$\boldsymbol{x}_{\mathrm{s},ij}$ |  location of $ij$ interface type
Greek:
$\delta$ |  one dimensional Dirac delta function
$\delta_{\text{S,}ij}$ |  surface delta function for $ij$ interface type
$\widehat{\Delta\sigma}$ |  sum of change in surface energies occurring over all dissipation events
$\delta_{\mathrm{S},ij}$ |  surface delta function for $ij$ interface type
$\mu$ |  viscosity
$\hat{\Phi}$ |  gravitational potential function
$\rho$ |  density
$\sigma$ |  energy per unit area (interfacial tension)
$\sigma_{ij}$ |  energy per unit area (interfacial tension) of $ij$ interface (between phases $i$ and $j$)
$\overrightarrow{\sigma}$ |  compound energy per unit area solid surface entering the CV from the right and under phase $\bigcirc$2
$\overleftarrow{\sigma}$ |  compound energy per unit area solid surface leaving the CV on the left and under phase $\bigcirc$1
$\sum_{i<j}$ |  sum taken over all interface types ($=\sum_{j=1}\sum_{i=1}^{j-1}$)
$\mathsf{T}_{\mathrm{M}}$ |  material stress tensor
$\mathsf{T}_{\mathrm{S}}$ |  interfacial stress tensor
$\tau_{\mathrm{cap}}$ |  total duration of dissipation events occurring during CV advance
$\theta_{\mathrm{a}}$ |  macroscopic advancing angle of phase $\bigcirc$1 in phase $\bigcirc$2
$\theta_{\mathrm{r}}$ |  macroscopic receding angle of phase $\bigcirc$1 in phase $\bigcirc$2
$\theta_{\mathrm{e}}$ |  equilibrium angle of phase $\bigcirc$1 in phase $\bigcirc$2

[Acknowledgements]An early version of the work was presented at the 82nd ACS
Colloid and Surface Science Symposium (Harvie et al., 2008). The author
acknowledges conversations and collaborations with many people over this
timeframe regarding the application of this theory, including with Prof Franz
Grieser, Prof Xuehua Zhang, A/Prof Brigitte Stadler, Prof William Ducker, Prof
David Dunstan, Prof Paul Mulvaney, Dr Annette Haebich, Dr Srinivas Mettu,
A/Prof Hong Zhao, Ms Mary Jane, Mr Pawan Kumar and Prof Suman Chakraborty.
Pawan Kumar is currently completing a PhD at the University of Melbourne on
the application of this theory, supported by a Melbourne-India Postgraduate
Program scholarship.

[Funding]The majority of this theory was developed in 2007 while the author
was between employment contracts. The work formed the basis for several
unsuccessful Australian Research Council Discovery Project applications
between the years of 2008 and 2017 (DP0881399, DP0988840, DP150104699,
DP160103697 and DP170104610). The author gratefully acknowledges the support
of an Melbourne Institute of Materials Interdisciplinary Seed-Funding Scheme
from the University of Melbourne in 2010 that was used to generate
experimental results to support this theory.

[Declaration of interests]The authors report no conflict of interest.

[Author ORCIDs]D.J.E. Harvie, https://orcid.org/0000-0002-8501-1344

##  Appendix A Appendix: Properties of the surface delta function
($\delta_{\mathrm{S}}$)

In this appendix we derive three properties of the surface delta function that
are used in Section 2 in the derivation of the multiphase mechanical energy
balance.

###  A.1 Volume integral of surface delta function

We start with some definitions: For an interface that lies between a specific
combination of immiscible materials the surface delta function is given by
(Lafaurie et al., 1994)

$\delta_{\mathrm{S}}(\boldsymbol{x}_{\mathrm{}})=\delta(q_{1}),$ (81)

where $q_{1}$ is the distance between $\boldsymbol{x}_{\mathrm{}}$ and the
closest point on the interface surface, and $\delta$ is the one dimensional
Dirac delta function. As $\delta=0$ when $q_{1}\neq 0$, equation (81) shows
that $\delta_{\mathrm{S}}$ is nonzero only on the interface surface.

For the analysis that follows we require a more specific relationship between
$\boldsymbol{x}_{\mathrm{}}$ and $q_{1}$ that is consistent with equation
(81): Given that $q_{1}=0$ defines the interface surface, we specify the
location of any point $\boldsymbol{x}_{\mathrm{}}$ that is near the surface
via a series of coordinates $\boldsymbol{q}_{\mathrm{}}=(q_{1},q_{2},q_{3})$
such that

$\boldsymbol{x}_{\mathrm{}}(\boldsymbol{q}_{\mathrm{}})=\boldsymbol{x}_{\mathrm{S}}(q_{2},q_{3})+q_{1}\boldsymbol{n}_{\mathrm{S}}(q_{2},q_{3}).$
(82)

Here $q_{2}$ and $q_{3}$ are a pair of convected surface coordinates (Aris,
1962) that uniquely locate a material particle at
$\boldsymbol{x}_{\mathrm{S}}$ that lies on the interface surface, and
$\boldsymbol{n}_{\mathrm{S}}$ is a unit normal to the surface at
$\boldsymbol{x}_{\mathrm{S}}$ which is (consistently) directed into one of the
phases. As $\boldsymbol{x}_{\mathrm{S}}$ moves with the material, when $q_{2}$
and $q_{3}$ are held constant we have

$\frac{d\boldsymbol{x}_{\mathrm{S}}}{dt}=\boldsymbol{v}_{\mathrm{}}$ (83)

where $\boldsymbol{v}_{\mathrm{}}$ is the local material velocity.

Lines of constant $q_{2}$ and $q_{3}$ define the surface coordinate lines. As
the surface coordinates move with the material, the coordinate lines will not
in general be orthogonal to each other, even if they are initially. Provided
that all material strain rates remain finite however, they will not become
coincident: This is important as it means that provided
$\boldsymbol{x}_{\mathrm{S}}$ can be defined uniquely in terms of $q_{2}$ and
$q_{3}$ at some particular time, a unique relationship between
$\boldsymbol{x}_{\mathrm{S}}$ and ($q_{2}$,$q_{3}$) will be realisable for all
time (on a smooth and continuous surface).

To derive equations (3) and (7) we utilise a volume $V$ that contains two
immiscible materials that are separated by such a single smooth and continuous
interface — surface $S$ (see figure 7). The surface completely spans $V$ such
that the circumference of $S$ occurs along the boundary of $V$. As the
velocity within $S$ is equal to that of the material (by equation (83)), the
boundary location of $S$ is constant for all time when expressed in terms of
the convected surface coordinates $q_{2}$ and $q_{3}$. Formally $V$ is
constructed by projecting both above and below $S$ in the direction of
$\boldsymbol{n}_{\mathrm{s}}$ a distance $\epsilon$. Hence within $V$,
$-\epsilon<q_{1}<\epsilon$, and provided that $\epsilon$ is small enough, the
relationship between $\boldsymbol{x}_{\mathrm{}}$ and
$\boldsymbol{q}_{\mathrm{}}$ expressed by equation (82) will be unique.

$\epsilon$$V$$\boldsymbol{x}_{\mathrm{S}}(q_{2},q_{3})$$\boldsymbol{x}_{\mathrm{}}(\boldsymbol{q}_{\mathrm{}})$$\boldsymbol{\hat{a}}_{\mathrm{1}}=\boldsymbol{n}_{\mathrm{S}}$$q_{1}$$\boldsymbol{\hat{a}}_{\mathrm{3}}$$\boldsymbol{\hat{a}}_{\mathrm{2}}$$\boldsymbol{\hat{a}}_{\mathrm{4}}$$S$$\epsilon$

Figure 7: The smooth interface surface $S$ separates two immiscible materials
within a volume $V$. Expressed in terms of the convected surface coordinates
$(q_{2},q_{3})$, $S$ is constant for all time.

To derive equation (3) we integrate the surface delta function over $V$ and
express the integral in terms of the new coordinate system
$\boldsymbol{q}_{\mathrm{}}$:

$\int_{V}\delta_{\mathrm{S}}(\boldsymbol{x}_{\mathrm{}})d\boldsymbol{x}_{\mathrm{}}=\int_{V}\delta(q_{1})\left[\frac{\partial\boldsymbol{x}_{\mathrm{}}}{\partial
q_{1}}\cdot\left(\frac{\partial\boldsymbol{x}_{\mathrm{}}}{\partial
q_{2}}\times\frac{\partial\boldsymbol{x}_{\mathrm{}}}{\partial
q_{3}}\right)\right]d\boldsymbol{q}_{\mathrm{}}.$ (84)

As the Dirac delta function $\delta(q_{1})$ is nonzero only for $q_{1}=0$, the
Jacobian in this equation (the term in square brackets) need only be evaluated
for $q_{1}=0$. Hence, utilising equation (82) to evaluate the partial
derivatives equation (84) becomes

$\int_{V}\delta_{\mathrm{S}}(\boldsymbol{x}_{\mathrm{}})d\boldsymbol{x}_{\mathrm{}}=\int_{-\epsilon}^{\epsilon}\delta(q_{1})dq_{1}\int_{S}\boldsymbol{n}_{\mathrm{S}}\cdot\left(\boldsymbol{a}_{2}\times\boldsymbol{a}_{3}\right)dq_{2}dq_{3}$
(85)

where

$\boldsymbol{a}_{2}=\left.\frac{\partial\boldsymbol{x}_{\mathrm{}}}{\partial
q_{2}}\right|_{q_{1}=0}=\frac{\partial\boldsymbol{x}_{\mathrm{S}}}{\partial
q_{2}}\quad\text{and}\quad\boldsymbol{a}_{3}=\left.\frac{\partial\boldsymbol{x}_{\mathrm{}}}{\partial
q_{3}}\right|_{q_{1}=0}=\frac{\partial\boldsymbol{x}_{\mathrm{S}}}{\partial
q_{3}}$ (86)

are two non-conincident vectors that are tangential to $S$ at
$\boldsymbol{x}_{\mathrm{S}}$. By definition of the Dirac delta function the
first integral on the right of this equation (85) is equal to one (as
$\epsilon$ is a small positive number). For the second integral we note that
the vectors $\boldsymbol{a}_{2}$ and $\boldsymbol{a}_{3}$ are both orthogonal
to $\boldsymbol{n}_{\mathrm{S}}$, and hence without loss of generality we
define

$\boldsymbol{n}_{\mathrm{S}}=\frac{\boldsymbol{a}_{2}\times\boldsymbol{a}_{3}}{\lvert\boldsymbol{a}_{2}\times\boldsymbol{a}_{3}\rvert}.$
(87)

This allows the volume integral of $\delta_{\mathrm{S}}$ to be written as

$\int_{V}\delta_{\mathrm{S}}(\boldsymbol{x}_{\mathrm{}})d\boldsymbol{x}_{\mathrm{}}=\int_{S}\left|\frac{\partial\boldsymbol{x}_{\mathrm{S}}}{\partial
q_{2}}\times\frac{\partial\boldsymbol{x}_{\mathrm{S}}}{\partial
q_{3}}\right|dq_{2}dq_{3}=\int_{S}dS=A$ (88)

where $A$ is the area of surface $S$ (Kreyszig, 2006, p454, §10.6). Noting
that any volume containing arbitrary surfaces can be composed of volumes that
contain smooth and continuous surfaces and volumes that contain no surface (in
which equation (88) is trivially satisfied), equation (3) from the main text
results.

###  A.2 Surface delta function transport equation

To derive equation (7) we take the derivative of equation (88) with respect to
time. Recognising that the boundary location of $S$ is constant in terms of
$q_{2}$ and $q_{3}$, the time derivative commutes into the integral giving

$\frac{d}{dt}\int_{V}\delta_{\mathrm{S}}(\boldsymbol{x}_{\mathrm{}})d\boldsymbol{x}_{\mathrm{}}=\int_{S}\frac{d}{dt}\Bigl{[}\lvert\boldsymbol{a}_{2}\times\boldsymbol{a}_{3}\rvert\Bigr{]}dq_{2}dq_{3}.$
(89)

Performing the differentiation and (re)introducing the Dirac delta function
leads to

$\displaystyle\frac{d}{dt}\int_{V}\delta_{\mathrm{S}}(\boldsymbol{x}_{\mathrm{}})d\boldsymbol{x}_{\mathrm{}}$
$\displaystyle=\int_{-\epsilon}^{\epsilon}\delta(q_{1})dq_{1}\int_{S}\mathcal{A}(\boldsymbol{q}_{\mathrm{}})\lvert\boldsymbol{a}_{2}\times\boldsymbol{a}_{3}\rvert
dq_{2}dq_{3}$ (90)
$\displaystyle=\int_{V}\delta(q_{1})\mathcal{A}(\boldsymbol{q}_{\mathrm{}})\left[\frac{\partial\boldsymbol{x}_{\mathrm{}}}{\partial
q_{1}}\cdot\left(\frac{\partial\boldsymbol{x}_{\mathrm{}}}{\partial
q_{2}}\times\frac{\partial\boldsymbol{x}_{\mathrm{}}}{\partial
q_{3}}\right)\right]d\boldsymbol{q}_{\mathrm{}}$
$\displaystyle=\int_{V}\delta_{\mathrm{S}}(\boldsymbol{x}_{\mathrm{}})\mathcal{A}(\boldsymbol{x}_{\mathrm{}})d\boldsymbol{x}_{\mathrm{}}$
(91)

where

$\mathcal{A}=\frac{\left(\boldsymbol{a}_{2}\times\boldsymbol{a}_{3}\right)}{\lvert\boldsymbol{a}_{2}\times\boldsymbol{a}_{3}\rvert^{2}}\cdot\left[\frac{d\boldsymbol{a}_{2}}{dt}\times\boldsymbol{a}_{3}+\boldsymbol{a}_{2}\times\frac{d\boldsymbol{a}_{3}}{dt}\right]$
(92)

is a function of the local surface geometry.

To simplify the expression for $\mathcal{A}$ we recall from equation (85) that
the surface integral in equation (90) which contains $\mathcal{A}$ is
evalulated under the condition $q_{1}=0$. Assuming this condition we combine
equations (83) and (86) to derive the identity,

$\frac{d\boldsymbol{a}_{2}}{dt}=\frac{d}{dt}\left(\frac{\partial\boldsymbol{x}_{\mathrm{S}}}{\partial
q_{2}}\right)=\frac{\partial}{\partial
q_{2}}\left(\frac{d\boldsymbol{x}_{\mathrm{S}}}{dt}\right)=\frac{\partial\boldsymbol{v}_{\mathrm{}}}{\partial
q_{2}}=\frac{\partial\boldsymbol{x}_{\mathrm{}}}{\partial
q_{2}}\cdot\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}=\frac{\partial\boldsymbol{x}_{\mathrm{S}}}{\partial
q_{2}}\cdot\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}=\boldsymbol{a}_{2}\cdot\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}.$
(93)

Substituting this and an analygous result for ${d\boldsymbol{a}_{3}}/{dt}$
into equation (92) yields

$\mathcal{A}=\beta^{2}\left(\boldsymbol{\hat{a}}_{2}\times\boldsymbol{\hat{a}}_{3}\right)\cdot\bigl{[}{\left(\boldsymbol{\hat{a}}_{2}\cdot\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}\right)\times\boldsymbol{\hat{a}}_{3}+\boldsymbol{\hat{a}}_{2}\times\left(\boldsymbol{\hat{a}}_{3}\cdot\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}\right)}\bigr{]}$
(94)

where

$\boldsymbol{\hat{a}}_{2}=\frac{\boldsymbol{a}_{2}}{\lvert\boldsymbol{a}_{2}\rvert}\quad\text{and}\quad\boldsymbol{\hat{a}}_{3}=\frac{\boldsymbol{a}_{3}}{\lvert\boldsymbol{a}_{3}\rvert}$
(95)

are unit vectors in each of the two surface coordinate directions and
$\beta=\lvert\boldsymbol{\hat{a}}_{2}\times\boldsymbol{\hat{a}}_{3}\rvert^{-1}$.

We now define a forth unit vector $\boldsymbol{\hat{a}}_{4}$ which is
coplannar with $\boldsymbol{\hat{a}}_{2}$ and $\boldsymbol{\hat{a}}_{3}$ (and
hence tangential to $S$) such that the vectors
$(\boldsymbol{n}_{\mathrm{S}},\boldsymbol{\hat{a}}_{2},\boldsymbol{\hat{a}}_{4})$
form a right handed coordinate system at $\boldsymbol{x}_{\mathrm{S}}$ (as
shown in figure 7). As the orientation of $\boldsymbol{\hat{a}}_{4}$ obeys
$\boldsymbol{n}_{\mathrm{S}}=\boldsymbol{\hat{a}}_{2}\times\boldsymbol{\hat{a}}_{4}$,
the new vector can be expressed as

$\boldsymbol{\hat{a}}_{4}=\alpha\boldsymbol{\hat{a}}_{2}+\beta\boldsymbol{\hat{a}}_{3},\quad\text{or}\quad\boldsymbol{\hat{a}}_{3}=\frac{1}{\beta}\left(\boldsymbol{\hat{a}}_{4}-\alpha\boldsymbol{\hat{a}}_{2}\right),$
(96)

where $\alpha$ is a finite scalar and equation (87) has been used.
Substituting equation (96) into equation (94) yields after some simplification

$\mathcal{A}=\left(\boldsymbol{\hat{a}}_{2}\times\boldsymbol{\hat{a}}_{4}\right)\cdot\bigl{[}{\left(\boldsymbol{\hat{a}}_{2}\cdot\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}\right)\times\boldsymbol{\hat{a}}_{4}+\boldsymbol{\hat{a}}_{2}\times\left(\boldsymbol{\hat{a}}_{4}\cdot\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}\right)}\bigr{]}.$
(97)

Employing the identity (Bird et al., 2002, p814, §A.2)

$[\boldsymbol{u}_{\mathrm{}}\times\boldsymbol{v}_{\mathrm{}}]\cdot[\boldsymbol{w}_{\mathrm{}}\times\boldsymbol{z}_{\mathrm{}}]=(\boldsymbol{u}_{\mathrm{}}\cdot\boldsymbol{w}_{\mathrm{}})(\boldsymbol{v}_{\mathrm{}}\cdot\boldsymbol{z}_{\mathrm{}})-(\boldsymbol{u}_{\mathrm{}}\cdot\boldsymbol{z}_{\mathrm{}})(\boldsymbol{v}_{\mathrm{}}\cdot\boldsymbol{w}_{\mathrm{}})$
(98)

and noting that $\boldsymbol{\hat{a}}_{2}\cdot\boldsymbol{\hat{a}}_{4}=0$
leads to

$\mathcal{A}=\boldsymbol{\hat{a}}_{2}\boldsymbol{\hat{a}}_{2}:\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}+\boldsymbol{\hat{a}}_{4}\boldsymbol{\hat{a}}_{4}:\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}=(\mathsf{I}_{\mathrm{}}-\boldsymbol{n}_{\mathrm{S}}\boldsymbol{n}_{\mathrm{S}}):\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}$
(99)

where $\mathsf{I}_{\mathrm{}}$ is the unit tensor.

With $\mathcal{A}$ defined we return to the development of equation (91):
Noting that for $\delta_{\mathrm{S}}\neq 0$ the boundary of $V$ moves at the
local material velocity $\boldsymbol{v}_{\mathrm{}}$, the left hand side of
this equation can be expanded using the Leibnitz formula for differentiating a
volume integral and the Gauss-Ostrogradskii divergence theorem. On the right
hand side we substitute $\mathcal{A}$ from equation (99). These operations
yield

$\int_{V}\frac{\partial\delta_{\mathrm{S}}}{\partial
t}d\boldsymbol{x}_{\mathrm{}}+\int_{V}\boldsymbol{\nabla}_{\mathrm{}}\cdot\left(\delta_{\mathrm{S}}\boldsymbol{v}_{\mathrm{}}\right)d\boldsymbol{x}_{\mathrm{}}=\int_{V}\delta_{\mathrm{S}}(\mathsf{I}_{\mathrm{}}-\boldsymbol{n}_{\mathrm{S}}\boldsymbol{n}_{\mathrm{S}}):\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}d\boldsymbol{x}_{\mathrm{}}.$
(100)

Equation (100) is valid for a specific volume geometry that contains a single
smooth and continuous interface surface. However, equation (100), like
equation (88), is trivially satisfied in volumes that contain no interface
surface. As any volume can be composed of volumes that contain a smooth and
continuous interface surface and volumes that contain no interface surface,
equation (100) must hold for any arbitrary volume. Hence, equation (100) must
hold at all locations and equation (7) from the main text results.

James & Lowengrub (2004), based on the work of Batchelor (1967, p132), give an
equation that is similar to equation (7) but expressed using non-equivalent
terms: The equation of James & Lowengrub (2004) concerns the material
derivative of an infinitesimal surface element, whereas equation (7) concerns
the material derivative of the surface delta function.

###  A.3 Surface integral of surface delta function

In order to apply the mechanical energy balance of equation (10) we also need
to be able to evaluate the integral of the surface delta function over a
surface. In the following we derive equation (4) that is given in the main
text for this purpose.

The derivation is based on a small amount of interfacial area, $\Delta A$,
that has a length of $\Delta l$, and that is contained within a small element
of volume $\Delta V$. The volume element is thin in the direction of
$\boldsymbol{n}_{\mathrm{}}$, having a thickness of $\epsilon$ in this
direction, so that $\Delta V=\epsilon\Delta S$. $\Delta S$ is a small part of
a larger surface, $S$. Figures 8a and 8b describe this geometrical system
using a projection and cross-section, respectively.

$\delta_{\mathrm{S}}$
---
$\boldsymbol{n}_{\mathrm{}}$
---
$\boldsymbol{n}_{\mathrm{S}}$
---
$\Delta l$
---
$\Delta V$
---
$\Delta S$
---
$\epsilon$
---
$\Delta A$
---
(a)

$\delta_{\mathrm{S}}$
---
$\boldsymbol{n}_{\mathrm{}}$
---
$\boldsymbol{n}_{\mathrm{S}}$
---
$l$
---
$\Delta V$
---
$\Delta S$
---
$\epsilon$
---
$\Delta A$
---
(b)

Figure 8: The two-dimensional integral of the delta function

The starting point for the analysis is the definition of the surface delta
function, that is equation (3), applied to the small volume $\Delta V$. This
gives

$\Delta A=\int_{\Delta
V}\delta_{\mathrm{S}}dV=\int_{-\epsilon/2}^{\epsilon/2}\int_{\Delta
S}\delta_{\mathrm{S}}dSdx_{\boldsymbol{n}_{\mathrm{}}}\approx\epsilon\int_{\Delta
S}\delta_{\mathrm{S}}dS$ (101)

where $x_{\boldsymbol{n}_{\mathrm{}}}$ is a coordinate in the direction of
$\boldsymbol{n}_{\mathrm{}}$ centred on $\Delta A$. The last equality holds
true for small $\epsilon$. Meanwhile for small $\Delta A$
$\boldsymbol{n}_{\mathrm{S}}$ is approximately uniform over $\Delta A$ and the
geometry of the intersecting surfaces gives (see Figure 8b)

$\Delta A=\frac{\Delta
l\epsilon}{\sqrt{1-(\boldsymbol{n}_{\mathrm{S}}\cdot\boldsymbol{n}_{\mathrm{}})^{2}}}$
(102)

Equating equations (101) and (102) leads to

$\int_{\Delta S}\delta_{\mathrm{S}}dS=\frac{\Delta
l}{\sqrt{1-(\boldsymbol{n}_{\mathrm{S}}\cdot\boldsymbol{n}_{\mathrm{}})^{2}}}$
(103)

Finally, recognising that $\Delta S$ is a small section of a larger surface
$S$, and that $\Delta l$ is a small section of the entire intersection between
$S$ and $\delta_{\mathrm{S}}$ that has a length of $l$, equation (103) can be
generalised in the limit of $\Delta l\rightarrow 0$ to

$\int_{S}\delta_{\mathrm{S}}dS=\int_{0}^{l}\frac{dl^{\prime}}{\sqrt{1-(\boldsymbol{n}_{\mathrm{S}}\cdot\boldsymbol{n}_{\mathrm{}})^{2}}}$
(104)

where $l^{\prime}$ is a pathlength parameterisation of the curve that is
defined as the intersection between $S$ and $\delta_{\mathrm{S}}$, and
$\boldsymbol{n}_{\mathrm{S}}\cdot\boldsymbol{n}_{\mathrm{}}$ is a function of
$l^{\prime}$. Equation (104) is the most general form of this surface delta
identity, however by assuming that
$\boldsymbol{n}_{\mathrm{S}}\cdot\boldsymbol{n}_{\mathrm{}}$ is independent of
the pathlength $l^{\prime}$ equation (4) from the main text results.

##  Appendix B Appendix: Dissipation event order of magnitude energy analysis

As discussed in the main text, the objective of this section is to quantify in
an order of magnitude sense what energy terms are significant during the
dissipation events so that we can determine the ultimate destination for the
specific surface energy $\widehat{\Delta\sigma}$ that is liberated from the
dissipation events.

Summing the individual dissipation event energy balances expressed by equation
(34) over all $N$ dissipation events and substituting the order of magnitude
of $\sum_{k=1}^{N}\widehat{T}_{0,k}$ from equation (41) gives

$\mathcal{O}\left(\rho
v_{\mathrm{cv}}^{2}h_{\mathrm{rough}}\right)+\mathcal{O}\left(\rho
gh_{\mathrm{rough}}^{2}\right)+\widehat{\Delta\sigma}=\sum_{k=1}^{N}\widehat{T}_{0,k}=\sum_{i=1}^{6}\sum_{k=1}^{N}\widehat{T}_{i,k}$
(105)

We now examine each of the six $\sum_{k=1}^{N}\widehat{T}_{i,k}$ terms on the
RHS of this equation.

For $\widehat{T}_{1,k}$, as for the equilibrium analysis
$\boldsymbol{n}_{\mathrm{cv}}\cdot\boldsymbol{v}_{\mathrm{cv}}$ on
$S_{\mathrm{cv,end}}$, so the only contribution to this term comes from
$S_{\mathrm{cv,cir}}$. However on the circumference of the CV the interface is
not affected by the dissipation event (as it is
$\mathcal{O}\left(r_{\mathrm{cv}}\right)$ away from
$\boldsymbol{x}_{\mathrm{cap}}$) so that the value of the surface integral is
independent of whether the time is within an equilibrium or dissipation stage.
Noting then that $\sum_{k=1}^{N}\int_{{\widehat{\Delta
t}}_{k}}=\tau_{\mathrm{cap}}$ and $\sum_{k=1}^{N+1}\int_{\widetilde{\Delta
t}_{k}}=\tau-\tau_{\mathrm{cap}}$ the first dissipation term can be written as

$\displaystyle\sum_{k=1}^{N}\widehat{T}_{1,k}$
$\displaystyle=\frac{1}{A_{\mathrm{cv}}}\sum_{k=1}^{N}\int_{{\widehat{\Delta
t}}_{k}}\int_{S_{\mathrm{cv}}}\sum_{i<j}\sigma_{ij}\delta_{\text{S},ij}\boldsymbol{n}_{\mathrm{cv}}\cdot\boldsymbol{v}_{\mathrm{cv}}dSdt$
$\displaystyle=\frac{\tau_{\mathrm{cap}}}{\tau-\tau_{\mathrm{cap}}}\sum_{k=1}^{N+1}\widetilde{T}_{1,k}=\mathcal{O}\left(\frac{\tau_{\mathrm{cap}}}{\tau}\sum_{k=1}^{N+1}\widetilde{T}_{1,k}\right)$
(106)

where equation (16) has been utilised.

The second term on the RHS of equation (105) is concerned with movement of
kinetic energy through the CV boundary during the dissipation periods. Using
the stationary solid model, this term is

$\sum_{k=1}^{N}\widehat{T}_{2,k}=\frac{1}{A_{\mathrm{cv}}}\sum_{k=1}^{N}\int_{{\widehat{\Delta
t}}_{k}}\int_{S_{\mathrm{cv,fluid}}}\boldsymbol{n}_{\mathrm{cv}}\cdot\frac{1}{2}\rho{|\widehat{\boldsymbol{v}_{\mathrm{}}}|}^{2}(\boldsymbol{v}_{\mathrm{cv}}-\widehat{\boldsymbol{v}_{\mathrm{}}})dSdt$
(107)

By recognising the separation of velocities $v_{\mathrm{cv}}\ll
v_{\mathrm{cap}}$ when using equation (68) in the above, for order of
magnitude purposes products of $v_{\mathrm{cv}}$ and $v_{\mathrm{cap}}$ can be
ignored and the above term written as a contribution from the continuous and
dissipation interface movements as

$\displaystyle\sum_{k=1}^{N}\widehat{T}_{2,k}$
$\displaystyle=\mathcal{O}\left(\frac{\tau_{\mathrm{cap}}}{\tau-\tau_{\mathrm{cap}}}\sum_{k=1}^{N+1}\widetilde{T}_{2,k}\right)+\mathcal{O}\left(\frac{\rho
v_{\mathrm{cap}}^{3}\tau_{\mathrm{cap}}h_{\mathrm{rough}}^{3}}{A_{\mathrm{cv}}l_{\mathrm{cv}}}\right)$
$\displaystyle=\mathcal{O}\left(\frac{\tau_{\mathrm{cap}}}{\tau}\sum_{k=1}^{N+1}\widetilde{T}_{2,k}\right)+\mathcal{O}\left(\frac{\rho
v_{\mathrm{cap}}^{2}h_{\mathrm{rough}}^{2}}{l_{\mathrm{cv}}}\right)$ (108)

In evaluating the dissipation event term on the first line of this equation
(the second on RHS), consistent with the dissipation fluid model of equation
(68) $\widehat{\boldsymbol{v}_{\mathrm{}}}$ is only significant within an area
of $\mathcal{O}\left(h_{\mathrm{rough}}^{2}\right)$ on each end of the CV, and
as the dissipation events are evenly distributed over the length
$l_{\mathrm{cv}}$ of the CV, only a proportion
$\mathcal{O}\left(h_{\mathrm{rough}}/l_{\mathrm{cv}}\right)$ of the
dissipation events contribute to this surface integral. Further, from equation
(16) we have used
$\tau_{\mathrm{cap}}v_{\mathrm{cap}}=\mathcal{O}\left(l_{\mathrm{cv}}X_{\mathrm{cv}}/h_{\mathrm{rough}}\right)$.

The third term on the RHS of equation (105) represents transport of
gravitational potential energy over the CV boundary and is evaluated in a very
similar fashion to the second term. Recognising as previously that energy
transport due to specific dissipation movements only occurs over the central
$\mathcal{O}\left(h_{\mathrm{rough}}^{2}\right)$ area of
$S_{\mathrm{cv,end}}$, and that during these events
$\hat{\Phi}=\mathcal{O}\left(gh_{\mathrm{rough}}\right)$, this term can be
evaluated as

$\displaystyle\sum_{k=1}^{N}\widehat{T}_{3,k}$
$\displaystyle=\frac{1}{A_{\mathrm{cv}}}\sum_{k=1}^{N}\int_{{\widehat{\Delta
t}}_{k}}\int_{S_{\mathrm{cv,fluid}}}\boldsymbol{n}_{\mathrm{cv}}\cdot\rho\hat{\Phi}(\boldsymbol{v}_{\mathrm{cv}}-\widehat{\boldsymbol{v}_{\mathrm{}}})dSdt$
$\displaystyle=\mathcal{O}\left(\frac{\tau_{\mathrm{cap}}}{\tau}\sum_{k=1}^{N+1}\widetilde{T}_{3,k}\right)+\mathcal{O}\left(\frac{\rho
gh_{\mathrm{rough}}^{3}}{l_{\mathrm{cv}}}\right)$ (109)

where assumptions consistent with those used for equation (108) have been
employed, and the first and second terms on the RHS of the second line of this
equation represent contributions from the continuous and dissipation
velocities occurring during the dissipation periods, respectively.

Term $\sum_{k=1}^{N}\widehat{T}_{4,k}$, like
$\sum_{k=1}^{N}\widehat{T}_{1,k}$, is concerned with interfacial behaviour at
the CV boundary. Recognising that only the interface between the two fluids
can experience a non-zero velocity, and that dissipation interfacial movements
are again confined to an area of
$\mathcal{O}\left(h_{\mathrm{rough}}^{2}\right)$ on $S_{\mathrm{cv,end}}$ this
term can be written as

$\displaystyle\sum_{k=1}^{N}\widehat{T}_{4,k}$
$\displaystyle=-\frac{1}{A_{\mathrm{cv}}}\sum_{k=1}^{N}\int_{{\widehat{\Delta
t}}_{k}}\int_{S_{\mathrm{cv,fluid}}}\sigma_{12}\delta_{\text{S,}12}\boldsymbol{n}_{\text{S,}12}\boldsymbol{n}_{\text{S,}12}:\widehat{\boldsymbol{v}_{\mathrm{}}}\boldsymbol{n}_{\mathrm{cv}}dSdt$
$\displaystyle=\mathcal{O}\left(\frac{\tau_{\mathrm{cap}}}{\tau}\sum_{k=1}^{N+1}\widetilde{T}_{4,k}\right)+\mathcal{O}\left(\sigma\frac{h_{\mathrm{rough}}}{l_{\mathrm{cv}}}\right)$
(110)

where $\tau_{\mathrm{cap}}v_{\mathrm{cap}}$ has again been evaluated using
equation (16).

The final two terms from the RHS of equation (105) are concerned with material
stresses, specifically representing the boundary work and internal viscous
dissipation occurring on and within the CV, respectively. As per previously in
the absence of solid velocities these terms only have contributions from the
fluid regions. For the boundary work term we substitute the Newtonian stress
equation (14) and dissipation period velocity, velocity gradient and pressure
expressions of equations (68), (69) and (70), respectively, into equation (27)
applied over the dissipation periods, giving

$\displaystyle\sum_{k=1}^{N}\widehat{T}_{5,k}$
$\displaystyle=-\frac{1}{A_{\mathrm{cv}}}\sum_{k=1}^{N}\int_{{\widehat{\Delta
t}}_{k}}\int_{S_{\mathrm{cv,fluid}}}\widehat{p}\widehat{\boldsymbol{v}_{\mathrm{}}}\cdot\boldsymbol{n}_{\mathrm{cv}}dSdt$
$\displaystyle+\frac{1}{A_{\mathrm{cv}}}\sum_{k=1}^{N}\int_{{\widehat{\Delta
t}}_{k}}\int_{S_{\mathrm{cv,fluid}}}\mu\left[\widehat{\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}}+(\widehat{\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}})^{T}\right]:\widehat{\boldsymbol{v}_{\mathrm{}}}\boldsymbol{n}_{\mathrm{cv}}dSdt$
$\displaystyle=\mathcal{O}\left(\frac{\tau_{\mathrm{cap}}}{\tau-\tau_{\mathrm{cap}}}\sum_{k=1}^{N+1}\widetilde{T}_{5,k}\right)+\mathcal{O}\left(\frac{v_{\mathrm{cap}}\tau_{\mathrm{cap}}h_{\mathrm{rough}}}{A_{\mathrm{cv}}l_{\mathrm{cv}}}\right.$
$\displaystyle\left.\int_{S_{\mathrm{cv,end}},r<h_{\mathrm{rough}}}\left[\frac{\mu
v_{\mathrm{cap}}}{\max(r,h_{\mathrm{mol}})}+\rho
v_{\mathrm{cap}}^{2}+\frac{\sigma}{h_{\mathrm{rough}}}\right]dS\right)$ (111)

The final integral in this equation is taken over the ends of the CV that are
within $\mathcal{O}\left(h_{\mathrm{rough}}\right)$ of its centreline, with
$r$ representing the distance to the centreline. Evaluating this integral
leads to

$\displaystyle\sum_{k=1}^{N}\widehat{T}_{5,k}$
$\displaystyle=\mathcal{O}\left(\frac{\tau_{\mathrm{cap}}}{\tau}\sum_{k=1}^{N+1}\widetilde{T}_{5,k}\right)$
$\displaystyle+\mathcal{O}\left(\rho
v_{\mathrm{cap}}^{2}\frac{h_{\mathrm{rough}}^{2}}{l_{\mathrm{cv}}}\right)+\mathcal{O}\left(\mu
v_{\mathrm{cap}}\frac{h_{\mathrm{rough}}}{l_{\mathrm{cv}}}\right)+\mathcal{O}\left(\sigma\frac{h_{\mathrm{rough}}}{l_{\mathrm{cv}}}\right)$
(112)

The viscous dissipation term $\sum_{k=1}^{N}\widehat{T}_{6,k}$ is evaluated
similarly. Noting that velocities within the solid are zero, and that the
fluid is incompressible, as for the corresponding equilibrium term analysis
there is no contribution from the pressure term $\widehat{p}$ and the
dissipation velocity gradient can be integrated over a region of volume
$\mathcal{O}\left(h_{\mathrm{rough}}^{3}\right)$ within the fluid, giving

$\displaystyle\sum_{k=1}^{N}\widehat{T}_{6,k}$
$\displaystyle=-\frac{1}{A_{\mathrm{cv}}}\sum_{k=1}^{N}\int_{{\widehat{\Delta
t}}_{k}}\int_{V_{\mathrm{cv,fluid}}}\mu\left[\widehat{\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}}+(\widehat{\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}})^{T}\right]:\widehat{\boldsymbol{\nabla}_{\mathrm{}}\boldsymbol{v}_{\mathrm{}}}dVdt$
$\displaystyle=\mathcal{O}\left(\frac{\tau_{\mathrm{cap}}}{\tau}\sum_{k=1}^{N+1}\widetilde{T}_{6,k}\right)+\mathcal{O}\left[\mu
v_{\mathrm{cap}}\ln\left(\frac{h_{\mathrm{rough}}}{h_{\mathrm{mol}}}\right)\right]$
(113)

Equation (71) in the main text results from substituting equations (106),
(108), (109), (110), (112) and (113) into equation (105).

## References

  * Aris (1962) Aris, Rutherford 1962 Vectors, tensors, and the basic equations of fluid mechanics. Dover.
  * Batchelor (1967) Batchelor, G. K. 1967 An introduction to fluid dynamics. Cambridge University Press.
  * Bico et al. (2002) Bico, José, Thiele, Uwe & Quéré, David 2002 Wetting of textured surfaces. Colloids and Surfaces 206, 41–46.
  * Bird et al. (2002) Bird, R. Byron, Stewart, Warren E. & Lightfoot, Edwin N. 2002 Transport phenomena, 2nd edn. John Wiley and Sons.
  * Blake & Haynes (1969) Blake, T. D. & Haynes, J. M. 1969 Kinetics of liquidliquid displacement. Journal of colloid and interface science 30 (3), 421–423.
  * Boruvka & Neumann (1977) Boruvka, L. & Neumann, A. W. 1977 Generalization of the classical theory of capillarity. Journal of Chemical Physics 66 (12), 5464–5476.
  * Brackbill et al. (1992) Brackbill, J. U., Kothe, D. B. & Zemach, C. 1992 A continuum method for modelling surface tension. Journal of Computational Physics 100, 335–354.
  * Brandon et al. (2003) Brandon, Simon, Haimovich, Nir, Yeger, Einat & Marmur, Abraham 2003 Partial wetting of chemically patterned surfaces: The effect of drop size. J. Colloid Interface Sci. 263, 237–243.
  * Brochard-Wyart & Gennes (1992) Brochard-Wyart, F. & Gennes, P. G. de 1992 Dynamics of partial wetting. Advances in Colloid and Interface Science 39, 1–11.
  * Butt et al. (2017) Butt, Hans-Jürgen, Gao, Nan, Papadopoulos, Periklis, Steffen, Werner, Kappl, Michael & Berger, Rüdiger 2017 Energy Dissipation of Moving Drops on Superhydrophobic and Superoleophobic Surfaces. Langmuir 33 (1), 107–116.
  * Butt et al. (2022) Butt, Hans-Jürgen, Liu, Jie, Koynov, Kaloian, Straub, Benedikt, Hinduja, Chirag, Roismann, Ilia, Berger, Rüdiger, Li, Xiaomei, Vollmer, Doris, Steffen, Werner & Kappl, Michael 2022 Contact angle hysteresis. Current Opinion in Colloid & Interface Science 59, 101574.
  * Callies & Quéré (2005) Callies, Mathilde & Quéré, David 2005 On water repellency. Soft Matter 1, 55–61.
  * Cassie & Baxter (1944) Cassie, A. B. D. & Baxter, S. 1944 Wettability of porous surfaces. Transactions of the Faraday Society 40, 546–551.
  * Cox (1983) Cox, R. G. 1983 The spreading of a liquid on a rough solid surface. Journal of Fluid Mechanics 131, 1–26.
  * Dahler & Scriven (1961) Dahler, J. S. & Scriven, L. E. 1961 Angular momentum of continua. Nature 192, 36–37.
  * de Gennes (1985) de Gennes, P. G. 1985 Wetting: statics and dynamics. Reviews of Modern Physics 57 (3), 827–863.
  * Delmas et al. (2011) Delmas, Mathieu, Monthioux, Marc & Ondarçuhu, Thierry 2011 Contact Angle Hysteresis at the Nanometer Scale. Physical Review Letters 106 (13), 136102.
  * Dorrer & Rühe (2008) Dorrer, Christian & Rühe, Jürgen 2008 Drops on Microstructured Surfaces Coated with Hydrophilic Polymers: Wenzel’s Model and Beyond. Langmuir 24 (5), 1959–1964.
  * Eral et al. (2013) Eral, H. B., ’t Mannetje, D. J. C. M. & Oh, J. M. 2013 Contact angle hysteresis: a review of fundamentals and applications. Colloid and Polymer Science 291 (2), 247–260.
  * Extrand (2002) Extrand, C. W. 2002 Model for contact angles and hysteresis on rough and ultrphobic surfaces. Langmuir 18, 7991–7999.
  * Feng et al. (2002) Feng, Lin, Li, Shuhoung, Li, Yingshun, Li, Huanjun, Zhang, Lingjuan, Zhai, Jin, Song, Yanlin, Liu, Biqian, Jiang, Lei & Zhu, Daoben 2002 Super-hydrophobic surfaces: From natural to artifcial. Advanced Materials 14, 1857–1860.
  * Feng et al. (2004) Feng, Lin, Mai, Zhongyi Zhang Zhenhong, Ma, Yongmei, Liu, Biqian, Jiang, Lei & Zhu, Daoben 2004 A super-hydrophobic and super-oleophilic coating mesh film film for the separation of oil and water. Angew. Chem. Int. Ed. 43, 2012–2014.
  * Fetzer & Ralston (2011) Fetzer, Renate & Ralston, John 2011 Exploring defect height and angle on asymmetric contact line pinning. The Journal of Physical Chemistry C 115 (30), 14907–14913.
  * Forsberg et al. (2010) Forsberg, Pontus S. H., Priest, Craig, Brinkmann, Martin, Sedev, Rossen & Ralston, John 2010 Contact line pinning on microstructured surfaces for liquids in the wenzel state. Langmuir 26 (2), 860–865.
  * Gao & McCarthy (2007) Gao & McCarthy, Thomas J. 2007 Reply to “Comment on How Wenzel and Cassie Were Wrong by Gao and McCarthy”. Langmuir 23 (26), 13243–13243.
  * Gao & McCarthy (2007) Gao, Lichao & McCarthy, Thomas J. 2007 How Wenzel and Cassie Were Wrong. Langmuir 23 (7), 3762–3765.
  * Harvie et al. (2008) Harvie, Dalton J. E., Zhang, Xuehua, Stadler, Brigitte Maria & Ducker, William 2008 Contact angle hysteresis on sparsely structured surfaces. In 82nd ACS Colloid and Surface Science Symposium. Raleigh, NC, USA, presentation.
  * Huh & Scriven (1971) Huh, Chun & Scriven, L.E 1971 Hydrodynamic model of steady movement of a solid/liquid/fluid contact line. Journal of Colloid and Interface Science 35 (1), 85–101.
  * James & Lowengrub (2004) James, Ashley J. & Lowengrub, John 2004 A surfactant-conserving volume-of-fluid method for interfacial flows with insoluble surfactant. Journal of Computational Physics 201, 685–722.
  * Jiang et al. (2019) Jiang, Youhua, Xu, Wei, Sarshar, Mohammad Amin & Choi, Chang-Hwan 2019 Generalized models for advancing and receding contact angles of fakir droplets on pillared and pored surfaces. Journal of Colloid and Interface Science 552, 359–371.
  * Joanny & de Gennes (1984) Joanny, J. F. & de Gennes, P. G. 1984 A model for contact angle hysteresis. J. Phys. Chem. 81 (1), 552–562.
  * Johnson Jr & Dettre (1964) Johnson Jr, Rulon E. & Dettre, Robert H. 1964 I: Study of an idealized rough surface. Contact Angle, Wettability, and Adhesion: Advances in Chemistry Series 43, 112–135.
  * Kreyszig (2006) Kreyszig, Erwin 2006 Advanced engineering mathematics, 9th edn. John Wiley and Sons.
  * Kuiken (1995) Kuiken, Gerard D. C. 1995 The symmetry of the stress tensor. Industrial and Engineering Chemistry Research 34 (10), 3568–3572, arXiv: http://pubs.acs.org/doi/pdf/10.1021/ie00037a046.
  * Lafaurie et al. (1994) Lafaurie, Bruno, Nardone, Carlo, Scardovelli, Ruben, Zaleski, Stéphane & Zanetti, Gianluigi 1994 Modelling merging and fragmentation in multiphase flows with SURFER. Journal of Computational Physics 113, 134–147.
  * Li et al. (2016) Li, Dandan, Xue, Yahui, Lv, Pengyu, Huang, Shenglin, Lin, Hao & Duan, Huiling 2016 Receding dynamics of contact lines and size-dependent adhesion on microstructured hydrophobic surfaces. Soft Matter 12 (18), 4257–4265.
  * Li et al. (2013) Li, Kan, Ju, Jie, Xue, Zhongxin, Ma, Jie, Feng, Lin, Gao, Song & Jiang, Lei 2013 Structured cone arrays for continuous and effective collection of micron-sized oil droplets from water. Nature Communications 4.
  * Marmur (1997) Marmur, Abraham 1997 Line tension and the intrinsic contact angle in solid-liquid-fluid systems. J. Colloid Interface Sci. 186, 462–466.
  * Marmur (2006) Marmur, Abraham 2006 Soft contact: measurement and interpretation of contact angles. Soft Matter 2, 12–17.
  * Marmur (2022) Marmur, Abraham 2022 The Contact Angle Hysteresis Puzzle. Colloids and Interfaces 6 (3), 39.
  * McHale (2007) McHale, G. 2007 Cassie and wenzel: Were they really so wrong? Langmuir 23 (15), 8200–8205, publisher: American Chemical Society.
  * Moffatt (1964) Moffatt, H. K. 1964 Viscous and resistive eddies near a sharp corner. Journal of Fluid Mechanics 18 (1), 1–18.
  * Mohammad Karim (2022) Mohammad Karim, Alireza 2022 A review of physics of moving contact line dynamics models and its applications in interfacial science. Journal of Applied Physics 132 (8), 080701.
  * Nosonovsky (2007) Nosonovsky, Michael 2007 On the Range of Applicability of the Wenzel and Cassie Equations. Langmuir 23 (19), 9919–9920.
  * Öner & McCarthy (2000) Öner, Didem & McCarthy, Thomas J. 2000 Ultrahydrophobic surfaces, effects of topography length scales on wettability. Langmuir 16, 7777–7782.
  * Öpik (2000) Öpik, U. 2000 Contact-angle hysteresis caused by a random distribution of weak heterogeneities on a solid surface. Journal of Colloid and Interface Science 223 (2), 143 – 166.
  * Panchagnula & Vedantam (2007) Panchagnula, Mahesh V. & Vedantam, Srikanth 2007 Comment on How Wenzel and Cassie Were Wrong by Gao and McCarthy. Langmuir 23 (26), 13242–13242.
  * Patankar (2003) Patankar, Neelesh A. 2003 On the modelling of hydrophobic contact angles on rough surfaces. Langmuir 19, 1249–1253.
  * Petrov & Petrov (1992) Petrov, P. & Petrov, I. 1992 A combined molecular-hydrodynamic approach to wetting kinetics. Langmuir 8 (7), 1762–1767.
  * Pomeau & Vannimenus (1985) Pomeau, Y & Vannimenus, J 1985 Contact angle on heterogeneous surfaces: Weak heterogeneities. Journal of Colloid and Interface Science 104 (2), 477–488.
  * Pompe et al. (1999) Pompe, T., Fery, A. & Herminghaus, S. 1999 Measurement of contact line tension by analysis of the three-phase boundary with nanometer resolution. J. Adhesion Sci. Technol. 13 (10), 1155–1164.
  * Priest et al. (2009) Priest, Craig, Albrecht, Trent W. J., Sedev, Rossen & Ralston, John 2009 Asymmetric wetting hysteresis on hydrophobic microstructured surfaces. Langmuir 25 (10), 5655–5660.
  * Priest et al. (2007) Priest, Craig, Sedev, Rossen & Ralston, John 2007 Asymmetric wetting hysteresis on chemical defects. Physical Review Letters 99 (2), 026103.
  * Priest et al. (2013) Priest, Craig, Sedev, Rossen & Ralston, John 2013 A quantitative experimental study of wetting hysteresis on discrete and continuous chemical heterogeneities. Colloid and Polymer Science 291 (2), 271–277.
  * Raj et al. (2012) Raj, Rishi, Enright, Ryan, Zhu, Yangying, Adera, Solomon & Wang, Evelyn N. 2012 Unified Model for Contact Angle Hysteresis on Heterogeneous and Superhydrophobic Surfaces. Langmuir 28 (45), 15777–15788.
  * Robbins & Joanny (1987) Robbins, M. O. & Joanny, J. F. 1987 Contact Angle Hysteresis on Random Surfaces. EPL (Europhysics Letters) 3 (6), 729.
  * Song et al. (2022) Song, Qingrui, Liu, Kun, Sun, Wei, Jiao, Yunlong, Wang, Zhaochang, Liu, Xiaojun & Ye, Jiaxin 2022 Contact angle hysteresis and lateral adhesion strength on random rough surfaces. Physics of Fluids 34 (8), 082015.
  * Sun et al. (2005) Sun, T. L., Feng, L., Gao, X. F. & Jiang, L. 2005 Bioinspired surfaces with special wettability. Accounts of Chemical Research 38, 644–652.
  * Voinov (1976) Voinov, O.V. 1976 Hydrodynamics of wetting. Translated from Izvestiya Akademii Nauk SSSR (5), 76–84.
  * Wenzel (1936) Wenzel, Robert N. 1936 Resistance of solid surfaces to wetting by water. Industrial and Engineering Chemistry 28 (8), 988–994.
  * Wu et al. (2002) Wu, Yunying, Sugimura, Hiroyuki, Inoue, Yasushi & Takai, Osamu 2002 Thin films with nanotextures for transparent and ultra water-repellent coatings produced from trimethylmethoxysilane by microwave plasma CVD. Chem. Vap. Dep. 8 (2), 47–49.


# Outcome-based Evaluation of Systematic Review Automation

Wojciech Kusa<EMAIL_ADDRESS>0000-0003-4420-4147 TU
WienViennaAustria ,  Guido Zuccon<EMAIL_ADDRESS>0000-0003-0271-5563 The
University of QueenslandBrisbaneAustralia ,  Petr Knoth<EMAIL_ADDRESS>0000-0003-1161-7359 The Open UniversityMilton KeynesThe United Kingdom and
Allan Hanbury<EMAIL_ADDRESS>0000-0002-7149-5843 TU
WienViennaAustria

(2023)

###### Abstract.

Current methods of evaluating search strategies and automated citation
screening for systematic literature reviews typically rely on counting the
number of relevant publications (i.e. those to be included in the review) and
not relevant publications (i.e. those to be excluded). Significant importance
is put into promoting the retrieval of all relevant publications through great
attention to recall-oriented measures, and demoting the retrieval of non-
relevant publications through precision-oriented or cost metrics. This
established practice, however, does not accurately reflect the reality of
conducting a systematic review, because not all included publications have the
same influence on the final outcome of the systematic review. More
specifically, if an important publication gets excluded or included, this
might significantly change the overall review outcome, while not including or
excluding less influential studies may only have a limited impact. However, in
terms of evaluation measures, all inclusion and exclusion decisions are
treated equally and, therefore, failing to retrieve publications with little
to no impact on the review outcome leads to the same decrease in recall as
failing to retrieve crucial publications.

We propose a new evaluation framework that takes into account the impact of
the reported study on the overall systematic review outcome. We demonstrate
the framework by extracting review meta-analysis data and estimating outcome
effects using predictions from ranking runs on systematic reviews of
interventions from CLEF TAR 2019 shared task. We further measure how closely
the obtained outcomes are to the outcomes of the original review if the
arbitrary rankings were used. We evaluate 74 runs using the proposed framework
and compare the results with those obtained using standard IR measures. We
find that accounting for the difference in review outcomes leads to a
different assessment of the quality of a system than if traditional evaluation
measures were used. Our analysis provides new insights into the evaluation of
retrieval results in the context of systematic review automation, emphasising
the importance of assessing the usefulness of each document beyond binary
relevance.

systematic reviews, citation screening, evaluation, study outcomes, effect
based evaluation, information retrieval

††journalyear: 2023††copyright: rightsretained††conference: Proceedings of the
2023 ACM SIGIR International Conference on the Theory of Information
Retrieval; July 23, 2023; Taipei, Taiwan††booktitle: Proceedings of the 2023
ACM SIGIR International Conference on the Theory of Information Retrieval
(ICTIR ’23), July 23, 2023, Taipei, Taiwan††doi:
10.1145/3578337.3605135††isbn: 979-8-4007-0073-6/23/07††ccs: Information
systems Evaluation of retrieval results††ccs: Information systems Information
retrieval††ccs: Information systems Retrieval effectiveness††ccs: Information
systems Specialized information retrieval

##  1\. Introduction

A systematic literature review is a well-established and rigorous methodology
for synthesising and evaluating the evidence on a specific research question,
which is particularly important in the field of medicine (Jo et al., 2009).
However, it is also gaining importance in other areas such as social sciences
and engineering (Castillo and Grbovic, 2022; Petticrew and Roberts, 2008;
Keele et al., 2007; Bilotta et al., 2014). The process involves a systematic
search, critical appraisal, and synthesis of the available literature on a
topic. During the critical appraisal step, every included publication has its
weight and effect calculated based on the outcomes reported by that
publication. This information influences the final outcome of the review.

One of the essential steps in conducting a systematic review is the process of
citation screening, in which a large number of publications are initially
identified through a literature search and then screened to determine those
relevant to the review (Bannach-Brown et al., 2019; Tsafnat et al., 2018).
This process can be time-consuming and labour-intensive, involving making
thousands of eligibility decisions. Given the importance of citation screening
in systematic literature reviews, there have been numerous attempts to
automate the process (O’Mara-Eves et al., 2015). Previous studies have
investigated the use of automated citation screening methods for systematic
literature reviews by utilising various natural language processing (NLP),
machine learning (ML), and information retrieval (IR) methods to retrieve,
rank, or classify references (O’Mara-Eves et al., 2015; van Dinter et al.,
2021; Kusa et al., 2022; Howard et al., 2016; Scells et al., 2020b; Kanoulas
et al., 2019; Scells et al., 2020a, 2021; Wang et al., 2022, 2023b, 2023a).

To understand the effectiveness of automated citation screening methods,
practitioners have relied on metrics based on the notions of recall, precision
and cost – and of a binary assessment of relevance111Every publication to be
included in the review is labelled as relevant, while every excluded
publication is non-relevant. (Kusa et al., 2022; van Dinter et al., 2021;
O’Mara-Eves et al., 2015). This practice assigns to every publication to be
included in the review the same importance. So, for example, if method $M_{1}$
identifies as potentially relevant publications $\\{A,B,C\\}$ while method
$M_{2}$ identifies publications $\\{A,D,E\\}$, and the ground truth is that
the relevant publications are $\\{A,B,D\\}$, then $M_{1}$ and $M_{2}$ achieve
the same recall, precision and cost. However, we argue, that the two sets
$\\{A,B,C\\}$ and $\\{A,D,E\\}$ may not be equally important, and thus
identifying either of $B$ or $D$ may not be equivalent if the outcomes of the
review were considered. In fact, if excluded, some publications can
significantly change a review’s conclusion to the extent that the conclusion
might be the opposite (e.g., from favouring a drug to favouring a placebo)
(Nussbaumer-Streit et al., 2018, 2020). On the other hand, not including other
publications might have only a small quantitative impact on the outcomes of
the review.

We argue that a holistic evaluation of retrieval and automated citation
screening methods for systematic review creation should not only consider the
concepts of recall, precision and cost, but also the quality of the outcomes
generated from the analysis of the automatically included publications.
Following this direction, we propose a new evaluation framework that considers
inclusion and exclusion information and meta-analysis data from reviews
created by Cochrane – the largest organisation responsible for creating
systematic literature reviews in medicine,222https://www.cochrane.org to
estimate outcomes and weights of included publications. This information can
be used to assess the quality of ranking and classification methods. This
framework allows for assessing automatic approaches from the angle of how
closely their _outcomes_ – not just their set of included publications – are
to the outcomes of the original review. By comparing the outcomes of the
automated model to those of the original review, we can gain a better
understanding of the quality of the automated approach and its effect on the
final outcome of the review.

We propose five aspects of analysis focusing on different features of review
outcomes. We explore initial experiments on the CLEF TAR 2019 dataset
(Kanoulas et al., 2019). Our simulation results show that by randomly removing
one publication per review (average recall of 92% publications), 95% of
outcomes remain unchanged. However, after removing five publications (average
recall of 63%), 76% of the outcomes are still the same, showing that the
relationship between recall and achieved outcomes is not linear. We also show
that the outcome-based evaluation emphasises different aspects of the models’
performance than the traditional IR evaluation measures. We finally propose
multi-objective optimisation to handle the problem of non-estimable outcomes.

We believe this new evaluation approach will provide a better understanding of
the impact of automatic literature screening methods on the outcome of
systematic literature reviews and help identify areas in which these methods
can be improved.

##  2\. Related Work

The effectiveness of automatic approaches for search strategy creation and
systematic review screening has been traditionally evaluated using binary
relevance ratings (O’Mara-Eves et al., 2015; Kanoulas et al., 2019; van Dinter
et al., 2021), often sourced at the title and abstract screening level, rather
than at the full-text level.

When the screening problem is treated as a ranking task (e.g., for the sub-
task of screening prioritisation or stopping prediction), then rank-based
metrics and metrics at a fixed cut-off are commonly used, e.g., $nDCG@n$,
$Precision@n$, $Recall@n$, last relevant found (Scells et al., 2017; Howard et
al., 2020). Cost-based and economic-based metrics are also used, especially in
the context of the query formulation task in the CLEF TAR shared task
(Kanoulas et al., 2019, 2017, 2018), e.g., total cost (TC) or total cost with
a weighted penalty (TCW). The TREC Total Recall track (Grossman et al., 2016)
also used a cut-off based metric, $recall@aR+b$, which is defined as the
recall achieved when $aR+b$ documents have been identified, where $R$ is the
number of relevant documents in the collection and $a$ and $b$ are parameters.
When $a=1$ and $b=0$, $recall@aR+b$ is equivalent to R-precision. In the
patent domain, the PRES score has been proposed which takes into account
achieved recall and the user’s search effort (Magdy and Jones, 2010).

When the screening problem is treated as a classification task, metrics based
on the confusion matrix and the notion of Precision and Recall are commonly
used (van Dinter et al., 2021; O’Mara-Eves et al., 2015): aside from Precision
and Recall, metrics include variations of the harmonised mean between the two,
i.e. Fβ–score, utility, U19 (Wallace et al., 2010b, a, 2011), sensitivity-
maximising thresholds (Dalal et al., 2013), and AUC (Cohen et al., 2010).
Another metric, Work Saved over Sampling (WSS), measures the amount of work
saved when using machine learning models to screen irrelevant publications
(Cohen et al., 2006; Matwin et al., 2010; Kontonatsios et al., 2020; Kusa et
al., 2022). The True Negative Rate (TNR) and nreTNR (normalised rectified TNR)
were proposed as an alternative as it addresses some of the limitations of WSS
regarding averaging scores from multiple datasets (Kusa et al., 2023a, b).

Nussbaumer-Streit et al. (2018) compared repeated literature searches using 14
abbreviated approaches (combinations of various databases with and without
searches of reference lists) on a sample of 60 Cochrane systematic reviews of
clinical interventions. They re-calculated the main summary-of-findings table
of each Cochrane review and asked original review authors whether the
conclusions changed compared to the original review. They found that in only
2% of cases (95% CI: 0%–9%), combining one database with another or with
searches of reference lists was falsely reaching an opposite conclusion
compared to comprehensive searches. This outcome shows that identifying _all_
relevant studies is not always crucial for obtaining the same review results.

Automated citation screening has become increasingly popular in systematic
literature reviews due to its potential to reduce the time and cost required.
However, current evaluation methods for these methods are limited to binary
relevance assessment, where each publication is considered either relevant or
irrelevant, and do not account for the impact of each publication on the
review outcome. This is a vital issue, as the assumption that all relevant
publications are equally important to the final outcome of the systematic
review is not necessarily valid. Without an accurate assessment of the
importance of each document, the conclusions of a systematic review may be
biased or incomplete. To address this issue, in this paper, we propose a novel
methodology for assessing citation screening based on evaluating outcome
differences, which enables us to determine the impact of each publication on
the systematic review.

##  3\. Evaluation Framework

This paper proposes a new evaluation framework for automated citation
screening. Our framework includes three steps which are detailed in the
following subsections. The first step is data extraction, where we extract
statistics of the studies included in the review and match studies to
publications. The second step is model evaluation, where we use the extracted
data to estimate the review’s outcomes for rankings or classifications of the
citation list. The third step is the analysis of the results, where we compare
the outcomes obtained from the alternative rankings to the outcomes of the
original review. Our proposed framework allows for a more nuanced evaluation
of automated citation screening methods. By considering the impact of each
publication on the review’s outcomes, we can identify which publications are
most important to retrieve and prioritise them accordingly. Next we describe
each step in detail.

###  3.1. Data Extraction

Cochrane systematic reviews distinguish between _study_ and _publication_. A
study is a distinct piece of research conducted to answer a specific research
question or investigate a particular hypothesis. It typically involves a group
of participants, data collection methods, and specific objectives.
Publications, on the other hand, are the atomic units which reviewers screen.
Each study can be reported by several publications, such as journal articles,
conference proceedings, or research reports. Each publication may present
different aspects or findings of the same study, but they are all derived from
the same underlying research. We assume that a study has been found if at
least one publication reporting it was successfully retrieved.

For every review, based on its Cochrane review ID, we identify its
corresponding RevMan file and list of included publications. A RevMan file is
the format used by Cochrane containing all statistical data about studies and
outcomes included in the review. Outcomes of Cochrane reviews are reported in
the following hierarchy: one comparison can have several outcomes, and one
outcome can consist of a few subgroups. We extract all metadata from the
RevMan files, such as the comparisons, outcomes and subgroups and the results
of every included study. Note that the use of RevMan files is for experimental
convenience, but is not a requirement from the framework: the required data
could be provided in other formats. Furthermore, Cochrane recently announced
that future systematic literature reviews would contain statistical data in
more common csv and ris formats.333https://www.cochrane.org/news/cochrane-
improving-way-we-manage-and-share-data-associated-our-reviews

Cochrane reports a list of included publications and studies which correspond
to them. Traditionally, retrieval was conducted at the level of publications
(Kanoulas et al., 2017, 2018, 2019). In order to be able to re-use previous
relevance judgments, we need to assign PubMed IDs to these publications. Our
process for matching PubMed IDs to publications is based on four steps in the
following order:

  * •

We check if the PubMed ID information is provided on the Cochrane references
webpage.

  * •

We conduct search in PubMed using
Entrez444https://www.ncbi.nlm.nih.gov/search/ by searching for the same title
and authors.

  * •

We search for the PubMed ID in
SemanticScholar555https://www.semanticscholar.org using publication DOI from
Cochrane references webpage.

  * •

We search again in PubMed, this time with a relaxed requirement by searching
for an exact match in the title only.

###  3.2. Model Evaluation

When conducting a meta-analysis, for every outcome, each study has its weight
and effect size calculated first (respectively columns 6 and 7 on example
forest plots in Figure 1). Effect size is an essential statistical concept in
the analysis of research data (Higgins et al., 2019). It is a measure that
quantifies the magnitude of difference between two groups in a study.
Researchers use a variety of effect measures to compare outcome data between
two intervention groups, including odds ratios and mean differences.

For instance, in ratio effect measures, a value of 1 represents no difference
between the groups (Deeks et al., 2019; Deeks and Higgins, 2010). On the other
hand, in difference measures, a value of 0 represents no difference between
the groups. Values higher or lower than these “null” values may indicate
either benefit or harm of an experimental intervention, depending on the order
of the interventions in the comparison and the nature of the outcome. Every
estimate is expressed with a measure of uncertainty, such as a confidence
interval (CI) or standard error (SE).

Effects depend on the number of events reported by that study, whereas weights
assigned to each study are influenced by other studies included in this
outcome. So when removing one study from the meta-analysis, only the weights
of the remaining studies will change, but their effect sizes will stay the
same (compare Figures 1(a) and 1(c)). There are several types of outcomes
reported by Cochrane, in our study, we focus on the dichotomous and continuous
outcomes only and calculate them following the approach by Deeks and Higgins
(2010).

Our framework takes arbitrary ranking or classification runs and calculates
the final outcomes of the review based on publications included in the run.
When evaluating a classification run or a search result, we take all
publications predicted as relevant. When evaluating ranking runs, we need to
assume a cut-off point. Previous studies working on systematic review
automation used either cut-off at r% of recall (Cohen et al., 2006; Kusa et
al., 2023a), or at d% of total dataset size (Kanoulas et al., 2017, 2018).

###  3.3. Results Analysis

(a) Hypothetical review outcome with 5 included studies.

(b) Not including studies B, C, D and E still keep the review outcome
approximately the same (absolute difference: 0.02, relative difference:
0.0076).

(c) Not including study C will overestimate the review outcome, yet it will be
within the 95% CI range.

(d) Not including studies A, C, D and E will overestimate the review outcome,
and it will be above the 95% CI range of the original outcome.

(e) Not including studies A, B, C and D will change the study outcome – from
‘favours control’ to ‘favours experimental’.

(f) Not including any study makes the outcome non-estimable.

Figure 1. Different versions of review outcomes represented as forest plots.
Each row is a single study. Columns from the right represent, respectively:
(1) the study identifier, (2) number of events in the experimental group
(e.g., patients with specific symptoms or adverse events), (3) experimental
group size, (4) number of events in the control group, (5) control group size,
(6) the weight of a study, and (7) effect size of a study: a difference (e.g.,
risk ratio or standardised mean difference) in events between experimental or
control group. Simulations and figures done using RevMan Web, available at
http://revman.cochrane.org.

We analyse the results by examining the outcomes generated by the run and
compare them with the outcomes obtained by the original review (Figure 1). We
extend the analysis done by Nussbaumer-Streit et al. (2018), who proposed two
categories of “changed conclusions”: (1) if the new review drew the opposite
conclusion, (2) if it is not possible to draw a conclusion or the new
conclusion has less certainty. We distinguish five aspects of analysis for
review outcomes against the original review (Figure 1(a)). The first two of
these aspects are real-valued, whereas the remaining three are categorical:

  1. (1)

_Magnitude of difference_ — By how much are the outcomes different in their
effect size (Figure 1(a) versus 1(b))? In other words, what is the numerical
_impact_ on the review outcome when certain studies are not included? This is
measured by calculating the relative difference in effect size between the
original outcome $O_{o}$ and predicted outcome $O_{p}$:
$MoD=\frac{\|O_{o}-O_{p}\|}{\|O_{o}\|}$. When $O_{o}=0$ and $O_{p}\neq 0$, we
assume $MoD=100\%$; otherwise, when $O_{o}=O_{p}=0$, we set $MoD=0\%$.
Similarly, when the predicted outcome cannot be estimated, we assume
$MoD=100\%$.

  2. (2)

_Distance from CI_ — Is the new outcome within the Confidence Interval (CI) of
the original outcome (Figure 1(c))? The answer is a distance between the
predicted outcome $O_{p}$ and the closest of the pair
($CI_{lower},CI_{upper}$):

$\Delta_{CI}=\begin{cases}\|O_{p}-CI_{lower}\|&\text{if
$O_{p}<CI_{lower}$,}\\\ \|O_{p}-CI_{upper}\|&\text{if $O_{p}>CI_{upper}$,}\\\
0&\text{otherwise.}\end{cases}$

  3. (3)

_Overestimation/underestimation_ — Is the outcome overestimated or
underestimated compared to the original one (Figure 1(d))? We first check if
the calculated outcome is equal (due to the limits of precision of data
reported in RevMan files, we use the relative and absolute tolerance of
$10^{-5}$ and $10^{-6}$ respectively). Then, if the outcome is different, we
check if the result is higher than the original (overestimation) or lower
(underestimation). The answer has three options: “overestimated”,
“underestimated”, and “equal”.

  4. (4)

_Sign_ — Does the outcome have the same sign as the original one (Figure
1(e))? In other words, are the new conclusions opposite to the original ones?
The answer is binary: “yes”/“no”. This aspect corresponds to the first
category from Nussbaumer-Streit et al. (2018).

  5. (5)

_Estimability_ — Is it possible to calculate the outcome (Figure 1(f))? An
outcome cannot be calculated if there are no included studies concerning it.
The answer is binary: “yes”/“no”.

##  4\. Experiment Setup

Contrary to the traditional evaluation based on retrieving relevant
publications, with our framework we envision the evaluation in an outcome-
based approach. Specifically, we do not treat a dataset as a collection of
systematic reviews but rather a collection of outcomes. The problem of
conducting a systematic review is multi-dimensional. One can think of it as
having several outcomes reporting different dimensions of the review, and the
evaluation of the user’s needs is conducted independently from each outcome’s
perspective. We do not want to average across reviews, each containing a
different number of outcomes. We add or average these outcome-level results
instead.

Before we present the results, we first discuss the used dataset and models.

###  4.1. Dataset and Models

We used a collection of 38 systematic reviews of interventions from the CLEF
TAR 2019 training and test datasets (Kanoulas et al., 2019). Each review
consists of a Cochrane ID, a protocol, and a list of publications described by
their PubMedIDs with qrels both on the title and abstract level and a full-
text level. We enhanced the dataset by collecting RevMan files and information
about the data and analysis as described in Section 3.1.

Out of 38 reviews in CLEF TAR 2019, our script found studies and outcomes for
32 reviews (17 in the training subset and 15 in the test subset). We summarise
the statistics of the 32 reviews we consider in Table 1. There is a
significant discrepancy in the number of outcomes reported by the reviews,
ranging from as few as 2 or 3 outcomes in small reviews to 128 outcomes in the
largest one. Moreover, the majority of these outcomes come from just one or
two studies, which presents an additional challenge.

These 32 reviews report 1640 included publications, out of which we managed to
find PubMed IDs for 1175 of them (71.6%). Next, we wanted to match
publications identified with our script to the CLEF TAR 2019 qrels based on
the PubMed ID. There were, in total, 778 relevant documents on the full-text
level identified in the CLEF TAR for these 32 reviews. We successfully merged
741 publications (95.2% of the total in CLEF TAR); there are only 37
publications in CLEF TAR 2019 qrels which we do not have in our records.

Table 1. Statistics of the considered dataset.  |  | CLEF TAR 2019
---|---|---
Dataset split | Training | Test
Reviews’ type |  — Interventional —
# Reviews | 17 | 15
Outcomes per review | Min | 2 |  | 3 |
Median | 9 |  | 15 |
Max | 41 |  | 128 |
Studies per outcome | Min | 1 |  | 1 |
Median | 2 |  | 2 |
Max | 55 |  | 40 |

We use 34 official CLEF TAR 2019 runs from three teams. The teams used a
variety of ranking methods, including traditional BM25, interactive BM25,
continuous active learning, relevance feedback, and various stopping criteria.
Additionally, we included 40 runs based on the reproducibility of the active
learning method by Yang et al. (2022). In total, we evaluate 74 runs, but for
the sake of brevity, in this paper, we present the results on a subset of 28
runs, as some of the runs were very similar to each other. Our model requires
full-text assessments, and thus, we use qrels from the full-text level,
despite the fact that runs have been trained on titles and abstracts. While
this might not be fair towards the evaluated systems, our experiments aim not
to establish which systems are better but to provide an example of the
operationalisation of our framework and its implications.

##  5\. Outcome-based Evaluation

We first run a simulation study to understand the results of our evaluation
framework better in a controlled manner. Then, we discuss the usage of the
evaluation framework with retrieval and classification runs on CLEF TAR 2019
collection.

###  5.1. Preliminary Simulation

We are interested in executing a preliminary study to understand the effect
our outcome-oriented evaluation has on the analysis of systematic review
automation methods.

Table 2. Initial results of the simulation on the publication level. Outcomes
are aggregated across 32 systematic reviews and are averaged from 20 different
random seeds.

|  |  | N relevant publications removed from the review
---|---|---|---
Analysis Aspect | gold | 1 | 2 | 3 | 4 | 5 | 10 | 15 | 20 | 30 | 50 | 100
1 | Mean relative difference | 0.0 | 0.9 | 2.5 | 5.3 | 7.1 | 10.0 | 18.3 | 26.2 | 36.5 | 54.9 | 65.5 | 84.5
2 | Mean distance from CI | 0.000 | 0.002 | 0.003 | 0.004 | 0.007 | 0.008 | 0.013 | 0.042 | 0.102 | 0.018 | 0.008 | 0.083
3 | Equal outcome | 824 | 786 | 750 | 706 | 657 | 623 | 496 | 410 | 340 | 256 | 164 | 80
Different | 0 | 38 | 73 | 117 | 167 | 200 | 328 | 413 | 483 | 567 | 659 | 743
\- Underestimated | 0 | 17 | 27 | 38 | 57 | 66 | 98 | 103 | 90 | 55 | 58 | 23
\- Overestimated | 0 | 20 | 45 | 79 | 109 | 134 | 229 | 309 | 393 | 512 | 601 | 720
4 | Have same sign | 824 | 815 | 800 | 774 | 756 | 735 | 663 | 597 | 516 | 365 | 277 | 121
Have different sign | 0 | 9 | 24 | 49 | 67 | 88 | 160 | 227 | 307 | 458 | 546 | 702
5 | Reported outcomes | 824 | 816 | 804 | 781 | 767 | 743 | 675 | 610 | 529 | 371 | 284 | 128
Missing outcomes | 0 | 7 | 20 | 43 | 56 | 80 | 148 | 213 | 294 | 452 | 539 | 695
Average $Recall$ for publications | 1.00 | 0.92 | 0.84 | 0.75 | 0.70 | 0.63 | 0.45 | 0.35 | 0.28 | 0.22 | 0.14 | 0.05
Average $Recall$ for studies | 1.00 | 0.97 | 0.91 | 0.80 | 0.77 | 0.68 | 0.53 | 0.43 | 0.37 | 0.31 | 0.22 | 0.12

We simulate the evaluation framework by taking the set of included
_publications_ for each review and randomly removing $[1,2,3,$ $4,5,10,$
$15,20,30,$ $50,100]$ publications from the set and then re-calculating the
outcomes. In other words, we are interested in exploring the impact of false
negatives on the final review outcome. We compare the outcomes with the ‘gold’
outcomes from the original review. Results from all 32 systematic reviews are
reported in Table 2. In our analysis, we consider the metrics from all five
analysis aspects (Section 3.3), as well as the Recall.

Figure 2 presents box plots of averaged relative difference (aspect (1))
values from our simulation at a cut-off at 20% of the total number of
documents. These results validate our expectations regarding the behaviour of
this aspect of analysis as the relative difference grows with the number of
removed publications. On the other hand, the distance to confidence intervals
(aspect (2), Figure 3) does not show any specific trend on the CLEF 2019
reviews.

Figure 2. Box plots presenting relative difference values from 20 simulations
on the publication level. Note that the x-axis does not preserve the linear
step. Figure 3. Box plots presenting distance to confidence intervals values
from 20 simulations on the publication level. Note that the x-axis does not
preserve the linear step.

Out of all the metrics, the one that changes the most when varying the number
of removed publications is estimability (5). As more publications are removed,
it becomes more and more challenging to calculate outcomes, predominantly
because half of the original outcomes relied on one or two studies. At the
very extreme, when 100 publications are removed from every review, only 15% of
outcomes are still estimable.

The measure of overestimation and underestimation (3) is showing growing
trends with more publications being removed. Already not including one
publication per review (achieving an average recall of 92% for publications
and 97% for studies) changed 38 outcomes (4.6% of the total number of
outcomes). This shows that the commonly used threshold of 95% Recall does not
enforce preserving the same outcomes of the review. We also notice that the
sign (4) aspect is not very descriptive across the simulations as it is mainly
influenced by non-estimable outcomes.

###  5.2. Evaluation with actual runs

In this section, we use the prediction on the test subset of the dataset from
runs described in Section 4.1 and evaluate them using our framework. We
further consider two baselines:

gold:

– the best possible run which returns all relevant studies from the original
review first.

max-with-qrels:

– this run takes into account the limitations of the CLEF TAR collection and
our PubMed articles matching process. It uses all relevant studies identified
in the CLEF TAR 2019 qrels as relevant and places them first.

We follow the evaluation procedure of CLEF TAR and calculate the following
traditional evaluation measures: Mean Average Precision ($MAP$), last relevant
found, Recall@k% of top-ranked publications, with k in [5, 10, 20, 30, 50],
Work Saved over Sampling at r% of recall with r in [95%, 100%] ($WSS@95\%$,
$WSS@100\%$), $nDCG@20\%$ of top-ranked publications and Area Under Recall
Curve ($AURC$). CLEF TAR as their primary reporting measure used $MAP$;
therefore, we will treat $MAP$ as the reference measure when sorting runs. We
do not evaluate baselines with traditional measures, yet for the purpose of
sorting, we assume that they achieved the highest MAP score.

We calculate the relative difference in study outcomes (analysis aspect (1) in
Section 3.3) for every outcome in all reviews. The lower the average score is,
the better the runs, as their effect differs less from the original review
effect. As considered runs were rankings, we follow the same procedure as for
Recall and nDCG, namely we calculate the relative difference at k% of top-
ranked publications with k in [5, 10, 20, 30, 50].

Figure 4. Box plot presenting runs with their relative difference in study
outcomes for an evaluation with a cut-off at 30% of the total number of
documents for each review. Runs are sorted by their MAP score. The orange
circle denotes the mean relative difference $\mathbf{@30\%}$. The X-axis is
cut at 30, while the outliers exist up to the value of 100; we cut for
visualisation purposes. Figure 5. Linear regression fits between relative
difference at 20% cut-off of documents and other evaluation measures scores.
Correlations for relative difference at other cut-offs follow similar trends.

Figure 4 presents a box plot of relative difference per outcome calculated at
30% cut-off of dataset size for 15 test CLEF TAR reviews. Except for the best
run, all other runs changed their rank when ordered using their mean relative
difference score compared to the MAP-based ranking. While top runs, according
to MAP scores, have low variability, there are runs among the top 10 which
show considerable fluctuation. This means there are specific reviews for which
these runs will lead to significantly different decisions about the outcome.
This behaviour is comparable for relative difference at other cut-offs $k$.

What is also interesting is that the mean relative difference at $30\%$ cut-
off for the _max-with-qrels_ baseline run equals 6.24. Furthermore, for the
relative difference score calculated at 100% of documents, this baseline score
is also not equal to 0. This means that the limitations of the CLEF TAR
collection and qrels establish a lower bound for the best achievable value of
relative difference.

Figure 5 presents correlation between relative difference calculated at 20%
cut-off of dataset size and evaluation measures used at CLEF TAR 2019. The
score correlates positively with the last relevant found, but there is a
negative correlation with all other measures. This confirms our intuition that
a higher average relative difference score across outcomes means a worse model
effectiveness, as the ideal ‘best’ model should achieve a difference of 0.

###  5.3. Pareto Frontier Optimisation

Based on the simulation results, we note a problem with non-estimable
outcomes. Should these outcomes be assigned a zero score or maybe an infinite
value? This raises the issue of handling these values in the evaluation
process for calculating relative difference scores. In our study, we assigned
a zero value to non-estimable outcomes, which allowed us to assume that the
relative difference equals 100%. Nevertheless, this yields the problem of when
the actual outcome is equal to the zero value (i.e., the study does not favour
the experimental nor the control group), as the difference, in this case,
would also be zero. One way to overcome the issue of non-estimable outcomes
would be to evaluate both estimability and relative difference implemented,
for instance, using the Pareto frontier (Lotov and Miettinen, 2008).

Figure 6. Visualisation of the Pareto frontier for two objectives: (1) number
of non-estimable outcomes on the x-axis and (2) sum of relative difference for
estimable outcomes on the y-axis. Both objectives are to be minimised. Runs
are evaluated at a cut-off at 5% of the total number of documents for each
review. Non-dominated runs are marked with a blue colour.

Figure 6 presents the Pareto frontier evaluated at a cut-off at 5% of the
total number of documents. On the x-axis, we show the number of non-estimable
outcomes for each run. On the y-axis, there is a sum of relative difference
for estimable outcomes. We min-max normalise the sums including the gold
baseline run (gold represents the best achievable score of $(0,0)$). Both
objectives should be minimised, i.e., we want to have as few non-estimable
outcomes as possible and for all estimated outcomes, the difference would be
as close to zero as possible. Contrary to the previous evaluations, we can
notice that no single run would dominate on both dimensions.

##  6\. Limitations

The primary objective of this paper was to introduce the concept of evaluating
automated methods for systematic reviews based on their impact on review
outcomes, rather than relying on binary qrels. In this section, we reflect on
the potential limitations that arise when attempting to fully operationalise
our proposed framework.

Do not optimise models using this measure. A practice that can be observed
across the field is treating evaluation measures as an optimisation objective.
We believe that our evaluation approach should not be used for optimising
models. The notion of difference in study outcomes is only known a-posteriori
when the review is completed. Using absolute differences in study outcomes as
an optimisation objective might lead to over-fitting to biases in data.

Other types of systematic reviews. We focus only on systematic reviews of
interventions which have a clear structure and evaluate the effectiveness of
specific treatments, programs, or policies by comparing experimental setups
with control groups. However, there are several other types of systematic
reviews, such as diagnostic test accuracy reviews, prognostic reviews, and
qualitative research reviews, each of which presents unique challenges for
automation and evaluation (Kanoulas et al., 2019). Future work should
investigate how this outcome-based evaluation framework can be extended to
these other types of reviews.

Different outcome types. While our proposed evaluation framework focuses on
continuous and dichotomous outcomes, other types of outcomes may be reported
in systematic reviews, including ordinal, count, and time-to-event data. In
our analysis, however, we found that continuous and dichotomous outcomes
comprised most of the outcomes in the dataset we studied, accounting for 92%
of all reported outcomes across 32 CLEF TAR 2019 reviews.

We believe that our evaluation framework could be generalised to incorporate
other types of outcomes. Additionally, while we attempted to closely follow
the evaluation protocols from the Cochrane handbook, some shortcuts were taken
during the implementation process (for 2.4% of outcomes our effect
calculations yielded marginally different results). In future work, ideally,
access to RevMan or another official program for calculating study outcomes
would be needed to make sure that all outcome types are covered.

Title and abstract screening. We work on the outcomes extracted from the full-
text screening and use relevance judgments from full-text screening to judge
the runs. However, most models are trained on titles and abstracts, which
might make this an unfair comparison.

##  7\. Conclusion

This paper puts forward a novel, outcome-based evaluation framework for
assessing the effectiveness of automatic search strategies and citation
screening methods in the context of systematic literature reviews. Our
proposed framework evaluates the quality of these methods based on how closely
the outcomes of their included publications match the actual review outcomes.
We believe that this approach offers a more accurate reflection of real-world
scenarios where not all included publications have the same impact on the
final review outcome.

In addition to proposing the framework, we explore five analysis aspects that
it enables, including measuring the numerical difference in predicted
systematic review outcomes. We run initial experiments to simulate the impact
of false negatives on reviews’ outcomes showing that five missing publications
per review can change 24% of outcomes. We also compare the evaluation results
obtained using our framework with those obtained using traditional evaluation
methods on CLEF TAR 2019 runs, highlighting the differences in focus between
the two approaches.

Overall, we believe this framework represents a step forward in developing
more effective and realistic methods for evaluating automation methods in the
context of systematic literature reviews in medicine and in other domains in
which the importance of systematic reviews is increasing.

###### Acknowledgements.

This work was supported by the EU Horizon 2020 ITN/ETN on Domain Specific
Systems for Information Extraction and Retrieval – DoSSIER (H2020-EU.1.3.1.,
ID: 860721). This research is partially funded by the Australian Research
Council, Discovery Project DP210104043.

## References

  * (1)
  * Bannach-Brown et al. (2019) Alexandra Bannach-Brown, Piotr Przybyła, James Thomas, Andrew S. C. Rice, Sophia Ananiadou, Jing Liao, and Malcolm Robert Macleod. 2019\.  Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error.  _Systematic Reviews 2019 8:1_ 8, 1 (1 2019), 1–12.  https://doi.org/10.1186/S13643-019-0942-7
  * Bilotta et al. (2014) Gary S Bilotta, Alice M Milner, and Ian Boyd. 2014.  On the use of systematic reviews to inform environmental policies.  _Environmental Science & Policy_ 42 (2014), 67–77.
  * Castillo and Grbovic (2022) Stefanie Castillo and Petar Grbovic. 2022.  The APISSER Methodology for Systematic Literature Reviews in Engineering.  _IEEE Access_ 10 (2022), 23700–23707.
  * Cohen et al. (2010) Aaron M Cohen, Kyle Ambert, and Marian McDonagh. 2010\.  A prospective evaluation of an automated classification system to support evidence-based medicine and systematic review. In _AMIA annual symposium proceedings_ , Vol. 2010. American Medical Informatics Association, 121\.
  * Cohen et al. (2006) A. M. Cohen, W. R. Hersh, K. Peterson, and Po Yin Yen. 2006\.  Reducing workload in systematic review preparation using automated citation classification.  _Journal of the American Medical Informatics Association_ 13, 2 (3 2006), 206–219.  https://doi.org/10.1197/jamia.M1929
  * Dalal et al. (2013) Siddhartha R Dalal, Paul G Shekelle, Susanne Hempel, Sydne J Newberry, Aneesa Motala, and Kanaka D Shetty. 2013.  A pilot study using machine learning and domain knowledge to facilitate comparative effectiveness review updating.  _Medical Decision Making_ 33, 3 (2013), 343–355.
  * Deeks et al. (2019) Jonathan J Deeks, Julian PT Higgins, Douglas G Altman, and Cochrane Statistical Methods Group. 2019\.  Analysing data and undertaking meta-analyses.  _Cochrane handbook for systematic reviews of interventions_ (2019), 241–284.
  * Deeks and Higgins (2010) Jonathan J. Deeks and Julian P. T. Higgins. 2010.  Statistical algorithms in Review Manager 5.  _Statistical Methods Group of The Cochrane Collaboration_ 1, 11 (2010).
  * Grossman et al. (2016) Maura R Grossman, Gordon V Cormack, and Adam Roegiest. 2016\.  TREC 2016 Total Recall Track Overview.. In _TREC_.
  * Higgins et al. (2019) Julian PT Higgins, Tianjing Li, and Jonathan J Deeks. 2019\.  Choosing effect measures and computing estimates of effect.  _Cochrane handbook for systematic reviews of interventions_ (2019), 143–176.
  * Howard et al. (2016) Brian E. Howard, Jason Phillips, Kyle Miller, Arpit Tandon, Deepak Mav, Mihir R. Shah, Stephanie Holmgren, Katherine E. Pelch, Vickie Walker, Andrew A. Rooney, Malcolm Macleod, Ruchir R. Shah, and Kristina Thayer. 2016.  SWIFT-Review: A text-mining workbench for systematic review.  _Systematic Reviews_ 5, 1 (5 2016), 1–16.  https://doi.org/10.1186/s13643-016-0263-z
  * Howard et al. (2020) Brian E. Howard, Jason Phillips, Arpit Tandon, Adyasha Maharana, Rebecca Elmore, Deepak Mav, Alex Sedykh, Kristina Thayer, B. Alex Merrick, Vickie Walker, Andrew Rooney, and Ruchir R. Shah. 2020\.  SWIFT-Active Screener: Accelerated document screening through active learning and integrated recall estimation.  _Environment International_ 138 (5 2020), 105623\.  https://doi.org/10.1016/J.ENVINT.2020.105623
  * Jo et al. (2009) Akers Jo, Aguiar-Ibáñez Raquel, Burch Jane, Chambers Duncan, Eastwood Alison, Fayter Debra, Hempel Susanne, Light Kate, Rice Stephen, Rithalia Amber, Stewart Lesley, Stock Christian, Wilson Paul, and Woolacott Nerys. 2009.  _Systematic Reviews: CRD’s guidance for undertaking reviews in health care_.  CRD, University of York, York.  www.york.ac.uk/inst/crd
  * Kanoulas et al. (2017) Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2017\.  CLEF 2017 technologically assisted reviews in empirical medicine overview.  _CEUR Workshop Proceedings_ 1866 (9 2017), 1–29.  https://pureportal.strath.ac.uk/en/publications/clef-2017-technologically-assisted-reviews-in-empirical-medicine-
  * Kanoulas et al. (2018) Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2018\.  CLEF 2018 technologically assisted reviews in empirical medicine overview.  _CEUR Workshop Proceedings_ 2125 (7 2018).  https://pureportal.strath.ac.uk/en/publications/clef-2018-technologically-assisted-reviews-in-empirical-medicine-
  * Kanoulas et al. (2019) E. Kanoulas, Dan Li, Leif Azzopardi, and René Spijker. 2019\.  CLEF 2019 Technology Assisted Reviews in Empirical Medicine Overview. In _CLEF_.
  * Keele et al. (2007) Staffs Keele et al. 2007\.  _Guidelines for performing systematic literature reviews in software engineering_.  Technical Report. Technical report, ver. 2.3 ebse technical report. ebse.
  * Kontonatsios et al. (2020) Georgios Kontonatsios, Sally Spencer, Peter Matthew, and Ioannis Korkontzelos. 2020.  Using a neural network-based feature extraction method to facilitate citation screening for systematic reviews.  _Expert Systems with Applications: X_ 6 (7 2020), 100030\.  https://doi.org/10.1016/j.eswax.2020.100030
  * Kusa et al. (2022) Wojciech Kusa, Allan Hanbury, and Petr Knoth. 2022\.  Automation of Citation Screening for Systematic Literature Reviews Using Neural Networks: A Replicability Study. In _Advances in Information Retrieval_ , Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg, and Vinay Setty (Eds.). Springer International Publishing, Cham, 584–598.  https://arxiv.org/abs/2201.07534v1
  * Kusa et al. (2023a) Wojciech Kusa, Aldo Lipani, Petr Knoth, and Allan Hanbury. 2023a.  An Analysis of Work Saved over Sampling in the Evaluation of Automated Citation Screening in Systematic Literature Reviews.  _Intelligent Systems with Applications_ 18 (2023), 200193.  https://doi.org/10.1016/j.iswa.2023.200193
  * Kusa et al. (2023b) Wojciech Kusa, Aldo Lipani, Petr Knoth, and Allan Hanbury. 2023b.  VoMBaT: A Tool for Visualising Evaluation Measure Behaviour in High-Recall Search Tasks. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’23)_. ACM, Taipei, Taiwan, 5\.  https://doi.org/10.1145/3539618.3591802
  * Lotov and Miettinen (2008) Alexander V Lotov and Kaisa Miettinen. 2008.  Visualizing the Pareto Frontier.  _Multiobjective optimization_ 5252 (2008), 213–243.
  * Magdy and Jones (2010) Walid Magdy and Gareth JF Jones. 2010.  PRES: a score metric for evaluating recall-oriented information retrieval applications. In _Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval_. 611–618.
  * Matwin et al. (2010) Stan Matwin, Alexandre Kouznetsov, Diana Inkpen, Oana Frunza, and Peter O’Blenis. 2010.  A new algorithm for reducing the workload of experts in performing systematic reviews.  _Journal of the American Medical Informatics Association_ 17, 4 (7 2010), 446–453.  https://doi.org/10.1136/JAMIA.2010.004325
  * Nussbaumer-Streit et al. (2020) B Nussbaumer-Streit, I Klerings, AI Dobrescu, E Persad, A Stevens, C Garritty, C Kamel, L Affengruber, VJ King, and G Gartlehner. 2020.  Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.  _Journal of clinical epidemiology_ 118 (2020), 42–54.
  * Nussbaumer-Streit et al. (2018) Barbara Nussbaumer-Streit, Irma Klerings, Gernot Wagner, Thomas L. Heise, Andreea I. Dobrescu, Susan Armijo-Olivo, Jan M. Stratil, Emma Persad, Stefan K. Lhachimi, Megan G. Van Noord, Tarquin Mittermayr, Hajo Zeeb, Lars Hemkens, and Gerald Gartlehner. 2018.  Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study.  _Journal of Clinical Epidemiology_ 102 (2018), 1–11.  https://doi.org/10.1016/j.jclinepi.2018.05.022
  * O’Mara-Eves et al. (2015) Alison O’Mara-Eves, James Thomas, John McNaught, Makoto Miwa, and Sophia Ananiadou. 2015.  Using text mining for study identification in systematic reviews: a systematic review of current approaches.  _Systematic reviews_ 4, 1 (2015), 1–22.
  * Petticrew and Roberts (2008) Mark Petticrew and Helen Roberts. 2008.  _Systematic reviews in the social sciences: A practical guide_.  John Wiley & Sons.
  * Scells et al. (2020a) Harrisen Scells, Guido Zuccon, and Bevan Koopman. 2020a.  You can teach an old dog new tricks: Rank fusion applied to coordination level matching for ranking in systematic reviews. In _Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part I 42_. Springer, 399–414.
  * Scells et al. (2021) Harrisen Scells, Guido Zuccon, and Bevan Koopman. 2021\.  A comparison of automatic Boolean query formulation for systematic reviews.  _Information Retrieval Journal_ 24, 1 (2021), 3–28.
  * Scells et al. (2020b) Harrisen Scells, Guido Zuccon, Bevan Koopman, and Justin Clark. 2020b.  Automatic Boolean Query Formulation for Systematic Review Literature Search. In _The Web Conference 2020 - Proceedings of the World Wide Web Conference, WWW 2020_. ACM, New York, NY, USA, 1071–1081.  https://doi.org/10.1145/3366423.3380185
  * Scells et al. (2017) Harrisen Scells, Guido Zuccon, Bevan Koopman, Anthony Deacon, Leif Azzopardi, and Shlomo Geva. 2017\.  A test collection for evaluating retrieval of studies for inclusion in systematic reviews.  _SIGIR 2017 - Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (8 2017), 1237–1240.  https://doi.org/10.1145/3077136.3080707
  * Tsafnat et al. (2018) Guy Tsafnat, Paul Glasziou, George Karystianis, and Enrico Coiera. 2018.  Automated screening of research studies for systematic reviews using study characteristics.  _Systematic Reviews 2018 7:1_ 7, 1 (4 2018), 1–9.  https://doi.org/10.1186/S13643-018-0724-7
  * van Dinter et al. (2021) Raymon van Dinter, Bedir Tekinerdogan, and Cagatay Catal. 2021\.  Automation of systematic literature reviews: A systematic literature review.  _Information and Software Technology_ 136 (8 2021), 106589\.  https://doi.org/10.1016/j.infsof.2021.106589
  * Wallace et al. (2010a) Byron C. Wallace, Kevin Small, Carla E. Brodley, and Thomas A. Trikalinos. 2010a.  Active learning for biomedical citation screening.  _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (2010), 173–181.  https://doi.org/10.1145/1835804.1835829
  * Wallace et al. (2011) Byron C Wallace, Kevin Small, Carla E Brodley, and Thomas A Trikalinos. 2011.  Who should label what? instance allocation in multiple expert active learning. In _Proceedings of the 2011 SIAM international conference on data mining_. SIAM, 176–187.
  * Wallace et al. (2010b) Byron C Wallace, Thomas A Trikalinos, Joseph Lau, Carla Brodley, and Christopher H Schmid. 2010b.  Semi-automated screening of biomedical citations for systematic reviews.  _BMC bioinformatics_ 11, 1 (2010), 1–11.
  * Wang et al. (2022) Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2022.  Automated MeSH Term Suggestion for Effective Query Formulation in Systematic Reviews Literature Search.  _Intelligent Systems with Applications_ (2022), 200141.
  * Wang et al. (2023a) Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2023a.  Can ChatGPT write a good boolean query for systematic review literature search?  _arXiv preprint arXiv:2302.03495_ (2023).
  * Wang et al. (2023b) Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2023b.  Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search. In _Proceedings of the 26th Australasian Document Computing Symposium_ (Adelaide, SA, Australia) _(ADCS ’22)_. Association for Computing Machinery, New York, NY, USA, Article 4, 10 pages.  https://doi.org/10.1145/3572960.3572980
  * Yang et al. (2022) Eugene Yang, Sean MacAvaney, David D Lewis, and Ophir Frieder. 2022.  Goldilocks: Just-right tuning of BERT for technology-assisted review. In _European Conference on Information Retrieval_. Springer, 502–517.