# Improved determination of the $D\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ coherence
factor and associated hadronic parameters from a combination of
$e^{+}e^{-}\to\psi(3770)\to c\bar{c}$ and $pp\to c\bar{c}X$ data

T. Evans  S.T. Harnew  J. Libby  S. Malde  J. Rademacker  G. Wilkinson
<EMAIL_ADDRESS>University of Oxford, Denys Wilkinson Building,
Keble Road, OX1 3RH, United Kingdom  University of Bristol, Bristol, BS8 1TL,
United Kingdom  Indian Institute of Technology Madras, Chennai 600036, India

###### Abstract

Measurements of the coherence factor $R_{K3\pi}$, the average strong-phase
difference $\delta^{K3\pi}_{D}$ and mean amplitude ratio $r_{D}^{K3\pi}$ for
the decay $D\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ are presented. These parameters
are important inputs to the determination of the unitarity triangle angle
$\gamma$ in $B^{-}\to DK^{-}$ decays, where $D$ designates a superposition of
$D^{0}$ and $\bar{D}{}^{0}$ mesons decaying to a common final state. The
results are based on a combined fit to observables obtained from a re-analysis
of the CLEO-c $\psi(3770)$ data set and those measured in a $D^{0}\bar{D}^{0}$
mixing study performed by the LHCb collaboration.

###### keywords:

charm decay, quantum correlations, $\mathit{CP}$ violation

††journal: Physics Letters B

##  1 Introduction

Knowledge of the $D\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ coherence factor and
associated hadronic parameters is necessary for a measurement of the unitarity
triangle angle $\gamma$ (also denoted $\phi_{3}$) in $B^{-}\to DK^{-},D\to
K^{-}\pi^{+}\pi^{+}\pi^{-}$ decays. (The symbol $D$ is used to denote a
neutral charm-meson that is not in a flavour eigenstate, or where the flavour
eigenvalue is not relevant for the discussion.)

The coherence factor $R_{K3\pi}$ and average strong-phase difference
$\delta^{K3\pi}_{D}$ for the decay $D^{0}\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ are
defined as follows [1]:

$R_{K3\pi}e^{-i\delta_{D}^{K3\pi}}=\frac{\int\mathcal{A}^{\ast}_{K^{-}\pi^{+}\pi^{+}\pi^{-}}(\mathbf{x})\mathcal{A}_{K^{+}\pi^{-}\pi^{+}\pi^{-}}(\mathbf{x})\mathrm{d}\mathbf{x}}{A_{K^{-}\pi^{+}\pi^{+}\pi^{-}}A_{K^{+}\pi^{-}\pi^{+}\pi^{-}}}\,,$
(1)

where $\mathcal{A}_{K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}}(\mathbf{x})$ is the decay
amplitude of $D^{0}\to K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ at a point in multi-
body phase space described by parameters $\mathbf{x}$, and

$A_{K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}}^{2}=\int|\mathcal{A}_{K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}}(\mathbf{x})|^{2}\mathrm{d}\mathbf{x}.$
(2)

Therefore $A_{K^{-}\pi^{+}\pi^{+}\pi^{-}}$ is the Cabibbo-favoured (CF)
amplitude, averaged over phase space, and $A_{K^{+}\pi^{-}\pi^{+}\pi^{-}}$ is
the corresponding doubly Cabibbo-suppressed (DCS) quantity. The coherence
factor takes a value between 0 and 1. It is also useful to define the
parameter
$r_{D}^{K3\pi}={A_{K^{+}\pi^{-}\pi^{+}\pi^{-}}}/{A_{K^{-}\pi^{+}\pi^{+}\pi^{-}}}$.
In this Letter the term ‘hadronic parameters’ of the decay $D\to
K^{-}\pi^{+}\pi^{+}\pi^{-}$ is employed to refer collectively to $R_{K3\pi}$,
$\delta_{D}^{K3\pi}$ and $r_{D}^{K3\pi}$. Throughout the discussion charge
conjugation is implicit, the good approximation is made that
$\mathit{CP}$-violation can be neglected in $D^{0}$ mixing and decay [2], and
expressions are given in the convention
$\mathit{CP}|D^{0}\rangle=|\bar{D}^{0}\rangle$.

The role of the hadronic parameters in measurements sensitive to $\gamma$ can
be appreciated by considering the decay of $B^{\mp}$ mesons to a neutral
charm-meson, reconstructed in the inclusive $K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$
final state, and a charged kaon. Neglecting small corrections from
$D^{0}\bar{D}^{0}$ mixing, which can be included in a straightforward manner
[3], the decay rates are given by:

$\displaystyle{\Gamma(B^{\mp}\to(K^{\pm}\pi^{\mp}\pi^{+}\pi^{-})_{D}K^{\mp})}$
$\displaystyle\propto$ $\displaystyle(r_{B})^{2}+(r_{D}^{K3\pi})^{2}+$ (3)
$\displaystyle
2r_{B}r_{D}^{K3\pi}R_{K3\pi}\cos{(\delta_{B}+\delta_{D}^{K3\pi}\mp\gamma)}\,.$

Here $r_{B}\sim 0.1$ is the absolute ratio of $B^{-}\to{\bar{D}}^{0}K^{-}$ to
$B^{-}\to D^{0}K^{-}$ amplitudes. The phase difference between these two
amplitudes is $(\delta_{B}-\gamma)$, where $\delta_{B}$ is a
$\mathit{CP}$-conserving strong phase. The coherence factor, which is replaced
by unity in the equivalent expression for a single point in phase space or
two-body $D$-meson decays, modulates the size of the interference term that
carries the dependence on $\gamma$.

The hadronic parameters can be measured in the decays of coherently produced
$D\bar{D}$ pairs at the $\psi(3770)$ resonance [1], such as are available in
the data sets collected by the CLEO-c and BESIII experiments. A double-tag
technique is employed where one meson is reconstructed in the signal decay
$D\to K^{-}\pi^{+}\pi^{+}\pi^{-}$, and the other in a tagging mode, for
example, a $\mathit{CP}$ eigenstate. This method was applied by the CLEO
collaboration to obtain first constraints on $R_{K3\pi}$ and
$\delta^{K3\pi}_{D}$, together with $R_{K\pi\pi^{0}}$ and
$\delta^{K\pi\pi^{0}}_{D}$, the analogous parameters for $D\to
K^{-}\pi^{+}\pi^{0}$ decays [4]. A later analysis augmented the set of tags
with the decay $D\to K^{0}_{\rm S}\pi^{+}\pi^{-}$ in order to improve the
sensitivity to the hadronic parameters of both signal modes [5]. These results
have been exploited by the LHCb collaboration to interpret a set of
measurements with the decays $B^{-}\to DK^{-}$, $D\to
K^{-}\pi^{+}\pi^{+}\pi^{-}$ [6] and $D\to K^{-}\pi^{+}\pi^{0}$ [7], in
combination with observables from other modes, to yield
$\gamma=(73^{+9}_{-10})^{\circ}$ [8].

Observables sensitive to $D^{0}\bar{D}^{0}$ mixing in multibody $D$-meson
decays are also affected by the same hadronic parameters [9, 10, 11]. The
time-dependent ratio between $D^{0}\to K^{+}\pi^{-}\pi^{+}\pi^{-}$ and
$D^{0}\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ decay rates is

$R(t)\approx(r_{D}^{K3\pi})^{2}-a\frac{t}{\tau}+b\left(\frac{t}{\tau}\right)^{2},$
(4)

where $t$ is the proper decay time, $\tau$ is the mean $D^{0}$ lifetime,
$a=R_{K3\pi}(y\cos\delta_{D}^{K3\pi}$ $-x\sin\delta_{D}^{K3\pi})$,
$b=\frac{1}{4}(x^{2}+y^{2})$, and $x$ and $y$ are the commonly used mixing
parameters, defined for example in Ref. [2]. As noted in Refs. [10, 11], a
measurement of $r_{D}^{K3\pi}$, $a$ and $b$ can either be used to determine
$x$ and $y$, given external information on $R_{K3\pi}$ and
$\delta_{D}^{K3\pi}$, or employed to set constraints on the latter parameters,
provided $x$ and $y$ are known. Both interpretations are presented in Ref.
[12], which reports a first observation of $D^{0}\bar{D}^{0}$ mixing with this
multibody final state.

Improved knowledge of $R_{K3\pi}$, $\delta^{K3\pi}_{D}$ and $r_{D}^{K3\pi}$ is
also needed to help understand the contribution of $D\to
K^{-}\pi^{+}\pi^{+}\pi^{-}$ decays to the width difference in the
$D^{0}\bar{D}^{0}$ system [13].

This Letter reports a new analysis of $D\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ decays
in the CLEO-c data set, and benefits from an updated Monte Carlo simulation
sample to correct a biased estimate of the background contamination in several
sets of double tags that afflicted the results reported in Refs. [4, 5]. In
addition, the quasi-$\mathit{CP}$-eigenstate $D\to\pi^{+}\pi^{-}\pi^{0}$ [14]
is included as a new tagging mode to augment the sensitivity of the analysis.
Furthermore, an error is corrected in the formalism used to interpret the
yields found with the $D\to K^{0}_{\rm S}\pi^{+}\pi^{-}$ tags 111The error is
present in Ref. [5] and also in the previously posted version of this paper..
The observables from this new analysis are then used to determine updated
constraints on the hadronic parameters. Finally, a global fit is made to the
observables from this study and those measured in $D^{0}\bar{D}^{0}$ mixing by
LHCb, to obtain more precise results for the $D\to K^{-}\pi^{+}\pi^{+}\pi^{-}$
hadronic parameters. Since one of the re-measured $\psi(3770)$ observables
couples the $K^{-}\pi^{+}\pi^{+}\pi^{-}$ and $K^{-}\pi^{+}\pi^{0}$ systems, it
is also possible to determine updated values of $R_{K\pi\pi^{0}}$,
$\delta_{D}^{K\pi\pi^{0}}$ and $r_{D}^{K\pi\pi^{0}}$.

##  2 Measuring the hadronic parameters with CLEO-c data

In this section the observables sensitive to the hadronic parameters are
reviewed, and a new analysis of the CLEO-c data set is presented that updates
the measured values with respect to those reported in Ref. [5]. Information on
the hadronic parameters is then obtained from a fit to the updated
measurements.

###  2.1 Observables

Two categories of observables exist, both based on double-tagged measurements
in which one $D$ meson decays to the signal mode $K^{-}\pi^{+}\pi^{+}\pi^{-}$,
and the other decays to a tagging mode. In the first category the tagging
modes are two-body or higher multiplicity final states, which are treated in
an inclusive fashion. In the second case, the tagging mode is the self-
conjugate final-state $K^{0}_{\rm S}\pi^{+}\pi^{-}$ and yields are measured in
different phase-space bins of this tagging decay, defined in the plane of the
Dalitz plot.

In the category involving inclusive tags, so-called $\rho$ observables are
constructed, which are the ratio of the measured double-tag yields, after
background subtraction and efficiency correction, to the yields expected in
the absence of quantum-correlations.

$\bm{\rho^{K3\pi}_{\mathit{CP}\pm}}$

These are the ratios where the tagging mode is a $\mathit{CP}$-even eigenstate
($\rho^{K3\pi}_{\mathit{CP}+}$) or a $\mathit{CP}$-odd eigenstate
($\rho^{K3\pi}_{\mathit{CP}-}$). Neglecting corrections from
$D^{0}\bar{D}^{0}$ mixing, which enter through the definitions of the
branching fractions used in the normalisation factors,
$\rho^{K3\pi}_{\mathit{CP}\pm}=1\mp
2r_{D}^{K3\pi}R_{K3\pi}\cos\delta_{D}^{K3\pi}/$ $[1+(r_{D}^{K3\pi})^{2}].$
Precise definitions and mixing-corrected expressions for this, and subsequent
relations, can be found in Ref. [4].

$\bm{\Delta^{K3\pi}_{\mathit{CP}}}$

This is a $\mathit{CP}$-invariant observable defined
$\Delta^{K3\pi}_{\mathit{CP}}\equiv\pm
1\times(\rho^{K3\pi}_{\mathit{CP}\pm}-1).$ It allows the results for the
$\mathit{CP}$-even and $\mathit{CP}$-odd tags to be combined together.
Neglecting mixing,
$\Delta^{K3\pi}_{\mathit{CP}}=-2r_{D}^{K3\pi}R_{K3\pi}\cos\delta_{D}^{K3\pi}/[1+(r_{D}^{K3\pi})^{2}]$.

$\bm{\rho^{K3\pi}_{LS}}$

This is the ratio involving $K^{-}\pi^{+}\pi^{+}\pi^{-}$ vs.
$K^{-}\pi^{+}\pi^{+}\pi^{-}$ events, i.e. those where the kaons are of like
sign (LS). Neglecting mixing, $\rho^{K3\pi}_{LS}=1-(R_{K3\pi})^{2}.$

$\bm{\rho^{K3\pi}_{K\pi,LS}}$

This is the ratio involving $K^{-}\pi^{+}\pi^{+}\pi^{-}$ vs. $K^{-}\pi^{+}$
events. Neglecting mixing,
$\rho^{K3\pi}_{K\pi,LS}=1\,-\,2(r_{D}^{K3\pi}/r_{D}^{K\pi})R_{K3\pi}\cos(\delta_{D}^{K\pi}-\delta_{D}^{K3\pi})/\left[1+(r_{D}^{K3\pi}/r_{D}^{K\pi})^{2}\right]$,
where $r_{D}^{K\pi}$ is the ratio between the DCS and CF $D\to K^{-}\pi^{+}$
amplitudes and $\delta^{K\pi}_{D}$ the accompanying strong-phase difference.

$\bm{\rho_{K\pi\pi^{0},LS}^{K3\pi}}$

This is the ratio involving $K^{-}\pi^{+}\pi^{+}\pi^{-}$ vs.
$K^{-}\pi^{+}\pi^{0}$ events. Neglecting mixing,
$\rho_{K\pi\pi^{0},LS}^{K3\pi}=1\,-\,2(r_{D}^{K3\pi}/r_{D}^{K\pi\pi^{0}})R_{K3\pi}R_{K\pi\pi^{0}}\cos(\delta_{D}^{K\pi\pi^{0}}-\delta_{D}^{K3\pi})/$
$[1+(r_{D}^{K3\pi}/r_{D}^{K\pi\pi^{0}})^{2}].$

The deviation of any of the $\rho$ observables from unity or
$\Delta^{K3\pi}_{\mathit{CP}}$ from zero is indicative of a non-zero coherence
factor. Recalling that $r_{D}^{K3\pi}$, $r_{D}^{K\pi}$ and
$r_{D}^{K\pi\pi^{0}}\sim 0.05$, it is expected that
$|\Delta^{K3\pi}_{\mathit{CP}}|<0.1$, whereas larger effects can occur for the
like-sign observables.

The observables involving $K^{0}_{\rm S}\pi^{+}\pi^{-}$ decays comprise the
yields of double tags, after background subtraction and bin-to-bin relative
efficiency correction, in eight pairs of bins in the plane of the Dalitz plot
symmetric about the line $m[K^{0}_{\rm S}\pi^{+}]^{2}=m[K^{0}_{\rm
S}\pi^{-}]^{2}$. The binning scheme follows the ‘equal $\Delta\delta_{D}$’
definition of Ref. [15, 16], where the partitioning is guided by an amplitude
model developed by the BaBar collaboration [17]. The observables are denoted
$Y_{i}$, where the subscript gives the bin number ($i=-8\to 8$, excluding 0).
The values of $Y_{i}$ differ from those expected in the incoherent case in a
manner that is dependent on the values of the coherence factor and average
strong-phase difference of the signal mode [5]:

$\displaystyle Y_{i}$ $\displaystyle=$ $\displaystyle
H_{K3\pi}{\Big{(}}K_{i}+(r_{D}^{K3\pi})^{2}K_{-i}-$ (5) $\displaystyle
2r_{D}^{K3\pi}\sqrt{K_{i}K_{-i}}R_{K3\pi}[c_{i}\cos\delta_{D}^{K3\pi}-s_{i}\sin\delta_{D}^{K3\pi}]{\Big{)}}.$

(In Ref. [5] this expression is written with an incorrect sign before the term
$s_{i}\sin\delta_{D}^{K3\pi}$). Here $H_{K3\pi}$ is a bin-independent
normalisation factor and $K_{i}$ is the fractional yield of $D^{0}$ decays
that fall into bin $i$. The parameters $c_{i}$ and $s_{i}$ are the amplitude-
weighted averages over bin $i$ of $\cos(\Delta\delta_{D})$ and
$\sin(\Delta\delta_{D})$, respectively, where $\Delta\delta_{D}$ is the
strong-phase difference between the $D^{0}\to K^{0}_{\rm S}\pi^{+}\pi^{-}$ and
$\bar{D}^{0}\to K^{0}_{\rm S}\pi^{+}\pi^{-}$ amplitudes at a single point in
the Dalitz plot. All these $D\to K^{0}_{\rm S}\pi^{+}\pi^{-}$ quantities are
defined ignoring $D^{0}\bar{D}{}^{0}$ mixing effects, which is appropriate for
double-tag yields arising from $\psi(3770)$ mesons produced at rest in the
laboratory, as is the case for the CLEO-c experiment [18].

###  2.2 Yield determination and results for observables

An 818 $\rm pb^{-1}$ data set of $e^{+}e^{-}$ collisions produced by the
Cornell Electron Storage Ring (CESR) at $\sqrt{s}=3.77$ GeV and collected with
the CLEO-c detector is analysed. The CLEO-c detector is described in detail
elsewhere [19]. In addition, simulated Monte Carlo samples are studied to
assess possible background contributions and to determine efficiencies. The
EVTGEN package [20] is used to generate the decays and GEANT [21] is used to
simulate the CLEO-c detector response.

To ensure full understanding of all the inputs to the analysis, the selection
of all double tags involving the decay $D\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ is
re-performed, although the selection criteria are intended to be identical to
those reported in Refs. [4, 5]. The full list of final states reconstructed is
given in Table 1, with $\pi^{0}\to\gamma\gamma$, $K^{0}_{\rm
S}\to\pi^{+}\pi^{-}$, $\phi\to K^{+}K^{-}$, $\eta\to\gamma\gamma$,
$\eta\to\pi^{+}\pi^{-}\pi^{0}$ and
$\eta^{\prime}\to\eta(\gamma\gamma)\pi^{+}\pi^{-}$. A single new addition to
the list of tag modes is the abundant decay $D\to\pi^{+}\pi^{-}\pi^{0}$, which
has recently been measured to be very close to a $\mathit{CP}$ eigenstate,
with a $\mathit{CP}$-even fraction of $F_{+}^{\pi\pi\pi^{0}}=0.973\pm 0.017$
[14]. The selection requirements for this new mode are similar to those
reported in Ref. [14]. In particular, a $K^{0}_{\rm S}$ veto is applied to the
$\pi^{+}\pi^{-}$ combination, as described in Ref. [4], in order to suppress
contamination from $D\to K^{0}_{\rm S}\pi^{0}$ decays. This veto rejects
candidates where the two-track vertex is significantly displaced from the
beamspot, or either track has a significant impact parameter.

Table 1: $D$-meson final states reconstructed in the analysis. Type | Final states
---|---
Flavoured |  $K^{-}\pi^{+}$ $K^{-}\pi^{+}\pi^{+}\pi^{-}$ $K^{-}\pi^{+}\pi^{0}$
$\mathit{CP}$ even |  $K^{+}K^{-}$, $\pi^{+}\pi^{-}$, $K^{0}_{\rm S}\pi^{0}\pi^{0}$, $K_{\rm L}^{0}\pi^{0}$, $K_{\rm L}^{0}\omega$, $\pi^{+}\pi^{-}\pi^{0}$
$\mathit{CP}$ odd |  $K^{0}_{\rm S}\pi^{0}$, $K^{0}_{\rm S}\omega$, $K^{0}_{\rm S}\phi$, $K^{0}_{\rm S}\eta$, $K^{0}_{\rm S}\eta^{\prime}$
Self conjugate | $K^{0}_{\rm S}\pi^{+}\pi^{-}$

The most significant change in the analysis concerns the sample of simulated
inclusive $D^{0}\bar{D}^{0}$ events used to estimate the contamination from
specific background decays that occur in or close to the kinematic region
where the signal peaks. The sample in the new analysis is a factor of two
larger than that used in the original studies and benefits from updated
knowledge of branching fractions. The singly Cabibbo-suppressed modes $D\to
K^{0}_{\rm S}K^{\mp}\pi^{\pm}$ (here specifying explicitly the two final
states) are a dangerous source of background for the like-sign
$\rho^{K3\pi}_{LS}$, $\rho^{K3\pi}_{K\pi,LS}$ and
$\rho^{K3\pi}_{K\pi\pi^{0},LS}$ observables since $\frac{{\cal B}(D^{0}\to
K^{0}_{\rm S}K^{+}\pi^{-})}{{\cal B}(D^{0}\to K^{0}_{\rm
S}K^{-}\pi^{+})}\sim{\cal O}(1)$, whereas $\frac{{\cal B}(D^{0}\to
K^{+}\pi^{-}\pi^{-}\pi^{+})}{{\cal B}(D^{0}\to
K^{-}\pi^{+}\pi^{+}\pi^{-})}\sim{\cal O}(10^{-3})$ [22]. These modes were
incorrectly simulated in the old Monte Carlo sample, being generated at a rate
that was a factor of three lower than the measured branching fractions [22],
and with a resonant substructure that poorly matches experimental results [23,
24]. Both of these deficiencies are corrected in the new simulation. Figure 1
shows the invariant mass of the $\pi^{+}\pi^{-}$ combinations, summed over all
like-sign double tags, for both data and simulation. The selection
requirements include a $K^{0}_{\rm S}$ veto on the $\pi^{+}\pi^{-}$
combination in the signal-decay candidate to suppress $D\to K^{0}_{\rm
S}K^{\mp}\pi^{\pm}$ contamination. A clear peak is seen from the residual
background surviving the $K^{0}_{\rm S}$ veto, which is well modelled by the
new Monte Carlo sample, but was previously described poorly. (The old Monte
Carlo simulation also contained other deficiencies, apparent from poor
agreement in other regions of the $\pi^{+}\pi^{-}$ mass spectrum, but these
did not impact directly upon the analysis.)

Figure 1: Invariant mass of the $\pi^{+}\pi^{-}$ pairs from the $D\to
K^{-}\pi^{+}\pi^{+}\pi^{-}$ candidates summed over the
$K^{-}\pi^{+}\pi^{+}\pi^{-}$ vs. $K^{-}\pi^{+}\pi^{+}\pi^{-}$,
$K^{-}\pi^{+}\pi^{+}\pi^{-}$ vs. $K^{-}\pi^{+}$ and
$K^{-}\pi^{+}\pi^{+}\pi^{-}$ vs. $K^{-}\pi^{+}\pi^{0}$ double tags. Data are
shown by points, and the simulation by the filled histogram. Left: old
simulation used in Ref. [4]. Right: new simulation.

The event yields after background subtraction are presented in Table 2. When
appropriate, the contamination from peaking background is corrected for the
small effects of quantum-correlation, which are not simulated in the Monte
Carlo. The contribution of non-peaking background is determined in a data-
driven manner, as in the original analysis [4]. The yields for
$K^{-}\pi^{+}\pi^{+}\pi^{-}$ vs. $K^{-}\pi^{+}\pi^{+}\pi^{-}$,
$K^{-}\pi^{+}\pi^{+}\pi^{-}$ vs. $K^{-}\pi^{+}$ and
$K^{-}\pi^{+}\pi^{+}\pi^{-}$ vs. $K^{-}\pi^{+}\pi^{0}$ are all significantly
lower than previously reported, because of the revised estimate for the level
of $D\to K^{0}_{\rm S}K^{\mp}\pi^{\pm}$ contamination, which is the dominant
source of background for these double tags, and is now determined to comprise
around 36%, 30% and 30%, respectively, of the selected events in each of the
three samples. For all the other classes of double tags, where the mean purity
is in excess of 95%, the differences in results with respect to the earlier
analysis are either negligible, or small and well understood.

Table 2: Measured double-tagged yields and statistical uncertainties after background subtraction. Mode | $K^{-}\pi^{+}\pi^{+}\pi^{-}$ | $K^{-}\pi^{+}$
---|---|---
$K^{+}\pi^{-}\pi^{+}\pi^{-}$ | 4006.3 $\pm$ 65.0 |  xx–
$K^{-}\pi^{+}\pi^{+}\pi^{-}$ |  x19.7 $\pm$ 6.2 |  xx–
$K^{+}\pi^{-}$ | 5203.7 $\pm$ 72.7 | 1723.1 $\pm$ 41.8
$K^{-}\pi^{+}$ |  x26.6 $\pm$ 6.2 |  xx–
$K^{+}\pi^{-}\pi^{0}$ | 10598.0 $\pm$ 104.8 |  xx–
$K^{-}\pi^{+}\pi^{0}$ |  x53.1 $\pm$ 9.1 |  xx–
$K^{+}K^{-}$ |  x542.0 $\pm$ 23.4 |  xx–
$\pi^{+}\pi^{-}$ |  x244.2 $\pm$ 15.9 |  xx–
$K^{0}_{\rm S}\pi^{0}\pi^{0}$ |  x299.5 $\pm$ 18.3 |  x223.5 $\pm$ 15.5
$K^{0}_{\rm L}\pi^{0}$ |  x839.4 $\pm$ 30.6 |  x703.0 $\pm$ 27.9
$K^{0}_{\rm L}\omega$ |  x302.8 $\pm$ 19.0 |  x247.3 $\pm$ 17.0
$\pi^{+}\pi^{-}\pi^{0}$ | 1280.0 $\pm$ 37.2 |  x951.9 $\pm$ 31.4
$K^{0}_{\rm S}\pi^{0}$ |  x701.3 $\pm$ 26.9 |  x472.5 $\pm$ 21.8
$K^{0}_{\rm S}\omega$ |  x340.7 $\pm$ 19.8 |  x202.0 $\pm$ 15.3
$K^{0}_{\rm S}\phi$ |  x57.5 $\pm$ 8.0 |  x47.8 $\pm$ 7.3
$K^{0}_{\rm S}\eta(\gamma\gamma)$ |  x135.0 $\pm$ 12.1 |  x67.2 $\pm$ 8.4
$K^{0}_{\rm S}\eta(\pi^{+}\pi^{-}\pi^{0})$ |  x37.5 $\pm$ 7.2 |  x27.2 $\pm$ 5.8
$K^{0}_{\rm S}\eta^{\prime}(\pi^{+}\pi^{-}\eta)$ |  x40.1 $\pm$ 6.4 |  x31.7 $\pm$ 5.7
$K^{0}_{\rm S}\pi^{+}\pi^{-}$ | 2206.4 $\pm$ 48.6 |  xx–

The normalisation for the like-sign observables is performed through
measurement of the corresponding opposite-sign yields (e.g.
$K^{-}\pi^{+}\pi^{+}\pi^{-}$ vs. $K^{+}\pi^{-}\pi^{+}\pi^{-}$), which are
negligibly modified by quantum-correlation effects, and from knowledge of the
ratios of the relevant charm-meson branching fractions. The normalisation for
the $\mathit{CP}$ observables $\rho^{K3\pi}_{\mathit{CP}\pm}$ is performed in
one of two ways. For the $D\to K^{+}K^{-}$ and $D\to\pi^{+}\pi^{-}$ tags the
expected number of events in the incoherent limit is calculated through
knowledge of the branching ratios, the total number of $D\bar{D}$ events, as
determined through the yield of opposite sign double-tags, and the
reconstruction efficiency, as measured from simulation. For the other tags,
where the branching fractions and reconstruction efficiencies are less well
known, the normalisation makes use of the corresponding number of events where
the $\mathit{CP}$-tag is reconstructed together with a $D\to K^{-}\pi^{+}$
decay. This treatment requires corrections for quantum-correlation effects in
the normalisation mode, which introduces minor dependence on the
$D^{0}\bar{D}^{0}$ mixing parameters and the ratio $r_{D}^{K\pi}$ between the
DCS and CF $D\to K^{-}\pi^{+}$ amplitudes and accompanying strong-phase
difference $\delta^{K\pi}_{D}$. Again, full details can be found in Ref. [4].
The external values used in these determinations, and their sources, are
summarised in Table 3. There are several small updates with respect to the
values used in Ref. [5], none of which induce significant changes on the
results.

Table 3: Values of branching fractions and other parameters used in the determination of the CLEO-c observables. Parameter | Value | Reference
---|---|---
$\mathcal{B}(D^{0}\to K^{-}\pi^{+}\pi^{+}\pi^{-})$ | $(8.29\pm 0.20)\%$ | [25]
$\frac{\mathcal{B}(D^{0}\to K^{+}\pi^{-}\pi^{-}\pi^{+})}{\mathcal{B}(D^{0}\to K^{-}\pi^{+}\pi^{+}\pi^{-})}$ | $(3.25\pm 0.11)\times 10^{-3}$ | [22]
$\frac{\mathcal{B}(D^{0}\to K^{+}\pi^{-}\pi^{0})}{\mathcal{B}(D^{0}\to K^{-}\pi^{+}\pi^{0})}$ | $(2.20\pm 0.10)\times 10^{-3}$ | [22]
$(r_{D}^{K\pi})^{2}$ | $(0.349\pm 0.004)\%$ | [2]
$\delta_{D}^{K\pi}$ | $(191.8^{+\phantom{0}9.5}_{-14.7})^{\circ}$ | [2]
$x$ | $(0.37\pm 0.16)\%$ | [2]
$y$ | $(0.66^{+0.07}_{-0.10})\%$ | [2]
$\mathcal{B}(D^{0}\to K^{+}K^{-})$ | $(3.96\pm 0.08)\times 10^{-3}$ | [22]
$\mathcal{B}(D^{0}\to\pi^{+}\pi^{-})$ | $(1.402\pm 0.026)\times 10^{-3}$ | [22]
$F^{\pi\pi\pi^{0}}_{+}$ | $0.973\pm 0.017$ | [14]

The assignment of systematic uncertainties for the inclusive tags follows the
same procedure as applied in Ref. [4], where contributions arise from
uncertainties in the external parameters, the finite size of the samples used
in the various normalisations, knowledge of reconstruction efficiencies
(relevant only for the double tags involving the modes $D\to K^{+}K^{-}$ and
$D\to\pi^{+}\pi^{-}$), assumptions involved in the $D\to K^{-}\pi^{+}$
normalisation procedure, and the potential bias from non-uniform acceptance
across the phase space of the signal mode. For $\rho^{K3\pi}_{\mathit{CP}\pm}$
and $\Delta_{\mathit{CP}}^{K3\pi}$ the leading source of systematic error
comes from the uncertainties in the yields of the $D\to K^{-}\pi^{+}$
normalisation samples. An important new component, dominant for the like-sign
observables, accounts for a $\pm 20\%$ uncertainty in the level of residual
contamination from $D\to K^{0}_{\rm S}K^{\mp}\pi^{\pm}$ decays. In addition,
small new contributions are assigned associated with the finite knowledge of
the $\mathit{CP}$-impurity in $D\to\pi^{+}\pi^{-}\pi^{0}$ and the potential
non-$\phi$ contribution to the $D\to K^{0}_{\rm S}K^{+}K^{-}$ sample [15], and
incomplete understanding of the effects of quantum-correlation on the
background sources.

Separate values of $\rho^{K3\pi}_{\mathit{CP}\pm}$ and
$\Delta^{K3\pi}_{\mathit{CP}}$ are calculated for each $\mathit{CP}$ tag. In
the case of $\pi^{+}\pi^{-}\pi^{0}$ a correction factor of
$(2F^{\pi\pi\pi^{0}}_{+}-1)^{-1}$ is applied to the raw result for
$\Delta^{K3\pi}_{\mathit{CP}}$ to account for residual $\mathit{CP}$-odd
contributions to this tagging mode. The individual results for
$\rho^{K3\pi}_{\mathit{CP}+}$ and $\rho^{K3\pi}_{\mathit{CP}-}$ are displayed
in Fig. 2. The results for the mean value of these quantities, and for the
$\mathit{CP}$-invariant observable $\Delta^{K3\pi}_{\mathit{CP}}$, evaluated
taking full account of correlations are given in Table 4. The $\chi^{2}$ for
the twelve measurements in the $\Delta^{K3\pi}_{\mathit{CP}}$ combination is
10.3, which indicates good compatibility. The result for
$\Delta^{K3\pi}_{\mathit{CP}}$ is around $1\sigma$ lower than formerly. This
shift can almost wholly be attributed to the inclusion of the
$D\to\pi^{+}\pi^{-}\pi^{0}$ tag, which returns a value for
$\rho^{K3\pi}_{\mathit{CP}+}$ lower than that of the other $\mathit{CP}$-even
tags, although still compatible. If this contribution is excluded then the
average result becomes $\Delta^{K3\pi}_{\mathit{CP}}=0.087\pm 0.018\pm 0.023$,
which is in excellent agreement with that found in Ref. [5].

Figure 2: Results for the $\rho^{K3\pi}_{\mathit{CP}+}$ and
$\rho^{K3\pi}_{\mathit{CP}-}$ observables for each tag. The error bars give
the total uncertainty on the individual measurements. The blue bands represent
the 1 $\sigma$ bound on the averaged results for each observable.

All the values of the like-sign observables, also presented in Table 4, are
around 1.5$\sigma$ lower than before. These shifts are a consequence of the
improved understanding of the background involving $D\to K^{0}_{\rm
S}K^{\mp}\pi^{\pm}$ decays.

Table 4: Measured values of the $\rho$ and $\Delta$ observables, as determined in the current analysis and reported in the previous analysis [5]. Here the first uncertainty is statistical and the second systematic. Observable | Measured value |  | Previous result
---|---|---|---
$\rho^{K3\pi}_{\mathit{CP}+}$ |  $1.061$ $\pm$ $0.019$ $\pm$ $0.028$ |  |  $1.087$ $\pm$ $0.024$ $\pm$ $0.029$
$\rho^{K3\pi}_{\mathit{CP}-}$ |  $0.926$ $\pm$ $0.027$ $\pm$ $0.042$ |  |  $0.934$ $\pm$ $0.027$ $\pm$ $0.046$
$\Delta^{K3\pi}_{\mathit{CP}}$ |  $0.063$ $\pm$ $0.015$ $\pm$ $0.021$ |  |  $0.084$ $\pm$ $0.018$ $\pm$ $0.022$
$\rho^{K3\pi}_{LS}$ |  $0.757$ $\pm$ $0.239$ $\pm$ $0.122$ |  |  $1.116$ $\pm$ $0.227$ $\pm$ $0.073$
$\rho^{K3\pi}_{K\pi,LS}$ |  $0.719$ $\pm$ $0.168$ $\pm$ $0.077$ |  |  $1.018$ $\pm$ $0.177$ $\pm$ $0.054$
$\rho_{K\pi\pi^{0},LS}^{K3\pi}$ |  $0.919$ $\pm$ $0.158$ $\pm$ $0.098$ |  |  $1.218$ $\pm$ $0.169$ $\pm$ $0.062$

The correlation matrix for the like-sign and $\Delta^{K3\pi}_{\mathit{CP}}$
observables and for those of the analogous quantities for the decay $D\to
K^{-}\pi^{+}\pi^{0}$ may be found in A.

Each double-tagged event involving $D\to K^{0}_{\rm S}\pi^{+}\pi^{-}$ decays
is subjected to a mass-constrained fit of the tagging candidate in order to
determine more reliably its location in the Dalitz plot, and hence the bin
assignment. The background contamination in this sample is below 10% in all
bins. A large sample of Monte Carlo signal events is used to determine the
relative bin-to-bin efficiencies, which all differ by less than 5%. The
resulting values of the $Y_{i}$ observables, after background subtraction and
relative efficiency correction, are presented in Table 5. All sources of
systematic bias are negligible compared with the statistical uncertainties.

Table 5: The $K^{0}_{\rm S}\pi^{+}\pi^{-}$-tagged signal yields, corrected for relative bin-to-bin efficiency effects, normalised to the bin of highest efficiency. Bin | $Y_{i}$ | Bin | $Y_{i}$
---|---|---|---
1 | 354.4 $\pm$ 21.3 |  $-$1 | 180.4 $\pm$ 15.0
2 | 217.2 $\pm$ 16.1 |  $-$2 | 56.8 $\pm$ 8.2
3 | 183.8 $\pm$ 13.7 |  $-$3 | 45.8 $\pm$ 7.0
4 | 62.1 $\pm$ 8.3 |  $-$4 | 41.7 $\pm$ 6.8
5 | 179.2 $\pm$ 14.5 |  $-$5 |  x96.9 $\pm$ 10.6
6 | 110.4 $\pm$ 11.3 |  $-$6 | 33.7 $\pm$ 6.3
7 | 290.8 $\pm$ 18.9 |  $-$7 | 35.7 $\pm$ 6.5
8 | 293.5 $\pm$ 19.1 |  $-$8 | 75.9 $\pm$ 9.4

###  2.3 Fit to the coherence factor and average strong-phase difference

The measured values of the $D\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ observables,
reported in Tables 4 and 5, are input to a $\chi^{2}$ fit to determine
$R_{K3\pi}$, $\delta_{D}^{K3\pi}$ and $r_{D}^{K3\pi}$. The observable
$\rho_{K\pi\pi^{0},LS}^{K3\pi}$ couples the $K^{-}\pi^{+}\pi^{+}\pi^{-}$
results to those of the $D\to K^{-}\pi^{+}\pi^{0}$ system. Therefore the
observables specific to the latter decay, with values taken from Ref. [5], are
also included in the fit, and $R_{K\pi\pi^{0}}$, $\delta_{D}^{K\pi\pi^{0}}$
and $r_{D}^{K\pi\pi^{0}}$ treated as free parameters. 222In previous studies
[4, 5] $r_{D}^{K3\pi}$ and $r_{D}^{K\pi\pi^{0}}$ were not fit parameters. The
change of strategy is motivated by the importance of these parameters in the
$\gamma$ determination, and by the additional information made available
through the LHCb $D\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ mixing analysis [12]. All
known sources of correlation are accounted for.

Full expressions relating the inclusive observables to the underlying physics
parameters can be found in Ref. [4], and involve not only the hadronic
parameters of the two multi-body signal modes, but also $r_{D}^{K\pi}$,
$\delta_{D}^{K\pi}$, $x$ and $y$. These four latter parameters are therefore
also floated in the fit, but with Gaussian constraints to the values of
external measurements. The ratio of the ‘wrong sign’ to the ‘right sign’
decay-time integrated branching fractions, $\frac{\mathcal{B}(D^{0}\to
K^{+}\pi^{-}\pi^{-}\pi^{+})}{\mathcal{B}(D^{0}\to
K^{-}\pi^{+}\pi^{+}\pi^{-})}$, is an important additional measurement. This
observable can be related to $r_{D}^{K3\pi}$ and the other two hadronic
parameters, together with $x$ and $y$, through the time-integrated form of Eq.
4, as given in Ref. [5] (and analogously for the $D^{0}\to
K^{-}\pi^{+}\pi^{0}$ quantities). The values of the external measurements
taken for the ratios of the branching fractions, and for the Gaussian
constraints of the additional fit parameters, are listed in Table 3.

Equation 5 is used to interpret the $Y_{i}$ observables in terms of
$R_{K3\pi}$, $\delta_{D}^{K3\pi}$ and $r_{D}^{K3\pi}$ (and an analogous
expression is employed for the $K^{-}\pi^{+}\pi^{0}$ case). Here also there
are additional parameters, all associated with the $K^{0}_{\rm
S}\pi^{+}\pi^{-}$ system, that are floated in the fit with Gaussian
constraints. The external values taken for the flavour-tagged fractions
$K_{i}$ are those reported in Ref. [5], and arise from a study of the results
of amplitude models developed by the BaBar and Belle collaborations [26, 27,
28, 17]. The values of the strong-phase parameters $c_{i}$ and $s_{i}$ come
from quantum-correlated measurements performed by CLEO [15].

The best fit values and the correlations for the parameters of interest are
given in Tables 6 and 7, respectively. Note that the precision on
$r_{D}^{K3\pi}$ and $r_{D}^{K\pi\pi^{0}}$ is limited by the knowledge of the
ratios of the branching fractions, rather than the CLEO-c data. Also shown are
the previously reported results from Ref. [5]. A significant change is
observed in the value of the $D\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ coherence
factor, which is about 1$\sigma$ higher than previously. This shift is mainly
driven by the change in the like-sign observables. A large change is also
found in the central value of $\delta_{D}^{K3\pi}$, which is the result of the
correction implemented in Eq. 5. As expected there is very little change in
the $D\to K^{-}\pi^{+}\pi^{0}$ results, as here the only input observable that
has evolved is $\rho^{K3\pi}_{K\pi\pi^{0},LS}$. The reduced $\chi^{2}$ of the
fit is 29.5/33, to be compared with 44.4/33 for the previous analysis. Hence
the compatibility of the input observables has improved.

Table 6: Results from the fit to the CLEO-c observables. The uncertainties are the combination of the statistical and systematic uncertainties. The results from the previous update [5], where available, are also shown for comparison. Parameter | Fitted value | Previous result
---|---|---
$R_{K3\pi}$ | $0.53^{+0.18}_{-0.21}$ | $0.32^{+0.20}_{-0.28}$
$\delta_{D}^{K3\pi}$ | $(125^{+22}_{-14})^{\circ}$ | $(255^{+21}_{-78})^{\circ}$
$r_{D}^{K3\pi}$ | $(5.50\pm 0.12)\times 10^{-2}$ | –
$R_{K\pi\pi^{0}}$ | $0.82\pm 0.06$ | $0.82\pm 0.07$
$\delta_{D}^{K\pi\pi^{0}}$ | $(199^{+13}_{-14})^{\circ}$ | $(164^{+20}_{-14})^{\circ}$
$r_{D}^{K\pi\pi^{0}}$ | $(4.48\pm 0.12)\times 10^{-2}$ | –
Table 7: Correlation coefficients between the parameters in the fit to the CLEO-c observables. | $R_{K3\pi}$ | $\delta_{D}^{K3\pi}$ | $r_{D}^{K3\pi}$ | $R_{K\pi\pi^{0}}$ | $\delta_{D}^{K\pi\pi^{0}}$ | $r_{D}^{K\pi\pi^{0}}$
---|---|---|---|---|---|---
$R_{K3\pi}$ | $1.00$ | $-0.47$ | $-0.45$ | $\phantom{-}0.06$ | $-0.01$ | $\phantom{-}0.08$
$\delta_{D}^{K3\pi}$ |  | $\phantom{-}1.00$ | $\phantom{-}0.04$ | $\phantom{-}0.02$ | $\phantom{-}0.24$ | $\phantom{-}0.06$
$r_{D}^{K3\pi}$ |  |  | $\phantom{-}1.00$ | $-0.03$ | $-0.06$ | $-0.04$
$R_{K\pi\pi^{0}}$ |  |  |  | $\phantom{-}1.00$ | $\phantom{-}0.25$ | $-0.03$
$\delta_{D}^{K\pi\pi^{0}}$ |  |  |  |  | $\phantom{-}1.00$ | $-0.04$
$r_{D}^{K\pi\pi^{0}}$ |  |  |  |  |  | $\phantom{-}1.00$

Scans of the $(R_{K3\pi},\delta_{D}^{K3\pi})$ and
$(R_{K\pi\pi^{0}},\delta_{D}^{K\pi\pi^{0}})$ parameter space are shown in Fig.
3. In making these plots the values of $R$ and $\delta_{D}$ are fixed, while
all other parameters are refitted to obtain $\Delta\chi^{2}$, the change in
$\chi^{2}$ with respect to the lowest value found.

|
---|---
Figure 3:  Scans of $\Delta\chi^{2}$ for the fit to the updated CLEO-c
observables in the (left) $(R_{K3\pi},\delta_{D}^{K3\pi})$ and (right)
$(R_{K\pi\pi^{0}},\delta_{D}^{K\pi\pi^{0}})$ parameter space, showing the
$\Delta\chi^{2}=1,4$ and $9$ intervals.

##  3 Constraints on the coherence factor and average strong-phase difference
from LHCb data

The LHCb collaboration has performed a study of the time-dependence of the
ratio between $D^{0}\to K^{+}\pi^{-}\pi^{+}\pi^{-}$ and $D^{0}\to
K^{-}\pi^{+}\pi^{+}\pi^{-}$ decay rates [12]. Several sets of results are
reported, including those given in Table 8 from a fit for $r_{D}^{K3\pi}$ and
the parameters $a$ and $b$, assuming the functional form given in Eq. 4.

Table 8: Results from the ‘unconstrained’ time-dependent $D^{0}\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ analysis of LHCb [12]. Parameter | Result
---|---
$r_{D}^{K3\pi}$ | $(5.67\pm 0.12)\times 10^{-2}$
$a$ | $(0.3\pm 1.8)\times 10^{-3}$
$b$ | $(4.8\pm 1.8)\times 10^{-5}$

Figure 4 shows a $\Delta\chi^{2}$ scan of the $(R_{K3\pi},\delta_{D}^{K3\pi})$
parameter space obtained from the LHCb results, and imposing Gaussian
constraints on the mixing parameters $x$ and $y$ according to the measured
values in Table 3. The shape of the contours is significantly different to
those obtained from the fit to the CLEO-c observables, therefore motivating a
combined fit of both sets of measurements.

Figure 4:  Scan of $\Delta\chi^{2}$ in $(R_{K3\pi},\delta_{D}^{K3\pi})$
parameter space corresponding to the results of the time-dependent $D^{0}\to
K^{-}\pi^{+}\pi^{+}\pi^{-}$ analysis of LHCb.

##  4 Combined fit

The fit described in Sect. 2.3 is repeated with the LHCb
$D^{0}\bar{D}^{0}$-mixing results (reported in Table 8) included as additional
input measurements. The best fit values for the hadronic parameters, and the
associated correlations, are presented in Tables 9 and 10, respectively. The
reduced $\chi^{2}$ of the fit is 33.5/36. Figure 5 shows the three possible
sets of two-dimensional scans in the $D\to K^{-}\pi^{+}\pi^{+}\pi^{-}$
hadronic-parameter space; also shown is a scan of
$(R_{K\pi\pi^{0}},\delta_{D}^{K\pi\pi^{0}})$. The inclusion of the LHCb
observables improves the precision of the $D\to K^{-}\pi^{+}\pi^{+}\pi^{-}$
coherence factor, but lowers the central value with respect to that returned
by the CLEO-c fit. In this region the 1$\sigma$ bound on $\delta_{D}^{K3\pi}$
is weaker, although the results for this parameter become significantly more
Gaussian in behaviour. The reduction in the uncertainty on $r_{D}^{K3\pi}$ is
largely driven by the correlation with the mixing parameters $x$ and $y$,
which are constrained through external measurements in the fit. As expected
there are only minor changes in the $D\to K^{-}\pi^{+}\pi^{0}$ results
compared to those obtained from the fit to the CLEO-c observables alone.

Table 9: Results from the combined fit to the updated CLEO-c and LHCb observables. The uncertainties are the combination of the statistical and systematic uncertainties.  Parameter | Fitted value
---|---
$R_{K3\pi}$ | $0.43^{+0.17}_{-0.13}$
$\delta_{D}^{K3\pi}$ | $(128^{+28}_{-17})^{\circ}$
$r_{D}^{K3\pi}$ | $(5.49\pm 0.06)\times 10^{-2}$
$R_{K\pi\pi^{0}}$ | $0.81\pm 0.06$
$\delta_{D}^{K\pi\pi^{0}}$ | $(198^{+14}_{-15})^{\circ}$
$r_{D}^{K\pi\pi^{0}}$ | $(4.47\pm 0.12)\times 10^{-2}$
Table 10: Correlation coefficients between the parameters from the combined fit to the updated CLEO-c and LHCb observables. | $R_{K3\pi}$ | $\delta_{D}^{K3\pi}$ | $r_{D}^{K3\pi}$ | $R_{K\pi\pi^{0}}$ | $\delta_{D}^{K\pi\pi^{0}}$ | $r_{D}^{K\pi\pi^{0}}$
---|---|---|---|---|---|---
$R_{K3\pi}$ | $1.00$ | $-0.67$ | $-0.48$ | $\phantom{-}0.03$ | $-0.05$ | $-0.04$
$\delta_{D}^{K3\pi}$ |  | $\phantom{-}1.00$ | $\phantom{-}0.12$ | $\phantom{-}0.02$ | $\phantom{-}0.15$ | $\phantom{-}0.08$
$r_{D}^{K3\pi}$ |  |  | $\phantom{-}1.00$ | $-0.04$ | $-0.02$ | $-0.03$
$R_{K\pi\pi^{0}}$ |  |  |  | $\phantom{-}1.00$ | $\phantom{-}0.23$ | $-0.04$
$\delta_{D}^{K\pi\pi^{0}}$ |  |  |  |  | $\phantom{-}1.00$ | $-0.03$
$r_{D}^{K\pi\pi^{0}}$ |  |  |  |  |  | $\phantom{-}1.00$
|
---|---
|
Figure 5:  Scans of $\Delta\chi^{2}$ for the combined fit to the updated
CLEO-c and LHCb observables in the (clockwise from top left)
$(R_{K3\pi},\delta_{D}^{K3\pi})$, $(R_{K3\pi},r_{D}^{K3\pi})$,
$(\delta_{D}^{K3\pi},r_{D}^{K3\pi})$ and
$(R_{K\pi\pi^{0}},\delta_{D}^{K\pi\pi^{0}})$ parameter space.

##  5 Conclusions

A re-analysis of the CLEO-c $\psi(3770)$ data set has yielded an updated set
of observables sensitive to the hadronic parameters of the decay $D\to
K^{-}\pi^{+}\pi^{+}\pi^{-}$, some of which are significantly different to
those reported previously [4, 5]. These observables have been input to a
combined fit, together with measurements from a recent LHCb $D^{0}\bar{D}^{0}$
mixing analysis [12]. Results are obtained for $R_{K3\pi}$ and $r_{D}^{K3\pi}$
that are significantly more precise than those derived from the CLEO-c
observables alone. New values and constraints are also determined for the
hadronic parameters of the decay $D\to K^{-}\pi^{+}\pi^{0}$. These results
will be valuable for improving sensitivity to the unitarity triangle angle
$\gamma$ with analyses exploiting $B^{-}\to DK^{-}$ decays. The combined fit
can be re-performed when future measurements of the $\psi(3770)$ observables
become available from the BESIII collaboration, or when improved
$D^{0}\bar{D}^{0}$ mixing results are reported by either LHCb or Belle-II.

## Acknowledgments

This analysis was performed using CLEO-c data. The authors of this Letter
(some of whom were members of CLEO) are grateful to the collaboration for the
privilege of using these data. We also thank Roy Briere, David Cassel, Tim
Gershon and Sheldon Stone for their careful reading of the draft manuscript
and valuable suggestions. We are grateful for support from the UK Science and
Technology Facilities Council, the UK India and Education Research Initiative
and the European Research Council under FP7.

##  Appendix A Correlation matrix for the observables measured with the CLEO-c
data

The correlation matrix for the $D\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ and $D\to
K^{-}\pi^{+}\pi^{0}$ inclusive observables is presented in Table 11. The
$Y_{i}$ observables are all uncorrelated.

Table 11: Correlation matrix for the $D\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ and $D\to K^{-}\pi^{+}\pi^{0}$ inclusive observables. | $\Delta^{K3\pi}_{\mathit{CP}}$ | $\rho^{K3\pi}_{LS}$ | $\rho^{K3\pi}_{K\pi,LS}$ | $\rho_{K\pi\pi^{0},LS}^{K3\pi}$ | $\Delta^{K\pi\pi^{0}}_{\mathit{CP}}$ | $\rho^{K\pi\pi^{0}}_{LS}$ | $\rho^{K\pi\pi^{0}}_{K\pi,LS}$
---|---|---|---|---|---|---|---
$\Delta^{K3\pi}_{\mathit{CP}}$ | 1.00 | 0.01 | 0.01 | 0.00 | 0.60 | 0.00 | 0.00
$\rho^{K3\pi}_{LS}$ |  | 1.00 | 0.16 | 0.19 | 0.00 | 0.00 | 0.00
$\rho^{K3\pi}_{K\pi,LS}$ |  |  | 1.00 | 0.20 | 0.00 | 0.00 | 0.00
$\rho_{K\pi\pi^{0},LS}^{K3\pi}$ |  |  |  | 1.00 | 0.03 | 0.02 | 0.00
$\Delta^{K\pi\pi^{0}}_{\mathit{CP}}$ |  |  |  |  | 1.00 | 0.03 | 0.01
$\rho^{K\pi\pi^{0}}_{LS}$ |  |  |  |  |  | 1.00 | 0.01
$\rho^{K\pi\pi^{0}}_{K\pi,LS}$ |  |  |  |  |  |  | 1.00

## References

  * [1] D. Atwood and A. Soni, Role of charm factory in extracting CKM phase information via $B\to DK$, Phys. Rev.  D 68 (2003) 033003, arXiv:hep-ph/0304085.
  * [2] Y. Amhis et al. (HFAG), Averages of b-hadron, c-hadron, and $\tau$-lepton properties as of summer 2014, arXiv:1412.7515, online updates at http://www.slac.stanford.edu/xorg/hfag. Numbers taken from ‘Charm 2015’ update.
  * [3] M. Rama, Effect of $D-\bar{D}$ mixing in the extraction of $\gamma$ with $B^{-}\to D^{0}K^{-}$ and $B^{-}\to D^{0}\pi^{-}$ decays, Phys. Rev. D 89 (2014) 014021, arXiv:1307.4384 [hep-ex].
  * [4] N. Lowrey et al. (CLEO collaboration), Determination of the $D^{0}\to K^{-}\pi^{+}\pi^{0}$ and $D^{0}\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ coherence factors and average strong-phase differences using quantum-correlated measurements, Phys. Rev. D 80 (2009) 031105(R), arXiv:0903.4853 [hep-ex].
  * [5] J. Libby et al., New determination of the $D^{0}\to K^{-}\pi^{+}\pi^{0}$ and $D^{0}\to K^{-}\pi^{+}\pi^{+}\pi^{-}$ coherence factors and average strong-phase differences using quantum-correlated measurements, Phys. Lett. B 731 (2014) 197, arXiv:1401.1904 [hep-ex].
  * [6] R. Aaij et al. (LHCb collaboration), Observation of the suppressed ADS modes $B^{\pm}\to[\pi^{\pm}K^{\mp}\pi^{+}\pi^{-}]_{D}K^{\pm}$ and $B^{\pm}\to[\pi^{\pm}K^{\mp}\pi^{+}\pi^{-}]_{D}\pi^{\pm}$, Phys. Lett. B 723 (2013) 44, arXiv:1304.4646 [hep-ex].
  * [7] R. Aaij et al. (LHCb collaboration), A study of $\mathit{CP}$ violation in $B^{\mp}\to Dh^{\mp}$ ($h=K,\pi$) with the modes $D\to K^{\mp}\pi^{\pm}\pi^{0}$, $D\to\pi^{+}\pi^{-}\pi^{0}$ and $D\to K^{+}K^{-}\pi^{0}$, Phys. Rev. D 91 (2015) 112014, arXiv:1504.05442 [hep-ex].
  * [8] LHCb collaboration, Improved constraints on $\gamma$: CKM2014 update, LHCb-CONF-2014-004; R. Aaij et al. (LHCb collaboration), Measurement of the CKM angle $\gamma$ from a combination of $B\to Dh$ analyses, Phys. Lett. B 726 (2013) 151, arXiv:1305.2050 [hep-ex].
  * [9] S. Malde and G. Wilkinson, $D^{0}-\bar{D}{}^{0}$ mixing studies with the decays $D^{0}\to K^{0}_{S}K^{\mp}\pi^{\pm}$, Phys. Lett. B 701 (2011) 353, arXiv:1104.2731 [hep-ph].
  * [10] S. Harnew and J. Rademacker, Charm mixing as input for model-independent determination of the CKM phase $\gamma$, Phys. Lett. B 728 (2014) 296, arXiv:1309.0134 [hep-ph].
  * [11] S. Harnew and J. Rademacker, Model independent determination of the CKM phase $\gamma$ using input from $D^{0}-\bar{D}^{0}$ mixing, JHEP 03 (2015) 169, arXiv:1412.7254 [hep-ph].
  * [12] R. Aaij et al. (LHCb collaboration), First observation of $D^{0}-\bar{D}^{0}$ oscillations in $D^{0}\to K^{+}\pi^{-}\pi^{+}\pi^{-}$ decays and a measurement of the associated coherence parameters, Phys. Rev. Lett. 116 (2016) 241801, arXiv:1602.07224 [hep-ex].
  * [13] T. Gershon, J. Libby and G. Wilkinson, Contributions to the width difference in the neutral $D$ system from hadronic decays, Phys. Lett. B 750 (2015) 338; arXiv:1506.08594 [hep-ph].
  * [14] S. Malde et al., First determination of the $\mathit{CP}$ content of $D\to\pi^{+}\pi^{+}\pi^{-}\pi^{-}$ and updated determination of the $\mathit{CP}$ contents of $D\to\pi^{+}\pi^{-}\pi^{0}$ and $D\to K^{+}K^{-}\pi^{0}$, Phys. Lett. B 747 (2015) 9, arXiv:1504.05878 [hep-ex]; M. Nayak et al., First determination of the $\mathit{CP}$ content of $D\to\pi^{+}\pi^{-}\pi^{0}$ and $D\to K^{+}K^{-}\pi^{0}$, Phys. Lett. B 740 (2015) 1, arXiv:1410.3964 [hep-ex].
  * [15] J. Libby et al. (CLEO collaboration), Model-independent determination of the strong-phase difference between the decays $D^{0}$ and $\bar{D}{}^{0}\to K^{0}_{\rm S,L}h^{+}h^{-}$ ($h=\pi,K)$ and its impact on the measurement of the CKM angle $\gamma$, Phys. Rev. D 82 (2010) 112006, arXiv:1010.2817 [hep-ex].
  * [16] A. Bondar and A. Poluektov, Feasibility study of model-independent approach to $\phi_{3}$ measurement using Dalitz plot analysis, Eur. Phys. J. C 47 (2006) 347, arXiv:hep-ph/0510246; A. Bondar and A. Poluektov, The use of quantum-correlated $D^{0}$ decays for $\phi_{3}$ measurement, Eur. Phys. J. C 55 (2008) 51, arXiv:0801.0840 [hep-ex].
  * [17] B. Aubert et al. (BaBar collaboration), Improved measurement of the CKM angle $\gamma$ in $B^{\mp}\to D^{(*)}K^{(*)\mp}$ decays with a Dalitz plot analysis of $D$ decays to $K^{0}_{S}\pi^{+}\pi^{-}$ and $K^{0}_{S}K^{+}K^{-}$, Phys. Rev. D 78 (2008) 034023., arXiv:0804.2089 [hep-ex].
  * [18] A. Bondar, A. Poleuktov and V. Vorobiev, Charm mixing in the model-independent analysis of correlated $D^{0}-\bar{D}{}^{0}$ decays, Phys. Rev. D 82 (2010) 034033, arXiv:1004.2350 [hep-ex].
  * [19] Y. Kubota et al. (CLEO collaboration), The CLEO II detector, Nucl. Instrum. Meth. A 320 (1992) 66; D. Peterson et al., The CLEO III drift chamber, Nucl. Instrum. Meth. A 478 (2002) 142; M. Artuso et al., Construction, pattern recognition and performance of the CLEO III LiF-TEA RICH detector, Nucl. Instrum. Meth. A 502 (2003) 91; R.A. Briere et al. (CLEO-c/CESR-c Taskforces and CLEO-c Collaboration), CLEO-c and CESR-c: a new frontier of weak and strong interactions, Cornell LEPP Report CLNS Report No. 01/1742 (2001).
  * [20] D.J. Lange, The EvtGen particle decay simulation package, Nucl. Instrum. Meth. A 462 (2001) 152.
  * [21] R. Brun et al., GEANT 3.21, CERN Program Library Long Writeup W5013, unpublished.
  * [22] Particle Data Group, K.A. Olive et al., Review of particle physics, Chin. Phys. C 88 (2014) 090001.
  * [23] J. Insler et al., Studies of the decays $D^{0}\to K^{0}_{\rm S}K^{-}\pi^{+}$ and $D^{0}\to K^{0}_{\rm S}K^{+}\pi^{-}$, Phys. Rev. D 85 (2012) 092016, arXiv:1203.3804 [hep-ex].
  * [24] R. Aaij et al. (LHCb collaboration), Studies of the resonance structure in $D^{0}\to K^{0}_{\rm S}K^{\pm}\pi^{\mp}$ decays, Phys. Rev. D 93 (2016) 052018, arXiv:1509.06628 [hep-ex].
  * [25] X. Shi et al. (CLEO collaboration), Updated measurements of absolute $D^{\pm}$ and $D^{0}$ hadronic branching fractions and $\sigma(e^{+}e^{-})\to D\overline{D}$, Phys. Rev. D 85 (2012) 092016, arXiv:1312.6775 [hep-ex].
  * [26] A. Poluektov et al. (Belle collaboration), Evidence for direct $\mathit{CP}$ violation in the decay $B\to D^{(*)}K$, $D\to K^{0}_{\rm S}\pi^{+}\pi^{-}$ and measurement of the CKM phase $\phi_{3}$, Phys. Rev. D 81 (2010) 112002, arXiv:1003.3360 [hep-ex].
  * [27] P. del Amo Sanchez et al. (BaBar collaboration), Evidence for direct $\mathit{CP}$ violation in the measurement of the CKM angle $\gamma$ with $B^{\mp}\to D^{(*)}K^{(*)\mp}$ decays, Phys. Rev. Lett. 105 (2010) 121801, arXiv:1005.1096 [hep-ex].
  * [28] B. Aubert et al. (BaBar collaboration), Measurement of the Cabibbo-Kobayashi-Maskawa angle $\gamma$ in $B^{\mp}\to D^{(*)}K^{\mp}$ decays with a Dalitz analysis of $D^{0}\to K^{0}_{\rm S}\pi^{-}\pi^{+}$, Phys. Rev. Lett. 95 (2005) 121802, arXiv:hep-ex/0504039.


11institutetext: Dept. of Computing, Imperial College London, London UK
22institutetext: FAU Erlangen-Nuremberg, Erlangen, Germany

# Is More Data All You Need? A Causal Exploration

Athanasios Vlontzos 11 Hadrien Reynaud 11 Bernhard Kainz 1122

###### Abstract

Curating a large scale medical imaging dataset for machine learning
applications is both time consuming and expensive. Balancing the workload
between model development, data collection and annotations is difficult for
machine learning practitioners, especially under time constraints. Causal
analysis is often used in medicine and economics to gain insights about the
effects of actions and policies. In this paper we explore the effect of
dataset interventions on the output of image classification models. Through a
causal approach we investigate the effects of the quantity and type of data we
need to incorporate in a dataset to achieve better performance for specific
subtasks. The main goal of this paper is to highlight the potential of causal
analysis as a tool for resource optimization for developing medical imaging ML
applications. We explore this concept with a synthetic dataset and an
exemplary use-case for Diabetic Retinopathy image analysis.

###### Keywords:

Causality Data Analysis.

##  1 Introduction

Translating deep learning methods to new applications in the clinic often
start with two questions that are very difficult to answer: How much data to
collect and what data aspects need more attention than others to meet clinical
performance expectations. In diagnostic settings, performance is often
characterized by a biased metric, for example an expectation towards zero
false negatives so that no signs of disease are missed but some leeway towards
false positives, which can be mitigated with further diagnostic tests.
However, this commonly leads to situations where end-users request from
machine learning practitioners to make specific interventions on well-
performing models, for example to make a deep neural network more sensitive
towards one specific class of disease or to change predictions for a selected
group of patients, while keeping its sensitivity and specificity intact for
other classes.

Active learning [8], where a probabilistic uncertainty based approach to
decide on new samples to label, is one option to make such interventions on a
working model but it has been shown that the introduced bias is not
necessarily beneficial and might harm a model’s specificity [14]. Furthermore,
there is currently no method to estimate for how long further (active)
learning should go ahead or how many more samples of specific classes have to
be collected until the expected change can be observed. This has critical
implications for practical translation of such methods into the clinical
practice since time, costs and amount of data cannot be estimated in advance,
which in turn conflicts with the need for data minimization as recommended by
General Data Protection Regulations [31].

We believe that methods like ours should be integral parts of regulation
approval processes. As models undergo fine-tuning and retraining in the
process of development and commercialization we should maintain the same high
standards of accuracy and robustness. We need hence to be able to provide
guarantees that the resulting ML models cannot degrade in performance.

Therefore, the need arises to be able to reason about the data needs of an
application and decide upon the best allocation of resources. Moving forward
from the well known active learning paradigm we are looking at more targeted
intervention scenario and provide in this paper a causal approach that allows
to estimate how much _extra_ data is needed for targeted interventions on
trained deep learning models. We show on a synthetic dataset and an exemplary
large Diabetic Retinopathy (DR) medical imaging dataset how to use our
approach.

Contribution: We treat the aforementioned scenarios as a counterfactual meta
analysis upon a static model. Our goal is to highlight causal analysis as a
potential alternative to active learning and showcase the powerful insights it
could yield. Interestingly, we found that it is not always advantageous to
increase the size of a dataset.

Related Work: Recent works on the field of model performance analysis have
primarily been focused on determining the required number of samples to
achieve. [15] developed a inverse power law model to predict model performance
with different data sizes. [11, 7] performed empirical studies on the learning
behavior of classifiers to determine sample size requirements. None, however,
of the above methods are able to determine the effect of interventions on
individual samples, which is the primary concern of this investigation.
Causality, on the other hand, is the field of analysis of causal relationships
between variables. The field was expanded to computer science by the works of,
among others, J. Pearl [24], however few works exist on the intersection of
machine learning, medical imaging and causal analysis. Recently, discussed as
useful for medical image analysis [10], More commonly such approaches are
found in fields like econometrics [16], epidemiology [12] and clinical
medicine[20, 25, 22]. We are borrowing inspiration from these works to argue
the potential advantages of causal analysis of medical imaging machine
learning algorithms.

##  2 Method

Preliminaries In this section we will first introduce key concepts and main
mathematical tools of our analysis, finally in Sections 4 and 5 we will detail
our results and discuss the significance of our analysis.

We work in the Structural Causal Models (SCM) framework. Chapter $7$ of [24]
gives an in-depth discussion. For an up-to-date, self-contained review of
counterfactual inference and Pearl’s Causal Hierarchy, see [6].

######  Definition 1 (Structural Causal Model)

A structural causal model (SCM) specifies a set of latent variables
$U=\\{u_{1},\dots,u_{n}\\}$ distributed as $P(U)$, a set of observable
variables $=\\{v_{1},\dots,v_{m}\\}$, a directed acyclic graph (DAG), called
the _causal structure_ of the model, whose nodes are the variables $U\cup V$,
a collection of functions $F=\\{f_{1},\dots,f_{n}\\}$, such that
$v_{i}=f_{i}(PA_{i},u_{i}),\text{ for }i=1,\dots,n,$ where $PA$ denotes the
parent observed nodes of an observed variable.

The collection of functions $F$ and the distribution over latent variables
induces a distribution over observable variables: $P(V=v):=\sum_{\\{u_{i}\mid
f_{i}(PA_{i},u_{i})\,=\,v_{i}\\}}\\\ P(u_{i})$ In this manner, we can assign
uncertainty over observable variables despite the fact the underlying dynamics
are deterministic.

Moreover, the $do$-operator forces variables to take certain values,
regardless of the original causal mechanism. Graphically, $do(X=x)$ means
deleting edges incoming to $X$ and setting $X=x$. Probabilities involving
$do(x)$ are normal probabilities in submodel $M_{x}$:
$P(Y=y\mid\text{do}(X=x))=P_{M_{x}}(y)$.

Counterfactual inference The latent distribution $P(U)$ allows one to define
probabilities of counterfactual queries, $P(Y_{y}=y)=\sum_{u\mid
Y_{x}(u)=y}P(u).$ For $x\neq x^{\prime}$ one can also define joint
counterfactual probabilities,
$P(Y_{x}=y,Y_{x^{\prime}}=y^{\prime})=\sum_{u\mid Y_{x}(u)=y,\text{
}\&Y_{x^{\prime}}(u)=y^{\prime}}P(u).$ Moreover, one can define a
counterfactual distribution given seemingly contradictory evidence. Given a
set of observed evidence variables $E$, consider the probability
$P(Y_{{x}}={y}^{\prime}\mid E={e})$.

Definition 2(Counterfactual):_The counterfactual sentence “ $Y$ would be $y$
(in situation $U=u$), had $X$ been $x$”, denoted $Y_{x}(u)=y$, corresponds to
$Y=y$ in submodel $M_{x}$ for $U=u$._

Despite the fact that this query may involve interventions that contradict the
evidence, it is well-defined, as the intervention specifies a new submodel.
Indeed, $P(Y_{x}={y}^{\prime}\mid E={e})$ is given by [24]
$\sum_{{u}}P(Y_{{x}}({u})={y}^{\prime})P({u}|{e})\,.$ There are two main ways
to resolve this type of questions; the Abduction-Action-Prediction paradigm
and the Twin Network paradigm shown respectively in ML literature among others
in [10, 30]. In short given SCM $M$ with latent distribution $P(U)$ and
evidence ${e}$, the conditional probability $P(Y_{{x}}\mid{e})$ is evaluated
as follows: 1) _Abduction:_ Infer the posterior of the latent variables with
evidence ${e}$ to obtain $P(U\mid{e})$, 2) _Action:_ Apply $\text{do}({x})$ to
obtain submodel $M_{x}$, 3) _Prediction:_ Compute the probability of $Y$ in
the submodel $M_{x}$ with $P(U\mid{e})$. Meanwhile the Twin Network paradigm
casts the resolution of counterfactual queries to Bayesian feed-forward
inference by extending the SCM to represent both factual and counterfactual
worlds at once [5].

Analysis framework As discussed above and expanded in Section 5 there exists a
wide range of possible methods that enable accurate estimation of
counterfactual quantities in real world scenarios [26, 10, 30]. For the scope
of this paper we are assuming perfect knowledge and as such we revert to the
mathematical tools that counterfactual inference methods approximate. We are
going to be mainly concerned about the effect of datasets on the
classification outcome of the samples. We treat the model architecture and its
hyper parameters as confounders that affect only the outcome and are invariant
between different treatments of our dataset. We intervene on the size and
composition of our dataset leaving all other parameters the same. In Figure 1
we show a sample directed acyclic graph (DAG) for the causal relationships
between the variables we take into account for our analysis. For simplicity
for each counterfactual we explore only interventions on one of the possible
treatments.

Figure 1: A DAG showing the causal relationships between the factors we are
analyzing - Yellow: Observed variables - Blue: Unobserved Variables. For
simplicity we intervene on one of the possible treatment variables during each
experiment.

Our main research question is a counterfactual one; given a sample was
incorrectly classified would this sample be correctly classified if we had
trained our model with a different dataset? Mathematically this resembles the
probability Sufficiency $P_{S}=P(Y_{X=T}=y\mid X=T^{\prime},Y=y^{\prime})$ and
can be described as:

$P(Y=1,do(X=\hat{\mathcal{D}})\mid Y=0,X=\mathcal{D},Z)$ (1)

where Y is the outcome of whether or not the sample was correctly classified;
X represents the treated dataset $\mathcal{D}$ and Z the confounders like the
architecture of the model. Our analysis regards the model as a black box and
its underlying architecture complexity is not a constraint. We use this
question as an example for a potential causal analysis on a medical image
model. We highlight that multiple counterfactual questions can be asked,
concerning for example the model architecture or a specific feature of the
data. We chose interventions on the overall statistics of the dataset as an
entry level counterfactual question with evident real world impact on ML
practice.

##  3 Evaluation

Datasets: We use two datasets, one synthetic and one real medical imaging
database. As synthetic data we use the MorphoMNIST [9] dataset. This dataset
provides a series of morphological operations upon the digits of the well
known MNIST dataset [19]. Each of the generated datasets is modified by random
fractures and swellings. In order to control this perturbation in our dataset
throughout the experiments each sample retains the same morphological
perturbation. As this dataset is fully controlled and synthetic we can draw
new never seen before samples each time, simulating new data gathering
regimes.

Furthermore, we chose Kaggle’s Retinopathy open source dataset [1] as medical
imaging dataset. We treat each of the images as a gray scale image and resize
it down to $128\times 128px$. As this is a multi-class dataset with heavy
imbalance, we focus on the following labels for Diabetic Retinopathy (DR)
levels : ”No DR”, ”Mild” and ”Moderate” cases.

Model architecture: As a first step we train a classifier with the full
training set. The goal of this step is to determine an architecture and set of
hyper-parameters that are able to provide acceptable classification results.
Having determined such parameters we fix them for each counterfactual query as
to determine the true causal effect of our dataset interventions on the
probability of correct classification of a sample. For the synthetic case we
opt for a 3 layered multi-layered Perceptron while for the medical use case a
simple multi scale residual convolutional net. The residual network is
comprised of 4 residual blocks with decreasing resolution. While in the
synthetic case we opt for a simple model and in the medical one with a
significantly more complex one, we note that this is not a parameter that
imposes any constraints on our analysis. Exact architectures can be found in
the Appendix and implementations will be made public together with the
codebase.

Interventions: Interventions are focused on the size and composition of each
dataset. First we explore the effect of the size of the dataset. Given that
both the synthetic and medical tasks have an abundance of data we create a
series of datasets with $[100,1000,5000,10000]$ samples for the synthetic
dataset and $[100,500,1000,2000,4000]$ samples from the medical dataset. We
use the intervened datasets to train our models and a static test dataset that
includes $20\%$ of the full dataset. This serves as a gold standard evaluation
set. Moreover, we intervene upon the dataset by modulating the number of
samples in selected classes as well as the percentage of the base dataset by
which we will increase a class. In other words given a base dataset of length
$N_{dataset}$ with approximate class balance and an upsampling percentage of
$10\%$ with class $2$, we will add $10\%\times N_{dataset}$ that has unseen
samples of class $2$. We explore upsampling all available classes by one of
the following percentages $0\%,5\%,10\%,20\%,30\%,50\%$

For example assuming an intervention of dataset size from $100$ to $1000$ we
are going to be calculating the measure:

$\begin{split}Query&=P(Y_{do(N=1000)}=1\mid Y=0,N=100,Z)\\\
&=\frac{P(Y=1,N=1000,Z)\wedge P(Y=0,N=100,Z)}{P(Y=0,N=100,Z)}\end{split}$ (2)

Note that this simplification of the query is possible as we assume full
knowledge of the simple DAG of Figure 1. In more complex scenarios proper
controlling for front- and back-door criteria should be happen. In addition in
our simple scenario each of the aforementioned probabilities can be calculated
as the average number of events _i.e._ for a model trained in the regime of
confounders $Z$ and $1000$ training samples:

$P(Y=1,N=1000,Z)=\frac{\\#\text{correctly classified samples}}{\\#\text{total
samples}}$ (3)

##  4 Results

Synthetic Data First we evaluate the synthetic task. In Table 1(a) we show the
effect of including more data regardless of the type of data given a base
dataset. We notice for example that if we had 5k samples and switched to a
dataset with 10k incorporating randomly selected samples that would give us
only a 9% chance of correctly classifying a sample that was previously
misclassified. As a general observation we can see that the more samples are
contained in our _base dataset_ the smaller the probability of a specific
sample being correctly classified after the intervention of including more
images. A proxy to this insight can be seen in Table 1(b) where we show the
average F1 score for all classes for different sized datasets. We see that
indeed the overall improvement between 5 and 10 thousand samples is quite
small. Meanwhile in Figures 3 and 3 we show the probability of a sample
“flipping” from being misclassified to being correctly classified versus the
class we upsample and the percentage of upsampling. In both cases we look at
any given sample without taking into account the ground truth label of it. It
is evident from both figures that there is no single class or percentage of
upsampling that is key for this dataset – in other words, there is no category
of samples that contain key information to help the classification task,
rather all classes seem to be necessary.

Table 1: MorphoMNIST if we change the number of samples regardless of class
and percentage. (A) The probability of changing a misclassified sample to a
correctly classified one is shown.(B) F1 performance of different sized base
datasets

(a) | # of samples
---
From / To
100 | 1000 | 5000 | 10000
100 | 0 | 23.17% | 27.18% | 27.72%
1000 | 4.17% | 0 | 17.57% | 18.93%
5000 | 3.95% | 7.22% | 0 | 9.46%
10000 | 3.94% | 7.20% | 7.25% | 0

(b) | F1
---|---
100 | 30.606%
1000 | 75.97%
5000 | 85.54%
10000 | 86.84%

Figure 2: Probability of flip; Treatment: Upsampling class - see legend

Figure 3: Probability of flip; Treatment: Upsampling percentage of base
dataset - see legend

Thus far we have increased class samples and adding more data to our base
dataset without taking into consideration the true class of the misclassified
data. In the next step we look into informed interventions where we include a
larger number of samples where the majority of them are from the class which
was misclassified. In Table 2 we show the effects of different dataset sizes
and percentages of upsampling on the probability of a specific sample being
correctly classified after the intervention. Compared to incorporating more
data randomly or upsampling a class regardless of the misclassfied class we
observe a significantly increased effect. If, conversely, we do not specify
the extend we upsample the misclassified class we average $67.56\%$
probability of correctly classifying a sample after the intervention across
all possible percentages and dataset sizes. It is evident, thus, that we can
obtain a higher probability of sufficiency if our interventions on the dataset
are targeted. We further note that during this analysis we look at the inverse
probability of incorrectly classifying a sample that was correctly determined
before treatment, in all our cases this probability did not exceed $3-4\%$
indicating that our chosen interventions did not have a negative effect upon
the model performance.

(a) | 100 | 1000 | 5000 | 10000
---|---|---|---|---
full dataset | 7.71 | 21.18 | 23.35 | 24.48
100 | 0 | 29.16 | 30.12 | 30.32
1000 | 19.34 | 0 | 24.30 | 24.92
5000 | 18.70 | 17.41 | 0 | 18.58
10000 | 18.49 | 16.94 | 16.69 | 0

(b) | 0.05 | 0.1 | 0.2 | 0.3 | 0.5
---|---|---|---|---|---
full dataset | 8.13 | 9.21 | 11.49 | 13.86 | 15.72
0.05 | 0 | 11.50 | 14.36 | 17.16 | 19.61
0.1 | 9.99 | 0 | 14.64 | 17.21 | 19.66
0.2 | 10.64 | 12.03 | 0 | 17.53 | 19.83
0.3 | 11.13 | 12.68 | 15.27 | 0 | 20.02
0.5 | 11.86 | 13.27 | 15.87 | 18.27 | 0

Table 2: Informed Interventions (A) the effect of different dataset sizes (B)
the effect of different upsampling percentages

Retinopathy: we follow a similar analysis for the medical image data where we
classify some of the most abundant categories of the open source Retinopathy
dataset. In the interest of space we only include some results here, full
tables can be found in the Appendix. In Table 3 we show the two most
interesting results. In this real world dataset we observe that incorporating
more of the moderate DR class leads to all together better classification
performance regardless of the dataset size. On the other hand modulating the
overall number of samples under an informed sampling regime seems to be driven
primarily by our informed sampling than the actual changes in number of data.
We note that increasing the datasize from 100 to 2000 by randomly sampling
from the available classes only provides a $\sim 10\%$ chance of a sample
flipping. Medical imaging datasets can offer interesting insights if looked
under a causal prism. It is possible to identify inter-dependencies of classes
and features and hence able to plan the dataset acquisition and annotations
more efficiently.

##  5 Discussion

We analyzed the effect of dataset size and composition on the probability of a
specific misclassified sample to become correctly classified after our
interventions. We have observed a wide expected range for this causal
probability. If used in practice to analyse a phenomenon and determine the
best allocation of resources certain thresholds that make sense have to be
determined by the users. Contrary to the well known active learning paradigm
where we are interested in the effect of an intervention on the overall
metrics of our task; by assuming a causal perspective we are able to estimate
the effect of interventions or counterfactuals on an individual sample. This
ability, enables a finer grain analysis of our interventions and their
effects. For the purposes of our analysis we have assumed complete knowledge
of the behavior of our models under different data regimes. This however, is
not a valid assumption in real life model development. In such cases, the
practitioner could employ a method from literature to estimate or bound the
above probability of causation.

If we do not condition on the knowledge that the sample was initially
misclassified and we are solely interested in the interventional probability
if it will be correctly classified, we aim to learn the conditional average
treatment effects: $\mathbb{E}(Y_{X=1}\mid Z)-\mathbb{E}(Y_{X=0}\mid Z)$.
Examples of methods that can estimate these include PerfectMatch [26],
DragonNet [29], PropensityDropout [3], Treatment-agnostic representation
networks (TARNET) [27], Balancing Neural Networks [17]. Other machine learning
approaches to estimating interventional queries made use of GANs, such as
GANITE [34] and CausalGAN [18], Gaussian Processes [32, 2], Variational
Autoencoders [20], and representation learning [35, 4, 33].

If, on the other hand, we wish to answer the same query in Equation 1 we need
to utilize methods able to handle counterfactual queries. Recent work [23]
proposes normalising flows and variational inference to compute counterfacual
queries using abduction-action-prediction. [22] used the Gumbel-Max trick to
estimate counterfactuals, again using abduction-action-prediction. While this
methodology satisfies generalisations of identifiability constraints.
Additional work by [12] devised a non-parametric method to compute the
Probability of Necessity using an influence-function-based estimator. A
limitation of this approach is that a separate estimator must be derived and
trained for each counterfactual query. Finally, [30] estimate counterfactual
probabilities via means of a deep twin network while imposing identifiability
constraints in the case of binary treatments and outcomes.

A major difference between the aforementioned causally enabled methods to
prior active learning is the ability of causal methods to answer individual
treatment effect queries. Utilizing methods like [30, 28, 21] one is able to
estimate the effect of a treatment at an individual level. In our example case
that translates the assessment of the probability of a sample to be correctly
identified after the treatment in the model’s training regime is applied. This
increase in analytical resolution admits reasoning about the allocation of
resources at a level that was not possible with aggregate level statistics.

(a) | Healthy | Mild | Moderate
---|---|---|---
Healthy | 0 | 12.03 | 18.75
Mild | 12.95 | 0 | 14.74
Moderate | 16.52 | 11.31 | 0

(b) | 100 | 500 | 1000 | 2000
---|---|---|---|---
100 | 0 | 18.28 | 18.16 | 19.77
500 | 18.23 | 0 | 18.00 | 19.70
1000 | 18.33 | 18.04 | 0 | 18.79
2000 | 18.88 | 18.71 | 17.59 | 0

Table 3: Dataset Interventions on the Retina dataset (A) the effect of
different upsampling classes (B) the effect informed interventions and dataset
sizes

##  6 Conclusion & Future Work

With this paper we hope to stimulate a new topic of discussion in the ML and
Medical imaging community around causal analysis and how it can help us
optimize resource allocations. Being able to quantify the per sample effect of
an intervention is necessary to better understand a given task. Besides
economical constraints, the proven environmental impact of our field [13]
means that we cannot opt for increasingly larger models when the expected
returns are minimal. Causally analyzing the task at hand can provide estimates
of performance vs. computational and economical resources without the need to
run the experiments.

This work can be considered as an introductory preparatory work. While other
methods provide argumentation in favor of using causality to enable robust and
explainable ML algorithms we showcase the use of causality and causal analysis
in a business intelligence task. We believe that future work can include
methodological contributions as causally enabled methods to provide estimates
of the amount and distributions of data in a dataset. Moreover, we aim at
working together with regulators to set out the appropriate thresholds of
confidence such that our analysis can inform regulatory procedures, ensuring
that a given model is not going to be degraded upon retraining or fine-tuning
with data that abide by the identified causal relationships.

##  7 Acknowledgments

The authors would like to acknowledge and thank the MAVEHA (EP/S013687/1)
project, Ultromics Ltd. and the UKRI Centre for Doctoral Training in
Artificial Intelligence for Healthcare (EP/S023283/1). The authors also
received GPU donations from NVIDIA.

## References

  * [1] (2022), $https://www.kaggle.com/c/diabetic-retinopathy-detection/overview$
  * [2] Alaa, A.M., van der Schaar, M.: Bayesian inference of individualized treatment effects using multi-task gaussian processes. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. pp. 3427–3435 (2017)
  * [3] Alaa, A.M., Weisz, M., Van Der Schaar, M.: Deep counterfactual networks with propensity-dropout. arXiv preprint arXiv:1706.05966 (2017)
  * [4] Assaad, S., Zeng, S., Tao, C., Datta, S., Mehta, N., Henao, R., Li, F., Duke, L.C.: Counterfactual representation learning with balancing weights. In: International Conference on Artificial Intelligence and Statistics. pp. 1972–1980. PMLR (2021)
  * [5] Balke, A., Pearl, J.: Probabilistic evaluation of counterfactual queries. In: AAAI (1994)
  * [6] Bareinboim, E., Correa, J.D., Ibeling, D., Icard, T.: On pearl’s hierarchy and the foundations of causal inference. Tech. rep., Columbia University, Stanford University (2020)
  * [7] Beleites, C., Neugebauer, U., Bocklitz, T., Krafft, C., Popp, J.: Sample size planning for classification models. Analytica chimica acta 760, 25–33 (2013)
  * [8] Budd, S., Robinson, E.C., Kainz, B.: A survey on active learning and human-in-the-loop deep learning for medical image analysis. Medical Image Analysis 71, 102062 (2021)
  * [9] Castro, D.C., Tan, J., Kainz, B., Konukoglu, E., Glocker, B.: Morpho-MNIST: Quantitative assessment and diagnostics for representation learning. Journal of Machine Learning Research 20(178) (2019)
  * [10] Castro, D.C., Walker, I., Glocker, B.: Causality matters in medical imaging. Nature Communications 11(1), 1–10 (2020)
  * [11] Cho, J., Lee, K., Shin, E., Choy, G., Do, S.: How much data is needed to train a medical image deep learning system to achieve necessary high accuracy? arXiv preprint arXiv:1511.06348 (2015)
  * [12] Cuellar, M., Kennedy, E.H.: A non-parametric projection-based estimator for the probability of causation, with application to water sanitation in kenya. Journal of the Royal Statistical Society: Series A (Statistics in Society) 183(4), 1793–1818 (2020)
  * [13] Dhar, P.: The carbon impact of artificial intelligence. Nature Machine Intelligence 2 (2020). https://doi.org/10.1038/s42256-020-0219-9
  * [14] Farquhar, S., Gal, Y., Rainforth, T.: On statistical bias in active learning: How and when to fix it. In: International Conference on Learning Representations (2020)
  * [15] Figueroa, R.L., Zeng-Treitler, Q., Kandula, S., Ngo, L.H.: Predicting sample size required for classification performance. BMC medical informatics and decision making 12(1), 1–10 (2012)
  * [16] Imbens, G.W., Rubin, D.B.: Causal inference in statistics, social, and biomedical sciences. Cambridge University Press (2015)
  * [17] Johansson, F., Shalit, U., Sontag, D.: Learning representations for counterfactual inference. In: International Conference on Machine Learning. pp. 3020–3029 (2016)
  * [18] Kocaoglu, M., Snyder, C., Dimakis, A.G., Vishwanath, S.: Causalgan: Learning causal implicit generative models with adversarial training. In: International Conference on Learning Representations (2018)
  * [19] LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)
  * [20] Louizos, C., Shalit, U., Mooij, J., Sontag, D., Zemel, R., Welling, M.: Causal effect inference with deep latent-variable models. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. pp. 6449–6459 (2017)
  * [21] Muellerr, S., Li, A., Pearl, J.: Causes of effects: Learning individual responses from population data. arXiv preprint arXiv:2104.13730 (2021)
  * [22] Oberst, M., Sontag, D.: Counterfactual off-policy evaluation with gumbel-max structural causal models. In: International Conference on Machine Learning. pp. 4881–4890. PMLR (2019)
  * [23] Pawlowski, N., Castro, D.C., Glocker, B.: Deep structural causal models for tractable counterfactual inference. arXiv preprint arXiv:2006.06485 (2020)
  * [24] Pearl, J.: Causality (2nd edition). Cambridge University Press (2009)
  * [25] Richens, J.G., Lee, C.M., Johri, S.: Improving the accuracy of medical diagnosis with causal machine learning. Nature communications 11(1), 1–9 (2020)
  * [26] Schwab, P., Linhardt, L., Karlen, W.: Perfect match: A simple method for learning representations for counterfactual inference with neural networks. arXiv preprint arXiv:1810.00656 (2018)
  * [27] Shalit, U., Johansson, F.D., Sontag, D.: Estimating individual treatment effect: generalization bounds and algorithms. arXiv preprint arXiv:1606.03976 (2016)
  * [28] Shalit, U., Johansson, F.D., Sontag, D.: Estimating individual treatment effect: generalization bounds and algorithms. International Conference on Machine Learning (2017)
  * [29] Shi, C., Blei, D.M., Veitch, V.: Adapting neural networks for the estimation of treatment effects. arXiv preprint arXiv:1906.02120 (2019)
  * [30] Vlontzos, A., Kainz, B., Gilligan-Lee, C.M.: Estimating the probabilities of causation via deep monotonic twin networks. arXiv preprint arXiv:2109.01904 (2021)
  * [31] Voigt, P., Von dem Bussche, A.: The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing 10(3152676), 10–5555 (2017)
  * [32] Witty, S., Takatsu, K., Jensen, D., Mansinghka, V.: Causal inference using gaussian processes with structured latent confounders. In: International Conference on Machine Learning. pp. 10313–10323. PMLR (2020)
  * [33] Yao, L., Li, S., Li, Y., Huai, M., Gao, J., Zhang, A.: Representation learning for treatment effect estimation from observational data. Advances in Neural Information Processing Systems 31 (2018)
  * [34] Yoon, J., Jordon, J., Van Der Schaar, M.: Ganite: Estimation of individualized treatment effects using generative adversarial nets. In: International Conference on Learning Representations (2018)
  * [35] Zhang, Y., Bellot, A., Schaar, M.: Learning overlapping representations for the estimation of individualized treatment effects. In: International Conference on Artificial Intelligence and Statistics. pp. 1005–1014. PMLR (2020)