# Bayesian sample size determination using robust commensurate priors with
interpretable discrepancy weights

Lou E. Whitehead1*,
James M. S. Wason1,
Oliver Sailer2,
Haiyan Zheng 3
1\. Biostatistics Research Group, Population Health Sciences Institute,
Newcastle University, Newcastle upon Tyne, UK
2\. Boehringer Ingelheim Pharma GmbH & Co. KG, Biberach, Germany
3\. Department of Mathematical Sciences, University of Bath, Bath, UK
*Corresponding Author Details:
Lou E. Whitehead; Address: Biostatistics Research Group,
Population Health Sciences Institute,
Ridley 1 Building, Richardson Road, Newcastle upon Tyne NE1 7RU, UK
Telephone number: +44(0)0191-208-6000; E-mail address:
<EMAIL_ADDRESS>
###### Abstract

Randomized controlled clinical trials provide the gold standard for evidence
generation in relation to the effectiveness of a new treatment in medical
research. Relevant information from previous studies may be desirable to
incorporate in the design of a new trial, with the Bayesian paradigm providing
a coherent framework to formally incorporate prior knowledge. Many established
methods involve the use of a discounting factor, sometimes related to a
measure of ‘similarity’ between historical sources and the new trial. However,
it is often the case that the sample size is highly nonlinear in those
discounting factors. This hinders communication with subject-matter experts to
elicit sensible values for borrowing strength at the trial design stage.

Focusing on a sample size formula that can incorporate historical data from
multiple sources, we propose a linearization technique such that the sample
size changes evenly over values of the discounting factors (hereafter referred
to as ‘weights’). Our approach leads to interpretable weights that directly
represent the dissimilarity between historical and new trial data on the
probability scale, and could therefore facilitate easier elicitation of expert
opinion on their values.

Inclusion of historical data in the design of clinical trials is not common
practice. Part of the reason might be difficulty in interpretability of
discrepancy parameters. We hope our work will help to bridge this gap and
encourage uptake of these innovative methods.

Keywords: Bayesian sample size determination; Commensurate priors; Historical
borrowing; Prior aggregation; Uniform shrinkage.

##  1 Introduction

In clinical drug development randomized controlled trials (RCTs) are regarded
as the gold standard for evaluating the effectiveness of new treatments or
interventions. Randomization of trial participants to the new treatment or a
control group aims to reduce bias and provides a rigorous tool to examine
whether a causal relationship exists between an intervention and outcome
(Hariton and Locascio,, 2018). Sample size calculations are an essential part
of clinical trial design, with a sample needing to be at least large enough to
meet the study objectives but also small enough to minimize (for example)
ethical or cost concerns (Julious,, 2023). In the frequentist paradigm, the
number of participants recruited onto a study is often chosen to control the
type I error rate (the rate of incorrectly declaring a treatment effective)
and/or power (the rate of correctly declaring a treatment effective) to pre-
specified levels, based on assumptions about the sampling distribution of the
data and the size of the treatment effect considered clinically meaningful.

Designing a trial with a large enough sample size to achieve the frequentist
power can sometimes be infeasible, especially when there are limited numbers
of participants available. This might be the case in early phase settings,
rare disease trials, or trials in pediatric populations, etc. Pre-trial
information, from historical studies conducted under similar circumstances, or
elicited directly from expert opinion, could be useful to overcome this
challenge, with the Bayesian paradigm offering a powerful tool to formalize
this approach. In the Bayesian framework, a prior distribution is formed for a
parameter of interest, which is then updated by the observed data to form a
posterior distribution from which inferences can be made. Instead of designing
a trial around frequentist type I error rates and power, Bayesian designs rely
on alternative metrics for success; for instance, specification of posterior
decision thresholds (the level of confidence we desire to have that a
treatment is effective or futile), or the width or coverage probabilities of
Bayesian credible intervals. The application of Bayesian methodology for trial
design to the specific areas noted above has been considered; see, for
example, Zheng and Hampson, (2020) for first-in-man trials, Hampson et al.,
(2014) for trials in very rare diseases, and Wadsworth et al., (2018) for
pediatric studies.

Bayesian methods for clinical trial design incorporating historical data can
be categorized according to the method of constructing a prior distribution
for a parameter of interest. These range from ‘irrelevance’, where a prior is
formed without reference to previous studies, to ‘equal’, which is equivalent
to pooling the historical data with the new study. A common compromise between
these two extremes is the ‘exchangeablility’ approach, where a prior is formed
by assuming that the parameter of interest in the new trial has been generated
from the same underlying distribution as the parameter(s) in the historical
trial(s) (Spiegelhalter et al.,, 2004). The meta-analytic predictive (MAP)
prior (Neuenschwander et al.,, 2010) is based on this assumption, with the
authors noting the importance of careful selection of relevant historical data
to render the exchangeability assumption plausible. A robust extension
(Schmidli et al.,, 2014) aims to effectively discount historical data in the
case of prior/data conflict by using a weighted mixture distribution
consisting of the MAP prior and a weakly informative component. Alternatives
are the ‘power prior’ (Ibrahim and Chen,, 2000), in which historical evidence
is discounted by taking its likelihood to a power, and the ‘commensurate
prior’ (Hobbs et al.,, 2011, 2012), where historical information is
downweighted by a commensurability parameter to form a predictive prior for
the new study. The commensurability parameter directly parameterizes the
discrepancy between the historical source and new data, with a prior formed by
assuming parameters from historical data are ‘potentially biased’
(Spiegelhalter et al.,, 2004) with respect to the current trial.

The importance of carefully selecting historical trials to be included for
planning a new trial is well understood. If the assumption of similarity is
not satisfied, this can result in increased mean square error of point
estimates due to bias and either reduced power or increased type I error rate
depending on the direction of the bias (Viele et al.,, 2014). A seminal paper
by Pocock, (1976) provided a set of criteria for assessing the comparability
between historical and current trials. Expert elicitation can play an
important role in assessing comparability and helping to choose model
parameters but the elicitation process is not trivial (Dias et al.,, 2017).
Johnson et al., (2010) review different methods to elicit beliefs for Bayesian
priors. Hampson et al., (2014) develop a methodology for systematic
elicitation from clinicians regarding beliefs concerning treatment efficacy
which is then used to establish priors for unknown model parameters.

This paper focuses on the design of a two-arm RCT incorporating historical
data. We follow the series of research in sample size determination based on
‘commensurate priors’ by Zheng et al., 2023a . The use of discrepancy weights
on the probability scale quantifying the relevance of information from
multiple historical sources was recommended. This methodology was later
extended to basket trials in Zheng et al., 2023b . Basket trials are an
innovative trial design investigating a single treatment in multiple subtrials
simultaneously (Woodcock and LaVange,, 2017). In the setting of borrowing from
historical data, the use of discrepancy weights offers a relatively simple way
to incorporate expert elicited opinions on the relevance of each data source
to the new trial.

This paper is organized as follows. In Section 2, we outline the problem,
which is namely that while the discrepancy weights are intended to behave
uniformly on the probability scale they in fact exhibit undesirable highly
nonlinear behaviour with respect to the amount of information they represent.
This is caused by the method used to aggregate information from multiple
sources into a single prior. This problem hinders interpretability and makes
elicitation of weights difficult. In Section 3, an alternative method of
aggregation is proposed, from which a Bayesian sample size formula is derived.
A technique for linearization is subsequently provided such that the weights
provide uniform shrinkage with respect to the sample size. The aim is to make
interpretability simpler and thereby facilitate easier elicitation of such
values. Section 4 provides a hypothetical motivating example illustrating the
method using a real-life clinical trial. Section 5 presents a brief simulation
study confirming pre-specified statistical properties are preserved across a
range of trial scenarios with sample sizes determined according to our
formula. We finish with a discussion highlighting areas for future research in
Section 6.

##  2 Problem Formulation

Consider planning a two-arm randomized controlled superiority trial (referred
to as ‘new trial’ in the following) to evaluate an investigational treatment
or intervention. Let $Y_{ij}$ be the measured post-randomization outcomes in
the new trial for patient $i=1,...,n_{j}$ in treatment group $j=T,C$.
Explicitly, $j=T$ refers to the experimental treatment group and $j=C$ refers
to the control group. We assume outcomes are normally distributed with common
variance in the outcome measures such that
$Y_{ij}\sim\mathcal{N}(\mu_{j},\sigma_{0}^{2})$. By the central limit theorem,
the groupwise sample means follow an asymptotic normal distribution,
$\bar{Y}_{j}\sim\mathcal{N}(\mu_{j},\frac{\sigma_{0}^{2}}{n_{j}})$.
Considering the distribution of the difference in group means leads to

$\bar{Y}_{T}-\bar{Y}_{C}=\bar{Y}_{\Delta}\sim\mathcal{N}\left(\mu_{\Delta},\frac{\sigma_{0}^{2}}{nR(1-R)}\right),$

where $\mu_{\Delta}=\mu_{T}-\mu_{C}$ is our primary inferential target.
$n=\sum_{j=T,C}n_{j}$ are the total number of trial participants randomized
(to treatment or control) at the initiation of the trial and $R=n_{T}/n$ is
the proportion randomly assigned to the experimental treatment arm.

In the Bayesian framework with no borrowing from historical data (for assumed
known $\sigma_{0}^{2}$), a prior for $\mu_{\Delta}$ is specified,

$\mu_{\Delta}\sim\mathcal{N}\left(\mu_{0},s_{0}^{2}\right),$

which is then updated by the trial data to give a posterior distribution,

$\mu_{\Delta}|\bm{y_{new}}\sim\mathcal{N}\left(d_{\theta_{0}},\sigma^{2}_{\theta_{0}}\right).$

The posterior mean is given by

$d_{\theta_{0}}=\frac{s_{0}^{-2}\cdot\mu_{0}+(\bar{y}_{T}-\bar{y}_{C})\cdot
nR(1-R)/\sigma_{0}^{2}}{s_{0}^{-2}+nR(1-R)/\sigma_{0}^{2}}$ (1)

and the posterior variance is

$\sigma^{2}_{\theta_{0}}=\left(\frac{1}{s_{0}^{2}}+\frac{nR(1-R)}{\sigma_{0}^{2}}\right)^{-1}.$
(2)

###  2.1 Formulating Priors from Multiple Historical Sources

Suppose instead that there are multiple sources of historical data that are
relevant to incorporate in the planning of the new trial and that from each
source, $q=1,...,Q$, a distribution for mean treatment effect has been
obtained, $\lambda_{q}\sim\mathcal{N}(\theta_{q},\tau^{2}_{q})$. We define
$\mu_{\Delta(q)}$ as the prediction for $\mu_{\Delta}$ in the new trial based
on the information from trial $q$ alone. A set of $Q$ predictive prior
distributions for $\mu_{\Delta}$ are formed centered on each $\theta_{q}$,

$\displaystyle\mu_{\Delta(1)}\sim\mathcal{N}(\theta_{1},\xi_{1}^{2}),\ \ ...,\
\ \mu_{\Delta(Q)}\sim\mathcal{N}(\theta_{Q},\xi_{Q}^{2}).$ (3)

Predictive variances are $\xi_{q}^{2}=\tau^{2}_{q}+\nu_{q}^{-1}$, where
$\nu_{q}$ parameterizes the ‘commensurability’ (Zheng et al., 2023b, ) between
$\theta_{q}$ and $\mu_{\Delta}$ in terms of precision (further details are
given in the following section).

###  2.2 Estimating $\xi^{2}_{q}$

To quantify the relevance of each historical data source in respect of the new
experiment, Zheng et al. introduce discrepancy parameters,
$\bm{w_{q}}=\\{w_{1},....,w_{Q}\\}$. These are prior weights $\in[0,1]$
intended to represent preliminary skepticism on the probability scale about
how commensurate $\theta_{q}$ and $\mu_{\Delta}$ are (Zheng et al., 2023a, ;
Zheng et al., 2023b, ). These weights are incorporated into a Gamma mixture
prior for the precision parameter, $\nu_{q}$:

$\displaystyle\nu_{q}\sim
w_{q}\>Ga(a_{01},b_{01})+(1-w_{q})\>Ga(a_{02},b_{02}),$ (4)

with $w_{q}\,\epsilon\,[0,1],\>\>{a_{01},a_{02}>1}$. This mixture prior is
favoured for robust inferences as it offers flexible downweighting or
borrowing from source $q$ depending on the value of $w_{q}$. Briefly, the
values of $a_{01},b_{01}$ are chosen such that the first Gamma mixture
component has its mass on small values, therefore when $w_{q}\to 1$, data from
source $q$ is increasingly discounted. Similarly, the values of
$a_{02},b_{02}$ are chosen such that the second Gamma mixture component has
its mass on large values, therefore setting $w_{q}\to 0$ results in a greater
degree of incorporation of information from source $q$. Setting $w_{q}=1$ or
$w_{q}=0$ results in the extremes of no borrowing or fully pooling,
respectively. It is anticipated in a real application that values of
$w_{q}\,\epsilon\,[0,1]$ are chosen either: (a) in collaboration with a
subject-matter expert(s) to reflect the degree of (dis)similarity between each
historical trial and the new experiment, or, (b) using a distributional
discrepancy measure such as the Hellinger distance (Dey and Birmiwal,, 1994)
to calculate $w_{q}$ empirically.

As detailed in Zheng et al., 2023a , the Gamma mixture prior in (4) can be
approximated by matching the first two moments of a unimodal t mixture
distribution. This leads to an approximation of the between-trial variance
(i.e., between source $q$ and the new experiment),

$\displaystyle\nu^{-1}_{q}\simeq\frac{w_{q}b_{01}}{a_{01}-1}+\frac{(1-w_{q})b_{02}}{a_{02}-1}.$

We note that if we were being fully Bayesian we would keep the prior for
$\nu_{q}$ in its distributional form, however in this paper we are looking to
propose an asymptotically approximate sample size formula and so we make a
simplifying assumption. The predictive variance between each source $q$ and
the new trial is therefore estimated as

$\displaystyle\xi_{q}^{2}=\tau_{q}^{2}+\frac{w_{q}b_{01}}{a_{01}-1}+\frac{(1-w_{q})b_{02}}{a_{02}-1}.$

###  2.3 Aggregating Multiple Predictive Distributions to Form a Collective
Prior

In Zheng et al., 2023a , an informative commensurate predictive prior (CPP) is
formed by aggregating the $Q$ predictive distributions in (3) into a
collective prior such that
$\mu_{\Delta}\sim\mathcal{N}(\theta_{CPP},\sigma^{2}_{CPP})$ using the
convolution operator for the sum of normal random variables (Grinstead and
Snell,, 1997), where

$\displaystyle\theta_{CPP}=\sum_{q=1}^{Q}p_{q}\theta_{q},\ \
\sigma^{2}_{CPP}=\sum_{q=1}^{Q}p^{2}_{q}\xi^{2}_{q}.$

$p_{q}$ are synthesis weights, set to a decreasing function of $w_{q}$, such
that are all between $0$ and $1$ and sum to $1$. In Zheng et al., 2023a ,

$\displaystyle
p_{q}=\frac{\exp(-w_{q}^{2}/c_{0})}{\sum_{q=1}^{Q}\exp(-w_{q}^{2}/c_{0})},$
(5)

where $c_{0}$ is a concentration parameter; further details on the $p_{q}$
function and how to choose $c_{0}$ are provided in Zheng and Wason, (2022) and
Zheng et al., 2023a . The CPP is updated by the trial data to give the
posterior,

$\mu_{\Delta}|\bm{y_{q}},\bm{y_{new}}\sim\mathcal{N}\left(d_{\theta_{1}},\sigma^{2}_{\theta_{1}}\right),$

In the same way as Equations (1) and (2), the posterior mean and variance are
given by

$d_{\theta_{1}}=\frac{\sigma^{-2}_{CPP}\cdot\theta_{CPP}+(\bar{y}_{T}-\bar{y}_{C})\cdot
nR(1-R)/\sigma_{0}^{2}}{\sigma^{-2}_{CPP}+nR(1-R)/\sigma_{0}^{2}}$

and

$\sigma^{2}_{\theta_{1}}=\left(\frac{1}{\sigma^{2}_{CPP}}+\frac{nR(1-R)}{\sigma_{0}^{2}}\right)^{-1}.$

###  2.4 Varying $w_{q}$ to Alter the Amount of Information from Source $q$

For fixed $\bm{\tau^{2}_{q}},a_{01},b_{01},a_{02},b_{02}$, $c_{0}$, the CPP
precision $\sigma_{CPP}^{-2}$ is a function of $\bm{w_{q}}$ and is a measure
of the amount of prior information on the treatment effect in the new trial
(which varies depending on the values of $\bm{w_{q}}$),

$\displaystyle\sigma^{-2}_{CPP}$
$\displaystyle=\left[\sum_{q=1}^{Q}p^{2}_{q}\xi^{2}_{q}\right]^{-1}$
$\displaystyle=\left[\sum_{q=1}^{Q}p^{2}_{q}\left(\tau_{q}^{2}+\frac{w_{q}b_{01}}{a_{01}-1}+\frac{(1-w_{q})b_{02}}{a_{02}-1}\right)\right]^{-1}.$
(6)

In Figure 1, we visualize how $\sigma_{CPP}^{-2}$ varies according to
$\bm{w_{q}}$ in an example when $Q=2$. For illustrative purposes, values of
all other parameters are held fixed
($\sigma_{0}^{2}=0.8,R=0.5,\delta=0.5,\tau_{1}^{2}=\tau_{2}^{2}=0.1,a_{01}=1.1,b_{01}=1.1,a_{02}=1\times
10^{6},b_{02}=1,c_{0}=0.05$). It can be seen that for $w_{1}=w_{2}=0$,
corresponding to full incorporation of information from both historical
sources, the CPP precision is maximized (as desired). Similarly, for
$w_{1}=w_{2}=1$, corresponding to full discounting of information from both
sources, the CPP precision is minimized (as desired).

We nonetheless also see the (undesirable) highly nonlinear nature of
$\sigma_{CPP}^{-2}$ with respect to varying $\bm{w_{q}}$ in two respects.
Firstly, it is clear that the majority of the change in prior precision occurs
for small values of $w_{q}$; beyond around $w_{q}>0.2$, there is almost no
discernible change in $\sigma_{CPP}^{-2}$. This results in a large loss of
information and complicates the interpretability of $w_{q}$ as a degree of
(dis)similarity on the probability scale. This issue occurs in varying degrees
for any value of $Q$ and regardless of what values the other parameters are
fixed at. Secondly, when $Q>1$, local minima/maxima can be seen along the
boundaries when either $w_{1}=0$ or $w_{2}=0$. This non-monotonic behaviour
occurs whenever $Q>1$ due the method of prior aggregation as well as the
higher order terms in $w_{q}$ introduced by the synthesis weighting function
(Equation (5)).

Figure 1: CPP precision, $\sigma_{CPP}^{-2}$ (Equation 2.4), with respect to
varying discrepancy weights, $w_{1}$ and $w_{2}$, for borrowing from two
historical sources, $q=1,2$.

These issues hinder communication with subject matter experts to elicit
sensible values for $\bm{w_{q}}$ at the trial design stage. Therefore, in this
paper we seek to specify a functional transformation of each $w_{q}\to
w^{\prime}_{q}$, where $w^{\prime}_{q}=f(w_{q})$, such that specifying any
$w_{q}\,\epsilon\,[0,1]$ corresponds directly to incorporating
$((1-w_{q})\times 100)\%$ of the information from source $q$. In order to
achieve this, an alternative method of prior aggregation (and therefore a new
way of formulating the CPP precision) is first necessary. This is due to the
non-monotonicity of Equation (2.4), since to define a transformation of a
variable with respect to a function, that function must first be a 1-to-1
(i.e., bijective) function. The alternative method of prior aggregation is
reflected in our proposed sample size function, detailed in the next section
along with the subsequent transformation procedure.

##  3 Methods

###  3.1 Alternative Method of Prior Aggregation

Following the set of predictive priors in (3), we propose an alternative
method of prior aggregation suggested in Winkler, (1981). This results in a
new collective prior, $\mu_{\Delta}\sim
N(\theta_{CPP^{*}},\sigma^{2}_{CPP^{*}})$, where

$\theta_{CPP^{*}}=\sum_{q=1}^{Q}p^{(*)}_{q}\theta_{q},\ \
\sigma^{2}_{CPP^{*}}=\left(\sum_{q=1}^{Q}\xi^{-2}_{q}\right)^{-1},$
$p^{(*)}_{q}=\frac{\xi^{-2}_{q}}{\left(\sum_{q=1}^{Q}\xi^{-2}_{q}\right)}.$

It is this method of prior aggregation which forms the basis of our Bayesian
sample size calculation derived in the following section. As in Section 2.3,
the CPP mean, $\theta_{CPP^{*}}$, is a weighted linear sum of the means from
(3). As before, synthesis weights $p^{(*)}_{q}$ sum to $1$ and are all between
$0$ and $1$. However, synthesis weights $p^{(*)}_{q}$ are now based on
predictive precisions, $\xi^{-2}_{q}$ (rather than $\bm{w_{q}}$ as in Equation
(5)). This preserves the desirable property that a smaller $w_{q}$ corresponds
to a larger $p^{(*)}_{q}$.

The CPP variance, $\sigma^{2}_{CPP^{*}}$, is the reciprocal of the sum of the
predictive precisions, $\xi^{-2}_{q}$. Again, this preserves the desirable
property that a smaller $w_{q}$ results in source $q$ receiving a larger
weight in $\sigma^{-2}_{CPP^{*}}$. The formulation of the CPP mean and
variance in this manner is exactly in line with the theory of Bayesian
updating of normal distributions with conjugate priors, with an initial
noninformative prior for $\mu_{\Delta}$ (as discussed in Winkler, (1981)). The
motivation to use this method of aggregation in our particular application is
that terms in the CPP precision relating to each $w_{q}$ remain linearly
independent of each other, i.e.,

$\displaystyle\sigma^{-2}_{CPP^{*}}$
$\displaystyle=\sum_{q=1}^{Q}\xi^{-2}_{q}$
$\displaystyle=\sum_{q=1}^{Q}\left[\left(\tau_{q}^{2}+\frac{w_{q}b_{01}}{a_{01}-1}+\frac{(1-w_{q})b_{02}}{a_{02}-1}\right)^{-1}\right].$
(7)

The nonlinearity of $\sigma^{-2}_{CPP^{*}}$ in Equation (3.1) with respect to
varying $\bm{w_{q}}$ now has a simpler pattern when compared with Equation
(2.4). Specifically, its first derivative with respect to each $w_{q}$ is
always negative and therefore it is monotonically decreasing. This is
visualized in Figure 2 using the same parameters as Figure 1 for borrowing
from two historical datasets.

Figure 2: CPP precision, $\sigma_{CPP^{*}}^{-2}$ (Equation (3.1)), with
respect to varying discrepancy weights, $w_{1}$ and $w_{2}$, for borrowing
from two historical sources, $q=1,2$.

As in Section (2), the CPP is updated by the trial data to give the posterior,

$\mu_{\Delta}|\bm{y_{q}},\bm{y_{new}}\sim\mathcal{N}\left(d_{\theta^{*}},\sigma^{2}_{\theta^{*}}\right),$

where

$d_{\theta^{*}}=\frac{\sigma^{-2}_{CPP^{*}}\cdot\theta_{CPP^{*}}+(\bar{y}_{T}-\bar{y}_{C})\cdot
nR(1-R)/\sigma_{0}^{2}}{\sigma^{-2}_{CPP^{*}}+nR(1-R)/\sigma_{0}^{2}}$ (8)

and

$\sigma^{2}_{\theta^{*}}=\left(\frac{1}{\sigma^{2}_{CPP^{*}}}+\frac{nR(1-R)}{\sigma_{0}^{2}}\right)^{-1}.$
(9)

###  3.2 Bayesian Decision Making Framework

We now introduce a Bayesian decision making framework proposed in Whitehead et
al., (2008). For pre-specified posterior decision thresholds $\eta$ and
$\zeta$, we seek a sample size to guarantee we have sufficient evidence to
conclude either efficacy or futility respectively.

In terms of the efficacy criteria, for a generic posterior distribution
$\mu_{\Delta}\sim\mathcal{N}(d_{\theta},\sigma^{2}_{\theta})$, we consider the
probability that the treatment effect is greater than zero, i.e.,

$\mathbb{P}(\mu_{\Delta}>0)=1-\Phi\left(-\frac{d_{\theta}}{\sigma_{\theta}}\right)=\Phi\left(\frac{d_{\theta}}{\sigma_{\theta}}\right),$

where $\Phi(\cdot)$ denotes the standard normal cumulative distribution
function. Therefore, we will conclude convincing evidence of treatment benefit
when $\frac{d_{\theta}}{\sigma_{\theta}}\geq z_{\eta}$, where $z_{\eta}$
satisfies $\Phi(z_{\eta})=\eta$. For the futility consideration, we examine
the posterior probability that the treatment effect is less than some
minimally clinically important amount, $\delta$, i.e.,

$\mathbb{P}(\mu_{\Delta}\leq\delta)=\Phi\left(\frac{\delta-
d_{\theta}}{\sigma_{\theta}}\right).$

Therefore, convincing evidence of treatment futility occurs when
$\frac{\delta-d_{\theta}}{\sigma_{\theta}}\geq z_{\zeta}$, where $z_{\zeta}$
satisfies $\Phi(z_{\zeta})=\zeta$.

###  3.3 Bayesian Sample Size Formula

Following the same approach detailed in Zheng et al., 2023b , to reach a
decisive conclusion regarding treatment effectiveness, we require a large
enough sample size such that either $d_{\theta}/\sigma_{\theta}\geq z_{\eta}$
or $(\delta-d_{\theta})/\sigma_{\theta}\geq z_{\zeta}$, i.e.,

$\frac{d_{\theta}}{\sigma_{\theta}}+\frac{(\delta-
d_{\theta})}{\sigma_{\theta}}\geq z_{\eta}+z_{\zeta}.$

Simplifying and rearranging, this is equivalent to requiring that

$\frac{1}{\sigma_{\theta}^{2}}\geq\left(\frac{z_{\eta}+z_{\zeta}}{\delta}\right)^{2}.$
(10)

We see that the left hand side of (10) is equal to the posterior precision.
Replacing $\sigma^{2}_{\theta}$ with the variance in (2), we therefore obtain
a Bayesian sample size formula in the case of no borrowing,

$n\geq\frac{\sigma_{0}^{2}}{R(1-R)}\left(\left(\frac{z_{\eta}+z_{\zeta}}{\delta}\right)^{2}-\frac{1}{s_{0}^{2}}\right).$
(11)

Note that if we wished to consider a purely frequentist formulation of the
problem, then the necessary sample size is simply,

$n\geq\frac{\sigma_{0}^{2}}{R(1-R)}\left(\frac{z_{1-\alpha}+z_{1-\beta}}{\delta}\right)^{2},$
(12)

where $\alpha$ and $\beta$ are the usual parameters set to control type I and
type II error rates respectively.

Replacing $\frac{1}{s_{0}^{2}}$ in (11) with $\sigma_{CPP^{*}}^{-2}$ from
(3.1), we obtain our sample size calculation informed by $Q$ sources of
historical data,

$\displaystyle
n\geq\frac{\sigma_{0}^{2}}{R(1-R)}\left(\left(\frac{z_{\eta}+z_{\zeta}}{\delta}\right)^{2}-\sum_{q=1}^{Q}\left[\left(\tau_{q}^{2}+\frac{w_{q}b_{01}}{a_{01}-1}+\frac{(1-w_{q})b_{02}}{a_{02}-1}\right)^{-1}\right]\right),$
(13)

with $w_{q}\,\epsilon\,[0,1],\>\>{a_{01},a_{02}>1}$. We note explicitly the
assumptions embedded into this sample size formula, which are common to many
normal models. The validity of the sample size calculation depends on these
assumptions being satisfied:

  1. 1.

Common (and known) variance in outcomes from the new trial.

  2. 2.

Independence of observations.

  3. 3.

Homoscedasticity and normality of residuals.

For non-normal data, a suitably adapted formula based on the approach of
constructing a normal test statistic in the Generalized Linear Model (GLM)
framework via a transformation could also be appropriate while being aware of
the limitations. For example, a sample size formula for Bernoulli data and
time-to-event data could be derived by the same method detailed in Sections
C-D of the ‘Supplementary Data’ in Zheng et al., 2023b .

###  3.4 Interpretable Discrepancy Weights

We now detail the linearization steps which result in $\bm{w_{q}}$ that are
directly interpretable on the probability scale. The idea is very similar to
the idea of functional uniform priors proposed in Bornkamp, (2012, 2014) for
nonlinear regression, in which a method for formulating a prior for a
parameter of interest is proposed such that that it is uniform in the space of
functional shapes of the underlying nonlinear function. We start by isolating
each nonlinear part of the sample size function in Equation (13) with respect
to $w_{q}$ (for fixed $\tau_{q}^{2},a_{01},b_{01},a_{02},b_{02}$). These are
the predictive precision terms making up $\sigma^{-2}_{CPP^{*}}$ in Equation
(3.1), i.e.,

$\xi_{q}^{-2}(w_{q})=\left(\tau_{q}^{2}+\frac{w_{q}b_{01}}{a_{01}-1}+\frac{(1-w_{q})b_{02}}{a_{02}-1}\right)^{-1}.$
(14)

Step 1: Perform linear interpolation on (14):

$h(w_{q})=(1-w_{q})\xi_{q}^{-2}(0)+w_{q}\xi_{q}^{-2}(1)=\xi_{q}^{-2}(0)+w_{q}(\xi_{q}^{-2}(1)-\xi_{q}^{-2}(0)).$
(15)

This essentially ‘draws a line’ between $\xi_{q}^{-2}(w_{q}=0)$ and
$\xi_{q}^{-2}(w_{q}=1)$ so that changes in $\xi_{q}^{-2}$ (and therefore the
corresponding sample size) are spread evenly across the full range of
$w_{q}\,\epsilon\,[0,1]$. This also necessarily ensures that the mapping
$w_{q}\to w^{\prime}_{q}$ preserves the property that $w_{q}=0\to
w_{q}^{\prime}=0$ and $w_{q}=1\to w_{q}^{\prime}=1$.

Step 2: Find the inverse of (14). This allows calculation of any $w_{q}$ value
corresponding to a given $\xi_{q}^{-2}$:

$g(\xi_{q}^{-2})=(\xi_{q}^{-2})^{(-1)}$ (16)

Step 3: Substitute the linearized $\xi_{q}^{-2}$ values obtained in (15) into
(16):

$f(w_{q})=g(h(w_{q}))=w^{\prime}_{q},$ (17)

$f(w_{q})$ is now the necessary transformation of $w_{q}\to w^{\prime}_{q}$.
Now, if we obtain expert elicited values of $\bm{w_{q}}$, corresponding to a
percentage degree of discrepancy between each historical source and the new
trial, we can use Equation (13) with $\bm{w^{\prime}_{q}}$ to incorporate
$((1-w_{q})\times 100)\%$ of the information from the corresponding historical
dataset in the planning of the new trial.

This transformation is possible due to the proposed method of prior
aggregation. Unlike the original $\sigma^{-2}_{CPP}$ in Equation (2.4), the
proposed $\sigma^{-2}_{CPP^{*}}$ is a monotonic function in $w_{q}$, and each
of the $\xi_{q}^{-2}$ terms (which form $\sigma^{-2}_{CPP^{*}}$) are linearly
independent of each other (i.e., separated by the addition operator). Our
transformation procedure can easily be extended to any number of sources,
$q=1,...,Q$, with each functional transformation of $w_{q}\to
f(w_{q})=w_{q}^{\prime}$ being performed independently with no additional
complexity.

The effect is visualized in Figure 3 and Figure 4, which compare the sample
size function with respect to $w_{q}$ before and after the functional
transformation of $w_{q}\to f(w_{q})=w_{q}^{\prime}$. These examples are for
the simplest cases of borrowing from one and two historical datasets.

Figure 3: Sample size, $n$, with respect to varying $w_{1}$, for borrowing
from a single source of data, both before (left) and after (right) functional
transformation of $w_{1}\to f(w_{1})=w^{\prime}_{1}$. Note that as required,
minimum and maximum sample sizes remain identical in both cases. Figure 4:
Sample size $n$ (vertical axis) borrowing from two historical sources, with
respect to varying $w_{1}$ and $w_{2}$. The left figure is before the
functional transformation of $\bm{w_{q}}\to\bm{w^{\prime}_{q}}$, i.e.,
$n(\bm{w_{q}},\bm{\tau^{2}_{q}})$, the right figure is after, i.e.,
$n(\bm{w^{\prime}_{q}},\bm{\tau^{2}_{q}})$.

Sample sizes corresponding to
$n(w_{1}=0,w_{2}=0),n(w_{1}=1,w_{2}=0),n(w_{1}=0,w_{2}=1)$ and
$n(w_{1}=1,w_{1}=1)$ remain as fixed after the transformation of $w_{q}\to
w^{\prime}_{q}$ as required. This will be the case for borrowing from any
number of sources, i.e., borrowing from $Q$ sources will have $2^{Q}$ fixed
points corresponding to each unique combination of
$w_{1}=\\{0,1\\},...,w_{Q}=\\{0,1\\}$. Between these fixed points, via the
proposed transformation, the change in the sample size is evenly distributed
across $w_{1},...,w_{Q}\,\epsilon\,[0,1]$. As before, the sample size is
minimized with full incorporation of information from all sources, i.e.,
$\\{w_{1},...,w_{Q}\\}=0$ and maximized with full discounting of information
from both sources, i.e., $\\{w_{1},...,w_{Q}\\}=1$. For values of $w_{q}$ in
between the two extremes, $((1-w_{q})\times 100)\%$ of the information from
source $q$ will be incorporated - therefore achieving interpretability on the
probability scale as desired.

##  4 Motivating Example

MET-PREVENT is a UK-based trial designed to provide evidence as to whether
metformin can improve physical performance in people with sarcopenia and
physical prefrailty/frailty (Rennie et al.,, 2022). MET-PREVENT is a parallel
group, randomized, double-blind, placebo-controlled trial. The primary outcome
measure is the between-group difference in the four metre walk speed at 4
months measured on a continuous scale (metres per second). The target
population is older adults ($\geq 65$ years old) with sarcopenia and physical
prefrailty or frailty. A previous Indonesian study in a similar population
(adults $\geq 60$ with pre-frailty) found that mean gait speed of patients
taking metformin compared to placebo was increased by 0.13 (SD 0.24) metres
per second (Laksmi et al.,, 2017).

The two treatments to be compared in the new trial are denoted $T$ (for
metformin) and $C$ (for the placebo control). The 4 metre walk speed at 4
months post-randomization will be denoted by $y_{ij},i=1,...,n_{j},j=T,C$, and
$y_{i,j}$ will be treated as normally distributed with mean $\mu_{j}$ and
common (known) variance $\sigma_{0}^{2}$, as in Section 3.3. The observed
difference in means $\bar{Y}_{T}-\bar{Y}_{C}=\bar{Y}_{\Delta}$ is assumed to
be normally distributed,
$\bar{Y}_{\Delta}\sim\mathcal{N}(\mu_{\Delta},\sigma_{0}^{2}/nR(1-R))$ with
positive values indicating an advantage for metformin. Based on the results of
the Indonesian trial, $\sigma_{0}^{2}=0.24^{2}$ (Laksmi et al.,, 2017).

Consider first a frequentist formulation of the sample size calculation. To
detect a minimum clinically important difference between treatment groups of
$\delta=0.1$ metres per second with a one-sided type I error rate
$\alpha=0.05$ and power $1-\beta=0.80$, the total sample size required is
minimized by equal allocation to treatment and control groups, i.e., $R=0.5$.
For these parameters, Equation (12) yields a total sample size of $n=144$
(rounded up to the nearest even integer). Our Bayesian sample size calculation
with no borrowing, Equation (11), gives the same result setting a large
$s_{0}^{2}$ (e.g., $s_{0}^{2}=100$), with $\eta=0.95$ and $\zeta=0.80$.

Let us now imagine a (contrived) scenario in which we have information
available from 5 small RCTs in the same drug which we wish to use for
calculating the sample size of the MET-PREVENT trial. We assume each
hypothetical historical trial has been summarized in the form of
$\lambda_{q}|\bm{y_{q}}\sim\mathcal{N}(\theta_{q},\tau_{q}^{2}),q=1,...,5$.
For illustrative purposes, suppose
$\lambda_{1}\sim\mathcal{N}(0.10,0.15^{2})$,
$\lambda_{2}\sim\mathcal{N}(0.15,0.10^{2})$,
$\lambda_{3}\sim\mathcal{N}(0.05,0.10^{2})$,
$\lambda_{4}\sim\mathcal{N}(0.10,0.12^{2})$, and
$\lambda_{5}\sim\mathcal{N}(0.10,0.13^{2})$. We then seek the opinion of a
clinical expert regarding the (dis)similarity on the percentage scale of each
data source to the new experiment. We note that it might be easier to elicit
these quantities as a degree of relevance (rather than degree of skepticism);
for example, if an expert thinks data source $q$ is $25\%$ relevant to the new
trial then we set $w_{q}=0.75$. Note that the process of eliciting expert
opinions is a complex topic outside the scope of this paper; for further
information see, for example, Quigley et al., (2018). We therefore suppose
that we have obtained a corresponding set of $\bm{w_{q}}\,\epsilon\,[0,1]$,
with $w_{1}=0.15,w_{2}=0.13,w_{3}=0.15,w_{4}=0.17,w_{5}=0.28$. This set would
imply a desire to incorporate a reasonable amount of information from sources
$1,...,4$, with less so incorporated from source $5$.

Firstly, using Equation (17) to transform $\bm{w_{q}}\to\bm{w^{\prime}_{q}}$
results in $\bm{w^{\prime}_{q}}=(3.60\times 10^{-4},1.36\times
10^{-4},1.60\times 10^{-4},2.68\times 10^{-4},5.96\times 10^{-4})$. Then,
using $\bm{w^{\prime}_{q}}$ and $\bm{\tau^{2}_{q}}$ in Equation (13) (setting
$a_{01}=1.1,b_{01}=1.1,a_{02}=1\times 10^{6},b_{02}=1$), the total sample size
required to detect a minimum clinically important difference between treatment
groups of $\delta=0.1$ metres per second for $\eta=0.95$ and $\zeta=0.80$ is
$n=72$ (rounded up to the nearest even integer). This represents a potential
saving in sample size of 72 subjects over the sample size calculated in the
frequentist manner or using the Bayesian formula with no borrowing.

Note, that if we just used the ‘raw’ $\bm{w_{q}}$ in Equation (13), we are
faced with the issue of over-discounting (described in previous sections),
resulting in a sample size of $142$: without transforming $\bm{w_{q}}$ they
are not interpretable on the (uniform) probability scale. Also note that if we
wished to include information to the specified degree (without transforming
$\bm{w_{q}}$ ), then we would have to have elicited values of
$\bm{w_{q}}=(3.60\times 10^{-4},1.36\times 10^{-4},1.60\times
10^{-4},2.68\times 10^{-4},5.96\times 10^{-4})$, which which would not be easy
to elicit from an expert, even if they had substantial statistical knowledge.

Finally, we also note that the actual sample size for the MET-PREVENT trial of
66 patients, i.e., 33 subjects per arm (not accounting for dropout rate), was
smaller due to the differing estimand. The treatment effect parameter targeted
in the actual MET-PREVENT trial is the mean of the post-randomization outcomes
adjusted for baseline using an ANCOVA design (Walters et al.,, 2019), assuming
a correlation between pre- and post-baseline measurements of $\rho=0.8$.
Bayesian methods would also become more efficient when adjusting for a
correlated baseline.

##  5 Performance Evaluation

We present a brief simulation study to verify that the proposed sample size
function and linearization method achieve the pre-specified statistical
properties across a range of scenarios. According to the Bayesian decision
framework in Section 3.2, the sample size should be large enough to guarantee
a conclusion of either efficacy, $\mathbb{P}(\theta>0)\geq\eta$, or, if not,
futility $\mathbb{P}(\theta\leq\delta)\geq\zeta$.

###  5.1 Basic Settings

Four base configurations, $k=1,...,4$, of hypothetical historical data are
generated, shown in Table 1. In each configuration, historical information
from 5 independent sources has been summarized in the form of
$\lambda_{q}|\bm{y_{q}}\sim\mathcal{N}(\theta_{q},\tau_{q}^{2})$, for
$q=1,...,5$. We suppose that $\bm{w_{q}}=w_{1},...,w_{5}$ have been elicited
to implement the proposed approach for borrowing information from the
respective source.

Table 1: Configurations $1,...,4$, of hypothetical historical data sources, $q=1,...,5$, assumed to have been independently summarized by a $\mathcal{N}(\theta_{q},\tau_{q}^{2})$ distribution. Each source is accompanied by a $w_{q}$ for borrowing of information, summarizing pre-experimental information about $\mu_{\Delta}$. |  | Historical data source, $q$: |
---|---|---|---
Config. | Parameter |  $1$ |  $2$ |  $3$ |  $4$ |  $5$
| $\theta_{q}$ |  0.260 |  0.240 |  0.370 |  0.340 |  0.320
$1$ | $\tau_{q}^{2}$ |  $0.250$ |  $0.230$ |  $0.220$ |  $0.360$ |  $0.260$
| $w_{q}$ |  0.252 |  0.319 |  0.140 |  0.306 |  0.149
| $\theta_{q}$ |  0.260 |  0.240 |  0.370 |  0.340 |  0.320
$2$ | $\tau_{q}^{2}$ |  0.150 |  0.150 |  0.150 |  0.150 |  0.150
| $w_{q}$ |  0.252 |  0.319 |  0.140 |  0.306 |  0.149
| $\theta_{q}$ |  0.260 |  0.170 |  0.440 |  0.150 |  -0.120
$3$ | $\tau_{q}^{2}$ |  0.250 |  0.640 |  0.970 |  1.540 |  0.590
| $w_{q}$ |  0.325 |  0.203 |  0.171 |  0.180 |  0.272
| $\theta_{q}$ |  0.260 |  0.170 |  0.440 |  0.150 |  -0.120
$4$ | $\tau_{q}^{2}$ |  0.250 |  0.150 |  0.400 |  0.890 |  0.220
| $w_{q}$ |  0.537 |  0.306 |  0.054 |  0.220 |  0.350

The minimally clinically important difference between treatment arms in the
new trial is set to be $\delta=0.4$. Probability boundaries for decision
making in terms of efficacy are $\eta=0.95$, and for futility, $\zeta=0.80$.
We assume a common (known) variance in outcome measures of
$\sigma^{2}_{0}=0.4$. For each configuration of historical data, a sample size
is calculated for the new trial: first, using Equation (17) to transform
$\bm{w_{q}}\to\bm{w^{\prime}_{q}}$, and then Equation (13), with
$\bm{w^{\prime}_{q}}$ and $\bm{\tau^{2}_{q}}$ (setting
$a_{01}=1.1,b_{01}=1.1,a_{02}=1\times 10^{3},b_{02}=1$). Note that, although
$\eta$ and $\zeta$ have been set to be equivalent to the often used
$(1-\alpha)$ and $(1-\beta)$ in the frequentist paradigm (which are set to
control type I error rate and power respectively), it must be remembered that
these values do not represent the same quantity. As discussed in Whitehead et
al., (2008), there is no reason to assume any form of equivalence since their
meanings are fundamentally different.

For the new trial, assuming equal allocation to treatment and control,
$R=0.5$, outcomes in the control group are generated for each configuration
according to $Y_{iC}\sim\mathcal{N}(0,0.4),i=1,...n_{k}/2$. Outcomes in the
treatment group are generated according to
$Y_{iT}\sim\mathcal{N}(\mu_{\Delta},0.4),i=1,...n_{k}/2$. For each simulation
replicate, true treatment effects are set to be one of the following:

  1. 1.

Effective treatment, $\mu_{\Delta}=0.4$.

  2. 2.

Ineffective treatment, $\mu_{\Delta}=0$.

A Bayesian analysis model will be applied to each simulation replicate, with
prior set according to the CPP from each configuration. Evidence of treatment
efficacy will be concluded if $\mathbb{P}(\mu_{\Delta}>0)\geq 0.95$. If
$\mathbb{P}(\mu_{\Delta}>0)<0.95$ then, according to our pre-specified
criteria, it should be the case that $\mathbb{P}(\mu_{\Delta}\leq\delta)\geq
0.80$. Results are summarized for (1) and (2) respectively by calculating the
percentage of trials in which a decisive conclusion can be reached by
averaging across 10,000 simulated trial replicates. This results in a total of
8 scenarios. The Bayesian analysis model is fitted analytically using
Equations (8) and (9) in R version 4.2.1 (2022-06-23).

###  5.2 Results

Table 2 gives transformed values of $w_{q}\to w^{\prime}_{q}$ via the method
described in Section 3.4.

Table 2: Transformed values of $w_{q}\to w^{\prime}_{q}$. |  | Historical data source, $q$: |
---|---|---|---
Config. |  |  $1$ |  $2$ |  $3$ |  $4$ |  $5$
$1$ | $w_{q}$ |  0.252 |  0.319 |  0.140 |  0.306 |  0.149
| $w^{\prime}_{q}$ |  $7.46\times 10^{-3}$ |  $9.54\times 10^{-3}$ |  $3.20\times 10^{-3}$ |  $1.38\times 10^{-2}$ |  $4.04\times 10^{-3}$
$2$ | $w_{q}$ |  0.252 |  0.319 |  0.140 |  0.306 |  0.149
| $w^{\prime}_{q}$ |  $4.54\times 10^{-3}$ |  $6.30\times 10^{-3}$ |  $2.20\times 10^{-3}$ |  $5.94\times 10^{-3}$ |  $2.37\times 10^{-3}$
$3$ | $w_{q}$ |  0.325 |  0.203 |  0.171 |  0.180 |  0.272
| $w^{\prime}_{q}$ |  $1.06\times 10^{-2}$ |  $1.38\times 10^{-2}$ |  $1.65\times 10^{-2}$ |  $2.63\times 10^{-2}$ |  $1.87\times 10^{-2}$
$4$ | $w_{q}$ |  0.537 |  0.306 |  0.054 |  0.220 |  0.350
| $w^{\prime}_{q}$ |  $2.52\times 10^{-2}$ |  $5.94\times 10^{-3}$ |  $2.00\times 10^{-3}$ |  $2.07\times 10^{-2}$ |  $1.05\times 10^{-2}$

Table 3 displays sample sizes for each configuration calculated using Equation
(13) (rounded up to the nearest even integer to allow for equal allocation
between treatment groups), along with the corresponding CPP used for analysis.

Table 3: Commensurate predictive priors for treatment effect in the new experiment, $\mu_{\Delta}\sim\mathcal{N}(\theta_{CPP^{*}},\sigma^{2}_{CPP^{*}})$, along with corresponding sample sizes (rounded up to nearest even integer for $R=0.5$) for configurations $1,...,4$. line Config. | $\theta_{CPP^{*}}(\bm{w^{\prime}_{q}},\bm{\theta_{q}},\bm{\tau^{2}_{q}})$ | $\sigma^{2}_{CPP^{*}}(\bm{w^{\prime}_{q}},\bm{\tau^{2}_{q}})$ | $n$
---|---|---|---
1 | 0.308 | 0.066 | 38
2 | 0.309 | 0.039 | 22
3 | 0.186 | 0.150 | 52
4 | 0.164 | 0.078 | 42

Table 4 displays the percentage of simulated trials concluding that
experimental treatment is effective (% Eff.) or futile (% Fut.) for each
configuration in scenarios where $\mu_{\Delta}=0.4$ and $\mu_{\Delta}=0$. The
percentage effective is defined as the percentage of trials out of 10,000
simulations in which $\mathbb{P}(\theta>0)\geq 0.95$, while the percentage
futile is the percentage of trials out of 10,000 simulations in which
$\mathbb{P}(\theta>0)<0.95$ and $\mathbb{P}(\theta\leq 0.4)\geq 0.80$. The
total percentage is $100\%$ in all scenarios, demonstrating that the pre-
specified statistical properties are upheld by the proposed method.

Table 4: Percentage of simulated trials that conclude E is efficacious or futile when $\mu_{\Delta}=0.4$ and $\mu_{\Delta}=0$. The prior for the treatment effect parameter in each trial is as specified in Table 3. |  |  $\mu_{\Delta}=0.4$: |  |  $\mu_{\Delta}=0$: |
---|---|---|---|---|---
Config. | n | % Eff. | % Fut. | Total % | % Eff. | % Fut. | Total %
1 | 38 | 79.0 | 21.0 | 100 | 12.5 | 87.5 | 100
2 | 22 | 80.7 | 19.3 | 100 | 24.9 | 75.1 | 100
3 | 52 | 75.9 | 24.1 | 100 | 5.28 | 94.7 | 100
4 | 42 | 67.5 | 32.5 | 100 | 5.60 | 94.4 | 100

##  6 Discussion

Our central goal in this paper has been to offer a solution for the problem of
nonlinear discrepancy weights in a Bayesian sample size formula. Following our
proposed method, given a set of historical data sources, clinical expert(s)
only have to specify values on the percentage scale representing the
(dis)similarity between each historic data source and the current trial to
incorporate a specified amount of information. We hope that these ideas can
encourage effective communication between statisticians and subject matter
experts to elicit sensible values for these weights.

Focus in this work is on the design of a two-armed trial where there is prior
information on the difference in means between treatment and control arms. We
acknowledge that it is more common for methods for borrowing from historical
data to consider borrowing only for the control arm (i.e., using historical
control information to supplement or replace a concurrent control). The
methods presented here could be adapted to this case such that a commensurate
prior would be formed for the arm-based statistic(s). As noted in Zheng et
al., 2023a , selection of historical data on a single arm should be done
carefully to avoid bias that may affect the inference of the difference in
means. Our sample size formula and linearization technique could also be
extended to a basket trial scenario (for concurrent borrowing between
subtrials) in which a sample size is sought for each subtrial, $k=1,...,K$,
with sample sizes being solved as a system of $K$ simultaneous equations.

The proposed method specifies a singular prior for both the design and
analysis of the new experiment. There may be instances where it is desirable
to adjust the analysis prior according to the observed similarity between the
historic datasets and current trial. In this case, a distributional distance
metric $\epsilon[0,1]$ such as the Hellinger distance might be useful in
determining $w_{q}$ for the analysis. However, as noted in Zheng et al., 2023b
, when $w_{q}$ is set to a larger value in the analysis than in the design
(i.e., less borrowing is implemented than planned), a smaller percentage of
trials will conclude with a decisive decision. Conversely, using smaller
$w_{q}$ in the analysis than the design would lead to a more precise posterior
distribution which may result in ambiguous conclusions regarding efficacy or
futility.

In our approach we have restricted focus to known variance in outcome measure
($\sigma_{0}^{2}$) as this is common in many settings. In addition, we
approximated $\nu_{q}$ by making some simplifying assumptions, which resulted
in a closed form for the sample size calculation. One avenue for development
would be a more fully Bayesian approach in which priors are specified for both
$\sigma_{0}^{2}$ and $\nu_{q}$. The sample size in the new trial would then be
sought by controlling average properties of posterior interval probabilities
of $\mu_{\Delta}$ with respect to $0$ and $\delta$, as in Zheng et al., 2023a
.

In conclusion, historical data from a range of sources is often available in
the planning of a new trial. We offer statisticians a way to incorporate this
information in a flexible manner at the design stage with the support of a
clinical expert(s).

## Acknowledgements

Dr Zheng’s contribution to this work was supported by Cancer Research UK
(RCCPDF\100008). James M. S. Wason is funded by NIHR Research Professorship
(NIHR301614).

## Supplementary Materials

R code for reproducing the Motivating Example and Performance Evaluation is
posted online at GitHub: https://github.com/lou-e-whitehead/BayesianSSD

## References

  * Bornkamp, (2012) Bornkamp, B. (2012).  Functional uniform priors for nonlinear modeling.  Biometrics, 68(3):893–901.
  * Bornkamp, (2014) Bornkamp, B. (2014).  Practical considerations for using functional uniform prior distributions for dose-response estimation in clinical trials.  Biometrical Journal, 56(6):947–962.
  * Dey and Birmiwal, (1994) Dey, D. K. and Birmiwal, L. R. (1994).  Robust Bayesian analysis using divergence measures.  Statistics & Probability Letters, 20(4):287–294.
  * Dias et al., (2017) Dias, L. C., Morton, A., and Quigley, J. (2017).  Elicitation: The Science and Art of Structuring Judgement.  International Series in Operations Research & Management Science. Springer International Publishing.
  * Grinstead and Snell, (1997) Grinstead, C. M. and Snell, J. L. (1997).  Introduction to probability.  American Mathematical Soc.
  * Hampson et al., (2014) Hampson, L. V., Whitehead, J., Eleftheriou, D., and Brogan, P. (2014).  Bayesian methods for the design and interpretation of clinical trials in very rare diseases.  Statistics in Medicine, 33(24):4186–4201.
  * Hariton and Locascio, (2018) Hariton, E. and Locascio, J. J. (2018).  Randomised controlled trials – the gold standard for effectiveness research.  BJOG: An International Journal of Obstetrics & Gynaecology, 125(13):1716–1716.
  * Hobbs et al., (2011) Hobbs, B. P., Carlin, B. P., Mandrekar, S. J., and Sargent, D. J. (2011).  Hierarchical commensurate and power prior models for adaptive incorporation of historical information in clinical trials.  Biometrics, 67(3):1047–1056.
  * Hobbs et al., (2012) Hobbs, B. P., Sargent, D. J., and Carlin, B. P. (2012).  Commensurate Priors for Incorporating Historical Information in Clinical Trials Using General and Generalized Linear Models.  Bayesian Analysis, 7(3):639 – 674.
  * Ibrahim and Chen, (2000) Ibrahim, J. G. and Chen, M.-H. (2000).  Power prior distributions for regression models.  Statistical Science, pages 46–60.
  * Johnson et al., (2010) Johnson, S. R., Tomlinson, G. A., Hawker, G. A., Granton, J. T., and Feldman, B. M. (2010).  Methods to elicit beliefs for Bayesian priors: a systematic review.  Journal of clinical epidemiology, 63(4):355–369.
  * Julious, (2023) Julious, S. A. (2023).  Sample sizes for clinical trials.  CRC Press.
  * Laksmi et al., (2017) Laksmi, P. W., Setiati, S., Tamin, T. Z., Soewondo, P., Rochmah, W., Nafrialdi, and Prihartono, J. (2017).  Effect of metformin on handgrip strength, gait speed, myostatin serum level, and health-related quality of life: A double blind randomized controlled trial among non-diabetic pre-frail elderly patients.  Acta Medical Indonesia, 49(2):118–127.
  * Neuenschwander et al., (2010) Neuenschwander, B., Capkun-Niggli, G., Branson, M., and Spiegelhalter, D. J. (2010).  Summarizing historical information on controls in clinical trials.  Clinical Trials, 7(1):5–18.  PMID: 20156954.
  * Pocock, (1976) Pocock, S. J. (1976).  The combination of randomized and historical controls in clinical trials.  Journal of Chronic Diseases, 29(3):175–188.
  * Quigley et al., (2018) Quigley, J., Quigley, J. B., Dias, L. C., and Morton, A. (2018).  Elicitation the science and art of structuring judgement.  Springer.
  * Rennie et al., (2022) Rennie, K. J., Witham, M., Bradley, P., Clegg, A., Connolly, S., Hancock, H. C., Hiu, S., Marsay, L., McDonald, C., Robertson, L., Simms, L., Steel, A. J., Steves, C. J., Storey, B., Wason, J., Wilson, N., von Zglinicki, T., and Sayer, A. A. P. (2022).  Met-prevent: metformin to improve physical performance in older people with sarcopenia and physical prefrailty/frailty – protocol for a double-blind, randomised controlled proof-of-concept trial.  BMJ Open, 12(7).
  * Schmidli et al., (2014) Schmidli, H., Gsteiger, S., Roychoudhury, S., O’Hagan, A., Spiegelhalter, D., and Neuenschwander, B. (2014).  Robust meta-analytic-predictive priors in clinical trials with historical control information.  Biometrics, 70(4):1023–1032.
  * Spiegelhalter et al., (2004) Spiegelhalter, D. J., Abrams, K. R., and Myles, J. P. (2004).  Bayesian approaches to clinical trials and health-care evaluation, volume 13.  John Wiley & Sons.
  * Viele et al., (2014) Viele, K., Berry, S., Neuenschwander, B., Amzal, B., Chen, F., Enas, N., Hobbs, B., Ibrahim, J. G., Kinnersley, N., Lindborg, S., Micallef, S., Roychoudhury, S., and Thompson, L. (2014).  Use of historical control data for assessing treatment effects in clinical trials.  Pharmaceutical Statistics, 13(1):41–54.
  * Wadsworth et al., (2018) Wadsworth, I., Hampson, L. V., and Jaki, T. (2018).  Extrapolation of efficacy and other data to support the development of new medicines for children: A systematic review of methods.  Statistical Methods in Medical Research, 27(2):398–413.  PMID: 26994211.
  * Walters et al., (2019) Walters, S. J., Jacques, R. M., dos Anjos Henriques-Cadby, I. B., Candlish, J., Totton, N., and Xian, M. T. (2019).  Sample size estimation for randomised controlled trials with repeated assessment of patient-reported outcomes: What correlation between baseline and follow-up outcomes should we assume?  Trials, 20(1).
  * Whitehead et al., (2008) Whitehead, J., Valdés-Márquez, E., Johnson, P., and Graham, G. (2008).  Bayesian sample size for exploratory clinical trials incorporating historical data.  Statistics in Medicine, 27(13):2307–2327.
  * Winkler, (1981) Winkler, R. L. (1981).  Combining probability distributions from dependent information sources.  Management Science, 27(4):479–488.
  * Woodcock and LaVange, (2017) Woodcock, J. and LaVange, L. M. (2017).  Master protocols to study multiple therapies, multiple diseases, or both.  New England Journal of Medicine, 377(1):62–70.  PMID: 28679092.
  * Zheng and Hampson, (2020) Zheng, H. and Hampson, L. V. (2020).  A Bayesian decision-theoretic approach to incorporate preclinical information into phase I oncology trials.  Biometrical Journal, 62(6):1408–1427.
  * (27) Zheng, H., Jaki, T., and Wason, J. M. S. (2023a).  Bayesian sample size determination using commensurate priors to leverage preexperimental data.  Biometrics, 79(2):669–683.
  * (28) Zheng, H., Grayling, M. J., Mozgunov, P., Jaki, T., and Wason, J. M. S. (2023b).  Bayesian sample size determination in basket trials borrowing information between subsets.  Biostatistics, 24(4):1000 – 1016.
  * Zheng and Wason, (2022) Zheng, H. and Wason, J. M. S. (2022).  Borrowing of information across patient subgroups in a basket trial based on distributional discrepancy.  Biostatistics, 23(1):120–135.


# TIPICAL - Type Inference for Python In Critical Accuracy Level

1st Jonathan Elkobi  GPS
UC San Diego
San Diego, USA
<EMAIL_ADDRESS>2nd Bernd Gruner  Institute of Data Science
German Aerospace Center
Jena, Germany
<EMAIL_ADDRESS>3rd Tim Sonnekalb  Institute of Data Science
German Aerospace Center
Jena, Germany
<EMAIL_ADDRESS>4th Clemens-Alexander Brust  Institute of Data Science
German Aerospace Center
Jena, Germany
<EMAIL_ADDRESS>
###### Abstract

Type inference methods based on deep learning are becoming increasingly
popular as they aim to compensate for the drawbacks of static and dynamic
analysis approaches, such as high uncertainty. However, their practical
application is still debatable due to several intrinsic issues such as code
from different software domains will involve data types that are unknown to
the type inference system.

In order to overcome these problems and gain high-confidence predictions, we
thus present TIPICAL, a method that combines deep similarity learning with
novelty detection. We show that our method can better predict data types in
high confidence by successfully filtering out unknown and inaccurate predicted
data types and achieving higher F1 scores to the state-of-the-art type
inference method Type4Py. Additionally, we investigate how different software
domains and data type frequencies may affect the results of our method.

###### Index Terms:

type inference, novelty detection, machine learning, cross-domain

##  I Introduction

Dynamic programming languages can be enriched by optional type annotations to
enable more precise program analysis and early detection of type-related run-
time errors [1, 2]. Our objective is to develop a workable technique for
Python programmers to use on a daily basis that will enhance their routine
workflow through the annotation of optional data types, a process known as
type inference. Giving type recommendations to the user in real-time as well
as automatically after writing the code will accomplish this. Due to its
automation, the first case, however, necessitates a method that only annotates
with high certainty of correctness, as the harm that inaccurate type hints can
do may be greater than the benefit of the positive one.

Static and dynamic type inference techniques suffer from low precision due to
applied abstraction or missing coverage [3]. Recent deep learning-based
methods aim to overcome these issues and provide promising results [4, 5, 6,
7]. However, these systems struggle with problems occurring in practical
applications for example data types unknown to the system or source code from
other software domains [8].

First, the problem of unknown classes, or the inability to accurately predict
unseen data types, is a prevalent issue in the field of machine learning. This
is due to the lack of representation of such data types in the training set,
rendering prediction of these data types ineffective. To mitigate this issue,
we propose a method to filter out unknown data types based on their
characteristic features. This approach not only enables the identification of
unknown data types but also improves the overall reliability and quality of
data type annotations by eliminating inaccurate predictions.

In practical usage, type inference systems are often utilized in a variety of
software domains. However, this can exacerbate the problem of unknown data
types and lead to decreased accuracy of predictions due to dataset shifts [9,
10]. Therefore, in our research, we investigate the effect of different
software domains on the performance of our type inference method.

We thus present TIPICAL \- Type Inference for Python In Critical Accuracy
Level, an extension of the Type4Py method [6] for obtaining accurate results
that address both the unknown data types problem and the domain shift in order
to maximize the practical application of deep learning-based type inference.
In our experiments, we show that our method can successfully filter out
unknown and inaccurate predicted data types and improve the results compared
to the state-of-the-art type inference method Type4Py.

Furthermore, we investigate the impact of the data type frequencies and
different software domains. For the evaluation, we use the recent datasets
CrossDomainTypes4Py [8] and ManyTypes4Py [11]. In order to ensure the
reproducibility of our experiments, we make our experimental pipeline publicly
available111https://gitlab.com/dlr-dw/type-inference.

##  II Related Work

Figure 1: Deciding the threshold using the validation set

Several studies use deep learning techniques for type inference. In this work,
we focus on methods that are designed for Python projects. DLType [12],
TypeWriter [13], and PyInfer [14] are the first deep learning-based methods in
that field. They suffer from the problem that they can only predict a limited
number of data types due to their architecture. These methods are limited to
the 500 or 1000 most frequent data types that occur in the training dataset.

Typilus [4], and Type4Py [6] address this problem and can predict all data
types which are present in the training dataset. There are potential
approaches where new types can be recognized through additional static
analysis, as demonstrated in HiTyper [15]. However, there is still the issue
that data types that are rarely or not in the training dataset cannot be
predicted[8].

Nevertheless, Novelty detection has not been actively pursued as a solution to
the problem. Therefore we develop TIPICAL to mitigate these problems. As a
basis for our method, we use Type4Py, because according to the evaluation of
Mir et al. [6], it is state-of-the-art and the source code is publicly
available.

##  III Methodology

In this section, we briefly explain the type inference method Type4Py, which
is the basis for our method. Afterward, the structure and the functioning of
TIPICAL are presented.

###  III-A Type4Py Inference System

The Type4Py framework serves as the foundation for our research, as detailed
in the original paper [6]. The Type4Py system utilizes code tokens, identifier
names, and available data types (visible type hints) as input. These code
tokens and identifier names are embedded by Word2Vec [16] and processed
separately through recurrent neural networks. Then the resulting
representations are concatenated with the visible type hints and further
processed through a fully connected layer to generate a feature vector. This
feature vector is then used for a k-nearest neighbor search in the type
cluster to predict all data types using the training dataset. However, it
should be noted that this method may not accurately predict unknown data types
that are not represented in the training dataset, as well as those that are
inaccurately predicted due to limitations of using the nearest neighbor as a
classification method

Figure 2: Applying the threshold to make predictions on the test set

###  III-B TIPICAL

As a result, we sought to develop TIPICAL, a comprehensive system that would
enhance the usage of type inference by utilizing novelty detection to filter
out inaccurate predictions and unpredictable data types. Using the workflow
presented in Figure 1, we find a threshold for filtering, We then use the same
cluster centers threshold on the test set to filter out predictions.

####  III-B1 Determining the Threshold

First, we determine the cluster centers of each known data type using the
training dataset, defined as follows:

$\bar{\vec{x}}=\frac{\sum_{i=1}^{n}\vec{x_{i}}}{n},$ (1)

where $\vec{x}$ is the feature vector, $n$ is the total number of vectors and
$\bar{\vec{x}}$ is the cluster center vector. Next, we determine the top two
nearest cluster centers for each vector in the validation set, and their
distances $d1$ and $d2$.

$\Delta d=d_{2}-d_{1}$ (2)

Then $\Delta d$ is calculated, as seen in Equation 2. It can be interpreted as
cost-effective active learning or the proxy of the entropy of the distribution
of the distances between all of the cluster centers.

After which, using these facts, we develop a threshold-based method to
determine whether or not the closest cluster center can accurately predict the
data type. The reasoning behind this is that if two cluster centers are
roughly the same distance from the vector, it may be difficult to distinguish
which datatype is the correct one or even they could be an unknown data type
that is just not represented in the training set, whereas if the closest
cluster center is significantly closer to the vector than the second one, it
will almost certainly make the correct prediction. The outliers are labeled as
non-predicted for further research directions as can be seen later in the
conclusion.

####  III-B2 Making the Predictions

Afterward, for the prediction itself, we use the nearest cluster center as the
predicted data type. In order to make accurate predictions, we maximize the F1
score on the validation set by determining until which value of $\Delta d$ we
are going to filter out the vectors. Finally, as can be seen in the second
part of Figure 1, we apply the same approach to the test set using our
previous findings. We calculate $d1$, $d2$, and $\Delta d$ for all the test
set vectors. Further, we apply the threshold that we determined from the
training set and filter out lower $\Delta d$ from our final predictions,
producing only high-certainty predictions that will provide our end-user with
the most accurate predictions, as will be demonstrated in the following
section.

##  IV Experiments and Evaluation

Following the pipeline of TIPICAL described in Section III-B, we present our
experiments on the ManyTypes4Py and the CrossDomainTypes4Py datasets.
Moreover, we expanded the scope of the experiments of the original papers[6,
8] to further study the effects of different software domains, and the unknown
data types issues.

In addition to creating a comprehensive method for real-life machine learning-
based type inference with high certainty, we conducted the experiments to
answer the following research questions:

  1. 1.

How do different software domains affect the predictable and unknown data type
distribution according to the entropy proxy - $\Delta d$?

  2. 2.

How does the nearest cluster center-based method accuracy of the data types
predictions correspond to $\Delta d$?

  3. 3.

Can TIPICAL create higher certainty predictions than the predictions of the
Type4Py system?

  4. 4.

How does the frequency of data types affect its predictability in TIPICAL?

###  IV-A Datasets and Domains

We use the CrossDomainTypes4Py [8] and ManyTypes4Py [11] datasets for our
experiments. As described by Gruner et al. [8], these contain a total of at
least three different software domains. CrossDomainTypes4Py consists of the
scientific calculation (cal) domain with 4,783 repositories and the web
development (web) domain with 3,129. ManyTypes4Py, on the other hand, contains
5,382 repositories, which are from various domains and are therefore
considered general (mtp).

###  IV-B Experiment Setup

We adapt the existing cross-domain Type4Py implementation from Gruner et al.
[8] in order to conduct the research. We employ PyTorch, a deep learning
framework, using Python 3.6. We use the same hyperparameters as Mir et al [6].

For our experiments, we created the following four cross-domain setups:

  1. 1.

Setup Cal2Mtp - Scientific Calculation to General

  2. 2.

Setup Mtp2Cal - General to Scientific Calculation

  3. 3.

Setup Cal2Web - Scientific Calculation to Web Development

  4. 4.

Setup Web2Cal - Web Development to Scientific Calculation

###  IV-C Research Questions and Results

RQ1: How do different software domains affect the predictable and unknown data
types distribution according to the entropy proxy - $\Delta d$?
Each setup consists of two experiments, which are conducted three times to
calculate the average of the results. In the first experiment, the system is
trained on the first mentioned domain and evaluated on the second domain
(example reference: Cal2Mtp.1). The second experiment is for comparison and
performs the training and the evaluation on the second mentioned domain
(example reference: Cal2Mtp.2). In total we get eight results from our four
setups.

Figure 3: Histogram of precitable vs. unknown data types distribution according to $\Delta d$ for setup Cal2Mtp.1  Figure 4: Histogram of precitables VS unknown data types distribution according to $\Delta d$ for Setup Cal2Mtp.2  TABLE I: Method Scores Comparison Methods/Setups | Cal2Mtp.1 | Cal2Mtp.2 | Mtp2Cal.1 | Mtp2Cal.2 | Cal2Web.1 | Cal2Web.2 | Web2Cal.1 | Web2Cal.2
---|---|---|---|---|---|---|---|---
TIPICAL [F1] | 67.09% | 88.05% | 87.41% | 64.26% | 88.65% | 77.45% | 76.77% | 86.17%
Type4Py [F1] | 43.96% | 44.69% | 45.27% | 58.74% | 47.41% | 49.33% | 47.20% | 55.35%
Type4Py only predictables [F1] | 65.26% | 64.55% | 63.27% | 73.71% | 67.18% | 67.17% | 65.09% | 71.47%

Figures 3 & 4 show that using examples from various software domains has no
discernible impact on the target domains. The amount of unpredictable types,
which sharply increases due to the size of the source dataset as a whole
rather than a change in the use case itself, is another intriguing development
that follows from this. Nevertheless, those findings are in favor of our
approach since, by removing the lower scores of the $\Delta d$, we will also
remove most of the unknown data types since they are closer to at least two
cluster centers within a comparable distance. Hence we provide a novelty
detection method for practical use.

RQ2: How does a nearest cluster center-based method accuracy of the data types
predictions corresponds to $\Delta d$?

Figure 5: Histogram of accurate and inaccurate predictions distribution
according to $\Delta d$ for Setup Web2Cal.1  Figure 6: Histogram of accurate
and inaccurate predictions distribution according to $\Delta d$for Setup
Web2Cal.2

Figures 5 & 6 illustrate that various domains do not affect the accuracy of
the target domains of the closest cluster center’s prediction. However, the
closest cluster center for the right predictions examples causes the
distribution of $\Delta d$ to be shifted to a greater distinction. As a
result, we can conclude that a domain change use case might be predicted with
a better degree of certainty using our method. In addition, those results
support our method because by filtering the lower scores of $\Delta d$. We
will also eliminate the majority of incorrect predictions since they are
located nearer to at least two cluster centers than the accurate ones.

Figure 7: Histogram of common and rare types distributed across the
subtraction of the 2 Nearest cluster centers for Setup 3.1

RQ3: Can TIPICAL create higher certainty predictions than the predictions of
the Type4Py system?

TABLE II: Example Categories Distribution Samples/Setups | Cal2Mtp.1 | Cal2Mtp.2 | Mtp2Cal.1 | Mtp2Cal.2 | Cal2Web.1 | Cal2Web.2 | Web2Cal.1 | Web2Cal.2
---|---|---|---|---|---|---|---|---
Predicted Samples | 38.31% | 30.62% | 32.48% | 41.83% | 30.09% | 37.10% | 35.10% | 27.50%
Samples Predicted Accurate | 24.62% | 26.96% | 28.77% | 27.67% | 25.93% | 28.73% | 26.96% | 24.67%
Predictable Samples Predicted Inaccurate | 11.55% | 2.62% | 2.18% | 11.60% | 3.39% | 5.39% | 5.84% | 2.16%
Unknown Samples Predicted | 2.14% | 1.04% | 1.53% | 2.56% | 0.77% | 2.98% | 2.30% | 0.66%
Non-Predicted Examples | 61.69% | 69.38% | 67.52% | 58.17% | 69.91% | 62.90% | 64.90% | 72.50%
Samples Non-Predicted Accurate | 2.96% | 3.12% | 2.82% | 2.95% | 3.67% | 5.08% | 4.26% | 3.39%
Predictable Samples Non-Predicted Inaccurate | 46.52% | 36.26% | 35.19% | 43.42% | 47.04% | 31.29% | 33.43% | 49.80%
Unknown Samples Non-Predicted | 12.22% | 30.00% | 29.52% | 11.80% | 19.20% | 26.53% | 27.21% | 19.31%

Table 1 presents a comparison of the final results of our proposed method,
TIPICAL, with the benchmark method. It is evident from this table that TIPICAL
produces significantly better high-certainty type annotations for the end
user, even across different software domains. On the other hand, the benchmark
method exhibits a significant amount of noise, resulting from numerous
inaccurate predictions of both predictable and unpredictable data types. To
obtain the final F1 scores, our method employs a filtering approach, as
described earlier, by utilizing a threshold determined from the validation
group of the training dataset, effectively reducing the majority of the noise.
Finally, TIPICAL reach better results in all of the experiments, with an
average of 79.48% F1 score, which is 30.49% improvement from the average of
the Type4Py method.

To further demonstrate the effectiveness of our proposed method, TIPICAL, in
comparison to the Types4Py technique, Table 1 presents a comparison of the
results obtained when only predicting the predictable data types in the test
dataset. Even in this scenario where the predictable data types are known,
TIPICAL outperforms Types4Py in 7 out of 8 experiments. On average, our method
gets 12.27% F1 score. This serves as evidence that TIPICAL, which does not
make any assumptions about the target dataset, is more effective than methods
that rely on prior knowledge of the predictable data types.

RQ4: How does the frequency of data types affect its predictability in
TIPICAL?

The frequency of a data type influences how accurately predictable it is.
Figures 7 and 8 show that common types (types that appear in the training more
than 100 times [6]) are presented after the threshold at a higher rate than
rare or unknown types. Additionally, although the rare type has a longer tail,
we can see similarities between the distribution of the unknown and rare
types. Because the calculated cluster centers of the common data types more
closely resemble the novel label features, we can also predict the common data
types more accurately than the rare types. Furthermore, the distribution of
the common type is impacted by different software domains, whereas the
distribution of the rare type is not. This may help to explain why some cases
have decreased accuracy.

##  V Limitations

Figure 8: Histogram of common and rare type distributions across the
subtraction of the 2 Nearest cluster centers for Setup 3.2

In order to perform a comparative analysis with the current benchmark of
Type4Py, we conducted a repetition of previous experiments while disregarding
the validation data type. This enabled us to make a comparison with the
methodology employed by Type4Py.

As demonstrated in Table 2, our method predicts approximately one-third of the
examples within the dataset. Although, as indicated by the ”Samples Non-
Predicted Accurate” row, a trade-off of 3.53% of accurate predictions must be
made on average to apply the threshold as TIPICAL is designed to predict only
a portion of the information. Nonetheless, by excluding, on average, 92.63% of
the unknown data types (21.97% of the dataset) and 40.37% of potentially
inaccurate predictions, as indicated by the ”Unknown Samples Non-Predicted”
and ”Predictable Samples Non-Predicted Innacurate” rows, respectively, we
observed an increase in the confidence of our predictions by 30.49% (F1) in
comparison to the Type4Py method. Our findings were consistent with this
conclusion, as exemplified by setup 1.2 where TIPICAL demonstrated a 43.36%
improvement in performance while only predicting 30.62% of the examples. As a
result, our research confirms that a lower prediction count results in higher
scores.

##  VI Conclusion

This study offers a thorough procedure for carrying out the most precise deep
learning-based type inference, TIPICAL. It reaches an average 79.48% F1 score,
which is 30.49% improvement from the Types4Py method. We accomplish this by
adding steps to the Type4Py pipeline’s final stages that address the two main
issues with the current approaches: predictability and uncertainty. TIPICAL is
a novel approach to type inference in Python, utilizing a combination of
machine learning and novelty detection techniques to improve the accuracy of
predictions. The system is built upon the Type4Py framework, which uses the
closest neighbor as a sole method for classification, which can lead to
inaccuracies and unpredictable data types. To mitigate this issue, TIPICAL
employs a threshold-based method to determine the accuracy of predictions by
comparing the distance to the closest cluster center to the second closest.
This approach results in the filtering out of lower certainty predictions,
thus maximizing the overall accuracy of predictions on both the validation and
test sets. Consequently, the best current method for use in practice that
needs high certainty of data type annotation is produced. Nevertheless, with
our 8 experiments across 3 domains, we focus on 3 subjects. Real-world
software domain changes, which mainly affect the accuracy of the predictable
data types but not the prediction percentage of the unpredictable data types,
lead to the conclusion that our technique can compete well even then.
Additionally, the fact that the main types are common and easier to predict,
further increases the validity of our method, mainly due to the robustness of
predicting well-represented types in the training set.

Our findings suggest that applying the type inference automatically using
TIPICAL would be preferable because it will cause less harm, however, it may
not be as effective for real-time suggestions when the developer may review
all of the type annotations.

Last but not least, we can suggest further developing this method by utilizing
lifelong learning techniques to improve the general predictability of labels
and enrich the predictable types. Moreover, another easy to achieve better
results is to use only the common types from the training set to create the
cluster centers, hence, the sureness of the cluster center will raise, and the
approach will predict probably higher F1 scores, but with fewer samples. Due
to the countless possible types, for example, by providing an example that is
below the threshold so that a domain expert can label it and repeat the entire
process indefinitely[17]. Researchers can also use a broad method to determine
the genuine entropy, then follow the instructions in our method to increase
the method’s effectiveness and further study it.

## References

  * [1] S. Hanenberg, S. Kleinschmager, R. Robbes, É. Tanter, and A. Stefik, “An empirical study on the impact of static typing on software maintainability,” _Empirical Software Engineering_ , vol. 19, pp. 1335–1382, 2013.
  * [2] Z. Gao, C. Bird, and E. T. Barr, “To type or not to type: Quantifying detectable bugs in javascript,” in _2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)_ , 2017, pp. 758–769.
  * [3] T. S. Heinze, A. Møller, and F. Strocco, “Type safety analysis for dart,” in _Proceedings of the 12th Symposium on Dynamic Languages_ , ser. DLS 2016\. New York, NY, USA: Association for Computing Machinery, 2016, p. 1–12. [Online]. Available: https://doi.org/10.1145/2989225.2989226
  * [4] M. Allamanis, E. T. Barr, S. Ducousso, and Z. Gao, “Typilus: Neural type hints,” in _Proceedings of the 41st acm sigplan conference on programming language design and implementation_ , ser. PLDI 2020. New York, NY, USA: Association for Computing Machinery, 2020, p. 91–105. [Online]. Available: https://doi.org/10.1145/3385412.3385997
  * [5] I. V. Pandi, E. T. Barr, A. D. Gordon, and C. Sutton, “Opttyper: Probabilistic type inference by optimising logical and natural constraints,” 2021.
  * [6] A. M. Mir, E. Latoškinas, S. Proksch, and G. Gousios, “Type4py: Practical deep similarity learning-based type inference for python,” in _Proceedings of the 44th International Conference on Software Engineering_ , 2022, pp. 2241–2252.
  * [7] K. Jesse, P. T. Devanbu, and T. Ahmed, “Learning type annotation: Is big data enough?” in _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ser. ESEC/FSE 2021. New York, NY, USA: Association for Computing Machinery, 2021, p. 1483–1486. [Online]. Available: https://doi.org/10.1145/3468264.3473135
  * [8] B. Gruner, T. Sonnekalb, T. S. Heinze, and C.-A. Brust, “Cross-domain evaluation of a deep learning-based type inference system,” _arXiv preprint arXiv:2208.09189_ , 2022.
  * [9] F. Maia Polo, R. Izbicki, J. Lacerda, J. Ibieta-Jimenez, and R. Vicente, “A unified framework for dataset shift diagnostics,” 05 2022.
  * [10] S. G. Finlayson, A. Subbaswamy, K. Singh, J. Bowers, A. Kupke, J. Zittrain, I. S. Kohane, and S. Saria, “The clinician and dataset shift in artificial intelligence,” _The New England journal of medicine_ , vol. 385, no. 3, p. 283, 2021.
  * [11] A. M. Mir, E. Latoskinas, and G. Gousios, “Manytypes4py: A benchmark python dataset for machine learning-based type inference,” in _IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)_. IEEE Computer Society, May 2021, pp. 585–589.
  * [12] C. Boone, N. de Bruin, A. Langerak, and F. Stelmach, “Dltpy: Deep learning type inference of python function signatures using natural language context,” 2019.
  * [13] M. Pradel, G. Gousios, J. Liu, and S. Chandra, “Typewriter: Neural type prediction with search-based validation,” in _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ser. ESEC/FSE 2020. New York, NY, USA: Association for Computing Machinery, 2020, p. 209–220. [Online]. Available: https://doi.org/10.1145/3368089.3409715
  * [14] S. Cui, G. Zhao, Z. Dai, L. Wang, R. Huang, and J. Huang, “Pyinfer: Deep learning semantic type inference for python variables,” 2021.
  * [15] Y. Peng, Z. Li, C. Gao, B. Gao, D. Lo, and M. Lyu, “Hityper: A hybrid static type inference framework with neural prediction,” 2021.
  * [16] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” _arXiv preprint arXiv:1301.3781_ , 2013\.
  * [17] C. Käding, E. Rodner, A. Freytag, and J. Denzler, “Watch, ask, learn, and improve: a lifelong learning cycle for visual recognition,” in _ESANN_ , 2016\.