# Extending the coefficient of variation for measuring heterogeneity following
a meta-regression

Maxwell Cairns, Luke A. Prendergast
Department of Mathematics and Statistics
La Trobe University
Melbourne, Australia
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Meta-regression is often used to form hypotheses about what is associated with
heterogeneity in a meta-analysis and to estimate the extent to which effects
can vary between cohorts and other distinguishing factors. However, study-
level variables, called moderators, that are available and used in the meta-
regression analysis will rarely explain all of the heterogeneity. Therefore,
measuring and trying to understand residual heterogeneity is still important
in a meta-regression, although it is not clear how some heterogeneity measures
should be used in the meta-regression context. The coefficient of variation,
and its variants, are useful measures of relative heterogeneity. We consider
these measures in the context of meta-regression which allows researchers to
investigate heterogeneity at different levels of the moderator and also
average relative heterogeneity overall. We also provide CIs for the measures
and our simulation studies show that these intervals have good coverage
properties. We recommend that these measures and corresponding intervals could
provide useful insights into moderators that may be contributing to the
presence of heterogeneity in a meta-analysis and lead to a better
understanding of estimated mean effects.

##  1 Introduction

In meta-analyses, the presence of heterogeneity is common and is seen by many
as a nuisance that contributes to estimator variability. However, if we can
identify one or more moderating variables, called moderators, that can account
for at least some of the heterogeneity, then heterogeneity may give valuable
insights. A successful moderator analysis can identify variables, the
independent variables (IVs) or moderators, that vary over studies and
contribute to variation in the dependent variable (DV) which is the effect
measure of interest. No single study may have manipulated a particular
variable as an IV, but if that variable varies over studies, and is identified
as a moderator, then the meta-analysis is giving fresh insight that may guide
future research.

A very useful type of moderator analysis is meta-regression. A meta-regression
is analogous to linear regression where we model the DV against moderators of
interest. For example, de Bruin et al., (2009) uses the standard quality of
participant care as a moderating variable.

There are two common types of moderators: numerical moderators and factor
moderators. A numerical moderator simply takes numerical values for $x_{i}$
for the $i$-th study and $\beta_{1}$ is interpreted as the change in the true
mean effect for each unit change of $x_{i}$; for example, the data considered
by Colditz et al., (1994) which considers the effect of a tuberculosis
vaccine. For this meta-analysis a numeric moderator is absolute latitude as a
geographical indicator of where a study was conducted. A factor moderator,
either ordinal or nominal, takes values for different levels of the moderator.
In Colditz et al., (1994), allocation method (random, alternate or systemic)
is a factor moderator indicating how treatment was allocated for each study.
In general, when we are estimating effects we are interested in moderators
which measure sample characteristics (e.g., average or median cohort age). We
would not be interested in moderators such as source characteristics (e.g.,
researcher characteristics) although the question of whether this helps
explain some heterogeneity may still be relevant. Card, (2015) provides a good
discussion of the different moderator types, which may provide useful guidance
in regards to identifying potential moderators of interest. Additional
information and advice on conducting meta-regression analyses is provided in
Borenstein et al., (2011); Baker et al., (2009); Higgins & Thompson, (2004);
Thompson & Higgins, (2002).

While meta-regression can explain some heterogeneity, it is often the case
that there is still residual heterogeneity remaining. Therefore, studying
heterogeneity following a meta-regression is still important and we believe
that a useful measure of heterogeneity to complement meta-regression is the
coefficient of variation (CV) (Takkouche et al.,, 1999, 2013). Here, the CV is
the the between-studies standard deviation divided by the absolute effect. A
small CV suggests that, while heterogeneity may be present, we do not expect
the effects to vary much relative to their magnitude. On the other hand, a
large CV tells us that the effects can be noticeably different, and even
contradictory to one another (e.g., a negative effect in some cases, and
positive in others). In this paper we consider using the CV in the context of
meta-regression. This can allow us to obtain a deeper understanding of
heterogeneity, and explore questions such as ‘at which geographical locations
is the vaccine effect likely to be most variable and therefore less reliable?’
and ‘what is the average heterogeneity in this meta-analysis?’.

The structure of the rest of this paper is as follows. In Section 2 we provide
background material on the meta-regression model, inverse variance weights
estimation and estimators of the heterogeneity variance. In Section 3 we
discuss the CV as a measure of heterogeneity. In Section 4 we consider mean CV
and associated variances before considering confidence intervals in Section 5.
Simulations are provided in Section 6 that illustrate very good coverage of
our preferred intervals and examples are also considered before the conclusion
in Section 7.

##  2 Meta-analysis and meta-regression analysis

In this section we briefly introduce background material on the meta-
regression model, inverse variance weights estimation and estimators of the
heterogeneity variance.

###  2.1 The models

Let $Y_{1},\ldots,Y_{k}$ be estimators of an effect for $k$ independent
studies. In the presence of zero heterogeneity, the fixed effect model (FEM)
is

$Y_{i}=\beta+\epsilon_{i},\;\;i=1,\ldots,k,$

where $\epsilon_{i}\sim N(0,v_{i})$ is the sampling error and $v_{i}$ is the
variance of the estimator $Y_{i}$ for the $i$-th study. The FEM is rarely
justified, since differences in studies (e.g., treatment dosage, population
demographics etc.) can reasonably be assumed to result in differences of the
effects to be estimated. These differences in the effects are simply referred
to as heterogeneity. To allow for this heterogeneity, we can define the random
effects model (REM) as

$Y_{i}=\beta+\gamma_{i}+\epsilon_{i},$ (1)

where $\gamma_{i}\sim N(0,\tau^{2})$ is the random effect that allows effects
to vary due to unexplained heterogeneity.

A simple meta-regression is modelled in a similar way to simple linear
regression. That is, we have an intercept ($\beta_{0}$), a slope coefficient
($\beta_{1}$) and $x_{1},\ldots,x_{k}$ are observed moderator values for the
$k$ studies. The simple REM meta-regression model is

$Y_{i}=\beta_{0}+\beta_{1}\cdot x_{i}+\gamma_{i}+\epsilon_{i},$ (2)

where $\beta_{0}=E(Y|x=0)$ is the mean effect for an arbitrary study when the
moderator is equal to zero. More generally, $E(Y|x)=\beta_{0}+\beta_{1}\cdot
x$ and given estimates of $\beta_{0}$ and $\beta_{1}$ we can estimate the mean
effects for given values of the moderator.

###  2.2 Inverse variance weight estimation

A pooled effect estimator of $\beta$ in the FEM and REM in the form of a
weighted average of the $Y_{1},\ldots,Y_{k}$ is

$\widehat{\beta}=w_{1}Y_{1}+\ldots+w_{k}Y_{k},$

where $w_{1},\ldots,w_{k}$ are weights that sum to one. Note that
$E(\widehat{\beta})=\beta$ where under the FEM, $\beta$ is the common effect
over all studies and the mean effect under the REM.

Let $W_{i}=1/\text{Var}(Y_{i})$, the inverse of the variance for the effect
estimator for the $i$-th study. The inverse variance weights (IVW) estimator
uses weights that minimises $\text{Var}(\widehat{\beta})$ and these weights
are given by $w_{i}=W_{i}/\sum^{k}_{i=1}W_{i}$. Under the REM in (1), we have
$\text{Var}(Y_{i})=v_{i}+\tau^{2}$ and so $W_{i}=1/(v_{i}+\tau^{2})$. In both
cases the IVW estimator and its variance are

$\widehat{\beta}=\frac{\sum^{k}_{i=1}W_{i}Y_{i}}{\sum^{k}_{i=1}W_{i}}=\sum^{k}_{i=1}w_{i}Y_{i},\;\;\text{Var}(\widehat{\beta})=\sum^{k}_{i=1}\frac{1}{W_{i}}.$
(3)

Let $\bm{\beta}=[\beta_{0},\beta_{1}]^{\top}$,
$\mathbf{W}=\text{diag}(W_{1},\ldots,W_{k})$ be a diagonal matrix of the un-
scaled weights, $\mathbf{X}$ be the matrix of moderators where the $i$-th row
is $[1,x_{i}]$ and $\mathbf{Y}=[Y_{1},\ldots,Y_{k}]$. Then the pooled
estimator of $\bm{\beta}$ for the meta-regression model is the weighted least
squares estimator where

$\widehat{\bm{\beta}}=\left[\begin{array}[]{c}\widehat{\beta}_{0}\\\
\widehat{\beta}_{1}\end{array}\right]=\left(\mathbf{X}^{\top}\mathbf{W}\mathbf{X}\right)^{-1}\mathbf{X}^{\top}\mathbf{W}\mathbf{Y},\;\text{Var}(\widehat{\bm{\beta}})=\left[\begin{array}[]{cc}\text{Var}(\widehat{\beta}_{0})&\text{Cov}(\widehat{\beta}_{0},\widehat{\beta}_{1})\\\
\text{Cov}(\widehat{\beta}_{0},\widehat{\beta}_{1})&\text{Var}(\widehat{\beta}_{1})\end{array}\right]=\left(\mathbf{X}^{\top}\mathbf{W}\mathbf{X}\right)^{-1}.$
(4)

Note that the weights for the REM depend on $\tau^{2}$, the variance of the
random effect. In practice this is unknown and needs to be estimated. Once
estimated, $\tau^{2}$ is replaced by its estimator $\widehat{\tau}^{2}$ in the
above formula. We next discuss some measures of $\tau^{2}$.

###  2.3 Estimators of heterogeneity variance

There are many estimators of $\tau^{2}$, including maximum-likelihood and
restricted maximum-likelihood approaches, as well as methods of moments and
unbiased methods. Here, we will describe two methods. Discussion of other
approaches can be found in Veroniki et al., (2016); Langan et al., (2019).

The most commonly used method is the DL measure proposed by DerSimonian &
Laird, (1986). Let $SF_{j}=\sum_{i=1}^{k}(W_{i,FE})^{j}$ where $W_{i,FE}$ is
the weight assigned to the $i$-th study of a fixed-effect (FE) meta-analysis.
The authors define

$\widehat{\tau}^{2}=\text{max}\left\\{0,\frac{Q-(k-1)}{SF_{1}}-\frac{SF_{2}}{SF_{1}}\right\\},$
(5)

where $Q=\sum W_{i,FE}(Y_{i}-\widehat{\beta}_{FE})^{2}$ and
$\widehat{\beta}_{FE}$ is the meta-estimate of the FE analysis. This method is
commonly referred to as the DerSimonian and Laird (DL) method.

A second estimator that we consider is the REML estimator. Veroniki et al.,
(2016); Langan et al., (2019) recommend the REML estimator as an estimator for
$\tau^{2}$, however it is not available in all analysis packages. Further
details about the REML estimator can be found in Sidik & Jonkman, (2005).
Throughout this paper we use the DL and REML estimators of $\tau^{2}$. It is
important to note however, that the measures will work with any estimator of
$\tau^{2}$.

##  3 The coefficient of variation

A very common measure of uncertainty in a meta-analysis is $I^{2}$ (Higgins &
Thompson,, 2002) defined to be

$I^{2}=\frac{\tau^{2}}{\tau^{2}+\sigma_{Y}^{2}},$

where $\sigma_{Y}^{2}$ is a typical within-studies variance. However, $I^{2}$
can be small even when heterogeneity is large (i.e. this can happen when a
typical sampling error variance is large) or large when heterogeneity is small
(such as when sampling errors are small). For more on this see Borenstein et
al., (2017). Hence, it is important that $I^{2}$ is interpreted correctly as a
measure of heterogeneity relative to total variability in the specific meta-
analysis.

A measure which may be very useful for quantifying heterogeneity is the
coefficient of variation (CV)(Takkouche et al.,, 1999, 2013) defined to be

$\text{CV}_{B}=\frac{\tau}{|\beta|}.$

A large CVB indicates that heterogeneity is large relative to the magnitude of
the effect. When this occurs the effects can even be contradictory, depending
on, e.g., cohorts differences such as age and gender, conditions under which a
treatment is implemented etc. On the other hand, a small CVB indicates that we
have a much more accurate understanding of the magnitude of the effect, even
when noting such differences. Hence, the coefficient of variation can be a
useful tool to help describe heterogeneity, perhaps as a complement to other
measures such as $I^{2}$ (Higgins & Thompson,, 2002). However, when $\beta$ is
very small, the CV can be huge and such cases there may be reluctance to use
it. Additionally, given the prominent use of $I^{2}$, that is on a simple-to-
understand scale, not everyone may be comfortable reporting CVB. Therefore,
Cairns & Prendergast, (2020) transformed the CV as

$M_{1}=\frac{\tau}{\tau+|\beta|}\text{ and
}M_{2}=\frac{\tau^{2}}{\tau^{2}+\beta^{2}}$

and provided several confidence intervals (CIs) that had good coverage
properties in a wide variety of scenarios. In that paper they discussed
several properties which we believe could be important for measures of
heterogeneity. Importantly, the paper noted that we can directly link
$\text{CV}_{B}$, $M_{1}$ and $M_{2}$ in the following ways:

$\log(\text{CV}_{B})=\text{logit}(M_{1})=\text{logit}(M_{2})/2,$ (6)

and

$M_{1}=\frac{\text{CV}_{B}}{1+\text{CV}_{B}}\;\;\text{and}\;\;M_{2}=\frac{\text{CV}^{2}_{B}}{1+\text{CV}^{2}_{B}}.$
(7)

##  4 CV estimation in the context of meta-regression

In this section we will briefly discuss the CV in the context of meta-
regression. For a given moderator value $x$, we define
$\beta_{x}=\beta_{0}+\beta_{1}x$ as the true value to be estimated. Note that
we can similarly consider more than one moderator variable, such as in
multiple linear regression, including factor moderators that require the $x$
(or multiple $x$-s) to be dummy variables indicating factor level. Then,
$\text{CV}_{B}=\frac{\tau}{|\beta_{x}|}$ and we can estimate the CV by

$\widehat{\text{CV}}_{B}=\frac{\widehat{\tau}}{|\widehat{\beta}_{x}|}.$ (8)

Note that if changes to a moderator can effect significant change to the mean
effect, then very different values for the CV may be result for different
$x$-s. For example, consider two scenarios:

Small $\widehat{\text{CV}}_{B}$

since we estimate that $\tau$ is small relative to $\beta_{x}$, then for this
$x$ we have a good understanding of the magnitude of the effect (e.g., we
estimate that the magnitude of $\beta_{x}$ will not vary much for studies
conducted for this value of $x$).

Large $\widehat{\text{CV}}_{B}$

since we estimate that $\tau$ is large relative to $\beta_{x}$, then for this
$x$ we have a poor understanding of the magnitude of the effect (e.g., we
estimate that the magnitude of $\beta_{x}$ will vary a lot for studies
conducted for this value of $x$).

Note that in the above we refer to the fact that we are only estimating
relative heterogeneity and our conclusions need to reflect that. Hence, to
provide some rigour we need CIs which we consider in the next section.
Variances for estimators of $\tau$ are readily available, and we can easily
obtain a variance for $\widehat{\beta}_{x}$ as

$\text{Var}(\widehat{\beta}_{x})=\text{Var}(\widehat{\beta}_{0}+x\widehat{\beta}_{1})=\text{Var}(\widehat{\beta}_{0})+x^{2}\cdot\text{Var}(\widehat{\beta}_{1})+2\cdot
x\cdot\text{Cov}(\widehat{\beta}_{0},\widehat{\beta}_{1}),$ (9)

where the variances and covariances are given in (4).

Takkouche et al., (2013) considered Wald CIs for the log of CVB. For the
estimator of $M_{1}$ (and $M_{2}$), Cairns & Prendergast, (2020) showed that
improved intervals were obtained when these were transformed using the logit
transformation and some different interval methods to be discussed later.
Hence, using their variance of the logit transformed estimation and also the
link between this and the log of the CV estimator shown in (6), we have

$\text{Var}\left[\text{log}(\widehat{\text{CV}}_{B})\right]=\text{Var}\left[\text{logit}(\widehat{M}_{1})\right]\approx\text{Var}(\widehat{\tau}^{2})\cdot\frac{1}{4\tau^{4}}+\text{Var}(\widehat{\beta_{x}})\cdot\frac{1}{\beta_{x}^{2}}.$

Note in the above we have considered a single fixed $x$, but we may wish to
explore different $x$’s. For example, if we assume that $\text{CV}_{B}$ has
been calculated for each level of a moderator, then one overall measure is an
average of these $\text{CV}_{B}$’s. We now consider several possibilities: a
weighted average of $\log$ values, a weighted average and a geometric mean.
For these averages we focus directly on the CVB, however the average of log
values can also be considered as the average logit of estimators of $M_{1}$.

Let $\bm{x}=[x_{1},\ldots,x_{r}]^{\top}$ be a vector of moderator values. This
can either be chosen by the researcher, or they may be the specific moderator
values observed for the $k$ studies (in which case $r=k$). Further, let
$\bm{\omega}=[\omega_{1},\ldots,\omega_{r}]^{\top}$ be weights that will be
assigned to each $x_{r}$ for the purpose of weighted average of the
heterogeneity measures computed at these moderator values. Finally, let
$\text{CV}_{B,i}$ be value of $\text{CV}_{B}$ calculated for each $x_{i}$.
Then the weighted average of $\log(\text{CV}_{B})$ estimates and the
corresponding geometric mean are of the form

$\widehat{\text{waLCV}}=\bm{\omega}^{\top}\cdot\log\left(\begin{bmatrix}\widehat{\text{CV}}_{B,1}\\\
\cdot\\\ \cdot\\\ \cdot\\\
\widehat{\text{CV}}_{B,r}\end{bmatrix}\right),\;\;\widehat{\text{GM}}=\exp(\widehat{\text{waLCV}}),$
(10)

where $\omega_{1}=\dots=\omega_{r}=1/r$. Finally, we define the weighted
average of CVB’s as

$\widehat{\text{waCV}}=\bm{\omega}^{\top}\cdot\begin{bmatrix}\widehat{\text{CV}}_{B,1}\\\
\cdot\\\ \cdot\\\ \cdot\\\ \widehat{\text{CV}}_{B,r}\end{bmatrix}.$ (11)

Let $\bm{\widehat{\beta}}_{X}$ be a column vector of estimated values of the
$\beta_{x}$’s corresponding to the $x_{i}$’s (e.g., the $i$-th element is the
estimated $\beta_{x}$ at $x_{i}$), $\bm{\widehat{\Sigma}}$ be the variance-
covariance matrix of $\bm{\widehat{\beta}}_{X}$,
$\text{sign}(\bm{\widehat{\beta}}_{X})$ be a column vector of corresponding
signs ($\pm 1$) of the elements of $\bm{\widehat{\beta}}_{X}$,
$\bm{\widehat{\beta}}^{\downarrow}_{X}$ be the vector where element
$\widehat{\beta}_{i}^{\downarrow}=1/\widehat{\beta}_{x_{i}}$ and
$|\bm{\widehat{\beta}}^{\downarrow}_{X}|$ be the vector comprised of elements
$|\widehat{\beta}_{x_{i}}^{\downarrow}|$. Then, we use the following notation:
we let $\odot$ be the Hadamard Product (in which matrices are multiplied
together element-wise), let $\bm{I}_{H}$ be the multiplicative identity of the
Hadamard product (i.e., a $r\times r$ matrix containing $1$’s in every entry)
and $\cdot$ be the usual matrix multiplication. We find the following theorem:

######  Theorem 4.1.

An approximate variance-covariance matrix for $r$ dependent log transformed
$\text{CV}_{B}$ estimators is

$\bm{V}_{vlcv}=\left[\frac{\text{Var}(\widehat{\tau}^{2})}{4\cdot\widehat{\tau}^{4}}\cdot\bm{I}_{H}\right]+\left[(\text{sign}(\bm{\widehat{\beta}}_{X})\cdot\text{sign}(\bm{\widehat{\beta}}_{X})^{\top})\odot(|\bm{\widehat{\beta}}^{\downarrow}_{X}|\cdot|\widehat{\bm{\beta}}^{\downarrow}_{X}|^{\top})\odot\bm{\widehat{\Sigma}}\right].$

Similarly, we can calculate a variance-covariance matrix for $r$ dependent
$\text{CV}_{B}$’s.

######  Theorem 4.2.

Let $\bm{\widehat{\beta}}^{\downarrow 2}$ be the column vector containing
elements $(\beta_{i}^{\downarrow})^{2}$. Then an approximate variance-
covariance matrix for $k$ dependent $\text{CV}_{B}$ estimators is

$\displaystyle\bm{V}_{vcv}$
$\displaystyle=\left[\left(\frac{\text{Var}(\widehat{\tau}^{2})}{4\cdot\widehat{\tau}^{2}}\cdot\bm{I}_{H}\right)\odot(|\bm{\widehat{\beta}}^{\downarrow}_{X}|\cdot|\bm{\widehat{\beta}}^{\downarrow}_{X}|^{\top})\right]$
$\displaystyle+\left[(\widehat{\tau}^{2}\cdot\bm{I}_{H})\odot(\text{sign}(\bm{\widehat{\beta}}_{X})\cdot\text{sign}(\bm{\widehat{\beta}}_{X})^{\top})\odot(\bm{\widehat{\beta}}^{\downarrow
2}_{X}\cdot\bm{\widehat{\beta}}_{X}^{{\downarrow
2}^{\top}})\odot\bm{\widehat{\Sigma}}\right].$

The proofs of Theorems 4.1 and 4.2 can be found in Appendix A and B
respectively.

When performing a factor moderator analysis, the method to choose weights is
very important. A factor moderator provides a vector (i.e., $\bm{x}$)
containing a value for each level of the moderator. A researcher could then
choose to use equal weights across each level of the vector, they could set
the weights equal to the proportion of times a value appears in the data set
or they could choose the weights based on knowledge of the research area in
practice (e.g., giving a small weight to a factor level that is known to
represent a very rare occurrence or type). Let us refer to the example seen in
the introduction. Looking at the allocation vector in the Colditz et al.,
(1994) data set, we see there are three levels: random, alternate and
systemic. Therefore, if the researcher chooses equal weights we would have
$\omega_{1}=\omega_{2}=\omega_{3}=\frac{1}{3}$. If the researcher chose to
define the weights as the proportion of the times the level appeared, they
would have $\omega_{1}\approx 0.538$, $\omega_{2}\approx 0.154$ and
$\omega_{3}=0.308$. Finally, the researcher may decide that a random
allocation is much more likely than what is represented in the data set, and
therefore increasing that weight while decreasing others (e.g., choosing
$\omega_{1}=0.7$, $\omega_{2}=0.1$ and $\omega_{3}=0.2$).

##  5 Confidence intervals

Cairns & Prendergast, (2020) proposed four methods of calculating CIs in the
context of meta-analysis. These methods naturally extend to the meta-
regression framework. Using the metafor (Viechtbauer,, 2010) package in R (R
Core Team,, 2020), we can readily calculate a CI for $\beta_{x}$. We provide
code for this in the OSF page found here. We use this method of calculating
intervals for $\beta_{x}$ in all substitution based methods of calculating
intervals. In Sections 5.1, 5.2 and 5.3 we briefly define the intervals we use
in this paper.

In the sections below we define CIs for $M_{1}$. CIs for CVB and $M_{2}$ can
be found using the transformations seen in Equations 6 and 7.

###  5.1 PropImp

Let $f(X,Y)$ be a function that is monotonic in both $X$ and $Y$, where
$(X_{z(\alpha)}^{L},X_{z(\alpha)}^{U})$ is a $(1-\alpha)\times 100\%$ CI for
$X$ and $z(\alpha)=\Phi^{-1}(1-\alpha/2)$ where $\Phi^{-1}$ is the standard
normal inverse distribution function; e.g., $\Phi^{-1}(0.975)\approx 1.96$.
The CI for $Y$, $(Y_{z(\alpha)}^{L},Y_{z(\alpha)}^{U})$, is similarly defined.
Then the PropImp interval (Newcombe,, 2011) for $M_{1}$ would have the form

$L=\min_{0\leq\theta_{1}\leq\pi/2}\frac{\widehat{\tau}_{z\sin(\theta_{1})}^{L}}{\widehat{\tau}_{z\sin(\theta_{1})}^{L}+|\widehat{\beta}_{z\cos(\theta_{1})}|^{U}},\
U=\max_{0\leq\theta_{2}\leq\pi/2}\frac{\widehat{\tau}_{z\sin(\theta_{2})}^{U}}{\widehat{\tau}_{z\sin(\theta_{2})}^{U}+|\widehat{\beta}_{z\cos(\theta_{2})}|^{L}},$

remembering that $\beta$ is referring to $\beta_{x}=\beta_{0}+\beta_{1}x$. The
full PropImp algorithm for the CV is discussed in Cairns & Prendergast,
(2020).

###  5.2 Alpha adjusted intervals

Similarly, let $[L_{\tau}(\alpha),U_{\tau}(\alpha)]$ and
$[L_{\beta_{x}}(\alpha),U_{\beta_{x}}(\alpha)]$ denote $(1-\alpha)\times 100$%
CIs for $\tau$ and $\beta_{x}$ respectively. Then when using the method
prescribed in Newcombe, (2011); Lloyd, (1990) for calculating CIs using CIs
for both variables, we see intervals for $M_{1}$ of the form

$\left[\frac{L_{\tau}(\alpha)}{L_{\tau}(\alpha)+U_{|\beta_{x}|}(\alpha)},\
\frac{U_{\tau}(\alpha)}{U_{\tau}(\alpha)+L_{|\beta_{x}|}(\alpha)}\right].$
(12)

This method of calculating CIs works by assigning $\alpha=0.8342$ to make
intervals less conservative. As such, we are using the same formula seen in
Equation 12 but are calculating a CI(0.8342, 0.8342).

###  5.3 Wald-type intervals

A Wald-type (WT) CI is a possible alternative to the substitution methods we
describe above. Here we consider 95% CIs and therefore choose
$z_{1-\alpha/2}=z_{0.975}=1.96$ as our pivot, however other values can be used
to achieve different levels of confidence. Therefore a WT interval for the
logit of $M_{1}$ in the context of meta-regression has the form

$\text{Logit}(\widehat{M_{1}})\pm
1.96\sqrt{\text{Var}\left[\text{Logit}(\widehat{M_{1}})\right]},$

with variance

$\text{Var}\left[\text{logit}(\widehat{M}_{1})\right]\approx\text{Var}(\widehat{\tau}^{2})\cdot\frac{1}{4\tau^{4}}+\text{Var}(\widehat{\beta_{x}})\cdot\frac{1}{\beta_{x}^{2}},$

where Var$(\widehat{\beta_{x}})$ is from (9). We then back transform to the
regular scale using the usual methods. Additionally, given the link between
the logit of $M_{1}$ and the log of CVB given in (6), the above is also a
confidence for $\log(\text{CV}_{B})$.

###  5.4 Confidence intervals for mean values

CIs of mean values may also be of interest. Here we consider CIs for the case
where we take a weighted or geometric mean of the estimated $\beta_{i}$’s.
These intervals are regular WT intervals of their respective values and the
required variance matrices can be found in Theorems 4.1 and 4.2. A weighted
average of $\log(\text{CV}_{B})$’s would have the interval

$\widehat{\text{waLCV}}\pm
1.96\sqrt{\bm{\omega}^{\top}\cdot\bm{V}_{vlcv}\cdot\bm{\omega}}.$ (13)

A CI for a geometric mean of $\text{CV}_{B}$’s has the form

$\text{exp}\left\\{\widehat{\text{waLCV}}\pm
1.96\sqrt{\bm{\omega}^{\top}\cdot\bm{V}_{vlcv}\cdot\bm{\omega}}\right\\},$
(14)

where we assume equal weights (i.e., $\omega_{1}=\ldots=\omega_{r}=1/r$). A CI
for a weighted average of $\text{CV}_{B}$’s is defined as follows:

$\widehat{\text{waCV}}\pm
1.96\sqrt{\bm{\omega}^{\top}\cdot\bm{V}_{vcv}\cdot\bm{\omega}},$ (15)

where $\bm{V}_{vcv}$ is the variance-covariance matrix defined in Theorem 4.2.

##  6 Simulations and examples

In this section we display the results of several simulation studies. These
studies were performed using the metafor package (Viechtbauer,, 2010) in R (R
Core Team,, 2020). We simulate our data based on the settings from real data
sets. We randomly generate our $Y_{i}$’s from a normal distribution with mean
$\beta_{0}+\beta_{1}x_{i}$ and variance $\tau^{2}+v_{i}$ (remembering that
$v_{i}$ is the within-study variance for the $i$-th study). We will now
briefly describe the data sets used to create our simulations. The Studies of
the Effectiveness of the BCG Vaccine Against Tuberculosis (BCG) data set
(Colditz et al.,, 1994) contains 13 studies. We performed simulations using
the numeric moderator absolute latitude and a second set of simulations using
the factor moderator allocation. The next data set, also containing 13
studies, was the Standard Care Quality (SCQ) and HAART-adherence (HAART) data
set (de Bruin et al.,, 2009). For simulations based on this data set we used
the standard quality of care as a moderating variable. We also performed
simulations using the factor moderator ethnicity as a moderating variable. Due
to the size of the data sets we do not describe them fully here. However they
can be viewed using the code provided on our OSF page.

###  6.1 Simulation results

We will now discuss the results of the above simulations.

####  6.1.1 Numeric moderator

We begin by providing the results of the simulations using numeric moderators.
For simplicity, when computing intervals for a specified moderator value, we
have set the moderator values equal to those observed. However, it is of
course possible to choose any value for the moderators.

Table 1: Coverages calculated using 10,000 trials with moderator values chosen
to be the absolute latitude values found in the Colditz et al., (1994) data
set.

Mods | 13 | 18 | 19 | 27 | 33 | 42 | 44 | 52 | 55
---|---|---|---|---|---|---|---|---|---
$\alpha_{adj}$ | 0.937 | 0.952 | 0.951 | 0.937 | 0.922 | 0.914 | 0.914 | 0.916 | 0.917
$\alpha_{adj}$ [REML] | 0.936 | 0.957 | 0.956 | 0.946 | 0.931 | 0.923 | 0.922 | 0.926 | 0.927
Prop | 0.970 | 0.977 | 0.977 | 0.972 | 0.967 | 0.964 | 0.964 | 0.965 | 0.966
Prop [REML] | 0.971 | 0.980 | 0.980 | 0.977 | 0.973 | 0.970 | 0.969 | 0.970 | 0.970
WT | 0.939 | 0.987 | 0.990 | 0.999 | 0.997 | 0.995 | 0.995 | 0.995 | 0.995
WT [REML] | 0.921 | 0.978 | 0.982 | 0.997 | 0.992 | 0.987 | 0.987 | 0.988 | 0.989

In Table 1 we see coverages for four types of CI for $M_{1}$ for simulations
with absolute latitude as a moderating variable based on the Colditz et al.,
(1994) example. In general, coverage is strong for each method and typically
slightly more conservative when using the REML estimator of $\tau$. We see
that coverage for the $\alpha$-adjusted intervals appears to be slightly
liberal. The PropImp intervals provide strong coverage for each value of the
moderator and the intervals based on the REML estimator tend to be slightly
more conservative than the intervals calculated with the DL estimator. This is
potentially due to the negative bias shown in the DL estimator when $\tau^{2}$
is large (Veroniki et al.,, 2016).

Table 2: Average widths calculated using 10,000 trials with absolute latitude
as a moderating variable. Note that the widths for CVB based intervals and WT
intervals are calculated using medians.

Mods | 13 | 18 | 19 | 27 | 33 | 42 | 44 | 52 | 55
---|---|---|---|---|---|---|---|---|---
$\alpha_{adj}$ ($\text{CV}_{B}$) | $>1000$ | 7.328 | 4.621 | 1.098 | 0.703 | 0.486 | 0.460 | 0.384 | 0.363
$\alpha_{adj}$ [REML] ($\text{CV}_{B}$) | $>1000$ | 8.840 | 5.318 | 1.161 | 0.741 | 0.512 | 0.485 | 0.406 | 0.384
$\alpha_{adj}$ ($M_{1}$) | 0.735 | 0.651 | 0.625 | 0.417 | 0.336 | 0.279 | 0.271 | 0.247 | 0.240
$\alpha_{adj}$ [REML] ($M_{1}$) | 0.718 | 0.642 | 0.619 | 0.418 | 0.338 | 0.283 | 0.276 | 0.254 | 0.247
Prop ($\text{CV}_{B}$) | $>1000$ | $>1000$ | 18.822 | 1.406 | 0.852 | 0.591 | 0.559 | 0.465 | 0.440
Prop [REML] ($\text{CV}_{B}$) | $>1000$ | $>1000$ | 43.359 | 1.512 | 0.902 | 0.620 | 0.587 | 0.493 | 0.468
Prop ($M_{1}$) | 0.814 | 0.754 | 0.731 | 0.496 | 0.397 | 0.332 | 0.323 | 0.295 | 0.287
Prop [REML] ($M_{1}$) | 0.798 | 0.743 | 0.723 | 0.499 | 0.399 | 0.335 | 0.327 | 0.302 | 0.295
WT (CVB) | 0.874 | 0.649 | 0.608 | 0.420 | 0.359 | 0.309 | 0.300 | 0.271 | 0.261
WT [REML] (CVB) | 0.860 | 0.634 | 0.591 | 0.393 | 0.332 | 0.283 | 0.276 | 0.251 | 0.243
WT ($M_{1}$) | 0.874 | 0.649 | 0.608 | 0.420 | 0.359 | 0.309 | 0.300 | 0.271 | 0.261
WT [REML] ($M_{1}$) | 0.860 | 0.634 | 0.591 | 0.393 | 0.332 | 0.283 | 0.276 | 0.251 | 0.243

In Table 2 we show the average widths corresponding to the coverages seen in
Table 1. We show results for CVB, $M_{1}$ and $M_{2}$. We note that
particularly for small $x_{i}$-s the CVB intervals are very wide. However,
this is because the CVB values are all extremely large, e.g., from 10000
trials the average $\widehat{\text{CV}}_{B}\approx 13$ when $x=13$ and in this
situation estimates of the transformed versions $M_{1}$ and $M_{2}$ may be
preferred.

Table 3: Coverages calculated using 10,000 trials with moderator values chosen
to be the standard quality of care values found in the de Bruin et al., (2009)
data set.

Mods | 3.67 | 4.80 | 7.47 | 7.80 | 8.60 | 10.47 | 12.07 | 15.07 | 20.00 | 21.00 | 22.67 | 22.80 | 27.80
---|---|---|---|---|---|---|---|---|---|---|---|---|---
$\alpha_{adj}$ | 0.951 | 0.950 | 0.944 | 0.943 | 0.941 | 0.937 | 0.933 | 0.928 | 0.934 | 0.935 | 0.937 | 0.937 | 0.942
$\alpha_{adj}$ [REML] | 0.953 | 0.953 | 0.948 | 0.947 | 0.946 | 0.943 | 0.939 | 0.932 | 0.938 | 0.940 | 0.942 | 0.942 | 0.946
Prop | 0.974 | 0.973 | 0.973 | 0.972 | 0.972 | 0.973 | 0.973 | 0.971 | 0.972 | 0.972 | 0.973 | 0.973 | 0.972
Prop [REML] | 0.973 | 0.973 | 0.974 | 0.974 | 0.973 | 0.974 | 0.975 | 0.974 | 0.974 | 0.974 | 0.974 | 0.974 | 0.973
WT | 0.966 | 0.967 | 0.969 | 0.969 | 0.969 | 0.969 | 0.969 | 0.968 | 0.968 | 0.968 | 0.968 | 0.968 | 0.966
WT [REML] | 0.961 | 0.961 | 0.962 | 0.963 | 0.964 | 0.965 | 0.964 | 0.963 | 0.964 | 0.964 | 0.963 | 0.963 | 0.961

Table 4: Average widths calculated from 10,000 trials using SCQ as a
moderator. The data was simulated from the de Bruin et al., (2009) data set.
Note that the widths for CVB based intervals and WT intervals are calculated
using medians.

Mods | 3.67 | 4.80 | 7.47 | 7.80 | 8.60 | 10.47 | 12.07 | 15.07 | 20.00 | 21.00 | 22.67 | 22.80 | 27.80
---|---|---|---|---|---|---|---|---|---|---|---|---|---
$\alpha_{adj}$ ($\text{CV}_{B}$) | 0.464 | 0.433 | 0.375 | 0.369 | 0.356 | 0.330 | 0.313 | 0.293 | 0.285 | 0.286 | 0.287 | 0.287 | 0.294
$\alpha_{adj}$ [REML] ($\text{CV}_{B}$) | 0.510 | 0.476 | 0.410 | 0.403 | 0.388 | 0.358 | 0.339 | 0.317 | 0.310 | 0.311 | 0.314 | 0.314 | 0.326
$\alpha_{adj}$ ($M_{1}$) | 0.241 | 0.229 | 0.205 | 0.203 | 0.197 | 0.186 | 0.178 | 0.171 | 0.171 | 0.173 | 0.175 | 0.176 | 0.186
$\alpha_{adj}$ [REML] ($M_{1}$) | 0.253 | 0.240 | 0.214 | 0.212 | 0.205 | 0.193 | 0.186 | 0.178 | 0.180 | 0.181 | 0.185 | 0.185 | 0.198
Prop ($\text{CV}_{B}$) | 0.582 | 0.534 | 0.448 | 0.439 | 0.420 | 0.386 | 0.367 | 0.346 | 0.333 | 0.334 | 0.337 | 0.337 | 0.357
Prop [REML] ($\text{CV}_{B}$) | 0.649 | 0.595 | 0.493 | 0.483 | 0.460 | 0.419 | 0.394 | 0.368 | 0.363 | 0.365 | 0.372 | 0.373 | 0.400
Prop ($M_{1}$) | 0.292 | 0.274 | 0.241 | 0.238 | 0.230 | 0.216 | 0.208 | 0.200 | 0.200 | 0.202 | 0.206 | 0.206 | 0.223
Prop [REML] ($M_{1}$) | 0.309 | 0.289 | 0.252 | 0.249 | 0.240 | 0.225 | 0.215 | 0.206 | 0.209 | 0.211 | 0.217 | 0.217 | 0.238
WT (CVB) | 0.246 | 0.239 | 0.224 | 0.222 | 0.219 | 0.212 | 0.207 | 0.200 | 0.195 | 0.194 | 0.194 | 0.194 | 0.194
WT [REML] (CVB) | 0.236 | 0.227 | 0.210 | 0.209 | 0.205 | 0.197 | 0.192 | 0.186 | 0.182 | 0.183 | 0.183 | 0.183 | 0.186
WT ($M_{1}$) | 0.246 | 0.239 | 0.224 | 0.222 | 0.219 | 0.212 | 0.207 | 0.200 | 0.195 | 0.194 | 0.194 | 0.194 | 0.194
WT [REML] ($M_{1}$) | 0.236 | 0.227 | 0.210 | 0.209 | 0.205 | 0.197 | 0.192 | 0.186 | 0.182 | 0.183 | 0.183 | 0.183 | 0.186

In Table 3 we again see typically good coverage when the settings from de
Bruin et al., (2009) are used. While the $\alpha$-adjusted coverages are too
liberal with the DL, coverage is again slightly more conservative using the
REML estimator. We consider the average widths of the intervals in Table 4. We
see that the size of the CIs can vary with average widths appearing quite
small across the different methods in comparison to the widths seen in Table
2. This is because the estimates to the CVB-s are typically smaller with less
relative heterogeneity. Again we see that the $\alpha$-adjusted intervals are
the narrowest. However, narrow interval width comes at the expense of the
intervals being too liberal to slightly liberal as we saw in Table 1, in
particular when using the DL estimator. That said, the slightly liberal
coverage when using the REML estimator and narrow CIs than the PropImp
approach means that the $\alpha$-adjusted intervals with the REML estimator
have performed reasonably well. The slightly wider PropImp intervals align
with the slightly conservative coverage seen in Table 3.

Figure 1: These plots display $M_{1}$ against absolute latitude (top-left),
$\text{CV}_{B}$ against absolute latitude (bottom-left) as well as several
associated CIs and $\widehat{\beta}_{x}$ against absolute latitude (top-
right). These plots highlight the advantage of transforming the CV measures to
the $[0,1]$ scale when the effect ($\beta_{x}$) is close to zero.

In general, we see that the widths of the REML intervals are slightly wider
than the DL based intervals but this also corresponds to closer-to-nominal
interval estimator. Hence, when considering both coverage and width, we
recommend using the REML estimator. We also saw that in some examples, huge
widths of the intervals were often obtained for CVB estimation and we noted
that this was because of large relative heterogeneity leading to an extremely
large upper bound. Hence, this is one reason why intervals for $M_{1}$ and
$M_{2}$ may be preferred. To gain a better understanding of this, we see in
the top-left plot of Figure 1 a plot of the $M_{1}$ estimate and its
$\alpha_{adj}$, PropImp and WT intervals using the DL estimator for $\tau$
(similar results for the REML estimator). In this example, we used the Colditz
et al., (1994) data set but defined the moderating variable to be integer
values between 10 and 60 inclusive. The WT intervals are the widest
(particularly for smaller moderator values) followed by the PropImp intervals,
with the $\alpha_{adj}$ intervals providing the shortest intervals. This is
what we would expect given our coverages seen in Table 1. The issues with
$\text{CV}_{B}$ being unbounded become obvious in the bottom-left plot in
Figure 1 where we can see the upper bounds of the CIs are very large. In the
top-right plot of Figure 1 we see the observed estimated effects against
absolute latitude. This helps us highlight the issues with the large values of
CVB for small values of $\widehat{\beta}_{x}$. When the estimated effects are
close to zero, as is the case for small moderator values, the CVB measure will
become very large and the interval upper bound even larger. Hence, we can
still portray very large relative heterogeneity using $M_{1}$ (e.g., a large
estimate and upper interval bound of one), without the nuisance of reporting
extremely large (even infinite) values for CVB. This is perhaps less
problematic for factor moderators which are less likely to have a viable
moderator value that results in an estimated effect approximately equal to
zero, due to the non-continuous nature of the regression model.

####  6.1.2 Factor moderator

In Table 5 we display the results of simulations for $\text{CV}_{B}$’s as
described above. Note that these coverages will be the same for $M_{1}$ and
$M_{2}$, however the width will differ.

Table 5: Coverages and median widths calculated using 10,000 trials with
several factor moderators using CVB’s for both DL and REML estimators of
$\tau^{2}$. We show single level data for the two data sets Colditz et al.,
(1994); de Bruin et al., (2009).
Data | Level | WT | $\alpha_{adj}$ | PropImp
---|---|---|---|---
| Alternate | 0.899 | 0.935 | 0.969
| Alternate [REML] | 0.911 | 0.944 | 0.971
Colditz et al., (1994) | Random | 0.950 | 0.949 | 0.975
| Random [REML] | 0.963 | 0.951 | 0.975
| Systematic | 0.905 | 0.947 | 0.974
| Systematic [REML] | 0.921 | 0.952 | 0.978
| Other | 0.963 | 0.941 | 0.972
| Other [REML] | 0.959 | 0.945 | 0.972
de Bruin et al., (2009) | Caucasian | 0.965 | 0.934 | 0.970
| Caucasian [REML] | 0.961 | 0.938 | 0.972

Table 5 results show generally good coverage though the WT intervals can be
quite liberal (e.g., 0.899).

####  6.1.3 Geometric and weighted means

In this section we consider the results of simulations performed for geometric
and weighted means. The $\alpha$-adjusted and PropImp methods no longer have
only two substitutions to perform (i.e. the interval for $\tau$ and the
interval for the effect) but $r+1$ interval substitutions (one for $\tau$ and
then one for each of the moderator values chosen to form the average of the
effects where $r=k$ if the observed moderators are chosen). Hence, we consider
the more straightforward WT intervals based on the variance matrices presented
in Theorems 4.2 and 4.1.

We consider results for two factor moderators (allocation and ethnicity) and
two numeric moderators (absolute latitude and standard quality of care). In
these simulations we found that using the REML estimator of $\tau^{2}$
performed best. We performed simulations for the numeric moderators, based on
the initial data before replicating the data set once, twice and three times
(to increase the number of studies present in the analysis). For example, the
Ablat results seen in Table 6 contains 13 studies, we replicate the data once
to run simulations with $k=26$, twice for simulations with $k=39$ studies and
three times for simulations with $k=52$ studies.

Table 6: Coverages calculated using 10,000 trials with various moderating
variables using CVB’s for both DL and REML estimators of $\tau^{2}$. CIs were
calculated using the WT intervals. We show single level data for the two data
sets Colditz et al., (1994); de Bruin et al., (2009). In this example we chose
20 values from between the minimum of the corresponding moderator to the
maximum (not including zero). Here, $k$ represents the number of studies
included in the meta-regression (by adding replications of the original data
set).
Data | Moderator | waCV | waLCV | GM
---|---|---|---|---
| Allocation | 0.920 | 0.963 | 0.956
| Allocation [REML] | 0.933 | 0.970 | 0.965
| Ablat ($k=13$) | 0.947 | 0.998 | 0.998
| Ablat[REML]($k=13$) | 0.909 | 0.996 | 0.996
Colditz et al., (1994) | Ablat ($k=26$) | 0.926 | 0.995 | 0.995
| Ablat[REML]($k=26$) | 0.908 | 0.990 | 0.990
| Ablat ($k=39$) | 0.922 | 0.995 | 0.995
| Ablat[REML]($k=39$) | 0.914 | 0.977 | 0.977
| Ablat ($k=52$) | 0.926 | 0.989 | 0.989
| Ablat[REML]($k=52$) | 0.919 | 0.968 | 0.968
| Ethnicity | 0.910 | 0.965 | 0.965
| Ethnicity [REML] | 0.920 | 0.961 | 0.961
| SCQ ($k=13$) | 0.910 | 0.965 | 0.965
| SCQ [REML] ($k=13$) | 0.920 | 0.961 | 0.961
de Bruin et al., (2009) | SCQ ($k=26$) | 0.931 | 0.956 | 0.956
| SCQ [REML] ($k=26$) | 0.941 | 0.955 | 0.955
| SCQ ($k=39$) | 0.939 | 0.957 | 0.957
| SCQ [REML] ($k=39$) | 0.946 | 0.956 | 0.956
| SCQ ($k=52$) | 0.944 | 0.955 | 0.955
| SCQ [REML] ($k=52$) | 0.951 | 0.957 | 0.957

In Table 6 we consider the results of several simulations. Note that the
coverage of $waLCV$ and $GM$ for the numeric moderators are the same because
the weights are equal in these simulations. We begin by showing the coverage
for a geometric mean and a weighted average for two different factor
simulations (described above). We see that coverage can be quite liberal for
the $waCV$ intervals though log-transformed intervals and geometric means have
good coverage. Continuing on, we display the results for two sets of
continuous numeric moderators. Looking at the Colditz et al., (1994) results,
the $waCV$ intervals show liberal coverage which increases with the number of
studies. For the $waLCV$ and $GM$ intervals, coverage tends to begin
conservatively and become closer to the nominal as more studies are added. The
coverages for the de Bruin et al., (2009) data set are generally good with
$waCV$ intervals tending to be liberal.

###  6.2 Examples

In this section we will give the reader an example of how these measures may
be used in practice. First, consider the data set seen in Colditz et al.,
(1994). Remember that this data set contains 13 studies and we use absolute
latitude as a moderating variable (see Section 4 for more detailed
information). Performing a meta-regression, we see the following:
$\widehat{\beta}_{0}=0.260$, $\widehat{\beta}_{1}=-0.029$ and
$\widehat{\tau}^{2}=0.063$. A quick calculation provides the values of CVB,
$M_{1}$ and $M_{2}$, for each value of absolute latitude. We see these results
in Table 7.

Table 7: Contains values for heterogeneity measures when using absolute
latitude as a moderating variable. These are examples of output a researcher
might receive when using these measures in practice.

Mods | 13 | 18 | 19 | 27 | 33 | 42 | 44 | 52 | 55
---|---|---|---|---|---|---|---|---|---
$\beta_{x}$ | -0.120 | -0.120 | -1.027 | -0.296 | -0.530 | -0.968 | -1.027 | -1.260 | -1.348
CVB | 2.089 | 0.944 | 0.851 | 0.475 | 0.357 | 0.260 | 0.245 | 0.200 | 0.187
CVB [REML] | 2.178 | 1.015 | 0.917 | 0.517 | 0.390 | 0.285 | 0.269 | 0.219 | 0.205
$M_{1}$ | 0.676 | 0.486 | 0.460 | 0.322 | 0.263 | 0.206 | 0.197 | 0.166 | 0.157
$M_{1}$ [REML] | 0.685 | 0.504 | 0.478 | 0.341 | 0.280 | 0.222 | 0.212 | 0.180 | 0.170

  * •

CIs using DL for specific moderators using PropImp intervals and a CI for the
weighted average:

  * •

$13:$ CVB $=[0.445,>>1000]$ and $M_{1}=[0.308,1.000]$.

  * •

$55:$ CVB $=[0.094,0.735]$ and $M_{1}=[0.086,0.424]$.

  * •

$waCV=0.742[0.000,1.863]$.

These results tell us that heterogeneity relative to the size of the effect is
larger for smaller values of absolute latitude. Hence, effects measured at
larger latitudes (e.g., 55) would be relatively stable in terms of magnitude.
However, for lower latitude values such as within the tropics, the magnitude
of the effect may vary considerably to the extent that small, moderate, or
large effects may be observed. Hence, while the effects themselves may vary
similarly at any given latitude, given that we are interested in magnitude of
the effects then we need to consider heterogeneity relative to the size of the
effect. These results indicate to the researcher that results of the usage of
this vaccine near the equator could vary considerably. In addition, further
research into the causes (and subsequent fixes) of the variation in the
effectiveness of the vaccine near the equator could be considered.

##  7 Conclusion

We consider the use of the CV in meta-analysis in the context of meta-
regression. We have used the CV for fixed values of the moderator, as well as
means of the CV over several moderator values. We also considered transformed
values of the CV and associated CIs as was considered by Cairns & Prendergast,
(2020). It is of interest to be able to assess how much adding moderators
affects the amount of explained heterogeneity of a meta-analysis relative to
the size of the effect. This is particularly useful when such an investigation
is used in conjunction with a consideration of how much that moderator affects
the uncertainty of the meta-analysis (i.e., a discussion of $I^{2}$) creating
strong hypothesis generating discussions. CIs for $\text{CV}_{B}$, $M_{1}$ and
$M_{2}$ show strong coverage. Simulations show that PropImp intervals for CVB,
$M_{1}$ and $M_{2}$ provide reliable, slightly conservative intervals in all
cases. Thus, we would recommend that in addition to $I^{2}$, either CVB and
$M_{1}$ should be reported, along with corresponding CIs.

All data and code used in this paper are available at https://osf.io/f68za/

## Appendix

### A.1. Proof of Theorem 4.1

We will prove Theorem 4.1 by calculating the $ij$-th entry. Let
$\log(\widehat{\text{CV}}_{i})=\log(\widehat{\tau})-\log(|\widehat{\beta}_{i}|)$
and
$\log(\widehat{\text{CV}}_{j})=\log(\widehat{\tau})-\log(|\widehat{\beta}_{j}|)$
and $\widehat{\tau}=\sqrt{\widehat{\tau}^{2}}$ (i.e. so that we consider
$\widehat{\text{CV}}_{j}$ a function of $\widehat{\tau}^{2}$ and write
$\widehat{\text{CV}}_{j}=\sqrt{\widehat{\tau}^{2}}/|\widehat{\beta}_{j}|$).
Noting that in general $d|x|/(dx)=\text{sign}(x)$, then a first order Taylor
approximation has the form

$\log(\widehat{\text{CV}})_{i}\approx\log(\sqrt{\tau^{2}})-\log(|\beta_{i}|)+(\widehat{\tau}^{2}-\tau^{2})\cdot\frac{1}{2\cdot\tau^{2}}-(\widehat{\beta}_{i}-\beta_{i})\cdot\frac{1}{|\beta_{i}|}\cdot\text{
sign}(\beta_{i}),$

where an approximate $\log(\text{CV}_{j})$ can be found in the same way.
Therefore the $ij$-th element of the variance-covariance matrix for two
dependent $\log(\text{CV}_{B})$’s has the form

$\displaystyle\text{Cov}\left[\log(\sqrt{\tau^{2}})\right.$
$\displaystyle-\log(|\beta_{i}|)+(\widehat{\tau}^{2}-\tau^{2})\cdot\frac{1}{2\cdot\tau^{2}}-(\widehat{\beta}_{i}-\beta_{i})\cdot\frac{1}{|\beta_{i}|}\cdot\text{
sign}(\beta_{i}),$
$\displaystyle\left.\log(\sqrt{\tau^{2}})-\log(|\beta_{j}|)+(\widehat{\tau}^{2}-\tau^{2})\cdot\frac{1}{2\cdot\tau^{2}}-(\widehat{\beta}_{j}-\beta_{j})\cdot\frac{1}{|\beta_{j}|}\cdot\text{
sign}(\beta_{j})\right].$

Using the fact that
$\text{Cov}(aW+bX,cY+dZ)=\text{Cov}(aW,cY)+\text{Cov}(aW,dZ)+\text{Cov}(bX,cY)+\text{Cov}(bX,dZ)$,
we see that

$\displaystyle\text{Cov}\left[\right.$
$\displaystyle\log(\widehat{CV}_{i}),\left.\log(\widehat{CV}_{j})\right]\approx\
\text{Cov}\left[(\widehat{\tau}^{2}-\tau^{2})\cdot\frac{1}{2\cdot\tau^{2}},(\widehat{\tau}^{2}-\tau^{2})\cdot\frac{1}{2\cdot\tau^{2}}\right]$
$\displaystyle+\text{Cov}\left[(\widehat{\tau}^{2}-\tau^{2})\cdot\frac{1}{2\cdot\tau^{2}},-(\widehat{\beta}_{j}-\beta_{j})\cdot\frac{1}{|\beta_{j}|}\cdot\text{
sign}(\beta_{j})\right]$
$\displaystyle+\text{Cov}\left[-(\widehat{\beta}_{i}-\beta_{i})\cdot\frac{1}{|\beta_{i}|}\cdot\text{
sign}(\beta_{i}),(\widehat{\tau}^{2}-\tau^{2})\cdot\frac{1}{2\cdot\tau^{2}}\right]$
$\displaystyle+\text{Cov}\left[-(\widehat{\beta}_{i}-\beta_{i})\cdot\frac{1}{|\beta_{i}|}\cdot\text{
sign}(\beta_{i}),-(\widehat{\beta}_{j}-\beta_{j})\cdot\frac{1}{|\beta_{j}|}\cdot\text{
sign}(\beta_{j})\right].$

Since $\widehat{\tau}$ and $\widehat{\beta}$ are independent so that the
second and third terms above are zero, and since
E$[\widehat{\tau}^{2}]\approx\tau^{2}$ and E$[\widehat{\beta}]\approx\beta$,
then by the definition of the covariance function

$\displaystyle\text{Cov}[$
$\displaystyle\log(\widehat{\text{CV}}_{i}),\log(\widehat{\text{CV}}_{j})]\approx\
\text{Cov}\left[(\widehat{\tau}^{2}-\tau^{2})\cdot\frac{1}{2\cdot\tau^{2}},(\widehat{\tau}^{2}-\tau^{2})\cdot\frac{1}{2\cdot\tau^{2}}\right]$
$\displaystyle+\text{Cov}\left[-(\widehat{\beta}_{i}-\beta_{i})\cdot\frac{1}{|\beta_{i}|}\cdot\text{
sign}(\beta_{i}),-(\widehat{\beta}_{j}-\beta_{j})\cdot\frac{1}{|\beta_{j}|}\cdot\text{
sign}(\beta_{j})\right]$
$\displaystyle=E\left[\frac{(\tau^{2}-\widehat{\tau}^{2})^{2}}{4\cdot(\tau^{2})^{2}}\right]+E\left[\frac{1}{|\beta_{i}|\cdot|\beta_{j}|}\cdot\text{
sign}(\beta_{i})\text{
sign}(\beta_{j})(\widehat{\beta_{i}}-\beta)(\widehat{\beta_{j}}-\beta)\right]$
$\displaystyle=\frac{\text{Var}(\widehat{\tau}^{2})}{4\cdot\tau^{4}}+\frac{1}{|\beta_{i}|\cdot|\beta_{j}|}\cdot\text{
sign}(\beta_{i})\text{
sign}(\beta_{j})\cdot\text{Cov}(\widehat{\beta}_{i},\widehat{\beta}_{j}).$

Thus a variance-covariance matrix for $k$ dependent
$\log(\widehat{\text{CV}}_{B})$’s is the matrix whose $ij$-th element is

$\text{Cov}[\log(\widehat{\text{CV}}_{i}),\log(\widehat{\text{CV}}_{j})].$

By introducing the notations just prior to Theorem 4.1, it is simple to
represent this in matrix form.

### A.2. Proof of Theorem 4.2

Recall the methods used in Appendix A. We will prove Theorem 4.2 by
calculating the $ij$-th entry. Let
$\widehat{\text{CV}}_{i}=\frac{\widehat{\tau}}{|\widehat{\beta}_{i}|}$ and
$\widehat{\text{CV}}_{j}=\frac{\widehat{\tau}}{|\widehat{\beta}_{j}|}$ and
$\widehat{\tau}=\sqrt{\widehat{\tau}^{2}}$ (i.e., we consider
$\widehat{\text{CV}}_{j}$ a function of $\widehat{\tau}^{2}$ and write
$\widehat{\text{CV}}_{j}=\sqrt{\widehat{\tau}^{2}}/|\widehat{\beta}_{j}|$).
Noting that in general $d|x|/(dx)=\text{sign}(x)$, then a first order Taylor
approximation has the form

$\widehat{\text{CV}}_{i}\approx\frac{\sqrt{\tau^{2}}}{|\beta_{i}|}+\frac{\widehat{\tau}^{2}-\tau^{2}}{|\beta_{i}|}\cdot\frac{1}{2\sqrt{\tau^{2}}}-(\widehat{\beta}_{i}-\beta_{i})\cdot\frac{\sqrt{\tau^{2}}}{\beta_{i}^{2}}\cdot\text{
sign}(\beta_{i}),$

where an approximate $\text{CV}_{j}$ can be found in the same way. Therefore
the $ij$-th element of the variance-covariance matrix for two dependent
$\text{CV}_{B}$’s has the form

$\displaystyle\text{Cov}\left[\frac{\sqrt{\tau}^{2}}{|\beta_{i}|}\right.$
$\displaystyle+\frac{\widehat{\tau}^{2}-\tau^{2}}{|\beta_{i}|}\cdot\frac{1}{2\sqrt{\tau^{2}}}-(\widehat{\beta}_{i}-\beta_{i})\cdot\frac{\sqrt{\tau^{2}}}{\beta_{i}^{2}}\cdot\text{
sign}(\beta_{i}),$
$\displaystyle\left.\frac{\sqrt{\tau^{2}}}{|\beta_{j}|}+\frac{\widehat{\tau}^{2}-\tau^{2}}{|\beta_{j}|}\cdot\frac{1}{2\sqrt{\tau^{2}}}-(\widehat{\beta}_{j}-\beta_{j})\cdot\frac{\sqrt{\tau^{2}}}{\beta_{j}^{2}}\cdot\text{
sign}(\beta_{j})\right].$

Using the fact that
$\text{Cov}(aW+bX,cY+dZ)=\text{Cov}(aW,cY)+\text{Cov}(aW,dZ)+\text{Cov}(bX,cY)+\text{Cov}(bX,dZ)$,
we see that

$\displaystyle\text{Cov}($
$\displaystyle\widehat{\text{CV}}_{i},\widehat{\text{CV}}_{j})\approx\
\text{Cov}\left[\frac{\widehat{\tau}^{2}-\tau^{2}}{|\beta_{i}|}\right.\cdot\frac{1}{2\sqrt{\tau^{2}}}-(\widehat{\beta}_{i}-\beta_{i})\cdot\frac{\sqrt{\tau^{2}}}{\beta_{i}^{2}}\cdot\text{
sign}(\beta_{i}),$
$\displaystyle\qquad\qquad\qquad\qquad\quad\left.\frac{\widehat{\tau}^{2}-\tau^{2}}{|\beta_{j}|}\cdot\frac{1}{2\sqrt{\tau^{2}}}-(\widehat{\beta}_{j}-\beta_{j})\cdot\frac{\sqrt{\tau^{2}}}{\beta_{j}^{2}}\cdot\text{
sign}(\beta_{j})\right]$
$\displaystyle=\text{Cov}\left[\frac{\widehat{\tau}^{2}-\tau^{2}}{|\beta_{i}|}\cdot\frac{1}{2\sqrt{\tau^{2}}},\frac{\widehat{\tau}^{2}-\tau^{2}}{|\beta_{j}|}\cdot\frac{1}{2\sqrt{\tau^{2}}}\right]$
$\displaystyle+\text{Cov}\left[\frac{\widehat{\tau}^{2}-\tau^{2}}{|\beta_{i}|}\cdot\frac{1}{2\sqrt{\tau^{2}}},-(\widehat{\beta}_{j}-\beta_{j})\cdot\frac{\sqrt{\tau^{2}}}{\beta_{j}^{2}}\cdot\text{
sign}(\beta_{j})\right]$
$\displaystyle+\text{Cov}\left[-(\widehat{\beta}_{i}-\beta_{i})\cdot\frac{\sqrt{\tau^{2}}}{\beta_{i}^{2}}\cdot\text{
sign}(\beta_{i}),\frac{\widehat{\tau}^{2}-\tau^{2}}{|\beta_{j}|}\cdot\frac{1}{2\sqrt{\tau^{2}}}\right]$
$\displaystyle+\text{Cov}\left[-(\widehat{\beta}_{i}-\beta_{i})\cdot\frac{\sqrt{\tau^{2}}}{\beta_{i}^{2}}\cdot\text{
sign}(\beta_{i}),-(\widehat{\beta}_{j}-\beta_{j})\cdot\frac{\sqrt{\tau^{2}}}{\beta_{j}^{2}}\cdot\text{
sign}(\beta_{j})\right].$

Thus, remembering from Appendix 4A that $\widehat{\tau}$ and $\widehat{\beta}$
are independent so that the second and third terms above are zero, and
E$[\widehat{\tau}^{2}]\approx\tau^{2}$ and E$[\widehat{\beta}]\approx\beta$,
then by the definition of the covariance function

$\displaystyle\text{Cov}($
$\displaystyle\widehat{\text{CV}}_{i},\widehat{\text{CV}}_{j})\approx\
\text{Cov}\left[\frac{\widehat{\tau}^{2}-\tau^{2}}{|\beta_{i}|}\cdot\frac{1}{2\sqrt{\tau^{2}}},\frac{\widehat{\tau}^{2}-\tau^{2}}{|\beta_{j}|}\cdot\frac{1}{2\sqrt{\tau^{2}}}\right]$
$\displaystyle\qquad\qquad\quad+\text{Cov}\left[-(\widehat{\beta}_{i}-\beta_{i})\cdot\frac{\sqrt{\tau^{2}}}{\beta_{i}^{2}}\cdot\text{
sign}(\beta_{i}),-(\widehat{\beta}_{j}-\beta_{j})\cdot\frac{\sqrt{\tau^{2}}}{\beta_{j}^{2}}\cdot\text{
sign}(\beta_{j})\right]$
$\displaystyle=E\left[\frac{(\tau^{2}-\widehat{\tau}^{2})^{2}}{4\cdot\tau^{2}\cdot|\beta_{i}||\beta_{j}|}\right]+E\left[\frac{\tau^{2}}{\beta_{i}^{2}\cdot\beta_{j}^{2}}\cdot\text{
sign}(\beta_{i})\text{
sign}(\beta_{j})(\widehat{\beta_{i}}-\beta)(\widehat{\beta_{j}}-\beta)\right]$
$\displaystyle=\frac{\text{Var}(\widehat{\tau}^{2})}{4\cdot\tau^{2}\cdot|\beta_{i}||\beta_{j}|}+\frac{\tau^{2}}{\beta_{i}^{2}\cdot\beta_{j}^{2}}\cdot\text{
sign}(\beta_{i})\text{
sign}(\beta_{j})\cdot\text{Cov}(\widehat{\beta}_{i},\widehat{\beta}_{j}).$

Thus a variance-covariance matrix for $k$ dependent
$\widehat{\text{CV}}_{B}$’s is the matrix whose $ij$-th element is
$\text{Cov}(\widehat{\text{CV}}_{i},\widehat{\text{CV}}_{j})$. By introducing
the notations just prior to Theorem 4.1, it is simple to represent this in
matrix form.

## References

  * Baker et al., (2009) Baker, W. L., Michael White, C., Cappelleri, J. C., Kluger, J., Coleman, C. I., & From the Health Outcomes, Policy, and Economics (HOPE) Collaborative Group. 2009.  Understanding heterogeneity in meta-analysis: The role of meta-regression.  International Journal of Clinical Practice, 63(10), 1426–1434.
  * Borenstein et al., (2011) Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. 2011\.  Introduction to meta-analysis.  John Wiley & Sons.
  * Borenstein et al., (2017) Borenstein, M., Higgins, J. P. T., Hedges, L. V., & Rothstein, H. R. 2017\.  Basics of meta-analysis: I2 is not an absolute measure of heterogeneity.  Research Synthesis Methods, 8(1), 5–18.
  * Cairns & Prendergast, (2020) Cairns, M., & Prendergast, L. A. 2020\.  On ratio measures of heterogeneity for meta-analyses.  Research Synthesis Methods.  https://onlinelibrary.wiley.com/doi/10.1002/jrsm.1517.
  * Card, (2015) Card, N. A. 2015\.  Applied meta-analysis for social science research.  Guilford Publications.
  * Colditz et al., (1994) Colditz, G. A., Brewer, T. F., Berkey, C. S., Wilson, M. E., Burdick, E., Fineberg, H. V., & Mosteller, F. 1994\.  Efficacy of BCG vaccine in the prevention of tuberculosis: Meta-analysis of the published literature.  JAMA, 271(9), 698–702.
  * de Bruin et al., (2009) de Bruin, M., Viechtbauer, W., Hospers, H. J., Schaalma, H. P., & Kok, G. 2009\.  Standard care quality determines treatment outcomes in control groups of HAART-adherence intervention studies: Implications for the interpretation and comparison of intervention effects.  Health Psychology, 28(6), 668.
  * DerSimonian & Laird, (1986) DerSimonian, R., & Laird, N. 1986\.  Meta-analysis in clinical trials.  Controlled Clinical Trials, 7, 177–188.
  * Higgins & Thompson, (2002) Higgins, J. P. T., & Thompson, S. G. 2002\.  Quantifying heterogeneity in a meta-analysis.  Statistics in Medicine, 21, 1539–1558.
  * Higgins & Thompson, (2004) Higgins, J. P. T., & Thompson, S. G. 2004\.  Controlling the risk of spurious findings from meta-regression.  Statistics in Medicine, 23(11), 1663–1682.
  * Langan et al., (2019) Langan, D., Higgins, J. P. T., Jackson, D., Bowden, J., Veroniki, A. A., Kontopantelis, E., Viechtbauer, W., & Simmonds, M. 2019\.  A comparison of heterogeneity variance estimators in simulated random-effects meta-analyses.  Research Synthesis Methods, 10(1), 83–98.
  * Lloyd, (1990) Lloyd, C. J. 1990\.  Confidence intervals from the difference between two correlated proportions.  Journal of the American Statistical Association, 85(412), 1154–1158.
  * Newcombe, (2011) Newcombe, R. G. 2011\.  Propagating imprecision: Combining confidence intervals from independent sources.  Communications in Statistics – Theory and Methods, 40(17), 3154–3180.
  * R Core Team, (2020) R Core Team. 2020.  R: A Language and Environment for Statistical Computing.
  * Sidik & Jonkman, (2005) Sidik, K., & Jonkman, J. N. 2005\.  Simple heterogeneity variance estimation for meta-analysis.  Journal of the Royal Statistical Society: Series C (Applied Statistics), 54(2), 367–384.
  * Takkouche et al., (1999) Takkouche, B., Cadarso-Suarez, C., & Spiegelman, D. 1999\.  Evaluation of old and new tests of heterogeneity in epidemiologic meta-analysis.  American Journal of Epidemiology, 150(2), 206–215.
  * Takkouche et al., (2013) Takkouche, B., Khudyakov, P., Costa-Bouzas, J., & Spiegelman, D. 2013\.  Confidence intervals for heterogeneity measures in meta-analysis.  American Journal of Epidemiology, 178(6), 993–1004.
  * Thompson & Higgins, (2002) Thompson, S. G., & Higgins, J. P. T. 2002\.  How should meta-regression analyses be undertaken and interpreted?  Statistics in Medicine, 21(11), 1559–1573.
  * Veroniki et al., (2016) Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P. T., Langan, D., & Salanti, G. 2016\.  Methods to estimate the between-study variance and its uncertainty in meta-analysis.  Research Synthesis Methods, 7, 55–79.
  * Viechtbauer, (2010) Viechtbauer, W. 2010\.  Conducting meta-analyses in R with the metafor package.  Journal of Statistical Software, 36(3), 1–48.