11institutetext: NeuroPoly Lab, Institute of Biomedical Engineering,
Polytechnique Montreal, Montreal, QC, Canada. 22institutetext: Functional
Neuroimaging Unit, CRIUGM,
Universite de Montreal, Montreal, QC, Canada.

# Deep semi-supervised segmentation with weight-averaged consistency targets

Christian S. Perone 11 0000-0002-1894-6924
Julien Cohen-Adad 1122 0000-0003-3662-9532

###### Abstract

Recently proposed techniques for semi-supervised learning such as Temporal
Ensembling and Mean Teacher have achieved state-of-the-art results in many
important classification benchmarks. In this work, we expand the Mean Teacher
approach to segmentation tasks and show that it can bring important
improvements in a realistic small data regime using a publicly available
multi-center dataset from the Magnetic Resonance Imaging (MRI) domain. We also
devise a method to solve the problems that arise when using traditional data
augmentation strategies for segmentation tasks on our new training scheme.

##  1 Introduction

In the past few years, we witnessed a large growth in the development of Deep
Learning techniques, that surpassed human-level performance on some important
tasks [1], including health domain applications [2]. A recent survey [3] that
examined more than 300 papers using Deep Learning techniques in medical
imaging analysis, made it clear that Deep Learning is now pervasive across the
entire field. In [3], they also found that Convolutional Neural Networks
(CNNs) were more prevalent in the medical imaging analysis, with end-to-end
trained CNNs becoming the preferred approach.

It is also evident that Deep Learning poses unique challenges, such as the
large amount of data requirement, which can be partially mitigated by using
transfer learning [4] or domain adaptation approaches [5], especially in the
natural imaging domain. However, in medical imaging domain, not only the image
acquisition is expensive but also data annotations, that usually requires a
very time-consuming dedication of experts. Besides that, other challenges are
still present in the medical imaging field, such as privacy and
regulations/ethical concerns, which are also an important factor impacting the
data availability.

According to [3], in certain domains, the main challenge is usually not the
availability of the image data itself, but the lack of relevant
annotations/labeling for these images. Traditionally, systems like Picture
Archiving and Communication System (PACS) [3], used in the routine of most
western hospitals, store free-text reports, and turning this textual
information into accurate or structured labels can be quite challenging.
Therefore, the development of techniques that could take advantage of the vast
amount of unlabeled data is paramount for advancing the current state of
practical applications in medical imaging.

Semi-supervised learning is a class of learning algorithms that can take
leverage not only of labeled samples but also from unlabeled samples. Semi-
supervised learning is halfway between supervised learning and unsupervised
learning [6], where the algorithm uses limited supervision, usually only from
a few samples of a dataset together with a larger amount of unlabeled data.

In this work, we propose a simple deep semi-supervised learning approach for
segmentation that can be efficiently implemented. Our technique is robust
enough to be incorporated in most traditional segmentation architectures since
it decouples the semi-supervised training from the architectural choices. We
show experimentally on a public Magnetic Resonance Imaging (MRI) dataset that
this technique can take advantage of unlabeled data and provide improvements
even in a realistic scenario of small data regime, a common reality in medical
imaging.

##  2 Semi-supervised segmentation using Mean Teacher

Given that the classification cost for unlabeled samples is undefined in
supervised learning, adding unlabeled samples into the training procedure can
be quite challenging. Traditionally, there is a dataset
$\textbf{X}=(x_{i})_{i\in[n]}$ that can be divided into two disjoint sets: the
samples $\textbf{X}_{l}=(x_{1},\ldots,x_{l})$ that contains the labels
$\textbf{Y}_{l}=(y_{1},\dots,y_{l})$, and the samples
$\textbf{X}_{u}=(x_{l+1},\ldots,x_{l+u})$ where the labels are unknown.
However, if the knowledge available in $p(x)$ that we can get from the
unlabeled data also contains information that is useful for the inference
problem of $p(y|x)$, then it is evident that semi-supervised learning can
improve upon supervised learning [6].

Many techniques were developed in the past for semi-supervised learning,
usually creating surrogate classes as in [7], adding entropy regularization as
in [8] or using Generative Adversarial Networks (GANs) [9]. More recently,
other ideas also led to the development of techniques that added perturbations
and extra reconstruction costs in the intermediate representations [10] of the
network, yielding excellent results. A very successful method called Temporal
Ensembling [11] was also recently introduced, where the authors explored the
idea of a temporal ensembling network for semi-supervised learning where the
predictions of multiple previous network evaluations were aggregated using an
exponential moving average (EMA) with a penalization term for the predictions
that were inconsistent with this target, achieving state-of-the-art results in
several semi-supervised learning benchmarks.

In [12], the authors expanded the Temporal Ensembling method by averaging the
model weights instead of the label predictions by using Polyak averaging [13].
The method described in [12] is a student/teacher model, where the student
model architecture is replicated into the teacher model, which in turn, get
its weights updated as the exponential moving average of the student weights
according to:

$\displaystyle\theta^{\prime}_{t}=\alpha\theta^{\prime}_{t-1}+(1-\alpha)\theta_{t}$
(1)

where $\alpha$ is a smoothing hyperparameter, $t$ is the training step and
$\theta$ are the model weights. The goal of the student is to learn through a
composite loss function with two terms: one for the traditional classification
loss and another to enforce the consistency of its predictions with the
teacher model. Both the student and teacher models evaluate the input data by
applying noise that can come from Dropout, random affine transformations,
added Gaussian noise, among others.

In this work, we extend the mean teacher technique [12] to semi-supervised
segmentation. To the best of our knowledge, this is the first time that this
semi-supervised method was extended for segmentation tasks. Our changes to
extend the mean teacher [12] technique for segmentation are simple: we use
different loss functions both for the task and consistency and also propose a
new method for solving the augmentation issues that arises from this technique
when used for segmentation. For the consistency loss, we use a pixel-wise
binary cross-entropy, formulated as

$\displaystyle\mathcal{C}(\theta)={}\mathbb{E}_{x\in\textbf{X}}\left[-y\log(p)+(1-y)\
log(1-p)\right],$ (2)

where $p\in[0,1]$ is the output (after sigmoid activation) of the student
model $f(x;\theta)$ and $y\in[0,1]$ is the output prediction for the same
sample from the teacher model $f(x;\theta^{\prime})$, where $\theta$ and
$\theta^{\prime}$ are student and teacher model parameters respectively. The
consistency loss can be seen as a pixel-wise knowledge distillation [14] from
the teacher model. It is important to note that both labeled samples from
$\textbf{X}_{l}$ and unlabeled samples from $\textbf{X}_{u}$ contribute for
the consistency loss $\mathcal{C}(\theta)$ calculation. We used binary cross-
entropy, instead of the mean squared error (MSE) used by [12] because the
binary cross-entropy provided an improved model performance for the
segmentation task. We also experimented with confidence thresholding as in
[15] on the teacher predictions, however, it didn’t improve the results.

For the segmentation task, we employed a surrogate loss for the Dice
Similarity Coefficient, called the Dice loss, which is insensitive to
imbalance and was also employed by [16] on the same segmentation task domain
we experiment in this paper. The Dice Loss, computed per mini-batch, is
formulated as

$\displaystyle
L(\theta)=-\frac{2\sum_{i}p_{i}y_{i}}{\sum_{i}p_{i}+\sum_{i}y_{i}},$ (3)

where $p_{i}\in[0,1]$ is the $i^{th}$ output (after sigmoid non-linearity) and
$y_{i}\in\\{0,1\\}$ is the corresponding ground truth. For the segmentation
loss, only labeled samples from $\textbf{X}_{l}$ contribute for the
$\mathcal{L}(\theta)$ calculation. As in [12], the total loss used is the
weighted sum of both segmentation and consistency losses. An overview
detailing the components of the method can be seen in the Figure 1, while a
description of the training algorithm is described in the Algorithm 1.

Figure 1:  An overview with the components of the proposed method based on the mean teacher technique. (1) A data augmentation procedure $g(x;\phi)$ is used to perturb the input data (in our case, a MRI axial slice), where $\phi$ is the data augmentation parameter (i.e. $\mathcal{N}(0,\,\phi$) for a Gaussian noise), note that different augmentation parameters are used for student and teacher models. (2) The student model. (3) The teacher model that is updated with an exponential moving average (EMA) from the student weights. (4) The consistency loss used to train the student model. This consistency will enforce the consistency between student predictions on both labeled and unlabelled data according to the teacher predictions. (5) The traditional segmentation loss, where the supervision signal is provided to the student model for the labeled samples.  Algorithm 1 Semi-supervised segmentation algorithm.  Require: $x_{i}$ | = training samples |
---|---|---
Require: $y_{i}$ | = labels for the labeled inputs $i\in\textbf{Y}_{l}$
Require: $t$ | = global step (initialized with zero)
Require: $w(t)$ = consistency weight ramp-up function
Require: $f_{\theta}(\cdot)$ = neural network model with parameters $\theta$
Require: $g_{\phi}(\cdot)$ = stochastic input augmentation procedure with
parameters $\phi$
for $k$ in $[1,\mathit{num\\_epochs}]$ do
for each minibatch $B$ do
$z_{i\in B}\leftarrow f_{\theta}(g_{\phi}(x_{i\in B}))$ |  |  $\triangleright$ evaluate augmented inputs with student model
$\tilde{z}_{i\in B}\leftarrow f_{\theta^{\prime}}(g_{\phi^{\prime}}(x_{i\in B}))$ |  |  $\triangleright$ teacher model evaluation w/ different perturbations
$\mathit{loss}{}\leftarrow$ |  ${}\,\mathcal{L}(z,y)+w(t)\frac{1}{|B|}\sum_{i\in B}\mathcal{C}(z_{i},\tilde{z}_{i})$ |  $\triangleright$ supervised and unsupervised loss components
update $\theta$ using, e.g., Adam |  |  $\triangleright$ update student model parameters
$t\leftarrow t+1$ |  |  $\triangleright$ increment the global step counter
$\theta^{\prime}_{t}\leftarrow$ |  $\alpha\theta^{\prime}_{t-1}+(1-\alpha)\theta_{t}$ |  $\triangleright$ update teacher model parameters with using EMA
end for
end for

###  2.1 Segmentation data augmentation

In segmentation tasks, data augmentation is very important, especially in the
medical imaging domain where data availability is limited, variability is high
and translational equivariance is desirable. Traditional augmentation methods
such as affine transformations (rotation, translation, etc) that change the
spatial content of the input data, as opposed to pixel-wise additive noise,
for example, are also applied with the exact same parameters on the label to
spatially align input and ground truth, both subject to a pixel-wise loss.
This methodology, however, is unfeasible in the mean teacher training scheme.
If two different augmentations (one for the student and another for the
teacher) causes spatial misalignment, the spatial content between student and
teacher predictions will not match during the pixel-wise consistency loss.

To avoid the misalignment during the consistency loss, such transformations
can be applied with the same parametrization both to the student and teacher
model inputs. However, this wouldn’t take advantage of the stronger invariance
to transformations that can be introduced through the consistency loss. For
that reason, we developed a solution that applies the transformations in the
teacher in a delayed fashion. Our proposed method is based on the application
of the same augmentation procedure $g(x;\phi)$ before the model forward pass
only for the student model, and then after model forward pass in the teacher
model predictions, making thus both prediction maps aligned for the
consistency loss evaluation, while still taking leverage of introducing a much
stronger invariance to the augmentation between student and teacher models.
This is possible because we do backpropagation of the gradients only for the
student model parameters.

##  3 Experiments

###  3.1 MRI Spinal Cord Gray Matter Segmentation

In this work, in order to evaluate our technique on a realistic scenario, we
use the publicly available multi-center Magnetic Resonance Imaging (MRI)
Spinal Cord Gray Matter Segmentation dataset from [17].

####  3.1.1 Dataset

The dataset is comprised of 80 healthy subjects (20 subjects from each center)
and obtained using different scanning parameters and also multiple MRI
systems. The voxel resolution of the dataset ranges from 0.25x0.25x2.5 mm up
to 0.5x0.5x5.0 mm. A sample of one subject axial slice image can be seen in
Figure 1. We split the dataset in a realistic small data regime: only 8
subjects are used as training samples, resulting in 86 axial training slices.
We used 8 subjects for validation, resulting in 90 axial slices. For the
unlabeled set we used 40 subjects, resulting in 613 axial slices and for the
test set we used 12 subjects, resulting in 137 slices. All samples were
resampled to a common space of 0.25x0.25 mm.

####  3.1.2 Network Architecture

To evaluate our technique, we used a very simple U-Net [18] architecture with
15 layers, Batch Normalization, Dropout and ReLU activations. U-Nets are very
common in medical imaging domain, hence the architectural choice for the
experiment. We also used a 2D slice-wise training procedure with axial slices.

####  3.1.3 Training procedure

For the supervised-only baseline, we used Adam optimizer with $\beta_{1}=0.9$
and $\beta_{2}=0.999$, mini-batch size of 8, dropout rate of 0.5, Batch
Normalization momentum of 0.9 and L2 penalty of $\lambda=0.0008$. For the data
augmentation, we used rotation angle between $None$ and $None$ and pixel-wise
additive Gaussian noise sampled from $\mathcal{N}(0,\,0.01)$. We used a
learning rate $\eta=0.0006$ given the small mini-batch size, also subject to a
initial ramp-up of 50 epochs and subject to a cosine annealing decay as used
by [12]. We trained the model for 1600 epochs.

For the semi-supervised experiment, we used the same parameters of the
aforementioned supervised-only baseline, except for the L2 penalty of
$\lambda=0.0006$. We used an EMA $\alpha=0.99$ during the first 50 epochs,
later we change it to $\alpha=0.999$. We also employed a consistency weight
factor of 2.9 subject to a ramp-up in the first 100 epochs. We trained the
model for 350 epochs.

####  3.1.4 Results

As we can see in Table 1, our technique not only improved the results on 5/6
evaluated metrics but also reduced the variance, showing a better regularized
model in terms of precision/recall balance. The model also showed a very good
improvement on overlapping metrics such as Dice and mean intersection over
union (mIoU). Given that we exhausted the challenge dataset [17] to obtain the
unlabeled samples, a comparison with [16] was unfeasible given different
dataset splits. We leave this work for further explorations given that
incorporating extra external data would also mix domain adaptation issues into
the evaluation.

Table 1: Result comparison for the Spinal Cord Gray Matter segmentation
challenge using our semi-supervised method and a pure supervised baseline.
Results are 10 runs average with standard deviation in parenthesis where bold
font represents the best result. Dice is the Dice Similarity Coefficient and
mIoU is the mean intersection over union. Other metrics are self-explanatory.

| Dice | mIoU | Accuracy | Precision | Recall | Specificity
---|---|---|---|---|---|---
Supervised |  | 67.915
---
(0.313)
| 53.679
---
(0.327)
| 99.745
---
(0.005)
| 57.948
---
(0.788)
| 92.495
---
(0.907)
| 99.775
---
(0.010)
Semi-supervised |  | 70.209
---
(0.229)
| 55.509
---
(0.253)
| 99.792
---
(0.003)
| 64.732
---
(0.773)
| 86.112
---
(0.936)
| 99.846
---
(0.006)

##  4 Related Work

Only a few works were developed in the context of semi-supervised
segmentation, especially in the field of medical imaging. Only recently, a
U-Net was used as auxiliary embedding in [19], however, with focus on domain
adaptation and using a private dataset.

In [20], they use a Generative Adversarial Networks (GAN) for the semi-
supervised segmentation of natural images, however, they employ unrealistic
dataset sizes when compared to the medical imaging domain datasets, along with
ImageNet pre-trained networks.

In [21] they propose a technique using adversarial training, but they focus on
the knowledge transfer between natural images with pixel-level annotation and
weakly-labeled images with image-level information.

##  5 Conclusion

In this work we extended the semi-supervised mean teacher approach for
segmentation tasks, showing that even on a realistic small data regime, this
technique can provide major improvements if unlabeled data is available. We
also devised a way to maintain the traditional data augmentation procedures
while still taking advantage of the teacher/student regularization. The
proposed technique can be used with any other Deep Learning architecture since
it decouples the semi-supervised training procedure from the architectural
choices.

It is evident from these results that future explorations of this technique
can improve the results even further, given that even with a small amount of
unlabeled samples, we showed that the technique was able to provide
significant improvements.

##  6 Acknowledgements

Funded by the Canada Research Chair in Quantitative Magnetic Resonance Imaging
(JCA), the Canadian Institute of Health Research [CIHR FDN-143263], the Canada
Foundation for Innovation [32454, 34824], the Fonds de Recherche du Québec -
Santé [28826], the Fonds de Recherche du Québec - Nature et Technologies
[2015-PR-182754], the Natural Sciences and Engineering Research Council of
Canada [435897-2013], IVADO, TransMedTech, the Quebec BioImaging Network and
NVIDIA Corporation for the donation of a GPU.

## References

  * [1] He, K., Zhang, X., Ren, S., Sun, J.:  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  Proceedings of the IEEE International Conference on Computer Vision 11-18-Dece (2016) 1026–1034
  * [2] Rajpurkar, P., Hannun, A.Y., Haghpanahi, M., Bourn, C., Ng, A.Y.:  Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks.  arXiv preprint (2017)
  * [3] Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., van der Laak, J.A., van Ginneken, B., Sánchez, C.I.:  A survey on deep learning in medical image analysis.  Medical Image Analysis 42 (2017) 60–88
  * [4] Yosinski, J., Clune, J., Bengio, Y., Lipson, H.:  How transferable are features in deep neural networks?  Advances in Neural Information Processing Systems 27 (Proceedings of NIPS) 27 (2014) 1–9
  * [5] Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V., Dogan, U., Kloft, M., Orabona, F., Tommasi, T.:  Domain-Adversarial Training of Neural Networks.  Journal of Machine Learning Research 17 (2016) 1–35
  * [6] Olivier, C., Schölkopf, B., Zien, A.:  Semi-Supervised Learning.  Interdisciplinary sciences computational life sciences 1(2) (2006) 524
  * [7] Lee, D.H.:  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  ICML 2013 Workshop: Challenges in Representation Learning (2013) 1–6
  * [8] Grandvalet, Y., Bengio, Y.:  Semi-supervised Learning by Entropy Minimization.  Advances in Neural Information Processing Systems - NIPS’04 17 (2004) 529–536
  * [9] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.:  Improved Techniques for Training GANs.  (2016)
  * [10] Rasmus, A., Valpola, H., Berglund, M.:  Semi-Supervised Learning with Ladder Network.  arXiv (2015) 1–17
  * [11] Laine, S., Aila, T.:  Temporal Ensembling for Semi-Supervised Learning
  * [12] Tarvainen, A., Valpola, H.:  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  (2017)
  * [13] Polyak, B.T., Juditsky, A.B.:  Acceleration of Stochastic Approximation by Averaging.  SIAM Journal on Control and Optimization 30(4) (1992) 838–855
  * [14] Hinton, G., Vinyals, O., Dean, J.:  Distilling the Knowledge in a Neural Network.  (mar 2015)
  * [15] French, G., Mackiewicz, M., Fisher, M.:  Self-ensembling for domain adaptation.  (2017) 1–15
  * [16] Perone, C.S., Calabrese, E., Cohen-Adad, J.:  Spinal cord gray matter segmentation using deep dilated convolutions.  (oct 2017)
  * [17] Prados, F., Ashburner, J., Blaiotta, C., Brosch, T., Carballido-Gamio, J., Cardoso, M.J., Conrad, B.N., Datta, E., Dávid, G., Leener, B.D., Dupont, S.M., Freund, P., Wheeler-Kingshott, C.A.G., Grussu, F., Henry, R., Landman, B.A., Ljungberg, E., Lyttle, B., Ourselin, S., Papinutto, N., Saporito, S., Schlaeger, R., Smith, S.A., Summers, P., Tam, R., Yiannakas, M.C., Zhu, A., Cohen-Adad, J.:  Spinal cord grey matter segmentation challenge.  NeuroImage 152 (2017) 312–329
  * [18] Ronneberger, O., Fischer, P., Brox, T.:  U-net: Convolutional networks for biomedical image segmentation.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 9351 (2015) 234–241
  * [19] Baur, C., Albarqouni, S., Navab, N.:  Semi-supervised Deep Learning for Fully Convolutional Networks.  Miccai-2017 (2017) 311–319
  * [20] Souly, N., Spampinato, C., Shah, M.:  Semi Supervised Semantic Segmentation Using Generative Adversarial Network.  In: 2017 IEEE International Conference on Computer Vision (ICCV), IEEE (oct 2017) 5689–5697
  * [21] Xiao, H., Wei, Y., Liu, Y., Zhang, M., Feng, J.:  Transferable Semi-supervised Semantic Segmentation.  (2017)


# Evaluating Gender Bias in Machine Translation

Gabriel Stanovsky Paul G. Allen School of Computer Science & Engineering,
University of Washington, Seattle, USA  Allen Institute for Artificial
Intelligence, Seattle, USA  Noah A. Smith Paul G. Allen School of Computer
Science & Engineering, University of Washington, Seattle, USA  Allen Institute
for Artificial Intelligence, Seattle, USA  Luke Zettlemoyer Paul G. Allen
School of Computer Science & Engineering, University of Washington, Seattle,
USA

###### Abstract

We present the first challenge set and evaluation protocol for the analysis of
gender bias in machine translation (MT). Our approach uses two recent
coreference resolution datasets composed of English sentences which cast
participants into non-stereotypical gender roles (e.g., ‘‘The doctor asked the
nurse to help _her_ in the operation’’). We devise an automatic gender bias
evaluation method for eight target languages with grammatical gender, based on
morphological analysis (e.g., the use of female inflection for the word
‘‘doctor’’). Our analyses show that four popular industrial MT systems and two
recent state-of-the-art academic MT models are significantly prone to gender-
biased translation errors for all tested target languages. Our data and code
are publicly available at shorturl.at/dimuD.

##  1 Introduction

Learned models exhibit social bias when their training data encode stereotypes
not relevant for the task, but the correlations are picked up anyway. Notable
examples include gender biases in visual SRL (cooking is stereotypically done
by women, construction workers are stereotypically men; Zhao et al., 2017),
lexical semantics (‘‘man is to computer programmer as woman is to homemaker’’;
Bolukbasi et al., 2016), and natural language inference (associating women
with gossiping and men with guitars; Rudinger et al., 2017).

The doctor asked the nurse to help her in the procedureEl doctor le pidio a la
enfermera que le ayudara con el procedimiento

Figure 1:  An example of gender bias in machine translation from English (top)
to Spanish (bottom). In the English source sentence, the nurse’s gender is
unknown, while the coreference link with ‘‘her’’ identifies the ‘‘doctor’’ as
a female. On the other hand, the Spanish target sentence uses morphological
features for gender: ‘‘ _el_ doctor’’ (male), versus ‘‘ _la_ enfermer _-a_ ’’
(female). Aligning between source and target sentences reveals that a
stereotypical assignment of gender roles changed the meaning of the translated
sentence by changing the doctor’s gender.

In this work, we conduct the first large-scale multilingual evaluation of
gender-bias in machine translation (MT), following recent small-scale
qualitative studies which observed that online MT services, such as Google
Translate or Microsoft Translator, also exhibit biases, e.g., translating
nurses as females and programmers as males, regardless of context Alvarez-
Melis and Jaakkola (2017); Font and Costa-Jussà (2019). Google Translate
recently tried to mitigate these biases by allowing users to sometimes choose
between gendered translations Kuczmarski (2018).

As shown in Figure 1, we use data introduced by two recent coreference gender-
bias studies: the Winogender Rudinger et al. (2018), and the WinoBias Zhao et
al. (2018) datasets. Following the Winograd schema Levesque (2011), each
instance in these datasets is an English sentence which describes a scenario
with human entities, who are identified by their role (e.g., ‘‘the doctor’’
and ‘‘the nurse’’ in Figure 1), and a pronoun (‘‘her’’ in the example), which
needs to be correctly resolved to one of the entities (‘‘the doctor’’ in this
case). Rudinger et al. (2018) and Zhao et al. (2018) found that while human
agreement on the task was high (roughly 95%), coreference resolution models
often ignore context and make socially biased predictions, e.g., associating
the feminine pronoun ‘‘her’’ with the stereotypically female ‘‘nurse.’’

We observe that for many target languages, a faithful translation requires a
similar form of (at least implicit) gender identification. In addition, in the
many languages which associate between biological and grammatical gender
(e.g., most Romance, Germanic, Slavic, and Semitic languages; Craig, 1986;
Mucchi-Faina, 2005; Corbett, 2007), the gender of an animate object can be
identified via morphological markers. For instance, when translating our
running example in Figure 1 to Spanish, a valid translation may be: ‘‘ _La
doctora_ le pidio a la enfermera que le ayudara con el procedimiento,’’ which
indicates that the doctor is a woman, by using a feminine suffix inflection
(‘‘doctor _a_ ’’) and the feminine definite gendered article (‘‘ _la_ ’’).
However, a biased translation system may ignore the given context and
stereotypically translate the doctor as male, as shown at the bottom of the
figure.

Following these observations, we design a challenge set approach for
evaluating gender bias in MT using a concatenation of Winogender and WinoBias.
We devise an automatic translation evaluation method for eight diverse target
languages, without requiring additional gold translations, relying instead on
automatic measures for alignment and morphological analysis (Section 2). We
find that four widely used commercial MT systems and two recent state-of-the-
art academic models are significantly gender-biased on all tested languages
(Section 3). Our method and benchmarks are publicly available, and are easily
extensible with more languages and MT models.

##  2 Challenge Set for Gender Bias in MT

We compose a challenge set for gender bias in MT (which we dub ‘‘WinoMT’’) by
concatenating the Winogender and WinoBias coreference test sets. Overall,
WinoMT contains 3,888 instances, and is equally balanced between male and
female genders, as well as between stereotypical and non-stereotypical gender-
role assignments (e.g., a female doctor versus a female nurse). Additional
dataset statistics are presented in Table 1.

We use WinoMT to estimate the gender-bias of an MT model, $M$, in target-
language $L$ by performing following steps (exemplified in Figure 1):

(1) Translate all of the sentences in WinoMT into $L$ using $M$, thus forming
a bilingual corpus of English and the target language $L$.

(2) Align between the source and target translations, using _fast_align_ Dyer
et al. (2013), trained on the automatic translations from from step (1). We
then map the English entity annotated in the coreference datasets to its
translation (e.g., align between ‘‘the doctor’’ and ‘‘el doctor’’ in Figure
1).

(3) Finally, we extract the target-side entity’s gender using simple
heuristics over language-specific morphological analysis, which we perform
using off-the-shelf tools for each target language, as discussed in the
following section.

This process extracts the translated genders, according to $M$, for all of the
entities in WinoMT, which we can then evaluate against the gold annotations
provided by the original English dataset.

This process can introduce noise into our evaluation in steps (2) and (3), via
wrong alignments or erroneous morphological analysis. In Section 3, we will
present a human evaluation showing these errors are infrequent.

| Winogender | WinoBias | WinoMT
---|---|---|---
Male | 240 | 1582 | 1826
Female | 240 | 1586 | 1822
Neutral | 240 | 0 | 240
Total | 720 | 3168 | 3888
Table 1: The coreference test sets and resulting WinoMT corpus statistics (in
number of instances).

##  3 Evaluation

| Google Translate | Microsoft Translator | Amazon Translate∗ | SYSTRAN
---|---|---|---|---
| Acc | $\Delta_{G}$ | $\Delta_{S}$ | Acc | $\Delta_{G}$ | $\Delta_{S}$ | Acc | $\Delta_{G}$ | $\Delta_{S}$ | Acc | $\Delta_{G}$ | $\Delta_{S}$
ES | 53.1 | 23.4 | 21.3 | 47.3 | 36.8 | 23.2 | 59.4 | 15.4 | 22.3 | 45.6 | 46.3 | 15.0
FR |  63.6 | 6.4 | 26.7 | 44.7 | 36.4 | 29.7 | 55.2 | 17.7 | 24.9 | 45.0 | 44.0 | 9.4
IT | 39.6 | 32.9 | 21.5 | 39.8 | 39.8 | 17.0 | 42.4 | 27.8 | 18.5 | 38.9 | 47.5 | 9.4
RU | 37.7 | 36.8 | 11.4 | 36.8 | 42.1 | 8.5 | 39.7 | 34.7 | 9.2 | 37.3 | 44.1 | 9.3
UK | 38.4 | 43.6 | 10.8 | 41.3 | 46.9 | 11.8 | \-- | \-- | \-- | 28.9 | 22.4 | 12.9
HE | 53.7 | 7.9 | 37.8 | 48.1 | 14.9 | 32.9 | 50.5 | 10.3 | 47.3 | 46.6 | 20.5 | 24.5
AR | 48.5 | 43.7 | 16.1 | 47.3 | 48.3 | 13.4 | 49.8 | 38.5 | 19.0 | 47.0 | 49.4 | 5.3
DE | 59.4 | 12.5 | 12.5 |  74.1 | 0.0 | 30.2 |  62.4 | 12.0 | 16.7 |  48.6 | 34.5 | 10.3
Table 2:  Performance of commercial MT systems on the WinoMT corpus on all tested languages, categorized by their family: Spanish, French, Italian, Russian, Ukrainian, Hebrew, Arabic, and German. _Acc_ indicates overall gender accuracy (% of instances the translation had the correct gender), $\Delta_{G}$ denotes the difference in performance ($F_{1}$ score) between masculine and feminine scores, and $\Delta_{S}$ is the difference in performance ($F_{1}$ score) between pro-stereotypical and anti-stereotypical gender role assignments (higher numbers in the two latter metrics indicate stronger biases). Numbers in bold indicate best accuracy for the language across MT systems (row), and underlined numbers indicate best accuracy for the MT system across languages (column). ∗Amazon Translate does not have a trained model for English to Ukrainian.  | Acc | $\Delta_{G}$ | $\Delta_{S}$
---|---|---|---
FR Ott et al. (2018) | 49.4 | 2.6 | 16.1
DE Edunov et al. (2018) | 52.5 | 7.3 | 8.4
Table 3: Performance of recent state-of-the-art academic translation models
from English to French and German. Metrics are the same as those in Table 2.

In this section, we briefly describe the MT systems and the target languages
we use, our main results, and their human validation.

###  3.1 Experimental Setup

#### MT systems

We test six widely used MT models, representing the state of the art in both
commercial and academic research: (1) Google
Translate,111https://translate.google.com (2) Microsoft
Translator,222https://www.bing.com/translator (3) Amazon
Translate,333https://aws.amazon.com/translate (4)
SYSTRAN,444http://www.systransoft.com (5) the model of Ott et al. (2018),
which recently achieved the best performance on English-to-French translation
on the WMT’14 test set, and (6) the model of Edunov et al. (2018), the WMT’18
winner on English-to-German translation. We query the online API for the first
four commercial MT systems, while for the latter two academic models we use
the pretrained models provided by the Fairseq
toolkit.555https://github.com/pytorch/fairseq

#### Target languages and morphological analysis

We selected a set of eight languages with grammatical gender which exhibit a
wide range of other linguistic properties (e.g., in terms of alphabet, word
order, or grammar), while still allowing for highly accurate automatic
morphological analysis. These languages belong to four different families: (1)
Romance languages: Spanish, French, and Italian, all of which have gendered
noun-determiner agreement and spaCy morphological analysis support Honnibal
and Montani (2017). (2) Slavic languages (Cyrillic alphabet): Russian and
Ukrainian, for which we use the morphological analyzer developed by Korobov
(2015). (3) Semitic languages: Hebrew and Arabic, each with a unique alphabet.
For Hebrew, we use the analyzer developed by Adler and Elhadad (2006), while
gender inflection in Arabic can be easily identified via the _ta marbuta_
character, which uniquely indicates feminine inflection. (4) Germanic
languages: German, for which we use the morphological analyzer developed by
Altinok (2018).

ESFRITRUUKHEARDE$20$$40$$60$$80$$100$$67$$80$$52$$44$$46$$76$$60$$69$$46$$54$$30$$33$$35$$38$$44$$57$Accuracy
(%)StereotypicalNon-Stereotypical Figure 2: Google Translate’s performance on
gender translation on our tested languages. The performance on the
stereotypical portion of WinoMT is consistently better than that on the non-
stereotypical portion. The other MT systems we tested display similar trends.

###  3.2 Results

Our main findings are presented in Tables 2 and 3. For each tested MT system
and target language we compute three metrics with respect to their ability to
convey the correct gender in the target language. Ultimately, our analyses
indicate that all tested MT systems are indeed gender biased.

First, the overall system _Accuracy_ is calculated by the percentage of
instances in which the translation preserved the gender of the entity from the
original English sentence. We find that most tested systems across eight
tested languages perform quite poorly on this metric. The best performing
model on each language often does not do much better than a random guess for
the correct inflection. An exception to this rule is the translation
accuracies on German, where three out of four systems acheive their best
performance. This may be explained by German’s similarity to the English
source language Hawkins (2015).

In Table 2, $\Delta_{G}$ denotes the difference in performance ($F_{1}$ score)
between male and female translations. Interestingly, all systems, except
Microsoft Translator on German, perform significantly better on male roles,
which may stem from these being more frequent in the training set.

Perhaps most tellingly, $\Delta_{S}$ measures the difference in performance
($F_{1}$ score) between stereotypical and non-stereotypical gender role
assignments, as defined by Zhao et al. (2018) who use statistics provided by
the US Department of Labor.666https://www.bls.gov/cps/cpsaat11.htm This metric
shows that all tested systems have a significant and consistently better
performance when presented with pro-stereotypical assignments (e.g., a female
nurse), while their performance deteriorates when translating anti-
stereotypical roles (e.g., a male receptionist). For instance, Figure 2
depicts Google Translate absolute accuracies on stereotypical and non-
stereotypical gender roles across all tested languages. Other tested systems
show similar trends.

| Original | +Adj | $\Delta$
---|---|---|---
ES | 53.1 | 63.5 | +10.4
RU | 37.7 | 48.9 | +11.2
UK | 38.4 | 42.9 | +4.5
Table 4: Performance of Google Translate on Spanish, Russian, and Ukranian
gender prediction accuracy (% correct) on the original WinoMT corpus, versus a
modified version of the dataset where we add sterotypical gender adjectives
(see Section 3.3).

###  3.3 Fighting Bias with Bias

Finally, we tested whether we can affect the translations by automatically
creating a version of WinoMT with the adjectives ‘‘handsome’’ and ‘‘pretty’’
prepended to male and female entities, respectively. For example, the sentence
in Figure 1 will be converted to: ‘‘The _pretty_ doctor asked the nurse to
help her in the operation’’. We are interested in evaluating whether this
‘‘corrects’’ the profession bias by mixing signals, e.g., while ‘‘doctor’’
biases towards a male translation, ‘‘pretty’’ tugs the translation towards a
female inflection. Our results show that this improved performance in some
languages, significantly reducing bias in Spanish, Russian, and Ukrainian (see
Table 4). Admittedly, this is impractical as a general debiasing scheme, since
it assumes oracle coreference resolution, yet it attests to the relation
between coreference resolution and MT, and serves as a further indication of
gender bias in MT.

Source | [Target lang.] Predicted translation |  Phenomenon
---|---|---
The janitor does not like  the baker because  she always messes up the kitchen. |  [ES] Al conserje no le gusta  el panadero porque  ella siempre desordena la cocina. |  Biased translation, giving ‘‘baker’’ a male inflection, with a mismatched pronoun reference.
The janitor does not like  the pretty baker because  she always messes up the kitchen. |  [ES] Al conserje no le gusta  la panadera bonita porque  ella siempre desordena la cocina. |  Adding a stereotypically female adjective ‘‘fixes’’ the translation.
The counselor asked  the guard a few questions and praised  her for the good work. |  [FR] Le conseiller a posé quelques questions à  la garde et  l’a louée pour le bon travail. |  French uses ‘‘garde’’ for both male and female guards, allowing for a more direct translation from English.
Table 5: Examples of Google Translate’s output for different sentences in the
WinoMT corpus. Words in  blue,  red, and  orange indicate male, female and
neutral entities, respectively.

###  3.4 Human Validation

We estimate the accuracy of our gender bias evaluation method by randomly
sampling 100 instances of all translation systems and target languages,
annotating each sample by two target-language native speakers (resulting in
9,600 human annotations). Each instance conformed to a format similar to that
used by our automatic gender detection algorithm: human annotators were asked
to mark the gender of an entity within a given target-language sentence.
(e.g., see ‘‘el doctor’’ as highlighted in the Spanish sentence in Figure 1).
By annotating at the sentence-level, we can account for both types of possible
errors, i.e., alignment and gender extraction.

We compare the sentence-level human annotations to the output of our automatic
method, and find that the levels of agreement for all languages and systems
were above 85%, with an average agreement on 87% of the annotations. In
comparison, human inter-annotator agreement was 90%, due to noise introduced
by several incoherent translations.

Our errors occur when language-specific idiosyncrasies introduce ambiguity to
the morphological analysis. For example, gender for certain words in Hebrew
cannot be distinguished without diacritics (e.g., the male and female versions
of the word ‘‘baker’’ are spelled identically), and the contracted determiner
in French and Italian (_l’_) is used for both masculine and feminine nouns. In
addition, some languages have only male or female inflections for professions
which were stereotypically associated with one of the genders, for example
‘‘sastre’’ (tailor) in Spanish or ‘‘soldat’’ (soldier) in French, which do not
have female inflections. See Table 5 for detailed examples.

##  4 Discussion

#### Related work

This work is most related to several recent efforts which evaluate MT through
the use of _challenge sets_. Similarly to our use WinoMT, these works evaluate
MT systems (either manually or automatically) on test sets which are specially
created to exhibit certain linguistic phenomena, thus going beyond the
traditional BLEU metric Papineni et al. (2002). These include challenge sets
for language-specific idiosyncrasies Isabelle et al. (2017), discourse
phenomena Bawden et al. (2018), pronoun translation Müller et al. (2018);
Webster et al. (2018), or coreference and multiword expressions Burchardt et
al. (2017).

#### Limitations and future work

While our work presents the first large-scale evaluation of gender bias in MT,
it still suffers from certain limitations which could be addressed in follow
up work. First, like some of the challenge sets discussed above, WinoMT is
composed of synthetic English source-side examples. On the one hand, this
allows for a controlled experiment environment, while, on the other hand, this
might introduce some artificial biases in our data and evaluation. Ideally,
WinoMT could be augmented with natural ‘‘in the wild’’ instances, with many
source languages, all annotated with ground truth entity gender. Second,
similar to any medium size test set, it is clear that WinoMT serves only as a
proxy estimation for the phenomenon of gender bias, and would probably be easy
to overfit. A larger annotated corpus can perhaps provide a better signal for
training. Finally, even though in Section 3.3 we show a very rudimentary
debiasing scheme which relies on oracle coreference system, it is clear that
this is not applicable in a real-world scenario. While recent research has
shown that getting rid of such biases may prove to be very challenging Elazar
and Goldberg (2018); Gonen and Goldberg (2019), we hope that this work will
serve as a first step for developing more gender-balanced MT models.

##  5 Conclusions

We presented the first large-scale multilingual quantitative evidence for
gender bias in MT, showing that on eight diverse target languages, all four
tested popular commercial systems and two recent state-of-the-art academic MT
models are significantly prone to translate based on gender stereotypes rather
than more meaningful context. Our data and code are publicly available at
shorturl.at/dimuD.

## Acknowledgments

We would like to thank Mark Yatskar, Iz Beltagy, Tim Dettmers, Ronan Le Bras,
Kyle Richardson, Ariel and Claudia Stanovsky, and Paola Virga for many
insightful discussions about the role gender plays in the languages evaluated
in this work, as well as the reviewers for their helpful comments.

## References

  * Adler and Elhadad (2006) Meni Adler and Michael Elhadad. 2006.  An unsupervised morpheme-based HMM for Hebrew morphological disambiguation.  In _ACL_.
  * Altinok (2018) Duygu Altinok. 2018.  DEMorphy, German language morphological analyzer.  _CoRR_ , abs/1803.00902.
  * Alvarez-Melis and Jaakkola (2017) David Alvarez-Melis and Tommi S. Jaakkola. 2017.  A causal framework for explaining the predictions of black-box sequence-to-sequence models.  In _EMNLP_.
  * Bawden et al. (2018) Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018.  Evaluating discourse phenomena in neural machine translation.  In _NAACL-HLT_.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. 2016.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  In _NIPS_.
  * Burchardt et al. (2017) Aljoscha Burchardt, Vivien Macketanz, Jon Dehdari, Georg Heigold, Jan-Thorsten Peter, and Philip Williams. 2017.  A linguistic evaluation of rule-based, phrase-based, and neural mt engines.  _The Prague Bulletin of Mathematical Linguistics_ , 108(1):159--170.
  * Corbett (2007) Greville G Corbett. 2007.  Gender and noun classes.
  * Craig (1986) Colette G Craig. 1986.  _Noun Classes and Categorization: Proceedings of a Symposium on Categorization and Noun Classification_ , volume 7.  John Benjamins Publishing Company.
  * Dyer et al. (2013) Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013.  A simple, fast, and effective reparameterization of ibm model 2.  In _HLT-NAACL_.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding back-translation at scale.  _arXiv preprint arXiv:1808.09381_.
  * Elazar and Goldberg (2018) Yanai Elazar and Yoav Goldberg. 2018.  Adversarial removal of demographic attributes from text data.  In _EMNLP_.
  * Font and Costa-Jussà (2019) Joel Escudé Font and Marta R. Costa-Jussà. 2019.  Equalizing gender biases in neural machine translation with word embeddings techniques.  _CoRR_.
  * Gonen and Goldberg (2019) Hila Gonen and Yoav Goldberg. 2019.  Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them.  _HLT-NAACL_.
  * Hawkins (2015) John A Hawkins. 2015.  _A Comparative Typology of English and German: Unifying the Contrasts_.  Routledge.
  * Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  _To appear_.
  * Isabelle et al. (2017) Pierre Isabelle, Colin Cherry, and George F. Foster. 2017.  A challenge set approach to evaluating machine translation.  In _EMNLP_.
  * Korobov (2015) Mikhail Korobov. 2015.  Morphological analyzer and generator for Russian and Ukrainian languages.  In Mikhail Yu. Khachay, Natalia Konstantinova, Alexander Panchenko, Dmitry I. Ignatov, and Valeri G. Labunets, editors, _Analysis of Images, Social Networks and Texts_ , volume 542 of _Communications in Computer and Information Science_ , pages 320--332. Springer International Publishing.
  * Kuczmarski (2018) James Kuczmarski. 2018.  Reducing gender bias in google translate.
  * Levesque (2011) Hector J. Levesque. 2011.  The Winograd schema challenge.  In _AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning_.
  * Mucchi-Faina (2005) Angelica Mucchi-Faina. 2005.  Visible or influential? language reforms and gender (in) equality.  _Social Science Information_ , 44(1):189--215.
  * Müller et al. (2018) Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018.  A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation.  _CoRR_.
  * Ott et al. (2018) Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018.  Scaling neural machine translation.  _arXiv preprint arXiv:1806.00187_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _ACL_.
  * Rudinger et al. (2017) Rachel Rudinger, Chandler May, and Benjamin Van Durme. 2017.  Social bias in elicited natural language inferences.  In _EthNLP@EACL_.
  * Rudinger et al. (2018) Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018.  Gender bias in coreference resolution.  In _NAACL-HLT_.
  * Webster et al. (2018) Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018.  Mind the gap: A balanced corpus of gendered ambiguous pronouns.  _Transactions of the Association for Computational Linguistics_ , 6:605--617.
  * Zhao et al. (2017) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017\.  Men also like shopping: Reducing gender bias amplification using corpus-level constraints.  In _EMNLP_.
  * Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018\.  Gender bias in coreference resolution: Evaluation and debiasing methods.  In _NAACL-HLT_.


Simulation study of  Q statistic with constant weights for testing and estimation of  heterogeneity of  standardized mean differences in meta-analysis

Ilyas Bakbergenuly, David C. Hoaglin,  and Elena Kulinskaya

Cochran's $Q$ statistic is routinely used for testing heterogeneity in meta-analysis. Its expected value  is also used for estimation of between-study variance $\tau^2$. Cochran's $Q$, or $Q_{IV}$, uses estimated inverse-variance weights which makes approximating its distribution rather complicated.

As an alternative,  we are investigating a new $Q$ statistic,  $Q_F$, whose constant weights use only the studies' effective sample sizes. For standardized mean difference as the measure of effect, we study, by simulation, approximations to distributions of $Q_{IV}$ and $Q_F$, as the basis for tests of heterogeneity and for new point and interval estimators of the between-study variance $\tau^2$. These include new DerSimonian-Kacker (2007)-type moment estimators based on the first moment of $Q_F$, and novel median-unbiased estimators of $\tau^2$.

§ MOMENT ESTIMATION OF $Q$

We consider the following random-effects model (REM): For Study $i$ ($i = 1, \ldots, K$),  the estimate of the effect is $\hat\theta_i \sim G(\theta_i, v_i^2)$, where the effect-measure-specific distribution $G$ has mean $\theta_i$ and variance $v_i^2$, and $\theta_i \sim N(\theta, \tau^2)$.
The $\hat\theta_i$ are unbiased estimates of the true conditional effects $\theta_i$, and the $v_i^2 = \var(\hat\theta_i | \theta_i)$ are the corresponding true conditional variances.

Cochran's $Q$ statistic is a weighted sum of squared deviations of the estimated effects $\hat\theta_i$ from their weighted mean $\bar\theta_w = \sum w_i\hat\theta_i/\sum w_i$:

\begin{equation} \label{Q}
Q=\sum w_i(\hat\theta_i-\bar\theta_w)^2.\end{equation}

In <cit.> $w_i$ is the reciprocal of the estimated variance of $\hat{\theta}_i$. In meta-analysis those $w_i$ come from the fixed-effect model. We denote this standard version of $Q$ by $Q_{IV}$.  In our simulations, we examine a version of $Q$, discussed by <cit.>, in which the $w_i$ are arbitrary positive constants. We denote this version by $Q_{F}$.

Define $W = \sum w_i$,  $q_i = w_i / W$, and $\Theta_i = \hat\theta_i - \theta$.  In this notation, and expanding $\bar\theta_w$, Equation (<ref>) can be written  as

\begin{equation} \label{Q1}
Q = W \left[ \sum q_i (1 - q_i) \Theta_i^2 - \sum_{i \not = j} q_i q_j \Theta_i \Theta_j \right].
\end{equation}

Under the above REM, it is straightforward to obtain the first moment of $Q$ as

\begin{equation} \label{M1Q}
\e(Q) = W \left[ \sum q_i (1 - q_i) \var(\Theta_i) \right] = W \left[ \sum q_i (1 - q_i) (\e(v_i^2) + \tau^2) \right].
\end{equation}

Rearranging the terms gives the moment-based estimator of $\tau^2$

\begin{equation} \label{tau_DSK}
\hat\tau^2_{M} = \frac{Q / W - \sum q_i (1 - q_i) \e(v_i^2)} {\sum q_i (1 - q_i)}.
\end{equation}

This equation is similar to Equation (6) in <cit.>; they use the estimate $s_i^2$ instead of $\e(v_i^2)$.  We study this estimator of $\tau^2$, which uses the conditional estimated variances, $\hat v_i^2$, in the sample-size-based weights, and denote it by $\hat\tau^2_{SSC}$ or simply SSC. We also study the corresponding estimator that uses the unconditional estimated variances, $\hat\e(v_i^2) $, which we denote by SSU.

§ APPROXIMATIONS FOR THE DISTRIBUTION OF $Q$

We also study approximations for the distributions of $Q$ statistics. For SMD, $Q_F$ is a quadratic form in $t$ variates.  The  <cit.> algorithm for the exact distribution of a quadratic form in normal variables  may provide a satisfactory approximation, especially for larger sample sizes. To apply it, we plug in estimated variances. We investigate the quality of that approximation, which we denote by F SW, and the two-moment approximation (M2 SW) based on the gamma distribution.

The null distribution of $Q_{IV}$ is usually approximated by the chi-square distribution with $K - 1$ degrees of freedom.  For mean difference and standardized mean difference, however, this approximation is not accurate for small sample sizes (<cit.>). For SMD <cit.> provided an improved approximation to the null distribution of $Q_{IV}$ based on a chi-square distribution with degrees of freedom equal to the estimate of the corrected first moment; we denote this approximation by KDB. <cit.> used the Farebrother approximation as the “exact” distribution of $Q_{IV}$. We denote this approximation by BJ.
When $\tau^2 = 0$, the BJ approximation to the distribution of $Q_{IV}$ reduces to the $\chi^2_{K - 1}$  distribution. For comparison, our simulations include these three approximations.

§ OTHER POINT AND INTERVAL ESTIMATORS OF $\TAU^2$

Distributions of the $Q$ statistics depend on the $\theta_i$ and the $v_i^2$ and on the between-study variance $\tau^2$. We denote such a distribution by $F(\cdot | \theta_i, v_i^2,\;\tau^2)$. In practice, an approximation uses estimated parameters, and we calculate its value at the observed value of $Q$, obtaining the approximate upper-tail p-value
$p(Q | \tau^2) = 1 - F(Q | \hat\theta_i, \hat v_i^2,\;\tau^2)$.

A confidence interval for $\tau^2$ at confidence level $1 - \alpha$ can be obtained as
$$\{\tau^2: p(Q | \tau^2) \in [\alpha/2, 1 - \alpha/2]\}.$$

Similarly, a point estimator of $\tau^2 $ can be found as
$$\hat\tau^2_{med} = \min(0,\; \{\tau^2: F(Q|\tau^2)=0.5\})$$

Our simulations for SMD use the Farebrother approximation to the distribution of $Q$ in two versions. One uses the conditional variances of the $\hat\theta_i$ (i.e., $\hat v_i^2$), and the other uses the unconditional variances (i.e.,  $\hat \e(v_i^2)$).  The corresponding point estimators of $\tau^2$, based on the median, are denoted by SMC and SMU, respectively.  The corresponding intervals are denoted by FPC and FPU. The P in their names reminds us that they are profile intervals.

For comparison our simulations include four point estimators that use inverse-variance weights: DerSimonian-Laird (DL), restricted maximum likelihood (REML), Mandel-Paule (MP), and an estimator (KDB) based on the work of <cit.>, which uses an improved first moment of $Q$.

We also include three interval estimators: the Q-profile (QP) interval, the profile-likelihood (PL) interval, and the KDB interval, which is based on the chi-square distribution with the corrected first moment.

§ STUDY-LEVEL ESTIMATION OF STANDARDIZED MEAN DIFFERENCE

Each of the $K$ studies consists of two arms, treatment (T) and control (C), with sample sizes $n_{iT}$ and $n_{iC}$. The total sample size in Study $i$ is $n_i = n_{iT} + n_{iC}$, and the ratio of the control sample size to the total is $f_i = n_{iC}/n_{i}$. We define the effective sample size in Study $i$ as $\tilde{n}_i = n_{iC} n_{iT} / n_i$.

The subject-level data in each arm are assumed to be normally distributed with means $\mu_{iT}$ and $\mu_{iC}$ and equal variances $\sigma_{i}^2$. The sample means are $\bar{x}_{ij}$, and the sample variances are $s^2_{ij}$, for $i = 1,\ldots, K$ and $j = C$ or $T$.

The standardized mean difference effect measure is

$$\delta_{i} = \frac{\mu_{iT} - \mu_{iC}} {\sigma_{i}}.$$

The unbiased estimator of $\delta_i$ is Hedges's $g$:

\begin{equation} \label{eq:delta}
{g}_i = J(m_i) \frac{\bar{x}_{iT} - \bar{x}_{iC}} {s_{i}},
\end{equation}

where the standard deviation, $\sigma_i$, is estimated by the square root of the pooled sample variance $s_i^2$, $m_{i} = n_{iT} + n_{iC} - 2$, and the factor
$J(m) = {\Gamma \left( \frac{m} {2} \right) } / {\sqrt{\frac{m}{2}}\Gamma \left( \frac{m - 1} {2} \right) }$
corrects for bias.

For the variance of $g_i$ we use the unbiased estimator

\begin{equation} \label{eq:g_var}
v_{i}^2 = \frac{n_{iT} + n_{iC}} {n_{iT } n_{iC}} + \left(1 - \frac{(m_{i} - 2)} {m_{i}J(m_{i})^2} \right) g^2_{i},
\end{equation}

derived by <cit.>. The literature contains several other estimators of the variance of $g_i$ and its biased counterpart, $d_i$. <cit.> provide a comprehensive assessment.

The sample SMD ${g}_i$ has a scaled non-central $t$-distribution with non-centrality parameter
$\gamma_i = \tilde{n}_{i}^{1/2} \delta_i$ <cit.>:

\begin{equation} \label{eq:g_dist}
\sqrt{\tilde{n}_i} J(m_i)^{-1} g_i \sim t_{m_i} (\tilde{n}_{i}^{1/2} \delta_i).
\end{equation}

§ SKETCH OF THE SIMULATIONS

The design of the simulations follows that described in <cit.>. Briefly,
we vary five parameters: the overall true SMD ($\delta$), the between-studies variance ($\tau^2$), the number of studies ($K$), the studies' total sample size (both equal, $n$, and unequal, $\bar{n}$), and the proportion of observations in the Control arm ($f$). Table <ref> lists the values of each parameter. We use a total of $10,000$ repetitions for each combination of parameters.

We generate the true effect sizes $\delta_{i}$ from a normal distribution:  $\delta_{i} \sim N(\delta, \tau^2)$. We generate the values of Hedges's estimator ${g}_{i}$ directly from the appropriately scaled non-central $t$-distribution with non-centrality parameter $\tilde{n}_{i}^{1/2} \delta_i$:

\begin{equation}
\sqrt{\tilde{n}_i} J(m_i)^{-1} g_i \sim t_{m_i} (\tilde{n}_{i}^{1/2} \delta_i).
\end{equation}

Values of parameters in the simulations for SMD

Parameter     Equal study sizes     Unequal study sizes

$K$ (number of studies)     5, 10, 30     5, 10, 30

$n$ or $\bar{n}$  (average (individual) study size —      20, 40, 100, 250,640, 1000     30 (12,16,18,20,84),

total of the two arms)     30, 50, 60, 70     60 (24,32,36,40,168),

For  $K = 10$ and $K = 30$, the same set         100 (64,72,76,80,208),

of unequal study sizes is used        160 (124,132,136,140,268)

twice or six times, respectively.

$f$ (proportion of each study in the control arm)     1/2, 3/4    1/2, 3/4

$\delta$ (true value of the SMD)     0, 0.2, 0.5, 1, 2     0, 0.2, 0.5, 1, 2

$\tau^{2}$ (variance of random effects)     0, 0.5, 1, 1.5, 2, 2.5     0, 0.5, 1, 1.5, 2, 2.5

§ SUMMARY

In estimating $\tau^2$, our moment estimator based on $Q_F$ with effective-sample-size weights is almost unbiased, the Mandel-Paule estimator has some negative bias in some situations, and the DerSimonian-Laird and restricted-maximum-likelihood estimators have considerable negative bias.

All 95% interval estimators have coverage that is too high when $\tau^2 = 0$, but otherwise the Q-profile interval performs very well.

An approximation based on an algorithm of Farebrother follows both the null and the alternative distributions of $Q_F$ reasonably well, whereas the usual chi-square approximation for the null distribution of $Q_{IV}$ and Biggerstaff-Jackson approximation to its alternative distribution are poor.

The test for heterogeneity based on $Q_F$ and F SW has error rates that somewhat exceed the nominal 5%, and the test based on $Q_{IV}$ and the chi-square approximation has error rates that are noticeably too low.

When $\tau^2 > 0$, F SW provides robust empirical levels at all values of $\tau^2$ and $\delta$. Those levels, however, are somewhat higher than the nominal .05 (for larger $K$). M2 SW has below-nominal levels, which decrease further for larger $\delta$ but do not depend on $\tau^2$. The levels of the BJ approximation are even lower, and they decrease further as $\tau^2$ increases.

§ DETAILED RESULTS

The appendices contain plots of the results of the simulations:

* Appendix A. Plots for bias and coverage of estimators of $\tau^2$

* Appendix B. Plots of error in approximations for the distribution of $Q$

* Appendix C. Empirical level of the test for heterogeneity ($\tau^2$ = 0 versus $\tau^2 > 0$) based on approximations for the distribution of $Q$, plotted vs sample size

* Appendix D: Empirical level of the test for heterogeneity ($\tau^2 \leq \tau_{0}^2$ versus $\tau^2 > \tau_{0}^2$) based on approximations for the distribution of $Q$, plotted vs $\tau_{0}^2$

* Appendix E. Power of the test for heterogeneity ($\tau^2 = 0$ versus $\tau^2 > 0$) based on approximations for the distribution of $Q$

§ APPENDICES

* Appendix A. Plots for bias and coverage of estimators of $\tau^2$

* Appendix B. Plots of error in approximations for the distribution of $Q$

* Appendix C. Empirical level of the test for heterogeneity ($\tau^2$ = 0 versus $\tau^2 > 0$) based on approximations for the distribution of $Q$, plotted vs sample size

* Appendix D: Empirical level of the test for heterogeneity ($\tau^2 \leq \tau_{0}^2$ versus $\tau^2 > \tau_{0}^2$) based on approximations for the distribution of $Q$, plotted vs $\tau_{0}^2$

* Appendix E. Power of the test for heterogeneity ($\tau^2 = 0$ versus $\tau^2 > 0$) based on approximations for the distribution of $Q$

§ A. BIAS AND COVERAGE OF ESTIMATORS OF $\TAU^2$

§ BIAS OF ESTIMATORS OF $\TAU^2$

Each figure corresponds to a value of $\delta$, a value of $f$, and either equal or unequal sample sizes.

For each combination of a value of $n$ or $\bar{n}$ and a value of $K$, a panel plots bias of estimators of $\tau^2$ versus $\tau^2$.

The point estimators of $\tau^2$ are

* SSC (constant effective-sample-size weights and conditional variances)

* SSU (constant effective-sample-size weights and unconditional variances)

* SMC (the median $\tau^2$ value in Farebrother approximation with conditional moments)

* SMU (the median $\tau^2$ value in Farebrother approximation with unconditional moments)

* DL (standard DerSimonian-Laird: inverse-variance weights)

* REML (restricted maximum likelihood)

* MP (Mandel-Paule)

* KDB (inverse-variance method based on corrected first moment of null distribution of $Q$)

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 0$, $f = .5$, and equal sample sizes

Bias of estimators of the between-studies variance $\tau^2 $ for $\delta = 0$, $f = .5$, and unequal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 0.2$, $f = .5$, and equal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 0.2$, $f = .5$, and unequal sample sizes

Bias of estimators of the between-studies variance $\tau^2 $ for $\delta = 0.5$, $f = .5$, and equal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta= 0.5$, $f = .5$, and unequal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 1$, $f = .5$, and equal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 1$, $f = .5$, and unequal sample sizes

Bias of estimators of the between-studies variance $\tau^2 $ for $\delta = 2$, $f = .5$, and equal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 2$, $f = .5$, and unequal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 0$, $f = .75$, and equal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 0$, $f = 0.75$, and unequal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 0.2$, $f = .75$, and equal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 0.2$, $f = .75$, and unequal sample eizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 0.5$, $f = .75$, and equal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 0.5$, $f = .75$, and unequal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 1$, $f = .75$, and equal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 1$, $f = .75$, and unequal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 2$, $f = .75$, and equal sample sizes

Bias of estimators of the between-studies variance $\tau^2$ for $\delta = 2$, $f = .75$, and unequal sample sizes

§ COVERAGE OF INTERVAL ESTIMATORS OF THE BETWEEN-STUDIES VARIANCE ($\TAU^2$)

The interval estimators of $\tau^2$ are

* QP (Q profile interval)

* PL (Profile likelihood interval)

* KDB (inverse-variance method based on corrected first moment of null distribution of $Q$)

* FPC (profile based on Farebrother approximation with conditional moments)

* FPU (profile based on Farebrother approximation with unconditional moments)

Coverage of $\tau^2$ for $\delta = 0$, $f = .5$, and equal sample sizes

Coverage of $\tau^2$ for $\delta = 0$, $f = .5$, and unequal sample sizes

Coverage of $\tau^2$ for $\delta = 0.2$, $f = .5$, and equal sample sizes

Coverage of $\tau^2$ for $\delta = 0.2$, $f = .5$, and unequal sample sizes

Coverage of $\tau^2$ for $\delta = 0.5$, $f = .5$, and equal sample sizes

Coverage of $\tau^2$ for $\delta = 0.5$, $f = .5$, and unequal sample sizes

Coverage of $\tau^2$ for $\delta = 1$, $f = .5$, and equal sample sizes

Coverage of $\tau^2$ for $\delta = 1$, $f = .5$, and unequal sample sizes

Coverage of $\tau^2$ for $\delta = 2$, $f = .5$, and equal sample sizes

Coverage of $\tau^2$ for $\delta = 2$, $f = .5$, and unequal sample sizes

Coverage of $\tau^2$ for $\delta = 0$, $f = .75$, and equal sample sizes

Coverage of $\tau^2$ for $\delta = 0$, $f = .75$, and unequal sample sizes

Coverage of $\tau^2$ for $\delta = 0.2$, $f = .75$, and equal sample sizes

Coverage of $\tau^2$ for $\delta = 0.2$, $f = .75$, and unequal sample sizes

Coverage of $\tau^2$ for $\delta = 0.5$, $f = .75$, and equal sample sizes

Coverage of $\tau^2$ for $\delta = 0.5$, $f = .75$, and unequal sample sizes

Coverage of $\tau^2$ for $\delta = 1$, $f = .75$, and equal sample sizes

Coverage of $\tau^2$ for $\delta = 1$, $f = .75$, and unequal sample sizes

Coverage of $\tau^2$ for $\delta = 2$, $f = .75$, and equal sample sizes

Coverage of $\tau^2$ for $\delta = 2$, $f = .75$, and unequal sample sizes

§ B. ERROR IN APPROXIMATIONS FOR NULL DISTRIBUTION OF $Q$

Each figure corresponds to a value of $\delta$, a value of $f$, and either equal or unequal sample sizes.

For each combination of a value of $n$ or $\bar{n}$ and a value of $K$, a panel plots the error, $\hat{P}(\hat{F}(Q) > 1 - p) - p$, for the upper-tail probabilities $p$ = .001, .0025, .005, .01, .025, .05, .1, .25, .5, .75, .9, .95, .975, .99, .995, .9975, .999.

The approximations for the distribution of $Q$ are

* F SW (Farebrother approximation, effective-sample-size weights)

* M2 SW (Two-moment approximation, effective-sample-size weights)

* $\chi_{K - 1}^2$ (Chi-square, IV weights)

* KDB (Chi-square approximation based on corrected first moment, IV weights)

Approximation error for $\delta = 0$, $f = .5$, and equal sample sizes

Approximation error for $\delta = 0$, $f = .5$, and unequal sample sizes

Approximation error for $\delta = 0.2$, $f = .5$, and equal sample sizes

Approximation error $\delta = 0.2$, $f = .5$, and unequal sample sizes

Approximation error $\delta = 0.5$, $f = .5$, and equal sample sizes

Approximation error for $\delta = 0.5$, $f = .5$, and unequal sample sizes

Approximation error for $\delta = 1$, $f = .5$, and equal sample sizes

Approximation error for $\delta = 1$, $f = .5$, and unequal sample sizes

Approximation error for $\delta = 2$, $f = .5$, and equal sample sizes

Approximation error for $\delta = 2$, $f = .5$, and unequal sample sizes

Approximation error for $\delta = 0$, $f = .75$, and equal sample sizes

Approximation error for $\delta = 0$, $f = .75$, and unequal sample sizes

Approximation error for $\delta = 0.2$, $f = .75$, and equal sample sizes

Approximation error for $\delta = 0.2$, $f = .75$, and unequal sample sizes

Approximation error for $\delta = 0.5$, $f = .75$, and equal sample sizes

Approximation error for $\delta = 0.5$, $f = .75$, and unequal sample sizes

Approximation error for $\delta = 1$, $f = .75$, and equal sample sizes

Approximation error for $\delta = 1$, $f = .75$, and unequal sample sizes

Approximation error for $\delta = 2$, $f = .75$, and equal sample sizes

Approximation error for $\delta = 2$, $g = .75$, and unequal sample sizes

§ C. EMPIRICAL LEVEL OF THE TEST FOR HETEROGENEITY ($\TAU^2$ = 0 VERSUS $\TAU^2 > 0$) BASED ON APPROXIMATIONS FOR THE DISTRIBUTION OF $Q$, PLOTTED VS SAMPLE SIZE

In sets of figures for $f = .5$ and $f = .75$, each figure corresponds to a value of $\alpha$ (= .001, .005, .01, .05) and either equal sample sizes or unequal sample sizes. (For all figures, $\tau^2 = 0$.)

For each combination of a value of $\delta$ (omitting $\delta = 0.2$) and a value of $K$, a panel plots the empirical level versus $n$ or $\bar{n}$.

The approximations for the distribution of $Q$ are

* F SW (Farebrother approximation, effective-sample-size weights)

* M2 SW (Two-moment approximation, effective-sample-size weights)

* $\chi^2_{K - 1}$ (Chi-square, IV weights)

* KDB (Chi-square approximation based on corrected first moment, IV weights)

§.§ C1. $f = .5$

Empirical level at $\alpha = .001$ of test for heterogeneity vs sample size

Empirical level at $\alpha = .001$ of test for heterogeneity vs average sample size

Empirical level at $\alpha = .005$ of test for heterogeneity vs sample size

Empirical level at $\alpha = .005$ of test for heterogeneity vs average sample size

Empirical level at $\alpha = .01$ of test for heterogeneity vs sample size

Empirical level at $\alpha = .01$ of test for heterogeneity vs average sample size

Empirical level at $\alpha = .05$ of test for heterogeneity vs sample size

Empirical level at $\alpha = .05$ of test for heterogeneity vs average sample size

§.§ C2. $f = .75$

Empirical level at $\alpha = .001$ of test for heterogeneity vs sample size

Empirical level at $\alpha = .001$ of test for heterogeneity vs average sample size

Empirical level at $\alpha = .005$ of test for heterogeneity vs sample size

Empirical level at $\alpha = .005$ of test for heterogeneity vs average sample size

Empirical level at $\alpha = .01$ of test for heterogeneity vs sample size

Empirical level at $\alpha = .01$ of test for heterogeneity vs average sample size

Empirical level at $\alpha = .05$ of test for heterogeneity vs sample size

Empirical level at $\alpha = .05$ of test for heterogeneity vs average sample size

§ D. EMPIRICAL LEVEL OF THE TEST FOR HETEROGENEITY ($\TAU^2 \LEQ \TAU_{0}^2$ VERSUS $\TAU^2 > \TAU_{0}^2$) BASED ON APPROXIMATIONS FOR THE DISTRIBUTION OF $Q$, PLOTTED VS $\TAU_{0}^2$

In sets of figures for $\alpha = .01$ and $\alpha =  .05$, each figure corresponds to a value of $\delta$, a value of $f$, and either equal sample sizes or unequal sample sizes.

For each combination of a value of $n$ or $\bar{n}$ and a value of $K$, a panel plots the empirical level versus $\tau_{0}^2$.

The approximations for the distribution of $Q$ are

* F SW (Farebrother approximation, effective-sample-size weights)

* M2 SW (Two-moment approximation, effective-sample-size weights)

* BJ (Biggerstaff and Jackson approximation, IV weights)

§.§ D1. Empirical level at $\alpha = .01$

Empirical level for $\delta = 0$, $f = .5$, and equal sample sizes

Empirical level for $\delta = 0$, $f = .5$, and unequal sample sizes

Empirical level for $\delta = 0.2$, $f = .5$, and equal sample sizes

Empirical level for $\delta = 0.2$, $f = .5$, and unequal sample sizes

Empirical level for $\delta = 0.5$, $f = .5$, and equal sample sizes

Empirical level for $\delta = 0.5$, $f = .5$, and unequal sample sizes

Empirical level for $\delta = 1$, $f = .5$, and equal sample sizes

Empirical level for $\delta = 1$, $f = .5$, and unequal sample sizes

Empirical level for $\delta = 2$, $f = .5$, and equal sample sizes

Empirical level for $\delta = 2$, $f = .5$, and unequal sample sizes

Empirical level for $\delta = 0$, $f = .75$, and equal sample sizes

Empirical level for $\delta = 0$, $f = .75$, and unequal sample sizes

Empirical level for $\delta = 0.2$, $f = .75$, and equal sample sizes

Empirical level for $\delta = 0.2$, $f = .75$, and unequal sample sizes

Empirical level for $\delta = 0.5$, $f = .75$, and equal sample sizes

Empirical level for $\delta = 0.5$, $f = .75$, and unequal sample sizes

Empirical level for $\delta = 1$, $f = .75$, and equal sample sizes

Empirical level for $\delta = 1$, $f = .75$, and unequal sample sizes

Empirical level for $\delta = 2$, $f = .75$, and equal sample sizes

Empirical level for $\delta = 2$, $f = .75$, and unequal sample sizes

§.§ D2. Empirical level at $\alpha = .05$ (only $f = .5$)

Empirical level for $\delta = 0$, $f = .5$, and equal sample sizes

Empirical level for $\delta = 0$, $f = .5$, and unequal sample sizes

Empirical level for $\delta = 0.2$, $f = .5$, and equal sample sizes

Empirical level for $\delta = 0.2$, $f = .5$, and unequal sample sizes

Empirical level for $\delta = 0.5$, $f = .5$, and equal sample sizes

Empirical level for $\delta = 0.5$, $f = .5$, and unequal sample sizes

Empirical level for $\delta = 1$, $f = .5$, and equal sample sizes

Empirical level for $\delta = 1$, $f = .5$, and unequal sample sizes

Empirical level for $\delta = 2$, $f = .5$, and equal sample sizes

Empirical level for $\delta = 2$, $f = .5$, and unequal sample sizes

§ E. POWER OF THE TEST FOR HETEROGENEITY ($\TAU^2 = 0$ VERSUS $\TAU^2 > 0$) BASED ON APPROXIMATIONS FOR THE DISTRIBUTION OF $Q$

In sets of figures for $\alpha = .01$ and $\alpha = .05$, each figure corresponds to a value of $\delta$, a value of $f$, and either equal sample sizes or unequal sample sizes.

For each combination of a value of $n$ or $\bar{n}$ and a value of $K$, a panel plots power versus $\tau^2$.

The approximations for the distribution of $Q$ are

* F SW (Farebrother approximation, effective-sample-size weights)

* M2 SW (Two-moment approximation, effective-sample-size weights)

* $\chi_{K-1}^2$ (Chi-square, IV weights)

* KDB (Chi-square approximation based on corrected first moment, IV weights)

§.§ E1. Power when $\alpha = .01$

Power for $\delta = 0$, $f = .5$, and equal sample sizes

Power for $\delta = 0$, $f = .5$, and unequal sample sizes

Power for $\delta = 0.2$, $f = .5$, and equal sample sizes

Power for $\delta = 0.2$, $f = .5$, and unequal sample sizes

Power for $\delta = 0.5$, $f = .5$, and equal sample sizes

Power for $\delta = 0.5$, $f = .5$, and unequal sample sizes

Power for $\delta = 1$, $f = .5$, and equal sample sizes

Power for $\delta = 1$, $f = .5$, and unequal sample sizes

Power for $\delta = 2$, $f = .5$, and equal sample sizes

Power for $\delta = 2$, $f = .5$, and unequal sample sizes

Power for $\delta = 0$, $f = .75$, and equal sample sizes

Power for $\delta = 0$, $f = .75$, and unequal sample sizes

Power for $\delta = 0.2$, $f = .75$, and equal sample sizes

Power for $\delta = 0.2$, $f = .75$, and unequal sample sizes

Power for $\delta = 0.5$, $f = .75$, and equal sample sizes

Power for $\delta = 0.5$, $f = .75$, and unequal sample sizes

Power for $\delta = 1$, $f = .75$, and equal sample sizes

Power for $\delta = 1$, $f = .75$, and unequal sample sizes

Power for $\delta = 2$, $f = .75$, and equal sample sizes

Power for $\delta = 2$, $f = .75$, and unequal sample sizes

§.§ E2. Power when $\alpha = .05$

Power for $\delta = 0$, $f = .5$, and equal sample sizes

Power for $\delta = 0$, $f = .5$, and unequal sample sizes

Power for $\delta = 0.2$, $f = .5$, and equal sample sizes

Power for $\delta = 0.2$, $f = .5$, and unequal sample sizes

Power for $\delta = 0.5$, $f = .5$, and equal sample sizes

Power for $\delta = 0.5$, $f = .5$, and unequal sample sizes

Power for $\delta = 1$, $f = .5$, and equal sample sizes

Power for $\delta = 1$, $f = .5$, and unequal sample sizes

Power for $\delta = 2$, $f = .5$, and equal sample sizes

Power for $\delta = 2$, $f = .5$, and unequal sample sizes

Power for $\delta = 0$, $f = .75$, and equal sample sizes

Power for $\delta = 0$, $f = .75$, and unequal sample sizes

Power for $\delta = 0.2$, $f = .75$, and equal sample sizes

Power for $\delta = 0.2$, $f = .75$, and unequal sample sizes

Power for $\delta = 0.5$, $f = .75$, and equal sample sizes

Power for $\delta = 0.5$, $f = .75$, and unequal sample sizes

Power for $\delta = 1$, $f = .75$, and equal sample sizes

Power for $\delta = 1$, $f = .75$, and unequal sample sizes

Power for $\delta = 2$, $f = .75$, and equal sample sizes

Power for $\delta = 2$, $f = .75$, and unequal sample sizes