# Visualizing Tests for Equality of Covariance Matrices

Michael Friendly and Matthew Sigal

###### Abstract

This paper explores a variety of topics related to the question of testing the
equality of covariance matrices in multivariate linear models, particularly in
the MANOVA setting. The main focus is on graphical methods that can be used to
address the evaluation of this assumption. We introduce some extensions of
data ellipsoids, hypothesis-error (HE) plots and canonical discriminant plots
and demonstrate how they can be applied to the testing of equality of
covariance matrices. Further, a simple plot of the components of Box’s M test
is proposed that shows _how_ groups differ in covariance and also suggests
other visualizations and alternative test statistics. These methods are
implemented and freely available in the heplots and candisc packages for R.
Examples from the paper are available in supplementary materials.

Keywords— Box M test; HE plots; MANOVA; graphics

##  1 Introduction

> _To make the preliminary test on variances is rather like putting to sea in
> a rowing boat to find out whether conditions are sufficiently calm for an
> ocean liner to leave port._ — G. E. P. Box (1953)

This paper concerns the extension of tests of homogeneity of variance from the
classical univariate ANOVA setting to the analogous multivariate (MANOVA)
setting. Such tests are a routine but important aspect of data analysis, as
particular violations can drastically impact model estimates (Lix & Keselman,
1996). In the multivariate context, the following questions and topics are of
main interest here:

  * •

Visualization: How can we visualize differences among group variances and
covariance matrices, perhaps in a way that is analogous to what is done to
visualize differences among group means? Multivariate linear models (MLMs)
present additional challenges for data visualization because we often want to
see the effects for a collection of response variables _simultaneously_ ,
which pushes the boundaries of typical graphical displays. As will be
illustrated, differences among covariance matrices can be comprised of spread
in overall size (“scatter”) and shape (“orientation”). When there are more
than a few response variables, what low-dimensional views can show the most
interesting properties related to the equality of covariance matrices?

  * •

Other test statistics: Test statistics for MANOVA and for equality of
covariance matrices are based on properties of eigenvalues of various
matrices. Available tests statistics for mean differences suggest alternatives
for the question of equality of covariance matrices.

The following subsections provide a capsule summary of the issues in this
topic. Most of the discussion is couched in terms of a one-way design for
simplicity, but the same ideas can apply to two-way (and higher) designs,
where a “group” factor is defined as the product combination (interaction) of
two or more factor variables. When there are also numeric covariates, this
topic can also be extended to the multivariate analysis of covaraiance
(MANCOVA) setting. This can be accomplished by applying these techniques to
the residuals from predictions by the covariates alone.

###  1.1 Homogeneity of Variance in Univariate ANOVA

In classical (Gaussian) univariate ANOVA models, the main interest is
typically on tests of mean differences in a response $y$ according to one or
more factors. The validity of the typical $F$ test, however, relies on the
assumption of _homogeneity of variance_ : all groups have the same (or
similar) variance,

$\sigma_{1}^{2}=\sigma_{2}^{2}=\cdots=\sigma_{g}^{2}\>.$

It turns out that the $F$ test for differences in means is relatively robust
to violation of this assumption (Harwell, Rubinstein, Hayes, & Olds, 1992), as
long as the group sizes are roughly equal.111If group sizes are greatly
unequal and homogeneity of variance is violated, then the $F$ statistic is too
liberal ($p$ values too large) when large sample variances are associated with
small group sizes. Conversely, the $F$ statistic is too conservative if large
variances are associated with large group sizes.

A variety of classical test statistics for homogeneity of variance are
available, including Hartley’s $F_{max}$ (Hartley, 1950), Cochran’s _C_
(Cochran, 1941),and Bartlett’s test (Bartlett, 1937), but these have been
found to have terrible statistical properties (Rogan & Keselman, 1977), which
prompted Box’s famous quote.

Levene (1960) introduced a different form of test, based on the simple idea
that when variances are equal across groups, the average absolute values of
differences between the observations and group means will also be equal, i.e.,
substituting an $L_{1}$ norm for the $L_{2}$ norm of variance. In a one-way
design, this is equivalent to a test of group differences in the means of the
auxilliary variable $z_{ij}=|y_{ij}-\bar{y}_{i}|$.

More robust versions of this test were proposed by Brown & Forsythe (1974).
These tests substitute the group mean by either the group median or a trimmed
mean in the ANOVA of the absolute deviations, and should be almost always
preferred to Levene’s version (which unfortunately was adopted as the default
in some software such as SPSS). See Conover, Johnson, & Johnson (1981) for an
early review and Gastwirth, Gel, & Miao (2009) for a general discussion of
these tests. In what follows, we refer to this class of tests as “Levene-type”
tests and suggest a multivariate extension described in the supplementary
appendix (Section 6).

###  1.2 Homogeneity of variance in MANOVA

MANOVA focuses on testing differences among mean vectors,

$H_{0}:\bm{\mu}_{1}=\bm{\mu}_{2}=\cdots=\bm{\mu}_{g}\>.$

However, the standard test statistics (Wilks’ Lambda, Hotelling-Lawley trace,
Pillai-Bartlett trace, Roy’s maximum root) rely upon the analogous assumption
that the within-group covariance matrices for all groups are equal,

$\mathbf{\Sigma}_{1}=\mathbf{\Sigma}_{2}=\cdots=\mathbf{\Sigma}_{g}\>.$

In the multivariate setting, there has been considerable attention to the
sensitivity of these tests to both non-normality and lack of equality of
covariance matrices, largely through simulation studies (e.g., Finch & French,
2013; Hakstian, Roed, & Lind, 1979). Most of these have been conducted in the
simple case of two-group designs (where Hotelling’s $T^{2}$ is the equivalent
of all the standard tests) or in one-way designs. A classic study in this area
is Olson (1974), that recommended:

> for protection against nonnormality and heterogeneity of covariance
> matrices, the largest-root test should be avoided, while the Pillai-Bartlett
> trace test may be recommended as the most robust of the MANOVA tests, with
> adequate power to detect true differences in a variety of situations (p.
> 894).

We mention in passing that, with a burgeoning interesting in robust methods
over the last few decades, there have been a variety of proposals for how to
conduct robust tests of differences on mean vectors, mostly in the one-way
MANOVA setting (e.g., Aelst & Willems, 2011; Todorov & Filzmoser, 2010).
Generally speaking, these involve using more robust alternatives for mean
vectors (medians, trimmed means, rank-based methods) and for covariance
matrices (e.g., minimum covariance determinant (MCD) and minimum volume
ellipsoid (MVE)).

Yet, there has not been as much attention paid to the second-order problem of
assessing equality of covariance matrices. Box’s M test, described below,
remains the main procedure readily available in statistical software for this
problem. The properties and alternatives to Box’s test have not been widely
studied (some exceptions are O’Brien, 1992; Tiku & Balakrishnan, 1984).

However, beyond issues of robustness, the question of equality of covariance
matrices is often of general interest itself. For instance, variability is
often an important issue in studies of strict equivalence in laboratories
comparing across multiple patient measurements and in other applied contexts
(see Gastwirth et al., 2009 for other exemplars). Moreover the outcome of such
tests often have important consequences for the details of a main method of
analysis. Just as the Welsh $t$-test (Welch, 1947) is now commonly used and
reported for a two-group test of differences in means under unequal variances,
a preliminary test of equality of covariance matrices is often used in
discriminant analysis to decide whether linear (LDA) or quadratic discriminant
analysis (QDA) should be applied in a given problem. In such cases, the data
at hand should inform the choice of statistical analysis to utilize.

###  1.3 Assessing heterogeneity of covariance matrices: Box’s M test

Box (1949) proposed the following likelihood-ratio test (LRT) statistic for
testing the hypothesis of equal covariance matrices,

$M=(N-g)\ln|\mathbf{S}_{p}|-\sum_{i=1}^{g}(n_{i}-1)\ln|\mathbf{S}_{i}|\>,$ (1)

where $N=\sum n_{i}$ is the total sample size and
$\mathbf{S}_{p}=(N-g)^{-1}\sum_{i=1}^{g}(n_{i}-1)\mathbf{S}_{i}$ is the pooled
covariance matrix. $M$ can thus be thought of as a ratio of the determinant of
the pooled $\mathbf{S}_{p}$ to the geometric mean of the determinants of the
separate $\mathbf{S}_{i}$.

In practice, there are various transformations of the value of $M$ to yield a
test statistic with an approximately known distribution (Timm, 1975). Roughly
speaking, when each $n_{i}>20$, a $\chi^{2}$ approximation is often used;
otherwise an $F$ approximation is known to be more accurate.

Asymptotically, $-2\ln(M)$ has a $\chi^{2}$ distribution. The $\chi^{2}$
approximation due to Box (1949, 1950) is that

$X^{2}=-2(1-c_{1})\ln(M)\quad\sim\quad\chi^{2}_{df}$

with $df=(g-1)p(p+1)/2$ degrees of freedom, and a bias correction constant:

$c_{1}=\left(\sum_{i}\frac{1}{n_{i}-1}-\frac{1}{N-g}\right)\frac{2p^{2}+3p-1}{6(p+1)(g-1)}\>.$

In this form, Bartlett’s test for equality of variances in the univariate case
is the special case when there is only one response variable, so Bartlett’s
test is sometimes used as univariate follow-up to determine which response
variables show heterogeneity of variance.

Yet, like its univariate counterpart, Box’s test is well-known to be highly
sensitive to violation of (multivariate) normality and the presence of
outliers. For example, Tiku & Balakrishnan (1984) concluded from simulation
studies that the normal-theory LRT provides poor control of Type I error under
even modest departures from normality. O’Brien (1992) proposed some robust
alternatives, and showed that Box’s normal theory approximation suffered both
in controlling the null size of the test and in power. Zhang & Boos (1992)
also carried out simulation studies with similar conclusions and used
bootstrap methods to obtain corrected critical values.

###  1.4 Visualizing heterogeneity

The goal of this article is to use the above background as a platform for
discussing approaches to visualizing and testing the heterogeneity of
covariance matrices in multivariate designs. While researchers often rely on a
single number to determine if their data have met a particular threshold, such
compression will often obscure interesting information, particularly when a
test concludes that differences exist, and one is left to wonder “why?”. It is
within this context where, again, visualizations often reign supreme. In fact,
we find it somewhat surprising that this issue has not been addressed before
graphically in any systematic way.

In this article, we propose three visualization-based approaches to questions
of heterogeneity of covariance in MANOVA designs: (a) direct visualization of
the information in the $\bm{S}_{i}$ and $\bm{S}_{p}$ using _data ellipsoids_
to show size and shape as minimal schematic summaries; (b) a simple dotplot of
the components of Box’s M test: the log determinants of the $\bm{S}_{i}$
together with that of the pooled $\bm{S}_{p}$. Extensions of these simple
plots raise the question of whether measures of heterogeneity other than that
captured in Box’s test might also be useful; and, (c) the connection between
Levene-type tests and an ANOVA (of centered absolute differences) suggests a
parallel with a multivariate extension of Levene-type tests and a MANOVA. We
explore this with a version of Hypothesis-Error (HE) plots we have found
useful for visualizing mean differences in MANOVA designs.

Accordingly, the following sections introduce and apply our conceptual
framework for general graphical methods for visualizing data in relation to
MLM-related questions and their applications. This is based on the simple
ideas that: (a) a _data ellipsoid_ provides a visual summary of location and
scatter of a multivariate sample; (b) these can be combined in various ways to
give visual tests of group differences in means and covariance matrices; and,
(c) when there are more than just a few response variables, a reduced-rank
(canonical) transformation provides an appealing way to visualize these
effects in an optimal low-dimensional approximation.

Section 4 introduces some novel visualizations of the components related to
Box’s test, which in turn suggest other possible test statistics that deserve
further study. Section 2 in the Supplementary Appendix describes a
multivariate generalization of Levene’s test within the HE plot framework that
yields attractive and useful displays.

A different graphical approach to the main question is to consider
multivariate dispersion in terms of distances of the points from their
centroids; this is illustrated in the Supplementary Materials. These methods
are all implemented in R (R Core Team, 2015), principally in the heplots and
candisc packages.222The complete R code for our examples is provided in the
Supplementary Materials, and are hosted online at
https://mattsigal.github.io/eqcov_supp/.

##  2 Visualizing covariance matrices

Before diving into details and statistical tests, it is useful to see how to
visualize covariance matrices themselves. We do this using the graphical
analog of minimally sufficient statistics $(\bar{\bm{y}}_{i},\bm{S}_{i})$ for
the MANOVA problem— a _minimally sufficient graphical display_. This graphical
principle has been called _visual thinning_ (Friendly, 2007): reducing a
graphical display to the essentials of what you want to see by relying upon
statistics that most efficiently capture the parameters of interest. In
multivariate displays, this usually means replacing data points by well-chosen
visual summaries.

###  2.1 Data ellipsoids

The essential idea (Dempster, 1969; Friendly, Monette, & Fox, 2013) is that
for a $p$-dimensional sample, $\bm{Y}_{n\times p}$, the $p\times p$ covariance
matrix $\bm{S}$ can be represented by the $p$-dimensional _concentration_ or
_data ellipsoid_ , $\mathcal{E}_{c}$ of size (“radius”) $c$. This is defined
as the set of all points $\bm{y}$ satisfying

$\mathcal{E}_{c}(\overline{\bm{y}},\bm{S}):=\\{\bm{y}:(\bm{y}-\bar{\bm{y}})^{\mathsf{T}}\,\bm{S}^{-1}\,(\bm{y}-\bar{\bm{y}})\leq
c^{2}\\}\>.$ (2)

It is readily seen that the quadratic form in Eqn. (2) corresponds to the set
of points whose squared Mahalanobis distances
$D^{2}_{M}(\bm{y})=(\bm{y}-\bar{\bm{y}})^{\mathsf{T}}\,\bm{S}^{-1}\,(\bm{y}-\bar{\bm{y}})$,
from the centroid of the sample,
$\bar{\bm{y}}=(\bar{y}_{1},\bar{y}_{2},\dots,\bar{y}_{p})^{\mathsf{T}}$, are
less than or equal to $c^{2}$.

When the variables are multivariate normal, the data ellipsoid approximates a
contour of constant density in their joint distribution. In this case
$D^{2}_{M}(\bm{y})$ has a large-sample $\chi^{2}_{p}$ distribution, or, in
finite samples, approximately $[p(n-1)/(n-p)]F_{p,n-p}$. Hence, in the
bivariate case, taking $c^{2}=\chi^{2}_{2}(0.95)=5.99\approx 6$ encloses
approximately 95% of the data points under normal theory. A 68% coverage data
ellipse with $c^{2}=\chi^{2}_{2}(0.68)=2.28$ gives a bivariate analog of the
standard $\bar{x}\pm 1s_{x}$ and $\bar{y}\pm 1s_{y}$ intervals. See Friendly
et al. (2013) for properties of data ellipsoids and their use to interpret a
wide variety of problems and applications in multivariate linear models.

In practice, $p$-dimensional data ellipsoids can be viewed in variable space
via 2D or 3D projections, or for all $p$ variables, in a pairwise scatterplot
matrix of 2D projections. Alternatively, they can be viewed in the space of
any linear transformation $\bm{Y}\bm{T}\mapsto\bm{Y}^{\star}$, where the
principal components transformation provides useful views in low-D projections
accounting for maximal total variance.

###  2.2 Simple example: Iris data

It is easiest to illustrate these ideas using the well-known Iris data set
(Anderson, 1935), which pertains to four measures (sepal width and height, and
petal width and height) of three species of iris flowers from the Gaspe
Peninsula. One approach to visualizing within group variability is to begin
with an enhanced scatterplot that adds a standard (68%) data ellipse for each
group. Then imagine taking away the data points (and other enhancements)
leaving only the data ellipses, and add the corresponding data ellipse for the
pooled sample variance covariance matrix $\bm{S}_{p}$. This gives a visual
summary of group means and of the within-group covariance, and is shown in the
right panel of Figure 1. In this plot the variances and covariances look
similar for the Versicolor and Virginca groups, but the Setosa group differs
by exhibiting a higher correlation between sepal length and width and a
smaller variance on sepal length.

Finally, we center all the ellipses at the origin in order to focus _only_ on
size and shape of the within-group covariances, so that these can be
_directly_ compared visually.333This example seems at first glance to be a
special case, because all variables are measured in the same units. However,
the units do not matter in most of our plots because the axis ranges are taken
from the data and scale units are not equated. The plots in Figure 1 would
look identical except for tick labels if we transformed sepal length from
centimeters to inches. For these two variables, we can now see that the
covariance of _Virginca_ is nearly identical to $\bm{S}_{p}$, while
_Versicolor_ has somewhat greater variance on sepal length.444Such plots are
produced by the covEllipses() function in the heplots package.

Figure 1: Covariance ellipses for the Iris data. Left: separate groups and the
pooled within-group covariance matrix; right: all covariance matrices centered
at the origin.

This method becomes particularly useful when we look at the data ellipses for
all pairs of variables in scatterplot matrix format. As in the right panel of
Figure 1, we center these ellipsoids at the origin. The display in Figure 2
shows only size (variance) and shape (correlation) differences, which speak
directly to the question of homogeneity of covariance matrices.

Figure 2: Pairwise data ellipses for the Iris data, centered at the origin.
This view makes it easy to compare the variances and covariances for all pairs
of variables.

It can now be seen precisely _how_ the covariance matrix for Setosa differs
from those of the other species. The within group correlations differ for all
pairs of variables, and as well, the variances are noticeably smaller for
petal width and petal length. In addition, while _Versicolor_ and _Virginca_
have similar shapes, close to that of the pooled covariance matrix, in most
panels (particularly for petal width), _Virginca_ exhibits greater variance.

####  2.2.1 More general models

In these plots, the centered views in Figure 2 correspond to an analysis of
the covariance matrices among the _residuals_ from the MLM predicting the four
responses from the species variable. Consequently, the same ideas apply in
more general models. For example, in a MANCOVA setting, the model may include
one or more quantitative covariates.555For instance, in R notation, mod1 <\-
lm(cbind(y1, y2, y3) ~ Group + x1 + x2), where three response variables are
being predicted by the grouping factor and two covariates. The analyses
suggested above could then be applied to the residuals from this model.
Likewise, in a two-way MANOVA design, with factors $A$ and $B$, we could treat
the combinations of these factors as the “group” variable and view the
pairwise data ellipses.666For example, we could estimate such a model within R
using mod2 <\- lm(cbind(y1, y2, y3) ~ A:B) and then generate the pairwise
covariance data ellipses with covEllipses(residuals(mod2), variables=1:3).

###  2.3 Low-rank views

With $p>3$ response variables, a simple alternative to the pairwise 2D
projections shown in Figure 2 is the projection into the principal component
space accounting for the greatest amounts of total variance in the data. For
the Iris data, a simple PCA of the covariance matrix shows that nearly 98% of
total variance in the data is accounted for in the first two dimensions.

Figure 3 shows the plots of the covariance ellipsoids for the first two
principal component scores, uncentered (left panel) and centered (right
panel). The dominant PC1 (92% of total variance) essentially orders the
species by a measure of overall size of their sepals and petals. In the
centered view, it can again be seen how _Setosa_ differs in covariance from
the other two species, and that while _Virginca_ and _Versicolor_ both have
similar shapes to the pooled covariance matrix, Versicolor has somewhat
greater variance on PC1.

Figure 3: Covariance ellipsoids for the first two principal components of the
iris data. Left: Uncentered, showing group means on the principal components;
right: centered at the origin.

We note that PCA is focused on a low-rank approximation to account for _total
variance_ of the data. In the MANOVA context, the main question concerns
between-group variance (differences among means) relative to within-group
variance. For this question, views in canonical space provide the same
advantages, as described in the Supplemental Appendix.

####  2.3.1 Small dimensions can matter

For the Iris data, the first two principal components account for 98% of total
variance, so we might think we are done here. Yet, it turns out that in a
variety of multivariate contexts small dimensions can matter. For example,
Friendly & Kwan (2009) showed that problems of multicollinearity in regression
models could be readily viewed as near singularities that exist in the space
of the _smallest_ principal component dimensions, but cannot be seen in the
larger dimensions. Similarly, multivariate outliers often do not appear in
bivariate views of the data in variable space, but can stand out like sore
thumbs in the space of the smallest PCA dimensions.777A simple yet powerful
demonstration: Generate triples, $(x_{1},x_{2},x_{3})$ as $U[0,1]$ and scale
each set to unit sum, so all points lie on the simplex $x_{1}+x_{2}+x_{3}=1$.
Then add a few outliers within a unit sphere of radius $r\leq 0.05$ centered
at the origin. The outliers will not stand out in any univariate or bivariate
plots along the coordinate axes, but will be dramatic when viewed along the
third principal component, which is orthogonal to the plane of the simplex.

As we will see, Box’s M test, because it is a (linear) function of all the
eigenvalues of the between and within group covariance matrices, is also
subject to the influence of the smaller dimensions, where differences among
$\bm{S}_{i}$ and of $\bm{S}_{p}$ can lurk.

Figure 4: Covariance ellipsoids for the last two principal components.

Figure 4 shows the covariance ellipsoids in (PC3, PC4) space. PC3 contrasts
Sepal.Length against the other variables; PC4 contrasts Sepal.Length and
Petal.Width vs. Sepal.Width and Petal.Length. The main point is that. even
though these dimensions contribute little to total variance, there are more
pronounced differences in the within-group shapes (correlations) relative to
the pooled covariance, and these contribute to a rejection of homogeneity by
Box’s M test. Here we see that the correlation for Virginca is of opposite
sign from the other two groups. (The total sample covariance, ignoring
Species, is of course uncorrelated in all principal component dimensions.)

##  3 Other examples

In what follows, it will be instructive to use two other empirical examples to
illustrate our graphical methods: (a) one where it turns out that there are
important differences among group means but little evidence for heterogeneity
of covariances; (b) another where there are both differences in means and
heterogeneity, but the number of response variables is large, which makes
understanding these effects more difficult.

###  3.1 Skulls data

For comparison with what we have seen for the Iris data, the Skulls data set
provides an example where there are also substantial differences among the
means of groups, but little evidence for heterogeneity of their covariance
matrices.

The data concern four physical measurements of size and shape made on 150
Egyptian skulls from five epochs ranging from 4000 BC to 150 AD. The measures
are: maximal breadth (mb), basibregmatic height (bh), basialveolar length
(bl), and nasal height (nh) of each skull. See
http://www.redwoods.edu/instruct/agarwin/anth_6_measurements.htm for the
formal definitions of these measures, and Figure 5 for a diagram of what they
pertain to. The question of interest in this analysis is whether and how these
measurements changed over time. Systematic changes over time in means and/or
covariances is of interest because it could indicate interbreeding among
migrant populations (or the influence of other factors, such as diet).

Figure 5: Diagram of the skull measurements for the Egyptian skulls data set.
Maximal breadth and basibregmatic height are the basic measures of “size” of a
skull. Basialveolar length and nasal height are important anthropometric
measures of skull “shape”.

A MANOVA of this data set shows a highly significant effect of the epoch
factor (Pillai trace = 0.3533, approx.
$F(16,434.45)=3.512,p<0.000001$).888This can be conducted as a MLM in R as
follows: lm(cbind(mb, bh, bl, nh) ~ epoch, data=Skulls). Treating epoch as an
ordered factor yields an even strong test for linear trend in the means over
time, and all non-linear trends are effectively null. The conclusion so far is
that for these measures of skull size and shape, there are approximately
systematic changes over time.

Figure 6 shows the centered covariance ellipsoids for all epochs and for the
pooled data.999Such figures can be generated using the covEllipses() function
from the heplots package. For the most part, these are all coincident,
indicating equal covariance matrices. Only for the variable basialveolar
length does any epoch differ perceptibly, where it has slightly greater
variance in the earliest epoch (4000BC). It can also be seen that these four
measures are relatively uncorrelated within each epoch.

Figure 6: Pairwise data ellipses for the Skulls data, centered at the grand
mean. Those for the pooled data are shaded.

###  3.2 Wine data

The Wine data101010This data set is contained in the candisc package, and is
originally from the UCI Machine Learning Repository
(http://archive.ics.uci.edu/ml/datasets/Wine). is a classic in the machine
learning literature as a high-D classification problem, but is also of
interest for examples of MANOVA and discriminant analysis. These data are the
results of a chemical analysis of wines grown in the same region in Italy but
derived from three different cultivars of grapes: Barolo, Grignolino, and
Barbera. The analysis determined the quantities of 13 constituents found in
each of the three types of wines. The total sample size is $N=178$, but the
data are unbalanced ($n_{i}=59,71,48$).

Figure 7: Boxplots of the distributions of the Wine variables, by Cultivar.
How do the means differ? How do the variances differ?

By way of introduction to this data set, the set of boxplots in Figure 7 for
all of the response variables by Cultivar gives a useful overview. It is easy
to see that most of the variables differ substantially among the cultivars,
but the pattern of differences in means or medians is complex across the
variables. There is also a substantial number of outliers for some of the
variables, particularly MalicAcid and Proa. It is much harder to characterize
how the wines differ in variance, though differences on some variables appear
pronounced (e.g., MalicAcid, Flav, Color).

##  4 Visualizating Box’s M test

The covariance ellipse plots we have seen in earlier examples (e.g., Figure 1
and Figure 6) are useful schematic summaries, but in some cases, a simpler
visual summary might be more useful. Eqn. (1) suggests that the simplest
visualization might focus on the components of Box’s M test, for example, a
dot plot of the log determinants of the covariance matrices $\bm{S}_{i}$
together with that of the pooled $\bm{S}_{p}$. To the extent that the
covariance matrices are all equal, so too should the values on which Box’s
test are based.

An important virtue of these plots is that they can show _how_ the groups
differ from each other, and from the pooled covariance matrix on the scalar
measure $\ln|\mathbf{S}|$. In this way, they can suggest more specific
questions or hypotheses regarding the equality of covariance matrices,
analogous to the use of contrasts and linear hypotheses for testing
differences among group mean vectors.

Such plots are far more useful with confidence intervals around the
$\ln|\mathbf{S}_{i}|$ and $\ln|\mathbf{S}_{p}|$. Recently, Cai, Liang, & Zhou
(2015) have suggested an asymptotic, central-limit theorem approximation to
the distribution of $\log|\mathbf{S}|$.111111Box’s M test is calculated by the
function boxM() in heplots. These plots are produced by the plot() method for
"boxM" objects.

To illustrate, Box’s M test gives an approximate chi-square,
$\chi^{2}(20)=140.94,p<2.2\times 10^{-16}$ for the Iris data, while the Skulls
data gives $\chi^{2}(40)=45.67,p=0.248$. The correponding plots for these
tests are shown in Figure 8. For the Iris data (left), _Setosa_ stands out
having a substantially smaller covariance matrix (by $\log|\mathbf{S}|$) than
the other species. The intervals for _Versicolor_ and _Virginica_ overlap with
that for $\ln|\mathbf{S}_{p}|$, but seem to differ from each other.

Figure 8: Plots of log determinants of the components of Box’s M test with
asymptotic 95% confidence intervals. Left: The Iris data shows substantial
heterogeneity; right: the Skulls data shows some small differences, but no
evidence for heterogeneity.

In contrast, for the Skulls data (Section 3.1), the plot of the log
determinants in Figure 8 (right) shows that the 95% confidence intervals for
the $\ln|\bm{S}_{i}|$ all overlap with each other and with that for the pooled
$\ln|\bm{S}_{p}|$.

Although these differences among covariance matrices are not significant by
Box’s M test, we can use this example to illustrate how such plots can suggest
scientifically meaningful hypotheses regarding the equality of covariance
matrices, analogous to what we are accustomed to doing with tests for mean
differences. For the sake of an example, assume that changes in variances and
covariances of such skull measurements is of interest, and that we were able
to obtain a sample 10 times as large from each epoch, giving the same pattern
of results, but with standard errors divided by $\sqrt{10}$. We might then try
to interpret the general decrease in $\ln|\bm{S}_{i}|$ from the earliest epoch
to 200BC. Were skulls becoming more homogeneous over time? But, what happened
in the 150 AD sample?

For completeness, and use below, Figure 9 shows the same type of plot for the
Wine data. Box’s M test is overwhelmingly significant,
$\chi^{2}(182)=684.2,p<2.2\times 10^{-16}$, and the figure shows why: the
$\ln|\bm{S}_{i}|$ for Barbera and Barolo differ substantially from that of the
pooled $\ln|\bm{S}_{p}|$ and from that of Grignolino, which is closest to that
of the pooled covariance matrix.
But the Wine data has 13 response variables; the Supplementary Appendix
describes some other graphical methods that help to understand this in a 2D
canonical space.

Figure 9: Plot of log determinants of the covariance matrices for the Wine
data.

###  4.1 Eigenvalue plots

Having made some progress with visualizing the components of Box’s M test, it
is natural to ask if other plots or other test statistics can address these
relationships in a more nuanced manner. In the MLM, the various test
statistics (Wilks’ $\Lambda$, Hotelling-Lawley and Pillai Trace criteria,
Roy’s maximum root test) are all functions of the eigenvalues of a hypothesis
matrix $\bm{H}$ relative to an error matrix $\bm{E}$. So too, all reasonable
test statistics for equality of covariance matrices are functions of the
eigenvalues of the $\mathbf{S}_{i}$ and $\mathbf{S}_{p}$.

Another sensible plot is therefore an analog of a _scree plot_ of eigenvalue
versus dimension number, with separate curves for each matrix, similar to
their use in exploratory factor analysis.

Figure 10 shows such a plot for the Wine data. This dataset is comprised of 13
response variables, so Box’s test is based on 13 eigenvalues. To preserve
resolution, we show eigenvalues on the log scale, and in two separate panels.

Figure 10: Scree plots of log eigenvalues of the covariance matrices for the
Wine data. Those for the pooled covariance matrix are shown with a heavier
line, marked ‘p’.

It can be seen that the eigenvalues on the largest dimensions do not differ
very much across the groups. However, they differ progressively more among the
groups on the dimensions with small eigenvalues. The differences among the
groups in the right panel of Figure 10 are similar to what was seen in the the
log determinants plot (Figure 9): Grignolino (group 2) is quite close to the
result for the pooled covariance matrix, while the Barolo (group 1) and
Barbera (group 3) wines differ. This demonstrates that Box’s M test is indeed
sensitive to differences among the smaller eigenvalues.

###  4.2 Other test statistics

As we saw above (Section 2), the question of equality of covariance matrices
can be expressed in terms of the similarity in size and shape of the data
ellipses for the individual group $\bm{S}_{i}$ relative to that of
$\bm{S}_{p}$. Box’s M test uses just one possible function to describe this
size: the logs of their determinants.

When $\bm{\Sigma}$ is the covariance matrix of a multivariate vector $\bm{y}$
with eigenvalues $\lambda_{1}\geq\lambda_{2}\geq\dots\lambda_{p}$, the
properties shown in Table 1 represent methods of describing the size and shape
of the ellipsoid in $\mathbb{R}^{p}$. More general theory and statistical
applications of the geometry of ellispoids is given by Friendly et al. (2013).

Table 1: Statistical and geometrical properties of “size” of an ellipsoid Size | Conceptual formula | Geometry | Function
---|---|---|---
(a) Generalized variance: | $\mathrm{det}(\bm{\Sigma})=\prod_{i}\lambda_{i}$ | area, (hyper)volume | geometric mean
(b) Average variance: | $\mathrm{tr}(\bm{\bm{\Sigma}})=\sum_{i}\lambda_{i}$ | linear sum | arithmetic mean
(c) Average precision: | $1/\mathrm{tr}(\bm{\bm{\Sigma}^{-1}})=1/\sum_{i}(1/\lambda_{i})$ |  | harmonic mean
(d) Maximal variance: | $\lambda_{1}$ | maximum dimension | supremum

Hence, for a sample covariance matrix $\bm{S}$, $|\bm{S}|$ is a measure of
generalized variance and $\ln|\bm{S}|$ is a measure of average variance across
the $p$ dimensions.

The "boxM" methods in heplots can compute and plot all of the functions of the
eigenvalues in Table 1. The results are shown in Figure 11.

Figure 11: Plot of eigenvalue statistics of the covariance matrices for the
Wine data.

Except for the absence of error bars, the plot for log product in Figure 11
(upper left panel) is the same as that in Figure 9. In principle, it is
possible to add such confidence intervals for all these measures through the
use of bootstrapping, but this has not yet been implemented.

For this data set, the pattern of points in the plot for Box’s M is also more
or less the same as that for the precision measure. The plots for the sum of
and maximum eigenvalue are also similar to each other, but differ from those
of the two measures in the left column of Figure 11. The main point is that
these are not _all the same_ , so different functions reflect different
patterns of the eigenvalues.121212In analogous examples for other data sets we
also see different patterns over measures, but these are not the same as in
this example.

The sum of eigenvalues is similar in form to the Hotelling-Lawley trace
criterion used in MLM test and the maximum eigenvalue is similar to Roy’s
test, and all these test statistics have convenient $F$ approximations. Plots
such as Figure 11 and those we have tried for other examples suggest that it
would be useful to develop analogous tests for homogeneity of covariance
matrices, with possibly different properties of error rates and power against
specified alternatives. We do not pursue this topic here, but leave it open
for further research.

##  5 Concluding Remarks

The idea for this paper arose from discussion in a graduate course on
multivariate data analysis in connection with the topic of the assumptions of
MANOVA (independence, multivariate normality of residuals, homogeneity of
covariance matrices), diagnostic tests for these, and remedies when they are
violated. The instructor (the first author) had presented Box’s M test, with
the caveat of the opening quotation by George Box that the test was not robust
against mild-to-moderate non-normality, while the Pillai-Bartlett trace test
for mean differences does possess this robustness.

One student commented that this was akin to recommending a screening test for
lung cancer that was more sensitive to influenza. Another student asked,
“Well, if I run Box’s test and it shows significance, how can I decide if my
MANOVA results are valid? How can I determine which groups differ in
covariance matrices and on which variables?”

These seemed to be perfectly reasonable questions, and while we were aware of
the modifications that might account for non-normality (O’Brien, 1992; Tiku &
Balakrishnan, 1984), these are not typically available in standard software
and therefore their theory is cold comfort to applied researchers. More
importantly, we were struck with how little information is provided by the
result of such a significance test, regardless of its Type I error and power
properties. What was lacking in the null hypothesis $p$-value was any
_insight_ into the nature of group differences in covariance matrices.

The approach we have outlined here stems from the mantras of exploratory data
analysis (EDA) in a multivariate setting:

> The greatest value of a picture is when it forces us to notice what we never
> expected to see (Tukey, 1977, p. vi).

> The purpose of [data] display is comparison (recognition of phenomena), not
> numbers (Tukey, 1990).

We began with the idea of displaying covariance matrices directly in terms of
data ellipsoids that serve as normal theory minimally sufficient visual
summaries. We mentioned, but did not illustrate, that classical estimates of
$\bm{S}$ could be replaced by robust MVE and MCD alternatives in the presence
of multivariate outliers.

The simple dot plot (Figure 9) of the components of Box’s M test answers the
first question: How do the groups differ in covariance matrices? It does more
than this, however, because it suggests that _other_ functions of the
eigenvalues of the $\bm{S}_{i}$ and $\bm{S}_{p}$ might provide alternative
measures of homogeneity of covariance matrices (Figure 11), and that their
distribution across the orthogonal dimensions of within-group variation
provides valuable insight into the properties of these measures and
statistical tests.

The HE plot framework we have described provides some answers to the second
question: If the multivariate responses do differ among groups in their
variances and covariances, which responses contribute to this and how do they
differ? We formulated an extension of the univariate Levene–Brown-Forsythe
test to the MANOVA setting, with the property that HE plots and canonical
discriminant HE plots for the question of mean differences apply directly to
the question of homogeneity of covariance matrices. Pairwise HE plots in
variable space show which responses differ in scatter, and their projection
into canonical space (Figure 2 in the supplementary appendix) provides a
convenient 2D representation, often with quite a simple interpretation.

Finally, we do not fear that this paper might attract the opprobrium that we
“flung data onto many canvases to see what stuck.” To the contrary, we were
struck by the relative degree of consistency of the resulting plots across
these various visual approaches. When covariance ellipsoids showed differences
among groups in variable space or principal components space, we could
understand how these differences were reflected in the Box’s M plots. All of
this is much more satisfying than a $p$-value, robust or otherwise.

##  6 Supplementary Materials

A Supplemental Appendix describes the HE plot framework for tests of mean
differences and gives some further examples. It also describes a multivariate
extension of Levene’s test within this framework. The R packages candisc and
heplots are freely available from the Comprehensive R Archive Network,
http://cran.us.r-project.org/. Complete R scripts for the Iris, Wine, and
Skulls examples are available at https://mattsigal.github.io/eqcov_supp/.

##  7 Acknowledgements

We are grateful to Augustine Wong and Harrison Zhou for discussions on the
asymptotic distribution of log determinants of covariance matrices. Wayne
Oldford provided much useful feedback on an initial draft of the paper. Two
anonymous reviewers and the associate editor gave detailed critical and
constructive feedback that helped us greatly to revise the article.

## References

Aelst, S. V., & Willems, G. (2011). Robust and efficient one-way MANOVA tests.
_Journal of the American Statistical Association_ , _106_(494), 706–718.
http://doi.org/10.1198/jasa.2011.tm09748

Anderson, E. (1935). The irises of the Gaspé peninsula. _Bulletin of the
American Iris Society_ , _35_ , 2–5.

Bartlett, M. S. (1937). Properties of sufficiency and statistical tests.
_Proceedings of the Royal Society of London. Series A_ , _160_(901), 268–282.
http://doi.org/10.2307/96803

Box, G. E. P. (1949). A general distribution theory for a class of likelihood
criteria. _Biometrika_ , _36_(3-4), 317–346.
http://doi.org/10.1093/biomet/36.3-4.317

Box, G. E. P. (1950). Problems in the analysis of growth and wear curves.
_Biometrics_ , _6_ , 362–389.

Box, G. E. P. (1953). Non-normality and tests on variances. _Biometrika_ ,
_40_(3/4), 318–335. http://doi.org/10.2307/2333350

Brown, M. B., & Forsythe, A. B. (1974). Robust tests for equality of
variances. _Journal of the American Statistical Association_ , _69_(346),
364–367. http://doi.org/10.1080/01621459.1974.10482955

Cai, T. T., Liang, T., & Zhou, H. H. (2015). Law of log determinant of sample
covariance matrix and optimal estimation of differential entropy for high-
dimensional gaussian distributions. _Journal of Multivariate Analysis_ , _137_
, 161–172. http://doi.org/https://doi.org/10.1016/j.jmva.2015.02.003

Cochran, W. G. (1941). The distribution of the largest of a set of estimated
variances as a fraction of their total. _Annals of Eugenics_ , _11_(1), 47–52.
http://doi.org/10.1111/j.1469-1809.1941.tb02271.x

Conover, W. J., Johnson, M. E., & Johnson, M. M. (1981). A comparative study
of tests for homogeneity of variances, with applications to the outer
continental shelf bidding data. _Technometrics_ , _23_(4), 351–361.
http://doi.org/10.1080/00401706.1981.10487680

Dempster, A. P. (1969). _Elements of continuous multivariate analysis_.
Reading, MA: Addison-Wesley.

Finch, H., & French, B. (2013). A monte carlo comparison of robust MANOVA test
statistics. _Journal of Modern Applied Statistical Methods_ , _12_(2), 35–81.
http://doi.org/10.22237/jmasm/1383278580

Friendly, M. (2007). HE plots for multivariate general linear models. _Journal
of Computational and Graphical Statistics_ , _16_(2), 421–444.
http://doi.org/10.1198/106186007X208407

Friendly, M., & Kwan, E. (2009). Where’s Waldo: Visualizing collinearity
diagnostics. _The American Statistician_ , _63_(1), 56–65.

Friendly, M., Monette, G., & Fox, J. (2013). Elliptical insights:
Understanding statistical methods through elliptical geometry. _Statistical
Science_ , _28_(1), 1–39. http://doi.org/http://dx.doi.org/10.1214/12-STS402

Gastwirth, J. L., Gel, Y. R., & Miao, W. (2009). The impact of Levene’s test
of equality of variances on statistical theory and practice. _Statistical
Science_ , _24_(3), 343–360. http://doi.org/10.1214/09-STS301

Hakstian, A. R., Roed, J. C., & Lind, J. C. (1979). Two-sample t-2 procedure
and the assumption of homogeneous covariance matrices. _Psychological
Bulletin_ , _86_(6), 1255–1263. http://doi.org/10.1037/0033-2909.86.6.1255

Hartley, H. O. (1950). The use of range in analysis of variance. _Biometrika_
, _37_(3–4), 271–280. http://doi.org/10.1093/biomet/37.3-4.271

Harwell, M. R., Rubinstein, E. N., Hayes, W. S., & Olds, C. C. (1992).
Summarizing monte carlo results in methodological research: The one- and two-
factor fixed effects ANOVA cases. _Journal of Educational and Behavioral
Statistics_ , _17_(4), 315–339. http://doi.org/10.3102/10769986017004315

Levene, H. (1960). Robust tests for equality of variances. In I. Olkin, S. G.
Ghurye, W. Hoeffding, W. G. Madow, & H. B. Mann (Eds.), _Contributions to
probability and statistics: Essays in honor of Harold Hotelling_ (pp.
278–292). Stanford, Calif: Stanford University Press.

Lix, J., L.M. Keselman, & Keselman, H. (1996). Consequences of assumption
violations revisited: A quantitative review of alternatives to the one-way
analysis of variance F test. _Review of Educational Research_ , _66_(4),
579–619. http://doi.org/10.3102/00346543066004579

Olson, C. L. (1974). Comparative robustness of six tests in multivariate
analysis of variance. _Journal of the American Statistical Association_ ,
_69_(348), 894–908. http://doi.org/10.1080/01621459.1974.10480224

O’Brien, P. C. (1992). Robust procedures for testing equality of covariance
matrices. _Biometrics_ , _48_(3), 819–827. Retrieved from
http://www.jstor.org/stable/2532347

R Core Team. (2015). _R: A language and environment for statistical
computing_. Vienna, Austria: R Foundation for Statistical Computing. Retrieved
from http://www.R-project.org/

Rogan, J. C., & Keselman, H. J. (1977). Is the ANOVA f-test robust to variance
heterogeneity when sample sizes are equal?: An investigation via a coefficient
of variation. _American Educational Research Journal_ , _14_(4), 493–498.
http://doi.org/10.3102/00028312014004493

Tiku, M., & Balakrishnan, N. (1984). Testing equality of population variances
the robust way. _Communications in Statistics - Theory and Methods_ ,
_13_(17), 2143–2159. http://doi.org/10.1080/03610928408828818

Timm, N. H. (1975). _Multivariate analysis with applications in education and
psychology_. Belmont, CA: Wadsworth (Brooks/Cole).

Todorov, V., & Filzmoser, P. (2010). Robust statistic for the one-way MANOVA.
_Computational Statistics & Data Analysis_, _54_(1), 37–48.
http://doi.org/10.1016/j.csda.2009.08.015

Tukey, J. W. (1977). _Exploratory data analysis_. Reading, MA: Addison Wesley.

Tukey, J. W. (1990). Data-based graphics: Visual display in the decades to
come. _Statistical Science_ , _5_(3), 327–339.

Welch, B. L. (1947). The generalization of “student’s” problem when several
different population varlances are involved. _Biometrika_ , _34_(1–2), 28–35.
http://doi.org/10.1093/biomet/34.1-2.28

Zhang, J., & Boos, D. D. (1992). Bootstrap critical values for testing
homogeneity of covariance matrices. _Journal of the American Statistical
Association_ , _87_(418), 425–429. Retrieved from
http://www.jstor.org/stable/2290273


# Metallised 3D printed plastic resonator demonstrates superconductivity below
4 K

J. Bourhill<EMAIL_ADDRESS>Quantum Technologies and Dark Matter
Labs, Department of Physics, University of Western Australia, 35 Stirling Hwy,
6009 Crawley, Western Australia.  G. Cochet  Elliptika (GTID) 2 Rue Charles
Jourde, 29200 Brest, France  J. Haumant  Elliptika (GTID) 2 Rue Charles
Jourde, 29200 Brest, France  V. Vlaminck  IMT Atlantique, Technopole Brest-
Iroise, CS 83818, 29238 Brest Cedex 3, France  Lab-STICC (UMR 6285), CNRS,
Technopole Brest-Iroise, CS 83818, 29238 Brest Cedex 3, France  A. Manchec
THALES, 10 Av. 1ère Dfl, 29200 Brest, France  M. E. Tobar  Quantum
Technologies and Dark Matter Labs, Department of Physics, University of
Western Australia, 35 Stirling Hwy, 6009 Crawley, Western Australia.  V.
Castel  IMT Atlantique, Technopole Brest-Iroise, CS 83818, 29238 Brest Cedex
3, France  Lab-STICC (UMR 6285), CNRS, Technopole Brest-Iroise, CS 83818,
29238 Brest Cedex 3, France

###### Abstract

We report the first observation of a superconducting transition in a 3D
printed, metallised-plastic device. A cylindrical cavity is 3D printed from a
photosensitive polymer resin and then a 20 $\mu$m layer of tin deposited. A
resonant TE microwave mode at 13.41 GHz is observed to reduce its losses by an
order of magnitude once it is cooled below 3.72 K; the superconducting
transition temperature of tin, with the mode’s $Q$ factor increasing from
$2.7\times 10^{4}$ to $4.0\times 10^{5}$.

##  I Introduction

The ease, accessibility, cost and versatility of photosensitive liquid resin
stereolithography (SLA) 3D printing is vastly superior when compared to
traditional subtractive manufacturing techniques. It is not uncommon for 3D
printers with micrometer sized print resolutions to be found in the personal
home, which is not the case for their cousins; Selective Laser Melting (SLM)
3D printers, which use metallic powders for additive manufacturing, and whose
base cost and material cost are substantially higher. Plastic resin 3D
printing has also been demonstrated to be an extraordinarily portable
technology, with the ability to move a printer from one location to another
without the need for arduous set-up and pack-down procedures, or overly
cumbersome printing units – a typical printer weighs on the order of 20 kg.
This fact makes them an intriguing technology to explore for their
applications to defence and space exploration wherein manufacturing from the
same device may be required at changing locations with short lead times.

It has been previously demonstrated that microwave resonant cavities with
equivalent performance to their traditionally manufactured counterparts can be
produced by plastic 3D SLA printing followed by metallisation [1, 2]. This
result has opened up the possibility of rapid, low cost, and highly
reproducible production of a wide variety of microwave devices; such as
filters, isolators, RF and magnetic field shielding, and more complicated
systems for coupling microwave photonics to additional degrees of freedom to
form a hybrid system [2, 1, 3].

At the same time, SLM printing has been demonstrated to produce metallic
cavities and structures which display superconducting transitions when cooled
below their critical temperatures, $T_{c}$, first in aluminium [4] and then in
niobium [5]. These advances have been used to construct non-trivial resonant
devices [6, 7] with unique properties which could not have been manufactured
through subtractive manufacturing methods, which will greatly benefit from
transitioning into superconducting devices at low temperatures.

Superconducting cavities find numerous uses in physics. Once transitioned,
they trap and store resonant microwave radiation and reduce losses, resulting
in devices with very high quality factor ($Q$), narrow bandwidth, and long
coherence times [8]. These cavities find application in particle accelerators
[9, 10, 11], sensing [12, 13, 14], metrology and precision RF sources [15,
16], and for testing fundamental physics. In particular, tests of the speed of
light and the constancy of fundamental constants [15, 17, 18, 19] as well as
the search for hidden sector particles and other dark matter candidates [20,
21, 22], depend on such cavities. They are also essential in cavity quantum
electrodynamics (CQED) experiments to shield qubit devices, thereby providing
a reduced density of states for the qubit to radiate into [23, 24].
Superconducting cavities are often precision-machined from extremely high
purity aluminium or niobium at great material and labour costs in order to
achieve optimal surface preparation.

Here, we demonstrate superconductivity in a tin (Sn) coated, 3D-printed-
plastic microwave resonant cavity exhibited via the sharp increase of the $Q$
factor of the cavity’s lowest-order transverse magnetic mode (TE011) at the
$T_{c}$ of Sn, $3.72$ K [25]. Sn is a type-I superconductor and has an
intermediate $T_{c}$ located below that of niobium (Nb, $T_{c}=9.2$ K), but
above Aluminium (Al, $T_{c}=1.2$ K) – the two most commonly used elemental
superconductors. Given that most commercially available liquid helium based
cryogenic systems can reach temperatures below 4 K, often down to or below at
least 3.5 K, Sn offers an advantage over Al given it can reach
superconductivity in a standard helium–4 system without the need for dilution
refrigeration, whilst its cost, availability and ease of machinability give it
certain advantages over Nb. Sn is readily available and specimens of very high
purity can be prepared. If necessary they can be grown as single crystals
[26]. These factors make it one of the most convenient superconducting
materials.

Whilst the observed $Q$ factor of the resonant TE011 mode at mK temperatures
is by no means groundbreaking, this study demonstrates an important proof-of-
concept of manufacturing superconducting devices from plastic 3D printed
structures coated in metal. There are no real limitations other than those
imposed by electrochemistry on the type of metal that can be used to coat the
plastic or the thickness of this layer. In fact, there exists a great deal of
parameter space unexplored in these types of devices in order to optimise the
achieved conductivity and hence $Q$ factor.

Figure 1: The 3D printed, metallised plastic can with internal dimensions
$h=30$ mm, $D=29.375$ mm (a) disassembled as lid and body, and (b) fully
assembled with flanged SMA input probes and copper cold-finger attachment. (c)
Schematic illustration of the experimental setup in the dilution refrigerator
demonstrating cold attenuation on the input line and isolator and
amplification (HEMT and room temperature) on the output line.

##  II System Description

###  II.1 Cavity manufacturing method

A hollow cylinder is first 3D printed with the
FormLabs${}^{\mbox{\scriptsize{\textregistered}}}$ Form
3B+${}^{\mbox{\scriptsize{\textregistered}}}$ SLA 3D printer, using the
BioMed–Clear–Resin${}^{\mbox{\scriptsize{\textregistered}}}$. This printer has
a 25 $\mu$m resolution for the vertical axis. The internal diameter and height
of the cavity were both designed to be 30 mm, with radial wall thickness 7 mm.
It is printed in two pieces - a main body and a lid (see Fig. 1 (a)), with the
lid being 5 mm thick and the base of the main body, which has two holes for
the coaxial in– and output probes, is 15 mm thick.

After printing, the cavity was cleaned of residual resin in an isopropyl
alcohol bath and then treated in an ultraviolet chamber including a heat
treatment at 60∘ C for 60 minutes. A conventional 3-D metallisation procedure
of plastic elements has been adapted in order to produce a full metallisation
of the plastic cavity by Elliptika${}^{\mbox{\scriptsize{\textregistered}}}$
and described as follows:

i) dry etching by sandblasting (increases surface roughness, increasing pores
for Pd2+ adsorption);
ii) surface activation with Pd2+ solution [27];
iii) autocatalytic bath of copper: Pd particles act as catalyst sites and
permit the growth of a homogeneous layer of Cu, which spontaneously reaches 3
$\mu$m thickness;
iv) standard electrodeposition process of Cu (10 $\mu$m);
v) Sn electrodeposition finish of 20 $\mu$m.

It should be noted that thicker layers of Cu and Sn are deposited in steps
(iv) and (v) when compared to previous implementations of this procedure [1],
which is achieved by simply increasing the run times of the electrodeposition
stages. The increase in Cu thickness is to ensure that sufficient heat
conduction is achieved in the outer metal layer, given that the bulk plastic
body of the system is a poor conductor of heat, it is important if we want the
conducting walls of the cavity to reach $T_{c}$ that the cryostat can
efficiently conduct heat away from them. The Sn layer is made thicker to
ensure it acts as a bulk superconductor, well above the percolation limit
[28].

Post metallisation, it is measured that the cavity height is $h=29.7$ mm,
whilst the cavity diameter is $D=29.24$. Deviations from the design size are
to be expected given the metal layer thickness and the 25 $\mu$m resolution of
the 3D printer.

###  II.2 Experimental method

Two coaxial cables terminated in SMA flange mounts act as the input and output
RF coupling ports, with the cables inserted through the base of the cavity. It
is important when setting the probe couplings at room temperature to ensure
that they are very weak. This is because as $Q$ increases, so do the losses
from the ports, and we do not wish to over-couple to the mode of interest once
transitioned, thereby loading it and indirectly lowering its $Q$ value. In
fact, as was done in [4], we use the opposite probe “type” in order to
minimise coupling losses as much as possible: straight coaxial probes parallel
to the cylinder axis are used, which should excite $E_{z}$ fields despite our
mode of interest being a TE mode.

The cavity has a rubidium oxide temperature sensor mounted to its base,
ensuring direct measurement of the real cavity surface temperature, and
together they are mounted to the mixing chamber (MXC) plate of a dilution
refrigerator. The input microwave lines are heavily attenuated, whilst the
output signal is amplified with a cryogenic Low Noise
Factory${}^{\mbox{\scriptsize{\textregistered}}}$ amplifier at the 4 K stage,
and a room temperature amplifier, as depicted in Fig. 1(c). This setup ensures
good signal-to-noise ratio of the output spectrum of the cavity when measured
with a Vector Network Analyser (VNA). Data is recorded during the condense
procedure of the dilution fridge, in which the temperature of the MXC changes
from $\sim 5$ K to $\sim 20$ mK, an ideal window to observe the
superconducting transition of Sn.

##  III Results

A sample of the $S_{21}$ transmission spectra recorded at different
temperatures is displayed in Fig. 2(a) and (b). It can be observed from the
first figure that within the observed 100 MHz frequency span there exists 3
modes, each of which becomes more resolved as the temperature decreases; a
direct result of the increase in conductivity of the conducting boundaries.

The mode of interest is the central, high $Q$ TE011 mode with frequency
$\omega/2\pi=13.413$ GHz. It is determined from finite element simulation that
this is indeed the mode we are looking at. A zoomed in picture of the mode is
shown in Fig. 2(b) as well as theoretical fits using a Fano model [29] and the
resulting calculated $Q$ factor. It is clear that an increase in $Q$ as well
as peak transmission is associated with a decrease in temperature.

Figure 2: (a) S21 transmission spectra over a 100 MHz span at various
temperatures with the central highlighted region zoomed in (b). (c) $Q$ factor
of the mode extracted from fitting as a function of temperature with the
superconducting transition temperature of Sn labelled in red.

By plotting the fitted $Q$ values against temperature in Fig. 2(c), we observe
that a jump in $Q$ occurs around $T_{c}=3.72$ K; the superconducting
transition for Sn. Over an order of magnitude improvement in $Q$ factor is
achieved; rising from a $T>T_{c}$ value of $Q\sim 2.7\times 10^{4}$ to a
$T<T_{c}$ value of $Q=4.0\times 10^{5}$.

Being a type-I superconductor, Sn has a typically low critical magnetic field
value of 0.03 T [26] above which the superconducting properties of the metal
are lost. Indeed, in a second experimental run in which the cavity is placed
inside an American Magnetics™ superconducting magnet mounted to the 4K plate
of the fridge, simply switching the magnet on instantly reduces the mode
amplitude and $Q$-factor due to some residual magnetisation of the magnet upon
power up.

##  IV Discussion

A microwave resonance inside an ideal empty metallic structure will be
dominated by surface resistance losses. The impact of these losses can be
calculated from the so called “geometric-factor” $G$, where

$G=\mu_{0}\omega\frac{\iiint|\vec{H}^{2}|dV}{\iint|\vec{H}_{\tau}^{2}|dS},$
(1)

where $\mu_{0}$ is the vacuum permeability, $\omega$ is the resonant angular
frequency, $\vec{H_{\tau}}$ is the tangential magnetic field of the resonant
mode, $S$ is the surfaces of the resonator, $\vec{H}$ is the magnetic field of
the resonant mode and $V$ the cavity volume. The quantity $G$ is related to
the $Q$ factor via $G=QR_{s}$, where $R_{s}$ is the surface resistance in
ohms. Essentially, the expression $G$ characterises the ratio of the resonant
mode’s magnetic field within the volume of the cavity compared to that at the
surface, where it induces current in the metallic walls and hence experiences
resistive loss.

The geometric factor $G$ obtained through the FEM is approximately 389
$\Omega$ at 13.413 GHz, allowing us to estimate the surface resistance at 972
$\mu\Omega$ below $T_{c}$ and 14.4 m$\Omega$ above $T_{c}$. This
superconducting resistivity is of equal order to previously achieved values in
SLM 3D printed Al and Nb resonators [4, 5], however an order of magnitude
greater than values measured in 500 nm thick Sn samples [28].

The measured $Q$ factor is likely limited by two factors; surface roughness
and electrical contact between the lid and the cavity. The former could be
improved by polishing the Sn layer of the cavity, either mechanically or
electrochemically. Conductivity between the two segments of the resonator
would likely also be improved with a smoother surface finish of the connecting
surfaces.

In our results we note that on either side of the highlighted mode are two
more strongly coupled TM modes – their transmission amplitudes are much larger
than the central mode, and their $Q$ factors lower as a result of higher $G$
and coupling losses.

##  V Conclusion

In conclusion, our study marks a decisive advance in the combination of
superconductivity and additive manufacturing. We have successfully
demonstrated the first-ever superconducting transition in a 3D-printed
metallized plastic device. Cooling the cavity below 3.72 K, the
superconducting transition temperature of tin, resulted in a substantial
reduction in losses and a noticeable increase in the $Q$ factor, from
$2.7\times 10^{4}$ to $4.0\times 10^{5}$, at a TE mode resonant at 13.413 GHz.
This breakthrough offers huge potential for any application or industry that
requires fast, on-site, short lead-time manufacturing. The adaptability and
versatility of this technology offers a promising route to meeting the dynamic
demands of certain industries and the access to a new design parameter space
for superconducting devices, unburdened by the limitations of subtractive
manufacturing.

## Acknowledgments

This work was jointly funded by the Région Bretagne through the project OSCAR-
SAD18024, by the UWA Research COLLABORATION AWARDS (RCA) grant “Investigation
of 3D printed microwave cavities at cryogenic temperature”, and by the
Australian Research Council Centre of Excellence for Engineered Quantum
Systems, CE170100009 and the Centre of Excellence for Dark Matter Particle
Physics, CE200100008..

## References

  * Castel  _et al._ [2019] V. Castel, S. B. Ammar, A. Manchec, G. Cochet, and J. B. Youssef, Strong coupling of magnons to microwave photons in three-dimensional printed resonators, IEEE Magnetics Letters 10, 1 (2019).
  * Bourhill  _et al._ [2020a] J. Bourhill, V. Castel, A. Manchec, and G. Cochet, Universal characterization of cavity–magnon polariton coupling strength verified in modifiable microwave cavity, Journal of Applied Physics 128, 073904 (2020a), https://doi.org/10.1063/5.0006753 .
  * Bourcin  _et al._ [2023] G. Bourcin, J. Bourhill, V. Vlaminck, and V. Castel, Strong to ultrastrong coherent coupling measurements in a yig/cavity system at room temperature, Phys. Rev. B 107, 214423 (2023).
  * Creedon  _et al._ [2016] D. L. Creedon, M. Goryachev, N. Kostylev, T. B. Sercombe, and M. E. Tobar, A 3d printed superconducting aluminium microwave cavity, Applied Physics Letters 109, 032601 (2016), https://doi.org/10.1063/1.4958684 .
  * McAllister  _et al._ [2021] B. T. McAllister, J. Bourhill, W. H. J. Ma, T. Sercombe, M. Goryachev, and M. E. Tobar, Characterization of cryogenic material properties of 3-d-printed superconducting niobium using a 3-d lumped element microwave cavity, IEEE Transactions on Instrumentation and Measurement 70, 1 (2021).
  * Bourhill  _et al._ [2023] J. F. Bourhill, E. C. I. Paterson, M. Goryachev, and M. E. Tobar, Searching for ultralight axions with twisted cavity resonators of anyon rotational symmetry with bulk modes of nonzero helicity, Phys. Rev. D 108, 052014 (2023).
  * Tobar  _et al._ [2022] M. E. Tobar, C. A. Thomson, W. M. Campbell, A. Quiskamp, J. F. Bourhill, B. T. McAllister, E. N. Ivanov, and M. Goryachev, Comparing instrument spectral sensitivity of dissimilar electromagnetic haloscopes to axion dark matter and high frequency gravitational waves, Symmetry 14, 10.3390/sym14102165 (2022).
  * Turneaure and Weissman [1968] J. P. Turneaure and I. Weissman, Microwave surface resistance of superconducting niobium, Journal of Applied Physics 39, 4417 (1968), https://doi.org/10.1063/1.1656986 .
  * Grimm  _et al._ [2005] T. Grimm, A. Aizaz, M. Johnson, W. Hartung, F. Marti, D. Meidlinger, M. Meidlinger, J. Popielarski, and R. York, New directions in superconducting radio frequency cavities for accelerators, IEEE Transactions on Applied Superconductivity 15, 2393 (2005).
  * Posen  _et al._ [2020] S. Posen, A. Romanenko, A. Grassellino, O. Melnychuk, and D. Sergatskov, Ultralow surface resistance via vacuum heat treatment of superconducting radio-frequency cavities, Phys. Rev. Appl. 13, 014024 (2020).
  * Romanenko  _et al._ [2020] A. Romanenko, R. Pilipenko, S. Zorzetti, D. Frolov, M. Awida, S. Belomestnykh, S. Posen, and A. Grassellino, Three-dimensional superconducting resonators at $t=20$ mk with photon lifetimes up to $\tau=2$ s, Phys. Rev. Appl. 13, 034032 (2020).
  * Blair  _et al._ [1995] D. G. Blair, E. N. Ivanov, M. E. Tobar, P. J. Turner, F. van Kann, and I. S. Heng, High sensitivity gravitational wave antenna with parametric transducer readout, Phys. Rev. Lett. 74, 1908 (1995).
  * Bourhill  _et al._ [2020b] J. Bourhill, N. C. Carvalho, M. Goryachev, S. Galliou, and M. E. Tobar, Generation of coherent phonons via a cavity enhanced photonic lambda scheme, Applied Physics Letters 117, 164001 (2020b), https://doi.org/10.1063/5.0023624 .
  * Carvalho  _et al._ [2017] N. C. Carvalho, J. Bourhill, M. E. Tobar, and O. D. Aguiar, Sensitivity characterisation of a parametric transducer for gravitational wave detection through optical spring effect, Classical and Quantum Gravity 34, 175001 (2017).
  * Turneaure  _et al._ [1983] J. P. Turneaure, C. M. Will, B. F. Farrell, E. M. Mattison, and R. F. C. Vessot, Test of the principle of equivalence by a null gravitational red-shift experiment, Phys. Rev. D 27, 1705 (1983).
  * Stein [1980] S. Stein, Space applications of superconductivity: resonators for high stability oscillators and other applications, Cryogenics 20, 363 (1980).
  * Lipa  _et al._ [2003] J. A. Lipa, J. A. Nissen, S. Wang, D. A. Stricker, and D. Avaloff, New limit on signals of lorentz violation in electrodynamics, Phys. Rev. Lett. 90, 060403 (2003).
  * Choi [2021] J. R. Choi, The rayleigh–lorentz invariant for superconducting resonators and optimal adiabatic qubit-information detection, Scientific Reports 11, 13722 (2021).
  * Nagel  _et al._ [2015] M. Nagel, S. R. Parker, E. V. Kovalchuk, P. L. Stanwix, J. G. Hartnett, E. N. Ivanov, A. Peters, and M. E. Tobar, Direct terrestrial test of lorentz symmetry in electrodynamics 10-18, Nature Communications 6, 8174 (2015).
  * Iwazaki [2020] A. Iwazaki, A new method for detecting axion with cylindrical superconductor, Physics Letters B 811, 135861 (2020).
  * Parker  _et al._ [2013] S. R. Parker, G. Rybka, and M. E. Tobar, Hidden sector photon coupling of resonant cavities, Phys. Rev. D 87, 115008 (2013).
  * Povey  _et al._ [2011] R. G. Povey, J. G. Hartnett, and M. E. Tobar, Microwave cavity hidden sector photon threshold crossing, Phys. Rev. D 84, 055023 (2011).
  * Rigetti  _et al._ [2012] C. Rigetti, J. M. Gambetta, S. Poletto, B. L. T. Plourde, J. M. Chow, A. D. Córcoles, J. A. Smolin, S. T. Merkel, J. R. Rozen, G. A. Keefe, M. B. Rothwell, M. B. Ketchen, and M. Steffen, Superconducting qubit in a waveguide cavity with a coherence time approaching 0.1 ms, Phys. Rev. B 86, 100506 (2012).
  * Paik  _et al._ [2011] H. Paik, D. I. Schuster, L. S. Bishop, G. Kirchmair, G. Catelani, A. P. Sears, B. R. Johnson, M. J. Reagor, L. Frunzio, L. I. Glazman, S. M. Girvin, M. H. Devoret, and R. J. Schoelkopf, Observation of high coherence in josephson junction qubits measured in a three-dimensional circuit qed architecture, Phys. Rev. Lett. 107, 240501 (2011).
  * De Haas  _et al._ [1935] W. De Haas, J. De Boer, and G. Van den Berg, The electrical resistance of cadmium, thallium and tin at low temperatures, Physica 2, 453 (1935).
  * Eisenstein [1954] J. Eisenstein, Superconducting elements, Rev. Mod. Phys. 26, 277 (1954).
  * Rhoda [1959] R. N. Rhoda, Electroless palladium plating, Transactions of the IMF 36, 82 (1959), https://doi.org/10.1080/00202967.1959.11869775 .
  * Beutel  _et al._ [2016] M. H. Beutel, N. G. Ebensperger, M. Thiemann, G. Untereiner, V. Fritz, M. Javaheri, J. Nägele, R. Rösslhuber, M. Dressel, and M. Scheffler, Microwave study of superconducting sn films above and below percolation, Superconductor Science and Technology 29, 085011 (2016).
  * Fano [1961] U. Fano, Effects of configuration interaction on intensities and phase shifts, Phys. Rev. 124, 1866 (1961).