# Using Proxy Pattern-Mixture Models to Explain Bias
in Estimates of COVID-19 Vaccine Uptake
from Two Large Surveys

Rebecca R. Andridge
The Ohio State University College of Public Health
1841 Neil Ave., Columbus, OH 43220
<EMAIL_ADDRESS>
(July 31, 2023)

###### Abstract

Recently, attention was drawn to the failure of two very large internet-based
probability surveys to correctly estimate COVID-19 vaccine uptake in the
United States in early 2021. Both the Delphi-Facebook CTIS and Census
Household Pulse Survey (HPS) overestimated uptake substantially, by 17 and 14
percentage points in May 2021, respectively. These surveys had large numbers
of respondents but very low response rates ($<$10%), thus, non-ignorable
nonresponse could have had substantial impact. Specifically, it is plausible
that “anti-vaccine” individuals were less likely to participate given the
topic (impact of the pandemic on daily life). In this paper we use _proxy
pattern-mixture models (PPMMs)_ to estimate the proportion of adults (18+) who
received at least one dose of a COVID-19 vaccine, using data from the CTIS and
HPS, under a non-ignorable nonresponse assumption. Data from the American
Community Survey provide the necessary population data for the PPMMs. We
compare these estimates to the true benchmark uptake numbers and show that the
PPMM could have detected the direction of the bias and provide meaningful bias
bounds. We also use the PPMM to estimate vaccine hesitancy, a measure for
which we do not have a benchmark truth, and compare to the direct survey
estimates.

Keywords: nonresponse bias, survey data

Running Head: _Using PPMMs to Estimate COVID-19 Vaccine Uptake_

##  1 Introduction

In the absence of nonresponse, carefully designed probability samples provide
a principled way of producing unbiased estimates of population quantities such
as proportions and means. Random selection of individuals into a sample, where
every population unit has a known, non-zero probability of selection, ensures
that the sample represents the population in expectation. Federal statistical
agencies in the United States and abroad rely on such surveys to produce
official estimates of population-level characteristics that play an important
in policy-making and business strategies (Hastak2001). These large,
government-sponsored surveys are generally large and expensive, requiring
years of development (e.g., field-testing) as well as careful post-survey
analysis before official statistics are released.

The COVID-19 pandemic created a unique challenge in that it created a sudden,
unanticipated need for data to describe both the incidence of disease and how
the pandemic was impacting daily life. In this paper we analyze two large
surveys that were implemented quickly in response to the pandemic: the U.S.
Census Bureau’s Household Pulse Survey (HPS) (Fields2020) and the Delphi-
Facebook COVID-19 Trends and Impact Survey (CTIS) (Salomon2021). The HPS was a
government-sponsored survey, whereas the CTIS was a collaboration between
academia and a private company. Both surveys were large probability samples
that repeatedly collected information on a range of pandemic-related topics;
we focus on the estimation of vaccine uptake in early 2021 when vaccines first
became available in the U.S.. Average sample sizes (number of respondents) for
the HPS was approximately 75,000 per wave and for CTIS it was approximately
250,000 per week.

Despite their large sizes, both the Census HPS and Delphi-Facebook CTIS
produced substantially biased estimates of vaccine uptake in the U.S. in early
2021 (Nguyen2021; Bradley2021). As shown in Figure 1, the weighted estimates
from these surveys consistently overestimated vaccine uptake (the percentage
of U.S. adults reporting receiving at least one dose of a COVID-19 vaccine) as
compared to benchmark data retrospectively available from the U.S. Centers for
Disease Control and Prevention (CDC) (CDCdata). Bradley2021 decomposed the
error in the survey estimates of vaccine update for both surveys using the
framework of Meng2018, emphasizing the danger of very large samples leading to
very precise (negligible confidence interval length) but severely biased
results.

Figure 1: Survey weighted estimates of COVID-19 vaccine uptake for adults in
the U.S. in 2021 compared to CDC benchmark data (grey line), plotted by the
end date of each survey wave. Intervals are 95% CIs; for Delphi-Facebook CTIS
the CIs are too small to be visible.

Importantly, while these two surveys resulted in large samples, they had very
small response rates. In the period from January through May 2021, unweighted
response rates for the HPS were in the range of 6.6-7.8% (HPStechdoc).
Response rates are not available for the CTIS, but daily cooperation
rates111Percent of all Facebook users who logged onto Facebook and received an
invitation on a given day who then answered the symptoms question on that day
were approximately 0.5-1.5% (CITSmethods). With such small response rates, the
protection against bias afforded by probability sampling is erased, and these
surveys in many ways resemble nonprobability samples (e.g., convenience
samples). A detailed analysis of nonresponse for the HPS (HPSNR2020) showed
that response rates differed across demographic domains (e.g., age, race,
ethnicity). Post-survey weighting adjustments were used for both surveys to
attempt to correct for differential nonresponse, but were limited to a small
set of demographic characteristics. Given that these weighting adjustments
failed to produce unbiased estimates, and with such small response rates, we
hypothesized that a _non-ignorable_ nonresponse mechanism might have been
responsible at least in part for the biased estimates.

In the context of measuring vaccine uptake, if an individual’s propensity to
respond to either the HPS or CTIS is at least in part a function of their
vaccine status, this constitutes a _non-ignorable_ nonresponse mechanism.
Specifically, it is plausible that people who were “anti-vaccine” (and thus
were unvaccinated) were less likely to complete these surveys on the impact of
the COVID-19 pandemic on daily life. One could also hypothesize that
individuals who were anti-vaccine might also be suspicious of the government
and thus less likely to respond to the HPS, which was an official government-
sponsored survey.

In order to assess whether this type of non-ignorable nonresponse may have
been occurring, we use previously developed _proxy pattern-mixture models
(PPMMs)_ (AndridgeLittle2011; AndridgeLittle2020), which allow for estimation
under a non-ignorable nonresponse assumption, to estimate vaccine uptake using
data from both surveys. In Section 2 we describe the HPS and CTIS in more
detail. In Section 3 we briefly review the PPMM, and present results from
applying it to estimate vaccine uptake in Section 4. In Section 5 we use the
PPMM to estimate vaccine hesitancy, a measure for which we do not have a
benchmark truth. We conclude in Section 6 with discussion of how the PPMM
could have been used prospectively as part of a nonresponse bias assessment
and describe factors that would facilitate such analyses in the future.

##  2 Details on the COVID-19 Vaccine Surveys

###  2.1 Census Household Pulse Survey

The Census Household Pulse Survey (HPS) was an experimental data product of
the U.S. Census Bureau that was developed in the early phase of the COVID-19
pandemic in conjunction with (https://www.census.gov/data/experimental-data-
products/household-pulse-survey.html). The first phase of this survey launched
on April 23, 2020 with the goal of quickly and efficiently collecting data
about how the pandemic was affecting the lives of individuals residing in the
United States, and was still ongoing as of March 2023. Survey questions asked
about experiences that may be affected by the pandemic, with a focus on
employment status, food security, housing security, physical and mental
health, and educational disruption (Fields2020). Starting in January 2021,
when COVID-19 vaccines became available, questions were added about
vaccination status and intention. Table 1 lists the questions used to estimate
vaccine uptake and vaccine hesitancy.

Given the goal of quick survey deployment and results dissemination as well as
the context (during the pandemic), all data collection was via web. The HPS
consisted of repeated, stratified, cross-sectional random samples with a
target population of all adults (18+) residing in housing units in the U.S.
(excluding Puerto Rico). As with many demographic surveys conducted by federal
statistical agencies, the HPS sampled households from the Census Bureau’s
Master Address File (MAF). However, due to the online-only design, only
addresses on the MAF that had a linked cell phone number and/or email address
(from the Census Bureau Contact Frame) were eligible for sampling due to the
online-only survey design. Approximately 80% of housing units on the MAF had a
cell phone and/or email address (https://www.census.gov/programs-
surveys/household-pulse-survey/technical-documentation.html). Initially,
samples were drawn weekly from the MAF, with a shift to bi-weekly samples in
August 2020. The sample was stratified by geographic area (50 states,
Washington D.C., top 15 Metropolitan Statistical Areas). Sampled individuals
were contacted by text and/or email with a request to complete the survey.

Table 1: Survey questions about vaccine uptake and intention in the Census
Household Pulse Survey and Delphi-Facebook CTIS, January 2021 - May 2021
Census Household Pulse Survey
---
Uptake |  Question: “Have you received a COVID-19 vaccine?”
|  Response Options: “Yes”, “No”
Intention |  Question: “Once a vaccine to prevent COVID-19 is available to you, would you…” [only asked if did not respond “Yes” to uptake question]
|  Response Options: “Definitely get a vaccine”, “Probably get a vaccine”, “Be
unsure about getting a vaccine”*, “Probably NOT get a vaccine”, “Definitely
NOT get a vaccine”
Delphi-Facebook CTIS
Uptake |  Question: “Have you had a COVID-19 vaccination?”
|  Response Options: “Yes”, “No”, “I don’t know”
Intention |  Question: “If a vaccine to prevent COVID-19 were offered to you today, would you choose to get vaccinated?” [only asked if did not respond “Yes” to uptake question]
|  Response Options: “Yes, definitely”, “Yes, probably”, “No, probably not”,
“No, definitely not”
*option added mid-April 2020

We analyzed iterations of the HPS conducted from January 6, 2021 through May
10, 2021. During this time period, approximately 1,000,000 housing units were
sampled in each data collection period with 68,000-80,000 respondents per
wave.

Several post-survey adjustments were made to the HPS base weights to produce
the final analytic weights, including adjustments for nonresponse,
undercoverage, and a conversion from household-level to person-level weights
(Fields2020). As a last step, an iterative raking procedure was used to ensure
that weighted totals match the U.S. adult population with respect to specified
demographic characteristics. Specifically, weights were raked to two sets of
population totals from the 2019 American Community Survey: educational
attainment by age and sex222Surveys conducted by the U.S. federal government
historically have collected sex as a binary variable and without nuance, i.e.,
conflating it with gender. We acknowledge this limitation. within state, and
race/ethnicity by age and sex within state.

###  2.2 Delphi-Facebook COVID-19 Trends and Impact Survey

The Delphi-Facebook COVID-19 Trends and Impact Survey (CTIS)
(https://delphi.cmu.edu/covid19/ctis/) was developed in the early phase of the
COVID-19 pandemic as a collaboration between Meta (Facebook’s parent company)
and the University of Maryland and Carnegie Mellon University (Barkay2020).
The survey launched on April 6, 2020 and ended on June 25, 2022. The stated
main goal of the surveys was to collect real-time indicators of symptom
severity, both individual and household-level (Kreuter2020). Starting in
January 2021, questions about vaccination status and intention were added,
with the exact wording as shown in Table 1.

The CTIS was a large, stratified, cross-sectional random samples, drawn daily,
with a target population of all adults (18+). The survey was implemented in
over 200 countries; we only use data from the U.S. in our analyses. The
sampling frame was all Facebook users (18+) who had been active on Facebook in
the previous month. Samples were drawn daily, and the survey invitation was
shown at the top of the Facebook feed for selected individuals(Salomon2021).
In the U.S. the sample was stratified by state.

We pooled the daily CTIS samples into weeks and analyzed the weeks ending in
January 16, 2021 through May 8, 2021. During this time period, an average of
approximately 290,000 respondents provided at least partial responses to the
survey each week.

Multiple post-survey adjustments were made to the CTIS base weights to account
for nonresponse and non-coverage (due to the fact that not all of the target
population are Facebook users) (CITSweights). First, inverse propensity score
weighting was used to adjust for nonresponse within the sampling frame (the
Facebook user base) using age and gender as predictors of response status.
Then post-stratification was used to ensure weighted totals match the target
population with respect to age by sex within state using the Current
Population Survey 2018 March Supplement for population totals (Barkay2020).

##  3 Methodology: The Proxy Pattern-Mixture Model

The proxy pattern-mixture model (PPMM) was originally proposed by
AndridgeLittle2011 as a tool for assessing the potential impact of non-
ignorable nonresponse on estimating means of continuous variables, primarily
in the context of item nonresponse. It was subsequently extended to estimating
proportions by AndridgeLittle2020. The PPMM has also been used as the basis
for indices that quantify the potential for non-ignorable selection bias for
means (LittleWest2020), proportions (Andridge2019), and regression
coefficients (West2021) estimated from nonprobability samples. Our goal is to
estimate a proportion – the proportion of U.S. adults who have had at least
one dose of a COVID-19 vaccine – thus we use the binary PPM of
AndridgeLittle2020 in our analyses. We briefly describe their methodology here
in the context of estimating vaccine uptake and refer readers to
AndridgeLittle2020 for additional details.

Let $Y_{i}$ be the binary indicator of whether individual $i$ in the
population of U.S. adults (18+) has received at least one dose of a COVID-19
vaccine. A single iteration of either the HPS or CTIS collects $Y_{i}$ from a
subset of the population, and let $S_{i}$ be the sample inclusion indicator
that takes the value $S_{i}=1$ if the individual is sampled and responds
(provides a value of $Y_{i}$) and 0 otherwise. Since only a small fraction of
sampled individuals responded to the survey, the $S_{i}$ we observe is a
combination of the design-based sample inclusion probability (which we know)
and an unknown response propensity (which we do not know). Thus the
probability density of $S_{i}$ is unknown without additional assumptions. In
the PPMM analysis we will make assumptions about the distribution of $S_{i}$
through a principled sensitivity analysis. In what follows, we refer to the
units with $S=1$ as the “responding sample,” and note that the units with
$S=0$ include both individuals who were sampled but did not respond and
individuals who were not sampled.

Crucial to the implementation of the PPMM, we must also observe covariate
information at the individual level for the responding individuals and in
aggregate for the rest of the population. Let $Z_{i}=(Z_{i1},Z_{i2},\dots
Z_{ip})$ be a set of $p$ covariates collected on the survey, which for our
purposes will be limited to information we can also obtain in aggregate for
the U.S. population, i.e., demographic data. In the PPMM approach, this
covariate data for respondents is reduced to a single _proxy_ variable $X$ by
regressing $Y$ on $Z$ using a probit regression model and taking $X$ to be the
estimated linear predictor from this regression. Importantly, individual-level
values of $X_{i}$ are available for all responding individuals $(S_{i}=1)$, as
their $Z$ values can be plugged into the estimated probit regression equation.
We do not observe $X_{i}$ for nonresponding individuals, but if we have the
mean and variance of $Z$ for this part of the population from an external
source then we can estimate the mean and variance of $X$ for the nonresponding
portion of the population. Despite the large sample sizes of the HPS and CITS
surveys, the samples are considerably smaller than the size of the full
population, i.e., sampling fractions are small. Therefore, estimates of the
mean and variance of $Z$ for the entire population of U.S. adults are
effectively the same as estimates for the part of this population that did not
respond to a single wave or week of the HPS or CTIS surveys.

The basic idea of the PPMM is that we can measure the degree of bias present
for the respondent sample mean of the proxy $X$ by comparing it to the
population-level mean of $X$ (based on the aggregate information for $Z$). If
$X$ is correlated with $Y$, then this provides some information about the
potential bias in the respondent sample mean of $Y$. If $X$ and $Y$ are highly
correlated, then a small bias in $X$ suggests (but does not guarantee) a small
bias in $Y$. If, however, $X$ and $Y$ are weakly correlated (which would occur
if the covariates $Z$ that create $X$ are not very predictive of $Y$) then we
simply do not have much evidence for or against bias in the respondent sample
mean of $Y$. Fortunately, many studies have shown that demographics available
in aggregate at the national level such as age, sex, race/ethnicity, and
education are moderately associated with COVID-19 vaccine acceptance (e.g.,
Reiter2020; Haile2022).

The PPMM does not directly model $Y$ and $X$, but instead introduces a
normally distributed latent variable, $U$, such that $Y=1$ when $U>0$, and
models the joint distribution of $U$ and $X$. Specifically, AndridgeLittle2020
use a bivariate normal pattern-mixture model for the joint distribution of $U$
and $X$ given $S$, in which the mean and variance parameters are distinct for
$S=1$ and $S=0$. Parameters of this joint distribution are fully identified
for the responding sample, with the exception of the mean and variance of the
latent $U$ which cannot be separately identified; as in AndridgeLittle2020 we
fix the variance of $U$ at one. For the nonresponding portion of the
population $(S=0)$ we can identify the mean and variance of $X$, but not the
parameters describing the distribution of $U$ or the correlation between $X$
and $U$.

The unidentified parameters of the PPMM can be identified by making an
assumption about the distribution of $S$ and with the introduction of a
sensitivity parameter, $\phi$. AndridgeLittle2020 show that the PPMM is just
identified if we assume that the probability an individual is sampled and
responds is an unspecified function of a known linear combination of $X$ and
$U$, plus potentially other observed covariates $V$ that are independent of
$U$ (and $Y$) and $X$:

$\Pr(S=1|U,X,V)=g\left((1-\phi)X^{*}+\phi U,V)\right)$ (1)

Here $X^{*}$ is the proxy, $X$, rescaled to have the same variance as $U$ for
$S=1$, and $\phi\in[0,1]$ is the sensitivity parameter. For a specified value
of $\phi$, the parameters of the PPMM are just identified, and thus the
overall mean of $Y$ can be estimated as a weighted (by the responding
fraction) average of estimates of $E[Y|S=1]=E[U>0|S=1]$ and
$E[Y|S=0]=E[U>0|S=0]$. Though there is no information in the data with which
to estimate $\phi$, certain values of $\phi$ correspond to specific types of
response mechanisms, thus enabling a reasonable, bounded sensitivity analysis.
Specifically, $\phi=0$ corresponds to a missing at random assumption
(Rubin1987), where the probability of response is only a function of $X$ and
$V$, which are observed – this is an ignorable response mechanism. If
$\phi>0$, then response depends at least in part on $U$, and therefore on $Y$
– a non-ignorable response mechanism.

AndridgeLittle2020 provide an explicit formula for the overall mean of $Y$
under the PPMM as a function of the parameters of the underlying normally-
distributed latent $U$ for respondents $(\mu_{u}^{(1)})$ and nonrespondents
$(\mu_{u}^{(0)},\sigma_{uu}^{(0)})$ and the fraction of the population that
responded ($\pi$),

$\displaystyle\mu_{y}$
$\displaystyle{}=\pi\Phi\left(\mu_{u}^{(1)}\right)+(1-\pi)\Phi\left(\mu_{u}^{(0)}\big{/}\sqrt{\sigma_{uu}^{(0)}}\right),$
(2)

where $\Phi(z)$ denotes the CDF of the standard normal distribution evaluated
at $z$. With the identifying restriction in (1), the mean and variance of $U$
for nonrespondents are given by

$\displaystyle\mu_{u}^{(0)}$
$\displaystyle{}=\mu_{u}^{(1)}+\left(\frac{\phi+(1-\phi)\rho_{ux}^{(1)}}{\phi\rho_{ux}^{(1)}+(1-\phi)}\right)\left(\frac{\mu_{x}^{(0)}-\mu_{x}^{(1)}}{\sqrt{\sigma_{xx}^{(1)}}}\right)$
(3) $\displaystyle\sigma_{uu}^{(0)}$
$\displaystyle{}=1+\left(\frac{\phi+(1-\phi)\rho_{ux}^{(1)}}{\phi\rho_{ux}^{(1)}+(1-\phi)}\right)^{2}\left(\frac{\sigma_{xx}^{(0)}-\sigma_{xx}^{(1)}}{\sigma_{xx}^{(1)}}\right).$
(4)

Here $\mu_{x}^{(j)}$ and $\sigma_{xx}^{(j)}$ are the mean and variance of the
proxy $X$ for $S=j,j=\\{0,1\\}$ and $\rho_{ux}^{(1)}$ is the correlation
between $U$ and $X$ in the respondent sample.

Insight into how the PPMM works can be seen by closer inspection of Equations
(2)-(4). In (3), the mean of latent $U$ for the nonresponding portion of the
population $(\mu_{u}^{(0)})$ is the respondent mean $(\mu_{u}^{(1)})$, shifted
by a factor that depends on the sensitivity parameter $\phi$, the strength of
the proxy as captured by the correlation between $X$ and $U$ in the respondent
sample $(\rho_{ux}^{(1)})$, and how different the proxy mean is for
respondents $(\mu_{x}^{(1)})$ and nonrespondents $(\mu_{x}^{(0)})$. Larger
differences in proxy means between respondents and nonrespondents will lead to
larger shifts of the mean of $U$. The amount of shift is also governed by
$\phi$, and at the two extremes of $\phi=0$ and $\phi=1$ the first term in the
parentheses in (3) is $\rho_{ux}^{(1)}$ and $1/\rho_{ux}^{(1)}$, respectively.
Thus, the larger the correlation $\rho_{ux}^{(1)}$, the smaller the range of
the shift as $\phi$ goes from 0 to 1. If the proxy is weak, however, this term
will produce a wide range for $\mu_{u}^{(0)}$ as $\phi$ is varied. A similar
shifting occurs for the variance of $U$ for nonrespondents as seen in (4).

For model estimation we use the Bayesian approach described by
AndridgeLittle2020, which puts non-informative priors on all identified
parameters in the PPMM to obtain draws of the overall mean of $Y$ via a Gibbs
sampler. Since the data contain no information to inform $\phi$, we use a
Uniform(0,1) prior, which generates a 95% credible interval for the mean of
$Y$ that effectively averages over all possible values of $\phi$. The
posterior median serves as an estimate of the mean of $Y$ for $\phi=0.5$,
which was recommended by LittleWest2020 as a “point index” if a single point
estimate is desired under a non-ignorable response mechanism.

##  4 Applying the PPMM to Estimate Vaccine Uptake

As described in Section 3, application of the PPMM requires aggregate
information for covariates $Z$ that are also available in the HPS and CTIS
survey data. We used the American Community Survey (ACS) 2019 data obtained
via IPUMS USA (IPUMSACS) for population-level data on the following covariates
available in both the HPS and CTIS: age, gender, education, race, and
ethnicity. The categories for all of these covariates differed slightly
between HPS and CTIS, so separate estimates of the population mean and
variance were made using the ACS that matched each survey; see Supplemental
Table S1 for the coding of variables across data sources. We note that income
was also available in both the HPS and the ACS, but as is typical for this
variable it had relatively high rates of missingness in the survey data with
approximately 25% of respondents not providing their income, and thus we
elected not to use this to create the proxy.

Our responding sample for each survey was taken to be the set of records that
had information on vaccination status $(Y)$ and complete covariate data $(Z)$,
as the PPMM requires complete data for the respondent sample. We followed the
procedures used by the respective surveys when producing their vaccination
estimates in terms of how missing data in $Y$ was handled. For the HPS, an
individual with a missing $Y$ value was assumed to be a “no, not vaccinated”
and was included in the sample, whereas for the CTIS an individual with
missing $Y$ was dropped from the sample ($\approx$6-7%). For covariate data,
the publicly available HPS data had our $Z$ variables already singly imputed
(since they were part of the Census’ weighting adjustments) and thus there
were no records with missing $Z$ values. In contrast, the CTIS suffered from
missing data for the demographic variables that came at the very end of the
survey, with approximately 15% additional records being dropped. Due to the
very large size of the CTIS surveys, analysis sample sizes were still very
large, ranging from 167,000 to 290,000 across weeks. We note that the survey
weights provided with each survey are not used for the PPMM analyses, and
instead the responding sample is treated effectively as a non-probability
sample.

As previously noted, sampling fractions for both the HPS and CTIS were small
and thus we used the mean and variance of $Z$ from the ACS for the
nonrespondent portion of the population, though technically these values are
for the full population. Additionally, we treat the means from the ACS as
though they were “known” despite them being estimates themselves; future work
is needed to incorporate uncertainty about the $Z$ at the population level
into PPMM estimation.

As a benchmark truth for the proportion of the population that had received at
least one dose of a COVID-19 vaccine we used the vaccination uptake statistics
available from the CDC as used by Bradley2021 and available via their GitHub
repository (https://github.com/vcbradley/ddc-vaccine-US). As noted in
Bradley2021, this benchmark data itself is potentially subject to error,
though retroactive corrections are included in these counts.

Figure 2: Posterior medians for the biserial correlation $(\rho^{(1)})$
between COVID-19 vaccination uptake (binary $Y$) and proxy $X$ for the
selected sample under the proxy pattern-mixture model. Bounds shown are 95%
credible intervals (too small to see for Delphi-Facebook CTIS).

Figure 2 shows the estimated proxy strength, i.e., the estimated correlation
between $U$ and $X$ for respondents in both the HPS and CTIS during January
through May of 2021. In the earlier waves, when vaccines were first available
only to limited groups (e.g., older adults), the model that builds the proxy
is relatively weak (around $\hat{\rho}_{ux}^{(1)}=0.25$). As vaccines became
more widely available, the proxy strength increases, to a high of slightly
larger than $\hat{\rho}_{ux}^{(1)}=0.5$, with a small decrease in April and
May.

Figure 3 shows the estimates of vaccine uptake under the PPMM with a
Uniform(0,1) prior on the sensitivity parameter $\phi$ for both surveys,
compared to the CDC benchmark and the direct survey (weighted) estimates.
Several patterns are evident in the results. First, the upper endpoint of the
credible intervals corresponding to $\phi=0$ is nearly identical to the
weighted estimates for HPS, which is expected since the covariates $Z$ that
created the proxy are the same as those used in the weighting adjustments. For
the CTIS, the interval endpoint is slightly lower than the direct estimates,
as a result of our PPMM using _more_ information than the survey weights which
only used age and gender, since education and race/ethnicity (used in the
PPMM) were predictive of vaccine uptake.

Figure 3: Estimates of vaccine uptake using the proxy pattern-mixture model
(PPMM) with a Uniform(0,1) prior on the sensitivity parameter $\phi$, for both
the Census HPS and the Delphi-Facebook CTIS. Shown are the posterior medians
with 95% credible intervals. The grey line is the benchmark CDC data (the
“truth”).

Second, the PPMM credible intervals cover the benchmark truth for both surveys
in all waves/weeks, while the direct survey estimates only cover the truth
twice (the first two waves of the HPS). Importantly, the PPMM correctly
detects the _direction_ of bias for both surveys in all waves/weeks, i.e., the
PPMM indicates that the direct estimates were overestimating the true
proportion of adults who had at least one vaccine dose. For the CTIS, the
posterior median proportion (corresponding to $\phi=0.5$) is remarkably close
to the truth across all waves; for the HPS this “point index” value is too low
(i.e., overcorrects the bias) in the earlier waves when the HPS direct
estimates are not as biased.

Finally, the PPMM credible intervals are much wider than the confidence
intervals for the survey estimates despite the very large sample sizes. This
is a desirable property, since one of the problems highlighted by Bradley2021
is the “big data paradox” of Meng2018: “The bigger the data, the surer we fool
ourselves.” The relatively larger intervals of the PPMM reflect the strength –
or weakness – of the proxy model. Since the covariate data $Z$ are only
moderately associated with $Y$, our confidence in how much non-ignorable
nonresponse bias might be present is only moderate, corresponding to larger
credible intervals.

##  5 Applying the PPMM to Estimate Vaccine Hesitancy

We also used the PPMM, with the same set of covariates $Z$ and same external
population source, to estimate the proportion of U.S. adults who were vaccine
hesitant for both the HPS and CTIS data. Individuals who reported that they
would “probably not” or “definitely not” choose to be vaccinated or were
“unsure” (HPS only) were coded as being vaccine hesitant (see Table 1 for
exact question wording and response options). Individuals who either had
received a vaccine dose or who “definitely” or “probably” would do so were
coded as not being vaccine hesitant.

Proxy strength for the models for vaccine hesitancy was relatively stable both
across time and between surveys. The posterior median for $\rho^{(0)}$ for the
HPS ranged from 0.392 to 0.415 across waves. For the CTIS, $\rho^{(0)}$ was
largest at the earliest time point (0.391) and slightly declined across the
time, with the smallest posterior median at the last time point (0.332). As
such, the proxy for vaccine hesitancy was generally weaker than the proxy for
vaccine uptake. The full set of estimates are available in Supplemental Figure
S1.

Results of applying the PPMM are shown in Figure 4. As one might hypothesize,
given that vaccine uptake was overestimated by these surveys, the PPMM
suggests that vaccine hesitancy is _underestimated_ by a relatively stable
amount across time. Using the posterior median as a point estimate under a
non-ignorable response mechanism, the results suggest that vaccine hesitancy
is being underestimated by around 9 percentage points on average for the HPS
and by around 7 percentage points on average for the CTIS. As expected due to
the relatively weak proxy, the credible intervals are large, averaging
approximately 40 percentage points wide for HPS and 30 percentage points wide
for CTIS. Nonetheless, this provides some evidence that the survey estimates
may be too optimistic when it comes to estimating vaccine hesitancy if
nonresponse is non-ignorable.

Figure 4: Estimates of vaccine hesitancy using the proxy pattern-mixture model
with a Uniform(0,1) prior on the sensitivity parameter $\phi$, for both the
Census HPS and the Delphi-Facebook CTIS. Shown are the posterior medians with
95% credible intervals.

##  6 Discussion

In this analysis of two large surveys that substantially overestimated vaccine
uptake in the U.S. in early 2021, the PPMM correctly detected the direction of
bias for all survey waves. This suggests that non-ignorable nonresponse is a
plausible explanation for the bias – individuals who were not vaccinated were
less likely to respond to these surveys. In addition to correctly detecting
the direction of bias, median posterior estimates from the PPMM, corresponding
to $\phi=0.5$ (previously suggested as a way to obtain a single estimate under
the PPMM) were remarkably accurate. For the Delphi-Facebook CTIS, PPMM
estimates with $\phi=0.5$ were close to the retrospectively available
benchmark truth in all survey waves. For the Census HPS, estimates for
$\phi=0.5$ were very close to the truth in the last two waves, when the true
bias was the largest.

The success of the PPMM in the vaccine uptake context is in part due to the
fact that the factors available at the population level, i.e., demographics,
were moderately predictive of the outcomes of interest. If other outcomes on
the same surveys are not as strongly associated with demographic
characteristics then the proxies will be weaker. Having a weak proxy means
that credible intervals from the PPMM will be relatively wide, and the
analysis will be less informative. Nonetheless, the present analysis
highlights the fact that demographic data alone can in fact provide enough
information for a meaningful sensitivity analysis and provide reasonable
bounds on the potential bias.

Importantly, the data necessary for a sensitivity analysis based on the PPMM
are data that would be readily available in most scenarios. The only
additional data needed beyond the survey microdata itself (from respondents)
are population-level means and variances for the variables that create the
proxy. In most cases these would be available while the survey data is first
being analyzed. In fact, in many cases these population margins will be the
same as what would be used for post-survey weighting adjustments.

Another reason for the success of the PPMM in our context is that the target
population is a relatively stable and clearly defined population for which
summary statistics are readily available. This may not always be the case. For
example, when applying the PPMM to pre-election polling we found very strong
proxies $(\rho^{(0)}\geq 0.9)$ (WestAndridge2023). However, the challenge
there was in defining the population of interest. A pre-election poll attempts
to make inference to a dynamic population of “likely voters.” Finding
aggregate data for such a population is a major challenge, unlike the
relatively simple task of finding demographic summaries for all adults in the
U.S. in the vaccine uptake application.

Overall, this _retrospective_ analysis provides evidence that the PPMM could
be used as a method for _prospective_ assessment of the potential for non-
ignorable nonresponse bias. In most cases, a benchmark truth will not be
available, but this application suggests that the PPMM can in fact capture the
truth in a “real data” setting. Our analysis also provides support for
LittleWest2020’s recommendation of $\phi=0.5$ as a reasonable point estimate,
a “moderately non-ignorable” mechanism that falls halfway between the
ignorable $(\phi=0)$ and most extremely non-ignorable $(\phi=1)$ sensitivity
bounds.

## Data Availability

Census HPS microdata are publicly available for download from
https://www.census.gov/data/experimental-data-products/household-pulse-
survey.html. Delphi-Facebook CTIS individual-level microdata are available to
eligible academic and nonprofit researchers with fully executed data use
agreements, see https://dataforgood.facebook.com/dfg/docs/covid-19-trends-and-
impact-survey-request-for-data-access. The HPS data used in this paper, along
with code to replicate the analyses, are available at
https://github.com/randridge/PPMA, along with code only for the Delphi-
Facebook analyses.


# Selective Undercut of Undoped Optical Membranes for Spin-Active Color
Centers in 4H-SiC

###### Abstract.

Silicon carbide (SiC) is a semiconductor used in quantum information
processing, microelectromechanical systems, photonics, power electronics, and
harsh environment sensors. However, its high temperature stability, high
breakdown voltage, wide bandgap, and high mechanical strength are accompanied
by a chemical inertness which makes complex micromachining difficult.
Photoelectrochemical etching is a simple, rapid means of wet processing SiC,
including the use of dopant selective etch stops that take advantage of mature
SiC homoepitaxy. However, dopant selective photoelectrochemical etching
typically relies on highly doped material, which poses challenges for device
applications such as quantum defects and photonics that benefit from low
doping to produce robust emitter properties and high optical transparency. In
this work, we develop a new, selective photoelectrochemical etching process
that relies not on high doping but on the electrical depletion of a fabricated
diode structure, allowing the selective etching of an n-doped substrate wafer
versus an undoped epitaxial ($N_{a}=1(10)^{14}cm^{-3}$) device layer. We
characterize the photo-response and photoelectrochemical etching behavior of
the diode under bias and use those insights to suspend large ($>100\mu m^{2}$)
undoped membranes of SiC. We further characterize the compatibility of
membranes with quantum emitters, performing comparative spin spectroscopy
between undoped and highly doped membrane structures, finding the use of
undoped material improves ensemble spin lifetime by $>3x$. This work enables
the fabrication of high-purity suspended thin films suitable for scalable
photonics, mechanics, and quantum technologies in SiC.

Jonathan Dietz, Amberly Xie, Aaron M. Day, Evelyn L. Hu

John A. Paulson School of Engineering and Applied Sciences
Harvard University, Cambridge, MA. 02138, USA

##  1\. Introduction

A high-yield, systematic means of producing high-quality thin-films is an
outstanding challenge in realizing scalable quantum and classical technology.
The availability of single crystal thin films with high purity and optical
quality has led to large advances in miniaturizing typically bulk optical
components with equivalent integrated photonic devices and the development of
micro-electromechanical circuits.[1, 2, 3, 4] One such approach to creating
thin films in semiconductor materials with effective p and n-type doping, is
dopant selective photoelectrochemical (PEC) or electrochemical etching of one
doping type over the other in an epitaxial or implanted homojunction.[5] Such
etching strategies have been designed in silicon, gallium nitride, gallium
arsenide, and silicon carbide homojunctions.[6, 7, 8, 9] However, these
approaches rely on heavily doped device layers, as lowering the doping of the
device layer reduces etching selectivity.[9] A dopant selective
electrochemical etch that does not require high device layer doping is
desirable in applications where high doping degrades performance or design
flexibility, such as electronics, photonics, and mechanics or in material
systems where effective doping is not available for either p-type or n-type,
such as zinc oxide or diamond respectively.[10, 11, 12, 13]

This work presents such a process, with a particular application to Silicon
Carbide (SiC), a wide band-gap semiconductor widely studied for use in
photonics, mechanics, and quantum information[14, 15, 16, 17]. Consequently,
SiC is such a material that could benefit from a PEC etching process that
results in a low doped thin film. For example, SiC hosts several solid-state
quantum defect qubits which show lifetime limited optical linewidths, long
spin coherence, and high initialization fidelity which could form the basis of
quantum memory nodes in a quantum network.[18, 19] Dopant-selective PEC has
successfully created suspended photonics in SiC with integrated defects, yet
these devices and defects demonstrate degraded optical and spin performance,
respectively due to the required use of high doping concentration.[20, 21, 22]
This requirement is predicated on the ability to independently control band
bending in each material doping type with respect to the chemical potential of
decomposition in the etching electrolyte, which is more favorable in layers
with large differences in doping.[23]

We present a modified PEC process, that uses the bias of a simple transparent
Shottky barrier diode to enable independent electrochemical control of each
material layer. We show selective etching between an undoped layer of 4H-SiC
and a highly doped substrate, demonstrating high etch rates ($>4\mu m/hr$),
low surface roughness ($\sim 1nm$ RMS), and large areas ($>100\mu m^{2}$) of
undercut material. We further demonstrate the immediate device performance
advantages of this process over a process that relies high doping of the
device layer alone, comparing the quantum spin performance of ensembles of
silicon monovacancies in doped and undoped layers . When an undoped layer is
used, lower strain, longer spin coherence, and lower inhomogeneous broadening
is observed. The reliance of this technique on an externally fabricated diode
device rather than solely on material properties indicates that this technique
could be adapted to other material systems where junctions can be fabricated.

##  2\. Main

Figure 1. (a) Scanning electron microscope (SEM) image of a $100\mu m^{2}$
membrane with support structures imaged at 30keV to reveal substrate
structure. (b) focused ion beam (FIB) cross section of membrane showing
uniform membrane thickness. (c) Electrochemical impedance spectroscopy of an
undoped, $500nm$ thick epilayer of SiC showing low p-type conductivity,
determined from linear fitting ($1/C^{2}$) and measured using an Al contact.
(d) configuration of the PEC cell used to perform CV and etching of 4H-SiC
structures. $C$, $R$, and $W$ are the Pt counter, Ag/AgCl reference, and SiC
working electrodes respectively. u-SiC denotes the undoped nature of the p-
layer (e) I-V curves of the photodiode formed by the ITO contacted undoped
layer and Ni contacted substrate. Inset: illumination dependent photo-biasing
of the diode.

Due to its highly chemically inert nature, suspended SiC photonic and
mechanical devices are often created using SmartCut, grind-and-polish, deep
backside etching, or via dopant selective photo-electro-chemical etching
(PEC).[2, 24, 25, 26, 27] These techniques have enabled fabrication of high-
quality devices, but performance is typically limited by wafer non-uniformity,
yield, excess strain and charge noise. PEC usually relies on high doping
contrast between epitaxial layers to selectively etch, for example, a highly
n-doped substrate from a comparably p-doped epilayer, with the epilayer as the
ultimate host of both photonic devices and quantum defects. However, free
carrier absorption from highly doped material and charge noise creates optical
propagation loss and limits qubit spin coherence.[22] Previously fabricated
devices demonstrated high quality optical resonances, but suffered from high
strain, roughened surfaces and poor defect coherence.[22, 28]

The photoelectrochemical etching of single crystal SiC is well understood for
both n and p doping types in alkaline solutions. In both cases, etching occurs
through the elimination of silicon through the formation of a soluble silicate
product and carbon monoxide via the charge exchange of holes with hydroxide
ions in solution[29]:

(1) $SiC+8OH^{-}+6h^{+}\rightarrow[Si(OH)_{2}O_{2}]^{2-}+CO+3H_{2}O$

In p-type material, electrochemical etching can occur without illumination,
because holes are a majority carrier in the material.[30] Because the valence
band in p-type material typically bends in energy downwards towards the
chemical potential of the electrolyte, the onset of etching occurs when the
cell bias lowers the barrier holes experience in reaching this interface. By
contrast, in n-type material, etching does not occur unless the SiC is
illuminated to generate the requisite holes, and though the onset of etching
can occur without cell bias, it is often beneficially controlled by the
application of such a bias.[29] Etching rate is limited by the rate of
reaction at the semiconductor-electrolyte interface, controlled by the
quantity and diffusion rate of ions in solution and holes in the semiconductor
towards the interface. The diffusion rate of ions to the interface is
controlled by mass transport in the bulk of the solution and the potential
drop that ions experience at the Helmholtz double layer. In the semiconductor,
the diffusion of holes to the interface is controlled by the built-in voltage
of the semiconductor-electrolyte junction and the voltage difference between
the semiconductor (working) electrode and the counter electrode. The number of
holes is largely determined by the intensity of illumination.
Photoelectrochemical etching of silicon carbide is stable if anodic oxidation
does not outpace the dissolution by charge exchange, in which case the etch is
passivated by an insulating layer of $SiO_{2}$.[29] Additionally, the etching
rate can be controlled by crystallography. In hexagonal polytypes of SiC, the
carbon-face of the material is known to be significantly more reactive than
the silicon-face of the material and consequently has a significantly faster
PEC etch.[31] p-type (n-type) material may be selectively etched against
relatively n-type (p-type) material by appropriate selection of cell bias by
setting a more negative (positive) cell bias which fixes the energy position
of the valence band relative to the electrolyte‘s chemical potential,
potentially yielding suspended structures if the substrate is targetted in the
etch (Figure 1a-b). In a homojunction that combines both doping types this
relies on a high doping difference (e.g. $N_{a}=1(10)^{14}cm^{-3}$ vs.
$N_{d}=1(10)^{14}cm^{-3}$) and allows a $0.5V$ potential window for
selectively etching relatively thick layers of SiC.[23]

Figure 2. (a) Cyclic voltamograms ($20mVs^{-1}$) of etched mesa structures
under open circuit and various diode reverse biases, measured on the n-type
contact (red, orange, and yellow) or p-type contact (indigo, purple). (b) Etch
rates for different KOH concentrations as a function of diode reverse bias.
(c) FIB cross section of mesa structures before PEC etching and after 2 hours
of PEC etching with a working electrode bias of 0.2V and an partially depleted
($-0.75V$) and fully depleted ($-1.5V$) reverse bias.

Unfortunately, in addition to the constraints on applications imposed by the
high doping spread necessary for selective etching, doping selectivity in a
homojunction depends on the ability to mutually bias both doping types in the
cell. While this assumption is reasonable in thick layers or under low
illumination conditions, it breaks down in epilayer thicknesses approaching
useful thicknesses in photonic devices because of strong photo-biasing of the
homojunction. To assess this effect, we construct a junction from a thin,
$500nm$ undoped layer of 4H-SiC epitaxially deposited onto an n-type
($3(10)^{18}$ nitrogen doped) substrate wafer. Contact is made to the n-type
substrate by thermally evaporating $200nm$ of nickel on the substrate backside
followed by subsequent rapid thermal annealing for 5 minutes at
950${}^{\circ}C$ in an argon atmosphere. Contact is made to the unintentional
doped layer by thermally evaporating $200nm$ of aluminum onto the epilayer,
with a circular aperture in the center of the sample left masked. The mask is
then stripped, and contact is made by rapid thermal annealing for 5 minutes at
950${}^{\circ}C$ in an argon atmosphere. The sample is then immersed in $2M$
KCl solution with the Al contact covered by a silicone o-ring. The carrier
concentration and conductivity type of the epilayer is determined using
electrochemical impedance spectroscopy-based Mott Schottky analysis[32]:

(2)
$\frac{1}{C^{2}_{SC}}=\frac{2}{\epsilon_{0}\epsilon_{r}N_{a,d}e}(V-V_{FB}-\frac{k_{B}T}{e})$

where $C_{SC}$ is the capacitance of the space charge region, $\epsilon_{0}$
and $\epsilon_{r}$ are the vacuum and relative (9.6 for 4H-SiC)
permittivities, $N_{a,d}$ is the acceptor or donor concentration,
respectively. $e$ is the electron charge, $k_{B}$ is the Boltzmann constant
and $T$ is the temperature in degrees Kelvin. By linear fitting to the slope
of the measured capacitance in cathodic and anodic sweeps, it is possible to
determine the acceptor and donor concentrations, respectively. The net doping
is found to be $1.15(10)^{14}cm^{-3}$, as shown in Figure 1c, hereafter
referred to as the "p-type" layer.

For PEC-enabled release of etched structures, indium tin oxide (ITO) is
adopted as an optically transparent contact to the p-type layer through which
the underlying SiC can be illuminated while still providing uniform electrical
connectivity across the cell. For diode characterization, cyclic voltammetry,
and PEC etching, ITO is sputtered at a rate of $3nm/min$ to form a $100nm$
layer, ramping the substrate to 600${}^{\circ}C$ over the course of
deposition. Substrate heating was found to be necessary to form a suitably
conductive contact to the p-type epilayer. Illumination for all measurements
and etching is provided by a $500W$ Xe-Hg Arc lamp with its visible spectrum
filtered by a $400nm$ short-pass filter to reduce excess heating of the
sample, producing a peak intensity of $1W\dot{c}m^{-2}$. Before fabricating
membrane devices we first characterize the photo-response of the diode
structure, to determine the degree to which the epilayer is photo-biased as
the lamp intensity is varied. In the dark, the current-voltage (IV) curve
shows a turn-on voltage of $0.5V$. Under increasing illumination, the diode
shows a typical photodiode response, with a peak photo-bias of $490mV$
measured at a lamp intensity of $1W\dot{c}m^{-2}$ (Figure 1e).

To characterize the dopant-selective undercut process, mesas are first defined
by patterning the ITO layer using positive tone (PMMA495 C6) electron-beam
lithography. The pattern is transferred to the ITO by inductively coupled
plasma reactive ion etching in H2/CF4, and then subsequently transferred into
the SiC to a depth of $2\mu m$ using dry etching in SF6/O2. ITO is found to be
a highly selective hard mask, as oxide reduction is continually countered by
plasma oxidation via the oxygen present in the SiC etching gas mixture,
yielding a selectivity of 50:1. PEC etching is performed in $40-500mM$
potassium hydroxide (KOH) electrolyte which is continuously circulated from a
10${}^{\circ}C$ reservoir to ensure a constant electrolyte concentration and
to extract waste heat, with no supporting electrolyte added. The counter
electrode is a wound Pt wire and the reference electrode is an Ag/AgCl
electrode in 3M KCl. The membranes are undercut using a PEC cell, with two
separate electrical circuits present. The first is the potentiostat, used to
bias either the n-type or p-type SiC as a working electrode against the Pt
counter electrode. The second is the junction bias, controlled by attaching a
voltage source to the p-type and n-type contacts. To assess the impact of
photo-biasing on the etching kinetics of the junction, cyclic voltammetry (CV)
is performed under various junction bias conditions with a sweep rate of
$20mVs^{-1}$ (Figure 2a). Under full-illumination ($1W\dot{c}m^{-2}$) and no
external bias applied to the junction, the p-type and n-type CV curves
recorded are similar except that the p-type curve is shifted anodic relative
to the n-type curve (Figure 2a.i). This is contrary to rigorous studies of
thick layers of 4H-SiC which found that n-type curves should be anodic
relative to p-type curves.[23] Furthermore, no distinct oxidation or reduction
peaks are observed corresponding to p-type or n-type etching, confirming that
the strong photo-biasing of the cell has eliminated etch selectivity. However,
by noting that the photo-biasing occurs because of the presence of the
junction formed by the epilayer and substrate, it can be concluded that the
photo-biasing can be completely suppressed by reverse biasing the junction to
correct for the photo-generated voltage. By closing the junction circuit
between the ITO and Ni junction contacts, the CV curves converge (Figure
2a.ii). While closing the junction circuit may restore selectivity in highly
doped material, we must further increase the junction bias in our case of an
undoped epilayer. The junction bias depletes the epilayer of carriers and
creates selectivity between regions of different doping where it is not
possible to achieve rapid selective etching without diode biasing. As greater
reverse bias is progressively applied, the quasi-Fermi level in the valence
band of the p-type layer rises relative to that in the n-type layer and the
dissolution potential into the electrolyte, and the p-type CV curve shifts
cathodic (Figure 2a.iii-v). At the same time, the characteristic oxidation and
reduction peaks associated with controlled etching reappear, now at cell
biases significantly different enough to support selective etching.

Figure 3. (a) PDMS membrane release process for AFM characterization (b) AFM
height maps for different electrolyte concentrations with corresponding root
mean squared roughness values ($R_{q}$) after a 2 hour release process with
$0.2V$ cell bias and $1W\dot{c}m^{2}$ illumination.  Figure 4. Spin
spectroscopy of weak ensembles of $V_{Si}^{-}$ in (red) undoped epilayer and
(blue) p-i-p doped structures. (a) Pulse sequences used for (i.) ODMR, (ii.)
FID and (iii.) spin dephasing measurements. (b-c) ODMR spectrums with fitted
triple Lorentzian spectrum based on natural ${}^{29}Si$ concentration in the
nearest neighbouring site of the defects. Inset: Fitted power dependent line-
width of the central peak. (d-e) Free induction decay ($T_{2}^{*}$) fitted to
a decaying sinusoid with an envelope of $exp((-t/T_{2}^{*})^{2})$ measured via
a Ramsey sequence $2MHz$ off-resonance and spin dephasing time ($T_{2}$)
fitted to an exponential decay of $exp((-t/T_{2})^{2})$ by Hahn echo sequence
on-resonance. All measurements performed at $6K$.

To assess doping selectivity in realistic structures, a series of $4\mu m$
wide mesas are undercut in $40mM$ and $200mM$ KOH at different junction
reverse biases (Figure 2), with a cell bias of $0.2V$, selected to ensure that
the n-type electrode is not passivated.[29] Lateral etching rates and membrane
morphology are determined using focused-ion-beam (FIB) cross sectioning. In
both electrolyte concentrations, as the reverse bias increases, p-type etching
is suppressed before becoming fully passivated, past a reverse bias of
$-1.3V$(Figure 2b). Cross sectional images corroborate this finding, showing
partial undercutting of the p-type layer at moderate reverse biases
($-0.75V$), and full passivation at high reverse biases($-1.5V$) (Figure 2c).
This result demonstrates that at sufficient reverse bias, diffusion of photo-
generated carriers can be predominantly controlled not by differences in
doping but by external biasing.

Smoothly-etched surfaces are highly desirable for photonic and mechanical
structures, as scattering loss and surface damping are therefore reduced. We
investigate the impact that electrolyte concentration has on the surface
roughness of the underside of our membranes. The membranes shown in Figure 2c
are fully suspended and then lifted out using polydimethylsiloxane (PDMS)
stamping (Figure 3a). The surface roughness of the membrane underside is then
measured by tapping mode atomic force microscopy. We find that the
concentration of electrolyte strongly controls the surface quality of the
underside of the membranes, with lower concentration ($40mM$ KOH) yielding a
RMS surface roughness value of $1.43nm$ (Figure 3b-d), whereas higher
concentrations of KOH yield rougher surfaces ($5.51nm$ for $500mM$ KOH).
Previous work suggests that roughness increases in a regime where oxide
formation exceeds the rate of oxide removal, indicating this balance may be
more difficult to establish at high electrolyte concentrations.[23] Similar
work also showed that above $500mM$ concentrations, only porous SiC is formed
by PEC.[31]

To assess the potential for the material to host quantum spin registers, the
reverse bias undercut process is paired with spin resonance and spin lifetime
measurements. A suspended, 500nm thick undoped membrane formed using the
process described, etched in $40mM$ KOH, is compared to another,
$500nm$ thick suspended membrane comprised of p-i-p-n ($150nm$ p-type, $200nm$
undoped, $150nm$ p-type, n-type substrate) material used in previous work,
that is undercut under identical conditions.[28] Prior work found that slight
shifts in the ZPL, ODMR and spin coherence times in photoelectrochemical
etched structures was a result of high doping, porosity and strain induced by
the PEC process.[22] We select the silicon monovacancy for study as it can be
flexibly introduced and has been shown to have high quality optical and spin
properties in nanostructures.[33, 34] Each sample is identically implanted
with He ions of energy $45keV$ at a dose of $10^{11}\,ions\cdot cm^{-2}$
followed by quenching from 650${}^{\circ}C$ after annealing in air for 30
minutes, to form a weak ensemble of silicon monovacancies ($V_{Si}^{-}$). The
samples are then loaded into a $4K$ flow cryostat and fitted with a microwave
wire antenna to excite ground state spin resonances. The charge state of the
defects is initialized by an off resonant helium neon laser with a power of
$3mW$, probed by $28dbm$ of input RF power, and photoluminescence is measured
on the phonon sideband (>$925nm$) of the defects using near resonant
excitation with a $916nm$ external cavity diode laser with a power of $100\mu
W$.(Figure 4a)

To compare the spin performance of emitters in both the heavily doped and
undoped material, power dependent optically detected magnetic resonance
(ODMR), free induction decay times ($T_{2}^{*}$), and spin dephasing times
($T_{2}$) are measured for each ensemble (Figure 4a). The ground state
($S=3/2$) spin levels, which are typically separated by a zero-field splitting
of $70MHz$ are separated further by a moderate magnetic field of $60G$ to a
final splitting of $240MHz$ to reduce the effect of heteronuclear spin flip
processes that limit the $T_{2}$ and $T_{2}^{*}$ times of ensemble of
emitters. Assessing the $m_{s}=-\frac{1}{2}$ to $m_{s}=-\frac{3}{2}$
transition, ODMR studies in Figure 4b,c show similar performance, with
emitters in both material types demonstrating narrow inhomogeneous broadening
that saturate to a minimum at similar drive powers. The $T_{2}^{*}$ time of
$1.64\pm 0.02\mu s$ measured in the undoped material (Figure 4d) is longer
than that in the doped material (Figure 4e), approaching the theoretically
predicted low field limit of ($1.9\mu s$) for monovacancy spins in a naturally
abundant ${}^{29}Si$,${}^{13}C$ nuclear bath.[35] This demonstrates that the
use of undoped material reduces the presence of influences that induce fast
dephasing of the spins, which previously limited PEC-etched device performance
[22]. In contrast, the $T_{2}$ time is similar between the two samples,
showing that in both cases on longer timescales the defects are limited by
similar slow, dephasing processes. This implies that doping or material
defects, and not material porosity or fabrication damage, primarily drives the
degradation of $V_{Si}^{-}$ spin properties in PEC processed material.

##  3\. Conclusion

We have demonstrated a means of dopant-selective PEC etching that
significantly relaxes constraints placed on doping density to achieve etch
selectivity–thereby enabling the suspension of low-surface roughness, large-
area membranes with an improved charge-noise environment to host quantum
emitters. This approach reduces the epitaxial steps necessary for high purity,
PEC-suspended photonics and mechanics in SiC to one step with the addition of
a commonly used hard-mask (ITO) and a circuit to bias the junction formed
between the epilayer and substrate. We investigate the photo-electrochemical
attributes of SiC diodes under external biasing and show that smooth membranes
of undoped silicon carbide can be suspended at rates exceeding $4um/hr$. We
additionally note that the demonstrated 500nm device layer is defined by
epitaxial growth and can therefore be flexibly changed to suit application.
Finally, we perform comparative spin spectroscopy on suspended doped and
undoped materials, demonstrating that undoped material beneficially reduces
the effect of fast spin noise on quantum defects and is compatible with PEC
processing. The development of a means of reliably suspending large undoped
membranes of SiC opens up new opportunities leveraging 4H-SiC’s high yield
strength for nanomechanical oscillators and applications that integrate
quantum emitters into electronic and photonic devices.[36] Beyond SiC, this
technique relaxes the material constraints for dopant selective etching,
suggesting a means of selectively etching other inert material where only a
single doping type can be reliably created, such as diamond.

##  4\. Methods

PEC etching is performed in $40-500mM$ potassium hydroxide (KOH) electrolyte
which is continuously circulated from a 10${}^{\circ}C$ reservoir to ensure a
constant electrolyte concentration and to extract waste heat, with no
supporting electrolyte added. The counter electrode is a wound Pt wire and the
reference electrode is an Ag/AgCl electrode in 3M KCl. For spin measurements,
each sample is identically implanted with He ions of energy $37keV$ at a dose
of $10^{11}\,ions\cdot cm^{-2}$ followed by quenching from 650${}^{\circ}C$
after annealing in air for 30 minutes, to form a weak ensemble of silicon
monovacancies ($V_{Si}^{-}$). The samples are then loaded into a $4K$ flow
cryostat and fitted with a microwave wire antenna to excite ground state spin
resonances. The charge state of the defects is initialized by an off resonant
helium neon laser with a power of $3mW$, and photoluminescence is measured on
the phonon sideband (>$925nm$) of the defects using near resonant excitation
with a $916nm$ external cavity diode laser with a power of $100\mu W$.

##  5\. Acknowledgements

Research reported in this publication was supported as part of the AWS Center
for Quantum Networking’s research alliance with the Harvard Quantum Initiative
(or HQI). Portions of this work were performed at the Harvard University
Center for Nanoscale Systems (CNS); a member of the National Nanotechnology
Coordinated Infrastructure Network (NNCI), which is supported by the National
Science Foundation under NSF award no. ECCS-2025158. A.M.D. acknowledges
funding from the Science and Technology Center for Integrated Quantum
Materials, NSF Grant No. DMR-1231319. J.R.D acknowledges funding from NSF
RAISE-TAQS Award 1839164.

## References

  * [1] Hongwei Qu  “CMOS MEMS Fabrication Technologies and Devices” Number: 1 Publisher: Multidisciplinary Digital Publishing Institute  In _Micromachines_ 7.1, 2016, pp. 14  DOI: 10.3390/mi7010014
  * [2] Yi Zheng et al.  “High-quality factor, high-confinement microring resonators in 4H-silicon carbide-on-insulator” Publisher: Optica Publishing Group  In _Optics Express_ 27.9, 2019, pp. 13053–13060  DOI: 10.1364/OE.27.013053
  * [3] Di Zhu et al.  “Integrated photonics on thin-film lithium niobate” Publisher: Optica Publishing Group  In _Advances in Optics and Photonics_ 13.2, 2021, pp. 242–352  DOI: 10.1364/AOP.411024
  * [4] Xinghan Guo et al.  “Tunable and Transferable Diamond Membranes for Integrated Quantum Technologies” Publisher: American Chemical Society  In _Nano Letters_ 21.24, 2021, pp. 10392–10399  DOI: 10.1021/acs.nanolett.1c03703
  * [5] Y. Lindén, L. Tenerz, J. Tirén and B. Hök  “Fabrication of three-dimensional silicon structures by means of doping-selective etching (DSE)”  In _Sensors and Actuators_ 16.1, 1989, pp. 67–82  DOI: 10.1016/0250-6874(89)80006-X
  * [6] C.J.M. Eijkel, J. Branebjerg, M. Elwenspoek and F.C.M. Van de Pol  “A new technology for micromachining of silicon: dopant selective HF anodic etching for the realization of low-doped monocrystalline silicon structures” Conference Name: IEEE Electron Device Letters  In _IEEE Electron Device Letters_ 11.12, 1990, pp. 588–589  DOI: 10.1109/55.63048
  * [7] J.. Shor et al.  “Dopant-selective etch stops in 6H and 3C SiC”  In _Journal of Applied Physics_ 81.3, 1997, pp. 1546–1551  DOI: 10.1063/1.363890
  * [8] C. Youtsey, G. Bulman and I. Adesida  “Dopant-selective photoenhanced wet etching of GaN”  In _Journal of Electronic Materials_ 27.4, 1998, pp. 282–287  DOI: 10.1007/s11664-998-0400-0
  * [9] Reena Khare and Evelyn L. Hu  “Dopant Selective Photoelectrochemical Etching of GaAs Homostructures” Publisher: IOP Publishing  In _Journal of The Electrochemical Society_ 138.5, 1991, pp. 1516  DOI: 10.1149/1.2085818
  * [10] R. Soref and B. Bennett  “Electrooptical effects in silicon” Conference Name: IEEE Journal of Quantum Electronics  In _IEEE Journal of Quantum Electronics_ 23.1, 1987, pp. 123–129  DOI: 10.1109/JQE.1987.1073206
  * [11] R.. Mihailovich and J.. Parpia  “Low temperature mechanical properties of boron-doped silicon” Publisher: American Physical Society  In _Physical Review Letters_ 68.20, 1992, pp. 3052–3055  DOI: 10.1103/PhysRevLett.68.3052
  * [12] Anderson Janotti and Chris G. de Walle  “Fundamentals of zinc oxide as a semiconductor”  In _Reports on Progress in Physics_ 72.12, 2009, pp. 126501  DOI: 10.1088/0034-4885/72/12/126501
  * [13] Hongchao Yang, Yandong Ma and Ying Dai  “Progress of structural and electronic properties of diamond: a mini review” Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/26941112.2021.1956287  In _Functional Diamond_ 1.1, 2022, pp. 150–159  DOI: 10.1080/26941112.2021.1956287
  * [14] Daniil M. Lukin, Melissa A. Guidry and Jelena Vučković  “Integrated Quantum Photonics with Silicon Carbide: Challenges and Prospects” Publisher: American Physical Society  In _PRX Quantum_ 1.2, 2020, pp. 020102  DOI: 10.1103/PRXQuantum.1.020102
  * [15] Yaoyao Long, Zhenming Liu and Farrokh Ayazi  “4H-Silicon Carbide as an Acoustic Material for MEMS” Conference Name: IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control  In _IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control_ 70.10, 2023, pp. 1189–1200  DOI: 10.1109/TUFFC.2023.3282920
  * [16] Gang Zhang, Yuan Cheng, Jyh-Pin Chou and Adam Gali  “Material platforms for defect qubits and single-photon emitters”  In _Applied Physics Reviews_ 7.3, 2020, pp. 031308  DOI: 10.1063/5.0006075
  * [17] Jonathan R. Dietz et al.  “Spin-acoustic control of silicon vacancies in 4H silicon carbide” Number: 10 Publisher: Nature Publishing Group  In _Nature Electronics_ 6.10, 2023, pp. 739–745  DOI: 10.1038/s41928-023-01029-4
  * [18] Roland Nagy et al.  “High-fidelity spin and optical control of single silicon-vacancy centres in silicon carbide” Number: 1 Publisher: Nature Publishing Group  In _Nature Communications_ 10.1, 2019, pp. 1954  DOI: 10.1038/s41467-019-09873-9
  * [19] Christopher P. Anderson et al.  “Electrical and optical control of single spins integrated in scalable semiconductor devices” Publisher: American Association for the Advancement of Science  In _Science_ 366.6470, 2019, pp. 1225–1230  DOI: 10.1126/science.aax9406
  * [20] Andrew P. Magyar et al.  “High quality SiC microdisk resonators fabricated from monolithic epilayer wafers”  In _Applied Physics Letters_ 104.5, 2014, pp. 051109  DOI: 10.1063/1.4863932
  * [21] David O. Bracher, Xingyu Zhang and Evelyn L. Hu  “Selective Purcell enhancement of two closely linked zero-phonon transitions of a silicon carbide color center” Publisher: Proceedings of the National Academy of Sciences  In _Proceedings of the National Academy of Sciences_ 114.16, 2017, pp. 4060–4065  DOI: 10.1073/pnas.1704219114
  * [22] Alexander L. Crook et al.  “Purcell Enhancement of a Single Silicon Carbide Color Center with Coherent Spin Control” Publisher: American Chemical Society  In _Nano Letters_ 20.5, 2020, pp. 3427–3434  DOI: 10.1021/acs.nanolett.0c00339
  * [23] Shojan P. Pavunny et al.  “On the doping concentration dependence and dopant selectivity of photogenerated carrier assisted etching of 4H–SiC epilayers”  In _Electrochimica Acta_ 323, 2019, pp. 134778  DOI: 10.1016/j.electacta.2019.134778
  * [24] David O. Bracher and Evelyn L. Hu  “Fabrication of High-Q Nanobeam Photonic Crystals in Epitaxially Grown 4H-SiC” Publisher: American Chemical Society  In _Nano Letters_ 15.9, 2015, pp. 6202–6207  DOI: 10.1021/acs.nanolett.5b02542
  * [25] B. Jiang, N.. Opondo and S.. Bhave  “Semi-insulating 4H-SiC lateral bulk acoustic wave resonators” Number: 11 Publisher: American Institute of Physics  In _Applied Physics Letters_ 118.11, 2021, pp. 114002  DOI: 10.1063/5.0045232
  * [26] Daniil M. Lukin et al.  “4H-silicon-carbide-on-insulator for integrated quantum and nonlinear photonics” Number: 5 Publisher: Nature Publishing Group  In _Nature Photonics_ 14.5, 2020, pp. 330–334  DOI: 10.1038/s41566-019-0556-6
  * [27] Jonah Heiler et al.  “Spectral stability of V2 centres in sub-micron 4H-SiC membranes” Publisher: Nature Publishing Group  In _npj Quantum Materials_ 9.1, 2024, pp. 1–8  DOI: 10.1038/s41535-024-00644-4
  * [28] Jonathan R. Dietz and Evelyn L. Hu  “Optical and strain stabilization of point defects in silicon carbide”  In _Applied Physics Letters_ 120.18, 2022, pp. 184001  DOI: 10.1063/5.0087805
  * [29] D.. Dorp, J.. Weyher and J.. Kelly  “Anodic etching of SiC in alkaline solutions”  In _Journal of Micromechanics and Microengineering_ 17.4, 2007, pp. S50  DOI: 10.1088/0960-1317/17/4/S04
  * [30] Kohei Adachi et al.  “Single-crystalline 4H-SiC micro cantilevers with a high quality factor”  In _Sensors and Actuators A: Physical_ 197, 2013, pp. 122–125  DOI: 10.1016/j.sna.2013.04.014
  * [31] Masashi Kato, Masaya Ichimura, Eisuke Arai and Perumalsamy Ramasamy  “Electrochemical Etching of 6H-SiC Using Aqueous KOH Solutions with Low Surface Roughness” Publisher: IOP Publishing  In _Japanese Journal of Applied Physics_ 42.7R, 2003, pp. 4233  DOI: 10.1143/JJAP.42.4233
  * [32] Kevin Sivula  “Mott–Schottky Analysis of Photoelectrodes: Sanity Checks Are Needed” Publisher: American Chemical Society  In _ACS Energy Letters_ 6.7, 2021, pp. 2549–2551  DOI: 10.1021/acsenergylett.1c01245
  * [33] Aaron M. Day et al.  “Laser writing of spin defects in nanophotonic cavities” Number: 6 Publisher: Nature Publishing Group  In _Nature Materials_ 22.6, 2023, pp. 696–702  DOI: 10.1038/s41563-023-01544-x
  * [34] Charles Babin et al.  “Fabrication and nanophotonic waveguide integration of silicon carbide colour centres with preserved spin-optical coherence” Publisher: Nature Publishing Group  In _Nature Materials_ 21.1, 2022, pp. 67–73  DOI: 10.1038/s41563-021-01148-3
  * [35] Li-Ping Yang et al.  “Electron spin decoherence in silicon carbide nuclear spin bath” Publisher: American Physical Society  In _Physical Review B_ 90.24, 2014, pp. 241203  DOI: 10.1103/PhysRevB.90.241203
  * [36] Leo Sementilli et al.  “Ultralow Dissipation Nanomechanical Devices from Monocrystalline Silicon Carbide” arXiv:2404.13893 [cond-mat, physics:physics]  arXiv, 2024  DOI: 10.48550/arXiv.2404.13893


# Designing an Adaptive Application-Level Checkpoint Management System for
Malleable MPI Applications

Jophin John1, Michael Gerndt2
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>
Chair of Computer Architecture and Parallel Systems, Technische Universität
München
Garching (near Munich), Germany

###### Abstract

Dynamic resource management opens up numerous opportunities in High
Performance Computing. It improves the system-level services as well as
application performance. Checkpointing can also be deemed as a system-level
service and can reap the benefits offered by dynamism. A checkpointing system
can have better resource availability by integrating with a malleable resource
management system. In addition to fault tolerance, the checkpointing system
can cater to the data redistribution demand of malleable applications during
resource change. Therefore, we propose iCheck, an adaptive application-level
checkpoint management system that can efficiently utilize the system and
application level dynamism to provide better checkpointing and data
redistribution services to applications.

###### Index Terms:

Fault Tolerance, Adaptive Checkpointing, RDMA, Malleability, MPI

##  I Introduction

Static resource management, still predominant in High-Performance Computing,
takes away many optimization opportunities otherwise catered by dynamic
resource management. This static behavior is visible from system services to
applications [1, 2]. The potential offered by dynamic applications using
dynamic resources spans beyond better system utilization [3, 4, 5, 6], and
checkpointing is no exception [7, 8, 9]. Trends in dynamism research bring
malleability[10, 11, 12, 13, 14, 15] into MPI[16] and resource management
infrastructure.

To yield the benefits of such a malleable infrastructure, the checkpointing
system must be able to react to the dynamic needs of resource managers and
applications by adapting itself to the changing system requirements. This can
result in faster checkpoint transfers and efficient utilization of
checkpointable resources (available memory, parallel file system bandwidth).
Furthermore, a checkpointing system can conduct data redistribution during
resource change, thereby addressing one of the key challenges in writing
malleable applications.

Hence, we posit checkpointing systems as a resource and data management system
that can simultaneously handle the nuances in checkpointing requirements of
multiple applications and interact with resource managers to provide seamless
checkpointing capabilities. Towards this, we propose iCheck, a multi-level
adaptive checkpoint management system that can reconfigure its checkpointing
resources on the fly and offers application-level checkpointing and data
redistribution services to malleable MPI applications. This approach of
application-level checkpointing as an adaptable resource management and data
distribution service brings novel contributions to the checkpointing domain.

##  II iCheck design

iCheck has a core system and a library. iCheck core runs on a dynamically
configurable number of dedicated nodes (iCheck nodes) of the HPC system and
communicates with connected applications using the iCheck library (See Figure
1). Application checkpoints are transferred to the memory of iCheck nodes
using Remote Direct Memory Access (RDMA) and later written into the Parallel
File System (PFS). The core system consists of an Agent, Manager, and
Controller. The agent performs the functionality of checkpoint read/write
(using libfabric[17]) and data redistribution (for malleable implementations).
Multiple agents can be assigned to a single application, and iCheck can
dynamically change the agent count to obtain an optimum checkpoint transfer
rate. The manager manages the node-level activities of the software, such as
launching the agents and monitoring and predicting the node usage parameters
(e.g., memory usage, bandwidth usage). The controller has a global view and
performs the agent and node selection for connected applications based on the
iCheck agent scheduling policies. These policies consider various system
metrics (available memory, checkpoint frequency and size, and bandwidth usage)
and can impact the overall checkpointing performance. The controller may also
request the resource manager for additional resources based on resource
availability. In addition, the controller will also orchestrate the writing of
the checkpoint data into PFS by minimizing the effect on running applications.

Figure 1: Components in iCheck system

The iCheck core and iCheck library components in an application continually
interact during the application life cycle. A generic workflow of these
interactions is as follows:

During the start of the application:

  1. 1.

An application registers itself with the controller.

  2. 2.

The controller decides the number of agents and the iCheck nodes in which they
will be launched.

  3. 3.

The controller contacts the corresponding managers with the agent information.

  4. 4.

The managers launch the agents in iCheck nodes and notify the controller.

  5. 5.

The agents wait for the application processes to connect.

  6. 6.

The controller provides the agent information to the application.

  7. 7.

The application then connects with the agent.

  8. 8.

The application and the agents register memory for RDMA.

  9. 9.

The application and the agents perform checkpoint transfer operations.

  10. 10.

Application finishes either with or without error.

During the restart:

  1. 1.

The application contacts the controller for checkpoint information.

  2. 2.

The controller passes the agent information.

  3. 3.

The application can either start by using the checkpoint from the agents or
start new.

During the data redistribution:

  1. 1.

The application contacts the agent for data transfer.

  2. 2.

The agents redistribute the data based on the mapping provided by the
application.

iCheck, with its agent-based checkpoint retrieval model, easily integrates the
asynchronous checkpoint retrieval capability in its library. Since the agents
use RDMA, the application does not need to block for data transfer rather it
can continue the execution immediately after notifying the agents about the
checkpoints.

##  III iCheck and Malleability

iCheck is malleable both at the system level and application level. At the
system level, iCheck can horizontally scale its resources (available memory,
number of agents) by attaching/removing iCheck nodes with the help of a
resource manager. At the application level, iCheck can dynamically reconfigure
the number of agents associated with connected applications.

###  III-A iCheck and Malleable Infrastructure

We have used the malleable infrastructure from the InvasiC project[11, 18, 5]
for our investigation into designing a malleable checkpointing system. It
consists of an MPI implementation (an extension of MPICH with four additional
methods) that can change the number of processes during the application
execution and a malleable resource manager (extension of the Slurm [19]) that
can orchestrate the resource changes in MPI.

To support malleable MPI applications, the application-level checkpointing
systems should be able to handle the changes in resources, i.e., whenever new
processes are added, they should be able to reinitialize the system (most of
the application-level checkpointing systems [20, 21, 22, 23, 24] needs to be
modified). iCheck provide this capability out-of-the-box. Whenever a new set
of processes are added to the application, iCheck can add new agents, thereby
maintaining the checkpointing performance or reinitializing with the same
agents. Additionally, using special APIs (icheck_add_adapt(),
icheck_redistribute() as in Listing 1), users can provide their data
distribution mapping that can be used by iCheck when a resource change
happens. iCheck also provides a special API call icheck_probe_agents() (line
29 in Listing 1) that can be used to query for agent change to improve the
checkpointing time.

To facilitate system-level malleability in iCheck, we have created an iCheck-
aware job scheduling plugin in the resource manager (RM). The following
interactions with iCheck are supported by the new plugin. 1 RM can give
resources (iCheck nodes) to the iCheck based on the request and availability.
For example, when iCheck runs out of memory in a node, the controller can
request more memory and get additional nodes from RM. 2 RM can retake nodes
from iCheck. For example, to support resource requirements of a priority job
or to satisfy power requirements. 3 RM can ask the controller to migrate
resources to a different iCheck node. 4 RM can pass application-specific
information to the controller. For example, it can inform the controller about
an impending resource change of an application so that agents can prepare for
data redistribution ahead of time.

###  III-B iCheck library and a simple malleable MPI application

The pseudocode of a naive iCheck enabled malleable MPI application is shown in
Listing 1. The four malleable MPI routines behave as follows. MPI_Init_adapt()
(line 3) will register the application with the resource manager and returns
the type of the MPI process. A process can be initial (created as part of an
initial application launch) or joining (created as part of an application
expansion). Initial processes continue the application execution (skipping
lines 10-15) and regularly call MPI_Probe_adapt() (line 17) to check for any
resource change. On the other hand, joining processes immediately calls the
MPI_Comm_adapt_begin() (line 11) collective function and waits for initial
processes to join. Meanwhile, the MPI_Probe_adapt() (in line 17) call informs
initial processes about the resource change triggered by the malleable
resource manager. Initial processes, then calls the collective
MPI_Comm_adapt_begin() function (line 19). Once both the initial and joining
processes call MPI_Comm_adapt_begin() routine (line 11 and line 19), they can
perform data redistribution and share application-specific data and control
information among them. After that, they call MPI_Comm_adapt_commit() (lines
14 and 22) collective function to finish the resource adaptation and together
resume the application execution.  ⬇ 1#include<icheck.h> 2int main() { 3
MPI_Init_adapt(…, type) 4 float data[SIZE]; 5 icheck_init(…, type); 6
icheck_add_adapt("data",data,…,BLOCK); 7 if(checkpoint_available && no_adapt){
8 icheck_restart(); 9 } 10 if (type == joining) { 11 MPI_Comm_adapt_begin(…);
12 icheck_redistribute("data",data, 13 new_size, BLOCK) 14
MPI_Comm_adapt_commit(); 15 } 16 while (true){ 17
MPI_Probe_adapt(resource_change,…); 18 if (resource_change) { 19
MPI_Comm_adapt_begin(…); 20 icheck_redistribute("data",data, 21 new_size,
BLOCK) 22 MPI_Comm_adapt_commit(); 23 } 24 /*Read/Write data[]*/ 25
if(iteration%100) 26 icheck_commit(); 27 /*Check for agent change*/ 28
if(iteration%1000) 29 icheck_probe_agents(); 30 } 31 icheck_finalize(…); 32
MPI_Finalize(); 33} Listing 1: Psuedocode of a malleable application with
iCheck

icheck_init() (line 5) registers the application with the controller.
icheck_add_adapt() (line 6) is used to add the data to be checkpointed along
with the mapping needed for data redistribution during a resource adaptation.
If present, the initial processes can restart and restore the previously
stored checkpoint icheck_restart() (lines 7-9). Otherwise, it continues the
application execution. icheck_commit() (in line 26) will trigger agents to
perform asynchronous checkpoint transfer to iCheck nodes. During an
application expansion, initial and joining processes will call the
icheck_redistribute() (line 12 and line 20) to get the proper data to resume
the application execution. Currently, iCheck only supports naive data
redistribution schemes like block (line 12 and line 20) or cyclic during a
resource change.

##  IV related work

There are a plethora of checkpointing works based on different categories like
application-level or system-level, process recovery or data recovery,
multilevel or single-level, coordinated or uncoordinated, and blocking or non-
blocking [25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 24, 20, 21,
22, 23]. Here we focus on works closest to iCheck.

iCheck is an adaptive asynchronous multilevel application-level in-memory
checkpointing system using RDMA that provides data distribution service and
fault tolerance. To the best of our knowledge, no single system exists with
all the above characteristics for application-level checkpointing. Existing
application-level checkpointing libraries (for example, [22, 21, 23]) cannot
reconfigure their resources across multiple applications to optimize their
checkpointing activity. In contrast, iCheck can centrally manage checkpoints
and improve the overall checkpointing performance by dynamically scaling its
resources.

Regarding multilevel RDMA capability in checkpointing, the closest work to
iCheck is by Sato et al.[24]. iCheck differs in the following aspects. iCheck
can simultaneously cater to the needs of multiple applications and reconfigure
its resources for RDMA capability on the fly. Further, iCheck does not use
extra processes inside the compute nodes of applications to transfer the
checkpoint.

VeloC [20] is a related work to iCheck with regards to the adaptivity in
checkpointing. It is an adaptive asynchronous checkpointing system where
adaptivity refers to efficiently selecting the underlying checkpoint storage
based on the available heterogeneous solutions. In iCheck, adaptivity refers
to the flexibility in scaling the resources of our system.

##  V conclusion and future work

Our current work on iCheck focuses on analyzing the malleable characteristics
of complex applications and adapting iCheck to support the intricacies of data
distribution in such applications. The current naive data redistribution
schemes in iCheck are insufficient to utilize malleability in complex
applications (for example, data redistribution inside molecular dynamics
applications during a resource change).

Furthermore, we intend to extend the current iCheck-aware job scheduling
plugin inside RM. The current experimental plugin prioritizes iCheck and will
allocate nodes to iCheck based on the controller’s request and node
availability. We will extend it by incorporating a malleable resource
management policy that removes nodes from running applications based on their
performance [18], ensuring fairness to running jobs and jobs waiting in the
queue.

Our prototype implementation is currently deployed on a virtual cluster and
analyzed with synthetic malleable applications. We plan to evaluate iCheck on
SuperMUC-NG[39] with real scientific applications developed using malleable
MPI.

##  VI Acknowledgment

The research leading to these results was funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation)-Projektnummer
146371743-TRR 89: Invasive Computing.

## References

  * [1] “Jsc.” [Online]. Available: https://hbp-hpc-platform.fz-juelich.de/?page_id=1836
  * [2] D. H. Ahn, J. Garlick, M. Grondona, D. Lipari, B. Springmeyer, and M. Schulz, “Flux: a next-generation resource management framework for large hpc centers,” in _2014 43rd International Conference on Parallel Processing Workshops_. IEEE, 2014, pp. 9–17.
  * [3] T. Desell, K. Maghraoui, and C. Varela, “Malleable applications for scalable high performance computing,” _Cluster Computing_ , vol. 10, pp. 323–337, 09 2007.
  * [4] M. Chadha, “Adaptive resource-aware batch scheduling for hpc systems.”
  * [5] J. John, S. Narváez, and M. Gerndt, “Invasive computing for power corridor management,” _Parallel Computing: Technology Trends_ , vol. 36, p. 386, 2020\.
  * [6] M. Abhishek, “Dynamic allocation of high performance computing resources,” _International Journal of Advanced Trends in Computer Science and Engineering_ , vol. 9, pp. 3538–3543, 06 2020.
  * [7] C. George and S. S. Vadhiyar, “Adft: An adaptive framework for fault tolerance on large scale systems using application malleability,” _Procedia Computer Science_ , vol. 9, pp. 166–175, 2012, proceedings of the International Conference on Computational Science, ICCS 2012. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1877050912001391
  * [8] K. El Maghraoui, B. K. Szymanski, and C. Varela, “An architecture for reconfigurable iterative mpi applications in dynamic environments,” in _Parallel Processing and Applied Mathematics_ , R. Wyrzykowski, J. Dongarra, N. Meyer, and J. Waśniewski, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, pp. 258–271.
  * [9] J. Posner, “Load balancing, fault tolerance, and resource elasticity for asynchronous many-task systems,” Ph.D. dissertation, Kassel, Universität Kassel, Fachbereich Elektrotechnik / Informatiik, 12 2021.
  * [10] I. A. Comprés Ureña and M. Gerndt, “Towards elastic resource management,” in _Tools for High Performance Computing 2017_ , C. Niethammer, M. M. Resch, W. E. Nagel, H. Brunst, and H. Mix, Eds. Cham: Springer International Publishing, 2019, pp. 105–127.
  * [11] I. A. Comprés Ureña, “Resource-elasticity support for distributed memory hpc applications,” Dissertation, TU Munich, Munich, 2017. [Online]. Available: http://mediatum.ub.tum.de?id=1362721
  * [12] L. V. Kalé, S. Kumar, and J. DeSouza, “A malleable-job system for timeshared parallel machines,” in _2nd IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGRID’02)_. IEEE, 2002, pp. 230–230.
  * [13] D. Projects, “Programming environment for european exascale systems,” https://www.deep-projects.eu/.
  * [14] admire, “Adaptive multi-tier intelligent data manager for exascale,” https://research.zdv.uni-mainz.de/research/admire/.
  * [15] J. I. Aliaga, M. Castillo, S. Iserte, I. Martín-Álvarez, and R. Mayo, “A survey on malleability solutions for high-performance distributed computing,” _Applied Sciences_ , vol. 12, no. 10, 2022. [Online]. Available: https://www.mdpi.com/2076-3417/12/10/5231
  * [16] L. Clarke, I. Glendinning, and R. Hempel, “The mpi message passing interface standard,” in _Programming Environments for Massively Parallel Distributed Systems_ , K. M. Decker and R. M. Rehmann, Eds. Basel: Birkhäuser Basel, 1994, pp. 213–218.
  * [17] “Libfabric library.” [Online]. Available: https://github.com/ofiwg/libfabric
  * [18] M. Chadha, J. John, and M. Gerndt, “Extending slurm for dynamic resource-aware adaptive batch scheduling,” in _2020 IEEE 27th International Conference on High Performance Computing, Data, and Analytics (HiPC)_ , 2020, pp. 223–232.
  * [19] A. B. Yoo, M. A. Jette, and M. Grondona, “Slurm: Simple linux utility for resource management,” in _Workshop on Job Scheduling Strategies for Parallel Processing_. Springer, 2003, pp. 44–60.
  * [20] B. Nicolae, A. Moody, E. Gonsiorowski, K. Mohror, and F. Cappello, “Veloc: Towards high performance adaptive asynchronous checkpointing at large scale,” 05 2019, pp. 911–920.
  * [21] F. Shahzad, J. Thies, M. Kreutzer, T. Zeiser, G. Hager, and G. Wellein, “Craft: A library for easier application-level checkpoint/restart and automatic fault tolerance,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. PP, 08 2017.
  * [22] M. Gamell, D. S. Katz, H. Kolla, J. Chen, S. Klasky, and M. Parashar, “Exploring automatic, online failure recovery for scientific applications at extreme scales,” _International Conference for High Performance Computing, Networking, Storage and Analysis, SC_ , vol. 2015, pp. 895–906, 01 2015\.
  * [23] A. Moody, G. Bronevetsky, K. Mohror, and B. R. d. Supinski, “Design, modeling, and evaluation of a scalable multi-level checkpointing system,” in _Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis_ , ser. SC ’10. USA: IEEE Computer Society, 2010, p. 1–11. [Online]. Available: https://doi.org/10.1109/SC.2010.18
  * [24] K. Sato, K. Mohror, A. Moody, T. Gamblin, B. Supinski, N. Maruyama, and S. Matsuoka, “Design and modeling of a non-blocking checkpointing system,” 11 2012, pp. 1–10.
  * [25] W. Bland, A. Bouteiller, T. Herault, G. Bosilca, and J. Dongarra, “Post-failure recovery of mpi communication capability: Design and rationale,” _The International Journal of High Performance Computing Applications_ , vol. 27, no. 3, pp. 244–254, 2013. [Online]. Available: https://doi.org/10.1177/1094342013488238
  * [26] J. Stearley, J. Laros, K. Ferreira, K. Pedretti, R. Oldfield, R. Riesen, and R. Brightwell, “rmpi : increasing fault resiliency in a message-passing environment.” 04 2011.
  * [27] G. Zheng, L. Shi, and L. V. Kale, “Ftc-charm++: An in-memory checkpoint-based fault tolerant runtime for charm++ and mpi,” in _Proceedings of the 2004 IEEE International Conference on Cluster Computing_ , ser. CLUSTER ’04. USA: IEEE Computer Society, 2004, p. 93–103.
  * [28] G. E. Fagg and J. Dongarra, “Ft-mpi: Fault tolerant mpi, supporting dynamic applications in a dynamic world,” in _Proceedings of the 7th European PVM/MPI Users’ Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface_. Berlin, Heidelberg: Springer-Verlag, 2000, p. 346–353.
  * [29] G. Rodríguez, M. Martín, P. González, J. Touriño, and R. Doallo, “Cppc: A compiler-assisted tool for portable checkpointing of message-passing applications,” _Concurrency and Computation: Practice and Experience_ , vol. 22, pp. 749–766, 04 2010.
  * [30] S. Chakraborty, I. Laguna, M. Emani, K. Mohror, D. K. Panda, M. Schulz, and H. Subramoni, “Ereinit: Scalable and efficient fault-tolerance for bulk-synchronous mpi applications,” _Concurrency and Computation: Practice and Experience_ , vol. 32, no. 3, p. e4863, 2020, e4863 cpe.4863. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/cpe.4863
  * [31] N. Sultana, A. Skjellum, I. Laguna, M. Farmer, K. Mohror, and M. Emani, “Mpi stages: Checkpointing mpi state for bulk synchronous applications,” 09 2018, pp. 1–11.
  * [32] B. Ramkumar and V. Strumpen, “Portable checkpointing for heterogeneous architectures,” 07 1997, pp. 58 – 67.
  * [33] R. Arora, P. Bangalore, and M. Mernik, “A technique for non-invasive application-level checkpointing,” _The Journal of Supercomputing_ , vol. 57, pp. 227–255, 09 2010.
  * [34] R. Arora and T. N. Ba, “Italc: Interactive tool for application-level checkpointing,” in _Proceedings of the Fourth International Workshop on HPC User Support Tools_ , ser. HUST’17. New York, NY, USA: Association for Computing Machinery, 2017. [Online]. Available: https://doi.org/10.1145/3152493.3152558
  * [35] J. Plank, M. Beck, G. Kingsley, and K. Li, “Libckpt: Transparent checkpointing under unix,” _TCON_ , 12 1995.
  * [36] T. Islam, K. Mohror, S. Bagchi, A. Moody, B. Supinski, and R. Eigenmann, “Mcrengine: A scalable checkpointing system using data-aware aggregation and compression,” _Scientific Programming_ , vol. 21, 01 2013.
  * [37] K. Sato, K. Mohror, A. Moody, T. Gamblin, B. Supinski, N. Maruyama, and S. Matsuoka, “A user-level infiniband-based file system and checkpoint strategy for burst buffers,” 05 2014, pp. 21–30.
  * [38] D. Kimpe, K. Mohror, A. T. Moody, B. C. V. Essen, M. B. Gokhale, R. B. Ross, and B. R. de Supinski, “Integrated in-system storage architecture for high performance computing,” in _ROSS ’12_ , 2012.
  * [39] “Supermuc-ng.” [Online]. Available: https://doku.lrz.de/display/PUBLIC/SuperMUC-NG