# Distributed sequential federated learning

Z. F<EMAIL_ADDRESS>X. Y. Zhang222<EMAIL_ADDRESS>
International Institute of Finance, The School of Management,
University of Science and Technology of China, China

and

Y-c I<EMAIL_ADDRESS>
Institute of Statistical Science,
Academia Sinica, Taipei 11529, Taiwan

###### Abstract

The analysis of data stored in multiple sites has become more popular, raising
new concerns about the security of data storage and communication. Federated
learning, which does not require centralizing data, is a common approach to
preventing heavy data transportation, securing valued data, and protecting
personal information protection. Therefore, determining how to aggregate the
information obtained from the analysis of data in separate local sites has
become an important statistical issue. The commonly used averaging methods may
not be suitable due to data nonhomogeneity and incomparable results among
individual sites, and applying them may result in the loss of information
obtained from the individual analyses. Using a sequential method in federated
learning with distributed computing can facilitate the integration and
accelerate the analysis process. We develop a data-driven method for
efficiently and effectively aggregating valued information by analyzing local
data without encountering potential issues such as information security and
heavy transportation due to data communication. In addition, the proposed
method can preserve the properties of classical sequential adaptive design,
such as data-driven sample size and estimation precision when applied to
generalized linear models. We use numerical studies of simulated data and an
application to COVID-19 data collected from 32 hospitals in Mexico, to
illustrate the proposed method.

Keywords: Adaptive sampling; Data Communication; COVID-19; Random average;
Sequential sampling

##  1 Introduction

When data are independently collected in many separate sites, data
transportation and machine/data communication are essential issues. In
addition to efficiency issues in data transportation and/or communication,
information security has become even more crucial (Damiani et al., 2015; Huang
et al., 2020). If maintaining a centralized data warehouse is impractical due
to cost, security and/or other reasons, federated learning is a feasible
option that enables data scientists to manage and utilize all data for their
analysis tasks. The key to this type of method is avoiding obstacles due to
data communication and transportation. However, the discussions of federated
learning in the machine learning literature, focus more on the perspectives of
information technicality (Yan et al., 2013; Jordan et al., 2019; Li et al.,
2018; Goetz et al., 2019; Li et al., 2020b; Zhou & Tang, 2020), and there is a
lack of consideration of statistical issues.

The common features of these methods are the analysis of data separately at
each site, and the integration of the results from individual sites together
in a convenient way. However, the nonhomogeneity among individual site data
may pose a challenge for result integration via usual methods such as weighted
average and statistical robust methods that bypass this issue by treating the
results obtained from the separated sites as random observations (McMahan et
al., 2017). Although some methods can deal with outliers, they are not
efficient and effective in terms of sample size and will usually not retain
all the information from the individual analyses. The issues of legitimately
and effectively merging the local results from individual data sites and the
corresponding statistical properties urgently require further (Dwork & Aaron,
2011; Carlini et al., 2019; Zhou & Tang, 2020). Thus, developing a method that
addresses the original issues raised in the common federated learning
discussion and can obtain better and more precise information of interest is
the aim of this research.

In this paper, we propose a novel sequential method that takes advantage of
both distributed computing and sequential sampling. We adopt the idea of
sequential analysis within each local site as a means of efficiently and
effectively using the data in each site and then study how to integrate the
results from the sepqrate analyses based on their formation contents in terms
of both quantity and quality, instead of using the common weighted average
which mostly depends on the prefixed sample sizes of the local sites. Because
the data sets were collected before-hand, sampling is not an obstacle as it is
in conventional clinical trials where recruiting new subjects into a study is
rather complicated (Woodroofe, 1982; Bartroff et al., 2013; Whitehead, 2014).
We apply the proposed method to generalized linear models and use logistic
regression models to illustarte the details of our methods. We focus on
classification problems, present the numerical results using simulated data
and a real data set including data collected in 32 different hospitals in
Mexico, and then summarize our findings.

##  2 Federate sequential learning

We apply the proposed method to the generalized linear model (GLM, Nelder &
Wedderburn, 1972; McCullagh & Nelder, 1989), including logistic regression
models, to illustrate the concept.

Suppose there are $M$ separate data sites, and $(Y,\bm{{X}})$ are the response
and covariate variables, respectively. Then, for site $j$, with sample size
$n_{i}$, $(Y_{ij},\bm{{X}}_{ij})$, $j=1,\cdots,M$, $i=1,\cdots,n_{j}$ denotes
its $i$th pair of observations. Assume that the data of each individual site
satisfy a generalized linear model with a link function $\mu$ such that
$E(Y_{j}|\bm{{X}}_{j})=\mu(\bm{{X}}_{j}^{\top}\bm{{\beta}}_{j})$ and
$var(Y_{j}|\bm{{X}}_{j})=\nu(\bm{{X}}_{j}^{\top}\bm{{\beta}}_{j})>0$, where
$\bm{{\beta}}_{j}$ $j=1,\cdots,M$, is an unknown length $p_{j}$ parameter
vector. Define $\bm{{U}}$ as a $p_{0}$ vector of common variables of interest
within all sites. Let $\bm{{V}}_{j}$ be a $p_{j}-p_{0}$ vector of other
variables in site $j$. Note that the length of $\bm{{V}}_{j}$ may vary among
sites. Define $\bm{{X}}_{j}=(\bm{{U}}^{\top},\bm{{V}}_{j}^{\top})^{\top}$,
then with
$\bm{{\beta}}_{j}=(\bm{{\theta}}^{\top},\bm{{\eta}}_{j}^{\top})^{\top}$, we
have that

$\displaystyle
E(Y_{j}|\bm{{X}}_{j})=\mu(\bm{{\theta}}^{\top}\bm{{U}}+\bm{{V}}_{j}^{\top}\bm{{\eta}}_{j}),$
(1) $\displaystyle
var(Y_{j}|\bm{{X}}_{j})=\nu(\bm{{\theta}}^{\top}\bm{{U}}+\bm{{V}}_{j}^{\top}\bm{{\eta}}_{j})>0.$
(2)

###### Remark 2.1

The generalized linear model family includes many popular models, where
logistic regression models are commonly used in many fields. In these models,
equations (1) and (2), implies that for $j=1,\cdots,M$,

$\displaystyle\mu(\bm{{\theta}}^{\top}\bm{{U}}+\bm{{V}}_{j}^{\top}\bm{{\eta}}_{j})=\frac{\exp{\bm{{\theta}}^{\top}\bm{{U}}+\bm{{V}}_{j}^{\top}\bm{{\eta}}_{j})}}{1+\exp{(\bm{{\theta}}^{\top}\bm{{U}}+\bm{{V}}_{j}^{\top}\bm{{\eta}}_{j})}},$
$\displaystyle\nu(\bm{{\theta}}^{\top}\bm{{U}}+\bm{{V}}_{j}^{\top}\bm{{\eta}}_{j})=\mu(\bm{{\theta}}^{\top}\bm{{U}}+\bm{{V}}_{j}^{\top}\bm{{\eta}}_{j})(1-\mu(\bm{{\theta}}^{\top}\bm{{U}}+\bm{{V}}_{j}^{\top}\bm{{\eta}}_{j})).$
(3)

For a linear regression model, (1) and (2) are

$\displaystyle\mu(\bm{{\theta}}^{\top}\bm{{U}}+\bm{{V}}_{j}^{\top}\bm{{\eta}}_{j})=\bm{{\theta}}^{\top}\bm{{U}}+\bm{{V}}_{j}^{\top}\bm{{\eta}}_{j},$
$\displaystyle\nu(\bm{{\theta}}^{\top}\bm{{U}}+\bm{{V}}_{j}^{\top}\bm{{\eta}}_{j})=\sigma_{j}^{2}>0.$
(4)

If the sample sizes of the individual procedures are prefixed, then there are
many classical methods for combining the results of $M$ procedures. Among
them, some voting schemes and the concepts of weighted and robust statistical
methods are popular choices. Because these methods are based on prefixed
sample sizes, the variability among the estimates from different sites can be
high when there are large differences in the sample sizes between sites. It is
difficult to reduce the variation, even by choosing the same sample sizes for
all sites, when the data quality of individual pools varies, which is common
in some applications such as the context of the coronavirus pandemic. On the
other hand, if results from individual sites have the required asymptotic
statistical properties, then it is possible to combine them effectively and
efficiently.

We use sequential procedures to retain both parameter estimation and
prediction power. This is novel in sequential analysis, to the best of our
knowledge. For generalized linear models, we use confidence set estimation
with a prescribed accuracy for the parameter of interest, say $\bm{{\theta}}$
for the estimation quality. The index for prediction may vary depending on the
types of response variables. For logistic regression models, in addition to
the size of the confidence set, we control the prediction performances via the
area under the receiver operation characteristic (ROC) curve, named AUC, while
for linear regression models, we calculate the mean square error (MSE) and/or
corresponding $R^{2}$ statistics and use them as performance indexes.

Although the sample size of each procedure is random due to the stopping
criterion and there is a lack of discussion about combining the results of
many random sample-sized procedures, prioritizing the required statistical
properties, instead of the sample sizes, seems more reasonable for analyzing
combined data sets.

Here, we study valid integration of the results of $M$ procedures such that
the final result has the desired property as in a conventional sequential
procedure, and use the fixed-size confidence set estimation of logistic
regression and linear models as examples. Because we can independently conduct
$M$ estimation procedures without a centralized data center, the proposed
method retains the key features of federated learning, such as protecting the
personal information of the subjects at each site and reducing the data
communication requirements. We now describe the $j$th sequential procedure,
one of the $M$ sequential procedures, and the method for integrating $M$
independent results below.

###### Remark 2.2

By runing $M$ estimation procedures separately, we can decrease the
computational load of each computer and, in fact, accelerate the computing
efficiency by using smaller sample sizes in the individual computing
facilities of local sites, since all sites can run their own procedures
simultaneously. Thus, the proposed method can also benefit from a distributed
computing framework, and each local site can be free to maintain its own high-
power high-cost computing facility. This feature of the distributed sequential
methods will certainly promote the uses of the federated learning methods.

###  2.1 Sequential estimate with fixed-sized confidence set and reserved
prediction precision

To combine the results, and overcome the issue of the nonhomogeneity among
local sites, we require the confidence estimation of each procedure to achieve
a given size and a prescribed accuracy. Instead, we allow the sample size used
in each procedure to be variable and random, depending on the sequential
sampling strategy and the quality of local data. The idea of sequential
analysis is an appropriate tool for these purposes. Because we conduct $M$
procedures separately and independently with their own local data pools, we
will just specify one of them in detail, say the $j$th procedure, and then
focus more on how to combine results.

Let $C_{jn}=\\{(y_{ji},\bm{{x}}_{ji}),i=1,...,n\\}$ be the recruited data
subset of the original data set $\mathcal{D}$ at stage $n$. Then at this
stage, we compute the maximum quasi-likelihood estimate (MQLE) for
$\bm{{\beta}}$ in (1) and (2), and denote them as
$\tilde{\bm{{\beta}}}_{jn}=(\tilde{\bm{{\theta}}}_{jn}^{\top},\tilde{\bm{{\eta}}}_{jn}^{\top})^{\top}$,
which are the solutions to the estimating equation (McCullagh & Nelder, 1989):

$\displaystyle
ln(\tilde{\bm{{\beta}}}_{jn})\equiv\sum_{i=1}^{n}\dot{\mu}(\bm{{X}}_{j}^{\top}\bm{{\beta}}_{j})w(\bm{{X}}_{j}^{\top}\bm{{\beta}}_{j})[y_{ji}-\mu(\bm{{x}}_{ji}^{\top}\tilde{\bm{{\beta}}}_{jn})]\bm{{x}}_{ji}=0,$
(5)

where $\dot{\mu}(t)=d\mu(t)/dt$ is the first derivative of $\mu(t)$ and
$w(t)=\nu^{-1}(t)$. Let $\bm{{L}}_{j}$ be a $p_{0}\times p_{j}$ matrix that
consists of the first $p_{0}$ rows of the diagonal matrix,
$\text{diag}\\{I_{j1},\cdots,I_{jp_{j}}\\}$, with $I_{j1}=\cdots=I_{jp_{0}}=1$
and $I_{jk}=0,k=p_{0}+1,\cdots,p_{j}$. Then, the estimates of the parameters
of interest are
$\tilde{\bm{{\theta}}}_{jn}=\bm{{L}}_{j}\tilde{\bm{{\beta}}}_{jn}$. Assume the
following conditions are satisfied.

(A1) $\sup_{i}||\bm{{x}}_{i}||<\infty$ for all variable $\bm{{x}}_{i}$, and
$E|\epsilon_{i}|^{\zeta}<\infty$ with some $\zeta>2$, where
$\epsilon_{i}=Y_{i}-\mu(\bm{{x}}_{ji}^{\top}\bm{{\beta}}_{j0})$ is the error
term.

(A2)
$\lim_{n\rightarrow\infty}\sum_{i=1}^{n}\bm{{x}}_{ji}\\{\dot{\mu}(\bm{{x}}_{ji}^{\top}\bm{{\beta}}_{j0})^{2}/\nu(\bm{{x}}_{ji}^{\top}\bm{{\beta}}_{j0})\\}\bm{{x}}_{ji}^{\top}/n=\Sigma_{j}$,
where $\Sigma_{j}$ is a positive definite matrix and $\bm{{\beta}}_{j0}$ is
the true value of $\bm{{\beta}}_{j}$.

Then MQLE $\tilde{\bm{{\beta}}}_{jn}$ of the GLM is a strong consistent
estimate of $\bm{{\beta}}_{j}$ and has the asymptotic normal property (Chen et
al., 1999; Chang, 1999). It follows that

$\displaystyle\sqrt{n}(\tilde{\bm{{\theta}}}_{jn}-\bm{{\theta}}_{0})\longrightarrow
N(0,{\bm{{L}}_{j}}\Sigma_{j}^{-1}{\bm{{L}}_{j}})~{}~{}~{}~{}\text{\rm in
distribution as }n\rightarrow\infty,$ (6)

where $\bm{{\theta}}_{0}$ is the true value of the common parameters of
interest, $\bm{{\theta}}$.

Performance measure and its estimate Without loss of generality, here we let
$A_{j}$ be a prediction measure, and $\hat{A}_{j}$ be a strongly consistent
estimator of it with variance estimate $v_{Aj}$. It is straightforward to show
that $(\hat{A}_{j}-A_{j})/\sqrt{v_{Aj}}$ converges to $N(0,1)$ in
distribution, as $n$ goes to $\infty$. Let
$\hat{y}^{n}_{ji}=\mu(\bm{{x}}_{ji}^{\top}\tilde{\bm{{\beta}}}_{jn})$ be the
fitted values of $y_{ji}$ when using the data set $C_{jn}$.

Let $S_{1}=\\{\hat{y}_{ji}:y_{ji}=1\\}$ and
$S_{0}=\\{\hat{y}_{ji}:y_{ji}=0\\}$, and $n_{0}$ and $n_{1}$ be the sizes of
$S_{0}$ and $S_{1}$, respectively, Then for a logistic regression model
described by (2.1), Zhou et al. (2009) shows that

$\displaystyle\widehat{AUC}_{j}=\frac{1}{n_{0}n_{1}}\sum_{v_{1}\in
S_{1}}\sum_{v_{2}\in S_{0}}I(v_{1}\geq v_{2}),$ (7)

is an estimate of the AUC using the data set $C_{jn}$, where $I(\cdot)$ is an
indicator function. Let ${\varepsilon}{Aj}$ be the variance of
$\widehat{AUC}_{j}$. It follows that
$(\widehat{AUC}_{j}-AUC_{j})/\sqrt{v_{Aj}}$ converges in distribution to
$N(0,1)$ as $n$ tends to $\infty$, where $AUC_{j}$ is the true AUC under
(2.1).

Sequential estimation procedure Let $C_{jn_{0}}$ be the initial data set with
size $n_{0}$ of procedure $j$, and $a$ be the square root of the $1-\alpha$
quantile of a $\chi^{2}_{p_{0}}$ chi-square distribution with $p_{0}$ degrees
of freedom. Let $\tilde{a}_{j}>0$, $j=1,\cdots,M$, such that
$\sum_{j=1}^{M}\tilde{a}_{j}^{2}=a^{2}.$ Values of $\tilde{a}_{j}$ are taken
based on the customer’s choice. For example, usually
$\tilde{a}^{2}_{j}=a^{2}/M$. When one data site $j$ has a small data size,
$\tilde{a}_{j}$ should be assigned a small value to avoid a sequential
procedure that cannot stop even using all samples at the site. Different
$\\{\tilde{a}_{j}:j=1,\cdots,M\\}$ values do not affect the statistical
properties of the final parameter estimation, as long as the sequential
procedures satisfy for all $M$ sites. Let $a_{p}$ be the $1-\alpha/2$ quantile
of $N(0,1)$. Then, for a given $d_{1},d_{2}>0$, a stopping rule
$\tilde{N}_{j}$ is defined as

$\displaystyle\tilde{N}_{j}=N_{d_{1},d_{2}}\equiv\inf\left\\{k:~{}k\geq
n_{0}~{}~{}\mbox{and}~{}~{}{\mu}_{jk}\leq\frac{d_{1}^{2}k}{\tilde{a}_{j}^{2}}~{}~{}\mbox{and}~{}~{}{\varepsilon}{A_{j}}\leq\left(\frac{d_{2}}{a_{p}}\right)^{2}\right\\},$
(8)

where
${\mu}_{jk}=\lambda_{max}[k\bm{{L}}_{j}\bm{{\Sigma}}_{jk}^{-1}\bm{{L}}_{j}]$,
$\bm{{\Sigma}}_{jk}=\sum_{i=1}^{k}\bm{{x}}_{ji}\\{\dot{\mu}(\bm{{x}}_{ji}^{\top}\bm{{\beta}}_{j0})^{2}/\nu(\bm{{x}}_{ji}^{\top}\bm{{\beta}}_{j0})\\}\bm{{x}}_{ji}^{\top}$,
and $\lambda_{max}(A)$ denotes the maximum eigenvalue of matrix $A$.

In (8), $\mu_{jk}\leq(d_{1}^{2}k)/\tilde{a}_{j}^{2}$ controls the precision of
the estimates for the parameters of interest, and
${\varepsilon}{Aj}\leq({d_{2}}/{a_{p}})^{2}$ maintains the prediction accuracy
of the model. When $d_{1}$ and $d_{2}$ decrease, the sample size
$\tilde{N}_{j}$ increases, and then we estimate
$\tilde{\bm{{\theta}}}_{j\tilde{N}_{j}}$ with greater accuracy and prediction
accuracy based on $A_{j}$. It follows from (8) that practitioners can choose
$d_{1}$ and $d_{2}$ based on their application needs.

Suppose we are at the $(k-1)$st stage, $k>n_{0}$, of the sequential estimation
procedure for the $j$th site, and have recruited $k-1$ samples into the
estimation procedure. If the inequalities in (8) are satisfied with data set
$C_{j{k-1}}$, then we stop recruiting new samples and report the current
results. Otherwise, we select an additional sample from the data pool of the
$j$th site and use all $k$ samples in $C_{j{k}}$ to calculate new estimates
$\tilde{\bm{{\beta}}}_{jk}$ and $\nu_{jk}$. This sequential recruiting process
is repeated until the inequalities in (8) are satisfied. It has been proven
that the estimates of parameters for GLM have uniform continuity in
probability (u.c.i.p.) property (Woodroofe, 1982; Chang & Martinsek, 1992;
Chang, 2011). Moreover, the property u.c.i.p. implies that the estimates are
asymptotically normally distributed as the sample size goes to infinity. Thus,
for each procedure $j$, $\tilde{\bm{{\theta}}}_{j\tilde{N}_{j}}$ has the
following asymptotic properties:

$\displaystyle\sqrt{\tilde{N}_{j}}(\tilde{\bm{{\theta}}}_{j}-\bm{{\theta}}_{0})\longrightarrow
N(0,{\bm{{L}}_{j}}\Sigma_{j}^{-1}{\bm{{L}}_{j}})~{}~{}~{}~{}\text{\rm in
distribution as }d_{1}\rightarrow 0,$
$\displaystyle\frac{\hat{A}_{j}-A_{j}}{\sqrt{v_{Aj}}}\longrightarrow
N(0,1)~{}~{}~{}~{}\text{\rm in distribution as }d_{2}\rightarrow 0.$

As mentioned before, we independently conduct $M$ estimation procedures for
constructing a confidence set of prefixed size for $\bm{{\theta}}$ in (2.1)
and each procedure sequentially recruits samples from its corresponding data
pool without replacement. The fixed-sized (fixed-precision) feature of these
$M$ estimates allows us to legitimately combine them into one final result
with the desired statistical properties. We will state our integration method
below.

###  2.2 Combining sequential estimations from data centers

Equation (8) states that $\tilde{a}_{j}^{2}$ only depends on ${p}_{0}$ – the
number of the variables of interest. Moreover, if $\tilde{a}_{j}^{2}$ becomes
larger, then we need to recruit more samples. Thus, we may adjust the sample
proportion among $M$ sites according to the data collection status for each
site through the value of $\tilde{a}_{j}$ as long as
$\sum_{j=1}^{M}\tilde{a}_{j}^{2}=a^{2}$. When site $k$ has a much smaller
number of subjects compared to others, we can take $\tilde{a}_{k}^{2}$ with a
smaller value such that it can provide appropriate information with its
limited data size via the predefined stopping criterion.

When all stopping rules $\tilde{N}_{j}$, $j=1,\cdots,M$, are fulfilled, we
stop recruiting new data into our estimation procedures. Let

$\displaystyle\hat{N}$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{M}\tilde{N}_{j},$ (9)

and

$\displaystyle\hat{\bm{{\theta}}}$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{M}\rho_{j}\tilde{\bm{{\theta}}}_{j\tilde{N}_{j}},$
(10)

where $\rho_{j}=\tilde{N}_{j}/\hat{N}$. Because these stopping times
$\tilde{N}_{j}$, $j=1,\cdots,M$, are random, the sum of stopping time
$\hat{N}$ indicates the total (random) samples used in $M$ processes together,
and $\hat{\bm{{\theta}}}$ is a “randomly weighted” estimate of
$\bm{{\theta}}_{0}$ with random weights $\\{\rho_{j}\\}$, $j=1,\cdots,M$.

###### Lemma 2.1

Assume that for each $j$, $\\{(\bm{{x}}_{ji},y_{ji}),i\geq 1\\}$ follows the
GLM (1) and (2), and Conditions (A1) and (A2) hold. Then we have the estimate
$\hat{\bm{{\theta}}}$ with the random weights $\rho_{j}$ that achieves the
minimal covariance asymptotically if all data centers have the same covariates
and homogeneous covariance $\Sigma_{1}=\Sigma_{2}=\cdots=\Sigma_{M}$.

The proof of Lemma 1 is in Supplementary. When all sites have homogeneous data
with the same covariates, the proposed random weight estimate achieves
asymptotical efficiency under the weight combination of the parameter
estimates from $M$ sites. Especially when $\tilde{a}_{j}$ are different, the
proposed estimate has smaller asymptotical variance than the usual average of
the parameter estimates from $M$ sites.

###### Remark 2.3

In practice, the data pool in each local site may contain some specific
variables for their own, such as administrative purposes, which are usually
not of interest for the analysis. Thus, focusing on a set of common variables
among all sites is reasonable.

Because $M$ procedures independently use their own data pools, and each
estimate has the u.c.i.p. property, it follows that, as $d_{1}\rightarrow 0$,

$\displaystyle\sqrt{\hat{N}}\left(\sum_{j=1}^{M}\rho_{j}{\bm{{L}}_{j}}\Sigma_{j}^{-1}{\bm{{L}}_{j}}\right)^{-1/2}\left(\hat{\bm{{\theta}}}-\bm{{\theta}}_{0}\right)\longrightarrow
N(0,\bm{{I}}_{p_{0}})\hbox{ \rm in distribution},$ (11)

where $\bm{{I}}_{p_{0}}$ is an identity matrix with rank $p_{0}$. Equation
(11) implies that

$\displaystyle(\hat{\bm{{\theta}}}-\bm{{\theta}}_{0})^{\top}\tilde{\bm{{\Sigma}}}^{-1}(\hat{\bm{{\theta}}}-\bm{{\theta}}_{0})\longrightarrow\chi_{p_{0}}^{2},\hbox{
as }d_{1}\rightarrow 0,$ (12)

where
$\tilde{\bm{{\Sigma}}}=\left(\sum_{j=1}^{M}\rho^{2}_{j}{\bm{{L}}_{j}}\Sigma_{j\tilde{N}_{j}}^{-1}{\bm{{L}}_{j}}\right)$.
Let $\bm{{Z}}=(z_{1},\cdots,z_{p_{0}})^{T}$, then

$\displaystyle R_{\hat{N}}\\!\\!=\\!\left\\{\bm{{Z}}\in
R^{p_{0}}\\!\\!:~{}~{}\\!\\!\frac{S_{\hat{N}}}{\hat{N}}\leq\frac{d_{1}^{2}}{\mu_{\hat{N}}}\right\\}$
(13)

defines a confidence set for $\bm{{\theta}}_{0}$, where
$S_{\hat{N}}=(\bm{{Z}}-\hat{\bm{{\theta}}})^{\top}\tilde{\bm{{\Sigma}}}^{-1}(\bm{{Z}}-\hat{\bm{{\theta}}})$,
$\mu_{\hat{N}}=\lambda_{max}\left(\hat{N}\tilde{\bm{{\Sigma}}}\right)$, and
the length of the maximum axis of the confidence set $R_{\hat{N}}$ is $2d$.
For the stopping time $\hat{N}$ and confidence set $R_{\hat{N}}$, we have the
following Theorem 2.1, and its proof is included in the Supplementary
materials.

###### Theorem 2.1

Assume that for each $j$, $\\{(\bm{{x}}_{ji},y_{ji}),i\geq 1\\}$ follows the
GLM (1) and (2), and Conditions (A1) and (A2) are satisfied. Then

$\displaystyle(i)\lim_{d_{1}\rightarrow
0}\frac{d_{1}^{2}\hat{N}}{a^{2}{\mu}}=1,\hbox{\rm almost surely},$
$\displaystyle(ii)\lim_{d_{1}\rightarrow 0}P(\bm{{\theta}}_{0}\in
R_{\hat{N}})=1-\alpha,$ $\displaystyle(iii)\lim_{d_{1}\rightarrow
0}\frac{d^{2}E(\hat{N})}{a^{2}{\mu}}=1,$

where ${\mu}$ is the maximum eigenvalue of
$\sum_{j=1}^{M}\rho_{j}{\bm{{L}}_{j}}\Sigma_{j}^{-1}{\bm{{L}}_{j}}$, and
$a^{2}$ is the $1-\alpha$ quantile of $\chi_{p_{0}}^{2}$.

This theorem shows that the proposed method has the prespecified coverage
probability $1-\alpha$, and the ratio of the (random) total sample size to the
best “theoretical” (but unknown) size is asymptotically equal to 1. Chow &
Robbins (1965) referred to these two properties as asymptotic consistency and
asymptotic efficiency, respectively.

The proposed method for aggregating the information among heteroscedastic data
sites goes beyond the conventionally discussed challenges that drove the
development of federated learning. In practice, the sizes and quality of data
are different among different data sites. In a confidence set estimation
scenario, it may not be possible to achieve the prefixed accuracy even when
using all data in a site, which has only a small amount of data. With the
proposed procedure, we can assign smaller $\tilde{a}_{j}$ to such sites and
then effectively integrate all results from variant sites. Thus, the data
usage is more efficient compared to that of other combining methods.

###### Remark 2.4

We can also modify the current methods by adopting a multiple-stage sequential
method instead so that we can find more information about individual sites
when there is little information available in advance. Please refer to Park &
Chang (2016) and the references therein.

###  2.3 Adaptive sampling strategy

Active learning is a long-standing topic in the machine learning literature.
Researchers may use conventional experimental design criteria, such as some
optimal design criteria, A-optima and D-optima criteria, for recruiting the
most “informative” samples from the collected data (Woods et al., 2006; Deng
et al., 2009; Montgomery, 2009; Chen et al., 2020).

Let
$\mathbf{A}=\sum_{i=1}^{k}\bm{{x}}_{ji}\\{\dot{\mu}(\bm{{x}}_{ji}^{\top}\bm{{\beta}}_{j0})^{2}/\nu(\bm{{x}}_{ji}^{\top}\bm{{\beta}}_{j0})\\}\bm{{x}}_{ji}^{\top}$
for $j$th data site at the $k$th stage, where $\\{\bm{{x}}_{ji}:i=1,...,k\\}$
denotes the (active) set of the selected samples up to the $k$th stage. Let
$\mathbf{U}$ be an inactive sample pool as in Settles (2010) (see also Chen et
al., 2020; Li et al., 2020a). The A-optima criterion will allocate a new
observation $\bm{{x}}_{j}^{*}$ from the given data pool such that

$\displaystyle\bm{{x}}_{j}^{*}={\mathrm{argmin}}_{x\in\mathbf{U}}\mathrm{tr}\\{(\mathbf{A}+\bm{{x}}\\{\dot{\mu}(\bm{{x}}^{\top}\bm{{\beta}}_{j0})^{2}/\nu(\bm{{x}}^{\top}\bm{{\beta}}_{j0})\\}\bm{{x}}^{\top})^{-1}\\},$

where notation $\mathrm{tr}(\cdot)$ is the trace function. Thus, we use the
A-optima criterion to select a new observation at each stage in the modeling
process and repeat this procedure until the stopping criterion (8) is
satisfied. We then integrate results from all sites similar to those used in
the random sampling procedure. Following the arguments of Wang & Chang (2013),
we have the required asymptotic properties under an adaptive sampling
scenario. The details are then omitted.

###### Remark 2.5

Although we have mainly discussed the confidence set estimation problems, this
method can also accelerate the analysis of big data sets and not just for
federated learning scenarios. It can be easily implemented on different
computing frameworks due to its distributed computing feature. For example, we
can simply partition a big data pool into M subdata sets and then fit the
desired models separately using individual subdata sets. Thus, the proposed
method goes beyond the original perspectives for developing federated learning
from computing viewpoints.

##  3 Numerical studies

###  3.1 Simulation studies

The performance of the proposed distributed sequential federated learning
estimation (DSFL) method is investigated through simulation studies. We
generate data using a logistic regression model:

$\displaystyle
P(Y=1|\bm{{X}})=\mu(\bm{{X}}^{\top}\bm{{\beta}})=\frac{\exp{(\beta_{0}+\bm{{X}}^{\top}\bm{{\beta}})}}{1+\exp{(\bm{{X}}^{\top}\bm{{\beta}})}},$
(14)

where $\bm{{\beta}}\in R^{p_{k}-1}$ is the parameter vector of this model. We
set $M=5$ for our simulation study, and use real data to evaluate our methods
with more sites later.

Below we list the parameters used in the simulation studies:

  * (1)

Let B1 denote the case that
$\bm{{\beta}}_{k}=(\beta_{k0},\bm{{\theta}}^{\top},\bm{{\beta}}_{k}^{\top})^{\top}$,
$\bm{{\theta}}=(2.0,1.0)^{\top}$, $k=1,\cdots,5$ where
$\bm{{\theta}}=(2.0,1.0)^{\top}$ is the same for all sites, and B2 denote the
case that the parameter vectors are different among sites with $\beta_{k0}$,
and $\bm{{\beta}}_{k}^{\top}$ as follows:

$\displaystyle(\beta_{k0},\bm{{\theta}}^{\top},\bm{{\beta}}_{1}^{\top})^{\top}$
$\displaystyle=(-2,$ $\displaystyle 2,1.0,1.0,0)^{\top},$
$\displaystyle(\beta_{k0},\bm{{\theta}}^{\top},\bm{{\beta}}_{2}^{\top})^{\top}$
$\displaystyle=(-2,$ $\displaystyle 2,1.0,1.0,0.5)^{\top},$
$\displaystyle(\beta_{k0},\bm{{\theta}}^{\top},\bm{{\beta}}_{3}^{\top})^{\top}$
$\displaystyle=(-2,$ $\displaystyle 2,1.0,1.0,0.5,0)^{\top},$
$\displaystyle(\beta_{k0},\bm{{\theta}}^{\top},\bm{{\beta}}_{4}^{\top})^{\top}$
$\displaystyle=(-1.5,$ $\displaystyle 2,1.0,1.0,0)^{\top},$
$\displaystyle(\beta_{k0},\bm{{\theta}}^{\top},\bm{{\beta}}_{5}^{\top})^{\top}$
$\displaystyle=(-2.5,$ $\displaystyle 2,1.0,1.0,1.0)^{\top}.$

Note that in addition to the values, the length of $\bm{\beta}_{3}$ is also
different from the others.

  * (2)

For $p1$, $\rho_{k}=1/5$, for all $k=1,\cdots,5$, and for $p2$, we have
$\rho_{1},\cdots,\rho_{4}=1/10$, and $\rho_{5}=6/10$.

  * (3)

Random vector $X$ under $h1$ follows
$N(0,{\rm{diag}}(\phi_{ki},i=1,\cdots,p_{k}-1))$ with $p_{k}=5$, $\phi_{ki}=1$
for $i=1,\cdots,p_{k}-1,k=1,\cdots,5$. For $h2$, $X$ follows
$N(0,{\rm{diag}}(\phi_{ki},i=1,\cdots,p_{k}-1))$ with different $p_{k}$,
$\phi_{23}=\phi_{24}=4,\phi_{43}=\phi_{44}=2,\phi_{53}=\phi_{54}=4$, and the
other $\phi_{ki}$ are equal to 1, for $k=1,\cdots,5$.

In our studies with the synthesized data, the sizes of the confidence set for
parameters $(\beta_{1},\beta_{2})$ are $d_{1}=0.4,0.3$ and $0.2$, and the
parameter for controlling the AUC of each model is $d_{2}=0.06,0.05,0.04$. We
set the significance level at $\alpha=0.05$, and the results were based on 200
replications for all cases.

Table 1 presents the stopping times, coverage frequency (CF), and average AUC
for models with either random (R) or adaptive (A) sample selection methods,
using the parameter setup B1. The stopping time, $N$, increases as $d_{1}$
goes to 0, and then the coverage frequency approaches the theoretical value,
0.95. The procedures with adaptive sampling via the A-optimal design criterion
have smaller stopping times than their random sampling counterparts. When
$\rho=1/5$ for each site, as in $p1$, the stopping times for all 5 sites are
similar. However, when the values of $\rho$ are different, as in $p2$, Sites
1-4 ($\rho_{1}=\cdots=\rho_{4}=0.1$) have much smaller sample sizes than Site
5 ($\rho_{5}=0.6$). This shows that we can control the size of the sample used
at each site via choosing an appropriate $\rho$.

Tables 2 and 3 show the absolute bias of the parameter estimation of interest
from the proposed method, parameter estimates by combining 5 sites with
equivalent weights, and each site for the R and A selection methods. We found
that compared with combining results from all 5 data sites, the results via
single site data have much larger biases, and standard deviations. When
$d_{1}$ is small, the proposed method has similar biases and standard
deviation under the $p1$ setup, compared to that of the method via straight
average. However, under the $p2$ setup, the proposed method have smaller
biases and standard deviations than the straight average method. The
differences are even more obvious when an adaptive sampling strategy is
adopted. As expected, the stopping times of all 5 sites are similar under
$p1$, and vary under $p2$.

We use both $h1$ and $h2$ to study the heterogeneity effects due to other
variables. For each data site, regression parameters have different lengths
with their values as described in $B2$ above. We consider 4 combinations of
parameters: $p1$ and $h1~{}(s1)$, $p2$ and $h1~{}(s2)$, $p1$ and $h2~{}(s3)$,
and $p2$ and $h2~{}(s4)$. Table 4 reports the stopping times, coverage
frequency (CF), and average AUC of 5 sites with $d_{1}=0.2$ and $d_{2}=0.05$.
Table 5 shows the corresponding absolute bias for the estimates of the
parameters of interest obtained via the proposed and straight average methods
for each site. For cases $s2$ and $s4$, the proposed method has smaller biases
and standard deviations than those of the simple average and methods using
only single site data.

The results based on the synthesized data confirm that the proposed sequential
federated learning yeilds a satisfactory result for estimating the parameters
of $\beta_{1}$ and $\beta_{2}$ in all setups, and retains the demanding
statistical properties in this study.

Table 1: stopping times, AUC and coverage frequency with homogeneous covariates of each data site for the simulated data. |  |  |  | Stopping time |  |  |
---|---|---|---|---|---|---|---
| $d_{2}$ | $d_{1}$ |  | $N$ | $N_{1}$ | $N_{2}$ | $N_{3}$ | $N_{4}$ | $N_{5}$ |  | AUC | CF
R | 0.06 | 0.4 | p1 | 1127.420(116.651) | 219.750(52.976) | 227.640(58.507) | 228.335(53.052) | 227.215(54.243) | 224.480(55.704) |  | 0.903(0.009) | 0.915
|  |  | p2 | 1193.870(123.002) | 141.160(31.880) | 144.850(31.103) | 148.820(33.206) | 146.380(32.687) | 612.660(98.263) |  | 0.907(0.009) | 0.965
|  | 0.3 | p1 | 1838.760(185.119) | 361.895(83.340) | 370.995(82.755) | 370.225(78.068) | 371.235(74.345) | 364.410(86.074) |  | 0.901(0.008) | 0.895
|  |  | p2 | 1862.495(173.527) | 201.465(47.522) | 198.705(48.032) | 204.145(45.102) | 207.550(50.823) | 1050.630(140.413) |  | 0.903(0.008) | 0.955
|  | 0.2 | p1 | 3962.110(270.694) | 795.065(113.044) | 793.180(117.097) | 781.050(118.549) | 802.765(117.263) | 790.050(121.204) |  | 0.902(0.005) | 0.955
|  |  | p2 | 3954.730(259.459) | 409.295(84.372) | 407.690(79.709) | 408.225(79.819) | 410.225(90.593) | 2319.295(208.466) |  | 0.901(0.007) | 0.960
| 0.05 | 0.4 | p1 | 1180.350(110.364) | 241.200(55.236) | 231.700(43.742) | 237.860(49.061) | 234.910(51.000) | 234.680(39.495) |  | 0.904(0.009) | 0.920
|  |  | p2 | 1292.470(125.559) | 170.245(42.899) | 171.965(39.977) | 173.190(40.141) | 176.115(41.536) | 600.955(107.197) |  | 0.908(0.008) | 0.965
|  | 0.3 | p1 | 1845.155(173.025) | 377.325(84.753) | 368.590(68.487) | 369.100(78.389) | 365.540(77.612) | 364.600(79.501) |  | 0.902(0.008) | 0.925
|  |  | p2 | 1918.810(150.884) | 217.020(43.842) | 214.135(41.127) | 219.315(42.147) | 218.955(42.511) | 1049.385(139.777) |  | 0.905(0.007) | 0.925
|  | 0.2 | p1 | 3987.365(271.466) | 807.735(121.610) | 792.770(101.409) | 796.120(131.255) | 799.635(127.288) | 791.105(126.285) |  | 0.902(0.005) | 0.915
|  |  | p2 | 3950.750(252.432) | 408.005(79.620) | 399.740(77.757) | 409.415(81.766) | 400.175(77.379) | 2333.415(211.235) |  | 0.901(0.006) | 0.960
0.04 | 0.4 | 0.4 | p1 | 1409.665(115.474) | 283.315(46.822) | 280.755(51.205) | 285.885(54.894) | 279.535(43.948) | 280.175(50.207) |  | 0.907(0.008) | 0.945
|  |  | p2 | 1592.525(158.412) | 244.215(64.961) | 256.545(72.169) | 249.130(68.352) | 247.605(62.649) | 595.030(93.138) |  | 0.908(0.008) | 0.955
|  | 0.3 | p1 | 1899.810(140.624) | 374.900(63.191) | 378.050(61.751) | 391.030(71.765) | 378.685(67.319) | 377.145(67.664) |  | 0.903(0.007) | 0.920
|  |  | p2 | 2133.280(164.460) | 268.430(52.070) | 282.930(54.898) | 275.985(51.456) | 272.100(49.524) | 1033.835(127.392) |  | 0.905(0.007) | 0.960
|  | 0.2 | p1 | 3953.785(267.599) | 774.375(113.544) | 786.030(122.170) | 810.615(123.860) | 798.300(118.161) | 784.465(124.124) |  | 0.902(0.006) | 0.935
|  |  | p2 | 3953.585(251.092) | 411.455(69.285) | 414.440(75.603) | 410.125(70.166) | 414.435(72.966) | 2303.130(198.681) |  | 0.901(0.006) | 0.940
A | 0.06 | 0.4 | p1 | 779.195(60.423) | 157.370(27.523) | 153.775(26.695) | 153.010(24.152) | 155.595(27.593) | 159.445(30.977) |  | 0.897(0.006) | 0.950
|  |  | p2 | 839.985(77.668) | 117.980(21.386) | 115.330(18.882) | 116.465(21.355) | 116.295(20.234) | 373.915(59.929) |  | 0.901(0.006) | 0.885
|  | 0.3 | p1 | 1181.445(90.720) | 238.465(43.166) | 234.650(37.981) | 230.065(40.849) | 234.105(46.879) | 244.160(40.479) |  | 0.893(0.006) | 0.960
|  |  | p2 | 1231.420(97.393) | 146.555(26.299) | 146.260(25.736) | 144.695(26.000) | 147.710(24.105) | 646.200(82.551) |  | 0.895(0.006) | 0.920
|  | 0.2 | p1 | 2458.665(151.224) | 492.260(61.309) | 498.565(62.355) | 481.495(64.580) | 488.935(63.845) | 497.410(62.629) |  | 0.886(0.004) | 0.965
|  |  | p2 | 2515.910(144.705) | 259.490(47.387) | 261.895(46.299) | 261.305(47.419) | 260.930(45.253) | 1472.290(111.568) |  | 0.888(0.005) | 0.950
| 0.05 | 0.4 | p1 | 872.290(58.272) | 172.975(25.499) | 172.095(25.739) | 179.750(26.169) | 176.220(27.657) | 171.250(26.573) |  | 0.901(0.005) | 0.930
|  |  | p2 | 972.265(83.208) | 153.410(32.018) | 145.595(30.334) | 146.720(30.308) | 148.950(33.610) | 377.590(54.576) |  | 0.904(0.006) | 0.945
|  | 0.3 | p1 | 1203.885(83.544) | 238.290(39.105) | 237.205(34.790) | 244.670(38.999) | 240.440(41.670) | 243.280(38.201) |  | 0.893(0.005) | 0.925
|  |  | p2 | 1328.320(89.670) | 166.100(24.505) | 168.325(24.836) | 170.980(28.018) | 168.410(24.977) | 654.505(72.764) |  | 0.898(0.005) | 0.930
|  | 0.2 | p1 | 2447.690(140.464) | 489.660(63.909) | 482.050(64.855) | 491.925(65.949) | 492.500(61.799) | 491.555(64.902) |  | 0.885(0.004) | 0.965
|  |  | p2 | 2551.055(149.139) | 262.870(44.496) | 269.955(44.021) | 272.190(47.966) | 271.170(40.905) | 1474.870(115.197) |  | 0.890(0.005) | 0.935
0.04 | 0.4 | 0.4 | p1 | 1286.435(116.231) | 259.770(52.710) | 253.700(48.164) | 257.625(52.416) | 254.005(47.705) | 261.335(52.356) |  | 0.900(0.006) | 0.985
|  |  | p2 | 1379.530(125.618) | 248.665(58.304) | 255.315(63.360) | 245.350(63.381) | 249.430(58.984) | 380.770(50.671) |  | 0.899(0.007) | 0.995
|  | 0.3 | p1 | 1420.685(84.562) | 284.455(36.414) | 284.250(39.525) | 282.260(35.245) | 284.255(35.636) | 285.465(39.358) |  | 0.896(0.005) | 0.955
|  |  | p2 | 1667.640(132.199) | 253.015(53.753) | 259.550(54.184) | 257.760(49.870) | 254.880(56.937) | 642.435(78.943) |  | 0.896(0.006) | 0.975
|  | 0.2 | p1 | 2463.705(150.922) | 495.770(68.153) | 495.555(66.413) | 488.955(64.063) | 492.855(65.264) | 490.570(65.247) |  | 0.886(0.004) | 0.930
|  |  | p2 | 2657.900(139.943) | 295.455(32.093) | 300.855(38.706) | 299.955(32.010) | 301.330(32.319) | 1460.305(115.925) |  | 0.891(0.004) | 0.950
∗ Standard deviations are in parentheses. R and A stand for Random and
Adaptive samplings, respectively.
Table 2: Results of absolute bias of parameter estimate with random selection strategy and homogenuous covariates in each data center. | $d_{2}$ | $d_{1}$ |  |  | Proposed | Avearge | Site 1 | Site 2 | Site 3 | Site 4 | Site 5
---|---|---|---|---|---|---|---|---|---|---|---
| 0.06 | 0.4 | p1 | $\beta_{1}$ | 0.141(0.104) | 0.114(0.087) | 0.250(0.180) | 0.262(0.186) | 0.253(0.179) | 0.238(0.189) | 0.257(0.174)
|  |  |  | $\beta_{2}$ | 0.100(0.073) | 0.089(0.067) | 0.216(0.158) | 0.200(0.130) | 0.195(0.130) | 0.196(0.134) | 0.215(0.154)
|  |  | p2 | $\beta_{1}$ | 0.129(0.100) | 0.147(0.109) | 0.291(0.231) | 0.306(0.225) | 0.305(0.222) | 0.297(0.220) | 0.145(0.112)
|  |  |  | $\beta_{2}$ | 0.098(0.065) | 0.110(0.079) | 0.255(0.188) | 0.256(0.177) | 0.254(0.200) | 0.251(0.193) | 0.112(0.086)
|  | 0.3 | p1 | $\beta_{1}$ | 0.105(0.081) | 0.092(0.073) | 0.204(0.151) | 0.214(0.157) | 0.199(0.157) | 0.181(0.147) | 0.222(0.159)
|  |  |  | $\beta_{2}$ | 0.076(0.056) | 0.070(0.055) | 0.156(0.112) | 0.144(0.109) | 0.141(0.101) | 0.153(0.117) | 0.175(0.132)
|  |  | p2 | $\beta_{1}$ | 0.101(0.076) | 0.109(0.084) | 0.251(0.203) | 0.249(0.186) | 0.240(0.180) | 0.261(0.200) | 0.119(0.087)
|  |  |  | $\beta_{2}$ | 0.072(0.054) | 0.087(0.062) | 0.208(0.148) | 0.205(0.132) | 0.205(0.159) | 0.207(0.152) | 0.088(0.066)
|  | 0.2 | p1 | $\beta_{1}$ | 0.063(0.047) | 0.061(0.046) | 0.132(0.101) | 0.138(0.100) | 0.138(0.105) | 0.140(0.101) | 0.140(0.112)
|  |  |  | $\beta_{2}$ | 0.047(0.034) | 0.046(0.035) | 0.089(0.075) | 0.096(0.074) | 0.101(0.071) | 0.100(0.078) | 0.109(0.084)
|  |  | p2 | $\beta_{1}$ | 0.061(0.045) | 0.077(0.055) | 0.197(0.151) | 0.179(0.134) | 0.174(0.134) | 0.201(0.164) | 0.078(0.060)
|  |  |  | $\beta_{2}$ | 0.045(0.036) | 0.060(0.046) | 0.155(0.115) | 0.145(0.106) | 0.152(0.106) | 0.151(0.115) | 0.059(0.046)
| 0.05 | 0.4 | p1 | $\beta_{1}$ | 0.136(0.103) | 0.113(0.086) | 0.252(0.196) | 0.218(0.171) | 0.227(0.188) | 0.236(0.193) | 0.223(0.177)
|  |  |  | $\beta_{2}$ | 0.097(0.070) | 0.092(0.066) | 0.199(0.148) | 0.176(0.126) | 0.181(0.143) | 0.192(0.146) | 0.167(0.125)
|  |  | p2 | $\beta_{1}$ | 0.124(0.089) | 0.144(0.098) | 0.280(0.222) | 0.293(0.221) | 0.304(0.223) | 0.277(0.218) | 0.154(0.126)
|  |  |  | $\beta_{2}$ | 0.084(0.063) | 0.096(0.073) | 0.242(0.183) | 0.236(0.186) | 0.220(0.149) | 0.228(0.169) | 0.112(0.087)
|  | 0.3 | p1 | $\beta_{1}$ | 0.101(0.071) | 0.088(0.066) | 0.212(0.152) | 0.188(0.139) | 0.205(0.151) | 0.201(0.150) | 0.198(0.155)
|  |  |  | $\beta_{2}$ | 0.074(0.054) | 0.070(0.052) | 0.156(0.116) | 0.135(0.108) | 0.146(0.107) | 0.152(0.120) | 0.145(0.107)
|  |  | p2 | $\beta_{1}$ | 0.093(0.071) | 0.100(0.074) | 0.244(0.174) | 0.220(0.162) | 0.249(0.175) | 0.229(0.165) | 0.119(0.094)
|  |  |  | $\beta_{2}$ | 0.068(0.051) | 0.080(0.057) | 0.204(0.152) | 0.199(0.150) | 0.198(0.126) | 0.188(0.137) | 0.087(0.065)
|  | 0.2 | p1 | $\beta_{1}$ | 0.067(0.053) | 0.061(0.049) | 0.142(0.107) | 0.117(0.091) | 0.142(0.119) | 0.143(0.104) | 0.143(0.108)
|  |  |  | $\beta_{2}$ | 0.046(0.035) | 0.044(0.034) | 0.105(0.074) | 0.092(0.076) | 0.094(0.080) | 0.110(0.074) | 0.102(0.079)
|  |  | p2 | $\beta_{1}$ | 0.057(0.050) | 0.073(0.052) | 0.191(0.137) | 0.174(0.136) | 0.184(0.137) | 0.171(0.138) | 0.077(0.063)
|  |  |  | $\beta_{2}$ | 0.045(0.032) | 0.056(0.044) | 0.131(0.100) | 0.145(0.104) | 0.159(0.110) | 0.130(0.109) | 0.059(0.043)
| 0.04 | 0.4 | p1 | $\beta_{1}$ | 0.137(0.097) | 0.128(0.088) | 0.220(0.176) | 0.224(0.171) | 0.234(0.198) | 0.225(0.183) | 0.212(0.174)
|  |  |  | $\beta_{2}$ | 0.090(0.069) | 0.085(0.067) | 0.175(0.137) | 0.185(0.146) | 0.179(0.134) | 0.176(0.138) | 0.197(0.140)
|  |  | p2 | $\beta_{1}$ | 0.115(0.090) | 0.150(0.115) | 0.270(0.239) | 0.301(0.233) | 0.292(0.249) | 0.306(0.227) | 0.144(0.101)
|  |  |  | $\beta_{2}$ | 0.083(0.065) | 0.099(0.079) | 0.202(0.156) | 0.208(0.178) | 0.205(0.176) | 0.212(0.194) | 0.108(0.083)
|  | 0.3 | p1 | $\beta_{1}$ | 0.109(0.073) | 0.093(0.064) | 0.180(0.127) | 0.186(0.136) | 0.200(0.150) | 0.178(0.141) | 0.185(0.142)
|  |  |  | $\beta_{2}$ | 0.075(0.053) | 0.069(0.050) | 0.143(0.113) | 0.155(0.119) | 0.147(0.114) | 0.158(0.117) | 0.170(0.118)
|  |  | p2 | $\beta_{1}$ | 0.092(0.073) | 0.110(0.088) | 0.216(0.176) | 0.247(0.182) | 0.238(0.190) | 0.249(0.172) | 0.118(0.080)
|  |  |  | $\beta_{2}$ | 0.068(0.052) | 0.077(0.062) | 0.174(0.139) | 0.180(0.147) | 0.180(0.148) | 0.172(0.144) | 0.086(0.064)
|  | 0.2 | p1 | $\beta_{1}$ | 0.064(0.048) | 0.061(0.046) | 0.136(0.093) | 0.140(0.112) | 0.146(0.107) | 0.134(0.098) | 0.145(0.103)
|  |  |  | $\beta_{2}$ | 0.050(0.034) | 0.049(0.034) | 0.109(0.079) | 0.103(0.076) | 0.105(0.077) | 0.107(0.076) | 0.110(0.085)
|  |  | p2 | $\beta_{1}$ | 0.061(0.046) | 0.077(0.056) | 0.160(0.118) | 0.187(0.136) | 0.182(0.135) | 0.172(0.128) | 0.076(0.062)
|  |  |  | $\beta_{2}$ | 0.051(0.036) | 0.060(0.045) | 0.132(0.093) | 0.137(0.112) | 0.138(0.115) | 0.138(0.104) | 0.063(0.044)
∗ Standard deviations are in parentheses. |
Table 3: Results of absolute bias of parameter estimate with adaptive selection strategy and homogenuous covariates in each data center. | $d_{2}$ | $d_{1}$ |  |  | Proposed | Avearge | Site 1 | Site 2 | Site 3 | Site 4 | Site 5
---|---|---|---|---|---|---|---|---|---|---|---
| 0.06 | 0.4 | p1 | $\beta_{1}$ | 0.140(0.098) | 0.114(0.086) | 0.248(0.177) | 0.232(0.170) | 0.225(0.165) | 0.238(0.172) | 0.264(0.191)
|  |  |  | $\beta_{2}$ | 0.075(0.057) | 0.067(0.051) | 0.167(0.113) | 0.139(0.107) | 0.149(0.114) | 0.139(0.101) | 0.161(0.119)
|  |  | p2 | $\beta_{1}$ | 0.149(0.114) | 0.182(0.127) | 0.341(0.248) | 0.308(0.242) | 0.322(0.244) | 0.346(0.263) | 0.165(0.120)
|  |  |  | $\beta_{2}$ | 0.097(0.071) | 0.113(0.082) | 0.220(0.164) | 0.212(0.163) | 0.209(0.167) | 0.211(0.159) | 0.109(0.081)
|  | 0.3 | p1 | $\beta_{1}$ | 0.097(0.072) | 0.089(0.062) | 0.190(0.148) | 0.179(0.129) | 0.195(0.148) | 0.208(0.161) | 0.192(0.141)
|  |  |  | $\beta_{2}$ | 0.053(0.043) | 0.049(0.040) | 0.124(0.091) | 0.114(0.084) | 0.118(0.089) | 0.119(0.090) | 0.122(0.091)
|  |  | p2 | $\beta_{1}$ | 0.107(0.070) | 0.122(0.085) | 0.261(0.196) | 0.258(0.188) | 0.247(0.184) | 0.259(0.191) | 0.122(0.100)
|  |  |  | $\beta_{2}$ | 0.065(0.050) | 0.079(0.056) | 0.174(0.139) | 0.161(0.127) | 0.169(0.130) | 0.160(0.123) | 0.078(0.062)
|  | 0.2 | p1 | $\beta_{1}$ | 0.065(0.049) | 0.064(0.046) | 0.126(0.091) | 0.133(0.100) | 0.136(0.108) | 0.133(0.104) | 0.129(0.097)
|  |  |  | $\beta_{2}$ | 0.039(0.028) | 0.037(0.029) | 0.080(0.057) | 0.083(0.061) | 0.084(0.067) | 0.084(0.060) | 0.085(0.068)
|  |  | p2 | $\beta_{1}$ | 0.061(0.043) | 0.077(0.057) | 0.197(0.141) | 0.178(0.138) | 0.193(0.137) | 0.188(0.145) | 0.075(0.052)
|  |  |  | $\beta_{2}$ | 0.045(0.032) | 0.055(0.038) | 0.128(0.097) | 0.124(0.098) | 0.120(0.087) | 0.116(0.086) | 0.056(0.038)
| 0.05 | 0.4 | p1 | $\beta_{1}$ | 0.127(0.099) | 0.118(0.090) | 0.220(0.168) | 0.218(0.159) | 0.266(0.182) | 0.239(0.193) | 0.202(0.149)
|  |  |  | $\beta_{2}$ | 0.078(0.060) | 0.075(0.056) | 0.139(0.107) | 0.144(0.118) | 0.171(0.119) | 0.152(0.119) | 0.142(0.110)
|  |  | p2 | $\beta_{1}$ | 0.144(0.109) | 0.186(0.129) | 0.308(0.232) | 0.327(0.258) | 0.349(0.255) | 0.342(0.292) | 0.155(0.114)
|  |  |  | $\beta_{2}$ | 0.075(0.057) | 0.091(0.070) | 0.193(0.153) | 0.174(0.145) | 0.184(0.148) | 0.190(0.149) | 0.094(0.075)
|  | 0.3 | p1 | $\beta_{1}$ | 0.099(0.070) | 0.085(0.064) | 0.190(0.136) | 0.178(0.124) | 0.197(0.141) | 0.203(0.148) | 0.184(0.149)
|  |  |  | $\beta_{2}$ | 0.062(0.044) | 0.057(0.040) | 0.125(0.093) | 0.121(0.090) | 0.128(0.096) | 0.126(0.094) | 0.123(0.096)
|  |  | p2 | $\beta_{1}$ | 0.102(0.076) | 0.127(0.088) | 0.239(0.173) | 0.262(0.192) | 0.276(0.221) | 0.252(0.182) | 0.114(0.088)
|  |  |  | $\beta_{2}$ | 0.062(0.045) | 0.073(0.058) | 0.154(0.118) | 0.157(0.115) | 0.161(0.138) | 0.157(0.117) | 0.073(0.051)
|  | 0.2 | p1 | $\beta_{1}$ | 0.060(0.043) | 0.058(0.044) | 0.129(0.101) | 0.136(0.102) | 0.136(0.105) | 0.125(0.094) | 0.132(0.103)
|  |  |  | $\beta_{2}$ | 0.038(0.028) | 0.038(0.028) | 0.085(0.071) | 0.087(0.066) | 0.087(0.066) | 0.085(0.059) | 0.087(0.069)
|  |  | p2 | $\beta_{1}$ | 0.070(0.050) | 0.084(0.064) | 0.199(0.136) | 0.193(0.135) | 0.205(0.140) | 0.171(0.135) | 0.080(0.055)
|  |  |  | $\beta_{2}$ | 0.044(0.029) | 0.053(0.040) | 0.125(0.092) | 0.110(0.083) | 0.126(0.095) | 0.115(0.087) | 0.051(0.039)
| 0.04 | 0.4 | p1 | $\beta_{1}$ | 0.111(0.087) | 0.122(0.093) | 0.213(0.180) | 0.217(0.187) | 0.229(0.176) | 0.236(0.181) | 0.201(0.172)
|  |  |  | $\beta_{2}$ | 0.065(0.053) | 0.070(0.055) | 0.134(0.116) | 0.139(0.108) | 0.123(0.104) | 0.134(0.109) | 0.123(0.101)
|  |  | p2 | $\beta_{1}$ | 0.105(0.077) | 0.134(0.094) | 0.237(0.203) | 0.237(0.231) | 0.274(0.241) | 0.234(0.219) | 0.159(0.119)
|  |  |  | $\beta_{2}$ | 0.063(0.048) | 0.080(0.059) | 0.149(0.125) | 0.134(0.129) | 0.159(0.156) | 0.159(0.142) | 0.092(0.065)
|  | 0.3 | p1 | $\beta_{1}$ | 0.096(0.073) | 0.091(0.068) | 0.190(0.135) | 0.186(0.142) | 0.168(0.133) | 0.186(0.128) | 0.189(0.145)
|  |  |  | $\beta_{2}$ | 0.057(0.044) | 0.055(0.041) | 0.111(0.092) | 0.112(0.090) | 0.105(0.085) | 0.117(0.086) | 0.118(0.088)
|  |  | p2 | $\beta_{1}$ | 0.092(0.068) | 0.115(0.082) | 0.198(0.164) | 0.222(0.197) | 0.218(0.171) | 0.226(0.177) | 0.120(0.099)
|  |  |  | $\beta_{2}$ | 0.056(0.043) | 0.070(0.052) | 0.138(0.107) | 0.134(0.114) | 0.134(0.116) | 0.150(0.115) | 0.080(0.056)
|  | 0.2 | p1 | $\beta_{1}$ | 0.066(0.052) | 0.062(0.051) | 0.143(0.108) | 0.144(0.105) | 0.134(0.103) | 0.145(0.098) | 0.133(0.105)
|  |  |  | $\beta_{2}$ | 0.041(0.031) | 0.040(0.029) | 0.085(0.065) | 0.092(0.068) | 0.083(0.063) | 0.085(0.061) | 0.089(0.066)
|  |  | p2 | $\beta_{1}$ | 0.061(0.044) | 0.074(0.057) | 0.158(0.122) | 0.178(0.136) | 0.177(0.137) | 0.170(0.120) | 0.072(0.057)
|  |  |  | $\beta_{2}$ | 0.042(0.031) | 0.053(0.035) | 0.113(0.083) | 0.114(0.083) | 0.121(0.086) | 0.112(0.085) | 0.057(0.043)
∗ Standard deviations are in parentheses. |
Table 4: Simulation results about stopping times, AUC and coverage frequency with heterogeneity covariates, $d_{1}=0.2$ and $d_{2}=0.05$. |  |  |  | Stopping time |  |  |
---|---|---|---|---|---|---|---
|  |  |  | $N$ | $N_{1}$ | $N_{2}$ | $N_{3}$ | $N_{4}$ | $N_{5}$ |  | AUC | CF
R |  |  | s1 | 3973.980(276.329) | 792.765(118.450) | 798.200(117.615) | 781.180(113.771) | 717.270(112.102) | 884.565(141.991) |  | 0.902(0.006) | 0.950
|  |  | s2 | 4218.450(282.242) | 420.235(83.695) | 413.230(82.100) | 409.250(82.803) | 370.620(69.013) | 2605.115(250.404) |  | 0.903(0.007) | 0.955
|  |  | s3 | 4215.865(261.601) | 796.765(110.019) | 867.045(141.630) | 799.680(120.935) | 771.295(103.102) | 981.080(139.137) |  | 0.916(0.005) | 0.940
|  |  | s4 | 4579.100(284.174) | 405.350(86.785) | 453.955(87.486) | 409.570(82.998) | 392.575(74.388) | 2917.650(231.654) |  | 0.916(0.006) | 0.965
A |  |  | s1 | 2479.915(132.460) | 503.405(63.221) | 488.560(67.064) | 509.740(59.487) | 483.295(57.845) | 494.915(64.279) |  | 0.885(0.004) | 0.960
|  |  | s2 | 2574.330(160.415) | 267.140(46.490) | 263.540(44.746) | 274.190(45.693) | 262.870(41.629) | 1506.590(121.938) |  | 0.888(0.005) | 0.905
|  |  | s3 | 2446.545(149.305) | 494.795(64.300) | 476.625(63.512) | 500.510(64.792) | 482.090(58.726) | 492.525(61.964) |  | 0.889(0.004) | 0.935
|  |  | s4 | 2513.765(149.527) | 268.940(48.173) | 260.395(44.636) | 267.365(42.222) | 251.965(45.746) | 1465.100(113.487) |  | 0.893(0.005) | 0.940
∗ Standard deviations are in parentheses. R and A stand for Random and Adaptive samplings, respectively. |
Table 5: Results of absolute bias of parameter estimate with with heterogeneity covariates, $d_{1}=0.2$ and $d_{2}=0.05$. |  |  |  |  | Proposed | Avearge | Site 1 | Site 2 | Site 3 | Site 4 | Site 5
---|---|---|---|---|---|---|---|---|---|---|---
R |  |  | s1 | $\beta_{1}$ | 0.065(0.047) | 0.061(0.047) | 0.137(0.104) | 0.138(0.098) | 0.127(0.097) | 0.138(0.103) | 0.146(0.108)
|  |  |  | $\beta_{2}$ | 0.048(0.037) | 0.047(0.037) | 0.099(0.069) | 0.106(0.083) | 0.104(0.082) | 0.105(0.077) | 0.111(0.081)
|  |  | s2 | $\beta_{1}$ | 0.061(0.051) | 0.073(0.054) | 0.195(0.136) | 0.191(0.142) | 0.196(0.138) | 0.174(0.133) | 0.082(0.063)
|  |  |  | $\beta_{2}$ | 0.047(0.034) | 0.057(0.041) | 0.148(0.113) | 0.140(0.103) | 0.131(0.097) | 0.153(0.121) | 0.067(0.049)
|  |  | s3 | $\beta_{1}$ | 0.062(0.048) | 0.058(0.043) | 0.133(0.102) | 0.145(0.112) | 0.138(0.095) | 0.139(0.098) | 0.128(0.095)
|  |  |  | $\beta_{2}$ | 0.046(0.034) | 0.045(0.033) | 0.096(0.073) | 0.100(0.082) | 0.099(0.078) | 0.095(0.073) | 0.108(0.076)
|  |  | s4 | $\beta_{1}$ | 0.058(0.046) | 0.069(0.053) | 0.204(0.137) | 0.180(0.132) | 0.177(0.150) | 0.179(0.131) | 0.073(0.053)
|  |  |  | $\beta_{2}$ | 0.045(0.033) | 0.058(0.043) | 0.162(0.126) | 0.146(0.103) | 0.149(0.107) | 0.154(0.108) | 0.054(0.039)
A |  |  | s1 | $\beta_{1}$ | 0.058(0.043) | 0.054(0.042) | 0.127(0.105) | 0.135(0.112) | 0.125(0.089) | 0.125(0.091) | 0.131(0.109)
|  |  |  | $\beta_{2}$ | 0.040(0.029) | 0.039(0.029) | 0.087(0.062) | 0.085(0.069) | 0.081(0.061) | 0.078(0.058) | 0.081(0.062)
|  |  | s2 | $\beta_{1}$ | 0.067(0.054) | 0.082(0.065) | 0.198(0.134) | 0.189(0.136) | 0.182(0.146) | 0.181(0.133) | 0.078(0.059)
|  |  |  | $\beta_{2}$ | 0.045(0.034) | 0.054(0.040) | 0.119(0.091) | 0.122(0.089) | 0.124(0.091) | 0.115(0.087) | 0.058(0.041)
|  |  | s3 | $\beta_{1}$ | 0.062(0.051) | 0.060(0.049) | 0.133(0.102) | 0.139(0.098) | 0.132(0.107) | 0.122(0.092) | 0.129(0.096)
|  |  |  | $\beta_{2}$ | 0.036(0.031) | 0.036(0.030) | 0.082(0.068) | 0.072(0.063) | 0.083(0.062) | 0.084(0.059) | 0.077(0.061)
|  |  | s4 | $\beta_{1}$ | 0.065(0.049) | 0.080(0.057) | 0.216(0.145) | 0.191(0.132) | 0.175(0.126) | 0.193(0.138) | 0.074(0.057)
|  |  |  | $\beta_{2}$ | 0.041(0.032) | 0.051(0.035) | 0.117(0.093) | 0.116(0.091) | 0.116(0.088) | 0.117(0.091) | 0.051(0.041)
∗ Standard deviations are in parentheses. R and A stand for Random and
Adaptive samplings, respectively.

###  3.2 COVID-19 data

We use a COVID-19 data set collected in Mexico by Mexican health authorities
to demonstrate our method. Please note that this data set may be updated all
any time. We downloaded this dataset in April 2021, and it contained 6659184
registers of the COVID-19 patients at that time. The data were obtained from
an epidemiological study of suspected cases of the viral respiratory disease
at that time. The data were identified in the medical units of the 32 health
sectors.Outpatients and inpatients were identified based on the clinical
diagnosis at admission. The data set includes some personal and health
information of the study subjects, such as gender, age, and medical history,
including pneumonia, diabetes, chronic obstructive pulmonary(COPD), asthma,
immunosuppression (Immu), hypertension, cardiovascular disease (Card), chronic
renal failure (CRF), other diseases diagnoses, as well as other factors such
as obesity smoking status, exposure to other cases of SARS CoV-2(EOC) and
COVID-19 status (positive or negative). We treat COVID-19 status as the
response variable and the others as covariates in our analysis. Other
variables, such as ID, birthplace of subject, sector catalog, type of care,
and date of onset of symptoms, are not included in our analysis. All variables
in this dataset except age, are binary, where 1 represents “Y”, and 0
represents “N”, and for gender, 1 indicates females, and 0 indicates males.

In this numerical study, we aim to find the association between COVID-19
status (positive or negative) and the other variables. In addition, some
variables are of particular interest. For example, one research question of
interest was as follows: are subjects with diabetes or obesity more likely to
be affected with COVID-19 compared with other subjects? To assess this, we
adopt a logistic regression model for our analysis.

After omitting subjects with missing values, 5816861 subjects remains in the
dataset, and the range of sample sizes of 32 different sites are from 21746 to
2396133. Because of the huge differences in sample size among the different
sites, we use 2 sampling strategies: (C1) equal proportion among all sites;
that is, each site contributes 1/32, of the samples, and (C2) 1/100 for sites
4, 6, 7, 18, and equal proportion, 6/175, for the rest. The reason we use
strategy (C2) is that the total data sizes of sites 4, 6, 7, and 18 are less
than 30000. By comparing the results based on these two sampling strategies,
we can assess the strength of the proposed method.

When maintaining the confidentiality of personal health information and
minimizing the data communication burden are essential, each health sector
uses its own data to build a logistics model. As mentioned before, we
separately train a logistic model for each health sector, report the stopping
time and parameter estimates, and use the AUC to control the prediction
performance of each model. We then integrate all the results as described
previously.

For comparison purposes (the baseline model), we simply use all data to
estimate parameters and then use the estimates and AUC value as references. We
consider three parameter combinations to illustrate our method as follows:

  * All: all parameter variables are equally important;

  * Part 1: Only five variables, pneumonia, COPD, asthma, CRF and EOC, are of interest;

  * Part 2: The 10-variable case. Suppose that variables gender, age, diabetes, asthma, hypertension, other diseases diagnoses, cardiovascular, obesity, CRF and smoking status are of interest.

Tables 6 and 7 report the stopping times and AUC obtained by applying the
proposed method to the COVID-19 data. Table 6 shows that the proposed method
yields larger AUCs than the baseline model. The models with 3 different
variable combinations have similar AUCs because we set $d_{2}=0.05$ for the
AUC for the 3 different variable sets. Models with a random selection strategy
usually require more than 100000 samples before the stopping criteria are
fulfilled. Thus, sites with smaller data sizes, such as sites 4, 6, 7 and 18,
could not achieve the preset precision of parameter estimation with their own
data. Models using an adaptive selection method usually reach the stopping
criteria with much smaller sample sizes.

When the number of variables of interest becomes large, the stopping time also
increases. Hence, if all parameters are of interest, then the stopping times,
the required sample sizes, are much larger than those of Part 1 and Part 2.
From the definition of the stopping criterion, it is clear that when $d_{1}$
is small, the proposed sequential methods would demand more samples. Table 6
shows that designs C1 (equal proportions) and C2 (different proportions) have
similar total stopping times. However, Table 7 shows that under C2, Sites 4,
6, 7 and 18 use much smaller sample sizes than under C1, because C2 provides a
smaller proportion for these sites. One of the advantages of early stopping
for these cases is better control the samples used for modeling even for sites
with limited samples; especially when adopting an adaptive sampling method, we
can even “guarantee” the quality of samples.

Table 8 reports parameter estimates for COVID-19 data with equal proportions
C1. We only list estimates of parameters of interest; hence, for Part 1, only
estimates for the variables pneumonia, COPD, asthma, CRF, and EOC are
reported. These tables also show that the estimates of the models focusing
only on the parameters of interest are consistent with those of the baseline
model.

Table 9 shows the results for the different proportions C2, which lead to a
similar conclusion with the equal proportion C1. Hence, we do not go into the
details here. Based on Tables 8 and 9, we find that adaptive selection yields
parameter estimates consistent with those of the models using random
selection. However, Table 6 shows that the models with adaptive sampling use
much smaller sample sizes. All the numerical results based on this COVID-19
data analysis confirm that the proposed distributed sequential federated
learning method performs well, and has good potential to be extended for other
applications.

We briefly summarize the results of our analysis using the COVID-19 data
below. However, please note that this was not a well-designed study, and we
mainly use it to illustrate our method. Comparing the results of Part 2 with
that of the baseline model, we found that the estimates of age, diabetes and
obesity in both models are significantly greater than 0, which may indicate
that elderly subjects with diabetes or obesity are more likely to be infected.
In addition, based on this data analysis, we find that the parameter estimates
for gender, Card and smoking status are significantly less than 0, which may
suggest that female subjects with cardiovascular disease and a history of
smoking have smaller probability of being infected. The results of our
analysis also shows that the parameter estimates for pneumonia and EOC are
statistically significantly greater than 0, and those for asthma and CRF are
significantly less than 0. These results suggest that subjects with pneumonia
and EOC and without asthma and CRF are more likely to be infected. Similar
results for age, diabetes, obesity, gender, smoking status and asthma on
COVID-19 inflection were also found in the studies of Hernández-Garduño
(2020); Rashedi et al. (2020); Louis et al. (2020); Liu et al. (2020); Memon &
Biswas (2022).

###### Remark 3.1

The proposed method determines “how many data” and “which subdata-sets”
selected from heterogeneous sources of data when a stopping criterion is
satisfied. During sequential procedures, in addition to requiring that the
estimation of parameters of interest achieves the prespecified level of
accuracy, the method guarantees that the model achieves the prespecified
precision, such as prediction for the classification model.

###### Remark 3.2

We construct individual sequential sampling strategies considering varying
data sizes of different data centers, to avoid sequential procedures with
small data sizes that can not be stopped even when all data run out.

Table 6: Stopping times and AUC for Covid data with $d_{2}=0.05$. |  |  | Stopping time |  | AUC |  |
---|---|---|---|---|---|---|---
|  | $d_{1}$ | All parameters | Part 1 | Part 2 |  | All parameters | Part 1 | Part 2 |  | Baseline
R | C1 | 0.4 | 121680 | 51780 | 55580 |  | 0.627 | 0.633 | 0.631 |  | 0.598
|  | 0.3 | 199380 | 84280 | 91580 |  | 0.625 | 0.629 | 0.627 |  | 0.598
|  | 0.2 | 432780 | 170680 | 183380 |  | 0.622 | 0.625 | 0.625 |  | 0.598
| C2 | 0.4 | 123580 | 51580 | 56380 |  | 0.628 | 0.635 | 0.632 |  | 0.598
|  | 0.3 | 203280 | 84580 | 92680 |  | 0.626 | 0.632 | 0.629 |  | 0.598
|  | 0.2 | 434780 | 172880 | 185380 |  | 0.622 | 0.626 | 0.627 |  | 0.598
A | C1 | 0.4 | 16940 | 16250 | 16390 |  | 0.673 | 0.673 | 0.672 |  | 0.598
|  | 0.3 | 18610 | 16480 | 16990 |  | 0.668 | 0.672 | 0.672 |  | 0.598
|  | 0.2 | 27270 | 18020 | 20480 |  | 0.662 | 0.670 | 0.666 |  | 0.598
| C2 | 0.4 | 17080 | 16280 | 16450 |  | 0.673 | 0.672 | 0.672 |  | 0.598
|  | 0.3 | 18750 | 16550 | 17100 |  | 0.670 | 0.672 | 0.672 |  | 0.598
|  | 0.2 | 26920 | 18230 | 20720 |  | 0.663 | 0.670 | 0.668 |  | 0.598
∗ R and A stand for Random and Adaptive samplings, respectively. |
Table 7: Stopping times of centers 4, 6, 7, and 18 with data size less than 30000 for Covid data. |  |  | All parameters |  | Part 1 |  | Part 2
---|---|---|---|---|---|---|---
|  | $d_{1}$ | Site 4 | Site 6 | Site 7 | Site 18 |  | Site 4 | Site 6 | Site 7 | Site 18 |  | Site 4 | Site 6 | Site 7 | Site 18
R | C1 | 0.4 | 3915 | 2515 | 4415 | 1615 |  | 2015 | 915 | 2315 | 515 |  | 2515 | 1115 | 1515 | 715
|  | 0.3 | 5815 | 4615 | 6715 | 3315 |  | 3715 | 1615 | 3815 | 1015 |  | 3815 | 1815 | 2615 | 1215
|  | 0.2 | 14715 | 12215 | 12215 | 7615 |  | 5215 | 3215 | 6815 | 2215 |  | 8415 | 3915 | 5715 | 2715
| C2 | 0.4 | 1315 | 1115 | 1715 | 715 |  | 615 | 515 | 715 | 515 |  | 1315 | 615 | 715 | 515
|  | 0.3 | 2515 | 1815 | 2615 | 1015 |  | 915 | 615 | 1115 | 515 |  | 1915 | 815 | 915 | 515
|  | 0.2 | 4215 | 3615 | 5115 | 2015 |  | 2515 | 1215 | 2615 | 715 |  | 2915 | 1415 | 1915 | 1015
A | C1 | 0.4 | 525 | 445 | 385 | 465 |  | 515 | 445 | 385 | 465 |  | 515 | 445 | 385 | 465
|  | 0.3 | 605 | 455 | 555 | 465 |  | 515 | 445 | 385 | 465 |  | 545 | 445 | 385 | 465
|  | 0.2 | 1025 | 805 | 1025 | 705 |  | 585 | 445 | 455 | 465 |  | 735 | 505 | 595 | 495
| C2 | 0.4 | 515 | 445 | 385 | 465 |  | 515 | 445 | 385 | 465 |  | 515 | 445 | 385 | 465
|  | 0.3 | 515 | 445 | 385 | 465 |  | 515 | 445 | 385 | 465 |  | 515 | 445 | 385 | 465
|  | 0.2 | 545 | 445 | 425 | 465 |  | 515 | 445 | 385 | 465 |  | 515 | 445 | 385 | 465
∗ R and A stand for Random and Adaptive samplings, respectively. |  |  |  |  |  |
Table 8: Parameter estimation for CoVid data with $d_{2}=0.05$ and equal proportion C1. |  | $d_{1}$ |  | Gender | Pneumonia | Age | Diabetes | COPD | Asthma | Immu | Hypertension | Other disease | Card | Obesity | CRF | Smoke | EOC
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
R | All | 0.4 | Est. | -0.162 | 1.085 | 0.013 | 0.155 | -0.252 | -0.160 | -0.374 | 0.026 | -0.120 | -0.337 | 0.234 | -0.301 | -0.095 | 0.275
|  |  | Sd | 0.013 | 0.027 | 0.000 | 0.023 | 0.080 | 0.043 | 0.083 | 0.021 | 0.055 | 0.058 | 0.019 | 0.060 | 0.028 | 0.013
|  | 0.3 | Est. | -0.169 | 1.097 | 0.012 | 0.159 | -0.294 | -0.150 | -0.295 | 0.033 | -0.124 | -0.312 | 0.242 | -0.275 | -0.131 | 0.267
|  |  | Sd | 0.010 | 0.021 | 0.000 | 0.018 | 0.060 | 0.034 | 0.062 | 0.016 | 0.041 | 0.044 | 0.015 | 0.046 | 0.021 | 0.010
|  | 0.2 | Est. | -0.163 | 1.092 | 0.012 | 0.148 | -0.276 | -0.130 | -0.331 | 0.032 | -0.138 | -0.265 | 0.236 | -0.356 | -0.182 | 0.256
|  |  | Sd | 0.007 | 0.014 | 0.000 | 0.012 | 0.039 | 0.022 | 0.042 | 0.011 | 0.028 | 0.030 | 0.010 | 0.031 | 0.015 | 0.007
| Part1 | 0.4 | Est. | - | 1.089 | - | - | -0.060 | -0.158 | - | - | - | - | - | -0.375 | - | 0.279
|  |  | Sd | - | 0.042 | - | - | 0.129 | 0.067 | - | - | - | - | - | 0.097 | - | 0.020
|  | 0.3 | Est. | - | 1.122 | - | - | -0.231 | -0.200 | - | - | - | - | - | -0.296 | - | 0.297
|  |  | Sd | - | 0.033 | - | - | 0.103 | 0.052 | - | - | - | - | - | 0.074 | - | 0.016
|  | 0.2 | Est. | - | 1.108 | - | - | -0.305 | -0.139 | - | - | - | - | - | -0.248 | - | 0.282
|  |  | Sd | - | 0.023 | - | - | 0.067 | 0.036 | - | - | - | - | - | 0.050 | - | 0.011
| Part 2 | 0.4 | Est. | -0.165 | - | 0.013 | 0.160 | - | - | - | 0.026 | -0.180 | -0.346 | 0.209 | -0.308 | -0.116 | -
|  |  | Sd | 0.019 | - | 0.001 | 0.034 | - | - | - | 0.030 | 0.081 | 0.091 | 0.028 | 0.089 | 0.041 | -
|  | 0.3 | Est. | -0.172 | - | 0.012 | 0.182 | - | - | - | 0.027 | -0.151 | -0.334 | 0.220 | -0.317 | -0.121 | -
|  |  | Sd | 0.014 | - | 0.001 | 0.026 | - | - | - | 0.024 | 0.063 | 0.068 | 0.022 | 0.069 | 0.032 | -
|  | 0.2 | Est. | -0.166 | - | 0.013 | 0.161 | - | - | - | 0.019 | -0.134 | -0.294 | 0.224 | -0.298 | -0.126 | -
|  |  | Sd | 0.010 | - | 0.000 | 0.019 | - | - | - | 0.017 | 0.044 | 0.046 | 0.015 | 0.048 | 0.022 | -
A | All | 0.4 | Est. | -0.174 | 1.100 | 0.014 | 0.099 | -0.236 | -0.082 | -0.427 | 0.037 | -0.267 | -0.391 | 0.235 | -0.280 | -0.170 | 0.441
|  |  | Sd | 0.034 | 0.051 | 0.001 | 0.048 | 0.065 | 0.062 | 0.066 | 0.045 | 0.062 | 0.063 | 0.044 | 0.063 | 0.053 | 0.036
|  | 0.3 | Est. | -0.186 | 1.034 | 0.014 | 0.098 | -0.265 | -0.102 | -0.350 | 0.036 | -0.258 | -0.357 | 0.265 | -0.231 | -0.191 | 0.451
|  |  | Sd | 0.032 | 0.045 | 0.001 | 0.043 | 0.054 | 0.052 | 0.055 | 0.041 | 0.052 | 0.053 | 0.040 | 0.053 | 0.047 | 0.034
|  | 0.2 | Est. | -0.173 | 0.915 | 0.014 | 0.142 | -0.215 | -0.010 | -0.257 | 0.051 | -0.167 | -0.298 | 0.275 | -0.195 | -0.201 | 0.487
|  |  | Sd | 0.026 | 0.031 | 0.001 | 0.031 | 0.037 | 0.036 | 0.038 | 0.030 | 0.035 | 0.036 | 0.030 | 0.036 | 0.034 | 0.028
| Part1 | 0.4 | Est. | - | 1.084 | - | - | -0.057 | -0.058 | - | - | - | - | - | -0.125 | - | 0.449
|  |  | Sd | - | 0.055 | - | - | 0.084 | 0.075 | - | - | - | - | - | 0.082 | - | 0.036
|  | 0.3 | Est. | - | 1.104 | - | - | -0.118 | -0.072 | - | - | - | - | - | -0.187 | - | 0.446
|  |  | Sd | - | 0.054 | - | - | 0.073 | 0.068 | - | - | - | - | - | 0.070 | - | 0.036
|  | 0.2 | Est. | - | 1.058 | - | - | -0.262 | -0.084 | - | - | - | - | - | -0.254 | - | 0.441
|  |  | Sd | - | 0.046 | - | - | 0.056 | 0.054 | - | - | - | - | - | 0.055 | - | 0.034
| Part 2 | 0.4 | Est. | -0.175 | - | 0.014 | 0.106 | - | - | - | 0.015 | -0.270 | -0.352 | 0.239 | -0.187 | -0.206 | -
|  |  | Sd | 0.035 | - | 0.001 | 0.050 | - | - | - | 0.047 | 0.072 | 0.074 | 0.045 | 0.073 | 0.056 | -
|  | 0.3 | Est. | -0.175 | - | 0.014 | 0.097 | - | - | - | 0.027 | -0.248 | -0.368 | 0.235 | -0.292 | -0.173 | -
|  |  | Sd | 0.034 | - | 0.001 | 0.048 | - | - | - | 0.045 | 0.061 | 0.062 | 0.044 | 0.062 | 0.053 | -
|  | 0.2 | Est. | -0.182 | - | 0.014 | 0.089 | - | - | - | 0.036 | -0.238 | -0.338 | 0.285 | -0.206 | -0.215 | -
|  |  | Sd | 0.031 | - | 0.001 | 0.039 | - | - | - | 0.038 | 0.046 | 0.046 | 0.037 | 0.047 | 0.043 | -
B |  |  | Est. | -0.109 | 1.318 | 0.008 | 0.176 | -0.167 | -0.077 | -0.189 | 0.094 | 0.063 | -0.201 | 0.339 | -0.256 | -0.242 | 0.063
|  |  | Sd | 0.002 | 0.004 | 0.000 | 0.003 | 0.010 | 0.006 | 0.011 | 0.003 | 0.007 | 0.008 | 0.003 | 0.009 | 0.003 | 0.002
∗ COPD: Chronic obstructive pulmonary; CRF: Chronic renal failure; EOC:
Exposed to other cases diagnosed as SARS CoV-2.
∗ R and A stand for Random and Adaptive samplings, respectovely, and B is
Baseline estimation.
Table 9: Parameter estimation for CoVid data with $d_{2}=0.05$ and different proportion C2. |  | $d_{1}$ |  | Gender | Pneumonia | Age | Diabetes | COPD | Asthma | Immu | Hypertension | Other disease | Card | Obesity | CRF | Smoke | EOC
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
R | All | 0.4 | Est. | -0.156 | 1.094 | 0.012 | 0.163 | -0.256 | -0.173 | -0.378 | 0.038 | -0.147 | -0.318 | 0.221 | -0.313 | -0.101 | 0.243
|  |  | Sd | 0.012 | 0.027 | 0.000 | 0.023 | 0.081 | 0.043 | 0.083 | 0.021 | 0.055 | 0.057 | 0.019 | 0.060 | 0.027 | 0.013
|  | 0.3 | Est. | -0.151 | 1.086 | 0.012 | 0.160 | -0.294 | -0.156 | -0.332 | 0.040 | -0.152 | -0.326 | 0.239 | -0.266 | -0.133 | 0.238
|  |  | Sd | 0.010 | 0.021 | 0.000 | 0.018 | 0.060 | 0.033 | 0.062 | 0.016 | 0.041 | 0.044 | 0.015 | 0.046 | 0.021 | 0.010
|  | 0.2 | Est. | -0.150 | 1.078 | 0.012 | 0.156 | -0.280 | -0.144 | -0.355 | 0.037 | -0.159 | -0.264 | 0.236 | -0.334 | -0.186 | 0.231
|  |  | Sd | 0.007 | 0.014 | 0.000 | 0.012 | 0.040 | 0.022 | 0.042 | 0.011 | 0.028 | 0.030 | 0.010 | 0.032 | 0.015 | 0.007
| Part1 | 0.4 | Est. | - | 1.086 | - | - | -0.067 | -0.199 | - | - | - | - | - | -0.341 | - | 0.256
|  |  | Sd | - | 0.043 | - | - | 0.129 | 0.067 | - | - | - | - | - | 0.097 | - | 0.020
|  | 0.3 | Est. | - | 1.114 | - | - | -0.151 | -0.227 | - | - | - | - | - | -0.296 | - | 0.264
|  |  | Sd | - | 0.034 | - | - | 0.103 | 0.052 | - | - | - | - | - | 0.074 | - | 0.015
|  | 0.2 | Est. | - | 1.096 | - | - | -0.296 | -0.152 | - | - | - | - | - | -0.249 | - | 0.252
|  |  | Sd | - | 0.023 | - | - | 0.067 | 0.036 | - | - | - | - | - | 0.050 | - | 0.011
| Part 2 | 0.4 | Est. | -0.160 | - | 0.012 | 0.156 | - | - | - | 0.022 | -0.200 | -0.366 | 0.206 | -0.260 | -0.105 | -
|  |  | Sd | 0.018 | - | 0.001 | 0.033 | - | - | - | 0.030 | 0.082 | 0.089 | 0.028 | 0.089 | 0.041 | -
|  | 0.3 | Est. | -0.163 | - | 0.012 | 0.176 | - | - | - | 0.039 | -0.167 | -0.312 | 0.210 | -0.289 | -0.129 | -
|  |  | Sd | 0.014 | - | 0.000 | 0.026 | - | - | - | 0.024 | 0.063 | 0.067 | 0.022 | 0.069 | 0.032 | -
|  | 0.2 | Est. | -0.158 | - | 0.012 | 0.165 | - | - | - | 0.023 | -0.162 | -0.306 | 0.222 | -0.293 | -0.147 | -
|  |  | Sd | 0.010 | - | 0.000 | 0.018 | - | - | - | 0.017 | 0.043 | 0.046 | 0.015 | 0.048 | 0.022 | -
A | All | 0.4 | Est. | -0.170 | 1.101 | 0.014 | 0.094 | -0.241 | -0.094 | -0.418 | 0.026 | -0.260 | -0.373 | 0.239 | -0.302 | -0.185 | 0.440
|  |  | Sd | 0.034 | 0.050 | 0.001 | 0.048 | 0.063 | 0.061 | 0.065 | 0.045 | 0.061 | 0.063 | 0.044 | 0.062 | 0.053 | 0.036
|  | 0.3 | Est. | -0.178 | 1.030 | 0.015 | 0.079 | -0.230 | -0.126 | -0.331 | 0.047 | -0.237 | -0.337 | 0.253 | -0.242 | -0.192 | 0.453
|  |  | Sd | 0.032 | 0.044 | 0.001 | 0.043 | 0.053 | 0.052 | 0.053 | 0.041 | 0.051 | 0.051 | 0.040 | 0.052 | 0.047 | 0.034
|  | 0.2 | Est. | -0.166 | 0.913 | 0.014 | 0.117 | -0.211 | -0.009 | -0.232 | 0.051 | -0.170 | -0.288 | 0.284 | -0.208 | -0.196 | 0.463
|  |  | Sd | 0.027 | 0.031 | 0.001 | 0.031 | 0.037 | 0.036 | 0.038 | 0.031 | 0.036 | 0.036 | 0.030 | 0.037 | 0.034 | 0.028
| Part1 | 0.4 | Est. | - | 1.081 | - | - | -0.079 | -0.069 | - | - | - | - | - | -0.146 | - | 0.450
|  |  | Sd | - | 0.055 | - | - | 0.083 | 0.074 | - | - | - | - | - | 0.077 | - | 0.036
|  | 0.3 | Est. | - | 1.094 | - | - | -0.143 | -0.084 | - | - | - | - | - | -0.223 | - | 0.448
|  |  | Sd | - | 0.053 | - | - | 0.070 | 0.066 | - | - | - | - | - | 0.068 | - | 0.036
|  | 0.2 | Est. | - | 1.035 | - | - | -0.244 | -0.101 | - | - | - | - | - | -0.271 | - | 0.439
|  |  | Sd | - | 0.045 | - | - | 0.055 | 0.053 | - | - | - | - | - | 0.054 | - | 0.034
| Part 2 | 0.4 | Est. | -0.173 | - | 0.014 | 0.108 | - | - | - | 0.022 | -0.279 | -0.327 | 0.235 | -0.180 | -0.204 | -
|  |  | Sd | 0.035 | - | 0.001 | 0.050 | - | - | - | 0.046 | 0.070 | 0.072 | 0.045 | 0.071 | 0.056 | -
|  | 0.3 | Est. | -0.173 | - | 0.014 | 0.100 | - | - | - | 0.026 | -0.262 | -0.362 | 0.230 | -0.303 | -0.186 | -
|  |  | Sd | 0.034 | - | 0.001 | 0.047 | - | - | - | 0.044 | 0.060 | 0.061 | 0.044 | 0.062 | 0.052 | -
|  | 0.2 | Est. | -0.178 | - | 0.014 | 0.095 | - | - | - | 0.054 | -0.234 | -0.297 | 0.267 | -0.245 | -0.220 | -
|  |  | Sd | 0.031 | - | 0.001 | 0.039 | - | - | - | 0.037 | 0.045 | 0.045 | 0.037 | 0.046 | 0.043 | -
∗ COPD: Chronic obstructive pulmonary; CRF: Chronic renal failure; EOC: Exposed to other cases diagnosed as SARS CoV-2. |
∗ R and A stand for Random and Adaptive samplings, respectively.

##  4 Discussion and conclusion

From a statistical perspective, we introduce the idea of the distributed
sequential method to the scenarios of federated learning applications without
alternating its computing framework. Thus, we can conduct the sequential
procedures separately using individual data pools. The distributed computing
feature naturally bypasses latent issues attributed to data communication and
data management (Lindell, 2005; Feigenbaum et al., 2001; Dwork & Aaron, 2011;
Yan et al., 2013; Li et al., 2018; Carlini et al., 2019). The advantages of
the sequential estimation based methods are that they are data driven and have
a feature that guarantees the precision of the estimate when sampling is
stopped. This feature makes the final ensemble result more stable than other
combination methods. The adaptive sampling strategy, which is founded on
statistical experimental design and information theory allows us to select the
most informative samples from a massive data pool, which is a common situation
in many data collection scenarios, such as the pandemic data used in our
study. Although we only discuss the estimation problem, this idea can be
extended to other statistical inferences, which will be examined in furture
studies.

## Acknowledgment

This research is supported in part by research grants from the National
Natural Science Foundation of China (No. 11971457), and National Science and
Technology Council of Taiwan (No. 111-2118-M-001 -003 -MY2 and No.
108-2118-M-001 -004 -MY3).

## Appendix

## Proofs of main results

Throughout the proofs, notation $C$ denotes a generic positive constant, which
does not depend on $n$.

### A. Properties of sequential estimation for procedure $j$

When the stopping criterion (8) holds for procedure $j$, we record the
stopping time and stop sampling. The confidence ellipsoid for
$\bm{{\theta}}_{0}$ is

$\displaystyle R_{\tilde{N}_{j}}=\\{z\in
R^{p_{0}}:\frac{S_{\tilde{N}_{j}}}{\tilde{N}_{j}}\leq\frac{d_{1}^{2}}{\mu_{j\tilde{N}_{j}}}\\},$
(15)

where
$S_{\tilde{N}_{j}}=(\bm{{z}}-\tilde{\bm{{\theta}}}_{j\tilde{N}_{j}})^{\top}(\bm{{L}}_{j}\bm{{\Sigma}}_{j\tilde{N}_{j}}^{-1}\bm{{L}}_{j})^{-1}(\bm{{z}}-\tilde{\bm{{\theta}}}_{j\tilde{N}_{j}})$,
and $\bm{{z}}=(z_{1},\cdots,z_{p_{0}})^{\top}$. Length of maximum axis of this
ellipsoid is

$D=2\left(\frac{\tilde{N}_{j}d_{1}^{2}}{\mu_{j\tilde{N}_{j}}}\right)^{1/2}\lambda^{1/2}_{max}(\tilde{N}_{j}(\bm{{L}}_{j}\bm{{\Sigma}}_{j\tilde{N}_{j}}^{-1}\bm{{L}}_{j}))=2d_{1},$

where $\lambda_{max}(A)$ is the maximum eigenvalue of matrix $A$. Then we have
following Lemmas.

Proof of Lemma 1 Since $\tilde{a}_{j}^{2}>0$, for all $j$, are constants, the
definition of the stopping time implies that for each $j$,
$\tilde{N}_{j}/\hat{N}$ converges to some constant $\gamma_{j}>0$. Let
$\theta^{*}=\sum_{j=1}^{M}w_{j}\tilde{\bm{{\theta}}}_{j\tilde{N}_{j}}$, where
$w_{j}$, $\sum_{j=1}^{M}w_{j}=1$ are given weights. Then variance of
$\theta^{*}$ is

$\sum_{j=1}^{M}w^{2}_{j}{\bm{{L}}_{j}}\Sigma_{j\tilde{N}_{j}}^{-1}{\bm{{L}}_{j}}=\sum_{j=1}^{M}w^{2}_{j}\tilde{N}_{j}^{-1}{\bm{{L}}_{j}}(\Sigma_{j\tilde{N}_{j}}/\tilde{N}_{j})^{-1}{\bm{{L}}_{j}}.$

Let

$G_{N}(w_{1},\cdots,w_{M})=\hat{N}\sum_{j=1}^{M}w^{2}_{j}\tilde{N}_{j}^{-1}{\bm{{L}}_{j}}(\Sigma_{j\tilde{N}_{j}}/\tilde{N}_{j})^{-1}{\bm{{L}}_{j}},$

then it follows that as $d_{1}$ tends to 0,

$G_{N}(w_{1},\cdots,w_{M})\longrightarrow
G(w_{1},\cdots,w_{M})=\sum_{j=1}^{M}w^{2}_{j}\gamma_{j}^{-1}{\bm{{L}}_{j}}\Sigma_{j}^{-1}{\bm{{L}}_{j}}.$

If we use the same covariates for all data sites, then they have a homogeneous
covariance $\Sigma_{1}=\Sigma_{2}=\cdots=\Sigma_{M}$, asymptotically. We then
minimize $G(w_{1},\cdots,w_{M})$ with respect to $w_{1},\cdots,w_{M}$ subject
to $\sum_{j}w_{j}=1$, which gives that $w_{j}=\gamma_{j}$, $j=1,\cdots,M$. It
follows that $\hat{\bm{{\theta}}}$ with the weights $\rho_{j}$ achieves the
minimal covariance, asymptotically.

LEAMMA C1.  Assume that the conditions of Theorem 1 are satisfied, and
$\tilde{N}_{j}$ is as defined in (8). Then

$\displaystyle\lim_{d\rightarrow
0}\frac{d^{2}\tilde{N}_{j}}{\tilde{a}_{j}^{2}\mu}=1~{}~{}~{}~{}\text{\rm
almost surely},$ (16) $\displaystyle\lim_{d\rightarrow 0}P(\theta_{0}\in
R_{\tilde{N}_{j}})=1-\tilde{\alpha}_{j},$ (17)
$\displaystyle\lim_{d\rightarrow
0}\frac{d^{2}E(\tilde{N}_{j})}{\tilde{a}_{j}^{2}\mu}=1,$ (18)

where $\tilde{\alpha}_{j}$ satisfies
$P(\chi_{p}^{2}>\tilde{a}_{j}^{2})=\tilde{\alpha}_{j}$, and $\mu$ is the
maximum eigenvalue of matrix $\bm{{L}}_{j}\Sigma^{-1}\bm{{L}}_{j}$.

Proof. For a single linear regression model, Wang & Chang (2013) obtained
asymptotic consistency and asymptotic efficiency of the sequential estimation.
Similar to Wang & Chang (2013), proof of this lemma for logistic regression
model (14) is straightforward and is omitted here.

### B. Proof of Theorem 1

If the stopping criterion defined in (8) holds for the procedure $j$ , then we
have used $\tilde{N}_{j}$ observations, and obtained a confidence set
$R_{\tilde{N}_{j}}$ as defined in (15). Via Lemma C1

$\displaystyle\lim_{d_{1}\rightarrow
0}\frac{d_{1}^{2}\tilde{N}_{j}}{\tilde{a}_{j}^{2}\mu}=1~{}~{}~{}~{}\text{\rm
almost surely},$ (19) $\displaystyle\lim_{d_{1}\rightarrow 0}P(\theta_{0}\in
R_{\tilde{N}_{j}})=1-\tilde{\alpha}_{j},$ (20)
$\displaystyle\lim_{d_{1}\rightarrow
0}\frac{d_{1}^{2}E(\tilde{N}_{j})}{\tilde{a}_{j}^{2}\mu}=1,$ (21)

where $\tilde{\alpha}_{j}$ satisfies
$P(\chi_{p}^{2}>\tilde{a}_{j}^{2})=\tilde{\alpha}_{j}$. Equations (19) and
(21) implies that as $d_{1}\rightarrow 0$,

$\displaystyle
d_{1}^{2}\tilde{N}_{j}\longrightarrow\tilde{a}_{j}^{2}\mu~{}~{}~{}~{}\text{\rm
almost surely},$ $\displaystyle
d_{1}^{2}E(\tilde{N}_{j})\longrightarrow\tilde{a}_{j}^{2}\mu.$

Because $\tilde{a}^{2}_{1}+\tilde{a}^{2}_{2}=a^{2}$, it follows that

$\displaystyle
d_{1}^{2}\hat{N}=d_{1}^{2}(\tilde{N}_{1}+\tilde{N}_{2})\longrightarrow(\tilde{a}_{1}^{2}+\tilde{a}_{2}^{2})\mu=a^{2}\mu~{}~{}~{}~{}\text{\rm
almost surely},$ $\displaystyle
d_{1}^{2}E(\hat{N})=d_{1}^{2}E(\tilde{N}_{1}+\tilde{N}_{2})\longrightarrow(\tilde{a}_{1}^{2}+\tilde{a}_{2}^{2})\mu=a^{2}\mu,$

which implies that

$\displaystyle\lim_{d_{1}\rightarrow
0}\frac{d_{1}^{2}\hat{N}}{a^{2}\mu}=1~{}~{}~{}~{}\text{\rm almost surely},$
$\displaystyle\lim_{d_{1}\rightarrow
0}\frac{d_{1}^{2}E(\hat{N})}{a^{2}\mu}=1.$

For simplicity, we suppose that $j=1,2$, in the following matrix algebra,
which csn be easily extended to the case for $j=1,\cdots,M$. Matrix
$\bm{{L}}_{j}\bm{{\Sigma}}_{j\tilde{N}_{j}}^{-1}\bm{{L}}_{j}$ can be used to
estimate covariance of $\tilde{\bm{{\theta}}}_{j\tilde{N}_{j}}$. Since two
procedures are independent, we can use
$\rho_{1}^{2}\bm{{L}}_{j}\bm{{\Sigma}}_{1N_{1}}^{-1}\bm{{L}}_{j}+\rho_{2}^{2}\bm{{L}}_{j}\bm{{\Sigma}}_{2N_{2}}^{-1}\bm{{L}}_{j}$
to estimate variance of $\hat{\bm{{\theta}}}$. We already have that
$\sqrt{\tilde{N}_{j}}(\tilde{\bm{{\theta}}}_{\tilde{N}_{j}}-\bm{{\theta}}_{0})$
has asymptotic normality as $d_{1}$ tends to 0. It follows that as
$d_{1}\rightarrow 0$, $\hat{\bm{{\theta}}}$ follows an asymptotic normal
distribution, and

$\displaystyle(\hat{\bm{{\theta}}}-\bm{{\theta}}_{0})^{\top}\left[\rho_{1}^{2}\bm{{L}}_{j}\bm{{\Sigma}}_{1N_{1}}^{-1}\bm{{L}}_{j}+\rho_{2}^{2}\bm{{L}}_{j}\bm{{\Sigma}}_{2N_{2}}^{-1}\bm{{L}}_{j}\right]^{-1}(\hat{\bm{{\theta}}}-\bm{{\theta}}_{0})\longrightarrow\chi_{p_{0}}^{2}.$
(22)

By definition of $\mu_{\hat{N}}$, $\mu_{\hat{N}}\rightarrow\mu$ and
${\hat{N}d_{1}^{2}}/{\mu_{\hat{N}}}\rightarrow a^{2}$ almost surely, as
$d_{1}\rightarrow 0$. Thus, (22) implies that

$\displaystyle\lim_{d_{1}\rightarrow 0}P(\bm{{\theta}}_{0}\in R_{\hat{N}})$
$\displaystyle=\lim_{d_{1}\rightarrow
0}P\left((\hat{\bm{{\theta}}}-\bm{{\theta}}_{0})^{\top}\left[\rho_{1}^{2}\bm{{L}}_{j}\bm{{\Sigma}}_{1N_{1}}^{-1}\bm{{L}}_{j}+\rho_{2}^{2}\bm{{L}}_{j}\bm{{\Sigma}}_{2N_{2}}^{-1}\bm{{L}}_{j}\right]^{-1}(\hat{\bm{{\theta}}}-\bm{{\theta}}_{0})\leq\frac{\hat{N}d_{1}^{2}}{\mu_{\hat{N}}}\right)$
$\displaystyle=\lim_{d_{1}\rightarrow
0}P\left((\hat{\bm{{\theta}}}-\bm{{\theta}}_{0})^{\top}\left[\rho_{1}^{2}\bm{{L}}_{j}\bm{{\Sigma}}_{1N_{1}}^{-1}\bm{{L}}_{j}+\rho_{2}^{2}\bm{{L}}_{j}\bm{{\Sigma}}_{2N_{2}}^{-1}\bm{{L}}_{j}\right]^{-1}(\hat{\bm{{\theta}}}-\bm{{\theta}}_{0})\leq
a^{2}\right)$ $\displaystyle=1-\alpha.$

This completes the proof of Theorem 1.

## References

  * Bartroff et al. (2013) Bartroff, J., Lai, T. & Shih, M. (2013).  Sequential Experimentation in Clinical Trials.  Springer, New York, USA.
  * Carlini et al. (2019) Carlini, N., Liu, C., Erlingsson, U., Kos, J. & Song, D. (2019).  The secret sharer: Evaluating and testing unintended memorization in neural networks.  Security Symposium , 267–284.
  * Chang (2011) Chang, Y.-c. I. (2011).  Sequential estimation in generalized linear models when covariates are subject to errors.  Metrika 73, 93–120.
  * Chang & Martinsek (1992) Chang, Y.-c. I. & Martinsek, A. T. (1992).  Fixed Size Confidence Regions for Parameters of a Logistic Regression Model.  The Annals of Statistics 20, 1953 – 1969.
  * Chang (1999) Chang, Y. I. (1999).  Strong consistency of maximum quassi-likelihood eistimate in generalized linear models via a last time.  Statistics & Probability Letters 45, 237–246.
  * Chen et al. (1999) Chen, K., Hu, I. & Ying, Z. (1999).  Strong consistency of maximum quasi-likelihood estimators in generalized linear models with fixed and adaptive designs.  Ann. of Statist. 27, 1155–1163.
  * Chen et al. (2020) Chen, Z., Wang, Z. & Chang, Y. (2020).  Sequential adaptive variables and subject selection for gee methods.  Biometrics 76, 496–507.
  * Chow & Robbins (1965) Chow, Y. & Robbins, H. (1965).  On the asymptotic theory of fixed-width sequential confidence intervals for the mean.  Ann. Math. Statist. 36, 457–462.
  * Damiani et al. (2015) Damiani, A., Vallati, M., Gatta, R., Dinapoli, N., Jochems, A., Deist, T., Soest, J. v., Dekker, A. & Valentini, V. (2015).  Distributed learning to protect privacy in multi-centric clinical studies.  In Conference on artificial intelligence in medicine in europe. Springer.
  * Deng et al. (2009) Deng, X., Joseph, V., Sudjianto, A. & Wu, C. F. (2009).  Active learning through sequential design, with applications to detection of money laundering.  Journal of the American Statistical Association 104, 969–981.
  * Dwork & Aaron (2011) Dwork, C. & Aaron, R. (2011).  The algorithmic foundations of differential privacy.  Found. Trends Theor. Comput 9, 211–407.
  * Feigenbaum et al. (2001) Feigenbaum, J., Ishai, Y., Malkin, T., Nissim, K., Strauss, M. J. & Wright, R. N. (2001).  Secure multiparty computation of approximations.  In International Colloquium on Automata, Languages, and Programming. Springer.
  * Goetz et al. (2019) Goetz, J., Malik, K., Bui, D., Moon, S., Liu, H. & Kumar, A. (2019).  Active federated learning.  arXiv preprint: arXiv:1909.12641 .
  * Hernández-Garduño (2020) Hernández-Garduño, E. (2020).  Obesity is the comorbidity more strongly associated for covid-19 in mexico. a case-control study.  Obes Res Clin Pract. 14, 375–379.
  * Huang et al. (2020) Huang, L., Yin, Y., Fu, Z., Zhang, S., Deng, H. & Liu, D. (2020).  Loadaboost: Loss-based adaboost federated machine learning with reduced computational complexity on iid and non-iid intensive care data.  Plos one 15, e0230706.
  * Jordan et al. (2019) Jordan, M., Lee, J. & Yang, Y. (2019).  Communication-efficient distributed statistical inference.  Journal of the American Statistical Association 114, 668–681.
  * Li et al. (2018) Li, C., Zhou, P., Xiong, L., Wang, Q. & Wang, T. (2018).  Differentially private distributed online learning.  IEEE Transactions on Knowledge and Data Engineering 30, 1440–1453.
  * Li et al. (2020a) Li, J., Chen, Z., Wang, Z. & Chang, Y. (2020a).  Active learning in multiple-class classification problems via individualized binary models.  Computational Statistics & Data Analysis 145, 106–119.
  * Li et al. (2020b) Li, T., Sahu, A. K., Talwalkar, A. & Smith, V. (2020b).  Federated learning: Challenges, methods, and future directions.  IEEE Signal Processing Magazine 37, 50–60.
  * Lindell (2005) Lindell, Y. (2005).  Secure multiparty computation for privacy preserving data mining.  In Encyclopedia of Data Warehousing and Mining. IGI global, pp. 1005–1009.
  * Liu et al. (2020) Liu, S., Zhi, Y. & Ying, S. (2020).  Covid-19 and asthma: Reflection during the pandemic.  Clin Rev Allergy Immunol. 59, 78–88.
  * Louis et al. (2020) Louis, R., Calmes, D., Frix, A. & Schleich, F. (2020).  Covid-19 and asthma.  Rev Med Liege. 75, 130–132.
  * McCullagh & Nelder (1989) McCullagh, P. & Nelder, J. A. (1989).  Generalized Linear Models, 2nd Edition.  Chapman & Hall, New York.
  * McMahan et al. (2017) McMahan, B., Moore, E., Ramage, D., Hampson, S. & y Arcas, B. A. (2017).  Communication-efficient learning of deep networks from decentralized data.  Artificial intelligence and statistics. PMLR , 1273–1282.
  * Memon & Biswas (2022) Memon, S. & Biswas, D. (2022).  Covid-19 and diabetes mellitus: From pathophysiology to clinical management.  Cureus 14, e31895.
  * Montgomery (2009) Montgomery, D. C. (2009).  Design and Analysis of experiments.  Hoboken, NJ, USA: JohnWiley&Sons., 7th ed.
  * Nelder & Wedderburn (1972) Nelder, J. A. & Wedderburn, R. W. M. (1972).  Generalized linear models.  Journal of the Royal statistical society, series A 135, 370–384.
  * Park & Chang (2016) Park, E. & Chang, Y. (2016).  Multiple-stage sampling procedure for covariate-adjusted response-adaptive designs.  Statistical Methods in Medical Research 25, 1490–1511.
  * Rashedi et al. (2020) Rashedi, J., Mahdavi, P. B., Asgharzadeh, V., Pourostadi, M., Samadi, K. H., Vegari, A., Tayebi-Khosroshahi, H. & Asgharzadeh, M. (2020).  Risk factors for covid-19.  Infez Med. 28, 469–474.
  * Settles (2010) Settles, B. (2010).  Active learning literature survey.  University of Wisconsin, Madison 52, 11.
  * Wang & Chang (2013) Wang, Z. & Chang, Y.-c. I. (2013).  Sequential estimate for linear regression models with uncertain number of effective variables.  Metrika 76, 949–978.
  * Whitehead (2014) Whitehead, J. (2014).  Sequential Methods for Clinical Trials.  John Wiley & Sons, Ltd.
  * Woodroofe (1982) Woodroofe, M. (1982).  Nonlinear renewal theory in sequential analysis.  CBMS-NSF regional conference series in applied mathematics.
  * Woods et al. (2006) Woods, D., Lewis, S., Eccleston, J. & Russell, K. (2006).  Designs for generalized linear models with several variables and model uncertainty.  Technometrics 48, 284–292.
  * Yan et al. (2013) Yan, F., Sundaram, S., Vishwanathan, S. & Qi, Y. (2013).  Distributed autonomous online learning: Regrets and intrinsic privacy-preserving properties.  IEEE Transactions on Knowledge and Data Engineering 25, 2483–2493.
  * Zhou et al. (2009) Zhou, X., McClish, D. & Obuchowski, N. (2009).  Statistical methods in diagnostic medicine.  John Wiley & Sons.
  * Zhou & Tang (2020) Zhou, Y. & Tang, S. (2020).  Differentially private distributed learning.  INFORMS Journal on Computing 32, 779–789.