# DemOpts: Fairness corrections in COVID-19 case prediction models

Naman Awasthi
Saad Mohammad Abrar,
Daniel Smolyak,
Vanessa Frias-Martinez

###### Abstract

COVID-19 forecasting models have been used to inform decision making around
resource allocation and intervention decisions e.g., hospital beds or stay-at-
home orders. State of the art deep learning models often use multimodal data
such as mobility or socio-demographic data to enhance COVID-19 case prediction
models. Nevertheless, related work has revealed under-reporting bias in
COVID-19 cases as well as sampling bias in mobility data for certain minority
racial and ethnic groups, which could in turn affect the fairness of the
COVID-19 predictions along race labels. In this paper, we show that state of
the art deep learning models output mean prediction errors that are
significantly different across racial and ethnic groups; and which could, in
turn, support unfair policy decisions.

We also propose a novel de-biasing method, DemOpts, to increase the fairness
of deep learning based forecasting models trained on potentially biased
datasets. Our results show that DemOpts can achieve better error parity that
other state of the art de-biasing approaches, thus effectively reducing the
differences in the mean error distributions across more racial and ethnic
groups.

##  1 Introduction

Forecasting the number of COVID-19 cases, hospitalizations or deaths is
crucial to inform decision making. For example, COVID-19 forecasts can be used
by hospitals to evaluate medical needs and required resources such as supplies
or beds; or by public health officials to inform closure policies at various
geographical scales. In the US, COVID-19 forecasts have been used at the state
and county levels to inform social distancing or masking, such as the publicly
available forecasts on the COVID-19 Forecast Hub that the CDC routinely uses
in their communications (for Disease Control and Prevention 2023; CDC 2020).

Related work for the past three years has shown a diverse variety of COVID-19
forecasting approaches, from compartmental models (Zou et al. 2020; Pei and
Shaman 2020) to statistical (Chiang, Liu, and Mohler 2020; Galasso, Cao, and
Hochberg 2022) or deep learning methods (Arik et al. 2020a; Zhang-James et al.
2021; Le et al. 2020a; Lucas, Vahedi, and Karimzadeh 2023a). These models,
always trained with past COVID-19 cases publicly available from, for example,
NYT or JHU (Times 2021; Dong, Du, and Gardner 2020), frequently use
complementary datasets with the objective of improving forecasting accuracy.
In fact, analysing the forecasting models from over $50$ teams in the COVID-19
Forecast Hub, $39\%$ use demographic data - either directly from the ACS or
indirectly via community vulnerability indices like the CCVI (Smittenaar et
al. 2021); and $52\%$ of the models incorporate human mobility data from
Safegraph, Google or Apple, among others (Google 2022; Apple 2022; Labs 2023).

The majority of publications focused on COVID-19 case prediction have reported
results around the accuracy of the models i.e., minimizing the difference
between the predicted cases and the actual number of cases reported.
Nevertheless, prior work has shown that the accuracy of COVID-19 predictions
can depend on various social determinants, including race or ethnicity (Gursoy
and Kakadiaris 2022), income, or age (Erfani and Frias-Martinez 2023),
revealing worse performance for protected attributes and pointing to a lack on
COVID-19 predictive fairness that can affect resource allocation and decision
making. This lack of predictive fairness might be related to bias in the
datasets used to train the model i.e., bias in COVID-19 case reporting or bias
in mobility data. In fact, prior work has shown COVID-19 case bias due to
under-reporting issues in minority communities whereby missing racial data or
misclassified race has been a source of errors (Douglas et al. 2021) as well
as inadequate testing for minority groups across the US, such as
Hispanic/Latino communities (Del Rios et al. 2022). Additionally, prior work
has also revealed sampling bias in mobility data with Black and elder
communities being under-represented because of the way mobility data is
collected (via smart phones and mobile app use) (Coston et al. 2021).

Given the presence of bias in the training datasets frequently used by
COVID-19 forecast models, and prior work demonstrating that COVID-19
prediction accuracy can vary across social determinants, we posit that it
becomes critical to devise methods to prevent data biases from percolating
into the COVID-19 forecasts so as to guarantee fair decision making based on
case predictions. Mitigating bias in COVID-19 forecast models can be done
through pre-processing or in-processing approaches i.e., via bias mitigation
in the training datasets, applying correction methods to COVID-19 counts
(Angulo, Finelli, and Swerdlow 2021; Jagodnik et al. 2020); or via de-biasing
methods embedded in the predictive models that attempt to reduce data and
model bias during training (Yan, Seto, and Apostoloff 2022; Wang and Singh
2023; Yang, Soltan, and Clifton 2022; Estiri et al. 2022). In this paper, we
focus on in-processing approaches given their scarcity in the COVID-19
literature, and propose DemOpts (Demographic Optimization) a de-biasing method
designed to achieve COVID-19 case prediction error parity across racial and
ethnic groups in the context of deep learning models i.e., guarantee that
county prediction errors are not significantly different across racial and
ethnic groups. Although there exist a diverse set of COVID-19 predictive
approaches, we focus on deep learning models, because these are the most
frequently used models in the machine learning community (Meraihi et al.
2022); and narrow down our choice to transformer-based architectures in
particular, because they are state of the art in time series predictions (Lim
et al. 2021a).

The main objective of DemOpts is to improve the fairness of the COVID-19 case
predictions at the county level by achieving error parity in a regression
setting (Gursoy and Kakadiaris 2022). DemOpts proposes a novel de-biasing
approach that leverages county racial and ethnic data during training to
modify conventional deep learning loss functions so as to penalize the model
for statistically significant associations between the predictive error and
the race or ethnicity distribution of a county.

Like state of the art de-biasing methods for regression settings (such as
Individual (Berk et al. 2017), Group (Berk et al. 2017) and Sufficiency-based
fairness correction (Shah et al. 2022)) DemOpts can work in multimodal
contexts, allowing for deep learning models to be trained with different types
of input data besides the COVID-19 cases, including the use of mobility or
demographic data, which are frequently used in COVID-19 prediction models.
However, unlike state of the art de-biasing methods for regression, DemOpts is
designed to de-bias predictions based on the relationship between the
prediction errors and the percentage of racial and ethnic groups in that
county, effectively considering multiple protected groups per county in the
de-biasing process, instead of assigning a county to a unique protected race
or ethnicity.

Thus, the main contributions of this paper are:

  * •

We present DemOpts, a novel de-biasing method for deep learning architectures,
that attempts to increase the fairness of the COVID-19 county case predictions
by achieving error parity i.e., guarantee that prediction errors are similar
across racial and ethnic groups.

  * •

The DemOpts architecture is designed to optimize for error parity across race
and ethnicity using a novel multi-label approach that allows each county to be
characterized by its own racial and ethnic group distribution during the de-
biasing process, instead of by a unique label.

  * •

We propose a novel evaluation protocol for the COVID-19 context and we show
that: (i) state of the art COVID-19 county case prediction models based on
transformer architectures lack error parity when no de-biasing method is
applied i.e., prediction errors are statistically significantly different
across racial and ethnic groups; (ii) DemOpts applied to transformer-based
architectures improves the error parity of COVID-19 county case prediction
models, increasing the similarity between mean prediction errors across racial
and ethnic groups, and (iii) DemOpts de-biasing approach performs better than
existing de-biasing methods for regression settings, namely, individual
fairness correction, group fairness correction and sufficiency-based penalty
for fairness correction.

The rest of the paper is structured as follows. We first discuss the related
literature, followed by a description of the DemOpts method to achieve error
parity in transformer based architectures, and its evaluation protocol for the
COVID-19 context, including two metrics to measure error parity. We finalize
the paper presenting the evaluation of DemOpts: first describing the datasets
used, and then showing how DemOpts improves fairness prediction by increasing
error parity across racial and ethnic groups.

##  2 Related Literature

In this section, we cover three areas that are of relevance to the research
proposed in this paper: deep learning models to forecast time series data, the
presence of bias in COVID-19 datasets used by COVID-19 case forecasting
methods; and approaches to measure and improve the fairness of predictions in
regression settings.

### Deep learning based Forecasting models

Deep learning models have started to become popular in time series prediction
tasks. The available methods include, (i) Autoregressive models, which are
modifications of recurrent neural networks (RNNs) such as Long Short Term
Memory (LSTM) Networks or Gated Recurrent Networks (GRN)(Hochreiter and
Schmidhuber 1997); (ii) Graph-based neural networks which encapsulate spatio-
temporal aspects of data in implementations such as Graph Attention Networks
(GANNs)(Velickovic et al. 2017), Spatio-temporal Graph Convolution network
(ST-GCN)(Yu, Yin, and Zhu 2018a), NBConv (Duan et al. 2017) or GGConv(Yu, Yin,
and Zhu 2018b); and (iii) transformers, which have gained success in various
applications such as computer vision (Dosovitskiy et al. 2021; Kirillov et al.
2023), natural language processing (Radford et al. 2018; Devlin et al. 2018),
speech (Latif et al. 2023) or tabular data (Liu et al. 2021; Wang et al. 2020;
Yin et al. 2020). There is a large body of work that utilizes Transformer-
based architecture models (Vaswani et al. 2017) to forecast time series with
state of the art performance including LogTrans (Li et al., 2019)(Li et al.
2019), Informer (Zhou et al., 2021) (Zhou et al. 2020), Autoformer (Wu et al.,
2021) (Wu et al. 2021), FEDformer (Zhou et al., 2022)(Zhou et al. 2022),
Pyraformer (Liu et al., 2022)(Liu et al. 2022), and PatchTST(Nie et al. 2023).
In this paper, we specifically focus on the Temporal Fusion Transformer
architecture (TFT) (Lim et al. 2021b) since it allows us to easily incorporate
exogenous variables (like mobility data) as well as static variables (like
demographic data) on top of the COVID-19 time series.

### Bias in Mobility and COVID-19 Data

Human mobility data has been used to characterize human behaviors in the built
environment (Vieira et al. 2010; Hernandez et al. 2017; Frias-Martinez and
Virseda 2013; Rubio et al. 2010; Wu, Frias-Martinez, and Frias-Martinez 2021),
for public safety (Wu et al. 2022, 2023), during epidemics and disasters
(Wesolowski et al. 2012; Bengtsson et al. 2015; Hong et al. 2017; Isaacman,
Frias-Martinez, and Frias-Martinez 2018; Ghurye, Krings, and Frias-Martinez
2016; Hong and Frias-Martinez 2020), as well as to support decision making for
socio-economic development (Frias-Martinez, Virseda, and Frias-Martinez 2010;
Fu et al. 2018; Frias-Martinez, Virseda, and Gomero 2012; Hong, Frias-
Martinez, and Frias-Martinez 2016; Frias-Martinez et al. 2012). During the
COVID-19 pandemic, human mobility has also played a central role in driving
decision making, acknowledging the impact of human movement on virus
propagation (Arik et al. 2020b; Lucas, Vahedi, and Karimzadeh 2023b; Le et al.
2020b; Erfani and Frias-Martinez 2023; Badr and Gardner 2021; Abrar et al.
2023).

Bias in mobility and COVID-19 data is being increasingly discussed due to the
exponential growth of COVID-19 forecasting models in the literature, and to
publicly available mobility data. Mobility data has been reported to suffer
from sampling bias given that digital traces are being collected from mobile
apps installed on smart phones, which limits the types of individuals from
whom mobility data is being collected. In fact, prior work has revealed
sampling bias in SafeGraph mobility data with Black and elder individuals
being under-represented in the datasets (Coston et al. 2021). Critical to the
research proposed in this paper, prior work has exposed biases in COVID-19
forecasting models (Tsai et al. 2022), and researchers have shown that
COVID-19 county prediction improvements associated to the use of mobility data
tend to take place in counties with lower presence of protected racial and
ethnic groups (Abrar et al. 2023). On the other hand, COVID-19 under-reporting
bias has been discussed in the literature (Douglas et al. 2021; Gross et al.
2020) and points to multiple causes, including inadequate testing across
certain minority groups such as Hispanic/Latinos (Del Rios et al. 2022); or
lack of consistency in reporting race and ethnicity for COVID-19 cases, which
has generated a lot of missing or incorrect racial data in COVID-19 case
statistics, as reported by the CDC (CDC 2023). Reducing the impact of mobility
or COVID-19 case bias in COVID-19 case predictions, as we do in this paper, is
of critical importance to support decision making processes focused on
resource allocation during pandemics, so as to reduce harms and guarantee that
decisions are fair and just across racial and ethnic groups.

### Fairness Metrics and Fairness Corrections in Machine learning models

Transformer-based COVID-19 case forecast models require the use of fairness
metrics for regression settings, given that the loss optimization process in
gradient based deep learning architectures uses real number predictions
instead of classes. Agarwal et al. (Agarwal, Dudík, and Wu 2019), Fitzsimons
et al. (Fitzsimons et al. 2019) or Gursoy et al. (Gursoy and Kakadiaris 2022)
outline the different aspects of fairness in regression settings, and propose
a set of fairness metrics for regression-type models. For this paper, we use
the error parity metric proposed in (Gursoy and Kakadiaris 2022). Error parity
requires error distributions to be statistically independent from racial and
ethnic groups. We expand this definition, and relax the statistical
significance requirement, to be able to also evaluate whether the proposed
DemOpts method can at least reduce the differences in error distributions
across racial and ethnic groups, even when they are still be statistically
significantly different.

To correct for bias and unfair performance in deep learning models,
researchers have used pre-processing and in-processing correction approaches.
Pre-processing approaches focus on creating a better input for learning deep
neural network models by removing bias from the datasets (Brunet et al.
2018),(Calmon et al. 2017); and there have been successful efforts focused on
de-biasing under-reporting COVID-19 datasets to estimate actual cases or
deaths before they are fed into predictive models (Jagodnik et al. 2020;
Albani et al. 2021). On the other hand, in-processing approaches to improve
the fairness of deep learning models, like the one we use in this paper, focus
on the model and its regularization, usually adding a bias correction term in
the loss function (Wang and Singh 2023; Das and Dooley 2023; Yan, Seto, and
Apostoloff 2022).

In this paper, we will compare our proposed de-biasing approach against three
state-of-the-art methods for de-biasing in regression settings: Individual
fairness correction (Berk et al. 2017), Group Fairness correction (Berk et al.
2017) and sufficiency based penalty for fairness correction (Shah et al.
2022). Individual and group fairness calculate penalties by determining over-
estimations across different groups and weighting the loss by a factor
proportional to the over-estimations; while sufficiency based regularizers
propose to make the loss independent of sensitive data attributes by
simultaneously training a joint model and subgroup specific networks to
achieve fair predictions (Shah et al. 2022).

##  3 DemOpts Method

Our modeling focus is on deep learning models, which are the most frequently
used approach for COVID-19 county case forecasts in the machine learning
community (Meraihi et al. 2022). We specifically focus on the Temporal Fusion
Transformer (TFT) model introduced in (Lim et al. 2021b) for several reasons.
First, this model is state of the art in interpretable time series prediction
(Lim et al. 2021b). Second, this model allows for the use of static reals as
input to the model (attributes that do not change over the duration of the
training process such as demographic percentages or population statistics);
and third, the model works well with time-dependent features including
COVID-19 cases or mobility data whereby past data influences future
statistics.

Figure 1: Flow Diagram for the DemOpts method.

Training a deep learning model has the following steps: (1) forward pass on
the training data, (2) computation of loss and (3) backward pass to change
weights of the model. DemOpts modifies conventional loss functions to penalize
the model for any statistically significant association between the county
prediction loss (error) and the county racial and ethnic groups. In other
words, DemOpts performs a race-based optimization on the error during model
training using county demographic racial and ethnic data. To achieve that,
DemOpts follows a three step process (see Figure 1 for a diagram and Algorithm
1 for the pseudocode):

### Step 1: Calculate Loss

To thoroughly model errors in time series, we use quantile predictions instead
of point-value predictions. Quantile predictions are measured for seven
quantiles ([0.02, 0.1, 0.25, 0.5, 0.75, 0.9, 0.98]) to gain insights into the
uncertainty ranges and confidence intervals of the COVID-19 county case
predictive models. When using quantile predictions, the error is computed
using the quantile loss, also known as pinball loss (PBL), and defined as
follows:

$PBL_{q}(y_{ip},y_{i})=\begin{cases}q*(y_{i}-y_{ip})&\text{if }y_{i}\geq
y_{ip}\\\ (q-1)*(y_{i}-y_{ip})&\text{if }y_{i}<y_{ip}\end{cases}$ (1)

For quantile $q$, the PBL for the prediction of a given input $X_{i}$ is
$PBL_{q}(y_{ip},y_{i})$, where $y_{i}$ is the ground truth and $y_{ip}$ is the
predicted value. The average over all quantiles can be represented as
$PBL(y_{ip},y_{i})=\frac{1}{|q|}\sum_{q}PBL_{q}(y_{ip},y_{i})$.

### Step 2: Identify Dependencies between Prediction Errors and Race and
Ethnicity

To achieve error parity i.e., mean errors being independent from racial and
ethnic groups, DemOpts first determines the relationship between errors and
race and ethnic labels. For that purpose, DemOpts fits a regression model
between the prediction losses $PBL(y_{ip},y_{i})$ across datapoints $i$ and
their corresponding county race and ethnicity labels $D_{i}$:

$\displaystyle PBL(y_{ip},y_{i})$ $\displaystyle=\beta*D_{i}+\alpha$ (2)
$\displaystyle\qquad\text{with}\quad D_{i}$
$\displaystyle=\begin{bmatrix}d_{1},d_{2},d_{3},d_{4},\text{lookahead}\end{bmatrix}$

where $d_{i}$ are the corresponding county demographic features extracted from
the U.S. census data and represented as the percentage of each racial and
ethnic group of the county corresponding to datapoint $i$, and lookahead
refers to the number of days into the future the COVID-19 case prediction was
generated for. In matrix representation: $PBL(Y_{ip},Y_{i})=\beta*D+\alpha$.
Once the regression model is fit, both regression coefficients ($\beta$) and
their statistical significance ($p-value$) are passed on to step 3, to modify
the adjusted loss and attempt to decouple race from the errors (loss).

### Step 3: Adjust the Loss

DemOpts modifies the conventional loss of deep learning models by adjusting
for racial or ethnic bias in the error i.e., the loss is increased whenever a
statistically significant regression coefficient for a race or ethnicity is
found in Step 2 (at level $p-value=0.05$). By increasing the loss, DemOpts
attempts to reduce the dependency between errors and race and ethnicity i.e.,
make the errors similar across racial and ethnic groups. Specifically, the
loss is adjusted by the product of the prior loss $PBL(y_{ip},y_{i})$, the
percentage race or ethnicity $D_{j}$ that holds a significant relationship
with the error, and its coefficient $\beta_{j}$ in absolute value:

$\displaystyle
L_{adj}=PBL(y_{ip},y_{i})+\sum_{j}H(pval_{j})(|\beta_{j}|\\*D_{j}\\*L)$ (3)
$\displaystyle\text{with}\qquad H(x)=\begin{cases}1&\text{if }x<0.05,\\\
0&\text{if }x\geq 0.05\end{cases}$

Algorithm 1 DemOpts Training

1:Input: Training set (X, D, Y), Learning rate (lr), Number of epochs
(epochs), threshold

2:Output: Trained model (M)

3:X : COVID-19 Timeseries data for all counties

4:Y : COVID-19 cases in future for all counties

5:D : Demographic data for all counties

6:Initialize model parameters randomly

7:for epoch in range(0, epochs) do

8: // sample from $X,D,Y$ of size b

9: for $(X_{b},D_{b},Y_{bt})$ in $(X,D,Y)$ do

10: // Forward propagation

11: $Y_{bp}=M(X_{b})$

12: //Calculate QuantileLoss

13: $L_{b}=QuantileLoss(Y_{bp},Y_{bt})$

14: //Find association

15: $olsreg=OLS.fit(D_{b},L_{b})$

16: $pvals,betas=olsreg.pvals,olsreg.coef$

17: // additional penalty on loss

18: for $index$ in $|pvals|$ do

19: $pval_{i},beta_{i}=pvals[index],betas[index]$

20: // Get the corresponding demographic percentage column and all rows

21: $D_{b,idx}=D_{b}[:,index]$

22: if $pval_{i}<threshold$ then //this ensures significant association

23: $L_{b}+=L_{b}*|beta_{i}|*D_{b,idx}$

24: end if

25: end for

26: $backpropagate(M,L_{b})$

27: end for

28:end for

29:return TFT

##  4 DemOpts Evaluation Protocol

In this section, we describe a novel protocol to evaluate DemOpts. For that
purpose, we first describe the COVID-19 county case prediction model we use,
and the different de-biasing approaches we evaluate on that prediction model.
Next, we describe the error parity metrics we use to evaluate the fairness of
each prediction model; and finally, we present the approach to analyze whether
DemOpts improves the error parity metrics when compared to other state-of-the-
art de-biasing approaches for regression settings.

### Predictive Model and De-Biasing Approaches

We use the Temporal Fusion Transformer model (TFT) with the conventional
pinball loss function as our baseline model ($TFT_{baseline}$) to predict the
number of COVID-19 county cases for a given day. Input data to the TFT model
include past COVID-19 cases per county, mobility data from SafeGraph and race
and ethnicity data for the county (further details about these datasets are
provided in the next section).

We also train and test another TFT enhanced with the DemOpts de-biasing
method, $TFT_{DemOpts}$, that adjusts the loss computation to attempt to
eliminate or reduce the dependencies between error and race so as to achieve
error parity. In addition, we train and test three more TFTs enhanced with
state-of-the-art de-biasing methods for regression settings namely, individual
fairness $TFT_{Individual}$ (Berk et al. 2017), group fairness $TFT_{Group}$
(Berk et al. 2017), and sufficiency based regularizer $TFT_{Sufficiency}$
(Shah et al. 2022). Individual and group fairness methods calculate penalties
by determining over-estimations across different groups and weighting the loss
by a factor proportional to the over-estimations; while the sufficiency based
regularizer trains a joint model and group-specific networks to achieve fair
predictions. We replicate their methodology and adapt it to the forecasting
setting by keeping TFT as the common network in the training process.

### Measuring Model Fairness

We choose error parity as our fairness metric (Gursoy and Kakadiaris 2022)
with a focus on evaluating whether the distribution of predictive errors at
the county level is independent of county race and ethnicity i.e., prediction
errors are not statistically significantly different across racial and ethnic
groups. To measure the fairness of each of the models $TFT_{baseline}$,
$TFT_{DemOpts}$, $TFT_{Individual}$, $TFT_{Group}$ and $TFT_{Sufficiency}$, we
propose a two-step process.

Table 1: Majority label counts  Majority label | Count
---|---
Asian | 6
Black | 127
Hispanic | 126
White | 2825

Step 1: Associate errors to county race or ethnicity. To carry out the
fairness analysis, we need to associate the PBL error of each county with race
and ethnicity labels. However, that would require access to race-stratified
COVID-19 case data at the county level, unfortunately not available due to
systemic data collection failures during the pandemic (Kader and Smith 2021).
Hence, we propose to associate each county and its error to the majority race:
we label each county with the race or ethnicity that has the highest
population percentage in that county. Following this procedure and using data
from the 2019 U.S. Census, our fairness analysis will consider the following
race and ethnic groups: Asian, Black, Hispanic and White. Table 1 shows the
distribution of U.S. counties into these four racial and ethnic groups, and
Figure 2 shows a color-coded map with the majority racial or ethnic group for
each county. During the fairness analysis, we will refer to majority White
counties as the unprotected group, and majority Black, Hispanic or Asian
counties as the protected groups.

Figure 2: Counties with majority based labels.

In addition, we normalize each county PBL error by 1,000 county population
people. The normalization by county population allows us to scale the errors
appropriately, since higher population counties will have higher case counts
and thus, higher magnitude of error. Normalizing by population fairly compares
error per unit population of one county with another.

$\displaystyle NormPBL(y_{pi},y_{ti})$
$\displaystyle=\frac{1000*PBL(y_{pi},y_{ti})}{pop_{i}}$ (4)

where $y_{i}$ is the ground truth, $y_{ip}$ is the predicted value, and
$pop_{i}$ is the county population.

Step 2: Compute fairness metric. Once PBLs have been associated with racial
and ethnic groups in the U.S., we can compute the error parity i.e., the
fairness metric focused on evaluating whether the prediction errors are
different across race and ethnicity. We propose two metrics to measure the
error parity of COVID-19 county case predictions: hard error parity and soft
error parity. Next, we explain how we implement them and why both are needed.

Hard Error Parity Metric. Model predictions exhibit hard error parity when no
statistical significant difference exists between county case normalized mean
prediction errors (NormPBL) across racial or ethnic groups. In other words,
normalized mean PBL errors across counties of different racial and ethnic
groups are similar and hence not biased by race or ethnicity. To test for the
hard error parity of a prediction model, we propose to run one-way ANOVA
followed by post-hoc Tukey HSD tests between the county normalized mean error
distributions of all racial and ethnic groups. ANOVA tests have been shown to
be an adequate choice even in violation of normality, and in the presence of
unequal sample sizes, like our majority race/ethnic distributions; thus, we
choose this parametric test due to its superior strength (Blanca Mena et al.
2017; Zimmerman 1987).

Rejecting the null hypothesis for ANOVA would point to significantly different
mean error values across some racial or ethnic groups, and to a lack of
perfect hard error parity. The subsequent analysis of the post-hoc Tukey-HSD
test would reveal the pairs of racial and ethnic groups whose mean error
values are significantly different, and their numerical difference. The Tukey
test also highlights the pairs of racial and ethnic groups for which the mean
error is not statistically significantly different, pointing to instances
where hard error parity exists for that model. However, as Table 1 shows, the
number of points in some distributions might not be sufficient to reveal
statistically significant results (Nature 2019). Thus, we also propose a
relaxed definition of error parity: soft error parity.

Soft Error Parity Metric. Instead of measuring the statistical significance of
the relationship between county race labels and county errors, we propose to
use the Accuracy Equity Ratio metric (AER) (Castelnovo et al. 2022). AER
computes the ratio between the errors of the protected and unprotected groups
as follows:

$\displaystyle AER_{pg}$
$\displaystyle=\frac{AvgNormPBL(y_{p},y_{t},pg)}{AvgNormPBL(y_{p},y_{t},unpg)}$
(5)

where subscript $pg$ indicates counties labeled as protected group (Black,
Hispanic or Asian), $unpg$ indicates counties labeled as the unprotected group
(White), and AvgNormPBL is the average of the normalized PBL across all
counties for a given racial group $g$ ($pg$ or $unpg$):

$\displaystyle AvgNormPBL(y_{p},y_{t},g)$ $\displaystyle=\sum_{i\in
c_{g}}\frac{NormPBL(y_{pi},y_{ti})}{\lvert c_{g}\rvert}$ (6)

As defined, the $AER$ metric goes from $0$ to $\infty$. $AER$ values in the
range $0$ to $1$ indicate comparatively lower normalized PBL for protected
groups, which means the model predictions could be biased - have higher errors
- for White majority counties; while $AER$ values larger than one indicate
that the model could be biased against the protected group i.e., the
prediction errors are larger for counties with majority Black, Hispanic or
Asian population. Values close to one indicate parity in error distribution
between the protected group counties and the majority White counties. We claim
that a predictive model achieves soft error parity for a given protected group
when the $AER$ value is close to one, that is, the mean predictive error
between that protected group and the White race is similar.

### DemOpts Over State-of-the-Art

To assess whether DemOpts is a better de-biasing approach than state-of-the-
art methods, we need to compare the error parity metrics of the COVID-19
county case prediction model enhanced with the DemOpts method $TFT_{DemOpts}$
against the error parity metrics of the same prediction model enhanced with
the other de-biasing approaches (individual $TFT_{Individual}$, group
$TFT_{Group}$ or sufficiency $TFT_{Sufficiency}$) as well as with the baseline
COVID-19 county case prediction model without any de-biasing approach,
$TFT_{Baseline}$.

Next we describe how we carry out this analysis for the hard and soft error
parity metrics.

Hard Error Parity Evaluation.

We compute the hard error parity metric for each of the COVID-19 county case
prediction model, using one-way ANOVA and the post-hoc Tukey-HSD test. An
exploration of the statistical significance of the mean error difference for
each pair of racial and ethnic groups will reveal whether applying DemOpts to
the COVID-19 case prediction model produces less instances of significant mean
prediction error differences than any of the other de-biasing methods or the
baseline. In other words, a decrease in the number of significantly different
mean PBL errors between races would point to an achievement of hard error
parity for more racial and ethnic groups that other state-of-the-art de-
biasing approaches, or than the baseline.

Soft Error Parity Evaluation. To assess whether DemOpts applied to a COVID-19
case prediction model has higher soft error parity than any of the other
state-of-the-art de-biasing approaches, we propose to compare the AER values
for each protected race and ethnic group across the five models:
$TFT_{DemOpts}$, $TFT_{Individual}$, $TFT_{Group}$, $TFT_{Sufficiency}$ and
$TFT_{Baseline}$. Since AER values represent the quotient between the
normalized mean prediction errors of a protected race/ethnicity versus White
counties, the model with more AER values closer to one will be the approach
with the highest soft error parity. To measure AER’s distance to one, we
compute the $distance=|1-AER_{race}|$ for each race and ethnic group, which
represents the distance to a perfect soft parity error of $1$. Distances
closer to zero reveal better soft parities i.e., soft parity values closer to
one.

##  5 DemOpts Evaluation Results

We first present the datasets and the models used to train and test the
COVID-19 county case prediction models with and without de-biasing approaches.
We finalize discussing the hard and soft error parity analysis for
$TFT_{DemOpts}$ by comparing it against the other state-of-the-art de-biasing
methods and against the baseline.

### Datasets

In this paper, we train TFT COVID-19 county case prediction models for the
U.S. using COVID-19 case data, as well as mobility and demographic data.
Mobility data has been used by prior work in an attempt to inform case
prediction via human mobility behaviors, under the assumption that the way
people move might have an impact on the spreading of the epidemic. On the
other hand, demographic data either raw from the census, or combined in
different types of vulnerability indices, has also been shown to be helpful in
predicting COVID-19 prevalence, given the fact that COVID-19 has heavily
affected vulnerable populations (Gross et al. 2020).

COVID-19 Case Data. We use the COVID-19 case data compiled by the New York
Times at the county level (Times 2021). We account for delayed reporting, by
using the 7-day daily rolling average of COVID-19 cases (computed as the
average of its current value and 6 prior days) instead of raw counts. As
stated in the Introduction, case numbers might not be reflective of the actual
spread of COVID-19 for specific racial and ethnic groups, and such under-
reporting bias could in turn affect the fairness of the COVID-19 predictions
(Douglas et al. 2021; Del Rios et al. 2022).

(a)

(b)

Figure 3: (a) Case counts per 1000 population for all ethnicities and race.
(b) Case counts per 1000 population using majority based labelling of
counties. Note scale difference in y-axis. Figure 4: Mobility for all
ethnicity and races.

Figure 3(a) reflects the daily COVID-19 reported cases throughout the data
collection period and Figure 3(b) shows the case temporal distribution per
race and ethnicity.

Fairness Method | F(x,y)
---|---
Baseline | 1195.398**
Demopts | 668.769**
Group | 1455.528**
Individual | 1469.698**
Sufficiency | 1195.651**
Table 2:  ANOVA F-test statistic comparing the mean prediction error for each
TFT prediction model: baseline and TFTs enhanced with de-biasing methods. All
tests were significant with p-value $<0.01$ (**) with DF (3,3080).

SafeGraph Mobility Data. SafeGraph open sourced the mobility patterns of smart
phone app users at the onset of the pandemic. These data points are curated by
tracking the movements of millions of pseudonymized users via mobile app SDKs.
Based on the data available, we use the daily O-D (origin-destination) county
to county flows (Kang et al. 2020). O-D flows represent the volume of trips
between pairs of counties across the United States for each day. For O-D
flows, we only use SafeGraph inflow (i.e. the mobility to the county). The
inflow mobility is measured as changes in volumes of flows with respect to a
baseline of normal behavior computed by SafeGraph using mobility data between
the period February 17, 2020 to March 7, 2020. Prior work has shown sampling
bias in mobility datasets revealing that not all races and ethnicities are
equally represented (Coston et al. 2021; Schlosser et al. 2021). It has also
been shown that sampling bias in the mobility data can negatively impact
downstream tasks such as COVID-19 forecasting (Abrar et al. 2023). While the
addition of mobility could potentially help improve the prediction accuracy
and support better decision making, it introduces bias. Our empirical analysis
of DemOpts aims to understand whether the de-biasing method proposed in this
paper can improve the fairness of COVID-19 county case predictive models when
mobility data is used as input into the predictive model. Figure 3(a) depicts
the daily average mobility across all counties in the US throughout the data
collection period.

Race and Ethnicity Data. We retrieve the race and ethnicity data from each
county in the U.S. from the American Community survey (ACS). The ACS survey
collects data annually from all 50 states, Puerto Rico and Washington DC. We
use the population race and ethnicity information for each county, and
consider the following labels: Asian, Black, Hispanic and White.

| Fairness Method | Baseline | DemOpts | Group | Individual | Sufficiency
---|---|---|---|---|---|---
Group1 | Group2 |  |  |  |  |
Asian | Black | -0.118 | 1.327 | -0.202 | -0.126 | -0.119
| Hispanic | -2.302** | -0.771 | -2.659** | -2.507** | -2.297**
| White | -2.064** | -0.968 | -2.515** | -2.517** | -2.061**
Black | Hispanic | -2.184** | -2.098** | -2.457** | -2.381** | -2.178**
| White | -1.946** | -2.295** | -2.313** | -2.391** | -1.942**
Hispanic | White | 0.238 | -0.197 | 0.144 | -0.01 | 0.236
Table 3: Hard error parity analysis. Each number represents the difference
between the mean normalized PBL loss for each pair of racial and ethnic
groups, and its statistical significance i.e., whether the difference is
statistically significant or not (with ** p-value $<0.01$, * p-value $<0.1$).
Bolded numbers represent the instances where DemOpts has removed a significant
difference in mean PBL errors, improving over the baseline and all the other
de-biasing methods. DemOpts achieves hard error parity for Asian counties.

### COVID-19 Case Prediction Models

We use COVID-19 case data as well as SafeGraph mobility data from March 18,
2020 to November 30, 2020 for the training (207 days) and testing (49 days) of
the TFT COVID-19 county case prediction models. The forecast task is the
prediction of the number of COVID-19 cases for a given county for day $X+1$ to
$X+49$ i.e., the following 3 months (long-term forecasting with lookahead
values from $1$ to $49$). Specifically, we train and test: (1) the
$TFT_{baseline}$, a TFT prediction model without a de-biasing method; (2) the
$TFT_{Individual}$, $TFT_{Group}$ and $TFT_{Sufficiency}$, TFT prediction
models with state-of-the-art de-biasing methods and (3) $TFT_{DemOpts}$, a TFT
prediction model enhanced with our proposed de-biasing method. All five models
are trained and tested for exactly the same temporal range; and all are
implemented using the pytorch-forecasting library. Although COVID-19 county
case data as well as mobility data are available for longer periods of time,
we decided to limit the period of analysis to a time before COVID-19 vaccines
were available, given that after that event, research has revealed a very
unclear relationship between mobility data and post-vaccines COVID-19 case
volumes (Gatalo et al. 2021). Once all models have been trained, we use the
prediction errors (PBL) per racial and ethnic group to analyze and compare
their hard and soft error parity.

Fairness Method | Baseline | Demopts | Group | Individual | Sufficiency
---|---|---|---|---|---
Group |  |  |  |  |
Asian | 0.811 | 0.248 | 0.842 | 0.850 | 0.811
Black | 0.764 | 0.588 | 0.774 | 0.807 | 0.764
Hispanic | 0.093 | 0.051 | 0.048 | 0.003 | 0.093
Table 4: Soft error parity analysis. Each number represents the distance
($|1-AER_{race}|$) for each protected group and de-biasing method. For each
protected race/ethnicity, distances closer to zero represent higher soft error
parity (signaled in bold font). $TFT_{DemOpts}$ achieves the highest soft
error parity for two of the three protected races under study.
$TFT_{Individual}$ achieves the best soft error parity for the Hispanic
counties when compared to prediction errors in White counties.

### Hard Error Parity Analysis

ANOVA tests of the normalized mean PBL error distributions across race and
ethnic groups for each de-biasing approach were all significant, pointing to a
dependency between race and the normalized prediction errors. Table 2 shows
the F-statistic and test significance for each of the prediction models with
and without de-biasing approaches. The significant ANOVA tests reveal that
perfect hard error parity is not achieved by any of the de-biasing methods. In
other words, for some racial and ethnic groups there exist statistically
significant differences between their mean PBL prediction errors and those of
other racial and ethnic groups; and this effect happens for the
$TFT_{baseline}$ as well as across all the other predictive models enhanced
with a de-biasing approach.

Nevertheless, post-hoc Tukey-HSD tests revealed interesting nuanced results,
showing significant differences in errors only between specific pairs of
racial and ethnic groups.

Table 3 shows the post-hoc Tukey-HSD test results for each COVID-19 case
predictive model: the baseline and each of the four models enhanced with a de-
biasing approach. Each row represents the output of the post-hoc test i.e.,
the difference between the normalized mean PBL error of Group 1 and Group2
i.e., $NormPBL_{Group1}-NormPBL_{Group2}$. If the difference is positive, it
means that the normalized mean predictive error is higher for Group 1; if the
difference is negative, the normalized PBL is higher for Group 2. The
asterisks indicate whether the difference is statistically significant or not.

The first relevant observation in looking at the table is that the baseline
model, focused on predicting COVID-19 county cases with no de-biasing
approach, is highly biased, with statistically significant differences between
the mean normalized errors across all pairs of races, except for the
comparison between Asian and Black counties as well as Hispanic and White
counties, for which there is no statistically significant difference between
the prediction errors. These results reveal that there is no racial group or
ethnicity that achieves hard error parity, and motivates our exploration of
whether state-of-the-art de-biasing methods or our proposed DemOpts can
improve the hard error parity results of the baseline model.

Looking at Table 3, we can observe that predictive models enhanced with the
Individual, Group, or Sufficiency de-biasing methods do not improve the hard
error parity over the baseline. In fact, each pair of racial and ethnic groups
whose prediction error distributions are significantly different for the
baseline (rows with asterisks in the Baseline column), remain significantly
different for the Individual, Group and Sufficiency de-biasing methods (rows
with asterisks in the Individual, Group and Sufficiency columns). Looking at
the significant mean PBL differences between racial and ethnic groups for the
baseline and the state-of-the art de-biasing models, we observe that all
coefficients have similar values, signaling similar significant mean PBL
differences between racial and ethnic groups (with values between $1.942$ and
$2.659$ error cases by 1,000 population). The sign of the coefficients reveals
higher mean PBL errors for Hispanic and White counties when compared to Asian
counties or Black counties; and higher mean PBL errors for White counties when
compared to Hispanic counties across all five models (baseline and four de-
biasing approaches). For example, Hispanic and White counties have mean
prediction errors $2.302$ and $2.064$ higher, when compared to Asian counties
and while using the baseline model; and Hispanic and White counties have
errors $2.184$ and $1.946$ higher when compared to Black counties and while
using the baseline model.

Fairness Method | Baseline | Demopts | Group | Individual | Sufficiency
---|---|---|---|---|---
Group |  |  |  |  |
Asian | 0.482 | 2.938 | 0.472 | 0.444 | 0.479
Black | 0.600 | 1.611 | 0.674 | 0.570 | 0.598
Hispanic | 2.784 | 3.709 | 3.131 | 2.951 | 2.776
White | 2.546 | 3.906 | 2.987 | 2.961 | 2.540
Table 5: Average prediction error (PBL) for each racial and ethnic group and
for each TFT COVID-19 county case prediction model: baseline model, and models
enhanced with a de-biasing method (Individual, Group, Sufficiency and
DemOpts). DemOpts achieves fairness by increasing mean errors for the Asian
and Black groups.

Notably, all predictive models including the baseline and those enhanced with
a de-biasing method ($TFT_{DemOpts}$, $TFT_{Group}$, $TFT_{Individual}$ and
$TFT_{Sufficiency}$) achieve hard error parity between Asian and Black
counties and between Hispanic and White counties i.e., the mean error
difference between these counties is not significant. But even more
interesting is the fact that DemOpts is the only de-biasing method that
achieves hard error parity in more cases than the baseline, effectively
removing some of the associations between race and ethnicity and the
normalized mean error distribution (PBL). Specifically, DemOpts removes the
significant difference between the prediction errors of Asian and White
counties, and of Asian and Hispanic counties (see bolded values in the Table),
effectively achieving hard error parity for Asian counties i.e., the mean PBL
in Asian counties is always similar to the mean error in counties of all the
other racial and ethnic groups. And these DemOpts improvements take place
while maintaining the $TFT_{baseline}$ hard error parity between Asian and
Black and Hispanic and White counties - also present in the other three de-
biasing methods. In other words, DemOpts improves the hard error parity of
COVID-19 county case predictions for two more racial and ethnic pairs than any
of the other de-biasing methods.

Finally, when looking specifically into the hard error parity between
protected (Asian, Black and Hispanic) and unprotected groups (White), DemOpts
achieves hard error parity for Asian and Hispanic i.e., their mean prediction
errors are not significantly different with respect to the White race; while
the baseline and the other three de-biasing methods only achieve hard error
parity for the Hispanic group when compared to White. These findings with
respect to the White group lead us to evaluate the soft error parity of the
different models, to understand, for example, if DemOpts achieves the best
soft error parity for the Black group (since hard error parity was not
achieved), or to see if DemOpts has better soft error parity than other de-
biasing methods for Asian or Hispanic groups. Next, we explore the soft error
parity metric for the TFT baseline and for all TFT models enhanced with de-
biasing approaches.

### Soft Error Parity Analysis

Table 4 shows the distance to the perfect soft error parity for each of the
de-biasing approaches and across all protected racial and ethnic groups. As we
can observe, DemOpts has the smallest values - closest distances to perfect
soft error parity - for Asian and Black counties; while the Individual de-
biasing method almost achieves perfect soft error parity for the Hispanic
counties. In other words, the errors for Asian and Black counties are the
closest to errors in White counties for the proposed DemOpts method, while the
Individual de-biasing model achieves errors for Hispanic counties that are the
closest to the White group. In addition, it is important to highlight that the
Group and Sufficiency de-biasing methods achieve soft error parities that are
similar to the $TFT_{baseline}$ which is not enhanced with any de-biasing
method. Overall, these results reveal that DemOpts is the de-biasing approach
that improves the most the soft error parity of COVID-19 county case
prediction models, with errors for Asian and Black counties being the closest
to errors in White counties; while the Individual de-biasing method achieves
the closest errors to the White race for Hispanic counties only.

### Why is DemOpts better?

The results have shown that DemOpts is the only de-biasing approach to achieve
both hard or soft error parity for all three racial minority groups when
compared to the White race. In an attempt to understand why DemOpts succeeds
in increasing both hard and soft error parity in the context of COVID-19
county case predictions, and when compared to other de-biasing methods, we
computed the average PBL for each racial and ethnic group and for each
predictive model enhanced, or not, with a de-biasing method (see Table 5).

We can observe that DemOpts achieves better hard and soft error parity metrics
because it considerably increases the errors for Asian and Black counties with
respect to the baseline, until the differences with Hispanic and White are
made not statistically significant (hard error parity) or closer to the White
mean errors (soft error parity). This result also points to another
interesting insight: the fact that DemOpts’ optimization could not decrease
prediction errors while trying to improve fairness, when fairness is measured
via statistical significance, showing a fairness-accuracy trade-off that has
been reported previously in the literature (Kim, Chen, and Talwalkar 2020).
Finally, it is also important to clarify that, in practice, the prediction
error increases brought about by DemOpts are not that large, with increases
between $1-2.5$ error cases by 1,000 people. We posit that these small
increases are acceptable if that is the requirement to guarantee hard and soft
error parity across protected and unprotected racial and ethnic groups.

##  6 Conclusion

In the past four years, researchers have worked profusely on the creation of
accurate COVID-19 case prediction models using not only historical COVID-19
cases but also complementary data such as human mobility or socio-demographic
information. However, there exists prior work showing that the accuracy of
COVID-19 predictions can depend on various social determinants, including race
and ethnicity, income, or age, revealing worse performance for protected
attributes and pointing to a lack on COVID-19 predictive fairness that could
affect resource allocation and decision making.

In this paper, we show that state of the art architectures in COVID-19 case
predictions (TFT models) incur in unfair prediction error distributions, and
we design a novel de-biasing approach to increase the fairness of the
predictions in the context of COVID-19 county case predictions. The new
proposed de-biasing approach, DemOpts, modifies the loss function in deep
learning models to reduce the dependencies between error distributions and
racial and ethnic labels. Our results show that DemOpts improves the most both
the hard and soft error parity of COVID-19 county case predictions when
compared to state-of-the-art de-biasing methods.

## References

  * Abrar et al. (2023) Abrar, S. M.; Awasthi, N.; Smolyak, D.; and Frias-Martinez, V. 2023.  Analysis of performance improvements and bias associated with the use of human mobility data in COVID-19 case prediction models.  _ACM Journal on Computing and Sustainable Societies_.
  * Agarwal, Dudík, and Wu (2019) Agarwal, A.; Dudík, M.; and Wu, Z. S. 2019.  Fair Regression: Quantitative Definitions and Reduction-based Algorithms.
  * Albani et al. (2021) Albani, V.; Loria, J.; Massad, E.; and Zubelli, J. 2021.  COVID-19 underreporting and its impact on vaccination strategies.  _BMC Infectious Diseases_ , 21: 1–13.
  * Angulo, Finelli, and Swerdlow (2021) Angulo, F. J.; Finelli, L.; and Swerdlow, D. L. 2021.  Estimation of US SARS-CoV-2 infections, symptomatic infections, hospitalizations, and deaths using seroprevalence surveys.  _JAMA network open_ , 4(1): e2033706–e2033706.
  * Apple (2022) Apple. 2022.  COVID-19 Mobility Trends Reports.
  * Arik et al. (2020a) Arik, S.; Li, C.-L.; Yoon, J.; Sinha, R.; Epshteyn, A.; Le, L.; Menon, V.; Singh, S.; Zhang, L.; Nikoltchev, M.; et al. 2020a.  Interpretable sequence learning for COVID-19 forecasting.  _Advances in Neural Information Processing Systems_ , 33: 18807–18818.
  * Arik et al. (2020b) Arik, S.; Li, C.-L.; Yoon, J.; Sinha, R.; Epshteyn, A.; Le, L.; Menon, V.; Singh, S.; Zhang, L.; Nikoltchev, M.; et al. 2020b.  Interpretable sequence learning for COVID-19 forecasting.  _Advances in Neural Information Processing Systems_ , 33: 18807–18818.
  * Badr and Gardner (2021) Badr, H. S.; and Gardner, L. M. 2021.  Limitations of using mobile phone data to model COVID-19 transmission in the USA.  _The Lancet Infectious Diseases_ , 21(5): e113.
  * Bengtsson et al. (2015) Bengtsson, L.; Gaudart, J.; Lu, X.; Moore, S.; Wetter, E.; Sallah, K.; Rebaudet, S.; and Piarroux, R. 2015.  Using mobile phone data to predict the spatial spread of cholera.  _Scientific reports_ , 5(1): 1–5.
  * Berk et al. (2017) Berk, R.; Heidari, H.; Jabbari, S.; Joseph, M.; Kearns, M.; Morgenstern, J.; Neel, S.; and Roth, A. 2017.  A convex framework for fair regression.  _arXiv preprint arXiv:1706.02409_.
  * Blanca Mena et al. (2017) Blanca Mena, M. J.; Alarcón Postigo, R.; Arnau Gras, J.; Bono Cabré, R.; and Bendayan, R. 2017.  Non-normal data: Is ANOVA still a valid option?  _Psicothema, 2017, vol. 29, num. 4, p. 552-557_.
  * Brunet et al. (2018) Brunet, M.-E.; Alkalay-Houlihan, C.; Anderson, A.; and Zemel, R. 2018.  Understanding the Origins of Bias in Word Embeddings.
  * Calmon et al. (2017) Calmon, F. P.; Wei, D.; Ramamurthy, K. N.; and Varshney, K. R. 2017.  Optimized Data Pre-Processing for Discrimination Prevention.
  * Castelnovo et al. (2022) Castelnovo, A.; Crupi, R.; Greco, G.; Regoli, D.; Penco, I. G.; and Cosentini, A. C. 2022.  A clarification of the nuances in the fairness metrics landscape.  _Scientific Reports_ , 12(1): 4209.
  * CDC (2020) CDC. 2020.  Forecast Hub.  https://covid19forecasthub.org/.
  * CDC (2023) CDC. 2023.  Health Disparities.  Last accessed January 2024.
  * Chiang, Liu, and Mohler (2020) Chiang, W.-H.; Liu, X.; and Mohler, G. 2020.  Hawkes process modeling of COVID-19 with mobility leading indicators and spatial covariates (preprint).
  * Coston et al. (2021) Coston, A.; Guha, N.; Ouyang, D.; Lu, L.; Chouldechova, A.; and Ho, D. E. 2021.  Leveraging Administrative Data for Bias Audits: Assessing Disparate Coverage with Mobility Data for COVID-19 Policy.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, 173–184. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450383097.
  * Das and Dooley (2023) Das, R.; and Dooley, S. 2023.  Fairer and More Accurate Tabular Models Through NAS.
  * Del Rios et al. (2022) Del Rios, M.; Puente, S.; Vergara-Rodriguez, P.; and Sugrue, N. 2022.  Invisibilidad de los latinos en la pandemia.  _AMA Journal of Ethics_ , 289–295.
  * Devlin et al. (2018) Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _CoRR_ , abs/1810.04805.
  * Dong, Du, and Gardner (2020) Dong, E.; Du, H.; and Gardner, L. 2020.  An interactive web-based dashboard to track COVID-19 in real time.  _The Lancet Infectious Diseases_ , 20(5): 533–534.
  * Dosovitskiy et al. (2021) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  arXiv:2010.11929.
  * Douglas et al. (2021) Douglas, M. D.; Respress, E.; Gaglioti, A. H.; Li, C.; Blount, M. A.; Hopkins, J.; Baltrus, P. T.; Willock, R. J.; Caplan, L. S.; Dawes, D. E.; et al. 2021.  Variation in reporting of the race and ethnicity of COVID-19 cases and deaths across US states: April 12, 2020, and November 9, 2020.  _American Journal of Public Health_ , 111(6): 1141–1148.
  * Duan et al. (2017) Duan, L.; Hu, T.; Cheng, E.; Zhu, J.; and Gao, C. 2017.  Deep convolutional neural networks for spatiotemporal crime prediction.  In _Proceedings of the International Conference on Information and Knowledge Engineering (IKE)_ , 61–67. csce.ucmss.com.
  * Erfani and Frias-Martinez (2023) Erfani, A.; and Frias-Martinez, V. 2023.  A fairness assessment of mobility-based COVID-19 case prediction models.
  * Estiri et al. (2022) Estiri, H.; Strasser, Z. H.; Rashidian, S.; Klann, J. G.; Wagholikar, K. B.; McCoy Jr, T. H.; and Murphy, S. N. 2022.  An objective framework for evaluating unrecognized bias in medical AI models predicting COVID-19 outcomes.  _Journal of the American Medical Informatics Association_ , 29(8): 1334–1341.
  * Fitzsimons et al. (2019) Fitzsimons, J.; Al Ali, A.; Osborne, M.; and Roberts, S. 2019.  A General Framework for Fair Regression.
  * for Disease Control and Prevention (2023) for Disease Control, C.; and Prevention. 2023.  COVID-19 Forecasting and Mathematical Modeling.  Accessed: 2023-12-25.
  * Frias-Martinez et al. (2012) Frias-Martinez, V.; Soto, V.; Virseda, J.; and Frias-Martinez, E. 2012.  Computing cost-effective census maps from cell phone traces.  In _Workshop on pervasive urban applications_.
  * Frias-Martinez and Virseda (2013) Frias-Martinez, V.; and Virseda, J. 2013.  Cell phone analytics: Scaling human behavior studies into the millions.  _Information Technologies & International Development_, 9(2): pp–35.
  * Frias-Martinez, Virseda, and Frias-Martinez (2010) Frias-Martinez, V.; Virseda, J.; and Frias-Martinez, E. 2010.  Socio-economic levels and human mobility.  In _Qual meets quant workshop-QMQ_ , 1–6.
  * Frias-Martinez, Virseda, and Gomero (2012) Frias-Martinez, V.; Virseda, J.; and Gomero, A. 2012.  Mobilizing education: evaluation of a mobile learning tool in a low-income school.  In _Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services_ , 441–450.
  * Fu et al. (2018) Fu, C.; McKenzie, G.; Frias-Martinez, V.; and Stewart, K. 2018.  Identifying spatiotemporal urban activities through linguistic signatures.  _Computers, Environment and Urban Systems_ , 72: 25–37.
  * Galasso, Cao, and Hochberg (2022) Galasso, J.; Cao, D. M.; and Hochberg, R. 2022.  A random forest model for forecasting regional COVID-19 cases utilizing reproduction number estimates and demographic data.  _Chaos, Solitons & Fractals_, 156: 111779.
  * Gatalo et al. (2021) Gatalo, O.; Tseng, K.; Hamilton, A.; Lin, G.; and Klein, E. 2021.  Associations between phone mobility data and COVID-19 cases.  _The Lancet Infectious Diseases_ , 21(5): e111.
  * Ghurye, Krings, and Frias-Martinez (2016) Ghurye, J.; Krings, G.; and Frias-Martinez, V. 2016.  A framework to model human behavior at large scale during natural disasters.  In _2016 17th IEEE International Conference on Mobile Data Management (MDM)_ , volume 1, 18–27. IEEE.
  * Google (2022) Google. 2022.  COVID-19 Community Mobility Reports.
  * Gross et al. (2020) Gross, C. P.; Essien, U. R.; Pasha, S.; Gross, J. R.; Wang, S.-y.; and Nunez-Smith, M. 2020.  Racial and ethnic disparities in population-level Covid-19 mortality.  _Journal of general internal medicine_ , 35: 3097–3099.
  * Gursoy and Kakadiaris (2022) Gursoy, F.; and Kakadiaris, I. A. 2022.  Error Parity Fairness: Testing for Group Fairness in Regression Tasks.  _arXiv preprint arXiv:2208.08279_.
  * Hernandez et al. (2017) Hernandez, M.; Hong, L.; Frias-Martinez, V.; Whitby, A.; and Frias-Martinez, E. 2017.  Estimating poverty using cell phone data: evidence from Guatemala.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S.; and Schmidhuber, J. 1997.  Long short-term memory.  _Neural computation_ , 9(8): 1735–1780.
  * Hong, Frias-Martinez, and Frias-Martinez (2016) Hong, L.; Frias-Martinez, E.; and Frias-Martinez, V. 2016.  Topic models to infer socio-economic maps.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 30.
  * Hong and Frias-Martinez (2020) Hong, L.; and Frias-Martinez, V. 2020.  Modeling and predicting evacuation flows during hurricane Irma.  _EPJ Data Science_ , 9(1): 29.
  * Hong et al. (2017) Hong, L.; Fu, C.; Torrens, P.; and Frias-Martinez, V. 2017.  Understanding citizens’ and local governments’ digital communications during natural disasters: the case of snowstorms.  In _Proceedings of the 2017 ACM on web science conference_ , 141–150.
  * Isaacman, Frias-Martinez, and Frias-Martinez (2018) Isaacman, S.; Frias-Martinez, V.; and Frias-Martinez, E. 2018.  Modeling human migration patterns during drought conditions in La Guajira, Colombia.  In _Proceedings of the 1st ACM SIGCAS conference on computing and sustainable societies_ , 1–9.
  * Jagodnik et al. (2020) Jagodnik, K. M.; Ray, F.; Giorgi, F. M.; and Lachmann, A. 2020.  Correcting under-reported COVID-19 case numbers: estimating the true scale of the pandemic.  _medRxiv_ , 2020–03.
  * Kader and Smith (2021) Kader, F.; and Smith, C. L. 2021.  Participatory approaches to addressing missing COVID-19 race and ethnicity data.  _International Journal of Environmental Research and Public Health_ , 18(12): 6559.
  * Kang et al. (2020) Kang, Y.; Gao, S.; Liang, Y.; Li, M.; Rao, J.; and Kruse, J. 2020.  Multiscale dynamic human mobility flow dataset in the US during the COVID-19 epidemic.  _Scientific data_ , 7(1): 390.
  * Kim, Chen, and Talwalkar (2020) Kim, J. S.; Chen, J.; and Talwalkar, A. 2020.  FACT: A diagnostic for group fairness trade-offs.  In _International Conference on Machine Learning_ , 5264–5274. PMLR.
  * Kirillov et al. (2023) Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A. C.; Lo, W.-Y.; Dollár, P.; and Girshick, R. 2023.  Segment Anything.  arXiv:2304.02643.
  * Labs (2023) Labs, D. 2023.  Mobility changes in response to COVID-19.
  * Latif et al. (2023) Latif, S.; Zaidi, A.; Cuayahuitl, H.; Shamshad, F.; Shoukat, M.; and Qadir, J. 2023.  Transformers in Speech Processing: A Survey.  arXiv:2303.11607.
  * Le et al. (2020a) Le, M.; Ibrahim, M.; Sagun, L.; Lacroix, T.; and Nickel, M. 2020a.  Neural relational autoregression for high-resolution COVID-19 forecasting.  _Facebook AI Research_.
  * Le et al. (2020b) Le, M.; Ibrahim, M.; Sagun, L.; Lacroix, T.; and Nickel, M. 2020b.  Neural relational autoregression for high-resolution COVID-19 forecasting.  _Facebook AI Research_.
  * Li et al. (2019) Li, S.; Jin, X.; Xuan, Y.; Zhou, X.; Chen, W.; Wang, Y.-X.; and Yan, X. 2019.  Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting.  In Wallach, H.; Larochelle, H.; Beygelzimer, A.; d'Alché-Buc, F.; Fox, E.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Lim et al. (2021a) Lim, B.; Ark, S.; Loeff, N.; and Pfister, T. 2021a.  Temporal fusion transformers for interpretable multi-horizon time series forecasting.  _International Journal of Forecasting_ , 37(4): 1748–1764.
  * Lim et al. (2021b) Lim, B.; Arık, S. O.; Loeff, N.; and Pfister, T. 2021b.  Temporal Fusion Transformers for interpretable multi-horizon time series forecasting.  _International Journal of Forecasting_ , 37(4): 1748–1764.
  * Liu et al. (2021) Liu, Q.; Chen, B.; Guo, J.; Lin, Z.; and Lou, J. 2021.  TAPEX: Table Pre-training via Learning a Neural SQL Executor.  _CoRR_ , abs/2107.07653.
  * Liu et al. (2022) Liu, S.; Yu, H.; Liao, C.; Li, J.; Lin, W.; Liu, A. X.; and Dustdar, S. 2022.  Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting.  In _International Conference on Learning Representations_.
  * Lucas, Vahedi, and Karimzadeh (2023a) Lucas, B.; Vahedi, B.; and Karimzadeh, M. 2023a.  A spatiotemporal machine learning approach to forecasting COVID-19 incidence at the county level in the USA.  _International Journal of Data Science and Analytics_ , 15(3): 247–266.
  * Lucas, Vahedi, and Karimzadeh (2023b) Lucas, B.; Vahedi, B.; and Karimzadeh, M. 2023b.  A spatiotemporal machine learning approach to forecasting COVID-19 incidence at the county level in the USA.  _International Journal of Data Science and Analytics_ , 15(3): 247–266.
  * Meraihi et al. (2022) Meraihi, Y.; Gabis, A. B.; Mirjalili, S.; Ramdane-Cherif, A.; and Alsaadi, F. E. 2022.  Machine learning-based research for covid-19 detection, diagnosis, and prediction: A survey.  _SN computer science_ , 3(4): 286.
  * Nature (2019) Nature. 2019.  5 tips for dealing with non-significant results.  Accessed: 2024-1-12.
  * Nie et al. (2023) Nie, Y.; H. Nguyen, N.; Sinthong, P.; and Kalagnanam, J. 2023.  A Time Series is Worth 64 Words: Long-term Forecasting with Transformers.  In _International Conference on Learning Representations_.
  * Pei and Shaman (2020) Pei, S.; and Shaman, J. 2020.  Initial simulation of SARS-CoV2 spread and intervention effects in the continental US.  _MedRxiv_ , 2020–03.
  * Radford et al. (2018) Radford, A.; Narasimhan, K.; Salimans, T.; Sutskever, I.; et al. 2018.  Improving language understanding by generative pre-training.
  * Rubio et al. (2010) Rubio, A.; Frias-Martinez, V.; Frias-Martinez, E.; and Oliver, N. 2010.  Human mobility in advanced and developing economies: A comparative analysis.  In _2010 AAAI Spring Symposium Series_.
  * Schlosser et al. (2021) Schlosser, F.; Sekara, V.; Brockmann, D.; and Garcia-Herranz, M. 2021.  Biases in human mobility data impact epidemic modeling.
  * Shah et al. (2022) Shah, A.; Bu, Y.; Lee, J. K.; Das, S.; Panda, R.; Sattigeri, P.; and Wornell, G. W. 2022.  Selective Regression under Fairness Criteria.  In Chaudhuri, K.; Jegelka, S.; Song, L.; Szepesvari, C.; Niu, G.; and Sabato, S., eds., _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , 19598–19615. PMLR.
  * Smittenaar et al. (2021) Smittenaar, P.; Stewart, N.; Sutermaster, S.; Coome, L.; Dibner-Dunlap, A.; Jain, M.; Caplan, Y.; Campigotto, C.; and Sgaier, S. K. 2021.  A COVID-19 Community Vulnerability Index to drive precision policy in the US.  _medRxiv_.
  * Times (2021) Times, T. N. Y. 2021.  Coronavirus (Covid-19) Data in the United States.  https://github.com/nytimes/covid-19-data.  Accessed: 2023-12-15.
  * Tsai et al. (2022) Tsai, T. C.; Arik, S.; Jacobson, B. H.; Yoon, J.; Yoder, N.; Sava, D.; Mitchell, M.; Graham, G.; and Pfister, T. 2022.  Algorithmic fairness in pandemic forecasting: lessons from COVID-19.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention Is All You Need.
  * Velickovic et al. (2017) Velickovic, P.; Cucurull, G.; Casanova, A.; Romero, A.; Lio, P.; Bengio, Y.; et al. 2017.  Graph attention networks.  _stat_ , 1050(20): 10–48550.
  * Vieira et al. (2010) Vieira, M. R.; Frias-Martinez, E.; Bakalov, P.; Frias-Martinez, V.; and Tsotras, V. J. 2010.  Querying spatio-temporal patterns in mobile phone-call databases.  In _2010 Eleventh International Conference on Mobile Data Management_ , 239–248. IEEE.
  * Wang and Singh (2023) Wang, Y.; and Singh, L. 2023.  Mitigating demographic bias of machine learning models on social media.
  * Wang et al. (2020) Wang, Z.; Dong, H.; Jia, R.; Li, J.; Fu, Z.; Han, S.; and Zhang, D. 2020.  Structure-aware Pre-training for Table Understanding with Tree-based Transformers.  _CoRR_ , abs/2010.12537.
  * Wesolowski et al. (2012) Wesolowski, A.; Eagle, N.; Tatem, A. J.; Smith, D. L.; Noor, A. M.; Snow, R. W.; and Buckee, C. O. 2012.  Quantifying the impact of human mobility on malaria.  _Science_ , 338(6104): 267–270.
  * Wu et al. (2021) Wu, H.; Xu, J.; Wang, J.; and Long, M. 2021.  Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting.  _CoRR_ , abs/2106.13008.
  * Wu et al. (2022) Wu, J.; Abrar, S. M.; Awasthi, N.; Frias-Martinez, E.; and Frias-Martinez, V. 2022.  Enhancing short-term crime prediction with human mobility flows and deep learning architectures.  _EPJ Data Science_ , 11(1): 53.
  * Wu et al. (2023) Wu, J.; Abrar, S. M.; Awasthi, N.; and Frias-Martinez, V. 2023.  Auditing the fairness of place-based crime prediction models implemented with deep learning approaches.  _Computers, Environment and Urban Systems_ , 102: 101967.
  * Wu, Frias-Martinez, and Frias-Martinez (2021) Wu, J.; Frias-Martinez, E.; and Frias-Martinez, V. 2021.  Spatial sensitivity analysis for urban hotspots using cell phone traces.  _Environment and Planning B: Urban Analytics and City Science_.
  * Yan, Seto, and Apostoloff (2022) Yan, B.; Seto, S.; and Apostoloff, N. 2022.  FORML: Learning to Reweight Data for Fairness.
  * Yang, Soltan, and Clifton (2022) Yang, J.; Soltan, A. A. S.; and Clifton, D. A. 2022.  Algorithmic Fairness and Bias Mitigation for Clinical Machine Learning: A New Utility for Deep Reinforcement Learning.  _medRxiv_.
  * Yin et al. (2020) Yin, P.; Neubig, G.; tau Yih, W.; and Riedel, S. 2020.  TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data.  In _Annual Conference of the Association for Computational Linguistics (ACL)_.
  * Yu, Yin, and Zhu (2018a) Yu, B.; Yin, H.; and Zhu, Z. 2018a.  Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_ , 3634–3640. International Joint Conferences on Artificial Intelligence Organization.
  * Yu, Yin, and Zhu (2018b) Yu, B.; Yin, H.; and Zhu, Z. 2018b.  Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting.(2018).
  * Zhang-James et al. (2021) Zhang-James, Y.; Hess, J.; Salkin, A.; Wang, D.; Chen, S.; Winkelstein, P.; Morley, C. P.; and Faraone, S. V. 2021.  A seq2seq model to forecast the COVID-19 cases, deaths and reproductive R numbers in US counties.  _Research Square_.
  * Zhou et al. (2020) Zhou, H.; Zhang, S.; Peng, J.; Zhang, S.; Li, J.; Xiong, H.; and Zhang, W. 2020.  Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting.  _CoRR_ , abs/2012.07436.
  * Zhou et al. (2022) Zhou, T.; Ma, Z.; Wen, Q.; Wang, X.; Sun, L.; and Jin, R. 2022.  FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting.  _CoRR_ , abs/2201.12740.
  * Zimmerman (1987) Zimmerman, D. W. 1987.  Comparative power of Student t test and Mann-Whitney U test for unequal sample sizes and variances.  _The Journal of Experimental Educational_ , 171–174.
  * Zou et al. (2020) Zou, D.; Wang, L.; Xu, P.; Chen, J.; Zhang, W.; and Gu, Q. 2020.  Epidemic model guided machine learning for COVID-19 forecasts in the United States.  _MedRxiv_ , 2020–05.