# Addressing Racial Bias in Facial Emotion Recognition

Alex Fan  Xingshuo Xiao  Peter Washington

###### Abstract

Fairness in deep learning models trained with high-dimensional inputs and
subjective labels remains a complex and understudied area. Facial emotion
recognition, a domain where datasets are often racially imbalanced, can lead
to models that yield disparate outcomes across racial groups. This study
focuses on analyzing racial bias by sub-sampling training sets with varied
racial distributions and assessing test performance across these simulations.
Our findings indicate that smaller datasets with posed faces improve on both
fairness and performance metrics as the simulations approach racial balance.
Notably, the F1-score increases by $27.2\%$ points, and demographic parity
increases by $15.7\%$ points on average across the simulations. However, in
larger datasets with greater facial variation, fairness metrics generally
remain constant, suggesting that racial balance by itself is insufficient to
achieve parity in test performance across different racial groups.

Facial Emotion Recognition, AffectNet, CAFE, Racial Bias, Bias Simulation

##  1 Introduction

Emotion recognition, commonly referred to as facial expression recognition
(FER), encompasses the identification and analysis of facial expressions
displayed by individuals in images or videos. This complex procedure consists
of three key stages: face detection, feature extraction, and emotion
classification. (Ko, 2018). FER finds extensive utility across various domains
including human-computer interaction (HCI) (Picard, 1999), media analytics
(Zhao et al., 2019), robotics (Tao & Tan, 2005), and health informatics (Voss
et al., 2019; Washington et al., 2022).

Historically, automatic emotion recognition predominantly relied on the
extraction of domain-specific features such as facial action units (Hamm et
al., 2011). However, with the rapid progression of machine learning (ML) and
deep learning (DL) techniques, deep neural networks (DNNs) have emerged as a
prominent approach for developing facial emotion recognition models. Such
models necessitate expansive datasets to ensure robustness and accuracy in
their predictions. In recent years, researchers have proposed DL models which
leverage more expansive datasets for training (Li & Deng, 2018). Despite the
impressive achievements of deep learning methods in FER, a significant
challenge arises from the presence of racial bias in such models. This issue,
which is well documented in existing literature (Chen & Joo, 2021b; Domnich &
Anbarjafari, 2021; Sham et al., 2023b; Xu et al., 2020b), necessitates
immediate attention to mitigate discriminatory outcomes and to provide
equitable opportunities for individuals of diverse ethnicities and skin
colors. Addressing and rectifying the biases within DNNs is of paramount
importance for real-world translation of such models.

Biases within DNNs can primarily be attributed to two fundamental sources: the
training data and the algorithms themselves (Mehrabi et al., 2022b). Given
that models learn from input data, any biases present within the underlying
datasets are inherently ingrained within the learning process of the
algorithms. Furthermore, the design of feature extraction processes for these
models may introduce biases that disproportionately affect different racial
groups. An illustrative example is the consideration of skin color as a
learned feature extracted during the deep learning process, which can
ultimately lead to unfair predictions.

To address the issue of racial bias in FER datasets, we conduct a simulation
study on AffectNet and CAFE datasets (Mollahosseini et al., 2017; LoBue &
Thrasher, 2015). In each simulation, we select a specific race as the
simulated race, and ensure other races equal representations. We train the FER
model using sub-sampled data with varying proportions of the simulated race,
measuring the accuracy, F1-scores, and fairness metrics. We find that the
racial balance of the training set has some influence on the test race-
specific F1-score, but mitigating balance alone is insufficient to address
other types of bias such as annotator bias.

##  2 Related Works

###  2.1 Deep-learning-based Emotion Recognition

Deep learning models serve as the foundation for each stage of FER (Ko, 2018).
Training DNNs for FER requires the utilization of diverse datasets,
encompassing varying numbers of labels and data types (Mollahosseini et al.,
2017; LoBue & Thrasher, 2015; Lucey et al., 2010; Lyons et al., 1998; Li et
al., 2017; Goodfellow et al., 2013). Li and Deng provided an overview of
popular datasets designed specifically for deep emotion recognition (2018).

In the realm of deep-learning-based FER approaches, convolutional neural
networks (CNNs) and recurrent neural networks (RNNs) are frequently employed.
Khorrami et al. demonstrated that CNNs excel at accurately extracting facial
action units (FAUs), thereby yielding promising classification performance on
the CK+ dataset (2015). Lopes et al. proposed a combination of CNNs and image
pre-processing techniques, such as cropping and normalization, to reduce the
number of convolutional layers and alleviate the need for extensive training
data. This approach resulted in improved overall accuracy and computational
efficiency on the CK+, JAFFE, and BU3DFE datasets (2017). Agrawal and Mittal
introduced a novel CNN model that investigated the influence of kernel size
and the number of filters on the final classification accuracy, leading to
further improvements in performance (2020). Recent studies have also explored
the integration of generative adversarial networks (GANs) within CNNs for data
augmentation and training purposes (Peng et al., 2020; Porcu et al., 2020).

Incorporating CNNs with RNNs, Zhu et al. proposed a novel FER approach that
integrates features learned from each layer of a CNN within a bidirectional
RNN architecture (2017). RNNs have also proven effective in capturing dynamic
facial actions within multimodal FER, as demonstrated by Majumder et al.
(Majumder et al., 2019; Hasani & Mahoor, 2017).

Despite achieving high overall accuracy, the performance across different
racial groups has yet to be thoroughly and systematically studied. While
multicultural FER models have demonstrated their effectiveness in improving
accuracy, their impact on fairness has yet to be explicitly addressed (Sohail
et al., 2022; Ali et al., 2016).

###  2.2 Fairness in Machine Learning

Numerous research endeavors have been dedicated to addressing issues of
unfairness within ML systems (Du et al., 2020; Mehrabi et al., 2022b; Oneto &
Chiappa, 2020; Mehrabi et al., 2022a). The underlying causes of biases in ML
have been explored in several publications (Chouldechova & Roth, 2018;
Martínez-Plumed et al., 2019). Evaluation metrics, shaped by social contexts,
are employed to gauge the fairness of these models (Chouldechova & Roth, 2018;
Castelnovo et al., 2021).

ML fairness improvement methods are generally categorized into pre-processing,
in-processing, and post-processing techniques, contingent upon the stage at
which the fairness correction method is applied (Du et al., 2020; Mehrabi et
al., 2022b; Oneto & Chiappa, 2020; Mehrabi et al., 2022a). Additionally, AI
fairness toolkits have been developed, harnessing these methods (Bird et al.,
2020; Bellamy et al., 2018; Wexler et al., 2020; Saleiro et al., 2018).

Several prior studies have delved into the examination and alleviation of
racial bias in FER. Raina et al. utilized artificial facial images and
observed racial bias in FER models (2022). Sham et al. conducted an
investigation into racial bias in popular state-of-the-art FER methods and
revealed that the presence of uneven or insufficient representation within the
training data leads to biased performance outcomes (2023a).

Additionally, biases in expression labeling within datasets, influenced by the
impact of races on emotion perceptions, was identified as contributors to
unfairness (Rhue, 2018; Chen & Joo, 2021a). Conversely, Chen and Joo’s study
did not report any systematic labeling biases for races, attributing the
absence to imbalanced racial representations in the dataset (2021a).

Although some methods have demonstrated effectiveness in correcting FER racial
bias (Xu et al., 2020a), the results highly depend on the datasets and models.
Due to the involvement of highly subjective labels and high-dimensional inputs
in emotion recognition, the field has yet to address the issue of fairness
comprehensively in this domain and similar areas with complex and
heterogeneous data streams.

##  3 Methods

###  3.1 Datasets

We employ two datasets to investigate racial bias. The first dataset is the
Child Affective Facial Expression (CAFE) dataset, a collection of images
featuring children posing specific emotions (LoBue & Thrasher, 2015). The
second dataset is AffectNet, a widely recognized large-scale dataset for
general facial emotion recognition (Mollahosseini et al., 2017).

To align the datasets, we filter the examples within AffectNet to include only
those with emotion labels matching those in the CAFE dataset, specifically
neutral, sadness, happiness, surprise, anger, disgust, and fear. Additionally,
we exclude grayscale images, which also contributes to more accurate race
estimates. We calculate the per-pixel squared error (summed across the three
channels) using the mean pixel value of each image. Images with an average
per-pixel squared error below a threshold are considered grayscale and are
removed from the training set. Consequently, the final training size for
AffectNet is N = 259,280. A similar procedure was followed for the validation
and test sets, resulting in sizes of N = 1,700 and 1,484, respectively.

To separate the faces in CAFE from their white background, we utilize OpenCV
bounding boxes, following a methodology that AffectNet uses during its data
collection process (Mollahosseini et al., 2017; Bradski, 2000). We exclude
images in which a face could not be adequately bounded, resulting in 1,178
usable images. Because the CAFE dataset involves participants posing for
multiple emotions, we opt to split the data at the participant level when
generating the training, validation, and test sets, resulting in sizes of N =
713, 227, and 222, respectively.

###  3.2 Race Estimates

Table 1: The racial distribution in CAFE’s training set. Participants self-
report their race prior to data collection. A plurality of participants
identify as European-American, indicating the potential for downstream racial
bias in the model.
Race | Count | Proportion
---|---|---
European-American | 281 | 0.404
African-American | 142 | 0.205
East Asian | 103 | 0.148
Latino | 86 | 0.124
South Asian | 82 | 0.118
Table 2: The racial distribution in AffectNet’s training set. AffectNet does
not provide race information, so this distribution is estimated using a race
model built from FairFace. European-American faces account for a much larger
proportion of the dataset relative to CAFE’s composition.
Race | Count | Proportion
---|---|---
European-American | 174,382 | 0.673
African-American | 19,131 | 0.074
East Asian | 15,833 | 0.061
Latino | 23,488 | 0.091
Middle Eastern | 18,120 | 0.070
South Asian | 4,786 | 0.018
Southeast Asian | 3,540 | 0.014

We require race labels to analyze the bias. CAFE provides participants’ self-
reported ground truth race labels: European-American, African-American, (East)
Asian, Latino, and South Asian. However, for the AffectNet dataset, race
information is not available; we approach this problem by estimating race
labels. We utilize models trained on labeled race datasets, specifically
FairFace, which exhibits greater racial balance compared to similar datasets,
and models trained on FairFace demonstrate improved performance on non-white
faces relative to other models and datasets (Karkkainen & Joo, 2021). Our
model uses the paper’s original weights to predict the race labels, and the
counts and proportions of the two datasets are presented in Tables 1 and 2. As
expected, European-American faces make up the majority of the training set
distributions for both CAFE and AffectNet, comprising 40.4% and 67.3% of their
respective datasets.

The FairFace-based model categorizes the AffectNet faces into seven
categories: European-American, African-American, (East) Asian, Latino, South
Asian, Middle Eastern, and Southeast Asian. It is possible to exclude the
Middle Eastern and Southeast Asian groups, from the AffectNet experiments to
align the racial categories with those of CAFE. However, we choose to retain
all races in the experiments assuming that there could be latent information
from these additional race categories that affects the training process.

Figure 1:  Example of the procedure for simulating East Asian representation
in a dataset. This represents a contrived case where East Asian is over-
represented in the training set. In actual simulations, $N$ is much larger,
with our largest experiment using $N=3500$.

###  3.3 Simulating Racial Composition

To investigate the impact of racial representation in the training set on test
performance, we select a specific race (henceforth referred to as the
simulated race) and vary their proportion. The sampling process ensures an
equal representation for non-simulated races by sampling $N$ examples for each
race. Then we set a ratio ($R$) of the simulated race to a non-simulated race
and sample $N*R$ examples of the simulated race. All sampling is done without
replacement. We analyze the effects of under-representation and over-
representation of the simulated race by varying the ratio.

In the simulations, we fine-tune a ResNet-50 model on the sub-sampled training
set. The model’s performance is evaluated on the validation set throughout the
epochs, and the weights yielding the highest accuracy on the validation set
are used to evaluate on the test set. While accuracy serves as one evaluation
metric, we also consider the race-specific F1-score, which is calculated after
filtering to the simulated race. Additionally, we explore fairness metrics and
extend their applicability to multi-class classification problems.

Demographic parity and equality of odds are two fairness principles that are
commonly used in the algorithmic fairness literature (Barocas et al., 2019).
Demographic parity ensures that the positive prediction rate remains
consistent across sensitive attributes. One approach to quantify this
principle is the use of the ratio of the smallest positive prediction rate to
the largest. A value of one suggests that the model achieves demographic
parity (Bird et al., 2020). To implement this metric, we transform the multi-
class problem into multiple one-versus-rest sub-problems. For each emotion, we
compute the demographic parity ratio across the different races and then
average these ratios to obtain an overall measure of demographic parity.

Equality of odds requires parity in both true-positive and false-positive
rates across sensitive attributes. A similar procedure to that used for the
demographic parity ratio can be employed to derive two separate ratios: the
true-positive parity ratio and the false-positive parity ratio (Bird et al.,
2020). The equalized odds ratio is determined by selecting the smaller of the
two ratios. This indicates that equality of odds is achieved when both parity
ratios closely approximate one.

##  4 Experiments

We conduct three sets of simulations in our study. The first simulation
focuses on the CAFE dataset, which we sample at the participant level. Given
the limited size of the training set, we set the number of participants $N$ to
be $5$ and the ratio $R$ ranges from 0 to 2.0 with increments of 0.2. This
approach ensures that each simulation comprises a whole number of
participants, and the sizes of each non-simulated races are approximately
equal, with 40-50 observations per racial group.

The second simulation involves the AffectNet dataset with $N$ set to 50
observations, and $R$ follows the same range and increments as in the previous
simulation. This simulation tests the consistency of results between AffectNet
and CAFE in the context of a small dataset size regime.

In the third simulation, we use the AffectNet dataset with $N$ set to 3500
observations, allowing for a larger training set. This simulation investigates
whether the trends observed in the first and second simulations generalize to
larger data regimes, which is relevant for the typical application of facial
emotion recognition.

Throughout the experiments, we maintain consistent training hyperparameters.
We use a fixed learning rate of $1e-4$ with an Adam optimizer (with
$\beta_{1}=0.9$ and $\beta_{2}=0.999$) and L2-weight decay on the model
parameters. Cross-entropy loss is backpropagated at each batch step, and each
simulation undergoes training for 5 epochs. To account for the emotion label
imbalance within the sub-sampled dataset, we apply a weight to the loss
function based on the number of ground truth labels. These experiments were
conducted using an NVIDIA K80 GPU unit.

###  4.1 Results

#### CAFE Simulation

We display metrics from various levels of $R$ in four race simulations:
African-American, East Asian, European-American, and Latino. We calculate
weighted F1-score and accuracy at a race level by filtering the predictions to
the simulated race. We calculate demographic parity ratio and equalized odds
ratio without any filters. Our hypothesis anticipates that race-specific
F1-score and fairness metrics would improve as the dataset becomes more
racially balanced. However, as the simulations over-sample and the simulated
race becomes over-represented, we expect fairness metrics to plateau or
decline, as the model’s fairness performance for the non-simulated races is
likely to deteriorate.

Figure 2:  CAFE racial composition simulations with all test metrics. Each
cell shows a varied simulated race with all non-simulated races held constant.
CAFE is sampled at the participant level ($N=5$). Every race shows improvement
in test performance for the race-specific F1-score and demographic parity
ratio when the simulations move towards racial balance.
Figure 3:  CAFE racial composition simulations with unaggregated F1-scores
show racial balance is correlated with increases in most emotion-specific
F1-scores. The exceptions are ‘surprise’ and ‘disgust’.

The simulations conducted on the CAFE dataset align with expectations on some
metrics. Figure 2 shows that the F1-score and the demographic parity ratio
increase ($+27.2\%$ and $+15.7\%$ points respectively on average) as the
dataset becomes balanced and stabilize when the dataset over-samples the
simulated race. On the other hand, the equalized odds ratio exhibits greater
inconsistency, with only the Latino simulations displaying a clear upward
trend, while the other races exhibit random or downward trends.

Additionally, in Figure 3 we present the disaggregated F1-scores for each race
and label before their aggregation. The disaggregated results reveal that a
significant portion of the F1-score improvement stems from emotions such as
neutral, sad, and fear. Interpreting changes in East Asian F1-scores proves
challenging, possibly due to the limited presence of Asian participants in the
test set. Moreover, surprise and disgust appear to be more challenging
emotions to predict, which could explain the seemingly random or marginal
trends observed.

#### AffectNet Small Simulation

Figure 4:  AffectNet racial composition simulations using small subsample
sizes ($N=50$) with all test metrics. Race-specific F1-score and demographic
parity ratio have poor performance likely due to overfitting, and the lack of
an observable trend in any of the simulations suggests minimal correlation
between racial balance and test performance.

The performance on the small sub-sample of AffectNet, achieving 15.2% race-
specific F1-score and 0.286 demographic parity ratio on average, is noticeably
inferior to the CAFE simulations. The limited size of the training set and the
substantially greater variation in emotion distribution from the “wild” images
in AffectNet likely contribute to this discrepancy. Despite the potential for
model overfitting, the overall trend shown in Figure 4 indicates that the
model’s performance does not significantly change as the dataset becomes more
racially balanced, as evidenced by the nearly random trends observed in the
F1-score and fairness metrics.

#### AffectNet Large Simulation

Figure 5:  AffectNet racial composition simulations using larger subsample
sizes ($N=3500$) with all test metrics. Although the larger training set
resolves overfitting to a degree, the simulations still lack a visible trend
between racial balance and test performance.

The performance on the larger AffectNet simulation, although overfitting less,
does not exhibit increases in F1-score and fairness metrics as the dataset
becomes racially balanced. Figure 5 shows the race-specific F1-score
fluctuating around 55% on average for all race simulations with no visible
trends. Both demographic parity ratio and equalized odds ratio also stay
roughly constant or even trend downwards.

Furthermore, the unaggregated F1-scores demonstrate minimal variation, with
the exception of anger, which experiences a temporary improvement when the
Asian simulations are over-sampled.

Figure 6:  The unaggregated F1-scores for the AffectNet simulations show
similar constant trends regardless of racial composition. ‘Angry’ has a
moderate increase in the East Asian simulations, but only when they are over-
sampled.

European-American

African-American

East Asian

Latino

South Asian

Middle Eastern

Southeast Asian

Figure 7:  A random sample of image/race examples from the FairFace race model
estimates. Qualitatively, estimates of European-American, African-American,
East Asian, and Latino individuals appear more accurate than of South Asian,
Middle Eastern and Southeast Asian individuals.

##  5 Discussion

The simulations conducted on CAFE indicate that racial balance of the training
set has some influence on the test race-specific F1-scores. However, only
certain fairness metric trends align with our expectations. Demographic parity
ratio increases as the training set becomes balanced and then plateaus, but
equalized odds ratio, a stricter metric, shows random trends through different
racial distribution simulations.

The AffectNet simulations exhibit mostly random trends across scenarios.
Nevertheless, even for CAFE, biases persist within the models since the
fairness metrics fail to approach a value of 1. This suggests that there may
be sources of bias that the simulations are unable to capture. One possible
explanation could be biased race estimations obtained from the FairFace model.
Upon examining a small sample of faces categorized by this model, we observe
that White, Black, Latino, and East Asian faces are estimated accurately.
Other race and ethnicity categories, however, appear to be more prone to
misclassification, particularly Middle Eastern faces based on lighter skin
tones and Indian faces based on darker skin tones. This bias could potentially
impact the training process even though the simulations focus on races
estimated with reasonable accuracy. However, since the race labels for
AffectNet are not available, it is challenging to ascertain the extent of
estimator bias. As part of future extensions, it may be worthwhile to exclude
the less-consistent racial categories to explore whether the estimator bias
contributes to the lack of trends in the AffectNet simulation.

Another source of bias that cannot be addressed through the simulation of
racial compositions is annotation bias. There is evidence in the psychology
literature suggesting that individuals are less accurate in determining facial
expressions for races different from their own, resulting in potentially
disproportionate labeling biases (Zhongqing Jiang et al., 2023; Chen & Joo,
2021a). Annotation bias could be particularly problematic for AffectNet since
its annotations were derived from only 12 labelers, with most labels being
annotated by a single individual111The authors note that a two annotators
labeled a subset of the data and there is agreement between the two annotators
ranging from 50.8% on neutral labels to 79.6% on happy labels. (Mollahosseini
et al., 2017). It would be possible to quantify and analyze this bias by
collecting data on each observation, the race of the labelers, and the
annotation process. CAFE, in fact, provides information about the aggregate
race distribution of the labelers (LoBue & Thrasher, 2015), and this
information has been incorporated into the training process in prior work
(Washington et al., 2021).

Given the persistence of these non-compositional biases within AffectNet,
traditional bias mitigation techniques like loss re-weighting or fairness
regularization may not yield strong results. This underscores the need for the
fairness community to explore alternative methods of bias mitigation,
particularly in settings involving high-dimensional image inputs and
subjective labels.

##  6 Conclusion

There is an ongoing need for addressing fairness in facial emotion
recognition. By simulating different racial distributions within the training
sets, we demonstrate the impact of compositional racial imbalance on test
performance and disparities in the CAFE dataset. Moreover, we extend this
analysis to AffectNet, which comprises non-posed, “in the wild” expressions.
The results reveal the persistent presence of bias across simulations of
racial composition in the training set, with no improvement observed in the
performance of race-specific F1-scores even when enforcing racial balance
within the simulation. To further advance research in this domain, we propose
exploring additional avenues of inquiry, such as re-simulating the AffectNet
experiments while excluding racial groups that are inaccurately estimated
during the pre-processing stage.

## References

  * Agrawal & Mittal (2020) Agrawal, A. and Mittal, N.  Using cnn for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy.  _The Visual Computer_ , 36(2):405–412, 2020.
  * Ali et al. (2016) Ali, G., Iqbal, M. A., and Choi, T.-S.  Boosted nne collections for multicultural facial expression recognition.  _Pattern Recognition_ , 55:14–27, 2016.
  * Barocas et al. (2019) Barocas, S., Hardt, M., and Narayanan, A.  _Fairness and Machine Learning: Limitations and Opportunities_.  fairmlbook.org, 2019.  http://www.fairmlbook.org.
  * Bellamy et al. (2018) Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., and et al., S. H.  AI fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias.  _CoRR_ , abs/1810.01943, 2018.  URL http://arxiv.org/abs/1810.01943.
  * Bird et al. (2020) Bird, S., Dudík, M., Edgar, R., Horn, B., Lutz, R., Milan, V., Sameki, M., Wallach, H., and Walker, K.  Fairlearn: A toolkit for assessing and improving fairness in AI.  Technical Report MSR-TR-2020-32, Microsoft, May 2020.
  * Bradski (2000) Bradski, G.  The OpenCV Library.  _Dr. Dobb’s Journal of Software Tools_ , 2000.
  * Castelnovo et al. (2021) Castelnovo, A., Crupi, R., Greco, G., Regoli, D., Penco, I. G., and Cosentini, A. C.  The zoo of fairness metrics in machine learning.  2021\.
  * Chen & Joo (2021a) Chen, Y. and Joo, J.  Understanding and mitigating annotation bias in facial expression recognition.  (arXiv:2108.08504), Aug 2021a.  URL http://arxiv.org/abs/2108.08504.
  * Chen & Joo (2021b) Chen, Y. and Joo, J.  Understanding and mitigating annotation bias in facial expression recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 14980–14991, 2021b.
  * Chouldechova & Roth (2018) Chouldechova, A. and Roth, A.  The frontiers of fairness in machine learning.  _arXiv preprint arXiv:1810.08810_ , 2018.
  * Domnich & Anbarjafari (2021) Domnich, A. and Anbarjafari, G.  Responsible ai: Gender bias assessment in emotion recognition.  _arXiv preprint arXiv:2103.11436_ , 2021.
  * Du et al. (2020) Du, M., Yang, F., Zou, N., and Hu, X.  Fairness in deep learning: A computational perspective.  (arXiv:1908.08843), Mar 2020.  URL http://arxiv.org/abs/1908.08843.
  * Goodfellow et al. (2013) Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al.  Challenges in representation learning: A report on three machine learning contests.  In _Neural Information Processing: 20th International Conference, ICONIP 2013, Daegu, Korea, November 3-7, 2013. Proceedings, Part III 20_ , pp. 117–124. Springer, 2013.
  * Hamm et al. (2011) Hamm, J., Kohler, C. G., Gur, R. C., and Verma, R.  Automated facial action coding system for dynamic analysis of facial expressions in neuropsychiatric disorders.  _Journal of Neuroscience Methods_ , 200(2):237–256, 2011.  ISSN 0165-0270.  doi: https://doi.org/10.1016/j.jneumeth.2011.06.023.
  * Hasani & Mahoor (2017) Hasani, B. and Mahoor, M. H.  Facial expression recognition using enhanced deep 3d convolutional neural networks.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. IEEE, Jul 2017.  doi: 10.1109/cvprw.2017.282.
  * Karkkainen & Joo (2021) Karkkainen, K. and Joo, J.  Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 1548–1558, 2021.
  * Khorrami et al. (2015) Khorrami, P., Paine, T., and Huang, T.  Do deep neural networks learn facial action units when doing expression recognition?  In _Proceedings of the IEEE international conference on computer vision workshops_ , pp. 19–27, 2015.
  * Ko (2018) Ko, B.  A brief review of facial emotion recognition based on visual information.  _Sensors (Basel, Switzerland)_ , 18, 2018.
  * Li & Deng (2018) Li, S. and Deng, W.  Deep facial expression recognition: A survey.  2018\.  doi: 10.48550/ARXIV.1804.08348.  URL https://arxiv.org/abs/1804.08348.
  * Li et al. (2017) Li, S., Deng, W., and Du, J.  Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2584–2593, 2017.  doi: 10.1109/CVPR.2017.277.
  * LoBue & Thrasher (2015) LoBue, V. and Thrasher, C.  The child affective facial expression (cafe) set: validity and reliability from untrained adults.  _Front. Psychol._ , 5, 2015.
  * Lopes et al. (2017) Lopes, A. T., de Aguiar, E., De Souza, A. F., and Oliveira-Santos, T.  Facial expression recognition with convolutional neural networks: Coping with few data and the training sample order.  _Pattern Recognition_ , 61:610–628, 2017.  ISSN 0031-3203.  doi: https://doi.org/10.1016/j.patcog.2016.07.026.
  * Lucey et al. (2010) Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I.  The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression.  In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops_ , pp. 94–101, 2010.  doi: 10.1109/CVPRW.2010.5543262.
  * Lyons et al. (1998) Lyons, M., Akamatsu, S., Kamachi, M., and Gyoba, J.  Coding facial expressions with gabor wavelets.  In _Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition_ , pp. 200–205, 1998.  doi: 10.1109/AFGR.1998.670949.
  * Majumder et al. (2019) Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A., and Cambria, E.  Dialoguernn: An attentive rnn for emotion detection in conversations, 2019\.
  * Martínez-Plumed et al. (2019) Martínez-Plumed, F., Ferri, C., Nieves, D., and Hernández-Orallo, J.  Fairness and missing values.  (arXiv:1905.12728), May 2019.  URL http://arxiv.org/abs/1905.12728.
  * Mehrabi et al. (2022a) Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and Galstyan, A.  A survey on bias and fairness in machine learning.  (arXiv:1908.09635), Jan 2022a.  URL http://arxiv.org/abs/1908.09635.
  * Mehrabi et al. (2022b) Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and Galstyan, A.  A survey on bias and fairness in machine learning.  (arXiv:1908.09635), Jan 2022b.  URL http://arxiv.org/abs/1908.09635.
  * Mollahosseini et al. (2017) Mollahosseini, A., Hasani, B., and Mahoor, M. H.  Affectnet: A database for facial expression, valence, and arousal computing in the wild.  _IEEE Transactions on Affective Computing_ , 2017.
  * Oneto & Chiappa (2020) Oneto, L. and Chiappa, S.  _Fairness in Machine Learning_ , volume 896, pp. 155–196.  2020\.  doi: 10.1007/978-3-030-43883-8˙7.  URL http://arxiv.org/abs/2012.15816.
  * Peng et al. (2020) Peng, Z., Li, J., and Sun, Z.  Emotion recognition using generative adversarial networks.  In _2020 International Conference on Computer Engineering and Intelligent Control (ICCEIC)_ , pp. 77–80, 2020.  doi: 10.1109/ICCEIC51584.2020.00023.
  * Picard (1999) Picard, R. W.  Affective computing for hci.  In _HCI (1)_ , pp. 829–833. Citeseer, 1999.
  * Porcu et al. (2020) Porcu, S., Floris, A., and Atzori, L.  Evaluation of data augmentation techniques for facial expression recognition systems.  _Electronics_ , 9(11), 2020.  ISSN 2079-9292.  doi: 10.3390/electronics9111892.  URL https://www.mdpi.com/2079-9292/9/11/1892.
  * Raina et al. (2022) Raina, R., Monares, M., Xu, M., Fabi, S., Xu, X., Li, L., Sumerfield, W., Gan, J., and de Sa, V. R.  Exploring biases in facial expression analysis.  In _NeurIPS 2022 Workshop on Synthetic Data for Empowering ML Research_ , 2022.  URL https://openreview.net/forum?id=jrddoq9NDP6.
  * Rhue (2018) Rhue, L.  Racial influence on automated perceptions of emotions.  _SSRN Electronic Journal_ , 2018.  ISSN 1556-5068.  doi: 10.2139/ssrn.3281765.  URL https://www.ssrn.com/abstract=3281765.
  * Saleiro et al. (2018) Saleiro, P., Kuester, B., Stevens, A., Anisfeld, A., Hinkson, L., London, J., and Ghani, R.  Aequitas: A bias and fairness audit toolkit.  _arXiv preprint arXiv:1811.05577_ , 2018.
  * Sham et al. (2023a) Sham, A. H., Aktas, K., Rizhinashvili, D., Kuklianov, D., Alisinanoglu, F., Ofodile, I., Ozcinar, C., and Anbarjafari, G.  Ethical ai in facial expression analysis: racial bias.  _Signal, Image and Video Processing_ , 17(2):399–406, Mar 2023a.  ISSN 1863-1703, 1863-1711.  doi: 10.1007/s11760-022-02246-8.
  * Sham et al. (2023b) Sham, A. H., Aktas, K., Rizhinashvili, D., Kuklianov, D., Alisinanoglu, F., Ofodile, I., Ozcinar, C., and Anbarjafari, G.  Ethical ai in facial expression analysis: Racial bias.  _Signal, Image and Video Processing_ , 17(2):399–406, 2023b.
  * Sohail et al. (2022) Sohail, M., Ali, G., Rashid, J., Ahmad, I., Almotiri, S. H., AlGhamdi, M. A., Nagra, A. A., and Masood, K.  Racial identity-aware facial expression recognition using deep convolutional neural networks.  _Applied Sciences_ , 12(1), 2022.  ISSN 2076-3417.  doi: 10.3390/app12010088.  URL https://www.mdpi.com/2076-3417/12/1/88.
  * Tao & Tan (2005) Tao, J. and Tan, T.  Affective computing: A review.  In _Affective Computing and Intelligent Interaction: First International Conference, ACII 2005, Beijing, China, October 22-24, 2005. Proceedings 1_ , pp. 981–995. Springer, 2005.
  * Voss et al. (2019) Voss, C., Schwartz, J., Daniels, J., Kline, A., Haber, N., Washington, P., Tariq, Q., Robinson, T. N., Desai, M., Phillips, J. M., et al.  Effect of wearable digital intervention for improving socialization in children with autism spectrum disorder: a randomized clinical trial.  _JAMA pediatrics_ , 173(5):446–454, 2019.
  * Washington et al. (2021) Washington, P., Kalantarian, H., Kent, J., Husic, A., Kline, A., Leblanc, E., Hou, C., Mutlu, C., Dunlap, K., Penev, Y., et al.  Training affective computer vision models by crowdsourcing soft-target labels.  _Cognitive computation_ , 13:1363–1373, 2021.
  * Washington et al. (2022) Washington, P., Kalantarian, H., Kent, J., Husic, A., Kline, A., Leblanc, E., Hou, C., Mutlu, O. C., Dunlap, K., Penev, Y., et al.  Improved digital therapy for developmental pediatrics using domain-specific artificial intelligence: Machine learning study.  _JMIR Pediatrics and Parenting_ , 5(2):e26760, 2022.
  * Wexler et al. (2020) Wexler, J., Pushkarna, M., Bolukbasi, T., Wattenberg, M., Viegas, F., and Wilson, J.  The what-if tool: Interactive probing of machine learning models.  _IEEE Transactions on Visualization and Computer Graphics_ , 26(01):56–65, Jan 2020.  ISSN 1941-0506.  doi: 10.1109/TVCG.2019.2934619.
  * Xu et al. (2020a) Xu, T., White, J., Kalkan, S., and Gunes, H.  Investigating bias and fairness in facial expression recognition.  (arXiv:2007.10075), Aug 2020a.  URL http://arxiv.org/abs/2007.10075.
  * Xu et al. (2020b) Xu, T., White, J., Kalkan, S., and Gunes, H.  Investigating bias and fairness in facial expression recognition.  In _Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16_ , pp. 506–523. Springer, 2020b.
  * Zhao et al. (2019) Zhao, S., Wang, S., Soleymani, M., Joshi, D., and Ji, Q.  Affective computing for large-scale heterogeneous multimedia data: A survey.  _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ , 15(3s):1–32, 2019.
  * Zhongqing Jiang et al. (2023) Zhongqing Jiang, Guillermo Recio, Wenhui Li, Peng Zhu, Jiamei He, and Werner Sommer.  The other-race effect in facial expression processing: Behavioral and ERP evidence from a balanced cross-cultural study in women.  _Int J Psychophysiol_ , 183:53–60, 2023.
  * Zhu et al. (2017) Zhu, X., Li, L., Zhang, W., Rao, T., Xu, M., Huang, Q., and Xu, D.  Dependency exploitation: A unified cnn-rnn approach for visual emotion recognition.  In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17_ , pp. 3595–3601, 2017.  doi: 10.24963/ijcai.2017/503.