# Integrating Curricula with Replays: Its Effects on Continual Learning

Tee Ren Jie,1,2 Mengmi Zhang 2, 3

###### Abstract

Humans engage in learning and reviewing processes with curricula when
acquiring new skills or knowledge. This human learning behavior has inspired
the integration of curricula with replay methods in continual learning agents.
The goal is to emulate the human learning process, thereby improving knowledge
retention and facilitating learning transfer. Existing replay methods in
continual learning agents involve the random selection and ordering of data
from previous tasks, which has shown to be effective. However, limited
research has explored the integration of different curricula with replay
methods to enhance continual learning. Our study takes initial steps in
examining the impact of integrating curricula with replay methods on continual
learning in three specific aspects: the interleaved frequency of replayed
exemplars with training data, the sequence in which exemplars are replayed,
and the strategy for selecting exemplars into the replay buffer. These aspects
of curricula design align with cognitive psychology principles and leverage
the benefits of interleaved practice during replays, easy-to-hard rehearsal,
and exemplar selection strategy involving exemplars from a uniform
distribution of difficulties. Based on our results, these three curricula
effectively mitigated catastrophic forgetting and enhanced positive knowledge
transfer, demonstrating the potential of curricula in advancing continual
learning methodologies. Our code and data are available:
https://github.com/ZhangLab-DeepNeuroCogLab/Integrating-Curricula-with-Replays

## Introduction

Continual learning enables consecutive task acquisition without forgetting
previously trained tasks (Kirkpatrick et al. 2017). This adaptability is vital
for autonomous systems in dynamic environments, such as updating a grocery
classification model with new products without retraining it on previous
products. However, a significant challenge in continual learning is
catastrophic forgetting, where knowledge from recent tasks interferes with
earlier ones (Parisi et al. 2019), leading to performance degradation on
earlier tasks after training on a task sequence.

To resolve this problem, there are three primary types of continual learning
methods commonly employed in the field: regularization-based methods introduce
regularization terms to mitigate catastrophic forgetting by preserving
important parameters during training (Chaudhry et al. 2018; Zenke, Poole, and
Ganguli 2017; Aljundi et al. 2018; Benzing 2022); rehearsal-based methods
store and replay a subset of previous data during training to maintain
knowledge from previous tasks (Rolnick et al. 2019; Chaudhry et al. 2019;
Riemer et al. 2018; Vitter 1985; Rebuffi et al. 2017; Castro et al. 2018) and
parameter isolation methods isolate specific parameters for each task to
prevent interference between tasks (Yoon et al. 2017; Hung et al. 2019;
Ostapenko et al. 2019). Rehearsal-based methods have proven highly effective
in continual learning. However, existing approaches typically involve randomly
selecting and rehearsing data from previous tasks. Limited research explores
the incorporation of meaningful curricula into replay methods.

In parallel, in the curriculum learning literature, various approaches have
focused on weakly supervised (Guo et al. 2018), unsupervised (Azad et al.
2023), and reinforcement learning tasks (Narvekar et al. 2020). These studies
demonstrate that curricula improve generalization abilities, task
performances, and convergence speed (Wu, Dyer, and Neyshabur 2021; Soviany et
al. 2022) during training. However, they primarily address intra-class
difficulty and example scheduling within a single task, neglecting the impact
of class presentation sequences across multiple tasks. Recent research has
explored curricula in continual learning scenarios without data replays (Singh
et al. 2022). In complement to this work, our study investigates the role of
curricula specifically during replay in continual learning, while keeping the
curricula consistent for the feed-forward training process.

Exploring optimal curricula offers countless possibilities, and in our study,
we take initial steps to investigate a limited set of potential curricula. We
draw inspiration from two sources to guide the design of these curricula.
Firstly, neuroscience research has revealed that neural activity patterns
associated with past experiences are replayed in specific orders during rest
or sleep, which is believed to contribute to memory consolidation and spatial
navigation (Drieu and Zugaro 2019). Secondly, pedagogy studies indicate that
repetitive practice and revisiting previous knowledge with increasing
difficulty enhance long-term memory integration in students (Zhan et al.
2018).

Specifically, we propose three types of curricula for replays and examine
their impact on catastrophic forgetting and positive knowledge transfer: (1)
the interleaved frequency of replayed exemplars with training data, (2) the
replay sequence of exemplars, and (3) the strategy for selecting exemplars
into the replay buffer. The experimental findings align with cognitive
psychology principles, highlighting the advantages of frequently interleaving
between training data and replayed exemplars, incorporating easy-to-hard
rehearsals, and selecting exemplars from a uniform distribution of
difficulties for replay. These observations present a promising avenue for
advancing continual learning methods. It also provides insights into the
underlying mechanisms of replay strategies in mitigating forgetting and
facilitating knowledge transfer across tasks.

## Related Works

### Replay Methods in Continual Learning

Extensive research has focused on utilizing replay methods to address the
issue of catastrophic forgetting. Conventional replay methods, such as iCaRL
(Rebuffi et al. 2017) and ER (Rolnick et al. 2019), involve explicit training
on previously saved data, while several variants, like DGR (Shin et al. 2017)
and Pseudo-Recursal (Atkinson et al. 2018), replay on artificially synthesized
samples by generative models, resembling data from previous tasks. Although
these replay methods have made significant contributions in reducing
catastrophic forgetting, they paid little attention to the incorporation of
meaningful curricula into replay methods. Most methods randomly interleave the
replay samples with the training data, without exploring the optimal mixing
strategies (Rolnick et al. 2019; van de Ven, Siegelmann, and Tolias 2020;
Ketz, Kolouri, and Pilly 2019). In our work, we systematically studied the
effect of interleaving curricula, which involves mixing training data and
replay samples within a pre-defined interleave interval.

### Curriculum Learning

Curriculum learning methods can be broadly categorized into two groups. The
first group involves manual curriculum design by humans before training
(Soviany et al. 2020; Chen and Gupta 2015), but these methods typically rely
on human expertise and struggle to generalize to new domains. The second group
consists of models that can autonomously design curricula without human
intervention (Graves et al. 2017; Fan et al. 2018). However, the application
of these methods to enhance model performance has received limited attention
in the continual learning setting. Here, we highlight two factors to consider
when applying curricula on the replay methods in continual learning. Firstly,
while curriculum learning has demonstrated efficacy in enhancing
generalization and training speed within a single task, the objective of
curriculum learning in the context of continual learning is to retain
knowledge from previous tasks while acquiring new knowledge from the current
task. Secondly, unlike within-task curriculum learning, models in continual
learning only have access to data from the current task, making it challenging
to create a comprehensive between-task curriculum that encompasses the entire
dataset. Here, we took initial steps in this direction by exploring automated
methods to determine the sequence of replay samples and introducing the sample
selection strategy which finds the best replay samples for building a
curriculum.

## Experiments

We investigated the effect of three types of replay curricula in the class
incremental learning (CIL) setting. We first introduce CIL, and then elaborate
on the three replay curricula individually.

Problem Setting. The objective of CIL is to teach a unified classification
model $\Theta$ to recognize sets of object classes incrementally over time.
Specifically, an image dataset $D$, consisting of $N$ object classes, is split
into subsets $\\{D_{1},...,D_{t},...,D_{T}\\}$ of images and presented over a
sequence of $T$ tasks. In each task $t$, the model only has access to training
data in $D_{t}$, consisting of samples from distinct classes $C_{t}$, and
$(x_{i,t},y_{i,t})$ is the $i$-th (image, label) pair in $D_{t}$. The model
$\Theta$ can run multiple passes over $D_{t}$ in task $t$. The model stops
training on $D_{t}$ after its performance on the validation set saturates,
considering the five most recent epochs.

We implemented the naive replay method where some raw images and their
corresponding labels are selected from previous tasks and are stored in the
replay buffer $R_{t}$. These data in $R_{t}$ are inter-leaved with $D_{t}$ for
rehearsals. There are three types of replay curricula involved in this study:
(1) the interleave frequency; (2) the rehearsal sequence of $R_{t}$ in CIL;
and (3) the image selection for $R_{t}$. $R_{t}$ is kept at a constant size of
1200 over all the tasks. See Appendix for more training details. As an upper
bound, we also include the offline method where the model $\Theta$ is trained
on the entire dataset $\Theta$ from $\\{D_{1},...,D_{T}\\}$ over multiple
epochs without any continual learning.

Datasets. We conducted experiments to investigate the use of these three types
of curricula in replay methods on the two image datasets ciFAIR-10 and
ciFAIR-100 (Barz and Denzler 2019). ciFAIR-10 dataset contains 10 object
classes. The protocol asks the model $\Theta$ to incrementally learn 2 object
classes in each task. There are a total of 5 tasks. ciFAIR-100 dataset
contains 100 object classes. The CIL protocol asks the model $\Theta$ to
incrementally learn 5 object classes in each task. There are a total of 20
tasks. Both datasets have a total of 60,000 images, with 50,000 images used
for training and 10,000 images used for testing. The conclusions drawn from
the experiments on both datasets are consistent. Without loss of generality,
we focus on all the experiments and result analysis in ciFAIR-100 in the main
text. See Appendix for more implementation details and results on ciFAIR-10.

Evaluation Metrics. To assess the continual learning performance of the model
$\Theta$, we followed (Singh et al. 2022) and introduce 2 standard evaluation
metrics. We define Forgetfullness (F) as the percentage decrease in
classification accuracy on the test instances from $C_{1}$ between the
$\Theta_{t}$ (model after being trained on $D_{t}$) and $\Theta_{1}$. An ideal
$\Theta_{t}$ could maintain the same classification accuracy on $C_{1}$ over
tasks; i.e. $\forall t,F_{t}=0$. The higher F is, the more $\Theta$ suffers
from catastrophic forgetting. To assess the overall classification performance
of $\Theta$ over tasks, we also report the continual average classification
accuracy (Avg. Accu.). Avg. Accu. is computed as the average accuracy on all
the test instance from $C_{i}$, where $i\in\\{1,2,...,t\\}$. For simplicity,
we report the averaged F and Avg. Accu. over all the tasks.

Experimental Controls. Within each experiment, only one variable of interest
changes while the rest of the experiment conditions are fixed as control
variables. As the previous study has shown that the sequence of class
presentations affects the continual learning performance (He, Wang, and Chen
2022), we use the same class presentation sequence in all three experiments.
The same MobileNetV3 (small) network architecture is used as the backbone for
the model $\Theta$ for all experiments. In every experiment, the total number
of training samples and the total number of replay samples exposed to $\Theta$
remain the same across all experiment variables. Each experiment is conducted
with 4 runs initialized with 4 random seeds, where the seeds are used to vary
the controlled variables. The average performance across all 4 runs is
reported.

### Interleave Divisions during Rehearsals

The number of interleaving divisions refers to the number of splits of in
$D_{t}$ and $R_{t}$. It indicates how often the model $\Theta$ rehearses on
$R_{t}$, while learning on a subset of $D_{t}$. For example, for interleaving
division number 400, $D_{t}$ is split into 400 groups where each group
contains an equal number of $(x_{i,t},y_{i,t})$ (image, label) pairs, and
these (image, label) pairs are randomly selected from $D_{t}$ without
replacement. Correspondingly, $R_{t}$ is also split into 400 groups with the
same splitting criteria as $D_{t}$. At each training epoch, the model
$\Theta_{t}$ at task $t$ is repeatedly trained with one group of $D_{t}$
followed by one group of $R_{t}$, until the entire $D_{t}$ and $R_{t}$ are
exhaustively seen by $\Theta_{t}$. We titrate the interleave division numbers
with the range of 1, 8, 60, 120, and 300. The training data is interleaved
with replay data and then presented to the model in sequence. Different
interleave division numbers result in different data presentation sequences;
hence, different curricula.

### Rehearsal Sequence of Replay Samples

We use the interleave divisions 1 and 600 for all the experiments in this
subsection and vary the rehearsal sequence of data samples in $R_{t}$ by
taking into account the two factors: the sample difficulty levels and the
increasing or decreasing directions of sample difficulty levels.

To measure whether a sample is easy or hard to learn, we introduce two
difficulty measures: (1) the confidence score difficulty metrics and (2) the
distance vector difficulty metrics. The confidence score difficulty metrics
were used to assess whether a teacher network with full knowledge of the
entire dataset $D$ predicted high or low confidence of the given sample
belonging to its ground truth class label. Specifically, each image within
$R_{t}$ was input to a teacher network. The teacher network is based on a
MobileNetV3 (small) architecture, pre-trained on the entire dataset $D$. After
this, the confidence score for the ground truth class of each sample was
extracted from the teacher network’s output. $R_{t}$ was then sorted according
to its individual sample’s confidence score, where a higher confidence score
means that the sample is easier to learn for $\Theta$.

However, in CIL setting, having a teacher network with full access to the
whole dataset is impractical, as the data is incrementally available over
tasks. Hence, we employed the distance vector difficulty metrics, used widely
in literature (Singh et al. 2022; Huang et al. 2022). Intuitively, if the
sample is closer to other samples in the memory buffer, it is easier for
$\Theta$ to learn and generalize to other samples as well. The 2nd last layer
from a ResNet-50 model (He et al. 2016), pretrained on the ImageNet dataset,
was used to extract the feature vector of each sample in $R_{t}$. A Euclidean
distance matrix was created, where the pairwise Euclidean distance for all the
samples based on their feature vectors was computed. We then compute the sum
of each row of the matrix and denote this column vector as a distance vector.
Each element in this distance vector represents how much a particular sample
differs from all other samples in the feature space. A smaller value in the
distance vector means that the particular replay sample is easier to learn for
$\Theta$.

We introduce a series of rehearsal sequences in the orders of either easy-to-
hard samples or hard-to-easy samples, where the difficulty levels of each
sample are determined by either the confidence score difficulty metrics or the
distance vector difficulty metrics. As the previous study has shown that the
class orders are also essential for continual learning (He, Wang, and Chen
2022), here we also explore the effect of the class orders during replays.
When we design the rehearsal sequence based on class difficulties in $R_{t}$,
we adapt the two sample-level difficulty measures above to compute class-level
difficulty measures by taking the average over all samples of the same class
in $R_{t}$. We then sort all the samples in $R_{t}$ by their class difficulty
metrics, regardless of their individual sample difficulty scores. Samples in
$R_{t}$ sorted by their class difficulties were then presented to the model
$\Theta$ in either the easy-to-hard or hard-to-easy orders.

### Selection of Samples for Replay Buffer

In common practice, selecting samples for $R_{t+1}$ from task $t$ is often
conducted in a random manner (Rolnick et al. 2019; Bagus and Gepperth 2021).
In contrast to the previous works, we vary the sample selection criteria for
$R_{t+1}$ as follows: selecting only the easiest samples from task $t$ for
$R_{t+1}$, selecting the hardest samples from task $t$ for $R_{t+1}$, and
selecting samples that are uniformly distributed across difficulty levels from
task $t$ for $R_{t+1}$. The difficulty levels are judged based on the
confidence scores and the distance vectors defined in the previous subsection.
We use interleave division numbers 1 and 600 for all the experiments in this
subsection.

## Results

### Frequent Replays Enhance Performances

Inter. Div. | 1 | 8 | 60 | 120 | 300
---|---|---|---|---|---
F | 63.9 | 62.6 | 57.4 | 55.1 | 56.1
Avg.Accu | 39.9 | 40.7 | 44.6 | 46.6 | 46.6
Table 1: Continual learning performance on ciFAIR-100 as a function of
interleave divisions (Inter. Div.). The higher Avg. Accu., the better; the
lower F, the better. The best are in bold. The offline accuracy is 62.0%. See
Experiments Section for evaluation metrics and experiments in interleave
divisions.

We report F and Avg. Accu. as a function of interleave divisions in Table 1.
Notably, we observed that interleave divisions are important factors
influencing the continual learning performance of the replay method with the
larger interleave divisions leading to better performances, as indicated by
the decreasing F and increasing Avg. Accu. over all the tasks. It is possible
that the model parameters at large division numbers are updated more
frequently for both the current task and all previous tasks, resulting in
minimal forgetting. However, we also note that the continual learning
performance saturates at interleave division number 120. This implies that
increasing interleave divisions beyond optimal values brings no extra benefits
in continual learning.

### Easy-To-Hard Rehearsal Sequences are Beneficial

Figure 1: Continual learning performance on ciFAIR-100 dataset with different
rehearsal sequences. Average classification accuracy over all tasks for models
trained with different rehearsal sequences is reported. The higher the average
accuracy, the better. See the legend on the right for conventions of different
rehearsal sequence conditions. For example, a light grey bar with a red
boundary without the texture of slanted bar patterns (the leftmost bar)
indicates that the model is trained with an easy-to-hard rehearsal sequence
sorted by the instance-level difficulties based on distance vector metrics.
See Experiments Section for details. See Appendix for the results reported in
F.

We studied the models trained with different rehearsal sequences sorted in
easy-to-hard or hard-to-easy curricula based on sample-level or class-level
difficulty measures computed from either the confidence scores or distance
vectors. We reported the Avg. Accu. results in Figure 1 and F scores in
Appendix and made four key observations. First, aligning with the observations
in Table 1 and the discussion from the previous subsection, large interleave
divisions benefit continual learning models with higher average accuracy and
less forgetting. Second, rehearsal sequences sorted by instance-level
difficulties lead to much better continual learning performances (compare red
bars versus blue bars). Third, the confidence score is a better evaluation
metric measuring instance-level difficulties, as shown by the bars with and
without texture patterns. Finally, the models trained with the easy-to-hard
rehearsal sequences outperform the ones with reversed rehearsal sequences
(compare light versus dark grey bars). It is possible that easy-to-hard
rehearsal sequences make the models converge faster on the previous tasks due
to more stable gradient updates; hence, the sequences lead to minimal
forgetting and higher classification accuracy. We also compared the continual
learning performance for both the offline method and the continual learning
method with various curricula and observed that there still exists a large
performance gap between these two.

### Replays with Only Hard Data Hurt Performances

Figure 2: Continual learning performance on ciFAIR-100 dataset under different
replay sample selection strategies. Average classification accuracy over all
tasks for models trained with different replay samples selected using
different strategies. The interpretation of legends and design conventions are
similar to Figure 1. See Experiments Section for details. See Appendix for the
results in F.

Here, we explored the effect of different sample selection strategies for
replay samples in terms of the sample difficulty levels based on distance
vectors or confidence scores. From Figure 2, Our observations indicate that
exclusively choosing the most challenging replay samples leads to inferior
performance compared to selecting the easiest samples or incorporating samples
with a balanced distribution of difficulty levels. Selecting samples with a
uniform distribution of difficulty levels yields the best continual learning
performance. This outcome may be attributed to the fact that difficult replay
samples result in less flat loss landscapes, which in turn make the training
process more challenging and slower to converge (Wu, Dyer, and Neyshabur
2021). A curriculum for training the models to rehearse from the easiest to
the hardest samples is the best, as it balances the greater precision in data
fitting due to the hardest samples and the fast convergence speed during
training due to the easier samples. Similar to the previous subsection, we
also noted that the confidence score is a better measure of sample difficulty
levels than the distance vectors.

## Conclusion

Our study examines the role of curricula during replays in the class-
incremental learning setting in continual learning. We designed and conducted
a series of controlled experiments to study the three key questions on
replays: how often is the replay, what data should be replayed, and in what
sequence to replay. Across the two common image datasets, our experimental
results shed light on the underlying principles of replay methods in continual
learning and reveal the good curricula design choices for replay methods.
These curricula designs not only facilitate positive knowledge transfers
(which has been explored in existing curriculum learning literature), but also
mitigate catastrophic forgetting (a significant problem we need to solve in
continual learning). Specifically, we found that (1) replays should happen
frequently; (2) only rehearsing on the most difficult exemplars hurts
continual learning performances; and (3) rehearsals on samples with increasing
difficulty eliminate forgetting more than its reversed difficulty orders.
There are numerous other possible choices of curricula designs for replay
methods, such as a unified difficulty metric considering both confidence
scores and distance vectors or the use of a student feedback loop to update
the difficulty scores. In the future, we will look into the role of curricula
under stringent continual learning conditions, such as learning with limited
training time or noisy data. We will also conduct experiments on other large-
scale datasets and apply our replay curriculum to existing replay-based
continual learning algorithms.

## Acknowledgements

This research is supported by the National Research Foundation, Singapore
under its AI Singapore Programme (AISG Award No: AISG2-RP-2021-025), its NRFF
award NRF-NRFF15-2023-0001, Mengmi Zhang’s Startup Grant from Agency for
Science, Technology, and Research (A*STAR), and Early Career Investigatorship
from Center for Frontier AI Research (CFAR), A*STAR. The authors declare that
they have no competing interests.

## References

  * Aljundi et al. (2018) Aljundi, R.; Babiloni, F.; Elhoseiny, M.; Rohrbach, M.; and Tuytelaars, T. 2018\.  Memory aware synapses: Learning what (not) to forget.  In _Proceedings of the European conference on computer vision (ECCV)_ , 139–154.
  * Atkinson et al. (2018) Atkinson, C.; McCane, B.; Szymanski, L.; and Robins, A. V. 2018.  Pseudo-Recursal: Solving the Catastrophic Forgetting Problem in Deep Neural Networks.  _CoRR_ , abs/1802.03875.
  * Azad et al. (2023) Azad, A. S.; Gur, I.; Faust, A.; Abbeel, P.; and Stoica, I. 2023.  CLUTR: Curriculum Learning via Unsupervised Task Representation Learning.
  * Bagus and Gepperth (2021) Bagus, B.; and Gepperth, A. 2021.  An investigation of replay-based approaches for continual learning.  In _2021 International Joint Conference on Neural Networks (IJCNN)_ , 1–9. IEEE.
  * Barz and Denzler (2019) Barz, B.; and Denzler, J. 2019.  Do we train on test data? Purging CIFAR of near-duplicates.  _CoRR_ , abs/1902.00423.
  * Benzing (2022) Benzing, F. 2022.  Unifying importance based regularisation methods for continual learning.  In _International Conference on Artificial Intelligence and Statistics_ , 2372–2396. PMLR.
  * Castro et al. (2018) Castro, F. M.; Marín-Jiménez, M. J.; Guil, N.; Schmid, C.; and Alahari, K. 2018.  End-to-end incremental learning.  In _Proceedings of the European conference on computer vision (ECCV)_ , 233–248.
  * Chaudhry et al. (2018) Chaudhry, A.; Dokania, P. K.; Ajanthan, T.; and Torr, P. H. 2018.  Riemannian walk for incremental learning: Understanding forgetting and intransigence.  In _Proceedings of the European conference on computer vision (ECCV)_ , 532–547.
  * Chaudhry et al. (2019) Chaudhry, A.; Rohrbach, M.; Elhoseiny, M.; Ajanthan, T.; Dokania, P. K.; Torr, P. H.; and Ranzato, M. 2019.  On tiny episodic memories in continual learning.  _arXiv preprint arXiv:1902.10486_.
  * Chen and Gupta (2015) Chen, X.; and Gupta, A. 2015.  Webly Supervised Learning of Convolutional Networks.  In _2015 IEEE International Conference on Computer Vision (ICCV)_ , 1431–1439.
  * Drieu and Zugaro (2019) Drieu, C.; and Zugaro, M. 2019.  Hippocampal Sequences During Exploration: Mechanisms and Functions.  _Frontiers in cellular neuroscience_ , 13: 232.
  * Fan et al. (2018) Fan, Y.; Tian, F.; Qin, T.; Li, X.; and Liu, T. 2018.  Learning to Teach.  _CoRR_ , abs/1805.03643.
  * Graves et al. (2017) Graves, A.; Bellemare, M. G.; Menick, J.; Munos, R.; and Kavukcuoglu, K. 2017.  Automated Curriculum Learning for Neural Networks.  In Precup, D.; and Teh, Y. W., eds., _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , 1311–1320. PMLR.
  * Guo et al. (2018) Guo, S.; Huang, W.; Zhang, H.; Zhuang, C.; Dong, D.; Scott, M. R.; and Huang, D. 2018.  Curriculumnet: Weakly supervised learning from large-scale web images.  In _Proceedings of the European conference on computer vision (ECCV)_ , 135–150.
  * He, Wang, and Chen (2022) He, C.; Wang, R.; and Chen, X. 2022.  Rethinking class orders and transferability in class incremental learning.  _Pattern Recognition Letters_ , 161: 67–73.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep Residual Learning for Image Recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 770–778.
  * Huang et al. (2022) Huang, P.; Xu, M.; Zhu, J.; Shi, L.; Fang, F.; and Zhao, D. 2022.  Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation.  arXiv:2210.10195.
  * Hung et al. (2019) Hung, C.-Y.; Tu, C.-H.; Wu, C.-E.; Chen, C.-H.; Chan, Y.-M.; and Chen, C.-S. 2019\.  Compacting, picking and growing for unforgetting continual learning.  _Advances in Neural Information Processing Systems_ , 32.
  * Ketz, Kolouri, and Pilly (2019) Ketz, N.; Kolouri, S.; and Pilly, P. K. 2019.  Using World Models for Pseudo-Rehearsal in Continual Learning.  _CoRR_ , abs/1903.02647.
  * Kirkpatrick et al. (2017) Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; Hassabis, D.; Clopath, C.; Kumaran, D.; and Hadsell, R. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences_ , 114(13): 3521–3526.
  * Narvekar et al. (2020) Narvekar, S.; Peng, B.; Leonetti, M.; Sinapov, J.; Taylor, M. E.; and Stone, P. 2020\.  Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey.  _J. Mach. Learn. Res._ , 21(1).
  * Ostapenko et al. (2019) Ostapenko, O.; Puscas, M.; Klein, T.; Jahnichen, P.; and Nabi, M. 2019.  Learning to remember: A synaptic plasticity driven framework for continual learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 11321–11329.
  * Parisi et al. (2019) Parisi, G. I.; Kemker, R.; Part, J. L.; Kanan, C.; and Wermter, S. 2019.  Continual lifelong learning with neural networks: A review.  _Neural Networks_ , 113: 54–71.
  * Rebuffi et al. (2017) Rebuffi, S.-A.; Kolesnikov, A.; Sperl, G.; and Lampert, C. H. 2017.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2001–2010.
  * Riemer et al. (2018) Riemer, M.; Cases, I.; Ajemian, R.; Liu, M.; Rish, I.; Tu, Y.; and Tesauro, G. 2018\.  Learning to learn without forgetting by maximizing transfer and minimizing interference.  _arXiv preprint arXiv:1810.11910_.
  * Rolnick et al. (2019) Rolnick, D.; Ahuja, A.; Schwarz, J.; Lillicrap, T.; and Wayne, G. 2019.  Experience replay for continual learning.  _Advances in Neural Information Processing Systems_ , 32.
  * Shin et al. (2017) Shin, H.; Lee, J. K.; Kim, J.; and Kim, J. 2017.  Continual learning with deep generative replay.  _Advances in neural information processing systems_ , 30.
  * Singh et al. (2022) Singh, P.; Li, Y.; Sikarwar, A.; Lei, W.; Gao, D.; Talbot, M. B.; Sun, Y.; Shou, M. Z.; Kreiman, G.; and Zhang, M. 2022.  Learning to Learn: How to Continuously Teach Humans and Machines.
  * Soviany et al. (2020) Soviany, P.; Ardei, C.; Ionescu, R.; and Leordeanu, M. 2020.  Image Difficulty Curriculum for Generative Adversarial Networks (CuGAN).  In _2020 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 3452–3461. Los Alamitos, CA, USA: IEEE Computer Society.
  * Soviany et al. (2022) Soviany, P.; Ionescu, R. T.; Rota, P.; and Sebe, N. 2022.  Curriculum Learning: A Survey.  _International Journal of Computer Vision_ , 130(6): 1526–1565.
  * van de Ven, Siegelmann, and Tolias (2020) van de Ven, G. M.; Siegelmann, H. T.; and Tolias, A. S. 2020.  Brain-inspired replay for continual learning with artificial neural networks.  _Nature Communications_ , 11(1): 4069.
  * Vitter (1985) Vitter, J. S. 1985.  Random sampling with a reservoir.  _ACM Transactions on Mathematical Software (TOMS)_ , 11(1): 37–57.
  * Wu, Dyer, and Neyshabur (2021) Wu, X.; Dyer, E.; and Neyshabur, B. 2021.  When Do Curricula Work?  In _International Conference on Learning Representations_.
  * Yoon et al. (2017) Yoon, J.; Yang, E.; Lee, J.; and Hwang, S. J. 2017.  Lifelong learning with dynamically expandable networks.  _arXiv preprint arXiv:1708.01547_.
  * Zenke, Poole, and Ganguli (2017) Zenke, F.; Poole, B.; and Ganguli, S. 2017.  Continual learning through synaptic intelligence.  In _International conference on machine learning_ , 3987–3995. PMLR.
  * Zhan et al. (2018) Zhan, L.; Guo, D.; Chen, G.; and Yang, J. 2018.  Effects of Repetition Learning on Associative Recognition Over Time: Role of the Hippocampus and Prefrontal Cortex.  _Frontiers in human neuroscience_ , 12: 277.

## Appendix

### Experimental Details

For experiments on both ciFAIR-10 and ciFAIR-100, PyTorch’s default
implementation of cross entropy loss was used for object classification tasks.
The SGD algorithm was used as the optimizer. The learning rate was set at a
constant of 0.001. Momentum was fixed at 0.9. A batch size of 32 is used.

For ciFAIR-10, we employ a 2-layer 2D-convolutional network with 6 and 16
channels in the successive layers, followed by 3 fully connected layers with
400, 120 and 84 hidden units respectively. ReLU was used as the activation
function.

We follow the standard training and testing data splits from the original
ciFAIR-10. In every task, the model is trained for 250 epochs. Each experiment
is conducted with 20 runs initialized with 20 random seeds, where the seeds
are used to vary the controlled variables. The average performance across all
20 runs is reported.

For ciFAIR-100, PyTorch’s implementation of MobileNetV3 (small) was used,
including the default layers and activation function. We used a custom
training, validation, and test data splits with a ratio of 9:1:2, and a
stopping criteria for training depending on the validation loss. The
ciFAIR-100 images were upscaled to 72x72 using PyTorch’s bicubic interpolation
function before training.

### More Results and Analysis

We reported the continual learning performance on ciFAIR-10 dataset of the
models trained with the three types of curricula as elaborated in Experiments
Section. See Table S1 for interleave divisions, Figure S1 for rehearsal
sequences, and Figure S3 for sample selections. All the tables and figures on
ciFAIR-10 dataset follow the same design conventions as the corresponding
tables and figures on ciFAIR-100 dataset in the main text. The conclusions
from the results of ciFAIR-10 dataset are consistent with the ones on the
ciFAIR-100 dataset.

Inter. Div. | 1 | 8 | 50 | 200 | 400
---|---|---|---|---|---
F | 62.5 | 59.3 | 57.1 | 57.4 | 57.2
Avg.Accu | 53.5 | 54.9 | 55.8 | 55.8 | 55.8
Table S1: Continual learning performance on ciFAIR-10 as a function of
interleave divisions (Inter. Div.). The higher Avg. Accu., the better; the
lower F, the better. The best are in bold. See Experiments Section for
evaluation metrics and experiments in interleave divisions. Figure S1:
Continual learning performance on ciFAIR-10 dataset under different rehearsal
sequences. Average classification accuracy and Forgetfullness scores (F) over
all tasks for models trained with different rehearsal sequences are reported.
The higher the average accuracy, the better. The lower the F, the better. The
design conventions follow the ones in Figure 1.  Figure S2: Continual learning
performance on ciFAIR-100 dataset under different rehearsal sequences.
Complementary to the average accuracy results in Figure 1, we also report the
results of F scores on the ciFAIR-100 dataset. The results of Avg. Accu. are
duplicated from Figure 1 and are also shown here again for completeness.
Figure S3: Continual learning performance on ciFAIR-10 dataset under different
replay sample selection strategies. Average classification accuracy and
Forgetfullness scores (F) over all the tasks for models trained with different
rehearsal samples based on different selection strategies are reported. The
higher the average accuracy, the better. The lower the F, the better. The
design conventions follow the ones in Figure 2.  Figure S4: Continual learning
performance on ciFAIR-100 dataset under different replay sample selection
strategies. Complementary to the average accuracy results in Figure 2, we also
report the results of F scores on the ciFAIR-100 dataset. The results of Avg.
Accu. are duplicated from Figure 2 and are also shown here again for
completeness.


# Balancing Fairness and Accuracy in Sentiment Detection using Multiple Black
Box Models

Abdulaziz A. Almuzaini  Department of Computer ScienceRutgers UniversityNew
Brunswick, NJ<EMAIL_ADDRESS>and  Vivek K. Singh  School of Communication
and InformationRutgers UniversityNew Brunswick, NJ<EMAIL_ADDRESS>
(2020)

###### Abstract.

Sentiment detection is an important building block for multiple information
retrieval tasks such as product recommendation, cyberbullying detection, and
misinformation detection. Unsurprisingly, multiple commercial APIs, each with
different levels of accuracy and fairness, are now available for sentiment
detection. While combining inputs from multiple modalities or black-box models
for increasing accuracy is commonly studied in multimedia computing
literature, there has been little work on combining different modalities for
increasing fairness of the resulting decision. In this work, we audit multiple
commercial sentiment detection APIs for the gender bias in two-actor news
headlines settings and report on the level of bias observed. Next, we propose
a ”Flexible Fair Regression” approach, which ensures satisfactory accuracy and
fairness by jointly learning from multiple black-box models. The results pave
way for fair yet accurate sentiment detectors for multiple applications.

sentiment analysis, black-box models, fairness, fusion

††copyright: acmcopyright††journalyear: 2020††doi:
10.1145/1122445.1122456††conference: FATE/MM ’20: ACM MM 2020; October, 2020;
US††booktitle: FATE/MM ’20, October, 2020, US††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Ensemble methods††ccs:
Computing methodologies Regularization

##  1\. Introduction

There is an increasing concern amongst researches that machine learning models
developed with the best of intentions may exhibit biases, promote inequality,
or perform unfairly for unprivileged groups. With the increasing usage of such
models, a considerable amount of research has been conducted to recognize and
address these issues and their social impact (Mehrabi et al., 2019; Sun et
al., 2019). When models show signs of bias, then these models are referred as
”unfair”.

Sentiment detection is an important building block for multiple applications
such as content moderation, product recommendation, misinformation detection
and recently language generation (Park et al., 2018; Hutchinson et al., 2020;
Raisi and Huang, 2019; Wolf et al., 2019). To build these models, large
training corpora from varied sources are used. Unfortunately, training data
might already contain some bias and that bias can transmit to the learning
phase; thus unprivileged groups get unfairly impacted. While the developers of
such models often assess success by measuring the accuracy factor, a few have
examined the fairness aspect.

Due to the aforementioned popularity, large companies such as Google 111Google
Cloud https://cloud.google.com/natural-language, Amazon 222Amazon Comprehend
https://aws.amazon.com/comprehend/ and IBM 333Ibm Watson
https://www.ibm.com/watson provide black-box models that can be easily
incorporated into any applications to provide a sentiment score for a given
text. Although sentiment detection plays a significant role in such tasks,
potential discriminatory treatments might exist for different populations and
since these providers are perceived as trustworthy, a severe impact could hurt
large populations (O’neil, 2016). For example, when a model often predicts a
text snippet as toxic when a female pronoun is present but fails to do so for
the male pronoun, this could amplify stereotypes, disenfranchise certain
groups, and yield systemic misogyny (or conversely misandry) (Singh et al.,
2020b).

An important reason for success in multimedia computing is having multiple
models (often involving different datasets) combined together to achieve
better results. This approach has been widely used for combining multiple weak
forecast models, such as in weather forecast. Similarly, for machine learning,
”ensemble learning” and ”multi-modal fusion” methods show astonishing results
in terms of co-learning and accuracy enhancement (Mendes-Moreira et al., 2012;
Atrey et al., 2010). Recently, ensemble deep learning models have been
utilized for various applications such as image, video, and speech recognition
(Lee et al., 2017; Deng and Platt, 2014).

Recognizing the importance of fairness in such applications, multiple
researchers have proposed various metrics and bias mitigation methods (Mehrabi
et al., 2019; Sun et al., 2019; Nozza et al., 2019; Alasadi et al., 2019).
Many such methods either massage the incoming data (pre-processing approaches)
or change the optimization parameters within the white-box machine learning
model (in-processing) (Calmon et al., 2017). As both of these are not easily
possible in multiple applications (e.g., news sentiment detection using Google
API), we focus on a post-processing approach for combining the results from
multiple black box APIs (which we also refer to as ”modalities” in this work).

In this paper, we examine the fairness aspect of three popular sentiment
detection black-box models on crime news headlines in which misogynistic
and/or misandristic bias might exist and we propose a method to mitigate this
bias by combining the output from different models. Specifically, we examine
the sentiment detection across ”gendered interaction” in news, such as _”a
woman hurts a man in a bus”_ vs _”a man hurts a woman in a bus”_. Here,
”gendered interaction” indicates that there are two actors both with clearly
identified and differing gender, and there is an interaction taking place
between them. In such settings, a model that produces a positive score for the
first sentence but a negative score for the second is considered biased
towards women (perpetrators) and unfair for men (victims) and vice versa. We
apply these tests on crime news headlines dataset that has been collected
specifically for this study. Experimental results show each of the publicly
available APIs has inherent gender bias and also inaccuracies. On the positive
side, the proposed ”Flexible Fair Regression” approach was found to be useful
to ameliorate both fairness and accuracy concerns.

Our main contributions in this paper are:

  * •

To examine the fairness aspect of publicly available sentiment detection APIs
that have been used extensively in various applications. We report that each
of these models have inherent gender bias.

  * •

To propose an optimization method ”Flexible Fair Regression” to easily allow
balancing between bias and accuracy when combining the outputs from multiple
(semi-accurate and semi-biased) black box models.

  * •

To share the newly created approach and resulting dataset for quantifying bias
in ”gendered interaction” scenarios
444https://github.com/abdulazizasz/fairness_sentiment.

Note that we consider the use of binary gender as a limitation of this work.
The use of gender neutral pronouns and those inclusive of non-binary
identities is still not common enough in news headlines and hence the problem
of bias with binary gendered pronouns remains an important challenge
(Badjatiya et al., 2019; Nozza et al., 2019).

The rest of the paper is organized as follows. Section 2 provides an overview
of the related work in bias detection and mitigation. Then, Section 3
describes the methods including the bias measurement approach and the proposed
bias mitigation strategy. The experimental setup and results are provided in
Sections 4 and 5. Finally, in Section 6, a summary and future directions are
shared.

##  2\. Related Work

There is significant research work devoted to fairness in algorithms and they
can be divided into two categories: (1) bias measurement, and (2) bias
mitigation.

For the first category, a commonly used strategy is to compare the treatment
differences of different privileged groups (Kiritchenko and Mohammad, 2018;
Rudinger et al., 2017). Muliple efforts use the approach of ”word-swapping”
where different sensitive variables are swapped in a fixed context. For
example, in (Park et al., 2018) authors use a template ”I hate ¡identity¿
people” by replacing the sensitive variable with different identities such as
(gay, jewish, african, etc). This process has been used to measure bias in
sentiment detection (Shen et al., 2018; Kiritchenko and Mohammad, 2018),
coreference resolution (Rudinger et al., 2018) and language models (Huang et
al., 2019). This technique provides a practical approach to examine the
treatment for different sentences when a sensitive variable plays an important
role in forming the sentence. Another set of metrics that have been widely
used for classification tasks are measuring the differences of accuracy, False
Positive Rate (FPR), False Negative Rate (FNR) and so on (Park et al., 2018;
Dixon et al., 2018). Unfortunately, these metrics are only applicable for
classification tasks, whereas in sentiment detection the output value is
typically a continuous score, therefore different statistical measures must be
utilized to examine unfair treatment.

Multiple studies have examined unfair treatments in settings in which a single
sensitive variable exists in a text such as (”I hate women”, or ”Jews are
bad”). Additionally, Rudinger et al., (Rudinger et al., 2018) in coreference
resolution examine the bias of an inferred pronoun in a text that has (two
actors). However, none have studied the bias issue in sentiment detection in
cases where there are two different actors involved in an interaction.

For the second category, bias mitigation strategies can be used in different
levels: pre-processing, in-processing and post-processing. Calmon et al.,
(Calmon et al., 2017) propose a de-biasing method which uses a probabilistic
transformation that edits the features and labels in the data with group
fairness. Another pre-processing method presented by Zemel et al., (Zemel et
al., 2013) focuses on learning a fair representation technique that finds a
latent representation which encodes the data well but obfuscates information
about sensitive attributes. Kamishima et al., (Kamishima et al., 2012) propose
an in-processing algorithm known as Prejudice Remover to decrease bias by
adding a discrimination–aware regularization term to the learning objective
function. Celis et al., (Celis et al., 2019) put forward the idea of a meta
fair classifier that takes fairness metric as part of the input and returns a
classifier optimized with respect to a fairness metric. Other efforts such as
(Pleiss et al., 2017; Hardt et al., 2016) bring forward the idea of calibrated
equalized odds which is a post-processing technique that optimizes over
calibrated classifier score.

Past work on multimodal fusion has proposed methods to enhance the accuracy by
combining multiple modalities in data-levels, feature-levels or the decision-
levels (Mendes-Moreira et al., 2012; Atrey et al., 2010). These methods are
also feasible in tasks when dealing with different black-box models that have
different levels of accuracy and different level of bias. Although these
approaches have shown great success in the past, they are yet to be studied
with the goal of fairness enhancement. Fairness in multimedia computing is a
relatively nascent but fast-growing (Alasadi et al., 2019; Singh et al.,
2020a) field and this work helps motivate and ground the need for fairness
considerations in fusion research. The approach in this paper is inspired by
multi-modal fusion proposals for co-learning with weak learners to enhance the
accuracy (Mendes-Moreira et al., 2012; Atrey et al., 2010; Deng and Platt,
2014; Lee et al., 2017). It also adapts the in-processing technique for bias
mitigation using a regularizer to create a post-processing approach which can
work well with multiple black box models in sentiment detection case.

##  3\. Methodology

###  3.1. Preliminaries

We formulate the problem of fair sentiment detection as the following. We have
$k$ independent black-box models and their sentiment scores $x_{k}\in[-1,1]$
and a ground truth score $y\in[-1,1]$. Besides that, let there be a sensitive
variable $S$ that can divide the dataset $D=\\{x_{i},y_{i}\\}^{N}_{i=1}$ into
different groups, e.g., $S_{male}$, $S_{female}$. To simplify the settings, we
can combine multiple $k$ column vectors modalities $\\{x_{1},\ x_{2},\
...x_{k}\\}$ in a matrix $X$.

In such a setting the goal of the algorithm is to minimize the loss as
measured via a combination of accuracy error, bias penalty, and over-fitting
penalty. We describe the operationalization of the different terms above in
the following subsections.

###  3.2. Measuring Bias

To measure the bias/fairness, we examine the mean difference of the scores for
each black-box models with respect to a sensitive variable $S$, e.g., for a
binary variable ${S^{+},S^{-}}$ as follows:

$Mean\ Difference\ (x,k)\ =\frac{\sum\limits_{x_{i}\in
S^{+}}x^{k}_{i}}{|S^{+}|}-\frac{\sum\limits_{x_{i}\in
S^{-}}x^{k}_{i}}{|S^{-}|}$

A fair black-box model will result in zero score which means the same
sentiment score has been produced regardless of the sensitive variable. Since
we are focusing on the scores deviation among different groups we can use the
Mean Absolute Difference/Error (MAE). Other methods proposed in the literature
for measuring the bias can also be useful candidates (e.g. Correlation and
delta of prediction accuracy (Calders et al., 2013)) in other settings.

###  3.3. Balancing Accuracy and Fairness

In this project, we use linear regression to find the best parameter to
combine these independent black-box models $X$ with respect to the target
scores $y$. The linear regression can be formulated as an optimization problem
to find the parameter $w$ that minimizes the following function:

(1) $\displaystyle\underset{w}{\textbf{minimize}}$ $\displaystyle
MSE(w)=\frac{1}{N}\sum_{i=1}^{N}(w*x_{i}-y_{i})^{2}$

Many variants of linear regression add a regularizer function to the
regression, which keeps a check on the number of parameters being used in the
modeling (Hoerl and Kennard, 1970). Further, recent efforts have proposed
adding a ”fairness regularizer” to the regression (Berk et al., 2017). Here,
we adapt that approach to define a regularizer which penalizes the function
when the sentiment scores differ between different groups. For a binary
variable, a model regularizes the difference between $S^{+}$ and $S^{-}$.

In other words, the modified objective function is trying to find the optimal
parameter $w$ that minimizes the Mean Squared Error (MSE) along with the
minimum bias between different groups. To simplify the objective function, a
bias matrix $\Delta$ which contains the sentiment score difference for each
modality is calculated. For instance, for a modality $k$ and a binary
sensitive variable $S$, the bias vector can be calculated as follows:

$\delta_{k}=|x_{S+}^{k}-x_{S-}^{k}|$

Similarly, calculating the bias for other modalities and combining them in one
matrix $\Delta$ yields:

$\displaystyle\Delta=\begin{bmatrix}&&\\\ \delta_{i}^{1}&...&\delta_{i}^{k}\\\
&&\end{bmatrix}$

Using the $w$ and $\Delta$ the ”fairness” penalty function $P$ is:

$P(w)=\frac{1}{N}\sum_{i=1}^{N}(w*\delta_{i})^{2}$

Now the optimization function for ”Flexible Fair Regression” is:

(2) $\displaystyle\underset{w}{\textbf{minimize}}$ $\displaystyle
L(w)=MSE(w)+\beta P(w)+\lambda\|w\|^{2}$

where $\|w\|^{2}$ is the $L_{2}$ norm. Hyper parameters $\lambda$ and $\beta$
are used to control over-fitting and the fairness trade-off. We call this
approach ”flexible fair regression” as it supports fairness in regression, and
allows a system designer to flexibly pick the relative importance and
thresholds that they want to assign to fairness and accuracy.

##  4\. Experimental Setup

In this section, we describe the process of the data-collection, annotations,
baselines, and the bias-reduction results.

Figure 1. Constructing a template, the two versions and getting the APIs
scores

###  4.1. Dataset

To construct a dataset, we collected crime news headlines from Google News API
555https://news.google.com. To do so, we used the API search criteria to
retrieve only the news headlines that contain abusive verbs such as (_kill,
murder, slap_ , etc) along with at least two different subjects (_man,
woman_). Using a carefully designed list of abusive verbs (Wiegand et al.,
2018), we collect a large number of data-points (crime news). Since in this
project we are tackling the predictive learning problem as a linear
regression, we need to label and assign a sentiment score to the collected
dataset. Utilizing Figure-Eight 666www.figure-eight.com, we asked 10
annotators to label and score each sentence (template) and to avoid bias in
the annotation process, we anonymize the subjects in the sentence. Then, every
annotator is presented with an anonymized template (see Fig. 1) and is asked
to provide two pieces of information following the Valence-Arousal model
(Russell, 1980): (1) A valence label to a sentence such as ”Positive or
Negative” (2) An arousal score on a scale from 1 to 10. Thus, a sentence with
a positive label and a score of 10 will have a sentiment score close to +1
whereas a sentence with a negative label and an arousal score of 5 will have a
sentiment score -0.5. This process yielded scores in the range of [-1, 1].

Removing inconsistent annotators, and using a seed of 200 templates, we did
gender swapping in the sentences where the first sentence has men as
perpetrators and women as victims and vice-versa. By applying different gender
identities such as (”man-woman”, ”male-female”, etc), we used 25 of such terms
provided by (Gonen and Goldberg, 2019). Thus, our corpus contains 10000 news
headline sentences. The resulting dataset was then scored using Google,
Amazon, and IBM APIs (see Fig. 1). Finally, we split the dataset into training
and testing sets in a ratio of 70:30.

A sample of a dataset is shown in Table 1. For each black box model, we have a
sentiment score for the two versions of the template along with the gender and
the ground-truth score. From Table 1, we get the scores for each template by
taking the average between two genders since we are aiming at finding the
optimal score for each template regardless of the gender (see Table 2).

Table 1. A sample from the dataset Sentence | $k_{1}$ | $k_{2}$ | $k_{3}$ | $S$ | $y$
---|---|---|---|---|---
man hurts woman in .. | $-0.9$ | $-0.5$ | $0.6$ | $m$ | $-0.7$
woman hurts man in .. | $-0.7$ | $-0.8$ | $-0.9$ | $f$ | $-0.7$
Table 2. Training dataset for each template regardless of the gender Sentence | $k_{1}$ | $k_{2}$ | $k_{3}$ | $y$
---|---|---|---|---
”[S1] hurts [S2] in .. | $-0.8$ | $-0.65$ | $0.15$ | $-0.7$

###  4.2. Baselines

Since the scores that are generated from these black-box models are not always
accurate, a fusion process is used to increase the accuracy by combining
different modalities. Related works (e.g., (Atrey et al., 2010; Mendes-Moreira
et al., 2012)) provide various methods that are practical for fusing
independent modalities. In this project we are experimenting with three such
methods:

Unweighted Average is the basic fusion process that assumes that independent
modalities are equal in terms of accuracy. Thus, the predicted sentiment score
is calculated as follows:

$\hat{y_{i}}=\frac{1}{N}(x_{i}^{google}+x_{i}^{amazon}+x_{i}^{ibm})$

Weighted Average weights each modality based on its accuracy (for the training
set):

$\hat{y_{i}}=w_{Google}*x_{i}^{google}+w_{Amazon}*x_{i}^{amazon}+w_{Ibm}*x_{i}^{ibm}$

where $\sum_{k=1}^{3}w_{k}=1$.

Multiple Regression is similar to our proposed method but without the
”fairness” penalty term. In other word, black-box models outputs can be
treated as features for another learning model.

All of the above methods are only used to optimize for the accuracy part and
not considering the fairness aspect.

Fairness Optimization is an additional baseline that optimizes (only) for
fairness. We weight each modality by its fairness scores in the training data.
The ”fairness” weight for a modality $k$ is calculated as follows:

(3) $w^{k}=\sum_{i=1}^{N}\
\mathds{1}\\{|x_{male_{i}}^{k}-x_{female_{i}}^{k}|\leq\tau\\}\ $

Here, if the sentiment scores differ by less than $\tau=10\%$, we consider it
as a fair treatment for that template.

##  5\. Results And Discussion

###  5.1. Auditing Sentiment Detection APIs

As a first step, we analyzed the results from the different black box models
(sentiment detection APIs) to see if they are accurate and if there is a
difference in the results obtained for sentences which are same, except for
the genders represented for the perpetrators and the victims.

To evaluate accuracy (compared to ground truth labels obtained from multiple
human labelers) we use Root Mean Squared Error (RMSE). To measure bias we use
Mean Absolute Error (MAE). Table 3 shows the errors in accuracy and bias
levels for each individual modality. The range for sentiment scores was [-1,
1] and we see that there are noteworthy accuracy and bias issues with each
modality. Further, a pairwise t-test comparing the mean sentiment scores
across genders yielded statistically significant differences in all three
modalities. These issues motivate the need to combine the outputs from
different modalities to improve both accuracy and fairness.

Table 3. Accuracy and Fairness in the original dataset API | Acc. Error | Bias
---|---|---
Google | $0.5611$ | $0.0590$
Amazon | $0.6939$ | $0.0581$
IBM | $0.7441$ | $0.0545$

###  5.2. Improving Accuracy and Fairness

Table 4. Mitigation Methods Analysis Model Name | Acc. Error | Bias
---|---|---
Multiple Regression | 0.5362 | 0.0738
Unweighted Average | 0.6302 | 0.0435
Weighted Average | 0.6153 | 0.0447
Fairness Optimization | 0.7051 | 0.0173
Our Method | 0.6026 | 0.0400

We implemented the Flexible Fair Regression approach (Eq. 2) on the created
dataset using Python. Table 4 provides the summary of the results. We can
easily see the trade-off between the accuracy error and bias among these
models (lower is better in both cases). Multiple Regression is performing well
in terms of accuracy (Low RMSE) but has to contend with higher value for bias.
Both Weighted Average and Unweighted Average methods yield higher errors in
terms of accuracy than Multiple Regression but yield lower levels of bias.
Note that it is also possible to optimize only for fairness (Fairness
Optimization), and this can reduce the bias to very low value (close to zero).
However, this comes with the price of a high accuracy error (see Fig. 2).

Lastly, our method allows for flexible trade-off between accuracy and fairness
(see Eq. 2). The trade-off depends on the choice of weight parameter $\beta$
and different values of $\beta$ yield different points on the purple curve in
Fig. 3. The axes for the figure are accuracy error and bias levels. Lower is
better for both of them and hence, the points on the lower left corner are
ideal. As shown, all the points of the purple curve (our approach with
different $\beta$ values) either coincide with other baselines or strictly
dominate them (i.e., yield better performance in terms of both accuracy and
fairness).

Figure 2. The accuracy error (RMSE) and the bias score (MAE) is shown for
different baselines methods along with our method. The lower the bar the
better. _”UA: Unweighted Average, WA: Weighted Average, FO: Fairness
Optimization, ML: Multiple Regression, OM: Our Method”_

Figure 3. Using the intersected point, we choose the closest $\beta$ which is
$\beta=.002$ in Our Method

Figure 4. Trade-off between accuracy and fairness by manipulating the
parameter $\tau$ (e.g., $\tau=10\%$)

The Multiple Regression and Fairness Optimization approaches can be considered
extreme cases of our flexible fair regression approach, such that they
optimize only for accuracy or only for fairness. Any point on the purple curve
(varying values of $\beta$) will provide a trade-off between these two
extremes. To find a specific suitable candidate for $\beta$ value, we use
Figure 3 to jointly consider the achievements of Fairness Optimization (FO)
and Multiple Regression (MR). Hence, an ideal solution (”Utopia Point” marked
as ”X” in Fig. 3; unlikely to be achievable in practice), will yield accuracy
error as low as MR and bias as low as FO (see Fig. 3). Hence, we consider the
point closest (minimum distance) to the ”Utopia Point” to be a suitable
candidate to pick the $\beta$ value. In the current work, $\beta$ value of
0.002 is the closest point, which yields the results shown in Table 4 and
Figure 2. Note that this result is pareto-optimal in the sense that there is
no other feasible point that is lower in both accuracy error and bias. This
can be seen from the points in Fig. 3 (which also includes points for models
which use just one modality) and Table 4.

Another possible approach to joint optimization of two factors is to budget a
fixed ”cost of fairness” (Berk et al., 2017). For example, losing a portion of
accuracy could lead to a gain in fairness. In Figure 4, we provide an
illustration of trade-off; a ${\sim}10\%$ accuracy loss could yield a
${\sim}38\%$ reduction in bias. Hence, a practitioner with an assigned $10\%$
accuracy budget could gain up to $38\%$ in terms of fairness. Other plausible
budgets and impact can also be easily computed using this approach to allow
for such decision making.

##  6\. Conclusion

In this work, we deploy a regularized objective function that combines
independent black box models to ensure an accurate and a fair learning model
for sentiment detection. Since we are dealing with disparate and independent
black-box models, a fusion process helps combine multiple sources’ results and
build a more robust score for each template regardless of the subjects’
genders. The proposed approach yields a family of pareto-optimal solutions
compared to other baseline approaches. Further, our ”fairness” penalty
function performs well in terms of bias reduction and is more flexible
compared to other baselines.

An important limitation of this work is the focus on binary genders. Another
critical challenge is that of constructing large and practical dataset, i.e.,
templates that cover enough amount of context in which abusive verbs occur. In
this study, we only investigate how sentiment detection APIs are dealing with
crime news headlines. Additionally, the annotation process might result in
biased scores; thus, to mitigate this bias, we take the average of the scores
from different annotators and removed inconsistent annotators. Lastly, since
we are solving an optimization function, a convexity of equation is assumed
and a solver has been used to find the optimal minima.

Despite the limitations, this paper marks a significant step forward toward
fairness and accuracy in sentiment detection literature. The paper advances
the fairness literature to consider multiple actor ”gendered interactions”,
which has use cases in news analysis, abuse detection, and misinformation
detection. The public dataset and the proposed flexible approach can allow for
fairness in a wide variety of scenarios where semi-accurate and semi-fair
black box models need to be combined to obtain fair yet accurate predictions.

## Acknowledgments

This material is in part based upon work supported by the National Science
Foundation under Grant SES-1915790.

## References

  * (1)
  * Alasadi et al. (2019) Jamal Alasadi, Ahmed Al Hilli, and Vivek K Singh. 2019\.  Toward Fairness in Face Matching Algorithms. In _Proceedings of the 1st International Workshop on Fairness, Accountability, and Transparency in MultiMedia_. 19–25.
  * Atrey et al. (2010) Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb El Saddik, and Mohan S Kankanhalli. 2010.  Multimodal fusion for multimedia analysis: a survey.  _Multimedia systems_ 16, 6 (2010), 345–379.
  * Badjatiya et al. (2019) Pinkesh Badjatiya, Manish Gupta, and Vasudeva Varma. 2019\.  Stereotypical bias removal for hate speech detection task using knowledge-based generalizations. In _The World Wide Web Conference_. 49–59.
  * Berk et al. (2017) Richard Berk, Hoda Heidari, Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, Seth Neel, and Aaron Roth. 2017\.  A convex framework for fair regression.  _arXiv preprint arXiv:1706.02409_ (2017).
  * Calders et al. (2013) Toon Calders, Asim Karim, Faisal Kamiran, Wasif Ali, and Xiangliang Zhang. 2013.  Controlling attribute effect in linear regression. In _2013 IEEE 13th international conference on data mining_. IEEE, 71–80.
  * Calmon et al. (2017) Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. 2017.  Optimized pre-processing for discrimination prevention. In _Advances in Neural Information Processing Systems_. 3992–4001.
  * Celis et al. (2019) L Elisa Celis, Lingxiao Huang, Vijay Keswani, and Nisheeth K Vishnoi. 2019.  Classification with fairness constraints: A meta-algorithm with provable guarantees. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_. 319–328.
  * Deng and Platt (2014) Li Deng and John C Platt. 2014.  Ensemble deep learning for speech recognition. In _Fifteenth Annual Conference of the International Speech Communication Association_.
  * Dixon et al. (2018) Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018.  Measuring and mitigating unintended bias in text classification. In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_. 67–73.
  * Gonen and Goldberg (2019) Hila Gonen and Yoav Goldberg. 2019.  Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them.  _arXiv preprint arXiv:1903.03862_ (2019).
  * Hardt et al. (2016) Moritz Hardt, Eric Price, and Nati Srebro. 2016.  Equality of opportunity in supervised learning. In _Advances in neural information processing systems_. 3315–3323.
  * Hoerl and Kennard (1970) Arthur E Hoerl and Robert W Kennard. 1970.  Ridge regression: Biased estimation for nonorthogonal problems.  _Technometrics_ 12, 1 (1970), 55–67.
  * Huang et al. (2019) Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2019.  Reducing sentiment bias in language models via counterfactual evaluation.  _arXiv preprint arXiv:1911.03064_ (2019).
  * Hutchinson et al. (2020) Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020.  Unintended machine learning biases as social barriers for persons with disabilitiess.  _ACM SIGACCESS Accessibility and Computing_ 125 (2020), 1–1.
  * Kamishima et al. (2012) Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. 2012.  Fairness-aware classifier with prejudice remover regularizer. In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 35–50.
  * Kiritchenko and Mohammad (2018) Svetlana Kiritchenko and Saif M Mohammad. 2018.  Examining gender and race bias in two hundred sentiment analysis systems.  _arXiv preprint arXiv:1805.04508_ (2018).
  * Lee et al. (2017) Inwoong Lee, Doyoung Kim, Seoungyoon Kang, and Sanghoon Lee. 2017\.  Ensemble deep learning for skeleton-based action recognition using temporal sliding lstm networks. In _Proceedings of the IEEE international conference on computer vision_. 1012–1020.
  * Mehrabi et al. (2019) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2019\.  A survey on bias and fairness in machine learning.  _arXiv preprint arXiv:1908.09635_ (2019).
  * Mendes-Moreira et al. (2012) Joao Mendes-Moreira, Carlos Soares, Alípio Mário Jorge, and Jorge Freire De Sousa. 2012\.  Ensemble approaches for regression: A survey.  _Acm computing surveys (csur)_ 45, 1 (2012), 1–40.
  * Nozza et al. (2019) Debora Nozza, Claudia Volpetti, and Elisabetta Fersini. 2019\.  Unintended bias in misogyny detection. In _IEEE/WIC/ACM International Conference on Web Intelligence_. 149–155.
  * O’neil (2016) Cathy O’neil. 2016\.  _Weapons of math destruction: How big data increases inequality and threatens democracy_.  Broadway Books.
  * Park et al. (2018) Ji Ho Park, Jamin Shin, and Pascale Fung. 2018.  Reducing gender bias in abusive language detection.  _arXiv preprint arXiv:1808.07231_ (2018).
  * Pleiss et al. (2017) Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. 2017.  On fairness and calibration. In _Advances in Neural Information Processing Systems_. 5680–5689.
  * Raisi and Huang (2019) Elaheh Raisi and Bert Huang. 2019.  Reduced-Bias Co-trained Ensembles for Weakly Supervised Cyberbullying Detection. In _International Conference on Computational Data and Social Networks_. Springer, 293–306.
  * Rudinger et al. (2017) Rachel Rudinger, Chandler May, and Benjamin Van Durme. 2017\.  Social bias in elicited natural language inferences. In _Proceedings of the First ACL Workshop on Ethics in Natural Language Processing_. 74–79.
  * Rudinger et al. (2018) Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018.  Gender bias in coreference resolution.  _arXiv preprint arXiv:1804.09301_ (2018).
  * Russell (1980) James A Russell. 1980\.  A circumplex model of affect.  _Journal of personality and social psychology_ 39, 6 (1980), 1161.
  * Shen et al. (2018) Judy Hanwen Shen, Lauren Fratamico, Iyad Rahwan, and Alexander M Rush. 2018.  Darling or babygirl? investigating stylistic bias in sentiment analysis.  _Proc. of FATML_ (2018).
  * Singh et al. (2020a) Vivek K Singh, Elisabeth André, Susanne Boll, Mireille Hildebrandt, and David A Shamma. 2020a.  Legal and ethical challenges in multimedia research.  _IEEE MultiMedia_ 27, 2 (2020), 46–54.
  * Singh et al. (2020b) Vivek K Singh, Mary Chayko, Raj Inamdar, and Diana Floegel. 2020b.  Female Librarians and Male Computer Programmers? Gender Bias in Occupational Images on Digital Media Platforms.  _Journal of the Association for Information Science and Technology_ (2020).
  * Sun et al. (2019) Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019\.  Mitigating gender bias in natural language processing: Literature review.  _arXiv preprint arXiv:1906.08976_ (2019).
  * Wiegand et al. (2018) Michael Wiegand, Josef Ruppenhofer, Anna Schmidt, and Clayton Greenberg. 2018.  Inducing a lexicon of abusive words–a feature-based approach.  (2018).
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019\.  Transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ (2019).
  * Zemel et al. (2013) Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013.  Learning fair representations. In _International Conference on Machine Learning_. 325–333.