# Exploiting Semantic Role Contextualized Video Features
for Multi-Instance Text-Video Retrieval
EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022

Burak Satar1,2
Zhu Hongyuan1
Hanwang Zhang2
Joo Hwee Lim1,2 1Institute for Infocomm Research, A*STAR, Singapore  2School
of Computer Science and Engineering, NTU, Singapore
{burak_satar, zhuh<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this report, we present our approach for EPIC-KITCHENS-100 Multi-Instance
Retrieval Challenge 2022. We first parse sentences into semantic roles
corresponding to verbs and nouns; then utilize self-attentions to exploit
semantic role contextualized video features along with textual features via
triplet losses in multiple embedding spaces. Our method overpasses the strong
baseline in normalized Discounted Cumulative Gain (nDCG), which is more
valuable for semantic similarity. Our submission is ranked 3rd for nDCG and
ranked 4th for mAP.

##  1 Introduction

With the rise of videos uploaded by users via social media channels, cross-
modal retrieval of video data and natural language descriptions has gained
popularity. The goal of video-to-text retrieval, given a query action segment,
is to rank captions in a gallery set so that those with a higher rank are more
semantically related to the video action. Text-to-video retrieval, on the
other hand, ranks videos based on a query caption.

While most methods [8, 10, 3] use one joint embedding space to align video and
text features, recent methods [1, 14] use multiple embedding spaces to match
video features into the noun and verb embedding spaces along with the textual
features but did not consider their interactions. Moreover, it is relatively
easy to parse the textual features into verb and noun levels since an off-the-
shelf toolkit could be used. However, mapping a video feature into the object
and action levels is still challenging, which corresponds to noun and verb
levels in text.

Inspired by [11, 5], we implement self-attentions to exploit visual features
on top of the baseline, JPoSE [14] by leveraging the contexts from nouns and
verbs of the text query with details in the following section. While we
outperform the strong baseline in normalized Discounted Cumulative Gain
(nDCG), which is more beneficial for semantic similarity, we fall short in
mean Average Precision (mAP), a traditional technique for binary relevance.
Our approach is ranked third for nDCG, while it is ranked fourth for mAP. We
also analyze various failure examples to save the time of the following
researchers on this task.

##  2 Method

We follow the baseline work [14]: we create a pair of functions that map
videos and texts into a joint embedding space, in which embeddings for matched
texts and videos should be close together, and embeddings for mismatched texts
and videos should be far apart, given a video and a query text. A suitable
embedding space should also ensure that related videos/texts stay close
together.

With this motivation, we first parse caption into the noun $t_{i}^{1}$ and
verb $t_{i}^{2}$ levels, followed by linear layers. We utilize linear layers
to embed corresponding video features $v_{i}^{1}$, $v_{i}^{2}$ and use a self-
attention layer to exploit contextualized features. Then, we concatenate
textual and visual features to compute the distance between these
representations, $\hat{v_{i}}$ and $\hat{t_{i}}$. L1 and L2 refer to triplet
losses. The more details of the loss functions are in Eq. 1 and baseline paper
[14] and the architecture details are in Fig. 1

In Eq. 1, the first two rows refer to cross-modal losses, and the last two
rows indicate within-modal losses. $\theta$ function denotes two fully
connected layers. $\delta$ function signifies two linear layers and one self-
attention layer. $m$ refers to the constant margin, while $d$ is the distance
function. While $i$ refers to the selected video, $j$ and $k$ denote positive
and negative samples, respectively.

Figure 1: We first parse caption into the noun $t_{i}^{1}$ and verb
$t_{i}^{2}$ levels, followed by linear layers. We utilize linear layers to
embed corresponding video features $v_{i}^{1}$, $v_{i}^{2}$ and use a self-
attention layer to exploit contextualized features. Then, we concatenate
textual and visual features to compute the distance between these
representations, $\hat{v_{i}}$ and $\hat{t_{i}}$. L1 and L2 refer to triplet
losses.

$\begin{split}L&=\lambda_{v,t}\sum_{i,j,k}\max(0,d(\delta_{v_{i}},\theta_{t_{j}})-d(\delta_{v_{i}},\theta_{t_{k}})+m)\\\
&+\lambda_{t,v}\sum_{i,j,k}\max(0,d(\theta_{t_{i}},\delta_{v_{j}})-d(\theta_{t_{i}},\delta_{v_{k}})+m)\\\
&+\lambda_{v,v}\sum_{i,j,k}\max(0,d(\delta_{v_{i}},\delta_{v_{j}})-d(\delta_{v_{i}},\delta_{v_{k}})+m)\\\
&+\lambda_{t,t}\sum_{i,j,k}\max(0,d(\theta_{t_{i}},\theta_{t_{j}})-d(\theta_{t_{i}},\theta_{t_{k}})+m)\end{split}$
(1)

Eq. LABEL:encoder shows that the visual features $V_{i}$ are fed into the
self-attention layer to encode into $z_{s}$. Then, a feed-forward layer FF
outputs the final contextualized appearance feature. Normalization of the
layer is done under the Norm function.

$\displaystyle
z_{s}=\textrm{Norm}\big{(}\textrm{MultiHead}(V_{i},V_{i},V_{i})+V_{i}\big{)}$
(2) $\displaystyle E_{v}=\textrm{Norm}\big{(}\textrm{FF}(z_{s})+z_{s}\big{)}$

For multi-headed attention layers, we follow [12], as formulated in Eq. 3. All
of the W matrices are learned during the training procedure. Since it is a
self-attention layer, query Q is the same as key K and value V. After each
attention layer, layer normalization and the residual connection are
implemented.

$\displaystyle\textrm{MultiHead}(Q,K,V)=\textrm{Concat}(\textrm{Head}_{1},...,\textrm{Head}_{h})W^{O}$
(3)
$\displaystyle\textrm{Head}_{i}=\textrm{Attention}(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V})$
$\displaystyle\textrm{Attention}(Q,K,V)=\sigma\Big{(}\frac{QK^{T}}{\sqrt{d}}V\Big{)}$

Table 1: Multi-instance retrieval results on the EPIC-KITCHENS-100 test split.
T2V and V2T stand for Text-to-Video and Video-to-Text retrieval, respectively.
While the above part of the table compares with baselines, the lower part
shares the result of this year’s competition.

Comparison to Baselines
---
| mean Average Precision (mAP) | normalised Discounted Cumulative Gain (nDCG)
Method | Average | T2V | V2T | Average | T2V | V2T
MI-MM | 27.58 | 23.08 | 32.09 | 42.10 | 40.48 | 43.72
JPoSE* | 43.95 | 38.18 | 49.71 | 53.40 | 51.60 | 55.21
JPoSE [14] | 44.01 | 38.11 | 49.91 | 53.53 | 51.55 | 55.51
DCRL [6] | 44.23 | 38.49 | 49.96 | 53.56 | 51.83 | 55.28
Our Method | 42.81 | 38.10 | 47.52 | 55.33 | 54.12 | 56.55
Comparison to Other Users
| mean Average Precision (mAP) | normalised Discounted Cumulative Gain (nDCG)
User | Average | T2V | V2T | Average | T2V | V2T
haoxiaoshuai | 44.02 (3) | 38.34 (3) | 49.69 (3) | 53.06 (4) | 51.31 (4) | 54.82 (4)
Our Method | 42.81 (4) | 38.10 (4) | 47.52 (4) | 55.33 (3) | 54.12 (3) | 56.55 (3)
afalcon | 49.77 (1) | 44.39 (1) | 55.15 (1) | 61.02 (2) | 58.88 (2) | 63.16 (2)
kevin.lin | 47.39 (2) | 40.95 (2) | 53.84 (2) | 61.44 (1) | 59.60 (1) | 63.29 (1)

##  3 Experiments

###  3.1 Implementation Details

While $\lambda_{t,v}$ equals 2.0, the other constant margins equal 1.0. The
batch size is 64, and the learning rate is 0.01.

Dataset. We undertake experiments on the EPIC-KITCHENS-100 dataset [2], which
is a collection of unscripted egocentric action data across the world, to
demonstrate the efficiency of our strategy.

Features. We use the video features extracted by TBN [7]. Each one is an
nx25x1024 matrix holding a python dictionary containing the RGB, flow, and
audio features, where n is the number of video clips. The number of training
and test set pairs is 67217 and 9668, respectively. Each feature is followed
by temporal mean pooling, making the shape nx1x1024. We utilize the textual
features given by [14] using a Word2Vec model trained on the Wikipedia corpus.
spaCy parser [4] is used to disentangle the text caption into different PoS
tags. The model is trained with the default values of the baseline.

Evaluation Metrics. We utilize two assessment metrics, mAP and nDCG, on the
test set to evaluate submissions for action retrieval. Mean Average Precision
(mAP) was employed for retrieval baselines because it allows the whole ranking
to be analyzed on binary relevance. nDCG has already been used to retrieve
information [13]. It necessitates the use of similarity scores throughout the
entire test set.

###  3.2 Results

Table 1 shows the comparison between our method and the baselines. It also
compares with the methods attended to this year’s challenge. While our method
overpass all the baselines on nDCG, it falls short on mAP. The MI-MM approach
projects both modalities onto a shared action space using linear layers via
max-margin loss, which is a simplified version of [9]. The JPoSE approach [14]
uses a triplet loss to separate captions into the verb and noun spaces. The
JPoSE* refers to our implementation. DCRL [6] considers both inter-modal and
intra-modal constraints at the same time to retain both cross-modal semantic
similarity and modality-specific consistency in the embedding space.

Failure cases. We also share failure cases which could be helpful for other
researchers. For every experiment, we give the results approximately compared
to baseline JPoSE [14]. 1) If we implement self-attention to the textual
features as it is done to the video features, the results decrease around
2-3%. 2) When we increase the batch size or embedding size, the results
decrease 1-2%. 3) We get 1-2% lower results when applying temporal max-pooling
rather than mean pooling.

##  4 Conclusion

In this report, we propose an approach to exploit contextualized video
features via self-attentions and disentangling them into multiple embedding
spaces. It also parses text into corresponding embedding spaces, and then the
similarity between representations is calculated via triplet loss. While our
strategy outperforms the strong baseline in normalized Discounted Cumulative
Gain (nDCG), a semantic similarity measurement, it falls short in mean Average
Precision (mAP), a standard measure of binary relevance. For nDCG, our
proposal is ranked third, and for mAP, it is ranked fourth. We plan to exploit
each video feature separately via novel fusion methods as well as utilize
domain-specific features such as hand-object relations for future work.

## References

  * [1] S. Chen, Y. Zhao, Q. Jin, and Q. Wu.  Fine-grained video-text retrieval with hierarchical graph reasoning.  In CVPR, 2020.
  * [2] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.  International Journal of Computer Vision (IJCV), 2021.
  * [3] J. Dong, X. Li, C. Xu, S. Ji, Y. He, G. Yang, and X. Wang.  Dual encoding for zero-example video retrieval.  In CVPR, pages 9338–9347, 2019.
  * [4] English spaCy parser.  https://spacy.io/.
  * [5] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid.  Multi-modal Transformer for Video Retrieval.  In European Conference on Computer Vision (ECCV), 2020.
  * [6] Xiaoshuai Hao, Yucan Zhou, Dayan Wu, Wanqian Zhang, Bo Li, and Weiping Wang.  Multi-feature graph attention network for cross-modal video-text retrieval.  In Proceedings of the 2021 International Conference on Multimedia Retrieval, ICMR ’21, page 135–143, New York, NY, USA, 2021. Association for Computing Machinery.
  * [7] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen.  Epic-fusion: Audio-visual temporal binding for egocentric action recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [8] Y. Liu, S. Albanie, A. Nagrani, and A. Zisserman.  Use what you have: Video retrieval using representations from collaborative experts.  In arXiv, 2019.
  * [9] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman.  End-to-End Learning of Visual Representations from Uncurated Instructional Videos.  In CVPR, 2020.
  * [10] N. C. Mithun, J. Li, F. Metze, and A. K. Roy-Chowdhury.  Learning joint embedding with multimodal cues for cross-modal video-text retrieval.  In Proceedings of the 2018 ACM on ICMR, page 19–27, 2018.
  * [11] Burak Satar, Zhu Hongyuan, Xavier Bresson, and Joo Hwee Lim.  Semantic role aware correlation transformer for text to video retrieval.  In 2021 IEEE International Conference on Image Processing (ICIP), pages 1334–1338, 2021.
  * [12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need.  In NIPS, 2017.
  * [13] Michael Wray, Hazel Doughty, and Dima Damen.  On semantic similarity in video retrieval.  In CVPR, 2021.
  * [14] M. Wray, D. Larlus, G. Csurka, and D. Damen.  Fine-grained action retrieval through multiple parts-of-speech embeddings.  In ICCV, 2019.


# Unveiling the Intricate Symphony of Nonlinear Pulsation Mode Interactions in
High-Amplitude $\delta$ Scuti Stars

Jia-Shu Niu (牛家树) Institute of Theoretical Physics, Shanxi University, Taiyuan
030006, China  State Key Laboratory of Quantum Optics and Quantum Optics
Devices, Shanxi University, Taiyuan 030006, China  Hui-Fang Xue (薛会芳)
Department of Physics, Taiyuan Normal University, Jinzhong 030619, China
Institute of Computational and Applied Physics, Taiyuan Normal University,
Jinzhong 030619, China

###### Abstract

People can diagnose the interiors of stars by sensing their pulsations.
Pulsation modes, which are determined by the internal state and structure of a
star, are typically considered stable over short timescales. These independent
pulsation modes have been used in asteroseismology to reconstruct the interior
structure of stars. Here, we report the discovery of peculiar pulsation mode
interaction details in the high-amplitude $\delta$ Scuti star KIC 6382916
(J19480292+4146558), challenging the reliability of independent pulsation
modes as indicators of the star’s internal structure. Through analysis of
archival data, we found distinct variations in amplitudes and frequencies of
three independent pulsation modes and their harmonics/combinations over
approximately 20 days. These variations can reach amplitudes of about 140% and
frequency variations of about 12%. Correlation analysis of amplitude and
frequency variations revealed additional pulsation mode interaction details
and patterns. Notably, our findings regarding the phenomena related to
harmonics of independent pulsation modes challenge the traditional
understanding in this area. These discoveries serve as cornerstones for future
research and advance nonlinear asteroseismology.

††software: astropy (Astropy Collaboration et al., 2022), seaborn (Waskom,
2021), Lightkurve (Lightkurve Collaboration et al., 2018), NumPy (Harris et
al., 2020), SciPy (Virtanen et al., 2020), matplotlib (Hunter, 2007)

##  1 Introduction

$\delta$ Scuti stars belong to the class of short-period pulsating variable
stars, with periods ranging from 15 minutes to 8 hours and spectral classes
A-F. They can be found at the intersection of the main sequence (MS) and the
lower section of the classical Cepheid instability strip on the Hertzsprung-
Russell diagram. The pulsations in $\delta$ Scuti stars are self-excited by
the $\kappa$ mechanism, which arises from the partial ionization of helium in
the outer layers (Kallinger et al., 2008; Handler, 2009; Guenther et al.,
2009; Uytterhoeven et al., 2011; Holdsworth et al., 2014; Steindl et al.,
2022).

High-amplitude $\delta$ Scuti stars (HADS) constitute a subclass of $\delta$
Scuti stars with relatively larger amplitudes ($\Delta V\geq 0_{\cdot}^{m}1$)
and slower rotation speeds ($v\sin i\leq 30\ \mathrm{km/s}$) in most cases.
However, as more samples of HADS have been accumulated, these classical
criteria have become less clear (see e.g., Balona et al. (2012)). Most HADS
pulsate with one or two radial pulsation modes (Niu et al., 2013, 2017; Bowman
et al., 2021; Daszyńska-Daszkiewicz et al., 2022), while some exhibit three
radial pulsation modes (Wils et al., 2008; Niu & Xue, 2022; Xue et al., 2023)
or even non-radial pulsation modes (Poretti et al., 2011; Xue & Niu, 2020).

For a long time, the stability of the amplitudes and frequencies of HADS has
been assumed based on the limited quality of observational data. However,
recent advancements in continuous time-series photometric data from space
telescopes, such as Kepler and TESS, have presented an opportunity to explore
the variations of pulsation modes over several years. The analysis of high-
quality data has revealed pulsation mode variations (including amplitudes,
frequencies, and phases) in various types of stars, including RR Lyrae stars
(Carrell et al., 2021), $\delta$ Scuti stars (Murphy et al., 2012; Breger &
Montgomery, 2014; Bowman et al., 2016), pulsating hot B subdwarf stars (Zong
et al., 2016a, 2018), pulsating DB white dwarf stars (Zong et al., 2016b), and
slowly pulsating B stars (Van Beeck et al., 2021). These variations not only
provide additional information to understand pulsation modes themselves (such
as the excitation and selection mechanisms, and mode interactions), but also
shed light on stellar structures and evolution. Despite HADS being primarily
pulsating in low-order radial p-modes (fundamental, first overtone, or second
overtone modes), their pulsation modes also exhibit distinct variations over
several years. Furthermore, the determination of linear period variation rates
(Bowman et al., 2021; Niu & Xue, 2022) and even quadratic amplitude variation
rates (Niu & Xue, 2022) from these photometric data offer further insights to
improve current stellar evolution and pulsation theories.

KIC 6382916 (J19480292+4146558), also known as GSC 03144-595, is an HADS that
was extensively monitored by the Kepler space telescope. The light curves of
this star show three independent pulsation modes. Initially, these three modes
were identified as non-radial pulsation modes with $l=1$ by Ulusoy et al.
(2013). Later, Mow et al. (2016) identified that these modes are actually the
fundamental, first overtone, and second overtone radial p-modes. Recently, Niu
& Xue (2022) conducted a comprehensive study on the pulsation mode variations
of KIC 6382916 over a period of about 4 years. Through a comparison between
observations and theoretical calculations, they identified the first and
second pulsation modes, labeled as $f_{0}$ and $f_{1}$, as the fundamental and
first overtone modes, respectively. The third pulsation mode, labeled as
$f_{2}$, was found to be the non-radial part of a resonating integration mode
(RI mode), generated by the resonance between a radial p-mode and a nonradial
p-g mixed mode. Furthermore, Niu & Xue (2022) discovered that almost all the
$f_{2}$-related combinations have partners with frequencies approximately
$\Delta\omega=0.0815\ \,\mathrm{c/d}$ away from them in the frequency domain.
These pairs follow the frequency relation $f_{2}-\Delta\omega=2f_{1}-f_{0}$.

In this study, we extracted the same pulsation modes as Niu & Xue (2022) using
the short-cadence (SC) photometric data of KIC 6382916 from Kepler telescope,
obtained from BJD 2455064 to 2455091 (Quarter 02). We aimed to study the
variations and interactions of these modes in a short timescale. Table 1
presents the frequencies and amplitudes of the 23 selected pulsation modes, as
well as their absolute and relative variations ($\Delta A$, $\Delta
A/\bar{A}$, $\Delta f$, and $\Delta f/\bar{f}$) and the Pearson correlation
coefficients ($\rho_{A,f}$). We were able to extract variation information
from the light curves for 19 of these pulsation modes. Further details can be
found in the Supplementary Materials.

Table 1: Overall Indicators of the 23 Pulsation Modes.

ID | Frequency | $\sigma_{f}$ | Amplitude | $\sigma_{A}$ | S/N | $\Delta f$ | $\Delta f/\bar{f}$ | $\Delta A$ | $\Delta A/\bar{A}$ | $\rho_{A,f}$ | Mark
---|---|---|---|---|---|---|---|---|---|---|---
| ($\,\mathrm{c/d}$) | ($\,\mathrm{c/d}$) | (mmag) | (mmag) |  | ($\,\mathrm{c/d}$) |  | (mmag) |  |  |
F0 | 4.9099 | 0.0004 | 81.2 | 1.6 | 51.9 | 0.013 | 0.003 | 4.094 | 0.05 | -0.84 | $f_{0}$
F1 | 6.4320 | 0.0004 | 79.8 | 1.4 | 58.2 | 0.009 | 0.001 | 5.676 | 0.07 | 0.52 | $f_{1}$
F2 | 11.3418 | 0.0004 | 30.9 | 0.6 | 55.5 | 0.014 | 0.001 | 2.241 | 0.07 | -0.03 | $f_{0}+f_{1}$
F3 | 1.5222 | 0.0004 | 16.3 | 0.3 | 53.7 | 0.029 | 0.019 | 4.359 | 0.27 | 0.97 | $-f_{0}+f_{1}$
F4 | 9.8197 | 0.0004 | 17.0 | 0.3 | 53.7 | 0.014 | 0.001 | 2.405 | 0.14 | -0.67 | $2f_{0}$
F5 | 8.0374 | 0.0005 | 12.0 | 0.3 | 42.1 | 0.084 | 0.01 | 9.496 | 0.75 | -0.92 | $f_{2}$
F6 | 12.8652 | 0.0004 | 13.3 | 0.2 | 62.1 | 0.039 | 0.003 | 5.090 | 0.37 | 0.97 | $2f_{1}$
F7 | 16.2519 | 0.0004 | 10.4 | 0.2 | 55.9 | 0.016 | 0.001 | 0.893 | 0.09 | 0.02 | $2f_{0}+f_{1}$
F8 | 7.9539 | 0.0004 | 7.0 | 0.1 | 55.4 | 0.092 | 0.012 | 4.618 | 1.36 | 0.80 | $-f_{0}+2f_{1}$
F9 | 3.3886 | 0.0005 | 5.1 | 0.1 | 46.6 | 0.072 | 0.021 | 3.347 | 0.61 | -0.91 | $2f_{0}-f_{1}$
F10 | 17.7748 | 0.0004 | 5.36 | 0.09 | 62.8 | 0.042 | 0.002 | 2.235 | 0.41 | 0.94 | $f_{0}+2f_{1}$
F11 | 12.9455 | 0.0005 | 3.33 | 0.07 | 47.0 | 0.124 | 0.01 | 2.141 | 1.25 | -0.87 | $f_{0}+f_{2}$
F12 | 22.6843 | 0.0004 | 4.94 | 0.09 | 58.4 | 0.024 | 0.001 | 0.895 | 0.18 | 0.67 | $2f_{0}+2f_{1}$
F13 | 14.4704 | 0.0007 | 3.5 | 0.1 | 32.9 | 0.100 | 0.007 | 2.742 | 0.79 | -0.71 | $f_{1}+f_{2}$
F14∗ | 1.6044 | 0.0006 | 2.79 | 0.07 | 40.7 | — | — | — | — | — | $-f_{1}+f_{2}$
F15∗ | 3.1222 | 0.0005 | 2.49 | 0.05 | 45.9 | — | — | — | — | — | $-f_{0}+f_{2}$
F16∗ | 3.3063 | 0.0008 | 1.98 | 0.07 | 30.2 | — | — | — | — | — | $f_{0}+f_{1}-f_{2}$
F17 | 14.7297 | 0.0005 | 2.38 | 0.06 | 42.5 | 0.071 | 0.005 | 1.069 | 0.47 | -0.50 | $3f_{0}$
F18 | 9.5582 | 0.0006 | 2.00 | 0.05 | 38.3 | 0.118 | 0.012 | 0.949 | 0.5 | -0.19 | $-f_{0}+f_{1}+f_{2}$
F19 | 19.2982 | 0.0005 | 2.69 | 0.06 | 48.7 | 0.076 | 0.004 | 1.988 | 0.67 | 0.92 | $3f_{1}$
F20 | 24.2067 | 0.0004 | 3.04 | 0.06 | 52.4 | 0.044 | 0.002 | 1.495 | 0.48 | 0.96 | $f_{0}+3f_{1}$
F21 | 21.1616 | 0.0005 | 1.97 | 0.04 | 48.4 | 0.052 | 0.002 | 0.833 | 0.41 | -0.77 | $3f_{0}+f_{1}$
F22∗ | 3.043 | 0.001 | 1.32 | 0.06 | 22.7 | — | — | — | — | — | $-2f_{0}+2f_{1}$

Note: ∗ denotes the pulsation modes which we cannot extract their amplitude
and frequency variations because of the strong interactions and low SNRs.

##  2 Results and Discussions

###  2.1 Resonating Integration Mode

Based on the proposed framework of the Resonating Integration (RI) mode by Niu
& Xue (2022), two important behaviors were predicted on short timescales for
the pulsation modes F5 ($f_{2}$) and F8 ($-f_{0}+2f_{1}$): (i) as the non-
radial part of a RI mode, F5 should exhibit amplitude and frequency
modulations; (ii) as the superposition and resonance of the radial part of a
RI mode and the combination of $f_{0}$ and $f_{1}$, F8 should have a dual
identity and exhibit frequency splitting while also being modulated (if two
parts of it are out of sync).

Figure 1 presents the amplitude and frequency variations, as well as the
interaction details of F5 and F8. Subfigure (a) and (b) show that F5 and F8
exhibit distinct amplitude and frequency modulations as predicted. Subfigure
(c) reveals that F8 splits into two parts after approximately 235 days. The
first part, which is opposite to the modulation phase of F5, approaches and
resonates with F5. The second part, which is the same as the modulation phase
of F5, seems to approach and resonate with another pulsation mode. After about
246 days, these two parts of F8 merge, and the amplitude of F5 remains low due
to decoupling from the first part of F8. However, for F8, its amplitude is not
completely anti-correlated with F5, as evident from the pit between 246 and
249 days. This indicates that the first part of F8 has its own modulation
rhythm, independent of F5, which arises from $f_{0}$ and $f_{1}$.

Furthermore, in subfigures (a) and (b), the amplitude and frequency
fluctuations of F5 (around 241-242 days) and F8 (around 240-243 days) are
caused by strong resonant interactions with adjacent pulsation modes, making
it difficult to accurately distinguish them in these time regions.111The
amplitude and frequency variations of F14, F15, F16, and F22 could not be
extracted for the same reason. In the lower frequency region, there are more
pulsation modes and complex interactions (see in Appendix, Figure A6).

Figure 1: Variation of the amplitudes and frequencies of F5 and its partner
F8. Subfigure (a) represents the amplitude and frequency variations of F5;
subfigure (b) represents the amplitude and frequency variations of F8;
subfigure (c) is the time-frequency diagram which presents interaction details
in $7.5-8.5\ \,\mathrm{c/d}$.

In Figures A2, A3, A4, and A5, it can be observed that the amplitudes and
frequencies of many pulsation modes exhibit strong (anti)correlation.
Specifically, the pulsation modes F3, F5, F6, F9, F10, F19, and F20 show
significant (anti)correlations, with their $|\rho_{A,f}|$ values exceeding
9.0. The amplitude and frequency (anti)correlations of these modes can be
consistently explained by the framework provided by F5 and F8. The intrinsic
frequency modulation of these modes leads to resonances between these modes
and their partners. These resonances result in the growth of amplitudes.
Although the modulation in frequency appears to be of first nature, it
presents another problem that needs to be addressed, which comes from the non-
correlation of the frequencies and their harmonics.

###  2.2 Harmonics and Combinations

In pulsating stars, such as Cepheids (Rathour et al., 2021), RR Lyrae stars
(Kurtz et al., 2016), $\delta$ Scuti stars (Breger & Montgomery, 2014), high-
amplitude $\delta$ Scuti stars (Niu et al., 2017), SX Phoenicis stars (Xue &
Niu, 2020), $\gamma$ Dor stars (Kurtz et al., 2015), pulsating white dwarfs
(Wu, 2001), $\beta$ Cep stars (Degroote et al., 2009), and slowly pulsating B
stars (Pápics et al., 2017), harmonics of independent pulsation modes are
quite common. These harmonics arise from the nonsinusoidal nature of light
curves and indicate the nonlinearity of the star’s pulsation. Therefore,
harmonics are not considered as intrinsic stellar pulsation modes (Brickhill,
1992; Wu, 2001) and they should mimic the behaviors of their parent pulsation
modes. However, recent research has shown that as the order of the harmonics
increases, the amplitude and frequency of the harmonics exhibit uncorrelated
variations compared to their parent pulsation mode (Niu et al., 2023).

For KIC 6382916, the amplitude and frequency variations of the fundamental and
first overtone pulsation modes ($f_{0}$ and $f_{1}$) along with their first
and second harmonics ($2f_{0}$, $3f_{0}$, $2f_{1}$, and $3f_{1}$) are
presented in Figures 2 and 3. It is evident that the harmonics of $f_{0}$ and
$f_{1}$ show uncorrelated amplitude and frequency variations compared to their
parent modes, even at the first and second harmonics. This challenges the
common understanding derived from stellar pulsation theory, thus broadening
our perception of harmonics.

Furthermore, the amplitude and frequency variations of $2f_{1}$ and $3f_{1}$
exhibit a high degree of correlation, indicating that $2f_{1}$ behaves like an
independent parent pulsation mode of $3f_{1}$. In general, the pulsation mode
$3f_{1}$ can be represented by $3\cdot(f_{1})$ and $(2f_{1})+(f_{1})$, which
are equivalent representations in common. However, in this case, the correct
representation of $3f_{1}$ is $(2f_{1})+(f_{1})$. This challenges the usual
frequency identification of harmonics in the pre-whitening process and
subsequently the identification of independent pulsation modes. Moreover, if
we adhere to the principle that frequency modulation is of first nature and
explain amplitude modulations through resonances, the physical origin of
frequency modulation in $2f_{1}$ becomes a question. It should be noted that
$f_{1}$ does not exhibit obvious frequency modulation like $2f_{1}$.

The aforementioned case is not unique. It is observed that the pulsation modes
F10 ($f_{0}+2f_{1}$) and F20 ($f_{0}+3f_{1}$) also exhibit highly correlated
amplitude and frequency variations. This suggests that $f_{0}+3f_{1}$ can be
represented as the sum of $(f_{0}+2f_{1})$ and $(f_{1})$, and $f_{0}+2f_{1}$
should be considered as an independent parent pulsation mode of
$f_{0}+3f_{1}$.

Figure 2: Amplitude and frequency variations of the independent pulsation mode
$f_{0}$ and its first and second harmonics ($2f_{0}$ and $3f_{0}$).

Figure 3: Amplitude and frequency variations of the independent pulsation mode
$f_{1}$ and its first and second harmonics ($2f_{1}$ and $3f_{1}$).

###  2.3 Overview of the Interactions

The interactions between the pulsation modes can be clearly represented in the
amplitude and frequency interaction diagrams of the 19 pulsation modes shown
in Figure 4. These diagrams cluster together pulsation modes with similar
amplitude and frequency variation patterns. It is worth noting that these
interactions are complex, indicating that the harmonics and combinations are
more intricate than commonly assumed, and they contain rich information about
the nonlinear characteristics. Furthermore, the resulting interaction
relationships differ from those observed in longer timescales. Within the
interaction diagrams, only F6/F19 and F10/F20 exhibit the most similar
behaviors in terms of both amplitude and frequency variations. This
observation provides valuable insights into reevaluating the physical nature
of the harmonics and combinations, as discussed earlier.

Figure 4: Interaction diagrams of amplitudes and frequencies of the 19
pulsation modes.

In the interaction diagram of amplitudes, although the modes related to
$f_{2}$ (F5, F11, F13, and F18) are not clustered together as observed
previously in Niu & Xue (2022), they exhibit anti-correlated behavior with the
modes in the first cluster (F7, F12, F8, F6, and F19). This suggests the
presence of energy transfer between the $f_{2}$-related and some
$f_{1}$-related modes. Specifically, if we consider F7 ($2f_{0}+f_{1}$) and
F12 ($2f_{0}+2f_{1}$) as $(2f_{0}-f_{1})+(2f_{1})$ and
$(2f_{0}-f_{1})+(3f_{1})$, respectively222This decomposition is inspired by
the relationship between $2f_{1}$ and $3f_{1}$., we can observe that the modes
in the first cluster are all related to $2f_{1}$ and $3f_{1}$. Based on the
relation $3f_{1}\equiv(2f_{1})+(f_{1})$, it is evident that the most
significant energy transfer channel of the first cluster is $2f_{1}$ rather
than $f_{1}$.

In the interaction diagram of frequencies, it can be observed that F3 and F2
exhibit the strongest interactions with other pulsation modes. F3
($-f_{0}+f_{1}$) has the largest number of anti-correlated pulsation modes. On
the other hand, F2 ($f_{0}+f_{1}$) is anti-correlated to F0 ($f_{0}$), F1
($f_{1}$), and F7 ($2f_{0}+f_{1}$), which is quite strange if we note that the
first two are the parent modes of F2. This finding challenges our traditional
understanding of combination modes in pulsating stars, suggesting the presence
of an undiscovered mechanism when these modes combine.

The frequency anti-correlation between F0 ($f_{0}$) and F2 ($f_{0}+f_{1}$) is
particularly noteworthy. Almost all pulsation modes are affected by a small
frequency modulation of approximately 0.004 $\,\mathrm{c/d}$ as evident in F0
and F2. This modulation exhibits a period of 0.6570 day and appears to be the
background in the time-frequency diagram (see, e.g., Figures A6). The physical
origin of this modulation is still unknown.

###  2.4 Summary

In summary, a comprehensive analysis of the pulsation modes in KIC 6382916
reveals a striking analogy to the surface of the sea. On larger timescales,
the pulsation modes exhibit a seemingly calm or gradual trend. However, when
observed on smaller timescales, they undergo violent fluctuations, unveiling
intricate pulsation mode interactions. These interactions manifest as
amplitude and frequency modulations, presenting a dual identity of certain
modes and posing challenges to conventional concepts like harmonics and
combinations. Despite the complexity inherent in these phenomena, discernible
patterns emerge, offering valuable guidance for future research in nonlinear
asteroseismology. This profound understanding of the pulsation modes in KIC
6382916 provides a foundation for unraveling the mysteries of stellar
pulsation and opens up exciting avenues for further exploration in the field.

## Acknowledgments

J.S.N. acknowledges support from the National Natural Science Foundation of
China (NSFC) (No. 12005124 and No. 12147215). H.F.X. acknowledges support from
the Scientific and Technological Innovation Programs of Higher Education
Institutions in Shanxi (STIP) (No. 2020L0528) and the Applied Basic Research
Programs of Natural Science Foundation of Shanxi Province (No.
202103021223320). The authors acknowledge the Kepler Science team and everyone
who has contributed to making the Kepler mission possible. Funding for the
Kepler mission is provided by NASA’s Science Mission Directorate.

## References

  * Astropy Collaboration et al. (2022) Astropy Collaboration, Price-Whelan, A. M., Lim, P. L., et al. 2022, ApJ, 935, 167, doi: 10.3847/1538-4357/ac7c74
  * Balona et al. (2012) Balona, L. A., Lenz, P., Antoci, V., et al. 2012, MNRAS, 419, 3028, doi: 10.1111/j.1365-2966.2011.19939.x
  * Bowman et al. (2021) Bowman, D. M., Hermans, J., Daszyńska-Daszkiewicz, J., et al. 2021, MNRAS, 504, 4039, doi: 10.1093/mnras/stab1124
  * Bowman et al. (2016) Bowman, D. M., Kurtz, D. W., Breger, M., Murphy, S. J., & Holdsworth, D. L. 2016, MNRAS, 460, 1970, doi: 10.1093/mnras/stw1153
  * Breger & Montgomery (2014) Breger, M., & Montgomery, M. H. 2014, ApJ, 783, 89, doi: 10.1088/0004-637X/783/2/89
  * Brickhill (1992) Brickhill, A. J. 1992, MNRAS, 259, 519, doi: 10.1093/mnras/259.3.519
  * Carrell et al. (2021) Carrell, K., Wilhelm, R., Olsen, F., et al. 2021, ApJ, 916, L12, doi: 10.3847/2041-8213/ac1202
  * Daszyńska-Daszkiewicz et al. (2022) Daszyńska-Daszkiewicz, J., Walczak, P., Pamyatnykh, A. A., & Szewczuk, W. 2022, MNRAS, 512, 3551, doi: 10.1093/mnras/stac646
  * Degroote et al. (2009) Degroote, P., Briquet, M., Catala, C., et al. 2009, A&A, 506, 111, doi: 10.1051/0004-6361/200911782
  * Guenther et al. (2009) Guenther, D. B., Kallinger, T., Zwintz, K., et al. 2009, ApJ, 704, 1710, doi: 10.1088/0004-637X/704/2/1710
  * Handler (2009) Handler, G. 2009, in American Institute of Physics Conference Series, Vol. 1170, Stellar Pulsation: Challenges for Theory and Observation, ed. J. A. Guzik & P. A. Bradley, 403–409, doi: 10.1063/1.3246528
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., et al. 2020, Nature, 585, 357, doi: 10.1038/s41586-020-2649-2
  * Holdsworth et al. (2014) Holdsworth, D. L., Smalley, B., Gillon, M., et al. 2014, MNRAS, 439, 2078, doi: 10.1093/mnras/stu094
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science and Engineering, 9, 90, doi: 10.1109/MCSE.2007.55
  * Kallinger et al. (2008) Kallinger, T., Zwintz, K., & Weiss, W. 2008, A&A, 488, 279, doi: 10.1051/0004-6361:20079207
  * Kurtz et al. (2016) Kurtz, D. W., Bowman, D. M., Ebo, S. J., et al. 2016, MNRAS, 455, 1237, doi: 10.1093/mnras/stv2377
  * Kurtz et al. (2015) Kurtz, D. W., Shibahashi, H., Murphy, S. J., Bedding, T. R., & Bowman, D. M. 2015, MNRAS, 450, 3015, doi: 10.1093/mnras/stv868
  * Lightkurve Collaboration et al. (2018) Lightkurve Collaboration, Cardoso, J. V. d. M., Hedges, C., et al. 2018, Lightkurve: Kepler and TESS time series analysis in Python, Astrophysics Source Code Library, record ascl:1812.013.  http://ascl.net/1812.013
  * Mow et al. (2016) Mow, B., Reinhart, E., Nhim, S., & Watkins, R. 2016, AJ, 152, 17, doi: 10.3847/0004-6256/152/1/17
  * Murphy et al. (2012) Murphy, S. J., Grigahcène, A., Niemczura, E., Kurtz, D. W., & Uytterhoeven, K. 2012, MNRAS, 427, 1418, doi: 10.1111/j.1365-2966.2012.22062.x
  * Niu et al. (2013) Niu, J.-S., Fu, J.-N., & Zong, W.-K. 2013, Res. Astron. Astrophys. , 13, 1181, doi: 10.1088/1674-4527/13/10/004
  * Niu et al. (2023) Niu, J.-S., Liu, Y., & Xue, H.-F. 2023, AJ, 166, 43, doi: 10.3847/1538-3881/acda2b
  * Niu & Xue (2022) Niu, J.-S., & Xue, H.-F. 2022, ApJ, 938, L20, doi: 10.3847/2041-8213/ac9407
  * Niu et al. (2017) Niu, J.-S., Fu, J.-N., Li, Y., et al. 2017, MNRAS, 467, 3122, doi: 10.1093/mnras/stx125
  * Pápics et al. (2017) Pápics, P. I., Tkachenko, A., Van Reeth, T., et al. 2017, A&A, 598, A74, doi: 10.1051/0004-6361/201629814
  * Poretti et al. (2011) Poretti, E., Rainer, M., Weiss, W. W., et al. 2011, A&A, 528, A147, doi: 10.1051/0004-6361/201016045
  * Rathour et al. (2021) Rathour, R. S., Smolec, R., & Netzel, H. 2021, MNRAS, 505, 5412, doi: 10.1093/mnras/stab1603
  * Smith et al. (2012) Smith, J. C., Stumpe, M. C., Van Cleve, J. E., et al. 2012, PASP, 124, 1000, doi: 10.1086/667697
  * Steindl et al. (2022) Steindl, T., Zwintz, K., & Vorobyov, E. 2022, Nature Communications, 13, 5355, doi: 10.1038/s41467-022-32882-0
  * Stumpe et al. (2012) Stumpe, M. C., Smith, J. C., Van Cleve, J. E., et al. 2012, PASP, 124, 985, doi: 10.1086/667698
  * Ulusoy et al. (2013) Ulusoy, C., Ulas, B., Gülmez, T., et al. 2013, MNRAS, 433, 394, doi: 10.1093/mnras/stt731
  * Uytterhoeven et al. (2011) Uytterhoeven, K., Moya, A., Grigahcène, A., et al. 2011, A&A, 534, A125, doi: 10.1051/0004-6361/201117368
  * Van Beeck et al. (2021) Van Beeck, J., Bowman, D. M., Pedersen, M. G., et al. 2021, A&A, 655, A59, doi: 10.1051/0004-6361/202141572
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, Nature Methods, 17, 261, doi: 10.1038/s41592-019-0686-2
  * Waskom (2021) Waskom, M. L. 2021, Journal of Open Source Software, 6, 3021, doi: 10.21105/joss.03021
  * Wils et al. (2008) Wils, P., Rozakis, I., Kleidis, S., Hambsch, F. J., & Bernhard, K. 2008, A&A, 478, 865, doi: 10.1051/0004-6361:20078992
  * Wu (2001) Wu, Y. 2001, MNRAS, 323, 248, doi: 10.1046/j.1365-8711.2001.04224.x
  * Xue & Niu (2020) Xue, H.-F., & Niu, J.-S. 2020, ApJ, 904, 5, doi: 10.3847/1538-4357/abbc12
  * Xue et al. (2023) Xue, W., Niu, J.-S., Xue, H.-F., & Yin, S. 2023, Research in Astronomy and Astrophysics, 23, 075002, doi: 10.1088/1674-4527/accdbc
  * Zong et al. (2018) Zong, W., Charpinet, S., Fu, J.-N., et al. 2018, ApJ, 853, 98, doi: 10.3847/1538-4357/aaa548
  * Zong et al. (2016a) Zong, W., Charpinet, S., & Vauclair, G. 2016a, A&A, 594, A46, doi: 10.1051/0004-6361/201629132
  * Zong et al. (2016b) Zong, W., Charpinet, S., Vauclair, G., Giammichele, N., & Van Grootel, V. 2016b, A&A, 585, A22, doi: 10.1051/0004-6361/201526300

##  Appendix A Methods

###  A.1 Photometric Data Reduction

This study utilized short-cadence (SC) photometric data of KIC 6382916
obtained from the Kepler space telescope (Stumpe et al., 2012; Smith et al.,
2012). The data span from BJD 2455064 to 2455091 (Quarter 02, with a small
portion of discontinuous data removed, approximately 2 days in length) and
were downloaded from the Mikulski Archive for Space Telescope
(MAST)333http://archive.stsci.edu/kepler. The data, in the form of reduced BJD
and magnitudes, were then normalized to have a zero mean. Figure A1 provides
an overview of the normalized SC data in both the time and frequency domains.

Figure A1: Overview of the normalized SC data in time domain and frequency
domain of KIC 6382916. The region surrounded by the red rectangular boxes is
selected to zoom in the details of the observed light curves. The 3
independent frequencies are marked as colored dots.

All the aforementioned normalized SC data were pre-whitened to extract the
frequencies and amplitudes of the pulsation modes. The process was terminated
once all the frequencies studied in Niu & Xue (2022) (a total of 23
frequencies) had been identified (see Table 1). For ease of comparison, the
IDs of the pulsation modes in this work were arranged in the same order as in
Niu & Xue (2022), rather than in descending order of amplitudes based on the
SC data.

The Short-Time Fourier Transformation (Bowman et al., 2016; Zong et al., 2018;
Niu & Xue, 2022; Niu et al., 2023) was then applied to the normalized SC data
to determine the variations of the amplitudes and frequencies. In this
process, a time window of 6 days was moved from the start to the end time of
the SC data, with a step of 0.02 days. At each step, the pre-whitening process
was conducted to extract the amplitudes and frequencies of the specific 23
pulsation modes. The phase was considered as a free parameter and was not the
focus of this study.

Next, we collected the amplitudes and frequencies from each moving window for
each of the 19 pulsation modes in 23. The times were defined as the midpoints
of the windows. The remaining 4 pulsation modes (F14, F15, F16, and F22)
exhibited complex interactions with the nearby pulsation modes, making it
difficult to identify and distinguish them from each other (see, e.g., Figure
A6). The variations in the amplitudes and frequencies of the 19 pulsation
modes are shown in Figures A2, A3, A4, and A5.

In this study, the pre-whitening process was conducted using Fourier
decomposition, which can be represented by the formula:

$m=m_{0}+\sum A_{i}\sin\left[2\pi(f_{i}t+\phi_{i})\right]~{},$ (A1)

where $m_{0}$ is the shifted value, $A_{i}$ is the amplitude, $f_{i}$ is the
frequency, and $\phi_{i}$ is the corresponding phase. The uncertainties of the
amplitudes ($\sigma_{A}$), frequencies ($\sigma_{f}$), and phases
($\sigma_{\phi}$) throughout this study were estimated using the framework
introduced in Niu & Xue (2022) and Niu et al. (2023).

###  A.2 Overall Indicators of Pulsation Modes

Table 1 presents the absolute and relative variations of these 19 pulsation
modes. The absolute variation is defined as the difference between the maximum
and minimum values of the mode ($\Delta f\equiv
f_{\mathrm{max}}-f_{\mathrm{min}}$ and $\Delta A\equiv
A_{\mathrm{max}}-A_{\mathrm{min}}$), while the relative variation is the ratio
of the difference to the mean values ($\Delta
f/\bar{f}\equiv(f_{\mathrm{max}}-f_{\mathrm{min}})/\bar{f}$ and $\Delta
A/\bar{A}\equiv(A_{\mathrm{max}}-A_{\mathrm{min}})/\bar{A}$). The relative
variations of the amplitudes and the absolute variations of the frequencies
are also presented in Figures A2, A3, A4, and A5.

The $\rho_{A,f}$ in Table 1 and Figures A2, A3, A4, and A5 represents the
Pearson correlation coefficient between the amplitude and frequency for each
of the 19 pulsation modes.

###  A.3 Time-frequency diagram

The time-frequency diagram (Subfigure (c) of Figure 1 and Figure A6) was
plotted with a time window of 6 days, which was moved from the start to the
end time of the SC data, with a step of 0.01 days. The power (square of
amplitude) is indicated by the color bar.

Please note that the pulsation modes marked in the time-frequency diagram are
for illustration purposes only and may not be accurate. This is because it is
difficult to identify a pulsation mode with drastic variations in amplitude
and frequency, let alone strong interaction with nearby modes.

##  Appendix B Figures and Tables

Figure A2: Amplitude and frequency variations of the 19 pulsation modes, Part
I.

Figure A3: Amplitude and frequency variations of the 19 pulsation modes, Part
II.

Figure A4: Amplitude and frequency variations of the 19 pulsation modes, Part
III.

Figure A5: Amplitude and frequency variations of the 19 pulsation modes, Part
IV. Figure A6: Time-frequency diagram which presents interaction details in
$2-3\ \,\mathrm{c/d}$.


# Exploring and Mitigating Gender Bias in Recommender Systems with Explicit
Feedback

Shrikant Saxena
Department of Computer Science
Indian Institute of Technology (IIT)
Ropar, India
<EMAIL_ADDRESS>
&Shweta Jain
Department of Computer Science
Indian Institute of Technology (IIT) Ropar, India
<EMAIL_ADDRESS>
###### Abstract

Recommender systems are indispensable because they influence our day-to-day
behavior and decisions by giving us personalized suggestions. Services like
Kindle, Youtube, and Netflix depend heavily on the performance of their
recommender systems to ensure that their users have a good experience and to
increase revenues. Despite their popularity, it has been shown that
recommender systems reproduce and amplify the bias present in the real world.
The resulting feedback creates a self-perpetuating loop that deteriorates the
user experience and results in homogenizing recommendations over time.
Further, biased recommendations can also reinforce stereotypes based on gender
or ethnicity, thus reinforcing the filter bubbles that we live in. In this
paper, we address the problem of gender bias in recommender systems with
explicit feedback. We propose a model to quantify the gender bias present in
book rating datasets and in the recommendations produced by the recommender
systems. Our main contribution is to provide a principled approach to mitigate
the bias being produced in the recommendations. We theoretically show that the
proposed approach provides unbiased recommendations despite biased data.
Through empirical evaluation on publicly available book rating datasets, we
further show that the proposed model can significantly reduce bias without
significant impact on accuracy. Our method is model agnostic and can be
applied to any recommender system. To demonstrate the performance of our
model, we present the results on four recommender algorithms, two from the
K-nearest neighbors family, UserKNN and ItemKNN, and the other two from the
matrix factorization family, Alternating least square and Singular value
decomposition.

_Keywords_ Book Recommender systems, Gender Bias, Fairness

##  1 Introduction

Recommender systems influence a significant portion of our digital activity.
They are responsible for keeping the user experience afresh by recommending
varied items from a catalog of millions of items and also adapt their
recommendations according to the personality and taste of the user. Therefore,
a sound recommender system may go a long way in improving user experience
quality, hence the user retentivity of a digital outlet.

Recommender systems have historically been judged on their accuracyHerlocker
et al. (2004); Shani and Gunawardana (2011). When it is concerned with other
factors such as novelty, user satisfaction, and diversityHurley and Zhang
(2011); Ziegler et al. (2005a); Knijnenburg et al. (2012), the focus continues
to be just on the satisfaction of the information needs of the users. Although
of immense importance to the relevance of a recommender system, these criteria
do not capture the complete picture. In recent years, the public and academic
community have scrutinized artificial intelligence systems regarding their
fairness. It has been observed that the results generated by various
recommender systems reflect the social biases that exist in human
stratumEkstrand et al. (2018); Shakespeare et al. (2020); Boratto et al.
(2019). Scholars have focused on identifying, quantifying, and mitigating the
bias present in the results generated by recommendation systems. Burke (2017)
presents a taxonomy of classes for fair recommendation systems. The author
suggests different recommendation settings with fairness requirements such as
fairness for only users, fairness for only items, and fairness for both users
and items. Our work falls into fairness for only items category where bias is
shown by a particular set of users against a specific set of items in the
dataset. In particular, we are interested in studying and eliminating users’
biasedness against the items associated with a specific gender in
recommendation systems.

Bias prevention approaches can be classified according to the phase of the
data mining process in which they operate: pre-processing, in-processing, and
post-processing methods. Pre-processing methods aim to control distortion of
the training set. In particular, they transform the training dataset so that
the discriminatory biases contained in the dataset are smoothed, hampering the
mining of unfair decision models from the transformed data. In-processing
methods modify recommendation algorithms such that the resulting models do not
entail unfair decisions by introducing a fairness constraint in the
optimization problem. Lastly, post-processing methods act on the extracted
data mining model results instead of the training data or algorithm. The
method presented in our work is a hybrid of a pre-processing phase and a post-
processing phase.

Two prominent studies have focused on gender bias in recommender systems. The
work by Shakespeare et al. (2020) establishes the existence of bias in the
results of the music recommender systems, and the work by Ekstrand et al.
(2018) focuses on bias shown by Collaborative Filtering (CF) algorithms while
recommending books written by women authors. Both the studies establish that
the CF algorithms produced biased results after being fed the biased data from
various socio-cultural factors. While both the works focus just on showing the
existence of bias in the presence of the users’ implicit feedback, we also
consider the explicit feedback ratings and the bias that may arise out of it.
Thus, our model handles the case when the items associated with specific
gender might have received worse feedback than they otherwise ought to achieve
by a set of users. We go one step further and propose a model to mitigate
these biases by quantifying a particular user’s bias and debiasing his or her
feedback ratings. We theoretically show that the debiased ratings are unbiased
estimators of the true preference of the user. Once the ratings are debiased,
they are fed into the recommender algorithms as input to produce
recommendations for the desired set of users. Since the recommender system is
now fed with the debiased ratings, the resulting recommendations are free from
the bias factor and avoid a self-perpetuating loop in the future.

The bias of an individual user reflects his or her taste. However, the KNN
based algorithms produce recommendations based on similar characteristics
between a set of users and naive implementation of these algorithms reflects
the bias of one user in the recommendations produced for the other user. While
not directly comparing the rating history of different users or items, Matrix
Factorization algorithms rely on deriving latent factors, which depend on the
rating history. Both the approaches make the system increasingly biased and
homogenized after users interact with their biased recommendations and
generate data for the next iteration. The above discussion suggests that
though it is necessary to reflect the user’s preference in the recommendations
produced for him or her to achieve accuracy, it is equally necessary to
prevent the bias of one user from reflecting in the recommendations of another
similar user. Our research focuses on this particular objective.

Our debiased ratings assure that the biases of one user do not affect other
users; however, it may lead to loss of accuracy because of not reflecting the
user’s own preferences. We introduce a new step called preference correction
which injects the user’s preference parameter into his/her own debiased
recommendation to maintain the accuracy of the system. On the publicly
available Book-Crossing dataset (Ziegler et al. (2005b)) and Amazon Book
Review dataset (Ni et al. (2019)), we empirically show that this approach
retained the significant reduction in bias and had minimal effect on the
accuracy of the system. The bias reflected in the recommendations produced by
the UserKNN, ItemKNN, ALS, and SVD algorithms is reduced by as much as
$42.39\%$, $37.65\%$, $26.51\%$, and $41.43\%$ respectively for the Amazon
dataset and by $37.82\%$, $30.73\%$, $24.99\%$, and $32.34\%$ for the Book-
Crossing dataset. When measured with respect to Root Mean Squared Error(RMSE),
the final accuracy loss in the case of the Amazon dataset comes out to be
$7.8\%$, $11.96\%$, $12.49\%$, and $10.38\%$ respectively for the four
algorithms. In the case of the Book-Crossing dataset, the RMSE loss comes out
to be $13.86\%$, $18.13\%$, $11.41\%$, and $12.89\%$ respectively. In
particular, the following are our main contributions.

###  1.1 Contributions

  * •

We propose a model to quantify the gender bias in the recommender system when
explicit feedback is present.

  * •

We propose a principled approach to debias the ratings given and theoretically
show that the debiased ratings represent the unbiased estimator of the true
preference of the user.

  * •

We empirically evaluate our model on publicly available book datasets and show
that the approach significantly reduced the biasedness in the system. To show
the generality of our proposed approach, we show the results on four
algorithms, UserKNN, ItemKNN, ALS, and SVD.

  * •

In order to further enhance the accuracy of the debiased system, we propose an
approach of preference correction that respects the user’s own preferences
towards his/her recommendations. We show that the final recommender system
significantly reduces the bias in the system while not deteriorating the
accuracy much.

##  2 Related Works

The problem of gender bias and discrimination has received lots of attention
in recent works Hajian et al. (2016). Many proposals like Pedreschi et al.
(2008), Pedreschi et al. (2009), Ruggieri et al. (2010), Thanh et al. (2011),
Mancuhan and Clifton (2014), Ruggieri et al. (2014) are dedicated to detecting
and measuring the existing biases in the datasets while other efforts Kamiran
et al. (2010), Kamiran et al. (2012), Hajian and Domingo-Ferrer (2013), Hajian
et al. (2014a), Hajian et al. (2014b), Dwork et al. (2011), Zemel et al.
(2013) are focused on ensuring that data mining models do not produce
discriminatory results even though the input data may be biased. Most of these
works focus on the classical problem of classification. Amatriain et al.
(2011) discuss the application of various classification methods like Support
Vector Machines, Artificial Neural Networks, Bayesian classifiers, and
decision trees in recommender systems. Their findings indicated that a more
complex classifier need not give a better performance for recommender systems,
and more exploration is needed in this direction.

When considering "fairness for only users" according to the taxonomy presented
by Burke (2017), Boratto et al. (2019) and Tsintzou et al. (2018) discuss the
bias with respect to the preferential recommending of certain items only to
the users of a specific gender. While weighted regularization matrix
factorization studied in Boratto et al. (2019) is only appropriate for
implicit feedback, the Group Utility Loss Minimization proposed in Tsintzou et
al. (2018) works only with respect to the UserKNN algorithm. Both the papers
address the issue of gender bias by employing post-processing algorithms that
work only in limited settings. Though Boratto et al. (2019) and Tsintzou et
al. (2018) have addressed the issue of fairness of recommender systems with
respect to gender, they have done so from the perspective of recommending
certain items only to the users of a specific gender. The difference between
their work and our study lies in the fact that we focus on the more direct
issue of gender bias in recommendations shown to items associated with a
specific gender.

Figure 1: Model schematics

Shakespeare et al. (2020) in their research highlight the artist gender bias
in music recommendations produced by Collaborative Filtering algorithms. The
work traces the causes of disparity to variations in input gender
distributions and user-item preferences, highlighting the effect such
configurations can have on user’s gender bias after recommendation generation.
Mansoury et al. (2020) discuss the biases from the perspective of a specific
group of individuals (for example, a particular gender) receiving less
calibrated and hence unfair recommendations. Ekstrand et al. (2018) explores
the gender bias present in the book rating dataset. Our work is different from
the works by Shakespeare et al. (2020), Mansoury et al. (2020) and Ekstrand et
al. (2018) in primarily two factors: (i) we consider explicit feedback as
opposed to the implicit feedback, and (ii) we propose a principled approach to
debias the ratings and theoretically show that the debiased ratings are
unbiased estimators of true ratings.

The research by Leavy et al. (2020) focuses on algorithmic gender bias and
proposes a framework whereby language-based data may be systematically
evaluated to assess levels of gender bias prevalent in training data for
machine learning systems. Our work is different from this study as this study
is focused on evaluating gender bias in the language and textual data
settings, while ours deals with gender bias in a more traditional user-item
rating setting.

##  3 The Model

Consider a recommender system having $\mathcal{U}=\\{1,2,\ldots,U\\}$ users
and $\mathcal{I}=\\{1,2,\ldots,I\\}$ items. Let $\mathbb{D}$ and $\mathbb{A}$
denote the set of items associated with disadvantaged group and advantaged
group, respectively. For example, in a book recommender system, the books
represent the items; $\mathbb{D}$ and $\mathbb{A}$ represent the set of books
written by women and men authors respectively. With respect to book
recommender system, researchers have already shown that the data is biased
against female authors’ books Ekstrand et al. (2018).

Let $r_{ui}\in[1,R]$ denote the rating that user $u$ has given to the item
$i$. As opposed to previous works, we consider explicit feedback wherein
biases may not only arise from not giving the rating to the item but may also
come from giving a bad rating to the item. The user profile
$p_{u}=\\{X_{u},R_{u}\\}$ represents the set of books ($X_{u}$) and the
ratings ($R_{u}=\\{r_{ui}\\}_{i\in X_{u}}$) that user $u$ has given to those
items.

The proposed recommender system first pre-processes the data that: 1) finds
the log-bias $\theta_{u}$ of each user $u$ and 2) generates the debiased
rating $d_{ui}$ of each user $u$ and item $i$ using the computed bias in the
first step. We then theoretically show that the debiased ratings generated are
unbiased estimators of the true preferences of the user for the items rated by
them. Thus, the debiased dataset can then be fed into various recommender
algorithms to generate an unbiased predicted rating of a user $u$ for the item
$i$, denoted by $\tilde{d}_{ui}$. This debiasing step ensures that the
existing biases are not boosted further in the system. Our debiasing model is
independent of any recommendation algorithm. We show the performance of our
debiasing model on both K-nearest neighbors-based algorithms (UserKNN,
ItemKNN) as well as matrix factorization-based algorithms (Alternating Least
Square and Singular Value Decomposition) to produce the recommendations.

In the next step, we use preference corrector to reintroduce the preferences
of a particular user $u$ to his/her own recommendations. This is achieved via
producing a user specific rating $\tilde{r}_{ui}$ from the debiased rating
$\tilde{d}_{ui}$. The recommendations are re-ranked according to the adjusted
ratings, and the recommendations are presented to the user. This step ensures
that the system does not lose accuracy for not considering the preferences of
the users. Figure 1 shows the schematic diagram of our model. Consider that
the ratings $r_{ui}$ are continuous values ranging from $1$ to $R$, then
mathematically, a biased recommender system can be represented as follows:

  1. 1.

Each user $u$, while rating an item $i$, scales down the maximum rating $R$ by
$e^{p_{ui}}$. $p_{ui}$ is a random variable, drawn from a distribution
function $P_{u}(I)$, which has a mean value of $\alpha_{u}$. $p_{ui}$
represents the logarithm of the true preference of the user $u$ for the item
$i$. For the sake of brevity, we call it log-preference of the user $u$ for
the item $i$. Hence $e^{p_{ui}}$ is a representation of the true preference of
user $u$ for the item $i$.

  2. 2.

In case the item is associated with the disadvantaged group, the user $u$
further scales down the rating of the item by a factor $e^{q_{ui}}$. $q_{ui}$
is a random variable, drawn from a distribution function $Q_{u}(I)$ having a
mean value of $\beta_{u}$. $q_{ui}$ represents the logarithm of the biasedness
of the user $u$ shown to the item $i$. For the sake of brevity, we call it the
log-bias of the user $u$ for the book $i$. Hence $e^{q_{ui}}$ represents the
biasedness of the user $u$ for the book $i$.

  3. 3.

For each user $u$, $\beta_{u}$ is sampled from the a distribution function
$\Omega(x)$ which governs the global log-bias tendency of the users. We denote
the mean value of $\Omega(x)$ by $\gamma$.

Thus, ratings $r_{ui}$ can be expressed as:

$r_{ui}=\begin{cases}R/e^{p_{ui}},&\textit{if $i$ is associated with
advantaged group}\\\ R/e^{p_{ui}}e^{q_{ui}},&\textit{if $i$ is associated with
disadvantaged group}\end{cases}$ (1)

We now present a detailed description of each of the step.

###  3.1 Estimating the mean value for log-bias

The geometric mean of the ratings given by a user $u$ to the items associated
with disadvantaged and advantaged groups, denoted by $r_{ud}$ and $r_{ua}$
respectively, are given by the following expressions:

$\displaystyle r_{ud}=\left(\prod_{i\in\mathbb{D}\cap
X_{u}}r_{ui}\right)^{1/|\mathbb{D}\cap X_{u}|}\ \text{ and }\
r_{ua}=\left(\prod_{i\in\mathbb{A}\cap X_{u}}r_{ui}\right)^{1/|\mathbb{A}\cap
X_{u}|}$

Further, the log bias in the user profile $p_{u}$, is given by
$\theta_{u}=\ln\left(\frac{r_{ua}}{r_{ud}}\right)$.

We use geometric mean to compute the average rating of a user due to the
following reasons: 1) It is less biased towards very high scores as compared
to arithmetic meanNeve and Palomares (2019) and 2) when cold users are
involved, aggregating recommendations using the geometric mean is more robust
as compared to arithmetic meanValcarce et al. (2020).

The below lemma shows that $\theta_{u}$ is an unbiased estimator of
$\beta_{u}$.

######  Lemma 1.

The expectation of log-bias, $\theta_{u}$ in the user profile $p_{u}$
represents the mean value of the log-bias, $\beta_{u}$ of the user $u$.

###### Proof.

Let us denote $m=|\mathbb{D}\cap X_{u}|$ and $n=|\mathbb{A}\cap X_{u}|$ to be
the number of items associated with disadvantaged and advantaged group
respectively in user profile $p_{u}$. Then,

$\displaystyle\theta_{u}$
$\displaystyle=\ln\left(\frac{r_{ua}}{r_{ud}}\right)=\ln\left[\frac{\left(\prod_{y=1}^{m}e^{p_{uy}}e^{q_{uy}}\right)^{\frac{1}{m}}}{\left(\prod_{x=1}^{n}e^{p_{ux}}\right)^{\frac{1}{n}}}\right]$
(Using equation 1)
$\displaystyle=\frac{1}{m}\sum_{y=1}^{m}q_{uy}+\frac{1}{m}\sum_{y=1}^{m}p_{uy}-\frac{1}{n}\sum_{x=1}^{n}p_{ux}$

Taking expectation both sides:

$\mathbb{E}[\theta_{u}]=\mathbb{E}\left[\frac{1}{m}\sum_{y=1}^{m}q_{uy}+\frac{1}{m}\sum_{y=1}^{m}p_{uy}-\frac{1}{n}\sum_{x=1}^{n}p_{ux}\right]$
(2)

Using linearity of expectation and some simplification, we get:

$\displaystyle\mathbb{E}[\theta_{u}]$
$\displaystyle=\frac{1}{m}\sum_{y=1}^{m}\mathbb{E}[q_{uy}]+\frac{1}{m}\sum_{y=1}^{m}\mathbb{E}[p_{uy}]-\frac{1}{n}\sum_{x=1}^{n}\mathbb{E}[p_{ux}]$
$\displaystyle=\frac{1}{m}\sum_{y=1}^{m}\beta_{u}+\frac{1}{m}\sum_{y=1}^{m}\alpha_{u}-\frac{1}{n}\sum_{x=1}^{n}\alpha_{u}$

Thus, $\mathbb{E}[\theta_{u}]=\beta_{u}$. ∎

Once we get the log biasedness tendencies of users, we use them to produce the
debiased ratings for the given dataset.

###  3.2 Debiasing the Dataset

The debiased rating of the item $i$ associated with disadvantaged group and
rated by user $u$ is given as $d_{ui}=r_{ui}e^{\theta_{u}}$ We now provide the
main theorem of our paper.

######  Theorem 2.

$\ln(d_{ui})$ is the unbiased estimator of the log of the true rating of the
item $i$.

###### Proof.

$\ln(d_{ui})=\theta_{u}+\ln(r_{ui})=\theta_{u}+\ln R-p_{ui}-q_{ui}$. Last
equality is obtained from Equation 1. Taking expectation both sides:

$\displaystyle\mathbb{E}(\ln(d_{ui}))$
$\displaystyle=\mathbb{E}[\theta_{u}]+\mathbb{E}[\ln
R]-\mathbb{E}[q_{ui}]-\mathbb{E}[p_{ui}]$ $\displaystyle=\beta_{u}+\ln
R-\beta_{u}-\alpha_{u}$ (Using Lemma 1) $\displaystyle=\ln
R-\alpha_{u}=\ln\left(\frac{R}{e^{\alpha_{u}}}\right)$

As we can see, the expected value of $\ln(d_{ui})$ contains only the term
representing the true preference of the item for user $u$. ∎

Thus, instead of $r_{ui}$, ratings $d_{ui}$ are fed into the recommender
system to generate the predicted unbiased ratings $\tilde{d}_{ui}$. Simply
removing the bias from the user’s rating could severely affect the system’s
accuracy because the bias of an individual user reflects their taste. However,
the debiasing step helps prevent the bias of one user from affecting the
recommendation of other users. Next, we use preference corrections by
correcting the predicted rating of the user with respect to his/her own
preference parameter.

###  3.3 Preference Correction to Improve Accuracy

Note that when the users are inherently biased against a group of items,
$\mathcal{D}$ then showing the items from $\mathcal{D}$ naively to these users
will severely affect the accuracy of the system. The goal of this work is not
just to promote the exposure of the items among the two groups but is to not
let the bias of one user creep into the bias of the other user. This was
achieved via debiasing the dataset. Once the debiased ratings are generated,
the accuracy of the system is maintained by introducing a correction factor.
Although providing us with higher accuracy, the idea to re-introduce the
correction factor may lead to an overall increase in the individual biases.
This on a prima-facie may look self-defeating, but we need to note that final
ratings still have significantly less bias than original ratings. If we do not
introduce the correction factor, the users might flock to a substantial bias
platform due to poor accuracy.

The correction is achieved via multiplying the predicted ratings of items
associated with disadvantaged group by a factor $e^{-\theta_{u}}$. Thus, the
final recommended ratings will be given as
$\tilde{r}_{ui}=\tilde{d}_{ui}e^{-\theta_{u}}$. Similar to the calculation of
bias in the dataset, we can now compute the bias in the recommendation
profile.

###  3.4 Bias in recommendation profile

We generate recommendations for the users in the test set $\mathcal{T}$. The
recommendation profile for a user $u\in\mathcal{T}$ is denoted by
$\tilde{p}_{u}=\\{\tilde{X}_{u},\tilde{R}_{u}\\}$, which represents the set of
recommended books ($\tilde{X}_{u}$) for the user $u$ and their predicted
ratings ($\tilde{R}_{u}=\\{\tilde{r}_{ui}\\}_{i\in\tilde{X}_{u}}$). Let the
set of items associated with disadvantaged and advantaged groups be denoted by
$\tilde{\mathbb{D}}$ and $\tilde{\mathbb{A}}$ respectively. The average
predicted ratings of the items associated with disadvantaged and advantaged
groups, denoted by $\tilde{r}_{ud}$ and $\tilde{r}_{ua}$ respectively, are
given by:
$\tilde{r}_{ud}=\left(\prod_{i\in\tilde{\mathbb{D}}\cap\tilde{X_{u}}}\tilde{r}_{ui}\right)^{1/|\tilde{\mathbb{D}}\cap\tilde{X}_{u}|}\
\text{ and }\
\tilde{r}_{ua}=\left(\prod_{i\in\tilde{\mathbb{A}}\cap\tilde{X_{u}}}\tilde{r}_{ui}\right)^{1/|\tilde{\mathbb{A}}\cap\tilde{X}_{u}|}$
where $\tilde{r}_{ui}$ is the predicted rating given to item $i$ in the
recommendation-profile generated for a user $u$. The log-bias in the
recommendation-profile $p_{u}$, denoted by $\tilde{\theta}_{u}$, is then given
by $\tilde{\theta}_{u}=\ln\left(\frac{\tilde{r}_{ua}}{\tilde{r}_{ud}}\right)$.
For an unbiased recommendation-profile, $\tilde{\theta}_{u}=0$. A profile
biased against disadvantaged groups will have $\tilde{\theta}_{u}>0$. We can
then compute the overall bias of the recommender system by taking the average
overall users, and this average gives us the estimated value of $\gamma$.

##  4 Dataset

To evaluate the proposed model, we run experiments on two publicly available
book rating datasets, the Book-Crossing dataset, originally put together by
Ziegler et al. (2005b) and the Amazon Book Review dataset, put together by Ni
et al. (2019). We further process this dataset through the following stages:

###  4.1 Book Author Identification

Their unique ISBNs identify the books in both datasets. We identified the
authors of the books present in the datasets via their ISBN numbers using the
following three API services: _Google Books API_ APIs (2 24), _ISBNdb API_
ISBNDB (2 27), and _Open Library API_ OpenLibrary (3 02). We could not
identify the authors of some of the books. Hence we discarded those books from
the dataset.

###  4.2 Author Gender Identification

We identified the genders of the authors via their first names. We used
_Genderize.io_ the gender of a name (03 5), an API service dedicated to
identifying the gender given the first name of the person. We used a minimum
confidence threshold of $90\%$ for gender identification. We could not
identify the gender of some of the authors. We discard the books written by
those authors from the dataset.

###  4.3 Filtering

We filtered the Book-Crossing dataset to include only those books with at
least $50$ ratings and only those users who have rated at least $50$ books.
Amazon dataset was significantly larger as compared to the Book-Crossing
dataset. We filtered it to include only those books with at least $100$
ratings and only those users who have rated at least $100$ books. We did this
filtering so that recommender algorithms have much data to produce accurate
recommendations. The statistics of filtered datasets are mentioned in Table 1.
The number of books written by male authors is almost equal to that of female
authors for both datasets.

Statistic | Amazon | Book-Crossing
---|---|---
Number of male authored books | 58369 | 829
Number of female authored books | 58220 | 806
Number of users | 44792 | 376
Table 1: Dataset details

##  5 Experimental Results

###  5.1 Input Bias

We show the distributions of log-bias tendency ($\theta_{u}$) of the users in
the Amazon dataset and the Book-Crossing dataset in Figure 2. We observe that
the mean log-bias tendency over all the users in the Amazon dataset is higher
(0.176) than that of the Book-Crossing dataset (0.157)111code is available at
https://github.com/venomNomNom/genderBias.git .

(a) Amazon dataset

(b) Book-Crossing dataset

Figure 2: User log-bias in the original dataset

###  5.2 Output Bias

We randomly separate $20\%$ of users in each dataset as the test group. We
generate the recommendations for the users in the test group using two
K-nearest neighbors-based algorithms, UserKNN and ItemKNN, and two matrix
factorization-based algorithms, Alternating Least Square and Singular Value
Decomposition. These algorithms were selected because the accuracy and ranking
relevancy of the recommendations produced by them were among the highest
values compared with other algorithms. Hence coupling our model with them
would best highlight the effects brought about by the same. We calculate the
estimated value of log-bias ($\tilde{\theta}_{u}$) and accuracy in the
recommendations separately for each algorithm applied on the two datasets. For
this, we use two error measures, the Root Mean Squared Error (RMSE) and the
Mean Absolute Error (MAE), and two ranking relevance parameters, Normalized
Discounted Cumulative Gains and Mean Reciprocal Rank.

We first begin plotting the log-bias ($\tilde{\theta}_{u}$) distribution for
the recommendations produced by the algorithms without employing our debiased
model in Figures 3 and 6 for Amazon and Book-Crossing datasets respectively.
We compute the log-bias by feeding biased ratings $r_{ui}$ to the four
algorithms. Table 2 shows the exact values for comparison with other cases. We
next deploy our model partially. We leave out the preference correction phase
and produce the recommendations using the algorithms mentioned before by
feeding the debiased ratings $d_{ui}$ to these algorithms. We estimate the
mean log-bias tendency in the recommendations $\tilde{\theta}_{u}$ using
debiased ratings produced by the algorithms $\tilde{d}_{ui}$. The log-bias
($\tilde{\theta_{u}}$) distribution for the recommendations produced by the
algorithms for the Amazon as well as Book-Crossing dataset after partial
deployment of the model is depicted in the Figure 4 and Figure 7 with the
exact values provided in the Table 2. As can be seen, there is a significant
reduction in log-bias tendency ($64.38\%$) in the Amazon dataset and
($53.67\%$) in Book-Crossing dataset for the UserKNN algorithm. However, we
also see an increase in error rates on both datasets. This is because the test
data itself contains biases.

Finally, we deploy our complete model and repeat the experiment. The log-bias
($\tilde{\theta_{u}}$) distribution for the recommendations produced by the
algorithms after deployment of the complete model is depicted in Figures 5 and
8 with the values in the Table 2. As can be seen, there is still a significant
reduction in mean log-bias tendency, which reduces by $42.39\%$ in the Amazon
dataset and by $37.82\%$ in the case of the Book-Crossing dataset for UserKNN
algorithm. The accuracy loss, however, is insignificant, making this trade-off
advantageous. Figure 9 presents the percentage gain in bias reduction for both
the dataset. The percentage loss in accuracy is depicted in figures 10 and 11
for Amazon and Book-Crossing datasets respectively. The percentage loss in
ranking relevancy metrics are depicted in figures 12 and 13 respectively.

(a) UserKNN

(b) ItemKNN

(c) ALS

(d) SVD

Figure 3: Output log-bias in AZ dataset without employing the model

(a) UserKNN

(b) ItemKNN

(c) ALS

(d) SVD

Figure 4: Output log-bias in AZ dataset with debiasing

(a) UserKNN

(b) ItemKNN

(c) ALS

(d) SVD

Figure 5: Output log-bias in AZ dataset with reinserting the biases

(a) UserKNN

(b) ItemKNN

(c) ALS

(d) SVD

Figure 6: Output log-bias in BX dataset without employing the model

(a) UserKNN

(b) ItemKNN

(c) ALS

(d) SVD

Figure 7: Output log-bias in BX dataset with debiasing

(a) UserKNN

(b) ItemKNN

(c) ALS

(d) SVD

Figure 8: Output log-bias in BX dataset with reinserting the biases |  | AZ | BX
---|---|---|---
Case | Algorithm |  | Mean
---
log-bias
RMSE | MAE | NDCG |  | Reciprocal
---
Rank
| Mean
---
log-bias
RMSE | MAE | NDCG |  | Reciprocal
---
Rank
Recommendations without model | UserKNN | 0.137 | 0.808 | 0.693 | 0.452 | 0.498 | 0.122 | 1.580 | 1.178 | 0.264 | 0.272
ItemKNN | 0.129 | 0.736 | 0.580 | 0.597 | 0.643 | 0.106 | 1.511 | 1.304 | 0.313 | 0.412
ALS | 0.164 | 0.873 | 0.829 | 0.281 | 0.447 | 0.158 | 1.815 | 1.642 | 0.235 | 0.370
SVD | 0.175 | 0.790 | 0.753 | 0.342 | 0.471 | 0.169 | 1.761 | 1.626 | 0.277 | 0.296
Recommendations without preference correction phase | UserKNN | 0.049 | 1.103 | 0.921 | 0.0224 | 0.0278 | 0.057 | 2.468 | 1.754 | 0.0232 | 0.0245
ItemKNN | 0.063 | 1.076 | 0.873 | 0.0229 | 0.0204 | 0.054 | 2.463 | 2.055 | 0.0142 | 0.0271
ALS | 0.093 | 1.281 | 1.1211 | 0.0161 | 0.0394 | 0.087 | 2.752 | 2.175 | 0.0421 | 0.0736
SVD | 0.071 | 1.257 | 1.183 | 0.0138 | 0.0206 | 0.072 | 2.601 | 1.979 | 0.0261 | 0.0240
Recommendations with preference correction phase | UserKNN | 0.079 | 0.871 | 0.738 | 0.3982 | 0.4462 | 0.076 | 1.799 | 1.298 | 0.2099 | 0.2317
ItemKNN | 0.080 | 0.824 | 0.661 | 0.5236 | 0.6121 | 0.073 | 1.785 | 1.516 | 0.2538 | 0.3560
ALS | 0.121 | 0.982 | 0.903 | 0.2391 | 0.3853 | 0.119 | 2.022 | 1.768 | 0.1626 | 0.2831
SVD | 0.103 | 0.872 | 0.847 | 0.2989 | 0.4159 | 0.114 | 1.988 | 1.731 | 0.2258 | 0.2481
Table 2: Summary of Results

(a) AZ dataset

(b) BX dataset

Figure 9: Bias reduction

(a) In terms of RMSE

(b) In terms of MAE

Figure 10: Accuracy loss for AZ dataset

(a) In terms of RMSE

(b) In terms of MAE

Figure 11: Accuracy loss for BX dataset

(a) In terms of NDCG

(b) In terms of Reciprocal Rank

Figure 12: Ranking relevancy loss for AZ dataset

(a) In terms of NDCG

(b) In terms of Reciprocal Rank

Figure 13: Ranking relevancy loss for BX dataset

We next conduct significance testing to validate the log-bias reduction. Table
3 shows the p-values obtained from left-tail significance tests on the log-
bias of the recommendations made for the users in the sample. We can see from
the p-value for the Amazon datasets that the bias reduction is significant.
For the Book-Crossing dataset, the significance of the bias reduction is less
pronounced. One of the prominent reasons for this is that the test sample size
for the Book-Crossing dataset was relatively small due to the small number of
users in the dataset. In essence, the utility of the recommender system is
maintained while reducing the log-bias tendency in the recommendations.

| AZ | BX
---|---|---
Algorithm | $\bar{x}$ | $\mu$ | $\sigma$ | $z$ | $p$ | $\bar{x}$ | $\mu$ | $\sigma$ | $z$ | $p$
UserKNN | 0.079 | 0.137 | 0.307 | -17.90 | $<10^{-5}$ | 0.076 | 0.122 | 0.343 | -1.164 | 0.122
ItemKNN | 0.080 | 0.129 | 0.381 | -12.06 | $<10^{-5}$ | 0.073 | 0.106 | 0.362 | -0.780 | 0.218
ALS | 0.121 | 0.164 | 0.394 | -10.46 | $<10^{-5}$ | 0.119 | 0.158 | 0.464 | -0.738 | 0.230
SVD | 0.103 | 0.175 | 0.354 | -19.27 | $<10^{-5}$ | 0.114 | 0.169 | 0.335 | -1.413 | 0.079
Table 3: Significance test results for bias reduction

We further observe that the bias reduction is more in the case of UserKNN
based recommendations than the ItemKNN based recommendations. This observation
can be attributed to the fact that our model addresses the bias originating
from the distortion in ratings from the users’ side. It compares the ratings
of an item given by a particular user with the appropriately scaled average of
ratings given by other users to that item in the dataset. It, therefore,
resonates with the UserKNN algorithm, which predicts the ratings of an item
for a particular user based on the ratings of that item for his or her peers.
The ItemKNN algorithm, on the other hand, predicts the ratings of an item for
a particular user based on the ratings given to similar items by that user.
The model does not sit squarely with ItemKNN. Thus the bias reduction in
UserKNN is more as compared to that in the case of ItemKNN. We further observe
that the bias reduction is more in the case of the AZ dataset as compared to
the BX dataset. This observation can be attributed to the AZ dataset having a
higher input mean log-bias tendency. Further, the AZ dataset has a
significantly larger number of users and items which leads to a more accurate
estimation of user bias scores and, therefore, more effective bias mitigation.

We observe that accuracy and ranking relevancy loss is, in general, higher for
ItemKNN as compared to UserKNN. This is due to the fact that the model
quantifies the bias of users by comparing the ratings given by them to
particular items with a scaled average of ratings given by their peers to
those items. This resonates with the UserKNN algorithm, which predicts user
ratings for particular items based on the ratings of similar users. Thus the
model is better oriented towards the UserKNN algorithm, giving better accuracy
and bias reduction in its case. In the case of matrix factorization
algorithms, the accuracy and ranking relevancy losses are relatively
comparable. It is not clear which one of the two algorithms is more coherent
with the model.

We further observe that accuracy loss on BX dataset is higher than that of AZ
dataset. This observation can be attributed to the fact that the user and item
base of the AZ dataset is higher as compared to the BX dataset. Thus, the bias
score estimates are more accurate, which provides more accurate predictions of
the item scores for the users when reinserted into the recommendations.

##  6 Conclusion and Future Work

We proposed a model to quantify and mitigate the bias in the explicit feedback
given by the users to different items. We theoretically showed that the
debiased ratings produced by our model are unbiased estimators of the true
preference of the users for the books. With the help of comprehensive
experiments on two publically available book datasets, we show a significant
reduction in the bias (almost 40%) with just 10% decrease in accuracy using
the UserKNN algorithm. Similar trends were observed for other algorithms such
as ItemKNN, ALS, and SVD. Our model is independent of these algorithms’
choices and can be applied with any recommendation algorithm. We used book
recommender system because we were able to generate the gender information
from publicly available APIs. Our model is not restricted to book recommender
system as long as protected attribute information about the items is known. We
leave extension of the model to missing protected attribute as an interesting
future work. It will be an interesting direction to see if the ideas from fair
classification literature with missing protected attributes Coston et al.
(2019) can be leveraged. We further did not address the bias originating from
fewer ratings for a female-authored book than a male-authored one. We leave
extending the model to the bias originating from lesser number of ratings and
extensively studying the model for other recommender systems as the future
directions.

## References

  * (1)
  * Amatriain et al. (2011) Xavier Amatriain, Alejandro Jaimes, Nuria Oliver, and Josep Pujol. 2011.  _Data Mining Methods for Recommender Systems_.  39–71.  https://doi.org/10.1007/978-0-387-85820-3_2
  * APIs (2 24) Google Books APIs. Accessed: 2021-02-24.  https://developers.google.com/books
  * Boratto et al. (2019) Ludovico Boratto, Gianni Fenu, and Mirko Marras. 2019\.  _The Effect of Algorithmic Bias on Recommender Systems for Massive Open Online Courses_.  457–472.  https://doi.org/10.1007/978-3-030-15712-8_30
  * Burke (2017) Robin Burke. 2017\.  Multisided Fairness for Recommendation.  (07 2017).
  * Coston et al. (2019) Amanda Coston, Karthikeyan Natesan Ramamurthy, Dennis Wei, Kush R Varshney, Skyler Speakman, Zairah Mustahsan, and Supriyo Chakraborty. 2019.  Fair transfer learning with missing protected attributes. In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_. 91–98.
  * Dwork et al. (2011) Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Rich Zemel. 2011.  Fairness Through Awareness.  _CoRR_ abs/1104.3913 (04 2011).  https://doi.org/10.1145/2090236.2090255
  * Ekstrand et al. (2018) Michael Ekstrand, Mucun Tian, Mohammed Kazi, Hoda Mehrpouyan, and Daniel Kluver. 2018.  Exploring Author Gender in Book Rating and Recommendation.  (08 2018).  https://doi.org/10.1145/3240323.3240373
  * Hajian et al. (2016) Sara Hajian, Francesco Bonchi, and Carlos Castillo. 2016\.  Algorithmic Bias: From Discrimination Discovery to Fairness-aware Data Mining. 2125–2126.  https://doi.org/10.1145/2939672.2945386
  * Hajian and Domingo-Ferrer (2013) Sara Hajian and Josep Domingo-Ferrer. 2013.  A Methodology for Direct and Indirect Discrimination Prevention in Data Mining.  _IEEE Transactions on Knowledge and Data Engineering_ (07 2013).  https://doi.org/10.1109/TKDE.2012.72
  * Hajian et al. (2014a) Sara Hajian, Josep Domingo-Ferrer, and Oriol Farràs. 2014a.  Generalization-based privacy preservation and discrimination prevention in data publishing and mining.  _Data Mining and Knowledge Discovery_ (09 2014).  https://doi.org/10.1007/s10618-014-0346-1
  * Hajian et al. (2014b) Sara Hajian, Josep Domingo-Ferrer, Anna Monreale, Dino Pedreschi, and Fosca Giannotti. 2014b.  Discrimination- and privacy-aware patterns.  _Data Mining and Knowledge Discovery_ 29 (11 2014).  https://doi.org/10.1007/s10618-014-0393-7
  * Herlocker et al. (2004) Jonathan L. Herlocker, Joseph A. Konstan, Loren G. Terveen, and John T. Riedl. 2004.  Evaluating Collaborative Filtering Recommender Systems.  _ACM Trans. Inf. Syst._ 22, 1 (Jan. 2004), 5–53.  https://doi.org/10.1145/963770.963772
  * Hurley and Zhang (2011) Neil Hurley and Mi Zhang. 2011.  Novelty and Diversity in Top-N Recommendation – Analysis and Evaluation.  _ACM Trans. Internet Technol._ 10, 4, Article 14 (March 2011), 30 pages.  https://doi.org/10.1145/1944339.1944341
  * ISBNDB (2 27) ISBN Database | ISBNDB. Accessed: 2021-02-27.  https://isbndb.com/isbn-database
  * Kamiran et al. (2010) Faisal Kamiran, Toon Calders, and Mykola Pechenizkiy. 2010\.  Discrimination Aware Decision Tree Learning. 869–874.  https://doi.org/10.1109/ICDM.2010.50
  * Kamiran et al. (2012) Faisal Kamiran, Asim Karim, and Xiangliang Zhang. 2012\.  Decision Theory for Discrimination-Aware Classification.  _Proceedings - IEEE International Conference on Data Mining, ICDM_ , 924–929.  https://doi.org/10.1109/ICDM.2012.45
  * Knijnenburg et al. (2012) Bart Knijnenburg, Martijn Willemsen, soncu gantner, and newell. 2012\.  Explaining the user experience of recommender systems.  _User Modeling and User-Adapted Interaction_ 22 (10 2012), 441–504.  https://doi.org/10.1007/s11257-011-9118-4
  * Leavy et al. (2020) Susan Leavy, Gerardine Meaney, Karen Wade, and Derek Greene. 2020\.  Mitigating Gender Bias in Machine Learning Data Sets.  (05 2020).
  * Mancuhan and Clifton (2014) Koray Mancuhan and Chris Clifton. 2014.  Combating discrimination using Bayesian networks.  _Artificial Intelligence and Law_ 22 (06 2014).  https://doi.org/10.1007/s10506-014-9156-4
  * Mansoury et al. (2020) Masoud Mansoury, Himan Abdollahpouri, Jessie Smith, Arman Dehpanah, Mykola Pechenizkiy, and Bamshad Mobasher. 2020.  Investigating Potential Factors Associated with Gender Discrimination in Collaborative Recommender Systems.  (02 2020).
  * Neve and Palomares (2019) James Neve and Ivan Palomares. 2019.  Latent factor models and aggregation operators for collaborative filtering in reciprocal recommender systems. In _Proceedings of the 13th ACM Conference on Recommender Systems_. 219–227.
  * Ni et al. (2019) Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019.  Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects. 188–197.  https://doi.org/10.18653/v1/D19-1018
  * OpenLibrary (3 02) Developers API | OpenLibrary. Accessed: 2021-03-02.  https://openlibrary.org/developers/api
  * Pedreschi et al. (2008) Dino Pedreschi, Salvatore Ruggieri, and Franco Turini. 2008\.  Discrimination-aware data mining.  _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 560–568.  https://doi.org/10.1145/1401890.1401959
  * Pedreschi et al. (2009) Dino Pedreschi, Salvatore Ruggieri, and Franco Turini. 2009\.  Measuring Discrimination in Socially-Sensitive Decision Records. 581–592.  https://doi.org/10.1137/1.9781611972795.50
  * Ruggieri et al. (2014) Salvatore Ruggieri, Sara Hajian, Faisal Kamiran, and Xiangliang Zhang. 2014.  Anti-discrimination Analysis Using Privacy Attack Strategies.  https://doi.org/10.1007/978-3-662-44851-9_44
  * Ruggieri et al. (2010) Salvatore Ruggieri, Dino Pedreschi, and Franco Turini. 2010\.  Data Mining for Discrimination Discovery.  _TKDD_ 4 (05 2010).  https://doi.org/10.1145/1754428.1754432
  * Shakespeare et al. (2020) Dougal Shakespeare, Lorenzo Porcaro, Emilia Gómez, and Carlos Castillo. 2020.  Exploring Artist Gender Bias in Music Recommendation.  (09 2020).
  * Shani and Gunawardana (2011) Guy Shani and Asela Gunawardana. 2011.  _Evaluating Recommendation Systems_. Vol. 12.  257–297.  https://doi.org/10.1007/978-0-387-85820-3_8
  * Thanh et al. (2011) Binh Thanh, Salvatore Ruggieri, and Franco Turini. 2011\.  k-NN as an Implementation of Situation Testing for Discrimination Discovery and Prevention.  _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 502–510.  https://doi.org/10.1145/2020408.2020488
  * the gender of a name (03 5) Genderize.io | Deteremine the gender of a name. Accessed: 2021-03-5.  https://genderize.io/
  * Tsintzou et al. (2018) Virginia Tsintzou, Evaggelia Pitoura, and Panayiotis Tsaparas. 2018.  Bias Disparity in Recommendation Systems.  (11 2018).
  * Valcarce et al. (2020) Daniel Valcarce, Alejandro Bellogín, Javier Parapar, and Pablo Castells. 2020.  Assessing ranking metrics in top-N recommendation.  _Information Retrieval Journal_ 23 (2020), 411–448.
  * Zemel et al. (2013) R. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork. 2013.  Learning fair representations.  _30th International Conference on Machine Learning, ICML 2013_ (01 2013), 1362–1370.
  * Ziegler et al. (2005a) Cai-Nicolas Ziegler, Sean M. McNee, Joseph A. Konstan, and Georg Lausen. 2005a.  Improving Recommendation Lists through Topic Diversification _(WWW ’05)_. Association for Computing Machinery, New York, NY, USA, 22–32.  https://doi.org/10.1145/1060745.1060754
  * Ziegler et al. (2005b) Cai-Nicolas Ziegler, Sean M. McNee, Joseph A. Konstan, and Georg Lausen. 2005b.  Improving Recommendation Lists through Topic Diversification. In _Proceedings of the 14th International Conference on World Wide Web_ (Chiba, Japan) _(WWW ’05)_. Association for Computing Machinery, New York, NY, USA, 22–32.  https://doi.org/10.1145/1060745.1060754