# Good Secretaries, Bad Truck Drivers?
Occupational Gender Stereotypes in Sentiment Analysis

Jayadev Bhaskaran
ICME, Stanford University, USA
<EMAIL_ADDRESS>
Isha Bhallamudi
Dept. of Sociology, UC Irvine, USA
<EMAIL_ADDRESS>

###### Abstract

In this work, we investigate the presence of occupational gender stereotypes
in sentiment analysis models. Such a task has implications for reducing
implicit biases in these models, which are being applied to an increasingly
wide variety of downstream tasks. We release a new gender-balanced
dataset111Link to dataset: https://bit.ly/2HLSKnf of $800$ sentences
pertaining to specific professions and propose a methodology for using it as a
test bench to evaluate sentiment analysis models. We evaluate the presence of
occupational gender stereotypes in $3$ different models using our approach,
and explore their relationship with societal perceptions of occupations.

##  1 Motivation

Social Role Theory (Eagly and Steffen, 1984) shows that our ideas about gender
are shaped by observing, over time, the roles that men and women occupy in
their daily lives. These ideas can crystallize into rigid stereotypes about
how men and women ought to behave, and what work they can and cannot do.
Gendered stereotypes are powerful precisely for this reason: they define
desirable and expected traits, roles and behaviors in people, and go beyond
description to prescription. Such biases from the social world, when they map
onto machine learning models, serve to reinforce and propagate stereotypes
further.

In this paper, we look specifically at occupational gender stereotypes in the
context of sentiment analysis. Sentiment analysis is increasingly being
applied for recruitment, employee retention and job satisfaction in the
corporate world (Costa and Veloso, 2015). Given the prevalence of occupational
gender stereotypes, our study primarily deals with the question of whether
sentiment analysis models display and propagate these stereotypes. To
contextualize and ground our study, we first provide a summary of the relevant
sociological literature on occupational gender stereotypes.

###  1.1 Background

Sociological studies as early as 1975 (Shinar, 1975) investigate gender
stereotypes of occupations, and rank occupations in terms of how “masculine”,
“feminine” or neutral they are perceived to be. Cejka and Eagly (1999)
successfully predicted the gender distribution of occupations based on beliefs
about how specific _gender-stereotypical_ attributes (such as “masculine
physical”) contribute to occupational success. Such beliefs - that success in
a _male_ dominated profession, for example, requires _male-specific_ traits -
directly contribute to sex segregation in occupations. The study also found
that high occupational prestige and wages are _strongly correlated_ with
masculine images. Together, this goes to show that occupational structure is
deeply shaped by gender. More recently, Haines et al. (2016) investigate how
and whether gender stereotypes have changed between 1983 and 2014, and find
conclusive evidence that occupational gender stereotypes have _persisted
strongly_ through the ages and remain stable. There is ample sociological
evidence to show that occupational gender stereotypes have _not_ undergone
substantial modification since the entry of women into the workplace, and that
they remain pervasive and widely held by both men _and_ women (Glick et al.,
1995; Haines et al., 2016).

Since occupational gender stereotypes are shaped by subjective factors and
_not_ objective reality, they remain resistant to contrary evidence. Theories
such as the backlash hypothesis (Rudman and Phelan, 2008) further explain
their persistence: this theory shows how women in the workplace must
disconfirm female stereotypes in order to be perceived as competent leaders,
yet traits of ambition and capability in women evoke negative reactions which
present a barrier to every level of occupational success.

The implications of occupational gender stereotypes are profound. Children and
adolescents are particularly sensitive to gendered language used to describe
occupations and form rigid occupational gender stereotypes based on this
(Vervecken et al., 2013). In adults, occupational gender stereotypes directly
contribute to the existence of unequal compensation and discriminatory hiring.
They also lead to self-fulling prophecies: for instance, individuals may not
apply to certain jobs in the first place because they think they don’t fit the
gender stereotype for occupational success in that field (Kay et al., 2015).

In the following section, we discuss relevant prior work on gender bias from
the NLP literature. In Section 3 we describe our methodology, dataset, and
experiments in greater detail. In Section 4, we present and analyze our
results, and finally, Section 5 describes possible directions of future work
and concludes222Source code for this paper:
github.com/jayadevbhaskaran/gendered-sentiment.

##  2 Prior Work

Word embeddings have been the bedrock of neural NLP models ever since the
arrival of word2vec (Mikolov et al., 2013), and a variety of topics related to
biases with word embeddings have been studied in prior literature. Garg et al.
(2018) show the presence of stereotypes in word embeddings through the ages,
while Bolukbasi et al. (2016) demonstrate explicit examples of social biases
that are introduced into word embeddings trained on a large text corpus. Prior
work has also dealt with occupational gender stereotypes in different areas of
NLP. Caliskan et al. (2017) formulate a method to test biases (including
gender stereotypes) in word embeddings, while Rudinger et al. (2018)
investigate such stereotypes in the context of coreference resolution. There
have also been efforts to _debias_ word embeddings (Bolukbasi et al., 2016)
and come up with _gender neutral_ word embeddings (Zhao et al., 2018). These
efforts, however, have attracted criticism suggesting that they do not
actually _debias_ embeddings but instead _redistribute_ the bias across the
embedding landscape (Gonen and Goldberg, 2019).

Figure 1: Simple diagram of our task definition.

Recent trends have been towards replacing fixed word embeddings with large
pretrained _contextual_ representations as building blocks for NLP tasks. The
rise of this paradigm is characterized by the use of language models for
pretraining, exemplified by models such as ELMo (Peters et al., 2018), ULMFit
(Howard and Ruder, 2018), GPT (Radford, 2018), and BERT (Devlin et al., 2018).

These models have shown marked improvements over word vector based approaches
for a variety of tasks. However, their complexity leads to a tradeoff in terms
of interpretability. Recent works have investigated gender biases in such deep
contextual representations (May et al., 2019; Basta et al., 2019) as well as
their applications to coreference resolution (Zhao et al., 2019; Webster et
al., 2018); however, no prior work has dealt with such models in the context
of occupational gender stereotypes in sentiment analysis.

Kiritchenko and Mohammad (2018) introduce the Equity Evaluation Corpus, a
dataset used for measuring racial and gender biases in sentiment analysis-like
systems. It was initially used to evaluate systems that predicted emotion and
valence of Tweets (Mohammad et al., 2018). We use a similar approach to create
a new dataset for measuring gender differences with a specific focus on
occupational gender stereotypes. Our approach is _model-independent_ and can
be used for _any_ sentiment analysis system, irrespective of model complexity.

##  3 Methodology

We create a dataset of $800$ sentences, each with the following structure:
noun is a/an profession. Here, noun corresponds to a _male_ or _female_ noun
phrase, such as “This boy”/“This girl”, and profession is one of $20$
different professions. Each sentence is an assertion of fact, and by itself
does not seek to exhibit either positive or negative sentiment. Our dataset is
balanced across genders and has $20$ noun phrases for each gender, leading to
a total of $400$ sentences per gender.

The rationale behind our selection of the $20$ professions is to include a
variety of gender distribution characteristics and occupation types, in
correspondence with US Current Population Survey 2018 (CPS) data (Current
Population Survey, 2018) and prior literature (Haines et al., 2016). We select
$5$ professions that are male-dominated (_truck driver, mechanic, pilot, chef,
soldier_) and $5$ that are female-dominated (_teacher, flight attendant,
clerk, secretary, nurse_) - with domination meaning greater than $70\%$ share
in the job distribution. Next, we add professions that are slightly male-
dominated (_scientist, lawyer, doctor_) and slightly female-dominated
(_writer, dancer_), with slight domination meaning a $60-65\%$ share in the
job distribution. We also add _professor_ , which does not have a clear
definition as per CPS but has been known to have different gender splits at
senior and junior levels. Finally, we include two professions that show an
approximately neutral divide (_tailor, gym trainer_) and two which have
experienced significant changes in their gender distribution over time
(_baker, bartender_), with an increasing female representation in recent times
(Haines et al., 2016). As mentioned previously, we also select our set of
occupations with an eye towards representing a range of occupation types.

We evaluate $3$ sentiment analysis models through our experiments. Each model
is trained on the Stanford Sentiment Treebank $2$ train dataset (Socher et
al., 2013), which contains phrases from movie reviews along with binary
($0/1$) sentiment labels. We then evaluate each model on our new corpus and
measure the difference in mean predicted positive class probabilities between
sentences with male nouns and those with female nouns. We test $3$ hypotheses
(one for each model), with the _null hypotheses_ indicating _no difference_ in
means between sentences with _male_ and _female_ nouns. Fig. 1 illustrates our
experimental setup.

Our evaluation methodology is very similar to that used in Kiritchenko and
Mohammad (2018). For each system, we predict the positive class probability
for each sentence. We then apply a _paired_ t-test (since each pair contains a
_male_ and _female_ version of the same template sentence) to measure if the
mean predicted positive class probabilities are different across genders,
using a significance level of $0.01$. Since we test _three_ hypotheses (one
for each system), we apply Bonferroni correction (Bonferroni, 1936) to the
$p$-values that we obtain. In other words, the null hypothesis is rejected
only for calculated $p$-values less than $0.01/3$. We note that we do not
perform any correction to account for the fact that the sentences within each
gender are _not_ iid, and only vary in the noun and profession words.

The $3$ models that we evaluate are as follows:

  * •

M.1: Bag-of-words + Logistic Regression (baseline): We build a simple bag-of-
words model, apply tf-idf weighting, and use logistic regression (implemented
using scikit-learn (Pedregosa et al., 2011)) to classify sentiment. This model
is a very simple approach that has nevertheless been found to work well in
practice for sentiment analysis tasks, and we use it as our baseline model.

  * •

M.2: BiLSTM: We use a bidirectional LSTM implemented in Keras (Chollet et al.,
2015) to predict sentiment. The words in a sentence are represented by
$300$-dimensional GloVe embeddings (Pennington et al., 2014). This model is
more sophisticated than the baseline and captures some contextual information
and long-term dependencies (Hochreiter and Schmidhuber, 1997). This model also
allows us to investigate gender differences that might be introduced through
word embeddings, as described in Bolukbasi et al. (2016).

  * •

M.3: BERT (Devlin et al., 2018): We use a pretrained (uncased) BERT-Base
model333Model source: https://bit.ly/2S8w6Jt and finetune it on the SST-2
dataset. This shows near state-of-the-art performance on a wide variety of NLP
tasks, including sentiment analysis (Devlin et al., 2018).

While analysing the results of our experiments, we measure overall predicted
mean positive probabilities (across genders) for each of the $20$ professions
in our newly created dataset, to identify which professions are rated as
_high-sentiment_ by these models. This helps us investigate relationships
between societal perceptions of occupations and corresponding sentiment
predictions from the models.

Model | Dev Acc. | F - M
---|---|---
M.1 (BoW+LogReg) | 0.827 | 0.035**
M.2 (BiLSTM) | 0.841 | 0.077**
M.3 (BERT) | 0.930 | -0.040**
Table 1: Results. Dev Acc. represents accuracy on SST-2 dev set. F - M
represents difference between means of predicted positive class probabilities
for sentences with _female_ nouns and sentences with _male_ nouns. ** denotes
statistical significance with $p<0.01$ (after applying Bonferroni correction).

We also examine differences in sentiment among equivalent gender pairs (such
as _bachelor_ and _spinster_) for the $20$ pairs in our dataset, to
investigate differences in predicted sentiment between different sets of
male/female noun pairs.

Finally, we examine differences between _male_ and _female_ nouns for each
individual occupation, to understand which occupations are susceptible to
gender stereotyping.

##  4 Results/Analysis

The main results of our experiments are shown in Table 1. Our null hypothesis
is that the predicted positive probabilities for _female_ and _male_ sentences
have identical means. We notice that M.1 (Bag-of-words + Logistic Regression)
and M.2 (BiLSTM) show a statistically significant difference between the two
genders, with higher predicted positive class probabilities for sentences with
_female_ nouns. This effectively represents the biases seen in the SST-2 train
dataset. The dataset has $1182$ sentences containing a _male_ noun with a mean
sentiment of $\mathbf{0.535}$, and $601$ sentences containing a _female_ noun
with a mean sentiment of $\mathbf{0.599}$. Thus, biases present in training
data can get propagated through machine learning models, and our approach can
help identify these.

On the contrary, M.3 (BERT) shows that sentences with _male_ nouns have a
statistically significant higher predicted positive class probability than
sentences with _female_ nouns. One possible reason for this might be biases
that propagate from the pretraining phase in BERT. This finding indicates a
promising direction of future work: investigating the effects of gender biases
in the large pretraining corpus versus those in the smaller fine-tuning corpus
(in our case, the SST-2 train dataset).

Model | Top 3 professions
---|---
BoW+LogReg | Secretary, Teacher, Writer
BiLSTM | Dancer, Secretary, Scientist
BERT | Scientist, Chef, Dancer
Model | Bottom 3 professions
BoW+LogReg | Truck Dr., Fl. Att., (many)
BiLSTM | Truck Dr., Gym Tr., Nurse
BERT | Truck Dr., Clerk, Tailor
Table 2: Top $3$ and bottom $3$ professions per model, based on predicted
positive class probability (agnostic of gender). Note: For BoW+LogReg, (many)
denotes all the professions that did not appear in the SST-2 train dataset.

###  4.1 Social Stereotypes of Occupations

We now look at mean distributions of positive class probability (across
genders) for each profession, as shown in Table 2. We notice that _secretary_
shows up as a high positive sentiment profession in both M.1 and M.2. On
further investigation, we notice that this artefact arises because of the 2002
movie _Secretary_ , starring Maggie Gyllenhall, that has a number of positive
reviews that form a part of the SST-2 train dataset. However, M.3 (BERT) seems
to be impervious to this, indicating that extensive pretraining could have the
potential to remove certain corpus-specific effects that might have lingered
in shallower models.

The profession with the lowest average sentiment score across all $3$ models
is _truck driver_ ; other low scoring professions include _clerk_ , _gym
trainer_ and _flight attendant_. We also note that the highest scoring
profession (average sentiment $0.99$) with M.3 (BERT) is _scientist_ and the
lowest (average sentiment $0.34$) is _truck driver_ , disturbingly reflective
of societal stereotypes about white-collar and blue-collar jobs.

To explore this further, we look at data from the Current Population Survey of
the US Bureau of Labor Statistics (Current Population Survey, 2018). Fig. 2
shows the relationship between median weekly earnings (for occupations where
data is available) and average positive sentiment predicted by BERT. While
there are some outliers, the figure shows a positive correlation between
earnings and sentiment, indicating that the model may have incorporated
societal perceptions around different occupations. We note that this is only a
rough analysis, as not all occupations directly correspond to entries from the
survey data.

Figure 2: Median weekly earnings (Current Population Survey, 2018) vs. mean
predicted positive probability using M.3 (BERT), per profession.

###  4.2 Gendered Stereotypes

We attempt to analyze differences in gender within occupations by studying the
predictions of M.3 (BERT), which incorporates the largest amount of external
data. First, we analyze differences in mean positive class probability between
sentences with male and female nouns for each profession. We notice that
_pilot_ has the highest _positive_ difference between female and male noun
sentences (i.e., _female_ is _higher_), while _flight attendant_ has the _most
negative_ difference (i.e., _male_ is _higher_). This provides an interesting
dichotomy: _pilot_ is a male-dominated profession, while _flight attendant_ is
a female-dominated one.

To test whether these are just artefacts of generic gender bias in the model
or specific to occupational gendered stereotypes, we replace profession with
“person” to create $20$ sentence pairs such as “This man/this woman is a
person.”, and predict the sentiment for these $20$ pairs. We notice that the
difference between _female_ and _male_ noun sentences for the control
experiment is $\mathbf{0.039}$, showing that sentences with _female_ nouns in
the control group exhibit _higher_ positive sentiment that those with _male_
nouns. The three occupations with the most negative difference (i.e., _female_
sentences have lower positive sentiment) are _flight attendant_
($\mathbf{-0.132}$), _bartender_ ($\mathbf{-0.126}$), and _clerk_
($\mathbf{-0.116}$). Of these, _flight attendant_ ($72\%$) and _clerk_
($86\%$) are female-dominated professions (Current Population Survey, 2018),
while _bartender_ ($55\%$) is a profession that has been shifting from male to
female-dominated in recent times (Haines et al., 2016).

Finally, we study differences between corresponding pairs of female and male
nouns, using predictions from M.3 (BERT). Out of the $20$ pairs in our
dataset, the pair with the greatest difference in mean positive class
probability is _spinster_ and _bachelor_ , with $\mathbf{spinster-
bachelor=-0.404}$ ($p<0.01$). This reflects societal perceptions of _spinster_
as someone who is characterized as alone, lonely and resembling an “old maid”,
versus _bachelor_ as someone who might be young, carefree and fun-loving
(Nieuwets, 2015). This is an example of _semantic pejoration_ seen in society,
where the female form of the noun (i.e., _spinster_) gradually acquires a
negative connotation. Notably, this pejorative behavior may have also leaked
into the model, reflecting societal gender stereotypes.

##  5 Conclusion/Future Work

In this paper, we introduce a new dataset that can be used to test the
presence of occupational gender stereotypes in _any_ sentiment analysis model.
We then train $3$ sentiment analysis models and evaluate them using our
dataset. Following that, we analyze our results, exploring social stereotypes
of occupations as well as gendered stereotypes. We find that all $3$ models
that we study exhibit differences in mean predicted positive class probability
between genders, though the directions vary. We also see that simpler models
may be more susceptible to biases seen in the training dataset, while deep
contextual models may exhibit biases potentially introduced during
pretraining.

One promising avenue for future work is to explore occupational stereotypes in
deep contextual models by analyzing their training corpora. This could also
help identify techniques to mitigate biases in such models, since they could
be relatively impervious to biases introduced by fine-tuning (especially on
smaller datasets).

From a sociological perspective, we plan to investigate occupational gender
stereotypes in downstream applications such as automated resume screening.
Such a task assumes greater importance with the increased use of these systems
in today‘s world. There is prior work on ethnic bias in such tools (Derous and
Ryan, 2018), and we believe that there is significant value in exploring and
characterizing gender biases in these systems.

## Acknowledgements

We thank Johan Ugander for helping motivate the initial phases of this work.
We also thank the anonymous reviewers for their thoughtful feedback and
suggestions.

## References

  * Basta et al. (2019) Christine Basta, Marta R. Costa-jussà, and Noe Casas. 2019.  Evaluating the Underlying Gender Bias in Contextualized Word Embeddings.  _arXiv e-prints_.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016\.  Man is to Computer Programmer As Woman is to Homemaker? Debiasing Word Embeddings.  In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , NIPS’16, pages 4356–4364, USA. Curran Associates Inc.
  * Bonferroni (1936) Carlo Bonferroni. 1936.  Teoria statistica delle classi e calcolo delle probabilita.  _Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commericiali di Firenze_ , 8:3–62.
  * Caliskan et al. (2017) Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017.  Semantics derived automatically from language corpora contain human-like biases.  _Science_ , 356(6334):183–186.
  * Cejka and Eagly (1999) Mary Ann Cejka and Alice H. Eagly. 1999.  Gender-Stereotypic Images of Occupations Correspond to the Sex Segregation of Employment.  _Personality and Social Psychology Bulletin_ , 25(4):413–423.
  * Chollet et al. (2015) François Chollet et al. 2015.  Keras.  https://keras.io.
  * Costa and Veloso (2015) Andre Costa and Adriano Veloso. 2015.  Employee analytics through sentiment analysis.  In _Brazilian Symposium on Databases_.
  * Current Population Survey (2018) Current Population Survey. 2018.  39\. Median weekly earnings of full-time wage and salary workers by detailed occupation and sex.  Bureau of Labor Statistics, United States Department of Labor.
  * Derous and Ryan (2018) Eva Derous and Ann Marie Ryan. 2018.  When your resume is (not) turning you down: Modelling ethnic bias in resume screening.  _Human Resource Management Journal_ , 29(2):113–130.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language understanding.  _CoRR_ , abs/1810.04805.
  * Eagly and Steffen (1984) Alice H Eagly and Valerie J Steffen. 1984.  Gender stereotypes stem from the distribution of women and men into social roles.  _Journal of personality and social psychology_ , 46(4):735.
  * Garg et al. (2018) Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018.  Word embeddings quantify 100 years of gender and ethnic stereotypes.  _Proceedings of the National Academy of Sciences_ , 115(16):E3635–E3644.
  * Glick et al. (1995) Peter Glick, Korin Wilk, and Michele Perreault. 1995.  Images of occupations: Components of gender and status in occupational stereotypes.  _Sex Roles_ , 32(9):565–582.
  * Gonen and Goldberg (2019) Hila Gonen and Yoav Goldberg. 2019.  Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them.  _arXiv e-prints_.
  * Haines et al. (2016) Elizabeth L. Haines, Kay Deaux, and Nicole Lofaro. 2016.  The Times They Are a-Changing… or Are They Not? A Comparison of Gender Stereotypes, 1983-2014.  _Psychology of Women Quarterly_ , 40(3):353–363.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Comput._ , 9(8):1735–1780.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Fine-tuned Language Models for Text Classification.  _CoRR_ , abs/1801.06146.
  * Kay et al. (2015) Matthew Kay, Cynthia Matuszek, and Sean A. Munson. 2015.  Unequal representation and gender stereotypes in image search results for occupations.  In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ , CHI ’15, pages 3819–3828, New York, NY, USA. ACM.
  * Kiritchenko and Mohammad (2018) Svetlana Kiritchenko and Saif M. Mohammad. 2018.  Examining gender and race bias in two hundred sentiment analysis systems.  _CoRR_ , abs/1805.04508.
  * May et al. (2019) Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019\.  On measuring social biases in sentence encoders.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed Representations of Words and Phrases and their Compositionality.  In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 26_ , pages 3111–3119. Curran Associates, Inc.
  * Mohammad et al. (2018) Saif M. Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018.  Semeval-2018 Task 1: Affect in Tweets.  In _Proceedings of International Workshop on Semantic Evaluation (SemEval-2018)_ , New Orleans, LA, USA.
  * Nieuwets (2015) Astrid Nieuwets. 2015.  _Fallen Females: On the Semantic Pejoration of Mistress and Spinster_.  Bachelor’s thesis, Utrecht University.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global Vectors for Word Representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  _CoRR_ , abs/1802.05365.
  * Radford (2018) Alec Radford. 2018.  Improving Language Understanding by Generative Pre-Training.
  * Rudinger et al. (2018) Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018.  Gender bias in coreference resolution.  _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_.
  * Rudman and Phelan (2008) Laurie A. Rudman and Julie E. Phelan. 2008.  Backlash effects for disconfirming gender stereotypes in organizations.  _Research in Organizational Behavior_ , 28:61 – 79.
  * Shinar (1975) Eva H Shinar. 1975.  Sexual stereotypes of occupations.  _Journal of Vocational Behavior_ , 7(1):99 – 111.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1631–1642.
  * Vervecken et al. (2013) Dries Vervecken, Bettina Hannover, and Ilka Wolter. 2013.  Changing (S)expectations: How gender fair job descriptions impact children’s perceptions and interest regarding traditionally male occupations.  _Journal of Vocational Behavior_ , 82(3):208 – 220.
  * Webster et al. (2018) Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018.  Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns.  _Transactions of the Association for Computational Linguistics_ , 6:605–617.
  * Zhao et al. (2019) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019.  Gender Bias in Contextualized Word Embeddings.  _CoRR_ , abs/1904.03310.
  * Zhao et al. (2018) Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018.  Learning Gender-Neutral Word Embeddings.  _CoRR_ , abs/1809.01496.


[Lin and Michailidis, 2017]
Lin J, Michailidis G (2017).
\newblock \enquote{Regularized estimation and testing for high-dimensional
multi-block vector-autoregressive models.}
\newblock \emph{Journal of machine learning research}, \textbf{18}.

[Matteson and James, 2014]
Matteson DS, James NA (2014).
\newblock \enquote{A nonparametric approach for multiple change point analysis
of multivariate data.}
\newblock \emph{Journal of the American Statistical Association},
\textbf{109}(505), 334--345.

[Meier \emph{et~al.}, 2021]
Meier A, Cho H, Kirch C (2021).
\newblock \emph{\pkg{mosum}: Moving Sum Based Procedures for Changes in the
\newblock \proglang{R} package version 1.2.5,
\urlprefix\url{https://CRAN.R-project.org/package=mosum}.

[Michailidis and d’Alch{\'e} Buc, 2013]
Michailidis G, d’Alch{\'e} Buc F (2013).
\newblock \enquote{Autoregressive models for gene regulatory network inference:
Sparsity, stability and causality issues.}
\newblock \emph{Mathematical biosciences}, \textbf{246}(2), 326--334.

[Negahban \emph{et~al.}, 2012]
Negahban SN, Ravikumar P, Wainwright MJ, Yu B (2012).
\newblock \enquote{A unified framework for high-dimensional analysis of $ M
$-estimators with decomposable regularizers.}
\newblock \emph{Statistical science}, \textbf{27}(4), 538--557.

[Nicholson \emph{et~al.}, 2020]
Nicholson WB, Wilms I, Bien J, Matteson DS (2020).
\newblock \enquote{High dimensional forecasting via interpretable vector
\newblock \emph{Journal of Machine Learning Research}, \textbf{21}(166), 1--52.

[Primiceri, 2005]
Primiceri GE (2005).
\newblock \enquote{Time varying structural vector autoregressions and monetary
\newblock \emph{The Review of Economic Studies}, \textbf{72}(3), 821--852.

[Rigaill \emph{et~al.}, 2019]
Rigaill G, Hocking T, Maidstone R, Koskas M, Fearnhead P (2019).
\newblock \emph{\pkg{fpop}: Segmentation using Optimal Partitioning and
Function Pruning}.
\newblock \proglang{R} package version 2019.08.26,
\urlprefix\url{https://CRAN.R-project.org/package=fpop}.

[Roy \emph{et~al.}, 2017]
Roy S, Atchad{\'e} Y, Michailidis G (2017).
\newblock \enquote{Change point estimation in high dimensional Markov
random-field models.}
\newblock \emph{Journal of the Royal Statistical Society: Series B (Statistical
Methodology)}, \textbf{79}(4), 1187--1206.

[Safikhani and Shojaie, 2020]
Safikhani A, Shojaie A (2020).
\newblock \enquote{Joint Structural Break Detection and Parameter Estimation in
High-Dimensional Nonstationary VAR Models.}
\newblock \emph{Journal of the American Statistical Association}, pp. 1--14.

[Schwarz \emph{et~al.}, 1978]
Schwarz G, \emph{et~al.} (1978).
\newblock \enquote{Estimating the dimension of a model.}
\newblock \emph{The annals of statistics}, \textbf{6}(2), 461--464.

[Stock and Watson, 2016]
Stock JH, Watson MW (2016).
\newblock \enquote{Dynamic factor models, factor-augmented vector
autoregressions, and structural vector autoregressions in macroeconomics.}
\newblock In \emph{Handbook of macroeconomics}, volume~2, pp. 415--525.

[Trujillo \emph{et~al.}, 2017]
Trujillo LT, Stanfield CT, Vela RD (2017).
\newblock \enquote{The effect of electro-encephalogram (EEG) reference choice
on information-theoretic measures of the complexity and integration of EEG
\newblock \emph{Frontiers in neuroscience}, \textbf{11}, 425.

[Truong \emph{et~al.}, 2020]
Truong C, Oudre L, Vayatis N (2020).
\newblock \enquote{Selective review of offline change point detection methods.}
\newblock \emph{Signal Processing}, \textbf{167}, 107299.

[Velu \emph{et~al.}, 1986]
Velu RP, Reinsel GC, Wichern DW (1986).
\newblock \enquote{Reduced rank models for multiple time series.}
\newblock \emph{Biometrika}, \textbf{73}(1), 105--118.

[Wang \emph{et~al.}, 2019]
Wang D, Yu Y, Rinaldo A, Willett R (2019).
\newblock \enquote{Localizing changes in high-dimensional vector autoregressive
\newblock \emph{arXiv preprint arXiv:1909.06359}.

[Zeileis \emph{et~al.}, 2002]
Zeileis A, Leisch F, Hornik K, Kleiber C (2002).
\newblock \emph{\pkg{strucchange}: An \proglang{R} Package for Testing for
Structural Change in Linear Regression Models}.
\newblock \proglang{R}~package version~1.5-2,
\urlprefix\url{https://CRAN.R-project.org/package=strucchange}.

[Zhu \emph{et~al.}, 2015]
Zhu X, Xie Y, Li J, Wu D (2015).
\newblock \enquote{Change point detection for subprime crisis in American
banking: From the perspective of risk dependence.}
\newblock \emph{International Review of Economics \& Finance}, \textbf{38},

\end{thebibliography}

\end{document}


# Squeezed State as a “Quasi-Neutral” One-Component Plasma

Y. Bliokh, J.G. Leopold, and Ya.E. Krasik  Physics Department, Technion,
Israel Institute of Technology, Haifa 320003, Israel

###### Abstract

We present a one-dimensional model which gives a novel physical interpretation
of the specific equilibrium state of the magnetized electron cloud in the
Malmberg-Penning trap. In the model, the strong magnetic field prevents radial
expansion of the electrons, while the electrostatic potential well restricts
their axial motion. A force-free steady-state is formed when the space-charge
field of the electrons accumulated in the potential well, compensates the
external electrostatic field. This state is known in another context as a
squeezed state. It is shown that the equilibrium electron density distribution
is correlated to the potential well’s shape. Small perturbations of the
steady-state survive over a long period of time in the form of specific
propagating waves.

## Introduction

Initially, the term squeezed state of electrons was introduced in Ignatov-1994
, in a system studied numerically where an over-critical magnetized electron
beam is injected into a conducting tube of section of increasing radii. It was
shown that under certain conditions, a virtual cathode can be tailored to
appear in the section with larger cross-section which then moves towards the
electron source, placed in the section of smaller radius, leaving behind it a
dense cloud of low-energy electrons in a specific state, which the authors
named “squeezed beam”. This is a state characterized by strong contraction of
the electron phase space associated with a longitudinal degree of freedom.

A similar phenomenon, that is, strong contraction of the phase space was
noticed in Bettega-2007 , where the electron injection in the Malmberg-Penning
trap was studied numerically and experimentally. This contracted state has
similar characteristics to the squeezed beam state – low electron energies and
equal or almost equal forward and backward electron fluxes. Because of this,
the term “squeezed state” looks more adequate than “squeezed beam” and will be
used below.

The squeezed state attracted recently particular interest in connection with
its possible application in microwave electronics as a source of electrons
instead of traditional cathodes Shamiloglu ; Fuks ; Siman-Tov .

In this paper, we propose a new theory of a squeezed state formation, which
allows the determination of such parameters as longitudinal distribution of
the electron density, accumulated charge, etc. To be specific, we study the
evolution of electrons continuously emitted in the Malmberg-Penning trap
because this situation allows a rather simple and illustrative interpretation.

##  I The squeezed state

Let us consider the Malmerg-Penning trap formed by a conducting cylinder of
radius $R$ and two electrodes placed on the axis and separated by a distance
$L\gg R$. The electrodes have negative potentials (not necessarily equal)
relative to the cylinder potential. Electrons are emitted from one electrode
and propagate in a longitudinal magnetic field, sufficiently strong to prevent
radial expansion. The second electrode’s negative potential reflects the
electrons completely or partially. Thus, the emitted electrons are trapped
both in the radial and longitudinal directions and form a negatively charged
non-expanding column between electrodes. Electron motion in the transverse
plane is described by azimuthal rotation in the crossed axial magnetic and
radial electric fields of the electron space charge. Longitudinal motion is
defined by oscillation in the potential well, $\varphi(z)$, between end
electrodes.

Let the radius $r_{d}$ of the electron cloud be small, $r_{d}\ll R$, and
assume that the electron density is homogeneous in the transverse plane
Driscoll ; Dubin-1999 . In this case, the radial dependence of the
longitudinal component of electric field, $E_{z}$, which governs axial motion,
can be neglected, and the problem of the electron longitudinal motion can be
reduced to be one-dimensional.

Apart from the external field, the electron motion is affected by the self-
consistent Coulomb electric field of the space charge. The presence of the
conducting walls strongly modifies the force between interacting electrons. In
contrast to free space, this force decreases much faster than $\propto d^{-2}$
with the distance $d$ between electrons. Radial homogeneity of the problem
allows one to consider the electron cloud as an ensemble of discs of radius
$r_{d}\ll R$ with homogeneous surface charge density $\sigma$. The condition
$r_{d}\ll R$ guaranties that the force of the Coulomb interaction is
distributed almost homogeneously across the disc surface so that the
distributed force can be replaced by its integrated value and the discs can be
considered as rigid. Azimuthal rotation in the crossed radial electric and
axial magnetic fields does not violate this assumption.

Let us replace a $dz$-thick cylindrical slice of the electron column at the
distance $z_{0}$ from the emitting electrode by a charged disc with surface
charge density $\sigma_{0}=en(z_{0})dz$, where $n(z_{0})$ is the electron
density. The longitudinal electric field $E_{z}$ produced by this disc at some
point $(z,r)$, can be written as follows Smythe :

$E_{z}(z,r,\phi)=\frac{4\pi\sigma_{0}r_{d}}{R}\,\sum_{n}e^{-\nu_{n}|z-z_{0}|/R}{\rm
sign}(z-z_{0})\frac{J_{0}(\nu_{n}r/R)J_{1}(\nu_{n}r_{d}/R)}{\nu_{n}J_{1}^{2}(\nu_{n})},$
(1)

where $J_{0,1}$ are Bessel functions of zero and first order respectively, and
$\nu_{n}$ are the zeros of $J_{0}$ function, $J_{0}(\nu_{n})=0$. One can
verify that the dependence of the field $E_{z}$ on radius $r$ is very weak in
the region $r\leq r_{d}\ll R$ near the axis, so that it is possible to use
averaged over the disc surface value of the field Williams ; Petersen :

$\displaystyle E_{z}(z,r)\Rightarrow\bar{E}_{z}(z)=\frac{2\pi}{\pi
r_{d}^{2}}\int rdrdE_{z}(z,r)$
$\displaystyle\bar{E}_{z}(z)=8\pi\sigma_{0}\,\sum_{n}e^{-\nu_{n}|z-z_{0}|/R}{\rm
sign}(z-z_{0})\frac{J_{1}^{2}(\nu_{n}r_{d}/R)}{\nu_{n}^{2}J_{1}^{2}(\nu_{n})}\equiv
8\pi\sigma_{0}f(|z-z_{0}|){\rm sign}(z-z_{0}).$ (2)

One can write the longitudinal equation of motion for an electron in the
presence of the external field $E_{\rm ext}(z)=-\partial\varphi(z)/\partial z$
and the electric field of the space charge as:

$\frac{d^{2}z}{dt^{2}}=\frac{e}{m}\left[-\frac{\partial\varphi(z)}{\partial
z}+8\pi e\int_{0}^{L}dz_{0}n(z_{0})f(|z-z_{0}|){\rm sign}(z-z_{0})\right]$ (3)

Note that here $e>0$ is the elementary charge (though we shall refer below to
electrons).

A typical dependence $f(x)$ on $x$, where $x=|z-z_{0}|/R$, is shown in Fig. 1.

Figure 1:  Function $f(x)$, $x=|z-z_{0}|/R$. Grey line – calculated using Eq.
(1); dots – approximation by one exponent, $f(x)=a\exp(-\lambda x)$; asterisks
– approximation by two exponents,
$f(x)=a_{1}\exp(-\lambda_{1}x)+a_{2}\exp(-\lambda_{2}x)$.

Thus, the electric field of the space charge decreases almost exponentially
with the distance from charged slice. Because the effect being studied does
not depend on the fine structure of the force, we will use below the
exponential approximation:

$f(x)=a\exp(-\lambda x).$ (4)

For example, for $r_{d}/R=0.1$, $a\simeq 0.25$ and $\lambda\simeq 13$.

Introducing the dimensionless variables

$\xi=z/R,\,\tau=\frac{t}{R}\sqrt{\frac{2e\varphi_{m}}{m}},$

where $\varphi_{m}$ is the potential well depth, the equation of motion Eq.
(3) takes the form:

$\frac{d^{2}\xi}{d\tau^{2}}=-\frac{1}{2}\frac{d\Phi}{d\xi}+4\frac{R^{2}}{r_{d}^{2}}\int_{0}^{\mathcal{L}}\,d\xi_{0}\,\eta(\xi_{0})f(|\xi-\xi_{0}|){\rm
sign}(\xi-\xi_{0}).$ (5)

Here $\Phi=\varphi/\varphi_{m}$ and $\eta=(\pi er_{d}^{2}/\varphi_{m})n$ are
the normalized potential and electron density, respectively, and
$\mathcal{L}=L/R$.

If the characteristic spatial scale $\lambda^{-1}$ of the Coulomb interaction
is small compared to the characteristic scale of the density variation (which
is the same as the spatial scale of the potential $\Phi$ variation), and point
$\xi$ is placed far from the well edges, the integral in Eq. (5) can be
evaluated as follows:

$\int_{0}^{\mathcal{L}}d\xi_{0}\,\eta(\xi_{0})f(|\xi-\xi_{0}|){\rm
sign}(\xi-\xi_{0})\simeq-2a\frac{d\eta(\xi)}{d\xi}\int_{0}^{\infty}\,dx\,x\exp(-\lambda
x)=-2a\frac{d\eta(\xi)}{d\xi}\frac{1}{\lambda^{2}}.$ (6)

Now Eq. (5) can be written as

$\frac{d^{2}\xi}{d\tau^{2}}=-\frac{1}{2}\frac{d\Phi}{d\xi}-8\frac{R^{2}}{r_{d}^{2}}\frac{a}{\lambda^{2}}\frac{d\eta}{d\xi}.$
(7)

Equation (7) shows that there is a steady-state distribution of electron
density $\eta_{\rm sq}(\xi)$ whose space-charge electric field compensates the
external field completely:

$\eta_{\rm
sq}(\xi)=\eta_{0}-\frac{1}{16}\frac{r_{d}^{2}}{R^{2}}\frac{\lambda^{2}}{a}\Phi(\xi).$
(8)

Let $\Phi(0)=0$. Then $\eta_{0}=\eta_{\rm sq}(0)$. Relation (7) is not valid
near the well edges and the integral at $\xi=0$ can be evaluated in a
different way:

$\left.\int_{0}^{\mathcal{L}}d\xi_{0}\,\eta(\xi_{0})f(|\xi-\xi_{0}|){\rm
sign}(\xi-\xi_{0})\right|_{\xi=0}\simeq-a\eta_{0}\int_{0}^{\infty}\,d\xi_{0}\,\exp(-\lambda\xi_{0})=-a\eta_{0}\frac{1}{\lambda}.$
(9)

Thus, at steady-state

$-\frac{1}{2}\left.\frac{d\Phi}{d\xi}\right|_{\xi=0}-4\frac{R^{2}}{r_{d}^{2}}\frac{a}{\lambda}\eta_{0}=0$
(10)

and

$\eta_{\rm
sq}(\xi)=\frac{1}{16}\frac{r_{d}^{2}}{R^{2}}\frac{\lambda^{2}}{a}\left[2\frac{\lambda_{\Phi}}{\lambda}-\Phi(\xi)\right],$
(11)

where $\lambda_{\Phi}=d\Phi/d\xi|_{\xi=0}\ll\lambda$. Expression (11)
describes the density distribution of the squeezed state of the electron cloud
in a longitudinal potential well.

The squeezed state develops in the following way. Let the electrons be emitted
from the left boundary with some initial velocity into an external potential
well with an absorbing right boundary as it is shown in Fig. 2. A single
electron can either reach the absorbing boundary if its energy exceeds the
maximal value of the potential or be reflected back and then disappear at the
emitting boundary. Note that, the time-dependent longitudinal distribution of
the electrons produces a time varying space-charge electric field, so that the
electron’s total energy is not conserved. If many electrons are simultaneously
present in the well, the self-consistent space-charge field can modify their
motion significantly. Part of the electrons increases their energy and
overcome the potential barrier (if it exists). There are also electrons whose
energy decreases and these become trapped in the well even if their initial
energy exceeds the barrier height. Accelerated electrons leave the system,
while the decelerated particles are accumulated in the well. The accumulated
electrons decrease the depth of the effective potential well, $\varphi_{\rm
eff}$, which includes the contribution of the space charge. Finally, the well
becomes completely filled with low-energy electrons, so that the effective
potential $\varphi_{\rm eff}\simeq{\rm const}$ and $E_{\rm eff}\simeq 0$, a
situation which defines a squeezed state.

Figure 2:  Model geometry of the problem. Two electrodes are placed inside a
conducting tube. The potentials of the electrodes are negative with respect to
the tube and have slightly different values. The potential distribution along
the axis is represented by the red line.

##  II Numerical simulations

The equation of motion Eq. (5) in its discrete form was solved numerically.
The energy of the injected electrons (macro particles) is chosen to be small
relative to the maximum potential well energy. The number of emitted electrons
per unit time is kept constant that corresponds to the saturated thermionic
emission. Note that soon after the beginning of emission, part of the
electrons returns immediately after injection and is absorbed, so that
actually the emitted current is space-charge limited. The potential of the
right boundary of the well can be chosen to be either slightly less or more
than the initial energy of the injected electrons.

At early times [Fig. 3(a)], electron trajectories in phase space,
$(\xi,u=d\xi/d\tau)$, are very near the single electron regular closed
trajectory. At the absorbing boundary, $\xi=6$, electrons are either absorbed
or return toward the emission point. This behavior keeps going for quite some
time. Later the trajectory curve’s interior is filled with decreasing energy
electrons [Fig. 3(b)].

Figure 3:  Phase space at different normalized times (shown at the left top
corner of each frame) from the beginning of the particles injection.
Trajectories during a time interval equal to the period of a single particle
round trip are depicted. Initially, trajectories are close to the single
particle trajectory (a) and fill in gradually the interior (b). Later,
occupied phase space shrinks along $u$-axis [(c), (d), note the scale change].
This stage is characterized by vortex-type structures which move along the
$\xi$-axis, merge, and split.

The number of electrons whose energy is large enough in order to escape from
the well decreases gradually, the velocity spread $\delta u$ decreases [Fig.
3(c)] until the phase space area diminishes close to the $u=0$ line [Fig.
3(d)]. The electron density distribution in this squeezed state is depicted in
Fig. 4a, looks like the mirror reflection of the external potential, as it is
predicted by Eq. (11) in Section II.

Figure 4: (a) Potential well (red) and density distribution in the squeezed
state (blue). (b) External (blue) and resultant force in the well (red).
Dimensionless time $\tau=20000$.

The resulting force distribution along the well at some point in time moment
is shown in Fig. 4b. Spatio-temporal evolution of the electron density and
electric field are shown in Fig. 5.

Figure 5:  Spatial-temporal evolution of the electron density (left) and
resulting electric field (right) in developed squeezed state. The
dimensionless time is counted from $\tau=20000$, corresponding phase space is
shown in Fig. 3d.

Notice that small fluctuations of the electron density and electric field
remain in the system over a long period of time (see also Section III).

##  III The squeezed state as a one-component plasma

The results of the numerical simulation of Section II confirm our assertion
that the trapped electrons, filling the potential well, screen almost
completely the external electric field. On the other hand, one can consider
this state as the result of electron space charge compensation by the external
electric field. This is exactly the situation occurring in a two-component
quasi-neutral plasma: electrons and ions compensate mutually their space
charges and form a force-free configuration with equal densities. Thus, a
squeezed state may be considered as a charged, non-neutral plasma.

The electron cloud in the Malmberg-Penning trap was also treated as a “plasma
with a single sign of charge” in Refs. ONeil-1988 ; ONeil-1999 ; Dubin-1999 ,
but the authors applied this analogy considering radial equilibrium of the
cloud. In this framework, the axial magnetic field plays the role of ion
background compensating the electrons space-charge. Instead, we use this
terminology – charged plasma – keeping in mind axial equilibrium of the
electron cloud, while that magnetic field provides its radial confinement.
Note that some similarity between the squeezed state and plasma was mentioned
in Refs. Dubinov-2016 ; Dubinov-2020 .

The interpretation of the squeezed state as a quasi-neutral plasma allows one
to suppose that the observed spatio-temporal fluctuations, which are visible
in Fig. 5, are nothing but a specular kind of Langmuir waves. Indeed, one can
distinguish wave patterns in the density and field distributions in Fig. 5.

As seen in Fig. 3, the relaxation to the squeezed state is very slow. For
example, relaxation from the state depicted in Fig. 3(a), to the state shown
in Fig. 3(d), occurs during several thousands of round trip periods. However,
the relation Eq. (11) between the potential and the density distributions
allows one to prepare an initial macro particles distribution very close to
steady-state one. This initially prepared state relaxes fast to steady-state
with a very low noise level, as shown in Fig. 6. This allows the study of the
effect of various perturbations in much longer systems at a small cost in
computation time.

Figure 6:  Phase space in the squeezed state. Compare with Fig. 3 c,d

For example, let us consider that the steady state is perturbed by additional
particles which appear suddenly in a small region near the center of the
potential well depicted in the top panel of Fig. 7. The uncompensated space-
charge field, produced by these excess particles, removes particles from the
well through the absorbing edges and the total number of particles in the
squeezed state relaxes rapidly to its equilibrium value. However, the field
and density perturbations remain in the system in the form of propagating
waves clearly visible in the bottom panel of Fig. 7.

Figure 7:  (a) – External potential well. (b) – Electric field evolution. Note
that the wave phase velocity is constant in the central region, where
$\Phi\sim{\rm const}$, and decreases near the system edges, where the
equilibrium density is minimal. Dashed line corresponds to $c_{\rm
sq}=1/\sqrt{2}$.

The evolution of small perturbations of the dimensionless steady-state density
$\tilde{\eta}$ and velocity $\tilde{u}$ is described by the equation of motion
and the continuity equation:

$\displaystyle\frac{\partial\tilde{u}}{\partial\tau}=-8\frac{R^{2}}{r_{d}^{2}}\frac{a}{\lambda^{2}}\frac{\partial\tilde{\eta}}{\partial\xi},$
$\displaystyle\frac{\partial\tilde{\eta}}{\partial\tau}+\eta_{\rm
sq}(\xi)\frac{\partial\tilde{u}}{\partial\xi}=0,$ (12)

where $\eta_{\rm sq}(z)$ is the equilibrium density distribution, defined by
Eq. (11). In the short-wavelength approximation, Eqs.(12) describe waves whose
phase velocity $c_{\rm sq}$ depends on the local unperturbed density.:

$c_{\rm sq}^{2}=8\frac{R^{2}}{r_{d}^{2}}\frac{a}{\lambda^{2}}\eta_{\rm
sq}(\xi).$ (13)

In the central part of the system, where the influence of the boundaries is
negligibly small, so that $\Phi(\xi)\simeq-1$, the phase velocity is constant
and it follows from Eq. (11) that:

$c_{\rm sq}^{2}\simeq 1/2,$ (14)

which is confirmed by the results of numerical simulations presented in Fig.
7.

##  IV Conclusions

A squeezed state can develop when electrons are continuously emitted at one or
both edges of a longitudinal potential well. The properties of this state of
an electron cloud is similar to that of a one-dimensional quasi-neutral plasma
where the external electrostatic potential substitutes the role of the ions.
Quasi-neutrality of an ordinary, two-component plasma, is replaced by
equilibrium between the confining external potential field and the repulsive
Coulomb force of the space charge. It is important to note that our
interpretation of the equilibrium condition allows one to determine the
maximum number of charged particles which can be accumulated in a given
potential trap.

## Acknowledgement

The research at the Technion was supported by Technion Defense Grant No.
2029541 and ONR Grant No. N00014-19-1-2155.

## References

  * (1) A.M. Ignatov and V.P. Tarakanov, Phys. Plasmas 1, 741 (1994).
  * (2) G. Bettega, F. Cavaliere, M. Cavenago et. all., Phys. Plasmas 14 , 042104 (2007).
  * (3) M.I. Fuks, S. Prasad, and E. Schamiloglu, IEEE Trans. Plasma Sci. 44, 1298 (2016).
  * (4) M.I. Fuks and E. Schamiloglu, Phys. Rev. Lett. 122, 224801 (2019).
  * (5) M. Siman-Tov, J.G. Leopold, Y.P. Bliokh, and Ya.E. Krasik, Phys. Plasmas 27, 083103 (2020).
  * (6) C.F. Driscoll, J.H. Malmberg, and K.S. Fine, Phys.Rev.Lett. 60, 1290 (1988).
  * (7) D.H.E. Dubin and T.M. O’Neil, Rev. Mod. Phys. 71, 87 (1999).
  * (8) W.R. Smythe, Static and Dynanic Electricity, (1950).
  * (9) C.B. Williams and M.H. MacGregor, IEEE Trans. Nuc. Sci. NS-14, 581 (1967).
  * (10) G.W. Petersen and W.J. Gallagher, IEEE Trans. Nuc. Sci. NS-16, 214 (1969).
  * (11) T.M. O’Neil, AIP Conference Proceedings 175, 1 (1988).
  * (12) T.M. O’Neil, Phys. Today 52, 24 (1999).
  * (13) A.E. Dubinov, A.G. Petrik, S.A. Kurkin, N.S. Frolov, A.A. Koronovskii, and A.E. Hramov, Phys. Plasmas 23, 042105 (2016).
  * (14) A.E. Dubinov, V.D. Selimir, and V.P. Tarakanov, Plasma Phys. Rep. 46, 1108 (2020).


# Drug Interaction Vectors Neural Network: DrIVeNN

Modeling Polypharmacy Adverse Drug Events and a Case Study in Cardiovascular
Disease Treatment

Natalie Wang<EMAIL_ADDRESS>Johns Hopkins UniversityBaltimoreMDUSA and
Casey Overby Taylor<EMAIL_ADDRESS>Johns Hokins UniversityBaltimoreMDUSA

###### Abstract.

Polypharmacy, the concurrent use of multiple drugs to treat a single
condition, is common in patients managing multiple or complex conditions.
However, as more drugs are added to the treatment plan, the risk of adverse
drug events (ADEs) rises rapidly. Many serious ADEs associated with
polypharmacy only become known after the drugs are already in use. It is
impractical to test every possible drug combination during clinical trials.
This issue is particularly prevalent among older adults with cardiovascular
disease (CVD) where polypharmacy and ADEs are commonly observed. In this
research, our primary objective was to identify key drug features and build
and evaluate a model for modeling polypharmacy ADEs. Our secondary objective
was to assess our model on a domain-specific case study.

We developed a two-layer neural network that incorporated drug features such
as molecular structure, drug-protein interactions, and mono drug side effects
(DrIVeNN). We assessed DrIVeNN using publicly available side effect databases
and determined Principal Component Analysis (PCA) with a variance threshold of
0.95 as the most effective feature selection method. DrIVeNN performed
moderately better than state-of-the-art models like RESCAL, DEDICOM, DeepWalk,
Decagon, DeepDDI, KGDDI, and KGNN in terms of AUROC for the drug-drug
interaction prediction task.

We also conducted a domain-specific case study centered on the treatment of
cardiovascular disease (CVD). When the best performing model architecture was
applied to the CVD treatment cohort, there was a significant increase in
performance from the general model. We observed an average AUROC for CVD drug
pair prediction increasing from 0.826 (general model) to 0.975 (CVD specific
model). Our findings indicate the strong potential of domain-specific models
for improving the accuracy of drug-drug interaction predictions. In
conclusion, this research contributes to the advancement of predictive
modeling techniques for polypharmacy ADEs.

polypharmacy prediction, neural networks, adverse drug events

††copyright: none††ccs: Computing methodologies Neural networks††ccs:
Computing methodologies Neural networks††ccs: Applied computing Bioinformatics

##  1\. Introduction

Polypharmacy, characterized by the concurrent use of multiple drugs to treat a
single condition, is a common in patients managing complex and terminal
diseases. However, as more drugs are added to the treatment plan, the risk of
adverse drug events (ADEs) increases rapidly.(Tatonetti et al., 2012) These
ADEs can have severe consequences on patient health and well-being.
Additionally, the prevalence of polypharmacy has been steadily increasing over
the years, reaching a rate of 37% in 2022 compared to 8.2% in 1999-2000 and
15% in 2011-2012.(Kantor et al., 2015)(Delara et al., 2022) In clinical
studies, it is infeasible to test every possible combination of drugs for
ADEs, so serious polypharmacy-related ADEs may only become known once the
drugs already in use, leading to a substantial healthcare burden. In the
United States alone, these ADEs contribute to an estimated $62 billion and
nearly 150,000 premature deaths.(Garber and Brownlee, 2019) As the number of
available drugs continues to escalate, traditional experimental methods for
assessing drug-drug interactions (DDIs) struggle to keep pace, necessitating
innovative computational approaches to accelerate our understanding.

We sought to build a novel deep learning model for DDI prediction. Numerous
studies show the potential of deep learning models for the DDI prediction
task. These include RESCAL, DEDICOM, DeepWalk, Decagon, DeepDDI, KGDDI, and
KGNN(Nickel et al., 2011; Harshman et al., 1982; Perozzi et al., 2014; Zitnik
et al., 2018; JY et al., 2018; Karim et al., 2019; Lin et al., 2020). Through
a review of the literature, we extracted some key features and takeaways from
each model, which served as a basis of comparison for our model.

Prior models use deep learning as a tool to discover hidden interactions
between drug features in many different ways. For example, RESCAL uses a
latent factorization method to represent drugs and interactions as low-
dimensional vectors. Another model, DEDICOM, uses tensor decomposition to
uncover latent features in drug-drug interaction data. Additionally, DeepWalk
uses a graph embedding technique to capture structural information of drug
interaction networks.

Other notable models include Decagon, a tensor factorization decoder, and
DeepDDI which incorporated domain knowledge via drug chemical structures and
protein sequence information. KGDDI and KGNN both exploit knowledge graphs to
capture semantic relationships between drugs, proteins, and diseases. These
two models use graph neural networks to capture complex interactions. In our
research, we utilized these state-of-the-art models as baselines for
comparison with our proposed model.

Additionally, we built and evaluated a disease-specific model. We used
cardiovasular disease (CVD) as a case study for other conditions that often
require polypharmacy. Older adults with CVD are particularly prone to
polypharmacy. A retrospective chart review at a tertiary care center found
among older adults (65 years and above) with a history of CVD and admitted to
the cardiology service, the prevalence of polypharmacy was 95%, hyper-
polypharmacy 69%, and at least one severe potential drug-drug interaction
(DDI) 77.5%.(Sheikh-Taha and Asmar, 2021) To the best of our knowledge, a
domain-specific model has not yet been investigated for the drug-drug
interaction task.

Our overall approach expands on existing work by integrating diverse sources
of drug-related data and a new model structure. The Drug Interaction Vectors
Neural Network (DrIVeNN) model we build uses graph neural networks to learn
complex patterns from datasets consisting of known drug interactions, drug
targets, and molecular structures. We also drew inspiration from a study
conducted by Chen et al (2021), who developed a deep learning method called
MUFFIN that incorporates chemical structures, drug-target interactions, and
known drug side effects to predict DDIs.(Chen et al., 2021) Their results
demonstrate the effectiveness of multi-modal data integration for the DDI
prediction task and motivated us to include these same data types for our
model. Lastly, DrIVeNN incorporates an ensemble method for prediction,
inspired by the works of Masumashah et al (2021).(Masumshah et al., 2021)

##  2\. Objectives

  * •

Build and evaluate the DrIVeNN model with diverse drug feature vectors for the
drug-drug interaction prediction task.

  * •

Evaluate DrIVeNN for the DDI prediction task in the CVD domain, as a case
study for domain-specific DDI prediction tasks.

##  3\. Methods

###  3.1. Datasets

To address our first objective, we employed the datasets compiled and
preprocessed by Zitnik et al(Zitnik et al., 2018) which contained drug-drug
interactions, drug-protein interactions, and mono drug side effects. The
authors used the STITCH (Search Tool for InTeractions of CHemicals) database
as a primary source for drug-protein interactions. STITCH is a comprehensive
database of protein-chemical interaction data with original data sources for
each interaction.(Szklarczyk et al., 2016) To capture mono drug side effects,
Zitnik et al(Zitnik et al., 2018) used two distinct databases: SIDER (Side
Effect Resource) and OFFSIDES. SIDER contains drug-side effect associations
obtained from drug label text, while OFFSIDES contains drug-side effect
associations generated from adverse event reporting systems.(Kuhn et al.,
2016; Tatonetti et al., 2012) To obtain the structural attributes of drugs, we
used the SMILES (Simplified Molecular Input Line Entry System) representation
of each drug to construct a molecular graph and employed a graph neural
network to generate structural representations.

To address our second objective, we utilized the Inxight Drugs API to conduct
a comprehensive search for drugs prescribed for three major cardiovascular
diseases: myocardial infarction (MI), congestive heart failure (CHF), and
coronary artery disease (CAD).(Siramshetty et al., 2022) Through this
analysis, we found 30 distinct principal forms of drugs for MI, 22 for CHF,
and 18 for CAD. To integrate this data with our existing datasets, we gathered
Unique Ingredient Identifiers (UNIIs) and InCHIKeys for each cardiovascular
disease treatment drug. We chose these because they were the most frequently
occurring in the dataset. For drugs in our other drug datasets, which were
identified solely by PubChem IDs, we used a two-step process to obtain their
corresponding UNIIs. First, we downloaded UNII drug records from open.fda.gov
and matched records that contained both UNII and PubChem IDs, thus obtaining
UNIIs for the drugs in our dataset. However, approximately 300 drugs from our
original dataset still lacked UNII matches so we manually used the PubChem
Lookup tool to find to find their drug names and InCHIKeys, then the Global
Substance Registration System (GSRS) to search for their UNIIs based on their
drug names and InCHIKeys.(Kim et al., 2023; Peryea et al., 2021) There is an
overview of this process shown in Figure 1. The CVD treatment drugs we
identified that were also present in our DDI datasets were Enoxaparin Sodium,
Niacin, Aspirin, Carvediolol, Metoprolol, Nitroglycerin, Ramipril, Valsartan,
Amlodipine, Ticlopidine, Chlorothizide, Ethacrynic Acid, Indapamide,
Metolazone, Ramipril, and Spironolactone.

Figure 1. DrIVeNN Cardiovascular Disease Data Pipeline.

Figure 1 summarizes the CVD data pipeline. To gather a list of CVD treatment
drugs, we performed the following steps: 1. Used the Inxight Drugs API to find
drugs indicated for MI, CHF, and CAD, 2. Removed duplicates and non-principal
forms of the drugs, and 3. Selected unique drugs within the three sets that
contained UNIIs, this became our CVD treatment drug dataset. The steps in the
second part of the pipeline were: 1. Matched PubChem IDs in DDI dataset with
UNIIs using UNII records from fda.gov, 2. Manually search unmatched drugs from
the DDI dataset for matching UNIIs using PubChem Lookup and GSRS, and 3.
Matched CVD drugs found in the first part of the pipeline and our DDI drugs
based on UNIIs then found all DDIs that included at least one CVD drug to
create our CVD DDI dataset.

###  3.2. Data-Driven Motivation for this Approach

During our exploratory data analysis, we made several observations that served
as our motivation for creating a domain-specific model.

First, we noticed a difference between the ten most common mono side effects
of all drugs and the ten most common mono side effects of drugs identified as
CVD treatment drugs. Only two side effects, ”emotional distress” and ”blood
creatinine increased” overlapped both groups. In the context of this study, a
mono side effect refers to an ADE that is known to be caused by a drug (as
opposed to a drug-drug interaction which is an ADE known to be caused by two
drugs). This observation suggests potential differences in the drug features
of CVD treatment drugs compared to other drugs, indicating the need for a
specialized model for this domain.

We also observed the median number of drug-drug interactions (DDIs) for
different drug sets. The median number of DDIs per all drug pairs was found to
be 53. In contrast, drug pairs that contained at least one CVD treatment drug
exhibited a higher median number of DDIs, 65 interactions, and the drug pairs
the consisted of two CVD drugs, reached a median of 124 interactions. This
notable increase in the likelihood of ADEs for CVD treatment drugs further
motivated us to explore this domain.

###  3.3. Data Processing

Each drug was associated with three distinct features: drug structure
features, drug-protein interaction features, and mono drug side effect
features. We created two drug feature datasets, one with drug structure
features and one without. To extract the drug structure features, we utilized
dgllife, a Python package designed for deep learning on graphs.(Li et al.,
2021) We used a message-passing graph neural network, MPNN, that was
pretrained for molecules to extract corresponding drug structure embeddings
from our drug dataset. For the drug-protein interaction features and mono drug
side effect features, we applied Principle Component Analysis (PCA) as a
feature extraction method(Pedregosa et al., 2011). We also applied
normalization techniques and UMAP (Uniform Manifold Approximation and
Projection) to address sparsity in the data(Pedregosa et al., 2011). To
capture the full representation of each drug, we concatenated the three types
of features into a single drug feature vector (Fig. 2). To represent a drug-
drug pair, we summed the individual drug features following protocols from
similar studies (Fig. 3a)(Masumshah et al., 2021).

Figure 2. Drug feature matrix creation for drugs $d_{1},d_{2},...,d_{n}$. This
summarizes the drug feature matrix creation process which includes: 1.
Applying MPNN, a graph neural network, to the drug structure dataset, 2.
Training and applying PCA(0.95) to the drug-protein dataset and mono drug side
effect dataset, and 3. Concatenating the three separate processed datasets
into the drug feature matrix where each row represents one drug.

Figure 2 summarizes the drug feature matrix creation process which includes:
1. Applying MPNN, a graph neural network, to the drug structure dataset, 2.
Training and applying PCA(0.95) to the drug-protein dataset and mono drug side
effect dataset, and 3. Concatenating the three separate processed datasets
into the drug feature matrix where each row represents one drug.

Figure 3. Drug Pair Prediction: 3a. Drug pair representation for drugs
$(d_{i},d_{j})$. Element-wise summation was performed on the individual drug
representations to create one representative vector. This process was done for
all drug pairs in the dataset. 3b. DDI prediction process for a single drug
pair and for a single side effect

This graphic shows the drug pair representation and prediction process.

To construct our classification dataset, we implemented negative sampling
following the established protocol from other studies.(Zitnik et al., 2018;
Trouillon et al., 2016; Mikolov et al., 2013) For each side effect, there are
corresponding drug-drug pairs that were identified as causative factors, these
are our positive edges. To generate negative edges, we randomly selected drug
pairs that were not associated with the given side effect. This approach
allowed us to create a balanced dataset that included both positive and
negative instances, helping with the classification task and ensuring more
complete coverage of potential drug-drug interactions.

###  3.4. Model Training

For every type of side effect, we partitioned the corresponding drug-drug
pairs into train, validation, and test sets using an 80/10/10 ratio,
respectively. This partitioning ensured the model was trained on a
sufficiently large dataset while also providing separate datasets for
validation and final evaluation. During the training phase, our objective was
to minimize the binary cross-entropy loss function. Each training run
consisted of 50 epochs, allowing the model to learn and adjust its parameters
over multiple iterations. We used a pretrained GNN for the data extraction
step, so our model training consists of learning weights for our predictive
feed-forward neural network.

###  3.5. Experiments

In our initial experiment, we examined different feature selection methods to
address sparsity observed in the dataset and improve computational efficiency.
We evaluated various levels of Principle Component Analysis (PCA) and a
combination of normalization and Uniform Manifold Approximation and Projection
(UMAP). PCA was chosen as a dimensionality reduction technique to retain the
most informative components of the drug features, while mitigating the effect
of sparse data on our analysis. We utilized normalization and UMAP as
complementary approaches. Because our drug feature vectors were derived from
multiple sources, we first applied normalization to ensure features were on
the same scale for fair comparisons. We chose UMAP due to its well-known
effectiveness at preserving both local and global structure within the data
while reducing its dimensionality.(McInnes et al., 2018) For this experiment,
we used the model architecture developed by Masumashah et al, a three-layer
neural network with 300, 200, and 100 nodes for each respective
layer(Masumshah et al., 2021). This architecture has demonstrated promising
performance in previous studies and served as a foundation for our
experiments. One key difference between that study and ours is the
incorporation of molecular drug data into the model.

In the second experiment, we performed hyperparameter tuning on the model
using the Hyperband algorithm which is a competition-based approach to
efficiently explore hyperparameter space.(Li et al., 2016) We considered
variables such as number of layers, neurons per layer, the inclusion of batch
normalization, and dropout regularization. Dropout was considered to prevent
overfitting and promote generalization while batch normalization was
incorporated to normalize inputs within each batch which can lead to faster
convergence.

For our third experiment, we conducted training using our best-performing
feature selection methods and model architecture on our cardiovascular disease
treatment dataset. This dataset is comprised of drug pairs where at least one
drug was specifically used for cardiovascular disease treatment. By focusing
on this domain-specific dataset, we aimed to assess the performance and
applicability of our approach within a specific medical context.

##  4\. Evaluation and Performance

For each model examined in our study, we evaluated its performance using the
following metrics: AUROC (Area Under ROC Curve) and AUPR (Area Under the
Precision-Recall Curve). We used AUROC to provide an assessment of the model’s
discriminatory power and AUPR to capture the trade-off between precision and
recall. We believe AUROC is a robust evaluation metric because our test set is
balanced. As mentioned in the data processing step, we employed negative
sampling techniques to get a balanced dataset. We calculated the average of
these metrics across the different side effects to provide a more
comprehensive evaluation of the model’s performance. We also evaluated each
model on training time.

Additionally, we investigated side effects for which the model performed
exceptionally well and those where its performance was comparatively poorer.
To further understand the significance of these side effects, we employed
Saedr scores as a means of assessing the severity of side effects grouped by
model prediction accuracy. Saedr scores, developed by Lavertu et al, are
designed to quantify the severity of adverse drug events using social media
network analysis.(Lavertu et al., 2021) We categorized side effects into three
distinct bins based on the model prediction accuracy: AUROC 0.85-0.90, AUROC
0.90-0.95, and AUROC 0.95-0.99. Then we evaluated and compared the average
Saedr scores associated with each of these bins.

##  5\. Results

###  5.1. Feature Selection

The initial dimensions of our drug feature matrix, before any feature
selection, were (645, 18279) across 964 side effects. During our first
experiment, we assessed the impact of different feature selection methods on
the performance of our model (Table 1). We observed as the variance captured
by PCA increased and as molecular embeddings were incorporated into the input
matrix, there was a moderate improvement in AUROC and AUPRC. We chose to
proceed with PCA 0.95 as the chosen feature selection method and with
molecular embeddings for our subsequent experiments. In the future, it may
also be interesting to look more closely at the performance of some of the PCA
datasets with lower variance as they would utilize less computational
resources and time.

Table 1. Evaluation of Feature Selection Methods |  | With Molecular Embeddings |  | Without Molecular Embeddings |
---|---|---|---|---|---
Feature Selection | Drug Feature Dimensions | AUROC | AUPRC | AUROC | AUPRC
UMAP + Norm | $\left(645,2\right)$ | 0.516 | 0.522 | 0.532 | 0.545
PCA(0.85) | $\left(645,667\right)$ | 0.901 | 0.824 | 0.897 | 0.819
PCA(0.90) | $\left(645,733\right)$ | 0.900 | 0.823 | 0.901 | 0.821
PCA(0.95) | $\left(645,825\right)$ | 0.903 | 0.826 | 0.902 | 0.825
PCA(0.99) | $\left(645,972\right)$ | 0.906 | 0.829 | 0.905 | 0.825

###  5.2. Hyperparameter Tuning

Upon completion of the hyperparameter turning process, we analyzed the results
obtained for each side effect. To ensure a more generalizable mode, we
selected the overall best model parameters by identifying the most common
parameters across all side effects. This approach aimed to find a balance
between individual side effect performance and the ability to capture patterns
across collective characteristics. The selected hyperparameters which reflect
our overall best model configuration are highlighted in Table 2.

Table 2. Selected Hyperparameter Values Number of Layers | Neurons Per Layer | Batch Norm | Dropout
---|---|---|---
2 | 300, 100 | Yes | No

###  5.3. Overall Model Performance

In our analysis, we found that our hyperparameter-tuned model, DrIVeNN,
slightly outperformed some commonly used baselines for AUROC (Table 3). Please
note that the performance values for AUROC and AUPRC for other models were
obtained from previously published works.(Masumshah et al., 2021) On CPU, the
DrIVeNN model took about 2 hours and 30 minutes to train.

Table 3. Evaluation of General Model Performance Model | AUROC | AUPRC
---|---|---
RESCAL | 0.693 | 0.613
DEDICOM | 0.705 | 0.637
DeepWalk | 0.761 | 0.737
DeepDDI | 0.830 | 0.503
Decagon | 0.874 | 0.825
KGDDI | 0.891 | 0.653
KGNN | 0.896 | 0.658
DrIVeNN_all | 0.901 | 0.821

###  5.4. Domain-Specific Results

The performance evaluation of our domain-specific model, specifically focused
on cardiovascular disease treatment drug pairs, yielded promising results. We
observed a significant increase in AUROC and AUPRC values, reaching 0.9725 and
0.952 respectively (Table 4). For comparison, we evaluated the performance of
the general DrIVeNN_all model on the same datasets. The general model had an
average AUROC of 0.901 for the test set created from all drug pairs, and only
0.701 for the test set created from CVD drug pairs. These findings show our
domain-specific model performs significantly better than a general model on
domain-specific drug pairs. These findings highlight the potential of domain-
specific models to increase predictive performance for drug-drug interaction
prediction. Additionally, on CPU, our domain-specific model trained in under
20 minutes.

Table 4. Evaluation of Domain-Specific Models Model | Test Set | AUROC | AUPRC
---|---|---|---
DrIVeNN_all | All Drug Pairs | 0.901 | 0.821
DrIVeNN_all | CVD Drug Pairs | 0.826 | 0.802
DrIVeNN_cvd | All Drug Pairs | 0.701 | 0.683
DrIVeNN_cvd | CVD Drug Pairs | 0.975 | 0.952

###  5.5. Side Effect Severity

We conducted an analysis of side effects associated with CVD-related treatment
polypharmacy, specifically focusing on the prediction AUROC of these side
effects from our model, DrIVeNN_cvd. To address the severity of these side
effects, we utilized Saedr scores as a metric. Out of the 3,196 unique side
effects caused by CVD-related treatment polypharmacy, we were able to obtain
Saedr scores for 2,794 side effects. This subset of side effects became the
focus of our analysis. In the future, we would like to explore additional
severity metrics and work on obtaining a larger domain-specific dataset to
allow for a more comprehensive analysis of drug-drug interactions within the
domain.

After excluding the first group (¡0.85 AUROC) from our analysis because it
contained only one side effect, we observed that, on average, DrIVeNN_cvd
demonstrated slightly higher AUROC scores for side effects with higher
severity scores (Table 5). This finding suggests that our model exhibits
improved performance on side effects that are associated with greater
severity, as measured by Saedr scores. To provide illustrative examples, we
have included a selection of side effects from each group in the table below
to showcase the variation in severity and corresponding AUROC scores.

Table 5. Severity of Side Effects by AUROC Bin AUROC Bin | Median Saedr Score | Selected Side Effects in Group | Total Side Effects
---|---|---|---
$\left(0.85,0.90\right)$ | 0.525 | carpal tunnel, hematoma, constipation, glaucoma | 8
$\left(0.90,0.95\right)$ | 0.588 | cardiac murmur, angina, excess potassium, incontinence | 32
$\left(0.95,0.99\right)$ | 0.605 | cardiac ischemia, heart attack, phlebothrombosis, ventricular fibrillation | 239

##  6\. Discussion

The drug-drug interaction (DDI) prediction task is a critical area of research
that plays a role in ensuring patient safety and healthcare management.
Traditionally, DDI prediction has been approached in two distinct ways:
predicting whether a side effect will occur as a result of drug interactions
and predicting the specific side effect that may occur. Our study focuses on
the second aspect. By targeting the prediction of specific side effects, we
have a more granular approach to understanding the potential outcomes of
polypharmacy. In our evaluation, we introduced Saedr scores as a metric to
assess the severity of side effects with lower and higher prediction accuracy
to gain insights into the potential impact and clinical relevance of this
model. This approach allows us to identify where areas of improvement in
predication accuracy may have significant clinical implications.

This study presented DrIVeNN, a novel approach to DDI prediction that
incorporated diverse drug features. In our methodology, we used PCA with a
variance threshold of 0.95 for feature selection then for each drug pair, a
representation was created by summing the selected features. This served as
the input for our model which had a performance comparable to state-of-the-art
baselines. Additionally, our findings indicate that our domain-specific model,
DrIVeNN_cvd, outperformed our general model in DDI prediction. This indicates
the potential for improved accuracy to predict DDIs with domain-specific
models. To the best of our knowledge, DrIVeNN_cvd represents the first domain-
specific model developed for the DDI prediction task. Exploring additional
domains and developing tailored models could lead to further improvements in
predictive accuracy.

The first notable limitation is the size of our domain-specific dataset.
Although we made substantial effort to collect relevant data for
cardiovascular disease treatment, the dataset’s size may restrict the full
exploration of the DDIs in the domain. Future work may include compiling a
more comprehensive and expansive domain-specific dataset. In addition, it is
worth exploring other medical domains where polypharmacy is common, for
example, the mental health domain. This could offer insights into the
generalizability and effectiveness of domain-specific models. Another
limitation of our study is that we evaluated our findings solely on the
dataset of known side effects. While this dataset provides a foundation for
initial model evaluation, it does not fully encompass the complexities of
real-world patient data. To better understand the potential to use this model
to predict DDIs in a real-world clinical setting, we believe that evaluations
on patient data sources, such as electronic health records (EHRs) are needed.
Doing so would provide more insight into the applicability of the model in
clinical settings and an evaluation framework for other models.

Some other potential avenues for future research includes a more comprehensive
exploration of both the time and space complexities of the model.
Additionally, trying local Principal Component Analysis (PCA) could prove
beneficial. While global PCA was employed due to its widespread application in
contexts such as ours, it is worth noting its use comes with strong linearity
assumptions. Consequently, global PCA might offer a better fit for
representing the data.

The potential impact of DrIVeNN extends beyond drug development and clinical
decision-making. Accurate DDI prediction can assist healthcare professionals
in proactively mitigating adverse effects, optimizing drug combinations, and
maximizing therapeutic outcomes. Additionally, pharmaceutical companies may
benefit from using our model to optimize drug discovery pipelines, prioritize
candidate drugs with lower interaction risks, and minimize costly experimental
testing.

The main contributions of this research paper can be summarized as follows:

  * •

Novel Drug Feature Vectors and Model: Different from baseline models, we used
drug feature vectors including drug structures, drug-protein interactions, and
mono drug side effects. We also propose a feature selection method and model
architecture for the drug-drug interaction task.

  * •

Domain-Specific Model Performance: We evaluated our model in two ways: with
all available drug pairs (as others have), and second, with a focus on
polypharmacy in CVD treatment (new in this study). Experimental results
obtained for this evaluation are noteworthy as they highlight the promising
potential of the effectiveness of domain-specific models in DDI prediction.

##  7\. Conclusion

In conclusion, our research has highlighted the potential of domain-specific
models for drug-drug interaction (DDI) prediction. By introducing DrIVeNN as a
new model that utilizes diverse drug features and feature selection
techniques, we have demonstrated competitive performance for the DDI
prediction task, comparable to existing state-of-the-art models in terms of
AUROC and AUPRC. We also have preliminary results suggesting our domain-
specific model specifically tailored for cardiovascular disease treatment
(DrIVeNN_cvd), exhibits potential advantages in predictive accuracy.
Furthermore, we observed that on average, DrIVeNN_cvd performs better on more
severe side effects, as quantified by the Saedr score system. This finding may
suggest other underlying factors related to severity that influence predictive
accuracy. Exploring these factors may provide more insights into the nature of
drug-drug interactions.

## References

  * (1)
  * Chen et al. (2021) Yujie Chen, Tengfei Ma, Xixi Yang, Jianmin Wang, Bosheng Song, and Xiangxiang Zeng. 2021\.  MUFFIN: multi-scale feature fusion for drug–drug interaction prediction.  _Bioinformatics_ 37 (9 2021), 2651–2658.  Issue 17. https://doi.org/10.1093/bioinformatics/btab169
  * Delara et al. (2022) Mahin Delara, Lauren Murray, Behnaz Jafari, Anees Bahji, Zahra Goodarzi, Julia Kirkham, Zia Chowdhury, and Dallas P. Seitz. 2022\.  Prevalence and factors associated with polypharmacy: a systematic review and Meta-analysis.  _BMC Geriatrics_ 22 (12 2022).  Issue 1. https://doi.org/10.1186/s12877-022-03279-x
  * Garber and Brownlee (2019) J. Garber and S. Brownlee. 2019.  Medication Overload: America’s Other Drug Problem.  https://doi.org/10.46241/LI.WOUK3548
  * Harshman et al. (1982) Richard A. Harshman, Paul E. Green, Yoram Wind, and Margaret E. Lundy. 1982.  A Model for the Analysis of Asymmetric Data in Marketing Research.  _Marketing Science_ 1 (5 1982), 205–242.  Issue 2. https://doi.org/10.1287/mksc.1.2.205
  * JY et al. (2018) Ryu JY, Kim HU, and Lee SY. 2018.  Deep learning improves prediction of drug-drug and drug-food interactions.  _Proc Natl Acad Sci U S A_ 115 (2018), E4304–E4311.  Issue 18. https://doi.org/doi:10.1073/pnas.1803294115
  * Kantor et al. (2015) Elizabeth D. Kantor, Colin D. Rehm, Jennifer S. Haas, Andrew T. Chan, and Edward L. Giovannucci. 2015.  Trends in prescription drug use among adults in the United States from 1999-2012.  _JAMA - Journal of the American Medical Association_ 314 (11 2015), 1818–1831.  Issue 17. https://doi.org/10.1001/jama.2015.13766
  * Karim et al. (2019) Md. Rezaul Karim, Michael Cochez, Joao Bosco Jares, Mamtaz Uddin, Oya Beyan, and Stefan Decker. 2019\.  Drug-Drug Interaction Prediction Based on Knowledge Graph Embeddings and Convolutional-LSTM Network.  _Proceedings of the 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics_ , 113–123.  https://doi.org/10.1145/3307339.3342161
  * Kim et al. (2023) Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, Leonid Zaslavsky, Jian Zhang, and Evan E Bolton. 2023.  PubChem 2023 update.  _Nucleic Acids Research_ 51 (1 2023), D1373–D1380.  Issue D1. https://doi.org/10.1093/nar/gkac956
  * Kuhn et al. (2016) Michael Kuhn, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. 2016.  The SIDER database of drugs and side effects.  _Nucleic Acids Research_ 44 (2016), D1075–D1079.  Issue D1. https://doi.org/10.1093/nar/gkv1075
  * Lavertu et al. (2021) Adam Lavertu, Hamamzy Tymor, and Russ Altman. 2021\.  Quantifying the Severity of Adverse Drug Reactions Using Social Media: Network Analysis.  _Journal of Medical Internet Research_ (2021).  https://doi.org/10.2196/27714
  * Li et al. (2016) Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2016.  Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization.  (3 2016).
  * Li et al. (2021) Mufei Li, Jinjing Zhou, Jiajing Hu, Wenxuan Fan, Yangkang Zhang, Yaxin Gu, and George Karypis. 2021.  DGL-LifeSci: An Open-Source Toolkit for Deep Learning on Graphs in Life Science.  _ACS Omega_ 6 (10 2021), 27233–27238.  Issue 41. https://doi.org/10.1021/acsomega.1c04017
  * Lin et al. (2020) Xuan Lin, Zhe Quan, Zhi-Jie Wang, Tengfei Ma, and Xiangxiang Zeng. 2020.  KGNN: Knowledge Graph Neural Network for Drug-Drug Interaction Prediction.  _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence_ , 2739–2745.  https://doi.org/10.24963/ijcai.2020/380
  * Masumshah et al. (2021) Raziyeh Masumshah, Rosa Aghdam, and Changiz Eslahchi. 2021\.  A neural network-based method for polypharmacy side effects prediction.  _BMC Bioinformatics_ 22 (12 2021).  Issue 1. https://doi.org/10.1186/s12859-021-04298-y
  * McInnes et al. (2018) Leland McInnes, John Healy, and James Melville. 2018\.  UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction.  (2 2018).  http://arxiv.org/abs/1802.03426
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed Representations of Words and Phrases and their Compositionality, C J Burges, L Bottou, M Welling, Z Ghahramani, and K Q Weinberger (Eds.).  _Advances in Neural Information Processing Systems_ 26\.  https://proceedings.neurips.cc/paper_files/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf
  * Nickel et al. (2011) Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011\.  A Three-Way Model for Collective Learning on Multi-Relational Data.  https://icml.cc/2011/papers/438_icmlpaper.pdf
  * Pedregosa et al. (2011) Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, and Vincent Dubourg. 2011.  Scikit-learn: Machine learning in Python.  _the Journal of machine Learning research_ 12 (2011), 2825–2830.
  * Perozzi et al. (2014) Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014\.  DeepWalk: online learning of social representations.  _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 701–710.  https://doi.org/10.1145/2623330.2623732
  * Peryea et al. (2021) Tyler Peryea, Noel Southall, Mitch Miller, Daniel Katzel, Niko Anderson, Jorge Neyra, Sarah Stemann, Dac-Trung Nguyen, Dammika Amugoda, Archana Newatia, Ramez Ghazzaoui, Elaine Johanson, Herman Diederik, Larry Callahan, and Frank Switzer. 2021.  Global Substance Registration System: consistent scientific descriptions for substances related to health.  _Nucleic Acids Research_ 49 (1 2021), D1179–D1185.  Issue D1. https://doi.org/10.1093/nar/gkaa962
  * Sheikh-Taha and Asmar (2021) Marwan Sheikh-Taha and Myriam Asmar. 2021.  Polypharmacy and severe potential drug-drug interactions among older adults with cardiovascular disease in the United States.  _BMC Geriatrics_ 21 (12 2021).  Issue 1. https://doi.org/10.1186/s12877-021-02183-0
  * Siramshetty et al. (2022) Vishal B Siramshetty, Ivan Grishagin, Dac-Trung Nguyen, Tyler Peryea, Yulia Skovpen, Oleg Stroganov, Daniel Katzel, Timothy Sheils, Ajit Jadhav, Ewy A Mathé, and Noel T Southall. 2022.  NCATS Inxight Drugs: a comprehensive and curated portal for translational research.  _Nucleic Acids Research_ 50 (1 2022), D1307–D1316.  Issue D1. https://doi.org/10.1093/nar/gkab918
  * Szklarczyk et al. (2016) Damian Szklarczyk, Alberto Santos, Christian von Mering, Lars Juhl Jensen, Peer Bork, and Michael Kuhn. 2016\.  STITCH 5: augmenting protein–chemical interaction networks with tissue and affinity data.  _Nucleic Acids Research_ 44 (1 2016), D380–D384.  Issue D1. https://doi.org/10.1093/nar/gkv1277
  * Tatonetti et al. (2012) Nicholas P. Tatonetti, Patrick P. Ye, Roxana Daneshjou, and Russ B. Altman. 2012.  Data-driven prediction of drug effects and interactions.  _Science Translational Medicine_ 4 (3 2012).  Issue 125. https://doi.org/10.1126/scitranslmed.3003377
  * Trouillon et al. (2016) Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016.  Complex Embeddings for Simple Link Prediction.  _CoRR_ abs/1606.06357 (2016).  http://arxiv.org/abs/1606.06357
  * Zitnik et al. (2018) Marinka Zitnik, Monica Agrawal, and Jure Leskovec. 2018\.  Modeling polypharmacy side effects with graph convolutional networks.  _Bioinformatics_ 34, i457–i466.  Issue 13. https://doi.org/10.1093/bioinformatics/bty294