#  $\lambda$-Scaled-Attention: A Novel Fast Attention Mechanism for Efficient
Modeling of Protein Sequences

Ashish Ranjan, Md Shah Fahad, and Akshay Deepak  Ashish Ranjan (corresponding
author), Md Shah Fahad and Akshay Deepak are with the Department of Computer
Science & Engineering, National Institute of Technology Patna, India. E-mail:
{ashish.cse16, shah.cse16<EMAIL_ADDRESS>
###### Abstract

Attention-based deep networks have been successfully applied on textual data
in the field of NLP. However, their application on protein sequences poses
additional challenges due to the weak semantics of the protein words, unlike
the plain text words. These unexplored challenges faced by the standard
attention technique include (i) vanishing attention score problem and (ii)
high variations in the attention distribution. In this regard, we introduce a
novel $\lambda$-scaled attention technique for fast and efficient modeling of
the protein sequences that addresses both the above problems. This is used to
develop the $\lambda$-scaled attention network and is evaluated for the task
of protein function prediction implemented at the protein sub-sequence level.
Experiments on the datasets for biological process (BP) and molecular function
(MF) showed significant improvements in the F1 score values for the proposed
$\lambda$-scaled attention technique over its counterpart approach based on
the standard attention technique (+2.01% for BP and +4.67% for MF) and state-
of-the-art ProtVecGen-Plus approach (+2.61% for BP and +4.20% for MF).
Further, fast convergence (converging in half the number of epochs) and
efficient learning (in terms of very low difference between the training and
validation losses) were also observed during the training process.

###### Index Terms:

$\lambda$-Scaled Attention Technique, $\lambda$-Scaled Attention Network,
Attention Technique, Long-Short Term Memory, Gated Recurrent Unit, Protein
Sequence.

##  1 Introduction

Automatic functional characterization of proteins is one of the key challenges
in computational biology. The proteins are fundamental to the biological
interactions. They are found helpful in, among other things, studying the
attack of viruses [1, 2, 3] and harmful bacteria [3] on the organism.
Proteins, in their elementary form, are linear chains of amino acids called
“protein sequences”, commonly also referred to as “the language of life” [4].
In the past few years, undoubtedly, protein sequences have become a cheap and
reliable source for the protein studies. The surplus amount of protein
sequences – generated due to the blooming of sequencing technologies [5] – has
marked a transitional shift toward the data-driven machine learning approaches
[6, 7] for the characterization of proteins.

The evolution of deep-learning has further taken the automated
characterization process to the next level by significantly bringing down the
effort needed to process protein sequences. In particular, the deep recurrent
neural networks (DRNN), e.g., long-short term memory (LSTM) networks, have
shown great potential with protein sequences for the function prediction task
[8, 9, 10]. In protein sequences, which are also represented as a string of
n-mers [8, 11], the order of amino acids/n-mers has a significant effect in
determining their function(s). These orderly arrangements are believed to
create orderly dependencies/patterns proven useful in determining their
function(s). DRNN helps capture such orderly dependencies between the amino
acids/n-mers of a protein sequence.

However, it should be noted that protein words (either amino acid or n-mers)
that make up a protein sequence have a different significance in uncovering
the underlying functional context in the sequence. The set of key significant
protein words, when combined, further provides a clear and strong indication
of the functional context. The functional context refers to the “small
pattern” common among the protein sequences of the same functional family.
These are created due to the orderly dependencies between the sub-set of
protein words making up a protein sequence, e.g., sequence motifs [12]. Thus,
there are two important aspects to efficient processing of protein sequences:
(i) learning the orderly dependencies between the protein words, and (ii)
capturing the importance of individual protein words toward revealing the
underlying functional context.

While LSTM can capture the orderly dependencies between the protein words, it
has no mechanism to highlight the significance of words. This causes
masking/losing the critical information (possibly, containing the functional
context), resulting in a sub-optimal intermediate representation. Therefore,
the attention mechanism [13] that builds an intermediate representation by
prioritizing the key informative words, regardless of their position in the
sentence, will be useful in such cases and has been explored in this work.
Attention-based DRNN has already become a standard solution for a wide variety
of tasks, such as document classification [14] and sentiment classification
[15, 16]. They have a profound effect on the produced intermediate
representations.

Howsoever, the application of attention technique on protein sequences has its
own set of challenges. This is because, unlike the plain text words which have
strong context-specific semantics, protein words (i.e., amino acid/n-mers)
have a very weak context-specific semantics. Thus, while a small sub-set of
words is sufficient to identify the context of a plain text sentence, possibly
a large sub-set of protein words is needed to identify a functional context in
the protein sequence. The large sub-set of protein words helps to make a
strong case for the underlying functional context. The challenges faced by the
attention technique are as follows:

  * •

Vanishing attention score problem: Attention [13] technique, in general,
produces a distribution that helps identify the significance of words in a
sentence. While for the plain text sentences, standard attention [13] performs
reasonably well with only a few significant words to focus, it struggle to
reproduce the same performance for the protein sequences. The reason for this
is the vanishing of the attention scores caused due to large sub-set of
significant protein words that form a functional context.

For example, consider the distribution of attention scores as shown in Figures
1 and 2. In Figure 1, where there are only a few significant words in the
sentence, the attention score is high (for $w_{2}=0.75$). However, when the
number of significant words in a sentence is increased, the attention scores
become small (see Figure 2). A further increase in the number of significant
words will soon start to produce diminished attention scores.

In the latter case (Figure 2), when the attention scores become too small, it
may cause a restricted flow of information down the network. We call this
problem, the “vanishing attention score” problem.

  * •

High variations in the attention distributions: The attention score
distributions for different protein sequences, in general, varies
significantly. This creates problem capturing the high variations among the
allocated scores, making the training process less efficient.

Figure 1: Case I: Distribution of attention scores with only a small number of
significant words {w2, w3}. Figure 2: Case II: Distribution of attention
scores with a relatively large number of significant words {w2, w3, w4, w5}.

To address the above discussed issues, a novel scaled-attention technique that
produces attention scores via the scale-up operation is proposed in this work
that helps eliminate the “vanishing attention score problem”, irrespective of
the number of significant protein words in the protein sequence. The proposed
solution also helps in reducing the high variation in the attention
distribution. In light of the above discussion, the major contributions of the
paper are:

  * •

A novel $\lambda$-scaled attention technique for the fast and efficient
modeling of protein sequences. The $\lambda$ parameter is useful in
controlling the convergence of model while training. A high value of $\lambda$
will push for fast convergence, while a low value of $\lambda$ will cause slow
convergence.

  * •

The proposed $\lambda$-scaled attention technique is next used to develop a
general deep neural network architecture for the protein sequence, called the
“$\lambda$-scaled attention network”, which is evaluated for the protein
function prediction task implemented at the protein sub-sequence level.

  * •

Experiments on the datasets for biological process (BP) and molecular function
(MF) demonstrate that the proposed $\lambda$-scaled attention technique has a
major impact on improving the overall performance. Compared to its counterpart
approach based on standard attention [13] with segment size = 100, it shows
improvements in F1-score by a margin of +2.01% (BP) and +4.67% (MF),
respectively. The corresponding improvements over state-of-the-art multi-
segment based ProtVecGen-Plus approach [8] are +2.61% (BP) and +4.20% (MF),
respectively.

  * •

Further, during training, the proposed $\lambda$-scaled attention technique
converges faster (taking around half the amount of time taken by the baseline
methods) and is efficient (in terms of very low difference between the
training and validation losses) when compared to both the simple baseline and
baseline+attention based classifiers (see Sec. 4.3.2 for more details).

The organization of rest of the paper is as follows: Section 2 outlines the
proposed $\lambda$-Scaled Attention Network, following which Section 3 discuss
the use of the $\lambda$-Scaled Attention Network for protein function
prediction. This follows results and discussion in Section 4. Finally, Section
5 concludes the paper.

##  2 $\lambda$-Scaled Attention Network

In this section, we introduce a novel $\lambda$-scaled attention network that
can be applied efficiently for a wide range of classification tasks using
protein sequences. However, in this paper the proposed network has been
evaluated for the task of protein function prediction. The proposed
architecture is composed of the following components: (1) Protein word
embedding layer, (2) Deep recurrent processing layer, (3) Protein word-level
$\lambda$-scaled attention layer, and (4) Dense output layer. Moreover, we
also added dropout layers [17] before the deep recurrent processing and dense
output layers, which helps avoid the over-fitting problem. The corresponding
amount of disconnections with each dropout layers were taken as 0.3% and 0.2%,
respectively. These are described in Sections 2.3 – 2.6. Before their
description, the preliminary notations and protein sequence tokenization (as
the pre-processing step) are discussed in Sections 2.1 – 2.2.

###  2.1 Notations

Let $\\{GO_{1},GO_{2},...,GO_{K}\\}$ denote the set of ‘$K$’ $GO$-terms
(either for biological process or molecular function). Let
$P=\\{(P_{1},Y_{1}),(P_{2},Y_{2}),...,(P_{m},Y_{m})\\}$ denote a database of
$m$ labeled protein sequences, where $P_{i}$ denotes a protein sequence and
$Y_{i}$ = $\\{y_{i1},y_{i2},...,y_{ik}...,y_{iK}\\}$ denotes one-hot encoding
representing the set of $GO$-term annotations corresponding to the protein
sequence $P_{i}$, such that $y_{ik}=1$ if $P_{i}$ exhibits $GO_{k}$ and 0
otherwise.

###  2.2 Protein Sequence Tokenization

The tokenization of protein sequences involves decomposing protein sequences
into a string of amino acids or n-mers, which are assumed to be the protein
words for the protein sequences. Since the length of protein sequences is
highly variable in nature, for simplification, we assume the maximum
permissible length for protein sequences to be max_seq_len. Protein sequences
with length $>$ max_seq_len are truncated, while padding is done for shorter
protein sequences. The maximum number of possible protein words in a sequence
is:

$T=\textit{max\\_seq\\_len}-n+1$ (1)

where, n is the size of n-mers / protein words.

###  2.3 Protein Word Embedding Layer

Tokenized protein sequences are passed through the embedding layer that helps
in creating meaningful dense representations for the protein words in the
sequence. The output from the embedding layer is a matrix of size [$T$ x d],
where $T$ denotes the maximum number of possible protein words in a sequence
(Equation 1) and d denotes the embedding dimension.

###  2.4 Deep Recurrent Processing Layer

To accomplish the learning of the orderly dependencies between the protein
words, deep recurrent neural networks (DRNN), such as GRU [18] / LSTM [19]
network, are used to process protein sequences. These networks are linear
chain of recurrent units (e.g., GRU cell / LSTM cell) that process sequences,
word-by-word, and produce a hidden state vector ($h_{t}$) corresponding to
every protein word.

Let at any time-step (say $t$), $x_{t}$ represent the current input and
$h_{t-1}$ represent the previous hidden state. Then, the recurrent unit
accepts $x_{t}$ and $h_{t-1}$ as the input and produces $h_{t}$ as the
corresponding output. Note that $x_{t}$ actually refers to the protein word
embedding occurring at time-step $t$. A brief description of both GRU and LSTM
cells are given below.

a) Gated Recurrent Unit (GRU). The major components with the GRU cells are:
update gate and reset gate. The update gate helps to measure the usefulness of
the past information ($h_{t-1}$) for the current time-step $t$. The output of
the update gate, denoted $z_{t}$, is described in Eq. 2. The reset gate, on
the other hand, helps to forget the past information that may be irrelevant in
the future. The output of the reset gate, denoted $r_{t}$, is described in Eq.
3.

$z_{t}=\sigma(W_{z}x_{t}+U_{z}h_{t-1}+b_{z})$ (2)
$r_{t}=\sigma(W_{r}x_{t}+U_{r}h_{t-1}+b_{r})$ (3)

where,
– $W_{z}$, $W_{r}$, $U_{z}$, and $U_{r}$ denote weight matrices,
– $b_{z}$ and $b_{r}$ denote bias vectors.

The outputs $z_{t}$ and $r_{t}$ help generate the hidden state vector $h_{t}$
as follows:

$h_{t}=[z_{t}\odot h_{t-1}]+[(1-z_{t})\odot\tilde{h_{t}}]$ (4)

where, $\tilde{h_{t}}$ denotes the current memory content and is computed as
follows:

$\tilde{h_{t}}=\tanh(W_{h}x_{t}+(r_{t}\odot U_{h}h_{t-1})+b_{h})$ (5)

where,
– $W_{h}$ and $U_{h}$ denote weight matrices,
– $b_{h}$ denotes the bias vector, and
– $\odot$ denotes the element-wise multiplication.

Figure 3: $\lambda$-Scaled Attention Network: $\lambda$-scaled-attention based
deep recurrent network for the protein sequences.

b) Long Short-Term Memory (LSTM). The key component of an LSTM cell is the
memory element called cell state that maintains the global information. Other
major components with an LSTM cell include: forget gate and input gate, which
assist cell state in maintaining the global information. The forget gate
assists cell state to forget the irrelevant past information. The output of
the forget gate, denoted as $f_{t}$, is described in Eq. 6. The input gate
assists in adding new information to the cell state. The outputs of the input
gate, denoted as $i_{t}$ and $\bar{C_{t}}$, are described in Eq. 7.

$f_{t}=\sigma(W_{f}[h_{t-1},x_{t}]+b_{f})$ (6)

$\displaystyle i_{t}=\sigma(W_{i}[h_{t-1},x_{t}]+b_{i})$ (7)
$\displaystyle\bar{C_{t}}=tanh(W_{c}[h_{t-1},x_{t}]+b_{c})$

where,
– $W_{f}$, $W_{i}$, and $W_{c}$ denotes weight matrices,
– $b_{f}$, $b_{i}$, and $b_{c}$ denote bias vectors.

The cell state is updated as follows:

$C_{t}=(f_{t}\odot C_{t-1})+(i_{t}\odot\bar{C_{t}})$ (8)

where, $\odot$ represents element-wise multiplication.

The hidden state $h_{t}$ is obtained based on the dot product between the
updated cell state $C_{t}$ and the output logic $o_{t}$. They are described
as:

$o_{t}=\sigma(W_{o}[h_{t-1},x_{t}]+b_{o})$ (9) $h_{t}=o_{t}\odot tanh(C_{t})$
(10)

where, $W_{o}$ and $b_{o}$ are the weight matrix and bias vector respectively,
with the output gate.

Bi-directional recurrent processing: In this work, the deep recurrent
processing layer is implemented with the bi-directional configuration [20]
comprising of forward and backward DRNN’s. The bi-directional property allows
computing the hidden state for the protein words, which is based on both the
preceding as well as the following words in a protein sequence. Let
$[x_{1},x_{2},...,x_{t},...,x_{T}]$ represent the ordered protein words in a
protein sequence, where $T$ is the total number of time-steps equivalent to
the maximum possible protein words in a sequence (Eq. 1).

The forward DRNN reads the protein sequences starting from $x_{1}$ to $x_{T}$
and the backward DRNN reads the protein sequences backwards from $x_{T}$ to
$x_{1}$. Let $h_{t}^{f}$ and $h_{t}^{b}$ represent the hidden states for the
word $x_{t}$ obtained from the forward and backward DRNNs, respectively. The
final hidden state for the protein word $x_{t}$ is the concatenation of
$h_{t}^{f}$ and $h_{t}^{b}$, i.e., $h_{i}$ = [$h_{t}^{f}$, $h_{t}^{b}$]. The
dimension of the hidden states from both the forward and backward DRNN is
taken as 70.

###  2.5 $\lambda$-Scaled Attention Layer

Here we introduce a novel $\lambda$-scaled attention technique to address both
the vanishing attention score problem and the variations in the attention
score distribution. The proposed attention technique allows to find a good
representation of the protein sequences by aggregating the weighted
representation of their protein words.

Let $[h_{i1},h_{i2},...,h_{it},...,h_{iT}]$ represent the list of hidden state
vectors for the $i^{th}$ protein sequence, where $h_{it}$ is the hidden state
vector corresponding to the protein word occurring at time $t$. Formally, the
proposed $\lambda$-scaled attention technique can be defined as consisting of
following steps:

  1. 1.

Each hidden state vector $h_{it}$, where $t\in\\{1,2,...,T\\}$, is fed forward
through a single layer perceptron to obtain the corresponding hidden
representation $u_{it}$ as:

$u_{it}=tanh(W_{a}h_{it}+b_{a})$ (11)

where, $W_{a}$ and $b_{a}$ represent the weight matrix and bias element,
respectively.

  2. 2.

Next, the scaled attention score $\alpha_{it}^{{}^{\prime}}$ for the protein
words occurring at time-steps $t\in\\{1,2,...,T\\}$ is calculated as follows:

$\alpha_{it}^{{}^{\prime}}=\frac{\alpha_{it}}{\alpha_{it}^{max}};\hskip
17.07164pt\text{s.t.}\hskip 5.69054pt\alpha_{it}^{{}^{\prime}}\leq 1$ (12)

where,

$\alpha_{it}=\frac{exp(u_{it}^{T})}{\sum_{t=1}^{T}exp(u_{it}^{T})}$ (13)

$\alpha_{it}^{max}=max[\alpha_{i1},\alpha_{i2},...,\alpha_{it},...,\alpha_{iT}]$
(14)

Equation 13 represents the standard softmax operation, which is used to find
the distribution of attention scores $\alpha_{it},\text{where
}t\in\\{1,2,...,T\\}$. Following this in Eq. 14, the value of the maximum
attention score $\alpha_{it}^{max}$ is computed from the distribution. These
are used in Eq. 12 to compute the scaled-up attention scores; the maximum
possible scaled-up attention score is 1.

The scaling operation in Eq. 12 creates the amplified attention scores,
thereby, eliminating the vanishing attention score problem. Moreover, scaling
also ensures that attention scores for each time-step are now on the same
scale, which reduces the variations in the attention score distribution.

  3. 3.

Optionally, a scalar parameter “$\lambda$” can also be multiplied to the
scaled attention scores:
$[\alpha_{i1}^{{}^{\prime}},\alpha_{i2}^{{}^{\prime}},...,\alpha_{it}^{{}^{\prime}},...,\alpha_{iT}^{{}^{\prime}}]$,
which limits the maximum value a scaled attention score can take. Such
restriction on the maximum possible scaled attention score is useful in
controlling the training rate. A higher value of “$\lambda$” will push for
faster training, while a lower value of “$\lambda$” will slow-down the
training rate.

  4. 4.

The final vector $s_{i}$ for the $i^{th}$ protein sequence is the aggregation
of the weighted hidden states. The weighted hidden states are computed by
multiplying the scaled attention scores $\alpha_{it}^{{}^{\prime}}$s with the
corresponding hidden states $h_{it}$s:

$s_{i}=\sum_{t=1}^{T}\alpha_{it}^{{}^{\prime}}h_{it}$ (15)

The benefits of the proposed $\lambda$-scaled attention are as follows:

  * •

It helps overcome the vanishing attention score problem, which otherwise
causes restricted flow of the information.

  * •

It further helps in reducing the high variations in the attention
distribution, making the training procedure a more efficient one.

###  2.6 Dense Output Layer

Following the attention layer is a fully-connected dense output layer, where
the number of neurons is equal to the number of GO-terms (i.e., K). The output
neurons are implemented with the sigmoid activations in order to deal with the
multi-label scenario. Note that the protein function prediction is a multi-
label classification problem. Other parameters include, “binary-cross entropy”
as the loss function and “adam” [21] as the gradient optimizer.

##  3 Protein Function Prediction

In this section, we discuss how we utilize the proposed $\lambda$-scaled
attention network for the task of protein function prediction. We choose to
use the $\lambda$-scaled attention network at the protein sub-sequence level
for classifying protein sub-sequences. This is because, in general, protein
words forming up the functional context(s) within the protein sequences do not
share long-distance dependency. The advantage of using protein sub-sequences
is also discussed in our earlier work [8]. The classified protein sub-
sequences are eventually used to determine the function(s) of their parent
sequence.

The proposed solution has the following steps: 1) Global Protein Sequence
Representation: an approach to generate global representation for protein
sequences based on small segments of protein sequences and the proposed
$\lambda$-scaled attention network, and 2) Classification Model : a multi-
layer perceptron (MLP) network based classifier to predict the function(s) of
unlabeled protein sequences. These are described next.

###  3.1 Global Protein Sequence Representation

This is a two-step procedure: (1) Training a $\lambda$-scaled attention
network using the protein segments, and (2) Utilizing the trained
$\lambda$-scaled attention network for producing the global representation for
the protein sequences.

####  3.1.1 Training a $\lambda$-scaled attention network using the protein
segments

The segmented dataset is used to train the proposed $\lambda$-scaled attention
network, which is constructed from the training dataset
$P=\\{P_{i},Y_{i}\\}_{i=1}^{m}$ as follows:

  * •

As shown in Figure 4, each protein sequence $P_{i}$ with the label-set $Y_{i}$
is decomposed into equal-sized protein segments. The set of such protein
segments corresponding to protein $P_{i}$ is denoted as
$\phi_{i}=\\{p_{i,1},p_{i,2},...,p_{i,j},...\\}$, where $p_{i,j}$ represents
the $j^{th}$ segment of protein $P_{i}$. Further, the decomposition happens in
an overlapping manner. Padding is done to the last protein segment if it is
smaller than the segment size.

  * •

Any two adjacent segments, say $p_{i,j}$ and $p_{i,j+1}$, are chosen such that
they have at least 50% of region in common.

  * •

Each segment $p_{i,j}\in\phi_{i}$ is assigned the same label-set $Y_{i}$ as
the parent protein sequence $P_{i}$. This constitutes the new segmented
training dataset as $(p_{i,j},Y_{i})$, where $i\in\\{1,2,..,m\\}$ and
$j\in\\{1,2,...\\}$.

Next, the newly constructed segmented dataset is used for the training of the
$\lambda$-scaled attention network as already discussed in Section 2.

Figure 4: Protein sequence segmentation.

####  3.1.2 Utilizing the trained $\lambda$-scaled attention network for
producing the representations for the protein sequences

The trained $\lambda$-scaled attention network based segment classifier from
previous Section 3.1.1 is used to produce the global representation for
protein sequences in both the training and testing datasets. This involves the
following steps:

  * •

The protein sequences are decomposed into protein segments as discussed before
in the Section 3.1.1. Let $\phi_{i}=\\{p_{i,1},p_{i,2},...\\}$ represent the
segments for the $i^{th}$ protein sequence.

  * •

The decomposed protein segments from the set $\phi_{i}$ are classified using
the pre-trained segment classifier. An output from the segment classifier is a
$K$-dimensional vector that represents the corresponding protein segment
$\textbf{p}_{i,j}$.

  * •

Next, the average of segment representations is computed to formulate the
intermediate sequence representation $\textbf{P}_{i}^{{}^{\prime}}$ as:

$\textbf{P}_{i}^{{}^{\prime}}\longleftarrow\frac{1}{|\phi_{i}|}\sum_{p_{i,j}\in\phi_{i}}\textbf{p}_{i,j}\hskip
17.07164pt\textit{\\# Averaging}$ (16)

For the large number of long protein sequences, only a few segments are found
supportive to the true output classes. The average operation, in general, may
cause lowering of the support for the true output classes. Thus, the values of
$\textbf{P}_{i}^{{}^{\prime}}$ are scaled-up as follows:

$\textbf{P}_{i}\longleftarrow\frac{\textbf{P}_{i}^{{}^{\prime}}}{max(\textbf{P}_{i}^{{}^{\prime}})}\hskip
25.60747pt\textit{\\# Scaling}$ (17)

###  3.2 Classification Model

The problem of protein function prediction was modeled as a multi-label
classification problem using the multi-layer perceptron (MLP) network. The MLP
network consists of a single hidden layer followed by the output layer. The
output neurons are implemented with the sigmoid activations in order to deal
with the multi-label scenario. The K-dimensional feature representations for
the protein sequences, as obtained in Section 3.1, are used for the training
of the perceptron model. Other parameters include “binary-cross entropy” as
the loss function and “adam” as the gradient optimizer.

##  4 Result and Discussion

This section presents the findings of the proposed system and discusses them
in depth. The proposed network architecture was implemented using Keras
2.0.6111https://github.com/keras-team/keras with a TensorFlow backend.

###  4.1 Datasets

For evaluation purposes, we utilized the protein sequences from the UniProtKB
database [22], where Gene Ontology (GO) [23] based terminology is used to
indicate the output labels (i.e., protein functions) for the protein
sequences. Protein sequence datasets with GO-terms defined across two
different domains, Molecular Function (MF) and Biological Process (BP), were
used for experiments. These two datasets are the same as the ones used in [8].

The BP dataset consists of 58310 protein sequences (with 295 GO-terms for the
biological process), while the MF dataset comprises of 43218 protein sequences
(with 135 GO-terms for the molecular function). In both the datasets, the
number of protein sequences corresponding to each GO-term is $\geq$ 200\. The
datasets are available for download at https://bit.ly/2RMsyOV. For all the
experiments, the datasets were split into the train and test datasets with 75%
being in the training dataset, while the remaining 25% belonging to the
testing dataset.

###  4.2 Metrics

The proposed solution was evaluated using metrics such as average precision,
average recall, and average F1-score [24, 25]; each metric being computed by
taking the average over the corresponding performances of each individual test
sample in the test dataset, with $n$ test samples. Let $Y_{i}$ be the set of
true $GO$-terms and $Y_{i}^{{}^{\prime}}$ be the set of predicted $GO$-terms.
Then, the used metrics are defined as:

  * •

Average Precision:

$\text{{Avg. Precision}}=\frac{1}{n}\sum_{i=1}^{n}\frac{\arrowvert
Y_{i}\bigcap Y_{i}^{{}^{\prime}}\arrowvert}{\arrowvert
Y_{i}^{{}^{\prime}}\arrowvert}$ (18)

  * •

Average Recall:

$\text{{Avg. Recall}}=\frac{1}{n}\sum_{i=1}^{n}\frac{\arrowvert Y_{i}\bigcap
Y_{i}^{{}^{\prime}}\arrowvert}{\arrowvert Y_{i}^{{}^{\prime}}\arrowvert}$ (19)

  * •

Average F1-Score:

$\text{{Avg. F1-score}}=\frac{1}{n}\sum_{i=1}^{n}\frac{2\arrowvert
Y_{i}\bigcap Y_{i}^{{}^{\prime}}\arrowvert}{\arrowvert
Y_{i}\arrowvert+\arrowvert Y_{i}^{{}^{\prime}}\arrowvert}$ (20)

TABLE I: Classification Accuracy with LSTM (Base. = Baseline, Attn. = Attention) |  |  $\lambda$-Scaled Attention Network
---|---|---
Seg. size |  Base. |  Base.+ Attn. |  $\lambda$ = 1.0 |  $\lambda$ = 0.7 |  $\lambda$ = 0.5 |  $\lambda$ = 0.3 |  $\lambda$ = 0.1
Dataset : Biological Process
80 |  53.71 |  55.57 |  56.83 |  56.77 |  56.69 |  56.86 |  57.27
100 |  52.33 |  54.95 |  56.96 |  56.85 |  57.09 |  57.01 |  56.96
120 |  51.51 |  55.41 |  57.15 |  57.04 |  57.16 |  57.04 |  56.39
Dataset : Molecular Function
80 |  66.13 |  67.41 |  69.89 |  69.86 |  69.57 |  69.02 |  69.01
100 |  64.41 |  65.25 |  69.92 |  69.77 |  69.17 |  68.99 |  68.01
120 |  62.26 |  65.93 |  69.80 |  69.67 |  69.29 |  68.93 |  67.32
TABLE II: Classification Accuracy with GRU (Base. = Baseline, Attn. = Attention) |  |  $\lambda$-Scaled Attention Network
---|---|---
Seg. size |  Base. |  Base.+ Attn. |  $\lambda$ = 1.0 |  $\lambda$ = 0.7 |  $\lambda$ = 0.5 |  $\lambda$ = 0.3 |  $\lambda$ = 0.1
Dataset : Biological Process
80 |  54.21 |  56.48 |  57.43 |  57.14 |  57.09 |  56.94 |  56.35
100 |  52.66 |  55.60 |  57.28 |  57.16 |  57.11 |  57.00 |  56.28
120 |  51.95 |  55.28 |  57.34 |  57.33 |  57.28 |  56.87 |  56.33
Dataset : Molecular Function
80 |  66.72 |  66.67 |  69.99 |  70.09 |  69.58 |  69.33 |  67.70
100 |  61.75 |  64.95 |  70.22 |  69.81 |  69.72 |  69.09 |  66.73
120 |  61.59 |  64.96 |  69.69 |  69.68 |  69.53 |  69.25 |  67.81

(a) Training Loss

(b) Validation Loss

Figure 5: Biological Process: LSTM based training and validation loss curve.

###  4.3 $\lambda$-scaled attention network performance analysis

In this section, we comprehensively evaluate the effect of the proposed
$\lambda$-scaled attention network onto the function prediction task by
comparing it with the two baseline models used at the sub-sequence level. The
baseline models are as follows:

  1. 1.

Baseline classifier: This model does not use attention technique.
Specifically, it only uses the last hidden state from the deep recurrent
processing layer, denoted $h_{T}$, as the intermediate representation for the
input protein segment.

  2. 2.

Baseline + attention-based classifier: This model includes the standard
attention technique proposed by Bahdanau et. al. [13]. Intermediate
representation for the input protein segment is the aggregation of the
weighted hidden states.

There are two aspects to evaluation: (i) the performance evaluation based on
the overall F1-scores, and (ii) the evaluation of the training process with
respect to learning and convergence rate. Learning is the evaluation of the
validation loss with respect to training loss. The lesser is the gap between
the training loss and validation loss, the more effective learning is, and
vice versa.

Moreover, we also evaluated the proposed $\lambda$-scaled attention network by
varying the embedding dimensions for the protein words (discussed in Section
4.3.3). Note that the final classification model (i.e., an MLP network),
already discussed in Section 3.2, is the same in all the experiments.

####  4.3.1 Superior overall performances

Here, overall performances (with respect to the F1-score) of the proposed
$\lambda$-scaled attention network are evaluated against both the baseline
models. Note that each of GRU and LSTM networks is explored as the deep
recurrent processing layer in the above classifiers. The comparison results
are shown in Tables I (with LSTM unit) and II (with GRU unit) for both the BP
and MF datasets.

The experiments using the LSTM as the deep recurrent processing layer (Table
I) demonstrate that the proposed $\lambda$-scaled attention network based
classifiers (with $\lambda$ = 1.0) are superior to both the baseline models.
For the BP dataset, with segment sizes 80, 100, and 120, the improvements
recorded over the baseline classifiers were +3.12%, +4.63%, and +5.64%,
respectively. The corresponding improvements over the baseline+attention-based
classifiers were +1.26%, +2.01%, and +1.74%, respectively.

Similarly, with the MF dataset (Table I), the observed improvements are as
follows: compared to the baseline classifiers and with segment sizes 80, 100,
and 120, improvements of +3.76%, +5.51%, and +7.54% were recorded,
respectively. The corresponding improvements with respect to the
baseline+attention-based classifiers are +2.48%, +4.67%, and +3.87%,
respectively. Moreover, the above observations are also true for all the
$\lambda$ values with the proposed $\lambda$-scaled attention network based
classifiers.

Further experiments using the GRU as the deep recurrent processing layer
(Table II) also suggest the superior results with the proposed
$\lambda$-scaled attention network based classifiers. These improvements hold
irrespective of the $\lambda$ value.

####  4.3.2 Efficient learning and faster convergence

The curves for the training and validation losses is used to study the effect
of the proposed $\lambda$-scaled attention technique on the training
efficiency and the convergence of the sub-sequence classifier. For a
comparative analysis, the curves showing the training and validation losses of
different classifiers are shown in Figures 5 (BP) and 6 (MF).

(a) Training Loss

(b) Validation Loss

Figure 6: Molecular Function: LSTM based training and validation loss curve.

A high gap between the training and validation loss curves for the baseline
classifiers (Figures 5 (BP) and 6 (MF)) demonstrate a poor learning. The
learning, however, is improved for the baseline+attention-based classifiers,
but losses are still high. A significant reduction in the losses are observed
for the proposed $\lambda$-scaled attention network based classifiers,
demonstrating the best learning among all.

Moreover, Figures 5 (BP) and 6 (MF) also highlight the faster convergence of
the proposed $\lambda$-scaled attention network based classifier, when
compared to both the baseline and baseline+attention based classifiers. The
convergence with the proposed $\lambda$-scaled attention network takes roughly
half the amount of time the other baseline methods are taking. The convergence
of the classifiers under the proposed $\lambda$-scaled attention network,
however, is delayed as the $\lambda$ value is decremented.

TABLE III: Classification Accuracy for the $\lambda$-scaled attention network based classifiers with $\lambda=1$ ($d$ = embedding dimension) | LSTM | GRU
---|---|---
Dataset |  Seg. Size |  d = 32 |  d = 48 |  d = 64 |  d = 80 |  d = 32 |  d = 48 |  d = 64 |  d = 80
Biological |  80 |  56.74 |  58.93 |  59.73 |  60.17 |  57.43 |  59.32 |  59.69 |  59.96
Process |  100 |  57.26 |  58.69 |  59.88 |  60.09 |  57.28 |  58.66 |  59.35 |  59.51
|  120 |  57.09 |  58.87 |  59.75 |  59.91 |  57.34 |  58.57 |  59.18 |  59.68
Molecular |  80 |  69.73 |  71.58 |  71.61 |  71.59 |  69.99 |  70.63 |  71.53 |  71.39
Function |  100 |  70.11 |  71.09 |  71.15 |  71.57 |  70.22 |  70.79 |  71.21 |  71.25
|  120 |  70.08 |  71.12 |  70.93 |  71.15 |  69.69 |  70.38 |  70.72 |  70.84
TABLE IV: Overall Classification Accuracy (* mark indicates proposed methods) |  |  Biological Process (%) |  Molecular Function (%)
---|---|---|---
S. No. |  Approach |  Recurrent cell /embedding dimension |  Average Recall |  Average Precision |  Average F1-Score |  Average Recall |  Average Precision |  Average F1-Score
1 |  MLDA [7] |  – |  49.42 |  52.61 |  49.27 |  58.29 |  60.20 |  57.91
2 |  ProtVecGen-100 [8] |  LSTM/32 |  53.15 |  54.42 |  52.11 |  63.93 |  65.25 |  63.39
3 |  ProtVecGen-Plus [8] |  LSTM/32 |  56.42 |  56.65 |  54.65 |  66.93 |  67.42 |  65.91
4 |  ProtVecGen-Plus + MLDA [8] |  – |  58.19 |  58.80 |  56.68 |  68.62 |  68.27 |  67.12
5 |  Proposed-100* |  LSTM/32 |  58.35 |  59.96 |  57.26 |  70.86 |  71.95 |  70.11
6 |  Proposed-100* |  GRU/32 |  58.54 |  59.95 |  57.28 |  71.01 |  72.12 |  70.22
7 |  Proposed-100* |  LSTM/64 |  60.59 |  63.17 |  59.88 |  71.85 |  73.07 |  71.15
8 |  Proposed-100* |  GRU/64 |  59.95 |  62.71 |  59.35 |  71.95 |  73.06 |  71.21

####  4.3.3 With respect to parameter embedding dimension

Word embedding creates dense representation for the protein words, where each
dimension represents a hidden attribute crucial to the meaning of the word. A
higher dimension is often advantageous creating a more meaningful
representations for the word, however, the increase in the dimension also
causes a significant increase in the number of trainable parameters. Here, the
proposed $\lambda$-scaled attention network based classifier is evaluated by
varying the embedding dimension of the protein words. While the default
embedding dimension (d) is $d=32$, the proposed $\lambda$-scaled attention
network based classifier (with $\lambda=1.0$) is also evaluated for the other
values of the $d=[48,64,80]$. The results are shown in Table III with both GRU
and LSTM as the deep recurrent layer. The number of trainable parameters when
$d$ = 32 is $\approx$ 5.20 million.

As shown in Table III, the average F1-score increases significantly upon
increasing the embedding dimension of the protein words from $d$ = 32 to $d$ =
48. The number of trainable parameters when $d$ = 48 is $\approx$ 7.76
million. With LSTM, for the biological process dataset, the margins of
improvement are +2.19%, +1.43%, and +1.78% with segment sizes 80, 100, and
120, respectively. The corresponding improvements for the molecular function
dataset are +1.85%, +0.98%, and +1.04%, respectively. Similar performance
enhancements were observed with the GRU as the deep recurrent layer. Further
increasing the embedding dimension (i.e., when $d$ = 64) only results in small
improvements in the average F1-score, while a marginal/no improvement is
observed by increasing the embedding dimension up to $d$ = 80.

We, therefore, choose to stick to the results obtained using $d$ = 64 (with
$\approx$ 10.33 million trainable parameters), which has significantly less
trainable parameters than $d$ = 80 (with $\approx$ 12.90 million parameters).

###  4.4 Performance comparison with work from the literature

In Table IV we have listed performances of the proposed method, denoted as
proposed-100* (LSTM/32), along with the existing work from the literature for
protein function prediction, where ‘LSTM’ and ‘32’ indicates recurrent cell
and embedding dimension, respectively. The work from the literature includes:
(i) MLDA approach [7], which is based on the tf-idf, (ii) ProtVecGen-100 [8],
which is a single-segment based approach, (iii) state-of-the-art ProtVecGen-
Plus [8], which is a multi-segment based approach, and (iv) ProtVecGen-
Plus+MLDA [8], which is a hybrid approach.

Additionally, their performances were also evaluated with respect to their
ability to deal with the protein sequences of different lengths. This is done
by splitting the test datasets into the following groups based on their
sequence lengths: G1: (0 – 200), G2: (201 – 500), G3: (501 – 1000), and G4:
$>$ 1000\. The range for each group specifies the permissible number of amino
acid residues in a sequence. These are shown in Figures 7a (BP) and 7b (MF).

As shown in Table IV, compared to the classifier trained on the MLDA features
[7], the classifiers trained with the proposed-100* (LSTM/32) features have
improved significantly: the margins of improvement in the average F1-scores
are +7.99% for the BP dataset and +12.20% for the MF dataset. The
proposed-100* (LSTM/32) features also showed huge improvements when validated
for the protein sequences in the groups G2, G3, and G4 (See Figures 7a (BP)
and 7b (MF)).

In addition, the proposed-100* (LSTM/32) features effectively outperformed
both the single-segment based ProtVecGen-100 [8] and a multi-segment based
ProtVecGen-Plus [8] features with the respective improvements of +5.15% and
+2.61% for the BP dataset. The corresponding improvements for the MF dataset
are +6.72% and +4.20%, respectively. Besides this, the evaluation of the
proposed-100* (LSTM/32) on protein sequences of different lengths, as shown in
Figures 7a (BP) and 7b (MF), clearly establishes the superiority of the
proposed features over them.

Finally, comparison with the hybrid ProtVecGen-Plus+MLDA [8] approach also
shows better results for the proposed-100* (LSTM/32) features by margins of
+0.58% (for BP) and +2.99% (for MF). The recall and precision values also
exhibit similar behavior. Furthermore, the proposed-100* (LSTM/32) based
classifier has again proven to be superior when validated against protein
sequences of different lengths, as shown in Figures 7a (BP) and 7b (MF). The
proposed-100* (GRU/32) features also exhibit similar improved performances.
The best overall results, however, are observed with the proposed-100*
(LSTM/64) and proposed-100* (GRU/62) features.

(a) Biological Process

(b) Molecular Function

Figure 7: Length-wise Evaluation: For the proposed methods, the configurations
are denoted in brackets.

##  5 Conclusion

In the paper, we explored the attention technique for the protein sequences
that help in recognizing key protein words from the sequences. These keywords
are useful in converting sequences to a meaningful representation for the
classification task. However, in general, the standard attention technique
suffers due to the weak semantics of the protein words. This poses a
difficulty for the standard attention technique to construct a good
representation for protein sequences. The novel challenges faced by the
standard attention technique addressed within this work include (i) vanishing
attention score problem and (ii) high variations in the attention
distribution.

In order to deal with the above challenges, we introduced a novel
$\lambda$-scaled attention technique to find a good representation of protein
sequences, where $\lambda$ is a parameter that assists in controlling the
training process. The proposed attention technique helps not only in solving
the vanishing attention score problem, but also helps in reducing the high
variations in the attention distribution. This has a significant effect
towards making the training more efficient and faster, when compared to the
standard attention technique. The proposed $\lambda$-scaled attention
technique is further used to build the $\lambda$-scaled attention network for
the protein function prediction task at the protein sub-sequence level.

Overall, for the protein function prediction task, the proposed
$\lambda$-scaled attention technique demonstrated a significant effect on the
predictive performances outperforming other existing approaches by substantial
margins. More importantly, performances across the protein sequences of
different lengths are consistently maintained.

## References

  * [1] Li, F., 2016. “Structure, function, and evolution of coronavirus spike proteins”. _Annual review of virology_ , 3, pp.237-261.
  * [2] Walls, A., Park, Y.J., Tortorici, M.A., Wall, A., Mcguire, A. and Veesler, D., 2020. “Structure, Function, and Antigenicity of the SARS-CoV-2 Spike Glycoprotein”. _Cell_ , 181(2), pp.281-292.
  * [3] Durmus Tekir, S., Cakir, T. and Ulgen, K., 2012. “Infection strategies of bacterial and viral pathogens through pathogen–human protein–protein interactions”. _Frontiers in Microbiology_ , 3, p.46.
  * [4] Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F. and Rost, B., 2019. “Modeling aspects of the language of life through transfer-learning protein sequences”. _BMC bioinformatics_ , 20(1), p.723.
  * [5] Metzker, Michael L. “Sequencing technologies—the next generation.” _Nature Reviews Genetics_ 11, no. 1 (2010): 31.
  * [6] Clark, Wyatt T., and Predrag Radivojac.“Analysis of protein function and its prediction from amino acid sequence.” _Proteins: Structure, Function, and Bioinformatics_ 79.7 (2011): 2086-2096.
  * [7] Wang, H., Yan, L., Huang, H., & Ding, C. (2017). “From Protein Sequence to Protein Function via Multi-Label Linear Discriminant Analysis”. _IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)_ , 14(3), 503-513.
  * [8] Ranjan, A., Fahad, M.S., Fernandez-Baca, D., Deepak, A. and Tripathi, S., 2019. “Deep Robust Framework for Protein Function Prediction using Variable-Length Protein Sequences”. _IEEE/ACM transactions on computational biology and bioinformatics_. doi: 10.1109/TCBB.2019.2911609.
  * [9] Hu, S., Ma, R. and Wang, H., 2019. “An improved deep learning method for predicting DNA-binding proteins based on contextual features in amino acid sequences”. _PloS one_ , 14(11), p.e0225317.
  * [10] Yi, H.C., You, Z.H., Zhou, X., Cheng, L., Li, X., Jiang, T.H. and Chen, Z.H., 2019. “ACP-DL: a deep learning long short-term memory model to predict anticancer peptides using high-efficiency feature representation”. _Molecular Therapy-Nucleic Acids_ , 17, pp.1-9.
  * [11] Cao, R., Freitas, C., Chan, L., Sun, M., Jiang, H. and Chen, Z., 2017. “ProLanGO: protein function prediction using neural machine translation based on a recurrent neural network”. _Molecules_ , 22(10), p.1732.
  * [12] Ben-Hur, A. and Brutlag, D., 2006. “Sequence motifs: highly predictive features of protein function”. In _Feature Extraction_ . Springer, Berlin, Heidelberg, pp. 625-645.
  * [13] Bahdanau, D., Cho, K. and Bengio, Y., 2014. “Neural machine translation by jointly learning to align and translate”. _arXiv preprint arXiv_ :1409.0473.
  * [14] Yang, Z., Yang, D., Dyer, C., He, X., Smola, A. and Hovy, E., 2016, June. “Hierarchical attention networks for document classification”. _In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies_ , pp. 1480-1489.
  * [15] Chen, H., Sun, M., Tu, C., Lin, Y. and Liu, Z., 2016, November. “Neural sentiment classification with user and product attention.” _In Proceedings of the 2016 conference on empirical methods in natural language processing_ (pp. 1650-1659).
  * [16] Zhou, X., Wan, X. and Xiao, J., 2016, November. Attention-based LSTM network for cross-lingual sentiment classification. In Proceedings of the 2016 conference on empirical methods in natural language processing (pp. 247-256).
  * [17] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R., 2014. “Dropout: a simple way to prevent neural networks from overfitting.” _The Journal of Machine Learning Research_ , 15(1), pp.1929-1958.
  * [18] Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H. and Bengio, Y., 2014, October. “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation”. _In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ (pp. 1724-1734).
  * [19] Hochreiter, S. and Schmidhuber, J., 1997.“Long short-term memory”. _Neural computation_ , 9(8), pp.1735-1780.
  * [20] Graves, A. and Schmidhuber, J., 2005. “Framewise phoneme classification with bidirectional LSTM and other neural network architectures.” _Neural Networks_ , 18(5-6), pp.602-610.
  * [21] Kingma, D.P. and Ba, J., 2014. “Adam: A method for stochastic optimization”. _arXiv preprint arXiv:1412.6980_.
  * [22] UniProt Consortium, 2014. “UniProt: a hub for protein information”. _Nucleic Acids Research_ , 43(D1), pp.D204-D212.
  * [23] Ashburner, M., Ball, C.A., Blake, J.A., Botstein, D., Butler, H., Cherry, J.M., Davis, A.P., Dolinski, K., Dwight, S.S., Eppig, J.T. and Harris, M.A., 2000. “Gene Ontology: tool for the unification of biology”. _Nature Genetics_ , 25(1), p.25.
  * [24] Sorower, M.S., 2010. “A literature survey on algorithms for multi-label learning”. _Oregon State University, Corvallis_ , 18, pp.1-25.
  * [25] Zhang, M.L. and Zhou, Z.H., 2014. “A review on multi-label learning algorithms”. _IEEE transactions on Knowledge and Data Engineering_ , 26(8), pp.1819-1837.


[a,1]Jenny List 11footnotetext: on behalf of the Linear Collider Collaboration

# Polarised Beams at Future e+e- Colliders

###### Abstract

Beam polarisation is an integral part of the physics case of future Linear
Colliders. In this contribution, important examples from Higgs coupling
measurements, top and electroweak physics at high energies, the $Z$ pole
program as well as from searches for production of new particles will be
reviewed. The full exploitation of its advantages requires the polarisation to
be known at the permille-level. The polarimetry concept which has been
developed for the ILC to achieve the required precision based on the
combination of Compton polarimeters, spin-tracking simulations and a global
analysis of collision data will be presented.

##  1 Introduction

Longitudinally polarised beams are a special feature of Linear Colliders and
are considered an integral part of the physics case, since, due to their
$SU(2)_{L}\times U(1)$ gauge structure, electroweak interactions are highly
sensitive to the chirality of fermions. The polarisation is defined as the
number asymmetry between right- and left-handed particles in a bunch,
$P=+80\%$ corresponding to $90\%$ right-handed and $10\%$ left-handed
particles. For CLIC, the electron beam, polarised to a degree of $80\%$, is
planned to collide with unpolarised positrons, as it was the case at the SLC.
For ILC also the positron beam is foreseen to be polarised, to a degree of
$30\%$, upgradable to $60\%$. With both its beams polarised, the ILC can thus
be considered as “four colliders in one”, depending on which of the four
configurations displayed in figure 1(a) is enriched by the chosen polarisation
signs. Figure 1(b) summarizes the four main benefits of polarised beams for
the physics program of a future $e^{+}e^{-}$ collider. The first three, namely
the enhancement of signals, the suppression of backgrounds and the provision
of additional observables for the analysis of the chiral properties of SM and
BSM processes have been addressed often in the literature, see e.g. ref. [1]
for an in-depth review and ref. [2] specifically for ILC at 250 GeV. The forth
one, the control of systematic uncertainties, is less widely known, but
equally important. It hinges upon the fact that for each process, the “wrong”
polarisation configuration yields a control sample with very little signal
content. In combination with the capability of the accelerator to revert the
sign of the polarisation on much faster time-scales than those of typical
nuisance effects (drifts of calibrations, changes of alignment, etc.), data
sets with different helicity configurations can be taken under otherwise
identical experimental conditions. This allows for a nearly perfect
cancellation of systematic effects between data sets with different
polarisation signs. Similar reasoning can be applied for theoretical
uncertainties, since the prediction of the effect of reversing the
polarisation sign decouples from many sources of theoretical uncertainties. In
other words, theory and detector systematics can be traded against
polarisation systematics, which could arise from the finite knowledge of the
actual beam polarisation values. These, however, can be very well controlled,
as will be described in Sec. 3. It should be noted that the reversal of the
positron polarisation provides the redundacy required for controling nuisance
effects on observables which depend on the electron polarisation.

(a)

(b)

Figure 1:  (a) Spin configurations in collisions of longitudinally polarised
electrons and positrons. (b) Overview of physics benefits of beam
polarisation.

##  2 Role of Polarisation in the Physics Case of Future $e^{+}e^{-}$
Colliders

The left-right asymmetry $A_{\mathrm{LR}}(ZH)$ of the cross section for
Higgsstrahlung, $e^{+}e^{-}\to ZH$, plays a very important role for
constraining physics beyond the SM via effective field theory-based
interpretations of Higgs precision measurements. Two of the relevant diagrams
at dimension-6, namely the SM diagram with a $Z$ exchange and the (hypothetic)
diagram with a photon exchange behave oppositely under spin reversal of the
incoming electron (positron): while the SM diagram flips its sign, the photon-
exchange diagram keeps its sign. Thus by measuring $A_{\mathrm{LR}}(ZH)$, the
degeneracy of the relevant EFT operators can be lifted [3]. Quantitatively
this additional information corresponds to a gain of about $2.5$ in
luminosity: In Higgs coupling determinations, e.g. $2$ ab-1 of polarised yield
the same level of statistical precision as $5$ ab-1 of unpolarised data [4].

For bread-and-butter electroweak precision measurements like $e^{+}e^{-}\to
f\bar{f}$, including $e^{+}e^{-}\to t\bar{t}$ at sufficiently high energies,
beam polarisation both adds important information and can be traded against
theory uncertainties. At the $Z$ pole, the asymmetry between the couplings of
the right-handed and left-handed electron to the $Z$, $A_{\mathrm{e}}$, is
directly obtained from the left-right asymmetry of the total cross section,
for which all visible $Z$ decay modes can be exploited. Compared to the
extraction of $A_{\tau}$ from final-state polarisation of $\tau$-leptons from
$Z\to\tau^{+}\tau^{-}$, this corresponds to an effective gain in luminosity of
a factor of $100$, rendering typical precisions at ILC’s GigaZ only a factor
of about $3$ less precise than at FCCee’s TeraZ, despite a factor $1000$
difference in assumed luminosity. As can be seen in figure 2(a), ILC would
improve upon current knowledge by factors between 5 and 100, depending on the
observable. Above the $Z$ pole, e.g. in the Higgs factory mode at $250$ GeV,
beam polarisation is crucial in order to disentangle the contributions from
$Z$ and photon exchange, similar to the Higgs example above. Furthermore it
turns out that the radiative corrections to the left-right-forward-backward
asymmetry $A^{f}_{\mathrm{FB,LR}}$ are seven times smaller than those on the
unpolarised forward-backward asymmetry $A^{f}_{\mathrm{FB}}$. Thus, polarised
beams also lead to a reduction of the impact of the residual theoretical
uncertainties from higher orders [5].

(a)

(b)

Figure 2:  (a) Expected precisions on various EWPOs from ILC’s polarised GigaZ
compared to current knowledge and FCCee’s unpolarised TeraZ [5]. (b)
Sensitivity reach in the energy scale for new physics $\Lambda$ for WIMP
production mediated by a vector operator at the 95% confidence level for
various assumptions on center-of-mass energy, integrated luminosity and beam
polarisation [6].

The power of combining data sets with different polarisation sign combinations
for suppressing the impact of systematic uncertainties has been studied for
instance in the search for WIMP production in the mono-photon channel. As
shown in figure 2(b), the sensitivity unpolarised data sets does hardly grow
with additional luminosity. With the mix of polarisations signs foreseen at
the ILC, already $2$ ab-1 outperform $10$ ab-1 unpolarised data. The other
speciality of Linear Colliders, namely to reach higher energies, is obviously
also very important for this kind of search, which probes energy scales up to
$3$ TeV at $\sqrt{s}=500$ GeV, and up to $5$ TeV at $\sqrt{s}=1$ TeV.

##  3 Polarimetry Concept at the ILC

In order to fully exploit the physics benefits of beam polarisation, the
actual polarisation values need to be known very precisely, at the level of
one permille. This is achieved for the ILC by combining the following items:
a) Fast, time-resolved measurements of two Compton polarimeters per beam,
$1.7$ km up- and $140$ m downstream of the $e^{+}e^{-}$ interaction point
reach a systematic precision of $0.25\%$, with the statistical uncertainty
becoming negligible on the timescale of a few seconds. The polarimeter design
and the R&D which demonstrates that the envisaged level of precision can be
reached is summarized in ref. [7]. b) The polarisation at the $e^{+}e^{-}$
interaction point can be obtained by interpolating the up- and downstream
polarimeter measurents. Spin tracking calculations have demonstrated the
ability to cross calibrate with a precision better than $0.1\%$ [8], dominated
by the relative alignment of orbit and spin vectors at the polarimeter
locations. The depolarisation in collision will be monitored by comparing the
up- and downstream measurements for colliding and non-colliding bunches. The
time-resolved measurements will be averaged, weighted according to the
instantaneous luminosity, to obtain the long-term averages relevant for the
interpretation of collision data. c) The absolute scale of this long-term
averages is in addition controlled by the collision data themselves, e.g. by
fitting the four average polarisation values (electron beam and positron beam
each with both signs) to cross sections and distributions which strongly
depend on the polarisation – either obtaining the polarisation values alone,
or by treating them as nuisance parameters in a fit which extracts them along-
side the actual physics observables. A prominent example often studied in the
past is $W^{+}W^{-}$ production, where it is well established that the beam
polarisations and charged triple gauge couplings can be extracted
simultaneously without a net loss in precision [9].

(a)

(b)

Figure 3:  (a) The absolute polarisation values do not need to be equal when
flipping the sign, since differences can be resolved with sufficient
precision. Magenta triangles: input values; colour scale: number of fit
outcomes on toy data sets. (b) The time evolution of the precision of the
average polarisations from collision data with operation time. The dashed
lines show the improvement when including the polarimeter measurements. Here,
the original ILC500 running scenario is assumed. In the staged scenario, only
the order of runs is changed. Both from ref. [10].

More recently, the power of combining all relevant $e^{+}e^{-}\to f\bar{f}$
and $e^{+}e^{-}\to
f\bar{f^{\prime}}f^{\prime\prime}\bar{f^{\prime\prime\prime}}$ channels has
been explored [10]. Figure 3(a) shows that in this approach even differences
in the absolute polarisation values for positive and negative sign can be
reproduced with high precision, reproducing the input values indicated by the
magenta triangles. Despite the power of the collision data, the additional
information from the polarimeters still plays an important role, as can be
seen from the time evolution of the precision in figure 3(b), where the full
lines are obtained from collision data only, while the dashed lines include
the polarimeter measurements.

With the combination of polarimeters, spin tracking and polarisation analysis
of the collision data, the system provides sufficient redundancy to detect
systematic biases in one of the methods, and to reach the goal of $0.1\%$
precision on the polarisation values [10]. It should be stressed that in
particular for the redundancy aspect, the positron polarisation and the
ability to reverse the polarisation inbetween bunch trains are essential.

##  4 Conclusions

Beam polarisation is an integral part of the physics case of future Linear
Colliders like the ILC. Important examples comprise Higgs coupling
measurements, top and electroweak physics at high energies, the $Z$ pole
program as well as searches for production of new particles. All these
require, however, the knowledge of the luminosity-weighted average
polarisation values at the permille-level. For the ILC, a detailed concept has
been developed to reach this goal. It is based on time-resolved measurements
of two Compton polarimeters per beam, spin-tracking across the 2 km of beam
delivery system and a global analysis of collision data. With their
combination, the polarisation is well under control, so that its physics
benefits can be fully exploited.

## References

  * [1] G. Moortgat-Pick et al. “The Role of polarized positrons and electrons in revealing fundamental interactions at the linear collider,” Phys. Rept. 460 (2008), 131-243 doi:10.1016/j.physrep.2007.12.003 [arXiv:hep-ph/0507011 [hep-ph]].
  * [2] K. Fujii et al. [LCC Physics Working Group], “The role of positron polarization for the inital $250$ GeV stage of the International Linear Collider,” [arXiv:1801.02840 [hep-ph]].
  * [3] T. Barklow, K. Fujii, S. Jung, R. Karl, J. List, T. Ogawa, M. E. Peskin and J. Tian, “Improved Formalism for Precision Higgs Coupling Fits,” Phys. Rev. D 97 (2018) no. 5, 053003 doi:10.1103/PhysRevD.97.053003 [arXiv:1708.08912 [hep-ph]].
  * [4] P. Bambade et al. [Linear Collider Collaboration], “The International Linear Collider: A Global Project,” [arXiv:1903.01629 [hep-ex]].
  * [5] K. Fujii et al. [LCC Physics Working Group], “Tests of the Standard Model at the International Linear Collider,” [arXiv:1908.11299 [hep-ex]].
  * [6] M. Habermehl, M. Berggren and J. List, “WIMP Dark Matter at the International Linear Collider,” Phys. Rev. D 101 (2020) no. 7, 075053 doi:10.1103/PhysRevD.101.075053 [arXiv:2001.03011 [hep-ex]].
  * [7] B. Vormwald, J. List and A. Vauth, “A calibration system for Compton polarimetry at $e^{+}e^{-}$ linear colliders,” JINST 11 (2016) no. 01, P01014 doi:10.1088/1748-0221/11/01/P01014 [arXiv:1509.03178 [physics.ins-det]].
  * [8] M. Beckmann, J. List, A. Vauth and B. Vormwald, “Spin Transport and Polarimetry in the Beam Delivery System of the International Linear Collider,” JINST 9 (2014), P07003 doi:10.1088/1748-0221/9/07/P07003 [arXiv:1405.2156 [physics.acc-ph]].
  * [9] I. Marchesini, “Triple gauge couplings and polarization at the ILC and leakage in a highly granular calorimeter,” PhD Thesis, Hamburg University 2011, DESY-THESIS-2011-044, https://bib-pubdb1.desy.de/record/94888.
  * [10] R. Karl, “From the Machine-Detector Interface to Electroweak Precision Measurements at the ILC — Beam-Gas Background, Beam Polarization and Triple Gauge Couplings,” PhD Thesis, Hamburg University 2019, doi:10.3204/PUBDB-2019-03013.


Distributional reinforcement learning (DRL) enhances the understanding of the effects of the randomness  in the environment by letting agents learn the distribution of a random return, rather than its expected value as in standard RL.

At the same time, a main challenge in DRL is that policy evaluation in DRL typically relies on the representation of the return distribution, which needs to be carefully designed.

In this paper, we address this challenge for a special class of DRL problems that rely on discounted linear quadratic regulator (LQR) for control, advocating for a new distributional approach to LQR, which we call distributional LQR. Specifically, we provide  a closed-form expression of the distribution of the random return which, remarkably, is applicable to all exogenous disturbances on the dynamics, as long as they are independent and identically distributed (i.i.d.).

While the proposed exact return distribution consists of infinitely many random variables, we show that this distribution can be approximated by a finite number of random variables, and the associated approximation error can be analytically bounded under mild assumptions.

Using the approximate return distribution, we propose a zeroth-order policy gradient algorithm for risk-averse LQR using the Conditional Value at Risk (CVaR) as a measure of risk.

Numerical experiments are provided to illustrate our theoretical results.

Distributional LQR, distributional RL, policy evaluation, risk-averse control

This work is supported in part by the Knut and Alice Wallenberg Foundation, the Swedish Strategic Research Foundation, the Swedish Research Council,  AFOSR under award #FA9550-19-1-0169, and  NSF under award CNS-1932011.

urlstyle[Barth-Maron et al., 2018]
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan,
Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap.
Distributed distributional deterministic policy gradients.
arXiv preprint arXiv:1804.08617, 2018.

[Bellemare et al., 2017]
Marc G Bellemare, Will Dabney, and Rémi Munos.
A distributional perspective on reinforcement learning.
In Proceedings of International Conference on Machine
Learning, pages 449–458. PMLR, 2017.

[Bellemare et al., 2023]
Marc G. Bellemare, Will Dabney, and Mark Rowland.
Distributional Reinforcement Learning.
MIT Press, 2023.

[Chapman and Lessard, 2021]
Margaret P Chapman and Laurent Lessard.
Toward a scalable upper bound for a CVaR-LQ problem.
IEEE Control Systems Letters, 6:0 920–925, 2021.

[Dabney et al., 2018]
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos.
Implicit quantile networks for distributional reinforcement learning.
In Proceedings of International Conference on Machine
Learning, pages 1096–1105. PMLR, 2018a.

[Dabney et al., 2018]
Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos.
Distributional reinforcement learning with quantile regression.
In Proceedings of AAAI Conference on Artificial Intelligence,
volume 32, 2018b.

[Dean et al., 2020]
Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu.
On the sample complexity of the linear quadratic regulator.
Foundations of Computational Mathematics, 200
(4):0 633–679, 2020.

[Fazel et al., 2018]
Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi.
Global convergence of policy gradient methods for the linear
quadratic regulator.
In Proceedings of International Conference on Machine
Learning, pages 1467–1476. PMLR, 2018.

[Kim and Yang, 2021]
Kihyun Kim and Insoon Yang.
Distributional robustness in minimax linear quadratic control with
Wasserstein distance.
arXiv preprint arXiv:2102.12715, 2021.

[Kishida and Cetinkaya, 2022]
Masako Kishida and Ahmet Cetinkaya.
Risk-aware linear quadratic control using conditional value-at-risk.
IEEE Transactions on Automatic Control, 2022.

[Li et al., 2021]
Yingying Li, Yujie Tang, Runyu Zhang, and Na Li.
Distributed reinforcement learning for decentralized linear quadratic
control: A derivative-free policy optimization approach.
IEEE Transactions on Automatic Control, 2021.

[Malik et al., 2019]
Dhruv Malik, Ashwin Pananjady, Kush Bhatia, Koulik Khamaru, Peter Bartlett, and
Martin Wainwright.
Derivative-free methods for policy optimization: guarantees for
linear quadratic systems.
In Proceedings of 22nd International Conference on Artificial
Intelligence and Statistics, pages 2916–2925. PMLR, 2019.

[Rockafellar et al., 2000]
R Tyrrell Rockafellar, Stanislav Uryasev, et al.
Optimization of conditional value-at-risk.
Journal of Risk, 2:0 21–42, 2000.

[Singh et al., 2020]
Rahul Singh, Qinsheng Zhang, and Yongxin Chen.
Improving robustness via risk averse distributional reinforcement
In Proceedings of Learning for Dynamics and Control
Conference, pages 958–968. PMLR, 2020.

[Singh et al., 2022]
Rahul Singh, Keuntaek Lee, and Yongxin Chen.
Sample-based distributional policy gradient.
In Proceedings of Learning for Dynamics and Control
Conference, pages 676–688. PMLR, 2022.

[Tang et al., 2019]
Yichuan Charlie Tang, Jian Zhang, and Ruslan Salakhutdinov.
Worst case policy gradients.
arXiv preprint arXiv:1911.03618, 2019.

[Tsiamis et al., 2021]
Anastasios Tsiamis, Dionysios S Kalogerias, Alejandro Ribeiro, and George J
Linear quadratic control with risk constraints.
arXiv preprint arXiv:2112.07564, 2021.

[Tu and Recht, 2018]
Stephen Tu and Benjamin Recht.
Least-squares temporal difference learning for the linear quadratic
In Proceedings of International Conference on Machine
Learning, pages 5005–5014. PMLR, 2018.

[Van Parys et al., 2015]
Bart PG Van Parys, Daniel Kuhn, Paul J Goulart, and Manfred Morari.
Distributionally robust control of constrained stochastic systems.
IEEE Transactions on Automatic Control, 610
(2):0 430–442, 2015.

[Yaghmaie et al., 2022]
Farnaz Adib Yaghmaie, Fredrik Gustafsson, and Lennart Ljung.
Linear quadratic control using model-free reinforcement learning.
IEEE Transactions on Automatic Control, 2022.

[Zhang et al., 2022]
Yan Zhang, Yi Zhou, Kaiyi Ji, and Michael M Zavlanos.
A new one-point residual-feedback oracle for black-box learning and
Automatica, 136:0 110006, 2022.

[Zheng et al., 2021]
Yang Zheng, Luca Furieri, Maryam Kamgarpour, and Na Li.
Sample complexity of linear quadratic Gaussian (LQG) control for
output feedback systems.
In Proceedings of Learning for Dynamics and Control
Conference, pages 559–570. PMLR, 2021.