# Sparsity and Sentence Structure
in Encoder-Decoder Attention of Summarization Systems

Potsawee Manakul and Mark J. F. Gales
Department of Engineering, University of Cambridge
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Transformer models have achieved state-of-the-art results in a wide range of
NLP tasks including summarization. Training and inference using large
transformer models can be computationally expensive. Previous work has focused
on one important bottleneck, the quadratic self-attention mechanism in the
encoder. Modified encoder architectures such as LED or LoBART use local
attention patterns to address this problem for summarization. In contrast,
this work focuses on the transformer’s encoder-decoder attention mechanism.
The cost of this attention becomes more significant in inference or training
approaches that require model-generated histories. First, we examine the
complexity of the encoder-decoder attention. We demonstrate empirically that
there is a sparse sentence structure in document summarization that can be
exploited by constraining the attention mechanism to a subset of input
sentences, whilst maintaining system performance. Second, we propose a
modified architecture that selects the subset of sentences to constrain the
encoder-decoder attention. Experiments are carried out on abstractive
summarization tasks, including CNN/DailyMail, XSum, Spotify Podcast, and
arXiv.111Our code is available at
https://github.com/potsawee/encdec_attn_sparse.

##  1 Introduction

The Transformer architecture Vaswani et al. (2017) with large-scale pre-
training has become the de-facto approach for a wide range of NLP tasks, from
classification Devlin et al. (2019) to seq2seq Raffel et al. (2020). Training
and inference using large transformer models can be computationally expensive
because the self-attention’s time and memory grow quadratically with sequence
length. Hence, there has been significant interest in efficient transformer
architectures. A number of approaches have been proposed to tackle the
quadratic complexity, and a comprehensive survey on efficient transformers has
been compiled in Tay et al. (2020). Most existing approaches are developed for
encoder-only architectures. For seq2seq tasks, efficient models such as
BigBird Zaheer et al. (2020) or LED Beltagy et al. (2020) consist of an
efficient encoder with the vanilla decoder. For long-document summarization,
this combination has been shown effective because the major bottleneck is the
encoder self-attention Manakul and Gales (2021). The attention mechanisms in
the decoder consist of self-attention and encoder-decoder attention.
Techniques such as local attention are applicable to self-attention in both
the encoder and decoder, while this work focuses on the encoder-decoder
attention.

When humans produce a summary, the information conveyed by each word/part in
the summary is likely drawn from some key sentences in the original document.
Inspired by this, we hypothesize that if the encoder-decoder attention is
constrained dynamically to salient sentences, the computation cost will be
reduced. For instance, sentence-level structures for the encoder-decoder
attention have been shown effective in the traditional RNN encoder-decoder
attention Cohan et al. (2018); Li et al. (2019); Manakul et al. (2020)

In this work, first, we compare the decoder’s cost in the training and
inference stages. We study the sparsity of the encoder-decoder attention in a
common transformer-based abstractive summarization model. An approximation
method to exploit this sparsity is described, and an empirical upper bound
performance is given. Second, we propose a modified decoder architecture that
can dynamically select salient input sentences to constrain the encoder-
decoder attention without having to compute complete attention at inference
time. Techniques to train our proposed model are described, and compared to
the full attention baseline performance and empirical upper bound.

##  2 Models and Data

Vanilla Transformers. We use BART Lewis et al. (2020) and local-attention BART
(LoBART) Manakul and Gales (2021) as our base models. BART’s maximum input
length is 1024, while that of LoBART is 4096 with attention width of 1024.
BART is fine-tuned to CNN/DailyMail and XSum, and LoBART is fine-tuned to
Podcast and arXiv.

Data. CNN/DailyMail Hermann et al. (2015) and XSum Narayan et al. (2018) are
used with BART, while long-document arXiv Cohan et al. (2018) and Spotify
Podcast Clifton et al. (2020) are used with LoBART. More details about models,
data, and training are provided in Appendix A.

##  3 Attention in the Transformer

Time and memory are dominated by the encoder self-attention, and models such
as LoBART adopt local attention in its encoder to mitigate this bottleneck,
while keeping the original decoder Manakul and Gales (2021). Training is fast
because attention is highly parallelizable. However, during inference, the
decoder uses its histories, becoming less parallelizable. To understand when
the decoder might become a bottleneck, we fix the input length $N$ and measure
the computational time as a function of the target length $M$:

${\tt time}=\bar{c}_{1}+\bar{c}_{2}M+\bar{c}_{3}M^{2}$ (1)

in three operating modes: i) Forward+Backward, e.g. at training time; ii)
Forward only, e.g. forward-pass where the input to the decoder is provided in
advance; iii) Inference, e.g. the decoder using its own back histories as the
input.

Through a curve-fitting method, the results in Table 1 show that the relative
decoder cost during inference is almost one order of magnitude larger than
that during training, e.g. forward+backward or forward only. More details are
provided in Appendix B, where we also show that the encoder-decoder attention
cost is greater than the decoder self-attention cost. Therefore, this work
will focus on the encoder-decoder attention.

Mode |  $\bar{c}_{2}/\bar{c}_{1}$ ($10^{-3}$) |  $\bar{c}_{3}/\bar{c}_{1}$ ($10^{-6}$)
---|---|---
Forward+Backward | 1.08 | 0.17
Forward only | 1.14 | 0.25
Inference | 9.96 | 1.30
Table 1: Empirical computational time as a function of the target length $M$
where $\bar{c}_{1},\bar{c}_{2},\bar{c}_{3}$ are the coefficients in Eq. 1. The
analysis is based on BART from Wolf et al. (2020) and the input length is
1024.

###  3.1 Encoder-Decoder Attention

Let $M$ = the summary length, $N$ = the input length, $N_{1}$ = #sentences,
and $N_{2}$ = the average number of words in a sentence, e.g. $N=N_{1}N_{2}$.
The standard encoder-decoder attention in Eq. 2 (scaling factor omitted) where
$\mathbf{Q}\in\mathcal{R}^{M\times D}$ and
$\mathbf{K},\mathbf{V}\in\mathcal{R}^{N\times D}$ has the complexity:
$\mathcal{O}(MN)=\mathcal{O}(MN_{1}N_{2})$. Note that we fix the
representation dimension $D$, so $D$ is omitted in our complexity notation.

$\mathbf{A}=\text{softmax}(\mathbf{Q}\mathbf{K}^{T})\mathbf{V}$ (2)

If the attention is concentrated on some $r$ sentences,222Motivated by the
observations shown in Appendix E. by selecting appropriate $r$, the speed of
the encoder-decoder attention can be improved by a factor of $N_{1}/r$ in
average. This is equivalent to:

${\mathbf{A}}\approx\hat{\mathbf{A}}=\text{softmax}(\mathbf{Q}\hat{\mathbf{K}}^{T})\hat{\mathbf{V}}$
(3)

where $\hat{\mathbf{K}},\hat{\mathbf{V}}\in\mathcal{R}^{rN_{2}\times D}$,
resulting in $\mathcal{O}(MrN_{2})$.

###  3.2 Sparsity of Encoder-Decoder Attention

Let the subscript $(i,j)$ denote the position of the $j$-th word in the $i$-th
input sentence, e.g.
$\mathbf{K}=[\underbrace{\mathbf{k}_{1,1},\mathbf{k}_{1,2},\mathbf{k}_{1,J_{1}}}_{\text{sent}1},...,\underbrace{\mathbf{k}_{i,1},\mathbf{k}_{i,J_{i}}}_{\text{sent}i},...,\underbrace{\mathbf{k}_{N_{1},1},\mathbf{k}_{N_{1},J_{N_{1}}}}_{\text{sent}N_{1}}]$.
At inference time, the outputs are generated sequentially:
$\mathbf{a}_{m}$$=$$\text{softmax}(\mathbf{q}_{m}\mathbf{K}^{T})\mathbf{V}$,
so $r$ sentences can be determined independently for each $\mathbf{q}_{m}$.
Consider the following sum of attention weights as the saliency at decoding
step $m$ of sentence $i$:333We discuss the details of multi-head attention on
$\alpha^{\tt s}_{m,i}$ and other operations such as entropy in Appendix A.4

$\alpha^{\tt
s}_{m,i}=\frac{1}{Z_{m}}\sum_{j=1}^{J_{i}}\exp(\mathbf{q}_{m}\cdot\mathbf{k}_{i,j})$
(4)

where $Z_{m}=\sum_{\forall i^{\prime}}\sum_{\forall
j^{\prime}}\exp(\mathbf{q}_{m}\cdot\mathbf{k}_{i^{\prime},j^{\prime}})$. We
then compute $\sum_{i}\alpha^{\tt s}_{m,i}$ up to $r$ sentences ranked by
$\alpha^{\tt s}_{m,i}$. The results in Fig. 1(a) show that $r$=25 is required
to achieve the sum of attention weights at 90%. In addition to the vanilla
model, we can fine-tune BART explicitly to make the attention sparse using:

$\mathcal{L}_{\tt A}=\mathcal{L}_{{\tt xent}}+\gamma\mathcal{L}_{{\tt
sparse}}$ (5)

where $\mathcal{L}_{{\tt xent}}$ is the teacher-forced cross entropy loss,
$\mathcal{L}_{{\tt
sparse}}=\frac{1}{M}\sum_{m=1}^{M}\text{H}(\boldsymbol{\alpha}^{{\tt
s}}_{m})$, and entropy $\text{H}(\boldsymbol{\alpha}^{{\tt
s}}_{m})=-\sum_{i=1}^{N_{1}}{\alpha}^{{\tt s}}_{m,i}\log{\alpha}^{{\tt
s}}_{m,i}$. We show in Fig. 1(b) that the fine-tuned models ($\gamma$=0.1 &
$\gamma$=1.0) retain close to 100% of attention weights for small $r$.
Subsequently, we investigate how selecting $r$ sentences impacts the
summarization performance.

(a) Layers 1,4,8,12

(b) $\mathcal{L}_{\tt A}$-tuned (Layer1)

Figure 1: The sum of attention weights against the number of retained
sentences ($r$) evaluated on CNNDM.

To obtain an empirical upper bound performance of Eq. 3, for each
$\mathbf{q}_{m}$, we can get ideal $\mathbf{k},\mathbf{v}$ corresponding to
the top $r$ sentences ranked by $\alpha^{\tt s}_{m,i}$:

$\mathcal{I}^{r}_{m}=[(i,j)\hskip 3.55658pt\text{s.t.}\hskip
3.55658pti\in\text{top-}r(\alpha^{\tt s}_{m,i})]$ (6)

$\hat{\mathbf{K}}_{m}=[\mathbf{k}_{i,j}:(i,j)\in\mathcal{I}^{r}_{m}]$, and the
same method is applied to obtain $\hat{\mathbf{V}}_{m}$.

System | $r$ | $\mathcal{I}^{r}_{m}$ | R1 | R2 | RL
---|---|---|---|---|---
Vanilla | All | N/A | 44.03 | 20.92 | 40.99
($\gamma=0.0$) | 5 | Ideal | 43.94 | 20.82 | 40.81
| 5 | Random | 39.06 | 14.32 | 36.07
$\gamma=0.1$ | 5 | Ideal | 44.22 | 21.01 | 41.19
$\gamma=1.0$ | 5 | Ideal | 43.61 | 20.46 | 40.60
Table 2: Sparsity and Selection ($\mathcal{I}^{r}_{m}$) on CNNDM.

The results in Table 2 show that:

  * •

For the vanilla model, despite the sum of attention weights being around 50%
at $r$=5 (Fig. 1(a)), the model is sufficiently sparse, and constraining to
$r$ ideal sentences ($\text{All}\rightarrow$
$\mathcal{I}^{r,\text{Ideal}}_{m}$) results in a small performance
degradation.

  * •

Forcing for sparsity (Fig.1(b)) does not yield a significant performance
improvement; but this forcing also makes the model more sensitive to random
selection (results in Appendix C).

Thus, for summarization, there is an observable sparsity, which allows us to
reduce the cost of encoder-decoder attention with a minimal degradation. Next,
we investigate how to build an efficient form of approximator to obtain
salient sentences.

(a) BART(1k) & CNNDM

(b) BART(1k) & XSum

(c) LoBART(4k) & Podcast

(d) LoBART(4k) & arXiv

Figure 2: Performance (ROUGE-1) of BART & LoBART. The integrated training is
based on $\mathcal{I}^{r,\text{Apx}}_{m}$.

##  4 Sentence-Level Structure for Encoder-Decoder Attention

In Section 3.1, we use ideal selection $\mathcal{I}^{r}_{m}$ (Eq. 6), which
requires computing $\alpha^{\tt s}_{m,i}$ (Eq. 4) using all input words. This
process cannot make the decoder more efficient. By exploiting the sentence
structure in the document, we propose the following partition for the
sentence-level attention score (Eq. 4) to allow a compact approximation:

${\alpha}^{\tt s}_{m,i}\approx\tilde{\alpha}^{\tt
s}_{m,i}=\text{softmax}\left(f_{1}({\mathbf{q}_{m}})\cdot
f_{2}({\mathbf{k}_{i,1},...,\mathbf{k}_{i,J_{i}}})\right)$

(7)

where $\sum_{i=1}^{N_{1}}\tilde{\alpha}^{\tt s}_{m,i}=1.0$. Essentially, we
modify the standard encoder-decoder attention such that it performs sentence
selection based on $\tilde{\alpha}^{\tt s}_{m,i}$ (Eq. 7) and computes subset
attention $\hat{\mathbf{A}}$ (Eq. 3).

###  4.1 Complexity of Modified Attention

The modified encoder-decoder attention consists of two components: i)
sentence-level attention over $N_{1}$ sentences; ii) word-level attention over
$rN_{2}$ words. Let $p$ denote a unit of matrix multiplication cost and $q$
denote a unit of softmax cost. The costs associated with attention are:

  1. i)

Sentence-level (Eq.7): $pMN_{1}D+qMN_{1}$

  2. ii)

Word-level (Eq.3): $2pMrN_{2}D+qMrN_{2}$

The additional cost associated with the sentence-level representation on the
encoder side grows with the input length $N$=$N_{1}N_{2}$. Thus, as opposed to
$\mathcal{O}(MN_{1}N_{2})$ in the case of vanilla encoder-decoder attention,
the overall complexity of the modified attention is
$\mathcal{O}(MN_{1}+k_{w}MrN_{2}+k_{e}N_{1}N_{2})$, where
$k_{w}\approx\frac{2pD+q}{pD+q}$ and $k_{e}$ depends on the exact form of
sentence-level representation computation.

###  4.2 Model-based Neural Approximator

To utilize the simple partition and sentence-level structure in Eq. 7, we use
a linear mapping for $f_{1}$ and a bidirectional RNN for $f_{2}$ as follows:

$\displaystyle f_{1}(\mathbf{q}_{m})$
$\displaystyle=\mathbf{q}_{m}\mathbf{W}^{\tt Q}$ (8) $\displaystyle
f_{2}(\mathbf{k}_{i,1},...,\mathbf{k}_{i,J_{i}})$
$\displaystyle=\mathbf{y}_{i}\mathbf{W}^{\tt K}$ (9)
$\displaystyle\mathbf{y}_{i}=\text{RNN}(\mathbf{k}$
${}_{i,1},...,\mathbf{k}_{i,J_{i}})$ (10)

As illustrated in Fig. 3, the base transformer model is extended by augmenting
two layers: i) sentence-level encoder-decoder attention computing
$\tilde{\alpha}^{\tt s}_{m,i}$ in Eq. 7; ii) sentence encoder computing the
sentence-level representation in Eq. 10. The details about model parameters
are provided in Appendix A.

Figure 3: Modified architecture with model-based approximator where the base
model is BART/LoBART. Model-based neural approximator is shown in orange.

###  4.3 KL Loss and Integrated Training

Let $\boldsymbol{\theta}_{\text{dec}}$ denote the original decoder, and
$\tilde{\boldsymbol{\theta}}$ denote the neural approximator. We train
$\tilde{\boldsymbol{\theta}}$ by minimizing:

$\mathcal{L}_{\tt
KL}=\frac{1}{M}\sum_{m=1}^{M}\text{KL}\left(\boldsymbol{\alpha}_{m}^{{\tt
s}}||\boldsymbol{\tilde{\alpha}}_{m}^{{\tt s}}\right)\vspace{-0.05cm}$ (11)

where $\text{KL}(.)=\sum_{i=1}^{N_{1}}{\alpha}_{m,i}^{{\tt
s}}\log({{\alpha}_{m,i}^{{\tt s}}}/{{\tilde{\alpha}}_{m,i}^{{\tt s}}})$. In
addition, we can integrate $\boldsymbol{\theta}_{\text{dec}}$ in the training
process. With the teacher-forced cross entropy loss $\mathcal{L}_{\tt xent}$,
we define the integrated training loss:

$\mathcal{L}_{\tt I}=\mathcal{L}_{\tt{xent}}+\lambda\mathcal{L}_{\tt{KL}}$
(12)

We cannot optimize $\mathcal{L}_{\tt I}$ in an end-to-end fashion because the
top-$r$ operation in Eq. 6 is not differentiable. Hence, we interleave the
training, i.e. update $\boldsymbol{\theta}_{\text{dec}}$ at fixed
$\tilde{\boldsymbol{\theta}}$ and update $\tilde{\boldsymbol{\theta}}$ using
$\mathcal{L}_{\tt KL}$ only:

$\Delta\boldsymbol{\theta}_{\text{dec}}=\nabla_{\boldsymbol{\theta}_{\text{dec}}}\mathcal{L}_{\tt
I}|_{\tilde{\boldsymbol{\theta}}}=\nabla_{\boldsymbol{\theta}_{\text{dec}}}\mathcal{L}_{\tt{xent}}|_{\tilde{\boldsymbol{\theta}}}+\lambda\nabla_{\boldsymbol{\theta}_{\text{dec}}}\mathcal{L}_{\tt{KL}}|_{\tilde{\boldsymbol{\theta}}}$

(13)
$\Delta\tilde{\boldsymbol{\theta}}=\nabla_{\tilde{\boldsymbol{\theta}}}\mathcal{L}_{\tt
I}=\cancelto{0}{\nabla_{\tilde{\boldsymbol{\theta}}}\mathcal{L}_{\tt{xent}}}+\lambda\nabla_{\tilde{\boldsymbol{\theta}}}\mathcal{L}_{\tt{KL}}$
(14)

Because during training, we compute both ${\alpha}^{\tt s}_{m,i}$ (ideal) and
$\tilde{\alpha}^{\tt s}_{m,i}$ (approx), we can use either in the top-$r$
selection. Also, inspired by scheduled sampling Bengio et al. (2015), we try
mixing them: ${\alpha}^{\tt s}_{m,i}$ with probability
$1-\frac{\text{step}}{\text{epoch\\_size}}$, otherwise $\tilde{\alpha}^{\tt
s}_{m,i}$.

###  4.4 System Performance

System | Train | Inference | R1 | R2 | RL
---|---|---|---|---|---
Vanilla | ✗ | $\mathcal{I}_{m}^{r\text{,Rnd}}$ | 39.06 | 14.32 | 36.07
Vanilla | ✗ | $\mathcal{I}_{m}^{r\text{,Idl}}$ | 43.94 | 20.82 | 40.81
KL-only | $\mathcal{L}_{\tt KL}$ | $\mathcal{I}_{m}^{r\text{,Apx}}$ | 43.02 | 20.02 | 39.89
Int-Idl |  $\mathcal{L}_{\tt I}$($\mathcal{I}_{m}^{r\text{,Idl}}$) | $\mathcal{I}_{m}^{r\text{,Apx}}$ | 43.03 | 20.04 | 40.05
Int-Apx |  $\mathcal{L}_{\tt I}$($\mathcal{I}_{m}^{r\text{,Apx}}$) | $\mathcal{I}_{m}^{r\text{,Apx}}$ | 43.72 | 20.40 | 40.70
Int-Mix |  $\mathcal{L}_{\tt I}$(Mix) | $\mathcal{I}_{m}^{r\text{,Apx}}$ | 43.31 | 20.21 | 40.35
Table 3: Performance on CNNDM where $r$=5 for both training and inference. KL-
only = $\tilde{\boldsymbol{\theta}}$ trained on $\mathcal{L}_{\tt KL}$; Int =
($\boldsymbol{\theta}_{\text{dec}}$&$\tilde{\boldsymbol{\theta}}$) trained on
$\mathcal{L}_{\tt I}$. Rnd=random, Idl=ideal, Apx=approximation,
Mix=scheduled(Idl/Apx).

In Table 3, we provide two vanilla baselines: random
($\mathcal{I}_{m}^{r\text{,Rnd}}$) obtained by random selection; ideal
($\mathcal{I}_{m}^{r\text{,Idl}}$) obtained by Eq. 6. The results show that
the KL-only system clearly outperforms the random selection baseline, and the
performance degradation of the KL-only system can be reduced by our integrated
training. The results verify the effectiveness of our modified decoder that
attends to a subset of sentences. Also, Table 3 shows that it is best to use
$\mathcal{I}_{m}^{r\text{,Apx}}$ as reference in integrated training. This
result is likely because we initialized integrated training from the KL-only
model. In addition, we apply the modified architecture to BART trained on XSum
($\leq$1k words), and to LoBART trained on Podcast and arXiv ($\leq$4k words).
The results in Fig. 2 confirm that the performance of our proposed method
converges to that of the full attention baseline across all models and
datasets.

In addition, $r^{*}\approx\hskip 4.0pt$5,10,30,30, respectively.444$r^{*}$
denotes $r$ at which the ideal selection system’s performance plateaus/reaches
the full attn baseline’s performance. Although XSum has fewer sentences in
average compared to CNNDM, $r^{*}_{\text{XSum}}$$>$$r^{*}_{\text{CNNDM}}$ as
XSum is more abstractive. For longer summarization tasks as shown by Podcast
and arXiv, the performance degradation appears larger, meaning that the task
of constraining to salient sentences in longer tasks is more challenging, and
larger $r$ is required.

#### Sensitivity of $r$ in Integrated Training

We train BART in three settings: $r^{\text{train}}$=2,5,10, and we show the
performance level w.r.t. the model with $r^{\text{train}}$=5 in Fig. 4. The
results show that setting $r^{\text{train}}$ beyond $r^{*}$ is not necessarily
beneficial as shown by the model with $r^{\text{train}}$=10 in the CNNDM
result, and it is best to set $r^{\text{train}}$ close to
$r^{\text{inference}}$.

(a) CNNDM

(b) XSum

Figure 4: $\Delta$R1 (Y-axis) against $r$ at inference (X-axis).

###  4.5 Further Discussion on Sentence-Level Encoder-Decoder Attention

The results in Section 4.4 demonstrate empirically that a neural network can
predict sparsity, therefore, allowing sentence selection. Our novel framework
requires an addition of modules to the original attention mechanism, and the
real gain in speed will depend on the balance of the sparsity against the
computational cost of the additional modules. Consequently, the challenge is
to make these additional modules highly efficient. Because the particular
network realization selected in this paper is to show the feasibility of our
framework, its limitations and possible improvements are discussed as follows.

#### Limitations and Possible Improvements

The model choice of using RNN for the sentence encoder in Eq. 10 leads to a
large additional computational cost (specifically large $k_{e}$) because the
computational cost of RNN grows with $N_{1}N_{2}D^{2}$. Because the goal is to
obtain a sentence-level representation, there is an opportunity to replace RNN
by a hierarchical attention that runs over sentences, which could instead lead
to a computational cost that grows with $N_{1}N_{2}D$. Additional sentence-
level query and key mappings in Eq. 8 and Eq. 9 also incur a large
computational cost.

#### Model-free Approximator

Lastly, we re-visit the sentence-level attention in Eq. 4, which have been
approximated by Eq. 7 and the model-based approximator. It is a challenge to
attempt via a model-free algebraic approximation, which does not require any
training and has little additional inference-time cost. We examined various
forms, and we present one model-free approach as well as experimental results
in Appendix D, but the current form has worse summarization performance than
our model-based approach.

##  5 Related Work

The discrepancy between low/moderate attention weight sparsity and good sparse
approximation could be because a considerable amount of the attention weight
is assigned to special tokens, e.g. ‘.’ in all sentences, but their vector
norm is small, which was observed in Kobayashi et al. (2020).

The sparse attention (Eq. 3) with ideal selection (Eq. 6) can be considered as
content selection, which has been shown to improve summarization Gehrmann et
al. (2018); Hsu et al. (2018). Recently, head-wise masks are applied to
encoder-decoder attention at inference time, and a performance improvement is
reported Cao and Wang (2021).

Voita et al. (2019) observed that heads are redundant, and Clark et al. (2019)
found that a head in BERT rarely attends to several consecutive tokens. Based
on these, Huang et al. (2021) applies a stride pattern in the encoder-decoder
attention, reducing its cost by a factor of the stride size, and this method
is likely complementary to our work.

##  6 Conclusion

We show that the computational cost of the transformer decoder becomes more
significant at inference time. Towards reducing this cost, first, we show that
there is sparsity in the encoder-decoder attention that allows us to reduce
the computational cost with a minimal degradation. Second, we partition the
sentence-level attention score, and we augment the standard decoder by adding
a neural network to approximate the attention over sentences, allowing
sentence selection. We show that the summarization performance of our approach
converges to that of the full attention baseline, while switching the
complexity from $\mathcal{O}(MN_{1}N_{2})$ to
$\mathcal{O}(MN_{1}+k_{w}MrN_{2}+k_{e}N_{1}N_{2})$.

## Acknowledgments

This paper reports on research supported by ALTA institute, Cambridge
Assessment English, University of Cambridge, and Cambridge International & St
John’s College Scholarship. Thanks to Yiting Lu for interesting discussions.
Thanks to the anonymous reviewers for their helpful comments.

## References

  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015.  Scheduled sampling for sequence prediction with recurrent neural networks.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 1171–1179.
  * Cao and Wang (2021) Shuyang Cao and Lu Wang. 2021.  Attention head masking for inference time content selection in abstractive summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5008–5016, Online. Association for Computational Linguistics.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019.  What does BERT look at? an analysis of BERT’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 276–286, Florence, Italy. Association for Computational Linguistics.
  * Clifton et al. (2020) Ann Clifton, Sravana Reddy, Yongze Yu, Aasish Pappu, Rezvaneh Rezapour, Hamed Bonab, Maria Eskevich, Gareth Jones, Jussi Karlgren, Ben Carterette, and Rosie Jones. 2020.  100,000 podcasts: A spoken English document corpus.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5903–5917, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018.  A discourse-aware attention model for abstractive summarization of long documents.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 615–621, New Orleans, Louisiana. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gehrmann et al. (2018) Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018.  Bottom-up abstractive summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4098–4109, Brussels, Belgium. Association for Computational Linguistics.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 1693–1701.
  * Hsu et al. (2018) Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018\.  A unified model for extractive and abstractive summarization using inconsistency loss.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 132–141, Melbourne, Australia. Association for Computational Linguistics.
  * Huang et al. (2021) Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021.  Efficient attentions for long document summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1419–1436, Online. Association for Computational Linguistics.
  * Jones et al. (2020) Rosie Jones, Ben Carterette, Ann Clifton, Maria Eskevich, Gareth J. F. Jones, Jussi Karlgren, Aasish Pappu, Sravana Reddy, and Yongze Yu. 2020.  Trec 2020 podcasts track overview.  In _The 29th Text Retrieval Conference (TREC) notebook_.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kobayashi et al. (2020) Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020.  Attention is not only a weight: Analyzing transformers with vector norms.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7057–7075, Online. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2019) Manling Li, Lingyu Zhang, Heng Ji, and Richard J. Radke. 2019.  Keep meeting summaries on topic: Abstractive multi-modal meeting summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2190–2196, Florence, Italy. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Manakul and Gales (2021) Potsawee Manakul and Mark Gales. 2021.  Long-span summarization via local attention and content selection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6026–6041, Online. Association for Computational Linguistics.
  * Manakul et al. (2020) Potsawee Manakul, Mark J.F. Gales, and Linlin Wang. 2020.  Abstractive Spoken Document Summarization Using Hierarchical Model with Multi-Stage Attention Diversity Optimization.  In _Proc. Interspeech 2020_ , pages 4248–4252.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 8024–8035.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Tay et al. (2020) Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020.  Efficient transformers: A survey.  _arXiv preprint arXiv:2009.06732_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Voita et al. (2019) Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5797–5808, Florence, Italy. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020.  Big bird: Transformers for longer sequences.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 17283–17297. Curran Associates, Inc.

##  Appendix A Reproducibility Details

###  A.1 Models

BART/LoBART: We use the HuggingFace’s implementation Wolf et al. (2020),
including BART models fine-tuned to
CNNDM555https://huggingface.co/facebook/bart-large-cnn and
XSum666https://huggingface.co/facebook/bart-large-xsum. We take LoBART from
Manakul and Gales (2021), including LoBART(4k)+MCS fine-tuned to Podcast and
arXiv. MCS is the multitask content selection system for handling the
Podcast/arXiv input documents that exceed 4096 words.

Modified Architecture: As shown in Fig. 3, the modified architecture consists
of a sentence encoder and a sentence-level encoder-decoder attention. The
sentence encoder, which approximates $f_{2}(.)$ in Eq. 7, is a two-layer bi-
directional GRU Cho et al. (2014) with hidden dimension of 1024. The sentence-
level encoder-decoder attention has linear mapping weights $\mathbf{W}^{\tt
Q}$ (query) and $\mathbf{W}^{\tt K}$ (key), and both weights have the same
dimension as and are initialized from their corresponding word-level encoder-
decoder attention’s linear mapping weights in the base model. The total number
of additional parameters is 58.8M.

###  A.2 Data

CNNDM: We follow the standard train/valid/test split of 287,113/13,368/11,490.

XSum: We follow the standard train/valid/test split of 204,045/11,332/11,334.

Spotify Podcast: We follow the data processing and split in Jones et al.
(2020), resulting in train/valid/test splits of 60,415/2,189/1,027.

arXiv: We follow the standard train/valid/test split of 203,037/6,436/6,440.

Our data processing is based on the byte-pair-encoding (BPE) tokenizer same as
the BART-large tokenizer, and we use the NLTK toolkit for sentence splitting.
**In Fig. 2 and Fig. 4, to reduce computational cost, we use first 2,000
samples of each test set, except Podcast which contains less than 2,000
samples.

Dataset | $N$ | $N_{1}$ | $M$ | $N/M$
---|---|---|---|---
CNNDM | 870 | 28.7 | 67.4 | 14.2
XSum | 489 | 17.4 | 27.9 | 18.2
Podcast | 5727 | 330.8 | 86.6 | 143.9
arXiv | 8584 | 237.0 | 364.9 | 45.3
Table 4: Data statistics (average over corpus). $N$=input length,
$N_{1}$=#sentences, $M$=summary length.

###  A.3 Training and Inference

We use PyTorch Paszke et al. (2019) in our experiments. All training
experiments use the Adam optimizer Kingma and Ba (2015) with $\beta_{1}$=0.9,
$\beta_{2}$=0.999, and the learning rate is:

$\text{lr}=0.002\times\text{min}(\text{step}^{-0.5},\text{step}\times\text{warmup}^{-1.5})$

where we use 20,000 warmup steps. In all experiments, we set batch size to 1,
and gradient accumulation to 2 steps. We evaluate the training loss on the
validation set every 20,000 steps, and stop the training if the validation
loss does not improve 3 times. All training experiments converged within 1
epoch. All experiments were carried out in 32-bit precision on either one V100
(32GB) GPU, or one RTX 2080Ti (11GB) GPU.

At inference time, we use the standard setting: beam search of width 4, and
length penalty of 2.0 Wu et al. (2016) for all experiments. The ROUGE Lin
(2004) scoring tool is pyrouge.777https://pypi.org/project/pyrouge/

###  A.4 Multi-head attention

In all of the equations and expressions in the paper, we omit the heads for
simplicity. Both BART and LoBART models have 16 heads. In Fig. 1, we average
$\alpha_{m,i}^{\tt s}$ over heads, before the summation. When computing an
uncertainty measure such as entropy $\text{H}(.)$ or KL-divergence
$\text{KL}(.)$, we compute the measure for each head separately and take the
average. In obtaining $\mathcal{I}^{r}_{m}$, we average $\alpha_{m,i}^{\tt s}$
over heads, before the top-$r$ operation, i.e. all heads get assigned the same
subset of sentences, but the differences are across layers and decoding
timesteps.

###  A.5 KL Loss and Integrated Training

The target ${\alpha}_{m,i}^{{\tt s}}$ is re-normalized to encourage higher
sparsity as follows:

$\boldsymbol{\alpha}_{m}^{{\tt
s}}\leftarrow\text{softmax}\left(\frac{\log(\boldsymbol{\alpha}_{m}^{{\tt
s}})}{T}\right)$ (15)

where temperature $T$ is set to 0.5. For integrated training, we set $\lambda$
in $\mathcal{L}_{\tt I}$ to 0.2. We initialize integrated training experiments
from KL-only models.

##  Appendix B Time Analysis

For each mode in Table 1, we take 6 samples of $M$ and the average time of 100
iterations. Curve fitting yields R-squared of at least 0.994.

Computational time as function of $M$ and $N$ is ${\tt
time}=c_{1}+c_{2}M+c_{3}N+c_{4}MN+c_{5}M^{2}+c_{6}N^{2}$. The coefficients are
obtained by a least-squares regression, e.g.
$\mathbf{c}^{*}=(\mathbf{P}^{T}\mathbf{P})^{-1}\mathbf{P}^{T}\mathbf{t}$ where
$\mathbf{P}$ is the matrix of $M,N$ associated with the coefficients, and
$\mathbf{t}$ contains the time measures. We collect 30 samples of the average
time of 100 F+B passes, spanning $N\in[256,1024]$ and $M\in[50,300]$. The
normalized coefficients are: $c_{1}=1.00,c_{2}=3.78\times
10^{-3},c_{3}=3.15\times 10^{-3},c_{4}=1.47\times 10^{-6},c_{5}=7.26\times
10^{-7},c_{6}=7.79\times 10^{-7}$. Because $c_{4}\approx 2c_{5}$ and $N>M$,
the enc-dec attention cost is greater than the decoder self attention cost.

##  Appendix C Sensitivity to Random Selection

System | R1 | R2 | RL
---|---|---|---
Vanilla ($\gamma=0.0$) | 39.06 | 14.32 | 36.07
$\mathcal{L}_{\tt A}$-tuned ($\gamma=0.1$) | 28.43 | 7.72 | 24.00
$\mathcal{L}_{\tt A}$-tuned ($\gamma=1.0$) | 21.38 | 4.22 | 17.69

Table 5: Impact of sparsity on the sensitivity to random selection based on
CNNDM with $r=5$.

##  Appendix D Model-free Approximation for Eq. 4

Since $\alpha^{\tt
s}_{m,i}=\frac{1}{Z_{m}}\sum_{j=1}^{J_{i}}\exp(\mathbf{q}_{m}\cdot\mathbf{k}_{i,j})$
is used to rank input sentences, the normalization term, $Z_{m}$, can be
dropped. The encoder-decoder attention has $\mathcal{O}(MN_{1}N_{2})$
complexity because $\mathbf{q}_{m}\cdot\mathbf{k}_{i,j}$ is computed for every
$m$ and $(i,j)$ pair. Hence, if we can group $\mathbf{k}_{i,j}$ into
sentences, the complexity could potentially be reduced. We try the following
approximation of unnormalized $\alpha^{\tt s}_{m,i}$:

$\displaystyle\sum_{j=1}^{J_{i}}\exp$
$\displaystyle(\mathbf{q}_{m}\cdot\mathbf{k}_{i,j})$
$\displaystyle=\sum_{j=1}^{J_{i}}\prod_{d=1}^{D}\exp({q}_{m,d}{k}_{i,j,d})$
(16)
$\displaystyle\approx\sum_{j=1}^{J_{i}}\sum_{d=1}^{D}\exp({q}_{m,d}{k}_{i,j,d})$
(17)
$\displaystyle\approx\sum_{j=1}^{J_{i}}\sum_{d=1}^{D}\phi({q}_{m,d})\phi({k}_{i,j,d})$
(18)
$\displaystyle=\phi(\mathbf{q}_{m})\cdot\left(\sum_{j=1}^{J_{i}}\phi(\mathbf{k}_{i,j})\right)$
(19)

where $d=\\{1,...,D\\}$ is the hidden dimension, and $\phi(.)=\text{ELU}(.)+1$
(or other form such as $\exp$ and ReLU). The model-free method reduces
complexity from $\mathcal{O}(MN_{1}N_{2})$ to
$\mathcal{O}(MN_{1}+k_{w}MrN_{2}+k_{e}N_{1}N_{2})$ where $k_{e}$ is now much
smaller compared to the model-based approach. Based on Eq. 19, we provide
model-free results in Table 6.

Selection Method | R1 | R2 | RL
---|---|---|---
Ideal (Eq.6) | 43.94 | 20.82 | 40.81
Best Model-based | 43.72 | 20.40 | 40.70
Random | 39.06 | 14.32 | 36.07
Model-free (Eq.19) | 40.01 | 17.28 | 36.95

Table 6: Model-free results on CNNDM ($r=5$).

Our model-free approach is better than the random selection baseline, but it
is significantly worse than both the ideal selection baseline and the model-
based approach. The reasons for this poor performance are: (i) the
approximation from (16) to (17) requires the following condition to be true:
$A_{1}A_{2}>B_{1}B_{2}\rightarrow A_{1}+A_{2}>B_{1}+B_{2}$, so it is
inaccurate when the values are not in a similar range; (ii) the approximation
from (17) to (18) is inaccurate for non-positive values. In conclusion, this
experiment investigates an alternative challenging method, which would not
require any training and would be computationally cheaper at inference time.
Although the current algebra does not work well, we hope that our initial
study might draw more interests into this type of model-free approach to
exploit the sentence structure in seq2seq tasks such as abstractive
summarization.

##  Appendix E Word-level and Sentence-level Attention Weight Plots

Constraining the encoder-decoder attention to $r$ sentences is motivated by
the observations of sentence-level attention (in Fig. 5(b), 6(b), 7(b), 8(b)).
Note that we average over all heads for the plots.

For instance, Fig. 5 shows that the decoder attends particularly to input
sentences #1,#2,#13 in the summary generation. Compared to Fig. 5, Fig. 6
shows a wider spread of the attention over sentences in a more abstractive
task. When using LoBART, Fig. 7 and Fig. 8 show a similar trend of the
sparsity to BART scenarios. These figures also explain Fig. 1(a) (Section 3)
that $\sum_{i}^{\mathcal{I}^{r}_{m}}\alpha_{m,i}^{\tt s}$ is only low/moderate
because most sentences get assigned some attention weights, despite being non-
salient.

(a) Word-Level

(b) Sentence-Level ($\alpha_{m,i}^{\tt s}$)

Figure 5: Example of BART’s encoder-decoder attention evaluated on CNNDM test
set.

(a) Word-Level

(b) Sentence-Level ($\alpha_{m,i}^{\tt s}$)

Figure 6: Example of BART’s encoder-decoder attention evaluated on XSum test
set.

(a) Word-Level

(b) Sentence-Level ($\alpha_{m,i}^{\tt s}$)

Figure 7: Example of LoBART’s encoder-decoder attention evaluated on Podcast
test set.

(a) Word-Level

(b) Sentence-Level ($\alpha_{m,i}^{\tt s}$)

Figure 8: Example of LoBART’s encoder-decoder attention evaluated on arXiv
test set.


The Implication of Enhanced Cosmic Ray Diurnal Anisotropy on the Global
Simultaneity of Forbush Decreases: A Statistical Approach–References

# The Implication of Enhanced Cosmic Ray Diurnal Anisotropy on the Global
Simultaneity of Forbush Decreases: A Statistical Approach

O. Okike1,4<EMAIL_ADDRESS>J. A. Alhassan2,4 and I. O.
Eya2,3,4
1Department of Industrial Physics  Ebonyi State University  Abakaliki
Nigeria.
2Department of Physics and Astronomy  University of Nigeria  Nsukka  Nigeria.
3Physics/Electronics Technique – Department of Science Laboratory Technology
University of Nigeria  Nsukka  Nigeria.
4Astronomy and Astrophysics Lab. Faculty of Physical Sciences  University of
Nigeria  Nsukka

(Accepted November 05, 2022, Received October 02, 2022; 2022)

###### Abstract

The short-term rapid CR flux depressions, generally referred to as Forbush
decreases (FDs), are the most spectacular time-intensity CR variation. The
need for analytical transformation of the observational CR time series data,
to account for FDs and other recurrence tendencies such as periodicities and
cycles, was noted since the 1930s’. Nevertheless, it has been recently
observed that harmonic analysis, which is capable of transforming raw CR data
into different frequencies, is rarely exploited. Predominant in the literature
are the ordinary Fourier and power spectral analyses, generally used to
calculate the positive vectors (amplitude and phase) of the periodic diurnal
CR anisotropy. In the two approaches, the days immediately connected with FDs
are frequently removed to minimize unusual changes in the amplitude of the
vectors as well as a spurious time of maximum. However, there is a paucity of
publications that adjust for the influence of enhanced CR diurnal anisotropy
on the magnitude and timing of FDs. Recently, in an attempt to test the global
FD event simultaneity, a combination of numerical filtering and fast Fourier
transform techniques was deployed to account for these superposition
tendencies in daily CR data, including the intractable CR diurnal anisotropy.
However, an extremely sensitive version of the software would be required to
analyze high-resolution CR hourly averages. As a way of achieving accurate
detection and precise timing of FD signals, the computer algorithms were
technically improved and employed in a long-term statistical investigation. To
appreciate the implemented extremely sensitive statistical technique, several
validation analyses, including FD-based solar-terrestrial correlation,
comparison of FD catalogues, and Frobush event simultaneity test were
conducted.

###### keywords:

Cosmic rays, cosmic ray modulation, Forbush decreases, algorithm,
simultaneity, diurnal anisotropy

##  1 Introduction

Investigation of temporal changes in the galactic cosmic ray (GCR) radiation
at the Earth started in the 1930s following the pioneering work of S. E.
Forbush (Forbush, 1938). Among the various CR intensity-time changes such as
daily, 27-day, 11- and 22-year periodic cycles, the sharp and non-recurring
(although they may be recurrent when connected with coronal holes) intensity
decreases, the so- called Forbush decreases (FDs), which seem to have gained
more attention. Lockwood (1971) is an excellent review of the literature on
FDs covering the initial understanding of Forbush events from 1938-1970
whereas (Venkatesan et al., 1992; Cane et al., 1996; Cane, 2000) are more
focused on causes of the phenomenon. Several interesting FD-related subjects
(e.g. anisotropies as well as their attendant bias on FD magnitude estimation,
magnitudes of FDs, CR modulation during FDs, FD dependence on rigidity and
altitude, the the link between FD and geomagnetic disturbances, longitudinal
and latitudinal dependence of FDs, FD onset time, precursory
increases/decreases that occur before the arrival of the solar events that
generate FDs, the recovery of intensity level during FDs and simultaneity of
FDs), was reviewed by Lockwood (1971).

Almost each of these topics is still hotly discussed among scientists,
especially in connection with Sun-weather studies (e.g Badruddin et al., 1991;
Ananth et al., 1993; Pankaj et al., 1994; Pudovkin & Veretenenko, 1995; Marcz,
1997; Kane, 2010; Pankaj & Singh, 2005; Bondo et al., 2010; Svensmark et al.,
2009, 2012; Scott et al., 2016; Svensmark et al., 2016; Okike, 2019; Okike &
Umahi, 2019; Okike et al., 2021). For example, Okike et al. (2021)
investigated solar-terrestrial relations concerning FD onset time/time of
maximal decreases using the superposed epoch method of analysis. Besides the
customary superposition analysis, correlation and regression analyses have
also been used to test the connection between FDs and solar-geophysical
parameters (see Okike, 2019; Okike & Umahi, 2019; Belov et al., 2002; Belov,
2008; Belov et al., 2014; Lingri et al., 2016). Despite the volumes of work on
the influence of the Sun (using FD as an indicator) on the Earth’s weather,
technology, and human life, definitive conclusions have yet to be reached. A
review of the existing publications suggests that the definition of FDs,
causes of FDs, Forbush event identification and timing, the implication of CR
anisotropy on FD event characteristics, simultaneity and magnitude of Forbush
events as well as what determines them are some of the major sources of
misunderstanding among CR researchers (Belov et al., 2002; Oh et al., 2008; Oh
& Yi, 2009; Okike & Collier, 2011; Laken et al., 2012; Lee et al., 2013; Belov
et al., 2018; Okike & Umahi, 2019; Abunina, 2020; Okike, 2020, 2020). In the
current work, we intend to further investigate the phenomena of global
simultaneity of Forbush events.

##  2 Data

The hourly CR data are sourced from http://cr0.izmiran.rssi.ru/ while data on
IZMIRAN FEs are taken from
http://spaceweather.izmiran.ru/eng/dbs.html. The latter will be referred to as
the Forbush-effects and interplanetary disturbances database (FEID,
hereafter.) Solar wind speed (SWS) and interplanetary magnetic field (IMF)
data used are taken from https://omniweb.gsfc.nasa.gov/html/ow_data.html.
Accessing SWS and IMF data is straightforward. The principal investigators
provide a simple interface that helps researchers produce plots or download
text files of the selected data resolutions. Hourly, daily, 27-day, and yearly
averaged data are available on the website. There is equally a window for data
availability checks. The hourly data used in the current work were downloaded
by filling out the selection icons for SWP and IMF as well as the submission
box.

##  3 Method of Analysis

The manual method of event identification is considered a case study approach.
Specific events that happen within the period of interest are isolated and
examined in detail. How much selection takes place remains an open question in
such method (Pittocks, 1978). A statistical approach to FD event
identification is attempted here.

Figure 1 shows the CR flux variation for a full solar cycle (Solar Circle 23)
for THUL and INVK stations. The dips are indications of FDs (see (Harrison et
al., 2011). There are several of them. We have marked some of the large events
for reference purposes. It is also evident that the largest FDs happen around
the solar cycle maximum (2001). Instead of isolating each of the pits and
separately calculating their magnitude, onset time, and time of minimum, the
programs developed by Okike & Umahi (2019)(Paper I, hereafter) and Okike
(2020) will be enhanced and used to calculate the event magnitude and time of
occurrence of all the FDs that happen within the period.

Detection of FDs from hourly data requires a relatively more efficient and
sensitive algorithm than those used in the analysis of daily data. It is
interesting to note that two versions of Okike (2020) codes $-$ which takes CR
hourly means as input data $-$ are recently developed and implemented in Okike
(2021) and Alhassan et al. (2022). Using one and five years’ data
respectively, Okike (2021) and Alhassan et al. (2022) demonstrated that
different versions of our FD algorithm can detect FDs from hourly CR counts.
An attempt is made to briefly describe the codes in the next section.

###  3.1 A Brief Description of the FD Location Algorithms

Okike (2020) employed two different FD selection programs. The first code
(referred to as Prog1, hereafter) was developed by a group of researchers (see
Okike & Umahi, 2019). The second program (hereafter, Prog2) was recently
developed and first implemented by Okike (2020).

Prog1 employs a combination of the traditional filtering method (Harrison et
al., 2011; Laken & Calogovic, 2013; Tezari, 2016) and fast Fourier transform
techniques. The two major signals that could complicate the identification of
the rapid and non-periodic fluctuations (e.g. FDs and GLEs) in CR raw data are
solar cycle variations and the periodic as well as the enhanced diurnal
variations. The effect of the 11-year solar cycle is evident in Figure 1.
While it is easy to filter the solar cycle effect using a high pass filter,
adjusting for the contribution of CR diurnal anisotropy requires some data
transformation. Before passing the raw data to the Fourier transform
subroutines, the data values were first subtracted from the running mean to
remove the effect of solar cycle variations. The running mean is calculated
using a special function (roll mean) in R for statistical computing software
(Team, R Core, 2014).

After removing the solar cycle effects, the next challenge is how to deal with
the superposed ordered/periodic and random/non-periodic components of the
data. The Fourier transform code handles this stage. It separates the data
into two components $-$ regular and irregular frequency parts. The regular
frequency part accounts for the diurnal variation whereas the irregular
component is composed of the short-term random intensity changes, including
FDs and GLEs.

The two separate components are of interest in the current work. Amplitudes of
FDs and diurnal waves are calculated respectively from the irregular and
ordered frequency signals. Although this could be done manually, it will be
tedious and time-consuming. Another program capable of simultaneously tracking
event time and magnitude is designed to do the calculation. The algorithm
scans input signals for possible maxima/peaks and dips/pits/minima. The peaks
(note that most of the peaks have been removed in Figure 1 for plotting
purposes) may be GLEs or anomalous CR modulations/enhancement (Dorman et al.,
2018). Since we are interested in the depressions, the program is instructed
to calculate the time and magnitude of the intensity reductions.

Prog2 operates basically in the same way as Prog1. There are, however, a few
similarities as well as differences in the operation of Prog2 and Prog1. The
high pass filter and rollmean functions are integrated into both software. FD
location subroutines are also implemented in both programs. The major
difference between Prog1 and Prog2 is that the influence of anisotropy is not
taken into consideration in Prog2.

We note that the date/time format implemented by Okike (2020) is of the form
"YYYY.mm.dd" whereas the format in the current work is of the form "YYYY.mm.dd
HH: mm". This is to enable both Prog1 and Prog2 to take hourly data as its
input signal.

Figure 1: Raw CR intensity variation for THUL (blue) and INVK (red) stations
for the Solar Cycle 23

###  3.2 Application of Prog1 and Prog2

####  3.2.1 Low Solar Activity Period

Figure 2: Comparison of raw, Fourier transformed (FTS) and diurnal wave at
INVK station for the year 1996. The filled blue and black circles are
indications of FDs associated with the (FTS) and the raw data. Figure 3: Same
as Figure 2 for THUL station.

This section aims to demonstrate the performance of the two FD detection
algorithms. For clarity’s sake, two separate years (1996 and 2003) are used to
demonstrate the operations of Prog1 and Prog2. The two years represent the
periods of low (1996) and 2003 (high) solar activities. Figures 2 and 3 show
the variation patterns of the raw data, the Fourier transformed component
(FTS), and the diurnal wave at THUL and INVK stations. The blue and black
filed circles are FDs detected by Prog1 and Prog2. It is evident from Figure 1
that CR intensity reductions in 1996 are very small compared to the period of
high solar activity. Manually selecting FDs from the raw data might be very
difficult. Inspection of Figures 2 and 3 shows that the FDs associated with
the raw data are fewer (30 and 21 at INVK and THUL respectively) in comparison
with those detected from the FTS (69 and 90 at INVK and THUL respectively). A
small threshold (CR (%) $\leq-2$) was employed here.

Whereas other researchers (e.g Laken et al., 2011) that searched for FDs
within the Solar Cycle 23 did not identify FDs in the year 1996, the IZMIRAN
group recorded 97 FDs the same year. This is possible with the aid of the GSM
which accounts for CR anisotropy. The red curve in the two diagrams is
representative of diurnal anisotropy at INVK and THUL stations. Separating
them from the raw data allows us to detect the FDs that are otherwise,
obscured. There are interesting differences and similarities in the trend of
the diurnal anisotropy at the two stations. The largest amplitude of the
anisotropy at INVK and THUL are respectively -2.02% and -1.74%. It is,
however, better to present similar diagrams for a period of high solar
activity for comparative purposes.

Figure 4: Same as Figure 2 for the year 2003. Figure 5: Same as Figure 3 for
the year 2003.

####  3.2.2 High Solar Activity Period

Figures 4 and 5 present the relationship for the raw data, FTS, and diurnal
wave for the year 2003 for INVK and THUL stations respectively. The blue and
red horizontal lines indicate the selection baseline and the amplitude of the
largest anisotropy that happened toward the end of 2003. As noted in Figures 2
and 3, there are noticeable differences between the raw data and FTS. For
example, the FTS indicates larger depressions than the raw data. The position
of the small event of 2/3/2003 is marked in both diagrams. The arrow shows a
small dip pointing below the horizontal black line for the FTS whereas a
similar dip for the raw data is above the line in Figure 4. We have a somewhat
different scenario in Figure 5. The corresponding depression in the raw data
is below the zero line. The implications are that the event may not be
detected from the raw data. But It may be detected from the raw data and the
FTS in Figure 5.

However, whether Prog1 detects the event or not depends on the detection
threshold. It is evident in both diagrams that the small event is not picked
by the algorithms. This is due to the large normalization baseline (CR (%)
$\leq-6$) used here. Several dips larger than those that appear within the
time of the event of 2/3/2003 are also not detected. When the threshold was
increased to (CR (%) $\leq-0.001$) Prog1 detected the event at 14:00 on
3/3/2003 at the INVK and THUL stations. Its magnitude is respectively -1.3 and
-1.73%. The event seems to be completely obscured by anisotropy and thus, it
is not detected from the raw data. The small dip around the event in Figure 5
was another event that happened at 06:00 on 5/3/2003. Its magnitude is -0.36%.

The intensity variation trend of the diurnal wave in comparison with the raw
and FTS is also quite interesting. There are two large depressions in Figures
4 and 5. The two dips are accompanied by two large amplitudes of the diurnal
wave. The largest FDs attracted the largest amplitudes of anisotropy in the
two diagrams. This is not, however, the case in the period of low solar
activity. The large amplitudes of the diurnal anisotropy in Figures 2 and 3
seem not to track the large FDs.

###  3.3 Coincident and Non-Coincident Algorithms

Event simultaneity could be defined regarding the complete FD profile or
either of the parts of Forbush events such as the onset, main phase, FD
minimum, and the recovery phase at different CR stations as was the case in Oh
et al. (2008) and (Okike & Collier, 2011). In the light of Okike (2020), an
event is considered to be simultaneous in the current work if its time of
minimum happens at the same universal time at INVK and THUL CR stations.
Conversely, an event is regarded as non-simultaneous if its FD minimum time
differs at the two stations.

A simple coincident or matching code was developed by (Okike & Nwuzor, 2020).
The algorithm searches for equivalent FD magnitude or identical time in two or
more event catalogs. The magnitude and event date of all the FDs from separate
stations are first transformed into a data frame or matrix of equal or unequal
length. The data are then passed to the algorithm, specifying the magnitude or
event date/time as the key search term. For the timing of globally
simultaneous or non-simultaneous FDs, the event date is specified as the
reference search parameter. In the current work, another simple code $-$ non-
coincident algorithm $-$ which operates differently is also developed. The two
subroutines are used for event identification. One searches for simultaneous
events while the second identifies non-simultaneous FDs.

(Okike, 2020) used the coincident code to select simultaneous FDs from seven
event catalogs. Each list from the seven CR stations investigated contains a
large number of FDs with a minimum and maximum of 238 and 386 FDs
respectively. The simple algorithm was recently used to select simultaneous
FDs as well as the amplitude of the accompanying diurnal anisotropy (see
Okike, 2020).

Here, the catalogs of FDs selected from INVK and THUL by Prog1 and Prog2 serve
as input to the coincident code.

##  4 Results and Discussions

###  4.1 FD Event Selection by Prog1

Figure 6: Same as Figures 2 for the Solar Cycle 23. Figure 7: Same as Figures
3 for the Solar Cycle 23. The blue circles are indications of FDs.

Figures 6 and 7 present the results of FD event selection (Prog1) for Solar
Cycle 23 at INVK and THUL stations respectively. The black and red signals are
the Fourier-transformed CR data (FTS) and the diurnal wave. The blue circles
are reflections of the magnitude and timing of FDs. The raw data and the
associated FDs are not reported here.

Some of the large FDs in Figure 1 are also marked in these diagrams for
comparative reasons. In the three figures, the series of FDs that happened
towards the end of October 2003 is the largest. In Figure 1, the second to
this event in magnitude is the FD of 16/7/2000. However, after removing the
11-year solar cycle effects, Figures 6 and 7 suggest that the event of 19
January 2005 is the second in the rank of FDs that happened within the period.
This is followed by the event of 16/7/2000.

The number of FDs detected at INVK and THUL are respectively 884 and 980. A
common threshold of $\mathrm{CR}(\%)\leq-3.5$ (Okike, 2019, e.g) is used for
event selection at the two stations. It should be noted that the number of FDs
identified may vary when different normalization baselines are used. With a
baseline of $\mathrm{CR}(\%)\leq-3$ and $\mathrm{CR}(\%)\leq-2.5$, for
example, (Okike, 2019) respectively identified 3905 and 6797 FDs from CLMX
data between 1957-2006.

The amplitudes of the diurnal waves exhibit interesting patterns at the two
stations. They tend to follow the 11-year solar cycle, pointing to the periods
of high solar activities (e.g. 2000 and 2003). Two peaks of the waves are
evident in the diagrams. The two largest amplitudes happened in the years 2001
and 2003. The two years are associated with periods of high solar activity
(e.g Tezari, 2016). The year 2001 is the year of solar maximum. While the
amplitudes of these waves are nearly equal in Figure 6 (-17.88%), Figure 7
shows that the amplitude is larger in 2003 (-17.54%) than in 2001. It is also
worth noting that the amplitudes of the waves decreased around 2002 at the two
stations.

The amplitudes of the diurnal wave in Figures 6 and 7 reflect those in Figures
2, 3, 4 and 5. Whereas small amplitudes are registered in the year 1996
(Figures 2 and 4), large amplitudes of the diurnal wave are registered at the
two stations in 2003 $-$ year of high solar activity (Figures 4 and 5). They
also reflect the result presented by (Okike, 2020) using daily averaged CR
data.

Figures 6 and 7 also allows the reader to consider FD event simultaneity at
the two stations. The exact time is indicated for the four events that are
marked in the diagrams. The time differences between the four large events at
INVK and THUL stations are in the range of $0\leq\mathrm{UT}\leq 3$, where UT
represents the universal time. In the light of Okike (2020) where events are
said to be simultaneous if points of minimal CR depressions are observed on
the same day (regardless if the differences of the FD minimum at different
stations are several hours), the four events might be taken as simultaneous
FDs. In the current work, however, only the event of 16/7/2000 whose time
difference at the two stations is 0 (zero) is simultaneous at INVK and THUL.

###  4.2 Simultaneous/Non-Simultaneous FDs and the Associated
Solar/Geophysical Parameters

Table 1 presents the list of 61 large FDs that happened at the same universal
time at INVK and THUL stations (List of the low magnitude ($>$ -3%) events
that are simultaneous at the two stations are not presented). Solar wind data
(IMF and SW) and Dst associated with the FDs are also listed. The Table allows
us to comment on the CR intensity modulations at the two stations. Inspection
of the first 6 events suggests that THUL measures larger FDs compared with
INVK. Nevertheless, the magnitude of the 7th event is larger at INVK than at
THUL. On the other hand, the magnitude of the 8th event is nearly equal at the
two stations. This points to the varieties of FDs and their different
manifestations at different locations (Belov, 2008; Okike & Collier, 2011).
This is also reflected in Figure 1. A close inspection of the diagram shows
that INVK sees larger intensity variations for some events. The event of
31/10/2003 is apparently larger at INVK in the diagram. The event is the 44th
in Table 1. Its magnitude at INVK and THUL are respectively -42.80% and
-40.71%. Another event that seems to be a simultaneous FD at the two stations
in Figure 1 is the event of 16/7/2000. The longer tail of the event at THUL
suggests that its magnitude is larger than at INVK. This difference is also
reflected in Table 1. The magnitude of the events at INVK and THUL are
respectively -28.17 and -30.70%.

The mean, maximum, minimum, the 25 and 75% of the FDs at INVK are 10.42 $\pm$
0.90%, 42.81%, 3.89%, 5.69 (25%, 25 percentile) and 13.06 (75%, 75
percentile). For THUL stations, the mean, maximum, minimum, 25, and 75% of the
FDs are 10.31 $\pm$ 0.88%, 40.71%, 3.73%, 6.05 (25%) and 12.54 (75%).

Given the large number of FDs detected at INVK (884) and THUL (980), one may
infer from the result presented in Table 1 that non-simultaneous FDs are much
more in number than events that happen simultaneously at the two stations.
While Table 1 reports that only 7 FDs are simultaneous in the year 2003, table
3 of Okike (2020) shows that simultaneous FDs are more in number, depending on
pairs of stations of interest. The number of simultaneous FDs at Sanae (SNAE)
and Moscow (MOSC) stations is, for instance, 16. For South Pole (SOPO) and
ClMX stations, 33 events are simultaneously detected. Okike (2020) attributes
the large number difference between simultaneous FDs selected with daily and
hourly data to the speculated increase in diurnal anisotropy (see Lockwood,
1971; Dumbovic et al., 2011; Belov et al., 2018, and the references therein)
in hourly averaged CR data. The phase shift (Pomerantz et al., 1971) arising
from CR anisotropy impacts significantly on the global simultaneity of FDs.
Additionally, (Alhassan et al., 2022) speculate that binning artifact may play
a significant role in the outcome of the simultaneity test between two
temporal resolutions.

The number of FDs at INVK which are not simultaneously detected at THUL
station is 823 while 919 events (result not presented), at THUL, do not happen
at the same universal time at INVK. Although one could arrive at these numbers
by simple arithmetic, a subroutine of the coincident algorithm, capable of
isolating non-simultaneous FDs from two event catalogs is required to separate
the 61 simultaneous events from the large number of FDs at each station.

The mean, maximum, minimum, the 25 and 75% of the FDs at INVK are 7.11 $\pm$
0.16%, 44.56%, 3.50%, 4.23 (25%, 25 percentile) and 8.04 (75%, 75 percentile).
For THUL stations, the mean, maximum, minimum, 25, and 75% of the FDs are 7.09
$\pm$ 0.15%, 44.44%, 3.50%, 4.27 (25%) and 7.84 (75%). A comparison of these
statistics with those of the simultaneous FDs suggests that the magnitude of
the simultaneous FDs is, on average, larger than those of the non-simultaneous
events. This agrees with the indications of Oh et al. (2008) and (Okike &
Collier, 2011). The time difference between the largest event at INVK (44.56%)
and THUL (44.44%) is an hour (see Figures 6 and 7).

Table 1: List of large and simultaneous FDs at INVK and THUL CR stations and the associated geomagnetic storm index (Dst) solar wind data . Order | Year | Hour | INVK (%) | THUL (%) | IMF (nT) | SW (Km/s) | Dst (nT)
---|---|---|---|---|---|---|---
1 | 1996-10-09 | 14 | -4.95 | -5.84 | 4.70 | 494 | -26
2 | 1998-05-10 | 07 | -4.74 | -7.68 | 6.30 | 481 | -46
3 | 1998-06-14 | 05 | -4.90 | -6.90 | 11.40 | 348 | -10
4 | 1998-08-24 | 10 | -9.32 | -10.57 | 3.50 | 423 | -20
5 | 1998-08-28 | 02 | -13.93 | -15.26 | 6.50 | 589 | -84
6 | 1998-11-10 | 00 | -4.38 | -6.37 | 11.30 | 432 | -93
7 | 1999-01-23 | 13 | -15.48 | -13.17 | 13.10 | 578 | -9
8 | 1999-08-23 | 09 | -7.74 | -7.59 | 9.40 | 402 | -48
9 | 1999-09-16 | 02 | -5.26 | -4.14 | 6.20 | 548 | -18
10 | 1999-10-26 | 10 | -4.81 | -4.19 | 5.20 | 470 | -24
11 | 1999-12-12 | 20 | -10.51 | -12.09 | 8.10 | 657 | -17
12 | 1999-12-14 | 00 | -9.32 | -12.37 | 12.00 | 442 | -33
13 | 2000-03-01 | 07 | -5.25 | -6.18 | 9.00 | 518 | -20
14 | 2000-05-22 | 09 | -6.47 | -7.39 | 6.40 | 445 | 4
15 | 2000-06-10 | 06 | -13.29 | -14.43 | 8.10 | 546 | -44
16 | 2000-06-24 | 06 | -5.08 | -6.20 | 12.90 | 574 | -31
17 | 2000-07-16 | 02 | -28.17 | -30.70 | 45.90 | 970 | -278
18 | 2000-07-19 | 00 | -13.54 | -15.08 | 3.90 | 571 | -33
19 | 2000-08-13 | 02 | -9.09 | -7.81 | 19.80 | 554 | -72
20 | 2000-11-08 | 04 | -6.63 | -3.73 | 16.00 | 445 | -34
21 | 2001-04-04 | 20 | -7.41 | -6.05 | 13.40 | 713 | -14
22 | 2001-04-12 | 09 | -21.74 | -21.85 | 19.60 | 645 | -134
23 | 2001-08-31 | 02 | -8.56 | -10.26 | 6.60 | 412 | -9
24 | 2001-10-06 | 02 | -6.56 | -5.92 | 3.40 | 421 | -14
25 | 2001-11-24 | 18 | -12.17 | -12.11 | 22.00 | 827 | -205
26 | 2001-11-26 | 01 | -9.37 | -13.01 | 4.60 | 638 | -86
27 | 2002-01-04 | 00 | -10.79 | -11.04 | 5.70 | 320 | -7
28 | 2002-01-07 | 00 | -6.72 | -6.59 | 7.60 | 330 | 2
29 | 2002-01-10 | 18 | -7.87 | -4.52 | 18.60 | 581 | -37
30 | 2002-01-13 | 10 | -9.23 | -9.29 | 5.10 | 528 | -24
31 | 2002-01-17 | 05 | -5.19 | -4.83 | 5.40 | 358 | -9
32 | 2002-01-19 | 11 | -4.98 | -5.71 | 13.00 | 372 | 24
33 | 2002-03-20 | 17 | -8.41 | -8.00 | 17.20 | 520 | 11
34 | 2002-03-24 | 16 | -10.37 | -9.39 | 14.10 | 441 | -94
35 | 2002-04-18 | 09 | -10.59 | -5.20 | 13.80 | 492 | -111
36 | 2002-04-20 | 03 | -7.39 | -6.29 | 18.60 | 563 | -93
37 | 2002-07-26 | 14 | -5.69 | -5.57 | 7.20 | 429 | -14
38 | 2002-09-08 | 02 | -7.63 | -5.64 | 19.30 | 495 | -152
39 | 2002-11-21 | 06 | -5.60 | -6.31 | 24.50 | 531 | -79
40 | 2003-05-30 | 13 | -13.40 | -12.54 | 16.50 | 573 | -63
41 | 2003-08-18 | 04 | -4.25 | -4.35 | 19.70 | 492 | -53
42 | 2003-10-23 | 01 | -5.25 | -4.65 | 9.90 | 524 | -13
43 | 2003-10-28 | 10 | -10.70 | -7.50 | 12.90 | 759 | -32
44 | 2003-10-31 | 11 | -42.80 | -40.71 | 14.40 | 1003 | -55
45 | 2003-11-20 | 15 | -18.12 | -18.41 | 55.80 | 614 | -171
46 | 2003-11-28 | 01 | -13.08 | -14.03 | 5.20 | 418 | -6
47 | 2004-02-03 | 08 | -7.05 | -6.15 | 6.40 | 614 | -34
48 | 2004-11-07 | 21 | -6.66 | -7.35 | 45.10 | 660 | -74
49 | 2005-01-06 | 02 | -7.25 | -8.75 | 6.80 | 568 | -11
50 | 2005-01-21 | 22 | -31.58 | -27.83 | 22.40 | 901 | -89
51 | 2005-05-14 | 01 | -3.89 | -4.09 | 4.60 | 525 | -18
52 | 2005-05-15 | 07 | -23.07 | -22.04 | 54.30 | 922 | -229
53 | 2005-05-16 | 03 | -16.37 | -15.78 | 14.10 | 704 | -99
54 | 2005-05-17 | 03 | -13.06 | -13.97 | 7.00 | 572 | -68
55 | 2005-06-16 | 10 | -4.58 | -4.05 | 18.00 | 538 | -22
56 | 2005-08-06 | 22 | -9.21 | -9.14 | 8.80 | 595 | -29
57 | 2005-08-08 | 04 | -10.10 | -6.65 | 4.70 | 571 | -15
58 | 2005-08-24 | 17 | -13.58 | -11.94 | 23.70 | 657 | -133
59 | 2005-09-22 | 13 | -7.36 | -7.57 | 5.50 | 326 | -19
60 | 2006-12-10 | 09 | -10.44 | -10.15 | 4.90 | 625 | -24
61 | 2006-12-17 | 01 | -14.90 | -16.11 | 6.70 | 650 | -36
Table 2: Correlation results of FDs and the associated solar and geomagnetic data. The symbols on the diagrams are as follows: "r" represents the Pearson’s product-moment correlation coefficient, N represents the number of FDs in a group, "ALL" denote all the non-simultaneous FDs at INVK, THUL, and for the IZMIRAN catalogue, "t" for the t static, "R2" is the coefficient of determination, FD_invk and FD_thul stand for FDs at INVK and THUL stations respectively, and "none" indicate relations that are not significant at any level. Order | Variables | r | t | R2 | Sig level (%)
---|---|---|---|---|---
$-$ | Simultaneous FDs (N = 61) |  |  |  |
1 | FD_invk/FD_thul | 0.97 | 30.07 | 0.94 | 99.8
2 | FD_invk/IMF | 0.40 | 3.32 | 0.16 | 99.8
3 | FD_thul/IMF | 0.37 | 3.09 | 0.14 | 99
4 | FD_invk/SW | 0.74 | 8.56 | 0.55 | 99.8
5 | FD_thul/SW | 0.71 | 7.65 | 0.50 | 99.8
6 | FD_ink/DST | -0.50 | 4.41 | 0.25 | 99.8
7 | FD_thul/DST | -0.50 | 4.38 | 0.25 | 99.8
8 | IMF/DST | -0.74 | 8.45 | 0.55 | 99.8
9 | SW/DST | -0.60 | 5.75 | 0.40 | 99.8
10 | SW/IMF | 0.52 | 4.67 | 0.27 | 99.8
$-$ | Non-Simultaneous (ALL, INVK, N = 823) |  |  |  |
1 | FD_invk/IMF | 0.0 | 0.0 | 0.0 | none
2 | FD_invk/SW | 0.26 | 7.91 | 0.07 | 99.8
3 | FD_invk/DST | -0.39 | 12.13 | 0.15 | 99.8
$-$ | Group A (INVK, N = 478) |  |  |  |
1 | FD_invk/IMF | 0.0 | 0.1 | 0.0 | none
2 | FD_invk/SW | 0.36 | 8.33 | 0.13 | 99.8
2 | FD_invk/DST | -0.34 | 7.93 | 0.12 | 99.8
$-$ | Group B (INVK, N = 345) |  |  |  |
1 | FD_invk/IMF | -0.04 | 0.71 | 0.0 | 50
2 | FD_invk/SW | 0.02 | 0.43 | 0.0 | none
3 | FD_invk/DST | -0.01 | 0.22 | 0.0 | none
$-$ | Non-Simultaneous (ALL, THUL, N = 919) |  |  |  |
1 | FD_thul/IMF | -0.01 | 0.40 | 0.00 | none
2 | FD_thul/SW | 0.22 | 6.74 | 0.05 | 99.8
3 | FD_thul/DST | -0.45 | 15.99 | 0.22 | 99.8
$-$ | Group A (THUL, N = 527) |  |  |  |
1 | FD_thul/IMF | 0.33 | 7.91 | 0.11 | 99.8
2 | FD_thul/SW | 0.43 | 10.84 | 0.18 | 99.8
3 | FD_thul/DST | -0.45 | 11.65 | 0.21 | 99.8
$-$ | Group B (THUL, N = 392) |  |  |  |
1 | FD_thul/IMF | 0.03 | 0.66 | 0.0 | none
2 | FD_thul/SW | 0.02 | 0.32 | 0.0 | none
3 | FD_thul/DST | 0.0 | 0.0 | 0.0 | none
$-$ | IZMIRAN (ALL FDs, N = 1476) |  |  |  |
1 | FD/IMF | 0.0 | 0.05 | 0.0 | none
2 | FD/SW | 0.20 | 7.88 | 0.04 | 99.8
3 | FD/DST | -0.05 | 2.01 | 0.003 | 95
$-$ | Group A (N = 134) |  |  |  |
1 | FD/IMF | 0.44 | 5.67 | 0.0 | 99.8
2 | FD/SW | 0.48 | 6.34 | 0.04 | 99.8
3 | FD/DST | -0.09 | 1.07 | 0.003 | 50
$-$ | Group B (N = 1342) |  |  |  |
1 | FD/IMF | -0.01 | 0.37 | 0.0 | none
2 | FD/SW | -0.02 | 0.58 | 0.04 | none
3 | FD/DST | -0.04 | 1.28 | 0.003 | 80

##  5 Summary

Although articles investigating Forbush effects (Belov, 2008) started
appearing in the literature in the 1930s (Forbush, 1937), the selection of FDs
from a given NM data is one of the difficult aspects of the subject. This is
due to the ever-present CR diurnal anisotropies. Though the bias implications
of the periodic and the enhanced diurnal anisotropy on timing and magnitude as
well as on the number of FDs detected in a given period/data has long been
recognized (e.g Lockwood, 1971; Cane et al., 1996; Pomerantz et al., 1971;
Sari et al., 1978), several of the existing FD lists are prepared without
adjusting for CR anisotropy. This could explain the conflicting and misleading
conclusions in some past articles utilizing different FD catalogs (see Marcz,
1997; Okike & Umahi, 2019; Okike, 2020, 2019, for discussion).

Graphical illustration of FD global simultaneity was indeed attempted by some
researchers in the early 1970’s (e.g Lockwood, 1971), Oh et al. (2008)
asserted that they were the first to investigate the global simultaneous/non-
simultaneous FDs as well as the associated solar wind data. Besides the Oh et
al. research team (see also Oh & Yi, 2009; Lee et al., 2013, 2015) and the
contribution from Okike & Collier (2011), Okike (2020), and Alhassan et al.
(2022), we are not aware of other publications that are dedicated to the
analysis of global simultaneity of FDs.

While other works that investigate the characteristics of simultaneous and
non-simultaneous FDs at different points on Earth failed to take the impact of
anisotropy on FD profiles into consideration, the recent article of Okike
(2020) carried out a detailed analysis of CR anisotropy to examine the
properties of the simultaneous and non-simultaneous FDs. The current work is a
follow-up on the new approach.

Some of the main differences between this and Okike (2020) lie in the
differences in the data resolution and period. While Okike (2020) employed
daily averaged data over ten CR stations for one year, the current work uses
hourly averaged CR data over two NMs for a full solar cycle. Since this is a
preliminary effort to investigate FD global event simultaneity using FDs
automatically selected from hourly averaged data, result validation was of
topmost priority.

A close inspection of Table 1 and the associated correlation results in Table
2 suggests that the FDs identified by the computer code might be related to
solar as well as geophysical data. Although there is no one-to-one
relationship between FDs and Dst, for example, some of the large FDs in the
Table are linked with large negative Dst. A detailed investigation of solar
wind conditions generating such events might be taken up as case studies. The
present analysis may be repeated using FDs selected by Prog2. All the events
presented and discussed in the result section are identified by Prog1.

## Acknowledgements

Huge thanks are due to those maintaining the following websites:
http://cr0.izmiran.ru/common,
http://spaceweather.izmiran.ru/eng/dbs.html
and
https://omniweb.gsfc.nasa.gov/html/ow_data.html.

## References

  * Abunina (2020) Abunina, M.A., Belov, A.V., Eronshenko, E.A., Abunin, A.A., G., Y.V., Melkumyan, A.A., Shlyk, N.S., Pryamushikina, I.I., 2020, Research. Solar Physics, 295
  * Alhassan et al. (2022) Alhassan, J.A., Okike, O., Chukwude, A.E., 2022, J. Astrophys. Astr. 43(6),
  * Ananth et al. (1993) Ananth, A.G., Venkatesan, D., Solar Physics 143, 373. https://doi.org/10.1016/j.asr.2009.04.004
  * Badruddin et al. (1991) Badruddin, Venkatesan, D., Zhu, B.Y., 1991, Solar Physics 134, 203
  * Belov et al. (2002) Belov, A.V., Eroshenko, E.A., Oleneva, V.A., Struminsky, A.B., Yanke, V.G. Advances in Space Research 27(3), 625
  * Belov et al. (2014) Belov, A., Abunin, A., Abunina, M., Eroshenko, E., Oleneva, V., Yanke, V., Papaioannuou, A., Maromichalaki, H., Gopalswamy, N., Yashiro, S. 2014, Solar Physics 289(10)
  * Belov et al. (2018) Belov, A.V., Eroshenko, E.A., Yanke, V.G., Oleneva, V.A., Abunina, M.A., A., A.A., 2018, Geomagnetism and Aeronomy 58(3), 356
  * Belov (2008) Belov, A.V., 2008, Proceedings IAU Symposium 257, https://doi.org/10.1017/S1743921309029676
  * Bondo et al. (2010) Bondo, T., Enghoff, M.B., Svensmark, H., Atmospheric Chemistry and Physics, 10, 2765
  * Cane et al. (1996) Cane, H.V., Richardson, I.G., von Rosenvinge, T.T., JGR,1996, 101(A10), 21561, https://doi.org/10.1029/96JA01964
  * Cane (2000) Cane, H.V., 2000, Space Science Reviews, 93, 55
  * Dorman et al. (2018) Dorman, L., Velinov, P., Tomava, D., Alexander, M., Mateev, L., 2018, Space, Ecology, Safety,
  * Dumbovic et al. (2011) Dumbovic, M., Vrsnak, B., Calogovic, J., Karlica, M., 2011, A & A 531(A91),1
  * Forbush (1937) Forbush, S.E., Physical Review, 51, 1108
  * Forbush (1938) Forbush, S.E. 1938, Review, 1938 54(12), 975. https://doi.org/10.1103/PhysRev.54.975
  * Harrison et al. (2011) Harrison, R.G., Ambaum, M.H.P., Lockwood, M., 2011, Proc. of the Roya Society 467, 2777
  * Kane (2010) Kane, R.P., 2010, Ann. Geophys 28, 479
  * Laken & Calogovic (2013) Laken, B.A., Calogovic, J., 2013, J. Space Weather Space Clim 3(A29), https://doi.org/10.1051/swsc/2013051
  * Laken et al. (2011) Laken, B.A., Kniveton, D.R., 2011, J. Atmos. Solar Terr. Phys 73, 71
  * Laken et al. (2012) Laken, B.A., Palle, E., Calogovic, J., Dunne, E.M., 2012, J. Space Weather Space Clim 2(A18). https://doi.org/10.1051/swsc/2012018
  * Lee et al. (2013) Lee, S., Oh, S.Y., Yi, Y., 2013, JGR 118, 608
  * Lee et al. (2015) Lee, S., Oh, S., Yu, Y., et al., 2015, J. Astron. Space Sci 33(1), 33
  * Lingri et al. (2016) Lingri, D., Mavromichalaki, H., Belov, A., Eroshenko, E., Yanke, V.G., Abunin, A., Abunina, M., 2016, Solar Physics, 1
  * Lockwood (1971) Lockwood, J.A., 1971, Space Science Reviews 12, 658
  * Marcz (1997) Marcz, F., 1997, Journal of Atmospheric and Solar-Terrestrial Physics, 59(9), 957
  * Oh & Yi (2009) Oh, S.Y., Yi, Y., 2009, JGR, 114, 1
  * Oh et al. (2008) Oh, S.Y., Yi, Y., Kim, H.Y., 2008, Journal of Geophysical Research 113(A01103). https://doi.org/10.1029/2007JA012333
  * Okike & Collier (2011) Okike, O., Collier, A.B.. 2011, Journal of Atmospheric and Solar-Terrestrial Physics 73, 796
  * Okike & Nwuzor (2020) Okike, O., Nwuzor, O.C., 2020, MNRA 493(2), 1948 https://doi.org/10.1093/mnras/staa370
  * Okike & Umahi (2019) Okike, O., Umahi, A.E., 2019, Journal of Atmospheric and Solar-Terrestrial Physics 189, 35 https://doi.org/10.1016/j.jastp.2019.04.002
  * Okike & Umahi (2019) Okike, O., Umahi, A.E., 2019, Solar Physics 294(2) . https://doi.org/10.1007/s11207-019-1405-y
  * Okike et al. (2021) Okike, O., Nwuzor, O.C., Odo, F.C., Iyida, E.U., Ekpe, J.E., Chukwude, A.E., 2021, MNRAS, 502, 300–312 (2021). https://doi.org/10.1093/mnras/staa4002
  * Okike (2019) Okike, O., 2019, JGR: Space Physics 124, 1–16. https://doi.org/10.1029/2018JA026456
  * Okike (2019) Okike, O., 2019, Okike, O. ApJ, 882(15)
  * Okike (2020) Okike, O., 2020, MNRAS, 491, 3793–3804
  * Okike (2020) Okike, O., 2020, JSTP 211,. https://doi.org/10.1016/j.jastp.2020.105460
  * Okike (2020) Okike, O., 2020, Data in Brief 33, https://doi.org/10.1016/j.dib.2020.106463
  * Okike (2021) Okike, O., 2021, ApJ 60,
  * Pankaj & Singh (2005) Pankaj, K.S., 2005, Singh, N.S., Chin. J. Astron. Astrophys 5(2), 198
  * Pankaj et al. (1994) Pankaj, K.S., Shukla, R.P., Solar Physics 154, 177
  * Pittocks (1978) Pittocks, A.B., 1978, Review of Geophysics and Space Physics 16(3), 400
  * Pomerantz et al. (1971) Pomerantz, M.A., Duggal, P.S., 1971, Space Science Reviews 12, 75
  * Pudovkin & Veretenenko (1995) Pudovkin, M.I., Veretenenko, S.V., 1995, Journal of Atmospheric and Solar-Terrestrial Physics 57(11), 1349. https://doi.org/10.1016/0021-9169(94)00109-2
  * Sari et al. (1978) Sari, J.W., Venkatesan, D., Lanzerotti, J.J., Maclennan, C.G., 1978, J. Geophys. Res. 83, 5139
  * Scott et al. (2016) Scott, C.J., Harrison, R.G., Owens, M.J., Lockwood, M., Barnard, L., 2014, Environ. Res. Lett 9(055004)
  * Svensmark et al. (2009) Svensmark, H., Bondo, T., Svensmark, J., 2009, Geophys. Res. Letters 36(L15101). https://doi.org/10.1029/2009GL038429
  * Svensmark et al. (2012) Svensmark., J., Enghoff, M.B., Svensmark, H. Atmos. Chem. Phys. Discuss 12, 3595–3617. https://doi.org/10.5194/acpd-12-3595-2012
  * Svensmark et al. (2016) Svensmark, J., Enghoff, M.B., Shaviv, N.J., Svensmark, H., 2016, JGR: Space Physics 121, 8152, https://doi.org/10.1002/2016JA022689
  * Team, R Core (2014) Team, R Core: R: 2014, A Language and Environment for Statistical Computing. R Foundation for Statistical Computing
  * Tezari (2016) Tezari, A., Mavromichalaki, H., Katsinis, D., Kanellakopoulos, A., Kolovi, S., Plainaki, C., Andriopoulou, M., 2016, Annales Geophysicae 34, 1053
  * Venkatesan et al. (1992) Venkatesan, D., Badruddin, Ananth, A.G., Pillai, S., 1992, Solar Physics 137, 345