# Image-Text Matching with Multi-View Attention

Rui Cheng
Alibaba Group
<EMAIL_ADDRESS>Wanqing Cui
CAS Key Lab of Network Data Science and Technology,
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
University of Chinese Academy of Sciences, Beijing, China
<EMAIL_ADDRESS>

###### Abstract

Existing two-stream models for image-text matching show good performance while
ensuring retrieval speed and have received extensive attention from industry
and academia. These methods use a single representation to encode image and
text separately and get a matching score with cosine similarity or the inner
product of vectors. However, the performance of the two-stream model is often
sub-optimal. On the one hand, a single representation is challenging to cover
complex content comprehensively. On the other hand, in this framework of lack
of interaction, it is challenging to match multiple meanings which leads to
information being ignored. To address the problems mentioned above and
facilitate the performance of the two-stream model, we propose a multi-view
attention approach for two-stream image-text matching MVAM (Multi-View
Attention Model). It first learns multiple image and text representations by
diverse attention heads with different view codes. And then concatenate these
representations into one for matching. A diversity objective is also used to
promote diversity between attention heads. With this method, models are able
to encode images and text from different views and attend to more key points.
So we can get representations that contain more information. When doing
retrieval tasks, the matching scores between images and texts can be
calculated from different aspects, leading to better matching performance.
Experiment results on MSCOCO and Flickr30K show that our proposed model brings
improvements over existing models. Further case studies show that different
attention heads can focus on different contents and finally obtain a more
comprehensive representation.

##  1 Introduction

Figure 1: Examples of retrieved images with CLIP and our model MVAM-CLIP. CLIP
retrieved images ignore some important information of texts. MVAM-CLIP
retrieved images are more consistent with the texts.

Cross-modal image-text retrieval, due to its great application prospects, such
as product search and information retrieval, becomes an important task for
multi-modal understanding and attracts much attention. There are two typical
architectures for cross-modal retrieval: single-stream and two-stream. Single-
stream modelsLi et al. (2020a); Lu et al. (2019); Qi et al. (2020); Chen et
al. (2019); Li et al. (2020b) match image and text with full self-attention in
a unified encoder. Although they tend to attend higher accuracy, such methods
suffer from low inference efficiency since each pair of possible candidates
needs to be recomputed during inference, which greatly limits the application
in practical scenes. Two-stream models Faghri et al. (2017); Zhang et al.
(2020), such as SCANLee et al. (2018), CLIPRadford et al. (2021) and ALIGNJia
et al. (2021), perform self-attention over image and text separately and
inputs from two modalities are encoded respectively. The two representations
only need to be combined at the end to get matching scores. With high
inference efficiency, these methods show extraordinary talents in practical
application.

However, two-stream models only achieve sub-optimal results due to the lack of
closer image-text interactions and insufficient comprehensive representation
ability. Firstly, only a single representation is used to encode the image or
text beforehand, which is difficult to cover sufficient information needed for
retrieval, especially when the input text or image is complex. Secondly,
because of unknowing what to attend to during representation, the poor content
representation ability will be further amplified when cross-modal interaction
is lacking. As shown in Figure 1, even for such a powerful model as CLIP,
which is trained with large-scale corpus, it is not able to encode all
required information. For example, given the query text "A kitchen with two
windows and two metal sinks", CLIP omits the modeling of "two windows" and
returns an image that doesn’t match exactly.

In order to enhance the representation ability in two-stream models, in this
paper, we propose a novel Multi-View Attention Model (MVAM), which firstly
represents images and texts in diverse views separately with different
attention heads. And then these multi-view representations will be integrated
as the final representation of the image or text. In order to further promote
the diversity between different views of attention and get diversified
features of the input, we also introduce a diversity objective. As shown in
Figure 1, for the query "A kitchen with two windows and two metal sinks",
being able to encode more detailed information without ignoring any content,
our model retrieves the best matching image successfully.

As a pluggable method, MVAM can seamlessly connect with any two-stream model
and improve the representation ability, so as to help achieve better retrieval
performance. Empirical results on two popular benchmarks, i.e., MSCOCO Lin et
al. (2014) and Flickr30K Young et al. (2014), show that our model can
significantly raise the performance of the existing methods. Besides, compared
to other methods, our model does not need training on large-scale multi-modal
datasets.

The contributions of this work can be concluded as the following: (1) We
present a novel model MVAM. It encodes multi-modal data into multiple views to
enhance the representation of the two-stream matching model and help get
better performance. (2) We introduce a diversity loss, which is able to help
model learning image and text representations from multiple views. (3) We
exhaustively quantify the capability of our method on two benchmarks under
various settings and conduct comprehensive ablation studies and analyses. We
also give some case studies to help intuitively understand our method.

##  2 Related Work

In this section, we first review the existing two-stream method for image-text
matching. Then we revisit work on multi-representation learning, which is
mainly concentrated in the field of uni-modal.

###  2.1 Two-stream Image-text Matching method

According to the architectures of image encoder, the two-stream models can be
divided into two categories: Faster-RCNN-based models and ViT-based models.

Faster-RCNN based models firstly use Faster-RCNN models Ren et al. (2015);
Anderson et al. (2018) to detect bounding boxes and visual features of objects
in images. Then these regional features are encoded by an image encoder.
Karpathy and Fei-Fei (2015), SCAN Lee et al. (2018) and Wang et al. (2019)
encode the text and image by Bi-directional RNN and RCNN respectively and
computes the similarity score via the interaction of multi-modal
representations. LightningDOT Sun et al. (2021) and WenLan Huo et al. (2021)
follows the pre-trained vision-language models Lu et al. (2019); Chen et al.
(2019); Li et al. (2020b) uses a transformer to encode the regional features
as well as corresponding location. VisaulSparta Lu et al. (2021) practiced an
inverted index on text-to-image search, which cached the relevance of all
candidate images and all words in vocabulary as the lookup table. In online
retrieval, it uses bag-of-words modeling queries to achieve a fast retrieval
speed.

ViT based models Dosovitskiy et al. (2020) use ViT as the image encoder, which
firstly splits an image into multiple patches and uses a transformer-based
model to encode it into visual patch representations. CLIP Radford et al.
(2021) and ALIGN Jia et al. (2021) are the two most known ViT-based cross-
model matching models. They establish relations between image and text by
training on large-scale unlabeled image-text pairs from the web. Among them,
CLIP is an open-source model, which achieves excellent performance on multiple
image-text matching tasks.

###  2.2 Multi-Representation Learning

To extract different key components of a complex text, Lin et al. (2017) uses
multiple self-attention to encode instances into multi-vectors and promotes
the diversity between vectors through a penalization term. In the field of
information retrieval, Poly-Encoder Humeau et al. (2019) encodes the query
into multiple vectors by attending over different attention and polymerizing
them according to the attention to each candidate doc. ME-BERT Luan et al.
(2021) encodes candidate doc into multiple vectors, and the matching score is
determined by the highest similarity between them and the query. Further,
ColBERT Khattab and Zaharia (2020) extends this approach to both query and
candidate.

To encode images, Li et al. (2018) employs multiple spatial attention to
represent an image as multi-vectors and regularizes the diversity between
multi-attentions to discover different body parts or accessories for better
person re-identification.

Figure 2: The network architecture of MVAM with ViT encoder, which firstly
splits an image into a sequence of flattened 2D patches as the inputs of the
image encoder and tokenizes a text into a sequence of tokens as the inputs of
the text encoder, and encodes their outputs into multi-vector by the multiple
views on image and text. We concatenate multi-vectors as the representation of
the image and the text and take the cosine similarity between them as the
matching score.

##  3 Method

In this section, we first introduce the architecture and training objective of
two-stream models. Then we describe the details of the Multi-View Attention
Model as well as our multi-view diversity loss.

###  3.1 Two-stream Image-text Matching Model

All of our models are built on top of the two-stream matching model, which
consists of an image encoder and a text encoder. Given a pair of images and
sentences, it takes the visual regions of the image (or visual tokens for ViT-
based image encoder) and textual tokens of the sentence to their respective
encoding modules and extracts their representations. The cosine similarity of
representations can be seen as the matching score of each pair.

####  3.1.1 Image Encoder

We consider two typical architectures for the image encoder following the
recent progress in designing matching models: (1) ViT based image encoder
Dosovitskiy et al. (2020) and (2) Faster-RCNN based image encoder Ren et al.
(2015); Anderson et al. (2018).

FRCNN Based Image Encoder convert images into object-level embeddings. Similar
to previous work Chen et al. (2019), for an input image $V$, we first use a
pre-trained Faster R-CNN model to detect the salient $N$ image regions and
extract the pooled ROI feature $v_{i}\in\mathbb{R}^{2048}$ for each region.
Besides, we also consider its location feature $l_{i}\in\mathbb{R}^{7}$, which
is the normalized top-left and bottom-right coordinates of the region,
together with the region width, height, and area. The embeddings of the image
is $V_{frcnn}={(v_{1},l_{1},),...,(v_{N},l_{N})}$. Then we add a special token
$[IMG]$ at the beginning, which denotes the representation of the entire
image.The contextual region representations
$H_{v}={h_{[IMG]},h_{v_{1}},...,h_{v_{N}}}$ is obtained with a 12-layer
transformer network. Generally, the pooled $h_{[IMG]}$ can be regarded as the
semantic representation $R_{v}$ of image:

$H_{v}=Transformer([[IMG];V_{frcnn}])$ (1) $R_{v}=Pool(h_{[IMG]})$ (2)

in which the pooling layer is a fully connected layer with a tanh function.

ViT Based Image Encoder convert images into token-level embeddings. For an
input image $V\in\mathbb{R}^{H\times W\times C}$, where $(H,W)$ is the
resolution of the image and $C$ is the number of channels, we first split it
into a sequence of flattened 2D patches
$V_{p}\in\mathbb{R}^{N\times(P^{2}C)}$, in which $N=HW//P^{2}$ is the number
of patches and $(P,P)$ is the resolution. Then to get contextual
representations, we add a special token $[IMG]$ before the patches. Similar to
FRCNN based image encoder, we use a transformer to encode the patches to get
their hidden states and get the final image representation by pooling the
hidden states of $[IMG]$ token.

####  3.1.2 Text Encoder

We use BERT Devlin et al. (2018a) as our text encoder, which is a transformer-
based model pre-trained on large-scale Wikipedia and Bookcorpus. An input
sentence $T$ is firstly split into a sequence of tokens
${[CLS],t_{1},...,t_{L},[SEP]}$ by BERT Tokenizer. $[CLS]$ and $[SEP]$ are
special tokens that mark start and end. Then we use a transformer to encode
the inputs into $L+2$ hidden states
$H_{t}={h_{[CLS]},h_{t_{1}},...,h_{t_{L}},h_{[SEP]}}$ and get the text
representation $R_{t}$ by pooling the hidden states of $[CLS]$ token.

$H_{t}=Transformer([[CLS];T;[SEP]])$ (3) $R_{t}=Pool(h_{[CLS]})$ (4)

###  3.2 Training Objective

Following CLIP Radford et al. (2021) and LightningDOT Sun et al. (2021), we
jointly train the image encoder and text encoder to predict the image-text
matching score $Score(V,T)$. The score is represented by the inner product of
the image representation and the text representation.

$Score(V,T)=\langle R_{v},R_{t}\rangle$ (5)

We use a contrastive learning objective to train the two encoders. Given a
batch of $B\times B$ image-text pairs, the training objective is to maximize
the matching scores of the $B$ matched image-text pairs, i.e. in-batch
contrastive loss $loss_{cl}$. It consists of the image-to-text retrieval loss
$loss_{i2t}$ and the text-to-image loss $loss_{t2i}$.

Specifically, for image-to-text retrieval, we compute the similarity between
each image and all texts, where $Score_{i,j}$ represents the matching score of
$i$th image and $j$th text. For the image-to-text retrieval of the $i$th image
as query, the positive example is the $i$th text and the other texts in a
batch are negative examples. The loss of image-to-text retrieval is:

$loss_{i2t}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{e^{Score(V_{i},T_{i})}}{\sum_{j=1}^{B}e^{Score(V_{i},T_{j})}}$
(6)

Similarly, we calculate the text-to-image retrieval loss $loss_{t2i}$ and the
final contrastive loss is:

$loss_{cl}=\frac{1}{2}(loss_{i2t}+loss_{t2i})$ (7)

###  3.3 Encode with Multi-View Attention Model (MVAM)

To enhance the representation ability of the two-stream match model and encode
more information, we propose a Multi-View Attention Model (MVAM) to encode
multi-modal data in different views with multi-attention heads. The image or
text is represented by multiple vectors in our model, instead of one vector by
traditional two-stream models. To promote diversity among multiple views, we
use a diversity loss, which can further improve the quality of representation
and brings better matching performance.

####  3.3.1 Model Architecture

The architecture of MVAM with ViT encoder is shown in Figure 2. Given an
image-text pair $(V,T)$, after encode them into hidden states
$H_{v}={h_{[IMG]},h_{v_{1}},...,h_{v_{N}}}$ and
$H_{t}={h_{[CLS]},h_{t_{1}},...,h_{t_{L}},h_{[SEP]}}$ with ViT-based encoder,
instead of using pooled hidden states $R_{v}$ or $R_{t}$ as representations,
we obtain the multi-viewed representations by multi-attention heads.

Take the image as an example. To obtain diverse visual representation in $m$
views, we learn $m$ view codes $(c_{v}^{1},...,c_{v}^{m})$ as the queries for
attention, where $c_{v}^{i}\in\mathbb{R}^{D}$ and $D$ is the dimension of
$h_{v_{i}}$. Besides, we propose $m$ view attention
$(a_{v}^{1},...,a_{v}^{m})$, where $a_{v}^{i}$ is calculated by image hidden
state $H_{v}$ and the $i$th view code.

$a_{v}^{i}=softmax(H_{v}\ast c_{v}^{i})$ (8)

Then, we obtain $m$ diverse view features $(F_{v}^{1},...,F_{v}^{m})$, where
$F_{v}^{i}$ is computed through the weighted sum of $H_{v}$ by the $i$th view
attention $a_{v}^{i}$.

$F_{v}^{i}=\sum\limits_{j=0}^{N}{a_{v}^{(i,j)}*H_{v}^{j}}$ (9)

Finally, we concatenate the $m$ diverse view features together to form the
final representation $multi-R_{v}$ of an image as follows:

$multi-R_{v}=[F_{v}^{1};F_{v}^{2};...;F_{v}^{m}]$ (10)

Similarity, we can get the multi-view representation $multi-R_{t}$ of text:

$multi-R_{t}=[F_{t}^{1};F_{t}^{2};...;F_{t}^{m}]$ (11)
$F_{t}^{i}=\sum\limits_{j=0}^{L+1}{a_{t}^{(i,j)}*H_{t}^{j}}$ (12)
$a_{t}^{i}=softmax(H_{t}\ast c_{t}^{i})$ (13)

###  3.4 Multi-View Diversity Loss

To get representation containing comprehensive information and expand the
diversity of multi-view attention, we constrain the $m$ view codes to focus on
different aspects. Following Li et al. (2018); Lin et al. (2017), we use a
diversity loss to enhance the differences between the multi-view attention
weights so that the multiple view codes could generate various highlights of
an image or a text.

Take the image view-attention as an example, we obtain the similarity between
each view attention by the multiplication of $A_{v}$ and its transpose, where
$A_{v}$ is $[a_{v}^{1},...,a_{v}^{m}]$. Then, we subtract it by an identity
matrix and regard it as a measure of diversity. The diversity grows as the
decrease of non-diagonal values of the similarity matrix. So that we introduce
a diversity loss $loss_{div}^{v}$ using:

$loss_{div}^{v}=\parallel(A_{v}A_{v}^{T}-I)\parallel_{F}^{2}$ (14)

Where $I$ is a $m$-dimensional identity matrix, which is used to remove the
$m$ view attention’s self-correspondences on the diagonal of the similarity
matrix. $\parallel\bullet\parallel_{F}$ stands for the Frobenius norm of a
matrix. We apply the same similar operation to text multi-view attention
diversity, and obtain the $loss_{div}^{t}$.

The final loss is shown as the Eq. (16), we multiplie the sum of
$loss_{div}^{v}$ and $loss_{div}^{v}$ by a coefficient $\beta$, and minimize
it together with the contrastive loss from Eq. (7).

$loss_{div}=loss_{div}^{v}+loss_{div}^{t}$ (15)
$loss=loss_{cl}+\beta(loss_{div})$ (16)

Table 1: Performance of all models on MSCOCO (1K/5K). We bold the highest scores separately based on each base model. In brackets is the increase of average performance compared with base models with attention pooling. Tasks | MSCOCO-1K | MSCOCO-5K
---|---|---
Recall | i2t@1 | i2t@5 | i2t@10 | t2i@1 | t2i@5 | t2i@10 | avg | i2t@1 | i2t@5 | i2t@10 | t2i@1 | t2i@5 | t2i@10 | avg
PVSE | 69.2 | 91.6 | 96.6 | 55.2 | 86.5 | 93.7 | 82.13 | 45.2 | 74.3 | 84.5 | 32.4 | 63.0 | 75.0 | 62.4
SCAN | 72.7 | 94.8 | 98.4 | 58.8 | 88.4 | 94.8 | 84.65 | 50.4 | 82.2 | 90.0 | 38.6 | 69.3 | 80.4 | 68.48
VisualSparta | - | - | - | 68.7 | 91.2 | 96.2 | - | - | - | - | 45.1 | 73.0 | 82.5 | -
LightningDOT | - | - | - | - | - | - | - | 60.1 | 85.1 | 91.8 | 45.8 | 74.6 | 83.8 | -
Base-ViT |
+[CLS] Pooling | 65.65 | 90.52 | 96.07 | 54.06 | 85.88 | 93.6 | 80.96 | 40.13 | 70.43 | 82.23 | 30.66 | 61.63 | 74.29 | 59.9
+Attn Pooling | 65.79 | 90.62 | 95.87 | 55.12 | 86.44 | 93.83 | 81.28 | 41.56 | 70.67 | 81.61 | 32.59 | 62.9 | 75.24 | 60.76
+MVAM | 70.02 | 92.09 | 96.75 | 56.69 | 87.35 | 94.46 | 82.89 ($\uparrow$1.62) | 46.6 | 75.33 | 84.64 | 33.54 | 64.53 | 76.46 | 63.52 ($\uparrow$2.76)
Base-FRCNN |  |  |
+[CLS] Pooling | 69.4 | 92.96 | 97.52 | 56.79 | 87.81 | 94.82 | 83.22 | 43.38 | 75.52 | 85.88 | 32.96 | 64.62 | 77.01 | 63.23
+Attn Pooling | 72.3 | 93.92 | 97.7 | 58.53 | 88.22 | 94.74 | 84.24 | 47.46 | 77.66 | 87.34 | 35.25 | 66.48 | 78.02 | 65.37
+MVAM | 77.0 | 95.34 | 98.28 | 62.11 | 90.04 | 95.64 | 86.40($\uparrow$2.17) | 54.08 | 82.46 | 90.06 | 39.08 | 69.94 | 80.92 | 69.42($\uparrow$4.06)
Base-CLIP |  |  |
+[CLS] Pooling | 80.86 | 96.08 | 98.44 | 67.54 | 91.74 | 96.5 | 88.53 | 62.93 | 85.32 | 91.66 | 47.49 | 74.81 | 84.06 | 74.38
+Attn Pooling | 81.64 | 96.62 | 98.48 | 68.36 | 91.85 | 96.78 | 88.96 | 65.11 | 87.38 | 93.20 | 48.62 | 75.66 | 84.38 | 75.73
+MVAM | 83.14 | 97.04 | 99.02 | 69.35 | 92.33 | 97.12 |  89.67($\uparrow$0.71) | 66.81 | 88.3 | 93.88 | 49.47 | 76.48 | 84.9 |  76.64($\uparrow$0.92)
Table 2: Performance of all models on Flickr30K datasets. Tasks | FLickr30K
---|---
Recall | i2t@1 | i2t@5 | i2t@10 | t2i@1 | t2i@5 | t2i@10 | avg
SCAN | 67.4 | 90.3 | 95.8 | 48.6 | 77.7 | 85.2 | 77.5
VisualSparta | - | - | - | 57.1 | 82.6 | 88.2 | -
LightningDOT | 83.9 | 97.2 | 98.6 | 69.9 | 91.1 | 95.2 | 89.32
Base-ViT |
+[CLS] Pooling | 48.01 | 74.87 | 84.58 | 39.0 | 71.05 | 80.86 | 66.4
+Attn Pooling | 55.2 | 83.0 | 89.9 | 43.66 | 73.96 | 83.16 | 71.48
+MVAM | 55.57 | 82.53 | 90.09 | 44.54 | 75.04 | 84.49 | 72.04 ($\uparrow$0.56)
Base-FRCNN |
+[CLS] Pooling | 51.2 | 78.4 | 86.3 | 39.98 | 71.36 | 81.32 | 68.09
+Attn Pooling | 57.3 | 83.25 | 91.22 | 47.6 | 77.73 | 85.99 | 73.85
+MVAM | 69.3 | 90.4 | 94.5 | 53.14 | 80.68 | 88.32 | 79.39 ($\uparrow$5.54)
Base-CLIP |
+[CLS] Pooling | 89.6 | 98.3 | 99.5 | 76.32 | 93.88 | 96.54 | 92.36
+Attn Pooling | 90.5 | 98.1 | 99.4 | 75.06 | 93.38 | 96.3 | 92.12
+MVAM | 91.8 | 98.7 | 99.4 | 76.44 | 94.22 | 97.16 |  92.95 ($\uparrow$0.83)
Table 3: The comparisons of our model with ensemble methods. Tasks | MSCOCO-1K | MSCOCO-5K
---|---|---
ModelType | i2t@1 | i2t@5 | i2t@10 | t2i@1 | t2i@5 | t2i@10 | i2t@1 | i2t@5 | i2t@10 | t2i@1 | t2i@5 | t2i@10
Base-FRCNN-ensemble | 73.54 | 94.16 | 97.74 | 59.75 | 89.32 | 95.34 | 49.6 | 79.42 | 87.78 | 36.31 | 67.87 | 79.52
MVAM-FRCNN | 77.0 | 95.34 | 98.28 | 62.11 | 90.04 | 95.64 | 54.08 | 82.46 | 90.06 | 39.08 | 69.94 | 80.92
Table 4: Ablation results of the number of views. Tasks | MSCOCO-1K | MSCOCO-5K
---|---|---
#Views | i2t@1 | i2t@5 | i2t@10 | t2i@1 | t2i@5 | t2i@10 | i2t@1 | i2t@5 | i2t@10 | t2i@1 | t2i@5 | t2i@10
1 | 72.3 | 93.92 | 97.7 | 58.53 | 88.22 | 94.74 | 47.46 | 77.66 | 87.34 | 35.25 | 66.48 | 78.02
8 | 76.26 | 94.8 | 98.22 | 61.27 | 90.12 | 95.79 | 53.52 | 81.28 | 89.66 | 38.52 | 69.24 | 80.62
16 | 77.0 | 95.34 | 98.28 | 62.11 | 90.04 | 95.64 | 54.08 | 82.46 | 90.06 | 39.08 | 69.94 | 80.92
32 | 75.92 | 95.22 | 98.26 | 61.44 | 89.97 | 95.71 | 53.9 | 81.72 | 89.9 | 38.28 | 69.35 | 80.59

##  4 Experiments

We carry out extensive experiments to evaluate the effects of the Multi-View
Attention Model (MVAM) based on various two-stream models on image-text
matching task. To distinguish MVAM from the conventional ensemble models, we
also conduct a detailed ablation study.

###  4.1 Datasets

We use MSCOCOLin et al. (2014) 111https://cocodataset.org and Flickr30KYoung
et al. (2014) 222https://bryanplummer.com/Flickr30kEntities/ datasets for
training and evaluating our model. There are 123K images in the MSCOCO dataset
and 31K images in the Flickr30K dataset. Each image has 5 caption texts.
Following Karpathy and Fei-Fei (2015), the MSCOCO dataset is divided into
114k/5k/5k for training, validation, and testing. We also use a common split
of 1k tests for a more detailed evaluation. The Flickr30K dataset is divided
into 29K/1K/1K for training, validation, and testing.

###  4.2 Metric

As is common in cross-modal retrieval, we consider the performance of image-
to-text retrieval (i2t) and text-to-image retrieval (i2i), and the performance
is measured by recall at $top-k$ candidates ($R@k$).

###  4.3 Baselines

We select PVSE Song and Soleymani (2019), SCAN Lee et al. (2018), LightningDOT
Sun et al. (2021) and VisaulSparta Lu et al. (2021) from prior works as our
baselines. And, we implement 3 additional two-stream models based on 3 image
encoders: Base-FRCNN, Base-ViT, and Base-CLIP.

Base-FRCNN takes the $\text{BERT}_{base}$ as the text encoder and uses a
FRCNN-based image encoder. The text encoder is initialized from bert-base-
uncased Devlin et al. (2018b). The image encoder is a randomly initialized
transformer.

Base-ViT uses the same text encoder as Base-FRCNN but uses a ViT based image
encoder. The ViT is initialized from ViT-B/16 Wu et al. (2020); Deng et al.
(2009).

Base-CLIP uses the same architecture as CLIP, and the parameters are
initialized from the pre-trained CLIP(ViT-B/16) Radford et al. (2021).

For a more comprehensive comparison, we design two ways to get the final
representations of images and texts, i.e. [CLS] pooling and attention pooling.
The former uses a linear projection following a tanh function to map the
hidden states of [CLS] (or [IMG]) to a 1024-dim vector. The latter first maps
the hidden states to 1024-dim vectors with a linear projection layer and then
aggregate them by an attention head with only one view code.

###  4.4 Implementation Details

To compare with the 3 base models and prior works, we implement 3 MVAM
variants: MVAM-FRCNN, MVAM-ViT, and MVAM-CLIP, which are respectively based on
Base-FRCNN, Base-ViT, and Base-CLIP. For all models, we design $16$ view codes
for images or texts to encode the hidden states produced by the base model
into $16$-view features. Coefficient $\beta$ for diversity is set to 10. All
experiments are run on 8$\times$ NVIDIA V100 GPUs. We train our models using a
batch size of 1024 over 20 epochs. The learning rates of Base-FRCNN, Base-ViT,
MVAM-FRCNN, and MVAM-ViT are $5e-5$. While in Base-CLIP, the learning rate is
set to $5e-8$ for preserving the multi-modal alignment learned in CLIP. For
MVAM-CLIP, We train it in 2 stages: firstly training view codes with the fixed
Base-CLIP using $5e-5$ learning rate, secondly fine-tuning all parameters of
MVAM-CLIP using $5e-8$ learning rate. The best development set accuracy from 3
random restarts is reported, with the standard deviation.

###  4.5 Experiments Results

The results of all models on MSCOCO and Flickr30K are shown in Table 1 and
Table 2. Base-ViT, Base-FRCNN, and Base-CLIP are the base models, "+[CLS]
pooling" and "+Attn pooling" denote the two baseline methods with different
pooling functions, and "+MVAM" denotes our method. We can see that MVAM makes
great progress based on varied two-stream matching models. What’s more, the
MVAM-CLIP outperforms PVSE, SCAN, LightningDOT and VisualSparta, in which
LightningDOT and VisualSparta are pre-trained on more multi-modal datasets
with several learning objectives.

Compared with Base-ViT, Base-FRCNN, and Base-CLIP with attention pooling, we
observe 1.62%, 2.17%, 0.71% increased average scores through MVAM on
MSCOCO-1K, and 2.76%, 4.06%, 0.92% increased average scores on MSCOCO-5K. The
increased average scores on Flickr30K are 0.56%, 5.54%, and 0.83%. Besides,
Our method has the greatest improvement for the FRCNN-based model, which means
MVAM is more effective on region-based image features. We guess that this is
because the object region-based features of FRCNN contain more complete
semantic information than ViT, so it is more conducive to our method to focus
on key points.

Please note that even though multiple views are used, our method is still a
strict two-stream model, and only needs to calculate a cosine similarity for
each pre-computed image-text representation pairs during inference, so it does
not increase the inference time.

###  4.6 Qualitative Case Analysis

Figure 3: The retrieved top5 images of CLIP and MVAM-CLIP. We lay out images
according to the retrieval score and the ground-truth images are in green
boxes. For long and complex text queries, the images retrieved by MVAM-CLIP
are more suitable.

Figure 3 shows the images retrieved based on the input text queries with CLIP-
based models on MSCOCO dataset. As we can see, the images retrieved by MVAM-
CLIP are more consistent with the text queries. Even for long and complex
query inputs, our method can still retrieve the correct image. As the first
example shows, although the CLIP model retrieves an image containing "white
fondant cake on pedestal", the top-ranked image and text do not exactly match
due to ignoring a lot of details in the text, such as "white and green roses",
"dark wire wide-mesh birdcage", and "a bird perched". In contrast, the top-1
MVAM-CLIP retrieved image completely satisfies the request of the query. This
shows that the MVAM, which utilizes multi-views to encode images and texts, is
able to encode more details into the image and text and can help the two-
stream model to perform well especially when retrieving complex queries.

##  5 Ablation Study and Model Analysis

In the following paragraphs, we sequentially present our ablation studies and
analysis of (1) the effectiveness of multi-view, (2) the number of views, and
(3) multi-view interpretability. The necessity of multi-view diversity loss
can be found in Appendix A.

In these experiments, to exclude the influence of the pre-trained cross-modal
alignment knowledge in CLIP, we choose the FRCNN-based model for analysis.
Besides, the region-based features in RCNN are more helpful to understand the
regions concerned by different views attention in our method.

###  5.1 The Effectiveness of Multi-view

In order to validate that the performance improvement owes to the introduction
of multi-views rather than ensemble, which integrates the outputs of several
different models to predict the result, we carry out the following ablation
studies with the Base-FRCNN model using attention pooling on MSCOCO. For the
ensemble setting, we use $16$ independent Base-FRCNN model with $64$-dim
representation vector through attention pooling and concatenate them into a
$1024$-dim vector for image-text matching. The $16$ independent Base-FRCNN
models are based on the same network architectures, and randomly initialized
parameters with $16$ seed during training.

The results are shown in Table 3. As we can see, our method outperforms the
ensemble models by a large margin. This proves it is the higher quality
representation learned with multi-view attention rather than the model
ensemble that boosts the performance.

Figure 4: The visualization of attention from different views. The left side
provides the attention scores over text for all the 16 attention heads of
MVAM, and the darker the grid color represents the greater the attention
value. The right side shows the regions in the image that four of the view
attention heads pay most attention to.

###  5.2 The Numbers of Views

We run extensive experiments on studying the influence of views to search for
the best settings. We train the MVAM-FRCNN with different number of view codes
$m$. Results are reported in Table 4. We can see that as the number of views
increases, the matching accuracy shows a trend of first increasing and then
decreasing. Among them, the best performance is obtained when the number of
views is 16.

We speculate that the reason may be that if the number of views is too small,
some key content in the image or sentence may be ignored, which is not able to
extract enough information for image-text matching. If the number of views is
too large, there may be redundancy in the representation. Besides, longer
representation vectors may lead to the model being more difficult to train.

###  5.3 Multi-view Interpretability

To find out what our model "see" from multiple views, we analyze the attention
map of the image and text by multi-view over an example. An attention
visualization is provided in Figure 4. As we can see, the view attention
exhibits a sparse and diverse pattern. For a single view, each view focuses on
certain regions and presents a relatively sparse attention map. Comparing
different views, the content concerned with different views is different. This
illustrates that our method is able to encode images and texts from different
aspects and attend to more salient regions, which promotes the representations
to contain more information. By jointly observing the attention map between
image and text from different views, it can be found a one-to-one mapping
relationship. For example, View-4 of the text focuses on "holding onto" and
View-4 of the image also pays attention to the region of "holding a rope".
This demonstrates that MVAM could consider complex content comprehensively,
and match multiple meanings in images and texts, which provides more
interaction for two-stream matching.

##  6 Conclusions

In this paper, we propose a Multi-View Attention Model (MVAM) to improve the
representation quality of the two-stream model for image-text matching.
Instead of encoding the image or text into a single vector, our model encodes
them with the aggregation of multi representations from diverse views, which
is able to encode more comprehensive information and bring more interaction
for two-stream matching. As a pluggable module, MVAM can be combined with a
variety of two-stream models. Experiment results on MSCOCO and Flickr30K show
that with higher-quality representations, our method can improve the matching
performance by a significant margin. In addition, we conduct extensive
experiments on studying the hyperparameters and loss functions of MVAM to
search for the best model settings. The visualization of the multiple view
attention for image and text helps us to understand what the different views
concern and proves that our model is able to integrate diverse information
from both images and texts.

##  7 Limitations

The limitation of our work is that we did not apply it to video-text matching.
As temporal data, a video often contains too much information, so its caption
is usually related to some keyframes, i.e. the most matching aspect. MVAM
might be a solution worth trying to attend to more key points of a video from
different views. Future work entails applying MVAM to video-text matching, and
more matching tasks with two-stream architecture.

## References

  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6077–6086.
  * Chen et al. (2019) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019.  Uniter: Learning universal image-text representations.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee.
  * Devlin et al. (2018a) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018a.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Devlin et al. (2018b) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018b.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Faghri et al. (2017) Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2017.  Vse++: Improving visual-semantic embeddings with hard negatives.  _arXiv preprint arXiv:1707.05612_.
  * Humeau et al. (2019) Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019.  Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring.  _arXiv preprint arXiv:1905.01969_.
  * Huo et al. (2021) Yuqi Huo, Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, Guoxing Yang, Jingyuan Wen, Heng Zhang, Baogui Xu, Weihao Zheng, et al. 2021.  Wenlan: Bridging vision and language by large-scale multi-modal pre-training.  _arXiv preprint arXiv:2103.06561_.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning_ , pages 4904–4916. PMLR.
  * Karpathy and Fei-Fei (2015) Andrej Karpathy and Li Fei-Fei. 2015.  Deep visual-semantic alignments for generating image descriptions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3128–3137.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  Colbert: Efficient and effective passage search via contextualized late interaction over bert.  In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_ , pages 39–48.
  * Lee et al. (2018) Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018.  Stacked cross attention for image-text matching.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 201–216.
  * Li et al. (2020a) Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020a.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 11336–11344.
  * Li et al. (2018) Shuang Li, Slawomir Bak, Peter Carr, and Xiaogang Wang. 2018.  Diversity regularized spatiotemporal attention for video-based person re-identification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 369–378.
  * Li et al. (2020b) Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020b.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _European Conference on Computer Vision_ , pages 121–137. Springer.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer.
  * Lin et al. (2017) Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017.  A structured self-attentive sentence embedding.  _arXiv preprint arXiv:1703.03130_.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _Advances in neural information processing systems_ , 32.
  * Lu et al. (2021) Xiaopeng Lu, Tiancheng Zhao, and Kyusong Lee. 2021.  Visualsparta: An embarrassingly simple approach to large-scale text-to-image search with weighted bag-of-words.  _arXiv preprint arXiv:2101.00265_.
  * Luan et al. (2021) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021.  Sparse, dense, and attentional representations for text retrieval.  _Transactions of the Association for Computational Linguistics_ , 9:329–345.
  * Qi et al. (2020) Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. 2020.  Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data.  _arXiv preprint arXiv:2001.07966_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28.
  * Song and Soleymani (2019) Yale Song and Mohammad Soleymani. 2019.  Polysemous visual-semantic embedding for cross-modal retrieval.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1979–1988.
  * Sun et al. (2021) Siqi Sun, Yen-Chun Chen, Linjie Li, Shuohang Wang, Yuwei Fang, and Jingjing Liu. 2021.  Lightningdot: Pre-training visual-semantic embeddings for real-time image-text retrieval.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 982–997.
  * Wang et al. (2019) Zihao Wang, Xihui Liu, Hongsheng Li, Lu Sheng, Junjie Yan, Xiaogang Wang, and Jing Shao. 2019.  Camp: Cross-modal adaptive message passing for text-image retrieval.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5764–5773.
  * Wu et al. (2020) Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. 2020.  Visual transformers: Token-based image representation and processing for computer vision.
  * Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _Transactions of the Association for Computational Linguistics_ , 2:67–78.
  * Zhang et al. (2020) Qi Zhang, Zhen Lei, Zhaoxiang Zhang, and Stan Z Li. 2020.  Context-aware attention network for image-text retrieval.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3536–3545.

##  Appendix A Appendix

###  A.1 Diversity Loss Analysis

We consider two diversity losses: base diversity loss (ours) Lin et al. (2017)
and square root diversity loss Li et al. (2018).

Base diversity loss $loss_{div}$ forces the area of interest not to overlap
and encourages them to be concentrated in a single region. The implementation
of the base diversity loss is in Eq. (15).

Square root diversity loss $loss_{div}^{\prime}$ is a variant of base
diversity loss. It allows more large salient regions from a higher level. The
loss function of square root diversity loss is as follows:

$loss_{div}^{\prime}=\parallel(EE^{T}-I)\parallel_{F}^{2}$ (17) $E=\sqrt{A}$
(18)

The results of MVAM-FRCNN trained without diversity loss, with base diversity
loss, and with square root diversity loss are shown in Table 5 and Table 6.
The results reveal that the diversity of multi-view plays an important role in
our method. Moreover, the performance with base diversity loss is superior. We
surmise this is because the text in datasets like COCO or Flickr30K is mostly
descriptive, so it would be more effective to encourage the model to focus on
isolated regions. But for some more abstract retrievals, it may be better to
use the square root diversity loss, which needs to be explored in the future.

Table 5: Ablation results of diversity loss on MSCOCO-1k datasets. Tasks | MSCOCO-1k
---|---
Diversity Loss | i2t@1 | i2t@5 | i2t@10 | t2i@1 | t2i@5 | t2i@10
w/o diversity loss | 76.18 | 95.4 | 98.38 | 61.74 | 90.08 | 95.88
square root diversity loss | 76.7 | 94.96 | 98.22 | 61.77 | 90.06 | 95.73
base diversity loss | 77.0 | 95.34 | 98.28 | 62.11 | 90.04 | 95.64
Table 6: Ablation results of diversity loss on MSCOCO-5k datasets. Tasks | MSCOCO-5k
---|---
Diversity Loss | i2t@1 | i2t@5 | i2t@10 | t2i@1 | t2i@5 | t2i@10
w/o diversity loss | 54.06 | 81.98 | 90.04 | 38.64 | 69.65 | 80.74
square root diversity loss | 54.14 | 82.44 | 90.32 | 38.82 | 69.8 | 80.73
base diversity loss | 54.08 | 82.46 | 90.06 | 39.08 | 69.94 | 80.92


# Flares from merged magnetars:
their prospects as a new population of gamma-ray counterparts of binary
neutron star mergers

Shu-Xu Yi  Zhen Zhang  Xilu Wang  Key Laboratory of Particle Astrophysics,
Institute of High Energy Physics, Chinese Academy of Sciences, 19B Yuquan
Road, Beijing 100049, China

(October 2022)

###### Abstract

Long-lived massive magnetars are expected to be remnants of some binary
neutron star (BNS) mergers. In this paper, we argue that the magnetic powered
flaring activities of these merged magnetars would occur dominantly in their
early millisecond-period-spin phase, which is in the timescale of days. Such
flares endure significant absorption by the ejecta from the BNS collision, and
their detectable energy range is from 0.1-10 MeV, in a time-lag of $\sim$ days
after the merger events indicated by the gravitational wave chirps. We
estimate the rate of such flares in different energy ranges, and find that
there could have been 0.1-10 cases detected by Fermi/GBM. A careful search for
$\sim 10$ milliseconds spin period modulation in weak short gamma-ray bursts
(GRBs) may identify them from the archival data. Future MeV detectors can
detect them at a rate from a few to tens per year. The recent report on the
Quasi-Period-Oscillation found in two BASTE GRBs should not be considered as
cases of such flares, for they were detection in a lower energy range and with
a much shorter period spin modulation.

††thanks<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
##  1 Introduction

Magnetars are a kind of neutron stars (NSs) which have extremely strong
magnetic fields. The magnetar’s magnetic field can be as strong as $\sim
10^{13-15}$ G (Ferrario & Wickramasinghe, 2008; Rea & Esposito, 2011;
Mereghetti et al., 2015; Turolla et al., 2015; Kaspi & Beloborodov, 2017),
while that of an ordinary NS is $\sim 10^{10-12}$ G (but see “low-magnetic
field” magnetars (Rea et al., 2010; Turolla & Esposito, 2013)). The typical
radiation activities are believed to be powered by the huge energy reservoir
in the magnetic fields of magnetars, rather than their rotational energy or
gravitational energy as those in spin powered or accretion powered NSs. Such
magnetars radiation activities were observed as anomalous X-ray pulsars (AXPs)
and Soft Gamma-ray Repeaters (SGRs). AXPs appear to be isolated pulsars with
X-ray emission, whose spin down luminosity are thought to be insufficient to
power their observed luminosity (Fahlman & Gregory, 1981; Gavriil et al.,
2002; Kaspi & Gavriil, 2004); SGRs are thought to be magnetars which give off
bursts in gamma-ray at irregular time intervals (Golenetskii et al., 1984;
Norris et al., 1991; Hartmann, 1995; Thompson & Duncan, 1995; Mereghetti,
2008). Besides, there are rare cases, where much more energetic flares are
emitted from magnetars, which are referred to as “Giant Flares” (Palmer et
al., 2005; Hurley et al., 2005; Minaev & Pozanenko, 2020; Zhang et al., 2020;
Roberts et al., 2021; Svinkin et al., 2021).

The latter two flaring activities are believed to originate from magnetic
energy releasing from occasional magnetic field recombination of magnetars.
There are various theories to explain the underlying triggers of such
recombination. Following the dichotomy of Sharma et al. (2023), the first
class of mechanisms attributes the trigger to crustal destructive or defective
events (the “star quake” paradigm, see models e.g., Blaes et al. 1989;
Thompson & Duncan 1996; Levin 2006; Beloborodov 2020; Bransgrove et al. 2020;
Yuan et al. 2020); while the second attributes to the reconfiguration in the
twisted magnetosphere due to some instabilities (the “solar flare” paradigm,
e.g., Lyutikov 2003; Komissarov et al. 2007; Ripperda et al. 2019; Mahlmann et
al. 2019).

Unlike those isolated evolved magnetars, there is a population of magnetars
that were born in the remnants of binary neutron star (BNS) collisions. We
refer to these magnetars as merged-magnetars, which are the focus of this
manuscript. It is widely believed that, BNS collision, if the remnant is not
massive enough to cause a prompt collapse into a black hole (BH), will result
a massive magnetar in millisecond time scale (Duncan & Thompson, 1992; Usov,
1992; Thompson, 1994; Yi & Blackman, 1998; Blackman & Yi, 1998; Kluźniak &
Ruderman, 1998; Nakamura, 1998; Spruit, 1999; Wheeler et al., 2000; Ruderman
et al., 2000). Such a magnetar inherits most of the orbital angular momentum
of the progenitor NSs, therefore possesses short spin period of milliseconds
initially, much shorter than those of typical magnetars of seconds. Because of
their faster spin, larger mass and younger age, it’s intuitively to suspect
that they are even less stable than isolated evolved magnetars, and therefore
could be more likely to emit gamma-ray flares.

In this work, we will investigate the possibility that flares from merged-
magnetars be identified, and especially as electromagnetic wave counterparts
(EMC) of gravitational wave (GW) signals of BNS mergers. The manuscript is
arranged as follows: In section II, we give semi-quantitative arguments that
the flare activities should dominantly occur in the merged magnetar’s early
phase. Then in the next section, the detectable event rate of merged-magnetar
flares is estimated from a population of BNS mergers. In section III, we
consider their potential to be identified as EMC of GW, following by a section
where the absorption from ejected matter is taken into consideration. We
discuss several relevant aspects and summarize the main findings in the last
two sections.

##  2 Flare mechanisms

In the “star quake” paradigm, where gamma-ray flares are triggered by crustal
defective events, when the magnetar is fast spinning down due to the gigantic
magnetic breaking torque, the centrifugal force decreases rapidly. The crust
of the NS will re-adjust to its new balance configuration, where the
centrifugal force is counteracting against the self-gravity. In this re-
configuring process, crustal defective events can be expect to be much
frequent than when the NS entered the later slowly spinning-down stage (such
spin-down induced star quake was first discuss by Baym & Pines 1971); in the
second paradigm, where gamma-ray flares are attributes to the instabilities in
the magnetosphere, the boundary of the magnetar’s magnetosphere (near the
light cylinder, whose radius $R_{\rm LC}=cP$, where $c$ is the velocity of
light, and $P$ is the spin period) expand fast as its spin period rapidly
increase. The magnetic field lines near the boundary will have to re-adjust
accordingly, and thus to be expect to be more likely to give-off gamma-ray
flares111unlike in those conventional models in the “solar flare” paradigm,
where the magnetic twist leaks from the NS interior into the magnetosphere,
through the slow crustal deformation, here the magnetic twist fast expands
along with the magnetosphere. Therefore we refer to it as the “magnetosphere
instability” paradigm for distinction. In fact, from our following semi-
quantitative estimation we show that, the rate of gamma-ray flares in the fast
spinning-down millisecond period phase is much higher than its second period
phase, so that the total flare energy released in the former phase (with the
time scale of days) is comparable or higher than the latter phase (with the
time scale of 105 years).

(a) Crust crack scenarios: Crustal defective events occurs more frequently in
the stage when the magnetar fast spins down, since the centrifugal force which
is counteracting the NS’ self-gravity is rapidly decreasing.

(b) Magnetosphere instability scenarios: Instabilities will be triggered
frequently in the stage when the magnetar fast spins down, the boundary of the
magnetar’s magnetosphere is rapidly expanding.

Figure 1: Illustrations of two scenarios in which the fast spinning-down
millisecond magnetar is more probable to have gamma-ray flares

In the crust crack scenarios, let us assume that each crustal destructive
event which is energetic enough to trigger a magnetar gamma-ray flare has a
characteristic energy $E_{\rm crack}$. Denote the change in the centrifugal
force as $\Delta F_{\rm cent}$ in a small interval of time $\Delta t$. The
corresponding linear deformation of the NS is $\Delta l$, which we suppose is
proportional to $\Delta F_{\rm cent}$ as

$\Delta l=\frac{\Delta F_{\rm cent}}{\kappa},$

where $\kappa$ is the elastic factor of the crust. The work done by the
gravity is thus:

$\Delta E=\frac{\Delta F_{\rm cent}}{\kappa}F_{\rm G},$ (1)

where $F_{\rm G}$ denote the gravitational force acting on the crust, which
remains almost constant as the deformation is small. If we equalize the work
done by gravity with that released in crust cracks, we have the following
equation:

$\frac{\Delta F_{\rm cent}}{\kappa}F_{\rm G}=\Delta N_{\rm crack}E_{\rm
crack},$ (2)

where $\Delta N_{\rm crack}$ is the number of crust cracks in $\Delta t$,
where their ratio is the magnetar gamma-ray flare rate

$R_{\rm B}=\frac{\Delta N_{\rm crack}}{\Delta t}$

, which, according to equation (2), has the following proportional
relationship:

$R_{\rm B}\propto\dot{F}_{\rm cent}\propto\frac{\dot{P}}{P^{3}}.$ (3)

Note that if the spinning-down torque is dominated by the magnetic dipole
braking222GW braking will only dominate over the magnetic braking when the
spin period is less than 0.1 s (Usov, 1992; Blackman & Yi, 1998). In this
case, GW braking will fast spin-down the magnetar to a regime where the
magnetic braking takes over (Zhang & Mészáros, 2001)., then we have:

$B^{2}_{\rm s}\propto P\dot{P},$ (4)

where $B_{\rm s}$ the surface magnetic field strength, we assume to be an
constant. As a result, equation (3) becomes:

$R_{\rm B}\propto\frac{1}{P^{4}}.$ (5)

Now, since in the first phase, the spin period is in the order of $\sim 10$
ms, and in the second phase it is $\sim$s, the $R_{\rm B}$ in the first phase
can be eight orders of magnitude larger than that in the second phase. On the
other hand, the time-span of the first phase is about $10^{-8}-10^{-7}$ of
that of the second phase. Therefore, the magnetar gamma-ray flare energy
releasing in the first phase is the same or one order of magnitude larger than
that in the second phase, as we claimed in above.

If we are in the magnetosphere instability scenarios, in a short time interval
$\Delta t$, the boundary of the magnetosphere (where close and open magnetic
field lines transit) expand in distance: $\Delta R_{\rm LC}$. The volume which
has been swept is:

$\Delta V=4\pi R_{\rm LC}^{2}\Delta R_{\rm LC}.$ (6)

The volume times the magnetic field energy density is the energy got evolved.
This energy is likely to be release by process such as magnetic field re-
connection near the light cylinder. We have:

$\Delta E\propto B_{r=R_{\rm LC}}^{2}R_{\rm LC}^{2}\Delta R_{\rm LC},$ (7)

where $B_{r=R_{\rm LC}}$ is the magnetic field strength at the light cylinder.
For a dipole magnetic field,

$B_{r=R_{\rm LC}}\propto\frac{B_{\rm s}~{}}{R_{\rm LC}^{3}}.$

Consequently, equation (7) can be reformed into:

$\Delta E\propto B^{2}_{\rm s}\frac{\Delta R_{\rm LC}}{R^{4}_{\rm
LC}}\propto\frac{\dot{P}}{P^{4}}\propto\frac{1}{P^{5}}.$ (8)

Using the similar argument as in the crust crack scenarios, we can see the
ratio of the $R_{\rm B}$ between the first and second phase can be ten orders
of magnitude, and thus the ratio of the corresponding total energy releasing
can be $10^{3}$ in the magnetosphere instability scenarios.

##  3 The rate of flares from the population of merged magnetars

Define the magnetar flare number density distribution from a single merged-
magnetar as: $n(\tau,E)$, where $\tau$ is the age of the magnetar, and $E$ is
the energy release during the flare. The total number of flares above some
certain energy limit ($E_{\rm limit}$) during the life time of the magnetar
is:

$N_{\rm B}=\int_{E_{\rm limit}}^{\infty}\int_{0}^{\infty}n(\tau,E)d\tau dE.$
(9)

and the total energy released is:

$E_{\rm B}=\int_{0}^{\infty}\int_{0}^{\infty}n(\tau,E)Ed\tau dE.$ (10)

which should be less than the total energy stored in the magnetosphere.

Now the rate of bursts from all merged-magnetars in the local Universe within
a sphere shell of radius from $D$ to $D+dD$, in the energy range from $E$ to
$E+dE$ is:

$d^{2}R_{B}=4\pi D^{2}dDR_{\rm
m}\big{(}\int^{\infty}_{0}n(\tau,E)d\tau\big{)}dE.$ (11)

where $R_{\rm m}$ is the merger rate density of double neutron stars whose
remnants are NSs instead of prompt collapsed BHs. The above equation can be
further formulated to:

$d^{2}R_{B}=4\pi D^{2}dDR_{\rm
m}\big{(}\int^{\infty}_{0}n(\tau,E)d\tau\big{)}\frac{dE}{dF}dF,$ (12)

where $F$ is the fluence. Since $E=4\pi D^{2}F$, we have from the above
equation that:

$\frac{d^{2}R_{\rm B}}{dFdD}=(4\pi D^{2})^{2}dDR_{\rm
m}\big{(}\int^{\infty}_{0}n(\tau,E)d\tau\big{)}.$ (13)

As a result, the rate of such bursts with in a limiting distance $D_{\rm u}$
and above a limiting fluence is:

$R_{\rm B}=(4\pi)^{2}R_{\rm m}\int_{F_{\rm limit}}^{\infty}\int_{0}^{D_{\rm
u}}\big{(}\int^{\infty}_{0}n(\tau,E(F))d\tau\big{)}D^{4}dDdF$ (14)

where $F_{\rm limit}$ is the fluence limit of the gamma-ray detector.

The volumetric integral in equation (11) should be limit in local Universe,
where the merger rate density can be viewed as a constant, and cosmic
expansion has a negligible effect. When considering the joint observation of
such flares with GW detection of the BNS mergers, the integral over the
luminosity distance in equation (12) should be truncated at the BNS horizon of
the GW detector.

The key quantities are $n(\tau,E)$ and $R_{\rm m}$, the latter we formulated
with:

$R_{\rm m}=\eta\mathcal{R}_{\rm m}$

, where $\mathcal{R}_{\rm m}$ is the merger rate density of all BNS
population, and $\eta$ is the fraction of those have long-lived magnetar
remnants. $\mathcal{R}_{\rm m}$ can be constrained by previous GW observation
at $39-1900$ Gpc-3s-1 (The LIGO Scientific Collaboration et al., 2021).

Here we suppose that the age and energy dependence of $n(\tau,E)$ is
separable:

$n(\tau,E)=n(\tau)f(E),$

where $f(E)$ is the normalized probability density distribution of the energy
of flares, for which we assume a power-law form of:

$f(E)=f_{0}E^{-\beta},E_{l}<E<E_{u}$ (15)

Studies (Cheng et al., 2020) found the index in broad consistent with that
expected from a SOC process ($\beta=5/3$), and the normalization factor

$f_{0}=(\beta-1)E_{\rm l}^{\beta-1}$

Now the equation (14) can be simplified to:

$R_{\rm B}=R_{\rm m}(4\pi)^{2-\beta}N_{\rm B}f_{0}\frac{F^{1-\beta}_{\rm
limit}}{(\beta-1)}\frac{D^{5-2\beta}_{\rm u}}{(5-2\beta)}$ (16)

The total energy to be released should be limited by the magnetic energy
stored in the magnetosphere:

$E_{\rm
mag}\geq\int_{\tau_{l}}^{\tau_{u}}n(\tau)d\tau\int_{E_{l}}^{E_{u}}Ef(E)dE\sim
N_{\rm B}\overline{E},$ (17)
$\overline{E}=\int_{E_{l}}^{E_{u}}Ef(E)dE\sim\frac{\beta-1}{2-\beta}\big{(}\frac{E_{u}}{E_{l}}\big{)}^{1-\beta}E_{u}.$
(18)

the approximant in the above equation is valid only when $1<\beta<2$. Cheng et
al. (2020) found $\beta\sim 1.66$, which meets the above mentioned conditions.
If we assume that the flares in ordinary SGR and GFs follows the same energy
distribution law $f(E)$, the energy of those flares can range more than five
orders of magnitudes, with $E_{\rm u}$ corresponds to the most energetic giant
flare is $E\sim 10^{46}$ ergs. Therefore, from equation (18) we find that:
$\overline{E}\sim 4\times 10^{43}E^{\beta-1}_{l,41}\,\rm ergs$, where
$E_{l,42}$ is the lower energy end of the $f(E)$ in unit of $10^{41}$ ergs.

The magnetic energy stored in the magnetosphere is (Zhang et al., 2022):

$E_{\rm mag}\sim 8\times 10^{46}B^{2}_{15}\,\rm ergs,$ (19)

where $B$ is the surface magnetic field strength of the magnetar scaled with
$10^{15}$ G. If we equalize the both sides of inequality (17), we can have a
rough estimation of $N_{\rm B}$ as:

$N_{\rm B}\sim 2\times
10^{3}\frac{B^{2}_{15}\,\rm{ergs}}{E^{\beta-1}_{l,42}},$ (20)

Taking the $N_{\rm B}$ from above, and taking $E_{l}=10^{42}$ergs, the
expression of $f_{0}$ into equation (16), we have:

$R_{\rm B}=2\times 10^{3}\frac{(4\pi)^{2-\beta}}{5-2\beta}R_{\rm m}D^{3}_{\rm
u}\big{(}\frac{10^{41}\,\text{ergs}}{F_{\rm limit}D^{2}_{\rm
u}}\big{)}^{\beta-1}.$ (21)

If we insert the numbers into above equation with $\beta=5/3$, we obtain that:

$R_{\rm B}=5\times 10^{-3}\eta\mathcal{R}_{\rm
m}B^{2}_{15}D_{u,100}^{5/3}F^{-2/3}_{\rm{limit},-8}\,\text{yr}^{-1},$ (22)

where $\mathcal{R}_{\rm m}$ is in unit of yr-1/Gpc3, $D_{u,100}$ is the
distance limit in unit of 100 Mpc, $F_{\rm{limit},-8}$ is the fluence limit in
unit of $10^{-8}$ ergs/cm2. The detection horizon $D_{u}$ is limited by the
fluence cut of a gamma-ray detector as:

$D_{\rm u,\gamma}\sim 300\,F_{\rm limit,-8}^{-1/2}\,\text{Mpc}$ (23)

when taking the $E_{u}=10^{46}$ ergs, which corresponds to a conservative
estimation of the total magnetic energy stored in the magnetosphere (Zhang et
al., 2022).

##  4 Merged-magnetar flares as EM counterpart of GW events and its spin
period modulation

A magnetar which was born with a millisecond spin period will experience two
evolutionary phases. In its first phase of millisecond period of spinning, the
magnetar’s spin period rapidly slowed down to seconds by the strong magnetic
braking torque. In the later phase, when the spin period is settled to second
scale and evolve less rapid. We can define a transition time between the first
and the second phases:

$\tau_{\rm trans}\sim\,\frac{P_{10\,\rm{ms}}^{2}}{B^{2}_{0,15}}\,{\rm day}.$
(24)

As argued in previous sections, the bursts rate before $\tau_{\rm trans}$
overwhelms that after it. Therefore, the rate in equation (22) is mostly
describe those bursts happens before $\tau_{\rm trans}$. Equivalently, those
bursts to be detected is likely to following a GW chirp from BNS merger with a
time lag $\tau_{\rm lag}<\tau_{\rm trans}$. On the other hand, $\tau_{\rm
lag}$ should be larger than $\tau_{\rm limit}$, which is the time limit less
than which, the ejecta from the BNS merger is still optically thick, thus the
flares from the magnetars will be largely absorbed and the temporal structure
within the flares is smeared. $\tau_{\rm limit}$ is also in time scale of days
(Li & Paczyński 1998, and see discussion in following section).

Since the duration of a typical magnetar GF is $\sim 0.1$-1 s, the flares
detected in this phase can exhibit significant spin modulation, which can
serve as an unambiguous evidence of the existence of a merged magnetar. In
this case, the $D_{\rm u}$ in equation (22) is the minimum between the gamma-
ray detection horizon and the GW horizon:

$D_{u}=\min\big{(}D_{\rm u,\gamma},D_{\rm GW}\big{)}$ (25)

The flare rate as function of fluence limit is plotted in figure 2, see figure
caption for the detailed description of the plot. When plotting figure 2, we
calculate the rate using equation (22) with Monte Carlo samplings of $\eta$,
$R_{\rm m}$ and $D_{\rm GW}$: $\eta$ is uniformly random sampled in log-space
from 0.01 to 0.1; $R$ is sampled from a log-Gaussian distribution with
1-$\sigma$ upper and lower limits correspond to 39 and 1900 yr-1/Gpc3; $D_{\rm
GW}$ is sampled from a Gaussian random with mean 300 Mpc and a standard
deviation of 40 Mpc, which corresponds to the BNS detection horizon of a GW
detector network with LIGO-Virgo-KAGRA (LVK) in O4 period333as simulated here:
https://emfollow.docs.ligo.org/userguide/capabilities.html.

Figure 2: The flare rate as function of fluence limit is plotted in figure 2.
The blue band are the possible range of bursts rate which are associated with
GW observation in LVK O4, and the dashed dark lines indicate those of bursts
regardless of GW counterparts. The upper and lower limits of the range
correspond to the 86% quantiles (1-sigma) in a Morte Carlo simulation. The
rate is calculating with the following choice of parameters: $\eta$ is
uniformly random sampled in log-space from 0.01 to 0.1; $R$ is sampled from a
log-Gaussian distribution with 1-$\sigma$ upper and lower limits correspond to
39 and 1900 yr-1/Gpc3; $D_{\rm GW}$ is sampled from a Gaussian random with
mean 300 Mpc and a standard deviation of 40 Mpc.

##  5 Absorption by the BNS ejecta

During the collision of BNS, abundant material will be ejected from both the
tidal tail and the disk (Bovard et al., 2017; Just et al., 2015). Actually BNS
is the confirmed site for rapid neutron capture nucleosynthesis ($r$-process)
(Abbott et al., 2017a, b; Cowperthwaite et al., 2017; Kasen et al., 2017),
which is responsible for about half of the elements heavier than iron measured
in our solar system (Burbidge et al., 1957; Sneden et al., 2008). Thus, it is
expected that the BNS will be surrounded by dense $r$-process material at
early time, with a total ejected mass ranging from $\sim 0.005-0.1M_{\odot}$
(Bovard et al., 2017; Radice et al., 2018; Côté et al., 2018; Just et al.,
2015; Fernández et al., 2015), and is optically thick to the flare gamma-ray
radiation from the center remnant largely due to the Compton scattering. In
the mean time, as the $r$-process material is ejected from BNS merger with
high speed, the ejecta will become optically thin at $\sim$ days after the
merging event (Li & Paczyński, 1998; Korobkin et al., 2020; Wang et al.,
2020b). The kilonova models of GW170817 observation suggested that such ejecta
has a speed ranges from $0.1c$ to $0.3c$ on average (e.g., Kasen et al., 2017;
Rosswog et al., 2018; Wollaeger et al., 2018; Watson et al., 2019), as
expected from previous theoretical work (e.g., Li & Paczyński, 1998; Tanaka &
Hotokezaka, 2013).

Our calculation in previous section did not include the absorption of the
surrounding BNS ejecta to the flare radiation. When such effect is included,
and together with a finite work energy range of the detector, it is equivalent
that the limit fluence in equations (22) and (23) is replaced by a
$\tilde{F}_{\rm limit}$, which related with the original $F_{\rm limit}$ as:

$\tilde{F}_{\rm limit}=\frac{F_{\rm limit}}{1-\xi}$ (26)

Here we define an absorption factor $\xi$ to describe the effect of the
surrounding ejecta in absorbing the high-energy photons from the BNS flare,
which is a function of time after BNS collision ($\tau$) and is defined as:

$1-\xi(\tau)=\frac{F_{\rm observed}(\tau)}{F_{\rm emitted}}=\frac{\int^{E_{\rm
high}}_{E_{\rm low}}f_{\rm observed,E}(\tau)dE}{\int^{E_{\rm high}}_{E_{\rm
low}}f_{E}dE},$ (27)

which is the ratio between the flux after absorption (observed) at time $\tau$
and the total emitted flux from the flare, $E_{\rm low}$ and $E_{\rm high}$
denote the energy range where a specific detector works.
$f_{E}=dE_{\gamma}/dEdAdt$ is the differential energy flux emitted from the
flare. We assume a spectrum shape of $f_{E}$ as a power law of index -0.2 with
an exponentially-cutoff at $0.48$ MeV, i.e.,
$f_{E}=f_{0}E^{-0.2}exp(-E/0.48\rm{MeV})$, and $f_{0}$ is the normalization
factor with $\int^{\infty}_{0}f_{E}dE=F_{\rm total}=L_{\rm total}/4\pi D^{2}$,
where $D$ is the distance of the BNS, and $L_{\rm total}$ is the luminosity of
the magnetar flare. This spectrum shape is taken from that of the GF from
magnetar SGR 1806-20 (Palmer et al., 2005).

For approximation, we assume a uniform spherical $r$-process ejecta
distribution as in Wang et al. (2020a, b) to calculate the observed flare
emission (after the ejecta absorption). Then, the emitted gamma rays after
propagation through the ejecta (absorption due to Compton scattering with the
ejecta material) is

$f_{\rm observed,E}(\tau)=f_{E}e^{-\rho_{\rm ej}\kappa(E)l}$ (28)

where $\rho_{\rm ej}$ is the ejecta density, $\kappa(E)$ is the opacity of the
ejecta to a photon with energy $E$, path-length $l$ is the distance of the
photons travelling rhough the ejecta, for this case, $l\sim v\tau$, with $v$
to be the expanding/ejected velocity of the ejecta. Only non-scattered photons
are included in the observed gamma-ray signal here; scattered photons are
ignored as their effects are minimal at late times when the ejecta is nearly
optically thin.

Figure 3: Plot of the flare spectra flux $f_{E}/f_{0}$ vs energy E before
(emitted, black) and after absorption by a 0.01 $M_{\odot}$ BNS merger ejecta
with 0.3c expansion velocity and a robust main $r$-process components, at 5
different times after BNS collision: 0.2 day (red), 0.3 day (orange), 0.6 day
(green), 1 day (cyan), 2 day (blue)

.

We obtain the $r$-process nuclei abundance distribution in the BNS merger
ejecta using the nuclear reaction network code Portable Routines for
Integrated nucleoSynthesis Modeling, or PRISM (Mumpower et al., 2018) as in
Wang et al. (2020b). We adopt a BNS merger dynamical ejecta with robust
$r$-process productions (Rosswog et al., 2013) for the baseline calculation.
The opacity values for the total BNS collision ejecta are calculated following
the method in Wang et al. (2020b), and the opacity values of individual
$r$-process nuclei are adopted from the XCOM
website444https://www.nist.gov/pml/xcom-photon-cross-sections-database. The
resulting spectra after absorption by the ejecta at 5 different times (0.2
day, 0.3 day, 0.6 day, 1day and 2 day) after BNS collision are shown in figure
3. Compared to the emitted spectrum shown as black line, we conclude that the
detection window of such bursts should be in energy range from $\sim$1 MeV to
10 MeV, and in the time window between 0.5 and 2 days after BNS collision.

We note that in addition to the burst flare radiation, the BNS collision
ejecta itself also emit gamma-ray photons through the decays of the
radioactive $r$-process nuclei. The total gamma radiation rate from the
$r$-process ejecta is estimated to be $\epsilon_{0}(\tau)\sim 2\times
10^{10}{\rm erg\ g}^{-1}s^{-1}(\tau/{\rm day})^{-1/3}$ (Metzger & Berger,
2012; Korobkin et al., 2020), and the $r$-process gamma-ray energy at $\sim
1$day is then $\sim 10^{41}$ erg/s for a 0.01 $M_{\odot}$ BNS merger ejecta.
Thus, such signal would be small compared to the flare emissions, and the BNS
ejecta spectrum shapes with nuclear decay lines (Korobkin et al., 2020; Wang
et al., 2020b) are also different from the flare signal discussed here.

Figure 4: Plot of the absorption factor $\xi$ versus time ($\tau$) for 4
different energy bands: 45keV-10MeV (black); 10-100keV(blue);
100keV-1MeV(green); 1MeV-10MeV(red). The solid lines are the absorption
results for a 0.01 $M_{\odot}$ BNS ejecta with 0.3c expansion velocity and a
robust main $r$-process components, the color shades indicate uncertainties
due to the variations in the BNS ejecta properties including mass, velocity
and composition.

Then we conduct the integral in equation (27) to obtain $\xi$ as function of
$\tau$ in Figure 4. The uncertainty bands are due to variations in BNS ejecta
properties, including velocity, ejecta mass and the components. Here we varied
the ejecta mass between 0.005 to 0.03 $M_{\odot}$, the velocity between 0.1c
to 0.3c. To test the sensitivity of the signal to the ejecta component, we
adopted the parameterized BNS outflow conditions (Just et al., 2015; Radice et
al., 2018) with a range of initial electron fractions as in Wang et al.
(2020b), so that the ejecta component varies from the weak $r$-process (no
third peak and heavier actinides elements) to robust $r$-process (with
actinides). From Figure 4, we can see that at the detection window discussed
above, the corresponding absorption factor is $\xi\sim 0.5-1$ with an order of
magnitude uncertainty. The burst detection rate after absorption of BNS ejecta
considered is re-plotted in Figure 5. When plotting the figure, we calculate
the rate with a $\xi$ randomly drawn from its corresponding range evaluated
above with uniform distribution.

During the variation test, we find that the flare signal is more sensitive to
the ejecta velocity and mass. Therefore, on the other hand, the detection rate
obtained in the real observation could enable us to put a constraint on the
BNS ejecta property.

Figure 5: Same plot as the band for GW counterpart in figure 2, but with
absorption considered for energy ranges from 1 MeV to 10 MeV and from 100 keV
to 1 MeV.

##  6 Discussion

###  6.1 Potential cases in archival data from Fermi/GBM sGRB catalogue

The Fermi/GBM detector has an energy range of $\sim 0.01-1$ MeV, and a fluence
limit (for a 1 s bursts) of $\sim 2\times 10^{-8}$ erg/cm2 555Following the
practice of Hendriks et al. (2022), we use the lowest observed fluence of the
sGRB catalogue (Narayana Bhat et al., 2016) as the fluence limit of second
duration bursts.. It has been monitoring GRBs for $\sim 10$ years. From our
estimation, there should have been $\sim 0.1-10$ such bursts detected in its
bursts catalogue. As mentioned above, such flares may exhibit spin modulation.
Although there has been search for QPOs in song of Fermi/GBM’s bright GRBs
(Dichiara et al., 2013) and found none positive results, a more careful survey
focusing on those weak short bursts with a fast increasing period $\sim 0.1$ s
might identify such bursts in the archival data, although with foreseeable
difficulties due to their fewer photon counts. Recently, Chirenti et al.
(2023) reported the detection of kHz QPOs in two archival sGRBs of the Burst
and Transient Source Experiment (BATSE). BATSE works in lower energy range
from 50 KeV to 300 keV, in which we expect significant absorption if they were
merged-magnetar-flares. Besides, the found QPO are above 1 kHz, which is much
higher than we expect for the stable merged-magnetar-flares. Therefore, these
two sGRBs with QPOs should not be considered as cases of the proposed merger-
magnetar-flares.

###  6.2 Prospects for next generation MeV detectors

For the next generation MeV telescope, such as
COSI666https://cosi.ssl.berkeley.edu,
AMEGO777https://asd.gsfc.nasa.gov/amego/, or
MeVGRO888https://indico.icranet.org/event/1/contributions/777/, the energy
range between $\sim 0.1-10$ MeV will be well covered, and the detectors’
sensitivities at this energy range are expected to be at the level between
$\sim 3\times 10^{-12}$ erg cm-2 s-1 and $10^{-10}$ erg cm-2 s-1, which are at
least 1-2 orders of magnitude better than current and previous MeV detectors
like INTEGRAL999https://www.cosmos.esa.int/web/integral and
COMPTEL101010https://heasarc.gsfc.nasa.gov/docs/cgro/comptel/. Thus, we expect
a much larger detection rate up to $\sim 100$ per year as a new population of
GW gamma-ray counterparts.

The main sources of the uncertainties in the burst rate estimation is from: 1)
the local rate density of the BNS collision; 2) the fraction of BNS merger
which leaves long-lived magnetar, or $\xi$ as we denote in this paper; A
future detection of a population of such bursts, together with multi-messenger
observation from GW will in turn give inference on these aspects. The current
BNS $\mathcal{R}_{\rm{m}}$ estimation is based on two BNS events (GW170817 and
GW200311_115853) in LIGO-Virgo-Collaboration (LVC) O2-O3 period (The LIGO
Scientific Collaboration et al., 2021). In the LVK-O4 period, it is estimated
that $36^{+49}_{-22}$ BNS mergers shall be detected111111as reported in
https://emfollow.docs.ligo.org/userguide/capabilities.html#datadrivenexpectations,
using the same methodology in Abbott et al. (2020) but with updated input
models for detector network and sources., which will return in a much tighter
constraints on $\mathcal{R}_{\rm{m}}$. The value of $\xi$ depends on the
equation of state of NS matter, and also the mass function of NS. The latter
can be better constraints by a larger sample of BNS observed with GWs. If
multiple bursts could be observed from a single merger magnetar, the
dependence of the bursts properties on its period spin could be study, which
will provide valuable insights into the emission mechanism of magnetar
activities.

###  6.3 Solidification of the crust of the newly born magnetar

In order to justify the domination of the magnetar activity in its rapid
braking stage in age of $\sim$days under the star quake paradigm, we made an
order-of-magnitude estimation of the burst rate in the introduction section. A
crucial presumption was that the elastic property of the neutron star crust
unchanged. First we need to check whether the surface of the NS has already
cooled enough to have a solidified crust. According to Negele & Vautherin
(1973); Haensel & Pichon (1994); Douchin & Haensel (2000), the melting
temperature of the NS crust lies well above 108 K, and Lattimer et al. (1994)
showed that the core of a newly born NS can fast cool down to temperature $T$
on the timescale:

$t=20\,(T/10^{9}\,\text{K})^{-4}\,\text{s}.$ (29)

Therefore, in the age of $\sim$ day, the new born magnetar already has a solid
crust. As for its elastic properties, their are in general not constants and
temperature dependent (e.g. Strohmayer et al. 1991). Therefore, a more careful
quantitative calculation of the burst rate as function of the magnetar’s age
should consider a realistic cooling curve of the NS’s crust and its
temperature-dependent elastic properties.

##  7 Summary

From our above argument and calculation, we conclude that flares from merged
massive magnetars can be expected as a population of gamma-ray transients,
which associate GW chirp events of BNS mergers. Such a gamma-ray counterpart
of GW may look like a short gamma-ray bursts (sGRB) according to its duration,
but it can be found with several distinct features:

  * •

it tends to be weak in flux, and the time-lag between the burst and GW chirp
is $\sim$1-2 days, rather than $\sim$ s as in sGRB;

  * •

its spectrum has a lower energy cut at $\sim$100 keV, due to absorption of the
ejecta.

Beside, it may show spin modulation with a significant spin down, although the
potential of observing a such short scale temporal structure significantly is
very challenging in reality.

Due to the absorption by the ejecta from the BNS collision, such flares are
optimally to be observed in the energy range from 0.1 to 10 MeV. The estimated
detection rate is increasing towards a fluence flux limit, with a lower law
with a index -1.5; While the rate of such bursts as GW association will also
be limited by the detecting reach of GW detector networks, when the fluence
limit of the HE detector is below some turn-over sensitivity. Below this turn-
over fluence limit, the rate follows another power law with index -2/3. When
observing with a detector of energy range 0.1-1 MeV, the turn-over flux limit
is at $\sim 2\times 10^{-9}$ erg/cm2, while than for a detector of 1-10 MeV is
at $\sim 10^{-8}$ erg/cm2. Based on our evaluated from a population of BNS, a
GRB monitor with energy range of $\sim 0.1-1$ MeV and a fluence limit of $\sim
2\times 10^{-8}$ erg/cm2 could detect such flares as gamma-ray counterparts of
GW events at a rate from 0.01 to 1 per year; while a MeV detector working in a
range from $\sim 1-10$ MeV with a fluence limit $\sim 10^{-9}$ erg/cm2 could
detect such signals a few to a few tens per year.

We would like to acknowledge the insightful discussions we had with Profs.
Shuang-Nan Zhang and Ming-Yu Ge. This work is supported by the National Key
R&D Program of China (2021YFA0718500). SXY acknowledge the support from the
Chinese Academy of Sciences (Grant No. E329A3M1). The work of X.W. is
supported in part by the Chinese Academy of Sciences (Grant No. E329A6M1).

## References

  * Abbott et al. (2017a) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2017, Physical Review Letters, 119, 161101
  * Abbott et al. (2017b) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2017, ApJ, 848, L12
  * Abbott et al. (2020) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2020, Living Reviews in Relativity, 23, 3. doi:10.1007/s41114-020-00026-9
  * Baym & Pines (1971) Baym, G. & Pines, D. 1971, Annals of Physics, 66, 816. doi:10.1016/0003-4916(71)90084-4
  * Beloborodov (2020) Beloborodov, A. M. 2020, ApJ, 896, 142. doi:10.3847/1538-4357/ab83eb
  * Blaes et al. (1989) Blaes, O., Blandford, R., Goldreich, P., et al. 1989, ApJ, 343, 839. doi:10.1086/167754
  * Blackman & Yi (1998) Blackman, E. G. & Yi, I. 1998, ApJ, 498, L31. doi:10.1086/311311
  * Bovard et al. (2017) Bovard, L., Martin, D., Guercilena, F., et al. 2017, Phys. Rev. D, 96, 124005
  * Bransgrove et al. (2020) Bransgrove, A., Beloborodov, A. M., & Levin, Y. 2020, ApJ, 897, 173. doi:10.3847/1538-4357/ab93b7
  * Burbidge et al. (1957) Burbidge, E. M., Burbidge, G. R., Fowler, W. A., & Hoyle, F. 1957, Reviews of Modern Physics, 29, 547
  * Cheng et al. (2020) Cheng, Y., Zhang, G. Q., & Wang, F. Y. 2020, MNRAS, 491, 1498. doi:10.1093/mnras/stz3085
  * Chirenti et al. (2023) Chirenti, C., Dichiara, S., Lien, A., et al. 2023, Nature, 613, 253. doi:10.1038/s41586-022-05497-0
  * Côté et al. (2018) Côté, B., Fryer, C. L., Belczynski, K., et al. 2018, ApJ, 855, 99. doi:10.3847/1538-4357/aaad67
  * Cowperthwaite et al. (2017) Cowperthwaite, P. S., Berger, E., Villar, V. A., et al. 2017, ApJ, 848, L17
  * Dichiara et al. (2013) Dichiara, S., Guidorzi, C., Frontera, F., et al. 2013, ApJ, 777, 132. doi:10.1088/0004-637X/777/2/132
  * Douchin & Haensel (2000) Douchin, F. & Haensel, P. 2000, Physics Letters B, 485, 107. doi:10.1016/S0370-2693(00)00672-9
  * Duncan & Thompson (1992) Duncan, R. C. & Thompson, C. 1992, ApJ, 392, L9. doi:10.1086/186413
  * Fahlman & Gregory (1981) Fahlman, G. G. & Gregory, P. C. 1981, Nature, 293, 202. doi:10.1038/293202a0
  * Fernández et al. (2015) Fernández, R., Kasen, D., Metzger, B. D., et al. 2015, MNRAS, 446, 750. doi:10.1093/mnras/stu2112
  * Ferrario & Wickramasinghe (2008) Ferrario, L. & Wickramasinghe, D. 2008, MNRAS, 389, L66. doi:10.1111/j.1745-3933.2008.00527.x
  * Gavriil et al. (2002) Gavriil, F. P., Kaspi, V. M., & Woods, P. M. 2002, Nature, 419, 142. doi:10.1038/nature01011
  * Golenetskii et al. (1984) Golenetskii, S. V., Ilinskii, V. N., & Mazets, E. P. 1984, Nature, 307, 41. doi:10.1038/307041a0
  * Haensel & Pichon (1994) Haensel, P. & Pichon, B. 1994, A&A, 283, 313. doi:10.48550/arXiv.nucl-th/9310003
  * Hartmann (1995) Hartmann, D. H. 1995, A&A Rev., 6, 225. doi:10.1007/BF01837116
  * Hendriks et al. (2022) Hendriks, K., Yi, S.-X., & Nelemans, G. 2022, arXiv:2208.14156. doi:10.48550/arXiv.2208.14156
  * Hurley et al. (2005) Hurley, K., Boggs, S. E., Smith, D. M., et al. 2005, Nature, 434, 1098. doi:10.1038/nature03519
  * Just et al. (2015) Just, O., Bauswein, A., Ardevol Pulpillo, R., et al. 2015, MNRAS, 448, 541. doi:10.1093/mnras/stv009
  * Kasen et al. (2017) Kasen, D., Metzger, B., Barnes, J., et al. 2017, Nature, 551, 80.
  * Kaspi & Gavriil (2004) Kaspi, V. M. & Gavriil, F. P. 2004, Nuclear Physics B Proceedings Supplements, 132, 456. doi:10.1016/j.nuclphysbps.2004.04.080
  * Kaspi & Beloborodov (2017) Kaspi, V. M. & Beloborodov, A. M. 2017, ARA&A, 55, 261. doi:10.1146/annurev-astro-081915-023329
  * Kluźniak & Ruderman (1998) Kluźniak, W. & Ruderman, M. 1998, ApJ, 505, L113. doi:10.1086/311622
  * Komissarov et al. (2007) Komissarov, S. S., Barkov, M., & Lyutikov, M. 2007, MNRAS, 374, 415. doi:10.1111/j.1365-2966.2006.11152.x
  * Korobkin et al. (2020) Korobkin, O., Hungerford, A. M., Fryer, C. L., et al. 2020, ApJ, 889, 168. doi:10.3847/1538-4357/ab64d8
  * Lattimer et al. (1994) Lattimer, J. M., van Riper, K. A., Prakash, M., et al. 1994, ApJ, 425, 802. doi:10.1086/174025
  * Li & Paczyński (1998) Li, L.-X. & Paczyński, B. 1998, ApJ, 507, L59. doi:10.1086/311680
  * Levin (2006) Levin, Y. 2006, MNRAS, 368, L35. doi:10.1111/j.1745-3933.2006.00155.x
  * Lyutikov (2003) Lyutikov, M. 2003, MNRAS, 346, 540. doi:10.1046/j.1365-2966.2003.07110.x
  * Mahlmann et al. (2019) Mahlmann, J. F., Akgün, T., Pons, J. A., et al. 2019, MNRAS, 490, 4858. doi:10.1093/mnras/stz2729
  * Mereghetti et al. (2015) Mereghetti, S., Pons, J. A., & Melatos, A. 2015, Space Sci. Rev., 191, 315. doi:10.1007/s11214-015-0146-y
  * Mereghetti (2008) Mereghetti, S. 2008, A&A Rev., 15, 225. doi:10.1007/s00159-008-0011-z
  * Metzger & Berger (2012) Metzger, B. D. & Berger, E. 2012, ApJ, 746, 48. doi:10.1088/0004-637X/746/1/48
  * Minaev & Pozanenko (2020) Minaev, P. Y. & Pozanenko, A. S. 2020, Astronomy Letters, 46, 573. doi:10.1134/S1063773720090042
  * Mumpower et al. (2018) Mumpower, M. R., Kawano, T., Sprouse, T. M., et al. 2018, ApJ, 869, 14
  * Nakamura (1998) Nakamura, T. 1998, Progress of Theoretical Physics, 100, 921. doi:10.1143/PTP.100.921
  * Narayana Bhat et al. (2016) Narayana Bhat, P., Meegan, C. A., von Kienlin, A., et al. 2016, ApJS, 223, 28. doi:10.3847/0067-0049/223/2/28
  * Negele & Vautherin (1973) Negele, J. W. & Vautherin, D. 1973, Nucl. Phys. A, 207, 298. doi:10.1016/0375-9474(73)90349-7
  * Norris et al. (1991) Norris, J. P., Hertz, P., Wood, K. S., et al. 1991, ApJ, 366, 240. doi:10.1086/169556
  * Palmer et al. (2005) Palmer, D. M., Barthelmy, S., Gehrels, N., et al. 2005, Nature, 434, 1107. doi:10.1038/nature03525
  * Radice et al. (2018) Radice, D., Perego, A., Hotokezaka, K., et al., 2018, ApJ, 869, 130
  * Rea et al. (2010) Rea, N., Esposito, P., Turolla, R., et al. 2010, Science, 330, 944. doi:10.1126/science.1196088
  * Rea & Esposito (2011) Rea, N. & Esposito, P. 2011, High-Energy Emission from Pulsars and their Systems, 21, 247. doi:10.1007/978-3-642-17251-9_21
  * Ripperda et al. (2019) Ripperda, B., Porth, O., Sironi, L., et al. 2019, MNRAS, 485, 299. doi:10.1093/mnras/stz387
  * Roberts et al. (2021) Roberts, O. J., Veres, P., Baring, M., et al. 2021,
  * Rosswog et al. (2013) Rosswog, S., Piran, T., & Nakar, E. 2013, MNRAS, 430, 2585. doi:10.1093/mnras/sts708
  * Rosswog et al. (2018) Rosswog, S., Sollerman, J., Feindt, U., et al. 2018, A&A, 615, A132
  * Ruderman et al. (2000) Ruderman, M. A., Tao, L., & Kluźniak, W. 2000, ApJ, 542, 243. doi:10.1086/309537
  * Thompson (1994) Thompson, C. 1994, MNRAS, 270, 480. doi:10.1093/mnras/270.3.480
  * Thompson & Duncan (1995) Thompson, C. & Duncan, R. C. 1995, MNRAS, 275, 255. doi:10.1093/mnras/275.2.255
  * Thompson & Duncan (1996) Thompson, C. & Duncan, R. C. 1996, ApJ, 473, 322. doi:10.1086/178147
  * Sharma et al. (2023) Sharma, P., Barkov, M., Lyutikov, M. 2023,arXiv: 2302.08848
  * Sneden et al. (2008) Sneden, C., Cowan, J. J., & Gallino, R. 2008, ARA&A, 46, 241
  * Spruit (1999) Spruit, H. C. 1999, A&A, 341, L1. doi:10.48550/arXiv.astro-ph/9811007
  * Strohmayer et al. (1991) Strohmayer, T., Ogata, S., Iyetomi, H., et al. 1991, ApJ, 375, 679. doi:10.1086/170231
  * Svinkin et al. (2021) Svinkin, D., Frederiks, D., Hurley, K., et al. 2021, Nature, 589, 211. doi:10.1038/s41586-020-03076-9
  * Tanaka & Hotokezaka (2013) Tanaka, M., & Hotokezaka, K. 2013, ApJ, 775, 113
  * The LIGO Scientific Collaboration et al. (2021) The LIGO Scientific Collaboration, the Virgo Collaboration, the KAGRA Collaboration, et al. 2021, arXiv:2111.03634. doi:10.48550/arXiv.2111.03634
  * Turolla et al. (2015) Turolla, R., Zane, S., & Watts, A. L. 2015, Reports on Progress in Physics, 78, 116901. doi:10.1088/0034-4885/78/11/116901
  * Turolla & Esposito (2013) Turolla, R. & Esposito, P. 2013, International Journal of Modern Physics D, 22, 1330024-163. doi:10.1142/S0218271813300243
  * Usov (1992) Usov, V. V. 1992, Nature, 357, 472. doi:10.1038/357472a0
  * Wang et al. (2020a) Wang, X., N3AS Collaboration, Fields, B. D., et al. 2020, ApJ, 893, 92. doi:10.3847/1538-4357/ab7ffd
  * Wang et al. (2020b) Wang, X., N3AS Collaboration, Vassh, N., et al. 2020, ApJ, 903, L3. doi:10.3847/2041-8213/abbe18
  * Watson et al. (2019) Watson, D., Hansen, C. J., Selsing, J., et al. 2019, Nature, 574, 497
  * Wheeler et al. (2000) Wheeler, J. C., Yi, I., Höflich, P., et al. 2000, ApJ, 537, 810. doi:10.1086/309055
  * Wollaeger et al. (2018) Wollaeger, R. T., Korobkin, O., Fontes, C. J., et al. 2018, MNRAS, 478, 3298
  * Yi & Blackman (1998) Yi, I. & Blackman, E. G. 1998, ApJ, 494, L163. doi:10.1086/311192
  * Yuan et al. (2020) Yuan, Y., Beloborodov, A. M., Chen, A. Y., et al. 2020, ApJ, 900, L21. doi:10.3847/2041-8213/abafa8
  * Zhang & Mészáros (2001) Zhang, B. & Mészáros, P. 2001, ApJ, 552, L35. doi:10.1086/320255
  * Zhang et al. (2020) Zhang, H.-M., Liu, R.-Y., Zhong, S.-Q., et al. 2020, ApJ, 903, L32. doi:10.3847/2041-8213/abc2c9
  * Zhang et al. (2022) Zhang, Z., Yi, S.-X., Zhang, S.-N., Xiong, S.-L., and Xiao, S., 2022, ApJ, 939, L25. doi:10.3847/2041-8213/ac9b55