# Sequence-to-Sequence Spanish Pre-trained Language Models

Vladimir Araujo1, Maria Mihaela Trusca1, Rodrigo Tufiño2, Marie-Francine
Moens1
1KU Leuven, 2Universidad Politécnica Salesiana
<EMAIL_ADDRESS>
###### Abstract

In recent years, substantial advancements in pre-trained language models have
paved the way for the development of numerous non-English language versions,
with a particular focus on encoder-only and decoder-only architectures. While
Spanish language models encompassing BERT, RoBERTa, and GPT have exhibited
prowess in natural language understanding and generation, there remains a
scarcity of encoder-decoder models designed for sequence-to-sequence tasks
involving input-output pairs. This paper breaks new ground by introducing the
implementation and evaluation of renowned encoder-decoder architectures,
exclusively pre-trained on Spanish corpora. Specifically, we present Spanish
versions of BART, T5, and BERT2BERT-style models and subject them to a
comprehensive assessment across a diverse range of sequence-to-sequence tasks,
spanning summarization, rephrasing, and generative question answering. Our
findings underscore the competitive performance of all models, with BART and
T5 emerging as top performers across all evaluated tasks. As an additional
contribution, we have made all models publicly available to the research
community, fostering future exploration and development in Spanish language
processing.

##  1 Introduction

Spanish ranks among the most extensively used languages globally. This fact
has captured the interest of the NLP community, prompting efforts towards
resource development for this NLP domain. Consequently, a number of pre-
trained language models tailored for Spanish have emerged in recent years,
predominantly employing encoder-only Cañete et al. (2020); De la Rosa et al.
(2022); Araujo et al. (2023) and decoder-only Gutiérrez-Fandiño et al. (2022)
architectures. These models have demonstrated exemplary performance in natural
language understanding across several tasks Cañete et al. (2020) and
benchmarks Araujo et al. (2022). Nevertheless, there has been limited
advancement in addressing tasks revolving around generating new sentences
depending on a given input, such as summarization, generative question
answering, dialogue, or translation.

Encoder-decoder models primarily serve for addressing sequence-to-sequence
tasks, and over recent years, numerous architectures have emerged. The
pretraining of these models is often based on the full transformer
architecture Vaswani et al. (2017) and entails more intricate learning
objectives than those of encoder or decoder-only models individually. For
instance, BART Lewis et al. (2020) is specifically trained to reconstruct text
that has been intentionally corrupted, while T5 Raffel et al. (2020) is
designed to adeptly fill in missing sections of text, simulating a scenario
where text spans have been omitted. These models have predominantly been
developed for the English language, and there have been recent efforts to pre-
train them in languages other than English Kamal Eddine et al. (2021); Sarti
and Nissim (2022). Unfortunately, when it comes to the Spanish language, there
is a notable scarcity of such models that may be valuable to the NLP
community.

In this paper, with the goal of democratizing sequence-to-sequence models for
the NLP community in Spanish, we present BARTO and T5S, which are the Spanish
versions of the BART and T5 models. We pre-train these models exclusively on
Spanish corpora, aligning with their self-supervised methodology. Furthermore,
we introduce models in the style of BERT2BERT Rothe et al. (2020), utilizing
well-established BERT Devlin et al. (2019) and RoBERTa Liu et al. (2019)
models in Spanish as baselines. Subsequently, we curate a diverse set of
sequence-to-sequence tasks to comprehensively assess the capabilities of our
models. Our findings reveal that all models deliver competitive performance
across these tasks. Notably, BARTO and T5S stand out as top performers,
particularly excelling in tasks involving extended sequence summarization,
split-and-rephrase tasks, and generative question answering. We are making
these models available to the public to promote further research and
application of these resources in Spanish
NLP111https://github.com/vgaraujov/Seq2Seq-Spanish-PLMs.

##  2 Related Work

###  2.1 Language-specific Pre-trained Language Models

Pre-trained language models represent a class of advanced language models
trained through self-supervised learning on large text corpora, making them
versatile for various applications. Notably, two prominent models are BERT
Devlin et al. (2019), an encoder-only model, and GPT Radford and Sutskever
(2018); Radford et al. (2019), a decoder-only model. These models have
established robust baselines for a wide range of NLP tasks in the English
language.

Numerous language-specific BERT-based and GPT-based models have emerged in
recent times. Examples include CamemBERT Martin et al. (2020) and GPT-fr
Simoulin and Crabbé (2021) tailored for French, RobBERT Delobelle et al.
(2020) designed for Dutch, FinBERT Virtanen et al. (2019) for Finnish,
GePpeTto Mattei et al. (2020) for Italian, and several others. These models
have consistently outperformed their multilingual counterparts, highlighting
the value of their existence for language-specific tasks.

In the context of the Spanish language, we find BETO Cañete et al. (2020) and
ALBETO Cañete et al. (2022), a BERT and ALBERT model, respectively, pre-
trained on the SUC corpus Cañete et al. (2020). BERTIN De la Rosa et al.
(2022), a RoBERTa base model trained on the Spanish portion sampled from mC4
Xue et al. (2021). Furthermore, MarIA Gutiérrez-Fandiño et al. (2022)
introduces a family of models, including RoBERTa and GPT-2 models trained on
the corpus crawled by the National Library of Spain. A more recent model,
RigoBERTa Serrano et al. (2022), follows the DeBERTa He et al. (2020)
architecture and was trained with several corpora, including OSCAR, SUC, and
mC4-es. Nevertheless, a notable gap exists in the availability of encoder-
decoder models exclusively trained with Spanish data.

###  2.2 Sequence-to-Sequence Pre-trained Language Models

A sequence-to-sequence model aims to map a fixed-length input with a fixed-
length output where the length of the input and output may differ Sutskever et
al. (2014). It comprises an encoder, which concurrently processes the entire
input sequence, and a decoder, which receives the representations computed by
the encoder and generates the output sequence in an autoregressive manner.
These model types have proven to be valuable in addressing tasks including
machine translation, dialogue systems, question answering, and text
summarization.

Following the paradigm of pre-training and self-supervision, several models
have been proposed. One of the first models is MASS Song et al. (2019), which
uses a transformer to reconstruct an input sequence where a contiguous span of
tokens is masked and mapped to a sequence consisting of the missing tokens.
Later, T5 Raffel et al. (2020) proposed a pre-train on a multitask combination
of supervised and self-supervised tasks, the latter being a task to complete
fill-in dropped-out spans of text from documents. BART Lewis et al. (2020) is
slightly similar to T5 but only uses a self-supervised objective in which
spans are masked from the input, but the complete output is predicted to
improve the decoder’s language modeling ability. Moreover, Rothe et al. (2020)
proposed the utilization of encoder or decoder-only pre-trained checkpoints
for initializing new encoder-decoder models, showcasing competitive
performance compared to purely encoder-decoder pre-trained models.

More recently, there has been a notable surge in endeavors to deploy sequence-
to-sequence models for languages beyond English. BART, for instance, has been
released for many other languages, including French Kamal Eddine et al.
(2021), Greek Evdaimon et al. (2023), Indic Dabre et al. (2022), Arabic Kamal
Eddine et al. (2022) and various other languages Shao et al. (2021); Tran et
al. (2022); La Quatra and Cagliero (2023). Furthermore, T5 has been pre-
trained in Italian Sarti and Nissim (2022), Arabic Nagoudi et al. (2022),
Indic Aralikatte et al. (2023), among others. While the aforementioned recent
models encompass a broad range of languages, the availability of Spanish
models remains limited.

##  3 Sequence-to-Sequence Spanish Pre-trained Language Models

In this section, we begin by presenting our data collection and preparation
procedures for pre-training our models. Subsequently, we provide detailed
descriptions of each model and outline the corresponding pre-training
processes.

###  3.1 Pre-training Data

We employ the OSCAR 21.09 corpus, which includes a deduplicated Spanish
dataset of approximately 160GB of text. Furthermore, we utilize the mC4-es
corpus Xue et al. (2021), specifically adopting the Gaussian perplexity
sampling subset proposed by De la Rosa et al. (2022), which boasts an
extensive 500GB text dataset and has demonstrated superior model consistency.
Additionally, we incorporate SUC, the corpus utilized for pre-training BETO,
comprising around 14GB of raw text from diverse sources. Note that we exclude
Wikipedia text from SUC, instead opting for an updated Spanish Wikipedia
dump222https://dumps.wikimedia.org/eswiki/latest/, resulting in approximately
10GB of text.

As established by prior research Liu et al. (2019); Raffel et al. (2020), the
corpus quality significantly impacts the outcomes of pre-training models.
Consequently, we closely follow the preprocessing methodologies previously
established for both English Raffel et al. (2020) and Spanish models
Gutiérrez-Fandiño et al. (2022); Serrano et al. (2022). Below, we describe the
procedure.

  1. 1.

Document-level Formatting: We ensure that all data adheres to a document-level
format, which means that each instance is a document containing several
contiguous coherent sentences. This is crucial for enabling the models to
capture extensive contextual dependencies.

  2. 2.

Data Filtering: To enhance data quality, we employ straightforward and cost-
effective filtering methods. For example, we eliminate very short documents
based on sentence and document length, filter out text containing repeated
characters or special characters not commonly used in Spanish, and exclude
pages containing code and sensitive content.

  3. 3.

Encoding Correction: Some text samples may employ inconsistent encodings or
exhibit encoding issues. To address this, we utilize the
ftfy333https://ftfy.readthedocs.io/ tool to rectify encoding errors.

  4. 4.

Deduplication: As a final step, we employ a deduplication process across all
corpora using the text-dedup444https://github.com/ChenghaoMou/text-dedup
library. Due to its computational intensity, this step is performed at the end
of the data preparation pipeline.

###  3.2 BARTO Model

BARTO follows the BERT base architecture Lewis et al. (2020), which consists
of an encoder and a decoder with 6 layers each. Also, it has 12 attention
heads and 768 hidden dimensions in both the encoder and decoder. BARTO is pre-
trained by denoising the corrupted input documents. As suggested by Lewis et
al. (2020), we use a combination of text infilling and sentence permutation
transformations for robust performance.

We use sentencepiece555https://github.com/google/sentencepiece library to
build a BPE tokenizer of 50,264 tokens. Furthermore, we rely on the
fairseq666https://github.com/facebookresearch/fairseq library to perform the
training. BARTO is pre-trained for 100,000 steps on 8 NVIDIA A100 GPUs with
input texts of 1024 and a batch size of 2048. We use the Adam optimizer Kingma
and Ba (2015), a warm-up of 10,000 steps, a dropout of 0.1, and FP16 to speed
up training.

###  3.3 T5S Model

T5S follows the T5.1.1777https://github.com/google-research/text-to-text-
transfer-transformer/blob/main/released_checkpoints.md base version of the T5
model, which includes some improvements. This model consists of an encoder and
decoder with 12 layers, 12 attention heads, and 768 hidden dimensions each.
Like T5.1.1, we pre-train only using the denoising objective by filling in
dropped-out spans of text from documents.

We use the sentencepiece library to build a unigram tokenizer of 32,000
tokens. We rely on the nanoT5888https://github.com/PiotrNawrot/nanoT5 library,
which allows pre-training T5 models on a limited budget. Our T5S is pre-
trained for 80,000 steps on 4 GPUs NVIDIA A100 with input texts of 512 and a
batch size of 320. Additionally, we use the AdamW optimizer Loshchilov and
Hutter (2019), a warm-up of 10,000 steps, a dropout of 0, and BP16 to speed up
training.

###  3.4 BERT2BERT-style Models

Our BERT2BERT-style models follow the procedure proposed by Rothe et al.
(2020), which consists of initializing encoder-decoder models with pre-trained
encoder and/or decoder-only checkpoints. We use two configurations: BERT2BERT,
which is an encoder initialized by a BERT-type checkpoint paired with a
decoder initialized with the same checkpoint, and BERTShare, which is similar
to BERT2BERT but the parameters between the encoder and decoder are shared.

We rely on the transformers library Wolf et al. (2020) to initialize models
based on two well-known architectures, BERT and RoBERTa models. On the one
hand, by leveraging the BETO checkpoint, we initialize a BETO2BETO and
BETOShare model. On the other hand, by leveraging the RoBERTa checkpoint from
MarIA, we initialize a RoBERTa2RoBERTa and RoBERTaShare. Note that these
models do not need to continue pre-training but rather fine-tune them directly
in downstream tasks. We will delve into this process in more detail in the
following section.

##  4 Evaluation

This section outlines the downstream tasks selected for benchmarking our
sequence-to-sequence models. These datasets involve generative tasks, where
input and output sequence texts are provided.

###  4.1 Generative Tasks

#### Abstractive Summarization

Summarization involves creating a concise version of a document while
retaining its key information. We consider short and long-form abstractive
summarization tasks to evaluate our models. On the one hand, we considered
MLSUM Scialom et al. (2020) and WikiLingua Ladhak et al. (2020), which are
datasets with short articles and summaries. MLSUM is a collection of 226k
articles with an average number of $\sim$1325 tokens, while WikiLingua counts
76k articles with $\sim$800 tokens on average. The average summary length
represents approximately 2.5% and 9.5% of the length of MLSUM and WikiLingua
articles, respectively. On the other hand, we use XL-Sum Hasan et al. (2021)
and EUR-Lex-Sum Aumiller et al. (2022), which contain longer and more complex
articles of about $\sim$18560 and $\sim$28972 tokens. The summary proportion
of the XL-Sum and Eur-Lex-Sum datasets is approximately 3.06% and 6.74%,
respectively. Although the proportions are similar to those of MLSUM and
WikiLingua, note that due to the longer length of the articles of XL-Sum and
EUR-Lex-Sum, their summaries are more extended, making these data sets more
challenging.

#### Split and Rephrase

The split-and-rephrase task assumes rewriting the content of a long sentence
into shorter and less verbose sentences. To evaluate this task, we use the
Spanish subset of the BiSECT dataset Kim et al. (2021). The subset counts 290k
instances defined based on the OPUS collection Tiedemann and Nygaard (2004).
The average number of tokens within the input sentences is approximately 43,
while after rephrasing into two sentences, the average increases to 45 tokens
across a pair of sentences.

#### Generative Question Answering

To the best of our knowledge, there is currently no dataset tailored for
abstractive question answering in Spanish. In line with prior work , we
utilize discriminative question answering datasets to train the models to
generate the correct answers rather than predicting the specific token
positions of the answer. We rely on MLQA Lewis et al. (2019) and SQAC
Gutiérrez-Fandiño et al. (2021) datasets for this evaluation. MLQA presents a
collection of parallel multi-lingual articles extracted from Wikipedia and
offers a development set and test set professionally translated into Spanish.
Unlike MLQA, SQAC was created exclusively for Spanish evaluation and contains
articles extracted purely from Spanish sources.

| MLSUM | WikiLingua
---|---|---
| R1 | R2 | RL | R1 | R2 | RL
beto2beto | 28.46/28.09 | 10.87/10.34 | 22.89/22.51 | 38.02/37.92 | 17.57/17.43 | 29.38/29.24
betoShare | 28.51/27.84 | 10.90/10.19 | 22.99/22.30 | 37.74/37.68 | 17.41/17.29 | 29.19/29.03
roberta2roberta | 27.94/27.69 | 9.66/9.25 | 21.92/22.07 | 35.68/35.58 | 14.53/14.51 | 26.49/26.37
robertaShare | 28.43/27.86 | 10.17/9.53 | 22.54/21.92 | 35.83/35.70 | 14.95/14.75 | 26.85/26.62
t5s | 29.18/28.60 | 11.42/10.82 | 23.78/23.20 | 36.83/36.63 | 18.25/18.01 | 30.88/32.05
barto | 29.65/29.12 | 10.96/10.32 | 22.71/22.18 | 39.48/39.37 | 19.65/19.46 | 32.33/30.63
Table 1:  Summary task results on the development/test sets for all models using the ROUGE metric. | XLSum | EUR-Lex-Sum
---|---|---
| R1 | R2 | RL | R1 | R2 | RL
beto2beto | 28.76/28.88 | 8.92/9.02 | 20.85/21.03 | 42.76/43.46 | 13.89/14.17 | 22.77/23.07
betoShare | 28.96/29.24 | 9.17/9.22 | 21.08/21.27 | 41.66/42.76 | 13.42/13.73 | 22.66/22.95
roberta2roberta | 26.92/27.22 | 6.98/7.32 | 19.01/19.23 | 44.70/45.63 | 14.58/14.93 | 22.86/23.06
robertaShare | 26.89/27.08 | 6.99/7.15 | 18.94/19.11 | 44.24/44.22 | 13.84/13.90 | 22.55/22.65
t5s | 28.13/28.31 | 10.29/10.48 | 21.61/21.70 | 62.95/61.01 | 47.18/45.27 | 53.08/50.47
barto | 31.02/31.26 | 10.68/10.72 | 21.96/23.81 | 66.49/65.91 | 49.99/48.39 | 56.01/54.15
Table 2:  Long-form summary task results on the development/test sets for all
models using the ROUGE metric.

###  4.2 Fine-tuning

We follow the fine-tuning procedures proposed for the English version models
Lewis et al. (2020); Raffel et al. (2020). Because BARTO and T5S have an
autoregressive decoder, they can be directly fine-tuned for sequence
generation tasks. Specifically, their encoders take a complete input, and then
their decoders generate a target output autoregressively. For BETO2BETO and
similar models, we initialize a transformer with the BETO checkpoint in both
the encoder and decoder. Note that the transformer decoder has cross-attention
layers that are randomly initialized. Subsequently, we fine-tune these models
following a similar process as BARTO and T5S.

We fine-tune all the models on an RTX 3090 GPU for each task using the
transformers library implemented in PyTorch. For a fair comparison, we use the
same hyperparameters with the exception of the batch size, learning rate, and
the number of training epochs. While optimal hyperparameter settings can be
task-dependent, we found a range of values that proved effective across tasks:

  * •

Batch size: 4, 8, 16.

  * •

Learning rate: 3e-5, 5e-5.

  * •

Epochs: 3, 6.

##  5 Results

#### Abstractive Summarization

Table 1 presents a comparison of the results achieved by T5S, BARTO, and
BERT2BERT-style models on the MLSUM and WikiLingua tasks, measured in terms of
ROUGE scores Lin (2004). In particular, BARTO shows the top performance with
an average of 25.48 across the tasks and ROUGE metrics. T5S is the second one
with an average of 24.97, showing a difference of 2.1% with respect to BARTO.
Interestingly, in some cases, T5S outperforms BARTO on the ROUGE-L metric,
indicating that the words in the generated summary appear in exactly the same
order as the target summary. Finally, BETO2BETO is the third one, with an
average ROUGE of 24.39 and a difference of 4.4% with respect to BARTO.

The results for long-form summarization are presented in Table 2. In the case
of XLSum, both BARTO and BETOShare demonstrate superior performance compared
to the other models. However, BARTO exhibits an 8.5% difference advantage over
BETOShare across these metrics. Regarding EUR-Lex-Sum, the contrast between
BARTO and T5S compared to the other models is particularly striking. BARTO is
again the top performer, followed by T5S with a 6.3% difference.
Interestingly, BARTO shows a 69% difference with respect to RoBERTa2RoBERTa.
These results can be attributed to the specific demands of EUR-Lex-Sum, which
necessitates both the processing of lengthy articles and the generation of
extensive summaries. BART and T5S exhibit superior performance in this task,
primarily due to document-formatting utilization during pre-training. This
approach equips the models with enhanced capabilities for processing extensive
sequences, particularly when dealing with longer documents.

| BiSECT
---|---
| SARI | BLEU
beto2beto | 49.45/49.27 | 37.79/37.14
betoShare | 49.72/49.37 | 38.38/37.62
roberta2roberta | 50.98/50.56 | 36.00/35.22
robertaShare | 51.49/51.09 | 37.19/36.16
t5s | 55.95/55.62 | 43.12/42.30
barto | 50.45/50.13 | 39.48/38.97
Table 3:  Split-and-rephrase task results on the development/test sets for all models using SARI and BLEU. | SQAC | MLQA
---|---|---
| R1 | R2 | RL | R1 | R2 | RL
beto2beto | 23.84/24.19 | 10.89/11.10 | 22.87/23.14 | 35.01/33.10 | 23.91/22.29 | 34.36/32.37
betoShare | 28.61/29.22 | 15.08/15.52 | 27.69/28.17 | 33.77/33.37 | 23.61/23.24 | 33.08/32.63
roberta2roberta | 25.42/24.86 | 12.43/12.16 | 24.53/23.78 | 32.21/31.01 | 20.50/18.90 | 31.58/30.12
robertaShare | 33.21/33.28 | 20.37/20.52 | 32.52/32.45 | 29.21/27.78 | 16.65/16.48 | 28.39/26.69
t5s | 72.57/71.20 | 53.81/53.91 | 72.44/71.12 | 66.45/64.81 | 44.16/40.61 | 66.35/64.67
barto | 77.92/77.00 | 58.88/58.45 | 77.78/76.87 | 68.69/66.45 | 44.08/40.87 | 68.57/66.34
Table 4:  Generative question answering results on the development/test sets
for all models using the ROUGE metric.

#### Split and Rephrase

Table 3 shows the comparison of the models in terms of SARI Xu et al. (2016)
and BLEU Papineni et al. (2002) scores for the BiSECT dataset. We can observe
that T5S exhibits the highest performance for both metrics, showing a
difference of 9.7% with respect to BARTO. Interestingly, our BARTO model
achieves the fourth-highest score for SARI and the second-highest score for
BLEU. The excelling performance in the SARI metric of T5S could be related to
the results on the previous task associated with generating a similar sequence
order of words as the target, associated with SARI that explicitly measures
the goodness of words that are added, deleted, and kept. We also hypothesize
the span-filling objective of T5 fosters the ability to split sentences,
explaining the improved performance.

#### Generative Question Answering

Table 4 presents the outcomes of our experiments focused on generative
question answering. Although SQAC and MLQA were originally designed for
discriminative tasks, our results indicate that they serve as a suitable
benchmark even for generative question answering.

BARTO and T5S show the best performance on all metrics and tasks. Notably,
BARTO and T5S present an advantage difference of 3 and 5, respectively, with
respect to , the third best performer, RoBERTaShare. This significant
difference could be explained by the faster adaptation of BARTO and T5S to the
task, while the others may need more fine-tuning steps. Also, the fact that
the self-supervised objective of BART and T5 also results in an improvement
for this task compared to those of BERT and RoBERTa Lewis et al. (2020);
Raffel et al. (2020).

##  6 Conclusions

This work presents a range of sequence-to-sequence models along with a diverse
set of datasets to facilitate their evaluation. Specifically, we introduce
BART, T5, and BERT2BERT-based models, exclusively pre-trained in Spanish. Our
evaluation, encompassing tasks such as summarization, split-and-rephrase, and
generative question answering, has illustrated the capability of these models
to effectively tackle these challenges, with BARTO and T5S emerging as the top
performers.

We believe this work establishes new benchmarks for future research in
encoder-decoder architectures within the Spanish language domain. Looking
ahead, we envision the pre-training of larger-scale versions of BARTO and T5S,
as well as the release of other valuable variants like LED Beltagy et al.
(2020), FlanT5 Chung et al. (2022), CPT Shao et al. (2021), and more.
Furthermore, we see potential in addressing missing sequence-to-sequence tasks
by creating dedicated datasets. Our aim is to contribute to the ongoing
advancement of NLP in Spanish and beyond.

## References

  * Aralikatte et al. (2023) Rahul Aralikatte, Ziling Cheng, Sumanth Doddapaneni, and Jackie Chi Kit Cheung. 2023\.  Varta: A large-scale headline-generation dataset for Indic languages.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 3468–3492, Toronto, Canada. Association for Computational Linguistics.
  * Araujo et al. (2022) Vladimir Araujo, Andrés Carvallo, Souvik Kundu, José Cañete, Marcelo Mendoza, Robert E. Mercer, Felipe Bravo-Marquez, Marie-Francine Moens, and Alvaro Soto. 2022.  Evaluation benchmarks for Spanish sentence representations.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 6024–6034, Marseille, France. European Language Resources Association.
  * Araujo et al. (2023) Vladimir Araujo, Marie-Francine Moens, and Alvaro Soto. 2023.  Learning sentence-level representations with predictive coding.  _Machine Learning and Knowledge Extraction_ , 5(1):59–77.
  * Aumiller et al. (2022) Dennis Aumiller, Ashish Chouhan, and Michael Gertz. 2022.  EUR-lex-sum: A multi- and cross-lingual dataset for long-form summarization in the legal domain.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 7626–7639, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.
  * Cañete et al. (2020) José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez. 2020.  Spanish pre-trained bert model and evaluation data.  _PML4DC at ICLR 2020_.
  * Cañete et al. (2022) José Cañete, Sebastian Donoso, Felipe Bravo-Marquez, Andrés Carvallo, and Vladimir Araujo. 2022.  ALBETO and DistilBETO: Lightweight Spanish language models.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 4291–4298, Marseille, France. European Language Resources Association.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.
  * Dabre et al. (2022) Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh Khapra, and Pratyush Kumar. 2022.  IndicBART: A pre-trained model for indic natural language generation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1849–1863, Dublin, Ireland. Association for Computational Linguistics.
  * De la Rosa et al. (2022) Javier De la Rosa, Eduardo Ponferrada, Manu Romero, Paulo Villegas, Pablo González de Prado Salas, and María Grandury. 2022.  Bertin: Efficient pre-training of a spanish language model using perplexity sampling.  _Procesamiento del Lenguaje Natural_ , page 13–23.
  * Delobelle et al. (2020) Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020.  RobBERT: a Dutch RoBERTa-based Language Model.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3255–3265, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Evdaimon et al. (2023) Iakovos Evdaimon, Hadi Abdine, Christos Xypolopoulos, Stamatis Outsios, Michalis Vazirgiannis, and Giorgos Stamou. 2023.  Greekbart: The first pretrained greek sequence-to-sequence model.
  * Gutiérrez-Fandiño et al. (2021) Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Marc Pàmies, Joan Llop-Palao, Joaquin Silveira-Ocampo, Casimiro Pio Carrino, Aitor Gonzalez-Agirre, Carme Armentano-Oller, Carlos Rodriguez-Penagos, and Marta Villegas. 2021.  Maria: Spanish language models.  _arXiv preprint arXiv:2107.07253_.
  * Gutiérrez-Fandiño et al. (2022) Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Marc Pàmies, Joan Llop-Palao, Joaquín Silveira-Ocampo, Casimiro Pio Carrino, Aitor Gonzalez-Agirre, Carme Armentano-Oller, Carlos Rodriguez-Penagos, and Marta Villegas. 2022.  Maria: Spanish language models.  _Procesamiento del Lenguaje Natural_ , 68(0):39–60.
  * Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021.  XL-sum: Large-scale multilingual abstractive summarization for 44 languages.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4693–4703, Online. Association for Computational Linguistics.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced bert with disentangled attention.  _ArXiv_ , abs/2006.03654.
  * Kamal Eddine et al. (2021) Moussa Kamal Eddine, Antoine Tixier, and Michalis Vazirgiannis. 2021.  BARThez: a skilled pretrained French sequence-to-sequence model.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9369–9390, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kamal Eddine et al. (2022) Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. 2022.  AraBART: a pretrained Arabic sequence-to-sequence model for abstractive summarization.  In _Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP)_ , pages 31–42, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Kim et al. (2021) Joongwon Kim, Mounica Maddela, Reno Kriz, Wei Xu, and Chris Callison-Burch. 2021\.  BiSECT: Learning to split and rephrase sentences with bitexts.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6193–6209, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederick P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_.
  * La Quatra and Cagliero (2023) Moreno La Quatra and Luca Cagliero. 2023.  Bart-it: An efficient sequence-to-sequence model for italian text summarization.  _Future Internet_ , 15(1).
  * Ladhak et al. (2020) Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020.  WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4034–4048, Online. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lewis et al. (2019) Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019.  Mlqa: Evaluating cross-lingual extractive question answering.  _arXiv preprint arXiv:1910.07475_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Martin et al. (2020) Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020.  CamemBERT: a tasty French language model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7203–7219, Online. Association for Computational Linguistics.
  * Mattei et al. (2020) Lorenzo De Mattei, Michele Cafagna, Felice Dell’Orletta, Malvina Nissim, and Marco Guerini. 2020.  Geppetto carves italian into a language model.
  * Nagoudi et al. (2022) El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022.  AraT5: Text-to-text transformers for Arabic language generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Radford and Sutskever (2018) Alec Radford and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rothe et al. (2020) Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020.  Leveraging pre-trained checkpoints for sequence generation tasks.  _Transactions of the Association for Computational Linguistics_ , 8:264–280.
  * Sarti and Nissim (2022) Gabriele Sarti and Malvina Nissim. 2022.  It5: Large-scale text-to-text pretraining for italian language understanding and generation.
  * Scialom et al. (2020) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020.  MLSUM: The multilingual summarization corpus.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8051–8067, Online. Association for Computational Linguistics.
  * Serrano et al. (2022) Alejandro Vaca Serrano, Guillem Garcia Subies, Helena Montoro Zamorano, Nuria Aldama Garcia, Doaa Samy, David Betancur Sanchez, Antonio Moreno Sandoval, Marta Guerrero Nieto, and Alvaro Barbero Jimenez. 2022.  Rigoberta: A state-of-the-art language model for spanish.
  * Shao et al. (2021) Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Hang Yan, Fei Yang, Li Zhe, Hujun Bao, and Xipeng Qiu. 2021.  Cpt: A pre-trained unbalanced transformer for both chinese language understanding and generation.  _arXiv preprint arXiv:2109.05729_.
  * Simoulin and Crabbé (2021) Antoine Simoulin and Benoit Crabbé. 2021.  Un modèle transformer génératif pré-entrainé pour le______ français (generative pre-trained transformer in______ (French) we introduce a French adaptation from the well-known GPT model).  In _Actes de la 28e Conférence sur le Traitement Automatique des Langues Naturelles. Volume 1 : conférence principale_ , pages 246–255, Lille, France. ATALA.
  * Song et al. (2019) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019.  MASS: Masked sequence to sequence pre-training for language generation.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 5926–5936. PMLR.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  In _Advances in Neural Information Processing Systems_ , volume 27. Curran Associates, Inc.
  * Tiedemann and Nygaard (2004) Jörg Tiedemann and Lars Nygaard. 2004.  The OPUS corpus - parallel and free: http://logos.uio.no/opus.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)_ , Lisbon, Portugal. European Language Resources Association (ELRA).
  * Tran et al. (2022) Nguyen Luong Tran, Duong Le, and Dat Quoc Nguyen. 2022.  BARTpho: Pre-trained sequence-to-sequence models for vietnamese.  In _Interspeech 2022_. ISCA.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Virtanen et al. (2019) Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019.  Multilingual is not enough: Bert for finnish.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xu et al. (2016) Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016\.  Optimizing statistical machine translation for text simplification.  _Transactions of the Association for Computational Linguistics_ , 4:401–415.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.


[b]Amarjit Soni

# Meeting the Challenges for Relic Neutrino Detection

P. S. Bhupal Dev

###### Abstract

Inspired by Gounaris-Sakurai and Lee-Zumino, we postulate that the weak vector
and axial vector currents are dominated by $J^{PC}=1^{--}$ and $1^{++}$
resonances respectively in the appropriate channels of $\nu+\bar{\nu}$
annihilation into quark-antiquark pairs when an ultrahigh-energy incoming
$\nu\ (\bar{\nu})$ strikes a relic $\bar{\nu}\ (\nu)$. Despite this and some
other ideas, it appears the detection of relic neutrinos with just the
Standard Model interactions seems extremely difficult at existing or future
neutrino telescopes. Thus any positive signal would be due to some non-
standard interactions of neutrinos.

##  1 Introduction

Clearly the detection of cosmic (relic) neutrino background is an extremely
important problem in Particle Physics, and as is well known, it is extremely
challenging and that is what makes it very interesting [1, 2, 3]. In this
proceeding, we will study the possibility of detecting relic neutrinos via
their resonant annihilation with ultrahigh energy (UHE) neutrinos. Recall that
the effective temperature of the cosmic neutrino background is $T_{\nu}\simeq
1.945~{}{\rm K}\simeq 1.7\times 10^{-4}~{}{\rm eV}$. From the neutrino
oscillation data, we of course only know two mass-square differences, while
the absolute value of the neutrino mass is still unknown. For our numerical
illustrations below, we will assume that the neutrino mass is
$\approx\sqrt{\Delta m_{\rm atm}^{2}}\simeq 0.05~{}{\rm eV}$ which is clearly
much larger than the effective temperature for relic neutrino, so it is almost
sitting at rest. But in reality there should be three masses for the three
neutrino species. As a result we should expect three curves (or peaks/dips, if
relevant) as functions of incident colliding neutrino energies.

Because the neutrino masses are of ${\cal O}(\lesssim 0.05~{}{\rm eV})$ the
colliding neutrino has to possess ultra-high energies even for production of
the lightest Standard Model (SM) resonances, such as $e^{+}e^{-}$ or
$\mu^{+}\mu^{-}$ pairs, or $\rho^{0},\ \phi$ mesons etc; see Table 1. Thus,
the squared center-of-mass energy needed is

$\displaystyle s=2m_{\nu}E_{\nu}\,,\quad{\rm or},\quad
E_{\nu}\simeq\left(\frac{s}{{\rm GeV}^{2}}\right)\times 10^{10}~{}{\rm
GeV}\,,$ (1)

where $E_{\nu}$ is the incoming UHE neutrino energy. Because such large
energies are needed the incident flux rates tend to be low. This is the main
reason for exceedingly small event rates, as shown below.

Particle produced | Incident neutrino energy needed | comments
---|---|---
$e^{+}e^{-}$ | 10 TeV | IceCube detected up to a few PeV
$\mu^{+}\mu^{-}$ | 447 PeV
$\pi^{+}\pi^{-}$ | 784 PeV
$\rho$ | 6 EeV | Balloon experiments detected up to about EeV
$\phi$ | 10 EeV
$J/\Psi$ | 96 EeV
$\Upsilon$ | 1 ZeV
$Z$ | 83 ZeV
NSI $Z^{\prime}$ (50 MeV) | 25 PeV | Detectable at future neutrino experiments
NSI $Z^{\prime}$ (1.8 GeV) | 32 EeV
Table 1: Incident neutrino energies needed for resonant production of various
SM particles, as well as two examples of beyond SM resonances. Recall that 1
EeV = $10^{18}$ eV and 1 ZeV=$10^{21}$ eV.

##  2 Standard Model Resonances

Recall that in the SM [4], left-handed components of $\nu$ and $\ell^{-}$ are
doublets of $SU(2)_{L}$: $\psi_{i}=(\nu_{i},\ \ell_{i}^{-})^{T}$, where
$i=1,2,3$ is the family index. After electroweak symmetry breaking, the
relevant piece of the Lagrangian reads

$\displaystyle{\cal L}\supset-
eQ_{i}\bar{\psi}_{i}\gamma^{\mu}\psi_{i}A_{\mu}-\frac{g}{2\cos\theta_{w}}\bar{\psi}_{i}\gamma^{\mu}(g_{V}^{i}-g_{A}^{i}\gamma_{5})\psi_{i}Z_{\mu}\,,$
(2)

where $A_{\mu}$ and $Z_{\mu}$ are the photon and the $Z$-boson fields
respectively, $g$ is the $SU(2)_{L}$ gauge coupling, $\theta_{w}$ is the weak
mixing angle, $e=g\sin\theta_{w}$ is the positron electric charge, $Q_{i}$ is
the fermion charge relative to the positron, and $g_{V,A}$ are the vector and
axial-vector couplings for the $Z$-boson.

We are all very familiar with $e^{+}e^{-}$ collisions. For the center-of-mass
energy $\sqrt{s}$ much less than the $Z$ mass, the $e^{+}e^{-}\to f\bar{f}$
($f$ being any SM fermion) cross-section falls as $1/s$ for dimensional
reasons. Therefore, the ratio $R$ of the cross-section into quark-antiquark
pairs divided by the cross-section into $\mu^{+}\mu^{-}$ pairs is a constant
that depends on the underlying quark degrees of freedom; see e.g. Figure 52.2
in PDG [4]. One can readily see that for $s\ll m_{Z}^{2}$, the $e^{+}e^{-}$
cross-section is dominated by the vector meson resonances with
$J^{PC}=1^{--}$, such as $\rho$, $\omega$, $\phi$, $J/\Psi$, $\Upsilon$ etc,
as predicted long ago by Gounaris and Sakurai [5].

In much the same way, when it comes to $\nu\bar{\nu}$ collisions, the weak
current now is either of vector or axial-vector type. There is nothing
particularly sacred about the vectors only. Specifically, inspired by the
general idea of Lee and Zumino [6] we postulate that in analogy with the weak
vector current, the axial-vector current will likewise be dominated by the
corresponding $J^{PC}=1^{++}$ resonances, such as $a_{1}(1260)$,
$f_{1}(1285)$, $f_{1}(1420)$, $f_{1}(1510)$, $\chi_{c1}(3872)$, $\chi_{b1}$,
etc.

We use the Breit-Wigner resonance formula [4] for calculating the cross-
section:

$\displaystyle\sigma(\nu_{i}\bar{\nu}_{i}\to X^{*}\to{\rm
anything})=\frac{8\pi}{m_{X}^{2}}{\rm BR}(X\to\nu_{i}\bar{\nu}_{i}){\rm
BR}(X\to{\rm
anything})\frac{s\Gamma_{X}^{2}}{\left(s-m_{X}^{2}\right)^{2}+\frac{s^{2}\Gamma_{X}^{2}}{m_{X}^{2}}}\,,$
(3)

where BR stands for the branching ratio of the resonance particle $X$ to
neutrinos and other final states. For the vector meson case, we use the
following expressions for ${\rm BR}(V\to\nu_{i}\bar{\nu}_{i})$:

$\displaystyle\Gamma(\rho\to\nu_{i}\bar{\nu}_{i})$
$\displaystyle=\frac{G_{F}^{2}}{24\pi}\left(1-2\sin^{2}\theta_{w}\right)^{2}f_{\rho}^{2}m_{\rho}^{3}\,,$
(4) $\displaystyle\Gamma(\omega\to\nu_{i}\bar{\nu}_{i})$
$\displaystyle=\frac{G_{F}^{2}}{96\pi}\left(-\frac{4}{3}\sin^{2}\theta_{w}\right)^{2}f_{\omega}^{2}m_{\omega}^{3}\,,$
(5) $\displaystyle\Gamma(\phi\to\nu_{i}\bar{\nu}_{i})$
$\displaystyle=\frac{G_{F}^{2}}{96\pi}\left(-1+\frac{4}{3}\sin^{2}\theta_{w}\right)^{2}f_{\phi}^{2}m_{\phi}^{3}\,,$
(6) $\displaystyle\Gamma(\Psi\to\nu_{i}\bar{\nu}_{i})$
$\displaystyle=\frac{G_{F}^{2}}{96\pi}\left(1-\frac{8}{3}\sin^{2}\theta_{w}\right)^{2}f_{\Psi}^{2}m_{\Psi}^{3}\,,$
(7) $\displaystyle\Gamma(\Upsilon\to\nu_{i}\bar{\nu}_{i})$
$\displaystyle=\frac{G_{F}^{2}}{96\pi}\left(-1+\frac{4}{3}\sin^{2}\theta_{w}\right)^{2}f_{\Upsilon}^{2}m_{\Upsilon}^{3}\,,$
(8)

where $G_{F}$ is the Fermi constant and $f_{X}$ is the $X$-meson decay
constant. The meson masses, decay widths and decay constants (when directly
available) are taken from the PDG [4]. For the higher resonances, we have
estimated the decay constants using the ratio of their $V\to e^{+}e^{-}$ decay
rates from the PDG and using the formula

$\displaystyle\Gamma(V\to
e^{+}e^{-})=\frac{4\pi}{3}\frac{\alpha^{2}}{m_{V}}f_{V}^{2}c_{V}\,,$ (9)

where the coefficients $c_{V}$ are given in Appendix C of Ref. [7]. For
instance, the decay constant for $\omega^{\prime}\ (1420)$ is estimated as

$\displaystyle\left(\frac{f_{\omega^{\prime}}}{f_{\omega}}\right)^{2}=\frac{\Gamma(\omega^{\prime}\to
e^{+}e^{-})}{\Gamma(\omega\to
e^{+}e^{-})}\frac{m_{\omega^{\prime}}}{m_{\omega}}=\frac{{\rm
BR}(\omega^{\prime}\to e^{+}e^{-})}{{\rm BR}(\omega\to
e^{+}e^{-})}\frac{\Gamma_{\omega^{\prime}}}{\Gamma_{\omega}}\frac{m_{\omega^{\prime}}}{m_{\omega}}=\frac{6.6\times
10^{-7}}{7.39\times 10^{-5}}\frac{290~{}{\rm MeV}}{8.68~{}{\rm
MeV}}\frac{1410~{}{\rm MeV}}{782.66~{}{\rm MeV}}.$

The cross-sections Eq. 3 as a function of the incoming neutrino energy (or the
center-of-energy) are shown in Figure 1 for the SM vector (left panel) and
axial-vector (right panel) meson resonances. The $Z$-peak is also shown for
comparison. It is clear that none of these meson resonances go beyond the
$Z$-continuum. This makes the observational prospects of any of these
resonances in neutrino scattering rather bleak, as we will show in the next
section (see also Ref. [8]).

Figure 1: Vector (left) and axial-vector (right) meson resonances in
$\nu-\bar{\nu}$ scattering. The $Z$-resonance is also shown (in cyan) for
comparison.

##  3 Event Spectrum

Given the cross section and the incident neutrino flux, the total number of
events in a neutrino telescope like IceCube can be calculated as

$\displaystyle N=T\cdot N_{A}\cdot\Omega\cdot V\cdot\int_{E_{\rm min}}^{E_{\rm
max}}dE\Phi(E)\sigma(E)\,,$ (10)

where $T$ is the exposure time (which we take 10 years), $V=1~{}{\rm km}^{3}$
is the detector volume, $N_{A}=6.022\times 10^{23}~{}{\rm cm}^{-3}$ is the
water-equivalent of Avogadro number, $\Omega=2\pi$ sr is the solid angle of
coverage (only events above horizon are considered; the UHE neutrinos coming
from below will be severely attenuated by Earth), $\Phi$ is the cosmogenic
neutrino flux (in units of ${\rm GeV}^{-1}{\rm cm}^{-2}{\rm s}^{-1}{\rm
sr}^{-1}$) for which we use the model given in Ref. [9]. The ranges of
integration $E_{\rm min}$ and $E_{\rm max}$ specify the bin size in the event
spectrum. The event spectrum for the sum of all SM resonances is shown in
Figure 2 left panel. The dashed lines are for the $Z$-resonance alone and the
solid ones are including all the vector and axial-vector resonances shown in
Figure 1. The red lines are for the current IceCube detector volume and the
blue ones are for a future IceCube Gen-II with ten times the IceCube volume.
It is clear from this figure that with the detectors of today or even with
much large ones that may be available in the foreseeable future, it is
extremely unlikely that relic neutrinos will be detected with the SM
interactions only. Thus, if such detectors are able to "see" relic neutrinos
they are very likely to have some non-standard interactions (NSI). We will
study two examples of NSI in the next section to serve as illustrations.

Figure 2: Event rate at current and future versions of IceCube. Left panel: SM
only. Right panel: With NSI.

##  4 Two examples of NSI

We consider a simple NSI model where a light vector boson $Z^{\prime}$ couples
to the neutrinos:

$\displaystyle{\cal L}\supset
g_{L}^{\prime}\bar{\nu}_{i}\gamma^{\mu}P_{L}\nu_{j}Z^{\prime}_{\mu}\,.$ (11)

Note that the couplings can be either flavor-diagonal ($i=j$) or non-diagonal
($i\neq j$). The corresponding $\nu\bar{\nu}$ cross section is given by [10]

$\displaystyle\sigma(s)=\frac{g_{L}^{\prime
4}}{3\pi}\frac{s}{\left(s-m_{Z^{\prime}}^{2}\right)^{2}+m_{Z^{\prime}}^{2}\Gamma_{Z^{\prime}}^{2}}\,,$
(12)

where $\Gamma_{Z^{\prime}}=g_{L}^{\prime 2}m_{Z^{\prime}}/24\pi$ is the
$Z^{\prime}$ decay width (including the charged lepton channels). The
corresponding event spectrum, calculated using Eq. (10), is shown in Figure 2
right panel for a benchmark value of $g_{L}^{\prime}=0.01$ and for two
different values of $m_{Z^{\prime}}=50$ MeV (red) and 1.8 GeV (blue). As in
the left panel, the dashed (solid) lines are for 10 years of IceCube (IceCube
Gen-II) exposure. We see that the number of events in the $\nu\bar{\nu}$
scattering can be sizable in presence of this NSI, but it remains to be seen
whether such large NSI effects are robustly excluded by other laboratory
experiments. This is currently under investigation.

##  5  Summary

  * •

Detection of relic neutrinos is clearly extremely important and also extremely
challenging.

  * •

We have studied the prospects of relic neutrino detection via their resonant
scattering with UHE neutrinos.

  * •

Since neutrino masses are so very small, the colliding neutrino energies
required even for production of muons, pions, $\rho$ etc are hundreds of PeVs,
consequently the incoming cosmogenic fluxes are very low.

  * •

Inspired by the works of Gounaris-Sakurai [5], and bearing in mind the work of
Lee-Zumino (see, “Field-current identities" [6]) in regard to photons and QED,
we generalize it to the weak vector and weak axial-vector currents. Despite
these additions, the cross-section still appears to be dominated by the
Z-boson. However, the collision energy needed for the $Z$ is extremely high,
$\approx 10^{14}~{}{\rm GeV}$, as discussed originally in Ref. [11] and the
corresponding flux exceedingly low rendering the experimental observation a
daunting challenge.

  * •

Consequently, any significant signal of relic neutrinos that is seen in a
detector, for the foreseeable future, is very likely to be a result of NSI.
Since there now exist serious anomalies in muon $(g-2)$, in flavor physics and
in neutrino physics [12], NSI may well be affecting neutrinos [13]. Motivated
by such thoughts, we gave two examples of light $Z^{\prime}$ of mass 50 MeV
and of 1.8 GeV coupling to neutrinos, in which case it turns out that the
$\nu\bar{\nu}$ scattering rates can in principle be enhanced to an observable
level.

## Acknowledgments

This work of BD was supported in part by the U.S. Department of Energy under
Grant No. DE-SC0017987 and by a Fermilab Intensity Frontier Fellowship. The
work of AS was supported in part by the U.S. DOE contract #DE-SC0012704.

## References

  * Gelmini [2005] G. B. Gelmini, Phys. Scripta T 121, 131 (2005), arXiv:hep-ph/0412305 .
  * Yanagisawa [2014] C. Yanagisawa, Frontiers in Physics 2, 30 (2014).
  * Vogel [2015] P. Vogel, AIP Conf. Proc. 1666, 140003 (2015).
  * Zyla  _et al._ [2020] P. A. Zyla _et al._ (Particle Data Group), PTEP 2020, 083C01 (2020).
  * Gounaris and Sakurai [1968] G. J. Gounaris and J. J. Sakurai, Phys. Rev. Lett. 21, 244 (1968).
  * Lee and Zumino [1967] T. D. Lee and B. Zumino, Phys. Rev. 163, 1667 (1967).
  * Bharucha  _et al._ [2016] A. Bharucha, D. M. Straub, and R. Zwicky, JHEP 1608, 098 (2016), arXiv:1503.05534 [hep-ph] .
  * Paschos and Lalakulich [2002] E. A. Paschos and O. Lalakulich, (2002), arXiv:hep-ph/0206273 .
  * Ahlers  _et al._ [2010] M. Ahlers, L. A. Anchordoqui, M. C. Gonzalez-Garcia, F. Halzen, and S. Sarkar, Astropart. Phys. 34, 106 (2010), arXiv:1005.2620 [astro-ph.HE] .
  * Altmannshofer  _et al._ [2016] W. Altmannshofer, C.-Y. Chen, P. S. B. Dev, and A. Soni, Phys. Lett. B 762, 389 (2016), arXiv:1607.06832 [hep-ph] .
  * Weiler [1982] T. J. Weiler, Phys. Rev. Lett. 49, 234 (1982).
  * Fischer  _et al._ [2021] O. Fischer _et al._ , (2021), arXiv:2109.06065 [hep-ph] .
  * Dev  _et al._ [2019] P. S. B. Dev _et al._ , SciPost Phys. Proc 2, 001 (2019), arXiv:1907.00991 [hep-ph] .