# Making the Relation Matters: Relation of Relation Learning Network for
Sentence Semantic Matching

Kun Zhang†, Le Wu†$\ddagger$, Guangyi Lv§, Meng Wang†$\ddagger$, Enhong Chen§,
Shulan Ruan§
Corresponding author

###### Abstract

Sentence semantic matching is one of the fundamental tasks in natural language
processing, which requires an agent to determine the semantic relation among
input sentences. Recently, deep neural networks have achieved impressive
performance in this area, especially BERT. Despite their effectiveness, most
of these models treat output labels as meaningless one-hot vectors,
underestimating the semantic information and guidance of relations that these
labels reveal, especially for tasks with a small number of labels. To address
this problem, we propose a _Relation of Relation Learning Network (R 2-Net)_
for sentence semantic matching. Specifically, we first employ BERT to encode
the input sentences from a global perspective. Then a CNN-based encoder is
designed to capture keywords and phrase information from a local perspective.
To fully leverage labels for better relation information extraction, we
introduce a self-supervised relation of relation classification task for
guiding _R 2-Net_ to consider more about relations. Meanwhile, a triplet loss
is employed to distinguish the intra-class and inter-class relations in a
finer granularity. Empirical experiments on two sentence semantic matching
tasks demonstrate the superiority of our proposed model. As a byproduct, we
have released the codes to facilitate other researches.

##  1 Introduction

Sentence semantic matching is a fundamental Natural Language Processing (NLP)
task that tries to infer the most suitable label for a given sentence pair.
For example, Natural Language Inference (NLI) targets at classifying the input
sentence pair into one of the three relations (i.e., Entailment,
Contradiction, Neutral) (Kim et al. 2018). Paraphrase Identification (PI) aims
at identifying whether the input sentence pair expresses the same meaning
(Dolan and Brockett 2005). Figure 1 gives some examples with different
semantic relations from different datasets.

Figure 1: Some examples from SNLI and SciTail datasets.

As a fundamental technology, sentence semantic matching has been applied
successfully into many NLP fields, e.g., information retrieval (Clark et al.
2016), question answering (Liu et al. 2018), and dialog system (Serban et al.
2016). Currently, most work leverages the advancement of representation
learning techniques (Devlin et al. 2018; Vaswani et al. 2017) to tackle this
task. They focus on input sentences and design different architectures to
explore sentence semantics comprehensively and precisely. Among all these
methods, BERT (Devlin et al. 2018) plays an important role. It adopts multi-
layer transformers to make full use of large corpus (i.e., BooksCorpus and
English Wikipedia) for the powerful pre-trained model. Meanwhile, two self-
supervised learning tasks (i.e., Masked LM and Next Sentence Prediction) are
designed to better analyze sentence semantics and capture as much information
as possible. Based on BERT, plenty of work has made a big step in sentence
semantic modeling (Liu et al. 2019; Radford et al. 2018).

In fact, since relations are the predicting targets of sentence semantic
matching task, most methods do not pay enough attention to the relation
learning. They just leverage annotated labels to represent relations, which
are formulated as one-hot vectors. However, these independent and meaningless
one-hot vectors cannot reveal the rich semantic information and guidance of
relations (Zhang et al. 2018), which will cause an information loss.
Gururangan et al. (2018) has observed that different relations among sentence
pairs imply specific semantic expressions. Taking Figure 1 as an example, most
sentence pairs with “contradiction” relation contain negation words (e.g.,
nobody, never). “entailment” relation often leads to exact numbers being
replaced with approximates (person, some). “Neutral” relation will import some
correct but irrelevant information (e.g., absorb carbon dioxide). Moreover,
the expressions between sentence pairs with different relations are very
different. Therefore, the comparison and contrastive learning among different
relations (e.g., pairwise relation learning) can help models to learn more
about the semantic information implied in the relations, which in turn helps
to strengthen the sentence analysis ability of models. They should be treated
as more than just meaningless one-hot vectors.

One of the solutions for better relation utilization is the embedding method
inspired by Word2Vec. Some researchers try to jointly encode the input
sentences and labels in the same embedding space for better relation
utilization during sentence semantic modeling (Du et al. 2019; Wang et al.
2018a). Despite the progress they have achieved, label embedding method
requires more data and parameters to achieve better utilization of relation
information. It still cannot fully explore the potential of relations due to
the small number of relation categories or the lack of explicit label
embedding initialization (Wang et al. 2018a).

To this end, in this paper, we propose a novel _Relation of Relation Learning
Network (R 2-Net)_ approach to make full use of relation information in a
simple but effective way. In concrete details, we first utilize pre-trained
BERT to model semantic meanings of the input words and sentences from a global
perspective. Then, we develop a CNN-based encoder to obtain partial
information (keywords and phrase information) of sentences from a local
perspective. Next, inspired by self-supervised learning methods in BERT
training processing, we propose a Relation of Relation (R2) classification
task to enhance the learning ability of _R 2-Net_ for the implicit common
features corresponding to different relations. Moreover, a triplet loss is
used to constrain the model, so that the intra-class and inter-class relations
are analyzed better. Along this line, input sentence pairs with the same
relations will be represented much closer and vice versa further apart.
Relation information is properly integrated into sentence pair modeling
processing, which is in favor of tackling the above challenges and improving
the model performance. Extensive evaluations of two sentence semantic matching
tasks (i.e., NLI and PI) demonstrate the effectiveness of our proposed _R
2-Net_ and its advantages over state-of-the-art sentence semantic matching
baselines.

##  2 Related Work

In this section, we mainly introduce the related work from two aspects: 1)
Sentence Semantic Matching, and 2) Label Embedding for Text Classification.

###  2.1 Sentence Semantic Matching

With the development of various neural network technologies such as CNN (Kim
2014), GRU (Chung et al. 2014), and the growing importance of the attention
mechanism (Vaswani et al. 2017; Parikh et al. 2016), plenty of methods have
been exploited for sentence semantic matching on large datasets like SNLI
(Bowman et al. 2015), SciTail (Khot et al. 2018), and Quora (Iyer et al.
2017). Traditionally, researchers try to fully use neural network technologies
to model semantic meanings of sentences in an end-to-end fashion. Among them,
CNNs focus on the local context extraction with different kernels, and RNNs
are mainly utilized to capture the sequential information and semantic
dependency. For example, Mou et al. (2016a) employed a tree-based CNN to
capture the local context information in sentences. Kun et al. (2018) combined
CNN and GRU into a hybrid architecture, which utilizes the advantages of both
networks. They used CNN to generate phrase-level semantic meanings and GRU to
model the word sequence and dependency between sentences.

Recently, attention-based methods have shown very promising results on many
NLP tasks, such as machine translation (Bahdanau et al. 2014), reading
comprehension (Zheng et al. 2019), and NLI (Bowman et al. 2016). Attention
helps to extract the most important parts in sentences, capture semantic
relations, and align the elements of two sentences properly (Cho, Courville,
and Bengio 2015; Zhang et al. 2017). It has become an essential component for
improving model performance and sentence understanding. Early attempts focus
on designing different attention methods that are suitable for specific tasks,
like inner-attention (Liu et al. 2016), co-attention (Kim et al. 2018), and
multi-head attention (Shen et al. 2017). To fully explore the potential of
attention mechanism, Zhang et al. (2019) proposed a dynamic attention
mechanism, which imitates human reading behaviors to select the most important
word at each reading step. This method has achieved impressive performance.
Another direction is pre-trained methods. Devlin et al. (2018) used very large
corpus and multi-layer transformers to obtain a powerful per-trained BERT.
This method leverages multi-head self-attention to encode sentences and
achieves remarkable performances on various NLP tasks. With the powerful
representation ability, pre-trained BERT model has accelerated the NLP
research.

However, most of these methods only focus on the input sentences and treat the
labels as meaningless one-hot vectors, which ignores the potential of label
information (Zhang et al. 2018). There still remains plenty of space for
further improvement on sentence semantic matching.

###  2.2 Label Embedding for Text Classification

As an extremely important part of training data, labels contain much implicit
information that needs to be explored. In computer vision, researchers have
proposed label embedding methods to make full use of label information.

However, research on explicit label utilization in NLP is still a relatively
new domain. One possible reason is that there are not that many labels in NLP
tasks. Thus, label information utilization is only considered on the task with
relatively a large number of labels or multi-task learning. For example, Zhang
et al. (2018) proposed a multi-task label embedding method for better implicit
correlations and common feature extraction among related tasks. Du et al.
(2019) designed an explicit interaction model to analyze the fine-grained
interaction between word representations and label embedding. They have
achieved impressive performance on text classification tasks. In addition,
Wang et al. (2018a) and Pappas and Henderson (2019) transferred the text
classification task to a label-word joint embedding problem. They leveraged
the semantic vectors of labels to guide models to select the important and
relevant parts of input sentences for better performance. The above work
demonstrates the superiority of explicit label utilization and inspires us to
make better use of label information.

Figure 2: Architecture of _Relation of Relation Learning Network (R 2-Net)_.

##  3 Problem Statements

In this section, we will introduce the definition of sentence semantic
matching task and our proposed relation of relation classification task.

###  3.1 Sentence Semantic Matching

Sentence semantic matching task can be formulated as a supervised
classification. Given two input sentences
$\bm{s}^{a}=\\{\bm{x}_{1}^{a},\bm{x}_{2}^{a},...,\bm{x}_{l_{a}}^{a}\\}$ and
$\bm{s}^{b}=\\{\bm{x}_{1}^{b},\bm{x}_{2}^{b},...,\bm{x}_{l_{b}}^{b}\\}$, where
$\bm{x}_{i}^{a}$ and $\bm{x}_{j}^{b}$ are feature tokens for each sentence.
The goal of this task is to train a classifier $\xi$, which is capable of
computing the conditional probability $P(y|\bm{s}^{a},\bm{s}^{b})$ and
predicting the relation for input sentence pair based on the probability.

$\begin{split}&P(y|\bm{s}^{a},\bm{s}^{b})=\xi(\bm{s}^{a},\bm{s}^{b}),\\\
&y^{*}=argmax_{y\in\mathcal{Y}}P(y|\bm{s}^{a},\bm{s}^{b}),\end{split}$ (1)

where the true label $y\in\mathcal{Y}$ indicates the semantic relation between
the input sentence pair. $\mathcal{Y}=\\{entailment,contradiction,neutral\\}$
for NLI task and $\mathcal{Y}=\\{Yes,No\\}$ for PI task.

###  3.2 Relation of Relation Classification

Gururangan et al. (2018) has observed that relations can be helpful to reveal
some implicit features or patterns for semantic understanding and matching. In
order to properly and fully utilize relation information, we propose a
Relation of Relation (R2) classification task to guide models to understand
sentence relation more precisely. Given two input sentence pairs
$(\bm{s}^{a}_{1},\bm{s}^{b}_{1})$ and $(\bm{s}^{a}_{2},\bm{s}^{b}_{2})$, the
goal is to learn a classifying function $\mathcal{F}$ with the ability to
identify whether these two input pairs have the same semantic relation:

$\begin{split}\mathcal{F}((\bm{s}^{a}_{1},\bm{s}^{b}_{1}),(\bm{s}^{a}_{2},\bm{s}^{b}_{2}))=\begin{cases}1,&\mbox{if
}y_{1}=y_{2},\\\ 0,&\mbox{if }y_{1}\not=y_{2},\end{cases}\end{split}$ (2)

where $y_{1}$ and $y_{2}$ stand for the semantic relations of two input
sentence pairs, respectively.

In order to make full use of relation information and do better sentence
semantic matching, the following important questions should be considered:

  * •

Since relations are the predicting targets, how to make full use of relation
information to improve model performance properly without leaking it?

  * •

How to integrate R2 task into matching task effectively for relation usage and
performance improvement?

To this end, we propose _R 2-Net_ to properly and fully utilize relation
information, and tackle the above issues. Next, we will introduce the
technical details of _R 2-Net_.

##  4 Relation of Relation Learning Network (R2-Net)

The overall architecture of _R 2-Net_ is shown in Figure 2(A). To better
describe how _R 2-Net_ tackles the above tasks and integrates R2 task to
enhance the model ability on sentence semantic matching, similar to Section 3,
we also elaborate the technical details from two aspects: 1) Sentence Semantic
Matching Part; 2) Relation of Relation Learning Part.

###  4.1 Sentence Semantic Matching Part

This part focuses on identifying the most suitable label for a given input
sentence pair. Specifically, for an input sentence pair, we first utilize
powerful BERT to generate sentence semantic representation globally.
Meanwhile, we develop a CNN-based encoder to capture the keywords and phrase
information from a local perspective. Thus, the input sentence pair can be
encoded in a comprehensive manner. Based on the comprehensive representation,
we leverage a multi-layer perceptron to predict the corresponding label.

#### Global Encoding.

With the full usage of large corpus and multi-layer transformers, BERT (Devlin
et al. 2018) has accomplished much progress in many NLP tasks. Thus, we select
BERT to generate sentence semantic representations for the input. Moreover,
inspired by ELMo (Peters et al. 2018), we also use the weighted sum of all the
hidden states of words from different transformer layers as the final
contextual representations of input words in sentences.

Specifically, we first split the input sentence pair $(\bm{s}^{a},\bm{s}^{b})$
into BPE tokens (Sennrich et al. 2015). Then, we concatenate two sentences to
the required format, in which “[SEP]” is adopted to concatenate two sentences
and “[CLS]” is added at the beginning and the end of the whole sequence. Then,
we use multi-layer transformer blocks to obtain the representations of words
and sentences in the input. Moreover, as illustrated in Figure 2(B), suppose
there are $L$ layers in the BERT. The contextual word representations in the
input sentence pairs is then a pre-layer weighted sum of transformer block
output, with the weights $\alpha_{1},\alpha_{2},...,\alpha_{L}$.

$\begin{split}\bm{h}_{0}^{l},\bm{H}^{l}&=TransformerBlock(\bm{s}^{a},\bm{s}^{b}),\\\
\bm{H}&=\sum_{l=1}^{L}\alpha_{l}\bm{H}^{l},\quad\bm{v}_{g}=\bm{h}_{0}^{L},\\\
\end{split}$ (3)

where $\bm{h}_{0}^{l}$ denotes the representation of first token “[CLS]” at
the $l^{th}$ layer, and $\bm{v}_{g}$ denotes the global semantic
representation of the input. $\bm{H}^{l}$ represents the sequence features of
the whole input. $\alpha_{l}$ is the weight of the $l^{th}$ layer in BERT and
will be learned during model training.

#### Local Encoding.

The semantic relation within the sentence pair is not only connected with the
important words, but also affected by the local information (e.g., phrase and
local structure). Though Bert leverages multi-layer transformers to perceive
important words to the sentence pair, it still has some weaknesses in modeling
local information. To alleviate these shortcomings, we develop a CNN-based
local encoder to extract the local information from the input.

Figure 2(C) illustrates the structure of this local encoder. The input of this
encoder is the output features $\bm{H}$ from global encoding. We use
convolution operations with different composite kernels (e.g., bigram and
trigram) to process these features. Each operation with different kernels is
capable of modeling patterns with different sizes (e.g., new couple, tall
person). Thus, we can obtain robust and abstract local features of the input
sentence pair. Next, we leverage average pooling and max pooling to enhance
these local features and concatenate them before sending them to a non-linear
transformation. Suppose we have $K$ different kernel sizes, this process can
be formulated as follows:

$\begin{split}\bm{H}^{k}&=CNN_{k}(\bm{H}),\quad k=1,2,...,K,\\\
\bm{h}^{k}_{max}&=max(\bm{H}^{k}),\bm{h}^{k}_{avg}=avg(\bm{H}^{k}),\\\
\bm{h}_{concat}&=[\bm{h}^{1}_{max};\bm{h}^{1}_{avg};...;\bm{h}^{K}_{max};\bm{h}^{K}_{avg}],\\\
\bm{v}_{l}&=ReLu(\bm{W}\bm{h}_{concat}+\bm{b}),\\\ \end{split}$ (4)

where $CNN_{k}$ denotes the convolution operation with the $k^{th}$ kernel.
$[\cdot;\cdot]$ is the concatenation operation. $\bm{v}_{l}$ represents the
local semantic representation of the input. $\\{\bm{W},\bm{b}\\}$ are
trainable parameters. $ReLu(\cdot)$ is the activation function.

After getting the global representation $\bm{v}_{g}$ and local representation
$\bm{v}_{l}$, we investigate the different fusion methods to integrate them
together, including simple concatenation, weighed concatenation, as well as
weighted sum. Finally, we obtain that simple concatenation is flexible and can
achieve comparable performance without adding more training parameters. Thus,
we employ the concatenation $\bm{v}=[\bm{v}_{g};\bm{v}_{l}]$ as the final
semantic representation of the input sentence pair.

Table 1: Performance (accuracy) of models on different NLI dataset. Model | Full test | Hard test | SICK test
---|---|---|---
(1) CENN (Zhang et al. 2017) | 82.1% | 60.4% | 81.8%
(2) CAFE (Tay, Tuan, and Hui 2017) | 85.9% | 66.1% | 86.1%
(3) Gumbel TreeLSTM (Choi, Yoo, and Lee 2018) | 86.0% | 66.7% | 85.8%
(4) Distance-based SAN (Im and Cho 2017) | 86.3% | 67.4% | 86.7%
(5) DRCN (Kim et al. 2018) | 86.5% | 68.3% | 87.4%
(6) DRr-Net (Zhang et al. 2019) | 87.5% | 71.2% | 87.8%
(7) Dynamic Self-Attention (Yoon, Lee, and Lee 2018) | 87.4% | 71.5% | 87.7%
(8) Bert-base (Devlin et al. 2018) | 90.3% | 80.8% | 88.5%
(9) _R 2-Net_ | 91.1% | 81.0% | 89.2%

#### Label Prediction.

This component is adopted to predict the label of input sentence pair, which
is an essential part of traditional sentence semantic matching methods. To be
specific, the input of this component is the semantic representation $\bm{v}$.
We leverage a two-layer MLP to make the final classification, which can be
formulated as follows:

$\begin{split}P(y|(\bm{s}^{a},\bm{s}^{b}))=MLP_{1}(\bm{v}).\\\ \end{split}$
(5)

###  4.2 Relation of Relation Learning Part

This part aims at properly and fully using relation information of input
sentence pairs to enhance the model performance on sentence semantic matching.
In order to achieve this goal, we employ two critical modules to analyze the
pairwise relation and triplet based relation simultaneously. Next, we will
describe each module in detail.

#### Relation of Relation Classification.

Inspired by self-supervised learning methods in BERT, we intend _R 2-Net_ to
make full use of relation information among input sentence pairs in a similar
way. Therefore, we introduce R2 classification task into sentence semantic
matching. Instead of just identifying the most suitable relation of input
sentence pairs, we plan to obtain more knowledge about the input sentence pair
by analyzing the pairwise relation between the semantic representations
($\bm{v}_{1}$ for pair $(\bm{s}^{a}_{1},\bm{s}^{b}_{1})$, and $\bm{v}_{2}$ for
pair $(\bm{s}^{a}_{2},\bm{s}^{b}_{2})$). Since a learnable nonlinear
transformation between representations and loss substantially improves the
model performance (Chen et al. 2020), we first transfer $\bm{v}_{1}$ and
$\bm{v}_{2}$ with a nonlinear transformation. Then, we leverage heuristic
matching (Chen et al. 2017a) to model the similarity and difference between
$\bm{v}_{1}$ and $\bm{v}_{2}$. Next, we send the result $\bm{u}$ to a MLP with
one hidden layer for final classification. This process is formulated as
follows:

$\begin{split}&\bar{\bm{v}}_{1}=ReLu(\bm{W}_{r}\bm{v}_{1}+\bm{b}_{r}),\\\
&\bar{\bm{v}}_{2}=ReLu(\bm{W}_{r}\bm{v}_{2}+\bm{b}_{r}),\\\
&\bm{u}=[\bar{\bm{v}}_{1};\bar{\bm{v}}_{2};(\bar{\bm{v}}_{1}\odot\bar{\bm{v}}_{2});(\bar{\bm{v}}_{1}-\bar{\bm{v}}_{2})],\\\
&P(\hat{y}|(\bm{s}^{a}_{1},\bm{s}^{b}_{1}),(\bm{s}^{a}_{2},\bm{s}^{b}_{2}))=MLP_{2}(\bm{u}),\\\
\end{split}$ (6)

where concatenation can retain all the information (Zhang et al. 2017). The
element-wise product is a certain measure of “similarity” of two sentences
(Mou et al. 2016b). Their difference can capture the degree of distributional
inclusion in each dimension (Weeds et al. 2014). $\hat{y}\in\\{1,0\\}$
indicates whether two input sentence pairs have same relation.

#### Triplet Distance Calculation.

Apart from leveraging R2 classification task to learn pairwise relation
information, we also intend to learn intra-class and inter-class information
from the triplet based relation. Thus, we also introduce a triplet loss
(Schroff et al. 2015) into _R 2-Net_. As a fundamental similarity function,
triplet loss is widely applied in information retrieval area (Liu 2009), and
is able to reduce the distance of input pairs with the same relation and
increase the distance of these with different relations. Therefore, we first
calculate the corresponding distances in this module. To be specific, the
inputs of this component are three semantic representations: $\bm{v}_{a}$ for
anchor pair $(\bm{s}^{a}_{a},\bm{s}^{b}_{a})$, $\bm{v}_{p}$ for positive pair
$(\bm{s}^{a}_{p},\bm{s}^{b}_{p})$, $\bm{v}_{n}$ for negative pair
$(\bm{s}^{a}_{n},\bm{s}^{b}_{n})$. In order to obtain better results, we first
transform them into a common space with a full connection layer (Chen et al.
2020). Then, we calculate the distance between anchor and positive pairs, and
the distance between anchor and negative pairs, respectively. This process is
formulated as follows.

$\begin{split}&\bar{\bm{v}}_{a}=ReLu(\bm{W}_{d}\bm{v}_{a}+\bm{b}_{d}),\\\
&\bar{\bm{v}}_{p}=ReLu(\bm{W}_{d}\bm{v}_{p}+\bm{b}_{d}),\\\
&\bar{\bm{v}}_{n}=ReLu(\bm{W}_{d}\bm{v}_{n}+\bm{b}_{d}),\\\
&d_{ap}=Dist(\bar{\bm{v}}_{a},\bar{\bm{v}}_{p}),\quad
d_{an}=Dist(\bar{\bm{v}}_{a},\bar{\bm{v}}_{n}),\\\ \end{split}$ (7)

where $\\{\bm{W}_{d},\bm{b}_{d}\\}$ are trainable parameters. $Dist(\cdot)$ is
the distance calculation function.

##  5 Experiments

In this section, the details about model implementation are firstly presented.
Then, five benchmark datasets on which the model is evaluated are introduced.
Next, a detailed analysis about the model and experimental results is made.

Table 2: Experimental Results (accuracy) on SciTail dataset. Model | SciTail test
---|---
(1) CAFE (2017) | 83.3%
(2) ConSeqNet (2018b) | 85.2%
(3) BiLSTM Max-Out (2018) | 85.4%
(4) HBMP (Talman et al.2018) | 86.0%
(5) DRr-Net (2019) | 87.4%
(6) Transformer LM (2018) | 88.3%
(7) Bert-base (2018) | 92.0%
(8) _R 2-Net_ | 92.9%

###  5.1 Training Details

#### Loss Function.

As is mentioned in Section 3, both sentence semantic matching and R2 task can
be treated as classification tasks. Thus, we employ Cross-Entropy as the loss
for each input as follows:

$\begin{split}L_{s}&=-\bm{y}_{i}\mathrm{log}P(y_{i}|(\bm{s}^{a}_{i},\bm{s}^{b}_{i})),\\\
L_{R^{2}}&=-\hat{\bm{y}}_{i}\mathrm{log}P(\hat{y}|((\bm{s}^{a}_{1},\bm{s}^{b}_{1}),(\bm{s}^{a}_{2},\bm{s}^{b}_{2}))_{i}),\\\
\end{split}$ (8)

where $\bm{y}_{i}$ is the one-hot vector for the true label of the $i^{th}$
instance. $\hat{\bm{y}}_{i}$ is the one-hot vector for the true relation of
relations of the $i^{th}$ instance pair.

Moreover, in order to learn more from relations and achieve better
performance, we also introduce the triplet loss to force _R 2-Net_ to better
analyze the intra-class and inter-class information among sentence pairs with
same or different relations:

$\begin{split}L_{d}=max((d_{ap}-d_{an}+\alpha)_{i},0),\\\ \end{split}$ (9)

where $\alpha$ is the margin. $(\cdot)_{i}$ denotes the $i^{th}$ triplet pair.

Since these three loss functions require different number of inputs, we modify
the input of _R 2-Net_ to have three input sentence pairs (i.e., anchor pair,
positive pair, and negative pair), as shown in Figure 2(A). Then, we calculate
$L_{s}^{1},L_{s}^{2},L_{s}^{3}$ for label prediction loss of each input pair,
randomly sample two groups from the input to calculate
$L_{R^{2}}^{1},L_{R^{2}}^{2}$ for R2 task loss, and use three input pairs to
calculate $L_{d}$ for triplet loss. Finally, we treat the weighed sum of these
losses with a hyper-parameter $\beta$ as the loss function for entire model as
follows:

$\begin{split}L=\frac{1}{N}\sum_{i=1}^{N}(\beta\frac{L_{s}^{1}+L_{s}^{2}+L_{s}^{3}}{3}+(1-\beta)(\frac{L_{R^{2}}^{1}+L_{R^{2}}^{2}}{2}+L_{d})).\\\
\end{split}$ (10)

Table 3: Experimental Results (accuracy) on Quora and MSRP datasets. Model | Quora test | MSRP test
---|---|---
(1) CENN (Zhang et al. 2017) | 80.7% | 76.4%
(2) L.D.C (Wang, Mi, and Ittycheriah 2016) | 85.6% | 78.4%
(3) REL-TK (Filice et al. 2015) | - | 79.1%
(4) BiMPM (Wang, Hamza, and Florian 2017) | 88.2% | -
(5) pt-DecAttachar.c (Tomar et al. 2017) | 88.4% | -
(6) DIIN (Gong, Luo, and Zhang 2017) | 89.1% | -
(7) DRr-Net (Zhang et al. 2019) | 89.8% | 82.9%
(8) DRCN (Kim et al. 2018) | 90.2% | 82.5%
(9) BERT-base (Devlin et al. 2018) | 91.0% | 84.2%
(10) _R 2-Net_ | 91.6% | 84.3%

#### Model Implementation.

We have tuned the hyper-parameters on validation set for best performance, and
have used early-stop to select the best model. Since _R 2-Net_ has different
hyper-parameter settings on different datasets, we list some common hyper-
parameters as follows.

We apply the BERT-base with 12 layers, hidden size 768, and 12 heads. The
kernel sizes of CNN in local encoding unit are $d_{k}=1,2,3$. The hidden state
size of MLP in _R 2-Net_ is $d_{m}=300$. The distance we use in distance
calculation component is Euclidean Distance. The margin $\alpha$ in the
triplet loss is $\alpha=0.2$. For the pre-trained BERT, we set the learning
rate $10^{-5}$ and use AdamW to fine-tune the parameters. For the rest of
parameters, we set the initial learning rate to be $10^{-3}$ and decrease its
value as the model training. An Adam optimizer with $\beta_{1}=0.9$ and
$\beta_{2}=0.999$ is adopted to optimize these parameters. The entire model is
implemented with PyTorch and
Transformers111https://github.com/huggingface/transformers, and is trained on
two Nvidia Tesla V100-SXM2-32GB GPUs.

###  5.2 Data Description

In this section, we give a brief introduction of the datasets on which we
evaluate all models. They are as follow:

  * •

SNLI: SNLI dataset (Bowman et al. 2015) contains $570,152$ human annotated
sentence pairs. The premise sentences are drawn from the captions of Flickr30k
corpus (Young et al. 2014), and the hypothesis sentences are manually
composed. Despite the original test set, we also select the challenging hard
subset (Gururangan et al. 2018) to evaluate the models.

  * •

SICK: SICK dataset (Marelli et al. 2014) contains $10,000$ English sentence
pairs, generated from 8K ImageFlickr dataset (Hodosh et al. 2013) and STS MSR-
video description dataset222https://www.cs.york.ac.uk/semeval-2012/. Each
sentence pair is generated from randomly selected subsets of the above sources
and manually labeled with the label set as SNLI did.

  * •

SciTail: SciTail dataset (Khot et al. 2018) is created from multiple-choice
science exams and web sentences. It has $27,026$ examples with $10,101$
Entailment examples and $16,925$ Neutral examples.

  * •

Quora: Quora dataset (Iyer et al. 2017) contains over $400,000$ potential
question duplicate pairs, which are drawn from Quora
website333https://www.quora.com/. This dataset has balanced positive and
negative labels, indicating whether the line truly contains a duplicate pair.

  * •

MSRP: MSRP dataset (Dolan and Brockett 2005) consists of $5,801$ sentence
pairs with a binary label. The sentences are distilled from a database of
$13,127,938$ sentence pairs, extracted from $9,516,684$ sentences in $32,408$
news clusters from the web.

Figure 3: Visualization of representation $\bm{v}$ from _R 2-Net_ and BERT-
base models.

###  5.3 Experimental Results

In this section, we will give a detailed analysis about models and
experimental results. We have to note that we use accuracy on different test
sets to evaluate the model performance.

#### Performance on NLI task.

We compared our proposed _R 2-Net_ to several published state-of-the-art
baselines on different NLI datasets. All results are summarized in Table 1 and
Table 2. Several observations are presented as follows.

  * •

It is clear that _R 2-Net_ achieves highly comparable performance over all the
datasets: SNLI, SICK, and SciTail. Specifically, _R 2-Net_ first fully uses
BERT and CNN-based encoder to get a comprehensive understanding of sentence
semantics from global and local perspectives. This is one of the reasons that
_R 2-Net_ outperforms other BERT-free baselines by a large margin. Another
important reason is that _R 2-Net_ employs R2 task and triplet loss to make
full use of relation information. Along this line, _R 2-Net_ is capable of
obtaining intra-class and inter-class knowledge among sentence pairs with the
same or different relations. Thus, it can achieve better performance than all
baselines, including the BERT-base model.

  * •

_R 2-Net_ has more stable performance on the challenging NLI hard test, in
which the pairs with obvious identical words are removed (Gururangan et al.
2018). Despite the obvious indicators, these still have implicit patterns for
relations among sentence pairs. By considering R2 task and triplet loss, _R
2-Net_ has the ability to fully use relation information and obtain the
implicit information, which leads to a better performance.

  * •

BERT-base model (Devlin et al. 2018) outperforms other BERT-free baselines by
a large margin. The main reasons can be grouped into two parts. First, BERT
takes advantages of multi-layer transformers to learn sentence patterns and
sentence semantics on a large corpus. Second, BERT adopts two self-supervised
learning tasks (i.e., MLM and NSP) to better analyze the important words
within a sentence and semantic connection between sentences. However, BERT
still focuses on the input sequence, underestimating the rich semantic
information that relations imply. Therefore, its performance is not as good as
that _R 2-Net_ achieves.

  * •

Among BERT-free baselines, DRr-Net (Zhang et al. 2019) and dynamic self-
attention (Yoon, Lee, and Lee 2018) achieve impressive performances. First,
their performance proves that multi-layer structure and CNN have better
ability to model sentence semantics from global and local perspectives. Then,
they all develop a dynamic attention mechanism to improve self-attention
mechanism. However, their encoding capability of extracting features or
generating semantic representations is still not comparable with BERT. This
observation inspires us that using powerful BERT as a basic encoder will be a
better choice.

Table 4: Ablation performance (accuracy) of _R 2-Net_. Model | SNLI test | SciTail test
---|---|---
(1) Bert-base | 90.3% | 92.0%
(2)  _R 2-Net_ (w/o local encoder) | 90.7% | 92.6%
(3)  _R 2-Net_ (w/o R2 task learning) | 90.5% | 92.3%
(4)  _R 2-Net_ (w/o triplet loss) | 90.9% | 92.6%
(5)  _R 2-Net_ |  91.1% |  92.9%

#### Performance on PI task.

Apart from NLI task, we also select PI task to evaluate the model performance.
PI task concerns whether two sentences express the same meaning and has broad
applications in question answering communities444https://www.quora.com/
555https://www.zhihu.com/. Table 3 reports the performance of models on
different datasets. We also list the observations as follows:

  * •

_R 2-Net_ still achieves highly competitive performance over other baselines.
The results demonstrate that R2 task and triplet loss is effective in helping
our proposed model to learn more about relations and improve the model
performance, even if the number of relations is small.

  * •

Almost all models have a better performance on Quora dataset than MSRP
dataset. One possible reason is that Quora dataset has more data than MSRP
dataset (over 400k sentences pairs v.s. 5,801 sentence pairs). In addition to
the data size, inter-sentence interaction is probably another reason. Lan et
al. (2018) observes that Quora dataset contains many sentence pairs with less
complicated interactions (many identical words in sentence pairs). Meanwhile,
_R 2-Net_ also achieves better improvement on Quora dataset, indicating that
more data or better label utilization method is needed for further performance
improvement on MSRP dataset.

#### Ablation Performance.

The overall experiments have proved the superiority of _R 2-Net_. However,
which part plays a more important role in performance improvement is still
unclear. Therefore, we perform an ablation study to verify the effectiveness
of each part, including CNN-based local encoder, R2 task classification, and
triplet loss. The results are illustrated in Table 4. Note that we select
BERT-base as the baseline to compare the importance of each part. According to
the results, we can observe varying degrees of model performance decline.
Among all of them, R2 task has the biggest impact, and triple loss has a
relatively small impact on the model performance. These observations prove
that R2 task is more important for relation information utilization.

#### Case Study.

To provide some intuitionistic examples for explaining why our model gains a
better performance than other baselines, we sample 700 sentence pairs from
SNLI dataset and send them to _R 2-Net_ and BERT-base models to generate the
semantic representation $\bm{v}$. Then, we leverage t-sne (Maaten and Hinton
2008) to visualize these representations with the same parameter settings.
Figure 3(A)-(B) report the results of _R 2-Net_ and BERT-base models,
respectively. By comparing two figures, we can obtain that the representations
generated by _R 2-Net_ have closer inter-class distances. Moreover, the
representations have more obvious distinctions between different classes.
These observations not only explain why our proposed _R 2-Net_ achieves
impressive performance, but also demonstrates that proper usage of relation
information is able to guide models to analyze sentence semantics more
comprehensively and precisely, which is in favor of tackling sentence semantic
matching.

##  6 Conclusion

In this paper, we presented a simple but effective method named _R 2-Net_ for
sentence semantic matching. This method not only uses powerful BERT and CNN to
encode sentences from global and local perspectives, but also makes full use
of relation information for better performance enhancement. Specifically, we
design a R2 classification task to help _R 2-Net_ for learning the implicit
common knowledge from the pairwise relation learning processing. Moreover, a
triplet loss is employed to constrain _R 2-Net_ for better triplet based
relation learning and intra-class and inter-class information analyzing.
Extensive experiments on NLI and PI tasks demonstrate the superiority of _R
2-Net_. In the future, we plan to combine the advantages of label embedding
method for better sentence semantic comprehension.

##  7 Acknowledgments

This research was partially supported by grants from the National Natural
Science Foundation of China (Grant No. 62006066, 61725203, 61972125, 61732008,
and U19A2079), the Major Program of the National Natural Science Foundation of
China (91846201), the Fundamental Research Funds for the Central Universities,
HFUT. The author wants to thank Cui Gongrongxiu for constructive criticism on
the earlier draft.

## References

  * Bahdanau, Cho, and Bengio (2014) Bahdanau, D.; Cho, K.; and Bengio, Y. 2014.  Neural Machine Translation by Jointly Learning to Align and Translate.  _CoRR_ abs/1409.0473.
  * Bowman et al. (2015) Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015.  A large annotated corpus for learning natural language inference.  In _EMNLP_.
  * Bowman et al. (2016) Bowman, S. R.; Gauthier, J.; Rastogi, A.; Gupta, R.; Manning, C. D.; and Potts, C. 2016.  A Fast Unified Model for Parsing and Sentence Understanding.  In _ACL_.
  * Chen et al. (2017a) Chen, Q.; Zhu, X.; Ling, Z.; Wei, S.; Jiang, H.; and Inkpen, D. 2017a.  Enhanced LSTM for Natural Language Inference.  In _ACL_ , 1657–1668.
  * Chen et al. (2017b) Chen, Q.; Zhu, X.-D.; Ling, Z.-H.; Wei, S.; Jiang, H.; and Inkpen, D. 2017b.  Recurrent Neural Network-Based Sentence Encoder with Gated Attention for Natural Language Inference.  In _RepEval@EMNLP_ , 36–40.
  * Chen et al. (2020) Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. E. 2020.  A Simple Framework for Contrastive Learning of Visual Representations.  _ArXiv_ abs/2002.05709.
  * Cho, Courville, and Bengio (2015) Cho, K.; Courville, A. C.; and Bengio, Y. 2015.  Describing Multimedia Content Using Attention-Based Encoder-Decoder Networks.  _IEEE Trans. Multimedia_ 17: 1875–1886.
  * Choi, Yoo, and Lee (2018) Choi, J.; Yoo, K. M.; and Lee, S.-g. 2018.  Learning to compose task-specific tree structures.  In _AAAI_.
  * Chung et al. (2014) Chung, J.; Gulcehre, C.; Cho, K.; and Bengio, Y. 2014.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling.  _CoRR_ abs/1412.3555.
  * Clark et al. (2016) Clark, P.; Etzioni, O.; Khot, T.; Sabharwal, A.; Tafjord, O.; Turney, P. D.; and Khashabi, D. 2016.  Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions.  In _AAAI_.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ .
  * Dolan and Brockett (2005) Dolan, W. B.; and Brockett, C. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _IWP_.
  * Du et al. (2019) Du, C.; Chen, Z.; Feng, F.; Zhu, L.; Gan, T.; and Nie, L. 2019.  Explicit interaction model towards text classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 6359–6366.
  * Filice, Da San Martino, and Moschitti (2015) Filice, S.; Da San Martino, G.; and Moschitti, A. 2015.  Structural representations for learning relations between pairs of texts.  In _ACL_ , 1003–1013.
  * Gong, Luo, and Zhang (2017) Gong, Y.; Luo, H.; and Zhang, J. 2017.  Natural Language Inference over Interaction Space.  _CoRR_ abs/1709.04348.
  * Gururangan et al. (2018) Gururangan, S.; Swayamdipta, S.; Levy, O.; Schwartz, R.; Bowman, S. R.; and Smith, N. A. 2018.  Annotation Artifacts in Natural Language Inference Data.  In _NAACL-HLT_ , 107–112.
  * He, Gimpel, and Lin (2015) He, H.; Gimpel, K.; and Lin, J. 2015.  Multi-perspective sentence similarity modeling with convolutional neural networks.  In _EMNLP_ , 1576–1586.
  * Hodosh, Young, and Hockenmaier (2013) Hodosh, M.; Young, P.; and Hockenmaier, J. 2013.  Framing image description as a ranking task: Data, models and evaluation metrics.  _Journal of Artificial Intelligence Research_ 47: 853–899.
  * Im and Cho (2017) Im, J.; and Cho, S. 2017.  Distance-based Self-Attention Network for Natural Language Inference.  _CoRR_ abs/1712.02047.
  * Iyer, Dandekar, and Csernai (2017) Iyer, S.; Dandekar, N.; and Csernai, K. 2017.  First quora dataset release: Question pairs.
  * Khot, Sabharwal, and Clark (2018) Khot, T.; Sabharwal, A.; and Clark, P. 2018.  SciTail: A textual entailment dataset from science question answering.  In _AAAI_.
  * Kim et al. (2018) Kim, S.; Hong, J.-H.; Kang, I.; and Kwak, N. 2018.  Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information.  _CoRR_ abs/1805.11360.
  * Kim (2014) Kim, Y. 2014.  Convolutional neural networks for sentence classification.  _arXiv preprint arXiv:1408.5882_ .
  * Kun et al. (2018) Kun, Z.; Guangyi, L.; Le, W.; Enhong, C.; Qi, L.; and Han, W. 2018.  Image-Enhanced Multi-Level Sentence Representation Net for Natural Language Inference.  In _IEEE ICDM_.
  * Lan and Xu (2018) Lan, W.; and Xu, W. 2018.  Neural network models for paraphrase identification, semantic textual similarity, natural language inference, and question answering.  In _COLING_ , 3890–3902.
  * Liu et al. (2018) Liu, Q.; Huang, Z.; Huang, Z.; Liu, C.; Chen, E.; Su, Y.; and Hu, G. 2018.  Finding Similar Exercises in Online Education Systems.  In _SIGKDD_ , 1821–1830. ACM.
  * Liu (2009) Liu, T.-Y. 2009.  Learning to rank for information retrieval.  In _Found. Trends Inf. Retr._ , volume 3, 225–331.
  * Liu et al. (2019) Liu, X.; He, P.; Chen, W.; and Gao, J. 2019.  Multi-task deep neural networks for natural language understanding.  _arXiv preprint arXiv:1901.11504_ .
  * Liu et al. (2016) Liu, Y.; Sun, C.; Lin, L.; and Wang, X. 2016.  Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention.  _CoRR_ abs/1605.09090.
  * Maaten and Hinton (2008) Maaten, L. v. d.; and Hinton, G. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9(Nov): 2579–2605.
  * Marelli et al. (2014) Marelli, M.; Bentivogli, L.; Baroni, M.; Bernardi, R.; Menini, S.; and Zamparelli, R. 2014.  Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment.  In _SemEval_ , 1–8.
  * Mihaylov et al. (2018) Mihaylov, T.; Clark, P.; Khot, T.; and Sabharwal, A. 2018.  Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering.  In _EMNLP_.
  * Mou et al. (2016a) Mou, L.; Men, R.; Li, G.; Xu, Y.; Zhang, L.; Yan, R.; and Jin, Z. 2016a.  Natural Language Inference by Tree-Based Convolution and Heuristic Matching .
  * Mou et al. (2016b) Mou, L.; Men, R.; Li, G.; Xu, Y.; Zhang, L.; Yan, R.; and Jin, Z. 2016b.  Natural Language Inference by Tree-Based Convolution and Heuristic Matching.  In _ACL_ , 130–136.
  * Pappas and Henderson (2019) Pappas, N.; and Henderson, J. 2019.  GILE: A Generalized Input-Label Embedding for Text Classification.  _Transactions of the Association for Computational Linguistics_ 7: 139–155.
  * Parikh et al. (2016) Parikh, A. P.; Täckström, O.; Das, D.; and Uszkoreit, J. 2016.  A Decomposable Attention Model for Natural Language Inference.  In _EMNLP_ , 2249–2255.
  * Peters et al. (2018) Peters, M. E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; and Zettlemoyer, L. 2018.  Deep contextualized word representations.  _ArXiv_ abs/1802.05365.
  * Radford et al. (2018) Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018.  Improving Language Understanding by Generative Pre-Training.
  * Schroff, Kalenichenko, and Philbin (2015) Schroff, F.; Kalenichenko, D.; and Philbin, J. 2015.  FaceNet: A unified embedding for face recognition and clustering.  _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ 815–823.
  * Sennrich, Haddow, and Birch (2015) Sennrich, R.; Haddow, B.; and Birch, A. 2015.  Neural machine translation of rare words with subword units.  _arXiv preprint arXiv:1508.07909_ .
  * Serban et al. (2016) Serban, I. V.; Sordoni, A.; Bengio, Y.; Courville, A. C.; and Pineau, J. 2016.  Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models.  In _AAAI_ , volume 16.
  * Shen et al. (2017) Shen, T.; Zhou, T.; Long, G.; Jiang, J.; Pan, S.; and Zhang, C. 2017.  DiSAN: Directional Self-Attention Network for RNN/CNN-free Language Understanding.  _CoRR_ abs/1709.04696.
  * Talman, Yli-Jyrä, and Tiedemann (2018) Talman, A.; Yli-Jyrä, A.; and Tiedemann, J. 2018.  Natural language inference with hierarchical bilstm max pooling architecture.  _arXiv preprint arXiv:1808.08762_ .
  * Tay, Tuan, and Hui (2017) Tay, Y.; Tuan, L. A.; and Hui, S. C. 2017.  A Compare-Propagate Architecture with Alignment Factorization for Natural Language Inference.  _CoRR_ abs/1801.00102.
  * Tomar et al. (2017) Tomar, G. S.; Duque, T.; Täckström, O.; Uszkoreit, J.; and Das, D. 2017\.  Neural Paraphrase Identification of Questions with Noisy Pretraining.  In _SWCN@EMNLP_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _NIPS_ , 5998–6008.
  * Wang et al. (2018a) Wang, G.; Li, C.; Wang, W.; Zhang, Y.; Shen, D.; Zhang, X.; Henao, R.; and Carin, L. 2018a.  Joint Embedding of Words and Labels for Text Classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2321–2331.
  * Wang et al. (2018b) Wang, X.; Kapanipathi, P.; Musa, R.; Yu, M.; Talamadupula, K.; Abdelaziz, I.; Chang, M.; Fokoue, A.; Makni, B.; Mattei, N.; and Witbrock, M. J. 2018b.  Improving Natural Language Inference Using External Knowledge in the Science Questions Domain.  In _AAAI_.
  * Wang, Hamza, and Florian (2017) Wang, Z.; Hamza, W.; and Florian, R. 2017.  Bilateral Multi-Perspective Matching for Natural Language Sentences.  _CoRR_ abs/1702.03814.
  * Wang, Mi, and Ittycheriah (2016) Wang, Z.; Mi, H.; and Ittycheriah, A. 2016.  Sentence Similarity Learning by Lexical Decomposition and Composition.  In _COLING_.
  * Weeds et al. (2014) Weeds, J.; Clarke, D.; Reffin, J.; Weir, D.; and Keller, B. 2014.  Learning to distinguish hypernyms and co-hyponyms.  In _COLING_ , 2249–2259.
  * Yoon, Lee, and Lee (2018) Yoon, D.; Lee, D.; and Lee, S. 2018.  Dynamic self-attention: Computing attention over words dynamically for sentence embedding.  _arXiv preprint arXiv:1808.07383_ .
  * Young et al. (2014) Young, P.; Lai, A.; Hodosh, M.; and Hockenmaier, J. 2014.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _TACL_ 2: 67–78.
  * Zhang et al. (2018) Zhang, H.; Xiao, L.; Chen, W.; Wang, Y.; and Jin, Y. 2018.  Multi-Task Label Embedding for Text Classification.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 4545–4553.
  * Zhang et al. (2017) Zhang, K.; Chen, E.; Liu, Q.; Liu, C.; and Lv, G. 2017.  A Context-Enriched Neural Network Method for Recognizing Lexical Entailment.  In _AAAI_ , 3127–3133.
  * Zhang et al. (2019) Zhang, K.; Lv, G.; Wang, L.; Wu, L.; Chen, E.; Wu, F.; and Xie, X. 2019.  Drr-net: Dynamic re-read network for sentence semantic matching.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 7442–7449.
  * Zheng et al. (2019) Zheng, Y.; Mao, J.; Liu, Y.; Ye, Z.; Zhang, M.; and Ma, S. 2019.  Human behavior inspired machine reading comprehension.  In _SIGIR_ , 425–434.


The Least Prime in Arithmetic an Progression
Andrew Fiori
University of Lethbridge
Andrew Fiori  acknowledges the support of NSERC Discovery Grant RGPIN-2020-05316.
The computations for this work were conducted on systems supported by the University of Lethbridge and purchased through an NSERC RTI grant.

In his 1979 paper Samuel Wagstaff studied the problem of bounding the first prime in an arithmetic progression.
In this paper we update a number of his computations using advances in hardware.
Based on this we refine his conjecture on Primes in Arithmetic Progression and provide further numerical evidence in support of it.
For instance we conjecture that for $n>3$ we have a bound of $3\phi(n)\log(n)\log(\phi(n))$ and verified this bound to $10^8$.

§ INTRODUCTION

In his 1979 paper, [12], Samuel Wagstaff studied the problem of bounding the first prime in an arithmetic progression.
His paper provides a heuristic and some limited data to justify a new conjectural upper bound on when the first prime would occur.
In the present work we aim to refine the conjecture and provide further data in support of it.
The computations would also be expected to have applications towards an explicit version of Linnik's theorem holding for all moduli as such a result almost certainly requires checking small moduli directly.

Given a modulus $n$ and a number $a<n$ with ${\rm gcd}(a,n)=1$ we shall denote by

\[ P(n,a) := {\rm min}( \{ p\; \text{prime}\;|\;  p = a \pmod{n} \})  \]

and by

\[ P(n) := \underset{(a,n)=1}{\rm max} P(n,a). \]

By Linnik's theorem [5] it is known that there exist an absolute constant $A$ such that $P(n) < n^A$.
For $n$ sufficiently large it has been shown by Xylouris [13] that one may take $A=5.2$, and it has  been conjectured by Kanold [3] that for all $n$ one has $A=2$. No fully explicit result holding for all $n$ is known.

Assuming the Generalized Riemann Hypothesis for $L$-functions of characters of modulus $n$ Titchmarsh [10] showed that $P(n) <\!\!< \phi(n)^2\log(n)^4$.
Meanwhile Turan [11] showed, under the same hypothesis, that for all $\delta > 0$ we have that $P(n,a) <\!\!< n\log(n)^{2+\delta}$ except for at most $o(\phi(n))$ many counterexamples.
Heath-Brown [2] speculated that $P(n) <\!\!<  n\log(n)^2$ whereas Wagstaff more precisely conjectured that $P(n) <\!\!<  \phi(n)\log(n)\log(\phi(n))$.

We note that lower bounds on $P(n)$ have also been studied, see for example [1, 4, 8, 9, 7].

§ HEURISTICS FOR BOUNDS

We shall briefly outline Wagstaff's heuristic.
Fix a function $m=m(n) <\!\!< \log(n)^3$ and set $X = X(n) =m(n) n\log(n)$.
For a given $a$ and $n$ Wagstaff estimates the probability that a number, $a+nk$ for $k=0,\ldots, \floor{X/n}$, is prime is approximately:

\begin{equation}\label{wagprime}  \frac{n}{\phi(n)\log(X)}. \end{equation}

Based on this (and assuming independence) one finds the probability that $P(n,a) > X$ is approximately:

\begin{equation}\label{wagpna}  \left( 1- \frac{n}{\phi(n)\log(X)}\right)^{m\log(n)}  \approx e^{-mn/\phi(n)}. \end{equation}

Hence for a given $n$ then one expects that the probability that $P(n) < X$ is approximately

\begin{equation}\label{wagp} (1- e^{-mn/\phi(n)})^{\phi(n)}.\end{equation}

Based on this Wagstaff makes the following conjecture:

As $n\rightarrow \infty$ for most $n$ we have

\[ P(n) \sim \phi(n)\log(n)\log(\phi(n)). \]

Based on his heuristic we will give a stronger conjecture.

The formula of equation (<ref>) can be thought of as the average probability that one of those numbers is prime. That is, by the prime number theorem the number of primes in that list should be approximately $\frac{1}{\phi(n)}\frac{X}{\log(X)}$ and there are $X/n$ numbers in the list.
We briefly revisit the formula in order to refine the approximation.
For each individual number, $a+nk$ for $k=0,\ldots, \floor{X/n}$, the probability it is prime is approximately

\begin{equation}\label{aprime}   \frac{n}{\phi(n)}\frac{1}{\log( a+kn )}. \end{equation}

It follows that the probability that none of them is prime is approximately:

\[  Y = \prod_{k=0}^{\floor{X/n}} \left( 1 -  \frac{n}{\phi(n)}\frac{1}{\log( a+kn )} \right). \]

We estimate $Y$ by

\begin{align*}
\log(Y)
&= \sum_{k=0}^{\floor{X/n}} \log\left( 1 -  \frac{n}{\phi(n)}\frac{1}{\log( a+kn )} \right) \\
&= \sum_{k=0}^{\floor{X/n}} \sum_{i=1}^\infty - \frac{1}{i}\left( \frac{n}{\phi(n)}\frac{1}{\log( a+kn )} \right)^i \\
&\sim \sum_{k=0}^{\floor{X/n}} \sum_{i=1}^3 - \frac{1}{i}\left( \frac{n}{\phi(n)}\frac{1}{\log( a+kn )} \right)^i \\
&\sim \frac{-1}{n}\frac{n}{\phi(n)}\int_{2}^X  \left(\frac{1}{\log(x)} +\frac{n}{2\phi(n)}\frac{1}{\log(x)^2} + \frac{n^2}{3\phi(n)^2}\frac{1}{\log(x)^3} \right){\rm dx}\\
&\sim\frac{-1}{\phi(n)} \left( \Li(x) + \frac{n}{2\phi(n)}\left(\Li(X) - \frac{X}{\log(X)}\right) + \frac{n^2}{6\phi(n)^2}\left(\Li(X) -\frac{X}{\log(X)}-\frac{X}{\log(X)^2}\right) \right) \\
&\sim\frac{-1}{\phi(n)} \left( \frac{X}{\log(X)} + \frac{X}{\log(X)^2} + \frac{2X}{\log(X)^3}  + \frac{n}{2\phi(n)}\left( \frac{X}{\log(X)^2} + \frac{2X}{\log(X)^3} \right) + \frac{n^2}{6\phi(n)^2}\left( \frac{2X}{\log(X)^3} \right) \right) \\
&\sim\frac{-1}{\phi(n)} \left( \frac{X}{\log(X)} +\left(1+\frac{n}{2\phi(n)}\right)\frac{X}{\log(X)^2} + \left(1+\frac{n}{2\phi(n)}+\frac{n^2}{6\phi(n)^2}\right)\frac{2X}{\log(X)^3}  \right)  \\
&\sim \frac{-X}{\log(X)\phi(n)}\left( 1 +\left(1+\frac{n}{2\phi(n)}\right)\frac{1}{\log(X)} + \left(1+\frac{n}{2\phi(n)}+\frac{n^2}{6\phi(n)^2}\right)\frac{2}{\log(X)^2}\right) \\
&\sim \frac{-X}{\log(X)\phi(n)}
\end{align*}

We thus might be inclined to estimate this by $\frac{-X}{\log(X)\phi(n)}$ and thus estimate $Y$ as:

\[ Y \approx e^{-X/(\log(X)\phi(n))}. \]

If we write $X=m(n) n \log(n)$ with $m(n) <\!\!< \log(n)^3$ then we have $\log(X) \sim \log(n)$ and

\[ X/(\log(X)\phi(n)) \sim m(n)n/\phi(n)  \]

and we recover Wagstaff's formula.

Now, we will be considering the case $X=X(n)=(1+\epsilon)\phi(n)\log(n)\log(\phi(n))$, so the case where $m(n) = (1+\epsilon)\log(\phi(n))\phi(n)/n$. If we put this directly into Wagstaff's formula we obtain:

\[   e^{-mn/\phi(n) } =\frac{1}{\phi(n)^{1+\epsilon}}. \]

However, in this case it is more accurate to write $\log(X) \sim \log(\phi(n))+2\log\log(n)$[We note that $\log(X) >\!\!> \log(n)$]  and we obtain

\begin{align*}
X/(\log(X)\phi(n)) &= (1+\epsilon)\phi(n)\log(n)\log(\phi(n))/\phi(n)(\log(\phi(n))+2\log\log(n)) \\
&\sim  (1+\epsilon)\log(n)/(1+\tfrac{2\log\log(n)}{\log(\phi(n))}) \\
&\sim  (1+\epsilon)\log(n)
\end{align*}

so that we may instead approximate

\begin{equation}\label{apna}  e^{-X/(\log(X)\phi(n)) }  \approx \frac{1}{n^{1+\epsilon}}. \end{equation}

It is important to also note some subtleties in taking exponentials of asymptotic formulas.
For example the denominator we dropped of $(1+\tfrac{2\log\log(n)}{\log(\phi(n))})$ is relatively far from $1$ for small $n$.
One can estimate better

\begin{align*} \log(n)/(1+\tfrac{2\log\log(n)}{\log(\phi(n))})
&\sim \log(n) - \tfrac{2\log\log(n)\log(n)}{\log(\phi(n))} + \tfrac{4\log\log(n)^2\log(n)}{\log(\phi(n))^2} + \cdots \\
&\sim  \log(n) - 2\log\log(n) +  \tfrac{4\log\log(n)^2}{\log(n)} + \cdots
\end{align*}

Upon exponentiation this might yield an additional factor of $\log(n)^{2+2\epsilon}$.

Additionally, while estimating $\log(Y)$ the error term we have discarded of size

\begin{align*}
\frac{-X}{\log(X)\phi(n)} \left(1 + \frac{n}{2\phi(n)}\right)\frac{1}{\log(X)}
\frac{-(1+\epsilon)\phi(n)\log(\phi(n))\log(n)}{(\log(\phi(n))+2\log\log(n))\phi(n)} \left(1 + \frac{n}{2\phi(n)}\right)\frac{1}{(\log(\phi(n))+2\log\log(n))} \\
\frac{-(1+\epsilon)}{1+\frac{2\log\log(n)}{\log(\phi(n))}} \left(1 + \frac{n}{2\phi(n)}\right)\frac{\log(n)}{(\log(\phi(n))+2\log\log(n))} \\
-(1+\epsilon)\left(1 + \frac{n}{2\phi(n)}\right)
\end{align*}

is potentially non-negligible once we exponentiate it.
In particular, if $\frac{n}{\phi(n)}$ is large[Recall that $1<n/\phi(n) <\!\!< \log\log(n)$], then, it becomes less likely that $P(n)$ is larger than $X$ by a factor of up to $\frac{1}{e\log(n)^{1/2}}$.
We note that the lower order terms, for instance of size $O(\frac{1}{\log(X)^3})$, in the exponent will tend not to influence the asymptotic.
However any other errors of constant size, for instance coming from the choice of left end point of integration, will multiply the final formula by a constant.

The final takeaway from these calculations is that the probability $P(n,a) > (1+\epsilon)\phi(n)\log(n)\log(\phi(n))$ should be between

\[ \frac{c_1}{\log(n)^{1/2}n^{1+\epsilon}} \quad\text{ and }\quad \frac{c_2\log(n)^{2+2\epsilon}}{\phi(n)^{1+\epsilon}}. \]

A good estimate might be of the form:

\[ c_3\left( \frac{1}{e^{1+\frac{n}{2\phi(n)}}n}\right)^{\frac{1+\epsilon}{1+\frac{2\log\log(n)}{\log(n)}}}. \]

§ COUNTING OUTLIERS

For $\epsilon>0$ and $x>0$ we define sets of $n$ for which $P(n)$ is a large outlier, that is:

\[ E_\epsilon = \{  n  \;|\;  P(n) > (1+\epsilon) \phi(n)\log(n)\log(\phi(n))\} \]

\[ E_{\epsilon}(x,y) =  \{ x <  n < y \;|\;  P(n) > (1+\epsilon) \phi(n)\log(n)\log(\phi(n))\}. \]

Based on (<ref>)  one finds the probability that $n \in  E_{\epsilon}$ is approximately:

\begin{equation}\label{wagpf} 1-\left(1- \frac{1}{\phi(n)^{1+\epsilon}}\right)^{\phi(n)} \end{equation}

whereas based on (<ref>) it would be

\begin{equation} \label{ap} 1-\left(1- \frac{1}{n^{1+\epsilon}}\right)^{\phi(n)}. \end{equation}

We shall make use of the following lemma:

For any $\epsilon > 0$ and for any $n\ge 2$ we have:

\[    \frac{1}{\phi(n)^{\epsilon}} -\frac{1}{2\phi(n)^{2\epsilon}} \leq  1-\left(1-\frac{1}{\phi(n)^{(1+\epsilon)}}\right)^{\phi(n)}  \leq \frac{1}{\phi(n)^{\epsilon}} \]

\[    \frac{\phi(n)}{n}\frac{1}{n^{\epsilon}}  - \frac{1}{2n^{2\epsilon}} \leq  1-\left(1-\frac{1}{n^{(1+\epsilon)}}\right)^{\phi(n)}  \leq \frac{\phi(n)}{n}\frac{1}{n^{\epsilon}} \leq  \frac{1}{n^{\epsilon}}. \]

Apply the binomial theorem to

\[ \left(1-\frac{1}{\phi(n)^{(1+\epsilon)}}\right)^{\phi(n)}  = 1 - \phi(n) \frac{1}{\phi(n)^{(1+\epsilon)}} + \sum_{k=2}^{\phi(n)} (-1)^k \begin{pmatrix} \phi(n) \\ k \end{pmatrix}  \frac{1}{\phi(n)^{k(1+\epsilon)}}. \]

We notice that the absolute value of the terms $a_k$ in the summation are decreasing and alternating hence any pair of terms $a_{2\ell}+a_{2\ell+1} > 0$ and hence (using also that the last term is positive) we have:

\[  \left(1-\frac{1}{\phi(n)^{(1+\epsilon)}}\right)^{\phi(n)} >  1 - \phi(n) \frac{1}{\phi(n)^{(1+\epsilon)}}. \]

It follows that

\[  1-\left(1-\frac{1}{\phi(n)^{1+\epsilon}}\right)^{\phi(n)}  < \frac{1}{\phi(n)^{\epsilon}}. \]

The argument for the lower bound is similar. Except one additionally groups the third and final terms of the sum.

Similar results hold if one replaces $\frac{1}{\phi(n)^{1+\epsilon}}$ or $\frac{1}{n^{1+\epsilon}}$ with other similar formulas for instance $\frac{1}{\log(n)^c n^{1+\epsilon}}$.

Based on the above it would seem reasonable to conjecture, for example, that for $\epsilon >0$ as $y \to\infty$ we have

\[ |E_\epsilon(x,y)|  \sim  \left(\sum_{n=x}^{y}  \left( 1-\left(1-\frac{1}{n^{1+\epsilon}}\right)^{\phi(n)}\right)\right)   \sim      \sum_{n=x}^{y}    \frac{\phi(n)}{n}\frac{1}{n^{\epsilon}}.  \]

However, as noted in Remark <ref> we expect that our estimates on probabilities could be off by a factor as large as $\log(n)^2$. In Table <ref> we will illustrate how well several different formulas work to estimate these counts.
That said, based on bounds on these probabilities we are able to make the following conjecture:

Let $\epsilon>0$. Then:

* For $\epsilon > 1$ as $y \to\infty$ we have

\[      \frac{1}{(1-\epsilon)\log(y)}(y^{1-\epsilon} - x^{1-\epsilon}) <\!\!<  |E_\epsilon(x,y)| <\!\!<  \frac{1}{\epsilon-1}(\log(x)^2x^{1-\epsilon}).  \]

* For $\epsilon=1$ as $y \to\infty$ we have

\[    \log(\log(y)) <\!\!<  |E_\epsilon(x,y)| <\!\!< (\log(y)^3 - \log(x)^3). \]

* For $\epsilon > 1$ as $y \to\infty$ we have

\[      \frac{1}{(1-\epsilon)\log(y)}(y^{1-\epsilon} - x^{1-\epsilon}) <\!\!<  |E_\epsilon(x,y)| <\!\!<  \frac{1}{1-\epsilon}(\log(y)^2y^{1-\epsilon} - \log(x)^2x^{1-\epsilon}).  \]

* $\underset{n\to\infty}{\limsup} \frac{P(n)}{\phi(n)\log(n)\log(\phi(n))} = 2$.

In particular, for all $\epsilon > 1$ we have that $ E_\epsilon$ is finite and for all $\epsilon \leq 1$ we have that $E_\epsilon$ is infinite

Passing from the estimates for a single $n$ to a conjecture about number of outliers involves an assumption of independence for different values of $n$ and $m$.
Though such an assumption is very reasonable for $n$ and $m$ relatively prime it certainly does not hold for $m=2n$ where $n$ is odd because in this case $\phi(n)=\phi(m)$ and virtually all of
the congruence classes under consideration are effectively the same.
However, provided $2{\rm gcd}(m,n) < m,n$ then it is highly unlikely that any arithmetic progressions modulo ${\rm gcd}(m,n)$ will control the size of either $P(m)$ or $P(n)$.

Indeed, even if we consider the case where half of congruences modulo $n$ have small primes coming from those modulo $d$, for some $d|n$ then considering the remaining classes gives

\[ 1-\left(1-\frac{1}{n^{1+\epsilon}}\right)^{\phi(n)/2} \sim  \frac{\phi(n)}{2}\frac{1}{n^{1+\epsilon}}. \]

And the expectation of being less than $(1+\epsilon)$ is only reduced by half. One can bound the expected impact of this as

\[   \prod_{p|n}\left( 1 - \frac{1}{p} \right) \]

which is bounded below by roughly $\frac{1}{\log\log(n)}$ but is only rarely this small.
One can see in Table  <ref> the impact having many small prime factors has.

We can similarly study small outliers. That is for $0<\epsilon < 1$ we could introduce:

\[ F_\epsilon = \{  n  \;|\;  P(n) < (1-\epsilon) \phi(n)\log(n)\log(\phi(n))\} \]

\[ F_{\epsilon}(x,y) =  \{ x <  n < y \;|\;  P(n) < (1-\epsilon) \phi(n)\log(n)\log(\phi(n))\}. \]

Here we may estimate the probability that $n\in F_\epsilon$ as

\begin{equation}\label{smallest}
\left(1- \frac{1}{n^{1-\epsilon}}\right)^{\phi(n)} =  \left(\left(1- \frac{1}{n^{1-\epsilon}}\right)^{n^{1-\epsilon}} \right)^{\phi(n)/n^{1-\epsilon}} \sim e^{- \phi(n)/n^{1-\epsilon}} < e^{-n^{\epsilon}/\log\log(n)}.
\end{equation}

Recalling Remark <ref> we note that better upper or lower bounds could easily be any number of things between

\[   e^{-cn^{\epsilon}/\log(n)\log\log(n)}\quad\text{ and }\quad e^{-c\log\log(n) n^{\epsilon}}. \]

We illustrate the numerics of several options in Table <ref>.

We do observe that in Table <ref> small outliers often have a large number of small prime factors which agrees with the expectation from (<ref>) that having $\phi(n)/n$ small would increase the probability of this.

It is also  important to note that the approximations in (<ref>) are not all that good for small $n$ so that it is not so surprising Table <ref> suggests these probabilities are significant over-estimates.

Based on this we conjecture:

Let $1>\epsilon > 0$. Then

* As $y\to\infty$ we have

\[  F_{\epsilon}(x,y)  <\!\!< \int_{x}^y e^{-n^{\epsilon}/\log(n)\log\log(n)} {\rm dn}  \]

* $\underset{n\to\infty}{\liminf} \frac{P(n)}{\phi(n)\log(n)\log(\phi(n))} = 1$.

In particular, for all $1>\epsilon>0$ we have that $ F_\epsilon$ is finite.

Based on Conjectures <ref> and <ref> as well as Tables <ref>, <ref>, <ref>, and <ref> we can make the following further conjectures:

For all $n>3$ we have

\[  P(n) < 3 \phi(n)\log(n)\log(\phi(n)) \]

and for all $n>570$ we have

\[  P(n) > 0.5 \phi(n)\log(n)\log(\phi(n)). \]

One could certainly be bolder and predict that

* for all $n>6$ that $P(n) < 2.4 \phi(n)\log(n)\log(\phi(n))$,

* for all $n>6840$ that  $P(n) > 0.6 \phi(n)\log(n)\log(\phi(n)) $, or

* for all $n>198660$  that  $P(n) > 0.7 \phi(n)\log(n)\log(\phi(n)) $.

However, based on our expectations it is also reasonable to think that there may well be at least one further counterexample to each of these. For example predicting for all $n>1623$ that $P(n) < 2.2 \phi(n)\log(n)\log(\phi(n))$, seemed reasonable until we had checked $n$ past $8\cdot 10^6$.

§ COMPUTING $P(N)$

There are two natural approaches to computing $P(n)$.

* For each $a<n$ with ${\rm gcd}(a,n)=1$ compute $P(n,a)$ by searching for a prime $p=a+kn$.
Computing $P(n,a)$ should involve checking $\log n$ many numbers and, conjecturally, a worst case of $(\log n)^2$.

Verifying if a number is prime has running time of (at least) $(\log n)$.
Consequently the algorithm has an expected running time of $O(\phi(n)(\log n)^2)$.

One can perform a time/space trade off to make the primality test $O(1)$ at the expense of precomputing a list of primes. This requires $O(n(\log(n))^2)$ space. The expected running time becomes $O(\phi(n)(\log n))$.

* Iterate over all primes and check if $p\pmod{n}$ is a new residue class until all residue classes are seen (ie: $\phi(n)$ of them).

One expects to need to iterate over primes up to at most $2\phi(n)(\log(n))^2$, There are roughly $n\log(n)$ such primes, and it is beneficial to precompute this list if you are computing many values of $P(n)$.

Assuming precomputation this approach has space complexity $n(\log(n))$ and running time $n\log(n)$.

One can also hybridize the two approaches:

Iterate over primes until you have encountered all except $\phi(n)/\log(n)$ of the congruence classes.  This step has expected running time $O(\phi(n)\log(\phi(n)))$.

For the remaining congruence classes compute $P(n,a)$ as above. This step has an expected running time $O(\phi(n)\log(n))$.

Regardless of the approach taken, the expected[this expectation is based on our conjectural answer to the problem being studied, this conjecture held in practice.] total running time to compute $P(n)$ up to $N$ is then $O(N^2(\log N))$ using $O(N(\log N)^2)$ space.

* All of these can be parallelized for different $n$, sharing any of the the precomputed prime lists.

* The hybrid option was the fastest experimentally. The intuition is that it attempts to minimize the wait time to finding a new prime in a congruence class.

* The time space trade offs become intractable for large $n$ but note that because one can use a bitset to record which $p<3N(\log(N))^2$ are prime actually storing the list of the first $O(N\log(N))$ primes will, for smaller values of $N$, actually be what uses more space.

* Our first implementation used PARI/GP, [6], it took roughly two months of cpu time and $38Gb$ of ram on an Intel(R) Xeon(R) CPU E5-2690 to compute the values given up to $3\cdot 10^6$.

We re-implemented the main parts of the algorithm in c, using the PARI c library.
The improved version computed $P(n)$ for the range $3\cdot 10^6$  to $5\cdot 10^6$ million using $10.7$Gb in $137.9$h of cpu time.
We computed values from $5\cdot 10^6$ million to $10^7$ million using $23.5$Gb and $630.9$h on the same system.
We computed values from $10^7$ million to $1.5\cdot 10^7 $ million using $47.3$Gb and $1125.7$h on the same system.

For the final computations we needed to further optimize memory usage and improve parallelism.
We used $114.4Gb$ and approximately $38000$h of cpu time on the same system to compute the values between $5\cdot 10^7$ and $10^8$.[Frequent power outages caused us to lose metadata, but not the results, for the computations between $1.5\cdot 10^7$ and $5\cdot 10^7$]

There remains room to improve both the memory usage and the use of parallelism in our implementation and this would be desirable before pushing the calculations significantly further.

Counts of large outliers with comparisons to various plausible expectations. This table is for $N=10^8$.
Note that for $\epsilon=1$ the third, forth and fifth column's are respectively $\log(N)$, $1$, and $\log(N)\log\log(N)$.
The value in parenthesis is the ratio comparing to the true count.

$(1-\epsilon)\log\log(N)(N^{1-\epsilon} -1)$
$ \sum_{k=1}^N \frac{1}{\phi(k)^{\epsilon}}$
$ \sum_{k=1}^N \frac{1}{k^{\epsilon}}$
$ \sum_{k=1}^N \frac{\phi(k)}{k^{\epsilon+1}}$

$0.0$     $63512264$     $99999999.00$ ($1.57$)     $5428680.97$  ($0.09$)    $291347395.78$  ($4.59$)     $99999998.00$  ($1.57$)     $99999998.00$ ($1.57$)      $60792708.76$ ($0.96$)

$0.1$     $19112007$     $17609923.25$ ($0.92$)     $955986.56$  ($0.05$)    $51306053.30$  ($2.68$)     $18677431.85$  ($0.98$)     $17609921.90$ ($0.92$)      $10705548.77$ ($0.56$)

$0.2$     $4124995$     $3139856.79$ ($0.76$)     $170452.81$  ($0.04$)    $9147891.08$  ($2.22$)     $3538070.83$  ($0.86$)     $3139855.45$ ($0.76$)      $1908803.23$ ($0.46$)

$0.3$     $832336$     $568723.10$ ($0.68$)     $30874.16$  ($0.04$)    $1656959.96$  ($1.99$)     $682012.03$  ($0.82$)     $568721.81$ ($0.68$)      $345741.42$ ($0.42$)

$0.4$     $167725$     $105157.89$ ($0.63$)     $5708.69$  ($0.03$)    $306374.78$  ($1.83$)     $134431.65$  ($0.80$)     $105156.67$ ($0.63$)      $63927.60$ ($0.38$)

$0.5$     $34331$     $19998.00$ ($0.58$)     $1085.63$  ($0.03$)    $58263.65$  ($1.70$)     $27298.10$  ($0.80$)     $19996.83$ ($0.58$)      $12156.63$ ($0.35$)

$0.6$     $7111$     $3959.73$ ($0.56$)     $214.96$  ($0.03$)    $11536.58$  ($1.62$)     $5780.22$  ($0.81$)     $3958.62$ ($0.56$)      $2406.56$ ($0.34$)

$0.7$     $1551$     $833.96$ ($0.54$)     $45.27$  ($0.03$)    $2429.73$  ($1.57$)     $1302.88$  ($0.84$)     $832.90$ ($0.54$)      $506.35$ ($0.33$)

$0.8$     $352$     $194.05$ ($0.55$)     $10.53$  ($0.03$)    $565.37$  ($1.61$)     $323.90$  ($0.92$)     $193.04$ ($0.55$)      $117.37$ ($0.33$)

$0.9$     $88$     $53.10$ ($0.60$)     $2.88$  ($0.03$)    $154.69$  ($1.76$)     $93.84$  ($1.07$)     $52.13$ ($0.59$)      $31.70$ ($0.36$)

$1.0$     $25$     $18.42$ ($0.74$)     $1.00$  ($0.04$)    $53.67$  ($2.15$)     $33.74$  ($1.35$)     $17.50$ ($0.70$)      $10.65$ ($0.43$)

$1.1$     $14$     $8.42$ ($0.60$)     $0.46$  ($0.03$)    $24.52$  ($1.75$)     $15.51$  ($1.11$)     $7.53$ ($0.54$)      $4.59$ ($0.33$)

$1.2$     $9$     $4.87$ ($0.54$)     $0.26$  ($0.03$)    $14.20$  ($1.58$)     $8.84$  ($0.98$)     $4.03$ ($0.45$)      $2.46$ ($0.27$)

$1.3$     $4$     $3.32$ ($0.83$)     $0.18$  ($0.05$)    $9.67$  ($2.42$)     $5.85$  ($1.46$)     $2.51$ ($0.63$)      $1.53$ ($0.38$)

$1.4$     $3$     $2.50$ ($0.83$)     $0.14$  ($0.05$)    $7.28$  ($2.43$)     $4.27$  ($1.42$)     $1.73$ ($0.58$)      $1.05$ ($0.35$)

$1.5$     $3$     $2.00$ ($0.67$)     $0.11$  ($0.04$)    $5.83$  ($1.94$)     $3.31$  ($1.10$)     $1.26$ ($0.42$)      $0.77$ ($0.26$)

Counts of small outliers with comparisons to our conjectured assymptotic bound. This table is for $N=10^8$.

$\epsilon$     $|F_\epsilon(3,N)|$       $\int e^{-k^\epsilon/(\log k\log\log k)}{\rm dk}$      $\int e^{-k^\epsilon/(\log k)}{\rm dk}$       $\int e^{-k^\epsilon/(\log\log k)}{\rm dk}$

$0.0$     $36487734$    $97998679.90$ ($2.69$)     $94401358.22$ ($2.59$)    $70447724.18$ ($1.93$)

$0.1$     $2164082$    $89136078.96$ ($41.19$)     $72001735.39$ ($33.27$)    $13616607.00$ ($6.29$)

$0.2$     $12027$    $51781016.45$ ($4305.40$)     $15555547.31$ ($1293.39$)    $13978.58$ ($1.16$)

$0.3$     $663$    $2970248.50$ ($4480.01$)     $53206.52$ ($80.25$)    $74.73$ ($0.11$)

$0.4$     $102$    $12962.17$ ($127.08$)     $638.85$ ($6.26$)    $4.95$ ($0.05$)

$0.5$     $17$    $448.27$ ($26.37$)     $52.74$ ($3.10$)    $0.61$ ($0.04$)

$0.6$     $2$    $46.10$ ($23.05$)     $9.27$ ($4.64$)    $0.08$ ($0.04$)

$0.7$     $0$    $7.49$ ($-$)     $2.13$ ($-$)    $0.01$ ($-$)

[1]  P. Erdos,
On some applications of Brun's method,
Acta Sci. Math. (Szeged) v. 13.1.
pp. 57–63.

[2] D.R. Heath-Brown,,
Almost-primes in arithmetic progressions and short intervals,
Mathematical Proceedings of the Cambridge Philosophical Society v. 83.3..
pp. 357–375

[3] H.-J. Kanold,
Über Primzahlen in arithmetischen Folgen,
Mathematische Annalen 156.
pp. 393–395.

[4] E. Landau,
Handbuch der Lehre von der Verteilung der Primzahlen,
Band I, Teubner, Leipzig-Berlin, 1909. Reprinted by Chelsea, New Yord, 1953.

[5] U. V. Linnik,
On the least prime in an arithmetic progression. I. The basic theorem,
Rec. Math. [Mat. Sbornik] N.S., 15 (57).
pp. 139–178.

[6]
The PARI Group, PARI/GP version , Univ. Bordeaux, 2022,

[7] C. Pomerance,
A note on the Least Prime in an Arithmetic Progression,
Journal of Number Theory, 12.
pp. 218-223.

[8] K. Prachar,
Über die kleinste Primzahl einer arithmetischen Reihe,
Reine Angew. Math. 2̱06.
pp. 3–4.

[9] A. Schinzel,
Remark on the paper of K. Prachar 'Über die kleinste arithmetischen Reihe,
Reine Angew. Math. 2̱10.
pp. 121–122.

[10] E. C. Titchmarsh,
A Divisor Problem,
Rend. Cire. Mat. Palermo, 5̱4.
pp. 414–429.

[11] P. Tur an,
Über die Primzahlen der arithmetischen Progression,
Acta Sci. Math. (Szeged) 8̱.
pp. 226-235.

[12] S. S. Wagstaff, Jr,
Greatest of the Least Primes in Arithmetic Progressions Having a Given Modulus,
Mathematics of Computation 33. No. 147,
(1979), pp, 1073–1080.

[13] T. Xylouris,
Über die Nullstellen der Dirichletschen L-Funktionen und die kleinste Primzahl in einer arithmetischen Progression,
Bonn, 2011. - Dissertation, Rheinische Friedrich-Wilhelms-Universität Bonn.

<https://nbn-resolving.org/urn:nbn:de:hbz:5N-27156 >

List of large outliers for $3 \leq n< 10^8$

$n$     $\phi(n)$     $a$     $P(n)$    $P(n)/\phi(n)\log(n)\log(\phi(n))$     factorization

List of small outliers for $3 \leq n< 10^8$

$n$     $\phi(n)$     $a$     $P(n)$    $P(n)/\phi(n)\log(n)\log(\phi(n))$     factorization


# Implementation of atmospheric proton spectrum in GEANT4 simulations for
space applications

A. Ilker Topuz1

(1Institute of Physics, University of Tartu, W. Ostwaldi 1, 50411, Tartu,
Estonia)

###### Abstract

A major part of cosmic rays consists of the primary protons, and this portion
plays a crucial role in the space applications such as shielding of
spacecrafts. In this study, the proton flux values measured at the top of the
atmosphere through the BESS-TeV spectrometer in 2004 are introduced into the
GEANT4 simulations by using a probability grid that generates the
corresponding discrete kinetic energies with a certain discrete probability.
The introduced scheme is tested over a set of the shielding materials such as
aluminum, copper, Kevlar, and polyethylene by computing the total absorbed
dose, which is the measure of the cumulative energy deposited in the
investigated target volumes by protons per unit mass in Gy. It is shown that
the present recipe provides the opportunity to use the discrete energy values
together with the experimental flux values, thereby demonstrating a beneficial
capability in the GEANT4 simulations for diverse space applications.

Keywords:  Proton spectrum; BESS-TeV spectrometer; GEANT4; Monte Carlo
simulations; Discrete energy spectrum.

##  1 Introduction

While the primary composition of cosmic rays includes different particles, the
primary cosmic ray protons constitute the dominant part together with the
alpha particles [1, 2]. Among the notable experimental studies that were
conducted for the measurement of the cosmic ray particles is the BESS-TeV
spectrometer in 2004 where the tabulated proton flux values are provided in
[m-2s-1sr-1GeV-1] at the discrete energies starting from 1.08 GeV up to 463
GeV [3].

By reminding that the major portion of the cosmic rays consists of the primary
protons, the energy spectrum as well as the corresponding population of these
primary particles becomes an important aspect in the development of shielding
materials for the space applications. Motivated by the implementation of the
experimental energy spectrum for the primary cosmic ray protons with the aim
of the space applications in the GEANT4 simulations [4], the proton energy
spectrum measured by the BESS-TeV spectrometer in 2004 is introduced into
GEANT4 by using a probability grid that converts the proton flux values into
the discrete probabilities. Following the implementation in the GEANT4 code,
the primary cosmic proton spectrum is checked over a number of spacecraft
materials including aluminum, copper, Kevlar, and polyethylene [5] by
calculating the total absorbed dose.

This study is organized as follows. In section 2, the implementation of the
discrete probabilities at the discrete energies via the probability grid is
shown in addition to the proton flux variation through the BESS-TeV
spectrometer for an energy interval lying between 0 and 68 GeV. While the
proton spectrum is tested over the shielding materials by computing the total
absorbed dose in section 3, the conclusions are drawn in section 4.

##  2 Implementation in GEANT4

The kinetic energies of the primary protons denoted by $E_{i}$ as well as the
corresponding flux values labeled as $\phi_{p,i}$ at the top of the atmosphere
are acquired from another study that is dedicated to the measurement of the
primary and atmospheric cosmic-ray spectra with the BESS-TeV spectrometer in
2004 [3] as listed in Table 1.

Table 1: BESS-TeV discrete experimental flux values and discrete probabilities
at the corresponding discrete energies between 0 and 68 GeV.

$E_{i}$ [GeV] |  $\phi_{p,i}$ [m-2s-1sr-1GeV-1]
---|---
0.00 | 0.000
1.08 | 350.000
1.26 | 322.000
1.47 | 298.000
1.71 | 271.000
2.00 | 241.000
2.33 | 208.000
2.71 | 174.000
3.16 | 145.000
3.69 | 119.000
4.30 | 95.200
5.01 | 73.500
5.84 | 56.300
6.81 | 43.400
7.93 | 31.500
9.25 | 22.500
10.80 | 15.900
12.60 | 11.200
14.70 | 7.710
17.10 | 5.330
19.90 | 3.630
23.20 | 2.480
27.10 | 1.620
31.60 | 1.090
36.80 | 0.717
42.90 | 0.484
50.00 | 0.315
58.30 | 0.207
68.00 | 0.134

$E_{i}$ [GeV] | $p_{i}$
---|---
0.00 | 0
1.08 | 0.139931881
1.26 | 0.128737331
1.47 | 0.119142002
1.71 | 0.108347257
2.00 | 0.096353095
2.33 | 0.083159518
2.71 | 0.069566135
3.16 | 0.057971779
3.69 | 0.047576840
4.30 | 0.038061472
5.01 | 0.029385695
5.84 | 0.022509043
6.81 | 0.017351553
7.93 | 0.012593869
9.25 | 0.008995621
10.80 | 0.006356905
12.60 | 0.004477820
14.70 | 0.003082499
17.10 | 0.002130963
19.90 | 0.001451294
23.20 | 0.000991517
27.10 | 0.000647685
31.60 | 0.000435788
36.80 | 0.000286660
42.90 | 0.000193506
50.00 | 0.000125939
58.30 | 0.000082759
68.00 | 0.000053573

First, the total proton flux between 0 and 68 GeV is computed by adding up
each flux value at a given kinetic energy over 28 bins, and the probability
for a particular kinetic energy is calculated through the ratio between the
corresponding flux value at this specific kinetic energy and the total proton
flux. Then, the discrete probabilities denoted by $p_{i}$ for the energy
interval lying between 0 and 68 GeV are determined as written in

$p_{i}=\frac{\phi_{p,i}}{\sum\limits_{i=0}^{28}\phi_{p,i}}~{}~{}~{}{\rm
with}~{}~{}~{}\sum\limits_{i=0}^{28}p_{i}=1$ (1)

As shown in Table 1, the proton energy spectrum is a fast decaying function,
the values of which are diminished by three orders of magnitude when the
kinetic energy is 68 GeV.

Figure 1: BESS-TeV proton flux values at the top of the atmosphere [3]: (a)
Variation of the experimental flux between 0 and 68 GeV, (b) Histogram of
discrete probabilities between 0 and 68 GeV, (c) Contrast between the input
dataset before processing with the probability grid and the output values
through the activation of our probability grid.

The variation of the experimental proton flux is shown in Fig. 1(a), while the
histogram of the discrete probabilities for the kinetic energies between 0 and
68 GeV is depicted in Fig. 1(b). The average kinetic energy for the cosmic
protons that exist at the top of the atmosphere is around 2.69 GeV for a
kinetic energy interval between 0 and 68 GeV according to the experimental
measurements via the BESS-TeV spectrometer in 2004. As illustrated in Fig.
1(a) and (b), the maximum probability occurs at 1.08 GeV, whereas 68 GeV
indicates the minimum probability in the present study. For the purpose of
implementing the discrete proton energy spectrum, the present strategy to
inject the incoming protons is subsequently integrated by means of
G4ParticleGun as can be found in the previous studies [6, 7, 8, 9]. By
recalling the unity condition, a grid is built by summing up the discrete
probabilities, the interval of which starts with 0 and ends in 1 as
illustrated in Fig. 2. Thus, each cell in this grid, i.e. the difference
between two points on the probability grid, specifies a discrete probability.

Figure 2: Probability grid that assigns the discrete kinetic energies via the
discrete probabilities in accordance with Table 1.

Then, a random number denoted by $\xi$ between 0 and 1 is generated by using
the pre-defined uniform number generator called G4UniformRand(). Finally, this
random number is scanned on the probability grid by checking the difference
between the grid points, and the particular discrete energy is assigned when
the random number matches with the associated cell. This operation is
summarized as explained in

${\rm
If}~{}\sum\limits_{i=0}^{x}p_{i}<\xi\leq\sum\limits_{i=0}^{x+1}p_{i}{\rm~{}for~{}any}~{}x\in\\{0,1,2,...,N-1\\},{\rm~{}then~{}Energy}=E_{x+1}$
(2)

In the wake of the above-mentioned implementation, the discrete probabilities
listed in Table 1 that are introduced in the GEANT4 code are verified, and the
comparison between Table 1 and the code output is demonstrated in Fig. 1(c).

##  3 Application for space shielding materials

The 28-bin discrete cosmic proton spectrum is tested on a list of shielding
materials that is composed of aluminum, copper, Kevlar, and polyethylene over
a detector setup as depicted in Fig. 3. The kinetic energies of the incoming
protons are collected from the top detector layer as well as the bottom
detector layer in order to calculate the energy loss due to the target
material. The dimensions of the detector layers are $100\times 0.4\times 100$
cm3, while the cubic volume of the shielding materials is 125 cm3. The
distance between the detector layers is 10 cm, and the target material is
situated at the center of the detector configuration.

xyCosmic protonTop detector layerBottom detector layerTarget100 cm10 cm5 cm
Figure 3: Simulation setup consisting of top detector layer as well as bottom
detector layer to compute the energy loss.

Through the detector setup demonstrated in Fig. 3, the total absorbed dose due
to the primary cosmic protons at the top of the atmosphere is determined by
using the ratio of the kinetic energy difference between the detector sections
over the mass of the target material as follows

$D_{\rm Total}=\sum\Delta E_{\rm Kinetic}/{\rm Mass}$ (3)

where the total absorbed dose in MeV/kg is converted into Gy (J/kg) by
utilizing a coefficient of $1.60217663\times 10^{-13}$.

A series of GEANT4 simulations are performed to compute the total energy loss
as well as the total absorbed dose by using the 28-bin discrete cosmic proton
spectrum, and the simulations properties are tabulated in Table
LABEL:Simulationproperties. A narrow planar multi-energetic mono-directional
beam that is generated at ([-0.5, 0.5] cm, 85 cm, [-0.5, 0.5] cm) via
G4ParticleGun is utilized, and the generated protons are propagating in the
vertically downward direction as shown by the black arrow in Fig. 3, i.e. from
the top edge of the simulation box through the bottom edge. The total number
of the generated cosmic protons is $5\times 10^{4}$ in every simulation. All
the materials in the simulation geometry are defined in agreement with the
GEANT4/NIST material database, and QBBC is the reference physics list used in
the present study.

Table 2: Simulation properties. Particle | proton
---|---
Beam direction | Vertical
Momentum direction | (0, -1, 0)
Source geometry | Planar
Initial position (cm) | ([-0.5, 0.5], 85, [-0.5, 0.5])
Number of particles | $5\times 10^{4}$
Energy interval (GeV) | [0, 68]
Energy cut-off (GeV) | 1.08
Average kinetic energy (GeV) | 2.69
Energy distribution | BESS-TeV
Target geometry | Cube
Target volume (cm3) | 5$\times$5$\times$5
Material database | G4/NIST
Reference physics list | QBBC

The proton tracking is maintained by G4Step, and the registered energy values
are post-processed by the aid of a Python script where the energy difference
is first calculated for every single primary proton exiting from the shielding
material, then the energy loss of each detected primary proton is divided by
the target mass and converted to Gy. Only the primary protons that hit the two
detector layers are taken into consideration, and the secondary protons
generated due to the inelastic interactions are neglected.

Regarding the simulations outcomes, the calculated total energy loss as well
as the computed total absorbed doses for aluminum, copper, Kevlar, and
polyethylene are listed in Table LABEL:Simulationoutcomes.

Table 3: Simulation outcomes over $5\times 10^{4}$ cosmic primary protons. Material |  $\sum\Delta E_{\rm Kinetic}$ [MeV] |  $D_{\rm Total}$ [Gy]
---|---|---
Aluminum | $1.011\times 10^{6}$ | $0.479\times 10^{-6}$
Copper | $2.274\times 10^{6}$ | $0.325\times 10^{-6}$
Kevlar | $0.694\times 10^{6}$ | $0.619\times 10^{-6}$
Polyethylene | $0.549\times 10^{6}$ | $0.749\times 10^{-6}$

It is observed that the maximum total energy deposition occurs in the case of
copper, whereas polyethylene yields the maximum total absorbed dose. Contrary
to this trend, the minimum total energy loss exists in the case of
polyethylene, while the minimum total absorbed dose is obtained in the
simulations of copper. Both Kevlar and polyethylene exhibit the same
dosimetric profile.

##  4 Conclusion

The present recipe might be beneficial in the cases where the discrete energy
values together the experimental flux values are provided. It is seen that a
discrete proton spectrum based on the experimental flux values might be at
disposal of the GEANT4 simulations for the space applications by using a
probability grid.

##

  * [1] M. Buchvarova, “Galactic cosmic rays above the earth’s atmosphere,” in Journal of Physics: Conference Series, vol. 2255, p. 012003, IOP Publishing, 2022\.
  * [2] D. P. Bowman, R. Scrandis, and E.-S. Seo, “Investigating cosmic ray elemental spectra and the atmospheric muon neutrino flux,” Advances in Space Research, vol. 70, no. 9, pp. 2703–2713, 2022.
  * [3] S. Haino, T. Sanuki, K. Abe, K. Anraku, Y. Asaoka, H. Fuke, M. Imori, A. Itasaki, T. Maeno, Y. Makida, et al., “Measurements of primary and atmospheric cosmic-ray spectra with the BESS-TeV spectrometer,” Physics Letters B, vol. 594, no. 1-2, pp. 35–46, 2004.
  * [4] S. Agostinelli, J. Allison, K. Amako, J. Apostolakis, H. Araujo, P. Arce, M. Asai, D. Axen, S. Banerjee, G. Barrand, et al., “GEANT4—a simulation toolkit,” Nuclear instruments and methods in physics research section A: Accelerators, Spectrometers, Detectors and Associated Equipment, vol. 506, no. 3, pp. 250–303, 2003.
  * [5] M. M. Finckenor, “Materials for spacecraft,” tech. rep., American Institute of Aeronautics and Astronautics, 2018.
  * [6] A. I. Topuz and M. Kiisk, “Towards energy discretization for muon scattering tomography in GEANT4 simulations: A discrete probabilistic approach,” arXiv preprint arXiv:2201.08804, 2022.
  * [7] A. I. Topuz, M. Kiisk, and A. Giammanco, “DOME: Discrete oriented muon emission in GEANT4 simulations,” Instruments, vol. 6, no. 3, p. 42, 2022\.
  * [8] A. I. Topuz, M. Kiisk, and A. Giammanco, “Particle generation through energy discretization and restrictive planes in GEANT4 simulations for potential applications of cosmic ray muon tomography,” in Journal of Physics: Conference Series, vol. 2438, p. 012150, IOP Publishing, 2023.
  * [9] A. I. Topuz, “COME: Cylindrical oriented muon emission in GEANT4 simulations,” arXiv preprint arXiv:2307.03754, 2023.


ȘS

Online Handbook of Argumentation for AI Volume 3 Edited by

Federico Castagna
Jack Mumford
Stefan Sarkadi
Andreas Xydis

December 2022

Preface

This volume contains revised versions of the papers selected for the third
volume of the Online Handbook of Argumentation for AI (OHAAI). Previously,
formal theories of argument and argument interaction have been proposed and
studied, and this has led to the more recent study of computational models of
argument. Argumentation, as a field within artificial intelligence (AI), is
highly relevant for researchers interested in symbolic representations of
knowledge and defeasible reasoning. The purpose of this handbook is to provide
an open access and curated anthology for the argumentation research community.
OHAAI is designed to serve as a research hub to keep track of the latest and
upcoming PhD-driven research on the theory and application of argumentation in
all areas related to AI. The handbook’s goals are to:

  1. 1.

Encourage collaboration and knowledge discovery between members of the
argumentation community.

  2. 2.

Provide a platform for PhD students to have their work published in a citable
peer-reviewed venue.

  3. 3.

Present an indication of the scope and quality of PhD research involving
argumentation for AI.

The papers in this volume are those selected for inclusion in OHAAI Vol.3
following a back-and-forth peer-review process. The volume thus presents a
strong representation of the contemporary state of the art research of
argumentation in AI that has been strictly undertaken during PhD studies.
Papers in this volume are listed alphabetically by author. We hope that you
will enjoy reading this handbook.

---
Editors
Federico Castagna
Jack Mumford
Stefan Sarkadi
Andreas Xydis
December 2022

Acknowledgements

We thank the senior researchers in the area of Argumentation and Artificial
Intelligence for their efforts in spreading the word about the OHAAI project
with early-career researchers.

We are especially thankful to Sylwia Polberg and the COMMA 2022 organisers for
collaborating with us at the conference summer school SSA 2022, during which
OHAAI ran a student programme that provided the attending students the
opportunity to present and discuss their PhD research amongst peers and
academics in a friendly forum.

We are also grateful to ArXiv for their willingness to publish this handbook.

Our sincere gratitude to Costanza Hardouin for her fantastic work in designing
the OHAAI logo.

We owe many thanks to Sanjay Modgil for helping to form the motivation for the
handbook, and to Elizabeth Black and Simon Parsons for their advice and
guidance that enabled the OHAAI project to come to fruition.

The success of the OHAAI project depends upon the quality feedback provided by
our reviewers. We have been fortunate in securing a diligent and thoughtful
program committee that produced reviews of a reliably high standard. Our
thanks go to our PC:

Andreas Brännström, Théo Duchatelle, Timotheus Kampik, Isabelle Kuhlmann, Lars
Malmqvist, Mariela Morveli-Espinoza, Stipe Pandžić, Christos Rodosthenous,
Robin Schaefer, Kenneth Skiba, Luke Thorburn, Antonio Yuste-Ginel, and Heng
Zheng.

Special thanks must go to the contributing authors:

Lars Bengel, Elfia Bezou-Vrakatseli, Lydia Blümel, Federico Castagna, Giulia
D’Agostino, Daphne Odekerken, Minal Suresh Patil, Jordan Robinson, Hao Wu, and
Andreas Xydis. Thank you for making the world of argumentation greater!

See pages - of PDFPapers/Bengel.pdf

See pages - of PDFPapers/Bezou-Vrakatseli.pdf

See pages - of PDFPapers/Blumel.pdf

See pages - of PDFPapers/Castagna.pdf

See pages - of PDFPapers/DAgostino.pdf

See pages - of PDFPapers/Odekerken.pdf

See pages - of PDFPapers/Patil.pdf

See pages - of PDFPapers/Robinson.pdf

See pages - of PDFPapers/Wu.pdf

See pages - of PDFPapers/Xydis.pdf