DEPARTMENT OF INFORMATION ENGINEERING AND COMPUTER SCIENCE

ICT International Doctoral School

Structural Self-Supervised Objectives for Transformers
Exploiting Unlabeled Data and Structural Self-Supervised Objectives to align
Pre-Training with Fine-Tuning

Luca Di Liello
---
Advisor
---
| Prof. Alessandro Moschitti
| University of Trento
Co-Advisor
---
| Prof. Olga Uryupina
| University of Trento
Mentor
---
| Siddhant Garg
| Amazon Alexa AI

July $2023$

Abstract

_In this Thesis, we leverage unsupervised raw data to develop more efficient
pre-training objectives and self-supervised tasks that align well with
downstream applications.

In the first part, we present three alternative objectives to BERT’s Masked
Language Modeling (MLM), namely Random Token Substitution (RTS), Cluster-based
Random Token Substitution (C-RTS), and Swapped Language Modeling (SLM). Unlike
MLM, all of these proposals involve token swapping rather than replacing
tokens with BERT’s [MASK]. RTS and C-RTS involve predicting the originality of
tokens, while SLM tasks the model at predicting the original token values.
Each objective is applied to several models, which are trained using the same
computational budget and corpora. Evaluation results reveal RTS and C-RTS
require up to 45% less pre-training time while achieving performance on par
with MLM. Notably, SLM outperforms MLM on several Answer Sentence Selection
and GLUE tasks, despite utilizing the same computational budget for pre-
training.

In the second part of the Thesis, we propose self-supervised pre-training
tasks that exhibit structural alignment with downstream applications, leading
to improved performance and reduced reliance on labeled data to achieve
comparable results. We exploit the weak supervision provided by large corpora
like Wikipedia and CC-News, challenging the model to recognize whether spans
of text originate from the same paragraph or document. To this end, we design
(i) a pre-training objective that targets multi-sentence inference models by
performing predictions over multiple spans of texts simultaneously, (ii) self-
supervised objectives tailored to enhance performance in Answer Sentence
Selection and its Contextual version, and (iii) a pre-training objective aimed
at performance improvements in Summarization.

Through continuous pre-training, starting from renowned checkpoints such as
RoBERTa, ELECTRA, DeBERTa, BART, and T5, we demonstrate that our models
achieve higher performance on Fact Verification, Answer Sentence Selection,
and Summarization. We extensively evaluate our proposals on different
benchmarks, revealing significant accuracy gains, particularly when annotation
in the target dataset is limited. Notably, we achieve state-of-the-art results
on the development set of the FEVER dataset and results close to state-of-the-
art models using much more parameters on the test set. Furthermore, our
objectives enable us to attain state-of-the-art results on ASNQ, WikiQA, and
TREC-QA test sets, across all evaluation metrics (MAP, MRR, and P@1). For
Summarization, our objective enhances summary quality, as measured by various
metrics like ROUGE and BLEURT. We maintain that our proposals can be
seamlessly combined with other techniques from recently proposed works, as
they do not require alterations to the internal structure of Transformer
models but only involve modifications to the training tasks. _

Keywords

self-supervised pre-training, structural objectives, transformer models,
answer sentence selection, fact verification, summarization

###### Contents

  1. 1 Introduction
    1. 1.1 Contributions
      1. 1.1.1 Alternative Efficient Pre-Training Objectives
      2. 1.1.2 Specialized Task-oriented Structural Objectives
        1. 1.1.2.1 Self-supervised Objectives for Multi-Sentence Inference
        2. 1.1.2.2 Self-Supervised Objectives for AS2
        3. 1.1.2.3 Self-supervised objectives for Contextual AS2
        4. 1.1.2.4 Self-supervised Objectives for Summarization
    2. 1.2 Implementation & Machines
    3. 1.3 Publications
    4. 1.4 Structure of this Thesis
  2. 2 Background
    1. 2.1 Natural Language Processing
      1. 2.1.1 Question Answering
        1. 2.1.1.1 Passage Retrieval
        2. 2.1.1.2 Answer Sentence Selection
        3. 2.1.1.3 Machine Reading Comprehension
      2. 2.1.2 Fact Verification
      3. 2.1.3 Summarization
    2. 2.2 Language Models
      1. 2.2.1 Transformers
        1. 2.2.1.1 Tokenization
        2. 2.2.1.2 Word Embeddings
        3. 2.2.1.3 Positional Embeddings
        4. 2.2.1.4 Transformer block
        5. 2.2.1.5 Transformer Models
      2. 2.2.2 Pre-Training with objectives
        1. 2.2.2.1 Token-Level Pre-Training Objectives
        2. 2.2.2.2 Sentence-Level Pre-Training Objectives
      3. 2.2.3 Fine-Tuning on downstream tasks
  3. 3 Related Work
    1. 3.1 Language Models
      1. 3.1.1 BERT
      2. 3.1.2 RoBERTa
      3. 3.1.3 ELECTRA
      4. 3.1.4 ALBERT
      5. 3.1.5 DeBERTa
      6. 3.1.6 BART
      7. 3.1.7 T5
    2. 3.2 Natural Language Processing Tasks
      1. 3.2.1 Answer Sentence Selection
      2. 3.2.2 Fact Verification
      3. 3.2.3 Summarization
    3. 3.3 Techniques
      1. 3.3.1 Efficient Objectives for Pre-Training
      2. 3.3.2 Multi-Sentence Inference
      3. 3.3.3 Exploiting Document Structure
  4. 4 Datasets
    1. 4.1 Pre-Training
    2. 4.2 Fine-Tuning
      1. 4.2.1 Answer Sentence Selection
      2. 4.2.2 Fact Verification
      3. 4.2.3 Summarization
    3. 4.3 Benchmark Suites
  5. 5 Alternative Efficient Objectives
    1. 5.1 Alternative Objectives
      1. 5.1.1 Random Token Substitution
      2. 5.1.2 Cluster-based Random Token Substitution
      3. 5.1.3 Swapped Language Modeling
    2. 5.2 Complexity Analysis
    3. 5.3 Experiments
      1. 5.3.1 Architectures
      2. 5.3.2 Pre-Training
      3. 5.3.3 Fine-Tuning
    4. 5.4 Results
      1. 5.4.1 Pre-Training
      2. 5.4.2 Fine-Tuning
        1. 5.4.2.1 Base Models
        2. 5.4.2.2 Small Models
      3. 5.4.3 RTS vs C-RTS
    5. 5.5 Pre-Training Computational Trends
  6. 6 Self-supervised Objectives for Multi-Sentence Inference
    1. 6.1 Limits of Pairwise Models
    2. 6.2 Multi-Sentence Transformer Model
      1. 6.2.1 Joint Encoder Architecture
        1. 6.2.1.1 Fixed Model
        2. 6.2.1.2 Flexible Model
      2. 6.2.2 Multi-Sentence Inference Tasks
      3. 6.2.3 Training and Inference using Joint Encoders
        1. 6.2.3.1 Predict a single label
        2. 6.2.3.2 Predict $k$ labels
      4. 6.2.4 Continuous Pre-Training of Joint Encoders
    3. 6.3 Experiments
      1. 6.3.1 Continuous Pre-Training
      2. 6.3.2 Fine-Tuning
    4. 6.4 Results
      1. 6.4.1 Answer Sentence Selection
        1. 6.4.1.1 Re-Rankers Chain
      2. 6.4.2 Fact Verification
    5. 6.5 Latency Analysis
  7. 7 Self-Supervised Objectives for AS2
    1. 7.1 The issue with common Objectives
    2. 7.2 Alternative Objectives for AS2
      1. 7.2.1 Spans in the Same Paragraph (SSP)
      2. 7.2.2 Spans in Paragraph (SP)
      3. 7.2.3 Paragraphs in the Same Document (PSD)
    3. 7.3 Experiments
      1. 7.3.1 Continuous Pre-Training
        1. 7.3.1.1 Comparison of objectives hardness
      2. 7.3.2 Fine-Tuning
    4. 7.4 Results
      1. 7.4.1 Base Models
      2. 7.4.2 Large Models
      3. 7.4.3 Only MLM Continuous Pre-Training
      4. 7.4.4 Few-Shot Setting
      5. 7.4.5 General Considerations
  8. 8 Self-supervised objectives for Contextual AS2
    1. 8.1 Contextual Models
      1. 8.1.1 Injecting Context into Transformers
    2. 8.2 Context-aware Pre-training Objectives
      1. 8.2.1 Static Document-level Context
      2. 8.2.2 Dynamic Paragraph-level Context
      3. 8.2.3 Dynamic Sentence-level Local Context
    3. 8.3 Experiments
      1. 8.3.1 Continuous Pre-Training
      2. 8.3.2 Fine-Tuning
    4. 8.4 Results
      1. 8.4.1 Base Models
      2. 8.4.2 Large Models
      3. 8.4.3 General Considerations
  9. 9 Self-supervised objectives for Summarization
    1. 9.1 Summarization-oriented Pre-training Objectives
      1. 9.1.1 Static Document-level Summary
    2. 9.2 Experiments
      1. 9.2.1 Continuous Pre-Training
      2. 9.2.2 Fine-Tuning
    3. 9.3 Evaluation
    4. 9.4 Results
      1. 9.4.1 Evaluation with Silver Summaries
      2. 9.4.2 General Considerations
  10. 10 Conclusions
    1. 10.1 Alternative Efficient Objectives
    2. 10.2 Specialized Objectives
      1. 10.2.1 Multi-Sentence Inference
      2. 10.2.2 Answer Sentence Selection
      3. 10.2.3 Contextual Answer Sentence Selection
      4. 10.2.4 Summarization
  11. A Background & Related Work
    1. A.1 Data Organization
    2. A.2 Language Models
      1. A.2.1 Recurrent Neural Networks
      2. A.2.2 Convolutional Neural Networks
    3. A.3 Tokenization methodologies
      1. A.3.1 Byte-Pair Encoding
      2. A.3.2 WordPiece
      3. A.3.3 UnigramLM
      4. A.3.4 Sentence-Piece
      5. A.3.5 Loss functions, Back-Propagation and Optimization
    4. A.4 Machine Reading Comprehension Branches
      1. A.4.1 Extractive Question Answering
      2. A.4.2 Generative Question Answering
      3. A.4.3 Abstractive Question Answering
  12. B Datasets
    1. B.1 GLUE
  13. C Alternative Efficient Objectives
    1. C.1 RTS vs C-RTS
    2. C.2 Negative Results
      1. C.2.1 C-RTS sampling always from the same Cluster
      2. C.2.2 Position-based techniques
      3. C.2.3 LM head-on ELECTRA’s discriminator
  14. D Self-Supervised Objectives for AS2
    1. D.1 Negative Results
      1. D.1.1 Additional positives from Large Language Model

##  Chapter 1 Introduction

In this Thesis, we focus on the development of alternative pre-training
objectives for Language Models (LMs), to improve both efficiency and final
accuracy on several downstream tasks. Language Models (LMs) are advanced
Machine Learning (ML) systems designed for Natural Language Processing (NLP)
and Natural Language Understanding (NLU). Nowadays, LMs are predominantly
based on deep learning architectures, particularly the Transformer [Vaswani et
al., 2017], as seen in models like GPT [Radford and Narasimhan, 2018] and BERT
[Devlin et al., 2019]. These models have brought about a revolution in the NLP
industry and achieved state-of-the-art performance across a wide range of
language-related applications.

The main drawback of LMs regards efficiency. Large Language Models (LLMs)
recently reached a remarkable size, with over 100 billion parameters in some
cases [Chowdhery et al., 2022; Zhang et al., 2022a; Scao et al., 2022]. The
computational requirements for both inference and pre-training of these models
are substantial, demanding a significant amount of computational resources [Xu
et al., 2021; Schwartz et al., 2020; Strubell et al., 2019]. For example, the
carbon footprint of BLOOM, a recently released model with 176B parameters, is
estimated to be equivalent to 30 tonnes of CO2 [Luccioni et al., 2022]. Other
models like GPT-3 [Brown et al., 2020], Gopher [Rae et al., 2021] and LLaMa 2
[Touvron et al., 2023], are estimated to require 500, 350 and 540 tonnes of
CO2 to be fully pre-trained, respectively.

The success of Language Models in NLP derives from two key factors: the
Transformer architecture and the Transfer-Learning (TL) training technique.

###### The Transformer architecture

The Transformer was initially proposed by Vaswani et al. [2017] as a new
architecture for Machine Translation (MT). The Transformer enables to capture
long-range dependencies and contextual information in text. It consists of
stacked neural layers, which allow the model to attend to different parts of
the input text and learn meaningful representations and relations. This is
performed by comparing multiple times the representation of each token with
all the others in the input sequence.

###### Transfer-Learning

Transfer-Learning (TL) is a widely used technique to train LMs by leveraging
knowledge gained from one task to improve performance on another related task
[West et al., 2007]. In traditional Machine Learning (ML) approaches, models
are trained directly on specific tasks using large amounts of labeled data.
However, TL is based on the assumption that most NLP tasks share common
patterns and features, for example the same language and similar
sentence/paragraph/document structures. TL involves two main steps: _pre-
training_ and _fine-tuning_ [Peters et al., 2018]. During pre-training, a
language model is trained from scratch on a vast amount of unlabeled data from
sources like Wikipedia and CommonCrawl to capture general language
understanding. In the fine-tuning step, the pre-trained model is specialized
on a specific task or domain using smaller amounts of labeled data. This
process allows the model to adapt its learned features to the target task,
resulting in improved performance without requiring extensive labeled data for
each individual application.

In the subsequent Sections, we present a comprehensive description of the
contributions of this Thesis, highlighting both the accomplishments and
limitations of the proposed methodologies. Additionally, we provide an in-
depth examination of the structure of this work, including the experimental
setup and the hardware employed. Lastly, we furnish an overview of the papers
published during my Ph.D., relating them to the various chapters of this
Thesis.

###  1.1 Contributions

This Thesis examines the pre-training of Language Models from various angles.
Initially, we demonstrate that the current state-of-the-art training tasks are
not efficient when pre-training time is a constraint. We address this issue by
providing alternative pre-training objectives which are both more efficient
and lead to higher language model accuracies. Secondly, we design new pre-
training tasks that align structurally with the downstream application. This
allows us to fine-tune models over downstream datasets using fewer data while
achieving better accuracies.

While today’s research is shifting toward Auto-Regressive (AR) LLMs that can
generate text for the user, we focus on Auto-Encoders (AE), which still
represent an important technique for information embedding, retrieval and
classification.

Out of all the downstream tasks examined in this research, special emphasis is
placed on Answer Sentence Selection (AS2), a branch of Question Answering (QA)
that constitutes the primary focus of my Ph.D. studies.

####  1.1.1 Alternative Efficient Pre-Training Objectives

In this research, we study the pre-training efficiency and time complexity of
LMs Auto-Encoders. We develop alternative pre-training tasks that reach higher
final accuracies while using less energy. Given the widespread utilization in
recent works [Liu et al., 2019b; He et al., 2020; Lan et al., 2020; Sanh et
al., 2020], we adopt the Masked Language Modeling (MLM) pre-training task of
BERT as the foundation for our experiments. MLM works by masking a small
fraction of the input tokens and by tasking the model at predicting their
original value. This procedure is expensive because the classification head of
the model must span over the whole vocabulary, which usually contains
thousands of tokens, to perform predictions. We design our objectives to
reduce the classification head size and improve training time. Our proposals
that target the inefficiency of MLM can be summarized as:

  * •

Random Token Substitution (RTS), in which input tokens are replaced with
others rather than being masked. Subsequently, the model should predict only
whether tokens are originals or replacements using a lightweight binary
classification head;

  * •

Cluster-base Random Token Substitution (C-RTS), which is similar to RTS but
replacements are harder to be detected because we exploit a simple statistical
approach to find the more challenging substitutions;

  * •

Swapped Language Modeling (SLM), in which tokens are always replaced with
others (and never masked) but the model should predict their original value,
as in MLM.

We pre-training different models with our objectives over the same corpora and
using a standard experimental setting. Then, we evaluate the effectiveness of
our proposals on several downstream tasks. We show that RTS and C-RTS reduce
pre-training time by up to 45% when compared with MLM while maintaining a
comparable level of accuracy. Finally, it becomes evident that MLM is a sub-
optimal technique, as SLM achieves greater accuracy in numerous tasks while
utilizing the same computational resources. More details are given in Chapter
5.

####  1.1.2 Specialized Task-oriented Structural Objectives

The four works presented here introduce specialized pre-training techniques
designed to adapt the model for specific downstream tasks during the pre-
training phase. Our approach involves combining these objectives with the
original training task in a continuous pre-training using various publicly
available checkpoints. Continuous pre-training is a phase in which a model is
trained further on large unlabeled datasets but with different objectives to
align better with specific tasks. Remarkably, this continuous pre-training
utilizes only 5% to 10% of the computational time and the same data as the
original training, ensuring that any observed improvements are not attributed
to the introduction of additional knowledge. Furthermore, we emphasize that
our techniques are orthogonal to other optimization methods, such as pre-fine-
tuning, which is training step performed before the real fine-tuning to
further specialize the model using labeled data. We acknowledge that exploring
the combination of our objectives with these alternative methods is a
promising direction for future research.

#####  1.1.2.1 Self-supervised Objectives for Multi-Sentence Inference

Most of the recently proposed LMs are pre-trained with token-level techniques,
such as Masked Language Modeling (MLM) or Token Detection (TD), and sometimes
also with some sentence-level tasks such as Next Sentence Prediction (NSP) or
Sentence Order Prediction (SOP) [Lan et al., 2020; Devlin et al., 2019; Clark
et al., 2020]. Token-level objectives task the model at reconstructing a
corrupted input while sentence-level objectives task the model at classifying
1 or 2 input sentences, for example by predicting their order in the original
document. This creates LM that perform well when the input is composed of at
most 2 text spans but struggle at processing more than 3 inputs at a time. We
call those models _Pairwise_ sentence classifiers.

In this work, we propose a novel architecture we denote as _Jointwise_ , that
is specifically trained to reason over multiple input text spans at the same
time. To train this multi-sentence model, we design a new pre-training task in
which the model is provided $k+1$ sentences and should predict whether
sentence $\\{s_{i}\\}_{0<i\leq k}$ belongs to the same paragraph as $s_{0}$.
We call this objective Multi-Sentences in the Same Paragraph (MSPP). This task
is self-supervised and does not require annotated data, because documents from
large corpora such as Wikipedia, the BookCorpus or OpenWebText are already
divided into several paragraphs by humans. Since every paragraph in a document
describes the same general topic from different perspectives, our objective
forces the model to reason at the semantic level over multiple text spans.

After the continuous pre-training, which is performed starting from a RoBERTa
[Liu et al., 2019b] checkpoint, we adapt our Jointwise architecture to perform
Answer Sentence Selection and Fact Verification. In AS2, the LM is given a
question and a set of possible answer candidate, and should re-rank them such
that correct answers receive the higher score. Fact Verification is instead
the task of determining whether a claim is supported or refuted by a set of
evidence sentences. For AS2, we provide the model with the question and $k$
candidates, and it reasons about them jointly to find the best answer. For
Fact Verification, we feed the model with the claim and a set of evidence
sentences, and the model takes advantage of the latter to jointly predict
whether the claim is supported or not.

By fine-tuning our models over different datasets, we show significant gains
in performance over the baseline Pairwise classifier, also outperforming
methods that exploit additional labeled data in some scenarios. This work is
presented in Chapter 6.

#####  1.1.2.2 Self-Supervised Objectives for AS2

There are many cases in which the Jointwise model described before does not
adapt well to the final task. For example, in Answer Sentence Selection there
may be too few answer candidates to fully exploit the Jointwise architecture
effectively. Moreover, the Jointwise inference speed might be slower than an
equivalent Pairwise model because of the quadratic complexity growth in the
input sequence length. This may be an issue in latency-constrained scenarios,
like virtual assistants. For those reasons, here we present self-supervised
objectives that target Pairwise model pre-training.

Pairwise models are usually pre-trained with weak supervision using sentence-
level objectives such as Next Sentence Prediction or Sentence Order
Prediction. We show that those objectives are solved with high accuracies by
Transformer models, thus providing weak signals while training. Moreover,
those objectives do not force the model at reasoning at the semantic level
because are mostly designed only to understand the order of sentences in a
given corpus.

We exploit the fact that large unlabeled corpora are divided into documents
and paragraphs by humans as a weak form of supervision. We design 3 self-
supervised tasks for pre-training that are harder to be solved than previous
sentence-level objectives and that force the model to reason at the semantic
level. In particular, we propose:

  * •

Sentences in the Same Paragraph (SSP): SSP challenges the model at predicting
whether two spans of text are extracted from the same paragraph of a document;

  * •

Sentences in Paragraph (SP): in SP, we extract a span of text from a paragraph
and we provide it to the model along with the remaining text in the paragraph.
Here, the model should predict if the span belongs to that paragraph in the
original document;

  * •

Paragraphs in the Same Document (PSD): PSD tasks the model with predicting
whether two entire paragraphs belong to the same original document.

After performing continuous pre-training starting from several checkpoints of
RoBERTa, ELECTRA and DeBERTa, we show that our pre-training objectives
outperform the baselines on many Answer Sentence Selection datasets and we
reach state-of-the-art performance on WikiQA and TREC-QA. Moreover, we test
our objectives in the few-shot setting, underlining the optimal performance of
our approaches when the data on the target task is scarce. All the details
about these experiments are provided in Chapter 7.

#####  1.1.2.3 Self-supervised objectives for Contextual AS2

Recently, Lauriola et al., 2021 and Han et al. [2021] showed that context can
help re-ranking the best answer candidate in the top position. In Answer
Sentence Selection happens that many answers are not well-formed or contain
unresolved entities or relations. Additional context can help in solving this
issue by adding details about them. In this work, our goal is to design pre-
training objectives that help the model in understanding the role of the
additional input context. We extend the original architectures of recently
released Transformer models such as RoBERTa and DeBERTa to allow them to
tokenize and process 3 input sequences: the question, the answer candidate and
the context.

The objectives we propose are extensions of the SSP task described before. We
define 3 different methods to gather additional context that will be fed into
the model:

  * •

Static Document-level Context (SDC): in SDC, we select the first paragraph of
a document as the context. The first paragraph is usually a summary of the
document content and thus it can help in solving broken references;

  * •

Dynamic Paragraph-level Context (DPC): DPC sets the context equal to the text
that remains in the paragraph after the sampling of the two spans for the SSP
objective;

  * •

Paragraphs in the Same Document (PSD): PSD defines the context as the pair of
sentences surrounding the second span that is fed to the SSP objective, which
corresponds to the answer in the final task. Notice that this aligns well with
the downstream AS2 application because the context is related to the answer
and not to the question.

As before, we perform continuous pre-training starting from several state-of-
the-art pre-trained models and then we evaluate our proposals on many
Contextual Answer Sentence Selection datasets. We show that the accuracy of
our models is superior to the baselines on all the benchmarks we consider, and
we also reach the state-of-the-art on ASNQ when combining our objectives with
larger models, such as DeBERTaV3Large. We describe in depth these experiments
and results in Chapter 8.

#####  1.1.2.4 Self-supervised Objectives for Summarization

Finally, we show that our methodologies can be adapted successfully also to
generative tasks. In particular, we analyze specialized pre-training
objectives designed for Summarization, in which a model is provided with a
document and should generate a coherent and concise summary of the main
document’s points. We leverage the weak supervision of large corpora to design
an objective that accustoms the model for the Summarization task already while
pre-training.

More specifically, we propose Static Document-level Summary (SDS), an
objective in which the model receives as input all the paragraphs of a
document except the first one, and it must learn to generate the latter.
Notice that we chose the first paragraph of the document because it usually
acts as a summary of the whole document’s content.

We perform continuous pre-training starting from BART and T5 checkpoints for a
few hundred thousand training steps. Then, we fine-tune our models and the
baselines on several datasets for Summarization and we show how our
methodologies can help to improve performance in Summarization. Furthermore,
given the inherent subjectivity of evaluating summaries, we employ a Large
Language Model with over 40 billion parameters to generate additional silver
summaries. This approach allows for a more robust comparison between models
and helps address the challenges associated with subjective evaluations. The
full description of those techniques and the results are given in Chapter 9.

###  1.2 Implementation & Machines

We provide the details of the hardware setup used all the experiments of this
work:

  * •

CPU: 2 x Intel Xeon Platinum 8275CL @ 3.00GHz (48 cores/96 threads)

  * •

RAM: 1152 GiB DDR4

  * •

GPU: 8 x NVIDIA A100 GPU with 40GB HBM2 memory, connected with NVLink

  * •

Cost: 32$/h on AWS as of May 2023

Regarding the development of the code, we based our project on different
libraries designed for Machine Learning:

  * •

_PyTorch_ [Paszke et al., 2019], described by the authors as “Tensors and
Dynamic neural networks in Python with strong GPU acceleration”, is a deep
learning library that offers high-performance tensors manipulation and auto-
differentiation;

  * •

_HuggingFace Transformers & Datasets_ [Wolf et al., 2020; Lhoest et al.,
2021], a large collection of state-of-the-art architectures, models and
datasets for NLP and CV;

  * •

_PyTorch Lightning_ [Falcon et al., 2019], a framework that allows researchers
to abstract from the hardware and pleasantly train and test Neural Networks in
different settings;

  * •

_TorchMetrics_ [Detlefsen et al., 2022], a collection of metrics to evaluate
models performance.

The framework we developed that can be used to reproduce all the experiments
described in this work and the pre-trained checkpoints will be released here:
https://github.com/lucadiliello/transformers-framework. We train all
experiments with _fp16_ (AMP) or _bf16_ , DeepSpeed and FuseAdam (which
equivalent to the AdamW optimizer but more efficient) [Rasley et al., 2020],
which decrease GPU memory consumption while improving training speed.

###### Evaluation and Statistical Analysis

In the tables displaying results on the downstream datasets, we emphasize our
proposed models by highlighting them in bold. Moreover, we include the
standard deviation for each fine-tuning on the target datasets. Unless
specified otherwise, we perform 5 runs with different initialization seeds.
Additionally, for each block in Tables, we underline results that are
statistically superior to the corresponding baseline and emphasize the highest
overall scores using bold formatting. To determine if there is a significant
performance improvement, we conduct a statistical t-test with a significance
level $\alpha=0.05$.

Finally, notice that we utilize various downstream benchmarks, such as ASNQ
and WikiQA, across multiple Chapters of this work. As the experiments were
conducted at different times during my Ph.D. pursuit, the baseline results
improved over time due to advancements in training frameworks, hardware
technologies, and bug fixes. We do not retrain every model with each new
software version release, maintaining the original results, even if there are
occasional misalignments among different Chapters.

###  1.3 Publications

In this Section, we provide a list of the publications at international
conferences that led to the creation of this work:

  * •

Effective Pretraining Objectives for Transformer-based Autoencoders
Luca Di Liello, Matteo Gabburo, Alessandro Moschitti.
Findings at The 2022 Conference on Empirical Methods in Natural Language
Processing (EMNLP), 2022

  * •

Paragraph-based Transformer Pre-training for Multi-Sentence Inference
Luca Di Liello, Siddhant Garg, Luca Soldaini, Alessandro Moschitti.
North American Chapter of the Association for Computational Linguistics
(NAACL), 2022.

  * •

Pre-training Transformer Models with Sentence-Level Objectives for Answer
Sentence Selection
Luca Di Liello, Siddhant Garg, Luca Soldaini, Alessandro Moschitti.
The 2022 Conference on Empirical Methods in Natural Language Processing
(EMNLP), 2022

  * •

Context-Aware Transformer Pre-Training for Answer Sentence Selection
Luca Di Liello, Siddhant Garg, Alessandro Moschitti.
The 61st Annual Meeting of the Association for Computational Linguistics
(ACL), 2023

###  1.4 Structure of this Thesis

This Thesis begins with an introduction to Natural Language Processing and
Transformer-based models, which is provided in Section 2. We describe in
detail the branches of NLP, with a special focus on Question Answering and
Answer Sentence Selection. Then, we give an outline of the Transformer
architecture, from the tokenization algorithms and the embedding strategies to
the structure of the Attention Mechanism. Follows an overview of various
categories of Transformer models, such as Auto-Encoders and Auto-Regressive
architectures, and a description of the most used training strategies and
tasks.

Chapter 3 describes the works related to the discoveries presented in this
Thesis. We start by providing an overview of several recently released state-
of-the-art models, which are our starting point for many experiments. Then, we
describe the most relevant works related to the tasks we address in this
Thesis, such as Answer Sentence Selection and Summarization. Finally, we list
works related to the methodologies we developed in this publication, such as
multi-sentence inference models and specialized continuous pre-training over
weakly supervised data.

In Chapter 4, we describe every dataset used in this Thesis, providing details
about the data collection process, the number of examples and the annotation
procedure.

From Chapter 5 to 9, we elaborate on the main discoveries of this work. In
particular, in Chapter 5 we analyze alternative pre-training objectives that
reduce training time while increasing final accuracy on several downstream
tasks. Then, in Chapter 6 we propose an innovative Transformer architecture
that can process many input spans of text at a time while performing a
different prediction for each of them. We also design a new training task
that, when combined with the proposed architecture, provides benefits in
Answer Sentence Selection and Fact Verification settings. In Chapter 7, we
study different pre-training objectives to improve models that reason over 2
spans of text by exploiting the weak supervision of large corpora, leading to
improvements on various AS2 datasets. Follows Chapter 8, in which we extend
the previous idea by adding context to the equation. We design custom pre-
training objectives that teach the models at exploiting the context to better
re-rank answer candidates while fine-tuning. Finally, in Chapter 9, we show
that the weak supervision of unlabeled datasets can be exploited also for
tasks different from Question Answering or Fact Verification. We describe
self-supervised objectives to improve the performance of Auto-Regressive
models in Summarization.

We present the conclusions of this Thesis in Chapter 10, with an overview of
the methods used, the best results achieved and a list of the possible future
research directions.

##  Chapter 2 Background

In this Section, we start by providing an overview of the Natural Language
Processing tasks that we address in this work, with a special focus on
Question Answering, Fact Verification and Summarization. Then, we describe the
most renowned architectures used for NLP. In particular, we elaborate on the
Transformer, which is, at the time of writing, the state-of-the-art for almost
all NLP tasks. Finally, we provide an overview of the most common Transformer
models referenced in this work by describing their internal architecture, the
data and the objectives used for pre-training. The mathematical notation and
formulas in this Chapter follow the convention of indexing starting from 1.

###  2.1 Natural Language Processing

Natural Language Processing is the study of algorithms that can process human-
generated text. NLP is a mix of computer science and linguistics: while the
first field provides the techniques to solve problems such as architectures
design, data collection, data processing and models evaluation, linguistics
studies the nature and the structure of languages, their syntax, semantics,
morphology and their ambiguities [Wikipedia contributors, 2004].

The first forms of symbolic NLP were developed in 1950. They exploited a
series of rules to encode the structure of a language and allowed for
automatic Machine Translation. However, systems were very limited and
performance was far below expectations. Twenty years later, with the
development of the first large vocabularies and ontologies, scientists
designed the first chatbots that could extract information from knowledge
bases. A few years later, in the 80s, many algorithms for automatic parsing,
semantic [Lesk, 1986] and morphology extraction [Koskenniemi, 1984] were
further developed. However, the main drawback of symbolic NLP was the need of
writing complex rules by hand and for every language.

In the 90s, NLP researchers started to use statistical analysis to develop
better systems, in particular, using Machine Learning algorithms. ML finally
removed the need to manually write rules-based parsers for each language,
which was complex and expensive. Great improvements in performance, especially
in Machine Translation, were enabled by the growth of the World Wide Web,
starting from 1995, which allowed large quantities of raw data to be publicly
available. The possibility to use large-scale datasets of unlabeled text
pushed on the development of new unsupervised, self-supervised and semi-
supervised encoding algorithms, e.g. Word Embeddings [Vinokourov et al.,
2002].

Recently, from the 2010s and onwards, algorithms based on Artificial Neural
Networks (ANN) have become the standard in NLP. The practice is to encode a
sentence by first splitting it into a sequence of tokens, which may be words,
sub-words or even single characters. Then, tokens are encoded in a vector
space and further processed together with various Neural Networks
architectures. While before 2017 scientists exploited mainly Convolutional
Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) [dos Santos et
al., 2016], recently Transformers [Vaswani et al., 2017] reached state-of-the-
art performance in almost all NLP tasks [Devlin et al., 2019; Liu et al.,
2019b; Radford and Narasimhan, 2018].

In the next Section, we provide an overview of the NLP task we address in this
work. In particular, the focus is on Answer Sentence Selection, which is a
branch of Question Answering that is rapidly gaining popularity, especially
for virtual assistants such as Amazon Alexa or Google Home.

####  2.1.1 Question Answering

Question Answering (QA) is the task of automatically answer to questions posed
in natural language. A QA pipeline is usually composed of multiple modules
that deal with (i) the retrieval of data that contain relevant information and
(ii) the extraction or generation of the correct answer span that is returned
to the user.

The challenges in QA regard the organization of data for the retrieval phase,
the parsing of unstructured text, the processing and disambiguation of the
question and the elaboration of documents to extract the correct answer. For
example, data for the retrieval phase can be organized in ontologies, graphs
or simple lists of documents. While the first two methodologies allow fast
access to key information, they are hard to create and maintain. On the other
hand, a list of documents is easy to compose and update. However, information
is stored in an unstructured way and it is not trivial to efficiently search
for it.

In this Thesis, we focus mostly on Open Domain Question Answering [Voorhees
and Tice, 2000], which is a branch of QA where questions are not restricted to
a single domain but may potentially span over all human knowledge. In other
more concrete words, ODQA is the task of replying to questions for which an
answer may be found on the web. Open Domain QA defines new challenges because
the system does not know the domain of the question a priori. Common
techniques to solve the problem include powerful search mechanisms to retrieve
relevant information and Machine Learning models with high generalization
capabilities to extract or generate the final answer.

The following Sections describe in depth the components of a QA pipeline.
Figure 2.1 shows a high-level perspective of how data are retrieved and
processed to find the best answer. In particular, after the gathering of
relevant documents through some Information Retrieval system, 4 different
Machine Reading Comprehension approaches could be exploited to obtain the
final answer. MRC is an umbrella term for methods that, given a user query,
process documents and try to extract or generate the correct answer. The main
difference between Answer Sentence Selection and other techniques is that in
AS2, the retrieved documents are split into smaller entities (e.g. sentences),
which are processed separately, while Extractive, Abstractive and Generative
QA usually process the whole input document to extract or generated the target
answer. While the latter methods have potentially more context to find the
correct answer span, Answer Selection techniques can work on much strict
latency constraints because input textual sequences are shorter and can be
processed in parallel [Garg et al., 2019].

Figure 2.1: Example of the common structure of an ODQA pipeline. Source
documents are usually millions or billions of data points organized as graphs
or lists. The Retrieval block finds a subset of relevant documents (a few
hundred or thousands) related to the question and divides them into smaller
entities for easier processing. Finally, Machine Reading Comprehension
techniques are used to extract the correct answer span that is returned to the
user.

#####  2.1.1.1 Passage Retrieval

Passage Retrieval is a branch of Information Retrieval that studies how to
organize, maintain, update and search over large bases of data points. In
Question Answering, a passage retrieval system is given a query and should
rank the available documents based on their relevance. Documents may contain
structured or unstructured text, depending on the application. The comparison
of the query with each document to compute a relevance score can be performed
in different ways. Among the most common, we cite: (i) tokens match, in which
the score is computed as the degree of overlap between the query and the
document tokens; (ii) similarity score, in which both the query and the
document are encoded in a vector space and some similarity metric (e.g. cosine
similarity) is used to compute the relevance.

###### Token matching techniques

Among the token matching techniques, TF-IDF [Rajaraman and Ullman, 2011] and
BM25 [Robertson and Zaragoza, 2009] are two non-parametric approaches that
recently gained popularity thanks to their effectiveness and efficiency.

TF-IDF stands for Term Frequency-Inverse Document Frequency, and it works by
calculating the importance of a word in a document or corpus by multiplying
the frequency of the word (TF) by the inverse frequency of the word in the
corpus (IDF). Words that appear frequently in a document but not in the corpus
will have a high TF-IDF score and are considered important to the document. In
Passage Retrieval, the term frequency (TF) of the query term in the document
is multiplied by the inverse document frequency (IDF) of the query term in the
corpus. The scores are then summed across all query terms to give a final
score for each document. In practice, a map between terms and documents is
pre-computed to speed up ranking.

BM25 builds over TF-IDF to improve ranking quality by also taking into account
the frequency of query terms in the document, the length of the document, and
other factors such as the frequency of query terms in the whole collection of
documents.

###### Representation similarity techniques

Regarding the techniques based on representation similarity, we cite Dense
Passage Retriever (DPR) [Karpukhin et al., 2020], ColBERT [Khattab and
Zaharia, 2020] and QUestion-Answer Database Retrieval (QUADRo) [Campese et
al., 2023]. DPR uses two BERT [Devlin et al., 2019] models to encode both
query and each document to a real-valued vector space and applies cosine
similarity to compute relevance. The BERT models are fine-tuned to minimize
and maximize the encoded distance between positive and negative pairs
respectively. In ColBERT, the authors encode the query and the document with a
BERT model to obtain vectorial representations, similar to DPR. However, to
compute the similarity score, they exploit a cheap interaction module that
performs a more fine-grained comparison between the query and the document
embeddings, leading to better re-ranking abilities. Finally, QUADRo operates
on a database of previously answered question-answer pairs rather than a pool
of documents. The authors encode the input query and millions of question-
answer pairs, minimizing the distances between the embeddings when the query
and the questions refer to the same concept, improving the retrieval and re-
ranking accuracy of dense passage retrieval methods.

Common techniques to speed up lookup in Information Retrieval systems are
indexes and inverted indexes, which use hash functions to map similar
documents or parts of them to the same hash value. Examples of ready-to-be-
used retrieval systems are ElasticSearch111https://www.elastic.co and Facebook
FAISS [Johnson et al., 2019]. The first allows users to quickly get relevant
documents by searching into sharded and distributed indexes by text matching,
while the latter allows search by similarity thanks to the vectorial encodings
of queries and documents.

For Question Answering, knowledge can be stored in many different modalities
depending on the quantity of data and whether they are structured or not. More
details about the different data organization methods are given in Appendix
A.1.

#####  2.1.1.2 Answer Sentence Selection

Answer Sentence Selection (AS2) is a ranking task in which a system is
provided with a question and a set of answer candidates, and it should rank
the latter based on how likely they answer the given question.

More formally, the system is provided with a question $q$ and a set of
possible answer candidates ${\mathbb{A}}=\\{a_{1},\dots,a_{n}\\}$. The goal is
to select the $a_{k}$ that best answers $q$. The common practice is to train a
binary classifier $\mathcal{M}$ that is fed with pairs
$(q,a_{i}),i\in\\{1,{\dots},n\\}$ and learns answers correctness
$s_{i}=\mathcal{M}(q,a_{i})$ by predicting whether $a_{i}$ answers $q$ or not.
At inference time, the best answer to $q$ is chosen by selecting the
candidates $a_{k}$ that scores the highest probability $s_{k}$ of being
correct:

$k=\operatorname*{arg\,max}_{i}\mathcal{M}(q,a_{i})$ (2.1)

In AS2, there are 2 main common practices used to compute scores $s_{i}$ for
each answer $a_{i}$ given question $q$: Bi-Encoders and Cross-Encoders. Bi-
Encoders are systems in which the question $q$ and the answer candidate
$a_{i}$ are encoded separately, using two different networks $\mathcal{M}_{1}$
and $\mathcal{M}_{2}$. If the networks are the same and share the same
internal parameters ($\mathcal{M}_{1}=\mathcal{M}_{2}$), we are referring to a
“Siamese Networks” system. Bi-Encoders create two embeddings
$e_{q}=\mathcal{M}_{1}(q)$ and $e_{a_{i}}=\mathcal{M}_{2}(a_{i})$ that are
independent from each others because no information from $q$ is used to
compute $e_{a_{i}}$ and vice-versa. The main advantage of Bi-Encoders is that
every answer candidate embedding can be computed efficiently in advance, even
before knowing the question. The final relevance score to determine whether
$a_{i}$ is a correct answer to $q$ is typically computed with a similarity
metric, such as cosine similarity:

$s_{k}=\frac{e_{q}\cdot e_{a_{i}}}{||e_{q}||\cdot||e_{a_{i}}||}$ (2.2)

A common example of siamese networks used for Bi-Encoders is represented by
Sentence-BERT [Reimers and Gurevych, 2019].

On the other hand, in the Cross-Encoder setting, a model $\mathcal{M}$
computes a joint representation that depends both on the question $q$ and the
answer candidate $a_{i}$:

$s_{k}=\mathcal{M}(q,a_{i})$ (2.3)

In this Thesis, we focus on Cross-Encoders because they provide the highest
level of performance even though they have slightly higher computational
requirements in practice. The higher computational cost arises from the
encoding of a longer input sequence, which usually is the concatenation of the
question and the answer candidate.

The evaluation of AS2 models requires metrics that measure the score quality
compared to an ideal ranking of the answers. In an ideal ranking, every
positive candidate has a higher score than all the negatives. Common metrics
for AS2 include Mean Average Precision (MAP), Mean Reciprocal Rank (MRR) and
Precision@K (P@K). MAP is a soft metric useful to evaluate the whole ranking
quality because it measures the Precision at different positions. MRR and P@1
are hard metrics that measure the position of the first correct answer
retrieved. In particular, Precision@1 is the fraction of times the model was
able to score a correct answer at the top of the ranking.

###### Contextual Answer Sentence Selection

Recent studies [Lauriola and Moschitti, 2021; Han et al., 2021] showed that
processing every answer candidate separately can lead to suboptimal results
because local or global context helps in classifying answer candidates,
especially when they are malformed or missing information or when they contain
ambiguities or unresolved references to external entities. Contextual AS2 is
an extension of AS2 in which the inputs of the language model are augmented
with additional context to better disambiguate between answer candidates. More
formally, in Contextual AS2 a model $\mathcal{M}$ receives in input a tuple
$(q,a_{i},c_{i})$ where $q$ and $a_{i}$ are the original question and the
answer candidate while $c_{i}$ is the additional context to be processed. As
before, the model should predict a ranking score
$s_{i}=\mathcal{M}(q,a_{i},c_{i})$. A typical approach to get the context is
to extract the sentences before and after the answer candidate in the original
document (local context) or to take the first paragraph of the same document
(global context).

#####  2.1.1.3 Machine Reading Comprehension

Machine Reading Comprehension (MRC) is a task in NLP that involves building
systems that can understand and reason about text in a similar way as humans
do. This involves understanding the meaning of words and phrases, as well as
the context in which they are used, and using this understanding to answer
questions or make decisions based on the text.

There are several approaches to MRC, including rule-based, Information
Retrieval-based, and Machine Learning-based systems. Machine Learning-based
systems are the most common, and often use deep learning techniques such as
neural networks to learn how to understand and reason about text. MRC has a
wide range of applications, including Information Retrieval, Question
Answering, Summarization, and dialogue systems and is an active area of
research.

In the context of Question Answering, the term MRC typically refers to
techniques in which the whole document is processed to find the target answer,
such as Extractive or Generative Question Answering. While those techniques
parse entire documents in a single step and have visibility over a lot of
contexts, the elaboration of long sequences is expensive and results in higher
latencies.

The three main approaches used to build a Machine Reading Comprehension
pipeline are summarized as follows (more details are provided in Appendix
A.4):

###### Extractive Question Answering

Extractive QA involves the identification and selection of relevant
information from a given text to directly answer a question. It relies on
algorithms that analyze the text, identify key sentences or phrases, and
choose the most appropriate portion as the answer. Extractive QA models do not
generate new content but rather extract existing information.

###### Generative Question Answering

Generative QA entails the generation of a response to a question from scratch.
It utilizes advanced language models, such as Recurrent Neural Networks or
Transformer models, to understand the question and generate a coherent and
contextually relevant answer.

###### Abstractive Question Answering

Abstractive QA goes beyond simple answer generation by summarizing relevant
information and generating a concise response. It involves understanding the
question and the source text, identifying key points, and generating a summary
that captures the essence of the information.

####  2.1.2 Fact Verification

Fact Verification is a task in NLP that requires verifying the accuracy of a
statement or a claim [Zhou et al., 2019; Liu et al., 2020; Zhong et al.,
2020]. This is accomplished by checking the sources of the information,
comparing them, and evaluating their credibility. In particular, Fact
Verification consists of 2 challenges: (i) the retrieval of relevant documents
that may contain textual evidence for the claim and (ii) the classification of
the claim as supported/refuted/neutral compared to the retrieved evidence.

The first part, which is an Information Retrieval task, is usually performed
with reliable search engines such as ElasticSearch or FAISS, as described in
Section 2.1.1.1. In this work, we address the second task, in which a model is
provided with a claim and a set of evidence texts and should classify whether
the claim is supported or not.

More formally, given a claim $c$ and the set of evidence text spans
$E=\\{e_{1},\dots,e_{n}\\}$ that were retrieved using some IR technique, the
objective is to predict whether $c$ is supported/refuted/neutral using $E$.
Specifically, to classify $c$, the model should find at least one evidence
$c_{i}$ supporting/refuting $c$.

Fact verification is an important task in NLP because it helps to ensure that
information is accurate and reliable, and it helps to prevent the spread of
misinformation. It is also an area of active research and development, as new
techniques and technologies are developed to improve the efficiency and
effectiveness of Fact Verification processes.

####  2.1.3 Summarization

In NLP, Summarization is the task that involves generating concise and
coherent summaries that capture the key information from longer documents. The
goal is to condense the content while retaining its essential meaning. There
two main branches are Extractive Summarization and Abstractive Summarization
[Torres-Moreno, 2014].

###### Extractive Summarization

In Extractive Summarization, the task is to identify and select the most
important sentences or phrases from the original text and arrange them to form
a summary. The selected sentences are typically extracted verbatim from the
source document without any modification. Extractive methods rely on
techniques such as sentence ranking, keyword extraction, and clustering to
determine the most salient information. This technique is often the easiest to
implement and can preserve the factual accuracy of the original text.

###### Abstractive Summarization

In this methodology, the model should generate a summary that may contain new
phrases, rephrased sentences, or paraphrased content that captures the meaning
of the original text. It involves understanding the source document and
generating concise and coherent summaries in a more human-like manner. This
task is usually performed by Generative Models, which are given the document
in input and generate the summary one token at a time. Abstractive
Summarization allows for more flexibility and creativity in generating
summaries but can be more challenging due to the need for advanced language
understanding and generation capabilities.

###  2.2 Language Models

Language models in NLP are employed as statistical estimators to predict the
likelihood of token sequences, assigning probabilities
$P(w_{1},w_{2},\dots,w_{k})$. These models find applications in tasks like
Machine Translation, Text Classification, and Text Generation, enabling the
classification or generation of human-like text.

Language Models can be constructed using various approaches, each offering its
own set of advantages and drawbacks. One straightforward method involves using
$n$-gram models [Harris, 1954], which estimate the probability of a word given
the preceding $n-1$ words in the sequence. For instance, a bigram model
($n=2$) estimates the probability of a word based solely on the preceding
word, while a trigram model ($n=3$) takes into account the probabilities of
the two preceding words. While $n$-gram models are relatively simple to
implement and can perform well with small datasets, they may exhibit decreased
accuracy compared to other approaches when dealing with large datasets or long
sequences of text.

An alternative approach involves neural language models, which rely on
Artificial Neural Networks (ANNs) and are trained on extensive text datasets.
Neural LMs can learn statistical patterns in the data and generate more
natural-sounding text compared to $n$-gram models. Additionally, they can
handle longer sequences and accommodate a larger vocabulary size. Nonetheless,
they may necessitate more computational resources for training and may require
more data to achieve satisfactory results.

For a long time, Recurrent Neural Networks [Elman, 1990] and Convolutional
Neural Networks [Lecun et al., 1998] have been the state-of-the-art for a
large variety of NLP tasks. Then, in 2017, Vaswani et al. [2017] proposed the
Transformer architecture that quickly became the standard de facto for most
NLP tasks such as Machine Translation, Summarization, Question Answering and
Natural Language Inference. For a detailed description of RNNs and CNNs, see
Appendix A.2

This Section proceeds by describing extensively the Transformer, which is the
building block of all the experiments reported in this Thesis. In Appendix
A.3.5, we provide an overview of the loss function, the back-propagation and
the various optimization algorithms commonly used to train Transformer models.

####  2.2.1 Transformers

The Transformer [Vaswani et al., 2017] is a type of Feed-Forward ANN based on
the Attention Mechanism, which allows every token to _attend_ to every other
token in the text. In particular, the input sequence is processed by a series
of Self-Attention layers, which allow the model to directly incorporate
information from the entire input sequence at each position. The Transformer
also introduces the concept of Multi-Head Attention, which is the usage of
multiple Self-Attention layers in parallel, each with its own set of
parameters. This allows the model to attend to different parts of the input
sequence simultaneously and to learn more complex relationships between the
input tokens.

In more detail, the initial steps to process a sequence of words with a
Transformer model involve tokenization and embedding. Tokenization is employed
to segment the input text into a sequence of tokens defined in a vocabulary,
such as words, sub-words, or characters. Embedding is instead a function that
maps these tokens into fixed-size vector representations. Subsequently, the
embedded sentence is fed into the Transformer model, which consists of a stack
of one or more Transformer blocks. The output of each block, ranging from the
first to the last, contains increasingly contextualized representations of
each token. Lastly, the token representations outputted by the last layer are
passed through a small neural network, typically referred to as a
Classification Head, to obtain predictions over a set of predefined labels or
to perform regression.

#####  2.2.1.1 Tokenization

The tokenization is the process of splitting the input text into a sequence of
tokens belonging to a fixed-size vocabulary ${\mathbb{V}}$. Usually, a
tokenization algorithm defines both the instructions to create the vocabulary
${\mathbb{V}}$ from raw text and the procedure to map the text into a sequence
of tokens belonging to ${\mathbb{V}}$. For example, a simple tokenizer may
divide the input text into individual words by splitting over whitespaces. It
may happen that some input text substrings cannot be tokenized using only the
tokens in ${\mathbb{V}}$: in those cases, an OOV (out of vocabulary) token is
assigned.

In Appendix A.3, we cover the four main tokenization algorithms used by recent
LMs, which are Byte-Pair Encoding (BPE) [Sennrich et al., 2016], WordPiece
[Schuster and Nakajima, 2012], UnigramLM [Kudo, 2018] and Sentence-Piece [Kudo
and Richardson, 2018].

#####  2.2.1.2 Word Embeddings

Computers are inherently designed to process numerical data rather than
strings. Therefore, every token generated by the tokenizer must be mapped to a
vector space before being input to the LM. To achieve this, an embedding layer
is utilized to convert categorical variables, such as tokens or integers, into
continuous vector representations.

A trivial embedding layer is called _one-hot_ encoding, in which every input
token is mapped to a binary vector. The binary vector corresponding to a token
$t_{i}$ has the same length of the vocabulary ${\mathbb{V}}$ and contains all
0s but from the position $i$ corresponding to the token $t_{i}$, which is set
to 1. For example, given a vocabulary ${\mathbb{V}}=\\{t_{1},\dots,t_{5}\\}$,
the one-hot encoding of $t_{3}$ is:

$\text{one\\_hot}(t_{3})=[0,0,1,0,0]$ (2.4)

The primary issue with one-hot encoding lies in the data sparsity and the
encoding size, which corresponds to the size of the vocabulary
$|{\mathbb{V}}|$, typically ranging from 30K to 120K and beyond in Transformer
models [Devlin et al., 2019; He et al., 2021].

A more sophisticated embedding layer should satisfy the following properties:
(i) create a dense representation of tokens and (ii) map related tokens to
close positions. Embedding layers are typically implemented as a matrix of
weights ${\bm{E}}\in{\mathbb{R}}^{|{\mathbb{V}}|\times d}$, where each row
${\bm{\mathrm{e}}}_{i}$ corresponds to the representation of token $t_{i}$.
While training, the embedding layer takes as input a token and returns the
corresponding row from the matrix as the embedding vector.

While in some architecture the embedding matrix ${\bm{E}}$ was learned before
training, for example with Gensim [Rehurek and Sojka, 2011] or word2vec
[Mikolov et al., 2013], Transformer models’ embedding layers are learned
during the training process of the model, allowing to extract the most
relevant information directly from the input data. Embedding layers are useful
for dealing with the large vocabularies of natural languages, allowing them to
represent each word with a fixed-size vector, which can be easily processed by
the Transformer. They also help to overcome the problem of sparsity in one-hot
encoded representation and can effectively capture semantic and syntactic
information of words.

#####  2.2.1.3 Positional Embeddings

The Transformer architecture, which we will deepen in Section 2.2.1.4, is
based on the Attention Mechanism and processes every input token
representation at the same time, regardless of the position in the input text.
This is different from recurrent architectures such as GRUs [Cho et al., 2014]
and LSTMs [Hochreiter and Schmidhuber, 1997], which intrinsically add
positional information by processing sequentially one token at a time.

For this reason, Transformer models need an additional technique to inject
positional information into the network. The original Transformer
implementation used cosine positional embeddings [Vaswani et al., 2017], in
which every positional embedding ${\bm{\mathrm{e}}}_{i}\in{\mathbb{R}}^{d}$ to
token $t_{i}$ is defined as:

${\bm{\mathrm{e}}}_{i,j}=\begin{cases}\sin\frac{i}{10000^{2j/d}}&\text{if $j$
is even}\\\ \cos\frac{i}{10000^{2j/d}}&\text{if $j$ is odd}\\\
\end{cases}\vspace{0.5em}$ (2.5)

However, recent models use either learnable absolute or relative positional
embeddings.

###### Learnable Absolute Positional Embeddings

Learnable absolute positional embeddings are similar to Word Embeddings, in
which they are composed of a matrix ${\bm{P}}\in{\mathbb{R}}^{s\times d}$,
where each row ${\bm{\mathrm{p}}}_{i}$ corresponds to the unique
representation of a position $i\in\\{1,\dots,L\\}$ in the continuous vector
space ${\mathbb{R}}^{d}$. The main advantage is that learnable positional
embeddings are easy to implement and work well. However, the sequence length
$L$ must be fixed a priori. Finally, the embedding ${\bm{\mathrm{h}}}_{i}$ of
each token $t_{i}$ is computed as the sum of the word and the positional
embedding:

${\bm{\mathrm{h}}}_{i}=\text{embedding}(t_{i})={\bm{\mathrm{e}}}_{i}+{\bm{\mathrm{p}}}_{i}$
(2.6)

We indicate with ${\bm{H}}\in{\mathbb{R}}^{L\times d}$ the embedding matrix of
size $d$ of a sequence of $L$ tokens that is provided in input to the
Transformers architecture. Learnable absolute positional embeddings are used
by many well-known models such as BERT [Devlin et al., 2019], RoBERTa [Liu et
al., 2019b], ALBERT [Lan et al., 2020], ELECTRA [Clark et al., 2020], DeBERTa
[He et al., 2020] and BART [Lewis et al., 2020].

###### Relative Positional Embeddings

Relative positional embeddings [Shaw et al., 2018] are fixed values that are
injected directly into the attention scores matrix to add positional
information. The Attention Mechanism, described in Section 2.2.1.4, computes a
score for every input tokens pair to estimate the relation between a token and
all the others. This information is stored in a matrix
${\bm{A}}\in{\mathbb{R}}^{L\times L}$ where $L$ is the input sequence length.
Relative embeddings are contained in a matrix
${\bm{P}}\in{\mathbb{R}}^{L\times L}$ that is added to ${\bm{A}}$. Every
position $i,j$ in ${\bm{P}}$ contains a value that represents the distance
between token $t_{i}$ and $t_{j}$. The actual computation of ${\bm{P}}$ varies
across models and architectures. Relative positional embeddings are used by
models such as T5 [Raffel et al., 2020], Transformer-XL [Dai et al., 2019] and
XLNet [Yang et al., 2019].

#####  2.2.1.4 Transformer block

The original Transformer block is composed of 2 or 3 sub-layers, based on
whether it is used as an encoder or as a decoder [Vaswani et al., 2017]. At
the core of the Transformer architecture, there is the Attention Mechanism,
which is used to modify the representation of every token based on all the
other tokens in the input sequence.

###### Attention Mechanism

The Attention Mechanism is an architecture used in neural networks to allow a
model to focus on a specific part of the input when processing it by comparing
every input token representation with the others. In recent years, several
Attention Mechanisms such as Additive Attention [Bahdanau et al., 2015] and
Multiplicative Attention [Luong et al., 2015] have been developed. The
Transformer uses a modified version of the Multiplicative Attention, called
Scaled Dot-Product Attention [Vaswani et al., 2017].

The Scaled Dot-Product Attention works as follows. First, each input hidden
state ${\bm{\mathrm{h}}}_{i}$ containing the representation in $d$ dimensions
of $t_{i}$ is projected from ${\mathbb{R}}^{d}$ to ${\mathbb{R}}^{d}$ using
three different learnable matrices of weights ${\bm{W}}_{q}$,
${\bm{W}}_{k},{\bm{W}}_{v}\in{\mathbb{R}}^{d\times d}$. More formally:

$\begin{split}{\bm{\mathrm{q}}}_{i}=&{\bm{W}}_{q}{\bm{\mathrm{h}}}_{i}\\\
{\bm{\mathrm{k}}}_{i}=&{\bm{W}}_{k}{\bm{\mathrm{h}}}_{i}\\\
{\bm{\mathrm{v}}}_{i}=&{\bm{W}}_{v}{\bm{\mathrm{h}}}_{i}\\\ \end{split}$ (2.7)

Then, projections are then packed into three matrices ${\bm{Q}}$, ${\bm{K}}$
and ${\bm{V}}$, called respectively _Queries_ , _Keys_ and _Values_ , and the
new hidden states ${\bm{H}}^{\prime}$ are computed with the Attention
Mechanism as follows:

${\bm{H}}^{\prime}=\text{Attention}({\bm{Q}},{\bm{K}},{\bm{V}})=\text{softmax}\left(\frac{{\bm{Q}}{\bm{K}}^{T}}{\sqrt{d}}\right){\bm{V}}\vspace{0.2em}$
(2.8)

Notice that ${\bm{A}}={\bm{Q}}{\bm{K}}^{T}\in{\mathbb{R}}^{L\times L}$
contains the attention scores and for every pair of tokens $t_{i},t_{j}$,
${\bm{A}}_{i,j}$ can be interpreted as the similarity between the two. The
scaling factor $\sqrt{d}$ is used to smooth the _softmax_ function and
stabilize gradient back-propagation by avoiding very large attention scores.
Finally, the multiplication of the softmaxed attention scores with ${\bm{V}}$
allows to influence every token representation with the attention scores, thus
creating a more contextualized hidden state ${\bm{H}}^{\prime}$.

###### Multi-Head Attention

In multi-head attention, the input hidden state is split into multiple heads
or subsets, and attention is performed separately on each head. Each head
performs a linear transformation on the input vectors to project them into a
new space, and then applies the Attention Mechanism to the transformed
vectors. The results of each head are then concatenated and projected back
into the original input space. More specifically, given the number of
attention heads $n$, the Queries, Keys and Values are computed with:

$\begin{split}{\bm{\mathrm{q}}}^{a}_{i}=&{\bm{W}}^{a}_{q}{\bm{\mathrm{h}}}_{i}\\\
{\bm{\mathrm{k}}}^{a}_{i}=&{\bm{W}}^{a}_{k}{\bm{\mathrm{h}}}_{i}\\\
{\bm{\mathrm{v}}}^{a}_{i}=&{\bm{W}}^{a}_{v}{\bm{\mathrm{h}}}_{i}\\\
\end{split}$ (2.9)

with $a\in\\{1,\dots,n\\}$ and ${\bm{W}}^{a}_{q}$, ${\bm{W}}^{a}_{k}$ and
${\bm{W}}^{a}_{v}\in{\mathbb{R}}^{d\times\frac{d}{n}}$. Then, attention is
computed as before for each triple $({\bm{Q}}^{a},{\bm{K}}^{a},{\bm{V}}^{a})$,
yielding to a set of new hidden states $\\{{{\bm{H}}^{\prime}}^{a}\\}$. The
final hidden state is then computed as a linear projection of the
concatenation along dimension $d$ of $\\{{{\bm{H}}^{\prime}}^{a}\\}$. More
specifically,
${\bm{H}}^{\prime}=\text{concat}({{\bm{H}}^{\prime}}^{1},\dots,{{\bm{H}}^{\prime}}^{n}){\bm{W}}_{0}$,
with ${\bm{W}}_{0}\in{\mathbb{R}}^{d\times d}$.

The advantage of multi-head attention is that it allows the model to capture
different patterns and features from the input sequence, and to learn more
complex relationships between the input and output. This can lead to improved
performance on several tasks such as Machine Translation, Language Modeling,
and Text Classification.

###### Feed-Forward Layer

The Feed-Forward layer is a sequence of two affine transformations that are
applied to each ${\bm{\mathrm{h}}}_{i}$ independently. The transformation
matrices are ${\bm{W}}_{1},{\bm{W}}_{2}\in{\mathbb{R}}^{d\times f}$ while the
biases are ${\bm{\mathrm{b}}}_{1}\in{\mathbb{R}}^{f}$ and
${\bm{\mathrm{b}}}_{2}\in{\mathbb{R}}^{d}$. The computation is performed with:

${\bm{\mathrm{h}}}_{i}^{\prime\prime}={\bm{W}}_{2}\max(0,{\bm{W}}^{T}_{1}{\bm{\mathrm{h}}}^{\prime}_{i}+{\bm{\mathrm{b}}}_{1})+{\bm{\mathrm{b}}}_{2}$
(2.10)

In common Transformer models such as BERT [Devlin et al., 2019] or RoBERTa
[Liu et al., 2019b], $f$ is called _intermediate_size_. The activation
function is not always a _relu_ function as in Equation 2.10, but can also be
_gelu_ or other custom activations depending on the model architecture.

###### Encoder

The Transformer block used in encoder layers is composed of an _Self-
Attention_ and a Feed-Forward layer. Self-Attention means that the Attention
Mechanism uses the same hidden state ${\bm{H}}$ to compute the projections
${\bm{Q}}$, ${\bm{K}}$ and ${\bm{V}}$.

###### Decoder

A decoder block is similar to the encoder block, with the addition of an extra
attention layer that enables the incorporation of information from the encoder
module. This additional block is called _Cross-Attention_. The output of the
encoder is used as ${\bm{K}}$ and ${\bm{V}}$ of every Cross-Attention layer in
the decoder, such that the hidden state is heavily influenced by the
information in output from the encoder.

###### Classification Head

The Classification Head is a module used to predict a label $y$ from a set of
possible labels ${\mathbb{C}}$. This is usually accomplished by putting a
linear layer on top of the output representation of the last Transformer Block
[Devlin et al., 2019; Lan et al., 2020]. More complex classification heads
composed of multiple linear and activation functions are used in models such
as RoBERTa [Liu et al., 2019b] and BART [Lewis et al., 2020].

#####  2.2.1.5 Transformer Models

(a) Encoder-only

(b) Encoder-Decoder

(c) Decoder-only

Figure 2.2: The three main Transformer configurations. $N$ is the number of
stacked layers in the encoder or the decoder.

Transformer-based can be categorized into three different classes based on
their internal architectures:

  * •

Encoder-only: the model is a stack of encoder blocks, such as ELECTRA [Clark
et al., 2020], RoBERTa [Liu et al., 2019b] or ALBERT [Lan et al., 2020]. These
models are usually trained with self-supervised objectives like BERT’s Masked
Language Modeling (MLM) or ELECTRA’s Token Detection (TD). The structure of an
Encoder-only model is shown in Figure 2.2(a).

  * •

Encoder-Decoder: the model comprises two sets of blocks: the encoder and the
decoder. The encoder blocks take the input and generate a representation of
it, while the decoder blocks generate a response for the user. For instance,
if the input is a question, the encoder would process it, and the decoder
would generate the answer. Similarly, in Summarization, if the input is a
document, the encoder would create its representation, and the decoder would
generate a summary. Examples of Encoder-Decoder models are BART [Lewis et al.,
2020] and T5 [Raffel et al., 2020]. Those models are usually pre-trained with
objectives that challenge the model at reconstructing, from the decoder
output, some corrupted input provided to the encoder. Figure 2.2(b) shows the
architecture of an Encoder-Decoder model.

  * •

Decoder-only: this class comprehends models composed only of decoder layers,
such as the GPT family [Radford and Narasimhan, 2018; Radford et al., 2019;
Brown et al., 2020], OPT [Zhang et al., 2022a] and Bloom [Scao et al., 2022].
Those models are usually pre-trained with GPT’s Causal Language Modeling (CLM)
objective, which consists in predicting the next token in a sequence. Figure
2.2(c) shows an example of Decoder-only architecture.

Transformers can also be divided between Auto-Encoder and Auto-Regressive
models based on the pre-training objective and the approach used for pre-
training:

  * •

Auto-Encoders: this class of Transformer models is well suited for tasks such
as Sequence or Token Classification because each token in the sequence can
attend to all the others. These models feature bi-directional attention and
can create good vectorial text representations. For example, when trained at
predicting some masked token $t_{i}$, the information from all the other
tokens in the sequence $\\{t_{j}\\}_{j\neq i}$ can be exploited. Auto-Encoders
perform well on classification tasks in which the model should incorporate
information from the whole input sequence to predict the target label.

  * •

Auto-Regressive: those models only use the tokens preceding the target token
$t_{i}$ as a source of information. In other words, they are constrained to
use only the tokens $\\{t_{j}\\}_{j<i}$ for predicting $t_{i}$. Auto-
Regressive models use masked attention layers to prevent the model from
exploiting information from future tokens. Given the nature of Auto-Regressive
Transformers, those models perform well in generative tasks, such as
Summarization and Generative Question Answering, by predicting a token at a
time.

Table 2.1 shows a summary of the architecture and type of well-known
Transformer models. Most Encoder-only models are Auto-Encoders while Decoder-
only models are primarily Auto-Regressive. Mixed models such as T5 and BART
use an Auto-Encoder to create a representation of some input text and an Auto-
Regressive decoder to output a response for the user.

Type | Encoder-only | Encoder-Decoder | Decoder-only
---|---|---|---
Auto-Encoder |  | BERT, RoBERTa, ALBERT,
---
DeBERTa, ELECTRA
|
Mixed | XLNet |  | T5, PEGASUS,
---
BART, BigBird
Auto-Regressive |  |  |  | GPT, Falcon, Bloom,
---
OPT, Transformer XL

Table 2.1: Configuration of the most renowned Transformer models.

####  2.2.2 Pre-Training with objectives

The pre-training phase of Transformers is a crucial step for accustoming
models to understand and generate text. It involves training on a large corpus
of text data to learn the statistical patterns and relationships present in
natural language. During pre-training, the model is exposed to a vast amount
of raw text, such as books, articles, and web pages. The text is typically
broken down into smaller units (see Section 2.2.1.1), which allows the model
to process and analyze the data more effectively.

Transformers utilize a self-supervised learning approach in pre-training,
which means that the training data itself provides the supervision signal for
learning. Based on the pre-training objective, such as Masked Language
Modeling (MLM) [Devlin et al., 2019], Causal Language Modeling (CLM) [Radford
and Narasimhan, 2018] or Token Detection (TD) [Clark et al., 2020], the model
is tasked at reconstructing the input which is corrupted in several ways. For
example, in MLM some of the tokens are randomly replaced with a special [MASK]
and the model’s task is to predict the original value given the surrounding
context. In CLM, the model should instead predict the next token in a sequence
given all the preceding. By training the model to reconstruct corrupted text,
it learns to capture the syntactic and semantic structures of the language.

The corpora used in pre-training consist of large amounts of raw data crawled
from the web. Examples are Wikipedia, the BookCorpus and the CommonCrawl.

Pre-training continues for several epochs or iterations until the model
achieves a satisfactory level of language understanding and predictions
accuracy. Once the pre-training phase is complete, the model can proceed to
the fine-tuning stage, where it is further adapted to specific downstream
tasks. In the following Sections, we provide an overview of the most cited and
used pre-training objective in the literature.

#####  2.2.2.1 Token-Level Pre-Training Objectives

Token-level objectives challenge the model to predict missing, replaced or
corrupted tokens by exploiting the knowledge of the surrounding context. The
most effective token-level pre-training objectives in the literature are
Masked Language Modeling, Token Detection, Causal Language Modeling and
Denoising.

###### Masked Language Modeling

Masked Language Modeling works by randomly replacing some of the input tokens,
i.e. 15%, with a special mask token, called [MASK]. The model is then tasked
to reproduce the original tokens using the information from the surrounding
context. This is usually also referred as the _cloze_ task. MLM for pre-
training language models was firstly proposed by BERT [Devlin et al., 2019]
and later was heavily utilized by many LMs such as RoBERTa [Liu et al.,
2019b], ALBERT [Lan et al., 2020] and XLM [Lample and Conneau, 2019]. MLM
requires bi-directional attention, thus it targets mainly Auto-Encoder models.
The main drawbacks of MLM are that: (i) original tokens are predicted assuming
they are independent of each other; (ii) there is a pre-training/fine-tuning
discrepancy because the [MASK] token appears only while pre-training and not
while fine-tuning; (iii) is not suited for text generation because tokens can
attend to the right.

###### Token Detection

Token Detection, which was initially proposed by ELECTRA [Clark et al., 2020],
consists in challenging the model at predicting whether input tokens are
originals or fakes created by another small Auto-Encoder network trained with
MLM. First, a small Auto-Encoder model trained with MLM (the _generator_) is
used to replace some masked tokens with plausible replacements. Those new
tokens are inserted in the original sequence in place of the [MASK] tokens.
Then, a large discriminative model (the _discriminator_) should classify every
input token and tell whether it is original or a replacement created by the
generator. After pre-training, the generator is discarded and the
discriminator is fine-tuned on the downstream task. The advantages of Token
Detection are that: (i) there is no discrepancy between pre-training and fine-
tuning because the discriminator is never fed with the [MASK] token; (ii) the
whole output of the discriminator is used to compute the loss, thus the
training is faster because of the more effective error back-propagation. TD
has been used to pre-train Auto-Encoder models such as ELECTRA, XLM-E [Chi et
al., 2022] and DeBERTaV3 [He et al., 2021].

###### Causal Language Modeling

Causal Language Modeling is a simple pre-training objective in which the task
is to predict the next token in a sequence. CLM has been extensively used for
pre-training Auto-Regressive models such as all the GPT architectures [Radford
and Narasimhan, 2018; Radford et al., 2019; Brown et al., 2020], OPT [Zhang et
al., 2022a] and Bloom [Scao et al., 2022]. The main drawbacks of CLM are that
in inference, predicting one token at a time is expensive, even if the user
applies caching techniques for previously computed ${\bm{K}}$ and ${\bm{V}}$
matrices.

###### Denoising

Denoising is a class of objectives mostly used to train Encoder-Decoder models
such as T5 [Raffel et al., 2020] and BART [Lewis et al., 2020]. The task
consists in corrupting the input text by removing a subset of tokens (and
possibly replacing them with a [MASK] token) before feeding the encoder. Then,
the decoder is tasked with generating the original sequence (BART) or just the
missing spans (T5). This task helps the model in learning to handle noisy and
incomplete input data and in generating coherent output sequences.

#####  2.2.2.2 Sentence-Level Pre-Training Objectives

Sentence-level pre-training objectives teach the model to reason at the
sentence level and not only on individual tokens. Follows a list of sentence-
level objectives commonly used in pre-training of Transformers, usually along
with some other token-level task.

###### Next Sentence Prediction

NSP was initially proposed by the authors of BERT [Devlin et al., 2019] and
consists in providing the model with two consecutive spans of text $s_{1}$ and
$s_{2}$ and tasking the model at predicting whether $s_{2}$ appeared after
$s_{1}$ in the original document. While this objective was shown to improve
results on some Pairwise tasks such as Natural Language Inference, Liu et al.
[2019b] showed that in longer pre-trainings, the effect of the NSP objective
is negligible. The main reason is that in long pre-trainings, NSP becomes a
trivial task that can be solved by checking the main topic of $s_{1}$ and
$s_{2}$, since negatives are created by sampling both spans from different
documents. Thus, the signal that is back-propagated through the model is very
weak.

###### Sentence Order Prediction

SOP is a harder alternative to NSP that was proposed by Lan et al. [2020]. In
SOP, the model is fed with two spans of text $s_{1}$ and $s_{2}$ and should
predict their order in the original document. Even if SOP is harder than NSP,
large LMs achieve very high accuracy on this task, thus more challenging
sentence-level objectives should be researched.

###### Sentence Structural Objective

SSO was first proposed by Wang et al. [2019b] and improves over NSP to create
a more challenging objective. In SSO, the model is given two spans $s_{1}$ and
$s_{2}$ and should perform a three-way classification to predict whether: (i)
the spans were sampled from different documents; (ii) $s_{1}$ precedes $s_{2}$
in the original document, (iii) $s_{1}$ follows $s_{2}$ in the original
document.

###### Sentences Shuffling

Another sentence-level objective is Sentence Shuffling [Lewis et al., 2020],
which consists in shuffling the input sentences before feeding them to the
encoder and tasking the decoder at reconstructing the original text.

####  2.2.3 Fine-Tuning on downstream tasks

Fine-tuning is the task of adapting a pre-trained model to perform well on
some specific task, such as Natural Language Inference (NLI), Question
Answering or Fact Verification.

For sentence-level classification tasks such as Natural Language Inference or
Answer Sentence Selection, predictions are performed by plugging a small
classification head on top of the Transformer model. For example, BERT [Devlin
et al., 2019] uses a linear layer
${\bm{L}}\in{\mathbb{R}}^{h\times|{\mathbb{C}}|}$ applied to the output
embedding of the first token to predict a label $y\in{\mathbb{C}}$, where
${\mathbb{C}}$ is the set of possible label classes. RoBERTa [Liu et al.,
2019b] takes instead the average of all token output embeddings before
performing the classification.

Regarding the token-level tasks such as POS Tagging, Named Entity Recognition
or Extractive Question Answering, a small classification head similar to that
described above is applied to all output token embeddings independently.

Finally, _Teacher-Forcing_ is used for the fine-tuning of generative
architectures. In this case, models are not enhanced with additional
parameters but are trained at generating the target sequence using the
original language modeling head. In particular, Teacher-Forcing is a training
technique where a generative model is trained by providing the correct output
at each step. It helps the model learn faster and generate higher-quality
sequences. During inference, the model generates output text based on past
predictions.

Generally, fine-tuning can be performed in different ways based on the
underlying model architecture and the task to be performed. We avoid
describing how every model is adapted for every task because this would be
very verbose and is out of the scope of this work.

##  Chapter 3 Related Work

This Chapter is divided into three main Sections. First, we describe all the
language model architectures we exploit in this work. In the second Section,
we report the works related to the tasks we address in this Thesis, such as
Answer Sentence Selection. In the last Section instead, we discuss the related
state-of-the-art techniques we compare with and from which we took
inspiration.

###  3.1 Language Models

Model | Type | # Params | Vocab | # Layers | Hidden size | # Att. heads | # Interm. size | # Tokens
---|---|---|---|---|---|---|---|---
BERTSmall | AE | 13M | 30K | 12 | 256 | 4 | 1024 | 65B
BERTBase | AE | 109M | 30K | 12 | 768 | 12 | 3072 | 131B
BERTLarge | AE | 335M | 30K | 24 | 1024 | 16 | 4096 | 131B
RoBERTaBase | AE | 125M | 50K | 12 | 768 | 12 | 3072 | 2097B
RoBERTaLarge | AE | 355M | 50K | 24 | 1024 | 16 | 4096 | 2097B
ELECTRASmall | AE | 18M | 30K | 12 | 256 | 4 | 1024 | 65B
ELECTRABase | AE | 109M | 30K | 12 | 768 | 12 | 3072 |  524B♣
ELECTRALarge | AE | 335M | 30K | 24 | 1024 | 16 | 4096 |  1835B♣
ALBERTBase | AE | 12M | 30K | 12 (repeating) | 768 | 12 | 3072 |  2097B♠
ALBERTLarge | AE | 17M | 30K | 24 (repeating) | 1024 | 16 | 4096 |  2097B♠
DeBERTaBase | AE | 139M | 50K | 12 | 768 | 12 | 3072 | 1048B
DeBERTaLarge | AE | 406M | 50K | 24 | 1024 | 16 | 4096 | 1048B
DeBERTaV3Base | AE | 184M | 128K | 12 | 768 | 12 | 3072 | 2097B
DeBERTaV3Large | AE | 434M | 128K | 24 | 1024 | 16 | 4096 | 2097B
BARTBase | AR | 139M | 50K | 6+6 | 768 | 12 | 3072 | 2097B
BARTLarge | AR | 406M | 50K | 12+12 | 1024 | 16 | 4096 | 2097B
T5Base | AR | 223M | 32K | 12+12 | 768 | 12 | 3072 | 1048B
T5Large | AR | 738M | 32K | 24+24 | 1024 | 16 | 4096 | 1048B

Table 3.1: Comparison of several state-of-the-art Transformer models. “Type”
indicates whether the model is an Auto-Encoder or Auto-Regressive model.
“Token” is the total number of tokens seen while pre-training. $\clubsuit$: We
report the number of tokens seen in pre-training of ELECTRABase++ and
ELECTRALarge-1.75M respectively, because those are the models that have been
released by the authors. $\spadesuit$: We report the number of tokens seen by
the models released in the corresponding repository.

The statistics about the model’s number of parameters, the data and the
setting used to pre-train the architectures exploited in this work are
reported in Table 3.1.

####  3.1.1 BERT

BERT [Devlin et al., 2019] is an architecture released by Google Research that
is built upon the Transformer architecture, which allows it to capture long-
range dependencies in text. BERT is initially pre-trained on large amounts of
unlabeled text data from sources like books and Wikipedia, using Masked
Language Modeling and Next Sentence Prediction objectives. Unlike previous
models that mainly relied on left-to-right or right-to-left context, BERT
leverages bi-directional context to perform predictions while pre-training or
fine-tuning. BERT was initially released in two versions based on the number
of parameters, the number of layers and the hidden state size. However,
recently the authors released also smaller, larger and multilingual variants.

####  3.1.2 RoBERTa

RoBERTa’s [Liu et al., 2019b] architecture is the same as BERT. It was
developed by Facebook AI Research in which the authors discovered that by
training BERT for longer, on more data and with different hyper-parameters,
the model could perform much better on a large number of downstream tasks.
More specifically, they (i) removed the Next Sentence Prediction objective;
(ii) applied dynamic masking in MLM; (iii) increased the batch size and the
learning rate; (iv) used a training corpus almost 10 times larger and (v)
changed the tokenizer from WordPiece with 30K tokens to BPE with 50K tokens.
RoBERTa has been initially released in different variants based on the number
of hidden layers, the hidden dimension and the total number of parameters.

####  3.1.3 ELECTRA

ELECTRA [Clark et al., 2020] is yet another Transformers-based model released
by Google Research which is trained as a discriminator rather than as a
generator. ELECTRA is composed of a small generator network trained with MLM
and a large discriminator model trained with Token Detection. The generator is
provided with text in which 15% of the tokens are masked and should find
reasonable replacements. On the other hand, the discriminator should identify
which tokens are original and which have been introduced by the generator.
After the pre-training, the generator is discarded and the discriminator is
fine-tuned for the desired task.

ELECTRA offers several advantages over BERT. One significant advantage is its
use of the entire discriminator output to compute the loss value, providing a
stronger signal for optimization. Additionally, the discriminator in ELECTRA
is never exposed to input sequences containing the [MASK] token, as the
generator replaces these tokens with challenging alternatives. This addresses
a key issue in BERT, where the [MASK] token is present during pre-training but
absent during fine-tuning, leading to a discrepancy between the two stages.

ELECTRA was shown to match BERT’s performance using only half of the
computational resources required by the latter.

####  3.1.4 ALBERT

ALBERT [Lan et al., 2020] is a Transformer-based model that was developed by
Google Research as a more efficient variant of BERT. ALBERT is designed to
reduce the memory footprint and computational requirements of BERT while
maintaining its effectiveness in various NLP tasks. ALBERT introduces three
key differences with BERT. First, the model shares the parameters of all the
layers, thus being much leaner in terms of memory footprint and faster in
training/inference. Secondly, they refactor the embedding layer into two
separate projection matrices that reduce the total number of parameters.
Third, the substitute BERT’s NSP objective with Sentence Order Prediction
(SOP), that was shown to be more challenging and more effective when the model
is fine-tuned on the downstream tasks. Finally, they pre-train the model for
longer and over more data than BERT. Thanks to these optimizations, they
achieve state-of-the-art results on a wide range of NLP tasks such as GLUE,
SQuAD and RACE.

####  3.1.5 DeBERTa

###### DeBERTa

is a Transformer-based architecture developed by Microsoft Research that uses
disentangled attention to improve the representation of tokens and positions
[He et al., 2020]. In other successful models such as BERT and RoBERTa, the
position and the token embeddings are summed together before being fed to the
Transformer layers. In DeBERTa, each token is represented with 2 distinct
vectors: 1 for the relative positions and 1 for the token representation. In
the Transformer layer, the Attention Mechanism is applied to tokens-to-tokens,
tokens-to-positions and positions-to-tokens. Position-to-position attention
scores are not computed because they are not informative. Moreover, DeBERTa is
pre-trained with an enhanced version of MLM that uses absolute positions
applied just before the language modeling head to help in predicting the right
token in long sequences.

DeBERTa is shown to outperform RoBERTa on various tasks using less
computational resources. The largest model released surpassed the human
baseline on SuperGLUE [Wang et al., 2019a] for the first time.

###### DeBERTaV2

further improves upon the original DeBERTa model111More information about
DeBERTaV2 can be found here: https://github.com/microsoft/DeBERTa. It
introduces two key modifications: disentangled attention is now utilized in
both Self-Attention and Cross-Attention (for decoding), and it utilizes an
additional training objective called document-level language modeling. These
enhancements are designed to improve the model’s understanding of long-range
dependencies and its generalization capabilities.

###### DeBERTaV3

is yet another improvement over DeBERTaV2 in which the model is trained in an
ELECTRA-like configuration, i.e. the architecture is composed of a small
generator network trained with MLM and a large discriminator trained with TD
[He et al., 2021]. At the time of writing, DeBERTaV3 is the most accurate
language model among all the architectures with less than one billion
parameters.

####  3.1.6 BART

BART (Bidirectional and Auto-Regressive Transformer) is a Sequence-To-Sequence
(Encoder-Decoder) language model developed by Facebook AI Research [Lewis et
al., 2020]. It is based on the Transformer architecture and is trained on
large amounts of unlabeled text data, where it learns to reconstruct
corrupted, shuffled or masked input sequences with Teacher-Forcing. This
process helps the model to acquire a rich understanding of language patterns
and semantics which resulted in state-of-the-art performance across many
generative tasks such as Machine Translation, Summarization and Question
Answering.

####  3.1.7 T5

T5 (Text-to-Text Transfer Transformer) is a versatile language model developed
by Google AI [Raffel et al., 2020]. It is trained using a “text-to-text”
framework, where a wide range of NLP tasks are cast as a text generation
problem. T5 is trained to map a source text, such as a prompt or question, to
a target text, which can be a summary, a translation, a sentiment analysis, or
any other entity. The pre-training objectives of T5 include tasks like
Translation, Summarization, Question Answering, Text Classification, and more.
By training on several different tasks, T5 learns to generalize across
different NLP domains and can be exploited for different applications in a
zero-shot setting by only adapting the input prompt, without changes to the
model architecture. T5 has achieved state-of-the-art results on various
benchmark datasets across multiple tasks such as Machine Translation,
outperforming previous models on standard translation benchmarks. T5 also
excels in Summarization, Question Answering, and Text Classification,
surpassing previous approaches and achieving competitive performance.

###  3.2 Natural Language Processing Tasks

This Section provides an overview of the works related to the tasks we address
in this Thesis.

####  3.2.1 Answer Sentence Selection

This Section relates to the experiments described in Chapters 5, 6, 7 and 8.
Answer Sentence Selection is a branch of Question Answering in which the task
is to re-rank a set of possible answer candidates such that the correct
answers to the given question are provided with the highest score.

Previous approaches to AS2 involved using CNNs to learn and score question and
answer representations. Severyn and Moschitti [2015] discusses the importance
of learning a similarity function between pairs of text spans in ranking tasks
such as Information Retrieval and AS2. The authors propose a CNN architecture
that learns the optimal representation of text pairs and a similarity function
in a supervised way. The model achieves state-of-the-art accuracy on the tasks
mentioned before and shows comparable results in tweet re-ranking without the
need for manual feature engineering or additional syntactic parsers.

Another relevant improvement in AS2 was achieved thanks to Alignment Networks
[Shen et al., 2017]. Alignment Networks allow the model to measure the
similarity between two sentences based on their interaction information.
Specifically, the authors exploit a similarity matrix built over the word
representation of both input sentences to discover alignments. Moreover, their
system automatically computes a weight for every word in the sentences based
on estimated importance. By applying this methodology to LSTM Networks, they
reach state-of-the-art results in different AS2 benchmarks, such as TREC-QA
and WikiQA.

Compare-and-Aggregate architectures for Question Answering have been
extensively studied [Wang and Jiang, 2017; Bian et al., 2017a; Yoon et al.,
2019]. Compare-and-Aggregate differs from the previous techniques in the way
the question and answer embeddings are used to perform the final prediction.
Previous methods usually exploit a single vector representing the question and
another vector for the answer to predict the target label. In Compare-and-
Aggregate instead, the prediction is performed by comparing smaller units
between the question and the answer, such as single-word embeddings. Then,
results from all comparisons are aggregated and used for the final decision.
It is worth mentioning that the authors of Yoon et al. [2019] mixed Clustering
techniques with Compare-and-Aggregate to improve final re-ranking even
further.

In Bonadiman and Moschitti [2020], the authors exploit separate encoders for
the question and the candidate answer for a more efficient re-ranking. They
show it is possible to achieve good accuracy with a training phase lasting
only a few minutes. However, the exploitation of lighter architectures such as
LSTMs leads to final lower results on various AS2 datasets when compared with
methods based on large Transformer models.

Recently, Garg et al. [2019] achieved state-of-the-art results with their new
methodology called Transfer-and-Adapt. In TandA, a large Transformers model is
first fine-tuned on a large QA corpus (ASNQ) and then it is adapted to the
downstream task, which can be a small dataset such as WikiQA and TREC-QA. The
advantage is that the model can exploit previously learned relations between
the question and the answer to perform better in the final task. This is
especially true on small target datasets such as WikiQA and TREC-QA, on which
TandA reaches state-of-the-art accuracy.

Another noteworthy contribution in the field of Answer Sentence Selection is
presented by Gabburo et al. [2022]. In their work, the authors utilize Auto-
Regressive models to generate answers based on the topk candidate sentences
retrieved by a Pairwise re-ranker. Building upon GenQA [Muller et al., 2022],
they introduce a self-supervised technique that transfers knowledge from AS2
to Generative QA. This technique challenges the model to reconstruct the top
answer retrieved by the AS2 system, leveraging only the next $k$ candidates in
the ranking, thereby eliminating the need for labeled datasets.

###### Contextual Answer Sentence Selection

The works described in this Section refer to experiments presented in Chapter
8. Contextual AS2 is an improvement over AS2 in which the model has access to
additional knowledge for each answer candidate in order to perform a better
re-ranking [Barlacchi et al., 2022]. The context is a very precious source of
information because it can help in solving entities and relations in the
answer candidate.

In Ghosh et al. [2016], the authors introduce CLSTM (Contextual LSTM), an
extension of the LSTM network that incorporates contextual features into the
model. Experimental results over three NLP tasks (Word Prediction, Next
Sentence Selection, and Sentence Topic Prediction) show that using both words
and topics as features improves CLSTM performance over corresponding
baselines. The study demonstrates that enhancing models with additional
context is essential to improve accuracy in tasks like Question Answering,
Sentence Completion, Paraphrase Generation, and Next Utterance Prediction in
dialog systems.

The authors of Tran et al. [2018] enhance AS2 systems with context. To the
best of our knowledge, this is the first work targeting specifically
Contextual AS2. The authors propose an innovative architecture called Context-
dependent Additive Recurrent Neural Network (CARNN) that incorporates strong
external signals for a better learning process. The authors show state-of-the-
art results on several datasets, including TREC-QA.

In Tan et al. [2018], the authors exploit GRU Networks to model answer
candidates and local context, improving performance on two AS2 datasets.
Unlike previous approaches, that treated question-answer pairs independently,
the authors emphasize the importance of contextual information within the
passage. They propose a hierarchical GRU model that incorporates context
information at both the word and sentence levels. The model outperforms state-
of-the-art methods on the WikiQA and SQuAD datasets, demonstrating the
effectiveness of their approach.

Lauriola et al., 2021 propose a Transformer encoder that uses both local and
global document-level context to better disambiguate between answer
candidates. Moreover, the authors perform experiments over the combination of
Transfer-Learning (using the TandA methodology) and Contextual AS2, reaching
state-of-the-art performance on several AS2 datasets, such as WikiQA.

Finally, Han et al. [2021], the authors exploit unsupervised retrieval
techniques to incorporate contextual information into Answer Sentence
Selection (AS2) models. Traditional AS2 models score question-answer pairs
individually without considering the document context. The authors propose an
approach that efficiently incorporates context by extracting relevant
sentences using unsupervised similarity techniques and feeding them into the
fine-tuned Transformer architecture. Their approach, leveraging a multi-way
attention architecture, improves the state-of-the-art in AS2 while maintaining
low system latency.

####  3.2.2 Fact Verification

Fact Verification is the task of predicting whether a claim is supported,
refuted or neutral compared to a set of evidence sentences. This Section is
related to the discoveries of Chapter 6.

An initial significant approach to address the Fact Verification task is
represented by GEAR [Zhou et al., 2019]. In GEAR, the authors address the
issue that previous methods have not effectively integrated and reasoned over
multiple pieces of evidence. The proposed solution, a graph-based evidence
aggregating and reasoning framework, allows the transfer of information on a
fully-connected evidence graph and the usage of different aggregators to
collect evidence information. The authors exploit contextual embeddings or
BERT to further enhance performance.

DOMLIN [Stammbach and Neumann, 2019] is another technique that targets Fact
Verification. In DOMLIN, the authors design a new system to select evidence
sentences that does not only consider the claim but also the previously
extracted evidence. They exploit BERT for both retrieval and claims
classification and show promising performance gains on the FEVER dataset.

Another relevant work regarding Fact Verification is Kernel Graph Attention
Network (KGAT) [Liu et al., 2020]. In this work, the authors build a graph
containing the evidence sentences and apply two different kernels to it: (i)
node kernels, which measure the relevance of the evidence compared to the
given claim and (ii) edge kernels, which propagate evidence in the graph for a
more sophisticated Fact Verification. The target is to create a network able
to discover claims that are syntactical and semantically acceptable but not
strongly supported by the retrieved evidence. KGAT reached state-of-the-art
performance on the FEVER benchmark.

In Transformers-XH [Zhao et al., 2020], the authors address the problem that
natural languages usually have a complex structure, which can be represented
with trees or graphs. Thus, they propose to extend the Attention Mechanism
with eXtra Hops, a technique that enables the model to attend also between
text sequences than only single tokens. This innovation improves inference
over tasks in which the model should reason over several text spans, such as
Multi-Hop Question Answering and Fact Verification, reaching state-of-the-art
performance on HotpotQA and FEVER.

Tymoshenko and Moschitti [2021] propose two interesting baselines for Fact
Verification. They show that when using Transformer-based models, simple
transformations of the output embeddings allow them to reach very good
accuracies. Specifically, they exploit the evidence sentence embeddings by
applying a max-pooling layer or by computing the weighted sum. Results on
FEVER show that max-pooling and weighted sum are effective techniques to
achieve state-of-the-art performance.

Finally, we describe two related works that use very large language models to
retrieve more evidence and to perform Fact Verification. In Stammbach and Ash
[2020-10], the authors improve DOMLIN by utilizing a large model for evidence
retrieval and by exploiting GPT-3 [Brown et al., 2020] few-shot learning
capabilities to summarize the retrieved evidence for each claim in the target
dataset. Their new architecture, called DOMLIN++, achieves state-of-the-art
results on the FEVER dataset. The other relevant work is DREAM [Zhong et al.,
2020], in which the authors propose a new methodology to reason about the
semantic structure of evidence sentences. By exploiting pre-trained language
models such as XLNet [Yang et al., 2019], they create a graph that describes
the semantic similarity of words. Then, they propagate the information through
the graph with Graph Convolutional Networks and Graph Attention Networks to
contextualize the representation of words in nodes. Exploiting the described
architecture, the authors achieve state-of-the-art results on FEVER on both
evaluation metrics, namely Label Accuracy and FEVER score.

####  3.2.3 Summarization

Abstractive Summarization comprehends a wide range of benchmarks, models and
techniques. In this Section, we try to summarize the most relevant works to
provide an overview of the actual state-of-the-art in this field.

Abstractive Text Summarization is a generative task and the actual state-of-
the-art is based on Auto-Regressive Transformer models in the Encoder-Decoder
or Decoder-only configuration. For this reason, most of the related works are
based on architectures such as BART or T5. For details about those
architectures, refer to Section 3.1, while for the differences between the
Encoder-Decoder and Decoder-only configurations, read Section 2.2.1.5.

The first work we review is SimCLS [Liu and Liu, 2021], which is a technique
that combines contrastive learning with generative models to create high-
quality summaries. The authors break down the generation process into two
separate stages. First, an Auto-Regressive model such as BART is trained and
exploited to generate $n$ summaries with Beam Search. Secondly, they design a
parametric function to evaluate each summary candidate independently. More
specifically, they train a RoBERTa model with a contrastive loss to measure
the similarity between the original document and each possible candidate.
Finally, the authors show that applying SimCLS to BART or PEGASUS improves
results on several benchmarks such as CNN/DailyMail and XSum.

PEGASUS [Zhang et al., 2020a] is another model specifically pre-trained for
Abstractive Summarization, in which an algorithm selects a subset of sentences
that are removed from the original document and used as pseudo-summaries. When
released, PEGASUS was shown to outperform previous SOTA on 12 Abstractive
Summarization benchmarks.

Another relevant work that addresses Abstractive Summarization is BRIO [Liu et
al., 2022]. In this work, the authors address the mismatch between training
and inference on generative models, called exposure bias. While training, the
model predicts the next token using Teacher-Forcing based on the assumption
that all the previous tokens are correct. In inference instead, the previous
tokens have been generated by the model itself and there is no guarantee they
are correct. Instead of training with maximum likelihood estimation, the
authors assume a non-deterministic output distribution of token probabilities
and propose an innovative training algorithm in which every summary generated
by the model is assigned a probability mass based on the quality. They reached
state-of-the-art performance on several Summarization datasets, such as
CNN/DailyMail and XSum.

SummaReranker [Ravaut et al., 2022] is a Mixture-of-Experts model that
addresses again the exposure bias problem by combining a model specialized for
Summarization with a re-ranker that chooses the best summary among those
generated by the former. By exploiting PEGASUS as the Summarization
architecture, they reached state-of-the-art models on different datasets such
as CNN/DailymMail, XSum and Reddit TIFU.

At the time of writing, the state-of-the-art model for Abstractive
Summarization on the datasets we consider is MoCa [Zhang et al., 2022b]. Here,
the authors address the exposure bias issue in inference, where the model
should choose the best summary among possible candidates that deviated from
the gold summary. The authors propose an alternative generation algorithm in
which the model generates several samples with beam search that are aligned to
the gold summaries while training, utilizing the momentum moving average
technique.

###  3.3 Techniques

####  3.3.1 Efficient Objectives for Pre-Training

In this Section we analyze recent techniques that try to reduce training time
and carbon footprint, to reach the same final accuracy as previous state-of-
the-art models. These works are related to our research about efficient and
effective objectives presented in Chapter 5.

Recently, many self-supervised pre-training objectives have been proposed to
train Transformer-based models. Those objectives are mostly self-supervised
because the pre-training phase is performed over large corpora of text without
explicit annotation.

An example of a work that targets reduced training time by proposing
architectural modifications is ALBERT, described in Section 3.1.4. In ALBERT
the parameters of every Transformer layer are tied to save memory, enabling
larger batch sizes. However, this approach reduces the expressive power of the
models, requiring longer pre-trainings. Another approach used by Sanh et al.
[2020] and Turc et al. [2019] is distillation. Distillation exploits a large
and already pre-trained teacher model to train a lighter architecture to
comparable accuracy, thus compressing the knowledge of the teacher into the
smaller model. Distillation reduces the final architecture size, but the pre-
training phase remains expensive due to the large teacher model.

Pre-training is typically a time-consuming process, taking weeks and requiring
costly machines [Liu et al., 2019b; Brown et al., 2020]. Therefore, it is
important to explore alternative training tasks for pre-training Transformers.
In Tay et al. [2020], the reader can find a comprehensive survey of recent
advancements in Transformer efficiency.

Regarding the learning objectives, SpanBERT [Joshi et al., 2020] introduces
two new training tasks: Span Masking (SM) and Span Boundary Objective (SBO).
SM masks contiguous spans of text instead of individual tokens, while SBO
predicts span content using only the output representation corresponding to
the boundary tokens. Another research about Transformers efficiency is
proposed by Zhang et al. [2020b], in which the authors suggest to adapt the
model to the final task during pre-training to enhance downstream performance,
which is related to the methods we propose in Chapter 7, 8 and 9.

In T5 [Raffel et al., 2020], the authors propose the use of Deshuffling [Liu
et al., 2019a] to pre-train an Auto-Regressive model. This technique involves
shuffling random spans of text and requiring the model to output tokens in the
original order. It has shown promising results across various benchmarks.

Although we focus solely on Auto-Encoder architectures, it is worth mentioning
the work of Izsak et al. [2021], which discusses the usage of multiple
optimizations for faster pre-training in Transformers. They also suggest that
using larger models with the same runtime can yield better results. However,
inference speed and efficiency decrease as the number of parameters of the
model increases.

####  3.3.2 Multi-Sentence Inference

Inference over multiple sentences has already been studied in the past, also
for different problems than Answer Sentence Selection. In this work, we do not
target new architectures for multi-sentence inference, but effective pre-
training strategies to improve models’ ability at collecting information from
several spans of text.

Answer Sentence Selection is a challenging task because the model needs to
effectively capture the semantic connections between questions and answers. In
[Bian et al., 2017b], the authors propose two improvements over previous
state-of-the-art methods, mostly based on Compare-Aggregate networks, which
are described in Section 3.2.1. First, they introduce Dynamic-Clip Attention,
which focuses on reducing noise in the attention matrix to better find
relations between the question and the answer tokens. Secondly, they feed
their architecture with a question and a set of answer candidates and task the
model with predicting the relative order of the latter. They reach state-of-
the-art results on WikiQA and TREC-QA.

Another relevant work about inference over multiple text spans is Ai et al.
[2018], which addresses the Information Retrieval task. In this work, the
authors demonstrate that ranking documents independently for each query is
suboptimal because they may follow a different distribution in the feature
space. For this reason, they develop a Deep Listwise Context Model, which is
applied to the most relevant documents retrieved for each query to improve
their final ranking. The proposed architecture is designed to encode multiple
documents and to model their interactions, and could be applied on top of
other ranking systems. By training with an attention-based loss function, they
achieve state-of-the-art results on several benchmarks for Information
Retrieval.

The most relevant related works targeting Answer Sentence Selection with
multiple answer candidates are Bonadiman and Moschitti [2020] and Zhang et al.
[2021]. The first work employs cheap and fast neural networks to rank multiple
sentences together. It leverages the original order of answer candidates when
extracted from source documents for a better ranking. When tested over WikiQA
or TREC-QA, the proposed architecture is hundreds of times faster than
Transformer-based models such as BERT, and achieves better performance than
previous approaches with a similar number of parameters. The second work
exploits mutual information and relations between several answer candidates to
improve the ranking quality. Specifically, their architecture performs a
three-way classification between answer candidate pairs and should predict
whether candidates support, refute or are neutral compared to the others,
similar to a Fact Verification pipeline. They combine state-of-the-art AS2
systems with their multi-classifier through a joint layer, which outputs
scores for the ranking. The proposed architecture obtains state-of-the-art
results on several AS2 datasets, such as WikiQA and TREC-QA.

####  3.3.3 Exploiting Document Structure

The intrinsic structure of documents has been leveraged in various studies.
For instance, RoBERTa [Liu et al., 2019b] generates input examples for pre-
training while avoiding crossing document boundaries. This approach yields a
more accurate model across multiple benchmarks because the model focuses on a
single topic at a time and can be fine-tuned more effectively for target
tasks.

Another work that exploits document structure is REALM [Guu et al., 2020], in
which the authors augment models pre-training with a neural retriever that
collects additional knowledge from large sources of text, such as Wikipedia.
By applying the described technique in both pre-training, fine-tuning and
inference, they incorporate useful additional knowledge from millions of semi-
structured documents, reaching state-of-the-art on several Open Domain
Question Answering benchmarks. This work differs from our pre-training setting
because it exploits external knowledge to improve the model’s accuracy.

DeCLUTR [Giorgi et al., 2021] is yet another approach that exploits the
document structure to create better sentence-level embeddings. They design a
self-supervised objective that challenges the model at predicting whether two
sentences are extracted from the same paragraph of a document. However, as
opposed to our discoveries, they do not exploit the Cross-Attention of
Transformer models and work only at the individual sentence level. The
continuous pre-training they apply starting from publicly available
checkpoints reduces the performance gap with models trained over labeled
datasets. They show improvements in accuracy over several tasks that require
reasoning over two input sentences.

##  Chapter 4 Datasets

The architecture and the training objectives proposed in this Thesis have been
evaluated on a wide range of benchmarks. In this Section, we describe the
domain, the number of examples and the annotation strategy of various
datasets. Moreover, we provide details about how data have been collected from
the original sources.

###  4.1 Pre-Training

Large corpora of raw text are of fundamental importance for models pre-
training. Statistics for all datasets are reported in Table 4.1.

Dataset | # Docs | # Words | # Words/Doc | # Para/Doc | # Sent/Para | Size
---|---|---|---|---|---|---
Wikipedia | 4,184,712 | 1,610,571,882 | 384.8 | 7.9 | 2.3 | 9.9GB
BookCorpus | 17,868 | 1,149,401,064 | 64,327.3 | 2005.9 | 2.5 | 6.2GB
Open-Web-Text | 8,013,769 | 6,401,507,179 | 798.8 | 20.7 | 1.9 | 38.8GB
CC-News | 149,954,415 | 62,676,498,100 | 418.0 | 16.5 | 1.3 | 385.4GB

Table 4.1: Statistics about the pre-training corpora exploited in this work.
Cleaning involves filtering HTML tags and JavaScript code from pages, images,
tables and other entities which are not raw text.

###### Wikipedia

The Wikipedia dataset is a collection of text data derived from the Wikipedia
online encyclopedia. It encompasses a diverse range of subjects, covering
fields like science, history, geography, arts, and more. It contains a
substantial amount of text, including introductory paragraphs, detailed
explanations, citations, and references, providing a comprehensive and
authoritative source of information for a wide array of topics111We publish
our pre-processed version of Wikipedia here:
https://huggingface.co/datasets/lucadiliello/english_wikipedia, which
corresponds to the dump of November 1, 2021..

###### BookCorpus

The BookCorpus dataset [Zhu et al., 2015] is a relatively small collection of
text data created by Google Research as a means to provide a diverse and
extensive source of textual information for Machine Learning purposes. We use
the Open version of this dataset in our experiments, which consists of
approximately 17,868 books from a wide range of genres and topics222We used
the version of the BookCorpus released here:
https://huggingface.co/datasets/bookcorpusopen. The books included in the
dataset were published between 2007 and 2020 and were released under and open-
source license.

###### OpenWebText

The OpenWebText dataset is a collection of text data derived from various
sources on the internet, excluding the Wikipedia website. It aims to capture a
broader range of language patterns, writing styles, and topics compared to
datasets focused solely on Wikipedia. The dataset is typically compiled by
scraping publicly available web pages, blogs, forums, news articles, and other
online sources. The original scraping script was created by OpenAI but it was
not publicly released. We use an open version of the OpenWebText, which is
based on Reddit dumps that are filtered based on the Karma score ($>=3$) of
the threads333We used the version of OpenWebText released here:
https://github.com/jcpeterson/openwebtext.

###### CC-News

CC-News is a dataset released by the Common Crawl Foundation that is
exclusively centered around news articles gathered from the web. CC-News aims
to offer a comprehensive and diverse compilation of news articles,
encompassing various topics, languages, and sources. The dataset covers
articles from a wide range of news websites, covering domains such as
politics, sports, technology, entertainment, and more. We parse the data of
the original CC-News dataset to keep a random subset of the English articles
published between 2016 and 2020444We release our version of CC-News here:
https://huggingface.co/datasets/lucadiliello/cc_news and we filter away
artifacts such as HTML code, headers, titles from the WARC files.

###  4.2 Fine-Tuning

####  4.2.1 Answer Sentence Selection

The main task addressed in this Thesis is Answer Sentence Selection. For this
task, we evaluate our proposed models and objectives on several public and two
industrial datasets. Moreover, we created 2 additional corpora derived from
well-known Question-Answering datasets, similar to how the authors in Garg et
al. [2019] created ASNQ. With a total of 8 datasets, we present an exhaustive
evaluation setting that includes datasets of varying sizes, ranging from very
few thousand examples to more than 10 million. Moreover, we include datasets
where sentence candidates are extracted from both single and multiple source
documents. We created _clean_ versions of each dataset, which means we removed
questions without at least a positive and a negative answer candidate in the
development and test set, which is the standard practice in AS2 [Garg et al.,
2019; dos Santos et al., 2016]. This allows us to better compare results
because otherwise, there would have been upper bounds to the AS2 metrics
dependent on the tested dataset.

###### ASNQ

The Answer Sentence Natural Question (ASNQ) is a large dataset for AS2 derived
from Google’s Natural Questions corpora [Kwiatkowski et al., 2019]. The
questions have been retrieved from Google’s search engine and the system is
required to reason over entire Wikipedia articles to find the correct answer.
NQ is a challenging dataset because it contains thousands of questions asked
by real users and answers extracted from long Wikipedia articles. The
annotations include both _long_ and _short_ answers. The short answers are
very small text spans that contain the exact answer to the question. Long
answers are instead paragraphs that contain the exact answer but also other
sentences and text. Most of the short answers are contained in a long answer.
There are very few rare cases in which the short answers are not contained in
the long answers, probably due to annotation mistakes. More specifically, NQ
contains 307,373 train and 7,830 development user questions and, for each
question, the most relevant Wikipedia article is provided. For the train set,
a single long answer annotation is available. For the development set instead,
the authors asked 5 different annotators to select a long answer. Regarding
the short answers, on average there are 0.42 annotations in the train set and
2.10 in the development set.

ASNQ is derived from NQ by Garg et al. [2019] with the following rules. First,
the authors split every retrieved Wikipedia document into a list of answer
sentence candidates with the NLTK tokenizer [Bird et al., 2009]. Then, they
assign a positive label to answer candidates which are entirely contained in a
long answer and that contain at least a short answer. Negative answer
candidates comprehend instead: (i) sentences in a long answer but not
containing a short answer; (ii) sentences containing a short answer but not
contained in a long answer; (iii) sentences both not in a long answer and not
containing a short answer. Notice that the formers are very hard negatives
because they are very close to the short answer (in the same paragraph) and
contained in a long answer. Moreover, the second case happens only because of
a few mistakes in the annotation process. Finally, the authors discard
questions not having at least a positive answer candidate.

The statistics for the resulting ASNQ dataset are reported in Table 4.2. For
the experiments in Chapter 5, we report results on the original ASNQ
development set, because the dataset does not provide a test set. The results
in Chapters 6, 7 and 8 are instead on the test set released by Soldaini and
Moschitti [2020], where the authors split the original development set for
both validation and testing.

###### WikiQA

Dataset | Train set | Development set | Test set
---|---|---|---
Q | QA | A/Q | +/Q | Q | QA | A/Q | +/Q | Q | QA | A/Q | +/Q
ASNQ | 57242 | 20377568 | 356.0 | 1.1 | 1336 | 463914 | 347.2 | 1.6 | 1336 | 466148 | 348.9 | 1.6
WikiQA | 2118 | 20360 | 9.6 | 0.5 | 122 | 1126 | 9.2 | 1.1 | 237 | 2341 | 9.9 | 1.2
TREC-QA | 1158 | 53310 | 46.0 | 5.4 | 69 | 1343 | 19.5 | 3.5 | 68 | 1442 | 21.2 | 3.5

Table 4.2: Statistics for ASNQ, WikiQA and TREC-QA. “Q” and “QA” stand for the
number of unique question and question-answer pairs. “A/Q” is the average
number of candidates per question while “+/Q” is the average number of
positive answers per question.

WikiQA is a small and challenging dataset for Answer Sentence Selection
released by Microsoft and based on queries asked to the Bing search engine
[Yang et al., 2015]. Based on research results, each question is associated
with a Wikipedia page. Sentences from the summary (first paragraph) are then
used as answer candidates and are annotated by crowd-workers. To ensure a high
annotation quality, each sentence is independently analyzed by three different
workers. Sentences with inconsistent labels were further parsed by a different
set of workers and finally, labels were assigned based on majority vote.
Statistics about the number of questions and answer candidates are given in
Table 4.2.

###### TREC-QA

TREC-QA is yet another popular dataset for Answer Sentence Selection created
from the TREC-8 to TREC-13 tracks of Question Answering [Wang et al., 2007].
The corpus is composed of factoid questions while answers are extracted from
web documents by participating teams. In particular, the authors of Wang et
al. [2007] use the tracks TREC-8 to TREC-12 for training and the track TREC-13
for development and testing. Manual annotation of positive candidates is
performed on both the development and test sets. In the training set, only the
answers to 100 questions received gold labels while the others were
automatically annotated, thus being more noisy. The split containing also the
manually annotated candidate answers is called _train-all_. We train over
train-all, which is the standard practice [dos Santos et al., 2016], because
even if it is noisier, it contains much more question-answer pairs. Negative
answer candidates are automatically extracted by selecting sentences with
overlapping words with the question. Statistics for TREC-QA are given in Table
4.2.

###### IQAD

IQAD (Internal Question Answering Dataset) is a large-scale internal dataset
of the Alexa Search team used to evaluate QA systems. The dataset was built
from Alexa traffic by de-identifying user questions asked to Alexa, which is a
popular virtual assistant. The de-identification process removes all
connections between questions and original Alexa customers. The dataset
contains about 220K general topic questions and a large set of answer
candidates extracted with ElasticSearch from a web index containing more than