# Challenges and Opportunities of Using Transformer-Based Multi-Task Learning
in NLP Through ML Lifecycle: A Survey

&Lovre Torbarina${}^{\;\;,}$${}^{\;\;,\;\omega}$
&Tin Ferkovic11footnotemark: 1${}^{\;\;,\;\omega}$
& Lukasz Roguski${}^{\;\omega}$
&Velimir Mihelcic${}^{\;\omega}$
&Bruno Sarlija${}^{\;\omega}$
&Zeljko Kraljevic${}^{\;\omega}$

${}^{\omega\;}$doXray B.V., Neede, Netherlands
<EMAIL_ADDRESS>
Equal contribution.Correspondence to: Lovre Torbarina
<EMAIL_ADDRESS>
###### Abstract

The increasing adoption of natural language processing (NLP) models across
industries has led to practitioners’ need for machine learning systems to
handle these models efficiently, from training to serving them in production.
However, training, deploying, and updating multiple models can be complex,
costly, and time-consuming, mainly when using transformer-based pre-trained
language models. Multi-Task Learning (MTL) has emerged as a promising approach
to improve efficiency and performance through joint training, rather than
training separate models. Motivated by this, we first provide an overview of
transformer-based MTL approaches in NLP. Then, we discuss the challenges and
opportunities of using MTL approaches throughout typical ML lifecycle phases,
specifically focusing on the challenges related to data engineering, model
development, deployment, and monitoring phases. This survey focuses on
transformer-based MTL architectures and, to the best of our knowledge, is
novel in that it systematically analyses how transformer-based MTL in NLP fits
into ML lifecycle phases. Furthermore, we motivate research on the connection
between MTL and continual learning (CL), as this area remains unexplored. We
believe it would be practical to have a model that can handle both MTL and CL,
as this would make it easier to periodically re-train the model, update it due
to distribution shifts, and add new capabilities to meet real-world
requirements.

##  1 Introduction

In recent years, advancements in natural language processing (NLP) have
revolutionized the way we deal with complex language problems. Consequently,
those advances have been significantly impacting the global industry, driving
growth in organizations that incorporate AI technologies as part of their core
business. To illustrate, McKinsey’s state of AI report for 2022 has reported
an increase of 3.8 times since 2017 in AI capabilities that organizations have
embedded within at least one function or business unit, where natural language
understanding (NLU) took third place among reported capabilities, just behind
computer vision (Chui et al., 2022). Furthermore, Fortune Business Insights
projected growth of global NLP from USD 20.80 billion in 2021 to USD 161.81
billion by 2029.111Fortune Business Insights - NLP Market Size As a result,
each NLP practitioner offering models through an API or using them internally,
alone, or together with other AI capabilities, must have a machine learning
(ML) system to efficiently manage these models. This involves having well-
established processes from training and verifying those models to deploying
them in production for end-users while continuously monitoring that those
models remain up-to-date with the most recent knowledge they are being trained
on.

This trend of widespread adoption of ML models by various practitioners
throughout industries, and the resulting need for ML systems to manage them
efficiently, was tackled in a survey of ML systems conducted by Paleyes et al.
(2022). The survey analyzed publications and blog posts reported by different
practitioners, providing insights into phases of an ML lifecycle and
challenges that commonly arise during those phases. The ML lifecycle refers to
the phases and processes involved in designing, developing, and deploying an
ML system. It encompasses the entire process, from defining the problem and
collecting data to deploying models and monitoring their performance. Model
learning and model deployment are two important phases in the ML lifecycle,
among others (Ashmore et al., 2021; Paleyes et al., 2022). To support
practitioners’ needs, the model learning phase should be equipped to handle
the training and updating of a large number of models, while the model
deployment phase must provide an easy and efficient way to integrate and serve
those models to run in production, that is, running as part of usual business
operations.

At the same time, it is a common practice in NLP production systems to utilize
pre-trained language models based on transformers (Vaswani et al., 2017) by
fine-tuning them for specific downstream tasks. While effective, the language
models have a large number of parameters that require significant
computational resources to fine-tune. Although fine-tuning pre-trained models
can be more data-efficient than training a model from scratch, the expertise
of annotators or domain experts may still be required to label a large number
of examples, particularly if there is a significant difference between the
downstream task and pre-training objectives (Wang et al., 2020). Therefore, it
is a costly and time-consuming procedure, especially if there is a need to
train and serve multiple models in production. To address the challenge of
training multiple models, researchers have been exploring Multi-Task Learning
(MTL) as a solution (Ruder, 2017). MTL trains a single model to learn multiple
tasks simultaneously while sharing part of the model parameters between them
(Caruana, 1997), making the process memory-efficient and, in some cases,
computationally more efficient than training multiple models. Additionally,
using a single model for multiple downstream tasks in a production system
could simplify the integration of ML models with ML systems and reduce
economic costs. This is due to the modular nature of MTL architectures that
promote code and model sharing, reuse, easier collaboration, and maintenance.
Moreover, the MTL model reduces idle time since the same model is used for
various tasks. Therefore, MTL approaches offer a promising solution to
mitigate some of the difficulties associated with managing multiple models in
ML production systems.

In this survey, we first provide an overview of transformer-based MTL
approaches in NLP (see Section 3). Second, we highlight the opportunities for
using MTL approaches across multiple stages of the ML lifecycle, specifically
focusing on the challenges related to data engineering, model development,
deployment, and monitoring (see Section 4). We focus solely on transformer-
based architectures. To the best of our knowledge, this is the first survey
that systematically discusses the benefits of using MTL approaches across
multiple ML lifecycle phases (see Section 2). Additionally, we encourage
further research on the connection between MTL and Continual Learning (CL). We
argue that having a model capable of handling both MTL and CL is practical as
it addresses the need for periodic re-training and continual updates in
response to distribution shifts and the addition of new capabilities in
production models.

The rest of the paper is organized as follows. In Section 2, we briefly review
related surveys and highlight the gaps addressed in our survey. In Section 3,
we give an overview of transformer-based MTL approaches. In Section 4, we
systematically analyze the benefits of using MTL through specific ML lifecycle
phases. And finally, in Section 5, we give a conclusion to our work.

##  2 Related Surveys

In this section, we give an overview of related work on MTL, ML systems, and
CL, and point out what has not been discussed so far regarding the connection
of MTL to both ML systems and CL.

###  2.1 Multi-Task Learning

Table 1: Discussed aspects per MTL survey. Aspects are indicated in bold. Year | MTL Survey
---|---
$2017$ |  $1$ \- (Ruder, 2017) $2$ \- (Zhang & Yang, 2017)
$2018$ |  $3$ \- (Zhang & Yang, 2018) $4$ \- (Thung & Wee, 2018)
$2019$ |  $5$ \- (Zhou, 2019)
$2020$ |  $6$ \- (Vafaeikia et al., 2020) $7$ \- (Worsham & Kalita, 2020) $8$ \- (Crawshaw, 2020)
$2021$ |  $9$ \- (Vandenhende et al., 2021) $10$ \- (Chen et al., 2021) $11$ \- (Upadhyay et al., 2021)
$2022$ |  $12$ \- (Samant et al., 2022) $13$ \- (Abhadiomhen et al., 2022)
$2023$ |  $14$ \- (Zhang et al., 2023)
Aspect \Survey | $1$ | $2$ | $3$ | $4$ | $5$ | $6$ | $7$ | $8$ | $9$ | $10$ | $11$ | $12$ | $13$ | $14$
Computational Model |
Traditional ML | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  |  |  |  | ✓ |
Deep Learning | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  | ✓
Architectures |
Learning to Share | ✓ |  |  | ✓ | ✓ | ✓ |  | ✓ | ✓ | ✓ |  |  |  | ✓
Universal Models |  |  |  |  | ✓ |  |  | ✓ | ✓ | ✓ |  |  |  | ✓
Optimization |
Loss Weighting | ✓ | ✓ | ✓ |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  | ✓ |  |
Regularization | ✓ | ✓ | ✓ | ✓ | ✓ |  |  | ✓ |  | ✓ |  |  | ✓ |
Task Scheduling |  |  |  |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  |  |  |
Gradient Modulation | ✓ |  |  |  |  | ✓ |  | ✓ | ✓ | ✓ |  |  |  |
Knowledge Distillation |  |  |  |  |  |  | ✓ | ✓ |  | ✓ |  | ✓ |  |
Multi-Objective Optimization |  |  |  |  |  |  |  | ✓ | ✓ | ✓ |  |  |  |
Task Relationship Learning |
Task Grouping | ✓ | ✓ | ✓ | ✓ |  |  | ✓ | ✓ | ✓ |  |  |  | ✓ | ✓
Relationships Transfer | ✓ |  |  | ✓ |  |  | ✓ | ✓ | ✓ |  |  |  |  |
Task Embeddings |  |  |  |  |  |  | ✓ | ✓ |  |  |  |  |  |
Connection to Learning Paradigm |
Reinforcement Learning |  | ✓ | ✓ |  | ✓ | ✓ |  | ✓ |  |  |  |  |  | ✓
Transfer Learning |  | ✓ | ✓ | ✓ |  |  |  |  |  |  |  |  |  |
Meta-Learning |  |  |  |  |  |  |  | ✓ |  |  | ✓ |  |  | ✓
Online Learning | ✓ | ✓ | ✓ |  |  |  |  |  |  |  |  |  |  |
Continual Learning |  |  |  |  |  |  |  |  |  |  |  |  |  |
Application Domain |
Natural Language Processing | ✓ | ✓ | ✓ | ✓ | ✓ |  | ✓ | ✓ |  | ✓ | ✓ | ✓ |  | ✓
Computer Vision | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  | ✓ | ✓ |  | ✓ |  |  |

The idea of MTL was explored in many research studies. In this section, we
provide an overview of related MTL surveys, address various aspects of MTL,
and list them along with their corresponding surveys in Table 1. 222The
broader version of the table is provided in Appendix Table 3. In the rest of
the section, we just mention related surveys and go over individual MTL
aspects (shown in bold) in more detail.333The broader version of the section
is provided in Appendix A.1.

Many application domains were studied in previous work, ranging from surveys
covering multiple domains (Ruder, 2017; Zhang & Yang, 2017; 2018; Thung & Wee,
2018; Vafaeikia et al., 2020; Crawshaw, 2020; Upadhyay et al., 2021;
Abhadiomhen et al., 2022), to those dedicated to a specific domain, such as
computer vision (Vandenhende et al., 2021) or natural language processing
(Zhou, 2019; Worsham & Kalita, 2020; Chen et al., 2021; Samant et al., 2022;
Zhang et al., 2023). Both traditional ML and deep learning computational
models were studied. The traditional ML was discussed primarily in older
studies, while deep learning was presented in all but one study.

MTL architectures were widely discussed in previous work. Hard and soft
parameter sharing (Ruder, 2017) was the most used architecture taxonomy, but
recent surveys refined the taxonomies for more precise categorization
(Crawshaw, 2020; Chen et al., 2021). Next, some MTL architectures were
categorized as learning-to-share (Ruder et al., 2017), which offers a more
adaptive solution by learning how to share parameters between tasks, rather
than having the sharing defined a priori. Additionally, some MTL architectures
were categorized as universal models, handling multiple modalities, domains,
and tasks using a single model (Kaiser et al., 2017; Pramanik et al., 2019).

Optimization techniques for MTL architectures were also widely discussed,
while loss weighting was the most common approach to mitigate MTL challenges.
Techniques include loss weighting by uncertainty (Kendall et al., 2018),
learning speed (Liu et al., 2019a; Zheng et al., 2019), or performance (Guo et
al., 2018; Jean et al., 2019), among others. Next, and closely related to
weighting task losses, is a task scheduling problem that involves choosing
tasks to train on at each step. Many techniques were used, from simple ones
that employ uniform or proportional task sampling, to the more complicated
ones, such as annealed sampling (Stickland & Murray, 2019) or approaches based
on active learning (Pilault et al., 2020). Finally, regularization approaches
(Long et al., 2017; Lee et al., 2018; Pascal et al., 2021), gradient
modulation (Lopez-Paz & Ranzato, 2017; Sinha et al., 2018), knowledge
distillation (Clark et al., 2019b), and multi-objective optimization (Lin et
al., 2019) were also applied to optimize MTL models.

Task relationship learning in MTL focuses on learning explicit representations
of tasks or relationships between them, and typically three categories of
methods were used. First, task grouping aims to divide a set of tasks into
groups to maximize knowledge sharing during joint training (Standley et al.,
2020). Second, transfer relationship learning determines when transferring
knowledge from one task to another will be beneficial for joint learning
(Zamir et al., 2018). Finally, task embedding methods aim to learn task
embedding space (Vu et al., 2020).

Previous works made connections to other learning paradigms, including
reinforcement learning, transfer learning, meta-learning, active and online
learning. To the best of our knowledge, there had been no prior work
systematically investigating connections between MTL and CL. We believe that a
connection between MTL and CL represents a promising research direction, as we
will motivate the need for this connection in Section 4.4.

###  2.2 ML Lifecycle and ML Systems

The unparalleled growth of advancements in ML methods in recent years, with
applications in NLP, computer vision, and others, has increased the complexity
of building ML systems that need to address the requirements of an ML
lifecycle. Previous works reviewed the challenges of such systems, often
defining ML lifecycle phases such as data management, model learning, and
model deployment, to study systematically the workings of ML systems and
identify challenges within and across phases (Vartak & Madden, 2018; Ashmore
et al., 2021; Paleyes et al., 2022; Huyen, 2022). For example, Vartak & Madden
(2018) defined ML lifecycle phases and analyzed model management challenges.
Furthermore, Ashmore et al. (2021) discussed assurance of ML for each phase,
while Paleyes et al. (2022) reviewed practitioners’ challenges at each phase
of an ML model deployment workflow in a broader scope than previous surveys.
In the rest of this section, we provide a few examples of challenges that
occur during typical phases of the ML lifecycle.

There are many challenges that occur during different phases of the ML
lifecycle. For example, data management is typically the early phase of the ML
lifecycle with associated challenges such as data collection and preprocessing
(Polyzotis et al., 2018; Sambasivan et al., 2021; Whang et al., 2023).
Subsequently, model learning and verification phases take place, presenting
challenges such as selecting (Ding et al., 2018) and training (Sun, 2020) a
model, and determining the most effective method for verifying it (Bernardi et
al., 2019; Schröder & Schulz, 2022), respectively. Then, a model deployment
phase takes place with challenges such as model integration (Sculley et al.,
2015; Renggli et al., 2019) into production. Finally, the monitoring phase
with challenges such as continuous model performance monitoring (Schröder &
Schulz, 2022) and updating a model over time (Ditzler et al., 2015;
Abdelkader, 2020).

Some challenges can impact several phases of the ML lifecycle, such as
collaboration among diverse teams and roles, including software and data
engineers, data scientists, and other stakeholders (Takeuchi & Yamamoto, 2020;
Nahar et al., 2022; Pei et al., 2022; Yang et al., 2022). Furthermore, there
are challenges of bias, fairness, and accountability in ethics (Mehrabi et
al., 2021; Kim & Doshi-Velez, 2021), various regulations set by law (Marchant,
2011; Politou et al., 2018) and adversarial attacks in security (Ren et al.,
2020; Rosenberg et al., 2021), among others.

Previous works addressed MTL aspects to varying degrees in specific phases,
either in a straightforward manner or indirectly. However, a systematic
discussion of the potential benefits of using MTL approaches to alleviate
challenges across different phases of the ML lifecycle has not been conducted.

###  2.3 Continual Learning

CL incrementally learns a sequence of tasks, with a goal to progressively
expand acquired knowledge and utilize it for subsequent learning (Chen & Liu,
2018). CL aims to overcome catastrophic forgetting (CF) and facilitate
knowledge transfer (KT) across tasks, where CF is the degradation of
performance on previous tasks when learning new ones, and KT is the ability to
apply knowledge from past tasks to new tasks (Ke & Liu, 2022). Previous
reviews on CL (Hsu et al., 2018; De Lange et al., 2021) categorized CL
settings based on the marginal output and input distributions $P(Y^{(t)})$ and
$P(X^{(t)})$ of a task $t$, with $P(X^{(t)})\neq P(X^{(t+1)})$. First, class
incremental learning is characterized by an expanding output space with
observed class labels such that ${Y^{(t)}}\subset{Y^{(t+1)}}$ and
$P(Y^{(t)})\neq P(Y^{(t+1)})$. Second, task incremental learning (TIL),
requires a task label $t$ to identify the separate output nodes $Y^{(t)}$ for
the current task $t$, where ${Y^{(t)}}\neq{Y^{(t+1)}}$. Lastly, incremental
domain learning defines tasks with equal class labels and probability
distributions, ${Y^{(t)}}={Y^{(t+1)}}$, and $P(Y^{(t)})=P(Y^{(t+1)})$.

CL approaches were also categorized into three main categories based on how
task-specific information is stored and utilized during the incremental
learning process. First, replay methods store samples in raw format or
generate pseudo-samples with a generative model, replaying them while learning
a new task to mitigate forgetting and prevent previous task interference.
Second, regularization-based methods, on the other hand, avoid storing raw
inputs and reduce memory requirements by introducing an extra regularization
term in the loss function to consolidate prior knowledge while learning new
data. Third, parameter isolation methods allocate different model parameters
to each task, either by adding new task-specific branches or masking out
previous task parts, to prevent forgetting and maintain task-specific
knowledge. We refer readers to Ke & Liu (2022) for more refined CL taxonomy
and details in NLP.

In CL, MTL was usually used as an upper-bound baseline which can use all of
the data from all of the tasks simultaneously (De Lange et al., 2021; Ke &
Liu, 2022). Since CL and MTL work in different learning settings, few works
tried to connect the two paradigms. Sun et al. (2020) presented a continual
pre-training framework named ERNIE $2.0$ which incrementally builds pre-
training tasks and then learns pre-trained models on these constructed tasks
via continual multi-task learning. Subsequently, ERNIE $2.0$ was tested
against CL and MTL pre-training approaches to evaluate the impact on
abstractive text summarization task but performed similarly to other
approaches (Kirstein et al., 2022). In Section 4.4, we motivate further
research on combining CL and MTL approaches, as pre-training approaches are
not sufficient for handling distribution shifts and adjusting models for new
business requirements in real-world scenarios.

##  3 Multi-Task Learning Approaches

###  3.1 Taxonomy

There were various MTL taxonomies covered in the surveys presented in Section
2.1. Ruder (2017) distinguishes between a hard and soft parameter sharing,
which proved to be an influential taxonomy, since it was used in later works
as well. Zhang & Yang (2018) defines three categories of multi-task supervised
learning – feature-, parameter-, and instance-based. Vandenhende et al. (2021)
distinguishes between encoder-focused and decoder-focused architectures. Chen
et al. (2021) discusses parallel, hierarchical, modular, and generative
adversarial architectures. We categorized transformer-based MTL approaches
into $3$ main categories based on differences in architectures: (1) Fully-
Shared Encoder, (2) Adapters, and (3) Hypernetworks (Figure 1).444In Appendix
B.1, we give a brief overview of prompt engineering approaches as a fourth
category. However, we do not include it in the main paper due to its need for
a large number of parameters to perform well, making it inaccessible to most
practitioners.

Figure 1: Simplified overview of MTL architectures. Sub-figure a) represents a
fully-shared encoder, b) an adapter, and c) a hypernetwork. Blue components
are trained jointly by all the tasks, green ones are task-specific, and grey
ones are kept frozen. A dotted adapter component suggests possible adapter
insertion positions.

###  3.2 MTL Approaches Overview

####  3.2.1 Fully-Shared Encoder

A simple and intuitive approach to MTL is to have a clear division between
shared and task-specific parameters. In such an approach, there is a shared
transformer-based encoder in lower layers, while the top layers consist of
different, task-specific layers (heads). One such approach, MT-DNN (Liu et
al., 2019b), batches all the GLUE tasks (Wang et al., 2018) together and
updates the model accordingly. The shared encoder is updated for all the
instances, while the task-specific heads are updated only for the instances of
a task that they are specific for. There are several downsides to this MT-DNN
approach. First, task interference is not taken into account and the authors
simply hope that the tasks would interact well, although some of them are in
different domains. Next, proportional random sampling is used, which could
lead to underfitting on low-resource datasets. Finally, the loss is calculated
in three different ways (for classification, regression, and ranking), and as
a result, it has different scales. Nevertheless, all the loss functions are
weighted equally. Notwithstanding these observations, their model outperforms
fine-tuning a different BERT (Devlin et al., 2019) model on most of the tasks.
Additionally, they tried fine-tuning this multi-task model further on each
task separately after training jointly on all the tasks, producing $N$ models
for $N$ tasks. That again gave an improvement and a state-of-the-art
performance at the time. However, an obvious downside is having a different
model for each task.

Another shared-encoder approach is pre-finetuning. Muppet (Aghajanyan et al.,
2021) shared an encoder in an MTL on 46 diverse datasets. Heterogeneous
batches have proved beneficial in handling noisy gradients from different
tasks. Furthermore, to have stable training, the data-point loss was divided
by log$(n)$, where $n$ denotes the cardinality of the label set for the
associated task. They maintained a natural distribution of datasets as other
approaches led to degraded performance. The authors found a threshold of
around 15 tasks, below which downstream fine-tuning performance is degraded,
and above which the performance improves linearly in the number of pre-
finetuning tasks. A similar approach with added decoder, EXT5 (Aribandi et
al., 2021), extended the mixture to 107 supervised tasks, formatted them for
encoder-decoder architectures, and performed pre-finetuning along with
unsupervised T5’s C4 span denoising (Raffel et al., 2020). Their mixture of
tasks also included NLP applications such as reading comprehension, closed-
book question answering, commonsense reasoning, dialogue, and summarization,
among others. This showed that encoder-decoder models like T5 are capable of
solving a wider range of NLP applications than encoder ones. However, task-
specific models still achieve a better performance than general ones (Chung et
al., 2022).

####  3.2.2 Adapters

Before being used in NLP, adapter residual modules were first introduced for
the visual domain (Rebuffi et al., 2017). Adapters are small, task-specific
modules that are typically inserted within network layers, but can also be
injected in parallel to them. In this survey, the network is always a
transformer-based architecture. Compared to transformers’ sizes, they add a
negligible number of parameters per task. The parameters of the original
network remain frozen unless stated otherwise, resulting in a high degree of
parameter sharing and a small number of trainable parameters. Consequently,
adapters for new tasks can be easily added without retraining the transformer
or other adapters. They learn task-specific layer-wise representation, are
small, scalable, and shareable, have modular representations and a non-
interfering composition of information (Pfeiffer et al., 2020b). Since the
respective adapters were trained separately, the necessity of sampling
heuristics due to skewed data set sizes no longer arises (Pfeiffer et al.,
2020b).

AdapterHub. AdapterHub (Pfeiffer et al., 2020b) is a framework that allows
dynamic usage of pre-trained adapters for different tasks and
languages.555https://adapterhub.ml The framework is built on top of the
HuggingFace Transformers library and enables quick and easy adaptation of
state-of-the-art pre-trained models. It allows for efficient parameter sharing
between tasks by training many task- and language-specific adapters, which can
be exchanged and combined post-hoc. One can choose to stack the adapters on
top of each other, combine them with attention (Pfeiffer et al., 2020a), or
replace them dynamically. Downloading, sharing, and training adapters require
minimal changes in the training scripts.

Bottleneck adapters. In AdapterHub’s documentation, three different bottleneck
adapter approaches were mentioned. Adapters can be inserted after both Multi-
Head Attention (MHA) and Feed-Forward (FF) block (Houlsby et al., 2019), only
after the FF block (Pfeiffer et al., 2020c), or in parallel to the Transformer
layers (He et al., 2021). Bottleneck adapters consist of a down-projection,
non-linearity (typically ReLU), and an up-projection back to the original
size. Residual connection is used, and layer normalization is applied
afterward.

Language adapters. In the MAD-X framework (Pfeiffer et al., 2020c), the
authors train (1) language adapters via masked language modeling (MLM) on
unlabelled target language data, and (2) task adapters by optimizing a target
task on labeled data in a source language with the most training data. Then,
adapters are stacked, allowing for a zero-shot cross-lingual transfer by
substituting the target language adapter at inference. Invertible adapters are
introduced to tackle the mismatch between the pre-trained model’s multilingual
vocabulary and target language vocabulary. Consequently, language adapters
could be useful when one had already trained a task adapter for a specific
task and now needs to perform inference for the same task, but on new data
from a different language.

Other. Projected Attention Layer (PAL) (Stickland & Murray, 2019) is a low-
dimensional multi-head attention layer added in parallel to the transformer
layers. Multi-head attention is applied on a down-projected input, after which
an up-projection to an original dimension is applied. These down- and up-
projection matrices are shared among layers, but not among tasks. The authors
fine-tune a pre-trained encoder along the PALs. This has downsides: (1)
forgetting of pre-trained knowledge is possible, (2) access to all the tasks
at training time is required, and (3) adding new tasks requires complete joint
retraining. Thus, this approach misses many of the characteristics of an
adapter.

AdapterFusion (Pfeiffer et al., 2020a) introduces a knowledge composition
phase, in which the previously trained adapters are combined. This approach
uses multiple adapters to maximize knowledge transfer between tasks without
suffering from the MTL drawbacks, such as catastrophic forgetting (Serra et
al., 2018) or task interference (Wu et al., 2020). It introduces a new set of
weights that learn to combine the adapters as a dynamic function of the target
task data by using attention. This shows its greatest downside – AdapterFusion
is trained for one task only.

Hu et al. (2021) argue that the original adapter bottleneck design (Houlsby et
al., 2019) introduces inference latency because the adapters are processed
sequentially, whereas large language models (LLMs) rely on hardware
parallelism. Their approach, LoRA (Low Rank Approximation) modifies attention
weights of query and value projection matrices by introducing trainable low-
rank decomposition matrices in parallel to the original computation. This
reduces inference latency, as the decomposition matrices can be merged with
the pre-trained weights for faster inference.

Figure 2: ML lifecycle phases. Image is taken from
Huyen (2022).

Table 2: ML Lifecycle phases and corresponding challenges.

ML Lifecycle Phase | Challenges
---|---
Project Scoping | Initial Requirements
Data Engineering | Labeling of Large Volumes of Data
| Cost of Annotators and Experts
| Lack of High-Variance Data
Model Development | Model Complexity
| Resource-Constrained Environments
| Computational Cost
| Environmental Impact
Deployment | Ease of integration
Monitoring | Distribution Shift
Business Analysis | New Requirements

####  3.2.3 Hypernetworks

A hypernetwork is a network that generates weights of another network (Ha et
al., 2016). This approach can alleviate a downside of adapters, which is a
lack of knowledge sharing. Hypernetwork allows sharing of knowledge across
tasks while adapting to individual tasks through task-specific parameter
generation.

CA-MTL (Pilault et al., 2020) modularizes a pre-trained network by either
adding task-conditioned layers or changing the pre-trained weights using the
task embedding. Their task-conditioned transformer-based network has four
components: (1) conditional attention, (2) conditional alignment, (3)
conditional layer normalization, and (4) conditional bottleneck. In (1), they
use block-diagonal conditional attention which allows the attention to account
for task-specific biases. Component (2) aligns the data of diverse tasks. In
(3), they adjust layer normalization statistics for specific tasks. Finally,
(4) facilitates weight sharing and boosts task-specific information flow from
lower layers. In addition, they use multi-task uncertainty sampling. This
favors tasks with the highest uncertainty by sampling a task whenever its
entropy increases, helping to avoid catastrophic forgetting. When introducing
a new task, they claim that only a new linear decoder head and a new task
embedding vector need to be added to re-modulate existing weights.

HyperFormer++ (Mahabadi et al., 2021) uses hypernetworks to generate the
weights of adapter and layer normalization parameters. These hypernetworks
condition on a task embedding, adapter position (after MHA or FF sub-layer),
and layer id within the T5 model (Raffel et al., 2020). During training, they
sample the tasks using temperature-based sampling. They state that for each
new task, their model only requires learning an additional task embedding.

HyperGrid (Tay et al., 2020) leverages a grid-wise decomposable hyper
projection structure which helps specialize regions in weight matrices for
different tasks. To construct the proposed hypernetwork, their method learns
the interactions and composition between a global, task-agnostic state and a
local, task-specific state. They equip position-wise FF sub-layers of a
Transformer with HyperGrid. They initialize a T5 model from a pre-trained
checkpoint and add additional parameters that are fine-tuned along the rest of
the network. The authors of the paper did not mention anything specific
regarding the ability to add a new task without re-training, as it appears to
be non-trivial.

##  4 MTL From ML Lifecycle Point of View

In this section, we discuss the challenges and opportunities of incorporating
into ML production systems MTL approaches instead of using multiple single-
task counterparts. Motivated by previous reviews on ML lifecycle (see Section
2.2), we define ML lifecycle phases to discuss challenges and opportunities in
a systematic manner.

Following Huyen (2022), we define six ML lifecycle phases: (1) Project
Scoping, (2) Data Engineering, (3) Model Development, (4) Deployment, (5)
Monitoring, and (6) Bussiness Analysis (Figure 2). Next, we mostly focus on
challenges that were discussed in a broader scope in Paleyes et al. (2022),
while we argue how MTL can alleviate them. Challenges per each phase of the ML
lifecycle are listed in Table 2. When discussing the challenges and
opportunities of using MTL approaches, we compare them to corresponding
single-task model solutions.

In the rest of the section, we first discuss data engineering and model
development phases in isolation. Then, we discuss an ML model updating problem
by indicating how certain aspects of the problem pose different challenges in
different phases of the ML lifecycle.

###  4.1 Data Engineering

The first phase we discuss is data engineering. This phase focuses on
preparing data that is needed to train a machine learning model, while we have
a particular interest in challenges related to the lack of labeled data (see
Table 2).

Challenges. The need for data augmentation can arise from various factors,
with one of the most problematic being the lack of labels in the data,
especially in real-world applications where labeled data may be scarce. It is
a common practice in NLP production systems to utilize pre-trained
transformer-based language models by fine-tuning them for specific downstream
tasks. However, if there is a significant gap between a downstream task and
the pre-training objectives, a larger amount of labeled data may still be
required to achieve the target performance (Wang et al., 2020). Obtaining this
data involves costly and time-consuming involvement of annotators and domain
experts. Additionally, the absence of high-variance data results in a model
that is unable to generalize well, such as adapting language models to low-
resource languages (Clark et al., 2019a).

Opportunities. The challenges posed by the lack of labeled data can be
alleviated using MTL approaches. For example, if a set of single-task models
is being used in a production system, training an MTL model instead can help
alleviate data sparsity by jointly learning to solve related tasks (Section
3.2.1). The benefits of MTL have been previously discussed in Caruana (1997);
Ruder (2017), including its ability to increase data-efficiency. First,
different tasks transfer different knowledge aspects to each other, enhancing
the representation’s ability to express the input text, which can be
beneficial for tasks with low-resource datasets (Section 3.2.1). However, some
MTL approaches may underperform in resource-constrained environments due to
inadequate optimization choices (Section 3.2.1). Additionally, the presence of
different noise patterns in each task acts as an implicit data augmentation
method, effectively increasing the sample size used for training, and leading
to a robust model with more general representations (Ruder, 2017). Finally,
using pre-finetuning (Section 3.2.1) could reduce convergence time, saving
computational resources.

###  4.2 Model Development

In the model development phase, we focus on two groups of challenges. The
first group refers to the _model selection_ problem, including issues related
to model complexity and resource constraints. The second group of challenges
is related to problems in _model training_ , such as the computational cost of
the training procedure and its impact on the environment. We refer readers to
(Gupta & Agrawal, 2022) for an overview of methods for efficient models in
text.

Model Selection Challenges. When selecting a model to handle tasks that end-
users are interested in, practitioners often face a dilemma regarding the
trade-off between model complexity and performance. Typically, complex models
have better performance, but they come with the risk of over-complicating the
design in the first place, leading to a longer development time and deployment
failure (Haldar et al., 2019). Furthermore, they may not be practical to use
in resource-constrained environments where they require high computational and
memory resources.

Model Selection Opportunities. MTL architectures, presented in Section 3.2,
have properties that can alleviate challenges related to the trade-off between
model complexity and performance. For example, consider replacing N single-
task models with a single MTL shared encoder model. The MTL model will have
close to N times smaller memory footprint, as the number of task-specific
parameters is negligible compared to the number of shared parameters. This
reduction results in a better fit to memory-constrained environments while
only having slightly worse performance than the single-task counterparts.
Similarly, saving $N$ adapters or a single hypernetwork is much more memory-
efficient than saving N single-task models.

Model Training Challenges. The training of machine learning models presents
several challenges that must be addressed by practitioners. One of the major
challenges is the high economic cost associated with training, which is due to
the computational resources required. In the field of natural language
processing, the cost of model training continues to rise, even as the cost of
individual floating-point operations decreases, due to factors such as the
growth in training dataset size, number of model parameters, and number of
operations involved in the training process (Sharir et al., 2020). The
training process also has a significant impact on the environment, leading to
increased energy consumption and greenhouse gas emissions (Strubell et al.,
2020). These challenges emphasize the need to address the economic and
environmental implications of training machine learning models.

Model Training Opportunities. Computational resource challenges could be
alleviated in some aspects by using MTL approaches to reduce the cost of model
training. First, in certain cases, smaller dataset sizes can be used due to
pre-finetuning or knowledge transfer between related tasks during joint
training of the MTL model, leading to more data-efficient training. Second,
the joint model is more parameter-efficient, resulting in a significant
reduction in the number of parameters required for multiple single-task
models.

Selection, Training, and Inference Trade-Off. The number of floating-point
operations is not reduced in some cases, and depends on the choice of MTL
architecture and the nature of the tasks. Different task-specific heads
require different inputs if tasks belong to different domains or have
different input encodings. However, if tasks belong to the same domain and
have the same input encoding, part of the computation can be shared among the
task-specific heads. For example, in a fully-shared encoder (Section 3.2.1),
the computation of the full encoder can be shared, while the computation in
the task-specific heads is negligible. Similarly goes for adapters (Section
3.2.2) – the large majority of the computation is shared, and only the
adapters are then dynamically plugged for performing different tasks on the
same input. Namely, the benefit of using adapters is a small number of
trainable parameters, thanks to a frozen encoder, which results in a faster
gradient back-propagation. A forward pass, on the other hand, takes more time
compared to an encoder with no adapters. Thus, when tasks don’t share the
datasets, using adapters results in a longer inference time compared to
single-task counterparts. AdapterFusion inference is even slower, as an input
must pass through all the available adapters. LoRA solves the inference
latency problem by using parameter composition.

###  4.3 Model Deployment

In the model deployment phase, our focus is on simplifying the integration of
trained ML models with existing ML systems that are running in production,
with a particular emphasis on straightforward implementation, seamless
collaboration, and ease of maintenance.

Challenges. The first challenge in model deployment is preparing the developed
model for use in a production environment. The initial versions of models are
often developed by stakeholders (e.g., ML researchers) who are different from
those responsible for deploying them into production (e.g., ML and DevOps
engineers). This means that the model code needs to be adapted to meet the
requirements of the operational production environment. Those requirements are
typically more strict and different from those available during the
development phase, so it is important to address operational aspects such as
scalability, security, and reliability. Hence, each additional model adds
complexity to the process, both for the stakeholders involved and for the ML
system infrastructure and computational resources in place. Another challenge
during model integration is incorporating the model into real-data processing
pipelines in production, whether it is for batch offline processes or handling
real-time user requests. During model training, the researcher often uses
preprocessed and clean datasets. However, when integrating the model into
production, it will be integrated into existing data pipelines. The more
complex the model’s input data requirements, or the more models that are used,
the more complex the data processing pipelines will need to be. Adapting the
models and existing data pipelines often requires collaboration between
different stakeholders or teams responsible for different parts of the ML
system and/or ML lifecycle phase. All of these factors directly impact the
increasing costs of maintenance, operations, support, and infrastructure.

Opportunities. MTL approach was presented by the team from Pinterest, where
the authors trained a universal set of image embeddings for three different
models, which simplified their deployment pipelines and improved performance
on individual tasks (Zhai et al., 2019). The key benefit is that the use of
MTL can reduce the number of parameters needed to support different models
required to solve a problem. This leads to smaller task-specific models and
less code to be adapted to the production environment, reducing the overhead
of potential changes in existing data pipelines. Reusing data, code, and
models can save time and simplify the model deployment process. Next, the
modular design of MTL architectures makes it easier to work with by enabling
code reuse. For example, the fully-shared encoder architecture (described in
Section 3.2.1) reuses the encoder between multiple task-specific heads.
Moreover, freezing the shared encoder would allow for working on separate
tasks independently and simultaneously, making cross-team collaboration easier
and more efficient. This idea was used in the HydraNet MTL model by the Tesla
AI team (Karpathy, 2021). However, freezing an encoder and updating only the
task-specific heads may decrease performance. Additionally, the concept behind
the AdapterHub (described in Section 3.2.2) was designed to allow users to
choose from a set of adapter modules, combine them in their preferred way, and
insert or replace them dynamically into state-of-the-art pre-trained models.
To conclude, the modular nature of MTL architectures has a positive impact on
making the incorporation of these architectures into ML software easier,
making it simpler to develop, collaborate, configure, and integrate into
deployment processes.

###  4.4 Model Updating Through Multiple ML Lifecycle Phases

Often it is necessary to update the ML model regularly after it has been
deployed and is running in production, in order to keep it aligned with the
most current changes in data and environment. The need for updating models is
one of the most important requirements of ML production systems (Pacheco et
al., 2018; Abdelkader, 2020; Lakshmanan et al., 2020; Paleyes et al., 2022;
Huyen, 2022; Wu & Xie, 2022; Nahar et al., 2022). In this section, we discuss
the challenges of model updating and how MTL approaches can alleviate these
challenges. Challenges of different phases of the ML lifecycle can trigger the
need for model updates. First, we identify these occurrences and discuss the
actions they trigger. Then, we discuss how MTL approaches can alleviate these
challenges and point out the current limitations of these approaches.

Model Updating Challenges. A distribution shift is one of the common reasons
for the need to update models. Distribution shift refers to changes observed
in the joint distribution of the input and output variables of an ML model
(Ditzler et al., 2015). Two problems must be solved to address the
distribution shift effectively. First, in the _monitoring phase_ , a mechanism
must be in place to detect changes in distribution or a drop in key
performance indicators, which will signal the need for an ML model update.
Second, in the _model development phase_ , there must be a way to continuously
learn and update the model in response to signals from the monitoring phase.

New business requirements often arise, requiring the model to have new
capabilities in addition to the existing ones. For example, a named entity
recognition model trained for 10 entity labels in news articles may require 5
new labels after 3 months of use. New requirements are introduced during the
_business analysis phase_ , which also triggers the need for an ML model
update. Unlike the initial requirements in the _project scoping phase_ ,
adding new requirements should be possible without re-training the full model.

Periodic or scheduled re-training and continual learning are the most common
approaches for adapting models to new data and requirements. Periodic re-
training refers to the process of re-training a model at a predetermined
interval, regardless of whether there have been any changes to the data or the
environment. The frequency of updating is determined beforehand and is
typically based on the amount of data and the desired level of model
performance. Striking a balance between updating the model regularly to
maintain good performance and avoiding over-updating the model to minimize
computational costs is crucial. It is important to note that different models
may require different re-training schedules, making it a situation-dependent
problem. Finding the optimal re-training schedule requires careful
consideration of the specific requirements and circumstances of each ML
system. Continual learning, on the other hand, refers to the ability of an ML
model to adapt to changes in the data and environment over time. This approach
involves continuously monitoring the performance of the model and updating it
as needed to ensure it remains accurate and up-to-date. The frequency of model
updates in continual learning is determined dynamically based on the changes
observed in the data and environment.

Model Updating Opportunities. Following Huyen (2022), we differentiate two
types of model updates: _data iteration_ and _model iteration_. Data iteration
refers to updating the model with new data while keeping the model
architecture and features the same, while model iteration refers to adding new
features to an existing model architecture or changing the model architecture
itself. To discuss the challenges and opportunities of MTL approaches for
model updating, we consider two scenarios. First, periodic re-training is done
every 6 or 12 months, with the goal of training each model to perform
optimally using all available data. Due to the high economic cost, we assume
this cannot be done more frequently. Second, between periodic retrains, there
may be situations where data iteration is necessary due to a distribution
shift, or model iteration is required to extend model capabilities in response
to a new business requirement.

We believe that incorporating MTL approaches into the update scenario would be
practical, in addition to the benefits discussed in Sections 4.1-4.3. MTL is
particularly well-suited for periodic re-training scenarios where the
objective is to obtain a single best-performing model on all tasks using all
available data, instead of training individual models for each task. However,
the challenge lies in how MTL approaches can manage the second scenario where
the MTL model must learn new tasks or domains sequentially over time. As a
result, the same model must be able to learn in both MTL and CL settings as
needed, and we refer to this setting as Continual MTL (CMTL).

Continual MTL. MTL and CL models both learn multiple tasks, however, MTL
learns them simultaneously, while CL learns them incrementally. As mentioned
in Section 2.3, the authors of Sun et al. (2020) combined CL and MTL to
further improve pre-training. Although the approach enhanced performance on
downstream tasks, it does not address the real-world scenario in which an MTL
model should be updated to support new downstream tasks or handle distribution
shifts. Moreover, the total number of sequential tasks must be known a priori
for the algorithm to determine an efficient training schedule. We believe that
the similarities between MTL and CL architectures could enable the
construction of a CMTL model. For example, most MTL architectures in Section
3.2 are transformer-based MTL architectures with task-specific heads, which
resemble the parameter-isolation architectures for the TIL setting in CL
(Section 2.3). In TIL, the task identifier is available during both training
and testing and is used to identify task-specific parameters in multi-headed
architectures, similar to MTL architectures (Ke & Liu, 2022). This similarity
can be seen in the use of adapter architectures (Section 3.2.2) in both MTL
(Stickland & Murray, 2019; Pfeiffer et al., 2020c; He et al., 2021) and CL (Ke
et al., 2021a; b), as well as in the use of hypernetworks (Section 3.2.3) in
both MTL (Pilault et al., 2020; Mahabadi et al., 2021) and CL (Von Oswald et
al., 2019; Jin et al., 2021).

Figure 3: (A). Adapter-BERT (Houlsby et al., 2019) uses adapters in a
transformer layer (Vaswani et al., 2017). Adapters are 2-layer networks with
skip-connections, added twice per layer. Only adapters (yellow) and layer norm
(green) are trainable while other modules (grey) are frozen. (B). B-CL
replaces adapters with CLA, containing a knowledge-sharing module (KSM) and
task-specific module (TSM), both with skip-connections. Image and modified
caption are taken from (Ke et al., 2021b).

For example, if we consider the BERT layer in Figure 3, we can observe that
the adapter layers for the MTL approach in Figure 3(A) are in the same
positions as the CL adapters in Figure 3(B) (Ke et al., 2021b). To switch from
MTL to CL we only need to change adapters. We believe that this similarity
could be a promising direction for further research on the development of a
CMTL model, but proper evaluation will not be possible without a good
benchmark.

Finally, we believe that a benchmark that accurately represents the challenges
of real-world systems could be beneficial for both researchers and
practitioners to evaluate CMTL models effectively. To accomplish this, we
propose combining and temporarily ordering tasks to simulate periodic re-
training and CL scenarios. The benchmark can be further refined to reflect
varying frequencies of periodic re-training, as well as the frequency of
incoming tasks and distribution shifts between re-training periods. Variation
in these scenarios could better represent real-world situations, where a
simpler CMTL model might be sufficient in some cases, while more advanced CMTL
models would be needed to handle environmental changes in other situations.

##  5 Conclusion

In this paper, we reviewed transformer-based MTL approaches in NLP and
explored the challenges and opportunities of those approaches in the context
of the ML lifecycle. We discussed how MTL can be a possible solution
addressing some of the key challenges in data engineering, model development,
deployment, and monitoring phases of the ML lifecycle, as compared to using
multiple single-task models.

We also discussed the opportunities for applying MTL to alleviate challenges
regarding model updating due to distribution shifts or evolving real-world
requirements, where the ability to learn multiple tasks simultaneously can be
leveraged to periodically update the model in response to changes in data and
environment. However, we also acknowledged the limitations of current MTL
approaches to handle updates sequentially, where CL is required. To address
this, we proposed the concept of CMTL, which aims to combine the benefits of
MTL and CL in a single model. We motivated creating a benchmark for the proper
evaluation of CMTL models, a benchmark that better represents the challenges
of production systems that can guide the development of those models.

In conclusion, MTL approaches offer many opportunities for improving the
efficiency and performance of ML systems in different phases of the typical ML
lifecycle. We believe that MTL will become an important part of practitioners’
toolbox seeking to address the challenges in their ML systems as highlighted
in this survey.

## References

  * Abdelkader (2020) Hala Abdelkader.  Towards robust production machine learning systems: Managing dataset shift.  In _2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE)_ , pp. 1164–1166. IEEE, 2020.
  * Abhadiomhen et al. (2022) Stanley Ebhohimhen Abhadiomhen, Royransom Chimela Nzeh, Ernest Domanaanmwi Ganaa, Honour Chika Nwagwu, George Emeka Okereke, and Sidheswar Routray.  Supervised shallow multi-task learning: analysis of methods.  _Neural Processing Letters_ , 54(3):2491–2508, 2022.
  * Aghajanyan et al. (2021) Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta.  Muppet: Massive multi-task representations with pre-finetuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 5799–5811, 2021.
  * Aribandi et al. (2021) Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al.  Ext5: Towards extreme multi-task scaling for transfer learning.  In _International Conference on Learning Representations_ , 2021.
  * Ashmore et al. (2021) Rob Ashmore, Radu Calinescu, and Colin Paterson.  Assuring the machine learning lifecycle: Desiderata, methods, and challenges.  _ACM Computing Surveys (CSUR)_ , 54(5):1–39, 2021\.
  * Bao et al. (2020) Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, and Hsiao-Wuen Hon.  UniLMv2: Pseudo-masked language models for unified language model pre-training.  In Hal Daumé III and Aarti Singh (eds.), _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pp. 642–652. PMLR, 13–18 Jul 2020.  URL https://proceedings.mlr.press/v119/bao20a.html.
  * Bernardi et al. (2019) Lucas Bernardi, Themistoklis Mavridis, and Pablo Estevez.  150 successful machine learning models: 6 lessons learned at booking. com.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, pp. 1743–1751, 2019.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  _CoRR_ , abs/2005.14165, 2020.  URL https://arxiv.org/abs/2005.14165.
  * Caruana (1997) Rich Caruana.  Multitask learning.  _Machine learning_ , 28(1):41–75, 1997.
  * Chen et al. (2021) Shijie Chen, Yu Zhang, and Qiang Yang.  Multi-task learning in natural language processing: An overview.  _arXiv preprint arXiv:2109.09138_ , 2021.
  * Chen & Liu (2018) Zhiyuan Chen and Bing Liu.  Lifelong machine learning.  _Synthesis Lectures on Artificial Intelligence and Machine Learning_ , 12(3):1–207, 2018.
  * Chui et al. (2022) Michael Chui, Bryce Hall, Helen Mayhew, and Alex Singla.  The state of ai in 2022–and a half decade in review, Dec 2022.  URL https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2022-and-a-half-decade-in-review.  Accessed: 2023-01-15.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Clark et al. (2019a) Christopher Clark, Mark Yatskar, and Luke Zettlemoyer.  Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 4069–4082, Hong Kong, China, November 2019a. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1418.  URL https://aclanthology.org/D19-1418.
  * Clark et al. (2019b) Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le.  Bam! born-again multi-task networks for natural language understanding.  _CoRR_ , abs/1907.04829, 2019b.  URL http://arxiv.org/abs/1907.04829.
  * Crawshaw (2020) Michael Crawshaw.  Multi-task learning with deep neural networks: A survey.  _arXiv preprint arXiv:2009.09796_ , 2020.
  * De Lange et al. (2021) Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(7):3366–3385, 2021.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Ding et al. (2018) Jie Ding, Vahid Tarokh, and Yuhong Yang.  Model selection techniques: An overview.  _IEEE Signal Processing Magazine_ , 35(6):16–34, 2018.
  * Ditzler et al. (2015) Gregory Ditzler, Manuel Roveri, Cesare Alippi, and Robi Polikar.  Learning in nonstationary environments: A survey.  _IEEE Computational Intelligence Magazine_ , 10(4):12–25, 2015.
  * González-Garduño & Søgaard (2018) Ana González-Garduño and Anders Søgaard.  Learning to predict readability using eye-movement data from natives and learners.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1), Apr. 2018.  doi: 10.1609/aaai.v32i1.11978.  URL https://ojs.aaai.org/index.php/AAAI/article/view/11978.
  * Guo et al. (2019) Han Guo, Ramakanth Pasunuru, and Mohit Bansal.  Autosem: Automatic task selection and mixing in multi-task learning.  _CoRR_ , abs/1904.04153, 2019.  URL http://arxiv.org/abs/1904.04153.
  * Guo et al. (2018) Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei.  Dynamic task prioritization for multitask learning.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * Gupta & Agrawal (2022) Manish Gupta and Puneet Agrawal.  Compression of deep learning models for text: A survey.  _ACM Transactions on Knowledge Discovery from Data (TKDD)_ , 16(4):1–55, 2022.
  * Ha et al. (2016) David Ha, Andrew Dai, and Quoc V. Le.  Hypernetworks, 2016.  URL https://arxiv.org/abs/1609.09106.
  * Haldar et al. (2019) Malay Haldar, Mustafa Abdool, Prashant Ramanathan, Tao Xu, Shulin Yang, Huizhong Duan, Qing Zhang, Nick Barrow-Williams, Bradley C Turnbull, Brendan M Collins, et al.  Applying deep learning to airbnb search.  In _proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & Data Mining_, pp. 1927–1935, 2019.
  * Han et al. (2021) Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun.  PTR: prompt tuning with rules for text classification.  _CoRR_ , abs/2105.11259, 2021.  URL https://arxiv.org/abs/2105.11259.
  * He et al. (2021) Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  _arXiv preprint arXiv:2110.04366_ , 2021.
  * He et al. (2022) Yun He, Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, Yaguang Li, Zhao Chen, Donald Metzler, Heng-Tze Cheng, and Ed H. Chi.  HyperPrompt: Prompt-based task-conditioning of transformers.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 8678–8690. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/he22f.html.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In _International Conference on Machine Learning_ , pp. 2790–2799. PMLR, 2019.
  * Hsu et al. (2018) Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira.  Re-evaluating continual learning scenarios: A categorization and case for strong baselines.  _arXiv preprint arXiv:1810.12488_ , 2018.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models, 2021.  URL https://arxiv.org/abs/2106.09685.
  * Huyen (2022) Chip Huyen.  _Designing Machine Learning Systems_.  ” O’Reilly Media, Inc.”, 2022.
  * Jean et al. (2019) Sébastien Jean, Orhan Firat, and Melvin Johnson.  Adaptive scheduling for multi-task learning.  _CoRR_ , abs/1909.06434, 2019.  URL http://arxiv.org/abs/1909.06434.
  * Jin et al. (2021) Xisen Jin, Bill Yuchen Lin, Mohammad Rostami, and Xiang Ren.  Learn continually, generalize rapidly: lifelong knowledge accumulation for few-shot learning.  _arXiv preprint arXiv:2104.08808_ , 2021.
  * Kaiser et al. (2017) Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit.  One model to learn them all, 2017.  URL https://arxiv.org/abs/1706.05137.
  * Karpathy (2021) Andrej Karpathy.  HydraNets - Tesla AI Day 2021, 8 2021.  URL https://www.youtube.com/watch?t=4284&v=j0z4FweCy4M&feature=youtu.be.  Accessed: 2023-02-15.
  * Ke & Liu (2022) Zixuan Ke and Bing Liu.  Continual learning of natural language processing tasks: A survey.  _arXiv preprint arXiv:2211.12701_ , 2022.
  * Ke et al. (2021a) Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu.  Achieving forgetting prevention and knowledge transfer in continual learning.  _Advances in Neural Information Processing Systems_ , 34:22443–22456, 2021a.
  * Ke et al. (2021b) Zixuan Ke, Hu Xu, and Bing Liu.  Adapting bert for continual learning of a sequence of aspect sentiment classification tasks.  _arXiv preprint arXiv:2112.03271_ , 2021b.
  * Kendall et al. (2018) Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * Kim & Doshi-Velez (2021) Been Kim and Finale Doshi-Velez.  Machine learning techniques for accountability.  _AI Magazine_ , 42(1):47–52, 2021.
  * Kirstein et al. (2022) Frederic Kirstein, Jan Philip Wahle, Terry Ruas, and Bela Gipp.  Analyzing multi-task learning for abstractive text summarization.  _arXiv preprint arXiv:2210.14606_ , 2022.
  * Lakshmanan et al. (2020) Valliappa Lakshmanan, Sara Robinson, and Michael Munn.  _Machine learning design patterns_.  O’Reilly Media, 2020.
  * Lee et al. (2018) Hae Beom Lee, Eunho Yang, and Sung Ju Hwang.  Deep asymmetric multi-task feature learning.  In Jennifer Dy and Andreas Krause (eds.), _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 2956–2964. PMLR, 10–15 Jul 2018.  URL https://proceedings.mlr.press/v80/lee18d.html.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  _CoRR_ , abs/2104.08691, 2021.  URL https://arxiv.org/abs/2104.08691.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 7871–7880, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.703.  URL https://aclanthology.org/2020.acl-main.703.
  * Li & Liang (2021) Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  _arXiv preprint arXiv:2101.00190_ , 2021.
  * Lin et al. (2019) Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong.  Pareto multi-task learning.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/685bfde03eb646c27ed565881917c71c-Paper.pdf.
  * Liu et al. (2021a) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _CoRR_ , abs/2107.13586, 2021a.  URL https://arxiv.org/abs/2107.13586.
  * Liu et al. (2019a) Shengchao Liu, Yingyu Liang, and Anthony Gitter.  Loss-balanced task weighting to reduce negative transfer in multi-task learning.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):9977–9978, Jul. 2019a.  doi: 10.1609/aaai.v33i01.33019977.  URL https://ojs.aaai.org/index.php/AAAI/article/view/5125.
  * Liu et al. (2021b) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang.  GPT understands, too.  _CoRR_ , abs/2103.10385, 2021b.  URL https://arxiv.org/abs/2103.10385.
  * Liu et al. (2019b) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao.  Multi-task deep neural networks for natural language understanding, 2019b.  URL https://arxiv.org/abs/1901.11504.
  * Liu et al. (2019c) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692, 2019c.  URL http://arxiv.org/abs/1907.11692.
  * Long et al. (2017) Mingsheng Long, ZHANGJIE CAO, Jianmin Wang, and Philip S Yu.  Learning multiple tasks with multilinear relationship networks.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/file/03e0704b5690a2dee1861dc3ad3316c9-Paper.pdf.
  * Lopez-Paz & Ranzato (2017) David Lopez-Paz and Marc' Aurelio Ranzato.  Gradient episodic memory for continual learning.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/file/f87522788a2be2d171666752f97ddebb-Paper.pdf.
  * Mahabadi et al. (2021) Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson.  Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks, 2021.  URL https://arxiv.org/abs/2106.04489.
  * Marchant (2011) Gary E Marchant.  _The growing gap between emerging technologies and the law_.  Springer, 2011.
  * McCann et al. (2018) Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher.  The natural language decathlon: Multitask learning as question answering.  _arXiv preprint arXiv:1806.08730_ , 2018.
  * Mehrabi et al. (2021) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  _ACM Computing Surveys (CSUR)_ , 54(6):1–35, 2021\.
  * Nahar et al. (2022) Nadia Nahar, Shurui Zhou, Grace Lewis, and Christian Kästner.  Collaboration challenges in building ml-enabled systems: Communication, documentation, engineering, and process.  In _Proceedings of the 44th International Conference on Software Engineering_ , pp. 413–425, 2022.
  * Pacheco et al. (2018) Fannia Pacheco, Ernesto Exposito, Mathieu Gineste, Cedric Baudoin, and Jose Aguilar.  Towards the deployment of machine learning solutions in network traffic classification: A systematic survey.  _IEEE Communications Surveys & Tutorials_, 21(2):1988–2014, 2018.
  * Paleyes et al. (2022) Andrei Paleyes, Raoul-Gabriel Urma, and Neil D Lawrence.  Challenges in deploying machine learning: a survey of case studies.  _ACM Computing Surveys_ , 55(6):1–29, 2022.
  * Parmar et al. (2022) Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, Murad Mohammad, and Chitta Baral.  In-BoXBART: Get instructions into biomedical multi-task learning.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pp. 112–128, Seattle, United States, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-naacl.10.  URL https://aclanthology.org/2022.findings-naacl.10.
  * Pascal et al. (2021) Lucas Pascal, Pietro Michiardi, Xavier Bost, Benoit Huet, and Maria A. Zuluaga.  Maximum roaming multi-task learning.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(10):9331–9341, May 2021.  doi: 10.1609/aaai.v35i10.17125.  URL https://ojs.aaai.org/index.php/AAAI/article/view/17125.
  * Pei et al. (2022) Zhongyi Pei, Lin Liu, Chen Wang, and Jianmin Wang.  Requirements engineering for machine learning: A review and reflection.  In _2022 IEEE 30th International Requirements Engineering Conference Workshops (REW)_ , pp. 166–175. IEEE, 2022.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller.  Language models as knowledge bases?  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 2463–2473, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1250.  URL https://aclanthology.org/D19-1250.
  * Pfeiffer et al. (2020a) Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych.  Adapterfusion: Non-destructive task composition for transfer learning.  _arXiv preprint arXiv:2005.00247_ , 2020a.
  * Pfeiffer et al. (2020b) Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych.  Adapterhub: A framework for adapting transformers.  _arXiv preprint arXiv:2007.07779_ , 2020b.
  * Pfeiffer et al. (2020c) Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder.  MAD-X: an adapter-based framework for multi-task cross-lingual transfer.  _CoRR_ , abs/2005.00052, 2020c.  URL https://arxiv.org/abs/2005.00052.
  * Pilault et al. (2020) Jonathan Pilault, Amine Elhattami, and Christopher Pal.  Conditionally adaptive multi-task learning: Improving transfer learning in nlp using fewer parameters & less data, 2020.  URL https://arxiv.org/abs/2009.09139.
  * Politou et al. (2018) Eugenia Politou, Efthimios Alepis, and Constantinos Patsakis.  Forgetting personal data and revoking consent under the gdpr: Challenges and proposed solutions.  _Journal of cybersecurity_ , 4(1):tyy001, 2018\.
  * Polyzotis et al. (2018) Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, and Martin Zinkevich.  Data lifecycle challenges in production machine learning: a survey.  _ACM SIGMOD Record_ , 47(2):17–28, 2018.
  * Pramanik et al. (2019) Subhojeet Pramanik, Priyanka Agrawal, and Aman Hussain.  Omninet: A unified architecture for multi-modal multi-task learning.  _CoRR_ , abs/1907.07804, 2019.  URL http://arxiv.org/abs/1907.07804.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67, 2020.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.  Learning multiple visual domains with residual adapters.  _Advances in neural information processing systems_ , 30, 2017.
  * Ren et al. (2020) Kui Ren, Tianhang Zheng, Zhan Qin, and Xue Liu.  Adversarial attacks and defenses in deep learning.  _Engineering_ , 6(3):346–360, 2020.
  * Renggli et al. (2019) Cedric Renggli, Bojan Karlaš, Bolin Ding, Feng Liu, Kevin Schawinski, Wentao Wu, and Ce Zhang.  Continuous integration of machine learning models with ease. ml/ci: Towards a rigorous yet practical treatment.  _Proceedings of Machine Learning and Systems_ , 1:322–333, 2019.
  * Rosenberg et al. (2021) Ishai Rosenberg, Asaf Shabtai, Yuval Elovici, and Lior Rokach.  Adversarial machine learning attacks and defense methods in the cyber security domain.  _ACM Computing Surveys (CSUR)_ , 54(5):1–36, 2021\.
  * Ruder (2017) Sebastian Ruder.  An overview of multi-task learning in deep neural networks.  _arXiv preprint arXiv:1706.05098_ , 2017.
  * Ruder et al. (2017) Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard.  Sluice networks: Learning what to share between loosely related tasks.  _arXiv preprint arXiv:1705.08142_ , 2, 2017.
  * Samant et al. (2022) Rahul Manohar Samant, Mrinal Bachute, Shilpa Gite, and Ketan Kotecha.  Framework for deep learning-based language models using multi-task learning in natural language understanding: A systematic literature review and future directions.  _IEEE Access_ , 2022.
  * Sambasivan et al. (2021) Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo.  “everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai.  In _proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , pp. 1–15, 2021.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush.  Multitask prompted training enables zero-shot task generalization, 2021\.  URL https://arxiv.org/abs/2110.08207.
  * Schröder & Schulz (2022) Tim Schröder and Michael Schulz.  Monitoring machine learning models: a categorization of challenges and methods.  _Data Science and Management_ , 5(3):105–116, 2022.
  * Sculley et al. (2015) David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison.  Hidden technical debt in machine learning systems.  _Advances in neural information processing systems_ , 28, 2015.
  * Serra et al. (2018) Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou.  Overcoming catastrophic forgetting with hard attention to the task.  In Jennifer Dy and Andreas Krause (eds.), _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 4548–4557. PMLR, 10–15 Jul 2018.  URL https://proceedings.mlr.press/v80/serra18a.html.
  * Sharir et al. (2020) Or Sharir, Barak Peleg, and Yoav Shoham.  The cost of training nlp models: A concise overview.  _arXiv preprint arXiv:2004.08900_ , 2020.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh.  AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 4222–4235, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.346.  URL https://aclanthology.org/2020.emnlp-main.346.
  * Sinha et al. (2018) Ayan Sinha, Zhao Chen, Vijay Badrinarayanan, and Andrew Rabinovich.  Gradient adversarial training of neural networks.  _CoRR_ , abs/1806.08028, 2018.  URL http://arxiv.org/abs/1806.08028.
  * Standley et al. (2020) Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese.  Which tasks should be learned together in multi-task learning?  In Hal Daumé III and Aarti Singh (eds.), _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pp. 9120–9132. PMLR, 13–18 Jul 2020.  URL https://proceedings.mlr.press/v119/standley20a.html.
  * Stickland & Murray (2019) Asa Cooper Stickland and Iain Murray.  Bert and pals: Projected attention layers for efficient adaptation in multi-task learning.  In _International Conference on Machine Learning_ , pp. 5986–5995. PMLR, 2019.
  * Strubell et al. (2020) Emma Strubell, Ananya Ganesh, and Andrew McCallum.  Energy and policy considerations for modern deep learning research.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 13693–13696, 2020.
  * Sun (2020) Ruo-Yu Sun.  Optimization for deep learning: An overview.  _Journal of the Operations Research Society of China_ , 8(2):249–294, 2020.
  * Sun et al. (2020) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang.  Ernie 2.0: A continual pre-training framework for language understanding.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pp. 8968–8975, 2020.
  * Takeuchi & Yamamoto (2020) Hironori Takeuchi and Shuichiro Yamamoto.  Business analysis method for constructing business–ai alignment model.  _Procedia Computer Science_ , 176:1312–1321, 2020.
  * Tay et al. (2020) Yi Tay, Zhe Zhao, Dara Bahri, Donald Metzler, and Da-Cheng Juan.  Hypergrid: Efficient multi-task transformers with grid-wise decomposable hyper projections.  _CoRR_ , abs/2007.05891, 2020.  URL https://arxiv.org/abs/2007.05891.
  * Thung & Wee (2018) Kim-Han Thung and Chong-Yaw Wee.  A brief review on multi-task learning.  _Multimedia Tools and Applications_ , 77(22):29705–29725, 2018.
  * Upadhyay et al. (2021) Richa Upadhyay, Ronald Phlypo, Rajkumar Saini, and Marcus Liwicki.  Sharing to learn and learning to share-fitting together meta-learning, multi-task learning, and transfer learning: A meta review.  _arXiv preprint arXiv:2111.12146_ , 2021.
  * Vafaeikia et al. (2020) Partoo Vafaeikia, Khashayar Namdar, and Farzad Khalvati.  A brief review of deep multi-task learning and auxiliary task learning.  _arXiv preprint arXiv:2007.01126_ , 2020.
  * Vandenhende et al. (2021) Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool.  Multi-task learning for dense prediction tasks: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ , 2021.
  * Vartak & Madden (2018) Manasi Vartak and Samuel Madden.  Modeldb: Opportunities and challenges in managing machine learning models.  _IEEE Data Eng. Bull._ , 41(4):16–25, 2018.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Von Oswald et al. (2019) Johannes Von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe.  Continual learning with hypernetworks.  _arXiv preprint arXiv:1906.00695_ , 2019.
  * Vu et al. (2020) Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer.  Exploring and predicting transferability across NLP tasks.  _CoRR_ , abs/2005.00770, 2020.  URL https://arxiv.org/abs/2005.00770.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_ , 2018.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _Advances in neural information processing systems_ , 32, 2019.
  * Wang et al. (2022) Liwen Wang, Rumei Li, Yang Yan, Yuanmeng Yan, Sirui Wang, Wei Wu, and Weiran Xu.  Instructionner: A multi-task instruction-based generative framework for few-shot ner, 2022.  URL https://arxiv.org/abs/2203.03903.
  * Wang et al. (2020) Sinong Wang, Madian Khabsa, and Hao Ma.  To pretrain or not to pretrain: Examining the benefits of pretraining on resource rich tasks.  _arXiv preprint arXiv:2006.08671_ , 2020.
  * Whang et al. (2023) Steven Euijong Whang, Yuji Roh, Hwanjun Song, and Jae-Gil Lee.  Data collection and quality challenges in deep learning: A data-centric ai perspective.  _The VLDB Journal_ , pp. 1–23, 2023.
  * Worsham & Kalita (2020) Joseph Worsham and Jugal Kalita.  Multi-task learning for natural language processing in the 2020s: where are we going?  _Pattern Recognition Letters_ , 136:120–126, 2020.
  * Wu & Xie (2022) Nan Wu and Yuan Xie.  A survey of machine learning for computer architecture and systems.  _ACM Computing Surveys (CSUR)_ , 55(3):1–39, 2022\.
  * Wu et al. (2020) Sen Wu, Hongyang R. Zhang, and Christopher Ré.  Understanding and improving information transfer in multi-task learning, 2020.  URL https://arxiv.org/abs/2005.00944.
  * Yang et al. (2022) Chenyang Yang, Rachel Brower-Sinning, Grace A Lewis, Christian Kästner, and Tongshuang Wu.  Capabilities for better ml engineering.  _arXiv preprint arXiv:2211.06409_ , 2022.
  * Yang & Hospedales (2016) Yongxin Yang and Timothy M. Hospedales.  Trace norm regularised deep multi-task learning.  _CoRR_ , abs/1606.04038, 2016.  URL http://arxiv.org/abs/1606.04038.
  * Zamir et al. (2018) Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese.  Taskonomy: Disentangling task transfer learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * Zhai et al. (2019) Andrew Zhai, Hao-Yu Wu, Eric Tzeng, Dong Huk Park, and Charles Rosenberg.  Learning a unified embedding for visual search at pinterest.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 2412–2420, 2019.
  * Zhang & Yang (2017) Yu Zhang and Qiang Yang.  A survey on multi-task learning.  _arXiv preprint arXiv:1707.08114_ , 2017.
  * Zhang & Yang (2018) Yu Zhang and Qiang Yang.  An overview of multi-task learning.  _National Science Review_ , 5(1):30–43, 2018\.
  * Zhang et al. (2023) Zhihan Zhang, Wenhao Yu, Mengxia Yu, Zhichun Guo, and Meng Jiang.  A survey of multi-task learning in natural language processing: Regarding task relatedness and training methods.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pp. 943–956, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics.  URL https://aclanthology.org/2023.eacl-main.66.
  * Zheng et al. (2019) Feng Zheng, Cheng Deng, Xing Sun, Xinyang Jiang, Xiaowei Guo, Zongqiao Yu, Feiyue Huang, and Rongrong Ji.  Pyramidal person re-identification via multi-loss dynamic training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Zhou (2019) Wenxuan Zhou.  An overview of models and methods for multi-task learning, Oct 2019.  URL https://shanzhenren.github.io/csci-699-replnlp-2019fall/lectures/W6-L1-Multi_Task_Learning.pdf.

##  Appendix A Additional Details for Related Surveys

###  A.1 Multi-Task Learning Surveys

Table 3: Discussed aspects per MTL survey. Aspects are indicated in bold. Year | MTL Survey
---|---
$2017$ |  $1$ \- (Ruder, 2017) $2$ \- (Zhang & Yang, 2017)
$2018$ |  $3$ \- (Zhang & Yang, 2018) $4$ \- (Thung & Wee, 2018)
$2019$ |  $5$ \- (Zhou, 2019)
$2020$ |  $6$ \- (Vafaeikia et al., 2020) $7$ \- (Worsham & Kalita, 2020) $8$ \- (Crawshaw, 2020)
$2021$ |  $9$ \- (Vandenhende et al., 2021) $10$ \- (Chen et al., 2021) $11$ \- (Upadhyay et al., 2021)
$2022$ |  $12$ \- (Samant et al., 2022) $13$ \- (Abhadiomhen et al., 2022)
$2023$ |  $14$ \- (Zhang et al., 2023)
Aspect \Survey | $1$ | $2$ | $3$ | $4$ | $5$ | $6$ | $7$ | $8$ | $9$ | $10$ | $11$ | $12$ | $13$ | $14$
Computational Model |
Traditional ML | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  |  |  |  | ✓ |
Deep Learning | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  | ✓
Learning Type |
Joint Learning | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Auxiliary Learning | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  |  | ✓ |  | ✓ |  | ✓
Architectures |
Taxonomy | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Learning to Share | ✓ |  |  | ✓ | ✓ | ✓ |  | ✓ | ✓ | ✓ |  |  |  | ✓
Universal Models |  |  |  |  | ✓ |  |  | ✓ | ✓ | ✓ |  |  |  | ✓
Optimization |
Loss Weighting | ✓ | ✓ | ✓ |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  | ✓ |  |
Regularization | ✓ | ✓ | ✓ | ✓ | ✓ |  |  | ✓ |  | ✓ |  |  | ✓ |
Task Scheduling |  |  |  |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  |  |  |
Gradient Modulation | ✓ |  |  |  |  | ✓ |  | ✓ | ✓ | ✓ |  |  |  |
Knowledge Distillation |  |  |  |  |  |  | ✓ | ✓ |  | ✓ |  | ✓ |  |
Multi-Objective Optimization |  |  |  |  |  |  |  | ✓ | ✓ | ✓ |  |  |  |
Task Relationship Learning |
Task Grouping | ✓ | ✓ | ✓ | ✓ |  |  | ✓ | ✓ | ✓ |  |  |  | ✓ | ✓
Relationships Transfer | ✓ |  |  | ✓ |  |  | ✓ | ✓ | ✓ |  |  |  |  |
Task Embeddings |  |  |  |  |  |  | ✓ | ✓ |  |  |  |  |  |
Supervision Level |
Supervised Learning | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Semi-supervised Learning |  | ✓ | ✓ |  |  |  |  |  |  | ✓ |  | ✓ |  | ✓
Self-supervised Learning |  | ✓ | ✓ |  |  |  | ✓ |  |  | ✓ |  | ✓ |  | ✓
Connection to Learning Paradigm |
Reinforcement Learning |  | ✓ | ✓ |  | ✓ | ✓ |  | ✓ |  |  |  |  |  | ✓
Transfer Learning |  | ✓ | ✓ | ✓ |  |  |  |  |  |  |  |  |  |
Domain Adaptation | ✓ |  |  |  |  |  | ✓ |  |  |  | ✓ |  |  |
Meta-Learning |  |  |  |  |  |  |  | ✓ |  |  | ✓ |  |  | ✓
Active Learning |  | ✓ | ✓ |  |  |  |  |  |  |  |  |  |  |
Online Learning | ✓ | ✓ | ✓ |  |  |  |  |  |  |  |  |  |  |
Continual Learning |  |  |  |  |  |  |  |  |  |  |  |  |  |
Benchmarks |
Benchmark Overview |  |  | ✓ |  |  |  | ✓ | ✓ | ✓ | ✓ |  | ✓ | ✓ |
Model Comparison |  |  | ✓ |  |  |  | ✓ | ✓ | ✓ | ✓ |  | ✓ | ✓ |
Application Domain |
Natural Language Processing | ✓ | ✓ | ✓ | ✓ | ✓ |  | ✓ | ✓ |  | ✓ | ✓ | ✓ |  | ✓
Computer Vision | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  | ✓ | ✓ |  | ✓ |  |  |
Healthcare |  | ✓ | ✓ | ✓ |  |  |  |  |  |  |  |  |  |
Bioinformatics | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  |  | ✓ |  |  |
Other | ✓ | ✓ | ✓ |  |  |  |  | ✓ |  |  |  |  | ✓ |

The idea of MTL has been explored in many research studies. In this section,
we provide an overview of related work in MTL surveys and address different
MTL aspects that were discussed. In Table 3, we list certain aspects and
indicate the survey in which they were discussed. In the rest of the section,
we just mention related surveys and go over individual MTL aspects (shown in
bold) in more detail.

Many application domains were studied in previous work, ranging from surveys
covering multiple domains (Ruder, 2017; Zhang & Yang, 2017; 2018; Thung & Wee,
2018; Vafaeikia et al., 2020; Crawshaw, 2020; Upadhyay et al., 2021;
Abhadiomhen et al., 2022), to those dedicated to a specific domain, such as
computer vision (Vandenhende et al., 2021) or natural language processing
(Zhou, 2019; Worsham & Kalita, 2020; Chen et al., 2021; Samant et al., 2022;
Zhang et al., 2023). Both traditional ML and deep learning computational
models were studied. The traditional ML was discussed primarily in older
studies, while deep learning was represented in all but one study.
Furthermore, most prior works provided an overview of the benchmarks for the
specific domain (McCann et al., 2018; Wang et al., 2019), and compare models
on them.

Learning types. Two learning types were mainly discussed. First, joint
learning was used in an MTL setting where all the tasks are equally important
(Kendall et al., 2018; Liu et al., 2019b). Here, the goal is to achieve an on-
par performance compared to their single-task learning (STL) counterparts.
Second, auxiliary learning was used when there was a single main task or a set
of them, while auxiliary tasks are only used to improve the performance of the
main tasks (González-Garduño & Søgaard, 2018; Wang et al., 2022). Although the
difference between the two learning types can be made, sometimes auxiliary
learning was not differentiated from joint learning.

Architectures. MTL model architectures were among the most discussed aspects
of MTL and one of the first aspects tackled in previous work. Distinguishing
between hard and soft parameter sharing (Ruder, 2017) is the most used
architecture taxonomy. In recent surveys, the method of sharing parameters
between tasks has improved, leading to the refinement of taxonomies to
categorize architectures more precisely (Crawshaw, 2020; Chen et al., 2021).
Next, some MTL architectures were categorized as learning-to-share approaches
(Ruder et al., 2017). Those works argue that it is better to learn parameter
sharing in architectures for MTL rather than hand-design where sharing
happens, as it offers a more adaptive solution to accommodate task
similarities at different parts of the network. Additionally, some MTL
architectures were categorized as universal models which can handle multiple
modalities, domains, and tasks with a single model (Kaiser et al., 2017;
Pramanik et al., 2019).

Optimization. Optimization techniques for MTL architectures were also
discussed in detail. To start with, the most common approach to mitigating MTL
challenges is loss weighting. Computing weights of task-specific losses is
crucial, as it helps optimize the loss function and considers the relative
importance of each task. There are various approaches to computing the loss
weights dynamically, including weighting by uncertainty (Kendall et al.,
2018), learning speed (Liu et al., 2019a; Zheng et al., 2019), or performance
(Guo et al., 2018; Jean et al., 2019), among others. Next, and closely related
to weighting task losses, is a task scheduling problem that involves choosing
tasks to train on at each step. Many techniques were used, from simple ones
that employ uniform or proportional task sampling, to the more complicated
ones, such as annealed sampling (Stickland & Murray, 2019) or approaches based
on active learning (Pilault et al., 2020). Regularization approaches were also
analyzed. Methods include (1) minimizing the L2 norm between the parameters of
soft-parameter sharing models (Yang & Hospedales, 2016), (2) placing prior
distributions on the network parameters (Long et al., 2017), (3) introducing
an auto-encoder term to the objective function (Lee et al., 2018), (4) MTL
variant of dropout (Pascal et al., 2021), and others. To continue, gradient
modulation techniques were employed to mitigate the problem of negative
transfer by manipulating gradients of contradictory tasks, either through
adversarial training (Sinha et al., 2018) or by replacing the gradient by its
modified version (Lopez-Paz & Ranzato, 2017). Another approach for optimizing
MTL models is by applying knowledge distillation (Clark et al., 2019b).
Finally, multi-objective optimization has been applied to the MTL setting to
obtain a set of Pareto optimal solutions on the Pareto frontier, providing
greater flexibility in balancing trade-offs between tasks (Lin et al., 2019).

Task relationship learning. The approach in MTL that focuses on learning the
explicit representation of tasks or relationships between them is task
relationship learning. This approach consists of three main categories of
methods. First, task grouping aims to divide a set of tasks into groups in
order to maximize knowledge sharing during joint training (Standley et al.,
2020). Second, transfer relationship learning involves methods that determine
when transferring knowledge from one task to another will be beneficial for
joint learning (Zamir et al., 2018; Guo et al., 2019). Finally, task embedding
methods aim to learn an embedding space for the tasks themselves (Vu et al.,
2020).

As for the supervision level, most studies focused on supervised approaches.
Some studies analyzed semi-supervised approaches that incorporate self-
supervised objectives, such as MLM. However, self-supervised approaches are
mainly discussed in the context of pre-training.

Most studies had made connections to other learning paradigms, including
reinforcement learning, transfer learning with a special emphasis on domain
adaptation, meta-learning, and active and online learning. However, only Ruder
(2017); Zhang & Yang (2017; 2018) explored the relationship between MTL and
online learning in the context of traditional ML. To the best of our
knowledge, there had been no prior work systematically investigating
connections between MTL and CL. We believe that a connection between MTL and
CL represents a promising research direction, as we will motivate the
importance of this connection in Section 4.

##  Appendix B Additional Multi-Task Learning Approaches

###  B.1 Prompts

Prompts embed a task in the input. The original input $\mathbf{x}$ is modified
using a template into a textual string prompt $\mathbf{x^{\prime}}$ that has
some unfilled slots, and then the LM is used to fill the information to obtain
a final string $\mathbf{\hat{x}}$, from which the final output $\mathbf{y}$
can be derived (Liu et al., 2021a). Prompting requires redesigning all the
inputs and outputs in order to treat the tasks as text-to-text problems. The
prompting approach proved to work the best in the zero- and few-show
scenarios, but the benefits dissipate in the high-resource settings (Parmar et
al., 2022; Wang et al., 2022). Also, performance scales well with an increase
in model size (Lester et al., 2021), making this approach not accessible to
everyone.

According to Liu et al. (2021a), there are many flavours to prompting. First,
models with different pre-training objectives can be used – left-to-right LM
(Brown et al., 2020), MLM (Liu et al., 2019c), prefix LM (Bao et al., 2020),
or encoder-decoder one (Lewis et al., 2020). Prompts can be engineered in a
cloze (Petroni et al., 2019) or prefix (Lester et al., 2021) shape, be hand-
crafted (Brown et al., 2020) or automated, which can again be discrete (Shin
et al., 2020) or continuous (Lester et al., 2021). Answer engineering searches
for an answer space and a mapping to the original output by deciding the
answer shape and choosing an answer design method. Furthermore, parameters can
be updated using different settings – tuning-free prompting (Brown et al.,
2020), fixed-LM prompt tuning (Li & Liang, 2021), fixed-prompt LM tuning
(Raffel et al., 2020), and prompt+LM tuning (Liu et al., 2021b). Finally,
training can be done in a few/zero-shot (Brown et al., 2020) or full-data
setting (Han et al., 2021). In the following paragraphs, we discuss some of
the previous prompting works.

HyperPrompt (He et al., 2022) is an approach that combines hypernetworks and
prompts. The key idea is to prepend task-conditioned trainable vectors to both
keys and values of MHA sub-layer at every Transformer layer. These task-
specific attention feature maps are jointly trained with the task-agnostic
representations. Key prompts interact with the original query, enabling tokens
to acquire task-specific semantics. Value prompts are prepended to the
original value vector, serving as task-specific memories for MHA to retrieve
information from. However, instead of having a key/value prompt for each task
and layer, authors initialize a global prompt $P$ for each task. They apply
two local hypernetworks $h_{k/v}$ (one for keys, the other for values) at each
Transformer layer in order to project this prompt into actual task- and layer-
specific key/value prompts. There are three variations examined in the paper –
HyperPrompt-Share, -Sep, and -Global. HyperPrompt-Global proved to be the
best, as it allows a flexible way to share information across tasks and
layers. It introduces task and layer embeddings, which are then fused into the
layer-aware task embedding. This embedding is then an input for the global
hypernetworks $H_{k/v}$ used as a generator of local hypernetworks $h_{k/v}$.
These local hypernetworks then generate hyper-prompts using a global prompt
$P$. Hyper-prompts are finally prepended with the original keys and values in
the MHA sub-layers. During training, no parameters are kept frozen. They
report better results compared to the competitive methods of HyperFormer++
(Mahabadi et al., 2021) and Prompt-Tuning (Lester et al., 2021).

In-BoXBART (Parmar et al., 2022) uses biomedical instructions which contain:
(1) definition (core explanation and detailed instructions about what needs to
be done), (2) prompt (short explanation of the task), and (3) examples
(input/output pairs with the explanation). Each instruction (effectively a
prompt) is prepended to the input instances. A problem of too long instances
arises.

InstructionNER (Wang et al., 2022) enriches the inputs with task-specific
instructions and answer options. Additionally, two auxiliary tasks are
introduced – entity extraction and entity typing, which directly help in
solving a NER task.

Sanh et al. (2021) allowed public to suggest prompts and came up with 2073
prompts for 177 datasets in total (12 prompts per dataset on avarage). They
shuffled and combined all the examples from the datasets prior to training. In
most of the cases, performance of their models improved as the number of
training datasets increased. Moreover, training on more prompts per dataset
resulted in a better and more robust generalization on unseen datasets. The
models they trained are based on LM-adapted T5 model (Lester et al., 2021).

Prefix tuning freezes the language model parameters and optimizes a small,
continuous task-specific vector called prefix (Li & Liang, 2021).
Consequently, only a prefix needs to be stored for each task, making prefix
tuning modular and space-efficient. This approach can be applied solely to
text generation models, such as GPT-2 (Radford et al., 2019) and BART (Lewis
et al., 2020). They state the intuition that the context can influence the
encoding of input $\mathbf{x}$ by guiding what to extract from $\mathbf{x}$;
and can influence the generation of output $\mathbf{y}$ by steering the next
token distribution. They optimize the prefix as continuous word embeddings,
instead of optimizing over discrete tokens. The reason for this is that the
discrete prompt needs to match the real word embedding, resulting in a less
expressive model.

Lester et al. (2021) used a fixed-length prompt of special tokens, where the
embeddings of these tokens are updated. This removes the requirement that the
prompts are parametrized by the model and allows them to have their own
trainable parameters. They test three different initialization techniques –
random uniform, sampled vocabulary, and class label. LM-adaptation pre-
training technique was used. Unlike adapters, which modify the actual function
that acts on the input, prompt tuning adds new input representations that
affect how input is processed, leaving the function fixed. They freeze the
pre-trained model. Finally, they try prompt ensembling (one prompt per model,
for each task), which showed improved performance compared to a single-prompt
average.

Challenges and Opportunities in ML Lifecycle. Prompts can also mitigate some
of the challenges of different ML lifecycle phases (see Table 2). To start
with, in a fixed-prompt LM tuning or a prompt+LM tuning setting, knowledge is
transferred from other tasks, which can be especially beneficial for low-
resource tasks. However, in such a scenario, different optimization techniques
need to be considered in order to avoid negative interference and other MTL
drawbacks (Appendix A.1). As for the model selection, it is more of a
challenge than an opportunity, as models based on prompts require large number
of parameters to begin with in order to perform well. Model training
complexity depends on the parameter update setting, with some settings
requiring no (Brown et al., 2020) or only few parameter updates (Li & Liang,
2021). One of the benefits when using prompts is the ability to handle all
tasks using a single text-to-text model, regardless of the input and output
encoding. When using prompts, the biggest challenge in model deployment is the
model’s size. Finally, model updating due to distribution shift, new data, or
business requirements (see Section 4.4) seems most plausible in the setting
where prompts are continuous and tuned (Li & Liang, 2021). However, this has
downsides, such as difficult optimization, non-monotonic performance change
with regard to the number of parameters, and reserving a part of sequence
length for adaptation (Hu et al., 2021).