￥¥

# A Survey on Effective Invocation Methods of Massive LLM Services

Can Wang1 Bolin Zhang1 Dianbo Sui1 Zhiying Tu1 Corresponding author. Xiaoyu
Liu 1 Jiabao Kang 1 1Harbin Institute of Technology,
<EMAIL_ADDRESS>{brolin<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Language models as a service (LMaaS) enable users to accomplish tasks without
requiring specialized knowledge, simply by paying a service provider. However,
numerous providers offer massive large language model (LLM) services with
variations in latency, performance, and pricing. Consequently, constructing
the cost-saving LLM services invocation strategy with low-latency and high-
performance responses that meet specific task demands becomes a pressing
challenge. This paper provides a comprehensive overview of the LLM services
invocation methods. Technically, we give a formal definition of the problem of
constructing effective invocation strategy in LMaaS and present the LLM
services invocation framework. The framework classifies existing methods into
four different components, including input abstract, semantic cache, solution
design, and output enhancement, which can be freely combined with each other.
Finally, we emphasize the open challenges that have not yet been well
addressed in this task and shed light on future research.

##  1 Introduction

Large Language Models (LLM) are becoming a fundamental tool for various
natural language processing tasks Yang et al. (2023), as they have shown
amazing emergent abilities, like in-context learning, multi-step reasoning,
instruction following and tool learning. Due to commercial reasons, the
potential risk of misuse and expensive tuning cost, LLMs, such as GPT-3, GPT-4
and Claude, are usually released as LLM services through application
programming interface (API) instead of open sourcing model weights Yu et al.
(2023), which is called Language Models as a Service (LMaaS).

Provider | LLM | Input Cost | Output Cost
---|---|---|---
OpenAI | gpt-4 | $30.0 | $60.0
gpt-4-turbo | $10.0 | $30.0
gpt-3.5-turbo-1106 | $1.00 | $2.00
Anthropic | Claude-2.0 | $11.02 | $32.68
Claude-instant-1.2 | $1.63 | $5.51
AI21 | Jurassic-2 Ultra | $15.0 | $15.0
Jurassic-2 Mid | $10.0 | $10.0
Jurassic-2 Light | $3.00 | $3.00
Textsynth | M2M100 1.2B | $0.15 | $3.00
GPT-J 6B | $0.20 | $5.00
Falcon 7B | $0.20 | $5.00
Mistral 7B | $0.20 | $2.00
Llama2 7B | $0.20 | $2.00
Flan-T5-XXL | $0.20 | $5.00
Falcon 40B | $3.30 | $10.00
Cohere | command | $1.00 | $2.00
command-light | $0.30 | $0.60
Baidu | Llama-2-13B-Chat | ￥6.00 | ￥6.00
Llama-2-70B-Chat | ￥35.0 | ￥35.0
ERNIE-Bot 4.0 | ￥150 | ￥300
ChatGLM2-6B-32K | ￥4.00 | ￥4.00
Llama-2-7B-Chat | ￥4.00 | ￥4.00
ERNIE-Bot | ￥12.0 | ￥12.0
BLOOMZ-7B | ￥4.00 | ￥4.00
ERNIE-Bot-turbo-0922 | ￥8.00 | ￥12.0

Table 1: Price list of different LMaaS. The cost is priced per 1 million
tokens. Note that Baidu’s LLM services are priced in Chinese Yuan ($￥$), while
other LLM services are priced in US Dollars ($). The data updated to 24
January 2024.

Via accessing these powerful LLMs as services through their opened API, novice
users do not need to possess extensive computational resources and expertise
in deep learning, as they can solve the tasks of interest by crafting task-
specific input query. However, invoking LLM services is not free and using
them for high-throughput applications can be very expensive. Estimated by
Claudia Slowik, a business supporting 15,000 customer interactions with text-
davinci-003 could have a monthly cost exceeding $14,400.

Typically, the cost of invoking LLM services consists of two components: (1)
input cost (proportional to the length of the input prompt), (2) output cost
(proportional to the length of the generated sequence). In Table 1, we present
the cost associated with using 25 different LLM services from some top-tier
providers, like OpenAI, Anthropic, AI21 and Textsynth. From the table, we
could find that the costs of different LLM services can vary by up to two
orders of magnitude: the input cost for 1 million tokens is $10 for OpenAI’s
GPT-4 but only $0.2 for Mistral 7B hosted by Textsyth.

In addition to cost considerations, various factors, including performance for
the same input query and response time, can also impact the user experience in
the usage of LLM services. Ahia et al. (2023); Lai et al. (2023) find that
different languages, prompt methods or the inclusion of simple enhancements
can also lead to notable alterations in performance. Meanwhile, Chen et al.
(2023) discovers that affordable LLMs often complement expensive ones. For
example, on the CoQA Reddy et al. (2019) dataset, GPT-4 makes a mistake about
11% of the questions, where cheaper and smaller GPJ-J can give right answers.

Considering the heterogeneity in pricing does not necessarily correlate with
the user experience, it is a great need to explore effective invocation
methods for LLM services in practice. As shown in Figure 1, we expect to make
use of massive LLM services to construct an effective invocation strategy
according to different methods, meeting targets in different scenarios. To
this end, we attempts to provide a comprehensive study of the development and
recent advances on effective invocation methods in LMaaS. In detail, We first
formalizes the task of constructing effective invocation strategy as a multi-
objective optimization problem. This entails simultaneous consideration of
latency, performance, and cost factors. Then, we propose a taxonomy to provide
a unified view on effective invocation methods in LMaaS where the existing
methods are categorized into: input abstract, semantic cache, solution design,
and output enhancement. These four components can be flexibly combined and
unified in a flexible framework. Finally, we highlight the challenges and
potential directions and hope our work can provide a useful roadmap for
beginners interested in this area and shed light on future research.

Figure 1: Vision of efficient invocation strategy construction for massive LLM
services.

forked edges, for tree= grow=east, reversed=true, anchor=base west, parent
anchor=east, child anchor=west, base=left, font=, rectangle, draw=hidden-draw,
rounded corners, align=left, minimum width=4em, edge+=darkgray, line
width=1pt, s sep=3pt, inner xsep=2pt, inner ysep=3pt, line width=0.8pt,
ver/.style=rotate=90, child anchor=north, parent anchor=south, anchor=center,
, where level=1text width=7.4em,font=,, where level=2text width=8.5em,font=,,
where level=3text width=5.8em,font=,, where level=4text width=5em,font=,, [
Invocation Strategies, ver [ Input
Abstraction (§3) [ Sentence
Simplification (§3.1) [ Extractive
methods [ TCRA-LLM Liu et al. (2023a), Mondrian Si et al. (2023),
Learned Token Pruning(LTP) Kim et al. (2022), , leaf, text width=25em ] ] [
Generative
methods [ Commercia-Models Ahia et al. (2023), TCRA-LLM Liu et al. (2023a),
OverPrompt Li et al. (2023) , leaf, text width=30em ] ] ] [ Prompt
Optimization (§3.2) [ Prompt
Selection [ LeanContext Arefeen et al. (2023), Cost-EffectiveL Zhou et al.
(2020),
Frugal-Prompting Santra et al. (2023) , leaf, text width=30em ] ] [ Prompt
Augmentation [ Black-Box Tuning Yu et al. (2023), Cost-EffectiveL Zhou et al.
(2020),
Vision-Transformer Haurum et al. (2023), Factual-Consistency Liu et al.
(2023c),
Chain-of-Thought Wu et al. (2023) , leaf, text width=34em ] ] ] ] [ Semantic
Technology (§4) [ Traditional
Cache (§4.1) [ GPTCache Bang (2023), Retrieval-based Dialogues Tao et al.
(2021),
Service-Caching Barrios and Kumar (2024), Optimal-Caching Zhu et al. (2023) ,
leaf, text width=34em ] ] [ Neural
Cache (§4.2) [ Cache-Distil Ramírez et al. (2023), VaryGen Rasool et al.
(2024),
Retrieval-based Dialogues Tao et al. (2021), leaf, text width=30em ] ] ] [
Solution
Design (§5) [ Score
Function (§5.1) [ Defined
metrics[ Cache-Distil Ramírez et al. (2023), MOT Yue et al. (2023),
Optimal Caching Zhu et al. (2023), Reward-guided Lu et al. (2023), leaf, text
width=30em ] ] [ Scorers[ FrugalGPT Chen et al. (2023), FORC Sakota et al.
(2023),
Model-Routing Shnitzer et al. (2023), EcoAssistant Zhang et al. (2023),
HYBRID LLM Anonymous (2024), AutoMix Madaan et al. (2023) , leaf, text
width=32em ] ] ] [ LLMs
Router (§5.2) [ Sequential
Structure [ FrugalGPT Chen et al. (2023), Cache-Distil Ramírez et al. (2023),
MOT Yue et al. (2023), EcoAssistant Zhang et al. (2023) , leaf, text
width=28em ] ] [ Other
Structure [ LLM-Blender Jiang et al. (2023), BRANCH-SOLVE-MERGE Saha et al.
(2023),
FORC Sakota et al. (2023), Reward-guided Lu et al. (2023),
AutoMix Madaan et al. (2023), MCDM Hosseinzadeh et al. (2020),
service-selection Manqele et al. (2017) , leaf, text width=34em ] ] ] ] [
Output
Enhancement (§6) [ Prompt-survey Liu et al. (2023b), FrugalMCT Chen et al.
(2022), Aligh-suevey Shen et al. (2023) , leaf, text width=44em ] ] ]

Figure 2: Taxonomy of effective invocation methods of LMaaS

The contributions of this survey can be concluded as follows:

  * •

Comprehensive Taxonomy. As shown in Figure 2, a taxonomy of effective
invocation methods in LMaaS is proposed, which categorizes existing methods
from four different aspects: input abstract, semantic cache, solution design
and output enhancement.

  * •

Flexible Framework. As shown in Figure 3, the framework can unify the four
type components, allowing each of them to work independently or
simultaneously, during the life cycle of the LLM service invocation.

  * •

Related Resources. To facilities the methods of this task, the price rules of
popular LMaaS products is present in Table. 1 and the paper list of existing
works is available. 111https://github.com/W-caner/Effective-strategy-for-LMaas

The rest of the survey is organized as follows. Section 2 describes the task
definition of constructing effective invocation strategy for LMaaS and
outlines the unified LLM services invocation framework. Section 3 reviews the
input abstract component, Section 4 reviews the semantic cache component,
Section 5 reviews the solution design component and Section 6 reviews the
output enhancement component. Section 7 emphasize the open challenges and
future direction of this task and summarizes the paper.

##  2 Background

###  2.1 Task Definition

In our topic, the problem is defined as how to construct an effective (low-
latency, high-performance, and cost-saving) invocation strategy $s$ given a
task $T$ among massive LLM services $LLMs$. The given task $T$ consists of
multiple identical query-answer pairs, represented as
$T=\\{(q_{1},a_{1}),(q_{2},a_{2}),...(q_{n},a_{n})\\}$, where $q$ represents
input query and $a$ represents output answer. Let, considering a fixed $LLM$,
a LLM service published through API. Input a query $q$, by invocation the
service, the process of getting the response $\tilde{a}$ can be represented
as:

$\tilde{a}=LLM(q)$ (1)

To characterize the concerns for the construction of effective invocation
strategy with a given query $q$ and LLM service $LLM$, we use three functions:
latency $f_{l}(LLM,q)$, performance $f_{p}(LLM,q)$, and cost $f_{c}(LLM,q)$.
These three functions are fixed values in a specific practical invocation and
can be estimated using certain methods. For example, $f_{l}$ might be a
function of the length of the input and output tokens. $f_{p}$ often uses a
metric function $r(·,·)$ to compare the difference between $a$ and
$\tilde{a}$. While $f_{c}$ involves two different pricing components we
mentioned before. We adopt the definition of the sum of the number of input
tokens multiplied by the price of input tokens and the number of generated
tokens multiplied by the price of generated tokens, as shown in the Eq. 2,
where $\alpha_{i}$ is a constant representing the unit price.

$f_{c}\triangleq\alpha_{1}||\tilde{a}||+\alpha_{2}||q||+\alpha_{3}$ (2)

Then we extend a single LLM service to $K$ different LLM services,
$LLM_{s}=\\{LLM_{1},LLM_{2},...LLM_{K}\\}$. Our problem is formalized as Eq.
3, where in the search space $S$, we seek the optimal invocation strategy $s$
that minimizes latency $f_{l}$, maximizes performance $f_{p}$, and minimizes
cost $f_{c}$ on task $T$. The best strategy $s$ includes a sequence of
selected LLM services, represented as
$s=\\{LLM_{1},LLM_{i},...,LLM_{k}\\},k\leq K$, which is highly flexible, such
as choosing a single service or accessing some in a specific order.

$\min\sum_{LLM_{i}\in s,q_{j}\in
T}F(f_{l}(LLM_{i},q_{j}),-f_{p}(LLM_{i},q_{j}),f_{c}(LLM_{i},q_{j}))$ (3)

This is a multi-objective optimization problem, and here, we combine them in a
simplified form using the function $F$. In the construction strategy of a
specific invocation, weighted averages may be used, or constraints may be
introduced, treating certain targets as conditions while optimizing others.
For example, in the scenario of limited funds, the cost $f_{c}$ is used as a
condition to obtain a invocation strategy with high-performance $f_{p}$ and
low-latency $f_{l}$.

###  2.2 LLM Services Invocation Framework

Similarly, we focus solely on the methods related to LLM services invocation
and do not consider others related to the internal details of LLM. According
to the different construction ways, the methods are summarized into four
categories, as shown in Figure 2.

Using the taxonomy, we proposed the effective LLM services invocation
framework illustrated in Figure 3, where different categories represented in
the form of components that can work independently or simultaneously.
Following the idea that building an effective invocation strategy requires an
understanding of the key resources involved in the LLM service life cycle Bai
et al. (2024), we divide the LLM services invocation into three phases :
before invocation, invocation and after invocation.

Before invocation, the user enters a query $q$, and we consider that in
general, $q$ consists of a question and multiple possible prompts. Where the
question represents the user’s goal, and the prompts are optional information
to help accomplish the goal.

The processing of the input query $q$ to express more meaningful information
in a more concise language is the first step to construct an effective
invocation strategy. The methods in this aspect are summarized as input
abstract (Section 3), which is divided into sentence simplification and prompt
optimization according to the different ways. The former reduce the latency
$f_{l}$ and cost $f_{c}$ by simplifying the query without changing its
semantics. The latter is used to improve the prompts for better performance
$f_{p}$.

Semantic cache (Section 4) is also an important strategy to improve service
performance, reduce latency and cost before invocation, which is divided into
traditional cache and neural cache according to different structures. It
checks whether there is a semantically similar query in the cache, if so, it
directly returns, otherwise it goes into the invocation phase.

Solution design (Section 5) aims to construct the best invocation solution $s$
by leveraging the complementary capabilities of massive LLM services. It
evaluates LLM services $LLM_{i}$ with a given query $q$, and the method of
evaluation is called scoring function. It is often done before the invocation,
such as an estimate of $f_{c}$ can be used to guide the design of a low-cost
solution. And in the invocation phase, the scoring function is used to guide
the organized routing between services, which is called LLM router. Through
different routing structures, the advantages of different services are
utilized to build a satisfactory solution for users.

After invocation, output enhancement (Section 6) focuses on the information
returned to the user. The output $\tilde{a}$ is adjusted to suit different
targets and returned in a suitable form. In addition, the input and output of
this invocation are stored into the semantic cache for future invocations.

Figure 3: LLM services invocation framework, shown by the phase of invocation.

##  3 Input Abstract

Input abstract is designed to reduce the length of the input query without
changing the semantics, while optimizing the prompt for better performance at
lower cost and latency to invoke a given LLM.

The generalization and in-context capabilities allow LLM services to get good
answers on untrained samplesDong et al. (2023). Thus, a variety of different
natural language tasks can be accomplished just by inputting different
queries. This also leads to the LLM service’s dependence on input when
invoked. After the service is selected, the input content and quality directly
affect the latency, price and performance of the service. For example,
concatenating the prompt ”just tell me the option, do not explain other
things” with the question as input to the LLM will generate shorter output,
reducing invocation cost and latency. However, it may at the same time cause
the LLM to lose its ability to think step by step, resulting in performance
degradation.

We grouped the methods into two categories based on different goals. Most LLM
services charge based on the length of tokens. Therefore, by reducing the
input length, sentence simplification can effectively reduce the use cost and
latency. Prompt optimization ensures the quality of the information and
improves the performance for the invocation.

###  3.1 Sentence Simplification

Sentence simplification aims to improve the performance of language models,
reduce latency and cost by reducing the complexity and length of language
expressions. In short, it is the process of making input more concise while
retaining its core meaning by modifying, removing, or replacing words,
phrases, or structures in a sentence.

This problem is similar to the summarization task, and many methods used in
summarization can be applied Huang et al. (2021); Watanangura et al. (2024);
Antony et al. (2023); Mridha et al. (2021). We collate the methods available
for LMaaS and divide them into extractive and generative methods based on
whether they derived entirely from the original input.

Extractive methods. From long original input, extractive methods select
sentences by extracting key sentences or phrases to form a new input, where
the content is entirely sourced from original. Pruning semantically irrelevant
tokens according to the relevance to the contexts is a good choice Liu et al.
(2023a). By employing an intermediate ”attacker” and using a greedy
invocation, iterative deletion and substitution of tokens are performed on the
input Si et al. (2023). Kim et al. (2022) based on the attention mechanism, to
remove unimportant tokens.

This approach is straightforward and efficient, making it very convenient for
immediate use. However, the extraction idea may ignore the global information.
Furthermore, it has limitations in tasks such as language translation, as it
cannot discern which parts need to be translated or deleted.

Generative methods. The generative methods refers to the compression and
rewriting of the content based on the original input, allowing for the
generation of new words. Language encoding is a simple processing method
applied to the input, Ahia et al. (2023) conducted extensive experiments with
different languages and tokenizers, where the cost varied by up to 5 times.
AE.studio 222Prompt Reducer-Cut Down GPT-4 Token Costs
(https://www.promptreducer.com/) employs encryption to provide an online
platform that sacrifices readability, reducing the length of input tokens by
half. Utilizing fast and low cost generative natural language models Liu et
al. (2023a); Li et al. (2023) also presents a viable option for sentence
simplification.

This category of methods is more flexible, as the generated sentences contain
less redundant information while preserving the main content. However, it may
introduce grammatical or factual errors. And this approach may rely on complex
structures such as graphs, trees, or neural networks.

###  3.2 Prompt Optimization

Prompt optimization is the design and adjustment of user-provided input
prompts to guide LLMS to produce output that more accurate, useful, or
tailored. The effectiveness of prompt optimization stems from the LLM’s
ability to learn from the few-shot or even zero-shot Liu et al. (2023b), where
appropriate prompts can complement the context of the task, highlight key
information, or improve the explain ability.

Based on the different granularity of optimization objective, we distinguish
two types of prompt optimization methods. By selecting or combining some
prompts, it is possible to guide LLM to handle various inputs more effectively
and efficiently. Prompt enhancement is concerned with the quality of the
content and wants to maximize the potential of the context.

Prompt selection. Prompt selection selects the most meaningful prompt from
possible prompts to accurately guide LLM. It removes the interference of
irrelevant prompts, and helps in efficient invocation. Zhou et al. (2020)
selecting representative samples and it can be highly beneficial in few-shot
tasks. Another way for prompt selection is to combine the prompts for the same
type tasks, allowing LLM to process prompt information shared by multiple
queries at once. Santra et al. (2023) combines various methods involving
instructions, examples, and additional context to propose a more compact
method for providing historical information in dialogues. Arefeen et al.
(2023) considers the concatenation of prompts and retrieves the most important
k sentences using comparative methods, enabling the shared use of prompts for
similar questions.

Prompt selection can directly guide LLM to focus on specific aspects of
information and understand user needs more accurately. For some generic tasks,
standard selection methods can be used without too much personalization.
However, for complex prompts, this approach does not maximize its potential
because no additional knowledge is introduced.

Prompt augmentation. Prompt augmentation considers the understanding ability
of LLM to elicit more accurate and desirable responses. Knowledge retrieval is
a direct method of enhancement, it helps achieve a comprehensive understanding
during model inference. Haurum et al. (2023) investigates the limitations of
factual knowledge in LLMs and optimizes the reasoning process with minimal
retrieval cost. Optimization through fine-tuning is a recent advance, Yu et
al. (2023); Zhou et al. (2020) proposes a black-box fine-tuning framework that
accesses only API to optimize continuous prompts using non-derivative methods.
Model alignment Liu et al. (2023c) and chain of thought reasoning Wu et al.
(2023), are also key focuses in prompt optimization.

The improvement in invocation performance through prompt augmentation is
significant, despite the likelihood of resulting in more complex processing
procedures. And the general method is difficult to explore, which requires
some professional knowledge.

##  4 Semantic Cache

Semantic cache is an approach to improve LLM invocation efficiency and
performance by storing and quickly retrieving semantic information. Different
from traditional data cache, semantic cache focuses more on storing high-level
semantics such as meaning, context and relationship of data, rather than just
raw data. The semantic cache is checked before the service is invoked. If hit,
the output given by the cache is returned without the follow cumbersome
procedures.

Cache technology often requires long-term data accumulation and is not
suitable for cold start scenarios. However, with the gradual increase in the
scale of LLM, it plays a more important role in accelerating computation,
reducing data transmission costs, and supporting high concurrent requests Miao
et al. (2023), providing users with low-cost, low-latency and high-performance
services.

There are two typical structures for the implementation of semantic cache in
LMaaS, and unlike other subsections, they generally cannot be used together.
Traditional caches use key-value pairs for storage and retrieval. When similar
input appears again, the system can quickly search the semantic cache by the
key and return the same value. Neural cache borrows ideas from neural networks
to response in a predictive rather than retrieval way. It learns semantic
relationships between input data without relying on a specific storage
structure.

###  4.1 Traditional Cache

The current paradigm of traditional cache consists of three parts: the cache
manager, similarity evaluator, and post processor. The cache manager is
responsible for storing content in the form key-value pairs, and managing
eviction. The similarity evaluator is used to determine if any of the keys in
the cache match the input query. The post processor organizes the final
response to be returned to the user. If no similar query is found in the
cache, the LLM service is invoked by the post processor to generate the output
and then the generated output is stored in the cache.

Bang (2023) represents a typical application of traditional cache, which
utilizes question embeddings for similarity matching and provides various
matching methods such as exact match and embedding distance. The open-source
application Zep 333Zep: Fast, scalable building blocks for LLM apps
(https://github.com/getzep/zep)also supports storage, aggregation, embedding,
and indexing of LLM applications. Through theoretical proof, Zhu et al. (2023)
gives the cache scheme with minimum expected cost considering the query
frequency. Furthermore, methods for query and conversations cache Tao et al.
(2021); Barrios and Kumar (2024) can be easily migrated to LMaaS.

Implementing traditional cache is usually relatively simple, requiring only
basic data structures such as hash that are easy to manage. This approach is
general, but it may not capture semantic similarity between inputs because it
relies heavily on the key matching.

###  4.2 Neural Cache

Neural cache uses neural networks or deep learning models to learn and store
data representations. It maps the input data into a high-dimensional space by
learning the representation of the data. The learned representation should
capture the semantic similarity of the input data so that similar inputs are
close in the representation space.

Ramírez et al. (2023) trains a student model using T5-base444T5-base model
(https://huggingface.co/docs/transformers/model_doc/t5) for providing early
feedback in classification tasks, and the model is periodically updated. To
address the effectiveness testing issue of semantic cache, Rasool et al.
(2024) generates similar input to hit the cache as much as possible.
Additionally, a retrieval-based dialogue response selection model can also
serve as an alternative choice. Tao et al. (2021) provides a survey that
categorizes most models into three framework and among them, representation-
based model can be used as neural cache.

These types of methods often outperform traditional cache especially in
domain-specific problems. However, their implementation and updates can be
relatively complex. And it is important to carefully consider the
effectiveness of the cache to avoid incurring unnecessary waste.

Figure 4: A simple invocation strategy composed of existing methods, using
Prompt Reducer in input abstract, Zep in semantic cache, FrugalGPT in solution
design, and nothing in output enhancement.

##  5 Solution Design

Solution design is an approach to leveraging LLM services with heterogeneous
costs and performance. It considers different scenarios and different targets,
dynamically selects one or more LLM services that are most suitable for a
specific invocation according to the query, and organizes them in some form to
provide flexible and efficient solutions. This methods allows users to select
LLM services that best suit their specific needs. When new queries arise or
requirements change, the configuration of the solution can be flexibly updated
to achieve the best performance and cost effectiveness.

Solution design has two main parts that work together to achieve dynamic LLM
services selection and routing. The scoring function is responsible for
evaluating the performance of each available LLM service, which can reflect
the concerned indicators of the invocation such as quality, speed, and so on.
The router, based on the evaluation results of the scoring function, performs
query routing between services and selects the appropriate one in a dynamic
manner.

###  5.1 Scoring Function

The scoring function is a comprehensive evaluation of LLM services given a
specific task or query, considering both targets and scenarios, and often used
to guide the routing path in the solution. It may be influenced by multiple
factors such as response time, query cost, accuracy of answers, etc. The
scoring function plays a decision-making role, and helps to understand the
relative performance of each LLM service so that it can make more intelligent
choices.

Defined metrics. Defined metrics provide a measurable way to achieve direct
quantification of the factors of concern. For instance, accuracy in
classification tasks, BLEU score in generation tasks, metrics like packet
loss, and quality of service (QoS) are all applicable indicators. Ramírez et
al. (2023) use interval sampling and predictive entropy to determine whether
to invoke LLM services for different time dimensions of the invocation.
Considering three sources of consistency, decision-making for LLM services is
performed through sampling and voting Yue et al. (2023). Cost expectations
between two models are calculated, Zhu et al. (2023) extend the selection
invocation to multiple LLMs. Reward ranking from answers provided by different
services is used as an evaluation criterion by Lu et al. (2023), incurring
minimal computational cost in the solution.

The defined metrics are intuitive and easily understandable. They are often
based on statistical data or experiments, providing high reliability and being
less susceptible to subjective factors. However, setting thresholds can be
challenging and may not adapt well to dynamic and changing environments.
Additionally, certain crucial factors may be difficult to capture with
specific metrics, leading to limitations in scoring.

Scorers. A scorer is a tool for scoring each LLM services based on metrics
that are not defined by a particular formula. The scorer utilizes prior
knowledge, training data, or rules to provide scores in an often less
interpretable manner, typically using smaller neural networks Chen et al.
(2023). AlBert is employed as a scorer, with the query and predicted output as
$x$, the accuracy of predicted output and label as $y$ during training Sakota
et al. (2023). Another methods involves using DistilBert as a scoring model,
with query and model ID as $x$, and whether it can solve the problem as $y$
during training Shnitzer et al. (2023). A comparison of LLM performances on
different benchmark datasets is conducted, Zhang et al. (2023) modeling it as
a binary selection problem, providing guiding suggestions. For specific tasks,
such as the execution results in the task of code generation Zhang et al.
(2023), the classifier according to the query difficulty in the task of
question and answer Anonymous (2024); Madaan et al. (2023), and estimate the
capabilities of LLM services in the task of dataset benchmark test Shnitzer et
al. (2023) are all reasonable scorers.

Compared to metrics defined by formulas, scorers can be updated based on real-
time data and feedback, demonstrating strong generalization across different
scenarios. However, this approach is equivalent to scoring using a more
powerful model, incurring scorers own training and usage costs. And it still
requires some labeled examples, leading to the fact that it makes sense only
when the query dataset is larger than the training.

###  5.2 LLM Router

LLM routing emphasizes the organizational structure between services,
connecting multiple independent services in a specific order logically. It
focus on constructing a flexible and reusable LLM service solution to address
continuously changing queries or targets. Depending on the different scoring
function and position used, LLM routing can construct target-oriented
solution, such as cost-oriented or performance-oriented.

Sequential Structure. The simplest method is to select one or several models
from the massive LLM services available and invoke them sequentially. A
scoring function is employed to decide whether to accept the answer or proceed
to the next step in routing Chen et al. (2023). When using a sequential
structure, the number of models is typically limited to three, and possible
options are determined through permutation, with pruning techniques applied
Ramírez et al. (2023); Yue et al. (2023). The use of small models as a cache,
with large models being invoked in sequence when cache misses occur, can be
considered a fixed sequential structure Ramírez et al. (2023); Yue et al.
(2023). For problems like code generation Zhang et al. (2023), an initial
response is obtained using a cost-effective LLM, and successful information is
tracked as context for subsequent queries.

This structure is simple and effective, and a limited number of permutations
can be searched quickly in the entire space. However, the sequential structure
may result in the invocation of all models in the sequential structure. And it
is difficult to extend the structure, which all models need to be rearranged
when adapting to new requirements.

Other Structure. A parallel structure, similar to the bagging and boosting in
machine learning, can enhance the correctness and consistency of LLM services,
with task decomposition and merging being key aspects Jiang et al. (2023). A
star-shaped structure, as seen in Sakota et al. (2023); Lu et al. (2023),
involves decision-making by a meta-model, allocating the current query to the
most suitable model. For the third category of unsolvable queries, pruning is
applied by Madaan et al. (2023) to prevent unnecessary expenses for
particularly challenging problems. The tree structure is considered promising,
combining aspects of both star-shaped and sequential structures. It initially
routes query to the most probable branch, and then invokes services in
sequence. Additionally, certain selection solutions specific to HTTP services
Hosseinzadeh et al. (2020); Manqele et al. (2017) are also noteworthy for
inspiration.

##  6 Output Enhancement

Output Enhancement refers to the process of further optimizing and adjusting
the output generated by the invocation. This process aims to improve the
syntactic correctness, semantic accuracy, and overall fluency of the generated
output to meet the needs of the user and the specific scenario.

To the best of our knowledge, the output enhancement methods still relies on
the methods mentioned above, but it emphasizes customization according to the
needs of specific tasks, improving the adaptability of the model to
application, reducing the need for subsequent manual intervention, and
providing users with low-latency, high-performance, and low-latency services.
For example, Liu et al. (2023b) guides the LLM to make concise answers can
reduce unnecessary output tokens. Aggregating responses from multiple low-cost
models is another way to improve quality Chen et al. (2022), and is often used
for multi-label tasks. Work on model alignment Shen et al. (2023) can also be
used to correct syntax and logic to reduce the need for subsequent manual
work.

##  7 Conclusion and Challenges

In conclusion, this paper has provided a comprehensive overview of effective
invocation methods in the realm of LMaaS. Through the establishment of a
taxonomy, we categorize existing methods into four categories: input abstract,
semantic cache, solution design, and output enhancement. Then we formalize the
problem of effective LLM services strategy construction, and propose a LLM
services invocation framework. Each component in the framework can work
independently or simultaneously to form effective strategy for LLM service
invocation that are low-latency, high-performance, and cost-saving.

Existing methods tend to focus on only one component of the framework, and we
can use them as plugins. A case is shown in Figure 4, a simple invocation
strategy constructed from three existing methods. The development prospects of
this field are promising, and here are some open challenges.

Input Abstract. In the input abstract component, one of the main challenges
faced is multi-modal input processing Yin et al. (2023). More comprehensive
and balanced methods are needed to shorten and optimize multiple types of
input such as text, image, and speech. Input abstract methods to dynamically
changing inputs are also worth exploring, such as real-time data streaming
Räth et al. (2023) or user interaction with the system. Furthermore, depending
on the granularity, the input abstract can also be divided into document
level, sentence level and phrase level. methods at different granularities may
interoperate and often compose multi-stage methods.

Semantic Cache. In the semantic cache part, how to design and select cache
methods Brais et al. (2021) more efficiently to accommodate different types of
inputs and queries is the main challenge faced in traditional cache, while
semantic representation Brito (2023) is concerned by neural cache.

Solution Design. In terms of solution design, the evaluation problem of LLM
services Chang et al. (2023) is an extension of scoring function, which needs
to pay more attention to adaptation and interpretability in the future. While
LLM router will focus on designing more powerful services integration methods,
not only focusing on the task itself, but also considering the requirements of
different resources Xu et al. (2024). A more effective combination of the two,
such as dynamic decision making, will lead to a better solution.

Output Enhancement. The importance of output enhancement is also gradually
seen by people. The balance between specification and diversity of output is a
key issue. When the task is completed, the user’s satisfaction is then an
important indicator to measure the service quality, and future research may
focus on building more intelligent and user-oriented Jeung and Huang (2023)
output enhancement methods.

Other Chanllenges. Basic work such as qualitative description and quantitative
comparison in experiments is still a gap to be filled, and the lack of data
sets makes there is no uniform standard for the comparison of service methods.
Some technical details, such as how to choose the tokenizer Alyafeai et al.
(2023) with the shortest input, the guidance on the cache size Vavouliotis et
al. (2022), and the choice of different pricing methods for the same LLM
service, need to be explored. In additionally, We specifically call for
attention to fairness Sah et al. (2024) and privacy issues Luo et al. (2024);
Utpala et al. (2023) in LMaaS. The efficient construction of methods using
middleware could potentially be exploited for personal gain or malicious
purposes. We look forward to future research further advancing the field,
providing users with low-latency, high-performance, and cost-effective LLM
services solution, and promoting the healthy development of the LMaaS
ecosystem.

## References

  * Ahia et al. [2023] Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov.  Do all languages cost the same? tokenization in the era of commercial language models.  In Proc. of EMNLP, 2023.
  * Alyafeai et al. [2023] Zaid Alyafeai, Maged Saeed AlShaibani, Mustafa Ghaleb, and Irfan Ahmad.  Evaluating various tokenizers for arabic text classification.  Neural Process. Lett., 2023.
  * Anonymous [2024] Anonymous.  Hybrid LLM: Cost-efficient and quality-aware query routing.  In Proc. of ICLR, 2024.
  * Antony et al. [2023] Dinu Antony, Sumit Abhishek, Sujata Singh, Siddu Kodagali, Narayana Darapaneni, Mukesh Rao, Anwesh Reddy Paduri, and Sudha BG.  A survey of advanced methods for efficient text summarization.  In 13th IEEE Annual Computing and Communication Workshop and Conference, CCWC 2023, Las Vegas, NV, USA, March 8-11, 2023, 2023.
  * Arefeen et al. [2023] Md. Adnan Arefeen, Biplob Debnath, and Srimat Chakradhar.  Leancontext: Cost-efficient domain-specific question answering using llms.  CoRR, 2023.
  * Bai et al. [2024] Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, Carl J. Yang, Yue Cheng, and Liang Zhao.  Beyond efficiency: A systematic survey of resource-efficient large language models.  CoRR, 2024.
  * Bang [2023] Fu Bang.  Gptcache: An open-source semantic cache for llm applications enabling faster answers and cost savings.  2023\.
  * Barrios and Kumar [2024] Carlos Barrios and Mohan Kumar.  Service caching and computation reuse strategies at the edge: A survey.  ACM Comput. Surv., 2024.
  * Brais et al. [2021] Hadi Brais, Rajshekar Kalayappan, and Preeti Ranjan Panda.  A survey of cache simulators.  ACM Comput. Surv., 2021.
  * Brito [2023] Eduardo Brito.  Explainable Resource-Aware Representation Learning via Semantic Similarity.  PhD thesis, 2023.
  * Chang et al. [2023] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie.  A survey on evaluation of large language models.  CoRR, 2023.
  * Chen et al. [2022] Lingjiao Chen, Matei Zaharia, and James Zou.  Efficient online ML API selection for multi-label classification tasks.  In Proc. of ICML, 2022.
  * Chen et al. [2023] Lingjiao Chen, Matei Zaharia, and James Zou.  Frugalgpt: How to use large language models while reducing cost and improving performance.  CoRR, 2023.
  * Dong et al. [2023] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui.  A survey on in-context learning, 2023.
  * Haurum et al. [2023] Joakim Bruslund Haurum, Sergio Escalera, Graham W. Taylor, and Thomas B. Moeslund.  Which tokens to use? investigating token reduction in vision transformers.  In Proc. of ICCV, 2023.
  * Hosseinzadeh et al. [2020] Mehdi Hosseinzadeh, Hawkar Kamaran Hama, Marwan Yassin Ghafour, Mohammad Masdari, Omed Hassan Ahmed, and Hemn Khezri.  Service selection using multi-criteria decision making: A comprehensive overview.  J. Netw. Syst. Manag., 2020.
  * Huang et al. [2021] Yi-Chong Huang, Xia-Chong Feng, Xiao-Cheng Feng, and Bing Qin.  The factual inconsistency problem in abstractive text summarization: A survey.  CoRR, 2021.
  * Jeung and Huang [2023] Jun Li Jeung and Yi-Ching Janet Huang.  Correct me if I am wrong: Exploring how AI outputs affect user perception and trust.  In Computer Supported Cooperative Work and Social Computing, CSCW 2023, Minneapolis, MN, USA, October 14-18, 2023, 2023.
  * Jiang et al. [2023] Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin.  Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.  In Proc. of ACL, 2023.
  * Kim et al. [2022] Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer.  Learned token pruning for transformers.  In Proc. of KDD, 2022.
  * Lai et al. [2023] Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Nguyen.  Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning.  In Proc. of EMNLP Findings, 2023.
  * Li et al. [2023] Jiazheng Li, Runcong Zhao, Yulan He, and Lin Gui.  Overprompt: Enhancing chatgpt capabilities through an efficient in-context learning approach.  CoRR, 2023.
  * Liu et al. [2023a] Junyi Liu, Liangzhi Li, Tong Xiang, Bowen Wang, and Yiming Qian.  TCRA-LLM: token compression retrieval augmented large language model for inference cost reduction.  In Proc. of EMNLP Findings, 2023.
  * Liu et al. [2023b] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  ACM Comput. Surv., 2023.
  * Liu et al. [2023c] Yixin Liu, Budhaditya Deb, Milagro Teruel, Aaron Halfaker, Dragomir Radev, and Ahmed Hassan Awadallah.  On improving summarization factual consistency from natural language feedback.  In Proc. of ACL, 2023.
  * Lu et al. [2023] Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou.  Routing to the expert: Efficient reward-guided ensemble of large language models.  CoRR, 2023.
  * Luo et al. [2024] Jinglong Luo, Yehong Zhang, Jiaqi Zhang, Xin Mu, Hui Wang, Yue Yu, and Zenglin Xu.  Secformer: Towards fast and accurate privacy-preserving inference for large language models.  CoRR, 2024.
  * Madaan et al. [2023] Aman Madaan, Pranjal Aggarwal, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, Shyam Upadhyay, Mausam, and Manaal Faruqui.  Automix: Automatically mixing language models.  CoRR, 2023.
  * Manqele et al. [2017] Lindelweyizizwe Manqele, Mqhele E. Dlodlo, Louis Coetzee, and George Sibiya.  A survey for service selection approaches in dynamic environments.  In IEEE AFRICON 2017, Cape Town, South Africa, September 18-20, 2017, 2017.
  * Miao et al. [2023] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, and Zhihao Jia.  Towards efficient generative large language model serving: A survey from algorithms to systems.  CoRR, 2023.
  * Mridha et al. [2021] Muhammad F. Mridha, Aklima Akter Lima, Kamruddin Nur, Sujoy Chandra Das, Mahmud Hasan, and Muhammad Mohsin Kabir.  A survey of automatic text summarization: Progress, process and challenges.  IEEE Access, 2021.
  * Ramírez et al. [2023] Guillem Ramírez, Matthias Lindemann, Alexandra Birch, and Ivan Titov.  Cache & distil: Optimising API calls to large language models.  CoRR, 2023.
  * Rasool et al. [2024] Zafaryab Rasool, Scott Barnett, David Willie, Stefanus Kurniawan, Sherwin Balugo, Srikanth Thudumu, and Mohamed Abdelrazek.  Llms for test input generation for semantic caches, 2024.
  * Räth et al. [2023] Timo Räth, Ngozichukwuka Onah, and Kai-Uwe Sattler.  Interactive data cleaning for real-time streaming applications.  In Proceedings of the Workshop on Human-In-the-Loop Data Analytics, HILDA 2023, Seattle, WA, USA, 18 June 2023, 2023.
  * Reddy et al. [2019] Siva Reddy, Danqi Chen, and Christopher D. Manning.  Coqa: A conversational question answering challenge.  Trans. Assoc. Comput. Linguistics, 2019.
  * Sah et al. [2024] Chandan Kumar Sah, Xiaoli Lian, and Muhammad Mirajul Islam.  Unveiling bias in fairness evaluations of large language models: A critical literature review of music and movie recommendation systems.  CoRR, 2024.
  * Saha et al. [2023] Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li.  Branch-solve-merge improves large language model evaluation and generation.  CoRR, 2023.
  * Sakota et al. [2023] Marija Sakota, Maxime Peyrard, and Robert West.  Fly-swat or cannon? cost-effective language model choice via meta-modeling.  CoRR, 2023.
  * Santra et al. [2023] Bishal Santra, Sakya Basak, Abhinandan De, Manish Gupta, and Pawan Goyal.  Frugal prompting for dialog models.  In Proc. of EMNLP Findings, 2023.
  * Shen et al. [2023] Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong.  Large language model alignment: A survey.  CoRR, 2023.
  * Shnitzer et al. [2023] Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin.  Large language model routing with benchmark datasets.  CoRR, 2023.
  * Si et al. [2023] Wai Man Si, Michael Backes, and Yang Zhang.  Mondrian: Prompt abstraction attack against large language models for cheaper API pricing.  CoRR, 2023.
  * Tao et al. [2021] Chongyang Tao, Jiazhan Feng, Rui Yan, Wei Wu, and Daxin Jiang.  A survey on response selection for retrieval-based dialogues.  In Proc. of IJCAI, 2021.
  * Utpala et al. [2023] Saiteja Utpala, Sara Hooker, and Pin-Yu Chen.  Locally differentially private document generation using zero shot prompting.  In Proc. of EMNLP Findings, 2023.
  * Vavouliotis et al. [2022] Georgios Vavouliotis, Gino Chacon, Lluc Alvarez, Paul V. Gratz, Daniel A. Jiménez, and Marc Casas.  Page size aware cache prefetching.  In Proc. of MICRO, 2022.
  * Watanangura et al. [2024] Patcharapruek Watanangura, Sukit Vanichrudee, On Minteer, Theeranat Sringamdee, Nattapong Thanngam, and Thitirat Siriborvornratanakul.  A comparative survey of text summarization techniques.  SN Comput. Sci., 2024.
  * Wu et al. [2023] Dingjun Wu, Jing Zhang, and Xinmei Huang.  Chain of thought prompting elicits knowledge augmentation.  In Proc. of ACL Findings, 2023.
  * Xu et al. [2024] Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, Qiyang Zhang, Zhenyan Lu, Li Zhang, Shangguang Wang, Yuanchun Li, Yunxin Liu, Xin Jin, and Xuanzhe Liu.  A survey of resource-efficient LLM and multimodal foundation models.  CoRR, 2024.
  * Yang et al. [2023] Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu.  Harnessing the power of llms in practice: A survey on chatgpt and beyond.  CoRR, 2023.
  * Yin et al. [2023] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen.  A survey on multimodal large language models.  CoRR, 2023.
  * Yu et al. [2023] Lang Yu, Qin Chen, Jiaju Lin, and Liang He.  Black-box prompt tuning for vision-language model as a service.  In Proc. of IJCAI, 2023.
  * Yue et al. [2023] Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao.  Large language model cascades with mixture of thoughts representations for cost-efficient reasoning.  CoRR, 2023.
  * Zhang et al. [2023] Jieyu Zhang, Ranjay Krishna, Ahmed Hassan Awadallah, and Chi Wang.  Ecoassistant: Using LLM assistant more affordably and accurately.  CoRR, 2023.
  * Zhou et al. [2020] Jianyi Zhou, Feng Li, Jinhao Dong, Hongyu Zhang, and Dan Hao.  Cost-effective testing of a deep learning model through input reduction.  In 31st IEEE International Symposium on Software Reliability Engineering, ISSRE 2020, Coimbra, Portugal, October 12-15, 2020, 2020.
  * Zhu et al. [2023] Banghua Zhu, Ying Sheng, Lianmin Zheng, Clark W. Barrett, Michael I. Jordan, and Jiantao Jiao.  On optimal caching and model multiplexing for large model inference.  CoRR, 2023.


# PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech
Representations

###### Abstract

While self-supervised speech representation learning (SSL) models serve a
variety of downstream tasks, these models have been observed to overfit to the
domain from which the unlabeled data originates. To alleviate this issue, we
propose PADA (Pruning Assisted Domain Adaptation). Before performing the
_target-domain_ ASR fine-tuning, we discover the redundant weights from pre-
trained wav2vec 2.0 models through various pruning strategies. We investigate
the effect of _Task-Agnostic_ and _Task-Aware_ pruning and propose a new
pruning paradigm called, _Cross-Domain Task-Aware Pruning_ (CD-TAW). CD-TAW
obtains the initial pruning mask from a well fine-tuned _out-of-domain_ (OOD)
model, thereby making use of the readily available fine-tuned models from the
web. The proposed CD-TAW method achieves up to 20.6% relative WER improvement
over our baseline when fine-tuned on a 2-hour subset of Switchboard data
without language model (LM) decoding.

Index Terms—  domain adaptation, pruning, self-supervised learning, automatic
speech recognition, telephone speech

##  1 Introduction

Over the past decade, Automatic Speech Recognition (ASR) has drawn the
attention of researchers from various fields owing to its potential
applications in various Natural Language Understanding (NLU) systems having
speech as the primary modality of communication [1, 2, 3, 4]. The advent of
Deep Neural Networks (DNNs) has pushed the state-of-the-art (SOTA) in speech
recognition in a variety of settings [5, 6]. However, DNNs are resource-
hungry, and building efficient ASR systems requires a lot of compute and
supervision in the form of labeled data. Thus, recent Self-Supervised Learning
(SSL) approaches [6, 7, 8, 9] which can learn representations from unlabeled
audio data directly, have been gaining much traction. The primary aim of SSL
is to use raw speech [6], or other low-level features like Filter Banks [10],
to learn high-level representations that prove to be effective in other
downstream speech processing tasks. SSL has shown considerable performance
boosts in building ASR systems, especially in settings where labeled data is
scarce (as low as 10 minutes), and has been known to generalize better than
supervised learning.

However, several recent studies have highlighted the drawbacks of SSL.
Firstly, the pretext tasks that the system is subjected to solve under the SSL
paradigm are compute-intensive [6] and require a lot of unlabeled data.
Secondly, a recent study reveals that similar to supervised learning, SSL too
gets biased to the domain from which the unlabeled data originates [11].
Thirdly, as SSL implicitly learns a language model and other semantic
information through the tasks it is subjected to solve [12], the
generalizability of these models is only to the extent where data from a
similar language or phonetic structure is introduced to it at the fine-tuning
stage. Thus, as correctly pointed out by [13], SSL for speech suffers from
scale problems, and SSL generalizability can be improved with more efficient
training procedures. Prior work for domain adaptation with self-supervised
models mostly employ _continued pre-training_ or _combined data pre-training_
approaches [11]. However, both assume the existence of high-resource unlabeled
target-domain data, which is not always the case in a real-world scenario (for
example, telephonic conversational speech is very difficult to procure due to
privacy issues and not more than 1000 hours is freely available online).

Building on the second and third problems mentioned above, in this paper, we
try to address the problem of domain bias in pre-trained SSL models and try to
devise an algorithm, that can allow pre-trained models trained on OOD data to
easily adapt to the target domain and with improved performance, using only
the supervised In-Domain data. To achieve this objective, we take the help of
Unstructured Magnitude Pruning (UMP), wherein we select model parameters of
“least magnitude”, which we hypothesize to be of “least importance” to the
domain, and propose to zero them out so that weights that are important for
the downstream task should emerge with gradient updates, and those that are
irrelevant should decrease in magnitude. This was first empirically proven in
[14]. The zeroed-out parameters are also kept trainable, which makes this
different from generic DNN pruning, where the pruning mask does not allow
gradient updates. More details about the different pruning strategies can be
found in Alg.1. We are motivated by the findings from the Lottery Ticket
Hypothesis [15], which suggests that a randomly initialized network contains
sparse sub-networks which can be trained in isolation to match the result of a
full model. Also, [16] explores pruning for monolingual language adaptation in
multilingual pre-trained models. To summarize, our main contributions are as
follows:

  * •

We analyze the performance of different pruning strategies and frequencies for
domain adaptation on pre-trained speech SSL models. We base our experiments on
the practical assumption that only limited amounts of target-domain labeled
data is available, and no other large-scale unlabeled corpus is available from
the target domain.

  * •

We propose _Cross-Domain Task-Aware pruning_ (CD-TAW), a first-of-its-kind
method that uses readily available models fine-tuned on high-resource labeled
OOD data to obtain the initial pruning mask relevant to the downstream ASR
task. Experimental results reveal that CD-TAW performs better than generic
fine-tuning, TAG, and TAW approaches.

Fig. 1: The white blocks represent the weights that have been zeroed out.

##  2 Related Work

###  2.1 Self-Supervised Speech Representation Learning

SSL for speech representation learning has been explored primarily under 3
main paradigms, each solving a form of Masked Acoustic Modelling (MAM). The
first and the most common in this space is based on Contrastive Predictive
Coding (CPC), which minimizes the InfoNCE loss [17, 6]. The second paradigm
learns by minimizing the Cross-Entropy loss, wherein the primary task is to
predict the correct pseudo-label assigned to a frame, masked at the input to
the model, leveraging the contextualized embedding of that frame obtained from
a transformer encoder [8, 9]. The third and the final paradigm solves the
reconstruction-based objective function [10, 18].

###  2.2 Pruning

The concept of pruning DNNs has been extensively studied in the past [15, 19,
20]. Authors of [21] show that large sparse models obtained through pruning
generally perform better when compared to smaller but dense models. One of the
first works that proposed pruning for domain adaptation was by [22]. However,
they propose an entirely different methodology using knowledge distillation
and employ it for text-based Neural Machine Translation (NMT). [16] propose a
pruning approach to improve monolingual ASR performance in multilingual SSL
pre-trained models. Authors of PARP [14] proposed the first work on pruning
large self-supervised speech pre-trained models. They intend to find a sparse
fine-tuned sub-network at some target sparsity level, with improved
performance than using a full model. However, our work differs from theirs
primarily from the perspective of the final task we try to solve. We approach
iterative UMP [15] from a domain adaptation perspective and devise our
algorithm by hypothesizing accordingly. Our end goal is to adapt the pre-
trained SSL model to the target-domain, and the focus is not on obtaining a
sparse model. One key difference from PARP is that our _task-aware_ pruning
strategies work better for domain adaptation. At the same time, PARP resorts
to _task-agnostic_ pruning strategies as they do not observe major differences
between the two, given that they fine-tune on the data from the same domain as
the pre-trained model’s unlabeled data. Another key difference from PARP is
that, while they progressively increase the pruning percentage to achieve the
target sparsity in PARP-P, we dynamically decay the pruning percentage for
better domain adaptation as explained in Section 3.2.4.

###  2.3 Domain Adaptation

Domain Adaptation (DA) for building efficient ASR systems has been a well-
studied topic in literature, with early work focusing on regularization [23,
24], teacher-student learning, [25, 26] or adversarial learning [27, 28].
Lately, unsupervised domain adaptation of ASR models has been gaining
traction, and researchers have been trying to find ways to use huge amounts of
unlabeled data from the target domain for domain adaptation [26, 29, 30].
Continued pre-training is another common approach used [31]. We emphasize that
our work is one of the first to approach domain adaptation from a pruning
perspective, involving no continued pre-training or OOD unlabeled data.

##  3 Proposed Methodology

###  3.1 Problem Formulation

Suppose we have a high-resource OOD unlabeled dataset P and a medium-to-high
resource OOD labeled dataset J, both from domain $D_{1}$. We also have a low-
resource labeled dataset L. Let $p(\theta)$ be the neural network pre-trained
using self-supervision on the dataset P from domain $D_{1}$ and let
$f(\theta_{j})$ represent the resultant neural network after CTC based ASR
fine-tuning has been performed on $p(\theta)$ on the dataset J from domain
$D_{1}$, such that $\theta$,$\theta_{j}\in\mathbb{R}^{d}$, represents the $d$
number of network parameters. As a part of this work, the models $p(\theta)$
and $f(\theta_{j})$ that we use are made available in the public domain
through the works on wav2vec-2.0 and fairseq [6, 32].

Our primary aim is to adapt our model $p(\theta)$, to domain $D_{2}$ by just
using the low-resource dataset L, with performance better than generic fine-
tuning of $p(\theta)$ on L. We achieve this objective by identifying the
weights having the least importance and zeroing them out. Weights relevant for
the downstream ASR task on L would emerge with gradient updates, and those
irrelevant are zeroed out to facilitate better adaptation of the model. As we
aim to get a model that delivers better performance without on any focus on
achieving a target sparsity, weights that have been zeroed out remain
trainable on L, thereby making use of the entire model capacity.

###  3.2 Pruning Assisted Domain Adaptation (PADA)

The generic pruning strategy in modern deep learning toolkits involves
obtaining a mask M for a subset of model parameters, which intuitively results
in multiplying by a value of zero in the forward pass step. Additionally, the
mask does not allow gradient updates for these parameters during the backward
pass. As part of this work, we explore different pruning strategies based on
UMP. The focus of PADA is to discover the “least important weights” in a pre-
trained SSL model $p(\theta)$. Unlike generic pruning, PADA does not retain
the mask and lifts the mask after a round of UMP, followed by zeroing out the
identified parameters, and keeps the parameters trainable. PADA can be
achieved in multiple ways, primarily differing in the procedure of obtaining
the initial pruning masks at the beginning of training and the rate of pruning
employed at every training iteration. Fig. 1 illustrates the three major
pruning strategies. In the following four subsections, we briefly discuss the
pruning strategies involved in PADA and the algorithm to implement the same.

####  3.2.1 Task-Agnostic Pruning (TAG)

In this approach we perform UMP at a pruning percentage $r_{1}$ directly on
$p(\theta)$, and the $r_{1}$ percent of weights with the least magnitude are
zeroed out, resulting in the model $p(\theta_{0})$. This approach does not
consider the downstream ASR fine-tuning that $p(\theta)$ would be subjected
to. Weights that have been zeroed out are identified to be of least importance
in terms of magnitude, only based on the pre-training task. One must note that
pre-training is used to learn representations that serve multiple downstream
tasks, and fine-tuning the model on specific downstream tasks and datasets
could lead to a different set of weights gaining importance.

####  3.2.2 Task-Aware Pruning (TAW)

The Task-Aware Pruning strategy aims to zero out those weights from
$p(\theta)$, that are of least importance specific to the downstream ASR task
performed on the dataset L. To this end, we first fine-tune $p(\theta)$ on L
which results in the fine-tuned model $p(\theta_{f})$. We then perform UMP at
a pruning percentage $r_{1}$ on $p(\theta_{f})$, to obtain the mask M. The
mask M carries the information about the least important weights when
$p(\theta)$ is exposed to the ASR downstream task on the dataset L. The exact
same weights specified by the mask M are zeroed out from $p(\theta)$,
resulting in the model $p(\theta_{0})$.

####  3.2.3 Cross-Domain Task-Aware Pruning (CD-TAW)

While TAW focuses on identifying and zeroing the weights of least importance
specific to the downstream ASR task on the dataset L, it is evident that such
an approach requires fine-tuning twice on the dataset L (once to obtain the
mask and the second time to fine-tune the pre-trained model carrying $r_{1}$
percent of zeroed weights). To alleviate this compute-heavy issue, we propose
a method where UMP is performed at a pruning percentage $r_{1}$ on
$f(\theta_{j})$, to obtain the mask M. Similar to TAW, the exact same weights
specified by the mask M are zeroed out from $p(\theta)$, resulting in the
model $p(\theta_{0})$. The advantage of using $f(\theta_{j})$ to obtain the
mask M is that, J is a relatively high resource dataset (though OOD) in
comparison with L. This results in the fine-tuning mask being more suited to
the downstream ASR task, as $f(\theta_{j})$ has seen more “task” related data.
The reason behind mentioning that CD-TAW requires only one round of fine-
tuning is that, when $f(\theta_{j})$ is readily available online, it saves the
explicit fine-tuning on the downstream ASR task to obtain a task-aware mask.

####  3.2.4 Algorithm

Alg. 1 shows the algorithm of PADA where we describe PADA with notations from
earlier subsections.

We fix the initial pruning percentage $r_{1}$. We then choose the pruning
frequency $P_{freq}$ from the 3 variants (Once / Iterative / Dynamic
Iterative). We also fix the total number of fine-tuning updates to $N$.

Algorithm 1 Pruning Assisted Domain Adaptation (PADA)

1:Based on the pruning strategy chosen (TAG / TAW / CD-TAW), obtain the model
$p(\theta_{0})$ after zeroing out $r_{1}$ percentage of weights.

2:if $P_{freq}$ is Once then

3: Fine-tune $p(\theta_{0})$ on L for $N$ model updates to obtain the final
domain adapted model $p(\theta_{N})$.

4:else if $P_{freq}$ is Iterative or Dynamic Iterative then

5: Choose the number of fine-tune updates $n$ after which you wish to re-prune
and zero out some model weights.

6: Fix the pruning percentages $r_{2},r_{3},\cdots,r_{k}$.

7: Set $updates=0$ ; $n1=0$ and $u=1$.

8: while $updates\leq N$ do

9: Train $p(\theta_{n1})$ on L, for $n$ updates to obtain $p(\theta_{n2})$.

10: Set $n1=n2$ ; $updates=updates+n$.

11: if $updates\leq N$ and $u<k$ then

12: Perform UMP on $p(\theta_{n1})$ at a pruning percentage $r_{u}$ and zero
out the pruned weights.

13: Set $u=u+1$.

14: end if

15: end while

16:end if

When an Iterative PADA is used, frequency $r_{i}$ corresponds to
$r_{1}=r_{1}=r_{2}=\cdots=r_{k}$. However, when the Dynamic Iterative PADA is
chosen, $r_{1}>r_{2}>r_{2}>\cdots>r_{k}$. Dynamic Iterative PADA has the
fundamental advantage in which fewer weights are zeroed out in the future
training iterations where the model has “already been trained on the target
domain” and needs “lesser space” to adapt. Precisely, each of the pruning
strategies mentioned in sections 3.2.1, 3.2.2, 3.2.3 differ only in step 1 of
the PADA algorithm. However, it is worth mentioning that step 1 of PADA is the
most important step, which makes all the difference in terms of performance.

##  4 Experimental Setup

###  4.1 Datasets and Pre-trained models

The pre-trained models which we choose for PADA are:

  * •

wav2vec-2.0 LV-60k: A LARGE model [6] pre-trained on 60k hours of Libri-Light
audio data.

  * •

wav2vec-2.0 LS-960: A BASE model [6] pre-trained on 960 hours of LibriSpeech
audio data.

  * •

XLSR-53: A LARGE model pre-trained on 56k hours of data from 53 different
languages [33]. The datasets used include Multilingual LibriSpeech (MLS),
CommonVoice (CV), and Babel.

The target domain datasets we choose for PADA are:

  * •

Switchboard data : It is a corpus of telephonic speech conversations [34], a
domain completely different from the Libri-Light data which is made up of read
speech from audiobooks [35]. We choose two subsets of the Switchboard data for
our experiments, one of 30 hours and another of 2 hours, representing the low-
resource and the extreme low-resource settings of domain adaptation. We report
the word error rate (WER) results on the Switchboard dev set.

  * •

Hindi Challenge Data : It is a corpus of Hindi read speech data released as a
part of the Hindi ASR Challenge111https://sites.google.com/view/asr-challenge,
which has 50 hours of speech data sourced from varied domains like politics,
sports, entertainment etc. We choose a 7-hour subset of this data for the
purpose of domain adaptation and report the character error rate (CER) results
on the evaluation set released as a part of the challenge.

Fig. 2: Comparison of various pruning strategies and frequencies on the
wav2vec-2.0 large LV-60k model.

###  4.2 Fine-tuning configuration

The baseline approach chosen for comparison with PADA is the one where we
directly fine-tune the pre-trained model on the target domain dataset L. This
approach is referred to as Direct Fine-tuning (DFT), the baseline in Table
4.2.

We add a task-specific linear layer on top of the model and jointly fine-tune
with CTC loss. As zeroing out weights is involved in PADA, we fine-tune the
weights during each iteration on both DFT and PADA with no freezing of layers
or updates being involved. The $N$ and $n$ values used in the Alg.1 are
mentioned in the Table 4.2.

Extensive experimentation on the 30h Switchboard data, as depicted in Fig. 2,
helps us conclude with the best settings to go for, with the different pruning
frequencies. The LARGE models use $r_{1}=40$, $r_{1}=r_{2}=r_{3}=30$,
$r_{1}=40,r_{2}=20,r_{3}=10$ whereas the BASE models use $r_{1}=30$,
$r_{1}=r_{2}$

Table 1: Performance of Direct Fine-tuning (DFT) compared to PADA with TAG,
TAW, and CD-TAW.

Pre-trained model |  DFT performance |  Pruning Strategy | Performance at different pruning frequencies
---|---|---|---
|  |  |  Once |  Iterative |  Dynamic Iterative
Finetuning on 30h Switchboard (WER) |  |  |  |  |
wav2vec-2.0 LV-60k $N=27000$ $n=2400$ |  |  TAG |  12.1 |  12.0 |  12.1
|  11.8 |  TAW |  11.8 |  11.7 |  11.8
|  |  CD-TAW(LS-100h) |  11.4 |  11.4 |  11.3
|  |  CD-TAW(LS-960h) |  11.1 |  11.2 |  11.0
Finetuning on 2h Switchboard (WER) |  |  |  |  |
wav2vec-2.0 LV-60k $N=10000$ $n=1000$ |  |  TAG |  22.4 |  22.2 |  22.5
|  22.2 |  TAW |  21.2 |  21.3 |  21.3
|  |  CD-TAW(LS-100h) |  19.0 |  19.1 |  18.9
|  |  CD-TAW(LS-960h) |  17.6 |  18.2 |  17.6
|  |  |  |  |
wav2vec-2.0 LS-960 $N=10000$ $n=1000$ |  |  TAG |  29.8 |  29.9 |  29.8
|  29.2 |  TAW |  29.7 |  29.7 |  29.8
|  |  CD-TAW(LS-100h) |  27.4 |  27.3 |  27.3
Finetuning on 7h Hindi data (CER) |  |  |  |  |
wav2vec-2.0 LV-60k $N=9000$ $n=1000$ |  |  TAG |  18.7 |  14.1 |  13.6
|  16.3 |  TAW |  13.2 |  13.4 |  13.2
|  |  CD-TAW(LS-100h) |  13.2 |  13.4 |  13.0
|  |  |  |  |
XLSR-53 $N=9000$ $n=1000$ |  |  TAG |  12.0 |  11.9 |  12.0
|  11.8 |  TAW |  11.4 |  11.5 |  11.3
|  |  CD-TAW(CV, Babel) |  12.0 |  11.9 |  12.0

$=r_{3}=30$ and $r_{1}=30,r_{2}=25,r_{3}=20,r_{4}=10$ for the _Once_ ,
_Iterative_ and _Dynamic Iterative_ pruning frequencies respectively. The rest
of the parameters follow standard configurations on fairseq [32] from the
works of [6, 33] and are made available on GitHub222https://github.com/Speech-
Lab-IITM/PADA 333Correspondence to Vasista Sai Lodagala:
<EMAIL_ADDRESS>
##  5 Results

As evident from Table 4.2, CD-TAW implemented with wav2vec-2.0 LV-60k and the
mask taken from 100hr LibriSpeech fine-tuning achieves a 20.6% relative WER
improvement on the 2h Switchboard fine-tuning and 19.8% relative CER
improvement on the 7h Hindi data fine-tuning.

##  6 Result Analysis

###  6.1 Key observations

In this section we try to elaborate and discuss on some of our key
observations from our results in Table 4.2. They are as follows:

  * •

CD-TAW almost always outperforms TAW. This corresponds to the fact that being
more “task-aware” is more efficient while making parameters free to adapt to
the new domain downstream than being “domain-aware”.

  * •

Dynamic Iterative Pruning in most cases outperforms other fixed pruning
frequency approaches. This re-confirms our hypothesis that, for domain
adaptation, iteratively decreasing the pruning rate across fine-tuning
iterations works well as the model gets more adapted to the new domain with
time.

  * •

CD-TAW benefits with larger OOD labeled datasets. We base this conclusion from
the Table 4.2 where CD-TAW on 2h Switchboard benefits more when the initial
mask M is taken from the model fine-tuned on 960h of LibriSpeech than 100h. It
may seem unfair to compare masks coming from models fine-tuned on few hours of
data with those fine-tuned on hundreds of hours of data. However, the point we
are trying to emphasize is that, using such readily available models fine-
tuned on large amounts of OOD data, brings better task-awareness and also
avoids the need to fine-tune the models multiple times as in the case of TAW.

  * •

Beyond domain adaptation, PADA also helps in cross-language adaptation. This
is very evident from our experiments on the Hindi 7h labeled data. Using CD-
TAW over TAG for fine-tuning on our low-resource Hindi dataset where the mask
was taken from a LirbiSpeech fine-tuned model, gives us an absolute
improvement of 5.5% CER.

  * •

Language supervision impacts CD-TAW performance. On fine-tuning the XLSR-53
model on the 7h Hindi data we notice that CD-TAW under-performs TAW as the
amount of Hindi data in the OOD dataset (CV and Babel) is minimal. This is
also in-lines with findings from [16].

Fig. 3: Layerwise MMA and IOU for 2h Switchboard fine-tuning.

###  6.2 Layer-wise Pruning Mask Comparison

Next, we try to find the similarity between a mask that provides us with the
best initial sub-network for fine-tuning our network on Switchboard using
PADA, and all other masks from our other experiments. As evident from Table
4.2, our proposed CD-TAW provides the best initial sub-network mask when the
mask is obtained from a converged model fine-tuned on 960hrs of LibriSpeech.
To compare the similarity between masks, we use Intersection Over Union (IOU)
and Mutual Mask Agreement (MMA) as the similarity measures. IOU for
quantifying subnetwork masks, is defined in [14] as follows:

$\operatorname{IOU}\left(m^{a},m^{b}\right)=\frac{\left|\left(m^{a}=1\right)\cap\left(m^{b}=1\right)\right|}{\left|\left(m^{a}=1\right)\cup\left(m^{b}=1\right)\right|}$
(1)

where $m^{a}$ and $m^{b}$ are the two masks we want to compare.

We find that IOU scores are not completely reflective of the similarity
between masks and define MMA as follows:

$\operatorname{Agg_{1}}\left(m^{a},m^{b}\right)=\left|\left(m^{a}=1\right)\cap\left(m^{b}=1\right)\right|$
$\operatorname{Agg_{0}}\left(m^{a},m^{b}\right)=\left|\left(m^{a}=0\right)\cap\left(m^{b}=0\right)\right|$

$\operatorname{MMA}\left(m^{a},m^{b}\right)=\frac{\operatorname{Agg_{1}}+\operatorname{Agg_{0}}}{N}$
(2)

where $N$ is the total number of weights in $m^{a}$ or $m^{b}$.

MMA provides a better comparison between masks compared to IOU because, while
IOU focuses only in the unmasked regions, MMA measures the similarity between
masks in both the masked and unmasked regions. For instance, let
$m^{a}=[1,0,1,0]$ and let $m^{b}=[1,1,0,0]$. In this case though the two masks
are similar in the first and last positions, the IOU score turns out to be
$0.33$, while MMA gives the desired score of $0.5$.

Fig.3 depicts the Layer-wise MMA and IOU scores between our best sub-network
mask and experiments from other masks. As clearly evident, both IOU and MMA
fall sharply towards the last layers, which also learn the highest task-
specific information [12] (in our case, ASR task through CTC fine-tuning).
Alternatively, our best-performing model, CD-TAW, re-confirms the hypothesis
that irrespective of the domain, the more “task-aware” the model is, the
better the initial sub-network mask. Finally, though CD-TAW from a model fine-
tuned on 100 hours performs worse than its 960hrs counterpart, it shows the
highest IOU and MMA values out of all the masks, which again asserts that
better masks can be obtained from more “task-awareness”.

##  7 Conclusion and Future Work

In this paper, we propose PADA, a novel paradigm for low-resource ASR domain
adaptation. Models pre-trained with SSL on high amounts of unlabeled OOD data
show significant improvements when fine-tuned on ASR using the PADA framework.
As part of the future work, we would like to explore if structured pruning and
unsupervised SSL domain adaptation can help boost the performance in this
learning paradigm.

##  8 Acknowledgement

A part of this work was funded by “Bhashini: National Language translation
Mission” project of the Ministry of Electronics and Information Technology
(MeitY), Government of India.

## References

  * [1] Santosh K Gaikwad, Bharti W Gawali, and Pravin Yannawar,  “A review on speech recognition technique,”  International Journal of Computer Applications, vol. 10, no. 3, pp. 16–24, 2010.
  * [2] Hermann Ney,  “Speech translation: Coupling of recognition and translation,”  in IEEE ICASSP 1999. IEEE, 1999, vol. 1, pp. 517–520.
  * [3] George Saon, Gakuto Kurata, Tom Sercu, Kartik Audhkhasi, Samuel Thomas, Dimitrios Dimitriadis, Xiaodong Cui, Bhuvana Ramabhadran, Michael Picheny, Lynn-Li Lim, et al.,  “English conversational telephone speech recognition by humans and machines,”  arXiv preprint arXiv:1703.02136, 2017.
  * [4] Hemant Yadav, Sreyan Ghosh, Yi Yu, and Rajiv Ratn Shah,  “End-to-end named entity recognition from english speech,”  arXiv preprint arXiv:2005.11184, 2020.
  * [5] Amodei et al.,  “Deep speech 2 : End-to-end speech recognition in english and mandarin,”  in ICML 2016, pp. 173–182.
  * [6] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  NeurIPS 2020, vol. 33, pp. 12449–12460, 2020.
  * [7] Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung yi Lee,  “SUPERB: Speech Processing Universal PERformance Benchmark,”  in Interspeech 2021, 2021, pp. 1194–1198.
  * [8] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [9] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al.,  “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”  arXiv preprint arXiv:2110.13900, 2021.
  * [10] Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee,  “Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,”  in IEEE ICASSP 2020, pp. 6419–6423.
  * [11] Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, and Michael Auli,  “Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training,”  in Interspeech 2021, 2021, pp. 721–725.
  * [12] Ankita Pasad, Ju-Chieh Chou, and Karen Livescu,  “Layer-wise analysis of a self-supervised speech representation model,”  in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 914–921.
  * [13] Awni Hannun,  “The history of speech recognition to the year 2030,”  arXiv preprint arXiv:2108.00084, 2021.
  * [14] Cheng-I Jeff Lai, Yang Zhang, Alexander H. Liu, Shiyu Chang, Yi-Lun Liao, Yung-Sung Chuang, Kaizhi Qian, Sameer Khurana, David Cox, and James Glass,  “Parp: Prune, adjust and re-prune for self-supervised speech recognition,”  in NeurIPS 2021.
  * [15] Jonathan Frankle and Michael Carbin,  “The lottery ticket hypothesis: Finding sparse, trainable neural networks,”  in ICLR 2019.
  * [16] Yizhou Lu, Mingkun Huang, Xinghua Qu, Pengfei Wei, and Zejun Ma,  “Language adaptive cross-lingual speech representation learning with sparse sharing sub-networks,”  arXiv preprint arXiv:2203.04583, 2022.
  * [17] Aaron van den Oord, Yazhe Li, and Oriol Vinyals,  “Representation learning with contrastive predictive coding,”  arXiv preprint arXiv:1807.03748, 2018.
  * [18] Andy T Liu, Shang-Wen Li, and Hung-yi Lee,  “Tera: Self-supervised learning of transformer encoder representation for speech,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 2351–2366, 2021.
  * [19] Trevor Gale, Erich Elsen, and Sara Hooker,  “The state of sparsity in deep neural networks,”  arXiv preprint arXiv:1902.09574, 2019.
  * [20] Song Han, Jeff Pool, John Tran, and William Dally,  “Learning both weights and connections for efficient neural network,”  NeurIPS 2015, vol. 28, 2015.
  * [21] Suyog Gupta Michael H. Zhu,  “To prune, or not to prune: Exploring the efficacy of pruning for model compression,”  in ICLR 2018.
  * [22] Shuhao Gu, Yang Feng, and Wanying Xie,  “Pruning-then-expanding model for domain adaptation of neural machine translation,”  in NAACL 2021.
  * [23] Dong Yu, Kaisheng Yao, Hang Su, Gang Li, and Frank Seide,  “Kl-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition,”  in IEEE ICASSP 2013, pp. 7893–7897.
  * [24] Hank Liao,  “Speaker adaptation of context dependent deep neural networks,”  in IEEE ICASSP 2013, pp. 7947–7951.
  * [25] Zhong Meng, Jinyu Li, Yifan Gong, and Biing-Hwang Juang,  “Adversarial teacher-student learning for unsupervised domain adaptation,”  in IEEE ICASSP 2018, pp. 5949–5953.
  * [26] Vimal Manohar, Pegah Ghahremani, Daniel Povey, and Sanjeev Khudanpur,  “A teacher-student learning approach for unsupervised domain adaptation of sequence-trained asr models,”  in IEEE SLT Workshop 2018, pp. 250–257.
  * [27] Yusuke Shinohara,  “Adversarial multi-task learning of deep neural networks for robust speech recognition.,”  in Interspeech 2016, pp. 2369–2372.
  * [28] Zhong Meng, Jinyu Li, Zhuo Chen, Yang Zhao, Vadim Mazalov, Yifan Gong, and Biing-Hwang Juang,  “Speaker-invariant training via adversarial learning,”  in IEEE ICASSP 2018, pp. 5969–5973.
  * [29] CS Anoop, AP Prathosh, and AG Ramakrishnan,  “Unsupervised domain adaptation schemes for building asr in low-resource languages,”  in IEEE ASRU Workshop 2021.
  * [30] Dongseong Hwang, Ananya Misra, Zhouyuan Huo, Nikhil Siddhartha, Shefali Garg, David Qiu, Khe Chai Sim, Trevor Strohman, Françoise Beaufays, and Yanzhang He,  “Large-scale asr domain adaptation using self-and semi-supervised learning,”  arXiv preprint arXiv:2110.00165, 2021.
  * [31] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith,  “Don’t stop pretraining: Adapt language models to domains and tasks,”  in ACL 2020, pp. 8342–8360.
  * [32] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli,  “fairseq: A fast, extensible toolkit for sequence modeling,”  in NAACL-HLT 2019: Demonstrations.
  * [33] Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli,  “Xls-r: Self-supervised cross-lingual speech representation learning at scale,”  arXiv, vol. abs/2111.09296, 2021.
  * [34] John J Godfrey, Edward C Holliman, and Jane McDaniel,  “Switchboard: Telephone speech corpus for research and development,”  in IEEE ICASSP 1992, vol. 1, pp. 517–520.
  * [35] Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al.,  “Libri-light: A benchmark for asr with limited or no supervision,”  in IEEE ICASSP 2020, pp. 7669–7673.