# Improving Article Classification with Edge-Heterogeneous Graph Neural
Networks

Khang Ly
Faculty of Science
University of Amsterdam
Amsterdam, Netherlands
<EMAIL_ADDRESS>
& Yury Kashnitsky
Research Content Operations
Elsevier B.V.
Amsterdam, Netherlands
<EMAIL_ADDRESS>
& Savvas Chamezopoulos
Research Content Operations
Elsevier B.V.
Amsterdam, Netherlands
<EMAIL_ADDRESS>
& Valeria Krzhizhanovskaya
Faculty of Science
University of Amsterdam
Amsterdam, Netherlands
<EMAIL_ADDRESS>

###### Abstract

Classifying research output into context-specific label taxonomies is a
challenging and relevant downstream task, given the volume of existing and
newly published articles. We propose a method to enhance the performance of
article classification by enriching simple Graph Neural Networks (GNN)
pipelines with edge-heterogeneous graph representations. SciBERT is used for
node feature generation to capture higher-order semantics within the articles’
textual metadata. Fully supervised transductive node classification
experiments are conducted on the Open Graph Benchmark (OGB) ogbn-arxiv dataset
and the PubMed diabetes dataset, augmented with additional metadata from
Microsoft Academic Graph (MAG) and PubMed Central, respectively. The results
demonstrate that edge-heterogeneous graphs consistently improve the
performance of all GNN models compared to the edge-homogeneous graphs. The
transformed data enable simple and shallow GNN pipelines to achieve results on
par with more complex architectures. On ogbn-arxiv, we achieve a top-15 result
in the OGB competition with a 2-layer GCN (accuracy 74.61%), being the
highest-scoring solution with sub-1 million parameters. On PubMed, we closely
trail SOTA GNN architectures using a 2-layer GraphSAGE by including additional
co-authorship edges in the graph (accuracy 89.88%). The implementation is
available at: https://github.com/lyvykhang/edgehetero-nodeproppred.

_Keywords_ Heterogeneous Graph Learning $\cdot$ Graph Neural Networks $\cdot$
Article Classification $\cdot$ Document Relatedness

##  1 Introduction

Article classification is a challenging downstream task within natural
language processing (NLP) (Mirończuk and Protasiewicz, 2018). An important
practical application is classifying existing or newly-published articles
according to specific research taxonomies. The task can be approached as a
graph node classification problem, where nodes represent articles with
corresponding feature mappings, and edges are defined by a strong signal of
article relatedness, e.g. citations/references. Conventionally, graph
representation learning is handled in two phases: unsupervised node feature
generation, followed by supervised learning on said features using the graph
structure. Graph neural networks (GNNs) can be successfully employed for the
second phase of such problems, being capable of preserving the rich structural
information encoded by graphs. In recent years, prolific GNN architectures
have achieved strong performance on citation network benchmarks (Kipf and
Welling, 2017; Hamilton et al., 2017; Veličković et al., 2018; Frasca et al.,
2020; Li et al., 2021).

We focus on combining textual information from articles with various
indicators of article relatedness (citation data, co-authorship, subject
fields, and publication sources) to create a graph with multiple edge types;
further referred to as an edge-heterogeneous graph, also known as multi-graphs
or multipartite networks (Barabási and Pósfai, 2017). We use two established
node classification benchmarks - the citation graphs ogbn-arxiv and PubMed -
and leverage their connection to large citation databases - Microsoft Academic
Graph (MAG) and PubMed Central - to retrieve the metadata fields and enrich
the graph structure with additional edge types (Hu et al., 2020; Sen et al.,
2008). For feature generation, SciBERT is used to infer embeddings based on
articles’ titles and abstracts (Beltagy et al., 2019), with the intention of
capturing higher-order semantics compared to the defaults, i.e. Skip-Gram or
bag-of-words. We test our transformed graphs with a variety of GNN backbone
models, converted to support heterogeneous input using the relational graph
convolutional network (R-GCN) framework (Schlichtkrull et al., 2018). In
essence, we approach a typically homogeneous task using heterogeneous
techniques. The method is intuitively simple and interpretable; we do not
utilize complex model architectures and training frameworks, focusing
primarily on data retrieval and preprocessing to boost the performance of
simpler models, thus maintaining a reasonably low computational cost and small
number of fitted parameters.

Figure 1: Illustration of the proposed edge heterogeneity, which enables the
neighboring feature aggregation for a node $X_{1}$ to be performed across a
variety of subgraphs, leveraging multiple signals of article relatedness
(References, Authorship, and Journal depicted here).

A considerable volume of research is devoted to article classification, graph
representation learning with respect to citation networks, and the adaptation
of these practices to heterogeneous graphs (Wu et al., 2019a; Bing et al.,
2022). However, the application of heterogeneous graph enrichment techniques
to article classification is not well-studied and presents a research
opportunity. Existing works on heterogeneous graphs often consider multiple
node types, expanding from article to entity classification; we exclusively
investigate the heterogeneity of paper-to-paper relationships to remain
consistent with the single-node type problem setting. The emergence of rich
metadata repositories for papers, e.g. OpenAlex, illustrates the relevance of
our research (Priem et al., 2022).

Scalability is often a concern with GNN architectures. For this reason,
numerous approaches simplify typical GNN architectures with varying
strategies, e.g. pre-computation or linearization, without sacrificing
significant performance in most downstream tasks (Frasca et al., 2020; Wu et
al., 2019b; Prieto et al., 2023). Other solutions avoid GNNs altogether,
opting for simpler approaches based on early graph-based techniques like label
propagation, which outperform GNNs in several node classification benchmarks
(Huang et al., 2021). The success of these simple approaches raises questions
about the potential impracticality of deep GNN architectures on large real-
world networks with a strong notion of locality, and whether or not such
architectures are actually necessary to achieve satisfactory performance.

Compared to simple homogeneous graphs, heterogeneous graphs encode rich
structural and semantic information, and are more representative of real-world
information networks and entity relationships (Bing et al., 2022). For
example, networks constructed from citation databases can feature relations
between papers, their authors, and shared keywords, often expressed in an RDF
triple, e.g. “paper $\xrightarrow{\text{(co-)authored by}}$ author,” “paper
$\xrightarrow{\text{includes}}$ keyword,” “paper $\xrightarrow{\text{cites}}$
paper.” Heterogeneous GNN architectures share many similarities with their
homogeneous counterparts; a common approach is to aggregate feature
information from local neighborhoods, while using additional modules to
account for varying node and/or edge types (Yang et al., 2022). Notably, the
relational graph convolutional network approach (R-GCN) by Schlichtkrull et
al. (2018) shows that GCN-based frameworks can be effectively applied to
modeling relational data, specifically for the task of node classification.
The authors propose a modeling technique where the message passing functions
are duplicated and applied individually to each relationship type. This
transformation can be generalized to a variety of GNN convolutional operators
in order to convert them into their relational (heterogeneous) counterparts.

##  2 Methodology

We propose an approach focusing on dataset provenance, leveraging their
linkage to large citation and metadata repositories, e.g. MAG and PubMed
Central, to retrieve additional features and enrich their graph
representations. The proposed method is GNN-agnostic, compatible with a
variety of model pipelines (provided they can function with heterogeneous
input graphs) and node embedding techniques (results are presented with the
provided features, plus the SciBERT embeddings). Figure 1 provides a high-
level overview of the method.

The tested GNN backbones (see section 3) are converted to support
heterogeneous input using the aforementioned R-GCN transformation defined by
Schlichtkrull et al. (2018), involving the duplication of the message passing
functions at each convolutional layer per relationship type; we employ the
PyTorch Geometric (PyG) implementation of this technique, using the mean as
the aggregation operator (Fey and Lenssen, 2019).

###  2.1 Datasets

Our experiments are conducted on two datasets: the Open Graph Benchmark (OGB)
ogbn-arxiv dataset and the PubMed diabetes dataset.

The OGB ogbn-arxiv dataset consists of 169,343 Computer Science papers from
arXiv, hand-labeled into 40 subject areas by paper authors and arXiv
moderators, with 1,166,243 reference links (Hu et al., 2020). Node features
are constructed from textual information by averaging the embeddings of words
(which are generated with the Skip-Gram model) in the articles’ titles and
abstracts. The dataset provides the mapping used between papers’ node IDs and
their original MAG IDs, which can be used to retrieve additional metadata.

The PubMed diabetes dataset consists of 19,717 papers from the National
Library of Medicine’s (NLM) PubMed database labeled into one of three
categories: “Diabetes Mellitus, Experimental,” “Diabetes Mellitus Type 1,” and
“Diabetes Mellitus Type 2,” with 44,338 references links (Sen et al., 2008).
Bag-of-words vectors from a dictionary of 500 unique words are provided as
node features. Similarly, the papers’ original PubMed IDs can be used to fetch
relevant metadata.

###  2.2 Data Augmentation

We used a July-2020 snapshot of the complete Microsoft Academic Graph (MAG)
index (240M papers) - since MAG (and the associated API) was discontinued
later - to obtain additional metadata (Zhang et al., 2022).111The data is
hosted by AMiner’s Open Academic Graph project. All chunks were downloaded
locally and metadata of IDs corresponding to papers in ogbn-arxiv were saved.
Potential indicators of paper relatedness include: authors, venue, and fields
of study. Other metadata (DOI, volume, page numbers, etc.) are not useful for
our purposes. Hence, we “heterogenize” the dataset with the following
additional edge types (in addition to the References edges):

  * •

(Co)-Authorship: Two papers are connected if they share an author, with a
corresponding edge weight indicating the number of shared authors. This is
based on the assumption that a given author tends to perform research on
similar disciplines.

  * •

Venue: Two papers are connected if they were published at the same venue
(conference or journal), assuming that specific conferences contribute to
relevant research areas.

  * •

Fields of Study (FoS): Two papers are connected if they share at least one
field, with an edge weight based on the number of shared fields. Fields of
study, e.g. “computer science,” “neural networks,” etc. are automatically
assigned with an associated confidence score (which we do not use), and each
paper can have multiple fields of study, making them functionally similar to
keywords.

An unprocessed version of the PubMed dataset preserving the original paper IDs
was used (Namata et al., 2012).222This version of the dataset is hosted by the
LINQS Statistical Relational Learning Group. The 2023 annual baseline on the
NLM FTP server is accessed to retrieve metadata. All files were downloaded
locally and metadata of matching IDs were extracted (19,716 records matched, 1
missing). A January-2023 snapshot of the complete PubMed citation database
(35M papers) was accessed for additional metadata. Relevant features include:
title, abstract, authors, published journal (indicated by unique NLM journal
IDs), and Medical Subject Headings (MeSH®). The latter is an NLM-controlled
hierarchical vocabulary used to characterize biomedical article content. As
with the Fields of Study, they are functionally similar to curated keywords.
As before, we use three additional edge types:

  * •

(Co)-Authorship: Two papers are connected if they share an author, with a
corresponding edge weight indicating the number of shared authors. Unlike MAG,
PubMed Central does not provide unique identifiers for authors, so exact
author names are used, which can lead to some ambiguity in a minority of
cases, e.g. two distinct authors with the same name.

  * •

Journal: Two papers are connected if they were published in the same journal.
The intuition is that journals publish papers on similar topics.

  * •

MeSH: Two papers are connected if they share at least one MeSH subject
heading, with an edge weight based on the number of shared subjects.

Since the ogbn-arxiv venue and fields and PubMed MeSH relationships result in
massive edge lists, posing out-of-memory issues on the utilized hardware, we
only create edges between up to $k$ nodes per unique venue/field/heading,
where $k$ is the mean number of papers per venue/field/heading, in order to
reduce the subgraphs’ sizes. All edge weights are normalized using logistic
sigmoid.

In a traditional citation network, the links are typically directed, but in
our experiments, they are undirected to strengthen the connections of
communities in the graph. The graph includes only one node type, “paper.”
Other approaches, notably in the citation recommendation domain, leverage node
types to represent authors and journals (Guo et al., 2017). However, this work
strictly concerns relationships between papers and not between papers and
other entities, in order to apply the homogeneous problem settings.
Practically, the resultant graph would contain too many nodes, while there the
number of features and metadata is insufficient to generate informative
representations of other node types, limiting their usefulness in the feature
aggregation step.

For textual node feature representation, embeddings of the concatenated paper
titles and abstracts are inferred using SciBERT (Beltagy et al., 2019); this
is inspired by the work of Cohan et al. (2020), in which SciBERT was
pretrained on a citation graph and used to generate high-quality document-
level embeddings for node classification purposes. Here, we utilize the base
model (scibert-scivocab-uncased) without additional fine-tuning.

###  2.3 Subgraph Properties

Some insights on the characteristics of the defined subgraphs can be derived
from Table 1, which lists the following: number of nodes and edges (post-
conversion to undirected) in the largest connected component (LCC), total
number of edges, average degree, network average clustering coefficient
(Schank and Wagner, 2005), and edge homophily ratio (fraction of edges
connecting nodes with the same label) (Ma et al., 2022). While the References
graphs do not exhibit the tight clustering typical of real-world information
networks, the strong signal of relatedness in the edges has nonetheless
ensured their compatibility with message passing GNN paradigms (Wu et al.,
2019a). This relatedness is also evident in the Authorship graphs, and the
high level of clustering confirms the initial hypothesis that researchers co-
author papers within similar topics. The topic-based relationships (FoS and
MeSH), include many edges formed between shared generic keywords, e.g.
“computer science,” leading to rather average homophily. The publication
source subgraphs (Venue and Journal) consist of isolated fully-connected
clusters per unique source, with no inter-cluster connections, as each paper
belongs to only one journal or venue. As with the topic-based relationships,
the research area covered by a given publication conference or journal can be
quite broad with respect to the paper labels.

Table 1: Properties of constructed subgraphs. Note that the References
subgraphs are the only ones without isolated nodes. Note that the network
average clustering coefficient computation accounts for isolated nodes (which
are treated as having “zero” local clustering), hence the value for sparser
subgraphs, e.g. ogbn-arxiv Venue, is notably lower than intuitively expected
(1, in the absence of isolated nodes).
|  | Nodes in LCC | Edges in LCC | Edges total | Avg. degree |  Avg. clustering coeff. | Homophily
---|---|---|---|---|---|---|---
ogbn-arxiv | References | 169,343 | 2,315,598 | 2,315,598 | 13.7 |  0.247 | 0.654
Authorship | 145,973 | 6,697,998 | 6,749,335 | 39.9 |  0.702 | 0.580
Venue | 63 | 3,906 | 600,930 | 3.5 |  0.112 | 0.077
FoS | 144,529 | 8,279,492 | 8,279,687 | 48.9 |  0.505 | 0.319
PubMed | References | 19,716 | 88,649 | 88,649 | 4.5 |  0.057 | 0.802
Authorship | 17,683 | 729,468 | 731,376 | 37.1 |  0.623 | 0.721
Journal | 2,213 | 4,895,156 | 11,426,930 | 579.6 |  0.940 | 0.414
MeSH | 18,345 | 1,578,526 | 1,578,530 | 80.1 |  0.456 | 0.550

(a)

(b)

Figure 2: Degree distribution, i.e. frequency of each degree value, of all
subgraphs for ogbn-arxiv (left) and PubMed (right), plotted on a log-log
scale. Points indicate the unique degree values.

Figure 2 shows the degree distribution of all edge type subgraphs in both
datasets, which gives a clear view of the subgraphs’ structures when
interpreted with the above metrics. The high frequency of large node degrees
in the PubMed Journal subgraph corresponds to large journals; the size of the
LCC (2,213) is the number of papers in the largest journal. While not visible
for the ogbn-arxiv Venue subgraph due to the aforementioned sampling, a
similar distribution would occur for large venues if all possible edges had
been included. In contrast, the lower occurrence of high degree nodes and low
clustering in the References subgraphs of both datasets indicates greater
average distance across the LCC compared to the other subgraphs; such a
structure stands to benefit the most from the multi-hop neighborhood feature
aggregation performed by GNNs. Relative to the References, the Authorship and
topic-based subgraphs (FoS and MeSH) exhibit increased skewness in the
distribution and higher average clustering, which indicates the presence of
more (near-)cliques, i.e. subsections of the graph wherein (almost) any two
papers share an author or topic. Hence, these subgraphs bear the closest
structural resemblance to small-world networks (Watts and Strogatz, 1998). The
impact of these degree distributions on classification performance is further
investigated in section 3.1.

##  3 Experiments and Results

We evaluate model performance on the task of fully supervised transductive
node classification. The metric is multi-class accuracy on the test set. The
proposed data preparation scheme is tested with several GNN architectures
commonly deployed in benchmarks. We consider two GCN setups (base one and with
a jumping knowledge module using concatenation as the aggregation scheme), as
well as GraphSAGE (Kipf and Welling, 2017; Xu et al., 2018; Hamilton et al.,
2017). We also run experiments with the simplified graph convolutional
operator (SGC) (Wu et al., 2019b). The increased graph footprint can lead to
scalability concerns, hence the performance of such lightweight and parameter-
efficient methods is of interest.

For ogbn-arxiv, the provided time-based split is used: train on papers
published until 2017, validate on those published in 2018, test on those
published since 2019. For PubMed diabetes, nodes of each class are randomly
split into 60% - 20% - 20% for training - validation - and testing for each
run (as performed by Pei et al. (2020) and Chen et al. (2020)). Ablation
experiments are also performed to examine the impact of the different edge
types (averaged across 3 runs) and to identify the optimal configuration for
both datasets, on which we then report final results (averaged across 10
runs). Experiments were conducted on a g4dn.2xlarge EC2 instance (32 GB RAM, 1
NVIDIA Tesla T4 16 GB VRAM). Models are trained with negative log-likelihood
loss and early stopping based on validation accuracy (patience of 50 epochs,
with an upper limit of 500 epochs). We also scale down the learning rate as
the validation loss plateaus. The SciBERT embeddings are pre-computed using
multi-GPU distributed inference.

###  3.1 Ablation Study

Ablation results for both datasets are presented in Table 2, separated by node
embedding type. First, all possible homogeneous subgraphs are inspected, as
this is the conventional input data for this task (see the first 4 rows). The
best performance is achieved on the References graphs, followed by the
Authorship graphs. Then we build upon the References graph by adding different
combinations of other subgraphs. The results demonstrate that transitioning to
edge-heterogeneous graphs can yield up to 2.87% performance improvement on
ogbn-arxiv and 2.05% on PubMed (see the difference between the yellow- and
green-highlighted cells). These results were obtained with a 2-layer GCN base,
using the following fixed hyperparameter values: optimizer weight decay of
0.001 and initial learning rate of 0.01, hidden layer dropout probability of
0.5, and hidden feature dimensionality of 128 (ogbn-arxiv) or 64 (PubMed).

Table 2:  Ablation study for both datasets, 3-run average test accuracy with a
2-layer GCN. The best results are highlighted in bold. Note the table feature
differences (Venue, FoS, and Skip-Gram for ogbn-arxiv, Journal, MeSH, and Bag-
of-Words for PubMed).

References | Authorship |  Venue / Journal |  FoS / MeSH | ogbn-arxiv | PubMed
---|---|---|---|---|---
SkipGram | SciBERT | Bag-of-Words | SciBERT
✓ |  |  |  | 0.6858 $\pm$ 0.0008 | 0.7177 $\pm$ 0.0024 | 0.8646 $\pm$ 0.0094 | 0.8635 $\pm$ 0.0043
| ✓ |  |  | 0.6099 $\pm$ 0.0008 | 0.6456 $\pm$ 0.0009 | 0.8026 $\pm$ 0.0054 | 0.7921 $\pm$ 0.0121
|  | ✓ |  | 0.4665 $\pm$ 0.0003 | 0.5998 $\pm$ 0.0043 | 0.5440 $\pm$ 0.0357* | 0.6137 $\pm$ 0.0090
|  |  | ✓ | 0.4828 $\pm$ 0.0088 | 0.5270 $\pm$ 0.0043 | 0.7499 $\pm$ 0.0047 | 0.7274 $\pm$ 0.0158
✓ | ✓ |  |  | 0.7079 $\pm$ 0.0023 | 0.7330 $\pm$ 0.0010 | 0.8851 $\pm$ 0.0050 | 0.8747 $\pm$ 0.0055
✓ |  | ✓ |  | 0.6795 $\pm$ 0.0035 | 0.7136 $\pm$ 0.0036 | 0.8745 $\pm$ 0.0103 | 0.8661 $\pm$ 0.0076
✓ |  |  | ✓ | 0.6914 $\pm$ 0.0007 | 0.7221 $\pm$ 0.0002 | 0.8737 $\pm$ 0.0106 | 0.8643 $\pm$ 0.0091
✓ | ✓ | ✓ |  | 0.6995 $\pm$ 0.0009 | 0.7334 $\pm$ 0.0008 | 0.8430 $\pm$ 0.0389* | 0.8696 $\pm$ 0.0046
✓ | ✓ |  | ✓ | 0.7145 $\pm$ 0.0024 | 0.7383 $\pm$ 0.0003 | 0.8301 $\pm$ 0.1035* | 0.8724 $\pm$ 0.0050
✓ |  | ✓ | ✓ | 0.6828 $\pm$ 0.0015 | 0.7189 $\pm$ 0.0001 | 0.8306 $\pm$ 0.0563* | 0.8656 $\pm$ 0.0104
✓ | ✓ | ✓ | ✓ | 0.7049 $\pm$ 0.0011 | 0.7327 $\pm$ 0.0004 | 0.8563 $\pm$ 0.0120* | 0.8726 $\pm$ 0.0038

*Indicates (significant) oversmoothing.

Cross-checking with the metrics in Table 1 implies improvements from edge-
heterogeneity roughly correspond to the edge homophily ratio of the utilized
subgraphs, as strong homophily is implicitly assumed by the neighborhood
aggregation mechanism of GCN-based models. Subsequently, their performance can
be erratic and unpredictable in graphs with comparatively low homophily (Kipf
and Welling, 2017; Ma et al., 2022). Since the R-GCN transformation collects
neighborhoods from input subgraphs with equal weighting, including a
comparatively noisy subgraphs (like ogbn-arxiv Venue) can worsen predictive
performance. Changing the R-GCN aggregation operator, e.g. from mean to
concatenation, does not alleviate this. This study suggests publication
sources do not yield beneficial subgraphs with their current definition; while
the aforementioned tight clustering could provide a strong signal for
classification with sufficient homophily, any potential benefit is absent
because these clusters are noisy. The topic-based subgraphs are more
structurally preferable, but noisy edges (from keywords tied to concepts that
are higher-level than the paper labels) reduce their usefulness in
classification. In all experimental settings, the largest gain from
heterogeneity comes from the co-authorship graph, often outperforming
configurations that use more subgraphs. These trends are expected, given the
characteristics discussed in section 2.3.

Replacing the PubMed bag-of-words vectors with SciBERT embeddings does not
improve raw accuracy in most cases. Though, the increased feature
dimensionality of SciBERT embeddings do stabilize convergence behavior in
configurations using multiple subgraphs. A paper might possess only a few non-
zero feature dimensions when using bag-of-words; combined with the additional
feature averaging step from the R-GCN transformation, the risk of
oversmoothing is increased even on shallow 2-layer networks in edge-
heterogeneous configurations (however, note that tested single-layer models
underfit and thus do not improve performance). In accordance with the findings
of Chen et al. (2020), these hypothesized oversmoothing effects are more
pronounced when using graphs with high average degree, i.e. the publication
source and topic-based subgraphs; nodes with high degree aggregate more
information from their neighbors, increasing the likelihood of homogenization
as network depth increases.

###  3.2 Optimal Configuration

Results with the optimal configuration identified from the ablation study on
ogbn-arxiv and PubMed are listed in Tables 3 and 4, respectively. Note that
the actual performance gains may vary slightly since the third-party
benchmarks were used with _directed_ References graphs and differently-tuned
hyperparameters.

Table 3:  Results on the best ogbn-arxiv configuration (References,
Authorship, Fields of Study subgraphs, and SciBERT embeddings) with
hyperparameters described in Appendix Appendix A. Hyperparameters. The
baseline results on the unmodified dataset and the improvement over the
baseline (test acc. minus baseline acc.) are also displayed. For GCN, GCN+JK,
and SAGE, these are taken from the official leaderboard.* The SGC baseline was
obtained by using the (undirected) References subgraph and provided SkipGram
features.

Model | Validation Accuracy | Test Accuracy | # Params | Baseline Accuracy | Improvement
---|---|---|---|---|---
GCN | 0.7586 $\pm$ 0.0012 | 0.7461 $\pm$ 0.0006 | 621,944 | 0.7174 $\pm$ 0.0029 | +2.87%
GCN+JK | 0.7629 $\pm$ 0.0007 | 0.7472 $\pm$ 0.0024 | 809,512 | 0.7219 $\pm$ 0.0021 | +2.53%
SAGE | 0.7605 $\pm$ 0.0007 | 0.7461 $\pm$ 0.0013 | 1,242,488 | 0.7149 $\pm$ 0.0027 | +3.12%
SGC | 0.7515 $\pm$ 0.0005 | 0.7419 $\pm$ 0.0004 | 92,280 | 0.6855 $\pm$ 0.0002 | +5.64%

* https://ogb.stanford.edu/docs/leader_nodeprop/#ogbn-arxiv.

Table 4:  Results on the best PubMed configuration (References and Authorship
subgraphs, and default Bag-of-Words vectors) with hyperparameters given in
Appendix Appendix A. Hyperparameters. The GCN and GCN+JK baselines are from
Chen et al. (2020) (no standard deviations reported), and the others were
obtained by using only the (undirected) References subgraph and provided Bag-
of-Words features.
Model | Validation Accuracy | Test Accuracy | # Params | Baseline Accuracy | Improvement
---|---|---|---|---|---
GCN | 0.8954 $\pm$ 0.0025 | 0.8938 $\pm$ 0.0040 | 129,286 | 0.8813 | +1.25%
GCN+JK | 0.8953 $\pm$ 0.0046 | 0.8939 $\pm$ 0.0064 | 34,499 | 0.8894 | +0.45%
SAGE | 0.9017 $\pm$ 0.0038 | 0.8988 $\pm$ 0.0047 | 129,155 | 0.8921 $\pm$ 0.0044 | +0.67%
SGC | 0.8715 $\pm$ 0.0038 | 0.8698 $\pm$ 0.0049 | 3,006 | 0.8607 $\pm$ 0.0042 | +0.91%

The results demonstrate that the additional structural information provided by
edge heterogeneity consistently improves final performance of a variety of
hetero-transformed GNN frameworks compared to their homogeneous counterparts
on both datasets, when making optimal subgraph choices (though, suboptimal
choices can still situationally improve performance). These improvements are
independent of the tested textual embedding methods, and can occur even when
the added subgraphs possess suboptimal graph properties, e.g. lower edge
homophily ratio and presence of isolated nodes, compared to the starting
References graph. SGC with ogbn-arxiv shows the strongest improvement over the
baseline. Most likely, the linear classifier relies less on graph structures,
and hence benefits more from the deeper textual semantics captured by SciBERT.
Notably, the best results are competitive with the SOTA, while operating on a
limited compute budget and low level of complexity (simple 2-layer GNN model
pipelines with comparatively few tunable parameters). On ogbn-arxiv, we
achieve a top-15 result with the GCN backbone, being the highest-scoring
solution with sub-1 million parameters. On PubMed with the aforementioned
splitting strategy, we closely trail the best performance reported by GCNII
(90.30%) and Geom-GCN (90.05%) (Chen et al., 2020; Pei et al., 2020), by using
a GraphSAGE backbone with just the added co-authorship subgraph input
(89.88%).

##  4 Conclusions and Future Work

In this paper, we propose a data transformation methodology leveraging
metadata retrieved from citation databases to create enriched edge-
heterogeneous graph representations based on various additional signals of
document relatedness: co-authorship, publication source, fields of study, and
subject headings. We also test the substitution of default node features with
SciBERT embeddings to capture higher-dimensionality textual semantics. By
nature, the methodology is GNN- and embedding-agnostic. Deploying optimal
configurations of the transformed graph with a variety of simple GNN pipelines
leads to consistent improvements over the starting data, and enables results
on par with the SOTA in full-supervised node classification. Overall, results
show that our methodology can be an effective strategy to achieve respectable
performance on datasets with readily-available article metadata, without
necessitating complex GNN architectures and lengthy (pre-)training procedures.

As the methodology is compatible with any hetero-transformable GNN backbone
and node embedding technique, we expect that deploying the transformed data
with SOTA GNN frameworks, e.g. RevGAT by Li et al. (2021) on ogbn-arxiv, will
lead to a greater raw performance. Though, the larger memory footprint of the
graph may complicate the application of such frameworks.

Refining the edge type definitions, e.g. connect papers that share at least
two fields of study and/or remove “generic” fields applicable to a majority of
papers in the set, can help de-noising and improving the properties of the
respective subgraphs. A custom aggregation scheme could be implemented for the
heterogeneous transformation dependent on individual subgraph properties, such
as a weighted average based on some metric of subgraph “quality,” e.g.
homophily. To mitigate the increased risk of heterogeneity-induced
oversmoothing, additional regularization techniques, e.g. DropEdge by Rong et
al. (2020), could be considered. Finally, applying parameter-efficient fine-
tuning (PEFT) techniques to the SciBERT model can improve feature separability
and thus classification performance (Duan et al., 2023); the effectiveness of
different transformer-based language models can also be investigated.

## Acknowledgement

The authors would like to thank  prof. Paul Groth for his supervision and
consultation throughout the project.

## References

  * Mirończuk and Protasiewicz [2018] Marcin Michał Mirończuk and Jarosław Protasiewicz.  A recent overview of the state-of-the-art elements of text classification.  _Expert Systems with Applications_ , 106:36–54, 2018.  ISSN 0957-4174.  doi:10.1016/j.eswa.2018.03.058.  URL https://www.sciencedirect.com/science/article/pii/S095741741830215X.
  * Kipf and Welling [2017] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In _International Conference on Learning Representations_ , 2017.  URL https://openreview.net/forum?id=SJU4ayYgl.
  * Hamilton et al. [2017] William L. Hamilton, Rex Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 1025–1035, Red Hook, NY, USA, 2017. Curran Associates Inc.  ISBN 9781510860964.
  * Veličković et al. [2018] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.  Graph Attention Networks.  _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=rJXMpikCZ.  accepted as poster.
  * Frasca et al. [2020] Fabrizio Frasca, Emanuele Rossi, Davide Eynard, Ben Chamberlain, Michael Bronstein, and Federico Monti.  Sign: Scalable inception graph neural networks, 2020.  URL https://arxiv.org/abs/2004.11198.
  * Li et al. [2021] Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun.  Training graph neural networks with 1000 layers.  _CoRR_ , abs/2106.07476, 2021.  URL https://arxiv.org/abs/2106.07476.
  * Barabási and Pósfai [2017] Albert-László Barabási and Márton Pósfai.  _Network science: graph theory_ , page 17–29.  Cambridge University Press, 2017.
  * Hu et al. [2020] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec.  Open graph benchmark: Datasets for machine learning on graphs.  _CoRR_ , abs/2005.00687, 2020.  URL https://arxiv.org/abs/2005.00687.
  * Sen et al. [2008] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad.  Collective classification in network data.  _AI Magazine_ , 29(3):93, Sep. 2008.  doi:10.1609/aimag.v29i3.2157.  URL https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2157.
  * Beltagy et al. [2019] Iz Beltagy, Kyle Lo, and Arman Cohan.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi:10.18653/v1/D19-1371.  URL https://aclanthology.org/D19-1371.
  * Schlichtkrull et al. [2018] Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling.  Modeling relational data with graph convolutional networks.  In Aldo Gangemi, Roberto Navigli, Maria-Esther Vidal, Pascal Hitzler, Raphaël Troncy, Laura Hollink, Anna Tordai, and Mehwish Alam, editors, _The Semantic Web_ , pages 593–607, Cham, 2018. Springer International Publishing.  ISBN 978-3-319-93417-4.
  * Wu et al. [2019a] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu.  A comprehensive survey on graph neural networks.  _CoRR_ , abs/1901.00596, 2019a.  URL http://arxiv.org/abs/1901.00596.
  * Bing et al. [2022] Rui Bing, Guan Yuan, Mu Zhu, Fanrong Meng, Huifang Ma, and Shaojie Qiao.  Heterogeneous graph neural networks analysis: a survey of techniques, evaluations and applications.  _Artificial Intelligence Review_ , pages 1–40, 12 2022.  doi:10.1007/s10462-022-10375-2.
  * Priem et al. [2022] Jason Priem, Heather Piwowar, and Richard Orr.  Openalex: A fully-open index of scholarly works, authors, venues, institutions, and concepts, 2022.  URL https://arxiv.org/abs/2205.01833.
  * Wu et al. [2019b] Felix Wu, Tianyi Zhang, Amauri H. Souza Jr., Christopher Fifty, Tao Yu, and Kilian Q. Weinberger.  Simplifying graph convolutional networks.  _CoRR_ , abs/1902.07153, 2019b.  URL http://arxiv.org/abs/1902.07153.
  * Prieto et al. [2023] Lucas Prieto, Jeroen Den Boef, Paul Groth, and Joran Cornelisse.  Parameter efficient node classification on homophilic graphs.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=LIT8tjs6rJ.
  * Huang et al. [2021] Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin R. Benson.  Combining label propagation and simple models out-performs graph neural networks.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.  URL https://openreview.net/forum?id=8E1-f3VhX1o.
  * Yang et al. [2022] Xiaocheng Yang, Mingyu Yan, Shirui Pan, Xiaochun Ye, and Dongrui Fan.  Simple and efficient heterogeneous graph neural network, 2022.  URL https://arxiv.org/abs/2207.02547.
  * Fey and Lenssen [2019] Matthias Fey and Jan E. Lenssen.  Fast graph representation learning with PyTorch Geometric.  In _ICLR 2019 Workshop on Representation Learning on Graphs and Manifolds_ , 2019.  URL https://arxiv.org/abs/1903.02428.
  * Zhang et al. [2022] Fanjin Zhang, Xiao Liu, Jie Tang, Yuxiao Dong, Peiran Yao, Jie Zhang, Xiaotao Gu, Yan Wang, Evgeny Kharlamov, Bin Shao, Rui Li, and Kuansan Wang.  Oag: Linking entities across large-scale heterogeneous knowledge graphs.  _IEEE Transactions on Knowledge and Data Engineering_ , pages 1–14, 2022.  doi:10.1109/TKDE.2022.3222168.
  * Namata et al. [2012] Galileo Mark Namata, Ben London, Lise Getoor, and Bert Huang.  Query-driven active surveying for collective classification.  In _International Workshop on Mining and Learning with Graphs_ , Edinburgh, Scotland, 2012.
  * Guo et al. [2017] Lantian Guo, Xiaoyan Cai, Fei Hao, Dejun Mu, Changjian Fang, and Libin Yang.  Exploiting fine-grained co-authorship for personalized citation recommendation.  _IEEE Access_ , 5:12714–12725, 2017.  doi:10.1109/ACCESS.2017.2721934.
  * Cohan et al. [2020] Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld.  SPECTER: Document-level representation learning using citation-informed transformers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2270–2282, Online, July 2020. Association for Computational Linguistics.  doi:10.18653/v1/2020.acl-main.207.  URL https://aclanthology.org/2020.acl-main.207.
  * Schank and Wagner [2005] Thomas Schank and Dorothea Wagner.  Approximating clustering coefficient and transitivity.  _Journal of Graph Algorithms and Applications_ , 9, 01 2005.  doi:10.7155/jgaa.00108.
  * Ma et al. [2022] Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang.  Is homophily a necessity for graph neural networks?  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=ucASPPD9GKN.
  * Watts and Strogatz [1998] Duncan J. Watts and Steven H. Strogatz.  Collective dynamics of ‘small-world’ networks.  _Nature_ , 393(6684):440–442, 1998.  doi:10.1038/30918.
  * Xu et al. [2018] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka.  Representation learning on graphs with jumping knowledge networks.  _CoRR_ , abs/1806.03536, 2018.  URL http://arxiv.org/abs/1806.03536.
  * Pei et al. [2020] Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang.  Geom-gcn: Geometric graph convolutional networks.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=S1e2agrFvS.
  * Chen et al. [2020] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li.  Simple and deep graph convolutional networks.  In Hal Daumé III and Aarti Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 1725–1735. PMLR, 13–18 Jul 2020.  URL https://proceedings.mlr.press/v119/chen20v.html.
  * Rong et al. [2020] Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang.  Dropedge: Towards deep graph convolutional networks on node classification.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=Hkx1qkrKPr.
  * Duan et al. [2023] Keyu Duan, Qian Liu, Tat-Seng Chua, Shuicheng Yan, Wei Tsang Ooi, Qizhe Xie, and Junxian He.  Simteg: A frustratingly simple approach improves textual graph learning, 2023.  URL https://arxiv.org/abs/2308.02565.

## Appendix A. Hyperparameters

The parameters for the learning rate scheduler
(torch.optim.lr_scheduler.ReduceLROnPlateau) were kept constant in all
experimental settings: 0.1 factor, 10 patience, 0.0001 threshold, 0.0001
min_lr, 20 cooldown.

Table 5: Hyperparameters used for reported ogbn-arxiv results in Table 3.
Model | Val. Acc. / Test Acc. | Baseline Acc.
---|---|---
GCN |  | num_layers: 2, hidden_channels: 256,
---
weight_decay: 0.001, lr: 0.01, dropout: 0.2
Existing leaderboard submission.
GCN+JK |  | num_layers: 2, hidden_channels: 256,
---
weight_decay: 0.003, lr: 0.01, dropout: 0
Existing leaderboard submission.
SAGE |  | num_layers: 2, hidden_channels: 256,
---
weight_decay: 0.001, lr: 0.01, dropout: 0.2
Existing leaderboard submission.
SGC | K: 2, weight_decay: 0, lr: 0.01 | K: 3, weight_decay: 0, lr: 0.1
Table 6: Hyperparameters used for reported PubMed results in Table 4.
Model | Val. Acc. / Test Acc. | Baseline Acc.
---|---|---
GCN |  | num_layers: 2, hidden_channels: 128,
---
weight_decay: 0.005, lr: 0.001, dropout: 0
Existing benchmark from Chen et al. [2020].
GCN+JK |  | num_layers: 2, hidden_channels: 32,
---
weight_decay: 0.005, lr: 0.001, dropout: 0
Existing benchmark from Chen et al. [2020].
SAGE |  | num_layers: 2, hidden_channels: 128,
---
weight_decay: 0.005, lr: 0.001, dropout: 0
| num_layers: 2, hidden_channels: 128,
---
weight_decay: 0.005, lr: 0.001, dropout: 0
SGC | K: 3, weight_decay: 0, lr: 0.1 | K: 3, weight_decay: 0, lr: 0.1


# Zero-shot Domain-sensitive Speech Recognition
with prompt-conditioning fine-tuning

###### Abstract

In this work, we propose a method to create domain-sensitive speech
recognition models that utilize textual domain information by conditioning its
generation on a given text prompt. This is accomplished by fine-tuning a pre-
trained, end-to-end model (Whisper) to learn from demonstrations with prompt
examples. We show that this ability can be generalized to different domains
and even various prompt contexts, with our model gaining a Word Error Rate
(WER) reduction of up to 33% on unseen datasets from various domains, such as
medical conversation, air traffic control communication, and financial
meetings. Considering the limited availability of audio-transcript pair data,
we further extend our method to text-only fine-tuning to achieve domain
sensitivity as well as domain adaptation. We demonstrate that our text-only
fine-tuned model can also attend to various prompt contexts, with the model
reaching the most WER reduction of 29% on the medical conversation dataset.

††footnotetext: *Equal contribution. †Work done during internship at MediaTek
Research. ‡ Project lead. Correspondence to
<EMAIL_ADDRESS>Code available at
https://github.com/mtkresearch/clairaudience

Index Terms—  Speech recognition, prompt-conditioning, text-only fine-tuning,
language model

##  1 Introduction

In the field of Automatic Speech Recognition (ASR), there has been a rapid
deployment of ASR systems to downstream applications in a multitude of
industry sectors, such as finance, healthcare, and transportation [1, 2, 3].
Recent work shows the potential of improving existing ASR systems with
identified domain scenarios [4, 2]. Domain-Prompts [4] tackles the challenge
by learning a small set of domain-specific parameters to rescore hypotheses
generated by a multi-domain ASR system whose inputs include domain names.
However, the strategy is challenging to scale to unseen domains due to the
resource-intensive requirements of retraining or maintaining copies of fine-
tuned models.

Meanwhile, in the natural language processing (NLP) field, pre-trained large
language models have exhibited excellent performance in various NLP tasks
through conditional prompts [5, 6, 7]. A recent study underscores the efficacy
of the prompting method where a text-classification model fine-tuned with
prompt learning allows for notable performance on datasets in unseen domains
at inference time [8]. Such an approach is appealing as it increases the
number of application domains without necessitating retraining. Concretely,
PADA [8] fine-tunes a T5 model [9] with prompts and domain scenarios such that
at inference time, the model can utilize prompt-generated hypotheses on unseen
domains for sentiment classification [8].

Inspired by the progress in NLP, in this work, we aim to develop a zero-shot
domain-sensitive speech recognition model that utilizes domain-specific
information without training on the target domain. Specifically, we formulate
the inclusion of the domain scenarios and tags, information easily obtainable
in applications, into a prompting task. We then fine-tune the Whisper model, a
large-scale pre-trained ASR model [10], with condition prompts. Through such
prompt-conditioning fine-tuning process, Whisper becomes sensitive to the
given prompts and utilizes available domain information effectively. We select
the Whisper model because it capitalizes rich Internet sources of paired
audio-transcript data and is trained with a simple encoder-decoder
architecture [11] which has been shown to scale well and flexible for
including prompts [12, 9]. During the Whisper’s training phase, the Whisper
decoder has processed around 6B tokens over a diverse set of web transcripts,
making the decoder a robust language model for the ASR system. We thus
postulate that the Whisper model has enough language ability to be further
fine-tuned for becoming domain-sensitive. We call Whisper models fine-tuned
with our text-prompts - Clairaudience. We show Clairaudience fine-tuned on
Gigaspeech dataset [13] exhibits domain sensitivity in leveraging prior
information in Medical [14], UWB-ATCC (ATC) [15], ATCO2 [16], Earnings-22 [3]
datasets. Empirical results on these domain-specific datasets present a
consistent reduction in terms of Word Error Rate (WER) from 17.4% to 33.0%.

Furthermore, acquiring large quantities of labeled data may be challenging,
and only unlabelled text is available for application domains [17, 18]. We
showcase that rendering Whisper domain sensitive using text-only data with
prompt-conditioning fine-tuning is achievable. We dub models fine-tuned with
the approach - Clairaudience-text-only. Experimental outcomes of
Clairaudience-text-only fine-tuned on Gigaspeech text demonstrate that the
model shows no forgetting to attend to audio features and yields zero-shot
improvements over the selected domains. The effectiveness of the text-only
results primes us to investigate whether the model can be improved by fine-
tuning on domain-text directly. This setting is most similar to the
conventional domain adaptation. We find that Whisper fine-tuned on domain text
achieves WER similar to but lower than Clairaudience over the ATC dataset and
the splited Medical dataset.

Below we summarize our contribution:

  * •

We address the challenge of incorporating domain-specific information, such as
scenarios and tags, into the Whisper model during inference without retraining
on unseen domains.

  * •

We propose to enable a domain-sensitive Whisper by fine-tuning the model with
(i) audio-transcript pairs or with (ii) text-only data on the GigaSpeech
dataset with text prompts that include domain tags generated by GPT3.5 [5].

  * •

Our experiments show that models fine-tuned with text prompts are zero-shot
domain-sensitive and showcase significant WER improvement over medical,
finance, and transportation datasets.

##  2 related work

Our work relates to several broad research areas, including prompt-
engineering, multi-domain ASR, contextual-biasing, and text-only domain
adaptation. In the applications to speech processing systems, prompt-
engineering has only attracted attention lately. Work in [19] prompts a frozen
Whisper to unseen tasks, such as audio-visual recognition, code-switched
speech recognition, and speech-translation. SpeechPrompt [20] applies a fine-
tuned wave2vec2 model to produce prompts as inputs to GPT2 for audio
classification. Additionally, text-only training has been extensively studied
in domain adaptation. Work of [21] fine-tunes an external LM at target domains
for re-scoring the outputs of ASR systems to achieve domain adaptation.
Shallow fusion linearly interpolates results between an external LM and an ASR
system [22]. In works employing contextual biasing, trained light-weight
adapters embed contextual phrases to bias the embedding process in models,
thereby enhancing models’ ability to recognize rare words [23, 24, 25].
Domain-Prompts [4], perhaps the most similar work to ours, takes as input the
audio and domain names and outputs re-scored transcription selected by a
prompt-tuned GPT2 with domain-specific parameters incorporated. However, we
differ from these works in that we consider the setting where multiple domains
are addressed by contextually prompting a fine-tuned model with domain
information.

##  3 Background

We built our approach upon the Whisper model; in this section, we provide a
brief overview of the Whisper model, focusing on its connection to prompting.

Whisper shows that ASR models trained with weakly supervised audio-transcript
at scale can transfer well to existing datasets zero-shot. The Whisper model
leverages encoder-decoder architecture [11], which has shown to scale well
[26] and showcase competitive few-shot performance on tasks without fine-
tuning [5]. During training, the model is exposed to various tasks such as
voice-activity detection, speech transcription, translation, and language
identification. The encoder module processes log-Mel spectrograms as inputs
and produces audio features. The decoder, in turn, takes these audio features
along with an additional prompt token sequence to generate audio transcripts.

The Whisper model can be conditioned to perform speech translation or
transcription by specifying a series of task-specific tokens as input prompts.
For instance, during inference, the token sequence can be
<|startoftranscript|> <|english|> <|transcribe|> for speech transcription,
while <|startoftranscript|> <|german|> <|translate|> can be employed for
speech translation from German to English. During the pre-training, the model
was trained with history text prepended as context to resolve ambiguous audio
with long-range transcription. We thus hypothesize that, by feeding custom
prompts as context text to a pre-trained Whisper, we can condition the model
to leverage domain scenarios/tags.

Fig. 1: Cliaraudience and Clairaudience-text-only. (a) During the train time
of the Clairaudience, the model takes in audio-text pairs with text having
domain prompts included. The model carries out next-token prediction of the
audio transcript. During the inference time, the model takes in audio-prompt
pairs where each prompt includes generic domain tags from target domains. The
model then generate transcripts autoregressively. (b) During the train time of
the Clairaudience-text-only, only the transformer decoder is trained. The
outputs of the cross attention at decoder layers are replaced with learnable
bias terms to represent audio features. During the inference time, the bias
terms are turned off, and cross-attention outputs are used instead. The
inference process of Clairaudience-text-only is the same as that of
Clairaudience and Whisper. Figure adapted from [10].

##  4 Prompt-Conditioning Fine-Tuning

Our objective is to fine-tune an end-to-end speech recognition model to have
the ability to utilize the information provided by a text prompt to reduce the
WER of the generated transcript. We expect the fine-tuned model will develop
the capability to differentiate between phrases with similar pronunciations
and recognize domain-specific words by conditioning on the text prompt. In
this paper, we focus on text prompts that contain comprehensive descriptions
of the broader context, scenario, and genre of the speech, information not
captured in the audio clip. Our approach can be easily extended to accommodate
other prompt formats, contexts, and even other modalities of prompts.

Due to the absence of publicly available speech recognition datasets
associating audio clips with text prompts, we have taken the initiative of
generating this dataset ourselves. An ideal approach will be to augment
Whisper’s original training data in order to avoid confounding our findings
with the improvement from training on extra data. As the dataset is not
released, we decided to use GigaSpeech [13], a dataset also collected from web
audio and exhibiting rich diversity in terms of speech context. We then
generate concise and diverse domain tags based on the audio transcripts,
resembling extensive work in large language models (LLM) for generating data
for supervision tasks [27, 28, 29]. For each full audio transcript, we query
GPT3.5 to generate ten domain tags in line with domain categories inherently
available in the Gigaspeech dataset [13].

With flexibility and extensibility in mind, we decided on inserting the prompt
between the `<|prev|>` token and `<|startoftranscript|>` token with the format
<|prev|>[ domain: {DOMAIN_1}, {DOMAIN_2}, ...]<|startoftranscript|> following
by the rest of decoder input sequence, where `{DOMAIN_n}` represent the domain
tags (see Figure 1). During training, the domain tags are sampled and
incorporated into prompt text added to the decoder input.

##  5 Text-only training

Whisper follows the encoder-decoder transformer architecture. At each decoder
block, cross attention is performed between the layer input and the encoder
output embedding, which depends on the audio clip. We remove this attention
mechanism by replacing the output of each cross-attention layer with trainable
biases, as illustrated in Figure 1. The addition of bias term is inspired by
classifier-free guidance [30]. The model is then tuned with conventional
sequence-to-sequence training.

As the model is fine-tuned with text-only data, the model may be prone to
catastrophic forgetting [31, 32]. Prior work [32] addresses this issue through
a two-stage process that initially utilizes audio-text fine-tuning,
subsequently transitioning to text-only fine-tuning. However, we found that
pre-trained Whisper is robust to catastrophic forgetting when the model is
prompt-conditioning fine-tuned on text-only data (See Section 7.1). Therefore,
we conduct all our text-only fine-tuning without interleaving any audio-text
fine-tuning.

Model | Domain Prompts | General Domain | Specific Domain
---|---|---|---
train | inference | GigaSpeech-Test | Medical | ATC | ATCO2 | Earnings-22
Whisper | No | No | 12.36 | 8.79 | 73.82 | 42.93 | 17.86
Yes | 18.88 | 6.61 | 56.04 | 45.76 | 26.38
Clairaudience | Yes | No | 11.28 | 7.29 | 61.44 | 35.73 | 17.49
Yes | 9.22 | 6.54 | 52.47 | 28.77 | 14.75
Clairaudience-text-only | Yes | No | 11.70 | 7.55 | 70.45 | 42.11 | 16.69
Yes | 17.04 | 6.25 | 56.70 | 41.07 | 22.93
Table 1: Comparison between not-tuned, audio-text tuned, and text-only tuned Whisper on prompt conditioning transcribing. Note that GigaSpeech and Earnings-22 are long-form audio datasets; Medical, ATC, ATCO2 are short-form audio datasets. The underlined scores mark the degraded performance on long-form datasets, further discussed in Section 7.2. Model | Domain Prompts | General Domain | Specific Domain
---|---|---|---
train | inference | GigaSpeech-Test | Earnings-22
Whisper | No | Yes | 11.61 | 16.34
Clairaudience-text-only | Yes | Yes | 11.67 | 15.84
Table 2: Evaluation of not-tuned and text-only tuned Whisper after pre-pending
0.5 seconds of empty audio pads to input audio features. The padding is for
simulating speech naturalness [33] to correct errors due to segmented audio
clips from long-form audio datasets.

##  6 EXPERIMENTAL SETUP

###  6.1 Evaluation Details

Our experiments are conducted using Whisper-large-v2 (1550M
parameters)111https://github.com/openai/whisper, the largest version of the
Whisper model family, as the base model and comparison baseline.

We generate transcripts with three beams on top candidates for reporting all
the results in the paper. Following Whisper’s design, we indicate the start of
prediction with <|startoftranscript|> token and specify the task as English
transcribing by setting the language token to <|english|> and the task token
to <|transcribe|>. A <|notimestamps|> token is then included to indicate that
timestamp prediction is not required. If a text prompt is given to the
decoder, it will be pre-pended to the <|startoftranscript|> token with a
<|prev|> token in the front. Below is an example of the decoder input during
inference time: <|prev|> [ domain: Finance, Real Estate, Investment]
<|startoftranscript|> <|english|> <|transcribe|>.

The performance is measured by Word Error Rate % (WER) and Word Error Rate
Reduction % (WERR) over the baseline system.

###  6.2 Data Preparation

Two types of datasets are required in our experiments: (i) a generic dataset
that resembles Whisper’s training data for fine-tuning with prompts and (ii)
domain-specific datasets where domain tags can be extracted or assigned for
evaluating the zero-shot performance of the models.

As described in Section 4, we perform prompt-conditioning fine-tuning on the
training set of GigaSpeech Medium (1000 hours) with GPT3.5 generated domain
tags. We observe loss convergence when the model is trained on 100 hours of
audio, so the amount of data required for tuning is much lower than that for
pre-training an end-to-end ASR system.

To evaluate the domain sensitivity of the tuned model, we select datasets from
three domains: medical conversation, air traffic control communication, and
financial meetings. For medical conversation, we choose the ”Medical Speech,
Transcription, and Intent” (Medical) [14] dataset from Kaggle, which contains
8.5 hours of audio-text pairs and additional domain tags for each pair. The
domain tags are used for constructing text prompts in the domain prompt
experiment. For air traffic control, we use UWB-ATCC (ATC) [15] and the subset
of ATCO2 [16] annotated with airport information, each containing 10 and 2
hours of clips of air traffic control communication, respectively. For
financial meetings, we use Earnings-22 [3], consisting of about 100 hours of
English earning calls. We will discuss our methods of extracting and assigning
prompt information for individual datasets in Section 7.1.

Model | Domain Adaptation | Target domain
---|---|---
Method | Training data | ATC | Medical-split
Whisper | - | - | 73.82 | 6.07
Whisper (DA) | Fine-tuning on target domains | audio-text | 24.04 | 4.44
1-13-5 Whisper (DA-text-only) | text-only | 51.84 | 5.27
Clairaudience | Prompting fine-tuned model | audio-text | 52.47 | 5.86
1-13-5 Clairaudience-text-only | text-only | 56.7 | 5.34
Table 3: Comparison in WER between methods that achieve domain adaptation.
Whisper (DA) and Whisper (DA-text-only) are fine-tuned Whisper models on
target datasets, i.e., ATC and Medical-split. Note that we split the medical
dataset into train-test sets.

##  7 Experiments and results

###  7.1 Prompt Conditioning Tuning

This experiment aims to investigate the generalizability of the prompt
conditioning ability acquired through audio-text and text-only fine-tuning on
GigaSpeech with prompts that include GPT-generated domain tags. We aim to
determine if this ability can be extended to diverse data genres and different
types of prompt content without further adaptation.

We compare the performance in three different settings: (i) Whisper, (ii)
Clairaudience, and (iii) Clairaudience-text-only. Our models are trained with
prompts, and we evaluate models with or without prompts. Note that all the
tunings mentioned above are conducted using GigaSpeech, and the model has yet
to be exposed to the datasets used in our evaluation.

In addition to GigaSpeech, the general domain dataset which the model is tuned
upon, we evaluate the three settings on four domain-specific datasets -
Medical, ATC, ATCO2, and Earnings-22 - with unique prompt content. The text
prompt generated for each dataset at inference time is explained as follows:

  * GigaSpeech-Test: We employ the identical prompt format as during training; instead of sampling, we provide the model with all ten domain phrases generated by GPT3.5. Noted that, in this section, we use the test set of GigaSpeech, which contains audio clips from a different set of web audio than that was used for training. Therefore, the model is evaluated using text prompts never encountered during training.

  * Medical: Each audio clip in the dataset is accompanied by a native symptom tag. In total, 25 concise tag phrases delineate the medical symptoms, including examples like ”Cough” and ”Stomach ache”. Leveraging these tags, we construct the text prompts associated with each audio clip.

  * ATC: As the dataset does not contain any information on individual audio, we use ”air traffic control” as the domain tag included in the text prompt for all the audio clips.

  * ATCO2: We construct the text prompts from the airport name, communication channel type, and airline call signs associated with each audio clip. An example of the domain content in a prompt will be ”Sion airport, control channel, callsigns: Alitalia Hansa KLM…”

  * Earnings-22: Similar to ATC, the dataset contains no information on individual audio. In this case, we use the domain phrases ”shareholders meeting”, ”earning call”, ”investment webinar”, and ”financial news”.

The results of prompt conditioning transcription are presented in Table 1. We
observe that Whisper before fine-tuning possesses some ability to utilize the
information from the text prompt to improve its transcribing accuracy.
Surprisingly, it can reduce the WER from 73.82% to 56.04% on ATC with a simple
prompt ”air traffic control”. However, on GigaSpeech-Test, ATCO2, and
Earnings-22, adding a prompt to a not-tuned Whisper negatively impacts its
prediction. In contrast, our audio-text tuned model (Clairaudience) shows
improvement on all the datasets. The model demonstrates a strong ability to
utilize the given text prompt to reduce the WER of its prediction. Consistent
with its performance on the GigaSpeech-Test, it achieves significant WER
improvement on all the datasets. As to the text-only model (Clairaudience-
text-only), it shows no catastrophic forgetting in attending to audio
features. The model demonstrates a weaker ability than the audio-text-tuned
model but still outperforms the prompted Whisper on Medical and ATCO2
datasets. It achieves a lower WER on the Medical dataset than the audio-text-
tuned model. However, similar to the prompted Whisper, the evaluation result
of Clairaudience-text-only with prompt is worse than that without prompt on
GagaSpeech-Test and Earnings-22. Such performance degradation is partly
attributed to segmented audio in long-form audio datasets (audio length > 30
seconds). On the other hand, Clairaudience is immune to the problem and
achieves the lowest WER.

Interestingly, we observe that evaluation of Clairaudience and Clairaudience-
text-only without prompts shows improvement over the domain-specific datasets,
even though the models are fine-tuned on a general domain dataset. We
postulate that such gain is contributed by better audio feature and text
feature alignment learned from the general domain dataset, which is
potentially presented in specific domains. Nevertheless, the outcomes of
Clairaudience indicate that using domain prompts during the inference stage
persistently reduces the WER on all domain-specific datasets, when contrasted
between inferences with and without prompts.

###  7.2 Speech Naturalness for Segmented Audio

Given that both prompted Whisper and Clairaudience-text-only show degraded
performance when evaluated on long-form audio datasets, i.e., GigaSpeech and
Earnings-22, we investigated the audio inputs and found that both models often
fail to transcribe segmented audios of unfinished sentences. Audios starting
with unfinished sentences result in models not outputting transcripts or
hallucinating. To solve the issue, we experiment with adding 0.5-second empty
audio to the beginning of the input audio. The intuition is that the padded
audio simulates the speech naturalness within sentences [33], which we assume
to be presented in Whisper’s pre-training data for unfinished sentences.
Results in Table 2 demonstrate that the heuristic effectively corrects errors.
Clairaudience-text-only even outperforms the prompted Whisper on Earnings-22,
consistent with results on other domain-specific datasets.

###  7.3 Text-Only Domain Adaptation

Given the results of Clairaudience-text-only, we have demonstrated the
effectiveness in prompting the text-only fine-tuned model in various domains.
In another research setting similar to ours - domain-adaptation, fine-tuning
models on target-domain is typical [18, 4, 17]. We thus investigate whether
the performance of the Whisper model can be improved by applying the text-only
fine-tuning process under the conventional domain adaption setting.

We adapt Whisper to air traffic control communication and medical conversation
with audio-text pair fine-tuning and our text-only fine-tuning method. We
split the Medical dataset into adaptation and evaluation sets with an 85:15
ratio, as the default datasets do not have train-test splits. Whisper is then
fine-tuned with the audio transcript or the transcript of the adaptation set
and assessed on the audio-transcript pairs of the evaluation set.

The result is presented in Table 3. As expected, the fine-tuned Whisper model
with audio-transcript pairs on the target domain (Whisper (DA)) achieves the
lowest WER. Compared with the not-tuned Whisper model, the text-only fine-
tuned model on target domains (Whisper (DA-text-only)) achieves a significant
20% and 13% WERR improvement on ATC and Medical-split, respectively. To our
surprise, the performance of Whisper (DA-text-only) is only 0.63% and 0.07% in
absolute WER difference to Clairaudience on ATC and Clairaudience-text-only on
Medical-split, respectively. The result underscores the effectiveness of
applying our prompt-conditioning fine-tuning process to a pre-trained end-to-
end ASR system in achieving performant ASR results through prompts instead of
the conventional domain adaptation method.

##  8 DISCUSSION

Our paper has explored a simple question: Can a fine-tuned ASR model with
prompt-conditioning leverages domain information as prompts to boost
performance across unseen domains without additional retraining? Our fine-
tuned models, Clairaudience and Clairaudience-text-only, improve performance
against a not-tuned Whisper model and surpass the evaluation results of a
prompted Whisper on most domain-specific datasets. Further experiments on
text-only domain adaptation reveal that a prompted Clairaudience model can
achieve similar performance in domain adaptation to the Whisper model fine-
tuned directly on target domains with text-only data.

As for limitations of our study, there is a degree of subjectivity in the
selected domains for evaluation, and our evaluation method work most robustly
to short-audio transcription. Our first limitation is that individual domain-
specific information might have appeared in the GigaSpeech training set. In
Table 1, we observe improvements over the inference setting without prompts
for the fine-tuned model. Such improvement could be attributed to the
aforementioned data overlapping between fine-tuning and testing data.
Furthermore, we observe that the performance of Clairaudicne, compared with
Whisper, exhibits a decrease of up to 0.95% in absolute WER on other general
domain datasets, namely Common Voice [34] and LibriSpeech [35]. The difference
could be attributed to distribution deviation in fitting the model to
GigaSpeech dataset, which might not be as comprehensive and diverse as the
Whisper pre-training dataset. Another limitation is that Whisper was trained
on audio segments (30 seconds) but not on arbitrarily long audio inputs due to
memory constraints. To handle long-form transcription, the Whisper model
inserts previously generated text between `<|prev|>` token and
<|startoftranscript|> token to inform the model of the speech that precedes
the current segment. Despite wrapping our text prompt with brackets, we
observe that the model tuned on text-only data inherits this mechanism and
mistakes the prompt as previous text when the input audio is part of an
unfinished sentence as discussed in Section 7.2. This behavior is even more
presented in the not-tuned model. Interestingly, despite having never seen the
setting where prompt and previous text coexisted, the model trained on audio-
text data is immune to such a problem. Therefore, we hypothesize that this
problem can be better solved by mixing text-only tuning with a small amount of
audio-text tuning or leveraging the voice detection task in Whisper as work in
[36] and leaving the study for future work.

##  9 CONCLUSION

In this paper, we present a method for fine-tuning end-to-end ASR models to
achieve generalization over multiple domains by prompting the fine-tuned model
with domain information. We demonstrate the efficacy of the method in models
fine-tuned with audio-transcript or with text-only data. Our experimental
outcome shows that: (i) Clairaudience, prompt-conditioning fine-tuned on
GigaSpeech, is zero-shot domain-sensitive on Medical, ATC, ATCO2, and
Earnings-22 datasets. It achieves the best WER in comparison to Whisper and
Clairaudience-text-only; (ii) Whisper can be rendered domain-sensitive with
text-only fine-tuning of its decoder. The resulting Clairaudience-text-only
model outperforms Whisper on Medical, ATCO2, and Earnings-22 datasets; (iii)
Through domain prompts, Clairaudience can perform similarly to Whisper (DA-
text-only), which is text-only fine-tuned directly on ATC and Medical-split
datasets. We hope that our paper will motivate further investigation into more
prompt-based ASR systems, zero-shot learning, and using text-only data to
improve end-to-end ASR systems.

## References

  * [1] Antonia Schulte, Rodrigo Suarez-Ibarrola, Daniel Wegen, Philippe-Fabian Pohlmann, Elina Petersen, and Arkadiusz Miernik,  “Automatic speech recognition in the operating room – An essential contemporary tool or a redundant gadget? A survey evaluation among physicians in form of a qualitative study,”  Annals of Medicine and Surgery, vol. 59, pp. 81–85, Nov. 2020.
  * [2] Iuliia Nigmatulina, Rudolf Braun, Juan Zuluaga-Gomez, and Petr Motlicek,  “Improving callsign recognition with air-surveillance data in air-traffic communication,” Aug. 2021,  arXiv:2108.12156 [cs, eess].
  * [3] Miguel Del Rio, Peter Ha, Quinten McNamara, Corey Miller, and Shipra Chandra,  “Earnings-22: A Practical Benchmark for Accents in the Wild,” Mar. 2022,  arXiv:2203.15591 [cs].
  * [4] Saket Dingliwal, Ashish Shenoy, Sravan Bodapati, Ankur Gandhe, Ravi Teja Gadde, and Katrin Kirchhoff,  “Domain Prompts: Towards memory and compute efficient domain adaptation of ASR systems,”  in Interspeech 2022. Sept. 2022, pp. 684–688, ISCA.
  * [5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei,  “Language models are few-shot learners,”  arXiv, , no. NeurIPS, 2020,  arXiv: 2005.14165.
  * [6] OpenAI,  “GPT-4 Technical Report,” Mar. 2023,  arXiv:2303.08774 [cs].
  * [7] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le,  “Finetuned Language Models Are Zero-Shot Learners,”  Sept. 2021,  arXiv: 2109.01652.
  * [8] Eyal Ben-David, Nadav Oved, and Roi Reichart,  “PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains,” Jan. 2022,  arXiv:2102.12206 [cs].
  * [9] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu,  “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,”  Tech. Rep., 2020,  arXiv: 1910.10683v3 Publication Title: Journal of Machine Learning Research Volume: 21.
  * [10] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever,  “Robust Speech Recognition via Large-Scale Weak Supervision,” Dec. 2022,  arXiv:2212.04356 [cs, eess].
  * [11] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in Neural Information Processing Systems, vol. 2017-Decem, no. Nips, pp. 5999–6009, 2017,  arXiv: 1706.03762.
  * [12] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever,  “Language Models are Unsupervised Multitask Learners,”  Tech. Rep., 2018.
  * [13] Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Yujun Wang, Zhao You, and Zhiyong Yan,  “GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio,” June 2021,  arXiv:2106.06909 [cs, eess].
  * [14] Figure Eight Inc.,  “Medical Speech, Transcription, and Intent,” 2019.
  * [15] Luboš Šmídl, Jan Švec, Daniel Tihelka, Jindřich Matoušek, Jan Romportl, and Pavel Ircing,  “Air traffic control communication (ATCC) speech corpora and their use for ASR and TTS development,”  Language Resources and Evaluation, vol. 53, no. 3, pp. 449–464, Sept. 2019.
  * [16] Igor Szoke, Santosh Kesiraju, Ondrej Novotny, Martin Kocour, Karel Vesely, and Jan ”Honza” Cernocky,  “Detecting English Speech in the Air Traffic Control Voice Communication,” Apr. 2021,  arXiv:2104.02332 [eess].
  * [17] Emiru Tsunoo, Yosuke Kashiwagi, Chaitanya Narisetty, and Shinji Watanabe,  “Residual Language Model for End-to-end Speech Recognition,” June 2022,  arXiv:2206.07430 [cs, eess].
  * [18] Keqi Deng and Philip C. Woodland,  “Adaptable End-to-End ASR Models using Replaceable Internal LMs and Residual Softmax,” Mar. 2023,  arXiv:2302.08579 [cs, eess].
  * [19] Puyuan Peng, Brian Yan, Shinji Watanabe, and David Harwath,  “Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization,” May 2023,  arXiv:2305.11095 [cs, eess].
  * [20] Kai-Wei Chang, Wei-Cheng Tseng, Shang-Wen Li, and Hung-yi Lee,  “SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks,” July 2022,  arXiv:2203.16773 [cs, eess].
  * [21] Linda Liu, Yile Gu, Aditya Gourav, Ankur Gandhe, Shashank Kalmane, Denis Filimonov, Ariya Rastrow, and Ivan Bulyko,  “Domain-aware Neural Language Models for Speech Recognition,” Feb. 2021,  arXiv:2101.03229 [cs, eess].
  * [22] Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio,  “Attention-Based Models for Speech Recognition,” June 2015,  arXiv:1506.07503 [cs, stat].
  * [23] Yaoxun Xu, Baiji Liu, Qiaochu Huang and, Xingchen Song, Zhiyong Wu, Shiyin Kang, and Helen Meng,  “CB-Conformer: Contextual biasing Conformer for biased word recognition,” Apr. 2023,  arXiv:2304.09607 [cs, eess].
  * [24] Kanthashree Mysore Sathyendra, Thejaswi Muniyappa, Feng-Ju Chang, Jing Liu, Jinru Su, Grant P. Strimel, Athanasios Mouchtaris, and Siegfried Kunzmann,  “Contextual Adapters for Personalized Speech Recognition in Neural Transducers,” May 2022,  arXiv:2205.13660 [cs].
  * [25] Tsendsuren Munkhdalai, Khe Chai Sim, Angad Chandorkar, Fan Gao, Mason Chua, Trevor Strohman, and Françoise Beaufays,  “Fast Contextual Adaptation with Neural Associative Memory for On-Device Personalized Speech Recognition,” Oct. 2021,  arXiv:2110.02220 [cs, eess].
  * [26] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei,  “Scaling Laws for Neural Language Models,”  Jan. 2020,  arXiv: 2001.08361.
  * [27] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto,  “Stanford Alpaca: An Instruction-following LLaMA model,” 2023,  Publication Title: GitHub repository.
  * [28] Ronen Eldan and Yuanzhi Li,  “TinyStories: How Small Can Language Models Be and Still Speak Coherent English?,” May 2023,  arXiv:2305.07759 [cs].
  * [29] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli,  “ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks,” Mar. 2023,  arXiv:2303.15056 [cs].
  * [30] Jonathan Ho and Tim Salimans,  “Classifier-Free Diffusion Guidance,” July 2022,  arXiv:2207.12598 [cs].
  * [31] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell,  “Overcoming catastrophic forgetting in neural networks,”  Proceedings of the National Academy of Sciences, vol. 114, no. 13, pp. 3521–3526, Mar. 2017.
  * [32] Zhong Meng, Yashesh Gaur, Naoyuki Kanda, Jinyu Li, Xie Chen, Yu Wu, and Yifan Gong,  “Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognition,” June 2022,  arXiv:2110.05354 [cs, eess].
  * [33] Shimeng Liu, Yoshitaka Nakajima, Lihan Chen, Sophia Arndt, Maki Kakizoe, Mark A. Elliott, and Gerard B. Remijn,  “How Pause Duration Influences Impressions of English Speech: Comparison Between Native and Non-native Speakers,”  Frontiers in Psychology, vol. 13, pp. 778018, Feb. 2022.
  * [34] Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber,  “Common Voice: A Massively-Multilingual Speech Corpus,” Mar. 2020,  arXiv:1912.06670 [cs].
  * [35] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: An ASR corpus based on public domain audio books,”  in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Apr. 2015, pp. 5206–5210,  ISSN: 2379-190X.
  * [36] Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman,  “WhisperX: Time-Accurate Speech Transcription of Long-Form Audio,” Mar. 2023,  arXiv:2303.00747 [cs, eess].