# RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses

Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni,
Xuanhui Wang and Michael Berdersky
Google Research
{hlz,zhenqin,jagerman,kaihuibj,maji,ljwinnie,jianmon,
<EMAIL_ADDRESS>
###### Abstract

Recently, substantial progress has been made in text ranking based on
pretrained language models such as BERT. However, there are limited studies on
how to leverage more powerful sequence-to-sequence models such as T5. Existing
attempts usually formulate text ranking as classification and rely on
postprocessing to obtain a ranked list. In this paper, we propose RankT5 and
study two T5-based ranking model structures, an encoder-decoder and an
encoder-only one, so that they not only can directly output ranking scores for
each query-document pair, but also can be fine-tuned with “pairwise” or
“listwise” ranking losses to optimize ranking performances. Our experiments
show that the proposed models with ranking losses can achieve substantial
ranking performance gains on different public text ranking data sets.
Moreover, when fine-tuned with listwise ranking losses, the ranking model
appears to have better zero-shot ranking performance on out-of-domain data
sets compared to the model fine-tuned with classification losses.

RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses

Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni,
Xuanhui Wang and Michael Berdersky Google Research
{hlz,zhenqin,jagerman,kaihuibj,maji,ljwinnie,jianmon,
<EMAIL_ADDRESS>

##  1 Introduction

Text ranking refers to the task of ranking a set of textual documents based on
their relevance to a given query or context. Learning a text ranking model is
a fundamental component of countless real world applications such as search
and question answering. Earliest explorations Liu (2009) mostly rely on
handcrafted numerical features to represent each query-document pair Qin and
Liu (2013); Chapelle and Chang (2011) and put more emphasis on the learning
algorithms such as ranking losses Qin et al. (2021). Progress on pretrained
language models in the past few years Devlin et al. (2019) and the release of
large-scale public data sets Bajaj et al. (2016); Kwiatkowski et al. (2019)
enable a series of work Lin et al. (2020); Nogueira et al. (2019); Han et al.
(2020) on text ranking models which directly encode textual query and document
using pretrained language models, noticeably BERT Devlin et al. (2019).

Recently, pretrained language models such as T5 Raffel et al. (2020) and GPT3
Brown et al. (2020) have shown superior performance in various NLP tasks
including sentiment analysis, coreference resolution, and translation. Such
models often have much larger size available than previous models such as BERT
Devlin et al. (2019) to store more hidden knowledge. They also mostly have a
sequence-to-sequence interface to unify different NLP tasks from
classification to text generation.

While BERT-based models have been well explored for text ranking Lin et al.
(2020); Nogueira et al. (2019); Han et al. (2020), how to leverage T5 for text
ranking is still under-explored and challenging. First, while many
classification and text generation tasks fit into the sequence-to-sequence
framework, it is more tricky for text ranking tasks: a text ranking model is
often expected to output a numerical ranking score $\hat{y}\in{\mathbb{R}}$
for each query-document pair. Second, it is important to train a text ranking
model with ranking losses to optimize its ranking performance, where the
losses take into account the ranking scores from multiple documents for each
query. This is different from the typical T5 fine-tuning strategy where the
objective is often formulated into a text generation loss for each single
input sequence independently.

A typical approach to use T5 for text ranking is to convert the problem into a
token generation problem. For example, Nogueira et al. (2020) fine-tune the T5
model to predict a “true” or “false” token for a relevant or irrelevant query-
document pair and then use a postprocessing step during inference to derive
ranking scores to rank candidate documents. Such an approach can be considered
a “pointwise” classification formulation. How to extend this approach to fine-
tune T5 with ranking losses is unclear.

In this paper, we propose RankT5 with the goal to support text ranking more
natively with T5 by outputting real numbers, instead of text tokens. We first
adapt the encoder-decoder structure for this goal. In addition, we also
propose an alternative structure which omits the decoder from T5 and outputs
real numbers based on the encoder, called the encoder-only structure. These
two structure variants allow us to fine-tune T5 with various ranking losses to
directly optimize ranking performance.

Experiments on MS MARCO and Natural Question data sets show that our RankT5
models fine-tuned with specialized ranking losses can significantly outperform
other T5 ranking models fine-tuned with classification losses and previously
proposed T5 adaptations for ranking Nogueira et al. (2020). We also discover
that models fine-tuned with some ranking losses tend to have better zero-shot
performance than models fine-tuned with classification losses.

##  2 Related Work

Leveraging pretrained language models for text ranking tasks is becoming the
state-of-the-art Lin et al. (2020). We review recent literature on utilizing
such models for text ranking tasks.

#### Model structure.

Pretrained language models accept a text sequence as input. A typical model
structure design is the cross-attention model structure, where a query and a
candidate document is concatenated into a sequence and fed into the model. It
allows the attention mechanism to fully capture the query-document
interaction. This model structure has been explored for BERT-like encoder-only
model Han et al. (2020); Nogueira et al. (2019); Gao et al. (2021). This is
also explored in the T5-like model Nogueira et al. (2020); Ju et al. (2021),
but the model is not directly fine-tuned with ranking losses for the optimal
ranking performance. Instead, the model is fine-tuned to generate “true” or
“false” tokens. The ranking scores are later derived in postprocessing from
the predicted logits. There are also model structures that take a triple with
one query and two documents as input Pradeep et al. (2021), but these models
can only be deployed in the _late ranking_ stage to work with tens of
documents because they require scoring all possible input document pairs,
which is not scalable. Our contributions are in the _early ranking_ stage
which scores thousands of input documents; they are thus complementary with
this work.

There are also works exploring ranking models based on generative likelihood
from language models. For example, existing works in Zhuang and Zuccon (2021);
Zhuang et al. (2021); Sachan et al. (2022); dos Santos et al. (2020) take a
candidate document as input and estimates the likelihood of generating the
query. Then they rank documents based on the query-likelihood or use it as an
auxiliary task Ju et al. (2021). These models convert ranking into a
generation task in order to leverage generative models such as T5, instead of
directly enabling T5 to output numerical scores.

There are a few other model structures proposed for retrieval tasks, where the
model needs to score hundreds of millions of documents in the entire corpus
almost instantly for each arriving query. These model structures emphasize
more on model efficiency. Typical examples for BERT-like models are dual-
encoder models Karpukhin et al. (2020) and encoders with late interaction
Khattab and Zaharia (2020); Gao et al. (2020); MacAvaney et al. (2020), where
the model computation of queries and documents are decomposed to allow
precomputations for documents in the corpus. Some other works also utilize the
predictive likelihood from generative T5-like models for retrieval. Examples
include using the likelihood of generating queries from documents Hui et al.
(2022) or using the likelihood of generative document IDs from queries Tay et
al. (2022). However, our paper focuses on the ranking task, where a small set
of candidates documents are already retrieved for each query. This allows us
to focus on cross-attention model structure and push for better ranking
performance.

#### Fine-tuning with ranking losses.

Early explorations Nogueira et al. (2019, 2020) of applying pretrained
language models on the document reranking task mainly use “pointwise” losses,
where the loss is calculated for each query-document pair independently. There
are some recent works that use a “listwise” loss Han et al. (2020); Gao et al.
(2021); Ren et al. (2021) which takes one positive and multiple negative
documents for each query and calculates the softmax loss over their ranking
scores. But they only fine-tune BERT, RoBERTa etc. There is no existing work
fine-tuning sequence-to-sequence models like T5 with _ranking losses_.

Some retriever models are fine-tuned using pairwise loss or softmax loss Xiong
et al. (2020); Karpukhin et al. (2020); Lu et al. (2021). However, in this
work we only focus on reranking models.

##  3 Preliminaries

#### Problem definition.

We provide the formalized definition of a ranking task. For each query
$q_{i}$, a list of candidate documents $D_{i}=(d_{i1},\ldots,d_{im})$ are
provided, which are usually the output from a retriever. The relevance labels
of candidate document with regard to the query are represented as
${\mathbf{y}}_{i}=(y_{i1},\ldots,y_{im})$ where $y_{ij}\geq 0$.

The objective is to train a ranking model $f$ which takes a query-document
pair as input and outputs a ranking score
$\hat{y}_{ij}=f(q_{i},d_{ij})\in\mathbb{R}$. We aim to optimize the ranking
metrics after we sort the documents $D_{i}$ for each query $q_{i}$ based on
their ranking scores.

In the ranking task, the number of documents in $D_{i}$ given for a query is
usually small and $f$ can be a high-capacity model such as a cross-attention
model. On the contrary, a retrieval task Qu et al. (2021) virtually needs to
score all the documents in a corpus with millions of documents. In this study
we only focus on the ranking task.

#### T5.

T5 Raffel et al. (2020) is a text-to-text pretrained generative language model
with an encoder-decoder structure based on transformers. The original T5 takes
a piece of text as input and outputs a sequence of text tokens in an
autoregressive manner. We can also obtain the output likelihood of each token
in the vocabulary ${\mathcal{V}}$ given the input text.

More formally, we denote the input to the T5 encoder as a text sequence
$s=[w_{1},\ldots,w_{l}]$ and the previously generated tokens from the T5
decoder as $t_{1:k-1}$ during the autoregressive decoding process. We
formalize the T5 model structure as:

$\displaystyle{\mathbf{p}}_{k}$ $\displaystyle=\text{T5}(s,t_{1:k-1})$
$\displaystyle=\text{Softmax}(\text{Dense}(\text{Dec}(\text{Enc}(s),t_{1:k-1})))$

where the output is a vector with the length of the vocabulary size
${\mathbf{p}}_{k}\in{\mathbb{R}}^{|{\mathcal{V}}|}$, representing the
predictive probability of each token in the vocabulary being generated at the
$k$-th position.

The following components are included in the T5 model structure:
$\text{Enc}(\cdot)$ is the encoder of T5 which outputs a sequence of embedding
vectors; $\text{Dec}(\cdot,\cdot)$ is the decoder of T5 which takes both the
encoder outputs and the previously generated tokens from the decoder as inputs
and outputs a single embedding vector; $\text{Dense}(\cdot)$ is a dense layer
that projects the embedding vector into a unnormalized logits vector with
length $|{\mathcal{V}}|$; and $\text{Softmax}(\cdot)$ normalizes the vector
into probabilities.

##  4 RankT5 Modeling

(a) monoT5 Nogueira et al. (2020)

(b) RankT5 (Encoder-Decoder)

(c) RankT5 (Encoder-Only)

Figure 1:  Illustration of the T5 ranking model structure from prior work and
the two variants of our proposed RankT5 model. The differences in training
mechanism is not reflected in this illustration.

###  4.1 Model structure

Applying T5 on generative tasks (e.g., summarization, translation) is
straightforward due to the sequence-to-sequence nature of these tasks. T5 can
also be straightforwardly adapted for classification tasks such as sentiment
analysis or textual entailment recognition Raffel et al. (2020). A commonality
of these tasks is that the output are text tokens or sequences of text tokens.
How to use T5 in a ranking model has not been studied widely because the
output are not tokens, but numerical scores used for sorting.

#### Existing work.

Nogueira et al. (2020) cast text ranking as classification to fit the T5 model
structure. They concatenate each query-document pair into an input sequence
and use “true” and “false” as target tokens to indicate whether they are
relevant or not; the T5 model is then fine-tuned as a text generation task.
After the model is fine-tuned, the ranking scores are obtained from the logits
of “true” and “false” by
$\hat{y}_{ij}=e^{{\mathbf{z}}_{({\texttt{true}})}}/(e^{{\mathbf{z}}_{({\texttt{true}})}}+e^{{\mathbf{z}}_{({\texttt{false}})}})$,
where ${\mathbf{z}}_{(w)}$ represents the logits of $w$ in the vocabulary
${\mathcal{V}}$. Figure 1(a) shows this model structure.

This existing method only obtains the numerical ranking score as a
postprocessing step after the fine-tuning is done. Therefore, the model is not
directly fine-tuned as a ranking model but as a text generation or
classification model, which may not optimize its ranking performance. We
propose to directly obtain the numerical ranking score as the model output.
Specifically, we implement two variants based on pretrained T5 models: an
encoder-decoder and an encoder-only model.

#### Input sequence.

For each candidate document $d_{ij}$ and its query $q_{i}$, we concatenate
them with prefix “Document:” and “Query:” respectively to construct the input
text $s_{ij}$:

$\displaystyle s_{ij}=\text{Query:}~{}q_{i}~{}\text{Document:}~{}d_{ij}$

The construction of input sequence is similar to Nogueira et al. (2020) except
that we do not include the “Relevant:” postfix. The postfix does not affect
the results in our experiments (Appendix A).

#### Encoder-decoder (EncDec).

This model variant is a simple adaptation of the T5 model by using the first
output token of the decoder. In this variant, we feed the input into a T5
model and obtain the unnormalized logits ${\mathbf{z}}$ over the entire
vocabulary:

$\displaystyle{\mathbf{z}}=\text{Dense}(\text{Dec}(\text{Enc}(s_{ij})))$ (1)

Notice that we omit the softmax layer over vocabulary so that the elements in
${\mathbf{z}}$ can be arbitrary real numbers. We also simplify the decoder
representation $\text{Dec}(\cdot,\cdot)$ into $\text{Dec}(\cdot)$ which only
takes the output from the encoder as its input, since we do not generate other
tokens from the decoder.

We then specify a special unused token in T5 vocabulary “<extra_id_10>” and
take its corresponding unnormalized logits as the ranking score:

$\displaystyle\hat{y}_{ij}={\mathbf{z}}_{({\texttt{<extra\\_id\\_10>}})}$ (2)

where we use the notation ${\mathbf{z}}_{(w)}$ to represent the logits
corresponding to the token $w\in{\mathcal{V}}$. The special token can be any
other unused token in the vocabulary. An illustration of this model structure
can be found in Figure 1(b).

While an experiment in Nogueira et al. (2020) shows that directly using a
single token’s logit (in their case
$\hat{y}_{ij}={\mathbf{z}}_{({\texttt{true}})}$) works poorly, notice that a
key difference in this work is that the training loss is a ranking loss
directly defined on top of the ranking score $\hat{y}_{ij}$, instead of using
the original text generation loss for T5 fine-tuning. We will provide more
details in Section 4.2.

#### Encoder-only (Enc).

Since we do not need to perform autoregressive decoding, the decoder of T5
could be unnecessary. Therefore, we also try an encoder-only variation.

We construct the input text $s_{ij}$ in a similar manner and feed them into
the T5 encoder. The output of the encoder $\text{Enc}(s_{ij})$ is a sequence
of embedding vectors $[{\mathbf{e}}_{1},\cdots,{\mathbf{e}}_{l}]$ with the
same sequence length of the input sequence.

$\displaystyle[{\mathbf{e}}_{1},\cdots,{\mathbf{e}}_{l}]=\text{Enc}(s_{ij})$
(3)

We use a pooling layer $\text{Pool}(\cdot)$ (e.g., using first-token’s
embedding or mean pooling) to aggregate them into a single embedding vector.
Then, we apply the dense layer which directly projects the embedding vector to
the ranking score $\hat{y}_{ij}$. To formally summarize the encoder-only
model:

$\displaystyle\hat{y}_{ij}=\text{Dense}(\text{Pool}(\text{Enc}(s_{ij})))$ (4)

Figure 1(c) summarizes the proposed model structure. The T5 encoder-only model
structure is very similar to BERT. The major difference is that the initial
model parameter checkpoint will be from T5 instead of BERT.

#### Remarks.

We propose to try the encoder-only structure because it is unclear how
important the decoder of T5 is for text ranking tasks, especially when there
is abundant labeled data to fine-tune the model. The decoder model structure
itself contains attention layers across all the embedding vectors from the
output of the encoder, essentially serving as a complicated pooling layer with
a lot more parameters. By replacing the decoder with a simple pooling layer in
the encoder-only structure, we remove both the potential hidden knowledge
learned by the decoder during the pretraining stage of T5, as well as the
extra modeling capability of pooling. Comparing the performance between these
two model structures can provide some insights on whether they are essential
for text ranking tasks.

###  4.2 Training

The adapted model structure enables the model to directly output the ranking
score for each query-document pair. We can thereby adopt the training manner
similar to learning-to-rank models.

For each query $q_{i}$ and a list of its candidate documents $D_{i}$, we
obtain the list of predicted ranking scores $\hat{{\mathbf{y}}}_{i}$ by
applying the model on each query-document pair $(q_{i},d_{ij})$ where
$d_{ij}\in D_{i}$. Then we can train the model by optimizing a ranking-based
training loss function $\ell({\mathbf{y}}_{i},\hat{{\mathbf{y}}}_{i})$ which
is defined on the list of predicted scores $\hat{{\mathbf{y}}}_{i}$ and the
list of ground-truth relevance labels ${\mathbf{y}}_{i}$. We study the
following ranking losses:

#### Pointwise cross entropy (PointCE).

We first try the pointwise cross-entropy loss function which is calculated
based on each query-document pair independently.

$\displaystyle\ell_{\text{PointCE}}({\mathbf{y}}_{i},\hat{{\mathbf{y}}}_{i})$
$\displaystyle=$
$\displaystyle-\sum_{j|y_{ij}=1}\log(\sigma(\hat{y}_{ij}))-\sum_{j|y_{ij}=0}\log(1-\sigma(\hat{y}_{ij}))$

where $\sigma(x)=\frac{1}{1+e^{-x}}$ is the logistic function.

The loss basically converts the ranking problem into a binary classification
problem for each query-document pair. This loss is only applicable when the
relevance labels are binary.

#### Pairwise logistic (Pair).

Instead of regarding each query-document pair independently, we can also focus
on pairwise comparison. For example, we can train the model using a pairwise
logistic ranking loss Burges et al. (2005):

$\displaystyle\ell_{\text{Pair}}({\mathbf{y}}_{i},\hat{{\mathbf{y}}}_{i})=$
$\displaystyle\sum_{j=1}^{m}\sum_{j^{\prime}=1}^{m}{\mathbb{I}}_{y_{ij}>y_{ij^{\prime}}}\log\big{(}1+e^{\hat{y}_{ij^{\prime}}-\hat{y}_{ij}}\big{)}$

where the ranking problem is converted into a binary classification problem on
the order of each candidate document pair in the same list with different
relevance labels.

#### Listwise softmax cross entropy (Softmax).

We can also define a listwise softmax cross entropy loss Bruch et al. (2019),
which takes the entire list into account. It can also be considered as a
simple version of ListNet Cao et al. (2007).

$\displaystyle\ell_{\text{Softmax}}({\mathbf{y}}_{i},\hat{{\mathbf{y}}}_{i})=-\sum_{j=1}^{m}y_{ij}\log\Big{(}\frac{e^{\hat{y}_{ij}}}{\sum_{j^{\prime}}e^{\hat{y}_{ij^{\prime}}}}\Big{)}$

#### Listwise poly-1 softmax cross entropy (Poly1).

We also try a recently proposed extended version of the softmax cross-entropy
loss called PolyLoss Leng et al. (2022). The idea is to adjust the weights of
polynomial terms in the Taylor expansion of a softmax cross entropy loss. A
simplified version only adjusted the first polynomial term:

$\displaystyle\ell_{\text{Poly1}}({\mathbf{y}}_{i},\hat{{\mathbf{y}}}_{i})$
$\displaystyle=\ell_{\text{Softmax}}({\mathbf{y}}_{i},\hat{{\mathbf{y}}}_{i})$
$\displaystyle+\sum_{j=1}^{m}\epsilon\cdot
y_{ij^{\prime}}\Big{(}1-\frac{e^{\hat{y}_{ij}}}{\sum_{j^{\prime}}e^{\hat{y}_{ij^{\prime}}}}\Big{)}$

where $\epsilon$ is a parameter specified by users, representing how much
extra coefficient to be placed for the first polynomial term.

#### Discussion.

In the previous work Nogueira et al. (2020) the monoT5 ranker is still fine-
tuned by a text generation loss. The generation loss for each token is defined
on a softmax function over the entire vocabulary, and two separate target
tokens “true” and “false” are used during the fine-tuning. Hence only using
the logits of single target token “true” as ranking scores does not align with
the fine-tuning set up and cannot achieves reasonable results. If they apply
the constraints during fine-tuning that the logits for the “false” target
token are always 0 and all the other tokens are excluded, they should be able
to only use the logits of “true” as ranking scores. When there are abundant
training data, it should be practically equivalent to our RankT5 with the
encoder-decoder structure fine-tuned with the PointCE loss. With these
commonalities, we expect our RankT5-EncDec model performs no worse than monoT5
when fine-tuned with the PointCE loss.

##  5 Experiment Setup

###  5.1 Data sets

#### MS MARCO.

We use the MS MARCO passage ranking data set Bajaj et al. (2016). The data set
contains around 530,000 queries in the “train” partition and around 6,800
queries in the “dev” partition. The candidates are from a corpus with more
than 8.8 million passages. For each query, relevant passages are labeled as 1
and others are labeled as 0. We use a dual-encoder retriever Ni et al. (2022)
fine-tuned on MS MARCO to retrieve the top-1000 passages for each query in
both the “train” and “dev” partitions as the candidate documents.

Notice that in this paper, the term “document” is used interchangeably with
“passage”. We do _not_ focus on long document ranking tasks such as the MS
MARCO document ranking task.

#### Natural Questions (NQ).

We use the Natural Questions data set Kwiatkowski et al. (2019), where there
are more than 50,000 queries in the “train” partition and 8,000 in the “dev”
partition. We adopt the preprocessing setup similar to Karpukhin et al. (2020)
to construct the corpus of passages. Similar to MS MARCO, binary relevance
labels are provided. We use a dual-encoder retriever Lu et al. (2021) fine-
tuned on NQ to retrieve the top-1000 passages for each query.

#### Training data construction.

We construct the training data by first selecting a document with label 1 for
each query and then uniformly randomly sampling $(m-1)$ documents from the
top-1000 retrieved documents with labels 0. Due to the size of T5 models and
hardware constraints, the maximum $m$ we can support is 36 with the batch size
32 while fine-tuning T5-Large models. We will show how $m$ impacts the ranking
performance in our experimental results.

For models with pointwise training losses, we upsample documents with label 1
in each query to the same number as documents with label 0 in order to achieve
the optimal performance.

###  5.2 Evaluation

We evaluate the performance on the “dev” partition on both data sets. We
perform model inference on the top-1000 documents retrieved by the dual-
encoder retriever of each data set respectively. Please note that our
evaluations are applied on top 1000 documents while our training only takes a
sample of 36 documents for each query.

We evaluate the performance by Mean Reciprocal Rank (MRR@10) Voorhees et al.
(1999) and Normalized Discounted Cumulative Gain (NDCG@$k$) Järvelin and
Kekäläinen (2002) where $k=5,10$.

###  5.3 Parameter configurations

We initialize the ranking model with pretrained T5 checkpoint. If not
specified, the default size we use in our experiments is T5-Large. For the
pooling layer in RankT5-Enc, we just follow BERT Devlin et al. (2019) and take
the embedding vector of the first token. These results do not differ when
using mean pooling (Appendix B).

For the MS MARCO data set, we set the maximum input sequence length to 128 as
we did not find significant differences between using 128 and 512. Similarly,
for the NQ data set we set it to $128+256=384$ which shows performance similar
to using 512. The batch size is set to 32 for both data sets. Notice that the
batch size here specifies the number of lists, and the effective query-
document pairs in a batch is $36\times 32=1152$. We use a constant learning
rate of $1\times 10^{-4}$ during fine-tuning. For the MS MARCO data set, we
fine-tune our models for 50k steps. For the NQ data set, most of our models
are fine-tuned for 100k steps, except the ones fine-tuned with pointwise
cross-entropy loss, which achieves the best performance at 25k steps. For
Poly1 loss, we simply set $\epsilon=1$ for all of our experiments.

The implementation is based on T5X, a T5
implementation111https://github.com/google-research/t5x in JAX. All the
ranking losses are implemented in Rax, a learning-to-rank
framework222https://github.com/google/rax Jagerman et al. (2022) in JAX.

##  6 Results

Table 1:  Comparing ranking performances of different ranking models. The best
performances are bolded. Results with † are statistically significantly
($p\leq 0.05$) better than monoT5.

Model | Loss | MS MARCO | NQ
---|---|---|---
MRR@10 | NDCG@5 | NDCG@10 | MRR@10 | NDCG@5 | NDCG@10
BERT | PointCE | 0.3867 | 0.4127 | 0.4530 | 0.5157 | 0.5515 | 0.5733
BERT | Softmax | 0.3928 | 0.4173 | 0.4580 | 0.5213 | 0.5566 | 0.5791
monoT5 | Generation | 0.4156 | 0.4448 | 0.4843 | 0.5406 | 0.5861 | 0.6079
RankT5-EncDec | PointCE | 0.4209 | 0.4496 | 0.4895† | 0.5403 | 0.5833 | 0.6079
Pair | 0.4177 | 0.4456 | 0.4852 | 0.5574† | 0.5957† | 0.6198†
Softmax | 0.4278† | 0.4573† | 0.4960† | 0.5687† | 0.6068† | 0.6291†
Poly1 | 0.4343† | 0.4640† | 0.5025† | 0.5647† | 0.6032† | 0.6255†
RankT5-Enc | PointCE | 0.4216† | 0.4509† | 0.4888 | 0.5441 | 0.5851 | 0.6099
Pair | 0.4206 | 0.4513† | 0.4891† | 0.5595† | 0.5980† | 0.6215†
Softmax | 0.4305† | 0.4582† | 0.4987† | 0.5620† | 0.6018† | 0.6231†
Poly1 | 0.4296† | 0.4586† | 0.4970† | 0.5689† | 0.6068† | 0.6279†

#### Overall comparison.

We compare the performances of our proposed rankers with different model
structures and different losses. We also train BERT rankers Han et al. (2020)
on the same data sets with the PointCE loss and the Softmax loss respectively.
The BERT rankers are initialized from BERT-Large checkpoint, which has similar
capacity as T5-Large encoders. We also train the monoT5 model Nogueira et al.
(2020) on our data sets and report the performance. Results are presented in
Table 1.

It can be observed that our proposed RankT5 variants achieve substantial
improvement in ranking performance. On the MS MARCO data set, our best model
improves MRR@10, NDCG@5 and NDCG@10 by more than +1.8% compared to monoT5. On
the NQ data set, the best performance from our proposed models also
outperforms monoT5 by +2.8% in terms of MRR@10 and +2.0% in terms of NDCG@5
and NDCG@10. This result verifies that by optimizing T5 models with ranking
losses, we can significantly improve their text ranking performance.

We can also observe that some ranking losses consistently outperform others.
For example, listwise losses like Softmax and Poly1 consistently outperform
PointCE and Pair on both data sets. For the encoder-decoder model structure on
MS MARCO and the encoder-only model structure on NQ, the Poly1 loss performs
significantly ($p\leq 0.05$) better than the Softmax loss on all metrics.

We also verify that the initial checkpoint of T5 shows advantage over the
initial checkpoint of BERT. RankT5-Enc fine-tuned with the Softmax loss
outperforms the BERT ranker fine-tuned with the Softmax loss on both data sets
with a large margin (+3.7% and +4.0% respectively). The same can be observed
for both models fine-tuned with the PointCE loss, which is also reported in
Nogueira et al. (2020). This result demonstrates the importance of choosing a
good initial checkpoint and justifies the necessity to adapt T5 for ranking.

We do not find a consistent winner between the encoder-decoder (EncDec) and
the encoder-only (Enc) model structure. It seems that when using the same
ranking losses, both model structures deliver similar performances. A possible
explanation is that the T5 decoder is less important when the model is fine-
tuned for text ranking tasks with sufficient training data.

(a) MS MARCO

(b) NQ

Figure 2:  Comparing performances with different T5 model sizes. Ranking
models are RankT5-EncDec fine-tuned with the Softmax and the PointCE loss. The
performance is measured by MRR@10 with 95% confidence interval.

#### Model size comparison.

We examine how the T5 model size affects the ranking performance. We fine-tune
the RankT5-EncDec model with the Softmax and the PointCE loss with different
sizes of T5 model checkpoints, which are “Base”, “Large” and “3B”. We evaluate
the model performances on both data sets measured by MRR@10. Results are
plotted in Figure 2.

The first observation is that the performance consistently improves when we
use larger model size. On the NQ data set, the largest model (3B) outperforms
the smallest model (Base) with a nearly +7% gap. It would be worth trying even
larger language models developed recently Chowdhery et al. (2022) in a similar
way for ranking tasks.

Another observation is that the models fine-tuned with the Softmax loss
consistently outperform the ones fine-tuned with the PointCE loss (all
statistically significant with $p\leq 0.05$) and the gaps remain relatively
stable across different model sizes. This might suggest that extra signals
captured by using the appropriate ranking loss cannot be compensated by simply
using larger pretrained checkpoints.

(a) MS MARCO

(b) NQ

Figure 3:  Comparing performances with different training list sizes. Ranking
models are RankT5-EncDec fine-tuned with Softmax loss. Similar models trained
on PointCE loss are also plotted as the dashed line. The performance is
measured by MRR@10.

#### Training list size.

Another potential factor with a great impact on the ranking performance is the
training data. We investigate how the size of each list in the training data
affects the ranking performance, particularly when ranking losses like the
Softmax loss is adopted. We prepare training data with different list sizes
$m$, where each list still contains $1$ positive document and $(m-1)$ negative
documents as described in Section 5. In particular, we use $m$ from
$\\{5,10,20,30,36\\}$ where 36 is the upper limit due to the hardware
constraint. Then we fine-tune the RankT5-EncDec model with the Softmax loss
using each training list size and compare their performances. Results are
illustrated in Figure 3.

It shows that on both data sets, the model performances improve with larger
list sizes. For the NQ data set, the improvement is more significant but seems
to saturate when the list size is over 30. For the MS MARCO data set, there
also seems to be a consistent uptrend of the performance when the list size
increases. These results suggest that using a large enough list size is
crucial to obtain the optimal performance when fine-tuning with losses like
the Softmax loss. We also plot the performances of ranking models fine-tuned
with the PointCE loss for reference. Notice that the list size needs to be at
least around 20 to 30 for the Softmax loss to beat the PointCE loss.

Table 2: Zero-shot performance comparison. Ranking models are RankT5-Enc fine-tuned on the MS MARCO data set with different losses. Performances are measured by NDCG@10. The best performance for each data set is bolded. Data set | PointCE | Softmax
---|---|---
TREC-COVID | 0.7522 | 0.8071
BioASQ | 0.5346 | 0.5635
NFCorpus | 0.3263 | 0.3810
NQ | 0.5959 | 0.6142
HotpotQA | 0.7126 | 0.7100
FiQA-2018 | 0.4156 | 0.4450
Signal-1M | 0.3153 | 0.3200
ArguAna | 0.2232 | 0.3300
Touché-2020 | 0.4594 | 0.4401
Quora | 0.8221 | 0.8309
DBPedia | 0.4345 | 0.4422
SCIDOCS | 0.1821 | 0.1806
FEVER | 0.8352 | 0.8316
Climate-FEVER | 0.2062 | 0.2152
SciFact | 0.7493 | 0.7499
Average | 0.5024 | 0.5241

#### Zero-shot results.

We also compare the zero-shot performances of our ranking models fine-tuned
with different ranking losses. We use a subset of BEIR Thakur et al. (2021)
data sets333Notice that the NQ data set in BEIR is has a different corpus and
query set from the NQ data set we used earlier. with easily accessible corpus.
We take our RankT5-Enc ranking models fine-tuned on the MS MARCO data set with
the PointCE loss and the Softmax loss respectively, and apply these models to
rerank top-1000 documents returned by BM25 Robertson and Zaragoza (2009).
Table 2 summarizes ranking performances measured by NDCG@10.

The ranking model fine-tuned with the Softmax loss outperforms the PointCE
loss on 11 out of the 15 data sets. On average, the Softmax loss achieves more
than +2.1% NDCG@10 (statistically significant with $p\leq 0.05$). These
results indicate that using the Softmax loss produces ranking models that
generalize better to out-of-domain data.

In particular, using the Softmax loss achieves larger improvement on data sets
with drastically different corpus. For example, TREC-COVID, BioASQ and
NFCorpus are all in the biomedical domain with a very different vocabulary
from the web corpus of MS MARCO. The average improvement on these corpora
reaches +4.6%. This might imply that fine-tuning the model with appropriate
ranking losses can enforce the model to put less emphasis on memorization, and
thus to better learn the abstract concept of “relevance” Lafferty and Zhai
(2003), regardless of what the underlying corpus is.

##  7 Conclusion

In this paper, we study to utilize pretrained T5 models on text ranking.
Unlike monoT5, which simply converts a text ranking problem into a
T5-compatible token generation task, our paper presents two model variants
that output numerical numbers based on the T5 architecture: an encoder-decoder
model and an encoder-only model. We thereby propose fine-tuning T5 with
ranking losses to maximize ranking metrics. We demonstrate significant
improvement when ranking losses are used to fine-tune the T5 model on the MS
MARCO and the NQ data sets. We further illustrate that such improvement can be
maintained in the zero-shot setting on out-of-domain data sets.

Our paper opens up several potential directions for further exploration.
First, more T5-based ranking model architectures can be studied. For example,
one can study different pooling strategies in the encoder-only model
structure. Second, it is unclear whether there is any difference between the
encoder-decoder and the encoder-only models when the training data is
insufficient. Third, it might be interesting to extend our training method
into the pretraining stage to produce better pretrained models.

## References

  * Bajaj et al. (2016) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016.  MS MARCO: A human generated machine reading comprehension dataset.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Bruch et al. (2019) Sebastian Bruch, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2019.  An analysis of the softmax cross entropy loss for learning-to-rank with binary relevance.  In _Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval_ , ICTIR ’19, page 75–78.
  * Burges et al. (2005) Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005.  Learning to rank using gradient descent.  In _Proceedings of the 22nd international conference on Machine learning_ , pages 89–96.
  * Cao et al. (2007) Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007.  Learning to rank: From pairwise approach to listwise approach.  In _Proceedings of the 24th International Conference on Machine Learning_ , page 129–136.
  * Chapelle and Chang (2011) Olivier Chapelle and Yi Chang. 2011.  Yahoo! learning to rank challenge overview.  In _Proceedings of the learning to rank challenge_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  PaLM: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _NAACL-HLT (1)_.
  * dos Santos et al. (2020) Cicero dos Santos, Xiaofei Ma, Ramesh Nallapati, Zhiheng Huang, and Bing Xiang. 2020\.  Beyond [CLS] through Ranking by Generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1722–1727.
  * Gao et al. (2020) Luyu Gao, Zhuyun Dai, and Jamie Callan. 2020.  Modularized transfomer-based ranking framework.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4180–4190.
  * Gao et al. (2021) Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021.  Rethink training of BERT rerankers in multi-stage retrieval pipeline.  In _Advances in Information Retrieval–43rd European Conference on IR Research_.
  * Han et al. (2020) Shuguang Han, Xuanhui Wang, Mike Bendersky, and Marc Najork. 2020.  Learning-to-Rank with BERT in TF-Ranking.  _CoRR_ , abs/2004.08476.
  * Hui et al. (2022) Kai Hui, Honglei Zhuang, Tao Chen, Zhen Qin, Jing Lu, Dara Bahri, Ji Ma, Jai Prakash Gupta, Cicero Nogueira dos Santos, Yi Tay, and Don Metzler. 2022.  ED2LM: Encoder-decoder to language model for faster document re-ranking inference.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3747–3758.
  * Jagerman et al. (2022) Rolf Jagerman, Xuanhui Wang, Honglei Zhuang, Zhen Qin, Michael Bendersky, and Marc Najork. 2022.  Rax: Composable learning-to-rank using JAX.  In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22)_.
  * Järvelin and Kekäläinen (2002) Kalervo Järvelin and Jaana Kekäläinen. 2002.  Cumulated gain-based evaluation of IR techniques.  _ACM Transactions on Information Systems (TOIS)_ , 20(4):422–446.
  * Ju et al. (2021) Jia-Huei Ju, Jheng-Hong Yang, and Chuan-Ju Wang. 2021.  Text-to-text multi-view learning for passage re-ranking.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1803–1807.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense Passage Retrieval for Open-Domain Question Answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  Col-BERT: Efficient and Effective Passage Search via Contextualized late Interaction over BERT.  In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_ , pages 39–48.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association of Computational Linguistics_.
  * Lafferty and Zhai (2003) John Lafferty and Chengxiang Zhai. 2003.  Probabilistic relevance models based on document and query generation.  In _Language modeling for information retrieval_ , pages 1–10. Springer.
  * Leng et al. (2022) Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Jay Shi, Shuyang Cheng, and Dragomir Anguelov. 2022.  Polyloss: A polynomial expansion perspective of classification loss functions.  In _International Conference on Learning Representations_.
  * Lin et al. (2020) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2020.  Pretrained transformers for text ranking: BERT and beyond.  _arXiv preprint arXiv:2010.06467_.
  * Liu (2009) Tie-Yan Liu. 2009.  _Learning to Rank for Information Retrieval_.  Now Publishers Inc.
  * Lu et al. (2021) Jing Lu, Gustavo Hernández Ábrego, Ji Ma, Jianmo Ni, and Yinfei Yang. 2021\.  Multi-stage training with improved negative contrast for neural passage retrieval.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6091–6103.
  * MacAvaney et al. (2020) Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020.  Efficient document re-ranking for transformers by precomputing term representations.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 49–58.
  * Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Wei Chang, et al. 2022.  Large dual encoders are generalizable retrievers.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020.  Document Ranking with a Pretrained Sequence-to-Sequence Model.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 708–718.
  * Nogueira et al. (2019) Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019.  Multi-stage document ranking with BERT.  _CoRR_ , abs/1910.14424.
  * Pradeep et al. (2021) Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021.  The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models.  _arXiv preprint arXiv:2101.05667_.
  * Qin and Liu (2013) Tao Qin and Tie-Yan Liu. 2013.  Introducing LETOR 4.0 datasets.  _CoRR_ , abs/1306.2597.
  * Qin et al. (2021) Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2021.  Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees?  In _International Conference on Learning Representations_.
  * Qu et al. (2021) Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021.  RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5835–5847.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:1–67.
  * Ren et al. (2021) Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021.  RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2825–2835.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  _The probabilistic relevance framework: BM25 and beyond_.  Now Publishers Inc.
  * Rosa et al. (2022) Guilherme Moraes Rosa, Luiz Bonifacio, Vitor Jeronymo, Hugo Abonizio, Marzieh Fadaee, Roberto Lotufo, and Rodrigo Nogueira. 2022.  No parameter left behind: How distillation and model size affect zero-shot retrieval.  _arXiv preprint arXiv:2206.02873_.
  * Sachan et al. (2022) Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022.  Improving passage retrieval with zero-shot question generation.  _arXiv preprint arXiv:2204.07496_.
  * Tay et al. (2022) Yi Tay, Vinh Q Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. 2022.  Transformer memory as a differentiable search index.  _arXiv preprint arXiv:2202.06991_.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Voorhees et al. (1999) Ellen M Voorhees et al. 1999.  The TREC-8 Question Answering Track Report.  In _Proceedings of the 8th Text REtrieval Conference (TREC-8),_ , volume 99, pages 77–82.
  * Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N Bennett, Junaid Ahmed, and Arnold Overwijk. 2020.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  In _International Conference on Learning Representations_.
  * Zhuang et al. (2021) Shengyao Zhuang, Hang Li, and Guido Zuccon. 2021.  Deep Query Likelihood Model for Information Retrieval.  In _The 43rd European Conference On Information Retrieval (ECIR)_.
  * Zhuang and Zuccon (2021) Shengyao Zhuang and Guido Zuccon. 2021.  TILDE: Term Independent Likelihood moDEl for Passage Re-ranking.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1483–1492.

##  Appendix A Choice of Target Tokens and Input Sequence

In Section 4.1, our proposed model structure with the encoder-decoder
structure adopts “<extra_id_10>” as the target token and uses its unnormalized
logit as the ranking score. We believe that the choice of target token does
not affect the model performance when we have sufficient training data. We
conduct experiments where we use “<extra_id_11>” or “true” as the target
token. We also try to use the original input sequence format in Nogueira et
al. (2020) where the input sequence contains the “Relevant:” postfix:

$\displaystyle
s_{ij}=\text{Query:}~{}q_{i}~{}\text{Document:}~{}d_{ij}~{}\text{Relevant:}$

and the target token is “true”. We fine-tune our EncDec model on the MS MARCO
data set with the Softmax loss using different target tokens or input sequence
formats. Results are shown in Table 3.

Table 3: Comparison of different target token choices and input sequence
formats. All models are RankT5-EncDec model fine-tuned with the Softmax loss.
The performances are reported on the MS MARCO data set.

Input sequence | Target token | MRR@10 | NDCG@10
---|---|---|---
w/o “Relevant:” | <extra_id_10> | 0.4277 | 0.4960
<extra_id_11> | 0.4272 | 0.4951
true | 0.4250 | 0.4937
w/ “Relevant:” | true | 0.4279 | 0.4953

It can be observed that the model performances are not sensitive to the choice
of target tokens. Whether or not to add the “Relevant:” postfix to the input
sequence also does not seem to show any effect. These results seem to align
with the observation in Nogueira et al. (2020) that the choice of target
tokens does not change the performance when there is sufficient training data.

##  Appendix B Pooling Strategies Comparison for RankT5-Enc

In Section 4.1, our proposed model structure with the encoder-only structure
uses a pooling layer. There could be different ways to instantiate the pooling
layers. The first one is to only take the embedding vector of the first token:

$\displaystyle\text{Pool}_{\text{First}}([{\mathbf{e}}_{1},\cdots,{\mathbf{e}}_{l}])={\mathbf{e}}_{1}$
(5)

Another common strategy is to take the mean of the embedding vectors of all
the tokens:

$\displaystyle\text{Pool}_{\text{Mean}}([{\mathbf{e}}_{1},\cdots,{\mathbf{e}}_{l}])=\frac{1}{l}\sum_{i=1}^{l}{\mathbf{e}}_{i}$
(6)

We conduct experiments comparing these pooling strategies on MS MARCO, where
the RankT5-Enc model is fine-tuned with the Softmax loss. Results are
summarized in Table 4.

Table 4: Comparison of different pooling strategies. All models are RankT5-Enc model fine-tuned with the Softmax loss. The performances are reported on the MS MARCO data set.  Pooling | MRR@10 | NDCG@10
---|---|---
First | 0.4305 | 0.4987
Mean | 0.4308 | 0.4993

These results suggest that different pooling strategies do not generate much
different results. This is probably because even just the first token is
already attended by all the other token with the transformer structure.
Moreover, in our experiments, there are sufficient training data to fine-tune
the model.

Table 5:  Comparing ranking performances of different ranking models with
additional metrics. The best performances are bolded. Results with † are
statistically significantly ($p\leq 0.05$) better than monoT5.

Model | Loss | MS MARCO | NQ
---|---|---|---
NDCG | MAP | Recall@5 | NDCG | MAP | Recall@5
BERT | PointCE | 0.5180 | 0.3932 | 0.5528 | 0.6164 | 0.5228 | 0.6849
BERT | Softmax | 0.5212 | 0.3978 | 0.5548 | 0.6215 | 0.5276 | 0.6895
monoT5 | Generation | 0.5421 | 0.4206 | 0.5945 | 0.6378 | 0.5434 | 0.7334
RankT5-EncDec | PointCE | 0.5462† | 0.4260† | 0.5973 | 0.6406 | 0.5457† | 0.7417†
Pair | 0.5438 | 0.4229 | 0.5911 | 0.6531† | 0.5629† | 0.7392
Softmax | 0.5520† | 0.4326† | 0.6059† | 0.6616† | 0.5740† | 0.7480†
Poly1 | 0.5569† | 0.4383† | 0.6143† | 0.6586† | 0.5701† | 0.7458†
RankT5-Enc | PointCE | 0.5467† | 0.4269† | 0.5973 | 0.6431 | 0.5496† | 0.7380
Pair | 0.5464† | 0.4256 | 0.6029† | 0.6547† | 0.5650† | 0.7418†
Softmax | 0.5539† | 0.4347† | 0.6052† | 0.6564† | 0.5674† | 0.7467†
Poly1 | 0.5536† | 0.4344† | 0.6059† | 0.6615† | 0.5744† | 0.7457†

##  Appendix C Ranking Performances with Additional Metrics

We provide some additional metrics to compare the performance of our RankT5
models with other baselines. The metrics we reported in Table 1 focus more
precision, which emphasizes ensuring the top positions of ranked lists are
relevant documents. In this appendix we also report metrics like NDCG without
cutoff or mean average precision (MAP), which would also penalize models if
any relevant document is ranked too low in the list. Specifically, we report
model performance measured by NDCG, MAP and Recall@5 in Table 5.

The conclusion is similar to Table 1. Our proposed RankT5 models still
outperform other baselines significantly on these metrics. Models fine-tuned
with Softmax or Poly1 still achieve better performance than models fine-tuned
with pointwise or pairwise losses. These results are not surprising as both
data sets have very few documents labeled for each query. Actually, for both
data sets, more than 94% of queries in the dev partition have only 1 document
labeled as relevant.

##  Appendix D Additional Experiments on Model Size Comparison

We report additional experiments of model size comparisons on RankT5-Enc
models. Similar to the experiments in Section 6, we fine-tune the RankT5-Enc
model with the Softmax and the PointCE loss with T5 model sizes of “Base”,
“Large” and “3B” and plot their MRR@10. Results are plotted in Figure 4.

(a) MS MARCO

(b) NQ

Figure 4:  Comparing performances with different T5 model sizes. Ranking
models are RankT5-Enc fine-tuned with the Softmax and the PointCE loss. The
performance is measured by MRR@10 with 95% confidence interval.

The results are similar to RankT5-EncDec results. The performance increases
with larger model sizes and the gap between the Softmax loss and the PointCE
loss remains similar.

##  Appendix E Additional Zero-Shot Experiments

We also conduct the zero-shot experiments using RankT5-Enc fine-tuned with
different model sizes. All the models are fine-tuned using the Softmax loss.
We compare the models with size “Base”, “Large” and “3B”. The results are
shown in Table 6.

Table 6: Zero-shot performance comparison. Ranking models are RankT5-Enc fine-tuned on the MS MARCO data set using the Softmax loss with different model sizes. Performances are measured by NDCG@10. Data set | Base | Large | 3B
---|---|---|---
TREC-COVID | 0.7896 | 0.8071 | 0.8237
BioASQ | 0.5627 | 0.5635 | 0.5795
NFCorpus | 0.3731 | 0.3810 | 0.3990
NQ | 0.5731 | 0.6142 | 0.6471
HotpotQA | 0.7269 | 0.7100 | 0.7536
FiQA-2018 | 0.4132 | 0.4450 | 0.4932
Signal-1M | 0.2909 | 0.3200 | 0.3191
ArguAna | 0.3094 | 0.3300 | 0.4069
Touché-2020 | 0.4449 | 0.4401 | 0.4869
Quora | 0.8138 | 0.8309 | 0.8192
DBPedia | 0.4373 | 0.4422 | 0.4598
SCIDOCS | 0.1760 | 0.1806 | 0.1918
FEVER | 0.8295 | 0.8316 | 0.8487
Climate-FEVER | 0.2462 | 0.2152 | 0.2753
SciFact | 0.7493 | 0.7499 | 0.7600
Average | 0.5157 | 0.5241 | 0.5509

It can be observed that on average the ranking model with larger size can also
perform better on out-of-domain data sets in the zero-shot setting.

We also compare the zero-shot performance of our RankT5-Enc model with
publicly reported zero-shot performance of monoT5 from Rosa et al. (2022). We
only compare the performance with model size Base and 3B since these are the
only sizes we have results from both models. The results can be found in Table
7.

Table 7: Comparing zero-shot performance of RankT5-Enc and monoT5. RankT5-Enc is fine-tuned on the MS MARCO data set using the Softmax loss. Performances are measured by NDCG@10. Data set | Base | 3B
---|---|---
monoT5 | RankT5 | monoT5 | RankT5
TREC-COVID | 0.7775 | 0.7896 | 0.7948 | 0.8237
BioASQ | 0.5240 | 0.5627 | 0.5740 | 0.5795
NFCorpus | 0.3570 | 0.3731 | 0.3837 | 0.3990
NQ | 0.5674 | 0.5731 | 0.6334 | 0.6471
HotpotQA | 0.6950 | 0.7269 | 0.7589 | 0.7536
FiQA-2018 | 0.4136 | 0.4132 | 0.5137 | 0.4932
Signal-1M | 0.2771 | 0.2909 | 0.3140 | 0.3191
ArguAna | 0.1321 | 0.3094 | 0.2876 | 0.4069
Touché-2020 | 0.2773 | 0.4449 | 0.2995 | 0.4869
Quora | 0.8230 | 0.8138 | 0.8407 | 0.8192
DBPedia | 0.4195 | 0.4373 | 0.4777 | 0.4598
SCIDOCS | 0.1649 | 0.1760 | 0.1970 | 0.1918
FEVER | 0.8018 | 0.8295 | 0.8495 | 0.8487
Climate-FEVER | 0.2451 | 0.2462 | 0.2802 | 0.2753
SciFact | 0.7356 | 0.7493 | 0.7773 | 0.7600
Average | 0.4807 | 0.5157 | 0.5321 | 0.5509

Results show that RankT5 fine-tuned with Softmax on average outperforms the
monoT5 with both model sizes. It again provides evidence that fine-tuning
ranking models with listwise ranking losses can help the model generalize
better to out-of-domain data, similar to our observation in Table 2.

##  Appendix F Limitations

One limitation of this work is that both data sets we used only have binary
relevance labels. There are other text ranking data sets with graded relevance
labels (e.g., 0 to 4), but they either are not publicly available or do not
have a large enough training set. Different fine-tuning strategies (training
data construction, ranking losses, learning rate etc.) might need to be
developed for our proposed models to achieve optimal performances on such data
sets.

Another limitation is that we only focus on T5. There are other sequence-to-
sequence language models like GPT-2 with different model structures (where the
encoder is not available). How to apply techniques proposed in this work to
those models remains to be further discussed.


# A GENERAL-PURPOSE DEEP LEARNING APPROACH TO MODEL TIME-VARYING AUDIO EFFECTS

###### Abstract

Audio processors whose parameters are modified periodically over time are
often referred as time-varying or modulation based audio effects. Most
existing methods for modeling these type of effect units are often optimized
to a very specific circuit and cannot be efficiently generalized to other
time-varying effects. Based on convolutional and recurrent neural networks, we
propose a deep learning architecture for generic black-box modeling of audio
processors with long-term memory. We explore the capabilities of deep neural
networks to learn such long temporal dependencies and we show the network
modeling various linear and nonlinear, time-varying and time-invariant audio
effects. In order to measure the performance of the model, we propose an
objective metric based on the psychoacoustics of modulation frequency
perception. We also analyze what the model is actually learning and how the
given task is accomplished.

##  1 Introduction

Modulation based or time-varying audio effects involve audio processors or
effect units that include a modulator signal within their analog or digital
implementation [1]. These modulator signals are in the low frequency range
(usually below $20$ Hz). Their waveforms are based on common periodic signals
such as sinusoidal, squarewave or sawtooth oscillators and are often referred
to as a Low Frequency Oscillator (LFO). The LFO periodically modulates certain
parameters of the audio processors to alter the timbre, frequency, loudness or
spatialization characteristics of the audio. This differs from time-invariant
audio effects which do not change their behavior over time. Based on how the
LFO is employed and the underlying signal processing techniques used when
designing the effect units, we can classify modulation based audio effects
into time-varying filters such as phaser or wah-wah; delay-line based effects
such as flanger or chorus; and amplitude modulation effects such as tremolo or
ring modulator [2].

The phaser effect is a type of time-varying filter implemented through a
cascade of notch or all-pass filters. The characteristic sweeping sound of
this effect is obtained by modulating the center frequency of the filters,
which creates phase cancellations or enhancements when combining the filter’s
output with the input audio. Similarly, the wah-wah is based on a bandpass
filter with a variable center frequency, usually controlled by a pedal. If the
center frequency is modulated by an LFO or an envelope follower, the effect is
commonly called auto-wah.

Delay-line based audio effects, as in the case of flanger and chorus, are
based on the modulation of the length of the delay lines. A flanger is
implemented via a modulated comb filter whose output is mixed with the input
audio. Unlike the phaser, the notch and peak frequencies caused by the
flanger’s sweep comb filter effect are equally spaced in the spectrum, thus
causing the known metallic sound associated with this effect. A chorus occurs
when mixing the input audio with delayed and pitch modulated copies of the
original signal. This is similar to various musical sources playing the same
instrument but slightly shifted in time. vibrato is digitally implemented as a
delay-line based audio effect, where pitch shifting is achieved when
periodically varying the delay time of the input waveform [3].

Tremolo is an amplitude modulation effect where an LFO is used to directly
vary the amplitude of the incoming audio, creating in this way a perceptual
temporal fluctuation. A ring modulator is also based on amplitude modulation,
but the modulation is achieved by having the input audio multiplied by a
sinusoidal oscillator with higher carrier frequencies. In the analog domain,
this effect is commonly implemented with a diode bridge, which adds a
nonlinear behavior and a distinct sound to this effect unit. Another type of
modulation based effect that combines amplitude, pitch and spatial modulation
is the Leslie speaker, which is implemented by a rotating horn and a rotating
woofer inside a wooden cabinet. This effect can be interpreted as a
combination of tremolo, Doppler effect and reverberation [4].

Most of these effects can be implemented directly in the digital domain
through the use of digital filters and delay lines. Nevertheless, modeling
specific effect units or analog circuits has been heavily researched and
remains an active field. This is because hardware effect units are
characterized by the nonlinearities introduced by certain circuit components.
Musicians often prefer the analog counterparts because the digital
implementations may lack this behavior, or because the digital simulations
make certain assumptions when modeling specific nonlinearities.

Virtual analog methods for modeling such effect units mainly involve circuit
modeling and optimization for specific analog components such as operational
amplifiers or transistors. This often requires assumptions or models that are
too specific for a certain circuit. Such models are also not easily
transferable to different effects units since expert knowledge of the type of
circuit being modeled is required, i.e. specific linear and nonlinear
components.

Prior to this work, deep learning architectures have not yet been implemented
to model time-varying audio effects. Thus, building on [5, 6], we propose a
general-purpose deep learning approach to model this type of audio effects.
Using convolutional, recurrent and fully connected layers, we explore how a
deep neural network (DNN) can learn the long temporal dependencies which
characterizes these effect units as well as the possibilities to match
nonlinearities within the audio effects. We include Bidirectional Long Short-
Term Memory (Bi-LSTM) neural networks and explore their capabilities when
learning time-varying transformations. We explore linear and nonlinear time-
varying emulation as a content-based transformation without explicitly
obtaining the solution of the time-varying system.

We show the model matching modulation based audio effects such as chorus,
flanger, phaser, tremolo, vibrato, tremolo-wah, ring modulator and Leslie
speaker. We investigate the capabilities of the model when adding further
nonlinearities to the linear time-varying audio effects. Furthermore, we
extend the applications of the model by including nonlinear time-invariant
audio effects with long temporal dependencies such as auto-wah, compressor and
multiband compressor. Finally, we measure performance of the model using a
metric based on the modulation spectrum.

The paper is structured as follows. In Section 2 we present the relevant
literature related to virtual analog of modulation based audio effects.
Section 3 gives details of our model, the modulation based effect tasks and
the proposed evaluation metric. Sections 4 and 5 show the analysis, obtained
results, and the respective conclusion.

##  2 Background

###  2.1 Virtual analog modeling of time-varying audio effects

Virtual analog audio effects aim to simulate an effect unit and recreate the
sound of an analog reference circuit. Much of the active research models
nonlinear audio processors such as distortion effects, compressors, amplifiers
or vacuum tubes [7, 8, 9]. With respect to modeling time-varying audio
effects, most of the research has been applied to develop white-box methods,
i.e. in order to model the effect unit a complete study of the internal
circuit is carried out. These methods use circuit simulation techniques to
characterize various analog components such as diodes, transistors,
operational amplifiers or integrated circuits.

In [10], phasers implemented via Junction Field Effect Transistors (JFET) and
Operational Transconductance Amplifiers (OTA) were modeled using circuit
simulation techniques that discretize the differential equations that describe
these components. Using a similar circuit modeling procedure, delay-line based
effects are modeled, such as flanger and chorus as implemented with Bucket
Brigade Delay (BBD) chips. BBD circuits have been widely used in analog delay-
line based effect units and several digital emulations have been investigated.
[11] emulated BBD devices through circuit analysis and electrical measurements
of the linear and nonlinear elements of the integrated circuit. [12] modeled
BBDs as delay-lines with fixed length but variable sample rate.

Based on BBD circuitry, a flanger effect was modeled in [13] via the nodal DK-
method. This is a common method in virtual analog modeling [14] where
nonlinear filters are derived from the differential equations that describe an
electrical circuit. In [15], a wah-wah pedal is implemented using the nodal
DK-method and the method is extended to model the temporal fluctuations
introduced by the continuous change of the pedal. In [16], the MXR Phase 90
phaser effect is modeled via a thorough circuit analysis and the DK-method.
This effect unit is based on JFETs, and voltage and current measurements were
performed to obtain the nonlinear characteristics of the transistors.

Amplitude modulation effects such as an analog ring modulator were modeled in
[17], where the diode bridge is emulated as a network of static
nonlinearities. [18] modeled the rotating horn of the Leslie speaker via
varying delay-lines, artificial reverberation and physical measurements from
the rotating loudspeaker. [19] also modeled the Leslie speaker and achieved
frequency modulation through time-varying spectral delay filters and amplitude
modulation using a modulator signal. In both Leslie speaker emulations,
various physical characteristics of the effect are not taken into account,
such as the frequency-dependent directivity of the loudspeaker and the effect
of the wooden cabinet.

In [20], gray-box modeling was proposed for linear time-varying audio effects.
This differs from white-box modeling, since the method was based on input-
output measurements but the time-varying filters were based on knowledge of
analog phasers. In this way, phaser emulation was achieved by multiple
measurements of the impulse response of a cascade of all-pass filters.

Another method to model time-varying audio effects is discretizing electrical
circuit elements via Wave Digital Filters (WDF). The Hammond organ
vibrato/chorus was modeled using WDFs in [21], and [22] performed circuit
modeling through WDFs to emulate modulation based effects that use OTAs.

Figure 1: Block diagram of the proposed model; adaptive front-end, Bi-LSTM and
synthesis back-end.

###  2.2 End-to-end deep neural networks

End-to-end deep learning is based on the idea that an entire problem can be
taken as a single indivisible task which must be learned from input to output.
Deep learning architectures using this principle have recently been researched
in the music information retrieval field [23, 24, 25], since the amount of
required prior knowledge may be reduced and engineering effort minimized by
learning directly from raw audio [26]. Recent work also demonstrated the
feasibility of these architectures for audio synthesis and audio effects
modeling. [27, 28] proposed models that synthesize audio waveforms and [29]
obtained a model capable of performing singing voice synthesis.

End-to-end deep neural networks for audio effects modeling were implemented in
[5], where Equalization (EQ) matching was achieved with convolutional neural
networks (CNN). Also, [6] presented a deep learning architecture for modeling
nonlinear processors such as distortion, overdrive and amplifier emulation.
The DNN is capable of modeling an arbitrary combination of linear and
nonlinear memoryless audio effects, but does not generalize to transformations
with long temporal dependencies such as modulation based audio effects.

##  3 Methods

###  3.1 Model

The model is entirely based on the time-domain and operates with raw audio as
the input and processed audio as the output. It is divided into three parts:
adaptive front-end, latent-space and synthesis back-end. A block diagram can
be seen in Fig. 1 and its structure is described in detail in Table 1. We
build on the architecture from [6], since we incorporate Bi-LSTMs into the
latent-space and we modify the structure of the synthesis back-end in order to
allow the model to learn nonlinear time-varying transformations.

###  3.2 Adaptive front-end

The front-end performs time-domain convolutions with the incoming audio. It
follows a filter bank architecture and is designed to learn a latent
representation for each audio effect modeling task. It consists of a
convolutional encoder which contains two CNN layers, one pooling layer and one
residual connection. This residual connection is used by the back-end to
facilitate the synthesis of the waveform based on the specific time-varying
transformation.

In order to allow the model to learn long-term memory dependencies, the input
consists of the current audio frame $x(t)$ concatenated with the $k$ previous
and $k$ subsequent frames. These frames are of size $N$ and sampled with a hop
size $\tau$. The input $\boldsymbol{x}$ is described by (1).

Table 1: Detailed architecture of a model with input frame size of $4096$
samples and $\pm 4$ context frames.

Layer | Output shape | Units | Output
---|---|---|---
Input | (9, 4096, 1) | . | $\boldsymbol{x}$
Conv1D | (9, 4096, 32) | 32(64) | $X_{1}$
Residual | (4096, 32) | . | $R$
Abs | (9, 4096, 32) | . | .
Conv1D-Local | (9, 4096, 32) | 32(128) | .
Softplus | (9, 4096, 32) | . | $X_{2}$
MaxPooling | (9, 64, 32) | . | $Z$
Bi-LSTM | (64, 128) | 64 | .
Bi-LSTM | (64, 64) | 32 | .
Bi-LSTM | (64, 32) | 16 | .
SAAF | (64, 32) | 25 | $\hat{Z}$
Unpooling | (4096, 32) | . | $\hat{X}_{3}$
Multiply | (4096, 32) | . | $\hat{X}_{2}$
Dense | (4096, 32) | 32 | .
Dense | (4096, 16) | 16 | .
Dense | (4096, 16) | 16 | .
Dense | (4096, 32) | 32 | .
SAAF | (4096, 32) | 25 | .
Abs | (4096, 32) | . | .
Global Average | (1, 32) | . | .
Dense | (1, 512) | 512 | .
Dense | (1, 32) | 32 | .
Multiply | (4096, 32) | . | $\hat{X}_{1}$
Add | (4096, 32) | . | $\hat{X}_{0}$
deConv1D | (4096, 1) | . | $\hat{y}$

$\boldsymbol{x}=x(t+j\tau),j=-k,...,k$ (1)

The first convolutional layer has $32$ one-dimensional filters of size $64$
and is followed by the absolute value as nonlinear activation function. The
operation performed by the first layer can be described by (2).

$\boldsymbol{X}_{1}=\boldsymbol{x}*\boldsymbol{W}_{1}$ (2)

Where $\boldsymbol{X}_{1}$ is the feature map after the input audio
$\boldsymbol{x}$ is convolved with the kernel matrix $\boldsymbol{W}_{1}$.
$\boldsymbol{R}$ is the corresponding row in $\boldsymbol{X}_{1}$ for the
frequency band decomposition of the current input frame $x(t)$. The back-end
does not directly receive information from the past and subsequent context
frames. The second layer has $32$ filters of size $128$ and each filter is
locally connected. We follow a filter bank architecture since each filter is
only applied to its corresponding row in $|\boldsymbol{X}_{1}|$ and so we
significantly decrease the number of trainable parameters. This layer is
followed by the softplus nonlinearity [30], described by (3).

$\boldsymbol{X}_{2}=\textit{softplus}(|\boldsymbol{X}_{1}|*\boldsymbol{W}_{2})$
(3)

Where $\boldsymbol{X}_{2}$ is the second feature map obtained after the local
convolution with $\boldsymbol{W}_{2}$, the kernel matrix of the second layer.
The max-pooling operation is a moving window of size $N/64$ applied over
$\boldsymbol{X}_{2}$, where the maximum value within each window corresponds
to the output.

By using the absolute value as activation function of the first layer and by
having larger filters $\boldsymbol{W}_{2}$, we expect the front-end to learn
smoother representations of the incoming audio, such as envelopes [24]. All
convolutions and pooling operations are time distributed, i.e the same
convolution or pooling operation is applied to each of the $2\cdot k+1$ input
frames.

###  3.3 Bidirectional LSTMs

The latent-space consists of three Bi-LSTM layers of $64$, $32$, and $16$
units respectively. Bi-LSTMs are a type of recurrent neural network that can
access long-term context from both backward and forward directions [31]. Bi-
LSTMs are capable of learning long temporal dependencies when processing
timeseries where the context of the input is needed [32].

The Bi-LSTMs process the latent-space representation $Z$. $Z$ is learned by
the front-end and contains information regarding the $2\cdot k+1$ input
frames. These recurrent layers are trained to reduce the dimension of $Z$,
while also learning a nonlinear modulation $\hat{Z}$. This new latent
representation is fed into the synthesis back-end in order to reconstruct an
audio signal that matches the time-varying task. Each Bi-LSTM has dropout and
recurrent dropout rates of $0.1$ and the first two layers have the hyperbolic
tangent as activation function.

The performance of CNNs in regression tasks has improved by using adaptive
activation functions [33]. So we add a Smooth Adaptive Activation Function
(SAAF) as the nonlinearity for the last layer. SAAFs consist of piecewise
second order polynomials which can approximate any continuous function and are
regularized under a Lipschitz constant to ensure smoothness. As shown in [6],
SAAFs can be used within deep neural networks to model nonlinearities in audio
processing tasks.

###  3.4 Synthesis back-end

The synthesis back-end accomplishes the reconstruction of the target audio by
processing the current input frame $x(t)$ and the nonlinear modulation
$\hat{Z}$. The back-end consists of an unpooling layer, a DNN block with SAAF
and Squeeze-and-Excitation (SE) [34] layers (DNN-SAAF-SE) and a final CNN
layer.

The DNN-SAAF-SE block consists of $4$ fully connected (FC) layers of $32$,
$16$, $16$ and $32$ hidden units respectively. Each FC layer is followed by
the hiperbolic tangent function except for the last one, which is followed by
a SAAF layer. Overall, each SAAF is locally connected and each function
consists of $25$ intervals between $-1$ to $1$.

The SE blocks explicitly model interdependencies between channels by
adaptively scaling the channel-wise information of feature maps [34]. The SE
dynamically scales each of the 32 channels and follows the structure from
[35]. It consists of a global average pooling operation followed by two FC
layers of $512$ and $32$ hidden units respectively. The FC layers are followed
by a rectifier linear unit (ReLU) and sigmoid activation functions
accordingly. Since the feature maps of the model are based on time-domain
waveforms, we incorporate an absolute value layer before the global average
pooling operation.

The back-end matches the time-varying task by the following steps. First, a
discrete approximation of $\boldsymbol{Z}$ ($\boldsymbol{\hat{X}}_{3}$) is
obtained by an upsampling operation. Then the feature map
$\boldsymbol{\hat{X}}_{2}$ is the result the element-wise multiplication of
the residual connection $\boldsymbol{R}$ and $\boldsymbol{\hat{X}}_{3}$. This
can be seen as a frequency dependent amplitude modulation between the learned
modulator $\boldsymbol{Z}$ and the frequency band decomposition
$\boldsymbol{R}$.

$\boldsymbol{\hat{X}_{2}}=\boldsymbol{\hat{X}}_{3}\cdot\boldsymbol{R}$ (4)

The feature map $\boldsymbol{\hat{X}}_{1}$ is obtained when the nonlinear and
channel-wise scaled filters from the DNN-SAAF-SE block are applied to the
modulated frequency band decomposition $\boldsymbol{\hat{X}}_{2}$. Then,
$\boldsymbol{\hat{X}}_{1}$ is added back to $\boldsymbol{\hat{X}}_{2}$, acting
as a nonlinear delay-line.

$\boldsymbol{\hat{X}_{0}}=\boldsymbol{\hat{X}}_{1}+\boldsymbol{\hat{X}}_{2}$
(5)

The last layer corresponds to the deconvolution operation, which can be
implemented by transposing the first layer transform. This layer is not
trainable since its kernels are transposed versions of $\boldsymbol{W}_{1}$.
In this way, the back-end reconstructs the audio waveform in the same manner
that the front-end decomposed it. The complete waveform is synthesized using a
hanning window and constant overlap-add gain.

All convolutions are along the time dimension and all strides are of unit
value. The models have approximately $300k$ trainable parameters, which,
within a deep learning context, represents a model that is not very large or
difficult to train.

Figure 2:  mae values for linear time-varying tasks with different input size
frames.

###  3.5 Training

The training of the model is performed in two steps. The first step is to
train only the convolutional layers for an unsupervised learning task, while
the second step consists of an end-to-end supervised learning task based on a
given time-varying target. During the first step only the weights of Conv1D
and Conv1D-Local are trained and both the raw audio $x(t)$ and wet audio
$y(t)$ are used as input and target functions. This means the model is being
prepared to reconstruct the input and target data in order to have a better
fitting when training for the time-varying task. Only during this step, the
unpooling layer of the back-end uses the time positions of the maximum values
recorded by the max-pooling operation.

Once the model is pretrained, the Bi-LSTM and DNN-SAAF-SE layers are
incorporated into the model, and all the weights of the convolutional,
recurrent, dense and activation layers are updated. Since small amplitude
errors are as important as large ones, the loss function to be minimized is
the mean absolute error between the target and output waveforms. We explore
input size frames from $1024$ to $8192$ samples and we always use a hop size
of $50\%$. The batch size consisted of the total number of frames per audio
sample.

Adam is used as optimizer and we perform the pretraining for $200$ epochs and
the supervised training for $500$ epochs. In order to speed convergence,
during the second training step we start with a learning rate of $5\cdot
10^{-5}$ and we reduce it by $50\%$ every $150$ epochs. We select the model
with the lowest error for the validation subset.

Figure 3:  mae and msed values with the test dataset for all the time-varying
tasks. _od_ , _ef_ and _mb_ mean overdrive, envelope follower and multiband
respectively.

###  3.6 Dataset

Modulation based audio effects such as chorus, flanger, phaser, tremolo and
vibrato were obtained from the IDMT-SMT-Audio-Effects dataset [36]. It
corresponds to individual 2-second notes and covers the common pitch range of
various 6-string electric guitars and 4-string bass guitars.

The recordings include the raw notes and their respective effected versions
for $3$ different settings for each effect. For our experiments, for each of
the above effects, we only use the setting $\\#2$ from where we obtained the
unprocessed and processed audio for bass guitar. In addition, processing the
bass guitar raw audio, we implemented an auto-wah with a peak filter whose
center frequency ranges from $500$ Hz to $3$ kHz and modulated by a $5$ Hz
sinusoidal.

Since the previous audio effects are linear time-varying, we further test the
capabilities of the model by adding a nonlinearity to each of these effects.
Thus, using the bass guitar wet audio, we applied an overdrive (gain$=+10$dB)
after each modulation based effect.

We also use virtual analog implementations of a ring modulator and a Leslie
speaker to process the electric guitar raw audio. The ring modulator
implementation111https://github.com/nrlakin/robot_voice/blob/master/robot.py
is based on [17] and we use a modulator signal of $5$ Hz. The Leslie speaker
implementation222https://ccrma.stanford.edu/software/snd/snd/leslie.cms is
based on [18] and we model each of the stereo channels.

Finally, we also explore the capabilities of the model with nonlinear time-
invariant audio effects with long temporal dependencies, such as compressors
and auto-wah. We use the compressor and multiband compressor from
SoX333http://sox.sourceforge.net/ to process the electric guitar raw audio.
The settings of the compressor are as follows: attack time 10 ms, release time
100 ms, knee 1 dB, ratio 4:1 and threshold -40 dB. The multiband compressor
has 2 bands with a crossover frequency of $500$ Hz, attack time: $5$ ms and
$625$ $\mu$s, decay time: $100$ ms and $12.5$ ms, knee: $0$ dB and $6$ dB,
ratio: $3$:$1$ and $6$:$1$ and threshold: $-30$ dB and $-60$ dB.

Similarly, we use an auto-wah
implementation444https://github.com/lucieperrotta/ASP with an envelope
follower and a peak filter which center frequency modulates between $500$ Hz
to $3$ kHz.

For each time-varying task we use $624$ raw and effected notes and both the
test and validation samples correspond to $5\%$ of this subset each. The
recordings were downsampled to $16$ kHz and amplitude normalization was
applied with exception to the time-invariant audio effects.

###  3.7 Evaluation

Two metrics were used when testing the models with the various test subsets.
Since the mean absolute error depends on the amplitude of the output and
target waveforms, before calculating this error, we normalize the energy of
the target and the output and define it as the energy-normalized mean absolute
error (mae).

We also propose an objective metric which mimics human perception of amplitude
and frequency modulation. The modulation spectrum uses time-frequency theory
integrated with the psychoacoustics of modulation frequency perception, thus,
providing long-term knowledge of temporal fluctuation patterns [37]. We
propose the modulation spectrum euclidean distance (msed), which is based on
the audio features from [38] and [39] and is defined as follows:

  * •

A Gammatone filter bank is applied to the target and output entire waveforms.
In total we use $12$ filters, with center frequencies spaced logarithmically
from $26$ Hz to $6950$ Hz.

  * •

The envelope of each filter output is calculated via the magnitude of the
Hilbert transform and downsampled to $400$ Hz.

  * •

A Modulation filter bank is applied to each envelope. In total we use $12$
filters, with center frequencies spaced logarithmically from $0.5$ Hz to $100$
Hz.

  * •

The Fast Fourier Transform (FFT) is calculated for each modulation filter
output of each Gammatone filter. The energy is normalized by the DC value and
summarized in the following bands: $0.5$-$4$ Hz, $4.5$-$10$ Hz, $10.5$-$20$ Hz
and $20.5$-$100$ Hz.

  * •

The msed metric is the mean euclidean distance between the energy values at
these $4$ bands.

(a)

(b)

(c)

(d)

(e)

Figure 4: Various internal plots for the test dataset of the tremolo modeling
task. 4a) Input, target and output frames of $4096$ samples and their
respective FFT magnitudes. 4b) For the input frame $x(t)$, respective $8$ rows
from $\boldsymbol{R}$. 4c) Following the filter bank architecture, respective
$8$ rows from $\boldsymbol{X}_{2}$. 4d) From $\boldsymbol{\hat{Z}}$,
corresponding $8$ modulator signals learned by the Bi-LSTM layer. 4e) In the
same manner, $8$ rows from $\boldsymbol{\hat{X}}_{0}$, which is the input to
the deconvolution layer prior to obtaining the output frame $\hat{y}(t)$.
Vertical axes in 4b)-4e) are unitless and horizontal axes correspond to time.

##  4 Results & Analysis

First, we explore the capabilities of Bi-LSTMs to learn long-term temporal
dependencies. Fig. 2 shows the $mae$ results of the test dataset for different
input frame sizes and various linear time-varying tasks. The most optimal
results are with an input size of $4096$ samples, since shorter frame sizes
represent a higher error and $8192$ samples do not represent a significant
improvement. Since the average modulation frequency in our tasks is $2$ Hz,
for each input size we select a $k$ that covers one period of this modulator
signal. Thus, for the rest of our experiments, we use an input size of $4096$
samples and $k=4$ for the number of past and subsequent frames.

The training procedures were performed for each type of time-varying and time-
invariant audio effect. Then, the models were tested with samples from the
test dataset and the audio results are available
online555https://mchijmma.github.io/modeling-time-varying/. Fig. 3 shows the
mae and msed for all the test subsets. To provide a reference, the mean mae
and msed values between input and target waveforms are $0.15$ and $0.11$
respectively. It can be seen that the model performed well on each audio
effect modeling task. Overall, the model achieved better results with
amplitude modulation and time-varying filter audio effects, although delay-
line based effects were also successfully modeled.

Fig. 4 visualizes the functioning of the model for the tremolo task. It shows
how the model processes the input frame $x(t)$ into the different frequency
maps $\boldsymbol{X}_{1}$ and $\boldsymbol{X}_{2}$, learns a modulator signal
$\boldsymbol{\hat{Z}}$, and applies the respective amplitude modulation. This
linear time-varying audio effect is easy to interpret. For more complex
nonlinear time-varying effects, a more in-depth analysis of the model is
required.

For selected linear and nonlinear time-varying tasks, Fig. 5 shows the input,
target, and output waveforms together with their respective modulation
spectrum. In the time-domain, it is evident that the model is matching the
target waveform. From the modulation spectrum it is noticeable that the model
introduces different modulation energies into the output which were not
present in the input and which closely match those of the respective targets.

The task becomes more challenging when a nonlinearity is added to a linear
time-varying transformation. Fig. 5d depicts results for the phaser-overdrive
task. Given the large overdrive gain the resulting audio has a lower-frequency
modulation. It can be seen that the model introduces modulations as low as
$0.5$ Hz. But the waveform is not as smooth as the target, hence the larger
$mae$ values. Although the $mae$ increased, the model does not significantly
reduce performance and is able to match the combination of nonlinear and
modulation based audio effects.

Much more complicated time-varying tasks, such as the ring modulator and
Leslie speaker virtual analog implementations were also successfully modeled.
This represents a significant result, since these implementations include
nonlinear modulation; ring modulator, or varying delay lines together with
artificial reverberation and Doppler effect simulation; and the Leslie
speaker.

(a)

(b)

(c)

(d)

Figure 5: Results with the test dataset for the following time-varying tasks:
5a) chorus, 5b) ring modulator, 5c) Leslie speaker (right channel) and 5d)
phaser-overdrive. For each subfigure and from top to bottom: input, target and
output waveforms and respective modulation spectrum plots.

Lastly, the model is also able to perform linear and nonlinear time-invariant
modeling. The long temporal dependencies of an envelope driven auto-wah,
compressor and multiband compressor are succesfully modeled. Furthermore, in
the latter case, the crossover filters are also matched. The msed may not be
relevant for these effects, but the low mae values represent that the model
also performs well here.

##  5 Conclusion

In this work, we introduced a general-purpose deep learning architecture for
modeling audio effects with long temporal dependencies. Using raw audio and a
given time-varying task, we explored the capabilities of end-to-end deep
neural networks to learn low-frequency modulations and to process the audio
accordingly. The model was able to match linear and nonlinear time-varying
audio effects, time-varying virtual analog implementations and time-invariant
audio effects with long-term memory.

Other white-box or gray-box modeling methods suitable for these time-varying
tasks would require expert knowledge such as specific circuit analysis and
discretization techniques. Moreover, these methods can not easily be extended
to other time-varying tasks, and assumptions are often made regarding the
nonlinear behavior of certain components. To the best of our knowledge, this
work represents the first black-box modeling method for linear and nonlinear,
time-varying and time-invariant audio effects. It makes less assumptions about
the audio processor target and represents an improvement of the state-of-the-
art in audio effects modeling.

Using a small amount of training examples we showed the model matching chorus,
flanger, phaser, tremolo, vibrato, auto-wah, ring modulator, Leslie speaker
and compressors. We proposed an objective perceptual metric to measure the
performance of the model. The metric is based on the euclidean distance
between the frequency bands of interest within the modulation spectrum. We
demonstrated that the model processes the input audio by applying different
modulations which closely match with those of the time-varying target.

Perceptually, most output waveforms are indistinguishable from their target
counterparts, although there are minor discrepancies at the highest
frequencies and noise level. This could be improved by using more convolution
filters, which means a higher resolution in the filter bank structures [6].
Moreover, as shown in [5], a cost function based on time and frequency can be
used to improve this frequency related issue, though listening tests may be
required.

The generalization can also be studied more thoroughly, since the model learns
to apply the specific transformation to the audio of a specific musical
instrument, such as the electric guitar or the bass guitar. In addition, since
the model strives to learn long temporal dependencies with shorter input size
frames, and also needs past and subsequent frames, more research is needed on
how to adapt this architecture to real-time implementations.

Real-time applications would benefit significantly from the exploration of
recurrent neural networks to model transformations that involve long-term
memory without resorting to large input frame sizes and the need for past and
future context frames. Although the model was able to match the artificial
reverberation of the Leslie speaker implementation, a thorough exploration of
reverberation modeling is needed, such as plate, spring or convolution
reverberation. In addition, since the model is learning a static
representation of the audio effect, ways of devising a parametric model could
also be explored. Finally, applications beyond virtual analog can be
investigated, for example, in the field of automatic mixing the model could be
trained to learn a generalization from mixing practices.

##  6 Acknowledgments

The Titan Xp GPU used for this research was donated by the NVIDIA Corporation.
EB is supported by a RAEng Research Fellowship (RF/128).

## References

  * [1] Joshua D Reiss and Andrew McPherson,  Audio effects: theory, implementation and application,  CRC Press, 2014.
  * [2] Udo Zölzer,  DAFX: digital audio effects,  John Wiley & Sons, 2011.
  * [3] Julius Orion Smith,  Physical audio signal processing: For virtual musical instruments and audio effects,  W3K Publishing, 2010.
  * [4] Clifford A Henricksen,  “Unearthing the mysteries of the leslie cabinet,”  Recording Engineer/Producer Magazine, 1981.
  * [5] Marco A. Martínez Ramírez and Joshua D. Reiss,  “End-to-end equalization with convolutional neural networks,”  in 21st International Conference on Digital Audio Effects (DAFx-18), 2018.
  * [6] Marco A. Martínez Ramírez and Joshua D. Reiss,  “Modeling of nonlinear audio effects with end-to-end deep neural networks,”  in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2019.
  * [7] Jyri Pakarinen and David T Yeh,  “A review of digital techniques for modeling vacuum-tube guitar amplifiers,”  Computer Music Journal, vol. 33, no. 2, pp. 85–100, 2009.
  * [8] Dimitrios Giannoulis, Michael Massberg, and Joshua D Reiss,  “Digital dynamic range compressor design - a tutorial and analysis,”  Journal of the Audio Engineering Society, vol. 60, no. 6, pp. 399–408, 2012.
  * [9] David T Yeh, Jonathan S Abel, and Julius O Smith,  “Automated physical modeling of nonlinear audio circuits for real-time audio effects part I: Theoretical development,”  IEEE transactions on audio, speech, and language processing, vol. 18, no. 4, pp. 728–737, 2010.
  * [10] Antti Huovilainen,  “Enhanced digital models for analog modulation effects,”  in 8th International Conference on Digital Audio Effects (DAFx-05), 2005.
  * [11] Colin Raffel and Julius Smith,  “Practical modeling of bucket-brigade device circuits,”  in 13th International Conference on Digital Audio Effects (DAFx-10), 2010.
  * [12] Martin Holters and Julian D Parker,  “A combined model for a bucket brigade device and its input and output filters,”  in 21st International Conference on Digital Audio Effects (DAFx-17), 2018.
  * [13] Jaromír Mačák,  “Simulation of analog flanger effect using BBD circuit,”  in 19th International Conference on Digital Audio Effects (DAFx-16), 2016.
  * [14] David T Yeh,  “Automated physical modeling of nonlinear audio circuits for real-time audio effects part II: BJT and vacuum tube examples,”  IEEE Transactions on Audio, Speech, and Language Processing, vol. 20, 2012.
  * [15] Martin Holters and Udo Zölzer,  “Physical modelling of a wah-wah effect pedal as a case study for application of the nodal dk method to circuits with variable parts,”  in 14th International Conference on Digital Audio Effects (DAFx-11), 2011.
  * [16] Felix Eichas et al.,  “Physical modeling of the mxr phase 90 guitar effect pedal,”  in 17th International Conference on Digital Audio Effects (DAFx-14), 2014.
  * [17] Julian Parker,  “A simple digital model of the diode-based ring-modulator,”  in 14th International Conference on Digital Audio Effects (DAFx-11), 2011.
  * [18] Julius Smith et al.,  “Doppler simulation and the leslie,”  in 5th International Conference on Digital Audio Effects (DAFx-02), 2002.
  * [19] Jussi Pekonen, Tapani Pihlajamäki, and Vesa Välimäki,  “Computationally efficient hammond organ synthesis,”  in 14th International Conference on Digital Audio Effects (DAFx-11), 2011.
  * [20] Roope Kiiski, Fabián Esqueda, and Vesa Välimäki,  “Time-variant gray-box modeling of a phaser pedal,”  in 19th International Conference on Digital Audio Effects (DAFx-16), 2016.
  * [21] Kurt J Werner, W Ross Dunkel, and François G Germain,  “A computational model of the hammond organ vibrato/chorus using wave digital filters,”  in 19th International Conference on Digital Audio Effects (DAFx-16), 2016.
  * [22] Ólafur Bogason and Kurt James Werner,  “Modeling circuits with operational transconductance amplifiers using wave digital filters,”  in 20th International Conference on Digital Audio Effects (DAFx-17), 2017.
  * [23] Jordi Pons et al.,  “End-to-end learning for music audio tagging at scale,”  in 31st Conference on Neural Information Processing Systems (NIPS), 2017.
  * [24] Shrikant Venkataramani, Jonah Casebeer, and Paris Smaragdis,  “Adaptive front-ends for end-to-end source separation,”  in 31st Conference on Neural Information Processing Systems (NIPS), 2017.
  * [25] Daniel Stoller, Sebastian Ewert, and Simon Dixon,  “Wave-u-net: A multi-scale neural network for end-to-end audio source separation,”  in 19th International Society for Music Information Retrieval Conference, 2018.
  * [26] Sander Dieleman and Benjamin Schrauwen,  “End-to-end learning for music audio,”  in International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2014.
  * [27] Soroush Mehri et al.,  “Samplernn: An unconditional end-to-end neural audio generation model,”  in 5th International Conference on Learning Representations. ICLR, 2017.
  * [28] Jesse Engel et al.,  “Neural audio synthesis of musical notes with wavenet autoencoders,”  34th International Conference on Machine Learning, 2017.
  * [29] Merlijn Blaauw and Jordi Bonada,  “A neural parametric singing synthesizer,”  in Interspeech 2017.
  * [30] Xavier Glorot, Antoine Bordes, and Yoshua Bengio,  “Deep sparse rectifier neural networks,”  in 14th International Conference on Artificial Intelligence and Statistics, 2011.
  * [31] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton,  “Speech recognition with deep recurrent neural networks,”  in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2013.
  * [32] Alex Graves and Jürgen Schmidhuber,  “Framewise phoneme classification with bidirectional lstm and other neural network architectures,”  Neural Networks, vol. 18, 2005.
  * [33] Le Hou et al.,  “Convnets with smooth adaptive activation functions for regression,”  in Artificial Intelligence and Statistics, 2017.
  * [34] Jie Hu, Li Shen, and Gang Sun,  “Squeeze-and-excitation networks,”  in IEEE Conference on Computer Vision and Pattern Recognition, 2018\.
  * [35] Taejun Kim, Jongpil Lee, and Juhan Nam,  “Sample-level cnn architectures for music auto-tagging using raw waveforms,”  in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2018.
  * [36] Michael Stein et al.,  “Automatic detection of audio effects in guitar and bass recordings,”  in 128th Audio Engineering Society Convention, 2010.
  * [37] Somsak Sukittanon, Les E Atlas, and James W Pitton,  “Modulation-scale analysis for content identification,”  IEEE Transactions on Signal Processing, vol. 52, 2004.
  * [38] Josh H McDermott and Eero P Simoncelli,  “Sound texture perception via statistics of the auditory periphery: evidence from sound synthesis,”  Neuron, vol. 71, 2011.
  * [39] Martin McKinney and Jeroen Breebaart,  “Features for audio and music classification,”  in 4th International Society for Music Information Retrieval Conference, 2003.