# Continuous Language Model Interpolation for Dynamic and Controllable Text
Generation

Sara Kangaslahti
Harvard University
<EMAIL_ADDRESS>&David Alvarez-Melis
Harvard University
<EMAIL_ADDRESS>
###### Abstract

As large language models (LLMs) have gained popularity for a variety of use
cases, making them adaptable and controllable has become increasingly
important, especially for user-facing applications. While the existing
literature on LLM adaptation primarily focuses on finding a model (or models)
that optimizes a single predefined objective, here we focus on the challenging
case where the model must dynamically adapt to diverse —and often changing—
user preferences. For this, we leverage adaptation methods based on linear
weight interpolation, casting them as continuous multi-domain interpolators
that produce models with specific prescribed generation characteristics on-
the-fly. Specifically, we use low-rank updates to fine-tune a base model to
various different domains, yielding a set of anchor models with distinct
generation profiles. Then, we use the weight updates of these anchor models to
parametrize the entire (infinite) class of models contained within their
convex hull. We empirically show that varying the interpolation weights yields
predictable and consistent change in the model outputs with respect to all of
the controlled attributes. We find that there is little entanglement between
most attributes and identify and discuss the pairs of attributes for which
this is not the case. Our results suggest that linearly interpolating between
the weights of fine-tuned models facilitates predictable, fine-grained control
of model outputs with respect to multiple stylistic characteristics
simultaneously.111Code: https://github.com/skangasl/continuous-lm-
interpolation

##  1 Introduction

Large language models (LLMs) are used for a diverse set of applications due to
their high performance across a wide spectrum of tasks (Bubeck et al., 2023).
In many common LLM use cases (such as chatbots), different users often have
distinct and continuously evolving preferences for the type of output they
want. For example, a user might want a creative and verbose response for
certain queries, but a concise and precise response for others. In practice, a
user may try different variations of the same query succesively until they
elicit a generation that matches their goal. This trial-and-error process can
be time-consuming and lacks guaranteed results, especially since minor word
changes in a prompt can have disproportionate impact on the output.
Additionally, expressing fine-grained continuous preferences (e.g., simplicity
of the response) is often difficult in —inherently discrete— natural language.
These challenges are exacerbated when the user has complex, multi-faceted
preferences (e.g., a specific combination of simplicity, formality, and
verbosity) that they expect the generation to satisfy all at once. As a
result, there is a pressing need for methods that allow for fine-grained and
predictable control over LLM text generation, and which can adapt on-the-fly
to mutable user preferences and constraints.

Prior work in controllable text generation (CTG) has largely focused on
optimizing for one set of control criteria through techniques such as
instruction tuning (Zhou et al., 2023), modifying the output probability
distributions (Pascual et al., 2021; Yang & Klein, 2021; Dekoninck et al.,
2024), changing model activations at inference time (Li et al., 2023),
learning modifications to the embeddings (Li & Liang, 2021; Han et al., 2023),
or training (Keskar et al., 2019; Krause et al., 2021). These methods,
however, do not naturally allow for the composition of multiple objectives and
lack fine-grained control, especially those that rely on the user expressing
preferences in natural language, for the reasons described above. Embedding
modification and inference-time approaches do not allow for as complex tuning
to the objective as fine-tuning based ones and often require additional
training for each control variable value. While fine-tuning to each desired
objective would likely allow for the most precise optimization, this is
computationally infeasible to do for each combination of control variables and
strengths of control in the entire (infinite) set of possible combinations.

With these challenges in mind, here we seek to enable dynamic and controllable
text generation in a manner that takes advantage of the strengths of fine-
tuning while remaining computationally feasible for dynamically changing
control variables. Recent work has demonstrated that multiple pre-trained or
fine-tuned models can be effectively composed through linear weight
interpolation (Wortsman et al., 2022; Ilharco et al., 2023). This has also
been shown to extend to models trained with parameter-efficient fine-tuning
(PEFT) methods (Zhang et al., 2023; Huang et al., 2024) such as low-rank
adaptation (Hu et al., 2021). We build upon and extend this line of work by
showing that linear weight interpolation can be used to obtain models with
specific mixtures of characteristics on-the-fly and without additional
training, effectively providing a continuous parametrization of the (infinite)
‘convex hull’ of a set of fine-tuned models. To do so, we fine-tune two
endpoint anchor models for each control attribute, one at each extreme of
attribute strength. We then interpolate along the vector between the weights
of these two models for each attribute before computing a weighted average
across all of the single-attribute interpolated models. Thus, varying the
interpolation and averaging weights gives us dense coverage of the model
parameter space, allowing us to create models tailored to any preference
profile spanned by the fine-tuned models. We evaluate linear weight
interpolation for multiple style attributes and demonstrate empirically that
changes in the interpolation and averaging weights yield predictable and
consistent responses in the level of each attribute in the generated text.

Figure 1: Overview of our continuous weight interpolation formulation. The
blue boxes represent the inputs controlled on-the-fly by the user. When the
user chooses $\alpha$ and $\lambda$, we first interpolate between the weights
$\theta_{+i}$ and $\theta_{-i}$ of two fine-tuned endpoint models for each
control $i$ using $\alpha_{i}$, then weight each interpolated model
$\theta_{\alpha_{i}}$ by $\lambda_{i}$ and sum over all
$\lambda_{i}\theta_{\alpha_{i}}$. In this manner, the user dynamically
controls attributes of the output by specifying $\alpha$ and $\lambda$.

A potential pitfall of this approach is that, as seen in prior work in the
vision domain (Ortiz-Jimenez et al., 2023), the weights for different single-
attribute interpolated models may be entangled. This could lead to unexpected
correlations between attributes in the averaged models. These correlations are
detrimental to CTG, as changing the interpolation weights for one attribute
could have an unexpected effect on the correlated attributes in the output
text. However, we find that there is surprisingly little entanglement between
the vast majority of control attributes and analyze the pairs of controls
where this is not the case.

In summary, our key contributions are: (1) we show how parameter-efficient
adaptation methods can be used to continuously interpolate between models
fine-tuned with various distinct generation objectives, allowing for on-the-
fly adaptation to user-specified generation preferences expressed in terms of
interpretable control variables; and (2) we demonstrate that changes in the
interpolation yield smooth and predictable changes in the properties of the
generated text across multiple sets of controls with limited entanglement.

##  2 Fine-tuning and weight interpolation

We evaluate the ability of weight interpolation to control the outputs of LLMs
on five commonly used style attributes defined in prior style transfer
literature (Jin et al., 2022): simplicity, formality, politeness, sentiment,
and humor. For every style characteristic, we first fine-tune two endpoint
‘anchor’ models, each of which optimizes for one extreme of the style
attribute. We then use these models as the basis of the interpolation scheme.

###  2.1 Datasets

For each style attribute, we fine-tune a separate anchor Llama2-7b model
(Touvron et al., 2023) on two datasets representing the extremes of the
attribute level. For simplicity, we use the TinyStories dataset (Eldan & Li,
2023) to fine-tune a simple model and novel chapters from the BookSum dataset
(Kryscinski et al., 2021) to fine-tune a complex model. We use the documents
classified as formal and informal in Grammarly’s Yahoo Answers Formality
Corpus (GYAFC) dataset (Rao & Tetreault, 2018) to fine-tune formal and
informal models. For the politeness attribute, we use the documents in the
highest and lowest politeness class in the work by Madaan et al. (2020) for
fine-tuning polite and impolite models, respectively. We fine-tune positive
and negative sentiment models using the Stanford Sentiment Treebank (SST-2)
dataset (Socher et al., 2013). For humor, we use the FlickrStyle dataset (Gan
et al., 2017) to fine-tune humorous and non-humorous models.

###  2.2 Fine-tuning

We employ Low-Rank Adaptation (LoRA) in order to fine-tune our models in a
parameter-efficient manner (Hu et al., 2021). In LoRA fine-tuning, at each
layer of the transformer model, the pretrained model weights are frozen and
low-rank decomposition matrices are learned to adapt the model in fine-tuning.
Denoting the pretrained language model weights as
$\theta_{PRE}\in\mathbb{R}^{d_{1}\times d_{1}}$, LoRA computes the updated
weights as follows:

$\theta=\theta_{PRE}+BA$ (1)

Here, $A\in\mathbb{R}^{k\times d_{2}}$ and $B\in\mathbb{R}^{d_{1}\times k}$
(with $k\ll d_{1},d_{2}$) are trainable parameters learned during fine-tuning.
We use LoRA as an adaptation method because it requires significantly fewer
parameters than traditional fine-tuning while maintaining similar performance,
so LoRA weights can be quickly modified and applied to large pretrained
language models. We use the parameters in Appendix A.1 for fine-tuning the
models and fine-tune two LoRA models per style characteristic, one on each of
the extreme classes outlined in 2.1.

###  2.3 Linear weight interpolation

We formulate linear weight interpolation between the LoRA fine-tuned models in
terms of interpolation weights $\alpha_{i}$ and attribute mixing weights
$\lambda_{i}$ as shown in Figure 1. We denote
$\theta_{+i}=\theta_{PRE}+B_{+i}A_{+1}$ and
$\theta_{-i}=\theta_{PRE}+B_{-i}A_{-1}$ as the two LoRA fine-tuned endpoint
anchor models for attribute $i$. Then, for a single attribute, we interpolate
along the vector between the two fine-tuned endpoint models by computing

$\displaystyle\theta_{\alpha_{i}}$
$\displaystyle=\theta_{PRE}+\alpha_{i}\theta_{+i}+(1-\alpha_{i})\theta_{-i}=\theta_{PRE}+\alpha_{i}B_{+i}A_{+i}+(1-\alpha_{i})B_{-i}A_{-i}$
(2)

We call $\alpha_{i}$ the interpolation weight for the $i$th attribute
dimension. We note that $\alpha_{i}=0$ and $\alpha_{i}=1$ correspond to
letting the interpolated model equal the fine-tuned models
$\theta_{\alpha_{i}}=\theta_{-i}$ and $\theta_{\alpha_{i}}=\theta_{+i}$,
respectively. Using Equation 2, we then combine multiple interpolated models
$\theta_{\alpha_{i}}$ by taking their weighted sum:

$\displaystyle\theta_{\alpha,\lambda}$
$\displaystyle=\sum_{i}\lambda_{i}\theta_{\alpha_{i}}=\theta_{PRE}+\sum_{i}\lambda_{i}(\alpha_{i}B_{+i}A_{+i}+(1-\alpha_{i})B_{-i}A_{-i})$
(3)

We denote $\lambda_{i}$ to be the mixing weight for the $i$th attribute and
constrain $\sum_{i}\lambda_{i}=1$. We note that the case with one attribute
dimension corresponds to the sum having a single term with $\lambda_{1}=1$.
With this formulation, we can construct any model in the convex hull of the
fine-tuned models by choosing appropriate interpolation weights $\alpha$ and
mixing weights $\lambda$.

(a) Interpolation between the anchor models

(b) Linear extrapolation beyond the anchor models

Figure 2: Effect of linear weight interpolation for a single attribute
dimension. For each style attribute, we report the attribute score when
linearly interpolating between the models with weight $\alpha$. We show the
effect of 2(a) interpolating between the two fine-tuned models and 2(b)
extrapolating beyond the fine-tuned models ($\alpha<0$ and $\alpha>1$).
Increasing $\alpha$ between the two fine-tuned models results in a smooth
increase in the attribute score. When extrapolating, there is a stable region
where this trend continues until a certain point (around $\alpha$ equal to
$-1$ and $2$), where performance degrades and the extrapolation is unstable.
Figure 3: Wikitext perplexity of linearly interpolated and extrapolated
models. We report the average perplexity (lower is better) of each model from
Figure 2(b) on the Wikitext test set. For all of the interpolated models
($\alpha\in[0,1]$), the perplexity is either better than or between the
performance of the two fine-tuned models. For the extrapolated models
($\alpha<0.0$ and $\alpha>1.0$), the perplexity increases rapidly beyond
$\alpha$ values of around $-1$ and $2$. We clip the y-axis at 7.0 for
readability (the full plot is shown in Figure 9).

###  2.4 Evaluation

To evaluate the generations of each interpolated model, we use a subset of 1k
randomly sampled prompts from the WritingPrompts dataset (Fan et al., 2018)
and generate 3 continuations for each prompt. We compute scores for each of
the attributes to evaluate the level of each control criterion. Similarly to
prior work on text style transfer (Xu et al., 2018), we fine-tune a RoBERTa
(Liu et al., 2019) classification head on each attribute and compute a sigmoid
over the output logits to obtain the probability of class $1$, which we report
as the attribute score. We label the documents such that an attribute score
closer to $1$ corresponds to a document that is more simple, formal, polite,
positive in sentiment, or humorous. We also compute perplexity on the test
split of the WikiText dataset (Merity et al., 2016) to evaluate model fluency.

##  3 Continuous Language Model Interpolation

We begin by investigating the linear interpolations between each pair of low-
rank fine-tuned anchor models (3.1). We then extend this analysis to the
convex hull of fine-tuned models for multiple attributes (3.2).

Figure 4: Effect of $\alpha_{i}$ and $\lambda_{i}$ on attribute scores for
multiple control dimensions. In each plot, we report the attribute scores for
multiple $\alpha_{i}$ values when $\alpha_{i}$ is varied for the attribute $i$
in the column title and kept constant at $0.0$ for the remaining attributes.
$\lambda_{i}$ for the control dimension $i$ for which the $\alpha_{i}$ weight
is being varied is listed in the row titles. The mixture weights $\lambda_{j}$
for all remaining $j$ are set uniformly so that the sum of $\lambda$ weights
is $1$ ($\lambda_{j}=(1.0-\lambda_{i})/4$). For all of the attribute
dimensions, increasing $\alpha_{i}$ results in an increasing attribute score
for that control dimension with limited effect on the other attribute scores.

###  3.1 Linear interpolation for a single attribute dimension

We first explore the effect of moving along the vector between a single pair
of fine-tuned anchor models. We note that $\alpha=0$ and $\alpha=1$ correspond
to the two fine-tuned anchor models, while $\alpha\in(0.0,1.0)$ is an
interpolation along the vector between the two models and
$\alpha\in(-\infty,0.0)\cup(1.0,\infty)$ is a linear extrapolation along the
vector between the models.

Linear interpolation: Figure 2 shows the effect of $\alpha$ on attribute
score. For all of the attributes, when interpolating between the two fine-
tuned models (Figure 2(a)), as $\alpha$ increases, there is a smooth increase
in the attribute score for all of the control dimensions. The model output
quality also remains high, as for every attribute the perplexity in the
interpolation region is either less than or between the perplexities of the
two fine-tuned models (Figure 3). These results indicate that for one control
attribute, interpolating between two endpoint models yields fine-grained
control over the model outputs. Furthermore, similarly to Dekoninck et al.
(2024), the trend of increase with $\alpha$ appears linear in some cases (and
nonlinear in others). For the majority of the attribute dimensions
(politeness, formality, and simplicity) we observe a linear increase in the
score as $\alpha$ increases in the interpolation region. On the other hand,
the other control dimensions (sentiment and humor) have a nonlinear increase
in attribute score with $\alpha$ due to plateaus at one or more of the
extremes.

Linear extrapolation: Figure 2(b) shows the attribute scores when
extrapolating linearly beyond the two fine-tuned models along the vector
between them. We find that even beyond the region of interpolation between the
two fine-tuned models, there is a small stable extrapolation regime up to
$\alpha$ values of around $-1$ and $2$ (Figure 2(b)). In this region, for many
of the attributes, the attribute score continues to behave predictably as
$\alpha$ is increased. However, beyond the stable extrapolation values, there
is an unstable extrapolation regime where the attribute score changes
unpredictably as $\alpha$ is varied. This is likely due to the model output
quality degrading, since as shown in Figure 3, the model perplexity increases
sharply starting near the edges of the stable extrapolation regime. While
prior work has shown that linear weight extrapolation can be used for tasks
such as model unlearning (Ilharco et al., 2023; Zhang et al., 2023), these
results provide a cautionary tale against extrapolating too far, as they
suggest that this ability only extends to a certain threshold before the
attribute score and model outputs become unpredictable due to poor quality
outputs. For the remainder of our experiments, we thus focus on the
interpolation regime.

###  3.2 Multi-dimensional interpolation

In real-world LLM applications, users often have diverse output preferences
across multiple control dimensions at once, and these preferences may change
dynamically for different inputs to the LLM. In this section, we show that
linear interpolation between fine-tuned parameter-efficient adapters can be
used to parametrize a whole convex hull of models, which can be used to
dynamically generate text with attribute levels specified on-the-fly.

####  3.2.1 Parametrization of the convex hull

Fine-grained analysis of the interpolation parameter $\alpha$: We find that
when interpolating across up to five attribute dimensions, modifying the
weight parameters $\lambda_{i}$ and $\alpha_{i}$ results in predictable, fine-
grained control over the attribute scores for the desired attributes while
having a comparatively small effect on the remaining attributes. Each spider
plot in Figure 4 shows that increasing the $\alpha_{i}$ parameter for
interpolating between the fine-tuned models increases the attribute score for
the $i$th attribute while the other scores remain fairly constant. Similarly,
as the model mixture parameter $\lambda_{i}$ increases, the effect on the
attribute score of changing $\alpha_{i}$ increases. While there is also more
effect on the other attributes as $\lambda_{i}$ increases, this effect is
still comparatively small in relation to the effect on the desired attribute.

(a) Effect of $\lambda$ on score when $\alpha_{i}=0$

(b) Effect of $\lambda$ on score when $\alpha_{i}=1$

Figure 5: Effect of $\lambda_{i}$ on the attribute scores for multiple
attribute dimensions. For each control dimension $i$ and combination of
parameters from Figure 4 where $\alpha_{i}=0$ (5(a)) and Figure 11 where
$\alpha_{i}=1$ (5(b)), we report the attribute score versus $\lambda_{i}$. In
5(a), as $\lambda_{i}$ increases, the attribute score decreases smoothly
toward the score for the $\alpha_{i}=0$ model, and in 5(b), as $\lambda_{i}$
increases, the attribute score increases smoothly toward the $\alpha_{i}=1$
score.

Fine-grained analysis of the mixing parameter $\lambda$: The effect of
$\lambda$ is further demonstrated in Figure 5(a), which plots the scores from
Figure 4 for each attribute dimension and $\lambda_{i}$ value when
$\alpha_{i}=0$ across all values of $\lambda_{j}$ for the other control
dimensions. We note that in this case, increasing $\lambda_{i}$ should
upweight the $\alpha_{i}=0$ model and thus decrease the attribute score.
Figure 5(b) is the analogous plot for $\alpha_{i}=1$, showing the scores
averaged across all combinations of weights from Figure 11 when $\alpha_{i}=1$
for the attribute dimension being plotted. In this case, increasing
$\lambda_{i}$ should increase the score, since the weight of the
$\alpha_{i}=1$ model is increasing. Combined, these plots shows that as
$\lambda_{i}$ increases, the output scores move smoothly toward the desired
extreme model for both the $\alpha_{i}=0$ and $\alpha_{i}=1$ case, showing
that the $\lambda$ parameter also provides fine-grained control over the model
outputs.

Figure 6: Effect of $\lambda_{i}$ on interpolation between the sentiment,
politeness, and simplicity dimensions for $\alpha_{i}=1$. The vertices of the
triangle represent the models with $\alpha_{i}=1$ for each of the three
attribute dimensions. The scores in the simplex of $\lambda$ weights between
the three control dimensions smoothly interpolate between the three extreme
models.

Figure 7: Effect of $\lambda_{i}$ on interpolation between the humor,
formality, and simplicity dimensions for $\alpha_{i}=1$. The vertices of the
triangle represent the models with $\alpha_{i}=1$ for each of the three
attribute dimensions. The scores in the simplex of $\lambda$ weights smoothly
interpolate between the three endpoints in the simplicity case, but the
averaged models are the most neutral in the other cases due to correlations
between the control dimensions.

Changing mixing parameters $\lambda$ for multiple attributes at once: We also
analyze the relationship throughout the whole simplex of $\lambda$ weights for
sets of three control dimensions in Figures 6 and 7 (as well as Figures 12-21
in the Appendix). For each set of three attributes listed, these plots show
the scores in the three dimensional simplex of mixing weights $\lambda$ for
which $\sum_{i}\lambda_{i}=1$. The value of the interpolation weight
$\alpha_{i}$ for each of the attributes is equal to 1 in Figures 6 and 7, so
increasing the $\lambda$ weight of each attribute should increase the
attribute score. We find that surprisingly, there is very limited entanglement
between the majority of the combinations of attributes (such as in Figure 6).
In these cases, we observe an approximately even increase in score as
$\lambda_{i}$ for a given attribute dimension increases, regardless of the
other $\lambda_{j}$ parameters.

However, in the case of humor in the humor-formality-simplicity simplex and
formality in the humor-formality-simplicity simplex with $\alpha_{i}=1$
(Figure 7) and the sentiment-politeness-formality simplex with $\alpha_{i}=1$
(Figure 17), we observe regions at the corners of the simplex that are close
to the other fine-tuned models and have a high attribute score. This is
because these other models are correlated with a positive attribute score, so
the mixture of models is the most neutral model. Nevertheless, this still has
a limited effect on the attribute score, since even in these cases with
correlations, the score still has the expected behavior unless the mixing
weight $\lambda_{j}$ is greater than around $0.4$ to $0.6$ for the correlated
control dimensions. This indicates that in practice, the model has smoothly
increasing attribute scores with $\lambda_{i}$ for all pairs of attributes
when $\lambda_{j}$ for the other attribute dimensions remains sufficiently
low.

These results demonstrate that as the parameters $\lambda_{i}$ and
$\alpha_{i}$ are increased for the $i$th attribute, there is a significant
effect on the attribute score for the $i$th control dimension and a limited
effect on the scores for the remaining attributes. Therefore, $\lambda_{i}$
and $\alpha_{i}$ parametrize the convex hull of models between all of the
attribute dimensions and yield fine-grained control over the model outputs
with respect to all of the attributes being considered.

####  3.2.2 Fine-tuned models and correlations

Given the results from the simplex plots, we analyze the relationships between
the fine-tuned endpoint models to better understand the attribute score
correlations. Figure 8, which plots the average cosine similarity between the
LoRA layers of each pair of models, shows that the LoRA weights are relatively
orthogonal to each other in most cases. We hypothesize that the lower
orthogonality between each pair of endpoint models for the same attribute is
because the models are trained on similar datasets. This is supported by the
fact that the simple and complex models are the most orthogonal of the pairs
of endpoint models and they are the only two models trained on different
datasets rather than different classes from the same dataset. In addition, the
humor models tend to deviate the most from orthogonality with the other models
(such as politeness), so this may provide a partial explanation for why some
of the other models were correlated with a higher humor score.

Figure 8: Cosine similarity of LoRA weights averaged across layers between
each pair of fine-tuned models. The LoRA weights are all relatively orthogonal
to each other, except some of the two endpoint models for the same attribute
are less orthogonal to each other, as well as the politeness and humorous
models.

##  4 Related work

###  4.1 Controllable text generation (CTG)

As it is crucial to constrain generated text in many downstream applications,
CTG has been a recent focus of NLP research. Methods such as CTRL (Keskar et
al., 2019) and GeDI (Krause et al., 2021) pretrain language models on text
prepended with control codes and generate text conditioned on the desired
control. However, these methods require pretraining a new model if new
controls are added, which is computationally expensive. To mitigate these
issues, a variety of methods have been proposed to perform CTG without
additional language model training. For example, Khalifa et al. (2021);
Pascual et al. (2021); Yang & Klein (2021); Dekoninck et al. (2024) constrain
language model outputs by modifying their output probability distributions. Li
& Liang (2021); Qian et al. (2022) learn prefixes and Han et al. (2023) train
a linear factor in the word embedding space. Subramani et al. (2022);
Hernandez et al. (2023); Li et al. (2023); Turner et al. (2023) control model
outputs by changing activations at inference time. Zhou et al. (2023) use
instruction tuning for CTG.

In this prior CTG research, only Dekoninck et al. (2024) show that their
method is composable and achieves fine-grained control over multiple
attributes at once. However, as this method requires composing multiple models
at inference time, the inference cost is significantly higher than inference
from a single weight-interpolated model, especially as the model size and
number of controlled attributes increases. In addition, combining low-rank
fine-tuning weights instead of probability distributions allows for more
complex relationships between the models to be taken into account when
composing them, which will likely allow for greater flexibility as the number
of controlled attributes increases.

###  4.2 Weight interpolation

Our work builds on prior work on linear weight interpolation, such as task
vectors (Ilharco et al., 2023), parameter-efficient task vectors (Zhang et
al., 2023), and model souping (Wortsman et al., 2022), as we use linear
interpolation and weighted model averaging as the basis for our analysis.
Prior work in this domain has focused mainly on improving multitask
performance when composing fully fine-tuned models (Matena & Raffel, 2021;
Yadav et al., 2023; Ortiz-Jimenez et al., 2023; Ramé et al., 2023) or
parameter-efficient fine-tuned models (Huang et al., 2024; Jiang et al.,
2024). However, these methods all differ from our work, since they focus on
combining model weights to improve a single multitask objective rather than
analyzing performance across a wide range of flexible, diverse objectives.
These approaches are orthogonal to our work and could be used in conjunction
with it to better combine the $\alpha$-interpolated models. Perhaps most
similar to our work are methods that interpolate between the weights of fine-
tuned models to control over a range of outputs (Gandikota et al., 2023;
Nylund et al., 2023). However, Gandikota et al. (2023) focus on the vision
domain and use a fine-tuning objective specific to diffusion models, and
Nylund et al. (2023) only analyze control over the time dimension.

##  5 Conclusion and future work

In this work, we show that continuous linear interpolation between low-rank
fine-tuned models can be used to parametrize the models in their convex hull.
We achieve fine-grained, predictable control over multiple attributes of style
at once by changing the interpolation weights between two anchor fine-tuned
models and the mixing weights between different interpolated attribute models.
We find that the interpolation profiles between models are smooth and there is
surprisingly little entanglement between the models for different control
dimensions. In other words, changing the weight for one attribute has a very
small effect on the scores for other attributes, especially for sufficiently
small mixing weights. As a result, we show that linear weight interpolation
can be used to dynamically adjust to diverse sets of changing preferences and
generate text that adheres to multiple controls simultaneously.

Limitations and future work: The main limitation of our work is that some
pairs of attributes are correlated, so when a correlated model has a large
mixing weight, it can unpredictably affect other control attributes. Thus, a
natural extension of this work would be to investigate whether this
correlation is inherent to the pair of tasks or if it can be eliminated. For
example, text that is more polite might always be more formal. However, it may
be the case that some correlations can be reduced by regularizing the LoRA
updates to be more orthogonal to each other or by merging the
$\alpha$-interpolated using more sophisticated methods that have recently
shown improvement over naive weight averaging in the multitask setting (Matena
& Raffel, 2021; Yadav et al., 2023; Ortiz-Jimenez et al., 2023; Ramé et al.,
2023).

Another potential focus of future work could be to extend the extrapolation
results to multiple control dimensions to analyze whether it is possible to
reliably generate text beyond the fine-tuned models when controlling multiple
attributes at once. This could be useful to further extend the range of
control over the model outputs.

##  6 Ethics Statement

Continuous weight interpolation may output text that contains existing biases
from the pre-trained models and fine-tuning datasets. It could also be used to
control the level of undesirable attributes such as toxicity. However, we
believe that this work is still beneficial overall, since it can be used to
improve the experience of LLM users for a variety of applications, and these
issues are faced by all pre-trained and fine-tuned language models.

##  7 Reproducibility

We provide code and the scripts used to run experiments at
https://github.com/skangasl/continuous-lm-interpolation. The fine-tuning
hyperparameters are included in Appendix A.1.

## References

  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang.  Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
  * Dekoninck et al. (2024) Jasper Dekoninck, Marc Fischer, Luca Beurer-Kellner, and Martin Vechev.  Controlled text generation via language model arithmetic.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=SLw9fp4yI6.
  * Eldan & Li (2023) Ronen Eldan and Yuanzhi Li.  Tinystories: How small can language models be and still speak coherent english?, 2023.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin.  Hierarchical neural story generation.  In Iryna Gurevych and Yusuke Miyao (eds.), _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 889–898, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-1082.  URL https://aclanthology.org/P18-1082.
  * Gan et al. (2017) Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng.  Stylenet: Generating attractive visual captions with styles.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 955–964, 2017.  doi: 10.1109/CVPR.2017.108.
  * Gandikota et al. (2023) Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau.  Concept sliders: Lora adaptors for precise control in diffusion models, 2023.
  * Han et al. (2023) Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, and Heng Ji.  Lm-switch: Lightweight language model conditioning in word embedding space, 2023.
  * Hernandez et al. (2023) Evan Hernandez, Belinda Z. Li, and Jacob Andreas.  Inspecting and editing knowledge representations in language models, 2023.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _CoRR_ , abs/2106.09685, 2021.  URL https://arxiv.org/abs/2106.09685.
  * Huang et al. (2024) Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin.  Lorahub: Efficient cross-task generalization via dynamic lora composition, 2024.
  * Ilharco et al. (2023) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi.  Editing models with task arithmetic, 2023.
  * Jiang et al. (2024) Weisen Jiang, Baijiong Lin, Han Shi, Yu Zhang, Zhenguo Li, and James T. Kwok.  Byom: Building your own multi-task model for free, 2024.
  * Jin et al. (2022) Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea.  Deep Learning for Text Style Transfer: A Survey.  _Computational Linguistics_ , 48(1):155–205, 04 2022.  ISSN 0891-2017.  doi: 10.1162/coli˙a˙00426.  URL https://doi.org/10.1162/coli_a_00426.
  * Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher.  CTRL: A conditional transformer language model for controllable generation.  _CoRR_ , abs/1909.05858, 2019.  URL http://arxiv.org/abs/1909.05858.
  * Khalifa et al. (2021) Muhammad Khalifa, Hady Elsahar, and Marc Dymetman.  A distributional approach to controlled text generation.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=jWkw45-9AbL.
  * Krause et al. (2021) Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani.  GeDi: Generative discriminator guided sequence generation.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pp. 4929–4952, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.findings-emnlp.424.  URL https://aclanthology.org/2021.findings-emnlp.424.
  * Kryscinski et al. (2021) Wojciech Kryscinski, Nazneen Fatema Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir R. Radev.  Booksum: A collection of datasets for long-form narrative summarization.  _CoRR_ , abs/2105.08209, 2021.  URL https://arxiv.org/abs/2105.08209.
  * Li et al. (2023) Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg.  Inference-time intervention: Eliciting truthful answers from a language model, 2023.
  * Li & Liang (2021) Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  _CoRR_ , abs/2101.00190, 2021.  URL https://arxiv.org/abs/2101.00190.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692, 2019.  URL http://arxiv.org/abs/1907.11692.
  * Madaan et al. (2020) Aman Madaan, Amrith Setlur, Tanmay Parekh, Barnabás Póczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W. Black, and Shrimai Prabhumoye.  Politeness transfer: A tag and generate approach.  _CoRR_ , abs/2004.14257, 2020.  URL https://arxiv.org/abs/2004.14257.
  * Matena & Raffel (2021) Michael Matena and Colin Raffel.  Merging models with fisher-weighted averaging.  _CoRR_ , abs/2111.09832, 2021.  URL https://arxiv.org/abs/2111.09832.
  * Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models.  _CoRR_ , abs/1609.07843, 2016.  URL http://arxiv.org/abs/1609.07843.
  * Nylund et al. (2023) Kai Nylund, Suchin Gururangan, and Noah A. Smith.  Time is encoded in the weights of finetuned language models, 2023.
  * Ortiz-Jimenez et al. (2023) Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard.  Task arithmetic in the tangent space: Improved editing of pre-trained models, 2023.
  * Pascual et al. (2021) Damian Pascual, Beni Egressy, Clara Meister, Ryan Cotterell, and Roger Wattenhofer.  A plug-and-play method for controlled text generation.  _CoRR_ , abs/2109.09707, 2021.  URL https://arxiv.org/abs/2109.09707.
  * Qian et al. (2022) Jing Qian, Li Dong, Yelong Shen, Furu Wei, and Weizhu Chen.  Controllable natural language generation with contrastive prefixes, 2022.
  * Ramé et al. (2023) Alexandre Ramé, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord.  Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards, 2023.
  * Rao & Tetreault (2018) Sudha Rao and Joel Tetreault.  Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer.  In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pp. 129–140, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-1012.  URL https://aclanthology.org/N18-1012.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (eds.), _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pp. 1631–1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.  URL https://aclanthology.org/D13-1170.
  * Subramani et al. (2022) Nishant Subramani, Nivedita Suresh, and Matthew E. Peters.  Extracting latent steering vectors from pretrained language models, 2022.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * Turner et al. (2023) Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid.  Activation addition: Steering language models without optimization, 2023.
  * Wortsman et al. (2022) Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022.
  * Xu et al. (2018) Jingjing Xu, Xu Sun, Qi Zeng, Xiaodong Zhang, Xuancheng Ren, Houfeng Wang, and Wenjie Li.  Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach.  In Iryna Gurevych and Yusuke Miyao (eds.), _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 979–988, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-1090.  URL https://aclanthology.org/P18-1090.
  * Yadav et al. (2023) Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal.  Ties-merging: Resolving interference when merging models, 2023.
  * Yang & Klein (2021) Kevin Yang and Dan Klein.  FUDGE: Controlled text generation with future discriminators.  In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 3511–3535, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.276.  URL https://aclanthology.org/2021.naacl-main.276.
  * Zhang et al. (2023) Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He.  Composing parameter-efficient modules with arithmetic operations, 2023.
  * Zhou et al. (2023) Wangchunshu Zhou, Yuchen Eleanor Jiang, Ethan Wilcox, Ryan Cotterell, and Mrinmaya Sachan.  Controlled text generation with natural language instructions, 2023.

##  Appendix A Appendix

###  A.1 Hyperparameters for fine-tuning

LoRA hyperparameter | Value
---|---
Batch size | 64
Learning rate | 5e-5
LoRA $r$ | $32$
LoRA $\alpha$ | $16$
LoRA dropout | $0.1$
Max sequence length | $128$
Quantization | 4 bit
Table 1: Parameters for LoRA fine-tuning. We use $20$ epochs for fine-tuning
the sentiment attribute models and $1$ epoch for the remaining fine-tuned
models.

###  A.2 Perplexity analysis

Figure 9: Wikitext perplexity of linearly interpolated and extrapolated
models. We report the average perplexity of each model from Figure 2(b) on the
Wikitext test set. For the extrapolated models not shown in Figure 3, the
perplexity increases rapidly.

###  A.3 Multi-dimensional scaling (MDS) analysis of fine-tuned models

We project the weights of the LoRA fine-tuned models, as well as some of the
interpolated models, into two dimensions using multi-dimensional scaling
(MDS). As shown in Figure 10, we find that the interpolating between the
endpoint fine-tuned models generally results in models that are closer to the
base model. This is expected behavior since we would anticipate that the base
model is fairly neutral with respect to all attributes.

Figure 10: Multi-dimensional scaling (MDS) plot for the fine-tuned models and
linear interpolations. This plot shows the 2-dimensional MDS projection of the
fine-tuned models and the models interpolated at intervals of $0.1$. This
corresponds to the models in Figure 2(a).

###  A.4 Additional multi-dimensional spider plots

Figure 11: Effect of $\alpha_{i}$ and $\lambda_{i}$ on attribute scores for
multiple dimensions. In each plot, we report the attribute scores for multiple
$\alpha_{i}$ values when $\alpha_{i}$ is varied for the attribute $i$ in the
column title and kept constant at $1.0$ for the remaining attributes.
$\lambda_{i}$ for the dimension $i$ for which the $\alpha_{i}$ weight is being
varied is listed in the row titles. The mixture weights $\lambda_{j}$ for all
remaining $j$ are set uniformly so that the sum of $\lambda$s weights is $1$
($\lambda_{j}=(1.0-\lambda_{i})/4$). For all of the dimensions, increasing
$\alpha_{i}$ results in an increasing attribute score for that dimension while
having limited effect on the other attribute scores.

###  A.5 Additional multi-dimensional $\lambda$ simplex plots

Figure 12: Effect of $\lambda_{i}$ on interpolation between the sentiment,
politeness, and humor dimensions for $\alpha_{i}=0$. The vertices of the
triangle represent the models with $\alpha_{i}=0$ for each of the three
dimensions.

Figure 13: Effect of $\lambda_{i}$ on interpolation between the sentiment,
politeness, and humor dimensions for $\alpha_{i}=0.5$. The vertices of the
triangle represent the models with $\alpha_{i}=0.5$ for each of the three
dimensions.

Figure 14: Effect of $\lambda_{i}$ on interpolation between the sentiment,
politeness, and humor dimensions for $\alpha_{i}=1$. The vertices of the
triangle represent the models with $\alpha_{i}=1$ for each of the three
dimensions.

Figure 15: Effect of $\lambda_{i}$ on interpolation between the sentiment,
politeness, and formality dimensions for $\alpha_{i}=0$. The vertices of the
triangle represent the models with $\alpha_{i}=0$ for each of the three
dimensions.

Figure 16: Effect of $\lambda_{i}$ on interpolation between the sentiment,
politeness, and formality dimensions for $\alpha_{i}=0.5$. The vertices of the
triangle represent the models with $\alpha_{i}=0.5$ for each of the three
dimensions.

Figure 17: Effect of $\lambda_{i}$ on interpolation between the sentiment,
politeness, and formality dimensions for $\alpha_{i}=1$. The vertices of the
triangle represent the models with $\alpha_{i}=1$ for each of the three
dimensions.

Figure 18: Effect of $\lambda_{i}$ on interpolation between the sentiment,
politeness, and simplicity dimensions for $\alpha_{i}=0$. The vertices of the
triangle represent the models with $\alpha_{i}=0$ for each of the three
dimensions.

Figure 19: Effect of $\lambda_{i}$ on interpolation between the sentiment,
politeness, and simplicity dimensions for $\alpha_{i}=0.5$. The vertices of
the triangle represent the models with $\alpha_{i}=0.5$ for each of the three
dimensions.

Figure 20: Effect of $\lambda_{i}$ on interpolation between the humor,
formality, and simplicity dimensions for $\alpha_{i}=0$. The vertices of the
triangle represent the models with $\alpha_{i}=0$ for each of the three
dimensions.

Figure 21: Effect of $\lambda_{i}$ on interpolation between the humor,
formality, and simplicity dimensions for $\alpha_{i}=0.5$. The vertices of the
triangle represent the models with $\alpha_{i}=0.5$ for each of the three
dimensions.

###  A.6 $L^{2}$ norms between pairs of fine-tuned models

In addition to computing the cosine similarities between LoRA weights in
Figure 8, we also use the average squared $L^{2}$ norm of the difference
between the LoRA updates to analyze the distances between models. We find that
the models fine-tuned on the classes with attribute score of 1 (positive
sentiment, polite, simple, formal, humorous) tend to be closer to the other
models than the models fine-tuned on classes with attribute score 0. We also
find that the polite and impolite LoRA fine-tuned endpoint models are the
farthest from the other models on average. This is consistent with the results
from the MDS plot (Figure 10).

Figure 22: Average pairwise squared $L^{2}$ norms between LoRA layers. The
fine-tuned models trained on the class with attribute score of 1 tend to be
closer to the other models than those trained on the class with attribute
score of 0. The polite and impolite models are the farthest from the other
models.


# Are ELECTRA’s Sentence Embeddings Beyond Repair?
The Case of Semantic Textual Similarity

Ivan Rep  David Dukić  Jan Šnajder
TakeLab, Faculty of Electrical Engineering and Computing, University of Zagreb
<EMAIL_ADDRESS>
{david.dukic<EMAIL_ADDRESS>
###### Abstract

While BERT produces high-quality sentence embeddings, its pre-training
computational cost is a significant drawback. In contrast, ELECTRA delivers a
cost-effective pre-training objective and downstream task performance
improvements, but not as performant sentence embeddings. The community tacitly
stopped utilizing ELECTRA’s sentence embeddings for semantic textual
similarity (STS). We notice a significant drop in performance when using the
ELECTRA discriminator’s last layer in comparison to earlier layers. We explore
this drop and devise a way to repair ELECTRA’s embeddings, proposing a novel
truncated model fine-tuning (TMFT) method. TMFT improves the Spearman
correlation coefficient by over 8 points while increasing parameter efficiency
on the STS benchmark dataset. We extend our analysis to various model sizes
and languages. Further, we discover the surprising efficacy of ELECTRA’s
generator model, which performs on par with BERT, using significantly fewer
parameters and a substantially smaller embedding size. Finally, we observe
further boosts by combining TMFT with a word similarity task or domain
adaptive pre-training.

Are ELECTRA’s Sentence Embeddings Beyond Repair?
The Case of Semantic Textual Similarity

Ivan Rep David Dukić Jan Šnajder TakeLab, Faculty of Electrical Engineering
and Computing, University of Zagreb<EMAIL_ADDRESS>{david.dukic,
<EMAIL_ADDRESS>

##  1 Introduction

Pre-trained language models (PLMs) have been a staple in NLP for years,
leveraging self-supervised objectives to improve representations for
downstream tasks. BERT (Devlin et al., 2019), one of the most widely used PLM,
uses a masked language modeling (MLM) objective for pre-training. The main
drawbacks of MLM are the substantial compute cost due to the low masking rate,
and the gap between the pre-training task and downstream tasks. To address
these defects, Clark et al. (2020) introduce ELECTRA, which substitutes MLM
with replaced token detection (RTD), achieving the same results as BERT but
with four times less compute. RTD uses a generator model that corrupts the
input, while a discriminator model distinguishes between corrupted and
original tokens.

Figure 1: A method for improving sentence embeddings with (2) TMFT on STS. We
apply mean pooling over the layer’s embeddings at index $l$. One of the
combinations can also be added for further improvement: (1a) TMFT on a WS
task, or (1b) DAPT using MLM.

With the rise of large language models and retrieval augmented generation, NLP
tasks such as semantic textual similarity (STS) have become increasingly
important. Aside from achieving high accuracy, STS models prefer small
embeddings for faster inference. To tackle efficiency, Reimers and Gurevych
(2019) present sentence transformers, a framework based on the transformer
model-agnostic bi-encoder architecture. Its effectiveness stems from encoding
text representations independently, followed by pooling. Finally, the
similarity score is calculated using a comparison operation between the two
embedded texts.

While some transformer models demonstrate high performance across all GLUE
tasks, their sentence embeddings may vary in quality. Sentence transformers
most often use PLMs pre-trained with a language modeling objective. In
contrast, models that use an ELECTRA-style pre-training task demonstrate
subpar performance in a bi-encoder setting (Reimers, 2021), but pre-training
such a model is more cost-effective. It is unclear whether this low
performance is due to pre-training and whether the embeddings can be improved.

To address this gap, we present a study of ELECTRA’s sentence embeddings
through layers. We examine the performance discrepancy between BERT and
ELECTRA’s discriminator across various model sizes and languages. We
hypothesize the last few layers of the discriminator are better suited for a
cross-encoder and perform poorly in a bi-encoder setting. Following this, we
devise truncated model fine-tuning (TMFT), shown in Figure 1, which utilizes
the transformer up to layer $l$, followed by pooling and fine-tuning the
extracted sentence embeddings. We show that the discriminator suffers from a
drop in STS performance when fine-tuning the final layer’s embeddings and that
this drop is consistent across all inspected model sizes and languages. When
truncating to certain layers, the embeddings obtained through TMFT
significantly outperform embeddings from ELECTRA discriminator’s last layer on
the STS Benchmark (STSB) (Cer et al., 2017). Moreover, these sentence
embeddings outperform the sentence embeddings of BERT fine-tuned on STSB up to
but not including the eighth layer. Further, we uncover the efficacy of the
ELECTRA generator model, whose variants perform on par with BERTs, while
having a significantly smaller embedding size and substantially fewer
parameters. Finally, we propose two improvements of the basic TMFT method:
prior TMFT on word similarity (WS) and prior domain adaptive pre-training
(DAPT) using MLM (cf. Figure 1).

Our contributions are: (1) a layer-wise analysis of ELECTRA’s sentence
embeddings across various model sizes and languages; (2) a novel TMFT method
for repairing ELECTRA’s embeddings that substantially improves performance on
STS, and uncovering the surprising efficacy of the generator model; (3) two
additional techniques in combination with TMFT for improving ELECTRA’s
embeddings even further.

##  2 Related Work

Reimers and Gurevych (2019) were the first to apply Siamese networks to
transformers. Since then, there has been significant research focused on
improving sentence embeddings. Fine-tuning on a task unrelated to STS has also
been explored, although not in much depth. Reimers and Gurevych (2019) first
fine-tune on a textual entailment task, followed by fine-tuning on STS. On the
other hand, DAPT is a well-known improvement technique that delivers large
gains in downstream task performance (Gururangan et al., 2020).

Similarly, there has been considerable interest in using different layers of a
PLM for producing sentence embeddings. Bommasani et al. (2020) assess the
layer-wise performance of various transformer models and pooling methods on
word similarity datasets. Huang et al. (2021) unveil what combinations of
hidden states perform the best for unsupervised STS, while Jawahar et al.
(2019) extract layer-wise structural characteristics encoded in BERT with
probing tasks. Finally, Ethayarajh (2019) explores how anisotropic the
embeddings of each layer are. Hosseini et al. (2023) combine different layer
representations to improve results, being the most relevant to our study.
Aside from not using ELECTRA, our work diverges as we fine-tune the model from
the input embeddings up to a certain layer and discard the upper part of the
model, while they combine precomputed layer representations by exploiting
dynamic programming.

##  3 Truncated Model Fine-Tuning

The usual approach for obtaining sentence embeddings is applying a pooling
operation over the last layer’s embeddings. We choose mean pooling as it
yields the best results, in line with previous work (Reimers and Gurevych,
2019). The TMFT method we propose for repairing ELECTRA’s embeddings reduces
to taking the $l$-th layer output followed by the pooling operation and fine-
tuning on STS.

Given a sentence $S=(s_{1},s_{2},\ldots,s_{N})$ the encoder outputs a tensor
$E\in\mathbb{R}^{L^{\prime}\times N\times d}$, where $L^{\prime}$ is the
number of layers including the input embeddings, since we treat them as a
layer. Then, we apply mean pooling $p$ over the $l$-th representation,
$p(E,l)=\frac{1}{N}\sum_{n=1}^{N}E_{l,n,:}$, where $E_{l,n,:}$ represents the
output token embedding of layer $l$ for token at position $n$. We apply the
same procedure to both sentences, compare them using cosine similarity, and
backpropagate on the mean squared error from layer $l$ to the model
input.111Fine-tuning based on the final representation and using prior
representations for inference does not yield satisfying results, and a similar
approach has already been tried out (Hosseini et al., 2023).

Furthermore, we propose combining one supervised and one unsupervised method
with TMFT for improved performance: (1) PLM fine-tuning on a WS task or (2)
DAPT on sentences using MLM. The intuition for the former is that the WS task
is crucial for assessing sentence similarity, while the latter builds upon
prior empirical evidence that PLMs based on MLM work well in a bi-encoder
setting (Reimers, 2021).

##  4 Experiments and Results

We evaluate each model on the STSB dataset from the GLUE benchmark (Wang et
al., 2018). We also run the experiments on machine-translated versions of STSB
in Korean, German, and Spanish. We choose these based on language-specific
datasets and PLM availability. We report the Spearman’s correlation
coefficient (SCC) as suggested by Reimers and Gurevych (2019). For WS
experiments, we use the set of all pairs present in the following datasets:
RG-65 (Rubenstein and Goodenough, 1965), WordSim-353 (10., 2002), SimLex-999
(Hill et al., 2015), and SimVerb-3500 (Gerz et al., 2016), with a random
$70$:$15$:$15$ train, validation, and test split, respectively. The
experiments with MLM are done on the merged set of sentences from SNLI (Bowman
et al., 2015) and MultiNLI (Williams et al., 2018) datasets.

We use the following models from Hugging Face Transformers (Wolf et al.,
2020): BERT tiny, mini, small, medium, base, and large. For the ELECTRA
discriminator and generator, we use the small, base, and large models. For
fine-tuning and pre-training, we use the AdamW optimizer Loshchilov and Hutter
(2017) and a learning rate of $2\mathrm{e}{-5}$, with weight decay set to
$1\mathrm{e}{-2}$. We apply gradient clipping to a max norm of $1.0$. For
sentence similarity fine-tuning, we use a batch size of 32 for 10 epochs,
while for WS, the batch size is set to 128 for 50 epochs. For DAPT, we use a
batch size of 32 with eight gradient accumulation steps for 10 epochs with the
masking probability of $0.15$. We report the results averaged across five
different seeds. All reported correlation coefficients correspond to the model
with the highest SCC on the validation set.

###  4.1 ELECTRA

(a) Truncated model FT

(b) WS FT + truncated model FT

(c) MLM + truncated model FT

Figure 2: Test set SCCs on the STSB dataset using TMFT with and without
improvements averaged across five different seeds. Subfigure 2(a) presents the
results using TMFT on STSB, 2(b) depicts TMFT on WS prior to TMFT on STSB, and
2(c) depicts MLM before TMFT on STSB. The shaded area corresponds to the
standard deviation. For detailed SCC values refer to Table 1 in Appendix C.

Applying ELECTRA to downstream tasks is usually done using the discriminator.
We decided not to follow this practice, as ELECTRA’s authors do not give
convincing reasons for discarding the generator. Hence, we conduct experiments
with both the ELECTRA discriminator and generator. The generator is similar to
BERT, except the generator’s input embeddings are tied to the discriminator in
pre-training. For comparison, we use BERT as a baseline, as it is a standard
choice for sentence embeddings and has a size similar to the discriminator.

Figure 2(a) presents the test set SCCs for TMFT applied to
ELECTRA${}_{\text{base}}$ discriminator, ELECTRA${}_{\text{base}}$ generator,
and BERT${}_{\text{base}}$. BERT shows a trend where the SCC almost
monotonously increases as the index of the fine-tuned layer embedding
increases. The same trend is present for the generator.
ELECTRA${}_{\text{base}}$ generator has $74.41$M ($69.07$%) parameters less
than BERT${}_{\text{base}}$, while maintaining comparable performance (cf.
Table 1 in Appendix C). This finding is consistent across all inspected
generator and BERT sizes (cf. Table 2 in Appendix C). The discriminator shows
a different trend, slowly increasing until the ninth layer, after which comes
a severe performance drop. A similar trend holds for experiments with Korean,
German, and Spanish (refer to Appendix A), as well as all inspected model
sizes (cf. Figure 8). The increase in the test SCCs between the ninth and the
last state is $11.28$ points and requires $21.26$M ($19.53$%) fewer
parameters. The best-performing state on the validation set is the third
state, and the increase compared to the last state on the test set is $8.36$,
requiring $63.79$M ($58.58$%) less parameters. Furthermore, we find that the
discriminator’s input embeddings perform better than BERT, while the output
embeddings outperform BERT up to but not including the eighth layer. For all
models, the biggest increase appears in the first layer, which we attribute to
the addition of contextualization.

###  4.2 Further Improvements

The first improvement we propose is TMFT on WS before TMFT on sentence
similarity. The downside of this method is that it requires data labeled for
WS. We consider only using the same layer for both fine-tuning procedures.
Figure 2(b) gives the test set SCCs on STSB for the proposed improvement. We
observe that BERT and the discriminator both benefit from this method, with
ELECTRA consistently outperforming BERT up to but not including the ninth
layer and alleviating the performance drop in the final layer.

Our second improvement is DAPT using MLM before TMFT on STS, regardless of the
model type. We only consider further pre-training using the last layer’s
embeddings. Figure 2(c) presents the test set SCCs on the STSB dataset.
Compared to not using improvements, the performance of almost all
representations has improved across all models, and the drop in the final
layers of the discriminator has been completely mitigated (cf. Figure 2(c) and
Table 1 in Appendix C).

###  4.3 Parameter-Performance Trade-off

Finally, we investigate the parameter-performance trade-off. The results show
the number of parameters the model has when using the best-performing
representation on the validation set and the corresponding test set SCCs (cf.
Figure 3 and Table 2). We consider only TMFT without improvements. Considering
the test set scores and the number of parameters, the best models are:
BERT${}_{\text{tiny}}$, BERT${}_{\text{mini}}$, ELECTRA${}_{\text{small}}$
generator, ELECTRA${}_{\text{base}}$ generator, ELECTRA${}_{\text{large}}$
generator, and BERT${}_{\text{large}}$. The difference in the number of
parameters and layers suggests that the depth of a transformer is essential
for retaining performance on STS. Figure 3 also demonstrates an improvement in
the number of parameters, and in the performance when comparing the last and
best representation for the discriminator.

Figure 3: Comparison of the number of parameters of the model and the test set
SCCs. The depicted models have the highest validation SCC value. All SCC
values are calculated across five different seeds. The figure also includes
the last layer representations for reference if they do not correspond to the
highest validation SCC. ELECTRA${}_{\text{large}}$ discriminator is excluded
as its value is too small ($35.53$). The gray line indicates the Pareto front.
For detailed test set SCC values refer to Table 2.

##  5 Conclusion

We analyze ELECTRA’s sentence embeddings for STS. We compare ELECTRA models to
a BERT baseline on the STSB dataset, presenting a novel truncated model fine-
tuning method, which brings substantial improvements for the discriminator.
The generator model performs on par with the BERT baseline while significantly
reducing the number of parameters and having a smaller embedding size. Future
work may include fine-tuning on a combination of layer representations, DAPT
using representations from prior layers, and conducting experiments on more
datasets for STS.

##  6 Limitations

Our study reveals substantial differences in model performance on a single
task (STS), but sentence embedding models can be used for other tasks as well.
We limited ourselves to a single dataset for STS, four datasets for WS, and
two datasets for MLM, due to computation restrictions. Experiments in Korean,
German, and Spanish were exclusively conducted using the truncated model fine-
tuning method. Another limitation of our work is the way we apply WS fine-
tuning prior to truncated model fine-tuning on STS. We only consider fine-
tuning the same layer’s embeddings for both procedures. Furthermore, in MLM,
prior to truncated model fine-tuning on STS, we only use the last layer’s
embeddings, as opposed to using previous layers. For all our experiments, we
fixed the hyperparameters. Finally, our study covers two families of models
exclusively: BERT and ELECTRA.

## References

  * 10\. (2002) 2002\.  Placing search in context: the concept revisited.  _ACM Trans. Inf. Syst._ , 20(1):116–131.
  * Bommasani et al. (2020) Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020.  Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4758–4781, Online. Association for Computational Linguistics.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015.  A large annotated corpus for learning natural language inference.  _CoRR_ , abs/1508.05326.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ethayarajh (2019) Kawin Ethayarajh. 2019.  How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 55–65, Hong Kong, China. Association for Computational Linguistics.
  * Gerz et al. (2016) Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016.  SimVerb-3500: A large-scale evaluation set of verb similarity.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2173–2182, Austin, Texas. Association for Computational Linguistics.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015.  SimLex-999: Evaluating semantic models with (genuine) similarity estimation.  _Computational Linguistics_ , 41(4):665–695.
  * Hosseini et al. (2023) MohammadSaleh Hosseini, Munawara Munia, and Latifur Khan. 2023.  BERT has more to offer: BERT layers combination yields better sentence embeddings.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 15419–15431, Singapore. Association for Computational Linguistics.
  * Huang et al. (2021) Junjie Huang, Duyu Tang, Wanjun Zhong, Shuai Lu, Linjun Shou, Ming Gong, Daxin Jiang, and Nan Duan. 2021.  WhiteningBERT: An easy unsupervised sentence embedding approach.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 238–244, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Reimers (2021) Nils Reimers. 2021.  Training state-of-the-art sentence embeddings models.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Rubenstein and Goodenough (1965) Herbert Rubenstein and John B. Goodenough. 1965.  Contextual correlates of synonymy.  _Commun. ACM_ , 8(10):627–633.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.

##  Appendix A Performance for Different Languages

Figure 4: Test set SCCs on Korean STSB across all layers. Figure 5: Test set
SCCs on German STSB across all layers. Figure 6: Test set SCCs on Spanish STSB
across all layers.

##  Appendix B Performance for Different Model Sizes

Figure 7: Test set SCCs on STSB across various ELECTRA generator model sizes.
Figure 8: Test set SCCs on STSB across various ELECTRA discriminator model
sizes. Figure 9: Test set SCCs on STSB across various BERT model sizes.

##  Appendix C Truncated Model Fine-Tuning Results on English

###  C.1 Truncated Model Fine-Tuning and Improvements

Model Layer Valid Test TMFT BERT${}_{\text{base}}$ 12 86.04/85.94 82.76/83.01
ELECTRA${}_{\text{D\>base}}$ 3 82.14/82.18 75.29/76.96
ELECTRA${}_{\text{G\>base}}$ 12 86.60/86.36 82.54/82.48 TMFT WS + TMFT STS
BERT${}_{\text{base}}$ 12 85.74/85.74 83.92/84.11 ELECTRA${}_{\text{D\>base}}$
3 82.59/82.45 76.87/77.40 ELECTRA${}_{\text{G\>base}}$ 12 86.47/86.20
82.58/82.69 MLM + TMFT STS BERT${}_{\text{base}}$ 12 85.89/85.67 82.70/82.64
ELECTRA${}_{\text{D\>base}}$ 7 84.04/83.72 79.93/79.92
ELECTRA${}_{\text{G\>base}}$ 11 85.55/85.21 80.99/80.91

Table 1: Comparison of models that have the highest validation set SCC using
TMFT. The reported scores are the SCCs and the Pearson correlation
coefficients multiplied by 100.

###  C.2 Overview of Best-Performing Models Using TMFT

Model | Layer | # Params (M) | Valid | Test
---|---|---|---|---
BERT${}_{\text{tiny}}$ | 2 | 4.37 | 78.23/77.58 | 69.71/70.57
BERT${}_{\text{mini}}$ | 4 | 11.10 | 83.10/82.47 | 75.64/76.36
BERT${}_{\text{small}}$ | 4 | 28.50 | 85.12/84.92 | 79.32/79.68
BERT${}_{\text{medium}}$ | 8 | 41.11 | 85.72/85.39 | 80.84/81.10
BERT${}_{\text{base}}$ | 12 | 107.72 | 86.04/85.94 | 82.76/83.01
BERT${}_{\text{large}}$ | 24 | 332.53 |  88.27/88.25 |  85.58/85.75
ELECTRA${}_{\text{D\>small}}$ | 1 | 4.76 | 79.72/79.26 | 68.82/69.63
ELECTRA${}_{\text{D\>base}}$ | 3 | 45.10 | 82.14/82.18 | 75.29/76.96
ELECTRA${}_{\text{D\>large}}$ | 12 | 182.94 |  84.36/84.40 |  80.54/80.71
ELECTRA${}_{\text{D\>small\>last}}$ | 12 | 13.45 | 74.02/73.15 | 66.54/67.00
ELECTRA${}_{\text{D\>base\>last}}$ | 12 | 108.89 | 72.32/71.56 | 66.93/67.35
ELECTRA${}_{\text{D\>large\>last}}$ | 24 | 334.09 | 35.96/32.66 | 35.53/33.09
ELECTRA${}_{\text{G\>small}}$ | 12 | 13.45 | 84.67/84.15 | 81.79/81.16
ELECTRA${}_{\text{G\>base}}$ | 12 | 33.31 | 86.60/86.36 | 82.54/82.48
ELECTRA${}_{\text{G\>large}}$ | 24 | 50.74 |  87.14/86.72 |  84.61/84.47

Table 2: An overview of the test set SCCs and Pearson correlation coefficients
multiplied by 100 for various model families and model sizes. The reported
test set values correspond to the model with the highest validation set SCC.
All correlation coefficient values are averaged over five seeds. For parameter
calculation, the pooler layer is excluded as we do not use it.

##  Appendix D Reproducibility

We conducted our experiments on an AMD Ryzen Threadripper 3970X 32-Core
Processor and a single RTX 3090 GPU with 24GB of RAM. Running the experiments
took around a week and a half (250 GPU hours).