# FreeLM: Fine-Tuning-Free Language Model

Xiang Li1†, Xin Jiang1†, Xuying Meng2, Aixin Sun3, Yequan Wang1$*$
1Beijing Academy of Artificial Intelligence, Beijing, China
2Institute of Computing Technology, Chinese Academy of Sciences, Beijing,
China
3School of Computer Science and Engineering, Nanyang Technological University,
Singapore
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Pre-trained language models (PLMs) have achieved remarkable success in NLP
tasks. Despite the great success, mainstream solutions largely follow the pre-
training then fine-tuning paradigm, which brings in both high deployment costs
and low training efficiency. Nevertheless, fine-tuning on a specific task is
essential because PLMs are only pre-trained with language signal from large
raw data. In this paper, we propose a novel fine-tuning-free strategy for
language models, to consider both language signal and teacher signal. Teacher
signal is an abstraction of a battery of downstream tasks, provided in a
unified proposition format. Trained with both language and strong task-aware
teacher signals in an interactive manner, our FreeLM model demonstrates strong
generalization and robustness. FreeLM outperforms large models e.g., GPT-3 and
InstructGPT, on a range of language understanding tasks in experiments. FreeLM
is much smaller with 0.3B parameters, compared to 175B in these models.

11footnotetext: Corresponding author22footnotetext: Indicates equal
contribution

##  1 Introduction

Pre-trained language models (PLMs), exampled by BERT Devlin et al. (2019), the
GPT series Radford et al. (2018, 2019); Brown et al. (2020) and their variants
Liu et al. (2019); Joshi et al. (2020); Sun et al. (2019); Raffel et al.
(2020), have been widely applied in various language processing tasks and
achieved remarkable success.

Figure 1: An overview of small models, pre-trained language models, and our
proposed fine-tuning-free language model.

Despite their great success, the pre-training then fine-tuning paradigm Devlin
et al. (2019); Radford et al. (2019) brings in very large costs for training
and deployment in enterprise-level applications. Even large companies are very
careful in using billion-parameter PLMs online Sanh et al. (2019), and remain
showing high interest in small models. Before the era of PLMs, small models
are trained for specific tasks. For some tasks, these task-dedicated models
may perform comparably to or even better than large PLMs with fine-tuning
Grinsztajn et al. (2022). Both the training and deployment costs are
manageable since the task-specific datasets and models are typically much
smaller. Nevertheless, additional efforts are essential to design task-
specific models, one for each task. Illustrated in Figure 1, pre-trained with
much larger datasets, PLMs gain high generalization, and can be fine-tuned to
diverse downstream tasks for language understanding and generation. The effort
of designing task-specific small models is now replaced by the huge costs of
pre-training on large datasets and the high deployment costs.

To reduce deployment costs, zero-shot e.g., GPT-3 Brown et al. (2020) and few-
shot models have been investigated. Their performance, particularly on
understanding tasks, remains unsatisfactory. One of the reasons is that the
PLMs are trained with only language signal which is not task-aware, i.e., the
training objective of PLMs does not well align with the task objective.
Recently, instruction-tuning-based models, i.e., InstructGPT Ouyang et al.
(2022) and FLAN Wei et al. (2022), further improves zero-shot performance.
Their main design transfers everything into language text, then uses self-
supervised language signal to train the model. In this way, the language
signal in their training becomes more task-aware and relatively stronger.
However, to achieve the best performance on specific tasks, adaptation remains
necessary.

The idea of enhancing task awareness in training motivates us to consider a
reasonably sized model that is capable of generalizing to a good range of pre-
defined tasks. The ultimate goal is to achieve good performance on all these
pre-defined tasks (and potentially unseen tasks) with low training and
deployment costs. We believe this is an ideal setting for an enterprise to
handle all their own specific tasks with a single model trained on their own
data, without additional fine-tuning.

The key challenge then becomes how to unify the pre-defined tasks as a strong
task-aware supervision signal for model training. To this end, we propose
FreeLM, a fine-tuning-free language model. Illustrated in Figure 1, FreeLM is
trained with both language signal and teacher signal in an iterative way.
During training, in different iterations, the model alternately learns (i) the
language signal from large raw data, and (ii) the teacher signal from unified
data prepared based on all pre-defined tasks. Benefiting from the interplay
between both signals, FreeLM is able to handle all pre-defined tasks with high
accuracy, and handle unseen tasks well as a typical pre-trained language model
does.

In addition to the fine-tuning-free strategy, a major contribution is that we
make the very first attempt to unify a good range of language tasks through a
unified data format, called proposition format. Datasets constructed for all
pre-defined tasks are then mapped to this unified proposition format, and
become the unified data in Figure 1. Through the proposition format, we
provide proposition correctness judgment to help the language model know the
facts. The unified format also implicitly facilitates mutual improvements
among these tasks, which further contributes to better generalization and
robustness.

To the best of our knowledge, this is the first attempt to propose an
effective fine-tuning-free strategy for large language model training. We also
demonstrate how to unify multiple tasks through a unified format to make the
PLM more task-aware through strong supervision signals. Through experiments,
we show that FreeLM greatly outperforms state-of-the-art models on language
understanding tasks, compared to GPT-2, 175B GPT-3 and InstructGPT. We also
show that the generation ability of FreeLM is even better than GPT-2. FreeLM
achieves good performance on unseen tasks, and has excellent generalization
and robustness.

##  2 Related Work

In this work, we focus on both language generation and language understanding
tasks. Thus, we only consider unidirectional Transformers like GPT as basic
models, and do not consider masked language models (e.g., BERT) or encoder-
decoder language models (e.g., T5). Next, we brief auto-regressive language
models and instruct-tuning-based models.

###  2.1 Auto-Regressive Language Models

Auto-regressive language models are trained to predict the next token based on
all previous tokens. The unidirectional nature empowers these models with
language generation capability. The impacts of model scale and model structure
are the two key research directions.

On model scale, the most typical and also the most influential auto-regressive
LMs are the GPT series, i.e., GPT-1 Radford et al. (2018), GPT-2 Radford et
al. (2019), and GPT-3 Brown et al. (2020). In particular, the success of GPT-3
has made researchers realize that the violent aesthetics of the model scale
and large raw data can have such a good generation performance. To make the
models bigger, even methods like MoE Nie et al. (2021) are proposed. However,
large scale brings in large costs and many challenges, particularly the high
costs of fine-tuning on downstream tasks.

To improve the model structure, GLM Du et al. (2022) is designed to utilize
the autoregressive blank infilling. Transformer-XL Dai et al. (2019) tries to
solve the longer-term dependency, and its variant XL-Net Yang et al. (2019)
hopes to learn bidirectional contexts. However, these models do not perform
better than GPT-3 as the models get larger. The vital reason remains how to
effectively and efficiently utilize data.

###  2.2 Instruct-Tuning Based Models

A larger model size does not mean that it can produce output that better meets
user expectations Ouyang et al. (2022). One solution is to fine-tune large
language models based on a wide range of tasks, human feedback, and so on. The
typical and most representative model is InstructGPT Ouyang et al. (2022).
Through reinforcement learning, data from tasks, and human feedback,
InstructGPT achieves excellent performance on both language understanding and
generation tasks. Experiments even show that InstructGPT can align with humans
compared with original GPT-3. EFL Wang et al. (2021), designed for few-shot
learning, reformulates potential NLP tasks into an entailment task based on
RoBERTa-large. Entailment task is hard to handle well more types of tasks,
while our unified proposition is more generic to fit more tasks. In addition,
EFL suffers from degradation in generation abilities.

In our model design, we aim to achieve better results on both understanding
and generation tasks without fine-tuning. InstructGPT adopts a method like
“all in language”. EFL gives up the generation ability. To our understanding,
these models do not effectively utilize the teacher supervision signal. In
this paper, we design an iterative training strategy on both language raw data
and task-aware data. The key idea here is to teach the model to be task-aware
data, while not forgetting its role of a language model, i.e., to model
language.

##  3 Task Unification

Our goal is to train a task-aware language model which learns from language as
a typical PLM does, and also learns from a good number of task-specific
datasets. Shown in Figure 1, FreeLM learns from both language data and unified
data, where the latter is the result of unifying pre-defined tasks.

#### Language Data.

For language data, the choice is relatively straightforward. We adopt
OpenWebText Gokaslan and Cohen (2019), an open-source replication of the
WebText Radford et al. (2019) dataset proposed by OpenAI. It is built by
scraping all outbound links from the social media platform Reddit, which
received at least $3$ karmas. We use the same settings as GPT-2 on this
dataset.

#### Unified Data in Proposition Format.

Our key motivation is to enable enterprises to handle all their own specific
tasks with a single model trained on their own data, without additional fine-
tuning. However, it is challenging to access and conduct experiments on
proprietary data. Even if that is feasible, our results will be hard to be
benchmarked against the current state-of-the-art. Without loss of generality,
we choose to unify seven well-defined and popular language tasks: 1. question
answering, 2. paraphrasing, 3\. topic classification, 4. story cloze, 5.
sentiment classification, 6. natural language inference, and 7. linguistic
acceptability.

We unify these seven tasks by transforming them into a “proposition
correctness judgment” task, to judge whether a proposition is true.
Specifically, for each task, we design some candidate templates, then use the
designed rule to transfer each instance into a proposition format:

“[tsk] task name [tsk] instance text(s) [sep] dynamic prompt [cls]”
---

The task name follows the original pre-define task name before unification
like “question answering”. Because each task has its own characteristic, hence
we use dynamic prompt to fit them well considering the ground-truth label.

Figure 2: A template example for the topic classification task. It also
describes the transform rule for task unification. Figure 3: The list of 30
datasets for 7 tasks; numbers in parentheses indicate dataset size.

We take the topic classification task as an example to explain the mapping,
also illustrated in Figure 2. We convert the class label into a natural
language description based on a template selected from a pre-designed template
pool. In Figure 2, Template 1 is chosen among the $N$ templates. Then we could
build a candidate proposition statement by appending the natural language
description to the original input text. The natural language description is
the class labels in this example task.

In our implementation, we utilize 30 datasets of the seven chosen tasks (see
Figure 3). For each dataset, we design at most $10$ templates to build the
proposition. Each item will randomly choose one template when building unified
data. According to the chosen template, the original input context is
concatenated with the label description to form a proposition. With the
unified formatted proposition as input, our FreeLM is responsible for giving a
true-or-false judgment.

FreeLM designs task unification by unified proposition format, which brings a
good generalization effect. In other words, FreeLM only has one proposition
correctness judgment task (except language task), which endows FreeLM with the
ability to handle almost all NLU tasks. As a comparison, multi-task needs
special design for each task to fit the multi-task model. More importantly,
multi-task could hardly gain generalization ability except the designed tasks.

##  4 FreeLM

###  4.1 Architecture and Iterative Training

Figure 4 shows the architecture of the proposed FreeLM. The upper part of the
figure shows training with language data, similar to the mainstream generative
language models. The lower part shows training with task-aware unified data,
through making correct judgments on unified propositions.

To train the model with both language and unified data, a straightforward
choice is to first train the model fully on language data, then fully on
unified data, similar to pre-train then fine-tuning. However, this strategy
would lead the model to focus too much on tasks, and reduce its language
capability and generalization. In our design, language signal and teacher
signal are used for training in an iterative manner, analogous to “first read,
then exercise”. Next, we detail the two training stages, which take turns to
train the model.

Figure 4: (a) The language stage utilizes the standard language training
method, and (b) the teacher stage uses the unified proposition signal.

#### Language Iterative Stage.

To keep the generation ability of language models, we choose the auto-
regressive language model, more specifically GPT-2, as our base model in
FreeLM. As shown in Figure 4(a), language signal guides the model to predict
the next token with the previous token(s). Given a text consists of $N$ tokens
$\mathcal{T}=\\{t_{1},\ldots,t_{N}\\}$, FreeLM computes the probability
$P_{L}$ of token $t_{k}$ with the previous $k-1$ tokens:

$P_{L}(t_{k}|t_{1},\dots,t_{k-1})=\operatorname{softmax}(W_{v}h_{t_{k-1}})$
(1)

Here, $h_{t_{k-1}}$ denotes the representation encoded by Transformer with the
previous tokens $\\{t_{1},\ldots,t_{k-1}\\}$ as input. $W_{v}$ represents the
learnable parameters.

#### Teacher Iterative Stage.

In the proposed fine-tuning-free structure, teacher signal aims to guide the
model to learn task-oriented knowledge. After unifying all tasks to
proposition correctness judgment, the model is expected to make correct
judgements about the propositions. Let $u=(T_{p},y)$ be a pair where $T_{p}$
refers to the proposition statement, and $y$ is the label of this proposition.
When the proposition is right, $y$ is True, otherwise False.

Figure 4(b) shows the mechanism of teacher signal from unified data. According
to the designed transfer rule (see Section 3), each instance in the unified
data is appended with a dedicated token “[cls]”. Given an instance, we can
calculate the representation $h_{[\text{cls}]}$ of token “[cls]” by the shared
Transformer of the language model. The proposition distribution
$\mathcal{P}_{P}$ is computed by a softmax classifier based on the learned
representation $h_{[\text{cls}]}$:

$\mathcal{P}_{P}(y|T_{p})=\operatorname{softmax}(W_{p}h_{[cls]}+b_{p}),$ (2)

where $W_{p}$ and $b_{p}$ are learnable parameters.

###  4.2 Training Objective

The training objective design has two parts. One is to maximize the likelihood
of predicted tokens, defined in Equation 3. The other is to minimize the
cross-entropy of judging proposition correctness, defined in Equation 4, where
$y$ is the ground truth of the given $T_{p}$.

$\displaystyle J(\theta)$ $\displaystyle=\sum\sum_{k}\log
P_{L}(t_{k}|t_{1},\dots,t_{k-1})$ (3) $\displaystyle U(\theta)$
$\displaystyle=\sum\text{cross-
entropy}\left(\mathcal{P}_{P}(y|T_{p}),y\right)$ (4)

Note that the two objectives are alternately performed in each iteration,
along with the two types of training data.

##  5 Experiments

Model | CoLA | SST-2 | MRPC | MRPC | QQP | QQP | MNLI | MNLI | QNLI | RTE | Average
---|---|---|---|---|---|---|---|---|---|---|---
Mat. Corr | Acc. | F1 | Acc. | F1 | Acc. | Mat. Acc. | M.m. Acc. | Acc. | Acc. | –
Fine-Tuning
GPT-2 w. F.T. | 56.52 | 94.95 | 87.93 | 82.84 | 87.91 | 90.83 | 85.87 | 85.83 | 91.03 | 68.23 | 83.19
Few-Shot & Zero-Shot
GPT-2 w. Z.S. | -3.26 | 51.83 | 81.22 | 68.38 | 52.88 | 36.19 | 35.60 | 34.30 | 46.40 | 51.62 | 45.52
GPT-2 w. F.S. | 0.60 | 63.30 | 81.34 | 68.62 | 44.46 | 46.80 | 31.60 | 35.00 | 48.60 | 49.09 | 46.94
GPT-3 w. Z.S. | 17.28 | 52.98 | 65.05 | 53.92 | 4.61 | 62.80 | 36.80 | 39.50 | 60.90 | 64.98 | 45.88
GPT-3 w. F.S. | -4.45 | 93.34 | 70.92 | 59.80 | 6.31 | 64.40 | 44.10 | 46.60 | 53.80 | 65.70 | 50.05
InstructGPT w. Z.S. | 63.81 | 92.08 | 83.75 | 77.94 | 64.20 | 78.70 | 46.20 | 48.90 | 71.89 | 86.64 | 71.41
InstructGPT w. F.S. | 56.72 | 95.18 | 77.41 | 70.83 | 72.98 | 81.20 | 71.70 | 72.89 | 81.30 | 82.67 | 76.29
FreeLM | 53.00 | 93.81 | 87.21 | 82.11 | 85.47 | 89.60 | 81.80 | 79.80 | 88.30 | 71.12 | 81.22

Table 1: Comparison with GPT-2, GPT-3 and InstructGPT on GLUE validation set.
F.T., Z.S. and F.S. denote fine-tuning, zero-shot and few-shot, repectively.
Best results are in bold face (without considering GPT-2 w. F.T.).

We evaluate FreeLM from two perspectives: language understanding performance,
and language generation performance. Then we perform a detailed analysis of
the impact of iterative processes, the impact of special tokens in the
proposition format, and the generalization ability on unseen data.

###  5.1 Training Datasets

As indicated in Section 3, for raw language data, we adopt the OpenWebText
Gokaslan and Cohen (2019). The unified data are composed with 30 datasets,
listed in Figure 3, from the seven tasks. Training sets are collected for all
these datasets, except Story Cloze 2016 & 2018, whose validation sets are
collected.

For each instance in unified data, we construct a positive sample (i.e., true
proposition) with the gold label according to the build method.
Simultaneously, we randomly choose another label to construct a false
proposition as a negative sample. In training set building, only one template
would be randomly chosen for each original instance. The maximum number of
training set proposition samples per dataset is set to $100K$. The final
unified training dataset contains about $1.76M$ samples.

###  5.2 Evaluation on Language Understanding

A key motivation of FreeLM is the understanding ability without fine-tuning.
For understanding ability, we evaluate FreeLM against strong baselines,
including 175B GPT-3, InstructGPT,111We use text-davinci-003 version through
OpenAI API. and GPT-2, under the settings of zero-shot and few-shot on these
baseline models.

We mainly evaluate FreeLM in fine-tuning-free scenarios. That is, our FreeLM
does not conduct further task-oriented fine-tuning after training. Also, as an
upper bound on the understanding ability with the same scale, we show the
results of GPT-2 fine-tuning.

#### Evaluation Metric.

We choose General Language Understanding Evaluation (GLUE) as the benchmark,
which consists of typical natural language understanding tasks, including
natural language inference (i.e., AX, MNLI Williams et al. (2018), RTE
Bentivogli et al. (2009), QNLI Wang et al. (2018), and WNLI Levesque et al.
(2012)), sentiment classification (i.e., SST-2 Socher et al. (2013) and STS-B
Cer et al. (2017)), paraphrase (i.e., MRPC Dolan and Brockett (2005) and QQP)
and linguistic acceptability (i.e., CoLA). Validation sets of each dataset are
used for evaluation. WNLI is excluded from the benchmark following the
previous work Devlin et al. (2019) because the GLUE
webpage222https://gluebenchmark.com/faq notes that there are issues with the
construction of this dataset. In addition, Ax is excluded due to the lack of
the validation set. As the regression task (i.e., STS-B) is not suitable for
FreeLM, we also exclude it from the benchmark.

We use the remaining seven tasks from the GLUE benchmark to evaluate. For
experimental efficiency and cost effectiveness on API, when evaluating GPT-3
and InstructGPT, we use the validation set of GLUE for evaluation, and sample
$1k$ instances for each task as the final evaluation set. All results are
compared fairly in the same setting.

#### Results against Baselines.

The GLUE results are reported in Table 1. On both the average metric and 7 out
of 10 evaluation metrics, FreeLM outperforms all baselines, revealing its
strong understanding ability. In particular, our FreeLM gains about $5$ points
improvement on average performance compared with the powerful InstructGPT. We
also observe that MRPC, QQP, MNLI and QNLI tasks obtain $7$ or more points
improvement in absolute accuracy. More importantly, the scale of our FreeLM is
only 0.3B. Even compared to the much larger GPT-3 with 175B parameters, FreeLM
achieves the state-of-the-art in the downstream tasks without fine-tuning,
thanks to the strong task-aware signal in training. Among baseline models,
InstructGPT has significant advantages over GPT-2 and GPT-3. Our results also
show that InstructGPT is an extremely strong model.

As reported in Table 1, our FreeLM nearly matches the performance of GPT-2
with fine-tuning. As GPT-2 is fine-tuned individually on each dataset, the
comparison in essence is between one FreeLM and $7$ dedicated GPT-2(s). In
this sense, GPT-2 with fine-tuning could represent an approximate upper bound
on 0.3B scale. InstructGPT, the strongest generative PLM, achieves the second-
best performance. As a comparison, FreeLM uses classification-based approach
while instruction tuning uses generation-based approach, which results in our
model being more effective. More importantly, the task unification (see
Section 3) by one unified proposition format brings a good generalization
effect. An interesting point is that FreeLM performs significantly worse than
InstructGPT, but significantly better than GPT-2 with fine-tuning, on the
accuracy of RTE. This result also suggests that the ability of language models
could be positively related to the scale.

Not directly reflected in Table 1, through experiments, we also notice that
FreeLM is robust and insensitive for inference. Interestingly, we find that
the results of the GPT family models predicted by few-shot or zero-shot are
sensitive to the settings of templates, parameters, and post-processing
methods. For example, we measure GPT-2 on the MRPC dataset with a zero-shot
setting. The results of the vocabulary probability distribution $top\\_k=50$
are used to predict the classification. If we calculate the average sum of the
probability ratios of positive and negative words, the accuracy is $33.08$.
However, if we get the word by the highest probability value, the accuracy is
more than doubled to $68.13$. The gap is so large that downstream tasks are
hard to use reliably. On the other hand, FreeLM is robust without the need of
expert knowledge or parameter tuning.

###  5.3 Evaluation on Language Generation

As aforementioned, while FreeLM gains better understanding ability, we hope
that the model retains its language generation ability, similar to GPT-2, for
learning from the same language data. Hence, we evaluate the generation
performance of FreeLM against GPT-2. The evaluation is conducted on three
public datasets, i.e., LAMBADA Paperno et al. (2016), WikiText2, and PTB. We
use perplexity (PPL) to evaluate the generation performance, following common
practice.

Model | LAMBADA | WikiText2 | PTB
---|---|---|---
FreeLM | 38.44 | 33.66 | 28.47
GPT-2 | 35.31 | 30.23 | 26.62
Table 2: Perplexity on FreeLM and GPT-2.

Case A
---
INPUT: Rick grew up in a troubled household. He never found good support in
family, and turned to gangs. It wasn’t long before Rick got shot in a robbery.
The incident caused him to turn a new leaf.
GPT-2: "I was a gang member, but I didn’t know it at the time," Rick said. "I had no idea what was going on. I just wanted to get away from it. That’s how I ended up on the streets. And that’s why I’m here today." |  FreeLM: "When I got out of prison, I was a different man," he says. "I was able to get my life back on track. I went back to school, got a job, started a family. And that’s what I’ve been doing ever since."
Case B
INPUT: Tony was happy to be going on his first cruise. He arrived at the docks
and was ready with all his luggage. When he walked around the corner he
couldn’t believe the size of it. He boarded and the ship left the docks.
GPT-2: When he got to his room he was surprised to see that the room was bigger than he thought it would be. There was a bed, a desk, and a TV. … |  FreeLM: "This is the first time I’ve ever been on a cruise ship," he said. "I’ve never seen anything like this before."
Case C
INPUT: Jasper has been avoiding getting a real job for months. He has hated
all the previous office jobs that he had. One day he sees a job opening at his
local butcher. He applies successfully for the job at the butcher.
GPT-2: "He said, ’I don’t want to hire you because you’re too old,’" Jasper says. "And I was like, I’m not old. I’ve been doing this for a long time, and I know what it’s like to be rejected. … |  FreeLM: "It’s a great job,” he says. “I love it. I love the people here. They treat me like a king. The people are so nice. It makes me feel like I can do anything I want to do in this world. …

Table 3: Case study on generation ability. Consistency and inconsistency
expressions are in blue and red colors.

#### Results against Baselines.

Table 2 reports the perplexity (PPL) of FreeLM and GPT-2. The PPL of FreeLM is
slightly higher than GPT-2. The minor gap could reflect that our model nearly
reaches GPT-2 in terms of generation. There are studies suggesting that PPL
does not fully reflect the generation ability of language models Wang et al.
(2022). Nevertheless, there are no good alternative metrics.

Since generation evaluation does not have a gold standard, we provide a case
study on a story cloze task. The input texts come from the StoryCloze dataset
Mostafazadeh et al. (2016). In order to make the comparison objective and
fair, we train a new FreeLM model by removing this dataset from the unified
data. Top-k random sampling with $k=50$ is used for generation, width of $5$
for beam search, and the maximum generated length is $256$. Table 3 shows
three generated samples.

First, we find that the readability of the text generated by FreeLM is good by
manual sampling inspection. There are no obvious grammatical errors, which
shows that FreeLM and GPT-2 perform at the same level in the grammatical
aspect. Second, Table 3 shows that there are many context inconsistency
phenomena in the texts generated by GPT-2. Our FreeLM could generate more
consistent text with the same texts as input. To clearly show the correct
logical association, we use the text in blue to highlight the relation.
Meanwhile, we use the text in red to warn the logical errors. Cases A and B
show that our FreeLM is capable of generating more consistent texts.
Interestingly, although there is no obvious grammatical error in Case B, the
generated text by GPT-2 almost has no direct logical relationship with the
input text. That is, it is not a high-quality generated result.

In simple words, our experiments show that the proposed FreeLM achieves the
same level of PPL as GPT-2. FreeLM could generate more consistent text. We
believe that if we train our model at a larger scale, better generation
ability can be achieved.

###  5.4 Detailed Analysis

We conduct three sets of experiments to study (i) the impact of the iterative
training, (ii) the proposition format, and (iii) the generalization ability on
unseen tasks.

Model | CoLA | SST-2 | MRPC | MRPC | QQP | QQP | MNLI | MNLI | QNLI | RTE | Average
---|---|---|---|---|---|---|---|---|---|---|---
Mat. Corr | Acc. | F1 | Acc. | F1 | Acc. | Mat. Acc | M.m. Acc | Acc. | Acc. | –
FreeLM | 52.53 | 93.81 | 87.21 | 82.11 | 86.32 | 89.82 | 80.70 | 80.93 | 88.61 | 71.12 | 81.32
w.o. L.S. | 40.14 | 92.43 | 85.71 | 79.17 | 85.66 | 89.11 | 77.56 | 78.57 | 87.28 | 71.48 | 78.71

Table 4: The effect of language signal for understanding, where L.S. refers to
language signal.

Model | CoLA | SST-2 | MRPC | MRPC | QQP | QQP | MNLI | MNLI | QNLI | RTE | Average
---|---|---|---|---|---|---|---|---|---|---|---
Mat. Corr | Acc. | F1 | Acc. | F1 | Acc. | Mat. Acc | M.m. Acc | Acc. | Acc. | –
FreeLM | 52.53 | 93.81 | 87.21 | 82.11 | 86.32 | 89.82 | 80.70 | 80.93 | 88.61 | 71.12 | 81.32
w.o. prefix | 48.15 | 92.78 | 86.76 | 81.37 | 86.06 | 89.53 | 80.23 | 80.32 | 88.76 | 71.84 | 80.58

Table 5: The effect of task prefix for understanding. Model | MRPC | MRPC | RTE | COPA | CB | Average
---|---|---|---|---|---|---
F1 | Acc. | Acc. | Acc. | Acc. | –
FreeLM | 87.21 | 82.11 | 71.12 | 73.00 | 73.21 | 77.33
$\text{FreeLM}_{\text{U}}$ | 81.22 | 72.06 | 66.79 | 68.00 | 68.64 | 71.34
GPT-2 w. F.T. | 87.93 | 82.84 | 68.23 | 54.00 | 73.21 | 73.24
Table 6: FreeLM vs. GPT-2 fine-tuning on unseen data.

#### The Iterative Training.

If we remove the teacher signal, FreeLM will degenerate into a general
language model. Based on the results in Section 5.2 and 5.3, we conclude that
the discarding of the teacher signal will lead to a reduction in the
performance of understanding tasks. At the opposite extreme, we remove the
language signal. The model could then only rely on the objective of
proposition correctness judgment for training. The comparison is under the
same training parameters and the same number of steps. Table 4 shows the
results compared to standard FreeLM. After removing language signal, almost
all evaluation metrics show a significant drop. Interestingly, the PPL of
WikiText2 could increase to $8,000$, even $9,000$, compared to a regular scale
$30$ from GPT-2. As a result, the generation ability is damaged seriously.

Figure 5: Effect of language signal on proposition loss.

We have shown that language signal could help FreeLM learn better. Also, to
evaluate whether FreeLM can learn faster, we give the loss performance during
training in Figure 5. We observe that after $20k$ steps, the proposition loss
of FreeLM decreases faster. It shows that the language signal has a positive
effect on task-oriented learning. In conclusion, language signal and teacher
signal could promote each other under the iterative training method.

#### The Proposition Format.

Task prefix, such as “[tsk] Topic Classification [tsk]”, could guide the model
to narrow down the search space. In the implementation, the set of task prefix
is {Linguistic Acceptability, Topic Classification, Story Cloze, Sentiment
Classification, Question Answering, Paraphrase, Natural Language Inference}.
Since the parameters of FreeLM are small, we utilize this kind of task prefix
to guide the model when meeting task-oriented understanding. To compare, we
train another FreeLM without task prefix. Other settings are the same as
FreeLM. Table 5 gives the comparison results.

We observe that the performance of most tasks is at a similar level. This
phenomenon could prove that FreeLM does not significantly rely on task prefix.
We believe that when the size of FreeLM exceeds $1B$, task prefix will have
little effect.

#### Generalization to Unseen Dataset.

To evaluate the generalization ability of FreeLM, we train a new model
$\text{FreeLM}_{\text{U}}$ by removing 4 datasets from unified data. They are
MRPC, RTE, COPA Gordon et al. (2012) and CB De Marneffe et al. (2019). Other
settings are exactly the same. Table 6 gives the results compared to standard
FreeLM.

We observe that the average score of $\text{FreeLM}_{\text{U}}$ nearly matches
the performance of GPT-2 with fine-tuning. More interestingly, the accuracy of
the $\text{FreeLM}_{\text{U}}$ on the COPA dataset has increased by $14$
points than GPT-2 with fine-tuning. Compared to the standard FreeLM, the
average drop is about $6$ points. It is an inspiring result because it reveals
that our FreeLM has a good generalization ability. We believe the reason is
that we unify various tasks into one proposition correctness judgment task.
The phenomenon also reminds us that the design of teacher signal is a key to
training a better language model.

##  6 Conclusion

With the aim of reducing costs in training and deployment, we design a novel
fine-tuning-free language model. The model training benefits from the self-
supervised language signal as a typical language model does. It also becomes
task-aware through the training on unified data. Evaluations show that FreeLM
retains its strong language capability as a language model in a similar scale
i.e., GPT-2. Experiments also show that FreeLM’s understanding performance on
the pre-defined tasks is significantly better than GPT-3 and instructGPT with
zero- or few-shot settings, even though our model is much smaller, 0.3B versus
175B parameters. FreeLM also generalizes well if a task can also be mapped to
the unified format. We believe the key reason behind the strong performance of
FreeLM is the task unification and the alternative training. If an enterprise
can design a similar unified data format for all their tasks, like the
proposition format in our setting, the proposed fine-tuning-free training
strategy could be a good consideration for cost saving.

## Limitation

There are three main limitations to this work. First, it is interesting to
study whether our model could lower the data size for training. If this
interesting point could be supported, computing costs could be further
reduced, and the large model could be greener. Another interesting point is to
study the impact of the order of task data. Lastly, we believe that our
proposed model could work better on a larger scale, but we do not prove that
due to the extremely high costs.

## Acknowledgments

This work was supported by the National Key R&D Program of China
(2022ZD0116300) and the National Science Foundation of China (NSFC No.
62106249).

## References

  * Bentivogli et al. (2009) Luisa Bentivogli, Bernardo Magnini, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. 2009.  The fifth PASCAL recognizing textual entailment challenge.  In _Proceedings of the Second Text Analysis Conference, TAC 2009, Gaithersburg, Maryland, USA, November 16-17, 2009_. NIST.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. 2019.  Transformer-xl: Attentive language models beyond a fixed-length context.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 2978–2988. Association for Computational Linguistics.
  * De Marneffe et al. (2019) Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019.  The commitmentbank: Investigating projection in naturally occurring discourse.  In _proceedings of Sinn und Bedeutung_ , volume 23, pages 107–124.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing, IWP@IJCNLP 2005, Jeju Island, Korea, October 2005, 2005_. Asian Federation of Natural Language Processing.
  * Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022.  GLM: general language model pretraining with autoregressive blank infilling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 320–335. Association for Computational Linguistics.
  * Gokaslan and Cohen (2019) Aaron Gokaslan and Vanya Cohen. 2019.  Openwebtext corpus.  http://Skylion007.github.io/OpenWebTextCorpus.
  * Gordon et al. (2012) Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012.  SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_ , pages 394–398, Montréal, Canada. Association for Computational Linguistics.
  * Grinsztajn et al. (2022) Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. 2022.  Why do tree-based models still outperform deep learning on tabular data? _CoRR_ , abs/2207.08815.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020.  Spanbert: Improving pre-training by representing and predicting spans.  _Trans. Assoc. Comput. Linguistics_ , 8:64–77.
  * Levesque et al. (2012) Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012.  The winograd schema challenge.  In _Principles of Knowledge Representation and Reasoning: Proceedings of the Thirteenth International Conference, KR 2012, Rome, Italy, June 10-14, 2012_. AAAI Press.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James F. Allen. 2016.  A corpus and evaluation framework for deeper understanding of commonsense stories.  _CoRR_ , abs/1604.01696.
  * Nie et al. (2021) Xiaonan Nie, Xupeng Miao, Shijie Cao, Lingxiao Ma, Qibin Liu, Jilong Xue, Youshan Miao, Yi Liu, Zhi Yang, and Bin Cui. 2021.  Evomoe: An evolutional mixture-of-experts training framework via dense-to-sparse gate.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Paperno et al. (2016) Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016.  The LAMBADA dataset: Word prediction requiring a broad discourse context.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1525–1534, Berlin, Germany. Association for Computational Linguistics.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter.  _CoRR_ , abs/1910.01108.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL_ , pages 1631–1642. ACL.
  * Sun et al. (2019) Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019.  ERNIE: enhanced representation through knowledge integration.  _CoRR_ , abs/1904.09223.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  _CoRR_ , abs/1804.07461.
  * Wang et al. (2021) Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021.  Entailment as few-shot learner.
  * Wang et al. (2022) Yequan Wang, Jiawen Deng, Aixin Sun, and Xuying Meng. 2022.  Perplexity from PLM is unreliable for evaluating text quality.  _CoRR_ , abs/2210.05892.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022.  Finetuned language models are zero-shot learners.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 5754–5764.

##  Appendix A Appendix

###  A.1 Implementation Detail

We list the implementation details of the proposed FreeLM to support
reproduction. To support research, we will release the corresponding codes and
trained parameters upon acceptance.

In our experiment of training FreeLM, we choose the medium version of GPT-2
with 345M parameters, which makes up of 12 transformer blocks. The model
dimension is 768. To speed up training and save energy, we initialize the
model with the pre-trained parameters provided by HuggingFace. All the
experiments are performed on 5 nodes each with eight 40GB NVIDIA Tesla A100
(40 cards total) for about 8 days. We train FreeLM for 8 epochs on OpenWebText
with a batch size of 4 on each card. Each epoch takes around 13.5 to 17.5
hours to complete. We use AdamW optimizer with a learning rate of 5e-5. The
mixing ratio of the language iterative stage and teacher iterative stage
samples is 2:1.


* We now compute the correction term on the left side, which corresponds to tropical curves that do not contain any contracted edge. Momentarily forgetting about the cross-ratio constraint, we get a $1$-parameter family of solutions. Each combinatorial type contributing a solution gives a $1$-parameter family. We look for combinatorial types for which this $1$-parameter family is unbounded, as we intersect with a very big cross-ratio. Since we precisely lack one marked point compared to the situation of Lemma <ref> giving the relative position of the marked points on the curves, we are in one of the following situations:

* There is an unbounded component containing two ends. We get a $1$-parameter family by deforming the path from one end to another. This situation is already described in [18]: the string is only comprised of two ends, and the unique adjacent bounded edge does not contribute to the cross-ratio, so that these combinatorial types do not contribute any solution.

* There is one of the $\psi L$-marking all of whose adjacent components are unbounded. We get a $1$-parameter family by moving the $\psi L$-marking along the line and the paths to the ends in the adjacent components along with it. In that case, we are as if there was only a unique $\psi L$-constraint, and the shape of the deformations that are not bounded are described by Lemma <ref>. Considering the exponent of the $\psi$-power is $1$, assuming the marking lies on the bottom end of its tropical line, the unique possibility is that there are vertical ends and a vertical edge of weight $2$ attached to it. The deformation makes the vertex go down. Unfortunately, the unique bounded edge does not contribute to the cross-ratio, and these solutions do not contribute either.

* Last, there is a unique bounded component which is adjacent to the two $\psi L$-markings. We get a $1$-parameter family by moving each $\psi L$-marking along their lines, the path between them inside the bounded component and the paths between the markings and the ends in the other adjacent components.

Up to symmetry, assume that the marking $Q'$ moves along the bottom end of $L'$ which is right to $L$. We prove step by step that the only possible deformation pattern is as depicted on the left of Figure <ref>.

* The vertex $Q'$ cannot be a flat horizontal vertex. By balancing, there is therefore at least one edge going down, which hence needs to be an bottom end.

* Similarly, as the vertex $Q'$ cannot be flat and vertical, still by balancing, there needs to be an edge going right, which hence is an end of slope $(1,1)$.

* As the vertex is trivalent, the remaining edge has slope $(-1,0)$.

* This latter edge cannot be an end since the curve is connected. As it misses the point region, the only possibility is that it meets $L$ at the other $\psi L$-marking: $Q$.

* By balancing, there needs to be a bottom end adjacent to $Q$. The balancing the remaining edge has slope $(-1,1)$.

* As this new edge still misses the point region, the only possibility is to split into a vertical edge that goes inside the point ragion and an end of slope $(-1,1)$. We exactly get the picture from Figure <ref>.

[line cap=round,line join=round,>=triangle 45,x=0.4cm,y=0.4cm]
[line width=1pt,blue] (0,0)–++(-3,0)++(3,0)–++(3,3) node[above right] $L$ ++(-3,-3)–++(0,-6)–++(0,-2);
[line width=1pt,blue] (1,-1)–++(-4,0)++(4,0)–++(3,3) node[above right] $L'$ ++(-3,-3)–++(0,-6);
[blue,fill=blue] (-1.5,1.5) circle[radius=1];
[line width=1pt] (0,-6)–++(-2,2)–++(-4,0);
[line width=1pt] (0,-8)–++(0,2)–++(1,0)–++(0,-2)++(0,2)–++(2,2);
(1) at (-2,2) $\Gamma_1$;
[line width=1pt] (-2,-4) to (1);
(0,-6) node $\bullet$ node[below left] $Q$;
(1,-6) node $\bullet$ node[below right] $Q'$;
[line cap=round,line join=round,>=triangle 45,x=0.4cm,y=0.4cm]
[line width=1pt,blue] (0,0)–++(-3,0)++(3,0)–++(3,3) node[above right] $L=L'$ ++(-3,-3)–++(0,-6)–++(0,-2);
[line width=1pt] (0.15,-6)–++(0,-2);
[line width=1pt] (-0.15,-6)–++(0,-2);
[blue,fill=blue] (-1.5,1.5) circle[radius=1];
[line width=1pt] (0,-6)–++(-2,2)–++(-4,0);
[line width=1pt] (0,-6)–++(2,2);
(1) at (-2,2) $\Gamma_1$;
[line width=1pt] (-2,-4) to (1);
(0,-6) node $\bullet$;

The deformation of a $k=2$-string as in Figure <ref> to a string with two $\psi L$ constraints at infinity.

The remaining curve is of degree $d-1$, passes through $n=3d-4=3(d-1)-1$ points and has a fixed down end of weight one, contributing a factor of $(d-1)\cdot N_{d-1}$, where $(d-1)$ accounts for the choice of the fixed down end. Furthermore, we need to pick the intersections with $L_1$ and $L_2$, giving another $(d-1)^2$ choices. Altogether, we obtain a contribution of

$$(Ie)^\mathrm{left}=3(d-1)^3\cdot N_{d-1}.$$

Step 2, the $(LL_1//L'L_2)$ right side:

* As in step 1, we assume $\Gamma$ has a contracted edge which splits the curve in two components $C_1$ and $C_2$ of respective degrees $d_1+d_2=d$. Let $n_1$ (resp. $n_2$) be the number of marked point on $C_1$ (resp. $C_2$).

* We cannot have $d_i=0$, as the marked ends $Q$ resp. $Q'$ need to be at a $4$-valent vertex, but the contracted bounded edge together with the two markings (one for $Q$ resp. $Q'$, one for $L_i$) yields only valency three.

* Assume now that $d_1\neq 0$, and the contracted edge is not adjacent to either of the $\psi L$-points. As $C_1$ is subject to $n_1$ point constraints and one $\psi L$ constraint, and $C_2$ to $n_2$ point constraints and one $\psi L$ constraint, we have

n_1\leqslant 3d_1-2 , \\
n_2\leqslant 3d_2-2.\\
\end{array}	 \right.$$

Adding the two relations, we see that $n=n_1+n_2\leqslant 3(d_1+d_2)-4=n$, so we have in fact equality. Thus, we can pick independently $C_1$ and $C_2$ subject to point constraints.
As both curves also meet a line condition without $\psi$, $L_i$, we can use the divisor equation Lemma <ref> to express these factors as $d_i\cdot \ang{\psi L}_{d_i}+N_{d_i}$ for $i=1,2$.
Furthermore, we pick an intersection point between $C_1$ and $C_2$ where to insert the contracted edge ($d_1d_2$). This contributes a term

$$(Ib)^\mathrm{right}=\sum_{d_1+d_2=d} \bino{3d-4}{3d_1-2} d_1 d_2 (d_1\cdot \ang{\psi L}_{d_1}+N_{d_1})\cdot (d_2\cdot \ang{\psi L}_{d_2}+N_{d_2}) .$$

* We now assume that the contracted edge is adjacent to the $\psi L$-point in $C_1$, but not to the $\psi L$-point in $C_2$.
The situation where the contracted edge is adjacent to the $\psi L$-point in $C_2$ but not the one in $C_1$ is completely analogous and symmetric (by exchanging $d_1$ and $d_2$, but we sum over all pairs satisfying $d_1+d_2=d$), which accounts for a factor of $2$ in our contributions below.

The valency of the vertex has to be $4$, of which two arise from the contracted bounded edge and the marked end itself. The remaining two edges point into opposite direction by the balancing condition, that is, the image just looks like a straight line. This image has to pass to the intersection point of $C_1$ and $C_2$, as it is adjacent to the contracted bounded edge, and through $L$ as it is adjacent to $Q$.
We now have

n_1\leqslant 3d_1-1 , \\
n_2\leqslant 3d_2-2.\\
\end{array}	 \right.$$

Adding the two rows, we do not get an equality anymore, as both sides differ by one. So we are in one of the following situations:

* We have $n_1=3d_1-1$. In this situation, we pick $C_1$ fixed by the points constraints ($N_{d_1}$), and an intersection point with $L_1$ ($d_1$). The curve $C_2$ has to meet one of the intersection points between $C_1$ and $L$, of which there are $d_1$. The curve $C_2$, in addition to passing through the remaining points, also still contains the second $\psi L$ condition, and passes through $L_2$, so using the divisor equation <ref> again we obtain a factor of $d_2\cdot \ang{\psi L}_{d_2}+N_{d_2}$.
We get a contribution of

$$(Ic)^\mathrm{right}=2\cdot \sum_{d_1+d_2=d} \bino{3d-4}{3d_1-1} d_1^2N_{d_1}\cdot (d_2\cdot \ang{\psi L}_{d_2}+N_{d_2}).$$

* We have $n_1=3d_1-2$. In this situation, $C_2$ is fixed by the $P$-constraints and $\psi L$-constraint, and passes through $L_2$ in addition ($d_2\cdot \ang{\psi L}_{d_2}+N_{d_2}$). The curve $C_1$ has then to pass through an intersection of $C_2$ with $L$ of which there are $d_2$, in addition to the $n_1$ points ($N_{d_1}$). We finally pick the intersection point of $C_1$ with $L_1$  ($d_1$).
We obtain

$$(Ic')^\mathrm{right}=2 \cdot \sum_{d_1+d_2=d} \bino{3d-4}{3d_1-2} d_1 d_2 N_{d_1}\cdot (d_2\cdot \ang{\psi L}_{d_2}+N_{d_2}).$$

* We finally assume that the contracted edge is adjacent to the $\psi L$-point in $C_1$ and to the $\psi L$-point in $C_2$. The images of both vertices adjacent to the contracted bounded edge look like straight lines, i.e. the intersection of $C_1$ and $C_2$ has to pass through $L\cap L'$.
We have

n_1\leqslant 3d_1-2 , \\
n_2\leqslant 3d_2-2.\\
\end{array}	 \right.$$

Thus, we have equality again. 	The curve $C_1$ is fixed by the point conditions together with the intersection point of $L$ and $L'$ ($N_{d_1}$), we can choose $d_i$ intersection poihts of $C_i$ with $L_i$ for $i=1,2$, and $C_2$ has to pass through the remaining points and the intersection point of $L$ and $L'$ ($N_{d_2}$).
Altogether, we have

$$(Id)^\mathrm{right}= \sum_{d_1+d_2=d} \bino{3d-4}{3d_1-2} d_1 d_2 N_{d_1}N_{d_2}.$$

On this side, we cannot have correction terms coming from strings going to infinity: the deformation has already been described on the left side, and as the $\psi L$-markings are close to each other, the cross-ratio is always bounded.

As the last step, we set the two sides equal and solve for our wanted enumerative invariant, obtaining the formula in the statement.

Using the initial value $\ang{\psi L,\psi L}_1=2$ (obtained from a line with vertex at the intersection of $L$ and $L'$, and both marked points $Q$ and $Q'$ adjacent to this vertex, yielding a $5$-valent vertex contributing a multiplicity factor of $2$), we can compute
$\ang{\psi L,\psi L}_2=17$ and $\ang{\psi L,\psi L}_3=302$, confirming the computation in Examples <ref> and <ref> obtained from Theorem <ref>.

[1]
Dan Abramovich and Qile Chen.
Stable logarithmic maps to Deligne-Faltings pairs. II.
Asian J. Math., 18(3):465–488, 2014.

[2]
Lars Allermann and Johannes Rau.
First steps in tropical intersection theory.
Mathematische Zeitschrift, 264(3):633–670, 2010.

[3]
Aubin Arroyo, Erwan Brugallé, and Lucía López de Medrano.
Recursive formulas for Welschinger invariants of the projective
Int. Math. Res. Not. IMRN, (5):1107–1134, 2011.

[4]
Benoît Bertrand, Erwan Brugallé, and Grigory Mikhalkin.
Tropical Open Hurwitz numbers.
Rend. Semin. Mat. Univ. Padova, 125:157–171, 2011.

[5]
Benoît Bertrand, Erwan Brugallé, and Grigory Mikhalkin.
Genus 0 characteristic numbers of tropical projective plane.
Compos. Math., 150(1):46–104, 2014.

[6]
Florian Block, Andreas Gathmann, and Hannah Markwig.
Psi-floor diagrams and a Caporaso-Harris type recursion.
Israel J. of Math., 2011.
doi: 10.1007/s11856-011-0216-0.

[7]
Thomas Blomme.
A Caporaso-Harris type Formula for relative refined invariants.
Preprint, arXiv:1912.06453, 2019.

[8]
Janko Böhm, Kathrin Bringmann, Arne Buchholz, and Hannah Markwig.
Tropical mirror symmetry for elliptic curves.
J. Reine Angew. Math., 732:211–246, 2017.

[9]
Pierrick Bousseau.
Tropical refined curve counting from higher genera and lambda
Invent. Math., 215(1):1–79, 2019.

[10]
Erwan Brugallé and Hannah Markwig.
Deformation and tropical Hirzebruch surfaces and enumerative
J. Algebraic Geom., 25(4):633–702, 2016.

[11]
Renzo Cavalieri, Paul Johnson, and Hannah Markwig.
Tropical Hurwitz numbers.
J. Algebr. Comb., 32(2):241–265, 2010.

[12]
Qile Chen.
Stable logarithmic maps to Deligne-Faltings pairs I.
Ann. of Math., 180(2):455–521, 2014.

[13]
Qile Chen.
The degeneration formula for logarithmic expanded degenerations.
J. Algebr. Geom., 23(2):341–392, 2014.

[14]
Sergey Fomin and Grigory Mikhalkin.
Labeled floor diagrams for plane curves.
J. Eur. Math. Soc., 12(6):1453–1496, 2010.

[15]
Andreas Gathmann, Michael Kerber, and Hannah Markwig.
Tropical fans and the moduli spaces of tropical curves.
Compositio Mathematica, 145(1):173–195, 2009.

[16]
Andreas Gathmann and Hannah Markwig.
The Caporaso–Harris formula and plane relative Gromov-Witten
invariants in tropical geometry.
Mathematische Annalen, 338(4):845–868, 2007.

[17]
Andreas Gathmann and Hannah Markwig.
The numbers of tropical plane curves through points in general
Journal für die reine und angewandte Mathematik (Crelles
Journal), 2007(602):155–177, 2007.

[18]
Andreas Gathmann and Hannah Markwig.
Kontsevich's formula and the WDVV equations in tropical geometry.
Adv. Math., 217:537–560, 2008.

[19]
Andreas Gross.
Correspondence theorems via tropicalizations of moduli spaces.
Commun. Contemp. Math., 18(3):1550043, 36, 2016.

[20]
Andreas Gross.
Intersection theory on tropicalizations of toroidal embeddings.
Proc. Lond. Math. Soc. (3), 116(6):1365–1405, 2018.

[21]
Mark Gross.
Mirror Symmetry for $\mathbb{P}^2$ and tropical geometry.
Adv. Math., 224(1):169–245, 2010.

[22]
Mark Gross, Rahul Pandharipande, Bernd Siebert, et al.
The tropical vertex.
Duke Mathematical Journal, 153(2):297–362, 2010.

[23]
Mark Gross and Bernd Siebert.
Mirror Symmetry via logarithmic degeneration data I.
J. Differential Geom., 72:169–338, 2006.

[24]
Mark Gross and Bernd Siebert.
Mirror Symmetry via logarithmic degeneration data II.
J. Algebraic Geom., 19(4):679–780, 2010.

[25]
Mark Gross and Bernd Siebert.
Logarithmic Gromov-Witten invariants.
J. Amer. Math. Soc., 26(2):451–510, 2013.

[26]
Ilia Itenberg, Viatcheslav Kharlamov, and Eugenii Shustin.
Logarithmic asymptotics of the genus zero Gromov-Witten invariants
of the blown up plane.
Geom. Topol., 9:483–491, 2005.

[27]
Michael Kerber and Hannah Markwig.
Intersecting Psi-classes on Tropical $M_{0,n} $.
International Mathematics Research Notices, 2009(2):221–240,

[28]
Travis Mandel and Helge Ruddat.
Descendant log Gromov-Witten invariants for toric varieties and
tropical curves.
Trans. Amer. Math. Soc., 373(2):1109–1152, 2020.

[29]
Hannah Markwig and Johannes Rau.
Tropical descendant Gromov-Witten invariants.
Manuscripta Math., 129(3):293–335, 2009.

[30]
Grigory Mikhalkin.
Enumerative tropical algebraic geometry in $\mathbb{R}^2$.
Journal of the American Mathematical Society, 18(2):313–377,

[31]
Grigory Mikhalkin.
Moduli spaces of rational tropical curves.
arXiv preprint arXiv:0704.0839, 2007.

[32]
Grigory Mikhalkin.
Quantum indices and refined enumeration of real plane curves.
Acta Mathematica, 219(1):135–180, 2017.

[33]
Takeo Nishinou and Bernd Siebert.
Toric degenerations of toric varieties and tropical curves.
Duke Math. J., 135:1–51, 2006.

[34]
Peter Overholser.
A descendent tropical Landau-Ginzburg potential for $\Bbb P^2$.
Commun. Number Theory Phys., 10(4):739–803, 2016.

[35]
Dhruv Ranganathan.
Skeletons of stable maps I: rational curves in toric varieties.
J. Lond. Math. Soc. (2), 95(3):804–832, 2017.

[36]
Dhruv Ranganathan.
Logarithmic Gromov-Witten theory with expansions.
Preprint, arXiv:1903.09006, 2019.

[37]
Ilya Tyomkin.
Tropical geometry and correspondence theorems via toric stacks.
Math. Ann., 353(3):945–995, 2012.