#  Baichuan2-Sum: Instruction Finetune Baichuan2-7B Model for Dialogue
Summarization

1st Jianfei Xiao* Viterbi School of Engineering
University of Southern California
Los Angeles, United States
<EMAIL_ADDRESS>2nd Yancan Chen* School of Computing
National University of Singapore
Singapore, Singapore
<EMAIL_ADDRESS>3rd Yimin Ou  College of Computing and Information Science
Cornell University
New York, United States.
<EMAIL_ADDRESS>4th Hanyi Yu  Viterbi School of Engineering
University of Southern California
Los Angeles, United States
<EMAIL_ADDRESS>5th Kai Shu  Viterbi School of Engineering
University of Southern California
Los Angeles, United States
<EMAIL_ADDRESS>6th Yiyong Xiao† Computer Science College
Taiyuan Normal University
Guangzhou, China
<EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) like LLaMA, Baichuan and Bloom models show
remarkable ability with instruction fine-tuning in many natural language
tasks. Nevertheless, for the dialogue summarization task, which aims to
generate summaries for different roles in dialogue, most of the state-of-the-
art methods foucus on small models (e.g BART and BERT). Existing methods try
to add task specified optimization on small models like adding global-local
centrality score to models. In this paper, we propose an instruction fine-
tuning model: Baichuan2-Sum, for role-oriented dialogue summarization. By
setting different instructions for different roles, the model can learn from
the dialogue interactions and output the desired summaries. Furthermore, we
applied NEFTune technique to add suitable noise during training, improving the
results. The experiments demonstrate that the proposed model achieves the new
state-of-the-art results on two public dialogue summarization datasets: CSDS
and SAMSUM. The Baichuan2-Sum model shows an improvement in Rouge scores on
both datasets compared to the previously best-performing model. Notably, for
the SAMSUM dataset, there is a 21% increase in the ROUGE-1 score, a 32%
increase in the ROUGE-2 score, and a 9% increase in the ROUGE-L score. We have
released our model and related codes to facilitate future studies in the
dialogue summarization task.

###### Index Terms:

Large language model, instruction fine-tuning, Baichuan2, Dialogue
summarization, NEFTune

##  I Introduction

Dialogue summarization, a natural language processing (NLP) task, aims to
generate a coherent summary from the content. With the development of the
internet, the variety of dialogues (including human-to-human and human-
computer conversations) and their volume have increased rapidly. It is
valuable to generate the summarization automatically for dialogues like online
customer service [1, 2] and meeting conversation [3, 4]. The figure 1 shows an
example of a customer and service dialogue and its corresponding summarization
from CSDS [2] dataset. ††footnotetext:  * These authors contributed equally.
† Corresponding Author.

Figure 1: A customer service dialogue example of CSDS dataset ††footnotetext:
Code address: https://github.com/aivolcano/LLM4DialogueSummarization

The generation of dialogue summaries presents many challenges, such as
interactions between multiple roles[5, 6], topic shifting between different
contexts[1, 7, 8] and understanding long dialogue context[9]. To conquer these
issues, the existing works focus on adding task specified optimization on
language models. Liang et al.[5] proposed a Role-Aware Centrality (RAC) model,
which requires re-weighting the hidden states by calculating the role-aware
centrality. Zou et al. [1] present a two-stage dialogue summarization method
with a saliency-aware neural topic model(SATM). Compared to the classical
transformer achitecture, SATM adopts their proposed topic-informed attention
mechanism. DIALOGLM [9] is a pre-trained Model designed for long dialogue
understanding and summarization. But the model is a pre-trained via a window-
based denosing task, which means it cannot be directly fine-tuned on open-
source NLP model weights. Besides, the backbone networks for majority of these
works are small models like PGN [10], BERT [11] and BART [12] rather than
large language models (LLMs) e.g. GPT-3 [13], LLaMA2 [14], Baichuan2 [15], Glm
[16]. The increase in parameters has brought significant improvements in the
capabilities for language models, resulting in more human-like text generation
for a range of text generation tasks, eg. machine translation, text
summarization and question answering. Hence, it is valuable to investigate
LLMs that can more effectively understand and summarize dialogues.

The contribution of this paper can be summarized as follows: 1) we present
BaichuanSum, a model based on Baichuan2, trained on dialogue dataset (CSDS [2]
and SAMSUM [17]), which achieves the new start-of-the-art performance for the
dialogue summarization task. 2). we create an instruction fine-tuning dataset
based on the original datasets, containing different instructions for various
summarization types. 3) we employ the Noisy Embedding Instruction Fine Tuning
(NEFT) [18] method to train the model, further improving the model performance
for this task. 4) Our work is publicly avalible on GitHub, offering good
extensibility. In addition to Baichuan2, the code also supports training and
evaluation for LLMs like LLaMA2 [15], Bloom [19] and ChatGLM [16].

##  II Related Works

###  II-A Dialogue Summarization Works

PGN (Pointer-Generator Networks) [10] is a sequence-to-sequence network for
summarization, featuring an encoder with a bidirectional LSTM layer and a
decoder with another bidirectional LSTM layer. Abigail et.al enhanced this
model in two innovative ways. Firstly, They implemented a hybrid pointer-
generator network, which enables the copying of words directly from the source
text through a pointing operation.Secondly, they utilized coverage mechanism
to track the information summarized, thus reducing repetition.

Liu et.al [20] proposed a new framework based on BERT for both extractive
(BERTExt) and abstractive (BERTAbs) summarization models. Since our task is
the abstractive summarization, only abstractive model is used as a baseline
model in the experiment part.

BART [12] is a denoising autoencoder for sequence-to-sequence tasks, including
machine translation and text summarization. It combines a bi-directional
encoder, similar to BERT and an auto-regressive decoder, like GPT. Bart is
trained by corrupting text through various noising operations such as Token
Masking, Token Deletion and Text Infilling, etc., and is then required to
reconstruct the original text.

BART-GLC (Global-Local Centrality) [8] is a topic-aware BART model for
dialogue summarization task. The model calculates the global and local
centrality score of each utterance using Kmeans algorithm. GLC score
(multiplication of global centrality score and local centrality score) is then
used to reweight the hidden state of the BART.

###  II-B Instruction Finetuning

Instruction fine-tuning [21, 22, 23] of large language models (LLMs) or large
multimodal models (LMMs) has been proved to improve the zero-shot learning
ability in many tasks. This technology enhances the ability of large language
models to follow instructions.

Wang et.al [21] introduce a framework called SELF-INSTRUCT for improving the
instruction-following capabilities of pretrained language models. The
framework generates instructions, input, and output samples from a language
model, filters invalid or similar ones, and uses them to finetune the original
model. The researchers apply the method to the vanilla GPT3 model and
demonstrate a 33% improvement over the original model on a task named SUPER-
NATURALINSTRUCTIONS.

After training on a 52K instruction-following dataset, Alpaca-7B [22] has
exhibited behaviors similar to those of OpenAI’s text-davinci-003.In this
task, due to certain limitation, we manually designed some instructions for
different roles and datasets, instead of automatically generating instructions
from LLMs.

The paper [23] shows the first attempt to use language-only GPT-4 to generate
multimodal language-image instruction-following data. Haotian et al. introduce
LLaVA, a instruction finetuned large multimodal model that connects a vision
encoder and an LLM for general-purpose visual and language understanding. The
experiments show that LLaVA demonstrates impressive multimodal chat abilities
and achieves a new state-of-the-art accuracy when fine-tuned on Science QA.

The work above inspires us to employ fine-tuning for the task of dialogue
summarization. We have created a fine-tuning dataset specifically for this
task.

###  II-C Noisy Embedding with NEFTune

Noisy Embedding technology aims to improve the model performance of pre-
trained models on downstream tasks by adding noise to the paramesters of pre-
trained model before fine-tuning. This method also applied on modern LLMs
[38]. When evaluated using AlpacaEval, the Llama2-7B model, after adding
noise, showed a 34.9% improvement on the Alpaca dataset compared to the model
without noise! [36, 37] Additionally, this technology reduces overfitting
effectively. In the case of longer outputs, the repetition rate does not raise
significantly; instead, it provides more details, thereby enhancing the
quality of the output text. Thanks to its simplicity and effectiveness in
enhancing model performance, Hugging Face website has integrated this
technology into Transformer Reinforcement Learning (TRL) library [39].

##  III Methodology

Figure 2 shows the main structure of our proposed Baichuan2-Sum model, which
consists of Noisy Embedding, Decoder-based Transformers with some
modifications, including xFormers, RMS Norm, and Rotary Positional Embedding
(RoPE). In this section, let us introduce them step by step.

###  III-A Tokenizer

Before further processing, Baichuan2 model requires Baichuan2 tokenizer to
break down the text into tokens, positional ids and attention mask.The
Baichuan2 model has faster inference speed and smaller memory cousumption
compared to other open source LLMs due to it has highest compression rate (See
Table 1). The compression rate of Baichuan2’s tokenizer is 0.498, which is
half that of LLaMA 2, resulting in Baichuan2’s speed being nearly twice as
fast.

The CSDS dataset is comprised ntirely of Chinese language corpora. Compared to
other tokenizers, Baichuan2 offers two advantages due to its comprehensive
vocabulary of Chinese words: 1) Under identical text conditions, the model
processes fewer tokens, leading to faster fine-tuning and predition; 2) The
shorter token lengths reduce the demands on GPU memory.

TABLE I: The vocab size and text compression rate of Baichuan2 tokenizer and other tokenizers Tokenizer | Vocab Size | Compression Rate
---|---|---
LLaMA 2 | 32,000 | 1.037
Bloom | 256,680 | 0.501
ChatGLM 2 | 64,794 | 0.527
Baichuan 1 | 64,000 | 0.570
Baichuan 2 | 125,696 | 0.498

###  III-B NEFTune: Noisy Embedding Instruction Fine-tuning

The tokens are converted into input embedding vectors, with some uniform noise
added to prevent model overfitting and to achieve better model performance
without incurring additional computational costs. NEFTune [18] is an effective
strategy for modeling. In every training step, NEFTune introduces noise to the
input embedding vector. This noise is generated from a uniform function within
the range of [-1, 1] and matches the shape of the embedding vector. The noise
is scaled by a factor of $\alpha/\sqrt{Ld}$, where $\alpha$ is a hyper-
parameter, and $L$ represents the sequence length and $d$ is the embedding
dimension.

Algorithm 1 NEFTune: Noisy Embedding Instruction Finetuning

Input: $D=\\{(x_{i},y_{i})\\}_{i=1}^{N}$ tokenized dataset, embedding layer
$emb(\cdot)$, rest of model $f_{\text{/}emb}(\cdot)$, model parameters
$\theta$, loss($\cdot$), optimizer $opt(\cdot)$

NEFT Hyperparameter: base noise scale $\alpha\in\mathbb{R}^{+}$

Initialize $\theta$ from a pretrained model.

repeat $(X_{i},Y_{i})\sim D$ $\triangleright$ sample a minibatch of data and
labels

$X_{emb}\leftarrow emb(X_{i}),\mathbb{R}^{B\times L\times d}$ $\triangleright$
batch size $B$, seq. length $L$, embedding dimension $d$

$\varepsilon\sim\text{Uniform}(-1,1),\mathbb{R}^{B\times L\times d}$
$\triangleright$ sample a noise vector

$X^{\prime}_{emb}\leftarrow
X_{emb}+\left(\frac{\alpha}{\sqrt{Ld}}\right)\varepsilon$ $\triangleright$ add
scaled noise to embedsa

$\hat{Y_{i}}\leftarrow f_{/emb}(X^{\prime}_{emb})$ $\triangleright$ make
prediction at noised embeddings

$\theta\leftarrow opt(\theta,\text{loss}(\hat{Y_{i}},Y_{i}))$ $\triangleright$
train step, e.g., grad descent

until stopping criteria met/max iterations.

aIf sequence lengths in a batch are not equivalent, then $L$ is a vector
$\in\mathbb{Z}_{>0}^{B}$ and the scaling factor
$\left(\frac{\alpha}{\sqrt{Ld}}\right)$ is computed independently for each
sequence in batch.

###  III-C Baichuan2 Model With NEFTune

After feeding the tokens into the model, the embedding is processed by the
Decoder architecture. This includes RMS Normalization, and the use of an
enhanced attention mechanism known as xFormers with Rotary Positional
Embedding (RoPE). It also involves a residual network structure and the SwiGLU
activation function. The Decoder processes the data in a loop which runs N
times, after which the token probability distribution is obtained via a
softmax function. Throughout this procedure, the NEFTune strategy is applied
to fine-tune the model, culminating in the output of our proposed model,
Baichuan2-Sum.

Figure 2 illustrates the framework of Baichuan2 model integrated with NEFTune.
Similar to GPT series models and LLaMA2, Baichuan2 consists exclusivly of
decoder-only Transformers. The proposed architecture can be mainly divided
into three parts: the Embedding part, the Transformer Block part, and the
Linear part. Although the composition is similar to the classical large
language models, the proposed model still exhibits some distinct differences.

Figure 2: Baichuan2 Achitecture with NEFTuning

Noisy Embedding: We enhance the model’s performance by introducing normal
distribution noise into embedding using NEFTune method. The sole tunable
parameter in NEFTune is $\alpha$, we set it to 5 after numerous experiments.
This setting demonstrated the best performance compared to other values.

Tokenization: Baichuan2 tokenizer has employed SentencePiece [34] and Byte-
Pair Encoding (BPE) [35] technologies. Unlike typical technology, it neither
normalizes text nor adds a dummy prefix. Instead, it splits numerical values
into single digits and adds a space token for code text with external spaces.
Additionally, some infrequent characters have been converted back to the UTF-8
bytes. Thanks to these advanced technologies, the Baichuan2 tokenizer boasts a
faster inference speed and lower memory consumption compared to other open-
source LLMs.

Rotary Positional Embedding (RoPE): The Baichuan2-7B model adopts rotary
positional embedding (RoPE) [24] instead of typical positional embedding. This
technique encodes absolute positions using a rotation matrix and explicitly
integrates relative position information into the self-attention mechanism.
RoPE is effective for handling long sequences as it can capture long-range
dependencies more efficiently, thus enhancing the model performance with
lengthy texts. Moreover, due to integrated with the self-attention mechanism,
RoPE adds positional context without significantly increasing computational
burden.

SwiGLU: Baichuan2 has employed SwiGLU [25], a switch-activated variant of
Gated Linear Unit (GLU), to achieve the optimal value in log-perplexity,
thereby enhancing its generation results.

xFormer: The authors of Baichuan 2 have adopted xFormers[26], which optimizes
attention with biasing capabilities, to perform memory-efficient attention
operations in the attention layers. This modification contributes to both
performance enhancements and efficiency gains in the model’s large-scale
training.

RMSNorm: RMSNorm [27] is utilized to prevent the issues of vanishing or
exploding gradients during training, leading to more stable training outcomes.
Additionally, in comparison to Layer Normalization, RMSNorm typically requires
less computational effort. This is becuase it computes only the root mean
square of the inputs, rather than their mean, making the process more
efficient, which lead to reducing computational time by 7% to 64% on different
models.

In table 3, we detail the parameters of the proposed model. The Baichuan2
model consists 32 transformer layers, each with 32 attention heads. It has a
hidden size of 4,096 and a feed-forwarding dimension of 11,008. These
dimensions are significantly larger than those of BART.

TABLE II: Details of Proposed Model Parameter | Value
---|---
Hidden Size | 4,096
FFN Size | 11,008
num heads | 32
num layers | 32
seq length | 4,096

##  IV Experiments

###  IV-A Dataset

We evaluate our model on two dialogue summarization datasets: CSDS1 [2] and
SAMSUM2 [17].

CSDS (Customer Service Dialogue Summarization) is a Chinese dataset focused on
role-based and real-world dialogue summarization. It not only produces overall
dialogue summaries but also creates summaries specific to different roles
within the dialogue. Specifically, the user summary represent the high-
frequency questions, and the agent summary for the quality of customer
service. The dataset includes 9,101 training samples and 800 samples each for
testing and validation.

SAMSUM (Samsung Abstractive Messenger Summarization dataset) contains about
16,000 English dialogue summaries. These messenger-style conversations,
crafted by linguists fluent in English, mirror the proportion of typical
themes found in real-life messenger conversations. The dataset varies in style
and register, ranging from informal and semi-formal to formal, and includes
the use of slang, emoticons, and typographical errors. The summaries have
aimed to encapsulate the discussions in a third-person narrative. The SAMSum
includes 14,732 training samples, 818 validation samples, and 819 test
samples.

Figure 3: Distribution of Samples in CSDS and SAMSUM Datasets Figure 4: Token
Length Distribution for CSDS Dataset Figure 5: Token Length Distribution for
SAMSUM Dataset

Figure 3 shows the training/validation/test sample distributions of both CSDS
and SAMSUM datasets. Meanwhile, in order to set the maximum token length for
the Baichuan2 model, we have separately compiled the token length
distributions for the dialogue and summary sections of the two datasets, as
shown in Figures 5 and 6. From these figures, it is evident that the summary
token length in both datasets are shorter than those of the dialogues, and the
length of the dialogues is mostly less than 1200 tokens. ††footnotetext:  1
https://github.com/xiaolinAndy/CSDS
2 https://huggingface.co/datasets/samsum

###  IV-B Instruction finetuning prompt

Table 3 shows the instruction fine-tuning prompts for different datasets.
Given that CSDS includes three types (Agent/User/All) of summarization, we
have designed three distinct prompt templates. The instructions for CSDS are
in Chinese, while those for SAMSUM are in English. This approach maintains the
consistency of the language used in the instruction, dialogue, and
summarization. We add the token “<s>” at the beginning of the template and the
same token at the end to indicate the start and end of the generated sequence,
respectively. These instruction templates style are designed to be
straightforward and easily applicable to new dataset, demonstrating the
transferability of this method.

TABLE III: Instruction finetuning Prompt Dataset | Type | Instruction Prompt Template
---|---|---
CSDS | Agent | <s>下面是一段电商公司的客服和用户之间的对话，请你给出客服的摘要。\n {{Dialouge}} \n {{Agent Summary}} <s>
User | <s>下面是一段电商公司的客服和用户之间的对话，请你给出用户的摘要。\n {{Dialouge}} \n {{User Summary}} <s>
All | <s>下面是一段电商公司的客服和用户之间的对话，请你给出全部的摘要。\n {{Dialouge}} \n {{All Summary}} <s>
SAMSUM | All | <s>Please make the summarization of following dialogue. \n {{Dialouge}} \n {{All Summary}} <s>

###  IV-C Evaluation Metrics

To assess model performance, we utilize both lexical and semantic-level
metrics. Lexical metrics, such as ROUGE-1/2/L [28] and BLEU[29], quantify
N-gram-based similarity between model-generated summaries and reference
summaries, spanning 1-gram to 4-gram evaluations that cover word accuracy to
phrase representation. For semantic similarity, BERTScore[30] calculates
cosine similarity between embeddings of reference tokens and candidate tokens,
derived from pre-trained language models. These metrics are employed to gauge
model performance. Displayed results are the averaged outcomes from multiple
model checkpoints.

###  IV-D Implementation Details

The initial weights 3 of the model were downloaded from Baichuan2’s model page
on Hugging Face website. Instead of training the full set of 7 billion
parameters of the model, we train only a small ratio of parameters using Low-
Rank Adaptation (LoRA) [31] method. LoRA involves freezing the pre-trained
model weights and adding trainable Low-Rank decomposition matrices to every
layer of the Transformer architecture. This approach significantly reduces the
number of trainable parameters for subsequent tasks.

The LoRA training for Baichuan2 is implemented by PEFT4 (Parameter-Efficient
Fine-Tuning) library. PEFT facilitates the effective adjustment of pre-trained
language models (PLMs) for a range of downstream applications, without the
necessity to fine-tune every parameter of the model. The latest version of the
PEFT library includes the implementations of LoRA [31], Prefix Tuning [32] and
P-Tuning [33]. Our configuration for LoRA is as follows: Lora rank is 9, Lora
target is W_pack and Lora Dropout is 0.9. ††footnotetext:  3
https://huggingface.co/baichuan-inc/Baichuan2-7B-Base
4 https://github.com/huggingface/peft
5 https://pypi.org/project/rouge-score/
6 https://github.com/mjpost/sacreBLEU
7 https://github.com/Tiiiger/bert_score
8 https://github.com/xnliang98/bart-glc
We fine-tune our model on the processed CSDS dataset for 9000 iterations,
using a learning rate of 5e-5 and a batch size of 2, with a gradient
accumulation step of 4. The Adam optimizer was employed without weight decay,
along with a cosine learning rate strategy that includes no warm-up steps.
Gradient accumulation is a practical method to save GPU memory. It involves
accumulating the gradients computed over multiple mini-batches before
conducting a single parameter update. To prevent gradient explosion, gradient
clipping by the L2 norm, as shown in the following formula, during training
process.

$\mathbf{g^{\prime}}=\frac{\mathbf{g}}{\max(1,\frac{\|\mathbf{g}\|}{\theta})}$

where $\mathbf{g}$ is the gradient vector and $\theta$ is the threshold, the
clipped gradient $\mathbf{g^{\prime}}$, $\|\mathbf{g}\|$ represents the L2
norm of the gradient vector.

The $\theta$ is set to 1 for clipping the L2 norm of gradient of whole model.
To save memory and accelerate the training speed, we apply FP16 data type for
all experiments. With the previous setting, we train Baichuan2-7B model with
only one RTX 4090 GPU card. Instruction finetuning on CSDS dataset completes
within 6 hours and on SAMSUM dataset completes within 4 hours. To sum up, the
specific details of experiments are shown in table 4.

TABLE IV: Details of experiments Parameter | Value
---|---
Learning Rate | 5e-5
Iteration | 9000
Batch Size | 2
Accumulation Step | 4
Maximum L2 Norm | 1
Optimizer | Adam
Learning Strategy | Cosine Learning
Lora Rank | 9
Lora Target | W_pack
Lora Dropout | 0.1

###  IV-E Automatic Evaluation Results and Analysis

We evaluates the all models introduced in the related work, including PGN,
BERT, BART-GLC and our proposed model Baichuan2Sum. BART-GLC8 is the previous
state-of-the-art model for the dialogue summarization. Each block has three
values, in a order of the following: final summary/user summary/agent summary.
According to the results in Table 5, Baichuan2Sum outperforms the other models
in all types of summary in terms of ROUGE-1/2/L and BERTScore metrics.
Although the BLEU score of Baichuan2Sum for the final summary and agent
summary is not as good as Bart-GLC, the gap between them is not very large.
For the text summarization task, especially for the diaglogue summarization
with multiple role, the ROUGE-1 scores of Baichuan2Sum (60.72/63.01/56.21) are
high for practical application in real-world scenarios.

TABLE V: Evaluation on CSDS Dataset CSDS | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU | BERTScore
---|---|---|---|---|---
PGN | 55.58/53.55/50.20 | 39.19/37.06/35.12 | 53.46/51.05/47.59 | 30.03/29.64/28.25 | 77.96/ 78.68/76.13
BERT | 53.87/52.72/49.57 | 37.59/36.39/33.82 | 52.40/50.44/46.83 | 29.90/30.17/26.99 | 78.52/ 79.23/76.39
BART | 59.07/58.78/53.89 | 43.72/43.59/40.24 | 57.11/56.86/50.85 | 34.33/34.26/31.88 | 79.74/ 80.67/77.31
BART-GLC | 60.07/61.42/54.59 | 44.67/45.83/40.02 | 58.10/59.25/52.43 | 35.89/36.43/32.58 | 80.10/ 81.83/77.61
Baichuan2-Sum | 60.36/62.97/56.08 | 45.33/47.05/41.11 | 58.69/60.67/53.55 | 33.90/36.92/30.79 | 80.53/82.10/77.61
Baichuan2-Sum(NEFTune) | 60.72/63.01/56.21 | 45.50/47.53/41.36 | 58.66/60.92/53.83 | 33.92/36.84/30.96 | 80.67/82.48/78.68

The Table 6 shows the evaluation results of both baseline models and
Baichuan2Sum model on SAMSUM dataset. Compared to CSDS dataset, SAMSUM dataset
is less complex, as it has a shorter average token length and only one type of
summarization, resulting in relatively higher scores over all metrics. The
Rouge-1, Rouge-2, and Rouge-L scores of Baichuan2 model have seen significant
improvement compared to other models, such as Bart-GLC. This indicates that
the generated text has a high degree of lexical similarity with the reference
text. Compared to other baseline models, Baichuan2Sum’s BLEU and BERT Score
are at a moderate level despite significantly leading in Rouge scores. This
could be due to a lack of diversity in the generated text.

TABLE VI: Evaluation on SAMSUM Dataset SAMSUM | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU | BERTScore
---|---|---|---|---|---
PGN | 40.08 | 15.28 | 36.63 | 37.49 | 80.67
BERT | 50.34 | 24.71 | 46.63 | 46.98 | 88.72
BART | 53.12 | 27.95 | 49.15 | 49.28 | 92.14
BART-GLC | 53.74 | 28.83 | 49.62 | 50.36 | 92.77
Baichuan2-Sum | 74.38 | 60.72 | 58.21 | 46.74 | 83.28
Baichuan2-Sum(NEFTune) | 74.51 | 60.87 | 58.26 | 46.51 | 84.19

###  IV-F Human Evaluation Results and Analysis

We adopt the human evaluation ways proposed by Liang et. al [8] to access the
performance on the CSDS and SAMSUM datasets. Specifically, we randomly select
100 samples from each dataset and invite five NLP researchers to conduct a
comparative analysis. The comparison is proposed model over previous best
model Bart-GLC and the result will fall into three cases: win, tie and loss,
meaning that our model is better, equal and weaker to the Bart-GLC model. The
evaluation standards focus on the three aspects:

  * •

Accuracy: This refers to how well the summary captures the essential
information and facts presented in the original text.

  * •

Coherence: A good summary should be coherent and logically structured. It
should flow smoothly, with ideas and sentences connected in a way that makes
sense.

  * •

Grammatical Correctness: The summary should be free of grammatical errors,
ensuring it is professionally presented.

We collect research results and draw the results in Figure 7. From the Figure
7, we can draw two conclusions. The Baichuan2Sum model demonstrates better
performance than BART-GLC on both datasets. Compared to the CSDS dataset, the
Baichuan2Sum model shows a greater lead in performance on the SAMSUM dataset.

Figure 6: Human Evaluation Result on CSDS and SAMSUM dataset

##  V Limitation

Even though our model achieves good result for the two dataset, there still
exists some limitations for our work. Due to the limitation of computer
resource, we only trained on the Baichuan2-7B model. Firstly, many recent NLP
studies show that model size can affect the result and a larger model usually
has a better performance. Hence, to improve the generation result, more large
language models which have more parameters can be explored. Secondly, the
experiment indicates that the proposed model still has room for improvement on
both the BLEU and BERT Score metrics. Given that the diversity of text
generation is positively correlated with these two metrics, we can use higher
temperature scores and top-p values during the model’s generation, or
increases the beam size for beam search.

##  VI Conclusion

In this paper, we instruction finetune the Baichuan2-7B model for two dialogue
summarization datasets. Before training model, we create various instruction
templates and generated corresponding datasets. Besides, the NEFTune technique
is applied to add noisy on embedding layer to improve the model performance.

In the CSDS dataset, the Baichuan2Sum (NEFTune) model generally outperforms
the other model across ROUGE-1, ROUGE-2, ROUGE-L, BLEU, and BERTScore metrics,
showing an approximately 1% increase in each. The model also demonstrates the
superior performance in all metrics on SAMSUM dataset, particularly
improvements of 21% in ROUGE-1, 32% in ROUGE-2, 9% in ROUGE-L compared to
other baselines. Additionally, human experiments show that our proposed model
has best performance in summarization task across accuracy, coherence, and
grammatical correctness metrics.

The model is also effective, which is trained on only one RTX4090 GPU in less
than 6 hours for CSDS and 4 hours for SAMSUM. Finally, to facilitate research
by a wider audience, we have made our code publicly available on GitHub and
provided details about the model and experiments in our paper.

## References

  * [1] Zou Y, Zhao L, Kang Y, et al. Topic-oriented spoken dialogue summarization for customer service with saliency-aware topic modeling[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(16): 14665-14673.
  * [2] Lin H, Ma L, Zhu J, et al. CSDS: A fine-grained Chinese dataset for customer service dialogue summarization[J]. ar**v preprint ar**v:2108.13139, 2021.
  * [3] Feng X, Feng X, Qin L, et al. Language model as an annotator: Exploring DialoGPT for dialogue summarization[J]. ar**v preprint ar**v:2105.12544, 2021.
  * [4] Feng X, Feng X, Qin B, et al. Dialogue discourse-aware graph convolutional networks for abstractive meeting summarization[J]. ar**v preprint ar**v:2012.03502, 2020, 17.
  * [5] Liang X, Bian C, Wu S, et al. Towards Modeling Role-Aware Centrality for Dialogue Summarization[C]//Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing. 2022: 43-50.
  * [6] Lin H, Zhu J, **ang L, et al. Other roles matter! enhancing role-oriented dialogue summarization via role interactions[J]. ar**v preprint ar**v:2205.13190, 2022.
  * [7] Zou Y, Lin J, Zhao L, et al. Unsupervised summarization for chat logs with topic-oriented ranking and context-aware auto-encoders[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(16): 14674-14682.
  * [8] Liang X, Wu S, Cui C, et al. Enhancing Dialogue Summarization with Topic-Aware Global-and Local-Level Centrality[J]. arXiv preprint arXiv:2301.12376, 2023.
  * [9] Zhong M, Liu Y, Xu Y, et al. Dialoglm: Pre-trained model for long dialogue understanding and summarization[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(10): 11765-11773.
  * [10] Liu P J, Manning C D. Get to the point: Summarization with pointer-generator networks[C]//Proc. Conf. Annu. Meeting Assoc. Comput. Linguistics (ACL). 2017: 1073-1083.
  * [11] Kenton J D M W C, Toutanova L K. Bert: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of naacL-HLT. 2019, 1: 2.
  * [12] Lewis M, Liu Y, Goyal N, et al. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020: 7871-7880.
  * [13] Brown T, Mann B, Ryder N, et al. Language models are few-shot learners[J]. Advances in neural information processing systems, 2020, 33: 1877-1901.
  * [14] Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models[J]. ar**v preprint ar**v:2307.09288, 2023.
  * [15] Yang A, **ao B, Wang B, et al. Baichuan 2: Open large-scale language models[J]. ar**v preprint ar**v:2309.10305, 2023.
  * [16] Du Z, Qian Y, Liu X, et al. Glm: General language model pretraining with autoregressive blank infilling[J]. ar**v preprint ar**v:2103.10360, 2021.
  * [17] Gliwa B, Mochol I, Biesek M, et al. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization[J]. ar**v preprint ar**v:1911.12237, 2019.
  * [18] Jain N, Chiang P, Wen Y, et al. NEFTune: Noisy Embeddings Improve Instruction Finetuning[J]. ar**v preprint ar**v:2310.05914, 2023.
  * [19] Workshop B S, Scao T L, Fan A, et al. Bloom: A 176b-parameter open-access multilingual language model[J]. ar**v preprint ar**v:2211.05100, 2022.
  * [20] Liu Y, Lapata M. Text summarization with pretrained encoders[J]. ar**v preprint ar**v:1908.08345, 2019.
  * [21] Wang Y, Kordi Y, Mishra S, et al. Self-instruct: Aligning language model with self generated instructions[J]. arXiv preprint arXiv:2212.10560, 2022.
  * [22] https://crfm.stanford.edu/2023/03/13/alpaca.html
  * [23] Liu H, Li C, Wu Q, et al. Visual instruction tuning[J]. arXiv preprint arXiv:2304.08485, 2023.
  * [24] Su J, Ahmed M, Lu Y, et al. Roformer: Enhanced transformer with rotary position embedding[J]. Neurocomputing, 2024, 568: 127063.
  * [25] Shazeer N. Glu variants improve transformer[J]. arXiv preprint arXiv:2002.05202, 2020.
  * [26] Rabe M N, Staats C. Self-attention Does Not Need $O(n^{2})$ Memory[J]. arXiv preprint arXiv:2112.05682, 2021.
  * [27] Zhang B, Sennrich R. Root mean square layer normalization[J]. Advances in Neural Information Processing Systems, 2019, 32.
  * [28] Lin C Y. Rouge: A package for automatic evaluation of summaries[C]//Text summarization branches out. 2004: 74-81.
  * [29] Papineni K, Roukos S, Ward T, et al. Bleu: a method for automatic evaluation of machine translation[C]//Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 2002: 311-318.
  * [30] Zhang T, Kishore V, Wu F, et al. Bertscore: Evaluating text generation with bert[J]. arXiv preprint arXiv:1904.09675, 2019.
  * [31] Hu E, Shen Y, Wallis P, et al. LoRA: Low-Rank Adaptation of Large Language Models[J]. 2021.
  * [32] Liu X, Ji K, Fu Y, et al. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks[J]. arXiv preprint arXiv:2110.07602, 2021.
  * [33] Liu X, Zheng Y, Du Z, et al. GPT understands, too[J]. AI Open, 2023.
  * [34] Kudo T, Richardson J. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing[J]. arXiv preprint arXiv:1808.06226, 2018.
  * [35] Sennrich R, Haddow B, Birch A. Neural machine translation of rare words with subword units[J]. arXiv preprint arXiv:1508.07909, 2015.
  * [36] Dubois Y, Li X, Taori R, et al. Alpacafarm: A simulation framework for methods that learn from human feedback[J]. arXiv preprint arXiv:2305.14387, 2023.
  * [37] Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models[J]. arXiv preprint arXiv:2307.09288, 2023.
  * [38] Jain N, Chiang P, Wen Y, et al. NEFTune: Noisy Embeddings Improve Instruction Finetuning[J]. arXiv preprint arXiv:2310.05914, 2023.
  * [39] Hugging Face. Enhance model’s performances using NEFTune. https://huggingface.co/docs/trl/main/en/sft_trainer, 2023
  * [40] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30.


while the decomposition used for $\mathcal{K}_{\mathrm{L}}$ is
$\mathbb{V}$-orthogonal, the decomposition for $\mathcal{K}_{\mathrm{KP}}$ is
$\ell^{2}$-orthogonal—the computation Eq. 4.30 and Lemma 4.33 exploit this
heavily. Nevertheless, we used the $\mathbb{V}$-inner product in the existence
result for $\mathcal{K}_{\mathrm{KP}}$. This geometric discrepancy is
reflected in condition (i) above.

  2. (ii)

Using the Cauchy–Schwarz inequality it is clear that $|\langle
t^{0},t^{\angle}\rangle_{\mathbb{V}}|<\|t^{0}\|_{\mathbb{V}}\|t^{\angle}\|_{\mathbb{V}}$
for all $t^{1}\in\mathbb{V}^{1}$, due to the fact that
$\mathbb{V}^{1}=\mathbb{V}^{0}\oplus\mathbb{V}^{\angle}$. The maximum of the
function $t^{1}\mapsto\frac{\langle
t^{0},t^{\angle}\rangle_{\mathbb{V}}}{\|t^{0}\|_{\mathbb{V}}\|t^{\angle}\|_{\mathbb{V}}}$
is attained on $\|t^{0}\|_{\mathbb{V}}=1$, $\|t^{\angle}\|_{\mathbb{V}}=1$ and
this maximum may be taken as the $0<g<1$ of condition (i).

  3. (iii)

In the coercive case, i.e. when Eq. 4.36 holds with $\alpha=0$, we have
$\Theta_{0}=I$, so that $\theta_{0}=\theta_{\angle}=0$. Letting
$\varepsilon\to 0$, condition (i) simplifies to

$\eta=\frac{(1-g)(\gamma_{0}-\frac{1}{2}M_{\delta}\delta)}{\Delta(t_{*}^{1})+M_{\delta}\delta}>\frac{1}{2}.$

This last condition in turn reduces to $\gamma_{0}>M_{\delta}\delta$ as $g$
and $\Delta(t_{*})$ approaches zero.

  4. (iv)

It is interesting to note that while the linear homotopy
$\mathcal{K}_{\mathrm{L}}$ involves the “targeted” solution $t_{*}^{1}$, the
KP homotopy $\mathcal{K}_{\mathrm{KP}}$ does not. In this sense,
$\mathcal{K}_{\mathrm{KP}}$ is “universal”.

  5. (v)

The result is straightforward to extend to the complex case.

  6. (vi)

A careful inspection of the proof shows that the result can be generalized to
the case when $\mathbb{V}^{1}=\mathbb{V}^{0}\oplus\mathbb{V}^{\angle}$ is an
arbitrary $\ell^{2}$-orthogonal direct sum decomposition as long as the
homotopy $\mathcal{K}_{\mathrm{KP}}$ is _defined_ via Eq. 4.34 (cf. Eq. 4.26).
In this more general case, $\Delta(t_{*}^{1})$ is given by a more complicated
expression.

The constant $\Delta(t_{*}^{1})$ also deserves some explanation. Roughly
speaking, the “defect” $\Delta(t_{*}^{1})$ measures how much the subspace
$\mathfrak{V}^{0}\subset\mathfrak{V}$ deviates from being an invariant
subspace of the operator
$(\mathcal{H}_{K}+[(T_{*}^{1})_{1}^{\dagger},\mathcal{H}_{K}])_{\mathfrak{V}}:\mathfrak{V}\to\mathfrak{V}$.
In addition, we can invoke Eq. 4.3 to write

$\Delta(t_{*}^{1})=\max_{\begin{subarray}{c}\|u^{0}\|_{\mathbb{V}}=1\\\
\|v^{\angle}\|_{\mathbb{V}}=1\end{subarray}}|\langle(\mathcal{W}_{K}+[(T_{*}^{1})_{1}^{\dagger},\mathcal{W}_{K}])U^{0}\Phi_{0},V^{\angle}\Phi_{0}\rangle|.$

Said differently,
$\Delta(t_{*}^{1})=\|(\mathcal{W}_{K}+[(T_{*}^{1})_{1}^{\dagger},\mathcal{W}_{K}])_{\mathfrak{V}}\|_{\mathcal{L}(\mathfrak{V}^{0},\mathfrak{V}^{\perp})}$.
Note that the term $[(T_{*}^{1})_{1}^{\dagger},\mathcal{W}_{K}]$ can be
eliminated via orbital rotations (since $(t_{*}^{1})_{1}=0$ can be achieved
according to the Thouless theorem), as it involves single excitations only, in
which case the amplitude dependence of $\Delta$ is removed. Hence, $\Delta$
quantifies how much the operator $(\mathcal{W}_{K})_{\mathfrak{V}}$ leaves
$\mathfrak{V}^{0}$ invariant.

We summarize the above existence result in the corollary below that holds
under the following structural assumptions.

Assumption (KPA). $\Delta(t_{*}^{1})$ can be made sufficiently small by an
appropriate choice of the orbital basis and truncation level $1\leq\rho<N$.

Assumption (KPB). There is a $\delta_{0}>0$ such that $M_{\delta_{0}}$ is
sufficiently small.

######  Corollary 4.36.

Suppose that $t_{*}^{1}\in\mathbb{V}^{1}$ is a non-degenerate zero of
$\mathcal{A}$ and that Assumptions (KPA) and (KPB) hold. If
$\|t_{*}^{\angle}\|_{\mathbb{V}}$ is sufficiently small, then there exists
$t_{**}^{1}\in\mathbb{V}^{1}$ in a neigborhood of $t_{*}^{1}$ such that
$\mathcal{K}_{\mathrm{KP}}(t_{**}^{1},0)=0$.

######  Remark 4.37.

Recall that $M_{\delta}$ only involves the second derivative
$\mathcal{A}^{\prime\prime}$ near $t_{*}^{1}$ (see Eq. 4.15), so that (KPB)
can be viewed as a “perturbative” assumption. Further, as we noted in Remark
4.9, $M_{\delta}$ involves the mapping
$\zeta\mapsto[[\mathcal{W}_{K}(t_{*}^{1}+\zeta),\cdot],\cdot]\Phi_{0}$.
Therefore, (KPB) may be viewed as the higher-order generalization of
assumption on the smallness of the local Lipschitz constant of the mapping
$\zeta\mapsto\mathcal{W}_{K}(t_{*}^{1}+\zeta)\Phi_{0}$ in [36, Assumption BII]
(see also Remark 4.41 (v)).

In the last step of the _proof_ of Theorem 4.34, we could have invoked Theorem
4.28 instead to obtain the existence of a connected component $\mathcal{C}$ of
the zero set $\mathcal{Z}(\mathcal{K}_{\mathrm{KP}})$, such that
$\mathcal{C}\cap(\mathcal{Z}(\mathcal{K}_{\mathrm{KP}}))_{j}\neq\emptyset$ for
$j=0,1$. Under the above assumptions, this provides a theoretical basis for
the “solution trajectories” observed in [32]. Further, combining Theorem 4.34
with Eq. 4.29, we get for $t_{**}^{1}=t_{**}^{1}(0)$,

$\sum_{t_{**}^{1}\in\mathcal{C}\cap(\mathcal{Z}(\mathcal{K}_{\mathrm{KP}}))_{0}}i(\mathcal{K}_{\mathrm{KP}}(\cdot,0),t_{**}^{1})=i(\mathcal{A},t_{*}^{1}).$

Since we do not have uniqueness for $t_{**}^{1}$ in this case, it is possible
that the left-hand side may contain multiple terms which sum up to
$i(\mathcal{A},t_{*}^{1})$. In particular,
$i(\mathcal{K}_{\mathrm{KP}}(\cdot,0),t_{**}^{1})$ does not need to be
$i(\mathcal{A},t_{*}^{1})$.

To close this section, we calculate the topological index for a non-degenerate
zero at the $\lambda=0$ endpoint of the KP homotopy. Note that the index at
$\lambda=1$ is simply given by Theorem 4.13 and Theorem 4.18.

Fix $t^{1}\in\mathbb{V}^{1}$ and define the operator
$\widehat{\mathcal{H}_{K}}(t^{1})$ (not to be confused with Eq. 4.10 in a
different context),

$\widehat{\mathcal{H}_{K}}(t^{1})=\mathcal{H}_{K}(t^{1})-\sum_{\alpha\in\Xi(G^{0})}\langle\mathcal{H}_{K}(t^{1})\Phi_{0},\Phi_{\alpha}\rangle
X_{\alpha}.$ (4.39)

Notice that
$\langle\widehat{\mathcal{H}_{K}}(t^{1})\Phi_{0},S^{0}\Phi_{0}\rangle=0$ for
all $s^{0}\in\mathbb{V}^{0}$. Define the linear mapping
$\widehat{\mathcal{H}_{K}}(t^{1})_{\mathfrak{V}^{0},\mathfrak{V}^{\angle}}:\mathfrak{V}^{0}\to\mathfrak{V}^{\angle}$
via
$\langle\widehat{\mathcal{H}_{K}}(t^{1})_{\mathfrak{V}^{0},\mathfrak{V}^{\angle}}\Psi,\Psi^{\prime}\rangle=\langle\widehat{\mathcal{H}_{K}}(t^{1})\Psi,\Psi^{\prime}\rangle$
for all $\Psi\in\mathfrak{V}^{0}$ and $\Psi^{\prime}\in\mathfrak{V}^{\angle}$.
We first calculate the derivative of $\mathcal{K}_{\mathrm{KP}}(\cdot,0)$.

######  Lemma 4.38.

The derivative
$\partial_{1}\mathcal{K}_{\mathrm{KP}}(t_{**}^{1},0):\mathbb{V}^{1}\to(\mathbb{V}^{1})^{*}$
of $\mathcal{K}_{\mathrm{KP}}(\cdot,0)$ at a zero $t_{**}^{1}$ is given by

$\displaystyle\langle\partial_{1}\mathcal{K}_{\mathrm{KP}}(t^{1}_{*},0)u^{1},v^{1}\rangle=\Big{\langle}\begin{pmatrix}\mathcal{H}_{K}(t_{**}^{0})_{\mathfrak{V}^{0}}-\mathcal{E}_{\mathrm{CC}}(t_{**}^{0})&0\\\
\widehat{\mathcal{H}_{K}}(t_{**}^{1})_{\mathfrak{V}^{0},\mathfrak{V}^{\angle}}&\widehat{\mathcal{H}_{K}}(t_{**}^{1})_{\mathfrak{V}^{\angle}}-\mathcal{E}_{\mathrm{CC}}(t_{**}^{1})\end{pmatrix}\begin{pmatrix}U^{0}\Phi_{0}\\\
U^{\angle}\Phi_{0}\end{pmatrix},\begin{pmatrix}V^{0}\Phi_{0}\\\
V^{\angle}\Phi_{0}\end{pmatrix}\Big{\rangle}$

for all $u^{1},v^{1}\in\mathbb{V}^{1}$.

###### Proof.

A calculation analogous to the one in the proof of Lemma 4.6 shows that
$D(t^{1}):=\partial_{1}\mathcal{K}_{\mathrm{KP}}(t^{1},0):\mathbb{V}^{1}\to(\mathbb{V}^{1})^{*}$
is given by

$\displaystyle\langle D(t^{1})u^{1},v^{1}\rangle$
$\displaystyle=\langle[\mathcal{H}_{K}(t^{0}),U^{0}]\Phi_{0},V^{0}\Phi_{0}\rangle+\langle[\mathcal{H}_{K}(t^{1}),U^{1}]\Phi_{0},V^{\angle}\Phi_{0}\rangle$
(4.40) $\displaystyle=:D_{1}(t^{1})+D_{2}(t^{1})$

for all $t^{1},u^{1},v^{1}\in\mathbb{V}^{1}$. We set $t^{1}=t^{1}_{**}$ and
evaluate $D_{1}$ and $D_{2}$. Write

$(U^{0})^{\dagger}V^{0}\Phi_{0}=\sum_{\alpha\in\Xi(G^{1})\cup\\{0\\}}\langle
U^{0}\Phi_{\alpha},V^{0}\Phi_{0}\rangle\Phi_{\alpha},$

from which,

$\displaystyle\langle
U^{0}\mathcal{H}_{K}(t_{**}^{0})\Phi_{0},V^{0}\Phi_{0}\rangle$
$\displaystyle=\langle\mathcal{H}_{K}(t_{**}^{0})\Phi_{0},(U^{0})^{\dagger}V^{0}\Phi_{0}\rangle$
$\displaystyle=\Bigg{(}\sum_{\alpha\in\Xi(G^{0})}+\sum_{\alpha\in\Xi(G^{0})^{c}}\Bigg{)}\langle\mathcal{H}_{K}(t_{**}^{0})\Phi_{0},\Phi_{\alpha}\rangle\langle
U^{0}\Phi_{\alpha},V^{0}\Phi_{0}\rangle+\mathcal{E}_{\mathrm{CC}}(t_{**}^{0})\langle
U^{0}\Phi_{0},V^{0}\Phi_{0}\rangle.$

Here, the first sum vanishes because of Eq. 4.32 and the second by the
orthogonality of $\mathbb{V}^{0}$ and $\mathbb{V}^{\angle}$. Consequently,

$D_{1}(t_{**}^{1})=\langle(\mathcal{H}_{K}(t_{**}^{0})-\mathcal{E}_{\mathrm{CC}}(t_{**}^{0}))U^{0}\Phi_{0},V^{0}\Phi_{0}\rangle.$

Analogously, Eq. 4.33 implies that

$\displaystyle\langle
U^{\angle}\mathcal{H}_{K}(t_{**}^{1})\Phi_{0},V^{\angle}\Phi_{0}\rangle$
$\displaystyle=\sum_{\alpha\in\Xi(G^{0})}\langle\mathcal{H}_{K}(t_{**}^{1})\Phi_{0},\Phi_{\alpha}\rangle\langle
X_{\alpha}U^{\angle}\Phi_{0},V^{\angle}\Phi_{0}\rangle+\mathcal{E}_{\mathrm{CC}}(t_{**}^{1})\langle
U^{\angle}\Phi_{0},V^{\angle}\Phi_{0}\rangle,$

and

$\langle
U^{0}\mathcal{H}_{K}(t_{**}^{1})\Phi_{0},V^{\angle}\Phi_{0}\rangle=\sum_{\alpha\in\Xi(G^{0})}\langle\mathcal{H}_{K}(t_{**}^{1})\Phi_{0},\Phi_{\alpha}\rangle\langle
X_{\alpha}U^{0}\Phi_{0},V^{\angle}\Phi_{0}\rangle.$

Thus,

$\displaystyle
D_{2}(t_{**}^{1})=\langle(\widehat{\mathcal{H}_{K}}(t_{**}^{1})-\mathcal{E}_{\mathrm{CC}}(t_{**}^{1}))U^{\angle}\Phi_{0},V^{\angle}\Phi_{0}\rangle+\langle\widehat{\mathcal{H}_{K}}(t_{**}^{1})U^{0}\Phi_{0},V^{\angle}\Phi_{0}\rangle,$

and the stated expression now follows. ∎

The preceding Lemma implies the index formula for the KP homotopy.

######  Theorem 4.39 (Index formula for KP – non-degenerate case).

Suppose that $t_{**}^{1}\in\mathbb{V}^{1}$ is a zero of
$\mathcal{K}_{\mathrm{KP}}(\cdot,0)$. Then $t_{**}^{1}$ is non-degenerate if
and only if the conditions

  1. (I)

$\mathcal{E}_{\mathrm{CC}}(t_{**}^{0})\not\in\sigma(\mathcal{H}_{K}(t_{**}^{0})_{\mathfrak{V}^{0}})$,

  2. (II)

$\mathcal{E}_{\mathrm{CC}}(t_{**}^{1})\not\in\sigma(\widehat{\mathcal{H}_{K}}(t_{**}^{1})_{\mathfrak{V}^{\angle}})$

both hold true, and in this case the topological index of
$\mathcal{K}_{\mathrm{KP}}(\cdot,0)$ at $t_{**}^{1}$ is given by
$i(\mathcal{K}_{\mathrm{KP}}(\cdot,0),t_{**}^{1})=(-1)^{\nu^{0}+\nu^{\angle}}$,
where

$\displaystyle\nu^{0}$
$\displaystyle=|\\{j:\mathcal{E}_{j}(\mathcal{H}_{K}(t_{**}^{0})_{\mathfrak{V}^{0}})\in\mathbb{R},\;\mathcal{E}_{j}(\mathcal{H}_{K}(t_{**}^{0})_{\mathfrak{V}^{0}})<\mathcal{E}_{\mathrm{CC}}(t_{**}^{0})\\}|,$
$\displaystyle\nu^{\angle}$
$\displaystyle=|\\{j:\mathcal{E}_{j}(\widehat{\mathcal{H}_{K}}(t_{**}^{1})_{\mathfrak{V}^{\angle}})\in\mathbb{R},\;\mathcal{E}_{j}(\widehat{\mathcal{H}_{K}}(t_{**}^{1})_{\mathfrak{V}^{\angle}})<\mathcal{E}_{\mathrm{CC}}(t_{**}^{1})\\}|.$

###### Proof.

The proof follows from Lemma 4.38 along similar lines as Theorem 4.13,

$\displaystyle i(\mathcal{K}_{\mathrm{KP}}(\cdot,0),t_{**}^{1})$
$\displaystyle=\operatorname{sgn}\prod_{j\geq
0}(\mathcal{E}_{j}(\mathcal{H}_{K}(t_{**}^{0})_{\mathfrak{V}^{0}})-\mathcal{E}_{\mathrm{CC}}(t_{**}^{0}))\operatorname{sgn}\prod_{j\geq
0}(\mathcal{E}_{j}(\widehat{\mathcal{H}_{K}}(t_{**}^{1})_{\mathfrak{V}^{\angle}})-\mathcal{E}_{\mathrm{CC}}(t_{**}^{1})).$

∎

###  4.6. An energy error estimate

In this this section we derive an energy error estimate for general
eigenstates for the KP homotopy using the results of Appendix B.

######  Theorem 4.40 (Energy error estimate).

Let $\mathbb{V}^{1}=\mathbb{V}(G^{\mathrm{full}})$ and suppose that
$t_{*}^{1}\in\mathbb{V}^{1}$ is a zero of $\mathcal{A}$, and that
$t_{**}^{1}\in\mathbb{V}^{1}$ is a zero of
$\mathcal{K}_{\mathrm{KP}}(\cdot,0)=0$. If the nonorthogonality condition
$\langle e^{T^{0}_{**}}\Phi_{0},e^{T_{*}^{1}}\Phi_{0}\rangle\neq 0$ holds
true, then

$|\mathcal{E}_{\mathrm{CC}}(t_{**}^{1})-\mathcal{E}_{\mathrm{CC}}(t_{*}^{1})|\leq
C(t_{**}^{1},t_{*}^{1})\|t_{**}^{\angle}\|_{\mathbb{V}},$ (4.41)

where

$C(t_{**}^{1},t_{*}^{1})=(C^{2}+M(t_{**}^{1}))\frac{\|\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}})^{\dagger}\Pi_{\mathfrak{V}^{\angle}}e^{T_{*}^{1}}\Phi_{0}\|_{\mathfrak{H}^{1}}}{|\langle
e^{T^{0}_{**}}\Phi_{0},e^{T_{*}^{1}}\Phi_{0}\rangle|}$

is bounded as $\|t_{**}^{\angle}\|_{\mathbb{V}}\to 0$. Here, $C$ is the norm
equivalence constant from Remark 2.6 and
$M(t_{**}^{1})=\max_{\xi\in[t_{**}^{0},t_{**}^{1}]}\|u^{1}\mapsto\Pi_{\mathfrak{V}^{\angle}}[\mathcal{W}_{K}(\xi),U^{1}]\|_{\mathcal{L}(\mathbb{V},\mathfrak{H}^{-1})}$.

###### Proof.

Setting $\Psi=e^{T_{*}^{1}}\Phi_{0}$ and $\lambda=0$ in Theorem B.1 (II), we
have

$|\mathcal{E}_{\mathrm{CC}}(t_{**}^{1})-\mathcal{E}_{\mathrm{CC}}(t_{*}^{1})|=\frac{|\langle(\mathcal{H}_{K}(t^{1}_{**})-\mathcal{H}_{K}(t^{0}_{**}))\Phi_{0},\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}})^{\dagger}\Pi_{\mathfrak{V}^{\angle}}e^{T_{*}^{1}}\Phi_{0}\rangle|}{|\langle
e^{T^{0}_{**}}\Phi_{0},e^{T_{*}^{1}}\Phi_{0}\rangle|}.$ (4.42)

Note that using Eq. 4.3 we may write

$\displaystyle(\mathcal{H}_{K}(t^{1}_{**})-\mathcal{H}_{K}(t^{0}_{**}))\Phi_{0}$
$\displaystyle=[\mathcal{F}_{K},T_{**}^{\angle}]\Phi_{0}+(\mathcal{W}_{K}(t^{1}_{**})-\mathcal{W}_{K}(t^{0}_{**}))\Phi_{0}$
$\displaystyle=\sum_{\gamma\in\Xi(G^{\angle})}\varepsilon_{\gamma}(t_{**}^{\angle})_{\gamma}\Phi_{\gamma}+(\mathcal{W}_{K}(t^{1}_{**})-\mathcal{W}_{K}(t^{0}_{**}))\Phi_{0}.$

Hence, we can bound the numerator of the right-hand side of Eq. 4.42 as

$\displaystyle\Big{|}\sum_{\gamma\in\Xi(G^{\angle})}\varepsilon_{\gamma}(t_{**}^{\angle})_{\gamma}\langle\Phi_{\gamma},\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}})^{\dagger}\Pi_{\mathfrak{V}^{\angle}}e^{T_{*}^{1}}\Phi_{0}\rangle\Big{|}+\Big{|}\langle(\mathcal{W}_{K}(t^{1}_{**})-\mathcal{W}_{K}(t^{0}_{**}))\Phi_{0},\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}})^{\dagger}\Pi_{\mathfrak{V}^{\angle}}e^{T_{*}^{1}}\Phi_{0}\rangle\Big{|}.$

Using the Cauchy–Schwarz inequality, the first term may be further bounded as

$\displaystyle\sum_{\gamma\in\Xi(G^{\angle})}\varepsilon_{\gamma}(t_{**}^{\angle})_{\gamma}\langle\Phi_{\gamma},\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}})^{\dagger}\Pi_{\mathfrak{V}^{\angle}}e^{T_{*}^{1}}\Phi_{0}\rangle\leq{\left|\kern-1.07639pt\left|\kern-1.07639pt\left|t_{**}^{\angle}\right|\kern-1.07639pt\right|\kern-1.07639pt\right|}{\left|\kern-1.07639pt\left|\kern-1.07639pt\left|\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}})^{\dagger}\Pi_{\mathfrak{V}^{\angle}}e^{T_{*}^{1}}\Phi_{0}\right|\kern-1.07639pt\right|\kern-1.07639pt\right|},$

where the Fock norm
${\left|\kern-1.07639pt\left|\kern-1.07639pt\left|\cdot\right|\kern-1.07639pt\right|\kern-1.07639pt\right|}$
was defined in Remark 2.6. For the second term, we use the intermediate value
inequality to obtain the bound
$M(t_{**}^{1})\|t_{**}^{\angle}\|_{\mathbb{V}}\|\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}})^{\dagger}\Pi_{\mathfrak{V}^{\angle}}e^{T_{*}^{1}}\Phi_{0}\|_{\mathfrak{H}^{1}}$.
∎

######  Remark 4.41.

  1. (i)

If the nonorthogonality condition holds and $t_{**}^{\angle}=0$, then
according to Eq. 4.32-Eq. 4.33, $t_{**}^{0}$ is an FCC solution such that
$\langle e^{T^{0}_{**}}\Phi_{0},e^{T_{*}^{1}}\Phi_{0}\rangle\neq 0$. In this
case, Eq. 4.41 implies that the energy error is zero:
$\mathcal{E}_{\mathrm{CC}}(t_{**}^{0})=\mathcal{E}_{\mathrm{CC}}(t_{*}^{1})$.

  2. (ii)

In the practically relevant case $\rho\geq 2$, using Eq. B.1, we have that
$\mathcal{E}_{\mathrm{CC}}(t_{**}^{1})=\mathcal{E}_{\mathrm{CC}}(t_{**}^{0})$
so the left-hand side of Eq. 4.41 does not involve $t_{**}^{\angle}$ at all.

  3. (iii)

The appearance of the quantity $\|t_{**}^{\angle}\|_{\mathbb{V}}$ allows us to
view the auxiliary equation Eq. 4.33 as providing an _a posteriori_ error
estimate.

  4. (iv)

If the nonorthogonality condition does not hold, i.e. $\langle
e^{T^{0}_{**}}\Phi_{0},e^{T_{*}^{1}}\Phi_{0}\rangle=0$, then
$e^{T^{0}_{**}}\Phi_{0}$ and $e^{T_{*}^{1}}\Phi_{0}$ represent different
eigenstates if $t_{*}^{1}$ is assumed to be non-degenerate. While
$e^{T^{0}_{**}}\Phi_{0}$ itself does not satisfy the Schrödinger equation, it
must be viewed as an approximation to an eigenstate _different_ from
$e^{T_{*}^{1}}\Phi_{0}$.

  5. (v)

Note that a local Lipschitz assumption (on a ball including $t_{**}^{1}$ and
$t_{**}^{0}$) with constant $L$ on the mapping
$t\mapsto\mathcal{W}_{K}(t)\Phi_{0}$ can be used to obtain the result of the
theorem with $M$ replaced by $L$. Such an assumption is akin to Assumption
B.II in [36] where it is used for guaranteeing the local strong monotonicity
of the CC mapping.

##  5\. Conclusions and further work

In this second part of a series of two articles, we analyzed the SRCC method.
We provided background material for the setting of the quantum-mechanical
problems the SRCC method is aimed at in Section 2. Then, in Section 3, we gave
a brief summary of topological degree theory since this served as our main
tool for the analysis.

The main discussion is contained in Section 4, where we began with the
definition of the SRCC mapping and some elementary considerations in Section
4.1. We then considered the local properties of the SRCC mapping in Section
4.2. It turned out that the topological index of the CC mapping (Theorem 4.13)
is connected with the nonvariational property of the CC method (Remark 4.16),
and the eigenvalues of the Fock operator (Proposition 4.17). In the degenerate
case, the classic Leray reduction formula provided the topological index
(Theorem 4.18). We also discussed the case when the cluster amplitudes are
allowed to be complex in Section 4.3.

In Section 4.4, we discussed how certain homotopies can be used to analyze the
CC method, in particular to prove the existence of a truncated CC solution
through the use of topological degree theory. This was done using an idea
well-known both in nonlinear analysis and in quantum chemistry: that an
appropriate homotopy “connects” the truncated problem with the exact problem
(essentially the Schrödinger equation), therefore one is able to infer
(homotopy-invariant) information regarding the former problem from the latter.
As an introductory example, we considered the linear homotopy (Theorem 4.31).

Next, in Section 4.5, motivated by the works of the chemists Kowalski and
Piecuch, we considered a homotopy that connects CC mappings corresponding to
different truncation levels. Using this, we proved an existence result for the
said homotopy (Theorem 4.34), which also implies the existence of a truncated
CC solution under certain assumptions. The index formula for the KP homotopy
was also derived in the non-degenerate case (Theorem 4.39). Using a known
result about the KP homotopy (Appendix B), we also derived an energy error
estimate in Section 4.6.

Finally, let us discuss some possible directions of research. Clearly, it
would be interesting to extend our analysis to the infinite-dimensional case.
An obvious next step would be the analysis of the JM-MRCC method (see Section
4.2 of Part I). It would be also interesting to look at the Extended CC (ECC)
[1, 24] and the Unitary CC (UCC) methods [3].

##  Appendix A Proof of Theorem 4.34

We first prove that $\mathcal{K}_{\mathrm{KP}}(\cdot,\lambda)\neq 0$ on
$\partial D$ for all $\lambda\in[0,1]$. Set $t^{1}=t_{*}^{1}+r^{1}$ and
$s^{1}=\Theta_{\alpha}r^{1}$ in Eq. 4.34 where $r^{1}\in\partial D$, and write

$\displaystyle\langle\mathcal{K}_{\mathrm{KP}}(t_{*}^{1}+r^{1},\lambda),\Theta_{\alpha}r^{1}\rangle=\langle\mathcal{A}(t_{*}^{0}+r^{0}+\lambda
t_{*}^{\angle}+\lambda
r^{\angle}),\Pi_{0}\Theta_{\alpha}r^{1}\rangle+\langle\mathcal{A}(t_{*}^{1}+r^{1}),\Pi_{\angle}\Theta_{\alpha}r^{1}\rangle$
$\displaystyle=\langle\mathcal{A}(t_{*}^{0}+\lambda
t_{*}^{\angle}+r^{0}+\lambda
r^{\angle})-\mathcal{A}(t_{*}^{1}+r^{1}),\Pi_{0}\Theta_{\alpha}r^{1}\rangle+\langle\mathcal{A}(t_{*}^{1}+r^{1}),\Theta_{\alpha}r^{1}\rangle=:\mathrm{(I)}+\mathrm{(II)}.$

For (I), Taylor expansion around $t_{*}^{1}$ leads to

$\displaystyle\mathrm{(I)}$
$\displaystyle=(\lambda-1)\langle\mathcal{A}^{\prime}(t_{*}^{1})(t_{*}^{\angle}+r^{\angle}),\Pi_{0}\Theta_{\alpha}r^{1}\rangle+\langle\mathcal{R}_{2}(t_{*}^{1},(\lambda-1)t_{*}^{\angle}+r^{0}+\lambda
r^{\angle})-\mathcal{R}_{2}(t_{*}^{1},r^{1}),\Pi_{0}\Theta_{\alpha}r^{1}\rangle$
$\displaystyle\geq(\lambda-1)\langle\mathcal{A}^{\prime}(t_{*}^{1})(t_{*}^{\angle}+r^{\angle}),\Pi_{0}\Theta_{\alpha}r^{1}\rangle-M\|t_{*}^{\angle}+r^{\angle}\|\|\Pi_{0}\Theta_{\alpha}r^{1}\|$
$\displaystyle\geq-(\Delta(t_{*}^{1})+M)\|t_{*}^{\angle}+r^{\angle}\|_{\mathbb{V}}\|\Pi_{0}\Theta_{\alpha}r^{1}\|_{\mathbb{V}},$

where we used the intermediate value inequality. Here, letting
$\Psi^{0}\in\mathfrak{V}^{0}$ correspond to the amplitude
$\Pi_{0}\Theta_{\alpha}r^{1}$,

$\displaystyle\langle\mathcal{A}^{\prime}(t_{*}^{1})(t_{*}^{\angle}+r^{\angle}),\Pi_{0}\Theta_{\alpha}r^{1}\rangle$
$\displaystyle=\langle(\mathcal{H}_{K}(t_{*}^{1})-\mathcal{E}_{\mathrm{CC}}(t_{*}^{1}))(T_{*}^{\angle}+R^{\angle})\Phi_{0},\Psi^{0}\rangle=\langle\mathcal{H}_{K}(t_{*}^{1})(T_{*}^{\angle}+R^{\angle})\Phi_{0},\Psi^{0}\rangle$
$\displaystyle=\langle(\mathcal{H}_{K}+[\mathcal{H}_{K},T_{*}^{1}]+\ldots)(T_{*}^{\angle}+R^{\angle})\Phi_{0},\Psi^{0}\rangle\leq\Delta(t_{*}^{1})\|t_{*}^{\angle}+r^{\angle}\|_{\mathbb{V}}\|\Pi_{0}\Theta_{\alpha}r^{1}\|_{\mathbb{V}},$

where the first equality is Eq. 4.11, and in the last inequality we exploited
that $\mathcal{H}_{K}$ decreases the excitation rank at most by 2 (so that the
single amplitudes $(t_{*}^{1})_{1}$ of $t_{*}^{1}$ only contribute). Also, the
following estimate holds,

$\displaystyle M$
$\displaystyle=\max_{\xi\in[(\lambda-1)t_{*}^{\angle}+r^{0}+\lambda
r^{\angle},r^{1}]}\|\partial_{2}\mathcal{R}_{2}(t_{*}^{1},\xi)\|_{\mathcal{L}(\mathbb{V},\mathbb{V}^{*})}$
$\displaystyle=\max_{\xi\in[(\lambda-1)t_{*}^{\angle}+r^{0}+\lambda
r^{\angle},r^{1}]}\|\mathcal{A}^{\prime}(t_{*}+\xi)-\mathcal{A}^{\prime}(t_{*})\|_{\mathcal{L}(\mathbb{V},\mathbb{V}^{*})}$
$\displaystyle\leq\max_{\xi\in[(\lambda-1)t_{*}^{\angle}+r^{0}+\lambda
r^{\angle},r^{1}]}\|\xi\|_{\mathbb{V}}\max_{\zeta\in[0,\xi]}\|\mathcal{A}^{\prime\prime}(t_{*}^{1}+\zeta)\|_{\mathcal{L}(\mathbb{V}\times\mathbb{V},\mathbb{V}^{*})}$
$\displaystyle\leq
M_{\delta}\max_{\xi\in[(\lambda-1)t_{*}^{\angle}+r^{0}+\lambda
r^{\angle},r^{1}]}\|\xi\|_{\mathbb{V}}\leq
M_{\delta}(\|r^{0}\|_{\mathbb{V}}+\|r^{\angle}\|_{\mathbb{V}}+\varkappa)\leq
M_{\delta}\delta.$

For (II), we have using Eq. 4.36, Eq. 4.37 and Taylor’s theorem,

$\displaystyle\mathrm{(II)}$
$\displaystyle=\langle\mathcal{A}^{\prime}(t_{*}^{1})r^{1},\Theta_{\alpha}r^{1}\rangle-\langle\mathcal{R}_{2}(t_{*}^{1},r^{1}),\Theta_{\alpha}r^{1}\rangle\geq\gamma_{\alpha}\|r^{1}\|_{\mathbb{V}}^{2}-\tfrac{1}{2}M_{\delta}\|r^{1}\|_{\mathbb{V}}^{2}\|\Theta_{\alpha}r^{1}\|_{\mathbb{V}}$
$\displaystyle\geq(\gamma_{\alpha}-\tfrac{1}{2}M_{\delta}\|\Theta_{\alpha}\|_{\mathcal{L}(\mathbb{V})}\|r^{1}\|_{\mathbb{V}})\|r^{1}\|_{\mathbb{V}}^{2}\geq(\gamma_{\alpha}-\tfrac{1}{2}M_{\delta}\|\Theta_{\alpha}\|_{\mathcal{L}(\mathbb{V})}\delta)\|r^{1}\|_{\mathbb{V}}^{2}.$

In summary, using the definitions of $\theta_{0}$ and $\theta_{\angle}$, and
setting
$\widetilde{\gamma}=\gamma_{\alpha}-\tfrac{1}{2}M_{\delta}\|\Theta_{\alpha}\|_{\mathcal{L}(\mathbb{V})}\delta$,

$\displaystyle\langle$
$\displaystyle\mathcal{K}_{\mathrm{KP}}(t_{*}^{1}+r^{1},\lambda),\Theta_{\alpha}r^{1}\rangle$
$\displaystyle\geq\widetilde{\gamma}\|r^{1}\|_{\mathbb{V}}^{2}-(\Delta(t_{*}^{1})+M_{\delta}\delta)\|t_{*}^{\angle}+r^{\angle}\|_{\mathbb{V}}\|\Pi_{0}\Theta_{\alpha}r^{1}\|_{\mathbb{V}}$
$\displaystyle\geq\widetilde{\gamma}\|r^{1}\|_{\mathbb{V}}^{2}-\frac{\Delta(t_{*}^{1})+M_{\delta}\delta}{2}(\|t_{*}^{\angle}+r^{\angle}\|_{\mathbb{V}}^{2}+\|\Pi_{0}\Theta_{\alpha}r^{1}\|_{\mathbb{V}}^{2})$
$\displaystyle\geq(1-g)\widetilde{\gamma}(\|r^{0}\|_{\mathbb{V}}^{2}+\|r^{\angle}\|_{\mathbb{V}}^{2})-\frac{\Delta(t_{*}^{1})+M_{\delta}\delta}{2}\left(\varkappa^{2}+2\varkappa\|r^{\angle}\|_{\mathbb{V}}+\|r^{\angle}\|_{\mathbb{V}}^{2}+\|\Pi_{0}\Theta_{\alpha}r^{1}\|_{\mathbb{V}}^{2}\right)$
$\displaystyle\geq\left((1-g)\widetilde{\gamma}-\frac{\Delta(t_{*}^{1})+M_{\delta}\delta}{2}\Big{(}1+\max\\{\varepsilon+2(1+\varepsilon^{-1})\theta_{0},2(1+\varepsilon^{-1})\theta_{\angle}\\}\Big{)}\right)$
$\displaystyle\quad\times(\|r^{0}\|_{\mathbb{V}}^{2}+\|r^{\angle}\|_{\mathbb{V}}^{2})-\frac{\Delta(t_{*}^{1})+M_{\delta}\delta}{2}(\varkappa^{2}+\sqrt{2}\varkappa(\delta-\varkappa))$
$\displaystyle\geq\left((1-g)\widetilde{\gamma}-\frac{\Delta(t_{*}^{1})+M_{\delta}\delta}{2}\max\\{\varepsilon+2(1+\varepsilon^{-1})\theta_{0},2(1+\varepsilon^{-1})\theta_{\angle}\\}\right)\frac{(\delta-\varkappa)^{2}}{2}-\frac{\Delta(t_{*}^{1})+M_{\delta}\delta}{2}\left(\varkappa+\frac{\sqrt{2}}{2}(\delta-\varkappa)\right)^{2}.$

The positivity of the last expression follows from Eq. 4.38. We also used the
bound

$\displaystyle\|r^{\angle}\|_{\mathbb{V}}^{2}+\|\Pi_{0}\Theta_{\alpha}r^{1}\|_{\mathbb{V}}^{2}\leq(1+\varepsilon^{-1})\|\Pi_{0}(\Theta_{\alpha}-I)r^{1}\|_{\mathbb{V}}^{2}+(1+\varepsilon)\|r^{0}\|_{\mathbb{V}}^{2}+\|r^{\angle}\|_{\mathbb{V}}^{2}$
$\displaystyle\leq
2(1+\varepsilon^{-1})(\|\Pi_{0}(\Theta_{\alpha}-I)\Pi_{0}\|_{\mathcal{L}(\mathbb{V})}^{2}\|r^{0}\|_{\mathbb{V}}^{2}+\|\Pi_{0}(\Theta_{\alpha}-I)\Pi_{\angle}\|_{\mathcal{L}(\mathbb{V})}^{2}\|r^{\angle}\|_{\mathbb{V}}^{2})+(1+\varepsilon)\|r^{0}\|_{\mathbb{V}}^{2}+\|r^{\angle}\|_{\mathbb{V}}^{2}$
$\displaystyle\leq(1+\max\\{\varepsilon+2(1+\varepsilon^{-1})\theta_{0},2(1+\varepsilon^{-1})\theta_{\angle}\\})(\|r^{0}\|_{\mathbb{V}}^{2}+\|r^{\angle}\|_{\mathbb{V}}^{2}).$

We can now apply the homotopy invariance of the degree to get
$\deg(\mathcal{K}_{\mathrm{KP}}(\cdot,\lambda),D,0)\equiv d\neq 0$ for all
$\lambda\in[0,1]$, with $\delta$ decreased if necessary.

##  Appendix B A short proof of the Kowalski–Piecuch theorem

To close this section, we present the main theorem101010They call it the
“Fundamental Theorem of $\beta$-Nested Equation Formalism”. of Kowalski and
Piecuch [32] in a somewhat different form. Define the _KP energy_ as

$\mathcal{E}_{\mathrm{KP}}(t^{1},\lambda)=\langle\mathcal{H}_{K}(t^{0}+\lambda
t^{\angle})\Phi_{0},\Phi_{0}\rangle=\langle\mathcal{H}_{K}e^{T^{0}+\lambda
T^{\angle}}\Phi_{0},\Phi_{0}\rangle,$

so that $\mathcal{E}_{\mathrm{KP}}(t^{1},0)=\mathcal{E}_{\mathrm{CC}}(t^{0})$
and $\mathcal{E}_{\mathrm{KP}}(t^{1},1)=\mathcal{E}_{\mathrm{CC}}(t^{1})$. If
$\rho\geq 2$, then according to (2.10) of Part I, we simply have

$\mathcal{E}_{\mathrm{KP}}(t^{1},\lambda)=\langle\mathcal{H}_{K}(t^{0})\Phi_{0},\Phi_{0}\rangle=\langle\mathcal{H}_{K}e^{T^{0}}\Phi_{0},\Phi_{0}\rangle,$
(B.1)

although we will not exploit this property in the proof.

######  Theorem B.1 (Kowalski–Piecuch).

  1. (I)

Suppose that $\Psi=(c_{0}I+C^{0})\Phi_{0}\in\mathfrak{H}^{1}_{K}$, with
$c_{0}\in\mathbb{R}$ and $c^{0}\in\mathbb{V}^{0}$, satisfies the (weak)
Schrödinger equation

$\langle\mathcal{H}_{K}\Psi,\Phi\rangle=\mathcal{E}\langle\Psi,\Phi\rangle\quad\text{for
all}\quad\Phi\in\mathfrak{H}^{1}_{K}$ (B.2)

for some $\mathcal{E}\in\mathbb{R}$ and that

$\langle e^{T^{0}_{**}(\lambda)+\lambda
T_{**}^{\angle}(\lambda)}\Phi_{0},\Psi\rangle\neq 0,$ (B.3)

where $t_{**}^{1}(\lambda)\in\mathbb{V}^{1}$ is a zero of
$\mathcal{K}_{\mathrm{KP}}(\cdot,\lambda)$ for _all_ $\lambda\in[0,1]$. Then

$\mathcal{E}_{\mathrm{KP}}(t_{**}^{1}(\lambda),\lambda)\equiv\mathcal{E}\quad\text{for
all}\quad\lambda\in[0,1].$

  2. (II)

Suppose that $\Psi=(c_{0}I+C^{1})\Phi_{0}$, with $c_{0}\in\mathbb{R}$ and
$c^{1}\in\mathbb{V}^{1}$, satisfies Eq. B.2 for some
$\mathcal{E}\in\mathbb{R}$. If Eq. B.3 holds true for _some_ $\lambda\in[0,1]$
and $t_{**}^{1}=t_{**}^{1}(\lambda)\in\mathbb{V}^{1}$ with
$\mathcal{K}_{\mathrm{KP}}(t_{**}^{1},\lambda)=0$, then

$\displaystyle\mathcal{E}_{\mathrm{KP}}(t_{**}^{1}(\lambda),\lambda)-\mathcal{E}$
$\displaystyle=\frac{\langle(\mathcal{H}_{K}(t^{1}_{**})-\mathcal{H}_{K}(t^{0}_{**}+\lambda
t^{\angle}_{**}))\Phi_{0},\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}+\lambda
T^{\angle}_{**}})^{\dagger}C^{\angle}\Phi_{0}\rangle}{\langle
e^{T^{0}_{**}+\lambda T_{**}^{\angle}}\Phi_{0},\Psi\rangle}$
$\displaystyle=(1-\lambda)\frac{\langle\Gamma(t_{**}^{1},\lambda)\Phi_{0},\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}+\lambda
T^{\angle}_{**}})^{\dagger}C^{\angle}\Phi_{0}\rangle}{\langle
e^{T^{0}_{**}+\lambda T_{**}^{\angle}}\Phi_{0},\Psi\rangle},$

where $\Gamma(t^{1},\lambda)$ is given by Eq. B.6 below.

Furthermore, in case the energy blows up, we have the following.

######  Theorem B.2.

Suppose that $\Psi=(c_{0}I+C^{1})\Phi_{0}$, with $c_{0}\in\mathbb{R}$ and
$c^{1}\in\mathbb{V}^{1}$, satisfies Eq. B.2 for some
$\mathcal{E}\in\mathbb{R}$. Furthermore, assume that
$t_{**}^{1}(\lambda)\in\mathbb{V}^{1}$ is a zero of
$\mathcal{K}_{\mathrm{KP}}(\cdot,\lambda)$ for all $\lambda$ in a neighborhood
of some $\lambda_{0}\in[0,1]$. If
$|\mathcal{E}_{\mathrm{KP}}(t_{**}^{1}(\lambda),\lambda)|\to\infty$ as
$\lambda\to\lambda_{0}$ and

$\langle\Gamma(t_{**}^{1}(\lambda),\lambda)\Phi_{0},\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}+\lambda
T^{\angle}_{**}})^{\dagger}C^{\angle}\Phi_{0}\rangle=\mathcal{O}\left(\frac{1}{1-\lambda}\right)\quad(\lambda\to\lambda_{0}),$
(B.4)

then

$\langle e^{T^{0}_{**}(\lambda)+\lambda
T_{**}^{\angle}(\lambda)}\Phi_{0},\Psi\rangle\to
0\quad(\lambda\to\lambda_{0}).$

Part (I) of Theorem B.1 says that if one can solve the Schrödinger equation
_exactly_ on $\mathbb{V}^{0}$ for an eigenvalue $\mathcal{E}$ and Eq. B.3
holds true for a zero $t_{**}^{1}(\lambda)\in\mathbb{V}^{1}$ of
$\mathcal{K}_{\mathrm{KP}}(\cdot,\lambda)$ for all $\lambda$, then the KP
energy $\mathcal{E}_{\mathrm{KP}}(t_{**}^{1}(\lambda),\lambda)$ is identically
$\mathcal{E}$. Notice that $t_{**}^{1}(1)$ is not required to represent
$\Psi$, i.e. $e^{T_{**}^{1}(1)}\Phi_{0}\neq\Psi$ is allowed. Also, no
regularity of the trajectory $\lambda\mapsto t_{**}^{1}(\lambda)$ is demanded.

Part (II) stipulates that the Schrödinger equation can be solved on
$\mathbb{V}^{1}$ with an eigenvalue $\mathcal{E}$ and that the
nonorthogonality condition Eq. B.3 holds true for some
$t_{**}^{1}\in\mathbb{V}^{1}$ zero of
$\mathcal{K}_{\mathrm{KP}}(\cdot,\lambda)$ for some $\lambda$. Then, the error
in the energy can be expressed by the stated formula. If one assumes the
hypothesis for all $\lambda\in[0,1]$ in a neighborhood of $1$, then we can
conclude that the KP energy
$\mathcal{E}_{\mathrm{KP}}(t_{**}^{1}(\lambda),\lambda)$ tends to
$\mathcal{E}$ smoothly, as $\lambda\to 1$. Again, no regularity of
$\lambda\mapsto t_{**}^{1}(\lambda)$ is needed.

Finally, Theorem B.2 considers the case when the KP energy diverges as
$\lambda\to\lambda_{0}$. Assuming the growth condition Eq. B.4, we can
conclude that the KP solution $e^{T^{0}_{**}(\lambda)+\lambda
T_{**}^{\angle}(\lambda)}\Phi_{0}$ becomes orthogonal to the eigenstate
$\Psi$.

For the proof, we need the following lemma which recasts the KP equations in
an “unlinked” form.

######  Lemma B.3.

Suppose that $t_{**}^{1}=t_{**}^{1}(\lambda)\in\mathbb{V}^{1}$ is such that
$\mathcal{K}_{\mathrm{KP}}(t^{1}_{**},\lambda)=0$ for some $\lambda\in[0,1]$.
Then,

$\langle(\mathcal{H}_{K}-\mathcal{E}_{\mathrm{KP}}(t_{**}^{1},\lambda))e^{T^{0}_{**}+\lambda
T_{**}^{\angle}}\Phi_{0},S^{1}\Phi_{0}\rangle=-\langle\mathcal{G}(t_{**}^{1},\lambda)\Phi_{0},\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}+\lambda
T^{\angle}_{**}})^{\dagger}S^{\angle}\Phi_{0}\rangle,$ (B.5)

where

$\mathcal{G}(t_{**}^{1},\lambda)=\mathcal{H}_{K}(t^{1}_{**})-\mathcal{H}_{K}(t^{0}_{**}+\lambda
t^{\angle}_{**}).$

Moreover,
$\mathcal{G}(t_{**}^{1},\lambda)=(1-\lambda)\Gamma(t_{**}^{1},\lambda)$, where

$\Gamma(t_{**}^{1},\lambda)=\sum_{k=1}^{2N}\frac{(1-\lambda)^{k-1}}{k!}e^{-(T_{**}^{0}+\lambda
T_{**}^{\angle})}[\mathcal{H}_{K},T_{**}^{\angle}]_{(k)}e^{T_{**}^{0}+\lambda
T_{**}^{\angle}}.$ (B.6)

###### Proof.

Assume that $\mathcal{K}_{\mathrm{KP}}(t^{1}_{**},\lambda)=0$, so using Eq.
4.34 we have

$\langle\mathcal{H}_{K}(t_{**}^{0}+\lambda
t_{**}^{\angle})\Phi_{0},S^{0}\Phi_{0}\rangle+\langle\mathcal{H}_{K}(t_{**}^{1})\Phi_{0},S^{\angle}\Phi_{0}\rangle=0\quad\text{for
all}\quad s^{1}\in\mathbb{V}^{1}.$

This can be rewritten as

$\langle\mathcal{H}_{K}(t_{**}^{0}+\lambda
t_{**}^{\angle})\Phi_{0},S^{1}\Phi_{0}\rangle=-\langle(\mathcal{H}_{K}(t_{**}^{1})-\mathcal{H}_{K}(t_{**}^{0}+\lambda
t_{**}^{\angle}))\Phi_{0},S^{\angle}\Phi_{0}\rangle\quad\text{for all}\quad
s^{1}\in\mathbb{V}^{1},$

or,

$\langle\mathcal{H}_{K}(t_{**}^{0}+\lambda
t_{**}^{\angle})\Phi_{0},S^{1}\Phi_{0}\rangle=-\langle\mathcal{G}(t_{**}^{1},\lambda)\Phi_{0},S^{\angle}\Phi_{0}\rangle\quad\text{for
all}\quad s^{1}\in\mathbb{V}^{1}.$

Then we can write

$\displaystyle\langle(\mathcal{H}_{K}-\mathcal{E}_{\mathrm{KP}}(t_{**}^{1},\lambda))e^{T^{0}_{**}+\lambda
T^{\angle}_{**}}\Phi_{0},S^{1}\Phi_{0}\rangle$ $\displaystyle=\langle
e^{-(T^{0}_{**}+\lambda
T^{\angle}_{**})}(\mathcal{H}_{K}-\mathcal{E}_{\mathrm{KP}}(t_{**}^{1},\lambda))e^{T^{0}_{**}+\lambda
T^{\angle}_{**}}\Phi_{0},(e^{T^{0}_{**}+\lambda
T^{\angle}_{**}})^{\dagger}S^{1}\Phi_{0}\rangle$ $\displaystyle=\langle
e^{-(T^{0}_{**}+\lambda T^{\angle}_{**})}\mathcal{H}_{K}e^{T^{0}_{**}+\lambda
T^{\angle}_{**}}\Phi_{0},\Pi_{\mathfrak{V}^{1}}(e^{T^{0}_{**}+\lambda
T^{\angle}_{**}})^{\dagger}S^{1}\Phi_{0}\rangle$
$\displaystyle=-\langle\mathcal{G}(t_{**}^{1},\lambda)\Phi_{0},\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}+\lambda
T^{\angle}_{**}})^{\dagger}S^{1}\Phi_{0}\rangle$
$\displaystyle=-\langle\mathcal{G}(t_{**}^{1},\lambda)\Phi_{0},\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}+\lambda
T^{\angle}_{**}})^{\dagger}S^{\angle}\Phi_{0}\rangle$

for all $s^{1}\in\mathbb{V}^{1}$. In the last step we used that
$\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}+\lambda
T^{\angle}_{**}})^{\dagger}$ maps $\mathfrak{V}^{0}$ to zero. The “moreover”
part is a simple expansion using Eq. 4.9,

$\displaystyle\mathcal{G}(t_{**}^{1},\lambda)$
$\displaystyle=e^{-(T^{0}_{**}+\lambda
T^{\angle}_{**})}(e^{-(1-\lambda)T^{\angle}_{**}}\mathcal{H}_{K}e^{(1-\lambda)T^{\angle}_{**}}-\mathcal{H}_{K})e^{T^{0}_{**}+\lambda
T^{\angle}_{**}}$
$\displaystyle=\sum_{k=1}^{2N}\frac{(1-\lambda)^{k}}{k!}e^{-(T^{0}_{**}+\lambda
T^{\angle}_{**})}[\mathcal{H}_{K},T^{\angle}_{**}]_{(k)}e^{T^{0}_{**}+\lambda
T^{\angle}_{**}}.$

∎

###### Proof of Theorem B.1.

We have using Lemma B.3,

$\displaystyle 0$
$\displaystyle=\langle(\mathcal{H}_{K}-\mathcal{E}_{\mathrm{KP}}(t_{**}^{1}(\lambda),\lambda))e^{T^{0}_{**}(\lambda)+\lambda
T_{**}^{\angle}(\lambda)}\Phi_{0},(c_{0}I+C^{0})\Phi_{0}\rangle$
$\displaystyle=(\mathcal{E}-\mathcal{E}_{\mathrm{KP}}(t_{**}^{1}(\lambda),\lambda))\langle
e^{T^{0}_{**}(\lambda)+\lambda T_{**}^{\angle}(\lambda)}\Phi_{0},\Psi\rangle.$

Similarly, for part (II)

$\displaystyle\langle(\mathcal{H}_{K}(t^{1}_{*})-\mathcal{H}_{K}(t^{0}_{**}+\lambda
t^{\angle}_{**}))\Phi_{0},\Pi_{\mathfrak{V}^{\angle}}(e^{T^{0}_{**}+\lambda
T^{\angle}_{**}})^{\dagger}C^{\angle}\Phi_{0}\rangle$
$\displaystyle=\langle(\mathcal{H}_{K}-\mathcal{E}_{\mathrm{KP}}(t_{**}^{1}(\lambda),\lambda))e^{T^{0}_{**}+\lambda
T_{**}^{\angle}}\Phi_{0},(c_{0}I+C^{0}+C^{\angle})\Phi_{0}\rangle$
$\displaystyle=(\mathcal{E}-\mathcal{E}_{\mathrm{KP}}(t_{**}^{1}(\lambda),\lambda))\langle
e^{T^{0}_{**}+\lambda T_{**}^{\angle}}\Phi_{0},\Psi\rangle.$

Theorem B.2 also follows from the previous equality. ∎

_Acknowledgements._ The authors would like to thank Fabian M. Faulstich and
Simen Kvaal for helpful discussions and comments on the manuscript. The useful
suggestions of the anonymous reviewer are gratefully acknowledged. This work
has received funding from the Norwegian Research Council through Grant Nos.
287906 (CCerror) and 262695 (CoE Hylleraas Center for Quantum Molecular
Sciences).

## References

  * [1] J. S. Arponen.  Variational principles and linked-cluster exp S expansions for static and dynamic many-body problems.  Ann. Phys., 151(2):311–382, 1983.
  * [2] V. Bach, E. H. Lieb, M. Loss, and J. P. Solovej.  There are no unfilled shells in unrestricted Hartree-Fock theory.  In The Stability of Matter: From Atoms to Stars, pages 309–311. Springer, 1997.
  * [3] R. J. Bartlett, S. A. Kucharski, and J. Noga.  Alternative coupled-cluster ansätze II. The unitary coupled-cluster method.  Chemical physics letters, 155(1):133–140, 1989.
  * [4] N. Benedikter.  Hartree-Fock theory.  Online lecture notes, 2017.
  * [5] A. Buffa.  Remarks on the discretization of some noncoercive operator with applications to heterogeneous Maxwell equations.  SIAM Journal on Numerical Analysis, 43(1):1–18, 2005.
  * [6] A. Buffa, R. Hiptmair, T. von Petersdorff, and C. Schwab.  Boundary element methods for Maxwell transmission problems in lipschitz domains.  Numerische Mathematik, 95(3):459–485, 2003.
  * [7] E. Cances, M. Defranceschi, W. Kutzelnigg, C. Le Bris, and Y. Maday.  Computational quantum chemistry: a primer.  Handbook of numerical analysis, 10:3–270, 2003.
  * [8] J. Cronin.  Fixed points and topological degree in nonlinear analysis, volume 11.  American Mathematical Soc., 1995.
  * [9] G. Dinca and J. Mawhin.  Brouwer degree and applications.  preprint, 2009.
  * [10] G. Dinca and J. Mawhin.  Brouwer Degree (The Core of Nonlinear Analysis).  Birkhäuser Basel, 2021.
  * [11] P. Drábek and J. Milota.  Methods of nonlinear analysis: applications to differential equations.  Springer Science & Business Media, 2007.
  * [12] F. M. Faulstich, A. Laestadius, O. Legeza, R. Schneider, and S. Kvaal.  Analysis of the tailored coupled-cluster method in quantum chemistry.  SIAM Journal on Numerical Analysis, 57(6):2579–2607, 2019.
  * [13] F. M. Faulstich, M. Máté, A. Laestadius, M. A. Csirik, L. Veis, A. Antalik, J. Brabec, R. Schneider, J. Pittner, S. Kvaal, et al.  Numerical and theoretical aspects of the DMRG-TCC method exemplified by the nitrogen dimer.  Journal of chemical theory and computation, 15(4):2206–2220, 2019\.
  * [14] G. Friesecke.  The multiconfiguration equations for atoms and molecules: charge quantization and existence of solutions.  Archive for rational mechanics and analysis, 169(1):35–71, 2003\.
  * [15] J. Geertsen, M. Rittby, and R. J. Bartlett.  The equation-of-motion coupled-cluster method: Excitation energies of Be and CO.  Chemical Physics Letters, 164(1):57–62, 1989.
  * [16] T. Helgaker, P. Jorgensen, and J. Olsen.  Molecular electronic-structure theory.  John Wiley & Sons, 2014.
  * [17] K. Jankowski and K. Kowalski.  Physical and mathematical content of coupled-cluster equations. II. on the origin of irregular solutions and their elimination via symmetry adaptation.  The Journal of chemical physics, 110(19):9345–9352, 1999.
  * [18] K. Jankowski and K. Kowalski.  Physical and mathematical content of coupled-cluster equations. IV. impact of approximations to the cluster operator on the structure of solutions.  The Journal of chemical physics, 111(7):2952–2959, 1999.
  * [19] K. Jankowski, K. Kowalski, I. Grabowski, and H. Monkhorst.  Correspondence between physical states and solutions to the coupled-cluster equations.  International journal of quantum chemistry, 75(4-5):483–496, 1999\.
  * [20] K. Jankowski, K. Kowalski, and P. Jankowski.  Multiple solutions of the single-reference coupled-cluster equations. I. H4 model revisited.  International Journal of Quantum Chemistry, 50(5):353–367, 1994\.
  * [21] F. Kossoski, A. Marie, A. Scemama, M. Caffarel, and P.-F. Loos.  Excited states from state specific orbital optimized pair coupled cluster.  arXiv preprint arXiv:2104.03746, 2021.
  * [22] K. Kowalski and K. Jankowski.  Full solution to the coupled-cluster equations: the H4 model.  Chemical physics letters, 290(1-3):180–188, 1998.
  * [23] A. Laestadius and F. M. Faulstich.  The coupled-cluster formalism–a mathematical perspective.  Molecular Physics, 117(17):2362–2373, 2019.
  * [24] A. Laestadius and S. Kvaal.  Analysis of the extended coupled-cluster method in quantum chemistry.  SIAM Journal on Numerical Analysis, 56(2):660–683, 2018.
  * [25] M. Lewin.  Geometric methods for nonlinear many-body quantum systems.  Journal of Functional Analysis, 260(12):3535–3595, 2011.
  * [26] M. Lewin.  Existence of Hartree–Fock excited states for atoms and molecules.  Letters in Mathematical Physics, 108(4):985–1006, 2018.
  * [27] E. H. Lieb and B. Simon.  On solutions to the Hartree-Fock problem for atoms and molecules.  The Journal of Chemical Physics, 61(2):735–736, 1974.
  * [28] P.-L. Lions.  Solutions of Hartree-Fock equations for Coulomb systems.  Communications in Mathematical Physics, 109(1):33–97, 1987.
  * [29] D. O’Regan, Y. J. Cho, and Y.-Q. Chen.  Topological degree theory and applications.  CRC Press, 2006.
  * [30] J. Paldus, M. Takahashi, and B. Cho.  Degeneracy and coupled-cluster approaches.  International Journal of Quantum Chemistry, 26(S18):237–244, 1984\.
  * [31] W. V. Petryshyn.  Approximation-solvability of nonlinear functional and differential equations, volume 171.  CRC Press, 1992.
  * [32] P. Piecuch and K. Kowalski.  In search of the relationship between multiple solutions characterizing coupled-cluster theories.  In Computational chemistry: reviews of current trends, pages 1–104. World Scientific, 2000.
  * [33] P. Piecuch, S. Zarrabian, J. Paldus, and J. Čížek.  Coupled-cluster approaches with an approximate account of triexcitations and the optimized-inner-projection technique. II. coupled-cluster results for cyclic-polyene model systems.  Physical Review B, 42(6):3351, 1990.
  * [34] V. V. Prasolov.  Problems and theorems in linear algebra, volume 134.  American Mathematical Soc., 1994.
  * [35] T. Rohwedder.  The continuous Coupled Cluster formulation for the electronic Schrödinger equation.  ESAIM: Mathematical Modelling and Numerical Analysis-Modélisation Mathématique et Analyse Numérique, 47(2):421–447, 2013.
  * [36] T. Rohwedder and R. Schneider.  Error estimates for the coupled cluster method.  ESAIM: Mathematical Modelling and Numerical Analysis-Modélisation Mathématique et Analyse Numérique, 47(6):1553–1582, 2013.
  * [37] R. Schneider.  Analysis of the projected coupled cluster method in electronic structure calculation.  Numerische Mathematik, 113(3):433–471, 2009.
  * [38] I. V. Skrypnik.  Methods for analysis of nonlinear elliptic boundary value problems, volume 139.  American Mathematical Soc., 1994.
  * [39] J. P. Solovej.  Many body quantum mechanics.  Lecture Notes., 2007.
  * [40] L. T. Watson, S. C. Billups, and A. P. Morgan.  Algorithm 652: Hompack: A suite of codes for globally convergent homotopy algorithms.  ACM Transactions on Mathematical Software (TOMS), 13(3):281–310, 1987.
  * [41] H. Yserentant.  Regularity and approximability of electronic wave functions.  Springer, 2010.
  * [42] P. Zabrejko.  Rotation of vector fields: definition, basic properties, and calculation.  In Topological nonlinear analysis II, pages 445–601. Springer, 1997\.
  * [43] E. Zeidler and P. R. Wadsack.  Nonlinear functional analysis and its applications: Fixed-point theorems/transl. by Peter R. Wadsack.  Springer-Verlag, 1993.
  * [44] T. P. Živković.  Existence and reality of solutions of the coupled-cluster equations.  International Journal of Quantum Chemistry, 12(S11):413–420, 1977\.
  * [45] T. P. Živković and H. J. Monkhorst.  Analytic connection between configuration–interaction and coupled-cluster solutions.  Journal of Mathematical Physics, 19(5):1007–1022, 1978.


# A flow-based IDS using Machine Learning in eBPF

Maximilian Bachl, Joachim Fabini, Tanja Zseby  Technische Universität Wien
<EMAIL_ADDRESS>
###### Abstract

eBPF is a new technology which allows dynamically loading pieces of code into
the Linux kernel. It can greatly speed up networking since it enables the
kernel to process certain packets without the involvement of a userspace
program. So far eBPF has been used for simple packet filtering applications
such as firewalls or Denial of Service protection. We show that it is possible
to develop a flow-based network intrusion detection system based on machine
learning entirely in eBPF. Our solution uses a decision tree and decides for
each packet whether it is malicious or not, considering the entire previous
context of the network flow. We achieve a performance increase of over 20%
compared to the same solution implemented as a userspace program.

##  I Introduction

###  I-A eBPF

eBPF is a technology which makes the Linux kernel programmable by enabling the
injection of pieces of code at many locations of the kernel code. eBPF can be
dynamically injected during runtime and is verified to make sure that it
cannot crash and cannot get caught in infinite loops. However, this
verification is only possible for programs that are not turing-complete. Thus
eBPF programs cannot contain features such as loops of arbitrary length but
instead loops must always have a maximum number of iterations. Also, backward
jumps in the code are generally not allowed. This means eBPF can only be used
to implement algorithms which do not require turing-completeness. eBPF
programs are usually written in C and are first compiled to eBPF bytecode.
Upon injection into the kernel, this eBPF bytecode is verified and dynamically
compiled to native code.

eBPF is especially suitable for packet processing: When a packet arrives at a
network interface, certain actions can be performed such as dropping the
packet. This is useful for programs such as firewalls [1] or tcpdump, which
records packets according to certain filters. For example, if only packets
coming from port 80 should be recorded, tcpdump will compile an eBPF program
which encodes this and will load it into the kernel. The kernel will then drop
all packets which don’t match the filter and only pass the correct ones to
tcpdump. The alternative would be that tcpdump receives every packet and
filters them itself. The drawback of this is that then each packet has to be
passed from the kernel to tcpdump, which involves copying the whole packet in
memory and also other computation steps. Thus, passing packets between the
kernel and programs should be avoided if possible because of performance
reasons. eBPF allows one to do that.

Because eBPF bytecode is compiled to native code, it should generally be as
fast as any other code in the kernel. A drawback, however, is that because
eBPF is verified, it can only use certain data structures. For example, an
eBPF program cannot use normal C arrays since they allow out-of-bounds
accesses. For example, in an array of length 10, C would allow accessing the
15th element even though the array only has 10 elements. Thus, eBPF programs
make use of special data structures which are safe. However, this can
potentially be a performance penalty since checking the bounds of an array
each time it is accessed requires extra work to be done by the CPU.

One alternative to using eBPF is using kernel modules. However, a drawback of
kernel modules is that they usually cannot be verified for stability and that
they have to be compiled for a specific kernel version. Moreover, developing
kernel modules is not straightforward and often it is not possible to extend
certain functionality in the kernel with a kernel module without changing the
kernel itself. Changing the whole kernel itself makes it necessary to
recompile the kernel, which is cumbersome.

###  I-B eBPF for a Machine Learning (ML)-based Intrusion Detection System
(IDS)

Some researchers have already investigated eBPF based IDSs or thought about
using ML in eBPF. [2] propose to use AI for detecting performance anomalies
with eBPF. However, they only propose a concept and do not provide an
evaluation of the benefit of their approach. [3], [4] and also [5] use eBPF to
develop solutions against Denial-of-Service attacks. They do not use ML.

One challenge of using eBPF for ML is that it is not turing-complete. However,
ML algorithms such as decision trees (DTs) or neural networks (NNs) do not
require loops in the implementation and thus don’t require turing-completeness
and can thus be implemented in eBPF. We decide to use DTs since tree-based
methods are a simple and effective ML method for IDSs [6]. As mentioned in the
previous sections, eBPF data structures need some additional processing
compared to the classic ones built into C. A question we want to answer in
this research is thus the following: Is eBPF faster than a solution
implemented as a normal program in userspace? The userspace program has the
disadvantage that all packets have to be passed between the kernel and the
program which is slow. The eBPF program has the disadvantage that it makes use
of potentially slower data structures. Thus, it is interesting to understand
whether in practice eBPF can be faster even for complex programs which make
use of data structures frequently.

##  II Evaluation

We envision an approach which keeps track of each network flow and analyzes
each packet in the context of the previous packets of the flow. For example,
certain attacks could be detected only when the fourth packet of the network
flow containing the attack arrives. eBPF has built-in hash tables which allow
storing information for each flow, which we use for this purpose. As the key
we use the classic five tuple of protocol type, source and destination IP and
source and destination port.

As features we use the source and destination port, the protocol identifier
(UDP, TCP, ICMP etc.), the packet length, the time since the last packet of
the flow and the direction of the packet (from sender to receiver or vice
versa) because these features have proven useful for network traffic analysis
[6]. Additionally, we include the mean of the packet size, the time since the
last packet and the direction for all packets received in the flow so far.
Furthermore, also the mean absolute deviation is computed for these three
features, since the standard deviation cannot be computed due to the absence
of advanced arithmetic operations in eBPF such as the square root operation. A
problem is that eBPF doesn’t allow floating point operations but only
integers. We thus implement the decision tree using fixed point arithmetic
using 64 bit signed integers. We use 16 bits for the part after the fixed
point.

We use the popular CIC-IDS-2017 dataset [7]. We train the decision tree using
scikit-learn with a maximum depth of 10 and a maximum number of leaves of 1000
using a train/test split of 2:1, which achieves an accuracy of 99% on the
testing dataset after training. This accuracy is comparable to the accuracy
achieved in related work [8].

To enable reproducibility and to encourage further experimentation by other
researchers, we make all the source code and other materials of this work
publicly available 111https://github.com/CN-TU/machine-learning-in-ebpf.

We implement the same IDS in userspace and also in eBPF and use the previously
trained DT. The code is completely equivalent except that data structures are
different because, as mentioned previously, many normal data structures are
not usable in eBPF. Also, some data structures that eBPF has, such as hash
maps, are not available in a normal C userspace program by default and thus we
use a simple hash map implementation from the Linux kernel for the userspace
version.

We emulate a network using Linux network name spaces. For this, we have one
server and one client, connected via a switch and configure the links to have
no delay in addition to the delay caused by the Linux kernel for forwarding
the packets. Furthermore, we set the maximum link speed to be unlimited. The
server and the client are connected using iPerf. Since the link speed is
unlimited, iPerf will send as fast as possible. The maximum speed is only
limited by how fast the computer can process the packets. The IDS is deployed
by opening a raw socket on the network interface of the server. All packets
passing through the server thus pass through the raw socket and are processed
by the IDS. The IDS is implemented either as a classic userspace program or by
using eBPF and run after one another (not at the same time). Both
implementations are run for 10 s. Instead of deploying the IDS on an end host,
it can also be deployed on a router or switch, which runs a recent Linux
version and thus can run eBPF.

TABLE I: Maximum number of packets each implementation can process (mean and standard deviation). Results averaged over 10 runs each. | Userspace | eBPF
---|---|---
| Mean | SD | Mean | SD
packets/s | 125 420 | 2627 | 152 274 | 1201

Table I shows that the userspace implementation inspects fewer packets per
second (125420) than the eBPF implementation (152274). The eBPF implementation
is thus more than 20% faster than the userspace one.

##  III Discussion

An interesting consideration is that for complex ML models, at some point the
overhead from using eBPF’s special data structures becomes larger than the
benefit that eBPF confers. For example, interesting future work would be to
test whether random forests (RFs) or deep NNs can also achieve a performance
advantage when implemented in eBPF. In this work, however, we could show that
for a simple decision tree model, eBPF provides a significant performance
advantage.

## References

  * [1] F. Risso and M. Tumolo, “Towards a faster Iptables in eBPF,” 2018.
  * [2] I. Ben-Yair, P. Rogovoy, and N. Zaidenberg, “AI & eBPF based performance anomaly detection system,” in SYSTOR, ACM, May 2019.
  * [3] H. M. Demoulin, I. Pedisich, N. Vasilakis, V. Liu, B. T. Loo, and L. T. X. Phan, “Detecting Asymmetric Application-layer Denial-of-Service Attacks In-Flight with FineLame,” USENIX, 2019.
  * [4] H. van Wieren, “Signature-Based DDoS Attack Mitigation: Automated Generating Rules for Extended Berkeley Packet Filter and Express Data Path,” 2019.
  * [5] Y. Choe, J.-S. Shin, S. Lee, and J. Kim, “eBPF/XDP Based Network Traffic Visualization and DoS Mitigation for Intelligent Service Protection,” in Advances in Internet, Data and Web Technologies, Springer, 2020.
  * [6] F. Iglesias, D. C. Ferreira, G. Vormayr, M. Bachl, and T. Zseby, “NTARC: A Data Model for the Systematic Review of Network Traffic Analysis Research,” Applied Sciences, Jan. 2020.
  * [7] I. Sharafaldin, A. Habibi Lashkari, and A. A. Ghorbani, “Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization,” in ICISSP, SCITEPRESS, 2018.
  * [8] A. Hartl, M. Bachl, J. Fabini, and T. Zseby, “Explainability and Adversarial Robustness for RNNs,” in BigDataService 2020, IEEE, Apr. 2020.