11institutetext: Language Technologies Research Center, KCIS, IIIT Hyderabad
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>
22institutetext: TCS Research, Hyderabad, India
22email<EMAIL_ADDRESS>
# LimGen: Probing the LLMs for Generating Suggestive Limitations of Research
Papers

Abdur Rahman Bin Md Faizullah 11 * * Ashok Urlana 22 * * Rahul Mishra 11

###### Abstract

Examining limitations is a crucial step in the scholarly research reviewing
process, revealing aspects where a study might lack decisiveness or require
enhancement. This aids readers in considering broader implications for further
research. In this article, we present a novel and challenging task of
Suggestive Limitation Generation (SLG) for research papers. We compile a
dataset called LimGen, encompassing 4068 research papers and their associated
limitations from the ACL anthology. We investigate several approaches to
harness large language models (LLMs) for producing suggestive limitations, by
thoroughly examining the related challenges, practical insights, and potential
opportunities. Our LimGen dataset and code can be accessed at
https://github.com/armbf/LimGen.

###### Keywords:

Limitations LLM Constrained Text Generation.

**footnotetext: Equal contribution.

##  1 Introduction

The process of reviewing research articles lies at the core of the scientific
publication process. It serves as a pivotal stage wherein experts meticulously
scrutinize the content, methodology, and findings of a study before it is
disseminated to the broader scientific community. This rigorous evaluation and
critique of the strengths and limitations of the research work ensures the
integrity, reproducibility, and credibility of the research being presented.
Identifying the potential limitations of the research papers is a very tedious
and time-consuming process that involves delving into the intricacies of the
research methodology, understanding the theoretical framework, and grasping
the nuances of the subject matter. This endeavor demands not only a broad
knowledge base but also keen attention to detail and analytical thinking.

Table 1: Comparison of actual vs LLM generated suggestive limitations (best
viewed in color). Magenta indicates the matching content between the actual
and LLM-generated limitations, whereas Blue represents the novel limitations.
Actual Limitation: Contamination is a significant problem in the era of
pretrained language models trained on large web corpora… Additionally, this
data’s distribution is fixed to pre-college Physics, Chemistry, and
Mathematics. There are more gradations and difficulty levels at which the
model can be evaluated, which have not been tested as part of our analysis.
---
Llama2-DPR: The limitations of the approach are: The dataset is limited to
only three subjects (Physics, Chemistry, and Mathematics) and does not cover
other important subjects like Biology, Geology, and Computer Science. The
dataset does not include any real-world applications or scenarios, limiting
the generalization of the model’s capabilities. The evaluation metric used is
based on the number of correct answers, which does not fully capture the
model’s reasoning abilities. The dataset does not include any adversarial
examples or challenging cases to test the model’s robustness and adaptability.

This paper presents a novel task of Suggestive Limitation Generation (SLG) for
research papers, which aims to generate a diverse array of potential
limitations specific to each paper, providing reviewers with valuable insights
to facilitate their assessment process. The task of SLG poses a greater
challenge compared to text summarization, question-answering (QnA), and open-
ended text generation tasks. It demands not only reasoning abilities but also
the capacity to discern and incorporate associations from a corpus of
previously encountered papers during the fine-tuning process. For example, in
Table 1, we show the actual limitation block from a research paper in row 1
and the corresponding generated limitations in row 2. Illustrated in magenta,
the proposed model demonstrates the capability to generate limitations akin to
those originally outlined in the paper (the dataset is limited to only three
subjects, Physics, Chemistry, and Mathematics) Additionally, it showcases the
ability to propose novel, valid limitations (depicted in blue) related to
adversarial examples, a facet not included by authors in original limitations.

The key idea behind this approach is to capitalize on cues regarding the
similarities or differences among research papers and learn to recommend
comparable limitations for a paper, especially when its underlying methodology
closely aligns with that of a set of other papers. To this end, we create a
dataset called LimGen, comprising 4068 research papers and corresponding
limitations from the ACL anthology. Subsequently, we probe many state-of-the-
art large language models (LLMs) in a multitude of experimental setups to
generate the suggestive limitations of the research papers. We conduct a very
thorough evaluation by utilizing automatic (rouge-based), human (manual
expert-based), and model assistive (LLM-based) approaches.

The key contributions of this work are: 1) To the best of our knowledge, we
are the first to propose the task of Suggestive Limitation Generation (SLG)
for research papers. 2) We release a SLG dataset LimGen, consisting of 4068
papers and corresponding limitations. 3) We propose and experiment with
several schemes to utilize LLMs for SLG. 4) We perform thorough evaluations
using automated, human, and LLM-driven methodologies.

##  2 Related work

Scientific document understanding poses a persistent challenge, primarily
attributable to its structure, diverse content modalities (such as tables and
figures), and the incorporation of citations within the text. The recent
emergence of large-scale scientific document summarization datasets were
automatically collected from the public repositories [3, 7]. Several works on
scientific documents encompass tasks such as abstract generation [12], delving
into the contributions outlined in a paper [17, 10], scientific papers
summarization [2] and formulate multi-perspective summaries by leveraging
reviews of the research papers [4, 24].

Moreover, few works delve into other forms of supervision for scientific
document understanding, including citations [18, 25], author-written
highlights [5], transcripts from conference presentations of the research
papers [14] and annotations [19]. In our work, we collected the research
papers and corresponding limitations from the ACL anthology. In contrast to
existing works, we attempt to generate suggestive limitations of the research
papers using LLMs. To the best of our knowledge, this is the first work
towards generating limitations of the research papers.

##  3 LimGen Dataset

###  3.1 Dataset collection

We obtain the dataset from ACL Anthology111https://aclanthology.org/ website.
We take advantage of the recent mandatory inclusion of the ‘limitations’
section in the research paper for the submission of Computational Linguistics-
related venues. We scrap the proceedings of EMNLP, ACL, and EACL venues of
2022 and 2023 years respectively. After obtaining the papers, the initial step
involves using scipdf_parser222https://github.com/titipata/scipdf_parser to
parse the PDFs. The parser segregates the content of the paper into section-
wise information. From the extracted sections, to create the ‘source’ text for
the SLG task, we discard some of the sections Abstract, Introduction, Related
Works, Acknowledgements, Conclusion, Ethics Statement, Appendix, References,
Limitations and preserve the main contribution of the paper in form of
methodology, experiments, results, and discussions, etc. We use the
corresponding actual ‘limitations’ section as the reference limitations. In
total, we utilize 4068 peer-reviewed short and long papers from three
different ACL venues.

As an initial exploratory analysis of the proposed LimGen dataset, we computed
several key statistics. Notably, the average length of the research papers
stands at approximately 5000 tokens and 187 sentences. Whereas the limitation
sections, average around 230 tokens, and 9 sentences. The longer length of the
papers poses a challenge for the large language models to process the longer
context length. For detailed statistics corresponding to each conference,
please refer to the provided Table 2.

Table 2: LimGen Dataset Statistics Number of research papers 4068
---
ACL 2022 | 1750 |  | #Avg words per paper | 5122
EMNLP 2022 | 1227 |  | #Avg sentences per paper | 188
EACL 2022 | 456 |  | #Avg words per limitation | 230
EMNLP 2023 | 635 |  | #Avg sentences per limitation | 10
Table 3: Manual analysis of 60 research papers; Relevance | Deduce Limitation | Future work or Limitation
---|---|---
Yes | No | Partial | Yes | No | Partial |  Yes |  No | Partial
37 | 3 | 20 | 12 | 13 | 35 |  15 |  13 | 32
Limitations related to
Methodology | Experimental setup | Dataset | Evaluation
41 | 17 | 22 | 7

###  3.2 Nature of the limitations

To understand the nature of the limitations in research papers, we conduct a
manual analysis of 60 papers. We maintain diversity (short, and long papers
from diverse venues) in paper selection to capture the stylistic variations of
the limitations present in the research papers.

The analysis aims to understand, 1) The relevance of the underlying limitation
to the research paper, 2) whether the given limitation can be deduced by
reading the paper or not?, 3) establish whether the mentioned limitation
represents a real constraint or suggests potential avenues for future
research, 4) classify the limitation according to its relevance to specific
sections of the paper, including Methodology, dataset, evaluation or
experimental setup. We present the outcome of our manual analysis in Table 3.
We note that within our sample, numerous papers regard their future prospects
as limitations. The extent of the limitation section ranges from mere
sentences to substantially lengthy paragraphs. Additionally, our observations
indicate that in over 50% of the papers, discerning the stated limitation
directly from the text is not straightforward. Furthermore, a significant
portion of these limitations are predominantly associated with the methodology
section of the paper. There are few instances, where the limitations cover
more than one section of the research paper.

##  4 Benchmark Experiments

###  4.1 Task formulation

This section introduces the Suggestive Limitation Generation (SLG) task
formulation. To produce the limitations of the papers, we approach the task as
a Seq2Seq problem. Precisely, we craft a model designed to intake a scientific
paper R as input and systematically produce a structured limitation block L =
l(1:n), where l(1:n) represents the combination of n limitations of R,
sequentially generated.

###  4.2 Methodology

This section describes the various approaches to generate the suggestive
limitations. We explore two suitable text generation paradigms to generate the
limitations of the research papers. Firstly, we consider the limitation
generation as a summarization task and utilize the summarization-specific pre-
trained models including BART333https://huggingface.co/facebook/bart-large-cnn
[15] and PEGASUS444https://huggingface.co/google/pegasus-large [26] to
generate the suggestive limitations. Given that the objective of the SLG task
surpasses the complexity of the summarization task, which typically involves
limited or constrained generation entropy, the SLG task demands a higher
degree of generation entropy. Unlike summarization, where the model’s task is
to condense information, in SLG, the model must infer and recommend
limitations from the source content, drawing from its understanding during
fine-tuning. Hence, the generative scheme becomes pivotal. To this end, as the
second paradigm, we utilize popular Large Language Models (LLMs) namely
T5555https://huggingface.co/google-t5/t5-base [20], Cerebras-GPT [9] and Llama
2 [22]. To experiment with both of these paradigms, we utilize the following
three schemes.

Figure 1: General architecture diagram for the suggestive limitation
generation. Figure 2: Architecture diagram for DPR fine-tuning. Figure 3:
Architecture diagram for Chain modeling.

####  4.2.1 Non-truncated research paper

In this scheme, we employ the entire research paper and its associated
limitations to experiment with both summarization-specific and generative
models, as illustrated in Figure 1. We fine-tune the summarization models such
as BART, PEGASUS and also utilize generative models including T5, Llama 2, and
Cerbras GPT to perform the zero-shot prompting and fine-tuning. To experiment
with the generative models in this scheme, we use the prompt depicted in Table
4. However, this approach is constrained by the models’ max input token limit,
resulting in a lack of comprehensive context for the research paper.

Table 4: Prompt for fine-tuning with the non-truncated research papers. |  Prompt: Generate limitations or shortcomings for the following scientific paper
---
{Paper Text}
Limitations:
End-of-Prompt

####  4.2.2 Dense Passage Retrieval (DPR)

To address the length constraint in the Non-truncated research paper scheme,
we employ the DPR approach. This approach processes only the relevant passages
for each sentence present in the limitation section. To obtain the relevant
passages, we utilize the following three-stage approach.

  1. 1.

Paragraph and sentence processing: We segment the research papers into
paragraphs and obtain the sentences of each paragraph by using
Spacy666https://spacy.io/api/tokenizer.

  2. 2.

Tokenization and paragraph management: We tokenize these passages using
BertTokenizer [8] to manage the length of each passage to maintain the max
input token limit of 1024 for Llama 2 and 2048 for Cerebras. We merge smaller
passages for optimization and split larger passages to adhere to the max input
token limit.

  3. 3.

Limitation sentence extraction and encoding: We encode limitation sentences
for each document and the passages using SentenceTransformer [21] (all-
MiniLM-L6-v2777https://www.sbert.net/). We compute the cosine similarity
between each sentence in the limitations and all sentences within each
passage. Further, discard the passages with a similarity score of less than
0.5. Ultimately, we identify the top three passages with the highest cosine
similarity for each sentence in the limitations.

Table 5: Prompt for DPR-based fine-tuning. |  Prompt: Generate limitations or shortcomings for the following passage from a
---
scientific paper
passage:{DPR paragraph}
A brief technical summary of the scientific paper for context:{Summary of the
paper}
Limitations:
End-of-Prompt

We fine-tune the model by employing individual limitation sentences as target
outputs, with the input comprising the top three passages retrieved through
DPR (Dense Passage Retrieval). These passages are specifically obtained using
the respective limitation sentence as the query. The prompt for the DPR-based
fine-tuning is depicted in Table 5. The fine-tuned model is utilized to
generate limitations for each passage extracted from the paper. These
limitations are then compiled to form the comprehensive set of limitations
associated with the research paper as shown in Figure 2. Both the
summarization and generative models utilize the DPR-retrieved passages for the
experiments. While generally effective, this approach may produce irrelevant
limitations due to the lack of a broader context of the entire research paper.
To tackle this challenge, we investigate the application of chain modeling
techniques as a prospective solution.

####  4.2.3 Chain Modeling

To overcome the constraints posed by the lack of contextual information in the
DPR approach, we implement the chain modeling approach inspired by
LangChain888https://github.com/LangChain-ai/LangChain. The chain modeling
consists of two stages. In stage 1 (limitation generation), we generate the
limitations for all passages of the research paper obtained by following the
steps 1 and 2 in DPR model creation (see Section 4.2.2). However, going
through individual passages, we observe that, the model lacks the
comprehensive context provided by the entire research paper. To overcome this,
in the prompt, we include the summary of the research paper along with each
passage. The prompt to obtain the summary is mentioned in Table 6. In stage 2
(refinement), we refine and distill all the generated limitations, eliminate
obvious duplicates, and standardize the language as depicted in Figure 3. The
prompt for both stages is shown in Table 7.

The chain modeling approach utilizes two distinct fine-tuned models. The
former one (LLM1 in Figure 3) is fine-tuned on DPR passages along with summary
of the research paper as input and limitation as output. The prompt for the
same is mentioned in Table 5. Whereas the latter one (LLM2 in Figure 3) is
fine-tuned on full non-truncated paper as input and corresponding paper
limitations as output using the prompt depicted in Table 4. This approach is
indicated as Llama2-DPR-Distilled in Table 9. Another approach using the full
non-truncated fine-tuned Llama 2 model for both LLM1 and LLM2, which is
indicated as Llama2-FT-Distilled (FT refers to Full-text) in Table 9. We also
implement chain modelling approach where the result of the previous iteration
is passed to the current iteration with the passage. But it is observed that
when the model hallucinates or causes repetitions, all the subsequent steps
are affected. For this setting, the corresponding results are reported in the
table 9 under Llama2-Continuous.

Table 6: Prompt for generating the summary of the research paper. |  Initial Prompt: Write a concise technical summary of the following: {first passage}
---
CONCISE TECHNICAL SUMMARY: {Summary of the paper}
End-of-Prompt
Iteration Prompt: Your job is to produce a final technical summary of a
research
paper. We have provided a summary generated by you up to a certain point:
{summary from the previous step}
We have the opportunity to refine the existing summary (only if needed)
with some more context below.
{next passage}
Given the new context, refine the original technical summary. Keep the
technical
summary less than 350 words. If the context isn’t useful in the technical
research
context, return the original summary. Do not ask any questions in the
response.
Refined Technical Summary:
End-of-Prompt

###  4.3 Experimental Setup

To conduct experiments with generative models with DPR and chain modeling
approaches, we use the Cerebras-GPT [9] 1.3B and Llama 2 [22] 7B versions. Due
to hardware constraints, we utilize the smaller versions of the LLMs and
perform the fine-tuning with the LoRA [11] configuration with 8-bit
quantization. We utilize the XTuring code
base999https://github.com/stochasticai/xTuring for the fine-tuning. For the
DPR approach, we limit the number of top passages passed to the model to 2 for
Llama 2 and 3 for Cerbras-GPT. Moreover, we utilize four Nvidia GeForce RTX
2080 Ti GPUs (11GB). Due to increased memory requirements for finetuning Llama
2 models for chain modeling with non-truncated research papers and processing
the DPR dataset with summaries as context, we temporarily upgrade to 4 NVIDIA
GeForce RTX 3090 GPUs. We use 80-10-10 split of the LimGen dataset for the
creation of train, valid, and test datasets.

##### Specifics for Chain Modelling:

The chain modeling approach requires more computation due to an increase in
the input context length and the need for higher inference speed to process
all the passages in a paper. To accommodate these requirements, we fine-tune
the Llama 2 model using Axolotl101010https://github.com/OpenAccess-AI-
Collective/axolotl with LoRA and FlashAttention [6]. Further, we use AWQ via
AutoAWQ111111https://github.com/casper-hansen/AutoAWQ with zero-point, group
size of 128, 4 bits, and GEMM version on vLLM [13] for efficient inference.

Table 7: Prompts for the chain modeling approach. |  Iteration Prompt: Your job is to take in a passage from a research paper
---
and a concise summary of that research and identify one or two main
limitations from the given passage using the summary as context.
Paper Passage: {passage}
Paper Summary: {summary}
Limitations:
End-of-Prompt
Distill Prompt:
The following is set of limitations:. {list of generated limitations}
Take these and distill them into a final, consolidated list of limitations:
End-of-Prompt

##  5 Experimental Results and Analysis

Summarization and generative approaches. The results for experiments with
summarization and generative models are listed in Table 8. Although the
objective of generating a summary and suggestive limitation may appear
similar, our experiments reveal that models trained specifically for
summarization do not effectively generate insightful limitations, often merely
extracting sentences from the texts. Whereas, in case of generative models, we
observe a significant decline in output quality when excluding ‘Limitations:’
keyword in the prompt. In the generation approach, Llama 2 and Cerebras models
demonstrate proficiency in limitations generation, but their effectiveness is
hindered by the truncation of the input paper due to token length limitations.
Moreover, they fail to generate limitations across different sections of the
research paper due to the handling of limited context length.

Dense Passage Retrieval. To overcome the limited context issue, we utilized
the relevant passages obtained from the DPR approach to fine-tine the model.
This model then iteratively takes passages and generates limitations for each
passage, which are collated to get the set of limitations for a paper. This
experiment’s performance is generally effective when a limitation could be
derived from a passage. However, despite the DPR approach being effective in
highlighting relevant limitations, it inadvertently points to less pertinent,
paragraph-level limitations due to the lack of overall context of the paper.

Chain modeling. In this approach, we experiment with fine-tuning the models by
providing the summary of the entire paper along with the passages to avoid the
overall context issue. After all the limitations are generated using the input
passages and summary, in the refinement stage, all the duplicates are
discarded.

Table 8: Limitation generation experimental results; BS refers to BERTScore. Model | Approach | Without DPR data | With DPR data
---|---|---|---
R-1 | R-2 | R-L | BS | R-1 | R-2 | R-L | BS
BART-large | Fine-tuning | 30.8 | 4.5 | 15.8 | 82.8 | 10.7 | 0.6 | 8.3 | 82.8
PEGASUS-large | 20.2 | 3.1 | 12.7 | 82.3 | 16.7 | 7.1 | 14.2 | 84.7
T5-base | 27.7 | 4.3 | 16.4 | 83.6 | 18.8 | 7.6 | 16.2 | 85.9
Llama-2-7b | Zero-shot | 21.3 | 3.3 | 12.1 | 81.9 | 16.7 | 5.2 | 8.9 | 82.4
Cerebras GPT2.7B | 17.6 | 2.1 | 12.1 | 78.9 | 19.8 | 4.8 | 10.3 | 80.5
Llama-2-7b | Fine-tuning | 21.4 | 3.1 | 12.7 | 81.1 | 34.8 | 11.0 | 17.7 | 83.5
Cerebras GPT2.7B | 16.9 | 1.9 | 12.0 | 79.1 | 32.4 | 9.6 | 15.9 | 83.4
Table 9: Limitation generation experimental results for chain modeling. The model utilized for distillation/refinement consistently involves Llama2 fine-tuned on Full paper. ‘Fine-tuning’ column indicates the type of the dataset used for fine-tuning. Chain Modeling | Fine-tuning | R-1 | R-2 | R-L | BS
---|---|---|---|---|---
| Llama2-Continuous
---
Full Paper | 24.3 | 6.2 | 15.6 | 82.9
| Llama2-DPR
---
DPR dataset | 33.5 | 9.9 | 16.3 | 83.2
Llama2-FT-Distilled | Full paper | 28.5 | 5.8 | 15.2 | 83.5
Llama2-DPR-Distilled | DPR dataset | 30.4 | 7.6 | 16.2 | 83.3

###  5.1 Evaluation

To assess the performance of the proposed models, we adopt the three types of
evaluation strategies, 1) Automatic evaluation, 2) LLM-base evaluation, and 3)
Human evaluation.

Automatic evaluation. We conduct the automatic evaluation by using the
standard evaluation metrics such as ROUGE [16] and BERTScore [27]. We use the
RoBERTa121212https://huggingface.co/FacebookAI/roberta-large large model for
the BERTScore calculation.

Table 10: Prompts for GPT-4-based evaluation of the generated limitations.
Evaluation Prompt: For the below sets of limitations created for the above
research paper, tell me if they are actually limitations and if each
limitation set is a proper limitation of the paper even though it might not be
mentioned as a limitation in the original paper. Rate them with respect to the
original limitations section in the above paper and with respect to the paper
itself. Also give each set a score of 1 to 5 on the above qualities, with 5
indicating very good limitations for the above paper. Defend the score.
---
{Limitations generated by each model}
End-of-Prompt

LLM-based Evaluation. We perform the LLM-based evaluation by utilizing the
GPT-4 [1] to evaluate the quality of the generated limitations. We use the
zero-shot prompting strategy to obtain the evaluation scores from GPT-4. We
instructed the GPT-4 to assign a score between 1 (least) to 5 (best). As shown
in Table 11, we observe that the ‘Llama2-FT-Distilled’ approach outperforms
the remaining models and the T5-base obtains the lowest score. It indicates
that summarization-specific models fail to generate limitations of research
paper due to their inherent nature of training objective. The prompt used to
perform the LLM-based evaluation is mentioned in Fig 10. We notice that GPT-4
does not thoroughly analyze the limitations. It tends to assign high scores to
general limitations of the model or approach, even if they may not be accurate
within the context of the provided research paper. When prompted again to
verify the limitations once more, it might fail to identify incorrect
limitations if they’re presented in a manner that implies they pertain to any
aspect discussed in the paper. When a specific limitation is singled out and
prompted for re-evaluation, GPT-4 shows improved performance but struggles
until the actual issue with the limitation is pointed out explicitly.
Human Evaluation. We perform the human evaluation of 50 research papers
(selected at random from the test set) and corresponding generated
limitations. The limitations are generated by four different systems including
Llama2-DPR, Llama2, T5-base, and Llama2-FT-Distilled approaches. We asked the
evaluators to rate Yes, No, or Partial for each of the following questions.

Q1. Whether the limitation generated by the model makes sense or not?, Q2. Is
there any overlap between the gold and generated limitation?, Q3. The
generated limitation is an actual limitation or not (can be a summary or
prospective work), Q4. The generated limitation contains any hallucinations,
repetitive sentences, or grammatical errors.

The results of the human evaluation are detailed in Table 12. Our findings and
manual evaluations suggest a superior performance of chain modeling using
Llama2 trained on the full paper dataset for both limitation generation and
reduction step (Llama2-FT-Distilled). This approach yields fewer but proper,
coherent limitations, effectively reducing non-limitation and speculative
content. The enhancement in the quality of generated limitations can be
credited to the comprehensive summary provided to the model, enabling it to
grasp the entirety of the paper’s context. Also, the distillation step of the
chain modeling adds coherence and structure to the results. The DPR dataset-
trained model (Llama2-DPR-Distilled) closely trails behind, likely owing to
its proficiency in generating highly pertinent limitations. However, it
occasionally produces generic limitations rather than focused ones, such as
“The model employs Micro-F1 scores as the primary evaluation metric, yet other
metrics might be more suitable depending on the particular task or
application”.

Table 11: Automatic evaluation using GPT-4 | T5-base |  | Llama2
---
| Llama2-DPR
---
| Llama2-FT-Distilled
---
Score | 2.71 | 3.60 | 3.12 | 4.10
Table 12: Human evaluation of LLMs generated limitations; For Q1-Q3, the higher values of ‘Yes’ are preferred, whereas for Q4 the higher values of ‘No’ are desired. | Llama2-DPR | Llama2 | T5-base | Llama2-FT-Distilled
---|---|---|---|---
| Yes |  No | Partial | Yes |  No | Partial | Yes |  No | Partial |  Yes |  No |  Partial
Q1 | 10 | 24 | 16 | 19 |  12 | 19 | 10 | 22 | 18 |  35 |  04 |  11
Q2 |  06 | 36 |  08 |  05 |  31 | 14 |  02 | 34 | 14 |  14 |  15 |  21
Q3 |  08 | 24 | 18 | 22 |  13 | 15 | 10 | 21 | 19 |  31 |  04 |  15
Q4 | 18 | 26 |  07 | 11 |  30 |  09 | 27 | 12 | 11 |  10 |  31 |  09

Error analysis. As part of human evaluation, Q4 helps to obtain insights for
the error analysis of the models. We notice that the ‘Llama2-FT-Distilled’
model generates better limitations compared to other models. The limitations
generated by ‘Llama2-DPR’ and ‘T5-base’ do not make sense and contain a lot of
noisy information or produce the summary of the research paper rather than
limitations. Moreover, the T5-base model is highly prone to directly copying
several phrases from the research paper. Llama2-DPR considers local
limitations, referring to limitations mentioned in a passage that refers to
another paper or approach but are not the limitations of the current paper. We
have also observed that more than 50% of limitations generated from T5-base
are prone to hallucination or contain repetitive phrases. Despite generating
lengthy limitations, most of the ‘Llama2-FT-Distilled’ model-generated
limitations make sense.

##  6 Challenges and Future work

In this section, we explore the key challenges and potential opportunities
associated with the SLG task.
Complexity of the SLG task. As illustrated in Table 3, it is often difficult
to infer the actual limitations solely from the content of the papers due to
the lack of detailed context surrounding these limitations. Consequently,
predicting such nuanced limitations poses a challenge for the model. Our
experiments reveal that both summarization and open-text generation strategies
struggle to generate these intricate limitations. Therefore, in our
experiments, we place emphasis on the fine-tuning phase, with the expectation
that the model will learn to discern similarities and differences across
research papers, enabling it to infer nuanced limitations more effectively.
Furthermore, the concept of similarity among papers can be explicitly modeled
by incorporating auxiliary information, such as citations.
Evaluation metrics suitability. Lexical overlapping-based metrics such as
ROUGE operate on n-gram-based matching, yet many generated limitations feature
valid novel sentences and phrase formulations compared to those mentioned in
the research papers. This disparity makes lexical matching-based metrics
imperfect for evaluating the task of SLG. As shown in Tables 8 and 9, We find
no notable discernable variation in the BERTScore values across the models.
However, our human evaluation reveals considerable variations in the quality
of the limitations generated by different models. Developing a tailored
evaluation metric for the SLG task stands out as the most promising path
forward.
Multi-modal content. We do not explore the utilization of non-textual elements
such as images and tables present in the research paper to generate the
limitations. Images such as architecture diagrams and plots, along with tables
like result tables and ablation tables, can provide supplementary context,
especially for grasping nuanced potential limitations.
Coherence and relevance. We observe that despite the apparent superior
performance of DPR-based models, they fail to generate coherent limitations.
The best Llama2-FT-Distilled model also generates a few speculative
limitations and has difficulty in generating highly relevant limitations for
every paper.
Open-ended generation of LLMs. As illustrated in Table 12, around 20% of the
limitations generated with the aid of LLMs are susceptible to issues such as
hallucination, repetitions, and grammatically incorrect sentence structures.
Controlling these issues leads to the generation of more faithful limitations.
Controllability. Recent advancements in controllable generation emphasize the
ability to address the specific intentions of users [23]. However, we note
that LLMs occasionally struggle to consistently generate specific structured
limitations. At times, the models produce limitations as bullet points, while
in other instances, they generate them as paragraphs.

##  7 Limitations

In this work, our focus is solely on extracting limitations from only textual
content within research papers. Nevertheless, incorporating non-textual
elements like tables and images could enhance the generation of more
contextually relevant limitations. Due to compute constraints, we experimented
with the Llama 2 7B and Cerebras-GPT 1.3B models. The automated evaluation
metrics used are insufficient for the complexity of this task, so we
supplemented our evaluation process with manual evaluations. Since the LimGen
dataset, sourced from the ACL anthology, it may exhibit bias towards
generating limitations specific to computational linguistic aspects. As a
result, its applicability to other domains like physics, chemistry, and
mathematical research studies could be limited. We listed the limitations
generated by our best model for this paper in Table 13. We observe that,
despite the broad coverage of the limitations, the model generates few of the
sub-standard limitations.

Table 13: Limitations generated by our best-performing model for our paper
(best viewed in color). Magenta indicates the appropriate limitations, whereas
Blue represents sub-standard limitations. Hallucination: The model may
generate limitations that are not present in the input text, which can lead to
inaccurate or irrelevant information in the generated limitations. Repetition:
The model may repeat similar phrases or sentences multiple times in the
generated limitations, which can result in a lack of diversity in the
generated limitations. Limited training data: The model may struggle to
generate limitations that are not present in the training data, which can
affect the quality of the output. Inadequate evaluation metrics: The authors
do not use adequate evaluation metrics to assess the performance of their
approach, which can affect the validity of their results. Lack of
consideration of the research question: The authors do not consider the
research question when generating limitations, which can lead to inaccurate or
irrelevant limitations. Lack of consideration of the methodology: The authors
do not consider the methodology used in the research paper when generating
limitations, which can lead to inaccurate or irrelevant limitations.
---

##  8 Conclusions

In this paper, we introduce the novel task of Suggestive Limitation Generation
(SLG) for research papers, aiming to provide reviewers with potential
limitations of the underlying work, thereby assisting in the review process.
We compile a dataset of 4068 research papers and corresponding limitations
from the ACL anthology. We propose an LLM-based baseline for SLG tasks and
conduct several ablation studies. Subsequently, we perform a thorough
evaluation of the proposed models with automatic, LLM-based, and human
evaluation schemes. Moving forward, our plans involve incorporating images and
tabular content in addition to text for the SLG task.

##  9 Ethics Statement

In the creation of the LimGen dataset, we did not collect any personal data.
Instead, we rely on the publicly accessible dataset from the ACL anthology.
This research did not involve any user data. In the release of LimGen, all
personal information pertaining to the authors of the research papers will be
excluded.

## References

  * [1] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
  * [2] Cachola, I., Lo, K., Cohan, A., Weld, D.S.: Tldr: Extreme summarization of scientific documents. In: Findings of the Association for Computational Linguistics: EMNLP 2020. pp. 4766–4777 (2020)
  * [3] Cohan, A., Dernoncourt, F., Kim, D.S., Bui, T., Kim, S., Chang, W., Goharian, N.: A discourse-aware attention model for abstractive summarization of long documents. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). pp. 615–621 (2018)
  * [4] Cohan, A., Feigenblat, G., Ghosal, T., Shmueli-Scheuer, M.: Overview of the first shared task on multi perspective scientific document summarization (mup). In: Proceedings of the Third Workshop on Scholarly Document Processing. pp. 263–267 (2022)
  * [5] Collins, E., Augenstein, I., Riedel, S.: A supervised approach to extractive summarisation of scientific papers. In: Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017). pp. 195–205 (2017)
  * [6] Dao, T., Fu, D., Ermon, S., Rudra, A., Ré, C.: Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35, 16344–16359 (2022)
  * [7] Delort, J.Y., Alfonseca, E.: Dualsum: a topic-model based approach for update summarization. In: Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics. pp. 214–223 (2012)
  * [8] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171–4186 (2019)
  * [9] Dey, N., Gosal, G., Zhiming, Chen, Khachane, H., Marshall, W., Pathria, R., Tom, M., Hestness, J.: Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster (2023)
  * [10] Hayashi, H., Kryściński, W., McCann, B., Rajani, N., Xiong, C.: What’s new? summarizing contributions in scientific literature. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. pp. 1019–1031 (2023)
  * [11] Hu, E.J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. In: International Conference on Learning Representations (2021)
  * [12] Kumarasinghe, D., de Silva, N.: Automatic generation of abstracts for research papers. In: Proceedings of the 34th Conference on Computational Linguistics and Speech Processing (ROCLING 2022). pp. 221–229 (2022)
  * [13] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C.H., Gonzalez, J., Zhang, H., Stoica, I.: Efficient memory management for large language model serving with pagedattention. In: Proceedings of the 29th Symposium on Operating Systems Principles. pp. 611–626 (2023)
  * [14] Lev, G., Shmueli-Scheuer, M., Herzig, J., Jerbi, A., Konopnicki, D.: Talksumm: A dataset and scalable annotation method for scientific paper summarization based on conference talks. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 2125–2131 (2019)
  * [15] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.: Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 7871–7880 (2020)
  * [16] Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. pp. 74–81. Association for Computational Linguistics, Barcelona, Spain (Jul 2004)
  * [17] Liu, M.H., Yen, A.Z., Huang, H.H., Chen, H.H.: Contributionsum: Generating disentangled contributions for scientific papers. In: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. pp. 5351–5355 (2023)
  * [18] Mao, Y., Zhong, M., Han, J.: CiteSum: Citation text-guided scientific extreme summarization and domain adaptation with limited supervision. In: Goldberg, Y., Kozareva, Z., Zhang, Y. (eds.) Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 10922–10935. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Dec 2022)
  * [19] Meng, R., Thaker, K., Zhang, L., Dong, Y., Yuan, X., Wang, T., He, D.: Bringing structure into summaries: a faceted summarization dataset for long scientific documents. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). pp. 1080–1089 (2021)
  * [20] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21(1), 5485–5551 (2020)
  * [21] Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3982–3992 (2019)
  * [22] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
  * [23] Urlana, A., Mishra, P., Roy, T., Mishra, R.: Controllable text summarization: Unraveling challenges, approaches, and prospects–a survey. arXiv preprint arXiv:2311.09212 (2023)
  * [24] Urlana, A., Surange, N., Shrivastava, M.: Ltrc@ mup 2022: Multi-perspective scientific document summarization using pre-trained generation models. In: Proceedings of the Third Workshop on Scholarly Document Processing. pp. 279–284 (2022)
  * [25] Yasunaga, M., Kasai, J., Zhang, R., Fabbri, A.R., Li, I., Friedman, D., Radev, D.R.: Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 7386–7393 (2019)
  * [26] Zhang, J., Zhao, Y., Saleh, M., Liu, P.: Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In: International Conference on Machine Learning. pp. 11328–11339. PMLR (2020)
  * [27] Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)


# Unisolvence of random Kansa collocation by Thin-Plate Splines for the
Poisson equation

F. Dell’Accio  University of Calabria, Rende (CS), Italy  A. Sommariva  M.
Vianello

###### Abstract

Existence of sufficient conditions for unisolvence of Kansa unsymmetric
collocation for PDEs is still an open problem. In this paper we make a first
step in this direction, proving that unsymmetric collocation matrices with
Thin-Plate Splines for the 2D Poisson equation are almost surely nonsingular,
when the discretization points are chosen randomly on domains with analytic
boundary.

##  1 Introduction

Kansa unsymmetric collocation, originally proposed in the mid ’80s [11], has
become over the years a popular meshless method for the discretization of
boundary value problems for PDEs. Despite its wide and successful adoption for
the numerical solution of a variety of physical and engineering problems (cf.
e.g. [4] with the references therein), a sound theoretical foundation
concerning unisolvence of the corresponding linear systems is still missing.
Indeed, it was shown by Hon and Schaback [10] that there exist point
configurations that lead to singularity of the collocation matrices, though
these are very special and “rare”cases. For this reason greedy and other
approaches have been developed to overcome the theoretical problem and ensure
invertibility, cf. e.g. [14, 18]. On the other hand, in the textbook [8] one
can read : “Since the numerical experiments by Hon and Schaback show that
Kansa’s method cannot be well-posed for arbitrary center locations, it is now
an open question to find sufficient conditions on the center locations that
guarantee invertibility of the Kansa matrix”, and the situation does not seem
to have changed so far.

In this paper we make a first step in this direction, proving that unsymmetric
collocation matrices with Thin-Plate Splines (without polynomial addition) for
the 2D Poisson equation are almost surely nonsingular, when the discretization
points are chosen randomly on domains with analytic boundary. Though TPS are
not the most adopted option for Kansa collocation, they have been often used
in the meshless literature, cf. e.g. [4, 5, 20] with the references therein.
One of their most relevant features is that they are scale invariant, thus
avoiding the delicate matter of the scaling choice with scale dependent RBF,
which is still an active research topic, cf. e.g. [2, 13]. On the other hand,
the fact that TPS without polynomial addition can guarantee unisolvence in the
interpolation framework has been recently recognized experimentally in [17]
and theoretically in [1, 6].

As we shall see, one of the key aspects is that Thin-Plate Splines
$\phi(\|P-A\|_{2})$, which correspond to the radial functions

$\phi(r)=r^{2\nu}\log(r)\;,\;\;\nu\in\mathbb{N}\;,$ (1)

are real analytic functions off their center $A$, due to analyticity of the
univariate functions $\log(\cdot)$ and $\sqrt{\cdot}$ in $\mathbb{R}^{+}$.
Analiticity together with the presence of a singularity at the center will be
the key ingredients of our unisolvence result by random collocation.

##  2 Unisolvence of random Kansa collocation

Consider the Poisson equation with Dirichlet boundary conditions (cf. e.g.
[7])

$\left\\{\begin{array}[]{l}\Delta u(P)=f(P)\;,\;P\in\Omega\\\
u(P)=g(P)\;,\;P\in\partial\Omega=\gamma([a,b])\;,\end{array}\right.$ (2)

where we assume that $\Omega\subset\mathbb{R}^{2}$ is a domain with analytic
boundary (a bounded connected open set whose boundary is an analytic curve),
namely a curve $\gamma:[a,b]\to\mathbb{R}^{2}\;,\;\gamma(a)=\gamma(b)$, that
is analytic and regular (i.e. $\gamma^{\prime}(t)\neq(0,0)$ for every
$t\in[a,b]$).

In Kansa collocation (see e.g. [8, 10, 11, 14, 18, 19]) one determines a
function

$u_{N}(P)=\sum_{j=1}^{n}{c_{j}\,\phi_{j}(P)}+\sum_{k=1}^{m}{d_{k}\,\psi_{k}(P)}\;,\;\;N=n+m\;,$
(3)

where

$\phi_{j}(P)=\phi(\|P-P_{j}\|_{2})\;,\;\;\\{P_{1},\dots,P_{n}\\}\subset\Omega\;,$
(4)
$\psi_{k}(P)=\phi(\|P-Q_{k}\|_{2})\;,\;\;\\{Q_{1},\dots,Q_{m}\\}\subset\partial\Omega\;,$
(5)

such that

$\left\\{\begin{array}[]{l}\Delta u_{N}(P_{i})=f(P_{i})\;,\;i=1,\ldots,n\\\
u_{N}(Q_{h})=g(Q_{h})\;,\;h=1,\ldots,m\;.\end{array}\right.$ (6)

The following facts will be used below. Defining $\phi_{A}(P)=\phi(\|P-A\|)$,
we have $\phi_{A}(B)=\phi_{B}(A)$ and $\Delta\phi_{A}(B)=\Delta\phi_{B}(A)$.
In fact, the Laplacian in polar coordinates centered at $A$ (cf. e.g. [7,
Ch.2]) is the radial function

$\Delta\phi_{A}={\frac{\partial^{2}\phi}{\partial^{2}r}}+{\frac{1}{r}}\frac{\partial\phi}{\partial
r}=4\nu r^{2(\nu-1)}(\nu\log(r)+1)\;.$ (7)

Moreover, $\phi_{A}(A)=0$ and $\Delta\phi_{A}(A)=0$ for $\nu\geq 2$, since
$\Delta\phi\to 0$ as $r\to 0$.

Kansa collocation can be rewritten in matrix form as

$\left(\begin{array}[]{cc}\Delta\Phi&\Delta\Psi\\\ \\\
\Phi&\Psi\end{array}\right)\left(\begin{array}[]{c}\mathbf{c}\\\ \\\
\mathbf{d}\end{array}\right)=\left(\begin{array}[]{c}\mathbf{f}\\\ \\\
\mathbf{g}\end{array}\right)$ (8)

where the block matrix is

$K_{N}=K_{N}(\\{P_{i}\\},\\{Q_{h}\\})=\left(\begin{array}[]{cc}\Delta\Phi&\Delta\Psi\\\
\\\ \Phi&\Psi\end{array}\right)$
$=\left(\begin{array}[]{ccccccc}0&\cdots&\cdots&\Delta\phi_{n}(P_{1})&\Delta\psi_{1}(P_{1})&\cdots&\Delta\psi_{m}(P_{1})\\\
\\\ \vdots&\ddots&&\vdots&\vdots&\cdots&\vdots\\\
\vdots&&\ddots&\vdots&\vdots&\cdots&\vdots\\\ \\\
\Delta\phi_{1}(P_{n})&\cdots&\cdots&0&\Delta\psi_{1}(P_{n})&\cdots&\Delta\psi_{m}(P_{n})\\\
\\\ \phi_{1}(Q_{1})&\cdots&\cdots&\phi_{n}(Q_{1})&0&\cdots&\psi_{m}(Q_{1})\\\
\\\ \vdots&\cdots&\cdots&\vdots&\vdots&\ddots&\vdots\\\ \\\
\phi_{1}(Q_{m})&\cdots&\cdots&\phi_{n}(Q_{m})&\psi_{1}(Q_{m})&\cdots&0\\\ \\\
\end{array}\right)$

and $\textbf{f}=\\{f(P_{i})\\}_{i=1,\ldots,n}$,
${\textbf{g}}=\\{g(Q_{h})\\}_{h=1,\ldots,m}$.

We can now state and prove our main result.

###### Theorem 1

Let $K_{N}$ be the TPS Kansa collocation matrix defined above, with $N=n+m\geq
2$, where $\\{P_{i}\\}$ is a sequence of independent uniformly distributed
random points in $\Omega$, and $\\{Q_{h}\\}$ a sequence of independent
uniformly distributed points on $\partial\Omega$. Namely,
$\\{Q_{h}\\}=\\{\gamma(t_{h})\\}$ with $\\{t_{h}\\}$ sequence of independent
identically distributed random abscissas in $(a,b)$ with respect to the
arclength density $\|\gamma^{\prime}(t)\|_{2}/L$, $L=length(\gamma([a,b]))$.

Then for every $N\geq 2$ the matrix $K_{N}$ is a.s. (almost surely)
nonsingular.

Proof. The proof proceeds by complete induction on $N$. For the induction
base, we prove that $\mbox{det}(K_{N})$ is a.s. nonzero for $N=2$, that is for
$n=2$ and $m=0$, or $n=0$ and $m=2$, or $n=1$ and $m=1$. In the first case,

$\mbox{det}(K_{2})=-\Delta\phi_{2}(P_{1})\Delta\phi_{1}(P_{2})=-(\Delta\phi_{1}(P_{2}))^{2}$
$=-16\nu^{2}\|P_{2}-P_{1}\|_{2}^{4\nu-4}\left(\nu\log(\|P_{2}-P_{1}\|_{2})+1\right)^{2}$

which vanishes iff $P_{2}=P_{1}$ (an event with null probability) or $P_{2}$
falls on (the intersection with $\Omega$ of) the curve
$\nu\log(\|P-P_{1}\|_{2})+1=0$, that is on the circle

$\|P-P_{1}\|_{2}^{2}=\exp(-2/\nu)\;.$

But this event has null probability, since any algebraic curve is a null set
in $\mathbb{R}^{2}$.

In the second case,

$\mbox{det}(K_{2})=-\psi_{2}(Q_{1})\psi_{1}(Q_{2})=-\psi_{1}^{2}(Q_{2})=-\psi_{1}^{2}(\gamma(t_{2}))\;.$

Now, given $P_{1}=\gamma(t_{1})$, the function
$\lambda(t)=\psi_{1}^{2}(\gamma(t))$ is analytic in $(a,t_{1})$ and in
$(t_{1},b)$. Then $\psi_{1}^{2}(\gamma(t_{2}))$ is zero iff $t_{2}=t_{1}$ (an
event that has null probability), or $t_{2}$ falls on the zero set of
$\lambda$ in $(a,t_{1})$ or $(t_{1},b)$. Again this event has null probability
since the zero set of an univariate analytic function in an open interval is a
null set (cf. [12, 15]).

As for the third case, assume that $Q_{1}$ is chosen on the boundary (randomly
or not) and that $P_{1}$ is chosen randomly in the interior. Since

$\mbox{det}(K_{2})=-\phi_{1}(Q_{1})\Delta\psi_{1}(P_{1})$
$=-4\nu\phi_{1}(Q_{1})\|P_{1}-Q_{1}\|_{2}^{2\nu-2}\left(\nu\log(\|P_{1}-Q_{1}\|_{2})+1\right)\;,$

and $\phi_{1}(Q_{1})\neq 0$ being $P_{1}\neq Q_{1}$, the determinant vanishes
if and only if $P_{1}$ falls on (the intersection with $\Omega$ of) the curve
$\nu\log(\|P-Q_{1}\|_{2})+1=0$, that is on the circle

$\|P-Q_{1}\|_{2}^{2}=\exp(-2/\nu)$

and again this event has null probability.

For the inductive step, we consider separately the case where a boundary point
is added, for which we define the matrix

$U(P)=\left(\begin{array}[]{cccccccc}0&\cdots&\cdots&\Delta\phi_{n}(P_{1})&\Delta\psi_{1}(P_{1})&\cdots&\Delta\psi_{m}(P_{1})&\Delta\phi_{1}(P)\\\
\\\ \vdots&\ddots&&\vdots&\vdots&\cdots&\vdots&\vdots\\\
\vdots&&\ddots&\vdots&\vdots&\cdots&\vdots&\vdots\\\ \\\
\Delta\phi_{1}(P_{n})&\cdots&\cdots&0&\Delta\psi_{1}(P_{n})&\cdots&\Delta\psi_{m}(P_{n})&\Delta\phi_{n}(P)\\\
\\\
\phi_{1}(Q_{1})&\cdots&\cdots&\phi_{n}(Q_{1})&0&\cdots&\psi_{m}(Q_{1})&\psi_{1}(P)\\\
\\\ \vdots&\cdots&\cdots&\vdots&\vdots&\ddots&\vdots&\vdots\\\ \\\
\phi_{1}(Q_{m})&\cdots&\cdots&\phi_{n}(Q_{m})&\psi_{1}(Q_{m})&\cdots&0&\psi_{m}(P)\\\
\\\ \phi_{1}(P)&\cdots&\cdots&\phi_{n}(P)&\psi_{1}(P)&\cdots&\psi_{m}(P)&0\\\
\\\ \end{array}\right)$

Observe that in this case $K_{N+1}=U(Q_{m+1})$. Indeed,
$\psi_{k}(Q_{h})=\psi_{h}(Q_{k})$ and
$\Delta\phi_{i}(Q_{m+1})=\Delta\psi_{m+1}(P_{i})$.

Differently, if an interior point is added, we define the matrix

$V(P)=\left(\begin{array}[]{cccccccc}0&\cdots&\cdots&\Delta\phi_{n}(P_{1})&\Delta\phi_{1}(P)&\Delta\psi_{1}(P_{1})&\cdots&\Delta\psi_{m}(P_{1})\\\
\\\ \vdots&\ddots&&\vdots&\vdots&\vdots&\cdots&\vdots\\\
\vdots&&\ddots&\vdots&\vdots&\vdots&\cdots&\vdots\\\ \\\
\Delta\phi_{1}(P_{n})&\cdots&\cdots&0&\Delta\phi_{n}(P)&\Delta\psi_{1}(P_{n})&\cdots&\Delta\psi_{m}(P_{n})\\\
\\\
\Delta\phi_{1}(P)&\cdots&\cdots&\Delta\phi_{n}(P)&0&\Delta\psi_{1}(P)&\cdots&\Delta\psi_{m}(P)\\\
\\\
\phi_{1}(Q_{1})&\cdots&\cdots&\phi_{n}(Q_{1})&\psi_{1}(P)&0&\cdots&\psi_{m}(Q_{1})\\\
\\\ \vdots&\cdots&\cdots&\vdots&\vdots&\vdots&\ddots&\vdots\\\ \\\
\phi_{1}(Q_{m})&\cdots&\cdots&\phi_{n}(Q_{m})&\psi_{m}(P)&\psi_{1}(Q_{m})&\cdots&0\\\
\\\ \end{array}\right)$

Observe that in this case $K_{N+1}=V(P_{n+1})$ since
$\psi_{k}(P_{n+1})=\phi_{n+1}(Q_{k})$ and
$\Delta\phi_{j}(P_{i})=\Delta\phi_{i}(P_{j})$.

Concerning the determinants, applying Laplace determinantal rule on the last
row of $U(P)$ we see that for every $\ell$, $1\leq\ell\leq m$, we get the
representation

$F(P)=\mbox{det}(U(P))=\delta_{N-1}\psi_{\ell}^{2}(P)+A(P)\psi_{\ell}(P)+B(P)$
(9)

where

$|\delta_{N-1}|=|\mbox{det}(K_{N-1}(\\{P_{i}\\},\\{Q_{h}\\}_{h\neq\ell}))|$
$A\in\mbox{span}\\{\phi_{j},\Delta\phi_{j},\psi_{k}\,;\,1\leq j\leq
n\,,\,1\leq k\leq m\,,\,k\neq\ell\\}$
$B\in\mbox{span}\\{\phi_{i}\Delta\phi_{j},\psi_{k}\phi_{i},\psi_{k}\Delta\phi_{i},\psi_{k}\psi_{h}\,;\,1\leq
i,j\leq n\,,\,1\leq k,h\leq m\,,\,k,h\neq\ell\\}\;.$

Similarly, developing $det(V(P))$ by the $(n+1)$-row we have

$G(P)=\mbox{det}(V(P))=-\mbox{det}(K_{N-1})(\Delta\phi_{n}(P))^{2}+C(P)\Delta\phi_{n}(P)+D(P)$
(10)

where

$C\in\mbox{span}\\{\Delta\phi_{j},\psi_{k},\Delta\psi_{k}\,;\,1\leq j\leq
n-1\,,\,1\leq k\leq m\\}$
$D\in\mbox{span}\\{\Delta\phi_{i}\Delta\phi_{j},\Delta\phi_{i}\Delta\psi_{h},\psi_{k}\Delta\phi_{i},\psi_{k}\Delta\psi_{h}\,;\,1\leq
i,j\leq n-1\,,\,1\leq k,h\leq m\\}\;.$

First, we prove that $G$ is not identically zero in $\Omega$ if
$\mbox{det}(K_{N-1})\neq 0$ (the latter a.s. holds by inductive hypothesis).
Let $P(t)=P_{n}+t(1,0)$, $t\in\mathbb{R}$, and $r(t)=\|P(t)-P_{n}\|_{2}=|t|$.
If $G\equiv 0$ then $G(P(t))\equiv 0$ in neighborhood of $t=0$. Then, we would
locally have

$u^{2}(t)=c(t)u(t)+d(t)\;,\;\;u(t)=\Delta\phi_{n}(P(t))\;,$ (11)

where $c(t)=C(P(t))/\mbox{det}(K_{N-1})$ and
$d(t)=D(P(t))/\mbox{det}(K_{N-1})$. Notice that both $c$ and $d$ are analytic
in a neighborhood of $t=0$, since $C$ and $D$ are analytic in a neighborhood
of $P_{n}$. By (11) and (7) we get

$u(t)=4\nu t^{2(\nu-1)}\left(\nu\log(|t|)+1\right)\;.$ (12)

Clearly $c$ cannot be identically zero there, otherwise $u^{2}$ would be
analytic at $t=0$ and thus would have an algebraic order of infinitesimal as
$t\to 0$, whereas by (12) we have $u^{2}(t)\sim
16\nu^{4}t^{4(\nu-1)}\log^{2}(|t|)$. Hence taking the Maclaurin expansion of
$c$ we get $c(t)\sim c_{s}t^{s}$ as $t\to 0$ for some $s\geq 0$, the order of
the first nonvanishing derivative at $t=0$. Now, $u^{2}(t)\sim
16\nu^{4}t^{4(\nu-1)}\log^{2}(|t|)$, whereas by $u^{2}\equiv cu+d$ we would
have $u^{2}(t)\sim 4\nu^{2}c_{s}t^{s+2(\nu-1)}\log(|t|)+d_{p}t^{p}$, where
either $d(0)\neq 0$ and $p=0$, or $d(0)=0$ and $p>0$ (the order of the first
nonvanishing derivative at $t=0$). Then we get a contradiction, since $u^{2}$
cannot have two distinct limits or orders of infinitesimal at the same point.

Moreover, $G$ is clearly continuous in $\Omega$ and analytic in
$\Omega\setminus\\{P_{1},\dots,P_{n}\\}$, since all the functions involved in
its definition (10) are analytic up to their own center. Consequently, if
$\mbox{det}(K_{N-1})\neq 0$ by continuity $G$ is not identically zero also in
$\Omega\setminus\\{P_{1},\dots,P_{n}\\}$.

Then, $\mbox{det}(K_{N+1})=\mbox{det}(V(P_{n+1}))=G(P_{n+1})$ is a.s. nonzero,
since the zero set of a not identically zero real analytic function on an open
connected set in $\mathbb{R}^{d}$ is a null set (cf. [15] for an elementary
proof). More precisely, denoting by $Z_{G}$ the zero set of $G$ in $\Omega$,
we have that

$Z_{G}=(Z_{G}\cap\\{P_{1},\dots,P_{n}\\})\cup(Z_{G}\cap(\Omega\setminus\\{P_{1},\dots,P_{n}\\}))\;.$

Hence $Z_{G}$ is a null set if $G\not\equiv 0$, because the first intersection
is a finite set, and the second is the zero set of a not identically zero real
analytic function. Considering the probability of the corresponding events and
recalling that $\mbox{det}(K_{N-1})\neq 0$ (which a.s. holds) implies
$G\not\equiv 0$, we can then write

$\mbox{prob}\\{\mbox{det}(K_{N+1})=0\\}=\mbox{prob}\\{G(P_{n+1})=0\\}$
$=\mbox{prob}\\{G\equiv 0\\}+\mbox{prob}\\{G\not\equiv 0\;\&\;P_{n+1}\in
Z_{G}\\}=0+0=0\;,$

and this branch of the inductive step is completed.

We turn now to the branch of the inductive step where a boundary point is
added. In this case we consider the function $F$ in (9) restricted to the
boundary, that is $F(P(t))$ with $P(t)=\gamma(t)$, $t\in(a,b)$, which for
every fixed $\ell\in\\{1,\dots,m\\}$ has the representation

$F(\gamma(t))=\mbox{det}(U(\gamma(t)))=\delta_{N-1}v^{2}(t)+A(\gamma(t))v(t)+B(\gamma(t))$

where

$v(t)=\psi_{\ell}(\gamma(t))=r_{\ell}^{2\nu}(t)\log(r_{\ell}(t))\;,\;\;r_{\ell}(t)=\|\gamma(t)-Q_{\ell}\|_{2}$
(13)

with $Q_{\ell}=\gamma(t_{\ell})$, $\;t_{\ell}\in(a,b)$. We claim that if
$\delta_{N-1}\neq 0$ (which a.s. holds by inductive hypothesis),
$F\circ\gamma$ cannot be identically zero in any of the two connected
components of $(a,b)\setminus\\{t_{1},\dots,t_{m}\\}$ (i.e., the subintervals)
having $t_{\ell}$ as extremum. Otherwise, we would have in a left or in a
right neighborhood of $t_{\ell}$

$v^{2}(t)=\alpha(t)v(t)+\beta(t)\;,$ (14)

where $\alpha(t)=A(\gamma(t))/\delta_{N-1}$ and
$\beta(t)=B(\gamma(t))/\delta_{N-1}$ are both analytic in a full neighborhood
of $t_{\ell}$. Notice that, since $\gamma^{\prime}(t_{\ell})\neq(0,0)$ (the
curve is regular),
$r_{\ell}(t)\sim\|\gamma^{\prime}(t_{\ell})\|_{2}|t-t_{\ell}|$ which by (13)
gives $v(t)\sim\|\gamma^{\prime}(t_{\ell})\|_{2}^{2\nu}(t-t_{\ell})^{2\nu}\l
og(|t-t_{\ell}|)$ and
$v^{2}(t)\sim\|\gamma^{\prime}(t_{\ell})\|_{2}^{4\nu}(t-t_{\ell})^{4\nu}\log^{2}(|t-t_{\ell}|)$
as $t\to t_{\ell}$. Now $\alpha$ cannot be identically zero in any left or
right neighborhood, otherwise $v^{2}\equiv\beta$ there and would have an
algebraic order of infinitesimal at $t_{\ell}$. Hence taking the Taylor
expansion of $\alpha$ we get $\alpha(t)\sim\alpha_{s}(t-t_{\ell})^{s}$ as
$t\to t_{\ell}$ for some $s\geq 0$, the order of the first nonvanishing
derivative at $t=t_{\ell}$. On the other hand, by $v^{2}\equiv\alpha v+\beta$
locally, we would have
$v^{2}(t)\sim\|\gamma^{\prime}(t_{\ell})\|_{2}^{2\nu}\alpha_{s}(t-t_{\ell})^{s+2\nu}\log(|t-t_{\ell}|)+\beta_{p}(t-t_{\ell})^{p}$,
where either $\beta(t_{\ell})\neq 0$ and $p=0$, or $\beta(t_{\ell})=0$ and
$p>0$ (the order of the first nonvanishing derivative at $t=t_{\ell}$). Again
we get a contradiction, since $v^{2}$ cannot have two distinct limits or
orders of infinitesimal at the same point.

The result is that $F\circ\gamma$ is a.s. not identically zero in any
connected component of $(a,b)\setminus\\{t_{1},\dots,t_{m}\\}$. Then,
$\mbox{det}(K_{N+1})=\mbox{det}(U(Q_{m+1}))=F(\gamma(t_{m+1}))$ is a.s.
nonzero. In fact, observe that $F\circ\gamma$ is analytic in
$(a,b)\setminus\\{t_{1},\dots,t_{m}\\}$, since $F$ is analytic in
$\mathbb{R}^{2}\setminus(\\{Q_{1},\dots,Q_{m}\\}\cup\\{P_{1},\dots,P_{n}\\})$.
Moreover, denoting by $Z_{F\circ\gamma}$ the zero set of $F\circ\gamma$ in
$(a,b)$, we have that

$Z_{F\circ\gamma}=(Z_{F\circ\gamma}\cap\\{t_{1},\dots,t_{m}\\})\cup(Z_{F\circ\gamma}\cap((a,b)\setminus\\{t_{1},\dots,t_{m}\\}))\;.$

Hence $Z_{F\circ\gamma}$ is a null set if ${F\circ\gamma}\not\equiv 0$,
because the first intersection is a finite set, and the second is the
componentwise finite union of the zero sets of a not identically zero real
analytic function on each connected component. Considering the probability of
the corresponding events and recalling that $\mbox{det}(K_{N-1})\neq 0$ (which
a.s. holds) implies $F\circ\gamma\not\equiv 0$, we can then write

$\mbox{prob}\\{\mbox{det}(K_{N+1})=0\\}=\mbox{prob}\\{F(Q_{m+1})=0\\}$
$=\mbox{prob}\\{F\circ\gamma\equiv 0\\}+\mbox{prob}\\{F\circ\gamma\not\equiv
0\;\&\;t_{m+1}\in Z_{F\circ\gamma}\\}=0+0=0\;,$

and also the boundary branch of the inductive step is completed. $\square$

###  2.1 Remarks on possible extensions

The result of Theorem 1 is a first step towards a theory of Kansa collocation
unisolvence, and could be extended in several directions within the random
framework. The first extension comes immediately from the fact that a null set
has also measure zero for any continuous measure with density (that is,
absolutely continuous with respect to the Lebesgue measure). We can state
indeed the following

###### Theorem 2

The assertion of Theorem 1 holds true if the points $\\{P_{i}\\}$ are
independent identically distributed with respect any continuous probability
measure with density on $\Omega$, say $\sigma\in L^{1}_{+}(\Omega)$, and the
abscissas $\\{t_{h}\\}$ are independent identically distributed with respect
any continuous probability measure with density on $(a,b)$, say $w\in
L^{1}_{+}(a,b)$.

This extension could be interesting whenever it is known that the solution has
steep gradients or other regions where it is useful to increase the
discretization density. Concerning the implementation of random sampling with
respect to continuous probability densities, we recall the well-known
“acceptance-rejection method”, cf. e.g. [3, 9, 16] with the references
therein.

More difficult but worth of further investigations are:

  * •

extension to $\Omega\subset\mathbb{R}^{d}$, $d\geq 3$;

  * •

extension to other analitic RBF up to the center, e.g. Radial Powers;

  * •

extension to piecewise analytic boundaries;

  * •

extension to other differential operators and/or boundary conditions.

The latter in particular could be challenging, since the operators involved in
the equation and in the boundary conditions may not be radial.

Acknowledgements.

Work partially supported by the DOR funds of the University of Padova, and by
the INdAM-GNCS 2024 Projects “Kernel and polynomial methods for approximation
and integration: theory and application software”.

This research has been accomplished within the RITA “Research ITalian network
on Approximation” and the SIMAI Activity Group ANA&A, and the UMI Group TAA
“Approximation Theory and Applications”.

## References

  * [1] L.P. Bos, A. Sommariva, M. Vianello, A note on polynomial-free unisolvence of polyharmonic splines at random points. https://arxiv.org/abs/2312.13710
  * [2] R. Cavoretto, A. De Rossi, Adaptive procedures for meshfree RBF unsymmetric and symmetric collocation methods, Appl. Math. Comput. 382 (2020), 125354.
  * [3] A. Chalkis, C. Katsamaki, J. Tonelli-Cueto, On the Error of Random Sampling: Uniformly Distributed Random Points on Parametric Curves, ISSAC ’22: Proceedings of the 2022 Intern. Symp. on Symb. and Alg. Comput., July 2022, pp. 273–282. https://arxiv.org/pdf/2203.02832.pdf
  * [4] W. Chen, Z.-J. Fu, C.S. Chen, Different Formulations of the Kansa Method: Domain Discretization, in W. Chen et al., Recent Advances in Radial Basis Function Collocation Methods, SpringerBriefs in Applied Sciences and Technology, 2014.
  * [5] P.P. Chinchapatnam, K. Djidjeli, P.B. Nair, Unsymmetric and symmetric meshless schemes for the unsteady convection–diffusion equation, Comput. Methods Appl. Mech. Engrg. 195 (2006), 2432–-2453.
  * [6] F. Dell’Accio, A. Sommariva, M. Vianello, Random sampling and unisolvent interpolation by almost everywhere analytic functions, Appl. Math. Lett. 145 (2023).
  * [7] L.C. Evans, Partial Differential Equations, Graduate Studies in Mathematics 19, AMS, 1998.
  * [8] G.E. Fasshauer, Meshfree Approximation Methods with Matlab, Interdisciplinary Mathematical Sciences, Vol. 6, World Scientific, 2007.
  * [9] B.D. Flury, Acceptance-Rejection Sampling Made Easy, SIAM Rev. 32 (1990), 474–476.
  * [10] Y.C. Hon, R. Schaback, On unsymmetric collocation by radial basis functions, Appl. Math. Comput. 119 (2001), 177–-186.
  * [11] E.J. Kansa, Application of Hardy’s multiquadric interpolation to hydrodynamics, in Proc. 1986 Simul. Conf., Vol. 4, pp. 111–-117.
  * [12] S.G. Krantz, H.R. Parks, A Primer of Real Analytic Functions, Second Edition, Birkhäuser, Boston, 2002.
  * [13] E. Larsson, R. Schaback, Scaling of radial basis functions, IMA Journal of Numerical Analysis (2023), drad035.
  * [14] L. Ling, R. Opfer, R. Schaback, Results on meshless collocation techniques, Eng. Anal. Bound. Elem. 30 (2006), 247–253.
  * [15] B.S. Mityagin, The Zero Set of a Real Analytic Function, Math. Notes 107 (2020), 529–530.
  * [16] N. Nguyen, G. Ökten, The acceptance-rejection method for low-discrepancy sequences, Monte Carlo Methods Appl. 22 (2016), 133–148.
  * [17] A. Pasioti, On the Constrained Solution of RBF Surface Approximation, MDPI Mathematics 10 (2022), 2582.
  * [18] R. Schaback, H. Wendland, Kernel techniques: From machine learning to meshless methods, Acta Numer. 15 (2006) 543–639.
  * [19] H. Wendland, Scattered Data Approximation, Cambridge Monogr. Appl. Comput. Math., vol. 17, Cambridge Univ. Press, Cambridge, 2005.
  * [20] M. Zerroukat, K. Djidjeli, A. Charafi, Explicit and implicit meshless methods for linear advection–diffusion-type partial differential equations, Int. J. Numer. Methods Eng. 48 (2000), 19–35.