# DTS-SQL: Decomposed Text-to-SQL with Small Large Language Models

Mohammadreza Pourreza
University of Alberta
<EMAIL_ADDRESS>
&Davood Rafiei
University of Alberta
<EMAIL_ADDRESS>

###### Abstract

Leading models for the text-to-SQL task heavily rely on proprietary Large
Language Models (LLMs), posing concerns over data privacy. Closing the
performance gap between small open-source models and large proprietary models
is crucial to mitigate this reliance. To this end, we introduce a novel two-
stage fine-tuning approach that decomposes the task into two simpler tasks.
Through comprehensive evaluation on two large cross-domain datasets and two
small LLMs, we show that this approach improves execution accuracy by 3 to 7
percent, effectively aligning the performance of open-source models with their
proprietary counterparts.

##  1 Introduction

Natural language interfaces for databases allow users to derive insights from
structured databases using natural language instead of complex SQL queries.
Leading open-source methods (Pourreza and Rafiei, 2023; Gao et al., 2023) for
this task heavily depend on proprietary Large language models (LLMs) like
GPT-4 and GPT-3.5-turbo, which have demonstrated superior performance in Text-
to-SQL benchamrks (Yu et al., 2018; Li et al., 2023c; Gan et al., 2021).
However, this reliance on large proprietary models has privacy and cost
implications. For instance, many large enterprises cannot share their customer
data with the model-providing companies due to privacy considerations.
Additionally, cost is a factor, especially for small businesses, in adopting
these models.

Recent attempts to utilize open-source LLMs (Gao et al., 2023) and fine-tune
them using question-SQL query pairs have fallen short of the zero-shot
performance of GPT-3.5-turbo. Table 1 presents a performance comparison of the
fine-tuned open-source LLMs on the Spider development set, contrasting with
methods that employ GPT-4’s prompting techniques. This paper aims to address
this disparity by introducing a novel two-step decomposed fine-tuning method,
employing two smaller LLMs, each with a paramater size of 7 billion. This
approach achieves a performance comparable to methods that are using GPT-4
with few-shot learning and well-designed prompts.

Model | EX | EM
---|---|---
Fine-tuning methods
Llama2 7B |  |
(Gao et al., 2023) | 66.7 | 63.9
Llama2 13B |  |
(Gao et al., 2023) | 67.0 | 62.7
Prompting methods
DAIL-SQL + GPT4 |  |
(Gao et al., 2023) | 84.4 | 74.4
DIN-SQL + GPT4 |  |
(Pourreza and Rafiei, 2023) | 74.2 | 60.1
Table 1: Perfromance comparison of the prompting methods and fineutning method
on Spider validation dataset

We evaluate the performance of our proposed method using two Text-to-SQL
benchmarks: Spider (Yu et al., 2018) and Spider-SYN (Gan et al., 2021) and two
7B LLMs: DeepSeek DeepSeek-AI (2024) and Mistral Jiang et al. (2023). Our
approach demonstrates a performance improvement of approximately 3 to 7
percent in execution accuracy compared to the conventional single-step fine-
tuning method employed in previous studies (Gao et al., 2023). This consistent
performance gain across both datasets highlights the generalizability of our
method. Moreover, our fine-tuning strategy, utilizing a 7 billion parameter
LLM, surpasses all previous open-source methods on the Spider development set
and achieves comparable results to the state-of-the-art open-source methods
using GPT-4 (Pourreza and Rafiei, 2023; Gao et al., 2023) on the Spider test
set. We have provided all the necessary code to replicate the results, along
with the models’ predicted SQL queries, in our GitHub repository
111https://anonymous.4open.science/r/DTS-SQL-2A42.

##  2 Methodology

A notable development in LLMs is their post-pretraining refinement, which
enhances their alignment with preferred behaviors, as documented by Mishra et
al. (2021); Victor et al. (2022); Thoppilan et al. (2022). Common methods of
alignment include Supervised Fine-Tuning (SFT) using human demonstrations, as
reported by Ouyang et al. (2022); Tunstall et al. (2023) and Reinforcement
Learning from Human Feedback (RLHF), as detailed by Christiano et al. (2017);
Ziegler et al. (2019); Stiennon et al. (2020); Bai et al. (2022).

The absence of extensive datasets containing either human or AI feedback (Lee
et al., 2023) has led to a predominant focus on supervised fine-tuning in the
text-to-SQL field. This approach necessitates a collection of specific
instructions or prompts along with their corresponding outputs or responses.
In the following section, we will delve into the established methods of
supervised fine-tuning for LLMs within the Text-to-SQL context. Subsequently,
we introduce our novel two-step fine-tuning approach, designed to enhance the
performance of models in the Text-to-SQL domain.

###  2.1 Supervised fine-tuning for Text-to-SQL

In this section, we explore the supervised fine-tuning process for Text-to-SQL
tasks, as practiced in the open-source community (Gao et al., 2023). Given a
set of databases $D_{i}$ comprising pairs of questions $q_{i}$ and
corresponding SQL queries $s_{i}$, the goal is to fine-tune a large language
model $M$ using a set of training data $T=\\{(q_{i},s_{i},D_{i})\\}$, where
$q_{i}$ and $s_{i}$ represent the natural language question and its associated
SQL query on database $D_{i}$. The objective of supervised fine-tuning is to
minimize the empirical loss defined as:

$\min_{\sigma,M^{*}}\frac{1}{|T|}\sum_{i=1}^{|T|}\mathcal{L}(M^{*}(\sigma_{f}(q_{i},D_{i},s_{i})),$
(1)

where $\mathcal{L}$ is the loss function used to measure the difference
between the SQL queries generated by the model and the actual, correct (ground
truth) queries. The function $\sigma_{f}$ determines the formatting of the
question, the database schema, and the SQL queries. A key challenge during
inference is that we do not know in advance among all of the tables inside the
database which tables are relevant to a given question for generating accurate
SQL queries. Therefore, a common approach in fine-tuning involves including
the schema of all tables within the prompts together with the question and SQL
pairs. This method serves a dual purpose: teaching the model to generate the
correct SQL query and to identify the relevant tables from among all the
provided tables. This approach of training for two objectives simultaneously
complicates the SQL generation task for LLMs, particularly for smaller models
with only a few billion parameters. Each task – generating SQL queries and
correctly linking to the relevant schema – demands its own reasoning process.
A significant proportion of errors in large language models can be attributed
to incorrect schema linking, highlighting this as a major challenge in the
field (Pourreza and Rafiei, 2023; Dong et al., 2023).

###  2.2 Decomposed Supervised Fine-tuning

We propose a two-stage fine-tuning process, which separates schema linking and
SQL generation, aiming to enhance the performance of NL-to-SQL systems.

####  2.2.1 Schema-linking Fine-tuning

Schema linking involves identifying the pertinent columns and tables in a
database in response to natural language queries. It has been demonstrated to
enhance cross-domain generalizability and facilitate the creation of intricate
queries (Lei et al., 2020). In prior studies, schema linking has primarily
been accomplished through in-context learning methods or implicitly during the
fine-tuning process for SQL generation (Pourreza and Rafiei, 2023; Cao et al.,
2021; Guo et al., 2019; Xu et al., 2021). In this work, we treat schema
linking as a distinct task and explicitly fine-tune LLMs to identify relevant
tables and columns when presented with a natural language query. Given the
training dataset $T=\\{(q_{i},s_{i},D_{i})\\}$, we extract all of the columns
and tables used in the SQL queries and create a new dataset of
$T=\\{(q_{i},T_{i},C_{i},D_{i})\\}$ where $T_{i}$ and $C_{i}$ represent lists
of tables and columns used in the SQL query $s_{i}$. The primary objective
during supervised fine-tuning for schema linking is to minimize the empirical
loss, as defined by the following equation:

$\min_{\sigma,M^{*}}\frac{1}{|T|}\sum_{i=1}^{|T|}\mathcal{L}(M^{*}(\sigma_{s}(q_{i},T_{i},C_{i},D_{i})),$
(2)

Here, $\mathcal{L}$ represents the loss related to the model’s next token
prediction, comparing the predicted column and table names with the actual
ground truth names.

####  2.2.2 SQL Generation Fine-tuning

After identifying the appropriate tables for SQL generation, the next step is
to utilize a model that constructs the SQL query based on the question and the
schema of the correct tables. Since we have already identified the potentially
correct tables using the schema-linking module, there is no need to include
all tables in the input for the SQL generation model. In contrast to previous
approaches for fine-tuning LLMs, we extract the relevant tables from the
training dataset $T=\\{(q_{i},s_{i},D_{i})\\}$ corresponding to the ground
truth SQL queries. We then fine-tune the LLM while minimizing the following
loss function:

$\min_{\sigma,M^{*}}\frac{1}{|T|}\sum_{i=1}^{|T|}\mathcal{L}(M^{*}(\sigma_{g}(q_{i},T_{i},s_{i})),$
(3)

The loss function is same as the loss function defined in Section 2.1. This
decomposition of the Text-to-SQL training process allows LLMs to be trained
with a singular objective. By segregating the schema-linking and SQL query
generation tasks, we improve the training process, enabling more focused and
effective fine-tuning.

##  3 Experiments

###  3.1 Models

Our methodology’s performance was assessed using two recent LLMs from distinct
architectures. These models are Mistral 7B (Jiang et al., 2023) and DeepSeek
7B (DeepSeek-AI, 2024). The DeepSeek model, sharing similar architecture with
the LLama model family (Touvron et al., 2023), has been pretrained on an
extensive dataset comprising 2 trillion tokens and supports a sequence length
of 4096. Mistral 7B, although not specifically pretrained for code generation,
surpasses many counterparts in its scale category (Jiang et al., 2023).

###  3.2 Hyperparameters

The two LLMs were trained on Nvidia Tesla A100 GPUs, employing a batch sizes
of 64 and 32 with a learning rate of 1*e-5 and 5*e-5 respectively. To enhance
the training efficiency, we incorporated Flash Attention techniques as
detailed in (Dao et al., 2022; Dao, 2023).

###  3.3 Datasets

We conducted our evaluation using cross-domain, challenging Text-to-SQL
datasets. Spider, was introduced by Yu et al. (2018) and includes 200 database
schemas. Of these, 160 schemas are allocated for training and development,
while the remaining 40 are set aside for testing purposes. Our second dataset
was Spider-Syn (Gan et al., 2021), which modifies the original Spider dataset
by replacing schema-related words with synonyms and removing explicit mentions
that link natural language queries (NLQs) to the database schema.

###  3.4 Metrics

In our evaluation of text-to-SQL models, we utilized exact set match accuracy
and execution accuracy. The former involves comparing the components of SQL
queries, such as select, where, having, group by, and order by clauses,
focusing on the matching of columns and predicates without considering the
order. The latter determines equivalence between a model-generated query and a
reference query if they produce identical results across various database
instances.

###  3.5 Results

####  3.5.1 Spider test set

As depicted in Table 2, our method employing DeepSeek 7B, when tested on the
Spider test dataset, achieves results comparable to state-of-the-art open-
source methods in terms of execution accuracy and exact set match accuracy.

Model | EX | EM
---|---|---
DAIL-SQL + GPT-4 |  |
(Gao et al., 2023) | 86.6 | -
DIN-SQL + GPT-4 |  |
(Pourreza and Rafiei, 2023) | 85.3 | 60
DTS-SQL + DeepSeek 7B |  |
Ours | 84.4 | 73.7
C3 + ChatGPT + Zero-Shot |  |
(Dong et al., 2023) | 82.3 | -
RESDSQL-3B + NatSQL |  |
(Li et al., 2023a) | 79.9 | 72
DIN-SQL + CodeX |  |
(Pourreza and Rafiei, 2023) | 78.2 | 57
DTS-SQL + Mistral |  |
Ours | 77.1 | 69.3
Graphix-3B + PICARD |  |
(Li et al., 2023b) | - | 74
Table 2: The comparison of different methods on test set of Spider.

####  3.5.2 Spider dev set

In Table 3, we showcase the results of our two-stage fine-tuning method on the
dev set of Spider. The performance is compared against two distinct scenarios:
firstly, a one-stage scenario where the model is fine-tuned on all tables
without employing our two-stage approach, and secondly, a perfect schema
linking scenario where we provide the ground truth tables to our fine-tuned
SQL generators. This latter scenario is denoted as the ’Upper Bound’ in the
table. Our two-stage model’s performance is measured by initially using our
fine-tuned schema linker model to identify potentially relevant tables, which
are then provided as context to the SQL generator model.

In Table 4, we offer a detailed comparison between our method and various
other baseline approaches. For the baselines, we selected diverse methods from
different families of approaches that are using LLMs and are available as open
source. Our two-stage decomposed approach with DeepSeek 7B attained state-of-
the-art performance on the Spider development set, surpassing all previous
methods that utilized prompting techniques and fine-tuning. Additionally, the
results of our two-stage method on Spider-SYN dataset is provided in the
appendix A section.

Model | Tuning | EX | EM
---|---|---|---
Mistral 7B | FT Tuning | 71.9 | 70.9
Mistral 7B | DTS-SQL | 78.6 | 73.3
Mistral 7B | Upper bound | 86.6 | 80.7
DeepSeek 7B | FT Tuning | 82.1 | 69.0
DeepSeek 7B | DTS-SQL | 85.5 | 79.1
DeepSeek 7B | Upper bound | 90.3 | 84.2
Table 3: Performance of the LLMs with different tuning methods on Spider dev set. FT stands for Full tables finetuning, Upper bound performance is the performance which we can achieve with a perfect schema linking. Model | EX | EM
---|---|---
Instruction tuning methods
DTS-SQL + Mistral 7B |  |
(our) | 78.6 | 73.3
DTS-SQL + DeepSeek 7B |  |
(our) | 85.5 | 79.1
Llama2 7B |  |
(Gao et al., 2023) | 66.7 | 63.9
Llama2 13B |  |
(Gao et al., 2023) | 67.0 | 62.7
Prompting methods
DIN-SQL + GPT4 |  |
(Pourreza and Rafiei, 2023) | 74.2 | 60.1
DIN-SQL + CodeX |  |
(Pourreza and Rafiei, 2023) | 69.9 | 57.2
DAIL-SQL + GPT4 |  |
(Gao et al., 2023) | 84.4 | 74.4
C3 + GPT-3.5 |  |
(Dong et al., 2023) | 81.8 | -
Table 4: Performance of different methods with LLMs on the dev set of Spider.

####  3.5.3 Schema-linking Performance

As discussed in Section 2, our approach employs two LLMs: one for schema
linking and another for SQL query generation. The schema-linking model plays a
pivotal role in our pipeline, as inaccuracies in table detection could hinder
the SQL generator’s ability to formulate the correct SQL queries. We fine-
tuned two models, based on the Deepseek and Mistral models, for schema
linking. Evaluation metrics, including exact set match, precision, and recall,
were used to assess their performance. Detailed information about these models
on two distinct datasets can be found in Table 5.

Model | Dataset | EX | PR | RE
---|---|---|---|---
DeepSeek | Spider | 93.1 | 98.4 | 97.7
Mistral | Spider | 91.1 | 97.5 | 97.8
DeepSeek | Spider-SYN | 87.6 | 94.6 | 94.7
Mistral | Spider-SYN | 85.3 | 91.2 | 90.5
Table 5: Performance of the schema-linker model on Spider and Spider-SYN dev
sets. PR stands for Precision, RE is recall, and EX is exact set match
accuracy.

##  4 Discussion

While our two-step approach has achieved state-of-the-art results on the
development set of Spider and demonstrated comparable performance to larger
models like GPT-4 on the test set, there is still significant room for
improvement, particularly for the schema-linking models. Currently, our
schema-linking models achieve roughly 90% exact set match accuracy. However,
as noted in Table 3, the substantial gap between the upper bound performance
of the SQL generator and that of DTS-SQL calls for further research into the
schema-linking. .

##  5 Conclusion

Before our research, small open-source models lagged behind large proprietary
models in performance on the text-to-SQL task. Our two-stage fine-tuning
approach breaks down the task into two simpler components, enabling small
open-source models to rival larger ones. Subsequent efforts could focus on
enhancing the performance of these stages and exploring improved methods for
transferring the output of one stage to the next.

## Limitations

This paper has placed its primary emphasis on enhancing the performance of
both small large language models for Text-to-SQL task. However, there remains
scope for further investigation and comparison of various techniques for
schema-linking. Exploring approaches like retrieval methods or in-context
learning when applied in conjunction with larger models such as GPT-4 for the
schema-linking task could yield valuable insights into identifying the most
effective methodologies for schema-linking.

## Ethics Statement

In this paper, we place a strong emphasis on the significance of ethical
considerations in every aspect of our research, from its inception to its
presentation. We wholeheartedly commit to adhering to the ACL Ethics Policy
and upholding ethical principles and guidelines throughout our research
journey.

We have taken proactive measures to minimize any potential biases or
discriminatory elements in our research design, data selection, and
interpretation of results. Our dedication to transparency, precision, and
fairness in reporting our findings is unwavering, and we have duly
acknowledged and cited the work of others to give proper credit.

By incorporating this ethics statement, we aim to underscore our unwavering
commitment to conducting research with integrity, respecting ethical
principles, and contributing responsibly to the advancement of knowledge in
our field.

## Acknowledgements

## References

  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_.
  * Cao et al. (2021) Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. 2021.  Lgesql: line graph enhanced text-to-sql model with mixed local and non-local relations.  _arXiv preprint arXiv:2106.01093_.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30.
  * Dao (2023) Tri Dao. 2023.  FlashAttention-2: Faster attention with better parallelism and work partitioning.  _arXiv preprint arXiv:2307.08691_.
  * Dao et al. (2022) Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022.  FlashAttention: Fast and memory-efficient exact attention with IO-awareness.  In _Advances in Neural Information Processing Systems_.
  * DeepSeek-AI (2024) DeepSeek-AI. 2024.  Deepseek llm: Scaling open-source language models with longtermism.  _arXiv preprint arXiv:2401.02954_.
  * Dong et al. (2023) Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, Jinshu Lin, Dongfang Lou, et al. 2023.  C3: Zero-shot text-to-sql with chatgpt.  _arXiv preprint arXiv:2307.07306_.
  * Gan et al. (2021) Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, and Pengsheng Huang. 2021.  Towards robustness of text-to-SQL models against synonym substitution.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2505–2515, Online. Association for Computational Linguistics.
  * Gao et al. (2023) Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023.  Text-to-sql empowered by large language models: A benchmark evaluation.  _arXiv preprint arXiv:2308.15363_.
  * Guo et al. (2019) Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019.  Towards complex text-to-sql in cross-domain database with intermediate representation.  _arXiv preprint arXiv:1905.08205_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023.  Rlaif: Scaling reinforcement learning from human feedback with ai feedback.  _arXiv preprint arXiv:2309.00267_.
  * Lei et al. (2020) Wenqiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua. 2020.  Re-examining the role of schema linking in text-to-sql.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6943–6954.
  * Li et al. (2023a) Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023a.  Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 13067–13075.
  * Li et al. (2023b) Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. 2023b.  Graphix-t5: Mixing pre-trained transformers with graph-aware layers for text-to-sql parsing.  _arXiv preprint arXiv:2301.07507_.
  * Li et al. (2023c) Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, et al. 2023c.  Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls.  _arXiv preprint arXiv:2305.03111_.
  * Mishra et al. (2021) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021.  Cross-task generalization via natural language crowdsourcing instructions.  _arXiv preprint arXiv:2104.08773_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Pourreza and Rafiei (2023) Mohammadreza Pourreza and Davood Rafiei. 2023.  Din-sql: Decomposed in-context learning of text-to-sql with self-correction.  _arXiv preprint arXiv:2304.11015_.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023.  Zephyr: Direct distillation of lm alignment.  _arXiv preprint arXiv:2310.16944_.
  * Victor et al. (2022) Sanh Victor, Webson Albert, Raffel Colin, Bach Stephen, Sutawika Lintang, Alyafeai Zaid, Chaffin Antoine, Stiegler Arnaud, Raja Arun, Dey Manan, et al. 2022.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_.
  * Xu et al. (2021) Kuan Xu, Yongbo Wang, Yongliang Wang, Zujie Wen, and Yang Dong. 2021.  Sead: End-to-end text-to-sql generation with schema-aware denoising.  _arXiv preprint arXiv:2105.07911_.
  * Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018.  Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task.  _arXiv preprint arXiv:1809.08887_.
  * Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_.

##  Appendix A Appendix

###  A.1 Spider-SYN dataset

To assess the efficacy of our proposed method, we evaluated its performance on
the development set of Spider-SYN. Although Spider-SYN possesses a distinct
training set, we opted to test our fine-tuned models directly on its
development set, without any additional tuning on the Spider-SYN training set.
The same performance gain is observed on this dataset (see Table 6) even
though the model was not directly trai ned in this dataset.

Model | Tuning | EX | EM
---|---|---|---
Mistral 7B | FT Tuning | 67.0 | 63.9
Mistral 7B | DTS-SQL | 71.1 | 64.6
Mistral 7B | Upper bound | 81.9 | 74.5
DeepSeek 7B | FT Tuning | 70.4 | 56.6
DeepSeek 7B | DTS-SQL | 76.2 | 68.9
DeepSeek 7B | Upper bound | 85.5 | 78.1
Table 6: Performance of the LLMs with different tuning methods on Spider-SYN
dev set. FT stands for Full tables finetuning, Upper bound performance is the
performance which we can achieve with a perfect schema linking.

###  A.2 Prompt

In conducting all our experiments on both models, we adhered to a standardized
prompt format to ensure consistency and facilitate reliable comparisons. The
chosen prompt format is well-established as effective in the Text-to-SQL
domain, as demonstrated in prior research by Gao et al. (2023). In this
format, we provided information about the foreign key constraints, primary
keys, and column types. Furthermore, to guide the models in understanding how
data is stored within the database, our prompt incorporated three sample rows,
showcasing data entries.

The specific prompt used for our experiments is as follows:

Figure 1: The prompt used for SQL generation. The database schema is where we
put the tables representations. Figure 2: The prompt used for Schema linking.
The database schema is where we put the tables representations. Figure 3: A
sample table representation. All of the table in a database are represented as
above and used in the prompts.