# Improving Generalization in Semantic Parsing
by Increasing Natural Language Variation

Irina Saparina  Mirella Lapata
Institute for Language, Cognition and Computation
School of Informatics, University of Edinburgh
10 Crichton Street, Edinburgh EH8 9AB
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Text-to-SQL semantic parsing has made significant progress in recent years,
with various models demonstrating impressive performance on the challenging
Spider benchmark. However, it has also been shown that these models often
struggle to generalize even when faced with small perturbations of previously
(accurately) parsed expressions. This is mainly due to the linguistic form of
questions in Spider which are overly specific, unnatural, and display limited
variation. In this work, we use data augmentation to enhance the robustness of
text-to-SQL parsers against natural language variations. Existing approaches
generate question reformulations either via models trained on Spider or only
introduce local changes. In contrast, we leverage the capabilities of large
language models to generate more realistic and diverse questions. Using only a
few prompts, we achieve a two-fold increase in the number of questions in
Spider. Training on this augmented dataset yields substantial improvements on
a range of evaluation sets, including robustness benchmarks and out-of-domain
data.111Model checkpoints and data are available at
github.com/saparina/Text2SQL-NLVariation

Improving Generalization in Semantic Parsing
by Increasing Natural Language Variation

Irina Saparina and Mirella Lapata Institute for Language, Cognition and
Computation School of Informatics, University of Edinburgh 10 Crichton Street,
Edinburgh EH8 9AB<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Semantic parsing is the task of mapping natural language utterances to
machine-interpretable expressions such as SQL queries or logical forms. It has
emerged as an important component in many natural language interfaces (Őzcan
et al., 2020) with applications in robotics (Dukes, 2014), question answering
(Zhong et al., 2017; Yu et al., 2018), dialogue systems Artzi and Zettlemoyer
(2011), and the Internet of Things (Campagna et al., 2017).

The release of the Spider dataset (Yu et al., 2018) marked an important
milestone in text-to-SQL semantic parsing. Apart from its considerable size,
Spider stands out for including complex and nested queries, and databases from
various domains. Importantly, it exemplifies a cross-domain generalization
setting, i.e., models trained on Spider are expected to parse natural language
questions for any given database, even in previously unseen domains. In
practice, models trained on Spider degrade significantly when tested on
different databases from _other_ datasets, for example, on real-world data
from Kaggle and Stack Exchange websites (Suhr et al., 2020; Lee et al., 2021;
Hazoom et al., 2021).

The linguistic composition of questions in Spider contributes to this
performance gap. Unlike real-world applications where user questions may be
concise, ambiguous, and necessitate commonsense reasoning or domain-specific
knowledge, questions in Spider are often overly explicit, directly mentioning
database entities even when such information is unnecessary for inferring the
underlying intent. An example is shown in Figure 1, the first question
includes redundant details (e.g., customer, first name, last name) which serve
as references to databases entities. Omitting these details would not change
the meaning of the question but rather make it more colloquial. Due to the
limited diversity of questions, Spider falls short in providing enough
examples for learning essential skills such as grounding and reasoning. As a
result, models tend to overfit to Spider-style questions, and even minor
perturbations in how questions are phrased lead to considerable performance
decrease, sometimes up to 22% (Gan et al., 2021b; Deng et al., 2021; Pi et
al., 2022; Chang et al., 2023).

Efforts to automatically increase its diversity often rely on text generation
models trained on the same Spider data and unavoidably inherit its
characteristics Zhong et al. (2020b); Wang et al. (2021); Wu et al. (2021);
Jiang et al. (2022). In this work, we propose to augment the training data for
text-to-SQL parsers with more realistic and diverse question reformulations.
We leverage the capabilities of large language models for rewriting utterances
and devise prompts designed to enhance model robustness against linguistic
variations. Our prompts consist solely of instructions and questions and are
easy to use. We train three state-of-the-art text-to-SQL parsers on Spider Yu
et al. (2018) with augmentations generated by our approach. Extensive
experiments show that a two-fold increase in the number of questions
substantially improves model generalization ability. Our augmentations
increase _robustness_ against question perturbations when models are evaluated
on the challenging Dr.Spider sets Chang et al. (2023) and deliver improvements
in a _zero-shot_ setting, when models are tested on out-of-domain datasets
like GeoQuery Zelle and Mooney (1996) and KaggleDBQA Lee et al. (2021).

Our contributions are three-fold: a proposal of rewrite operations to render
questions more diverse and natural; a methodology for augmenting existing
datasets based on the proposed reformulations; and empirical results
validating our approach improves generalization across models and datasets.

##  2 Related Work

#### Out-of-domain Generalization

Several datasets have been released to facilitate the development of models
with generalization capabilities. WikiSQL Zhong et al. (2017) is a large-scale
benchmark with different databases but only one table. As a result, WikiSQL
queries are relatively easy to parse due to the use of a limited set of
operations. Spider (Yu et al., 2018), contains multiple tables per database
which result in complex SQL queries.

Suhr et al. (2020) examine the performance of Spider-trained models on
datasets varying in terms of the questions being asked, the database
structure, and SQL style. They discover that a key challenge in achieving
generalization lies in linguistic variation, and propose augmenting Spider’s
training set with WikiSQL data. Our work addresses the problem of question
diversity in Spider, without compromising its complex query structures or
multi-table database nature. We evaluate our approach on GeoQuery (Zelle and
Mooney, 1996), a dataset similar to Spider in terms of database structure and
SQL queries but different in the style of questions. We also report results on
KaggleDBQA Lee et al. (2021), a dataset with real-world databases and
questions created by users with access to field descriptions rather than
database schemas.

Bird Li et al. (2023b) is a recently released a text-to-SQL benchmark, aiming
to highlight real-world challenges with large-scale databases which often
contain dirty and noisy values and how to express SQL queries to improve
execution speed. They show that incorporating manually annotated external
knowledge that includes synonyms improves performance. Our augmentations can
be viewed as an alternative to this approach; we _learn_ language variations
_without_ oracle knowledge.

#### Robustness to Perturbations

Another challenge for text-to-SQL parsers is robustness to small
perturbations. Previous studies evaluate robustness in the single-domain
setting (Huang et al., 2021) and across databases, e.g., by removing or
paraphrasing explicit mentions of database entities (Spider-Realistic; Deng et
al. 2021) or by substituting such mentions with synonyms (Spider-Syn; Gan et
al. 2021a). Other work explores the effect of perturbations in the database
schema Pi et al. (2022) and also in questions Ma and Wang (2021). Recently,
Chang et al. (2023) released Dr.Spider, a comprehensive robustness benchmark
with a wide range of perturbations in the database schema, questions, and SQL
semantics. We evaluate our approach on their “question sets” which cover a
broader range of language variations compared to previous efforts.

#### Data Augmentation

Several data augmentation and adversarial training techniques have been
proposed to support SQL queries executed on a single table Li et al. (2019);
Radhakrishnan et al. (2020) and multiple tables Zhong et al. (2020b); Wang et
al. (2021); Wu et al. (2021); Deng et al. (2021); Wu et al. (2021); Jiang et
al. (2022). Augmentations in earlier work Gan et al. (2021a); Deng et al.
(2021); Ma and Wang (2021); Huang et al. (2021) target specific linguistic
expressions like synonyms or paraphrases. We leverage the capabilities of
(very) large languages models (LLMs; Brown et al. 2020; Chowdhery et al. 2022)
to generate linguistically diverse natural language questions. Recent efforts
Dai et al. (2023); He et al. (2023) have shown that LLMs can serve as
annotators when given sufficient guidance and examples mainly for text
classification tasks.

##  3 Motivation

###  3.1 Problem Formulation

Semantic parsing aims to translate a natural language utterance into a formal
representation of its meaning. We focus on meaning representations in the form
of SQL queries that can be executed in some database to retrieve an answer or
denotation. In the cross-domain setting, the parser is not limited to a
specific database and can be in theory applied to arbitrary databases and
questions. In practice, this task is more or less complex depending on the
database in hand, i.e., the number of tables and values, the naming
conventions used for tables and columns, the way values are formatted, and
specific domain characteristics. We do not consider these challenges in this
work, focusing instead on generalization issues that arise from the variation
of questions in natural language.

###  3.2 Types of Utterances in Semantic Parsing

Recent work has demonstrated the importance of wording in semantic parsing,
indicating that certain question formulations can be more difficult to parse
than others (Radhakrishnan et al., 2020; Gan et al., 2021a; Deng et al., 2021;
Chang et al., 2023).

The level of difficulty for a question can be influenced by the amount of
task-specific background knowledge used to formulate it. For instance, users
familiar with SQL and the underlying database will have some idea of the
desired program, and will be able to articulate their intentions more
precisely, e.g., by providing explicit instructions. In contrast, users
unfamiliar with the task are more likely to ask general questions in a
colloquial style. Figure 1 illustrates different question formulations with
the same intent. The first question could have been posed by a user who is
well-versed in SQL and has knowledge of the database; it mentions specific
database entities and operations like summation and filtering, unlike the
second question which does not have any such details. More formally, we
distinguish between two types of utterances:

#### Utterances which demonstrate prior knowledge

are closely aligned with the desired programs, highlight logical structure
operations, and explicit references to database entities. Such utterances
resemble instructions, suggesting the user has some understanding of the
desired program. In Figure 1, the first question falls under this category,
presupposing knowledge of summation and filtering operations and the names of
entities (e.g., first_name, last_name) used in the target SQL query.

#### Utterances which do not demonstrate prior knowledge

are general descriptions of intent, expressed in a simple, colloquial
language. They do not provide intentional hints about the desired program, but
are often ambiguous, requiring additional reasoning based on domain or common
sense knowledge. In the examples shown in Figure 1, the second question
belongs to this category, it is laconic, underspecified, and inherently
natural.

Database: driving_school
---
Customers
customer_id | $\dots$ | first_name | last_name | $\dots$ | email_address
Lessons
lesson_id | $\dots$ | customer_id | lesson_time | $\dots$ | price

|  |  | Prior
---|---|---|---
Questions |  | SQL | DB
1. |  Calculate the total sum of lesson times filtering the results by selecting the customer with the first name "Rylan" and the last name "Goodwin". |  | ✓ | ✓
2. |  How long did Rylan Goodwin’s lesson last? |  | X | X
3. |  How long is the total lesson time taken by a customer with a first name as Rylan and a last name as Goodwin? |  | X | ✓

SQL Query
---
SELECT sum(T1.lesson_time) FROM Lessons AS T1 JOIN Customers AS T2 ON
T1.customer_id = T2.customer_id WHERE T2.first_name = "Rylan" AND T2.last_name
= "Goodwin".

Figure 1: Different types of questions that are related to the same database
(only relevant tables and columns are shown) and map to the same SQL query.

These types of utterances represent two important edge cases but do not cover
all possibilities. In the context of text-to-SQL semantic parsing, information
about the database schema and its contents can also be useful when formulating
questions. We thus introduce a third category that falls between having task-
specific knowledge and none at all.

#### Utterances which demonstrate knowledge of the database schema

are general descriptions of intent but with explicit references to related
database entities. This category differs from the previous two in the type of
prior knowledge used; users are familiar with the database schema and possibly
database content but have no expertise in query construction. The third
question in Figure 1 includes explicit references to the database table (e.g.,
customers) and its columns (e.g., lesson_time, first_name, last_name). Because
of that, questions may be less coherent and natural. In our example, the
question contains redundant details such as first name, last name, and
customer.

Questions in Spider (Yu et al., 2018) often include explicit mentions of
database elements (Deng et al., 2021). This is a by-product of Spider’s
creation process which encouraged annotators familiar with SQL to formulate
the questions more clearly and explicitly. In contrast, other datasets like
GeoQuery (Zelle and Mooney, 1996) or cross-domain KaggleDBQA (Lee et al.,
2021) contain less explicit questions with a smaller percentage of database
entity mentions. In this work, we automatically augment Spider’s training set
with more general and natural questions aiming to develop semantic parsing
models that can effectively handle all types of utterances mentioned above.

##  4 Data Generation

We augment the training set of Spider (Yu et al., 2018) by leveraging large
language models. Specifically, we exploit ChatGPT’s222chat.openai.com text
generation capabilities (gpt-3.5-turbo-0301) and ask it to rephrase Spider
questions (no SQL- or database-specific information is provided; see Table 1),
using three types of rewrite operations:

  1. 1.

Deletion of words or phrases which are redundant for understanding the
question’s intent. For this purpose, we use two instructions: the first one
simplifies the question, while the second one explicitly hides unnecessary
details that do not change the meaning. The first instruction affords ChatGPT
more freedom in rewriting the question. In Table 1, examples 1–2 show how
Spider questions are reformulated with these instructions.

  2. 2.

Substitution of words or phrases with simpler ones. We instruct ChatGPT to
replace words with their _synonyms_ and also to more generally attempt to
simplify by substituting a few words in the question. In Table 1, examples 3–4
show how questions are rewritten with these instructions.

  3. 3.

Rewriting of the entire question. Some questions can have the same meaning,
despite being significantly dissimilar in their surface realisation. For
example, the questions Where do most people live? and Which cities have the
largest population? are related to the same database about cities and express
the same intent but have no words in common. We instruct ChatGPT to provide
different ways of expressing a question. We empirically find that ChatGPT can
be too conservative at times and also include a prompt with examples to
encourage more drastic reformulations. In Table 1, questions 5–6 show example
outputs for these instructions.

1\. Instruction: |  Simplify
---|---
Original Question |  Which club has the most female students as their members? Give me the name of the club.
Output |  Which club has the most female students?
2\. Instruction: |  Simplify by hiding details
Original Question |  What is the title and credits of the course that is taught in the largest classroom (with the highest capacity)?
Output |  What course is taught in the biggest classroom and what are its credits?
3\. Instruction: |  Simplify using synonyms
Original Question |  What is the average duration in milliseconds of tracks that belong to Latin or Pop genre?
Output |  What is the mean length in milliseconds of Latin or Pop songs?
4\. Instruction: |  Simplify using substitutions
Original Question |  What are the locations that have gas stations owned by a company with a market value greater than 100?
Output |  Where are the gas stations owned by a company worth more than 100?
5\. Instruction: |  Express in a different way
Original Question |  What is the number of routes operated by the airline American Airlines whose destinations are in Italy?
Output |  How many routes does American Airlines have that fly to Italy?
6\. Instruction: |  Examples of the question simplification: <$\dots$>
Original Question |  What are the names of body builders whose total score is higher than 300?
Output |  Who are the body builders with a score over 300?
7\. Instruction: |  Paraphrase
Original Question |  Return the categories of music festivals that have the result "Awarded".
Output |  List the categories of music festivals that have been recognized with awards.
Table 1:  Different types of augmentation generated for Spider questions. The
full version of the instructions for generation is shown in Appendix A, Table
6.

We also ask ChatGPT to paraphrase questions (see example 7 in Table 1). This
instruction may be viewed as a generalization of previous reformulations,
however, in practice it is only somewhat helpful. ChatGPT often generates very
similar versions of the original question, retaining the same details, style
and structure following this instruction. We consider this conservative
paraphrasing strategy to be an advantage as almost all machine-generated
questions preserve the meaning of the original question. To verify the quality
of generated paraphrases, we compute the cosine similarity between the
original and generated questions.

##  5 Experimental Setup

Our experiments aim to evaluate the performance of models trained specifically
for cross-database text-to-SQL parsing. We are interested in two types of
generalization: robustness to controllable perturbations in utterances and
adaptation to new domains with different question styles. Perturbations allow
us to study more closely the impact of language variations, while new domains
provide a more realistic and challenging setting. We first describe the
datasets we use for training and evaluation and then briefly discuss the
semantic parsing models and metrics we employ in our experiments.

Augmentation Type | # examples
---|---
Simplify | 774
Simplify by hiding details | 1,136
Simplify using synonyms | 1,285
Simplify using substitutions | 1,316
Paraphrase | 1,130
Express in a different way | 1,065
Prompt with examples | 1,256
Total | 7,962
Table 2: Question reformulations generated for Spider; number of generations
per instruction.

###  5.1 Training Datasets

Our primary training dataset is Spider (Yu et al., 2018), which contains 7,000
questions to 140 different databases and 3,981 target queries.333We exclude
the single-domain datasets Yu et al. (2018) employ in addition to their data.
Although there can be more than one question for the same intent (usually
two), linguistic variations tend to be scanty and limited. We augment Spider
with additional questions using ChatGPT as an automatic annotator. For each
intent in the original training set, we generate two question reformulations
based on the types specified in Section 4. We choose the augmentation types
randomly and do not accept duplicates. Figure 2 shows the distribution of
cosine similarities between the original Spider questions and the generated
reformulations. We measure cosine similarity based on the SimCSE embeddings of
Gao et al. (2021). As can be seen, the majority of paraphrases are
semantically similar to the Spider question (the mean similarity is 0.88).
Experiments with different filtering thresholds (ranging from 0.5 to 0.7 with
a step of 0.05) revealed that storing all generated examples, effectively
adopting a threshold of 0.5, obtained best results. Additionally, we manually
inspected 100 reformulations and found only 6% to be incorrect (i.e.,
inaccurate expressions of intent). Analysis in Appendix B further shows that
our augmentations do not affect the nature of parsing errors.

The resulting training set contains 14,954 instances; statistics for each
category are in Table 2 and examples in Appendix E. The cost of calling the
ChatGPT API to obtain our augmentations is approximately 7.5$.

Figure 2: Distribution of cosine similarities between Spider questions and
generated reformulations.

###  5.2 Evaluation Datasets

The Spider development set consists of 1,034 questions to 20 databases and 564
target SQL Training Datasets. Since these questions share the same style and
level of detail as the training set, we instead focus on evaluation sets with
more natural and diverse language. Specifically, we present results on two
groups of evaluation sets. The first group are datasets derived from the
Spider development set, featuring identical SQL Training Datasets and
databases which allow us to assess the model’s resilience to variations in
linguistic expression. The second group are independent datasets which not
only differ in language usage but also in SQL style and database specifics.
This allows us to evaluate model performance in more realistic conditions.

#### Datasets Based on Spider

Chang et al. (2023) have recently released Dr.Spider, a comprehensive
robustness benchmark which includes 9 evaluation sets with 7,593 examples of
perturbations in natural language questions (NLQ sets). They have also created
evaluation sets for database and SQL perturbations which are out of scope for
this work. NLQ perturbation sets are based on the Spider development set, they
contain the same databases and gold Training Datasets, deviating only in terms
of the questions asked. They are generated with OPT (Zhang et al., 2022), a
large pretrained language model, and manually filtered by SQL experts. There
are three main categories of perturbations: change one or a few words that
refer to SQL keywords (for example, replace the word maximum referring to the
max SQL function with the largest), change references to columns (for example,
replace name of the countries referring to column CountryName with which
countries) and change references to database values (for example, replace
players from the USA referring to the value USA with American players).
Changes are made by replacing words with their synonyms or carrier phrases
(e.g., name of the countries and which countries). Note that our augmentations
target solely language variations and do not manipulate gold SQL queries.

#### Other Datasets

GeoQuery (Zelle and Mooney, 1996) is a single-domain semantic parsing dataset
with questions to a database of US geography. We use a version with SQL
queries as logical forms and query-based splits (Finegan-Dollak et al., 2018)
with a test set of 182 examples. GeoQuery questions are concise and their
interpretation often depends on domain knowledge. For example, in the question
what is the largest city in the smallest state in the usa, the largest city
implies the city with the largest population but the smallest state implies
the state with the smallest area.

KaggleDBQA (Lee et al., 2021) is a cross-domain text-to-SQL dataset for
testing models under more realistic conditions. It contains 272 examples
related to 8 real-world databases which can have abbreviated table and column
names and “dirty” values. Questions were collected with annotators having
access to column descriptions only, rather than the actual database schema
(the dataset provides these descriptions but we do not use them). This
simulates realistic database usage but also creates a challenge for semantic
parsers as questions cannot be easily aligned to target SQL queries. For
example, the question Which artist/group is most productive? to a database
with information on hip hop torrents should be parsed into query SELECT artist
FROM torrents GROUP BY artist ORDER BY count(groupName) DESC LIMIT 1, as
productive refers to the number of releases and column groupName contains
released titles.

###  5.3 Models

Current approaches frame text-to-SQL parsing as a sequence-to-sequence
problem. The input is the concatenation of question and database entities,
including table and column names, and content values extracted based on string
matching, and the output is an SQL query. Shaw et al. (2021) show that a pre-
trained T5-3B model (Raffel et al., 2020) fine-tuned on Spider Yu et al.
(2018) is a competitive text-to-SQL parser. Scholak et al. (2021) build on
this approach with PICARD, a method for constrained decoding that filters the
beam at each generation step, taking into account task-specific constraints
such as grammatical correctness and consistency with the database. Recently,
Li et al. (2023a) propose RESDSQL, an approach that decouples schema linking
from SQL parsing. They first filter relevant database entities and then use
T5-3B to generate a sketch (i.e., SQL keywords) and then the actual SQL query.
We use the best version of their model which also leverages NatSQL
intermediate representations Gan et al. (2021c).

We use the implementations from Scholak et al. (2021) and Li et al. (2023a)
for training models on augmented data and their released checkpoints for
training on the original Spider. All models are trained for 100 epochs; we use
a batch size of 200 for the base T5-3B to reduce the computational cost,
leaving all other hyperparameters unchanged. We train on a single NVIDIA A100
GPU.

Our approach to data augmentation is model agnostic but our experiments focus
on settings where the model is specifically trained or fine-tuned on text-to-
SQL data. An alternative is large language models which are trained on huge
text collections (including code) and able to translate natural language to
SQL, without further fine-tuning on task-specific data Rajkumar et al. (2022).
Since our augmentations are generated by ChatGPT, a model trained with
Reinforcement Learning for Human Feedback (Christiano et al., 2017), we
include it as a standalone baseline. Following Liu et al. (2023), we prompt
ChatGPT in a _zero-shot_ setting with the description of the database schema
followed by the question (the full prompt is shown in Appendix C). Large
language models like ChatGPT differ from task-specific models in many
respects, including potential use cases, resource requirements, transparency,
and accessibility and thus any comparison should be interpreted with a grain
of salt.

###  5.4 Metrics

We report performance as execution accuracy which compares the execution
results of gold and predicted queries (using the implementation of Zhong et
al. 2020a).

Firstly, we evaluate model _robustness to perturbations_ in questions, by
considering zero-shot parsing on Dr.Spider. Evaluation sets for Dr.Spider NLQ
fall in two categories: pre-perturbed sets are subsets of the Spider
development set, while post-perturbed sets are the same subsets but with
rewritten questions instead of the original Spider ones. Execution accuracy on
post-perturbed sets measures absolute robustness, while the difference in
execution accuracy between pre- and post-perturbed sets measures relative
robustness. We also evaluate _out-of-domain generalization_ by considering the
execution accuracy of zero-shot parsers on GeoQuery and KaggleDBQA.

##  6 Results

Our experiments compare models trained on the original Spider training data
against models trained on Spider with our augmentations. We also report
results for ChatGPT tested in a zero-shot mode. Appendix D provides additional
results (on more evaluation sets) and detailed versions of all tables.

###  6.1 Robustness to Question Perturbations

Table 3 reports average execution accuracy on evaluation sets from Dr.Spider
(Chang et al., 2023) containing perturbations in natural language questions.
We also present results on the original Spider development set. Pre/Post refer
to subsets before/after perturbations (post-perturbation sets are the same
Spider subsets but with the questions rewritten; absolute robustness).

We compare T5-3B with and without PICARD and RESDSQL models fine-tuned on the
original Spider data and our augmentations; we also provide results for
ChatGPT evaluated in the zero-shot setting. Our results show that ChatGPT is
most vulnerable to question reformulations among all models. Chang et al.
(2023) reach similar conclusions with Codex (Chen et al., 2021), another large
pre-trained language model, and hypothesize this is due to the training data
being biased towards docstrings (which is what most natural language
utterances look like on websites like GitHub).

Absolute robustness (accuracy on post-perturbed sets) improves by more than 3%
for augmented models compared to base models in almost all cases. Moreover,
the performance gap on pre- and post-perturbed data decreases which indicates
better relative robustness for augmented models. Augmented RESDSQL delivers
the highest post-perturbation accuracy of 72.5% and augmented PICARD
demonstrates the smallest gap between pre- and post-perturbations of 8.4%
confirming that our augmentations improve both absolute and relative
robustness.

Augmented models do not have an advantage over base models on the original
Spider development set (see the last row in Table 3). There are two reasons
for this: firstly, we augment questions only without adding new SQL queries,
and secondly, augmentations shift the language distribution by removing
specific details and rendering questions more natural, but the development set
remains closer to the original training set.

Augmented models do not have an advantage over base models on the original
Spider development set (see the last row in Table 3). There are two reasons
for this: firstly, we augment questions only without adding new SQL queries,
and secondly, augmentations shift the language distribution by removing
specific details and rendering questions more natural, but the development set
remains closer to the original training set.

|  | Dr.Spider NLQ
---|---|---
Model | Spider Dev $\uparrow$ | Pre $\uparrow$ | Post $\uparrow$ | Diff $\downarrow$
T5-3B | 74.4 | 70.3 | 58.9 | 11.4
+Augmented | 75.3 | 72.6 | 62.7 | 9.9
PICARD | 79.3 | 76.0 | 65.0 | 11.0
+Augmented | 79.3 | 76.7 | 68.3 | 8.4
RESDSQL | 84.1 | 84.7 | 69.3 | 15.4
+Augmented | 84.0 | 84.6 | 72.5 | 12.1
ChatGPT | 72.2 | 73.8 | 57.9 | 15.9
Table 3: Execution Accuracy on Spider development set and Dr.Spider NLQ subsets, before (Pre) and after perturbations (Post: absolute robustness) and the gap between them (Diff: relative robustness). All models are tested in a zero-shot setting; +Augmented refers to models fine-tuned on the augmented Spider data. |  | KaggleDBQA
---|---|---
Model | GeoQuery | Nuclear | Crime | Pesticide | Math | Baseball | Fires | WhatCD | Soccer | Avg
T5-3B | 54.4 | 59.4 | 48.2 | 16.0 | 7.1 | 20.5 | 43.2 | 7.3 | 16.7 | 27.3
+Augmented | 60.4 | 56.3 | 48.2 | 18.0 | 7.1 | 20.5 | 43.2 | 26.8 | 22.2 | 30.3
PICARD | 56.6 | 59.4 | 51.9 | 18.0 | 10.7 | 25.6 | 43.2 | 9.8 | 22.2 | 30.1
+Augmented | 62.6 | 56.3 | 48.1 | 22.0 | 14.3 | 25.6 | 43.2 | 24.4 | 27.8 | 32.7
RESDSQL | 56.6 | 59.4 | 48.1 | 16.0 | 25.0 | 23.1 | 43.2 | 17.1 | 22.2 | 31.8
+Augmented | 59.3 | 65.6 | 44.4 | 24.0 | 25.0 | 23.1 | 43.2 | 19.5 | 27.8 | 34.1
ChatGPT | 20.9 | 34.4 | 18.5 | 16.0 | 10.7 | 15.4 | 27.0 | 4.9 | 16.7 | 17.9
Table 4: Execution accuracy on GeoQuery test set (query splits) and different
databases from KaggleDBQA. All models are tested in a zero-shot setting;
+Augmented refers to models fine-tuned on the augmented Spider data.

###  6.2 Generalization to Other Datasets

Table 4 summarizes our results in the more challenging zero-shot setting.
Specifically, we evaluate model performance on two out-of-domain datasets,
namely GeoQuery Zelle and Mooney (1996) and KaggleDBQA Lee et al. (2021). Both
datasets differ from Spider in many respects, i.e., the types of questions
being asked, the style of SQL queries, and the database structure.

We find ChatGPT performs very poorly on these datasets compared to models
fine-tuned on Spider with or without augmentations. In all cases, augmented
models improve execution accuracy compared to base models. PICARD trained with
augmentations performs best on GeoQuery reaching an accuracy of 62.6% (a 6%
difference against the base model). Augmented RESDSQL performs best on
KaggleDBQA, which is more challenging, reaching an average accuracy of 34.1%.
Augmentations are generally helpful but not across all individual categories
(note that categories are represented by a limited number of examples per
database and even a small number of errors can result in a drop of several
percentage points). We suspect the low accuracy on KaggleDBQA is primarily due
to challenges that are unrelated to language variation. In particular, its
databases contain abbreviations which might be difficult to parse and SQL
queries exemplify operations which are not present in Spider (e.g., arithmetic
operators between columns).

###  6.3 Ablations and Analysis

We next investigate the impact of different types of question reformulations
introduced in Section 4, and also compare against related augmentation
methods: Gan et al. (2021a) manually annotate Spider-Syn with synonym
substitutions, whereas Ma and Wang (2021) introduce MT-teql, a framework for
generating semantics-preserving variants of utterances and database schemas.
We use a version of MT-teql that changes prefixes and aggregator mentions in
Spider questions. Additionally, we include a baseline which follows our
procedure for data generation but uses only one prompt: provide different ways
of expressing a question.

Table 5 shows the execution accuracy of T5-3B trained with and without
augmentations pertaining to Deletion, Substitution, Rewriting, and
Paraphrasing. We also include results with All augmentations combined. The
ablation study shows that different types of augmentation are helpful for
different datasets. On GeoQuery, models augmented with deletions and
substitutions perform best; substitutions also perform best on the NLQ sets of
Dr.Spider and KaggleDBQA. Paraphrasing-based augmentations are best for the
original Spider development set, with Rewriting trailing behind. Results
obtained with a single prompt (express in a different way) further illustrate
the need for diverse instructions. We also trained T5-3B with augmentations
from Spider-Syn (Gan et al., 2021a) and MT-teql (Ma and Wang, 2021). For a
fair comparison, we randomly sample MT-teql examples with question
transformations to match the training size obtained through our augmentations
(Spider-Syn and one-prompt baselines also match our training size). As can be
seen in Table 5, our combined augmentations outperform models trained on
Spider-Syn and MT-teql on all evaluation sets (Dr.Spider NLG, GeoQuery, and
KaggleDBQA) and the improvement comes from reformulating the questions rather
than increasing the training set.

|  | Dr.Spider NLQ |  |
---|---|---|---|---
Model | Spider Dev $\uparrow$ | Post $\uparrow$ | Diff $\downarrow$ | GeoQuery $\uparrow$ | KaggleDBQA $\uparrow$
T5-3B | 74.4 | 58.9 | 11.4 | 54.4 | 27.3
\+ Deletion | 74.7 | 59.7 | 11.5 | 56.0 | 28.7
\+ Substitution | 75.1 | 62.9 | 9.8 | 56.0 | 31.2
\+ Rewriting | 75.0 | 62.3 | 11.2 | 53.8 | 27.4
\+ Paraphrase | 75.3 | 61.4 | 11.6 | 41.8 | 25.9
\+ All (ours) | 75.3 | 63.2 | 9.9 | 60.4 | 30.3
\+ One Prompt | 74.4 | 60.4 | 15.4 | 40.7 | 29.2
\+ Spider-Syn | 75.6 | 59.2 | 13.2 | 49.5 | 27.0
\+ MT-teql* | 75.0 | 62.0 | 11.1 | 47.8 | 29.2
Table 5: Execution accuracy on Spider development set, Dr.Spider NLQ (Post:
absolute robustness; Diff: relative robustness), GeoQuery, and KaggleDBQA for
T5-3B base and trained with different augmentations including Spider-Syn (Gan
et al., 2021a) and sub-sampled (diacritic *) version of MT-teql (Ma and Wang,
2021).

The results in Table 5 reaffirm the observation that different evaluation sets
exemplify different linguistic variations and that there is no single type of
augmentation that represents them all. Rather, a _combination_ of
augmentations is needed to perform well _across_ datasets. This in turn
suggests that a model can acquire useful knowledge by being exposed to a
_diverse_ range of linguistic variations. We also observe that a model trained
on combined augmentations outperforms models trained on more specialized
datasets (i.e., Spider-Syn and MT-teql) which confirms that relying solely on
local transformations of the questions is not sufficient for better
generalization.

##  7 Conclusion

We propose to enhance the generalization capabilities of text-to-SQL parsers
by increasing natural language variation in the training data. We leverage a
large language model like ChatGPT to automatically generate a variety of
question reformulations, thereby augmenting existing datasets with more
natural and diverse questions. We evaluate state-of-the-art models trained
with and without our augmentations on a variety of challenging datasets
focusing on robustness (to perturbations) and out-of-domain generalization.
Across models and datasets we find that augmentations improve performance by a
wide margin. Our experiments further underscore the need for a broad range of
augmentations representing the full spectrum of rewrite operations. In the
future, we plan to explore the potential of large language models for
multilingual semantic parsing.

## Limitations

Our work aims to increase the robustness of semantic parsers against natural
language variation but does not handle problems related to SQL queries and
database structures that are also important for out-of-domain generalization.
We obtain augmentations using ChatGPT, a black-box model provided by OpenAI,
which limits its usage for non-academic purposes. Our augmentations are
unfiltered and may add a small amount of noise to training data. Moreover,
even though our proposed rewrite operations are diverse, they may still not
cover all possible reformulations. In fact, we found it challenging for
ChatGPT to generate wildly different expressions of the original intent.
Finally, this work does not consider multilingual or conversational semantic
parsing which we hope to explore in the future.

### Acknowledgments

We thank the meta-reviewer and anonymous reviewers for their constructive
feedback. The authors also thank Hao Zheng for insightful comments on earlier
versions of this work. We gratefully acknowledge the support of the UK
Engineering and Physical Sciences Research Council (grant EP/L016427/1).

## References

  * Artzi and Zettlemoyer (2011) Yoav Artzi and Luke Zettlemoyer. 2011.  Bootstrapping semantic parsers from conversations.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 421–432, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Proceedings of the 33st Annual Conference on Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Campagna et al. (2017) Giovanni Campagna, Rakesh Ramesh, Silei Xu, Michael Fischer, and Monica S. Lam. 2017.  Almond: The architecture of an open, crowdsourced, privacy-preserving, programmable virtual assistant.  In _Proceedings of the 26th International Conference on World Wide Web_ , WWW ’17, page 341–350, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Chang et al. (2023) Shuaichen Chang, Jun Wang, Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hanbo Li, Wuwei Lan, Sheng Zhang, Jiarong Jiang, Joseph Lilien, Steve Ash, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, and Bing Xiang. 2023.  Dr.spider: A diagnostic evaluation benchmark towards text-to-SQL robustness.  In _The 11th International Conference on Learning Representations_.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, Guss, et al. 2021.  Evaluating large language models trained on code.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  In _Proceedings of the 31st Annual Conference on Neural Information Processing Systems_ , volume 30, Long Beach, CA, USA. Curran Associates, Inc.
  * Dai et al. (2023) Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Lichao Sun, Quanzheng Li, Dinggang Shen, Tianming Liu, and Xiang Li. 2023.  Auggpt: Leveraging chatgpt for text data augmentation.
  * Deng et al. (2021) Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2021.  Structure-grounded pretraining for text-to-SQL.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1337–1350, Online. Association for Computational Linguistics.
  * Dukes (2014) Kais Dukes. 2014.  SemEval-2014 task 6: Supervised semantic parsing of robotic spatial commands.  In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_ , pages 45–53, Dublin, Ireland. Association for Computational Linguistics.
  * Finegan-Dollak et al. (2018) Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018.  Improving text-to-SQL evaluation methodology.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 351–360, Melbourne, Australia. Association for Computational Linguistics.
  * Gan et al. (2021a) Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, and Pengsheng Huang. 2021a.  Towards robustness of text-to-SQL models against synonym substitution.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2505–2515, Online. Association for Computational Linguistics.
  * Gan et al. (2021b) Yujian Gan, Xinyun Chen, and Matthew Purver. 2021b.  Exploring underexplored limitations of cross-domain text-to-SQL generalization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8926–8931, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Gan et al. (2021c) Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John R. Woodward, John Drake, and Qiaofu Zhang. 2021c.  Natural SQL: Making SQL easier to infer from natural language specifications.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2030–2042, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Hazoom et al. (2021) Moshe Hazoom, Vibhor Malik, and Ben Bogin. 2021.  Text-to-SQL in the wild: A naturally-occurring dataset based on stack exchange data.  In _Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021)_ , pages 77–87, Online. Association for Computational Linguistics.
  * He et al. (2023) Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2023.  Annollm: Making large language models to be better crowdsourced annotators.
  * Huang et al. (2021) Shuo Huang, Zhuang Li, Lizhen Qu, and Lei Pan. 2021.  On robustness of neural semantic parsers.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3333–3342, Online. Association for Computational Linguistics.
  * Jiang et al. (2022) Jiarong Jiang, Yiqun Hu, Wuwei Lan, Henry Zhu, Anuj Chauhan, Alexander Li, Lin Pan, Jun Wang, Chung-Wei Hang, Sheng Zhang, Marvin Dong, Joe Lilien, Patrick Ng, Zhiguo Wang, Vittorio Castelli, and Bing Xiang. 2022.  Importance of synthesizing high-quality data for text-to-sql parsing.  In _NeurIPS 2022 Workshop on SyntheticData4ML_.
  * Lee et al. (2021) Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. 2021.  KaggleDBQA: Realistic evaluation of text-to-SQL parsers.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2261–2273, Online. Association for Computational Linguistics.
  * Li et al. (2023a) Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023a.  Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql.  In _Proceedings of the 37th AAAI Conference on Artificial Intelligence_ , pages 13067–13075, Washington, DC, USA. AAAI Press.
  * Li et al. (2019) Jingjing Li, Wenlu Wang, Wei-Shinn Ku, Yingtao Tian, and Haixun Wang. 2019.  Spatialnli: A spatial domain natural language interface to databases using spatial comprehension.  In _Proceedings of the 27th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems_ , SIGSPATIAL ’19, page 339–348, New York, NY, USA. Association for Computing Machinery.
  * Li et al. (2023b) Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Chenhao Ma, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023b.  Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls.
  * Liu et al. (2023) Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S. Yu. 2023.  A comprehensive evaluation of chatgpt’s zero-shot text-to-sql capability.
  * Ma and Wang (2021) Pingchuan Ma and Shuai Wang. 2021.  Mt-teql: Evaluating and augmenting neural nlidb on real-world linguistic and schema variations.  _Proceedings of the VLDB Endowment_ , 15(3):569–582.
  * Őzcan et al. (2020) Fatma Őzcan, Abdul Quamar, Jaydeep Sen, Chuan Lei, and Vasilis Efthymiou. 2020.  State of the art and open challenges in natural language interfaces to data.  In _Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data_ , SIGMOD ’20, page 2629–2636, New York, NY, USA. Association for Computing Machinery.
  * Pi et al. (2022) Xinyu Pi, Bing Wang, Yan Gao, Jiaqi Guo, Zhoujun Li, and Jian-Guang Lou. 2022.  Towards robustness of text-to-SQL models against natural and realistic adversarial table perturbation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2007–2022, Dublin, Ireland. Association for Computational Linguistics.
  * Radhakrishnan et al. (2020) Karthik Radhakrishnan, Arvind Srikantan, and Xi Victoria Lin. 2020.  ColloQL: Robust text-to-SQL over search queries.  In _Proceedings of the First Workshop on Interactive and Executable Semantic Parsing_ , pages 34–45, Online. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rajkumar et al. (2022) Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022.  Evaluating the text-to-sql capabilities of large language models.
  * Scholak et al. (2021) Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021.  PICARD: Parsing incrementally for constrained auto-regressive decoding from language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9895–9901, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Shaw et al. (2021) Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021.  Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 922–938, Online. Association for Computational Linguistics.
  * Suhr et al. (2020) Alane Suhr, Ming-Wei Chang, Peter Shaw, and Kenton Lee. 2020.  Exploring unexplored generalization challenges for cross-database semantic parsing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8372–8388, Online. Association for Computational Linguistics.
  * Wang et al. (2021) Bailin Wang, Wenpeng Yin, Xi Victoria Lin, and Caiming Xiong. 2021.  Learning to synthesize data for semantic parsing.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2760–2766, Online. Association for Computational Linguistics.
  * Wu et al. (2021) Kun Wu, Lijie Wang, Zhenghua Li, Ao Zhang, Xinyan Xiao, Hua Wu, Min Zhang, and Haifeng Wang. 2021.  Data augmentation with hierarchical SQL-to-question generation for cross-domain text-to-SQL parsing.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8974–8983, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018.  Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3911–3921, Brussels, Belgium. Association for Computational Linguistics.
  * Zelle and Mooney (1996) John M Zelle and Raymond J Mooney. 1996.  Learning to parse database queries using inductive logic programming.  In _Proceedings of the 13th AAAI Conference on Artificial Intelligence_ , volume 2, pages 1050–1055, Portland, Oregon. AAAI Press.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  Opt: Open pre-trained transformer language models.
  * Zhong et al. (2020a) Ruiqi Zhong, Tao Yu, and Dan Klein. 2020a.  Semantic evaluation for text-to-SQL with distilled test suites.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 396–411, Online. Association for Computational Linguistics.
  * Zhong et al. (2020b) Victor Zhong, Mike Lewis, Sida I. Wang, and Luke Zettlemoyer. 2020b.  Grounded adaptation for zero-shot executable semantic parsing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6869–6882, Online. Association for Computational Linguistics.
  * Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher. 2017.  Seq2SQL: Generating structured queries from natural language using reinforcement learning.

##  Appendix A Data Generation

Table 6 shows the full versions of the prompts we use to generate the
augmentations defined in Section 4 for the Spider training set.

1\. Instruction: |  Simplify
---|---
Full version |  Simplify the following sentence: $\dots$
2\. Instruction: |  Simplify by hiding details
Full version |  Simplify the sentence by hiding unnecessary details that do not change the meaning: $\dots$
3\. Instruction: |  Simplify using synonyms
Full version |  Simplify the following sentence using synonyms: $\dots$
4\. Instruction: |  Simplify using substitutions
Full version |  Make the sentence simpler by substituting some words in $\dots$
5\. Instruction: |  Express in a different way
Full version |  What are different ways of expressing this question: $\dots$
6\. Instruction: |  Examples of the question simplification: <$\dots$>
Full version |  Examples of the question simplification:
|  Original: Find the names of stadiums whose capacity is smaller than the
average capacity.
|  Simplified: Which stadiums are smaller than the average?
|  Original: Show the fleet series of aircraft flown by pilots younger than
34.
|  Simplified: Return the fleet series of the planes whose captains are
younger than 34.
|  Original: Which cities have the largest population?
|  Simplified: Where do most people live?
|  Original: In which year was most of the ships built?
|  Simplified: When were most of the ships constructed?
|  Original: Tell me the number of orders with "Second time" as the order
detail.
|  Simplified: How many orders have "Second time" as an order detail?
|  Original: $\dots$
|  Simplified:
7\. Instruction: |  Paraphrase
Full version |  Give me a paraphrase of the following question: $\dots$
Table 6: Prompts used for data generation. |  |  | Augmented |  | Augmented |  | Augmented |
---|---|---|---|---|---|---|---|---
|  | T5-3B | T5-3B | PICARD | PICARD | RESDSQL | RESDSQL | ChatGPT
Perturbation Set | Pre | Post | Pre | Post | Pre | Post | Pre | Post | Pre | Post | Pre | Post | Pre | Post
NLQ | Keyword-synonym | 70.2 | 62.6 | 73.8 | 65.4 | 72.6 | 66.3 | 75.3 | 69.4 | 81.5 | 72.4 | 84.2 | 74.7 | 64.7 | 55.7
Keyword-carrier | 82.7 | 76.4 | 83.0 | 79.2 | 85.0 | 82.7 | 88.7 | 84.0 | 89.0 | 83.5 | 87.5 | 85.0 | 85.0 | 82.0
Column-synonym | 63.9 | 51.3 | 66.3 | 54.2 | 71.0 | 57.2 | 68.7 | 59.7 | 78.7 | 63.1 | 77.4 | 66.1 | 66.1 | 48.8
Column-carrier | 83.1 | 61.7 | 82.0 | 70.5 | 86.9 | 64.9 | 85.0 | 73.1 | 86.5 | 63.9 | 86.4 | 76.3 | 82.2 | 52.0
Column-attribute | 49.6 | 48.7 | 60.5 | 58.8 | 58.8 | 56.3 | 63.9 | 62.2 | 82.4 | 71.4 | 82.4 | 71.4 | 77.3 | 62.2
Column-value | 69.1 | 58.6 | 76.3 | 58.9 | 82.9 | 69.4 | 83.2 | 70.4 | 96.4 | 76.6 | 95.1 | 77.6 | 74.0 | 57.9
Value-synonym | 68.6 | 46.4 | 68.6 | 53.0 | 72.5 | 53.0 | 70.8 | 57.1 | 79.2 | 53.2 | 79.6 | 55.1 | 69.0 | 45.8
Multitype | 70.1 | 51.1 | 71.4 | 56.3 | 74.4 | 57.1 | 74.0 | 61.4 | 83.8 | 60.7 | 83.8 | 65.7 | 71.9 | 49.8
Others | 75.3 | 73.1 | 76.6 | 72.7 | 79.6 | 78.3 | 80.9 | 77.6 | 85.2 | 79.0 | 84.8 | 80.2 | 74.0 | 66.4
Average | 70.3 | 58.9 | 73.2 | 63.2 | 76.0 | 65.0 | 76.7 | 68.3 | 84.7 | 69.3 | 84.6 | 72.5 | 73.8 | 57.9
SQL | Comparison | 62.9 | 62.4 | 71.3 | 66.3 | 68.0 | 68.0 | 74.2 | 70.8 | 80.9 | 82.0 | 84.3 | 83.7 | 73.6 | 64.0
Sort-order | 75.0 | 70.3 | 76.0 | 75.5 | 79.2 | 74.5 | 78.1 | 76.6 | 88.0 | 85.4 | 88.5 | 83.3 | 66.7 | 57.8
NonDB-number | 77.1 | 73.3 | 71.8 | 77.1 | 83.2 | 77.1 | 73.3 | 77.9 | 87.8 | 85.5 | 90.8 | 90.8 | 90.8 | 90.1
DB-text | 59.5 | 58.3 | 59.9 | 61.6 | 64.7 | 65.1 | 66.2 | 66.7 | 77.2 | 74.3 | 91.5 | 75.0 | 67.5 | 68.2
DB-number | 83.9 | 83.7 | 79.8 | 78.8 | 86.3 | 85.1 | 84.6 | 83.2 | 88.8 | 88.8 | 91.5 | 91.2 | 82.7 | 79.8
Average | 71.7 | 69.6 | 71.8 | 71.9 | 76.3 | 74.0 | 75.3 | 75.0 | 84.5 | 83.2 | 89.3 | 84.8 | 76.3 | 72.0
DB | Schema-synonym | 66.4 | 46.9 | 67.8 | 52.8 | 73.0 | 56.5 | 73.4 | 61.9 | 81.3 | 68.3 | 80.9 | 70.4 | 67.6 | 56.0
Schema-abbreviation | 69.5 | 53.3 | 71.0 | 55.5 | 74.9 | 64.7 | 75.2 | 65.3 | 82.4 | 70.0 | 81.8 | 71.7 | 68.8 | 63.5
Content-equivalence | 84.6 | 40.8 | 72.3 | 46.1 | 88.7 | 43.7 | 86.9 | 37.2 | 90.3 | 40.1 | 91.9 | 41.4 | 81.2 | 46.3
Average | 73.5 | 47.0 | 72.3 | 46.1 | 78.9 | 55.0 | 78.5 | 54.8 | 84.7 | 59.5 | 84.9 | 61.1 | 72.5 | 55.3
All | 71.3 | 59.9 | 72.6 | 62.7 | 76.6 | 65.9 | 76.6 | 67.9 | 84.7 | 71.7 | 86.0 | 74.1 | 74.3 | 61.5
Table 7: Execution Accuracy on subsets taken from Dr.Spider (NLQ, DB, and SQL sets); model performance is shown before (Pre) and after perturbations (Post). We compare T5-3B, T5-3B+PICARD, and RESDSQL fine-tuned with and without augmentations, and zero-shot ChatGPT. |  | Augmented |  | Augmented |  | Augmented |
---|---|---|---|---|---|---|---
Dataset | T5-3B | T5-3B | PICARD | PICARD | RESDSQL | RESDSQL | ChatGPT
Realistic | 64.2 | 66.7 | 71.4 | 79.3 | 80.7 | 84.0 | 63.4
Spider-Syn | 62.4 | 70.8 | 69.8 | 72.8 | 76.9 | 79.2 | 58.6
GeoQuery dev | 59.1 | 64.2 | 64.2 | 68.6 | 59.7 | 54.1 | 25.8
Table 8: Execution accuracy on Spider-Realistic, Spider-Syn and GeoQuery dev
set for T5-3B with and without PICARD and RESDSQL trained with or without
augmentations.

##  Appendix B Error Analysis

In order to verify that our augmentations do not introduce new parsing errors,
we examined examples in the Spider development set which were correctly parsed
by a T5 model trained without augmentations but rendered incorrect after the
same T5 model was trained with augmentations. Based on a sample of 60
instances, we observed that the majority of errors are similar in nature and
symptomatic of a T5-trained semantic parser, e.g., errors in the output
columns or join operation.

The only type of error that might be due to our augmentations concerns minor
changes in values. Baseline T5 almost always copies values from the question
but T5 trained with augmentations can slightly change them, e.g., use the full
name instead of an abbreviation or lowercase instead of uppercase. We found
this occurs in 10% of cases. Database values are mentioned verbatim in Spider
questions but this could be different in real-world settings or other datasets
where some tolerance to surface variations might be advantageous.

##  Appendix C ChatGPT Zero-Shot Prompt

Below we show the prompt we used when evaluating the zero-shot ChatGPT on
text-to-SQL datasets following Liu et al. (2023):

⬇

### SQL tables, with their properties:

#

# stadium(Stadium_ID, Location, Name, Capacity, Highest, Lowest, Average)

# singer(Singer_ID, Name, Country, Song_Name, Song_release_year, Age, Is_male)

# concert(concert_ID, concert_Name, Theme, Stadium_ID, Year)

# singer_in_concert(concert_ID, Singer_ID)

#

### How many singers do we have? Return only a SQL query.

SELECT

##  Appendix D Additional Results

Table 7 shows our results on _all_ Dr.Spider perturbation subsets (NLQ refers
to subsets with perturbations in natural language questions, SQL and DB are
perturbations in SQL and database tokens). We compare three models trained
with and without augmentations: T5-3B, PICARD, and RESDSQL. We also employ
ChatGPT in a zero-shot setting. Overall, the best model is augmented RESDSQL
(74.1%) which is better than the base version by more than 2% on post-
perturbed sets. Augmented T5-3B and PICARD also improve robustness compared to
base models. Augmented RESDSQL delivers the best average results for all three
types of perturbations and performs best on the majority of individual
categories, even though our augmentations are _not_ designed to improve
robustness against SQL and DB perturbations.

Table 8 shows results on the additional evaluation sets, Spider-Realistic,
(Gan et al., 2021a) Spider-Syn with 1,034 examples, and GeoQuery dev set with
152 examples (query splits of Finegan-Dollak et al. 2018). Both evaluation
sets are based on the Spider development set, aiming to remove from the
questions explicit references to database entities. These references were
manually deleted or paraphrased in Spider-Realistic and replaced with synonyms
in Spider-Syn. Augmented RESDSQL obtains best results on both datasets (84.0%
on Spider-Realistic and 79.2% on Spider-Syn) and is better than the base
version by more than 4%. On the GeoQuery development set, the best model is
augmented PICARD with 68.6% accuracy. Across _all_ benchmarks, fine-tuned
text-to-SQL parsers significantly outperform zero-shot ChatGPT.

##  Appendix E Examples of Spider Augmentations

We provide samples of the augmented Spider training set. Questions are grouped
based on intent; types indicate whether they are in the original Spider
training set or were generated with instructions: simplify, simplify by hiding
details, simplify by synonyms, simplify by substitutions, express differently,
paraphrase, or by showing examples.

1. |  SQL query | SELECT personal_name FROM Students EXCEPT SELECT T1.personal_name FROM
---|---|---
|  | Students AS T1 JOIN Student_Course_Enrolment AS T2 ON T1.student_id = T2.student_id
|  Questions |  Find the personal names of students not enrolled in any course. |  Type: original
|  |  Which students not enrolled in any course? Find their personal names. |  Type: original
|  |  Find names of unenrolled students. |  Type: hide-details
|  |  Can you provide me with the names of the students who are not currently registered in any course? |  Type: paraphrase
2. |  SQL query | SELECT donator_name , sum(amount) FROM endowment GROUP BY donator_name
|  | ORDER BY sum(amount) DESC
|  Questions |  List each donator name and the amount of endowment in descending order of the amount of endowment |  Type: original
|  |  Enumerate contributors and their endowment sums in decreasing order |  Type: synonyms
|  |  List donors and their endowments in descending order |  Type: hide-details
3. |  SQL query | SELECT count(*) FROM CLASS AS T1 JOIN enroll AS T2 ON T1.class_code = T2.class_code
|  | WHERE T1.crs_code = ’ACCT-211’
|  Questions |  How many students enrolled in class ACCT-211? |  Type: original
|  |  What are the total number of students enrolled in ACCT-211? |  Type: original
|  |  How many pupils registered for course ACCT-211? |  Type: synonyms
|  |  How many students are enrolled in ACCT-211? |  Type: from-examples
4. |  SQL query | SELECT T2.roomName FROM Reservations AS T1 JOIN Rooms AS T2 ON T1.Room = T2.RoomId
|  | WHERE firstname LIKE ’%ROY%’
|  Questions |  Find the name of rooms booked by some customers whose first name contains ROY. |  Type: original
|  |  What are the name of rooms booked by customers whose first name has "ROY" in part? |  Type: original
|  |  What are the room names that have been reserved by customers with "ROY" in their first name? |  Type: paraphrase
|  |  What rooms did customers with "ROY" in their first name book? |  Type: substitutions
5. |  SQL query | SELECT T1.account_name , T1.other_account_details FROM Accounts AS T1 JOIN
|  | Customers AS T2 ON T1.customer_id = T2.customer_id WHERE
|  | T2.customer_first_name = ’Meaghan’ AND T2.customer_last_name = ’Keeling’
|  Questions |  Show the account name and other account detail for all accounts by the customer with first name Meaghan and last name Keeling. |  Type: original
|  |  What are the names and other details for accounts corresponding to the customer named Meaghan Keeling? |  Type: original
|  |  What are Meaghan Keeling’s account names and details? |  Type: simplify
|  |  I am looking for the account details and names associated with Meaghan Keeling. Can you help me with that? |  Type: paraphrase
6. |  SQL query | SELECT sum(acc_bal) FROM customer WHERE state = ’Utah’ OR state = ’Texas’
|  Questions |  Find the total account balance of each customer from Utah or Texas. |  Type: original
|  |  What are the total account balances for each customer from Utah or Texas? |  Type: original
|  |  Add up the account balances of customers who live in Utah or Texas. |  Type: express-differently
|  |  What is the total account balance for customers from Utah or Texas? |  Type: from-examples
7. |  SQL query | SELECT date_of_enrolment , date_of_completion FROM Student_Course_Enrolment
|  Questions |  List all the dates of enrollment and completion of students. |  Type: original
|  |  What are all the dates of enrollment and completion in record? |  Type: original
|  |  Provide a record of the enrollment and completion dates for all students. |  Type: paraphrase
|  |  What are the enrollment and completion dates of all students? |  Type: from-examples
8. |  SQL query | SELECT headquarter FROM manufacturers WHERE founder = ’James’
|  Questions |  Where is the headquarter of the company founded by James? |  Type: original
|  |  What is the headquarter of the company whose founder is James? |  Type: original
|  |  Where was the company founded by James headquartered? |  Type: express-differently
|  |  Where is the main office of the company established by James? |  Type: paraphrase

9. |  SQL query | SELECT max(Price) , max(Score) FROM WINE WHERE Appelation = ’St. Helena’
---|---|---
|  Questions |  What are the maximum price and score of wines produced by St. Helena appelation? |  Type: original
|  |  Give the maximum price and score for wines produced in the appelation St. Helena. |  Type: original
|  |  What is the topmost price and score that can be obtained by wines produced in St. Helena? |  Type: paraphrase
|  |  What is the highest price and score for St. Helena wines? |  Type: simplify
10. |  SQL query | SELECT degrees FROM campuses AS T1 JOIN degrees AS T2 ON t1.id = t2.campus
|  | WHERE t1.campus = ’San Francisco State University’ AND t2.year = 2001
|  Questions |  What are the degrees conferred in "San Francisco State University" in 2001. |  Type: original
|  |  What degrees were conferred in San Francisco State University in the year 2001? |  Type: original
|  |  What diplomas were granted at SF State in 2001? |  Type: synonyms
|  |  What degrees were given at San Francisco State University in 2001? |  Type: substitutions
11. |  SQL query | SELECT membership_card FROM member WHERE address = ’Hartford’ INTERSECT
|  | SELECT membership_card FROM member WHERE address = ’Waterbury’
|  Questions |  What is the membership card held by both members living in Hartford and ones living in Waterbury address? |  Type: original
|  |  What is the membership card for people in Hartford and Waterbury called? |  Type: substitutions
|  |  Is there a membership card that is valid for both Hartford and Waterbury residents? |  Type: express-differently
12. |  SQL query | SELECT kids FROM Reservations WHERE FirstName = ’ROY’ AND LastName = ’SWEAZY’
|  Questions |  How many kids stay in the rooms reserved by ROY SWEAZY? |  Type: original
|  |  Find the number of kids staying in the rooms reserved by a person called ROY SWEAZ. |  Type: original
|  |  How many children are staying in ROY SWEAZY’s reserved rooms? |  Type: from-examples
|  |  How many kids are in Roy Sweaz’s reserved rooms? |  Type: hide-details
13. |  SQL query | SELECT count(*) FROM products AS t1 JOIN product_characteristics AS t2
|  | ON t1.product_id = t2.product_id JOIN CHARACTERISTICS AS t3
|  | ON t2.characteristic_id = t3.characteristic_id WHERE t1.product_name = ’laurel’
|  Questions |  How many characteristics does the product named "laurel" have? |  Type: original
|  |  Count the number of characteristics of the product named ’laurel’. |  Type: original
|  |  How many features does "laurel" have? |  Type: simplify
|  |  How many qualities does the product "laurel" have? |  Type: substitutions
14. |  SQL query | SELECT customer_name FROM customers WHERE payment_method = (SELECT payment_method
|  | FROM customers GROUP BY payment_method ORDER BY count(*) DESC LIMIT 1)
|  Questions |  What are the names of customers using the most popular payment method? |  Type: original
|  |  Find the name of the customers who use the most frequently used payment method. |  Type: original
|  |  Who are the customers using the popular payment method? |  Type: hide-details
|  |  Who are the customers utilizing the most favored payment option? |  Type: synonyms
15. |  SQL query | SELECT TYPE FROM ship WHERE Tonnage > 6000 INTERSECT SELECT TYPE FROM ship
|  | WHERE Tonnage < 4000
|  Questions |  Show the types of ships that have both ships with tonnage larger than 6000 and ships with tonnage smaller than 4000. |  Type: original
|  |  What are the types of the ships that have both shiips with tonnage more than 6000 and those with tonnage less than 4000? |  Type: original
|  |  Display ships with tonnage above 6000 and below 4000. |  Type: simplify
|  |  Which types of ships have tonnage exceeding 6000 and also less than 4000? |  Type: express-differently
16. |  SQL query | SELECT customer_name FROM customers EXCEPT SELECT t1.customer_name FROM customers AS t1
|  | JOIN customer_addresses AS t2 ON t1.customer_id = t2.customer_id JOIN addresses AS t3
|  | ON t2.address_id = t3.address_id WHERE t3.state_province_county = ’California’
|  Questions |  Find the names of customers who are not living in the state of California |  Type: original
|  |  Discover the names of non-California customers. |  Type: substitutions
|  |  Who are the customers not residing in California? |  Type: from-examples


# The DKU-Tencent System for the VoxCeleb Speaker Recognition Challenge 2022

###### Abstract

This paper is the system description of the DKU-Tencent System for the
VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC22). In this challenge, we
focus on track1 and track3. For track1, multiple backbone networks are adopted
to extract frame-level features. Since track1 focus on the cross-age
scenarios, we adopt the cross-age trials and perform QMF to calibrate score.
The magnitude-based quality measures achieve a large improvement. For track3,
the semi-supervised domain adaptation task, the pseudo label method is adopted
to make domain adaptation. Considering the noise labels in clustering, the
ArcFace is replaced by Sub-center ArcFace. The final submission achieves 0.107
mDCF in task1 and 7.135% EER in task3.

Index Terms: speech recognition, cross-age, semi-supervised, domain-adaptation

##  1 Task1: Fully supervised speaker verification

###  1.1 Data Augmentation

We adopt the on-the-fly data augmentation [1] to add additive background noise
or convolutional reverberation noise for the time-domain waveform. The MUSAN
[2] and RIR Noise [3] datasets are used as noise sources and room impulse
response functions, respectively. The convolution operation is performed for
the reverberation with 40,000 simulated room impulse responses. We only use
RIRs from small and medium rooms. To further diversify training samples, we
apply amplification or playback speed change (pitch remains untouched) to
audio signals. Also, we apply speaker augmentation with speed perturbation
[4]. We speed up or down each utterance by a factor of 0.9 or 1.1, yielding
shifted pitch utterances that are considered from new speakers. As a result,
the training data includes $3,276,027(1,092,009\times 3)$ utterances from
$17,982(5,994\times 3)$ speakers. The probability of Noise/RIR/Tempo/Volume is
0.66.

###  1.2 Speaker Embedding Model

Generally, the speaker embedding model consists of three parts: backbone
network, encoding layer and loss function. The ArcFace [5], which could
increase intra-speaker distances while ensuring inter-speaker compactness, is
used as a loss function. The backbone and encoding layer are reported as
follows.

####  1.2.1 Backbone Network

In this module, we introduce four different speaker verification systems,
including the SimAM-ResNet34[6], the ResNet101[7], ResNet152, and
Res2Net101[8]. The acoustic features are 80-dimensional log Mel-filterbank
energies with a frame length of 25ms and hop size of 10ms. The extracted
features are mean-normalized before feeding into the deep speaker network.

####  1.2.2 Encoding Layer

The encoding layer could transform the variable length feature maps into a
fixed dimension vector, that is, the frame-level feature is converted to the
segment-level feature. In this module, we adopt the statistic pooling[9],
standard deviation pooling and attentive statistic pooling[10]. The statistic
pooling layer computes the mean and standard deviation of the output feature
maps along with the frequency dimension. The standard deviation pooling layer
only calculates the standard deviation.

###  1.3 Training Strategy and Implement Detail

The speaker embedding model is trained through two stages. In the first stage,
the baseline system adopts the SGD optimizer. We adopt the ReduceLROnPlateau
scheduler with 2 epochs of patience. The init learning rate starts from 0.1,
the minimum learning rate is 1.0e-4, and the decay factor is 0.1. The margin
and scale of ArcFace are set as 0.2 and 32, respectively. We perform a linear
warmup learning rate schedule at the first 2 epochs to prevent model vibration
and speed model training. The input frame length is fixed at 200 frames. In
the second stage, we following the SpeakIn training protocol[11] with large
margin fine-tune (LMFT)[12]. In the LMFT stage, we also remove the speaker
augmentation and reduce the probability of data augmentation to 0.33. What’s
more, the margin is increased from 0.2 to 0.5. According to the speaker
embedding model size and the memory limit of GPU, the frame length is expended
from 200 to 400 or 600. The learning rate decays from 1.0e-4 to 1.0e-5.

Table 1: The baseline system performance of track1. Model | Vox-O(cleaned) | VoxSRC22 val | Vox-CA | VoxSRC22 eval
---|---|---|---|---
EER[%] | mDCF0.05 | EER[%] | mDCF0.05 | EER[%] | mDCF0.05 | EER[%] | mDCF0.05
SimAM-ResNet34 |  |  |  |  |  |  |  |
\+ LMFT | 0.542 | 0.034 | 1.789 | 0.108 | 6.944 | 0.263 | 3.585 | 0.219
++ AS-Norm (Top 400) | - | - | - | - | 6.852 | 0.249 | 3.400 | 0.200
+++ QMF | - | - | - | - | 6.801 | 0.240 | 2.478 | 0.139

###  1.4 Score Calibration and Normalization

The cosine similarity is the back-end scoring method. After scoring, results
from all trials are subject to score normalization. We utilize Adaptive
Symmetric Score Normalization (AS-Norm) [13] in our systems. The imposter
cohort consists of the average of the length normalized utterance-based
embeddings of each training speaker, i.e., a speaker-wise cohort with 5994
embeddings.

Quality Measure Functions could calibrate the scores and improve the system
performance. As [14] described, we adopt the two qualities set: speech
duration and magnitude rate. For speech duration, the value of
$|log(d_{u}-d_{min})|$ is adopt as a QMF. $d_{u}$ indicates the duration of
enrollment or test utterance. $d_{min}$ is slightly lower than the minimum
test utterance duration to prevent the $log(.)$ approaching NaN. The magnitude
rate is formula as:$|log(\frac{\|\mathbf{z}_{e}\|}{\|\mathbf{z}_{t}\|})|$. As
shown in [15], the embedding magnitude could contain quality information about
the original utterance. The experimental result also verifies the magnitude
information is effective.

###  1.5 Results

####  1.5.1 Baseline System

In this subsection, we will show baseline system performance and how to
improve performance step by step. We adopt the SimAM-ResNet34 as our baseline
system, which is easy to train and fast to fit. At first, we validate the
baseline system under the VoxSRC22 validation and evaluation set. It’s easily
observed that there is a large gap between the val set and eval set. To
prevent the model overfit the validation set, we follow our previous work
[16], which is focused on the cross-age speaker verification, and adopt the
Vox-CA15 trial as our validate trial. The design detail is shown in the [16].
Therefore, tuning the hyper-parameter is operated on the Vox-CA. The AS-Norm
achieves a relative 10% improvement, but the QMF achieves a 30% improvement.
According to the weight of logistic regression, the coefficient of magnitude
rate is much higher than that of the similarity score between enrollment and
test embedding. After all back-end processing, the system achieves the
relative 35% improvement in mDCF. The other models will adopt the same
operation, and Table.2 only reports the final results.

####  1.5.2 Fusion System

Table.2 presents the performance of final models we used. At first pre-
training stage, we have multiple combinations of the backbone network and
encoding layer, the models in Table.2 are the best ones on VoxCeleb-O
(cleaned) trials. Comparing models 1 and 2, we find the lower EER in Vox-CA
and the better mDCF in VoxSRC22 eval. Therefore, the final score is the sum of
models 2, 3, 4 and 5 with average weight. The final result achieves the 2.078%
EER and 0.107 mDCF.

Table 2: The performance of various systems in the track3. ID & Model | Vox-CA | VoxSRC22 eval
---|---|---
EER[%] | mDCF0.05 | EER[%] | mDCF0.05
1 SimAM-ResNet34 | 6.801 | 0.240 | 2.478 | 0.139
2 ResNet101-ASP | 6.494 | 0.224 | 2.139 | 0.122
3 ResNet152-Stat | 6.654 | 0.240 | - | -
4 ResNet152-ASP | 6.538 | 0.224 | - | -
5 Res2Net101-Std | 6.534 | 0.244 | - | -
Fusion(1+2+3+4) | - | - | 2.085 | 0.117
Fusion(2+3+4+5) | - | - | 2.078 | 0.107
Figure 1: SSE versus the number of cluster K employed in unsupervised target
domain data

##  2 Task3: Semi-supervised domain adaptation

Semi-supervised domain adaptation is a new task proposed this year. We are
following the FFSVC baseline system method we proposed [17]. The pseudo-label
method is adopted for a large-scale set of unlabeled data from the target
domain. The detail is shown in the following.

###  2.1 Pseudo Label Method

In this case, the VoxCeleb data is considered as the out-of-domain labeled
data and the permitted CN-Celeb data is treated as the in-domain unlabeled
data. The labeled data is used to pre-train a model for pseudo-labeling. For
semi-supervised learning operation in the transfer learning phase, the speaker
embedding from unlabeled data is extracted using the pre-trained model and the
following clustering algorithms to generate pseudo-labels for the unlabeled
data. The following describes the algorithm generating pseudo-labels:

  * •

Step 1. Cleaning audios. Since too short duration audios may not contain text
information, only audios greater than 1s in length are retained.

  * •

Step 2. Extract embedding. Extract all speaker embeddings $Z\in R^{N\times d}$
from the permitted CN-Celeb dataset using the pre-trained speaker model.

  * •

Step 3. Clustering. Run a clustering algorithm with the different number of
clusters K to obtain centroid matrix $C\in R^{d}$ for each K.

  * •

Step 4. Calculate the sum of square error (SSE) and observe the ‘elbow’ of the
SSE curve to determine the number of clusters K .

  * •

Step 5. Pseudo labeling. Create the pseudo labels for the target domain
unlabeled data.

  * •

Step 6. Fine-tuning. Use the pseudo-labels data together with the labeled data
into the speaker embedding model to fine-tune the model.

  * •

Step 7. Repeat Step 2 with the fine-tuned model from Step 6 as the pre-trained
model.

Table 3: The performance of various systems in the track3. ID & Model | Iteration | VoxSRC22 val | VoxSRC22 eval
---|---|---|---
EER[%] | mDCF0.05 | EER[%] | mDCF0.05
1 SimAM-ResNet34 | - |  |  |  |
\+ pseudo label (ArcFace) | Round1 | 10.726 | 0.458 | - | -
\+ pseudo label (Sub-center ArcFace) | Round1 | 9.735 | 0.415 | 9.747 | 0.4985
++ AS-Norm | Round1 | 9.290 | 0.389 | - | -
+++ QMF | Round1 | 8.459 | 0.397 | 8.332 | 0.449
\+ pseudo label (Sub-center ArcFace) | Round2 | 10.010 | 0.434 | - | -
2 ResNet101-ASP |  |  |  |  |
\+ pseudo label (Sub-center ArcFace) | Round1 | 8.425 | 0.385 | - | -
++ AS-Norm | Round1 | 8.275 | 0.367 | - | -
+++ QMF | Round1 | 8.065 | 0.374 | - | -
3 ResNet152-Stat |  |  |  |  |
\+ pseudo label (Sub-center ArcFace) | Round1 | 8.165 | 0.375 | - | -
++ AS-Norm | Round1 | 7.875 | 0.356 | - | -
+++ QMF | Round1 | 7.455 | 0.381 | - | -
4 ResNet152-ASP |  |  |  |  |
\+ pseudo label (Sub-center ArcFace) | Round1 | 8.335 | 0.369 | - | -
++ AS-Norm | Round1 | 8.020 | 0.347 | - | -
+++ QMF | Round1 | 7.730 | 0.365 | - | -
5 Res2Net101-Std |  |  |  |  |
\+ pseudo label (Sub-center ArcFace) | Round1 | 8.440 | 0.387 | - | -
++ AS-Norm | Round1 | 8.345 | 0.362 | - | -
+++ QMF | Round1 | 7.810 | 0.390 | - | -
6 SE-ResNet101-ASP |  |  |  |  |
\+ pseudo label (Sub-center ArcFace) | Round1 | 8.680 | 0.398 | - | -
++ AS-Norm | Round1 | 8.560 | 0.375 | - | -
+++ QMF | Round1 | 8.345 | 0.391 | - | -
7 ResNet221Stat |  |  |  |  |
\+ pseudo label (Sub-center ArcFace) | Round1 | 9.160 | 0.376 | - | -
++ AS-Norm | Round1 | 8.810 | 0.357 | - | -
+++ QMF | Round1 | 8.090 | 0.372 | - | -
8 RepVGG-ASP |  |  |  |  |
\+ pseudo label (Sub-center ArcFace) | Round1 | 8.570 | 0.382 | - | -
++ AS-Norm | Round1 | 8.445 | 0.360 | - | -
+++ QMF | Round1 | 8.270 | 0.372 | - | -
Fusion(1+2+3+4+5+6+7+8) | - | 7.000 | 0.326 | 7.153 | 0.389

Generating pseudo label by clustering. We adopt the K-means algorithm as the
clustering algorithm to generate the pseudo labels. The learning objective of
K-means is set to minimize the within-cluster sum-of-squares criterion:

$\underset{\mathbf{C}}{min}\frac{1}{N}\sum_{i=1}^{N}\underset{k}{min}\|\mathbf{z}_{i}-\mathbf{C}_{k}\|^{2}$
(1)

where $\mathbf{z}_{i}\in\mathbb{R}^{d}$ is the $d$-dimensional speaker
embedding of the $i^{th}$ sample. The cluster with the closest centroid to
$z_{i}$ in terms of the L2-norm distance is assigned as the pseudo-label for
sample i.

Determine the number of clusters. Inspired by the works of Cai et al.[4, 18],
we determine the number of clusters by the ‘elbow’ method. Given
$\mathbf{z}_{k,a}$, the assigned $a^{th}$ embedding of the $k^{th}$ cluster.
The total SSE of $N$ elements is:

${SSE}=\sum_{i=1}^{k}\sum_{a=1}^{A}\|\mathbf{z}_{k,a}-\mathbf{C}_{k}\|^{2}$
(2)

Fig. 1 shows the curve of SSE results under different $K$s. The SSE
monotonically increases as the number of clusters $K$ increases. SSE tends to
flatten with some $K$ onwards and forming an ‘elbow’ of the curve. Such
‘elbow’ indicates that the intra-cluster has little variation and is
increasingly over-fitting. From Fig. 1, the ‘elbow’ is distributed between
1600 and 2000. The specific implementation of the baseline system has been
released111https://github.com/FFSVC/FFSVC2022_Baseline_System. Therefore, the
experiment conducts with K=1600,1800,2000.

###  2.2 Training Strategy and Implement Detail

In this task, the training consists of two parts: pre-training and fine-
tuning. The pre-training stage is the implementation of the track1. We adopt
the pre-trained model used in track1. In the fine-tuning stage, different from
the fine-tuning method of the FFSVC task2 baseline, only permitted CN-Celeb
data is fed into the pre-trained speaker model. We also attempt to train with
the mix of Vox2dev together with CN-Celeb, but the performance is decline.

In contrast to task1, we adopt more speaker embedding models in this task.
Although the performance of the additional speaker model is not the best in
the VoxCeleb-O, the scenarios of target domain data are complex. We hope that
the different models can complement each other. Therefore, we add
ResNet221-Stat, RepVGG-ASP[19, 11] and SE-ResNet101-ASP models to our
candidate model list. In addition, since clustering inevitably generates noisy
labels, the ArcFace is replaced by Sub-Center ArcFace[20]. Since the data
distribution of track3 validation is the same as the evaluation set, QMF-based
score calibration is operated on the validation set. For score normalization,
20,000 utterances are randomly selected from the target domain unlabeled data
as the cohort. What’s more, consider the long-time utterances in validation
and evaluation set, we also truncate the audio for a maximum of 20s.

###  2.3 Results

Table.3 reports the system performance in track3. First, the SimAM-ResNet34
model is adopted as the baseline system. Sub-center ArcFace achieves about a
relative 10% improvement over ArcFace. Since the validation and evaluation set
have the same data distribution, score calibration and normalization could
significantly improve the system performance. It is worth noting that the
performance decays in the second round. We also conduct the second round of
fine-tuning on other models, the experimental results are consistent.
Therefore, all semi-supervised speaker embedding models only operate one round
of clustering and fine-tuning. The final submitted results achieve the 7.153%
EER in the evaluation set.

##  3 Conclusions

This paper describes the our system in Track 1 and 3. For track 3, the pseudo-
label method is introduced in this challenge to make semi-supervised domain
adaptation. To mitigate the noisy label generated by clustering, the Sub-
Center ArcFace is adopted to replace the ArcFace. After the AS-Norm and score
calibration operation, the final fusion system achieves the 7.153% EER in the
evaluation set.

## References

  * [1] W. Cai, J. Chen, J. Zhang, and M. Li, “On-the-Fly Data Loader and Utterance-Level Aggregation for Speaker and Language Recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , pp. 1038–1051, 2020.
  * [2] D. Snyder, G. Chen, and D. Povey, “MUSAN: A Music, Speech, and Noise Corpus,” _arXiv:1510.08484_.
  * [3] T. Ko, V. Peddinti, D. Povey, M. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in _Proc. ICASSP_ , 2017, pp. 5220–5224.
  * [4] W. Wang, D. Cai, X. Qin, and M. Li, “The DKU-DukeECE Systems for VoxCeleb Speaker Recognition Challenge 2020,” _arXiv.2010.12731_.
  * [5] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “ArcFace: Additive Angular Margin Loss for Deep Face Recognition,” in _Proc. CVPR_ , 2019, pp. 4685–4694.
  * [6] X. Qin, N. Li, C. Weng, D. Su, and M. Li, “Simple attention module based speaker verification with iterative noisy label detection,” in _Proc. ICASSP_ , 2022, pp. 6722–6726.
  * [7] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in _Proc. CVPR_ , 2016, pp. 770–778.
  * [8] S.-H. Gao, M.-M. Cheng, K. Zhao, X.-Y. Zhang, M.-H. Yang, and P. Torr, “Res2net: A new multi-scale backbone architecture,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 2, pp. 652–662, 2021\.
  * [9] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “x-vectors: Robust DNN Embeddings for Speaker Recognition,” in _Proc. ICASSP_ , 2018, pp. 5329–5333.
  * [10] K. Okabe, T. Koshinaka, and K. Shinoda, “Attentive Statistics Pooling for Deep Speaker Embedding,” _Proc. Interspeech_ , 2018.
  * [11] M. Zhao, Y. Ma, M. Liu, and M. Xu, “The speakin system for voxceleb speaker recognition challange 2021,” _arXiv preprint arXiv:2109.01989_ , 2021.
  * [12] J. Thienpondt, B. Desplanques, and K. Demuynck, “The idlab voxsrc-20 submission: Large margin fine-tuning and quality-aware score calibration in dnn based speaker verification,” in _Proc. ICASSP_ , 2021, pp. 5814–5818.
  * [13] P. Matějka, O. Novotný, O. Plchot, L. Burget, M. Sánchez, and J. Černocký, “Analysis of Score Normalization in Multilingual Speaker Recognition,” in _Proc. Interspeech_ , 2017.
  * [14] J. Thienpondt, B. Desplanques, and K. Demuynck, “The idlab voxceleb speaker recognition challenge 2021 system descriptionn,” _arXiv preprint arXiv:2104.02370_ , 2021.
  * [15] Y. Zheng, D. K. Pal, and M. Savvides, “Ring loss: Convex feature normalization for face recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 5089–5097.
  * [16] X. Qin, N. Li, C. Weng, D. Su, and M. Li, “Cross-age speaker verification: Learning age-invariant speaker embeddings,” _arXiv preprint arXiv:2207.05929_ , 2022.
  * [17] X. Qin, M. Li, H. Bu, S. Narayanan, and H. Li, “The 2022 far-field speaker verification challenge: Exploring domain mismatch and semi-supervised learning under the far-field scenario,” _arXiv preprint arXiv:2209.05273_ , 2022.
  * [18] D. Cai and M. Li, “The DKU-DukeECE System for the Self-Supervision Speaker Verification Task of the 2021 VoxCeleb Speaker Recognition Challenge,” _arXiv:2010.14751_.
  * [19] X. Ding, X. Zhang, N. Ma, J. Han, G. Ding, and J. Sun, “Repvgg: Making vgg-style convnets great again,” in _Proc. CVPR_ , 2021, pp. 13 733–13 742.
  * [20] J. Deng, J. Guo, T. Liu, M. Gong, and S. Zafeiriou, “Sub-center arcface: Boosting face recognition by large-scale noisy web faces,” in _Proc. ECCV_ , 2020, pp. 741–757.