# CSCD-IME: Correcting Spelling Errors Generated by Pinyin IME

Yong Hu, Fandong Meng, Jie Zhou
WeChat AI, Tencent Inc, China
<EMAIL_ADDRESS>

###### Abstract

Chinese Spelling Correction (CSC) is a task to detect and correct spelling
mistakes in texts. In fact, most of Chinese input is based on pinyin input
method, so the study of spelling errors in this process is more practical and
valuable. However, there is still no research dedicated to this essential
scenario. In this paper, we first present a Chinese Spelling Correction
Dataset for errors generated by pinyin IME (CSCD-IME), including 40,000
annotated sentences from real posts of official media on Sina Weibo.
Furthermore, we propose a novel method to automatically construct large-scale
and high-quality pseudo data by simulating the input through pinyin IME. A
series of analyses and experiments on CSCD-IME show that spelling errors
produced by pinyin IME hold a particular distribution at pinyin level and
semantic level and are challenging enough. Meanwhile, our proposed pseudo-data
construction method can better fit this error distribution and improve the
performance of CSC systems. Finally, we provide a useful guide to using pseudo
data, including the data scale, the data source, and the training strategy
111https://github.com/nghuyong/cscd-ime.

##  1 Introduction

Chinese spelling correction (CSC) is a task to detect and correct spelling
mistakes in Chinese sentences. It has always been an important Chinese NLP
task and also gradually becomes an indispensable sub-module for many
downstream tasks, such as search query correction (Gao et al., 2010), named
entity recognition (Wan et al., 2011), and machine translation (Zheng et al.,
2020).

Further exploring the process of input, unlike alphabetic languages, the input
of Chinese characters must rely on an input method (or input method editor,
commonly abbreviated IME 222https://en.wikipedia.org/wiki/Input_method). Among
all Chinese IMEs, pinyin IME
333https://en.wikipedia.org/wiki/Pinyin_input_method is the most popular one,
used by more than 97% of Chinese people (Chen and Lee, 2000; Zhang et al.,
2021). We can infer that most spelling errors are made by people when they
input through pinyin IME, such as typing the wrong pinyin, clicking the wrong
words, etc. It will be more practical and valuable to pay attention to this
scenario. Therefore, this paper focuses on spelling errors generated by pinyin
IME.

Figure 1: It is hard to repoduce the sampled spelling error from SIGHAN
(misspelling “错误” as “错勿”) by pinyin IME, no matter what input form is used.

However, there is still no professional benchmark dataset for errors generated
by pinyin IME. In this situation, previous research all use SIGHAN13-15 (Wu et
al., 2013; Yu et al., 2014; Tseng et al., 2015) as baseline datasets, but they
cannot accurately evaluate the real performances of CSC systems. (1) The main
reason is that errors in SIGHAN datasets come from mistakes in essays written
by teenage students (SIGHAN13) or Chinese as a foreign language (CFL) learners
(SIGHAN14-15), which has a big gap with errors generated by native speakers
through pinyin IME. For example, as shown in Figure 1, many errors in SIGHAN
would hardly occur if people use pinyin IME because the modern pinyin IME
always has a strong language model and would not recommend very unreasonable
candidate words. (2) In addition, sentences in SIGHAN datasets are in
traditional Chinese, and even after converting to Simplified Chinese, there is
still a big difference in the expression. (3) The data size is too small, for
example, SIGHAN13 has only 1,000 test samples, which may make the evaluation
results unreliable.

Figure 2: A real post of official media in Sina Weibo, where the “效力于” is
misspelled as ”效力与”.

Therefore, in the paper, we first present a Chinese Spelling Correction
Dataset for errors generated by pinyin IME (CSCD-IME). It contains 40,000
manually labeled data, which is ten times larger than SIGHAN datasets. As far
as we know, this is also the largest dataset for the CSC task. As shown in
Figure 2, sentences in the dataset are all real posts of official media (such
as Sina Sports) on Sina Weibo and all in Simplified Chinese. We could believe
that these errors are generated by editors in the process of typing through
pinyin IME, which is consistent with errors made by native speakers in the
real input scenario. In order to study the error distribution in depth, we
design a tagging system at the pinyin and semantic levels. The comparative
analysis shows that the error distribution of CSCD-IME is quite different from
existing datasets, with a much higher proportion of word-level errors.

In previous research, additional pseudo data is usually used to improve the
performance of CSC systems, which is often constructed by ASR based methods
(Wang et al., 2018) or confusion set based methods (Liu et al., 2021; Zhang et
al., 2021). However, these construction methods still have a gap with the real
Chinese input scene. Therefore, we propose a novel pseudo-data construction
method based on pinyin IME. Specifically, we simulate the input process
through pinyin IME and add sampled noise to generate pseudo data. In this way,
pseudo data can be constructed at scale and high quality.

In the end, we experiment with several strong CSC baseline models and perform
a series analysis. Experiments show that CMCD-IME is a challenging dataset and
our proposed pseudo-data construction method can better fit the real error
distribution than previous methods and improve the performance of CSC systems.
We also discover how to better use pseudo data, and provide a useful strategy
in the appendix, including the data scale, the data source, and the training
strategy.

Dataset | train size | dev size | test size | all size | avg len. | err. ratio | avg err./sent.
---|---|---|---|---|---|---|---
SIGHAN13 | 700 | - | 1,000 | 1,700 | 60.94 | 77.11% | 1.20
SIGHAN14 | 3,437 | - | 1,062 | 4,499 | 49.66 | 86.19% | 1.52
SIGHAN15 | 2,339 | - | 1,100 | 3,439 | 31.10 | 81.82% | 1.33
CSCD-IME | 30,000 | 5,000 | 5,000 | 40,000 | 57.42 | 46.02% | 1.09
Table 1: Data statistics, including the size of the training set, the size of
development set, the size of the test set, the size of the all dataset, the
averaged number of characters per sentence, the ratio of error sentences, and
the averaged error characters in each error sentence.

##  2 Related Work

CSC Dataset: CSC datasets can be divided into two categories according to the
user group: CFL learners and native speakers. Among them, the former has
richer data resources, including SIGHAN14-15 (Yu et al., 2014; Tseng et al.,
2015), NLPCC2018 (Zhao et al., 2018), NLPTEA2020 (Rao et al., 2020a) and YACLC
(Wang et al., 2021). Although the latter has a broader application scenario,
the related data sources are fewer, only SIGHAN13 (Wu et al., 2013) and
CTC2021 (Zhao et al., 2022). Moreover, these two datasets are very small, with
almost 1,000 test sentences. In addition, SIGHAN13 is a traditional Chinese
dataset, and CTC2021 is a Chinese grammar correction dataset. These are all
different from the real scenario of Simplified Chinese spelling error
correction.

CSC Data Augmentation: In order to make up for the lack of labeled data,
previous studies usually build additional pseudo data to increase the
performance. The mainstream method is based on the confusion set (Liu et al.,
2021; Zhang et al., 2021, 2020), the pseudo data constructed in this way is
extensive in size but relatively low in quality because of the big gap from
the true error distribution. Another relatively high-quality construction
method is based on ASR or OCR method (Wang et al., 2018). However, this method
requires additional labeled ASR or OCR data and is inconsistent with the
pinyin IME based input scenario.

CSC models: In recent years, BERT-style (Devlin et al., 2018) models have
dominated the research of the CSC task (Hong et al., 2019; Liu et al., 2022;
Zhu et al., 2022; Huang et al., 2021; Zhang et al., 2020; Bao et al., 2020;
Liu et al., 2021). However, due to the lack of large-scale and high-quality
datasets, the performance of these models is greatly limited.

##  3 CSCD-IME

In this section, we will show how to build the CSCD-IME and discover the error
distribution.

###  3.1 Data Collection

Our goal is to build a dataset that fits the real Chinese spelling error
correction scenario where most spelling errors are generated by pinyin IME. We
cleverly discovered a large-scale data source with this feature: LCSTS (Hu et
al., 2015), which is a large-scale Chinese short text summarization dataset.
Sentences in this dataset all come from posts of popular Chinese media on Sina
Weibo, such as People’s Daily. These accounts are verified on Weibo and
labeled with a blue ‘V’. There are usually special editors responsible for
editing and publishing news and information. We can infer that these editors
generally use pinyin IME to publish posts, so spelling errors in this editing
progress are generated by pinyin IME, which is consistent with our target
scenario. In addition, LCSTS is a large-scale and general dataset, including
over 2 million posts and covering multiple domains such as politics,
economics, military, movies, games, etc. Therefore, we use LCSTS as our data
source.

Tags | Description
---|---
Pinyin | same pinyin |  the pinyin of the correct is the same with of the wrong e.g. 现再(zai) -¿ 现在(zai); now
fuzzy pinyin |  the pinyin of the correct is the fuzzy pinyin of the wrong e.g. 现宅(zhai) -¿ 现在(zai); now
similar pinyin |  the edit distance of two pinyin is 1 e.g. 现砸(za) -¿ 现在(zai); now
dissimilar pinyin |  the edit distance of two pinyin is greater than 1 e.g. 现太(tai) -¿ 现在(zai); now
Semantic | entity word |  the correct word is a named entity, such as a place name e.g. 卢(lu)山 -¿ 庐(lu)山; Lushan Mountain
normal word |  the wrong word is a valid Chinese word e.g. 开心地驴友(lv, you) -¿ 开心地旅游(lv, you); travel happily
special char |  the error is a Chinese high-frequency error-prone character e.g. 开心的(de)旅游 -¿ 开心地(de)旅游; travel happily
normal char |  the wrong word is not a valid Chinese word or the length is 1 e.g. 开心地旅有(you) -¿ 开心地旅游(you); travel happily
Table 2: The tagging system at pnyin level and semantic level.

###  3.2 Data Selection

We split posts in LCSTS into sentence levels and obtain over 8 million
sentences. Obviously, it is not realistic to manually label all these
sentences, and most of the sentences are completely correct, so we use an
error detection model and rules to filter out these correct sentences.

Detection Model: Given a source sequence ${\rm\bf
X}=(x_{w_{1}},x_{w_{2}},...,x_{w_{N}})$, the detection model is to check
whether a character $w_{i}(1\leq i\leq N)$ is correct or not. We use the label
1 and 0 to mark the misspelled character and the correct character,
respectively. The detection model can be formalized as follows:

${\rm\bf y_{d}}=sigmoid(W^{T}(E({\rm\bf e})))$ (1)

where ${\rm\bf e}=(e_{w_{1}},e_{w_{2}},...,e_{w_{N}})$ is the word embedding
and ${E(*)}$ is the pre-trained encoder. The output ${\rm\bf
y_{d}}=(y_{d_{1}},y_{d_{2}},...,y_{d_{N}})$ is a probability sequence, and
$y_{d_{i}}\in(0,1)$ means the probability that character $x_{w_{i}}$ is
erroneous.

Training: We follow the successful experience (Wang et al., 2020) of the
NLPTEA2020 task (Rao et al., 2020b) and use a Chinese ELECTRA-Large
discriminator model 444https://github.com/ymcui/Chinese-ELECTRA (Clark et al.,
2020) to initialize the detection model. Following previous research, we train
the detection model on SIGHAN13-15’s training data and Wang’s pseudo data
(Wang et al., 2018) and save the best checkpoint by SIGHAN13-15’s test data.

Filtering: We then use the trained detection model to filter out correct
sentences. For the input sentence, we can obtain the error probability of each
character ${\rm\bf y_{d}}=(y_{d_{1}},y_{d_{2}},...,y_{d_{N}})$. Previous
studies have shown that the detection model will not work well on special
high-frequency characters (such as “的”, “地”, “得” etc.) because of the poor
labeling of these characters in SIGHAN (Xu et al., 2021), and is also easy to
over-check low-frequency entity words (such as person names, place names,
etc.) (Zhang et al., 2020). Therefore, we use a Chinese lexical analysis tool
(LAC) (Jiao et al., 2018) to detect these special and entity characters in the
input sentence and divide characters into three categories:
$C_{speicial},C_{entity},C_{normal}$. We then calculate the maximum error
probability of characters in each category (if the category is empty, the
maximum probability is 0), and only if all the max error probabilities of each
category are less than the corresponding threshold is the sentence considered
correct. This can be formalized as follows:

$\left\\{\begin{aligned} &max(\\{y_{d_{i}}|w_{i}\in
C_{special}\\})<\delta_{special}\\\ &max(\\{y_{d_{i}}|w_{i}\in
C_{entity}\\})<\delta_{entity}\\\ &max(\\{y_{d_{i}}|w_{i}\in
C_{normal}\\})<\delta_{normal}\end{aligned}\right.$ (2)

where $\delta_{special}$, $\delta_{entity}$ and $\delta_{normal}$ are
thresholds.

We use the above method to filter out about 91.2% of the sentences and keep
about 700,000 sentences that may have spelling errors. We randomly select
2,000 filtered sentences to check whether these sentences are really correct,
and the result shows that the accuracy is 99.2%, which is in line with our
expectations. For the rest of about 700,000 sentences, we cumulatively
selected about 50,000 random sentences for manual annotation.

###  3.3 Data Annotation

We recruit a group of educated native speakers for data annotation, and each
sentence will be annotated twice by different people to ensure the annotation
quality. Specifically, the annotator needs to detect and correct spelling
errors in the sentence and give the final correct sentence.

In order to further clear the labeling rules and reduce objections, sentences
that may be ambiguous (Zhang et al., 2022) will be discarded directly. There
are three scenarios: (1) The sentence is very unsmooth, and the annotator does
not understand the meaning of the sentence; (2) There are multiple possible
correction results for the typo in the sentence, and no unique correction
result with a pinyin edit distance less than 2 could be found; (3) The
sentence contains complex grammatical errors. Therefore, the sentence that is
not discarded during the labeling process is semantically clear and has a
unique correction result.

In the end, we obtain 40,000 labeled and qualified sentences, which make up
the CSCD-IME dataset. The whole dataset is randomly divided into the training
set, development set, and test set according to the ratio of 6:1:1.

Figure 3: The comparison of error distribution (%) at pinyin level (above) and
semantic level (below). If the proportion is less than 2%, the specific value
is not indicated in the figure.

###  3.4 Basic Statistics Analysis

As shown in Table 1, the data size of CSCD-IME is much larger than existing
datasets. In terms of the averaged sentence length and averaged errors in
sentences, CSCD-IME is closer to SIGHAN13, with a longer sentence length and
fewer errors because both SIGHAN13 and CSCD-IME are datasets for native
speakers, whereas SIGGHAN14 and SIGHAN15 are datasets for CFL learners. As for
the proportion of positive and negative samples, CSCD-IME is more balanced.

###  3.5 Error Distribution Analysis

In order to analyze the error distribution of spelling mistakes produced by
pinyin IME in depth, we first design a tagging system at pinyin level and
semantic level, and then conduct the error distribution study on CSCD-IME and
previous datasets.

Tag defination: As shown in Table 2, we design four pinyin-level and four
semantic-level tags. For pinyin-level tags, we define different tags according
to the degree of pinyin similarity between the wrong and the correct
character. In practice, we use pypinyin555https://github.com/mozillazg/python-
pinyin to obtain the pinyin info and add the pinyin-level tag. Note that for
fuzzy pnyin, we follow the definition of Sogou IME
666https://pinyin.sogou.com/help.php?list=3&q=5. As for semantic-level tags,
they can be roughly divided into two categories: word-level tags (entity word
and normal word) and character-level tags (special char and normal char). In
practice, we first seg the corrected sentence by LAC (Jiao et al., 2018) and
get the corresponding word of the wrong character. For a given word, we then
add the semantic tag based on LAC’s POS tag and an entity dictionary. Note
that the special char refers to “他/她/它/的/地/得”.

Pinyin-level analysis: As shown in the upper part of Figure 3, in all four
datasets, the same pinyin error is the largest proportion and the dissimilar
pinyin error is the smallest one. Compared with SIGHAN datasets, this feature
of CSCD-IME is more prominent, it has higher proportion of same pinyin errors
(up to 82.4%) and lower percentage of dissimilar pinyin errors (only 2.2%).
This is because pinyin IME could auto fix the pinyin typoes based on the input
context (Jia and Zhao, 2014).

Semantic-level analysis: As shown in the lower part of Figure 3, the
proportion of special char error in CSCD-IME and SIGHAN13 is lower than that
in SIGHAN14 and SIGHAN15, this means native speakers could make fewer special
char errors than CFL learners. It also can be observed that the percentage of
the word-level errors (entity word and normal word) in CSCD-IME is
significantly higher than those in SIGHAN datasets and is nearly doubled. This
is because if we input through the whole word (Figure 1(b)), pinyin IME will
generally recommend a valid word rather than a very strange “word”. Note that
the high proportion of word-level errors is the most important feature of
CSCD-IME. This is also the biggest difference with SIGHAN datasets and makes
CSCD-IME challenging. We will conduct a deep analysis between word-level and
character-level errors in Section 5.

Figure 4: The process of generating pseudo data by pinyin IME (the orange box
indicates the selected noise).

##  4 Data Augmentation

It is expensive to manually label data for the CSC task (Wang et al., 2018),
so how to construct pseudo data has always been a valuable topic. In this
section, we propose a novel method to construct large-scale and high-quality
pseudo data.

###  4.1 Data Preparation

The basic principle of pseudo-data construction is to add noise to correct
sentences and generate sentences with spelling errors, so we need to prepare
completely correct sentences first. In fact, this kind of text data is common
on the Internet, such as sentences in Wikipedia or classic books, which also
ensures that we could build large-scale dataset. In practice, we better choose
the in-domain text that is relevant to the actual application scenario.

###  4.2 Noise Simulation

Given a correct sentence, we use the pinyin IME to simulate the input of this
sentence and add noise in the input process to construct pseudo data.

We first sample a pinyin noise $\upsilon_{pinyin}$, a token granularity noise
$\upsilon_{token}$, and a number of errors noise $\upsilon_{num}$. As
discussed in Section 3.5, the $\upsilon_{pinyin}$ could be one of same pinyin,
fuzzy pinyin, similar pinyin and dissimilar pinyin, the $\upsilon_{token}$
could be one of word and character, and the $\upsilon_{num}$ is a number
greater than 0. For the convenience of expression, we use token to represent
the Chinese character or word. We can obtain the distribution of
$\upsilon_{pinyin}$, $\upsilon_{token}$, and $\upsilon_{num}$ from the prior
knowledge of the scene or the labeled data.

In the generation process, we first determine the number of errors to generate
based on the sampled $\upsilon_{num}$. For each error, as shown in Figure 4,
we randomly select a word (Figure 4(a-c)) or a character (Figure 4(d-e)) from
the correct sentence based on the sampled $\upsilon_{token}$. Then we first
type the correct text before the selected token, and enter correct (Figure
4(a)) or wrong pinyin (Figure 4(c)) of the selected token based on the sampled
$\upsilon_{pinyin}$. If the first token recommended by pinyin IME is the
correct token, we randomly select the second or the third token as noise
(Figure 4(b)). If not, we directly select the first token as noise (Figure
4(a)). Finally, we replace the correct token with the noise token in the
original sentence.

###  4.3 Language Model Filtering

It can be observed that the pseudo data generated by this method may also be
correct sentences (Figure 4(b)), so we use an n-gram language model (LM) to
perform secondary filtering. Specifically, we calculate the perplexity (PPL)
value of the generated sentence and the original sentence, and only when the
PPL value is relatively improved by $\delta$ after adding the noise do we
consider the generated noise is indeed erroneous. This can be formalized as
follows:

$\frac{PPL(noised)-PPL(origin)}{PPL(origin)}>\delta$ (3)

In practice, the value of $\delta$ is adjusted by the selected language model.
This additional step ensures that we could build a high-quality dataset.

Finally, based on the correct sentences from the LCSTS dataset (Section 3.2),
the error distribution from the CSCD-IME dataset (Section 3.4, 3.5) and the
Google pinyin IME 777https://en.wikipedia.org/wiki/Google_Pinyin, we build a
large-scale and high-quality pseudo dataset LCSTS-IME-2M, including 2 million
sentences.

Models | Sentence-level | Character-level
---|---|---
Detection | Correction | Detection | Correction
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
BERT | 79.16 | 65.83 | 71.88 | 70.55 | 58.66 | 64.06 | 83.00 | 67.01 | 74.15 | 73.59 | 59.41 | 65.75
+LCSTS-ime-2M | 78.98 | 73.60 | 76.20 | 75.63 | 70.47 | 72.96 | 82.19 | 75.75 | 78.84 | 78.84 | 72.67 | 75.63
+Wang-cs-271K | 79.76 | 59.53 | 68.17 | 74.40 | 55.54 | 63.60 | 83.99 | 59.77 | 69.84 | 77.60 | 55.22 | 64.53
+Wang-asr-271K | 78.18 | 64.39 | 70.62 | 74.01 | 60.96 | 66.86 | 81.67 | 65.23 | 72.53 | 77.07 | 61.55 | 68.44
+Wang-ime-271K | 79.72 | 64.18 | 71.11 | 76.86 | 61.88 | 68.56 | 82.9 | 65.39 | 73.11 | 79.84 | 62.98 | 70.41
*Wang-cs-271K | 37.21 | 22.10 | 27.73 | 25.22 | 14.98 | 18.80 | 39.69 | 23.22 | 29.30 | 26.50 | 15.51 | 19.57
*Wang-asr-271K | 52.27 | 44.42 | 48.03 | 43.79 | 37.21 | 40.23 | 55.87 | 46.32 | 50.65 | 46.56 | 38.61 | 42.22
*Wang-ime-271K | 52.41 | 44.33 | 48.04 | 46.87 | 39.64 | 42.95 | 56.07 | 49.72 | 52.70 | 49.69 | 44.07 | 46.71
soft-masked BERT | 80.87 | 64.78 | 71.94 | 74.42 | 59.62 | 66.20 | 84.46 | 65.35 | 73.68 | 77.50 | 59.97 | 67.62
+LCSTS-ime-2M | 79.19 | 74.86 | 76.97 | 75.75 | 71.60 | 73.62 | 82.39 | 77.93 | 80.10 | 78.63 | 74.37 | 76.44
+Wang-cs-271K | 79.74 | 62.22 | 69.9 | 75.52 | 58.92 | 66.19 | 83.33 | 62.50 | 71.43 | 78.59 | 58.94 | 67.36
+Wang-asr-271K | 79.17 | 66.52 | 72.30 | 75.66 | 63.57 | 69.09 | 83.21 | 67.64 | 74.62 | 79.47 | 64.6 | 71.26
+Wang-ime-271K | 79.96 | 66.87 | 72.83 | 77.21 | 64.57 | 70.32 | 83.57 | 68.63 | 75.37 | 80.64 | 66.22 | 72.72
*Wang-cs-271K | 37.23 | 17.54 | 23.85 | 23.87 | 11.25 | 15.29 | 34.54 | 18.55 | 24.14 | 22.02 | 11.83 | 15.39
*Wang-asr-271K | 46.74 | 38.86 | 42.44 | 41.41 | 34.43 | 37.60 | 47.32 | 41.97 | 44.49 | 42.02 | 37.26 | 39.50
*Wang-ime-271K | 56.03 | 51.63 | 53.74 | 51.55 | 47.50 | 49.45 | 59.22 | 57.40 | 58.30 | 54.69 | 53.01 | 53.84
PLOME | 79.78 | 57.23 | 66.65 | 78.09 | 56.01 | 65.23 | 83.48 | 57.99 | 68.44 | 81.49 | 56.61 | 66.81
+LCSTS-ime-2M | 81.20 | 72.21 | 76.44 | 79.05 | 70.30 | 74.42 | 84.21 | 73.81 | 78.67 | 82.00 | 71.88 | 76.60
Table 3: The performance (%) of baseline models on CSCD-IME with or without different pseudo datasets. Note that ”+” means pretraining on pseudo data and then finetuning on CSCD-IME’s training data, and ”*” means no finetuning step, evaluating directly after pretraining on pseudo data. Models | char level | word level | $\Delta$
---|---|---|---
BERT | 72.82 | 71.07 | -1.75
soft-masked BERT | 75.09 | 72.71 | -2.38
PLOME | 77.77 | 72.78 | -4.99
Table 4: The f1-score (%) of correction at character level on character-level
errors and word-level errors, and the decline of the latter compared with the
former.

##  5 Experiment

In this section, we provide the benchmark of CSCD-IME and verify the
effectiveness of our pseudo-data construction method.

###  5.1 Basic Settings

The basic experimental setup includes data, evaluation methods, and baseline
models.

Data: We use CSCD-IME as the basic dataset, including the training set,
development set, and test set. Note that for pseudo data, we conduct two-step
training: pretraining and then finetuning, using only pseudo data in the
pretraining step, and only manually labeled data in the finetuning step.

Metrics: We use the f1-score of detection and correction at sentence level and
character level as evaluation metrics. The sentence-level metrics follow the
calculation in FASPell (Hong et al., 2019). As for character-level metrics, we
do not follow the method in SpellGCN (Cheng et al., 2020). We calculate all
characters for correction metric instead of only those been correctly detected
characters.

Baselines: We choose the following three strong baseline models. BERT (Devlin
et al., 2018) directly fine-tunes the standard masked language model to
generate fixed-length corrections. Soft-Masked BERT (Zhang et al., 2020)
utilizes a detection model to help the correction model learn the right
context. PLOME (Liu et al., 2021) integrates the phonetic and visual features
into the pre-trained model, and it has an additional pretraining step on a
large-scale confusion set based pseudo dataset.

###  5.2 CSCD-IME Benchmark

In this part, we provide basic benchmark performances of CSCD-IME, including
two versions: without any pseudo data and with large-scale pseudo data LCSTS-
IME-2M.

As shown in Table 3, if we don’t add any pseudo data, the overall f1-score of
correction at character level could be only nearly 67%; when introducing 2
million pseudo data, the performance could increase a lot, but the best is
still just over 76%. There is still a big room for improvement.

We further analyze the model’s performance (without any pseudo data version)
on different semantic tags. Note that here we only analyze errors with same
pinyin to reduce the impact of pinyin.

As shown in Table 4, all three models perform worse on word-level errors than
on character-level errors, and the maximum gap could reach 5%. The iteration
of the CSC model in recent years, for example, from BERT to PLOME, reflects in
the improvement of character-level errors but no improvement in word-level
errors. For word-level errors, the model should have a strong understanding of
context and entities, so these errors are more difficult and require further
effort. Compared with existing datasets, the biggest feature of CSCD-IME is
the high proportion of word-level errors, so from this point of view, CSCD-IME
is challenging.

###  5.3 Comparition of Pseudo Datasets

In this subsection, we compare pseudo datasets constructed by different
methods. Note that PLOM already has a pretraining step on large-scale pseudo
data, so this baseline is not used here.

We choose existing ASR(/OCR)-based pseudo data (Wang et al., 2018) as
experiment dataset, including over 271K sentences, abbreviated as Wang-
asr-271K. This pseudo dataset is also the most famous one, which is always
used to improve performance in previous research. We further extract correct
sentecnes from Wang-asr-271K to construct confusion set based pseudo data
Wang-cs-271K and pinyin IME based pseudo data Wang-ime-271K.

As shown in Table 3, our proposed pinyin IME based pseudo-data construction
method is more effective than all existing methods. Compared with Wang-
asr-271K and Wang-cs-271K, adding Wang-ime-271K (ours) always obtain the best
f1-score on both sentence-level and character-level metrics. To observe the
effect of each method more clearly, if we only use pseudo data in the training
progress, our method would significantly outperform other methods. The key to
the quality of pseudo data is the error distribution of pseudo data, as shown
in Figure 6, our method can better fit the real scene. More analysis can be
checked in Appendix A.

###  5.4 Impact of LM Post-Filtering

LM threshold ($\delta$) | precision | recall | f1
---|---|---|---
w/o | 41.17 | 40.66 | 40.91
-20% | 44.52 | 49.01 | 46.66
0% | 49.69 | 44.07 | 46.71
20% | 50.64 | 26.46 | 34.76
50% | 57.52 | 9.38 | 16.12
Table 5: The correction results (%) at character level for pseudo data with
different LM filtering strategies.

In this part, we investigate the effect of LM filtering in Section 4.3. We
reconstruct Wang-ime-271K and adopt different LM filtering strategies. We
choose the basic BERT model to conduct the experiment, and only use pseudo
data in the training progress to more clearly study the differences.

As shown in Table 5, if we do not use LM filtering, more unexcepted noises
will be introduced, for example, the generated pseudo data may also be a
completely correct sentence. However, when the threshold is too low, even less
than 0, the generated errors will be more confused, and the model would have
good recall but poor precision. In contrast, if the threshold is too high, the
generated errors are relatively simple, and the precision is better, but the
recall is lower. Therefore, LM filtering is necessary and we also need to
choose a moderate threshold.

###  5.5 Error Analysis

We detailed analyze errors that can not be correctly handled in all three
baseline models and summarize these errors into the following three main
types.

The over-correction of expressions with low-frequency (Liu et al., 2022).
Since the backbone model is BERT and its pre-trained task is masked token
recovering, these CSC models tend to overcorrect valid low-frequency
expressions to more frequent expressions, such as people’s names, place names,
new Internet words, etc. For instance, “墨尔本矩形体育场” (Melbourne Rectangular
Stadium) is a location entity and is over-corrected as “墨尔本巨形体育场” (Melbourne
Mega Stadium).

The missing correction of expressions requires strong contextual reasoning
ability. If the wrong word in the sentence is a high-frequency word, the model
needs to combine the semantic information of the context to correct it. For
example, for the sentence “几辆车停在原地争执，后面的车辆也无法进程” (since several cars parked
and argued, cars behind could not enter the city), where the wrong word “进程”
(process) is a common word, but it is wrong to fill in this sentence, based on
the context, it should be “进城” (enter the city) here.

The failing correction of grammar-related errors. The CSC task also contains a
certain number of grammar-related errors, such as the use of function words,
auxiliary words, and verb-object fixed collocations, and these baseline models
cannot effectively handle such errors. For example, for the phrase “准确的传达情绪”
(convey emotions accurately), the auxiliary word between the adverb and the
verb should be “地” not “的” according to Chinese grammar rules.

##  6 Conclusion

In this paper, we focus on the most common mistakes in practical CSC
applications, i.e., spelling errors generated by pinyin IME. We first build a
new dataset CSCD-IME for this field, which is also the largest dataset for the
CSC task. Furthermore, we propose a novel method to construct large-scale and
high-quality pseudo data based on pinyin IME. A series of analyses and
experiments show that CSCD-IME holds a particular error distribution and is a
challenging dataset. Our proposed pseudo-data construction method can better
fit this real error distribution and is more efficient than all previous
methods. We hope this data resource and our discovery will advance follow-up
research.

## References

  * Bao et al. (2020) Zuyi Bao, Chen Li, and Rui Wang. 2020.  Chunk-based chinese spelling check with global optimization.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2031–2040.
  * Chen and Lee (2000) Zheng Chen and Kai-Fu Lee. 2000.  A new statistical approach to chinese pinyin input.  In _Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics_ , pages 241–247.
  * Cheng et al. (2020) Xingyi Cheng, Weidi Xu, Kunlong Chen, Shaohua Jiang, Feng Wang, Taifeng Wang, Wei Chu, and Yuan Qi. 2020.  Spellgcn: Incorporating phonological and visual similarities into language models for chinese spelling check.  _arXiv preprint arXiv:2004.14166_.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Gao et al. (2010) Jianfeng Gao, Xiaolong Li, Daniel Micol, Chris Quirk, and Xu Sun. 2010.  A large scale ranker-based system for search query spelling correction.  _international conference on computational linguistics_.
  * Hong et al. (2019) Yuzhong Hong, Xianguo Yu, Neng He, Nan Liu, and Junhui Liu. 2019.  Faspell: A fast, adaptable, simple, powerful chinese spell checker based on dae-decoder paradigm.  In _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_ , pages 160–169.
  * Hu et al. (2015) Baotian Hu, Qingcai Chen, and Fangze Zhu. 2015.  Lcsts: A large scale chinese short text summarization dataset.  _arXiv preprint arXiv:1506.05865_.
  * Huang et al. (2021) Li Huang, Junjie Li, Weiwei Jiang, Zhiyu Zhang, Minchuan Chen, Shaojun Wang, and Jing Xiao. 2021.  Phmospell: Phonological and morphological knowledge guided chinese spelling check.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5958–5967.
  * Jia and Zhao (2014) Zhongye Jia and Hai Zhao. 2014.  A joint graph model for pinyin-to-chinese conversion with typo correction.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1512–1523.
  * Jiao et al. (2018) Zhenyu Jiao, Shuqi Sun, and Ke Sun. 2018.  Chinese lexical analysis with deep bi-gru-crf network.  _arXiv preprint arXiv:1807.01882_.
  * Liu et al. (2022) Shulin Liu, Shengkang Song, Tianchi Yue, Tao Yang, Huihui Cai, TingHao Yu, and Shengli Sun. 2022.  Craspell: A contextual typo robust approach to improve chinese spelling correction.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3008–3018.
  * Liu et al. (2021) Shulin Liu, Tao Yang, Tianchi Yue, Feng Zhang, and Di Wang. 2021.  Plome: Pre-training with misspelled knowledge for chinese spelling correction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2991–3000.
  * Rao et al. (2020a) Gaoqi Rao, Erhong Yang, and Baolin Zhang. 2020a.  Overview of nlptea-2020 shared task for chinese grammatical error diagnosis.  _Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications_.
  * Rao et al. (2020b) Gaoqi Rao, Erhong Yang, and Baolin Zhang. 2020b.  Overview of nlptea-2020 shared task for chinese grammatical error diagnosis.  In _Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications_ , pages 25–35.
  * Tseng et al. (2015) Yuen-Hsien Tseng, Lung-Hao Lee, Li-Ping Chang, and Hsin-Hsi Chen. 2015.  Introduction to sighan 2015 bake-off for chinese spelling check.  _CIPS-SIGHAN Joint Conference on Chinese Language Processing_.
  * Wan et al. (2011) Xiaojun Wan, Liang Zong, Xiaojiang Huang, Tengfei Ma, Houping Jia, Yuqian Wu, and Jianguo Xiao. 2011.  Named entity recognition in chinese news comments on the web.  In _Proceedings of 5th international joint conference on natural language processing_ , pages 856–864.
  * Wang et al. (2018) Dingmin Wang, Yan Song, Jing Li, Jialong Han, and Haisong Zhang. 2018.  A hybrid approach to automatic corpus generation for chinese spelling check.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2517–2527.
  * Wang et al. (2020) Shaolei Wang, Baoxin Wang, Jiefu Gong, Zhongyuan Wang, Xiao Hu, Xingyi Duan, Zizhuo Shen, Gang Yue, Ruiji Fu, Dayong Wu, et al. 2020.  Combining resnet and transformer for chinese grammatical error diagnosis.  In _Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications_ , pages 36–43.
  * Wang et al. (2021) Yingying Wang, Cunliang Kong, Liner Yang, Yijun Wang, Xiaorong Lu, Renfen Hu, Shan He, Zhenghao Liu, Yun Chen, Erhong Yang, et al. 2021.  Yaclc: A chinese learner corpus with multidimensional annotation.  _arXiv preprint arXiv:2112.15043_.
  * Wu et al. (2013) Shih-Hung Wu, Chao-Lin Liu, and Lung-Hao Lee. 2013.  Chinese spelling check evaluation at sighan bake-off 2013.  _CIPS-SIGHAN Joint Conference on Chinese Language Processing_.
  * Xu et al. (2021) Heng-Da Xu, Zhongli Li, Qingyu Zhou, Chao Li, Zizhen Wang, Yunbo Cao, Heyan Huang, and Xian-Ling Mao. 2021.  Read, listen, and see: Leveraging multimodal information helps chinese spell checking.  _arXiv preprint arXiv:2105.12306_.
  * Yu et al. (2014) Liang-Chih Yu, Lung-Hao Lee, Yuen-Hsien Tseng, and Hsin-Hsi Chen. 2014.  Overview of sighan 2014 bake-off for chinese spelling check.  _CIPS-SIGHAN Joint Conference on Chinese Language Processing_.
  * Zhang et al. (2021) Ruiqing Zhang, Chao Pang, Chuanqiang Zhang, Shuohuan Wang, Zhongjun He, Yu Sun, Hua Wu, and Haifeng Wang. 2021.  Correcting chinese spelling errors with phonetic pre-training.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2250–2261.
  * Zhang et al. (2020) Shaohua Zhang, Haoran Huang, Jicong Liu, and Hang Li. 2020.  Spelling error correction with soft-masked bert.  _arXiv preprint arXiv:2005.07421_.
  * Zhang et al. (2022) Yue Zhang, Zhenghua Li, Zuyi Bao, Jiacheng Li, Bo Zhang, Chen Li, Fei Huang, and Min Zhang. 2022.  Mucgec: a multi-reference multi-source evaluation dataset for chinese grammatical error correction.  _arXiv preprint arXiv:2204.10994_.
  * Zhao et al. (2022) Honghong Zhao, Baoxin Wang, Dayong Wu, Wanxiang Che, Zhigang Chen, and Shijin Wang. 2022.  Overview of ctc 2021: Chinese text correction for native speakers.  _arXiv preprint arXiv:2208.05681_.
  * Zhao et al. (2018) Yuanyuan Zhao, Nan Jiang, Weiwei Sun, and Xiaojun Wan. 2018.  Overview of the nlpcc 2018 shared task: Grammatical error correction.  _international conference on natural language processing and chinese computing_.
  * Zheng et al. (2020) Renjie Zheng, Mingbo Ma, Baigong Zheng, Kaibo Liu, and Liang Huang. 2020.  Opportunistic decoding with timely correction for simultaneous translation.  _arXiv preprint arXiv:2005.00675_.
  * Zhu et al. (2022) Chenxi Zhu, Ziqiang Ying, Boyu Zhang, and Feng Mao. 2022.  Mdcspell: A multi-task detector-corrector framework for chinese spelling correction.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1244–1253.

##  Appendix A Pseudo Data Analysis

###  A.1 Error Distribution

As shown in Figure 6, we study the error distribution of pseudo data generated
by different methods at the pinyin level and semantic level. It can be
observed that our pseudo-data construction method is most consistent with
CSCD-IME, this also means that our method matches the real input scenario by
pinyin IME well. In contrast, the confusion set based method and ASR based
method have a big gap with the true error distribution.

a quick change in the rules of the game
---
O | 游戏规则的快速改变
cs | 游戏规则德快速改变
asr | 游戏规则得快速改变
ime | 游戏规则得快速改变
simple, fashionable and moderately style
O | 简约时尚的风格适中的
cs | 简约时尚的风格誓中的
asr | 简约时尚的风格是中的
ime | 简约时尚的风格始终的
and regulation is not as good as life insurance
O | 且监管也不如寿险完善
cs | 且监管也不如寿险碗善
asr |  其监管也不如寿险完善
ime | 且监管也不如寿险玩善
Table 6: The pseudo data generated by different methods. Note that O means the
origin sentence, cs, asr and ime denote the corresponding noised sentence from
Wang-cs-271K, Wang-asr-271K and Wang-ime-271K, respectively.

###  A.2 Case Study

We sample more examples in Table 6. It can be seen that the confusion set
based method may produce errors that are completely out of context, which is
far from the real input scene. ASR based method is better, but the constructed
errors are mostly character-level errors. Meanwhile, since ASR based method
doesn’t have a LM filtering module, the generated noise may also be correct,
such as the third case in Table 6. In contrast, our method could generate
high-quality pseudo data, including word-level and character-level errors.

##  Appendix B Guide to Using Pseudo Data

In this section, we study how to better use pseudo data and provide several
recommendations. Note that PLOM already has a pretraining step on large-scale
pseudo data, so this baseline is not used here.

###  B.1 How to Choose Data Source

In this part, we study the influence of data sources when we build pseudo
dataset. We randomly split a same scale sub-dataset LCSTS-ime-271K from our
constructed LCSTS-IME-2M, and then compare with Wang-ime-271K. In order to
clearly do the comparition, we only use pseudo data in the training progress.

As shown in Table 7, compared with Wang-ime-271K, adding LCSTS-ime-271K could
obtain a greater promotion. Since the data of LCSTS-ime-271K and CSCD-IME all
come from posts in Sina Weibo, they share the same data distribution, such as
language style, average length, etc.

Therefore, we better choose the data source that is consistent with the real
scenario.

Figure 5: The correction results (%) at character level with different sizes
of pseudo data, including 0, 200K, 500K, 1M and 2M.

###  B.2 How to Choose Data Size

In this subsection, we check the effect of data size. Specifically, we
randomly split CSCD-IME-2M into sub-datasets with different scales.

As shown in Figure 5, generally, with the size of the pseudo data gradually
increasing, the overall f1 score also increases. When the scale of pseudo data
is not very large, especially from 0 to 200K, the improvement is significant.
But when the scale of pseudo data continues to increase, the performance
increases slowly or even begins to decline.

Therefore, we better build a large-scale pseudo dataset, containing at least
one million sentences.

###  B.3 How to Choose Training Method

In previous studies (Liu et al., 2021; Zhang et al., 2020; Cheng et al., 2020;
Zhang et al., 2021), the pseudo data is directly trained with labeled data
together. In this subsection, we compare two training strategies: (1) directly
adding pseudo data to the training set (the popular method in previous
research) (2) pretraining on pseudo data and then finetuning on labeled data
(the method in our work). We carry out experiments on different sizes of
pseudo-data: 200K and 2M.

As shown in Table 8, compared with training together, training in two stages
can always achieve better results, especially when the size of pseudo data is
large. This is because compared with manually labeled data, pseudo data would
be noisier, the finetuning step could allow the model to better fit the true
error distribution.

Therefore, we better use the two-step training method, i.e., pretraining and
then finetuning.

Figure 6: The comparison of error distribution (%) at pinyin level (above) and semantic level (below) of different pseudo datasets. If the proportion is less than 2%, the specific value is not indicated in the figure. Models | Sentence-level | Character-level
---|---|---
Detection | Correction | Detection | Correction
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
BERT |  |  |  |  |  |  |  |  |  |  |  |
*Wang-ime-271K | 52.41 | 44.33 | 48.04 | 46.87 | 39.64 | 42.95 | 56.07 | 49.72 | 52.7 | 49.69 | 44.07 | 46.71
*LCSTS-ime-271K | 57.13 | 62.09 | 59.51 | 51.34 | 55.8 | 53.48 | 59.89 | 67.29 | 63.38 | 53.45 | 60.05 | 56.56
soft-masked BERT |  |  |  |  |  |  |  |  |  |  |  |
*Wang-ime-271K | 47.80 | 40.60 | 43.91 | 41.82 | 35.52 | 38.41 | 47.53 | 45.73 | 46.61 | 41.61 | 40.03 | 40.81
*LCSTS-ime-271K | 59.49 | 58.36 | 58.92 | 53.39 | 52.37 | 52.87 | 63.54 | 62.94 | 63.23 | 56.67 | 56.13 | 56.40
Table 7: The performance (%) of baseline models on CSCD-IME with different pseudo data sources. Models | Sentence-level | Character-level
---|---|---
Detection | Correction | Detection | Correction
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
BERT |  |  |  |  |  |  |  |  |  |  |  |
+w/o-prtrain-200K | 77.83 | 66.48 | 71.71 | 73.0 | 62.35 | 67.26 | 81.37 | 68.59 | 74.44 | 76.25 | 64.28 | 69.76
+w-prtrain-200K | 78.22 | 68.78 | 73.20 | 74.22 | 65.26 | 69.45 | 81.67 | 70.85 | 75.87 | 77.34 | 67.09 | 71.85
+w/o-prtrain-2M | 70.28 | 75.25 | 72.68 | 66.42 | 71.12 | 68.69 | 72.33 | 79.00 | 75.52 | 68.53 | 74.84 | 71.55
+w-prtrain-2M | 78.98 | 73.6 | 76.20 | 75.63 | 70.47 | 72.96 | 82.19 | 75.75 | 78.84 | 78.84 | 72.67 | 75.63
soft-masked BERT |  |  |  |  |  |  |  |  |  |  |  |
+w/o-prtrain-200K | 75.9 | 70.30 | 72.99 | 72.01 | 66.7 | 69.25 | 79.27 | 73.81 | 76.44 | 74.98 | 69.82 | 72.31
+w-prtrain-200K | 78.97 | 70.91 | 74.72 | 75.19 | 67.52 | 71.15 | 82.15 | 73.38 | 77.52 | 78.25 | 69.90 | 73.84
+w/o-prtrain-2M | 70.90 | 77.77 | 74.18 | 67.46 | 73.99 | 70.57 | 72.59 | 82.44 | 77.20 | 68.86 | 78.20 | 73.24
+w-prtrain-2M | 79.19 | 74.86 | 76.97 | 75.75 | 71.60 | 73.62 | 82.39 | 77.93 | 80.10 | 78.63 | 74.37 | 76.44
Table 8: The performance (%) of baseline models on CSCD-IME with or without
pretraining.