# Self-Supervised Text Erasing with Controllable Image Synthesis

Gangwei Jiang  University of Science and Technology of China
<EMAIL_ADDRESS>,  Shiyao Wang  Alibaba Group shiyao.wsy@alibaba-
inc.com ,  Tiezheng Ge  Alibaba Group<EMAIL_ADDRESS>,  Yuning
Jiang  Alibaba Group<EMAIL_ADDRESS>,  Ying Wei  City University
of Hong Kong<EMAIL_ADDRESS>and  Defu Lian  University of Science and
Technology of China<EMAIL_ADDRESS>
(2018)

###### Abstract.

Recent efforts on scene text erasing have shown promising results. However,
existing methods require rich yet costly label annotations to obtain robust
models, which limits the use for practical applications. To this end, we study
an unsupervised scenario by proposing a novel Self-supervised Text Erasing
(STE) framework that jointly learns to synthesize training images with erasure
ground-truth and accurately erase texts in the real world. We first design a
style-aware image synthesis function to generate synthetic images with diverse
styled texts based on two synthetic mechanisms. To bridge the text style gap
between the synthetic and real-world data, a policy network is constructed to
control the synthetic mechanisms by picking style parameters with the guidance
of two specifically designed rewards. The synthetic training images with
erasure ground-truth are then fed to train a coarse-to-fine erasing network.
To produce better erasing outputs, a triplet erasure loss is designed to
enforce the refinement stage to recover background textures. Moreover, we
provide a new dataset (called PosterErase), which contains 60K high-resolution
posters with texts and is more challenging for the text erasing task. The
proposed method has been extensively evaluated with both PosterErase and the
widely-used SCUT-Enstext dataset. Notably, on PosterErase, our unsupervised
method achieves 5.07 in terms of FID, with a relative performance of 20.9%
over existing supervised baselines.

text erasing, self-supervised, reinforcement learning

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing methodologies
Reconstruction††ccs: Computing methodologies Unsupervised learning

##  1\. Introduction

Text erasing has attracted increasing interest because of its wide range of
applications such as privacy protection (Inai et al., 2014), image/video
editing (Wu et al., 2019), and image restoration (Nakamura et al., 2017a). It
aims to erase text on the stroke level by filling it with a semantically
plausible background (Nakamura et al., 2017b; Tursun et al., 2019). Most
previous works (Nakamura et al., 2017a; Zhang et al., 2019a; Tursun et al.,
2019, 2020; Liu et al., 2020) focus on designing a high-quality network to
remove the text from natural images. For example, EnsNet (Zhang et al., 2019a)
is the first end-to-end framework to remove text at a whole image level.
MTRNet (Tursun et al., 2019) utilizes an auxiliary mask to improve the text
detection branch. EraseNet (Liu et al., 2020) presents a two-stage coarse-to-
fine network with an additional segmentation head, which achieves state-of-
the-art performance. Although they have obtained remarkable improvements,
their erasing quality strongly relies on a great amount of annotated data,
which requires significant economic and labor costs.

One important method to mitigate this issue is to use synthetic data. Gupta et
al. (Gupta et al., 2016) used an offline generation mechanism to synthesize
training image pairs. As shown in Figure 1 (a), text styles are randomly drawn
from a pre-defined style space. Then an image without the synthetic text will
be regarded as the erasure ground-truth of the image blending the synthetic
text. However, remarkable distribution divergence between synthetic text style
and existing text style in images leads to suboptimal results, as shown in
Figure 1 (b). The situation becomes even worse when being adopted for high-
resolution and complex poster images (see the last two rows of Figure 1 (b)).

Figure 1. While the SynthText (Gupta et al., 2016) is synthesized by uniform
sampling from a pre-defined space, STE learns the style distribution of the
real-world data by a policy network. Several erasing results on real-world
data are shown on the right.

To tackle this issue, two mainstream approaches are proposed. One is to
directly generate synthetic images as realistic as real-world images based on
the generative adversarial networks (Zhu et al., 2017; Hoffman et al., 2018;
Murez et al., 2018). The other is to minimize representation discrepancy
between synthetic images and real-world images for style alignment (Zoph and
Le, 2016; Brock et al., 2018; Zhang et al., 2019b). However, there are many
and diverse text styles in images for erasing, making it challenging to learn
a GAN network or directly minimize representation discrepancy (Guan et al.,
2021).

To address these issues, in this paper, we propose Self-Supervised Text
Erasing (STE) framework, which consists of two modules: image synthesis and
text erasing. In the synthesis module, in addition to generating from a
customize synthesis space as in previous work, we leverage the Maximally
Stable Extremal Regions (MSER) method (Matas et al., 2004) to extract and
replicate text regions from real-world images. The above customization and
replication can provide sufficient variety to approximate the original text
style. Then, considering the text style gap between the synthetic and real-
world data, a policy network is constructed to control the synthesis function
under the guidance of two well-designed rewards. The rewards are calculated in
terms of realistic and difficulties of the current selected style, encouraging
the synthesis function to provide samples matching the target distribution
while keeping the diversity. In the erasing module, we use a coarse-to-fine
generative model to erase the text and fill missing pixels with appropriate
textures. However, the current refinement network is incapable of accurately
recovering detailed information when the text or background is complex. So we
propose a triplet erasure loss (TEL) to solve the issue of blurry results. The
TEL explicitly enforces the refined results away from the coarse outputs while
keeping close to the ground truth. It effectively boosts the refinement
network to generate more texture details and semantics than the coarse
network. The synthesis module and erasing module are optimized alternately in
the training process. Last but not the least, we collect 60K high-resolution
poster images from the e-commerce platform to embrace more challenging
scenarios for text erasing. On the whole, our STE method gains significant
improvements compared to the supervised methods in both poster and scene text.

The contributions are summarized as follows:

  * •

We propose a novel framework (STE) for text erasing, which incorporates a
synthesis function and a policy network that can produce both unbiased and
diverse synthetic data. The controllable synthesis module ensures the
stability and unbiased training, and effectively promotes the performance of
the erasing model.

  * •

The triplet erasure loss is presented to enforce the refinement network to
generate more detailed and vivid content by pushing the results away from the
coarse ones. It successfully improves the erasing results of text in a complex
style.

  * •

Additionally, the first high-resolution poster text dataset for erasing is
constructed (PosterErase111will be released in the camera-ready version),
containing 60k images with text detection annotations. We conduct extensive
experiments on both our dataset and the public scene text dataset. Benefiting
from the improvement of data synthesis and erasing model, our method
significantly outperforms all other models.

##  2\. Related Work

Text Erasing: Early text erasing frameworks (Khodadadi and Behrad, 2012; Wagh
and Patil, 2015) usually have two stages based on traditional text detection
and image inpainting techniques while being limited to easy and single-color
text. Nakamura et al. (Nakamura et al., 2017a) initially designed a deep
encoder-decoder network to erase text patch by patch, which makes a big
success on account of the outstanding ability of deep learning. Zhang et al.
(Zhang et al., 2019a) presented an end-to-end architecture to erase text
across the full image based on the conditional generative adversarial network
(Mirza and Osindero, 2014). Inspired by the pix2pix model (Isola et al.,
2017), Tursun et al. (Tursun et al., 2019) applied text masks as auxiliary
information to achieve efficient and stable training. More recently, several
works (Liu et al., 2020; Tursun et al., 2020) improve the accuracy by
explicitly modeling a branch that predicts the text regions and build the
state-of-the-art text erasing models.

Training the above erasing models requires high-quality labeled pairs which
are expensive and inefficient (Liu et al., 2020). To address this issue,
synthesizing data from non-text images (Zhang et al., 2019a; Tursun et al.,
2020; Liu et al., 2020) has been the most common alternative method. But the
above studies primarily focus on the setting where training and test data are
both synthetic, thus leaving the issue of dataset shift unattended. In
addition, Zdenek et al. (Zdenek and Nakayama, 2020) used a separately trained
scene text detector and an inpainting network to achieve weak-supervised
learning. However, the result of this two-stage training is not favorable,
since the inpainting stage loses information around text pixels.

Text Image Synthesis: Data synthesis is an economical and efficient method for
data collection in deep learning, and it has been successfully applied in
fields such as text detection (Gupta et al., 2016), semantic segmentation (Ros
et al., 2016; Richter et al., 2016), medical analysis (Frid-Adar et al.,
2018), and so on. In text image synthesis, Wang et al. (Wang et al., 2012) and
Jaderberg et al. (Jaderberg et al., 2014) first used synthetic images in text
recognition tasks. Gupta et al. (Gupta et al., 2016) developed an efficient
engine that inserts diverse text at semantically relevant locations to
synthesize text images. In (Zhan et al., 2018), Zhan et al. took semantic
coherence, visual attention, and adaptable text appearance into consideration
achieving verisimilar text image synthesis.

A more advanced line of work resorts to the generative adversarial network
(GAN). GA-DAN (Zhan et al., 2019) synthesizes data by modeling the
representation in geometry and appearance spaces. Wu et al. (Wu et al., 2019)
and Yang et al. (Yang et al., 2020) developed an end-to-end trainable style
retention network to modify text in images. These GAN-based methods obtain
higher synthesis quality; however, they suffered from high computational
overhead and generate images with lots of artifacts when out of the training
distribution (e.g., text in shadow style).

Domain Adaptation: Domain adaptation aims to study the problem of domain
shift. The majority of existing research falls into two categories. Methods
like (Long et al., 2015; Ganin et al., 2016; Long et al., 2017; Saito et al.,
2018; Zhang et al., 2019b; Xu et al., 2019) align the distribution of a source
and target domain in the feature space. The second category aligns input
images of two domains at the pixel level. Zhu et al. (Zhu et al., 2017;
Hoffman et al., 2018) achieved the consistency of structure and semantics by
adding consistency loss on the basis of GAN models. GA-DAN (Zhan et al., 2019)
adds a multi-modal spatial learning model for the shifts in both geometry and
appearance spaces. However, the performance of these alignment methods will
decline when confronted with large variance (Guan et al., 2021). Meanwhile,
the focus on the target makes them difficult to obtain the ability outside the
distribution.

Figure 2. Our Self-supervised Text Erasing framework. Part (a) presents the
workflow of the controllable synthesis module. The policy network
$\mathcal{A}$ is optimized with the two rewards in dashed boxes and directs
the function $F$ to generate training samples. Part (b) illustrates the
learning of erasing module, while the discriminator and mask prediction part
is omitted.

##  3\. Self-supervised Text Erasing

###  3.1. Overview

Our goal is to learn a text erasing network $G$, which best erases texts in a
target domain (original texts in the real-world image). Given the annotated
images collection $\\{(I,I_{gt})\\}$ where $I$ and $I_{gt}$ denote the real-
world and corresponding annotated images, the previous supervised methods
(Zhang et al., 2019a; Tursun et al., 2020; Liu et al., 2020) train $G$ by
solving the following optimization problem:

(1) $\min_{\theta}\mathcal{L}(G_{\theta}(I),I_{gt}),$

where $\theta$ are the parameters of erasing model $G$, and $\mathcal{L}$
denotes the whole erasure loss function generally consisting of adversarial
loss $\mathcal{L}_{adv}$, reconstruction loss $\mathcal{L}_{rec}$, and so on.
But in this paper, we assume that we have only available to the unlabeled
image collection $\\{I\\}$ and study an unsupervised scenario. To train $G$,
we generate the synthetic image $I_{syn}$ on $I$ using a controllable
synthesis module, and leverage $\\{(I_{syn},I)\\}$ as the training pairs (see
examples in Fig. 2). We learn our model $G$ with the synthetic dataset
$\\{(I_{syn},I)\\}$, and evaluate it on a held-out labeled test set from the
target domain, which is disjoint from $\\{I\\}$ and is never used for
training.

Specifically, we present an overview of our self-supervised text erasing
framework in Fig. 2. It composes of two major parts: (a) the controllable
synthesis module and (b) the text erasing module. In the synthesis module,
real-world image $I$ will be processed by synthesis function $F$ (Sec. 3.2),
resulting in the synthesized image $I_{syn}\equiv F(I,s)$ with text instance
in a specified style $s$. Then, $I_{syn}$ and its corresponding original $I$
can be regarded as a pair of input and ground-truth for training the text
erasing model $G$:

(2) $\min_{\theta}\mathcal{L}(G_{\theta}(F(I,s)),I).$

Moreover, to align the source styles (synthetic text) with the target one
(original text), we use a policy network $\mathcal{A}$ to select an
appropriate style $s\equiv\mathcal{A}(I)$ for function $F$, which is
implemented by an LSTM (Hochreiter and Schmidhuber, 1997) and optimized by the
environmental feedback, including text difficult $R_{diff}$ and style
realistic reward $R_{real}$ (Sec. 3.3).

In the text erasing module, our model is built upon a two-stage coarse-to-fine
network called EraseNet (Liu et al., 2020), where the erasing model $G$ is
construct by the coarse and refinement networks. Given the synthetic image
$I_{syn}$ and a binary mask $M_{syn}$ indicating the synthesized region, the
coarse network first hallucinates a rough prediction $I_{c}$. Then, the
refinement network generates more detailed images denoted as $I_{r}$. And
finally, the result $I_{pred}$ which used for loss calculation is composited
by $I_{r}$ and $I_{syn}$ conditioned on $M_{syn}$. However, the current
refinement network is incapable of generating detailed texture when the coarse
network recovers indistinguishable content. So we propose a novel triplet
erasure loss (TEL) $\mathcal{L}_{te}$ to ensure the refined results $I_{r}$
are closer to ground truth $I$ and generate more content than the $I_{c}$
(Sec.3.4).

The synthesis and erasing modules are jointly trained in an end-to-end manner,
and thus reach better erasing learning. Specifically, the model $G$ is
optimized by gradient descent while the policy network $\mathcal{A}$ is
optimized with reinforcement learning. The whole loss function of our STE
method can be summarized as follows:

(3)
$\begin{array}[]{cl}\min_{\theta}&\mathcal{L}(G_{\theta}(F(I,\mathcal{A}_{w^{*}}(I))),I)\\\
\text{where}&w^{*}\equiv\arg\max_{w}\underset{s\sim\mathcal{A}_{w}(I)}{\mathbb{E}}R(I,s),\\\
\end{array}$

$R$ denotes the weighted sum of reward function $R_{real}$ and $R_{diff}$, and
$w$ are the parameters of the policy network $\mathcal{A}$.

###  3.2. Style-Aware Synthesis Function

The detailed procedure of our synthesis function $F$ is shown in Fig. 3. Given
an image $I$, we first utilize text detection and recognition methods (Baek et
al., 2019; Du et al., 2020) to obtain text information like position, content,
blank place, and so on. Then, we generate text with a specific style $s$ and
render it in a blank place. Different from past methods (Gupta et al., 2016;
Zhan et al., 2018) which only use a limited customization mechanism to
generate text, we propose a simple yet powerful replication mechanism to
enrich the synthesis ability. Details are as follows:

Figure 3. An example procedure of the synthesis functions $F$. Specifically,
$e_{0}$ in style $s$ denotes the choice of two mechanisms, and other elements
lead the generation of the customization method. The replication method above
fails to capture the pixel of the border around the text.

Customization mechanism: It was proposed in (Gupta et al., 2016), where the
text style is decomposed into multiple separated units. According to the
influence on the text style, the units can be roughly divided into the
following three categories:

  * •

Appearance unit: color, font, size, Gaussian blurring, alpha blending, and
Poisson blending.

  * •

Geometry unit: italic, curve, and perspective transformation.

  * •

Structure unit: shadow, border, and their components.

By selecting the operation in each unit, the style parameters $s$ are
determined and the corresponding graphic is rendered on the image (see the
output $I_{syn}$ in Fig. 3 as an example). Although the customization
mechanism can create a variety of text style, they are loosely connected to
the target styles and limited by the pre-defined parameter spaces. For
example, The text under natural light in the first line of Fig. 1 (b) cannot
be synthesized by it. To address the aforementioned issue, we first expand the
space used in (Gupta et al., 2016) to match the huge variance in different
scenarios and further propose the second replication mechanism.

Replication mechanism: It aims to synthesize samples by copying the original
text in the target distribution. To achieve this, we employ the Maximally
Stable Extremal Regions (MSER) method (Matas et al., 2004) to extract text
pixels from the original area (see the top part of Fig. 3). The MSER method
enjoys the benefit of maintaining the target text pattern which is not limited
by the space of the engine and successfully makes up for the problems incurred
by the first mechanism. Nevertheless, it may suffer from the poor extraction
of style structures, e.g., the complex art forms like border, shadow, etc (see
Fig. 3 or the supplementary material for more examples). Therefore, it is
important to balance the advantages and disadvantages of these two mechanisms.

Combining these two mechanisms, our synthesis function $F$ outshines in the
following aspects: (a) $F$ creates samples fast and automatically; (b) the
synthetic data from $F$ automatically resembles the target style by
replication, and (c) $F$ is capable of synthesizing a wide range of text
images by utilizing these two complementary mechanisms. Consequently, the
synthetic source data from $F$ cover the real target domain to the greatest
extent and thus provides the possibility to capture the variance in the target
domain. With the synthesis function $F$ built, the synthetic samples $I_{syn}$
with original image $I$ will be fed into the erasing module as a training
pair.

###  3.3. Controllable Synthesis Module

Through uniformly sampling in the large space of synthesis function $F$, we
can get an offline synthetic dataset SynthRC. But the huge variance of it
makes the naive training or previous synthetic-to-realistic methods (Zoph and
Le, 2016; Brock et al., 2018; Zhang et al., 2019b; Zhu et al., 2017; Hoffman
et al., 2018; Murez et al., 2018) harder to perform well. In this regard, we
design a controllable synthesis module to generate realistic and harder
training data in an online manner, capturing the variance in the target
domain.

####  3.3.1. Search Space

The synthesis function $F$ provides various styles, all of which constitute
the search space of the policy network. Let $s=\\{e_{1},e_{2},...,e_{N}\\}$
denote a sample in the space where the $e_{i}$ denote the $i^{th}$ style
element and $N$ is the number of elements. When each element is assigned a
value, the synthesis function will generate a unique text image (see an
example in Fig. 3). In particular, we have 20 elements, including MSER, font,
color, etc., and each has a distinct selection range. Details can be found in
the supplementary material. Consequently, multiple elements are composited
combinatorially, leading to an exponential explosion in the number of the
search space with almost $10^{5}$ candidates. Furthermore, our goal is to
generate styles for each image, resulting in an unacceptable search space.

####  3.3.2. Style Optimization via REINFORCE

The goal of the policy network is to find a suitable synthetic style for each
image in the large search space. It is a typical discrete optimization problem
since the element selection in the synthesis function are not differentiable.
So we formulate the selection as a reinforcement learning problem and apply
the REINFORCE algorithm (Williams, 1992). Concretely, the policy network is
implemented as an LSTM (Hochreiter and Schmidhuber, 1997). At each synthesis
process, the policy network will observe the input image $I$ as the state and
predict action corresponding to a discrete parameter for each element (see the
policy network part in Fig. 2). Notably, we construct our policy network as an
image-aware agent, which is different from automatic data augmentation using a
dataset-aware agent (Cubuk et al., 2019, 2020). And the objective of the
policy network is to maximize the reward function as Eq. 3, and the
optimization equation is as follows:

(4)
$\nabla_{w}\underset{s\sim\mathcal{A}_{w}}{\mathbb{E}}R(I,s)\approx\frac{1}{M*N}\sum_{m=1}^{M}\sum_{n=1}^{N}R_{m}\nabla_{w}\log
p^{n}_{m},$

where $M$ is the updating batch size, $N$ is the number of element and
$p^{n}_{m}$ represents the probability of the choice of the $n^{th}$ element
in style $s_{m}$. $R$ and $R_{m}$ denotes the reward function and the reward
value of style $m$. The training of the policy network is inspired by (Zoph
and Le, 2016; Cubuk et al., 2019; Zhang et al., 2019c), and the detailed
parameter updates can be found in the supplementary materials. Furthermore, to
accommodate the hierarchical relationship in the search space (e.g., when the
border style is not selected, the element controlling the border width does
not work), the policy network is actually optimized using a masking mechanism
as follows:

(5)
$\nabla_{w}\underset{s\sim\mathcal{A}_{w}}{\mathbb{E}}R(I,s)\approx\frac{1}{M*N}\sum_{m=1}^{M}\sum_{n=1}^{N}H^{n}_{m}R_{m}\nabla_{w}\log
p^{n}_{m},$

where $H_{m}^{n}$ denotes the hierarchical relationship value which which is
set to 1 if the element works and 0 otherwise. Details can be found in the
supplementary material.

####  3.3.3. Reward Setting

To facilitate the learning of erasing module, the generated style $s$ is
expected to meet the following requirements: (a) $s$ should be realistic and
match with the target distribution; (b) $s$ should be challenging for the
current model, providing informative gradient; and (c) $s$ should not be
extremely difficult, e.g., the style that generates irregular samples is not
recommended. Based on these guidelines, we designed the following rewards:

Style realistic reward: To capture the target distribution, we implement a
text discriminator $D_{text}$ to guide the data synthesis (see Fig. 2).
Specifically, $D_{text}$ is built to predict the domain of the text region in
the synthetic image $I_{syn}$. This module takes the feature map $g(I_{syn})$
of the generator $G$ as input and is compatible with any generative model
plugged in.

The key to the style of realistic discriminator design is to configure the gap
between the synthetic text on the training set and the original text on the
validation set collected from the target distribution. So we apply an
adversarial mechanism here. In the training phase, the text discriminator
$D_{text}$ aims to distinguish between the original and synthetic texts,
predicting the original text as 1 and other regions as 0. On the contrary, in
the reward calculation phase, the closer the prediction of the synthetic text
area is to 1, the greater the reward. Then the reward is formulated as (also
check $R_{real}$ branch in Fig. 2):

(6) $R_{real}(I,s)=-\left[L_{dice}(D_{text}(g(I_{syn})),M_{syn})\right],$

where $L_{dice}$ is the dice loss (Milletari et al., 2016) for mask image and
$M_{syn}$ is the mask of synthetic text region. The realistic reward
discourages the style shifts and ensures that the data synthesized is within
the same distribution of target data.

Text difficult reward: Inspired by the success of adversarial training in
distributionally robust optimization (Sinha et al., 2017; Heinze-Deml and
Meinshausen, 2017; Volpi et al., 2018), we consider increasing the training
loss of the erasing model with hard and diverse synthetic samples. Training by
the cases with larger losses guarantees robust performance against data
distributions. Meanwhile, to reduce the risk of training collapse, we refer to
the approach in (Mao et al., 2017; Li et al., 2020), and finally propose a
text difficult reward as follows (also check $R_{diff}$ branch in Fig. 2):

(7) $R_{diff}(I,s)=-\left|1-exp\left[L-\alpha L_{mean}\right]\right|,$

where $L$ is the L1-distance between output image $I_{pred}=G(F(I,s))$ and
original image $I$, and $L_{mean}$ is the exponential average of historical
loss. During training, $\alpha>1$ controls the difficult level of generated
samples, constraining that the value of $L$ stays in a certain range, i.e.,
$\alpha L_{mean}$.

Finally, the REINFORCE is employed to update the policy network using the
weighted sum of the above rewards, while $R_{real},R_{diff}$ are normalized.
In practice, we update the policy network $\mathcal{A}$ for every certain
number of model iterations, which makes the time overhead caused by the guided
synthesis module negligible.

(8) $R=\alpha_{1}R_{real}+\alpha_{2}R_{diff},$

###  3.4. Erasing Module with Triplet Erasure Loss

In this section, we will discuss how our erasing module and the newly designed
loss work. The model in STE aims to explore the in-variance between target and
source, to build the erasing ability of real-world data. However, as the
labels in the target domain are not given, they may still suffer from erasure
blurring, even when adopting the coarse-to-fine generator (see in Fig. 2). The
reason likely lies in that the current losses do not differentiate outputs of
the two stages, causing the model to learn similar representations, which
leads to tiny changes or fuzzy textures in the outcomes.

Inspired by the recent progress of contrastive learning, we propose our novel
triplet erasure loss (TEL) to explicitly enforce the refinement network to
learn different representations from the coarse network. It can be formulated
as:

(9)
$\mathcal{L}_{te}=\frac{||I_{r}-I||_{1}}{||I_{r}-I||_{1}+||I_{r}-I_{c}||_{1}^{\gamma}},$

where $I,I_{c},I_{r}$ are the label image, coarse output, and refined output,
respectively. $\gamma$ is the scaling factor that emphasizes the impact of
hard samples. Intuitively, TEL ensures the refined results are closer to
ground truth and generates better images than the coarse results. Minimizing
TEL addresses the blurring erasure problem from two perspectives. The first is
that minimizing the term $||I_{r}-I_{c}||_{1}$ directly constrains the refined
image standing close to its corresponding ground truth. The second is that it
guides the refinement network to take a further leap from $I_{c}$ and
generates more detailed results by maximizing the distance between them.
Consequently, the loss adds a pushing force from the ”negative samples”
$I_{c}$, and the ground truth image provides an optimal direction to pull
$I_{r}$ close to it, making the refinement network easier to converge.
Notably, these two objectives mutually improve via optimization.

The training process for the erasing module has been described in Sec 3.1.
Unlike the previously supervised learning (Zhang et al., 2019a; Liu et al.,
2020), our training data $\\{(I_{syn},I)\\}$ contains both labeled synthetic
text and unlabeled target text in the same image. So when calculating losses,
our model is expected to focus only on the synthetic text region. Thus, we
combine the output image $I_{r}$ and input image $I_{syn}$ conditioned on the
synthetic mask $M_{syn}$ to create our final prediction $I_{pred}$:

(10) $I_{pred}=M_{syn}*I_{syn}+(1-M_{syn})*I_{r}.$

Specifically, $M_{syn}$ is generated by the synthesis function without any
prediction.

Finally, we adopt the adversarial loss $\mathcal{L}_{adv}$, reconstruction
loss $\mathcal{L}_{rec}$, perceptual loss $\mathcal{L}_{perc}$, style loss
$\mathcal{L}_{sty}$, mask refine loss $\mathcal{L}_{m}$, and triplet erasure
loss $\mathcal{L}_{te}$ as the learning objectives. The whole loss function
for erasing model can be summarized as follows:

(11)
$\begin{array}[]{cc}\mathcal{L}(I_{pred},I)=&\lambda_{1}\mathcal{L}_{adv}+\lambda_{2}\mathcal{L}_{rec}+\lambda_{3}\mathcal{L}_{perc}\\\
&+\lambda_{4}\mathcal{L}_{sty}+\lambda_{5}\mathcal{L}_{m}+\lambda_{6}\mathcal{L}_{te},\end{array}$

where $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, $\lambda_{4}$,
$\lambda_{5}$, and $\lambda_{6}$ are trade-off hyper-parameters.

Figure 4. Qualitative results for the PosterErase (the first two rows) and
SCUT-Enstext (the latter two rows) based on EraseNet. More cases are included
in the supplementary material.

##  4\. Experiment

In this section, we conduct experiments on PosterErase and SCUT-Enstext (Liu
et al., 2020) datasets to investigate the erasing quality, synthesis ability,
and robustness of our self-supervised text erasing method. Unless otherwise
specified, the reported results below are the performance tested on the
PosterErase dataset.

###  4.1. Dataset

SCUT-Enstext: It is a widely used benchmark for scene text erasing collected
by (Liu et al., 2020). It contains 2,748 training images and 813 test images
with their annotations. Since our method is a self-supervised framework, it
can benefit from increased amounts of unlabeled text images. So we adopt
10,166 scene text images without annotations from ICDAR2019-Art dataset (Chng
et al., 2019) as auxiliary information. All the images are distinct and
resized to $512\times 512$.

PosterErase: The dataset is collected on the e-commerce platform and consists
of posters mainly with Chinese text. The dataset contains 60,000 training
images and 400 test images. Each image is released with its text information,
including the bounding box and text content. We also provide the annotated
images for the test set, carefully processed by human experts using Adobe
Photoshop (PS). Through the test dataset, we can quantitatively evaluate the
quality of the model. Each image is in the size of $750\times 513$ and is
resized to $768\times 512$ as the model input.

###  4.2. Experimental Settings & Evaluation Metrics

We use Adam (Kingma and Ba, 2014) to train models, where $\beta_{1}=0.5$ and
$\beta_{2}=0.9$. The learning rate is set to 0.0001, 0.00001, and 0.00005 in
the generator, discriminator, and policy network, respectively. We set the
batch size to 32. In the synthesis function, we adapt the CRAFT (Baek et al.,
2019) and PaddleOCR (Du et al., 2020) model to detect and recognize text. The
policy network receives the image representations extracted by VGG-16 as
input. It is implemented by a two-layer LSTM and updated every 150 batches of
erasing network updating. In each update process, 100 images in the training
set will be used to calculate the reward for text styles. For the construction
of the erasing network, we followed the settings in the papers (Liu et al.,
2020), and the experimental results are obtained from the same pre-trained
initial point.

Three commonly used image generation metrics are used to evaluate the quality
of the method: PSNR, SSIM and, FID. PSNR describes the error between pixels,
SSIM (Wang et al., 2004) evaluates the structural similarity of the two
images, and FID (Heusel et al., 2017) compares the quality of the generated
pictures from the feature level. The larger PSNR, SSIM, and the smaller FID
symbolize the higher the quality of model generation. In addition, we also
provide the visualization results (Fig. 4) to show the model’s effect
directly.

Table 1. Comparison with baselines in PosterErase and SCUT-Enstext dataset. Higher PSNR, SSIM, and Lower FID is better. | PosterErase | SCUT-Enstext (Liu et al., 2020)
---|---|---
SSIM($\uparrow$) | PSNR($\uparrow$) | FID($\downarrow$) | SSIM($\uparrow$) | PSNR($\uparrow$) | FID($\downarrow$)
Text Erase | Pix2pix (Isola et al., 2017) | 0.9160 | 26.828 | 7.672 | 0.5343 | 23.865 | 17.05
EnsNet (Zhang et al., 2019a) | 0.9399 | 30.196 | 7.409 | 0.8245 | 30.785 | 6.548
MtrNet++ (Tursun et al., 2020) | 0.8326 | 26.921 | 10.11 | 0.8754 | 30.982 | 7.379
EraseNet (Liu et al., 2020) | 0.9389 | 34.034 | 6.411 | 0.8844 | 32.092 | 5.567
SceneTextErase (Zdenek and Nakayama, 2020) | 0.8165 | 28.990 | 11.27 | 0.8758 | 32.307 | 6.260
Synthetic to Realistic | SynthText (Gupta et al., 2016) | 0.9363 | 32.042 | 6.599 | 0.8796 | 31.839 | 6.069
DANN (Ganin et al., 2016) | 0.9433 | 35.728 | 5.625 | 0.8837 | 32.939 | 5.395
PCD (Yue et al., 2019) | 0.9457 | 35.601 | 5.622 | 0.8823 | 32.751 | 5.542
AFN (Xu et al., 2019) | 0.9463 | 36.098 | 5.269 | 0.8827 | 32.650 | 5.675
AA (Zhang et al., 2019c) | 0.9353 | 33.477 | 6.219 | 0.8812 | 32.669 | 5.928
Ours | SynthRC | 0.9418 | 34.707 | 5.945 | 0.8829 | 32.765 | 5.480
STE | 0.9548 | 37.249 | 5.070 | 0.8867 | 33.198 | 4.990
STE+Finetune | 0.9648 | 39.914 | 4.093 | 0.8915 | 34.139 | 4.318

###  4.3. Comparison with the state-of-the-Art

In this subsection, we show the performance of the related methods in Table 1
and Figure 4. The baseline methods can be divided into two groups: supervised
text erasing models and synthetic-to-real adaptation methods.

  * •

Supervised text erasing models: the first includes the various
supervised/weak-supervised erasing models, including Pix2pix (Isola et al.,
2017), EnsNet (Zhang et al., 2019a), MtrNet++ (Tursun et al., 2020), EraseNet
(Liu et al., 2020) and SceneTextErase (Zdenek and Nakayama, 2020). They are
trained on the annotated data and then tested. For the PosterErase dataset
which only contains 150 training pairs, we take the well-trained model on
SCUT-Enstext as initialization and achieve better results.

  * •

Synthetic-to-real adaptation models: the second type focuses on the
utilization of synthetic data. SynthText is the offline synthetic dataset
proposed by (Gupta et al., 2016), while DANN (Zoph and Le, 2016), PCD (Yue et
al., 2019), AFN (Xu et al., 2019), and AA (Zhang et al., 2019c) are a series
of methods dealing with dataset shift. Specifically, DANN and PCD capture the
consistency of features between domains, AFN adopts feature-norm across
domains, and AA applies adversarial data augmentation. These methods are
trained on the synthetic dataset SynthRC and then tested on real-world images.

The hyper-parameters of each baseline are carefully designed to ensure the
best possible results.

We compare Self-supervised Text Erasing to the above baselines on both
PosterErase and SCUT-Enstext datasets. Specifically, SynthRC is the offline
synthesis method proposed in this paper which is based on replication and
customization mechanism. The experimental results are reported in Table 1. The
key finding is that the STE algorithm achieved excellent erasing performance
without annotated data by effectively solving the inconsistency between the
synthetic samples and the target data while providing diversity. Other
findings include the following:

Quantitative evaluation: Firstly, our algorithm outperforms the state-of-the-
art supervised methods in text erasing, with the improvements on FID of 20.9%
and 10.3% in PosterErase and SCUT-Enstext. This is due to the fact that the
supervised models are learned by limited labeled data, while STE provides the
method for introducing richer training materials. Second, when compared with
the algorithms in the synthetic-to-realistic topic, our method still achieves
significant improvements on FID of 3.8% and 7.5% in two datasets. By
regulating the synthesis process, our method could match the variance in
target from the large space of synthesis function and provide generation
ability to out-of-distribution images. Thus, the erasing model can be trained
stably and fast, and perform well in the target scene. Finally, we tried to
evaluate our STE in a semi-supervised setup by unsupervised pretraining
followed by finetuning using a small number of labeled samples. The result in
the last raw of Table 1 demonstrates that our self-supervised model with
massive unlabeled data provide an effective prior for model initialization and
help to yield better performing.

Quantitative comparison: We also present several erasing examples on both
datasets in Fig. 4. The erasing results of EraseNet (Liu et al., 2020) and
DANN (Ganin et al., 2016) are compared to our outcomes as they have the best
erasing performance in the two types of baselines. It is evident that STE
generates the most visually appealing results with the fewest noticeable
artifacts. More findings are as follows. First, the model from supervised
training(i.e., EraseNet in the figure) performs poorly in detecting the text
area and completing the background. This is primarily due to the limited scale
of training data, which makes the model difficult in generalizing to new
cases. Second, the DANN lacks the capacity to process special style texts
(e.g., the shadow on the case in the second row of the first column and the
border on the cases in the first to second rows of the second column have not
been erased), but our method improves the erasing results of these images by
altering the synthetic data distribution.

###  4.4. Ablation Study

To validate the effectiveness of each component, we conduct comprehensive
ablation studies in this section. The majority of the experiments were
performed on the PosterErase.

Figure 5. Qualitative analysis of triplet erasure loss on both PosterErase
(the first row) and SCUT-Enstext (the second row) datasets.

Triplet erasure loss: We employ triplet erasure loss $\mathcal{L}_{te}$ to
help refinement network in producing clear texture. To demonstrate its
superiority, we compare it with the baseline without using $\mathcal{L}_{te}$.
The numerical results in Table 2 show that with the help of TEL (the $\lambda$
in $\mathcal{L}_{te}$ is set to 2 to get the best performance), the FID metric
drops considerably. In addition, the qualitative study in Fig. 5 shows that
triplet erasure loss recovers more details, which further benefits the quality
of the erasing results.

Table 2. Ablation study on triplet erasure loss (TEL).  | PosterErase | SCUT-Enstext
---|---|---
| psnr | fid | psnr | fid
w/o $\mathcal{L}_{te}$ | 36.796 | 5.277 | 32.947 | 5.264
with $\mathcal{L}_{te}$ | 37.249 | 5.070 | 33.198 | 4.990

Effectiveness of style optimization components: Table 3 shows the impact of
each component in the optimization process of the controllable synthesis
module. We study three variants: 1) using only realistic reward $R_{real}$; 2)
using both two rewards $R_{real}$ and $R_{diff}$; 3) using the image as the
environmental state in REINFORCE. Experiments show that the complete
optimization design has gained 2.09 improvements on PSNR. We also find that
$R_{real}$ helps in finding the style in target distribution, whereas
$R_{diff}$ is more likely to yield complex samples, and the final reward
function effectively combines the benefits of both. Here $R_{real}$ takes the
greatest role in guiding the synthesis of training data. On the other hand,
considering the image as a state enriches the ability of the synthesis module
to model text styles, capturing the personalized demands, and hence improves
the synthesis performance.

Table 3. The influence of the component in the optimization of the controllable synthesis module. Image-aware indicates that the optimization takes the image as the state.  $R_{match}$ | $R_{diff}$ | $image-aware$ | PSNR($\uparrow$) | FID($\downarrow$)
---|---|---|---|---
|  |  | 34.71 | 5.946
$\checkmark$ |  |  | 36.07 ($\uparrow$1.36) | 5.612 ($\downarrow$0.334)
$\checkmark$ | $\checkmark$ |  | 36.29 ($\uparrow$1.58) | 5.483 ($\downarrow$0.463)
$\checkmark$ | $\checkmark$ | $\checkmark$ | 36.80 ($\uparrow$2.09) | 5.277 ($\downarrow$0.669)
Figure 6. Visualization results in the synthesis probabilities of style
elements. The horizontal axis shows the elements in the search space. The
vertical axis shows the proportion of each parameter in the corresponding
element.

Text style output by controllable synthesis module: A major contribution of
our work is to guide the synthesis function to automatically produce realistic
and harder training pairs. Therefore, we visualized the synthesis
probabilities of each element in the output text style. The probability value
is obtained by counting the element selection of 2,000 images under the
guidance of the policy network. The results are shown in Fig. 6. The parameter
choices of each element are represented by ”choice0-11”, where the detailed
information is listed in the supplementary material. Generally, the higher the
number, the deeper the degree of operation. For example, ”Alpha0-3” represents
0-30% of the background involved during alpha blending. We can see that there
is a significant difference in the elements of gaussian, alpha, curve, and
perspective between the two datasets, which is congruent with our common sense
of natural and poster scenarios. The results reveal that our controllable
synthesis module is effective in resolving the dataset shift issue.

Robustness to different style spaces: We also check the performance of the STE
in different spaces of the synthesis function $F$ to confirm its
effectiveness. The results are summarized in Fig. 7. We construct four pre-
defined spaces and detailed information is available in the supplemental
material. As the style space rises, the variance in the source domain increase
and the performance of baselines improves and then declines. Instead, our STE
continually maintains efficiency and significantly outperforms them. Here, the
drop of SynthRC on space2, 3 is related to the introduction of the
”Perspective” element which rarely occurs in the poster scene.

Figure 7. The performance of STE and baselines on different style spaces on PosterErase. The space expands with the increase of the serial number. Table 4. Ablation study on elements of the synthesis function. Bas., Rep. and Cus. are abbreviations of basic, replication, and customization respectively. The reported metrics is PSNR.  Bas. | Rep. | Cus. | PosterErase | SCUT-Enstext
---|---|---|---|---
$\checkmark$ |  |  | 27.518 | 31.887
$\checkmark$ | $\checkmark$ |  | 31.110 | 32.841
$\checkmark$ | $\checkmark$ | $\checkmark$ | 34.707 | 32.765

Analysis on the synthesis function: We demonstrate the effectiveness of each
component in the synthesis function (also see Sec. 3.2), and the results are
reported in Table 4. By adding the replication mechanism, and the
customization mechanism in the synthesis function, the performance of the
trained model can raise from 27.518 to 34.707. Specifically, they are all
trained without the guides of the policy network. Such a result shows that
with the various text styles in the target task that can be effectively
included in the synthesis function, higher performance of the model training
can be achieved. Notably, replication works better in natural scenes. This is
because the text changes of natural scenes are concentrated on appearance and
geometry, which can be well copied by MSER.

##  5\. Conclusion

In this paper, we propose a novel framework, named Self-supervised Text
Erasing (STE), to learn the generated training image pairs in an unsupervised
fashion for the text erasing task. Specifically, our proposed STE method is
composed of two newly developed modules: text synthesis and text erasing. The
synthesis module is responsible for fast generating training samples as well
as learning a policy network for steering the synthesis process by selecting
more realistic and harder training data. The text erasing module employs a so-
called triplet erasure loss to better recover background textures. The two
modules are alternately updated in the training process. We collect 60K high-
resolution poster images from the e-commerce platform to embrace more
challenging scenarios for text erasing. Extensive experiments clearly show
that our proposed unsupervised STE method outperforms existing supervised
baselines, which demonstrates the effectiveness of the text synthesis and
erasing strategy in our STE.

## References

  * (1)
  * Baek et al. (2019) Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. 2019.  Character region awareness for text detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 9365–9374.
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018\.  Large scale GAN training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ (2018).
  * Chng et al. (2019) Chee Kheng Chng, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, Junyu Han, Errui Ding, et al. 2019\.  Icdar2019 robust reading challenge on arbitrary-shaped text-rrc-art. In _2019 International Conference on Document Analysis and Recognition (ICDAR)_. IEEE, 1571–1576.
  * Cubuk et al. (2019) Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. 2019.  Autoaugment: Learning augmentation strategies from data. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 113–123.
  * Cubuk et al. (2020) Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. 2020\.  Randaugment: Practical automated data augmentation with a reduced search space. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_. 702–703.
  * Du et al. (2020) Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, et al. 2020\.  Pp-ocr: A practical ultra lightweight ocr system.  _arXiv preprint arXiv:2009.09941_ (2020).
  * Frid-Adar et al. (2018) Maayan Frid-Adar, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. 2018.  Synthetic data augmentation using GAN for improved liver lesion classification. In _2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018)_. IEEE, 289–293.
  * Ganin et al. (2016) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016.  Domain-adversarial training of neural networks.  _The journal of machine learning research_ 17, 1 (2016), 2096–2030.
  * Guan et al. (2021) Dayan Guan, Jiaxing Huang, Shijian Lu, and Aoran Xiao. 2021\.  Scale variance minimization for unsupervised domain adaptation in image segmentation.  _Pattern Recognition_ 112 (2021), 107764.
  * Gupta et al. (2016) Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. 2016\.  Synthetic data for text localisation in natural images. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2315–2324.
  * Heinze-Deml and Meinshausen (2017) Christina Heinze-Deml and Nicolai Meinshausen. 2017.  Conditional variance penalties and domain shift robustness.  _arXiv preprint arXiv:1710.11469_ (2017).
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017\.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ 30 (2017).
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Hoffman et al. (2018) Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. 2018\.  Cycada: Cycle-consistent adversarial domain adaptation. In _International conference on machine learning_. PMLR, 1989–1998.
  * Inai et al. (2014) Kohei Inai, Mårten Pålsson, Volkmar Frinken, Yaokai Feng, and Seiichi Uchida. 2014\.  Selective concealment of characters for privacy protection. In _2014 22nd International Conference on Pattern Recognition_. IEEE, 333–338.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017.  Image-to-image translation with conditional adversarial networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1125–1134.
  * Jaderberg et al. (2014) Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014.  Synthetic data and artificial neural networks for natural scene text recognition.  _arXiv preprint arXiv:1406.2227_ (2014).
  * Khodadadi and Behrad (2012) Mohammad Khodadadi and Alireza Behrad. 2012.  Text localization, extraction and inpainting in color images. In _20th Iranian Conference on Electrical Engineering (ICEE2012)_. IEEE, 1035–1040.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Li et al. (2020) Ruihui Li, Xianzhi Li, Pheng-Ann Heng, and Chi-Wing Fu. 2020\.  Pointaugment: an auto-augmentation framework for point cloud classification. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6378–6387.
  * Liu et al. (2020) Chongyu Liu, Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Yongpan Wang. 2020\.  EraseNet: End-to-End Text Removal in the Wild.  _IEEE Transactions on Image Processing_ 29 (2020), 8760–8775.
  * Long et al. (2015) Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. 2015\.  Learning transferable features with deep adaptation networks. In _International conference on machine learning_. PMLR, 97–105.
  * Long et al. (2017) Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. 2017\.  Deep transfer learning with joint adaptation networks. In _International conference on machine learning_. PMLR, 2208–2217.
  * Mao et al. (2017) Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. 2017\.  Least squares generative adversarial networks. In _Proceedings of the IEEE international conference on computer vision_. 2794–2802.
  * Matas et al. (2004) Jiri Matas, Ondrej Chum, Martin Urban, and Tomás Pajdla. 2004\.  Robust wide-baseline stereo from maximally stable extremal regions.  _Image and vision computing_ 22, 10 (2004), 761–767.
  * Milletari et al. (2016) Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. 2016\.  V-net: Fully convolutional neural networks for volumetric medical image segmentation. In _2016 fourth international conference on 3D vision (3DV)_. IEEE, 565–571.
  * Mirza and Osindero (2014) Mehdi Mirza and Simon Osindero. 2014.  Conditional generative adversarial nets.  _arXiv preprint arXiv:1411.1784_ (2014).
  * Murez et al. (2018) Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim. 2018\.  Image to image translation for domain adaptation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4500–4509.
  * Nakamura et al. (2017a) Toshiki Nakamura, Anna Zhu, Keiji Yanai, and Seiichi Uchida. 2017a.  Scene Text Eraser. In _2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)_ , Vol. 01. 832–837.  https://doi.org/10.1109/ICDAR.2017.141
  * Nakamura et al. (2017b) Toshiki Nakamura, Anna Zhu, Keiji Yanai, and Seiichi Uchida. 2017b.  Scene text eraser. In _2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)_ , Vol. 1. IEEE, 832–837.
  * Richter et al. (2016) Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. 2016.  Playing for data: Ground truth from computer games. In _European conference on computer vision_. Springer, 102–118.
  * Ros et al. (2016) German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. 2016\.  The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3234–3243.
  * Saito et al. (2018) Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. 2018.  Maximum classifier discrepancy for unsupervised domain adaptation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3723–3732.
  * Sinha et al. (2017) Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. 2017.  Certifying some distributional robustness with principled adversarial training.  _arXiv preprint arXiv:1710.10571_ (2017).
  * Tursun et al. (2020) Osman Tursun, Simon Denman, Rui Zeng, Sabesan Sivapalan, Sridha Sridharan, and Clinton Fookes. 2020\.  MTRNet++: One-stage mask-based scene text eraser.  _Computer Vision and Image Understanding_ 201 (2020), 103066.
  * Tursun et al. (2019) Osman Tursun, Rui Zeng, Simon Denman, Sabesan Sivapalan, Sridha Sridharan, and Clinton Fookes. 2019\.  Mtrnet: A generic scene text eraser. In _2019 International Conference on Document Analysis and Recognition (ICDAR)_. IEEE, 39–44.
  * Volpi et al. (2018) Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, and Silvio Savarese. 2018\.  Generalizing to unseen domains via adversarial data augmentation.  _arXiv preprint arXiv:1805.12018_ (2018).
  * Wagh and Patil (2015) Priyanka Deelip Wagh and DR Patil. 2015.  Text detection and removal from image using inpainting with smoothing. In _2015 International Conference on Pervasive Computing (ICPC)_. IEEE, 1–4.
  * Wang et al. (2012) Tao Wang, David J Wu, Adam Coates, and Andrew Y Ng. 2012\.  End-to-end text recognition with convolutional neural networks. In _Proceedings of the 21st international conference on pattern recognition (ICPR2012)_. IEEE, 3304–3308.
  * Wang et al. (2004) Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004\.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ 13, 4 (2004), 600–612.
  * Williams (1992) Ronald J Williams. 1992\.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine learning_ 8, 3 (1992), 229–256.
  * Wu et al. (2019) Liang Wu, Chengquan Zhang, Jiaming Liu, Junyu Han, Jingtuo Liu, Errui Ding, and Xiang Bai. 2019.  Editing text in the wild. In _Proceedings of the 27th ACM international conference on multimedia_. 1500–1508.
  * Xu et al. (2019) Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. 2019\.  Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 1426–1435.
  * Yang et al. (2020) Qiangpeng Yang, Jun Huang, and Wei Lin. 2020.  Swaptext: Image based texts transfer in scenes. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 14700–14709.
  * Yue et al. (2019) Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni-Vincentelli, Kurt Keutzer, and Boqing Gong. 2019.  Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2100–2110.
  * Zdenek and Nakayama (2020) Jan Zdenek and Hideki Nakayama. 2020.  Erasing scene text with weak supervision. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 2238–2246.
  * Zhan et al. (2018) Fangneng Zhan, Shijian Lu, and Chuhui Xue. 2018.  Verisimilar image synthesis for accurate detection and recognition of texts in scenes. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 249–266.
  * Zhan et al. (2019) Fangneng Zhan, Chuhui Xue, and Shijian Lu. 2019.  Ga-dan: Geometry-aware domain adaptation network for scene text detection and recognition. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 9105–9115.
  * Zhang et al. (2019a) Shuaitao Zhang, Yuliang Liu, Lianwen Jin, Yaoxiong Huang, and Songxuan Lai. 2019a.  Ensnet: Ensconce text in the wild. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 801–808.
  * Zhang et al. (2019c) Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong. 2019c.  Adversarial autoaugment.  _arXiv preprint arXiv:1912.11188_ (2019).
  * Zhang et al. (2019b) Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. 2019b.  Bridging theory and algorithm for domain adaptation. In _International Conference on Machine Learning_. PMLR, 7404–7413.
  * Zhu et al. (2017) Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017.  Unpaired image-to-image translation using cycle-consistent adversarial networks. In _Proceedings of the IEEE international conference on computer vision_. 2223–2232.
  * Zoph and Le (2016) Barret Zoph and Quoc V Le. 2016.  Neural architecture search with reinforcement learning.  _arXiv preprint arXiv:1611.01578_ (2016).