11institutetext: NAVER Corp 11email: {hyeon-su.lee,
<EMAIL_ADDRESS>
# The Surprisingly Straightforward Scene Text Removal Method With Gated
Attention and Region of Interest Generation: A Comprehensive Prominent Model
Analysis

Hyeonsu Lee 11 0000-0002-6317-9883 Chankyu Choi 11 0000-0002-9166-2100

###### Abstract

Scene text removal (STR), a task of erasing text from natural scene images,
has recently attracted attention as an important component of editing text or
concealing private information such as ID, telephone, and license plate
numbers. While there are a variety of different methods for STR actively being
researched, it is difficult to evaluate superiority because previously
proposed methods do not use the same standardized training/evaluation dataset.
We use the same standardized training/testing dataset to evaluate the
performance of several previous methods after standardized re-implementation.
We also introduce a simple yet extremely effective Gated Attention (GA) and
Region-of-Interest Generation (RoIG) methodology in this paper. GA uses
attention to focus on the text stroke as well as the textures and colors of
the surrounding regions to remove text from the input image much more
precisely. RoIG is applied to focus on only the region with text instead of
the entire image to train the model more efficiently. Experimental results on
the benchmark dataset show that our method significantly outperforms existing
state-of-the-art methods in almost all metrics with remarkably higher-quality
results. Furthermore, because our model does not generate a text stroke mask
explicitly, there is no need for additional refinement steps or sub-models,
making our model extremely fast with fewer parameters. The dataset and code
are available at https://github.com/naver/garnet.

Figure 1: Visualization of the proposed model’s input and output.
Visualization of the results of applying attention to the text stroke and
applying attention to the text stroke surrounding regions found by the GA. We
obtained satisfying STR results using RoIG.

##  1 Introduction

Scene text removal (STR) is a task of erasing text from natural scene images,
which is useful for privacy protection, text editing in images/videos, and
Augmented Reality (AR) translation.

A lot of current STR research utilizes the deep learning method. Early studies
[26] attempted to erase all text in an image without using a text region mask,
but this produced imprecise, dissatisfactory results such as some regions
being blurred out. Furthermore, it was impossible to selectively erase letters
in a certain region with this method, which drastically limited applications.
In order to overcome these limitations, recent studies [19, 25] in the field
built text removal models after generating text region masks through manual or
automatic means. This lead to comparably better quality results and wider
applications.

However, these past studies did not use the same standardized training dataset
and evaluation dataset, making it impossible to evaluate performance in a fair
manner. For instance, some papers trained on different subsets of synthetic
data[6]. Furthermore, they do not have the same input image dimensions, which
affects speed, accuracy, and model parameters. Thus, it is difficult to select
a previously proposed model for the unique requirements of a specific
application when there is no standardized comparison of the model size and
speed.

In this paper, we perform a fair comparison with both qualitative (text
removal quality) and quantitative (model size, inference time) metrics by re-
implementing prominent previously proposed models with the same standards,
training them with the same training dataset, and evaluating them on the same
evaluation dataset. Furthermore, we introduce the Gated Attention (GA) and the
Region of Interest Generation (RoIG). GA is a simple yet extremely effective
method that uses attention to focus on the text stroke and its surroundings,
while RoIG drastically improves the training efficiency of the STR model.

Figure 1 shows GA found the Text Stroke Region (TSR) and the Text Stroke
Surrounding Region (TSSR) after training the pseudo mask and how RoIG
generates the results only for text mask regions. More details are described
in Section 4.

Our contributions can be summarized as follows:

  1. 1.

We re-implemented previously proposed methods with the same input image size,
trained them on the same training dataset, and compared their performance with
the same evaluation dataset. We also performed comparisons of speed and model
size.

  2. 2.

We proposed a method called GA which not only distinguishes between the
background and the text stroke, but also utilizes attention to identify both
TSR and TSSR. To the best of our knowledge, there is no other previous study
that considered both TSR and TSSR while performing STR.

  3. 3.

We proposed a method called RoIG, which helps the network focus on only the
region with text instead of the entire image to train the model more
efficiently.

  4. 4.

The proposed method generates higher quality results than existing state-of-
the-art methods on both synthetic and real datasets. It is also significantly
lighter and faster than most other prominent previously proposed popular
methods because our model does not have additional refinement steps or sub-
models needed to generate a text stroke mask.

##  2 Related Works

Table 1: A side-by-side comparison of how our proposed model differs from previous works. Method | Use | Selective | Stage | Stroke | Surround | RoI
---|---|---|---|---|---|---
text box | removal |  | localization | localization | Generation
EnsNet [26] | x | x | 1 | x | x | x
MTRNet [20] | o | o | 1 | x | x | x
MTRNet++ [19] | o | o | 2 | o | x | x
EraseNet [10] | x | x | 2 | x | x | x
Tang et al. [17] | o | o | 2 | o | x | o(crop)
Ours | o | o | 1 | o | o | o

The major trend in scene text removal before the emergence of deep learning
was traditional rule-based methods [2, 18], which are often hand-crafted and
require prior domain knowledge.

Recently, deep learning-based text removal has been proposed by adopting
popular GAN-based methods. EnsNet [26] is simple and fast because it doesn’t
need any auxiliary inputs. However, its results are blurry and of low quality.
Furthermore, its practical applications are limited because it is impossible
to only erase text in a certain region. MTRNet [20] requires the generation of
text box region masks through manual or automatic means. These studies show
that text region masks can improve the network’s performance but cannot
guarantee high-quality results. MTRNet++ [19] and EraseNet [10] proposed a
coarse-refinement two-stage network. While the results are of higher quality,
the model is much bigger, slower, and too complicated.

Zdenek et al. [25] used an auxiliary text detector to retrieve the text box
mask, then attempted to erase text through a general inpainting method.
However, they were unable to generate results of satisfactory quality because
they did not consider qualities specific to text like the text strokes. Tang
et al. [17] can train their model in a rather efficient manner because they
erase text by cropping only the text regions. However, this method ignores all
global contexts other than the cropped region and has difficulty precisely
cropping the region of curved texts.

We designed a model that can focus on only the text region area while also
managing to take the global context of the entire image into consideration.
This model does not require an additional refinement process or a sub-model
dedicated to text stroke localization, leading to a drastically faster and
lighter model.

Table 2: A comparison of our model’s performance with previous STR models on real data (SCUT-EnsText [10]). We compare the results claimed in the original papers of the previous prominent STR models with the results we re-measured in an equal environment. We realize that there are fundamental differences in models and that it would be unfair to evaluate them under potentially disadvantageous circumstances. To make things truly fair, we also pasted the text region (produced by the model using a text box mask) over the original image, then evaluated the results yet again. The values right of “/” reflect experiments conducted after this adjustment was made. Our model produces superior results in all metrics regardless. The best score is highlighted in bold. | Method | Train | Size | Image-Eval
---|---|---|---|---
Data | PSNR | SSIM | AGE
Reported | Scene Text Eraser [13] | Real | 256 | 25.47/- | 90.14/- | 6.01/-
EnsNet [26] | Real | 512 | 29.53/- | 92.74/- | 4.16/-
MTRNet [20] | Syn(75%) | 256 | -/- | -/- | -/-
MTRNet++ [19] | Syn(95%) | 256 | -/- | -/- | -/-
EraseNet [10] | Real | 512 | 32.30/37.26 | 95.42/96.86 | 3.02/-
Our experiment | EnsNet [26] | Real+Syn | 512 | 31.05/32.99 | 94.78/95.16 | 2.67/1.85
MTRNet [20] | Real+Syn | 256 | 30.61/36.06 | 89.85/95.72 | 3.92/1.21
| Real+Syn | 512 | 32.46/36.89 | 95.86/96.41 | 3.12/0.97
MTRNet++ [19] | Real+Syn | 256 | 35.29/37.40 | 96.31/96.68 | 1.26/1.09
| Real+Syn | 512 | 34.86/36.50 | 96.32/96.51 | 1.48/1.35
EraseNet [10] | Real+Syn | 512 | 30.54/37.16 | 96.27/97.53 | 3.07/1.20
EraseNet [10] \+ M | Real+Syn | 512 | 34.29/40.18 | 97.73/97.98 | 2.28/0.69
| Ours | Real+Syn | 512 | -/41.37 | -/98.46 | -/0.64

##  3 Comprehensive Prominent Model Analysis

In this section, we analyze the difference between previous methods and ours.
We also show the results of evaluating previous methods with one standardized
dataset.

Table 1 outlines the specific differences between our method and previously
proposed methods. First, our proposed method takes a text box mask as the
input to our model, leading to results of significantly better quality as well
as the option for users to selectively erase only the text that they wish to.
Second, our proposed method can localize the TSR and TSSR to erase text in a
surgical manner. Finally, the use of RoIG makes our proposed method show off
significantly better results than previous methods without even implementing a
coarse-refinement 2-stage network.

Table 2, Table 3 and Table 4 show the performance of several previous STR
methods on real and synthetic data. Details of the experiment and re-
implementation are in Section 5 and Section 5.3.

After performing an objectively fair comparison, we found that our method
generates higher quality results than existing state-of-the-art methods on
both synthetic and real datasets. It is also significantly lighter and faster
than any other method except EnsNet [26].

Table 3: A comparison of our model’s performance with previous STR models on real data (SCUT-EnsText [10]). R, P, and F refer to recall, precision and F-score, respectively. The best score is highlighted in bold. | Method | Train | Size | Detection-Eval | GPU | Params
---|---|---|---|---|---|---
Data | P | R | F | Time(ms)
Reported | Scene Text Eraser [13] | Real | 256 | 40.9 | 5.9 | 10.2 |  |
EnsNet [26] | Real | 512 | 68.7 | 32.8 | 44.4 | - | 12.4M
MTRNet [20] | Syn(75%) | 256 | - | - | - | - | 54.4M
MTRNet++ [19] | Syn(95%) | 256 | - | - | - | - | 18.7M
EraseNet [10] | Real | 512 | 53.2 | 4.6 | 8.5 | - | -
Our experiment | EnsNet [26] | Real+Syn | 512 | 73.1 | 54.7 | 62.6 | 12.0 | 12.4M
MTRNet [20] | Real+Syn | 256 |  |  |  | 21.9 | 50.3M
| Real+Syn | 512 | 69.8 | 41.1 | 51.2 | 51.3 |
MTRNet++ [19] | Real+Syn | 256 |  |  |  | 51.3 |
| Real+Syn | 512 | 58.6 | 20.5 | 30.4 | 238.7 |
EraseNet [10] | Real+Syn | 512 | 40.8 | 6.3 | 10.9 | 47.4 | 17.8M
EraseNet [10] \+ M | Real+Syn | 512 | 37.3 | 6.1 | 10.3 | 47.4 | 17.8M
| Ours | Real+Syn | 512 | 15.5 | 1.0 | 1.8 | 14.9 | 12.4M
Table 4: A comparison of our model’s performance with previous STR models on synthetic data (Oxford [6]). The notation is the same as Table 2 and Table 3. Method | Input | Image-Eval | Detection-Eval
---|---|---|---
Size | PSNR | SSIM | AGE | P | R | F
Original Image | 512 |  |  |  | 71.3 | 51.5 | 59.8
EnsNet [26] | 512 | 36.67/39.74 | 97.71/97.94 | 1.25/0.77 | 55.1 | 14.0 | 22.3
MTRNet [20] | 256 | 30.96/37.69 | 90.95/95.83 | 4.17/1.22 |  |  |
| 512 | 35.49/40.03 | 97.10/97.69 | 2.18/0.80 | 58.6 | 13.7 | 22.2
MTRNet++ [19] | 256 | 37.40/40.26 | 97.02/97.25 | 0.86/0.79 |  |  |
| 512 | 38.31/40.64 | 97.82/97.94 | 0.81/0.73 | 64.0 | 15.9 | 25.4
EraseNet [10] | 512 | 34.35/41.73 | 98.01/98.62 | 1.81/0.66 | 30.8 | 0.6 | 1.2
EraseNet [10] \+ M | 512 | 36.37/42.98 | 98.50/98.75 | 1.72/0.56 | 31.4 | 0.0 | 1.4
Ours | 512 | -/43.64 | -/98.64 | -/0.55 | 18.9 | 0.1 | 0.3

##  4 Methodology

###  4.1 Motivation

Previous STR models [20, 19] used a text box region as well as a text stroke
region in an attempt to perform precise text removal. However, TSSR was mostly
overlooked. After finding inspiration from observing how humans must alternate
between paying attention to the text stroke regions and the surrounding
regions of the text while manually performing STR, we devised the GA.
Meanwhile, because all previous studies performed STR on the entire image,
artifacts frequently occurred in non-text regions. Thus, we devised the RoIG,
which allows our STR model to only generate a result image from within the
text box region instead of wasting resources attempting to perform STR on the
full image.

Figure 2: The overall architecture of the proposed model. The top box, Region-
of-Interest Generation, shows that our model generates images focusing only on
the text region. The box on the bottom left, Loss for RoIG, how our loss
function helps the model focus only on the text region. The $L_{R}$ only use
$I_{gen}$ with $M$, and $L_{tv}$, $L_{p}$ and $L_{s}$ use $I_{comp}$, so the
outside of text regions in $I_{gen}$ does not participate in loss calculation.
On bottom right box, The GA calculates the stroke attention and stroke
surrounding attention mask for every layer of the generator’s encoder, then
automatically calculates the importance of each. Each attention map is
supervised by pseudo mask. More detailed information on GA is in Figure 3.

###  4.2 Model architecture

Figure 2 shows the architecture of our proposed method. The generator G takes
the image and corresponding text box mask as its input and produces a non-text
image that is visually plausible. Following GAN-based methods [12, 8], the
discriminator D takes both the input of the generator and the target images as
its input and differentiates between real images and images produced by the
generator. The objective functions of the generator and discriminator are as
follows:

$L_{adv}=\mathbb{E}_{x}\left[logD(x,G(x))\right]$ (1)
$\begin{split}L_{D}=\mathbb{E}_{x,y}\left[logD(x,y)\right]+\mathbb{E}_{x}\left[(1-logD(x,G(x)))\right]\end{split}$
(2)

where $x$ is the input image concatenated with a text box mask and y is the
target image.

Generator. The generator has an FCN-ResNet18 backbone and skip connections
[14] between the encoder and decoder. The model is composed of five
convolution layers paired with five deconvolution layers with a kernel size of
4, stride of 2, and padding of 1. The convolution pathway is composed of two
residual blocks [7], which contains the proposed Gated Attention (GA) module.

Discriminator. For training, we use a local-aware discriminator proposed in
EnsNet [26], which only penalizes the erased text patches. The discriminator
is trained with locality-dependent labels, which indicate text stroke regions
in the output tensor. It guides the discriminator to focus on text regions.

###  4.3 Gated Attention

Localizing both the TSR and TSSR is imperative to perform surgical STR. In
order to do this without letting the size of our model blow up, we used
spatial attention[22] instead of a separate image segmentation branch[19].
Table 2 shows how our proposed model is significantly faster and smaller than
MTRNet++ [19].

Figure 3 shows the architecture of the GA. The module takes the feature map as
its input and generates a TSR and TSSR feature map, then adjusts the
proportion of these two feature maps through gate parameters. The process of
GA is as follows:

$F_{i}^{{}^{\prime}}=(MaxPool(F_{i}^{In})\oplus AvgPool(F_{i}^{In})\oplus
M_{box})$ (3) $\begin{split}F_{i}^{t}=W_{i}^{t}\cdot F_{i}^{{}^{\prime}}\\\
F_{i}^{s}=W_{i}^{s}\cdot F_{i}^{{}^{\prime}}\end{split}$ (4)
$A_{i}^{out}=\sigma(\alpha_{i}F_{i}^{t}+\beta_{i}F_{i}^{s})$ (5)
$F_{i}^{out}=F_{i}^{In}A_{i}^{out}$ (6)

where $i$ denotes the $ith$ layer in the encoder, $F^{In}$ and $F^{out}$
denote the input and output feature maps, $W^{t}$ and $W^{s}$ denote 7x7
convolution filters that extract TSR and TSSR features, $F^{t}$ and $F^{s}$
denote extracted feature maps for localized TSR and TSSR, $\alpha$ and $\beta$
denote gate parameters, and $\sigma$ and $A^{out}$ denote the Sigmoid
activation function and the attention score map.

Figure 1 shows the text box mask, TSR mask, and the TSSR mask necessary for
this. We generated a pseudo text stroke mask, automatically calculated by
taking the pixel value difference between the input image and ground truth
image. The TSR masks help train the TSR attention module to distinguish the
TSR. The TSSR mask is the intersection region between the exterior of the
pseudo text stroke mask and the interior of the text box mask. It makes the
TSSR attention module train to focus on the colors and textures of the TSSR.
Note that the attention method that we used differs from S. Woo et al. [22],
which was trained in a weakly-supervised manner. We show that our method is
superior in Section 5.4. The GA learns the gate parameter on its own, and can
thus adjust the respective attention ratios allocated to TSR and TSSR during
training. The loss function is designed to be applied only within the text box
regions, not to the entire score map. The loss function to train GA is as
follows:

$\begin{split}L_{att}^{t}=\begin{cases}-Gt_{i}^{t}log(S_{i}^{t})&\mbox{if
}M_{i}^{Box}>0,\\\ 0&\mbox{otherwise}\end{cases}\\\
L_{att}^{s}=\begin{cases}-Gt_{i}^{s}log(S_{i}^{s})&\mbox{if }M_{i}^{Box}>0,\\\
0&\mbox{otherwise}\end{cases}\end{split}$ (7)
$L_{att}=L_{att}^{t}+L_{att}^{s}$ (8)

where $Gt_{i}^{t}$ and $Gt_{i}^{s}$ are the $ith$ pixel of the ground truth
mask representing TSR and TSSR. $S_{i}^{t}$ and $S_{i}^{s}$ are the $ith$
pixel of the TSR and TSSR attention score maps. $M_{i}^{Box}$ is text box
mask. The shape of $M$, $Gt$ and $S$ is $(H_{n},W_{n},1)$. $H_{n}$ and $W_{n}$
denote the sizes of feature map in the $nth$ layer.

Figure 3: Architecture of the Gated Attention. The GA calculates the TSR and
TSSR attention masks, then automatically calculates the importance of each
through gate parameters $\alpha,\beta$.

###  4.4 Region-of-Interest Generation

Most STR methods attempt to perform in-painting of TSR as well as
reconstruction of the entire image. However, our approach can skip the
reconstruction of non-masked regions altogether because if the text box region
is given as an input to the STR model, there is no need to render the entire
image for the output. Therefore, we modified the loss function so that our
model’s generator only has to focus on the text box region during training.
Note that the generator’s loss is only calculated with respect to the text box
region. Every other region is considered don’t care and therefore is
irrelevant during training. Because all regions other than the text box
obtained from the generator’s output value are not used, we blurred them in
Fig 3. This makes training significantly easier.

In total, we modified four loss functions for RoIG: RoI Regression,
Perceptual, Style, and Total Variation Loss.

RoI Regression Loss. We modified regression loss to only consider text
regions. The proposed Region-of-Interest Regression Loss is defined as:

$L_{R}(M,I_{out},I_{gt})=\sum_{i=1}^{n}\lambda_{i}\odot
M_{i}\parallel{I_{out(i)}-I_{gt(i)}}\parallel_{1}$ (9)

where $I_{out(i)}$ is the output of the $ith$ deconvolution pathway. We use
the output of the 3rd, 4th, and 5th layers in the deconvolution pathway.
$M_{i}$ and $I_{gt(i)}$ are the box mask and ground truth that was resized to
the same scale as $I_{out(i)}$. $\lambda_{i}$ is the weight for scale. We set
$\lambda_{i}$ to 0.6, 0.8, 1.0.

Perceptual Loss. Perceptual Loss [9] is used to make the generated image more
realistic. It reduces the difference between the high-level features of the
two images extracted using the pre-trained ImageNet. Some works [11, 26, 10]
used a composited output to address the discrepancy between text-erased
regions and the background. We use only the text regions in the generated
image by pasting it into the input image. We address the discrepancy between
the text box and input images by designing the loss function to use two
composited outputs generated using the box and stroke mask, respectively.
Perceptual Loss is defined as:

$\displaystyle I_{boxComp}=I_{in}(1-M_{Box})+I_{out}M_{Box}$ (10)
$\displaystyle I_{StrokeComp}=I_{in}(1-M_{Stroke})+I_{out}M_{Stroke}$
$\begin{split}L_{P}=\sum_{n=1}^{N-1}\parallel{A_{n}(I_{boxComp})-A_{n}(I_{gt})}\parallel_{1}\\\
+\sum_{n=1}^{N-1}\parallel{A_{n}(I_{StrokeComp})-A_{n}(I_{gt})}\parallel_{1}\end{split}$
(11)

where $I_{in}$, $I_{out}$ refer to the input image and the generator’s output
image, $I_{BoxComp}$, $I_{StrokeComp}$ are generated images composited of box
mask $M_{box}$ and stroke mask $M_{stroke}$ respectively. $I_{gt}$ is the
ground truth image, and $A_{n}$ refers to the activation of the $nth$ layer in
network. We use the pool1, pool2 and pool3 layers of the VGG-16 [15]
pretrained on ImageNet [3].

Style Loss. Style Loss [5] considers the global texture of the entire image
and is used to further improve the visual quality of the output. It is
calculated using the Gram matrix of feature maps. Like perceptual loss, we use
two composited outputs. Style loss is defined as:

$\begin{split}L_{S}=\sum_{n=1}^{N-1}\parallel{\frac{1}{H_{n}W_{n}C_{n}}[(\phi(A_{n}(I_{boxComp}))-\phi(A_{n}(gt))]}\parallel_{1}\\\
+\sum_{n=1}^{N-1}\parallel{\frac{1}{H_{n}W_{n}C_{n}}[(\phi(A_{n}(I_{strokeComp}))-\phi(A_{n}(gt))]}\parallel_{1}\end{split}$
(12)

where $\phi(x)=x^{T}x$ is a gram matrix operator and $A_{n}$ is the activation
of the $nth$ layers of VGG-16 [15]. We use same layers as Perceptual Loss.

Total Variation Loss. J. Johnson et al. [9] proposed total variation loss for
global denoising. As our model generates images using RoIG method, the loss
function uses composited images which are generated using box masks. The total
variation loss is as follows:

$\begin{split}L_{t}=\sum_{i,j}\parallel{I^{i,j+1}_{Comp}-I^{i,j}_{Comp}}\parallel_{1}\\\
+\parallel{I^{i+1,j}_{Comp}-I^{i,j}_{Comp}}\parallel_{1}\end{split}$ (13)

where $i$,$j$ are the pixel positions.

Total Loss function. The total loss function for train generator is as
follows:

$\begin{split}L_{G}=\lambda_{r}L_{R}+\lambda_{p}L_{P}+\lambda_{s}L_{S}+\lambda_{t}L_{t}\\\
+\lambda_{adv}L_{adv}+\lambda_{att}L_{Att}\end{split}$ (14)

where $\lambda_{r}$ to $\lambda_{att}$ represent the weight of each loss. We
set $\lambda_{i}$ to 100, 0.5, 50.0, 25.0, 1, 10.

##  5 Experiment

###  5.1 Dataset and Evaluation Metrics

For training and evaluation, we use both synthetic and real datasets.

Synthetic data. The Oxford Synthetic text dataset [6] is adopted for training
and evaluation. The dataset contains around 800,000 images composed of 8,000
text-free images. We randomly selected 95% images for training, selected
10,000 images for testing, and used the rest for validation. Note that the
background images in the train set and test set are mutually exclusive.

Real data. SCUT-EnsText [10] is a real dataset for scene text removal. The
dataset, which was manually generated from Chinese and English text images,
contains 2,749 train and 813 test images. In this paper, we adopted these
images for training and evaluation.

Preprocessing. We need stroke-level segmentation masks to train our model.
However, the existing datasets do not provide stroke-level segmentation masks.
Therefore, we created it automatically by calculating the pixel value
difference between the input image and the ground truth image. To suppress
noise, we set a threshold of 25.

We combined synthetic and real datasets. In total, we used 738,113 images for
training. The test set was used separately to distinguish between performance
on real and synthetic datasets.

Evaluation Metrics T. Nakamura et al. [13] proposed an evaluation method using
an auxiliary text detector. An auxiliary text detector obtains detection
results on the images with text removed. Then, it evaluates the model
performance by calculating Precision, Recall, and F-score values. A lower
value means that the texts are better erased. In this paper, we use Detection
Eval [21] as an evaluation metric and CRAFT [1] as an auxiliary detector.
However, that method only indicates how much text has been erased, not output
quality. S. Zhang et al. [26] proposed using the evaluation method that is
used in image inpainting. They used PSNR, SSIM, MSE, AGE, pEPs, pCEPs to
evaluate image quality. The higher the value of PSNR and SSIM, and the lower
the value of other metrics, the better the quality of the output image. We use
PSNR, SSIM, and AGE for evaluation.

###  5.2 Implementation details

We trained our model for 6 epochs with batch size 30 on the combined dataset.
The Adam optimizer with $\beta$ (0.9, 0.999) was used. We set the initial
learning rate to 0.0005 and divided it by 5 every 50,000 steps. PyTorch and
NVIDIA Tesla M40 GPUs were used in all experiments.

###  5.3 Re-implementation details

In this section, we provide details of the re-implementation of previous
methods. We re-implemented EnsNet [26] by modifying the code implemented in
mxnet and trained the model with the same hyperparameters used in their paper.
We re-implemented MTRNet [20] by converting the code implemented in TensorFlow
to PyTorch and trained the model with the same batch size and epoch as
mentioned in their paper. We used the official implementation of MTRNet++ [19]
and EraseNet [10]. We trained them with the same batch size and epoch as
mentioned in their respective papers. EraserNet [10] \+ M refers the model
using mask as its input with image.

###  5.4 Ablation Study

In this section, we validate the effectiveness of our contributions: Gated
Attention (GA) and Region-of-Interest Generation (RoIG)

BaseLine. We combined the text region mask with the EnsNet [26] model to make
a baseline. This results in a quality improvement of the output image as well
as the added functionality of flexibly removing only specific characters at
the user’s discretion. All of our experiments including the baseline use a
4-channel input by concatenating the 3-channel RGB and a 1-channel mask.

Table 5: Result quality comparison with ablation studies. SA, TSRA, TSSRA, GA, and RoIG refer to Simple Attention [22], Text Stroke Region Attention, Text Stroke Surrounding Region Attention, Gated Attention, and Region of Interest Generation respectively. The notation is the same as Table 2. Method | Image Eval | Detection Eval
---|---|---
PSNR | SSIM | AGE | P | R | F
Original Image |  |  |  | 79.8 | 67.2 | 73.0
EnsNet [26] | 31.05/32.99 | 94.78/95.16 | 2.67/1.85 | 73.1 | 54.7 | 62.6
BaseLine (EnsNet + M) | 35.65/37.28 | 96.67/96.78 | 1.50/0.89 | 63.3 | 33.2 | 43.6
BaseLine + SA | 35.73/37.24 | 96.53/96.62 | 1.52/0.89 | 65.5 | 34.4 | 45.1
BaseLine + TSRA | 36.07/38.06 | 97.23/97.34 | 1.46 /0.84 | 54.3 | 21.1 | 30.4
BaseLine + TSSRA | 36.12/38.45 | 97.29/97.46 | 1.51/0.84 | 39.1 | 6.0 | 10.5
BaseLine + GA | 36.38/38.82 | 97.56/97.72 | 1.46/0.80 | 30.6 | 3.9 | 6.9
BaseLine + RoIG | -/40.82 | -/98.19 | -/0.66 | 27.5 | 3.1 | 5.6
BaseLine + GA + RoIG | -/41.37 | -/98.46 | -/0.64 | 15.5 | 1.0 | 1.8
Figure 4: Comparison of the output results after ablation. Image from left to
right: Input, Baseline, Baseline + TSRA, Baseline + TSSRA, Baseline + GA, and
Baseline + GA + RoIG. Figure 5: Visualization of the attention masks for
differing encoding layers. The images progressively go from low-level features
to high-level features. (a) is a visualization of simple spatial attention
[22], (b) and (c) is a visualization of the text stroke and the surrounding
attention that the GA generated. (d) is a visualization of how the GA used the
gate parameter to aggregate TSA and TSSA. Spatial attention does a poor job
finding the text strokes and the surrounding regions if simply applied in STR.
In comparison, we can see that the GA pays more attention to the text stroke
regions in low-level features while paying more attention to the surrounding
regions of the text in high-level features.

Attention. First, we performed the following three experiments to observe the
effect of each attention on the STR results: Simple Attention (SA) [22], Text
Stroke Region Attention (TSRA), and Text Stroke Surrounding Region Attention
(TSSRA).

Table 5 shows how the application of only SA does not improve the quality of
STR. Figure 5’s (a) shows how the application of only SA in STR does not lead
to localization of the text stroke and the surrounding regions of the text
stroke properly. However, both the TSRA and the TSSRA obtained higher PSNR and
SSIM results. In particular, the evaluation result of the TSSRA was
significantly better than the TSRA in Detection Eval. This shows that TSSR is
more important. Figure 4 demonstrates that results produced by the TSSRA
reduce more artifacts than the TSRA. TSRA helps locate the TSR because it is a
target for inpainting, but is inappropriate when focusing on the TSSR to erase
text. On the other hand, TSSRA is appropriate to focus on the TSSR, and the
model utilizes this to fill the TSR and generate higher quality output.

Furthermore, We found that having the GA module picking optimal ratios from an
ensemble of both TSRA and TSSRA, then aggregating the features as appropriate
was super effective. As Table 5 and Figure 4 clearly demonstrate, the
evaluation scores of STR were highest when using the GA while also leaving
almost no artifacts behind on the resulting image. In Figure 5, the GA module
puts more emphasis on the surrounding regions of text strokes rather than the
text strokes as it approaches higher-level features.

Region of Interest Generation. In order to measure the effects of RoIG, we
performed experiments with and without its use. Table 5 demonstrates that RoIG
significantly improves the quality of STR in all metrics. Figure 4 shows that
models with the application of RoIG left almost no artifacts with the best
overall results.

Figure 6: Comparison of output image results with other prominent STR models.
Image from left to right: Input, Ground Truth, EnsNet [26], MTRNet [20],
MTRNet++ [19], EraseNet [10], and Ours.

###  5.5 Comparison with previous methods

As we mentioned in Section 3, the proposed model maintains a competitive edge
with speed and model size while significantly improving the result quality
than previous STR models on both real and synthetic data. The first row of
Figure 6 shows that EnsNet[26] and EraseNet[10], both of which do not use an
explicit text box region, only partially erase text. The fourth row of the
Figure shows that MTRNet[20] and MTRNet++[19] do not successfully remove all
text from complex backgrounds without leaving behind artifacts or partially-
erased text. However, our proposed model with GA and RoIG successfully outputs
high-quality STR images without residual artifacts from images with small
text, curved text, and text on complex backgrounds, even without additional
refinement.

##  6 Conclusion

Although there was a lot of progress in the STR area, it was difficult to
establish the superiority of a model from the previously proposed methods
because there was no standardized and fair way to evaluate performance. In
this paper, we re-implemented prominent previously proposed methods, trained
and evaluated on respective standardized datasets, and evaluated their
accuracies, model size, and inference time in an objectively fair manner. We
proposed a simple yet highly impactful STR method with Gated Attention (GA)
and Region-of-Interest Generation (RoIG). GA uses attention on the text
strokes and the surrounding region’s colors and textures to surgically erase
text from images. RoIG makes the generator focus on only the region with text
instead of the entire image for more efficient training. Our method
significantly outperforms all existing state-of-the-art methods on all
benchmark datasets in terms of inference time and output image quality.

####  6.0.1 Acknowledgements.

We wish to thank Osman Tursun for providing codes of MTRNet and MTRNet++.

## References

  * [1] Baek, Y., Lee, B., Han, D., Yun, S., Lee, H.: Character region awareness for text detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9365–9374 (2019)
  * [2] Bertalmio, M., Bertozzi, A.L., Sapiro, G.: Navier-stokes, fluid dynamics, and image and video inpainting. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. vol. 1, pp. I–I. IEEE (2001)
  * [3] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [4] Doersch, C., Singh, S., Gupta, A., Sivic, J., Efros, A.A.: What makes paris look like paris? Communications of the ACM 58(12), 103–110 (2015)
  * [5] Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576 (2015)
  * [6] Gupta, A., Vedaldi, A., Zisserman, A.: Synthetic data for text localisation in natural images. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2315–2324 (2016)
  * [7] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [8] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1125–1134 (2017)
  * [9] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision. pp. 694–711. Springer (2016)
  * [10] Liu, C., Liu, Y., Jin, L., Zhang, S., Luo, C., Wang, Y.: Erasenet: End-to-end text removal in the wild. IEEE Transactions on Image Processing 29, 8760–8775 (2020)
  * [11] Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 85–100 (2018)
  * [12] Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  * [13] Nakamura, T., Zhu, A., Yanai, K., Uchida, S.: Scene text eraser. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). vol. 1, pp. 832–837. IEEE (2017)
  * [14] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
  * [15] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [16] Song, Y., Yang, C., Lin, Z., Liu, X., Huang, Q., Li, H., Kuo, C.C.J.: Contextual-based image inpainting: Infer, match, and translate. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 3–19 (2018)
  * [17] Tang, Z., Miyazaki, T., Sugaya, Y., Omachi, S.: Stroke-based scene text erasing using synthetic data for training. IEEE Transactions on Image Processing 30, 9306–9320 (2021)
  * [18] Telea, A.: An image inpainting technique based on the fast marching method. Journal of graphics tools 9(1), 23–34 (2004)
  * [19] Tursun, O., Denman, S., Zeng, R., Sivapalan, S., Sridharan, S., Fookes, C.: Mtrnet++: One-stage mask-based scene text eraser. Computer Vision and Image Understanding 201, 103066 (2020)
  * [20] Tursun, O., Zeng, R., Denman, S., Sivapalan, S., Sridharan, S., Fookes, C.: Mtrnet: A generic scene text eraser. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 39–44. IEEE (2019)
  * [21] Wolf, C., Jolion, J.M.: Object count/area graphs for the evaluation of object detection and segmentation algorithms. International Journal on Document Analysis and Recognition 8(4), 280–296 (2006)
  * [22] Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV). pp. 3–19 (2018)
  * [23] Xie, C., Liu, S., Li, C., Cheng, M.M., Zuo, W., Liu, X., Wen, S., Ding, E.: Image inpainting with learnable bidirectional attention maps. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8858–8867 (2019)
  * [24] Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4471–4480 (2019)
  * [25] Zdenek, J., Nakayama, H.: Erasing scene text with weak supervision. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2238–2246 (2020)
  * [26] Zhang, S., Liu, Y., Jin, L., Huang, Y., Lai, S.: Ensnet: Ensconce text in the wild. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 801–808 (2019)
  * [27] Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence 40(6), 1452–1464 (2017)

##  Appendix 0.A Appendix

Figure 7: Comparison of the quality of images. Image from left to right: input
image, ground truth image, LBAM pre-trained on Paris Street View, GC pre-
trained on Places2, LBAM fine-tuned on our dataset, Ours.

###  0.A.1 Comparison with general image inpainting

We compared our proposed method with general inpainting methods. LBAM [23],
Gated Convolution [24] are adopted for comparison. We trained LBAM [23], which
is pre-trained on Paris Street View [4], for 4 epochs on our combined dataset.
For general inpainting methods, the text-stroke which is located outside of
the box mask can affect the model’s performance. To solve this issue, we
provided the bounding box information, which was dilated 4 times with a 3x3
kernel, to the inpainting model. However, due to time constraints, we could
not train GC [24] on our combined dataset. Instead, we evaluated model, pre-
trained on Places2 [27], which has Contextual Attention module [16]. The
quantitative results are shown in Tab. 6. For comparison, we used composited
images generated by using box masks. Table 6 shows that our proposed method
outperforms the existing inpainting method in all metrics. The qualitative
results are shown in Figure 7. As shown in Figure 7, the results of LBAM [23]
are blurry and incomplete. When masked regions contain complex backgrounds,
the model can not reconstruct non-text information in masked regions properly,
while our methods can reconstruct non-text regions and only erase text-stroke
regions.

We acknowledge the lack of comparison with GC [24]. We planned to trained GC
[24] on our combined dataset in the future. In addition, we planned to apply
the Gated Convolution module to our model to compare performance between Gated
Convolution and our proposed module without the influence of Contextual
Attention [16].

Table 6: Comparison for SCUT-EnsText (Image Eval). For a fair comparison, we provided box masks, which were dilated four times with a 3x3 kernel, to our methods.  Method | data | Input | SCUT-EnsText
---|---|---|---
size | PSNR | SSIM | AGE
LBAM [23] | pre-trained | 512 | 34.20 | 96.13 | 1.6670
| pre-trained + Ours | 512 | 36.76 | 97.55 | 1.1404
GC [24] | pre-trained | 512 | 34.24 | 96.46 | 1.5049
Ours | Ours | 512 | 39.20 | 98.11 | 0.8302


XX XX 8 May/June

Department: SE4AI Editor: Tim Menzies<EMAIL_ADDRESS>
# AI-driven Development Is Here: Should You Worry?

Neil A. Ernst  University of Victoria  Gabriele Bavota  Università della
Svizzera italiana

(2019)

††journal: Software

From the Editor:

The benefits, as well as the draw-backs, of new AI technology needs to be
carefully scrutinized. For example, in this article, researchers review the
promise, and potential pitfalls, of AI tools that help programmer co-write
their code.

This “SE for AI” column publishes commentaries on the growing field of SE for
AI. Submissions are welcomed and encouraged (1,000–2,400 words, each figure
and table counts as 250 words, try to use fewer than 12 references, and keep
the discussion practitioner focused). Please submit your ideas to me at
timmieee.org.—Tim Menzies

AI-driven Development Environments (AIDEs) integrate the power of modern AI
into IDEs like Visual Studio Code and JetBrains IntelliJ. By leveraging
massive language models and the plethora of openly available source code,
AIDEs promise to automate many of the obvious, routine tasks in programming.
At the same time, AIDEs come with new challenges to think about, such as bias,
legal compliance, security vulnerabilities, and their impact on learn
programming.

Time was, programmers were laboriously entering machine instructions on punch
cards, to be painstakingly offered up to the machine for processing. Nowadays,
of course, we type code into editors, often hosted online, and get near-
instant feedback on the compilation/test outcomes of our latest change. Let’s
call this the Integrated Development Environment (IDE) Revolution.

Another revolution is now underway, call it the “AI-driven Development
Environment” (AIDE), one where a programmer can leverage the hive-mind of the
many many programmers that came before them. Large language models are now
available to make suggestions based on what hundreds of previous programmers
typed in.

Not only is the syntax of programming “natural”, but in fact a lot of
programming follows repetitive patterns. If you have used a form of
autocomplete in your IDE, you have seen a very primitive version of this idea
(autocomplete seems to have been popularized with Microsoft’s Visual-* tools
in the 90s). Autocomplete at its simplest looks for all valid completions,
such as possible library calls in List.. But things have progressed very
rapidly in this space. Think about preparing some data analysis code:

⬇

# import a file named ”observations.csv”

# and print the number of rows and columns

# then use the data to create

# a plot of the observations

import csv

import matplotlib.pyplot as plt

import numpy as np

\pardata = []

\parwith open(’observations.csv’) as csvfile:

reader = csv.reader(csvfile)

for row in reader:

data.append(float(row[0]))

\parprint(”Number of rows: ”, len(data))

print(”Number of columns: ”, len(data[0]))

\parplt.plot(data)

plt.show()

Spoiler alert: we didn’t write a single line of that code! Instead, we typed
the four lines of comments into Visual Studio Code, a lightweight IDE, and had
GitHub’s Copilot extension “write” the actual Python.

Copilot, and its underlying AI engine, OpenAI’s Codex, leverages the power of
large language models trained on billions of text data online. Codex further
tunes those language models with publicly available GitHub repositories. In
essence, since so many repositories contain similar lines to the ones we wrote
above, the language model can associate those lines with the comments we
wrote, and something that might seem like magic can appear.

This did not happen overnight, of course. Codex is the product of:

  * •

AI research into deep neural networks and language models;

  * •

Software repository mining for extracting data from repositories;

  * •

Investigations into the linguistic nature of source code (Hindle et al.,
2012);

  * •

Massive amounts of compute and data storage.

Codex and Copilot are not alone: several other AIDEs exist, including
TabNine111tabnine.com and Kite222kite.com. Intelligent fill in spreadsheets is
a close companion, as is program repair such as Facebook’s SapFix (Marginean
et al., 2019).

Codex (Chen et al., 2021) is built on OpenAI’s GPT-3 language model with 12B
parameters, and fine-tuned on 159 Gb of data from public GitHub repositories.
The nature of a language model is that it builds deep associations between
words in a high-dimensional space, learning complex patterns that produce
code. They do not require abstract representations of code like ASTs, working
on the token level directly. With suitable unit test coverage, generating 100
possible completions produced a solve rate (pass@100) of over 70% on a
benchmark of common completion tasks.

Codex promises to dramatically change the way we write and build software
systems. Like all AI technology, it has great promise, but also comes with
several new challenges for research and for practice.

## Promises of AI-driven DEs

Automate the mundane Much of software development is routine. Developers get a
bug report, track down the bug, and file a patch; they wire library code
together to leverage APIs; they need to display database records on a web page
and handle any updates. Much of software development is also staggeringly
complex and creative, too! Software is, as Grady Booch once wrote, “the
invisible thread … on which we weave the fabric of computing.” A key developer
task then is to carefully distinguish those tasks which are complex, and those
which are obvious or complicated (Snowden et al., 2020). AIDEs can remove the
accidental complexity from what are obvious tasks, just like the code showed
earlier. An AIDE like Copilot is already capable of automating these routine
tasks, and other technologies, such as the automated program repair work of
Facebook’s SapFix tool (Marginean et al., 2019), are tackling similar routine
tasks.

Automate API interactions Much of programming today is about framework and
API-driven development: connecting to a third party service, processing the
result, and sending the result back to the user. Just as often we work within
an existing architectural framework, for example for web or mobile
applications, and our programs are closely coupled with those library calls.
Many of these library calls are routine and repetitive for each new variant of
an app.

Teach Programming As programming languages and APIs proliferate, learning new
approaches and syntax becomes more challenging. Stack Overflow is invaluable
for specific answers to common, and not so common, programming problems. For
example, how does one configure a particular plotting library such as R’s
ggplot to change the background colour? But Stack Overflow, while incredibly
helpful (we certainly could not program without it), requires one to leave the
IDE to ask questions or perform search. AIDE promises the knowledge potential
of Stack Overflow while avoiding continuous context switches between the IDE
and the browser. And this will be useful for novices and experts alike. No
more ‘yak shaving’333Yak shaving refers to doing a series of trivial tasks
which distract you from the original, and important, goal. Compare with
_bikeshedding_. trying to figure out the correct series of syntax calls for a
given problem.

## Possible Challenges with AIDE

Like any software development, AIDE will come with a host of challenges to be
overcome, challenges in traditional software concerns such as defects and
security vulnerabilities, but in new areas as well.

Copyright and Licensing Codex is trained on (54 million) public GitHub
repositories, and the creators of these GitHub repositories agreed to Codex-
like usage of their code. However, the Codex context of use was something most
of us probably did not anticipate. Does Codex have the right to all the code
it was trained on? For output the language model is a series of weights, so in
theory, code produced is an amalgamation of the inputs. Accordingly to a
recent study (Ziegler, 2021) Codex rarely quotes code verbatim from the
training set and when this happens it is usually code largely reused across
open source projects. Does Codex-created code violate copyright? Is it fair
use? We don’t have an answer to this question, and open source licenses might
need to be revisited to explicitly regulate the usage of code for training
commercial code generation tools. Also, currently Codex output is the
intellectual work product of the person who activated Codex, but this is
currently because Codex is a beta, and these terms might change.

Learning to Program The nature of learning programming will change
dramatically with AIDEs. Whether these assistants will speed up or slow down
the learning process is currently an open question. On the one hand, novice
programmers can benefit from AIDEs by receiving recommendations useful to deal
with tasks they struggle with. On the other hand, the risk of not fully
understanding the received recommendations and just accept them is there. On
top of this, AIDEs do also pose challenges for instructors: Codex is already
so good it might surpass first year university students in introduction to
programming (CS1) courses. Some initial results in our testing show it is
relatively simple to get Codex to generate reasonable (passing) solutions. CS1
programming assignments must change to handle those students who can merely
pass the entire assignment spec to Codex for a solution.

Figure 1: Copilot portends a new generation of AI-based productivity. The
benefits, as well as the drawbacks, of this new approach need to be carefully
scrutinized. For example, asking Copilot to generate a list of names produces
the gray text; which is a list of predominantly English/American names
(indicating an interesting, if not perhaps troubling, bias in its training
data.)

Dataset Quality Plenty of code freely available online has flaws. For example,
much of it features student submissions, one-off explorations, or other low
quality work (Kalliamvakou et al., 2014). Like any trained model, Codex and
other AIDEs are only as good as the training data. And although Codex has done
extensive work filtering low-quality inputs, there remains code that has bugs,
that has technical debt, or that uses outdated APIs. Subtle security holes can
easily persist even in high-quality, high-volume repositories (consider the
OpenSSH Heartbleed incident), and recent work showed how deep learning models
can learn vulnerable code and inject it during autocompletion (Schuster et
al., 2020). AIDE demands that humans inspect its outputs carefully, but if we
use it to create code for a problem we don’t fully understand, we won’t be
able to understand its outputs either.

More worrisome is that the language model reflects the biases that we humans
have. For example, asking Copilot to generate a list of names produces a list
of predominantly English/American names (Fig. 1). Plotting suggestions
generate graphs that fail to accommodate people with color-blindness. This is
of course both a challenge for us, as much as it is for AIDE.

Sociotechnical Questions The IDE revolution produced a now well known paradigm
in computer programming, with continuous integration workflows dominant. But
AIDE will possibly change that as more and more of the work is routinized and
automated. More programmer time will be available for complex problem solving.
But that means our current knowledge of how humans and machines interact will
change. When Facebook rolled out their automated bug repair approach, one of
the biggest challenges was not the technical problem, but rather integrating
the repair bot into the humans that worked with them (Harman and O'Hearn,
2018).

Context and Complexity Mechanization—such as in steel-making or automotive—has
greatly improved productivity at the expense of those humans doing the
routine. AIDEs will likely be no different. To what degree will an AIDE be
able to carefully contextualize the solution for a specific problem? Where is
the line between the routine and simple, and the complex and contextual? Will
AI eventually design and write complex software solutions? Currently being
very clear with AIDE is essential for it to understand the context; but
developing and communicating a clear understanding of the problem is one of
the essentially complex problems in software engineering.

## What’s Next

The AIDE revolution has just begun, leaving open questions on what to expect
in future.

Language Models, Data, and Computational Power The rapid progress in the
capabilities of language models is difficult to quantify. A simple proxy for
it is the increasing number of parameters in the language models presented by
OpenAI in the last few years. In 2018 GPT-1 had 117M parameters. One year
later GPT-2 pushed the boundaries to 1.5B, and in 2020 GPT-3 reached 175B
parameters. Rumors place the next release (GPT-4) at an astonishing 100T
parameters (500 $\times$ GPT-3) (Romero, 2021). Similarly, the amount of
training data available for code-related tasks is increasing every day, as are
the computational capabilities of GPUs. Put together, these advances are
expected to substantially improve the support AIDEs can provide to developers.
To what extent will these improvements be affordable (in money and climate
terms), and accessible (for those with no data centres)?

Improving the Quality of Training Data As previously discussed, one of the
main challenges when dealing with data-driven assistants lies in the quality
of the training data. Manually checking all training instances is just not an
option, but can AI help AI? In other words, can we teach AI what a high-
quality training instance is? Whatever the underlying technology will be,
defining techniques to automatically filter out noisy and flawed training
instances is a cornerstone for AIDEs, and a focus for GitHub’s next iteration
of Copilot.

Code is Not (Just) Text Language models have been proposed in the context of
natural language processing, in which they are fed with a stream of tokens
representing the text to process. However, code is not just text and there is
active research investigating what the best representation is when feeding
code as input to language models. For example, structural information can be
extracted from the code Abstract Syntax Tree (AST) and used to boost the
model’s performance.

Consumer-related Customization While Copilot is able to provide code
recommendations that are tailored for the specific coding task at hand, no
customization is performed when it comes to the developer receiving such
recommendation. However, two developers having a different technical
background, coding history, and skills, may benefit from different
recommendations. For example, more expert developers working on real-time
software are likely to appreciate multi-threading solutions to a given task,
while newcomers may be confused by its usage. Customizing the recommendations
based on the target developer can substantially increase the usefulness of
AIDEs.

AIDE Learning Rate A last point worth discussing is the learning rate we can
expect from AIDEs, namely the pace at which they’ll be able to improve their
capabilities. All the above-discussed points can contribute to that, but it’s
unclear when the AI will be able to pass a _programmer’s Turing Test_ , for
example submitting pull requests that reviewers cannot distinguish from a
human’s submission.

An important truism in software development is Fred Brook’s maxim “there is no
silver bullet”, derived from his insight into essential (inherent) complexity
versus accidental (self-imposed) complexity. Like any new approach to our
challenging field, AIDE is unlikely to become a panacea for software
development. But it does seem to portend an important shift in how we develop
software, and just might remove some of the accidental complexity in our
projects.

## References

  * Chen et al. [2021] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba.  Evaluating large language models trained on code, 2021.
  * Harman and O'Hearn [2018] M. Harman and P. O'Hearn.  From start-ups to scale-ups: Opportunities and open problems for static and dynamic program analysis.  IEEE, Sept. 2018.  10.1109/scam.2018.00009.  URL https://doi.org/10.1109/scam.2018.00009.
  * Hindle et al. [2012] A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu.  On the naturalness of software.  In _Proceedings of the 34th International Conference on Software Engineering_ , page 837–847. IEEE Press, 2012.  ISBN 9781467310673.
  * Kalliamvakou et al. [2014] E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German, and D. Damian.  The promises and perils of mining github.  In _Proceedings of the 11th Working Conference on Mining Software Repositories_ , page 92–101, 2014.
  * Marginean et al. [2019] A. Marginean, J. Bader, S. Chandra, M. Harman, Y. Jia, K. Mao, A. Mols, and A. Scott.  SapFix: Automated end-to-end repair at scale.  IEEE, May 2019.  10.1109/icse-seip.2019.00039.
  * Romero [2021] A. Romero.  GPT-4 will have 100 trillion parameters — 500x the size of GPT-3.  https://bit.ly/3uLakBC, 2021.  Accessed: 2021-11-02.
  * Schuster et al. [2020] R. Schuster, C. Song, E. Tromer, and V. Shmatikov.  You autocomplete me: Poisoning vulnerabilities in neural code completion.  Technical Report arXiv:2007.02220, 2020.
  * Snowden et al. [2020] D. Snowden, Z. Goh, R. Greenberg, and B. Bertsch.  _Cynefin: Weaving Sense-Making into the Fabric of Our World_.  Cognitive Edge, 2020.
  * Ziegler [2021] A. Ziegler.  GitHub Copilot: Parrot or crow? a first look at rote learning in GitHub Copilot suggestions.  https://docs.github.com/en/github/copilot/research-recitation, 2021\.  Accessed: 2021-11-02.

Neil A Ernst is with University of Victoria, Canada. He conducts research into
software design and natural language understanding in software. Contact him at
<EMAIL_ADDRESS>

Gabriele Bavota is with Università della Svizzera italiana, Switzerland. He
conducts research into mining software repositories and recommender systems
for software developers. Contact him at<EMAIL_ADDRESS>