# APISR: Anime Production Inspired Real-World Anime Super-Resolution

Boyang Wang1 Fengyu Yang1,2* Xihang Yu1 Chao Zhang3 Hanbin Zhao3†
1University of Michigan 2Yale University 3Zhejiang University

###### Abstract

While real-world anime super-resolution (SR) has gained increasing attention
in the SR community, existing methods still adopt techniques from the
photorealistic domain. In this paper, we analyze the anime production workflow
and rethink how to use characteristics of it for the sake of the real-world
anime SR. First, we argue that video networks and datasets are not necessary
for anime SR due to the repetition use of hand-drawing frames. Instead, we
propose an anime image collection pipeline by choosing the least compressed
and the most informative frames from the video sources. Based on this
pipeline, we introduce the Anime Production-oriented Image (API) dataset. In
addition, we identify two anime-specific challenges of distorted and faint
hand-drawn lines and unwanted color artifacts. We address the first issue by
introducing a prediction-oriented compression module in the image degradation
model and a pseudo-ground truth preparation with enhanced hand-drawn lines. In
addition, we introduce the balanced twin perceptual loss combining both anime
and photorealistic high-level features to mitigate unwanted color artifacts
and increase visual clarity. We evaluate our method through extensive
experiments on the public benchmark, showing our method outperforms state-of-
the-art anime dataset-trained approaches. The code is available at
https://github.com/Kiteretsu77/APISR.

††footnotetext: † Corresponding author††footnotetext: * works done at
University of Michigan

##  1 Introduction

As an important subdiscipline of real-world super-resolution (SR), anime SR
focuses on restoring and enhancing low-quality low-resolution (LR) anime
visual art images and videos to high-quality high-resolution (HR) forms. It
has demonstrated significant practical impacts in the fields of entertainment
and commerce [56, 46, 61, 54, 48]. An emerging line of work has addressed the
problem by extending SR networks to capture multi-scale information or
learning an adaptive degradation model [56, 46]. We argue these methods lack
understanding of the anime domain as their techniques are directly
transplanted from the photorealistic SR approach.

Figure 1: Comparisons between proposed APISR and other SOTA anime SR methods.
Ours present clearer and sharper hand-drawn lines, better restoration with
more natural details, and do not present unwanted color artifacts. Zoom in for
best view.

In this paper, we thoroughly analyze the anime production process, exploring
ways to leverage its unique aspects for practical applications in anime SR.
The production workflow first starts with hand-drawing sketches on paper,
which are then colorized and enhanced by computer-generated imagery (CGI)
processing [69]. Then, these processed sketches are concatenated into a video.
Due to the fact that the drawing process is extremely labor-intensive and
human eyes are not sensitive to motions [37, 10], it is a standard practice to
reuse a single image across multiple consecutive frames when forming the
video. This procedure in production motivates us to rethink whether it is
necessary and efficient to use video networks and video datasets to train SR
networks in the anime domain.

To this end, we explore the use of image-based methods and datasets as a
unified super-resolution and restoration framework for both anime images and
videos. Creating an image dataset allows us more flexibility to exclusively
choose the least-compressed video frames as our potential dataset pool, rather
than gathering sequential frames that contain temporal distortions to create a
video dataset. Furthermore, by forming an image dataset, we can selectively
focus on the most informative frames, as anime videos typically possess less
information than photorealistic videos. If we randomly crop a patch from an
anime image, there is a high probability that it is a monochromatic area
signifying a lack of information. In light of these phenomena, we introduce an
anime image collection pipeline that focuses on keyframes in video, along with
an image complexity assessment-based selection criteria. This method is
designed to identify and select the least-compressed and the most informative
images from video sources. Using our pipeline, we propose Anime Production-
oriented Image (API) dataset for SR training.

In addition, we identify two new anime-specific challenges for real-world SR
tasks. First, in anime production, the clarity of hand-drawn lines is a highly
emphasized detail [56, 26, 6] as shown in Fig. 2 a, but hand-drawn lines are
easily weakened due to compression in internet transmission and physical aging
in production. This deterioration at the edges of lines exerts a substantial
negative impact on the visual effects. To address this, we start from the
perspective of restoration and enhancement. Concretely, we propose a
prediction-oriented compression module in the image degradation model to
simulate compression in internet transmission such that the model trained with
this self-supervised method can restore hand-drawn line distortions. In
addition, we propose a ground-truth (GT) enhancement approach to enhance
faint, aging hand-drawn lines, by merging hand-drawn lines extracted from the
overly sharpened GT images.

Second, we realize an issue of unwanted color artifacts in anime images, which
is a consequence of employing the GAN-based SR networks [17] (see Fig. 2 b).
These artifacts are presented as irregularly shaped colored spots with varying
intensities that are scattered randomly across generated images, which
significantly undermines visual perception. We attribute this issue to the
reason that image features of perceptual loss are trained on the
photorealistic image datasets, which is inconsistent in the anime domain. To
mitigate this issue, we conduct a comprehensive study of perceptual loss and
introduce balanced twin perceptual loss, which assembles perceptual features
from both the photorealistic domain and the anime domain by a balanced layer
scaling distribution.

Thus, we summarize our contributions as follows:

  * •

We propose a novel anime dataset curation pipeline that is capable of
collecting the least compressed and the most informative anime images from
video sources.

  * •

We propose an image degradation model to deal with harder compression
restoration challenges, especially for hand-drawn line distortions, and the
first methodologies in the anime domain to attentively enhance faint hand-
drawn lines.

  * •

We realize and address the unwanted color artifacts in GAN-based SR network
training caused by the domain inconsistency of the perceptual loss.

  * •

We thoroughly evaluate our method on the real-world anime SR dataset and show
that our method outperforms state-of-the-art anime dataset-trained SR
approaches by a large margin with only 13.3% training sample complexity of the
prior work.

Figure 2:  We identify two new anime-specific challenges: (a) Distorted and
faint hand-drawn lines frequently appear in real-world anime images. (b)
Unwanted color artifacts in AnimeSR [56] and VQD-SR [46]. Zoom in for the best
view.

##  2 Realated Work

#### Real-World Super-Resolution.

Classical SR methods [15, 7, 57, 58] typically employs a straightforward
approach, using a single bicubic downsampling operation to convert high-
resolution (HR) ground-truth (GT) images into their low-resolution (LR)
counterparts. Classical image restoration methods [31, 59, 60, 30] train
different weights for different tasks. In contrast, real-world SR is dedicated
to implementing a sophisticated degradation model by one model weight to
restore the diverse degradations found in the real-world scenario, such as
blurring, noise, and compression [24, 54, 65, 31, 30, 48].

Generic degradation model design can be broadly classified into two
categories: explicit models [24, 54, 65, 30, 48] and implicit models [56, 46,
64]. Explicit degradation models employ kernels and mathematical formulas to
simulate real-world degradation processes. On the other hand, the implicit
degradation models focus on training neural networks to capture the
distribution of real-world degradations. Nevertheless, implicit models face
challenges of interpretability and scalability. The efficacy of implicit
models lacks a clear rationale, and adapting them to new domains requires the
creation of bespoke datasets and extra training complexity.

#### Anime Processing.

Anime represents a distinctive form of visual art, often characterized by
exaggerated visual representation. Creators of anime typically start by
sketching line art, followed by 2D and 3D animation techniques, which include
elements like colorization, CGI effects, and frame interpolation. Notably,
recent research in the realm of anime has garnered substantial attention,
_e.g_., AI painting with anime content [67, 18, 22], vectorization of anime
images [68, 63], anime interpolation and inbetweening [40, 10, 42], anime
sketch colorization [6, 5, 50, 66, 13], 3D representation [41, 11], and anime
domain adaptation [26].

AnimeSR (NeurIPS 2022) [56] and VQD-SR (ICCV 2023) [46] are two recent
representative studies in the domain of real-world anime super-resolution
tasks. However, they have not fully addressed the unique challenges of low-
level anime restoration. This includes the faint hand-drawn lines and domain
inconsistency in the training of GAN-based networks. This paper conducts a
comprehensive exploration of several meticulously crafted approaches to the
anime SR domain.

##  3 Proposed Method

###  3.1 Anime Production-Oriented Image SR Dataset

In this section, we present the API (Anime Production-oriented Image) SR
dataset and its curation workflow. This curation leverages the characteristics
of anime videos to select the least compressed and the most informative
frames.

Figure 3: Histogram of (a) the average image data size comparison between
I-Frames and Non-I-Frames (P and B-Frame) in collected video sources and (b)
image complexity [16] comparison between proposed API and AVC [56] dataset.
Figure 4:  Image Quality Assessment (IQA) with HyperIQA [43] and Brisque [33]
vs. Image Complexity Assessment (ICA) with IC9600 [16]. IQA favors simple
scenes and gives low scores to images with strong CGI. However, ICA is the
opposite.  Figure 5:  Samples of API Super-Resolution Dataset. API includes
versatile CGI effects scenes (_e.g_., different lightning and special effects)
and presents high image complexity.  Figure 6:  The overview of our proposed
methods. (a) We proposed a prediction-oriented compression module in the
degradation model to simulate versatile compression degradations for a single
image input (detailed in Sec. 3.2). Proposed shuffled resize module is
randomly positioned to augment the representation of the degradation model.
(b) GT images are augmented with proposed hand-drawn line enhancement to
promote the generation of images with sharpened line edge details in training
(detailed in Sec. 3.3). (c) Proposed balanced twin perceptual loss avoids
unwanted color artifacts in GAN network training (detailed in Sec. 3.4).

I-Frame-based Image Collection. AnimeSR introduces AVC-Train, the first video-
based anime SR dataset, but they overlook the impact of compression during the
collection process, which leads VQD-SR to propose a post-processing technique
to enhance the dataset. Instead, we propose a novel method to select the least
compressed frames from the source level with minimum effort.

All videos on the internet are compressed and encapsulated with a video
compression standard (_e.g_., H.264 [36] and H.265 [44]) for a trade-off
between the quality and the data size. There are numerous video compression
standards, each with a complex engineering system, but they share a similar
backbone design. This characteristic motivates us to find the pattern that the
compression quality assigned to each frame is different. Video compression
designates some keyframes, known as I-Frames, as individual units for
compression. Empirically, I-Frames are the first frame of scene-changing
scenarios. These I-Frames are allocated with a high data size budget. On the
contrary, a higher compression ratio requires non-I-Frames, namely P-Frames
and B-Frames, to take I-Frames as the reference during compression, which
introduces temporal distortions. As shown in Fig. 3 a, among the anime videos
we collect, I-Frames on average have a much higher data size than other non-I-
Frames, which genuinely stand for higher quality. Thus, we use ffmpeg, a video
processing tool, to extract all I-Frames from the video source as an initial
pool.

Image Complexity-based Selection. To further select idealistic images from the
I-Frames pool, we need some criteria. A straightforward method involves
following AVC-Train to use the Image Quality Assessment (IQA) to rank and
choose frames with better scores. However, IQA ranking does not prefer anime
images with CGI effects but favors simple scenes with little information (see
Fig. 4). Thus, we argue that image complexity assessment (ICA) is a better
option in the anime domain.

ICA evaluates the level of intricacy in an image by scoring the amount and
variety of details present. Compared to IQA, ICA demonstrates greater
robustness against changes in saturation, lightning, contrast, and motion
blurring. The ICA metric we use is a recent rising analysis network, IC9600
[16]. In the anime domain, employing ICA presents two primary advantages.
First, many scenes in anime videos are characteristically monotonous (as
exemplified in Fig. 4 left), where the majority of pixels lack significant
information in training. IQA favors these simple images and gives higher score
compared to other images, but ICA enables the exclusion of these scenes,
which, in turn, contributes to a reduced training sample complexity. Second,
ICA is more adept at identifying meaningful scenes within anime production,
especially those featuring CGI effects, such as the dark scene in Fig. 4
right. These are scenarios where IQA methods typically falter. By collecting
versatile scenes, the network training can become more robust in handling
complex real-world anime inputs.

API Dataset. We began by manually sourcing 562 high-quality anime videos. From
these, we extracted all I-Frames as an initial selection pool. Utilizing the
image complexity assessment method mentioned above, we then selected the top
10 highest-scoring frames from the I-Frames pool of each video. After
discarding inappropriate images (_e.g_., nudity, violence, abnormality, and
anime images mixed with photorealistic content), 3,740 high-quality images are
obtained as our proposed dataset. Example images are shown in Fig. 5.
Moreover, as shown in Fig.3 b, the density of high image complexity scored
frames of our API dataset is remarkably superior to that of AVC-Train. More
analysis and data can be found in the supplementary materials.

720P Back-to-Original Production Resolution. While studying the anime
production pipeline, we observed that most anime productions follow a 720P
format (with an image height of 720 pixels). However, in real-world scenarios,
anime is often falsely upscaled to 1080P or other formats, for the sake of
standardizing multimedia formats. We empirically find that rescaling all anime
images back to the original 720P can provide feature density envisioned by the
creators with more compact anime hand-drawn lines and CGI information.

###  3.2 An Anime Practical Degradation model

In the real-world SR, the design of the degradation model is of great
importance. Based on the high-order degradation model [54] and a recent image-
based video compression restoration model [48], we propose two improvements to
restore distorted hand-drawn lines and versatile compression artifacts and to
augment the representation of the degradation model. Our degradation model is
shown in Fig. 6 a.

Figure 7:  H.264 [36] compression of regular multi-frame video compression and
our proposed single-frame compression. They exert similar degradations
(_e.g_., distortion to hand-drawn lines).

Prediction-Oriented Compression. Utilizing the image degradation model
presents a challenge in the anime restoration of video compression artifacts.
This is because previous real-world image SR methods employ JPEG, an old but
widely-used image compression standard, as the sole compression module in the
image degradation model. JPEG performs repetitive and independent compression
on all encoding units, without considering the existence of other units.
However, video compression algorithms, for higher compression ratios, apply
prediction algorithms to search for a reference with similar pixel content and
only compress their differences (residual), thereby reducing information
entropy. Prediction algorithms can search their reference spatially (intra-
prediction) or temporally (inter-prediction). Regardless of the category, the
intrinsic cause of distortion comes from the misalignment in residual due to
prediction limitation.

Hence, we argue that artifacts equivalent to real-world video compression
artifacts can be synthesized using a single image input in conjunction with a
prediction-oriented compression algorithm (_e.g_., WebP [38] and H.264). The
need for genuinely sequential frames is not necessary. To this end, we design
a prediction-oriented compression module within the image degradation model.
This module requires video compression algorithms to compress inputs on a
single-frame basis. Compared to VCISR [48], we don’t need multiple frames for
one turn of execution of compression. This methodology is theoretically
reasonable and practically viable from an engineering perspective. With a
single-frame input, video compression trivially applies intra-prediction to
compress the frame without using its inter-prediction functionality. Utilizing
this approach, the image degradation model is capable of synthesizing
compression artifacts akin to those observed in conventional multi-frame video
compression as shown in Fig. 7. Subsequently, by feeding these synthesized
images into the image SR network, the system can effectively learn the
patterns of versatile compression artifacts and engage in the restoration.

Shuffled Resize Module. Degradation models in the real-world SR domain
consider blurring, resize, noise, and compression modules. Blurring, noise,
and compression are real-world artifacts that can be synthesized with clear
mathematical models or algorithms. However, the logic of the resize module is
entirely different. Resize is not a part of natural image generation but is
introduced solely for SR-paired dataset purposes. Given this notion, we
believe that previous fixed resize module is not very suitable. We propose a
more robust and effective solution, which involves randomly placing resize
operations at various orders in the degradation model.

###  3.3 Anime Hand-Drawn Lines Enhancement

To enhance faint hand-drawn lines, directly employing global methods, such as
modifying the degradation model or sharpening the entire GT, is not an ideal
approach, as the network cannot learn with attention to hand-drawn line
changes. Thus, we choose to extract sharpened hand-drawn line information and
merge it back with GT to form pseudo-GT. By introducing this attentively
enhanced pseudo-GT to SR training, the network can generate sharpened hand-
drawn lines without the need to introduce additional neural network modules or
separate post-processing networks.

To extract hand-drawn lines, a direct approach is to apply a sketch extraction
model. However, current learning-based sketch extraction is often
characterized by a style transfer to the reference image, which distorts hand-
drawn line details and encompasses unrelated pixel content (_e.g_., shadows
and edges of CGI effects). Consequently, we need a more granular, pixel-by-
pixel methodology to extract hand-drawn lines. Thus, we utilize XDoG [55], a
pixel-by-pixel Gaussian kernel-based sketch extraction algorithm, to extract
edge maps from the sharpened GT. Nevertheless, XDoG edge maps are marred by
excessive noise, containing outlier pixels and fragmented line
representations. To address this ill-posed issue, we propose an outlier
filtering technique coupled with a custom-designed passive dilation method
(detailed in the supplementary materials). In this way, we yield a more
coherent and undisturbed representation of hand-drawn lines.

We empirically find that overly sharpened pre-processed GT makes the hand-
drawn line margins more noticeable than other unrelated shadow edge details,
which makes the outlier filter easier to distinguish their differences. Thus,
we propose three rounds of unsharp masking to the GT first. To sum up, the
formula is as follows:

$\displaystyle I_{\text{Sharp}}=f^{n}(I_{\text{GT}}),$ (1) $\displaystyle
I_{\text{Map}}=h(g(I_{\text{Sharp}})),$ (2) $\displaystyle I_{\text{pseudo-
GT}}=I_{\text{Sharp}}\cdot
I_{\text{Map}}+I_{\text{GT}}\cdot(1-I_{\text{Map}}),$ (3)

where $f$ is the sharpening function that recursively executes $n$ times, $g$
denotes XDoG edge detection and $h$ stands for post-processing techniques of
passive dilation with outlier filtering. $I_{\text{Map}}$ is a binary value
map. The visual pipeline is shown in Fig. 8.

Figure 8: Anime Hand-Drawn Lines Enhancement Pipeline.

###  3.4 Balanced Twin Perceptual Loss for Anime

The existence of unwanted color artifacts is attributed to the inconsistent
dataset domain in training between the generator and perceptual loss.
Currently, most SR models trained with GAN, including AnimeSR and VQD-SR, use
the same ImageNet [14] pre-trained VGG [39] network as the perceptual loss.
However, anime content, particularly those mixed with CGI and extensive
illustrations, differs significantly from photorealistic features in ImageNet.
To tackle this problem, we investigate perceptual loss and the subsequent
improvements made in their following works.

The core idea behind perceptual loss is to utilize high-level features
(_e.g_., segmentation, classification, recognition) to complement low-level
pixel features by comparing middle-layer feature outputs. In this regard, we
employ a pre-trained ResNet50 [21, 3] on anime object classification task with
Danbooru [4] dataset, a substantial and rich tagging anime illustration
database. Since the pre-trained network is ResNet50 instead of VGG, we propose
a similar middle-layer comparison (detailed in the supplementary material).
Overall, the formula is as follows:

$\displaystyle
L^{\phi}_{ResNet}(\hat{y},y)={\sum_{j}}\frac{w_{j}}{C_{j}H_{j}W_{j}}\left|\phi_{j}(\hat{y})-\phi_{j}(y)\right|,$
(4)

where $y$ and $\hat{y}$ are the pseudo-GT by Sec. 3.3 and the generated
images. $\phi_{j}$ represents the perceptual function that returns $j$th layer
output of ResNet50. $C_{j}$, $H_{j}$, and $W_{j}$ are dimensions of the layer
output and $w_{j}$ is the scaling factor for each layer. There are 5 middle-
layer feature outputs, which is the same quantity as VGG-based perceptual
loss. We also observe that the intensity of shallow feature layers in ResNet50
is very weak (see Fig. 9). To resemble a similar intensity balance as the VGG,
we apply a high $w_{j}$ to the early layers, which leads to stable training.

Figure 9:  The second middle-layer feature outputs comparison between VGG19
used by photo-realistic perceptual loss [27] and ResNet50 used by anime
recognition task [3, 4]. With scaling, ResNet50 presents a similar intensity
as the VGG outputs.

Notably, introducing the ResNet-based perceptual loss as the sole perceptual
loss can solve unwanted color artifacts and lead to quantitative improvements.
However, there may be instances of poor visual results. This is attributed to
the inherent bias in the Danbooru dataset, where most images are character
faces or relatively simple illustrations. Hence, we seek a tradeoff by using
real-world features as an auxiliary primer to guide the ResNet-based
perceptual loss in training. This approach results in visually appealing
images and also resolves the unwanted color issue. The overall loss function
for our GAN training is defined as follows:

$\displaystyle L=\alpha L_{1}+\beta L_{per}+\gamma L_{adv},$ (5)
$\displaystyle L_{per}=L_{ResNet}+\delta L_{VGG},$ (6)

where $L_{1}$, $L_{VGG}$, and $L_{adv}$ are L1 pixel loss, photorealistic VGG-
based perceptual loss, and the adversarial loss. $\alpha$, $\beta$, $\gamma$
and $\delta$ are weight parameters.

##  4 Experiment

###  4.1 Implementation Details

In our experiment, we employ our proposed API dataset as the training dataset
for the image network. The image network we utilize is a tiny version of GRL
[30] with the nearest convolution upsample module (detailed in the
supplementary).

To train the GAN, we follow the same two-stage training approach as prior
works [54, 53, 56, 65, 8]. In the first stage, we train the network with L1
pixel loss for 300K iterations. In the second stage, we introduce our balanced
twin perceptual loss and the adversarial loss, conducting an additional 300K
iterations. The weights of $\\{\alpha,\beta,\gamma,\delta\\}$ are
$\\{1,0.5,0.2,1\\}$ respectively. The layer weight of perceptual loss is
$\\{0.1,20,25,1,1\\}$ for ResNet and $\\{0.1,1,1,1,1\\}$ for VGG. Our
discriminator is the same three-scale patch discriminator [23, 51, 35] as in
AnimeSR [56] and VQD-SR [46]. We use the Adam optimizer [28] with a learning
rate of $2\times 10^{-4}$ in the first stage and $1\times 10^{-4}$ in the
second stage. A learning rate decay is applied every 100K iterations in both
stages. The entire training process was carried out on one Nvidia RTX 4090,
with HR patch sizes set at 256x256 and a batch size of 32.

As for the degradation model, we perform degradation on the whole HR image
first rather than directly on a cropped patch as in previous works [54, 30,
65, 48]. Within the degradation model, noise and blurring are configured
identically to Real-ESRGAN [54], and the first prediction-oriented compression
is implemented with JPEG [47] and WebP [38]. The second prediction-oriented
compression includes AVIF [19], JPEG [47], WebP [38], and single-frame
compression of MPEG2 [32], MPEG4 [2], H.264 [36], and H.265 [44]. The
probability of placing the resize module is equally divided among all
positions. Specific parameter settings can be found in our supplementary
materials.

Figure 10:  Qualitative comparisons on AVC-RealLQ [56] for $4\times$ scaling.
Zoom in for the best view.

###  4.2 Comparisons with State-of-the-art Methods

We compare our APISR quantitatively and qualitatively with other SOTA real-
world image and video SR methods, which include Real-ESRGAN [54], BSRGAN [65],
Real-BasicVSR [8], AnimeSR [56], and VQD-SR [46].

Table 1: Quantitative comparisons on AVC-RealLQ [56]. Bold text indicates the
best performance. (‘$*$’ denotes fine-tune on animation videos from [56])

Methods |  Params $\downarrow$ |  NIQE$\downarrow$ |  MANIQA$\uparrow$ |  CLIPIQA$\uparrow$
---|---|---|---|---
Real-ESRGAN* [54] | 16.70 | 8.281 | 0.381 | -
BSRGAN* [65] | 16.70 | 8.632 | 0.376 | -
RealBasicVSR* [8] | 6.30 | 8.621 | 0.362 | -
AnimeSR [56] | 1.50 | 8.109 | 0.462 | 0.539
VQD-SR [46] | 1.47 | 8.202 | 0.464 | 0.567
APISR (Ours) | 1.03 | 6.719 | 0.514 | 0.711

#### Quantitative Comparison.

Following previous real-world SR works [54, 8, 56, 46, 25], we conduct
inference on low-quality LR datasets to generate high-quality HR images and
evaluate them using no-reference metrics. The scaling factor is 4 for all
methods. To validate the effectiveness of our approach, our evaluation is
based on AVC-RealLQ [56], which has 46 video clips each with 100 frames. This
dataset is the only known dataset designed for real-world anime SR testing.
For no-reference metrics, we employ the same metrics used in VQD-SR and
AnimeSR, which are NIQE [34] and MANIQA [62]. We also incorporate other SOTA
learning-based image quality assessment metrics like CLIPIQA [49]. All metrics
are based on pyiqa[9] library.

As shown in Tab. 1, our model has the smallest network size, 1.03M parameters,
but has SOTA performance in all metrics among all image and video-based
methods. Apart from the various proposed methods that contribute to our
success, special acknowledgment is due to the design of the prediction-
oriented compression model, which enables us to train image datasets and image
networks to restore video compression degradations. Meanwhile, it is worth
mentioning that we achieved the result with only 13.3% and 25% of the training
sample complexity of AnimeSR [56] and VQD-SR [46]. This is especially thanks
to the introduction of image complexity assessment in dataset curation which
selects informative images to increase the efficacy of learning the
representation of anime images. Further, we require zero training on the
degradation model due to the explicit degradation model we design.

#### Qualitative Comparison.

As shown in Fig. 10, APISR greatly improves the visual quality than other
methods. In restoring heavily compressed images, our model exhibits
exceptional proficiency than all other methods, as exemplified in the first
row, where we have much fewer ringing artifacts. Moreover, owing to the
proposed hand-drawn lines enhancement, our generated images manifest increased
line density and clarity as observed in the second row. In addressing various
twisted lines and shadow artifacts, our model outperforms others in effective
restoration, evidenced by the third and fourth rows. This is thanks to our
improvement to the image degradation model where we provide a robust
restoration capability on compression and resize functionality. Meanwhile, due
to our proposed balanced twin perceptual loss, images generated by our GAN
network do not show unwanted color artifacts as in AnimeSR and VQD-SR, which
can be seen in the fifth row. Further, thanks to the versatile scenes
collected in our proposed dataset, we are capable of achieving effective
restoration in dark scenes. More visual results can be found in the
supplementary materials.

Table 2: Ablation study results of different training datasets. IQA stands for image quality assessment. ICA stands for image complexity assessment. Dataset | NIQE$\downarrow$ | MANIQA$\uparrow$ | CLIPIQA$\uparrow$
---|---|---|---
AVC-Train [56] | 7.681 | 0.476 | 0.658
Random Select | 8.006 | 0.446 | 0.625
I-Frame + IQA Select | 7.876 | 0.493 | 0.675
I-Frame + ICA Select | 6.912 | 0.499 | 0.683
I-Frame + ICA Select + 720P Rescale | 6.719 | 0.514 | 0.711
Table 3: Ablation study results of different degradation model. Degradation Model | NIQE$\downarrow$ | MANIQA$\uparrow$ | CLIPIQA$\uparrow$
---|---|---|---
High-Order [54] | 6.667 | 0.483 | 0.663
Random Order [65] | 6.975 | 0.491 | 0.674
Prediction-Oriented Compression | 7.133 | 0.506 | 0.709
Compression + Shuffled Resize | 6.719 | 0.514 | 0.711
Table 4:  Ablation study results of hand-drawn lines enhancement denoted as Sharpen and twin perceptual loss denoted as APL.  | NIQE$\downarrow$ | MANIQA$\uparrow$ | CLIPIQA$\uparrow$
---|---|---|---
Plain | 7.351 | 0.501 | 0.689
Plain + Sharpen | 7.182 | 0.504 | 0.707
Plain + Sharpen + APL | 6.835 | 0.512 | 0.708
Plain + Sharpen + APL + Balanced Scale | 6.719 | 0.514 | 0.711

###  4.3 Ablation Study

In this section, we conduct ablation studies to evaluate the substantial
impact of our proposed dataset, degradation model, and hand-drawn lines
enhancement with balanced twin perceptual loss. The inference dataset is still
AVC-RealLQ [56]. Visual comparisons are presented in the supplementary
materials.

#### Impact of the Dataset.

As shown in Tab. 2, we substitute our API training dataset with several
alternatives for comparative analysis: AVC-Train [56], frames randomly
selected from the same video source as our API, a collection of I-Frames with
IQA selection, and a collection of I-Frames with ICA selection. For a fair
comparison, we keep a similar intensity of the training dataset size. If we
take the AVC-Train video training dataset as an image dataset to train, we
include temporal distorted images and less informative frames, which makes the
performance hard to compete with the model trained with API in all metrics.
Randomly selected image datasets perform poorly because they lack attention to
high-quality frames in videos. With our I-Frame collection, we take off
temporally distorted frames and choose the least compressed frames, but IQA-
based selection limits the performance. With the same training iterations and
conditions, the dataset selected by ICA-based criteria leads to an improvement
over the dataset by IQA-based selection. With the 720P rescaling method, anime
images have more compact hand-drawn lines and CGI information than falsely
upscaled versions, and this back-to-original thinking boosts the performance
in all metrics.

#### Degradation Model.

As shown in Tab. 3, to validate the superiority of our degradation model, we
replace our proposed degradation model with the high-order degradation model
from the Real-ESRGAN [54] and random order degradation model from BSRGAN [65],
which share certain similarity as our methods. Our degradation model with
prediction-oriented compression model reaches an outstanding improvement in
MANIQA [62] and CLIPIQA [49] metrics. With our shuffled resize design, our
network becomes more robust to versatile real-world SR scenarios and the
performance can move one step further, especially the NIQE [34] metrics.

#### Benefits of proposed Enhancement and Perceptual Loss.

As shown in Tab. 4, we compare our model with the plain version that is not
trained with proposed hand-drawn lines enhancement and balanced twin
perceptual loss. The introduction of our hand-drawn lines enhancement presents
a significant improvement on CLIPIQA [49]. When we append ResNet perceptual
loss in GAN training, it shows outstanding improvement in NIQE [34]. Further,
with the proposed scaling on the early layers of the ResNet perceptual loss
part, two perceptual losses have reached a stable balance and the performance
moves one step further. This proves that a perceptual loss that is compatible
with the anime domain is very insightful and instructive.

##  5 Conclusion

In this paper, we thoroughly utilize the characteristics of anime production
knowledge and fully leverage it to enrich and enhance anime SR. To be
specific, we propose a high-quality and informative anime production-oriented
image (API) SR dataset with a novel dataset curation design. To restore and
enhance hand-drawn lines, we propose an image degradation model to restore
video compression artifacts and a pseudo-GT enhancement strategy. We further
address unwanted color artifacts by introducing a network trained with high-
level anime tasks to construct a balanced twin perceptual loss. Extensive
experiment results demonstrate our superiority over existing SOTA methods,
where we can restore harder real-world low-quality anime images.

## References

  * Agustsson and Timofte [2017] Eirikur Agustsson and Radu Timofte.  Ntire 2017 challenge on single image super-resolution: Dataset and study.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 126–135, 2017.
  * Avaro et al. [2000] Olivier Avaro, Alexandros Eleftheriadis, Carsten Herpel, Ganesh Rajan, and Liam Ward.  Mpeg-4 systems: overview.  _Signal Processing: Image Communication_ , 15(4-5):281–298, 2000.
  * Baas [2019] Matthew Baas.  Danbooru2018 pretrained resnet models for pytorch.  https://rf5.github.io, 2019.  Accessed: DATE.
  * Branwen and Gokaslan [2019] Gwern Branwen and Aaron Gokaslan.  Danbooru2019: A large-scale crowdsourced and tagged anime illustration dataset.  _Danbooru2017_ , 2019.
  * Cao et al. [2023] Yu Cao, Xiangqiao Meng, PY Mok, Xueting Liu, Tong-Yee Lee, and Ping Li.  Animediffusion: Anime face line drawing colorization via diffusion models.  _arXiv preprint arXiv:2303.11137_ , 2023.
  * Carrillo et al. [2023] Hernan Carrillo, Michaël Clément, Aurélie Bugeau, and Edgar Simo-Serra.  Diffusart: Enhancing line art colorization with conditional diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3485–3489, 2023.
  * Chan et al. [2021] Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Basicvsr: The search for essential components in video super-resolution and beyond.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4947–4956, 2021.
  * Chan et al. [2022] Kelvin CK Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy.  Investigating tradeoffs in real-world video super-resolution.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5962–5971, 2022.
  * Chen and Mo [2022] Chaofeng Chen and Jiadi Mo.  IQA-PyTorch: Pytorch toolbox for image quality assessment.  [Online]. Available: https://github.com/chaofengc/IQA-PyTorch, 2022.
  * Chen and Zwicker [2022] Shuhong Chen and Matthias Zwicker.  Improving the perceptual quality of 2d animation interpolation.  In _European Conference on Computer Vision_ , pages 271–287. Springer, 2022.
  * Chen et al. [2023] Shuhong Chen, Kevin Zhang, Yichun Shi, Heng Wang, Yiheng Zhu, Guoxian Song, Sizhe An, Janus Kristjansson, Xiao Yang, and Matthias Zwicker.  Panic-3d: Stylized single-view 3d reconstruction from portraits of anime characters.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 21068–21077, 2023.
  * Ci et al. [2018] Yuanzheng Ci, Xinzhu Ma, Zhihui Wang, Haojie Li, and Zhongxuan Luo.  User-guided deep anime line art colorization with conditional adversarial networks.  In _Proceedings of the 26th ACM international conference on Multimedia_ , pages 1536–1544, 2018.
  * Dai et al. [2024] Yuekun Dai, Shangchen Zhou, Qinyue Li, Chongyi Li, and Chen Change Loy.  Learning inclusion matching for animation paint bucket colorization, 2024.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Dong et al. [2015] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  _IEEE transactions on pattern analysis and machine intelligence_ , 38(2):295–307, 2015.
  * Feng et al. [2022] Tinglei Feng, Yingjie Zhai, Jufeng Yang, Jie Liang, Deng-Ping Fan, Jing Zhang, Ling Shao, and Dacheng Tao.  Ic9600: A benchmark dataset for automatic image complexity assessment.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Guo et al. [2023] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  _arXiv preprint arXiv:2307.04725_ , 2023.
  * Han et al. [2021] Jingning Han, Bohan Li, Debargha Mukherjee, Ching-Han Chiang, Adrian Grange, Cheng Chen, Hui Su, Sarah Parker, Sai Deng, Urvang Joshi, et al.  A technical overview of av1.  _Proceedings of the IEEE_ , 109(9):1435–1462, 2021.
  * Hati et al. [2019] Yliess Hati, Gregor Jouet, Francis Rousseaux, and Clément Duhart.  Paintstorch: a user-guided anime line art colorization tool with double generator conditional adversarial network.  In _Proceedings of the 16th ACM SIGGRAPH European Conference on Visual Media Production_ , pages 1–10, 2019.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14_ , pages 630–645. Springer, 2016.
  * Huang et al. [2023] Zhengyu Huang, Haoran Xie, Tsukasa Fukusato, and Kazunori Miyata.  Anifacedrawing: Anime portrait exploration during your sketching.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pages 1–11, 2023.
  * Isola et al. [2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1125–1134, 2017.
  * Ji et al. [2020a] Xiaozhong Ji, Yun Cao, Ying Tai, Chengjie Wang, Jilin Li, and Feiyue Huang.  Real-world super-resolution via kernel estimation and noise injection.  In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , 2020a.
  * Ji et al. [2020b] Xiaozhong Ji, Yun Cao, Ying Tai, Chengjie Wang, Jilin Li, and Feiyue Huang.  Real-world super-resolution via kernel estimation and noise injection.  In _proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pages 466–467, 2020b.
  * Jiang et al. [2023] Yuxin Jiang, Liming Jiang, Shuai Yang, and Chen Change Loy.  Scenimefy: Learning to craft anime scene via semi-supervised image-to-image translation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7357–7367, 2023.
  * Johnson et al. [2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14_ , pages 694–711. Springer, 2016.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Lee and Lee [2020] Yeongseop Lee and Seongjin Lee.  Automatic colorization of anime style illustrations using a two-stage generator.  _Applied Sciences_ , 10(23):8699, 2020.
  * Li et al. [2023] Yawei Li, Yuchen Fan, Xiaoyu Xiang, Denis Demandolx, Rakesh Ranjan, Radu Timofte, and Luc Van Gool.  Efficient and explicit modelling of image hierarchies for image restoration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18278–18289, 2023.
  * Liang et al. [2021] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.  Swinir: Image restoration using swin transformer.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1833–1844, 2021.
  * Mitchell et al. [1996] Joan L Mitchell, William B Pennebaker, Chad E Fogg, Didier J LeGall, Joan L Mitchell, William B Pennebaker, Chad E Fogg, and Didier J LeGall.  Mpeg-2 overview.  _MPEG Video Compression Standard_ , pages 171–186, 1996.
  * Mittal et al. [2012a] Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik.  No-reference image quality assessment in the spatial domain.  _IEEE Transactions on image processing_ , 21(12):4695–4708, 2012a.
  * Mittal et al. [2012b] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik.  Making a “completely blind” image quality analyzer.  _IEEE Signal processing letters_ , 20(3):209–212, 2012b.
  * Miyato et al. [2018] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  _arXiv preprint arXiv:1802.05957_ , 2018.
  * Schwarz et al. [2007] Heiko Schwarz, Detlev Marpe, and Thomas Wiegand.  Overview of the scalable video coding extension of the h. 264/avc standard.  _IEEE Transactions on circuits and systems for video technology_ , 17(9):1103–1120, 2007.
  * Shen et al. [2022] Wang Shen, Cheng Ming, Wenbo Bao, Guangtao Zhai, Li Chenn, and Zhiyong Gao.  Enhanced deep animation video interpolation.  In _2022 IEEE International Conference on Image Processing (ICIP)_ , pages 31–35. IEEE, 2022.
  * Si and Shen [2016] Zhanjun Si and Ke Shen.  Research on the webp image format.  In _Advanced graphic communications, packaging technology and materials_ , pages 271–277. Springer, 2016.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Siyao et al. [2021] Li Siyao, Shiyu Zhao, Weijiang Yu, Wenxiu Sun, Dimitris Metaxas, Chen Change Loy, and Ziwei Liu.  Deep animation video interpolation in the wild.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6587–6595, 2021.
  * Siyao et al. [2022] Li Siyao, Yuhang Li, Bo Li, Chao Dong, Ziwei Liu, and Chen Change Loy.  Animerun: 2d animation visual correspondence from open source 3d movies.  _Advances in Neural Information Processing Systems_ , 35:18996–19007, 2022.
  * Siyao et al. [2023] Li Siyao, Tianpei Gu, Weiye Xiao, Henghui Ding, Ziwei Liu, and Chen Change Loy.  Deep geometrized cartoon line inbetweening.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7291–7300, 2023.
  * Su et al. [2020] Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang.  Blindly assess image quality in the wild guided by a self-adaptive hyper network.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3667–3676, 2020.
  * Sullivan et al. [2012] Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand.  Overview of the high efficiency video coding (hevc) standard.  _IEEE Transactions on circuits and systems for video technology_ , 22(12):1649–1668, 2012.
  * Timofte et al. [2017] Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang.  Ntire 2017 challenge on single image super-resolution: Methods and results.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 114–125, 2017.
  * Tuo et al. [2023] Zixi Tuo, Huan Yang, Jianlong Fu, Yujie Dun, and Xueming Qian.  Learning data-driven vector-quantized degradation model for animation video super-resolution.  _arXiv preprint arXiv:2303.09826_ , 2023.
  * Wallace [1992] Gregory K Wallace.  The jpeg still picture compression standard.  _IEEE transactions on consumer electronics_ , 38(1):xviii–xxxiv, 1992.
  * Wang et al. [2024] Boyang Wang, Bowen Liu, Shiyu Liu, and Fengyu Yang.  Vcisr: Blind single image super-resolution with video compression synthetic data.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 4302–4312, 2024.
  * Wang et al. [2023a] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy.  Exploring clip for assessing the look and feel of images.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 2555–2563, 2023a.
  * Wang et al. [2023b] Ning Wang, Muyao Niu, Zhi Dou, Zhihui Wang, Zhiyong Wang, Zhaoyan Ming, Bin Liu, and Haojie Li.  Coloring anime line art videos with transformation region enhancement network.  _Pattern Recognition_ , 141:109562, 2023b.
  * Wang et al. [2018a] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8798–8807, 2018a.
  * Wang et al. [2018b] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Recovering realistic texture in image super-resolution by deep spatial feature transform.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 606–615, 2018b.
  * Wang et al. [2018c] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy.  Esrgan: Enhanced super-resolution generative adversarial networks.  In _Proceedings of the European conference on computer vision (ECCV) workshops_ , pages 0–0, 2018c.
  * Wang et al. [2021] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan.  Real-esrgan: Training real-world blind super-resolution with pure synthetic data.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1905–1914, 2021.
  * Winnemöller et al. [2012] Holger Winnemöller, Jan Eric Kyprianidis, and Sven C Olsen.  Xdog: An extended difference-of-gaussians compendium including advanced image stylization.  _Computers & Graphics_, 36(6):740–753, 2012.
  * Wu et al. [2022] Yanze Wu, Xintao Wang, Gen Li, and Ying Shan.  Animesr: Learning real-world super-resolution models for animation videos.  _arXiv preprint arXiv:2206.07038_ , 2022.
  * Xiao et al. [2020] Zeyu Xiao, Zhiwei Xiong, Xueyang Fu, Dong Liu, and Zheng-Jun Zha.  Space-time video super-resolution using temporal profiles.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 664–672, 2020.
  * Xiao et al. [2021] Zeyu Xiao, Xueyang Fu, Jie Huang, Zhen Cheng, and Zhiwei Xiong.  Space-time distillation for video super-resolution.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2113–2122, 2021.
  * Xiao et al. [2023a] Zeyu Xiao, Jiawang Bai, Zhihe Lu, and Zhiwei Xiong.  A dive into sam prior in image restoration.  _arXiv preprint arXiv:2305.13620_ , 2023a.
  * Xiao et al. [2023b] Zeyu Xiao, Yutong Liu, Ruisheng Gao, and Zhiwei Xiong.  Cutmib: Boosting light field super-resolution via multi-view image blending.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1672–1682, 2023b.
  * Xu et al. [2022] Shizhuo Xu, Vibekananda Dutta, Xin He, and Takafumi Matsumaru.  A transformer-based model for super-resolution of anime image.  _Sensors_ , 22(21):8126, 2022.
  * Yang et al. [2022] Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang.  Maniqa: Multi-dimension attention network for no-reference image quality assessment.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1191–1200, 2022.
  * Yao et al. [2016] Chih-Yuan Yao, Shih-Hsuan Hung, Guo-Wei Li, I-Yu Chen, Reza Adhitya, and Yu-Chi Lai.  Manga vectorization and manipulation with procedural simple screentone.  _IEEE transactions on visualization and computer graphics_ , 23(2):1070–1084, 2016.
  * Yuan et al. [2018] Yuan Yuan, Siyuan Liu, Jiawei Zhang, Yongbing Zhang, Chao Dong, and Liang Lin.  Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 701–710, 2018.
  * Zhang et al. [2021a] Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte.  Designing a practical degradation model for deep blind image super-resolution.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4791–4800, 2021a.
  * Zhang et al. [2021b] Lvmin Zhang, Chengze Li, Edgar Simo-Serra, Yi Ji, Tien-Tsin Wong, and Chunping Liu.  User-guided line art flat filling with split filling mechanism.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9889–9898, 2021b.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023.
  * Zhang et al. [2009] Song-Hai Zhang, Tao Chen, Yi-Fei Zhang, Shi-Min Hu, and Ralph R Martin.  Vectorizing cartoon animations.  _IEEE Transactions on Visualization and Computer Graphics_ , 15(4):618–629, 2009.
  * Zhao et al. [2022] Yang Zhao, Diya Ren, Yuan Chen, Wei Jia, Ronggang Wang, and Xiaoping Liu.  Cartoon image processing: A survey.  _IJCV_ , 2022.

Supplementary Material

In this supplementary material, Sec. A first presents more statistics and
details of our proposed anime image SR training dataset. Then, Sec. B shows
details about our implementations in super-resolution (SR) network training.
Specifically, Sec. B.1 presents the image SR network we used in our training.
Sec. B.2 presents details of post-processing techniques we use on the pseudo-
GT preparation for hand-drawn line enhancement. Sec. B.3 presents figures and
details of the ResNet50 [21] perceptual loss for our proposed balanced twin
perceptual loss. Sec. B.4 provides the hyperparameter setting for our proposed
prediction-oriented compression and shuffled resize module in the degradation
model. Finally, Sec. C provides more visual results of comparisons among SOTA
methods and ablation studies.

##  Appendix A API Dataset Details

Our Anime Production-oriented Image (API) SR dataset contains 3,740 high-
quality and informative images. This quantity is roughly the same quantity as
the previous photorealistic SR training dataset size [54, 65], which includes
DIV2K [1], Flickr2K [45], and OutdoorSceneTraining [52]. The aspect ratio and
resolution information before scaling are shown in Fig. 11.

##  Appendix B Implementation Details

###  B.1 Training Network Details

The generator network we deploy is GRL [30], a SOTA image SR network (CVPR
2023). GRL leverages interconnected relationships within various layers of
image structures through a Transformer-based framework, attaining improvement
in multiple tasks of SR and image restoration. The model we chose is its tiny
version, which has 0.91M parameters. To better adapt the real-world SR task,
we changed its upsampler module from the default pixel shuffle strategy to the
nearest neighbor interpolation with the convolution layer approach, which is
used for the base model version but not for the tiny version in their proposed
methods. We change the upsampler because the nearest neighbor interpolation
with the convolution layer is claimed to show fewer artifacts in the
upsampling process than the pixel shuffle strategy. The final network
parameter is 1.03M, which is the smallest network among all image and video-
based SOTA methods that we compare.

###  B.2 Hand-drawn line enhancement Details

In the hand-drawn line enhancement, we have proposed outlier filter and
passive dilate techniques to obtain a clean XDoG-extracted [55] hand-drawn
line edge map. XDoG is widely used in paired dataset preparation in anime
colorization [6, 5, 50, 22]. The extracted edge map by XDoG is a binary
output, where the white pixel stands for the active edge map region and the
black pixel stands for the unrelated region.

For the outlier filter, we use breadth-first search in eight directions to
recursively detect the surrounding pixels of all white pixels and turn white
pixel regions into black pixels if the total quantity of connected white
pixels is less than the threshold. We empirically set the threshold as 32.

For the dilation, we passively replace the black pixel with the white pixel if
it has more than 3 white pixel neighbors, which is different from independent
kernel-based active dilation methods in [29, 12, 20] that directly spread the
surrounding neighbors to be white pixels if the central pixel is white.
Compared to active dilation methods, our proposed passive dilation is more
concentrated on the hand-drawn lines region instead of covering unrelated
pixel information (see Fig. 13). Thus, we name our methods as passive
dilatation.

In the implementation, we will do an unsharp mask for the whole image first to
increase overall visualization sharpness and then apply two extra turns of
sharpening to the hand-drawn lines specifically based on the pipeline design
mentioned above. More implementation details can be found in our released
code.

###  B.3 Balanced Twin Perceptual Loss Details

As shown in Fig. 12, our proposed middle-layer output comparisons for ResNet50
[21] follow the idea proposed by ESRGAN [53] which compares feature map
outputs before the activation layer. Following VGG-based perceptual loss [27],
we compare the last convolution layer of each stage. There are five middle-
layer output comparisons, which are the same quantity as VGG-based perceptual
loss [27]. Thus, our proposed twin perpetual loss reaches a mutual balance in
training.

Figure 11:  API dataset extra statistics. Figure 12:  The overview of our
proposed middle-layer outputs of ResNet50 [21] perceptual loss trained by
Danbooru dataset [3]. Overall, ResNet50 can be summarized into five stages
which is similar to VGG [39]. $\phi_{j}$ represents the perceptual function
that returns $j$th layer output of ResNet50.  Figure 13:  Comparisons between
active and passive dilation. Our proposed passive dilation is more
concentrated on the hand-drawn line region without producing over-sharpened
pseudo-GT images as in active dilation methods.

###  B.4 Degradation Details

For the prediction-oriented compression module of the degradation model, we
deploy both the image compression with prediction mechanism (_i.e_., WebP [38]
and AVIF [19]) and single-frame video compression. Meanwhile, for the
robustness of the degradation model, we keep the JPEG [47]. The quality factor
range of JPEG, WebP, and AVIF is $[20,95]$ with encoding speed in the range of
$[0,6]$ for WebP and AVIF. The probability of fetching the value in the range
is equal.

For the stability of video compression processing, we choose the widely-used
video processing tools, ffmpeg, to perform the proposed single-frame
compression of MPEG2 [32], MPEG4 [2], H.264 [36], and H.265 [44]. In ffmpeg,
CRF is an engineering system to control the quantization level, and preset is
a speed control mechanism whose setting is directly related to compression
distortions. For MPEG2 and MPEG4, we empirically find that the quality factor
control (-qscale:v) is a better way to control single-frame compression, but
for H.264 and H.265, CRF is a better way to control. For MPEG2 and MPEG4, we
set the quality factor in the range $[8,31]$. For H.264 and H.265, we set the
CRF in the range $[23,38]$ and $[28,42]$ respectively. The preset for all of
them is $\\{slow,medium,fast,faster,superfast\\}$ with probability
$\\{0.05,0.35,0.3,0.2,0.1\\}$.

The first prediction-oriented compression includes JPEG [47] and WebP [38]
with a probability of $\\{0.4,0.6\\}$ respectively. The second prediction-
oriented compression includes JPEG [47], WebP [38], AVIF [19], and single-
frame compression of MPEG2 [32], MPEG4 [2], H.264 [36], and H.265 [44] with
probability of $\\{0.06,0.1,0.1,0.12,0.12,0.3,0.2\\}$ respectively. For the
first resize module, we set the scaling in the range of $[0.1,1.2]$ with
probability $\\{0.2,0.7,0.1\\}$ to scale up, scale down, or remain current
resolution. For the second resize module, we choose the range of $[0.15,1.2]$
with probability $\\{0.2,0.7,0.1\\}$. More implementation details can be found
in our released code.

##  Appendix C More Qualitative Comparisons

In this section, we present more qualitative results to verify the
effectiveness of our APISR among SOTA methods. Moreover, we provide visual
comparisons for the ablation studies.

#### Extra Qualitative Comparisons with SOTA methods.

Fig. 14 and Fig. 15 show extra qualitative comparisons on AVC-RealLQ [56]
datasets for $4\times$ scaling. This includes image-based Real-ESRGAN [54] and
BSRGAN [65], and video-based RealBasicVSR [8], AnimeSR [56], and VQD-SR [46].
Our APISR presents clearer and sharper hand-drawn lines (first example of Fig.
14, first and second examples of Fig. 15, and third example of Fig. 16),
better restoration with more natural details (second and third examples of
Fig. 14, and third example of Fig. 15), and does not present unwanted color
artifacts (first and second examples of Fig. 16).

#### Qualitative Comparisons of Ablation Studies.

Fig. 17, Fig. 18, and Fig. 19 shows the qualitative comparisons of ablations
studies.

As shown in Fig. 17, the network trained with AVC-Train [56] over-sharpens the
grid texture and produces annoying artifacts as denoted by the arrows in the
figure. Similarly, the network trained with the random sampled or IQA-based
sampled dataset can alleviate this artifact but is still hard to completely
remove it. However, when we introduce the ICA-based selection method with
I-Frame dataset collection, this artifact is greatly removed and the generated
image shows more natural details. This is thanks to versatile complex scenes
included in the dataset due to ICA-based selection. With 720P rescaling, fewer
ringing artifacts appear.

As shown in Fig. 18, the network trained with high-order [54] and random order
[65] degradation model presents ringing artifacts, rainbow effects, and color
distortions as denoted by the arrows in the figure. Nevertheless, introducing
our proposed prediction-oriented compression module in the degradation model
promotes the network to greatly restore these problems and generate more
natural details with less distorted hand-drawn lines. Moreover, with the
shuffled resize module in the degradation model, more distortions are restored
and present natural shadow details.

As shown in Fig. 19, the network trained with the plain version presents
unwanted color pixel artifacts and sparse hand-drawn line information as
denoted by the arrows in the figure. With the hand-drawn line enhancement, the
hand-drawn line around the eyes of the character is greatly intensified and
more details are generated. However, the unwanted color pixels still exist and
they are presented as an annoying artifact. With the twin perceptual loss, the
unwanted color pixels are greatly alleviated. Further, with the scaling to
early layers in ResNet perceptual loss, more shadow artifacts and distortions
are restored.

Figure 14:  Qualitative comparisons on AVC-RealLQ [56] for $4\times$ scaling.
Our APISR presents clearer and sharper hand-drawn lines, better restoration
with more natural details, and does not present unwanted color artifacts. Zoom
in for the best view. Figure 15:  Qualitative comparisons on AVC-RealLQ [56]
for $4\times$ scaling. Our APISR presents clearer and sharper hand-drawn
lines, better restoration with more natural details, and does not present
unwanted color artifacts. Zoom in for the best view. Figure 16:  Qualitative
comparisons on AVC-RealLQ [56] for $4\times$ scaling. Our APISR presents
clearer and sharper hand-drawn lines, better restoration with more natural
details, and does not present unwanted color artifacts. Zoom in for the best
view. Figure 17:  Qualitative comparisons of the first ablation study. IQA
stands for image quality assessment. ICA stands for image complexity
assessment. 720P stands for our proposed 720P rescaling. Zoom in for the best
view. Figure 18:  Qualitative comparisons of the second ablation study. Zoom
in for the best view. Figure 19:  Qualitative comparisons of the third
ablation study. Hand-drawn lines enhancement is denoted as Sharpen and twin
perceptual loss is denoted as APL. Balanced Scale presents the early layer
scaling to ResNet perceptual loss. Zoom in for the best view.


11institutetext: Instituto de Física de Cantabria (CSIC-UC). Avda. Los Castros
s/n. 39005 Santander, Spain

# A fast algorithm for simulating light curves of stars at extreme
magnification affected by microlensing.

J.M. Diego<EMAIL_ADDRESS>
We present a fast algorithm to produce light curves of distant stars
undergoing microlensing near critical curves. The need of these type of
algorithms is motivated by recent observations of microlensing events of
distant stars at high redshift and at extreme magnification factors. The
algorithm relies on a low-resolution computation of the deflection field due
to an ensemble of microlenses near critical curves, and takes advantage of the
slowly varying nature of the deflection field to infer the magnification of
the unresolved images.

###### Key Words.:

gravitational lensing – microlensing – dark matter – cosmology

##  1 Introduction

The observation of Icarus in 2018 (Kelly et al., 2018) by the Hubble Space
Telescope (HST) represented the beginning of a new branch of astrophysics: The
study of individual stars at cosmological distances. Icarus, at $z=1.49$
behind the cluster MACS J1149, was the first individual star ever observed
beyond redshift $z=1$ and pushed the record of the farthest star by a factor
$\approx 200$. Icarus was observed thanks to the combined effect of a large
macromodel magnification factor ($\mu\approx 500$) plus the boost given by a
microlens, momentarily intersecting the path of the photons (adding an extra
factor $\mu\approx 4$ to the total magnification), resulting in a net
magnification factor of $\mu\approx 2000$. At magnification factors of
$\mu=2000$, there is a net gain of more than 8 magnitudes, making stars that
would be undetected with current technology bright enough to be detected by
telescopes like HST. This technique allows us to reach a vast volume beyond
redshift $z=1$ comprising thousands of Gpc3. This volume is millions of times
larger than the volume accessible in our local universe without the aid of
lensing, and where we have been so far limited to study a relatively small
number of very bright stars. Stars undergoing microlensing events can be
identified by algorithms designed to search for transients such as supernovae
(SNe). Other stars that are being strongly magnified near critical curves have
been identified in recent years. Behind the galaxy cluster MACS J0416, several
candidate strongly lensed stars have been identified (Rodney et al., 2018;
Chen et al., 2019; Kaurov et al., 2019). Among these, one candidate in
particular, dubbed Warhol at $z=0.94$, has unambiguously identified as a star
(Chen et al., 2019; Kaurov et al., 2019). Like in the case of Icarus, Warhol
shows temporal variations probably due to microlenses. More recently, behind
the galaxy cluster PSZ1 G311, Diego (2019) identified an extremely magnified
star dubbed Godzilla and at redshift $z=2.37$. As opposed to Icarus and
Warhol, Godzilla was not identified due to temporal variations in the flux but
due to its unusually bright appearance, lack of counterimages, and unresolved
nature. Although a previously known object (Vanzella et al., 2020), Godzilla
was originally unrecognized as an extremely lensed star. The high brightness
in Godzilla is interpreted in Diego et al. (2022) as due to the fact that the
star is being observed during an outburst phase (these outbursts can last
decades in the observer frame). Due to its brightness ($m_{AB}\approx 22$),
Godzilla represents the first star at $z>1$ for which a spectrum was obtained
(Vanzella et al., 2020). The MUSE spectrum shows peculiar features also
present in massive luminous blue variable stars like a P-cygni profile (Diego
et al., 2022). An interesting fact regarding Godzilla is that no microlensing
flux fluctuations have been observed so far. This is consistent with the
magnification of the star being in the range of several thousands. In this
situation, as discussed in Diego et al. (2022) and Welch et al. (2022), there
is an effect known as the ”more-is-less” effect, where increassing the
magnification results in more overlapping microcaustics in the source plane.
Crossing one of these microcaustics results in a relatively small change in
flux, since the flux linked to other microcaustics is still significant, and
remain more or less constant during the microcaustic crossing.

The latest example of an extremely magnified star is also the most distant
one. Welch et al. (2022) reports the discovery of Earendel, a strongly
magnified star at a record breaking redshift $z=6.2$, peering into the end of
the reionization epoch. Although unlikely to be a Pop III star, Earendel
represents one of the missing links in the evolution of stellar populations.
Earendel was discovered behind the galaxy cluster WHL0137–08 ($z=0.566$) in
the strongly lensed galaxy Sunrise. This galaxy was first observed in images
from the RELICS program (Coe et al., 2019). Although spectral measurements of
Earendel are still not available, and photometric measurements are relatively
poor, it is believed that Earendel is a star with very low metallicity and
very luminous. Earlier estimates in Welch et al. (2022) would put Earendel in
pair with the most luminous stars known in our local volume (${\rm R}<40$
Mpc). As in the case of Godzilla discussed above, no significant flux
variations are appreciated during the two epochs (3 years apart) where
Earendel was observed. This can be explained by the same ”more-is-less” effect
since the estimated magnification of Earendel is also in the range of
thousands, resulting in multiple overlapping microcautics. However, small flux
fluctuations ($\Delta m\approx$0.5–1 magnitudes) are expected due to
microlensing but one needs to wait for deeper observations with JWST111A JWST
program to observe Earendel is already approved.

The recently launched JWST telescope arrives at the perfect moment to exploit
the recent discovery of extremely magnified distant stars. With its gold-
coated mirrors, JWST will lead us to a new golden age in astronomy, exposing
previously unknown distant objects with unprecedented detail. Some of these
objects will undoubtedly include extremely magnified stars, discovered through
the same technique so successfully proven by Hubble. As mentioned earlier,
observation of these distant stars represents the beginning of a new branch of
astrophysics where we will learn not only about the physics of the first and
second generation stars, but also about the small scale fluctuations in the
lens plane that perturb the magnification of the background stars. An
interesting aspect of extremely magnified distant stars is that the ubiquitous
presence of microlenses prohibits magnification factors much larger than a few
tens of thousands, even for relatively compact stars with sizes of several
solar radii Venumadhav et al. (2017); Diego et al. (2018). This is interesting
since it imposes a natural bias toward brighter stars. Without this
limitation, there would exist a degeneracy between the more common stars that
are fainter but more magnified and stars that are more luminous but less
magnified. A maximum value for the magnification then sets a limit to the
lowest possible luminosity of the star at a given redshift. Hence, JWST will
naturally pick up the brightest stars at high redshifts, offering a clean view
of the massive end of the primeval stellar mass function.

Microlensing events of these distant stars can provide a unique type of
information about the components of the gravitational lens effect. One type of
microlens, stars responsible for the intracluster light (or ICL), will
certainly have an impact on the magnification of distant stars. The role of
these microlenses has been extensively studied in the context of quasar
microlensing (Chang & Refsdal, 1979; Irwin et al., 1989; Esteban-Gutiérrez et
al., 2022), but given the much larger size of quasar accretion discs, the role
of stellar microlenses is significantly much smaller than in the case of
microlensing of distant stars. A background star can act as a pencil beam
scanning through the small scale anisotropies in the lens plane. The smaller
the pencil beam (that is, the smaller the star radius), the better the
resolution one can obtain in the lens plane. Although challenging due to the
ubiquitous presence of the more massive stellar microlenses, with good cadence
it would be possible to detect even rogue planets at the redshift of the lens
(Diego et al., 2018), which are not orbiting a star. High-cadence observations
of very bright stars, such as Godzilla, could potentially reveal the first
planet at cosmological distances ($z>0.4$).

Future studies of lensed distant stars will have to rely on accurate light
curves to interpret the results. For instance, since the maximum magnification
during a microcaustic crossing depends on the radius of the star, these events
can be directly used to set constraints on the radius of the star, and hence
its surface gravity. Since massive stars usually form binaries, the light
curve may reveal the existence of the binary since each star would cross a
microcaustic a different times. Frequent monitoring of the system can be used
to also derive orbital parameters of the system, since the relative distance
between the stars in relation to the microcaustics will depend on the orbital
phase.

For these studies we need accurate and fast computation of light curves with
microlensing events that can later be compared with actual observations.
Although similar tools already exist in the literature, they have often been
applied to regimes where the magnification from a macromodel is small or
negligible (for instance in microlensing within our own Galaxy), or the
magnification is moderate, usually in the range of a few to a few tens, such
as in the case of quasar microlensing. When considering extreme magnification
one needs to consider magnification factors of order $\mu\sim 1000$. Also,
since one wants to simulate light curves that extend several decades, the
simulation needs to be large enough in order to contain the path of a moving
star through a web of microcaustics during that period. One can get a sense of
the requirements from standard ray tracing approaches using simple estimates.
Adopting a typical relative velocity of 1000 km s${}^{-}1$ between the moving
star and the web of microcaustics, a distant star would move $\approx 6$
microarcseconds through the web of microcaustics in five decades. If one wants
to resolve a star with $\approx 10$ R⊙ or $R\approx 3\times 10^{-5}$
microarcsec, the number of pixels in the source plane needs to be of order
${\rm N}\sim 10^{10}$. Since we are considering magnifications factors of
order $\mu\sim 1000$, the corresponding number of pixels in the image plane
needs to be ${\rm N}\sim 10^{13}$. The disk space for the deflection field
would require $\sim 1$ Pb. Although technically doable, the computing
resources for simulating just one light curve would be very demanding.
Alternative more efficient approaches have been devised over the last years
that can cope with the simultaneous demand of large area in the image plane
and high resolution in the source plane. In Venumadhav et al. (2017) the
authors used an approach that exploits ideas from earlier work by Paczynski
(1986); Witt (1993) to track the microimages forming near critical curves. In
a more recent work, Meena et al. (2022) adopts ideas similar to the ones used
in zoom-in N-body simulations to increase the resolution around lensed images.
Both approaches produce robust light curves at high resolution and allows for
efficient computation of the light curves.

In this work we present an approach that differs from the ones listed above.
This method was originally employed to produce the light curves that were used
to interpret the Icarus microlensing event in Kelly et al. (2018) and Diego et
al. (2018). With some delay over this earlier results, (but still consistent
with the time dilation from stars at $z\approx 2$), we present in the next
sections the method used in Kelly et al. (2018) and Diego et al. (2018).

##  2 An efficient algorithm for light curves at extreme magnifications

The algorithm presented in this work takes advantage of the smoothness of the
deflection field and interpolates this field to rapidly find the multiple
images and their magnifications. In the next subsections we discuss the basic
equations from lensing theory and present cartoon examples that help
understand the main ideas behind the algorithm.

###  2.1 Formation of microimages around microlenses

The lens equation is defined in the canonical way

$\bf{\it\beta}=\bf{\it\Theta}-\bf{\it\alpha}(\bf{\it\Theta})$ (1)

where $\bf{\it\beta}$, and $\bf{\it\Theta}$ are angular positions in the
source and image plane respectively, and $\bf{\it\alpha}$ is the deflection
field. Since these fields are all vector fields, we can decompose them into
their cartesian components. In particular we are interested into the two
residual fields, $\bf{\it R}_{x}$ and $\bf{\it R}_{y}$ defined as;

$\displaystyle\bf{\it
R}_{x}=\bf{\it\beta}_{x}-(\bf{\it\Theta}_{x}-\bf{\it\alpha}_{x}(\bf{\it\Theta}))$
(2) $\displaystyle\bf{\it
R}_{y}=\bf{\it\beta}_{y}-(\bf{\it\Theta}_{y}-\bf{\it\alpha}_{y}(\bf{\it\Theta}))$

where the $x$ and $y$ subscripts represents the $x$ and $y$ components of the
fields.

If we consider a small source, for instance a star, with radius ${\rm R}_{*}$,
then images form at those locations where the following condition is met;

$d=\sqrt{\bf{\it R}_{x}^{2}+\bf{\it R}_{y}^{2}}<{\rm R}_{*}$ (3)

Images in the image plane can be found very efficiently through minimization
algorithms by taking advantage of the quadratic nature of $d^{2}$, which
quickly converges to a valley of positions where distances are below some
threshold in $d$. This threshold is typically set to a value larger than a few
times ${\rm R}_{*}$. A second step follows which precisely finds the number
and sizes of all microimages at the subpixel level.

In general, at extreme magnification values ($\mu\leavevmode\nobreak\ 1000$)
near a galaxy or cluster critical curve, the deflection field is a
contribution of a macromodel which produces a deflection field (with slope
close to 1 in the direction of maximum magnification) and very small
perturbations to this field from the microlenses near the critical curve. The
deflection field from microlenses scales as $r^{-1}$ with $r$ being the
distance to the microlens. Hence the perturbations from microlenses are also
smooth, with the exception of very small distances $r<<r_{E}$, where $r_{E}$
is the Einstein radius of the microlens. However, at this very short distances
the magnification factors are very small ($\mu<1$) and can be safely ignored.

Figure 1: Magnification map around a 1 ${\rm M}_{\odot}$ embedded in a
macrolens potential with magnification $\mu_{t}=100$ and $\mu_{r}=2$. The grey
scale shows the magnification with the critical curve around the microlens
forming a standing hour glass figure. The blue near horizontal thin curve
marks the region where the residual $R_{x}$ is nearly zero. The green wider
curve marks the corresponding region where the residual $R_{y}$ is close to
zero. Arcs form in regions where both blue and green curves overlap, that is,
when the condition in equation 3 is satisfied. The yellow rectangle encloses a
region similar to the cartoon representation of Figure 2.

Figure 1 shows a simple example of a microlensed image and how it relates to
the residuals $R_{x}$ and $R_{y}$. For this particular case, the macromodel
magnifies the source by a factor $\mu=\mu_{t}\times\mu_{r}=100\times 2=200$.
Without the microlens there would be one single arc 200 times larger than the
original source. For illustration purposes, in this figure we adopt a source
that is large enough so the arcs can be appreciated visually. In a real
situation where the source represents a star, the lensed microimages would in
general be much smaller than the pixel size. In particular, the pixel size
used for figure 1 is 30 nanoarcseconds which is orders of magnitude larger
than the typical size of a luminous star at cosmological distances. The source
is placed at $z=1.5$ and is modelled as Gaussian with dispersion 30
nanoarcsec. The images are simply given by $exp(-d^{2}/(2\sigma^{2})$, where
$d$ is given by 3. In the presence of the microlens (a star with 1 solar mass
at redshift $z=0.55$), and for the particular position of the source in
relation to the microlens, 3 microimages are formed (marked with arrows in the
figure). The total magnification of the microimages is $\mu=410$. That is, the
microlens boosts the magnification by a factor $\approx 2$. The critical curve
around the microlens is the vertical hour-glass shape. The grey scale in the
plot shows the magnification. The green and blue bands show regions where the
residuals $R_{x}$ and $R_{y}$ are close to zero. In particular, these bands
are given by $exp(-R_{x}^{2}/(2\sigma^{2})$, and
$exp(-R_{y}^{2}/(2\sigma^{2})$ respectively. The product of these two bands is
equal to $exp(-d^{2}/(2\sigma^{2})$, that is, it results in the three
microimages. The important result from this figure is to realize how both
$R_{x}$ and $R_{y}$ vary slowly, and they can be approximated by straight
lines once we consider scales comparable to the pixel scale. Under this
condition, the behaviour of $R_{x}$ and $R_{y}$ can be easily predicted at
scales smaller than the pixel scale, creating the foundation for the algorithm
presented in the next subsection.

###  2.2 Interpolating at the subpixel level

The previous section showed the simple example of a microlens producing three
microimages. The pixels potentially containing microimages can be rapidly
found by restricting the search to pixels where $d^{2}$ is below some
threshold $\epsilon$, for instance $\epsilon=100R_{*}$ or more generally
$\epsilon<1$ pixel, that is a fraction of a pixel. The sample of pixels that
meet this requirement needs to be updated only a few times, since changes in
$d$ are very small when the source position is varied by a distance comparable
to $R_{*}$. Restricting the search of microimages to the subset of pixels that
satisfy $d<\epsilon$, and given the smooth (and linear) dependency of $d$,
$R_{x}$ and $R_{y}$ with the source position, the small subsample of pixels
containing microimages can be easily identified since $R_{x}$ and $R_{y}$ must
contain a zero, i.e, the quantity $R_{x}-R_{y}$ changes sign between two
opposing edges in the pixel. This change of sign can be easily computed by
interpolating neighboring pixels. We refer to the subsample of pixels that
contain microimages as $P_{c}$. After the preselection of $P_{c}$ is
performed, the last element that remains is the determination of the
magnification of the microimages within these pixels.

Figure 2: Representation of $R_{x}$ and $R_{y}$ at the subpixel area. The
squares represent the pixel size. The pixel at position $(x,y)$, and
containing the arc is marked with solid black lines in the center. Neighboring
pixels are marked with dashed lines. The residual $R_{x}$ and $R_{y}$ are
marked with blue and green lines respectively. The thickness of the line
represents the points where $|R_{x}|=|R_{y}|=R_{*}$. The yellow rectangle is
the region represented in greater detail in figure 3

Through a combination of linear interpolations it is possible to determine the
size of the microimage within a pixel and hence the magnification. For
circular sources such as stars, the intrinsic size of the source is given by
$\pi R_{*}^{2}$ and the magnification of the microimage $i$ is given by
$\mu_{i}=A_{i}/piR_{*}^{2}$, where $A_{i}$ is the estimated area of the
microimage $i$. Adding up the magnifications from all individual microimages
we can compute the net magnification for that particular position,
$(\beta_{x},\beta_{y})$, of the source.

Taking advantage of the smoothness of the deflection field we can interpolate
it to find the fraction of a pixel that contains a counterimage. A cartoon
idea of the different steps in the algorithm is given in figures 2 and 3.

The size (or area $A_{i}$) of the microimage can be estimated by computing the
4 points ${\rm H}_{A}$, ${\rm H}_{B}$,${\rm H}_{C}$, and ${\rm H}_{D}$ shown
in figure 3 as the intersecting points between the lines $R_{x}=\pm R_{*}$ and
$R_{y}=\pm R_{*}$.

Alternatively one can use the intersection points ${\rm H}_{1}$, ${\rm
H}_{2}$,${\rm H}_{3}$, and ${\rm H}_{4}$.

The magnification of each microimage is found by computing the ratio of areas.

$A=\frac{A_{p}}{\pi R_{*}^{2}}$ (4)

Where $A_{p}$ is the area of the parallelogram defined by the intersection
points ${\rm H}_{A}$, ${\rm H}_{B}$,${\rm H}_{C}$, and ${\rm H}_{D}$. This
area needs to be corrected by the factor $\pi/4$ to account for the fact that
the lensed image does not occupy the entire area within the parallelogram. The
area $A_{i}$ is then given by;

$A_{i}=d({\rm H}_{3},{\rm H}_{4})\times{\rm h}\times\frac{\pi}{4}$ (5)

where $d({\rm H}_{3},{\rm H}_{4})$ is the distance between these points, and
${\rm h}$ is the distance between the two lines $R_{x}=R_{*}$ and
$R_{x}=-R_{*}$. This distance is given by ;

$h=\frac{|y_{1}-y_{2}|}{\sqrt{1+S_{x}^{2}}}$ (6)

where $S_{x}$ is the slope of the line $R_{x}=0$ and $y_{1}$, $y_{2}$ are
constants in the relation $y=y_{1,2}+S_{x}\,x$ which are determined through
interpolation. The point $y_{1}$ is obtained as the point where $R_{x}=R_{*}$
between pixels (i,j+1) and (i,j-1). Similarly, the point $y_{2}$ can be
obtained as the position where $R_{x}=-R_{*}$ between the same pixels.

Figure 3: Cartoon representation of the algorithm. Points H1 through H4 are
found through interpolation of the smooth ${\rm R}_{\rm x}$ and ${\rm R}_{\rm
y}$ 2-dimensional fields, and determining the intersection points at ${\rm
R}_{\rm x,y}=0,\pm{\rm R}_{*}$. The scale of the pixel is not shown but is
assumed to be larger than the size of this figure (see figure 2).

##  3 Comparison with ray-tracing

In order to test the performance of the algorithm we compute the deflection
field with a pixel scale of 30 nanoarcsec per pixel and place a microlens in
the centre with 1 $M_{\odot}$ (see figure 1). For the background star we adopt
a radius of 20 $R_{\odot}$, that is, the star is $\approx 2500$ times smaller
in area than the pixel. The star moves in a horizontal direction with respect
to the microlens shown in figure 1 and forms microimages along the $R_{y}=0$
region shown in the same figure. We compute the net magnification at each
source position by adding the magnification of each microimage using the
algorithm described above. The result is shown in figure 4 as a black solid
line. In a regular laptop, it takes less than a minute to compute this light
curve (1000 varying star positions in the source plane). For comparison we
show as a red dashed line the result obtained from the brute force approach of
inverse ray shooting, where pixel positions in the image plane are projected
back into the source plane. The standard ray shooting technique can not
resolve the caustics well and predicts smaller magnification values. In
contrast, our fast algorithm is able to properly resolve the caustics even for
sources as small as $R=20\,R_{\odot}$.

We also test the ability of the new algorithm to resolve small microlenses. We
place a small microlens with one Jupiter mass (or $\approx 0.001M_{\odot}$)
near the caustic at $\approx 0.1\mu$arcsec. In the image plane, this small
microlens is at a physical distance of $\approx 0.1$ parsec from the larger
microlens so it should be interpreted as a rogue planet not gravitationally
bounded to the larger microlens. The critical curve of the Jupiter mass
microlens is amplified by the combined magnification of the $1\,M_{\odot}$
microlens and the galaxy (or cluster) macrolens. In figure 5 we show the light
curve when the path of the background star intersects the small microcaustic
from the rogue planet. For this particular case we have considered an even
smaller background star with radius $R=5\,R_{\odot}$. The new algorithm is
able to resolve both, the smaller Jupiter mass microlens and the smaller
background star. As in figure 5, the red dashed line corresponds to the
standard ray shooting technique that can not resolve the small microlens and
fails at reproducing the magnification during the caustic crossing. The
example shown in figure 5 serves also to illustrate the limitations of the new
algorithm since small artifacts can be appreciated near the Jupiter mass
microcaustic. In this case, the assumption made about the linear behaviour of
the deflection field within the 30 nanoarcsec simulation pixel is not as
accurate as in the case of the larger microlens. To better reproduce this
case, a cubic interpolation or a smaller pixel size may be necessary.
Nevertheless, this test is interesting to show that the algorithm can be used
to efficiently produce light curves when a wide range of microlens masses are
involved. Additional light curves can be found in the original paper that made
use of this algorithm for the first time (Diego et al., 2018).

Figure 4: Comparison with standard ray shooting. The solid line shows the
result from our method for a star with $R=20\,R_{\odot}$. For comparison we
show as a red dotted line the result obtained from the ray-shooting method,
with a pixel scale similar to the one used for the simulation in the image
plane (30 nanoarcseconds). The trajectory of the star intersects the
microcaustic horizontally. The microlens has one solar mass and microimages
form along the curve $R_{y}=0$ shown in figure 2.

##  4 Conclusions

We have presented a new algorithm to compute light curves of stars grazing a
field of microcaustics near the caustic of a galaxy or cluster macrolens. In
these scenarios, given the large magnification factors, $\mu_{m}$, from the
macrolens (typically hundreds to thousands), the simulated region in the image
plane needs to be relatively large. In the source plane, this large region
gets compressed by the factor $mu_{m}$. If a luminous star is moving at a
typical relative velocity of $\sim 1000$ km s-1 with respect to the micro-
caustic network, it can cover $approx1$ microarcsecond in the source plane
during 10 years. The corresponding microimages would span a region $\mu_{m}$
times larger in the image plane. This means that in realistic scenarios the
simulated region must cover approximately 1 milliarcsecond in the image plane.
If one wants to resolve the magnification of stars (with typical sizes $\sim
10^{-11}"$) with standard ray shooting techniques the number of pixels to be
simulated would be prohibitive. In this work we present an algorithm that
relies on the smoothness of the deflection field in order to resolve
microimages at the sub-pixel level. Using simulations with a pixel size that
is orders of magnitude larger than the size of the stars, we show how it is
possible to quickly produce light curves of a star moving across a web of
microcaustics. The method is fast (less than one minute in a regular laptop
for 1000 positions in the source plane) and accurate. We show how stars at
cosmological distances as small as $5\,R_{\odot}$ can be resolved when the
original simulation is done with a 30 nanoarcsecond pixel. We also show how
small microlens with masses comparable to planets can be also resolved by the
algorithm.

Figure 5: Zoom in near a caustic with a rogue planet perturber. This plot
shows a small portion of the critical curve for the same trajectory and
microlens as in figure 2 but where a small perturber with mass equal to one
Jupiter mass intersects the trajectory of the counterimages near the right
peak at $\approx 0.1$ microarcsec in figure 2. For this plot the background
star is also smaller with radius $R=5\,R_{\odot}$. The red dotted lines shows
the result obtained with the ray shooting method that does not resolve the
small mass perturber.

###### Acknowledgements.

J.M.D. acknowledges the support of project PGC2018-101814-B-100
(MCIU/AEI/MINECO/FEDER, UE) Ministerio de Ciencia, Investigación y
Universidades. This project was funded by the Agencia Estatal de
Investigación, Unidad de Excelencia María de Maeztu, ref. MDM-2017-0765.

## References

  * Chang & Refsdal (1979) Chang, K. & Refsdal, S. 1979, Nature, 282, 561
  * Chen et al. (2019) Chen, W., Kelly, P. L., Diego, J. M., et al. 2019, ApJ, 881, 8
  * Coe et al. (2019) Coe, D., Salmon, B., Bradač, M., et al. 2019, ApJ, 884, 85
  * Diego (2019) Diego, J. M. 2019, A&A, 625, A84
  * Diego et al. (2018) Diego, J. M., Kaiser, N., Broadhurst, T., et al. 2018, ApJ, 857, 25
  * Diego et al. (2022) Diego, J. M., Pascale, M., Kavanagh, B. J., et al. 2022, arXiv e-prints, arXiv:2203.08158
  * Esteban-Gutiérrez et al. (2022) Esteban-Gutiérrez, A., Mediavilla, E., Jiménez-Vicente, J., et al. 2022, ApJ, 929, L17
  * Irwin et al. (1989) Irwin, M. J., Webster, R. L., Hewett, P. C., Corrigan, R. T., & Jedrzejewski, R. I. 1989, AJ, 98, 1989
  * Kaurov et al. (2019) Kaurov, A. A., Dai, L., Venumadhav, T., Miralda-Escudé, J., & Frye, B. 2019, ApJ, 880, 58
  * Kelly et al. (2018) Kelly, P. L., Diego, J. M., Rodney, S., et al. 2018, Nature Astronomy, 2, 334
  * Meena et al. (2022) Meena, A. K., Arad, O., & Zitrin, A. 2022, arXiv e-prints, arXiv:2203.08131
  * Paczynski (1986) Paczynski, B. 1986, ApJ, 301, 503
  * Rodney et al. (2018) Rodney, S. A., Balestra, I., Bradac, M., et al. 2018, Nature Astronomy, 2, 324
  * Vanzella et al. (2020) Vanzella, E., Meneghetti, M., Pastorello, A., et al. 2020, MNRAS, 499, L67
  * Venumadhav et al. (2017) Venumadhav, T., Dai, L., & Miralda-Escudé, J. 2017, ApJ, 850, 49
  * Welch et al. (2022) Welch, B., Coe, D., Diego, J. M., et al. 2022, Nature, 603, 815
  * Witt (1993) Witt, H. J. 1993, ApJ, 403, 530