††thanks: This manuscript has been authored by UT-Battelle, LLC under Contract
No. DE-AC05-00OR22725 with the U.S. Department of Energy. The publisher, by
accepting the article for publication, acknowledges that the U.S. Government
retains a non-exclusive, paid up, irrevocable, world-wide license to publish
or reproduce the published form of the manuscript, or allow others to do so,
for U.S. Government purposes. The DOE will provide public access to these
results in accordance with the DOE Public Access Plan
(http://energy.gov/downloads/doe-public-access-plan).

# Analyzing Machine Learning Performance in a Hybrid Quantum Computing and HPC
Environment

Samuel T. Bieberich  Texas A&M University, Wisenbaker Engineering Building
3128, 188 Bizzell St, College Station, TX 77843  Michael A. Sandoval  National
Center for Computational Sciences, Oak Ridge National Laboratory, P.O. Box
2008, Oak Ridge, TN 37831-6164, USA

###### Abstract

We explored the possible benefits of integrating quantum simulators in a
“hybrid” quantum machine learning (QML) workflow that uses both classical and
quantum computations in a high-performance computing (HPC) environment. Here,
we used two Oak Ridge Leadership Computing Facility HPC systems, Andes (a
commodity-type Linux cluster) and Frontier (an HPE Cray EX supercomputer),
along with quantum computing simulators from PennyLane and IBMQ to evaluate a
hybrid QML program – using a "ground up" approach. Using 1 GPU on Frontier, we
found $\sim$56% and $\sim$77% speedups when compared to using Frontier’s CPU
and a local, non-HPC system, respectively. Analyzing performance on a larger
dataset using multiple threads, the Frontier GPUs performed $\sim$92% and
$\sim$48% faster than the Andes and Frontier CPUs, respectively. More
impressively, this is a $\sim$226% speedup over a local, non-HPC system’s
runtime using the same simulator and number of threads. We hope that this
proof of concept will motivate more intensive hybrid QC/HPC scaling studies in
the future.

††preprint: AIP/123-QED

##  I Introduction

Ever since Alan Turing’s landmark paper "Computer Machinery and Intelligence"
in 1950turing , the field of computing has experienced unprecedented growth.
Seventy-three years later, High Performance Computing (HPC) machines, often
called "supercomputers," operate on the Exascale (1 quintillion operations per
second)exascale , and Quantum Computing has morphed from a science fiction
pipe dream to near-term realitysupremacy . With this growth in computing
power, it is only natural to look back at the initial propositions of
scientists like Turing and question whether computers can be programmed to
"think" in the same way as the human mind. While simulating the methods behind
the most complicated biological construct on Earth often has been viewed as
futility, the latest developments in Machine Learning have asserted that this
is not the case with modern computing equipment.

Machine Learning is a sub-field of Artificial Intelligence, or AI, a branch of
computer science that studies the ability of computers to emulate the
intellectual processes of human beings, such as (as per Britannica) "the
ability to reason, discover meaning, generalize, or learn from past
experience."britannica  Common applications include in Google Search
algorithms, Autocorrect, and traffic predictions.

Machine Learning, and in particular the method we utilized most in this
project, Convolutional Neural Networks (CNNs), allow computers to utilize data
to recognize patterns that would otherwise require an immense amount of hard
coding. For example, one of the most famous uses of CNNs is in image
recognition. By creating what is called a "neural network" composed of various
layers, each with different weights, linear algebra operations through the
network allow the machine to recognize the pixel patterns appearing in
different files and sort them into groups. After this is completed, accuracy
is determined by reading the labels of all of the photos through a process
called "supervised learning"pytorch-docs . The process is repeated by stepping
backward and then forwards again through the network, manipulating weights
until final accuracy across the widest range of data is optimized.

While modern HPC infrastructure is currently the primary method for which ML
programs are run, a promising option for the next era of the industry is
Quantum Machine Learning (QML). As datasets grow in the coming decades, the
utilization of Quantum Computers (QC), often considered critical in the next
generation of high-performance computing, is a natural progression of
computing power that Machine Learning (ML) will require. Quantum Computers
specifically are well equipped to handle processes like those in CNNs,
optimized for computing linear algebra matrix calculations and weighted cost
functions. They also are much more scalable than current HPC infrastructure,
slowed by the steady decline of Moore’s Lawmoore-end . Given this over-
idealized outlook, the question remains, why would HPC need to be involved at
all?

Quantum Computers can’t solve all computable problems, and while classical
computers share this fact, they have a much broader repertoire, easily
explained by their 6 added decades of relevance and R&D. The lack of Turing
Complete Quantum Computers prevents them from accurately simulating classical
architecture, meaning that, at least for the foreseeable future, HPC and QC
will be best utilized in tandem, rather than completely replacing one or the
othermeetiqm .

This project aims to confirm the modern use cases of hybrid quantum and
classical machine learning algorithms and analyze the practicality of running
said programs on real HPC hardware, serially and in parallel. We ran tests
from several programs on the computers at Oak Ridge Leadership Computing
Facility (OLCF) at Oak Ridge National Lab, setting benchmarks and determining
the current computational feasibility of hybrid HPC and QC programs in the
field of ML.

##  II Baseline ML Tests on HPC resources

To establish a suitable standard for evaluating hybrid algorithms, our initial
step involved conducting multiple tests based on PyTorch’s documentation,
specifically the "Transfer Learning for Computer Vision Tutorial"pytorch-docs
. PyTorchpytorch-citation  is a Python library that specializes in ML,
offering various datasets and tools to allow for training using CNNs, among
other methods, either locally on your personal computer or remotely on servers
or HPC clusters. As stated before, CNNs are often used for image recognition
algorithms, but oftentimes training a dataset is difficult when it is
sufficiently small. Transfer Learning takes this into account by taking the
training from the top layers of a large CNN, and then re-appropriating it for
a related taskTL . The final layers are then completed again, allowing for a
relatively accurate and fast result (given the correct dataset). For the first
example explored until Section III.3, the dataset is composed of approximately
240 images of ants and bees. Compared to ImageNetimagenet , which has 14
million images, this would normally be not enough data for a computer to train
off of. However, since this is just a small subset of ImageNet, the PyTorch
code uses the top layers from the main training dataset the prepare the stage
for the final, decisive transfer learning layers. Note that entering Section
III.3, we expand the dataset further by including more images from ImageNet.

Figure 1: Changes in runtime due to the backend used and training method
(Section II).

We started with running the program on a local laptop to compare the final
accuracy and runtimes for the two TL learning methods, "fine-tuning" the
Convolution Network, and fixed feature testing. Fine-tuning removes the final
layer and adds in the pre-trained model from ImageNet, whereas the fixed
feature extractor freezes the previous layers while finding the weights from
the larger datasetfixed-feature . This method takes less time with a slight
benefit to the final accuracy of the network (see Figure 1).

Notice that the tests in this section were run on four different backends: an
Andes CPU compute node, an Andes GPU node, an Andes CPU on a GPU node, and the
local laptop used for the experiments, an Apple Macbook Pro (2.3 GHz Quad-Core
Intel Core i7). AndesAndes-compute-nodes  is a Linux Cluster located at OLCF.
It hosts 704 batch (CPU) compute nodes, as well as 9 GPU compute nodes. This
program predictably ran much faster on Andes, particularly on the GPU node,
regardless of the backend. However, due to the limited amount of GPU nodes on
Andes, this approach is less feasible for larger training sets that require
distributed training methods.

Once affirming that these tests could be run locally and remotely on Andes, we
moved on to the hybrid program.

##  III Running HPC-QC Hybrid Programs

###  III.1 Pathfinding

In the second part of our work, we transitioned to a hybrid PennyLanepennylane
(v0.32.0) and PyTorch (v1.13.0) program using the same ants and bees subset
from ImageNet. PennyLane is a Python library, written and maintained by
Canadian quantum photonics company Xanadu***https://www.xanadu.ai/, which has
built a reputation for reliably working with several quantum cloud backends,
including those from IBMQ. It also features a large repository of QML
documentation, which we utilized for this project.

Figure 2: Accuracy (blue, left axis) and runtime (orange, right axis) versus
simulator used. (4 qubits, 30 epochs on Andes)

Figure 3: Accuracy (blue, left axis) and CPU runtime (orange, right axis)
versus qubits. (30 epochs on Andes CPU)

The hybrid program follows the same basic principles as the original PyTorch
code but instead allows the quantum qubits to do most of the work. After being
initialized to quantum states depending on the original training input, they
are put through 3D rotations and entanglements. These allow the qubits to
fundamentally follow the same procedures as their classical counterpart.

Using the code from PennyLanepennylane-docs , we set out to run the program
while modifying various parameters to determine the most optimal way to
balance the hybrid program. Across all the classical compute systems we
utilized, the three main code parameters explored were: 1) the quantum
backend, 2) the epoch count, and 3) the qubit count.

From a quantum backend perspective, we first tested several of the most common
quantum computing simulators to which PennyLane has access and compared the
accuracy and runtime of the quantum circuit, as exhibited in Figure 2. The
three PennyLane simulators we tested, default.qubit, lightning.qubit, and
lightning.gpu (run on an Andes GPU node) each had identical accuracy at
approximately 94%, with default.qubit having the shortest runtime by a
significant margin. However, switching to other resources caused the runtime
and accuracy to plummet in efficiency. Although the lightning.gpu did not
perform well on Andes, we suspect it would have performed better when scaling
across multiple GPUs and when used on a more GPU-centric machine (such as
Frontier); however, Frontier is unable to run the lightning.gpu device due to
lack of AMD support. Both the lightning and lightning.gpu devices are also
known to perform better with larger qubit counts than the 4 qubit tests we
performed O’Riordan_2022 .

Figure 4: Accuracy (blue, left axis) and CPU runtime (orange, right axis)
versus total epochs. (4 qubits on Andes CPU)

Exploring outside of PennyLane simulators, we tested IBM’s AER simulator
qiskit2024 . Accessed locally, it still took nearly ten times longer than the
default PennyLane simulator, and it operated at a less stable and lower
accuracy of approximately 93%. Switching to devices accessed over the Cloud,
we attempted jobs on IBMQ’s QASM simulator qiskit2024 . Unfortunately, we
determined that this was not possible, as the number of jobs the program
needed to submit to the simulator was far too much to work on one node of
Andes feasibly. Each time we exceeded the apparent IBMQ limits, as jobs
started to fail approximately 5 hours in (18,000-19,500 seconds), or slightly
less than one epoch through. The training algorithm used performs thousands of
tensor-based (or array-based) calculations, and by relying on the quantum
computer for each of them, the runtime is significantly increased. IBM also
uses a queue for their QASM simulator, like their quantum computing devices,
which though short (1-5 seconds), prevents the ML algorithm from completing in
a reasonable amount of time. We additionally tried to run on other IBMQ
backends before realizing the unfeasibility of this as well. The average
remote IBMQ queue time for a user at OLCF is close to 10 hours for one job,
and due to the fair-share queue policyfair-share , it can increase for
everyone in the project if many jobs are submitted. Thus, we deemed that this
QML training program could not run in a computationally reasonable amount of
time on a remote quantum computer (at least in our code’s current state).

The remaining code parameters that we independently adjusted were the epochs
and qubits. Epochs define how many times the training iterates through the
algorithm, increasing accuracy at the expense of runtimeepoch-define , while
qubits are the basic representation of encoded quantum data. When increasing
the number of qubits, the same limitations are present as increasing epochs.
It is worth noting that some of the most powerful quantum processors today
range from 30-430 qubits†††https://quantum-
computing.ibm.com/services/resources?tab=systems, but often feature long queue
times waiting for jobs to run, so we wanted to test and determine how
detrimental this would be for a basic QML program.

While keeping the qubits at the default value, 4, we increased epochs until we
saw a significant increase in runtime without a change in accuracy. This
occurred at approximately 120 Epochs, as per Figure 4. It is worth noting that
the recommended 30 Epochs operate 4 times faster with a 4% decrease in
accuracy, confirming that this could also be a sufficient benchmark for the
variation in simulated qubit count.

The qubits offered less expected results. Decreasing the qubit count to 3
(from the default 4) decreased runtime and increased accuracy. Increasing
qubit count steadily increased the accuracy to a plateau at approximately 96%,
but after 6 qubits, there was a greater gradient concerning runtime (Figure
3). Simulating with much more than 10 qubits (15-20) did not increase accuracy
at all, and runtime only continued to exponentially increase until stopping at
20, where errors prevented the program from running fully.

Note that, until this point, we were purely invoking the Python program via
python script.py  and not explicitly mapping to a specific number of cores or
threads on any system. In the next section we transition to distributed
training, thus enabling us to take more advantage of a given system
(especially the HPC systems).

###  III.2 Parallelization

Figure 5: Qubit runtime by system. Execution time for $\rm N=1$ (solid) and
$\rm N=8$ (dashed) threads across all systems.

The third part of our work focused on distributing the QML training across
multiple threads – either across multiple CPU cores or GPU devices. For this
suite of tests, we added FrontierFrontier-compute-nodes  – the world’s first
exascale computer – into our list of test systems. PyTorch is commonly
utilized on NVIDIA GPUs, meaning that a different build method was needed to
run the code on Frontier’s
AMD‡‡‡https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#amd-gpus
GPUs. Although pre-compiled binaries are available with AMD support, we
noticed a slight performance increase on Frontier when building from source
instead. Implementing the QML code required combining what we already had used
with a Distributed Data-Parallel (DDP) training method. Using DDP and a
distributed sampler, we were able to implement a workflow that allowed the
code to train across multiple CPUs, GPUs, and compute nodes. Note that due to
the small ants and bees dataset, multiple compute nodes were not needed in
this scenario, but are needed as the dataset is increased (see Section III.3).
While we explored both CPU and GPU distribution on Frontier, CPU training was
the focus on Andes due to the limited nature of the GPU compute nodes. We also
tested a similar workflow on the laptop with its multiple compute cores. These
tests were conducted using the default.qubit simulator from Pennylane to yield
results that are more comparable to the previous ones.

For all systems used (see Figure 5), we tested the performance of the code
when using $\rm N=$ 1 thread (effectively non-distributed training) compared
to $\rm N=$ 8 threads. The 8 threads are a convenient choice due to the
laptop’s maximum thread count (4 cores, two threads per core) and the maximum
number of GPUs per node on Frontier (4 GPUs, with 2 GCDs per GPU). For our
purposes, all GCDs on Frontier will be referred to as "GPUs". On the HPC
systems, distribution was achieved through the srun parallel job launcher,
while PyTorch’s native torchrun was used instead for the laptop slurm . For
tighter control over thread mapping on the HPC systems, MPI was utilized via
the mpi4py package mpi4py .

Figure 6: Execution time by qubit count and number of GPUs on Frontier. (30
epochs, smaller dataset)

As expected, the laptop performs well when only considering one compute core
(even outperforming Andes), but falls short when starting to distribute across
multiple cores. Frontier’s GPU performs the best in the $\rm N=1$ regime, but
has a trend that will eventually be outperformed by the Frontier CPU at larger
qubit counts. When considering $\rm N=8$, Frontier’s CPU starts to perform
quicker than the GPU at 6 qubits. The loss of performance at larger qubit
counts is a bit concerning, as the slope of the Frontier GPU performance is
drastically different when compared to other methods. This may be attributed
to GPU communication overheads in the quantum circuit every training iteration
(which worsens as the number of qubits is increased), or an underlying
communication issue between CPU and GPU torch devices in the circuit due to
using the default.qubit simulator. This may be mitigated by using the tailored
lightning.gpu simulator; however, it is currently unable to be installed on
Frontier due to hardware compatibility issues. The new lightning.kokkos
simulator, which should be compatible, was also not an option at the time of
testing. Due to the code hanging at certain GPU counts and not hanging when
running purely on the CPU with an equivalent number of threads, an underlying
GPU communication issue with the code seems highly likely. One other aspect is
that our code was never designed to be run at larger qubit counts and is thus
inefficient in that region. Even with Frontier’s GPU trend being eventually
surpassed by the Andes CPUs at larger qubit counts, all HPC systems maintain
better performance (including their slopes) than the laptop approach at $\rm
N=8$. The laptop results emphasize that although one can run this program
without HPC resources, laptop performance quickly falls behind as more and
more threads are needed for distributed training (further emphasized with a
larger dataset in Section III.3).

Figure 7: Accuracy of the smaller dataset versus number of GPUs on Frontier.
(4 qubits, 30 epochs, smaller dataset)

Note that the discrepancy between the Andes times listed in Figure 5 and
Figure 3 are due to now explicitly invoking 1 or 8 threads via srun as opposed
to purely doing  python script.py . The former leads to more consistent and
precise optimization, while the latter approach had inconsistent optimization
across compute cores and is limited to 1 compute node.

Focusing more on Frontier’s performance, we benchmarked a single compute node
across our qubit range. As exhibited in Fig. 6, the most optimal execution of
the circuit, in terms of runtime, is on 8 GPUs with 3 qubits. When utilizing
the full power of a compute node on Frontier with all 8 GPUs, the runtime
plateaus across all qubit counts. This is explained by the number of training
iterations being small due to the relatively small dataset, so adding more
GPUs to distribute the training provides less benefit. Note that at larger
thread counts, the distributed training masks the effect that larger qubit
counts have on runtime, while the quantum portion of the computations has more
of an impact at lower thread counts.

One thing to consider is that other hyperparameters, such as batch size and
layers, also present issues. In Section III.1, we used the default circuit
depth of 6 and batch size of 4, and we used these same benchmarks for
consistent comparative testing in Section III.2; however, in reality, these
are both very small values. CNNs often have a base of 15 layers but at this
point the Pennylane default.qubit simulator would have significant trouble
computing all of the rotations necessary on the input qubits. Remote backends
such as those used by IBMQ and Quantinuum may not be able to run those
circuits with success either due to their depth. The best quantum computers on
the market have high accuracy at a circuit depth range of only 10-20Q-Volume .
This means that any large QML datasets are best trained on simulators until
stronger, more coherent quantum computing resources are available, or when
better algorithms are implementeddepth .

Likewise, batches determine how often the model updates, and larger dataset
models often use anywhere from 32 to ’the size of the training set’
batchesbatch . If we tested with a larger dataset, these bounds would be more
appropriate. This is most significantly reflected in the accuracy of the
training falling while more GPUs are added to the workflow, as exhibited in
Fig. 7. While accuracy is maintained in the early stages from 1-3 GPUs, it
plummets as more are added and the sampler splits the dataset into smaller and
smaller parts. Due to keeping the batch size parameter constant, the
"effective batch size" (the product between the number of GPUs and batch size
per GPU) is directly proportional to the increase of GPUs. The effective batch
size increasing, combined with not increasing the learning rate parameter
accordingly, results in inconsistent, low image recognition accuracy.

###  III.3 Expanding the dataset

In an effort to analyze performance across multiple compute nodes, we expanded
the dataset to include more images from ImageNet. The overall dataset was
expanded to 4145 training images (up from 245), and also added an additional
class (ladybugs). Although ladybugs were added to the ants and bees set, the
small amount of classes still keeps the model in the realm of a "feature
extractor". For these tests, we keep the number of qubits (4) and epochs (30)
constant as we focus on analyzing performance of the default parameters on a
larger dataset. However, in an effort to combat the issue mentioned at the end
of Section III.2, we scaled the learning rate parameter appropriately when
distributing across more GPUs or CPUs2015deepresidual ; 1hourimagenet .
Varying the learning rate in this manner led to a relatively consistent
validation accuracy of $\sim$92% as we scaled up the number of devices being
used.

From a computational setup point of view, each MPI task was allocated the
maximum number of physical cores in each L3 cache region on a given node of a
system. This meant spanning each MPI task across two compute cores on Andes,
while spanning across seven compute cores on Frontier. Mapping tasks to cores
in this manner leads to a maximum of 16 tasks per node on Andes, and 8 tasks
per node on Frontier. Similar to Section III.2, CPU training was the focus on
Andes, while both CPU and GPU training was explored on Frontier.

Figure 8: Execution time versus number of threads on Andes and Frontier. (4
qubits, 30 epochs, larger dataset)

The performance of our hybrid program across multiple compute nodes can be
seen in Figure 8. As expected, the Frontier GPUs perform quite well overall,
performing $\sim$92% and $\sim$48% faster than the Andes and Frontier CPUs,
respectively, at $\rm N=8$. Not pictured in Figure 8 is the laptop’s
performance on the larger dataset using 8 threads, which had an execution time
of 2801 s. This leads to an astounding speed-up of $\sim$226% that the
Frontier GPUs have over the laptop at the same thread count. The Frontier CPUs
and GPUs start to converge around 72 threads (9 nodes), where the GPUs are
only $\sim$8% faster. Note that although not constant, the percent difference
between the Andes CPUs and Frontier CPUs is consistently around the 30%
region. Like in Section III.2, this suggests inefficient GPU communication in
our code that degrades the Frontier GPU tests as thread count is increased;
although, this time it was not the high qubit count but rather the shear
number of GPUs being used. With the absence of a GPU-based simulator, this
problem will only worsen as the dataset is expanded further, or when more
nodes ($\geq$10) are used with this dataset. For further details and access to
the code, please refer to the GitHub link provided
below§§§https://github.com/michael-sandoval/hybrid_quantum_hpc.

Although Andes clearly performs the slowest when compared to both Frontier
methods, it is able to use twice as many tasks per node due to the way we
mapped MPI tasks to cores (1 MPI task per L3 memory region), thus providing a
potential benefit to cost when comparing to the Frontier CPU method. However,
more testing must be done with the Frontier CPU approach, as Frontier has
twice as many physical cores per node than Andes, but utilizing more tasks per
node on Frontier would require packing tasks more tightly in a given L3
region, which could affect performance.

##  IV Discussion and Conclusions

Although ML has proven its value in the realm of HPC, this work sought out to
explore the potential value QML has in an HPC environment from a quantum
simulator standpoint. Taking a "ground up" approach, we started with a
tutorial code and gradually modified it and the dataset to fit in an HPC
environment. We highlighted the HPC benefits that QML intrinsically inherits
from classical ML (data distribution speed-up), while also demonstrating
potential quantum benefits (model accuracy at higher qubit counts) and
simulator pitfalls. One result we emphasize is the communication obstacles
when using either cloud-based or local simulators. More specifically, the
latency to remote, cloud-based simulators is concerning when simulators are
used every training iteration. Additionally, inefficient GPU communication
with local, CPU-based simulators pose a challenge within and across compute
nodes at increased qubit and thread counts, respectively. The communication
challenges emphasize how a simulator should be implemented, including its
framework, and ideally how it should be designed with HPC in mindxacc .

This research furthers the active interest in hybrid QC and HPC resources
located in the same location at computing centers worldwide. While QC and HPC
can operate independently, remote access, particularly for job-partitioned QML
algorithms, is best suited for well-developed workflows in computing centers.
Some suggestions are noted as follows:

  1. (A)

Run jobs on devices featuring both classical and quantum computational
abilities: QPUs, CPUs, and/or GPUs.

  2. (B)

Port jobs to centers hosting both classical and quantum computers at the same
locationmeetiqm .

  3. (C)

Reserve time on HPC and quantum devices (for the same period) ahead so that
the full power of the QC can be utilized during HPC runtime without queue-
induced latency.

  4. (D)

Design QML jobs that can be trained in parallel with multiple GPUs/nodes with
a GPU-capable simulator (in addition to any of the above).

Further research can also be done with larger datasets to determine the
feasibility of hybrid programs. Many large databases such as ImageNet require
strong CPU or GPU infrastructure, but other datasets, particularly those used
in the private sector for demographic mappinggoogle , self-driving
vehiclesself-driving , and forensic recognitiondigital-forensics , may require
large server architectures or many HPC machine nodes to train IT-KB-NDSU . QML
would be especially optimal for these use cases, offering not only speedup but
a practical application for early quantum computers in the Noisy Intermediate
Scale Quantum era.

Additionally, this research acts as an example for the benefit an HPC
environment can provide to quantum computing – even when only considering
quantum simulators (as opposed to a real quantum backend). Local non-HPC
machines can run simulators adequately in some scenarios, but quickly fall
behind when multiple threads are needed to perform computations (i.e., with
complex training methods or when using large datasets). In Section III.2, the
laptop is consistently outperformed in most scenarios, even when utilizing a
small dataset, as demonstrated. This trend is reinforced in Section III.3,
where the impracticality of using the laptop becomes apparent as we increase
the dataset size. Although there are much more powerful non-HPC machines
available when compared to the laptop we used in this research, the takeaway
remains that they will eventually hit feasible training limits much earlier
than a compute cluster will (even when only using a simulator). We acknowledge
that this research was based on a tutorial-like problem, but hope that this
proof of concept will motivate more intensive scaling studies in the future.

###### Acknowledgements.

This research was supported in part by an appointment to the Oak Ridge
National Laboratory Science Undergraduate Laboratory Internship Program,
sponsored by the U.S. Department of Energy and administered by the Oak Ridge
Institute for Science and Education. STB thanks Art Stewart for paper
reviewing services. We acknowledge the use of IBM Quantum services for this
work. The views expressed are those of the authors, and do not reflect the
official policy or position of IBM or the IBM Quantum team. This research used
resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge
National Laboratory, which is supported by the Office of Science of the U.S.
Department of Energy under Contract No. DE-AC05-00OR22725. We thank Oak Ridge
National Laboratory Leadership Computing Facility for allowing us access to
Quantum Processors at IBMQ, as well as the Andes Linux Cluster and Frontier
Exascale computers.

## References

## References

  * (1) A. M. Turing, “Computing machinery and intelligence,” Mind, vol. 59, no. 236, pp. 433–460, 1950.
  * (2) P. Messina, “The exascale computing project,” Computing in Science & Engineering, vol. 19, no. 3, pp. 63–67, 2017.
  * (3) F. Arute et al., “Quantum supremacy using a programmable superconducting processor,” Nature, vol. 574, p. 505–510, 2019.
  * (4) B. J. Copeland, “Artificial intelligence.” https://www.britannica.com/technology/artificial-intelligence/Reasoning, Jun 2023.
  * (5) S. Chilamkurthy, “Transfer learning for computer vision tutorial.” https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html, 2023.
  * (6) C. Tardi, “Moore’s law explained.” https://www.investopedia.com/terms/m/mooreslaw.asp, 2021.
  * (7) D. Berebichez, “The way forward: Bringing hpc and quantum computing together (part 1).” https://www.meetiqm.com/resources/blog/the-way-forward/, Nov 2022.
  * (8) A. Paszke et al., “Pytorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems 32, pp. 8024–8035, Curran Associates, Inc., 2019.
  * (9) Stanford Department of Computer Science, “Transfer learning.” https://cs231n.github.io/transfer-learning/, 2016.
  * (10) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee, 2009.
  * (11) A. Aprilliant, “Fixed feature extractor as the transfer learning method for image classification using mobilenet.” https://audhiaprilliant.medium.com/fixed-feature-extractor-as-the-transfer-learning-method-for-image-classification-using-mobilenet-b26376e25d49, Jan 2023.
  * (12) “Olcf user docs, andes user guide, compute nodes.” https://docs.olcf.ornl.gov/systems/andes_user_guide.html#compute-nodes.  Accessed: 2023-06-29.
  * (13) V. Bergholm et al., “Pennylane: Automatic differentiation of hybrid quantum-classical computations,” 2022.
  * (14) A. Mari, “Quantum transfer learning.” https://pennylane.ai/qml/demos/tutorial_quantum_transfer_learning, 11 2019.  Date Accessed: 2023-06-23.
  * (15) L. J. O’Riordan, “Performance of lightning.gpu vs lightning.qubit.” https://discuss.pennylane.ai/t/performance-of-lightning-gpu-vs-lightning-qubit/1976/2, journal=Xanadu Discussion Forum, Jun 2022.
  * (16) A. Javadi-Abhari, M. Treinish, K. Krsulich, C. J. Wood, J. Lishman, J. Gacon, S. Martiel, P. D. Nation, L. S. Bishop, A. W. Cross, B. R. Johnson, and J. M. Gambetta, “Quantum computing with Qiskit,” 2024.
  * (17) IBM Quantum Contributors, “Fair-share queuing.” https://quantum-computing.ibm.com/lab/docs/iql/manage/systems/queue/.
  * (18) Simplilearn, “What is epoch in machine learning?: Simplilearn.” https://www.simplilearn.com/tutorials/machine-learning-tutorial/what-is-epoch-in-machine-learning, May 2023.
  * (19) “OLCF User Docs, Frontier User Guide, Compute Nodes.” https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#frontier-compute-nodes.  Accessed: 2023-06-29.
  * (20) A. B. Yoo, M. A. Jette, and M. Grondona, “Slurm: Simple linux utility for resource management,” in Job Scheduling Strategies for Parallel Processing (D. Feitelson, L. Rudolph, and U. Schwiegelshohn, eds.), (Berlin, Heidelberg), pp. 44–60, Springer Berlin Heidelberg, 2003.
  * (21) L. Dalcin and Y.-L. L. Fang, “mpi4py: Status update after 12 years of development,” Computing in Science & Engineering, vol. 23, no. 4, pp. 47–54, 2021.
  * (22) Qiskit, “What is quantum volume, anyway?.” https://medium.com/qiskit/what-is-quantum-volume-anyway-a4dff801c36f, Mar 2021.
  * (23) G. Verdon, M. Broughton, and J. Biamonte, “A quantum algorithm to train neural networks using low-depth circuits,” 2019.
  * (24) J. Brownlee, “Difference between a batch and an epoch in a neural network.” https://machinelearningmastery.com/difference-between-a-batch-and-an-epoch/, Aug 2022.
  * (25) K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015.
  * (26) P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, “Accurate, large minibatch sgd: Training imagenet in 1 hour,” 2018.
  * (27) K. K. Yang, “Distributed tutorial.” https://github.com/yangkky/distributed_tutorial/tree/24467967c1c719110c33fccca69353ad8e5ae2e4/src, oct 2019.
  * (28) D. Steinkraus, I. Buck, and P. Simard, “Using gpus for machine learning algorithms,” in Eighth International Conference on Document Analysis and Recognition (ICDAR’05), pp. 1115–1120 Vol. 2, 2005.
  * (29) T. Nguyen, D. Lyakh, E. Dumitrescu, D. Clark, J. Larkin, and A. McCaskey, “Tensor network quantum virtual machine for simulating quantum circuits at exascale,” 2021.
  * (30) W. Sirko et al., “Continental-scale building detection from high resolution satellite imagery,” 2021.
  * (31) F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell, “Bdd100k: A diverse driving dataset for heterogeneous multitask learning,” 2020.
  * (32) S. Ferreira, M. Antunes, and M. E. Correia, “A dataset of photos and videos for digital forensics analysis using machine learning processing,” Data, vol. 6, p. 87, Aug 2021.
  * (33) I. K. Base, “Running machine learning on hpc clusters.” https://kb.ndsu.edu/page.php?id=107849, Dec 2020.


# GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution

Kelvin C.K. Chan1 Xintao Wang2 Xiangyu Xu1 Jinwei Gu3 Chen Change Loy1∗
1Nanyang Technological University  Singapore
2Applied Research Center  Tencent PCG 3SenseBrain
{chan0899, xiangyu.xu<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

We show that pre-trained Generative Adversarial Networks (GANs), , StyleGAN,
can be used as a latent bank to improve the restoration quality of large-
factor image super-resolution (SR). 00footnotetext: ∗Corresponding author
While most existing SR approaches attempt to generate realistic textures
through learning with adversarial loss, our method, Generative LatEnt bANk
(GLEAN), goes beyond existing practices by directly leveraging rich and
diverse priors encapsulated in a pre-trained GAN. But unlike prevalent GAN
inversion methods that require expensive image-specific optimization at
runtime, our approach only needs a single forward pass to generate the
upscaled image. GLEAN can be easily incorporated in a simple encoder-bank-
decoder architecture with multi-resolution skip connections. Switching the
bank allows the method to deal with images from diverse categories, , cat,
building, human face, and car. Images upscaled by GLEAN show clear
improvements in terms of fidelity and texture faithfulness in comparison to
existing methods as shown in Fig. 1. A high-resolution version of this paper
can be found at https://ckkelvinchan.github.io/.

Figure 1: Example of large-factor super-resolution (16${\times}$). (a) The
low-resolution input (LR). (b) ESRGAN [35] trains the SR generator from
scratch, which often produces artifacts and unnatural textures. (c) PULSE [27]
achieves more realistic results by GAN inversion, which, however, cannot
faithfully recover the structures of the ground-truth. (d) With the proposed
generative latent bank, GLEAN is able to generate output that not only is
close to the ground-truth, but also possesses realistic textures. (e) The
ground-truth (GT).

##  1 Introduction

In this study, we explore a new way to employ GAN [9] for image super-
resolution. We are interested in the regime of high magnification factors
($8{\times}$ to ${64\times}$), which typical SR methods fail to handle since
most details and textures are lost during downsampling. Since the problem is
severely underspecified, informative priors become inevitable in this setting,
especially in restoring the textural details. Studying large-factor image SR
is meaningful as it can potentially improve the state of the arts in SR, and
more generally conditional generative models for images.

The notion of GAN has been extensively used in SR with the aim to enrich
texture details in an upscaled image. There are two popular approaches to
deploy GANs for this task. The more common paradigm [21, 34, 35] trains a
generator to handle the upscaling task, where adversarial training is
performed by using a discriminator to differentiate real images from the
upscaled images produced by the generator. Another possible way to exploit GAN
for the task is by GAN inversion [1, 11, 27, 28]. In this setting, one will
need to ‘invert’ the generation process of a pre-trained GAN by mapping a
corrupted image back to the latent space. A restored image can then be
reconstructed from the optimal vector in the latent space.

While both methods are capable of generating more realistic results than
approaches that solely rely on $\ell_{2}$ loss, they have some inherent
shortcomings. The first paradigm typically trains the SR generator from
scratch using a combined objective function consisting of a fidelity term and
an adversarial loss. In this setting, the generator is responsible for both
capturing the natural image characteristics and maintaining the fidelity to
the ground-truth. This inevitably limits the capability of approximating the
natural image manifold. As a result, these methods often produce artifacts and
unnatural textures. As shown in Fig. 1, while ESRGAN [35] faithfully recovers
the structures ( pose, ear shape) of the cat, it struggles to produce
realistic textures.

The second paradigm resolves the aforementioned problem by making better use
of the latent space of GAN through optimization. However, as the low-
dimensional latent codes and the constraints in the image space are
insufficient to guide the restoration process, these methods often generate
images with low fidelity. As shown in Fig. 1, despite being realistic, the
output of a representative method, PULSE [27], fails to recover the structures
of the ground-truth faithfully. In addition, as the optimization is usually
conducted in an iterative manner for each image at runtime, these approaches
are often time-consuming.

In our approach, we leverage pre-trained GANs such as StyleGAN [16] to provide
rich and diverse priors for the task. Unlike most GAN inversion methods, which
also use pre-trained GANs, our method does not involve image-specific
optimization at runtime. Once trained, the model only needs a single forward
pass to upscale an image, which is more practical for applications that demand
fast response. The idea is partially inspired by the classic notion of
dictionary [40]. But unlike conventional approaches that construct a finite
and imagery-derived dictionary, we exploit GAN as a more effective way for
storing priors.

Conditioning and retrieving from a GAN-based dictionary is a new and non-
trivial question we need to address in this work. We show that pre-trained
GANs can be employed as a latent bank in a succinct encoder-bank-decoder
architecture. This novel architecture allows us to lift the burden of learning
both fidelity and texture generation simultaneously in a typical encoder-
decoder network since the latent bank already captures rich texture priors. In
addition, we show that it is pivotal to condition the bank by passing both the
latent vectors and multi-resolution convolutional features from the encoder to
achieve high-fidelity results. Symmetrically, multi-resolution cues need to be
passed from the bank to the decoder. We show the effectiveness of the proposed
method in handling images with challenging poses and structures apart from the
large magnification factor. We also demonstrate how the method can be
generalized to different categories, , human faces, cats, buildings, by
switching different pre-trained GAN latent banks.

Figure 2: Overview of GLEAN. In addition to the latent vectors
$\mathbf{c}_{i}$, the generator (, the generative latent bank) is also
conditioned on the multi-resolution features $f_{i}$. With a pre-trained GAN
capturing the natural image prior, this encoder-bank-decoder design lifts the
burden of learning both fidelity and naturalness in the conventional encoder-
decoder architecture. $E_{i}$, $S_{i}$ and $D_{i}$ denote the encoder blocks,
latent bank blocks and decoder blocks, respectively. This example corresponds
to an input size of $32{\times}32$ and an output size of $256{\times}256$.

##  2 Related Work

Image Super-Resolution. Many existing SR algorithms [4, 6, 7, 8, 12, 36, 44,
48] directly learn a mapping from the low-resolution images to high-resolution
images with a pixel-wise constraint ($\ell_{2}$ loss). While these methods
achieve remarkable results in terms of PSNR, training solely with pixel-wise
constraints often results in perceptually unconvincing outputs with severe
over-smoothing artifacts [21, 27]. To alleviate the problem, GANs [21, 29, 35,
37] are employed to approximate the natural image manifold, yielding more
photo-realistic results. However, as the generator needs to learn both
fidelity and natural image characteristics, unnatural artifacts could still be
observed in the outputs, especially if one trains the generator from scratch.

Recent interests have shifted to large-factor SR beyond the typical upscaling
factors ($2{\times}$ or $4{\times}$) [3, 13, 31, 46]. Dahl et al. [3] propose
a fully probabilistic pixel recursive network for upsampling extremely coarse
images with resolution $8{\times}8$. RFB-ESRGAN [31] builds upon ESRGAN and
adopts multi-scale receptive fields blocks for $16{\times}$ SR. VarSR [13]
achieves $8{\times}$ SR by matching the latent distributions of LR and HR
images to recover the missing details. Zhang et al. [46] perform $16{\times}$
reference-based SR on paintings with a non-local matching module and a wavelet
texture loss. To handle even larger magnification factors, one would need to
rely on stronger priors. SR methods specialized on large magnification factors
are typically dedicated to the human face category as one could exploit the
strong structural prior of faces. Facial priors including facial attributes
[22], facial landmarks [18, 26], and identity [10] have been studied. Our work
goes beyond previous works and pushes the limit to $64{\times}$ and
generalizes to more categories. Such a large magnification factor is
challenging due to its highly ill-posed nature.

GAN Inversion. Given a degraded image $x$, GAN inversion-based methods [1, 11,
27, 28] in general produce a natural image best approximating $x$ by
optimizing
$z^{*}=\operatorname*{argmin}\nolimits_{z\in\mathcal{Z}}\mathcal{L}\left(G(z),x\right)$,
where $\mathcal{Z}$ is the latent space and $\mathcal{L}(\cdot,\cdot)$ denotes
the task-specific objective function. For instance, PULSE [27] iteratively
optimizes the latent code of StyleGAN [16] with a pixel-wise constraint
between the input and output. mGANprior [11] optimizes multiple latent codes
to increase the expressiveness of the model. DGP [28] further finetunes the
generator together with the latent code to reduce the gap between the
distributions of the training and testing images. A common issue with GAN
inversion is that important spatial information may not be faithfully kept due
the low-dimensionality of the latent code. Thus, these methods often generate
undesirable results that do not resemble the ground-truth. Different from GAN
inversion, GLEAN conditions the pre-trained generator with both the latent
codes and multi-resolution convolutional features, providing additional
spatial guidance for restoration. In addition, GLEAN does not require
iterative optimization during inference.

##  3 Methodology

A GAN model that is trained on large-scale natural images captures rich
texture and shape priors. Previous studies [1, 11, 27, 28] have shown that
such priors can be harvested through GAN inversion to benefit various image
restoration tasks. Nonetheless, it remains underexplored how to exploit the
priors without the expensive optimization during inversion.

In this study, we devise GLEAN within a novel encoder-bank-decoder
architecture, which allows one to exploit the generative priors by needing
just a single forward pass. An overview of the architecture is depicted in
Fig. 2. Given a severely downsampled LR image, GLEAN applies an encoder to
extract latent vectors and multi-resolution convolutional features, which
capture important high-level cues as well as spatial structure of the LR
image. Such cues are used to condition the latent bank, which further produces
another set of multi-resolution features for the decoder. Finally, the decoder
generates the final output by integrating the features from both the encoder
and the latent bank. In this work, we adopt StyleGAN [16, 17] as the
generative latent bank due to its exceptional performance. The idea of latent
bank can be extended to other generators such as BigGAN [2].

###  3.1 Encoder

To generate the latent vectors, we first use an RRDBNet [35] (denoted as
$E_{0}$) to extract features $f_{0}$ from the input LR image. Then, we
gradually reduce the resolution of the features by:

$f_{i}=E_{i}(f_{i-1}),\quad i\in\\{1,\cdots,N\\},$ (1)

where $E_{i}$, ${i\in\\{1,\cdots,N\\}}$, denotes a stack of a stride-2
convolution and a stride-1 convolution. Finally, a convolution and a fully-
connected layer are used to generate the latent vectors:

$C=E_{N+1}(f_{N}),$ (2)

where $C$ is a matrix whose columns represent the latent vectors for the
StyleGAN.

The latent vectors in $C$ capture a compressed representation of the images,
providing the generative latent bank with high-level information. To further
capture the local structures of the LR image and to provide additional
guidance for structure restoration, we also feed multi-resolution
convolutional features $\\{f_{i}\\}$ into the latent bank.

###  3.2 Generative Latent Bank

Given the convolutional features $\\{f_{i}\\}$ and the latent vectors $C$, we
leverage a pre-trained generator as a latent bank to provide priors for
texture and detail generation. As StyleGAN is originally designed for image
generation tasks, it cannot be directly integrated into the proposed encoder-
bank-decoder framework. In this work, we adapt StyleGAN to our SR network by
making three modifications:

  1. 1.

Instead of taking one single latent vector as the input, each block of the
generator takes a different latent vector to improve expressiveness. More
specifically, we have $C{=}(\mathbf{c}_{0},\cdots,\mathbf{c}_{k-1})$ for $k$
blocks, where each $\mathbf{c}_{i}$ corresponds to one latent vector. We find
that this modification leads to outputs with fewer artifacts. This
modification is also seen in previous works [11, 38, 49].

  2. 2.

To allow conditioning on the additional features from the encoder, we use an
additional convolution in each style block for feature fusion:

$g_{i}=\begin{cases}S_{0}(\mathbf{c}_{0},f_{N}),&\text{if }i=0,\\\
S_{i}(\mathbf{c}_{i},g_{i-1},f_{N-i}),&\text{otherwise},\end{cases}$ (3)

where $S_{i}$ denotes the augmented style block with an additional
convolution, and $g_{i}$ corresponds to the output feature of the $i$-th
augmented style block.

  3. 3.

Instead of directly generating outputs from the generator, we output the
features $\\{g_{i}\\}$ and pass them to the decoder to better fuse the
features from the latent bank and encoder.

Advantages. The use of generative latent bank is reminiscent of the task of
reference-based SR [23, 24, 39, 45, 47], where external HR reference image(s)
are employed as an explicit imagery dictionary. While the external HR
information leads to marked improvements, the performance is sensitive to the
similarity between the inputs and references. This sensitivity may eventually
lead to degraded results when the reference images/components are not well
selected. Moreover, the size and diversity of those imagery dictionaries are
limited by the selected components, impeding the generalization to diverse
scenes in practice. In addition, computationally-intensive global matching
[47] or component detection/selection [23] is often required to aggregate
appropriate information from the references, hindering the applications to
scenarios with tight computational constraints. Instead of constructing an
imagery dictionary, GLEAN adopts a GAN-based dictionary conditioned on a pre-
trained GAN. Our dictionary does not depend on any specific components or
images. Instead, it captures the distribution of the images and has
potentially unlimited size and diversity. Furthermore, GLEAN is
computationally efficient without requiring global matching and reference
images/components selection.

###  3.3 Decoder

GLEAN uses an additional decoder with progressive fusion to integrate the
features from the encoder and latent bank to generate the output image. It
takes the RRDBNet features as inputs and progressively fuse the features with
the multi-resolution features from the latent bank:

$d_{i}=\begin{cases}D_{0}(f_{0})&\text{if }i=0,\\\
D_{i}(d_{i-1},g_{N-2+i})&\text{otherwise},\end{cases}$ (4)

where $D_{i}$ and $d_{i}$ denote a $3{\times}3$ convolution and its output,
respectively. Each convolution is followed by a pixel-shuffle [32] layer
except the final output layer. With the skip-connection between the encoder
and decoder, the information captured by the encoder can be reinforced and
hence the latent bank could focus more on the texture and detail generation.

###  3.4 Training

Similar to existing works [21, 34, 35], we adopt the standard $l_{2}$ loss,
perceptual loss [14], and adversarial loss for training. More details on the
loss function can be found in the appendix. To exploit the generative prior,
we keep the weights of the latent bank fixed throughout training. In our
preliminary experiments, finetuning the latent bank with the encoder and
decoder demonstrates no noticeable improvements. Moreover, it potentially
harms the generalizability of the model as the latent bank may eventually bias
to the training distribution. It is worth emphasizing that despite GLEAN is
trained with similar objectives as in existing works ( ESRGAN), the main
difference to these methods is that GLEAN leverages a pre-trained generator to
directly incorporate the priors into the network, further improving the output
quality. We show that the improvement is not due to additional parameters in
the generator by comparing GLEAN with ESRGAN+, a larger ESRGAN that has
similar FLOPs to GLEAN.

Figure 3: Comparisons on 16${\times}$ SR on CelebA-HQ [15]. Only GLEAN is able
to maintain high fidelity while synthesizing realistic textures and details:
GAN inversion methods fail to preserve the identity, and adversarial loss
methods struggle to synthesize fine details. ESRGAN+ denotes a larger version
with similar FLOPs to GLEAN. (Zoom-in for best view) Figure 4: Results of
16${\times}$ SR on other categories. GLEAN can be applied to various
categories by switching between StyleGANs trained on different categories.
(Zoom-in for best view) Figure 5: Results on larger scale factors. GLEAN
reconstructs realistic images highly similar to the GT for up to $64{\times}$
upscaling factor. (Zoom-in for best view)

##  4 Experiments

We adopt pre-trained StyleGAN111GenForce: https://github.com/genforce/genforce
[16] or StyleGAN2222BasicSR: https://github.com/xinntao/BasicSR [17]
(depending on the availability of pre-trained models) as our latent bank, and
use the publicly available codes of existing methods for the comparison in
this section. To maintain fairness, we train our model and baselines on the
same datasets, including FFHQ [16] and LSUN [41], so that the difference in
restoration quality is mainly caused by the algorithms instead of the training
distribution. Test set is strictly exclusive from the training. Detailed
experimental settings are provided in the appendix.

Qualitative comparison. The qualitative comparison on $16{\times}$ SR is shown
in Fig. 3. Guided by low-dimensional vectors and constraints in LR space, the
outputs of GAN inversion methods are unable to maintain a good fidelity. In
particular, PULSE [27] and mGANprior [11] fail to restore a face image with
the same identity. In addition, artifacts are observed in their outputs.
Through finetuning the generator during optimization, the result of DGP [28]
demonstrates significant improvements in both quality and fidelity. However, a
slight difference between the identities of the output and ground-truth is
still observed. For example, the eyes and lips show noticeable differences.

Methods trained with adversarial loss (SinGAN [30], ESRGAN+333A larger version
of ESRGAN with similar FLOPs to GLEAN. [35]) can preserve the local
structures, but fail in synthesizing convincing textures and details.
Specifically, SinGAN fails to capture the natural image style, producing a
painting-like image. Although ESRGAN+ is capable of generating a realistic
image, it struggles to synthesize fine details and introduces unnatural
artifacts in detailed regions. It is worth emphasizing that although ESRGAN+
achieves competitive results on human faces, its performances on other
categories such as cats and cars are less promising (see Fig. 1 and Fig. 4).
With the latent bank providing natural image priors, GLEAN succeeds in both
fidelity and naturalness. For example, when compared to ESRGAN+, GLEAN
reconstructs eyes with better shape and details. We further extend our method
to larger scale factors in Fig. 5. GLEAN successfully generates perceptually
convincing images resembling the ground-truth for up to $64{\times}$
upscaling.

Figure 6: Outputs with diverse poses and contents. Despite GLEAN is trained
with aligned human faces, it is able to reconstruct faithful images for non-
aligned and non-human faces. PULSE approximates the GT in low resolution
(bottom left), but its outputs are significantly different from the GT when
viewed in high resolution. Table 1: Cosine similarity of ArcFace Embeddings
[5]. GLEAN achieves a higher similarity than baselines. Bolded texts represent
the best performance.

| PULSE [27] | mGANprior [11] | DGP [28]
---|---|---|---
Similarity | 0.4047 | 0.5526 | 0.7341
| SinGAN [30] | ESRGAN+ [35] | GLEAN
Similarity | 0.7718 | 0.9599 | 0.9678

Table 2: Quantitative (PSNR/LPIPS) comparison on 16${\times}$ SR. GLEAN
outperforms other methods in most categories. ESRGAN+ denotes a larger version
of ESRGAN [35] having similar FLOPs to GLEAN. Bolded texts represent the best
performance.

| mGANprior [11] | PULSE [27] | ESRGAN+ [35] | GLEAN
---|---|---|---|---
Face [15] | 23.66/0.4661 | 21.83/0.4600 | 26.76/0.2787 |  26.84/0.2681
Cat [43] | 17.01/0.5556 | 19.78/0.5241 | 19.99/0.3482 |  20.92/0.3215
Car [20] | 14.53/0.7228 | 16.30/0.6491 | 19.42/0.3006 |  19.74/0.2830
Bedroom [41] | 16.38/0.5439 | 12.97/0.7131 |  19.47/0.3291 | 19.44/0.3310
Tower [41] | 15.96/0.4870 | 13.62/0.7066 | 17.86/0.3132 |  18.41/0.2850

Figure 7: Effects of the multi-resolution encoder features. Without the
convolutional features, the outputs can only resemble the global attributes (
hair color, pose). When adding the encoder features progressively, the network
can capture more local structures, better approximating the GT.

Robustness to poses and contents. Another appealing property of GLEAN is its
robustness to the changes in poses and contents. As shown in Fig. 6, guided by
the convolutional features, GLEAN is still able to construct realistic images
when the images are non-aligned and contain non-human faces despite it is
trained on aligned human faces. In contrast, the outputs of PULSE are biased
to aligned human faces. Its outputs can only approximate the ground-truths in
low resolution. Such robustness enables GLEAN to be applied to diverse
categories and scenes such as cats, cars, bedrooms, and towers. Examples are
shown in Fig. 4 and more results are provided in the appendix.

Quantitative comparison. To demonstrate the ability of GLEAN in producing
outputs with high fidelity, we extract 100 images from CelebA-HQ [15] and
compute the cosine similarity to the ground-truth on the ArcFace embedding
space [5]. As shown in Table 1, GLEAN achieves higher similarity than the
baseline methods, validating the superiority of GLEAN.

We additionally provide the quantitative comparison on different categories in
Table 2. For each category, we select 100 images and compute their average
PSNR and LPIPS [42]. It is observed that mGANprior and PULSE perform
significantly worse as they fail to restore the original objects. GLEAN
outperforms these methods in most categories, suggesting its effectiveness in
generating images with high quality and fidelity.

Figure 8: Effects of the latent bank features. The rich texture priors
captured in the generator lift the burden of the encoder in texture
generation. Improvements on both texture and structures are observed when
finer features are inserted into the decoder. (Zoom-in for best view) Figure
9: Contributions of the decoder. The decoder reinforces the spatial
information captured in the encoder features and aggregate them in a coarse-
to-fine manner, leading to enhanced quality.

##  5 Ablation Studies

Importance of multi-resolution encoder features. We demonstrate how the
convolutional features generated from the encoder assist in the restoration of
fine details and local structures. We start with only the latent vectors and
observe the transition when features are gradually introduced to the latent
bank as conditions. To discard the effects brought by the decoder, we test
with a variant of GLEAN where the generator directly produces the output
images. The comparison is depicted in Fig. 7.

When all convolutional features are discarded, GLEAN resembles the typical GAN
inversion methods that learn only the latent vectors. Similar to those
methods, the network is able to synthesize realistic images given the latent
vectors. However, guided only by low-dimensional vectors, in which spatial
information is not well-preserved, the network restores only the global
attributes such as hair color and poses, but fails to preserve finer details.
When providing coarse (from $4{\times}4$ to $16{\times}16$) convolutional
features to the latent bank, more details are recovered and the outputs are
better approximating the ground-truths. Further improvements in both quality
and fidelity are observed when finer features are passed to the latent bank.
The above observations corroborate our hypothesis that the convolutional
features are pivotal in guiding the restoration of fine details and local
structures, which cannot be reconstructed with only the latent vectors.

Effects of latent bank features. To understand the contributions of the latent
bank, we investigate the effects brought by the latent bank features. We start
by discarding all the latent bank features, and progressively pass the
features to the decoder. The comparison is shown in Fig. 8. Lacking
appropriate prior information, the network is responsible for both generating
realistic details and maintaining fidelity to the ground-truths. Such a
demanding objective eventually leads to outputs that contain flaws in both
structure restoration and texture generation. With the latent bank, the burden
of texture and details generation is reduced as the generator already captures
rich image priors. Therefore, improvements in both structures and textures are
observed when passing finer features to the decoder.

Importance of decoder. As shown in Fig. 9, without the decoder, despite being
perceptually convincing overall, the output image contains unpleasant
artifacts when zoomed in. The decoder allows the network to aggregate the
information in a coarse-to-fine manner, leading to more natural details. In
addition, the multi-scale skip-connections between the encoder and decoder
reinforce the spatial information captured in the encoder features so that the
latent bank could focus more on detail generation, further enhancing the
output quality.

(a) Comparison with DFDNet [23] (b) Comparison with SRNTT [47]

Figure 10: Comparison to imagery dictionary. (a) DFDNet fails to restore
components absent in the dictionary ( skin, hair), leading to incoherent
outputs. (b) SRNTT is unable to produce faithful fur textures.

Comparisons with reference-based methods. We assess the efficacy of the new
notion of GAN-based dictionary by comparing GLEAN with two representative
methods adopting an imagery dictionary for SR – SRNTT [47] and DFDNet [23].
Examples are shown in Fig. 10.

For DFDNet, we evaluate the performance on LR images with unknown
degradations444We further downsample the LR images to $64{\times}64$ to match
the input size of GLEAN.. Through pre-constructing a dictionary of facial
components ( eyes, lips), DFDNet shows remarkable performance on face
restoration. However, it cannot produce faithful results on parts absent in
the dictionary, such as skin and hair. Therefore, significant incoherence is
observed in the outputs. Despite GLEAN is trained on the bicubic kernel, it is
still capable of producing appealing outputs. More importantly, GLEAN is not
confined to improving the visual quality of specific components. Instead, the
entire image is super-resolved, leading to coherent and pleasing results. The
performance of GLEAN could be further improved by employing multiple
degradations during training.

For SRNTT, we follow the same settings and downsample the ground-truth images
using the bicubic kernel. With such low-resolution images ($32{\times}32$),
global matching becomes prohibitive, and hence SRNTT fails to transfer the
textures from HR reference images. As a result, SRNTT tends to provide blurry
textures. By capturing the distribution instead of specific imagery clues,
GLEAN does not rely on any explicit textural transferal procedure. This
enables the applicability to large-factor SR, where image matching is
extremely difficult. More importantly, with no external images employed, GLEAN
does not require any global matching to search for suitable textures/details.
This allows GLEAN to be applied to images with larger resolutions, where
global matching is computationally prohibitive.

Figure 11: Image Retouching. GLEAN can be used to eliminate unnatural
artifacts introduced by amateur retouching. (Zoom-in for best view)

##  6 Application – Image Retouching

In this section, we present one interesting application of GLEAN. In
interactive image retouching, users can manually edit the images based on
their preference. However, a perfect output requires tedious and precise
retouching. As a result, artifacts are common in the outputs, especially those
from amateur retouching. As a powerful super-resolver, GLEAN can be used as an
image retouching tool to eliminate unpleasant artifacts.

As shown in Fig. 11, the blending operation in the interactive editing
software produces a blurry and incoherent output. Thanks to the capability of
GLEAN in producing high quality and fidelity images, GLEAN is able to
eliminate the blurry region and generate a coherent output with natural
textures. Furthermore, with only a single forward pass for generation, it can
be easily incorporated into existing interactive editing software. More
examples will be shown in the appendix.

##  7 Conclusion

We have presented a new way to exploit pre-trained GANs for the task of large-
scale super-resolution, up to $64{\times}$ upscaling factor. We have shown
that a pre-trained GAN can be used as a generative latent bank in an encoder-
bank-decoder architecture. Reconstructing photorealistic HR images requires
just a single forward pass, thanks to effective ways in conditioning and
retrieving rich priors from the bank. The generality of the notion of GAN-
based dictionary allows GLEAN to be potentially extended to not only diverse
architectures but also various imaging tasks, such as image denoising,
inpainting and colorization.

Acknowledgement. This research was conducted in collaboration with SenseTime
and supported by the Singapore Government through the Industry Alignment Fund
- Industry Collaboration Projects Grant. It is also partially supported by
Singapore MOE AcRF Tier 1 (2018-T1-002-056) and NTU SUG.

## References

  * [1] David Bau, Hendrik Strobelt, William Peebles, Bolei Zhou, Jun-Yan Zhu, Antonio Torralba, et al.  Semantic photo manipulation with a generative image prior.  TOG, 2020.
  * [2] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  arXiv preprint arXiv:1809.11096, 2018.
  * [3] Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens.  Pixel recursive super resolution.  In ICCV, 2017.
  * [4] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang.  Second-order attention network for single image super-resolution.  In CVPR, 2019.
  * [5] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  ArcFace: Additive angular margin loss for deep face …  In CVPR, 2019.
  * [6] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Learning a deep convolutional network for image super-resolution.  In ECCV, 2014.
  * [7] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  TPAMI, 38(2):295–307, 2016.
  * [8] Chao Dong, Chen Change Loy, and Xiaoou Tang.  Accelerating the super-resolution convolutional neural network.  In ECCV, pages 391–407. Springer, 2016.
  * [9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NeurIPS, 2014.
  * [10] Klemen Grm, Walter J Scheirer, and Vitomir Štruc.  Face hallucination using cascaded super-resolution and identity priors.  TIP, 2019.
  * [11] Jinjin Gu, Yujun Shen, and Bolei Zhou.  Image processing using multi-code GAN prior.  In CVPR, 2020.
  * [12] Xiangyu He, Zitao Mo, Peisong Wang, Yang Liu, Mingyuan Yang, and Jian Cheng.  Ode-inspired network design for single image super-resolution.  In CVPR, 2019.
  * [13] Sangeek Hyun and Jae-Pil Heo.  VarSR: Variational super-resolution network for very low resolution images.  In ECCV, 2020.
  * [14] Justin Johnson, Alexandre Alahi, and Fei-Fei Li.  Perceptual losses for real-time style transfer and super-resolution.  In ECCV, pages 694–711. Springer, 2017.
  * [15] Tero Karras, Timo Ailo, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of GANs for improved quality, stability, and variation.  In ICLR, 2018.
  * [16] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, 2019.
  * [17] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of StyleGAN.  arXiv preprint arXiv:1912.04958, 2019.
  * [18] Deokyun Kim, Minseon Kim, Gihyun Kwon, and Dae-Shik Kim.  Progressive face super-resolution via attention to facial landmark.  In BMVC, 2019.
  * [19] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [20] Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei Li.  3D object representations for fine-grained categorization.  In ICCV, 2013.
  * [21] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In CVPR, 2017.
  * [22] Mengyan Li, Yuechuan Sun, Zhaoyu Zhang, Haonian Xie, and Jun Yu.  Deep learning face hallucination via attributes transfer and enhancement.  In ICME, 2019.
  * [23] Xiaoming Li, Chaofeng Chen, Shangchen Zhou, Xianhui Lin, Wangmeng Zuo, and Lei Zhang.  Blind face restoration via deep multi-scale component dictionaries.  In ECCV, 2020.
  * [24] Xiaoming Li, Wenyu Li, Dongwei Ren, Hongzhi Zhang, Meng Wang, and Wangmeng Zuo.  Enhanced blind face restoration with multi-exemplar images and adaptive spatial feature fusion.  In CVPR, 2020.
  * [25] Ilya Loshchilov and Frank Hutter.  SGDR: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [26] Cheng Ma, Zhenyu Jiang, Yongming Rao, Jiwen Lu, and Jie Zhou.  Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation.  In CVPR, 2020.
  * [27] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin.  PULSE: Self-supervised photo upsampling via latent space exploration of generative models.  In CVPR, 2020.
  * [28] Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo.  Exploiting deep generative prior for versatile image restoration and manipulation.  In ECCV, 2020.
  * [29] Mehdi S M Sajjadi, Bernhard Schölkopf, and Michael Hirsch.  Enhancenet: Single image super-resolution through automated texture synthesis.  In ICCV, pages 4501–4510, 2017.
  * [30] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli.  SinGAN: Learning a generative model from a single natural image.  In ICCV, 2019.
  * [31] Taizhang Shang, Qiuju Dai, Shengchen Zhu, Tong Yang, and Yandong Guo.  Perceptual extreme super resolution network with receptive field block.  In CVPRW, 2020.
  * [32] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In CVPR, pages 1874–1883, 2016.
  * [33] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In ICLR, 2015.
  * [34] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Recovering realistic texture in image super-resolution by deep spatial feature transform.  In CVPR, 2018.
  * [35] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang.  ESRGAN: Enhanced super-resolution generative adversarial networks.  In ECCVW, 2018.
  * [36] Xiangyu Xu, Yongrui Ma, and Wenxiu Sun.  Towards real scene super-resolution with raw images.  In CVPR, 2019.
  * [37] Xiangyu Xu, Deqing Sun, Jinshan Pan, Yujin Zhang, Hanspeter Pfister, and Ming-Hsuan Yang.  Learning to super-resolve blurry face and text images.  In ICCV, 2017.
  * [38] Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolei Zhou.  Generative hierarchical features from synthesizing images.  arXiv preprint arXiv:2007.10379, 2020.
  * [39] Xu Yan, Weibing Zhao, Kun Yuan, Ruimao Zhang, Zhen Li, and Shuguang Cui.  Towards content-independent multi-reference super-resolution: Adaptive pattern matching and feature aggregation.  In ECCV, 2020.
  * [40] Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma.  Image super-resolution via sparse representation.  TIP, 19(11):20861–2873, 2010.
  * [41] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao.  LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop.  arXiv preprint arXiv:1506.03365, 2015.
  * [42] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  arXiv preprint arXiv:1801.03924, 2018.
  * [43] Weiwei Zhang, Jian Sun, and Xiaoou Tang.  Cat head detection - how to effectively exploit shape and texture features.  In ECCV, 2008.
  * [44] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In ECCV, 2018.
  * [45] Yang Zhang, Ivor W Tsang, Yawei Luo, Changhui Hu, Xiaobo Lu, and Xin Yu.  Copy and Paste GAN: Face hallucination from shaded thumbnails.  In CVPR, 2020.
  * [46] Yulun Zhang, Zhifei Zhang, Stephen DiVerdi, Zhaowen Wang, Jose Echevarria, and Yun Fu.  Texture hallucination for large-scale painting super-resolution.  In ECCV, 2020.
  * [47] Zhifei Zhang, Zhaowen Wang, Zhe Lin, and Hairong Qi.  Image super-resolution by neural texture transfer.  In CVPR, 2019.
  * [48] Shangchen Zhou, Jiawei Zhang, Wangmeng Zuo, and Chen Change Loy.  Cross-scale internal graph neural network for image super-resolution.  In NeurIPS, 2020.
  * [49] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou.  In-domain GAN inversion for real image editing.  In ECCV, 2020.

Appendix

We first provide the implementation details of GLEAN in Sec. A. We then
provide additional qualitative results on various categories and scale factors
in Sec. B.1. Finally, we demonstrate the application of GELAN to the task of
image retouching in Sec. B.2.

##  Appendix A Training Details of GLEAN

We adopt pre-trained StyleGAN555GenForce: https://github.com/genforce/genforce
[16] or StyleGAN2666BasicSR: https://github.com/xinntao/BasicSR [17] as our
generative latent bank. In this section, we assume the latent bank is pre-
trained and present the training details of GLEAN ( the encoder-bank-decoder
network). Note that the weights of the latent bank are fixed when training
GLEAN to better employ the generative prior and to avoid biasing to the
training distribution.

We train GLEAN on five categories including human faces, cats, cars, towers,
and bedrooms. The training and test datasets used in our experiments are
summarized in Table 3. Since StyleGAN produces images with fixed size, we
resize the images in the datasets for our experiments.

Table 3: Datasets used in our experiments.

| Train | Test
---|---|---
Human faces | FFHQ [16] | CelebA-HQ [15]
Cats | LSUN-train [41] | CAT [43]
Cars | LSUN-train [41] | Cars [20]
Bedrooms | LSUN-train [41] | LSUN-validate [41]
Towers | LSUN-train [41] | LSUN-validate [41]

Following previous works [34, 35], the objective function for GLEAN consists
of three terms. MSE loss is used to guide the fidelity of the output images:

$\mathcal{L}_{mse}=\dfrac{1}{N}||\hat{y}-y||_{2}^{2},$ (5)

where $N$, $\hat{y}$, and $y$ denote the number of pixels, the output image,
and the ground-truth image, respectively. We further incorporate perceptual
loss [14] and adversarial loss [9] to improve the perceptual quality:

$\displaystyle\mathcal{L}_{percep}=\dfrac{1}{N}||f(\hat{y})-f(y)||_{2}^{2},$
(6) $\displaystyle\mathcal{L}_{gen}=\log\left(1-D(\hat{y})\right),$ (7)

where $f(\cdot)$ denotes the feature embedding space of the VGG16 [33]
network, and $D$ corresponds to the StyleGAN discriminator. The resulting
objective function is a weighted mean of the three losses:

$\mathcal{L}_{g}=\mathcal{L}_{mse}+\alpha_{percep}{\cdot}\mathcal{L}_{percep}+\alpha_{gen}{\cdot}\mathcal{L}_{gen}.$
(8)

In all our experiments, we set $\alpha_{percep}{=}\alpha_{gen}{=}10^{-2}$. For
the discriminator, we maximize

$\mathcal{L}_{d}=\log\left(1-D(\hat{y})\right)+\log D(y).$ (9)

We adopt Cosine Annealing Scheme [25] and Adam optimizer [19] in training. The
number of iterations is 300K and the initial learning rate is $10^{-4}$. The
batch size is 8 for human faces and 16 for other categories. We train our
models using two Nvidia V100 GPUs.

##  Appendix B Qualitative Results

###  B.1 Super-Resolution

Randomly-Selected Examples. In Fig. 12, we show the results of randomly-
selected examples from CelebA-HQ [15]. By optimizing only the latent codes,
PULSE [27] produces outputs with low-fidelity. In contrast, guided by the
encoder features and our generative latent bank, GLEAN achieves remarkable
quality and fidelity, demonstrating the effectiveness of our designs.

Scale Factors and Categories. GLEAN is extensible to various scale factors
(from $8{\times}$ to $64{\times}$) and categories ( faces, cats, cars,
bedrooms, towers). From Fig. 13 to Fig. 18, we see that GLEAN outperforms DGP
and ESRGAN+ in both fidelity and quality. It is noteworthy that the
performance of DGP and ESRGAN+ are less promising on categories other than
human faces.

###  B.2 Image Retouching

In interactive image retouching, users can manually edit the images based on
their preference. For instance, users can change the facial expression of an
object and perform geometric transformations for enlarging eyes. However, a
perfect output requires tedious and precise retouching. As a result, artifacts
are common in the outputs from amateur retouching.

GLEAN allows the possibility of performing realistic refinement of imperfect
retouching. More specifically, given a retouched image, we can first
downsample the image to a smaller resolution, where the artifacts vanished. We
can then upsample it back to the original resolution. With GLEAN as a powerful
super-resolver, we can obtain an output with unnatural artifacts suppressed.

As shown in Fig. 19, GLEAN is able to correct the unnatural artifacts
introduced by amateur retouching while being similar to the retouched images,
realistic, and coherent with the unaltered regions. In addition, since GLEAN
requires only a single forward pass, it can be used in interactive image
editing software to allow a more flexible retouching.

Figure 12: Comparison to PULSE on randomly-selected examples from CelebA-HQ
[15]. By optimizing only the latent vectors, the outputs of PULSE [27] differ
significantly from the ground-truths. With our novel designs, GLEAN produces
outputs highly similar to the ground-truths. Figure 13: Comparison with DGP
and ESRGAN+. The outputs of DGP show noticeable identity differences to the
ground-truths. ESRGAN+ shows unpleasant artifacts for the fine details. (Zoom-
in for best view) Figure 14: Performance of GLEAN on 16${\times}$,
32${\times}$, and 64${\times}$ SR. GLEAN is able to synthesize images well
resembling the ground-truths for up to $64{\times}$ upsampling. Figure 15:
(Top) Comparison with DGP and ESRGAN+ on Cats. DGP produces outputs with low
fidelity; ESRGAN+ fails to synthesize realistic textures. (Bottom) Performance
of GLEAN on 8${\times}$ SR. GLEAN produces realistic outputs that are highly
similar to the ground-truths. (Zoom-in for best view) Figure 16: (Top)
Comparison with DGP and ESRGAN+ on Cars. DGP produces outputs with low
fidelity; ESRGAN+ fails to synthesize realistic textures. (Bottom) Performance
of GLEAN on 8${\times}$ SR. GLEAN produces realistic outputs that are highly
similar to the ground-truths. (Zoom-in for best view) Figure 17: (Top)
Comparison with DGP and ESRGAN+ on Bedrooms. DGP produces outputs with low
fidelity; ESRGAN+ fails to synthesize realistic textures. (Bottom) Performance
of GLEAN on 8${\times}$ SR. GLEAN produces realistic outputs that are highly
similar to the ground-truths. (Zoom-in for best view) Figure 18: (Top)
Comparison with DGP and ESRGAN+ on Towers. DGP produces outputs with low
fidelity; ESRGAN+ fails to synthesize realistic textures. (Bottom) Performance
of GLEAN on 8${\times}$ SR. GLEAN produces realistic outputs that are highly
similar to the ground-truths. (Zoom-in for best view) Figure 19: Results on
image retouching. GLEAN can be used to correct unpleasant artifacts introduced
by amateur retouching. (Zoom-in for best view)