# E2GAN: Efficient Training of Efficient GANs for Image-to-Image Translation

Yifan Gong∗1,2, Zheng Zhan∗2, Qing Jin1, Yanyu Li1,2, Yerlan Idelbayev1, Xian
Liu1,
Andrey Zharkov1, Kfir Aberman1, Sergey Tulyakov1, Yanzhi Wang2, Jian Ren1
1 Snap Inc. 2 Northeastern University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

One highly promising direction for enabling flexible _real-time_ _on-device_
image editing is utilizing data distillation by leveraging large-scale text-
to-image diffusion models, such as Stable Diffusion, to generate paired
datasets used for training generative adversarial networks (GANs). This
approach notably alleviates the stringent requirements typically imposed by
high-end commercial GPUs for performing image editing with diffusion models.
However, unlike text-to-image diffusion models, each distilled GAN is
specialized for a specific image editing task, necessitating costly training
efforts to obtain models for various concepts. In this work, we introduce and
address a novel research direction: _can the process of distilling GANs from
diffusion models be made significantly more efficient?_ To achieve this goal,
we propose a series of innovative techniques. First, we construct a base GAN
model with generalized features, adaptable to different concepts through fine-
tuning, eliminating the need for training from scratch. Second, we identify
crucial layers within the base GAN model and employ Low-Rank Adaptation (LoRA)
with a simple yet effective rank search process, rather than fine-tuning the
entire base model. Third, we investigate the minimal amount of data necessary
for fine-tuning, further reducing the overall training time. Extensive
experiments show that we can efficiently empower GANs with the ability to
perform real-time high-quality image editing on mobile devices with remarkable
reduced training cost and storage for each concept. 111Project website:
https://yifanfanfanfan.github.io/e2gan/

###  1 Introduction

Recent development of diffusion-based image editing models has witnessed
remarkable progress in synthesizing contents containing photo-realistic
details with full of imagination [39, 38, 35, 36]. Albeit being creative and
powerful, these generative models typically require a huge amount of
computation even for inference and storage for saving weights. For example,
Stable Diffusion [38] has more than one billion parameters and takes $30$
seconds to conduct iterative denoising process to get one image on T4 GPU.
Such low-efficiency issue prohibits their real-time application on mobile
devices [24].

Exiting works try to tackle the problem through two main directions. One is
accelerating the diffusion models by designing efficient model architecture or
reducing the number of denoising steps [40, 26, 23, 16]. However, these
efforts still struggle to obtain models that can run in real-time on mobile
devices [24]. Another area focuses on data distillation, where diffusion
models are leveraged to create datasets to train other mobile-friendly models,
such as generative adversarial networks (GANs) for image-to-image translation
[54, 33]. Nevertheless, although GAN is efficient for on-device deployment,
each new concept still asks for the _costly training_ of a GAN model from
_scratch_.

In this work, we propose and aim to address a new research direction: can the
GAN models be trained efficiently under the data distillation pipeline to
perform real-time on-device image editing?

To tackle the challenge, we introduce E2GAN, powered with the following
techniques for the Efficient training and Efficient inference of GAN models
with the help of diffusion models:

  * •

First, we construct a base GAN model trained from various prompts and the
corresponded edited images obtained from diffusion models. It enables
efficient transfer learning for different new concepts by fine-tuning, rather
than training models from scratch, to reduce the training cost. Meanwhile, the
base GAN model achieves fast inference with fewer parameters on mobile devices
(as in Fig. 1 _Right_), and maintaining high-performance.

  * •

Second, we identify that only partial layers are necessary to be fine-tuned
for new concepts. LoRA is applied on these layers with a simple yet effective
rank search process, eliminating the need to fine-tune the entire base model
(as in Fig. 1 _Left_). It brings two advantages – both the training cost and
saving storage for each new concept are significantly reduced.

  * •

Third, we investigate the amount of data for fine-tuning the base model for
various concepts. The reducing the amount of training data helps reduce the
training cost and time for adapting the base model to new concepts.

Figure 1: Overview of E2GAN. _Left : Training Comparison._ Conventional GAN
training, such as pix2pix [13] and pix2pix-zero-distilled that distills Co-
Mod-GAN [54] using data from pix2pix-zero [33], requires all the weights
trained from scratch, while our efficient training significant reduces the
training cost by only fine-tuning $1\%$ weights with only _portion_ of
training data. _Right : Mobile Inference Comparison._ Our efficient on-device
model can achieve real-time ($30$FPS, iPhone 14) runtime and is faster than
pix2pix and diffusion model, while the pix2pix-zero-distilled model (Co-Mod-
GAN) is not supported on device.

We show extensive experiments results to demonstrate that by using our
approach, we can efficiently distill the image editing capability from a
large-scale text-to-image diffusion model into GAN models via data
distillation (examples in Fig. 5). The distilled GAN model showcases real-time
image editing capabilities on mobile devices. We hope our work can shed light
on how to democratize the diffusion models into efficient on-device computing.

###  2 Related Works

Generative Models learn the joint data distribution to generate new samples,
such as VAEs [18, 37], GANs [7, 55, 30], auto-regressive models [46, 41, 46,
27, 50], and diffusion models [43, 10, 29, 44, 45, 3]. Among these, diffusion
models demonstrate strong capability of generating images with high-fidelity
[36, 38], at the cost of bulky model size and numerous sampling steps during
inference. Several studies try to accelerate the image generation process of
the diffusion models [40, 26, 23]. However, they still struggle to achieve
real-time on-device generation. On the contrary, GANs are more efficient in
terms of model size and inference speed for image editing [22, 14, 47]. To
this end, we leverage the approach of data distillation to transfer knowledge
from diffusion models to lightweight GANs that are compatible with real-time
inference on mobile devices.

Efficient GANs. Exiting works actively explore the reduction of the inference
runtime for GANs by using various model compression techniques, such as
efficient architecture design [22, 14], network pruning and quantization [47,
48], and neural architecture search [47, 5]. On the other hand, the research
about training cost savings for GANs is quite limited, as most works typically
train all the parameters of a GAN model from scratch for the image-to-image
translation task, involving large computing efforts. This work aims to fine-
tune a very small portion, _i.e_. $1\%$, of the pre-trained models with the
partial training data to reduce the training cost. Thus, the training of GAN
can be tiny in terms of both parameters and data. There are many efforts on
efficient training [12, 19], in particular the sparse training [4, 21, 52].
However, these methods rely on the mask of trainable parameters, which in turn
is determined during training with a huge bunch of data. In contrast, our
method adopts pre-defined learnable components, and only fine-tunes on a small
fragment of data to make the transfer learning progress efficient and
effective.

###  3 Motivation

Though diffusion models possess the advantages of generating high-fidelity
images, the huge model size, high computation cost, and numerous sampling
steps pose significant challenges for implementation on widely adopted mobile
platforms with limited capacities. Even recent attempts at accelerating
diffusion models, such as SnapFusion [24], still require nearly 2 seconds to
generate a single image on an iPhone 14 Pro, as shown in Tab. 1. This
efficiency issue strictly hinders their real-time application, _e.g_., image
editing with $30$ frames per second (FPS).

Table 1: Comparison of model size, FLOPs, and latency for different works [24,
13, 33]. Co-Mod-GAN [54] is trained following the pipeline in Pix2pix-zero
[33]. Reported latency is averaged over $100$ runs on iPhone 14 Pro. The
training time of pix2pix and Co-Mod-GAN is measured on a single NVIDIA H100
GPU.

Model |  | Param
---
num
FLOPs | Latency |  | Train
---
time
SnapFusion | 861M |  $>$1T | 1956 ms | 7680 hours
Pix2pix with 9 RB | 11.4M | 56.9G | 21.0 ms | 16 min
Co-Mod-GAN | 79.2M | 98.2G | not supported | 110 min
|  |  |  |

In contrast, various efficient and mobile-friendly GAN designs exist. For
instance, the pix2pix model with 9 ResNet Blocks (RBs) takes only 21 ms to
generate an edited image on an iPhone 14 Pro. Recognizing the inefficiency in
directly accelerating diffusion models and the lightweight nature of certain
GANs, researchers have explored data distillation as an alternative research
direction. This approach involves transferring the knowledge of diffusion
models to GANs. Latest work pix2pix-zero [33] creates training data to train
Co-Mod-GAN for model acceleration, yet it is not supported on mobile devices.
Furthermore, the training time to obtain the Co-Mod-GAN for a new concept is
still costly, which takes 110 min as shown in Tab. 1.

To overcome the above-mentioned limitations, the objective of this work is to
achieve efficient distillation of diffusion models to mobile-friendly real-
time GANs. Specifically, efficient distillation refers to minimizing the
training efforts needed to obtain the GAN model for a new concept.
Furthermore, when deployed on a mobile device after efficient distillation,
the mobile-friendly real-time GANs should exhibit low latency ($<$33.3 ms) and
demand minimal storage for a new concept.

###  4 Methods

Here we present our approach that efficiently transfers the image editing
capability from large-scale text-to-image diffusion models to on-device real-
time GANs. In the following sections, we first give an overview of our
knowledge transfer pipeline (Sec. 4.1). Then, we study efficient training
strategies to get a series of on-device models with _fewer_ training costs and
_less_ storage, while maintaining high-quality image generation (Sec. 4.2).

####  4.1 Overview of Knowledge Transfer Pipeline

Benefits of Knowledge Transfer. Despite the powerful image generating
capability of text-to-image diffusion models, they suffer from large model
size and inefficient sampling [24]. For example, the denoising UNet of Stable
Diffusion [38] has $860$ millions of parameters. Even the latest efficient
diffusion model, SnapFusion [24], costs around $2$ seconds to generate a
single image on iPhone 14 Pro, limiting the real-time applications for
diffusion models. On the contrary, the image-to-image GANs are typically
light-weight with much fewer parameters that can synthesize one image with
only one forward pass. For instance, a pix2pix model with ResNet-based network
has a latency of around $21$ms on iPhone 14 Pro. Thus, these efficient
architectures are more compatible with the mobile devices to achieve real-time
image generation. To leverage the benefits of both models, we employ GANs as
the image editing model and transfer knowledge of diffusion models to GAN by
training the GANs with data created by the diffusion model. Such a data
distillation pipeline serves as the foundation for our whole framework.

Pipeline for Dataset Creation. To enable the data distillation, we use the
diffusion models to edit real images to obtain the edited images, forming
pairs of data along with the used text prompts to create the training
datasets, which then be utilized to train the image-to-image GAN model. The
real images come from FFHQ [15] and Flickr-Scenery [2], covering diverse
content and are challenging for content editing. For diffusion models, we
choose the recent works for image editing, such as Stable Diffusion [38],
Instruct-Pix2Pix (IP2P) [1], Null-text Inversion (NI) [28], ControlNet [53],
and InstructDiffusion [6].

Figure 2:  FID comparison of applying TBs in image generators trained on two
datasets (_Left:_ forest during autumn, _Right:_ forest in the dusk). Vertical
axis shows the position to inserting TBs. Pix2pix-zero-distilled uses pix2pix-
zero for creating datasets to train Co-Mod-GAN [35]. Figure 3: Overview of
E2GAN model architecture. The generator is composed of down/up-sampling
layers, 3 RBs, and 1 TB. The base generator is trained on multiple
representative concepts. New concepts are achieved by fine-tuning LoRA
parameters on crucial layers.

Training Objectives. With paired images and the associated prompts, we train
the efficient GANs for image translation by using the conventional adversarial
loss. Specifically, given the original image $\mathbf{x}$ and the editing
prompt $\mathbf{c}$, the image generator $\mathcal{G}$ and discriminator
$\mathcal{D}$ are jointly optimized as follows:

$\displaystyle\min_{\theta_{g}}\max_{\theta_{d}}\lambda\underbrace{\mathbb{E}_{\mathbf{x},\tilde{\mathbf{x}}^{c},\mathbf{z},\mathbf{c}}\left[\|\tilde{\mathbf{x}}^{c}-\mathcal{G}(\mathbf{x},\mathbf{z},\mathbf{c};\theta_{g})\|_{1}\right]}_{\textrm{$\ell_{1}$
loss}}+$ (1)
$\displaystyle\underbrace{\mathbb{E}_{\mathbf{x},\tilde{\mathbf{x}}^{c}}\left[\log\mathcal{D}(\mathbf{x},\tilde{\mathbf{x}}^{c};\theta_{d})\right]+\mathbb{E}_{\mathbf{x},{\mathbf{z}},\mathbf{c}}\left[\log(1-\mathcal{D}(\mathbf{x},\mathcal{G}(\mathbf{x},\mathbf{z},\mathbf{c};\theta_{g});\theta_{d}))\right]}_{\textrm{conditional
GAN loss}},$

where $\tilde{\mathbf{x}}^{c}$ denotes the image generated by the diffusion
model ${\epsilon}$ conditioned on the concept $\mathbf{c}$ of the target
style, _i.e._ ,
$\tilde{\mathbf{x}}^{c}=\mathbf{\epsilon}(\mathbf{x},\mathbf{c})$,
$\mathcal{G}$ and $\mathcal{D}$ denote the generator and discriminator
function parameterized by $\theta_{g}$ and $\theta_{d}$, respectively,
$\mathbf{z}$ is a random noise introduced to increase the stochasticity of
output, and $\lambda$ can be used to adjust the relative importance between
$\ell_{1}$ loss and the conditional GAN loss.

####  4.2 Efficient Training of GAN Models

Diffusion-based generative models can perform image editing on-the-fly, while
existing lightweight GAN-based networks require training to be adapted to the
new concept. The training of GAN models for various image translation tasks
require substantial computation costs. Additionally, there is high storage
demand for saving the trained weights. To mitigate such training and storage
costs, we introduce three main techniques to reduce the number of trainable
parameters and the demanded data for model fine-tuning: _First_ , we establish
a _base GAN model_ equipped with generalized features and representations,
ready to be leveraged for new concepts (Sec. 4.2.1). _Second_ , starting from
the base model, we identify key parameters to optimize during fine-tuning for
a new concept, bolstered by the application of Low-Rank Adaptation (LoRA) [11]
to further reduce the number of parameters (Sec. 4.2.2). _Third_ , we explore
the possibility of tiny fine-tuning where the training data are first
clustered and only those near the cluster centers are used (Sec. 4.2.3).

#####  4.2.1 Base GAN Model Construction

To obtain model weights for a new target concept with as few training efforts
as possible, we explore transfer learning from a pre-trained base GAN model,
instead of training from scratch. The base model should possess the capability
of a more general features and representations, which can be learned from
multiple image translation tasks, allowing the new concept to leverage
existing knowledge. Thus, we opt to train the base model on a mixed dataset
comprising diverse concepts.

Table 2: The model size, FLOPs, and latency of E2GAN. The reported latency is
an average of 100 runs measured on the GPU of an iPhone 14 Pro.

Model | Param num | FLOPs | Latency
---|---|---|---
3RB+1TB | 7.1M | 23.6G | 15.5 ms
3RB+2TB | 10.1M | 26.6G | 21.0 ms
|  |  |

The construction of the image-to-image model $\mathcal{G}$ serves as the first
step in obtaining such a base model. This model should fulfill three key
criteria: (1) the ability to learn multiple concepts; (2) achievement of real-
time inference on mobile devices; and (3) strong image generation
capabilities. We start from the classic ResNet generator with $9$ RBs that is
widely adopted [13, 55, 31]. To incorporate the text information of the
concept and facilitate a more holistic understanding of global shapes and
structure, we introduce Transformer Blocks (TBs) with self-attention and
cross-attention modules into the architecture. For expedited inference
purpose, we reduce the number of RBs from $9$ to $3$. The subsequent steps
involve determining the number and position of TBs.

Number of TBs. We train models with different architecture designs, _e.g_.
different number of TBs, and evaluate both the efficiency (in terms of model
size, FLOPs, and latency) and image generation capability (in terms of the FID
[9] between the images generated by GANs and diffusion models). The results
are presented in Tab. 2 and Fig. 2, respectively. Interestingly, we find that
one TB is enough for generating high quality images. Introducing more TBs does
not further improve the performance yet brings in more computation cost.
Notice that to reduce the inference cost of the introduced TB, we apply a
downampling operation to halve the feature map size before sending it into the
TB, and use an upsampling layer to recover the feature map size for the
following operations.

Position of TBs. Additionally, we find that the position of the TB is
important for the final performance of the image generation. First, the TB
should be placed between the last downsample layers and the first upsample
layers to avoid high computations on mobile devices, due to the high
resolution of features. Second, we apply attention to different positions of
the network bottleneck. Particularly, the TB can be inserted between one of
the following: (1) before the first RB; (2) after the first RB; (3) after the
second RB; and (4) after the third RB. As evident in Fig. 2, all these options
lead to a generator with better performance than the conventional CONV-only
networks used in pix2pix [13] and pix2pix-zero-distilled [33]. For our model,
we place the TB after the second RB.

Therefore, our architecture is finalized with an overall architecture in Fig.
3. It achieves faster inference speeds, reduces the number of parameters, and
lowers computational costs compared to existing image-to-image models, as
shown in in Tab. 2 and Fig. 2. With the architecture determined, the base
model is trained on a subset of concepts denoted as
$\mathcal{C}=\\{\mathbf{c}_{1},\cdots,\mathbf{c}_{K}\\}$, where each concept
$\mathbf{c}_{k}$ is selected among different concepts by K-means clustering
[25] based on the average of the CLIP image embedding [34] of uniformly
sampled images.

#####  4.2.2 Crucial Weights for Fine-Tuning

Figure 4: Crucial weights analysis via freezing partial weights in the base
model. (a) Number of parameters for each part of the base model; (b) Averaged
FID across $10$ different concepts on the Flicker-Scenery dataset when
freezing partial weights of base model. ‘-’ indicates fine-tune all the
weights; (c) The generated images when freezing each part of the base model.

To save the training and storage cost, we reduce the number of trainable
parameters during fine-tuning. Specifically, we pre-define trainable layers
that occupy a small portion of weights from the base model. Then, we apply
LoRA on top of the trainable layers. In this way, we only optimize $1.29\%$ of
the weights from the base model during the fine-tuning, greatly reducing the
training cost and storage cost for a new concept.

Inspired by the recent work of customized diffusion [20], where a pre-trained
diffusion model can be fine-tuned to a personalized version with updating only
partial weights, we explore the feasibility to identify the minimal set of
tunable weights for GAN. We aim to determine such a set that is sufficient for
fine-tuning the base model to a new target concept. To this end, we analyze
the components of the GAN model, which mainly consist of three parts: (1)
sampling layers (SL) with downsampling and upsampling; (2) transformer block
(TB); and (3) intermediate ResBlock (RB).

Identifying Crucial Layers. We systematically and empirically study the impact
of each part in the image-to-image task by freezing each part individually,
with results provided in Fig. 4. Combining Fig. 4(b) & (c), we see that SL
plays a more crucial role in maintaining the quality of generated images,
identified by the high FID score and low image quality. SL might be more
crucial for constructing the desired output texture, yet intermediate RB might
contain lower-level information that are common among styles. Meanwhile,
compared to RB, TB has a fewer amount of parameters ($1.58$M _v.s._ $3.54$M in
Fig. 4(a)), while it is more important in keeping performance ($123.6$ _v.s._
$111.3$ in Fig. 4(b)). Considering the situation with a limited training
budget, RB has a lower priority to be optimized.

LoRA on Crucial Layers. From the perspective of maintaining image generating
quality, it is better to include TB in training as self-attention modifies the
image with a better holistic understanding and the cross-attention module
takes the information from the given target concept. However, combing SL and
TB leads to $3.42$M parameters to be updated, taking account $47.90\%$ of the
entire model weights. To fine-tune the crucial layers with much fewer
trainable parameters, we investigate the best way of incorporating Low-Rank
Adaptation (LoRA) [11] into GAN training, which introduces two trainable low-
rank weight matrices besides the original weight for each layer identified as
crucial.

Rank for LoRA. With the leverage of LoRA, when fine-tuning to a new concept,
the weights of base model are _frozen_ , while only the two low-rank matrices
with much fewer parameters for each crucial layer are updated to save
computation and storage costs. For instance, for a CONV layer $i$ with weights
$\small\theta_{i}\in\mathbb{R}^{h\times w\times k_{h}\times k_{w}}$, we apply
two low-rank matrices with rank $r_{i}$, _i.e_.
$\small\theta_{i}^{A}\in\mathbb{R}^{h\times r_{i}\times k_{h}\times k_{w}}$
and $\small\theta_{i}^{B}\in\mathbb{R}^{r_{i}\times w\times 1\times 1}$, to
approximate the gradient update $\nabla\theta_{i}$. Given multiple crucial
layers, determining the appropriate rank for _each of them_ is important.
Prior works mostly rely on manual setting [11] for deciding the rank value,
due to a huge search space for the rank. However, in our task, the rank should
be pre-fixed for different concepts to avoid the rank search process when a
new concept comes. To tackle this challenge, we randomly sample $K$ concepts
and conduct a simple yet effective rank search process. For each concept, we
start by assigning $r_{i}$ as 1 for each crucial layer $i$, and upscale the
rank for every $e$ epochs by doubling the rank value, until $r_{i}$ reaches
the upper threshold $\tau_{i}$ for the layer $i$. The threshold $\tau_{i}$ is
determined by the size of the weight. We evaluate the FID performance at the
end of each $e$ training epochs. If the performance saturates, the rank value
from the best FID performance setting is returned as the rank for the concept.
Typically, a larger rank can provide more model capability. Thus, the largest
returned rank among the $K$ selected concepts is viewed as $r^{*}$ for the
future use of a new concept.

#####  4.2.3 Similarity Clustering to Reduce Training Data

Reducing the amount of training data can directly result in a reduction in the
training time. Thus, we aim to investigate data efficiency as a means of
decreasing the training cost in addition to the crucial weight update for
E2GAN. We find not all data are indispensable for reliable training, but only
a small subset is necessary. We obtain this small subset in an unsupervised
manner with a selection of the data crowding around the clustering center on
the whole dataset.

To identify the small subset of essential data, we conduct unsupervised
learning to analyze the structure of the training data. We first extract an
embedding for each image $\mathbf{x}$ with an extractor $\mathcal{E}$. Then,
we apply clustering on the embeddings by the K-Means algorithm [25] to obtain
$K<N$ clusters ($N$ it the total number of images), each with center
$\mu_{k}$. The embeddings within the same cluster have a closer distance among
each other, indicating a higher _similarity_ of the data points. To reduce the
data amount while maintaining data diversity for the good generalization
ability of the model, one data point, which is the closet to the center
$\mu_{k}$, is selected for each of the $K$ clusters.

With our data selection method using $K$ clusters, we further reduce the
number of training iterations by $N/K$ times. In contrast to prior methods
involving additional computations in the training process to shrink the
dataset [51, 49], our Similarity Clustering (SC) data reduction is tailored
for expediting the training of image editing tasks. It reduces the training
data volume directly before the training process without incurring any
additional costs during the training.

###  5 Experiments

In this section, we provide the detailed experimental settings and results of
our proposed method. More details as well as some ablation studies can be
found in the Appendix.

Figure 5: Qualitative comparisons on various tasks. The _leftmost_ column
shows two original images and the remaining columns present the corresponding
synthesized images in the target concept domain, where target prompts are
shown at the bottom row. We provide images generated by various models.

Paired Data Preparation. We verify our method on $1,000$ images from FFHQ
dataset [15] and Flickr-Scenery dataset [2] with image resolution as
$256\times 256$. The images in the target domain are generated with several
different text-to-image diffusion models, including Stable Diffusion [38],
IP2P [1], NI [28], ControlNet [53], and InstructDiffusion [6]. The generated
images with the best perceptual quality among diffusion models are selected to
form with the real images into paired datasets. To perform training and
evaluation of GAN models, we divide the image pairs from each target concept
into training/validation/test subsets with the ratio as $80\%$/$10\%$/$10\%$.

Baselines. We compare E2GAN with image-to-image translation methods like
pix2pix [13] (image generator with $9$ ResNet blocks) and pix2pix-zero-
distilled that distills Co-Mod-GAN [54] using data generated by pix2pix-zero
[33].

Training Setting. We follow the standard approach that alternatively updates
the generator and discriminator [7]. The training is conducted from an initial
learning rate $2e-4$ with mini-batch SGD using Adam solver [17]. The total
training epochs is set to $100$ for E2GAN, and $200$ for pix2pix [13] and
pix2pix-zero-distilled [33] for them to converge well. For SC (Sec. 4.2.3), we
choose the cluster number as $400$ and use the feature extractor $\mathcal{E}$
as FaceNet [42] on FFHQ dataset and CLIP image encoder [34] on Flicker Scenery
dataset. To train the base model, we use $20$ prepared tasks/datasets from
FFHQ dataset and $7$ from Flicker Scenery dataset.

Evaluation Metric. We compare the images generated by GAN-based models and
diffusion-based models by calculating Clean FID proposed by [32] on the test
sets.

####  5.1 Experimental Results

Qualitative Results. The synthesized images in the target domain obtained by
E2GAN and other methods are shown in Fig. 5. The original images are listed at
the leftmost column, and the synthesized images for the target concept
obtained by diffusion models, pix2pix, pix2pix-zero-distilled, and E2GAN are
shown from top to bottom. The tasks span a wide range, such as changing the
age, artistic styles, and editing the seasons. According to the results, E2GAN
is able to modify the original images to the target concept domain by updating
only the LoRA parameters. For instance, for the green lantern prompt on FFHQ
dataset, diffusion model fails to modify the image, pix2pix and pix2pix-zero-
distilled add colors to wrong areas, while E2GAN generates the image that fits
the prompt well. As for the add blossoms prompt on the Flicker Scenery
dataset, E2GAN preserves the structure of the original image better than other
models while editing the image as desired.

Figure 6:  Training cost comparison of conventional GAN and E2GAN. _Left_ :
Training FLOPs. _Middle_ : Training time. _Right_ : Number of parameters that
required gradient update, which also equals to the weights need to be saved
for a concept. Table 3:  FID comparison. FID is calculated between the images
generated by GAN-based approaches and diffusion models. Reported FID is
averaged across different concepts ($30$ for FFHQ and $10$ for Flicker
Scenery).

Method Dataset | FFHQ | Landscape
---|---|---
Pix2pix | 86.03 | 114.2
Pix2pix-zero-distilled | 87.76 | 132.6
E2GAN | 80.28 | 109.37
|  |

Quantitative Comparisons between E2GAN and other baseline methods are provided
in Tab. 3. Note that for each concept, pix2pix and pix2pix-zero-distilled are
trained on the whole training dataset of $800$ samples. E2GAN begins with a
base model and is fine-tuned with only $400$ samples to obtain models for
different target concepts. The results demonstrate E2GAN is able to reach a
even better FID performance than the conventional GAN training techniques like
pix2pix and pix2pix-zero-distilled, indicating high-fidelity of generated
images.

Training Cost Analysis between E2GAN and other approaches are provided in Fig.
6. Compared with pix2pix and pix2pix-zero-distilled, E2GAN greatly saves the
training FLOPs of $14\times$ and $25\times$, respectively, and accelerates the
training time by $4.8\times$ and $33\times$, respectively. Moreover, E2GAN
only requires updating $0.092$M parameters for a new concept, greatly saving
the storage requirement when training models for various tasks/concepts,
_i.e_. $869\times$ less than pix2pix-zero-distilled.

Notably, E2GAN requires _much fewer_ trainable parameters, training data, and
training time than other GAN-based approaches to reach even _better_
generation quality, _i.e_. E2GAN has lower FID than pix2pix on FFHQ ($80.28$
_v.s._ $86.03$). Furthermore, E2GAN enjoys a faster inference speed on mobile
devices (Tab. 1). The good performance of E2GAN is originated from our
effective framework design, including the efficient model architecture and
efficient training strategy (Sec. 4.2).

####  5.2 Ablation Analysis

We provide ablation analysis to understand the impact of each components. We
first study the effectiveness of the base model determination. After that, we
provide an analysis on the LoRA rank search. Finally, we discuss the effect of
our data selection.

Table 4:  Analysis (FID) of various base models on FFHQ.

Concept Base model | Ours |  | 20
---
random
| 200 art
---
concepts
| Single
---
concept
White walker | 40.18 | 53.92 | 40.32 | 51.99
Blond person | 48.01 | 52.77 | 61.50 | 55.58
Sunglasses | 38.49 | 40.54 | 41.37 | 44.12
Vangogh style | 71.82 | 78.58 | 68.21 | 78.06
|  |  |  |

Analysis of Base Model Determination. We study the impacts of our base model
determination method discussed in Sec. 4.2.1 by comparing our method with the
following three settings: (1) train the base model on $20$ _random_ concepts;
(2) train the base model on $200$ artist concepts; (3) train the base model on
a _single_ concept old person from the FFHQ dataset. The results are
demonstrated in Tab. 4. Note our method is obtained by training on $20$
selected representative concepts. The results indicate our base model
construction outperforms or matches the alternatives across the evaluated
concepts. This underscores the efficacy of our base model in enhancing
performance. In contrast, the single concept base model generally performs
worse. Furthermore, simply increasing the amount of concepts does not
necessarily leads to better performance as indicated by training the base
model with $200$ art concepts.

Table 5: Analysis of searching LoRA rank on the Flicker Scenery dataset. The
reported FID values are averaged over $10$ different target concepts.

Scheme | FID | # of Param
---|---|---
Our searched | 109.37 | 0.092M
Upcale 1$\times$ | 130.98 | 0.056M
Upcale 4$\times$ | 111.42 | 0.164M
Random | 129.87 | 0.100M
|  |

Analysis of LoRA on Crucial Layers. Tab. 5 presents an evaluation of the
effectiveness of our LoRA rank search on the Flicker Scenery dataset. The
table reports the FID averaged across $10$ different target concepts, as well
as the number of LoRA parameters for various schemes. We compare our method
with the other three settings: (1) upscale the rank $1\times$ for each crucial
layer by doubling the rank from the initialization until the rank reaches the
threshold; (2) upscale the rank $4\times$ for each crucial layer from the
initialization; and (3) random assign ranks for the crucial layers. Our
searched scheme achieves the lowest FID value of $109.37$ while maintaining a
relatively low number of parameters as $0.092$M. This demonstrates the
effectiveness of our LoRA rank search approach.

Figure 7:  Comparisons of Data Selection Rule. Prompts for the _left_ and
_right_ figures are old person and put on a pair of sunglasses, respectively.

Analysis of Cluster Number of Data Selection. To investigate our sampling rule
SC for obtaining training samples (proposed in Sec. 4.2.3 to reduce the number
of training data), we compare it with randomly sampling method. Random
sampling is implemented as shuffling the training data randomly and only
accessing the first $K$ examples as training data. The comparisons are
conducted with different number of training samples $K$. We show the results
in Fig. 7 and can draw the following observations. First, SC provides better
FID performance in all scenarios, indicating the effectiveness of our sampling
method by enriching data diversity. Second, the cluster number, _i.e_. the
number of target training samples, influences the SC performance to some
extent. More training examples (clusters) do not necessarily lead to better
performance. Furthermore, SC can work for a wide range of different number of
training samples by providing models with good FID performance.

###  6 Conclusion

This paper addresses the growing demand for efficient on-device image editing
by introducing a novel research direction, that is the efficient training of
efficient GAN models via distilling the large-scale text-to-image diffusion
models with data distillation. The proposed framework, E2GAN, incorporates a
hybrid training pipeline that can efficiently adapt a pre-trained text-
conditioned GAN model, which has real-time inference speed on mobile devices,
to different concepts, while significantly mitigating computational and
storage demands. Extensive experimental results validate the effectiveness of
E2GAN, that enables real-time image editing capabilities on mobile devices and
brings the power of diffusion models for efficient on-device computing.

Limitation and Discussion. Generating high-quality images using diffusion
models can be challenging for diverse prompts, which in turn restricts the
expansion of our training datasets. Moreover, utilizing diffusion models for
data collection remains an expensive endeavor. Developing efficient techniques
to rapidly construct well-paired and high-quality datasets from diffusion
models would greatly enhance the training of E2GAN. Furthermore, while our
approach does succeed in reducing training costs, alternative methods such as
hyper-networks [8] have the potential to eliminate the need for training. This
could result in the seamless obtaining real-time on-device GAN models as
needed without any fine-tuning.

### References

  * Brooks et al. [2022] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  _arXiv preprint arXiv:2211.09800_ , 2022.
  * Cheng et al. [2022] Yen-Chi Cheng, Chieh Hubert Lin, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, and Ming-Hsuan Yang.  Inout: Diverse image outpainting via gan inversion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11431–11440, 2022.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ , 34:8780–8794, 2021.
  * Evci et al. [2020] Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen.  Rigging the lottery: Making all tickets winners.  In _International Conference on Machine Learning_ , pages 2943–2952. PMLR, 2020.
  * Fu et al. [2020] Yonggan Fu, Wuyang Chen, Haotao Wang, Haoran Li, Yingyan Lin, and Zhangyang Wang.  Autogan-distiller: Searching to compress generative adversarial networks.  _arXiv preprint arXiv:2006.08198_ , 2020.
  * Geng et al. [2023] Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Han Hu, Dong Chen, et al.  Instructdiffusion: A generalist modeling interface for vision tasks.  _arXiv preprint arXiv:2309.03895_ , 2023.
  * Goodfellow et al. [2020] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Ha et al. [2017] David Ha, Andrew M. Dai, and Quoc V. Le.  Hypernetworks.  In _International Conference on Learning Representations_ , 2017.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Huang et al. [2019] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al.  Gpipe: Efficient training of giant neural networks using pipeline parallelism.  _Advances in neural information processing systems_ , 32, 2019.
  * Isola et al. [2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1125–1134, 2017.
  * Jin et al. [2021] Qing Jin, Jian Ren, Oliver J Woodford, Jiazhuo Wang, Geng Yuan, Yanzhi Wang, and Sergey Tulyakov.  Teachers do more than teach: Compressing image-to-image models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13600–13611, 2021.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4401–4410, 2019.
  * Kim et al. [2023] Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi.  On architectural compression of text-to-image diffusion models.  _arXiv preprint arXiv:2305.15798_ , 2023.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Köster et al. [2017] Urs Köster, Tristan Webb, Xin Wang, Marcel Nassar, Arjun K Bansal, William Constable, Oguz Elibol, Scott Gray, Stewart Hall, Luke Hornof, et al.  Flexpoint: An adaptive numerical format for efficient training of deep neural networks.  _Advances in neural information processing systems_ , 30, 2017.
  * Kumari et al. [2022] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  _arXiv preprint arXiv:2212.04488_ , 2022.
  * Lee et al. [2019] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr.  Snip: Single-shot network pruning based on connection sensitivity.  In _ICLR_ , 2019.
  * Li et al. [2020] Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Zhu, and Song Han.  Gan compression: Efficient architectures for interactive conditional gans.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5284–5294, 2020.
  * Li et al. [2022] Muyang Li, Ji Lin, Chenlin Meng, Stefano Ermon, Song Han, and Jun-Yan Zhu.  Efficient spatially sparse inference for conditional gans and diffusion models.  _arXiv preprint arXiv:2211.02048_ , 2022.
  * Li et al. [2023] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren.  Snapfusion: Text-to-image diffusion model on mobile devices within two seconds.  _arXiv preprint arXiv:2306.00980_ , 2023.
  * Lloyd [1982] Stuart Lloyd.  Least squares quantization in pcm.  _IEEE transactions on information theory_ , 28(2):129–137, 1982.
  * Meng et al. [2022] Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans.  On distillation of guided diffusion models.  _arXiv preprint arXiv:2210.03142_ , 2022.
  * Menick and Kalchbrenner [2018] Jacob Menick and Nal Kalchbrenner.  Generating high fidelity images with subscale pixel networks and multidimensional upscaling.  _arXiv preprint arXiv:1812.01608_ , 2018.
  * Mokady et al. [2022] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  _arXiv preprint arXiv:2211.09794_ , 2022.
  * Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pages 8162–8171. PMLR, 2021.
  * Park et al. [2019] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019.
  * Park et al. [2020] Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu.  Contrastive learning for unpaired image-to-image translation.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16_ , pages 319–345. Springer, 2020.
  * Parmar et al. [2022] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu.  On aliased resizing and surprising subtleties in gan evaluation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11410–11420, 2022.
  * Parmar et al. [2023] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu.  Zero-shot image-to-image translation.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pages 1–11, 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Rezende et al. [2014] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  In _International conference on machine learning_ , pages 1278–1286. PMLR, 2014.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Salimans et al. [2017] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma.  Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications.  _arXiv preprint arXiv:1701.05517_ , 2017.
  * Schroff et al. [2015] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 815–823, 2015.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning_ , pages 2256–2265. PMLR, 2015.
  * Song et al. [2020a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song et al. [2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Van Den Oord et al. [2016] Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.  Pixel recurrent neural networks.  In _International conference on machine learning_ , pages 1747–1756. PMLR, 2016.
  * Wang et al. [2020] Haotao Wang, Shupeng Gui, Haichuan Yang, Ji Liu, and Zhangyang Wang.  Gan slimming: All-in-one gan compression by a unified optimization framework.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16_ , pages 54–73. Springer, 2020.
  * Wang et al. [2019] Peiqi Wang, Dongsheng Wang, Yu Ji, Xinfeng Xie, Haoxuan Song, XuXin Liu, Yongqiang Lyu, and Yuan Xie.  Qgan: Quantized generative adversarial networks.  _arXiv preprint arXiv:1901.08263_ , 2019.
  * Wang et al. [2022] Zifeng Wang, Zheng Zhan, Yifan Gong, Geng Yuan, Wei Niu, Tong Jian, Bin Ren, Stratis Ioannidis, Yanzhi Wang, and Jennifer Dy.  Sparcl: Sparse continual learning on the edge.  _arXiv preprint arXiv:2209.09476_ , 2022.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ , 2022.
  * Yuan et al. [2021] Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al.  Mest: Accurate and fast memory-economic sparse training framework on the edge.  _Advances in Neural Information Processing Systems_ , 34:20838–20850, 2021.
  * Yuan et al. [2022] Geng Yuan, Yanyu Li, Sheng Li, Zhenglun Kong, Sergey Tulyakov, Xulong Tang, Yanzhi Wang, and Jian Ren.  Layer freezing & data sieving: Missing pieces of a generic framework for sparse training.  _Advances in Neural Information Processing Systems_ , 35:19061–19074, 2022.
  * Zhang and Agrawala [2023] Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models, 2023.
  * Zhao et al. [2021] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu.  Large scale image completion via co-modulated generative adversarial networks.  _arXiv preprint arXiv:2103.10428_ , 2021.
  * Zhu et al. [2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2223–2232, 2017.

Supplementary Material

##

###  Appendix A More Implementation Details

####  A.1 Details for Diffusion Model

We apply most recent diffusion-based image editing models to create paired
datasets, which include Stable Diffuison (SD) [38], Instruct-Pix2Pix (IP2P)
[1], Null-text inversion (NI) [28], ControlNet [53], and Instruct Diffusion
[6]. For all these models, we use the checkpoints or pre-trained weights
reported from their official websites222SD v1.5:
https://huggingface.co/runwayml/stable-diffusion-v1-5, IP2P: http://instruct-
pix2pix.eecs.berkeley.edu/instruct-pix2pix-00-22000.ckp, NI:
https://huggingface.co/CompVis/stable-diffusion-v1-4, ControlNet:
https://huggingface.co/lllyasviel/ControlNet/blob/main/models/control_sd15_normal.pth,
InstructDiffusion: https://github.com/cientgu/InstructDiffusion..

More specifically, for SD, the strength, guidance scale, and denoising steps
are set to $0.68$, $7.5$, and $50$, respectively. For IP2P, the images are
generated with $100$ denoising steps using a text guidance of $7.5$ and an
image guidance of $1.5$. For NI, each image is generated with $50$ denoising
steps and the guidance scale is $7.5$. The fraction of steps to replace the
self-attention maps is set in the range from $0.5$ to $0.8$ while the fraction
to replace the cross-attention maps is $0.8$. The amplification value for
words is $2$ or $5$, depending on the quality of the generation. For
ControlNet, the control strength, normal background threshold, denoising
steps, and guidance scale are $1$, $0.4$, $20$, and $9$, respectively. For
Instruct Diffusion, the denoising steps, text guidance, and image guidance are
set as $100$, $5.0$, and $1.25$, respectively.

####  A.2 Hyperparameters in LoRA Rank Search

During the process of searching LoRA rank, the rank $r_{i}$ for each crucial
layer $i$ is upscaled once for every $e$ epochs until $r_{i}$ reaches the
upper threshold $\tau_{i}$ for the layer $i$. In the experiments, $e$ is set
as $10$. The rank threshold $\tau_{i}$ is determined by the size of the layer.
More specifically, the crucial layers include: (1) four CONV-based upsampling
layers with the shape as ${[3,64,7,7],[64,128,3,3],[128,256,3,3]}$, and
${[256,256,3,3]}$; (2) four corresponding downsampling layers by transpose
CONV with the same set of weight shape as upsampling; and (3) transformer
blocks with projection matrices $q$,$k$,$v$ with shape as ${[256,256]}$, and
multi-layer perceptron (MLP) module with shape as ${[2048,256]}$ and
${[256,1024]}$. Based on the weight size, the rank threshold $\tau$ is set as
$1$, $4$, $16$, and $32$ for the four upsampling/downsampling layers,
respectively, and $1$ for the layers in the transformer block. After the
search process, the suitable rank is determined as $1$, $4$, $8$, $8$ for the
four upsampling/downsampling layers.

Table 6: FID comparison between E2GAN model architecture (3RB+1TB) and pix2pix
(9RB) under the setting of training-from-scratch.

Concept | E2GAN (3RB+1TB) | Pix2pix (9RB)
---|---|---
Angry person | 49.56 | 55.16
Pale person | 42.65 | 49.14
Tan person | 42.47 | 51.37
Young person | 51.27 | 56.10
|  |

###  Appendix B More Analysis for the Efficient Image-to-Image Model

####  B.1 Effectiveness of Model Architecture

Here we further show the effectiveness of our efficient model architecture
design in complementary to the results in Sec. 4.2.1. We compare our 3RB+1TB
design against the 9RB design used in pix2pix for several concept settings.
The results are shown in Tab. 6 with both models trained on the entire
training set of $800$ samples. From which we can see that the 3RB+1TB design
can reach higher FID with fewer parameters and FLOPs (as in Tab. 1). For
instance, a 3RB+1TB model in the target concept domain of pale person has a
FID as $42.65$, decreasing the FID value by $6.49$ compared to the 9RB model
of pix2pix.

####  B.2 Sampling Operations for Transformer Block

As mentioned in Sec. 4.2.1, we apply a downsampling operation with a CONV
layer to halve the feature map size before sending it into the transformer
block, and use an upsampling layer implemented by transpose CONV operation to
recover the feature map size for the following operations to reduce the amount
of computations. We conduct another set of experiments on the Flicker Scenery
dataset to see if these sampling operations can be replaced by pooling and
unpooling operations, such that a smaller model size can be reached. We first
train these two models on the selected prompts to get the base model. Then, we
fine-tune the entire model with all the training data for a new concept. The
comparision results are shown in Tab. 7. From the results, we can observe that
though applying pooling operations can reduce the number of parameters from
base model by $1.2$M, the FID performance becomes much worse. Thus, we use
CONV operation instead of pooling to tackle the feature map reduction and
recovery for the transformer block.

Table 7: FID performance of replacing the downsampling and upsampling layers
for the transformer block with Max Pool and Max Unpool operations.

Operation |  CONV + transpose CONV |  | Max Pool +
---
Max Unpool
Concept Model Size | 7.1M | 5.9M
Forest in the dark | 121.60 | 190.05
Impressionism painting | 88.52 | 135.96
Forest in the autumn | 88.82 | 141.29
|  |

###  Appendix C More Ablation Analysis for the Base Model

####  C.1 Pre-train with Multiple Concepts for Conventional GAN Training

We investigate if conventional GAN training such as pix2pix can benefit from
fine-tuning a pre-trained base model, as leveraged in E2GAN. To verify this,
we follow the same step as E2GAN to pre-train pix2pix with the selected $7$
prompts/datasets on the Flicker Scenery dataset. Then, the base model is fine-
tuned to adapt to other concepts. The results in Tab. 8 show that pix2pix does
not gain much benefits from pre-training. Moreover, the performance becomes
even worse, such as for the concept Vangogh style (FID degrades from $138.77$
to $151.20$ with a pre-trained base model). The results indicate that with our
efficient architecture design, our base model possess the capability of a more
general features and representations when trained on multiple concepts. The
transformer block with self-attention modifies the image with a better
holistic understanding and the cross-attention module takes the information
from the given target concept. Thus, our method allows the new concept to
better leverage existing knowledge, which is not possessed by prior methods.

Table 8: FID performance of fine-tuning from a pre-trained base model for
pix2pix on Flicker Scenery dataset.

Method | Pixpix | E2GAN
---|---|---
Concept Pretrain | ✓ | ✗ | ✓
Vangogh style | 151.2 | 138.77 | 117.41
Add blossoms | 157.76 | 150.96 | 146.42
Forest in the winter | 119.31 | 122.35 | 119.15
|  |  |

####  C.2 Autoencoder as Pre-trained Base Model

In E2GAN, we first train the GAN model with multiple diverse concepts to get a
pre-trained base model, and then fine-tune it to other concepts. We have shown
multiple base model settings in Sec. 5.2. One may wonder if the pre-trained
base model can be chosen as an auto-encoder, _e.g_. the base model encodes the
input data into a lower-dimensional representation and then decodes it back
into the original data, instead of being trained on other concepts. To verify
this, we conduct experiments by first training an auto-encoder on the original
images in the subset of FFHQ [15] with only the $\ell_{1}$ loss in Eq. 1, then
fine-tune the auto-encoder following the same method as fine-tuning a pre-
trained GAN. The results are compared in Tab. 9. We find that auto-encoder is
not comparable as fine-tuning a GAN trained on a single concept as old person,
not to mention our base model that is pre-trained on multiple concepts. For
instance, for the target style angry person, tuning from a base model pre-
trained to generate old person can give an FID as low as $54.48$, yet tuning
from the auto-encoder results in a much worse FID of $110.35$. This might due
to the simplicity of the auto-encoder, which only needs to generate the
original image and does not necessarily include other semantic information,
either coarse-grained global features, or fine-grained local details. In
constrast, the GAN models include more information like texture or color,
during training. From this observation, in E2GAN, we adopt a model pre-trained
on several concepts instead of using auto-encoder as base model.

Table 9: The FID performance of using autoencoder as the pre-trained base
model.

Base model Concept |  | Angry
---
person
| White
---
walker
Auto-encoder | 110.35 | 80.43
Old person | 54.48 | 51.99
Ours | 54.27 | 40.18
|  |

###  Appendix D Ablation on the Influence of Longer Training Time

Table 10: The FID comparison between training E2GAN for 100 epochs and 200
epochs.

Concept |  Train 100 epochs |  | Train 200
---
epochs
Forest in the dark | 115.32 | 114.17
Oil painting | 110.87 | 111.93
Forest in the spring | 122.77 | 124.91
|  |

E2GAN greatly saves the training time compared to conventional GAN training
while maintaining good image synthesize ability. To see if training longer can
lead to better performance, we add further experiments to increase the
training time by doubling the training epochs. The results can be found in
Tab. 10. The reported FID is evaluated on the model weights obtained at the
end of training. The results show that training longer will not bring obvious
performance improvements for E2GAN, but leads to more computation cost. The
results indicate that our efficient E2GAN is able to reach good performance
with fewer epochs compared to conventional GAN training.

###  Appendix E Additional Qualitative Results

We provide more example images generated by our approach and other baseline
methods in Fig. 8, 9, 10, 11, and 12.

Figure 8: Qualitative comparisons on various tasks. The _leftmost_ column
shows two original images and the remaining columns present the corresponding
synthesized images in the target concept domain, where target prompts are
shown at the bottom row. We provide images generated by various models. Figure
9: Qualitative comparisons on various tasks. The _leftmost_ column shows two
original images and the remaining columns present the corresponding
synthesized images in the target concept domain, where target prompts are
shown at the bottom row. We provide images generated by various models. Figure
10: Qualitative comparisons on various tasks. The _leftmost_ column shows two
original images and the remaining columns present the corresponding
synthesized images in the target concept domain, where target prompts are
shown at the bottom row. We provide images generated by various models. Figure
11: Qualitative comparisons on various tasks. The _leftmost_ column shows two
original images and the remaining columns present the corresponding
synthesized images in the target concept domain, where target prompts are
shown at the bottom row. We provide images generated by various models. Figure
12: Qualitative comparisons on various tasks. The _leftmost_ column shows two
original images and the remaining columns present the corresponding
synthesized images in the target concept domain, where target prompts are
shown at the bottom row. We provide images generated by various models.


* [167] Zachary D. Stephens, Skylar Y. Lee, Faraz Faghri, Roy H. Campbell, Chengxiang Zhai, Miles J. Efron, Ravishankar Iyer, Michael C. Schatz, Saurabh Sinha, and Gene E. Robinson.  Big Data: Astronomical or Genomical?  PLOS Biology, 2015.
  * [168] Erika Check Hayden.  Genome researchers raise alarm over big data.  Nature, 2015.
  * [169] Robert C. Edgar, Jeff Taylor, Victor Lin, Tomer Altman, Pierre Barbera, Dmitry Meleshko, Dan Lohr, Gherman Novakovsky, Benjamin Buchfink, Basem Al-Shayeb, Jillian F. Banfield, Marcos de la Peña, Anton Korobeynikov, Rayan Chikhi, and Artem Babaian.  Petabase-scale sequence alignment catalyses viral discovery.  Nature, 2022.
  * [170] Mantas Sereika, Rasmus Hansen Kirkegaard, Søren Michael Karst, Thomas Yssing Michaelsen, Emil Aarre Sørensen, Rasmus Dam Wollenberg, and Mads Albertsen.  Oxford Nanopore R10.4 long-read sequencing enables the generation of near-finished bacterial genomes from pure cultures and metagenomes without short-read or reference polishing.  Nature Methods, 2022.
  * [171] Yunhao Wang, Yue Zhao, Audrey Bollas, Yuru Wang, and Kin Fai Au.  Nanopore Sequencing Technology, Bioinformatics and Applications.  Nature Biotechnology, 2021.
  * [172] Leonard Schuele, Hayley Cassidy, Nilay Peker, John W. A. Rossen, and Natacha Couto.  Future potential of metagenomics in microbiology laboratories.  Expert Review of Molecular Diagnostics, 2021.
  * [173] Illumina.  NovaSeq X Series Specifications.  https://emea.illumina.com/systems/sequencing-platforms/novaseq-x-plus/specifications.html, 2023.
  * [174] Shadi Shokralla, Teresita M. Porter, Joel F. Gibson, Rafal Dobosz, Daniel H. Janzen, Winnie Hallwachs, G. Brian Golding, and Mehrdad Hajibabaei.  Massively parallel multiplex DNA sequencing for specimen identification using an Illumina MiSeq platform.  Scientific Reports, 2015.
  * [175] Haowen Zhang, Haoran Li, Chirag Jain, Haoyu Cheng, Kin Fai Au, Heng Li, and Srinivas Aluru.  Real-time Mapping of Nanopore Raw Signals.  Bioinformatics, 2021.
  * [176] Can Firtina, Nika Mansouri Ghiasi, Joel Lindegger, Gagandeep Singh, Meryem Banu Cavlak, Haiyu Mao, and Onur Mutlu.  RawHash: enabling fast and accurate real-time analysis of raw nanopore signals for large genomes.  Bioinformatics, 2023.
  * [177] Sam Kovaka, Yunfan Fan, Bohan Ni, Winston Timp, and Michael C Schatz.  Targeted Nanopore Sequencing by Real-time Mapping of Raw Electrical Signal with UNCALLED.  Nature Biotechnology, 2020.
  * [178] Onur Mutlu and Can Firtina.  Accelerating Genome Analysis via Algorithm-Architecture Co-Design.  In DAC, 2023.
  * [179] Alexander Payne, Nadine Holmes, Thomas Clarke, Rory Munro, Bisrat J. Debebe, and Matthew Loose.  Readfish enables targeted nanopore sequencing of gigabase-sized genomes.  Nature Biotechnology, 2021.
  * [180] Yuwei Bao, Jack Wadden, John R. Erb-Downward, Piyush Ranjan, Weichen Zhou, Torrin L. McDonald, Ryan E. Mills, Alan P. Boyle, Robert P. Dickson, David Blaauw, and Joshua D. Welch.  Squigglenet: real-time, direct classification of nanopore signals.  Genome Biology, 2021.
  * [181] Jens-Uwe Ulrich, Ahmad Lutfi, Kilian Rutzen, and Bernhard Y Renard.  ReadBouncer: precise and scalable adaptive sampling for nanopore sequencing.  Bioinformatics, 2022.
  * [182] Illumina.  NovaSeq 6000 System Specifications.  https://emea.illumina.com/systems/sequencing-platforms/novaseq/specifications.html, 2020.
  * [183] Miten Jain, Hugh E. Olsen, Benedict Paten, and Mark Akeson.  The Oxford Nanopore MinION: delivery of nanopore sequencing to the genomics community.  Genome Biology, 2016.
  * [184] Aaron Pomerantz, Nicolás Peñafiel, Alejandro Arteaga, Lucas Bustamante, Frank Pichardo, Luis A Coloma, César L Barrio-Amorós, David Salazar-Valenzuela, and Stefan Prost.  Real-time DNA Barcoding in a Rainforest Using Nanopore Sequencing: Opportunities for Rapid Biodiversity Assessments and Local Capacity Building.  GigaScience, 2018.
  * [185] Eric W Sayers, Richa Agarwala, Evan E Bolton, J Rodney Brister, Kathi Canese, Karen Clark, Ryan Connor, Nicolas Fiorini, Kathryn Funk, Timothy Hefferon, J Bradley Holmes, Sunghwan Kim, Avi Kimchi, Paul A Kitts, Stacy Lathrop, Zhiyong Lu, Thomas L Madden, Aron Marchler-Bauer, Lon Phan, Valerie A Schneider, Conrad L Schoch, Kim D Pruitt, and James Ostell.  Database resources of the National Center for Biotechnology Information.  Nucleic Acids Research, 2018.
  * [186] AMD.  AMD${}^{\text{\textregistered}}$ EPYC${}^{\text{\textregistered}}$ 7742 CPU.  https://www.amd.com/en/products/cpu/amd-epyc-7742.
  * [187] Micron Technology Inc.  4Gb: x4, x8, x16 DDR4 SDRAM Data Sheet, 2016.
  * [188] Serial ATA International Organization.  SATA revision 3.0 specifications.  https://www.sata-io.org.
  * [189] PCI-SIG.  PCI Express Base Specification Revision 4.0, Version 1.0.  https://pcisig.com/specifications.
  * [190] Samsung PM1735.  https://www.digitec.ch/en/s1/product/samsung-pm1735-3200-gb-pci-express-ssd-15678607.
  * [191] Samsung PM9A3.  https://www.digitec.ch/en/s1/product/samsung-pm9a3-3840-gb-m2-22110-ssd-16404342.
  * [192] Samsung 870 EVO.  https://www.digitec.ch/en/s1/product/samsung-870-evo-4000-gb-25-ssd-14599189.
  * [193] Daehwan Kim, Li Song, Florian P Breitwieser, and Steven L Salzberg.  Centrifuge.  http://www.ccb.jhu.edu/software/centrifuge/, 2020.
  * [194] Téo Lemane, Paul Medvedev, Rayan Chikhi, and Pierre Peterlongo.  kmtricks: efficient and flexible construction of Bloom filters for large sequencing data collections.  Bioinformatics Advances, 2022.
  * [195] Jarno N Alanko, Jaakko Vuohtoniemi, Tommi Mäklin, and Simon J Puglisi.  Themisto: a scalable colored k-mer index for sensitive pseudoalignment against hundreds of thousands of bacterial genomes.  Bioinformatics, 2023.
  * [196] Jason Fan, Noor Pratap Singh, Jamshed Khan, Giulio Ermanno Pibiri, and Rob Patro.  Fulgor: A Fast and Compact k-mer Index for Large-Scale Matching and Color Queries.  In WABI, 2023.
  * [197] Zhuowen Zou, Hanning Chen, Prathyush Poduval, Yeseong Kim, Mahdi Imani, Elaheh Sadredini, Rosario Cammarota, and Mohsen Imani.  BioHD: an efficient genome sequence search platform using hyperdimensional memorization.  In ISCA, 2022.
  * [198] Onur Mutlu, Saugata Ghose, Juan Gómez-Luna, and Rachata Ausavarungnirun.  A Modern Primer on Processing in Memory.  In Emerging Computing: From Devices to Systems: Looking Beyond Moore and Von Neumann. 2022.
  * [199] Saugata Ghose, Amirali Boroumand, Jeremie S Kim, Juan Gómez-Luna, and Onur Mutlu.  Processing-in-Memory: A Workload-Driven Perspective.  IBM Journal of Research and Development, 2019.
  * [200] Onur Mutlu, Saugata Ghose, Juan Gómez-Luna, and Rachata Ausavarungnirun.  Processing Data Where It Makes Sense: Enabling In-Memory Computation.  Microprocessors and Microsystems, 2019.
  * [201] Saugata Ghose, Kevin Hsieh, Amirali Boroumand, Rachata Ausavarungnirun, and Onur Mutlu.  Enabling the Adoption of Processing-in-memory: Challenges, Mechanisms, Future Research Directions.  arXiv, 2018.
  * [202] Marek Kokot, Maciej Długosz, and Sebastian Deorowicz.  KMC 3: counting and manipulating k-mer statistics.  Bioinformatics, 2017.
  * [203] Nikola Samardzic, Weikang Qiao, Vaibhav Aggarwal, Mau-Chung Frank Chang, and Jason Cong.  Bonsai: High-performance adaptive merge tree sorting.  In ISCA, 2020.
  * [204] Weikang Qiao, Licheng Guo, Zhenman Fang, Mau-Chung Frank Chang, and Jason Cong.  TopSort: A High-Performance Two-Phase Sorting Accelerator Optimized on HBM-based FPGAs.  In FCCM, 2022.
  * [205] Soundarya Jayaraman, Bingyi Zhang, and Viktor Prasanna.  Hypersort: High-performance Parallel Sorting on HBM-enabled FPGA.  In ICFPT, 2022.
  * [206] Gaëtan Benoit, Pierre Peterlongo, Mahendra Mariadassou, Erwan Drezen, Sophie Schbath, Dominique Lavenier, and Claire Lemaitre.  Multiple comparative metagenomics using multiset k-mer counting.  PeerJ Computer Science, 2016.
  * [207] Roderick Bovee and Nick Greenfield.  Finch: a tool adding dynamic abundance filtering to genomic minhashing.  The Journal of Open Source Software, 2018.
  * [208] Samsung.  Samsung SSD 980 PRO.  https://www.samsung.com/semiconductor/minisite/ssd/product/consumer/980pro/, 2020.
  * [209] Shaopeng Liu and David Koslicki.  CMash: fast, multi-resolution estimation of k-mer-based Jaccard and containment indices.  Bioinformatics, 2022.
  * [210] Camille Marchet, Christina Boucher, Simon J. Puglisi, Paul Medvedev, Mikaël Salson, and Rayan Chikhi.  Data structures based on k-mers for querying large collections of sequencing data sets.  Genome Research, 2021.
  * [211] Silvio Weging, Andreas Gogol-Döring, and Ivo Grosse.  Taxonomic analysis of metagenomic data with kASA.  Nucleic Acids Research, 2021.
  * [212] Anirban Nag, CN Ramachandra, Rajeev Balasubramonian, Ryan Stutsman, Edouard Giacomin, Hari Kambalasubramanyam, and Pierre-Emmanuel Gaillardon.  GenCache: Leveraging In-cache Operators for Efficient Sequence Alignment.  In MICRO, 2019.
  * [213] Daichi Fujiki, Arun Subramaniyan, Tianjun Zhang, Yu Zeng, Reetuparna Das, David Blaauw, and Satish Narayanasamy.  Genax: A Genome Sequencing Accelerator.  In ISCA, 2018.
  * [214] Myungsuk Kim, Jaehoon Lee, Sungjin Lee, Jisung Park, Youngsun Song, and Jihong Kim.  Improving Performance and Lifetime of Large-page NAND Storages Using Erase-free Subpage Programming.  In DAC, 2017.
  * [215] Atsuo Kawaguchi, Shingo Nishioka, and Hiroshi Motoda.  A flash-memory based file system.  In ATC, 1995.
  * [216] Micron.  Product Flyer: Micron 3D NAND Flash Memory.  https://www.micron.com/-/media/client/global/documents/products/product-flyer/3d_nand_flyer.pdf?la=en, 2016.
  * [217] David Danko, Daniela Bezdan, Evan E Afshin, Sofia Ahsanuddin, Chandrima Bhattacharya, Daniel J Butler, Kern Rei Chng, Daisy Donnellan, Jochen Hecht, Katelyn Jackson, et al.  A Global Metagenomic Map of Urban Microbiomes and Antimicrobial Resistance.  Cell, 2021.
  * [218] Peter J. Turnbaugh, Ruth E. Ley, Micah Hamady, Claire M. Fraser-Liggett, Rob Knight, and Jeffrey I. Gordon.  The human microbiome project.  Nature, 2007.
  * [219] Synopsys, Inc.  Design Compiler.  https://www.synopsys.com/implementation-and-signoff/rtl-synthesis-test/design-compiler-graphical.html.
  * [220] United Microelectronics Corporation.  UMK65LSCLLMVBBL_A - UMC 65 nm Low-K 1.2V/1.0V Low Leakage LVT Tapless Standard Cell Library, version A02, 2008.
  * [221] Cadence Design Systems, Inc.  Innovus Implementation System.  https://www.cadence.com/en_US/home/tools/digital-design-and-signoff/soc-implementation-and-floorplanning/innovus-implementation-system.html.
  * [222] Yoongu Kim, Weikun Yang, and Onur Mutlu.  Ramulator: A Fast and Extensible DRAM Simulator.  CAL, 2015.
  * [223] Yoongu Kim, Weikun Yang, and Onur Mutlu.  Ramulator Source Code.  https://github.com/CMU-SAFARI/ramulator.
  * [224] Arash Tavakkol, Juan Gómez-Luna, Mohammad Sadrosadati, Saugata Ghose, and Onur Mutlu.  MQSim: A Framework for Enabling Realistic Studies of Modern Multi-queue SSD Devices.  In FAST, 2018.
  * [225] Arash Tavakkol, Juan Gómez-Luna, Mohammad Sadrosadati, Saugata Ghose, and Onur Mutlu.  MQSim Source Code.  https://github.com/CMU-SAFARI/MQSim.
  * [226] Samsung.  LPDDR4.  https://semiconductor.samsung.com/dram/lpddr/lpddr4/.
  * [227] Saugata Ghose, Tianshi Li, Nastaran Hajinazar, Damla Senol Cali, and Onur Mutlu.  Demystifying Complex Workload-DRAM Interactions: An Experimental Study.  POMACS, 2019.
  * [228] Advanced Micro Devices.  AMD® $\mu$Prof.  https://developer.amd.com/amd-uprof/, 2021.
  * [229] Alexander Sczyrba, Peter Hofmann, Peter Belmann, David Koslicki, Stefan Janssen, Johannes Dröge, Ivan Gregor, Stephan Majda, Jessika Fiedler, Eik Dahms, Andreas Bremges, Adrian Fritz, Ruben Garrido-Oter, Tue Sparholt Jørgensen, Nicole Shapiro, Philip D. Blood, Alexey Gurevich, Yang Bai, Dmitrij Turaev, Matthew Z. DeMaere, Rayan Chikhi, Niranjan Nagarajan, Christopher Quince, Fernando Meyer, Monika Balvočiūtė, Lars Hestbjerg Hansen, Søren J. Sørensen, Burton K. H. Chia, Bertrand Denis, Jeff L. Froula, Zhong Wang, Robert Egan, Dongwan Don Kang, Jeffrey J. Cook, Charles Deltel, Michael Beckstette, Claire Lemaitre, Pierre Peterlongo, Guillaume Rizk, Dominique Lavenier, Yu-Wei Wu, Steven W. Singer, Chirag Jain, Marc Strous, Heiner Klingenberg, Peter Meinicke, Michael D. Barton, Thomas Lingner, Hsin-Hung Lin, Yu-Chieh Liao, Genivaldo Gueiros Z. Silva, Daniel A. Cuevas, Robert A. Edwards, Surya Saha, Vitor C. Piro, Bernhard Y. Renard, Mihai Pop, Hans-Peter Klenk, Markus Göker, Nikos C. Kyrpides, Tanja Woyke, Julia A. Vorholt, Paul Schulze-Lefert, Edward M. Rubin, Aaron E. Darling, Thomas Rattei, and Alice C. McHardy.  Critical Assessment of Metagenome Interpretation—A Benchmark of Metagenomics Software.  Nature Methods, 2017.
  * [230] Samsung.  Samsung 8 GB DRAM DDR4 8GB PC3200 UB 1Rx16 Samsung.  https://semiconductor.samsung.com/dram/module/udimm/m378a1g44ab0-cwe/.
  * [231] Samsung.  Samsung 128 GB DDR4 3200 LRDIMM ECC Registred.  https://semiconductor.samsung.com/dram/module/lrdimm/m386aag40am3-cwe/.
  * [232] Oxford Nanopore Technologies.  MinION Mk1B IT Requirements.  https://community.nanoporetech.com/requirements_documents/minion-it-reqs.pdf, 2021.
  * [233] Damla Senol Cali, Jeremie S Kim, Saugata Ghose, Can Alkan, and Onur Mutlu.  Nanopore Sequencing Technology and Tools for Genome Assembly: Computational Analysis of the Current State, Bottlenecks and Future Directions.  Briefings in Bioinformatics, 2018.
  * [234] Aaron Stillmaker and Bevan Baas.  Scaling Equations for the Accurate Prediction of CMOS Device Performance from 180 Nm to 7 Nm.  Integration, 2017.
  * [235] Xiaoquan Su, Jian Xu, and Kang Ning.  Parallel-meta: efficient metagenomic data analysis based on high-performance computation.  BMC Systems Biology, 2012.
  * [236] Xiaoquan Su, Xuetao Wang, Gongchao Jing, and Kang Ning.  GPU-Meta-Storms: computing the structure similarities among massive amount of microbial community samples using GPU.  Bioinformatics, 2013.
  * [237] Masahiro Yano, Hiroshi Mori, Yutaka Akiyama, Takuji Yamada, and Ken Kurokawa.  Clast: Cuda implemented large-scale alignment search tool.  BMC Bioinformatics, 2014.
  * [238] Antonio Saavedra, Hans Lehnert, Cecilia Hernández, Gonzalo Carvajal, and Miguel Figueroa.  Mining discriminative k-mers in DNA sequences using sketches and hardware acceleration.  IEEE Access, 2020.
  * [239] Tianqi Zhang, Antonio González, Niema Moshiri, Rob Knight, and Tajana Rosing.  GenoMiX: Accelerated Simultaneous Analysis of Human Genomics, Microbiome Metagenomics, and Viral Sequences.  In BioCAS, 2023.
  * [240] Gustavo Henrique Cervi, Cecília Dias Flores, and Claudia Elizabeth Thompson.  Metagenomic Analysis: A Pathway Toward Efficiency Using High-Performance Computing.  In ICICT, 2022.
  * [241] Tim Dunn, Harisankar Sadasivan, Jack Wadden, Kush Goliya, Kuan-Yu Chen, David Blaauw, Reetuparna Das, and Satish Narayanasamy.  SquiggleFilter: An Accelerator for Portable Virus Detection.  In MICRO, 2021.
  * [242] Po Jui Shih, Hassaan Saadat, Sri Parameswaran, and Hasindu Gamaarachchi.  Efficient real-time selective genome sequencing on resource-constrained devices.  GigaScience, 2023.
  * [243] Omar Ahmed, Massimiliano Rossi, Sam Kovaka, Michael C. Schatz, Travis Gagie, Christina Boucher, and Ben Langmead.  Pan-genomic matching statistics for targeted nanopore sequencing.  iScience, 2021.
  * [244] Wenqin Huangfu, Shuangchen Li, Xing Hu, and Yuan Xie.  RADAR: A 3D-ReRAM based DNA Alignment Accelerator Architecture.  In DAC, 2018.
  * [245] S Karen Khatamifard, Zamshed Chowdhury, Nakul Pande, Meisam Razaviyayn, Chris H Kim, and Ulya R Karpuzcu.  GeNVoM: Read Mapping Near Non-Volatile Memory.  TCBB, 2021.
  * [246] Xue-Qi Li, Guang-Ming Tan, and Ning-Hui Sun.  PIM-Align: A Processing-in-Memory Architecture for FM-Index Search Algorithm.  Journal of Computer Science and Technology, 2021.
  * [247] Shaahin Angizi, Jiao Sun, Wei Zhang, and Deliang Fan.  Aligns: A Processing-in-memory Accelerator for DNA Short Read Alignment Leveraging SOT-MRAM.  In DAC, 2019.
  * [248] Farzaneh Zokaee, Hamid R Zarandi, and Lei Jiang.  Aligner: A Process-in-memory Architecture for Short Read Alignment in ReRAMs.  IEEE Computer Architecture Letters, 2018.
  * [249] Advait Madhavan, Timothy Sherwood, and Dmitri Strukov.  Race Logic: A Hardware Acceleration for Dynamic Programming Algorithms.  SIGARCH Computer Architecture News, 2014.
  * [250] Haoyu Cheng, Yong Zhang, and Yun Xu.  Bitmapper2: A GPU-accelerated All-mapper Based on The Sparse Q-gram Index.  TCBB, 2018.
  * [251] Ernst Joachim Houtgast, Vlad-Mihai Sima, Koen Bertels, and Zaid Al-Ars.  Hardware Acceleration of BWA-MEM Genomic Short Read Mapping for Longer Read Lengths.  Computational Biology and Chemistry, 2018.
  * [252] Ernst Joachim Houtgast, VladMihai Sima, Koen Bertels, and Zaid AlArs.  An Efficient GPU-accelerated Implementation of Genomic Short Read Mapping with BWA-MEM.  SIGARCH Computer Architecture News, 2017.
  * [253] Alberto Zeni, Giulia Guidi, Marquita Ellis, Nan Ding, Marco D Santambrogio, Steven Hofmeyr, Aydın Buluç, Leonid Oliker, and Katherine Yelick.  Logan: High-performance GPU-based X-drop Long-read Alignment.  In IPDPS, 2020.
  * [254] Nauman Ahmed, Jonathan Lévy, Shanshan Ren, Hamid Mushtaq, Koen Bertels, and Zaid Al-Ars.  GASAL2: A GPU Accelerated Sequence Alignment Library for High-Throughput NGS Data.  BMC Bioinformatics, 2019.
  * [255] Takahiro Nishimura, Jacir L Bordim, Yasuaki Ito, and Koji Nakano.  Accelerating the Smith-waterman Algorithm Using Bitwise Parallel Bulk Computation Technique on GPU.  In IPDPSW, 2017.
  * [256] Edans Flavius de Oliveira Sandes, Guillermo Miranda, Xavier Martorell, Eduard Ayguade, George Teodoro, and Alba Cristina Magalhaes Melo.  CUDAlign 4.0: Incremental Speculative Traceback for Exact Chromosome-wide Alignment in GPU Clusters.  TPDS, 2016.
  * [257] Yongchao Liu and Bertil Schmidt.  GSWABE: Faster GPU-accelerated Sequence Alignment with Optimal Alignment Retrieval for Short DNA Sequences.  Concurrency and Computation: Practice and Experience, 2015.
  * [258] Yongchao Liu, Adrianto Wirawan, and Bertil Schmidt.  CUDASW++ 3.0: Accelerating Smith-Waterman Protein Database Search by Coupling CPU and GPU SIMD Instructions.  BMC Bioinformatics, 2013.
  * [259] Yongchao Liu, Douglas L Maskell, and Bertil Schmidt.  CUDASW++: Optimizing Smith-Waterman Sequence Database Searches for CUDA-enabled Graphics Processing Units.  BMC Research Notes, 2009.
  * [260] Yongchao Liu, Bertil Schmidt, and Douglas L Maskell.  CUDASW++ 2.0: Enhanced Smith-Waterman Protein Database Search on CUDA-enabled GPUs Based on SIMT and Virtualized SIMD Abstractions.  BMC Research Notes, 2010.
  * [261] Richard Wilton, Tamas Budavari, Ben Langmead, Sarah J Wheelan, Steven L Salzberg, and Alexander S Szalay.  Arioc: High-throughput Read Alignment with GPU-accelerated Exploration of The Seed-and-extend Search Space.  PeerJ, 2015.
  * [262] Amit Goyal, Hyuk Jung Kwon, Kichan Lee, Reena Garg, Seon Young Yun, Yoon Hee Kim, Sunghoon Lee, and Min Seob Lee.  Ultra-fast Next Generation Human Genome Sequencing Data Processing Using DRAGENTM Bio-IT Processor for Precision Medicine.  Open Journal of Genetics, 2017.
  * [263] Yu-Ting Chen, Jason Cong, Zhenman Fang, Jie Lei, and Peng Wei.  When Spark Meets FPGAs: A Case Study for Next-Generation DNA Sequencing Acceleration.  In HotCloud, 2016.
  * [264] Peng Chen, Chao Wang, Xi Li, and Xuehai Zhou.  Accelerating the Next Generation Long Read Mapping with the FPGA-based System.  TCBB, 2014.
  * [265] Yen-Lung Chen, Bo-Yi Chang, Chia-Hsiang Yang, and Tzi-Dar Chiueh.  A High-Throughput FPGA Accelerator for Short-Read Mapping of the Whole Human Genome.  TPDS, 2021.
  * [266] Daichi Fujiki, Shunhao Wu, Nathan Ozog, Kush Goliya, David Blaauw, Satish Narayanasamy, and Reetuparna Das.  SeedEx: A Genome Sequencing Accelerator for Optimal Alignments in Subminimal Space.  In MICRO, 2020.
  * [267] Subho Sankar Banerjee, Mohamed El-Hadedy, Jong Bin Lim, Zbigniew T Kalbarczyk, Deming Chen, Steven S Lumetta, and Ravishankar K Iyer.  ASAP: Accelerated Short-read Alignment on Programmable Hardware.  TC, 2019.
  * [268] Xia Fei, Zou Dan, Lu Lina, Man Xin, and Zhang Chunlei.  FPGASW: Accelerating Large-scale Smith–Waterman Sequence Alignment Application with Backtracking on FPGA Linear Systolic Array.  Interdisciplinary Sciences: Computational Life Sciences, 2018.
  * [269] Hasitha Muthumala Waidyasooriya and Masanori Hariyama.  Hardware-acceleration of Short-read Alignment Based on the Burrows-wheeler Transform.  TPDS, 2015.
  * [270] Yu-Ting Chen, Jason Cong, Jie Lei, and Peng Wei.  A Novel High-throughput Acceleration Engine for Read Alignment.  In FCCM, 2015.
  * [271] Enzo Rucci, Carlos Garcia, Guillermo Botella, Armando De Giusti, Marcelo Naiouf, and Manuel Prieto-Matias.  SWIFOLD: Smith-Waterman Implementation on FPGA with OpenCL for Long DNA Sequences.  BMC Systems Biology, 2018.
  * [272] Abbas Haghi, Santiago Marco-Sola, Lluc Alvarez, Dionysios Diamantopoulos, Christoph Hagleitner, and Miquel Moreto.  An FPGA Accelerator of the Wavefront Algorithm for Genomics Pairwise Alignment.  FPL, 2021.
  * [273] Luyi Li, Jun Lin, and Zhongfeng Wang.  PipeBSW: A Two-Stage Pipeline Structure for Banded Smith-Waterman Algorithm on FPGA.  ISVLSI, 2021.
  * [274] Tae Jun Ham, Yejin Lee, Seong Hoon Seo, U Gyeong Song, Jae W Lee, David Bruns-Smith, Brendan Sweeney, Krste Asanovic, Young H Oh, and Lisa Wu Wills.  Accelerating Genomic Data Analytics With Composable Hardware Acceleration Framework.  IEEE Micro, 2021.
  * [275] Lisa Wu, David Bruns-Smith, Frank A. Nothaft, Qijing Huang, Sagar Karandikar, Johnny Le, Andrew Lin, Howard Mao, Brendan Sweeney, Krste Asanović, David A. Patterson, and Anthony D. Joseph.  FPGA Accelerated Indel Realignment in the Cloud.  In HPCA, 2019.
  * [276] Ryan Marcus, Andreas Kipf, Alexander van Renen, Mihail Stoian, Sanchit Misra, Alfons Kemper, Thomas Neumann, and Tim Kraska.  Benchmarking learned indexes.  Proc. VLDB Endow., 2020.
  * [277] Arun Subramaniyan, Jack Wadden, Kush Goliya, Nathan Ozog, Xiao Wu, Satish Narayanasamy, David Blaauw, and Reetuparna Das.  Accelerated seeding for genome sequence alignment with enumerated radix trees.  In ISCA, 2021.
  * [278] Shengwen Liang, Ying Wang, Cheng Liu, Huawei Li, and Xiaowei Li.  InS-DLA: An In-SSD deep learning accelerator for near-data processing.  In FPL, 2019.
  * [279] Lingxi Wu, Minxuan Zhou, Weihong Xu, Ashish Venkat, Tajana Rosing, and Kevin Skadron.  Abakus: Accelerating k-mer Counting With Storage Technology.  TACO, 2023.