# Automatic Configuration for Optimal Communication Scheduling in DNN Training

Yiqing Ma1, Hao Wang1, Yiming Zhang2, Kai Chen1
1iSING Lab, Hong Kong University of Science and Technology
2National University of Defense Technology

###### Abstract

ByteScheduler partitions and rearranges tensor transmissions to improve the
communication efficiency of distributed Deep Neural Network (DNN) training.
The configuration of hyper-parameters (i.e., the partition size and the credit
size) is critical to the effectiveness of partitioning and rearrangement.
Currently, ByteScheduler adopts Bayesian Optimization (BO) to find the optimal
configuration for the hyper-parameters beforehand. In practice, however,
various runtime factors (e.g., worker node status and network conditions)
change over time, making the statically-determined one-shot configuration
result suboptimal for real-world DNN training.

To address this problem, we present a real-time configuration method (called
AutoByte) that automatically and timely searches the optimal hyper-parameters
as the training systems dynamically change. AutoByte extends the ByteScheduler
framework with a meta-network, which takes the system’s runtime statistics as
its input and outputs predictions for speedups under specific configurations.
Evaluation results on various DNN models show that AutoByte can dynamically
tune the hyper-parameters with low resource usage, and deliver up to 33.2%
higher performance than the best static configuration in ByteScheduler.

###### Index Terms:

Distributed training, communication scheduling, meta network.

##  I Introduction

Deep learning (DL) has been widely adopted for data mining and analytics. DL
drives rapid development of emerging applications such as face recognition
[20] and natural language processing (NLP) [12]. To support these
applications, the datasets become larger and larger and the models become more
and more complex [16], making DL training increasingly time-consuming. As a
result, large-scale training is often conducted in a distributed manner. In
distributed training, communication is usually the bottleneck [36]. Many
studies have been done to improve the communication performance for
distributed training, such as Parameter Server (PS) [33], All-reduce [38], and
Gradient Compression [35].

Most recently, communication scheduling [26, 19, 39] is proposed to further
improve the communication efficiency of distributed training. The key idea of
communication scheduling is to change the transmission order of different DNN
layers, in order to better parallelize the communication and computing tasks.
The state-of-the-art ByteScheduler [39] provides a generic communication
scheduling service for different DL frameworks (such as MXNet [10], PyTorch
[37] and Tensorflow [1]), which adopts tensor partitioning and priority-based
communication scheduling to realize pipelined communication and computation of
distributed training.

Based on experimental analysis, we observe that the configuration of hyper-
parameters (i.e., the partition size and the credit size) is critical to
ByteScheduler and inappropriate hyper-parameters may significantly degrade the
effectiveness of partitioning and rearrangement. Currently, ByteScheduler
adopts Bayesian Optimization (BO) [6] to accelerate the searching process by
finding a near optimal configuration of the hyper-parameters beforehand. BO
assumes the optimal values of the hyper-parameters stay constant throughout
the entire training procedure. In practice, however, various runtime factors,
such as worker node status and network conditions, change over time, making
the statically-determined one-shot configuration result suboptimal for dynamic
training environments.

To address this problem, in this paper we present a real-time configuration
method (called AutoByte) that automatically and timely searches the optimal
hyper-parameters as the training systems dynamically change. AutoByte extends
the ByteScheduler framework with a meta-network, which takes the system’s
runtime statistics as the model input and outputs predictions for speedups
under specific configurations. The runtime statistics include static
parameters (such as the numbers of workers and model types) and dynamic
parameters (such as layer-wise computation time and transmission rate which
characterize the dynamic computation and communication conditions). AutoByte
integrates offline training with online adaptation for its meta-network to
better fit the practical training scenarios. AutoByte executes
reconfigurations for the dynamically-generated optimal hyper-parameters based
on an optimization trigger mechanism, which can timely adjust the hyper-
parameters and always achieve high training performance as runtime factors
change.

Our evaluation on various DNN models shows that AutoByte can dynamically tune
the hyper-parameters to effectively improve the training performance. We show
that the configuration automatically generated by AutoByte performs very close
to the real optimal configuration found by Grid Search [39], and that when
multiple training jobs share the network bandwidth (as in real cloud
environments), AutoByte can quickly find the optimal configuration under
varying network bandwidth and improve the training performance by up to
19.2$\%$. We further evaluate AutoByte’s effectiveness for different bandwidth
settings and communication architectures (PS and All-reduce), and the results
show that AutoByte can outperform the state-of-the-art communication
scheduling algorithms by up to 33.2 $\%$ at low resource usage cost.

The rest of this paper is organized as follows. In $\S$2, we introduce the
necessary background of communication scheduling in distributed training. We
illustrate the downgrading caused by inappropriate partition size and credit
size in ByteScheduler with both diagrams and experiments. Meanwhile, we show
that the original hyper-parameters tuning technique Bayesian Optimization (BO)
in ByteScheduler fails to adapt with the dynamic factors changing, e.g.,
available bandwidth and computation resource, which incurs suboptimal
decisions. In $\S$III, we first explore the dynamic factors which impact the
choosing of optimal credit size and partition size. Then we introduce the
design of AutoByte, at its core, AutoByte uses a meta network optimizer to
automatically exploit the relation of input metrics and training speed. To
mitigate the extra overhead, AutoByte leverages an offline training and online
adapting strategy. In $\S$IV, we evaluate the speedup of AutoByte compare to
vanilla ML framework and ByteSheduler on three different models. We also show
the rapid adaptation and low overhead of AutoByte.

##  II Background & Motivation

(a) Computation and communication in distributed deep learning architecture

(b) A contrived example showing performance gain with tensor partitioning

(c) tensor partitioning with inappropriate partition size will cause
performance deficiency

Figure 1: Tensor partitioning. An appropriate partition size brings
significant performance gains.

(a) Credit-based priority scheduling allow higher-priority tensors to jump
ahead of the queue

(b) An appropriate credit size can fully utilize the network bandwidth and
deliver positive gains

(c) Inappropriate credit size brings negative effect

Figure 2: Credit-based priority scheduling. An appropriate credit size also
brings considerable performance gains.

In this section, we introduce the background of distributed training. We
describe the necessity of distributed training, the basic workflow, the common
distributed training architectures and classical communication scheduling
strategies.

###  II-A Distributed Deep Learning

For a deep learning task, a DNN model can be trained through an iterative way.
The iterations are repeated over the large training dataset until the loss
function is minimized and the model converges to an acceptable prediction
accuracy [11].

Forward and backward propagation. When dealing with the large training
dataset, the dataset is firstly partitioned into many mini-batches [17]. Then
in each iteration, one mini-batch travels through the model layer-by-layer and
generates a loss. This is the process of forward propagation (FP). After that,
the gradients are pushed through the last layer to the first layer. This is
the process of backward propagation (BP). Lastly, the calculated gradients are
used to update the model parameters by using Stochastic Gradient Descent (SGD)
[11]. Then the model can start to deal with the next mini-batch similarly.
Figure 1(a) shows the whole process.

Data parallelism.  Due to the high degree of computational complexity in
processing large amount of data and tuning billions of parameters, training
such a DNN model is tremendously time consuming [17]. These days distributed
training is all the rage. Especially, data parallel distribution is a popular
method for accelerating the training. The strategy of data parallelism is
partitioning the dataset into multiple devices. Each device has the same model
parameters but with different partitions of the dataset. In each iteration,
the calculated gradients are aggregated to update the model parameters, and
then broadcasted to all devices.

###  II-B Communication Architecture

In the parameter update step, devices need to share their gradients, which
involves network communication issues. Two kinds of communication
architectures are widely used. They are Parameters Server [33] and all-reduce
[38].

Parameter Server (PS). The PS architecture leverage parameter servers to
collect and aggregate the gradients. Each worker computes the gradients
locally and sends them to the servers through push step. Each server sums the
gradients from different workers and update the parameters. Each worker then
fetches the parameters from the servers through pull step. This architecture
fully utilizes the computation power and is able to guarantee the
synchronization of DNN training. A PS architecture also enables fault
tolerance [33] .

All-reduce. Unlike PS, All-reduce does not require additional servers for the
update of parameters. Instead, every worker collects the gradients from
others’ and update their parameters locally. One of the most popular all
reduce methods is Ring All-reduce [38]. A logical ring is formed among all
workers. Every worker receives gradients from its left neighbor worker and
sends the collected gradients to its right neighbor worker. Then they adds the
gradients to its own copy until all gradients are aggregated. Then they
broadcast the gradients along the ring. Ring all-reduce architecture can
alleviate the bottleneck. It is also bandwidth optimal.

###  II-C Communication Scheduling

The layer-wise structure of deep learning training makes it convenient to
parallel the communication tasks and computing tasks. The communication
scheduling strategies mainly targets at minimizing the network communication
time [32, 36, 50, 26, 19, 39, 28, 29, 46, 47, 48]. Poseidon [50] supports
overlapping communication process with backward propagation, reducing bursty
network communication. P3 [26] attempts to overlap communication with forward
propagation by layer partitioning and scheduling on MXNet PS architecture.
TicTac [19] proposes a similar idea that reduces the iteration time by
calculating the order of transmission. It guarantees near-optimal overlap of
communication and computation. It is only implemented on TensorFlow PS.
ByteScheduler [39] provides a generic communication scheduler for distributed
DNN training. It supports TensorFlow, PyTorch, and MXNet on both Parameter
Server and all-reduce architectures. PipeDream [36] also exploits the
scheduling on the model parallelism. it proposes to aggressively pipeline
mini-batch processing to maximize the utilization of computing resources.
However, its coarse-grained partitioning potentially misses optimization
opportunities and may fail to scale extremely large layers. TensorExpress [29]
schedules tensor packets in-network using P4 to mitigate network contention
and reduce training time.

To sum up, there are two main techniques for communication scheduling, namely,
tensor partition and priority-based scheduling [26, 39], which have remarkable
effects and thus been widely used.

Tensor partitioning. During the forward propagation, the computation of back
layers must wait for the completion of front layers [50]. However, during the
communication, tensors of back layers are transmitted first since they finish
earlier in the backward propagation [26]. To tackle this mismatching, recent
work propose tensor partitioning [39]. At its core, a tensor will be divided
into smaller chunks if its size is larger than a threshold, i.e, partition
size. With the help of tensor partitioning, we are able to switch the
transmission from back layers to front layers. The forward propagation can
start immediately after receiving the front layers’ tensors. Therefore, the
overlap of the communication and computation time is increased and the
training time is reduced.

Priority-based scheduling. As we mentioned above, forward propagation
processes the front layers first, therefore, a straightforward idea is to
prioritize the transmission of front layers. With the help of priority-based
scheduling, we can set the priority to be the index of the tensor’s layer.
Thus these front-layer tensors can be transmitted first and the forward
propagation can start earlier, which greatly mitigate the communication
overhead. Priority-based scheduling is also proved to be the theoretical
optimal [39].

###  II-D Hyper-Parameters for ByteScheduler

There are two tunable parameters which are essential to the training
performance: the partition size and the credit size [39].

Partition size. Figure 1 shows the effect of partition size. $S_{p}$ is the
threshold of splitting the communication tensor. As shown in Figure 1(b), when
the tensors are splitted into appropriate small pieces, the communication and
computation are well overlapped, which results in a shorter training
time,1.28$\times$ speedup. A larger partition size in Figure 1(c) brings about
performance deficiency(from 1.28$\times$ to 1.19$\times$). Intuitively, the
ideal status is that the partition size is infinitesimal. Then the backward
propagation and the former communication operation can start at the same time.
This can lead to a shortest training time.

However, there is a gap between theoretical upper bound and the real world
performance. The cost of tensor partition is not small enough to be ignored.
This partition overhead will bring about additional time consumption, making
infinitesimal partition size hard to satisfy in practice. There is usually an
inherent partition size which can achieve optimal training speed [39].

Credit size. Credit size $S_{c}$ is the size of a sliding window. It allows
the tensor in the credit window size to be transmitted in parallel. The credit
size is usually the multiple of the partition size. Intuitively, a larger
credit size allows multiple tensors sending together. It allows the tensors
filling the sending buffer in the network stack and fully utilize the network
bandwidth [39].

As shown in Figure 2. When the credit size equals to 1X in Figure 2(a), the
priority scheduling policy is stop-and-wait since it allows one tensor in the
sending buffer. Such a credit size is far from optimal in a 10Gbps network
because of low bandwidth utilization. When the credit size equals to 2X as
shown in Figure 2(b), the training time is reduced since a larger credit size
leads to better bandwidth utilization. It increases the training speed from
1.37$\times$ to 1.52$\times$. However, a larger credit size will undermine the
priority-scheduling. In Figure 2(c), when the credit size equals to 5X, the P0
tensors can not finish transmission before P1 tensors, thus F0 can not start
beforehand. A larger credit size performs worse than a smaller credit size
(the speedup drops to 1.12$\times$). Actually, credit-size is a critical
parameter to balance the trade-off between the network utilization and
priority scheduling.

###  II-E Motivating Examples

These two important parameters, partition size and credit size, affect
training performance. We use experiments to showcase this. We train four
typical models, ResNet [20], VGG [41], AlexNet [30] and DenseNet [25] with
MXNet [10] PS [33] and PyTorch [37] Allreduce [38] (NCCL [27]). In the
motivation experiments, we vary partition size from 1M to 8M and credit size
from 1X to 4X. For the dynamic bandwidth experiment, we vary the bandwidth
from 3Gbps to 20Gbps. The results are shown in Figure 3.

(a) Model influence

(b) Bandwidth influence

Figure 3: Different models and available bandwidth have influence on optimal
pair of $\langle S_{p},S_{c}\rangle$.

The results show two points. (1) Different ML jobs result in different optimal
pair of $\langle S_{p},S_{c}\rangle$. Different network environments do the
same. As shown in Figure 3(a), the optimal $\langle S_{p},S_{c}\rangle$ is
$\langle 1,4\rangle$ for ResNet, $\langle 4,1\rangle$ for VGG, $\langle
1,1\rangle$ for AlexNet and $\langle 8,2\rangle$ for DenseNet. Different ML
models have respective proportion of communication to computation, thus their
inherent parallel ratios are different. This reflects to different optimal
configuration of $\langle S_{p},S_{c}\rangle$.

Besides, in Figure 3(b) different bandwidth will have influence on the choice
of optimal $\langle S_{p},S_{c}\rangle$. The optimal $\langle
S_{p},S_{c}\rangle$ is $\langle 1,4\rangle$ for 20G bandwidth, $\langle
1,2\rangle$ for 8G bandwidth and $\langle 4,2\rangle$ for 5G and 3G bandwidth.
This is because in the real-world, scheduling and tensor partitioning have
networking-related overhead. And credit-based priority scheduling is designed
to fully utilize the network. Thus there exists an optimal pair of $\langle
S_{p},S_{c}\rangle$ in particular network setting.

(2) Optimal pair of $\langle S_{p},S_{c}\rangle$ can greatly improve the
training speed. As shown in Figure 3(a), the improvement of adjusting
configuration ranges from $5.4\%-99.0\%$ for different ML jobs and
$21.8\%-45.4\%$ for different network setting.

TABLE I: Categorization of approaches on parameter tuning

| Dynamic | Static | Cost
---|---|---|---
Methods | Traffic | GPUs | Architecture | Model | Network | Degree
Default |  |  |  |  |  | Low
Grid Search |  |  | $\checkmark$ | $\checkmark$ | $\checkmark$ | High
BO |  |  | $\checkmark$ | $\checkmark$ | $\checkmark$ | Low
AutoByte | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | Medium

Actually, current communication scheduling methods [26, 39] have used
different methods to tune these two knobs $S_{p}$ and $S_{c}$. Table I
provides an overview and categorization of these relevant methods. 1)Default
method. P3 [26] uses a default partition size of 160KB and credit size of 1X.
This method has no additional overhead. But this partition size is far from
optimal in 10Gbps network. This stable method may lead to performance
degradation in some particular network condition and ml models. 2) The Grid
Search method. It is a simple way to search the best credit size and partition
size. It enumerates all possible combinations of $\langle S_{p},S_{c}\rangle$.
But the cost of enumeration is too high. It is also ineffective where
background traffic exists. 3) Bayesian Optimization (BO). ByteScheduler [39]
adopts BO to tune $\langle S_{p},S_{c}\rangle$ together. BO can reach optimal
configuration with much less cost and give much more stable performance.
However, BO only works before the beginning of training. Once the train
starts, the $\langle S_{p},S_{c}\rangle$ stay constant. It can not deal with
the dynamic condition like bandwidth competition and computing power change.
4) AutoByte. Our proposed method should not only find the optimal pair for
particular ML model and network, but also keep the optimal performance when
sudden change happens.

The optimal pair are likely to vary with many factors as listed in table I.
The physical network bandwidth, gradient synchronization method, DNN model
types all have effects on these parameters. Sudden change of network bandwidth
and computation power also have effects. Besides, the cost of reconfiguration
should be within an acceptable range.

Design goal. The design of AutoByte has three objectives.

  * •

AutoByte should automatically choose the parameters to achieve optimal
training performance in different run-time environments.

  * •

AutoByte should adjust the parameters when dealing with network or computation
power change over the training courses to keep optimal performance.

  * •

AutoByte should have an acceptable search cost.

We will introduce how AutoByte achieves the three objectives in $\S$III.

##  III Design

Through conducting the motivation experiments, we have demonstrated that how
the DNN model and network bandwidth affect the optimal setting of the credit-
size and partition size. Therefore, we extend an existing priority
communication scheduling framework ByteScheduler to automatically configure
the hyper parameters. We call the plugin as AutoByte, which can adjust two
core parameters:1) partition-size and 2) credit-size. We build a Meta Network
Optimizer, which can predict the training speedup under different hyper
parameter’s configuration according to the runtime metrics, i.e., the
computation power and background traffic bandwidth. Thus we can choose the
optimal hyper-parameter setting and increase the training speedup.

However, dynamic tuning the two parameters is non-trivial due to several
reasons. Firstly, formally modeling and solving this problem is nearly
impossible, since we can not explicitly express the training performance as
the function of two parameters. Secondly, the distributed training system is
very complicated. This is reflected in many places. For example, different DNN
models vary in number of layers, each layer’s computation time and the size of
parameter.The tuning experience in one DNN model can not be directly applied
to another. Next the computation power and communication bandwidth settings
are vary in different runtime environment. Furthermore, it’s quite complex to
analyze particular cases such as computation straggling and communication
straggling.

To solve this problem, the first thought that comes to mind is to consistently
search for the best values and use newly profiled results. Grid search and
Bayesian Optimization(BO)[6] are both able to search for the optimal partition
size and credit size. But these method may incur significantly search costs.
The cost may even outweigh the improvement of training performance.

Therefore, we propose to design a meta-network. This network makes a
prediction of training speed under different pairs of parameter settings. Thus
we can select the optimal pair with the maximum training speedup at the cost
of one inference.

###  III-A Meta Network Optimizer

TABLE II: Important Runtime Metrics (Input Features)

Symbol | Shape | Specification
---|---|---
$S_{c}$ | $1$ | Credit size
$S_{p}$ | $1$ | Partition size
$n$ | $1$ | Total number of workers
$l$ | $1$ | Total number of backward propagation layers
$m$ | $m*1$ | Model type embedding vector
$arc$ | $a*1$ | Architecture type embedding vector
$B_{d}$ | $n*1$ | Network download speed for all workers
$B_{u}$ | $n*1$ | Network upload speed for all workers
T | $l*n$ | Layer-wise computation time for all workers
V | $n*1$ | Training speed (image/s) for all workers

Input runtime metrics. Our target is to learn the mapping from the current
environment and the partition size, credit size to the model training speed.
Besides this two key hyper-parameters, the credit size $S_{c}$ and the
partition size $S_{p}$, AutoByte need to collect runtime metrics and use them
as inputs to the meta network optimizer. We provide a detailed specification
of the input and output features in Table II, .i.e., the total number of
workers $n$, the total number of backward propagation layers $l$, the model
type of current job $m$, the architecture type of current work $arc$.

Since the network condition and computation power are actually ever-changing
during the training process, we should choose appropriate metrics to represent
the dynamic runtime environment. We measure local computation time and
communication time at the end of every mini-batch. Some background traffic may
preempt the bandwidth. Thus we need an index to show the real-time bandwidth.
So we choose the network upload speed for all workers $B_{u}$ and network
download speed $B_{d}$ to show the real-time network transmission speed. Since
the bandwidth metrics may fluctuate, we apply a n-dimension vector. On the
other hand, the computation power also needs to be monitored to prevent sudden
machine breakdown. Here we use a group of parameters to represent the real-
time computation power, namely the layer-wise backward computation time of all
workers $T=[T_{1},T_{2},....T_{n}]$. With these runtime metrics, the dynamic
changes of allocated computation power from GPU clusters and the network
bandwidth can be detected.

We denote the $V$ as the training speed, the objective of the meta network is
to learn the mapping

$\displaystyle f:(T,B,S_{c},S_{p})\rightarrow V,$ (1)

under the different settings of $S_{p}$ and $S_{c}$. Then we can choose the
best pair of $\langle S_{p},S_{c}\rangle$ to optimize the training speed.

(a) Meta Network Optimizer

(b) AutoByte Workflow

Figure 4: The left figure illustrates the structure of Meta Network Optimizer.
The right figure shows the workflow, and how Meta Network Optimizer
Optimization Trigger and Execution work together.

Meta network architecture. The proposed meta network architecture is shown as
Figure 4(a). There are totally four components. The first one is dynamic
computation power monitor. As analysis before, we use layer-wise computation
time to monitor the changing of computation power. It should be noticed that
as different training tasks usually result in different number of layers.
Therefore, we firstly embed layer-wise computation time $T$ into a fixed
dimension feature space to make the meta-network robust to various of models.
Afterwards, we apply two-layer LSTM to extract the sequential features in $T$.
The second one is dynamic background traffic (network) monitor. We use real-
time measured bandwidth series as $B_{d}$ $B_{\mu}$ to display network change.
The third one is a static fixed-length parameter group, i.e. total number of
workers. The fourth one is the alternative of $\langle S_{p},S_{c}\rangle$. We
concatenate the features extracted from the aforementioned four components.
After applying two dense layers on the concatenated feature, the training
speed $V$ is expected to be predicted.

As the objective of the meta-network is to predict precisely the speed up
vector $V$. we define the loss function of the meta-network as a typical
L2-loss function, i.e.,

$\displaystyle L(V,\overline{V})=\|V-\overline{V}\|_{L2}.$ (2)

Here, the $\overline{V}$ denote the observed average training speed. We set 10
iterations as a group. We calculate the average training speed for this group
to eliminate the error and jitters.

###  III-B Training of Meta Network

Dataset collection. To train this meta network, the first step is to collect
the training data. We use the testbed to run ByteScheduler to collect runtime
data. We have implemented the meta-network using Keras. To evaluate the meta-
network, we utilized ByteScheduler to train three typical models, ResNet50,
VGG16 and AlexNet. For the partition size experiment, we vary the partition
size from 4KB to 1024MB and credit size from 1X to 16X. To simulate the
dynamic system conditions, we dynamically adjusted the network bandwidth in a
wide range. We vary the bandwidth from 0.5, 1, 5, 10, 25Gbps at 20 iterations
interval to simulate the coexisting with background traffic. we respectively
collect each workers’ runtime statistics and record the average training speed
after 10 iterations. A total of 360000 iterations of meta-data samples have
been collected. We can use the training data to develop our meta model
AutoByte.

Offline training online adapting. One critical issue is that DNN is a data-
driven approach which can not handle the out of distribution problem. As the
distributed deep-learning system environment may vary in both hardware(GPU and
network) and software (model and DL architecture) settings, it is not
practical to acquire a perfect distributed training dataset.

One possible solution is to perform online training. we can online train and
update the meta-network on the target distributed deep learning task. However,
this also introduces system overhead as we are required to try exhaustive
parameter settings recurrently to meet with various of system conditions. This
method violates our original intention, i.e.,maximizing the training speed.

To overcome the issue, we proposed an offline training, online adapting
approach, whose key idea is use transfer learning to quickly adapt the meta-
network to the current environment with low system overhead. This method has
the following three advantages. 1)It does not bring about much additional
system overhead. 2) Transfer learning helps the meta network optimizer quickly
adapt to the current condition. 3) This method provides higher accuracy of
speed prediction than offline trained version.

###  III-C AutoByte workflow

The workflow of the AutoByte’s online framework is shown in Figure 4(b).
AutoByte contains three important components, Meta Network Optimizer,
Optimization Trigger and Execution. The Meta Network Optimizer estimates the
optimal configuration for the running ML job using runtime metrics. Following
the decision of Optimization Trigger, the Execution applies the necessary
changes dynamically to partition size and credit size without stopping the
running job.

Meta network optimizer. When AutoByte is deployed, for the Meta Network
Optimizer, it collects runtime statistics and turn those metrics to vector
inputs. Then the meta network will predict the speedup performance in
different system configurations based on runtime metrics. After finding the
configuration that is expected to be optimal, the Optimizer will generate a
better configuration. Through reconfiguration, AutoByte changes the system
configuration to the one with better performance. But we need to make
decisions such as when to calculate a better configuration, whether to execute
the reconfiguration or not. To say that, we need an Optimization Trigger.

Optimization trigger. In order to avoid the system from continuously
reconfiguring back and forth around the estimated optimum. Optimizer Trigger
predicts the performance benefit of a new configuration and skips that attempt
if the gain is less than a certain threshold. A threshold number from our
experience 5$\%$ is good enough to prevent the system from oscillating, while
allowing the system to undergo moderately-sized optimizations.

The Optimization Trigger will also monitor the difference between the predict
$V$ and the true speedup. When the difference is larger than a certain
threshold $10\%$. It will trigger the online adapting scheme. The Meta Network
Optimizer opportunistically tries to adapt the model to the current
environment. It will use transfer learning to update the offline trained model
and also update the dataset. Thus the optimizer can adjust to find an optimal
configuration including the new online dataset.

Execution. The execution of the optimization is relatively simple. The Meta
Network Optimizer executes a plan by simply invoking the Execution API that
reconfigures system transparently without stop training. Once the decision of
a reconfiguration is made, the execution will switch the configuration of
$\langle S_{p},S_{c}\rangle$. Each time when the configuration is changed, we
checkpoint and restart the training. The $S_{p}$ and $S_{c}$ can be modified
by substituting the execution code.

##  IV Evaluation

###  IV-A Methodology

Testbed setup: Our testbed has 8 nodes, each with 20 CPU cores, 64GB memory, 1
Tesla V100, and a Mellanox CX-5 single-port NICs. We use a leaf-spine topology
with 2 racks and 4 nodes in each rack. We use Mellanox SN2100 switch with the
Onyx 3.7.1134 operating system. The link bandwidth is 100Gbps and the
oversubscription ratio is 1. Our operation system is Ubuntu 18.04 with Linux
kernel version 4.15.0-135-generic. The Mellanox driver version is 4.6-1.0.1.1.

Benchmarks: We choose 3 typical models, ResNet50 [20], VGG16 [41] and Alexnet
[41]. We run our AutoByte in MXNet [10] with the PS [33] and PyTorch [37] with
NCCL [27] (Ring all-reduce [38]). For the PS, workers and servers are on the
same machines, and their numbers are equal. We measure the performance under
different NIC bandwidth. We chose TCP as our transport protocol. Since TCP
cannot saturate the link bandwidth, we only test on 0.5, 1, 5, 10, 25 Gbps. We
leave the RDMA test as the future work.

Compared Schemes: We compare AutoByte with the following two different
communication scheduling methods.

  * •

Baseline: We use the vanilla PS for MXNet and NCCL for PyTorch. For
illustration, the backward propagation and forward propagation are executed
sequentially without tensor partitioning and priority-scheduling.

  * •

ByteScheduler: ByteScheduler [39] is a generic communication scheduler, which
combines priority scheduling and tensor partitioning. We use the open-source
code of ByteSchedulerfor evaluation. We use two methods to set the partition
size and credit size, namely, grid search and Bayes optimization [6].

To clearly show how AutoByte’s Meta Network Optimizer works, especially how it
optimizes the parameter configuration when network resource changes and how
much overhead it incurrs. We first compare the Meta Network Optimizer with
other methods, i.e., the Grid Search and Bayesian Optimization (BO). We use
the code of BO method in ByteScheduler. We compare these three optimizers on
the speed and the cost of reconfiguration. We measure how much iterations they
need to find the optimal pair of $\langle S_{p},S_{c}\rangle$, and how much
extra overhead they bring.

We use training speed (samples/sec) as the main metric. We measure the average
training speed over the first 50 iterations after a warm-up of 10 iterations.

###  IV-B Overview of the experiments

We conduct the evaluation of AutoByte and illustrate the result in the
following four sections: (1) In $\S$ IV-C, we validate that AutoByte can
timely search for the optimal hyper-parameters during the training. We firstly
introduce the method we use to find the optimal configuration. Then we show
that AutoByte can quickly find a nearly optimal hyper parameters. (2) In $\S$
IV-D, we evaluate AutoByte when available bandwidth and computing resources
are changing, which are common in a cloud environment. We emulate the
resources changing by artificially adding a new training job. The experiment
shows that AutoByte outperformances ByteScheduler with BO by $18.2\%\sim
39.3\%$. (3) In $\S$ IV-E, we show the speedup of AutoByte in different
environments. We compare AutoByte with the baseline, ByteScheduler (with BO),
on three models under different network bandwidth, different architectures (PS
and Ring all-reduce). The result shows that with the help of Meta Network
Optimizer, AutoByte can improve the training speed with up to 94.4$\%$ and
33.2$\%$, compared with baseline and ByteScheduler. (4) In $\S$ IV-F, we
investigate the overhead introduced by AutoByte. We also compare the
reconfiguration speed of AutoByte with other searching methods, namely, BO and
grid search.

###  IV-C AutoByte finds the Optimal Hyper-parameters

In this section, we evaluate AutoByte to show that: 1) The output of AutoByte
can be very close to the optimal. 2) The hyper-parameters chosen by AutoByte
can fast converge to the optimal. 3) AutoByte can bring considerable speedup
to the training.

Figure 5: Training ResNet50 with 2 different configurations. The black line
indicates the optimum, the red line and the blue line show two different
configurations. The dotted lines show the performance without optimization.
The vertical lines represent the reconfiguration.  Figure 6: Training speed
under dynamic bandwidth condition. Blue line shows AutoByte’s ability to adapt
to dynamic network compared to BO drawn in black line, which assume the values
stay constant throughout the training. The areas filled in sky-blue denote the
reconfigurations.  Figure 7: Training speed under dynamic changing of
available resources. Black line and red line indicates the training of
AutoByte and ByteScheduler with BO, repectively. Vertical lines and areas
filled in orange represent the event of reconfiguration.

(a) ResNet50, PS, MXNet (b) VGG16, PS, MXNet (c) AlexNet, PS, MXNet

Figure 8: The training speed of ResNet, VGG, AlexNet models under different
bandwidth conditions in Parameter Server.

(a) ResNet50, Ring, PyTorch (b) VGG16, Ring, PyTorch (c) AlexNet, Ring,
PyTorch

Figure 9: The training speed of ResNet, VGG, AlexNet models under different
bandwidth conditions in All Reduce.

Optimal prediction. We can obtain the optimal setting of $\langle
S_{p},S_{c}\rangle$ by grid search. We first apply grid search to all possible
configurations $\langle S_{p_{i}},S_{c_{i}}\rangle$, to find the ground truth.
The step for partition size ($S_{p_{i}}$) is 2M, and for credit size
($S_{c_{i}}$) is 1X in grid search. We denote the optimal pair as $\langle
S_{p_{*}},S_{c_{*}}\rangle$. Figure 5 shows that the Meta Network Optimizer
could predict the optimal configurations very close to the ground truth even
with different initial parameters.

Fast convergence. In our setting, AutoByte conducts new prediction every 10
iterations. Figure 5 shows that the output of AutoByte can converge to the
optimal after three times prediction, which is corresponding to 30 iterations,
far less than the total iteration number in one epoch (often more than one
hundred). Therefore, in practice, AutoByte can quickly find the optimal
configuration.

Training improvement. As shown in Figure 5, for the red line, when change from
$\langle S_{p}\colon 2,S_{c}\colon 2\rangle$ to $\langle S_{p}\colon
1,S_{c}\colon 4\rangle$, AutoByte speedup the training by 12.6$\%$. For the
blue line, when change from $\langle S_{p}\colon 8,S_{c}\colon 3\rangle$ to
$\langle S_{p}\colon 1,S_{c}\colon 4\rangle$, the speedup is 81$\%$.

###  IV-D Optimization under Dynamic Resources

The above experiments are under a static environment. Actually, AutoByte can
further speedup the training with a dynamic environment. AutoByte can keep
track of dynamic network and computation resources in the cluster and timely
updates the parameter configuration.

In the first experiment, we train a ResNet50 model. We simulate throughput
variation through artificially changing the NIC bandwidth. We switch the
bandwidth between 3Gbps and 10Gbps every 20 iterations. At the beginning, the
bandwidth is 10Gbps. We compare the training speed of AutoByte with
ByteScheduler (BO). BO chooses the best credit size and partition size for the
initial environment. Once chosen, it assumes the value stay constant
throughout the training process. Figure 6 shows the average speed of training
for every 5 iterations. At the $20^{\mathrm{th}}$ iteration, AutoByte’s
configuration moves toward $\langle S_{p}\colon 4,S_{c}\colon 2\rangle$. This
configuration better fits the low bandwidth setting, achieving higher speedup
than ByteScheduler. At the $40^{\mathrm{th}}$ iteration, AutoByte’s
configuration returns to the previous choice $\langle S_{p}\colon
8,S_{c}\colon 3\rangle$. Thus the AutoByte can achieve 6.4$\%$-23$\%$
improvement in the training speedup.

In the second experiment, we simulate the changing of available bandwidth and
computation resources by adding new training jobs. We add a new training job
at $5^{\mathrm{th}}$ and $25^{\mathrm{th}}$ iterations. Since the job
initialization takes 15 iteration’s time, in the first $20^{\mathrm{th}}$
iterations, the measured job occupy the whole bandwidth (20Gbps) and the whole
GPU. In the next 20 iterations, two jobs share the resources. In the last 20
iterations, three jobs share resources. As shown in Figure 7, we measure the
average training speed of every 5 iterations. At the $20^{\mathrm{th}}$
iteration, AutoByte’s configuration moves toward $\langle S_{p}\colon
4,S_{c}\colon 2\rangle$. This configuration better fit the low bandwidth
setting. Thus achieve higher speedup than ByteScheduler with BO. At the
$40^{\mathrm{th}}$ iteration, AutoByte’s configuration turns to the $\langle
S_{p}\colon 2,S_{c}\colon 1\rangle$. and AutoByte achieve $8.8\%\sim 19.2\%$
higher performance.

###  IV-E Optimization under Various Environments

Figure 8 and 9 compare the training speed of baseline and ByteScheduler with
AutoByte on three models under different network bandwidth ranging from
500Mbps to 25 Gbps (500Mbps, 1Gbps, 5Gbps, 10Gbps, 25Gbps) under two different
architectures (PS $\&$ All-reduce), which are implemented by MXNet and PyTorch
respectively.

From Figure 8 (a)-(c) We observe that AutoByte can outperform baseline /
ByteScheduler with BO by 16.7$\%$ to 60.1$\%$ / 3.1$\%$ to 17.5$\%$ for
ResNet50, 30.2$\%$ to 94.4 $\%$ / 15.5$\%$ to 33.2$\%$ for VGG16, 40.7$\%$ to
79.7$\%$ / 10.6$\%$ to 31.1$\%$ for AlexNet. AutoByte’s meta network takes
into consideration of the model’s layer-wise information and dynamic change of
available resources. This is the reason AutoByte outperform baseline and
ByteScheduler (BO) . We could observe the same trend for Ring all-reduce in
PyTorch in Figure 9 (a)-(c), 15.1$\%$ to 144.4$\%$ improvements for baseline,
and 2.3$\%$ to 17.4 $\%$ improvements for ByteScheduler. This shows that
AutoByte is not only adapt to different run-time environments, but also
generic to different frameworks.

###  IV-F Reconfiguration Overhead and Speed

To make AutoByte ready deployable, we need to ensure a low overhead and fast
execution speed. To measure the overhead, we compare AutoByte with a constant
hyper-parameter setting. We find that their cpu utilization is almost the same
(difference is less than 1%). To measure the execution speed, we compare
AutoByte with grid search and BO, as shown in Figure 10, AutoByte reduce the
execution time by $28\%\sim 81\%$ compared to the grid search and the
performance is close to BO.

Figure 10: Search costs of different searching algorithms

##  V Related work

Communication optimization for distributed training. Generally speaking, there
are a wide range of approaches we can explore to optimize the communication
for distributed DNN training. These include, but are not limited to: 1) using
large mini-batch [17] and periodic communication [43] to reduce the
communication rounds; 2) using gradient compression technique, e.g., gradient
sparsification [35] and quantization [3], to reduce the taffic volume in each
iteration; 3) relaxing the synchronization requirement [23, 24, 47]; 4) taking
the intra-machine GPU topology into consideration [28]; 5) designing a
parameter exchanging scheme considering the network topology [45]; 6)
overlapping communication with computation [50, 39, 26]; 7) leveraging
advanced communication library, e.g., ZMQ [22] and NCCL [27]; 8) exploiting
fast network protocols, e.g., RDMA [18, 49]; 9) performing in-network
aggregation to reduce the in-network traffic volume [40, 31, 7]; 10)
minimizing network flow completion time by using congestion control [4, 8],
flow scheduling [5, 34] or coflow scheduling [13, 51, 44, 52]. We note that,
while some of these methods have already been integrated into distributed DNN
training systems, others remain to be explored in the future.

Automatic parameter configuration. Parameter configuration is necessary for
many applications, such as big data analytics [2], tuning machine learning
hyper-parameters [42], and databases [15]. Recently, many works focus on
automatic configure the parameters. For datacenter congestion control, PCC
[14] leverages the reinforcement learning to automatically set the congestion
windows. For the flow scheduling, AuTO [9] dynamically chooses the suitable
priority for each flow. For hyper-parameters tuning of machine learning,
AutoML [21] can automatically find the optimal setting.

##  VI Conclusion

In this paper, we have presented a methodology to automatically tune parameter
configuration of ByteScheduler communication scheduling system. We designed
AutoByte, which is composed of Meta Network Optimizer, Optimization Trigger
and Elastic Execution, to dynamically select the two critical parameters, the
partition size and the credit size. The Meta Network Optimizer estimates the
optimal configuration using runtime metrics. Following the decision of
Optimization Trigger, the Elastic Execution applies the necessary changes
dynamically to partition size and credit size without stopping the running
job. Thus, AutoByte can further increase the scheduling efficiency and
decrease the training time. Our evaluation shows that AutoByte frees ML
developers of choosing the right parameter configuration by tuning the system
configuration automatically.AutoByte can optimize the system configuration
when resource availability changes, reducing 33.2% training time in dynamic
network environment.

## References

  * [1] Mart’n Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al.  Tensorflow: A system for large-scale machine learning.  In 12th $\\{$USENIX$\\}$ symposium on operating systems design and implementation ($\\{$OSDI$\\}$ 16), pages 265–283, 2016.
  * [2] Omid Alipourfard, Hongqiang Harry Liu, Jianshu Chen, Shivaram Venkataraman, Minlan Yu, and Ming Zhang.  Cherrypick: Adaptively unearthing the best cloud configurations for big data analytics.  In 14th $\\{$USENIX$\\}$ Symposium on Networked Systems Design and Implementation ($\\{$NSDI$\\}$ 17), pages 469–482, 2017.
  * [3] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic.  Qsgd: Communication-efficient sgd via gradient quantization and encoding.  Advances in Neural Information Processing Systems, 30:1709–1720, 2017.
  * [4] Mohammad Alizadeh, Albert Greenberg, David A Maltz, Jitendra Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Sridharan.  Data center tcp (dctcp).  In Proceedings of the ACM SIGCOMM 2010 Conference, pages 63–74, 2010.
  * [5] Wei Bai, Li Chen, Kai Chen, Dongsu Han, Chen Tian, and Hao Wang.  Information-agnostic flow scheduling for commodity data centers.  In 12th $\\{$USENIX$\\}$ Symposium on Networked Systems Design and Implementation ($\\{$NSDI$\\}$ 15), pages 455–468, 2015.
  * [6] Eric Brochu, Vlad M. Cora, and Nando de Freitas.  A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning.  arXiv:1012.2599 [cs], December 2010.
  * [7] Li Chen, Ge Chen, Justinas Lingys, and Kai Chen.  Programmable switch as a parallel computing device.  arXiv preprint arXiv:1803.01491, 2018.
  * [8] Li Chen, Shuihai Hu, Kai Chen, Haitao Wu, and Danny HK Tsang.  Towards minimal-delay deadline-driven data center tcp.  In Proceedings of the Twelfth ACM Workshop on Hot Topics in Networks, pages 1–7, 2013.
  * [9] Li Chen, Justinas Lingys, Kai Chen, and Feng Liu.  Auto: Scaling deep reinforcement learning for datacenter-scale automatic traffic optimization.  In Proceedings of the 2018 conference of the ACM special interest group on data communication, pages 191–205, 2018.
  * [10] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang.  MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems.  arXiv:1512.01274 [cs], December 2015.
  * [11] J Michael Cherry, Caroline Adler, Catherine Ball, Stephen A Chervitz, Selina S Dwight, Erich T Hester, Yankai Jia, Gail Juvik, TaiYun Roe, Mark Schroeder, et al.  Sgd: Saccharomyces genome database.  Nucleic acids research, 26(1):73–79, 1998.
  * [12] Gobinda G Chowdhury.  Natural language processing.  Annual review of information science and technology, 37(1):51–89, 2003.
  * [13] Mosharaf Chowdhury, Yuan Zhong, and Ion Stoica.  Efficient coflow scheduling with varys.  In Proceedings of the 2014 ACM conference on SIGCOMM, pages 443–454, 2014.
  * [14] Mo Dong, Qingxi Li, Doron Zarchy, P Brighten Godfrey, and Michael Schapira.  $\\{$PCC$\\}$: Re-architecting congestion control for consistent high performance.  In 12th $\\{$USENIX$\\}$ Symposium on Networked Systems Design and Implementation ($\\{$NSDI$\\}$ 15), pages 395–408, 2015.
  * [15] Songyun Duan, Vamsidhar Thummala, and Shivnath Babu.  Tuning database configuration parameters with ituned.  Proceedings of the VLDB Endowment, 2(1):1246–1257, 2009.
  * [16] Luciano Floridi and Massimo Chiriatti.  Gpt-3: Its nature, scope, limits, and consequences.  Minds and Machines, 30(4):681–694, 2020.
  * [17] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  arXiv preprint arXiv:1706.02677, 2017.
  * [18] Chuanxiong Guo, Haitao Wu, Zhong Deng, Gaurav Soni, Jianxi Ye, Jitu Padhye, and Marina Lipshteyn.  Rdma over commodity ethernet at scale.  In Proceedings of the 2016 ACM SIGCOMM Conference, pages 202–215, 2016.
  * [19] Sayed Hadi Hashemi, Sangeetha Abdu Jyothi, and Roy H Campbell.  Tictac: Accelerating distributed deep learning with communication scheduling.  arXiv preprint arXiv:1803.03288, 2018.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
  * [21] Xin He, Kaiyong Zhao, and Xiaowen Chu.  Automl: A survey of the state-of-the-art.  Knowledge-Based Systems, 212:106622, 2021.
  * [22] Pieter Hintjens.  ZeroMQ: messaging for many applications.  ” O’Reilly Media, Inc.”, 2013.
  * [23] Qirong Ho, James Cipar, Henggang Cui, Seunghak Lee, Jin Kyu Kim, Phillip B Gibbons, Garth A Gibson, Greg Ganger, and Eric P Xing.  More effective distributed ml via a stale synchronous parallel parameter server.  In Advances in neural information processing systems, pages 1223–1231, 2013.
  * [24] Kevin Hsieh, Aaron Harlap, Nandita Vijaykumar, Dimitris Konomis, Gregory R Ganger, Phillip B Gibbons, and Onur Mutlu.  Gaia: Geo-distributed machine learning approaching $\\{$LAN$\\}$ speeds.  In 14th $\\{$USENIX$\\}$ Symposium on Networked Systems Design and Implementation ($\\{$NSDI$\\}$ 17), pages 629–647, 2017.
  * [25] Forrest Iandola, Matt Moskewicz, Sergey Karayev, Ross Girshick, Trevor Darrell, and Kurt Keutzer.  Densenet: Implementing efficient convnet descriptor pyramids.  arXiv preprint arXiv:1404.1869, 2014.
  * [26] Anand Jayarajan, Jinliang Wei, Garth Gibson, Alexandra Fedorova, and Gennady Pekhimenko.  Priority-based Parameter Propagation for Distributed DNN Training.  arXiv:1905.03960 [cs], May 2019.
  * [27] Sylvain Jeaugey.  Nccl 2.0.  In GPU Technology Conference (GTC), 2017.
  * [28] Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, and Chuanxiong Guo.  A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters.  page 18.
  * [29] Minkoo Kang, Gyeongsik Yang, Yeonho Yoo, and Chuck Yoo.  Tensorexpress: In-network communication scheduling for distributed deep learning.  In 2020 IEEE 13th international conference on cloud computing (CLOUD), pages 25–27. IEEE, 2020.
  * [30] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  ImageNet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, May 2017.
  * [31] ChonLam Lao, Yanfang Le, Kshiteej Mahajan, Yixi Chen, Wenfei Wu, Aditya Akella, and Michael Swift.  ATP: In-network aggregation for multi-tenant learning.  In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21), pages 741–761. USENIX Association, April 2021\.
  * [32] Woo-Yeon Lee, Yunseong Lee, Joo Seong Jeong, Gyeong-In Yu, Joo Yeon Kim, Ho Jin Park, Beomyeol Jeon, Wonwook Song, Gunhee Kim, Markus Weimer, et al.  Automating system configuration of distributed machine learning.  In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), pages 2057–2067. IEEE, 2019.
  * [33] Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su.  Scaling distributed machine learning with the parameter server.  In 11th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 14), pages 583–598, 2014.
  * [34] Ziyang Li, Wei Bai, Kai Chen, Dongsu Han, Yiming Zhang, Dongsheng Li, and Hongfang Yu.  Rate-aware flow scheduling for commodity data center networks.  In IEEE INFOCOM 2017-IEEE Conference on Computer Communications, pages 1–9. IEEE, 2017.
  * [35] Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally.  Deep gradient compression: Reducing the communication bandwidth for distributed training.  arXiv preprint arXiv:1712.01887, 2017.
  * [36] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia.  Pipedream: generalized pipeline parallelism for dnn training.  In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 1–15, 2019.
  * [37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  arXiv:1912.01703 [cs, stat], December 2019.
  * [38] Pitch Patarasuk and Xin Yuan.  Bandwidth optimal all-reduce algorithms for clusters of workstations.  Journal of Parallel and Distributed Computing, 69(2):117–124, 2009\.
  * [39] Yanghua Peng, Yibo Zhu, Yangrui Chen, Yixin Bao, Bairen Yi, Chang Lan, Chuan Wu, and Chuanxiong Guo.  A generic communication scheduler for distributed DNN training acceleration.  In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 16–29, Huntsville Ontario Canada, October 2019\. ACM.
  * [40] Amedeo Sapio, Marco Canini, Chen-Yu Ho, Jacob Nelson, Panos Kalnis, Changhoon Kim, Arvind Krishnamurthy, Masoud Moshref, Dan Ports, and Peter Richtarik.  Scaling distributed machine learning with in-network aggregation.  In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21), pages 785–808. USENIX Association, April 2021\.
  * [41] Karen Simonyan and Andrew Zisserman.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  arXiv:1409.1556 [cs], April 2015.
  * [42] Jasper Snoek, Hugo Larochelle, and Ryan P Adams.  Practical bayesian optimization of machine learning algorithms.  Advances in neural information processing systems, 25, 2012.
  * [43] Sebastian U Stich.  Local sgd converges fast and communicates little.  arXiv preprint arXiv:1805.09767, 2018.
  * [44] Hengky Susanto, Hao Jin, and Kai Chen.  Stream: Decentralized opportunistic inter-coflow scheduling for datacenter networks.  In 2016 IEEE 24th International Conference on Network Protocols (ICNP), pages 1–10. IEEE, 2016.
  * [45] Xinchen Wan, Hong Zhang, Hao Wang, Shuihai Hu, Junxue Zhang, and Kai Chen.  Rat-resilient allreduce tree for distributed machine learning.  In 4th Asia-Pacific Workshop on Networking, pages 52–57, 2020.
  * [46] Hao Wang, Jingrong Chen, Xinchen Wan, Han Tian, Jiacheng Xia, Gaoxiong Zeng, Weiyan Wang, Kai Chen, Wei Bai, and Junchen Jiang.  Domain-specific communication optimization for distributed dnn training.  arXiv preprint arXiv:2008.08445, 2020.
  * [47] Weiyan Wang, Cengguang Zhang, Liu Yang, Jiacheng Xia, Kai Chen, and Kun Tan.  Divide-and-shuffle synchronization for distributed machine learning.  arXiv preprint arXiv:2007.03298, 2020.
  * [48] Jiacheng Xia, Gaoxiong Zeng, Junxue Zhang, Weiyan Wang, Wei Bai, Junchen Jiang, and Kai Chen.  Rethinking transport layer design for distributed machine learning.  In Proceedings of the 3rd Asia-Pacific Workshop on Networking 2019, pages 22–28, 2019.
  * [49] Bairen Yi, Jiacheng Xia, Li Chen, and Kai Chen.  Towards zero copy dataflows using rdma.  In Proceedings of the SIGCOMM Posters and Demos, pages 28–30. 2017\.
  * [50] Hao Zhang, Zeyu Zheng, Shizhen Xu, Wei Dai, Qirong Ho, Xiaodan Liang, Zhiting Hu, Jinliang Wei, Pengtao Xie, and Eric P Xing.  Poseidon: An efficient communication architecture for distributed deep learning on $\\{$GPU$\\}$ clusters.  In 2017 $\\{$USENIX$\\}$ Annual Technical Conference ($\\{$USENIX$\\}$$\\{$ATC$\\}$ 17), pages 181–193, 2017.
  * [51] Hong Zhang, Li Chen, Bairen Yi, Kai Chen, Mosharaf Chowdhury, and Yanhui Geng.  Coda: Toward automatically identifying and scheduling coflows in the dark.  In Proceedings of the 2016 ACM SIGCOMM Conference, pages 160–173, 2016.
  * [52] Yangming Zhao, Kai Chen, Wei Bai, Minlan Yu, Chen Tian, Yanhui Geng, Yiming Zhang, Dan Li, and Sheng Wang.  Rapier: Integrating routing and scheduling for coflow-aware data center networks.  In 2015 IEEE Conference on Computer Communications (INFOCOM), pages 424–432. IEEE, 2015.


* [145] Z. Lu, I. Whalen, V. Boddeti, Y. Dhebar, K. Deb, E. Goodman, and W. Banzhaf, “Nsga-net: neural architecture search using multi-objective genetic algorithm,” in _Proceedings of the Genetic and Evolutionary Computation Conference_ , 2019, pp. 419–427.
  * [146] P. Ngatchou, A. Zarei, and A. El-Sharkawi, “Pareto multi objective optimization,” in _Proceedings of the 13th International Conference on, Intelligent Systems Application to Power Systems_ , 2005, pp. 84–91.
  * [147] X. Chu, B. Zhang, H. Ma, R. Xu, and Q. Li, “Fast, accurate and lightweight super-resolution with neural architecture search,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 59–64.
  * [148] J. Y. Ahn and N. I. Cho, “Neural architecture search for image super-resolution using densely constructed search space: Deconas,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 4829–4836.
  * [149] H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean, “Efficient neural architecture search via parameters sharing,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 4095–4104.
  * [150] Y. Guo, Y. Luo, Z. He, J. Huang, and J. Chen, “Hierarchical neural architecture search for single image super-resolution,” _IEEE Signal Processing Letters_ , vol. 27, pp. 1255–1259, 2020.
  * [151] Y.-C. Chen, C. Gao, E. Robb, and J.-B. Huang, “Nas-dip: Learning deep image prior with neural architecture search,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16_. Springer, 2020, pp. 442–459.
  * [152] R. Lee, Ł. Dudziak, M. Abdelfattah, S. I. Venieris, H. Kim, H. Wen, and N. D. Lane, “Journey towards tiny perceptual super-resolution,” in _European Conference on Computer Vision_. Springer, 2020, pp. 85–102.
  * [153] H. Liu, K. Simonyan, and Y. Yang, “Darts: Differentiable architecture search,” _arXiv preprint arXiv:1806.09055_ , 2018.
  * [154] H. Zhang, Y. Li, C. Gong, H. Chen, Z. Bai, and C. Shen, “Memory-efficient hierarchical neural architecture search for image restoration,” _arXiv preprint arXiv:2012.13212_ , 2020.
  * [155] Y. Wu, Z. Huang, S. Kumar, R. S. Sukthanker, R. Timofte, and L. Van Gool, “Trilevel neural architecture search for efficient single image super-resolution,” _arXiv preprint arXiv:2101.06658_ , 2021.
  * [156] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” _arXiv preprint arXiv:1710.10196_ , 2017.
  * [157] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4401–4410.
  * [158] Y. Wang, L. Wang, J. Yang, W. An, and Y. Guo, “Flickr1024: A large-scale dataset for stereo image super-resolution,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , 2019, pp. 0–0.
  * [159] E. Agustsson and R. Timofte, “Ntire 2017 challenge on single image super-resolution: Dataset and study,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 126–135.
  * [160] K. Zhang, D. Li, W. Luo, W. Ren, B. Stenger, W. Liu, H. Li, and M.-H. Yang, “Benchmarking ultra-high-definition image super-resolution,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 14 769–14 778.
  * [161] S. Gu, A. Lugmayr, M. Danelljan, M. Fritsche, J. Lamour, and R. Timofte, “Div8k: Diverse 8k resolution image dataset,” in _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_. IEEE, 2019, pp. 3512–3516.
  * [162] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 33, no. 5, pp. 898–916, 2010.
  * [163] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _2012 IEEE conference on computer vision and pattern recognition_. IEEE, 2012, pp. 3354–3361.
  * [164] D. Yi, Z. Lei, S. Liao, and S. Z. Li, “Learning face representation from scratch,” _arXiv preprint arXiv:1411.7923_ , 2014.
  * [165] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 3730–3738.
  * [166] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “Vggface2: A dataset for recognising faces across pose and age,” in _2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018)_. IEEE, 2018, pp. 67–74.
  * [167] X. Wang, K. Yu, C. Dong, and C. C. Loy, “Recovering realistic texture in image super-resolution by deep spatial feature transform,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 606–615.
  * [168] Z. Shen, W. Wang, X. Lu, J. Shen, H. Ling, T. Xu, and L. Shao, “Human-aware motion deblurring,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * [169] J. Yang, S. Shen, H. Yue, and K. Li, “Implicit transformer network for screen content image continuous super-resolution,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [170] C. Liu and D. Sun, “On bayesian adaptive video super resolution,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 36, no. 2, pp. 346–360, 2013.
  * [171] J. Y. Lin, R. Song, C.-H. Wu, T. Liu, H. Wang, and C.-C. J. Kuo, “Mcl-v: A streaming video quality assessment database,” _Journal of Visual Communication and Image Representation_ , vol. 30, pp. 1–9, 2015.
  * [172] S. Nah, T. H. Kim, and K. M. Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * [173] T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman, “Video enhancement with task-oriented flow,” _International Journal of Computer Vision (IJCV)_ , vol. 127, no. 8, pp. 1106–1125, 2019.
  * [174] X. Yang, W. Xiang, H. Zeng, and L. Zhang, “Real-world video super-resolution: A benchmark dataset and a decomposition based learning scheme,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 4781–4790.
  * [175] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 012–10 022.

Figure 18:  The SISR setting. The task is to super-resolve with a SR model a
given LR image. The resulting SR image is then compared with the target HR
image. The SR model learns to minimize the difference. Figure 19:  Pixel loss
concept. The SR model constructs the SR image and minimizes the pixel loss
function based on the difference between the SR and the HR image.  Figure 20:
GAN based learning objective [49]. It consists of a generator (SR model) and a
discriminator network. The generator approximates HR images while the
discriminator distinguishes between the HR and the approximated image. The
generator’s goal is to fool the discriminator.  Figure 21:  UDL based learning
objective [49]. It is a Dual Network approach with two steps. (1) The SR model
approximates the HR image by learning a mean and a variance using the ESU
loss. (2) The SR model approximates the HR image by predicting a mean and the
fixed variance value learned from step (1) and using the UDL loss.  Figure 22:
Content loss based learning objective. A separate network, e.g. an image
classifier CNN, is used to extract feature maps, which are then used for
content loss. It determines the difference between the feature maps extracted
from the approximated and the ground-truth HR image.

Figure 23:  Visualization of discussed interpolation-based upsampling methods: Nearest-neighbor (left), bilinear (middle), and bicubic Interpolation (right). It shows the spatial space of the image with a blue area and the ground-truth intensity values, which are the points connected to the edges with a solid line. The dashed lines indicate the interpolated intensity values between the ground-truth intensity values. Figure 24:  Visualization of the transposed convolution layer. It expands the given input with zero-padding and applies a convolution to process the features further. Figure 25:  Visualization of the upsampling method Sub-pixel Layer, adapted from Shi et al. [71]. It generates a deep feature map by using convolutional operations. Next, it rearranges the feature map to increase the spatial size while decreasing the channel dimension. Figure 26:  Visualization of network fusion. It uses multiple SR models to generate SR images, which are then combined to produce the final fused SR image.  TABLE I: Benchmark datasets for Super-Resolution (SISR and MISR). Purpose | Dataset | Amount of | Year of
---|---|---|---
| name | images | release
Classic SR Training | BSDS200 [2] | 500 | 2001
T91 [33] | 91 | 2010
General-100 [36] | 100 | 2016
Classic SR Testing | BSDS100 [2] | 100 | 2001
Set14 [1] | 14 | 2010
Set5 [34] | 5 | 2012
Urban100 [32] | 100 | 2015
Manga109 [35] | 109 | 2017
1K images | CelebA-HQ [156] | 30,000 | 2017
Flickr-Faces-HQ [157] | 70,000 | 2019
Flickr1024 [158] | 1024 | 2019
2K images | DIV2K [159] | 900 | 2017
Flickr2K [51] | 2650 | 2017
4K images | UHDSR4K [160] | 8,099 | 2021
8K images | DIV8K [161] | 1,504 | 2019
UHDSR8K [160] | 2,966 | 2021
other SR datasets | BSDS500 [162] | 500 | 2010
KITTI [163] | 3889 | 2012
CASIA Webfaces [164] | 494,414 | 2014
CelebA [165] | 202,599 | 2015
VGGFace2 [166] | 3,310,000 | 2018
PIRM [39] | 200 | 2018
OST300 [167] | 300 | 2018
HIDE [168] | 8,422 | 2019
| SCI1k [169] | 1,000 | 2021
Video SR datasets | VID4 [170] | 4 | 2013
MCL-V [171] | 12 | 2015
GOPRO [172] | 33 | 2017
Vimeo-90k [173] | 90,000 | 2019
RealVSR [174] | 500 | 2021
Figure 27:  Architecture design of RED-Net [105]. It is inspired by the U-Net
[106] architecture and diminishes vanishing gradient effects by using
residuals between the downsampling and upsampling paths.  Figure 28:
Architecture design of SRResNet [14]. It is inspired by ResNet [107] and
applies multiple residual units to propagate information from early layers to
later layers.  Figure 29:  Architecture design of IDN [46]. It splits the
feature maps in the middle part and processes one part further while the other
part is concatened to the residual information path. The notation “c” denotes
concatenation and “s” denotes the split operation.  Figure 30:  Architecture
design of SRFBN [69]. It utilizes FeedBack (FB) blocks that uses multiple
Iterative up-and-down upsamplings with dense connections. For each iteration,
the SRFBN generates a SR image. They are used to train the network with
curriculum learning.  Figure 31:  Architecture design of XLSR [123]. It
applies multi-paths derive different parts of the feature map and 1x1
convolution to combine the features pixel-wise again. The operator “s” denotes
a split operation.  Figure 32:  Architecture design of DSRN [112]. It uses a
dual-state design, where the state-to-state transitions (LR to LR and HR to
HR) are realized with residual unit like paths. To interchange information
between the to spaces, it uses a convolution layer to downsample information
(HR to LR) and a transposed convolution to upsample information (LR to HR).
Figure 33:  Visualization of GAN approaches WESPE [135] and CinCGAN [56].
WESPE learns LR to HR relationship with generator G1, D1 and D2. It also
learns the SR to LR relationship with G2 and content loss. CinCGAN works
similar but uses an cycle-in-cycle adaption of CycleGAN [136]. It learns the
noisy LR to clean LR relationship and the inverse direction. Upon this, it
learns the LR to HR and the HR to noisy LR relationship.

TABLE II: Comparison of upsampling frameworks (pre-, post-, progressive, and iterative up-and-down upsampling). Framework | Pro | Contra
---|---|---
Pre- | •Reduced learning difficulty | •Slow feature extraction
Upsampling | because it only needs to refine | due to spatial space
| •Can handle any random | •High memory consumption
| size input without | due to spatial space
| forfeiting performance |
Post- | •Fast in comparison | •Reduced complexity
Upsampling | to pre-Upsampling | •Harder to train due
|  | to reduced spatial size
|  | of feature extraction
|  | maps
|  | •Not suitable for
|  | multi-scale
Iterative | •Improved performance | •Harder to interpret
Up-and-down | in comparison to | •Harder to train due
Upsampling | pre- and post-upsampling | to parallel learning
| •Learns also the | of up- and downsampling
| degradation relationship |
Progressive | •Improved performance | •Harder to integrate
Upsampling | •Suitable for multi-scale | •Needs modified learning
|  | objective
TABLE III: Comparison (PSNR/SSIM) of SR approaches discussed in this work: Simple, residual, recurrent, attention-based, lightweight and wavelet networks, as well as NAS derived networks and unsupervised trained models. The ordering reflects roughly the performance ranking. However, some comparisons are not fair due to different settings, e.g., NAS vs unsupervised results. Also, some publications do not report results on the commonly used datasets. Therefore, they are not listed (e.g. WESPE [135], DSN [140], CinCGan [56]).  Model | Set5 | Set14 | BSDS100
---|---|---|---
x2 | x3 | x4 | x2 | x3 | x4 | x2 | x3 | x4
| PSNR/SSIM | PSNR/SSIM | PSNR/SSIM | PSNR/SSIM | PSNR/SSIM | PSNR/SSIM | PSNR/SSIM | PSNR/SSIM | PSNR/SSIM
Bicubic | 33.66/0.9542 | 30.39/0.8682 | 28.42/0.8104 | 30.24/0.8688 | 27.55/0.7742 | 26.00/0.7027 | 29.56/0.8431 | 27.21/0.7385 | 25.96/0.6675
DIP [138] | n.a. | n.a. | 28.90/n.a. | n.a. | n.a. | 27.00/n.a. | n.a. | n.a. | n.a.
NAS-DIP [151] | 35.90/n.a. | n.a. | 31.09/n.a. | 31.89/n.a. | n.a. | 28.37/n.a. | n.a. | n.a. | n.a.
SelfExSR [32] | 36.49/0.9537 | 32.58/0.9093 | 30.31/0.8619 | 32.22/0.9034 | 29.16/0.8196 | 27.40/0.7518 | 31.18/0.8855 | 28.29/0.7840 | 26.84/0.7106
SRCNN [42] | 36.66/0.9542 | 32.75/0.9090 | 30.48/0.8628 | 32.42/0.9063 | 29.28/0.8209 | 27.49/0.7503 | 31.36/0.8879 | 28.41/0.7863 | 26.90/0.7101
FSRCNN [36] | 37.00/0.9558 | 33.16/0.9140 | 30.71/0.8657 | 32.63/0.9088 | 29.43/0.8242 | 27.59/0.7535 | 31.51/0.8910 | n.a. | 26.97/0.7140
ESPCN [71] | n.a. | 33.13/n.a. | 30.90/n.a. | n.a. | 29.49/n.a. | 27.73/n.a. | n.a. | n.a. | n.a.
MZSR [142] | 37.25/0.9567 | n.a. | n.a. | n.a. | n.a. | n.a. | 31.64/0.8928 | n.a. | n.a.
ZSSR [92] | 37.37/0.9570 | 33.42/0.9188 | 31.13/0.8796 | 33.00/0.9108 | 29.80/0.8304 | 28.01/0.7651 | 31.65/0.8920 | 28.67/0.7945 | 27.12/0.7211
LapSRN [46] | 37.52/0.9591 | 33.81/0.9220 | 31.54/0.8852 | 32.99/0.9124 | 29.79/0.8325 | 28.09/0.7700 | 31.80/0.8952 | 28.82/0.7980 | 27.32/0.7275
VDSR [101] | 37.53/0.9587 | 33.66/0.9213 | 31.35/0.8838 | 33.03/0.9124 | 29.77/0.8314 | 28.01/0.7674 | 31.90/0.8960 | 28.82/0.7976 | 27.29/0.7251
DWSR [128] | 37.43/0.9568 | 33.82/0.9215 | 31.39/0.8833 | 33.07/0.9106 | 29.83/0.8308 | 28.04/0.7669 | 31.80/0.8940 | n.a. | 27.25/0.7240
DRCN [109] | 37.63/0.9588 | 33.82/0.9226 | 31.53/0.8854 | 33.04/0.9118 | 29.76/0.8311 | 28.02/0.7670 | 31.85/0.8942 | 28.80/0.7963 | 28.23/0.7233
MoreMNAS [10] | 37.57/0.9584 | n.a. | n.a. | 33.25/0.9142 | n.a. | n.a. | 31.94/0.8966 | n.a. | n.a.
RED [105] | 37.66/0.9599 | 33.82/0.9230 | 31.51/0.8869 | 32.94/0.9144 | 29.61/0.8341 | 27.86/0.7718 | 31.99/0.8974 | 28.93/0.7994 | 27.40/0.7290
DSRN [112] | 37.66/0.9590 | 33.88/0.9220 | 31.40/0.8830 | 33.15/0.9130 | 30.26/0.8370 | 28.07/0.7700 | 32.10/0.8970 | 28.81/0.7970 | 27.25/0.7240
DRRN [70] | 37.74/0.9591 | 34.03/0.9244 | 31.68/0.8888 | 33.23/0.9136 | 29.96/0.8349 | 28.21/0.7720 | 32.05/0.8973 | 28.95/0.8004 | 27.38/0.7284
CARN-M [118] | 37.53/0.9583 | 33.99/0.9236 | 31.92/0.8903 | 33.26/0.9141 | 30.08/0.8367 | 28.42/0.7762 | 31.92/0.8960 | 28.91/0.8000 | 27.44/0.7304
MemNet [97] | 37.78/0.9597 | 34.09/0.9248 | 31.74/0.8893 | 33.28/0.9142 | 30.00/0.8350 | 28.26/0.7723 | 32.08/0.8978 | 28.96/0.8001 | 27.40/0.7281
IDN [46] | 37.83/0.9600 | 34.11/0.9253 | 31.82/0.8903 | 33.30/0.9148 | 29.99/0.8354 | 28.25/0.7730 | 32.08/0.8985 | 28.95/0.8013 | 27.41/0.7297
FALSR [147] | 37.82/0.9595 | n.a. | n.a. | 33.55/0.9168 | n.a. | n.a. | 32.12/0.8987 | n.a. | n.a.
MWCNN [131] | 37.91/0.9600 | 34.17/0.9271 | 32.12/0.8941 | 33.70/0.9182 | 30.16/0.8414 | 28.41/0.7816 | 32.23/0.8999 | 29.12/0.8060 | 27.62/0.7355
NLRN [115] | 38.00/0.9603 | 34.27/0.9266 | 31.92/0.8916 | 33.46/0.9159 | 30.16/0.8374 | 28.36/0.7745 | 32.19/0.8992 | 29.06/0.8026 | 27.48/0.7306
CARN [118] | 37.76/0.9590 | 34.29/0.9255 | 32.13/0.8937 | 33.52/0.9166 | 30.29/0.8407 | 28.60/0.7806 | 32.09/0.8978 | 29.06/0.8034 | 27.58/0.7349
IMDN [122] | 38.00/0.9605 | 34.36/0.9270 | 32.21/0.8948 | 33.63/0.9177 | 30.32/0.8417 | 28.58/0.7811 | 32.19/0.8996 | 29.09/0.8046 | 27.56/0.7353
RFDN [120] | 38.05/0.9606 | 34.41/0.9273 | 32.24/0.8952 | 33.68/0.9184 | 30.34/0.8420 | 28.61/0.7819 | 32.16/0.8994 | 29.09/0.8050 | 27.57/0.7360
RFDN-L [120] | 38.08/0.9606 | 34.47/0.9280 | 32.28/0.8957 | 33.67/0.9190 | 30.35/0.8421 | 28.61/0.7818 | 32.18/0.8996 | 29.11/0.8053 | 27.58/0.7363
DeCoNAS [148] | 37.96/0.9594 | n.a. | n.a. | 33.63/0.9175 | n.a. | n.a. | 32.15/0.8986 | n.a. | n.a.
HNAS [150] | 38.11/0.9640 | n.a. | n.a. | 33.60/0.9200 | n.a. | n.a. | 32.17/0.9020 | n.a. | n.a.
ESRN [144] | 38.04/0.9607 | 34.46/0.9281 | 32.26/0.8957 | 33.71/0.9185 | 30.43/0.8439 | 28.63/0.7818 | 32.23/0.9005 | 29.15/0.8072 | 27.62/0.7378
SRFBN [69] | 38.11/0.9609 | 34.70/0.9292 | 32.47/0.8983 | 33.82/0.9196 | 30.51/0.8461 | 28.81/0.7868 | 32.29/0.9010 | 29.24/0.8084 | 27.72/0.7409
SRFBN+ [69] | 38.18/0.9611 | 34.77/0.9297 | 32.56/0.8992 | 33.90/0.9203 | 30.61/0.8473 | 28.87/0.7881 | 32.34/0.9015 | 29.29/0.8093 | 27.77/0.7419
MDSR [41] | 38.17/0.9605 | 34.77/0.9288 | 32.60/0.8982 | 33.92/0.9203 | 30.53/0.8465 | 28.82/0.7876 | 32.34/0.9014 | 29.30/0.8101 | 27.78/0.7425
EDSR [41] | 38.20/0.9606 | 34.76/0.9290 | 32.62/0.8984 | 34.02/0.9204 | 30.66/0.8481 | 28.94/0.7901 | 32.37/0.9018 | 29.32/0.8104 | 27.79/0.7437
WRAN [133] | 38.32/0.9630 | 34.79/0.9310 | 32.36/0.8990 | 34.21/0.9220 | 30.71/0.8520 | 28.60/0.7860 | 32.57/0.9070 | 29.36/0.8190 | 27.71/0.7420
DRLN [6] | 38.27/0.9616 | 34.78/0.9303 | 32.63/0.9002 | 34.28/0.9231 | 30.73/0.8488 | 28.94/0.7900 | 32.44/0.9028 | 29.36/0.8117 | 27.83/0.7444
HAN [87] | 38.33/0.9617 | 34.85/0.9305 | 32.75/0.9016 | 34.24/0.9224 | 30.77/0.8495 | 28.99/0.7907 | 32.45/0.9030 | 29.39/0.8120 | 27.85/0.7454
DRLN+ [6] | 38.34/0.9619 | 34.86/0.9307 | 32.74/0.9013 | 34.43/0.9247 | 30.80/0.8498 | 29.02/0.7914 | 32.47/0.9032 | 29.40/0.8125 | 27.87/0.7453
SwinIR [175] | 38.46/0.9624 | 35.04/0.9322 | 32.93/0.9043 | 33.07/0.9106 | 31.00/0.8542 | 29.15/0.7958 | 31.80/0.8940 | 29.49/0.8150 | 27.95/0.7494
WIDN [129] | 39.60/0.9830 | 34.48/0.9430 | 32.85/0.9290 | 34.44/0.9800 | 30.95/0.9310 | 29.75/0.9090 | 33.52/0.9790 | 29.99/0.9280 | 28.10/0.9100
CAR (EDSR) [7] | 38.94/0.9658 | n.a. | 33.88/0.9174 | 35.61/0.9404 | n.a. | 30.31/0.8382 | 33.83/0.9262 | n.a. | 29.15/0.8001