# Visibility-Aware Pixelwise View Selection for Multi-View Stereo Matching

Zhentao Huang
University of Guelph
<EMAIL_ADDRESS>Yukun Shi
University of Guelph
<EMAIL_ADDRESS>Minglun Gong
University of Guelph
<EMAIL_ADDRESS>
###### Abstract

The performance of PatchMatch-based multi-view stereo algorithms depends
heavily on the source views selected for computing matching costs. Instead of
modeling the visibility of different views, most existing approaches handle
occlusions in an ad-hoc manner. To address this issue, we propose a novel
visibility-guided pixelwise view selection scheme in this paper. It
progressively refines the set of source views to be used for each pixel in the
reference view based on visibility information provided by already validated
solutions. In addition, the Artificial Multi-Bee Colony (AMBC) algorithm is
employed to search for optimal solutions for different pixels in parallel.
Inter-colony communication is performed both within the same image and among
different images. Fitness rewards are added to validated and propagated
solutions, effectively enforcing the smoothness of neighboring pixels and
allowing better handling of textureless areas. Experimental results on the DTU
dataset show our method achieves state-of-the-art performance among non-
learning-based methods and retrieves more details in occluded and low-textured
regions.

##  1 Introduction

Multi-view stereo (MVS), which estimates dense 3D point clouds from a set of
calibrated input images, is an important research topic and supports many
downstream applications, such as autonomous driving, 3D reconstruction, and
virtual reality. Even though much progress has been made in recent years [7,
28, 38, 4], reconstructing accurate and complete 3D point cloud models remains
challenging due to obstacles caused by low texture, reflections, occlusions,
and repetitive patterns.

Inspired by the success of MVSNet [47], numerous learning-based methods [21,
48, 10, 39, 26] had been proposed in recent years and shown outstanding
performances. They had been ranked on the top of various MVS datasets [1, 49,
17]. However, it is questionable how well these learning-based methods can
adapt to scenes underrepresented in the training dataset and how much time
these models take for training and fine-tuning on a new dataset.

Recently, PatchMatch-based methods [8, 44, 45, 28] also show excellent
capability in depth map estimation. Following [3], these methods generally
have a four-step pipeline: random initialization, propagation, view selection,
and refinement. View selection is an important factor here because correct
matches can only be found from nearby unoccluded views, and occlusions are
common under the MVS setting. Yet, existing approaches often resort to ad-hoc
view selection methods (_e.g_. top-n views with the lowest matching cost [8])
without considering visibility constraints. Therefore, a motivating question
is whether we can make the view selection process visibility-aware and how
much benefit we can gain from such enhancement.

To this end, we develop a pixelwise view selection approach, which
progressively updates the source views used for each pixel. The selected views
will be used for both matching cost calculation and depth/normal consistency
check, which leads to a set of validated solutions. These validated solutions
are used to guide future view selections through visibility checks; see Figure
1.

Even with a proper set of source views selected, searching the optimal depth
and normal for each pixel is still a challenging problem. We address this
issue using three strategies: 1) A swarm-based optimization framework (the
Artificial Multi-Bee Colony algorithm [41]) is utilized to avoid being trapped
into local optimum; 2) Both intra-image and inter-image solution propagation
are employed to speed up the convergence; 3) A smoothness term is added into
intra-image propagation to handle low texture areas better.

Figure 1: Overview of the proposed algorithm, which iteratively performs view
selection, normal and depth optimization, and geometry consistency check. Once
the process converges, 3D point clouds estimated from different views are
fused together to produce the final model.

##  2 Related Work

Related works in the MVS field can be mainly divided into two aspects,
learning-based and non-learning-based (traditional MVS).

#### Learning-based Shape Reconstruction.

The huge success of deep neural networks in image classification has sparked
interest in introducing learning to other domains. For the 3D shape
reconstruction problem, a natural path for applying the standard convolutional
neural network (CNN) architecture is to estimate a 2.5D depth map for each
input image or to operate on 3D voxels. Depth map based approaches are
proposed to infer per-pixel depth either from single image[20] or to use
visual cues extracted from stereo pairs [9, 50]. Additional processing is then
needed to consolidate multiple depth maps into a single 3D point cloud[43,
22]. Voxel-based methods[6, 36] utilize 3D convolution operators to encode and
decode geometric features in discretized 3D space directly but are limited to
the relatively low voxel resolution. To address these limitations, 3D point
cloud based[37, 46] and implicit surface based[5, 24, 32] approaches are also
proposed. Latter research finds that implicit surfaces can be approximated by
fully connected networks[25], which leads to reconstruction using neural
implicit representation[42].

#### Non-learning-based Multi-view Stereo.

Non-learning-based MVS computes the matching cost of rectified image patches
by different methods, such as the Sum of Absolute Differences (SAD), Sum of
Squared Distances (SSD), or Normalized Cross-Correlation (NCC). According
to[29], generally, there are four types of methods used to represent scene:
volumetric based[18, 30], point cloud based[7, 19], mesh based[34, 35], and
depth map based[8, 44]. Recently, due to parallel ability and high
performance, PatchMatch[8, 44] is widely used in this field. The core idea of
Patchmatch[2] is to minimize the matching cost of every pixel and the plane in
disparity space and then effectively estimate depth maps for every image. [28]
estimates depth maps and pixelwise view selection jointly.

#### Depth Map Fusion.

There are two main types of Multi-view Stereo approaches categorized according
to the scene representation: volumetric-based methods and depth map-based
methods [40]. The depth map representation provides pixel-wise depth
information for every viewpoint. The 3D reconstruction represented by the
point cloud can be recovered by applying 3D fusion techniques to all the depth
maps. Unlike depth map-based approaches, which construct the scene indirectly,
volumetric representation depicts the scene based on volume occupancy in 3D
space. [31] proposed a graph cuts-based reconstruction that generates multi-
resolution volumetric meshes.

#### Artificial Multi-Bee-Colony Algorithm.

Artificial Bee Colony (ABC) algorithm[15] and its variants[51] are used to
handle constrained and unconstrained optimization problems[14]. Compared with
other population-based algorithms, the ABC algorithm can achieve equal or
better performance with fewer parameters. Thus, the ABC algorithm has been
used in many applications such as feature selection[11] and data
clustering[33].

Artificial Multi-Bee-Colony (AMBC) algorithm[41] searches k-nearest matches
via dedicated bee colonies. The communication in colonies can propagate proper
matches to escape local optima. AMBC is also able for parallel processing as
it makes no assumption about the neighbor or direction.

Our work leverages the idea proposed by AMBC to build the MVS framework. We
not only apply the between-colony communication idea for propagating solutions
between different pixels of the same image but also for propagation among
different images. In addition, rewards are added to validated intra-image
solution propagation, which allows simple yet effective enforcement of
smoothness constraint.

##  3 Overview

Given a set of input 2D images $I=\\{I_{i}|i=1\cdots N\\}$ with known camera
parameters $C=\\{C_{i}|i=1\cdots N\\}$, the goal of MVS is to estimate pixel-
wise depth maps $d=\\{d_{i}|i=1\cdots N\\}$ for every view and fuse them into
a 3D point cloud. Specifically, when processing a reference image
$I_{\mathit{ref}}$, MVS algorithms normally estimate a local fitting plane $P$
for each pixel $x$ in $I_{\mathit{ref}}$’s local coordinates, using some of
the remaining views as source images $I_{src}\in\\{I\\}-I_{\mathit{ref}}$. The
plane $P$ depicts both the depth and normal information of the local geometry,
which are denoted as $d_{\mathit{ref}}(x)$ and $\vec{n}_{\mathit{ref}}(x)$,
respectively.

Figure 1 illustrates an overview of our method. We construct a three-phased
process that evolves over cycles. An initial set of source views is selected
for each pixel in each reference image based on only camera parameters $C$.
These source views are used to compute matching costs, based on which an AMBC
algorithm is used to search for the optimal solution (depth and normal) for
each pixel. A geometry consistency check is then performed by projecting the
optimal solution found for a given pixel to all of its source views. The
solution is considered validated if both depth and normal are consistent with
the corresponding pixels in these source views. Validated solutions are
further used for: 1) guiding the future source view selection process as some
source views may be determined as occluded; 2) communicating both among pixels
in the same image and among different images; and 3) fusing into the final
point cloud.

##  4 Pixelwise View Selection

The selection of source views for matching cost calculation strongly impacts
the quality of reconstruction results. Previous work has proposed to use
triangulation angle, incident angle, and image resolution-based geometric
priors to perform pixelwise view selection [28]. While we acknowledge the
importance of resolution-based geometric prior in handling large-scale scenes,
its benefit for reconstructing a small set of objects is limited since all
input images have similar resolutions. Hence, we removed this term in our
implementation for simplicity. Instead, we add a visibility-based term, which
handles occlusions based on geometric information instead of heuristics.

The actual terms used for view selection are based on available information.
At the beginning of the process, we have yet to gain prior knowledge of scene
geometry. Therefore, when processing image $I_{i}$ as the reference view, only
the triangulation angle term is used for view selection. All nearby views
whose triangulation angle with $I_{i}$ is between $[10^{\circ},30^{\circ}]$
are selected into the source view set $\\{I_{src}\\}$; see Figure 2(a). It is
worth noting that the same set of views is used for all pixels in $I_{i}$.

Once the depth $d_{i}(x)$ and normal $\vec{n}_{i}(x)$ for each pixel $x$ in
$I_{i}$ are estimated, the incident angle term will be used. If a given view
$I_{j}$ has a poor incident angle, _i.e_. the angle between $\vec{n}_{i}(x)$
and the viewing vector of $I_{j}$ is greater than $80^{\circ}$, $I_{j}$ will
be removed from the source view set $\\{I_{src}\\}$; see Figure 2(b). As a
result, the set $\\{I_{src}\\}$ will be adaptively determined for different
pixels in $I_{i}$.

Finally, once validated depth and normal are found for different views
(details on solution validation will be discussed in Sec. 6), the visibility
term will be introduced. That is, for a given pixel $x$ in reference view
$I_{i}$, we will first backproject $x$ to a 3D scene point $X$ using the
estimated depth $d_{i}(x)$. The 3D point $X$ is then projected to each view
$I_{j}$ in set $\\{I_{src}\\}$. Without losing generality, here we assume the
projection of $X$ on image $I_{j}$ is pixel $y$. We consider the $X$ is
occluded in $I_{j}$ if and only if a validated solution is found at pixel $y$
and the depth $d_{j}(y)$ is smaller than the distance to 3D point $X$. $I_{j}$
will be removed from source view set $\\{I_{src}\\}$ if $X$ is occluded in
$I_{j}$; see Figure 2(c).

Figure 2: Illustration for the progressive view selection process: (a)
initially, only source views with poor triangulation angle ($C_{5}$) will be
filtered out from the set; (b) once the normal for $x$ is estimated, views
with poor incident angles ($C_{1}$) will be removed from the source view set;
and (c) validated solutions are used to further remove occluded views
($C_{4}$) from the source view set.

(a) COLMAP [28]

(b) Gipuma [8]

(c) Ours

(d) Ground Truth

Figure 3: Comparison of the reconstructed point cloud on DTU dataset. Our
approach outperform in less-textured regions, thanks to the added smoothness
reward term. Figure 4: An illustration on the effect of Pixelwise View
Selection (PVS). From left to right: input image, normal map without/with PVS,
source view set map without/with PVS (brightness indicates the number of views
in the source view set), depth map without/with PVS. Challenging areas are
marked in red boxes.

##  5 Artificial Multi-Bee Colony Algorithm

The original Artificial Bee Colony algorithm is a bio-inspired optimization
approach proposed by Karaboga [15]. Wang _et al_. [41] later employ multiple
bee colonies to optimize solutions for different pixels and use inter-colony
scout bees to facilitate solution propagation.

In MVS context, each solution has four parameters: three for the normal vector
and one for the depth value. By representing each solution as a food source,
the optimal results are searched by sending out three kinds of bees: employed
bees, onlooker bees, and scout bees. The parameterization is conducted in the
Euclidean scene space as Gipuma [8] does. Compared to the disparity space,
parameterization in scene space avoids the epipolar rectification, and it
could generate dense normals in the scene, which could be used for further
point cloud fusion [16].

In the Euclidean scene space, the plane equation for 3D object points
$\textbf{X}=[X,Y,Z]^{\top}$ would be $\vec{n}^{\top}\textbf{X}=-D$, where
$\vec{n}$ is the normal vector and $D$ is the distance to the origin. By
placing the reference camera at the origin, the depth $d$ at the pixel $x$
could be inferred with the plane parameters and the camera intrinsic
parameters:

$d=\frac{Df_{x}}{[x-u,f_{x}/f_{y}(y-v),f_{x}]\cdot\vec{n}}$ (1)

In this equation, $c_{x},c_{y}$ represent the optical center, and
$f_{x},f_{y}$ represent the focal length of the camera in pixels which are
parameters from camera intrinsic matrix $K$. Under this context, each solution
contains four parameters: three for $n$ and one for $d$. Then the pixel $x$ in
the reference image $K[I|0]$ is related to the corresponding point
$x^{\prime}$ in the source image $K^{\prime}[R|t]$ based on the plane-induced
homography [12]:

$H_{\pi}=K^{\prime}(R-\frac{1}{D}t\vec{n}^{\top})K^{-1}$ (2)

Figure 5: Left: red-black checkerboard pattern. Each block represents a pixel
in the image. The pixels in the same color are processed parallelly. Right:
the sample space (red) of a center pixel (black).

###  5.1 Random Initialization

We first randomly generate the same number of hypotheses as preset
$\mathit{FoodNumber}$ for each pixel. Each hypothesis contains three
parameters for the normal vector $\vec{n}$, and one parameter for plane depth
$D$. We follow [23] to uniformly sample the normal vector over the visible
hemisphere. Note that since plane depth $D$ is used here, the solutions of
pixels on the same plane will have the same set of parameters. Scene depth $d$
could be obtained via equation 1 when needed.

###  5.2 Matching Cost Evaluation

The similarity between two patches related via plane-induced homography
defines whether the hypothesis depicts the scene correctly. In the two-view
stereo, the matching cost is straightforward. When extending to multi-view
stereo, we adopt the following equation to aggregate the matching cost of
hypothesis $\langle d_{i}(x),\vec{n}_{i}(x)\rangle$ in pixel $x$ of the
reference image $i$:

$C(d_{i}(x),\vec{n}_{i}(x))=\begin{dcases}\frac{\sum\limits_{j\in\\{S_{src}\\}}m(i,j)}{|\\{S_{src}\\}|-1},&\text{if
}|\\{S_{src}\\}|>1\\\ +\infty,&\text{otherwise}\end{dcases}$ (3)

where $m(i,j)$ represents the matching cost between two patches from reference
view $i$ and source view $j$, and $\\{S_{src}\\}$ is the set of suitable
source views for pixel $x$. In this paper, we adopt bilaterally weighted
Normalized Cross-Correlation [28] as our matching cost function.

In Equation 3, the aggregation cost is divided by $|\\{S_{src}\\}|-1$ rather
than $|\\{S_{src}\\}|$ because we expect the results to be an unbiased sample
estimate that prefers a larger $\\{S_{src}\\}$ set. Which is more likely to
capture the true normal and depth information than a small set where only
several views produce the best results and the rest are invalid (_e.g_. out of
image boundaries).

In addition, we adopt a fitness value to each plane hypothesis:

$F(d_{i}(x),\vec{n}_{i}(x))=\frac{1}{1+C(d_{i}(x),\vec{n}_{i}(x))}$ (4)

The fitness for a hypothesis is higher if the aggregation matching cost is
lower. The trial count $T(d_{i}(x),\vec{n}_{i}(x))$ is added and set to zero
for all food sources in the initialization. It is designed to track whether
each food source is updated through iterations.

###  5.3 Employed Bees

The task for employed bees is to perform a search within the local colonies by
randomly perturbing each food source. For each food source, the perturbed food
source $y^{\prime}_{x}$ is generated by:

$y^{\prime}_{x}=y_{x}+R(-1,1)(y_{n}-y_{x})$ (5)

where $R(-1,1)$ returns a value uniformly distributed between -1 and 1, and
$y_{n}$ is another food source randomly selected within the colony. The
fitness of the perturbed hypothesis is then evaluated based on the combination
of Equation 3 and 4. If the fitness of perturbed food source
$F(y^{\prime}_{x})$ is greater than $F(y_{x})$, then $y_{x}$ is replaced by
$y^{\prime}_{x}$. Otherwise, set the trial count $T(x)=T(x)+1$.

###  5.4 Onlooker Bees

The task for onlooker bees is to perform searching in neighboring colonies.
Following [8], we adopt a red-black checkerboard pattern for sampling. It
divides the image into red and black groups. Pixels in the same color group
can be processed in parallel without interfering with others. Figure 5 shows a
center pixel’s pattern and sample space.

For every food source $y_{x}$ with a black label, randomly select a colony
with a red label following the pattern in Figure 5, and vice versa. Then
select the food source $y_{n}$ with the highest fitness value, and evaluate
the fitness of the food source $F(y_{n})$ in the current colony. If $F(y_{n})$
is greater than $F(y_{x})$, then $y_{x}$ is replaced by $y_{n}$. Otherwise,
set the trial count $T(y_{x})=T(y_{x})+1$.

###  5.5 Scout Bees

After several iterations after the initialization, the colony’s food sources
will converge to a relatively small range. Since employed bees and onlooker
bees only perturb or copy the existing solutions, the task for scout bees is
to perform global searching and avoid potential local optimum. In this paper,
we only want to perform the global searching for $\mathit{FoodNumber}-1$ food
sources. The reason is that the food source with the highest fitness value
should always be kept in the colony. Therefore, for the rest of the food
source, if their trial count $T(y_{x})$ exceeds a preset threshold (it is
empirically set to 10), then it is replaced by a randomly generated food
source. The idea behind this is that when a food source has not been updated
for certain iterations and is not the best in the colony, it may have been
stuck in the local optimum. The new food source is evaluated via Equation 4,
and the new trial count is set to 0.

##  6 Geometric Consistency Check

Due to noise and/or deviation from Lambertian property, mismatches sometimes
have lower matching costs (or higher fitness scores) than correct matches. To
filter out these mismatches, the consistency check is often applied. As shown
in Figure 1, our approach alternates between the depth/normal estimation stage
and the consistency check stage until the whole process converges. That is,
once normal and depth map calculation is completed for all views, the
algorithm will cross-check the obtained depth/normal among these views. The
solutions that pass the check will be marked as validated for the next
calculation cycle. It is worth noting that validated and unvalidated solutions
are continued to be refined in future optimizations. Unlike unvalidated
solutions, validated ones are used for: 1) enforcing smoothness constraint
during intra-image solution propagation; 2) propagating solutions among
different views; 3) providing visibility information during future source view
selection.

Ideally, if a solution $\langle d_{i}(x),\vec{n}_{i}(x)\rangle$ for pixel $x$
in image $I_{i}$ is correct, it should be consistent with all corresponding
solutions in the source views used for pixel $x$. That is, after applying
homography transform $H^{ij}$ between view $I_{i}$ and one of the source view
$I_{j}$ for pixel $x$, we should have $H^{ij}(d_{i}(x))=d_{j}(H^{ij}(x))$ and
$H^{ij}(\vec{n}_{i}(x))=\vec{n}_{j}(H^{ij}(x))$. In practice, we consider
solution $\langle d_{i}(x),\vec{n}_{i}(x)\rangle$ is consistent with source
view $I_{j}$ if the following two criteria hold:

$\displaystyle|H^{ij}(d_{i}(x))-d_{j}(H^{ij}(x))|$ $\displaystyle<$
$\displaystyle{\mathit{T}}_{depth},$ (6)
$\displaystyle\arccos(H^{ij}(\vec{n}_{i}(x))\cdot\vec{n}_{j}(H^{ij}(x)))$
$\displaystyle<$ $\displaystyle{\mathit{T}}_{normal}$

where ${\mathit{T}}_{depth}$ and ${\mathit{T}}_{normal}$ are two preset
thresholds (it is set to 0.01 mm and 30 degrees in practice). In addition,
instead of requiring all source views used for pixel $x$ to be consistent with
$\langle d_{i}(x),\vec{n}_{i}(x)\rangle$, we relax the constraint by allowing
a small percentage of these views do not satisfy the above conditions. That
is, $\langle d_{i}(x),\vec{n}_{i}(x)\rangle$ is labeled as verified if it is
consistent with 70% or more of the source views $\\{I_{src}\\}$ selected for
pixel $x$.

###  6.1 Propagation Between Views

Initially, solution propagation only appears in neighboring colonies within
the same image (referred as intra-image propagation) via onlooker bees. It is
one of the vital concepts in PatchMatch-based multi-view stereo. In this
paper, we propose inter-image propagation as well, which enables the solution
to propagate between pixels in different images related by consistency check.
Figure 6 shows an example of propagation between views. The key concept is to
propagate each solution that is consistent with at least one other source view
to views that do not have validated solutions at the corresponding locations.
This helps to speed up the convergence and prevent potential local optimum.

###  6.2 Smoothness Constraint

Not being able to handle textureless areas properly is a significant
limitation for the PatchMatch-based methods [27]. In binocular stereo, this
problem is often addressed by introducing an additional smoothness term, which
converts per-pixel optimization into global optimization. Solving global
optimization under the MVS setting can be highly computationally expensive.
Hence, we applied a simple yet effective approach, which adds fitness reward
to solutions propagated by onlooker bees. The key observation is that the
fitness values for the correct match and mismatches are similar in textureless
areas. Adding a small reward to the validated solutions propagated by onlooker
bees effectively encourages the same fitting plane being selected at the
current solution. The smoothness is therefore enforced, and flat textureless
surfaces can be properly modeled. For areas with distinct textures, the small
reward will not affect the search for optimal solutions.

##  7 Fusion

Follows [8, 28], after obtaining all the depth and normal maps, we fuse them
into a single point cloud. More specifically, for $N$ images in the scene, we
consequently select each image as the reference image and convert its depth
map to 3D points in the world coordinate, then project them to the rest $N-1$
views. If the relative depth difference is less than 0.01 mm, and the angle
between normals is smaller than 30 degrees, then it is counted as a consistent
view. If there exist more than three consistent views, then the point will be
accepted in the result. Finally, the points that are related by the depth and
normal estimates are averaged into a 3D point in the result point cloud.

Method |  Acc.(mm) |  Comp.(mm) |  Overall
---|---|---|---
Furukawa [7] |  0.605 |  0.842 |  0.724
Tola [38] |  0.307 |  1.097 |  0.702
COLMAP [28] |  0.400 |  0.532 |  0.664
Campbell [4] |  0.753 |  0.540 |  0.647
Gipuma [8] |  0.273 |  0.687 |  0.480
Ours |  0.385 |  0.388 |  0.386
Table 1: Quantitative results for non-learning-based approaches on full DTU dataset. Lower is better. Our method ranks first in terms of Completeness and Overall metrics.  Method |  Acc.(mm) |  Comp.(mm) |  Overall
---|---|---|---
Non-Learning-based |  |  |
Furukawa [7] |  0.613 |  0.941 |  0.777
Tola [38] |  0.342 |  1.190 |  0.766
Campbell [4] |  0.835 |  0.554 |  0.695
Gipuma [8] |  0.283 |  0.873 |  0.578
COLMAP [28] |  0.411 |  0.657 |  0.534
Ours |  0.405 |  0.381 |  0.393
Learning-based |  |  |
SurfaceNet [13] |  0.450 |  1.040 |  0.745
MVSNet [47] |  0.396 |  0.527 |  0.462
P-MVSNet [21] |  0.406 |  0.434 |  0.420
R-MVSNet [48] |  0.383 |  0.452 |  0.417
CasMVSNet [10] |  0.325 |  0.385 |  0.355
PatchMatchNet [39] |  0.427 |  0.277 |  0.352
UniMVSNet [26] |  0.352 |  0.278 |  0.315
Table 2: Quantitative results on DTU evaluation set. Both learning-based and
non-learning-based approaches are listed for impartial comparison.

##  8 Experiments

We evaluate our method on the DTU Robot Image dataset [1]. In this section, we
present the dataset details, evaluation results, ablation study for key
components, and implementation details. Noted that $\mathit{FoodNum}$ is set
to 10 in our experiments.

###  8.1 DTU Robot Image Dataset

As our main testing dataset, the DTU dataset contains 124 different scenes
captured by a structured light scanner mounted on an industrial robot arm.
Each scene has been taken from 49 or 64 positions with seven different
lighting conditions. In this paper, we select the most diffuse set. The image
resolution is $1600\times 1200$, and the camera calibration parameters are
provided.

Figure 6: Illustration of the Propagation Between Views. The solution in
reference view $C_{r}$ is consistent with two of its neighboring views:
$C_{2},C_{3}$. While the rest of the two are not consistent with $C_{r}$. Then
the solution will be propagated to $x_{1},x_{4}$ to check whether it is a
better solution to replace existing ones.  Figure 7: Visualization results of
the proposed method in scene 11, 95 and 100 in DTU dataset. From left to
right: Tola [38], Furukawa [7], Campbell [4], point cloud without Smoothness
Constraint, point cloud with Smoothness Constraint, ground truth. Figure 8:
Illustration on the effect of the Smoothness Constraint (SC) and Pixelwise
View Selection (PVS). From top to bottom: scene 19, 59, 100 from the DTU
dataset. From left to right: input image, normal map without SC & PVS, normal
map with SC, normal map with SC & PVS, depth map without SC & PVS, depth map
with SC, depth map with SC & PVS. Challenging areas are marked in red boxes.

###  8.2 Point Cloud Evaluation

Accuracy is measured as the distance from the reconstruction point cloud to
the structured light reference. Completeness is measured from the structured
light reference to the reconstruction point cloud. The overall score is
computed by averaging the accuracy and completeness. Note that the
reconstruction point cloud is downsampled to ensure unbiased evaluation since
strongly textured regions generally have dense 3D points. The evaluation
program is provided by the authors.

For this dataset, we present two versions of quantitative results: one for
comparison of the non-learning-based methods and the other for comparison of
both the learning-based and non-learning-based methods. For non-learning-based
method evaluation, we follow the protocol specified by the authors of the
dataset by testing 80 different scenes. For Learning-based method evaluation,
we follow [47] by using the validation set containing 22 different scenes for
impartial comparison. In Table 1, we present the quantitative results for non-
learning-based approaches on the full DTU dataset. Our approach performs the
best in both completeness and overall metrics. In Table 2, we present the
quantitative results for both learning-based and non-learning-based methods on
the DTU evaluation set. Our method shows competitive performance to the
learning-based methods.

###  8.3 Ablation Study

We here present the ablation study of the proposed method. The baseline method
is generated using AMBC with all three types of bees: employed bees, onlooker
bees, and scout bees. Our proposed baseline method with smoothness constraint,
pixelwise view selection, and propagation between views shows the best
performance. Note that propagation between views only speeds up the
convergence because it does not modify the solution space. Therefore it does
not have a significant impact on result accuracy.

Figure 4 demonstrates the effect of the pixelwise view selection in occluded
regions. The pixel-wise view selection could handle the occluded regions and
generate correct normal and depth estimation. Figure 7 shows a comparison of
the point clouds on scene 11, 95, and 100 in the DTU dataset. The proposed
method with smoothness constraint clearly outperforms in low-textured areas.
In addition, only a few noisy/incorrect points are introduced along the edges
of the surfaces. Figure 8 illustrates the effect of the combination of
smoothness constraint and pixelwise view selection. The best results are
obtained by applying both of them. The numerical results of the ablation study
are provided in Supplementary Materials.

##  9 Conclusion

In this paper, we present a visibility-aware pixelwise view selection method
for PatchMatch-based multi-view stereo. View selection is progressively
improved for individual pixels as more knowledge of scene geometry is
obtained. Selected views are used for both matching cost evaluation and
consistency check. When applying Artificial Multi-Bee Colony (AMBC) to search
optimal solutions for different pixels in parallel, between-colony onlooker
bees are used for intra-image and inter-image solution propagation. To tackle
the lack of photometric cues at low-textured regions, fitness rewards are
introduced on those solutions verified by consistency check. Experiments on
the DTU dataset demonstrate that our method achieves state-of-the-art
performance among the non-learning-based methods. Ablation study shows that
our two main components, visibility-aware pixelwise view selection, and
smoothness rewards, can notably improve the handling of occluded and low-
textured areas. The source code will be released after the acceptance of the
paper.

## References

  * [1] Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl.  Large-scale data for multiple-view stereopsis.  International Journal of Computer Vision, pages 1–16, 2016.
  * [2] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman.  Patchmatch: A randomized correspondence algorithm for structural image editing.  ACM Trans. Graph., 28(3):24, 2009.
  * [3] Michael Bleyer, Christoph Rhemann, and Carsten Rother.  Patchmatch stereo-stereo matching with slanted support windows.  In The British Machine Vision Conference (BMVC), volume 11, pages 1–11, 2011.
  * [4] Neill DF Campbell, George Vogiatzis, Carlos Hernández, and Roberto Cipolla.  Using multiple hypotheses to improve depth-maps for multi-view stereo.  In European Conference on Computer Vision, pages 766–779. Springer, 2008.
  * [5] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939–5948, 2019.
  * [6] Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese.  3d-r2n2: A unified approach for single and multi-view 3d object reconstruction.  In European Conference on Computer Vision, pages 628–644. Springer, 2016.
  * [7] Yasutaka Furukawa and Jean Ponce.  Accurate, dense, and robust multiview stereopsis.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(8):1362–1376, 2009.
  * [8] Silvano Galliani, Katrin Lasinger, and Konrad Schindler.  Massively parallel multiview stereopsis by surface normal diffusion.  In Proceedings of the IEEE International Conference on Computer Vision, pages 873–881, 2015.
  * [9] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 270–279, 2017.
  * [10] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan.  Cascade cost volume for high-resolution multi-view stereo and stereo matching.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2495–2504, 2020.
  * [11] Emrah Hancer, Bing Xue, Dervis Karaboga, and Mengjie Zhang.  A binary abc algorithm based on advanced similarity scheme for feature selection.  Applied Soft Computing, 36:334–348, 2015.
  * [12] Richard Hartley and Andrew Zisserman.  Multiple view geometry in computer vision.  Cambridge university press, 2003.
  * [13] Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang.  Surfacenet: An end-to-end 3d neural network for multiview stereopsis.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2307–2315, 2017.
  * [14] Dervis Karaboga and Bahriye Basturk.  Artificial bee colony (abc) optimization algorithm for solving constrained optimization problems.  In International Fuzzy Systems Association World Congress, pages 789–798. Springer, 2007.
  * [15] Dervis Karaboga et al.  An idea based on honey bee swarm for numerical optimization.  Technical report, Technical Report-tr06, Erciyes University, Engineering Faculty, Computer …, 2005.
  * [16] Michael Kazhdan and Hugues Hoppe.  Screened poisson surface reconstruction.  ACM Transactions on Graphics (ToG), 32(3):1–13, 2013.
  * [17] Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun.  Tanks and temples: Benchmarking large-scale scene reconstruction.  ACM Transactions on Graphics (ToG), 36(4):1–13, 2017.
  * [18] Kiriakos N Kutulakos and Steven M Seitz.  A theory of shape by space carving.  International Journal of Computer Vision, 38(3):199–218, 2000.
  * [19] Maxime Lhuillier and Long Quan.  A quasi-dense approach to surface reconstruction from uncalibrated images.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(3):418–433, 2005.
  * [20] Fayao Liu, Chunhua Shen, Guosheng Lin, and Ian Reid.  Learning depth from single monocular images using deep convolutional neural fields.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(10):2024–2039, 2015.
  * [21] Keyang Luo, Tao Guan, Lili Ju, Haipeng Huang, and Yawei Luo.  P-mvsnet: Learning patch-wise matching confidence aggregation for multi-view stereo.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10452–10461, 2019.
  * [22] Wendong Mao, Mingjie Wang, Hui Huang, and Minglun Gong.  A robust framework for multi-view stereopsis.  The Visual Computer, 38(5):1539–1551, 2022.
  * [23] George Marsaglia.  Choosing a point from the surface of a sphere.  The Annals of Mathematical Statistics, 43(2):645–646, 1972.
  * [24] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460–4470, 2019.
  * [25] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3504–3515, 2020.
  * [26] Rui Peng, Rongjie Wang, Zhenyu Wang, Yawen Lai, and Ronggang Wang.  Rethinking depth estimation for multi-view stereo: A unified representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8645–8654, 2022.
  * [27] Andrea Romanoni and Matteo Matteucci.  Tapa-mvs: Textureless-aware patchmatch multi-view stereo.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10413–10422, 2019.
  * [28] Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys.  Pixelwise view selection for unstructured multi-view stereo.  In European Conference on Computer Vision, pages 501–518. Springer, 2016.
  * [29] Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski.  A comparison and evaluation of multi-view stereo reconstruction algorithms.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 1, pages 519–528. IEEE, 2006.
  * [30] Steven M Seitz and Charles R Dyer.  Photorealistic scene reconstruction by voxel coloring.  International Journal of Computer Vision, 35(2):151–173, 1999.
  * [31] Sudipta N Sinha, Philippos Mordohai, and Marc Pollefeys.  Multi-view stereo via graph cuts on the dual of an adaptive tetrahedral mesh.  In 2007 IEEE 11th International Conference on Computer Vision, pages 1–8. IEEE, 2007.
  * [32] Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein.  Metasdf: Meta-learning signed distance functions.  Advances in Neural Information Processing Systems, 33:10136–10147, 2020.
  * [33] Qiuhang Tan, Hejun Wu, Biao Hu, and Xingcheng Liu.  An improved artificial bee colony algorithm for clustering.  In Proceedings of the Companion Publication of the 2014 Annual Conference on Genetic and Evolutionary Computation, pages 19–20, 2014.
  * [34] Jiapeng Tang, Xiaoguang Han, Junyi Pan, Kui Jia, and Xin Tong.  A skeleton-bridged deep learning approach for generating meshes of complex topologies from single rgb images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4541–4550, 2019.
  * [35] Jiapeng Tang, Xiaoguang Han, Mingkui Tan, Xin Tong, and Kui Jia.  Skeletonnet: A topology-preserving solution for learning mesh reconstruction of object surfaces from rgb images.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [36] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox.  Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2088–2096, 2017.
  * [37] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas.  Kpconv: Flexible and deformable convolution for point clouds.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6411–6420, 2019.
  * [38] Engin Tola, Vincent Lepetit, and Pascal Fua.  Daisy: An efficient dense descriptor applied to wide-baseline stereo.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(5):815–830, 2009.
  * [39] Fangjinhua Wang, Silvano Galliani, Christoph Vogel, Pablo Speciale, and Marc Pollefeys.  Patchmatchnet: Learned multi-view patchmatch stereo.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14194–14203, 2021.
  * [40] Xiang Wang, Chen Wang, Bing Liu, Xiaoqing Zhou, Liang Zhang, Jin Zheng, and Xiao Bai.  Multi-view stereo in the deep learning era: A comprehensive review.  Displays, 70:102102, 2021.
  * [41] Yunhai Wang, Yiming Qian, Yang Li, Minglun Gong, and Wolfgang Banzhaf.  Artificial multi-bee-colony algorithm for k-nearest-neighbor fields search.  In Proceedings of the Genetic and Evolutionary Computation Conference 2016, pages 1037–1044, 2016.
  * [42] Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou.  Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5610–5619, 2021.
  * [43] Shihao Wu, Peter Bertholet, Hui Huang, Daniel Cohen-Or, Minglun Gong, and Matthias Zwicker.  Structure-aware data consolidation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(10):2529–2537, 2017.
  * [44] Qingshan Xu and Wenbing Tao.  Multi-scale geometric consistency guided multi-view stereo.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5483–5492, 2019.
  * [45] Qingshan Xu and Wenbing Tao.  Planar prior assisted patchmatch multi-view stereo.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12516–12523, 2020.
  * [46] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan.  Pointflow: 3d point cloud generation with continuous normalizing flows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4541–4550, 2019.
  * [47] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  Mvsnet: Depth inference for unstructured multi-view stereo.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 767–783, 2018.
  * [48] Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan.  Recurrent mvsnet for high-resolution multi-view stereo depth inference.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5525–5534, 2019.
  * [49] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan.  Blendedmvs: A large-scale dataset for generalized multi-view stereo networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1790–1799, 2020.
  * [50] Zhichao Yin and Jianping Shi.  Geonet: Unsupervised learning of dense depth, optical flow and camera pose.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1983–1992, 2018.
  * [51] Guopu Zhu and Sam Kwong.  Gbest-guided artificial bee colony algorithm for numerical function optimization.  Applied Mathematics and Computation, 217(7):3166–3173, 2010.


#  _Zooming Out on Zooming In:_ Advancing Super-Resolution for Remote Sensing

Piper Wolters Favyen Bastani Aniruddha Kembhavi
Allen Institute for AI
<EMAIL_ADDRESS>
###### Abstract

Super-Resolution for remote sensing has the potential for huge impact on
planet monitoring by producing accurate and realistic high resolution imagery
on a frequent basis and a global scale. Despite a lot of attention, several
inconsistencies and challenges have prevented it from being deployed in
practice. These include the lack of effective metrics, fragmented and
relatively small-scale datasets for training, insufficient comparisons across
a suite of methods, and unclear evidence for the use of super-resolution
outputs for machine consumption. This work presents a new metric for super-
resolution, CLIPScore, that corresponds far better with human judgments than
previous metrics on an extensive study. We use CLIPScore to evaluate four
standard methods on a new large-scale dataset, S2-NAIP, and three existing
benchmark datasets, and find that generative adversarial networks easily
outperform more traditional L2 loss-based models and are more semantically
accurate than modern diffusion models. We also find that using CLIPScore as an
auxiliary loss can speed up the training of GANs by 18x and lead to improved
outputs, resulting in an effective model in diverse geographies across the
world which we will release publicly. The dataset, pre-trained model weights,
and code are available at this URL.

Figure 1: Display of the incredible power of super-resolution for remote
sensing imagery. High-resolution satellite imagery is not available for free
worldwide, and a public source such as NAIP is restricted to every 2-3 years
in the US. On the other hand, Sentinel-2 imagery is global, free, and has a
revisit rate of 5-10 days, so with super-resolution methods, we can generate
high-resolution imagery globally and frequently, especially in places that
have disproportionately less public imagery.

##  1 Introduction

High-resolution satellite and aerial imagery have the potential to change the
landscape of environmental and climate monitoring applications. Images from
these sensors provide the ability to count individual trees [65], classify
crop types and conditions [35], map out detailed land use categories [31] and
track glacial conditions [55]. Low resolution imagery limits these
observations to coarse designations of regions such as forest, farmland,
residential or ice. Public high-resolution imagery is available infrequently
in a few developed countries and commercial global imagery is expensive and
can be five or more years out of date. With the bounded spatio-temporal
coverage of public domain high-resolution imagery and cost-prohibitive
commercial imagery, many use cases cannot be scaled up globally.

This lack of free high-resolution imagery has prompted research in Super-
Resolution for Remote Sensing, where computer vision methods are used to
upsample low-resolution imagery. Past works include spline models [3],
convolutional neural networks [11, 26] and generative networks [54, 32].
Within this field are diverse datasets with varying satellites, spatial
resolutions [25, 42] and earth coverage [40, 42]. Despite this attention and
potential for impact, Super-Resolution (Super-Res) for remote sensing faces
many challenges that are limiting progress.

First, metrics for Super-Res have serious and well known issues, see Figure 2.
It is crucial to establish a metric that closely aligns with human perception
and reflects the accuracy and realism of Super-Res outputs. We run an
extensive human evaluation study to assess the quality of Super-Res outputs,
and then compare human preferences to those of multiple pixel-wise,
perceptual, and model-based metrics. We determine that model-based metrics
easily outperform pixel-wise ones, and that using modern and powerful visual
encoders is most effective. Based on this analysis and inspired by [20], we
propose a new metric for this domain, CLIPScore.

Second, existing remote sensing datasets are relatively small in comparison to
those for natural images (see Table 1), and often imagery is sourced from
commercial vendors, making it expensive to deploy models in practice. We
present a new large-scale public domain remote sensing Super-Res dataset,
covering 113K $km^{2}$ using free imagery from Sentinel-2 [1] and the US
National Agriculture Imagery Program (NAIP). There are at least 18 Sentinel-2
images for each NAIP image (to support multi-view super-resolution), along
with geographic and temporal metadata. This large amount of free imagery
allows us to determine that scaling up indeed increases performance, and
models can be continually updated with the reoccurring public data for years
to come.

Third, while many diverse models have been proposed, rarely are models
compared side-by-side, especially models that are biased towards the simple
and shallow metrics PSNR and SSIM [11, 50, 43, 34] with models that are more
geared towards perceptual quality [57, 32, 56, 2]. We perform comprehensive
experiments on four remote sensing Super-Res datasets with two L2 loss-based
methods, a GAN, and a diffusion model. We find that the L2 loss-based methods
lag behind by large margins and that the GAN outperforms other methods in
terms of our newly proposed metric, CLIPScore (and also the popular LPIPS
metric) on all four datasets. We also find that training GANs with a CLIPScore
based loss speeds up training dramatically (18x speedup) and further improves
performance. With the increasing emphasis on avoiding hallucinations by
generative methods, we run a human study to determine the accuracy of human
made structures and find that the GAN is superior to the diffusion model. This
body of evidence suggests that despite GANs falling out of favor to modern
diffusion models, particularly in the natural image domain, they nevertheless
perform really well and provide much faster inference for remote sensing
Super-Res.

Fourth, while there is some work exploring the use of Super-Res images as
input to downstream tasks [14] in lieu of the original low-resolution images,
the evidence of improvement over directly inputting the low-resolution images
is unclear. And although representation learning has received much interest in
computer vision, the use of Super-Res as a representation learning mechanism
is under-explored. We conduct a set of experiments to address both of these
directions and find that Super-Res is a powerful representation learning
mechanism that shows promise on downstream remote sensing tasks; but also find
that training models on Super-Res outputs may not yet be more effective than
training on the original low-resolution images for these tasks. Based on these
findings, we establish that although Super-Res outputs can have a large impact
for planet monitoring, they are primarily effective for human consumption
(visualization); more progress is required for them to be effective for
machine consumption.

We leverage our findings to build an effective model for Super-Res that works
well in diverse geographies. Furthermore, we deploy the model globally to
regularly compute up-to-date high-resolution imagery that is freely available.
All data, code, models, and pretrained weights are available at this URL.

In summary, our contributions are:

  1. 1.

CLIPScore, a new metric for Super-Res that strongly corresponds with human
preferences.

  2. 2.

S2-NAIP, a large-scale, public-domain dataset for remote-sensing Super-Res.

  3. 3.

An extensive evaluation of Super-Res methods on four remote sensing datasets,
and a determination that GANs are the current state-of-the-art.

  4. 4.

Demonstrating that transferring Super-Res weights to downstream tasks works
well, but machine consumption of Super-Res outputs is ineffective.

  5. 5.

A new global model for Super-Res that leverages the above contributions.

##  2 Related Work

Super-Res has a rich history in computer vision [12, 30, 69, 59], with recent
methods achieving very impressive results [24, 66]. This paper will focus on
Super-Res for remote sensing, hence we provide context for this domain.

Remote Sensing Super-Resolution Metrics. The two most widely used metrics in
Super-Res are Peak Signal to Noise Ratio (PSNR) and Structural Similarity
Index Measure (SSIM) [61]. PSNR is a pixel-based metric that is the inverse of
an L2 loss; and SSIM was proposed as a perceptual metric based on image
qualities like luminance and contrast. They are both simple functions, as
briefly addressed by [5, 25], that are unable to handle ambiguities like
shadow direction or nadir angle and physical changes like crop harvest cycles.
Figure 2 shows how these are not influenced by crucial perceptual nuances such
as blur. Märtens et al. [42] propose cPSNR, a variation of PSNR that handles
misalignment and brightness differences but the lack of contextual information
remains an issue.

Drawbacks with standard metrics have prompted perceptual metrics like Learned
Perceptual Image Patch Similarity (LPIPS) [68], which is proven to align with
human judgement and is adopted in the natural image domain [44, 6, 17]. Still,
much of the remote sensing community only reports SSIM or pixel-wise metrics
[49, 50, 41], including two of the most recently proposed super-resolution
benchmarks, SEN2VENµS and WorldStrat [39, 10]. We take inspiration from
CLIPScore [20] to find a new metric for Super-Res.

Remote Sensing Super-Resolution Datasets. Existing datasets can be categorized
based on the satellite sensors used, whether they are targeting single image
Super-Res [56, 39] or multi-image Super-Res [42, 10, 25], and the scale of the
data, both in $km^{2}$ covered and number of pixels.

Many datasets curate pairs of low-resolution and high-resolution images, from
one [42] or multiple [10, 34, 56, 39] sensors. In multi-sensor datasets, many
utilize Sentinel-2 as the low-resolution image source [10, 39, 25], likely
because it is free and maintains a good balance of spatial resolution and
temporal revisits. Other works obtain imagery from sources such as Google
Earth [13], PlanetScope [48], PeruSAT [45], China GF 1 [62] and WorldView
[25].

Even with the many datasets that are intended to be standard benchmarks for
remote sensing Super-Res [25, 10, 39, 56], there are still papers that
evaluate their methods on piecemeal datasets [46, 57, 45]. Also, synthetic
datasets are often generated by bicubicly downsampling high-resolution images
to create artificial pairs [36, 64, 2], which is unrealistic and biased by
standard metrics.

Of the proposed benchmark datasets, none of them are large-scale nor are they
entirely obtained from free sources. In this paper, we present S2-NAIP, a
large-scale dataset with over 1.2 million pairs of public domain images,
covering 113 thousand $km^{2}$. Table 1 shows information about existing
benchmarks in comparison to S2-NAIP.

| Sensors | Pairs | Input | HR Res. | $km^{2}$
---|---|---|---|---|---
S2-NAIP | S2,NAIP | 1,200K | 18 | 512x512 | 113K
SEN2VENµS | S2,VENµS | 133K | 1 | 256x256 | 0.806K
WorldStrat | S2,SPOT | 4K | 16 | 1054x1054 | 10K
OLI2MSI | LandSat,S2 | 5.225K | 1 | 480x480 | 0.2K
PROBA-V | PROBA-V | 2.368K | 19 | 384x384 | 3,000K
MuS2 | S2,WV-2 | 0.091K | 14 | 1965x1686 | 2.5K
Table 1: Metadata about existing datasets: the low-resolution (LR) and high-
resolution (HR) sensors used, the number of HR, LR pairs, the number of LR
input images, the spatial resolution of HR (HR Res.) and coverage in terms of
$km^{2}$.

Remote Sensing Super-Resolution Methods. Broadly, recent Super-Res methods
fall into the categories of convolutional [41, 50, 11], generative adversarial
[54, 56, 64], and diffusion models [32, 18, 2]. There is a lack of evaluations
across these three diverse method families, likely because it is difficult to
compare them without strong and consistent metrics. We provide a comprehensive
analysis on methods from each of these categories on four remote sensing
Super-Res datasets, enabling us to determine the most effective method type
for remote sensing Super-Res. A tangentially related line of work uses older
images of a reference image as input to the model along with the low
resolution image at the desired timestamp [19]; this produces impressive
results, but requires high resolution imagery at inference time, which limits
its use in practice.

##  3 Metrics

PSNR and SSIM are the most commonly used metrics to evaluate Super-Res outputs
for remote sensing. These metrics however, have many inherent issues. Previous
work in the natural image domain has acknowledged these, including the well-
known finding that blurring an image will cause significant perceptual change
but little difference will be reflected in PSNR [68]. Figure 2 showcases this
in the remote sensing domain. Although the two Super-Res outputs on the right
are quite poor, with one being blurry and the other being sharply downsampled
and disfiguring structures, they attain high PSNR and SSIM scores, far beyond
an image output from an ESRGAN model which resembles the ground truth image
far more closely to the human eye. We now present an extensive human
evaluation study to evaluate a suite of metrics and present an image
similarity metric based on CLIP [47].

Figure 2: Example of a target image (GT), an ESRGAN output at full resolution
as well as downsampled 16x, and a HighResNet output, with corresponding
metrics. Note that the four images are ordered from best to worst based on
human preference, and PSNR and SSIM increase in an opposite trend. Our
proposed CLIPScore more closely matches human judgement.

###  3.1 Super-Resolution Human Judgement Dataset

To find a metric that can effectively assess perceptual quality of remote
sensing Super-Res outputs, we run a human evaluation study to gather a dataset
of human judgements on our S2-NAIP dataset and WorldStrat [10]. Preferences
are gathered using Amazon Mechanical Turk (AMT), where each AMT worker was
asked to pick, between two model outputs, which one is closer to the target
image.

Pairs of outputs were chosen from a set of generated outputs from multiple
models on each of 20,000 datapoints from S2-NAIP and 381 datapoints from the
WorldStrat. A total of 11,524 pair-wise annotations were collected. More
details on this process are in supplementary Section A.1.

We computed many metrics on the model outputs and generated preferences for
each of the pairs that had been annotated. The metrics include PSNR, SSIM,
LPIPS, SAMScore [29], and measures of feature similarity from several web-
scale models like CLIP [47]. The level of agreement between each metric and
the human preferences across all pairs was computed (i.e., the percentage of
pairs where the metric and human pick the same model output as being more
similar to the target image). Results are shown in Figure 3.

Finding 1. _PSNR and SSIM, the most widely used metrics, are insufficient and
poorly correlate with human judgments._

The correspondences between human preferences and those computed by the
standard Super-Res metrics, PSNR, SSIM, and cPSNR, are very low, as shown in
Figure 3. In this study, an accuracy of 50% is equal to random guessing, so
metrics with accuracies less than or equal to this are unacceptable for
evaluation of Super-Res quality. This finding aligns with the counter-
intuitive values of these metrics in Figure 2. Remote sensing Super-Res should
not rely on metrics like PSNR or SSIM when there are other metrics that
achieve over 80% accuracy.

###  3.2 CLIP as an Image Similarity Metric

Inspired by CLIPScore [20], we propose a new metric for Super-Res that
measures the distance in image embedding space between the target image and
the generated image, using the web-scale CLIP [47] model. The score is equal
to the cosine similarity between the target and output features. This can be
done in just a few lines of code, as shown in the snippet below.

⬇

1import clip

2import torch.nn.functional as F

3

4# Substitute any CLIP model here

5clip_model = clip.load("RN50")

6

7gt_feats = clip_model(gt)

8super_res_feats = clip_model(super_res)

9

10score = F.cosine_similarity(gt_feats, super_res_features)

We assessed several CLIP variants, including CLIPA [27], SigLIP [67], MetaCLIP
[63], and EVA [53]. Additionally, we trained variations of CLIP from scratch
on satellite imagery, similar to [23], but found the performance of these, as
metrics, to be worse than the pretrained models. More details are provided in
supplementary Section A.2.

Finding 2. _CLIP is an effective super-resolution metric._

The results from the human-metric correspondence study show that the CLIP
models are effective at measuring perceptual quality, with at least 76%
agreement with human preference. CLIPA-v2 [28] performed best in this study,
with an impressive agreement accuracy of 84.6%. We propose this as a measure
of image similarity for Super-Res.

We use the open-source CLIPA-v2 model with the ViT-bigG-14 architecture,
pretrained on DataComp1b [15], from the open_clip codebase [8]. Note that this
model configuration is in the top-5 of best performing models on their set of
38 evaluation tasks including the remote sensing tasks Resisc45 [7] and FMoW
[9]. The large diversity in the DataComp1b training data is likely a
contributing factor to why this model works so well as a metric for satellite
imagery.

Figure 3: The level of accuracy between human preferences and those generated
by the various metrics. The x-axis is ordered from worst to best average
accuracy between the two datasets. The y-axis is adjusted to a range of 40% to
90% to better show the difference in accuracy across metrics.

##  4 Data

| S2-NAIP | WorldStrat | PROBA-V | OLI2MSI
---|---|---|---|---
Method | cPSNR $\uparrow$ | LPIPS $\uparrow$ |  CLIP $\uparrow$ | cPSNR $\uparrow$ | LPIPS $\uparrow$ |  CLIP $\uparrow$ | cPSNR $\uparrow$ | LPIPS $\uparrow$ |  CLIP $\uparrow$ | cPSNR $\uparrow$ | LPIPS $\uparrow$ |  CLIP $\uparrow$
SRCNN | 17.9039 | 0.3517 | 0.7 | 31.5746 | 0.4493 | 0.6196 | 23.9455 | 0.2112 | 0.7924 | 43.4113 | 0.7614 | 0.9285
HighResNet | 20.9171 | 0.199 | 0.6705 | 32.8787 | 0.4416 | 0.5998 | 22.5687 | 0.2137 | 0.8096 | 35.7374 | 0.86 | 0.94
ESRGAN | 22.6506 | 0.8406 | 0.8745 | 31.723 | 0.8299 | 0.9842 | 24.9336 | 0.2154 | 0.8465 | 36.3202 | 0.8709 | 0.9518
SR3 | 19.4706 | 0.6292 | 0.8223 | 30.545 | 0.7745 | 0.9232 | 23.7789 | 0.2115 | 0.8115 | 34.2208 | 0.839 | 0.9391
Table 2: Results for four methods on four datasets. The reported LPIPS scores
are converted to an accuracy, so with each of these metrics, higher is better.
ESRGAN achieves the best CLIPScore for all datasets, and that is the metric we
are prioritizing for this study.

Existing datasets are relatively small in scale, in terms of number of images
and coverage in terms of $km^{2}$, as reflected in Table 1. Also, most
datasets source a portion of their imagery from commercial satellites, making
it difficult in practice to extend the data and methods produced.

###  4.1 S2-NAIP Dataset

We build a new dataset, S2-NAIP, consisting of 1.2 million pairs of low-
resolution Sentinel-2 time series and high-resolution NAIP images. The goal
with this is to measure if an increased scale of training data will improve
model performance as well as to release a remote sensing Super-Res dataset
built entirely from public domain images.

NAIP imagery covers most of the United States with a revisit rate of 2-3
years, and Sentinel-2 is globally available with new imagery every 5-10 days.
We source the NAIP imagery from 2019-2020 as well as all spatially overlapping
Sentinel-2 images within two months of each NAIP image capture timestamp,
resulting in at least 18 Sentinel-2 images. We do not remove images with cloud
cover, so that the data maintains a real-world distribution.

Figure 4: Examples of target images (GT), one of the corresponding low-
resolution images (Low-Res), and Super-Res outputs from SRCNN, HighResNet,
Medium ESRGAN, and SR3 on samples from the S2-NAIP dataset. We recommend
zooming in for the full effect.

The full resolution of each NAIP tile is 512x512 pixels. Tiles correspond to
Web-Mercator tiles at zoom level 17, i.e., the world is projected to a 2D
plane and divided into a $2^{17}$ X $2^{17}$ grid, with each tile
corresponding to a grid cell. Each NAIP tile corresponds to a time series of
Sentinel-2 images, each having a resolution of 32x32 pixels. Geospatial and
temporal information for all imagery is provided.

With the goal of avoiding an overload of monotonous landscapes such as ocean
or desert, we sample NAIP tiles from within 20km of cities with populations of
at least 20k. This resulted in a balance of various man-made infrastructure
like water tanks, parking lots, and houses as well as farmland, mountainous
area, and forest.

We train an ESRGAN [58], as described in Section 5, to upsample the Sentinel-2
images by a factor of four to 128x128 pixels, and use NAIP tiles downsampled
by four as the target images. Experiments are run with 1, 3, 10, 30, and 100
percent of the S2-NAIP dataset to determine the difference in performance with
varied dataset sizes.

Additionally, we train three sizes of the ESRGAN model on each of these data
splits to see if increased model size in conjunction with more data is
correlated with better outputs. We define small, medium, and large versions of
the ESRGAN with 17mil, 87mil, and 347mil parameters, respectively. Although
the effects of data and model size are well studied in the natural image
domain, few comprehensive studies have been done on this in remote sensing.

Finding 3. _Performance scales with dataset and model size._

Somewhat unsurprisingly, we find that training on more of S2-NAIP results in
higher performance, as shown in Figure 5. Between the smallest and largest
data splits, there is a ten point improvement in CLIPScore for the smallest
model. This suggests that training on larger amounts of data will be
beneficial to Super-Res in remote sensing, justifying the creation of large-
scale datasets like our proposed S2-NAIP.

Model size also contributes to higher quality outputs. There is on average a
five point improvement between the small and large models across all five data
splits. This finding motivates further exploration into larger Super-Res
models for satellite imagery, especially with GANs that provide substantially
faster inference than diffusion models.

Figure 5: CLIPScore results on the S2-NAIP dataset, with three sizes of the
ESRGAN model and five data splits. Performance increases with more data and
larger models.

##  5 Method Study

We run a comprehensive study with SRCNN [12], HighResNet [11], [10], ESRGAN
[58], and Image Super-Resolution via Iterative Refinement (SR3); the latter
method is an adaption of Denoising Diffusion for Probablistic Models (DDPM)
[22] for Super-Res. The SR3 model has 97mil parameters so we choose to use the
medium ESRGAN from Figure 5 which has 87mil parameters, for a relatively fair
comparison. Minor model tweaks are made across datasets to account for
different upsample factors as well as for stabilization of training. Specific
training and model details can be found in supplementary Section A.3.

We train and evaluate these methods on our proposed large-scale S2-NAIP
dataset as well as the existing PROBA-V [42] and the recently proposed
benchmarks WorldStrat [10] and OLI2MSI [56]. We report CLIPScore as proposed
in Section 3 and base our evaluation on this metric, though we also report
cPSNR and LPIPS for the sake of comparison to previous work. Results are shown
in Table 2.

We further experimented with several variations of ESRGAN and describe the
findings that led to demonstrably better performance in Section 6. We also ran
experiments using Denoising Diffusion Implicit Models (DDIM) [51], which led
to much faster inference but slightly lesser quality outputs; as well as
Classifier Free Guidance [21], which did not clearly improve output quality.
Further details with sample outputs can be found in supplementary Section A.4.

Finding 4. _GANs are capable super-resolution methods._

Based on the cumulative CLIPScore results on four datasets as shown in Table
2, it is clear that ESRGAN is effective in producing high-quality Super-Res
outputs. Compared to training only on L2 loss, like SRCNN and HighResNet, the
outputs are sharper and more similar to the target images, as shown in Figure
4.

Although diffusion models have become increasingly popular in computer vision,
GANs are proving here to be very effective. It is also important to mention
the large gap in inference speed – GANs require just one forward pass while
diffusion models require several. Between our ESRGAN and SR3 models, the
ESRGAN is 200x faster.

Building Count Study. Qualitative assessments of the four models, as shown in
Figure 4, provides evidence that ESRGAN and SR3 produce high quality outputs.
A priority in remote sensing Super-Res is maintaining accuracy and avoiding
hallucinations [42]. To determine the accuracy of the two generative methods,
we run a human study on a held out set of 256 datapoints from S2-NAIP, with
the task of counting buildings. We choose this task because human-made
structures are of special interest in satellite imagery and are important to
represent accurately.

We have an expert annotator count the number of visible buildings in the
provided target images as well as the corresponding generated images from
ESRGAN and SR3. Of the buildings annotated in each model output, the buildings
were labeled as ground truth or hallucination, depending on if the building
exists in the target image or not.

Results from this study show that ESRGAN accurately generates the buildings
from the high-resolution target image 94.70% of the time, while SR3 only
81.77% of the time. Furthermore, the average hallucination rate for each
method is 0.41 and 0.43 buildings per image, respectively.

Finding 5. _Diffusion models generate realistic but inaccurate images._

Results from this study give us a glimpse into the difference in semantic
accuracy between two models that both generate high perceptual quality
outputs. ESRGAN is more accurate in generating the correct buildings relative
to the target image and has a smaller hallucination rate.

Example qualitative outputs from the building count study are shown in Figure
6, where red circles point out flaws in the SR3 column such as nonexistent
roads, the addition of buildings where there should be none, and a grassy
field where there is supposed to be a river.

Figure 6: Examples of high-resolution images from S2-NAIP and corresponding
ESRGAN and SR3 outputs. Red circles showcase qualitatively incorrect portions
generated by SR3.

##  6 Improving ESRGAN

We find ESRGAN to be the strongest Super-Res method of the four compared in
our study, and propose novel techniques to improve it. This includes adding an
auxiliary loss based on CLIPScore described in Section 6.1 and incorporating
domain knowledge into the training pipeline, described in supplementary
Section A.5. We hypothesize that these could be transferred to other GAN
models.

###  6.1 Training with CLIPScore

Similar to previous works that optimize a loss that is inversely related to a
metric such as LPIPS [68], we introduce a CLIPScore loss. We use an L1 loss to
minimize the distance between CLIP features of the target and the Super-Res
output. In our experiments, we use CLIP with ResNet50, as it is smaller and
thus faster than CLIPA-v2.

We train the small ESRGAN in Figure 5, from scratch, with the addition of the
CLIPScore loss. Figure 7 shows results of models trained with and without this
loss.

Figure 7: Plot of CLIPScore between a model trained with the CLIP-based loss
and one without. Note the y-axis is between 0.75 and 0.95 to more clearly show
the difference in performance.

Finding 6. _Incorporating CLIP significantly speeds up training and produces
more effective models._

We find that training a model with the addition of the CLIPScore-based loss
results in a 9 point improvement in CLIPScore in 18x less training time. This
decreases our training time from weeks to hours and the quality of Super-Res
outputs is vastly improved.

The Best Model. We compile all of the improvements described in Sections 6.1
and supplementary Section A.5 into one model, and the result is an impressive
0.959 CLIPScore. Figure 8 shows qualitative results between the small and
large ESRGAN models from Figure 5, compared to the best model.

Figure 8: Examples of target images (GT), one of the corresponding low-
resolution images (Low-Res), and outputs from the small, large, and best
ESRGAN. We recommend zooming in and looking at the fine differences between
the last two columns.

##  7 Super-Resolution for Downstream Tasks

In this section, we analyze two uses of Super-Res for downstream tasks: images
and features.

We compile a set of five downstream tasks corresponding to existing datasets:
the three SatlasPretrain fine-tuning datasets (solar farms, wind turbines, and
marine infrastructure) [4], PASTIS (crop type segmentation) [16], and
BigEarthNet (land cover segmentation) [52]. Because these five datasets are
built with Sentinel-2 imagery in mind, most features are relatively coarse, so
we also create a new urban land use segmentation dataset from OpenStreetMap
labels; details are in supplementary Section A.6.

###  7.1 Image Usage

Previous works such as PROBA-V and WorldStrat [42, 10] have argued that Super-
Res outputs could be used in place of low-resolution images for downstream
tasks like crop type and urban land use segmentation. In fact, the authors of
PROBA-V argue that, in remote sensing, Super-Res methods should be designed
for machine rather than human consumption, and suggest that models trained
without GAN losses may produce images with more accurate pixel values that are
better for machine consumption.

To evaluate this claim, we generate SRCNN, HighResNet, and ESRGAN Super-Res
outputs from the Sentinel-2 imagery in each of the downstream tasks, and then
use them as input to a model with a Swin-Base Transformer [33] backbone
pretrained on ImageNet. Second, we apply a model that inputs eight low-
resolution Sentinel-2 images to the same Swin-Base Transformer [33] backbone
pretrained on ImageNet. Results are shown in Table 3.

Finding 7. _Super-resolution outputs are ineffective for machine consumption._

Via this preliminary study we find that using Super-Res outputs does not
outperform inputting the original low-resolution images, as shown in Table 3.
Interestingly, between SRCNN, HighResNet, and ESRGAN, the ESRGAN outputs led
to the best average downstream performance. This suggests that optimizing L2
losses does not always imply better outputs for machine use and GANs should be
considered for Super-Res, regardless of whether the aim is machine or human
consumption.

###  7.2 Feature Usage

Transfer learning is a very common technique in computer vision for improving
the downstream performance, though using Super-Res weights is largely under-
explored.

We evaluate the effectiveness of this by fine-tuning a Swin-Base Transformer
backbone that is (a) randomly initialized; (b) pretrained on ImageNet; (c)
pretrained on SatlasPretrain [4]; and (d) pretrained on our S2-NAIP Super-Res
dataset. And though not directly comparable, we also apply a similar model
with a ResNet50 backbone pretrained using different self-supervised learning
methods, including CaCo [37], SeCo [38], and SSL4EO-S12 [60]; Swin weights
were not available for these methods. Results are shown in Table 4.

Finding 8. _Representations learned through super-resolution transfer to
downstream tasks._

On average, representations from Super-Res lead to the best downstream
performance compared to those of self-supervised learning methods, CaCo [37],
SeCo [38], and SSL4EO-S12 [60], and from the large-scale SatlasPretrain [4]
dataset. This encourages further exploration in using Super-Res
representations for downstream tasks.

Input | OSM | S1 | S2 | S3 | Big | PS | Avg
---|---|---|---|---|---|---|---
Low-Res | 0.464 | 0.782 | 0.888 | 0.591 | 0.963 | 0.348 | 0.673
SRCNN | 0.429 | 0.219 | 0.616 | 0.524 | 0.953 | 0.269 | 0.502
HighResNet | 0.467 | 0.661 | 0.732 | 0.521 | 0.957 | 0.287 | 0.604
ESRGAN | 0.486 | 0.723 | 0.706 | 0.524 | 0.954 | 0.299 | 0.615
Table 3: Results on 6 tasks (Our OpenStreetMap dataset=OSM, Satlas Tasks=Solar Farms, Wind Turbines, and Marine Infrastructure, BigEarthNet=Big, PASTIS=PS), using an ImageNet Swin Model with 8 low-resolution input images versus Super-Res outputs from SRCNN, HighResNet, and ESRGAN.  Arch | Method | OSM | Satlas1 | Satlas2 | Satlas3 | BigEarth | PASTIS | Avg
---|---|---|---|---|---|---|---|---
Res50 |  SeCo [38] | 0.4493 | 0.8468 | 0.8962 | 0.6008 | 0.9641 | 0.2596 | 0.6695
|  CaCo [37] | 0.4421 | 0.8384 | 0.8985 | 0.5985 | 0.9629 | 0.2591 | 0.6666
|  SSL4EO [60] | 0.4529 | 0.8213 | 0.8924 | 0.6029 | 0.9637 | 0.2426 | 0.6626
Swin | Random | 0.4446 | 0.7010 | 0.7391 | 0.5591 | 0.9607 | 0.3754 | 0.6299
| ImageNet | 0.4642 | 0.7820 | 0.8878 | 0.5906 | 0.9631 | 0.3484 | 0.6727
|  Satlas [4] | 0.4718 | 0.8818 | 0.9152 | 0.6021 | 0.9641 | 0.3594 | 0.6991
| Super-Res | 0.4759 | 0.8987 | 0.8958 | 0.5927 | 0.9637 | 0.3803 | 0.7012
Table 4: Transfer learning results on 6 downstream tasks (Our OpenStreetMap
dataset=OSM, BigEarthNet=BigEarth, and the three Satlas tasks are Solar Farms,
Wind Turbines, and Marine Infrastructure), comparing several pretraining
methods to Super-Res weights trained on our S2-NAIP dataset.

##  8 Deploying Super-Resolution Globally

With all of these findings, we have deployed global Super-Res outputs to
https://satlas.allen.ai/ for anyone to view. We hope this can further research
in AI for remote sensing as well as assist non-ML researchers with annotation
tasks such as identifying the drivers of deforestation or counting green
energy installations.

##  9 Conclusion

We explored, in depth, the metrics, datasets, and methods in the remote
sensing Super-Res field. We propose CLIPScore as a new metric and utilize it
to analyze three method types, and find that GANs are very effective for this
task. We introduce a new large-scale dataset, S2-NAIP, and determine the
benefit of scale. We determine the effectiveness of Super-Res images and
features for downstream tasks. Finally, these findings enable us to train a
large-scale model that achieves an impressive CLIPScore.

##  Appendix A Supplementary Material

The supplementary material can be accessed at this URL.

## References

  * sen [2022] Copernicus Sentinel Missions.  https://sentinel.esa.int/web/sentinel/home, 2022.  European Space Agency.
  * Ali et al. [2023] Anas M. Ali, Bilel Benjdira, Anis Koubâa, Wadii Boulila, and Walid El Shafai.  Tesr: Two-stage approach for enhancement and super-resolution of remote sensing images.  _Remote. Sens._ , 15:2346, 2023.
  * Anger et al. [2020] Jérémy Anger, Thibaud Ehret, Carlo de Franchis, and G. Facciolo.  Fast and accurate multi-frame super-resolution of satellite images.  _ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , pages 57–64, 2020.
  * Bastani et al. [2023] Favyen Bastani, Piper Wolters, Ritwik Gupta, Joe Ferdinando, and Aniruddha Kembhavi.  Satlaspretrain: A large-scale dataset for remote sensing image understanding.  _ICCV 2023_ , 2023.
  * Beaulieu et al. [2018] Mario Beaulieu, Samuel Foucher, Dan Haberman, and Colin Stewart.  Deep image- to-image transfer applied to resolution enhancement of sentinel-2 images.  _IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium_ , pages 2611–2614, 2018.
  * Bhat et al. [2021] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Deep burst super-resolution.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9205–9214, 2021.
  * Cheng et al. [2017] Gong Cheng, Junwei Han, and Xiaoqiang Lu.  Remote Sensing Image Scene Classification: Benchmark and State of the Art.  In _Proceedings of the IEEE_ , pages 1865–1883, 2017.
  * Cherti et al. [2022] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.  Reproducible scaling laws for contrastive language-image learning.  _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2818–2829, 2022.
  * Christie et al. [2018] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee.  Functional Map of the World.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Cornebise et al. [2022] Julien Cornebise, Ivan Orsolic, and Freddie Kalaitzis.  Open high-resolution satellite imagery: The worldstrat dataset - with application to super-resolution.  _ArXiv_ , abs/2207.06418, 2022.
  * Deudon et al. [2020] Michel Deudon, Alfredo Kalaitzis, Israel Goytom, Md Rifat Arefin, Zhichao Lin, Kris Sankaran, Vincent Michalski, Samira Ebrahimi Kahou, Julien Cornebise, and Yoshua Bengio.  Highres-net: Recursive fusion for multi-frame super-resolution of satellite imagery.  _ArXiv_ , abs/2002.06460, 2020.
  * Dong et al. [2014] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 38:295–307, 2014.
  * Dong et al. [2021] Runmin Dong, Lixian Zhang, and Haohuan Fu.  Rrsgan: Reference-based super-resolution for remote sensing image.  _IEEE Transactions on Geoscience and Remote Sensing_ , 60:1–17, 2021.
  * Frizza et al. [2022] Tristan Frizza, D. Dansereau, Nagita Mehr Seresht, and Michael Bewley.  Semantically accurate super-resolution generative adversarial networks.  _Comput. Vis. Image Underst._ , 221:103464, 2022.
  * Gadre et al. [2023] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander J. Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alexandros G. Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt.  Datacomp: In search of the next generation of multimodal datasets.  _ArXiv_ , abs/2304.14108, 2023.
  * Garnot and Landrieu [2021] Vivien Sainte Fare Garnot and Loic Landrieu.  Panoptic segmentation of satellite image time series with convolutional temporal attention networks.  _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 4852–4861, 2021.
  * Gu et al. [2022] Jinjin Gu, Haoming Cai, Chenyu Dong, Ruofan Zhang, Yulun Zhang, Wenming Yang, and Chun Yuan.  Super-resolution by predicting offsets: An ultra-efficient super-resolution network for rasterized images.  In _European Conference on Computer Vision_ , 2022.
  * Han et al. [2023] Lintao Han, Yuchen Zhao, Hengyi Lv, Yisa Zhang, Hailong Liu, Guoling Bi, and Qing Han.  Enhancing remote sensing image super-resolution with efficient hybrid conditional diffusion model.  _Remote. Sens._ , 15:3452, 2023.
  * He et al. [2021] Yutong He, Dingjie Wang, Nicholas Lai, William Zhang, Chenlin Meng, M. Burke, D. Lobell, and Stefano Ermon.  Spatial-temporal super-resolution of satellite imagery via conditional pixel synthesis.  In _Neural Information Processing Systems_ , 2021.
  * Hessel et al. [2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  _ArXiv_ , abs/2104.08718, 2021.
  * Ho [2022] Jonathan Ho.  Classifier-free diffusion guidance.  _ArXiv_ , abs/2207.12598, 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  _ArXiv_ , abs/2006.11239, 2020.
  * Jain et al. [2022] Umang Jain, Alex Wilson, and Varun Gulshan.  Multimodal contrastive learning for remote sensing tasks.  _ArXiv_ , abs/2209.02329, 2022.
  * Kang et al. [2023] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park.  Scaling up gans for text-to-image synthesis.  _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10124–10134, 2023.
  * Kowaleczko et al. [2023] Pawel Kowaleczko, Tomasz Tarasiewicz, Maciej Ziaja, Daniel Kostrzewa, Jakub Nalepa, Przemyslaw Rokita, and Michal Kawulok.  A real-world benchmark for sentinel-2 multi-image super-resolution.  _Sci Data_ , 10:644, 2023.
  * Laban et al. [2017] Noureldin Laban, Bassam Abdellatif, H. M. Ebied, Howida A. Shedeed, and Mohamed F. Tolba.  Performance enhancement of satellite image classification using a convolutional neural network.  In _International Conference on Advanced Intelligent System and Informatics_ , 2017.
  * Li et al. [2023a] Xianhang Li, Zeyu Wang, and Cihang Xie.  An inverse scaling law for clip training.  _ArXiv_ , abs/2305.07017, 2023a.
  * Li et al. [2023b] Xianhang Li, Zeyu Wang, and Cihang Xie.  Clipa-v2: Scaling clip training with 81.1% zero-shot imagenet accuracy within a $10, 000 budget; an extra $4, 000 unlocks 81.8% accuracy.  _ArXiv_ , abs/2306.15658, 2023b.
  * Li et al. [2023c] Yunxiang Li, Meixu Chen, Wenxuan Yang, Kai Wang, Jun Ma, Alan C. Bovik, and You Zhang.  Samscore: A semantic structural similarity metric for image translation evaluation.  In _Arxiv_ , 2023c.
  * Lim et al. [2017] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 1132–1140, 2017.
  * Liu et al. [2020] Chun Liu, Doudou Zeng, Hangbin Wu, Yin Wang, Shoujun Jia, and Liang Xin.  Urban land cover classification of high-resolution aerial imagery using a relation-enhanced multiscale convolutional network.  _Remote. Sens._ , 12:311, 2020.
  * Liu et al. [2022] Jinzhe Liu, Zhiqiang Yuan, Zhaoying Pan, Yiqun Fu, Li Liu, and Bin Lu.  Diffusion model with detail complement for super-resolution of remote sensing.  _Remote. Sens._ , 14:4834, 2022.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9992–10002, 2021.
  * Lu et al. [2019] Tao Lu, Jiaming Wang, Yanduo Zhang, Zhongyuan Wang, and Junjun Jiang.  Satellite image super-resolution via multi-scale residual deep neural network.  _Remote. Sens._ , 11:1588, 2019.
  * Lu et al. [2022] Tingyu Lu, Luhe Wan, and Lei Wang.  Fine crop classification in high resolution remote sensing based on deep learning.  In _Frontiers in Environmental Science_ , 2022.
  * Luo et al. [2023] Jinming Luo, Lei Han, Xianjie Gao, Xiuping Liu, and Weiming M. Wang.  Sr-feinr: Continuous remote sensing image super-resolution using feature-enhanced implicit neural representation.  _Sensors (Basel, Switzerland)_ , 23, 2023.
  * Mall et al. [2023] Utkarsh Mall, Bharath Hariharan, and Kavita Bala.  Change-aware sampling and contrastive learning for satellite images.  _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5261–5270, 2023.
  * Mañas et al. [2021] Oscar Mañas, Alexandre Lacoste, Xavier Giró i Nieto, David Vázquez, and Pau Rodríguez López.  Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data.  _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9394–9403, 2021.
  * Michel et al. [2022] Julien Michel, Juan Vinasco-Salinas, Jordi Inglada, and Olivier Hagolle.  Sen2venµs, a dataset for the training of sentinel-2 super-resolution algorithms.  _Data_ , 7:96, 2022.
  * Mnih [2020] Volodymyr Mnih.  Massachusetts buildings dataset.  https://www.kaggle.com/datasets/balraj98/massachusetts-buildings-dataset, 2020.
  * Molini et al. [2019] Andrea Bordone Molini, Diego Valsesia, Giulia Fracastoro, and Enrico Magli.  Deepsum: Deep neural network for super-resolution of unregistered multitemporal images.  _IEEE Transactions on Geoscience and Remote Sensing_ , 58:3644–3656, 2019.
  * Märtens et al. [2019] Marcus Märtens, Dario Izzo, Andrej Krzic, and Daniël Cox.  Super-resolution of proba-v images using convolutional neural networks.  _Astrodyn_ , 3:387–402, 2019.
  * Nayak et al. [2020] Gaurav Kumar Nayak, Saksham Jain, R. Venkatesh Babu, and Anirban Chakraborty.  Fusion of deep and non-deep methods for fast super-resolution of satellite images.  _2020 IEEE Sixth International Conference on Multimedia Big Data (BigMM)_ , pages 267–271, 2020.
  * Pérez-Pellitero et al. [2018] Eduardo Pérez-Pellitero, Mehdi S. M. Sajjadi, Michael Hirsch, and Bernhard Scholkopf.  Photorealistic video super resolution.  _ArXiv_ , abs/1807.07930, 2018.
  * Pineda et al. [2020] Ferdinand Pineda, Victor H. Andres Ayma, and César Beltrán.  A generative adversarial network approach for super-resolution of sentinel-2 satellite images.  _ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , pages 9–14, 2020.
  * Puri and Kotze [2022] Jaskaran Singh Puri and Andre Kotze.  Evaluation of srgan algorithm for superresolution of satellite imagery on different sensors.  _25th AGILE Conference on Geographic Information Science_ , 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , 2021.
  * Razzak et al. [2021a] Muhammed Razzak, Gonzalo Mateo-García, Luis G’omez-Chova, Yarin Gal, and Freddie Kalaitzis.  Multi-spectral multi-image super-resolution of sentinel-2 with radiometric consistency losses and its effect on building delineation.  _ArXiv_ , abs/2111.03231, 2021a.
  * Razzak et al. [2021b] Muhammed Razzak, Gonzalo Mateo-García, Luis G’omez-Chova, Yarin Gal, and Freddie Kalaitzis.  Multi-spectral multi-image super-resolution of sentinel-2 with radiometric consistency losses and its effect on building delineation.  _ArXiv_ , abs/2111.03231, 2021b.
  * Salvetti et al. [2020] Francesco Salvetti, Vittorio Mazzia, Aleem Khaliq, and Marcello Chiaberge.  Multi-image super resolution of remotely sensed images using residual feature attention deep neural networks.  _ArXiv_ , abs/2007.03107, 2020.
  * Song et al. [2020] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _ArXiv_ , abs/2010.02502, 2020.
  * Sumbul et al. [2019] Gencer Sumbul, Marcela Charfuelan, Begüm Demir, and Volker Markl.  Bigearthnet: A large-scale benchmark archive for remote sensing image understanding.  _IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium_ , pages 5901–5904, 2019.
  * Sun et al. [2023] Quan Sun, Yuxin Fang, Ledell Yu Wu, Xinlong Wang, and Yue Cao.  Eva-clip: Improved training techniques for clip at scale.  _ArXiv_ , abs/2303.15389, 2023.
  * Tayba and Rivas [2021] Mahee Noor Tayba and P. Munoz Rivas.  Enhancing the resolution of satellite imagery using a generative model.  _2021 International Conference on Computational Science and Computational Intelligence (CSCI)_ , pages 20–25, 2021.
  * Thomas et al. [2023] Daniel Jack Thomas, Benjamin Aubrey Robson, and Adina E. Racoviteanu.  An integrated deep learning and object-based image analysis approach for mapping debris-covered glaciers.  _Frontiers in Remote Sensing_ , 2023.
  * Wang et al. [2021a] Junwei Wang, Kun Gao, Zhenzhou Zhang, Chong Ni, Zibo Hu, Dayu Chen, and Qiong Wu.  Multisensor remote sensing imagery super-resolution with conditional gan.  _Journal of Remote Sensing_ , 2021a.
  * Wang and Sertel [2023] Peijuan Wang and Elif Sertel.  Multi-frame super-resolution of remote sensing images using attention-based gan models.  _Knowl. Based Syst._ , 266:110387, 2023.
  * Wang et al. [2018] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang.  Esrgan: Enhanced super-resolution generative adversarial networks.  In _ECCV Workshops_ , 2018.
  * Wang et al. [2021b] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan.  Real-esrgan: Training real-world blind super-resolution with pure synthetic data.  _2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)_ , pages 1905–1914, 2021b.
  * Wang et al. [2022] Yi Wang, Nassim Ait Ali Braham, Zhitong Xiong, Chenying Liu, Conrad M. Albrecht, and Xiao Xiang Zhu.  Ssl4eo-s12: A large-scale multi-modal, multi-temporal dataset for self-supervised learning in earth observation.  _ArXiv_ , abs/2211.07044, 2022.
  * Wang et al. [2004] Zhou Wang, Alan Conrad Bovik, Hamid R. Sheikh, and Eero P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ , 13:600–612, 2004.
  * Xiong et al. [2020] Yingfei Xiong, Shanxin Guo, Jinsong Chen, Xinping Deng, Luyi Sun, Xiaorou Zheng, and Wenna Xu.  Improved srgan for remote sensing image superresolution across locations and sensors.  _Remote. Sens._ , 12:1263, 2020.
  * Xu et al. [2023] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao (Bernie) Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer.  Demystifying clip data.  _ArXiv_ , abs/2309.16671, 2023.
  * Xu et al. [2022] Yongyang Xu, Wei Luo, Anna Hu, Zhong Xie, Xuejing Xie, and Liufeng Tao.  Te-sagan: An improved generative adversarial network for remote sensing super-resolution images.  _Remote. Sens._ , 14:2425, 2022.
  * Yao et al. [2021] Ling Yao, Tang Liu, Jun Qin, Ning Lu, and Chenghu Zhou.  Tree counting with high spatial-resolution satellite imagery based on deep neural networks.  _Ecological Indicators_ , 125:107591, 2021.
  * Yue et al. [2023] Zongsheng Yue, Jianyi Wang, and Chen Change Loy.  Resshift: Efficient diffusion model for image super-resolution by residual shifting.  _ArXiv_ , abs/2307.12348, 2023.
  * Zhai et al. [2023] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.  Sigmoid loss for language image pre-training.  _ArXiv_ , abs/2303.15343, 2023.
  * Zhang et al. [2018a] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 586–595, 2018a.
  * Zhang et al. [2018b] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Raymond Fu.  Residual dense network for image super-resolution.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2472–2481, 2018b.