# Geometry-Aware Reference Synthesis for Multi-View Image Super-Resolution

Ri Cheng  0000-0002-5866-6847 School of Computer Science, Shanghai Key
Laboratory of Intelligent Information Processing, Shanghai Collaborative
Innovation Center of Intelligent Visual Computing, Fudan
UniversityShanghaiChina<EMAIL_ADDRESS>,  Yuqi Sun  0000-0002-7179-5045
School of Computer Science, Shanghai Key Laboratory of Intelligent Information
Processing, Shanghai Collaborative Innovation Center of Intelligent Visual
Computing, Fudan UniversityShanghaiChina<EMAIL_ADDRESS>,  Bo Yan
0000-0003-0256-9682 School of Computer Science, Shanghai Key Laboratory of
Intelligent Information Processing, Shanghai Collaborative Innovation Center
of Intelligent Visual Computing, Fudan UniversityShanghaiChina
<EMAIL_ADDRESS>,  Weimin Tan  0000-0001-7677-4772 School of Computer
Science, Shanghai Key Laboratory of Intelligent Information Processing,
Shanghai Collaborative Innovation Center of Intelligent Visual Computing,
Fudan UniversityShanghaiChina<EMAIL_ADDRESS>and  Chenxi Ma
0000-0002-5577-5773 School of Computer Science, Shanghai Key Laboratory of
Intelligent Information Processing, Shanghai Collaborative Innovation Center
of Intelligent Visual Computing, Fudan UniversityShanghaiChina
<EMAIL_ADDRESS>
(2022)

###### Abstract.

Recent multi-view multimedia applications struggle between high-resolution
(HR) visual experience and storage or bandwidth constraints. Therefore, this
paper proposes a Multi-View Image Super-Resolution (MVISR) task. It aims to
increase the resolution of multi-view images captured from the same scene. One
solution is to apply image or video super-resolution (SR) methods to
reconstruct HR results from the low-resolution (LR) input view. However, these
methods cannot handle large-angle transformations between views and leverage
information in all multi-view images. To address these problems, we propose
the MVSRnet, which uses geometry information to extract sharp details from all
LR multi-view to support the SR of the LR input view. Specifically, the
proposed Geometry-Aware Reference Synthesis module in MVSRnet uses geometry
information and all multi-view LR images to synthesize pixel-aligned HR
reference images. Then, the proposed Dynamic High-Frequency Search network
fully exploits the high-frequency textural details in reference images for SR.
Extensive experiments on several benchmarks show that our method significantly
improves over the state-of-the-art approaches.

Super-resolution, Multi-view images

††copyright: acmcopyright††journalyear: 2022††copyright:
acmcopyright††conference: Proceedings of the 30th ACM International Conference
on Multimedia ; October 10–14, 2022; Lisboa, Portugal.††booktitle: Proceedings
of the 30th ACM International Conference on Multimedia (MM ’22), October
10–14, 2022, Lisboa, Portugal††price: 15.00††isbn:
978-1-4503-9203-7/22/10††doi: 10.1145/3503161.3547947††ccs: Computing
methodologies Computer vision††ccs: Computing methodologies Reconstruction
Figure 1. Multi-view image super-resolution. We propose GARS module for
extracting the HR details from all _LR Multi-View_. Then, our GARS module
synthesizes high-frequency textural details into _HR Reference Image_ that is
aligned with the bicubic of _LR Input View_. The word SQUARE is more textured
in _HR Reference Image_ than in the _LR Input View_.

##  1\. Introduction

Multi-view Multimedia (MVMM) uses numerous cameras to capture multi-view
images of the same scene from various angles. It is widely used in auto-
driving, street-view navigation, free-viewpoint video(Smolic et al., 2006; Lee
et al., 2015), and virtual reality(Brooks, 1999; Berg and Vance, 2017). Recent
MVMM applications call for high-resolution (HR) images to give a pleasing
visual experience and assist downstream tasks such as detection and tracking.
Nevertheless, HR multi-view images face substantial storage costs or bandwidth
constraints. To address this problem, we propose the Multi-View Image Super-
Resolution (MVISR) task, which increases the resolution of multi-view images
by using all low-resolution (LR) multi-view images in the same scene.

A naive approach to MVISR is to perform single image SR (SISR) (Dong et al.,
2016; Lim et al., 2017) on each view. They take a single view as input and
cannot utilize the relationship across all LR views. Other alternatives
include video SR (VSR) methods (Li et al., 2020; Chan et al., 2021). They
regard multi-views as video frames with little object motion, and so cannot
deal with the large-angle transformations between input views. Some reference-
based SR (Ref-SR) methods (Zhang et al., 2019b; Yang et al., 2020; Lu et al.,
2021a) can overcome angle constraints and extend the range of reference views.
However, Ref-SR is ineffective due to the difficulties involved with reference
image selection in all LR multi-views and reference alignment when the scale
and rotation differences between the Ref and LR input are large. In the end,
none of the approaches discussed previously can explicitly leverage all LR
multi-views and their associated geometry information.

Therefore, in this paper, we propose MVSRnet, which utilizes the geometry
information to exploit high-frequency (HF) reference information from all LR
views for each view SR. A camera that is closer to the objects can capture
sharper images than a distant one. It inspires us that a close-up view
contains more textural details that can provide sufficient reference for the
distant view. To realize this idea, we design Geometry-Aware Reference
Synthesis (GARS) module in MVSRnet. As illustrated in Figure 1, the word
SQUARE is blurred in the LR input view but sharp in other LR views. Our GARS
module takes LR multi-views as input to generate reference images with rich
textural details and well-aligned with the LR input view. To be specific, we
use structure-from-motion (Schonberger and Frahm, 2016) and multi-view stereo
(Schönberger et al., 2016; Yao et al., 2018) to extract scene geometry
information such as the camera pose and depth map. After that, GARS conducts
3D image warping to align all LR multi-views into the target view. Then, we
introduce a depth-guided patch-selection strategy to select close views on the
patch level and fuse them into HR multi-view reference images (MVRs).

Figure 2. Example of multi-view image super-resolution results. Our proposed
model outperforms SISR (RCAN (Zhang et al., 2018b) and HAN (Niu et al., 2020))
and VSR (MuCAN (Li et al., 2020) and IconVSR (Chan et al., 2021)) methods in
visual quality. For instance, by utilizing the synthesized reference image,
our model is able to recover finer details of the word SQUARE in the first
example and the center area of the tire in the second example.

On the other hand, MVRs synthesized from all LR views in GARS have artifacts
and some pixels that are not properly aligned with the LR input view due to
inaccuracies in the geometry information. Nearby views have a small difference
in viewing angle and hence suffer less from geometric inaccuracies. As a
result, we also synthesize near-view reference images (NVRs) with fewer
artifacts but fewer HF textural details using nearby views. Then, in order to
properly leverage MVRs and NVRs, we design the Raw Selection Module (RSM) and
Adaptation Selection Module (ASM) in our Dynamic High-Frequency Search (DHFS)
network to exploit the HF textural details by reducing the feature difference
dynamically. Finally, we perform experiments to demonstrate the proposed
MVSRnet SR capability, and some comparative experiments are shown in Figure 2.
Additionally, our work contributes to the Ref-SR methodology by proposing a
way for performing SR using multi-view LR images rather than HR images (Zhang
et al., 2019b) as reference.

This paper mainly has the following contributions:

  * •

For the first time, this paper proposes the MVSRnet, a learning-based network
that makes explicit use of geometric information to address the MVISR problem.
In addition, this work enriches the Reference-Based SR research.

  * •

The Geometry-Aware Reference Synthesis module synthesizes the HR reference
image from all LR multi-view images using geometry information. The reference
supports SR by containing rich high-frequency textural details and aligning
with the input view.

  * •

The Dynamic High-Frequency Search network is proposed for reconstructing the
high-resolution view by dynamically reducing the feature difference to exploit
high-frequency textural details from synthesized reference images.

##  2\. Related Work

###  2.1. Single Image Super-Resolution

Single Image Super-Resolution (SISR) (Dong et al., 2016; Lim et al., 2017;
Zhang et al., 2018b; Ledig et al., 2017; Wang et al., 2018; Ma et al., 2020)
aims to learn the mapping function between LR and HR images. However, SISR is
unable to recover HF details due to the destruction of HR textural clues in
the downsampling LR image.

Figure 3.  (a) The overview pipeline of the proposed MVSRnet. (b) The depth
value of truck door area in the warped depth map $D_{S\rightarrow T}^{hr}$ is
smaller (darker blue) than $D_{1\rightarrow T}^{hr}$, so the warped view
$I_{S\rightarrow T}^{hr}$ contains more truck door HR details than
$I_{1\rightarrow T}^{hr}$, corresponding to more truck door high-frequency
details in view $I_{S}^{lr}$ than $I_{1}^{lr}$.

###  2.2. Reference-Based Super-Resolution

Reference-based super-resolution (Ref-SR) approaches transfer the HR details
of the reference image to the LR input image. Several works (Shim et al.,
2020; Zheng et al., 2018) leverage deformable convolution (Dai et al., 2017;
Zhu et al., 2019) or estimated flow to align and extract reference features.
However, it will seriously affect the performance of these algorithms when
finding long-distance correspondence. The patch-based Ref-SR approaches (Zhang
et al., 2019b; Yan et al., 2020; Yang et al., 2020; Lu et al., 2021a; Jiang et
al., 2021) perform patch matching in the feature space by calculating the
cosine similarity between the LR patch and Ref patch. (Yang et al., 2020; Lu
et al., 2021a) formulate patch matching based on the transformer (Vaswani et
al., 2017). However, the difference in scale and rotation between the HR Ref
and LR inputs can cause matching errors (Jiang et al., 2021), which is serious
in multi-view settings. In addition, for the super-resolution of multi-view LR
images, there is no available view reference selection method. The SIFT-based
(Lowe, 1999) selection method used in constructing the CUFED5 (Zhang et al.,
2019b) dataset is only applicable to selecting HR references whose content is
similar to the LR input view.

###  2.3. Multi-View Super-Resolution

Several traditional approaches (Goldluecke and Cremers, 2009; Tsiminaki et
al., 2014) and learning-based approaches (Li et al., 2019; Richard et al.,
2019) study the multi-view texture SR problem. However, they only super-
resolve the texture of the object instead of the entire image. Stereo image SR
(Ying et al., 2020; Wang et al., 2019) utilizes additional information
provided from a second viewpoint, but it uses a much smaller number of
viewpoint than MVISR. Light field image SR (Bishop et al., 2009; Wanner and
Goldluecke, 2014; Zhang et al., 2019a; Wang et al., 2020) has also been
extensively studied in recent years, but the view span are limited compared to
the unstructured multi-views. Our MVSRnet differs from multi-view video SR (Lu
et al., 2021b; Li et al., 2016) in that they utilize HR reference frames,
whereas MVSRnet exclusively uses LR frames. However, obtaining HR images is
difficult due to substantial storage costs and bandwidth constraints in
reality. Sun _et al._ (Sun et al., 2021) addresses the space-angle SR problem,
but only two adjacent views are used to synthesize virtual view and SR. In
general, it is difficult to apply the methods discussed above to the MVISR
task directly.

##  3\. Proposed Method

This section describes the proposed network called MVSRnet, and the overview
of the MVSRnet is shown in Figure 3(a). We begin with a data preprocessing
step that estimates depth maps $\\{D_{i}^{lr}\\}_{i=1}^{S}$ and poses
$\\{P_{i}^{lr}\\}_{i=1}^{S}$ of all $S$ LR multi-views in Section 3.1. Then,
in Section 3.2, we introduce Geometry-Aware Reference Synthesis (GARS) that
uses geometry priors ($\\{D_{i}^{lr}\\}_{i=1}^{S}$ and
$\\{P_{i}^{lr}\\}_{i=1}^{S}$) to synthesize $V$ HR multi-view reference (MVR)
$\\{R_{i}^{M}\\}_{i=1}^{V}$ and HR near-view reference (NVR)
$\\{R_{i}^{N}\\}_{i=1}^{V}$. MVR stores more high-frequency (HF) textural
details than NVR since MVR is synthesised from all $S$ LR multi-view images
$\\{I_{i}^{lr}\\}_{i=1}^{S}$, not from $L$ nearby views like NVR. However,
there are fewer artifacts caused by erroneous warping in the NVR. Therefore,
in Section 3.3, we design Dynamic High-Frequency Search (DHFS) network that
dynamically selects and extracts the correct HF details in MVRs and NVRs to
super-resolve the LR input view $I_{T}^{lr}$ and get HR output view
$I_{T}^{hr}$. At last, loss functions are shown in Section 3.4.

###  3.1. Preprocessing

To obtain the geometry information of the multi-view scene, we utilize COLMAP
(Schonberger and Frahm, 2016) to compute camera poses by inputting all $S$ LR
multi-views. This is known as structure-from-motion (SfM), and we can get the
camera pose $\\{P_{i}^{lr}\\}_{i=1}^{S}$ of each viewpoint. The pose is
represented by the extrinsic parameters (rotation matrices and translation
vectors) and intrinsic parameters. We use the depth map of the LR input view
to align all LR multi-views to the LR input view. Therefore, we conduct the
multi-view stereo (MVS) in COLMAP to obtain the depth map for each LR view
$\\{D_{i}^{lr}\\}_{i=1}^{S}$ by inputting all LR multi-views and the SfM
output.

###  3.2. Geometry-Aware Reference Synthesis

The GARS module synthesizes references (Refs) from all multi-view images, and
we display the synthesis process of a Ref in Figure 3(b). The reference image
is aligned with the input view and contains high-frequency details.

Alignment Using Scene Geometry. When the reference image has a large
deformation in comparison to the input image, the alignment capabilities of
recent Ref-SR approaches are reduced. For instance, Figure 3 illustrates a
great differences in scale and rotation of the truck between the LR input view
$I_{T}^{lr}$ and view $I_{S}^{lr}$. Therefore, we use the multi-view scene
geometry to explicitly align all multi-view source images
$\\{I_{i}^{lr}\\}_{i=1}^{S}$ with the input view $I_{T}^{lr}$. Specifically,
we up-sample the depth map to the target output resolution (SR factor is
$\times 4$ in our experiment setting) based on the bicubic mode and modify the
camera intrinsic matrices according to the up-sampling factor. Then we apply
the 3D warping operation (McMillan, 1997) to obtain warped views
$\\{I_{i\rightarrow T}^{hr}\\}_{i=1}^{S}$ as follows:

(1) $I_{i\rightarrow
T}^{hr}=W\left(D_{i}^{hr},D_{T}^{hr},P_{i}^{hr},P_{T}^{hr},I_{i}^{lr}\right),$

where $i$ and $T$ refer to $i$-th multi-view images and input view. $D^{hr}$
and $P^{hr}$ denote the depth maps and the camera pose at the same resolution
as the target output solution. Specifically, the pixel coordinate
correspondence between the multi-view images and the input view in the 3D
warping operation is as the following formula:

(2) $D_{i}(p_{i})*p_{i}=K_{i}(R_{r}K_{T}^{-1}D_{T}(p_{T})*p_{T}+t_{r}),$

where $p$ denotes pixel homogeneous coordinates. Camera intrinsic matrices
$K$, rotation matrices $R$ and translation matrices $t$ are provided by camera
posed $P$ as presented in Eq.(1). $R_{r}=R_{i}R_{T}^{T}$,
$t_{r}=t_{i}-R_{r}t_{T}$ are relative rotation and translation matrices.

It is noted that we can directly apply 3D warping operation on LR depth maps
without up-sampling and obtain LR aligned views $\\{I_{i\rightarrow
T}^{lr}\\}_{i=1}^{S}$. However, the general sampling modes in warping
operation, such as nearest and bilinear, blurs and destructs high-frequency
details using the weighted average method when dealing with overlapping
pixels. One possible solution is to use a pre-trained network as a feature
extractor to conduct 3D warping operations in the LR feature space. However,
this leads to both spatially and temporally inefficient as hundreds of views
need to be extracted features in each training iteration.

HF Details Extraction Using Depth-Guided Patch-Selection Strategy. We should
extract useful HF details from warped views $\\{I_{i\rightarrow
T}^{hr}\\}_{i=1}^{S}$ that contain redundant details as the warped view
contains the LR details area as well. We assume that the view taken at close
range has more high-frequency textural details than the distant view.
Therefore, we apply the 3D warping operation on source depth maps
$W(D_{i}^{hr},D_{T}^{hr},P_{i}^{hr},$ $P_{T}^{hr},D_{i}^{lr})$ to obtain HR
warped depth maps $\\{D_{i\rightarrow T}^{hr}\\}_{i=1}^{S}$. Warped depth maps
present distance geometry information, i.e., the distance between the object
and the camera. The smaller the depth value, the closer the object in source
view $I_{i}^{lr}$ is to the camera. As a result, we can leverage the HR warped
depth maps to locate the HR details area in the warped views
$\\{I_{i\rightarrow T}^{hr}\\}_{i=1}^{S}$. For example, Figure 3(b) shows that
the truck door has smaller depth values (darker blue) in the warped depth map
$D_{S\rightarrow T}^{hr}$ than $D_{1\rightarrow T}^{hr}$. Hence, the HR warped
view $I_{S\rightarrow T}^{hr}$ contains more truck door HR details than
$I_{1\rightarrow T}^{hr}$, corresponding to more truck door high-frequency
details in view $I_{S}^{lr}$ than $I_{1}^{lr}$.

We unfold each warped depth map into patches $p_{i,j}$, $(i\in$
$[1,S],j\in\left[1,\left(hr_{h}\times hr_{w}\right)/(ps^{2})\right])$,
$hr_{h}$, $hr_{w}$, and $ps$ denote the target HR height, target HR width, and
patch size. $j$ denotes the $j$-th patch index. We also unfold each warped
view, and denote it as $I_{i\rightarrow T,j}^{hr}$. Then, we assign the depth
value in each patch to its patch mean $p_{i,j}^{mean}=mean(p_{i,j})$. This
alleviates the error of the depth value averaged in the edge area due to the
up-sampling and warping operation. Then, we obtain the high-frequency index
map $H$ from the index with the smallest depth value in $p_{i,j}^{mean}$ as
follows:

(3) ${H}_{j}=\underset{i}{\arg\min}\ p_{i,j}^{mean}.$

$H_{j}$ refers to the high-frequency patch index of warped views, and we
extract the high-frequency patch as the following formula:

(4) ${R}_{j}=I_{i\rightarrow T,j}^{hr},\ i=H_{j}.$

Now we can obtain the multi-view reference image $R$ by folding the multi-view
reference patches ${R}_{j}$.

Multiple References Synthesis. It is impossible to integrate all of the high-
frequency details into a reference image. As a result, we synthesize $V$
multi-view reference images (MVRs) denoted as $\\{R_{i}^{M}\\}_{i=1}^{V}$
using those the top $V$ high-frequency index maps. However, the depth errors
generated by MVS estimation and 3D warping causes inaccurate warped depth
values, leading to incorrect MVR selection and alignment. In addition, the
large angle difference between views magnifies alignment errors, resulting
artifacts in MVR. Therefore, we synthesize $V$ near-view reference images
(NVRs) $\\{R_{i}^{N}\\}_{i=1}^{V}$ from $L$ nearby views that have a small
viewing angle gap and similar lighting conditions as the LR input view. NVRs
suffer less from geometric inaccuracies, and the selection criteria for nearby
views are based on their order in the dataset. NVR includes less high-
frequency details than MVR, but it also contains fewer alignment errors, and
artifacts. As a result, both MVRs and NVRs can be used to cross-reference to
correct selection errors and accurately supply HF details to the SR model.

Figure 4.  Overview of the proposed Dynamic High-Frequency Search (DHFS)
network, which contains three Reference Extraction Module (REM). Each REM
includes a Raw Selection Module (RSM) and an Adaptation Selection Module (ASM)
to exploit the high-frequency textural details in reference images
dynamically.

###  3.3. Dynamic High-Frequency Search

MVR has both more HF textural details and more artifacts than NVR, and that
NVR is the inverse. Therefore, we propose the DHFS network to maximize the use
of references (MVR and NVR). As shown in Figure 4(a), we first concatenate the
MVRs and NVRs to the bicubic input view at the channel level, and use a
learnable feature extractor to extract their multi-scale ($\times 1$, $\times
2$, and $\times 4$) textural feature ($F_{M}^{ref}$ and $F_{N}^{ref}$). On the
other hand, we obtain the textural features $F_{T}$ from LR input view using
several residual layers. Then, we use three Reference Extraction Module (REM)
that do not share parameters. Each REM includes a Raw Selection Module (RSM)
and an Adaptation Selection Module (ASM) to exploit the high-frequency
textural details in reference images dynamically, as shown in Figure 4(b). We
adopt PixelShuffle (Shi et al., 2016) layer for up-sampling with factor 2.

Raw Selection Module. RSM first extracts high-frequency textural details as
residues from MVR features ($F_{T},F_{M}^{ref}$) and NVR features
($F_{T},F_{M}^{ref}$), and adds them to the feature of the LR input view
$F_{T}$, getting $F_{T}^{m}$ and $F_{T}^{n}$ respectively. Then, it aggregates
them into the final output $F_{T}^{rsm}$ via selection map $A_{m}$ and
$A_{n}$. Figure 4(c) shows this process, and the formula is as follows:

(5) $\displaystyle F_{T}^{m}=F_{T}+{Conv}_{1}\left(F_{T},F_{M}^{ref}\right)$
$\displaystyle F_{T}^{n}=F_{T}+{Conv}_{1}\left(F_{T},F_{N}^{ref}\right)$
$\displaystyle
A_{M},A_{N}={Softmax}\left({Conv}_{2}\left(F_{T}^{m},F_{T}^{n},F_{T}\right)\right)$
$\displaystyle
F_{T}^{rsm}={Resnet}\left(A_{M}F_{T}^{m}+A_{N}F_{T}^{n}\right).$

We find that RSM extracts more reference features from NVRs than from MVRs.
Reference features are generated by the feature extractor rather than residual
layers such as $F_{T}$, resulting in a feature difference between reference
feature and $F_{T}$. Due to the feature difference, it is difficult for the
network to extract high-frequency textural details from MVR features which
contain numerous artifacts and alignment errors. However, MVRs contain rich
high-frequency textural details that are extremely useful for reconstructing
the high-quality SR output view. As a result, we propose using ASM to reduce
feature difference and feature artifacts, increasing DHFS capabilities of
extracting MVR high-frequency details.

Adaptation Selection Module. As shown in Figure 4(d), ASM first adds residues,
calculated from ($F_{M}^{ref},F_{T}^{rsm}$) and ($F_{N}^{ref},F_{T}^{rsm}$),
to $F_{M}^{ref}$ and $F_{N}^{ref}$. Then, utilizing numerous residual layers,
ASM adapts reference features to $F_{T}^{rsm}$, reducing the feature
difference and artifact features in reference features. Finally, it takes the
adapted reference feature ($F_{M}^{a}$ and $F_{N}^{a}$) and $F_{T}^{rsm}$ as
the input of another RSM to extract high-frequency feature details. We define
the process in ASM as follows:

(6) $\displaystyle
F_{M}^{a}={Resnet}\left({Conv}_{2}\left(F_{M}^{ref}+{Conv}_{1}\left(f_{M}^{ref},F_{T}^{rsm}\right)\right)\right)$
$\displaystyle
F_{N}^{a}={Resnet}\left({Conv}_{2}\left(F_{N}^{ref}+{Conv}_{1}\left(F_{N}^{ref},F_{T}^{rsm}\right)\right)\right)$
$\displaystyle
F_{T}^{asm}={RSM}\left({F}_{M}^{a},F_{N}^{a},F_{T}^{rsm}\right).$

As a result, dynamic extraction via RSM and ASM makes effective use of the HF
textural details contained in the Refs.

| Model | Train | Playground | M60 | Truck | Total
---|---|---|---|---|---|---
SISR | EDSR (Lim et al., 2017) | 28.05 / 0.8294 / 0.226 | 28.33 / 0.8070 / 0.318 | 31.38 / 0.9072 / 0.165 | 28.19 / 0.8535 / 0.209 | 28.99 / 0.8493 / 0.230
RCAN (Zhang et al., 2018b) | 28.56 / 0.8351 / 0.228 | 29.68 / 0.8117 / 0.320 | 32.41 / 0.9126 / 0.163 | 28.93 / 0.8583 / 0.213 | 29.89 / 0.8544 / 0.231
HAN (Niu et al., 2020) | 28.43 / 0.8321 / 0.222 | 29.59 / 0.8098 / 0.315 | 32.25 / 0.9103 / 0.167 | 28.77 / 0.8551 / 0.207 | 29.76 / 0.8518 / 0.228
ESRGAN (Wang et al., 2018) | 25.53 / 0.7512 / 0.109 | 25.48 / 0.6810 / 0.151 | 29.52 / 0.8666 / 0.091 | 26.00 / 0.7893 / 0.097 | 26.63 / 0.7720 / 0.112
SPSR (Ma et al., 2020) | 25.68 / 0.7495 / 0.106 | 26.61 / 0.6900 / 0.133 | 29.85 / 0.8631 / 0.087 | 26.49 / 0.7920 / 0.090 | 27.16 / 0.7736 / 0.104
VSR | RBPN (Haris et al., 2019) | 28.35 / 0.8314 / 0.236 | 29.66 / 0.8131 / 0.321 | 32.23 / 0.9117 / 0.171 | 28.77 / 0.8563 / 0.217 | 29.75 / 0.8531 / 0.236
RSDN (Isobe et al., 2020) | 27.68 / 0.8148 / 0.240 | 29.05 / 0.7954 / 0.314 | 31.05 / 0.8886 / 0.189 | 28.23 / 0.8402 / 0.218 | 29.00 / 0.8347 / 0.240
MuCAN (Li et al., 2020) | 28.46 / 0.8363 / 0.215 | 29.66 / 0.8148 / 0.296 | 32.56 / 0.9175 / 0.150 | 28.90 / 0.8607 / 0.200 | 29.90 / 0.8573 / 0.215
IconVSR (Chan et al., 2021) | 28.31 / 0.8319 / 0.227 | 29.69 / 0.8172 / 0.310 | 32.36 / 0.9157 / 0.151 | 28.62 / 0.8538 / 0.218 | 29.75 / 0.8547 / 0.227
Ref-SR | TTSR (Yang et al., 2020) | 26.28 / 0.7661 / 0.118 | 27.08 / 0.7129 / 0.155 | 29.97 / 0.8626 / 0.108 | 26.80 / 0.8016 / 0.115 | 27.53 / 0.7858 / 0.124
TTSR-rec (Yang et al., 2020) | 28.16 / 0.8277 / 0.227 | 29.32 / 0.8039 / 0.319 | 31.84 / 0.9060 / 0.162 | 28.51 / 0.8509 / 0.214 | 29.46 / 0.8471 / 0.231
MASA (Lu et al., 2021a) | 25.48 / 0.7301 / 0.142 | 26.37 / 0.6822 / 0.152 | 28.89 / 0.8333 / 0.120 | 25.88 / 0.7722 / 0.121 | 26.66 / 0.7545 / 0.134
MASA-rec (Lu et al., 2021a) | 28.43 / 0.8375 / 0.208 | 29.41 / 0.8073 / 0.309 | 32.10 / 0.9119 / 0.152 | 28.74 / 0.8586 / 0.205 | 29.67 / 0.8538 / 0.218
$C^{2}$-Matching (Jiang et al., 2021) | 28.13 / 0.8247 / 0.150 | 28.95 / 0.7844 / 0.210 | 31.71 / 0.9004 / 0.119 | 28.37 / 0.8441 / 0.151 | 29.29 / 0.8384 / 0.158
$C^{2}$-Matching-rec (Jiang et al., 2021) | 28.71 / 0.8473 / 0.201 | 29.62 / 0.8160 / 0.305 | 32.32 / 0.9155 / 0.147 | 28.92 / 0.8648 / 0.194 | 29.89 / 0.8609 / 0.212
| MVSRnet | 27.57 / 0.8222 / 0.085 | 27.99 / 0.7694 / 0.104 | 31.73 / 0.9065 / 0.068 | 28.03 / 0.8505 / 0.082 | 28.83 / 0.8372 / 0.085
| MVSRnet-rec |  29.04 / 0.8588 / 0.183 |  29.93 / 0.8371 / 0.275 |  32.88 / 0.9278 / 0.127 |  29.19 / 0.8757 / 0.178 |  30.26 / 0.8748 / 0.191

Table 1.  Quantitative comparison on Tanks and Temples dataset. $\uparrow$PSNR
/ $\uparrow$SSIM / $\downarrow$LPIPS are used for evaluation. SISR, VSR, and
Ref-SR methods are grouped separately. ESRGAN, SPSR, TTSR, MASA,
$C^{2}$-Matching, and MVSRnet are GAN-based methods and the best results are
in bold.

| Model | BlendedMVS | GTAV
---|---|---|---
SISR | EDSR (Lim et al., 2017) | 27.80 / 0.7198 / 0.393 | 30.93 / 0.8478 / 0.211
RCAN (Zhang et al., 2018b) | 27.90 / 0.7227 / 0.393 | 31.23 / 0.8530 / 0.212
HAN (Niu et al., 2020) | 27.86 / 0.7211 / 0.396 | 31.13 / 0.8515 / 0.206
ESRGAN (Wang et al., 2018) | 25.50 / 0.6315 / 0.173 | 28.26 / 0.7746 / 0.105
SPSR (Ma et al., 2020) | 25.72 / 0.6406 / 0.168 | 28.37 / 0.7739 / 0.102
VSR | RBPN (Haris et al., 2019) | 27.79 / 0.7179 / 0.408 | 31.27 / 0.8559 / 0.218
RSDN (Isobe et al., 2020) | 26.38 / 0.6412 / 0.480 | 30.26 / 0.8365 / 0.223
MuCAN (Li et al., 2020) | 27.72 / 0.7162 / 0.400 | 31.24 / 0.8585 / 0.196
IconVSR (Chan et al., 2021) | 27.53 / 0.7089 / 0.408 | 31.41 / 0.8633 / 0.205
Ref-SR | TTSR (Yang et al., 2020) | 26.09 / 0.6567 / 0.201 | 28.80 / 0.7914 / 0.121
TTSR-rec (Yang et al., 2020) | 27.68 / 0.7193 / 0.396 | 30.79 / 0.8485 / 0.216
MASA (Lu et al., 2021a) | 25.84 / 0.6468 / 0.220 | 28.05 / 0.7699 / 0.131
MASA-rec (Lu et al., 2021a) | 27.82 / 0.7335 / 0.375 | 31.12 / 0.8613 / 0.196
$C^{2}$-Matching (Jiang et al., 2021) | 27.24 / 0.7039 / 0.281 | 30.57 / 0.8396 / 0.153
$C^{2}$-Matching-rec (Jiang et al., 2021) | 27.99 / 0.7377 / 0.370 | 31.21 / 0.8628 / 0.194
MVSRnet | 26.90 / 0.7080 / 0.182 | 30.27 / 0.8462 / 0.085
| MVSRnet-rec |  28.03 / 0.7457 / 0.360 |  31.60 / 0.8775 / 0.179

Table 2.  Quantitative comparison on BlendedMVS and GTAV datasets.
$\uparrow$PSNR / $\uparrow$SSIM / $\downarrow$LPIPS are used for evaluation.
SISR, VSR, and Ref-SR methods are grouped separately.

###  3.4. Loss Functions

Similar to most SR models, we adopt the reconstruction loss
$\mathcal{L}_{{rec}}$, adversarial loss $\mathcal{L}_{{adv}}$, and perceptual
loss $\mathcal{L}_{{per}}$ in our training stage. The $\mathcal{L}_{{adv}}$
and $\mathcal{L}_{{per}}$ are used to improve the visual quality, and we
define the overall loss as follows:

(7)
$\mathcal{L}_{{overall}}=\mathcal{L}_{{rec}}+\lambda_{adv}\mathcal{L}_{adv}+\lambda_{{per}}\mathcal{L}_{{per}}$

Specifically, we employ $\ell_{1}$-norm for reconstruction loss and adopt the
WGAN-GP (Gulrajani et al., 2017) structure to obtain adversarial loss. For
perceptual loss (Johnson et al., 2016), it is calculated on the VGG19
(Simonyan and Zisserman, 2015) features of conv3-2, conv4-2 ,and conv5-2.

##  4\. Experiments

###  4.1. Datasets and Training Details

Datasets. As the multi-view image super-resolution problem focuses on multi-
view settings, we compare our MVSRnet to the state of the art on three
challenging multi-view datasets: Tanks and Temples (Knapitsch et al., 2017),
BlendedMVS (Yao et al., 2020), and GTAV (Huang et al., 2018). The real-world
images captured in Tanks and Temples dataset and BlendedMVS are irregular
distribution. GTAV is a photo-realistic synthetic dataset containing urban
scenes captured in the video game Grand Theft Auto V. Specifically, we use 17
scenes from Tanks and Temples for training and 4 (Train, Playground, M60, and
Truck) for testing, the same as (Riegler and Koltun, 2020). In addition, the
camera pose and the depth map we use are provided by (Riegler and Koltun,
2020) for Tanks and Temple. Additionally, we evaluate ten scenes in BlendedMVS
and all scenes in GTAV, proving the generalization capacity of our MVSRnet.
The number of views in each scene of the Tanks and Temples and BlendedMVS
datasets is concentrated in 300-400 and 60-150, and the GTAV dataset is 100.
We set the SR factor as $\times 4$, and we generate LR multi-view images and
depth maps by bicubic and modify the camera intrinsic matrices according to
the SR factor.

Figure 5.  Qualitative comparisons. The top three examples are Playground,
M60, and Train scene in Tanks and Temple dataset. The fourth and fifth example
come from BlenedMVS and GTAV datasets, respectively. Zoom in for better
visualization.

Training Details. We first utilize the Geometry-Aware Reference Synthesis
module to synthesize MVRs and NVRs for each LR input view. Specifically, we
synthesize $V$ MVRs and $V$ NVRs from all LR multi-view images and $L$
adjacent views, respectively. $V$ is equal to 6, and we empirically set the
$L$ to 6 . Then, we use synthesized reference images and the LR input view to
train the Dynamic High-Frequency Search network from scratch on an Nvidia
GeForce RTX 3090 GPU. We implement our method in Pytorch, and the total number
of training iterations is 200,000, with a batch size of 2 and a LR patch size
of $80\times 160$ pixels. In addition, we decay the learning rate from 1e-4 to
1e-7 in a cosine annealing way, and we choose ADAM with parameters
$\beta_{1}=0.9$, and $\beta_{2}=0.999$ as optimizer. We set $\lambda_{adv}$
and $\lambda_{{per}}$ in Eq.(7) to 5e-3 and 1e-2.

###  4.2. Comparison with State-of-the-Art Methods

We compare our MVSRnet with recent SISR, VSR, and Ref-SR methods. We use five
SISR methods, including EDSR (Lim et al., 2017), RCAN (Zhang et al., 2018b),
HAN (Niu et al., 2020), ESRGAN (Wang et al., 2018), and SPSR (Ma et al.,
2020), four VSR methods, including RBPN (Haris et al., 2019), RSDN (Isobe et
al., 2020), MuCAN (Li et al., 2020), and IconVSR (Chan et al., 2021), and
three Ref-SR methods, including TTSR (Yang et al., 2020), MASA (Lu et al.,
2021a), and $C^{2}$-Matching (Jiang et al., 2021). We use the best results
from their pre-trained and fine-tuned models using our Tanks and Temples
dataset for display. Because RSDN and MuCAN do not supply training code, we
rely on the pre-trained models they provide. We employ an MVR synthesized from
the top-1 HF index maps as the input to Ref-SR methods because these methods
are incapable of processing hundreds of reference images. Our compared Ref-SR
methods may concatenate the references in a row or column to conduct multi-
reference SR, but the GPU runs out of memory when employing two synthesized
references. ESRGAN, SPSR, TTSR, MASA, and $C^{2}$-Matching are all GAN-based
models, whereas the remaining models rely solely on reconstruction loss. To
ensure a fair comparison, we train our model with overall loss (Eq.(7)) and
solely reconstruction loss and refer to them as MVSRnet and MVSRnet-rec,
respectively.

Quantitative Comparison. Table 1 presents the quantitative results of the
Tanks and Temples dataset. In Table 2, we also verify the generalization
capacity of our MVSRnet using BlendedMVS and GTAV, and our approach continues
to perform the best. The PSNR of GAN-based $C^{2}$-Matching in Table 1 and
Table 2 is higher than our MVSRnet, while MVSRnet reduces the LPIPS (Zhang et
al., 2018a) error by around 35%-46% when compared to $C^{2}$-Matching. LPIPS
is validated to human perception and is commonly used as a metric for
evaluating GAN-based models. While our LPIPS is comparable to that of SPSR in
BlendedMVS, our PSNR and SSIM are 1.18dB and 0.0674 higher, respectively. In
summary, our proposed method outperforms the state of the art by a significant
margin.

Qualitative Evaluation. The visual results shown in Figure 5 reveal that our
method is capable of producing finer details, and they lead to two
conclusions. (a) Our reference image generated from all LR multi-view images
is more beneficial for SR than the image self-reference (SISR) and adjacent
views (VSR). For example, the reference image of the fourth example contains
more realistic horse details than the bicubic input view. Thus, using our
reference image, MVSRnet generates more fine textural details of the horse
when compared to SISR methods (ESRGAN and SPSR) and VSR methods (MuCAN and
IconVSR). (b) Our DHFS network, including RSM and ASM, shows its effectiveness
and robustness in identifying and extracting useful features from the
reference images. Specifically, because our synthesized reference image is
used as an input, Ref-SR methods such as TTSR, MASA, and $C^{2}$-Matching can
produce textural results. They are, however, not robust to artifacts in the
synthesized reference. For example, in Figure 5, the reference image of the
third example contains blur artifacts, resulting in more blur areas in the
$C^{2}$-Matching image than in our MVSRnet image.

Figure 6.  Ablation study. (a) Effectiveness of MVR and NVR. Model 1 uses
neither MVR nor NVR, and Model 2 and 3 only use MVR or NVR. (b) Effectiveness
of RSM and ASM. The upper left and upper right are the selection map $A_{M}$
of features $F_{T}^{M}$ from RSM in Model 5 and from ASM in Model 6,
respectively. The lower left and lower right are the selection map $A_{M}$
from RSM and from ASM in MVSRnet.

###  4.3. Ablation Study

Muti-View Reference Image (MVR) and Near-View Reference Image (NVR). We
analyze the MVR and NVR that have complementary effects on SR tasks. The
ablation Model 1 does not input both MVR and NVR, and Model 2 and Model 3 do
not input NVR and MVR, respectively. As illustrated in Figure 6(a), MVR
contains high-frequency details (sharp number ‘713’) as well as noticeable
artifacts (the incorrect structure of letter ‘PAC’). In contrast, NVR has
fewer artifacts and details of ‘PAC’ and ‘713’ than MVR, but there are still
more high-frequency details than the bicubic input view. We discover that
MVSRnet recovers the exact number ‘713’ than Model 3 and produces fewer ‘PAC’
artifacts than Model 2. Additionally, we conduct quantitative ablation study,
and Table 3 reveals that MVSRnet has superior PSNR performance than Model 2
and Model 3 by 0.38dB and 0.45 dB, respectively. These experiments fully
demonstrate the complementary effects of MVR and NVR.

In addition, we search for the optimal number $V$ of MVR and NVR. The LR
details and artifacts areas grow in relation to the number $V$ of reference
images, making it challenging for the network to identify and extract relevant
high frequencies details. We perform ablation studies with $V$ to 1, 2, 4, 6,
and 8, and we achieve the best performance when we set $V$ to 6, as shown in
Table 4. We also conduct an ablation study on the hyper-parameter of patch
size $ps$ as 4/8/16/32/64 in GARS module, and achieve the best performance
when we set $ps$ to 16, as shown in Table 4. The reason is that when the patch
size is too small, there are numerous selection mistakes and block-boundary
artifacts in the synthesized reference image. When the block size is too
large, the selected patches are blurry and lack a significant amount of high-
frequency texture details.

Raw Selection Module (RSM) and Adaptation Selection Module (ASM). The ablation
Model 4 does not include both RSM and ASM, and Model 5 and Model 6 do not
include ASM and RSM, respectively. Figure 6(b) shows that selected areas
(denoted as white) in the selection map of ‘ASM in MVSRnet’ are more than ‘RSM
in Model 5’ and ‘ASM in Model 6’, indicating that more MVR HR feature details
are selected for SR. This is because a small difference between features can
help the network identify and extract useful features. As mentioned in Section
3.3, ASM implicitly decreases the difference between reference features and
input view features and reduces artifact features in reference features,
making it easier for the network to extract useful features in MVR. RSM, on
the other hand, uses a residual process to adapt input view features to
reference features, as indicated in Eq.(5). The different ways ASM and RSM
incorporate features makes feature extraction more sufficient. Thus, combining
RSM and ASM significantly improves the power of DHFS to extract and identify
HF details when compared to using only RSM or ASM. We also verify them using
the quantitative evaluation displayed in Table 3. MVSRnet improves the PSNR
value by 0.12dB and 0.11dB, respectively, when compared to Model 5 and 6,
demonstrating the usefulness of using both RSM and ASM.

Model | MVR | NVR | RSM | ASM | Parameters | Tanks and Temple
---|---|---|---|---|---|---
Model 1 |  |  | ✓ | ✓ | 19.51M | 27.33 / 0.7734 / 0.108
Model 2 | ✓ |  | ✓ | ✓ | 19.56M | 28.45 / 0.8215 / 0.089
Model 3 |  | ✓ | ✓ | ✓ | 19.56M | 28.38 / 0.8182 / 0.089
Model 4 | ✓ | ✓ |  |  | 19.73M | 28.68 / 0.8323 / 0.085
Model 5 | ✓ | ✓ | ✓ |  | 19.62M | 28.71 / 0.8324 / 0.084
Model 6 | ✓ | ✓ |  | ✓ | 19.46M | 28.72 / 0.8340 / 0.083
MVSRnet | ✓ | ✓ | ✓ | ✓ | 19.42M |  28.83 / 0.8372 / 0.085

Table 3.  Quantitative ablation study on the MVR and NVR and quantitative
ablation study on the RSM and ASM.

$V$ | Tanks & Temple | Patch Size | Tanks & Temple
---|---|---|---
1 | 28.49 / 0.8252 / 0.087 | 4 | 28.72 / 0.8335 / 0.082
2 | 28.73 / 0.8353 / 0.085 | 8 | 28.76 / 0.8336 / 0.084
4 | 28.71 / 0.8334 / 0.085 | 16 |  28.83 / 0.8372 / 0.085
6 |  28.83 / 0.8372 / 0.085 | 32 | 28.73 / 0.8323 / 0.086
8 | 28.56 / 0.8289 / 0.087 | 64 | 28.56 / 0.8273 / 0.087

Table 4. Columns 1 and 2: ablation study of the optimal number $V$ of MVR and
$V$ NVR. Columns 3 and 4: ablation study of the patch size $ps$ in GARS
module.

##  5\. Conclusion

We propose the MVSRnet for multi-view image super-resolution that aims to
super-resolve each view by utilizing all views as references. Our Geometry-
Aware Reference Synthesis module, in particular, can synthesize reference
images from all LR multi-view images by explicitly using geometry information.
The reference image contains high-frequency details and is aligned with the
input view. Then we implement a Dynamic High-Frequency Search network which
includes the Raw Selection Module and Adaptation Selection Module, to
dynamically select useful features in the synthesized reference features for
super-resolution. We have achieved best experimental results compared with
state-of-the-art methods and this work enriches the Reference-Based SR
research.

## References

  * (1)
  * Berg and Vance (2017) Leif Berg and Judy Vance. 2017.  Industry use of virtual reality in product design and manufacturing: a survey.  _Virtual Reality_ 21 (03 2017).  https://doi.org/10.1007/s10055-016-0293-9
  * Bishop et al. (2009) Tom E. Bishop, Sara Zanetti, and Paolo Favaro. 2009\.  Light field superresolution. In _2009 IEEE International Conference on Computational Photography (ICCP)_. 1–9.  https://doi.org/10.1109/ICCPHOT.2009.5559010
  * Brooks (1999) F.P. Brooks. 1999\.  What’s real about virtual reality?  _IEEE Computer Graphics and Applications_ 19, 6 (1999), 16–27.  https://doi.org/10.1109/38.799723
  * Chan et al. (2021) Kelvin C.K. Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. 2021.  BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 4947–4956.
  * Dai et al. (2017) Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. 2017.  Deformable Convolutional Networks. In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Dong et al. (2016) Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2016\.  Image Super-Resolution Using Deep Convolutional Networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 38, 2 (2016), 295–307.  https://doi.org/10.1109/TPAMI.2015.2439281
  * Goldluecke and Cremers (2009) Bastian Goldluecke and Daniel Cremers. 2009.  Superresolution texture maps for multiview reconstruction. In _2009 IEEE 12th International Conference on Computer Vision_. 1677–1684.  https://doi.org/10.1109/ICCV.2009.5459378
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014\.  Generative Adversarial Nets. In _NIPS_. 2672–2680.
  * Gulrajani et al. (2017) Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. 2017\.  Improved Training of Wasserstein GANs. In _NIPS_. 5769–5779.  http://papers.nips.cc/paper/7159-improved-training-of-wasserstein-gans
  * Haris et al. (2019) Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. 2019\.  Recurrent Back-Projection Network for Video Super-Resolution. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Huang et al. (2018) Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang. 2018.  DeepMVS: Learning Multi-View Stereopsis. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Isobe et al. (2020) Takashi Isobe, Xu Jia, Shuhang Gu, Songjiang Li, Shengjin Wang, and Qi Tian. 2020\.  Video Super-Resolution with Recurrent Structure-Detail Network. In _Computer Vision – ECCV 2020_. Cham, 645–660.
  * Jiang et al. (2021) Yuming Jiang, Kelvin C.K. Chan, Xintao Wang, Chen Change Loy, and Ziwei Liu. 2021.  Robust Reference-Based Super-Resolution via C2-Matching. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 2103–2112.
  * Johnson et al. (2016) Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In _Computer Vision – ECCV 2016_. Cham, 694–711.
  * Knapitsch et al. (2017) Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. 2017.  Tanks and temples: benchmarking large-scale scene reconstruction.  _ACM Trans. Graph._ 36, 4 (2017), 78:1–78:13.
  * Ledig et al. (2017) Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Lee et al. (2015) Chuen-Chien Lee, Ali Tabatabai, and Kenji Tashiro. 2015\.  Free viewpoint video (FVV) survey and future research direction.  _APSIPA Transactions on Signal and Information Processing_ 4 (2015), e15.  https://doi.org/10.1017/ATSIP.2015.18
  * Li et al. (2020) Wenbo Li, Xin Tao, Taian Guo, Lu Qi, Jiangbo Lu, and Jiaya Jia. 2020\.  MuCAN: Multi-correspondence Aggregation Network for Video Super-Resolution. In _Computer Vision – ECCV 2020_. Cham, 335–351.
  * Li et al. (2016) Yawei Li, Xiaofeng Li, Zhizhong Fu, and Wenli Zhong. 2016\.  Multiview Video Super-Resolution via Information Extraction and Merging. In _Proceedings of the 24th ACM International Conference on Multimedia_ (Amsterdam, The Netherlands) _(MM ’16)_. New York, NY, USA, 446–450.  https://doi.org/10.1145/2964284.2967260
  * Li et al. (2019) Yawei Li, Vagia Tsiminaki, Radu Timofte, Marc Pollefeys, and Luc Van Gool. 2019.  3D Appearance Super-Resolution With Deep Learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Lim et al. (2017) Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017.  Enhanced Deep Residual Networks for Single Image Super-Resolution. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_.
  * Lowe (1999) D.G. Lowe. 1999\.  Object recognition from local scale-invariant features. In _Proceedings of the Seventh IEEE International Conference on Computer Vision_ , Vol. 2. 1150–1157 vol.2.  https://doi.org/10.1109/ICCV.1999.790410
  * Lu et al. (2021a) Liying Lu, Wenbo Li, Xin Tao, Jiangbo Lu, and Jiaya Jia. 2021a.  MASA-SR: Matching Acceleration and Spatial Adaptation for Reference-Based Image Super-Resolution. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 6368–6377.
  * Lu et al. (2021b) Shao-Ping Lu, Sen-Mao Li, Rong Wang, Gauthier Lafruit, Ming-Ming Cheng, and Adrian Munteanu. 2021b.  Low-Rank Constrained Super-Resolution for Mixed-Resolution Multiview Video.  _IEEE Transactions on Image Processing_ 30 (2021), 1072–1085.  https://doi.org/10.1109/TIP.2020.3042064
  * Ma et al. (2020) Cheng Ma, Yongming Rao, Yean Cheng, Ce Chen, Jiwen Lu, and Jie Zhou. 2020\.  Structure-Preserving Super Resolution With Gradient Guidance. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * McMillan (1997) Leonard McMillan. 1997\.  _An image-based approach to three-dimensional computer graphics_.  Ph. D. Dissertation. Citeseer.
  * Niu et al. (2020) Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen. 2020.  Single Image Super-Resolution via a Holistic Attention Network. In _Computer Vision – ECCV 2020_. 191–207.
  * Richard et al. (2019) Audrey Richard, Ian Cherabier, Martin R. Oswald, Vagia Tsiminaki, Marc Pollefeys, and Konrad Schindler. 2019.  Learned Multi-View Texture Super-Resolution. In _2019 International Conference on 3D Vision (3DV)_. 533–543.  https://doi.org/10.1109/3DV.2019.00065
  * Riegler and Koltun (2020) Gernot Riegler and Vladlen Koltun. 2020.  Free View Synthesis. In _European Conference on Computer Vision_.
  * Riegler and Koltun (2021) Gernot Riegler and Vladlen Koltun. 2021.  Stable View Synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 12216–12225.
  * Schonberger and Frahm (2016) Johannes L. Schonberger and Jan-Michael Frahm. 2016.  Structure-From-Motion Revisited. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Schönberger et al. (2016) Johannes L. Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. 2016.  Pixelwise View Selection for Unstructured Multi-View Stereo. In _Computer Vision – ECCV 2016_. Cham, 501–518.
  * Shi et al. (2016) Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016\.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Shim et al. (2020) Gyumin Shim, Jinsun Park, and In So Kweon. 2020.  Robust Reference-Based Super-Resolution With Similarity-Aware Deformable Convolution. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman. 2015.  Very Deep Convolutional Networks for Large-Scale Image Recognition. In _International Conference on Learning Representations_.
  * Smolic et al. (2006) Aljoscha Smolic, Karsten Mueller, Philipp Merkle, Christoph Fehn, Peter Kauff, Peter Eisert, and Thomas Wiegand. 2006.  3D Video and Free Viewpoint Video - Technologies, Applications and MPEG Standards. In _2006 IEEE International Conference on Multimedia and Expo_. 2161–2164.  https://doi.org/10.1109/ICME.2006.262683
  * Sun et al. (2021) Yuqi Sun, Ri Cheng, Bo Yan, and Shili Zhou. 2021\.  Space-Angle Super-Resolution for Multi-View Images. In _MM ’21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021_. ACM, 750–759.  https://doi.org/10.1145/3474085.3475244
  * Tsiminaki et al. (2014) Vagia Tsiminaki, Jean-Sebastien Franco, and Edmond Boyer. 2014.  High Resolution 3D Shape Texture from Multiple Videos. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _Advances in Neural Information Processing Systems_ , Vol. 30.
  * Wang et al. (2019) Longguang Wang, Yingqian Wang, Zhengfa Liang, Zaiping Lin, Jungang Yang, Wei An, and Yulan Guo. 2019.  Learning Parallax Attention for Stereo Image Super-Resolution. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Wang et al. (2018) Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. 2018\.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks. In _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_.
  * Wang et al. (2020) Yingqian Wang, Longguang Wang, Jungang Yang, Wei An, Jingyi Yu, and Yulan Guo. 2020\.  Spatial-Angular Interaction for Light Field Image Super-Resolution. In _Computer Vision – ECCV 2020_. 290–308.
  * Wanner and Goldluecke (2014) Sven Wanner and Bastian Goldluecke. 2014.  Variational Light Field Analysis for Disparity Estimation and Super-Resolution.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 36, 3 (2014), 606–619.  https://doi.org/10.1109/TPAMI.2013.147
  * Yan et al. (2020) Xu Yan, Weibing Zhao, Kun Yuan, Ruimao Zhang, Zhen Li, and Shuguang Cui. 2020\.  Towards Content-Independent Multi-Reference Super-Resolution: Adaptive Pattern Matching and Feature Aggregation. In _Computer Vision – ECCV 2020_. 52–68.
  * Yang et al. (2020) Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. 2020.  Learning Texture Transformer Network for Image Super-Resolution. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Yao et al. (2018) Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. 2018.  MVSNet: Depth Inference for Unstructured Multi-view Stereo. In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Yao et al. (2020) Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. 2020\.  BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Ying et al. (2020) Xinyi Ying, Yingqian Wang, Longguang Wang, Weidong Sheng, Wei An, and Yulan Guo. 2020\.  A Stereo Attention Module for Stereo Image Super-Resolution.  _IEEE Signal Processing Letters_ 27 (2020), 496–500.  https://doi.org/10.1109/LSP.2020.2973813
  * Zhang et al. (2018a) Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018a.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhang et al. (2019a) Shuo Zhang, Youfang Lin, and Hao Sheng. 2019a.  Residual Networks for Light Field Image Super-Resolution. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhang et al. (2018b) Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018b.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks. In _ECCV_.
  * Zhang et al. (2019b) Zhifei Zhang, Zhaowen Wang, Zhe Lin, and Hairong Qi. 2019b.  Image Super-Resolution by Neural Texture Transfer. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zheng et al. (2018) Haitian Zheng, Mengqi Ji, Haoqian Wang, Yebin Liu, and Lu Fang. 2018.  CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping. In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Zhu et al. (2019) Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. 2019\.  Deformable ConvNets V2: More Deformable, Better Results. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.

##  Appendix A Loss Functions

Reconstruction Loss. The $\ell_{1}$-norm of the reconstruction loss is used as
follows:

(8) $\mathcal{L}_{\text{rec }}=\left\|I_{T}^{hr}-I_{T}^{gt}\right\|_{1},$

where $I_{T}^{hr}$ and $I_{T}^{gt}$ represent the output of our MVSRnet and
the high-resolution (HR) view of the ground truth, respectively.

Adversarial Loss. We use the WGAN-GP (Gulrajani et al., 2017) structure to
obtain adversarial loss (Goodfellow et al., 2014). WGAN-GP replaces the weight
clipping with a gradient norm penalty to achieve stable training and generate
high-quality samples. We define adversarial loss as follows:

(9) $\displaystyle\mathcal{L}_{D}=$
$\displaystyle\mathbb{E}_{\boldsymbol{I_{T}^{hr}}}[D(I_{T}^{hr})]-\mathbb{E}_{\boldsymbol{I_{T}^{gt}}}[D(I_{T}^{gt})]+$
$\displaystyle\lambda{\mathbb{E}_{\boldsymbol{\hat{I}}}}\left[\left(\left\|\nabla_{\hat{I}}D(\hat{I})\right\|_{2}-1\right)^{2}\right],$
(10) $\mathcal{L}_{G}=-{\mathbb{E}_{\boldsymbol{I_{T}^{hr}}}}[D(I_{T}^{hr})].$

where $\hat{I}$ is the random sample of our network output $I_{T}^{hr}$ and
ground truth $I_{T}^{gt}$ through combination, ($\hat{I}\leftarrow\epsilon
I_{T}^{gt}+(1-\epsilon)I_{T}^{hr},\epsilon\sim U[0,1]$).

Perceptual Loss. We utilize perceptual loss (Johnson et al., 2016) to generate
the photo-realist image, which we define as follows:

(11)
$L_{per}=\left\|\phi_{i}\left(I_{T}^{hr}\right)-\phi_{i}\left(I_{T}^{gt}\right)\right\|_{1},$

where $\phi_{i}$ denotes the $i$-th layer feature map of VGG19 (Simonyan and
Zisserman, 2015), and we use the features of conv3-2, conv4-2, and conv5-2 to
calculate the perceptual loss.

Table 5.  Comparison to multi-view super-resolution method.

Model | Tanks and Temple | BlendedMVS
---|---|---
SASRnet (Sun et al., 2021) | 28.99 / 0.8493 / 0.230 | 27.80 / 0.7198 / 0.393
MVSRnet-rec |  30.26 / 0.8748 / 0.191 |  28.03 / 0.7457 / 0.360

Table 6. The network structure of our Dynamic High-Frequency Search network.
MVR, NVR, and Bic denote the multi-view reference image, near-view reference
image, and the bicubic of the LR input view, respectively. We present the
convolution layer as Conv(input channels, output channels, kernel size,
stride, padding). $RB\times N$ is a residual block with $N$ residual layers,
and we adopt the residual layer with channel attention described in (Zhang et
al., 2018b). In addition, we use PixelShuffle to upsample, and use ReLU in
$RB$ and LeakyReLU other places.

| Id | Layer Name |  | Id | Layer Name |  | Id | Layer Name |  | Id |  | Layer
---
Name
Feature Extractor | 1-0 | Concat (MVRs, Bic) |  | 2-0 | LR Input View |  |  |  |  |  |
1-1 |  | Conv(19,64,3,1,1),
---
RB $\times$ 3
2-1 |  | Conv(3,64,3,1,1),
---
RB $\times$ 10
1-2 |  | Conv(64,128,4,2,1),
---
RB $\times$ 3
RSM (scale 1$\times$) | 2-2 |  | Concat($2-1, $1-3),
---
Conv(320,64,3,1,1)
RSM (scale 2$\times$) | 3-0 |  | Concat($2-18, $1-2),
---
Conv(192,64,3,1,1)
RSM (scale 4$\times$) | 4-0 |  | Concat($3-16, $1-1),
---
Conv(128,64,3,1,1)
1-3 |  | Conv(128,256,4,2,1),
---
RB $\times$ 3
2-3 | $2-1+$2-2 | 3-1 | $2-18+$3-0 | 4-1 | $3-16+$4-0
1-4 | Concat (NVRs, Bic) | 2-4 |  | Concat($2-1, $1-7),
---
Conv(320,64,3,1,1)
3-2 |  | Concat($2-18, $1-6),
---
Conv(192,64,3,1,1)
4-2 |  | Concat($3-16, $1-5),
---
Conv(128,64,3,1,1)
1-5 |  | Conv(19,64,3,1,1),
---
RB $\times$ 3
2-5 | $2-1+$2-4 | 3-3 | $2-18+$3-2 | 4-3 | $3-16+$4-2
1-6 |  | Conv(64,128,4,2,1),
---
RB $\times$ 3
2-6 |  | Concat($2-3,$2-5,$2-1),
---
Conv(192,64,3,1,1),
LeakyReLU,
Conv(64,2,3,1,1),
Softmax
3-4 |  | Concat($3-1,$3-3,$2-18),
---
Conv(192,64,3,1,1),
LeakyReLU,
Conv(64,2,3,1,1),
Softmax
4-4 |  | Concat($4-1,$4-3,$3-16),
---
Conv(192,64,3,1,1),
LeakyReLU,
Conv(64,2,3,1,1),
Softmax
1-7 |  | Conv(128,256,4,2,1),
---
RB $\times$ 3
2-7 |  | $2-3$\times$$2-6[0]+
---
$2-5$\times$$2-6[1],
RB $\times$ 8
3-5 |  | $3-1$\times$$3-4[0]+
---
$3-3$\times$$3-4[1],
RB $\times$ 6
4-5 |  | $4-1$\times$$4-4[0]+
---
$4-3$\times$$4-4[1],
RB $\times$ 4
|  |  |  ASM (scale 1$\times$) | 2-8 |  | Concat($1-3, $2-7),
---
Conv(320,256,3,1,1)
ASM (scale 2$\times$) | 3-6 |  | Concat($1-2, $3-5),
---
Conv(192,128,3,1,1)
ASM (scale 4$\times$) | 4-6 |  | Concat($1-1, $4-5),
---
Conv(128,64,3,1,1)
2-9 |  | $1-3+$2-8,
---
Conv(256,64,3,1,1),
RB $\times$ 8
3-7 |  | $1-2+$3-6,
---
Conv(128,64,3,1,1),
RB $\times$ 6
4-7 |  | $1-1+$4-6,
---
Conv(64,64,3,1,1),
RB $\times$ 4
2-10 |  | Concat($1-7, $2-7),
---
Conv(320,256,3,1,1)
3-8 |  | Concat($1-6, $3-5),
---
Conv(192,128,3,1,1)
4-8 |  | Concat($1-5, $4-5),
---
Conv(64,64,3,1,1)
2-11 |  | $1-7+$2-10,
---
Conv(256,64,3,1,1)
RB $\times$ 8
3-9 |  | $1-6+$3-8,
---
Conv(128,64,3,1,1)
RB $\times$ 6
4-9 |  | $1-5+$4-8,
---
Conv(64,64,3,1,1)
RB $\times$ 4
2-12 |  | Concat($2-7, $2-9),
---
Conv(128,64,3,1,1)
3-10 |  | Concat($3-5, $3-7),
---
Conv(128,64,3,1,1)
4-10 |  | Concat($4-5, $4-7),
---
Conv(128,64,3,1,1)
2-13 | $2-7+$2-12 | 3-11 | $3-5+$3-10 | 4-11 | $4-5+$4-10
2-14 |  | Concat($2-7, $2-11),
---
Conv(128,64,3,1,1)
3-12 |  | Concat($3-5, $3-9),
---
Conv(128,64,3,1,1)
4-12 |  | Concat($4-5, $4-9),
---
Conv(128,64,3,1,1)
2-15 | $2-7+$2-14 | 3-13 | $3-5+$3-12 | 4-13 | $4-5+$4-12
2-16 |  | Concat($2-13,$2-15,$2-7),
---
Conv(192,64,3,1,1),
LeakyReLU,
Conv(64,2,3,1,1),
Softmax
3-14 |  | Concat($3-11,$3-13,$3-5),
---
Conv(192,64,3,1,1),
LeakyReLU,
Conv(64,2,3,1,1),
Softmax
4-14 |  | Concat($4-11,$4-13,$4-5),
---
Conv(192,64,3,1,1),
LeakyReLU,
Conv(64,2,3,1,1),
Softmax
2-17 |  | $2-13$\times$$2-16[0]+
---
$2-15$\times$$2-16[1],
RB $\times$ 8
3-15 |  | $3-11$\times$$3-14[0]+
---
$3-13$\times$$3-14[1],
RB $\times$ 6
4-15 |  | $4-11$\times$$4-14[0]+
---
$4-13$\times$$4-14[1],
RB $\times$ 4
2-18 |  | Conv(64,256,3,1,1),
---
PixelShuffle(2),
Conv(64,64)
3-16 |  | Conv(64,256,3,1,1),
---
PixelShuffle(2),
Conv(64,64)
4-16 | Conv(64,3,3,1,1),

Table 7.  Comparison of the model size. _light_ denotes the light version of
our proposed network. Red and blue indicate first and second performance,
respectively. We include the parameter of discriminator in the total
parameters of the GAN-based model.

| Model | Parameters | Tanks and Temple | BlendedMVS | GTAV
---|---|---|---|---|---
SISR | EDSR (Lim et al., 2017) | 43.09M | 28.99 / 0.8493 / 0.230 | 27.80 / 0.7198 / 0.393 | 30.93 / 0.8478 / 0.211
RCAN (Zhang et al., 2018b) | 15.59M | 29.89 / 0.8544 / 0.231 | 27.90 / 0.7227 / 0.393 | 31.23 / 0.8530 / 0.212
HAN (Niu et al., 2020) | 16.07M | 29.76 / 0.8518 / 0.228 | 27.86 / 0.7211 / 0.396 | 31.13 / 0.8515 / 0.206
ESRGAN (Wang et al., 2018) | 31.20M | 26.63 / 0.7720 / 0.112 | 25.50 / 0.6315 / 0.173 | 28.26 / 0.7746 / 0.105
SPSR (Ma et al., 2020) | 39.30M | 27.16 / 0.7736 / 0.104 | 25.72 / 0.6406 / 0.168 | 28.37 / 0.7739 / 0.102
VSR | RBPN (Haris et al., 2019) | 12.77M | 29.75 / 0.8531 / 0.236 | 27.79 / 0.7179 / 0.408 | 31.27 / 0.8559 / 0.218
RSDN (Isobe et al., 2020) | 6.19M | 29.00 / 0.8347 / 0.240 | 26.38 / 0.6412 / 0.480 | 30.26 / 0.8365 / 0.223
MuCAN (Li et al., 2020) | 25.66M | 29.90 / 0.8573 / 0.215 | 27.72 / 0.7162 / 0.400 | 31.24 / 0.8585 / 0.196
IconVSR (Chan et al., 2021) | 8.71M | 29.75 / 0.8547 / 0.227 | 27.53 / 0.7089 / 0.408 | 31.41 / 0.8633 / 0.205
Ref-SR | TTSR (Yang et al., 2020) | 145.66M | 27.53 / 0.7858 / 0.124 | 26.09 / 0.6567 / 0.201 | 28.80 / 0.7914 / 0.121
TTSR-rec (Yang et al., 2020) | 6.73M | 29.46 / 0.8471 / 0.231 | 27.68 / 0.7193 / 0.396 | 30.79 / 0.8485 / 0.216
MASA (Lu et al., 2021a) | 18.99M | 26.66 / 0.7545 / 0.134 | 25.84 / 0.6468 / 0.220 | 28.05 / 0.7699 / 0.131
MASA-rec (Lu et al., 2021a) | 4.03M | 29.67 / 0.8538 / 0.218 | 27.82 / 0.7335 / 0.375 | 31.12 / 0.8613 / 0.196
$C^{2}$-Matching (Jiang et al., 2021) | 14.11M | 29.29 / 0.8384 / 0.158 | 27.24 / 0.7039 / 0.281 | 30.57 / 0.8396 / 0.153
$C^{2}$-Matching-rec (Jiang et al., 2021) | 8.87M | 29.89 / 0.8609 / 0.212 | 27.99 / 0.7377 / 0.370 | 31.21 / 0.8628 / 0.194
MVSRnet | 19.42M | 28.83 / 0.8372 / 0.085 | 26.90 / 0.7080 / 0.182 | 30.27 / 0.8462 / 0.085
| MVSRnet-_light_ | 9.67M | 28.48 / 0.8260 / 0.090 | 26.83 / 0.7003 / 0.190 | 29.89 / 0.8337 / 0.094
| MVSRnet-rec | 14.18M |  30.26 / 0.8748 / 0.191 |  28.03 /  0.7457 / 0.360 |  31.60 / 0.8775 / 0.179
| MVSRnet-_light_ -rec | 4.39M |  30.12 / 0.8712 / 0.199 |  28.00 / 0.7426 / 0.368 |  31.45 / 0.8735 / 0.186

##  Appendix B Compare to Multi-view Super-Resolution Method

As far as we know, our MVSRnet is the first to solve the multi-view image
super-resolution problem using low-resolution images. Existing multi-view SR
related methods rarely consider this problem. Some methods focus on multi-view
texture (Richard et al., 2019) or light field images (Zhang et al., 2019a) and
some other methods (Lu et al., 2021b; Li et al., 2016) require high-resolution
multi-view as a reference, which is not easily available in practice due to
high storage costs and bandwidth constraints. The most similar method to ours
is SASRnet (Sun et al., 2021). It uses two adjacent low-resolution views to
generate high-resolution source views and a novel view. In Tanks and Temples
and BlendedMVS dataset, they have a source view PSNR of 29.29dB and 27.27dB,
respectively, which is much lower than our 30.26dB and 28.03dB as shown in
Table 5. Moreover, SASRnet fails in GTAV and generate dark images because
their depth prediction module is sensitive to depth scale.

##  Appendix C Network Structures

We illustrate the network structure of the Dynamic High-Frequency Search
network in Table 6. It first uses a feature extractor to extract the features
of multi-view reference images and near-view reference images at 1$\times$,
2$\times$, and 4$\times$ scales, respectively. Then three Raw Selection
Modules (RSM) and three Adaptation Selection Modules (ASM) are used to select
the useful features from the reference features of different scales. We adopt
the same structure as (Jiang et al., 2021) for the discriminator.

##  Appendix D Model Size

This section compares the number of training parameters in Table 7. We provide
two models, MVSRnet-_light_ and MVSRnet-_light_ -rec, for comparing model
parameters. Specifically, for MVSRnet-_light_ , we reduce the number of
channels in the feature extractor and residual layer to half and add three
more residual layers to feature extractor. We also include the parameter of
the discriminator in the total parameters for the GAN-based model. As shown in
Table 7, the parameters of MVSRnet-_light_ -rec are only 0.36M more than MASA-
rec, but our PSNR is 0.45dB, 0.18dB, and 0.33dB more than MASA-rec in Tanks
and Temple, BlendedMVS, and GTAV, respectively. In summary, MVSRnet-_light_
not only has fewer parameters, but achieves better results than the state of
the arts to a large extent.

##  Appendix E More Visual Comparison

In this section, we provide more visual comparisons with SISR methods (RCAN
(Zhang et al., 2018b), ESRGAN (Wang et al., 2018), and SPSR (Ma et al.,
2020)), VSR methods (MuCAN (Li et al., 2020) and IconVSR (Chan et al., 2021)),
and Ref-SR methods (TTSR (Yang et al., 2020) and $C^{2}$-Matching (Jiang et
al., 2021)). Specifically, RCAN, MuCAN, IconVSR, and our MVSRnet-rec only use
the reconstruction loss, and ESRGAN, SPSR, TTSR, $C^{2}$-Matching, and our
MVSRnet are all GAN-based models. The visual comparison of Tanks and Temple,
BlendedMVS, and GTAV are shown in Figure 7 and Figure 8, Figure 9, and Figure
10, respectively.

##  Appendix F Limitation

We have two major limitations. First, we assume static sceneries like in most
multi-view work (Riegler and Koltun, 2021, 2020), and the denser the input
views, the better the scene geometry estimation. As a result, we will
concentrate on the MVISR task of dynamic scenes and design a strategy for
dealing with situations in which there are just a few multi-view images
accessible. Second, we preprocess all LR multi-view images to create
synthesized reference images, so an elaborated online reference selection
approach should be proposed.

Figure 7.  More qualitative comparisons with soft-of-the-art SR methods on the
Tanks and Temple (Knapitsch et al., 2017) dataset.  Figure 8.  More
qualitative comparisons with soft-of-the-art SR methods on the Tanks and
Temple (Knapitsch et al., 2017) dataset.  Figure 9.  More qualitative
comparisons with soft-of-the-art SR methods on the BlendedMVS (Yao et al.,
2020) dataset.  Figure 10.  More qualitative comparisons with soft-of-the-art
SR methods on the GTAV (Huang et al., 2018) dataset.


# A generalization of the alternating group: Permutations preserving
orientation of subsets of a fixed size

Alexei Vernitski

###### Abstract

We study permutations on $n$ elements preserving orientation (parity) of every
subset of size $k$. We describe all groups of these permutations.
Unexpectedly, these groups (except some special cases) are either trivial or
cyclic or dihedral.

##  1 Introduction

Fix a positive integer $n$. Denote $0,1,\dots,n-1$ by $[n]$. Elements of $[n]$
will be denoted by letters $a,b,c$. Mappings on $[n]$ will be denoted by
lower-case Greek letters. Denote the identity mapping on $[n]$ by $\iota_{n}$.
For a given $k$, denote the cyclic shift $a\mapsto(a+k)\;\mathrm{mod}\;n$ by
$\sigma_{k}$. Denote the cyclic group consisting of the permutations
$\sigma_{k}$ by $Z_{n}$. Denote the permutation $a\mapsto(n-1)-a$ reversing
the order of elements in $[n]$ by $\rho$. Denote the dihedral group generated
by $\sigma_{1}$ and $\rho$ by $D_{n}$.

A mapping $\alpha$ (which is not necessarily a permutation) on $[n]$ is called
order-preserving if from $a<b$ it follows that $\alpha(a)\leq\alpha(b)$. The
set of order-preserving mappings is closed with respect to composition, and,
hence, is a semigroup. This semigroup is one of the most frequently studied
semigroups in semigroup theory; its research originated in [1, 2, 3, 4, 5] and
led to hundreds of later publications.

One of the ways in which semigroup theorists have generalized the semigroup of
order-preserving mappings is the following. A mapping $\alpha$ on $[n]$ is
called orientation-preserving if from $a<b<c$ it follows that
$\alpha(a)\leq\alpha(b)\leq\alpha(c)$ or $\alpha(b)\leq\alpha(c)\leq\alpha(a)$
or $\alpha(c)\leq\alpha(a)\leq\alpha(b)$. The intuition behind this
construction is that the elements of $[n]$ are written clockwise around a
circle, like on a clock face, and a mapping is orientation-preserving if it
preserves the clockwise orientation of the elements of $[n]$. The set of
orientation-preserving mappings is a semigroup. It was introduced 25 years ago
[6, 7] and has been repeatedly studied since then, two relevant recent
publications being [8, 9].

As one compares the semigroup of order-preserving mappings and the semigroup
of orientation-preserving mappings, one can ask the following questions.

  * •

The semigroup of order-preserving mappings is defined using a condition on
pairs of elements of $[n]$, and the semigroup of orientation-preserving
mappings is defined using a condition on triples of elements of $[n]$. Is it
possible to generalize these conditions to $t$-tuples of elements of $[n]$?

  * •

The only permutation belonging to the semigroup of order-preserving mappings
is $\iota_{n}$. In other words, the set of permutations belonging to the
semigroup of order-preserving mappings is the one-element group
$\\{\iota_{n}\\}$. As to the semigroup of orientation-preserving mappings, the
set of permutations belonging to it is the cyclic group $Z_{n}$. How will this
generalize if we consider conditions on $t$-tuples of elements of $[n]$?

The aim of this article is to answer the questions above. We will introduce a
property which generalizes the properties of being order-preserving and
orientation-preserving. Then we will describe all groups of permutations
satisfying this generalized property.

##  2 The generalized definition

Consider a one-to-one partial mapping $\alpha$ on $[n]$ whose domain and image
both have size $t$. By an inversion we mean a pair $a,b$ such that $a<b$ and
$\alpha(a)>\alpha(b)$. We will say that $\alpha$ is even (odd) if the number
of inversions is even (odd). It is useful to note that if $t=n$ then this
definition of being even (odd) coincides with the usual definition of an even
(odd) permutation.

We define $\Gamma_{n}^{t}$, where $t=2,\dots,n$, as the group consisting of
those permutations on $[n]$ whose restriction to every $t$-element subset of
$[n]$ is even.

We also define $\Sigma_{n}^{t}$ as the semigroup consisting of those
permutations on $[n]$ whose restriction to every $t$-element subset of $[n]$
is either not one-to-one or one-to-one and even. Notice that the semigroup of
order-preserving mappings is $\Sigma_{n}^{2}$, and the semigroup of
orientation-preserving mappings is $\Sigma_{n}^{3}$. Thus, $\Sigma_{n}^{s}$ is
a generalization of both the semigroup of order-preserving mappings and the
semigroup of orientation-preserving mappings.

Notice that $\Gamma_{n}^{t}$ is the set of permutations belonging to
$\Sigma_{n}^{t}$. Recall that $\Gamma_{n}^{2}=\\{\iota_{n}\\}$ and
$\Gamma_{n}^{3}=Z_{n}$. Apart from these two groups, it is easy to spot yet
another famous group among groups $\Gamma_{n}^{t}$; this is $\Gamma_{n}^{n}$,
which is the alternating group. Speaking of the size of these groups,
$|\Gamma_{n}^{2}|=1$, $|\Gamma_{n}^{3}|=n$ and
$|\Gamma_{n}^{n}|=\frac{1}{2}(n!)$. What is the size $|\Gamma_{n}^{t}|$ if the
value of $t$ is in the range between $3$ and $n$? Examples above might be seen
as suggesting that the size $|\Gamma_{n}^{t}|$ increases monotonically as a
function of $t$. However, the real picture is different; let us illustrate it
by an example of $|\Gamma_{11}^{t}|$. The table below shows values of $t$ in
the top row and values of $|\Gamma_{11}^{t}|$ in the bottom row.

2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11
---|---|---|---|---|---|---|---|---|---
1 | 11 | 2 | 22 | 1 | 11 | 2 | 22 | 43,200 | 19,958,400

As you can see, the values of $|\Gamma_{n}^{t}|$ oscillate between $1$, $n$,
$2$ and $2n$ on the whole range $t=2,\dots,n-2$, and $t=n-1$ is a special
case. In the following sections we prove theorems which explain in detail how
these numbers arise.

##  3 Groups $\Gamma_{n}^{t}$ for $t=2,\dots,n-2$

Let $S$ be a subset of $[n]$. Let $\alpha$ be a mapping on $[n]$. Denote the
number of inversions of the restriction of $\alpha$ to $S$ by
$\mathrm{inv}(\alpha\upharpoonright S)$.

In Lemma 2 below, we fix $a\in[n]$ and then consider, firstly, some elements
$b_{i}\in[n]$ such that $\alpha(b_{i})$ lies between $\alpha(a)$ and
$\alpha(a+1)$ and secondly, some elements $c_{j}\in[n]$ such that
$\alpha(c_{j})$ does not lie between $\alpha(a)$ and $\alpha(a+1)$. Note that
$a+1$ is calculated modulo $n$. Depending on the relative position of $a$ and
$a+1$ (that is, $a<a+1$ or $a>a+1$), and depending on the relative position of
$\alpha(a)$ and $\alpha(a+1)$ (that is, $\alpha(a)<\alpha(a+1)$ or
$\alpha(a)>\alpha(a+1)$), one would have to consider several cases. It is
perfectly possible, but we can write a simpler proof using the following
definition.

Consider $a,b\in[n]$, with $b\neq a$ and $b\neq a+1$. We will say that
$\alpha(b)$ lies between $\alpha(a)$ and $\alpha(a+1)$ if
$\mathrm{inv}(\alpha\upharpoonright\\{a,a+1\\})\neq\mathrm{inv}(\alpha\upharpoonright\\{a,a+1,b\\})\;\mathrm{mod}\;2$.
This condition can be expressed in a different way as
$\mathrm{inv}(\alpha\upharpoonright\\{a,b\\})\neq\mathrm{inv}(\alpha\upharpoonright\\{a+1,b\\})\;\mathrm{mod}\;2$
or, even more conveniently,
$\mathrm{inv}(\alpha\upharpoonright\\{a,b\\})=\mathrm{inv}(\alpha\upharpoonright\\{a+1,b\\})+1\;\mathrm{mod}\;2$.
Hence, the following useful observation is true.

######  Lemma 1.

Let $a\in[n]$. Consider $b_{1},\dots,b_{r}\in[n]$ such that $\alpha(b_{i})$
lies between $\alpha(a)$ and $\alpha(a+1)$ and $c_{1},\dots,c_{s}\in[n]$ such
that $\alpha(c_{j})$ does not lie between $\alpha(a)$ and $\alpha(a+1)$. Let
$S=\\{b_{1},\dots,b_{r};c_{1},\dots,c_{s}\\}$. Let $S^{\prime}=S\cup\\{a\\}$
and $S^{\prime\prime}=S\cup\\{a+1\\}$. Then
$\mathrm{inv}(\alpha\upharpoonright
S^{\prime})=\mathrm{inv}(\alpha\upharpoonright
S^{\prime\prime})+r\;\mathrm{mod}\;2$.

######  Lemma 2.

$\Gamma_{n}^{t}\subseteq D_{n}$ for every $t=2,\dots,n-2$.

###### Proof.

Suppose a permutation $\alpha$ on $[n]$ does not belong to $D_{n}$. This can
be expressed by saying that there are $a,a+1$ such that $\alpha(a)$ is neither
$\alpha(a)-1$ nor $\alpha(a)+1$, where addition is modulo $n$. Hence, there
exists at least one $b\in[n]$ such that $\alpha(b)$ lies between $\alpha(a)$
and $\alpha(a+1)$. Consider all elements $b_{1},\dots,b_{p}\in[n]$ such that
$\alpha(b_{i})$ lies between $\alpha(a)$ and $\alpha(a+1)$. Consider all
elements $c_{1},\dots,c_{q}\in[n]$ such that $\alpha(c_{j})$ does not lie
between $\alpha(a)$ and $\alpha(a+1)$. Note that $p+q=n-2$ and $p\geq 1$. The
number $t-1$ is in the range $1,\dots,n-3$. Represent $t-1$ as a sum of an odd
positive integer $r$ in the range $1,\dots,p$ and a non-negative integer $s$
in the range $0,\dots,q$. Like in Lemma 1, consider
$S=\\{b_{1},\dots,b_{r};c_{1},\dots,c_{s}\\}$ and $S^{\prime}=S\cup\\{a\\}$
and $S^{\prime\prime}=S\cup\\{a+1\\}$. Since $r$ is odd, by Lemma 1, we have
$\mathrm{inv}(\alpha\upharpoonright
S^{\prime})\neq\mathrm{inv}(\alpha\upharpoonright
S^{\prime\prime})\;\mathrm{mod}\;2$. That is, if the restriction of $\alpha$
on $S^{\prime}$ is even then the restriction of $\alpha$ on $S^{\prime\prime}$
is odd, and if the restriction of $\alpha$ on $S^{\prime\prime}$ is even then
the restriction of $\alpha$ on $S^{\prime}$ is odd. Recall that for $\alpha$
to be in $\Gamma_{n}^{t}$, the restriction of $\alpha$ to every $t$-element
subset of $[n]$ should be even. Hence, $\alpha\notin\Gamma_{n}^{t}$. ∎

######  Lemma 3.

For every $k\neq 0$ and $t=2,\dots,n-2$, $\sigma_{k}\in\Gamma_{n}^{t}$ if an
only if $t$ is odd.

###### Proof.

Consider a $t$-element subset of $[n]$ consisting of elements
$a_{1}<\cdots<a_{t}$. Since $\sigma_{k}$ is a cyclic shift, we have
$\alpha(a_{k+1})<\cdots<\alpha(a_{t})<\alpha(a_{1})<\cdots<\alpha(a_{k})$ for
some $k$. The number of inversions of the restriction of $\alpha$ to
$a_{1},\dots,a_{t}$ is $k(t-k)$; if $t$ is odd, this product is even for every
$k$.

Conversely, suppose $t$ is even. Suppose $(t-1)+k\leq n-1$. Then elements
$0,\dots,t-1\in[n]$ are mapped to $k,\dots,(t-1)+k$, respectively, by
$\sigma_{k}$. Since $k\neq 0$, we have $\sigma_{k}(n-1)=k-1$. Then
$\mathrm{inv}(\sigma_{k}\upharpoonright\\{n-1,\dots,t-2\\})=t-1$, and $t-1$ is
an odd number. Hence, $\sigma_{k}\notin\Gamma_{n}^{t}$.

Now suppose $(t-1)+k>n-1$ and consider again how $\sigma_{k}$ acts on elements
$0,\dots,t-1\in[n]$. Notice that $0,\dots,n-k-1$ are mapped to $k,\dots,n-1$,
respectively, and $n-k,\dots,t-1$ are mapped to $0,\dots,k+t-n-1$,
respectively. Then
$\mathrm{inv}(\sigma_{k}\upharpoonright\\{0,\dots,t-1\\})=(n-k)(t-n+k)$.
Similarly, considering $\\{n-1,\dots,t-2\\}$, we notice that
$\mathrm{inv}(\sigma_{k}\upharpoonright\\{0,\dots,t-1\\})=(n-k+1)(t-n+k-1)$.
Recall that $t$ is even. Therefore, one of the two products $(n-k)(t-n+k)$ and
$(n-k+1)(t-n+k-1)$ is odd. Hence, $\sigma_{k}\notin\Gamma_{n}^{t}$. ∎

######  Lemma 4.

$\rho\in\Gamma_{n}^{t}$ if an only if $t=0\;\mathrm{mod}\;4$ or
$t=1\;\mathrm{mod}\;4$.

###### Proof.

As $\rho$ acts on a $t$-element subset of $[n]$ consisting of elements
$a_{1}<\cdots<a_{t}$, it reverses their order,
$\alpha(a_{t})<\cdots<\alpha(a_{1})$. Thus,
$\mathrm{inv}(\rho\upharpoonright\\{a_{1},\dots,a_{t}\\})=\frac{t(t-1)}{2}$.
The number expressed by this fraction is even if and only if
$t=0\;\mathrm{mod}\;4$ or $t=1\;\mathrm{mod}\;4$. ∎

From Lemmas 2, 3, 4 our main result follows.

######  Theorem 5.

For every $t=2,\dots,n-2$,

  * •

If $t=2\;\mathrm{mod}\;4$ then $\Gamma_{n}^{t}$ is the one-element group
$\\{\iota_{n}\\}$.

  * •

If $t=3\;\mathrm{mod}\;4$ then $\Gamma_{n}^{t}$ is the cyclic group $Z_{n}$.

  * •

if $t=0\;\mathrm{mod}\;4$ then $\Gamma_{n}^{t}$ is the two-element group
$\\{\rho,\iota_{n}\\}$.

  * •

if $t=1\;\mathrm{mod}\;4$ then $\Gamma_{n}^{t}$ is the dihedral group $D_{n}$.

##  4 Group $\Gamma_{n}^{n-1}$

In this section we will use a type of permutations called parity-alternating
permutations [10, 11]. A permutation $\alpha$ on $[n]$ is parity-alternating
if for every $a=0,\dots,n-2$ (we do not consider $a=n-1$) we have
$\alpha(a)\neq\alpha(a+1)\;\mathrm{mod}\;2$. Denote the set of all parity-
alternating permutations on $[n]$ by $PAP_{n}$. It is not difficult to find a
formula for the size of $PAP_{n}$, and it has been done in [10]. The size
$|PAP_{n}|$ is $2\cdot(\frac{n}{2})!^{2}$ if $n$ is even and
$(\frac{n-1}{2})!\cdot(\frac{n+1}{2})!$ if $n$ is odd. We will prove that
every permutation from $\Gamma_{n}^{n-1}$ is parity-alternating; to be more
precise, the permutations belonging to $\Gamma_{n}^{n-1}$ constitute exactly a
half of $PAP_{n}$; hence, $|\Gamma_{n}^{n-1}|=\frac{1}{2}\cdot|PAP_{n}|$.

Recall that $\Gamma_{n}^{n-1}$ consists of all permutations on $[n]$ such that
for every $a\in[n]$ the number
$\mathrm{inv}(\alpha\upharpoonright([n]\setminus\\{a\\})$ is even. It will be
useful to consider a set of all permutations on $[n]$ such that for every
$a\in[n]$ the number $\mathrm{inv}(\alpha\upharpoonright([n]\setminus\\{a\\})$
is odd (instead of even); we will denote this set by $-\Gamma_{n}^{n-1}$.

######  Lemma 6.

$PAP_{n}$ is a disjoint union of $\Gamma_{n}^{n-1}$ and $-\Gamma_{n}^{n-1}$.

###### Proof.

Consider a permutation $\alpha\in\Gamma_{n}^{n-1}$ or
$\alpha\in-\Gamma_{n}^{n-1}$. Consider $a=0,\dots,n-2$. Let
$S=[n]\setminus\\{a,a+1\\}$. Like in Lemma 1, let $S^{\prime}=S\cup\\{a\\}$
and $S^{\prime\prime}=S\cup\\{a+1\\}$. By Lemma 1,
$\mathrm{inv}(\alpha\upharpoonright
S^{\prime})=\mathrm{inv}(\alpha\upharpoonright
S^{\prime\prime})+r\;\mathrm{mod}\;2$. Since $\alpha\in\Gamma_{n}^{n-1}$ or
$\alpha\in-\Gamma_{n}^{n-1}$, both $\mathrm{inv}(\alpha\upharpoonright
S^{\prime})$ and $\mathrm{inv}(\alpha\upharpoonright S^{\prime\prime})$ are
even (or both are odd). Hence, $r$ is even. The number $r$ is the number of
elements of $[n]$ lying between $\alpha(a)$ and $\alpha(a+1)$, and since
$a\neq n-1$, we are speaking about the gap between $\alpha(a)$ and
$\alpha(a+1)$ in the usual arithmetic, not the modular arithmetic. Since $r$
is even, the difference between $\alpha(a)$ and $\alpha(a+1)$ is odd.
Therefore, by the definition of a parity-alternating permutation, $\alpha$ is
parity-alternating.

Conversely, for every $\alpha\in PAP_{n}$ and for every $a=0,\dots,n-2$, the
difference between $\alpha(a)$ and $\alpha(a+1)$ is odd, hence, the number $r$
of elements of $[n]$ lying between $\alpha(a)$ and $\alpha(a+1)$ is even.
Hence, $\mathrm{inv}(\alpha\upharpoonright
S^{\prime})=\mathrm{inv}(\alpha\upharpoonright
S^{\prime\prime})\;\mathrm{mod}\;2$. Hence, either for every $a\in[n]$ the
number $\mathrm{inv}(\alpha\upharpoonright([n]\setminus\\{a\\})$ is even, or
for every $a\in[n]$ the number
$\mathrm{inv}(\alpha\upharpoonright([n]\setminus\\{a\\})$ is odd. In the
former case, $\alpha$ belongs to $\Gamma_{n}^{n-1}$, and in the latter case,
$\alpha$ belongs to $-\Gamma_{n}^{n-1}$. ∎

######  Theorem 7.

$|\Gamma_{n}^{n-1}|=\frac{1}{2}\cdot PAP(n)$.

###### Proof.

Recall that in the definition of $\Gamma_{n}^{t}$ we assume that $t\geq 2$,
hence, when we consider $\Gamma_{n}^{n-1}$, we assume that $n\geq 3$. On the
set $PAP_{n}$, consider an involution $\Xi$ induced by applying the cycle
$(1\;3)$ before the permutation. That is, let $\xi$ be a permutation on $[n]$
swapping $1$ and $3$. For each $\alpha\in PAP_{n}$, the mapping $\Xi$ maps
$\alpha$ to $\alpha(\xi(\cdot))$. If $\alpha(1)$ is odd, $\alpha(2)$ is even,
$\alpha(3)$ is odd then $\alpha(\xi(1))$ is odd, $\alpha(\xi(2))$ is even,
$\alpha(\xi(3))$ is odd. Likewise, if $\alpha(1)$ is even, $\alpha(2)$ is odd,
$\alpha(3)$ is even then $\alpha(\xi(1))$ is even, $\alpha(\xi(2))$ is odd,
$\alpha(\xi(3))$ is even. In either case, we conclude that if $\alpha\in
PAP_{n}$ then $\Xi(\alpha)\in PAP_{n}$; thus, $\Xi$ is, indeed, an involution
on $PAP_{n}$. Note that since $\alpha(1)\neq\alpha(3)$, the involution has no
fixed points, that is, for every permutation $\alpha$ we have
$\Xi(\alpha)\neq\alpha$.

Now we will prove that $\Xi$ always maps an element of $\Gamma_{n}^{n-1}$ to
$-\Gamma_{n}^{n-1}$ and vice versa. Compare sets of inversions
$I^{\prime}=\mathrm{inv}(\alpha\upharpoonright[n]\setminus\\{1\\})$ and
$I^{\prime\prime}=\mathrm{inv}(\alpha(\xi(\cdot))\upharpoonright[n]\setminus\\{3\\})$.
Since $\xi$ swaps $1$ and $3$, the set $I^{\prime}$ contains an inversion
$2,3$ if and only if the set $I^{\prime\prime}$ does not contain an inversion
$1,2$. All other inversions in $I^{\prime}$ and $I^{\prime\prime}$ are the
same (subject to changing notation from $3$ to $1$ when comparing $I^{\prime}$
and $I^{\prime\prime}$). In terms of the number of elements, this means that
either $|I^{\prime}|$ is odd and $|I^{\prime\prime}|$ is even, or
$|I^{\prime}|$ is even and $|I^{\prime\prime}|$ is odd. Therefore, either
$\alpha\in-\Gamma_{n}^{n-1}$ and $\Xi(\alpha)\in\Gamma_{n}^{n-1}$, or
$\alpha\in\Gamma_{n}^{n-1}$ and $\Xi(\alpha)\in-\Gamma_{n}^{n-1}$.

The existence of the involution $\Xi$ proves the result.

∎

## References

  * [1] A. Y. Aizenstat, “The defining relations of the endomorphism semigroup of a finite linearly ordered set,” in _Sibirsk. Mat._ , vol. 3, 1962, pp. 161–169.
  * [2] J. Howie, “Products of idempotents in certain semigroups of transformations.” _Proceedings of the Edinburgh Mathematical Society_ , vol. 17, no. 3, 1971\.
  * [3] G. M. Gomes and J. M. Howie, “On the ranks of certain semigroups of order-preserving transformations,” in _Semigroup Forum_ , vol. 45. Springer, 1992, pp. 272–282.
  * [4] P. M. Higgins, “Combinatorial results for semigroups of order-preserving mappings,” in _Mathematical Proceedings of the Cambridge Philosophical Society_ , vol. 113, no. 2. Cambridge University Press, 1993, pp. 281–296.
  * [5] A. Vernitskii and M. Volkov, “The proof and the generalization of higgins’ theorem on divisors of semigroups of order-preserving mappings,” _Russian Mathematics-New York_ , vol. 39, no. 1, pp. 34–39, 1995.
  * [6] P. M. Catarino and P. M. Higgins, “The monoid of orientation-preserving mappings on a chain,” in _Semigroup Forum_ , vol. 2, no. 58, 1999, pp. 190–206.
  * [7] V. H. Fernandes, “The monoid of all injective orientation preserving partial transformations on a finite chain,” _Communications in Algebra_ , vol. 28, no. 7, pp. 3401–3426, 2000.
  * [8] P. M. Higgins and A. Vernitski, “Orientation-preserving and orientation-reversing mappings: a new description,” in _Semigroup Forum_ , vol. 104, no. 2. Springer, 2022, pp. 509–514.
  * [9] V. H. Fernandes and T. Paulista, “On the monoid of partial isometries of a cycle graph,” _Turkish Journal of Mathematics_ , vol. 47, no. 6, pp. 1746–1760, 2023.
  * [10] S. Tanimoto, “Parity-alternate permutations and signed eulerian numbers,” _arXiv preprint math/0612135_ , 2006.
  * [11] ——, “Combinatorics of the group of parity alternating permutations,” _Advances in Applied Mathematics_ , vol. 44, no. 3, pp. 225–230, 2010.