* [7] Z. Chen, & Z. J. Liu, Omni-Lie algebroids. Journal of Geometry and Physics 60, 799-808 (2010); arXiv:0710.1923 [math-ph].
  * [8] M. Boucetta, Riemannian geometry of Lie algebroids. Journal of the Egyptian Mathematical Society, 19, 57-70 (2011); arXiv:0806.3522 [math.DG].
  * [9] Y. Kosmann-Schwarzbach, & K. C. H. Mackenzie, Differential operators and actions of Lie algebroids. Proceedings of Quantization, Poisson Brackets and Beyond, (2002); arXiv:math/0209337 [math.DG].
  * [10] N. Bourbaki, Algébre. Hermann, Paris (1970).
  * [11] A. Weinstein, Omni-Lie algebras. Berkeley CPAM Preprint Series, 774, (1999); arXiv:math/9912190 [math.RT].
  * [12] R. J. Fernandes, Lie algebroids, holonomy and characteristic classes. Advances in Mathematics, 170 , 119-179 (2002); arXiv:math/0007132[math.DG].
  * [13] T. Dereli, & K. Doğan, Metric-connection geometries on pre-Leibniz algebroids: A search for geometrical structure in string models. Journal of Mathematical Physics 62, (2021); arXiv:2006.05957 [hep-th].
  * [14] I. Vaisman, Transitive Courant algebroids. International Journal of Mathematics and Mathematical Sciences 2005, 1737-1758 (2005); arXiv:math/0407399 [math.DG].
  * [15] A. J. Bruce, & J. Grabowski, Pre-Courant algebroids. Journal of Geometry and Physics 142, 254-273 (2019); arXiv:1608.01585 [math-ph].
  * [16] P. Ševera, Letters to Alan Weinstein about Courant algebroids. Unpublished manuscript (2017); arXiv:1707.00265 [math.DG].
  * [17] B. Jurc̆o, & J. Vysoký, Leibniz algebroids, generalized Bismut connections and Einstein-Hilbert actions. Journal of Geometry and Physics 97, 25-33 (2015); arXiv:1503.03069 [hep-th].
  * [18] T. Dereli, & K. Doğan, ‘Anti-commutable’ pre-Leibniz algebroids and admissible connections, (submitted) (2021); arXiv:2108.10199 [math.DG].
  * [19] K. Doğan, ‘Statistical geometry and Hessian structures on pre-Leibniz algebroids, Journal of Physics: Conference Series 2191, 012011 (2022); arXiv:2109.03916 [math.DG].
  * [20] J. Grabowski, D. Khudaverdyan, & N. Poncin, The supergeometry of Loday algebroids. The Journal of Geometric Mechanics 5 , 185-21 (2011); arXiv:1103.5852 [math.DG].
  * [21] K. C. H. Mackenzie, General theory of Lie groupoids and Lie algebroids. University Press, Cambridge (2005).
  * [22] D. Baraglia, Leibniz algebroids, twistings and exceptional generalized geometry. Journal of Geometry and Physics 62 , 903-934 (2012); arXiv:1101.0856 [math.DG].
  * [23] M. Grützmann, $H$-twisted Lie algebroids. Journal of Geometry and Physics 61, 476-484 (2011); arXiv:1005.5680 [math.DG].
  * [24] M. J. Lean, Dorfman connections and Courant algebroids. Journal de Mathématiques Pures et Appliquées 116, 1-39 (2018); arXiv:1209.6077 [math.DG].
  * [25] M. Grützmann, $H$-twisted Courant algebroids. (2011); arXiv:1101.0993 [math.DG].
  * [26] M. Hansen, & T. Strobl, First class constrained systems and twisting of Courant algebroids by a closed 4-form. Fundamental Interactions: A Memorial Volume for Wolfgang Kummer, World Scientific, 115–144 (2010); arXiv:0904.0711 [hep-th].
  * [27] Y. Sheng, & Z. Liu, Leibniz 2-algebras and twisted Courant algebroids. Communications in Algebra 41, 1929-1953 (2013); arXiv:1012.5515 [math-ph].
  * [28] J. Crilly, & V. Mathai, Exotic Courant algebroids and T-duality. Journal of Geometry and Physics 163, (2021); arXiv:1909.07127 [hep-th].
  * [29] A. Deser, & C. Sämann, Extended Riemannian geometry I: Local double field theory, Annales Henri Poincaré 19, 2297–2346 (2018); arXiv:1611.02772 [hep-th].
  * [30] A. G. Tortorella, The deformation $L_{\infty}$ algebra of a Dirac-Jacobi structure, (2021); arXiv:2111.07467 [math.DG].
  * [31] J. Grabowski, & G. Marmo, Jacobi structures revisited. Journal of Physics A: Mathematical and General 34, 10975–10990 (2001); arXiv:math/0111148 [math.DG].
  * [32] D. Baraglia, Conformal Courant algebroids and orientifold T-duality. International Journal of Geometric Methods in Modern Physics 10, 1-35 (2013); arXiv:1109.0875 [math.DG].
  * [33] C. Marle, Calculus on Lie algebroids, Lie groupoids and Poisson manifolds. Dissertationes Mathematicae 457, 1-57 (2008); arXiv:0806.0919 [math.DG].
  * [34] I. Vaisman, Lectures on the geometry of Poisson manifolds. Birkhäuser, Basel (1994).
  * [35] Y. Bi, L. Vitagliano, & T. Zhang, Higher omni-Lie algebroids. Journal of Lie Theory 29, 881-899 (2019); arXiv:1812.09496 [math.DG].
  * [36] K. C. H. Mackenzie, & P. Xu, Lie bialgebroids and Poisson groupoids. Duke Mathematical Journal 73, 415-452 (1984).
  * [37] Z. Liu, A. Weinstein, & P. Xu, Manin triples for Lie bialgebroids. Journal of Differential Geometry 45, 547-674 (1997); arXiv:1812.09496 [math.DG].
  * [38] T. Mokri, Matched pairs of Lie algebroids. Glasgow Mathematical Journal 39, (2009).
  * [39] G. S. Rinehart, Differential forms on general commutative algebras. Transactions of the American Mathematical Society 108, 195-222 (1963).


_Receive-validity:_ Consider any message $m$, where $p_{s}=m.\mathsf{sender}$,
$p_{r}=m.\mathsf{receiver}$ and $j=m.\mathsf{round}$, such that $m$ is
received or receive-omitted in $\mathcal{B}^{*}_{r}$. We distinguish three
cases:

    * –

Let $p_{r}\in A$. In this case, $m\in\mathit{to}_{r}$ (line 10) in the $j$-th
iteration of the for loop at line 8. Therefore, $m\in\mathit{sent}_{s}$ in the
$j$-th iteration of the for loop at line 8. Hence,
$m\in\mathsf{sent}(\mathcal{B}_{s}^{*},j)$ (line 13 or line 18).

    * –

Let $p_{r}\in B$. We further distinguish two scenarios:

      * *

Let $m$ be received in $\mathcal{B}_{r}^{*}$. In this case,
$m\in\mathsf{received}(\mathcal{B}_{r},j)$ (line 16). As
$\mathcal{E}_{0}^{B(k_{1})}$ satisfies receive-validity,
$m\in\mathsf{sent}(\mathcal{B}_{s},j)$. If $j<k_{1}$, then
$m\in\mathsf{sent}(\mathcal{B}_{s}^{*},j)$ as, until (and excluding) round
$k_{1}$, all processes send the same messages as in
$\mathcal{E}_{0}^{B(k_{1})}$. Otherwise, $s\in B$ (as $B$ is isolated from
round $k_{1}$ in $\mathcal{E}_{0}^{B(k_{1})}$), which implies that
$m\in\mathsf{sent}(\mathcal{B}_{s}^{*},j)$.

      * *

Let $m$ be receive-omitted in $\mathcal{B}_{r}^{*}$. In this case, $m\in
to_{r}$ (line 10) in the $j$-th iteration of the for loop at line 8. Hence,
$\mathit{m}\in\mathit{sent}_{s}$ in the $j$-th iteration of the for loop at
line 8, which implies that $m\in\mathsf{sent}(\mathcal{B}_{s}^{*},j)$.

    * –

Let $p_{r}\in C$. There are two scenarios:

      * *

Let $m$ be received in $\mathcal{B}_{r}^{*}$. In this case,
$m\in\mathsf{received}(\mathcal{B}_{r}^{\prime},j)$ (line 17). As
$\mathcal{E}_{b}^{C(k_{2})}$ satisfies receive-validity,
$m\in\mathsf{sent}(\mathcal{B}_{s}^{\prime},j)$. If $j<k_{2}$, then
$m\in\mathsf{sent}(\mathcal{B}_{s}^{*},j)$ as, until (and excluding) round
$k_{2}$, all processes send the same messages as in
$\mathcal{E}_{b}^{C(k_{2})}$. Otherwise, $s\in C$ (as $C$ is isolated from
round $k_{2}$ in $\mathcal{E}_{b}^{C(k_{2})}$), which implies that
$m\in\mathsf{sent}(\mathcal{B}_{s}^{*},j)$.

      * *

Let $m$ be receive-omitted in $\mathcal{B}_{r}^{*}$. In this case, $m\in
to_{r}$ (line 10) in the $j$-th iteration of the for loop at line 8. Hence,
$\mathit{m}\in\mathit{sent}_{s}$ in the $j$-th iteration of the for loop at
line 8, which implies that $m\in\mathsf{sent}(\mathcal{B}_{s}^{*},j)$.

  * •

_Omission-validity:_ Only processes in $B\cup C$ (potentially) receive-omit
some messages. As $\mathcal{F}^{*}=B\cup C$, omission-validity is satisfied.

Second, we prove the indistinguishability statement. Let $p_{i}\in B$. The
$\mathsf{merge}$ procedure (Algorithm 5) ensures that
$\mathsf{received}(\mathcal{B}_{i}^{*},j)=\mathsf{received}(\mathcal{B}_{i},j)$,
for every round $j\geq 1$. Moreover, for every round $j\geq 1$,
$\mathsf{state}(\mathcal{B}_{i}^{*},j)=\mathsf{state}(\mathcal{B}_{i},j)$ and
$\mathsf{sent}(\mathcal{B}_{i}^{*},j)\cup\mathsf{send\\_omitted}(\mathcal{B}_{i}^{*},j)=\mathsf{sent}(\mathcal{B}_{i},j)\cup\mathsf{send\\_omitted}(\mathcal{B}_{i},j)$.
The symmetric argument holds if $p_{i}\in C$. Hence the indistinguishability
statement holds.

Finally, every process $p_{B}\in B$ (resp., $p_{C}\in C$) exhibits the same
behavior (except for potentially new receive-omitted messages) in
$\mathcal{E}^{*}$ as in $\mathcal{E}_{0}^{B(k_{1})}$ (resp.,
$\mathcal{E}_{b}^{C(k_{2})}$). Therefore, group $B$ (resp., $C$) is isolated
from round $k_{1}$ (resp., $k_{2}$) in $\mathcal{E}^{*}$. ∎

###  A.3. Lemmas 4 and 6

First, we formally prove Lemma 4.

See 4

###### Proof.

Let $\mathcal{E}=[\mathcal{F},\mathcal{B}_{1},...,\mathcal{B}_{n}]$, where
$\mathcal{F}=Y\cup Z$. For every process $p_{i}\in Y$, we define
$\mathcal{M}_{X\to p_{i}}$:

$\mathcal{M}_{X\to
p_{i}}=\\{m\in\mathsf{all\\_receive\\_omitted}(\mathcal{B}_{i})\,|\,m.\mathsf{sender}\in
X\\}.$

For every set $Y^{\prime\prime}\subseteq Y$, let $\mathcal{M}_{X\to
Y^{\prime\prime}}=\bigcup\limits_{p\in Y^{\prime\prime}}\mathcal{M}_{X\to p}$.
As correct processes (i.e., processes from group $X$) send fewer than
$\frac{t^{2}}{32}$ messages in $\mathcal{E}$, $|\mathcal{M}_{X\to
Y}|<\frac{t^{2}}{32}$. Therefore, there does not exist a set $Y^{*}\subseteq
Y$ of $|Y^{*}|\geq\frac{|Y|}{2}$ processes such that, for every process
$p_{Y^{*}}\in Y^{*}$, $|\mathcal{M}_{X\to p_{Y^{*}}}|\geq\frac{t}{2}$. This
implies that there exists a set $Y^{\prime}\subseteq Y$ of
$|Y^{\prime}|>\frac{|Y|}{2}$ processes such that, for every process
$p_{Y^{\prime}}\in Y^{\prime}$, $|\mathcal{M}_{X\to
p_{Y^{\prime}}}|<\frac{t}{2}$.

Fix any process $p_{Y^{\prime}}\in Y^{\prime}$. By contradiction, suppose that
$p_{Y^{\prime}}$ does not decide $b_{X}$ in $\mathcal{E}$. We define a set of
processes $\mathcal{S}$:

$\mathcal{S}=\\{p_{s}\in\Pi\,|\,\exists
m\in\mathsf{all\\_receive\\_omitted}(\mathcal{B}_{Y^{\prime}}):m.\mathsf{sender}=p_{s}\\}.$

Note that $|\mathcal{S}\cap X|<\frac{t}{2}$ (since $|\mathcal{M}_{X\to
p_{Y^{\prime}}}|<\frac{t}{2}$) and $\mathcal{S}\subsetneq X\cup Z$. Let
$\mathcal{E}^{\prime}=[F^{\prime},\mathcal{B}_{1}^{\prime},...,\mathcal{B}_{n}^{\prime}]=\mathsf{swap\\_omission}(\mathcal{E},p_{Y^{\prime}})$.
By Lemma 5, $\mathcal{E}^{\prime}$ is an execution that is indistinguishable
from $\mathcal{E}$ to every process. Moreover, Lemma 5 proves that
$p_{Y^{\prime}}\notin\mathcal{F}^{\prime}$ (since $p_{Y^{\prime}}$ does not
commit any send omission fault) and that there exists a process $p_{X}\in
X\setminus\mathcal{S}$ (since $X\setminus\mathcal{S}\neq\emptyset$) such that
$p_{X}\notin\mathcal{F}^{\prime}$. Importantly, $p_{X}$ decides $b_{X}$ in
$\mathcal{E}^{\prime}$. Therefore, $\mathcal{E}^{\prime}$ violates either
_Termination_ (if $p_{Y^{\prime}}$ does not decide) or _Agreement_ (if
$p_{Y^{\prime}}$ decides $1-b_{X}$), which concludes the proof. ∎

Lastly, we prove Lemma 6.

See 6

###### Proof.

For $\mathcal{A}$ to satisfy _Termination_ and _Agreement_ , all processes
from group $A$ decide $b_{1}$ (resp., $b_{2}$) in $\mathcal{E}_{0}^{B(k_{1})}$
(resp., $\mathcal{E}_{b}^{C(k_{2})}$). Given the partition $(A\cup
C,B,\emptyset)$ of $\Pi$ and the execution $\mathcal{E}_{0}^{B(k_{1})}$, Lemma
4 proves that there exists a set $B^{\prime}\subseteq B$ of more than
$\frac{|B|}{2}$ processes such that every process $p_{B^{\prime}}\in
B^{\prime}$ decides $b_{1}$ in $\mathcal{E}_{0}^{B(k_{1})}$. Similarly, given
the partition $(A\cup B,C,\emptyset)$ of $\Pi$ and the execution
$\mathcal{E}_{b}^{C(k_{2})}$, Lemma 4 proves that there exists a set
$C^{\prime}\subseteq C$ of more than $\frac{|C|}{2}$ processes such that every
process $p_{C^{\prime}}\in C^{\prime}$ decides $b_{2}$ in
$\mathcal{E}_{b}^{C(k_{2})}$.

Let
$\mathcal{E}=\mathsf{merge}(\mathcal{E}_{0}^{B(k_{1})},\mathcal{E}_{b}^{C(k_{2})})$.
By Lemma 6, $\mathcal{E}$ is an infinite execution such that group $B$ (resp.,
$C$) is isolated from round $k_{1}$ (resp., $k_{2}$) and no process $p_{B}\in
B$ (resp., $p_{C}\in C$) distinguishes $\mathcal{E}$ from
$\mathcal{E}_{0}^{B(k_{1})}$ (resp., $\mathcal{E}_{b}^{C(k_{2})}$). Therefore,
all processes from $B^{\prime}$ (resp., $C^{\prime}$) decide $b_{1}$ (resp.,
$b_{2}$) in $\mathcal{E}$. Let $b_{A}$ be the decision of processes from group
$A$ in $\mathcal{E}$; such a decision must exist as $\mathcal{A}$ satisfies
_Termination_ and _Agreement_ and processes from group $A$ are correct in
$\mathcal{E}$. Given the partition $(A,B,C)$ of $\Pi$ and the newly
constructed execution $\mathcal{E}$, Lemma 4 proves that $b_{1}=b_{A}$.
Similarly, given the partition $(A,C,B)$ of $\Pi$ and the execution
$\mathcal{E}$, Lemma 4 shows that $b_{2}=b_{A}$. As $b_{1}=b_{A}$ and
$b_{A}=b_{2}$, $b_{1}=b_{2}$, which concludes the proof. ∎

##  Appendix B Proof of Lemma 3

This section provides a proof of Lemma 3 (introduced in § 4.2).

See 3

###### Proof.

We prove the lemma by contradiction. Hence, let $v^{\prime}\in\mathcal{V}_{O}$
be decided in $\mathcal{E}$, and let
$v^{\prime}\notin\bigcap\limits_{c^{\prime}\in\mathit{Cnt}(c)}\mathit{val}(c^{\prime})$.
Therefore, there exists an input configuration $c^{\prime}\in\mathcal{I}$ such
that (1) $c\sqsupseteq c^{\prime}$, and (2)
$v^{\prime}\notin\mathit{val}(c^{\prime})$. Observe that
$\pi(c^{\prime})\subsetneq\pi(c)$. Let $p_{i}$ be any process such that
$p_{i}\in\pi(c^{\prime})$; note that such a process exists as
$|\pi(c^{\prime})|\geq n-t$ and $n-t>0$. If $\mathcal{E}$ is an infinite
execution, then $\mathcal{E}_{i}\leftarrow\mathcal{E}$; otherwise, let
$\mathcal{E}_{i}$ be any infinite continuation of $\mathcal{E}$ such that
$\mathit{Correct}_{\mathcal{A}}(\mathcal{E}_{i})=\mathit{Correct}_{\mathcal{A}}(\mathcal{E})$.
Note that $c=\mathsf{input\\_conf}(\mathcal{E}_{i})$ and $p_{i}$ decides
$v^{\prime}$ in $\mathcal{E}_{i}$ (to satisfy _Termination_ and _Agreement_).

Consider now another infinite execution
$\mathcal{E}^{\prime}\in\mathit{execs}(\mathcal{A})$ which is identical to
$\mathcal{E}_{i}$, except that only processes in $\pi(c^{\prime})$ are correct
in $\mathcal{E}^{\prime}$. As $c\sqsupseteq c^{\prime}$, each process which
belongs to $\pi(c^{\prime})$ has identical proposals in $c^{\prime}$ and $c$,
which implies that $c^{\prime}=\mathsf{input\\_conf}(\mathcal{E}^{\prime})$.
Moreover, observe that (1)
$p_{i}\in\mathit{Correct}_{\mathcal{A}}(\mathcal{E}^{\prime})$ (as
$p_{i}\in\pi(c^{\prime})$), and (2) $p_{i}$ decides $v^{\prime}$ in
$\mathcal{E}^{\prime}$ as $\mathcal{E}^{\prime}$ and $\mathcal{E}_{i}$ are
indistinguishable to $p_{i}$. Thus, we reach a contradiction as a correct
process (namely, $p_{i}$) decides value
$v^{\prime}\notin\mathit{val}(c^{\prime})$ in execution $\mathcal{E}^{\prime}$
which corresponds to $c^{\prime}$, thus violating $\mathit{val}$.
Consequently, the lemma holds. ∎

##  Appendix C Proof of the Reduction (Algorithm 1)

In this section, we prove the correctness of the reduction (Algorithm 1) from
weak consensus to any solvable non-trivial agreement problem $\mathcal{P}$.
First, we prove that $v_{0}^{\prime}\neq v_{1}^{\prime}$ (see Table 2). Recall
that $v_{0}^{\prime}$ (resp., $v_{1}^{\prime}$) is decided in the fully
correct infinite execution $\mathcal{E}_{0}$ (resp., $\mathcal{E}_{1}$) of
$\mathcal{A}$ such that $\mathsf{input\\_conf}(\mathcal{E}_{0})=c_{0}$ (resp.,
$\mathsf{input\\_conf}(\mathcal{E}_{1})=c_{1}$), where $\mathcal{A}$ is the
fixed algorithm which solves $\mathcal{P}$.

###### Lemma 0.

$v_{0}^{\prime}\neq v_{1}^{\prime}$.

###### Proof.

Due to Lemma 3, $v_{1}^{\prime}$ is admissible according to $c_{1}^{*}$ (as
$c_{1}\sqsupseteq c_{1}^{*}$; see Table 2). As $v_{0}^{\prime}$ is not
admissible according to $c_{1}^{*}$, $v_{0}^{\prime}\neq v_{1}^{\prime}$. ∎

We are ready to prove the correctness of our reduction (Algorithm 1).

###### Lemma 0.

Algorithm 1 is a correct weak consensus algorithm with the same message
complexity as $\mathcal{A}$.

###### Proof.

_Termination_ and _Agreement_ of Algorithm 1 follow directly from the
properties of $\mathcal{A}$. Moreover, the message complexity of Algorithm 1
is identical to that of $\mathcal{A}$ as Algorithm 1 introduces no additional
communication. Thus, it is left to prove _Weak Validity_ , which we do by
analyzing two fully correct executions in which all processes propose the same
value:

  * •

Assume that every process is correct and proposes $0$ to weak consensus (line
3). Therefore, every process proposes its proposal from the input
configuration $c_{0}\in\mathcal{I}_{n}$ (of $\mathcal{P}$) to the underlying
algorithm $\mathcal{A}$ (line 5). As every fully correct execution is
completely determined by the proposal of processes, $\mathcal{A}$ exhibits the
execution $\mathcal{E}_{0}$ which decides $v_{0}^{\prime}$. Thus, all
processes decide $0$ from weak consensus (line 10).

  * •

Assume that every process is correct and proposes $1$ to weak consensus (line
3). Hence, every process proposes its proposal from the input configuration
$c_{1}\in\mathcal{I}_{n}$ (of $\mathcal{P}$) to the underlying algorithm
$\mathcal{A}$ (line 7). Hence, $\mathcal{A}$ exhibits the execution
$\mathcal{E}_{1}$ which decides $v_{1}^{\prime}$. By Lemma 1,
$v_{1}^{\prime}\neq v_{0}^{\prime}$, which implies that all processes decide
$1$ from weak consensus (line 12).

As _Termination_ , _Agreement_ and _Weak Validity_ are proven to be satisfied,
the lemma holds. ∎


# Combination of Single and Multi-frame Image Super-resolution: An Analytical
Perspective

Mohammad Mahdi Afrasiabi, Reshad Hosseini and Aliazam Abbasfar M. M.
Afrasiabi, R. Hosseini, and A. Abbasfar are with the Department of Electrical
and Computer Engineering, University of Tehran, Tehran, Iran.
Email<EMAIL_ADDRESS>
###### Abstract

Super-resolution is the process of obtaining a high-resolution image from one
or more low-resolution images. Single image super-resolution (SISR) and multi-
frame super-resolution (MFSR) methods have been evolved almost independently
for years. A neglected study in this field is the theoretical analysis of
finding the optimum combination of SISR and MFSR. To fill this gap, we propose
a novel theoretical analysis based on the iterative shrinkage and thresholding
algorithm. We implement and compare several approaches for combining SISR and
MFSR, and simulation results support the finding of our theoretical analysis,
both quantitatively and qualitatively.

###### Index Terms:

super-resolution, single image, multi-frame, analytical perspective,
theoretical analysis.

##  1 Introduction

Improving the resolution of images is one of the oldest research topics in
image processing and machine vision. The process of transforming one or more
low-resolution (LR) images to a high-resolution (HR) one is called super-
resolution (SR) [1]. Although the sensors (cameras) produced in recent years
have relatively high quality and resolution, the need to increase the
resolution of images is still substantial for the following reasons [2, 3, 4].
1) Increasing the resolution of images recorded in the past with low-
resolution cameras. 2) Increasing the resolution of medical or satellite
images, which are usually expensive, and progress in their imaging technology
is slow. 3) Combining the images taken by a handheld camera in a burst shot
mode to achieve a higher quality. 4) Increasing the resolution of surveillance
and security cameras, especially for facial recognition. 5) Saving the network
bandwidth by sending LR images at the source and converting them to HR ones at
the destination.

SR methods can be classified from different perspectives and in different ways
[2, 3]. In a general and conventional division, there are two families of SR
methods: SISR and MFSR. SISR works with only one input image while MFSR uses
several input images to reach the final result. Although at the first glance,
it may seem that SISR methods are special cases of MFSR ones, in general, the
approaches and tools of these methods have major differences111It is important
to mention that by MFSR methods, we mean those methods that use shift-and-add
[5] approach to solve the SR problem, and have not used SISR methods in their
algorithm. For the methods that use both MFSR and SISR approaches, we use a
different naming. [3].

The SR problem is inherently ill-posed. Especially in SISR, it needs image
priors to remove the problem from the under-determined condition [6]. The
initial SISR methods tried to increase the resolution of the LR image with the
help of different interpolation techniques [7, 8, 9]. Since the interpolation
operation has inherent smoothness, this approach was ineffective for
reconstructing high-frequency regions and edges. Although these methods do not
perform well, they are still used in many commercial applications due to their
simplicity. Later, methods were developed that used large number of image
patches (previously prepared) to learn efficient mappings between LR and HR
images [10, 11, 12, 13, 14]. These example-based methods had a much higher
performance than the interpolation-based ones. A branch of example-based
methods used the assumption of the sparsity of natural images and developed
methods based on solving the sparse coding (SC) problem with the help of LR
and HR dictionaries prepared in advance [6, 15]. These methods are based on
the assumption that the coefficients of the bases obtained by solving the SC
problem for the LR image are approximately equal to the coefficients obtained
for the HR one. This assumption holds for natural images under some mild
conditions [6]. Further, with the expansion of deep learning methods in
computer vision, the field of SR was not the exception [16, 17, 18, 19, 20],
and the first SR method based on deep learning was widely welcomed by
researchers [16]. The authors of [16] showed better performance obtained by
their proposed neural network framework compared to the previous SR methods.
They demonstrated that the proposed neural network is somehow equivalent to
the SC process, but since all its parameters are trained carefully, it
achieves higher performance.

MFSR was first introduced in [1] using a frequency domain approach only for
the translational motions. [21] proposed a fast method in pixel domain for an
invariant point spread function and under additive white Gaussian noise (AWGN)
condition like models in [22, 23]. Farsiu et al. in [5] described a new method
to consider other types of noise such as salt-and-pepper. [5] is a strong work
in the field of MFSR because of proposing a robust algorithm with clear
mathematical justifications.

Regardless of a few pieces of research that use a combination of single-image
and multi-frame SR (ComSR) techniques such as [24, 25, 26, 27] in recent
years, SISR and MFSR methods have been evolved separately and independently.
To the best of our knowledge, before the suggested method in [24], SC-based
methods were employed only in SISR. As proposed in [24] and [25], the LR
images are concatenated, and then the corresponding sparse code is found. In
[27], SR process is divided into two phases. At first, each LR image is
magnified and recovered using a SISR method, and then an MFSR method is
applied over the magnified images. We call this approach single first multi
last (SFML) ComSR approach. In [26], the process is done in the inverse order
of [27] where an MFSR method combines the LR images first, and then a SISR
method is applied on the resulting image. We call this multi first single last
(MFSL) ComSR approach.

To the best of our knowledge, researchers have not yet theoretically answered
the question of “what is the optimal way to combine SISR and MFSR methods?”.
Should the SISR be applied to the input images first, then the MFSR or vice
versa? In this article, with the help of mathematical analysis, we will show
the optimal way to combine SISR and MFSR. The main contribution of this paper
is that we present an analytical approach on how to combine SISR and MFSR. We
show that MFSL ComSR is optimal. Based on this analytical result, we also
propose a novel SR method which is robust to relatively high AWGN. Our
analytical result is validated by numerous experiments. Especially when the
up-scaling factor ($r$) is a multiplication of two smaller integers
($r=r_{1}\times r_{2}$), we show that the optimal method of combining the LR
images is still achieved by MFSL ComSR, that is by using MFSR (with $r$ up-
scale) firstly, then applying a SISR network.

##  2 Related Work

The related work is divided into three subsections, each for an approach of
solving the SR problem, namely SISR, MFSR and ComSR.

###  2.1 Single image super-resolution

SISR methods can be divided as follows.

1) Interpolation-based methods. Early SISR approaches simply use interpolation
(bi-linear, bi-cubic, Lanczos) and try to reconstruct new pixels by weighted
averaging of neighboring pixels [7, 8, 9]. It is obvious that these methods
are simple but have intrinsic smoothness (low-pass filtering). Therefore, they
are not able to recover high-frequency contents, especially at the edges.

2) Example-based (patch-based) methods. These methods use the existing paired
HR and LR training images to find a proper mapping between HR and LR patches.
Some of them obtain this mapping using internal learning (only internal
patches of down-scaled and original given image, as LR and HR patches,
correspondingly) [10, 11] and the other from external learning (given
databases of LR and HR patches) [12, 13, 14].

3) Sparse coding-based methods. These methods use the assumption of the
possible sparse representation of natural images [6, 15]. This assumption
comes from the inherent redundancy of natural images. These methods need to
train over-complete LR and HR dictionaries to solve an SC problem [15]. Each
patch of an LR image could be represented by a linear combination of a few
columns of the LR dictionary. The coefficients of this linear combination are
as same as those we need to generate an HR patch (from corresponding columns
of the HR dictionary) [15].

4) Deep learning-based methods. With the progress of deep learning approaches
in a broad spectrum of computer vision applications [28, 29, 30], a category
of SISR methods was developed using deep learning techniques. The pioneering
work of [16] applied a simple convolutional neural network (CNN) to solve the
SISR problem and outperformed the classical methods. This approach was matured
in [17] and then has been developed in various ways by using new architectures
[31, 18, 19, 32, 33, 34, 35] or proper loss functions [36, 37, 38, 39, 40].
Some new methods use real-world images instead of synthetic images to obtain
better results in real scenarios [41, 42, 40, 33]. In [33], statistics of
image priors are injected into the model and a bayesian framework is
investigated.

###  2.2 Multi-frame super-resolution

MFSR methods can be divided as follows.

1) Frequency domain-based methods. These methods transfer the problem into the
frequency domain (wavelet, DCT, and etc.) [1, 43, 44, 45, 46]. They model the
shifts between LR images in a simple translational model, thus avoiding high
computational complexity. These methods suffer from their high sensitivity to
complex shifts and model mismatch.

2) Iterative back propagation. These methods are inherently iterative. They
guess an initial estimate of the HR image by a simple intuition like averaging
of the LR images and then try to refine that using an iterative optimization
process [47, 48, 49, 50].

3) Regularization-based methods. These methods add some regularization to the
cost function of the MFSR problem. These regularization terms convey image
priors and hence result in more robust HR images with detail-preserving
features[51, 5, 52, 53]. In fact, this technique is equivalent to a maximum a
posteriori (MAP) solution with a special noise probability density function
[2].

4) Deep learning-based methods. These methods in MFSR are not as developed as
in SISR [54]. However, some noticeable architectures have been developed for
specific applications. [55] improves the resolution of human face images. In
this method, special features are extracted from each of the LR images and
finally, with the help of these features and the relative shifts of LR images,
the final image is reconstructed. [56] developed a recursive fusion framework
for combining satellite images. This method uses encoder-decoder architecture.
The method of [57], known as DeepSum, uses a CNN architecture and benefits
from spatial and temporal correlations between LR images. In this method, LR
images are registered implicitly. In [58, 59, 60], deep learning-based methods
have been developed and used in satellite and remote sensing applications. In
[54], for general applications (such as burst SR in handheld photography), a
method was proposed that explicitly uses the information of shifts between LR
images for their combination. The final fusion uses an attention-based
architecture.

###  2.3 Combined SISR and MFSR

Some researchers have made efforts to combine SISR and MFSR approaches. In
[24], a method is introduced that extends the SC framework to multi-frame. In
this approach, instead of solving the SC problem for one input image, the
problem is solved for several input images by first registering LR images and
then solving a new SC problem. This approach was proceeded in [25], wherein
two phases of registration and solving the SC problem are presented in the
form of solving a single optimization problem. Among other approaches on
ComSR, we can refer to [26], in which the input LR images are first converted
into a higher resolution image by a conventional MFSR method and then, the
result passes through a SISR network. On the other hand, method of [27] passes
the input LR images separately through the SISR network and then combines the
results of the previous step with the help of a conventional MFSR method. As
it is clear, this method has a higher computational complexity than [26] or
other existing methods. In [61], an optimization problem based on input LR
images and their corresponding semi-HR images (generated by a SISR method) was
proposed. This optimization problem has a closed form solution, which is
actually a weighted combination of LR and semi-HR images.

##  3 Background

In this section, we provide an overview of the methods needed for our analysis
explained in the next section. These methods are the MFSR method based on a
shift-and-add approach, the SISR method based on SC, and finally, the
iterative shrinkage and thresholding algorithm (ISTA).

###  3.1 MFSR based on shift-and-add

In SR applications, image degradation model is almost always described as [5]

$\mathbf{y}_{k}=\mathbf{R}\mathbf{F}_{k}\mathbf{H}\mathbf{x}+\mathbf{w}_{k},$
(1)

where $\mathbf{x}\in\mathbb{R}^{n}$ and $\mathbf{y}_{k}\in\mathbb{R}^{m}$ are
HR image (ground truth) and $k$-th LR image (from $N$ LR images),
respectively. For the sake of simplicity, all images are exhibited by vectors
rather than matrices and their columns are stacked lexicographically. Also,
$\mathbf{R}\in\mathbb{R}^{m\times n}$, $\mathbf{F}_{k}\in\mathbb{R}^{n\times
n}$, and $\mathbf{H}\in\mathbb{R}^{n\times n}$ are down-sampling, translation
(shift), and smoothing filter (blur) matrices (operators), respectively. AWGN
is represented by vector $\mathbf{w}_{k}\in\mathbb{R}^{m}$.

We define vector $\mathbf{z}=\mathbf{Hx}$. Actually, the vector $\mathbf{z}$
is the blurred version of the vector $\mathbf{x}$. A common way to obtain the
HR image is defining a cost function in the form of $\mathbf{L}_{2}$ norm [5,
21] such as

$\rho(\mathbf{z})=\sum_{k=1}^{N}\left\lVert\mathbf{RF}_{k}\mathbf{z-y}_{k}\right\rVert^{2}.$
(2)

So, the optimum solution ($\widehat{\mathbf{z}}$) is given by

$\widehat{\mathbf{z}}=\sum_{k=1}^{N}{\mathbf{F}^{T}_{k}}{\mathbf{R}^{T}}{\mathbf{y}_{k}}.$
(3)

###  3.2 SISR based on sparse coding

Let $\mathbf{x}\in\mathbb{R}^{n}$ and $\mathbf{y}\in\mathbb{R}^{m}$ be HR and
LR vectors (patches), respectively. Also, matrices
$\mathbf{D}_{H}\in\mathbb{R}^{n\times K}$ and
$\mathbf{D}_{L}\in\mathbb{R}^{m\times K}$ are HR and LR over-complete
dictionaries, respectively. These dictionaries have been learned from
thousands of training HR and LR pairs of patches [15]. SC-based methods state
that $\mathbf{y}$ can be represented as

$\mathbf{y}=\mathbf{D}_{L}\mathbf{\alpha},$ (4)

where $\mathbf{\alpha}\in\mathbb{R}^{K}$ is a sparse vector. Then, under mild
conditions, the vector $\mathbf{x}$ could be reconstructed [15] using

$\widehat{\mathbf{x}}=\mathbf{D}_{H}\mathbf{\alpha}.$ (5)

###  3.3 ISTA method

ISTA method states that the solution of the SC problem, that is $\alpha$ in
(4), could be obtained by the following fixed-point equation [62]

$\alpha^{(n+1)}=h_{\tau}\big{\\{}{\mathbf{D}^{T}_{L}}{\mathbf{y}}+{\mathbf{S}}{\alpha^{(n)}}\big{\\}},$
(6)

where $n$ and $h_{\tau}\\{.\\}$ indicate iteration index and shrinkage
operator, and initial point is set to zero $\alpha^{(0)}=0$. Matrix
$\mathbf{S}$ is defined as

${\mathbf{S}}=L{\mathbf{I}}-{\mathbf{D}^{T}_{L}}{\mathbf{D}_{L}},$ (7)

where $\mathbf{I}$ and $L$ are identity matrix and upper bound of the largest
eigenvalue of the matrix ${\mathbf{D}^{T}_{L}}{\mathbf{D}_{L}}$.

##  4 Optimality Analysis of MFSL ComSR

In this section, we show the optimality of MFSL ComSR method by deriving the
ISTA method for a collection of LR images. To this end, we first derive a
formulation of the ISTA method based on HR dictionary. Then, we show that the
fixed-point equation of the ISTA method for the collection of LR images is the
same as that of the single image ISTA method with a fused version of LR images
as its input. This fused image can be considered as the result of a
conventional MFSR method.

###  4.1 ISTA formulation based on $\mathbf{D}_{H}$

We represent the formulation of the ISTA method based on $\mathbf{D}_{H}$
rather than $\mathbf{D}_{L}$. We replace $\mathbf{D}_{L}$ in (6) by
${\mathbf{R}}{\mathbf{F}_{0}}{\mathbf{D}_{H}}$. In the case of SISR,
translation matrix $\mathbf{F}_{0}$ is an identity matrix, hence it is not
needed, but we intend to inject it in our formulation to use it the next
subsection.

$\displaystyle\alpha^{(n+1)}$
$\displaystyle=h_{\tau}\big{\\{}{\mathbf{D}^{T}_{L}}{\mathbf{y}}+{\mathbf{S}}{\alpha^{(n)}}\big{\\}}$
$\displaystyle=h_{\tau}\big{\\{}{\mathbf{D}^{T}_{H}}{\mathbf{F}^{T}_{0}}{\mathbf{R}^{T}}{\mathbf{y}}+L{\alpha^{(n)}}-{\mathbf{D}^{T}_{H}}{\mathbf{F}^{T}_{0}}{\mathbf{R}^{T}}{\mathbf{R}}{\mathbf{F}_{0}}{\mathbf{D}_{H}}{\alpha^{(n)}}\big{\\}}$
(8)

###  4.2 Optimality analysis

We use all LR frames ($\\{\mathbf{y}_{k}\\}_{k=1}^{N}$) effectively to
estimate the HR image ($\mathbf{x}$). Similar to MFSR methods, we define
vector $\mathbf{z}=\mathbf{Hx}$ for simplifying the problem. We define a new
vector $\mathbf{y}$ including all the LR images as

$\mathbf{y}^{T}=\begin{bmatrix}\mathbf{y}^{T}_{1}&\mathbf{y}^{T}_{2}&\dots&\mathbf{y}^{T}_{N}\\\
\end{bmatrix}.$ (9)

Inspired by SC-based methods [24], we define $\mathbf{D}_{L}$, the
corresponding LR dictionary of $\mathbf{y}$ including all LR dictionaries of
LR images as

$\mathbf{D}_{L}=\begin{bmatrix}\mathbf{D}_{1}\\\ \mathbf{D}_{2}\\\ \vdots\\\
\mathbf{D}_{N}\end{bmatrix}=\begin{bmatrix}{\mathbf{R}}{\mathbf{F}_{1}}{\mathbf{D}_{H}}\\\
{\mathbf{R}}{\mathbf{F}_{2}}{\mathbf{D}_{H}}\\\ \vdots\\\
{\mathbf{R}}{\mathbf{F}_{N}}{\mathbf{D}_{H}}\end{bmatrix}.$ (10)

Inspired by the ISTA method, we try to solve the SC problem of (4) using (9)
and (10). To this end, the matrix $\mathbf{S}$ of (7) needs to be computed.
First, we calculate the matrix ${\mathbf{D}^{T}_{L}}{\mathbf{D}_{L}}$ of (7),

$\displaystyle{\mathbf{D}^{T}_{L}}{\mathbf{D}_{L}}$
$\displaystyle=\sum_{k=1}^{N}{\mathbf{D}^{T}_{k}}{\mathbf{D}_{k}}=\sum_{k=1}^{N}{{\mathbf{D}^{T}_{H}}{\mathbf{F}^{T}_{k}}{\mathbf{R}^{T}}{\mathbf{R}}{\mathbf{F}_{k}}{\mathbf{D}_{H}}}$
$\displaystyle=\sum_{k=1}^{N}{{\mathbf{D}^{T}_{H}}{\mathbf{C}_{k}}{\mathbf{D}_{H}}}={\mathbf{D}^{T}_{H}}{\Big{[}\sum_{k=1}^{N}{\mathbf{C}_{k}}\Big{]}}{\mathbf{D}_{H}}$
$\displaystyle={\mathbf{D}^{T}_{H}}{\mathbf{C}}{\mathbf{D}_{H}},$ (11)

where ${\mathbf{C}_{k}}$ and ${\mathbf{C}}$ are defined respectively by

${\mathbf{C}_{k}}={\mathbf{F}^{T}_{k}}{\mathbf{R}^{T}}{\mathbf{R}}{\mathbf{F}_{k}},$
(12)

and

$\displaystyle{\mathbf{C}}=\sum_{k=1}^{N}{\mathbf{C}_{k}}.$ (13)

Therefore, the matrix $\mathbf{S}$ of (7) is

${\mathbf{S}}=L{\mathbf{I}}-{\mathbf{D}^{T}_{L}}{\mathbf{D}_{L}}=L{\mathbf{I}}-{\mathbf{D}^{T}_{H}}{\mathbf{C}}{\mathbf{D}_{H}}.$
(14)

Second, we calculate ${\mathbf{D}^{T}_{L}}{\mathbf{y}}$ of (6) as

${\mathbf{D}^{T}_{L}}{\mathbf{y}}=\sum_{k=1}^{N}{\mathbf{D}^{T}_{k}}{\mathbf{y}_{k}}=\sum_{k=1}^{N}{{\mathbf{D}^{T}_{H}}{\mathbf{F}^{T}_{k}}{\mathbf{R}^{T}}}{\mathbf{y}_{k}}={\mathbf{D}^{T}_{H}}{\mathbf{y}_{C}},$
(15)

where fused image (shift-and-add) ${\mathbf{y}_{C}}$ is defined as

${\mathbf{y}_{C}}=\sum_{k=1}^{N}{{\mathbf{F}^{T}_{k}}{\mathbf{R}^{T}}}{\mathbf{y}_{k}}.$
(16)

According to (6), (14), and (15), the fixed-point equation of ISTA is

$\displaystyle\alpha^{(n+1)}$
$\displaystyle=h_{\tau}\big{\\{}{\mathbf{D}^{T}_{L}}{\mathbf{y}}+{\mathbf{S}}{\alpha^{(n)}}\big{\\}}$
$\displaystyle=h_{\tau}\big{\\{}{\mathbf{D}^{T}_{H}}{\mathbf{y}_{C}}+L{\alpha^{(n)}}-{\mathbf{D}^{T}_{H}}{\mathbf{C}}{\mathbf{D}_{H}}{\alpha^{(n)}}\big{\\}}.$
(17)

The above fixed-point equation is very similar to single image ISTA fixed-
point equation of (4.1) with the difference that it has the fused image
$\mathbf{y}_{C}$ as its input and the matrix $\mathbf{C}$ in (4.2) is the
generalization of
${\mathbf{F}^{T}_{0}}{\mathbf{R}^{T}}{\mathbf{R}}{\mathbf{F}_{0}}$ in (4.1).
Therefore, we can conclude that MFSL ComSR method is optimal. In the following
subsection, we explain more on this.

###  4.3 Discussion on optimality analysis

We divide the discussion on optimality analysis into two parts. First, we
consider the case where the up-scaling factor is an indivisible integer such
as 3. Then, we further discuss the case for divisible integers such as 4 in
the next part.

####  4.3.1 Up-scaling factor is an indivisible integer

Figure 1: Various structures for combining SISR and MFSR (Up-scaling factor is
an indivisible integer such as 3.) Figure 2: Various structures for combining
SISR and MFSR (Up-scaling factor is a divisible integer such as 4.)

Assuming the up-scaling factor to be an indivisible integer such as 2 or 3,
one can consider either of four structures of Fig. 1 to do the SR. To the best
of our knowledge, the question of what is the best way (structure or method)
to combine SISR and MFSR (when the up-scaling factor is an indivisible
integer) has not yet been theoretically answered. Based on the examination of
(4.2), we understand that ${\mathbf{y}_{C}}$ should be built first, and then
SISR should be used to improve it. Considering (4.1) and (4.2), we realize
that:

  * -

(4.1) is a special case of (4.2).

  * -

It can be seen from (4.2) that it first combines the up-sampled and shifted
versions of the LR images (by generating ${\mathbf{y}_{C}}$). Then, it finds
the solution with the help of the fixed-point equation of the ISTA method.

  * -

Since (4.2) deals with the optimal solution of the sparse problem (through the
iterative fixed-point algorithm introduced in [62]), we can conclude that the
optimal approach of SR using multiple LR images needs the initial fused image
to be computed by the combination of up-sampled and shift-corrected LR images.
Therefore, we suggest using the MFSL ComSR method by applying MFSR first and
SISR after.

####  4.3.2 Up-scaling factor is a divisible integer

There are several structures for performing SR using SISR, MFSR, or
combinations of them when the up-scaling factor is a divisible integer such as
4 as shown in Fig. 2. These structures are: a) Just using SISR [16] with
$r=4$. b) Just using MFSR [5] with $r=4$. c) Applying SISR on each LR image
with $r=2$ and then combining them by MFSR with $r=2$, we call it S2M2 ComSR
[27]. d) Combining the LR images using MFSR with $r=2$ and then applying SISR
with $r=2$ to improve the result and remove the artifacts, we call it M2S2
ComSR [26]. e) Applying SISR on each LR image with $r=4$ and then combining
them by MFSR with $r=1$, we call it SFML ComSR [27]. f) Combining the LR
images using MFSR with $r=4$ and then applying SISR with $r=1$ (it does not
up-sample the image but uses SRCNN network to enhance the quality of output),
called MFSL ComSR [26].

Similar to the case of indivisible up-scaling factor, based on the examination
of (4.2), it can be concluded that ${\mathbf{y}_{C}}$ should be built first,
regardless of whether $r$ is divisible or indivisible. After that, SISR should
be used to improve the final quality and to remove artifacts.

##  5 Experimental Results

For the experiments, we implement two different versions of the MFSR ComSR
method. One is the MFSR ComSR method based on shift-and-add (16) in the case
of noisy images, and the other is the two-dimensional curve fitting method (on
LR images) in the case of noise-free images. In both versions, a registration
method of [63] is used to estimate the shift between an LR image and a
randomly chosen reference LR image. In the noise-free mode, it is enough to
fuse the LR images with two-dimensional curve fitting method by estimating the
pixel values in the main HR grid from shifted versions of LR images. In noisy
mode, the shift-and-add approach is used, meaning that we register LR images
based on the estimated shifts between them, and then we re-sample and add the
resulting images. The reason for the separation of MFSR ComSR into two modes
(noisy and noise-free) is that in the noise-free mode, the degradation model
of (1) does not include AWGN and the averaging in (16) causes the edges and
high frequency regions to be damaged.

For the SISR component in all experimented methods, we use the SRCNN method of
[16], a powerful method with low computational complexity. The network used in
our simulations is exactly the same of [16], which has been trained for the
SISR task. In other words, we do not retrain or fine-tune the network of [16]
in the simulations. Each average PSNR point is calculated over 20 independent
trials. In each trial, for a specific dataset and number of LR frames, the LR
images are generated by applying random shifts to the HR image of chosen
dataset, then after down-sampling, AWGN with specific amount of noise is
added222We provide the source codes of our simulations (implementations of
evaluated methods) at https://github.com/MMAfrasiabi/ComSR.

Before conducting our comprehensive simulations and experimental results, we
first examine the effect of ideal and practical image registration on the
results of various SR methods.

###  5.1 Effect of image registration methods

At first, we examine the effect of practical and ideal image registration on
the results of SR. We use the method of [63] for practical registration. As
illustrated in Table. I, for Set5 and in two modes of noise-free and low-
noise, the average PSNR values have been calculated for various methods, up-
scaling factors, and the number of LR images (N). In the methods that require
registration, there are two numbers in each cell of the table, which
respectively correspond to practical and ideal registration. From examining
the trend of the PSNR values, it can be concluded that even if we consider the
registration as practical, we only see an almost constant decrease in the PSNR
in various up-scaling factors and the number of LR images. Therefore, to
remove the effect of registration error and to observe purely the effect of
different structures and methods on SR performance, we use ideal registration
in the following simulations.

TABLE I: Examining the effect of ideal and practical image registration on SR methods. Average PSNRs (in dB) are represented on Set5 dataset for different methods. Various up-scaling factors and number of input LR images (N) are included. In items with two values, first and second ones indicate average PSNR for practical and ideal registration, respectively. Dataset | Scale | N | Bicubic | SISR | MFSR | SFML ComSR | Curve-Fit | MFSL ComSR | MFSL ComSR
---|---|---|---|---|---|---|---|---|---
SRCNN [16] | Farsiu [5] | Kawulok [27] |  | Optimal | Optimal
|  |  |  | (Curve-Fit) | (Shift-and-Add)
Set 5 | $\times 2$ | 2 | 33.67 | 36.69 | 33.41/33.42 | 36.13/36.13 | 33.45/33.50 | 36.28/36.31 | 36.23/36.24
3 | 33.44/33.45 | 36.18/36.18 | 33.49/33.51 | 36.39/36.40 | 36.30/36.34
$\times 3$ | 3 | 30.39 | 32.75 | 30.07/30.07 | 32.44/32.47 | 30.11/30.12 | 32.58/32.59 | 32.51/32.52
(Noise-Free) | 5 | 30.09/30.10 | 32.52/32.52 | 30.14/30.14 | 32.66/32.66 | 32.56/32.57
| 7 | 30.10/30.10 | 32.53/32.58 | 30.16/30.16 | 32.70/32.71 | 32.58/32.58
Set 5 | $\times 2$ | 2 | 32.11 | 32.97 | 32.60/32.62 | 33.63/33.88 | 32.39/32.42 | 33.55/33.64 | 34.09/34.17
3 | 32.91/32.94 | 34.64/34.63 | 32.52/32.57 | 33.86/33.95 | 34.83/34.92
$\times 3$ | 3 | 29.71 | 31.45 | 29.93/29.94 | 32.08/32.11 | 29.82/29.85 | 31.82/31.84 | 31.98/32.02
($\sigma_{n}=0.001$) | 5 | 29.93/30.00 | 32.22/32.29 | 29.83/29.89 | 31.95/31.97 | 32.22/32.29
| 7 | 29.96/30.01 | 32.36/32.38 | 29.88/29.91 | 31.97/32.01 | 32.25/32.38

###  5.2 Up-scaling factor is an indivisible integer

|
---|---
|
|

(a) Set5, Noise-Free (b) Set14, Noise-Free

(c) Set5, $\sigma_{n}=0.001$ (d) Set14, $\sigma_{n}=0.001$

(e) Set5, $\sigma_{n}=0.005$ (f) Set14, $\sigma_{n}=0.005$

Figure 3: Average PSNR (in dB) vs number of LR frames for up-scaling factor
$\times 2$. Simulation results on Set5 dataset for noise-free,
$\sigma_{n}=0.001$, and $\sigma_{n}=0.005$ are illustrated in (a), (c), and
(e), respectively. Simulation results on Set14 dataset for noise-free,
$\sigma_{n}=0.001$, and $\sigma_{n}=0.005$ are illustrated in (b), (d), and
(f), respectively.

|
---|---
|
|

(a) Set5, Noise-Free (b) Set14, Noise-Free

(c) Set5, $\sigma_{n}=0.001$ (d) Set14, $\sigma_{n}=0.001$

(e) Set5, $\sigma_{n}=0.005$ (f) Set14, $\sigma_{n}=0.005$

Figure 4: Average PSNR (in dB) vs number of LR frames for up-scaling factor
$\times 3$. Simulation results on Set5 dataset for noise-free,
$\sigma_{n}=0.001$, and $\sigma_{n}=0.005$ are illustrated in (a), (c), and
(e), respectively. Simulation results on Set14 dataset for noise-free,
$\sigma_{n}=0.001$, and $\sigma_{n}=0.005$ are illustrated in (b), (d), and
(f), respectively.

We evaluate the performance of the optimal method (MFSL ComSR) in comparison
with other existing ones, for up-scaling factors 2 and 3 in Fig. 3 and Fig. 4,
respectively. Each column in Fig. 3 or Fig. 4 corresponds to a dataset, and
each row corresponds to a specific amount of noise in the LR images. The
horizontal axis of each sub-graph shows the number of LR images, while the
vertical axis demonstrates the average PSNR. Following conclusions can be made
based on Fig. 3 and Fig. 4. 1) The more the number of LR images, the better
the performance, but it has a saturation trend. 2) The general trend of the
graphs is independent of the dataset. 3) The higher the noise of the LR
images, the lower the final PSNR. 4) The superiority of the MFSL ComSR method
over the others becomes more apparent when the noise increases. 5) Each mode
of the MFSL ComSR methods works better in either noisy or noise-free case. 6)
There is a relatively large gap between PSNRs of the methods that use SISR in
the last stage of their methods and those that do not. This observation
justifies the use of SISR after MFSR in MFSL ComSR.

|
---|---
|

(a) Set5, $\sigma_{n}=0.001$ (b) Set14, $\sigma_{n}=0.001$

(c) Set5, $\sigma_{n}=0.005$ (d) Set14, $\sigma_{n}=0.005$

Figure 5: Average PSNR (in dB) vs number of LR frames for up-scaling factor
$\times 4$. Simulation results are on Set5 and Set14 datasets for two
different amount of noises $\sigma_{n}=0.001$ and $\sigma_{n}=0.005$.

###  5.3 Up-scaling factor is a divisible integer

To examine the MFSL ComSR method thoroughly, we conduct some simulations when
$r$ is divisible. Fig. 5 depicts the evaluation results of the MFSL ComSR
method on different datasets with different amounts of noise. As it can be
seen from Fig. 5, MFSL ComSR has better performance than the other methods.
SFML ComSR method is very close to MFSL ComSR in terms of PSNR, but it
initially applies SISR to each LR image, so it has higher computational
complexity. In the case of the up-scaling factor of 4, the slope of the PSNR
curve significantly decreases for the large number of LR images, which is not
the case in up-scaling factors 2 or 3.

###  5.4 Simulations on real-world data

In this part, we test the MFSL ComSR method on the two real-world (non-
synthetic) data. In Fig. 6, the noisy Alpaca dataset of [5] (with 55 LR
images) is used for the up-scaling factor of 2. Up-scaling factor 4 is
investigated on our almost noise-free dataset (10 LR images captured by burst
shot tool of Xiaomi POCO X4 GT smartphone) in Fig. 7. Qualitative comparison
on these real data also validates the high performance of the MFSL ComSR
method. It can also be observed that shift-and-add version of the MFSL ComSR
method works better for the noisy Alpaca dataset, while curve-fit version of
the MFSL ComSR works better for our almost noise-free one.

|  |
---|---|---
|  |

(a) SISR (SRCNN)

(b) MFSR (Curve-Fit)

(c) MFSL ComSR (Curve-Fit version)

(d) MFSR (Shift-and-Add)

(e) SFML ComSR

(f) MFSL ComSR (Shift-and-Add version)

Figure 6: Applying different SR methods on a real-world dataset (Alpaca [5])
for up-scaling factor $\times 2$.

|  |
---|---|---
|  |
|  |

(a) Bicubic

(b) SISR

(c) MFSR (Curve-Fit)

(d) MFSR (Shift-and-Add)

(e) S2M2 ComSR

(f) M2S2 ComSR

(g) SFML ComSR

(h) MFSL ComSR (Shift-and-Add version)

(i) MFSL ComSR (Curve-Fit version)

Figure 7: Applying different SR methods on a real-world dataset (captured by
burst shot) for up-scaling factor $\times 4$.

##  6 Conclusion

In this paper, we presented how to optimally combine SISR and MFSR based on an
analytical perspective. Our mathematical analysis and simulation results
showed that the best performance is obtained by MFSL ComSR, the method that
combines LR images first (MFSR) and applies SISR to the resulting image. This
approach also has significantly lower computational complexity than the
methods that first apply SISR to all the input LR images. We proposed two
modes for MFSL ComSR, one for noisy images and the other for noise-free cases.

## References

  * [1] R. Tsai, “Multiframe image restoration and registration,” _Advance Computer Visual and Image Processing_ , vol. 1, pp. 317–339, 1984.
  * [2] K. Nasrollahi and T. B. Moeslund, “Super-resolution: a comprehensive survey,” _Machine Vision and Applications_ , vol. 25, no. 6, pp. 1423–1468, 2014.
  * [3] M. M. Khattab, A. M. Zeki, A. A. Alwan, A. S. Badawy, and L. S. Thota, “Multi-frame super-resolution: A survey,” in _IEEE International Conference on Computational Intelligence and Computing Research_. IEEE, 2018.
  * [4] B. Wronski, I. Garcia-Dorado, M. Ernst, D. Kelly, M. Krainin, C.-K. Liang, M. Levoy, and P. Milanfar, “Handheld multi-frame super-resolution,” _ACM Transactions on Graphics_ , vol. 38, no. 4, pp. 1–18, 2019.
  * [5] S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” _IEEE Transactions on Image Processing_ , vol. 13, no. 10, pp. 1327–1344, 2004.
  * [6] J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution as sparse representation of raw image patches,” in _IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2008\.
  * [7] H. Hou and H. Andrews, “Cubic splines for image interpolation and digital filtering,” _IEEE Transactions on Acoustics, Speech, and Signal Processing_ , vol. 26, no. 6, pp. 508–517, 1978.
  * [8] X. Li and M. T. Orchard, “New edge-directed interpolation,” _IEEE Transactions on Image Processing_ , vol. 10, no. 10, pp. 1521–1527, 2001.
  * [9] L. Zhang and X. Wu, “An edge-guided image interpolation algorithm via directional filtering and data fusion,” _IEEE Transactions on Image Processing_ , vol. 15, no. 8, pp. 2226–2238, 2006.
  * [10] D. Glasner, S. Bagon, and M. Irani, “Super-resolution from a single image,” in _IEEE International Conference on Computer Vision_. IEEE, 2009, pp. 349–356.
  * [11] G. Freedman and R. Fattal, “Image and video upscaling from local self-examples,” _ACM Transactions on Graphics_ , vol. 30, no. 2, pp. 1–11, 2011.
  * [12] W. T. Freeman, T. R. Jones, and E. C. Pasztor, “Example-based super-resolution,” _IEEE Computer Graphics and Applications_ , vol. 22, no. 2, pp. 56–65, 2002.
  * [13] H. Chang, D.-Y. Yeung, and Y. Xiong, “Super-resolution through neighbor embedding,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , vol. 1. IEEE, 2004.
  * [14] R. Timofte, V. De Smet, and L. Van Gool, “Anchored neighborhood regression for fast example-based super-resolution,” in _IEEE International Conference on Computer Vision_ , 2013, pp. 1920–1927.
  * [15] J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” _IEEE Transactions on Image Processing_ , vol. 19, no. 11, pp. 2861–2873, 2010.
  * [16] C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in _European Conference on Computer Vision_. Springer, 2014, pp. 184–199.
  * [17] ——, “Image super-resolution using deep convolutional networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 2, pp. 295–307, 2015.
  * [18] J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 1646–1654.
  * [19] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2472–2481.
  * [20] Z. Wang, J. Chen, and S. C. Hoi, “Deep learning for image super-resolution: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 10, pp. 3365–3387, 2020.
  * [21] M. Elad and Y. Hel-Or, “A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur,” _IEEE Transactions on Image Processing_ , vol. 10, no. 8, pp. 1187–1193, 2001.
  * [22] N. Nguyen, P. Milanfar, and G. Golub, “A computationally efficient image superresolution algorithm ieee trans,” _Image Processing_ , vol. 10, no. 4, pp. 573–583, 2001.
  * [23] ——, “Efficient generalized cross-validation with applications to parametric image restoration and resolution enhancement,” _IEEE Transactions on Image Processing_ , vol. 10, no. 9, pp. 1299–1308, 2001.
  * [24] T. Kato, H. Hino, and N. Murata, “Multi-frame image super resolution based on sparse coding,” _Neural Networks_ , vol. 66, pp. 64–78, 2015.
  * [25] ——, “Doubly sparse structure in image super resolution,” in _IEEE International Workshop on Machine Learning for Signal Processing_. IEEE, 2016, pp. 1–6.
  * [26] J. Wu, T. Yue, Q. Shen, X. Cao, and Z. Ma, “Multiple-image super resolution using both reconstruction optimization and deep neural network,” in _IEEE Global Conference on Signal and Information Processing_. IEEE, 2017, pp. 1175–1179.
  * [27] M. Kawulok, P. Benecki, S. Piechaczek, K. Hrynczenko, D. Kostrzewa, and J. Nalepa, “Deep learning for multiple-image super-resolution,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 17, no. 6, pp. 1062–1066, 2019.
  * [28] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Communications of the ACM_ , vol. 60, no. 6, pp. 84–90, 2017.
  * [29] K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep convolutional networks for visual recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 37, no. 9, pp. 1904–1916, 2015\.
  * [30] N. Zhang, J. Donahue, R. Girshick, and T. Darrell, “Part-based R-CNNs for fine-grained category detection,” in _European Conference on Computer Vision_. Springer, 2014, pp. 834–849.
  * [31] Z. Wang, D. Liu, J. Yang, W. Han, and T. Huang, “Deep networks for image super-resolution with sparse prior,” in _IEEE International Conference on Computer Vision_ , 2015, pp. 370–378.
  * [32] C. Ma, Y. Rao, J. Lu, and J. Zhou, “Structure-preserving image super-resolution,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 11, pp. 7898–7911, 2022.
  * [33] S. Gao and X. Zhuang, “Bayesian image super-resolution with deep modeling of image statistics,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 45, no. 2, pp. 1405–1423, 2023.
  * [34] C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–14, 2022.
  * [35] Y. Huang, J. Li, Y. Hu, X. Gao, and H. Huang, “Transitional learning: Exploring the transition states of degradation for blind super-resolution,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–16, 2022.
  * [36] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _European Conference on Computer Vision_. Springer, 2016, pp. 694–711.
  * [37] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang _et al._ , “Photo-realistic single image super-resolution using a generative adversarial network,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 4681–4690.
  * [38] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “ESRGAN: Enhanced super-resolution generative adversarial networks,” in _European Conference on Computer Vision Workshops_. Springer, 2019.
  * [39] A. Lugmayr, M. Danelljan, L. V. Gool, and R. Timofte, “SRFLOW: Learning the super-resolution space with normalizing flow,” in _European Conference on Computer Vision_. Springer, 2020, pp. 715–732.
  * [40] L. Wang, T.-K. Kim, and K.-J. Yoon, “Joint framework for single image reconstruction and super-resolution with an event camera,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 11, pp. 7657–7673, 2022.
  * [41] X. Xu, Y. Ma, W. Sun, and M.-H. Yang, “Exploiting raw images for real-scene super-resolution,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, pp. 1905–1921, 2022.
  * [42] S. Son, J. Kim, W.-S. Lai, M.-H. Yang, and K. M. Lee, “Toward real-world super-resolution via adaptive downsampling models,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 11, pp. 8657–8670, 2022.
  * [43] S. Rhee and M. G. Kang, “Discrete cosine transform based regularized high-resolution image reconstruction algorithm,” _Optical Engineering_ , vol. 38, no. 8, pp. 1348–1356, 1999.
  * [44] N. Nguyen and P. Milanfar, “A wavelet-based interpolation-restoration method for superresolution (wavelet superresolution),” _Circuits, Systems and Signal Processing_ , vol. 19, no. 4, pp. 321–338, 2000.
  * [45] P. Vandewalle, S. Süsstrunk, and M. Vetterli, “A frequency domain approach to registration of aliased images with application to super-resolution,” _EURASIP Journal on Advances in Signal Processing_ , 2006, 071459.
  * [46] H. Ji and C. Fermüller, “Robust wavelet-based super-resolution reconstruction: theory and algorithm,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 31, no. 4, pp. 649–660, 2008.
  * [47] M. Irani and S. Peleg, “Super resolution from image sequences,” in _10th International Conference on Pattern Recognition_ , vol. 2. IEEE, 1990, pp. 115–120.
  * [48] B. Bascle, A. Blake, and A. Zisserman, “Motion deblurring and super-resolution from an image sequence,” in _European Conference on Computer Vision_. Springer, 1996, pp. 571–582.
  * [49] M. Shen and P. Xue, “Low-power video acquisition with super-resolution reconstruction for mobile devices,” _IEEE Transactions on Consumer Electronics_ , vol. 56, no. 4, pp. 2520–2528, 2010.
  * [50] V. Patel, C. K. Modi, C. N. Paunwala, and S. Patnaik, “Hybrid approach for single image super resolution using ISEF and IBP,” in _International Conference on Communication Systems and Network Technologies_. IEEE, 2011, pp. 495–499.
  * [51] M. K. Ng and A. M. Yip, “A fast map algorithm for high-resolution image reconstruction with multisensors,” _Multidimensional Systems and Signal Processing_ , vol. 12, no. 2, pp. 143–164, 2001.
  * [52] M. K. Ng, H. Shen, E. Y. Lam, and L. Zhang, “A total variation regularization based super-resolution reconstruction algorithm for digital video,” _EURASIP Journal on Advances in Signal Processing_ , 2007.
  * [53] C. Chen, H. Liang, S. Zhao, Z. Lyu, S. Fang, and X. Pei, “Integrating the missing information estimation into multi-frame super-resolution,” _Circuits, Systems, and Signal Processing_ , vol. 35, no. 4, pp. 1213–1238, 2016.
  * [54] G. Bhat, M. Danelljan, L. Van Gool, and R. Timofte, “Deep burst super-resolution,” in _IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2021, pp. 9209–9218.
  * [55] F. Cao and M. Su, “Research on face recognition algorithm based on CNN and image super-resolution reconstruction,” in _IEEE International Conference on Big Data Security on Cloud, IEEE International Conference on High Performance and Smart Computing, and IEEE International Conference on Intelligent Data and Security_. IEEE, 2022, pp. 157–161.
  * [56] M. Deudon, A. Kalaitzis, I. Goytom, M. R. Arefin, Z. Lin, K. Sankaran, V. Michalski, S. E. Kahou, J. Cornebise, and Y. Bengio, “Highres-net: Recursive fusion for multi-frame super-resolution of satellite imagery,” _arXiv preprint arXiv:2002.06460_ , 2020.
  * [57] A. B. Molini, D. Valsesia, G. Fracastoro, and E. Magli, “Deepsum: Deep neural network for super-resolution of unregistered multitemporal images,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 58, no. 5, pp. 3644–3656, 2020.
  * [58] F. Dorr, “Satellite image multi-frame super resolution using 3d wide-activation neural networks,” _Remote Sensing_ , vol. 12, no. 22, 2020\.
  * [59] F. Salvetti, V. Mazzia, A. Khaliq, and M. Chiaberge, “Multi-image super resolution of remotely sensed images using residual attention deep neural networks,” _Remote Sensing_ , vol. 12, no. 14, 2020.
  * [60] T. An, X. Zhang, C. Huo, B. Xue, L. Wang, and C. Pan, “Tr-misr: Multiimage super-resolution based on feature fusion with transformers,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 15, pp. 1373–1388, 2022.
  * [61] M. M. A. Gonbadani and A. Abbasfar, “Combined single and multi-frame image super-resolution,” in _Iranian Conference on Electrical Engineering_. IEEE, 2020, pp. 1–6.
  * [62] I. Daubechies, M. Defrise, and C. De Mol, “An iterative thresholding algorithm for linear inverse problems with a sparsity constraint,” _Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences_ , vol. 57, no. 11, pp. 1413–1457, 2004\.
  * [63] J. L. Barron, D. J. Fleet, S. S. Beauchemin, and T. Burkitt, “Performance of optical flow techniques,” in _IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 1992, pp. 236–237.

|  Mohammad Mahdi Afrasiabi Mohammad Mahdi Afrasiabi received the B.Sc. and
M.Sc. degrees in electrical engineering (telecommunication) from University of
Tehran, Iran in 2013 and 2015, respectively. He is working currently as a PhD
candidate at University of Tehran in the field of telecommunication
engineering. His main research interests include analytical approaches for
solving signal and image processing problems, sparse coding, image super-
resolution, image enhancement, and applications of machine learning in
wireless communications.
---|---
|  Reshad Hosseini Reshad Hosseini received the B.Sc. degree in electrical
engineering (telecommunication) from the University of Tehran, Tehran, Iran,
in 2004, and the Ph.D. degree from the Faculty of Electrical Engineering and
Computer Science, Technical University of Berlin, Berlin, Germany, in 2012. He
is currently an Assistant Professor with the School of Electrical and Computer
Engineering, College of Engineering, University of Tehran. His professional
interest topics are machine learning, signal processing, and computational
vision. He is particularly interested in the mathematical foundation of these
fields, such as differential geometry, optimization, functional analysis, and
statistics. His current research interests include manifold optimization,
large-scale mixture models, 3-D reconstruction, neural system identification,
visual recognition using deep learning, and accelerating reinforcement
learning.
---|---
|  Aliazam Abbasfar Aliazam Abbasfar (Senior Member, IEEE) received the B.Sc.
(Highest Hons.) and M.Sc. degrees in electrical engineering from the
University of Tehran, Tehran, Iran, in 1992 and 1995, respectively, and the
Ph.D. degree in electrical engineering from the University of California at
Los Angeles (UCLA), Los Angeles, CA, USA, in 2005.,From 2001 to 2004, he held
positions as a Senior Design Engineer in the areas of communication system
design and digital design with various startup companies in California. Upon
graduation from UCLA, he joined Rambus Inc., Sunnyvale, CA, USA, where he was
a Principal Engineer working on high-speed data communications on wireline
serial and parallel links. He is currently an Associate Professor with the
University of Tehran. His main research interests include wireless and
wireline communications, error correcting codes, and VLSI for digital data
communications.
---|---