# Smooth covers of moduli stacks of Riemann surfaces with symmetry

Fabio Perroni  Dipartimento di Matematica e Geoscienze, Università degli Studi
di Trieste, via Valerio 12/1, 34127 Trieste, Italy<EMAIL_ADDRESS>
###### Abstract.

We construct explicitly a finite cover of the moduli stack of compact Riemann
surfaces with a given group of symmetries by a smooth quasi-projective
variety.

Dedicated to Fabrizio Catanese on the occasion of his 70th birthday

###### Contents

  1. 1 Introduction
  2. 2 Moduli spaces of curves with symmetry
    1. 2.1 On the homology of $M_{g}(G)$
  3. 3 Level structures and Teichmüller structures
    1. 3.1 Level structures and Teichmüller structures
  4. 4 Smooth covers of $\mathcal{M}_{g}(G,\tau)$

##  1\. Introduction

For a finite group $G$, the locus $M_{g}(G)$ (in the moduli space $M_{g}$) of
curves that have an effective action by $G$ plays an important role in the
study of the geometry of $M_{g}$ (for example its singularities [Cor87],
[Cat12]), in the study of Shimura varieties (see e.g. [FPP16], [MZ18],
[CFGP19] and the references therein), of totally geodesic subvarieties of
$M_{g}$ [EMMW20], and also in the classification of higher dimensional
varieties (see e.g. [Cat15], [FG16], [Cat17], [LLR20]). Similar loci in the
moduli space of higher dimensional varieties have been studied in [Li18].

To investigate the geometry of $M_{g}(G)$ it is more natural to introduce the
moduli stack $\mathcal{M}_{g}(G)$ of genus $g$ compact Riemann surfaces with
an effective action by $G$. Then we can obtain $M_{g}(G)$ as the image of a
finite morphism $\mathcal{M}_{g}(G)\to M_{g}$. In this paper we study some
geometric properties of $\mathcal{M}_{g}(G)$. If $g\geq 2$,
$\mathcal{M}_{g}(G)$ is a complex orbifold, whose connected components are in
bijection with the set $\mathbb{T}$ of topological types of the $G$-actions.
For any $\tau\in\mathbb{T}$, let $\mathcal{M}_{g}(G,\tau)$ be the
corresponding connected component. It is known that $\mathcal{M}_{g}(G,\tau)$
is isomorphic to the stack quotient $\left[\mathcal{T}_{g}^{\tau(G)}/{\rm
C}_{\Gamma_{g}}(\tau(G))\right]$, where $\mathcal{T}_{g}^{\tau(G)}$ is the
locus of points, in the Teichmüller space $\mathcal{T}_{g}$, that are fixed by
$\tau(G)$, and ${\rm C}_{\Gamma_{g}}(\tau(G))$ is the centralizer of $\tau(G)$
in the mapping class group $\Gamma_{g}$. Using the theory of level structures
we define a smooth quasi-projective variety $Z_{\tau}$ such that
$\mathcal{M}_{g}(G,\tau)\cong\left[Z_{\tau}/\bar{\rm C}_{\tau}\right]$, where
$\bar{\rm C}_{\tau}$ is a finite group. Hence $\mathcal{M}_{g}(G,\tau)$ is a
smooth Deligne-Mumford stack and $Z_{\tau}\to\mathcal{M}_{g}(G,\tau)$ is a
finite Galois cover. The disjoint union of the $Z_{\tau}$’s is a finite smooth
cover of $\mathcal{M}_{g}(G)$, since $\mathbb{T}$ is finite (see e.g.
[CLP16]). Notice that the existence of a smooth quasi-projective variety $Z$
and a finite flat morphism $Z\to\mathcal{M}_{g}(G)$ follows also from [KV04].

#### Acknowledgements

I would like to thank Fabrizio Catanese for introducing me to the subject of
moduli spaces of curves with symmetries, and for teaching me a lot about this.
I am grateful to Paola Frediani and Alessandro Ghigi for discussions that
motivated the present article.

The research was partially supported by the national projects PRIN 2015EYPTSB-
PE1 “Geometria delle varietà algebriche” and 2017SSNZAW 005-PE1 “Moduli Theory
and Birational Classification”, by the research group GNSAGA of INDAM and by
FRA of the University of Trieste.

##  2\. Moduli spaces of curves with symmetry

Throughout the article $G$ is a finite group and $g$ is an integer greater or
equal than $2$. Let $\mathcal{M}_{g}(G)$ be the stack, in the complex analytic
category, whose objects are pairs $(\pi\colon\mathcal{C}\to B,\alpha)$, where
$\pi\colon\mathcal{C}\to B$ is a family of compact Riemann surfaces of genus
$g$ and $\alpha\colon G\times\mathcal{C}\to\mathcal{C}$ is an effective
(holomorphic) action of $G$ on $\mathcal{C}$ such that, for any $a\in G$,
$\pi\circ\alpha(a,\\_)=\pi$. A morphism
$(\Phi,\varphi)\colon(\pi\colon\mathcal{C}\to
B,\alpha)\to(\pi^{\prime}\colon\mathcal{C}^{\prime}\to
B^{\prime},\alpha^{\prime})$ is a Cartesian diagram

${\mathcal{C}}$${\mathcal{C}^{\prime}}$${B}$${B^{\prime}}$$\scriptstyle{\Phi}$$\scriptstyle{\pi}$$\scriptstyle{\pi^{\prime}}$$\scriptstyle{\varphi}$

such that, for any $a\in G$,
$\Phi\circ\alpha(a,\\_)\circ\Phi^{-1}=\alpha^{\prime}(a,\\_)$.

Using the Teichmüller space $\mathcal{T}_{g}$, we are going to define a
complex orbifold structure on $\mathcal{M}_{g}(G)$. Given a compact,
connected, oriented topological surface of genus $g$, $\Sigma_{g}$, recall
that a Teichmüller structure on a Riemann surface $C$ is the isotopy class of
an orientation preserving homeomorphism $f\colon C\to\Sigma_{g}$, it will be
denoted with $[f]$. Two Riemann surfaces with Teichmüller structures
$(C,[f]),(C^{\prime},[f^{\prime}])$ are isomorphic, if there exists an
isomorphism $F\colon C\to C^{\prime}$ such that $[f]=[f^{\prime}\circ F]$. We
will denote with $[C,[f]]$ the class of $(C,[f])$. Then, $\mathcal{T}_{g}$ is
the set of isomorphism classes $[C,[f]]$ of compact Riemann surfaces of genus
$g$ with Teichmüller structures. The mapping class group of $\Sigma_{g}$,
denoted by $\Gamma_{g}$, is the group of all isotopy classes of orientation
preserving homeomorphisms of $\Sigma_{g}$. There is a natural action of
$\Gamma_{g}$ on $\mathcal{T}_{g}$, given by

$[\gamma]\cdot[C,[f]]=[C,[\gamma\circ
f]]\,,\quad\forall[\gamma]\in\Gamma_{g},\,[C,[f]]\in\mathcal{T}_{g}\,.$

Furthermore, for any $[C,[f]]\in\mathcal{T}_{g}$, the homomorphism

$\sigma_{[f]}\colon{\rm
Aut}(C)\to\Gamma_{g}\,,\quad\Phi\mapsto[f\circ\Phi\circ f^{-1}]$

is injective and its image is the stabilizer of $[C,[f]]$ in $\Gamma_{g}$,
that we denote by ${\rm Stab}_{\Gamma_{g}}([C,[f]])$. We collect in the
following theorem several results about the Teichmüller space, for a proof and
for more details we refer to [AC09], [ACG11].

######  Theorem 2.1.

$\mathcal{T}_{g}$ has a natural structure of a complex manifold which is
homeomorphic to the unit ball in $\mathbb{C}^{3g-3}$. The action of
$\Gamma_{g}$ on $\mathcal{T}_{g}$ is holomorphic and properly discontinuous.
The map

$\mathcal{T}_{g}\to M_{g}\,,\quad[C,[f]]\mapsto[C]\,,$

to the coarse moduli space of compact Riemann surfaces of genus $g$ yields an
isomorphism $\mathcal{T}_{g}/\Gamma_{g}\cong M_{g}$.

Furthermore there is a universal family of Riemann surfaces of genus $g$ with
Teichmüller structure

$\eta\colon\mathcal{X}_{g}\to\mathcal{T}_{g}\,.$

Let $\alpha$ be an effective action of $G$ on $C$, viewed as an injective
group homomorphism $\alpha\colon G\to{\rm Aut}(C)$. Let us choose an
orientation preserving homeomorphism $f\colon C\to\Sigma_{g}$. Then we have an
injective homomorphism

(1) $\rho:=\sigma_{[f]}\circ\alpha\colon G\to\Gamma_{g}\,$

such that $\rho(G)\subset{\rm Stab}_{\Gamma_{g}}([C,[f]])$. Notice that, if
$f^{\prime}\colon C\to\Sigma_{g}$ is another orientation preserving
homeomorphism, we obtain a different homomorphism $\rho^{\prime}\colon
G\to\Gamma_{g}$. However $[C,[f]]$ and $[C,[f^{\prime}]]$ belong to the same
$\Gamma_{g}$-orbit, so there exists $[\gamma]\in\Gamma_{g}$ such that
$[\gamma]\cdot[C,[f]]=[C,[f^{\prime}]]$ and
$\rho^{\prime}=[\gamma]\cdot\rho\cdot[\gamma]^{-1}$. This motivates the
following definitions (that are already present in the literature, in slightly
different forms, see e.g. [CLP16] and the references therein).

######  Definition 2.2.

Let $\alpha\colon G\to{\rm Aut}(C)$ be an injective homomorphism. Let $\rho$
be defined in (1). The topological type of the $G$-action $\alpha$ is the
class of $\rho$ in ${\rm Hom}^{\rm inj}(G,\Gamma_{g})/\Gamma_{g}$, where ${\rm
Hom}^{\rm inj}(G,\Gamma_{g})$ is the set of injective group homomorphisms from
$G$ to $\Gamma_{g}$, ${\rm Hom}^{\rm inj}(G,\Gamma_{g})/\Gamma_{g}$ is the
quotient under the action of $\Gamma_{g}$ by conjugation.

######  Definition 2.3.

The Teichmüller space of compact Riemann surfaces of genus $g$ with
$G$-actions is the set

$\mathcal{T}_{g}(G)=\\{([C,[f]],\rho)\in\mathcal{T}_{g}\times{\rm Hom}^{\rm
inj}(G,\Gamma_{g})\,|\,\rho(G)\subseteq{\rm Stab}_{\Gamma_{g}}([C,[f]])\\}\,.$

Notice that, if ${\rm Hom}^{\rm inj}(G,\Gamma_{g})\not=\emptyset$, then
$\mathcal{T}_{g}(G)\not=\emptyset$ by Nielsen realization problem [Ker83] and
that it carries the following action by $\Gamma_{g}$:

(2)
$[\gamma]\cdot([C,[f]],\rho)=([\gamma]\cdot[C,[f]],[\gamma]\cdot\rho\cdot[\gamma]^{-1})\,,$

for $[\gamma]\in\Gamma_{g}$ and $([C,[f]],\rho)\in\mathcal{T}_{g}(G)$.

######  Proposition 2.4.

$\mathcal{T}_{g}(G)$ is a complex manifold. Moreover there is an object of
$\mathcal{M}_{g}(G)$,
$(\eta(G)\colon\mathcal{X}_{g}(G)\to\mathcal{T}_{g}(G),\alpha)$, such that the
associated classifying morphism $\mathcal{T}_{g}(G)\to\mathcal{M}_{g}(G)$
induces an isomorphism

$\left[\mathcal{T}_{g}(G)/\Gamma_{g}\right]\cong\mathcal{M}_{g}(G)\,,$

where $\left[\mathcal{T}_{g}(G)/\Gamma_{g}\right]$ is the stack quotient
associated to the action (2). In particular $\mathcal{M}_{g}(G)$ has a
structure of complex orbifold in the sense of [ALR07], [ACG11, XII, §4.].

###### Proof.

It follows directly from Definition 2.3 that

$\mathcal{T}_{g}(G)=\bigsqcup_{\rho\in{\rm Hom}^{\rm
inj}(G,\Gamma_{g})}\mathcal{T}_{g}^{\rho(G)}\,,$

where $\mathcal{T}_{g}^{\rho(G)}$ is the locus of points fixed by $\rho(G)$.
Since $\rho(G)$ is a finite group, $\mathcal{T}_{g}^{\rho(G)}$ is a complex
submanifold of $\mathcal{T}_{g}$, so the first claim follows.

For any $\rho\in{\rm Hom}^{\rm inj}(G,\Gamma_{g})$, let
$\eta(\rho)\colon\mathcal{X}_{g}(\rho)\to\mathcal{T}_{g}^{\rho(G)}$ be the
restriction of the universal family. There is a natural effective action,
$\alpha(\rho)$, of $G$ on $\mathcal{X}_{g}(\rho)$ such that
$(\eta(\rho)\colon\mathcal{X}_{g}(\rho)\to\mathcal{T}_{g}^{\rho(G)},\alpha(\rho))$
is an object of $\mathcal{M}_{g}(G)$. Then we define
$(\eta(G)\colon\mathcal{X}_{g}(G)\to\mathcal{T}_{g}(G),\alpha)$ as the
disjoint union of these objects.

To prove the last statement recall that the objects of
$\left[\mathcal{T}_{g}(G)/\Gamma_{g}\right]$, over a base $B$, are pairs
$(p\colon P\to B,f\colon P\to\mathcal{T}_{g}(G))$, where $p\colon P\to B$ is a
principal $\Gamma_{g}$-bundle and $f$ is a $\Gamma_{g}$-equivariant
holomorphic map. Let $f^{*}(\eta(G))$ be the pull-back of the family
$\eta(G)$. The action of $\Gamma_{g}$ on $P$ extends to a free action on
$f^{*}(\eta(G))$. So $f^{*}(\eta(G))$ descends to a family
$\pi\colon\mathcal{C}\to B$ over $B$. By construction there is a $G$-action,
$\alpha$, on $\mathcal{C}$, in such a way that $(\pi\colon\mathcal{C}\to
B,\alpha)$ is an object of $\mathcal{M}_{g}(G)$. Similarly one associates to
every arrow of $\left[\mathcal{T}_{g}(G)/\Gamma_{g}\right]$ an arrow of
$\mathcal{M}_{g}(G)$ obtaining an equivalence of categories. ∎

Furthermore we have the following result.

######  Theorem 2.5.

$\mathcal{M}_{g}(G)$ is a smooth algebraic Deligne-Mumford stack with quasi-
projective coarse moduli space.

A proof of this theorem will be given in Section 4, where we define a groupoid
presentation of $\mathcal{M}_{g}(G)$,
$X_{1}\stackrel{{\scriptstyle[}}{{t}}]{s}{\rightrightarrows}X_{0}$, with
$X_{0}$ and $X_{1}$ smooth quasi-projective algebraic varieties, and such that
$X_{0}\to\mathcal{M}_{g}(G)$ is finite étale.

###  2.1. On the homology of $M_{g}(G)$

Let $\mathbb{T}\subset{\rm Hom}^{\rm inj}(G,\Gamma_{g})$ be a set of
representatives of topological types of $G$-actions (see Definition 2.2).
Notice that, for any $\tau\in\mathbb{T}$,

${\rm Stab}_{\Gamma_{g}}(\tau)={\rm C}_{\Gamma_{g}}(\tau(G))\,,$

where ${\rm C}_{\Gamma_{g}}(\tau(G))$ is the centralizer of $\tau(G)$ in
$\Gamma_{g}$. Therefore we have an homeomorphism

$\mathcal{T}_{g}(G)/\Gamma_{g}\cong\bigsqcup_{\tau\in\mathbb{T}}\mathcal{T}_{g}^{\tau(G)}/{\rm
C}_{\Gamma_{g}}(\tau(G))\,.$

######  Definition 2.6.

Let $\tau\in\mathbb{T}$. The moduli space of compact Riemann surfaces of genus
$g$ with $G$-action of topological type $\tau$ is defined as
$\mathcal{T}_{g}^{\tau(G)}/{\rm C}_{\Gamma_{g}}(\tau(G))$ and will be denoted
by $M_{g}(G,\tau)$.

We have the following theorem from [Har71], [Cat00] (see also [GDH92] and the
references therein).

######  Theorem 2.7.

$\mathcal{T}_{g}^{\tau(G)}$ is bi-holomorphic to $\mathcal{T}_{g^{\prime},d}$
where $g^{\prime}$ is the genus of $C/G$,
$[C,[f]]\in\mathcal{T}_{g}^{\tau(G)}$, and $d$ is the number of branch points
of the quotient map $C\to C/G$. In particular $\mathcal{T}_{g}^{\tau(G)}$ is
homeomorphic to the unit ball in $\mathbb{C}^{3g^{\prime}-3+d}$.

Let us give an interpretation of the previous theorem. Given a point
$[C,[f]]\in\mathcal{T}_{g}^{\tau(G)}$, let
$\alpha\colon\sigma_{[f]}^{-1}\circ\tau\colon G\to{\rm Aut}(C)$. Then
$f\colon(C,\alpha)\to(\Sigma_{g},f\circ\alpha\circ f^{-1})$ is
$G$-equivariant, so it yields an orientation-preserving homeomorphism of
$\theta\colon C^{\prime}/G\to\Sigma_{g}^{\prime}/G$, where
$C^{\prime}\subseteq C$ and $\Sigma_{g}^{\prime}\subseteq\Sigma_{g}$ are the
loci of points with trivial stabilizer. The bi-holomorphism in Theorem 2.7
sends $[C,[f]]$ to
$\left[\Sigma_{g}^{\prime}/G,[\theta]\right]\in\mathcal{T}_{g^{\prime},d}$.

We describe now the map $\mathcal{T}_{g^{\prime},d}\to M_{g}(G,\tau)$, which
is the composition of the inverse of the previous one,
$\mathcal{T}_{g^{\prime},d}\to\mathcal{T}_{g}^{\tau(G)}$, with the quotient
$\mathcal{T}_{g}^{\tau(G)}\to\mathcal{T}_{g}^{\tau(G)}/{\rm
C}_{\Gamma_{g}}(\tau(G))$. Let $p\colon\Sigma_{g}\to\Sigma_{g}/G$ be the
quotient map, let $y\in\Sigma_{g}^{\prime}/G$, and let $x\in p^{-1}(y)$. Let
$\mu\colon\pi_{1}\left(\Sigma_{g}^{\prime}/G,y\right)\to G$ be the monodromy
of the covering $p$ restricted to $\Sigma_{g}^{\prime}$. For any
$[D^{\prime},[\theta]]\in\mathcal{T}_{g^{\prime},d}$ the composition

$\mu\circ\theta_{*}\colon\pi_{1}(D^{\prime},\theta^{-1}(y))\to G\,$

gives a $G$-cover $C^{\prime}\to D^{\prime}$, hence a point $[C,\alpha]\in
M_{g}(G,\tau)$. Then the map $\mathcal{T}_{g^{\prime},d}\to M_{g}(G,\tau)$
sends $[D^{\prime},[\theta]]$ to $[C,\alpha]$.

Notice that the class of $\mu$,

$[\mu]\in{\rm Hom}(\pi_{1}\left(\Sigma_{g}^{\prime}/G,y\right),G)/G\,,$

where $G$ acts by conjugation, does not depend from the choice of $y$, and
that $\Gamma_{g^{\prime},d}$ acts on ${\rm
Hom}(\pi_{1}\left(\Sigma_{g}^{\prime}/G,y\right),G)/G$.

It follows from this that

$M_{g}(G,\tau)\cong\mathcal{T}_{g^{\prime},d}/{\rm
Stab}_{\Gamma_{g^{\prime},d}}([\mu])\,.$

In particular the homology of $M_{g}(G,\tau)$ with rational coefficients can
be computed as the homology of the group ${\rm
Stab}_{\Gamma_{g^{\prime},d}}([\mu])$:

$H_{n}(M_{g}(G,\tau);\mathbb{Q})\cong H_{n}({\rm
Stab}_{\Gamma_{g^{\prime},d}}([\mu]);\mathbb{Q})\,.$

Let now
$(\alpha_{1},\beta_{1},\ldots,\alpha_{g^{\prime}},\beta_{g^{\prime}},\gamma_{1},\ldots,\gamma_{d})$
be a geometric basis of the fundamental group
$\pi_{1}(\Sigma_{g^{\prime}}\setminus\\{y_{1},\ldots,y_{d}\\},y)$ (here we
follow the notation of [CLP16]). Setting
$a_{i}=\mu(\alpha_{i}),b_{i}=\mu(\beta_{i}),c_{j}=\mu(\gamma_{j})$, we obtain
an element

$[(a_{1},b_{1},\ldots,a_{g^{\prime}},b_{g^{\prime}},c_{1},\ldots,c_{d})]\in
G^{2g^{\prime}+d}/G\,,$

where $G$ acts diagonally by conjugation. This yields an injective
correspondence between classes $[\mu]$ of monodromies and elements of
$G^{2g^{\prime}+d}/G$. $\Gamma_{g^{\prime},d}$ acts on $G^{2g^{\prime}+d}/G$
(see e.g. [CLP16, Sec 2.]) and the homology of ${\rm
Stab}_{\Gamma_{g^{\prime},d}}([\mu])$ is isomorphic to the equivariant
homology of the orbit of
$[(a_{1},b_{1},\ldots,a_{g^{\prime}},b_{g^{\prime}},c_{1},\ldots,c_{d})]$
under the action of $\Gamma_{g^{\prime},d}$. This motivates the following

######  Question 2.8.

[CLP] Let $n,d\in\mathbb{N}$. Are there constants $a,b$ such that the
dimension of $H_{n}^{\Gamma_{g^{\prime},d}}(G^{2g^{\prime}+d}/G;\mathbb{Q})$
is independent of $g^{\prime}$, in the range $g^{\prime}>an+b$?

We proved in [CLP16] that the previous question has an affermative answer in
the case where $n=0$.

##  3\. Level structures and Teichmüller structures

In this section we recall some basic results and we fix the notation, for the
proofs and for more details we refer to [ACG11, Chapter XVI]. Given two groups
$G_{1}$ and $G_{2}$ we denote with ${\rm Hom}(G_{1},G_{2})$ the set of
homomorphisms from $G_{1}$ to $G_{2}$. Notice that there is an action of
$G_{2}$ on ${\rm Hom}(G_{1},G_{2})$, which is induced by the action of $G_{2}$
on itself by conjugation ($(g,h)\mapsto g^{-1}hg$). An exterior group
homomorphism from $G_{1}$ to $G_{2}$ is an element of the quotient set ${\rm
Hom}(G_{1},G_{2})/G_{2}$. For any $\varphi\in{\rm Hom}(G_{1},G_{2})$ we denote
with $\hat{\varphi}$ its class in ${\rm Hom}(G_{1},G_{2})/G_{2}$. We will say
that $\hat{\varphi}$ is an exterior group monomorphism (respectively
epimorphism, isomorphism) if $\varphi$ is a monomorphism (respectively
epimorphism, isomorphism).

Let $X$ be a path connected topological space. For any pair of points $x,y\in
X$ and for any continuous path $\gamma$ from $x$ to $y$, let
$\varphi_{\gamma}\colon\pi_{1}(X,x)\to\pi_{1}(X,y)$ be the isomorphism that
sends any $[c]\in\pi_{1}(X,x)$ to $[\gamma^{-1}\cdot c\cdot\gamma]$, where $c$
is a loop in $X$ based at $x$ and $[c]$ is its homotopy class.
Let $H$ be a group. We will identify two exterior homomorphisms
$\hat{\alpha}\in{\rm Hom}(\pi_{1}(X,x),H)/H$ and $\hat{\beta}\in{\rm
Hom}(\pi_{1}(X,y),H)/H$ if
$\hat{\alpha}=\widehat{\beta\circ\varphi_{\gamma}}$. Notice that this
definition does not depend on the choice of $\gamma$ and yields an equivalence
relation on the disjoint union

$\bigsqcup_{x\in X}{\rm Hom}(\pi_{1}(X,x),H)/H\,.$

The class of $\hat{\alpha}$ will be denoted with $[\alpha]$.

######  Definition 3.1.

A Teichmüller structure of level $H$ on $X$ is the class $[\alpha]$ of an
exterior epimorphism $\hat{\alpha}\in{\rm Hom}(\pi_{1}(X,x),H)/H$, for $x\in
X$.

Let now $C$ and $C^{\prime}$ be two compact Riemann surfaces. Let $[\alpha]$
and $[\alpha^{\prime}]$ be two Teichmüller structures of level $H$ on $C$ and
$C^{\prime}$, respectively. We say that the pairs $(C,[\alpha])$ and
$(C^{\prime},[\alpha^{\prime}])$ are isomorphic if there exists an isomorphism
$F\colon C\to C^{\prime}$ such that $[\alpha]=[\alpha^{\prime}\circ F_{*}]$,
where $F_{*}\colon\pi_{1}(C,x)\to\pi_{1}(C^{\prime},F(x))$ is the homomorphism
induced by $F$, $\alpha\colon\pi_{1}(C,x)\to H$ and
$\alpha^{\prime}\colon\pi_{1}(C^{\prime},F(x))\to H$ are representatives of
$[\alpha]$ and $[\alpha^{\prime}]$, respectively. The isomorphism class of
$(C,[\alpha])$ will be denoted $[C;\alpha]$. The set of isomorphism classes of
compact Riemann surfaces of genus $g$ with Teichmüller structures of level $H$
will be denoted ${}_{H}M_{g}$. When $g\geq 2$ it is possible to define a
structure of complex analytic space on ${}_{H}M_{g}$, called the moduli space
of genus $g$ curves with Teichmüller structure of level $H$. We refer to
[ACG11] for more details.

###  3.1. Level structures and Teichmüller structures

Let $[\psi]$ be a Teichmüller structure of level $H$ on $\Sigma_{g}$. The
following map defines a morphism of complex analytic spaces:

$t_{[\psi]}\colon\mathcal{T}_{g}\to{}_{H}M_{g}\,,\quad[C,[f]]\mapsto[C;\psi\circ
f_{*}]\,,$

where $f$ is a representative of $[f]$ and
$\psi\colon\pi_{1}(\Sigma_{g},f(x))\to H$ is a representative of $[\psi]$.

######  Remark 3.2.

As explained in [ACG11], the mapping class group $\Gamma_{g}$ acts on the set
of Teichmüller structures of level $H$ on $\Sigma_{g}$ as follows:
$[\psi]\cdot[\gamma]=[\psi\circ\gamma_{*}]$. Moreover every connected
component of ${}_{H}M_{g}$ coincides with $t_{[\psi]}(\mathcal{T}_{g})$, for
some $[\psi]$, we denote $t_{[\psi]}(\mathcal{T}_{g})$ by $M_{g}[\psi]$. Hence
we have the following decomposition,

${}_{H}M_{g}=\coprod_{[\psi]\,{\rm mod}\,\Gamma_{g}}M_{g}[\psi]\,.$

Let
$\Lambda_{[\psi]}:=\\{[\gamma]\in\Gamma_{g}\,|\,[\psi]\cdot[\gamma]=[\psi]\\}$.
Then

$M_{g}[\psi]=\mathcal{T}_{g}/\Lambda_{[\psi]}\,.$

Furthermore, if $\Lambda_{[\psi]}$ is a normal subgroup of $\Gamma_{g}$ and
$\Gamma_{g}[\psi]:=\Gamma_{g}/\Lambda_{[\psi]}$, then

$M_{g}=M_{g}[\psi]/\Gamma_{g}[\psi]\,.$

We report the following result from [ACG11], where a characteristic subgroup
of $\pi_{1}(\Sigma_{g},x)$ is a subgroup that is mapped to itself by every
automorphism of $\pi_{1}(\Sigma_{g},x)$.

######  Lemma 3.3.

If $\ker(\psi)$ is a characteristic subgroup, then $\Lambda_{[\psi]}$ is a
normal subgroup of $\Gamma_{g}$.

###### Proof.

Under our hypotheses any automorphism $\gamma$ of $\pi_{1}(\Sigma_{g})$
induces an automorphism $\bar{\gamma}$ of $H$ such that
$\psi\circ\gamma=\bar{\gamma}\circ\psi$. This yields a group homomorphism
${\rm Out}^{+}(\pi_{1}(\Sigma_{g}))\to{\rm Out}(H)$,
$[\gamma]\mapsto[\bar{\gamma}]$, whose kernel is $\Lambda_{[\psi]}$, under the
identification of ${\rm Out}^{+}(\pi_{1}(\Sigma_{g}))$ with $\Gamma_{g}$. ∎

In the case where $\ker(\psi)$ is a characteristic subgroup we can describe
the action of $\Gamma_{g}$ on $M_{g}[\psi]$ as follows ([ACG11]).

######  Lemma 3.4.

Let $\ker(\psi)$ be a characteristic subgroup of $\pi_{1}(\Sigma_{g})$. Then
$\Gamma_{g}$ acts on $M_{g}[\psi]$ as follows:

$[\gamma]\cdot[C;\alpha]=[C;\bar{\gamma}\circ\alpha]\,,$

where $\bar{\gamma}$ is the automorphism of $H$ defined in the proof of Lemma
3.3.

Let now $m\in\mathbb{Z}_{\geq 1}$ and let
$\chi_{m}\colon\pi_{1}(\Sigma_{g})\to
H_{1}(\Sigma_{g},\mathbb{Z}/m\mathbb{Z})$ be the composition of the natural
morphism $\pi_{1}(\Sigma_{g})\to H_{1}(\Sigma_{g},\mathbb{Z})$ with the
reduction modulo $m$. Then $H_{1}(\Sigma_{g},\mathbb{Z}/m\mathbb{Z})$ is a
strongly characteristic quotient of $\pi_{1}(\Sigma_{g})$ (i.e. there is only
one subgroup $K$ of $\pi_{1}(\Sigma_{g})$ such that $\pi_{1}(\Sigma_{g})/K$ is
isomorphic to $H_{1}(\Sigma_{g},\mathbb{Z}/m\mathbb{Z})$), in particular
$\ker(\chi_{m})$ is a characteristic subgroup of $\pi_{1}(\Sigma_{g})$. In
this situation we use the following notation:

$M_{g}[m]:=M_{g}[\chi_{m}]\,,\quad\Gamma_{g}[m]:=\Gamma_{g}[\chi_{m}]\,,\quad\Lambda_{[m]}:=\Lambda_{[\chi_{m}]}\,,\quad
t_{[m]}:=t_{[\chi_{m}]}.$

Furthermore we have that

$\Gamma_{g}[m]\cong{\rm Sp}_{2g}(\mathbb{Z}/m\mathbb{Z})$

and moreover $M_{g}[m]$ coincides with the set of isomorphism classes of pairs
$(C,\rho)$, where $\rho\colon
H_{1}(C,\mathbb{Z}/m\mathbb{Z})\to(\mathbb{Z}/m\mathbb{Z})^{2g}$ is a
symplectic isomorphism, with respect to the intersection form on
$H_{1}(C,\mathbb{Z}/m\mathbb{Z})$ and the standard symplectic form on
$(\mathbb{Z}/m\mathbb{Z})^{2g}$.

##  4\. Smooth covers of $\mathcal{M}_{g}(G,\tau)$

Let $\tau\colon G\to\Gamma_{g}$ be an injective homomorphism, we denote with
${\rm C}_{\tau}$ the centralizer of $\tau(G)$ in $\Gamma_{g}$, and with ${\rm
N}_{\tau}$ the normalizer of $\tau(G)$ in $\Gamma_{g}$. For any subgroup
$H\leq\Gamma_{g}$, its image under the quotient morphism
$\Gamma_{g}\to\Gamma_{g}[m]=\Gamma_{g}/\Lambda_{[m]}$ will be denoted by
$\bar{H}$.

######  Proposition 4.1.

Let $m\geq 3$ be an integer and
$Z_{\tau}:={\mathcal{T}_{g}^{\tau(G)}}/({\Lambda_{[m]}\cap{\rm C}_{\tau}})$.
Then $Z_{\tau}$ is a complex manifold and, for $M_{g}(G,\tau)$ defined in
Definition 2.6, the quotient morphism $\mathcal{T}_{g}^{\tau(G)}\to
M_{g}(G,\tau)$ induces a finite morphism

(3) $Z_{\tau}\to M_{g}(G,\tau)\,,$

which gives an isomorphism between $M_{g}(G,\tau)$ and $Z_{\tau}/\bar{\rm
C}_{\tau}$.

###### Proof.

The first claim follows from the fact that $\Lambda_{[m]}$ acts freely on
$\mathcal{T}_{g}$, since $m\geq 3$. To see this, let
$[\gamma]\in\Lambda_{[m]}$ and let $[C,[f]]\in\mathcal{T}_{g}$ such that
$[\gamma]\cdot[C,[f]]=[C,[f]]$. Then $[\gamma]\in{\rm Aut}(C)$. Let
$\bar{\gamma}_{*}$ be the induced automorphism of
$H_{1}(C,\mathbb{Z}/m\mathbb{Z})$. Since $[\gamma]\in\Lambda_{[m]}$,
$\bar{\gamma}_{*}={\rm Id}$, then $[\gamma]={\rm Id}_{C}$ ([ACG11, Prop.
(2.8), p. 512]).

The last claim follows from the fact that $\Lambda_{[m]}$ is a normal subgroup
of $\Gamma_{g}$ (Lemma 3.3) and ${\bar{\rm C}_{\tau}}=\frac{{\rm
C}_{\tau}}{{\rm C}_{\tau}\cap\Lambda_{[m]}}$, so

$M_{g}(G,\tau)=\mathcal{T}_{g}^{\tau(G)}/{\rm C}_{\tau}=Z_{\tau}/{\bar{\rm
C}_{\tau}}\,.$

∎

Now we prove that $Z_{\tau}$ is a quasi-projective algebraic variety. Although
this fact can be deduced from [GDH92] or [Cat12], we give here, for
completeness, an elementary proof by showing that there is a finite morphism
$Z_{\tau}\to\left(M_{g}[m]\right)^{\overline{\tau(G)}}$, where
$\left(M_{g}[m]\right)^{\overline{\tau(G)}}$ is the set of points of
$M_{g}[m]$ fixed by the action of $\overline{\tau(G)}$. By a finite morphism
of complex analytic spaces we mean a proper morphism with finite fibers.

######  Proposition 4.2.

Let $t_{[m]}\colon{\mathcal{T}}_{g}\to M_{g}[m]$ be the morphism defined in
Section 3.1, $m\geq 3$. Then the following statements hold true:

  * 1)

$t_{[m]}({\mathcal{T}}_{g}^{\tau(G)})=\left(M_{g}[m]\right)^{\overline{\tau(G)}}$;

  * 2)

let $t_{[m]|}$ be the restriction of $t_{[m]}$ to
${\mathcal{T}}_{g}^{\tau(G)}$, then the following diagram is commutative:

${{\mathcal{T}}_{g}^{\tau(G)}}$${Z_{\tau}}$${\left(M_{g}[m]\right)^{\overline{\tau(G)}}}$${Z_{\tau}/\left(\frac{\Lambda_{[m]}\cap{\rm
N}_{\tau}}{\Lambda_{[m]}\cap{\rm
C}_{\tau}}\right)}$$\scriptstyle{t_{[m]|}}$$\scriptstyle{q}$$\scriptstyle{p}$

where the horizontal arrow to the top is the quotient map by
$\Lambda_{[m]}\cap{\rm C}_{\tau}$, $q$ is the quotient map by
$\frac{\Lambda_{[m]}\cap{\rm N}_{\tau}}{\Lambda_{[m]}\cap{\rm C}_{\tau}}$ and
$p$ sends any equivalence class of the $(\Lambda_{[m]}\cap{\rm
N}_{\tau})$-action to the corresponding equivalence class of the
$\Lambda_{[m]}$-action (under the identification of
$Z_{\tau}/\left(\frac{\Lambda_{[m]}\cap{\rm N}_{\tau}}{\Lambda_{[m]}\cap{\rm
C}_{\tau}}\right)$ with
$\frac{\mathcal{T}_{g}^{\tau(G)}}{\Lambda_{[m]}\cap{\rm N}_{\tau}}$);

  * 3)

the morphism $p\circ q$ is finite, therefore $Z_{\tau}$ has a structure of
complex quasi-projective algebraic variety and $p\circ q$ is algebraic.

###### Proof.

1) Since $\Lambda_{[m]}$ is normal in $\Gamma_{g}$ (Lemma 3.3), $t_{[m]}$ is
$G$-equivariant, therefore
$t_{[m]}(\mathcal{T}_{g}^{\tau(G)})\subseteq\left(M_{g}[m]\right)^{\overline{\tau(G)}}$.
On the other hand, for any given $[C,\chi_{m}\circ
f_{*}]\in\left(M_{g}[m]\right)^{\overline{\tau(G)}}$ and $[\gamma]\in\tau(G)$,
from the equality

$[C,\chi_{m}\circ f_{*}]=\overline{[\gamma]}\cdot[C,\chi_{m}\circ f_{*}]$

it follows that $[C,[f]]$ and $[\gamma]\cdot[C,[f]]$ map to the same point of
$M_{g}[m]$. So, if $[\gamma]\not\in{\rm Stab}_{\Gamma_{g}}([C,[f]])$, it would
be an element of finite order of $\Lambda_{[m]}$, but this contradicts the
fact that $\Lambda_{[m]}$ acts freely on $\mathcal{T}_{g}$.

The statement in 2) follows from the fact that
$M_{g}[m]=t_{[m]}(\mathcal{T}_{g})=\mathcal{T}_{g}/\Lambda_{[m]}$ and from 1).

To prove 3), we show that $p\circ q$ is closed and has finite fibers. Let us
first show that $q$ has finite fibers, equivalently that
$\Lambda_{[m]}\cap{\rm C}_{\tau}$ has finite index in $\Lambda_{[m]}\cap{\rm
N}_{\tau}$. Notice that, for $\tau(G)=\\{h_{1},\ldots,h_{|\tau(G)|}\\}$, ${\rm
C}_{\tau}$ is the stabilizer of
$(h_{1},\ldots,h_{|\tau(G)|})\in\prod_{h\in\tau(G)}\tau(G)$ with respect to
the action of ${\rm N}_{\tau}$ given by conjugation on each factor. So $[{\rm
N}_{\tau}\,:\,{\rm C}_{\tau}]<\infty$, hence also $[\Lambda_{[m]}\cap{\rm
N}_{\tau}\,:\,\Lambda_{[m]}\cap{\rm C}_{\tau}]<\infty$. To see that $p$ has
finite fibers, notice that for any $[C,[\chi_{m}\circ
f_{*}]]\in\left(M_{g}[m]\right)^{\overline{\tau(G)}}$

$p^{-1}([C,[\chi_{m}\circ
f_{*}]])=\frac{\left(\Lambda_{[m]}\cdot[C,[f]]\right)\cap\mathcal{T}_{g}^{\tau(G)}}{\Lambda_{[m]}\cap{\rm
N}_{\tau}}\,,$

and, since $\Lambda_{[m]}$ acts freely on $\mathcal{T}_{g}^{\tau(G)}$ (see the
proof of Proposition 4.1), $p^{-1}([C,[\chi_{m}\circ f_{*}]])$ is in bijection
with

(4)
$\frac{\\{\lambda\in\Lambda_{[m]}\,|\,\lambda^{-1}\tau(G)\lambda\subseteq{\rm
Stab}_{\Gamma_{g}}([C,[f]])\\}}{\Lambda_{[m]}\cap{\rm N}_{\tau}}\,.$

The claim follows since the map from the quotient set in (4) to the set of
subgroups of ${\rm Stab}_{\Gamma_{g}}([C,[f]])$, induced by
$\lambda\mapsto\lambda^{-1}\tau(G)\lambda$, is injective.
The fact that $p\circ q$ is closed follows from the fact that, for any closed
subset $A\subseteq\mathcal{T}_{g}^{\tau(G)}$, the union of all $[\gamma]\cdot
A$, $[\gamma]\in\Gamma_{g}$, is closed in $\mathcal{T}_{g}$ (see e.g. [GDH92],
proof of Theorem 1). Finally, $Z_{\tau}$ and $p\circ q$ are algebraic by the
Generalized Riemann Existence Theorem of Grauert-Remmert [GR58] (see [Har77,
Theorem 3.2, Appendix B]), $Z_{\tau}$ is quasi-projective since
$\left(M_{g}[m]\right)^{\overline{\tau(G)}}$ is so. ∎

Proof of Thm. 2.5 From the proof of Proposition 2.4 we have that

$\mathcal{M}_{g}(G)=\bigsqcup_{\tau\in\mathbb{T}}[\mathcal{T}_{g}^{\tau(G)}/{\rm
C}_{\tau}]\,,$

where we use the notation of Section 2.1. Since ${\Lambda_{[m]}\cap{\rm
C}_{\tau}}$ acts freely on $\mathcal{T}_{g}^{\tau(G)}$ the stack quotient
$[\mathcal{T}_{g}^{\tau(G)}/({\Lambda_{[m]}\cap{\rm C}_{\tau}})]$ is
represented by the variety
$Z_{\tau}=\mathcal{T}_{g}^{\tau(G)}/({\Lambda_{[m]}\cap{\rm C}_{\tau}})$,
hence

$\mathcal{M}_{g}(G)=\bigsqcup_{\tau\in\mathbb{T}}\left[Z_{\tau}/{\bar{\rm
C}}_{\tau}\right]\,.$

The claim follows from the fact that the stack quotient of the algebraic
variety $Z_{\tau}$ (Proposition 4.1) by the finite group ${\bar{\rm
C}}_{\tau}$ is a Deligne-Mumford stack (see e.g. [LMB00, (4.6.1)]). ∎

## References

  * [AC09] Enrico Arbarello and Maurizio Cornalba, _Teichmüller space via Kuranishi families_ , Ann. Sc. Norm. Super. Pisa Cl. Sci. (5) 8 (2009), no. 1, 89–116. MR 2512202
  * [ACG11] Enrico Arbarello, Maurizio Cornalba, and Pillip A. Griffiths, _Geometry of algebraic curves. Volume II_ , Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], vol. 268, Springer, Heidelberg, 2011.
  * [ALR07] Alejandro Adem, Johann Leida, and Yongbin Ruan, _Orbifolds and stringy topology_ , Cambridge Tracts in Mathematics, vol. 171, Cambridge University Press, Cambridge, 2007.
  * [Cat00] Fabrizio Catanese, _Fibred surfaces, varieties isogenous to a product and related moduli spaces_ , Amer. J. Math. 122 (2000).
  * [Cat12] by same author, _Irreducibility of the space of cyclic covers of algebraic curves of fixed numerical type and the irreducible components of $Sing(\overline{\mathfrak{M}_{g}})$_, Advances in geometric analysis, Adv. Lect. Math. (ALM), vol. 21, Int. Press, Somerville, MA, 2012, pp. 281–306.
  * [Cat15] F. Catanese, _Topological methods in moduli theory_ , Bull. Math. Sci. 5 (2015).
  * [Cat17] Fabrizio Catanese, _Kodaira fibrations and beyond: methods for moduli theory_ , Jpn. J. Math. 12 (2017), no. 2, 91–174.
  * [CFGP19] Elisabetta Colombo, Paola Frediani, Alessandro Ghigi, and Matteo Penegini, _Shimura curves in the Prym locus_ , Commun. Contemp. Math. 21 (2019), no. 2.
  * [CLP] Fabrizio Catanese, Michael Lönne, and Fabio Perroni, _Homological stability for moduli spaces of curves with symmetries_ , (work in progress).
  * [CLP16] by same author, _Genus stabilization for the components of moduli spaces of curves with symmetries_ , Algebr. Geom. 3 (2016), no. 1, 23–49.
  * [Cor87] Maurizio Cornalba, _On the locus of curves with automorphisms_ , Ann. Mat. Pura Appl. (4) 149 (1987).
  * [EMMW20] Alex Eskin, Curtis T. McMullen, Ronen E. Mukamel, and Alex Wright, _Billiards, quadrilaterals, and moduli spaces_ , J. Amer. Math. Soc. 33 (2020), no. 4.
  * [FG16] Davide Frapporti and Christian Gleißner, _On threefolds isogenous to a product of curves_ , J. Algebra 465 (2016).
  * [FPP16] Paola Frediani, Matteo Penegini, and Paola Porru, _Shimura varieties in the Torelli locus via Galois coverings of elliptic curves_ , Geom. Dedicata 181 (2016).
  * [GDH92] G. González Díez and W. J. Harvey, _Moduli of Riemann surfaces with symmetry_ , Discrete groups and geometry (Birmingham, 1991), London Math. Soc. Lecture Note Ser., vol. 173, Cambridge Univ. Press, Cambridge, 1992, pp. 75–93.
  * [GR58] Hans Grauert and Reinhold Remmert, _Komplexe Räume_ , Math. Ann. 136 (1958), 245–318.
  * [Har71] W. J. Harvey, _On branch loci in Teichmüller space_ , Trans. Amer. Math. Soc. 153 (1971), 387–399.
  * [Har77] R. Hartshorne, _Algebraic geometry_ , Graduate Texts in Math., no. 52, Springer-Verlag, New York, 1977.
  * [Ker83] Steven P. Kerckhoff, _The Nielsen realization problem_ , Ann. of Math. (2) 117 (1983), no. 2, 235–265.
  * [KV04] Andrew Kresch and Angelo Vistoli, _On coverings of Deligne-Mumford stacks and surjectivity of the Brauer map_ , Bull. London Math. Soc. 36 (2004), no. 2, 188–192.
  * [Li18] Binru Li, _$G$ -marked moduli spaces_, Commun. Contemp. Math. 20 (2018), no. 6.
  * [LLR20] Ju A Lee, Michael Lönne, and Sönke Rollenske, _Double Kodaira fibrations with small signature_ , Internat. J. Math. 31 (2020), no. 7.
  * [LMB00] Gérard Laumon and Laurent Moret-Bailly, _Champs algébriques_ , Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge. A Series of Modern Surveys in Mathematics [Results in Mathematics and Related Areas. 3rd Series. A Series of Modern Surveys in Mathematics], vol. 39, Springer-Verlag, Berlin, 2000.
  * [MZ18] Abolfazl Mohajer and Kang Zuo, _On Shimura subvarieties generated by families of abelian covers of $\mathbb{P}^{1}$_, J. Pure Appl. Algebra 222 (2018), no. 4.


# A Dual Sensor Computational Camera for
High Quality Dark Videography

Yuxiao Cheng, Runzhao Yang, Zhihong Zhang, Jinli Suo, and Qionghai Dai All the
authors are with the Department of Automation, Tsinghua University. Jinli Suo
and Qionghai Dai are also affiliated with the Institute of Brain and Cognitive
Sciences, Tsinghua University. Emails<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>authors: Jinli Suo
and Qionghai Dai.

###### Abstract

Videos captured under low light conditions suffer from severe noise. A variety
of efforts have been devoted to image/video noise suppression and made large
progress. However, in extremely dark scenarios, extensive photon starvation
would hamper precise noise modeling. Instead, developing an imaging system
collecting more photons is a more effective way for high-quality video capture
under low illuminations. In this paper, we propose to build a dual-sensor
camera to additionally collect the photons in NIR wavelength, and make use of
the correlation between RGB and near-infrared (NIR) spectrum to perform high-
quality reconstruction from noisy dark video pairs. In hardware, we build a
compact dual-sensor camera capturing RGB and NIR videos simultaneously.
Computationally, we propose a dual-channel multi-frame attention network
(DCMAN) utilizing spatial-temporal-spectral priors to reconstruct the low-
light RGB and NIR videos. In addition, we build a high-quality paired RGB and
NIR video dataset, based on which the approach can be applied to different
sensors easily by training the DCMAN model with simulated noisy input
following a physical-process-based CMOS noise model. Both experiments on
synthetic and real videos validate the performance of this compact dual-sensor
camera design and the corresponding reconstruction algorithm in dark
videography.

###### Index Terms:

Computational photography, Video denoising, Low light video, Dark vision, RGB-
NIR, Dual-channel network.

##  1 Introduction

Under low light conditions, video photographers usually have to set high ISO
(sensitivity) and short exposure time to avoid motion blur, and thus suffer
from noise deterioration badly. To address this issue, researchers have been
attempting to develop various low-light image/video reconstruction methods in
the past decades. The basic idea of noise suppression or removal is
introducing prior information of nature scenes to recover the latent clean
version, and some methods can achieve excellent reconstruction results, such
as [1][2][3][4]. The reconstruction can be conducted on either a single image
or a video sequence. Generally speaking, multi-frame noise reduction methods
usually get better results than those based on a single frame, thanks to the
utilization of priors from temporal redundancy of dynamic scenes.

Figure 1: An example of paired RGB (top) and NIR (bottom) video frame captured
computationally by our approach under dark environment. The three segments in
each image show the extremely dark raw measurement, linearly scaled version
with severe noise and color distortion, and the clean final reconstruction,
respectively.

In spite of the extensive studies in spatial and temporal priors of nature
image/videos and their applications in reconstruction tasks, high-quality
imaging under extremely low illuminations is still quite challenging. For
example, in Fig. 1, the raw measurement is quite low, and after $\sim$10-time
magnification, only the general scene structure can be perceived while the
details are buried in severe noise and there exists color distortion. To
achieve more decent imaging under environments with poor illuminations, one
can resort to collecting more photons and introducing new priors. Actually,
there exist a bunch of near-infrared lights under low illuminations and a
typical camera sensor can respond to photons from wavelength from 350-1200nm
[5]. Therefore, recording the photons in NIR band additionally can improve the
quality of dark imaging. However, existing approaches are lacking in the
following aspects: (i) An infrared cut-off filter is placed before the image
sensor of RGB/gray camera to cut off NIR light for a natural tone matching the
perception of human eyes (RGB, roughly 400-700nm) [6]. As a result, severe
noise occurs due to the insufficient photons in visible wavelength. (ii) RGB
and NIR images share similar features on scene structure and relative
brightness, as shown in the two rows in Fig. 1, and it is possible to make use
of such cross-channel correlations between RGB and NIR images to enhance the
imaging quality algorithmically. However, current denoising methods rarely
utilize this cross channel prior.

Figure 2: Analysis of the cross channel correlation among three visible
channels (i.e., red, green, blue), and between RGB and NIR channels, with SSIM
and NMI as score metrics. These plots are calculated from 50 RGB/NIR image
pairs.

Inspired by the above analysis, we design a compact double-sensor camera
system to acquire paired RGB and NIR images at the same time, with a beam
splitter placed between the primary lens and image sensors. Using this setup,
we capture a dataset of RGB-NIR nature video pairs, and analyze the image
similarity among the 4 channels (R, G, B, and NIR), in terms of SSIM
(Structural Similarity) and NMI (Normalized Mutual Information) scores. The
results are plotted in Fig. 2 with the scattered dots concentrated in the
right-top region of each subfigure, which validates that the red, green, blue,
and infrared channels are indeed strongly correlated, and the correlations
between visible and NIR wavelength are only slightly lower than that among R,
G, and B channels. In other words, we can introduce priors across visible and
NIR channels to improve the final reconstruction.

Computationally, we design a Dual-channel Multi-frame Attention Network
(DCMAN) based on bi-directional ConvLSTM and CNN-based encoder-decoder with
mutual multi-level feature fusion [7]. In other words, the data from the two
arms contribute mutually to recovering the latent structure of nature videos
buried in the severe noise. By additionally introducing the photons in the
near-infrared wavelength, and taking advantage of spatial-temporal as well as
spectral prior information, we can achieve better reconstruction quality. Our
method takes noisy NIR and RGB videos as inputs and gives clean results of
both videos at the same time, as shown in Fig. 1. The result demonstrates that
the visual quality is largely raised with fine details being reconstructed.

Overall, this paper reports a computational photography approach for dark
video capturing and mainly contributes in following aspects:

  * •

We build a dual-sensor camera system capturing RGB and NIR video pairs
simultaneously, being able to increase the collected photons effectively for
higher imaging quality in dark environments.

  * •

We design a Dual-Channel Multi-frame Attention Network (DCMAN) with Guided
Skip Connections (GSCs) to utilize spatial, temporal, and spectral video
prior. This network can serve as a general structure applicable for similar
tasks taking multiple input channels.

  * •

The approach can be adapted to different sensors and noise levels easily by
learning the model from synthetic data, produced by adding noise to our
collected high quality RGB + NIR video dataset on real scenes.

  * •

The proposed approach can achieve superior performance to existing methods in
most dark scenarios.

##  2 Related Works

###  2.1 Video Denoising

Video denoising has been extensively studied in the past decades [8][9][10],
and the large amount of literature falls into several groups based on the ways
of specifying priors.

Statistically, the distribution of natural images/videos is discriminative
from noises globally or over local patches. One way of noise removal is
applying filters spatiotemporally or in a transformed domain after motion
compensation, making use of the fact that noise components are of high
frequency while the latent image/video concentrates at low frequency band
[11][12][13]. Another way is conducting video denoising over patches,
exploiting temporal and spatial patch similarity in the neighborhood. For
example, Buates et al. [14] propose to conduct patch-based denoising after
compensating motions among neighboring frames, and some other methods can
suppress noise without motion compensation, such as non-local means algorithm
[15], and V-BM3D [16] and its variants. V-BM3D [16] is the extension of patch-
based method BM3D [17], which extracts similar 2D patches from consecutive
video frames and then stacks them for filtering in a transformed domain.
Later, they extend V-BM3D to V-BM4D [18] searching similar 3D spatio-temporal
blocks instead of 2D patches, and VNLB [19] that conducts inference with
Bayesian estimation.

With the rapid development of Convolutional Neural Networks (CNN) and deep
learning techniques, learning-based image reconstruction exhibits better
performance and is attracting wide attention. In this stream, video denoising,
however, is much less explored than image denoising, since extracting spatio-
temporal features jointly is non-trivial. On the one hand, researchers try to
extend patch-based image denoising by introducing temporal information with
CNN feature fusion. Tassano et al. [20] propose DVDnet by explicitly splitting
the denoising process into two successive stages. Specifically, consecutive
frames are first individually denoised and then temporally denoised with
motion compensation. Later, building on DVDnet, they further propose
FastDVDnet [3] without flow estimation and achieved higher performance. Using
a similar two-stage method, Claus and Gemert [2] design ViDeNN, and Mildenhall
et al. [21] design a Kernel Prediction Network (KPN) to suppress noises in a
burst of dark images. On the other hand, some researchers explore Recurrent
Neural Networks for extracting temporal information. Chen et al. [22] develop
an RNN-based method to utilize temporal video prior for noise reduction. Wang
et al. [23] introduce an LSTM-based method, extracting both short and long-
term dependencies from image sequences. Godard et al. [4] propose a global
recurrent network and append it to existing CNN-based single image denoising
networks. Beyond these works on spatio-temporal feature extraction, we explore
the possibility of making use of spatial, temporal, as well as spectral
corrections in one end-to-end network—CNN for spatial information, LSTM for
temporal information, and finally network channel fusion for spectral RGB-NIR
information.

###  2.2 Noise Modeling

The noise in low light photography is complex and closely correlated with
camera sensors. A precise description of the sensor noise is crucial for the
final denoising performance. In the early stage, most of the denoising methods
assumes identically distributed (i.i.d.) additive white Gaussian noise (AWGN)
and are of limited performance. Recently, several new noise models are
explored, such as mixture of Gaussian (MoG) [24], Poisson-Gaussian [25] and
other physical-process-based mixture model [26][27][28][29]. Since high
sensitivity cameras are preferred in low light imaging, Wang et al. [23] model
CMOS noise with a more complex high sensitivity noise model and propose a
noise calibration method assisting generating plausible synthetic noisy
images/videos. In this paper, we build an integrated physical-process-based
noise model covering rich noise sources, with high performance and good
compatibility with different sensors.

###  2.3 RGB-NIR Imaging

The short-wavelength NIR images are of similar brightness and structures to
RGB images. Especially, under low-light conditions NIR images serve good
performance [30] and exhibit much more details invisible to eyes [31], so
introducing NIR to RGB cameras can be a promising way for low light
photography. Fortunately, widely used CCD and CMOS can detect photons in
short-wavelength NIR range up to 1200 nm, which largely facilitate RGB-NIR
hybrid imaging. Towards this direction, researchers have made some progress.
Some groups use NIR images to assist with high-quality RGB imaging. For
example, Krishnan and Fergus [5] and Zhuo et al. [32] enhance RGB images
captured with ambient light using images captured with NIR flashlight (dark
flash), whereas the former captures two images sequentially and the latter
uses two synchronized hybrid cameras to take the pair simultaneously.
Similarly, Sugimura et al. [33] construct an imaging system to take RGB and
NIR photos with different exposure times and recover clean color images from
those image pairs. Conversely, Han et al. [30] propose to super-resolve low-
resolution NIR images using a high-resolution RGB image. There are also works
trying to capture both high-resolution NIR and RGB images computationally, for
example, Hu et al. [34] propose RGB-NIR reconstruction techniques in a single
sensor with a novel RGB-NIR Color Filter Array (CFA). In spite of the above
progress, there is still no working can capture and enhance both RGB and NIR
for dark visual recording.

In this paper, we build a compact RGB/NIR imaging system to capture paired
videos, which is largely different from [33] that uses two separate cameras.
Such a design is of high stability and can be used in the cases requiring
lightweight imaging setups. The reconstruction is also different from the
above hybrid imaging by enhancing both RGB and NIR videos to maximize the
mutual assistance, instead of focusing on only one of them. In addition, a
high-quality RGB-NIR dataset is important for data-driven algorithms, but
several existing datasets, such as [35] and [36], either consists of only
single images (i.e., static scenes) or are constrained to a specific type of
scenes. To address these problems, we capture a dataset with paired RGB and
NIR nature videos.

###  2.4 Channel-Fusion Learning

Channel fusion is an effective way of taking multiple input data and learning
the latent structure from both intra- and inter- multiple data sources and can
incorporate CNN-based deep networks. Specifically, the informative feature
extracted by CNN can be fused at the input level, early level, or late level
[6][37]. However, to leverage features at different levels, researchers
propose networks with feature fusion across multiple levels [7]. The
advantages of such multi-level fusion have been successfully applied in high-
level vision tasks such as object or scene change detection [7] and [38], but
the studies are still at an early stage in the field of image reconstruction.

Figure 3: The structure of our proposed Dual-Channel Multi-Frame Attention
Network (DCMAN). Our network includes two channels: each channel consists of a
ConvLSTM module and a U-Net-shaped CNN module; two channels are fused with
Guided Skip Connections (GSCs). This network explores the spatial prior in
nature videos with the U-Net CNN module, temporal prior with the Bi-ConvLSTM
module, and spectral prior with inter-channel GSCs.

Meng et al. [39] use consecutive frames as dominant data channel while guided
maps serve as an assistant arm, which guides the network to focus more on
certain regions. Like in [7], these two channels are fused at multiple levels
to achieve high performance. Similarly, Zha et al. [40] design a dual-channel
multi-scale deep network to fuse the features from multi-spectral bands.
Inspired by that work, we can utilize RGB and NIR correlations explicitly. In
this paper, the proposed dual-channel multi-frame attention network (DCMAN) is
a symmetrical structure, which treats two channels—RGB and NIR—in a parallel
manner to mutually enhance each other’s quality, considering the similar
photon counts and noise levels at two corresponding input light paths.

##  3 Network Structure

We propose a Dual-Channel Multi-frame Attention Network (DCMAN) with two
channels taking noisy RGB and NIR inputs: each channel is equipped with a
ConvLSTM module and a U-Net-shaped CNN module to explore the spatio-temporal
redundancies for noise suppression; two channels are fused with Guided Skip
Connections at multiple layers to explore cross-spectrum prior in nature
videos.

The training dataset is composed of paired noisy and clean video patches
cropped at the same location in consecutive $2T+1$ frames, each entry can be
described as

$\left\\{<\mathbf{I}^{1\cdots(2T+1)}_{\text{rgb}},\mathbf{I}^{1\cdots(2T+1)}_{\text{nir}}>,<\mathbf{\widetilde{I}}^{1\cdots(2T+1)}_{\text{rgb}},\mathbf{\widetilde{I}}^{1\cdots(2T+1)}_{\text{nir}}>\right\\}$
(1)

Here $\mathbf{I}^{t}_{\text{rgb}}$ and $\mathbf{I}^{t}_{\text{nir}}$ denote a
clean RGB and corresponding NIR patch captured under good illumination,
$\mathbf{\widetilde{I}}^{t}_{\text{rgb}}$ and
$\mathbf{\widetilde{I}}^{t}_{\text{nir}}$ denote noisy counterparts generated
following the physical based noise model, and $t$ denotes the frame index. We
define the neighboring frames of $\mathbf{I}^{t}$ as

$\mathcal{N}(\mathbf{I}^{t})=\\{\mathbf{I}^{t-T},\mathbf{I}^{t-T+1},\cdots\mathbf{I}^{t-1},\mathbf{I}^{t+1},\cdots\mathbf{I}^{t+T-1},\mathbf{I}^{t+T}\\}.$
(2)

In the next subsections, we describe the processing of $t$th frame, and
omitted the subscript $t$ for simplicity.

###  3.1 Bi-ConvLSTM Subnet

Utilizing the redundancies among neighboring video frames, the noise
deterioration in a video frame can be corrected by neighboring frame patches,
including past and future ones. To explore the latent temporal correlation, we
first equip the network with two identical bi-directional ConvLSTM [39][41]
modules, which process the input RGB patch sequence
$\mathcal{N}(\mathbf{\widetilde{I}}_{\text{rgb}})$ and NIR patches
$\mathcal{N}(\mathbf{\widetilde{I}}_{\text{nir}})$ separately after an
affiliated pre-convolutional layers. The residual connections in bi-
directional ConvLSTM (bi-ConvLSTM) will guide the network to penalize large
differences between adjacent frame patches. Specifically, the bi-ConvLSTM
$F_{\text{lstm}}\left(\cdot\right)$ can be formulated as follows:

$\displaystyle\mathbf{\widetilde{I}}^{\text{lstm}}_{\text{rgb}}$
$\displaystyle=F_{\text{lstm}}\left(\mathcal{N}(\mathbf{\widetilde{I}}_{\text{rgb}});\theta^{\text{lstm}}_{\text{rgb}}\right)$
(3) $\displaystyle\mathbf{\widetilde{I}}^{\text{lstm}}_{\text{nir}}$
$\displaystyle=F_{\text{lstm}}\left(\mathcal{N}(\mathbf{\widetilde{I}}_{\text{nir}});\theta^{\text{lstm}}_{\text{nir}}\right),$
(4)

with $\theta^{\text{lstm}}_{\text{rgb}}$ and
$\theta^{\text{lstm}}_{\text{nir}}$ being the network parameters of
$F_{\text{lstm}}\left(\cdot\right)$ to be learned. The receptive field of this
module depends on the number of pre-Conv and bi-ConvLSTM layers. Usually the
motion between adjacent frames is relatively mild, a shallow bi-ConvLSTM
module would be sufficient.

###  3.2 Dual-Channel Encoder

To further extract the spatio-temporal-spectral information, we design a Dual-
Channel subnet based on encoder-decoder networks. The encoder module of each
channel has 4 encode layers, each layer takes output of the the former layer
as input. The input of the first layer
$\mathbf{\widetilde{I}}^{(0)}_{\text{rgb}}$ and
$\mathbf{\widetilde{I}}^{(0)}_{\text{nir}}$ are the output of bi-ConvLSTM
module, i.e.,
$\mathbf{\widetilde{I}}^{(0)}_{\text{rgb}}=\mathbf{\widetilde{I}}^{\text{lstm}}_{\text{rgb}}$
and
$\mathbf{\widetilde{I}}^{(0)}_{\text{nir}}=\mathbf{\widetilde{I}}^{\text{lstm}}_{\text{nir}}$.
The encoder can be formulated as

$\displaystyle\mathbf{\widetilde{I}}_{\text{rgb}}^{(k)}$ $\displaystyle=$
$\displaystyle
F_{\downarrow}^{(k)}\left(\mathbf{\widetilde{I}}^{(k-1)}_{\text{rgb}};\theta_{r\downarrow}^{(k)}\right)$
(5) $\displaystyle\mathbf{\widetilde{I}}_{\text{nir}}^{(k)}$ $\displaystyle=$
$\displaystyle
F_{\downarrow}^{(k)}\left(\mathbf{\widetilde{I}}^{(k-1)}_{\text{nir}};\theta_{n\downarrow}^{(k)}\right),\
k\in[1,4].$ (6)

In these two equations, $F_{\downarrow}^{(k)}$ is the encoder layer at the
$k$th level with $k\in[1,4]$, $\theta_{\text{rgb}\downarrow}^{(k)}$ and
$\theta_{\text{nir}\downarrow}^{(k)}$ are parameters of subnets in two
channels, repsectively.

###  3.3 Guided Skip Connections in Decoder

The RGB and NIR channels guide each other mutually with the help of the Guided
Skip Connections (GSCs), an extension to Skip Connections [42]. GSCs in
decoder module can not only synthesize high-resolution low-level features,
like the skip connections does, but can also combine features from another
channel (RGB or NIR). In our GSCs, the features from encoder are concatenated
and passed to the 4-layer decoder $F_{\uparrow}^{(k)}$ with $k\in[1,4]$,
specifically defined as

$\displaystyle\mathbf{\hat{I}}_{\text{rgb}}^{(k-1)}$ $\displaystyle=$
$\displaystyle
F_{\uparrow}^{(k)}\left(\left\\{\mathbf{\hat{I}}_{\text{rgb}}^{(k)},\mathbf{\widetilde{I}}_{\text{rgb}}^{(k)},\mathbf{\widetilde{I}}_{\text{nir}}^{(k)}\right\\}\
;\ \theta_{\text{rgb}\uparrow}^{(k)}\right)$ (7)
$\displaystyle\mathbf{\hat{I}}_{\text{nir}}^{(k-1)}$ $\displaystyle=$
$\displaystyle
F_{\uparrow}^{(k)}\left(\left\\{\mathbf{\hat{I}}_{\text{nir}}^{(k)},\mathbf{\widetilde{I}}_{\text{nir}}^{(k)},\mathbf{\widetilde{I}}_{\text{rgb}}^{(k)}\right\\}\
;\ \theta_{\text{nir}\uparrow}^{(k)}\right).$ (8)

Here
$\mathbf{\hat{I}}_{\text{rgb}}^{(4)}=\mathbf{\widetilde{I}_{\text{rgb}}^{(4)}},\
\mathbf{\hat{I}_{\text{nir}}^{(4)}}=\mathbf{\widetilde{I}}_{\text{nir}}^{(4)}$,
i.e., output of the encoder module. The final output of our network are

$\displaystyle\mathbf{\hat{I}}_{\text{rgb}}$ $\displaystyle=$ $\displaystyle
F^{(0)}\left(\left\\{\mathbf{\hat{I}}_{\text{rgb}}^{(0)},\mathbf{\widetilde{I}}_{\text{rgb}}^{(0)},\mathbf{\widetilde{I}}_{\text{nir}}^{(0)}\right\\}\
;\ \theta_{\text{rgb}}^{(0)}\right)$ (9)
$\displaystyle\mathbf{\hat{I}}_{\text{nir}}$ $\displaystyle=$ $\displaystyle
F^{(0)}\left(\left\\{\mathbf{\hat{I}}_{\text{nir}}^{(0)},\mathbf{\widetilde{I}}_{\text{nir}}^{(0)},\mathbf{\widetilde{I}}_{\text{rgb}}^{(0)}\right\\}\
;\ \theta_{\text{nir}}^{(0)}\right)$ (10)

###  3.4 Loss Function

The loss function for our network is defined to favor better image sharpness
and higher color fidelity jointly. Specifically, the loss function can be
written as

$\displaystyle\begin{aligned}
L=&\sum_{i}\lambda_{1}\left(\left\|\mathbf{\hat{I}}_{i,\text{rgb}}-\mathbf{I}_{i,\text{rgb}}\right\|_{1}+\left\|\mathbf{\hat{I}}_{i,\text{nir}}-\mathbf{I}_{i,\text{nir}}\right\|_{1}\right)+\\\
&\sum_{i}\lambda_{2}L_{\text{cos}}(\mathbf{\hat{I}}_{i,\text{rgb}},\mathbf{I}_{i,\text{rgb}}),\end{aligned}$
(11)

in which $i$ indexes patches in the training batch, and $L_{\text{cos}}$
denotes cosine embedding loss[43]

$\displaystyle\begin{aligned}
L_{\text{cos}}(\mathbf{x},\mathbf{y})=1-\cos(\mathbf{x},\mathbf{y}).\end{aligned}$
(12)

##  4 Integrated CMOS Noise Model

Figure 4: Comparison among the synthetic noisy images of the scene in (a)
following different noise models (b-c) and real noisy images captured by
different cameras (d-f). (a) Bright image captured under long exposure. (b)(c)
Images superimposed with Gaussian noise and Poisson noise, respectively.
(d)(e)(f) Noisy images captured with iPhone 11, Fujifilm X100T and Sony A7R2,
respectively.

Capturing clean-noisy paired data for model training is time-consuming and
inflexible for different cameras, so here we add noise to the high-quality
NIR-RGB database to synthesize training data instead. A precise noise modeling
is crucial for the final results. Many denoising algorithms assume the noise
to be Gaussian or Poisson [3][44]. However, test results in Fig. 4 show that
these models can not describe the true low light noise well. Besides, low-
light photos taken with different cameras vary a lot. These facts inspire us
to use an integrated physical-process-based noise model being able to
incorporate the various noise sources and compatible with different sensors.

A typical CMOS camera sensor takes three steps to convert photon hit on the
photosensor to the final digital images: the photosensor detects photons and
converts them to electrons, integrated circuits convert and amplify electrons
to voltage, and an ADC quantizes analog voltage signals to digital signals.
During these stages, various sources of noise exist. We build an integrated
noise model taking all the fundamental ones into account and adopt proper
probabilistic distribution for each term to approximate the physical process
best.

(i) Shot noise.  The shot noise is caused by the randomness of photon arrival
and can be modeled with Poisson distribution with the mean value being the
expected photon number $N$ [45]. The shot noise is determined by the signal
itself and the photons at different color channels vary. The shot noise at
channel $c$ is

$n^{\text{sht}}_{c}\sim P(N_{c}),c\in\\{{\text{R, G, B, NIR}}\\},$ (13)

with $P(\cdot)$ denoting the Poisson distribution.

(ii) Dark current.  Dark current is caused by the random generation of
electrons and holes, leading to a signal-independent noise [45]. The modeling
of this noise is, however, more complex. In some prior works, dark current is
assumed to follow a Gaussian or Poisson distribution [23][45]. However,
researchers in [28] observe the long-tail shape of noise data and propose to
model this noise by Tukey lambda distribution, which matches the true dark
current well. Inspired by this work, we model this noise using a clipped
Poisson distribution with better approximation and compatibility with
different sensors:

$n^{\text{dk}}=\max\\{0,n_{d}-N_{d}\\},\ n_{d}\sim P(N_{d}).$ (14)

Here $N_{d}$ denotes the expected number of dark current electrons of each
pixel.

(iii) Read noise and Dynamic streak noise.  During the conversion from
electrons to voltage signals, read noise appears. This kind of noise does not
depend on the signal and can be modeled with a Gaussian distribution and added
to the signal directly as

$n^{\text{rd}}\sim G(0,\sigma_{r}^{2}).$ (15)

In low light imaging, image quality would be further worsened by dynamic
streak noise (DSN) [23], which appears to be horizontal streaks in low light
noisy photos. This noise heavily depends on the pixel location. Combining shot
noise, dark current, read noise and DSN, the equation would be

$y_{r,c}=\beta_{r,c}(n^{{\text{sht}}}_{c}+n^{\text{dk}}+n^{{\text{rd}}})$ (16)

where $r$ denotes the row index of pixel and $\beta_{r,c}\sim
G(1,\sigma_{\beta})$.

Figure 5: Comparison of noisy images produced by different noise models with
the real images captured under low illumination. (a) Clean image patch for
simulation. (b) Results by additive white Gaussian noise. (c) By Poisson
noise. (d) By our physical-process-based noise model. (e) Real noisy image
patch of the same scene.

(iv) Quantization.  Under low illumination, one usually set a high information
gain to amplify the analog signal before quantization, and magnify further
digitally for better visualization and subsequent processing. Mathematically,
the voltage signals from camera sensor (with shot noise and dark current) are
amplified $K_{a,c}$ times before digitalized, and the quantized signals can be
expressed as

$y_{q,c}=\lfloor
K_{a,c}\beta_{r,c}(n^{{\text{sht}}}_{c}+n^{\text{dk}}+n^{{\text{rd}}})\rfloor,$
(17)

where $\lfloor\cdot\rfloor$ denotes the floor function. Afterwards, the images
can be amplified to fit a proper scale, e.g. [0, 255], with digital gain
$K_{d}$.

In this paper, we integrated all above terms to model the noise based on the
physical process of image recording. The model can be formulated as

$y_{c}=K_{d}(\lfloor
K_{a,c}\beta_{r,c}(n^{{\text{sht}}}_{c}+n^{\text{dk}}+n^{{\text{rd}}})\rfloor),$
(18)

in which the parameters of noise terms $n^{{\text{sht}}}_{c}$,
$n^{\text{dk}}$, and $n^{{\text{rd}}}$ are all defined and calibrated in a
pixel-wise manner.

To synthesize noisy low light images, we set parameter $N_{c}$ based on the
target flux and clean image pixel and simulate system gain $K=K_{a}K_{d}$.
Suppose we capture videos in extremely dark environments, and set the highest
analog amplification factor $K_{a,c}$ and a digital gain adjust
$K_{d}=\frac{K}{K_{a}}$ is set. Then the number of photons $N_{c}$ can be
infered as $N_{c}=\frac{{\bf{\text{I}}}_{c}}{K}$ to maintain the overall
brightness. Parameters are calibrated following the procedure described in
[23], and then randomly fluctuates within a range to improve robustness. The
ranges of the key parameters are shown in Tab. II. Note that image noise
intensity is mainly determined by the system gain $K$. Fig. 5 demonstrates the
simulated noisy data and comparison with results by other models. We can see
that our noise model provides simulation results much closer to real captured
low light video frames, especially in the dark regions.

##  5 Image Acquisition

###  5.1 The RGB-NIR Imaging System

Figure 6: Our RGB-NIR imaging system. (a) Diagram of our two sensor
acquisition system, with the sensor response of the visible band and near-
infrared plotted. (b) A photo of our prototype.

To capture paired RGB and NIR videos, we build an RGB+NIR camera with two 1"
CMOS image sensors (HIKROBOT MV-CH089-UC and MV-CH089-10UM) recording the
visible and near-infrared bands respectively. As illustrated in Fig. 6(a), the
target scene is firstly captured by the primary lens (Nikon BlueVision
BV-L1024, $f$=24mm, F-Mount), and then the outgoing light is split by a cubic
beam splitter which reflects RGB light (stops at roughly 700nm) and transmits
NIR light. The spectral transmission curve of the beam splitter and simulated
spectral response of red, green, blue, and near-infrared channels are also
plotted in Fig. 6(a). Later, two arms arrive at two orthogonally placed
sensors: the RGB sensor UC with Bayer CFA captures RGB images, and monochrome
sensor UM without a near-infrared cut-off filter captures monochromatic
images). These two arms are complementary in collecting photons falling within
the broad spectral region (350-1100 nm) of two CMOS sensors. Two cameras are
wired to a PC and are software synchronized. After capturing the raw video
pairs, we need to align the images from two sensors. Since the RGB and NIR
images share the same optical axis, there is no parallax and a precise
alignment can be achieved with a simple SIFT detector [46] and affine
transformation. The pixel resolution of the aligned video pair is around
$1280\times 720$.

Figure 7: Several examples from our RGB-NIR video dataset, with pixel-wise registration bewteen measurements in two wavelength bands. TABLE I: Comparison of performance in terms of PSNR (dB) / SSIM on our RGB-NIR video dataset. $K=K_{a}K_{d}$ denotes the total gain factor during the simulation process. S: Spatial, T: Temporal, C: Spectral. “Ours S” denotes the results from SCMAN_Fn1, with only spatial prior. “Ours S+” denotes the results from SCMAN_Fn7, with explicit exploration of temporal prior. “Ours S+T+C” denotes the results from DCMAN_Fn7 (or DCMAN), with spatio-temporal and spectral prior jointly. "Ours$-$LSTM" denotes the results by removing bi-ConvLSTM module from DCMAN_Fn7. Simu. |  V-BM4D [18] |  TOFlow [44] |  FastDVDnet [3] | Ours S | Ours S+T |  Ours$-$LSTM | Ours S+T+C
---|---|---|---|---|---|---|---
$K=10$ | 27.974 / 0.867 | 28.976 / 0.914 | 29.020 / 0.919 | 29.530 / 0.906 | 31.062 / 0.935 | 31.121 / 0.936 | 31.150 / 0.937
$K=20$ | 24.017 / 0.790 | 25.943 / 0.873 | 26.012 / 0.883 | 28.127 / 0.879 | 29.838 / 0.916 | 29.863 / 0.918 | 29.965 / 0.920
$K=40$ | 19.667 / 0.659 | 22.227 / 0.802 | 22.371 / 0.819 | 26.442 / 0.841 | 27.947 / 0.882 | 28.256 / 0.889 | 28.427 / 0.892
Figure 8: Simulation test results of our model compared with state of the art
methods V-BM4D [18], TOFlow [44], and FastDVDnet [3], and the degenerated
single-channel version of our deep network (SCMAN), i.e.,model without NIR
channel. (a) Simulated noisy frame. (b) V-BM4D. (c) TOFlow. (d) FastDVDnet.
(e) Ours. (f) Groundtruth. In (g) we show the comparison in the highlighted
region for better visualization, with the same layout as (a)-(f).

###  5.2 Data for Model Training and Testing

For simulating the clean-noisy data for supervised model training and
quantitative performance evaluation, we collect a dataset including 118 high-
quality RGB-NIR video clips with 11,444 frames of real nature scenes under
sufficient illuminance. Using a high-end commercial camera as a reference, we
adjust the color tone and image contrast by building a Look-Up-Table (LUT) to
improve the quality of our dataset. Several examples are shown in Fig. 7. The
training data for the network is generated by adding noise to the bright
dataset following the aforementioned CMOS noise model. For quantitative
evaluation, 104 videos are used for model training and the rest for testing.

To test the performance on real low light videos, we captured noisy input in
dark environments with around 0.125%$\sim$0.5% illuminance of daily bright
photography. These testing data are preprocessed in the same way as the
dataset and linearly scaled to normal brightness before being fed into the
trained deep network.

##  6 Experiments and Analysis

###  6.1 Experiment Settings

The training set for the DCMAN is synthesized by adding noise to our RGB-NIR
clean video pairs following the CMOS noise model, with the parameters listed
in Table. II. To raise the robustness towards different noise levels,
parameters $K_{a}$ and $K_{d}$ are randomly selected for each simulated low
light video.

TABLE II: Parameter settings of our CMOS noise model for generating paired training data. Each parameter is randomly selected from a uniform distribution over a range instead of a fixed value, which can help cover the intensity variation among different scenes. Parameters |  Distribution
---|---
$N_{d}$: Dark current electron number | $U(2,10)$
$\sigma_{\beta}$: Standard deviation of DSN | $U(0.02,0.08)$
$\sigma_{r}$: Standard deviation of read noise | $U(0.5,2)$
$K_{RGB}$: Gain in RGB channel | $U(10,40)$
$K_{NIR}$: Gain in NIR channel | $U(K_{RGB},3K_{RGB})$

We conduct experiments on a PC with Intel Core CPUs and NVIDIA GeForce GTX
1080Ti GPUs. The DCMAN network is trained with 20,000 paired video patches
randomly cropped from original synthesized videos with a patch size of
$120\times 120$ pixels. We use the Adam optimizer to train the network. The
batch size is 10, the learning rate is $10^{-4}$ and decreases by a factor of
0.1 every epoch. The color loss parameter $\lambda_{2}$ is 0.1 initially and
increases to 0.25 and 0.6 after the 20th and 30th epoch, respectively, and
$\lambda_{1}=1-\lambda_{2}$. We terminate the training after 40 epochs. The
whole training process takes around 40 hours.

As the first to discuss low light video reconstruction with RGB and NIR video
pairs, in order to prove our superior performance towards RGB video methods,
our method is compared with three state-of-the-art video denoising methods,
one filtering based and two data-driven methods: V-BM4D [18], FastDVDnet [3]
and TOFlow [44]. For a fair comparison, we conduct the following preprocessing
to optimize the performance of three benchmark methods. For V-BM4D,
information on noise level is required and we use its in-built noise estimator
to estimate the sigma value (standard variation of the Gaussian noise). For
FastDVDnet, we remove the noise map and re-train the model on RGB videos in
our RGB-NIR dataset to maintain fairness. TOFlow requires the optical flow
between adjacent frames, so we use the pre-trained flow estimation network
SpyNet [47] and fine-tune it on our dataset with superimposed Gaussian noise,
as mentioned in its original paper.

Figure 9: Comparison of SCMAN_Fn1 (Spatial information), SCMAN_Fn7
(Spatial+Temporal information) and DCMAN_Fn7 (Spatial+Temporal+Spectral
information) results to verify the effectiveness of our multi-frame dual-
channel network. (a) and (b) show the comparion on synthetic and real data,
respectively.

.

TABLE III: Performance on simulation test for several $F_{n}$ and system gain $K$. Analytical data are shown as $PSNR_{DCMAN}/PSNR_{SCMAN}(dB)$. $PSNR_{DCMAN}$ denotes PSNR of DCMAN (with NIR channel), and $PSNR_{SCMAN}$ denotes that of SCMAN (without NIR channel). Simultion Test |  $F_{n}=1$ |  $F_{n}=3$ |  $F_{n}=5$ |  $F_{n}=7$
---|---|---|---|---
$K=10$ | 29.851 / 29.530 | 30.826 / 30.603 | 31.166 / 30.889 | 31.150 / 31.062
$K=20$ | 28.515 / 28.127 | 29.521 / 29.192 | 29.898 / 29.533 | 29.965 / 29.838
$K=40$ | 26.875 / 26.442 | 27.875 / 27.475 | 28.275 / 27.847 | 28.427 / 27.947

###  6.2 Experiments on Synthetic Noisy Videos

To quantify our performance, we first apply our methods on the simulated test
dataset, using PSNR (Peak signal-to-noise ratio) and SSIM (The Structural
Similarity Index) as evaluation metrics. We also compare with three state-of-
the-art methods, as shown in Tab. I, including a patch-based video denoising
method V-BM4D [18], a deep-learning-based method FastDVDnet [3], and an optic-
flow-based deep learning method TOFlow [44].

The experiment is performed under the setting $K_{RGB}=K_{NIR}$. Other noise
parameters are set to match Tab. II. The results show that our method
outperforms other methods by a big gap. At $K=40$, the PSNR increment goes up
to 8.76 dB, 6.20 dB, and 6.06 dB higher than V-BM4D, TOFlow, and FastDVDnet,
respectively. The visual comparison in Fig. 8 displays similar advantages. One
can see that V-BM4D and TOFlow leave some noise and FastDVDnet tends to
produce oversmooth results, while the proposed approach preserves much more
details and better color fidelity.

The superior performance benefits from multiple aspects: firstly, more
information from the infrared channel is utilized in our methods, while V-BM4D
and FastDVDnet only use RGB input. Secondly, 7 adjacent frames are used to
denoise the center frame instead of 5 in FastDVDnet. Finally, two previous
deep-learning-based methods (TOFlow and FastDVDnet) are retrained on our data
but both assume Gaussian noise, while our models are trained with data
synthesized following an integrated CMOS noise model and calibrated
parameters.

###  6.3 Ablation Studies

Temporal prior information.  In our network, the LSTM module combines
information from $F_{n}=2T+1$ frames to denoise the center frame. To evaluate
the influence from temporal information utilization, we set $F_{n}$ to be 1,
3, 5, and 7 respectively to train 4 models DCMAN_Fn1, DCMAN_Fn3, DCMAN_Fn5,
and DCMAN_Fn7. Their performances are shown in Tab. III. Generally, the
denoise performance grows as $F_{n}$ increases. Empirically, in our final
network, we set $F_{n}=7$, which provides the network with a massive amount of
temporal prior assistance. Comparing DCMAN_Fn1 with DCMAN_Fn7 ($K=40$), i.e.,
the “Ours S” and “Ours S+T” columns in Tab. I, introducing temporal
information increases PSNR by 1.55 dB.

Spectral prior information.  One of the key contributions of our work is the
utilization of additional NIR information. To quantitatively evaluate the
performance improvement brought by the NIR information, we disable the NIR
channel in our network and remove the corresponding input to train a network
recovering only RGB video frames. The network is called Single-Channel Multi-
frame Attention Network (SCMAN), as shown in the two parallel insets of Fig.
3.

We firstly vary the frame number ($F_{n}=1,3,5,7$) to train SCMAN_Fn1,
SCMAN_Fn3, SCMAN_Fn5, and SCMAN_Fn7, with the results shown in Tab. III. We
can see that introducing NIR information indeed raises the performance at
almost all of the settings. Comparing SCMAN_Fn7 with DCMAN_Fn7 ($K=40$), we
can calculate that spectral information increases PSNR by 0.48 dB on our test
dataset, as shown in the “Ours S+T+C” column in Tab. I. We also compare the
reconstruction results from DCMAN_Fn7 and SCMAN_Fn7 visually in Fig. 9. The
conclusion consists well with the quantitative results, more details and fewer
artifacts occur in DCMAN results than in SCMAN.

Conv-LSTM.  Bidirectional Conv-LSTM modules are used to encode temporal
correlation among multiple frames. However, features from different frames may
also be combined directly with a multi-channel CNN, named CNN early fusion. To
quantify the contribution from the Conv-LSTM module, we design a new network
with CNN early fusion [37]. The result with Conv-LSTM and trivial CNN early
fusion are shown in the ‘Ours S+T+C’ and ‘’Ours$-$LSTM’ columns in Tab. I. It
is observed that on our data the Bi-ConvLSTM module increases PSNR by 0.17.

The design of the deep network.  The superior performance of our final model
can be attributed to both better network design, additional information from
the NIR channel, and adopting a physical-process-based noise model.
Considering that most state-of-the-art denoising methods are trained with
Gaussian or other simple noise models, we conduct an experiment here to verify
the advantages of the proposed network itself. Specifically, we train a DCMAN
and SCMAN model on additive white Gaussian noise (AWGN) and name it DCMAN_G
and SCMAN_G. Compared to state-of-the-art deep learning-based video denoising
methods, TOFlow [44] and FastDVDnet [3], retrained with synthetic data by
adding AWGN on our dataset, both our single-channel and bi-channel networks
perform much better, as shown in Tab. IV.

TABLE IV: Test results on image synthesized with Additive White Gaussian Noise (AWGN) compared with two deep learning based method TOFlow and FastDVDnet. DCMAN_G and SCMAN_G are our methods with and without NIR channel trained on simulated AWGN noise. Our model outperform those methods on both noise levels. PSNR(dB) / SSIM |  $\sigma=20$ |  $\sigma=40$
---|---|---
TOFlow | 31.792 / 0.941 | 29.328 / 0.902
FastDVDnet | 32.509 / 0.947 | 29.605 / 0.911
SCMAN_G | 32.833 / 0.952 | 30.259 / 0.923
DCMAN_G | 32.988 / 0.954 | 30.418 / 0.926

Integrated noise model.  Our training data are synthesized by a physical-
process-based integrated noise model. To validate the necessity of using such
a noise model incorporating various noise sources, we compare the performance
of DCMAN_G (Gaussian noise) and DCMAN (calibrated noise following the
integrated noise) on real low light images. The results are shown in Fig. 10,
from which one can see noticeable performance improvements brought by learning
from data generated by an integrated noise model with calibrated model
parameters.

Figure 10: Comparison of results by DCMAN_G (trained from data with Gaussian
noise) and DCMAN (trained with synthetic data following physical CMOS noise
model) on real low light videos.

###  6.4 Experiments on Real Captured Low Light Videos

Figure 11: Our RGB and NIR reconstruction results on a dark video frame
captured by our setup at night. Figure 12: Our RGB reconstruction results on
real low light video frames and comparison with state-of-the-arts. Here some
regions with large performance diversity are highlighted with red ellipses.

We test our approach on real low-light videos captured with our dual sensor
camera, the results are shown in Fig. 11-12, and please refer to supplementary
data for video demonstration. The images are captured at dark hours with
illumination of less than 1 lux. From the image in Fig. 11, one can see that
under such low illuminations the input raw images are severely deteriorated by
noise, with structure details washed out and color deviating largely. After
reconstruction, we can achieve striking quality improvement. The flat regions
(e.g., road, sky), structures (e.g., road lamp, driving line, car, traffic
sign), and textured regions (e.g., trees, shrubbery) are all recovered with
high quality. The high performance indicates that our approach is quite
promising in assisting night traffic surveillance and auto driving.

Comparatively, three previously published state-of-the-arts show limited
performance in such challenging scenarios, as shown in Fig. 12. The results
from V-BM4D and TOFlow tend to suffer from residual noise, while FastDVDnet
causes over-smoothness and cannot recover thin structures. Besides, all these
three methods have apparent color distortion, while our DCMAN removes purplish
color bias and preserves image details well. For example, in the 1st row, we
can notice that the details in the dark background are recovered, such as the
tree trunks and the shrubs. In the 2nd and 4th scenes, the details are
reconstructed at a much higher quality, e.g., the textures on the marble, and
the bike on the side of the road. Besides, the proposed approach produces high
performance on the dense striped patterns (the railing in the 5th row) that
are prone to oversmoothness and the highlight region (car wheel in the 6th
row). Our results are also advantageous in the texture regions, with more
details and less color distortion, such as the lawn regions in the 3rd scene.
Overall, the proposed method is of stronger noise suppression, better
structure preservation, and higher color fidelity. The superior performance is
mainly attributed to the additional information from the NIR channel and
successful exploration of the cross-channel and multi-frame prior, including
both the network design and effective training strategy.

##  7 Summary and Discussions

This paper reports a dual-channel computational dark videography approach with
superior performance than state-of-the-arts, by collecting more photons
optically and introducing cross channel priors computationally. The high
performance benefits from technical contributions in three folds: (i) We
design a compact RGB+NIR dual-sensor camera to largely increase the collected
photons of a conventional RGB camera by additionally capturing a NIR
wavelength band, and two paths are with pixel-wise registration and high
precision synchronization. (ii) We then propose a Dual-Channel Multi-frame
Attention Network (DCMAN) to enhance the RGB and NIR video pair in an end-to-
end manner by exploring the spatial, temporal, and spectral information
jointly. (iii) The model can be learned effectively from synthetic data
produced by superimposing noise on our high quality NIR+RGB dataset following
an integrated physical-process-based noise model and adapted to different
sensors.

So far, the application of our approach is somewhat limited due to its high
computing complexity. For a 360 $\times$ 640 video frame, we need around 700ms
to recover the high-quality video with an NVIDIA GeForce GTX 1080Ti GPU. A low
weight model is under development currently. In the hardware, one can replace
the Bayer pattern filter with a customized one with NIR transmittance to
collect more photons in dark environments, and developing corresponding
reconstruction algorithms is a worth studying topic. The approach can also be
extended further to other wavelength bands for different imaging scenarios or
tasks, such as UV, mid-NIR, etc.

## Acknowledgments

Our code will be soon available at https://github.com/ jarrycyx/dual-channel-
low-light-video.git

## References

  * [1] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 9446–9454.
  * [2] M. Claus and J. van Gemert, “Videnn: Deep blind video denoising,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 0–0.
  * [3] M. Tassano, J. Delon, and T. Veit, “Fastdvdnet: Towards real-time deep video denoising without flow estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1354–1363.
  * [4] C. Godard, K. Matzen, and M. Uyttendaele, “Deep burst denoising,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 538–554.
  * [5] D. Krishnan and R. Fergus, “Dark flash photography,” _ACM Trans. Graph._ , vol. 28, no. 3, p. 96, 2009.
  * [6] J. Jiang, X. Feng, F. Liu, Y. Xu, and H. Huang, “Multi-spectral RGB-NIR image classification using double-channel CNN,” _IEEE Access_ , vol. 7, pp. 20 607–20 613, 2019.
  * [7] P. Zhang, W. Liu, Y. Lei, and H. Lu, “Hyperfusion-Net: Hyper-densely reflective feature fusion for salient object detection,” _Pattern Recognition_ , vol. 93, pp. 521–533, 2019.
  * [8] Y. Zhang, X. Guo, J. Ma, W. Liu, and J. Zhang, “Beyond brightening low-light images,” _International Journal of Computer Vision_ , vol. 129, no. 4, pp. 1013–1037, Apr. 2021.
  * [9] C. Tian, Y. Xu, L. Fei, and K. Yan, “Deep learning for image denoising: A survey,” _arXiv:1810.05052 [cs]_ , Oct. 2018.
  * [10] R. S.R and N. P. Kavya, “Noise reduction in video sequences the state of art and the technique for motion detection,” _International Journal of Computer Applications_ , vol. 58, no. 8, pp. 31–36, Nov. 2012.
  * [11] F. Jin, P. Fieguth, and L. Winger, “Wavelet video denoising with regularized multiresolution motion estimation,” _EURASIP Journal on Advances in Signal Processing_ , vol. 2006, pp. 1–11, 2006.
  * [12] V. Zlokolica, A. Pizurica, and W. Philips, “Wavelet-domain video denoising based on reliability measures,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 16, no. 8, pp. 993–1007, 2006.
  * [13] J. C. Brailean, R. P. Kleihorst, S. Efstratiadis, A. K. Katsaggelos, and R. L. Lagendijk, “Noise reduction filters for dynamic image sequences: A review,” _Proceedings of the IEEE_ , vol. 83, no. 9, pp. 1272–1292, 1995.
  * [14] A. Buades, J.-L. Lisani, and M. Miladinović, “Patch-based video denoising with optical flow estimation,” _IEEE Transactions on Image Processing_ , vol. 25, no. 6, pp. 2573–2586, 2016.
  * [15] A. Buades, B. Coll, and J.-M. Morel, “Denoising image sequences does not require motion estimation,” in _IEEE Conference on Advanced Video and Signal Based Surveillance_. IEEE, 2005, pp. 70–74.
  * [16] K. Dabov, A. Foi, and K. Egiazarian, “Video denoising by sparse 3D transform-domain collaborative filtering,” in _European Signal Processing Conference_ , 3, pp. 145–149.
  * [17] M. Lebrun, “An analysis and implementation of the BM3D image denoising method,” _Image Processing On Line_ , vol. 2012, pp. 175–213, 2012.
  * [18] M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian, “Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms,” _IEEE Trans Image Process_ , vol. 21, no. 9, pp. 3952–66, Sep. 2012.
  * [19] P. Arias and J.-M. Morel, “Video denoising via empirical Bayesian estimation of space-time patches,” _Journal of Mathematical Imaging and Vision_ , vol. 60, no. 1, pp. 70–93, 2018.
  * [20] M. Tassano, J. Delon, and T. Veit, “Dvdnet: A fast network for deep video denoising,” in _IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 1805–1809.
  * [21] B. Mildenhall, J. T. Barron, J. Chen, D. Sharlet, R. Ng, and R. Carroll, “Burst denoising with kernel prediction networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2502–2510.
  * [22] X. Chen, L. Song, and X. Yang, “Deep rnns for video denoising,” in _Applications of Digital Image Processing XXXIX_ , vol. 9971. International Society for Optics and Photonics, 2016, p. 99711T.
  * [23] W. Wang, X. Chen, C. Yang, X. Li, X. Hu, and T. Yue, “Enhancing low light videos by exploring high sensitivity camera noise,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 4111–4119.
  * [24] F. Zhu, G. Chen, and P.-A. Heng, “From noise modeling to blind image denoising,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 420–429.
  * [25] M. Mäkitalo and A. Foi, “Noise parameter mismatch in variance stabilization, with an application to poisson–gaussian noise estimation,” _IEEE Transactions on Image Processing_ , vol. 23, no. 12, pp. 5348–5359, 2014.
  * [26] G. E. Healey and R. Kondepudy, “Radiometric CCD camera calibration and noise estimation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 16, no. 3, pp. 267–276, 1994.
  * [27] Y. Tsin, V. Ramesh, and T. Kanade, “Statistical calibration of CCD imaging process,” in _Proceedings Eighth IEEE International Conference on Computer Vision_ , vol. 1. IEEE, 2001, pp. 480–487.
  * [28] K. Wei, Y. Fu, J. Yang, and H. Huang, “A physics-based noise formation model for extreme low-light raw denoising,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2758–2767.
  * [29] H. Yue, C. Cao, L. Liao, R. Chu, and J. Yang, “Supervised raw video denoising with a benchmark dataset on dynamic scenes,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2301–2310.
  * [30] T. Y. Han, D. H. Kim, S. H. Lee, and B. C. Song, “Infrared image super-resolution using auxiliary convolutional neural network and visible image under low-light conditions,” _Journal of Visual Communication and Image Representation_ , vol. 51, pp. 191–200, 2018.
  * [31] Z. Chen, X. Wang, and R. Liang, “RGB-NIR multispectral camera,” _Optics Express_ , vol. 22, no. 5, pp. 4985–4994, 2014.
  * [32] S. Zhuo, X. Zhang, X. Miao, and T. Sim, “Enhancing low light images using near infrared flash images,” in _IEEE International Conference on Image Processing_. IEEE, 2010, pp. 2537–2540.
  * [33] D. Sugimura, T. Mikami, H. Yamashita, and T. Hamamoto, “Enhancing color images of extremely low light scenes based on RGB/NIR images acquisition with different exposure times,” _IEEE Trans Image Process_ , vol. 24, no. 11, pp. 3586–97, Nov. 2015.
  * [34] X. Hu, F. Heide, Q. Dai, and G. Wetzstein, “Convolutional sparse coding for RGB+NIR imaging,” _IEEE Trans Image Process_ , vol. 27, no. 4, pp. 1611–1625, Apr. 2018.
  * [35] X. Niu, H. Han, S. Shan, and X. Chen, “VIPL-HR: A multi-modal database for pulse estimation from less-constrained face video,” in _Asian Conference on Computer Vision_. Springer, 2018, pp. 562–576.
  * [36] M. Brown and S. Süsstrunk, “Multi-spectral SIFT for scene category recognition,” in _CVPR_. IEEE, 2011, pp. 177–184.
  * [37] J. Caballero, C. Ledig, A. Aitken, A. Acosta, J. Totz, Z. Wang, and W. Shi, “Real-time video super-resolution with spatio-temporal networks and motion compensation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 4778–4787.
  * [38] Y. Lei, D. Peng, P. Zhang, Q. Ke, and H. Li, “Hierarchical paired channel fusion network for street scene change detection,” _IEEE Trans Image Process_ , vol. 30, pp. 55–67, 2021.
  * [39] X. Meng, X. Deng, S. Zhu, X. Zhang, and B. Zeng, “A robust quality enhancement method based on joint spatial-temporal priors for video coding,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 31, no. 6, pp. 2401–2414, Jun. 2021.
  * [40] S. Zha, W. Jin, C. He, Z. Chen, G. Si, and Z. Jin, “Detecting of Overshooting Cloud Tops via Himawari-8 Imagery Using Dual Channel Multiscale Deep Network,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 14, pp. 1654–1664, 2020.
  * [41] S. Xingjian, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo, “Convolutional LSTM network: A machine learning approach for precipitation nowcasting,” in _Advances in Neural Information Processing Systems_ , 2015, pp. 802–810.
  * [42] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2015, pp. 234–241.
  * [43] T. Wilkinson and A. Brun, “Semantic and verbatim word spotting using deep neural networks,” in _International Conference on Frontiers in Handwriting Recognition (ICFHR)_. IEEE, 2016, pp. 307–312.
  * [44] T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman, “Video enhancement with task-oriented flow,” _International Journal of Computer Vision_ , vol. 127, no. 8, pp. 1106–1125, 2019.
  * [45] M. Konnik and J. Welsh, “High-level numerical simulations of noise in CCD and CMOS photosensors: Review and tutorial,” _arXiv preprint arXiv:1412.4031_ , 2014.
  * [46] D. G. Lowe, “Object recognition from local scale-invariant features,” in _Proceedings of the IEEE International Conference on Computer Vision_ , vol. 2. IEEE, 1999, pp. 1150–1157.
  * [47] A. Ranjan and M. J. Black, “Optical flow estimation using a spatial pyramid network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 4161–4170.

|  Yuxiao Cheng is an undergraduate student in the Department of Automation,
Tsinghua University, Beijing, China. His research interests include
computational imaging and computer vision.
---|---
|  Runzhao Yang received his B.E. degree in the School of Electrical
Engineering and Automation from Wuhan University, Wuhan, China, in 2020. He is
pursuing his Ph.D. degree in the Department of Automation at Tsinghua
University, Beijing, China. His research interests include computer vision,
data compression, and machine learning.
---|---
|  Zhihong Zhang received the B.Eng. degree from the School of Electronics
Engineering, Xidian University, Xi’an, China, in 2019. He is currently
pursuing the Ph.D. degree in the Department of Automation, Tsinghua
University. His research interests include computational imaging, computer
vision, and machine learning.
---|---
|  Jinli Suo received the BS degree in computer science from Shandong
University, Shandong, China, in 2004 and the Ph.D. degree from the Graduate
University of Chinese Academy of Sciences, Beijing, China, in 2010. She is
currently an associate professor with the Department of Automation, Tsinghua
University, Beijing, China. Her research interests include computer vision,
computational photography, and statistical learning. She is an Associate
Editor for the IEEE Transactions on Computational Imaging.
---|---
|  Qionghai Dai received the M.S. and Ph.D. degrees in computer science and
automation from Northeastern University, Shenyang, China, in 1994 and 1996,
respectively. He is currently a Professor with the Department of Automation,
an Adjunct Professor with the School of Life Sciences, Tsinghua University,
and an Academician with the Chinese Academy of Engineering. He has authored or
coauthored more than 200 conference and journal papers and two books. His
research interests include computational photography and microscopy, computer
vision and graphics, and intelligent signal processing. He is an Associate
Editor for the Journal of Visual Communication and Image Representation, the
IEEE Transactions on Neural Networks and Learning Systems, and the IEEE
Transactions on Image Processing.
---|---