# MetaSCI: Scalable and Adaptive Reconstruction for Video Compressive Sensing

Zhengjue Wang† Hao Zhang† Ziheng Cheng  Bo Chen*
National Laboratory of Radar Signal Processing  Xidian University  Xian  China
{zhengjuewang,zhanghao_xidian}@163.com,zhcheng<EMAIL_ADDRESS>Xin Yuan*
Bell Labs  NJ USA
<EMAIL_ADDRESS>
###### Abstract

To capture high-speed videos using a two-dimensional detector, video snapshot
compressive imaging (SCI) is a promising system, where the video frames are
coded by different masks and then compressed to a snapshot measurement.
Following this, efficient algorithms are desired to reconstruct the high-speed
frames, where the state-of-the-art results are achieved by deep learning
networks. However, these networks are usually trained for specific small-scale
masks and often have high demands of training time and GPU memory, which are
hence not flexible to $i$) a new mask with the same size and $ii$) a larger-
scale mask. We address these challenges by developing a Meta Modulated
Convolutional Network for SCI reconstruction, dubbed MetaSCI. MetaSCI is
composed of a shared backbone for different masks, and light-weight meta-
modulation parameters to evolve to different modulation parameters for each
mask, thus having the properties of fast adaptation to new masks (or systems)
and ready to scale to large data. Extensive simulation and real data results
demonstrate the superior performance of our proposed approach. Our code is
available at https://github.com/xyvirtualgroup/MetaSCI-CVPR2021.

†††Equal contributions. * Corresponding authors.

##  1 Introduction

(a) Adaptation to a new mask

(b) MetaSCI for large-scale SCI reconstruction by fast adaptation

Figure 1: Illustration of the fast adaptation property of MetaSCI, as training
on $256\times 256$ measurements and adapted to (a) a new $256\times 256$
measurement compressed by different masks, and (b) a $768\times 1792$
measurement compressed by large masks. The results in (a) is evaluated on a
benchmark data, Kobe [15]. Compared with BIRNAT [4], a SOTA deep model,
MetaSCI realizes much faster adaptation. In (b), we decompose the large-scale
SCI reconstruction task into 21 sub-tasks without overlap. MetaSCI is trained
on 4 sub-tasks, and then fast adapted to the others. After the adaptation
stage, MetaSCI can realize real-time reconstruction using feed-forward
mapping. By contrast, PnP [39] and GAP-TV [37], the only two existing methods
suitable for large-scale SCI, need iterative optimization for every
measurement.

High-speed video imaging system is desirable in our daily life, which often
faces challenges in capturing and saving high-dimensional (HD) data, , high
memory, bandwidth and power demand. Inspired by compressive sensing (CS) [2,
5] techniques, video snapshot compressive imaging (SCI) has attracted much
attention, which enjoys the advantages of low memory, low bandwidth, low power
and potentially low cost [39]. The video SCI system constructs a pipeline of
an optical hardware encoder and a software decoder [38]. In one exposure time,
the optical encoder modulates the HD data via dynamic masks and then
compresses multiple high-speed frames into a two-dimensional (2D) snapshot
measurement. The decoder, on the other hand, aims to recover or reconstruct
the high-speed video frames using advanced algorithms. This paper focuses on
video SCI reconstruction. More specifically, we develop a fast adaptive
decoder motivated by meta learning, which is flexible to different systems and
ready to scale to large data.

In general, a desirable decoder should have good properties in $i$) high
fidelity (often with PSNR $\geq$30dB) and $ii$) fast recovery. With more than
a decade of development, more and more optical encoders are constructed [9,
16, 25, 26, 27, 28, 29, 30, 31, 40, 41], which arouses more considerations for
a practical algorithm. As a new encoding system is built, one may often wonder
whether a well learned decoder can be fast adapted to this new encoder. A
relatively simple scenario is that the physical masks are changed but with the
same spatial size such as $256\times 256$ pixels. Even more challenging, the
physical masks are scaled to higher spatial dimension such as $512\times 512$
or even up to $2048\times 2048$ pixels when a high resolution system is built.
Therefore, the properties of $iii$) fast adaptation and $iv$) scalability are
also desired to make video SCI system being practical. However, existing SCI
reconstruction networks often lack fast adaptation, , not flexible. As the
mask changes, the network has to be re-trained, which again needs a long time,
as shown in Fig. 1a.

Bearing all these four-aspect concerns in mind, we propose a meta modulated
convolutional network for flexible SCI reconstruction, dubbed MetaSCI. With
the following contributions and appealing properties, MetaSCI will pave the
way of applying deep learning methods to large-scale SCI in our daily life.

  * •

We discuss the fast adaptation problem of video SCI in real applications,
which have not been studied before, especially for deep learning models.

  * •

To realize fast adaptation, we construct a multi-encoding-system training
regime and build MetaSCI. MetaSCI consists of a shared backbone for different
systems, and light-weight meta-modulation parameters that can evolve to
different modulation parameters for each individual system.

  * •

A hybrid learning algorithm is developed to train the network, with standard
gradient descend for the shared backbone and meta updates for meta parameters.

  * •

Besides achieving competing performance and fast adaptation on widely used
small-scale video benchmarks, using the attractive property of fast
adaptation, MetaSCI is the first deep model to perform real-time large-scale
video reconstruction as shown in Fig. 1b.

##  2 Related Work

Existing SCI reconstruction models can be divided into two categories,
optimization based ones and deep learning based ones.

The optimization based methods consider the ill-posed SCI reconstruction task
as a regularized optimization problem, and usually solve it via iterative
algorithms. Among traditional methods [1, 15, 34, 35, 37, 36], DeSCI in [15]
achieves the state-of-the-art (SOTA) reconstruction performance. Yet, it takes
more than one hour to recover a $256\times 256\times 8$ video, which precludes
DeSCI to be applied to a higher-dimensional scenario. To accelerate the
optimization speed, Yuan et al.. [39, 43] develop a plug-and-play (PnP)
framework. By plugging a pre-trained denoising network into every iteration of
the optimization, PnP achieves comparable PSNR scores and much faster
inference speed. Generally, the major drawback of optimization based models is
that one needs to perform iterative optimization for every new-coming snapshot
measurement (sometimes needs to fine-tune the parameters), making it time-
consuming as processing large amounts of snapshots, as shown in Fig. 1b.

Compared with optimization based methods, the most attractive property of deep
learning based ones is the real-time test speed. In [3, 4, 11, 12, 17, 18, 19,
20, 27], various end-to-end networks are proposed, whose input is the
measurement (and optionally masks) and the output is the recovered video,
optimized by reconstruction loss such as mean square error (MSE) and
adversarial loss [7]. Among them, the BIRNAT in [4] achieves comparable or
even superior PSNR than DeSCI. Even though, based on the bidirectional RNN
structure, BIRNAT needs a significant amount of GPU memories, scarcely
possible for large-scale SCI. More importantly, as the mask changes, the
network has to be re-trained, as shown in Fig. 1a. Unfortunately, the training
time is on the order of days or weeks.

Table 1: Property of typical SCI reconstruction algorithms.

Algorithm | Fidelity | Inference speed | Fast adaptation | Scalability
---|---|---|---|---
DeSCI [15] | High | Low | Middle | Low
PnP [39] | High | Middle | Middle | High
BIRNAT [4] | High | High | Low | Low
MetaSCI | High | High | High | High

To make more evident comparisons, Table 1 summarizes the property of some
typical SCI reconstruction algorithms111Note that though DeSCI and PnP can
easily adapt to new systems, one usually needs to fine-tune the parameters to
achieve good results. Therefore, we dub their adaptation to ‘middle’.,
highlighting our motivations to propose a scalable and adaptive reconstruction
model for video SCI.

Figure 2: Pipelines of video SCI (optical encoder) and our proposed MetaSCI
(software decoder) for reconstruction. A video scene, represented by a
sequence of images, is coded ($\odot$) by binary masks and then integrated
($\sum$) over time on a camera, forming a single-frame compressed measurement.
At the training stage, by minimizing MSE between recovered and original
videos, we use $M$ masks (tasks) to train our proposed network including
shared base parameters and meta parameters, where meta parameters evolve to
modulation parameters, modulating (see Fig. 3 for details) the shared base
parameters for respective tasks. After training, we get well-learned base and
meta parameters. During testing, we have a new mask, fixing the base
parameters, we only need to run a few iterations to update well-learned meta
parameters to fit our new mask, realizing fast adaptation.

##  3 Problem Statement

We first provide the mathematical model of video SCI, and then discuss the
importance of fast adaptation in SCI.

###  3.1 Mathematical Model of Video SCI

We assume that a scene with $B$ high-speed frames $\\{{\bf
X}_{b}\\}_{b=1}^{B}\in\mathbb{R}^{d_{x}\times d_{y}}$ is modulated by the
coding patterns (masks) $\\{{\bf C}_{b}\\}_{b=1}^{B}\in\mathbb{R}^{d_{x}\times
d_{y}}$. As shown in Fig. 2, a measurement ${\bf Y}\in\mathbb{R}^{d_{x}\times
d_{y}}$ is then obtained by

$\textstyle{\bf Y}=\sum_{b=1}^{B}{\bf X}_{b}\odot{\bf C}_{b}+{\bf Z},$ (1)

where $\odot$ and ${\bf Z}\in\mathbb{R}^{d_{x}\times d_{y}}$ denote the matrix
element-wise product and noise, respectively. Denoting the vectorization
operation on a matrix as $\mbox{Vec}(\cdot)$, (1) can be re-written as

$\boldsymbol{y}={\bf H}\boldsymbol{x}+\boldsymbol{z},$ (2)

where $\boldsymbol{y}=\mbox{Vec}({\bf Y})\in\mathbb{R}^{d_{x}d_{y}}$ and
$\boldsymbol{z}=\mbox{Vec}({\bf Z})\in\mathbb{R}^{d_{x}d_{y}}$.
Correspondingly, the video $\boldsymbol{x}$ is expressed as

$\boldsymbol{x}=\textstyle\left[\mbox{Vec}({\bf
X}_{1})^{T},\cdots,\mbox{Vec}({\bf
X}_{B})^{T}\right]^{T}\in\mathbb{R}^{d_{x}d_{y}B}.$ (3)

Unlike traditional CS methods [5, 23] using a dense sensing matrix, in SCI,
the sensing matrix ${\bf H}\in\mathbb{R}^{d_{x}d_{y}\times d_{x}d_{y}B}$ is
sparse and can be represented as a concatenation of $B$ diagonal matrices as
${\bf H}=\left[{\bf D}_{1},\cdots,{\bf D}_{B}\right]$, where $\left\\{{\bf
D}_{b}=\mbox{diag}\left(\mbox{Vec}({\bf
C}_{b})\right)\in\mathbb{R}^{d_{x}d_{y}\times d_{x}d_{y}}\right\\}_{b=1}^{B}$.
Therefore, the compression ratio of SCI is $B$, and the theoretical analysis
have been studies in [13].

###  3.2 Why Is Fast Adaptation Important?

Recently, deep-learning based SCI reconstruction models have achieved
promising results in both fidelity and inference speed [4, 22]. To be
concrete, given the mask matrices $\\{{\bf C}_{b}\\}_{b=1}^{B}$ and a training
dataset $\mathcal{Q}$ containing $N$ data pairs $\\{{\bf Y},{\bf X}\\}$, a
deep reconstruction network $f_{\boldsymbol{\Theta}}$ is learned by optimizing

$\textstyle\min_{\boldsymbol{\Theta}}\sum_{\\{{\bf Y},{\bf
X}\\}\in\mathcal{Q}}d\left({\bf X},f_{\boldsymbol{\Theta}}({\bf Y},\\{{\bf
C}_{b}\\}_{b=1}^{B})\right),$ (4)

where $d(\cdot)$ is a distance metric such as MSE, and $\boldsymbol{\Theta}$
denotes all trainable parameters. For every testing measurement, unlike
optimization based methods performing iterative inferences, deep-learning
methods [4, 22, 27] directly use the well-learned model
$f_{\boldsymbol{\Theta}}$ to perform real-time reconstruction.

However, as the system changes, , masks $\\{{\bf C}_{b}\\}_{b=1}^{B}$ become
$\\{{\bf C}^{{}^{\prime}}_{b}\\}_{b=1}^{B}$, the network
$f_{\boldsymbol{\Theta}}$ trained on $\\{{\bf C}_{b}\\}_{b=1}^{B}$ does not
work on the new system, as shown in Fig. 1a. Yet, in real applications, the
mask can be changed due to different practical reasons, such as mask drifting
or illumination change [16]. If ${\bf C}^{{}^{\prime}}_{b}$ has the same size
as ${\bf C}_{b}$, one still has the opportunity to re-train the model, at the
expense of time. Whereas, if ${\bf C}^{{}^{\prime}}_{b}$ has larger spatial
size than ${\bf C}_{b}$, directly re-training the network often faces enormous
challenges in both GPU memory and training time.

For some video SCI systems, , CACTI system [16], the measurement, mask and
signal are spatially decoupled. This provides a potential solution to the
large-scale video reconstruction problem, , decomposing the video into
multiple blocks and reconstructing them separately and in parallel. For
example, as shown in Fig. 1b, one can decompose a video with $768\times 1792$
pixels into 21 $256\times 256$ non-overlapping blocks222The number of blocks
increases with the video spatial size.. However, with limited computational
resources, for existing deep learning based methods, the training time for
these 21 models is on the order of months or even years. Therefore, if we
build a model with the fast adaptation property, we can also apply it to
realize large-scale video reconstruction via spatial decomposition. That is to
say, we train the model only on a small number of sub-masks and then perform
fast adaptation to all sub-masks, which will be discussed specifically in Sec.
4.5. In this manner, our model would be more flexible in real applications.

##  4 MetaSCI

MetaSCI considers data from multiple encoding systems as training. During
testing, with fast adaptation, the model is flexible to different systems
(with different masks) and ready to scale to large data. Toward this end, we
start by presenting the task definition of MetaSCI. Based on it, we construct
a CNN backbone and introduce meta parameters into the network architecture.
Then, a hybrid learning algorithm is developed to train the network.

###  4.1 Task Definition of MetaSCI

Since the optical encoder in SCI can typically be modeled accurately, it is
widely used to train the reconstruction network on simulated data and test on
real data [4]. In order to consider the adaptive capability into the network
learning, we simulate a multi-system training scenario, which can be realized
in multiple ways. For example, we can randomly generate $M$ sets of masks or
crop sub-masks from large-scale masks. Each set of masks corresponds to an
encoding system. Without loss of generality, we denote the masks of these $M$
encoding systems as $\\{{\bf C}^{m}_{b}\\}_{b=1,m=1}^{B,M}$, all of which have
the same spatial size. Correspondingly, for a ground-truth video ${\bf X}$,
its snapshot measurement in the $m$-th encoding system is denoted as ${\bf
Y}^{m}$.

For a specific task $m$, the reconstruction network aims to output a recovered
video $\widehat{{\bf X}}^{m}$ given the inputs ${\bf Y}^{m}$ and $\\{{\bf
C}^{m}_{b}\\}_{b=1}^{B}$. Jointly considering all these $M$ tasks, our goal is
to learn a reconstruction network, whose parameters can be fast adapted to a
specific/new task via a few numbers of updates on this task.

###  4.2 Fully Convolutional Network Backbone

We start by considering the network structure for a single task. Here we take
the $m$-th task as an example.

To enhance the motion information in the measurement and achieve better fusion
between the measurement and masks, an energy normalization method [4] is
applied, with the normalized measurement $\overline{{\bf Y}}^{m}$ derived by

$\textstyle\overline{{\bf Y}}^{m}={\bf
Y}^{m}\oslash\left(\sum\nolimits_{b=1}^{B}{\bf
C}_{b}^{m}\right),\vspace{-1mm}$ (5)

where $\oslash$ represents the matrix element-wise division. Considering the
fact that $\\{\overline{{\bf Y}}^{m}\odot{\bf C}_{b}^{m}\\}_{b=1}^{B}$ can be
used to approximate the real modulated frames $\\{{\bf X}_{k}\odot{\bf
C}_{b}^{m}\\}_{b=1}^{B}$ [4], we fuse all current visual information by

$\textstyle{\bf O}^{m}=\left[\overline{{\bf Y}}^{m}\,,\overline{{\bf
Y}}^{m}\odot{\bf C}_{1}^{m},...,\overline{{\bf Y}}^{m}\odot{\bf
C}_{B}^{m}\right]_{3},\vspace{-1mm}$ (6)

where ${\bf O}_{m}\in\mathbb{R}^{d_{x}\times d_{y}\times(B+1)}$, and
$[\cdot]_{3}$ denotes concatenation along the third dimension. Then, we
consider ${\bf O}^{m}$ as the input of our proposed fully CNN backbone
$\mathcal{F}_{\boldsymbol{\Theta}_{1}}(\cdot)$ and achieve reconstruction by

$\widehat{{\bf X}}^{m}=\mathcal{F}_{\boldsymbol{\Theta}_{1}}({\bf
O}^{m})\in\mathbb{R}^{d_{x}\times d_{y}\times
B},~{}m=1,\cdots,M,\vspace{-1mm}$ (7)

where $\boldsymbol{\Theta}_{1}$ denotes the network parameters. As shown in
Fig. 3a, the network has three Res-blocks [8] for better information
transmission, each of which has six convolutional layers and one residual
connection. Detailed network structure is given in the supplementary material
(SM). Different from BIRNAT [4] that builds a type of CNN only to reconstruct
the first frame and employs a time- and memory-consuming RNN to sequentially
reconstruct the rest frames, our proposed fully CNN is able to generate all
frames quickly in a memory efficient way.

Inspired by the success of meta learning in various fast adaptation tasks [6,
33], we hope to use meta learning to train our proposed backbone, so that it
can realize adaptive reconstruction. A straight forward way is to consider
every parameter in $\boldsymbol{\Theta}_{1}$ as a meta-parameter, which is
then optimized via a meta-learning algorithm [6]. During the test, with the
well-learned meta-parameters as initialization, the backbone is adapted to a
new system (or task) by a few numbers of updates. Though effective, suppose we
have $\tilde{M}$ new systems (tasks), the backbone would evolve as $\tilde{M}$
models, that means the volume of parameters would increase $\tilde{M}$ times.
A specific example is discussed in Sec. 4.5. This is undesired for the SCI
application on edge devices.

To address this challenge, we develop meta-modulation parameters based on our
proposed fully CNN. Specifically, the following methods are employed to
balance the speed and memory.

  * •

We assume the backbone is shared by all (training and testing) tasks, and
introduce task-specific parameters, whose volume is much smaller than the
backbone, as shown in Fig. 3b.

  * •

We propose the rank-one convolutional kernel modulation to save the memory
during training and also for fast adaptation, with details as follows.

(a) Fully convolutional network backbone

(b) Rank-one convolutional kernel modulation by meta parameters

Figure 3: (a) The brief structure of our proposed fully CNN backbone for SCI
reconstruction, where the numbers denote the number of kernels at each layer.
(b) Illustration of our proposed rank-1 convolutional kernel modulation, which
is applied to every convolutional layer in (a).

###  4.3 Rank-one Kernel Modulation

The basic operation of the backbone in (7) is the convolution written as ${\bf
Q}={\bf G}*{\bf W}+{\bf E}$, where ${\bf G}\in\mathbb{R}^{I_{x}\times
I_{y}\times C_{in}}$ is the input feature map, ${\bf
Q}\in\mathbb{R}^{I_{x}\times I_{y}\times C_{out}}$ is the output feature map,
${\bf W}\in\mathbb{R}^{k_{x}\times k_{y}\times C_{in}\times C_{out}}$ is the
convolutional kernel, $*$ denotes the convolutional operation, and ${\bf
E}\in\mathbb{R}^{C_{out}}$ is the bias. For multiple-task purpose, hereby,
instead of learning different kernels for different tasks, we utilize a task-
specific matrix $\boldsymbol{\Gamma}_{m}\in\mathbb{R}^{C_{in}\times C_{out}}$
to modulate the convolutional kernel ${\bf W}$. This is

${\bf Q}={\bf G}*({\bf W}\odot\boldsymbol{\Gamma}_{m})+{\bf E},\quad
m=1,\cdots,M,$ (8)

where $\odot$ denotes element-wise product with appropriate broadcasting.
Furthermore, to decrease the volume of the task-specific parameters, we assume
$\boldsymbol{\Gamma}_{m}$ is a rank-1 matrix, denoted by

$\boldsymbol{\Gamma}_{m}=\boldsymbol{\alpha}_{m}^{T}\boldsymbol{\beta}_{m},\quad\boldsymbol{\alpha}_{m}\in\mathbb{R}^{1\times
C_{in}},\quad\boldsymbol{\beta}_{m}\in\mathbb{R}^{1\times C_{out}}.$ (9)

Compared with $\boldsymbol{\Gamma}_{m}$ having $C_{in}\times C_{out}$
parameters, $\boldsymbol{\alpha}_{m}$ and $\boldsymbol{\beta}_{m}$ only have
$(C_{in}+C_{out})$ parameters333For our proposed fully CNN backbone with
$B=8$, the total number of meta-modulation parameters is $8.7$k with our
developed rank-1 kernel modulation, while it will be $537.8$k with full-matrix
modulation..

To perform fast adaptation from training to testing tasks, rather than
learning $\boldsymbol{\alpha}_{m}$ and $\boldsymbol{\beta}_{m}$ directly, we
construct meta-modulation parameters
$\tilde{\boldsymbol{\alpha}}\in\mathbb{R}^{1\times C_{in}}$ and
$\tilde{\boldsymbol{\beta}}\in\mathbb{R}^{1\times C_{out}}$ correspondingly,
such that $\tilde{\boldsymbol{\alpha}}$ and $\tilde{\boldsymbol{\beta}}$ can
evolve to $\\{\boldsymbol{\alpha}_{m}\\}_{m=1}^{M}$ and
$\\{\boldsymbol{\beta}_{m}\\}_{m=1}^{M}$, respectively, via a few number of
updates.

Denoting all meta-modulation parameters as $\boldsymbol{\Theta}_{2}$, all
learnable parameters of MetaSCI are
$\boldsymbol{\Theta}=\\{\boldsymbol{\Theta}_{1},\boldsymbol{\Theta}_{2}\\}$.
Next, we introduce how to train MetaSCI and then perform fast adaptation.

###  4.4 Training and Fast Adaptation

Training: Supposing that each task (corresponding to one set of masks) has $N$
measurements denoted by $\mathcal{T}=\left\\{{\bf Y}_{n}^{m},\\{{\bf
X}_{n,b}^{m},{\bf C}^{m}_{b}\\}_{b=1}^{B}\right\\}_{n=1,m=1}^{N,M}$ for $M$
tasks. The training objective is to minimize the MSE loss between real and
recovered videos:

$\textstyle\mathcal{L(\boldsymbol{\Theta};\mathcal{T})}=\sum\nolimits_{m=1,n=1,b=1}^{M,N,B}\left\|{\bf
X}_{n,b}^{m}-\widehat{{\bf X}}_{n,b}^{m}\right\|_{2}.$ (10)

Different from general network parameters, $\boldsymbol{\Theta}$ contains both
shared base parameters $\boldsymbol{\Theta}_{1}$ and meta-modulation
parameters $\boldsymbol{\Theta}_{2}$. Therefore, the adaptation of
$\boldsymbol{\Theta}_{2}$ to task-specific parameters should be considered
into the learning algorithm. Specifically, in each iteration, we sample a
mini-batch data $\mathcal{T}_{pre,m}$ for the $m$-th task, and run $U$ (often
small, set to $3$ in our experiments) iterations of standard gradient descend
to obtain task-specific parameters $\boldsymbol{\Theta}_{2,m}^{\prime}$. Then,
based on another mini-batch data $\mathcal{T}_{obj,m}$ and parameters
$\\{\boldsymbol{\Theta}_{1},\boldsymbol{\Theta}_{2,m}^{\prime}\\}$, we use
Adam [14] to update both $\boldsymbol{\Theta}_{1}$ and
$\boldsymbol{\Theta}_{2}$. Note that, the task-specified parameters
$\boldsymbol{\Theta}_{2,m}^{\prime}$ evolved from $\boldsymbol{\Theta}_{2}$ is
actually a function w.r.t. $\boldsymbol{\Theta}_{2}$. Thus, rather than
updating
$\\{\boldsymbol{\Theta}_{1},\\{\boldsymbol{\Theta}_{2,m}^{\prime}\\}_{m=1}^{M}\\}$,
we update $\\{\boldsymbol{\Theta}_{1},\boldsymbol{\Theta}_{2}\\}$ during
training. The entire training process is exhibited in Algorithm 1.

Algorithm 1 Training algorithm of MetaSCI

0: Step size $\beta$, number of inner-loop $U$.

1: Randomly initialize
$\boldsymbol{\Theta}=\\{\boldsymbol{\Theta}_{1},\boldsymbol{\Theta}_{2}\\}$.

2: while not done do

3: for all training tasks $m=1,\cdots,M$ do

4: Sample a mini-batch of data $\mathcal{T}_{pre,m}=\\{{\bf Y}_{n}^{m},\\{{\bf
X}_{n,b}^{m},{\bf C}^{m}_{b}\\}_{b=1}^{B}\\}_{n=1}^{N_{1}}$.

5: Initialize
$\boldsymbol{\Theta}_{2,m}^{\prime}\leftarrow\boldsymbol{\Theta}_{2}$

6: for $u=1$ to $U$ do

7:
$\mathcal{L}_{1}=\mathcal{L}(\\{\boldsymbol{\Theta}_{1},\boldsymbol{\Theta}_{2,m}^{\prime}\\};\mathcal{T}_{pre,m})$;

8:
$\boldsymbol{\Theta}_{2,m}^{\prime}\leftarrow\boldsymbol{\Theta}_{2,m}^{\prime}-\beta\nabla_{\boldsymbol{\Theta}_{2,m}^{\prime}}\mathcal{L}_{1}$.

9: end for

10: Sample another mini-batch of data $\mathcal{T}_{obj,m}$.

11: end for

12: Obtain loss:
$\mathcal{L}_{2}=\sum_{m}\mathcal{L}(\\{\boldsymbol{\Theta}_{1},\boldsymbol{\Theta}_{2,m}^{\prime}\\};\mathcal{T}_{obj,m})$.

13: Update all parameters
$\boldsymbol{\Theta}=\\{\boldsymbol{\Theta}_{1},\boldsymbol{\Theta}_{2}\\}$
via$\boldsymbol{\Theta}\leftarrow\boldsymbol{\Theta}-Adam[\mathcal{L}_{2}]$.

14: end while

Fast adaptation: After training, we obtain the well-learned base parameters
$\boldsymbol{\Theta}_{1}$ and meta-modulation parameters
$\boldsymbol{\Theta}_{2}$. During testing, aiming for fast adaptation, we fix
$\boldsymbol{\Theta}_{1}$ and only update $\boldsymbol{\Theta}_{2}$ for a new
task (with new masks).

Given $\tilde{M}$ new tasks, for task $\tilde{m}=1,\dots,\tilde{M}$, the model
parameters is represented by
$\\{\boldsymbol{\Theta}_{1},\boldsymbol{\Theta}_{2,\tilde{m}}\\}$.
$\boldsymbol{\Theta}_{2,\tilde{m}}$ is firstly initialized by
$\boldsymbol{\Theta}_{2}$. In every iteration, after sampling a mini-batch
data $\mathcal{T}_{ad,\tilde{m}}$, we use Adam to update
$\boldsymbol{\Theta}_{2,\tilde{m}}$. Algorithm 2 exhibits how to perform fast
adaptation.

Hereby, we describe the difference between our proposed algorithm (Algorithms
1 and 2) and [6]. In [6], all parameters are regraded as meta parameters,
while our propped model exploits a small number of meta parameters to modulate
the base backbone. As a result, as shown in Fig. 2 and Algorithm 2, when
performing fast adaptation, all $\tilde{M}$ tasks share a large fixed (thus no
need to update) backbone $\boldsymbol{\Theta}_{1}$ but with a small number of
task-specific parameters
$\\{\boldsymbol{\Theta}_{2,\tilde{m}}\\}_{\tilde{m}=1}^{\tilde{M}}$ that need
to be updated from meta parameters $\boldsymbol{\Theta}_{2}$. Considering the
application of MetaSCI for large-scale SCI reconstruction by fast adaptation
discussed in the following Sec. 4.5, this distinct property makes MetaSCI
memory efficient and can be performed in parallel for multiple new tasks.

Algorithm 2 Fast adaptation of MetaSCI

0:
$\boldsymbol{\Theta}=\\{\boldsymbol{\Theta}_{1},\boldsymbol{\Theta}_{2}\\}$:
the well-learned base and meta-modulation parameters from Algorithm 1.

1: Initialize $\boldsymbol{\Theta}_{2,\tilde{m}}$ = $\boldsymbol{\Theta}_{2}$,
$\tilde{m}=1,\cdots,\tilde{M}$ .

2: while not done do

3: for all testing tasks $\tilde{m}=1,\cdots,\tilde{M}$ do

4: Sample a mini-batch of data $\mathcal{T}_{ad,\tilde{m}}=\\{{\bf
Y}_{n}^{\tilde{m}},\\{{\bf X}_{n,b}^{\tilde{m}},{\bf
C}_{b}^{\tilde{m}}\\}_{b=1}^{B}\\}_{n=1}^{N_{2}}$.

5: end for

6: Obtain loss:
$\sum_{\tilde{m}}\mathcal{L}(\\{\boldsymbol{\Theta}_{1},\boldsymbol{\Theta}_{2,\tilde{m}}^{\prime}\\};\mathcal{T}_{ad,\tilde{m}})$

7: Update $\\{\boldsymbol{\Theta}_{2,\tilde{m}}\\}_{\tilde{m}=1}^{\tilde{M}}$
via Adam in parallel.

8: end while

###  4.5 Efficient Reconstruction for Large-scale SCI Using MetaSCI

Existing deep models, such as BIRNAT [4], are difficult to handle large-scale
videos due to limited GPU memory. Interestingly, besides adapting the model
for new masks quickly, fast adaptation makes MetaSCI feasible for large-scale
SCI reconstruction. Without loss of generality, in this section, we take
videos of size $2048\times 2048\times B$ as an example to illustrate the
efficient reconstruction using MetaSCI.

Basically, we can spatially decompose a $2048\times 2048\times B$ video into
64 non-overlapping $256\times 256\times B$ sub-videos444Actually, in SM, we
discuss the effect of overlapping size in recovering large-scale videos,
showing that appropriate overlapping will bring better reconstruction,
especially at boundaries., corresponding to 64 small tasks. At training stage,
we only randomly choose $M$ ($M\ll 64$) sub-videos as training tasks and
construct a training set to learn parameters $\boldsymbol{\Theta}$ by
Algorithm 1. And then, we use Algorithm 2 to perform fast adaptation for the
other $\tilde{M}$ ($\tilde{M}+M=64$) sub-videos. Finally, after aggregating
all $\tilde{M}+M$ sub-videos, we realize large-scale video SCI recovery in an
end-to-end manner.

In our experiments, to accelerate the training process, we find $M=4$ is
enough to achieve a good $\boldsymbol{\Theta}$. As a result, when adapting it
to other $\tilde{M}$ masks, since all tasks share a large fixed backbone with
a small number of learnable task-specific modulation parameters, it is
efficient to perform parallel fast adaptation on all $\tilde{M}$ masks with a
much smaller model555If we regard all parameters as meta parameters as [6], it
needs at least (without overlapping) $213.18$M parameters to recover videos
with size of $2048\times 2048\times 8$. Our proposed MetaSCI only needs
$3.89$M parameters.; please refer to Fig. 1 for an example.

##  5 Experiments

We evaluate the proposed MetaSCI on both simulated data [4, 15, 17] and real
data captured by the SCI cameras [27, 30]. Considering that most existing
methods can only work on small-scale datasets (benchmarks), for comprehensive
comparison, we first evaluate MetaSCI on the simulated benchmark datasets,
discussing the reconstruction performance and the adaptation speed to
different masks. Further, we show the appealing results of MetaSCI on large-
scale simulated and real datasets, while most existing methods fail because of
the limitations of memory or speed.

###  5.1 Implementation Details of MetaSCI

The high-speed training videos are acquired using the code provided by [4],
containing about 26,000 videos of size $256\times 256\times B$ cropped from
the DAVIS2017 dataset [24] with data augmentation. To simulate a multi-task
scenario, we randomly generate four different sets of binary masks of size
$256\times 256$, , $M=4$. This means that each task has about 26,000 training
samples. For Algorithm 1, the number of training epochs is set as 100; the
number of inner-loop $U$ is set as 3; step size $\beta=10^{-5}$; we use the
default Adam setting [14]. During the adaptation by Algorithm 2, we only need
4 epochs to achieve good results. For MetaSCI in reconstructing the large-
scale video, we decompose the videos into overlapping $256\times 256\times B$
sub-videos using a spatial interval of 128 pixels.

###  5.2 Counterparts and Evaluation Metrics

We compare our method with three representative optimization-based ones,
including GAP-TV [37], DeSCI [15], and PnP-FFDNet [39], and two deep-learning
based ones, including U-net [27] and BIRNAT [4]. Among them, DeSCI and BIRNAT
have achieved the SOTA performance.

Both peak-signal-to-noise ratio (PSNR) and structural similarity (SSIM) [32]
are employed to evaluate the performance. The adaptation and inference speed
are assessed by adaptation time and test time, respectively. The scalability
is evaluated on large-scale scenes.

Table 2: The results of PSNR in dB (left entry in each cell), SSIM (right
entry in each cell), and running time per measurement in seconds on $256\times
256\times 8$ simulation benchmarks. The results above double lines denote the
testing mask appears at training stage while the results under double lines
denote the test mask does NOT appear at the training stage. ‘FT’ and ‘AD’
represent ‘fine-tuning’ and ‘adaptation’, respectively. The AD time is
determined when training is converged. Note that, with the same AD time as
MetaSCI, BIRNAT only achieves about 27dB in average.

Algorithm | Kobe | Traffic | Runner | Drop | Aerial | Vehicle | Average | AD Time | Test Time
---|---|---|---|---|---|---|---|---|---
GAP-TV | 26.45, 0.845 | 20.89, 0.715 | 28.81, 0.909 | 34.74, 0.970 | 25.05, 0.828 | 24.82, 0.838 | 26.79, 0.858 | 0 | 4.2
DeSCI | 33.25, 0.952 | 28.72, 0.925 | 38.76, 0.969 | 43.22, 0.993 | 25.33, 0.860 | 27.04, 0.909 | 32.72, 0.935 | 0 | 6180
PnP-FFDNet | 30.50, 0.926 | 24.18, 0.828 | 32.15, 0.933 | 40.70, 0.989 | 25.27, 0.829 | 25.42, 0.849 | 29.70, 0.892 | 0 | 3.0
U-net | 29.79, 0.807 | 24.62, 0.840 | 34.12, 0.947 | 36.56, 0.949 | 27.18, 0.869 | 26.43, 0.882 | 29.45, 0.882 | 0 | 0.0312
BIRNAT | 32.71, 0.950 | 29.33, 0.942 | 38.70, 0.976 | 42.28, 0.992 | 28.99, 0.927 | 27.84, 0.927 | 33.31, 0.951 | 0 | 0.16
MetaSCI | 30.12, 0.907 | 26.95, 0.888 | 37.02, 0.967 | 40.61, 0.985 | 28.31, 0.904 | 27.33, 0.906 | 31.72, 0.926 | 0 | 0.025
U-net-w/o-FT | 20.13, 0.221 | 16.63, 0.165 | 23.15, 0.765 | 23.02, 0.502 | 22.85, 0.527 | 20.94, 0.486 | 21.12, 0.443 | 0 | 0.0312
U-net-w-FT | 29.81, 0.811 | 24.70, 0.843 | 34.31, 0.951 | 36.51, 0.950 | 26.98, 0.860 | 26.54, 0.890 | 29.81, 0.884 | 2013 | 0.0312
BIRNAT-w/o-FT | 21.45, 0.243 | 18.55, 0.186 | 26.67, 0.796 | 26.12, 0.539 | 24.22, 0.559 | 22.29, 0.509 | 23.22, 0.387 | 0 | 0.16
BIRNAT-w-FT | 32.73, 0.952 | 29.30, 0.941 | 38.83, 0.975 | 42.16, 0.989 | 28.93, 0.923 | 27.48, 0.907 | 33.23, 0.948 | 20376 | 0.16
MetaSCI | 30.10, 0.905 | 27.01, 0.891 | 37.01, 0.969 | 40.52, 0.982 | 28.35, 0.904 | 27.22, 0.901 | 31.70, 0.925 | 1004 | 0.025

Table 3: PSNR, SSIM and running time per measurement in seconds on large-scale
simulation data with $B$=8. Note that BIRNAT fails in these large-scale
datasets due to high demanding of GPU memory, as well as other deep learning
methods.

Size | Algorithm | Beauty | Bosphorus | HoneyBee | Jockey | ShakeNDry | Average | Test Time
---|---|---|---|---|---|---|---|---
$512\times 512$ | GAP-TV | 32.13, 0.857 | 29.18, 0.934 | 31.40, 0.887 | 31.01, 0.940 | 32.52, 0.882 | 31.25, 0.900 | 44.67
PnP-FFDNet | 30.70, 0.855 | 35.36, 0.952 | 31.94, 0.872 | 34.88, 0.955 | 30.72, 0.875 | 32.72, 0.902 | 14.22
MetaSCI | 35.10, 0.901 | 38.37, 0.950 | 34.27, 0.913 | 36.45, 0.962 | 33.16, 0.901 | 35.47, 0.925 | 0.12
Size | Algorithm | Beauty | Jockey | ShakeNDry | ReadyGo | YachtRide | Average | Test Time
$1024\times 1024$ | GAP-TV | 33.59, 0.852 | 33.27, 0.971 | 33.86, 0.913 | 27.49, 0.948 | 24.39, 0.937 | 30.52, 0.924 | 178.11
PnP-FFDNet | 32.36, 0.857 | 35.25, 0.976 | 32.21, 0.902 | 31.87, 0.965 | 30.77, 0.967 | 32.49, 0.933 | 52.47
MetaSCI | 35.23, 0.929 | 37.15, 0.978 | 36.06, 0.939 | 33.34, 0.973 | 32.68, 0.955 | 34.89, 0.955 | 0.59
Size | Algorithm | City | Kids | Lips | Night | RiverBank | Average | Test Time
$2048\times 2048$ | GAP-TV | 21.27, 0.902 | 26.05, 0.956 | 26.46, 0.890 | 26.81, 0.875 | 27.74, 0.848 | 25.67, 0.894 | 764.75
PnP-FFDNet | 29.31, 0.926 | 30.01, 0.966 | 27.99, 0.902 | 31.18, 0.891 | 30.38, 0.888 | 29.77, 0.915 | 205.62
MetaSCI | 32.63, 0.930 | 32.31, 0.965 | 30.90, 0.895 | 33.86, 0.893 | 32.77, 0.902 | 32.49, 0.917 | 2.38

###  5.3 Results on Simulated Small-scale Benchmarks

The widely used six videos [17], including Kobe, Traffic, Runner, Drop,
Vehicle, and Aerial, are considered, with spatial size $256\times 256$. We
follow the simulation setup in [4, 15, 39], compressing $B$=8 frames into a
snapshot measurement. Besides the standard testing (training and testing have
the same set of masks) as usual, we also conduct experiments to evaluate the
performance of adaptation to a new set of masks during the test.

Training and testing have the same masks. The quantitative results on the
standard testing are listed in Table 2 (the upper half). In general, compared
with optimization based ones (in group 1) that need iterative inferences, end-
to-end deep learning based methods (in group 2) have much ($\geq$150$\times$)
faster testing speed. Though BIRNAT have achieved the SOTA fidelity
performance, the recurrent structure (a sequential forward and then backward
frame-by-frame reconstruction) limits its testing speed. MetaSCI achieves
comparable PSNR and SSIM scores with BIRNAT and a superior ($\geq$6$\times$
shorter) testing speed. Some reconstructed frames are shown in the left of
Fig. 4. It can be seen that MetaSCI is able to recover fine details and have
little artifacts.

Training and testing have different masks. As mentioned in Sec. 3.2 and shown
in Fig. 1a, deep-learning based SCI reconstruction models are sensitive to
masks. When the masks change, directly employing the network trained on the
original masks to perform testing often results in poor recovery (see
U-net-w/o-FT and BIRNAT-w/o-FT in Tab. 2). Although one can fine-tune the
model of U-net or BIRNAT on the testing masks, it is time-consuming. With half
adaptation time, MetaSCI achieves superior fidelity performance than U-net-w-
FT. Using about $1/20$ adaptation time, MetaSCI achieves comparable fidelity
performance with BIRNAT-w-FT.

Figure 4: Reconstructed frames on multi-scale simulation benchmarks.

###  5.4 Results on Simulated Large-scale Data

Since limited researches provide large-scale data for SCI, we create a large-
scale benchmark, including $512\times 512\times 8$, $1024\times 1024\times 8$,
and $2048\times 2048\times 8$ videos cropped from the Ultra Video Group (UVG)
dataset [21]. Each scale have five different videos, and the compression ratio
is $B=8$. Details are provided in the SM.

As the increase of spatial size, existing deep-learning based models often
face challenges in both GPU memory and time expense. Among all existing works,
only GAP-TV, PnP-FFDNet, and our proposed MetaSCI can be applied to large-
scale SCI reconstruction. The quantitative comparisons are provided in Tab. 3.
Obviously, MetaSCI demonstrates the superior PSNR and SSIM scores, and also
testing speed. Specifically, MetaSCI outperforms GAP-TV 5.13dB in average, and
outperforms PnP-FFDNet 2.62dB in average. Further, MetaSCI accelerates the
test speed over 100$\times$ than GAP-TV and PnP-FFDNet. As selected
reconstructed frames shown in the right of Fig. 4, MetaSCI provides sharper
boundaries than GAP-TV and less artifacts than PnP-FFDNet. See recovered
videos in the SM.

###  5.5 Results on Real Data

We now apply the proposed MetaSCI to real data. Different from simulation,
real measurements always have noise inside and thus make the reconstruction
more challenging. In addition to this, the mask is not ideal due to nonuniform
illumination and other reasons. To handle large scale problems, as mentioned
before, for the sake of fast reconstruction, deep learning models need a huge
amount of training data and time plus GPU memory. Our proposed MetaSCI, on the
other hand, provides a solution to train the model on small-scale and
efficiently scale to large data as shown in Fig. 1b. We use the real data
captured by SCI cameras [27, 30], with the mask controlled by a digital
micromirror device. The real data Domino and Water Balloon are snapshot
measurements of size $512\times 512$, encoding $B=10$ frames. The UCF snapshot
measurement has $850\times 1100$ pixels [30], which is also a compression of
10 frames.

The results compared with other algorithms are shown in Fig. 5, where we can
see that the recovered frames of MetaSCI show sharper borders and finer
details (recovered videos are in SM). Meanwhile, on the large scale
($850\times 1100$) UCF data, the time cost of MetaSCI is the least after
adaptation. It is worth noting that, in this scale data, BIRNAT fails due to
the large memory requirement and we thus only compare with GAP-TV, DeSCI and
PnP-FFDnet. DeSCI takes more than 4 hours while MetaSCI only needs 0.51
seconds.

Figure 5: Reconstructed frames on real data, Domino, Water Balloon, and UCF.
The test time on the challenging UCF is: 0.51s (MetaSCI), 300.84s (GAP-TV),
15045s (DeSCI), 12.52s (PnP-FFDNet).

##  6 Conclusions and Future Work

Fast and high quality reconstruction plays a pivot role in inverse problems.
This paper takes one step further to devise flexible reconstruction networks
considering the application of video snapshot compressive imaging, where masks
can be different for different systems and thus it is desirable to have fast,
accurate and flexible reconstruction algorithms. Towards this end, we have
developed a meta modulated convolutional network for SCI reconstruction,
dubbed MetaSCI, which contains a shared fully CNN as the backbone but being
modulated by a small number of meta parameters for different tasks (masks). In
this manner, MetaSCI has accomplished fast adaptation using our proposed
algorithms. MetaSCI is the first end-to-end deep model to realize high-quality
and efficient large-scale SCI recovery.

The proposed MetaSCI is actually a backbone-free framework, in which different
deep models can be used. Furthermore, how to realize fast adaptation among
different compression ratios is also a practical problem, which would be
considered in the future.

In addition to the video SCI considered in this paper, our proposed MetaSCI
can also be used in the spectral SCI reconstruction where large-scale real
data are available [19, 20, 42] and different networks [10, 22, 44] have been
proposed for the inversion. MetaSCI is able to speed up the training process
and reduce the model size.

## References

  * [1] José M. Bioucas-Dias and Mário A. T. Figueiredo.  A new twist: Two-step iterative shrinkage/thresholding algorithms for image restoration.  IEEE Transactions on Image processing, 16(12):2992–3004, 2007.
  * [2] E. J. Candès, J. Romberg, and T. Tao.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information.  IEEE Transactions on Information Theory, 52(2):489–509, Feb 2006\.
  * [3] Z. Cheng, B. Chen, G. Liu, H. Zhang, R. Lu, Z. Wang, and X. Yuan.  Memory-efficient network for large-scale video compressive sensing.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
  * [4] Ziheng Cheng, Ruiying Lu, Zhengjue Wang, Hao Zhang, Bo Chen, Ziyi Meng, and Xin Yuan.  Birnat: Bidirectional recurrent neural networks with adversarial training for video snapshot compressive imaging.  In European Conference on Computer Vision, 2020.
  * [5] David L. Donoho.  Compressed sensing.  IEEE Transactions on information theory, 52(4):1289–1306, 2006\.
  * [6] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In ICML, 2017.
  * [7] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [9] Yasunobu Hitomi, Jinwei Gu, Mohit Gupta, Tomoo Mitsunaga, and Shree K. Nayar.  Video from a single coded exposure photograph using a learned over-complete dictionary.  In 2011 International Conference on Computer Vision, pages 287–294. IEEE, 2011.
  * [10] T. Huang, W. Dong, X. Yuan, J. Wu, , and G. Shi.  Deep gaussian scale mixture prior for spectral compressive imaging.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
  * [11] Michael Iliadis, Leonidas Spinoulas, and Aggelos K. Katsaggelos.  Deep fully-connected networks for video compressive sensing.  Digital Signal Processing, 72:9–18, 2018.
  * [12] Michael Iliadis, Leonidas Spinoulas, and Aggelos K. Katsaggelos.  Deepbinarymask: Learning a binary mask for video compressive sensing.  Digital Signal Processing, 96:102591, 2020.
  * [13] Shirin Jalali and Xin Yuan.  Snapshot compressed sensing: performance bounds and algorithms.  IEEE Transactions on Information Theory, 65(12):8005–8024, 2019\.
  * [14] Diederik P. Kingma and Jimmy Lei Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [15] Yang Liu, Xin Yuan, Jinli Suo, David J. Brady, and Qionghai Dai.  Rank minimization for snapshot compressive imaging.  IEEE transactions on pattern analysis and machine intelligence, 41(12):2990–3006, 2019.
  * [16] Patrick Llull, Xuejun Liao, Xin Yuan, Jianbo Yang, David Kittle, Lawrence Carin, Guillermo Sapiro, and David J. Brady.  Coded aperture compressive temporal imaging.  Optics express, 21(9):10526–10545, 2013.
  * [17] Jiawei Ma, Xiao-Yang Liu, Zheng Shou, and Xin Yuan.  Deep tensor admm-net for snapshot compressive imaging.  In Proceedings of the IEEE International Conference on Computer Vision, pages 10223–10232, 2019.
  * [18] Ziyi Meng, Shirin Jalali, and Xin Yuan.  Gap-net for snapshot compressive imaging.  arXiv: 2012.08364, December 2020.
  * [19] Z Meng, J Ma, and X Yuan.  End-to-end low cost compressive spectral imaging with spatial-spectral self-attention.  In European Conference on Computer Vision (ECCV)(August 2020), 2020\.
  * [20] Ziyi Meng, Mu Qiao, Jiawei Ma, Zhenming Yu, Kun Xu, and Xin Yuan.  Snapshot multispectral endomicroscopy.  Optics Letters, 45(14):3897–3900, 2020.
  * [21] Alexandre Mercat, Marko Viitanen, and Jarno Vanne.  Uvg dataset: 50/120fps 4k sequences for video codec analysis and development.  In Proceedings of the 11th ACM Multimedia Systems Conference, pages 297–302, 2020.
  * [22] Xin Miao, Xin Yuan, Yunchen Pu, and Vassilis Athitsos.  lambda-net: Reconstruct hyperspectral images from a snapshot measurement.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4058–4068. IEEE, 2019.
  * [23] Deanna Needell and Roman Vershynin.  Signal recovery from incomplete and inaccurate measurements via regularized orthogonal matching pursuit.  IEEE Journal of selected topics in signal processing, 4(2):310–316, 2010.
  * [24] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  arXiv preprint arXiv:1704.00675, 2017.
  * [25] Mu Qiao, Xuan Liu, and Xin Yuan.  Snapshot spatial–temporal compressive imaging.  Opt. Lett., 45(7):1659–1662, Apr 2020.
  * [26] Mu Qiao, Xuan Liu, and Xin Yuan.  Snapshot temporal compressive microscopy using an iterative algorithm with untrained neural networks.  Opt. Lett., 2021.
  * [27] Mu Qiao, Ziyi Meng, Jiawei Ma, and Xin Yuan.  Deep learning for video compressive sensing.  APL Photonics, 5(3):030801, 2020.
  * [28] Dikpal Reddy, Ashok Veeraraghavan, and Rama Chellappa.  P2c2: Programmable pixel compressive camera for high speed imaging.  In CVPR 2011, pages 329–336. IEEE, 2011.
  * [29] Y. Sun, X. Yuan, and S. Pang.  High-speed compressive range imaging based on active illumination.  Optics Express, 24(20):22836–22846, Oct 2016.
  * [30] Y. Sun, X. Yuan, and S. Pang.  Compressive high-speed stereo imaging.  Optics Express, 25(15):18182–18190, 2017.
  * [31] T.-H. Tsai, P. Llull, X. Yuan, D. J. Brady, and L. Carin.  Spectral-temporal compressive imaging.  Optics Letters, 40(17):4054–4057, Sep 2015.
  * [32] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [33] Zhengjue Wang, Bo Chen, Ruiying Lu, Hao Zhang, Hongwei Liu, and Pramod K. Varshney.  Fusionnet: An unsupervised convolutional variational network for hyperspectral and multispectral image fusion.  IEEE Transactions on Image Processing, 29:7565–7577, 2020.
  * [34] Jianbo Yang, Xuejun Liao, Xin Yuan, Patrick Llull, David J. Brady, Guillermo Sapiro, and Lawrence Carin.  Compressive sensing by learning a gaussian mixture model from measurements.  IEEE Transactions on Image Processing, 24(1):106–119, 2015.
  * [35] Jianbo Yang, Xin Yuan, Xuejun Liao, Patrick Llull, David J. Brady, Guillermo Sapiro, and Lawrence Carin.  Video compressive sensing using gaussian mixture models.  IEEE Transactions on Image Processing, 23(11):4863–4878, 2014.
  * [36] P. Yang, L. Kong, X. Liu, X. Yuan, and G. Chen.  Shearlet enhanced snapshot compressive imaging.  IEEE Transactions on Image Processing, 29:6466–6481, 2020.
  * [37] Xin Yuan.  Generalized alternating projection based total variation minimization for compressive sensing.  In 2016 IEEE International Conference on Image Processing (ICIP), pages 2539–2543. IEEE, 2016.
  * [38] X. Yuan, D. J. Brady, and A. K. Katsaggelos.  Snapshot compressive imaging: Theory, algorithms, and applications.  IEEE Signal Processing Magazine, 38(2):65–88, 2021.
  * [39] Xin Yuan, Yang Liu, Jinli Suo, and Qionghai Dai.  Plug-and-play algorithms for large-scale snapshot compressive imaging.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1447–1457, 2020.
  * [40] Xin Yuan, Patrick Llull, Xuejun Liao, Jianbo Yang, David J Brady, Guillermo Sapiro, and Lawrence Carin.  Low-cost compressive sensing for color video and depth.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3318–3325, 2014.
  * [41] X. Yuan and S. Pang.  Structured illumination temporal compressive microscopy.  Biomedical Optics Express, 7:746–758, 2016.
  * [42] X. Yuan, T. Tsai, R. Zhu, P. Llull, D. Brady, and L. Carin.  Compressive hyperspectral imaging with side information.  IEEE Journal of Selected Topics in Signal Processing, 9(6):964–976, Sep. 2015.
  * [43] Xin Yuan, Jinli Suo Yang Liu, Frédo Durand, and Qionghai Dai.  Plug-and-play algorithms for video snapshot compressive imaging.  arXiv: 2101.04822, Jan 2021.
  * [44] Siming Zheng, Yang Liu, Ziyi Meng, Mu Qiao, Zhishen Tong, Xiaoyu Yang, Shensheng Han, and Xin Yuan.  Deep plug-and-play priors for spectral snapshot compressive imaging.  Photon. Res., 9(2):B18–B29, Feb 2021.

##  Appendix A Network Structure

In this section, we provide the detailed network structure of the proposed
fully convolutional network backbone in Section 4.2, with the illustrations
shown in Fig. 6 and Table 4.

Figure 6: The brief structure of our proposed fully CNN backbone for SCI reconstruction, where the numbers denote the number of kernels at each layer. Table 4: Network Structure of the proposed fully CNN backbone for SCI reconstruction. | Module | Stride | Kernel Size | Output Size
---|---|---|---|---
$\times 1$ | Conv. + Leaky Relu | 1 | $5\times 5\times(B+1)\times 32$ | $256\times 256\times 32$
Conv. + Leaky Relu | 1 | $3\times 3\times 32\times 64$ | $256\times 256\times 64$
Conv. + Leaky Relu | 1 | $1\times 1\times 64\times 64$ | $256\times 256\times 64$
Conv. + Leaky Relu | 2 | $3\times 3\times 64\times 128$ | $128\times 128\times 128$
$\times 3$ | Conv. + Leaky Relu | 1 | $3\times 3\times 128\times 128$ | $128\times 128\times 128$
Conv. + Leaky Relu | 1 | $1\times 1\times 128\times 128$ | $128\times 128\times 128$
Conv. + Leaky Relu | 1 | $3\times 3\times 128\times 128$ | $128\times 128\times 128$
Conv. + Leaky Relu | 1 | $3\times 3\times 128\times 128$ | $128\times 128\times 128$
Summation | – | – | $128\times 128\times 128$
Conv. + Leaky Relu | 1 | $3\times 3\times 128\times 128$ | $128\times 128\times 128$
| Conv. + Leaky Relu | 1 | $1\times 1\times 128\times 128$ | $128\times 128\times 128$
$\times 1$ | dConv. + Leaky Relu | 1 | $3\times 3\times 64\times 128$ | $256\times 256\times 64$
Conv. + Leaky Relu | 1 | $3\times 3\times 64\times 32$ | $256\times 256\times 32$
Conv. + Leaky Relu | 1 | $1\times 1\times 32\times 16$ | $256\times 256\times 16$
Conv. + Leaky Relu | 1 | $3\times 3\times 16\times B$ | $256\times 256\times B$

##  Appendix B Simulated Large-scale Benchmark

Since limited researches provide large-scale data for SCI, we create a large-
scale benchmark, including $512\times 512\times 8$, $1024\times 1024\times 8$,
and $2048\times 2048\times 8$ videos cropped from the Ultra Video Group (UVG)
dataset [21]. Each scale have five different video sequences, and the
compression ratio is $B=8$. We download the original UVG dataset from
http://ultravideo.cs.tut.fi/#testsequences, with the parameters listed in
Table 5. Since the motion speed varies among different videos, for each video,
we sequentially choose 64 frames at an interval of ‘Step’ frame(s). For
example, for the video ‘Beauty’ of spatial size $512\times 512$, we extract 1
frame from every 5 frames in the original video, and totally obtain $64$
frames.

Table 5: Details of the Simulated Large-scale Benchmark.

Spatial Size | Name | Original data | Processing
---|---|---|---
Resolution | Bit depth | Format | Container | Step | Num. of frames | Num. of measurements
$512\times 512$ | Beauty | $1920\times 1080$ | 8 | YUV | RAW | 5 | 64 | 8
Bosphorus | $1920\times 1080$ | 8 | YUV | RAW | 5 | 64 | 8
HoneyBee | $1920\times 1080$ | 8 | YUV | RAW | 1 | 64 | 8
Jockey | $1920\times 1080$ | 8 | YUV | RAW | 1 | 64 | 8
ShakeNDry | $1920\times 1080$ | 8 | YUV | RAW | 3 | 64 | 8
$1024\times 1024$ | Beauty | $1920\times 1080$ | 8 | YUV | RAW | 5 | 64 | 8
Jockey | $1920\times 1080$ | 8 | YUV | RAW | 5 | 64 | 8
ReadyGo | $1920\times 1080$ | 8 | YUV | RAW | 5 | 64 | 8
ShakeNDry | $1920\times 1080$ | 8 | YUV | RAW | 3 | 64 | 8
YachtRide | $1920\times 1080$ | 8 | YUV | RAW | 9 | 64 | 8
$2048\times 2048$ | City | $3840\times 2160$ | 8 | YUV | RAW | 8 | 64 | 8
Kids | $3840\times 2160$ | 8 | YUV | RAW | 5 | 64 | 8
Lips | $3840\times 2160$ | 8 | YUV | RAW | 5 | 64 | 8
Night | $3840\times 2160$ | 8 | YUV | RAW | 5 | 64 | 8
RiverBank | $3840\times 2160$ | 8 | YUV | RAW | 5 | 64 | 8

##  Appendix C Discussion on the Overlapping Size in Large-scale Video
Reconstruction

Table 6: Number of sub-tasks on large-scale video reconstruction with different overlapping sizes. Overlapping size | Scale
---|---
$512\times 512$ | $1024\times 1024$ | $2048\times 2048$
0 | 4 | 16 | 64
64 | 9 | 25 | 100
128 | 9 | 49 | 196
192 | 25 | 100 | 400
Table 7: PSNR on large-scale video reconstruction with different overlapping sizes. Overlapping size | Scale
---|---
$512\times 512$ | $1024\times 1024$ | $2048\times 2048$
0 | 34.81 | 34.18 | 31.87
64 | 35.25 | 34.66 | 32.28
128 | 35.47 | 34.89 | 32.49
192 | 35.50 | 34.87 | 32.51