# Bitstream-corrupted Video Recovery:
A Novel Benchmark Dataset and Method

Tianyi Liu1 Kejun Wu1111https://blog.quindorian.org/2013/03/fixing-rtsp-
stream-corruption.html/ Yi Wang2 Wenyang Liu1 Kim-Hui
Yap1222https://www.youtube.com/watch?appd̄esktop&vM̄7wZxk7yPeQ Lap-Pui Chau2
1School of EEE, Nanyang Technological University, Singapore
2Department of EEE, The Hong Kong Polytechnic University, Hong Kong
{liut0038, wang1241<EMAIL_ADDRESS>
{kejun.wu<EMAIL_ADDRESS><EMAIL_ADDRESS>
Equal first contributionCorresponding authorYi Wang was with NTU when this
research was conducted and is currently with Hong Kong PolyU.

###### Abstract

The past decade has witnessed great strides in video recovery by specialist
technologies, like video inpainting, completion, and error concealment.
However, they typically simulate the missing content by manual-designed error
masks, thus failing to fill in the realistic video loss in video communication
(e.g., telepresence, live streaming, and internet video) and multimedia
forensics. To address this, we introduce the bitstream-corrupted video (BSCV)
benchmark, the first benchmark dataset with more than 28,000 video clips,
which can be used for bitstream-corrupted video recovery in the real world.
The BSCV is a collection of 1) a proposed three-parameter corruption model for
video bitstream, 2) a large-scale dataset containing rich error patterns,
multiple corruption levels, and flexible dataset branches, and 3) a plug-and-
play module in video recovery framework that serves as a benchmark. We
evaluate state-of-the-art video inpainting methods on the BSCV dataset,
demonstrating existing approaches’ limitations and our framework’s advantages
in solving the bitstream-corrupted video recovery problem. The benchmark and
dataset are released at https://github.com/LIUTIGHE/BSCV-Dataset.

##  1 Introduction

As Cisco’s report [10] shows, video traffic is expected to account for 82% of
all internet traffic by 2022, making it the commonest multimedia type on the
internet. However, due to unreliable channels and physical damage of the
storage medium, videos are vulnerable to generated errors in the case of
packet loss during transmission and context corruption during compression and
storage [5]. Meanwhile, malicious attacks on the video decoder ecosystem may
cause the risk of severe damage to video bitstreams [45]. Therefore, bitstream
damage during compression, storage, and transmission chains is a common and
crucial problem. The various types of damage factors yield different
corruption degrees and error patterns in decoded frames, which are
irreversible and unpredictable. Recovering the video content in corrupted
bitstreams is of vital importance but beset with difficulties.

Figure 1: Summary of the corruption pattern in video recovery problem.
Compared with the simulated video corruption in existing inpainting or error
concealment reseach, our dataset contains various realistic corruption
patterns including (1) block artifacts (artfs.), (2) color artifacts, (3)
duplication artifacts, (4) misalignment, (5) texture loss, (6) trailing
artifacts, which is closer to the corrupted videos1,2,3 in real world.

Researchers have been dedicated to video recovery at the encoding,
transmission, and decoding stages. Reed-Solomon codes [50] add redundant
information during the encoding stage to enable error correction for the
receiver. Checksum [14] is used in the transmission process to detect errors
and initiate re-transmission when errors are detected. These methods introduce
additional requirements and inflexibility in hardware design and system
reliability, and they cannot deal with long sequence loss in bitstreams. More
research has focused on visual-based solutions in the decoding stage due to
its intuitive and easy access to images, such as error concealment,
completion, frame interpolation, and video inpainting.

33footnotetext: https://www.youtube.com/watch?vl̄66kIS_-UmI

Typically, the error concealment is to mitigate the effect of errors on video
quality [25]. However, the error patterns are generally simulated by error
masks of the slice or block shapes directly on the decoded video content. The
fixed and simple error simulation limits the application scenarios, as the
error patterns in realistic scenarios are neither fixed nor simple. Frame
interpolation [37, 36, 4, 18, 59] is another visual-based solution. It
synthesizes intermediate frames from a given set of correctly decoded frames
to replace damaged or lost frames. Nonetheless, interpolation methods are
barely satisfactory when there exists a large scene motion between frames
[40], and when they encounter errors spread across a sequence of frames. Video
inpainting is similar to video completion, which aims to complete missing
regions in a given video [53]. Generally, video inpainting takes the
surrounding temporal and spatial content as a reference to fill corrupted
regions by learning underlying patterns and structural features of videos [29,
33]. However, the corrupted regions are commonly simulated by user-predefined
binary error masks instead of natural errors generated from the real
bitstream.

For the application scenarios of video storage, communication, and internet
video, the manually created masks have difficulty in reflecting the shapes and
patterns of real corruption. The requirements of video content coherence in
temporal and spatial dimensions are hard to meet when large motion and details
are missing across frames [29]. Therefore, the real bitstream and video
datasets, as well as video content recovery methods, are highly necessary and
urgently needed. So far, there is no large-scale dataset specialized for
bitstream-corrupted video recovery. Existing inpainting datasets are limited
to simulated error masks, and error concealment datasets are small-scale and
may require extracting motion information from bitstream, which is not always
available [23].

In this paper, we construct the first large-scale benchmark to facilitate the
research of bitstream-corrupted video (BSCV) recovery. Our BSCV dataset
includes more than 28,000 bitstream-corrupted video clips (over 3,500,000
frames), which are extracted and elaborated from the most popular video
inpainting datasets, YouTube-VOS [55] and DAVIS [39]. Specifically, we
compress these video clips into bitstreams using the most popular H.264 video
codec [21]. Segments in bitstreams are randomly removed to simulate the effect
of packet loss error and storage damage error on decoded videos, and these
error types are common in real-world multimedia communications [3]. The
simulated error patterns used in typical video recovery tasks and the real
error patterns of our dataset are shown in Fig. 1. It can be observed that the
video error types in our dataset are sophisticated and unpredictable, while
others are simple and fixed. Therefore, our dataset enables us to reveal the
problem of real-world video corruption in multimedia communications
completely. Furthermore, we also provide a specialized recovery method for
bitstream-corrupted video. The remaining semantic information in corrupted
regions is incorporated with the spatially and temporally adjacent information
to recover the corrupted regions.

The main contributions are as follows: (i) We construct BSCV, the first large-
scale dataset used for bitstream-corrupted video recovery in the real world.
The provided videos are decoded from real corrupted bitstreams, which are
generated by our three-parameter bitstream corruption model. The dataset
contains over 28,000 challenging corrupted video clips with realistic and
unpredictable error patterns, multiple corruption levels, and flexible dataset
branches. (ii) We propose a plug-and-play module to flexibly embed in existing
video inpainting frameworks. It enhances the feature representation capability
by extracting residual visual information from the corrupted region, achieving
higher recovery quality. (iii) We perform a comprehensive evaluation of our
dataset to reveal the limitation of existing video inpainting algorithms and
point out the future direction.

##  2 Related Works

Benchmark and dataset. To the best of our knowledge, currently, there is no
bitstream-corrupted video benchmark for the research of bitstream-corrupted
video recovery.

Dataset |  |  |  |  |
---|---|---|---|---|---
Name | Clip |  |  |  |
Numbers | Bitstream |  |  |  |
Provided | Bitstream |  |  |  |
Corruption | Mask |  |  |  |
Provided | Application |  |  |  |
Scenarios |  |  |  |  |
YUV Sequence [1] | 26 | $\times$ | × | $\times$ | VC, VR
Vimeo90K [54] | 90,000+1 | $\times$ | × | $\times$ | SR, ITP
REDS [34] | 300 | $\times$ | × | $\times$ | SR, DB
DAVIS [39] | 150 | $\times$ | × | ✓ | OS, INP
YouTube-VOS [55] | 4,000+ | $\times$ | × | ✓ | OS, INP
BSCV(Ours) | 28,000+ | ✓ | $\checkmark$ | ✓ | VR, SR, ITP, INP

  * *

VC: Video Coding, SR: Super Resolution, ITP: Interpolation, DB: Deblurring,
OS: Object Segmentation, INP: Inpainting, VR: Video Recovery

  * 1

Frame numbers are fixed at 7 frames.

Table 1: Comparisons among video datasets

As shown in Table 1, for conventional error concealment and corruption
recovery research, using a small set of YUV sequence [1] to test the algorithm
performance is a common practice. In that case, researchers usually simulate
different kinds of stripe or packet loss-caused masks on those video sequences
[9, 51, 26, 25]. However, the scale issue limits the application of deep
learning methods. Along with the development of video datasets for different
computer vision applications, some datasets such as Vimeo90K [54], REDS [34]
are proposed for video restoration tasks including super-resolution,
deblurring, and so on. Recently, deep learning-based video completion assumes
a very similar task setting with error concealment which is a sub-task of
video inpainting. By accepting arbitrary masks, learning-based video
inpainting can be trained by a large number of samples. The setting of the
mask is usually a fixed mask [53] or an object-like mask with limited size,
random shape, and motion [33, 58, 29, 60]. Most datasets involve the content
of the videos in the DAVIS [39] and YouTube-VOS [55]. However, DAVIS is still
a relatively small scale with only 150 video clips, and therefore it is
usually used for qualitative evaluation in video inpainting research.
Recently, YouTube-VOS has been a widely-used large-scale dataset for various
video inpainting algorithm training because of its content diversity.
Nevertheless, the large-scale dataset never considers real video corruption.
With the simulated mask setting, video inpainting and different kinds of video
recovery research are difficult to perform well in complex and unpredictable
video corruptions because of the gap between the human-predefined binary mask
and unpredictable mask supervision. Besides, modern video datasets are usually
packed in frame sequences, and bitstream-related research still hungers for
data in the current deep learning era.

Video restoration. As videos can be treated as multiple consecutive
images/frames, earlier works [43, 11] simply reuse the ideas from image
restoration with the temporal redundancy of neighboring frames fails to be
explored. To fully utilize temporal information, Xue et al. [54] proposed a
task-oriented flow to achieve feature alignment explicitly. Other studies
utilized dynamic upsampling filter [20] or deformable convolution [44] to
achieve implicit motion compensation. As for feature fusion, either a one-
stage direct fusion structure [44, 47] or multi-stage progressive fusion
structure [57] have been used in existing methods.

Video error concealment. Video error concealment, a commonly-used post-
processing technique at the decoder side, aims to recover the error regions in
decoded videos [51]. It can be divided into various categories in the
bitstream and pixel levels, including spatial, temporal, and hybrid spatial-
temporal methods [23, 25, 56]. Traditionally, at the bitstream level, missing
motion information can be estimated by surrounding motion vector and block
partitioning in the previous frame [8, 32]. At the pixel level, pixel-wise
processing is capable but relies on deficient spatial information [23].
Recently, deep learning-based methods still assume a traditional corruption
pattern and use experimental mask settings to simulate stripe or patch loss
[42, 52, 51, 9]. It makes these methods not suitable for recovering bitstream-
corrupted videos because the corruption caused by realistic packet loss is
generally unpredictable and irregular.

Video inpainting/completion. Video inpainting is to generate content in
unfilled regions of a video, which accepts arbitrarily defined masks to
indicate corrupted regions. Traditionally, video inpainting is considered as a
patch matching or pixel diffusion problem [19, 19, 12, 16, 35]. In the era of
deep learning, the patch-based method also makes significant success [33, 41].
Flow-guided generative methods are currently mainstream in video inpainting,
leveraging motion information for spatial and temporal relationships between
frames [53, 13, 58, 28, 22, 29, 61]. DFVI [53] was a pioneering work that
formulates the generative video inpainting problem as a pixel propagation task
rather than simply filling RGB values in corrupted regions. Li et al. [29]
built the traditional 3-stage video inpainting pipeline optimized jointly and
achieved an efficient end-to-end framework for video inpainting. In the
context of bitstream-corrupted video recovery, video inpainting is closely
related. However, existing research often overlooks the performance of
inpainting algorithms when dealing with dynamic and large masks. Consequently,
they fail to address complex recovery scenarios with significant corrupted
areas and partially remained content caused by bitstream corruption.

##  3 Bitstream-corrupted Video Dataset

Figure 2: Left: H.264 bitstream statistics and the proposed corruption model.
Right: Inter-frame correlations and error propagation in the frame domain.

H.264 bitstream and bitstream corruption. The most popular video codec, H.264,
was used by 85% of video developers in 2022 [2]. The compatibility of H.264
with a variety of devices and platforms empowers the delivery of video content
bitstream over the internet. The H.264 bitstream domain is shown in Fig. 2.
The typical format of H.264 bitstream consists of successive NALUs (network
abstraction layer units). A bitstream contains several bytes of start code
prefix and Header. The SPS (sequence parameter sets) and PPS (picture
parameter sets) also occupy a small number of bytes. By analyzing the
bitstream component, we find that the bytes of SPS, PPS, header, and start
code only take up a negligible proportion of NALU bytes. In contrast, the
frame data occupy a dominant proportion of an H.264 bitstream (usually more
than 99.9%).

The bitstream segments and packets are possibly corrupted or lost in the
chains of video storage, encoding, transmission, and decoding. Therefore,
video recovery from corrupted bitstreams is in surging demand. Due to the
significant proportion of frame data in a bitstream file, corruption is most
likely to occur in the frame data parts, which is the basic assumption in this
paper. As shown in the frame domain of Fig. 2, there exist inter-frame
correlations among frames of a video when encoding a video into the bitstream.
A frame can refer to other previously coded frames for high coding efficiency.
The inter-NALU dependencies are accordingly created in a bitstream. Thus,
error propagation among frames tends to be irregular and unpredictable.

Bitstream-corrupted video generation. Due to the popularity of H.264, video
clips are encoded by H.264 codec to generate bitstream files of these videos.
The coding configuration selects widely used close-GOP (group of pictures),
and the GOP sizes adopt 16 frames for a long-range reference. We simulate the
corruption pattern by removing specific segments of some NALUs in a bitstream,
as shown in the bottom part of Fig. 2. As for the start code and NALU headers,
we skip them because of their negligible proportion less than 0.01% of a H.264
bitstream and corruption on these parts may cause severe errors, e.g.,
sequential frame missing, even decoding failures. In such cases, bitstream
corrupted videos are not recoverable by existing vision-based methods. Based
on the analysis, we can randomly corrupt frames in visual level. Therefore, we
parameterize a three-parameter corruption model $(P,L,S)$, where the corrupted
fragments are defined by frame corruption probability $P$, corruption location
$L$, and fragment size $S$. The corrupted bitstreams are parsed and decoded by
H.264 decoder, generating videos with unpredictable regional errors.

Dataset construction and statistics.

Figure 3: Taking the BSCV dataset branch in the parameters of
$(1/16,0.4,4096)$ as an example for illustration. (a) The statistics of
corruption distribution. (b) The corruption level changes among frames for
some sampled videos.

By the above bitstream corruption, we construct a bitstream-corrupted video
(BSCV) dataset based on two commonly used datasets in video inpainting, i.e.,
YouTube-VOS [55] and DAVIS [39]. In detail, we extract 4,132 original videos
from YouTube-VOS and DAVIS datasets to generate the main branch.

DAVIS provides 480P videos with dense object segmentation annotation which is
mostly used for evaluation in prior works, we followed its application in this
paper as well. YouTube-VOS is mainly a 720P dataset which also contains
several 1080P videos. Our method is also applicable for those 1080P videos
which proves its scalability. Then we additionally provide an additional 1080P
branch with 256 videos from YouTube-UGC dataset [48]. It contains longer frame
sequence and higher resolution which could enrich the source of our dataset,
allowing further extension based on it. Further, we provide a small 4K branch
using videos from Videezy4K [17] as an reference example for the future
extension to higher resolution videos.

By setting the parameter combinations of the proposed corruption model,
multiple branches of the BSCV dataset can be generated. We also provide error
region masks in the dataset. Specifically, grayscale difference maps are
calculated by subtracting the corrupted videos from the corresponding original
videos decoded from the corruption-free bitstream. The slight changes below
the default threshold are suppressed, and the small outliers inside or outside
masks are removed by morphological filtering.

For the BSCV dataset branch in the parameters of $(1/16,0.4,4096)$, Fig. 3
illustrates the corruption statistics of the branch and the corruption degree
of randomly selected video samples. The area ratio of corrupted regions to
their corresponding frame is referred to as the “corrupted area ratio”. The
ratios range from 0-10%, 10-30%, and above 30% are defined as minor (min.),
moderate (mod.), and severe (sev.) corruption levels. The ratio of 0 is
defined as corruption-free (unc.). We observe that nearly 30% of frames are
corrupted for this example dataset branch. Compared with the existing video
inpainting tasks with fixed mask area settings (e.g., 1/16), the frame
corruption in our dataset is complex, variable, and unpredictable, making it
closer to realistic scenarios and more challenging.

We further analyze the rich error patterns of our dataset shown in Fig. 1.
Color artifacts occur when chrominance information is corrupted, which is more
severer than edge color bleeding in typical compression artifacts [31]. The
trailing artifacts come from the corruption of motion information, which
causes a floating trailing effect in subsequent frames. The texture
information corruption and error propagation may cause blocking artifacts.
Duplicate artifacts are common in intra-coding regions, which duplicate the
error pixels in the adjacent regions. More details on dataset construction and
analysis of error patterns can refer to the Supplementary Material.

Flexibility and extensibility. The constructed dataset and proposed three-
parameter corruption model can provide flexibility in dataset customization
and extensibility in application scenarios. By setting different parameter
combinations, it is flexible to construct custom datasets to meet specific
application scenarios, which is demonstrated in the experiment section. We
also developed a video recovery framework without relying on the motion,
partition, and residual information in case they are not available in a
corrupted bitstream. Thus, the provided dataset and recovery framework can
extend to broad bitstream corruption scenarios, such as packet loss during
transmission, segment corruption in compression, and deletion of partial data
in storage. The application scenarios are not limited to bitstream-related
video recovery. It is also suitable for local and cloud video processing
tasks, like video inpainting, completion, and manipulation.

##  4 Bitstream-corrupted Video Recovery Framework

In this section, we propose a specialized plug-and-play feature enhancement
module and implement it on existing video inpainting frameworks.

Figure 4: Overview of our bitstream-corrupted video recovery (BSCVR)
framework. Compared with existing methods, we follow the common practice by
inputting the corruption-free content as the basic information source when
constructing local features for recovery. We additionally enable a new input
channel for the corrupted region and extract the feature of its partial
contents which is completely ignored by existing methods. With transformer-
based architecture, the local feature can be enhanced by encoding the feature
of partial contents into it.

Framework overview. The overview of the proposed bitstream-corrupted video
recovery (BSCVR) framework is illustrated in Fig. 4. We propose a plug-and-
play feature enhancement module that provides an additional perception channel
to existing video inpainting frameworks. It extracts and fuses local features
from corrupted and corruption-free regions. By encoding the residual
information inside the corrupted regions into the local features, it can
enhance the feature completion and representation capability greatly compared
to existing video inpainting frameworks. Consequently, the enhanced feature
can provide a solid reference for the subsequent recovery process. Then a
flow-guided feature propagation module is used in [29] to propagate the
content. Combining with reference content from non-local frames, the content
generation module is implemented by stacking several temporal focal
transformers [29].

To be specific, given a corrupted video frame sequence
$\left\\{X^{t}\in\mathcal{R}^{3\times h\times w}|t=1,2,...,T\right\\}$, and
its corresponding mask sequence indicating the corrupted regions
$\left\\{M^{t}\in\mathcal{R}^{1\times h\times w}|t=1,2,...,T\right\\}$. The
video recovery framework is expected to recover the corrupted region with
spatially and temporally plausible content. According to Fig. 4, for the input
corrupted frame sequence, we use a context encoder ($E$) [38] to perform
region-based encoding. $\left\\{Q^{t}\in\mathcal{R}^{3\times h\times
w}|t=1,2,...,T_{l}\right\\}$ indicated by masks will be separately input into
the recovery framework. Then, we propose to use several transformer encoder
layers [46] to fuse and re-encode these two features. By attention-based
decoding and channel fusion, an intermediate feature is generated.
Consequently, with skip connection and output projection, the representative
capability of the resulting feature can be further enhanced by fusing multi-
scale and multi-level information. We then follow the approach of flow-guided
video inpainting to extract and complete optical flows from neighboring frames
to serve as guidance for feature alignment and propagation. Afterward, a
content generation module based on temporal focal transformer and soft
spliting will combine the enhanced, aligned, and propagated features of local
neighboring frames with the reference features of non-local frames’
corruption-free regions $\left\\{R^{t}\in\mathcal{R}^{3\times h\times
w}|t=1,2,...,T_{nl}\right\\}$ to generate content and finally reconstruct a
result frame sequence $\left\\{\hat{Y}^{t}\in\mathcal{R}^{3\times h\times
w}|t=1,2,...,T_{l}\right\\}$ through a decoder ($D$) module.

Training method. Inspired by existing video inpainting methods, we apply
multiple loss terms from different perspectives [29] to our BSCVR framework.
The overall loss function is composed of three terms $\mathcal{L}_{rec}$,
$\mathcal{L}_{adv}$, and $\mathcal{L}_{flow}$, expressed as:

$Loss=\mathcal{L}_{rec}+\mathcal{L}_{adv}+\mathcal{L}_{flow}\\\
=\left\|\mathrm{\hat{\textbf{Y}}}-\mathrm{\textbf{Y}}\right\|_{1}+(-E_{z\sim
P_{\hat{\textbf{Y}}}(z)}[D(z)])+\mathcal{L}_{flow}.$ (1)

Meanwhile, since the model is trained in GAN [15] paradigm, the applied
T-PatchGAN [7] based discriminator should minimize the loss, expressed as:

$\mathcal{L}_{\mathcal{D}}=E_{x\sim
P_{\textbf{Y}}(x)}[\mathrm{ReLU}(1-D(x))]+E_{x\sim
P_{\hat{\textbf{Y}}}(x)}[\mathrm{ReLU}(1+D(z))].$ (2)

Detailed descriptions of the methodology can be found in the Supplementary
Material.

##  5 Experiment

Table 2: Quantitative results of SOTA pre-trained video inpainting methods,
their corresponding models trained on our dataset (denoted by Star Mark*), and
our method. In our method, BSCVR-S means that the feature enhancement module
considers the input feature as a sequence like traditional Transformer [46],
and BSCVR-P indicates that the module considers the input as patches referring
to SwinIR [30]. The comparison is conducted under the 240P setting due to the
model capability of previous works. For the methods which are able to handle
arbitrary-resolution video, we calculate metrics based on the original frame
sequence, and we measured and demonstrate the runtime of the model under 720P
(former) / 480P (latter) input, respectively.

Test |  |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---
res. | Method | Accuracy |  |  |  |  |  |  |  | Efficiency
|  | YouTube-VOS (720P) Subset |  |  |  | DAVIS (480P) Subset |  |  |  | Runtime
|  | PSNR$\uparrow$ |  $SSIM$↑ | LPIPS$\downarrow$ | VFID$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | VFID$\downarrow$ | s/frame
240P | Input | 18.8749 | $0.8160$ | 0.1527 | 0.2015 | 18.4562 | 0.7921 | 0.1541 | 0.4189 | -
| STTN [58] | 29.3840 | $0.9174$ | 0.0465 | 0.0566 | 26.2172 | 0.8600 | 0.0638 | 0.1589 | 0.120
| STTN [58]* | 29.9172 | $0.9303$ | 0.0394 | 0.0544 | 26.5453 | 0.8722 | 0.0575 | 0.1534 | 0.120
| FuseFormer [33] | 28.8012 | $0.9047$ | 0.0549 | 0.0641 | 26.2547 | 0.8618 | 0.0659 | 0.1645 | 0.200
| FuseFormer [33]* | 29.8108 | $0.9328$ | 0.0381 | 0.0526 | 26.7367 | 0.8834 | 0.0531 | 0.1477 | 0.200
| E2FGVI-HQ [29] | 29.6866 | $0.9228$ | 0.0469 | 0.0555 | 26.7850 | 0.8765 | 0.0600 | 0.1513 | 0.160
| E2FGVI-HQ [29]* | 31.0030 | $0.9473$ | 0.0341 | 0.0479 | 27.6551 | 0.9018 | 0.0491 | 0.1387 | 0.160
| BSCVR-S (Ours)* | 31.8345 | $0.9584$ | 0.0262 | 0.0427 | 28.4211 | 0.9180 | 0.0381 | 0.1196 | 0.172
| BSCVR-P (Ours)* | 31.9534 | 0.9598 | 0.0258 | 0.0426 | 28.5430 | 0.9199 | 0.0375 | 0.1165 | 0.178
Ori- |  |  |  |  |  |  |  |  |  |
ginal | Input | 19.1490 | $0.8244$ | 0.1415 | 0.0575 | 18.4384 | 0.7979 | 0.1490 | 0.1999 | -
| E2FGVI-HQ [29] | 28.5039 | $0.8783$ | 0.0453 | 0.0126 | 25.7803 | 0.8236 | 0.0504 | 0.0468 | 0.192 / 0.176
| E2FGVI-HQ [29]* | 29.5666 | $0.9023$ | 0.3955 | 0.0161 | 26.6723 | 0.8611 | 0.0530 | 0.0577 | 0.192 / 0.176
| BSCVR-S (Ours)* | 30.2235 | 0.9185 | 0.0335 | 0.0143 | 27.2770 | 0.8809 | 0.0427 | 0.0500 | 0.250 / 0.203
| BSCVR-P (Ours)* | 29.9943 | $0.9144$ | 0.0343 | 0.0104 | 26.3564 | 0.8511 | 0.0416 | 0.0406 | 0.261/ 0.213

The proposed BSCVR and state-of-the-art (SOTA) video inpainting methods are
performed on the constructed BSCV dataset. We conduct comprehensive
quantitative and qualitative evaluations to demonstrate the effectiveness of
our dataset and method. The flexibility of our corruption model and the
robustness of our BSCVR framework are validated on multiple branches of BSCV.

(a) Qualitative results comparing our method with STTN [58], FuseFormer [33],
and E2FGVI-HQ [29] under unified 240P resolution.

(b) Qualitative results comparing our method with E2FGVI-HQ [29] trained on
our dataset under original video resolution.

Figure 5: Qualitative comparison of our method and SOTA video inpainting
methods on low (a) and high (b) resolutions. The involved corruption types
include (1) blocking artifacts, (2) color artifacts, (3) duplication
artifacts, (4) misalignment, (5) texture loss, (6) trailing Artifacts and
their combinations.

Experimental setting. We adopt the corruption parameter of $(1/16,0.4,4096)$,
and its corresponding statistics have been illustrated in Fig. 3. This setting
has moderate difficulties with adequate corruption types, which is suitable
for video inpainting methods. The corrupted region is usually recoverable yet
challenging. SOTA video inpainting methods are compared with our method to
recover corruption-free videos. STTN [58] and FuseFormer [33] downsample
videos to 240P due to the limitation of computational complexity. E2FGVI-HQ is
an upgraded version of E2FGVI [29], stating that it could take arbitrary
resolution and generate results with the original input resolution. They could
be viewed as the main competitor of our method. Noted that previous works set
the mask with “random shape and location” to augment inpainting data, and
those pre-trained models are trained with 50K iterations. In contrast,
training those methods on our dataset requires only 25K iterations to
converge. The detailed implementation of our method can refer to the
Supplementary Material.

Evaluation metrics. Regarding the quantitative evaluation, we measure the
performance of inpainting algorithms based on two aspects: inpainting quality
reconstruction and realism. Among them, Peak Signal-to-Noise Ratio (PSNR),
Structural Similarity Index (SSIM) [49], and Learned Perceptual Image Patch
Similarity (LPIPS) [62] using pre-trained AlexNet backbone [27] are mainly
used to measure reconstruction performance. Video Fr$\acute{e}$chet Inception
Distance (VFID) [24] is mainly used to measure performance in terms of
realism. It corresponds to the sets of all recovered videos and all reference
videos. The features are extracted from a pre-trained I3D backbone [6].

Quantitative evaluation. The quantitative results on the YouTube-VOS and DAVIS
subsets are shown in Table 2. Due to the model capability of some previous
works, we first follow the existing experimental setting of video inpainting
to downscale the original video in our dataset to 240P and conducted model
training and metric calculation. It can be observed that our methods achieve
better results in all metrics. However, compromising on video resolution is
not reasonable for the video recovery problem. Thus, we particularly compared
our method with the SOTA method E2FGVI-HQ [29] which is currently the only
method can handle the original resolution scenario. For the metrics of PSNR,
SSIM, LPIPS, and VFID, our method can achieve significant improvement, which
makes the bitstream-corrupted video recoverable, e.g., >30dB PSNR for YouTube-
VOS. Our method could comprehensively refine the content in corrupted regions
to guide plausible content generation and keep low computational complexity by
applying efficient model architecture referring to E2FGVI-HQ [29]. For the
results on different corruption parameters, we provided more experiments in
the Supplementary Material.

Param. |  |  |  |  |
---|---|---|---|---|---
(P, L, S) | Methods | Accuracy |  |  |
|  | DAVIS Subset |  |  |
|  | PSNR$\uparrow$ |  $SSIM$↑ | LPIPS$\downarrow$ | VFID$\downarrow$
(1/16, 0.4, 4096) | Input | 18.4384 | $0.7979$ | 0.1490 | 0.1999
| E2FGVI-HQ [29]* | 26.3734 | $0.8415$ | 0.0466 | 0.0444
| BSCVR-S (Ours) | 27.2770 | 0.8809 | 0.0427 | 0.0500
| BSCVR-P (Ours) | 26.3564 | $0.8511$ | 0.0416 | 0.0406
(1/16, 0.4, 2048) | Input | 18.2283 | $0.7798$ | 0.1536 | 0.1920
| E2FGVI-HQ [29]* | 26.0251 | $0.8423$ | 0.4777 | 0.0440
| BSCVR-S (Ours) | 26.2437 | 0.8554 | 0.0416 | 0.0401
| BSCVR-P (Ours) | 26.1057 | $0.8525$ | 0.0422 | 0.0407
(1/16, 0.2, 4096) | Input | 18.0789 | $0.7649$ | 0.1569 | 0.2067
| E2FGVI-HQ [29]* | 24.7468 | $0.7746$ | 0.0656 | 0.0546
| BSCVR-S (Ours) | 24.9832 | 0.7980 | 0.0570 | 0.0478
| BSCVR-P (Ours) | 24.8303 | $0.7940$ | 0.0579 | 0.0484
(2/16, 0.4, 4096) | Input | 17.9418 | $0.7616$ | 0.1592 | 0.1963
| E2FGVI-HQ [29]* | 24.3774 | $0.7934$ | 0.0623 | 0.0767
| BSCVR-S (Ours) | 24.5066 | 0.8077 | 0.0553 | 0.0698
| BSCVR-P (Ours) | 24.3808 | $0.8037$ | 0.0561 | 0.0705
(1/16, 0.4,8192) | Input | 18.6665 | $0.8170$ | 0.1450 | 0.1849
| E2FGVI-HQ [29]* | 26.0722 | $0.8371$ | 0.0486 | 0.0455
| BSCVR-S (Ours) | 26.2708 | 0.8518 | 0.0423 | 0.0417
| BSCVR-P (Ours) | 26.1231 | $0.8487$ | 0.0430 | 0.0418
(1/16, 0.8, 4096) | Input | 19.0062 | $0.8419$ | 0.1389 | 0.1874
| E2FGVI-HQ [29]* | 32.8311 | $0.9506$ | 0.0162 | 0.0264
| BSCVR-S (Ours) | 32.7959 | 0.9514 | 0.0147 | 0.0247
| BSCVR-P (Ours) | 32.7204 | $0.9509$ | 0.0149 | 0.0252
(4/16, 0.4, 4096) | Input | 17.8542 | $0.7587$ | 0.1610 | 0.1973
| E2FGVI-HQ [29]* | 22.7912 | $0.7480$ | 0.0738 | 0.1192
| BSCVR-S (Ours) | 22.7094 | 0.7570 | 0.0679 | 0.1079
| BSCVR-P (Ours) | 22.5480 | $0.7527$ | 0.0686 | 0.1079

Table 3: Performance comparison with E2FGVI-HQ on different dataset branches
under different corruption parameter combinations.

Qualitative Evaluation. For qualitative evaluation, we choose STTN [58],
FuseFormer [33], and E2FGVI-HQ [29] trained on our dataset as comparison
methods. The evaluation is conducted under 240P and original resolutions. Some
representative corruption patterns and their recovery results are visualized
in Fig. 5(a). The comparison methods are difficult to generate plausible
content to recover the corrupted region, while our proposed method can
generate clearer details with more informative textures and structures. Our
method and dataset limits the tendency of object removal when the mask is
relatively large, especially under the original resolution and the compared
results of our method with the SOTA method E2FGVI-HQ [29] are shown in Fig.
5(b). These results demonstrate the limitation of current methods on the
proposed dataset and the advantage of our high quality data and method. More
visualized results can be found in the Supplementary Material.

Figure 6: Corruption ratio distribution of the YouTube-VOS&DAVIS dataset
branches for experiments under different corruption parameter combinations.

Flexibility in Dataset construction. We use the three-parameter corruption
model $(P,L,S)$ in our dataset construction. We validate the model by
generating more branches of dataset with seven additional parameter settings:
$(1/16,0.4,2048)$, $(1/16,0.4,8192)$, $(1/16,0.4,4096)$, $(1/16,0.2,4096)$,
$(1/16,0.8,4096)$, $(2/16,0.4,4096)$ and $(4/16,0.4,4096)$. These settings
represent varying corruption probabilities, where $P=m/l$ implies that the
corruption happens in random $m$ frames out of $l$ frames of a GOP.

We analyze the corruption distribution by calculating the ratio of the
corrupted region to the frame resolution, as shown in Fig. 6. It indicates
that parameter $P$ has the most significant impact on corruption, leading to a
higher number of corrupted frames and more severe damage. For these dataset
branches, we compare our BSCVR and E2FGVI-HQ under the original resolution on
the DAVIS 480P subset. The results are listed in Tab. 3. It shows that the
proposed BSCVR consistently outperforms E2FGVI-HQ, validating the robustness
of our BSCVR on multiple dataset branches.

##  6 Conclusion

Aiming at the challenging problem of bitstream-corrupted video recovery in the
real world, we construct the first large-scale benchmark, BSCV. The BSCV
provides a bitstream corruption model, a realistic decoded video dataset, and
a video recovery framework, BSCVR. The bitstream corruption model enables to
flexibly generate dataset branches by specifying parameter combinations. The
dataset contains 28,000 realistic video clips decoded from corrupted
bitstreams with unpredictable error patterns and corruption levels. The BSCVR
offers a plug-and-play feature enhancement module to achieve high-quality
video recovery. Extensive experiments demonstrate that the proposed BSCVR
outperforms SOTA video inpainting methods quantitatively and qualitatively.
The flexibility of dataset construction and the robustness of our BSCVR
framework are also validated on various dataset branches. The benchmark
dataset is expected to benefit video recovery in video communication and
multimedia forensics. The future work will concentrate on designing more
reasonable bitstream corruption models, engaging more dataset sources, and
creating more effective recovery frameworks.

## Acknowledgement

This research / project is supported by the National Research Foundation,
Singapore, and Cyber Security Agency of Singapore under its National
Cybersecurity R&D Programme (NRF2018NCR-NCR009-0001). Any opinions, findings
and conclusions or recommendations expressed in this material are those of the
author(s) and do not reflect the views of National Research Foundation,
Singapore and Cyber Security Agency of Singapore.

## References

  * [1] Yuv sequence.  http://trace.eas.asu.edu/yuv/.  Accessed: 2023-06-06.
  * [2] The 6th annual bitmovin video developer report.  https://bitmovin.com/video-developer-report, 2022.
  * [3] Anthony O Adeyemi-Ejeye, Mohammed Alreshoodi, Laith Al-Jobouri, and Martin Fleury.  Impact of packet loss on 4k uhd video for portable devices.  Multimedia tools and applications, 78:31733–31755, 2019.
  * [4] Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang.  Depth-aware video frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3703–3712, 2019.
  * [5] V. Boussard, F. Golaghazadeh, S.Coulombe, F. X. Coudoux, and P. Corlay.  Robust h. 264 video decoding using crc-based single error correction and non-desynchronizing bits validation.  In 2020 IEEE International Conference on Image Processing (ICIP), pages 1098–1102. IEEE, 2020.
  * [6] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  * [7] Ya-Liang Chang, Zhe Yu Liu, Kuan-Ying Lee, and Winston Hsu.  Free-form video inpainting with 3d gated convolution and temporal patchgan.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9066–9075, 2019.
  * [8] Yueh-Lun Chang, Yuriy A Reznik, Zhifeng Chen, and Pamela C Cosman.  Motion compensated error concealment for hevc based on block-merging and residual energy.  In 2013 20th International Packet Video Workshop, pages 1–6. IEEE, 2013.
  * [9] Byungjin Chung and Changhoon Yim.  Bi-sequential video error concealment method using adaptive homography-based registration.  IEEE Transactions on Circuits and Systems for Video Technology, 30(6):1535–1549, 2020.
  * [10] Cisco.  Cisco visual networking index: Forecast and trends, 2017–2022.  https://twiki.cern.ch/twiki/pub/HEPIX/TechwatchNetwork/HtwNetworkDocuments/white-paper-c11-741490.pdf.
  * [11] Qiqin Dai, Seunghwan Yoo, Armin Kappeler, and Aggelos K Katsaggelos.  Dictionary-based multiple frame video super-resolution.  In 2015 IEEE International Conference on Image Processing (ICIP), pages 83–87. IEEE, 2015.
  * [12] Mounira Ebdelli, Olivier Le Meur, and Christine Guillemot.  Video inpainting with short-term windows: application to object removal and error concealment.  IEEE Transactions on Image Processing, 24(10):3034–3047, 2015.
  * [13] Chen Gao, Ayush Saraf, Jia-Bin Huang, and Johannes Kopf.  Flow-edge guided video completion.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, pages 713–729. Springer, 2020.
  * [14] Firouzeh Golaghazadeh, Stéphane Coulombe, François-Xavier Coudoux, and Patrick Corlay.  Checksum-filtered list decoding applied to h. 264 and h. 265 video error correction.  IEEE Transactions on Circuits and Systems for Video Technology, 28(8):1993–2006, 2017.
  * [15] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [16] Miguel Granados, James Tompkin, K Kim, Oliver Grau, Jan Kautz, and Christian Theobalt.  How not to be seen—object removal from videos of crowded scenes.  In Computer Graphics Forum, volume 31, pages 219–228. Wiley Online Library, 2012.
  * [17] Shi Guo, Xi Yang, Jianqi Ma, Gaofeng Ren, and Lei Zhang.  A differentiable two-stage alignment scheme for burst image reconstruction with large shift.  2022\.
  * [18] Ai-Mei Huang and Truong Q Nguyen.  A multistage motion vector processing method for motion-compensated frame interpolation.  IEEE transactions on image processing, 17(5):694–708, 2008.
  * [19] Jia-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Johannes Kopf.  Temporally coherent completion of dynamic video.  ACM Transactions on Graphics (TOG), 35(6):1–11, 2016.
  * [20] Younghyun Jo, Seoung Wug Oh, Jaeyeon Kang, and Seon Joo Kim.  Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3224–3232, 2018.
  * [21] Video Team JVT Joint et al.  Draft itu-t recommendation and final draft international standard of joint video specification.  ITU-T Rec. H. 264/ISO/IEC 14496-10 AVC, 2003.
  * [22] Jaeyeon Kang, Seoung Wug Oh, and Seon Joo Kim.  Error compensation framework for flow-guided video inpainting.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV, pages 375–390. Springer, 2022.
  * [23] Mohammad Kazemi, Mohammad Ghanbari, and Shervin Shirmohammadi.  A review of temporal video error concealment techniques and their suitability for hevc and vvc.  Multimedia Tools and Applications, 80:12685–12730, 2021.
  * [24] Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon.  Deep video inpainting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5792–5801, 2019.
  * [25] J. Koloda, J. Østergaard, S. H. Jensen, V. Sánchez, and A. M. Peinado.  Sequential error concealment for video/images by sparse linear prediction.  IEEE Transactions on Multimedia, 15(4):957–969, 2013.
  * [26] J. Koloda, A. M. Peinado, and V. Sánchez.  Kernel-based mmse multimedia signal reconstruction and its application to spatial error concealment.  IEEE Transactions on Multimedia, 16(6):1729–1738, 2014.
  * [27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [28] Dong Lao, Peihao Zhu, Peter Wonka, and Ganesh Sundaramoorthi.  Flow-guided video inpainting with scene templates.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14599–14608, 2021.
  * [29] Zhen Li, Cheng-Ze Lu, Jianhua Qin, Chun-Le Guo, and Ming-Ming Cheng.  Towards an end-to-end framework for flow-guided video inpainting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17562–17571, 2022.
  * [30] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.  Swinir: Image restoration using swin transformer.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1833–1844, 2021.
  * [31] Liqun Lin, Shiqi Yu, Liping Zhou, Weiling Chen, Tiesong Zhao, and Zhou Wang.  Pea265: Perceptual assessment of video compression artifacts.  IEEE Transactions on Circuits and Systems for Video Technology, 30(11):3898–3910, 2020.
  * [32] Ting-Lan Lin, Neng-Chieh Yang, Ray-Hong Syu, Chin-Chie Liao, and Wei-Lin Tsai.  Error concealment algorithm for hevc coded video using block partition decisions.  In 2013 IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC 2013), pages 1–5. IEEE, 2013.
  * [33] Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu Sun, Xiaogang Wang, Jifeng Dai, and Hongsheng Li.  Fuseformer: Fusing fine-grained information in transformers for video inpainting.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14040–14049, 2021.
  * [34] Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee.  Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [35] Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, and Patrick Pérez.  Video inpainting of complex scenes.  Siam journal on imaging sciences, 7(4):1993–2019, 2014.
  * [36] Simon Niklaus, Long Mai, and Feng Liu.  Video frame interpolation via adaptive convolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 670–679, 2017.
  * [37] Simon Niklaus, Long Mai, and Feng Liu.  Video frame interpolation via adaptive separable convolution.  In Proceedings of the IEEE international conference on computer vision, pages 261–270, 2017.
  * [38] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [39] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  arXiv:1704.00675, 2017.
  * [40] Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless.  Film: Frame interpolation for large motion.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII, pages 250–266. Springer, 2022.
  * [41] Jingjing Ren, Qingqing Zheng, Yuanyuan Zhao, Xuemiao Xu, and Chen Li.  Dlformer: Discrete latent transformer for video inpainting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3511–3520, 2022.
  * [42] Arun Sankisa, Arjun Punjabi, and Aggelos K Katsaggelos.  Video error concealment using deep neural networks.  In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 380–384. IEEE, 2018.
  * [43] Hiroyuki Takeda, Peyman Milanfar, Matan Protter, and Michael Elad.  Super-resolution without explicit subpixel motion estimation.  IEEE Transactions on Image Processing, 18(9):1958–1975, 2009.
  * [44] Yapeng Tian, Yulun Zhang, Yun Fu, and Chenliang Xu.  Tdan: Temporally-deformable alignment network for video super-resolution.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3360–3369, 2020.
  * [45] Willy R. Vasquez, Stephen Checkoway, and Hovav Shacham.  The most dangerous codec in the world: Finding and exploiting vulnerabilities in h.264 decoders.  In USENIX Security Symposium, 2023.
  * [46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [47] Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy.  Edvr: Video restoration with enhanced deformable convolutional networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [48] Yilin Wang, Sasi Inguva, and Balu Adsumilli.  Youtube ugc dataset for video compression research.  In 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP), pages 1–5. IEEE, 2019.
  * [49] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [50] Stephen B Wicker and Vijay K Bhargava.  Reed-Solomon codes and their applications.  John Wiley & Sons, 1999.
  * [51] Kejun Wu, Yi Wang, Wenyang Liu, Kim-Hui Yap, and Lap-Pui Chau.  A spatial-focal error concealment scheme for corrupted focal stack video.  In 2023 Data Compression Conference (DCC), pages 91–100, 2023.
  * [52] Chongyang Xiang, Jiajun Xu, Chuan Yan, Qiang Peng, and Xiao Wu.  Generative adversarial networks based error concealment for low resolution video.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1827–1831. IEEE, 2019.
  * [53] Rui Xu, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy.  Deep flow-guided video inpainting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3723–3732, 2019.
  * [54] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman.  Video enhancement with task-oriented flow.  International Journal of Computer Vision, 127:1106–1125, 2019.
  * [55] Linjie Yang, Yuchen Fan, and Ning Xu.  The 2nd large-scale video object segmentation challenge - video object segmentation track, October 2019.
  * [56] S. Ye, M. Ouaret, F. Dufaux, and T. Ebrahimi.  Hybrid spatial and temporal error concealment for distributed video coding.  In 2008 IEEE International Conference on Multimedia and Expo, pages 633–636. IEEE, 2008.
  * [57] Peng Yi, Zhongyuan Wang, Kui Jiang, Junjun Jiang, and Jiayi Ma.  Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 3106–3115, 2019.
  * [58] Yanhong Zeng, Jianlong Fu, and Hongyang Chao.  Learning joint spatial-temporal transformations for video inpainting.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pages 528–543. Springer, 2020.
  * [59] Jiefu Zhai, Keman Yu, Jiang Li, and Shipeng Li.  A low complexity motion compensated frame interpolation method.  In 2005 IEEE International Symposium on Circuits and Systems (ISCAS), pages 4927–4930. IEEE, 2005.
  * [60] Kaidong Zhang, Jingjing Fu, and Dong Liu.  Flow-guided transformer for video inpainting.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVIII, pages 74–90. Springer, 2022.
  * [61] Kaidong Zhang, Jingjing Fu, and Dong Liu.  Inertia-guided flow completion and style fusion for video inpainting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5982–5991, 2022.
  * [62] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.


§ STUDY AREA CHOICE

The first significant design decision made during the study was the district boundaries under which the length of the road network and air pollution would be aggregated. This problem is known as the Modifiable Aerial Unit Problem (MAUP)
[Wong, David WS (2004). The modifiable areal unit problem (MAUP), WorldMinds: Geographical perspectives on 100 problems. 571–575.]. Three schemas were considered for aggregating the study area into districts, each of which was motivated by a specific design philosophy.

* Ordnance Survey (OS) National Grid (motivated by geographic consistency) Figure <ref>– Considered to give a consistent size in each district with boundary areas considered including 1km/5km/10km district areas.

* Local Authority District (LAD) (motivated by administrative and policy capture) Figure <ref>– The use of Local Authority Districts as the district boundary allowed the road network maintained by the same political entity to be aggregated. The reasoning is that local legislators decide the choice of road built within an area, and so if there were a disposition to build a particular type of road, this would be captured by the district.

* Middle Super Output Areas (MSOA) (motivated by population metrics) Figure <ref>– The districts within the MSOA are based upon the population in an area, with each district having a similar population with varying geographical size.

Ordance Survey Grids (1km / 5km / 10km) MAUP examples

Local Authority District (LAD) MAUP examples

Middle Super Output Area (MSOA) MAUP examples

§ DATA DETAILS

§.§ Air Pollution Ground Observation Data

In 2019 there were 173 functional monitoring stations across the UK in the Automatic Urban and Rural Network (AURN). Each monitoring station only measures a subset of each of the 12 pollutants of interest in this study. Figure <ref> shows the count for the number of stations online in each given year during the 2014-2019 period coloured by pollutant.

Number of active AURN monitoring stations by year

Another critical aspect of the AURN is the type of location in which the station is placed. There are six different station location types [Tony Clark et al. (2020). Automatic Urban and Rural Network: Site Operator's Manual - 6.2.3 Site Classifications, Ricardo Energy and Environment.]. The stations used in this study were also restricted to stations within the area covered by the MSOA, meaning only stations within England and Wales were included, shown in Figure <ref>. Local Authority Environmental Health Offices are responsible for running the stations. While there are locations within the MSOA closer to a monitoring station outside the MSOA boundary, it was essential to restrict the stations to the top-level political authority due to the local authority running the station. This choice was made in case of differences in standard operating procedures by health officers within different local authorities [UK AIR Information Resource (2021). Automatic Urban and Rural Network (AURN), Department for Environment, Food and Rural Affairs. <https://uk-air.defra.gov.uk/networks/network-info?view=aurn>  "[Online; accessed August-2021]"].

AURN stations within MSOA boundary

§.§ Air Pollution Data Interpolation

Figure <ref> shows the 12 different pollutants scatter plots of the values for the Root Mean Square Error (RMSE) against different values of the power parameter, used during the interpolation leave one out validations process.

Root mean square error (RMSE) scatter plots - Interpolation Power Parameter vs RMSE

Figure <ref> shows the resulting interpolated raster from the monitoring station ground observations and Figure <ref> shows the confidence interval that was calculated for the corresponding raster for each of 12 pollutants, with the given pollutants power parameter that minimised the RMSE.

Of note is that some of the pollutants, such as Carbon Monoxide (CO), had a minimal amount of ground observations monitoring stations. As such, the interpolation resulted in estimating the mean of the stations everywhere across the study area, highlighted by a power parameter of 0.

Interpolation surface for pollutants at the annual level in 2019. Black points represent pollution ground observation monitoring stations location

Confidence interval for pollutants at the annual level in 2019. Black points represent pollution ground observation monitoring stations location

§.§ Target Vector Data Limitations

Table <ref> details each MSOAs aggregate pollution value calculated from the 2019 interpolated raster using the uniform point grid with a count for the number of points sampled.

multicolumn names=l,
col sep=comma,
string type,
header = has colnames,
columns=msoa11nm, Raster Value Pollution, Count,
columns/msoa11nm/.style=column type=l, column name=MSOA District Name,
columns/Raster Value Pollution/.style=column type=S[round-precision=0, table-format=5.0, table-number-alignment=left], column name=Aggregate Pollution Sum (µg/m$^3$),
columns/Count/.style=column type=S[round-precision=4, table-format=4.0,  table-number-alignment=left], column name=Point Sample Count,
every head row/.style=before row=, after row=,
every last row/.style=after row=
Sample (every 500th MSOA by sample point count) aggregate pollution sum for MSOA district boundaries with associated number of point samples from raster, that the model aims to estimate

Only the annual target vector was successfully implemented. Due to data limitations concerning the technique of interpolating the ground observation data, the variance of the ground observation measurements was lost, resulting in a target vector not representative of the ground observations seen at the monitoring stations. Figure <ref> shows the average standard deviation of the zones in the raster that were created. Figure <ref> shows the average standard deviation between the ground observation measurements by the AURN monitoring stations. It is clear from the plots that the trends within the ground observation measurements are lost during the interpolating and sampling process. This issue was then further compounded without having a second data source verify the interpolation technique at the daily or hourly period as had been done for the annual interpolation with the DEFRA air pollution model. Therefore due to the data limitations, only the annual average pollution per district was attempted to be estimated.

Mean standard deviation in the interpolated for the annual, daily and hourly time periods

Mean standard deviation in the ground observation for the annual, daily and hourly time periods

§ MODEL VARIANT DETAILS

§.§ Model Variant Feature Vector

Table <ref> shows an example feature vector for the length model. Table <ref> shows an example feature vector for the composition model.Table <ref> shows an example feature vector for the spatial variant of the composition model.

multicolumn names=l,
col sep=comma,
string type,
header = has colnames,
columns=msoa11nm, Length,
columns/msoa11nm/.style=column type=l, column name=MSOA District Name,
columns/Length/.style=column type=S[round-precision=0, table-format=6.0,  table-number-alignment=left], column name=Total Road Length (m),
every head row/.style=before row=, after row=,
every last row/.style=after row=
Example  vector for the length model for 2019, with every 500th MSOA selected ordered by total length (m).

multicolumn names=l,
col sep=comma,
string type,
header = has colnames,
columns=msoa11nm, motorway, primary_link, secondary_link, tertiary, trunk,
columns/msoa11nm/.style=column type=l, column name=MSOA District Name,
columns/motorway/.style=column type=S[round-precision=0, table-format=5.0,  table-number-alignment=left], column name=Total Road Motorway Length (m),
columns/primary_link/.style=column type=S[round-precision=0, table-format=4.0, table-number-alignment=left], column name=Total Road Primary Link Length (m),
columns/secondary_link/.style=column type=S[round-precision=0, table-format=3.0,  table-number-alignment=left], column name=Total Road Secondary Link Length (m),
columns/tertiary/.style=column type=S[round-precision=0, table-format=5.0,  table-number-alignment=left], column name=Total Road Tertiary Length (m),
columns/trunk/.style=column type=S[round-precision=0, table-format=5.0,  table-number-alignment=left], column name=Total Road Trunk Length (m),
every head row/.style=before row=, after row=,
every last row/.style=after row=
Example feature vector for the highway composition model for 2019, with every 500th MSOA selected with a selection of key roads total length.

multicolumn names=l,
col sep=comma,
string type,
header = has colnames,
columns=msoa11nm, primary, secondary_link, steps, N_motorway, N_tertiary_link, N_trunk,
columns/msoa11nm/.style=column type=l, column name=MSOA Name,
columns/primary/.style=column type=S[round-precision=0, table-format=5.0,  table-number-alignment=left], column name=Total Road (TR) Primary  Length (m),
columns/secondary_link/.style=column type=S[round-precision=0, table-format=3.0, table-number-alignment=left], column name=TR Secondary Link Length (m),
columns/steps/.style=column type=S[round-precision=0, table-format=3.0, table-number-alignment=left], column name=TR Steps Length (m),
columns/N_motorway/.style=column type=S[round-precision=0, table-format=5.0, table-number-alignment=left], column name=TR Neighbour Motorway Length (m),
columns/N_tertiary_link/.style=column type=S[round-precision=0, table-format=3, table-number-alignment=left], column name=TR Neighbour Tertiary Link Length (m),
columns/N_trunk/.style=column type=S[round-precision=0, table-format=6.0, table-number-alignment=left], column name=TR Neighbour Trunk Length (m),
every head row/.style=before row=, after row=,
every last row/.style=after row=
Example feature vector for the spatial vector model for 2019, with every 500th MSOA selected with a selection of key roads total length.

§.§ Model Variant Example Output Estimations

Table <ref> shows example pollution predictions for the length model for different MSOAs. Table <ref> shows example pollution predictions for the composition model for different MSOAs. Table <ref> shows example pollution predictions for the spatial model for different MSOAs.

multicolumn names=l,
col sep=comma,
string type,
header = has colnames,
columns=msoa11nm, Raster Value Pollution, Predicted 2018 Length, Length Difference,
columns/msoa11nm/.style=column type=l, column name=MSOA District Name,
columns/Raster Value Pollution/.style=column type=S[round-precision=2, table-format=5.2, table-number-alignment=left], column name=Actual Pollution (µg/m$^3$),
columns/Predicted 2018 Length/.style=column type=S[round-precision=2, table-format=-5.2, table-number-alignment=left], column name=Predicted Pollution (µg/m$^3$),
columns/Length Difference/.style=column type=S[round-precision=2, table-format=4.2, table-number-alignment=left], column name=Difference (µg/m$^3$),
every head row/.style=before row=, after row=,
every last row/.style=after row=
Results from the length model with every 500th MSOA shown and sorted by the difference between predicted and actual pollution.

multicolumn names=l,
col sep=comma,
string type,
header = has colnames,
columns=msoa11nm, Raster Value Pollution, Predicted 2018 Composition, Composition Difference,
columns/msoa11nm/.style=column type=l, column name=MSOA District Name,
columns/Raster Value Pollution/.style=column type=S[round-precision=2, table-format=4.2, table-number-alignment=left], column name=Actual Pollution (µg/m$^3$),
columns/Predicted 2018 Composition/.style=column type=S[round-precision=2, table-format=-5.2, table-number-alignment=left], column name=Predicted Pollution (µg/m$^3$),
columns/Composition Difference/.style=column type=S[round-precision=2, table-format=4.2, table-number-alignment=left], column name=Difference (µg/m$^3$),
every head row/.style=before row=, after row=,
every last row/.style=after row=
Results from the highway composition model with every 500th MSOA shown and sorted by the difference between predicted and actual pollution.

multicolumn names=l,
col sep=comma,
string type,
header = has colnames,
columns=msoa11nm, Raster Value Pollution, Predicted 2018 Spatial, Spatial Difference,
columns/msoa11nm/.style=column type=l, column name=MSOA Name,
columns/Raster Value Pollution/.style=column type=S[round-precision=2, table-format=4.2, table-number-alignment=left], column name=Actual Pollution (µg/m$^3$),
columns/Predicted 2018 Spatial/.style=column type=S[round-precision=2, table-format=-4.2, table-number-alignment=left], column name=Predicted Pollution (µg/m$^3$),
columns/Spatial Difference/.style=column type=S[round-precision=2, table-format=4.2, table-number-alignment=left], column name=Difference (µg/m$^3$),
every head row/.style=before row=, after row=,
every last row/.style=after row=
Results from the spatial model with every 500th MSOA shown and sorted by the difference between predicted and actual pollution.

§.§ Model Feature Selection

Table <ref> shows the mutual information between every 5th highway type and all 12 pollutants, with the final column, summation, providing the total mutual information for that highway type across each of the pollutants. The mutual information is also normalised between 0 and 1 and have been ordered from least to most total mutual information.

multicolumn names=l,
col sep=comma,
string type,
header = has colnames,
columns=Highway Type, MI PollutionCO, MI PollutionNO, MI PollutionNO2, MI PollutionNOX, MI PollutionNV10, MI PollutionNV25, MI PollutionO3, MI PollutionPM10, MI PollutionPM25, MI PollutionSO2, MI PollutionV10, MI PollutionV25, Summation,
columns/Highway Type/.style=column type=l, column name=Highway Type,
columns/MI PollutionCO/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=Mutual Information (MI) CO,
columns/MI PollutionNO/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name= MI NO,
columns/MI PollutionNO2/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=MI NO2,
columns/MI PollutionNOX/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=MI NOX,
columns/MI PollutionNV10/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=MI NV10,
columns/MI PollutionNV25/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=MI NV25,
columns/MI PollutionO3/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=MI O3,
columns/MI PollutionPM10/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=MI PM10,
columns/MI PollutionPM25/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=MI PM25,
columns/MI PollutionSO2/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=MI SO2,
columns/MI PollutionV10/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=MI V10,
columns/MI PollutionV25/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=MI V25,
columns/Summation/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=2.2, table-number-alignment=left], column name=Summation Mutual Information,
every head row/.style=before row=, after row=,
every last row/.style=after row=
Mutual information from every 5th highway type in relation to every pollutant ordered by total mutual information.

Table <ref> shows results from repeating the experiments for all three models while reducing the input data set through the removal of specific highway types reducing the data used to create the feature vector. The highway types that were removed were chosen by incrementing a threshold value and removing highway types that had a summation mutual information value below the threshold.

multicolumn names=l,
col sep=comma,
string type,
header = has colnames,
columns=R2 Length, R2Composition, R2 Spatial, Threshold Value, Highway Type Length,
columns/R2 Length/.style=column type=S[round-precision=2, table-format=1.2, table-number-alignment=left], column name=$R^2$ Score Length Model,
columns/R2Composition/.style=column type=S[round-precision=2, table-format=1.2, table-number-alignment=left], column name=$R^2$ Score Composition Model,
columns/R2 Spatial/.style=column type=S[round-precision=2, table-format=1.2, table-number-alignment=left], column name=$R^2$ Score Spatial Model,
columns/Threshold Value/.style=column type=S[round-precision=1, round-integer-to-decimal, table-format=-2.1, table-number-alignment=left], column name=Threshold Value,
columns/Highway Type Length/.style=column type=S[round-precision=0, table-format=2.0, table-number-alignment=left], column name=No. of Highway Types,
every head row/.style=before row=, after row=,
every last row/.style=after row=
Results from rerunning the three models with a reduced feature vector, with inclusion based on the total mutual information being higher than the threshold value.

Table <ref> shows the results from the same tests as Table <ref>, however with the starting set of highway types being restricted to highway types that have a mutual information value with every pollutant. The initial set of highway types numbered 22.

multicolumn names=l,
col sep=comma,
string type,
header = has colnames,
columns=R2 Length, R2Composition, R2 Spatial, Threshold Value, Highway Type Length,
columns/R2 Length/.style=column type=S[round-precision=2, table-format=1.2, table-number-alignment=left], column name=$R^2$ Score Length Model,
columns/R2Composition/.style=column type=S[round-precision=2, table-format=1.2, table-number-alignment=left], column name=$R^2$ Score Composition Model,
columns/R2 Spatial/.style=column type=S[round-precision=2, table-format=1.2, table-number-alignment=left], column name=$R^2$ Score Spatial Model,
columns/Threshold Value/.style=column type=S[round-precision=1, round-integer-to-decimal, table-format=-2.1, table-number-alignment=left], column name=Threshold Value,
columns/Highway Type Length/.style=column type=S[round-precision=0, table-format=2.0, table-number-alignment=left], column name=No. of Highway Types,
every head row/.style=before row=, after row=,
every last row/.style=after row=
Results from rerunning the three models with a reduced feature vector, with inclusion based on the total mutual information being higher than the threshold value. The initial set includes only highway types that have relevance to all pollutants.

Table <ref> shows the non normalised mutual information values used in the heatmap.

multicolumn names=l,
col sep=comma,
string type,
header = has colnames,
columns=Highway Type,living street,no,primary link,raceway,motorway link,pedestrian,trunk link, steps,road,motorway,cycleway,residential,primary,path,trunk,secondary,bridleway,service,tertiary, footway,unclassified,track,
columns/Highway Type/.style=column type=l, column name=Highway Type,
columns/living street/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=living street,
columns/no/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=no,
columns/primary link/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=primary link,
columns/raceway/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=raceway,
columns/motorway link/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=motorway link,
columns/pedestrian/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=pedestrian,
columns/trunk link/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=trunk link,
columns/steps/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=steps,
columns/road/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=road,
columns/motorway/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=motorway,
columns/cycleway/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=cycleway,
columns/residential/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=residential,
columns/primary/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=primary,
columns/path/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=path,
columns/trunk/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=trunk,
columns/secondary/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=secondary,
columns/bridleway/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=bridleway,
columns/service/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=service,
columns/tertiary/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=tertiary,
columns/footway/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=footway,
columns/unclassified/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=unclassified,
columns/track/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=1.2, table-number-alignment=left], column name=track,
every head row/.style=before row=, after row=,
every last row/.style=after row=
Non normalised inter road mutual information matrix.

Figure <ref> shows the full network representation of the 22 highway types with non zero mutual information with all 12 pollutants, and Figure <ref> the network state at key stages during the removal process for reducing the highway type inclusion list, detailed in Table <ref>.

Network model of the inter road mutual information

Removal Process for Inter Road Mutual Information Network

Removal process for network model of the inter road mutual information

multicolumn names=l,
col sep=comma,
string type,
header = has colnames,
columns=R2 Length, R2Composition, Inter Road Cumulative MutualInformation, Highway Type Length List, Road List,
columns/R2 Length/.style=column type=S[round-precision=2, table-format=1.2, table-number-alignment=left], column name=$R^2$ Length,
columns/R2Composition/.style=column type=S[round-precision=2, table-format=1.2, table-number-alignment=left], column name=$R^2$ Composition,
columns/Inter Road Cumulative MutualInformation/.style=column type=S[round-precision=2, table-format=2.2, table-number-alignment=left], column name=Inter Road Cumulative Mutual Information,
columns/Highway Type Length List/.style=column type=S[round-precision=0, table-format=2.0, table-number-alignment=left], column name=No. of Highway Types,
columns/Road List/.style=column type=l, column name=New Highway Type Included,
every head row/.style=before row=, after row=,
every last row/.style=after row=
Inter road mutual information tests.

§ INTER YEAR TESTS DETAILS

Figure <ref> shows the visualisation of the MSOAs selected for inclusion in the training set during the inter year tests. Table <ref> shows the population density for a range of MSOAs. Table <ref> details the results from the running of the length and composition model with a set of MSOAs restricted by population density distribution.

Visualisation of Random State 12345 Test and Training Regions

Visualisation of changing MSOA test and training sets

multicolumn names=l,
col sep=comma,
string type,
header = has colnames,
columns=msoa11nm, Population Density, Population Density Normalized,
columns/msoa11nm/.style=column type=l, column name=MSOA Name,
columns/Population Density/.style=column type=S[table-format=1.2e-1, round-precision=2, scientific-notation = true, round-integer-to-decimal, table-number-alignment=left], column name=Population Density (Individuals / m$^2$),
columns/Population Density Normalized/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=3.2, table-number-alignment=left], column name=Population Density Normalized (between 0-100),
every head row/.style=before row=, after row=,
every last row/.style=after row=
MSOA population density for every 900th MSOA.

multicolumn names=l,
col sep=comma,
string type,
header = has colnames,
columns=Test Region Size, Population Density Range Mean, R2 Length Mean, R2 Composition Mean,
columns/Test Region Size/.style=column type=S[round-precision=0, table-format=4.0, table-number-alignment=left], column name=Test Set Size,
columns/Population Density Range Mean/.style=column type=S[round-precision=2, round-integer-to-decimal, table-format=2.2, table-number-alignment=left], column name=Population Density Range Mean (Individuals / m$^2$),
columns/R2 Length Mean/.style=column type=S[round-precision=2, table-format=-1.2, table-number-alignment=left], column name=$R^2$ Length Model Mean,
columns/R2 Composition Mean/.style=column type=S[round-precision=2,scientific-notation = true, table-format=-1.2e-1, table-number-alignment=left], column name=$R^2$ Composition Model Mean,
every head row/.style=before row=, after row=,
every last row/.style=after row=
Length and composition model results from inter year tests from 16 random state tests.