# I2VC: A Unified Framework for
Intra- & Inter-frame Video Compression

Meiqin Liu
Institute of Information Science
Beijing Jiaotong University
Beijing, China 100044
<EMAIL_ADDRESS>&Chenming Xu∗
Institute of Information Science
Beijing Jiaotong University
Beijing, China 100044
<EMAIL_ADDRESS>&YuKai Gu∗
Institute of Information Science
Beijing Jiaotong University
Beijing, China 100044
<EMAIL_ADDRESS>Chao Yao
School of Computer and Communication Engineering
University of Science and Technology Beijing
Beijing, China 100083
<EMAIL_ADDRESS>
&Yao Zhao
Institute of Information Science
Beijing Jiaotong University
Beijing, China 100044
<EMAIL_ADDRESS>
Equal contribution Corresponding author

###### Abstract

Video compression aims to reconstruct seamless frames by encoding the motion
and residual information from existing frames. Previous neural video
compression methods necessitate distinct codecs for three types of frames
(I-frame, P-frame and B-frame), which hinders a unified approach and
generalization across different video contexts. Intra-codec techniques lack
the advanced Motion Estimation and Motion Compensation (MEMC) found in inter-
codec, leading to fragmented frameworks lacking uniformity. Our proposed
Intra- & Inter-frame Video Compression (I2VC) framework employs a single
spatio-temporal codec that guides feature compression rates according to
content importance. This unified codec transforms the dependence across frames
into a conditional coding scheme, thus integrating intra- and inter-frame
compression into one cohesive strategy. Given the absence of explicit motion
data, achieving competent inter-frame compression with only a conditional
codec poses a challenge. To resolve this, our approach includes an implicit
inter-frame alignment mechanism. With the pre-trained diffusion denoising
process, the utilization of a diffusion-inverted reference feature rather than
random noise supports the initial compression state. This process allows for
selective denoising of motion-rich regions based on decoded features,
facilitating accurate alignment without the need for MEMC. Our experimental
findings, across various compression configurations (AI, LD and RA) and frame
types, prove that I2VC outperforms the state-of-the-art perceptual learned
codecs. Impressively, it exhibits a 58.4% enhancement in perceptual
reconstruction performance when benchmarked against the H.266/VVC standard
(VTM). Official implementation can be found at https://github.com/GYukai/I2VC

##  1 Introduction

Video codec is designed to achieve high-quality reconstruction with the
available transmission requirements. ITU/MPEG video coding standards, such as
AVC/H.264 [53], HEVC/H.265 [49], VVC/H.266 [5], incorporate three
configurations, including All Intra (AI), Low Delay (LD), and Random Access
(RA), to adapt to different intra- and inter-frame dependencies with three
types of frames (I-frame, P-frame and B-frame), respectively. The current
neural video compression methods [3, 35, 58] also provide different frameworks
and some methods [14, 28] provide general optimization strategies for
corresponding structures, achieving excellent compression performance.

Specifically, as depicted in Figure 1(a) and Figure 1(b), due to the
additional motion codec of inter-frame compression, different compression
frameworks [3, 35, 57] are designed to cater to specific frame types. It is
impossible to share frameworks and weights between intra- and inter-frame
compression, as well as the B-frame and P-frame of inter-frame compression,
leading to model redundancy and weak generalization across existing
frameworks. Therefore, it is necessary to propose a framework to unify intra-
and inter-frame compression. The unified framework necessitates coherence in
the compression process across three types of frames with shared certain
parameter weights. Ultimately, flexible modeling of the three types of frames
allows a unified model to adapt three compression configurations in a Group of
Pictures (GoP). Some methods [16, 50, 23] integrate inter-frame correlations
directly by using reference frames or multi-modal information as a condition
of the diffusion models [19, 48, 45, 66]. However, these methods fail to
address the challenge of a unified framework, specifically how to leverage
inter-frame information to reduce bit-rate during inter-frame compression
without Motion Estimation and Motion Compensation (MEMC). For detailed
analysis and motivation, please refer to Appendix A and Appendix B.

Figure 1: A comparison of different video compression frameworks. I2VC employs
the reference feature $\hat{y}_{\text{ref}}$ as a prior to compress the target
feature $y_{i}$ and applies diffusion inversion to generate the diffusion
start feature $y_{i}^{T}$, thereby incorporating inter-frame correlations.

In this study, we introduce I2VC, a novel framework designed to uniformly
address the compression of three types of frames (I-frame, P-frame and
B-frame) across different video compression scenarios (AI, LD and RA). Unlike
traditional methods that rely on a motion codec for inter-frame compression,
I2VC achieves an integration of intra- and inter-frame compression by
reference-based coding and iterative generation, as illustrated in Figure
1(c). Specifically, a Spatio-Temporal Variable-rate Codec (STVC) and an
Implicit Inter-frame Feature Alignment (IIFA) module are designed. STVC, in
particular, uses the spatio-temporal significance of reference features to
guide the compression of each frame type, enhancing the model’s
generalization. Due to without a motion codec, I2VC employs a Latent Diffusion
Model (LDM) [45] for subtle inter-frame alignment. Notably, for inter-frame
compression, the reference feature undergoes a Denoising Diffusion Implicit
Model (DDIM) [48] inversion, setting the stage for targeted denoising of
motion areas based on decoded features, effectively achieving unified
compression without MEMC. Comparative experiments indicate that I2VC surpasses
VTM-19.0 [25] by an impressive 58.4% in perceptual construction performance
across all configurations. In summary, our main contributions are listed as
follows.

  * •

We introduce I2VC, a unified framework for Intra- and Inter-frame video
compression. The three types of frames (I-frame, P-frame and B-frame) across
different video compression configurations (AI, LD and RA) within a GoP are
uniformly solved by one framework.

  * •

We design a conditional coding scheme codec for three types of frames,
leveraging the spatio-temporal significance of reference features to unify
intra- and inter-frame correlations at variable rates.

  * •

We use DDIM inversion on reference features to selectively denoise motion-rich
areas, ensuring temporal consistency across frames through implicit feature
alignment, bypassing the need for MEMC.

##  2 Related Work

Learned video compression adopts a similar framework to traditional
compression, still requiring corresponding networks for three types of frames
(I-frame, P-frame and B-frame). Towards learned I-frame compression, Ballé _et
al._ [2, 3] introduce a Variational Auto-Encoder (VAE) incorporating a
factorized and hyperprior entropy models. Cheng _et al._ [11] explore a
Gaussian mixture entropy model to improve rate-distortion performance. Wang
_et al._ [51] propose a dual spatial prior checkerboard context model to
improve the probability estimation. Jiang _et al._ [24] propose a multi-
reference entropy model to achieve state-of-the-art performance in I-frame
coding. Recently, the development of generative models has improved perceptual
coding performance. Mentzer _et al._ [39] firstly utilize Generative
Adversarial Network (GAN) for I-frame coding. Some methods [61, 44] leverage
the diffusion process to optimize the qualitative reconstruction quality of
images in the decoder. Careil _et al._ [6] utilize pre-trained diffusion
models to improve the generalization for low-bit-rate I-frame coding. Ma _et
al._ [37] and Muckley _et al._ [41] respectively propose a privileged end-to-
end decoder and a non-binary discriminator to improve statistical fidelity of
perceptual image compression models.

Motivated by the advancement in the I-frame codec, P-frame compression methods
including motion codec and residual codec are initially proposed by Lu _et
al._ [22, 35]. For robust inter-frame prediction, Li _et al._ [30] propose a
Deep Contextual Video Compression (DCVC) to shift the paradigm from predictive
coding to conditional coding. Sheng _et al._ [47] investigate Temporal Context
Mining (DCVC-TCM) for P-frame compression enhancement. Moreover, Li _et al._
propose a series of variable-rate video coding methods, including DCVC-HEM
[31], DCVC-DC [32], and DCVC-FM [33], which continuously optimize the
performance and efficiency of the spatio-temporal checkerboard context model
in the hyperprior entropy model, greatly improving rate-distortion performance
and the variable range of bit-rate for P-frame compression. To exploit bi-
directional correlations, some methods directly extend P-frame codecs to
B-frame by using bi-directional MEMC [7, 58, 64] or video frame interpolation
[1, 55, 60]. The utilization of bi-directional references renders B-frame
compression frameworks incompatible with P-frame. Besides, generative networks
are utilized to achieve low bit-rate video compression. Yang _et al._ [59]
employ a recurrent conditional GAN to achieve state-of-the-art rate-perception
for P-frame compression. Li _et al._ [29] achieve extreme video compression
through pre-trained diffusion-based predictive frame generation.

##  3 Methodology

The proposed framework aims to adapt to the three types of frames with a
unified framework, and further achieve Intra- and Inter-frame video
compression. Due to the absence of motion codec in inter-frame compression for
unification, using only a codec to introduce inter-frame correlations poses a
challenge. We propose to use inter-frame reference features in a conditional
form to reconstruct the current frame and reduce the bit-rate. The details are
elaborated in the following section.

###  3.1 A Unified Framework for Video Compression

Given an input video sequence
$\left\\{x_{i}{\mid}i\in\\{0,1,\ldots,n\\}\right\\}$ consisting of $n$ frames,
video compression aims to reconstruct the high-quality video sequence
$\\{\hat{x}_{i}{\mid}i\in\\{0,1,\ldots,n\\}\\}$ while maintaining a high
compression ratio. To meet the above requirements, a unified framework based
on controllable condition diffusion for Intra- & Inter-frame Video
Compression, I2VC is proposed. As illustrated in Figure 2 and Algorithm 1
(Appendix C.1), the input frame $x_{i}$ is first transformed by 4 $\times$
down-sampling convolution $\mathcal{E}(\cdot)$ to latent $y_{i}$ for
complexity reduction. Then, the codec uses the reference features to guide the
compression of each frame type. Towards different types of frames, the input
frame feature $y_{i}$ is encoded to feature $\hat{y}_{i}$ with the reference
feature $\hat{y}_{\text{ref}}$ as the prior, formulated as:

$\hat{y}_{i}=D(\left\lfloor
E(y_{i},\hat{y}_{\text{ref}})\right\rceil,\hat{y}_{\text{ref}}),$ (1)

where the encoder $E(\cdot)$ and decoder $D(\cdot)$ means the spatio-temporal
codec with auto-regressive entropy model [30] as described in Section 3.2. The
codec structure remains consistent in both intra- and inter-frame modes,
wherein the spatio-temporal importance is guided by the reference feature
$\hat{y}_{\text{ref}}$ during inter-frame compression. The codec initially
unifies three types of frames and provides reference information for inter-
frame coding without the additional modules.

Figure 2: Overview of the unified framework for Intra- & Inter-frame of Video
Compression (I2VC). The solid box represents the intra-frame compression
framework, while the dashed box represents the additional reference
information required for inter-frame compression. The input frame $x_{i}$ is
encoded to fusion feature $\hat{y}_{i}$ and is referred to
$\hat{y}_{\text{ref}}$ in inter-frame compression. The random Gaussian noise
$y_{i}^{T}$ is transited to $y_{i}^{0}$ conditioned on $\hat{y}_{i}$ in $T$
denoising steps. In inter-frame compression, the masked diffusion inversion on
reference feature $\hat{y}_{\text{ref}}$ is employed as the initial state
$y_{i}^{T}$ to achieve implicit alignment. The output frame $\hat{x}_{i}$ is
up-sampled by the pre-trained LDM decoder.

Specifically, by using different reference features $\hat{y}_{\text{ref}}$,
three types of frames can be preliminary unified into one framework. During
the I-frame compression, as illustrated in the solid line of Figure 2,
reference feature $\hat{y}_{\text{ref}}$ is not used. In P-frame compression,
the previous decoded feature $\hat{y}_{i-1}$ is served as reference feature
$\hat{y}_{\text{ref}}$, i.e. $\hat{y}_{\text{ref}}=\hat{y}_{i-1}$. For B-frame
compression, the reference feature $\hat{y}_{\text{ref}}$ is synthesized by
the previous decoded feature $\hat{y}_{i-1}$ and the following decoded feature
$\hat{y}_{i+1}$, formulated as:

$\hat{y}_{\text{ref}}=O\cdot\hat{y}_{i-1}+(1-O)\cdot\hat{y}_{i+1},$ (2)

where $O\in[0,1]$ represents the occlusion coefficient between bi-directional
features $\hat{y}_{i-1}$ and $\hat{y}_{i+1}$.

Relying solely on the above conditional coding makes it difficult to fully
utilize inter-frame information to reduce bit-rate and reconstruct high-
quality frames. Hence, three types of frames are reconstructed using
conditional generation. The fusion feature $\hat{y}_{i}$ is used as the
condition to perform Markovian dynamics $p_{\theta}(y_{i}^{0:T})$ on the
initial state $y_{i}^{T}$ between a sequence of denoising steps
$t=T\rightarrow 1(T=30)$, formulated as:

$p_{\theta}(y_{i}^{0:T}{\mid}\hat{y}_{i})=p(y_{i}^{T})\prod_{t=1}^{T}p_{\theta}(y_{i}^{t-1}{\mid}y_{i}^{t},\hat{y}_{i}),$
(3)

where $p(y_{i}^{T})=\mathcal{N}(y_{i}^{T};0,1)$ represents the initial state
of the transition for intra-frame compression. And
$p(y_{i}^{T})=\mathcal{N}(\hat{y}_{\text{ref}}^{T^{\prime}};\mu_{\theta}(\hat{y}_{\text{ref}}^{T^{\prime}},T^{\prime}),\sigma_{\theta}^{2}(\hat{y}_{\text{ref}}^{T^{\prime}},T^{\prime}))$
denotes to conduct $\hat{y}_{\text{ref}}$ for DDIM inversion to
$\hat{y}_{\text{ref}}^{T^{\prime}}$ as the initial state in inter-frame
compression. $T^{\prime}=\frac{1}{2}T$ means the inverse noise level.
$p_{\theta}(y_{i}^{t-1}{\mid}y_{i}^{t})$ denotes the transition kernel through
pre-trained U-Net [46] in LDM [45], formulated as:

$p_{\theta}(y_{i}^{t-1}{\mid}y_{i}^{t})=\mathcal{N}(y_{i}^{t-1};\mu_{\theta}(y_{i}^{t},t,\hat{y}_{i}),\sigma_{\theta}^{2}(y_{i}^{t},t,\hat{y}_{i})),$
(4)

where the fusion feature $\hat{y}_{i}$ and the initial state
$\hat{y}^{T^{\prime}}_{\text{ref}}$ share similar spatial and semantic
structure, differing only in temporal motion dynamics. During the
approximation for the mean $\mu_{\theta}$ and variance $\sigma_{\theta}^{2}$
of the target feature $y_{i}$, $y_{i}^{t}$ is aligned to $y_{i}^{0}$
conditioned on $\hat{y}_{i}$, achieving implicit inter-frame feature
alignment. During inter-frame compression, this strategy incorporates inter-
frame correlations to reduce the bit-rate, achieving high-quality frame
reconstruction without MEMC. The details are described in Section 3.3. At
last, a $4\times$ up-sampling $\mathcal{D}(\cdot)$ of pre-trained LDM is
performed on the maximum likelihood feature $y^{0}_{i}$ to obtain the output
frame $\hat{x}_{i}$.

###  3.2 Spatio-Temporal Variable-rate Codec

As described in Section 3.1 and Figure 2, we propose a Spatio-Temporal
Variable-rate Codec (STVC) to redefine the correlation of intra- and inter-
frame in a unified network. Specifically, four Spatio-Temporal Guidance Units
(STGU) are conducted to use the significance of reference features across each
frame type. For intra-frame compression, only the self-spatial importance of
the target feature $y_{i}$ is adopted for feature compression, as shown in
Figure 3. To learn long-term dynamics from historical observations in inter-
frame compression, the reference feature $\hat{y}_{\text{ref}}$ is deployed to
perform spatio-temporal attention in STGU. Initially, the spatio-temporal
importance-guided mask $\omega_{i}$ is conditioned on $\hat{y}_{\text{ref}}$,
formulated as:

$\omega_{i}=\sigma(\operatorname{Conv}(\operatorname{Concat}(f_{i},\hat{y}_{\text{ref}}))),$
(5)

where $f_{i}$ is the concatenation of $\hat{y}_{\text{ref}}$ and $y_{i}$.
$\operatorname{Concat}(\cdot)$ denotes the concatenation function.
$\operatorname{Conv}(\cdot)$ denotes a 3$\times$3 convolution with ReLU.
$\sigma(\cdot)$ means the sigmoid function. The spatial correlations and
temporal dynamics of the target feature $y_{i}$ are selected by the mask
$\omega_{i}$ as $\tilde{f}_{i}$:

$\tilde{f}_{i}=f_{i}+\omega_{i}\cdot\operatorname{Conv}(f_{i}).$ (6)

The above Equation 5 and Equation 6 constitute the spatio-temporal attention
mechanism, which preliminary introduces reference features in the form of
conditional coding. Besides, the mask $\omega_{i}$ is employed to scale the
feature $\tilde{f}_{i}$ with the variable-rate coefficient $\lambda$,
formulated as:

$\hat{f}_{i}=\operatorname{MLP}(\operatorname{Concat}(\omega_{i},\lambda))\cdot\tilde{f}_{i},$
(7)

where $\operatorname{MLP}(\cdot)$ means the multi-layer perceptron.
$\hat{f}_{i}$ denotes the scaled feature that preserves spatio-temporal
deformations for subsequent implicit inter-frame alignment.

Figure 3: Overview of the Spatio-Temporal Variable-rate Codec (STVC). The
target feature $y_{i}$ is scaled and transmitted as the fusion feature
$\hat{y}_{i}$ with spatio-temporal attention on the reference feature
$\hat{y}_{\text{ref}}$. The outputs also include the spatio-temporal
importance-guided mask $\omega_{i}$.

###  3.3 Implicit Inter-frame Feature Alignment

After obtaining decoded features, we use conditional diffusion to generate
three types of frames. It can integrate intra- and inter-frame dependency, but
inevitably cause the limitation of temporal consistency. Therefore, as
illustrated in Figure 2 and Algorithm 1, DDIM inversion is implemented to
introduce inter-frame dependency for Implicit Inter-frame Feature Alignment
(IIFA) without MEMC.

In detail, the DDIM inversion in inter-frame compression on the reference
feature $\hat{y}_{\text{ref}}$ is guided by the spatio-temporal mask
$\omega_{i}$, formulated as:

$\hat{y}_{\text{ref}}^{t}=\sqrt{\alpha^{t}}\frac{\hat{y}_{\text{ref}}^{t-1}-\sqrt{1-\alpha^{t-1}}\omega_{i}\varepsilon_{\theta}}{\sqrt{\alpha^{t-1}}}+\sqrt{1-\alpha^{t}}\omega_{i}\varepsilon_{\theta},$
(8)

where $\hat{y}^{t}_{\text{ref}}$ represents the noised feature that has
undergone one DDIM inversion based on $\hat{y}^{t-1}_{\text{ref}}$, with the
initial state $\hat{y}^{0}_{\text{ref}}=\hat{y}_{\text{ref}}$. $\alpha^{t-1}$
and $\alpha^{t}$ denote the weight coefficients during the inversion kernel.
While a sequence of inverse steps (set as $t=1\rightarrow T^{\prime}$), IIFA
retains similar low-frequency contextual structures and adds noise to high-
frequency motion dynamics between the reference feature $\hat{y}_{\text{ref}}$
and the target feature $y_{i}$. Then, the initial state
$y_{i}^{T}\backsim\mathcal{N}(0,1)$ in I-frame compression for denoising
transition is replaced by the final noised feature
$\hat{y}^{T^{\prime}}_{\text{ref}}$ in the inter-frame compression.

This strategy allows step-by-step implicit temporal variations compensation
from $y^{T}_{i}$ to $y^{0}_{i}$ using the denoising kernel (Equation 4)
conditioned on the fusion feature $\hat{y}_{i}$, which includes spatial
correlations from $y_{i}$ and temporal dynamics from $\hat{y}_{\text{ref}}$.
Besides, the inverse steps $T^{\prime}$ is related to the motion complexity
between $\hat{y}_{\text{ref}}$ and $y_{i}$. Therefore, we set
$T^{\prime}=\frac{1}{2}T$ to improve the robustness of the inter-frame
compression model. More analysis and discussion on implicit inter-frame
alignment are described in Appendix C.2.

Figure 4: Rate-perception comparison of different video compression methods
with three configurations on Kodak, JCT-VC, and UVG datasets in terms of
LPIPS$\downarrow$.

###  3.4 Objective Function

We use unified object functions $\mathcal{L}$ to ensure Rate-Distortion (R-D)
and Rate-Perception (R-P) trade-off across different compression
configurations, formulated as:

$\mathcal{L}=\mathcal{R}(\hat{y}_{i})+\lambda\cdot(\mathcal{D}(x_{i},\hat{x}_{i})+\beta\cdot\mathcal{P}(x_{i},\hat{x}_{i})),$
(9)

where $\lambda$ and $\beta$ denote the scale factors among bit-rate
$\mathcal{R}$, distortion $\mathcal{D}$ and perception $\mathcal{P}$,
respectively. Thanks to the proposed variable-rate codec, only one model is
trained with $\lambda\in[8,512]$ for Mean Squared Errors (MSE) distortion and
$\beta=0.05$ for Learned Perceptual Image Patch Similarity (LPIPS) [67]. The
specific training strategy is described in Appendix C.3.

##  4 Experiments

###  4.1 Settings

We adopt the same training dataset Vimeo-90K Septuplet [56] as the previous
method [30] for a fair comparison. The Kodak [15], JCT-VC (HEVC Class B, C and
D) [4] and UVG [40] datasets are utilized to evaluate the performance of I2VC.
We make a comparison with traditional codec VTM-19.0 [25] and state-of-the-art
learned codecs across AI, LDP, LDB and RA configurations. Specifically, the
learned I-frame compression method DGML [11], learned generative I-frame
compression method HiFiC [39] and CDC [61], learned P-frame compression method
DCVC-HEM [31], DCVC-DC [32], and DCVC-FM [33], learned generative P-frame
compression method PLVC [59], learned B-frame compression method IBVC [55] and
its LPIPS model IBVC-LPS [55] are considered for comparisons. We test 32
frames of each video sequence for inter-frame compression. The GoP size is set
as 32. Precisely, there are 32 I-frame in I2VC-AI; 1 I-frame, 31 P-frame in
I2VC-LDP; 1 I-frame and 6 P-frame and 25 B-frame in I2VC-LDB; 2 I-frame and 30
B-frame in I2VC-RA. All experiments are performed on 4 NVIDIA GeForce RTX 3090
GPUs with Intel(R) Xeon(R) Gold 6248R CPUs.

###  4.2 Evaluation against State-of-the-art Methods

####  4.2.1 Quantitative Evaluations

It is noted in Figure 4 that I2VC achieves state-of-the-art R-P performance on
test datasets compared to VTM-19.0 [25] and deep generative compression
methods [39, 59]. Especially, I2VC reaches an average of 58.4% perceptual
improvements than VTM-19.0 [25] with a similar bit-rate across different
configurations. Besides, we compare the R-D performance of I2VC-AI and other
intra-frame compression methods on Kodak dataset. As displayed in Figure 5,
I2VC-AI can reach comparable MS-SSIM [52] and even better PSNR than the
training required diffusion-based CDC [61]. Considering inter-frame
compression, the pre-trained diffusion-based method [29] achieves a
reconstruction quality of 24.47dB using 0.06 Bits per pixel (Bpp) on UVG
dataset, while I2VC achieves an average reconstruction quality of 31.03dB
across three configurations with the same bit-rate. It demonstrates that I2VC
maintains the better fidelity of reconstructed videos while leveraging pre-
trained models. The experimental results indicate that I2VC can achieve
generalized perceptual reconstruction with acceptable fidelity across
different configurations with a unified framework. Furthermore, we display
Fréchet Inception Distance (FID) [17] performance in Figure 9 of Appendix D.1.

####  4.2.2 Qualitative Evaluation

We provide a qualitative comparison of Kodak and HEVC Class C datasets between
I2VC and other methods [25, 33, 39, 61] as depicted in Figure 6. For AI
configuration on Kodak dataset, I2VC exhibits fewer artifacts and achieves
similar visual quality with a lower bit-rate compared to VTM-19.0 (AI default)
[25]. For LD and RA configuration on Kodak dataset, I2VC achieves richer
visually pleasing textures than VTM-19.0 [25] with similar bit-rate and DCVC-
FM [33] with 1.3× the bit-rate of ours. It can be observed that VTM-19.0 [25]
loses some texture details of floor gaps in the second example. In contrast,
our model achieves sharp boundaries and realistic textures without excessive
smoothing and motion artifacts. It verifies the temporal consistency of I2VC
in inter-frame coding. We provide more visual examples in Figure 10 of
Appendix D.2.

Figure 5: Rate-distortion comparison of different intra-frame compression
methods on Kodak dataset in terms of PSNR$\uparrow$ and MS-SSIM$\uparrow$.
Figure 6: Qualitative comparison of different video compression configurations
on Kodak and HEVC Class C datasets.

###  4.3 Ablation Study

####  4.3.1 Ablation of the Unified Framework for Inter-frame Video
Compression

To verify the effectiveness of I2VC, we conduct three ablations on HEVC Class
C dataset regarding the LDP configuration. Model A (w/o Ref) is designed to
investigate the impact of STVC which utilizes reference features for
unification. Concretely, $\hat{y}_{\text{ref}}$ is not used as a prior
condition in Equation 1, the inter-frame compression is replaced by intra-
frame compression, as $\hat{y}_{i}=D(\left\lfloor E(y_{i})\right\rceil)$.
Model B (w/o Inv) is deployed to verify the effectiveness of IIFA on the
reference feature $\hat{y}_{\text{ref}}$. Random noise $\varepsilon\sim
N(0,1)$ is used as initial state for inter-frame compression. Additionally, we
conduct Model C (w/o Ref & Inv) without the reference feature
$\hat{y}_{\text{ref}}$ for STVC and IIFA. As depicted in Figure 7(a), I2VC-LDP
saves bit-rate with the same reconstruction quality as the ablation models.
And it indicates that the reference feature $\hat{y}_{\text{ref}}$ is vital to
I2VC to reduce the bit-rate and improve the perception quality.

(a) Ablation of the proposed inter-frame framework.

(b) Ablation of the masked DDIM inversion steps.

(c) Ablation of the bi-directional inter-frame compression.

Figure 7: Ablations of the proposed inter-frame framework, the masked DDIM
inversion steps, and the bi-directional inter-frame compression on HEVC Class
C dataset.

####  4.3.2 Ablation of the Implicit Inter-frame Feature Alignment

Model B (w/o Inv) in Section 4.3.1 has confirmed that the DDIM inversion can
introduce reference information to optimize the denoising process.
Furthermore, Model 1 to 4 under the LDP configuration on HEVC Class C dataset
with DDIM inversion steps of $0,\frac{1}{3}T,\frac{2}{3}T$, and $T$ are
conducted to validate the impact of DDIM inversion for IIFA. As shown in
Figure 7(b), I2VC-LDP with $T^{\prime}=\frac{1}{2}T$ achieves the best R-P
performance compared to Models 1 to 4. As depicted in Figure 8, I2VC-LDP
($T^{\prime}=\frac{1}{2}T$) has a clearer structural definition than Model 1
($T^{\prime}=0$) and preserves more semantic contents than Model 4
($T^{\prime}=T$). Therefore, the appropriate number of DDIM inversion steps on
the reference feature $\hat{y}_{\text{ref}}$ enables to add noise on the
temporal dynamics with structural maintenance, providing a reference for
inter-frame alignment in subsequent transition. Besides, we conduct an
ablation of direct frame reconstruction from the fusion feature $\hat{y}_{i}$
without diffusion denoising, but the reconstruction performance is
unacceptable and therefore not presented. Therefore, it demonstrates that the
proposed IIFA can fuse multi-frame information for video compression.

Figure 8: Visual example of $\hat{y}^{T^{\prime}}_{\text{ref}}$ about DDIM
inversion from PartyScene in HEVC Class C dataset.

####  4.3.3 Ablation of Bi-directional Inter-frame Compression

To investigate the impact of the occlusion coefficient on bi-directional
inter-frame compression, Model I-LDB (w/o Occ) and Model II-RA (w/o Occ) is
set up on HEVC Class $C$ dataset. Both models exclude occlusion coefficients
for direct fusion of bi-directional inter-frame reference features. As shown
in Figure 7(c), I2VC-LDB and I2VC-RA achieve the same perceptual quality while
bit-rate saving compared to Model I and Model II. It demonstrates the
effectiveness of the occlusion coefficient in the fusion of bi-directional
inter-frame reference information.

To investigate the influence of the GoP structures on the bi-directional
compression, we vary the number of P-frame in the LDB configuration and
I-frame in the RA configuration. I2VC-LDB (P-frame=6) has GOP length of 32
with 6 P-frame. Model III-LDB (P-frame=1) has a GoP length of 32 with 1
P-frame. Model IV-LDB (P-frame=11) has a GoP length of 32 with 11 P-frame. As
depicted in Figure 7(c), I2VC-LDB achieves the optimal reconstruction quality,
especially at a lower bit-rate. The experimental results demonstrate that an
appropriate GoP structure can regulate the distance between the reference
frame and the target frame, ensuring a trade-off between bit-rate and
reconstruction quality. The GoP length of I2VC-RA (I-frame=2), Model V-RA
(I-frame=7) and Model VI-RA (I-frame=12) is 32, and the number of I-frame is
2, 7 and 12, respectively. As shown in Figure 7(c), I2VC-RA achieves the best
reconstruction results, saving about 33% bit-rate compared to Model V with the
same reconstruction quality. It demonstrates that I2VC-RA can sufficiently
leverage bi-directional inter-frame correlations to save bit-rate.

##  5 Conclusion and Limitation

Current video compression methods require training distinct frameworks for
different configurations with three types of frames, resulting in model
redundancy and weak generalization. Thus, we propose a unified framework for
Intra- & Inter-frame Video Compression (I2VC), including a single spatio-
temporal variable-rate codec and an implicit inter-frame alignment. Initially,
the unified codec inter-frame dependency into conditional coding referred to
decoded reference features, preliminary integrating intra- and inter-frame
correlations to one framework. Subsequently, to resolve the absence of motion
information, reference features are applied for DDIM inversion as the initial
state of diffusion denoising instead of random noise. It allows for selective
denoising of motion-rich regions based on decoded features, facilitating
implicit inter-frame alignment without MEMC. The experimental results across
three configurations demonstrate that I2VC achieves an average of 58.4%
perceptual improvements over VTM-19.0 with the same bit-rate and exhibits
superior R-P performance compared to other learned video compression methods.
However, I2VC primarily relies on pre-trained LDM constrained by training
complexity and device memory. As a result, it exhibits certain limitations in
terms of different metrics. Future research will aim to train the complete
diffusion and inverse process to enhance reconstruction performance and
generalization.

## References

  * [1] David Alexandre, Hsueh-Ming Hang, and Wen-Hsiao Peng.  Hierarchical B-frame video coding using two-layer CANF without motion coding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10249–10258, 2023.
  * [2] Johannes Ballé, Valero Laparra, and Eero P Simoncelli.  End-to-end optimized image compression.  In Proceedings of the International Conference on Learning Representations (ICLR), pages 1–27, 2017.
  * [3] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston.  Variational image compression with a scale hyperprior.  In Proceedings of the International Conference on Learning Representations (ICLR), pages 1–23, 2018.
  * [4] Frank Bossen et al.  Common test conditions and software reference configurations.  JCTVC-L1100, 12(7):1, 2013.
  * [5] Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J Sullivan, and Jens-Rainer Ohm.  Overview of the versatile video coding (VVC) standard and its applications.  IEEE Transactions on Circuits and Systems for Video Technology, 31(10):3736–3764, 2021.
  * [6] Marlène Careil, Matthew J Muckley, Jakob Verbeek, and Stéphane Lathuilière.  Towards image compression with perfect realism at ultra-low bitrates.  In Proceedings of the International Conference on Learning Representations (ICLR), pages 1–21, 2024.
  * [7] Eren Çetin, M Akın Yılmaz, and A Murat Tekalp.  Flexible-rate learned hierarchical bi-directional video compression with motion refinement and frame-level bit allocation.  In Proceedings of the IEEE International Conference on Image Processing (ICIP), pages 1206–1210, 2022.
  * [8] Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra.  Pix2video: Video editing using image diffusion.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 23206–23217, 2023.
  * [9] Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu.  Stablevideo: Text-driven consistency-aware diffusion video editing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 23040–23050, 2023.
  * [10] Tsai-Shien Chen, Chieh Hubert Lin, Hung-Yu Tseng, Tsung-Yi Lin, and Ming-Hsuan Yang.  Motion-conditioned diffusion model for controllable video synthesis.  arXiv preprint arXiv:2304.14404, 2023.
  * [11] Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto.  Learned image compression with discretized Gaussian mixture likelihoods and attention modules.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7939–7948, 2020.
  * [12] Duolikun Danier, Fan Zhang, and David Bull.  LDMVFI: Video frame interpolation with latent diffusion models.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1472–1480, 2024.
  * [13] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7346–7356, 2023.
  * [14] Runsen Feng, Zongyu Guo, Zhizheng Zhang, and Zhibo Chen.  Versatile learned video compression.  arXiv preprint arXiv:2111.03386, 2021.
  * [15] R. Franzen.  Kodak lossless true color image suite, 2013.  http://r0k.us/graphics/kodak/.
  * [16] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood.  Flexible diffusion modeling of long videos.  In Advances in Neural Information Processing Systems (NeurIPS), pages 27953–27965, 2022.
  * [17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in Neural Information Processing Systems (NeurIPS), pages 6629–6640, 2017.
  * [18] Jonathan Ho, William Chan, Chitwan Saharia, et al.  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303, 2022.
  * [19] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In Advances in Neural Information Processing Systems (NeurIPS), pages 6840–6851, 2020.
  * [20] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  In Advances in Neural Information Processing Systems (NeurIPS), pages 8633–8646, 2022.
  * [21] Yaosi Hu, Zhenzhong Chen, and Chong Luo.  LaMD: Latent motion diffusion for video generation.  arXiv preprint arXiv:2304.11603, 2023.
  * [22] Zhihao Hu, Guo Lu, and Dong Xu.  FVC: A new framework towards deep video compression in feature space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1502–1511, 2021.
  * [23] Zhihao Hu and Dong Xu.  VideoControlNet: A motion-guided video-to-video translation framework by using diffusion model with ControlNet.  arXiv preprint arXiv:2307.14073, 2023.
  * [24] Wei Jiang, Jiayu Yang, Yongqi Zhai, Peirong Ning, Feng Gao, and Ronggang Wang.  MLIC: Multi-reference entropy model for learned image compression.  In Proceedings of the ACM International Conference on Multimedia (ACM MM), pages 7618–7627, 2023.
  * [25] JVET.  VTM-19.0, 2023.  https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/.
  * [26] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15954–15964, 2023.
  * [27] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Proceedings of the International Conference on Learning Representations (ICLR), pages 1–13, 2015.
  * [28] Théo Ladune and Pierrick Philippe.  AIVC: Artificial intelligence based video codec.  In Proceedings of the IEEE International Conference on Image Processing (ICIP), pages 316–320, 2022.
  * [29] Bohan Li, Yiming Liu, Xueyan Niu, Bo Bai, Lei Deng, and Deniz Gündüz.  Extreme video compression with pre-trained diffusion models.  arXiv preprint arXiv:2402.08934, 2024.
  * [30] Jiahao Li, Bin Li, and Yan Lu.  Deep contextual video compression.  In Advances in Neural Information Processing Systems (NeurIPS), pages 18114–18125, 2021.
  * [31] Jiahao Li, Bin Li, and Yan Lu.  Hybrid spatial-temporal entropy modelling for neural video compression.  In Proceedings of the ACM International Conference on Multimedia (ACM MM), pages 1503–1511, 2022.
  * [32] Jiahao Li, Bin Li, and Yan Lu.  Neural video compression with diverse contexts.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22616–22626, 2023.
  * [33] Jiahao Li, Bin Li, and Yan Lu.  Neural video compression with feature modulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–16, 2024.
  * [34] Zhenyi Liao and Zhijie Deng.  LOVECon: Text-driven training-free long video editing with ControlNet.  arXiv preprint arXiv:2310.09711, 2023.
  * [35] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao.  DVC: An end-to-end deep video compression framework.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11006–11015, 2019.
  * [36] Shilin Lu, Yanzhu Liu, and Adams Wai-Kin Kong.  TF-ICON: Diffusion-based training-free cross-domain image composition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2294–2305, 2023.
  * [37] Yiyang Ma, Wenhan Yang, and Jiaying Liu.  Correcting diffusion-based perceptual image compression with privileged end-to-end decoder.  arXiv preprint arXiv:2404.04916, 2024.
  * [38] Kangfu Mei and Vishal Patel.  VIDM: Video implicit diffusion models.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 9117–9125, 2023.
  * [39] Fabian Mentzer, George D Toderici, Michael Tschannen, and Eirikur Agustsson.  High-fidelity generative image compression.  In Advances in Neural Information Processing Systems (NeurIPS), pages 11913–11924, 2020.
  * [40] Alexandre Mercat, Marko Viitanen, and Jarno Vanne.  UVG dataset: 50/120fps 4k sequences for video codec analysis and development.  In Proceedings of the ACM International Conference on Multimedia (ACM MM), pages 297–302, 2020.
  * [41] Matthew J Muckley, Alaaeldin El-Nouby, Karen Ullrich, Hervé Jégou, and Jakob Verbeek.  Improving statistical fidelity for neural image compression with implicit local likelihood models.  In Proceedings of the International Conference on Machine Learning (ICML), pages 25426–25443, 2023.
  * [42] Bo Peng, Xinyuan Chen, Yaohui Wang, Chaochao Lu, and Yu Qiao.  Conditionvideo: Training-free condition-guided text-to-video generation.  arXiv preprint arXiv:2310.07697, 2023.
  * [43] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.  Fatezero: Fusing attentions for zero-shot text-based video editing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15932–15942, 2023.
  * [44] Lucas Relic, Roberto Azevedo, Markus Gross, and Christopher Schroers.  Lossy image compression with foundation diffusion models.  arXiv preprint arXiv:2404.08580, 2024.
  * [45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2022.
  * [46] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional networks for biomedical image segmentation.  In Proceedings of the Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 234–241, 2015.
  * [47] Xihua Sheng, Jiahao Li, Bin Li, Li Li, Dong Liu, and Yan Lu.  Temporal context mining for learned video compression.  IEEE Transactions on Multimedia, 25:7311–7322, 2023.
  * [48] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In Proceedings of the International Conference on Learning Representations (ICLR), pages 1–20, 2021.
  * [49] Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand.  Overview of the high efficiency video coding (HEVC) standard.  IEEE Transactions on Circuits and Systems for Video Technology, 22(12):1649–1668, 2012.
  * [50] Vikram Voleti, Alexia Jolicoeur-Martineau, and Chris Pal.  MCVD-masked conditional video diffusion for prediction, generation, and interpolation.  In Advances in Neural Information Processing Systems (NeurIPS), pages 23371–23385, 2022.
  * [51] Guo-Hua Wang, Jiahao Li, Bin Li, and Yan Lu.  EVC: Towards real-time neural image compression with mask decay.  In Proceedings of the International Conference on Learning Representations (ICLR), pages 1–23, 2023.
  * [52] Zhou Wang, Eero P Simoncelli, and Alan C Bovik.  Multiscale structural similarity for image quality assessment.  In Proceedings of the Asilomar Conference on Signals, Systems & Computers, pages 1398–1402, 2003.
  * [53] T. Wiegand, G.J. Sullivan, G. Bjontegaard, and A. Luthra.  Overview of the H.264/AVC video coding standard.  IEEE Transactions on Circuits and Systems for Video Technology, 13(7):560–576, 2003.
  * [54] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7623–7633, 2023.
  * [55] Chenming Xu, Meiqin Liu, Chao Yao, Weisi Lin, and Yao Zhao.  IBVC: Interpolation-driven B-frame video compression.  Pattern Recognition, 153:110465, 2024.
  * [56] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman.  Video enhancement with task-oriented flow.  International Journal of Computer Vision, 127(8):1106–1125, 2019.
  * [57] Ren Yang, Fabian Mentzer, Luc Van Gool, and Radu Timofte.  Learning for video compression with hierarchical quality and recurrent enhancement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6628–6637, 2020.
  * [58] Ren Yang, Fabian Mentzer, Luc Van Gool, and Radu Timofte.  Learning for video compression with recurrent auto-encoder and recurrent probability model.  IEEE Journal of Selected Topics in Signal Processing, 15(2):388–401, 2020.
  * [59] Ren Yang, Radu Timofte, and Luc Van Gool.  Perceptual video compression with recurrent conditional GAN.  In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 1512–1519, 2022.
  * [60] Ren Yang, Radu Timofte, and Luc Van Gool.  Advancing learned video compression with in-loop frame prediction.  IEEE Transactions on Circuits and Systems for Video Technology, 33(5):2410–2423, 2023.
  * [61] Ruihan Yang and Stephan Mandt.  Lossy image compression with conditional diffusion models.  In Advances in Neural Information Processing Systems (NeurIPS), pages 64971–64995, 2023.
  * [62] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt.  Diffusion probabilistic modeling for video generation.  Entropy, 25(10):1469, 2023.
  * [63] Siyuan Yang, Lu Zhang, Yu Liu, Zhizhuo Jiang, and You He.  Video diffusion models with local-global context guidance.  In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), page 1640–1648, 2023.
  * [64] M Akın Yılmaz and A Murat Tekalp.  End-to-end rate-distortion optimized learned hierarchical bi-directional video compression.  IEEE Transactions on Image Processing, 31:974–983, 2021.
  * [65] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin.  Video probabilistic diffusion models in projected latent space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18456–18466, 2023.
  * [66] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3836–3847, 2023.
  * [67] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 586–595, 2018.

##  Appendix A Analysis and Motivation

###  A.1 Learned Video Compression

Current learned video compression methods [3, 35, 57] leverage various inter-
frame dependencies to achieve three different coding configurations (AI, LD
and RA), catering to different scenarios. Specifically, the AI configuration
is often used for video sequences with sparse inter-frame correlation and low
frame rates. Each frame is an I-frame and independently compressed without
reference from other frames. The VAE codec [3] is used in I-frame compression
for output frame $\hat{x}_{i}$, formulated as:

$\hat{x}_{i}={D}(\lfloor{E}(x_{i})\rceil),$ (10)

where $E(\cdot)$ denotes the encoder to compress input frame $x_{i}$ as latent
feature. $D(\cdot)$ denotes the decoder which restores latent feature back to
$\hat{x}_{i}$.

The LD configuration is used for video coding in low-latency scenarios. Three
types of frames (I-frame, P-frame and B-frame) are included in this
configuration. The RA configuration is used for random access within video
streams, comprising I-frame and B-frame, which respectively utilize
unidirectional and bi-directional inter-frame references. A hybrid compression
framework incorporating motion codec and residual codec [35, 57] is employed
in P-frame and B-frame compression, the predicted frame $\tilde{x}_{i}$ and
output frame $\hat{x}_{i}$ are respectively formulated as:

$\displaystyle\tilde{x}_{i}=W(x_{i-1},\left[x_{i+1}\right],D_{\text{m}}(\left\lfloor
E_{\text{m}}(m_{x_{i-1}\rightarrow x_{i}},\left[m_{x_{i+1}\rightarrow
x_{i}}\right])\right\rceil)),$ (11)
$\displaystyle\hat{x}_{i}=D_{\text{r}}(\left\lfloor
E_{\text{r}}(x_{i}-\tilde{x}_{i})\right\rceil)+\tilde{x}_{i},$

where $E_{\text{m}}(\cdot)$ and $D_{\text{m}}(\cdot)$ represent the motion
codec to transmit the motion information between reference frame
${x_{i-1},[x_{i+1}]}$ and input frame $x_{i}$ to obtain predicted frame
$\tilde{x}_{i}$. ${E}_{\text{r}}(\cdot)$ and ${D}_{\text{r}}(\cdot)$ represent
the residual codec to transmit the residual information between predicted
frame $\tilde{x}_{i}$ and input frame $x_{i}$ to obtain output frame
$\hat{x}_{i}$. $[\cdot]$ indicates the variable is optional.

Though there are similarities between Equation 10 and Equation 11, the inter-
frame video compression framework includes an extra motion codec compared to
the intra-frame framework. Furthermore, there are also significant differences
within the inter-frame video compression frameworks. P-frame compression
necessitates only unidirectional reference frames, but B-frame compression
requires bi-directional reference frames, as shown in Equation 11. The MEMC
and motion codec in the B-frame coding framework are more complex, making
these three frameworks incompatible.

###  A.2 Video Diffusion Model

Existing video restoration and generation methods [10, 50, 12, 42, 23]
implement DDIM [48] to achieve the fusion of inter-frame dependency. During
the process of diffusion denoising, the maximization distribution
$p_{\theta}(x_{i}^{0})$ with variables $x_{i}^{1:T}$ of input frame $x_{i}$ is
formulated as:

$p_{\theta}(x_{i}^{0})=\int p_{\theta}(x_{i}^{0:T})dx_{i}^{1:T},$ (12)

where $p_{\theta}(x_{i}^{0:T})$ denotes the Markovian dynamics between a
sequence of transitional steps $t=T\rightarrow 1$, which can be specifically
formulated as:

$p_{\theta}(x_{i}^{0:T})=p(x_{i}^{T})\prod_{t=1}^{T}p_{\theta}(x_{i}^{t-1}{\mid}x_{i}^{t}),$
(13)

where $p(x_{i}^{T})=\mathcal{N}(x_{i}^{T};0,1)$ represents the initial
distribution with random Gaussian noise.
$p_{\theta}(x_{i}^{t-1}{\mid}x_{i}^{t})$ represents the state transition
kernel using U-Net [46], formulated as:

$p_{\theta}(x_{i}^{t-1}{\mid}x_{i}^{t})=\mathcal{N}(x_{i}^{t-1};\mu_{\theta}(x_{i}^{t},t),\sigma_{\theta}^{2}(x_{i}^{t},t)),$
(14)

where $\mu_{\theta}(x_{i}^{t},t)$ and $\sigma_{\theta}^{2}(x_{i}^{t},t)$
denote the mean and variance of the transition kernel. To better estimate the
distribution $p_{\theta}(x_{i}^{t-1}{\mid}x_{i}^{t})$, the method [50]
introduces reference frames as the controllable condition, formulated as:

$p_{\theta}(x_{i}^{t-1}{\mid}x_{i}^{t})=\mathcal{N}(x_{i}^{t-1};\mu_{\theta}(x_{i}^{t},t,\tau_{\theta}),\sigma_{\theta}^{2}(x_{i}^{t},t,\tau_{\theta})),$
(15)

where $\tau_{\theta}$ denotes the controllable condition, which can be
expressed as forward reference frame $x_{i-1}$ or bi-directional reference
frame $\left\\{x_{i-1},x_{i+1}\right\\}$ for video restoration in [50].
However, using only reference frames as controllable conditions introduces
higher uncertainty, leading to the essential training with diffusion process
$q_{\theta}(x_{i}^{t}{\mid}x_{i}^{t-1})$. Other methods use motion vector
[10], optical flow [23] and depth image [23], etc. combined with multi-modal
information as the controllable condition $\tau_{\theta}$, but resulting in
unrealistic video reconstruction with weak consistency.

##  Appendix B Related Work

Currently, some methods commonly utilize Denoising Diffusion Probabilistic
Models (DDPM) [19] and DDIM [48] to model the inter-frame correlations between
video frames. Yang _et al._ [62] directly utilize DDPM to predict future video
frames. Harvey _et al._ [16] propose a diffusion model to generate long-range
video conditioned on any sampled arbitrary subset of original video frames.
Voleti _et al._ [50] introduce a general video synthesis probabilistic
diffusion conditioned on past and/or future frames. Yang _et al._ [63] present
a local-global video diffusion model using 3D convolution to capture multi-
perception conditions for video synthesis. Danier _et al._ [12] propose a
video frame interpolation framework by latent diffusion models. Mei _et al._
[38] devise a video implicit diffusion model to generate motion using the
latent map of the first and latest frames. Chai _et al._ [9] utilize a pre-
trained propagator in diffusion for video editing. Yu _et al._ [65]
efficiently transform video to low-dimensional projected latent space for
training the diffusion model within limited resources.

Some methods leverage multi-modal conditions to incorporate inter-frame
dependency relationships. Ho _et al._ [18, 20] first propose a diffusion-based
text-to-video model and develop a large text-conditioned video generation
task. Chen _et al._ [10] use intended content and dynamics from users as the
diffusion condition to synthesize video. The above methods are limited by high
training complexity. Therefore, some methods [54, 26, 36] extend the pre-
trained diffusion models [45, 66] by incorporating inter-frame temporal
information using cross-attention. Furthermore, Esser _et al._ [13] control
structure by input video and content by input text based on pre-trained image
diffusion. Hu _et al._ [23] use optical-flow as a condition of ControlNet for
video-to-video translation. Hu _et al._ [21] introduce a motion generator by
latent diffusion model for video generation. Peng _et al._ [42] disentangle
the inter-frame dynamics into scenery motion components, such as pose, depth,
and segmentation which can be the condition for pre-trained ControlNet.
Moreover, some methods [8, 34, 43] introduce DDIM inversion to ensure the
inter-frame consistency of invariant content while modifying only the edited
content in video editing. However, the multi-modal conditions and high
complexity of cross-attention pose challenges for the aforementioned methods
in effectively recovering the original video in video compression tasks.

Algorithm 1 A unified framework for intra- & inter-frame video compression

1:$x_{i},\left[\hat{y}_{i-1}\right],\left[\hat{y}_{i+1}\right]$ // Input frame
and reference features from feature buffer.

2:$\hat{x}_{i},\hat{y}_{i}$ // Output frame and fusion feature for feature
buffer.

3:$y_{i}=\mathcal{E}(x_{i})$ // Feature domain.

4:// Frame types judgment for reference feature $\hat{y}_{\text{ref}}$ and the
initial state $y_{i}^{T}$.

5:if not $\hat{y}_{i-1}$ and not $\hat{y}_{i+1}$ then // I-frame compression.

6: $\hat{y}_{\text{ref}}\Leftarrow$ None // Without reference feature.

7: $y_{i}^{T}\Leftarrow\mathcal{N}(0,1)$ // Random noise.

8:else if $\hat{y}_{i-1}$ and not $\hat{y}_{i+1}$ then // P-frame compression.

9: $\hat{y}_{\text{ref}}\Leftarrow\hat{y}_{i-1}$ // Forward reference feature.

10: $y_{i}^{T}\Leftarrow$ DDIM_Inversion
$(\hat{y}_{\text{ref}},T^{\prime},\text{U-Net})$ // Equation 8 masked DDIM
inversion.

11:else if $\hat{y}_{i-1}$ and $\hat{y}_{i+1}$ then

12: $\hat{y}_{\text{ref}}\Leftarrow
O\cdot\hat{y}_{i-1}+(1-O)\cdot\hat{y}_{i+1}$ // Bi-directional reference
features.

13: $y_{i}^{T}\Leftarrow$ DDIM_Inversion
$(\hat{y}_{\text{ref}},T^{\prime},\text{U-Net})$ // Eqution 8 masked DDIM
inversion.

14:end if

15:// Spatio-temporal variable-rate codec for controllable condition.

16:$\hat{y}_{i}\Leftarrow D(\left\lfloor
E(y_{i},\hat{y}_{\text{ref}})\right\rceil,\hat{y}_{\text{ref}})$

17:// Implicit inter-frame feature alignment using pre-trained U-Net.

18:for $t=T...1$ do

19: $y_{i}^{t-1}\Leftarrow$ DDIM_Backward
$(y_{i}^{t},t,\hat{y}_{i},\text{U-Net})$ // Transition kernel in Equation 4.

20:end for

21:$\hat{x}_{i}=\mathcal{D}(y_{i}^{0})$ // Frame reconstruction.

22:return $\hat{x}_{i},\hat{y}_{i}$

##  Appendix C Methodology

###  C.1 Algorithm

The algorithm of proposed I2VC is illustrated in Section 3. The inputs are the
target frame $x_{i}$ and the optional reference features $\hat{y}_{i-1}$ and
$\hat{y}_{i+1}$ from the feature buffer. The outputs are the reconstructed
frame $\hat{x}_{i}$ and the fusion feature $\hat{y}_{i}$.

###  C.2 Implicit Inter-frame Feature Alignment

As described in Section 3.3, the state transition at each step
$p_{\theta}(y_{i}^{t-1}{\mid}y_{i}^{t})$ is considered as a motion
compensation from the initial state $\hat{y}^{T^{\prime}}_{\text{ref}}$
towards the target feature $y_{i}$. The final denoised feature $y_{i}^{0}$ and
the approximate motion $m_{\hat{y}_{\text{ref}}\rightarrow y_{i}}$ are
formulated as:

$\displaystyle
y_{i}^{0}=\hat{y}_{\text{ref}}^{T^{\prime}}-\sum_{1}^{T}\varepsilon_{\theta}(y_{i}^{t},t,\hat{y}_{i})=\hat{y}_{\text{ref}}+\sum_{1}^{T^{\prime}}\varepsilon_{\theta}(\hat{y}_{\text{ref}}^{t},t)-\sum_{1}^{T}\varepsilon_{\theta}(y_{i}^{t},t,\hat{y}_{i}),$
(16) $\displaystyle m_{\hat{y}_{\text{ref}}\rightarrow y_{i}}\approx
y_{i}^{0}-\hat{y}_{\text{ref}}=\sum_{1}^{T^{\prime}}\varepsilon_{\theta}(\hat{y}_{\text{ref}}^{t},t)-\sum_{1}^{T}\varepsilon_{\theta}(y_{i}^{t},t,\hat{y}_{i}),$

where $\varepsilon_{\theta}(\hat{y}_{\text{ref}}^{t},t)$ and
$\varepsilon_{\theta}(y_{i}^{t},t,\hat{y}_{i})$ represent the eliminated noise
with U-Net. The motion $m_{\hat{y}_{\text{ref}}\rightarrow y_{i}}$ is
estimated and compensated through the transition from the reference feature
$\hat{y}_{\text{ref}}$ to the denoised feature $y_{i}^{0}$. The reference
feature $\hat{y}_{\text{ref}}$ provides texture information and spatial
structure for the denoised feature $y_{i}^{0}$, and the transition kernels
provide implicit motion information.

I2VC-AI (I-frame training)
---
Steps | Iterations | LR | Frames | I weight | P weight | B weight
1 | 160K | 1$e^{-4}$ | I | Pre-trained${\dagger}$ | - | -
2 | 5K | 1$e^{-5}$ | I | AI step 1${\dagger}$ | - | -
I2VC-LDP (P-frame training)
1 | 160K | 1$e^{-4}$ | I,P | AI step 2* | AI step 2${\dagger}$ | -
2 | 5K | 1$e^{-4}$ | I,P,P | AI step 2* | LDP step 1${\dagger}$ | -
3 | 5K | 1$e^{-5}$ | I,P,P | AI step 2* | LDP step 2${\dagger}$ | -
I2VC-RA (B-frame training)
1 | 5K | 1$e^{-4}$ | I,B,I | AI step 2* | - | LDP step 3${\dagger}$
2 | 5K | 1$e^{-4}$ | I,B,B,B,I | AI step 2* | - | RA step 1${\dagger}$
3 | 5K | 1$e^{-5}$ | I,B,B,B,I | AI step 2* | - | RA step 2${\dagger}$
I2VC-LDB (B-frame training)
1 | 5K | 1$e^{-4}$ | I,P,B,P | AI step 2* | LDP step 3* | RA step 3${\dagger}$
2 | 5K | 1$e^{-5}$ | I,P,B,P | AI step 2* | LDP step 3* | LDB step 1${\dagger}$
Table 1: The proposed training strategy for different compression
configurations and frame types, where * denotes that the model parameters are
loaded from the corresponding step and frozen in training, and ${\dagger}$
denotes that only the U-Net is frozen.

Consequently, it is crucial to use appropriate DDIM inversion steps
$T^{\prime}$ for implicit MEMC. The large number of steps implies that the
target feature $y_{i}$ significantly diverges from the reference feature
$\hat{y}_{\mathrm{ref}}$, posing a greater need for the motion compensation
through controlled conditions $\hat{y}_{i}$. Conversely, the small step number
indicates minimal temporal variations between the target feature $y_{i}$ and
the reference feature $\hat{y}_{\text{ref}}$, allowing for greater use of the
reference feature $\hat{y}_{\text{ref}}$ to reduce bit-rate. To standardize
the number of DDIM inversion steps and enhance the model generalization, we
set the number of steps as $T^{\prime}=\frac{1}{2}T$.

###  C.3 Training Strategy

Table 1 suggests the different step-by-step training strategies for
corresponding video compression configurations. For I-frame training, I2VC-AI
loads weights from the pre-trained LDM [45] and first warms up the network by
loss function $\mathcal{L}$ with 160K iterations with a Learning Rate (LR) of
1$e^{-4}$. I2VC-AI is further fine-tuned through 5K iterations by a learning
rate (LR) of 1$e^{-5}$. For P-frame training, I2VC-LDP is initialized by I2VC-
AI, followed by a similar warm-up and fine-tuning steps, incorporating an
additional step with an extra P-frame to assimilate inter-frame dependency.
Thanks to the generalization of the unified framework I2VC, the B-frame
training of I2VC-RA and I2VC-LDB can be simplified as fine-tuning on I2VC-LDP.
All experiments are implemented on 4 NVIDIA GeForce RTX 3090 GPUs with
Intel(R) Xeon(R) Gold 6248R CPUs. We conduct a mini-batch size of 4. The Adam
optimizer [27] is utilized with $\beta_{1}=0.9$ and $\beta_{2}=0.999$.

Figure 9: Rate-perception comparison of different video compression methods
with three configurations on Kodak, JCT-VC and UVG datasets in terms of
FID$\downarrow$. Figure 10: Qualitative comparison of different video
compression configurations on Kodak and UVG datasets.

##  Appendix D Experiments

###  D.1 Quantitative Evaluation

As described in Figure 9, it is observed that I2VC yields the acceptable rate-
perception performance in terms of FID across three configurations, but does
not achieve state-of-the-art performance like LPIPS metric. This is because
the pre-trained LDM [45] is trained using LPIPS loss, resulting in weak
generalization on other metrics. However, the current FID performance still
demonstrates the capability of I2VC to consistently perform across the three
configurations of video compression. Therefore, future work will focus on
applying the training of diffusion models based on the findings of this study
to address this limitation.

###  D.2 Qualitative Evaluation

As depicted in Figure 10. For AI configuration on Kodak dataset, I2VC exhibits
clearer wood texture with a lower bit-rate than the VTM-19.0 (AI default) [25]
and HiFiC [39]. For LD and RA configuration on UVG dataset, I2VC significantly
reconstructs the street light on the left side. However, VTM-19.0 [25] and
DCVC-FM [33] with similar bit-rate of ours do not recover this street light.
It proves that the implicit inter-frame alignment strategy of I2VC can
effectively maintain the invariant feature structure in video sequences and
maintain temporal consistency.