# Benchmarking AIGC Video Quality Assessment: A Dataset and Unified Model

Zhichao Zhang, Xinyue Li, Wei Sun, Jun Jia, Xiongkuo Min, Zicheng Zhang,
Chunyi Li, Zijian Chen, Puyi Wang, Zhongpeng Ji, Fengyu Sun, Shangling Jui,
and Guangtao Zhai

###### Abstract

In recent years, artificial intelligence (AI) driven video generation has
garnered significant attention due to advancements in stable diffusion and
large language model techniques. Thus, there is a great demand for accurate
video quality assessment (VQA) models to measure the perceptual quality of AI-
generated content (AIGC) videos as well as optimize video generation
techniques. However, assessing the quality of AIGC videos is quite challenging
due to the highly complex distortions they exhibit (e.g., unnatural action,
irrational objects, etc.). Therefore, in this paper, we try to systemically
investigate the AIGC-VQA problem from both subjective and objective quality
assessment perspectives. For the subjective perspective, we construct a Large-
scale Generated Video Quality assessment (LGVQ) dataset, consisting of 2,808
AIGC videos generated by 6 video generation models using 468 carefully
selected text prompts. Unlike previous subjective VQA experiments, we evaluate
the perceptual quality of AIGC videos from three dimensions: spatial quality,
temporal quality, and text-to-video alignment, which hold utmost importance
for current video generation techniques. For the objective perspective, we
establish a benchmark for evaluating existing quality assessment metrics on
the LGVQ dataset, which reveals that current metrics perform poorly on the
LGVQ dataset. Thus, we propose a Unify Generated Video Quality assessment
(UGVQ) model to comprehensively and accurately evaluate the quality of AIGC
videos across three aspects using a unified model, which uses visual, textual
and motion features of video and corresponding prompt, and integrates key
features to enhance feature expression. We hope that our benchmark can promote
the development of quality evaluation metrics for AIGC videos. The LGVQ
dataset and the UGVQ metric will be publicly released.

###### Index Terms:

video generation, AIGC, video quality assessment, dataset, benchmark.

##  I Introduction

With the rapid development of generative models and digital media techniques,
artificial intelligence (AI)-generated content (AIGC) media, leveraging
generative models to automatically create media content, has gained
substantial attention in recent years. A typical form of AIGC media is text-
to-video (T2V) [1, 2, 3], also known as AIGC videos, where videos are
generated entirely based on textual descriptions. Because of the highly
simplistic generation process, AIGC videos can be widely used in industries
including film [4], gaming [5], advertising [6], and more. Although
significant advancements have been made in the field of AIGC videos, they
still face various quality issues. For example, AIGC videos may exhibit severe
spatial and temporal distortions, such as blurred objects and backgrounds,
poor action continuity, etc. Moreover, discrepancies between AIGC videos and
the original text may impede their effectiveness in real-world applications.
We show some typical distortions of AIGC videos in Fig. 1.

In the spatial domain, distortions include blurriness and irrational objects.
Blurriness results in poor sharpness, and irrational objects show forms that
defy physical logic. Temporal distortions involve motion blur, causing
significant distortion in rapidly moving parts, and frame jitter, leading to
incoherent frames and severe shaking. Alignment distortions encompass event
inconsistency, where the described action mismatches the video content, and
context inconsistency, where the main subject or object in the prompt is
inconsistent with the video.

Therefore, how to effectively evaluate the perceptual quality of AIGC videos
is crucial for measuring the progress of video generation techniques,
selecting the best AIGC videos from a set of candidates generated by T2V
models and optimizing the video generation techniques [7].

Figure 1: Typical distortion types of AIGC videos. Spatial distortions mainly
include (a) blurriness and (b) irrational objects, temporal distortions
include (c) motion blur and (d) frame jitter, and alignment distortions
include (e) event inconsistency and (f) context inconsistency.

Video quality assessment has been applied in various fields [8, 9, 10], most
methods focus on User-Generated Content (UGC). However, there is still no fair
metric for AIGC videos. In previous video generation studies [1, 2, 11, 3],
only a few metrics are utilized to evaluate the effectiveness of video
generation methods, such as IS [12], FID [13], FVD [14], KVD [15], CLIP [16],
CLIPScore [17], FCS [3]. However, FID, FVD, and KVD compare the distribution
of Inception [18] features of generated frames with that of a set of real
images/videos, thus failing to capture distortion-level and semantic-level
quality characteristics. IS [12] does not require reference videos for
comparison but relies on pre-trained classification models. Furthermore,
motion generation poses a great challenge for current video generation
techniques, yet FID [13] and FVD [14] are unable to quantify the impact of
temporal-level distortions on visual quality. CLIP-based methods such as
CLIPScore [17], and FCS [3] are frequently employed to assess the alignment
between the generated video and its prompt text.

Towards this goal, we establish a benchmark for AIGC-VQA to evaluate the
effectiveness of existing quality assessment metrics in assessing the
perceptual quality of AIGC videos. We first construct a Large-scale Generated
Video Quality assessment (LGVQ) dataset to subjectively evaluate the three
most critical quality dimensions (i.e., spatial quality, temporal quality, and
text-to-video alignment) of AIGC videos. To enhance the utility of the
benchmark, the generated video content should cover wide a range of real-world
scenes as possible. So, we structurally partition the text prompts into three
components: foreground, background, and motion. For each component, we include
typical elements that frequently occur in daily life. Specifically, the
foreground includes four categories: people, animals, plants, and man-made
objects, and the background is categorized into indoor scenes, outdoor natural
scenes, and outdoor man-made scenes. For motions, we focus on three types:
static, dynamic, and local movement (e.g., watching belongs to static, running
belongs to dynamic, and taking belongs to local movement). By combining
different words or phrases from the aforementioned types, we obtain $468$ text
prompts for video generation. Then, six mainstream T2V algorithms are selected
to generate $2,808$ AIGC videos. To obtain the perceptual quality labels of
these videos, we invite $54$ subjects to provide their ratings of the spatial
quality, temporal quality, and text-to-video alignment for each video.

Subsequently, we bench existing quality assessment metrics on the LGVQ dataset
to analyze their ability in assessing the quality of AIGC videos. Given that
LGVQ provides spatial quality, temporal quality, and text-to-alignment labels
for each AIGC video, we evaluate three categories of quality assessment
metrics for the AIGC-VQA benchmark: image quality assessment (IQA) methods for
spatial quality evaluation, video quality assessment (VQA) methods for
temporal quality evaluation, and CLIP-based methods for text-to-video
alignment evaluation. Specifically, we test $10$ IQA methods, $10$ VQA
methods, and $8$ alignment methods on LGVQ.

Therefore, we further propose a Unify Generated Video Quality assessment
(UGVQ) model to simultaneously evaluate the three aspects of the quality of
AIGC videos. UGVQ leverages the visual and textual features of CLIP [19] and
the motion features of SlowFast [20] as the quality-aware feature
representation. We use visual CLIP [19] features and the motion features of
SlowFast [20] as the frame-level spatial feature and the video-level temporal
feature, respectively. For the text-video alignment, we use textual CLIP [19]
features as the prompt semantics, and use the visual CLIP [19] features as the
semantics of the frames. Then, we propose a video-level spatial feature
extractor to extract spatial features from consecutive frames. To enhance the
quality-aware feature representation, we propose a feature fusion module to
fuse the spatial features, text features, and temporal features. Experiments
show that UGVQ exhibits the best performance in assessing all three quality
aspects of AIGC videos compared to existing quality metrics re-trained on the
LGVQ dataset, suggesting that UGVQ can be used for an effective and
comprehensive VQA metric for assessing the quality of AIGC videos.

##  II Related Works

###  II-A Video Generation Techniques

With the development of artificial intelligence, video generation, especially
text-to-video (T2V) generation, has made significant progress. T2V models can
be categorized into three types: GAN/VAE-based [21, 22, 23, 24],
Autoregressive-based [25, 26, 27, 28, 29, 30], and Diffusion-based [31, 32,
33, 34, 35, 36, 37, 38, 39, 40, 41].

Initially, researchers used Variational Autoencoders (VAE) or Generative
Adversarial Networks (GAN) for T2V generation. For instance, Li et al. [23]
trained a conditional video generative model combining VAE and GAN to extract
static and dynamic information from text. Deng et al. [24] proposed the
introspective recurrent convolutional GAN.

Autoregressive models (e.g., Transformer) have also been explored for T2V
generation. For instance, NÜWA [25] leverages a 3D transformer encoder-decoder
with a nearby attention mechanism for high-quality video synthesis. NÜWA-
Infinity [26] presents a “render-and-optimize” strategy for infinite visual
generation. CogVideo [27] utilizes pre-trained weights from the text-to-image
model and employs a multi-frame-rate hierarchical training strategy to enhance
text-video alignment. Phenaki [28] uses a variable-length video generation
method with a C-ViViT encoder-decoder structure to compress video into
discrete tokens.

Diffusion models [42] have significantly advanced text-to-image generation and
are now being explored for T2V generation [43, 44, 45]. The Video Diffusion
Model [31] applies the diffusion model to video generation using a 3D U-Net
architecture combined with temporal attention. LVDM [32] introduces a
hierarchical latent video diffusion model. Gen-1 [46] is a structure and
content-guided video diffusion model, training on monocular depth estimates
for control over structure and content. Tune-a-video [33] employs a
spatiotemporal attention mechanism to maintain frame consistency. Video
Crafter1 [37] uses a video VAE and a video latent diffusion process for lower-
dimensional latent representation and video generation. NÜWA-XL [41] uses two
diffusion models to generate keyframes and refine adjacent frames.

Despite these advancements, challenges like unrealistic motion, intermittent
object appearances, and unrealistic phenomena remain [47, 48]. Quality
assessment for T2V videos is crucial for measuring progress and further
promoting the development of T2V models.

###  II-B Quality Metrics for AIGC Videos

Spatial quality metrics aim to measure the frame-level visual quality of
videos. IS [12] and FID [13] are the most frequently used metrics to evaluate
spatial quality. However, many studies [49, 50] have indicated that IS and FID
exhibit poor correlation with human visual perception. On the other hand,
image quality assessment (IQA) is design to quantify the perceptual quality of
images. Many popular IQA methods, such as SSIM [51], UNIQUE [52], StairIQA
[53], MUSIQ [54], LIQE [55], etc., have demonstrated remarkable capability in
measuring the perceptual quality of natural images.

Temporal quality metrics are responsible for assessing the temporal coherence
of AIGC videos. Previous T2V studies utilize FVD [14] to gauge the disparity
between features extracted by pre-trained Inflated-3D Convnets (I3D) [56] from
generated videos and realistic videos. Similar to FID, FVD also demonstrates a
weak correlation with human visual perception. As related research, user-
generated content (UGC) VQA models [57], such as SimpleVQA [58], FastVQA [59],
DOVER [60], OV-PSNR [61], SSL [62], etc., have attempted to utilize action
recognition network (e.g. SlowFast [20], Video Swin Transformer [63]) to
represent the temporal quality feature. However, several studies [64, 65] have
shown that the current UGC VQA datasets pose little challenge to temporal
quality analyzers in UGC VQA models

Text-to-video alignment metrics evaluate the consistency between the generated
videos and textual descriptions. CLIP-based methods, such as CLIP [17], BLIP
[66], and viCLIP [67] are frequently used to evaluate the consistency between
the AIGC videos and their text prompts. While these methods are trained on
large-scale text-image datasets [68, 69] to maximize the similarity of
positive pairs, recent studies [70, 49] demonstrate that they have poor
consistency with human visual perception. Hence, some studies have constructed
human-rated text-to-image alignment datasets [71, 68, 69]. Based on these
datasets, they develop alignment assessment models, like ImageReward [71],
PickScore [68], HPSv1 [72], HPSv2 [69], etc., to evaluate the consistency
between the text prompts and AIGC images.

##  III Subjective Quality Assessment Study

Figure 2: Illustration of the MOSs probability density of the LGVQ dataset.

###  III-A LGVQ Dataset

We first construct LGVQ, a large-scale generated video dataset consisting of
diverse AIGC videos, to serve as the benchmark for testing the perceptual
quality of AIGC videos subjectively and objectively.

Prompts Selection. Existing benchmarks such as VBench [73] and FETV [74] offer
extensive but somewhat broad classifications. VBench, for example, categorizes
text prompts into eight classes: animal, architecture, food, human, lifestyle,
plant, scenery, and vehicles. Similarly, FETV identifies nine categories:
people, animals, vehicles, plants, artifacts, food, building, scenery, and
illustrations. These categories provide a solid foundation but often lack
specificity in terms of motion and dynamic events.

Generally, models perform significantly better on static prompts compared to
dynamic ones. Our method introduces a detailed division into static, dynamic,
and local movement. This refined classification allows for a more precise
depiction of the inherent motion in each scenario, facilitating a more
comprehensive and descriptive set of prompts.

TABLE I: Video formats generated by the six T2V models in the LGVQ dataset. Methods | Time Duration | FPS | Resolution
---|---|---|---
Gen-2 [75] | $96$ | $24$ | $1408\times 768$
Tune-a-video [3] | $24$ | $10$ | $512\times 512$
Video Crafter [1] | $16$ | $8$ | $256\times 256$
Text2Video-Zero [2] | $36$ | $12$ | $512\times 512$
HotShot [11] | $8$ | $4$ | $672\times 384$
Video Fusion [76] | $16$ | $8$ | $256\times 256$

Specifically, we decompose the prompts into three components: foreground,
background, and motion state. The foreground refers to the main subject of the
event, including four categories: people, animals, plants, and man-made
objects. The background refers to the environment and location where the event
occurs, divided into indoor scenes, outdoor natural scenes, and outdoor man-
made scenes. The motion state refers to the main motion pattern of the event,
mainly divided into static, dynamic, and local movement. Each prompt needs to
choose one or more elements from the components of foreground, background, and
motion types.

T2V Methods Selection. We select six SOTA text-to-video models, including
Gen-2 [75], Hotshot-XL [11], Video Fusion [76], Video Crafter [1], Text2Video-
Zero [2], and Tune-a-video [3], to generate the videos for each prompt. So,
there are a total of $2,808$ AIGC videos in LGVQ. The detailed information of
videos generated by six T2V methods is shown in Table.I.

###  III-B Subjective Quality Assessment Experiment

Figure 3: Comparison of MOSs of different generation elements. The quality
scores are adjusted to range from 20 to 70. Figure 4: Comparison of MOSs of
different generation methods. The quality scores are adjusted to range from 20
to 70.

We conducted a subjective quality assessment on the LGVQ dataset to derive
quality labels for each AIGC video. This experiment targeted the investigation
of main distortion types in AIGC videos on perceptual quality, as highlighted
in Fig.1, including three critical quality aspects: (1) Spatial quality
focuses on the visual appearance of individual frames. (2) Temporal quality
evaluates the coherence across video frames. (3) Text-to-video alignment
assesses the correspondence between the video content and the accompanying
text prompt.

Before the formal assessments, participants underwent a training session where
they reviewed sample videos that were not included in the formal experiment.
This session was intended to familiarize them with the evaluation criteria and
the rating interface. Example distortions and quality attributes were
discussed to calibrate participant expectations and rating standards.

We follow the recommended method in [77] to process the raw subjective
ratings. First, the rating of an image is considered as an outlier if it is
far from $2$ (if normal) or $\sqrt{20}$ (if nonnormal) standard deviations
about the mean rating of that image. A subject with more than $5$% outlier
ratings is also rejected. The MOSs distribution is illustrated in Fig. 2,
highlighting the variability and trends across the dataset.

###  III-C MOSs Analysis

The MOSs of Different Generation Elements. We calculated the average MOS
across six T2V generation models to evaluate the strengths and weaknesses of
current mainstream T2V models in generating various objects and actions. The
results are shown in Fig. 3. For the foreground elements, the spatial and
temporal quality of the human category is relatively low, which may be due to
the complexity of human actions, postures, and expressions. In contrast, the
spatial quality of animals, plants, and man-made objects is satisfactory. In
terms of motion state, whether it is spatial, temporal, or T2V alignment
quality, static objects clearly outperform locally moving objects, and fully
moving objects perform the worst. This indicates that when the subject needs
strong temporal changes, its temporal quality will decrease, and the spatial
quality of this moving subject will also be affected. In terms of alignment
quality, static and locally moving objects align better with the text prompts,
whereas fully moving objects present increased challenges in alignment,
leading to lower scores. For background elements, natural scenery performs
best in terms of spatial and temporal quality, followed by indoor scenes and
man-made outdoor scenes.

TABLE II: The benchmark of existing IQA, VQA, and text-to-image alignment
methods on the LGVQ dataset.

| Method | SRCC | KRCC | PLCC | RMSE
---|---|---|---|---|---
Spatial | UNIQUE (TIP, 2021) [52] | 0.130 | 0.089 | 0.231 | 13.471
MUSIQ (ICCV, 2021) [54] | 0.389 | 0.221 | 0.431 | 10.078
– | 0.327 | 0.211 | 0.404 | 10.504
StairIQA (JSTSP, 2023) [53] | 0.334 | 0.209 | 0.393 | 10.521
– | 0.320 | 0.209 | 0.393 | 10.521
– | 0.284 | 0.184 | 0.372 | 11.209
CLIP-IQA (AAAI, 2023) [78] | 0.320 | 0.213 | 0.346 | 11.302
CLIP-IQA+ (AAAI, 2023) [78] | 0.341 | 0.221 | 0.355 | 10.917
LIQE (CVPR, 2023) [55] | 0.174 | 0.116 | 0.209 | 13.665
NIQE (ISPL, 2012) [79] | 0.228 | 0.127 | 0.293 | 13.072
BRISQUE (TIP, 2012) [80] | 0.255 | 0.155 | 0.457 | 10.836
Temporal | TLVQM (TIP, 2019) [81] | 0.286 | 0.189 | 0.437 | 10.615
RAPIQUE (JSP, 2021) [82] | 0.313 | 0.212 | 0.451 | 10.208
– | 0.347 | 0.231 | 0.469 | 10.107
VSFA (ACMMM, 2019) [83] | 0.295 | 0.192 | 0.451 | 10.858
SimpleVQA (ACMMM, 2022) [58] | 0.271 | 0.182 | 0.419 | 10.945
– | 0.289 | 0.197 | 0.432 | 10.901
– | 0.304 | 0.205 | 0.448 | 10.889
FastVQA (ECCV, 2023) [59] | 0.374 | 0.255 | 0.473 | 10.130
– | 0.361 | 0.247 | 0.468 | 10.197
DOVER (ICCV, 2023) [60] | 0.254 | 0.164 | 0.514 | 11.001
– | 0.249 | 0.161 | 0.493 | 10.904
VIDEAL (TIP, 2021) [84] | 0.238 | 0.155 | 0.421 | 11.219
PatchVQ (CVPR, 2021) [85] | 0.275 | 0.181 | 0.439 | 10.910
Alignment | CLIP (ICML, 2021) [19] | 0.324 | 0.239 | 0.388 | 10.782
CLIPScore (EMNLP, 2021) [17] | 0.372 | 0.254 | 0.405 | 10.604
BLIP (ICML, 2022) [66] | 0.379 | 0.260 | 0.389 | 10.540
viCLIP (ArXiv, 2022) [67] | 0.397 | 0.280 | 0.421 | 9.599
ImageReward (NIPS, 2023) [71] | 0.369 | 0.255 | 0.371 | 10.546
PickScore (NIPS, 2023) [68] | 0.381 | 0.262 | 0.382 | 9.906
HPSv1 (ICCV, 2023) [72] | 0.248 | 0.171 | 0.339 | 11.283
HPSv2 (ArXiv, 2023) [69] | 0.325 | 0.223 | 0.395 | 10.434

The MOSs of Different Generation Methods. We also analyze the generation
effect of six T2V models from three evaluation dimensions: spatial, temporal,
and alignment. The results are shown in Fig. 4. For spatial quality, Gen-2
demonstrates outstanding performance across foreground, background, and motion
state categories, surpassing all other models significantly. Text2Video-Zero
follows closely, ranking second in all categories. Hotshot-XL and Video Fusion
are comparable in quality. Video Crafter and Tune-a-video exhibit the poorest
overall performance. For temporal quality, Gen-2 continues to excel overall.
Following closely are Hotshot-XL and Video Fusion, which exhibit similar
quality levels. Video Crafter performs moderately in terms of temporal
quality. In addition, we found that the performance of the static motion of
Hotshot-XL, Video Fusion, and Video Crafter is much higher than their
performance of the dynamic motion, indicating that they are difficult to
generate high-frequency movement. The Text2Video-Zero and Tune-a-video models
perform the worst in this aspect. For text-to-video alignment, the differences
between the 6 models are relatively small. Comparing all 10 aspects, the
overall quality ranking of the 6 models from high to low is Gen-2, Hotshot-XL,
Video Fusion, Text2Video-Zero, Video Crafter, and Tune-a-video.

##  IV AIGC VQA Benchmark

To provide a comprehensive and detailed evaluation for T2V models, we create
this benchmark by incorporating a wide range of quality metrics. Our benchmark
allows for an in-depth assessment of various aspects of AI-generated video,
including spatial and temporal quality, as well as text-to-video alignment.
This benchmark not only highlights the strengths and weaknesses of existing
models but also drives innovation by identifying areas that require
improvement.

###  IV-A Compared Quality Metrics.

For spatial quality, we test two handcrafted-based IQA models, i.e. NIQE [79],
and BRISQUE [80], and six deep learning-based IQA methods including UNIQUE
[52], MUSIQ [54], StairIQA [53], CLIP-IQA [78], LIQE [55], CLIP-IQA+ [78]. For
temporal quality, we test two handcrafted-based VQA models, i.e. TLVQA [81]
and RAPIQUE [82], and six deep learning-based VQA models, i.e. VSFA [83],
SimpleVQA [58], FastVQA [59], DOVER [60], VIDEAL [84] and PatchVQ [85]. For
text-to-video alignment, we evaluate four CLIP-based methods including CLIP
[19], CLIPScore [17], BLIP [66], and viCLIP [67], and four text-to-image
alignment assessment methods including ImageReward [71], PickScore [68], HPSv1
[72], and HPSv2 [69].

###  IV-B Results

We present the results in Table II, from which we observe that across all
three quality dimensions, the performance of all quality metrics is notably
poor on the LGVQ dataset. For spatial quality metrics, the method MUSIQ,
trained on the KonIQ-10K dataset, shows a notably higher SRCC value of 0.389,
indicating relatively better performance in capturing the image quality
perturbations specific to AI-generated content. Conversely, the method LIQE
exhibits the lowest SRCC score, possibly due to its reliance on less robust
feature representations for complex image distortions typical of AIGC videos.
For temporal quality metrics, the VSFA model trained on the KonViD-1k dataset,
leads with an SRCC of 0.295, suggesting mediocrity in dealing with temporal
quality variations. On the other hand, the DOVER model scores lower with an
SRCC of 0.254, which may stem from its approach that might be less adaptive to
the specific temporal dynamics and distortion patterns present in AIGC videos.
For text-to-video alignment, the method viCLIP, leveraging a large-scale
dataset InternVid-10M, achieves an SRCC of 0.280, showing poor performance in
aligning textual descriptions with video content. However, the method BLIP
shows a lesser SRCC value of 0.254, possibly indicating challenges in handling
the intricate semantic relationships required for effective text-to-video
mapping in AIGC scenarios.

Overall, the performance of all quality metrics is poor on the LGVQ dataset.
This suggests that existing methods might not yet be fully equipped to handle
the unique spatial and temporal distortion types associated with AIGC videos.
Additionally, the lower performance across the board might also indicate a
fundamental limitation in current methodologies to fully comprehend and
evaluate the diverse and complex action concepts depicted in AIGC videos.

Figure 5: The proposed UGVQ Framework. The spatial, temporal, and text feature
extractor is used to extract features from sampled frames, videos, and
prompts, respectively. The video feature extractor extracts video-level
spatial features from frame-level spatial features. The feature fusion module
fuses the video-level spatial feature, text feature, and temporal feature to
obtain the fusion feature. The quality regression module regresses all four
features to spatial, temporal, and alignment quality.

##  V Proposed Method

In this session, we introduce UGVQ (Text-to-Video Unify Generated Video
Quality), a structured framework designed to comprehensively assess video
quality. The UGVQ comprises three key modules: a feature extraction module, a
text-visual feature fusion module, and a quality regression module.

###  V-A Quality-aware Feature Extraction

To accurately evaluate the quality of AIGC video regarding spatial, temporal,
and alignment, we extract the features related to the quality of these three
aspects.

####  V-A1 Spatial-aware Feature

Given a video $v$ composed of $n$ frames, $v=\\{v_{1},v_{2},\dots,v_{n}\\}$.
First, we sample $k$ key frames from $n$ video frames. Then, we use the frame-
level spatial feature extractor to get the spatial feature of $k$ key frames.
The set of the frame-level spatial features is shown in Eq (1),

$\\{\mathrm{S_{f}}(v_{i}),i\in\\{1,2,\dots,k\\}\\},$ (1)

where $\mathrm{S_{f}}$ donate the frame-level spatial feature extractor.

A video is composed of multiple frames (images), however, the spatial feature
of a video is not a simple combination of the spatial feature of the frames.
Therefore, we train a video-level spatial feature extractor $\mathrm{S_{v}}$
to obtain the time series feature of a set of video frames as the spatial
features of a given video $v$ with $n$ frames. The structure diagram is shown
in Fig.6. We split the features of the frames according to different scales,
then use a feature extractor to obtain the video features of the two scales
respectively, and finally use MLP to fuse these two scales. The formula is
shown as Eq (2),

$s_{v}=\mathrm{S_{v}}(\\{\mathrm{S_{f}}(v_{i}),i\in\\{1,2,\dots,k\\}\\}),$ (2)

where $\mathrm{S_{v}}$ represents video-level spatial feature extractor, and
$s_{v}$ represents video-level spatial feature of video $v$.

####  V-A2 Temporal-aware Feature

We use the pre-trained action recognition model as a temporal-aware feature
extractor to obtain the motion features of each video. Action recognition
networks [20] are commonly used to obtain features related to motion detection
and motion classification, which can largely recognize the phenomenon of
motion and frame discontinuity in AIGC videos. Therefore, for video $v$ and
action recognition network, we can get temporal features $t_{v}$ by Eq (3).

$t_{v}=\text{T}(v),$ (3)

where T donate the action recognition network.

####  V-A3 Alignment-aware Feature

Figure 6: The proposed video-level spatial feature extractor.

For the AIGC video, the alignment of the prompt with the generated video
content is an important aspect of quality assessment. Obtaining semantic
features of the prompt and generated video is the key to judging the
alignment. In Session II, we mentioned that CLIP [19] is an effective method
to evaluate the similarity of text and image. In this work, we also use CLIP-
based methods [68] to obtain semantic features of prompt and video frame
respectively. For a given prompt $p$ and corresponding video $v_{p}$, we can
obtain the text feature through text semantics extractor
$\text{E}_{\text{p}}$, as shown in Eq (4),

$e_{p}=\text{E}_{\text{p}}(p),$ (4)

where $e_{p}$ represent the semantic feature of prompt $p$.

###  V-B Feature Fusion Module

Figure 7: The proposed fuse block. This module adjusts the two input features
to the same size by a 1x1 convolutional layer and then fuses the two input
features with the attention mechanism.

We propose the feature fusion module to fuse three kinds of features extracted
in Section V-A to enhance quality-aware feature representations. For a given
prompt $p$ and the corresponding generated video $v_{p}$, we can obtain the
spatial feature $s_{v}$ of video $v$ by Eq (1) and Eq (2), the temporal
feature $t_{v}$ by Eq (3), and the prompt semantic feature $e_{p}$ by Eq (4).
Then, we fuse $e_{v}$ and $e_{p}$ by Eq (5),

$j_{1}=\text{FFM}(s_{v},e_{p}),$ (5)

which can enhance the feature expression of the alignment-aware features. In
addition, we fuse temporal features of the video $t_{v}$ and prompt semantics
$e_{p}$ to improve the performance of alignment and temporal quality, as shown
in Eq (6),

$j_{2}=\text{FFM}(t_{v},e_{p}),$ (6)

where FFM denate the feature fusion module.

We finally fuse the above two fuse features to obtain the fusion feature by Eq
(7)

$j=\text{JFM}(j_{1},j_{2}),$ (7)

where $j$ donate the fused feature of $j_{1}$ and $j_{2}$.

The overall framework of the feature fusion module is shown in Fig(5), and the
fuse block is shown in Fig(7).

TABLE III: The performance of the proposed UGVQ method, and existing IQA, VQA, and text-to-image alignment methods on the LGVQ, FETV, and MQT datasets. Aspects | Methods | LGVQ | FETV | MQT
---|---|---|---|---
SRCC | KRCC | PLCC | RMSE | SRCC | KRCC | PLCC | RMSE | SRCC | KRCC | PLCC | RMSE
Spatial | UNIQUE [52] | 0.716 | 0.525 | 0.768 | 7.675 | 0.764 | 0.637 | 0.794 | 0.492 | 0.689 | 0.519 | 0.702 | 0.917
MUSIQ [54] | 0.669 | 0.491 | 0.682 | 8.397 | 0.722 | 0.613 | 0.758 | 0.607 | 0.678 | 0.498 | 0.689 | 0.938
StairIQA [53] | 0.701 | 0.521 | 0.737 | 7.984 | 0.806 | 0.643 | 0.812 | 0.476 | 0.695 | 0.535 | 0.710 | 0.904
CLIP-IQA [78] | 0.684 | 0.502 | 0.709 | 8.124 | 0.741 | 0.619 | 0.767 | 0.545 | 0.713 | 0.537 | 0.722 | 0.877
LIQE [55] | 0.721 | 0.538 | 0.752 | 7.597 | 0.765 | 0.635 | 0.799 | 0.507 | 0.722 | 0.546 | 0.738 | 0.851
Ours | 0.759 | 0.567 | 0.795 | 7.190 | 0.841 | 0.685 | 0.841 | 0.445 | - | - | - | -
Temporal | TLVQM [81] | 0.828 | 0.616 | 0.832 | 7.927 | 0.825 | 0.675 | 0.837 | 0.632 | 0.813 | 0.605 | 0.831 | 0.761
RAPIQUE [82] | 0.836 | 0.641 | 0.851 | 7.661 | 0.833 | 0.691 | 0.854 | 0.618 | 0.822 | 0.627 | 0.837 | 0.737
VSFA [83] | 0.841 | 0.643 | 0.857 | 6.942 | 0.839 | 0.705 | 0.859 | 0.515 | 0.834 | 0.630 | 0.851 | 0.712
SimpleVQA [58] | 0.857 | 0.659 | 0.867 | 6.747 | 0.852 | 0.726 | 0.862 | 0.469 | 0.848 | 0.644 | 0.856 | 0.684
FastVQA [59] | 0.849 | 0.647 | 0.843 | 7.548 | 0.842 | 0.714 | 0.847 | 0.562 | 0.842 | 0.638 | 0.849 | 0.689
DOVER [60] | 0.867 | 0.672 | 0.878 | 6.498 | 0.868 | 0.731 | 0.881 | 0.583 | 0.854 | 0.665 | 0.869 | 0.673
Ours | 0.893 | 0.703 | 0.907 | 5.695 | 0.897 | 0.753 | 0.907 | 0.412 | 0.898 | 0.733 | 0.909 | 0.642
Alignment | CLIPScore [17] | 0.446 | 0.301 | 0.453 | 11.212 | 0.607 | 0.498 | 0.633 | 0.768 | 0.772 | 0.611 | 0.783 | 0.928
BLIP [66] | 0.455 | 0.319 | 0.464 | 11.008 | 0.616 | 0.505 | 0.645 | 0.754 | 0.761 | 0.616 | 0.772 | 0.951
viCLIP [67] | 0.479 | 0.338 | 0.487 | 10.765 | 0.628 | 0.518 | 0.652 | 0.746 | 0.798 | 0.628 | 0.818 | 0.871
ImageReward [71] | 0.498 | 0.344 | 0.499 | 10.348 | 0.657 | 0.519 | 0.687 | 0.720 | 0.794 | 0.624 | 0.812 | 0.889
PickScore [68] | 0.501 | 0.353 | 0.515 | 9.995 | 0.669 | 0.533 | 0.708 | 0.697 | 0.823 | 0.649 | 0.831 | 0.821
HPSv1 [72] | 0.481 | 0.341 | 0.497 | 10.578 | 0.639 | 0.525 | 0.680 | 0.739 | 0.781 | 0.620 | 0.785 | 0.903
HPSv2 [69] | 0.504 | 0.357 | 0.511 | 10.104 | 0.686 | 0.540 | 0.703 | 0.705 | 0.819 | 0.643 | 0.821 | 0.852
Ours | 0.551 | 0.394 | 0.555 | 9.480 | 0.734 | 0.572 | 0.737 | 0.675 | 0.845 | 0.668 | 0.851 | 0.775

###  V-C Quality Regression Module

Through the above research, we obtained 3 quality-related features and one
fusion feature, namely the spatial-aware feature $s_{v}$, the temporal-aware
feature $t_{v}$, the alignment-aware feature $e_{p}$, and the fusion feature
$j$. To maximize the performance, we concat all four features and input them
into the quality regression model. The regression model was implemented by two
MLP layers. The quality regression model will ultimately output three scores
as spatial, temporal, and alignment quality of the prompt and the
corresponding video, as shown in Eq (8),

$\begin{split}s_{s},s_{t},s_{a}=\text{QRM}(\text{concat}(s_{v},t_{v},e_{p},j)),\end{split}$
(8)

where QRM donates the quality regression module, and $s_{s},s_{t},s_{a}$
represent the score of the spatial quality, temporal quality, and text-to-
video alignment.

##  VI Experiments

###  VI-A Experiment Settings

####  VI-A1 Test Dataset

We validate our proposed method on LGVQ, and the other two published datasets,
FETV [74] and MQT [86]. The FETV dataset contains 2,476 videos, which were
generated by 619 prompts and 4 text-to-video models with 4 quality dimensions,
static quality, temporal quality, overall alignment, and fine-grained
alignment. In this paper, we use static quality, temporal quality, and overall
alignment for our experiments. The MQT [86] dataset contains 1,005 videos,
which were generated by 201 prompts and 5 text-to-video models with 2 quality
dimensions, alignment, and perception. In our experiments, we change the
number of nodes in the last MLP in the quality regression module to predict
the alignment and perception quality in MQT [86].

TABLE IV: Ablation study of the proposed UGVQ methods. No.8 is the proposed
UGVQ model.

| Features | Spatial | Temporal | Alignment
---|---|---|---|---
No. | frame-level | text | motion | video-level | fusion | SRCC | KRCC | PLCC | RMSE | SRCC | KRCC | PLCC | RMSE | SRCC | KRCC | PLCC | RMSE
1 | ✔ |  |  |  |  | 0.706 | 0.563 | 0.738 | 7.476 | 0.814 | 0.625 | 0.828 | 8.024 | 0.483 | 0.358 | 0.526 | 9.858
2 |  | ✔ |  |  |  | 0.248 | 0.222 | 0.315 | 11.235 | 0.222 | 0.151 | 0.238 | 13.194 | 0.308 | 0.213 | 0.334 | 10.684
3 |  |  | ✔ |  |  | 0.581 | 0.433 | 0.628 | 9.181 | 0.876 | 0.683 | 0.892 | 6.241 | 0.471 | 0.351 | 0.496 | 9.864
4 | ✔ | ✔ |  |  |  | 0.739 | 0.579 | 0.795 | 7.107 | 0.837 | 0.649 | 0.843 | 7.277 | 0.511 | 0.369 | 0.531 | 9.834
5 | ✔ | ✔ | ✔ |  |  | 0.748 | 0.559 | 0.781 | 7.315 | 0.889 | 0.697 | 0.902 | 5.834 | 0.525 | 0.376 | 0.548 | 9.611
6 | ✔ |  | ✔ | ✔ |  | 0.752 | 0.561 | 0.787 | 7.210 | 0.889 | 0.698 | 0.901 | 5.838 | 0.511 | 0.368 | 0.533 | 9.742
7 | ✔ | ✔ | ✔ | ✔ |  | 0.753 | 0.563 | 0.790 | 7.208 | 0.891 | 0.699 | 0.903 | 5.781 | 0.543 | 0.386 | 0.551 | 9.499
8 | ✔ | ✔ | ✔ | ✔ | ✔ | 0.759 | 0.567 | 0.795 | 7.190 | 0.895 | 0.703 | 0.907 | 5.695 | 0.551 | 0.394 | 0.555 | 9.480

####  VI-A2 Training Protocols

To validate the performance of our proposed method comprehensively, all
experiments use a training/validation/testing mode. The model is selected
based on the best performance on the validation set, and the chosen model is
then evaluated on the test set. The average results from all trials represent
the final performance, reflecting the generalization ability of our method. In
our experiment, the ratio of the training, validation, and testing sets is
approximately 7:1:2. Additionally, for all three datasets, there were
duplicate prompts for video generation, meaning one prompt corresponded to
multiple videos. To validate the generalization of our method in prompt-
related tasks, all experiments use invisible prompt mode, which means that the
prompt in the test or validation set does not appear in the train set.

###  VI-B Performance Comparison

The performance of the IQA-based, VQA-based, and alignment-based models on
proposed LGVQ databases are listed in Table. III. The performance shows that
the proposed method has excellent performance on temporal quality, which
indicates that the UGVQ has a good effect on detecting the frame stabs and
motion incoherence in AIGC video, and satisfies human subjective perception.
For spatial quality, we do not directly use IQA-related pre-trained models to
generate spatial-aware features, but our method still performs well in terms
of spatial quality. The alignment quality is relatively poor among the three
aspects, which may be because it is inherently a cross-modal task, and
eliminating the spacing between modalities is more difficult than the other
two aspects. In addition, we directly used the pre-trained CLIP-based model to
extract semantic features of video frames and prompts without training,
resulting in a lack of discriminative features.

We conducted experiments on two other T2V datasets, MQT [86], and FETV [74],
respectively. We compare IQA-based, VQA-based, and alignment-based models on
FETV and MQT datasets, and the performance results are shown in Table.III.
Experimental results show that the UGVQ performs better on FETV and MQT
datasets.

###  VI-C Ablation Study

TABLE V: The performance of the proposed UGVQ method, and existing IQA, VQA,
and text-to-image alignment methods on cross-dataset evaluation.

Aspects | Methods | $LGVQ\rightarrow FETV$ | $FETV\rightarrow LGVQ$
---|---|---|---
SRCC | KRCC | PLCC | SRCC | KRCC | PLCC
Spatial | UNIQUE [52] | 0.389 | 0.272 | 0.383 | 0.360 | 0.252 | 0.353
MUSIQ [54] | 0.426 | 0.312 | 0.472 | 0.406 | 0.281 | 0.404
StairIQA [53] | 0.501 | 0.360 | 0.539 | 0.484 | 0.346 | 0.500
CLIP-IQA [78] | 0.503 | 0.361 | 0.543 | 0.493 | 0.355 | 0.501
LIQE [55] | 0.478 | 0.341 | 0.519 | 0.461 | 0.340 | 0.477
Ours | 0.553 | 0.406 | 0.555 | 0.521 | 0.359 | 0.524
Temporal | TLVQM [81] | 0.306 | 0.211 | 0.314 | 0.310 | 0.211 | 0.314
RAPIQUE [82] | 0.373 | 0.260 | 0.351 | 0.347 | 0.247 | 0.351
VSFA [83] | 0.396 | 0.272 | 0.364 | 0.388 | 0.274 | 0.398
SimpleVQA [58] | 0.501 | 0.366 | 0.511 | 0.419 | 0.279 | 0.407
FastVQA [59] | 0.482 | 0.324 | 0.494 | 0.397 | 0.272 | 0.364
DOVER [60] | 0.494 | 0.349 | 0.483 | 0.427 | 0.287 | 0.406
Ours | 0.512 | 0.368 | 0.535 | 0.442 | 0.296 | 0.432
Alignment | CLIPScore [17] | 0.234 | 0.167 | 0.252 | 0.168 | 0.112 | 0.205
BLIP [66] | 0.216 | 0.144 | 0.233 | 0.151 | 0.103 | 0.193
viCLIP [67] | 0.253 | 0.173 | 0.274 | 0.177 | 0.117 | 0.212
ImageReward [71] | 0.261 | 0.183 | 0.283 | 0.193 | 0.135 | 0.245
PickScore [68] | 0.259 | 0.178 | 0.284 | 0.181 | 0.124 | 0.229
HPSv1 [72] | 0.228 | 0.152 | 0.248 | 0.153 | 0.104 | 0.195
HPSv2 [69] | 0.263 | 0.189 | 0.285 | 0.201 | 0.140 | 0.243
Ours | 0.278 | 0.196 | 0.292 | 0.217 | 0.148 | 0.255

We conduct ablation experiments in this section to verify the effectiveness of
the 3 main components in our UGVQ method, i.e., the overall architecture of
UGVQ, the video-level spatial feature extraction, and the feature fusion
module. Moreover, we further verify the effect of each feature on the
performance of various qualities. The performance results are shown in
Table.IV.

Through No.1, No.2, and No.3 experiments in Table.IV, we find that frame
features, text features, and motion features all have a certain correlation
with the quality of spatial, temporal, and text-video alignment. Among them,
text features have a low correlation with the quality of all three aspects due
to the lack of corresponding video or frame features. Through No.4, we found
that the combination of text features and frame features significantly
improved the performance in text-video alignment. Through comparison with No.4
and No.5, it was found that the temporal performance in No.4 was still
adversely affected due to the lack of motion features. In No.5, motion
features significantly improved temporal performance, which indicates that the
motion features extracted by the action recognition model have a good effect
on detecting the frame stabs and motion incoherence in AIGC video. Through
No.5, No.7, and No.8, we found that our proposed video-level spatial feature
extractor and feature fusion module effectively improved the performance of
the three aspects of quality, and proved the effectiveness of the video
feature extractor and feature fusion module.

###  VI-D Cross-Dataset Evaluation

To verify the generalization performance of our proposed method, we conducted
cross-dataset experiments on other datasets. We set up two sets of
experiments. First, we train the UGVQ model on all the data in our proposed
LGVQ dataset and then test the performance on all the FETV data. Second, we
train the UGVQ model on the FETV dataset and then test it on LGVQ data. We run
the same experiments on IQA-based, VQA-based, and alignment-based methods. The
performance results are shown in Table.V. Compared with the IQA-based, VQA-
based, and alignment-based methods, we can find that our method has the best
spatial, temporal, and alignment performance. However, numerically, it still
needs to be improved.

##  VII Conclusion

In this paper, we establish a new quality assessment dataset (i.e. LGVQ) and
benchmark for AIGC videos. LGVQ consists of 2,808 AIGC videos generated by 6
T2V generation methods using 468 text prompts. We conduct a subjective quality
assessment experiment on LGVQ, evaluating the videos from three dimensions.
Then, we test IQA, VQA, and text-to-image alignment methods on the LGVQ
dataset, revealing their limitations in measuring the perceptual quality of
AIGC videos. To address this issue, we develop a unified generated video
quality assessment (i.e. UGVQ) model capable of simultaneously evaluating the
three quality dimensions of AIGC videos.

## References

  * [1] H. Chen, M. Xia, Y. He, Y. Zhang, X. Cun, S. Yang, J. Xing, Y. Liu, Q. Chen, X. Wang, C. Weng, and Y. Shan, “Videocrafter1: Open diffusion models for high-quality video generation,” 2023.
  * [2] L. Khachatryan, A. Movsisyan, V. Tadevosyan, R. Henschel, Z. Wang, S. Navasardyan, and H. Shi, “Text2video-zero: Text-to-image diffusion models are zero-shot video generators,” 2023.
  * [3] J. Z. Wu, Y. Ge, X. Wang, S. W. Lei, Y. Gu, Y. Shi, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou, “Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2023, pp. 7623–7633.
  * [4] R. Gu, H. Li, C. Su, and W. Wu, “Innovative digital storytelling with aigc: Exploration and discussion of recent advances,” _arXiv preprint arXiv:2309.14329_ , 2023.
  * [5] Y. Wang, Y. Pan, M. Yan, Z. Su, and T. H. Luan, “A survey on chatgpt: Ai-generated contents, challenges, and solutions,” _IEEE Open Journal of the Computer Society_ , 2023.
  * [6] D. Du, Y. Zhang, and J. Ge, “Effect of ai generated content advertising on consumer engagement,” in _International Conference on Human-Computer Interaction_. Springer, 2023, pp. 121–129.
  * [7] K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine, “Training diffusion models with reinforcement learning,” in _The Twelfth International Conference on Learning Representations_ , 2023.
  * [8] Y. Li, S. Meng, X. Zhang, M. Wang, S. Wang, Y. Wang, and S. Ma, “User-generated video quality assessment: A subjective and objective study,” _IEEE Transactions on Multimedia_ , vol. 25, pp. 154–166, 2023.
  * [9] X. Guan, F. Li, Y. Zhang, and P. C. Cosman, “End-to-end blind video quality assessment based on visual and memory attention modeling,” _IEEE Transactions on Multimedia_ , vol. 25, pp. 5206–5221, 2023.
  * [10] Y. Li, B. Chen, B. Chen, M. Wang, S. Wang, and W. Lin, “Perceptual quality assessment of face video compression: A benchmark and an effective method,” _IEEE Transactions on Multimedia_ , pp. 1–13, 2024.
  * [11] J. Mullan, D. Crawbuck, and A. Sastry, “Hotshot-xl,” 2023. [Online]. Available: https://github.com/hotshotco/hotshot-xl
  * [12] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [13] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017.
  * [14] T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly, “Towards accurate generative models of video: A new metric challenges,” 2019.
  * [15] M. Binkowski, D. J. Sutherland, M. Arbel, and A. Gretton, “Towards accurate generative models of video: A new metric & challenges,” _arXiv preprint arXiv:1812.01717_ , 2018.
  * [16] C. Wu, L. Huang, Q. Zhang, B. Li, L. Ji, F. Yang, G. Sapiro, and N. Duan, “Godiva: Generating open-domain videos from natural descriptions,” _arXiv preprint arXiv:2104.14806_ , 2021.
  * [17] J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y. Choi, “Clipscore: A reference-free evaluation metric for image captioning,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2021, pp. 7514–7528.
  * [18] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _Proceedings of the 32nd International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, F. Bach and D. Blei, Eds., vol. 37. Lille, France: PMLR, 07–09 Jul 2015, pp. 448–456.
  * [19] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” _Proceedings of the 38th International Conference on Machine Learning_ , 2021.
  * [20] C. Feichtenhofer, H. Fan, J. Malik, and K. He, “Slowfast networks for video recognition,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 6201–6210.
  * [21] G. Mittal, T. Marwah, and V. N. Balasubramanian, “Sync-draw: Automatic video generation using deep recurrent attentive architectures,” in _Proceedings of the 25th ACM international conference on Multimedia_ , 2017, pp. 1096–1104.
  * [22] Y. Pan, Z. Qiu, T. Yao, H. Li, and T. Mei, “To create what you tell: Generating videos from captions,” in _Proceedings of the 25th ACM international conference on Multimedia_ , 2017, pp. 1789–1798.
  * [23] Y. Li, M. Min, D. Shen, D. Carlson, and L. Carin, “Video generation from text,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 32, no. 1, 2018.
  * [24] K. Deng, T. Fei, X. Huang, and Y. Peng, “Irc-gan: Introspective recurrent convolutional gan for text-to-video generation.” in _IJCAI_ , 2019, pp. 2216–2222.
  * [25] C. Wu, J. Liang, L. Ji, F. Yang, Y. Fang, D. Jiang, and N. Duan, “Nüwa: Visual synthesis pre-training for neural visual world creation,” in _European conference on computer vision_. Springer, 2022, pp. 720–736.
  * [26] J. Liang, C. Wu, X. Hu, Z. Gan, J. Wang, L. Wang, Z. Liu, Y. Fang, and N. Duan, “Nuwa-infinity: Autoregressive over autoregressive generation for infinite visual synthesis,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 15 420–15 432, 2022.
  * [27] W. Hong, M. Ding, W. Zheng, X. Liu, and J. Tang, “Cogvideo: Large-scale pretraining for text-to-video generation via transformers,” in _The Eleventh International Conference on Learning Representations_ , 2022.
  * [28] R. Villegas, M. Babaeizadeh, P.-J. Kindermans, H. Moraldo, H. Zhang, M. T. Saffar, S. Castro, J. Kunze, and D. Erhan, “Phenaki: Variable length video generation from open domain textual descriptions,” in _International Conference on Learning Representations_ , 2022.
  * [29] J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, Y. Aytar, S. M. E. Bechtle, F. Behbahani, S. C. Chan, N. Heess, L. Gonzalez, S. Osindero, S. Ozair, S. Reed, J. Zhang, K. Zolna, J. Clune, N. de Freitas, S. Singh, and T. Rocktäschel, “Genie: Generative interactive environments,” in _Forty-first International Conference on Machine Learning_ , 2024.
  * [30] S. Ge, T. Hayes, H. Yang, X. Yin, G. Pang, D. Jacobs, J.-B. Huang, and D. Parikh, “Long video generation with time-agnostic vqgan and time-sensitive transformer,” in _European Conference on Computer Vision_. Springer, 2022, pp. 102–118.
  * [31] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 8633–8646, 2022.
  * [32] Y. He, T. Yang, Y. Zhang, Y. Shan, and Q. Chen, “Latent video diffusion models for high-fidelity long video generation,” _arXiv preprint arXiv:2211.13221_ , 2022.
  * [33] J. Z. Wu, Y. Ge, X. Wang, S. W. Lei, Y. Gu, Y. Shi, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou, “Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 7623–7633.
  * [34] D. Zhou, W. Wang, H. Yan, W. Lv, Y. Zhu, and J. Feng, “Magicvideo: Efficient video generation with latent diffusion models,” _arXiv preprint arXiv:2211.11018_ , 2022.
  * [35] Y. Zeng, G. Wei, J. Zheng, J. Zou, Y. Wei, Y. Zhang, and H. Li, “Make pixels dance: High-dynamic video generation,” _arXiv preprint arXiv:2311.10982_ , 2023.
  * [36] H. Chen, Y. Zhang, X. Cun, M. Xia, X. Wang, C. Weng, and Y. Shan, “Videocrafter2: Overcoming data limitations for high-quality video diffusion models,” _arXiv preprint arXiv:2401.09047_ , 2024.
  * [37] H. Chen, M. Xia, Y. He, Y. Zhang, X. Cun, S. Yang, J. Xing, Y. Liu, Q. Chen, X. Wang _et al._ , “Videocrafter1: Open diffusion models for high-quality video generation,” _arXiv preprint arXiv:2310.19512_ , 2023.
  * [38] U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni _et al._ , “Make-a-video: Text-to-video generation without text-video data,” _arXiv preprint arXiv:2209.14792_ , 2022.
  * [39] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans, “Imagen video: High definition video generation with diffusion models,” 2022.
  * [40] L. Khachatryan, A. Movsisyan, V. Tadevosyan, R. Henschel, Z. Wang, S. Navasardyan, and H. Shi, “Text2video-zero: Text-to-image diffusion models are zero-shot video generators,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 15 954–15 964.
  * [41] S. Yin, C. Wu, H. Yang, J. Wang, X. Wang, M. Ni, Z. Yang, L. Li, S. Liu, F. Yang _et al._ , “Nuwa-xl: Diffusion over diffusion for extremely long video generation,” _arXiv preprint arXiv:2303.12346_ , 2023.
  * [42] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in neural information processing systems_ , vol. 33, pp. 6840–6851, 2020.
  * [43] J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo _et al._ , “Improving image generation with better captions,” _Computer Science_ , vol. 2, no. 3, p. 8, 2023.
  * [44] L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 3836–3847.
  * [45] C. Zhang, C. Zhang, M. Zhang, and I. S. Kweon, “Text-to-image diffusion model in generative ai: A survey,” _arXiv preprint arXiv:2303.07909_ , 2023.
  * [46] P. Esser, J. Chiu, P. Atighehchian, J. Granskog, and A. Germanidis, “Structure and content-guided video synthesis with diffusion models,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 7346–7356.
  * [47] R. Sun, Y. Zhang, T. Shah, J. Sun, S. Zhang, W. Li, H. Duan, and B. Wei, “From sora what we can see: A survey of text-to-video generation.”
  * [48] J. Cho, F. D. Puspitasari, S. Zheng, J. Zheng, L.-H. Lee, T.-H. Kim, C. S. Hong, and C. Zhang, “Sora as an agi world model? a complete survey on text-to-video generation,” _arXiv preprint arXiv:2403.05131_ , 2024.
  * [49] M. Otani, R. Togashi, Y. Sawai, R. Ishigami, Y. Nakashima, E. Rahtu, J. Heikkilä, and S. Satoh, “Toward verifiable and reproducible human evaluation for text-to-image generation,” pp. 14 277–14 286, 2023.
  * [50] Z. Chen, W. Sun, Y. Tian, J. Jia, Z. Zhang, J. Wang, R. Huang, X. Min, G. Zhai, and W. Zhang, “Gaia: Rethinking action quality assessment for ai-generated videos,” _arXiv preprint arXiv:2406.06087_ , 2024.
  * [51] L. Zhang and H. Li, “Sr-sim: A fast and high performance iqa index based on spectral residual,” in _2012 19th IEEE international conference on image processing_. IEEE, 2012, pp. 1473–1476.
  * [52] W. Zhang, K. Ma, G. Zhai, and X. Yang, “Uncertainty-aware blind image quality assessment in the laboratory and wild,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 3474–3486, 2021.
  * [53] W. Sun, X. Min, D. Tu, S. Ma, and G. Zhai, “Blind quality assessment for in-the-wild images via hierarchical feature fusion and iterative mixed database training,” _IEEE Journal of Selected Topics in Signal Processing_ , 2023.
  * [54] J. Ke, Q. Wang, Y. Wang, P. Milanfar, and F. Yang, “Musiq: Multi-scale image quality transformer,” in _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 5128–5137.
  * [55] W. Zhang, G. Zhai, Y. Wei, X. Yang, and K. Ma, “Blind image quality assessment via vision-language correspondence: A multitask learning perspective,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2023, pp. 14 071–14 081.
  * [56] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6299–6308.
  * [57] X. Min, H. Duan, W. Sun, Y. Zhu, and G. Zhai, “Perceptual video quality assessment: A survey,” _arXiv preprint arXiv:2402.03413_ , 2024.
  * [58] W. Sun, X. Min, W. Lu, and G. Zhai, “A deep learning based no-reference quality assessment model for ugc videos,” in _Proceedings of the 30th ACM International Conference on Multimedia_ , 2022, p. 856–865.
  * [59] H. Wu, C. Chen, J. Hou, L. Liao, A. Wang, W. Sun, Q. Yan, and W. Lin, “Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling,” in _Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VI_. Berlin, Heidelberg: Springer-Verlag, 2022, p. 538–554.
  * [60] H. Wu, E. Zhang, L. Liao, C. Chen, J. H. Hou, A. Wang, W. S. Sun, Q. Yan, and W. Lin, “Exploring video quality assessment on user generated contents from aesthetic and technical perspectives,” in _International Conference on Computer Vision (ICCV)_ , 2023.
  * [61] P. Gao, P. Zhang, and A. Smolic, “Quality assessment for omnidirectional video: A spatio-temporal distortion modeling approach,” _IEEE Transactions on Multimedia_ , vol. 24, pp. 1–16, 2022.
  * [62] S. Mitra, S. Jogani, and R. Soundararajan, “Semi-supervised learning of perceptual video quality by generating consistent pairwise pseudo-ranks,” _IEEE Transactions on Multimedia_ , vol. 26, pp. 6215–6227, 2024.
  * [63] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * [64] Y. Fang, Z. Li, J. Yan, X. Sui, and H. Liu, “Study of spatio-temporal modeling in video quality assessment,” _IEEE Transactions on Image Processing_ , 2023.
  * [65] W. Sun, W. Wen, X. Min, L. Lan, G. Zhai, and K. Ma, “Analysis of video quality datasets via design of minimalistic video quality models,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2024.
  * [66] J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in _ICML_ , 2022.
  * [67] Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang, S. Xing, G. Chen, J. Pan, J. Yu, Y. Wang, L. Wang, and Y. Qiao, “Internvideo: General video foundation models via generative and discriminative learning,” _arXiv preprint arXiv:2212.03191_ , 2022.
  * [68] Y. Kirstain, A. Poliak, U. Singer, and O. Levy, “Pick-a-pic: An open dataset of user preferences for text-to-image generation,” in _Advances in Neural Information Processing Systems_ , vol. 36, 2023.
  * [69] X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li, “Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis,” 2023.
  * [70] M. Ding, W. Zheng, W. Hong, and J. Tang, “Cogview2: Faster and better text-to-image generation via hierarchical transformers,” _arXiv preprint arXiv:2204.14217_ , 2022.
  * [71] J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong, “Imagereward: Learning and evaluating human preferences for text-to-image generation,” in _Advances in Neural Information Processing Systems_ , 2023.
  * [72] X. Wu, K. Sun, F. Zhu, R. Zhao, and H. Li, “Human preference score: Better aligning text-to-image models with human preference,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2023, pp. 2096–2105.
  * [73] Z. Huang, Y. He, J. Yu, F. Zhang, C. Si, Y. Jiang, Y. Zhang, T. Wu, Q. Jin, N. Chanpaisit, Y. Wang, X. Chen, L. Wang, D. Lin, Y. Qiao, and Z. Liu, “Vbench: Comprehensive benchmark suite for video generative models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2024, pp. 21 807–21 818.
  * [74] Y. Liu, L. Li, S. Ren, R. Gao, S. Li, S. Chen, X. Sun, and L. Hou, “Fetv: A benchmark for fine-grained evaluation of open-domain text-to-video generation,” in _Advances in Neural Information Processing Systems_ , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 62 352–62 387.
  * [75] P. Esser, J. Chiu, P. Atighehchian, J. Granskog, and A. Germanidis, “Structure and content-guided video synthesis with diffusion models,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2023, pp. 7346–7356.
  * [76] Z. Luo, D. Chen, Y. Zhang, Y. Huang, L. Wang, Y. Shen, D. Zhao, J. Zhou, and T. Tan, “Notice of removal: Videofusion: Decomposed diffusion models for high-quality video generation,” in _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023, pp. 10 209–10 218.
  * [77] “Methodology for the subjective assessment of the quality of television pictures,” _International Telecommunication Union_ , 2002.
  * [78] J. Wang, K. C. Chan, and C. C. Loy, “Exploring clip for assessing the look and feel of images,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 2, pp. 2555–2563, Jun. 2023.
  * [79] A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “completely blind” image quality analyzer,” _IEEE Signal processing letters_ , vol. 20, no. 3, pp. 209–212, 2012.
  * [80] A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” _IEEE Transactions on image processing_ , vol. 21, no. 12, pp. 4695–4708, 2012.
  * [81] J. Korhonen, “Two-level approach for no-reference consumer video quality assessment,” _IEEE Transactions on Image Processing_ , vol. 28, no. 12, pp. 5923–5938, 2019.
  * [82] Z. Tu, X. Yu, Y. Wang, N. Birkbeck, B. Adsumilli, and A. C. Bovik, “Rapique: Rapid and accurate video quality prediction of user generated content,” _IEEE Open Journal of Signal Processing_ , vol. 2, pp. 425–440, 2021.
  * [83] D. Li, T. Jiang, and M. Jiang, “Quality assessment of in-the-wild videos,” in _Proceedings of the 27th ACM International Conference on Multimedia_ , ser. MM ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 2351–2359.
  * [84] Z. Tu, Y. Wang, N. Birkbeck, B. Adsumilli, and A. C. Bovik, “Ugc-vqa: Benchmarking blind video quality assessment for user generated content,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 4449–4464, 2021.
  * [85] Z. Ying, M. Mandal, D. Ghadiyaram, and A. Bovik, “Patch-vq: ’patching up’ the video quality problem,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021, pp. 14 019–14 029.
  * [86] I. Chivileva, P. Lynch, T. E. Ward, and A. F. Smeaton, “Measuring the quality of text-to-video model outputs: Metrics and dataset,” _arXiv preprint arXiv:2309.08009_ , 2023.