# Vulnerability Detection in Open Source Software: An Introduction

Stuart Millar, Rapid7 LLC<EMAIL_ADDRESS>S. Millar is with Rapid 7
LLC, Boston, MA, USA. This version dated March 26th 2017. For correspondence
please e-mail<EMAIL_ADDRESS>
###### Abstract

This paper is an introductory discussion on the cause of open source software
vulnerabilities, their importance in the cybersecurity ecosystem, and a
selection of detection methods. A recent application security report showed
44% of applications contain critical vulnerabilities in an open source
component, a concerning proportion. Most companies do not have a reliable way
of being directly and promptly notified when zero-day vulnerabilities are
found and then when patches are made available. This means attack vectors in
open source exist longer than necessary. Conventional approaches to
vulnerability detection are outlined alongside some newer research trends. A
conclusion is made that it may not be possible to entirely replace expert
human inspection of open source software, although it can be effectively
augmented with techniques such as machine learning, IDE plug-ins and
repository linking to make implementation and review less time intensive.
Underpinning any technological advances should be better knowledge at the
human level. Development teams need trained, coached and improved so they can
implement open source more securely, know what vulnerabilities to look for and
how to handle them. It is the use of this blended approach to detection which
is key.

###### Index Terms:

open source software, cyber security, vulnerability detection, static
analysis, dynamic analysis, software assurance, machine learning.

##  I Introduction

Open source software (OSS) is developed collaboratively in the public domain
with a licence granting rights to the user base that are usually reserved for
copyright holders. A well-known OSS licence is the GNU General Public Licence
that allows free distribution under the condition that further developments
are also free. In a globally connected software society, a sizeable amount of
development work is effectively crowd-sourced to an international community of
OSS developers, with little awareness of the potential security problems this
creates [1]. OSS libraries increase development speed but there is a tangible
increase in risk also, with the Heartbleed bug in OpenSSL being a prime
example. Research into vulnerability detection in OSS is crucial given its
prevalence with many companies using vulnerable OSS components and vulnerable
libraries being repackaged in software. This OSS uptake shows no sign of
reversing or slowing, with a recent survey [2] indicating that 43% of
respondents think OSS is superior to its commercial equivalent.

At the crux of OSS vulnerability is that today’s applications commonly use
thirty or more libraries which in turn can comprise up to 80% of the code in
any such application. These libraries have the same full privileges of the
application that use them, letting them access data, write to files or send
data to the internet. Anything the application can do, the library can do.
Some estimate that custom built Java applications contain 5-10 vulnerabilities
per 10,000 lines of code [3]. A library can have on average 10,000 to 200,000
lines of code, therefore the chances a library has never had a vulnerability
are very slim, with it perhaps being more likely in fact that it has not even
been examined for vulnerabilities. Hence libraries with no vulnerabilities
should not automatically be considered ‘safe’. Most vulnerabilities are
undiscovered and it might be said the only way to deal with the risk of
unknown vulnerabilities is to have someone who understands security manually
analyse the source code. Tool support provides hints but is not a replacement
for experts because the lack of context within libraries makes it very
difficult for current tools at time of writing to conclusively identify
vulnerabilities.

This paper contributes a short introductory discussion on vulnerability
detection in OSS, and is organised as follows: Section II is a background
overview and Section III contains information on conventional and emerging
detection methods. Conclusions are presented with ideas for future work in
Section IV.

##  II Background

A study carried out in 2012 [3] found that more than 50% of the Fortune Global
500 companies have downloaded vulnerable OSS components, security libraries
and web frameworks. This report analysed 113 million Java framework and
security library downloads by more than 60,000 commercial, government and non-
profit organisations from the Central Repository. Central is the software
industries most widely used repository of OSS with more than 300,000
libraries. It was found the vast majority of library flaws remain
undiscovered, the presence of a vulnerability (or an absence of one) is not a
security indicator, and that typical Java applications are likely to include
at least one vulnerable library. Furthermore, the same study showed most
organisations did not have a strong process in place for ensuring the
libraries they rely upon are up-to-date and free from vulnerabilities. The
authors of [3] stress there are no shortcuts and they go as far as saying the
only useful indicator of library security is a thorough review that finds
minimal vulnerabilities – in other words, software assurance, or the measure
of how safe the software is to use, needs to be generated internally. One
might say this is surprising, as in many other product or service industries,
this assurance – consider it some kind of warranty or seal of approval perhaps
– is offered up by the supplier without hesitation to help build trust and
sell to the customer. [4] adopt a similar stance, agreeing that recurring
vulnerabilities in software are due to reuse. This reuse includes the same
code base with an identical or very similar code structure, method calls and
variables. Interestingly these attributes form the basis of a proposed method
of detecting unreported vulnerabilities in one system by consulting knowledge
of reported vulnerabilities in other systems that reuse the same code.

Linus’ Law [5] is often quoted in relation to OSS, which is “given enough
eyeballs, all bugs are shallow”, meaning with a large enough number of
developers looking at code, errors can be found. However, this can questioned
from a scientific viewpoint, and an empirical study of Linus’ Law appeared to
show more collaboration meant more vulnerabilities, not less. [6] found that
files with changes from nine or more developers were sixteen times more likely
to have a vulnerability than files changed by fewer than nine developers. Thus
inherent collaborative nature of OSS creates potentially unavoidable
vulnerabilities that require addressing.

##  III OSS Vulnerability Detection

###  III-A Conventional Detection Methods

There are not currently an abundance of publications on vulnerability
detection in OSS. However, those that have been written thus far describe
three conventional methods – static analysis, dynamic analysis and code
reviews.

####  III-A1 Static Analysis

Many black-box static analysis techniques and tools scan source code and
detect vulnerabilities in software after it has been written, which encourages
late detection and produces a lot of false positives . [7] explicitly
referenced the cut and thrust of the software development process, saying that
external static tools for secure programming don’t fit into such a workflow,
since they don’t work with the IDE and are retrospective. [8] concur that
static analysis produces high levels of false positives, as do [9] and [10].
[11] point out it is hard to know which vulnerabilities a static analysis tool
deals with, and there are difficulties in obtaining and maintaining up-to-date
tooling.

[12] specifically wrote about the capability of static code analysis to detect
vulnerabilities, concluding that tools are not effective. They tested three
widely used commercial tools and found 27% of C/C++ vulnerabilities and 11% of
Java vulnerabilities in their dataset were missed by all three. In some cases,
they were comparable to or worse than random guessing. They too make the point
about tools being prone to false positives, and this consolidates the need to
find other methods of detection rather than rely solely on static analysis.
That is not to say static analysis is of little use, as some compliance
regulations require inventories of OSS components so that risks can be
addressed. Static tools can scan open source code and create an inventory, so
when a new vulnerability is disclosed, it is known which applications use the
vulnerable OSS. The OWASP Dependency-Check tool [13] analyses code and creates
reports on associated CVE entries.

####  III-A2 Dynamic Analysis

White-box dynamic analysis can also be called run-time analysis. Fuzzing is
often used here, where inputs are changed using random values to detect
unwanted behavior [11]. [14] researched the nuances of how vulnerabilities
were discovered by researchers, and how those same researchers shared their
findings with the OSS community. They found running a fuzzer and debugging was
the chosen method for developers exploring binary executables to find buffer
overflows. Vulnerability researchers tend to make their own fuzzing tools,
seeing it as part of the learning process and preferring this approach over
more systematic exploration methods. [9] notes the usefulness of fuzzing, and
that it needs only basic knowledge to undertake. However fuzzing does not
allow the control of program execution, large campaigns are needed for
results, and it is time consuming. [8] contends fuzzing doesn’t scale if
dynamic symbolic execution is used, as it explores code paths simultaneously
which could create large workloads. Symbolic execution uses symbolic values
for variables instead of concrete values to execute all paths in a program.

####  III-A3 Manual Code Reviews

These involve manual inspection of the source code in a white-box manner.
Consequently, this method requires a lot of human effort [10]. Working on
source code manually does however detect vulnerabilities [14], and recall that
[3] argued code reviews, conducted by someone with appropriate security
knowledge, may be the only way to properly deal with vulnerabilities.

Figure 1: Hub and spoke layout for distributed demand-driven security testing

###  III-B Emerging Detection Methods

The issues with the conventional methods in the main are that static analysis
produces too many false positives, dynamic analysis does not scale, and code
reviews are time consuming. Research into newer methods tries to address these
problems via some interesting and novel approaches.

####  III-B1 Distributed demand-driven security testing

Proposed by [8], this involves many clients using OSS and one main testing
server, in a hub and spoke style per Figure 1. When a new path in a program is
about to be exercised by user input, it is sent to the testing hub for
security testing. Symbolic execution is applied to the execution trace to
check potential vulnerabilities on this new path, and if one is detected then
a signature is generated and updated back to all the clients for protection.
If a path exercised by an input triggers any vulnerability that has already
been detected, the execution is terminated. This allows testing to focus on
paths being used and helps stop attackers exploiting unreported
vulnerabilities at a client site.

However, questions remain over how to handle large time and space overheads at
client sites, how sensitive data is transmitted and handled, and actual
implementation details are scarce. That said, the principle of increasing test
coverage of important paths as users exercise them is sound, and [8] offers an
interesting conclusion that machine learning can in future identify patterns
of bugs at the testing server and use them to predict problematic code.

####  III-B2 Use of Execution Complexity Metrics

[15] examined complexity metrics collected during code execution, considering
them potential indicators of vulnerable code locations. Table I describes
these metrics. They measure the frequency of function calls and duration of
execution functions using Callgrind, a Valgrind tool for profiling programs.
The collected data consists of the number of instructions executed on a run,
their relationship to source lines, and call relationship among functions
together with call counts. Firefox and Wireshark were analysed with Callgrind
to gather the metrics and results showed execution complexity metrics may be
better indicators of vulnerable code than the conventional static complexity
metric of lines of code, or LoC.

TABLE I: Execution complexity metrics defined in [15] Name |  Definition
---|---
NumCalls |  The number of calls to the functions defined in a file.
InclusiveExeTime |  Execution time for the set of functions, $S$, defined in a file including all the execution time spent by the functions called directly or indirectly by the functions in $S$.
ExclusiveExeTime |  Execution time for the set of functions, $S$, defined in a file excluding the execution time spent by the functions called by the functions in $S$.

Their initial results, shown in Table II, indicate the percentage of
vulnerable files in execution is higher than the percentage of vulnerable
files in total, and hence execution complexity metrics could be good
indicators of vulnerability. This can reduce the code inspection effort as
prioritisation can take place based on the metrics.

TABLE II: Execution statistics from [15] Program |  % of vulnerable files |  % of vulnerable files in executed files
---|---|---
Firefox |  3.8 |  11
Wireshark |  7.8 |  19

####  III-B3 IDE Plugins for Early Detection

[7] attempted to detect vulnerabilities earlier in the development process by
using an Eclipse Java plug-in, arguing developers should be aware of security
vulnerabilities as they are coding. To reduce false positives, they proposed
context-sensitive data flow analysis which uses a program’s context of
variables and methods when searching for vulnerabilities instead of pattern
matching. [16] presented interactive static analysis, also known as IDE static
analysis. They too developed an Eclipse Java plug-in for detecting code
patterns that gives a two-way interaction between the IDE and the developer.
According to [16], their tool detected multiple zero day vulnerabilities.
Figure 2 shows a screenshot where the developer is instructed to annotate
access control logic for a highlighted sensitive method call.

Figure 2: An IDE static analysis tool from [16]

####  III-B4 Machine Learning

Machine learning is a type of artificial intelligence where computers use
algorithms to learn iteratively, teaching themselves to recognise patterns.
Most OSS code is managed using version control systems like Git or CVS, with
vulnerable code inserted via commits from the developer to the main data
repository. But many tools can’t run on a small code snippet in an individual
commit, and checking the whole project is time consuming. [10] implemented a
type of machine learning algorithm called a Support Vector Machine (SVM) that
used metadata from commits made to OSS repositories. The SVM used features
from the metadata such as the number of added, deleted or modified functions
and how often a contributor had contributed to a given project before. Their
results showed that false positives were reduced by over 99% compared to those
generated by a static analysis tool - to be exact, their SVM driven tool
generated 36 false positives compared to 5,460 generated from the static
analysis tool. The goal of their work was to reduce the chance of
vulnerabilities getting from a vulnerable commit into the fully deployed
software. [9] also developed a machine learning tool to predict
vulnerabilities for large scale software like operating systems. They took the
popular Debian OS as an example, since it has 30,000 programs and 80,000 bug
reports. Clearly, code flaws can be hard to find manually in a code base of
that size, so the application of machine learning is of interest. Their
classification results were not conclusive but nevertheless, as an initial
study, they showed promise for large-scale vulnerability detection only using
binary executables, an approach which does not appear to have been attempted
elsewhere.

####  III-B5 Further Knowledge Formalisation and Linking Repositories

[17] discussed formalising knowledge representation to determine transitive
dependencies in software. The idea is the various vulnerability repositories
that exist online like the NIST National Vulnerability Database (NVD), or the
Common Weakness Enumeration (CWE) database can be linked and simultaneously
used to find out if a project is indirectly dependent on vulnerable
components.

##  IV Conclusions & Future Work

The global use of OSS presents such a huge number of attack vectors that
discovering novel techniques of vulnerability detection is an essential area
of research. Of the new methods mentioned in this paper, machine learning,
early detection IDE plug-ins and linking repositories show much promise for
future work. Machine learning lends itself well to feature-rich OSS which
speeds up classification of vulnerable code and reduces the time burden on
development teams. Early detection IDE plug-ins will help developers
implementing OSS to grow and consolidate their secure coding knowledge.
Linking repositories ensures better value from the separate, unconnected
datastores of vulnerabilities as they presently exist. It may also be possible
to use machine learning to levarage all these options in a modular system.
Improvements in OSS vulnerability detection might be quicker to realise than
one would think – [18] mention Pareto’s law, where 80% of effects can be
contributed to 20% of causes, and so identifying a small proportion of
problematic OSS code then focusing testing efforts using a selection of
detection methods could improve code quality and time-to-release, whilst
reducing development and maintenance costs. The exact mix of techniques might
vary from one OSS scenario to another but in the first instance a strategy
using a blend of methods that augment each other is likely to be considerably
more performant than one approach in isolation.

## References

  * [1] S. Koussa, “Techbeacon: 13 tools for checking the security risk of open-source dependencies,” https://techbeacon.com/13-tools-checking-security-risk-open-source-dependencies-0, [Online: last accessed March 2017].
  * [2] Black Duck Software, “2015 Future of Open Source Survey,” https://www.slideshare.net/blackducksoftware/2015-future-of-open-source-survey-results, [Online: last accessed March 2017].
  * [3] J. Williams, A. Dabirsiaghi, “The Unfortunate Reality of Insecure Libraries,” http://cdn2.hubspot.net/hub/315719/file-1988689661-pdf/download-files/The_Unfortunate_Reality_of_Insecure_Libraries.pdf?t=14901257
24196, [Online: last accessed March 2017].

  * [4] N. H. Pham, T. T. Nguyen, H. A. Nguyen, and T. N. Nguyen, “Detection of recurring software vulnerabilities,” in _Proceedings of the IEEE/ACM International Conference on Automated Software Engineering_ , ser. ASE ’10. New York, NY, USA: Association for Computing Machinery, 2010, p. 447–456. [Online]. Available: https://doi.org/10.1145/1858996.1859089
  * [5] Wikipedia, “Linus’ Law,” https://en.wikipedia.org/wiki/Linus%27s_Law, [Online: last accessed March 2017].
  * [6] A. Meneely and L. Williams, “Secure open source collaboration: An empirical study of linus’ law,” in _Proceedings of the 16th ACM Conference on Computer and Communications Security_ , ser. CCS ’09. New York, NY, USA: Association for Computing Machinery, 2009, p. 453–462. [Online]. Available: https://doi.org/10.1145/1653662.1653717
  * [7] L. Sampaio and A. Garcia, “Exploring context-sensitive data flow analysis for early vulnerability detection,” _Journal of Systems and Software_ , vol. 113, pp. 337–361, 2016. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0164121215002873
  * [8] D. Zhang, D. Liu, C. Csallner, D. Kung, and Y. Lei, “A distributed framework for demand-driven software vulnerability detection,” _Journal of Systems and Software_ , vol. 87, pp. 60–73, 2014. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0164121213002288
  * [9] G. Grieco, G. L. Grinblat, L. Uzal, S. Rawat, J. Feist, and L. Mounier, “Toward large-scale vulnerability discovery using machine learning,” in _Proceedings of the Sixth ACM Conference on Data and Application Security and Privacy_ , ser. CODASPY ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 85–96. [Online]. Available: https://doi.org/10.1145/2857705.2857720
  * [10] H. Perl, S. Dechand, M. Smith, D. Arp, F. Yamaguchi, K. Rieck, S. Fahl, and Y. Acar, “Vccfinder: Finding potential vulnerabilities in open-source projects to assist code audits,” in _Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security_ , ser. CCS ’15. New York, NY, USA: Association for Computing Machinery, 2015, p. 426–437. [Online]. Available: https://doi.org/10.1145/2810103.2813604
  * [11] N. Shahmehri, A. Mammar, E. Montes De Oca, D. Byers, A. Cavalli, S. Ardi, and W. Jimenez, “An advanced approach for modeling and detecting software vulnerabilities,” _Inf. Softw. Technol._ , vol. 54, no. 9, p. 997–1013, sep 2012. [Online]. Available: https://doi.org/10.1016/j.infsof.2012.03.004
  * [12] K. Goseva-Popstojanova and A. Perhinschi, “On the capability of static code analysis to detect security vulnerabilities,” _Information and Software Technology_ , vol. 68, pp. 18–33, 2015. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0950584915001366
  * [13] OWASP, “OWASP Dependency Check,” https://www.owasp.org/index.php/OWASP_Dependency_Check, [Online: last accessed March 2017].
  * [14] M. Hafiz and M. Fang, “Game of detections: How are security vulnerabilities discovered in the wild?” _Empirical Softw. Engg._ , vol. 21, no. 5, p. 1920–1959, oct 2016. [Online]. Available: https://doi.org/10.1007/s10664-015-9403-7
  * [15] Y. Shin and L. Williams, “An initial study on the use of execution complexity metrics as indicators of software vulnerabilities,” in _Proceedings of the 7th International Workshop on Software Engineering for Secure Systems_ , ser. SESS ’11. New York, NY, USA: Association for Computing Machinery, 2011, p. 1–7. [Online]. Available: https://doi.org/10.1145/1988630.1988632
  * [16] J. Zhu, J. Xie, H. R. Lipford, and B. Chu, “Supporting secure programming in web applications through interactive static analysis,” _Journal of Advanced Research_ , vol. 5, no. 4, pp. 449–462, 2014, cyber Security. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2090123213001422
  * [17] S. S. Alqahtani, E. E. Eghan, and J. Rilling, “Tracing known security vulnerabilities in software repositories – a semantic web enabled modeling approach,” _Science of Computer Programming_ , vol. 121, pp. 153–175, 2016, special Issue on Knowledge-based Software Engineering. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167642316000253
  * [18] M. English, C. Exton, I. Rigon, and B. Cleary, “Fault detection and prediction in an open-source software project,” in _Proceedings of the 5th International Conference on Predictor Models in Software Engineering_ , ser. PROMISE ’09. New York, NY, USA: Association for Computing Machinery, 2009. [Online]. Available: https://doi.org/10.1145/1540438.1540462


11institutetext: School of Computing, Dublin City University, Ireland
22institutetext: Insight Centre for Data Analytics, Dublin City University,
Ireland
22email<EMAIL_ADDRESS>
# Measuring the Quality of Text-to-Video Model Outputs: Metrics and Dataset

Iya Chivileva 11 Philip Lynch 11 Tomás E. Ward 22 0000-0002-6173-6607
Alan F. Smeaton 22 0000-0003-1028-8389

###### Abstract

Evaluating the quality of videos generated from text-to-video (T2V) models is
important if they are to produce plausible outputs that convince a viewer of
their authenticity. We examine some of the metrics used in this area and
highlight their limitations. The paper presents a dataset of more than 1,000
generated videos from 5 very recent T2V models on which some of those commonly
used quality metrics are applied. We also include extensive human quality
evaluations on those videos, allowing the relative strengths and weaknesses of
metrics, including human assessment, to be compared. The contribution is an
assessment of commonly used quality metrics, and a comparison of their
performances and the performance of human evaluations on an open dataset of
T2V videos. Our conclusion is that naturalness and semantic matching with the
text prompt used to generate the T2V output are important but there is no
single measure to capture these subtleties in assessing T2V model output.

###### Keywords:

Text-to-Video Models Video Synthesis Evaluation.

##  1 Introduction

Recent rapid advances in text-to-video (T2V) generation algorithms have
concentrated on improving the state of the art while producing a reliable
metric to measure the quality of generated videos is often an afterthought. It
is common for a model to be evaluated on 3 or 4 different metrics and in some
cases to also include human assessment. These evaluations show a large variety
in the quality of videos generated from T2V models. Considering this variance,
how to fairly evaluate the quality of generated videos is important but the
development of quality metrics has not received much attention.

In the field of AI-generated images, recent work by Li et al. [9] has compared
the quality of images generated from a collection of text-to-image (T2I)
models. This used automatically-computed quality metrics and using human
evaluations of perception and alignment of the image to the prompt used to
generate it. That work also included the release of a database of 2,982 T2I
images and the prompts and model parameters used to generate them as well as a
comparison between human evaluations and automatically created metrics. The
work in this paper follows a similar sequence to [9] except that we address
evaluating the quality of text-to-video instead of text-to-image models.

Our aim is assess how the number of metrics needed to produce a reliable
evaluation of the output from a T2V model could be reduced. This includes
comparing the outputs of automatic metrics with human evaluations. We begin
with an examination of open-source state-of-the-art T2V models and address the
limitations with current evaluation metrics including major pitfalls. We then
present the output of human assessment of the authenticity and realism of
videos which we refer to as video naturalness, plus the degree to which a
generated video aligns with the input prompt which we refer to as text
similarity. We also include the semantic matching between the original text
prompt and the content inside the generated video. In Section 4, we compare
the results of human evaluations with commonly used T2V metrics. Our findings
suggest that human evaluations mostly align with commonly used metrics, but
not always so.

##  2 Related Work

###  2.1 Text-to-Video Models

In 2022 the first open-source T2V model called Tune-a-Video was released by Wu
et al. [25] introducing a mechanism that uses the Stable Diffusion model [17]
for video generation. This model is built on state-of-the-art T2I diffusion
models and involves a tailored spatio-temporal attention mechanism and an
efficient one-shot tuning strategy. It served as an inspiration for the rapid
development of other open-source models including the following, which are
used in this paper.

  1. 1.

VideoFusion [12] in 2023 uses a decomposed diffusion process to resolve per-
frame noise as a base noise that is shared among all frames leading to
smoother video output.

  2. 2.

Text-to-Video Synthesis, also based on the work described in [12] in 2023 and
openly available is also a multi-stage text-to-video generation diffusion
model which consists of text feature extraction, a text feature-to-video
latent space diffusion model, and video latent space to video visual space.
This model adopts the Unet3D structure, and performs video generation through
an iterative denoising process from pure Gaussian noise video.

  3. 3.

Text2Video-Zero [8] in 2023 takes a low-cost zero-shot text-to-video
generation approach, leveraging the power of Stable Diffusion and tailoring it
for video. It achieves this by enriching the latent codes of frames with
motion dynamics and using a cross-frame attention of each frame on the first
frame to preserve the context, appearance, and identity of foreground objects.

  4. 4.

Aphantasia [4] also from 2023, is a collection of text-to-image tools, evolved
from the artwork of the same name. It is based on the CLIP model and Lucent
library, with FFT/DWT/RGB parameterizers and no-GAN generation.

###  2.2 Evaluation Metrics

The 3 most commonly used metrics for evaluating video quality are as follows.

1\. Inception Score (IS) [19] was developed as an alternative to human
evaluation and aims to measure both image quality and diversity. It relies on
the “inception network” [22] to generate a class probability distribution for
images. Higher-quality images should have a low entropy probability $P(y|x)$,
while diversity is measured with the marginal distribution of all images,
which should have high entropy.

2\. Fréchet Video Distance (FVD) [23] measures the distance between feature
activations of real and generated videos in the feature space of a pre-trained
video classifier, similar to the approach in the Fréchet Inception Distance
(FID) [6] which was developed for images. A lower FVD score indicates better
quality video generation indicating that both the reference and generated
videos have similar distributions for feature activations.

3\. CLIPSim [24] uses the CLIP [16] model to evaluate semantic matching
between an initial text prompt and a generated video. CLIP is a contrastive
learning-based model that creates a joint embedding space for images and text,
allowing the model to understand relationships between them. CLIPSim extends
CLIP to evaluate videos by finding the CLIP score of each frame in a video and
returning the average frame score.

Although the use of these metrics is common, there are concerns about their
use. IS has been criticised for its tendency to overfit on models trained
using ImageNet and its inability to distinguish between poor and high-quality
images [2] as demonstrated in 1(a) where IS assigned an almost perfect score
for the examples shown. FVD requires reference videos in order to generate a
score, which is challenging when comparing T2V models trained on different
datasets which hinders the broader applicability of this metric. The
underlying model used in CLIPSim, CLIP, has been criticised by the authors of
BLIP [10] for its reliance on noisy web image-text pairs arguing that a
smaller, filtered dataset should be used. Finally, the ability of Image-to-
Text models such as BLIP and CLIP to generate semantically similar captions
for images/frames that do not appear to match visually is another concern, as
illustrated in 1(b). Here the video was generated using the Aphantasia T2V
model [4] with the text prompt “A blue unicorn flying over a mystical land”
and by using CLIPSim we generated a similarity score of over 70%.

In summary, the metrics are used widely but have limitations which motivates
our interest in assessing them and how they compare to human assessments.

(a) Poor Quality Images from [2]

(b) CLIPSim Error

Figure 1: Example limitations of existing T2V quality metrics.

###  2.3 Image Naturalness

Image naturalness refers to how realistic and free of distortions or artefacts
an image appears. Naturalness is related to image quality, which encompasses
aspects such as sharpness, contrast, and colour accuracy, but image
naturalness specifically focuses on the realism of an image. Non-natural
images lack recognisable and interpretable real-world objects or scenes and
may include computer-generated graphics, abstract art, or heavily manipulated
photographs. They often serve artistic or functional purposes but do not
necessarily reflect the properties of natural images that are easily
interpreted by human vision.

The Naturalness Image Quality Evaluator (NIQE) is a no-reference image quality
assessment metric [13] based on the observation that natural images tend to
exhibit a unit-normal Gaussian characteristic in their luminance values. NIQE
uses a set of natural scene statistics (NSS) that captures the statistical
regularities present in natural scenes that are not present in unnatural or
distorted images. NIQE was trained on the LIVE image quality assessment
database [20], with 29 reference and 779 distorted images, each with 5
subjective quality scores.

BRISQUE (Blind/Referenceless Image Spatial QUality Evaluator) is an image
quality assessment measure [14] that also uses NSS to evaluate the quality of
a distorted image without requiring a reference image. BRISQUE extracts
36-dimensional feature vectors from 96 non-overlapping blocks of the distorted
image and maps these onto a reduced-dimensional space using principal
component analysis. The quality score is calculated using a support vector
regression (SVR) model trained on the LIVE IQA database [20].

The performance of NIQE and BRISQUE on real photo (a) and frames extracted
from various T2V models is shown in Figure 2 where the metrics evaluate images
on a scale of 0 to 100, higher scores indicate lower naturalness. We see
images (b) and (c) received the highest scores indicating poor naturalness and
non-natural images (d) and (e) received better scores than the low-quality
image of a dog in (b) and the image of an oil painting of a couple in (c),
which still represent recognisable objects. Although NIQE scores showed
slightly better results than BRISQUE, they were still unable to fully
differentiate between natural and non-natural images. Based on these
properties we opted to develop a new classifier to detect the naturalness of
an image, acknowledging that metrics like NIQE and BRISQUE are primarily
concerned with the visual quality of generated videos rather than their
naturalness.

(a) N:2.5/B:23.8

(b) N:7.7/B:54.4

(c) N:5.1/B:39.8

(d) N:3.5/B:15.7

(e) N:3.4/B:22.8

Figure 2: Image naturalness assessment with NIQE (N) and BRISQUE (B) scores.

##  3 An Ensemble Video Quality Metric

To address the limitations of image naturalness and modal biases in T2V bideo
quality metrics we propose an ensemble with the workflow shown in Figure 3.
The first of the two parts involves data generation, depicted in blue and
yellow boxes on the left side of the figure. Starting with an initial set of
text prompts, we generate a video for each using a T2V model under evaluation.
The videos are used to produce a set of captions for each using BLIP-2 [10].
The second part involves an ensemble of two metrics, the Text Similarity
Metric which calculates a similarity score between the original text prompt
and BLIP-generated captions. Next, we use the Naturalness Metric, a customised
XGBoost classifier that takes the generated video as input and outputs a
score. Both metric outputs are in the range $[0,1]$.

Figure 3: T2V-CL metric ensemble

A weighted average based on a linear regression model trained using manually
rated videos described later is used to combine the individual metrics.

###  3.1 Evaluating Image Naturalness

We developed a classifier for video naturalness for which we collected and
analysed several statistical measures from each video, including the
following:

  1. 1.

Texture score measures the degree of uniformity in an image’s texture since
natural images, such as landscapes or animal fur, tend to have more complex
textures than synthetic images. After converting to grayscale and applying a
Gaussian blur to reduce noise we apply Sobel edge detection in the x and y
directions and calculate the magnitude of the gradient. The variance of this
magnitude is the texture score.

  2. 2.

The sharpness score measures the amount of high-frequency content in an image,
indicative of the image’s level of detail. It is calculated by applying a
sharpening filter to the image and then taking the RMS difference between the
original and the filtered image.

  3. 3.

The colour distribution score is a measure of the uniformity of colour in an
image, exploiting the characteristic of a uniform or artificial colour
distribution in a non-natural image. It is calculated by applying K-means
clustering with K=2 to the A and B channels of the image’s LAB representation.
This score is the proportion of pixels in the cluster with the lowest A
channel value.

  4. 4.

The spectral score measures the extent to which an image differs from the
natural image statistics in the Fourier domain. The function calculates the
mean and standard deviation of each colour channel and computes the spectral
score as the sum of standard deviations divided by the sum of means.

  5. 5.

The entropy score uses the Shannon entropy formula [3], which measures the
level of randomness or disorder in pixel values. Natural images tend to have a
higher degree of order and lower entropy than non-natural ones.

  6. 6.

The contrast score measures differences between the lightest and darkest parts
of an image by dividing the standard deviation of pixel intensities by the
mean intensity.

  7. 7.

Oriented FAST and Rotated BRIEF (ORB) is a feature detection algorithm [18] to
compute statistics about the key points in an image including the mean and
standard deviations of the distances between key points and of the lengths of
the descriptors associated with those key points.

  8. 8.

The number and sizes of blobs is detected using the Laplacian of Gaussian
(LoG) method [11]. Blobs are regions in an image with a relatively uniform
intensity that stand out compared to the surrounding area.

Despite poor performance on some images, NIQE and BRISQUE scores are useful in
filtering very noisy and disordered images thus they are included in our
metric comparisons. To facilitate processing, a YUV444 video frame is reshaped
from planar to interleaved format, which represents colour information in
terms of brightness (Y) and colour (U and V), with 8 bits allocated to each
channel. NIQE scores are calculated for the grayscale frame and for the Y, U
and V channels in the YUV444 video frame separately as this provides a better
visual representation of the image [15] and a more accurate evaluation of
quality metrics like NIQE, which are more sensitive to variations in the
chrominance channels.

To train a classifier for image naturalness we also calculated a Modified
Inception Score (MIS) for each video which operates on a similar principle to
Inception Score outlined in Section 2.2 by calculating the mean probability
distributions of all frames in a generated video. We modified the IS metric to
return a larger value if the mean probability distribution in a video has low
entropy. Essentially, if the Inception model assigns a greater probability to
one particular class throughout the frames in a video, MIS will produce a
larger value. We achieved this by setting the marginal distribution to the
uniform distribution.

We collected all the video feature data described above from 187 videos
comprising 92 natural and 95 non-natural. We approached the naturalness
classifier task as a binary classification problem and manually assigned each
video a label indicating natural or not. We trained three classifiers,
AdaBoost, a Bagging classifier with a DecisionTree base and XGBoost. To
optimise the performance of each classifier, we employed GridSearch. We
evaluated the classifiers’ performance using F1 on training, validation, and
test sets. The XGBoost classifier performed the best on unseen data and was
used in the rest of the paper.

###  3.2 Evaluating Text Similarity

In the second part of our metric comparison we measure the semantic similarity
between the generated video captions and the original text prompt. The process
involves generating captions for each video frame using BLIP-2 and measuring
the similarity between each caption.

In our approach we combine BERT and Cosine similarity. Using the example
illustrated in Table 1 we see that BERT tends to over-perform as is designed
to capture more nuanced and complex semantic relationships between sentences
or captions, whereas Cosine only considers surface-level similarity based on
word overlap. By penalising the BERT similarity score with the Cosine
similarity score, we ensure that the combined similarity shown in Equation 1
reflects both surface-level and deeper semantic similarities between two
captions, thus providing a more accurate representation of their overall
similarity. After conducting an analysis and running multiple experiments, we
determined that the optimal ratio between BERT and Cosine similarities is 0.75
: 0.25.

|  Cosine | BERT | Combined
---|---|---|---
The sunrise was beautiful over the ocean |  0.00 | 0.45 | 0.22
The bulldozer was loud and destroyed the building
A hot air balloon in the sky |  0.28 | 0.76 | 0.65
Balloon full of water exploding in extreme slow motion
Table 1: Similarity scores for two pairs of sentences

$\text{Combined sim}=\begin{cases}0.25~{}(\text{Cos sim})+0.75~{}(\text{BERT
sim}),&\ if~{}\text{Cos sim}\neq 0\\\ 0.5~{}(\text{BERT sim})&\
otherwise\end{cases}$ (1)

Given that some frames in generated videos may exhibit significant distortions
or omissions or not contain recognisable objects such as in Figure 4 where two
frames do not include a dog, we calculate the weighted textual similarity for
a generated video of $n$ frames as
$\frac{1}{n}\sum_{i=1}^{n}w_{i}\cdot\text{sim}_{i}$. The weights are assigned
based on the frequency of each caption in the overall list of generated
captions.

Figure 4: Frames from a generated video with the prompt “A golden retriever
eating ice cream on a beautiful tropical beach at sunset”. Note that 2 of the
frames are missing the dog.

##  4 Evaluation

We now present the comparison of quality metrics for videos generated from T2V
models. We used 201 prompts and 5 T2V models outlined earlier in Section 2.1
to create 1,005 T2V model videos. We carefully selected the 201 prompts by
combining content generated by ChatGPT with manual curation. The compilation
covers a broad range of topics including influential figures, notable places,
and cultural events like Easter and the Brazilian Carnival. 87 of the prompts
are short (4 to 8 words), 43 are of average length (9 to 13 words) and 71 are
longer than 13 words. The collection of prompts offers a diverse range of
videos, spanning from practical scenarios to creative concepts. The videos
encompass a variety of actions, relationships, and visual styles. Example
frames from the collection of generated videos are shown in Figure 5. For each
of the 1,005 generated videos we computed two measures, naturalness and text
similarity score as described earlier.

Figure 5: Samples from our generated videos – rows show frames generated by
Text2Video-Zero, Text-to-Video Synthesis, Tune-a-Video, Aphantasia and Video
Fusion respectively while columns are frames from the same text prompts.

To obtain human quality scores for the generated videos we recruited
volunteers to rate videos remotely and in their own time with each person
given 10 days to complete the task across up to 10 sessions. The annotators
rated each video on a scale of 1 (low) to 10 (high) for two aspects, alignment
and perception. Alignment reflects the compatibility between the generated
video and the text of the original prompt which was provided, while perception
rates the overall perceptual quality of the video considering issues such as
clarity and sharpness, the presence of visual artefacts, blurriness, or
pixelation and whether colours represent real-world scenes and objects. 24 (16
male, 8 female) adult annotators, mostly graduate students, completed two
ratings of each video giving $1,005~{}videos\times 24~{}annotators\times
2~{}aspects$ = 48,240 quality ratings. Annotators were rewarded with a gift
token when they completed annotating the videos.

In assessing the quality of still images the “de facto” metric is mean opinion
score (MOS) [21] which is the mean of the opinions and ratings of human
evaluators gathered according to some numeric or qualitative scale. Despite
its popularity across various media including speech, audio, images, video,
multimedia, etc. it does have issues with its acceptability [21] because a
single number cannot capture a diversity of opinion scores. In [7] the authors
proposed that the standard deviation of opinion scores reflects the subjective
diversity while more recently [5] proposed that as well as the mean of the
opinion scores, researchers should assess quality in terms of the distribution
of opinion scores, not just the standard deviation.

Following the approach taken by Li et al. [9] where the authors developed a
quality metric for AI-generated text-to-image (T2I), we calculated a
“pid_delta” by computing the difference between the average rating (5) and the
mean of scores. We did this to see if the work in [9] for T2I images could be
applied to T2V videos and we refer to this as T2V-EC. Z-scores were computed
by subtracting the mean from each score and dividing by the standard deviation
and outliers were determined by comparing scores to a range defined by the
mean and standard deviation. Re-scaling then restored the original data range
by reversing the z-score transformation though negative values can result from
the initial shift towards 5 during normalisation, especially where original
scores were below 5. Figure 6 shows the distribution of adjusted MOS scores
for alignment and perception. An overall human evaluation score for each video
is the average adjusted MOS scores for alignment and perception. It is clear
that correlation between alignment and perception is not contingent upon the
rating values.

Figure 6: Distribution of adjusted MOS Scores.

Model | Alignment | Perception | Combined
---|---|---|---
Mean |  Std.Dev |  Mean |  Std.Dev | Human Score
Aphantasia | 4.016 | 0.841 | 3.221 | 0.692 | 0.362
Text2Video-Zero | 5.985 | 1.139 | 6.393 | 0.886 | 0.619
T2VSynthesis | 5.333 | 1.622 | 5.485 | 1.366 | 0.541
Tune-a-Video | 5.053 | 1.340 | 5.070 | 1.196 | 0.506
Video_Fusion | 4.995 | 1.686 | 5.139 | 1.507 | 0.507

Table 2: Human evaluation scores for five T2V models.

We now examine how human evaluation differentiated among the 5 models for
alignment, perception and for the combined human score (average), and these
are shown as means and standard deviations in Table 2 and as distributions in
Figure 7. What these show is that Aphantasia is the worst-performing across
perception and alignment, Text2Video-Zero is best and Tune-a-Video, Video
Fusion, and Text2Video Synthesis appear to be about the same

Figure 7: MOS score distributions across different models.

To support this interpretation we conducted a Tukey HSD (honestly significant
difference) test on the adjusted combined human evaluation scores to identify
differences among the 5 models. Tukey’s HSD [1] is a significance test to
simultaneously compare multiple means and distributions, MOS scores for 5 T2V
models in our case, in a single step and to find means whose differences are
greater than the expected standard error. The results of the test are shown in
Figure 8 and show that Aphantasia exhibits notably lower performance than the
others while Text2Video-Zero demonstrates significantly higher performance.
The other three models — Tune-a-Video, Video_Fusion, and Text2Video Synthesis
—– exhibit relatively similar levels of performance.

Figure 8: Tukey HSD of adjusted MOS Scores across different models.  Figure 9:
Comparison of model ranks by different metrics.

The combined human evaluation scores were compared against T2V-CL replicating
the work of Li et al. [9] for images, BLIPSim and Inception scores in Figure
9. The replicated T2V-CL metric shows a consistency in ranking with the other
metrics except for the Tune-a-Video model which has a higher rank by compared
to human evaluation. This discrepancy is because Tune-a-Video predominantly
produces cartoon-style videos, as shown in Figure 5 and our naturalness
classifier was trained on a range of example videos from a range of models
which did not include enough from the Tune-a-Video model.

In a final analysis we examined how prompt length influences model
performance. Figure 10 shows boxplots of adjusted MOS scores indicating the
ordering of model performance with Text2Video-Zero best and Aphantasia worst
in both boxplots and for all prompt lengths. They also show that almost always
the shorter the prompt length, the better the quality of the video. This is
explained by the fact that alignment to a longer prompt is more difficult for
a T2V model.

Figure 10: Boxplots of adjusted alignment (top) and perception (bottom) scores
across different prompt lengths

##  5 Conclusions

We investigated techniques for evaluating the quality of text-to-video (T2V)
model outputs including a critical analysis of commonly-employed metrics. We
examined the limitations of existing methods for evaluating video quality with
an emphasis on assessing the naturalness of T2V content as well as the degree
of semantic correspondence between videos and text prompts used to generate
them. We provide an open dataset of T2V videos from 5 models, with human
annotations of their quality111Code, prompts, examples and the 1,005 generated
videos from the 5 models used in this paper are available for public access at
https://tinyurl.com/4eufrek8. In summary we can say that there is some
consistency across the metrics and with human evaluations but not yet at the
level where we can eliminate the need for expensive and time-consuming human
assessment, though automatic metrics are a good proxy.

## References

  * [1] Abdi, H., Williams, L.J.: Tukey’s honestly significant difference (HSD) test. Encyclopedia of research design 3(1), 1–5 (2010)
  * [2] Barratt, S., Sharma, R.: A note on the inception score. arXiv preprint arXiv:1801.01973 (2018)
  * [3] Cover, T.M., Thomas, J.A.: Elements of information theory second edition solutions to problems. Internet Access pp. 19–20 (2006)
  * [4] Epstein, V.: Aphantasia text to video model. https://github.com/eps696/aphantasia, last Acccessed 29 August 2023
  * [5] Gao, Y., Min, X., Zhu, Y., Li, J., Zhang, X.P., Zhai, G.: Image quality assessment: From mean opinion score to opinion score distribution. In: Proceedings of the 30th ACM International Conference on Multimedia. p. 997–1005 (2022)
  * [6] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. AAdvances in Neural Information Processing Systems 30 (2017)
  * [7] Hoßfeld, T., Schatz, R., Egger, S.: SOS: The MOS is not enough! In: 2011 Third International Workshop on Quality of Multimedia Experience. pp. 131–136 (2011)
  * [8] Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R., Wang, Z., Navasardyan, S., Shi, H.: Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13439 (2023)
  * [9] Li, C., Zhang, Z., Wu, H., Sun, W., Min, X., Liu, X., Zhai, G., Lin, W.: AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment. arXiv preprint arXiv:2306.04717 (2023)
  * [10] Li, J., Li, D., Savarese, S., Hoi, S.: BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)
  * [11] Lindeberg, T.: Feature detection with automatic scale selection. International Journal of Computer Vision 30(2), 79–116 (1998)
  * [12] Luo, Z., Chen, D., Zhang, Y., Huang, Y., Wang, L., Shen, Y., Zhao, D., Zhou, J., Tan, T.: VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10209–10218 (2023)
  * [13] Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Transactions on image processing 21(12), 4695–4708 (2012)
  * [14] Mittal, A., Soundararajan, R., Bovik, A.C.: Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters 20(3), 209–212 (2012)
  * [15] Podpora, M., Korbas, G.P., Kawala-Janik, A.: Yuv vs rgb-choosing a color space for human-machine interaction. In: FedCSIS (Position Papers). pp. 29–34. Citeseer (2014)
  * [16] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 8748–8763. PMLR (2021)
  * [17] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10684–10695 (2022)
  * [18] Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: Orb: An efficient alternative to sift or surf. In: 2011 International conference on computer vision. pp. 2564–2571. Ieee (2011)
  * [19] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training gans. Advances in Neural Information Processing Systems 29 (2016)
  * [20] Sheikh, H.R., Bovik, A.C.: Image information and visual quality. IEEE Transactions on Image Processing 15(2), 430–444 (2006)
  * [21] Streijl, R.C., Winkler, S., Hands, D.S.: Mean opinion score (MOS) revisited: methods and applications, limitations and alternatives. Multimedia Systems 22(2), 213–227 (2016)
  * [22] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 2818–2826 (2016)
  * [23] Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., Gelly, S.: FVD: A new Metric for Video Generation (May 2019), iCLR Workshop on Deep Generative Models for Highly Structured Data
  * [24] Wu, C., Huang, L., Zhang, Q., Li, B., Ji, L., Yang, F., Sapiro, G., Duan, N.: Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806 (2021)
  * [25] Wu, J.Z., Ge, Y., Wang, X., Lei, W., Gu, Y., Hsu, W., Shan, Y., Qie, X., Shou, M.Z.: Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.11565 (2022)