# A crowdsourcing approach to video quality assessment

Babak Naderi, Ross Cutler

###### Abstract

The gold standard for measuring video quality is the subjective test, and the
most prevalent is the ITU-T Rec. P.910, a lab-based subjective standard in use
for the past two decades. However, in practice using ITU-T Rec. P.910 is slow,
expensive, and requires a lab, which all create barriers to usage. As a
result, most research papers that need to measure video quality don’t use
ITU-T Rec. P.910 but rather metrics that are not well correlated to subjective
opinion. We provide an open-source extension of ITU-T Rec. P.910 based on
crowdsourcing principles which address the speed, usage cost, and barrier to
usage issues. We implement Absolute Category Rating (ACR), ACR with hidden
reference (ACR-HR), Degradation Category Rating (DCR), and Comparison Category
Rating (CCR). It includes rater, environment, hardware, and network
qualifications, as well as gold and trapping questions to ensure quality. We
have validated that the implementation is both accurate and highly
reproducible compared to existing ITU-T Rec. P.910 lab studies.

###### Index Terms:

Subjective Quality, Video Quality, Crowdsourcing

##  I Introduction

Measuring the quality of video is an important task in many engineering areas,
such as video codec development, video enhancement, and video
telecommunication systems. As a result, there are many standards and metrics
for measuring video quality, though the gold standard is the subjective test
done in a controlled lab environment. The most prevalent of these standardized
subjective tests is the ITU-T Rec. P.910 [1]. However, using ITU-T Rec. P.910
in practice is slow due to the recruitment of test subjects and the limited
number of test subjects, and expensive due to paying qualified test subjects
and the cost of the test lab. The speed and cost result in the vast majority
of research papers not using ITU-T Rec. P.910 but rather objective functions
that are not well correlated to subjective opinion.

An alternative to lab-based subjective tests is to crowdsource the testing,
and there are several such systems that do this (see Table I). However, none
of these identified systems have the rater, environment conditions, and
hardware qualifications that ITU-T Rec. P.910 requires. In addition, these
existing systems have not been rigorously validated to show they are accurate
compared to a ITU-T Rec. P.910 lab-based study and give reproducible results.
We provide a crowdsourced implementation that includes rater, environment,
hardware, and network qualifications, as well as gold and trapping questions
to ensure quality. We include a validation study that shows it is both
accurate and highly reproducible compared to existing ITU-T Rec. P.910 lab
studies. The tool is open-sourced and can be used on the Amazon Mechanical
Turk platform for wide-scale usage. We recently used the tool in the CVPR 2022
CLIC challenge111http://compression.cc/ to provide the challenge metric for
machine learning-based video codecs.

A significant advantage of the crowdsourcing approach to labeling data is its
scalability due to low cost, availability of workers, and speed. Crowdsourcing
has been widely used to label very large image datasets like ImageNet [2], as
well as to label millions of audio clips for speech quality assessment for use
in academic challenges and speech enhancement development [3, 4, 5, 6, 7, 8,
9, 10, 11]. This crowdsourcing tool enables millions of video clips to be
accurately, inexpensively, and quickly labeled for video quality assessment,
which is necessary data to develop machine learning-based video codecs, video
quality enhancement, and to optimize video-based telecommunication systems.

The vast majority of published video compression research papers provide
results using the objective metrics Peak signal-to-noise ratio (PSNR) [12],
structural similarity index measure (SSIM) [13], multi-scale SSIM (MS-SSIM)
[14], and Video Multi-Method Assessment Fusion (VMAF) [15], e.g., [16, 17,
18]. However, it has been shown that PSNR, SSIM, and MS-SSIM are not well
correlated to subjective opinion [15, 19]. VMAF has been trained on
traditional DSP-based video codecs such as H.264/AVC [20], but has not been
shown to correlate with the newer deep learning-based codecs. Moreover, VMAF
hasn’t been trained for longer-term effects like recency, primacy, and
rebuffering [21]. Therefore for video codec research, there is no alternative
to subjective tests to evaluate and compare codecs at this time.

Section II provides an overview of related work, Section III describes our
tool’s implementation, Section IV gives the tool’s validation, and Section VI
gives conclusions.

##  II Related work

A recent review of subjective quality assessment tools is given in [22]. ITU-T
Rec. P.910 [1] provides a general subjective video quality assessment standard
for multimedia applications. ITU-T Rec. P.910 includes ACR, ACR-HR, DCR, and
paired comparison (PC) methods, as well as rater qualifications, environment
conditions, and video playback procedures. ITU-T Rec. P.911 is a counterpart
of ITU-T Rec. P.910 but for audiovisual signals. ITU-T Rec. P.912 [23]
provides a target-specific subjective video quality assessment standard, such
as for faces, license plates, etc. ITU-T Rec. P.913 [24] considers different
displays and testing environments and provides flexibility on the rating scale
and modality with mandatory reporting of test requirements [25]. ITU-T Rec.
P.918 [26] details subjective assessment methods for five perceptual video
quality dimensions, which can provide diagnostic information on the source of
observed degradation. Finally, ITU-R BT.500 [27] focuses on the video quality
of broadcast television signals in a highly controlled environment.

Tominaga et al. [28] conducted a comparison of eight different subjective
video quality assessment methods and found that ACR was the most suitable for
statistical reliability, assessment time, and ease of evaluation.

Keimel et al. [29] describes an open-source tool QualityCrowd that supports
ACR video quality assessment. QualityCrowd is extended by [30] to include a
Double Stimulus Continuous Quality Scale (DSCQS). Rainer et al. [31] describe
the tool WESP, an open-source tool that supports ACR, ACR-HR, DCR, and PC.

Jung et al. [32] provide a methodology to conduct remote subjective video
quality assessment studies in which the raters download videos and view them
manually on their devices. While the methodology suggests the rater checks
visual acuity and color blindness, it doesn’t provide any such tests. There
are also no tests for environmental conditions or hardware setup, which we
will show is essential.

TABLE I: Open-source crowdsourcing video quality assessment systems

Tool | Measures | Rater | Viewing | HW | Network | Accur. | Repro.
---|---|---|---|---|---|---|---
|  | qual. | cond. |  |  |  |
QualityCrowd | ACR, DSCQS | N | N | N | N | Y | N
[29, 30] |  |  |  |  |  |  |
WESP [31] | ACR, ACR-HR, | N | N | N | N | N | N
| DCR, PC |  |  |  |  |  |
avrateNG [33] | ACR | N | N | N | N | Y | N
Ours | ACR, ACR-HR, | Y | Y | Y | Y | Y | Y
| DCR, CCR |  |  |  |  |  |

##  III Implementation

Our implementation222https://github.com/microsoft/P.910 can be used either as
an integrated survey in Amazon Mechanical Turk (AMT) or as an external survey
deployed on a dedicated Web server333Currently the integrated surveys within
AMT are not able to playback videos in full-screen mode, so we strongly
recommend to deploy the test as an external survey.. Consequently, AMT or any
other crowdsourcing platform can be used for recruiting and paying test
participants. We also provide a lightweight container-based web application
that can serve the experiment. The web application can easily be deployed on
any Linux virtual machine. As a result, this implementation can be used for
both crowdsourcing tests or remote testing with a dedicated panel of
participants.

The open-source implementation includes the Absolute Category Rating (ACR),
ACR with hidden reference (ACR-HR), Degradation Category Rating (DCR), and
Comparison Category Rating (CCR). All methods can be used with either a five
or nine-point discrete Likert scale. We also followed and extend best
practices on video and speech quality assessment in crowdsourcing [34, 35] in
our implementation.

###  III-A Tools

We provide a set of program scripts to ease the interaction with the system
and avoid operation errors. The scripts are used to prepare the subjective
test (e.g., create customized trapping sequences), process the submitted
answers (i.e., data cleansing, aggregating ratings, and generating reports),
and interact with AMT (e.g., sending bonuses to workers).

In the first step (see Figure 1), trapping clips customized to the dataset
under the test should be created using the corresponding script. They will be
created by adding a text message, in the middle of selected existing video
sequences, asking participants to select a specific score when rating this
trapping clip. Next, a test configuration and URLs for test sequences (in the
case of DCR, CCR and ACR-HR also reference sequences), trapping clips, gold,
and training clips should be provided to the master script. It creates the
HTML template of the test, a list of variables (dynamic content), and a
configuration file for the result parser. Gold clips are video sequences whose
quality is known to the experimenter (either excellent or bad) [36]. A test
can be created in the HIT App Server by providing the HTML template and a list
of variables generated in the last step. Finally, a generic project
description and list of URLs can be downloaded and used to create a new test
in AMT (or any other crowdsourcing platform).

When the test is finished, the submitted answers to AMT and HIT App Server
should be provided to the result parser script in addition to the
configuration file which was created in step 2 (see Figure 1). It will perform
the data cleansing process, and aggregate the valid and reliable ratings over
the test sequence and over the test condition (i.e., Hypothetical Reference
Circuits444HRC is a treatment applied to source video to create the test
sequence. \- HRCs) if they exist. In addition, reports on the list of bonus
assignments, and lists of the submissions to be accepted/rejected or extended
will be generated.

Figure 1: Data Flow Diagram.

###  III-B Test components

The subjective test is organized in different sections from the participant’s
perspective (see Figure 2). Each section is designed to instruct the test
participant, qualify the participant, their environment, their setup, and
collect their votes. The instructions and ratings sections are included in all
tests, whereas the qualification, and calibration only need to be performed
once per test. The setup and training sections are shown (e.g., once per
hour). It is recommended to keep the crowdsourcing test session short (less
than 15 minutes), therefore we present a limited set of test sequences in the
rating section. We encourage participants to take part in more than one
session by providing monetary incentives and dynamic sections (i.e., the next
test session will be significantly shorter than the first one).

Figure 2: The crowdsourcing test from the participant’s perspective.

A set of automatic measurements are performed on the test’s loading time. The
experimenter can restrict test participants to specific viewing devices (i.e.,
mobile, PC, or both), minimum screen refresh rate, and minimum resolution.

####  III-B1 Video playback

We developed a video playback component based on HTML5. To avoid the
confounding effect of the network connection, it fully downloads all the video
materials on the browser’s local storage. The videos are also played in full-
screen mode, and the participants must watch the video until the end before
casting their votes. The player records the playback duration of videos which
will be used later in the data cleansing process. The experimenter can decide
whether the videos should be presented in their original size or scaled to
fill the participant’s viewing device. In the case of the DCR or CCR test, the
reference and processed video sequence will be shown in a sequence with a gray
screen in between for one second.

####  III-B2 Qualification

Within the qualification section, the eligibility of test participants is
evaluated. According to the ITU-T Rec. ITU-T Rec. P.910, participants should
be screened for normal color vision and normal or corrected-to-normal visual
acuity (i.e., no error on the 20/30 line of a standard eye chart) [37]. The
standard Ishihara test for color vision [38] includes 15 plates in the normal
and 6 plates in the short version, which are both too long for a crowdsourcing
test. In a prestudy, we invited 300 participants from AMT (34% male) and 191
participants (91% male) from two internet communities dealing with color
vision deficiencies. Both groups participated in the full Ishihara test in
which 6% of participants from AMT and 96% from the online forums were detected
as color blind. Applying a decision tree classifier with entropy as the
criterion revealed that only using Plates 3 and 4 reached 98% accuracy
(sensitivity 0.996, specificity 0.95). Consequently, we only use these two
plates in the qualification section.

For the visual acuity test, we use the Landolt ring optotypes as recommended
by ISO 8596:2017 [39]. In this test, participants are presented with broken
rings (like the letter C). The gap can be in 8 directions (every 45 degrees).
The diameter of the ring is 5 times the size of the gap. The visual acuity is
the inverse value of the gap size (in arc minutes) of the smallest identified
Landolt ring [39]. In each row (i.e., ring size) 5 samples should be presented
and the participant should answer 3 or more correctly to pass that size. Our
implementation of the visual acuity test consists of two steps: 1) setup, and
2) answer to up to 5 Landolt rings on the specific size. In the setup section,
the size of a pixel in the user’s screen is estimated by asking them to adapt
the size of a given picture (here a credit card) on their screen by clicking
on +/- buttons to reach the specified size. In addition, we asked them to sit
in the range of 50 to 75 cm from the screen during the test. Although the
maximum viewing distance of participants is limited to a distance in which
they can easily interact with their device, we cannot enforce a minimum
distance. Given the pixel size, the corresponding Landolt ring size is
calculated which correctly answering them in 50 cm distance shows a minimum of
20/30 visual acuity. The participant passes the visual acuity test if they
correctly select the direction of three out of the five Landolt rings.
Finally, participants can only continue to the next sections if they
successfully pass the qualification.

####  III-B3 Display calibration and instructions

Participants are asked to set the resolution of their device to default or
recommended value suggested by their operating system. We also ask them to
perform display color calibration using methods provided by their operating
system. This section provides information on how to perform these tasks for
Windows and Mac operating systems and will only be shown once during the test.

In the instruction section, a sample video for some of the perceptual quality
dimensions [40, 26] (i.e., fragmentation, discontinuity, and uncleanness) is
presented to the participants. They are also informed that impairments can
happen in a specific area or time within a video clip and the impairments are
not limited to the presented samples.

####  III-B4 Setup

In the setup section, the viewing conditions including the brightness of the
screen, room light, and viewing distance are evaluated. The ITU-T Rec. P.910
provides a detailed specification for viewing conditions that can only be
measured in a controlled laboratory environment using professional devices. We
created a brightness/light calibration task in which participants should count
the number of geometric shapes presented in a picture. They can insert their
answers and verify them. Given the correct answer, they can continue to the
next steps whereas on mismatch they are encouraged to modify the brightness of
their screen, lighting of the environment, or repeat the display calibration
task until they can insert the correct number for each picture. The presented
picture is a matrix of 4x4 squares. Each square has a different level of gray
background and may contain a triangle or circle in different sizes and
locations. The foreground color of the shapes is close to the background color
of the square. A pretest, with a limited group of experts, showed that the
calibration task performs best when the difference between foreground and
background colors is four integer points in RGB24 color space. Figure 3
illustrates an example matrix picture used in our tests. Within the setup
section, a second matrix is also presented for which no feedback is given. In
post-processing, submissions will be filtered based on their answers to the
second matrix question. It should be noted that to provide online filtering
and feedback, encrypted correct answers to some of the tests are included in
the HTML/JS code and a limited number of re-tries are provided in case of
failure. All the qualifications are also assessed separately in the post-
processing step.

Figure 3: Matrix of squares with 4 circles and 10 triangles used for
evaluating screen calibration.

Figure 4: Video playback control and rating scale implemented for ACR test.

The crowdsourcing test also includes a short paired-comparison task to
evaluate participants’ viewing distance. The task is inspired by [41] which is
the recommended method to evaluate crowd workers’ setup and environment in a
speech quality assessment test [36]. In this task, three pairs of images are
presented to the participants who are asked to sit at a distance of 1.5 times
their screen height. In each pair, one of the images is distorted with a blur
effect. Participants are asked to select the image of better quality. They can
choose between either of the images or select that both have the same quality.
In a separate experiment, we used blur effects with different radiuses from 1
pixel to 9. A small group of participants took part in six Just-Noticeable
Difference (JND) quality tests (adaptive staircase method 2 down–1 up) [42]
with two different screen sizes and three different viewing distances (too
close, expected, too far). Using the results, we selected three pairs for this
task: 1) the difference is mostly recognized if the participant is too close
to the screen, 2) the difference is correctly recognized if the participant is
seated at the expected distance, 3) the difference is still recognized even
they are too far from their screen. Given their response555For instance, if
one answers (2) wrongly but (3) correctly, it will be categorized as sitting
too far from their screen., there might be a feedback message asking them to
adjust their seating distance to 1.5 times their screen height. An example
from the paired-comparison questions is given in Figure 5.

Figure 5: One of the questions used in a paired-comparison test to check
viewing distance. Participants recognize the difference when they are too
close to their display.

####  III-B5 Training and rating

In the training section, participants should watch the videos from the
training set and rate their quality. The training set should represent the
dataset under the test and cover the entire range of the rating scale. We also
added a trapping clip to the training set, which instructs the participant to
make a specific selection with text in the video. In case participants provide
a wrong answer, they will be alerted and asked to watch the video again.
Finally, the training section is shown periodically (every 60 minutes) to keep
its anchoring effect following best practices in the speech quality domain
[3].

Within the rating section, participants rate the quality of ten video clips,
one trapping, and one gold clip. The trapping and gold clip will be inserted
automatically and will be used in the data cleansing step in post-processing.
The video playback controller and the rating scale are presented in Figure 4.

##  IV Validation

We used the VQEG HDTV datasets [43] to evaluate the validity and reliability
of our implementation. The datasets contain coding only and coding plus
transmission error impairments. They were created to validate objective video
quality models that predicted the quality of High Definition Television
(HDTV). The video materials and subjective data from experiments VQEGHD1-5 (in
a laboratory) are made publicly available in the Consumer Digital Video
Library666https://www.cdvl.org. Each experiment includes 168 video clips, with
24 shared between all experiments. Each experiment includes its specific
dataset where 21 HRCs were applied to 9 source videos (i.e., 144 processed
sequences). We used the datasets from experiments VQEGHD3 and VQEGHD5 for
which 40” (native resolution 1920x1080) and 24” (native resolution 1920x1200)
displays were used in the laboratory viewing sessions by 24 test participants,
respectively. The test sequences have various bit rates; 1-15 Mbps for VQEGHD3
and 2-16 Mbps for VQEGHD5. The test sequences created from two source clips in
the dataset VQEGHD5 were not included in the published package, leading to 136
sequences. Results of laboratory-based subjective tests were published in [43]
by VQEG.

To prepare the video clips for our crowdsourcing experiments, we re-encode the
video sequences using FFmpeg with the H.264/AVC codec keeping the original
frame rate, and resolution with a GOP of one second and Constant Rate Factor
of 17777 ffmpeg -i INPUT -y -preset veryslow -keyint_min 2 -g 24 -sc_threshold
0 -c:v libx264 -pix_fmt yuv420p -crf 17 OUT.mp4 .

We conducted six crowdsourcing studies with the two above-mentioned datasets.
In one of the tests, the VQEGHD5 dataset was used. In the rest of the five
tests, we used the VQEGHD3 dataset on five separate days, each with unique
raters, to evaluate the reproducibility of our implementation. In all tests,
participants with a minimum display resolution of 1280x720 and a refresh rate
of 30Hz were allowed.

###  IV-A Accuracy

In each experiment, 10 test sequences, one gold clip, and one trapping clip
were presented in one session and we aimed to collect 30 ratings per clip. In
the five experiments using the VQEGHD3 dataset an average of 71% submissions
passed the data cleansing step; the rest were not used due to the rater
providing a wrong answer to the gold question, longer playback duration,
failure on the second brightness check, low variance in ratings, or a wrong
verification code. On average we had 21 accepted votes per test sequence (with
a minimum of 15 ratings).

In the sixth experiment with the dataset VQEGHD5, 78.8% of submissions passed
the data cleansing step; the rest were not used due to similar issues. On
average we had 25 accepted votes per test sequence (with a minimum of 20
ratings).

The results are presented in Table II per test sequences and in Table III per
HRCs. We observed a strong correlation between laboratory and crowdsourcing
subjective tests ($\overline{r}=0.952$ per test sequence and $0.964$ per HRC).
The distribution of MOS values from crowdsourcing and laboratory tests, before
a First-Order Mapping (FOM)888FOM is typically used to removed bias and
gradient between scores from two subjective tests without changing the rank-
order[44]. Here, to visualize the effect, the scatter plots are illustrated
before without FOM however RMSEs are calculated after mapping., which is
illustrated in Figure 6.

TABLE II: Comparison between laboratory and crowdsourcing experiments
(sequence level).

Dataset | MOS | DMOS
---|---|---
| PCC | SPCC | RMSE FOM | PCC | SPCC | RMSE FOM
VQEG HDTV3 -run1 | 0.956 | 0.949 | 0.333 | 0.948 | 0.949 | 0.362
VQEG HDTV3 -run2 | 0.964 | 0.951 | 0.302 | 0.946 | 0.939 | 0.370
VQEG HDTV3 -run3 | 0.959 | 0.949 | 0.323 | 0.940 | 0.942 | 0.389
VQEG HDTV3 -run4 | 0.917 | 0.913 | 0.455 | 0.904 | 0.922 | 0.489
VQEG HDTV3 -run5 | 0.947 | 0.923 | 0.367 | 0.932 | 0.909 | 0.415
VQEG HDTV5 | 0.970 | 0.957 | 0.278 | 0.965 | 0.958 | 0.299

TABLE III: Comparison between laboratory and crowdsourcing tests in HRC level

Dataset | MOS |
---|---|---
| PCC | SPCC | RMSE | RMSE FOM
VQEG HDTV3 -run1 | 0.967 | 0.980 | 0.655 | 0.248
VQEG HDTV3 -run2 | 0.977 | 0.982 | 0.618 | 0.211
VQEG HDTV3 -run3 | 0.968 | 0.981 | 0.577 | 0.245
VQEG HDTV3 -run4 | 0.940 | 0.975 | 0.706 | 0.333
VQEG HDTV3 -run5 | 0.965 | 0.972 | 0.671 | 0.257

(a)

(b)

(c)

Figure 6: Distribution of MOS values between laboratory-based and
crowdsourcing-based experiments (a-b) and two crowdsourcing runs (c). (a)
Dataset VQEG HD3-run1 and (b) Dataset VQEG HD5.

###  IV-B Reproducibility

On average 63 unique workers participated in each run. We observed a
$PCC=0.971$ and $SRCC=0.95$ on average between the MOS values of sequences in
the five different runs. The correlation coefficients between the runs are
reported in Table IV. We also fitted a linear mixed-effects model (LMEMs) with
random intercept in which test sequences and runs are considered as fixed
factors and participants as a random factor. The result shows there was no
significant effect of runs on the subjective ratings
($\chi^{2}(4)=2.691,p=0.611$).

TABLE IV: Correlation coefficients between five runs of the VQEGHD3 dataset.
Pearson correlation coefficients are on the upper triangle and Spearman’s rank
correlation coefficients are on the lower triangle.

| Run 1 | Run 2 | Run 3 | Run 4 | Run 5
---|---|---|---|---|---
Run 1 |  | 0.984 | 0.987 | 0.957 | 0.977
Run 2 | 0.959 |  | 0.985 | 0.957 | 0.977
Run 3 | 0.974 | 0.969 |  | 0.952 | 0.972
Run 4 | 0.943 | 0.941 | 0.942 |  | 0.956
Run 5 | 0.954 | 0.947 | 0.942 | 0.933 |

###  IV-C Ablation study

We evaluated the effect of different components on the performance of the
crowdsourcing test. In pretests, we observed that the percentage of correct
answers to the brightness check increases by 19% when participants are asked
to perform screen calibration. Furthermore, in preliminary studies, we
observed that in 24.8% of submissions, the playback duration was longer than
1.15 times the original duration of the videos. After a software update, in
which videos are fully downloaded in advance, only 5.06% of cases showed
longer playback duration than expected. That change led to a 0.04% increase in
correlation coefficients to lab accuracy and a 0.1 reduction of RMSE.

We evaluated the performance of the gold clips, playback duration, brightness
check, and straightliner999Straitliners select the same rating for many
consecutive clips. detection by comparing the MOS values from the laboratory
test (as the ground truth) with the MOS values calculated from submissions
that passed or failed in each criterion. We combined submissions from all five
runs in the reproducibility study. The results are presented in Table V. The
correlation coefficients between the laboratory scores and the aggregated
scores that passed all the criteria are significantly higher than the other
groups. Similarly, the RMSEs were significantly lower.

TABLE V: Effect of integrated filtering criteria on the validity of the
crowdsourcing test.

Case | Avg.# Ratings | PCC | SRCC | RMSE | RMSE FOM
---|---|---|---|---|---
All passed | 104 | 0.96 | 0.96 | 0.62 | 0.31
Gold clips failed | 17 | 0.57 | 0.53 | 1.02 | 0.93
Playback duration failed | 8 | 0.62 | 0.57 | 1.12 | 0.89
Brightness check failed | 17 | 0.89 | 0.88 | 0.84 | 0.52
Straightliners | 6 | 0.29 | 0.30 | 1.53 | 1.09
All statistics are significantly different from their counterpart in All
passed at $\alpha=.05$

The rest of the components are evaluated in separate studies which are
detailed in the following sections.

####  IV-C1 Viewing distance

We conducted a crowdsourcing experiment with two settings in which
participants were asked 1) to sit at 1.5 times the height of their screen and
2) to sit as far as they could but still be able to interact with their device
and not be closer than three times of the height of their screen. The setup
section (including the viewing distance check) was forced to be shown in all
instances for both settings. Despite the original viewing distance test, no
feedback to participants was provided upon their answers. Submissions that
passed all the data cleansing criteria are aggregated based on their
performance in the viewing distance test. The results are presented in Table
VI. The correlation coefficient between the ratings from the submissions that
passed the viewing distance and the laboratory scores is significantly higher
than those that failed the test ($z=4.435$, $p<.001$).

TABLE VI: Effect of the viewing distance (VD) component on the validity of the
crowdsourcing test.

Case | PCC | SRCC | RMSE | RMSE FOM
---|---|---|---|---
VD - Passed | 0.93 | 0.92 | 0.76 | 0.41
VD - Failed | 0.83 | 0.78 | 0.95 | 0.62
VD - Failed (too far) | 0.87 | 0.84 | 0.81* | 0.58
VD - Failed (unknown category) | 0.79 | 0.72 | 1.07 | 0.70
* Beside this value, all statistics are significantly different from their counterpart in
VD- passed at $\alpha=.05$

####  IV-C2 Visual acuity test

In a new crowdsourcing test, the task was modified so that the Visual Acuity
Test (VAT) did not give any feedback on its finish, and despite its results,
participants could continue to the rating section. Meanwhile, the
qualification section (which contains the VAT) was included in all tasks. To
increase the diversity of participants, up to five tasks per worker were
allowed. We used the VQEG HD3 dataset for this test as well. In post-
processing, we grouped submissions based on their performance in the VAT
(passed or failed) and the rest of the quality control criteria. 13.3% of
submissions that passed all the other quality control criteria failed in VAT.
For each group, we aggregated the ratings per test sequence and compared the
resulting MOS values with scores from the laboratory tests. The results are
presented in Table VII. The correlation of laboratory scores to MOS values
from the group that passed the VAT and all the other criteria is higher than
the group that failed in VAT but passed the other criteria ($z=1.76$,
$p=0.039$).

TABLE VII: Effect of the Visual acuity test (VAT) component on the validity of
the crowdsourcing test.

Case | PCC | SRCC | RMSE | RMSE FOM
---|---|---|---|---
VAT Passed & All criteria passed | 0.91 | 0.90 | 0.88 | 0.47
VAT Failed & All criteria passed | 0.87 | 0.89 | 0.96 | 0.59
VAT Passed * | 0.91 | 0.90 | 0.97 | 0.48
VAT Failed * | 0.63 | 0.59 | 1.41 | 0.89
*No other criterion was considered.

(a)

(b)

(c)

Figure 7: Properties of crowd workers’ display. (a) Resolution, (b) Estimated
refresh rate (Hz), (c) Estimated display size (inch).

####  IV-C3 Display calibration and trapping clips

To validate the display calibration and trapping question, we conducted three
new crowdsourcing experiments. In the first experiment, we modified the test
and left out the display calibration task (and the brightness/light check).
The trapping clips from the training and rating sections were left out in the
second experiment. In the last one, the complete test was used. In all
experiments, the participants rated the VQEG HD3 dataset, and we aimed to
collect 30 votes per clip. The results are presented in Table VIII. The
correlation coefficients between laboratory and crowdsourcing scores were
significantly higher when the complete test was used. In the same case, the
RMSE after a first-order mapping was significantly lower. We also conducted
multiple Welch’s t-tests (one per test sequence) with Bonferroni correction to
determine which sequences are rated significantly different in the
crowdsourcing tests compared to the laboratory. In the case of complete test,
only 16% of the sequences were rated significantly different than the
laboratory results. Half of those cases belong to HRC 16 and 17, in which
videos were coded with 1.0 and 1.2 Mbps bitrates. We anticipate the difference
is due to the fact that the laboratory test was conducted using a 40” screen,
whereas crowd workers used their PC and laptop screens (typically 11 to 24”).
The distribution of display resolution and estimated display refresh rate of
crowd workers who participated in our studies are presented in Figure 7a and
Figure 7b, respectively.

TABLE VIII: Effect of calibration task and trapping clips.

Case | PCC | SRCC | RMSE | RMSE FOM | Sig. diff. to lab
---|---|---|---|---|---
Complete test | 0.95 | 0.95 | 0.72 | 0.34 | 16%
No Calibration | 0.91 | 0.89 | 0.80* | 0.47 | 23%
No Trapping clip | 0.92 | 0.92 | 0.88 | 0.46 | 29%
* Beside this value, all statistics are significantly different compared to the Complete test
at $\alpha=.05$

###  IV-D Number of ratings

We combined all the crowdsourcing studies in which the dataset VQEG HDTV3 and
the complete set of quality control mechanisms were used. We conducted a
bootstrapping simulation study to determine how the accuracy of MOS scores
changes depending on the number of valid votes per sequence. In the
bootstrapping process, we randomly selected N votes per sequence with
replacement and calculated the MOS values per sequence and consequently
correlation and RMSE between those MOS values and MOS values from Lab or MOS
values from crowdsourcing when all valid votes are used. We repeated this
process 200 times per the number of votes (N) and calculated the mean and 95%
confidence interval for all statistics. Figure 8a illustrated changes in
correlation coefficients and Figure 8b changes in RMSE by increasing the valid
number of votes used for calculating MOS. The results show that both
coefficients and RMSE plots saturated on about 40 votes.

(a)

(b)

Figure 8: Changes of different statistics by increasing the number of votes
used for calculating MOS, (a) Correlation coefficients, (b) RMSE.

##  V Test Methods

We compared the quality of compressed videos submitted to the CVPR 2022 CLIC
challenge using the ACR, DCR, and CCR test methods. Each of the 12 teams
compressed the 30 videos in the test set targeting 1 Mbps and 0.1 Mbps average
bitrates during the challenge, creating an overall 27 sets of clips including
the original sequences and two sets encoded by H.264 and AV1. On average, we
have collected 21.6 valid votes per sequence in the ACR test, 12.1 in the DCR
test (on a 9-point discrete scale), and 15.2 in the CCR test. As the original
sequences were included in the test set, we also calculated the Differential
MOS which is typically the result of the ACR with Hidden References (ACR-HR)
test method. Finally, we aggregated the ratings on the sequence (i.e., clip)
and also on the model (i.e., submission of a team for 0.1 MBps or 1 MBps task)
level. Table IX reports the correlation coefficients between aggregated
ratings. The highest correlation was observed between the ratings from ACR and
ACR-HR (single stimulus methods) and between the DCR and CCR methods (double-
stimulus methods). Correlations to the most used objective measures are also
reported in Table X, which shows a poor correlation with subjective scores.

The ITU-T Rec. P.910 recommends using double-stimulus test methods with
explicit reference when the transparency or fidelity of the underlying process
or transmission should be evaluated or when high-quality systems are under
test. These methods lead to longer test sessions or more participants and
consequently higher costs compared to the single-stimulus approach (about
$\times 2$). We also observed a large difference in the quality ranking of
sequences according to single- or double-stimulus tests when they contain 1)
imperfect source sequence (e.g., low image quality, or slow motioned), 2)
specific impairment due to encoding process (e.g., color changes or blur
effect on non-saliency area), 3) source sequence with bokeh.

TABLE IX: Pearson correlations (upper-triangle) and Spearman’s rank
correlations (lower-triangle) between ratings collected via different test
methods, aggregated in model and clip level.

Test | Model level |  | Clip level
---|---|---|---
method | ACR | ACR-HR | DCR | CCR |  | ACR | ACR-HR | DCR | CCR
ACR | - | 0.999 | 0.981 | 0.963 |  | - | 0.975 | 0.948 | 0.920
ACR-HR | 0.999 | - | 0.983 | 0.966 |  | 0.964 | - | 0.939 | 0.912
DCR | 0.976 | 0.981 | - | 0.990 |  | 0.948 | 0.935 | - | 0.962
CCR | 0.987 | 0.989 | 0.981 | - |  | 0.936 | 0.928 | 0.960 | -

TABLE X: Correlation between ratings collected via different subjective test
methods and prediction of objective measures in clip level.

Test | Pearson correlation |  | Spearman’s rank correlation
---|---|---|---
method | PSNR | MS-SSIM | VMAF |  | PSNR | MS-SSIM | VMAF
ACR | 0.691 | 0.757 | 0.914 |  | 0.691 | 0.830 | 0.896
ACR-HR | 0.725 | 0.773 | 0.919 |  | 0.731 | 0.855 | 0.903
DCR | 0.735 | 0.722 | 0.915 |  | 0.751 | 0.861 | 0.914
CCR | 0.706 | 0.688 | 0.893 |  | 0.713 | 0.843 | 0.910

##  VI Discussion and Conclusions

We have created the first crowdsourced implementation of ITU-T Rec. P.910 that
is shown to be as accurate as a lab study and is highly reproducible. In a set
of experiments, we showed that each test component significantly improves the
validity of the collected ratings.

It should be noted that the impact of the test components depends on the
reliability of crowd workers (and their environment and device) and the test
sequences under study. If a dedicated group of workers performs the test in a
suitable environment using proper devices, those components might look
redundant. However, they strongly increase the validity of collected scores
and robustness of the test method despite the group of test participants and
consequently make large-scale data collection possible. Furthermore, we let
the crowd workers watch the videos as often as they want, in contrast to the
strict one-time play-back policy and limited time for casting votes typically
used in laboratory tests. We chose this approach to compensate for the
uncontrolled nature of the crowd worker’s environment, allowing participants
to watch the videos again in case of unexpected interruptions in their flow.
Meanwhile, the framework logs the number of times a video sequence is played
back. Within our experiments, only 6.3% of videos were played back more than
once.

We plan to include the simultaneous presentation of sequence pairs for future
work and include ITU-T Rec. P.910 Annex E: An advanced data analysis technique
for tests under challenging conditions. In addition, this framework can be
easily extended to measure image quality assessment, which is also missing an
open-source crowdsourcing implementation with rater, environment, and device
qualifications as well as accuracy and reproducibility studies.

## References

  * [1] “ITU-T Recommendation P.910 Subjective video quality assessment methods for multimedia applications,” International Telecommunication Union, Tech. Rep., 2021.
  * [2] J. Deng, W. Dong, R. Socher, L.-J. Li, Kai Li, and Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition_. Miami, FL: IEEE, Jun. 2009, pp. 248–255.
  * [3] B. Naderi and R. Cutler, “An Open Source Implementation of ITU-T Recommendation P.808 with Validation,” _INTERSPEECH_ , pp. 2862–2866, Oct. 2020.
  * [4] R. Cutler, B. Naderi, M. Loide, S. Sootla, and A. Saabas, “Crowdsourcing approach for subjective evaluation of echo impairment,” in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020.
  * [5] B. Naderi and R. Cutler, “Subjective Evaluation of Noise Suppression Algorithms in Crowdsourcing,” in _INTERSPEECH_ , 2021.
  * [6] H. Dubey, V. Gopal, R. Cutler, A. Aazami, S. Matusevych, S. Braun, S. E. Eskimez, M. Thakker, T. Yoshioka, H. Gamper, and R. Aichner, “ICASSP 2022 Deep Noise Suppression Challenge,” in _ICASSP_ , 2022.
  * [7] R. Cutler, A. Saabas, T. Parnamaa, M. Purin, H. Gamper, S. Braun, and R. Aichner, “ICASSP 2022 Acoustic Echo Cancellation Challenge,” in _ICASSP_ , 2022\.
  * [8] C. K. A. Reddy, V. Gopal, and R. Cutler, “DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors,” in _INTERSPEECH_ , 2021.
  * [9] ——, “DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors,” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , May 2022, pp. 886–890, iSSN: 2379-190X.
  * [10] M. Purin, S. Sootla, M. Sponza, A. Saabas, and R. Cutler, “AECMOS: A speech quality assessment metric for echo impairment,” in _ICASSP_ , 2022\.
  * [11] L. Diener, S. Sootla, S. Branets, A. Saabas, R. Aichner, and R. Cutler, “INTERSPEECH 2022 Audio Deep Packet Loss Concealment Challenge,” in _INTERSPEECH_ , 2022, p. 5.
  * [12] R. Gonzalez and R. Woods, _Digital image processing_ , 3rd ed. Prentice Hall, 2006.
  * [13] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Transactions on Image Processing_ , vol. 13, no. 4, pp. 600–612, Apr. 2004.
  * [14] Z. Wang, E. Simoncelli, and A. Bovik, “Multiscale structural similarity for image quality assessment,” in _The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003_. Pacific Grove, CA, USA: IEEE, 2003, pp. 1398–1402.
  * [15] Z. Li, A. Aaron, I. Katsavounidis, A. Moorthy, and M. Manohara, “Toward A Practical Perceptual Video Quality Metric.” Tech. Rep., 2016. [Online]. Available: http://techblog.netflix.com/2016/06/toward-practical-perceptual-video.html.
  * [16] J. Lin, D. Liu, H. Li, and F. Wu, “M-LVC: Multiple Frames Prediction for Learned Video Compression,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Seattle, WA, USA: IEEE, Jun. 2020, pp. 3543–3551.
  * [17] Z. Hu, G. Lu, and D. Xu, “FVC: A New Framework towards Deep Video Compression in Feature Space,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Nashville, TN, USA: IEEE, Jun. 2021, pp. 1502–1511.
  * [18] O. Rippel, A. G. Anderson, K. Tatwawadi, S. Nair, C. Lytle, and L. Bourdev, “ELF-VC: Efficient Learned Flexible-Rate Video Coding,” in _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. Montreal, QC, Canada: IEEE, Oct. 2021, pp. 14 459–14 468.
  * [19] K. Seshadrinathan, R. Soundararajan, A. Bovik, and L. Cormack, “A Subjective Study to Evaluate Video Quality Assessment Algorithms,” in _Human Vision and Electronic Imaging_ , 2010.
  * [20] T. Wiegand, G. J. Sullivan, G. Bjøntegaard, and A. Luthra, “Overview of the H.264 / AVC Video Coding Standard,” _IEEE Transactions On Circuits And Systems For Video Technology_ , p. 19, 2003.
  * [21] Z. Li, C. Bampis, J. Novak, A. Aaron, K. Swanson, A. Moorthy, and J. De Cock, “VMAF: The Journey Continues,” Tech. Rep., 2018. [Online]. Available: https://netflixtechblog.com/vmaf-the-journey-continues-44b51ee9ed12
  * [22] M. Testolina and T. Ebrahimi, “Review of subjective quality assessment methodologies and standards for compressed images evaluation,” in _Applications of Digital Image Processing XLIV_ , vol. 11842\. SPIE, Aug. 2021, pp. 302–315.
  * [23] “Recommendation ITU-T P.912: Subjective video quality assessment methods for recognition tasks,” International Telecommunication Union, Tech. Rep., 2016.
  * [24] ITU-T Recommendation P.913, _Methods for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment_. Geneva: International Telecommunication Union, 2021.
  * [25] M. H. Pinson and L. Janowski, “A new subjective audiovisual & video quality testing recommendation,” _An Era of Change_ , p. 51, 2014.
  * [26] ITU-T Recommendation P.918, _Dimension-based subjective quality evaluation for video content_. Geneva: International Telecommunication Union, 2021.
  * [27] ITU-R Recommendation BT.500-14, _Methodologies for the subjective assessment of the quality of television images_. Geneva: International Telecommunication Union, 2019.
  * [28] T. Tominaga, T. Hayashi, J. Okamoto, and A. Takahashi, “Performance comparisons of subjective quality assessment methods for mobile video,” in _2010 Second International Workshop on Quality of Multimedia Experience (QoMEX)_ , Jun. 2010, pp. 82–87.
  * [29] C. Keimel, J. Habigt, C. Horch, and K. Diepold, “QualityCrowd-A Framework for Crowd-based Quality Evaluation,” in _Picture coding symposium_ , 2012.
  * [30] “Large-Scale Crowdsourcing Subjective Quality Evaluation of Learning-Based Image Coding,” _IEEE Visual Communications and Image Processing (VCIP 2021)_ , 2021.
  * [31] B. Rainer, M. Waltl, and C. Timmerer, “A web based subjective evaluation platform,” in _2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX)_ , Jul. 2013, pp. 24–25.
  * [32] J. Jung, M. Wien, and V. Baroncini, “ISO/IEC JTC 1/SC 29/AG 5: Guidelines for remote experts viewing sessions,” Nov. 2021.
  * [33] R. R. R. Rao, S. Göring, and A. Raake, “Towards High Resolution Video Quality Assessment in the Crowd,” in _2021 13th International Conference on Quality of Multimedia Experience (QoMEX)_ , Jun. 2021, pp. 1–6, iSSN: 2472-7814.
  * [34] T. Hossfeld, C. Keimel, M. Hirth, B. Gardlo, J. Habigt, K. Diepold, and P. Tran-Gia, “Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing,” _IEEE Transactions on Multimedia_ , vol. 16, no. 2, pp. 541–558, Feb. 2014. [Online]. Available: http://ieeexplore.ieee.org/document/6671455/
  * [35] ITU-T PSTR-CROWDS, _Subjective evaluation of media quality using a crowdsourcing approach_. Geneva: International Telecommunication Union, 2018.
  * [36] ITU-T Recommendation P.808, _Subjective evaluation of speech quality with a crowdsourcing approach_. Geneva: International Telecommunication Union, 2021.
  * [37] ITU-T Recommendation P.910, _Subjective video quality assessment methods for multimedia applications_. Geneva: International Telecommunication Union, 2021.
  * [38] J. Clark, “The ishihara test for color blindness.” _American Journal of Physiological Optics_ , 1924.
  * [39] “ophthalmic optics — visual acuity testing — standard and clinical optotypes and their presentation,” Geneva, CH, Standard.
  * [40] F. Schiffner and S. Möller, “Defining the relevant perceptual quality space for video and video-telephony,” in _2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX)_. IEEE, 2017, pp. 1–3.
  * [41] B. Naderi and S. Möller, “Application of just-noticeable difference in quality as environment suitability test for crowdsourcing speech quality assessment task,” in _2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX)_. IEEE, 2020, pp. 1–6.
  * [42] B. Treutwein, “Adaptive psychophysical procedures,” _Vision research_ , vol. 35, no. 17, pp. 2503–2522, 1995.
  * [43] Video Quality Experts Group, “Report on the validation of video quality models for high definition video content,” 2010. [Online]. Available: https://www.its.bldrdoc.gov/media/4212/vqeg_hdtv_final_report_version_2.0.zip
  * [44] ITU-T Recommendation P.1401, _Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models_. Geneva: International Telecommunication Union, 2020.