# HDR or SDR? A Subjective and Objective Study of Scaled and Compressed Videos

Joshua P. Ebenezer, Zaixi Shang, Yixu Chen, Yongjun Wu, Hai Wei, Sriram
Sethuraman, Alan C. Bovik J.P. Ebenezer, Z. Shang, and A.C. Bovik are with the
Laboratory for Image and Video Engineering, The University of Texas at Austin,
Austin, TX, 78712, USA e-mail<EMAIL_ADDRESS>Chen, Y. Wu, H.
Wei, and S. Sethuraman are with Amazon Prime Video.J.P. Ebenezer and Z. Shang
contributed equally to this work

###### Abstract

We conducted a large-scale study of human perceptual quality judgments of High
Dynamic Range (HDR) and Standard Dynamic Range (SDR) videos subjected to
scaling and compression levels and viewed on three different display devices.
HDR videos are able to present wider color gamuts, better contrasts, and
brighter whites and darker blacks than SDR videos. While conventional
expectations are that HDR quality is better than SDR quality, we have found
subject preference of HDR versus SDR depends heavily on the display device, as
well as on resolution scaling and bitrate. To study this question, we
collected more than 23,000 quality ratings from 67 volunteers who watched 356
videos on OLED, QLED, and LCD televisions. Since it is of interest to be able
to measure the quality of videos under these scenarios, e.g. to inform
decisions regarding scaling, compression, and SDR vs HDR, we tested several
well-known full-reference and no-reference video quality models on the new
database. Towards advancing progress on this problem, we also developed a
novel no-reference model called HDRPatchMAX, that uses both classical and bit-
depth sensitive distortion statistics more accurately than existing metrics.

###### Index Terms:

High dynamic range, video quality assessment, video compression

##  I Introduction

High Dynamic Range (HDR) videos have utilize deeper bit-depths to represent
brighter and darker luminance with wider color gamuts than Standard Dynamic
Range (SDR) videos. To obtain the full benefits of HDR, however, a display
must have the technology to accurately represent high contrasts and the
extremes of the dynamic range. SDR videos are gamma-encoded using the power
law described in BT 709 [1]. When shown on a TV, these values are decoded and
adjusted to the TV’s display capabilities using a look-up table. Although SDR
standards were originally created for cathode ray televisions having a maximum
display brightness of 100 nits, modern display devices can use their entire
brightness range (often much greater than 100 nits) to display SDR content,
since the digital values of SDR content are relative and not scene-referred.
The exact mapping between the SDR digital values and what the television
displays are able to display differs across products and is ordinarily
proprietary.

Videos following the HDR10 standard, on the other hand, are scene-referred and
absolute. The PQ EOTF (used in the HDR10 standard) specifies the absolute
luminance that the display must show for a particular digital value. If the
absolute luminance value that is required to be shown is greater than the
highest luminance value that the TV can display, a tonemapping function called
the Electrical-Electrical Transfer Function (EETF) is applied on the HDR
content so that clipping does not occur at highlights, while ensuring a smooth
roll-off of brightness values at the peak. EETFs differ among televisions and
are also usually proprietary.

Due to differences in how SDR and HDR signals are displayed on HDR-capable
displays, an SDR version of a content may have a higher average brightness
than the HDR version, depending on how each is graded and displayed. For
example, a content having a maximum brightness of 200 nits in HDR may be
graded in SDR such that the digital value of the maximum brightness is 255.
When the HDR and SDR versions are displayed on an HDR-capable device having a
peak brightness of 1000 nits, the SDR version may present a peak brightness
much larger than 200 nits while the HDR version will be displayed with a peak
brightness of 200 nits. The SDR version may therefore appear brighter, but it
may also be washed out or oversaturated. Higher peak brightness or higher
average brightness are not the only reasons why HDR content can be more
appealing than SDR. Indeed, SDR videos may suffer from defects such as
saturation, banding, low-contrast, etc., which are less likely to occur in HDR
videos.

In addition to these differences, HDR videos use 10 bit representations
(stored in 16 bits), while SDR videos have 8 bit representations. HDR videos
require twice the number of bytes that SDR videos of the same content have.
Because of this, HDR videos may be more susceptible to compression artifacts.

The tradeoffs between compression, contrast-representation, color-
representation, and brightness make the perceptual assessment of HDR and SDR
video quality content-dependent and display-dependent. Towards better
understanding these tradeoffs, we have conducted a detailed subjective and
objective assessment of HDR and SDR videos having the same contents. For the
subjective study, we recruited 67 participants who viewed and rated the
qualities of 356 HDR and SDR videos of 25 source contents, which were
processed by various combinations of scaling and compression using the x265
encoder. We also evaluate objective full-reference (FR) and no-reference (NR)
video quality assessment (VQA) on the new subjective database. We also present
the design of a new NR VQA model for the task of predicting the quality of
both HDR and SDR videos.

##  II Related Work

To the best of our knowledge, there do not exist any studies that compare the
subjective qualities of videos as the dynamic range, resolution, and
compression levels are all varied. Existing databases such as LIVE Livestream
[2], LIVE ETRI [3], LIVE YTHFR [4], AVT UHD [5], and APV LBMFR [6] study the
subjective quality of professionally-generated SDR videos under conditions of
downsampling, compression, and source distortions. Other datasets including
Konvid-1k [7], YouTube UGC [8], and LSVQ [9] study the quality of SDR user-
generated content. UGC databases are typically much larger than those that
study the quality of professional-grade content because they can be conducted
online via crowdsourcing owing to looser requirements on the display devices,
resolution, and bitrate. LIVE HDR [10], LIVE AQ HDR [11], and APV HDR Sports
[12] are recent databases that study the quality of professionally-created HDR
videos that have been downsampled and compressed at various resolutions and
bitrates.

Each of the above-mentioned databases study the quality of either SDR or HDR
videos, but not both., that have been subject to distortions. Here we present
the first subjective study that compares the quality of HDR and SDR videos of
the same content, that have been processed by downscaling and compression. We
conducted the study on a variety of display devices using different
technologies and having differing capabilities.

While subjective human scores from studies like the one we conducted are
considered the gold standards of video quality, conducting such studies is
expensive is not scalable. However, objective video quality metrics are
designed and trained to automatically predict video quality and can be quite
economic and scalable. These fall into two categories: Full-Reference (FR) and
No-Reference (NR) models. FR VQA models require as take as input both pristine
and distorted videos to measure the quality of the distorted videos. NR
metrics only have access to distorted videos when predicting quality, hence
designing them is a more challenging problem. NR VQA models are relevant for
video source inspection as well as when measuring quality with no available
source video.

PSNR measures the peak signal to noise ratio between a reference frame and a
distorted version of the same frame. SSIM [13] incorporates luminance,
contrast, and structure features to predict the quality of distorted images.
VMAF [14] models the statistics of the wavelet coefficients of video frames,
as well as the detail losses from distortions. SpEED [15] measures the
difference in entropy of bandpass coefficients of reference and distorted
videos. STRRED [16] models the statistics of space-time video wavelet
coefficients altered by distortions. STGREED [17] measures differences in
temporal and spatial entropy arising from distortions to model the quality of
videos having varying frame rates and bitrates.

BRISQUE [18], VBLIINDS [19], VIDEVAL [20], RAPIQUE [21], ChipQA [22], HDR
ChipQA [23] and NIQE [24] are NR video quality metrics that rely on
neurostatistical models of visual perception. Pristine videos are known to
follow certain regular statistics when processed using visual neural models.
Distortions predictably alter the statistics of perceptually processed videos,
allowing for the design of accurate VQA models. RAPIQUE combines features
developed under these models with (semantic) video features provided by a pre-
trained deep network. TLVQM [25] explicitly models common distortions such as
compression, blur, and flicker, using a variety of spatial and temporal
filters and heuristics.

##  III Details of Subjective Study

The study was conducted on 356 videos shown to 67 subjects. The videos were
generated from a set of 25 unique and pristine contents also converted to SDR,
all subjected to combinations of downscaling and HEVC compression. Three
different television technologies were used to conduct the study.

###  III-A Source Sequences

The 25 source sequences can be divided into 4 groups: Video on Demand (VoD),
Live Sports, HDR Demo Videos, and SJTU videos. Three of the source sequences
were “anchor” sequences taken from prior HDR VQA databases (LIVE HDR, LIVE AQ
HDR, and APV HDR Sports) to calibrate and combine their data with the present
database. All of the videos are represented in the BT2020 [26] color gamut and
were quantized using the SMPTE ST2084 [27] Opto-Electronic Transfer Function
(OETF), also known as the Perceptual Quantizer (PQ). All of the source
sequences have durations in the range 7s and 10s, with static metadata
conforming to the HDR10 standard. The video categories are described in detail
as follows:

####  III-A1 VoD

These 7 videos were professionally captured and graded for VoD streaming
services. The HDR and SDR versions of these videos were prepared by Amazon
Studios. One of the contents used in this category is an “anchor” video from
the LIVE AQ HDR database. The SDR versions were created and manually graded
with creative intent by professional graders.

####  III-A2 Live Sports

The 4 Live Sports videos were captured professionally by broadcasters at
stadiums hosting live matches of Soccer and Tennis. One content from this
category is an “anchor” video from the APV HDR Sports database. Due to the
low-latency requirements of live broadcasts, these videos were graded using
preset Lookup Tables (LUTs) for both HDR and SDR formats. The LUT used for the
HDR grading is a proprietary Amazon LUT, while the LUT used for the HDR to SDR
conversion is the open-source NBC LUT. All the videos were originally in
YUV422 10-bit format and were converted to the limited YUV420 10-bit format.

####  III-A3 HDR Demo Videos

These are a set of 8 open-source videos collected from 4kmedia.org. The videos
were created and graded by television manufacturers to showcase the
capabilities of HDR over SDR, and hence these contents have a high degree of
contrast and colorfulness in order to be eye-catching. The source videos are
in the limited YUV420 10-bit format. They were converted to their SDR versions
using the NBC LUT.

####  III-A4 SJTU videos

Six source contents belong to this category. These contents were taken from
the open-source SJTU HDR Video Sequence Dataset. They were recorded using a
Sony F65 camera and graded using the S-Gamut LUT. The videos are in the
limited YUV420 10-bit format and were also converted to their SDR versions
using the NBC LUT. All of the source contents in this category are also
present in the LIVE HDR database, although the distorted versions differ.

###  III-B Content Descriptors

The Spatial Information (SI), Temporal Information (TI), Colorfulness, and
Average Luminance Level were computed for each video sequence and plotted by
their groups in Fig. 1. The VoD contents can be characterized as having lower
average brightness levels, lower SI, lower TI, and a lower colorfulness index
than the other categories. The Live content has high TI. The Demo Videos have
a high level of contrast, colorfulness, and brightness since they were
designed to showcase HDR’s capabilities. The SJTU videos include many
different scenes and hence have a high degree of variability. Their SI and TI
are lower than that of Groups 2 and 3, although one video of fireworks was an
outlier in this group for all four content descriptors.

(a) SI

(b) TI

(c) Colorfulness

(d) Avg. Luminance

Figure 1: Content Descriptors, where 1=VoD videos, 2=Live Sports videos, 3=HDR
demo videos, and 4=SJTU videos. See text for details.

###  III-C Processing of Source Sequences

The VoD contents include studio-graded HDR and SDR source versions, while the
other content categories are pristine HDR versions which were also converted
to SDR versions using the NBC LUT. The HDR and SDR versions were encoded using
the bitrate-capped Constant Rate Factor (CRF) method of the x265 encoder. The
maximum bitrates (maxrate), CRFs, and buffer sizes for each resolution are
listed in Table I.

TABLE I:  Resolution | CRF for HDR | CRF for SDR | Maxrate (kbps) | Buf. Size (kb)
---|---|---|---|---
540p | 25 | 30 | 800 | 2000
720p | 17 | 24.5 | 3000 | 6000
1080p | 20 | 27 | 3000 | 8000
2160p | 25 | 28 | 6000 | 15000
1440p | 15 | 19 | 10000 | 20000
2160p | 17.5 | 22 | 15000 | 22500

Nine HDR videos of the EPL6 content (from the Live Sports category) were taken
from the APV HDR Sports database, nine HDR videos of the NightTraffic content
(from the SJTU category were taken from the LIVE HDR database, and nine HDR
videos of the BTFB-01h05m40s content were taken from the LIVE AQ HDR database.
Each group of nine videos included the pristine version as well as eight
compressed versions. These 27 “anchor” videos were used to estimate a mapping
between the scores from prior HDR databases to the scores in the new HDR vs
SDR database. Each of the anchor contents also had 6 compressed SDR versions
at the maxrates and resolutions shown in Table I, as well as a pristine SDR
version. Hence each anchor content was associated with 16 video sequences each
(9 HDR versions and 7 SDR versions).

The remaining 22 source contents each were associated a pristine HDR version,
a pristine SDR version, six compressed HDR versions and six compressed SDR
versions, for a total of 308 videos. These videos, combined with those from
the anchor content, yield a total of 356 videos in the new database. There are
thus a total of 181 HDR videos and 175 SDR videos in the database.

###  III-D Display Devices

Three televisions were selected for the study: the 65” Samsung S95T (TV1), 65”
Samsung Q90 (TV2), and the 55” Amazon Fire TV (TV3). All of the televisions
are capable of receiving and displaying HDR10 content. Peak luminance refers
to the maximum brightness a television can get when displaying HDR. Different
televisions advertise different “peak luminance” capabilities, but they may
only be able to generate the advertised value in a small portion of the screen
for a short duration. This is done to prevent damage to the display and to
reduce power consumption, and is referred to as Auto Brightness Limiting. We
refer to the the peak luminance as the instantaneous brightness of a white
rectangle displayed on an area covering 2% of the screen, as reported by [28].

The Samsung S95 has a quantum dot organic light-emitting diode (QD-OLED)
display. Each pixel emits its own light, and hence the device can show very
high contrasts. It has 86.93% coverage of the BT 2020 color space and a peak
luminance of 1028 cd/m2. The Samsung Q90T is a Quantum Dot display with a
Vertical Alignment (VA) LED backlight. Quantum Dots emit red and green colors
with high accuracy, and can hence produce more vivid colors than standard LED
TVs. The Samsung Q90 has 67.24% coverage of the BT 2020 color space and a peak
luminance of 1170 cd/m2. The Amazon Fire TV is an entry-level VA LED TV with a
54.25% coverage of the Rec 2020 color gamut and a peak luminance of 230 cd/m2.
The Samsung Q90T and the Amazon Fire TV have the full array local dimming
(FALD) feature that dims the backlight brightness in areas of the screen that
are meant to be darker in order to increase the contrast. However, due to the
backlight, brightness can still “bleed” from brighter areas of the screen to
darker areas, which can affect contrast. The Samsung S95, on the other hand,
is an OLED display, hence each pixel can be controlled individually for better
contrasts than FALD can allow. However, the Samsung Q90T can achieve higher
peak brightness values than the S95 because of the presence of the LED
backlight as well as the Quantum Dots, which amplify light.

A Windows PC with an NVIDIA 3090 GPU running the Windows 10 Operating System
was used to drive the televisions via a HDMI 2.1 cable. The PC and the
televisions had HDR enabled. The screen resolutions were set at 3840x2160 and
the refresh rate was set at 30 Hz. The VLC media player was used for video
playback.

###  III-E Subjects

A total of 67 students at the University of Texas at Austin volunteered to
participate in the human study. All the subjects were between the ages of 20
and 28. Approximately two-thirds of the subjects were male, and the remaining
third identified as female. A demographic survey revealed that 73% of the
subjects identified as Asian, 20% identified as White, and 7% identified as
Black.

Among these, 22 subjects were assigned to watch TV1, 21 were assigned to watch
TV2, and 24 were assigned to watch TV3. None of the subjects were told about
the other TVs or the nature of the study in order to eliminate biases. All the
subjects passed the Ishihara test for color-blindness and the Snellen test for
visual acuity when wearing their corrective lenses (if needed).

###  III-F Subjective Testing Design

We employed a Single Stimulus method for the study, as described in ITU-R BT
500.13 [29]. Each video was shown once to each subjects and a quality score
was collected from the subject immediately after the video was shown. The
videos were displayed in random order and the reference and distorted videos
were not identified or given different treatment. Videos of the same content
were not allowed to be adjacent in the viewing order, in order to reduce
memory biases. Each quality score was collected using an invisible integer
scale from 1-100 with a continuous slider that had 5 verbal markers: “Poor,”
“Bad,” “Fair,” “Good,” and “Excellent.” The slider was operated by a mouse.
The study was divided into two sessions of approximately 40 minutes each, and
the two sessions were separated by at least 24 hours to reduce viewer fatigue.

Before each test session began, a training session was conducted whereby the
subject was familiarized with the setup using videos that are not a part of
the database. A set of six videos of the same content at varying levels of
compression were shown to each subject, three being in HDR and three being in
SDR, such that the quality range present in the database was fairly
represented. Subjects were shown how to use the scoring mechanism. During the
training session, instructions were given on how to rate the videos based on
their subjective assessment of the quality, while avoiding judgments on the
aesthetic content. No other instructions or details about the study were given
to avoid biasing the participants.

##  IV Subjective Analysis

###  IV-A Internal Correlation

The internal correlations for the three groups were calculated as follows. Let
the score given by subject $i$ for video $j$ on TV $t$ be given by $u_{ijt}$.
The $Z$ score is computed as

$Z_{ijt}=\frac{u_{ijt}-\frac{\Sigma_{j}u_{ijt}}{N}}{\sqrt{\frac{\Sigma_{j}(u_{ijt}-\Sigma_{j}u_{ijt})^{2}}{N}}}.$
(1)

The subjects were randomly divided into two equal groups and the average $Z$
score computed for each video across all subjects in that group. The
correlation between the scores provided by these groups for all the videos was
computed over 100 trails with different random groupings. The median inter-
subject correlation for viewers of TV1 was found to be 0.95, for TV2 was 0.94,
and for TV3 was 0.93. These data indicate a high degree of internal
consistency and reliability.

###  IV-B Calculation of MOS

The Mean Opinion Scores (MOS) were obtained using the Maximum Likelihood
Estimation method proposed in ITU 910 [30]. The MOS is modelled as a random
variable

$U_{ijn}=\Psi_{jn}+\Delta_{i}+\nu_{i}X$ (2)

where $\Psi_{jn}$ is the true quality of video $j$ viewed on TV$n$,
$\Delta_{i}$ is the bias of subject $i$, $\nu_{i}$ represents the
inconsistency of subject $i$, and $X\sim N(0,1)$ are i.i.d. Gaussian random
variables. Given the scores $u_{ijn}$, the true score for each video on each
television is estimated by treating $\Psi_{jn},\Delta_{i},\nu_{i}$ as free
parameters that are solved so that the model in (2) is the best fit to the
observed MOS. Specifically, $\Psi_{jt}$ is by maximizing the log-likelihood of
the observations using the Newton-Raphson solver.

The Differential Mean Opinion Scores (DMOS) were calculated between the
distorted and reference videos by finding the differences in MOS as follows:

$DMOS_{jt}=\Psi_{jt}-\Psi_{j_{0}t},$ (3)

where $j_{0}$ is the index of the reference video corresponding to video $j$
viewed on television TV$n$. The MOS and DMOS are thus computed for each video
and separately for each television.

###  IV-C Analysis of Scores

The MOS of 20 videos of 4 contents (1 content from each source group and 5
videos per content) are plotted against bitrate in Figs. 2, 3, 4, and 5. The
Forge video is from group 1, EPL is from group 2, ColorDJ is from group 3, and
Porsche is from group 4. Screenshots from the SDR versions of the source
contents are shown in the first column. The points on each line correspond to
540p, 720p, 1080p, 1440p, 2160p, and 2160p in increasing order, following
Table I.

The Forge video shows a sword being forged in a smithy. The light from the hot
iron contrasts strongly with the darkness and shadows around it and in the
background. The HDR version of the Forge video is rated higher than the SDR
version on TV1, since details are clearer in the HDR version. However, on TV2
and TV3, due to their reduced ability to display contrasts, higher compression
of the HDR video offsets the relative quality gain from the increased
contrast, and the SDR version was thus rated as better. The HDR version
appears darker and dimmer than the SDR version, due to the way in which HDR is
displayed differently from SDR (as discussed in the Introduction) as well as
the way in which the video was graded. The lower average brightness may also
contribute to the lower perceived quality of the HDR version of the video
content. The lower average brightness of the HDR version, the reduced contrast
of TV2 and TV3, and the high spatial complexity of the scene, may also explain
why, at higher bitrates, the SDR version was still rated higher. Similar
observations can be made about the EPL video taken from group 2, where the HDR
version was rated better on TV1, while the SDR version was rated better at
lower bitrates on TV2 and TV3. At higher bitrates, the HDR version was rated
higher, as compressive artifacts had less visual impact, despite the high
temporal complexity of the soccer scene.

The SDR version of the ColorDJ video suffers from over-saturation and
overexposure due to the wide range of colors and brightnesses present. The HDR
version, on the other hand, does not exhibit over-saturation or overexposure
because of its greater bit-depth and wider color gamut. This may explain why
the HDR version was generally rated better on all the televisions.

The Porsche video has a bright red paint on the car which is accurately
represented in HDR, but looks saturated in SDR. The video is also not
spatially or temporally complex, which may be because of the superior ability
of HDR to represent contrasts and bright colors, outweighing the effects of
compression. However, there is a sharp drop in the MOS vs bitrate curve for
the 720p HDR version of the video, encoded at 1515 kbps. The encoder made the
decision to encode the 1080p version at 1300 kbps, yet the 1080p version is
still rated higher than the 720p version. This is likely because rescaling
artifacts are more prominent than the compressive artifacts on this content
due to its low spatial complexity.

(a) Screenshot from video “Forge”.

(b) MOS vs bitrate for TV1

(c) MOS vs bitrate for TV2

(d) MOS vs bitrate for TV3

Figure 2: MOS vs bitrate plots for the three tested televisions on the “Forge”
video.

(a) Screenshot from video “EPL”.

(b) MOS vs bitrate for TV1

(c) MOS vs bitrate for TV2

(d) MOS vs bitrate for TV3

Figure 3: MOS vs bitrate plots for the three tested televisions on the “EPL”
video.

(a) Screenshot from video “ColorDJ”.

(b) MOS vs bitrate for TV1

(c) MOS vs bitrate for TV2

(d) MOS vs bitrate for TV3

Figure 4: MOS vs bitrate plots for the three tested televisions on the
“ColorDJ” video.

(a) Screenshot from video “Porsche”.

(b) TV1

(c) TV2

(d) TV3

Figure 5: MOS vs bitrate plots for the three tested televisions on the
“Porsche” video.

The average difference between the MOS of the HDR and SDR versions of all the
videos are plotted against the maxrate of each television in Fig. 6. As may be
seen, at lower bitrates SDR was rated higher than HDR, while at higher
bitrates the difference becomes positive. The drops in the curve of 1080p
content at maxrate 3000 kbps and of 2160p at maxrate 6000 kbps are indicative
of how for an optimal bitrate ladder, these resolutions should be encoded at
higher bitrates. It is important to note that these compression levels were
included in the study to represent a wide range of quality, and not to
represent an optimal bitrate ladder. For TV1, HDR was better than SDR at even
low bitrates (3000 kbps) and the difference in quality increased to 5 MOS
units at the highest bitrate. For TV2 and TV3, SDR quality was better than HDR
quality at low bitrates but the differences decreased and became slightly
positive at higher bitrates. This also indicates how the capabilities of the
display devices strongly influence the quality of HDR relative to that of SDR
content.

(a) MOS vs bitrate for TV1

(b) MOS vs bitrate for TV2

(c) MOS vs bitrate for TV3

Figure 6: Average difference between MOS of corresponding HDR and SDR videos
vs maxrate for different televisions.

###  IV-D Combining Databases

The scores given to the 27 anchor videos from the LIVE HDR, LIVE AQ HDR, and
the APV HDR Sports VQA datasets were used to map scores from those databases
to the current database. A logistic function was fitted to map the scores
assigned to the anchor videos in those databases to the scores of the same
videos in the LIVE HDRvsSDR database for each television. The logistic
function is

$f(x)=\frac{a-b}{(1+\exp\frac{-(x-c)}{s})}+b,$ (4)

where $x$ are the scores of the videos in prior databases, $f(x)$ is the
mapping to scores in the LIVE HDRvsSDR database for a particular television,
and $a,b,c$ and $s$ were separately solved for on each database using the
anchor videos for that database. Since each television and each (prior)
database was used to generate a different fitting function, a total of nine
functions were derived from the data. The functions that map scores from the
LIVE HDR database and the LIVE AQ HDR database to the scores obtained from the
three televisions used in the LIVE HDRvsSDR database are plotted in Figs. 12
and 13 in supplementary material, respectively. The mappings and results for
the APV HDR Sports database cannot be shown, for proprietary reasons.

Deriving these functions from the scores of the anchor videos enables the
merging of the three LIVE databases into a single, large-scale database of
1066 videos that were collected in a controlled laboratory environment.

##  V Objective Assessment

###  V-A Full Reference Video Quality Assessment

We tested the Peak Signal to Noise Ratio (PSNR), the Structural Similarity
Index Measure (SSIM) [13], Multi-Scale SSIM (MS-SSIM) [31], Video Multimethod
Assessment Fusion (VMAF) [14], Spatio-Temporal Reduced Reference Entropic
Differences (STRRED) [16], Spatial Efficient Entropic Differencing (SpEED)
[15], and the Space-Time GeneRalized Entropic Difference (STGREED) [17] video
quality models on the newly created database. We evaluated the FR VQA
algorithms by computing the Spearman’s Rank Ordered Correlation Coefficient
(SRCC) between the scores predicted by the algorithms and the ground truth
DMOS. We also fit the predicted scores to the DMOS using a logistic function

$l(s)=\frac{\beta_{1}-\beta_{2}}{1+\exp(-\frac{(x-\beta_{3})}{\beta_{4}})+\beta_{5}},$
(5)

and then computed Pearson’s Linear Correlation Coefficient (LCC) and the Root
Mean Square Error (RMSE) between the fitted scores and the DMOS, following
standard practice [32]. The results are presented in Table II .

As can be seen, the FR metrics did not perform well on the task, with VMAF
achieving the highest SRCC of 0.56 on TV1, 0.55 on TV2, and 0.59 on TV3. In
addition, since none of these models incorporate modelling of the display
device, their predictions were the same on all three televisions, except for
STGREED which requires an SVR to be trained separately for each TV. These
results underscore the need for further research on FR VQA for HDR and SDR
content. The predictions are plotted against the scores obtained for TV1 in
Fig.7.

TABLE II: Median SRCC, LCC, and RMSE of FR VQA algorithms on the new LIVE
HDRvsSDR Database. Standard deviations are in parentheses. Best results for
each TV are in bold.

Method | TV1 | TV2 | TV3
---|---|---|---
| SRCC$\uparrow$ | PLCC$\uparrow$ | RMSE$\downarrow$ | SRCC$\uparrow$ | LCC$\uparrow$ | RMSE$\downarrow$ | SRCC$\uparrow$ | LCC$\uparrow$ | RMSE$\downarrow$
PSNR | 0.1056 | 0.3170 | 18.8984 | 0.0380 | 0.2673 | 19.1349 | 0.1351 | 0.2918 | 17.7954
STRRED[16] | 0.1865 | 0.3228 | 18.8593 | 0.1287 | 0.2691 | 19.1252 | 0.1166 | 0.2331 | 18.0922
SpEED[15] | 0.3209 | 0.4535 | 17.7103 | 0.2326 | 0.3796 | 17.4674 | 0.1771 | 0.3095 | 16.6531
MS-SSIM [31] | 0.4172 | 0.4391 | 17.9224 | 0.3071 | 0.3502 | 17.3550 | 0.3181 | 0.3544 | 17.3256
SSIM [13] | 0.4807 | 0.4921 | 17.3459 | 0.4184 | 0.4439 | 17.7940 | 0.3818 | 0.4021 | 17.0344
VMAF [14] | 0.5654 | 0.5850 | 16.1606 | 0.5547 | 0.5810 | 16.1618 | 0.5971 | 0.6057 | 14.8045
STGREED [17] | 0.4873(0.0908) | 0.5114(0.0643) | 17.3710(1.0645) | 0.4719(0.0902) | 0.4814(0.0635) | 17.3219(1.1705) | 0.4471(0.1191) | 0.4663(0.0758) | 16.1962(1.2141)

(a) PSNR

(b) STRRED

(c) SpEED

(d) MS-SSIM

(e) SSIM

(f) VMAF

Figure 7: Scatter plots of DMOS for videos shown in TV1 against FR VQA
predictions with logistic fit in orange.

###  V-B HDRPatchMAX

Motivated by the low performance of existing models, we also designed a new
HDR NR-VQA algorithm called HDRPatchMAX. HDRPatchMAX utilizes features that
are relevant to SDR and HDR quality, as well as to motion perception.

####  V-B1 NIQE features

The first set of features in HDRPatchMAX are the same as those in NIQE [24].
NIQE can be used alone as a blind metric, but we average the 36 features and
the distance measure across all frames in each video and use the resulting 37
features for subsequent training.

####  V-B2 PatchMAX

The 37 features obtained from NIQE performed strongly on the LIVE HDR, LIVE AQ
HDR, and LIVE HDRvsSDR dataset, which prompted us to develop a patch-based
feature extraction method. We start by partitioning the frame into non-
overlapping patches of size $P\times P$ and computing their Mean Subtracted
Contrast Normalization (MSCN) coefficients. The MSCN coefficients
$\hat{V}[i,j,k_{0}]$ of a luma channel of a video frame or patch
$V[i,j,k_{0}]$ are defined as :

$\hat{V}[i,j,k_{0}]=\frac{V[i,j,k_{0}]-\mu[i,j,k_{0}]}{\sigma[i,j,k_{0}]+C},$
(6)

where $i\in 1,2..,M$, $j=1,2..,N$ are spatial indices, $M$ and $N$ are the
patch height and width, respectively, the constant $C$ imparts numerical
stability, and where

$\mu[i,j,k_{0}]=\sum\limits_{m=-L}^{m=L}\sum\limits_{l=-L}^{l=L}w[m,l]V[i+m,j+l,k_{0}]$
(7)

and

$\begin{split}\sigma[i,j,k_{0}]&=(\sum\limits_{m=-L}^{m=L}\sum\limits_{l=-L}^{l=L}w[m,l](V[i+m,j+l,k_{0}]\\\
&-\mu[i,j,k_{0}])^{2})^{\frac{1}{2}}\end{split}$ (8)

are the local weighted spatial mean and standard deviation of luma,
respectively. The weights $w=\\{w[m,l]|m=-L,\dots,L,l=-L,\dots,L\\}$ are a 2D
circularly-symmetric Gaussian weighting function sampled out to 3 standard
deviations and rescaled to unit volume, and $K=L=3$.

The standard deviation that is computed during the MSCN operation is averaged
across each patch and is used as a proxy for each patch’s contrast. The
patches are then divided into three groups based on the percentile of their
standard deviation relative to the standard deviation of the other patches in
the frame. High contrast patches are defined as patches having standard
deviations above the $(100-T)$th percentile, medium contrast patches are
defined as patches with standard deviations between the $T$th and $(100-T)$th
percentiles, and low contrast patches are those having standard deviations
less than the $T$th percentile.

The MSCN coefficients of these patches can be reliably modelled as Generalized
Gaussian Distributions (GGD), defined as

$g_{1}(x;\alpha;\beta)=\frac{\alpha}{2\beta\Gamma(\frac{1}{\alpha})}\exp[-(\frac{|x|}{\beta})^{\alpha}]$
(9)

where $\Gamma(.)$ is the gamma function:

$\Gamma(\alpha)=\int_{0}^{\infty}t^{\alpha-1}\exp(-t)dt.$ (10)

The shape and variance parameters of the best fit to the MSCN coefficients,
$\alpha$ and $\beta$, are extracted and used as quality-aware features.

Following this, compute the products of neighboring pairs of pixels in each
patch to capture correlations between them as follows:

$\displaystyle\begin{split}H[i,j,k_{0}]&=\hat{V}[i,j,k_{0}]\hat{V}[i,j+1,k_{0}]\\\
V[i,j,k_{0}]&=\hat{V}[i,j,k_{0}]\hat{V}[i+1,j,k_{0}]\\\
D_{1}[i,j,k_{0}]&=\hat{V}[i,j,k_{0}]\hat{V}[i+1,j+1,k_{0}]\\\
D_{2}[i,j,k_{0}]&=\hat{V}[i,j,k_{0}]\hat{V}[i+1,j-1,k_{0}].\end{split}$ (11)

These are modelled as following an Asymmetric Generalized Gaussian
Distribution (AGGD):

$g_{2}(x;\nu,\sigma_{l}^{2},\sigma_{r}^{2})=\begin{cases}\frac{\nu}{(\beta_{l}+\beta_{r})\Gamma(\frac{1}{\nu})}\exp(-(-\frac{x}{\beta_{l}})^{\nu})&x<0\\\
\frac{\nu}{(\beta_{l}+\beta_{r})\Gamma(\frac{1}{\nu})}\exp(-(\frac{x}{\beta_{r}})^{\nu})&x>0,\end{cases}$
(12)

where

$\beta_{l}=\sigma_{l}\sqrt{\frac{\Gamma(\frac{1}{\nu})}{\Gamma(\frac{3}{\nu})}}\quad\mathrm{and}\quad\beta_{r}=\sigma_{r}\sqrt{\frac{\Gamma(\frac{1}{\nu})}{\Gamma(\frac{3}{\nu})}},$
(13)

and where $\nu$ controls the shape of the distribution and $\sigma_{l}$ and
$\sigma_{r}$ control the spread on each side of the mode. The parameters
($\eta,\nu,\sigma_{l}^{2},\sigma_{r}^{2}$) are extracted from the best AGGD
fit to the histograms of each of the pairwise products in (11), where

$\eta=(\beta_{r}-\beta_{l})\frac{\Gamma(\frac{2}{\nu})}{\Gamma(\frac{1}{\nu})}.$
(14)

The parameters of the best fit to each patch are extracted as quality
features, and are averaged separately for each of the three categories of
patches, yielding a total of 54 features. This procedure is performed at two
scales to yield a total of 108 features. In addition to this, we also compute
the average temporal standard deviation of these features over every non-
overlapping group of five consecutive frames, and use the additional 108
features as spatio-temporal quality features. The patch size $P$ and the
percentile threshold $T$ are treated as hyperparameters, and were chosen as
$P=20$ and $T=10$ based on the results shown in Table III.

There are three important motivations for the use of contrast-separated
feature aggregation. Firstly, one of the primary ways HDR differs from SDR is
in its representation of local contrast. SDR frames, being limited to 8 bits,
cannot represent edges and details as well as HDR frames are. However, feature
responses corresponding to this increased contrast visibility may be masked by
feature responses from other regions of the frame which may not benefit from
the more accurate quantization of HDR. Explicitly separating feature responses
by contrast prevents this masking effect. For example, in Fig. 8(b), high
contrast regions of an HDR frame are shown in white, medium contrast regions
are shown in gray, and low contrast regions are shown in black. In Fig. 8(e),
the same is shown for the SDR version of the same frame. In the SDR version,
there are underexposed regions in the woman’s t-shirt and in the back of the
suit worn by the man in the foreground, which are highlighted as low-contrast
regions by our proposed contrast-based segmentation. In the HDR version, the
same regions are not underexposed and low-contrast regions are instead only
located in the plain walls in the background. The statistics of these regions
will accordingly be different and quality-aware.

Secondly, NSS modelling suffers in the presence of regions of very low
contrast (such as the sky). Prior observations on the Gaussianity of MSCN
coefficients of pristine frames are strongly validated by most natural scenes,
very smooth areas lacking any texture or detail can present rare exceptions to
this natural law. Separating such areas from regions of higher contrast and
texture may therefore improve the validity and power of these models.

Thirdly, contrast masking is an important visual phenomenon whereby
distortions may become less visible in the presence of high contrast textures,
providing additional motivation for separately analyzing regions having
different contrasts. The MSCN operation serves to model the contrast-gain
masking that occurs in the early stages of the human visual system, while the
explicit separation of regions with different contrasts contributes to
additional modelling of this effect.

The MSCN coefficients of different patches grouped by their contrasts are
plotted in Fig. 9 for a frame from a compressed HDR video, along with the
corresponding SDR frame. The coefficients clearly differ and the best GGD fits
to these distributions will be accordingly quality-aware.

####  V-B3 HDRMAX

The HDRMAX feature set [33] is extracted by first passing the luma values of
each frame through an expansive nonlinearity, first introduced in HDR ChipQA.
Values in overlapping windows of size $20\times 20$ are first linearly
rescaled to $[-1,1]$ using the minimum and maximum values in each window. They
are then passed through an expansive nonlinearity defined as follows:

$f(x;\delta)=\begin{cases}\exp(\delta x)-1&x>=0\\\ 1-\exp((-\delta
x))&x<0\end{cases}$ (15)

where $\delta=4$ based on prior experiments [33] on the LIVE HDR database.
This nonlinearity amplifies local contrasts by suppressing the middle range of
values and by amplifying the extreme ends of the luminance range. Since HDR
excels in representing contrasts, this operation enhances endemic compression
distortions and forces subsequent feature responses to focus on them.

In Fig. 8(c) and Fig. 8(f), for example, the HDRMAX nonlinearity is applied to
the frames shown in Fig. 8(a) and Fig. 8(d), respectively. Local contrast is
amplified in the resulting images throughout the frame. Regions in SDR
suffering from underexposure, such as the woman’s t-shirt, and overexposure,
such as the tubelight in the room behind, present luminance values that are
near constant and are hence stretched to extremes with sharp boundaries that
highlight their defects. The corresponding regions in the HDR version are
smoother, and distortions in those regions highlight HDR’s enhanced ability to
represent contrast.

The MSCN coefficients of the nonlinearly processed frames are computed and
modelled as following a GGD, and their neighboring products are modelled as
following an AGGD, yielding 18 features. These features are extracted at two
scales. In addition, we also find the average standard deviation of these
features on every non-overlapping group of five frames, yielding 72 features.

(a) HDR Grayscale

(b) Contrast-based segmentation of HDR

(c) HDR after HDRMAX nonlinearity

(d) SDR Grayscale

(e) Contrast-based segmentation of SDR

(f) SDR after HDRMAX nonlinearity.

Figure 8: Comparing HDR and SDR.

(a) High Contrast Patches

(b) Medium Contrast Patches

(c) Low Contrast Patches

Figure 9: Comparing HDR and SDR MSCN coefficients of different contrast
groups.

####  V-B4 Space-Time Chips

We model spatio-temporal information using Space-Time (ST) Gradient Chips,
first introduced in [34] and further developed in [22]. Gradients carry
important information about edges and distortions and ST chips are
spatiotemporal slices of the gradient video that are designed to capture
temporal information in a quality-aware way. The gradient magnitude of each
frame is first computed using a $3\times 3$ Sobel operator. The MSCN
coefficients of the gradient magnitude are found using Eqn. (6). Following
this, a temporal bandpass filter $k(t)$ is applied:

$h(t)=t(1-at)\exp(-2at)u(t).$ (16)

This simple linear filter models the process of temporal decorrelation that
occurrs in the lateral geniculate nucleus (LGN) [35]. The visual signal passes
to area V1 from the LGN, where neurons are sensitive to motion over local
areas. Motivated by this, spatiotemporal slices of the MSCN coefficients of
the gradient video are selected from each spatial location by performing a
grid search over six directions, and by selecting the slice having the
kurtosis closest to that of a Gaussian.

Specifically, let $\hat{G}[i,j,k]$ denote MSCN coefficients of the gradient
magnitudes of video frames with temporal index $k$. For $k=T$, define a block
of 5 frames ending at time $T$, denoted
$\hat{G}_{T}={\hat{G}[i,j,k];k=T-4,\dots T}$. Then, within each space-time
volume $\hat{G}_{T}$, define six 5x5 space-time slices or chips, which
intersect all 5 frames, which are separated by an angle of $\frac{\pi}{6}$,
and are constrained so that the center of each chip coincides with the center
of the volume $\hat{G}_{T}$, and so that the normal vector of every chip lies
on the spatial plane. ST chips that are perpendicular to the direction of
motion will capture the motions of objects along that vector, where we tacitly
assume that motions of small spatiotemporal volumes is translational. We have
previously shown that the natural bandpass statistics of ST chips of high-
quality videos that are oriented perpendicular to the local direction of
motion reliably follow a Gaussian law, while those pointing in other
directions diverge from Gaussianity [22]. This observation follows from the
Gaussianity of MSCN coefficients of pristine frames. Accordingly, in each
$5\times 5\times 5$ volume, we select the $5\times 5$ ST chip that has the
least excess kurtosis. The ST Gradient chips coefficients are modelled as
following a GGD, and their neighboring pairwise products are modelled as
following an AGGD, yielding 18 features. This procedure is repeated at two
scales, yielding a total of 36 features.

HDRPatchMAX hence consists of 37 NIQE features, 108 PatchMAX features, 72
HDRMAX features, and 36 ST Gradient Chip features.

TABLE III: Effect of varying patch size and percentile thresholds on median SRCC for TV1. Patch size $P$ | Percentile threshold $T$ | SRCC
---|---|---
20 | 10 | 0.8347(0.0828)
20 | 20 | 0.8244(0.0902)
20 | 30 | 0.7917(0.1144)
120 | 10 | 0.8035(0.0997)
120 | 20 | 0.8003(0.1061)
120 | 30 | 0.8055(0.1035)
240 | 10 | 0.7838(0.1197)
240 | 20 | 0.7853(0.1123)
240 | 30 | 0.7838(0.1197)

###  V-C No-Reference Video Quality Assessment

We evaluated RAPIQUE [21], BRISQUE [18], TLVQ [25], VBLIINDS [19], ChipQA
[22], and HDR ChipQA [23], as well as HDRPatchMAX, on the new LIVE HDRvsSDR
Database. Again, we report the SRCC, PLCC, and RMSE metrics. The algorithms
were trained using a Random Forest Regressor. We also tested the Support
Vector Regressor but chose the Random Forest Regressor based on its better
performance. The videos are separated into training and test sets with an
80:20 ratio such that videos of the same content appeared in the same set.
Cross-validation was performed over the training set to select the best
hyperparameters (the number of estimators and the number of features) for the
random forest. The random forest with the best hyperparameters was then fitted
to the training set and evaluated on the test set. This procedure was repeated
100 times with different randomized train-test splits, with the median results
reported along with the standard deviations in Table IV. HDRPatchMAX
outperformed the other NR VQA algorithms on all the televisions. The
parameters $P$ and $T$ were chosen based on the performance of the PatchMAX
set of features as $P$ and $T$ were varied, as reported in Table III.

TABLE IV: Median SRCC, LCC, and RMSE of NR VQA algorithms on LIVE HDRvsSDR.
Standard deviations are in parentheses. Best results for each TV are in bold.

Dataset | TV1 | TV2 | TV3
---|---|---|---
| SRCC$\uparrow$ | PLCC$\uparrow$ | RMSE$\downarrow$ | SRCC$\uparrow$ | PLCC$\uparrow$ | RMSE$\downarrow$ | SRCC$\uparrow$ | PLCC$\uparrow$ | RMSE$\downarrow$
RAPIQUE | 0.4814(0.1315) | 0.5769(0.1099) | 17.1953(2.1580) | 0.5163(0.1382) | 0.5713(0.1332) | 17.5951(1.7378) | 0.5116(0.1455) | 0.5279(0.1402) | 16.4634(2.0299)
BRISQUE | 0.6923(0.1339) | 0.7337(0.1221) | 15.6301(2.7714) | 0.7355(0.1009) | 0.7736(0.0820) | 13.2101(2.4794) | 0.7285(0.1094) | 0.7583(0.0920) | 12.9504(2.5819)
TLVQM | 0.7340(0.1022) | 0.7865(0.0873) | 12.9524(2.3336) | 0.7422(0.0876) | 0.7976(0.0698) | 12.4863(2.0532) | 0.7734(0.0929) | 0.8087(0.0747) | 11.7250(1.9711)
ChipQA | 0.7396(0.1040) | 0.7810(0.0918) | 13.0415(2.3269) | 0.7450(0.1052) | 0.7933(0.0822) | 11.9339(1.8891) | 0.7943(0.1038) | 0.7912(0.0915) | 12.2313(2.0476)
HIGRADE | 0.7610(0.1196) | 0.7928(0.1076) | 12.6439(2.7403) | 0.7950(0.1189) | 0.8246(0.0970) | 11.4478(2.3973) | 0.7497(0.1074) | 0.7679(0.0898) | 12.5681(2.2225)
VBLIINDS | 0.7895(0.0867) | 0.8060(0.0793) | 13.4819(2.3147) | 0.8005(0.0825) | 0.8216(0.0698) | 12.0408(2.1253) | 0.8052(0.0833) | 0.8085(0.0665) | 11.2490(2.0545)
VIDEVAL | 0.7913(0.0815) | 0.8210(0.0722) | 11.8524(2.0402) | 0.8014(0.0862) | 0.8248(0.0702) | 11.8614(2.0022) | 0.8252(0.0719) | 0.8275(0.0648) | 11.0899(1.8907)
HDRChipQA | 0.7941(0.0954) | 0.8129(0.0913) | 11.9236(2.4481) | 0.8116(0.0774) | 0.8209(0.0767) | 12.0703(2.2669) | 0.8112(0.0899) | 0.8135(0.0867) | 11.5474(2.3609)
HDRPatchMAX | 0.8586(0.0723) | 0.8524(0.0654) | 10.8200(2.0941) | 0.8436(0.0714) | 0.8588(0.0636) | 11.1900(2.0942) | 0.8494(0.0691) | 0.8551(0.0627) | 10.2514(2.0489)

The scatter plots of the predictions made by the NR VQA algorithms against the
MOS obtained on the videos shown on TV1 are shown in Fig. 10. The scatter
plots were created by plotting the MOS against the mean quality predictions
produced by the NR VQA algorithms on each video in the test set over the 100
train-test splits, using the logistic fit in Eqn. 5 shown in orange.
HDRPatchMAX’s predictions are better aligned with MOS and have a more linear
fit than the other algorithms.

(a) BRISQUE

(b) ChipQA

(c) TLVQM

(d) VBLIINDS

(e) HIGRADE

(f) VIDEVAL

(g) HDR ChipQA

(h) HDRPatchMAX

Figure 10: Scatter plots of MOS against NR VQA predictions with logistic fit
in orange.

We also show a boxplot of the SRCCs obtained over the 100 train-test splits by
the NR VQA algorithms on TV1 in Fig. 11. HDRPatchMAX has a higher median SRCC
and a tighter spread of values than the other algorithms.

Figure 11: Boxplot of SRCCs obtained by NR VQA algorithms for 100 train-test
splits

We also evaluated each feature set in HDRPatchMAX individually, and report the
results in Table V for TV1. The PatchMAX set of features obtained the highest
performance and individually exceeded the performance of other NR VQA
algorithms. We also conducted an ablation study with HDRPatchMAX, removing
each feature set and evaluating the performance of the rest of the feature
sets on TV1, and report the results in Tab. VI. Despite the strong individual
performance of PatchMAX, its removal had minimal impact on the algorithm’s
performance, indicating the robustness and complementary nature of the other
feature sets. Similar results were observed on TV2 and TV3.

TABLE V: Individual feature performance on TV1. Feature | SRCC$\uparrow$ | PLCC$\uparrow$ | RMSE$\downarrow$
---|---|---|---
ST Grad. Chips | 0.5872(0.1380) | 0.6264(0.1347) | 17.4190(2.5293)
HDRMAX | 0.7688(0.0647) | 0.7900(0.0682) | 13.9499(1.8542)
NIQE | 0.7829(0.0896) | 0.8057(0.0809) | 13.5136(2.3201)
PatchMAX | 0.8347(0.0828) | 0.8376(0.0794) | 12.7068(2.0619)
TABLE VI: Ablation study on TV1. Feature set removed | SRCC$\uparrow$ | PLCC$\uparrow$ | RMSE$\downarrow$
---|---|---|---
PatchMAX | 0.8553(0.0719) | 0.8567(0.0661) | 11.9267(2.0255)
ST Grad. Chips | 0.8524(0.0721) | 0.8579(0.0665) | 12.0564(1.9838)
NIQE | 0.8428(0.0720) | 0.8405(0.0715) | 12.3666(1.9113)
HDRMAX | 0.8334(0.0777) | 0.8375(0.0709) | 12.5534(1.9670)

The random forest models trained on the data from TV1, TV2, and TV3, can be
considered as separate models for OLED displays, QLED displays, and LED
displays respectively. However, we also conducted a separate experiment where
each TV was assigned a numerical value (1 for TV1, 2 for TV2, and 3 for TV3)
that was concatenated onto each NR VQA algorithm as an additional feature, and
trained on the scores obtained from all the display devices, treating videos
shown on different display devices as different stimuli with different feature
vectors. The results are shown in Table VII. HDRPatchMAX once again is the
leading performer.

TABLE VII: Results on all display devices using TV indices as additional features. Algorithm | SRCC$\uparrow$ | PLCC$\uparrow$ | RMSE$\downarrow$
---|---|---|---
RAPIQUE | 0.5106(0.1306) | 0.5789(0.1106) | 16.0880(1.9977)
BRISQUE | 0.7091(0.1221) | 0.7500(0.0926) | 13.1656(2.2399)
ChipQA | 0.7319(0.1046) | 0.7748(0.0884) | 12.4513(2.0950)
HIGRADE | 0.7673(0.1105) | 0.7937(0.0947) | 12.3120(2.3021)
TLVQM | 0.7491(0.0988) | 0.7957(0.0817) | 12.0173(2.1908)
HIGRADE | 0.7673(0.1105) | 0.7937(0.0947) | 12.3120(2.3021)
VBLIINDS | 0.7850(0.0812) | 0.8098(0.0600) | 11.4007(1.7672)
HDRChipQA | 0.7864(0.0894) | 0.8050(0.0878) | 11.3890(2.3069)
VIDEVAL | 0.7969(0.0812) | 0.8243(0.0733) | 11.2577(1.9316)
HDRPatchMAX | 0.8367(0.0732) | 0.8437(0.0605) | 10.4734(1.9913)

###  V-D Results on combined databases

We evaluated the NR and FR VQA algorithms on the combined LIVE HDR, LIVE AQ
HDR, and LIVE HDRvsSDR databases consisting of 891 HDR videos and 175 SDR
videos. The Samsung Q90T (TV2) was used for the LIVE HDR and LIVE AQ HDR
studies, so the scores from the LIVE HDR and LIVE AQ HDR databases were mapped
to the scores obtained from from TV2 in the LIVE HDRvsSDR database using the
procedure described in Section IV-D, and then used as the ground truth for
evaluation. The results are presented in Table VIII for FR metrics and Table
IX for NR metrics. HDRPatchMAX was the best performer on the combined dataset
among NR VQA models while VMAF was the best performer among FR VQA models.

TABLE VIII: Median SRCC, LCC, and RMSE of FR VQA algorithms on the combined databases. Standard deviations are in parentheses. Best results are in bold. Method | SRCC$\uparrow$ | PLCC$\uparrow$ | RMSE$\downarrow$
---|---|---|---
PSNR | 0.3208 | 0.4777 | 16.5218
SSIM [13] | 0.3618 | 0.3726 | 17.4523
MS-SSIM [31] | 0.3693 | 0.3794 | 16.5699
SpEED[15] | 0.3235 | 0.4237 | 17.0150
STRRED[16] | 0.3588 | 0.3921 | 17.3007
VMAF[14] | 0.6710 | 0.6930 | 13.5578
STGREED [17] | 0.6182(0.0687) | 0.6150(0.0641) | 14.8379(1.0856)
TABLE IX: Median SRCC, LCC, and RMSE of FR VQA algorithms on the combined databases. Standard deviations are in parentheses. Best results are in bold. Algorithm | SRCC$\uparrow$ | PLCC$\uparrow$ | RMSE$\downarrow$
---|---|---|---
RAPIQUE | 0.5684(0.0828) | 0.5440(0.0736) | 16.0103(0.9990)
BRISQUE | 0.6551(0.0621) | 0.6374(0.0560) | 14.5969(1.0516)
ChipQA | 0.7319(0.1046) | 0.7748(0.0884) | 12.4513(2.0950)
HIGRADE | 0.7673(0.1105) | 0.7937(0.0947) | 12.3120(2.3021)
TLVQM | 0.6997(0.0601) | 0.6916(0.0547) | 13.7720(1.0247)
HIGRADE | 0.6780(0.0736) | 0.6670(0.0665) | 13.9237(1.2181)
VBLIINDS | 0.7501(0.0572) | 0.7394(0.0524) | 12.8156(1.2017)
VIDEVAL | 0.7588(0.0559) | 0.7526(0.0529) | 12.4759(1.0645)
HDRChipQA | 0.7699(0.0537) | 0.7702(0.0484) | 12.2328(1.1006)
HDRPatchMAX | 0.8064(0.0516) | 0.8065(0.0445) | 11.2318(0.9964)

##  VI Conclusion

We presented the first ever study on comparing HDR and SDR videos of the same
content encoded at different bitrates and resolutions on different display
devices. Our study shows that despite HDR’s theoretical capabilities over SDR,
its perceptual quality depends heavily on the display device used in practice.
We also evaluated several NR and FR VQA algorithms on the new database, and
presented a novel NR VQA algorithm called HDRPatchMAX that exceeds the current
state-of-the-art on this database. We hope that this spurs research on the
modelling of display devices in VQA algorithms as well as for optimal bitrate
ladders for streaming.

## Acknowledgment

This research was sponsored by a grant from Amazon.com, Inc., and by grant
number 2019844 for the National Science Foundation AI Institute for
Foundations of Machine Learning (IFML). The authors also thank the Texas
Advanced Computing Center (TACC) at The University of Texas at Austin for
providing HPC resources that have contributed to the research results reported
in this paper. URL: http://www.tacc.utexas.edu.

## References

  * [1] ITU, “BT.709 : Parameter values for the HDTV standards for production and international programme exchange,” Intl. Telecomm. Union, Tech. Rep., 2011\.
  * [2] Z. Shang, J. P. Ebenezer, Y. Wu, H. Wei, S. Sethuraman, and A. C. Bovik, “Study of the subjective and objective quality of high motion live streaming videos,” _IEEE Trans. Image Process._ , vol. 31, pp. 1027–1041, 2022.
  * [3] D. Y. Lee, S. Paul, C. G. Bampis, H. Ko, J. Kim, S. Y. Jeong, B. Homan, and A. C. Bovik, “A subjective and objective study of space-time subsampled video quality,” _arXiv preprint arXiv:2102.00088_ , 2021.
  * [4] P. C. Madhusudana, X. Yu, N. Birkbeck, Y. Wang, B. Adsumilli, and A. C. Bovik, “Subjective and objective quality assessment of high frame rate videos,” _IEEE Access_ , vol. 9, pp. 108 069–108 082, 2021.
  * [5] R. R. R. Rao, S. Göring, W. Robitza, B. Feiten, and A. Raake, “Avt-vqdb-uhd-1: A large scale video quality database for uhd-1,” in _2019 IEEE Intl. Symposium on Multimedia (ISM)_. IEEE, 2019, pp. 17–177.
  * [6] J. P. Ebenezer, Y. Chen, Y. Wu, H. Wei, and S. Sethuraman, “Subjective and objective quality assessment of high-motion sports videos at low-bitrates,” in _2022 IEEE Intl. Conf. on Image Process. (ICIP)_ , 2022, pp. 521–525.
  * [7] V. Hosu, F. Hahn, M. Jenadeleh, H. Lin, H. Men, T. Szirányi, S. Li, and D. Saupe, “The Konstanz natural video database (Konvid-1k),” in _Int. Conf. Quality of Multimedia Experience_ , 2017, pp. 1–6.
  * [8] Y. Wang, S. Inguva, and B. Adsumilli, “Youtube UGC dataset for video compression research,” in _IEEE Int. Workshop Multimed. Signal Process._ IEEE, 2019, pp. 1–5.
  * [9] Z. Ying, M. Mandal, D. Ghadiyaram, and A. Bovik, “Patch-VQ: ‘patching up’ the video quality problem,” in _IEEE Conf. Comp. Vision Pattern Recognit._ , 2021, pp. 14 014–14 024.
  * [10] Z. Shang, J. P. Ebenezer, A. C. Bovik, Y. Wu, H. Wei, and S. Sethuraman, “Subjective assessment of high dynamic range videos under different ambient conditions,” in _IEEE Intl. Conf. Image Process._ , 2022, pp. 786–790.
  * [11] Z. Shang, J. P. Ebenezer, Y. Wu, H. Wei, S. Sethuraman, and A. C. Bovik, “A study of subjective and objective quality assessment of HDR videos,” _submitted to IEEE Trans. Image Process._ , 2022.
  * [12] Z. Shang, Y. Chen, Y. Wu, H. Wei, and S. Sethuraman, “Subjective and objective video quality assessment of high dynamic range sports content,” in _Proceedings of the IEEE/CVF Winter Conf. on Applications of Computer Vision (WACV) Workshops_ , January 2023, pp. 556–564.
  * [13] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Trans. Image Process._ , vol. 13, no. 4, pp. 600–612, 2004.
  * [14] Netflix, _VMAF: The Journey Continues_ , 2018 (accessed December 28, 2021). [Online]. Available: https://netflixtechblog.com/vmaf-the-journey-continues-44b51ee9ed12
  * [15] C. G. Bampis, P. Gupta, R. Soundararajan, and A. C. Bovik, “Speed-qa: Spatial efficient entropic differencing for image and video quality,” _IEEE Signal Process. Letters_ , vol. 24, no. 9, pp. 1333–1337, 2017.
  * [16] R. Soundararajan and A. C. Bovik, “Video quality assessment by reduced reference spatio-temporal entropic differencing,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 23, no. 4, pp. 684–694, 2012.
  * [17] P. C. Madhusudana, N. Birkbeck, Y. Wang, B. Adsumilli, and A. C. Bovik, “St-greed: Space-time generalized entropic differences for frame rate dependent video quality prediction,” _IEEE Trans. Image Process._ , vol. 30, pp. 7446–7457, 2021.
  * [18] A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” _IEEE Trans. Image Process._ , vol. 21, no. 12, pp. 4695–4708, 2012.
  * [19] M. A. Saad, A. C. Bovik, and C. Charrier, “Blind prediction of natural video quality,” _IEEE Trans. Image Process._ , vol. 23, no. 3, pp. 1352–1365, 2014\.
  * [20] Z. Tu, Y. Wang, N. Birkbeck, B. Adsumilli, and A. C. Bovik, “UGC-VQA: Benchmarking blind video quality assessment for user generated content,” _IEEE Trans. Image Process._ , vol. 30, pp. 4449–4464, 2021.
  * [21] Z. Tu, X. Yu, Y. Wang, N. Birkbeck, B. Adsumilli, and A. C. Bovik, “RAPIQUE: Rapid and accurate video quality prediction of user generated content,” _arXiv preprint arXiv:2101.10955_ , 2021.
  * [22] J. P. Ebenezer, Z. Shang, Y. Wu, H. Wei, S. Sethuraman, and A. C. Bovik, “ChipQA: No-reference video quality prediction via space-time chips,” _IEEE Trans. Image Process._ , vol. 30, pp. 8059–8074, 2021.
  * [23] ——, “HDR-ChipQA: No-reference quality assessment for high dynamic range videos,” _submitted to IEEE Trans. Image Process._ , 2023.
  * [24] A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “completely blind” image quality analyzer,” _IEEE Signal Process. Lett._ , vol. 20, no. 3, pp. 209–212, 2012.
  * [25] J. Korhonen, “Two-level approach for no-reference consumer video quality assessment,” _IEEE Trans. Image Process._ , vol. 28, no. 12, pp. 5923–5938, 2019.
  * [26] ITU, “ BT.2020 : Parameter values for ultra-high definition television systems for production and international programme exchange,” Intl. Telecomm. Union, Tech. Rep., 2015.
  * [27] S. Miller, M. Nezamabadi, and S. Daly, “Perceptual signal coding for more efficient usage of bit codes,” _SMPTE Motion Imaging J._ , vol. 122, no. 4, pp. 52–59, 2013.
  * [28] Rtings, _Peak Brightness Measurement_ , 2023 (accessed February 28, 2023). [Online]. Available: https://www.rtings.com/tv/tests/picture-quality/sdr-peak-brightness
  * [29] ITU, “BT.500 : Methodologies for the subjective assessment of the quality of television images,” Intl. Telecomm. Union, Tech. Rep., 2019.
  * [30] ——, “ITU. 910 : Subjective video quality assessment methods for multimedia applications,” Intl. Telecomm. Union, Tech. Rep., 2008.
  * [31] Z. Wang, E. Simoncelli, and A. Bovik, “Multiscale structural similarity for image quality assessment,” in _Asilomar Conf. Signals, Syst., Comput._ , vol. 2, 2003, pp. 1398–1402 Vol.2.
  * [32] H. R. Sheikh, M. F. Sabir, and A. C. Bovik, “A statistical evaluation of recent full reference image quality assessment algorithms,” _IEEE Trans. Image Process._ , vol. 15, no. 11, pp. 3440–3451, 2006.
  * [33] J. P. Ebenezer, Z. Shang, Y. Wu, H. Wei, S. Sethuraman, and A. C. Bovik, “Making video quality assessment models robust to bit depth,” _submitted to IEEE Signal Process. Letters_ , 2023.
  * [34] J. P. Ebenezer, Z. Shang, Y. Wu, H. Wei, and A. C. Bovik, “No-reference video quality assessment using space-time chips,” in _IEEE Intl. Workshop Multimedia Signal Process._ , 2020, pp. 1–6.
  * [35] D. W. Dong and J. J. Atick, “Temporal decorrelation: a theory of lagged and nonlagged responses in the lateral geniculate nucleus,” _Netw.: Comput. Neural Syst._ , vol. 6, no. 2, pp. 159–178, 1995.