# Highly Efficient No-reference 4K Video Quality Assessment with Full-Pixel
Covering Sampling and Training Strategy

Xiaoheng Tan  0009-0009-0348-6214 South China University of Technology,
Guangzhou, China Alibaba Digital Media & Entertainment Group, Beijing, China
<EMAIL_ADDRESS>,  Jiabin Zhang  0000-0002-5876-3575 Alibaba Digital
Media & Entertainment Group, Beijing, China<EMAIL_ADDRESS>,
Yuhui Quan  0000-0002-2564-7703 South China University of Technology,
Guangzhou, China Pazhou Lab, Guangzhou, China<EMAIL_ADDRESS>,  Jing Li
0000-0001-8645-9709 Alibaba Digital Media & Entertainment Group, Beijing,
China<EMAIL_ADDRESS>,  Yajing Wu  0000-0003-4735-0377 State Key
Laboratory of Multimodel Artificial Intelligence Systems, Institute of
Automation, Chinese Academy of Sciences, Beijing, China<EMAIL_ADDRESS>and
Zilin Bian  0000-0002-0677-5013 Alibaba Digital Media & Entertainment Group,
Beijing, China<EMAIL_ADDRESS>
(2024; 2024)

# Highly Efficient No-reference 4K Video Quality Assessment with Full-Pixel
Covering Sampling and Training Strategy

Xiaoheng Tan  0009-0009-0348-6214 South China University of Technology,
Guangzhou, China Alibaba Digital Media & Entertainment Group, Beijing, China
<EMAIL_ADDRESS>,  Jiabin Zhang  0000-0002-5876-3575 Alibaba Digital
Media & Entertainment Group, Beijing, China<EMAIL_ADDRESS>,
Yuhui Quan  0000-0002-2564-7703 South China University of Technology,
Guangzhou, China Pazhou Lab, Guangzhou, China<EMAIL_ADDRESS>,  Jing Li
0000-0001-8645-9709 Alibaba Digital Media & Entertainment Group, Beijing,
China<EMAIL_ADDRESS>,  Yajing Wu  0000-0003-4735-0377 State Key
Laboratory of Multimodel Artificial Intelligence Systems, Institute of
Automation, Chinese Academy of Sciences, Beijing, China<EMAIL_ADDRESS>and
Zilin Bian  0000-0002-0677-5013 Alibaba Digital Media & Entertainment Group,
Beijing, China<EMAIL_ADDRESS>
(2024; 2024)

###### Abstract.

Deep Video Quality Assessment (VQA) methods have shown impressive high-
performance capabilities. Notably, no-reference (NR) VQA methods play a vital
role in situations where obtaining reference videos is restricted or not
feasible. Nevertheless, as more streaming videos are being created in ultra-
high definition (e.g., 4K) to enrich viewers’ experiences, the current deep
VQA methods face unacceptable computational costs. Furthermore, the resizing,
cropping, and local sampling techniques employed in these methods can
compromise the details and content of original 4K videos, thereby negatively
impacting quality assessment. In this paper, we propose a highly efficient and
novel NR 4K VQA technology. Specifically, first, a novel data sampling and
training strategy is proposed to tackle the problem of excessive resolution.
This strategy allows the VQA Swin Transformer-based model to effectively train
and make inferences using the full data of 4K videos on standard consumer-
grade GPUs without compromising content or details. Second, a weighting and
scoring scheme is developed to mimic the human subjective perception mode,
which is achieved by considering the distinct impact of each sub-region within
a 4K frame on the overall perception. Third, we incorporate the frequency
domain information of video frames to better capture the details that affect
video quality, consequently further improving the model’s generalizability. To
our knowledge, this is the first technology for the NR 4K VQA task. Thorough
empirical studies demonstrate it not only significantly outperforms existing
methods on a specialized 4K VQA dataset but also achieves state-of-the-art
performance across multiple open-source NR video quality datasets.

4K video quality assessment, 4K video sampling strategy, Network training,
Transformer

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
email; October 28 - November 1, 2024; Melbourne, Australia.††isbn:
978-1-4503-XXXX-X/18/06††journalyear: 2024††copyright:
acmlicensed††conference: Proceedings of the 32nd ACM International Conference
on Multimedia; October 28-November 1, 2024; Melbourne, VIC,
Australia††booktitle: Proceedings of the 32nd ACM International Conference on
Multimedia (MM ’24), October 28-November 1, 2024, Melbourne, VIC,
Australia††doi: 10.1145/3664647.3680907††isbn: 979-8-4007-0686-8/24/10††ccs:
Computing methodologies Image processing††ccs: Information systems Multimedia
information systems

##  1\. Introduction

Advancements in multimedia technology have facilitated the distribution of
videos across multiple platforms (Sun et al., 2022; Liu et al., 2023).
However, the quality of these videos varies significantly. Thus, Video Quality
Assessment (VQA) has become essential for accurately understanding and
predicting the quality of user experience (QoE) (Chen et al., 2020; Xu et al.,
2021; Wu et al., 2023b). Moreover, advances in hardware and the increasing
demand for high-quality, high-resolution videos have led to a surge in 4K
videos. Unfortunately, many 4K videos are not originally filmed in 4K but are
instead generated from lower resolutions using techniques such as up-sampling
and super-resolution (Lu et al., 2022). This has led to massive low-quality
pseudo-4K videos, notably diminishing user experience on social platforms and
escalating resource usage. Hence, it is crucial to research and understand the
distinctions among various 4K video types, including those from the source
medium, high-quality 4K restored videos, and mid-to-low quality pseudo-4K
videos. Nonetheless, despite the urgent necessity, a specific approach for no-
reference (NR) 4K VQA is currently lacking.

---
(a)
(b)
Figure 1.  Comparison of data sampling strategies in VQA methods on 4K videos.
(a) The commonly used sampling strategies and our proposed strategy. (b) The
content-covering percentage of sampling strategies on 4K videos. Both
traditional and grid-based strategies cover only a minimal amount of content,
where the grid-based method also accesses some global information. In
contrast, our strategy is capable of covering the entire content.

Challenges:

While current VQA techniques have shown considerable advancements (Li et al.,
2019; You and Korhonen, 2019; Wang et al., 2021; Xu et al., 2021; Lu et al.,
2022), they encounter various limitations when handling NR 4K videos because
of the demanding requirements in terms of high-resolution and computational
resources. Typically, early deep learning (DL) methods for VQA involve
traditional pre-processing of video frames, such as resizing or cropping,
before performing quality evaluation (You and Korhonen, 2019; Wang et al.,
2021; Lu et al., 2022). Actually, resizing a frame can cause the loss of
crucial details, and cropping may result in the omission of essential visual
information from the original 4K video. As a result, both actions have the
potential to compromise the performance of VQA.

To mitigate the impact of the above operations on VQA, FAST-VQA (Wu et al.,
2022) utilizes a novel data sampling strategy based on Grid Mini-patch
Sampling. This data sampling strategy enables the learning of local detail
information (through cropping) while retaining certain global information (as
grids are uniformly divided across the entire frame). This innovative approach
has been successfully applied to VQA, inspiring numerous subsequent methods
(e.g. (Wu et al., 2023b; Liu et al., 2024)). Among them, DOVER (Wu et al.,
2023b) leverages the network of FAST-VQA and combines it with aesthetic and
technical perspectives to better understand and predict the QoE on videos.
SAMA further improves upon FAST-VQA, enhancing its performance with minimal
additional resource consumption (Liu et al., 2024). Despite this, when
cropping within a uniformly distributed grid, the grid size increases notably
with higher resolutions such as 4K videos. Consequently, if the input data
size to the network stays constant (Wu et al., 2022), there is a rapid
decrease in the proportion of useful information acquired.

To better explain the different information retrieval rates of different
sampling strategies, a comparison is made and shown in Fig. 1. It is clear
that both traditional and grid-based strategies can only cover a tiny fraction
of the content in 4K videos. As a result, the grid-based strategy almost
entirely loses the global semantic information, as shown in Fig. 1(b).
Moreover, in the commonly seen 4K video resolution (i.e., 3840x2160), the
cropped patches occupy only about 0.6% of the grid areas. This restriction
significantly limits the effectiveness of 4K VQA and possibly for videos of
even greater resolutions in the future.

Motivation and Main Idea:

High-resolution 4K videos present a challenge to existing data sampling
methods in capturing adequate information. Hence, the vital issue we need to
handle is  how to maximize video information fed into the network with limited
computational resources.  To confront this core issue head-on, this paper
proposes a highly efficient and novel data sampling and training strategy for
NR 4K VQA named Full-Pixel Covering (FuPiC). This strategy ensures that the
network can capture nearly all content of the sampled frames while
guaranteeing an efficient training process. Generally, 4K video frames are
partitioned into non-overlapping patches in our study, ensuring that all
content of the frames could be fed into the network. Adopting this full
sampling approach, we thoughtfully develop a novel training strategy. Instead
of the conventional method of treating samples within one batch as
”supervisory units,” we opt to gather multiple output results from samples
originating from the same frame and supervise the combination of these
results. Stemming from this concept, FuPiC enables the network to receive and
learn from the entire content (the cropped patches occupy 100%, far exceeding
0.6%. as depicted in Fig. 1), simultaneously ensuring that GPU resources are
adequate for handling the learning process of videos with high-resolution.

As stated earlier, FuPiC combines the results from different samples and
supervises the combined result. However, the second challenge we need to face
is  how to effectively and reasonably combine these results. To tackle this
challenge, inspired by the subjective perception model of humans when
evaluating videos, a weighting and scoring scheme named region-aware scoring
scheme is proposed to focus on different regions within the video. In
subjective 4K video assessments, humans evaluate various regions, such as
focal areas, background, and blur-affected regions due to jitter, each
impacting overall quality differently. Thus, we consider partitioned patches
as a series of regions, using a neural network to learn their weights and
scores to estimate overall video quality.

FuPiC feeds all the information of 4K videos into the network, and whether the
network can more effectively capture and utilize these details is the third
point of concern in our study. One notable observation from various real-world
4K video samples is the difference in detail between high and low scoring
videos, where higher scoring videos tend to contain more intricate
information. To delve deeper into the impact of these details on 4K video
quality, we employed Haar Wavelet Transform to shift video frames from the
image domain to the frequency domain. We observed that videos with similar
scenes tend to have richer high-frequency information if they score higher,
which becomes especially apparent after applying Haar Wavelet processing.
Consequently, we pre-process video frames with Haar Wavelets and use a linear
embedding layer to handle information across different frequencies,
integrating this data for network processing. This approach, named multi-
frequency feature fusion, enhances the performance of the network in assessing
4K video quality, and notably, the inference time was reduced to 25% of the
original one.

Additionally, a novel 4K VQA dataset with a large variety of scenes and a
reasonable quality distributed range for targeted 4K VQA methods is
constructed. To better meet the practical demands of the 4K VQA task, our
dataset construction follows a NR paradigm instead of a reference-based one,
meaning that all video clips originate from heterogeneous content.
Specifically, our dataset encompasses 200 ten-second video clips sourced from
a diverse array of occupationally generated content (OGC), including movies,
television dramas, and TV shows from different eras and countries. Initially,
we selected 200 long 4K videos guided by a criteria that the era, the genre,
and the region of these videos should be distributed as balanced as possible.
Subsequently, we randomly extract 10 ten-second short video clips from each
long video, resulting in a total of 2000 clips. Following this, we employ a
sampling strategy similar to (Wang et al., 2019; Xu et al., 2021) to distill
these into 200 representative ten-second video clips while some key video
indicators are considered, including spatial activity, temporal activity,
noise, brightness, and contrast. Then we adopt the Pair Comparison method in
our subjective experiment, which can efficiently obtain an accurate mean
opinion score (MOS) for each 4K video clip in the dataset.

Contributions:

To summarize, the main contributions of this work include:

  * •

We propose a novel data sampling and training strategy, namely Full-Pixel
Covering (FuPiC), for NR 4K VQA. FuPiC enables the network to receive all
content information for learning, while ensuring an easy training process.

  * •

We introduce a weighting and scoring scheme for FuPiC. This scheme emulates
the subjective quality evaluation procedure, which can predict the frame score
and eventually the overall video score more accurately.

  * •

By transforming video frame patches into the frequency domain and integrating
information from various frequency domains into the network, we emphasize the
network’s attention on the influence of high-frequency information on video
quality. This approach enhances the network performance while mitigating the
computational burden.

  * •

We have constructed the dataset explicitly tailored for NR 4K VQA.

Our approach significantly outperforms other VQA methods on our 4K VQA
datasets. Additionally, its excellent performance is also demonstrated by the
experiments on other VQA datasets.

Figure 2. Overall of our proposed method. We utilize a Swin Transformer as the
Encoder to extract features.

##  2\. Related Work

###  2.1. Classical VQA Methods

Classic VQA methods (Moorthy and Bovik, 2011; Mittal et al., 2012;
Soundararajan and Bovik, 2012; Saad et al., 2012, 2014; Mittal et al., 2015;
Ghadiyaram and Bovik, 2017; Korhonen, 2019; Madhusudana et al., 2021; Tu et
al., 2021; Liao et al., 2022) often rely on handcrafted features for quality
evaluation. Among them, V-BLIINDS (Saad et al., 2014) is a spatio-temporal
Natural Scene Statistics (NSS) model that quantifies the NSS features of frame
differences and motion coherency characteristics to assess video quality;
VIIDEO (Mittal et al., 2015) embodies models that capture intrinsic
statistical regularities of natural videos to quantify disturbances introduced
by distortions; TLVQM (Korhonen, 2019) first computes temporal low-complexity
handcrafted features and then uses them to extract high-complexity features.
Moreover, CNN-TLVQM (Korhonen et al., 2020) integrates TLVQM with spatial
features extracted from a pre-trained convolutional neural network (CNN) for
VQA tasks; AVQBits (Rao et al., 2022) is a bitstream-based method encompassing
the ITU-T P.1204.3 model from the ITU-T P.1204 famliy (Raake et al., 2020).

###  2.2. Deep VQA Methods

Classical VQA methods are known for their inefficiency and struggle to perform
well amidst the rapid development of multimedia technologies. With the advent
of DL, there has been a significant emergence of deep VQA methods. These
methods leverage the power of deep neural networks (DNN) to provide more
accurate and efficient quality assessments.

In earlier research, the majority of efforts are based on employing CNN models
(You and Korhonen, 2019; Sun et al., 2022; Li et al., 2019; Wang et al., 2021;
Ying et al., 2021; Li et al., 2021; Xu et al., 2021; Chen et al., 2021; Li et
al., 2022) to delve into the spatial-temporal information within videos for
quality assessment. Li et al.(Li et al., 2019) utilizes a pre-trained model to
extract features and further expands this approach to MDTVSFA (Li et al.,
2021) to explore the effectiveness of training a unified model across multiple
datasets. Ying et al.(Ying et al., 2021) introduces a local-to-global
architecture for predicting the overall video quality and leverages the
proposed PVQ Mapper to better learn spatial and temporal features. Wang et
al.(Wang et al., 2021) investigates the impact of content, technical quality,
and compression level on video quality. Sun et al.(Sun et al., 2022) proposes
an efficient network that directly extracts quality-aware features from raw
pixels of video frames, while extracting the motion features for accurate
prediction.

Later, as the Visual Transformer evolves, Transformer-based methods (Liu et
al., 2023; Wu et al., 2022, 2023a, 2023b; Liu et al., 2024) begin to exhibit
strong performance in VQA tasks. However, the Visual Transformer takes a high
computational cost, often relying on scaling or cropping video samples to
reduce the cost. To overcome the limitations associated with traditional
scaling and cropping strategy, Wu et al.(Wu et al., 2022) introduces a grid-
based sampling method, allowing for the simple and efficient use of Video Swin
Transformer (Liu et al., 2022) for end-to-end training and fine-tuning, while
maintaining a focus on global information. Building on this grid-based
approach, DOVER (Wu et al., 2023b) explores the influence of Aesthetics and
Technical quality on subjective evaluations; SAMA (Liu et al., 2024) further
refines the grid-based sampling method by incorporating scaling and masking
strategies to encompass multi-scale information within regular-sized inputs.
Despite the improvements brought by grid-based sampling to DNN-based methods
in assessing the quality of 1080P videos, challenges persist when dealing with
higher-resolution content (4K and beyond). These methods face significant
limitations as they only sample a small fraction of the video’s content,
impacting their performance adversely.

###  2.3. Databases for Video Quality Assessment

Databases are crucial for VQA tasks. Many VQA Databases (Péchard et al., 2008;
Seshadrinathan et al., 2010; Hosu et al., 2017; Nuutinen et al., 2016;
Ghadiyaram et al., 2017; Sinno and Bovik, 2018; Wang et al., 2019; Ying et
al., 2021) are built to tackle the challenge of VQA problems. Among them,
early VQA databases (Péchard et al., 2008; Seshadrinathan et al., 2010)
usually consist of limited numbers of source reference sequences (SRC) and
then introduce different degradations such as compression artifacts or
transmission errors to generate distorted sequences. Subsequently, some in-
the-wild databases (Hosu et al., 2017; Sinno and Bovik, 2018; Ghadiyaram et
al., 2017; Wang et al., 2019) have significantly promoted the development of
VQA tasks, due to their large diversity of content, different levels of
degradations and diverse types of distortions. However, a considerable number
of the videos in these databases are of low resolution, which does not meet
the urgent requirements on the evaluation of high-resolution video sequences,
especially in the Quality Control (QC) of the delivery of Full-High Definition
(HD) or Ultra-HD source videos. For instance, KoNViD-1k (Hosu et al., 2017)
consists of 540P videos. With the increasing popularity of 4K videos, some
databases (Cheon and Lee, 2017; Mackin et al., 2018; Rao et al., 2019, 2023b,
2023a; Xing et al., 2022) that exclusively include 4K videos are proposed.
However, most of these databases are built by conducting compression and up-
scaling on a few SRC videos, resulting in the simplicity of video scenes.

###  2.4. Quality Assessment for 4K Images/Videos

Only a few DL-based methods (Zhu et al., 2021; Lu et al., 2022) have been
specifically designed for NR 4K image/video quality assessment tasks due to
the unique challenges posed by high-resolution content. Zhu et al.(Zhu et al.,
2021) proposes to select three cropped patches with size 240 $\times$ 240
within 4K images based on local variances for efficiently extracting features.
Lu et al.(Lu et al., 2022) utilizes texture complexity measure to select three
patches with size 240 $\times$ 240 as inputs into their proposed BTURA model
for 4K content quality assessment. Moreover, BTURA is trained by utilizing an
extra information that whether a 4K content is true or pseudo, which could
potentially restrict the practical applications of BTURA.

##  3\. Proposed Method

The pipeline of our proposed method is illustrated in Fig. 2. As shown in the
figure, the Full-Pixel Covering (FuPiC) sampling and training strategy is
implemented on the frames extracted from the 4K video. In this process, the
Swin Transformer is equipped as the encoder to extract the features for the
input frames, and a proposed region-aware scoring scheme is utilized to
predict the frame score based on the patch score. After FuPiC, the scores of
all extracted frames are aggregated to compute the final score for the video.
Moreover, we use multi-frequency feature fusion to improve the performance of
our method. In this section, we first briefly introduce the Swin Transformer
in Sec. 3.1. Following that, we introduce FuPiC in detail (Sec. 3.2). Then we
present the region-aware scoring scheme in Sec. 3.3. The multi-frequency
feature fusion is described in Sec. 3.4. Finally, we introduce the dataset we
constructed for the NR 4K VQA task in Sec. 3.5.

###  3.1. Brief Introduction of Swin Transformer

The Visual Transformer (Dosovitskiy et al., 2020) has achieved significant
success in the field of computer vision, including image/video quality
assessment (Ke et al., 2021; Liu et al., 2023; Wu et al., 2022, 2023a, 2023b;
Liu et al., 2024). However, the Visual Transformer conducts self-attention
across the entire content, leading to excessive computational burden. In
contrast, the Swin Transformer (Liu et al., 2021) performs self-attention
within windows and employs shift window operations to ensure that self-
attention covers the entire content. This method significantly reduces
computational complexity while maintaining model effectiveness, enabling GPUs
to handle larger pixel content as input. Therefore, our method utilizes the
Swin Transformer as the encoder to extract features from video frames.

###  3.2. FuPiC Sampling and Training Strategy

A basic Swin Transformer is typically trained on training samples of 224
$\times$ 224 or 384 $\times$ 384 pixels due to the limitations of windows and
the computational capabilities of standard GPUs.(Liu et al., 2021). Even when
opting for the larger size of 384 $\times$ 384 pixels, it remains
significantly smaller than the resolution of 4K videos. To tackle the issue,
the sampling strategy in FuPiC partitions each frame of 4K video into non-
overlapping patches with the size of 384 $\times$ 384 pixels as the training
samples.

Following the sampling strategy, the patches with the size of 384 $\times$ 384
can be inputted into the network to achieve training or inference. However,
the network training process typically supervises individual samples within
one batch, treating each sample as a ”supervision unit”. This leads to a
situation where each patch must be assigned a score for training.
Nevertheless, since each patch represents only a portion of a video frame, we
cannot assign the same score of the video to each patch. Therefore, the
training strategy of FuPiC is proposed to supervise the subset of training
samples in one batch rather than each individual training sample.
Specifically, the training strategy of FuPiC arranges for all the patches
(samples) from the same 4K frame to be in the same training batch, and the
frame ID of each patch is also inputted. In each training iteration, the
outputs for patches(training samples) are transformed into the outputs for
frame (subsets of samples) by Eq. (1).

(1)
$\mathbf{\mathit{O}}_{f}^{k}=\sum_{i=1}^{\mathbf{N}}\mathbf{\mathit{o}}_{i}^{k}/N,$

where $\mathbf{\mathit{o}}_{i}^{k}$ represents the network output of each
sample from the same frame, k is the frame ID, N indicates the number of
partitioned patches and $\mathbf{\mathit{O}}_{f}^{k}$ denotes the predicted
score for frame k. Subsequently, $\mathbf{\mathit{O}}_{f}^{k}$ can be
supervised by the subjective score $\mathbf{\mathit{S}}_{f}^{k}$ of frame k,
utilizing Mean Squared Error (MSE) loss.

To summarize, the proposed FuPiC sampling and training strategy can input all
the original content of the 4K video frame in the form of multiple patches
while treating a complete 4K frame as a ”supervision unit”.

|  |  |
---|---|---|---
(a) Spatial of low score | (b) H1 of low score | (c) H2 of low score | (c) H3 of low score
|  |  |
(e) Spatial of high score | (f) H1 of high score | (g) H2 of high score | (h) H3 of high score
Figure 3. Comparing on multi-frequency of two similar frames from different
videos.

###  3.3. Region-aware Scoring Scheme

Utilizing FuPiC sampling and training strategy, our method ensures the
entirety of 4K video content can be fed into the network for training, and
each frame is treated as a ”supervision unit”. However, Eq. (1) assumes that
each sample from this unit impacts the unit equally. This assumption
introduces a challenge, as different samples (patches) from the same frame may
not uniformly influence the quality of the frame. Consequently, it becomes
imperative to devise a scheme for aggregating the scores of individual
samples, obtained through parallel network outputs, into a final score for the
frame.

To tackle this challenge, a region-aware scoring scheme is proposed to
consider the samples as a series of regions within the entire frame.
Subjective quality scores provided by viewers are often influenced by distinct
regions within the video, such as areas of focus, background, zones impacted
by motion blur, and regions with intricate details. To mimic the varied
effects that different regions may exhibit, the network outputs weights
$\mathit{w}_{i}^{k}$ for all parallel input patches, reflecting their
potential influence on the frame’s quality. Once these weights are computed,
the contribution of each region to the overall quality
$\mathbf{\mathit{y}}_{j}^{k}$ and the region-aware score of frame k can be
calculated as:

(2)
$\displaystyle\mathbf{\mathit{y}}_{j}^{k}=\mathbf{\mathit{w}}_{j}^{k}/\sum_{i=1}^{\mathbf{N}}\mathbf{\mathit{w}}_{i}^{k},\quad\mathbf{\mathit{O}}_{f}^{k}=\sum_{i=1}^{\mathbf{N}}\mathbf{\mathit{y}}_{i}^{k}\times\mathbf{\mathit{o}}_{i}^{k}.$

With the region-aware score $\mathbf{\mathit{O}}_{f}^{k}$, the network can
still be trained by utilizing MSE loss. During inference, the scores for all
sampled frames with intervals of $t$ are calculated, and the final score is
obtained by averaging the frame scores, as shown in Eq. (3).

(3)
$\mathbf{\mathit{Q}}=(\sum_{j=1}^{\mathbf{T//t}}\mathbf{\mathit{O}}_{f}^{j})/(\lfloor
T/t\rfloor),$

where $\mathbf{\mathit{Q}}$ represents the final score of the video predicted
by the network, and T represents the total number of frames of the video. In
our method, $t$ is set to be 10.

###  3.4. Multi-frequency Feature Fusion

For high-quality 4K videos with richer high-frequency information, existing
VQA methods typically feed spatial information directly into the network for
training. However, the critical details might be overlooked by the convolution
and down-sampling in the shallow layers of DNNs, which leads to less accurate
quality predictions for high-quality 4K content.

To address this, multi-frequency feature fusion is proposed to capture
sufficient detail information, which utilizes the Haar Transform (HT) to
convert 4K videos from the spatial to the frequency domain. A standard HT
layer reduces the spatial size of the input while increasing the channel
number by a stride-2 convolution with four kernels including
${\bm{L}}{\bm{L}}^{\top},{\bm{H}}{\bm{L}}^{\top},{\bm{L}}{\bm{H}}^{\top},{\bm{H}}{\bm{H}}^{\top}$,
where ${\bm{L}}=\frac{1}{\sqrt{2}}[1,1]^{\top}$ and
${\bm{H}}=\frac{1}{\sqrt{2}}[1,-1]^{\top}$. The low-pass filter
${\bm{L}}{\bm{L}}^{\top}$ acts as the average pooling on feature maps while
the three high-pass filters capture edge-like information with different
orientations.

To accurately demonstrate the HT’s ability to capture details in 4K videos, we
select two 4K videos with similar scenes and extracted frames that are highly
similar to each other. By applying HT to these frames and comparing the
results, visualizations can be observed in Fig. 3. In this figure, the upper
half represents a frame from a higher-scoring 4K video, while the lower half
represents a frame from a lower-scoring 4K video. Direct observation of
spatial information reveals high quality in both cases. However, upon closer
examination of different frequency domains, numerous distinctions become
evident, with the higher-scoring video exhibiting richer detail. Therefore,
pre-extracting these abundant details in various frequency domains using HT
can prevent the loss of such detail information during Linear Embedding. This
approach enables the network to more effectively focus on the crucial details
essential for 4K VQA.

Specifically, we down-sample the patches half to their original scale through
HT. The spatial information of the patches is then represented in the low-
frequency component obtained through average pooling, while the high-frequency
details from the original scale are preserved in three different edge-like
high-frequency maps. Assume that the patches derived from a frame are
contained within the set ${\mathbb{P}}$ and we consider an element $p$ from
${\mathbb{P}}$ as an example. Suppose the original scale patch
$p\in\mathbb{R}^{3\times l\times l}$, where $l$ represents the height and
weight of $p$. Then, the four components resulting from the Haar Transform are
$p_{\text{avg}}$, $p_{h}^{1}$, $p_{h}^{2}$, and $p_{h}^{3}$, where
$p_{\text{avg}}$, $p_{h}^{1}$, $p_{h}^{2}$, and
$p_{h}^{3}\in\mathbb{R}^{3\times\frac{l}{2}\times\frac{l}{2}}$. We employ
linear embedding layers with shared-weight to transform these multi-frequency
maps into embeddings. The network then adaptively fuses these embeddings. The
specific process is as follows:

(4) $p_{\text{avg}},p_{h}^{1},p_{h}^{2},p_{h}^{3}=\text{Split}(\text{HT}(p)),$
(5)
${\bm{z}}=\alpha_{1}\text{LE}(p_{\text{avg}})+\alpha_{2}\text{LE}(p_{h}^{1})+\alpha_{3}\text{LE}(p_{h}^{2})+\alpha_{4}\text{LE}(p_{h}^{3}),$

where ${\bm{z}}$ represents the input embeddings of the encoder, LE represents
the Linear Embedding Layer and $\alpha_{i}$ represents the learnable
parameters. Multi-frequency feature fusion enables the network to focus on
capturing more detail information regarding edges (extracted from the high-
frequency maps). It is noteworthy that, apart from the computation involved in
the HT itself, we do not introduce any additional computational cost.
Moreover, the input size of the encoder is reduced to 1$/$4 of that without
multi-frequency feature fusion, leading to a significant decrease in
computational burden.

###  3.5. 4K Video Dataset

####  3.5.1. Data collection

200 video clips from the digital masters of movies, television dramas, and TV
shows are collected from an online video streaming platform, Youku.com, guided
by a criteria that the era, genre, and region of these video clips are
distributed as balanced as possible. Firstly, 10 ten-second video sequences
are randomly clipped from each of 200 source videos, resulting in a total of
2000 ten-second video clips. Then 200 representative video clips are sampled
from the 2000 clips, where the sampling strategy in (Wang et al., 2019; Xu et
al., 2021) is adopted. In the sampling process, several video indicators
obtained by using (AGH University of Science and Technology, n d) are
considered including spatial activity, temporal activity, noise, brightness,
and contrast. Finally, the distributions of video indicators are shown in Fig.
4, and some video clip samples are illustrated in Fig. 6. It should be noted
that these videos are from the digital masters of the video streaming
platform, thus, the original resolutions of these videos are ranged from 720P
to 4K, with almost the best quality level at the corresponding era/region and
the corresponding resolution. In addition, there is also the possibility that
the digital master has been post-processed, such as through super-resolution
or enhancement, and might have suffered various distortions. All these videos
are then up-scaled to 4K for evaluation.

Figure 4. Distribution percentage of video indicators. The normalized feature space of each indicator is uniformly divided into 5 Bins. Figure 5. Distribution percentage of MOS in our dataset. |  |  |
---|---|---|---
(a) | (b) | (c) | (d)
|  |  |
(e) | (f) | (g) | (h)
Figure 6. Frames of video samples in our dataset. Details can be seen more
clearly when zoomed in. The MOS scores of the videos that (a)$\sim$(b) belong
to are 0.08, 0.20, 0.29, 0.41, 0.58, 0.68, 0.72, and 0.81, respectively.

####  3.5.2. Subjective Experiment

In our experiment, the PC method (ITU-T RECOMMENDATION, 1999) is adopted where
the subject is forced to choose which of the two stimuli is preferred. PC is
sensitive to the conditions with small differences so that the accurate and
reliable subjective opinions for the 4K videos can be obtained. However, the
number of comparisons increases exponentially with the number of processed
video sequences (PVSs). So we adopt the strategies in (Li et al., 2018) to
limit the number of comparisons while maximizing the accuracy that can be
reached. Specifically, each observer needs to watch 100 pairs of ten-second 4K
videos and choose the better one from each pair. The two stimuli are displayed
in a side-by-side way on two 4K 27-inches screens (HP-Z27K G3). The viewing
distance is approximately 1.6H. All test environment follows ITU-R
BT500(Series, 2012). The comparison list for the subsequent observer is based
on the previous subjective results. On average, each stimulus is observed 100
times in the experiment. The quality scores are re-scaled from 0 to 1. The
distribution of the final MOS scores is shown in Fig. 5. Some video clips and
their subjective scores are presented in Fig. 6.

##  4\. Experiments

###  4.1. Experimental Settings

Experiments are carried out on our developed 4K dataset, which encompasses a
diverse range of scenes and quality levels. In addition, two frequently used
open-source VQA datasets are included to validate the applicability of our
method across different high-resolution VQA tasks. Specifically, one dataset
is LIVE-Qualcomm (Ghadiyaram et al., 2017) dataset, comprising 1080P videos,
while the other is the YouTube-UGC (Wang et al., 2019) dataset, where 35% of
the videos are either 1080P or 4K videos.

Five DL-based methods are used for comparison, including two methods without
special design on data sampling including VSFA (Gu et al., 2019) and BVQA-2022
(Li et al., 2022); three methods with grid-based sampling strategy including
FAST-VQA (Wu et al., 2022), DOVER (Wu et al., 2023b) and SAMA (Liu et al.,
2024).

###  4.2. Performance Criteria

We employ two metrics, including the Pearson Linear Correlation Coefficient
(PLCC) and the Spearman Rank-Order Correlation Coefficient (SRCC), to evaluate
the performance of VQA. The PLCC computes the linear predictability of the VQA
algorithm, and the SRCC assesses the prediction monotonicity. Their values are
in the range of [0,1] and the higher value means the better performance.

To guarantee unbiased dataset partitioning and equitable comparisons, we
randomly divide the datasets into an 80% training set and a 20% testing set,
performing this split 10 times and presenting the median SRCC and PLCC scores.
For LiveVQC, LIVE-Qualcomm, and YouTube-UGC datasets, we cite comparison
results directly from existing literature when available. Otherwise, we
retrain the comparison methods under their original optimization protocols.

Table 1. Performance comparison on three datasets. Bold: best; and Underline: 2nd-best. Method | Source | Data Sampling | LIVE-Qualcomm | YouTube-UGC | Our 4K dataset | _Weighted Average_
---|---|---|---|---|---|---
SRCC $\uparrow$ |  PLCC $\uparrow$ |  SRCC $\uparrow$ |  PLCC $\uparrow$ |  SRCC $\uparrow$ |  PLCC $\uparrow$ |  SRCC $\uparrow$ |  PLCC $\uparrow$
VSFA (Gu et al., 2019) | ACMMM2019 | Traditional | 0.737 | 0.732 | 0.724 | 0.743 | 0.718 | 0.737 | 0.725 | 0.740
BVQA-2022 (Li et al., 2022) | TCSVT2022 | Traditional | 0.817 | 0.828 | 0.831 | 0.819 | 0.814 | 0.822 | 0.827 | 0.821
FAST-VQA (Wu et al., 2022) | ECCV2022 | Grid-based | 0.819 | 0.851 | 0.855 | 0.852 | 0.851 | 0.867 | 0.849 | 0.853
DOVER (Wu et al., 2023b) | ICCV2023 | Grid-based | 0.736 | 0.789 | 0.890 | 0.891 | 0.838 | 0.862 | 0.862 | 0.874
SAMA (Liu et al., 2024) | AAAI2024 | Grid-based | 0.815 | 0.829 | 0.881 | 0.880 | 0.848 | 0.882 | 0.868 | 0.873
Proposed | Proposed | Proposed | 0.823 | 0.862 | 0.876 | 0.883 | 0.914 | 0.932 | 0.873 | 0.886

###  4.3. Performance Comparison

The quantitative comparison results are shown in Table 1. From Table 1, it
could be observed that on our dataset specifically constructed for 4K VQA, our
method shows significantly better performance over other methods.
Specifically, it outperforms the second-best method by 0.063 (an increase of
7.4%) and 0.050 (an increase of 5.7%) on SRCC and PLCC, respectively. This
proves the effectiveness of our approach in capturing all information of 4K
content. On LIVE-Qualcomm, a dataset that exclusively contains 1080P videos,
our method also achieves state-of-the-art performance, outperforming the
second-best method with improvements of 0.004 and 0.011 on SRCC and PLCC,
respectively. This demonstrates our method’s capability on accurately
predicting the quality of high-resolution videos that close to 4K resolution.
On YouTube-UGC, our method secures second place, scoring 0.011 lower on PLCC
than DOVER (Wu et al., 2023b). A possible reason is that our method is
designed for capturing detailed information, and loses its advantage in low-
resolution videos, while 42% of videos in YouTube-UGC are 360P or 480P.
Overall, the comparison experiments illustrate how our method is effective not
just for 4K but also for other high-resolution VQA.

It is worth noting that our method has served in one of the world’s most well-
known video streaming platforms, Youku.com. It has already provided reliable
video quality evaluations in practical business scenarios such as Ultra-HD
video QC, digital master QC, video enhancement/super-resolution evaluation,
etc. As for practical, the inference time of our method on a full 4K frame
averages 0.041s, achieved on a 16GB V100 GPU.

Table 2. Results of ablation studies for the proposed method. | Method Setting | SRCC | PLCC
---|---|---|---
(a) | Plain Model I | 0.759 | 0.790
(b) | Plain Model II | 0.819 | 0.824
(c) |  Plain Model _w_ / FuPiC | 0.834 | 0.853
(d) |  (c) _w_ / Region-aware Scheme | 0.872 | 0.893
(e) | Proposed Method | 0.914 | 0.932

###  4.4. Ablation Studies

####  4.4.1. Ablation Studies for the Proposed Method

To see the performance contribution of each component within our method, we
conduct ablation studies by forming the following baseline models. (a) “Plain
Model I”: We resize the 4K video frames and input them directly into the
network to obtain frame scores. (b) “Plain Model II”: We crop patches from the
4K video frames randomly and input them into the network, utilizing the video
score to supervise the corresponding patches. (c) “Plain Model _w_ / FuPiC”: A
baseline that only utilizes the proposed FuPiC sampling and training strategy.
(d) “(c) _w_ / Region-aware Scheme”: A baseline that utilizes both the
proposed FuPiC strategy and region-aware scoring scheme.

The results of the ablation studies are listed in Table 2, where the final
method outperforms all the baselines noticeably. (i) The performance of the
plain models (a) and (b) is significantly poor, which highlights that resizing
or cropping the video would cause the loss of crucial details or result in the
omission of essential visual information. (ii) The result of (c) shows
significant improvement compared with (a) and (b), proving the importance of
maximizing video information fed into the network. (iii) When comparing (c)
with (d), it is evident that not utilizing the region-aware scheme shows a
decline in SRCC and PLCC by 0.038 and 0.040, respectively. This indicates that
reasonably combining the results of the patches is highly effective. (iv)
Against (d), our final method increases SRCC and PLCC by 0.042 and 0.039,
respectively. This suggests that effectively capturing the details of 4K
videos from the frequency domain enables the network more accurate. Moreover,
we conduct an experiment on replacing the shared linear projection with
distinct linear projection, the results decrease by 0.035 and 0.027 on SRCC
and PLCC, respectively, suggesting that there is no need to utilize different
linear projection specifically designed for different frequency information.
Instead, a simple projection is enough.

Table 3. Results of ablation studies for non-local/position information. Method Setting | SRCC | PLCC
---|---|---
_w_ / Non-local Information | 0.914 | 0.930
_w_ / Positional Information | 0.906 | 0.918
Proposed Method | 0.914 | 0.932
---
Figure 7. Visualization of the region-aware scheme. The right side shows the
visualization of the final weight for each patch, where the deeper green
represents a higher weight.

####  4.4.2. Ablation Studies for Non-local/Position Information

In addition to the previous experiments, we set up two more baselines to study
whether the incorporation of non-local or positional information is useful to
our proposed method or not. (f) “ _w_ / Non-local Information”: We integrate
embeddings containing information from neighboring regions with those of the
current patch, feeding these combined features into the network. (g) “ _w_ /
Positional Information”: We introduce learnable position encodings based on
the spatial locations of the patches. These encodings are also merged with the
embeddings of the current patch before being fed into the network.

The results presented in Table 3 indicate that fully leveraging the detailed
information in 4K videos and reasonably combining the results from different
regions is sufficient. Adding non-local/positional information would increase
computational cost and even lead to a decline in performance.

###  4.5. Visualization on Region-aware Scoring

In addition to comparative results, The visualization of the weights for the
regions in 4K frame predicted by our Region-aware Scoring Scheme is
illustrated in Fig. 7. The figure shows that the network effectively focuses
on the relatively important regions, which proves highly beneficial for the 4K
VQA task.

##  5\. Conclusion

A novel and highly efficient NR 4K VQA method is presented in the paper. Its
main component is the proposed FuPiC sampling and training strategy, which can
feed all the original content of 4K frames into our VQA network under
computational resource limitations. A region-aware scoring scheme is designed
to mimic human subjective perception, where each sub-region contributes
differently to the overall score of the 4K frame. Additionally, a novel multi-
frequency feature fusion approach is developed to enhance the performance of
the proposed method and significantly reduce inference time. Experiments prove
the effectiveness of the proposed method on our constructed 4K video quality
dataset and other open-source video quality datasets, resulting in high
practicability.

## References

  * (1)
  * AGH University of Science and Technology (n d) AGH University of Science and Technology. n. d..  Video Quality Indicators.  http://vq.kt.agh.edu.pl/metrics.html.
  * Chen et al. (2021) Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. 2021.  Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment.  _IEEE Transactions on Circuits and Systems for Video Technology_ 32, 4 (2021), 1903–1916.
  * Chen et al. (2020) Pengfei Chen, Leida Li, Lei Ma, Jinjian Wu, and Guangming Shi. 2020.  RIRNet: Recurrent-in-recurrent network for video quality assessment. In _Proceedings of the ACM International Conference on Multimedia_. 834–842.
  * Cheon and Lee (2017) Manri Cheon and Jong-Seok Lee. 2017.  Subjective and objective quality assessment of compressed 4K UHD videos for immersive experience.  _IEEE Transactions on Circuits and Systems for Video Technology_ 28, 7 (2017), 1467–1480.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ (2020).
  * Ghadiyaram and Bovik (2017) Deepti Ghadiyaram and Alan C Bovik. 2017.  Perceptual quality prediction on authentically distorted images using a bag of features approach.  _Journal of Vision_ 17, 1 (2017), 32–32.
  * Ghadiyaram et al. (2017) Deepti Ghadiyaram, Janice Pan, Alan C Bovik, Anush Krishna Moorthy, Prasanjit Panda, and Kai-Chieh Yang. 2017.  In-capture mobile video distortions: A study of subjective behavior and objective algorithms.  _IEEE Transactions on Circuits and Systems for Video Technology_ 28, 9 (2017), 2061–2077.
  * Gu et al. (2019) Jie Gu, Gaofeng Meng, Cheng Da, Shiming Xiang, and Chunhong Pan. 2019.  No-reference image quality assessment with reinforcement recursive list-wise ranking. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 8336–8343.
  * Hosu et al. (2017) Vlad Hosu, Franz Hahn, Mohsen Jenadeleh, Hanhe Lin, Hui Men, Tamás Szirányi, Shujun Li, and Dietmar Saupe. 2017.  The Konstanz natural video database (KoNViD-1k). In _International Conference on Quality of Multimedia Experience (QoMEX)_. IEEE, 1–6.
  * ITU-T RECOMMENDATION (1999) P ITU-T RECOMMENDATION. 1999.  Subjective video quality assessment methods for multimedia applications.  (1999).
  * Ke et al. (2021) Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. 2021.  Musiq: Multi-scale image quality transformer. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5148–5157.
  * Korhonen (2019) Jari Korhonen. 2019.  Two-level approach for no-reference consumer video quality assessment.  _IEEE Transactions on Image Processing_ 28, 12 (2019), 5923–5938.
  * Korhonen et al. (2020) Jari Korhonen, Yicheng Su, and Junyong You. 2020.  Blind natural video quality prediction via statistical temporal features and deep spatial features. In _Proceedings of the ACM International Conference on Multimedia_. 3311–3319.
  * Li et al. (2022) Bowen Li, Weixia Zhang, Meng Tian, Guangtao Zhai, and Xianpei Wang. 2022.  Blindly assess quality of in-the-wild videos via quality-aware pre-training and motion perception.  _IEEE Transactions on Circuits and Systems for Video Technology_ 32, 9 (2022), 5944–5958.
  * Li et al. (2019) Dingquan Li, Tingting Jiang, and Ming Jiang. 2019.  Quality assessment of in-the-wild videos. In _Proceedings of the ACM International Conference on Multimedia_. 2351–2359.
  * Li et al. (2021) Dingquan Li, Tingting Jiang, and Ming Jiang. 2021.  Unified quality assessment of in-the-wild videos with mixed datasets training.  _International Journal of Computer Vision_ 129 (2021), 1238–1257.
  * Li et al. (2018) Jing Li, Rafal Mantiuk, Junle Wang, Suiyi Ling, and Patrick Le Callet. 2018.  Hybrid-MST: A hybrid active sampling strategy for pairwise preference aggregation.  _Advances in Neural Information Processing Systems_ 31 (2018).
  * Liao et al. (2022) Liang Liao, Kangmin Xu, Haoning Wu, Chaofeng Chen, Wenxiu Sun, Qiong Yan, and Weisi Lin. 2022.  Exploring the effectiveness of video perceptual representation in blind video quality assessment. In _Proceedings of the ACM International Conference on Multimedia_. 837–846.
  * Liu et al. (2023) Hongbo Liu, Mingda Wu, Kun Yuan, Ming Sun, Yansong Tang, Chuanchuan Zheng, Xing Wen, and Xiu Li. 2023.  Ada-DQA: Adaptive Diverse Quality-aware Feature Acquisition for Video Quality Assessment. In _Proceedings of the ACM International Conference on Multimedia_. 6695–6704.
  * Liu et al. (2024) Yongxu Liu, Yinghui Quan, Guoyao Xiao, Aobo Li, and Jinjian Wu. 2024.  Scaling and Masking: A New Paradigm of Data Sampling for Image and Video Quality Assessment.  _arXiv preprint arXiv:2401.02614_ (2024).
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin transformer: Hierarchical vision transformer using shifted windows. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 10012–10022.
  * Liu et al. (2022) Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. 2022.  Video swin transformer. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3202–3211.
  * Lu et al. (2022) Wei Lu, Wei Sun, Xiongkuo Min, Wenhan Zhu, Quan Zhou, Jun He, Qiyuan Wang, Zicheng Zhang, Tao Wang, and Guangtao Zhai. 2022.  Deep neural network for blind visual quality assessment of 4K content.  _IEEE Transactions on Broadcasting_ (2022).
  * Mackin et al. (2018) Alex Mackin, Mariana Afonso, Fan Zhang, and David Bull. 2018.  A study of subjective video quality at various spatial resolutions. In _IEEE International Conference on Image Processing_. IEEE, 2830–2834.
  * Madhusudana et al. (2021) Pavan C Madhusudana, Neil Birkbeck, Yilin Wang, Balu Adsumilli, and Alan C Bovik. 2021.  ST-GREED: Space-time generalized entropic differences for frame rate dependent video quality prediction.  _IEEE Transactions on Image Processing_ 30 (2021), 7446–7457.
  * Mittal et al. (2015) Anish Mittal, Michele A Saad, and Alan C Bovik. 2015.  A completely blind video integrity oracle.  _IEEE Transactions on Image Processing_ 25, 1 (2015), 289–300.
  * Mittal et al. (2012) Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. 2012.  Making a “completely blind” image quality analyzer.  _IEEE Signal Processing Letters_ 20, 3 (2012), 209–212.
  * Moorthy and Bovik (2011) Anush Krishna Moorthy and Alan Conrad Bovik. 2011.  Blind image quality assessment: From natural scene statistics to perceptual quality.  _IEEE Transactions on Image Processing_ 20, 12 (2011), 3350–3364.
  * Nuutinen et al. (2016) Mikko Nuutinen, Toni Virtanen, Mikko Vaahteranoksa, Tero Vuori, Pirkko Oittinen, and Jukka Häkkinen. 2016.  CVD2014—A database for evaluating no-reference video quality assessment algorithms.  _IEEE Transactions on Image Processing_ 25, 7 (2016), 3073–3086.
  * Péchard et al. (2008) Stéphane Péchard, Romuald Pépion, and Patrick Le Callet. 2008.  Suitable methodology in subjective video quality assessment: a resolution dependent paradigm. In _International Workshop on Image Media Quality and its Applications_. 6.
  * Raake et al. (2020) Alexander Raake, Silvio Borer, Shahid M Satti, Jörgen Gustafsson, Rakesh Rao Ramachandra Rao, Stefano Medagli, Peter List, Steve Göring, David Lindero, Werner Robitza, et al. 2020\.  Multi-model standard for bitstream-, pixel-based and hybrid video quality assessment of UHD/4K: ITU-T P. 1204.  _IEEE Access_ 8 (2020), 193020–193049.
  * Rao et al. (2023a) Rakesh Rao Ramachandra Rao, Silvio Borer, David Lindero, Steve Göring, and Alexander Raake. 2023a.  PNATS-UHD-1-Long: An Open Video Quality Dataset for Long Sequences for HTTP-based Adaptive Streaming QoE Assessment. In _2023 15th International Conference on Quality of Multimedia Experience (QoMEX)_. IEEE, 252–257.
  * Rao et al. (2023b) Rakesh Rao Ramachandra Rao, Steve Göring, Bassem Elmeligy, and Alexander Raake. 2023b.  AVT-VQDB-UHD-1-Appeal: A UHD-1/4K Open Dataset for Video Quality and Appeal Assessment Using Modern Video Codecs. In _2023 IEEE 25th International Workshop on Multimedia Signal Processing (MMSP)_. IEEE, 1–6.
  * Rao et al. (2022) Rakesh Rao Ramachandra Rao, Steve Göring, and Alexander Raake. 2022.  Avqbits—adaptive video quality model based on bitstream information for various video applications.  _IEEE Access_ 10 (2022), 80321–80351.
  * Rao et al. (2019) Rakesh Rao Ramachandra Rao, Steve Göring, Werner Robitza, Bernhard Feiten, and Alexander Raake. 2019.  AVT-VQDB-UHD-1: A large scale video quality database for UHD-1. In _2019 IEEE International Symposium on Multimedia (ISM)_. IEEE, 17–177.
  * Saad et al. (2012) Michele A Saad, Alan C Bovik, and Christophe Charrier. 2012.  Blind image quality assessment: A natural scene statistics approach in the DCT domain.  _IEEE Transactions on Image Processing_ 21, 8 (2012), 3339–3352.
  * Saad et al. (2014) Michele A Saad, Alan C Bovik, and Christophe Charrier. 2014.  Blind prediction of natural video quality.  _IEEE Transactions on Image Processing_ 23, 3 (2014), 1352–1365.
  * Series (2012) B Series. 2012.  Methodology for the subjective assessment of the quality of television pictures.  _Recommendation ITU-R BT_ 500, 13 (2012).
  * Seshadrinathan et al. (2010) Kalpana Seshadrinathan, Rajiv Soundararajan, Alan Conrad Bovik, and Lawrence K Cormack. 2010.  Study of subjective and objective quality assessment of video.  _IEEE Transactions on Image Processing_ 19, 6 (2010), 1427–1441.
  * Sinno and Bovik (2018) Zeina Sinno and Alan Conrad Bovik. 2018.  Large-scale study of perceptual video quality.  _IEEE Transactions on Image Processing_ 28, 2 (2018), 612–627.
  * Soundararajan and Bovik (2012) Rajiv Soundararajan and Alan C Bovik. 2012.  Video quality assessment by reduced reference spatio-temporal entropic differencing.  _IEEE Transactions on Circuits and Systems for Video Technology_ 23, 4 (2012), 684–694.
  * Sun et al. (2022) Wei Sun, Xiongkuo Min, Wei Lu, and Guangtao Zhai. 2022.  A deep learning based no-reference quality assessment model for ugc videos. In _Proceedings of the ACM International Conference on Multimedia_. 856–865.
  * Tu et al. (2021) Zhengzhong Tu, Yilin Wang, Neil Birkbeck, Balu Adsumilli, and Alan C Bovik. 2021.  UGC-VQA: Benchmarking blind video quality assessment for user generated content.  _IEEE Transactions on Image Processing_ 30 (2021), 4449–4464.
  * Wang et al. (2019) Yilin Wang, Sasi Inguva, and Balu Adsumilli. 2019.  YouTube UGC dataset for video compression research. In _IEEE International Workshop on Multimedia Signal Processing (MMSP)_. IEEE, 1–5.
  * Wang et al. (2021) Yilin Wang, Junjie Ke, Hossein Talebi, Joong Gon Yim, Neil Birkbeck, Balu Adsumilli, Peyman Milanfar, and Feng Yang. 2021.  Rich features for perceptual quality assessment of UGC videos. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 13435–13444.
  * Wu et al. (2022) Haoning Wu, Chaofeng Chen, Jingwen Hou, Liang Liao, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. 2022.  Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling. In _Proceedings of the European Conference on Computer Vision_. Springer, 538–554.
  * Wu et al. (2023a) Haoning Wu, Chaofeng Chen, Liang Liao, Jingwen Hou, Wenxiu Sun, Qiong Yan, Jinwei Gu, and Weisi Lin. 2023a.  Neighbourhood representative sampling for efficient end-to-end video quality assessment.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2023).
  * Wu et al. (2023b) Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. 2023b.  Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 20144–20154.
  * Xing et al. (2022) Fengchuang Xing, Yuan-Gen Wang, Hanpin Wang, Jiefeng He, and Jinchun Yuan. 2022.  DVL2021: An ultra high definition video dataset for perceptual quality study.  _Journal of Visual Communication and Image Representation_ 82 (2022), 103374.
  * Xu et al. (2021) Jiahua Xu, Jing Li, Xingguang Zhou, Wei Zhou, Baichao Wang, and Zhibo Chen. 2021.  Perceptual quality assessment of internet videos. In _Proceedings of the ACM International Conference on Multimedia_. 1248–1257.
  * Ying et al. (2021) Zhenqiang Ying, Maniratnam Mandal, Deepti Ghadiyaram, and Alan Bovik. 2021.  Patch-VQ:’Patching Up’the video quality problem. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 14019–14029.
  * You and Korhonen (2019) Junyong You and Jari Korhonen. 2019.  Deep neural networks for no-reference video quality assessment. In _IEEE International Conference on Image Processing_. IEEE, 2349–2353.
  * Zhu et al. (2021) Wenhan Zhu, Guangtao Zhai, Xiongkuo Min, Xiaokang Yang, and Xiao-Ping Zhang. 2021.  Perceptual quality assessment for recognizing true and pseudo 4K content. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2190–2194.


# Quantum non-Markovianity, quantum coherence and extractable work in a
general quantum process

Amin Mohammadi  Afshin Shafiee<EMAIL_ADDRESS>Research Group on
Foundations of Quantum Theory and Information, Department of Chemistry, Sharif
University of Technology P.O.Box 11365-9516, Tehran, Iran

###### Abstract

A key concept in quantum thermodynamics is extractable work, which specifies
the maximum amount of work that can be extracted from a quantum system.
Different quantities are used to measure extractable work, the most prevalent
of which are ergotropy and the difference between the non-equilibrium and
equilibrium quantum free energy. Using the former, we investigate the
evolution of extractable work when an open quantum system goes through a
general quantum process described by a completely-positive and trace-
preserving dynamical map. We derive a fundamental equation of thermodynamics
for such processes as a relation between the distinct sorts of energy change
in such a way the first and second laws of thermodynamics are combined. We
then identify the contributions made by the reversible and irreversible
processes in this equation and demonstrate that they are respectively
responsible for the heat flow and change in the extractable work during the
process. Furthermore, we discuss the potential benefit of this assignment in
favor of a clear explanation of the impact of quantum effects on the evolution
of extractable work. Specifically, we establish this by directly connecting
the extractable work with standard quantifiers of quantum non-Markovianity and
quantum coherence during the process. We illustrate these results with two
examples.

††preprint: AIP/123-QED

##  I Introduction

Thermodynamics can be thought of as an extension of classical mechanics to
contain concepts such as temperature and heat in the study of properties of
macroscopic systems lebon2008understanding. Depending on the underlying
mechanics, one may wonder how the transition from classical to quantum
mechanics will affect the laws of thermodynamics. It becomes even more
intriguing if the system under study exhibits quantum effects that have no
classical counterpart. Followed by advances in laboratory and quantum
technological implementations, much attention in recent years has been devoted
to understanding thermodynamics of small quantum systems gemmer2009quantum;
vinjanampathy2016quantum; binder2018thermodynamics; goold2016role;
deffner2019quantum; horodecki2013fundamental; brandao2015second;
cwiklinski2015limitations; linden2010small; scully2003extracting;
masanes2017general; taranto2023landauer; kammerlander2016coherence;
perarnau2015extractable; taranto2023landauer; alipour2022entropy;
soltanmanesh2020can; soltanmanesh2019clausius; myers2022quantum;
bhattacharjee2021quantum; quan2007quantum; dann2020quantum. One of the most
fundamental challenges in this new field of research is finding the general
definitions of work and heat for arbitrary quantum systems. Although standard
formulations of work and heat exist with limitations to weak coupling
conditions and Markovian dynamics alicki1979quantum, the situation in general
quantum evolutions is rather unclear. The difficulty arises from the fact that
both work and heat are path-dependent quantities that can not be described by
Hermitian operators talkner2007fluctuation, a standard way to define physical
observables in quantum mechanics. In the absence of a single, universal
concept, several definitions based on various criteria emerge. For example, a
variety of approaches from use of quantifiers like ergotropy
allahverdyan2004maximal and free energy difference skrzypczyk2014work;
muller2018correlating; garcia2020fluctuations for work extraction from non-
passivity of quantum states to statistical treatments attempting to measure
quantum work during a process by constructing appropriate probability
distributions esposito2009nonequilibrium; seifert2012stochastic;
campisi2011colloquium have been proposed.

In this regard, in a remarkable study Binder _et al._ binder2015quantum
established in an operational sense a relation in resemblance to the first law
of thermodynamics between work done, extractable work, heat and change of
internal energy for a quantum system evolving in a general quantum process
presented by a completely-positive and trace-preserving (CPTP) map. The
constituents of this relation are the difference in ergotropy between the
initial and final states of the map, adiabatic work and introduced operational
heat that their sum is equal to the change in internal energy of the open
system due to general quantum process.

In the present study, we establish similarly an energy balance relation for
general quantum processes in which the free energy work is used to measure the
extractable work from the system during the process. This relation replaces
the abovementioned operational heat with the change in entropy of the system
due to the fact that in contrast to ergotropy, the free energy work exploits
the resource from open quantum systems. We can think of this relation as the
fundamental thermodynamic equation combining the first and second laws of
thermodynamics for general quantum processes. We then show the evolution of
heat and the extractable work are determined respectively by the reversible
and irreversible change in the entropy of the system. The fact that
irreversibility governs the evolution of the extractable work underlies the
main results of this paper. We demonstrate how this fact can be used to link
directly non-Markovianity with extractable work and its rate of change during
evolution. In addition, We discuss how such a utility is also provided for
exploring the connection between the amount of extractable work and the time
evolution of quantum coherence, a well-known resource for many quantum
processes. In the end, as an illustration of our results, we investigate the
effects of quantum non-Markovianity and quantum coherence on the evolution of
the charging power for two types of quantum batteries: a two-qubit battery-
charger model and a two-qubit battery charged by quantum coherence due to one
photon mode mediation.

##  II Preliminaries

The first Law of Thermodynamics remarks the contribution of work and heat to
internal energy change in a thermodynamic process. In quantum thermodynamics,
depending on whether one considers a system out of passive or thermal
equilibrium (completely passive) states as a resource from which work can be
extracted, ergotropy and free energy difference are the two most widely used
work extractable quantifiers. More precisely, ergotropy
allahverdyan2004maximal quantifies the maximum work that can be extracted from
a system by transforming it from a non-passive to a passive state in a cyclic
unitary manner

$W_{e}=tr[\rho\emph{H}-\pi\emph{H}].$ (1)

The non-passive state $\pi$ is represented as $\pi=\Sigma$
$r_{n}|\varepsilon_{n}><\varepsilon_{n}|$ provided that the state $\rho$ and
Hamiltonian _H_ are expressed in their spectral decomposition respectively in
a decreasing and increasing order i.e., $\rho=\Sigma r_{n}|r_{n}><r_{n}|$ and
$\emph{H}=\Sigma\varepsilon_{n}|\varepsilon_{n}><\varepsilon_{n}|$ with
$r_{n+1}\leq r_{n}$ and $\varepsilon_{n+1}\geq\varepsilon_{n}$ $\forall n$.
Cyclicity, here, refers to the fact that Hamiltonian is identical in the
initial and final points of evolution. The ergotropy has been measured
experimentally in the quantum heat engines with spin van2020single and single
atom von2019spin as working fluids and recently in quantum batteries modeled
by low dimensional metal complexes cruz2022quantum. An extension of ergotropy
for open quantum systems and non-unitary evolutions has been made by Binder
_et al._ binder2015quantum. They recovered the first law of thermodynamics for
finite quantum systems undergoing general quantum processes:

$\Delta{E}=\Delta{W_{e}}+<W>_{ad}+<Q>_{op}.$ (2)

Here, $\Delta{W_{e}}=W_{e}$($\emph{H}_{\tau},\rho_{\tau}$) -
$W_{e}$($\emph{H}_{0}$,$\rho_{0}$) is change in ergotropy of the system during
the process ($\emph{H}_{0}$,$\rho_{0}$)$\to$($\emph{H}_{\tau},\rho_{\tau}$) in
which $\emph{H}_{0}$ and $\emph{H}_{\tau}$ are initial and final Hamiltonians
and $\rho_{0}$ and $\rho_{\tau}$ are initial and final quantum states that
denote inputs and outputs of CPTP map $\Lambda_{\tau}$ i.e.
$\rho_{\tau}=\Lambda_{\tau}\rho_{0}$. Although $\Delta{W_{e}}$ bears much
resemblance to the change of thermodynamic state functions as it depends only
on the states and Hamiltonians in the initial and final of a process, here, it
enters a genuine out-of-equilibrium contribution in the first law relation.
This is important because any generalization of thermodynamic laws to quantum
regimes must have the ability to describe both equilibrium and non-equilibrium
situations.

Also, $<W>_{ad}$ measures the change in internal energy due to a non-cyclic
unitary transformation that change in Hamiltonian is assumed to be adiabatic
in a quantum sense, i.e. at each instant of evolution eigenstates of
Hamiltonian remain eigenstates

$<W>_{ad}=tr[\pi_{t}\emph{H}_{t}]-tr[\pi_{m}\emph{H}_{0}]$ (3)

where $\pi_{t}$ and $\pi_{m}$ are passive states with respect to final and
initial Hamiltonians.

Finally, to account for heat contribution in the first law formulation, the
last term in Eq. (3) was defined as operational heat $<Q>_{op}$:

$<Q>_{op}=tr[\pi_{m}\emph{H}_{0}]-tr[\pi_{0}\emph{H}_{0}]$ (4)

which is the energy change in the transformation between two different passive
states $\pi_{0}$ and $\pi_{m}$ of the initial Hamiltonian.

##  III Fundamental equation of quantum thermodynamics

Free energy is one of the most important quantities in thermodynamics because,
according to Clausius’s statement of second law, it informs us of the possible
state transitions of a system and the optimal work can be extracted from the
system as a result. In consequence of generally extending the validity of
second law (in Clausius form) to quantum regime vinjanampathy2016quantum, one
can consider another extractable work quantifier that is given by the free
energy difference between non-equilibrium and equilibrium states of the system
in contact with a thermal bath skrzypczyk2014work; muller2018correlating;
garcia2020fluctuations

${W_{f}}=\emph{F}(\rho)-\emph{F}(\tau_{\beta})$ (5)

where $\tau^{\beta}=exp(-\beta\emph{H})/Z$ is the thermal equilibrium or Gibbs
state with inverse temperature $\beta$ and partition function Z and
$\emph{F}(\rho)=tr[\emph{H}\rho]-\beta^{-1}\emph{S}(\rho)$ is free energy with
$S$ denoting the entropy and $\emph{F}(\tau_{\beta})$ is equilibrium free
energy of the system. There have been some protocols to extract this kind of
quantum work in both individual skrzypczyk2014work; muller2018correlating and
asymptotically many copies of quantum systems brandao2013resource. Considering
the general process
$(\emph{H}_{0},\rho_{0})\to(\emph{H}_{t},\rho_{\tau}=\Lambda_{\tau}\rho_{0})$,
we write the change in the free energy work in initial and final points of
evolution:

$\Delta{W_{f}}={W_{f}}_{t}-{W_{f}}_{0}$ (6)

from which we obtain the following relation for a general time-dependent
Hamiltonian:

$\Delta{E}=\Delta{W_{f}}+<W>_{ad}+\beta^{-1}(\Delta{S}_{(}\rho)-\Delta{S}(\pi^{\beta})).$
(7)

As Eq. (2) was described as the first law of thermodynamics for general
quantum processes, we interpret Eq. (7) as the fundamental thermodynamic
equation for such processes that adds to the power of thermodynamic
description with combining the first and second laws. The first and second
terms similar to Eq. (2) are extractable and adiabatic works with the
difference that the former is quantified by free energy difference instead of
ergotropy and in the latter the instantaneous Gibbs equilibrium state is
singled out among possible passive states of system at time t. The main
difference, however, is the third term that shows the change in energy which
alters von Neumann entropies of the system :

$\Delta{S}(\rho)=S(\rho_{t})-S(\rho_{0}).$ (8)
$\Delta{S}(\pi^{\beta})=S(\pi_{t}^{\beta})-S(\pi_{0}^{\beta}).$ (9)

We note that the above equations cannot define the heat in a general CPTP
evolution before assigning the reversible and irreversible contributions in
entropy change. To this end, we add and subtract the expression
$tr(\rho_{0}log(\pi^{\beta}_{0})+\rho_{t}log(\pi^{\beta}_{t}))$ to Eqs. (8)
and (9) through that we can write
$\Delta{S}(\rho)-\Delta{S}(\pi^{\beta})=\Delta{S}_{Ir}-\Delta{S}_{R}$ by the
following definitions:

$\Delta{S}_{Ir}=S(\rho_{0}||\pi^{\beta}_{0})-S(\rho_{t}||\pi^{\beta}_{t})$
(10)

and

$\Delta{S}_{R}=tr[(\rho_{t}-\pi^{\beta}_{t})log(\pi^{\beta}_{t})]-tr[(\rho_{0}-\pi^{\beta}_{0})log(\pi^{\beta}_{0})].$
(11)

We argue Eq. (10), expressed in terms of change in relative entropy between
state $\rho$ and its correspondence equilibrium state at initial and final
times of transformation, can identify the irreversible change in the entropy
based on the intuition that states $\rho$ suffers from relaxation to
$\pi^{\beta}$ by the irreversible loss of its coherence. Then the relative
entropy measuring the closeness of two states nielsen2010quantum can also be
considered as a measure of irreversibility. In the case that the Hamiltonian
of the system remains unchanged during the evolution, Eq. (10) turns into
$\Delta{S}_{Ir}=S(\rho_{0}||\pi^{\beta}_{0})-S(\rho_{t}||\pi^{\beta}_{t})$
that is a famous expression for entropy production indicating the irreversible
term in the entropy balance relation. Because of the relative entropy’s
contractivity property when subject to the action of CPTP maps vedral2002role,
the irreversible entropy change defined by Eq. (10) is a positive amount. It
may, however, be negative for time-independent Hamiltonians, as was the case
with CPTP non-Markovian dynamics with a non-stationary thermal equilibrium
state marcantoni2017entropy. On the other hand, we introduce Eq. (11) which
includes the change in the state of the system between non-equilibrium and
equilibrium situations as the reversible change in entropy. It may also be
viewed as an extension of the quantum Hatano-Sasa inequality hatano2001steady;
vinjanampathy2015entropy, a well-known formulation of the second law for CPTP
maps which is expressed by $\Delta{S}\geq
tr[(\rho_{t}-\rho_{0})log(\pi^{\beta})]$, to CPTP maps with time-dependent
Hamiltonians.

Putting these together in Eq. (7), we define heat flow from the system
corresponding to the reversible change in entropy

$<Q>=-\beta^{-1}\Delta{S}_{R}=\Delta{E}-<W>_{ad}$ (12)

and moreover, find the time evolution of extractable work for the system is
governed by irreversible entropy change, that is:

$\Delta{W_{f}}=-\beta^{-1}\Delta{S}_{Ir}$ (13)

##  IV Effective Quantum Parameters on Extractable work of open quantum
systems

We begin this section by noting the rate at which the extractable work changes
can be defined concerning Eq. (13) as:

$P(t)=\lim_{\Delta{t}\to
0}\frac{\Delta{W_{f}}}{\Delta{t}}=-\beta^{-1}\frac{d{S}_{Ir}}{dt}.$ (14)

$P(t)$ is commonly referred to as charging power in the study of quantum
batteries, which employ $P(t)$ to investigate how quickly a battery may be
charged or discharged garcia2020fluctuations. We then go over how Eqs. (13)
and (14) can be used to highlight the thermodynamic importance of particular
quantum properties such as non-Markovianity and coherence as valuable
resources on account of the enhancement of extracted work.

###  IV.1 Quantum non-Markovianity

Irreversibility is a fundamental concept in the theory of open quantum systems
breuer2002theory. The interaction between the system and the bath causes the
system to continuously lose its quantum properties, which is typically
understood as an irreversible flow of information from the system to the
environment. However, in some situations, for instance, when there is a strong
system-bath interaction, the information that the system loses to the
environment can be recovered. In contrast to Markovian systems, which lose
information irreversibly, these systems that revive information are referred
to as non-Markovian open quantum systems breuer2002theory. Since quantum
properties are well known to be valuable work resources, it is anticipated
that reviving these properties and resulting in non-Markovianity will have an
advantage in thermodynamic studies. For example, some studies report positive
effects of non-Markovianity on quantum Landauer erasure work cost revivals
bylicka2016thermodynamic, the performance of various kinds of quantum heat
engines zhang2014quantum; abiuso2019non; camati2020employing;
ptaszynski2022non, and the charging of quantum batteries kamin2020non;
ghosh2021fast; tabesh2020environment.

According to thermodynamics, irreversibility is considered as a contribution
to the entropy balance relation of the system lebon2008understanding;
callen1991thermodynamics. As we can see from Eq. (13), this contribution
determines the change in extractable work from the system. The extractable
work calculated from this relation is monotonically decreasing throughout a
CTPT evolution as a result of an increase in irreversibility, which is
indicated by a positive value for $\Delta{S}_{Ir}$. However, the the rate of
change i.e. charging power can become in general negative in some time
intervals.

In this respect, we note a precise connection with non-Markovianity is
deducible when we take charging power into account and apply the result of
Chen _et al._ chen2017thermodynamic to measure the amount of non-Markovianity
during evolution. In accordance with the foregoing intuition of a highly close
relation between non-Markovianity and lack of irreversibly, Chen _et al._ show
that under the hypothesis of static environment, the non-Markovianity of open
quantum systems can be defined as:

$\emph{I}=-\frac{\partial S_{Ir}}{\partial t}.$ (15)

The static hypothesis is satisfied if the environment state doesn’t change
during evolution. Now the charging power can be simply written as:

$P(t)=\beta^{-1}\emph{I}.$ (16)

This evidently shows more quantum properties a system revives, as indicated by
a higher positive amount of non-Markovianity measure I, the more charging
power it recovers. The non-Markovianity advantage is especially apparent when
the system is thermalized using baths at a higher temperature.

###  IV.2 Quantum coherence

Quantum coherence, both fundamentally and practically, is one of the most
important quantum features as it distinguishes the quantum world from the
classical one and is an essential resource in many quantum processes
streltsov2017colloquium. When defined with relative to energy basis, quantum
coherence is a prominent resource for carrying out certain thermodynamic
tasks. Many studies have been conducted in recent years to look into the role
that coherence play in the work extraction protocols lostaglio2015description;
lostaglio2015quantum; korzekwa2016extraction; lostaglio2019introductory,
operation of quantum thermal machines rahav2012heat; scully2011quantum;
mitchison2015coherence; uzdin2015equivalence, the performance of quantum
batteriesgarcia2020fluctuations; kamin2020entanglement; mayo2022collective;
shi2022entanglement, etc. Here, we show that Eq. (13) is capable of
efficiently interpreting the impact of a system’s quantum coherence on the
work that can be extracted from it. This is provided by a direct connection
between the change in free energy work and relative entropy of coherence, a
widely used measure of quantum coherence baumgratz2014quantifying defined as:

$C_{r}(\rho)=S(\Delta\rho)-S(\rho)$ (17)

where $\Delta\rho$ denotes the state with the same diagonal elements as $\rho$
but zero off-diagonal ones. The relation providing the desired utility is
easily derived from Eq. (13) :

$\Delta{W_{f}}=\Delta{E}+\beta^{-1}(\Delta{C_{r}}(\rho)-\Delta{S}(\Delta\rho)-\log(\frac{Z_{t}}{Z_{0}}))$
(18)

where the second term involves the difference between the relative entropy of
coherence, the entropy of dephased state and the logarithm of partition
function at the final and initial times of evolution that the latter arises
from time variation of Hamiltonian and vanishes with time-independent
Hamiltonians. It is clear from this relation that the more coherent a quantum
system is at a given time $t$ compared to its initial state, the more work can
be extracted from it. For instance, this is the case when the system
coherently exchanges energy with another quantum system or is coherently
derived by an external field. In addition and in line with the work of Shi _et
al._ shi2022entanglement, one can recognize the Eq. (18) as a sum of two
parts: a coherent part includes the change in the relative entropy of
coherence $\Delta{W_{f}^{c}}=\beta^{-1}\Delta{C_{r}}(\rho)$ and an incoherent
part includes other terms
$\Delta{W_{f}^{i}}=\Delta{E}-\beta^{-1}(\Delta{S}(\Delta\rho)-\log(\frac{Z_{t}}{Z_{0}}))$.
The charging power can also be decomposed into coherent and incoherent parts,
$P(t)=P^{c}(t)+P^{i}(t)$ with

$P^{c}(t)=\beta^{-1}(\frac{\partial C_{r}(\rho(t))}{\partial t})$ (19)

and

$P^{i}(t)=\frac{\partial E}{\partial t}-\frac{\partial S(\rho(t))}{\partial
t}-\frac{\frac{\partial Z_{t}}{\partial t}}{Z_{t}}.$ (20)

Eq. (19) shows the revival of quantum coherence during the evolution,
associated with $P^{c}(t)>0$, enhances the charging power and can have a
dominant effect at higher temperatures.

##  V Examples

In this section, we discuss two examples to illustrate the utility of the
above results. Examples are considered as different types of quantum
batteries, an interesting case study in the field of quantum thermodynamics to
investigate the potential efficiency that may be acquired on energy storage of
a battery by quantum mechanical phenomena. We employ Eqs. (16) and (19) to
demonstrate the thermodynamics advantage of quantum non-Markovianity in
example 1 and quantum coherence in example 2 on the charging power evolution
of the related quantum batteries.

###  V.1 Example 1: Two-qubit battery-charger model

In the first example, we consider a two-qubit system as a battery-charger
model in which qubit 1 as a quantum battery is directly charged through
interaction with qubit 2 as a charger while both qubits are simultaneously
exposed to a common bosonic environment. The Hamiltonian of the system is
given by $\emph{H}=\emph{H}_{0}+f(t)\emph{H}_{I}$, with

$\emph{H}_{0}=\sum_{i=1}^{2}\omega_{0}\sigma_{i}^{+}\sigma_{i}^{-}+\sum_{k}\omega_{k}a_{k}^{\dagger}a_{k}$
(21)

and

$\emph{H}_{I}=(\alpha_{1}\sigma_{1}^{+}+\alpha_{2}\sigma_{2}^{+})\sum_{k}(g_{k}a_{k})+h.c$
(22)

where $\emph{H}_{0}$ includes free Hamiltonians of two qubits and environment
with $\omega_{0}$ and $\sigma_{i}^{\pm}$ being respectively for transition
frequency and Pauli raising and lowering operators for i-th qubits. Also,
$\omega_{k}$, $a_{k}^{\dagger}$ and $a_{k}$ are frequency, creation operator
and annihilation operator of k-th mode of environment. $\emph{H}_{I}$ is the
interaction Hamiltonian of two qubits with a common environment in which
$g_{k}\alpha_{i}$ denotes the coupling constant between the i-th qubit and
k-th mode of environment. $f(t)$ is a dimensionless function that controls the
coupling and decoupling of the charger to the battery with accepting
respectively the value 1 in charging time [0,$\tau$] and 0 elsewhere. This
model was solved analytically in one maniscalco2008protecting; francica2009off
and two mazzola2009sudden excitation spaces and was first studied as a quantum
battery-charger model in tabesh2020environment. For a bath initially in a
vacuum state, we consider the general the initial state of the whole system

$|\psi(0)>=(c_{01}|1>_{1}|0>_{2}+c_{02}|0>_{1}|1>_{2})\otimes|0>_{k}$ (23)

that its time evolution can be expressed in terms of one excitation basis
states as the following:

$|\psi(t)>=c_{1}(t)|1>_{1}|0>_{2}|0>_{E}+c_{2}(t)|0>_{1}|1>_{2}|0>_{E}\\\
+\sum_{k}c_{k}(t)|0>_{1}|0>_{2}|1>_{E}.$ (24)

The reduced density matrix for the quantum battery can be thereby written as :

$\rho_{1}=|c_{1}(t)|^{2}|1><1|_{1}+(1-|c_{1}(t)|^{2})|0><0|_{1}.$ (25)

We assume in the following that the bath has a Lorentzian spectral density

$J(\omega)=\frac{W^{2}}{\pi}\frac{\lambda}{(\omega-\omega_{0})^{2}+\lambda^{2}}$
(26)

where $\lambda$ is the width of the Lorentzian spectrum that is connected to
the environment correlation time $\frac{1}{\lambda}$ and W is an effective
coupling strength related in the limit of $\lambda\to 0$ to Rabi frequency of
two qubits $\Omega$. To distinguish the non-Markovian from Markovian dynamics
we employ dimensionless parameter $\emph{R}=\frac{\Omega}{\lambda}$ that shows
the Markovian and non-Markovian effects can be dominant respectively when
$\emph{R}\ll 1$ and $\emph{R}\gg 1$ tabesh2020environment.

We here suppose the initial state of the system as $|\psi(0)>=|0>_{1}|1>_{2}$
implying that the battery is empty and the charger has its maximum amount of
energy before the charging process. The details of the solution and analytical
expression for amplitude $|c_{1}(t)|^{2}$ can be found in
maniscalco2008protecting, whereby the state of the battery is easily
determined by Eq. (25). The Gibbs state is reached by
$|c_{1}(t)|^{2}=\frac{e^{-\beta\omega_{0}}}{Z}$ and
$1-|c_{1}(t)|^{2}=\frac{1}{Z}$

We then apply Eq. (16) to calculate the charging power of the battery during
the time evolution. The results have been plotted in Fig. 1 for two different
values of R taken as 0.3 and 30 to enable us to better compare the effects of
Markovian and non-Markovian dynamics on the calculated results. With
$\emph{R}=0.3$, as we would expect the dynamics to become mostly Markovian,
Fig. 1(a) shows that the charging power decreases monotonically over the given
time interval as a result of the monotonically decreasing non-Markovian
characteristic of the dynamics (measured by $I$ in Eq. (15)). On the other
hand, for $\emph{R}=30$, Fig. 1(b) shows revivals in the charging power of the
battery in consequence of the increase in non-Markovianity ($I$) demonstrating
that the system interpolates between Markovian and non-Markovian regimes.
These results exhibit obviously that the charging power is enhanced by virtue
of non-Markovian effects on the dynamics of the quantum battery a fact that is
well expressed in Eq. (16) by establishing a direct connection between two
desired quantities $P$ and $I$.

Figure 1: Time evolution of charging power $P(t)$ of quantum battery in
example 1 . (a) and (b) depict $\emph{R}=0.3$ and $\emph{R}=30$ respectively.
The model parameters are set as $\omega_{0}=1$, $\beta=0.1$, $\lambda=1$ and
$\alpha=\frac{1}{\sqrt{2}}$.

###  V.2 Example 2: Two-qubit battery–one photon charger model

To explain the role of quantum coherence, we consider in example 2 a two-qubit
battery model in which the battery has been charged by coherence mediated by
one photon mode. We investigate this model in two cases: in the first, the
model is assumed as a close system in which the battery and the photon evolve
unitarily. In the second, we regard the detrimental effects of spontaneous
emission brought on by the interaction of the battery with a Markovian
environment. The Hamiltonian of the system is given by
$\emph{H}=\emph{H}_{0}+f(t)\emph{H}_{I}$, with

$\emph{H}_{0}=\sum_{i=1}^{2}\omega_{0}\sigma_{i}^{+}\sigma_{i}^{-}+\omega_{p}a_{p}^{\dagger}a_{p}$
(27)

and

$\emph{H}_{I}=g(\sigma_{1}^{+}+\sigma_{2}^{+})(a_{p})+h.c$ (28)

where p denotes the photon mode and all operators and parameters are defined
before in Eqs. (21) and (22). The dynamic of the battery is described after
tracing out the photon degree of freedom by the Schrödinger equation

$\frac{\partial\rho_{b}}{\partial t}=i\ tr_{p}([\rho,\emph{H}])$ (29)

in the first case, and by a master equation ficek2002entangled;
dzsotjan2010quantum:

$\frac{\partial\rho_{b}}{\partial t}=i\
tr([\rho,\emph{H}^{{}^{\prime}}])+\sum_{i=1}^{2}\frac{\gamma_{ij}}{2}(2\sigma_{i}^{-}\rho_{b}\
\sigma_{j}^{+}-\sigma_{i}^{+}\sigma_{j}^{-}\rho_{b}-\rho_{b}\sigma_{i}^{+}\sigma_{j}^{-})$
(30)

in the second case. Here, $\emph{H}^{{}^{\prime}}$ is lamb shift Hamiltonian
describing the effective interaction between two qubits:

$\emph{H}^{{}^{\prime}}=\sum_{i=1}^{2}\omega_{0}\sigma_{i}^{+}\sigma_{i}^{-}+g_{12}(\sigma_{1}^{+}\sigma_{2}^{-}+\sigma_{2}^{+}\sigma_{1}^{-})$
(31)

with $g_{12}$ being the coupling constant. Also, $\gamma_{ii}$ is the
spontaneous emission rate for i-th qubit while $\gamma_{ij}$ represents the
contribution from two-qubit interactions. Here, we assume
$\gamma_{11}=\gamma_{22}=\gamma$ and use $g_{12}=g^{2}/\Delta$ and
$\gamma_{12}\approx 0$ in accordance with realistic situations in cavity
quantum electrodynamics (CQED) experiments gonzalez2011entanglement where
$\Delta$ denotes the detuning between frequencies of qubits and photonic
cavity mode. The results for two cases are plotted in Fig. 2. In Fig. 2(a),
one can see the battery in the first case exhibits a cycle of gain and loss of
quantum coherence continuing indefinitely due to the unitarity of evolution.
In the second case, however, even though the battery reaches its maximum
amount of quantum coherence more quickly than in the first case (thanks to the
coherent part of Eq. (30) in the first term), quantum coherence of battery
vanishes as a result of decoherence process induced by interaction with
Markovian bath.

The same behavior can be observed in Fig. 2(b) for the coherent part of the
charging power of quantum battery as it indicates the oscillations with equal
amplitude in case 1, while in case 2 the decoherence induced by the Markovian
environment leads to null coherent charging power. Finally, in Fig. 2(c), it
is shown that despite having initially markedly different values, the coherent
and total charging powers nearly reach zero at the same time. This supports
the main finding of Shi _et al._ shi2022entanglement that coherence must be
attained during charging in order to extract work from a battery in its
initial non-coherent state. The values of total charging power reveal that,
notwithstanding early revivals in quantum coherence, the system loses its
available work monotonically through interaction with the environment.

Figure 2: Time evolution of (a) relative entropy of coherence $C_{r}(\rho)$(b)
coherent charging power $P^{c}(t)$ of quantum battery in example 2. (1) and
(2) denote to cases 1 and 2 in text. Model parameters are taken as
$g=1,\omega_{0}=1,\omega_{p}=2$ and $\beta=0.1$. (c) depicts the coherent and
total charging power of the quantum battery in case 2. Model parameters are
the same as (b) and (c) except for $\beta=1$

##  VI Conclusions

Examining how a system’s quantum characteristics affect its thermodynamic
performance is crucial to understanding quantum thermodynamics. This is
particularly important in the context of work extraction to design more
efficient quantum thermodynamic devices like quantum heat engines and quantum
batteries. The main objective of this study is to make one more contribution
to addressing this problem by providing a formalism capable of highlighting
the effective quantum parameters on the dynamics of extractable work of an
open quantum system. Dynamics has been considered as a general quantum process
which means the only limitation to dynamics is that it must be physically
legitimate. Complete-positive and trace-preserving maps ensure this condition.
We have quantified the extractable work of the evolved system with the
differences between its instantaneous non-equilibrium and equilibrium free
energies that measure the maximum work that can be extracted from a quantum
system when it is in contact with a thermal bath. We have established a
fundamental thermodynamic equation that relates the change in internal energy
of the quantum system to the change in its extractable work and entropy. We
have demonstrated the salient fact about this equation appears when we write
the entropy change of the system as a sum of two contributions stemming from
reversible and irreversible processes. They respectively specify the heat flow
and the change in extractable work of the system during the process. We have
demonstrated how this correspondence between irreversibility and extractable
work can be used to explore the genuine quantum effects on the dynamics of the
latter. In particular, this is accomplished by establishing a direct
correspondence between extractable work and standard measures of quantum non-
Markovianity and quantum coherence. Our results, therefore, show that quantum
non-Markovianity and quantum coherence are resourceful for the thermodynamic
task of extracting work from an open quantum system evolving under a general
quantum process and open up the possibility of systematically studying the
efficiency of these quantum phenomena for technological purposes. We have
illustrated our results by taking their advantage to show that dynamics with
non-Markovian effects can improve the thermodynamic performance of an open
quantum battery by increasing its charging power during the evolution. Both
the battery and charger in this example are qubits. We have also demonstrated
the same benefit is realized when we look at the role of quantum coherence
induced by an optical charger in the charging power of a two-qubit battery.

###### Acknowledgements.

We thank Dr. Ali Soltanmanesh for his constructive comments on the
presentation of the paper.

## AUTHOR DECLARATIONS

### Conflict of Interest

The authors have no conflicts to disclose.

## Author Contributions

Amin Mohammadi proposed the idea and did the calculations, Afshin Shafiee
contributed to the development and completion of the idea, analyzing the
results and discussions. Amin Mohammadi and Afshin Shafiee participated in
writing the manuscript.

## Data Availability Statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## REFERENCES