# Subjective and Objective Quality Assessment for in-the-Wild Computer
Graphics Images

Zicheng Zhang<EMAIL_ADDRESS>,  Wei Sun<EMAIL_ADDRESS>,  Yingjie
Zhou<EMAIL_ADDRESS>,  Jun Jia<EMAIL_ADDRESS>,  Zhichao Zhang
<EMAIL_ADDRESS>Shanghai Jiao Tong UniversityChina ,  Jing Liu
<EMAIL_ADDRESS>Tianjin UniversityChina ,  Xiongkuo Min
<EMAIL_ADDRESS>and  Guangtao Zhai<EMAIL_ADDRESS>Shanghai
Jiao Tong UniversityChina

###### Abstract.

Computer graphics images (CGIs) are artificially generated by means of
computer programs and are widely perceived under various scenarios, such as
games, streaming media, etc. In practice, the quality of CGIs consistently
suffers from poor rendering during production, inevitable compression
artifacts during the transmission of multimedia applications, and low
aesthetic quality resulting from poor composition and design. However, few
works have been dedicated to dealing with the challenge of computer graphics
image quality assessment (CGIQA). Most image quality assessment (IQA) metrics
are developed for natural scene images (NSIs) and validated on databases
consisting of NSIs with synthetic distortions, which are not suitable for in-
the-wild CGIs. To bridge the gap between evaluating the quality of NSIs and
CGIs, we construct a large-scale in-the-wild CGIQA database consisting of
6,000 CGIs (CGIQA-6k) and carry out the subjective experiment in a well-
controlled laboratory environment to obtain the accurate perceptual ratings of
the CGIs. Then, we propose an effective deep learning-based no-reference (NR)
IQA model by utilizing both distortion and aesthetic quality representation.
Experimental results show that the proposed method outperforms all other
state-of-the-art NR IQA methods on the constructed CGIQA-6k database and other
CGIQA-related databases. The database is released at
https://github.com/zzc-1998/CGIQA6K.

Computer graphics images, in-the-wild distortions, image quality assessment,
no-reference

††copyright: acmlicensed††journal: TOMM††journalyear: 2023††journalvolume:
1††journalnumber: 1††article: 1††publicationmonth: 1††price: 15.00††doi:
10.1145/3631357††ccs: Computing methodologies Computer vision

##  1\. Introduction

With the rapid development of computer graphics rendering techniques, billions
of computer graphics images (CGIs) have been generated and perceived in
various applications. Unlike natural scene images (NSIs) that are captured
with cameras in the real world, CGIs are rendered through a 2D or 3D model
described by means of a computer program (Min et al., 2017; Pharr et al.,
2016; Zhang et al., 2021b, c; Fan et al., 2022; Zhang et al., 2022), which has
been widely used in architecture, video games, simulators, movies, etc.
Different from the generative models based on deep neural networks such as GAN
(Goodfellow et al., 2014), Nerf (Mildenhall et al., 2020), etc., computer
graphics directly synthesize images using mathematical and computational
techniques without learning from the specific training data. According to the
rendering techniques and application scenarios, CGIs can be divided into
photorealistic images and non-photorealistic images. Photorealistic images are
generated to achieve photorealism by modeling the real world (Greenberg et
al., 1997), while non-photorealistic images promote the prosperity of digital
art by enabling a wide variety of expressive styles. Regardless of the
rendering techniques and purposes, both photorealistic and non-photorealistic
images inevitably suffer from various distortions, such as texture loss caused
by limited computation resources, poor visibility caused by wrong exposure
settings, and blur caused by low rendering accuracy, etc. Additionally, a
large part of CGIs are transmitted through the network services like cloud
gaming and live broadcasting (Laghari et al., 2019). In most practical cases,
CGIs are constantly influenced by compression distortion, which damages the
user’s Quality of Experience (QoE) (Chen et al., 2013). Moreover, previous
studies (Hosu et al., 2017; Li et al., 2018; Murray et al., 2012) have shown
that the subjective aesthetic preferences of individuals can also influence
their quality ratings of images, particularly in relation to CGIs (Ling et
al., 2020).

In the last decade, many image quality assessment (IQA) models have been
proposed to tackle quality assessment issues. According to the involving
extent of reference images, image quality assessment can be categorized into
full-reference (FR), reduced-reference (RR), and no-reference (NR) methods
(Zhai and Min, 2020). A variety of IQA measures have been proposed (Mittal et
al., 2012a, b; Li et al., 2015; Saad et al., 2012; Min et al., 2018; Narvekar
and Karam, 2011; Min et al., 2017; Gu et al., 2014; Zhang et al., 2018; Ke et
al., 2021; Su et al., 2020; Wang et al., 2021; Sun et al., 2023; Peng et al.,
2021; Song et al., 2022; Jiang et al., 2022b; Hu et al., 2023; Zhang et al.,
2021a) and achieved huge success on IQA tasks for natural scene content.
However, several studies have proven that the state-of-the-art (SOTA) IQA
models designed for NSIs are not suitable for images with different statistics
distributions from natural scene content (Yang et al., 2015; Wang et al.,
2016; Ling et al., 2020; Yu et al., 2022). Compared with NSIs, CGIs often
contain more regular geometric shapes, simpler texture, and relatively less
content diversity. To verify the statistical difference between NSIs and CGIs,
we present 5 quality-related attributes distributions of over 10,073 in-the-
wild NSIs and 18,031 in-the-wild CGIs for comparison. The NSIs are sourced
from the in-the-wild KonIQ-10k IQA database (Hosu et al., 2020) and the CGIs
are collected through the LSCGB database (Bai et al., 2021). The quality-
related attributes include light, contrast, colorfulness, blur, and spatial
information (SI), the details description of which can be referred to in (Hosu
et al., 2017). Fig. 1 illustrates the quality-related distributions of the
NSIs and CGIs, from which we can see that the normalized probability
distributions of NSIs’ quality attributes tend to be more centered at zero
while the normalized probability distributions of CGIs’ quality attributes are
relatively irregular in skewness. Specifically, the CGIs contain relatively
lower illumination levels and lack colorfulness.

Thus, it is unreasonable to simply transfer IQA models based on NSIs scope to
computer graphics IQA (CGIQA). Moreover, the mainstream well-performing IQA
methods are mostly designed in the data-driven manner, which highly depends on
the quality of corresponding databases. To promote the development of natural
scene IQA (NSIQA), various NSI databases have been carried out (Sheikh et al.,
2006b; Ponomarenko et al., 2015; Larson and Chandler, 2010; Lin et al., 2019;
Jiang et al., 2022a). However, the progress of CGIQA database falls behind: 1)
A large part of the publicly available CGIQA-related databases are organized
in former times. The selected CGIs are sourced from limited types of media and
are generated by outdated rendering techniques, thus such CGIs may not be able
to cover the range of current CGIQA tasks. 2) The existing CGIQA-related
databases are relatively small in scale, which is difficult for supporting the
designing and training of algorithms based on the deep neural networks, which
have been the dominant methods for the IQA tasks. 3) Many previous works
mainly focus on full-reference (FR) methods and the distortions are manually
introduced to the reference images. However, the pristine reference CGIs are
usually not available in practical situations and the distortions are complex
and unpredictable, which therefore increases the need for evaluating the
quality of in-the-wild CGIs in the NR manner.

Therefore, to address the above challenges, we first establish a large-scale
CGIQA database containing 6,000 CGIs. In order to construct the benchmark for
the CGIQA task, we select several SOTA NR IQA models including handcrafted-
based and deep learning-based methods for comparison. What’s more, we propose
a deep learning-based NR IQA model through both distortion and aesthetic
aspects to help promote the performance on the proposed CGIQA database. The
experimental results show that the proposed method is more effective for
predicting the quality scores of CGIs. The major contributions of our work are
summarized as follows:

  * •

To the best of our knowledge, we construct the largest in-the-wild CGIQA
database (CGIQA-6k), which consists of 6,000 CGIs from games and movies along
with the subjective quality scores. Our database covers a wider range of
resolutions (480p$\sim$4K) and contains more diverse content as well as
authentic distortions.

  * •

We especially design a deep learning-based model by jointly employing
distortion and aesthetic quality representation. The distortion stream
integrates multi-stage feature fusion and multi-stage channel attention
mechanisms, aimed at improving the learning of distortion representation.
Meanwhile, the aesthetic stream leverages prior knowledge acquired from the
aesthetic quality assessment (AQA) databases. The integration of these two
streams empowers the model with an enhanced understanding of the various
quality levels exhibited by CGIs.

  * •

The proposed method and some mainstream NR IQA models are validated on the
CGIQA-6k database along with other CGIQA-related databases to provide the
benchmark for CGIQA tasks. The ablation study, cross-database validation, and
statistical test are conducted for further analysis. In-depth discussions
about the experimental performance are given as well.

The remainder of this paper is organized as follows. Section 2 gives a brief
introduction of the related work. Section 3 describes the establishment of the
CGIQA-6k database and the subjective experiment. Section 4 presents the
details of the proposed IQA model. Section 5 gives the experimental
performance of the proposed method and other SOTA NR IQA models. The ablation
study and statistical test are also conducted and discussed. Section 6
concludes this paper.

(a) Sample NSI

(b) Sample CGI

(c) NSI distributions (d) CGI distributions

Figure 1. The normalized probability distributions of the quality-related
attributes for NSIs and CGIs. The distributions are obtained from 10,073 NSIs
in the KonIQ-10k IQA database (Hosu et al., 2020) and 18,031 CGIs in the LSCGB
database (Bai et al., 2021) respectively. The ’color’ indicates the
colorfulness of the images and the ’SI’ (spatial information) stands for the
content diversity of the images.

##  2\. Related Work

In the section, we briefly summarize the development of NR IQA models as well
as the construction of CGIQA-related databases.

###  2.1. No-reference Image Quality Assessment

Generally speaking, the NR IQA models can be categorized into handcrafted-
based methods (extracting features in handcrafted manners) and deep learning-
based methods (extracting features using deep neural networks), both of which
have been confirmed to be effective for common IQA tasks. BRISQUE (Mittal et
al., 2012a) employs natural scene statistics (NSS) in the spatial field for
quality analysis. NIQE (Mittal et al., 2012b) makes use of measurable
deviations from statistical regularities observed in natural images for
evaluation. BLIINDS2 (Saad et al., 2012) further uses NSS approach in the
discrete cosine transform (DCT) domain for quality assessment. CPBD (Narvekar
and Karam, 2011) calculates the blur levels by computing the cumulative
probability of blur detection. BIBLE (Li et al., 2015) focuses on blur-
specific distortions based on discrete orthogonal moments. NFERM (Gu et al.,
2014) studies the quality of images by using free energy principle. UCA (Min
et al., 2017) is a unified content-type adaptive blind IQA measure aiming at
compression distortion.

With the development of deep neural networks, many deep learning-based IQA
methods have been proposed. DBCNN (Zhang et al., 2018) constitutes two streams
of deep neuron networks and deals with both synthetic and authentic
distortions. HyperIQA (Su et al., 2020) uses a self-adaptive hyper network to
deal with the challenges of distortion diversity and content variation for IQA
issues. MUSIQ (Ke et al., 2021) employs a multi-scale image quality
transformer to represent image quality levels at different granularities. MGQA
(Wang et al., 2021) and StairIQA (Sun et al., 2023) both hierarchically
integrate the features extracted from intermediate layers to take advantage of
both low-level and high-level visual information. Most of the mentioned IQA
measures mentioned above have shown strong ability of predicting quality
levels for NSIs on the traditional IQA databases such as LIVE (Sheikh et al.,
2006b), TID2013 (Ponomarenko et al., 2015), CSIQ (Larson and Chandler, 2010),
Kadid10K (Lin et al., 2019), etc.

###  2.2. No-reference Aesthetic Quality Assessment

Earlier methods in AQA use handcrafted feature extraction. For instance, they
utilize features like color arrangement, explosion, sharpness, visual
saliency, composition, and focusing, as mentioned by (Datta et al., 2006), (Ke
et al., 2006), (Nishiyama et al., 2011), (Sun et al., 2009), and (Zhang et
al., 2014).

Nowadays, the trend is to employ deep learning models for aesthetic
evaluation. Studies like (Lu et al., 2015), (Talebi and Milanfar, 2018), and
(Yang et al., 2019) take smaller image patches from resized images as input
for further feature extraction. However, a challenge with this approach is
that randomly chosen patches, even when combined, don’t capture both the local
and the overall image details. Addressing this, (Ma et al., 2017) creates an
innovative model that uses an object-based attribute graph to better
understand the image layout. (Zhang et al., 2019) develops an attention
mechanism to zoom in on specific aesthetic-related areas. (Yang et al., 2019)
uses a network to prioritize local views based on their saliency. (Sheng et
al., 2018) devises an attention module that gauges the importance of each
image patch. A notable approach by (Hou et al., 2020) incorporates region-of-
interest (RoI) features from a generic object detector, leading to a more
effective blend of overall and localized features. Another study (Cui et al.,
2018) posits that a single label might not sufficiently assess the aesthetic
quality of an image, and hence suggests using a range of quality levels. (Shu
et al., 2021) presents a method that uses partial attribute annotations to
address the manual attribute annotation shortage. (Kong et al., 2016) focuses
on Knowledge Distillation, enabling the AQA model to discern and extract more
relevant aesthetic patterns.

Both IQA and AQA aim to assess an image’s merits but differ in focus and
application. IQA concentrates on technical attributes like sharpness and noise
which is vital in fields requiring image clarity, like camera imaging, medical
imaging, and image compression/transmission systems. Feedback in IQA is
technical, suggesting adjustments in equipment settings or resolution. On the
other hand, AQA delves into an image’s artistic attributes like composition
and emotional impact, central in fields like photography or art. Feedback in
AQA is more abstract, recommending changes in color scheme or subject matter.
While both evaluations have objective measures, they inherently contain
subjective elements, influenced by individual perceptions.

Table 1. The comparison of previous CGIQA-related databases and our database. The scale indicates the number of stimuli with mean opinion scores (MOSs). Database | Year | Scale | Source | Scope | Resolution Range | Public
---|---|---|---|---|---|---
CCT (Min et al., 2017) | 2017 | 528 | PC game images | Compression distortion |  720P$\sim$1080P | Yes
GamingVideoSET (Barman et al., 2018) | 2018 | 90 | PC game videos | Compression distortion |  480P$\sim$1080P | Yes
KUGVD (Barman et al., 2019) | 2019 | 90 | PC game videos | Downsampling & Bitrates control |  480P$\sim$1080P | Yes
CGVDS (Zadtootaghaj et al., 2020) | 2020 | 225 | PC game videos | Compression distortion |  480P$\sim$1080P | Yes
TGQA (Ling et al., 2020) | 2020 | 1091 | Mobile game images | Aesthetic evaluation | 1080P | Yes
TGV (Wen et al., 2021) | 2021 | 1293 | Mobile game videos | Stull & Bitrates control |  480P$\sim$1080P | No
LIVE-YT-Gaming (Yu et al., 2022) | 2021 | 600 | PC game videos | UGC distortions |  360P$\sim$1080P | Yes
NBU-CIQAD (Chen et al., 2021) | 2021 | 2700 | Cartoon images | Brightness, Saturation, Contrast | 1080P | Yes
CGIQA-6k(ours) | 2023 | 6000 | Games & Movies | in-the-wild distortion |  480P$\sim$4K | Yes
Figure 2. Sample images from the CGIQA-6k database. Top three rows: CGIs from
games. Bottom three rows: CGIs from movies.

###  2.3. CGIQA-Related Databases

The CCT database (Min et al., 2017) is a cross-content-type database including
natural scene images, screen content images, and computer graphics images,
which mainly focuses on compression distortion. The GamingVideoSET database
(Barman et al., 2018) contains 576 compressed gaming videos from 24 raw gaming
videos and conducts the subjective quality assessment experiment on 90
compressed gaming videos. Similarly, the KUGVD database (Barman et al., 2019)
obtains 144 distorted gaming videos from 6 reference gaming videos and
provides subjective ratings for 90 distorted gaming videos. The CGVDS database
(Zadtootaghaj et al., 2020) is carried out for cloud gaming applications and
provides 225 gaming videos along with quality scores. The TGQA database (Ling
et al., 2020) focuses on the aesthetic assessment of mobile game images and
presents the subjective study for 1,091 mobile game images on multi-
dimensional aesthetic factors. The TGV database (Wen et al., 2021) generates
1,293 mobile gaming sequences encoded with three codecs along with quality
scores. The LIVE-YT-Gaming database (Yu et al., 2022) pays more attention to
the user-generated content (UGC) gaming videos and consists of 600 authentic
UGC gaming videos with subjective ratings. The NBU-CIQAD database (Chen et
al., 2021) deals with the quality assessment of cartoon images and contains
2,600 distorted cartoon images with quality scores. A detailed comparison of
the CGIQA-related quality assessment database is given in Table 1. Through
further observation, it can be clearly found that our database is the largest
in scale and covers the most common resolutions.

##  3\. Subjective Quality Assessment for Computer Graphics Images

Few large-scale databases for CGIs have been developed in the quality
assessment field. To further facilitate research, we construct a large-scale
database called CGIQA-6k and conduct the corresponding subjective experiment
to derive the subjective quality scores.

###  3.1. Data Collection

The previously published LSCGB database for CGI and NSI detection (Bai et al.,
2021) contained 18,031 CGIs sourced from popular 3D games and 3D movies. To
cover more range of resolutions, we personally collect 40,000 CGIs through the
frames of stream videos from YouTube, Netflix, and Bilibili with different
playback settings. Furthermore, we obtain about 10,000 CGIs by recording
screenshots of local game demos. To reduce the effect of uncorrelated
distractions, we specifically eliminate the life bars, mini maps, and
subtitles. To ensure the diversity of content, we also manually remove the
CGIs with close content. Considering different viewpoints can provide
different experiences even in interactive games, we include both first-person
view (POV) CGIs and third-person view (TOV) CGIs. The POV CGIs allow the
viewers to perceive the scene through the character’s eyes, providing the most
immersive feelings (Denisova and Cairns, 2015). The TOV CGIs give a broader
view of the environment and enable the viewers to have a clear sight of the
main character (Voorhees et al., 2012). These mentioned two views make up the
majority views of mainstream video games on the market. As the CGIs focus more
on storylines rather than interactivity, we choose mostly portrait CGIs and
landscape CGIs for evaluation, which are commonly found in movies and game
cutscenes.

(a) Game (b) Movie (c) All

Figure 3. The distribution of the MOSs of the CGIQA-6k database. (a)
represents the MOS distribution for game CGIs while (b) indicates the MOS
distribution for movie CGIs. (c) exhibits the MOS distribution for all CGIs.
Additionally, the mean and standard deviation values of the MOSs are marked on
the top right as well.

(a) Confusing exposure, MOS = 1.14

(b) Texture loss, MOS = 1.50

(c) Low rendering accuracy, MOS = 1.29

(d) Compression distortion, MOS = 1.51

Figure 4. Examples of some typical distortions of CGIs. The content of (a) is
poorly visible due to the confusing exposure and the hair texture is quite
ambiguous in (b). The components of (c) exhibit relatively simple geometry
shapes with jagged edges caused by low rendering accuracy and the compression
distortion damages the perceptual quality of (d). The subjective quality
scores for (a), (b), (c), and (d) are provided and the MOS ranges from 0-5.

###  3.2. Data Sampling

After conducting the aforementioned data collection process, we successfully
acquired approximately 55,000 CGIs. To ensure the preservation of the quality
distribution, we follow the recommended approach presented in previous works
(Wu et al., 2023; Hosu et al., 2017; Ying et al., 2021; Zhang et al., 2023b).
Accordingly, we proceed to randomly sample a subset of CGIs, while ensuring
that the distributions of quality-related attributes (including light,
contrast, colorfulness, blur, and spatial information (Hosu et al., 2017))
within the subset align with those observed in the original collected CGIs.
Finally, we obtain a total of 6,000 CGIs, which consists of 3,000 game CGIs
and 3,000 movie CGIs. Samples of the selected CGIs are exhibited in Fig. 2,
where the top three rows are game CGIs and the bottom three rows are movie
CGIs. Generally speaking, most CGIs have been processed through compression or
transmission systems, we do not introduce new distortions in addition and pay
most attention to the in-the-wild distortions of CGIs. Examples of typical
distortions are shown in Fig. 4: In CGI (a), the content is poorly visible due
to inappropriate exposure. Such issues with exposure could stem from incorrect
settings in the lighting model or improper simulation of reflection and
scattering effects during rendering. In CGI (b), the hair texture appears
ambiguous. Rendering hair is challenging in computer graphics because hair
consists of countless tiny fibers, each interacting with light. Components in
CGI (c) present relatively simplistic geometric shapes but suffer from jagged
edges, a consequence of low rendering accuracy. Furthermore, the perceptual
quality of the CGI is compromised by compression distortion. In computer
graphics, CGIs are often compressed to save on storage space or transmission
bandwidth. However, if the compression rate is too high or if an unsuitable
compression algorithm is used, noticeable distortions can be introduced,
degrading the perceived quality of CGIs. Examples of different aesthetic
quality levels are shown in Fig. 5.

(a) Higher aesthetic quality and more ambiguous, MOS = 2.71

(b) Lower aesthetic quality and clearer content, M0S = 2.45

Figure 5. Examples of CGIs with higher and lower aesthetic quality. Although
the content of (a) appears more vague and ambiguous while the details of (b)
appear clearer, (a) elicits a higher quality experience than (b) from the
aesthetic perspective in terms of MOS.

###  3.3. Subjective Experiment Methodology

To obtain the perceptual quality scores of CGIs, the subjective experiment is
conducted with the recommendations of ITU-R BT.500-13 (BT, 2002). All CGIs are
shown in random order with an interface designed by Python Tkinter on an iMac
monitor which supports a resolution up to 4096 $\times$ 2304\. The screenshot
of the interface is illustrated in Fig. 6, from which we can see that the
interface allows users to freely browse the previous and next CGIs, and record
the quality scores through a scroll bar. A total of 60 graduate students (38
males and 22 females) are invited to participate in the subjective experiment.
The viewers are seated at a distance of around 1.5 times the screen height
(45cm) in a laboratory environment with normal indoor illumination. The
quality scale scores from 0 to 5, with a minimum interval of 0.1.

The participants are instructed to assess the CGIs based on both distortion
and aesthetic quality, providing ratings with an overall quality score. The
whole experiment is split into 20 sessions, each of which includes the
subjective quality evaluation for 300 CGIs. In this way, we limit the
experiment time for each session to less than half an hour. Every session is
attended by at least 20 viewers so that every CGI is evaluated by at least 20
subjects, which generates more than 20$\times$6,000=120,000 quality ratings in
total.

Figure 6. Screenshot of the interface for subjective experiments.

###  3.4. Subjective Data Processing

After the subjective experiment, we collect all the quality ratings from the
subjects. Let $r_{ij}$ denote the raw rating judged by the $i$-th subject on
the $j$-th image, the z-scores are obtained from the raw ratings as follows:

(1) $z_{ij}=\frac{r_{ij}-\mu_{i}}{\sigma_{i}},$

where $\mu_{i}=\frac{1}{N_{i}}\sum_{j=1}^{N_{i}}r_{ij}$,
$\sigma_{i}=\sqrt{\frac{1}{N_{i}-1}\sum_{j=1}^{N_{i}}\left(r_{ij}-\mu_{i}\right)}$,
and $N_{i}$ is the number of images judged by subject $i$. Then we remove the
ratings from unreliable subjects by employing the recommended subject
rejection procedure described in the ITU-R BT.500-13 (BT, 2002). The mean
opinion score (MOS) of the image $j$ is computed by averaging the rescaled
z-scores:

(2) $MOS_{j}=\frac{1}{M}\sum_{i=1}^{M}z_{ij}^{{}^{\prime}},$

where $MOS_{j}$ indicates the MOS for the $j$-th CGI, $M$ is the number of the
valid subjects, and $z_{ij}^{{}^{\prime}}$ are the rescaled z-scores. The
corresponding MOS distributions are exhibited in Fig. 3. It can be evidently
seen that the MOS distributions follow Gaussian-like distribution. With closer
inspections, the movie CGIs tend to gain higher quality scores than the game
CGIs, which is consistent with the practical situations that games are usually
rendered in real-time manners while movies often have more resources for
rendering.

Figure 7. The framework of the proposed method, where the proposed MFF and MCA
modules are arranged in a series manner and the aesthetic stream backbone’s
weights are fixed during the training.

##  4\. Proposed Method

In this paper, we propose an effective deep learning-based method to evaluate
the quality of CGIs, the framework of which is clearly shown in Fig. 7. The
framework consists of two streams: the distortion stream and the aesthetic
stream. The distortion stream utilizes the multi-stage feature fusion (MFF)
module and the multi-stage channel attention (MCA) module to enhance the
learning of broad-range distortion representation, which has been proven
effective in many IQA tasks (Golestaneh and Kitani, 2020; You and Yan, 2022;
Liu et al., 2022b; Hu et al., 2020; Lu et al., 2022; Wang et al., [n. d.]).
The aesthetic stream employs the vision backbones pre-trained on the mobile
game aesthetic quality assessment TGQA (Ling et al., 2020) and general
aesthetic quality assessment AVA (Murray et al., 2012) databases to extract
aesthetic quality-aware features. Finally, the distortion and aesthetic
features are fused and regressed into quality scores.

###  4.1. Distortion Stream

####  4.1.1. Multi-stage Feature Fusion

Games and movies are among the most critical artistic mediums since they are
believed to pass conceptual ideas, spread emotional powers, and inspire
creative thinking (Gintere, 2019; Patton, 2013; Mackay, 2017; Mendiburu,
2012). Therefore, the CGIs obtained from games and movies require the viewers
to identify the semantic contents before fully understanding the images. In
previous quality assessment studies (Dhar et al., 2011; Kao et al., 2017;
Zhang et al., 2023a), the semantic information is stated to be closely
correlated with the high-level attributes. At the same time, due to poor
rendering and transmission loss, CGIs also suffer from low-level distortions
such as blur and texture damage. Therefore, we propose a multi-stage feature
fusion (MFF) module to make full use of the quality-aware information from
low-level to high-level in CGIQA tasks as well.

Assume that there are $N_{s}$ stages, and $F_{i}$ is the extracted feature map
from the $i$-th stage, where $i\in\\{0,1,\ldots,N_{s}-1\\}$. Considering that
the feature maps obtained from different stages have different resolutions, we
first apply the 2D adaptive average pooling with the target output size of
7$\times$7 to keep all feature maps having the same resolution:

(3) $\hat{F_{i}}=A_{7\times 7}F_{i},$

where $A_{7\times 7}$ represents the adaptive average pooling operations with
a target output size of 7$\times$7\. Then the pooled feature maps are
concatenated to form the combined feature maps $\hat{F}\in\mathbf{R}^{C\times
7\times 7}$ ($C$ is the sum of the number of channels):

(4) $\hat{F}=\mathop{\cup}\limits_{i=0}^{N_{s}-1}F_{i},$

where $\cup$ indicates the concatenation process. To dynamically explore the
spatial information across different channels, we use a convolution block that
consists of 3 convolution layers to generate the fused feature maps:

(5) $\hat{F_{s}}=W_{1\times 1}\times W_{3\times 3}\times W_{1\times
1}\times\hat{F},$

where $W_{1\times 1}$ and $W_{3\times 3}$ represent the 1$\times$1 and
3$\times$3 convolution operation. The first $1\times 1$ convolution layer is
used to reduce the channels to a quarter and generates feature maps
$W_{1\times 1}\times\hat{F}\in\mathbf{R}^{\frac{C}{4}\times 7\times 7}$, which
helps lower the computational complexity. Then the $3\times 3$ convolution
layer is applied to fuse the spatial information across channels and generates
feature maps $W_{3\times 3}\times W_{1\times
1}\times\hat{F}\in\mathbf{R}^{\frac{C}{4}\times 7\times 7}$. Finally, the
$1\times 1$ convolution layer is employed to adjust the number of channels and
we set the number of output channels as the same of the last stage’s feature
maps and we can get the fused feature maps
$\hat{F_{s}}\in\mathbf{R}^{C^{\prime}\times 7\times 7}$ ($C^{\prime}$ is the
number of adjusted channels).

####  4.1.2. Multi-stage Channel Attention

Each channel of the feature maps is believed to function as a feature detector
and focuses on representative meanings of the image (Woo et al., 2018). For
example, the channels may represent specific aspects, possibly such as
texture, color, illumination (low-level), and interactivity, understandability
(high-level), etc, in CGIQA tasks. It is apparent that the features reflected
by different channels have different impacts on human perception. Inspired by
many channel attention works (Hu et al., 2018; Li et al., 2019; Woo et al.,
2018; Park et al., 2018), we introduce a typical channel attention module to
improve the representation of the multi-stage fused feature maps as the multi-
stage channel attention (MCA) module.

At the beginning of this module, we simply squeeze the spatial dimension of
the obtained feature maps $\hat{F_{s}}$ using both average-pooling and max-
pooling. It is argued that average-pooling together with max-pooling can
gather both global and distinctive features, which can help improve the
representation power of channel-wise attention (Woo et al., 2018). The process
can be derived as:

(6) $\displaystyle\hat{F_{a}}$ $\displaystyle=A_{1\times 1}\hat{F_{s}},$
$\displaystyle\hat{F_{m}}$ $\displaystyle=M_{1\times 1}\hat{F_{s}},$

where $A_{1\times 1}$ and $M_{1\times 1}$ stand for the average-pooling and
max-pooling, and such squeeze process generates channel descriptors
$\hat{F_{a}}\in\mathbf{R}^{C^{\prime}\times 1\times 1}$ and
$\hat{F_{m}}\in\mathbf{R}^{C^{\prime}\times 1\times 1}$. Then the descriptors
are put through multi-layer perceptron (MLP) with one hidden layer, the
parameters of which are shared for $\hat{F_{a}}$ and $\hat{F_{m}}$. Similarly,
to reduce the computation complexity, the hidden activation size of the MLP is
set as $\mathbf{R}^{\frac{C^{\prime}}{r}\times 1\times 1}$, where $r$ is the
reduction ratio. The MLP output size is still set the same as the input
channel descriptors ($\mathbf{R}^{C^{\prime}\times 1\times 1}$). The channel
attention feature maps can be computed as the sum of forwarded channel
descriptors:

(7) $\hat{F_{c}}=\delta(MLP(\hat{F_{a}})\oplus MLP(\hat{F_{m}})),$

where $\oplus$ indicates the element-wise summation operation and $\delta$
denotes the sigmoid function. The final strengthened quality-wise feature
vector can then be computed with the elementwise multiplication and average
pooling:

(8) $\mathbf{{F_{D}}}=Avg(\hat{F_{c}}\otimes\hat{F_{s}}),$

where $\otimes$ indicates the element-wise multiplication, $Avg(\cdot)$
represents the average pooling operation, the final distortion quality-wise
feature vector $\mathbf{{F_{D}}}\in\mathbf{R}^{{C_{D}}\times 1}$, the multi-
stage channel attention feature maps
$\hat{F_{c}}\in\mathbf{R}^{C^{\prime}\times 1\times 1}$, and multi-stage fused
feature maps $\hat{F_{s}}\in\mathbf{R}^{C^{\prime}\times 7\times 7}$.

###  4.2. Aesthetic Stream

To specifically adapt to the AQA for CGIs and ensure evaluation robustness, we
pre-train the backbone on the TGQA (Ling et al., 2020) and AVA (Murray et al.,
2012) databases. Considering that the aesthetic quality is highly correlated
with the semantics (Li et al., 2023), we only employ the high-level features
(obtained from the last stage) for analysis. The backbone is first trained on
the AVA database and then trained on the TGQA database since the AVA database
is much larger in scale. The utilization of these two distinct databases and
the sequential training process allows for the acquisition of both general and
CGI-related aesthetic quality representations, contributing to a more
comprehensive understanding of aesthetic perception in the context of the
given task. Afterward, the aesthetic features can be derived as:

(9) $\mathbf{{F_{A}}}=Avg(F_{l}),$

where $F_{l}$ is the feature map from the last stage of the backbone and the
final aesthetic quality-wise feature vector
$\mathbf{{F_{A}}}\in\mathbf{R}^{{C_{A}}\times 1}$. It’s worth mentioning that
the pre-trained weights of the aesthetic stream backbone are fixed to avoid
the impact of distortion in the joint training.

###  4.3. Feature Fusion & Regression

With the extracted features, two Fully Connected (FC) layers are adopted as
the feature regression module, which consists of 1024 and 128 neurons
respectively:

(10) $\hat{Q}=FC(MLP(\mathbf{{F_{D}}})\cup MLP(\mathbf{{F_{A}}})),$

where $\hat{Q}$ represents the predicted quality score and the multi-layer
perception is used to align the channels of the distortion and aesthetic
features. The mean squared error (MSE) is utilized as the loss function:

(11) $Loss=\frac{1}{m}\sum_{i=1}^{m}\left(\hat{Q}_{i}-Q_{i}\right)^{2},$

where $Q$ represents the ground-truth quality score obtained from subjective
experiments and $m$ indicates the number of images in a mini batch.

##  5\. Experiment

In this section, we conduct experiments on the proposed CGIQA-6k and other
CGIQA-related databases. Ablation experiments are carried out to demonstrate
the contributions and effects of the proposed streams and modules. Cross-
database validation is conducted to confirm the generalization ability of the
proposed method. In-depth performance discussions are given as well.

###  5.1. Comparing Models

Since there are no pristine reference images in the proposed CGIQA-6k
database, we only select NR IQA models for comparison. The chosen models can
be divided into handcrafted-based models and deep learning-based models:

  * •

Handcrafted-based models : These models include BRISQUE (Mittal et al.,
2012a), NIQE (Mittal et al., 2012b), BIBLE (Li et al., 2015), BLIINDS2 (Saad
et al., 2012), BMPRI (Min et al., 2018), CPBD (Narvekar and Karam, 2011), UCA
(Min et al., 2017), and NFERM (Gu et al., 2014). Such models extract
handcrafted features through prior knowledge, and then employ the extracted
features to evaluate the quality levels of images.

  * •

Deep learning-based models: These models consist of DBCNN (Zhang et al.,
2018), MUSIQ (Ke et al., 2021), HyperIQA (Su et al., 2020), MGQA (Wang et al.,
2021), and StairIQA (Sun et al., 2023). These mentioned models obtain quality-
aware information by learning a feature representation from the labeled data
and have achieved competitive performance on previous IQA tasks.

###  5.2.  Evaluation Criteria

Four mainstream consistency evaluation criteria are used to measure the
correlation between the predicted scores and MOS, which include Spearman Rank
Correlation Coefficient (SRCC), Pearson Linear Correlation Coefficient (PLCC),
Kendall’s Rank Correlation Coefficient (KRCC), Root Mean Squared Error (RMSE).
The SRCC values represent the similarity between two groups of rankings, the
PLCC values describe the linear correlation of two sets of rankings, the KRCC
values denote the ordinal association between two measured quantities, and the
RMSE values stand for the average distance between the predicted scores and
labels.

Before obtaining the criteria values, a five-parameter logistic function is
applied to map the predicted scores according to the practices in (Sheikh et
al., 2006b):

(12)
$\hat{y}=\beta_{1}\left(0.5-\frac{1}{1+e^{\beta_{2}\left(y-\beta_{3}\right)}}\right)+\beta_{4}y+\beta_{5},$

where $\left\\{\beta_{i}\mid i=1,2,\ldots,5\right\\}$ are parameters to be
fitted, $y$ and $\hat{y}$ are the predicted scores and mapped scores
respectively. Besides, the MOSs for all databases are rescaled to a five-point
scale for validation.

Table 2. Performance comparison on the CGIQA-6k database and the corresponding subsets. The best performance results are marked in RED and the second performance results are marked in BLUE. Index | Model | Type | Game | Movie | All
---|---|---|---|---|---
SRCC$\uparrow$ |  PLCC$\uparrow$ |  KRCC$\uparrow$ |  RMSE$\downarrow$ |  SRCC$\uparrow$ |  PLCC$\uparrow$ |  KRCC$\uparrow$ |  RMSE$\downarrow$ |  SRCC$\uparrow$ |  PLCC$\uparrow$ |  KRCC$\uparrow$ |  RMSE$\downarrow$
A | BRISQUE | Hand-crafted-based | 0.4231 | 0.4246 | 0.2855 | 0.5460 | 0.4434 | 0.4316 | 0.3083 | 0.5590 | 0.3979 | 0.4126 | 0.2735 | 0.6531
B | NIQE | 0.1185 | 0.1016 | 0.0907 | 0.6188 | 0.0326 | 0.0114 | 0.0245 | 0.6423 | 0.1091 | 0.0777 | 0.0821 | 0.7035
C | BIBLE | 0.0828 | 0.0704 | 0.0542 | 0.6265 | 0.0268 | 0.0334 | 0.0171 | 0.6448 | 0.0064 | 0.0077 | 0.0031 | 0.7025
D | BLIINDS2 | 0.3696 | 0.3509 | 0.2499 | 0.5872 | 0.3670 | 0.3693 | 0.2490 | 0.5927 | 0.3530 | 0.3449 | 0.2373 | 0.6704
E | BMPRI | 0.2305 | 0.2378 | 0.1548 | 0.6010 | 0.2245 | 0.2514 | 0.1536 | 0.6046 | 0.1927 | 0.1933 | 0.1292 | 0.6944
F | CPBD | 0.1952 | 0.1857 | 0.1307 | 0.6331 | 0.1643 | 0.1321 | 0.1119 | 0.6204 | 0.1519 | 0.1536 | 0.1014 | 0.7006
G | UCA | 0.3044 | 0.3211 | 0.2118 | 0.5822 | 0.3595 | 0.3807 | 0.2484 | 0.5613 | 0.2126 | 0.2254 | 0.1409 | 0.6921
H | NFERM | 0.3660 | 0.4241 | 0.2513 | 0.5543 | 0.3801 | 0.3897 | 0.2592 | 0.6035 | 0.3978 | 0.3679 | 0.2666 | 0.6747
I | DBCNN | Deep learning-based | 0.5907 | 0.6439 | 0.4423 | 0.5055 | 0.6533 | 0.6402 | 0.4668 | 0.5492 | 0.6437 | 0.6568 | 0.4593 | 0.5460
J | MUSIQ | 0.6067 | 0.6158 | 0.4347 | 0.5256 | 0.7181 | 0.7140 | 0.5293 | 0.4442 | 0.7104 | 0.7250 | 0.5251 | 0.4870
K | HypherIQA | 0.6271 | 0.6371 | 0.4485 | 0.5533 | 0.6465 | 0.6822 | 0.4666 | 0.4902 | 0.7003 | 0.7224 | 0.5160 | 0.5140
L | MGQA | 0.7472 | 0.7586 | 0.5579 | 0.3816 | 0.7494 | 0.7573 | 0.5648 | 0.3992 | 0.7999 | 0.8053 | 0.5865 | 0.4110
M | StairIQA | 0.7461 | 0.7601 | 0.5567 | 0.3806 | 0.7410 | 0.7493 | 0.5552 | 0.3938 | 0.8117 | 0.8186 | 0.6191 | 0.4082
N | Proposed | 0.8169 | 0.8209 | 0.6431 | 0.3783 | 0.8234 | 0.8303 | 0.6577 | 0.3865 | 0.8552 | 0.8531 | 0.6444 | 0.3781
Figure 8. Mean SRCC values and standard error bars for methods compared on the
CGIQA-6k database and its corresponding subsets. A-N are of the same order as
in Table 2.

###  5.3. Experimental Setup

####  5.3.1. Aesthetic Pre-training Setup

  * •

Input Size & Backbone: The 224$\times$224 views are adopted as inputs by
random-cropping the resized images with dimensions of 256$\times$256\. The
Swin Transformer tiny (ST-t) (Liu et al., 2021) is initialized with the
weights pretrained on the ImageNet-22K database (Deng et al., 2009).

  * •

Optimizer: The Adam optimizer (Kingma and Ba, 2014) is employed with an
initial learning rate set as 1e-5. The default batch size is set as 32.

  * •

Pre-training Databases: The AVA (Murray et al., 2012) database, which includes
250,000 aesthetic images, and the TGQA (Ling et al., 2020) database, which
includes 1091 aesthetic mobile game images, are used as the pre-training
databases. To begin, the aesthetic stream is trained on the AVA database for
50 epochs. This initial training is aimed to acquire a comprehensive
understanding of general aesthetic quality representations. Following this,
the aesthetic stream is further trained on the TGQA database for an additional
10 epochs. This subsequent training stage focuses on enhancing the aesthetic
stream’s ability to capture aesthetic quality representations specific to
CGIs.

####  5.3.2. Main Experimental Setup

The details of the experimental setup are shown as follows:

  * •

Input Size & Backbone: Similarly, the 224$\times$224 views and the
ImageNet-22K database (Deng et al., 2009) initialized Swin Transformer tiny
(ST-t) (Liu et al., 2021) backbone are utilized. Specifically, the reduction
ratio $r$, the channel number $C_{D}$, and the channel number $C_{A}$
mentioned in Section 4.1.1, 4.1.2, and 4.2 are 16, 720, and 768 respectively.
For other comparing deep learning-based methods, the default training
parameters are utilized.

  * •

Optimizer: The Adam optimizer (Kingma and Ba, 2014) is employed with an
initial learning rate set as 1e-5. The default batch size is set as 32. Each
training process lasts for 40 epochs.

  * •

Training and Testing set: We randomly separate the databases into the training
sets and testing sets with a ratio of 8:2. We repeat the training process 10
times and the average results are recorded as the final performance.

Table 3. Performance results on the CCT-CGI, NBU-CIQAD, and LIVE-YT-Gaming databases. The best performance results are marked in RED and the second performance results are marked in BLUE. Index | Model | Type | CCT-CGI | NBU-CIQAD | LIVE-YT-Gaming
---|---|---|---|---|---
SRCC$\uparrow$ |  PLCC$\uparrow$ |  KRCC$\uparrow$ |  RMSE$\downarrow$ |  SRCC$\uparrow$ |  PLCC$\uparrow$ |  KRCC$\uparrow$ |  RMSE$\downarrow$ |  SRCC$\uparrow$ |  PLCC$\uparrow$ |  KRCC$\uparrow$ |  RMSE$\downarrow$
A | BRISQUE | Hand-crafted-based | 0.8468 | 0.8609 | 0.6633 | 0.3605 | 0.4858 | 0.4579 | 0.3744 | 0.6860 | 0.4815 | 0.4986 | 0.3397 | 0.7437
B | NIQE | 0.6888 | 0.6862 | 0.5078 | 0.6760 | 0.4932 | 0.4985 | 0.3412 | 0.8329 | 0.5412 | 0.5864 | 0.3852 | 0.6990
C | BIBLE | 0.5941 | 0.6272 | 0.4195 | 0.7204 | 0.1452 | 0.1071 | 0.0950 | 1.0008 | 0.1647 | 0.1344 | 0.1108 | 0.9782
D | BLIINDS2 | 0.8894 | 0.8790 | 0.7082 | 0.3124 | 0.3915 | 0.3825 | 0.2700 | 0.9216 | 0.4204 | 0.4839 | 0.2915 | 0.7294
E | BMPRI | 0.8700 | 0.8770 | 0.6751 | 0.3292 | 0.4001 | 0.4140 | 0.2703 | 0.9258 | 0.4461 | 0.4844 | 0.3118 | 0.7746
F | CPBD | 0.6612 | 0.6490 | 0.4799 | 0.4579 | 0.2295 | 0.2414 | 0.1860 | 0.8958 | 0.0184 | 0.0422 | 0.0112 | 0.8232
G | UCA | 0.8459 | 0.8854 | 0.6450 | 0.5822 | 0.2916 | 0.1642 | 0.0817 | 0.9554 | 0.2503 | 0.2910 | 0.1734 | 0.8891
H | NFERM | 0.3570 | 0.3340 | 0.2413 | 0.6094 | 0.0848 | 0.0736 | 0.0580 | 1.0048 | 0.2035 | 0.1684 | 0.1385 | 0.8782
I | DBCNN | Deep learning-based | 0.9111 | 0.9123 | 0.7365 | 0.2911 | 0.7295 | 0.7301 | 0.5296 | 0.7162 | 0.6832 | 0.5240 | 0.5035 | 0.7989
J | MUSIQ | 0.9077 | 0.9045 | 0.7481 | 0.2710 | 0.8419 | 0.8531 | 0.6481 | 0.5792 | 0.6107 | 0.4095 | 0.4435 | 0.8094
K | HypherIQA | 0.9122 | 0.9219 | 0.7437 | 0.2699 | 0.6297 | 0.6771 | 0.5383 | 0.5831 | 0.5200 | 0.5840 | 0.3705 | 0.7183
L | MGQA | 0.9061 | 0.9062 | 0.7825 | 0.2901 | 0.8916 | 0.8842 | 0.7113 | 0.4712 | 0.7266 | 0.7791 | 0.5376 | 0.5312
M | StairIQA | 0.9163 | 0.9142 | 0.7660 | 0.2454 | 0.9070 | 0.9041 | 0.7277 | 0.4311 | 0.6808 | 0.7024 | 0.4903 | 0.6032
N | Proposed | 0.9210 | 0.9230 | 0.7851 | 0.2220 | 0.9335 | 0.9332 | 0.7838 | 0.3547 | 0.7613 | 0.7733 | 0.5612 | 0.5212

###  5.4. Performance on the CGIQA-6k Database

The summary of experimental performance on the CGIQA-6k database and its
corresponding subsets is presented in Table 2. The table highlights the best
performance achieved for each column. Additionally, to further validate the
effectiveness and stability of various IQA methods, the mean SRCC values and
corresponding standard error bars are illustrated in Fig. 8. The following
observations can be made based on the aforementioned results: a) Deep
learning-based methods demonstrate significant advantages over handcrafted-
based methods in terms of both mean and standard deviation values. This
disparity arises from the fact that handcrafted-based methods are designed to
extract features suited for natural scene content rather than computer
graphics content. Conversely, deep learning-based methods have the capability
to learn more effective feature representations specifically tailored for
CGIs. b) With the exception of DBCNN (Zhang et al., 2018) and MUSIQ (Ke et
al., 2021), most deep learning-based methods exhibit improved performance
across the entire CGIQA-6k database compared to the separate subsets. This
observation indicates that learning from a larger sample size contributes to a
better understanding of CGIs. c) All deep learning-based methods achieve
better performance on the movie subset compared to the game subset. This can
be attributed to the movie CGIs being more refined, thereby making the
perceived distortion levels more discernible. Conversely, the game CGIs tend
to have lower initial quality levels due to limitations in computing
resources, resulting in relatively inconspicuous distortions. d) The proposed
method demonstrates the highest performance on the CGIQA-6k database,
including both subsets, and exhibits the smallest standard deviation values
among all IQA methods considered in the comparison. This suggests that the
proposed method performs more effectively and consistently.

###  5.5. Performance on other CGIQA-related Databases

####  5.5.1. Databases Selection

To further demonstrate the effectiveness of the proposed method, we also
select several existing CGIQA-related databases to compare the performance of
the proposed model and the mainstream IQA models, which include the CCT (Min
et al., 2017), NBU-CIQAD (Chen et al., 2021), LIVE-YT-Gaming (Yu et al.,
2022). A detailed introduction of the selected databases is given as follows:

  * •

CCT: The CCT database is a cross-content-type IQA database that deals with the
quality assessment for natural scene images (NSIs), computer graphic images
(CGIs), and screen content images (SCIs). We only adopt the 528 distorted CGIs
for validation and such subset of the CCT database is referred to as the CCT-
CGI database in this paper. The distortions are generated by HEVC (Sullivan et
al., 2012) and HEVC-SCC (Xu et al., 2015) standards.

  * •

NBU-CIQAD: The NBU-CIQAD database focuses on the quality assessment of cartoon
images, which has a high correlation with CGIs. The NBU-CIQAD database
consists of 1,800 cartoon images corrupted by single type of distortion and
800 cartoon images suffering from multiple types of distortions. The
distortions are caused by the manual adjustment of brightness, saturation, and
contrast.

  * •

LIVE-YT-Gaming: The LIVE-YT-Gaming database includes 600 authentic user-
generated content (UGC) gaming videos and 18,600 subjective quality ratings
collected from an online subjective study. We extract frames from the gaming
videos with a fixed interval of 60 frames and obtain 3,501 game images for
validation. The images share the same quality scores as the corresponding
videos.

We maintain the default experimental setup. Additionally, the distorted images
of the CCT-CGI and the NBU-CIQAD databases are synthesized from the
corresponding reference images. Therefore, we separate the training and
testing sets without content overlap.

####  5.5.2. Performance discussion

Table 3 presents the performance of the proposed method on other CGIQA-related
IQA databases. A detailed analysis of the results is provided below: a) The
CCT-CGI database utilizes manual compression distortions for evaluation. Both
handcrafted-based and deep learning-based methods demonstrate relatively good
performance. Notably, all deep learning-based methods achieve SRCC values
higher than 0.9. b) The NBU-CIQAD database focuses specifically on color
distortions. In this task, deep learning-based methods outperform handcrafted-
based methods. This can be attributed to the fact that handcrafted features
have a low correlation with color information, making deep learning approaches
more effective in this context. c) The LIVE-YT-Gaming database consists of
authentic distorted contents. As a result, all IQA models experience a
noticeable performance drop compared to other databases. Assessing authentic
distortions is considered more challenging, which contributes to the lower
performance across the models. These observations shed light on the strengths
of the proposed method in relation to other CGIQA-related IQA databases,
showcasing its performance in different distortion scenarios.

Table 4. Performance Results of the ablation study on the two streams. The mark ✓indicates that the referred stream is applied while $\times$ indicates that the referred stream is not applied. The aesthetic stream backbone weights are fixed and only linear regression is employed for the single aesthetic stream. Distortion | Aesthetic | CGIQA-6K | CCT-CGI | NBU-CIQAD | LIVE-YT
---|---|---|---|---|---
SRCC$\uparrow$ |  PLCC$\uparrow$ |  SRCC$\uparrow$ |  PLCC$\uparrow$ |  SRCC$\uparrow$ |  PLCC$\uparrow$ |  SRCC$\uparrow$ |  PLCC$\uparrow$
✓ | ✓ | 0.8552 | 0.8531 | 0.9210 | 0.9230 | 0.9335 | 0.9332 | 0.7613 | 0.7733
✓ | $\times$ | 0.8288 | 0.8353 | 0.9111 | 0.9033 | 0.9297 | 0.9292 | 0.7451 | 0.7471
$\times$ | ✓ | 0.3002 | 0.3091 | 0.4456 | 0.4245 | 0.5345 | 0.5375 | 0.3801 | 0.3834
Table 5. Performance Results of the ablation study on the attention modules. The mark ✓indicates that the referred module is applied while $\times$ indicates that the referred module is not applied. MFF | MCA | CGIQA-6K | CCT-CGI | NBU-CIQAD | LIVE-YT
---|---|---|---|---|---
SRCC$\uparrow$ |  PLCC$\uparrow$ |  SRCC$\uparrow$ |  PLCC$\uparrow$ |  SRCC$\uparrow$ |  PLCC$\uparrow$ |  SRCC$\uparrow$ |  PLCC$\uparrow$
✓ | ✓ | 0.8552 | 0.8531 | 0.9210 | 0.9230 | 0.9335 | 0.9332 | 0.7613 | 0.7733
✓ | $\times$ | 0.8399 | 0.8316 | 0.9107 | 0.9122 | 0.9201 | 0.9214 | 0.7331 | 0.7343
$\times$ | ✓ | 0.8410 | 0.8427 | 0.9200 | 0.9119 | 0.9276 | 0.9264 | 0.7467 | 0.7481
$\times$ | $\times$ | 0.8310 | 0.8368 | 0.9077 | 0.9086 | 0.9174 | 0.9163 | 0.7300 | 0.7321

###  5.6. Ablation Study

####  5.6.1. Two Streams Contribution

In this section, we investigate the contributions of the distortion and
aesthetic streams employed in the proposed model for evaluating the visual
quality of CGIs. The results of the ablation study for the two streams are
presented in Table 4, revealing the following findings: Firstly, employing
both streams together yields better performance compared to using each single
stream separately. This observation confirms the contributions of both the
distortion and aesthetic streams in evaluating CGI quality, emphasizing the
importance of considering both aspects. Secondly, the aesthetic stream
demonstrates significantly inferior performance compared to the distortion
stream. This finding suggests that distortions have a more substantial impact
on human perception when it comes to CGIs. It indicates that the presence of
distortions plays a more critical role in shaping the perceived quality of
CGIs, while the aesthetic attributes have a relatively less pronounced
influence.

####  5.6.2. Distortion Stream Modules Contribution

To quantify the contributions of each stream in the proposed method, we
conduct an ablation study while maintaining the default experiment setup. The
results of the ablation study are presented in Table 5, providing several
notable conclusions. Firstly, when the MFF and MCA modules are separately
attached, they outperform the bare backbone model. This suggests that both of
the proposed modules contribute to the final results, indicating their
individual effectiveness. Secondly, employing both the MFF and MCA modules
together achieves the highest performance, further confirming the
effectiveness of the proposed framework as a whole. Thirdly, employing only
the MFF module results in lower performance compared to employing only the MCA
module. This finding suggests that the MCA module seems to make a more
significant contribution to the proposed framework. It implies that channel
attention plays a crucial role, and human perception of CGIs is more easily
influenced by specific feature attributes.

Table 6. Performance results of the different backbones on the CGIQA-6k database. The proposed framework is adaptively applied to the backbones. The number of parameters and flops are exhibited as well. Backbone |  SRCC$\uparrow$ |  PLCC$\uparrow$ |  KRCC$\uparrow$ |  RMSE$\downarrow$ |  FLOPs$\downarrow$ |  Params$\downarrow$
---|---|---|---|---|---|---
VGG16 | 0.8212 | 0.8219 | 0.6087 | 0.4254 | 30.94G | 277.74M
ResNet50 | 0.8312 | 0.8306 | 0.6111 | 0.4114 | 8.27G | 52.14M
MobileNetV2 | 0.7875 | 0.7918 | 0.5855 | 0.4339 | 0.66G | 8.03M
ConvNeXt-tiny | 0.8419 | 0.8481 | 0.6551 | 0.3768 | 8.91G | 55.99M
Swin-tiny | 0.8552 | 0.8531 | 0.6444 | 0.3781 | 8.94G | 59.92M

###  5.7. Backbone Comparison

The proposed framework can be easily adapted to other CNN modules which
contain multiple stages. To further test the performance of different
backbones, we select VGG16 (Simonyan and Zisserman, 2014), ResNet50 (He et
al., 2016), MobileNetV2 (Sandler et al., 2018), and ConvNeXt-tiny (Liu et al.,
2022a) for comparison. The performance results on the CGIQA-6k database are
clearly exhibited in Table 6, from which we can make several useful
observations. First, the effectiveness of the proposed framework is
demonstrated by the competitive performance achieved by all backbones in
comparison to other NR IQA methods. This outcome highlights the capability of
the proposed framework to effectively address the task at hand. Second, the
Swin Transformer tiny backbone achieves the best performance among the
evaluated models. This finding suggests that the Swin Transformer tiny
architecture is particularly well-suited for the specific requirements of the
task. Additionally, the light-weight MobileNetV2 model also demonstrates
strong performance, indicating its potential for deployment on mobile devices
by only consuming 0.66G flops and 8.03M parameters. This finding highlights
the versatility of the proposed framework, which can accommodate various
popular CNN models, thus enabling broader applications in CGIQA tasks with
diverse demands.

Table 7. Experimental results for the cross-database validation. The NR IQA models are trained on the CGIQA-6K database. The default experimental setup is maintained. Model | CCT-CGI | NBU-CIQAD
---|---|---
SRCC$\uparrow$ |  PLCC$\uparrow$ |  SRCC$\uparrow$ |  PLCC$\uparrow$
MGQA | 0.7088 | 0.7086 | 0.2341 | 0.2797
StairIQA | 0.7211 | 0.7297 | 0.2275 | 0.2608
Proposed | 0.8092 | 0.8293 | 0.3061 | 0.3003

###  5.8. Cross-Database Validation

In this section, we perform cross-database validation to demonstrate the
robustness and generalization ability of the proposed method. We select the
most competitive NR IQA models, namely MGQA and StairIQA, and train all models
on the CGIQA-6K database. Subsequently, we evaluate the models on the CCT-CGI
and NBU-CIQAD databases (excluding the LIVE-YT-Gaming database, which focuses
on videos). The validation results are presented in Table 7, leading to the
following observations: a) The proposed method outperforms all the competitors
by a significant margin, indicating its strong generalization ability. This
suggests that the proposed method is capable of performing well on unseen
databases, highlighting its robustness and effectiveness in assessing CGI
quality. b) The cross-database validation results on the NBU-CIQAD database
are relatively unsatisfactory, and we provide insights into the reasons behind
this performance gap. The CGIQA-6K database primarily consists of CGIs and
encompasses a wide range of in-the-wild distortions. In contrast, the NBU-
CIQAD database consists solely of cartoon images and primarily focuses on
color-specific distortions. This significant disparity in content and
distortions between the CGIQA-6K and NBU-CIQAD databases leads to poor
performance. However, the CCT-CGI database comprises CGIs and primarily
focuses on compression distortion, which falls within the scope of in-the-wild
distortions. Consequently, the cross-database validation results on the CCT-
CGI database are more impressive.

(a) Game (b) Movie (c) All (d) CCT-CGI (e) NBU-CIQAD (f) LIVE-YT-Gaming

Figure 9. Statistical test results on the proposed CGIQA-6k, CCT-CGI, NBU-
CIQAD, and LIVE-YT-Gaming databases. A black/white block means the row method
is statistically worse/better than the column one. A gray block means the row
method and the column method are statistically indistinguishable. The metrics
are denoted by the same index as in Table 2 and Table 3 respectively.

###  5.9. Statistical Test

In this section, we conduct a statistical test to further validate the
effectiveness of the proposed method. We adhere to the experimental setup
outlined in (Sheikh et al., 2006a) and compare the disparity between the
predicted quality scores and the subjective ratings. The null hypothesis
posits that the residuals from one quality assessment model originate from the
same distribution and are, at a 95% confidence level, statistically
indistinguishable from the residuals of another quality assessment model. All
possible pairs of models are examined, and the outcomes are presented in Fig.
9. Upon closer examination, we observe that the proposed method exhibits
statistical superiority over all the compared NR IQA models on the CGIQA-6k
database and its corresponding subsets. Additionally, our method demonstrates
competitive performance on other CGIQA-related databases. Specifically, the
proposed method statistically outperforms 6, 12, and 13 NR IQA models on the
CCT-CGI, NBU-CIQAD, and LIVE-YT-Gaming databases, respectively. These findings
reinforce the efficacy of our proposed method, establishing its significant
statistical advantage in the domain of CGIQA and its associated databases.

##  6\. Conclusion

In this paper, we construct a large-scale in-the-wild CGIQA database named
CGIQA-6k. The database consists of 3,000 game CGIs and 3,000 movie CGIs and
covers a wider range of resolutions. Through meticulous subjective experiments
conducted in a controlled laboratory environment, we have obtained accurate
perceptual ratings for the CGIs in the database. This database serves as a
valuable resource for advancing research in the field. Furthermore, we propose
an effective deep learning-based NR IQA model that leverages both distortion
and aesthetic quality representation. Our model outperforms all other state-
of-the-art NR IQA methods on the CGIQA-6k database as well as other CGIQA-
related databases. The superior performance demonstrates the efficacy and
robustness of our approach in assessing CGI quality. In summary, this work
contributes to bridging the gap between evaluating NSIs and CGIs by
constructing a comprehensive CGIQA database and proposing a powerful deep
learning-based NR IQA model. The availability of the CGIQA-6k database will
facilitate further research and advancements in the field of CGIQA.

## References

  * (1)
  * Bai et al. (2021) Weiming Bai, Zhipeng Zhang, Bing Li, Pei Wang, Yangxi Li, Congxuan Zhang, and Weiming Hu. 2021.  Robust Texture-Aware Computer-Generated Image Forensic: Benchmark and Algorithm.  _IEEE Transactions on Image Processing_ 30 (2021), 8439–8453.
  * Barman et al. (2019) Nabajeet Barman, Emmanuel Jammeh, Seyed Ali Ghorashi, and Maria G Martini. 2019.  No-reference video quality estimation based on machine learning for passive gaming video streaming applications.  _IEEE Access_ 7 (2019), 74511–74527.
  * Barman et al. (2018) Nabajeet Barman, Saman Zadtootaghaj, Steven Schmidt, Maria G Martini, and Sebastian Möller. 2018.  GamingVideoSET: a dataset for gaming video streaming applications. In _2018 16th Annual Workshop on Network and Systems Support for Games (NetGames)_. IEEE, 1–6.
  * BT (2002) RECOMMENDATION ITU-R BT. 2002.  Methodology for the subjective assessment of the quality of television pictures.  _International Telecommunication Union_ (2002).
  * Chen et al. (2021) Hangwei Chen, Xiongli Chai, Feng Shao, Xuejin Wang, Qiuping Jiang, Mengxiang Chao, and Yo-Sung Ho. 2021.  Perceptual Quality Assessment of Cartoon Images.  _IEEE Transactions on Multimedia_ (2021).
  * Chen et al. (2013) Kuan-Ta Chen, Yu-Chun Chang, Hwai-Jung Hsu, De-Yu Chen, Chun-Ying Huang, and Cheng-Hsin Hsu. 2013.  On the quality of service of cloud gaming systems.  _IEEE Transactions on Multimedia_ 16, 2 (2013), 480–495.
  * Cui et al. (2018) C. Cui, H. Liu, T. Lian, L. Nie, L. Zhu, and Y. Yin. 2018.  Distribution-oriented aesthetics assessment with semantic-aware hybrid network.  _IEEE transactions on multimedia_ 21, 5 (2018), 1209–1220.
  * Datta et al. (2006) R. Datta, D. Joshi, J. Li, and J. Z. Wang. 2006.  Studying aesthetics in photographic images using a computational approach. In _European conference on computer vision_. Springer, 288–301.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 248–255.
  * Denisova and Cairns (2015) Alena Denisova and Paul Cairns. 2015.  First person vs. third person perspective in digital games: do player preferences affect immersion?. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_. 145–148.
  * Dhar et al. (2011) Sagnik Dhar, Vicente Ordonez, and Tamara L Berg. 2011.  High level describable attributes for predicting aesthetics and interestingness. In _CVPR 2011_. IEEE, 1657–1664.
  * Fan et al. (2022) Yu Fan, Zicheng Zhang, Wei Sun, Xiongkuo Min, Ning Liu, Quan Zhou, Jun He, Qiyuan Wang, and Guangtao Zhai. 2022.  A no-reference quality assessment metric for point cloud based on captured video sequences. In _IEEE International Workshop on Multimedia Signal Processing_. IEEE, 1–5.
  * Gintere (2019) Ieva Gintere. 2019.  A New Digital Art Game: The Art of the Future. In _SOCIETY. INTEGRATION. EDUCATION. Proceedings of the International Scientific Conference_ , Vol. 4. 346–360.
  * Golestaneh and Kitani (2020) S Alireza Golestaneh and Kris Kitani. 2020.  No-reference image quality assessment via feature fusion and multi-task learning.  _arXiv preprint arXiv:2006.03783_ (2020).
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014.  Generative adversarial nets.  _Advances in neural information processing systems_ 27 (2014).
  * Greenberg et al. (1997) Donald P Greenberg, Kenneth E Torrance, Peter Shirley, James Arvo, Eric Lafortune, James A Ferwerda, Bruce Walter, Ben Trumbore, Sumanta Pattanaik, and Sing-Choong Foo. 1997.  A framework for realistic image synthesis. In _Proceedings of the 24th annual conference on Computer graphics and interactive techniques_. 477–494.
  * Gu et al. (2014) Ke Gu, Guangtao Zhai, Xiaokang Yang, and Wenjun Zhang. 2014.  Using free energy principle for blind image quality assessment.  _IEEE Transactions on Multimedia_ 17, 1 (2014), 50–63.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Hosu et al. (2017) Vlad Hosu, Franz Hahn, Mohsen Jenadeleh, Hanhe Lin, Hui Men, Tamás Szirányi, Shujun Li, and Dietmar Saupe. 2017.  The Konstanz natural video database (KoNViD-1k). In _2017 Ninth international conference on quality of multimedia experience (QoMEX)_. IEEE, 1–6.
  * Hosu et al. (2020) V. Hosu, H. Lin, T. Sziranyi, and D. Saupe. 2020.  KonIQ-10k: An Ecologically Valid Database for Deep Learning of Blind Image Quality Assessment.  _IEEE Transactions on Image Processing_ 29 (2020), 4041–4056.
  * Hou et al. (2020) J. Hou, S. Yang, and W. Lin. 2020.  Object-level attention for aesthetic rating distribution prediction. In _Proceedings of the 28th ACM International Conference on Multimedia_. ACM, 816–824.
  * Hu et al. (2020) Bo Hu, Leida Li, Jinjian Wu, and Jiansheng Qian. 2020.  Subjective and objective quality assessment for image restoration: A critical survey.  _Signal Processing: Image Communication_ 85 (2020), 115839.
  * Hu et al. (2023) Bo Hu, Shuaijian Wang, Xinbo Gao, Leida Li, Ji Gan, and Xixi Nie. 2023.  Reduced-reference image deblurring quality assessment based on multi-scale feature enhancement and aggregation.  _Neurocomputing_ 547 (2023), 126378.
  * Hu et al. (2018) Jie Hu, Li Shen, and Gang Sun. 2018.  Squeeze-and-excitation networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7132–7141.
  * Jiang et al. (2022a) Qiuping Jiang, Yuese Gu, Chongyi Li, Runmin Cong, and Feng Shao. 2022a.  Underwater image enhancement quality evaluation: Benchmark dataset and objective metric.  _IEEE Transactions on Circuits and Systems for Video Technology_ 32, 9 (2022), 5959–5974.
  * Jiang et al. (2022b) Qiuping Jiang, Zhentao Liu, Ke Gu, Feng Shao, Xinfeng Zhang, Hantao Liu, and Weisi Lin. 2022b.  Single image super-resolution quality assessment: a real-world dataset, subjective studies, and an objective metric.  _IEEE Transactions on Image Processing_ 31 (2022), 2279–2294.
  * Kao et al. (2017) Yueying Kao, Ran He, and Kaiqi Huang. 2017.  Deep aesthetic quality assessment with semantic information.  _IEEE Transactions on Image Processing_ 26, 3 (2017), 1482–1495.
  * Ke et al. (2021) Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. 2021.  MUSIQ: Multi-scale Image Quality Transformer. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5148–5157.
  * Ke et al. (2006) Y. Ke, X. Tang, and F. Jing. 2006.  The design of high-level features for photo quality assessment. In _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , Vol. 1. IEEE, 419–426.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Kong et al. (2016) S. Kong, X. Shen, Z. Lin, R. Mech, and C. Fowlkes. 2016.  Photo aesthetics ranking network with attributes and content adaptation. In _European Conference on Computer Vision_. Springer, 662–679.
  * Laghari et al. (2019) Asif Ali Laghari, Hui He, Kamran Ali Memon, Rashid Ali Laghari, Imtiaz Ali Halepoto, and Asiya Khan. 2019.  Quality of experience (QoE) in cloud gaming models: A review.  _multiagent and grid systems_ 15, 3 (2019), 289–304.
  * Larson and Chandler (2010) Eric Cooper Larson and Damon Michael Chandler. 2010.  Most apparent distortion: full-reference image quality assessment and the role of strategy.  _Journal of electronic imaging_ 19, 1 (2010), 011006.
  * Li et al. (2018) Dingquan Li, Tingting Jiang, Weisi Lin, and Ming Jiang. 2018.  Which has better visual quality: The clear blue sky or a blurry animal?  _IEEE Transactions on Multimedia_ 21, 5 (2018), 1221–1234.
  * Li et al. (2023) Leida Li, Yipo Huang, Jinjian Wu, Yuzhe Yang, Yaqian Li, Yandong Guo, and Guangming Shi. 2023.  Theme-aware Visual Attribute Reasoning for Image Aesthetics Assessment.  _IEEE Transactions on Circuits and Systems for Video Technology_ (2023).
  * Li et al. (2015) Leida Li, Weisi Lin, Xuesong Wang, Gaobo Yang, Khosro Bahrami, and Alex C Kot. 2015.  No-reference image blur assessment based on discrete orthogonal moments.  _IEEE transactions on cybernetics_ 46, 1 (2015), 39–50.
  * Li et al. (2019) Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. 2019.  Selective kernel networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 510–519.
  * Lin et al. (2019) Hanhe Lin, Vlad Hosu, and Dietmar Saupe. 2019.  KADID-10k: A large-scale artificially distorted IQA database. In _2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX)_. IEEE, 1–3.
  * Ling et al. (2020) Suiyi Ling, Junle Wang, Wenming Huang, Yundi Guo, Like Zhang, Yanqing Jing, and Patrick Le Callet. 2020.  A subjective study of multi-dimensional aesthetic assessment for mobile game image. In _Proceedings of the 1st Workshop on Quality of Experience (QoE) in Visual Multimedia Applications_. 47–53.
  * Liu et al. (2022b) Yongxu Liu, Jinjian Wu, Leida Li, Weisheng Dong, and Guangming Shi. 2022b.  Quality Assessment of UGC Videos Based on Decomposition and Recomposition.  _IEEE Transactions on Circuits and Systems for Video Technology_ 33, 3 (2022), 1043–1054.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin transformer: Hierarchical vision transformer using shifted windows. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 10012–10022.
  * Liu et al. (2022a) Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. 2022a.  A convnet for the 2020s. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11976–11986.
  * Lu et al. (2022) Wei Lu, Wei Sun, Xiongkuo Min, Wenhan Zhu, Quan Zhou, Junxia He, Qiyuan Wang, Zicheng Zhang, Tao Wang, and Guangtao Zhai. 2022.  Deep Neural Network for Blind Visual Quality Assessment of 4K Content.  _IEEE Transactions on Broadcasting_ (2022).  https://doi.org/10.1109/TBC.2022.3221689
  * Lu et al. (2015) X. Lu, Z. Lin, X. Shen, R. Mech, and J. Z. Wang. 2015.  Deep multi-patch aggregation network for image style, aesthetics, and quality estimation. In _Proceedings of the IEEE international conference on computer vision_. 990–998.
  * Ma et al. (2017) S. Ma, J. Liu, and C. Wen Chen. 2017.  A-lamp: Adaptive layout-aware multi-patch deep convolutional neural network for photo aesthetic assessment. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4535–4544.
  * Mackay (2017) Daniel Mackay. 2017.  _The fantasy role-playing game: A new performing art_.  McFarland.
  * Mendiburu (2012) Bernard Mendiburu. 2012.  _3D movie making: stereoscopic digital cinema from script to screen_.  Routledge.
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2020.  Nerf: Representing scenes as neural radiance fields for view synthesis. In _European conference on computer vision_. Springer, 405–421.
  * Min et al. (2017) Xiongkuo Min, Kede Ma, Ke Gu, Guangtao Zhai, Zhou Wang, and Weisi Lin. 2017.  Unified blind quality assessment of compressed natural, graphic, and screen content images.  _IEEE Transactions on Image Processing_ 26, 11 (2017), 5462–5474.
  * Min et al. (2018) Xiongkuo Min, Guangtao Zhai, Ke Gu, Yutao Liu, and Xiaokang Yang. 2018.  Blind image quality estimation via distortion aggravation.  _IEEE Transactions on Broadcasting_ 64, 2 (2018), 508–517.
  * Mittal et al. (2012a) Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. 2012a.  No-reference image quality assessment in the spatial domain.  _IEEE Transactions on image processing_ 21, 12 (2012), 4695–4708.
  * Mittal et al. (2012b) Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. 2012b.  Making a “completely blind” image quality analyzer.  _IEEE Signal processing letters_ 20, 3 (2012), 209–212.
  * Murray et al. (2012) Naila Murray, Luca Marchesotti, and Florent Perronnin. 2012.  AVA: A large-scale database for aesthetic visual analysis. In _2012 IEEE conference on computer vision and pattern recognition_. IEEE, 2408–2415.
  * Narvekar and Karam (2011) Niranjan D Narvekar and Lina J Karam. 2011.  A no-reference image blur metric based on the cumulative probability of blur detection (CPBD).  _IEEE Transactions on Image Processing_ 20, 9 (2011), 2678–2683.
  * Nishiyama et al. (2011) M. Nishiyama, T. Okabe, I. Sato, and Y. Sato. 2011.  Aesthetic quality classification of photographs based on color harmony. In _CVPR 2011_. IEEE, 33–40.
  * Park et al. (2018) Jongchan Park, Sanghyun Woo, Joon-Young Lee, and In So Kweon. 2018.  Bam: Bottleneck attention module.  _arXiv preprint arXiv:1807.06514_ (2018).
  * Patton (2013) Ryan M Patton. 2013.  Games as an artistic medium: Investigating complexity thinking in game-based art pedagogy.  _Studies in Art Education_ 55, 1 (2013), 35–50.
  * Peng et al. (2021) Zhenyu Peng, Qiuping Jiang, Feng Shao, Wei Gao, and Weisi Lin. 2021.  Lggd+: Image retargeting quality assessment by measuring local and global geometric distortions.  _IEEE Transactions on Circuits and Systems for Video Technology_ 32, 6 (2021), 3422–3437.
  * Pharr et al. (2016) Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016.  _Physically based rendering: From theory to implementation_.  Morgan Kaufmann.
  * Ponomarenko et al. (2015) Nikolay Ponomarenko, Lina Jin, Oleg Ieremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, et al. 2015\.  Image database TID2013: Peculiarities, results and perspectives.  _Signal processing: Image communication_ 30 (2015), 57–77.
  * Saad et al. (2012) Michele A Saad, Alan C Bovik, and Christophe Charrier. 2012.  Blind image quality assessment: A natural scene statistics approach in the DCT domain.  _IEEE transactions on Image Processing_ 21, 8 (2012), 3339–3352.
  * Sandler et al. (2018) Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018.  Mobilenetv2: Inverted residuals and linear bottlenecks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4510–4520.
  * Sheikh et al. (2006a) H.R. Sheikh, M.F. Sabir, and A.C. Bovik. 2006a.  A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms.  _IEEE Transactions on Image Processing_ 15, 11 (2006), 3440–3451.  https://doi.org/10.1109/TIP.2006.881959
  * Sheikh et al. (2006b) Hamid R Sheikh, Muhammad F Sabir, and Alan C Bovik. 2006b.  A statistical evaluation of recent full reference image quality assessment algorithms.  _IEEE Transactions on image processing_ 15, 11 (2006), 3440–3451.
  * Sheng et al. (2018) K. Sheng, W. Dong, C. Ma, X. Mei, F. Huang, and B.-G. Hu. 2018.  Attention-based multi-patch aggregation for image aesthetic assessment. In _Proceedings of the 26th ACM international conference on Multimedia_. ACM, 879–886.
  * Shu et al. (2021) Y. Shu, Q. Li, L. Liu, and G. Xu. 2021.  Semi-supervised adversarial learning for attribute-aware photo aesthetic assessment.  _IEEE Transactions on Multimedia_ (2021).
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * Song et al. (2022) Tianshu Song, Leida Li, Pengfei Chen, Hantao Liu, and Jiansheng Qian. 2022.  Blind image quality assessment for authentic distortions by intermediary enhancement and iterative training.  _IEEE Transactions on Circuits and Systems for Video Technology_ 32, 11 (2022), 7592–7604.
  * Su et al. (2020) Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang. 2020.  Blindly assess image quality in the wild guided by a self-adaptive hyper network. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3667–3676.
  * Sullivan et al. (2012) Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. 2012.  Overview of the high efficiency video coding (HEVC) standard.  _IEEE Transactions on circuits and systems for video technology_ 22, 12 (2012), 1649–1668.
  * Sun et al. (2023) Wei Sun, Xiongkuo Min, Danyang Tu, Siwei Ma, and Guangtao Zhai. 2023.  Blind quality assessment for in-the-wild images via hierarchical feature fusion and iterative mixed database training.  _IEEE Journal of Selected Topics in Signal Processing_ (2023).
  * Sun et al. (2009) X. Sun, H. Yao, R. Ji, and S. Liu. 2009.  Photo assessment based on computational visual attention model. In _Proceedings of the 17th ACM international conference on Multimedia_. ACM, 541–544.
  * Talebi and Milanfar (2018) H. Talebi and P. Milanfar. 2018.  Nima: Neural image assessment.  _IEEE Transactions on Image Processing_ 27, 8 (2018), 3998–4011.
  * Voorhees et al. (2012) Gerald A Voorhees, Joshua Call, and Katie Whitlock. 2012.  _Guns, grenades, and grunts: First-person shooter games_.  Bloomsbury Publishing USA.
  * Wang et al. (2016) Shiqi Wang, Ke Gu, Xiang Zhang, Weisi Lin, Li Zhang, Siwei Ma, and Wen Gao. 2016.  Subjective and Objective Quality Assessment of Compressed Screen Content Images.  _IEEE Journal on Emerging and Selected Topics in Circuits and Systems_ 6, 4 (2016), 532–543.  https://doi.org/10.1109/JETCAS.2016.2598756
  * Wang et al. (2021) Tao Wang, Wei Sun, Xiongkuo Min, Wei Lu, Zicheng Zhang, and Guangtao Zhai. 2021.  A Multi-dimensional Aesthetic Quality Assessment Model for Mobile Game Images. In _2021 International Conference on Visual Communications and Image Processing (VCIP)_. IEEE, 1–5.
  * Wang et al. ([n. d.]) Tao Wang, Wei Sun, Wei Wu, Ying Chen, Xiongkuo Min, Wei Lu, Zicheng Zhang, and Guangtao Zhai. [n. d.].  A Deep Learning Based Multi-Dimensional Aesthetic Quality Assessment Method for Mobile Game Images.  _IEEE Transactions on Games_ ([n. d.]).  https://doi.org/10.1109/tg.2022.3212201
  * Wen et al. (2021) Shaoguo Wen, Suiyi Ling, Junle Wang, Ximing Chen, Lizhi Fang, Yanqing Jing, and Patrick Le Callet. 2021.  Subjective and Objective Quality Assessment of Mobile Gaming Video.  _arXiv preprint arXiv:2103.05099_ (2021).
  * Woo et al. (2018) Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. 2018.  Cbam: Convolutional block attention module. In _Proceedings of the European conference on computer vision (ECCV)_. 3–19.
  * Wu et al. (2023) Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. 2023.  Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives.  arXiv:2211.04894 [cs.CV]
  * Xu et al. (2015) Jizheng Xu, Rajan Joshi, and Robert A Cohen. 2015.  Overview of the emerging HEVC screen content coding extension.  _IEEE Transactions on Circuits and Systems for Video Technology_ 26, 1 (2015), 50–62.
  * Yang et al. (2015) Huan Yang, Yuming Fang, and Weisi Lin. 2015.  Perceptual Quality Assessment of Screen Content Images.  _IEEE Transactions on Image Processing_ 24, 11 (2015), 4408–4421.  https://doi.org/10.1109/TIP.2015.2465145
  * Yang et al. (2019) S. Yang, Q. Jiang, W. Lin, and Y. Wang. 2019.  Sgdnet: An end-to-end saliency-guided deep neural network for no-reference image quality assessment. In _Proceedings of the 27th ACM International Conference on Multimedia_. ACM, 1383–1391.
  * Ying et al. (2021) Zhenqiang Ying, Maniratnam Mandal, Deepti Ghadiyaram, and Alan Bovik. 2021.  Patch-VQ: ‘Patching Up’ the Video Quality Problem. In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE.
  * You and Yan (2022) Junyong You and Jie Yan. 2022.  Explore Spatial and Channel Attention in Image Quality Assessment. In _2022 IEEE International Conference on Image Processing (ICIP)_. IEEE, 26–30.
  * Yu et al. (2022) Xiangxu Yu, Zhengzhong Tu, Zhenqiang Ying, Alan C Bovik, Neil Birkbeck, Yilin Wang, and Balu Adsumilli. 2022.  Subjective Quality Assessment of User-Generated Content Gaming Videos. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 74–83.
  * Zadtootaghaj et al. (2020) Saman Zadtootaghaj, Steven Schmidt, Saeed Shafiee Sabet, Sebastian Möller, and Carsten Griwodz. 2020.  Quality estimation models for gaming video streaming services using perceptual video quality dimensions. In _Proceedings of the 11th ACM Multimedia Systems Conference_. 213–224.
  * Zhai and Min (2020) Guangtao Zhai and Xiongkuo Min. 2020.  Perceptual image quality assessment: a survey.  _Science China Information Sciences_ 63, 11 (2020), 1–52.
  * Zhang et al. (2014) L. Zhang, Y. Gao, R. Zimmermann, Q. Tian, and X. Li. 2014.  Fusion of multichannel local and global structural cues for photo aesthetics evaluation.  _IEEE Transactions on Image Processing_ 23, 3 (2014), 1419–1429.
  * Zhang et al. (2018) Weixia Zhang, Kede Ma, Jia Yan, Dexiang Deng, and Zhou Wang. 2018.  Blind image quality assessment using a deep bilinear convolutional neural network.  _IEEE Transactions on Circuits and Systems for Video Technology_ 30, 1 (2018), 36–47.
  * Zhang et al. (2019) X. Zhang, X. Gao, W. Lu, and L. He. 2019.  A gated peripheral-foveal convolutional neural network for unified image aesthetic prediction.  _IEEE Transactions on Multimedia_ 21, 11 (2019), 2815–2826.
  * Zhang et al. (2021a) Zicheng Zhang, Wei Sun, Xiongkuo Min, Tao Wang, Wei Lu, and Guangtao Zhai. 2021a.  No-Reference Quality Assessment for 3D Colored Point Cloud and Mesh Models.  _IEEE Transactions on Circuits and Systems for Video Technology_ (2021).
  * Zhang et al. (2021b) Zicheng Zhang, Wei Sun, Xiongkuo Min, Tao Wang, Wei Lu, Wenhan Zhu, and Guangtao Zhai. 2021b.  A no-reference visual quality metric for 3d color meshes. In _IEEE International Conference on Multimedia & Expo Workshops_. IEEE, 1–6.
  * Zhang et al. (2022) Zicheng Zhang, Wei Sun, Xiongkuo Min, Wei Wu, Ying Chen, and Guangtao Zhai. 2022.  Treating Point Cloud as Moving Camera Videos: A No-Reference Quality Assessment Metric.  _arXiv preprint arXiv:2208.14085_ (2022).
  * Zhang et al. (2023a) Zicheng Zhang, Wei Sun, Xiongkuo Min, Quan Zhou, Jun He, Qiyuan Wang, and Guangtao Zhai. 2023a.  MM-PCQA: Multi-modal learning for no-reference point cloud quality assessment.  _International Joint Conference on Artificial Intelligence_ (2023).
  * Zhang et al. (2021c) Zicheng Zhang, Wei Sun, Xiongkuo Min, Wenhan Zhu, Tao Wang, Wei Lu, and Guangtao Zhai. 2021c.  A No-Reference Evaluation Metric for Low-Light Image Enhancement. In _IEEE International Conference on Multimedia & Expo_. 1–6.
  * Zhang et al. (2023b) Zicheng Zhang, Wei Wu, Wei Sun, Dangyang Tu, Wei Lu, Xiongkuo Min, Ying Chen, and Guangtao Zhai. 2023b.  MD-VQA: Multi-Dimensional Quality Assessment for UGC Live Videos.  _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2023).