# Multi-pooled Inception features for no-reference image quality assessment

Domonkos Varga

###### Abstract

Image quality assessment (IQA) is an important element of a broad spectrum of
applications ranging from automatic video streaming to display technology.
Furthermore, the measurement of image quality requires a balanced
investigation of image content and features. Our proposed approach extracts
visual features by attaching global average pooling (GAP) layers to multiple
Inception modules of on an ImageNet database pretrained convolutional neural
network (CNN). In contrast to previous methods, we do not take patches from
the input image. Instead, the input image is treated as a whole and is run
through a pretrained CNN body to extract resolution-independent, multi-level
deep features. As a consequence, our method can be easily generalized to any
input image size and pretrained CNNs. Thus, we present a detailed parameter
study with respect to the CNN base architectures and the effectiveness of
different deep features. We demonstrate that our best proposal — called
MultiGAP-NRIQA — is able to provide state-of-the-art results on three
benchmark IQA databases. Furthermore, these results were also confirmed in a
cross database test using the LIVE In the Wild Image Quality Challenge
database.

##  1 Introduction

With the increasing popularity of imaging devices as well as the rapid spread
of social media and multimedia sharing websites, digital images and videos
have become an essential part of daily life, especially in everyday
communication. Consequently, there is a growing need for effective systems
that are able to monitor the quality of visual signals. Obviously, the most
reliable way of assessing image quality is to perform subjective user studies,
which involves the gathering of individual quality scores. However, the
compilation and evaluation of a subjective user study are very slow and
laborious processes. Furthermore, their application in a real-time system is
impossible. In contrast, objective image quality assessment (IQA) involves the
development of quantitative measures and algorithms for estimating image
quality.

Objective IQA is classified based on the availability of the reference image.
Full-reference image quality assessment (FR-IQA) methods have full access to
the reference image, whereas no-reference image quality assessment (NR-IQA)
algorithms possess only the distorted digital image. In contrast, reduced-
reference image quality assessment (RR-IQA) methods have partial information
about the reference image; for example, as a set of extracted features.
Objective IQA algorithms are evaluated on benchmark databases containing the
distorted images and their corresponding mean opinion scores (MOSs), which
were collected during subjective user studies. The MOS is a real number,
typically in the range 1.0–5.0, where 1.0 represents the lowest quality and
5.0 denotes the best quality. Furthermore, the MOS of an image is its
arithmetic mean over all collected individual quality ratings. As already
mentioned, publicly available IQA databases help researchers to devise and
evaluate IQA algorithms and metrics. Existing IQA datasets can be grouped into
two categories with respect to the introduced image distortion types. The
first category contains images with artificial distortions, while the images
of the second category are taken from sources with "natural" degradation
without any additional artificial distortions.

The rest of this section is organized as follows. In Subsection 1.1, we review
related work in NR-IQA with a special attention on deep learning based
methods. Subsection 1.2 introduces the contributions made in this study.

###  1.1 Related work

Many traditional NR-IQA algorithms rely on the so-called natural scene
statistics (NSS) [1] model. These methods assume that natural images possess a
particular regularity that is modified by visual distortion. Further, by
quantifying the deviation from the natural statistics, perceptual image
quality can be determined. NSS-based feature vectors usually rely on the
wavelet transform [2], discrete cosine transform [3], curvelet transform [4],
shearlet transform [5], or transforms to other spatial domains [6]. DIIVINE
[2] (Distortion Identification-based Image Verity and INtegrity Evaluation)
exploits NSS using wavelet transform and consists of two steps. Namely, a
probabilistic distortion identification stage is followed by a distortion-
specific quality assessment one. In contrast, He et al. [7] presented a sparse
feature representation of NSS using also the wavelet transform. Saad et al.
[3] built a feature vector from DCT coefficients. Subsequently, a Bayesian
inference approach was applied for the prediction of perceptual quality
scores. In [8], the authors presented a detailed review about the use of local
binary pattern texture descriptors in NR-IQA.

Another line of work focuses on opinion-unaware algorithms that require
neither training samples nor human subjective scores. Zhang et al. [9]
introduced the integrated local natural image quality evaluator (IL-NIQE),
which combines features of NSS with multivariate Gaussian models of image
patches. This evaluator uses several quality-aware NSS features, i.e., the
statistics of normalized luminance, mean subtracted and contrast-normalized
products of pairs of adjacent coefficients, gradient, log-Gabor filter
responses, and color (after the transformation into a logarithmic-scale
opponent color space).

Kim et al. [10] introduced a no-reference image quality predictor called the
blind image evaluator based on a convolutional neural network (BIECON), in
which the training process is carried out in two steps. First, local metric
score regression and then subjective score regression are conducted. During
the local metric score regression, non-overlapping image patches are trained
independently; FR-IQA methods such as SSIM or GMS are used for the target
patches. Then, the CNN trained on image patches is refined by targeting the
subjective image score of the complete image. Similarly, the training of a
multi-task end-to-end optimized deep neural network [11] is carried out in two
steps. Namely, this architecture contains two sub-networks: a distortion
identification network and a quality prediction network. Furthermore, a
biologically inspired generalized divisive normalization [12] is applied as
the activation function in the network instead of rectified linear units
(ReLUs). Similarly, Fan et al. [13] introduced a two-stage framework. First, a
distortion type classifier identifies the distortion type then a fusion
algorithm is applied to aggregate the results of expert networks and produce a
perceptual quality score.

In recent years, many algorithms relying on deep learning have been proposed.
Because of the small size of many existing image quality benchmark databases,
most deep learning based methods employ CNNs as feature extractors or take
patches from the training images to increase the database size. The CNN
framework of Kang et al. [14] is trained on non-overlapping image patches
extracted from the training images. Furthermore, these patches inherit the MOS
of their source images. For preprocessing, local contrast normalization is
employed. The applied CNN consists of conventional building blocks, such as
convolutional, pooling, and fully connected layers. Bosse et al. [15]
introduced a similar method. Namely, they developed a 12-layer CNN that is
trained on $32\times 32$ image patches. Furthermore, a weighted average patch
aggregation method was introduced in which weights representing the relative
importance of image patches in quality assessment are learned by a subnetwork.
In contrast, Li et al. [16] combined a CNN trained on image patches with the
Prewitt magnitudes of segmented images to predict perceptual quality.

Li et al. [17] trained a CNN on $32\times 32$ image patches and employed it as
a feature extractor. In this method, a feature vector of length $800$
represents each image patch of an input image and the sum of image patches’
feature vectors is associated with the original input image. Finally, a
support vector regressor (SVR) is trained to evaluate the image quality using
the feature vector representing the input image. In contrast, Bianco et al.
[18] utilized a fine-tuned AlexNet [19] as a feature extractor on the target
database. Specifically, image quality is predicted by averaging the quality
ratings on multiple randomly sampled image patches. Further, the perceptual
quality of each patch is predicted by an SVR trained on deep features
extracted with the help of a fine-tuned AlexNet [19]. Similarly, Gao et al.
[20] employed a pretrained CNN as a feature extractor, but they generate one
feature vector for each CNN layer. Furthermore, a quality score is predicted
for each feature vector using an SVR. Finally, the overall perceptual quality
of the image is determined by averaging these quality scores. In contrast,
Zhang et al. [21] trained first a CNN to identify image distortion types and
levels. Furthermore, the authors took another CNN, that was trained on
ImageNet, to deal with authentic distortions. To predict perceptual image
quality, the features of the last convolutional layers were pooled bi-linearly
and mapped onto perceptual quality scores with a fully-connected layer. He et
al. [22] proposed a method containing two steps. In the first step, a sequence
of image patches is created from the input image. Subsequently, features are
extracted with the help of a CNN and a long short-term memory (LSTM) is
utilized to evaluate the level of image distortion. In the second stage, the
model is trained to predict the patches’ quality score. Finally, a saliency
weighted procedure is applied to determine the whole image’s quality from the
patch-wise scores. Similarly, Ji et al. [23] utilized a CNN and an LSTM for
NR-IQA, but the deep features were extracted from the convolutional layers of
a VGG16 [24] network. In contrast to other algorithms, Zhang et al. [25]
proposed an opinion-unaware deep method. Namely, high-contrast image patches
were selected using deep convolutional maps from pristine images which were
used to train a multi-variate Gaussian model.

###  1.2 Contributions

Convolutional neural networks (CNNs) have demonstrated great success in a wide
range of computer vision tasks [26], [27], [28], including NR-IQA [14], [15],
[16], [29]. Furthermore, pretrained CNNs can also provide a useful feature
representation for a variety of tasks [30]. In contrast, employing pretrained
CNNs is not straightforward. One major challenge is that CNNs require a fixed
input size. To overcome this constraint, previous methods for NR-IQA [14],
[15], [16], [18] take patches from the input image. Furthermore, the
evaluation of perceptual quality was based on these image patches or on the
features extracted from them. In this paper, we make the following
contributions. We introduce a unified and content-preserving architecture that
relies on the Inception modules of pretrained CNNs, such as GoogLeNet [31] or
Inception-V3 [32]. Specifically, this novel architecture applies visual
features extracted from multiple Inception modules of pretrained CNNs and
pooled by global average pooling (GAP) layers. In this manner, we obtain both
intermediate-level and high-level representation from CNNs and each level of
representation is considered to predict image quality. Due to this
architecture, we do not take patches from the input image like previous
methods [14], [15], [16], [18]. Unlike previous deep architectures [22], [15],
[18] we do not utilize only the deep features of the last layer of a
pretrained CNN. Instead, we carefully examine the effect of different features
extracted from different layers on the prediction performance and we point out
that the combination of deep features from mid- and high-level layers results
in significant prediction performance increase. With experiments on three
publicly available benchmark databases, we demonstrate that the proposed
method is able to outperform other state-of-the-art methods. Specifically, we
utilized KonIQ-10k [33], KADID-10k [34], and LIVE In the Wild Image Quality
Challenge Database [35] databases. KonIQ-10k [33] is the largest publicly
available database containing 10,073 images with authentic distortions, while
KADID-10k [34] consists of 81 reference images and 10,125 distorted ones (81
reference images $\times$ 25 types of distortions $\times$ 5 levels of
distortions). LIVE In the Wild Image Quality Challenge Database [35] is
significantly smaller than KonIQ-10k [33] or KADID-10k [34]. For a cross
database test, also the LIVE In the Wild Image Quality Challenge Database [35]
is applied which contains $1,162$ images with authentic distortions evaluated
by over $8,100$ unique human observers.

The remainder of this paper is organized as follows. After this introduction,
Section 2 introduces our proposed approach. In Section 3, the experimental
results and analysis are presented, and a conclusion is drawn in Section 4.

##  2 Methodology

To extract visual features, GoogLeNet [31] or Inception-V3 [32] were applied
as base models. GoogLeNet [31] is a 22 layer deep CNN and was the winner of
ILSVRC 2014 with a top 5 error rate of 6.7 %. Depth and width of the network
was increased but not simply following the general method of stacking the
layers on each other. A new level of organization was introduced codenamed
Inception module (see Figure 1). In GoogLeNet [31] not everything happens
sequentially like in previous CNN models, pieces of the network work in
parallel. Inspired by a neuroscience model in [36] where for handling multiple
scales a series of Gabor filters were used with a two layer deep model. But
contrary to the beforementioned model all layers are learned and not fixed. In
GoogLeNet [31] architecture Inception layers are introduced and repeated many
times. Subsequent improvements of GoogLeNet [31] have been called
Inception-v$N$ where $N$ refers to the version number put out by Google.
Inception-V2 [32] was refined by the introduction of batch normalization [37].
Inception-V3 [32] was improved by factorization ideas. Factorization into
smaller convolutions means for example replacing a $5\times 5$ convolution by
a multi-layer network with fewer parameters but with the same input size and
output depth.

We chose the features of Inception modules for the following reasons. The main
motivation behind the construction of Inception modules is that salient parts
of images may very extremely. This means that the region of interest can
occupy very different image regions both in terms of size and location. That
is why, determining the convolutional kernel size in a CNN is very difficult.
Namely, a larger kernel size is required for visual information that is
distributed rather globally. On the other hand, a smaller kernel size is
better for visual information that is distributed more locally. As already
mentioned, the creators of Inception modules reflected to this challenge by
the introduction of multiple filters with multiple sizes on the same level.
Furthermore, visual distortions have a similar nature. Namely, the distortion
distribution is strongly influenced by image content [38].

Figure 1: Illustration of Inception module. It was restricted to filter sizes
$1\times 1$, $3\times 3$, and $5\times 5$. Subsequently, the outputs were
concatenated into a single vector that is the input for the next stage. Adding
of an alternative parallel pooling path was found to be beneficial. Applying
filters of $1\times 1$ convolution makes possible to reduce the volume before
the expensive $3\times 3$ and $5\times 5$ convolutions [31].  Figure 2: The
pipeline of the proposed method. An input image is run through on an ImageNet
database pretrained CNN body (GoogLeNet and Inception-V3 are considered in
this study) which carries out all its defined operations. Furthermore, global
average pooling (GAP) layers are attached to each Inception module to extract
resolution independent deep features at different abstraction levels. The
feature vectors obtained from the Inception modules are concatenated and an
SVR with radial basis function is applied to predict perceptual image quality.

###  2.1 Pipeline of the proposed method

The pipeline of the proposed framework is depicted in Figure 2. A given input
image to be evaluated is run through a pretrained CNN body (GoogLeNet [31] and
Inception-V3 [32] are considered in this study) which carries out all its
defined operations. Specifically, global average pooling (GAP) layers are
attached to the output of each Inception module. Similar to max- or min-
pooling layers, GAP layers are applied in CNNs to reduce the spatial
dimensions of convolutional layers. However, a GAP layer carries out a more
extreme type of dimensional reduction than a max- or min-pooling layer.
Namely, an $h\times w\times d$ block is reduced to $1\times 1\times d$. In
other words, a GAP layer reduces a feature map to a single value by taking the
average of this feature map. By adding GAP layers to each Inception module, we
are able to extract resolution independent features at different levels of
abstraction. Namely, the feature maps produced by neuroscience models inspired
[36] Inception modules have been shown representative for object categories
[31], [32] and correlate well with human perceptual quality judgments [39].
The motivation behind the application of GAP layers was the followings. By
attaching GAP layers to the Inception modules, we gain an architecture which
can be easily generalized to any input image resolution and base CNN
architecture. Furthermore, this way the decomposition of the input image into
smaller patches can be avoided which means that parameter settings related to
the database properties (patch size, number of patches, sampling strategy,
etc.) can be ignored. Moreover, some kind of image distortions are not
uniformly distributed in the image. These kind of distortions could be better
captured in an aspect-ratio and content preserving architecture.

As already mentioned, a feature vector is extracted over each Inception module
using a GAP layer. Let $\textbf{f}_{k}$ denote the feature vector extracted
from the $k$th Inception module. The input image’s feature vector is obtained
by concatenating the respective feature vectors produced by the Inception
modules. Formally, we can write
$\textbf{F}=\textbf{f}_{1}\oplus\textbf{f}_{2}\oplus...\oplus\textbf{f}_{N}$,
where $N$ denotes the number of Inception modules in the base CNN and $\oplus$
stands for the concatenation operator. In Section 3.3, we present a detailed
analysis about the effectiveness of different Inception modules’ deep features
as a perceptual metric. Furthermore, we point out the prediction performance
increase due to the concatenation of deep features extracted from different
abstraction levels.

Subsequently, an SVR [40] with radial basis function (RBF) kernel is trained
to learn the mapping between feature vectors and corresponding perceptual
quality scores.

Moreover, we also applied Gaussian process regression (GPR) with rational
quadratic kernel function [41] in Section 3.4.

###  2.2 Database compilation and transfer learning

Many image quality assessment databases are available online, such as TID2013
[42] or LIVE In the Wild [35], for research purposes. In this study, we
selected the recently published KonIQ-10k [33] database to train and test our
system, because it is the largest available database containing digital images
with authentic distortions. Furthermore, we present a parameter study on
KonIQ-10k [33] to find the best design choices. Our best proposal is compared
against the state-of-the-art on KonIQ-10k [33] and also on other publicly
available databases.

Figure 3: MOS distribution in KonIQ-10k database. It contains 10,073 RGB
images with authentic distortions and the corresponding MOS values which are
on a scale from 1.0 (worst image quality) to 5.0 (best image quality).

KonIQ-10k [33] consists of 10,073 digital images with the corresponding MOS
values. To ensure the fairness of the experimental setup, we selected randomly
6,073 images $(\sim 60\%)$ for training, 2,000 images $(\sim 20\%)$ for
validation, and 2,000 images $(\sim 20\%)$ for testing purposes. First, the
base CNN was fine-tuned on target database KonIQ-10k [33] using the above-
mentioned training and the validation subsets. To this end, regularly the base
CNN’s last 1,000-way softmax layer was removed and replaced by a 5-way one in
previous methods [18], because the training and validation subsets were
reorganized into five classes with respect to the MOS values: class $A$ for
excellent image quality $(5.0>MOS\geq 4.2)$, class $B$ for good image quality
$(4.2>MOS\geq 3.4)$, class $C$ for fair image quality $(3.4>MOS\geq 2.6)$,
class $D$ for poor image quality $(2.6>MOS\geq 1.8)$, and class $E$ for very
poor image quality $(1.8>MOS\geq 1.0)$. Subsequently, the base CNN was further
train to classify the images into quality categories. Since the MOS
distribution in KonIQ-10k [33] is strongly imbalanced (see Figure 3), there
would be very little number of images in the class for excellent images. That
is why, we took a regression-based approach instead of classification-based
approach for fine-tuning. Namely, we removed the base CNN’s last 1,000-way
softmax layer and we replaced it by a regression layer containing only one
neuron. Since GoogLeNet [31] and Inception-V3 [32] accept images with input
size of $224\times 224$ and $299\times 299$, respectively, twenty $224\times
224$-sized or $299\times 299$-sized patches were cropped randomly from each
training and validation images. Furthermore, these patches inherit the
perceptual quality score of their source images and the fine-tuning is carried
out on these patches. Specifically, we trained the base CNN further for
regression to predict the images patches MOS values which are inherited from
their source images. During fine-tuning Adam optimizer [43] was used, the
initial learning rate was set to 0.0001 and divided by 10 when the validation
error stopped improving. Further, the batch size was set to 28 and the
momentum was 0.9 during fine-tuning.

##  3 Experimental results and analysis

In this section, we demonstrate our experimental results. First, we give the
definition of the evaluation metrics in Section 3.1. Second, we describe the
experimental setup and the implementation details in Section 3.2. In Section
3.3, we give a detailed parameter study to find the best design choices of the
proposed method using KonIQ-10k [33] database. Subsequently, we carry out a
comparison to other state-of-the-art methods using KonIQ-10k [33], KADID-10k
[34], and LIVE In the Wild [35] publicly available IQA databases. Finally, we
present a so-called cross database test using LIVE In the Wild Image Quality
Challenge database [35].

###  3.1 Evaluation metrics

The performance of NR-IQA algorithms are characterized by the correlation
calculated between the ground-truth scores of a benchmark database and the
predicted scores. To this end, Pearson’s linear correlation coefficient (PLCC)
and Spearman’s rank order correlation coefficient (SROCC) are widely used in
the literature [44]. PLCC between datasets $A$ and $B$ is defined as

$PLCC(A,B)=\frac{\sum_{i=1}^{m}(A_{i}-\bar{A})(B_{i}-\bar{B})}{\sqrt{\sum_{i=1}^{m}(A_{i}-\bar{A})^{2}}\sqrt{\sum_{i=1}^{m}(B_{i}-\bar{B})^{2}}},$
(1)

where $\bar{A}$ and $\bar{B}$ denote the average of sets $A$ and $B$, and
$A_{i}$ and $B_{i}$ denote the $i$th elements of sets $A$ and $B$,
respectively. SROCC, it can be expressed as

$SROCC(A,B)=\frac{\sum_{i=1}^{m}(A_{i}-\hat{A})(B_{i}-\hat{B})}{\sqrt{\sum_{i=1}^{m}(A_{i}-\hat{A})^{2}}\sqrt{\sum_{i=1}^{m}(B_{i}-\hat{B})^{2}}},$
(2)

where $\hat{A}$ and $\hat{B}$ stand for the middle ranks.

###  3.2 Experimental setup and implementation details

As already mentioned, a detailed parameter study was carried out on the
recently published KonIQ-10k [33], which is the currently largest available
IQA database with authentic distortions, to determine the optimal design
choices. Subsequently, our best proposal is compared to the state-of-the-art
using other publicly available databases as well.

The proposed method was implemented in MATLAB R2019a mainly relying on the
functions of the Deep Learning Toolbox (formerly Neural Network Toolbox),
Image Processing Toolbox, and Statistics and Machine Learning Toolbox. Thus,
the parameter study was also carried out in MATLAB environment. More
specifically, it was evaluated by 100 random train-validation-test split of
the applied database and we report on the average of the PLCC and SROCC
values. As usual in machine learning, $\sim 60\%$ of the images was used for
training, $\sim 20\%$ for validation, and $\sim 20\%$ for testing purposes.
Moreover, for IQA databases containing artificial distortions the splitting of
the database is carried out with respect to the reference images, so no
semantic overlapping was between the training, validation, and test sets.
Further, the models were trained and tested on a personal computer with 8-core
i7-7700K CPU two NVidia Geforce GTX 1080 GPUs.

###  3.3 Parameter study

First, we conducted experiments to determine which Inception module in
GoogLeNet [31] or in Inception-V3 [32] is the most appropriate for visual
feature extraction to predict perceptual image quality. Second, we answer the
question whether the concatenation of different Inception modules’ feature
vectors improves the prediction’s performance or not. Third, we demonstrate
that fine-tuning of the base CNN architecture results in significant
performance increase. In this parameter study, we used KonIQ-10k database to
answer the above mentioned questions and to find the most effective design
choices. In the next subsection, our best proposal is used to carry out a
comparison to the state-of-the-art using other databases as well.

The results of the parameter study are summarized in Tables 1, 2, 3, and 4.
Specifically, Table 1 and 3 contains the results with GoogLeNet [31] and
Inception-V3 [32] base architectures without fine-tuning, respectively. On the
other hand, Table 2 and 4 summarizes the results when fine-tuning is applied.
In these tables, we reported on the average, the median, and the standard
deviation of the PLCC and SROCC values obtained after 100 random train-
validation-test splits using KonIQ-10k database. Furthermore, we report on the
effectiveness of deep features extracted from different Inception modules.
Moreover, the tables also contain the prediction performance of the
concatenated deep feature vector. From these results, it can be concluded that
the deep features extracted from the early Inception modules perform slightly
poorer than those of intermediate and last Inception modules. Although most
state-of-the-art methods [22], [15], [18] utilize the features of the last CNN
layers, it is worth to examine earlier layers as well, because the tables’
data indicate that the middle layers encode those information which are the
most powerful for perceptual quality prediction. We can also assert that
feature vectors containing both mid-level and high-level deep representations
are significantly more efficient than those of containing only one level’s
feature representation. Finally, it can be clearly seen that fine-tuning the
base CNN architectures also improves the effectiveness of the extracted deep
features. On the whole, the deeper Inception-V3 [32] provides more effective
features than GoogLeNet [31]. Our best proposal relies on Inception-V3 and
concatenates the features of all Inception modules. In the followings, we call
this architecture MultiGAP-NRIQA and compare it to other state-of-the-art in
the next subsection.

Another contribution of this parameter study may be the followings. It is
worth to study the features of different layers separately because the
features of intermediate layers may provide a better representation of the
given task than high-level features. Furthermore, the proposed feature
extraction method may be also superior in other problems where the task is to
predict one value only from the image data itself relying on a large enough
database.

In our environment (MATLAB R2019a, PC with 8-core i7700K CPU and two NVidia
Geforce GTX 1080), the computational times of the proposed MultiGAP-NRIQA
method are the followings. The loading of the base CNN and the $1024\times
768$-sized or the $512\times 384$ input image takes about $1.8s$. Furthermore,
the feature extraction from multiple Inception modules of Inception-V3 [32]
and concatenation takes on average $1.355s$ or $0.976s$ on the GPU,
respectively. Furthermore, the SVR regression takes $2.976s$ on average
computing on the CPU.

Table 1: Performance comparison of deep features extracted from GoogLeNet’s Inception modules without fine-tuning measured on KonIQ-10k. Average/median $(\pm std)$ values are reported over 100 random train-test splits. The best results are typed by bold.  Layer | Dimension | PLCC | SROCC
---|---|---|---
inception_3a-output | 256 | $0.845/0.845(\pm 0.006)$ | $0.842/0.841(\pm 0.007)$
inception_3b-output | 480 | $0.861/0.861(\pm 0.007)$ | $0.856/0.858(\pm 0.007)$
inception_4a-output | 512 | $0.876/0.876(\pm 0.004)$ | $0.872/0.872(\pm 0.006)$
inception_4b-output | 512 | $0.874/0.874(\pm 0.005)$ | $0.865/0.864(\pm 0.008)$
inception_4c-output | 512 | $0.875/0.877(\pm 0.006)$ | $0.865/0.865(\pm 0.006)$
inception_4d-output | 528 | $0.876/0.875(\pm 0.007)$ | $0.864/0.864(\pm 0.007)$
inception_4e-output | 832 | $0.872/0.871(\pm 0.006)$ | $0.861/0.862(\pm 0.005)$
inception_5a-output | 832 | $0.873/0.874(\pm 0.005)$ | $0.859/0.860(\pm 0.005)$
inception_5b-output | 1024 | $0.861/0.861(\pm 0.008)$ | $0.851/0.850(\pm 0.008)$
All concatenated | 5488 | $\textbf{0.889/0.889}(\pm 0.007)$ | $\textbf{0.879/0.877}(\pm 0.006)$
Table 2: Performance comparison of deep features extracted from GoogLeNet’s Inception modules with fine-tuning measured on KonIQ-10k. Average/median $(\pm std)$ values are reported over 100 random train-validation-test splits. The best results are typed by bold.  Layer | Dimension | PLCC | SROCC
---|---|---|---
inception_3a-output | 256 | $0.850/0.849(\pm 0.007)$ | $0.846/0.846(\pm 0.007)$
inception_3b-output | 480 | $0.866/0.866(\pm 0.006)$ | $0.861/0.862(\pm 0.007)$
inception_4a-output | 512 | $0.881/0.881(\pm 0.005)$ | $0.877/0.876(\pm 0.006)$
inception_4b-output | 512 | $0.877/0.876(\pm 0.005)$ | $0.870/0.870(\pm 0.006)$
inception_4c-output | 512 | $0.879/0.880(\pm 0.005)$ | $0.869/0.868(\pm 0.005)$
inception_4d-output | 528 | $0.880/0.880(\pm 0.006)$ | $0.869/0.868(\pm 0.005)$
inception_4e-output | 832 | $0.877/0.877(\pm 0.005)$ | $0.867/0.867(\pm 0.007)$
inception_5a-output | 832 | $0.878/0.878(\pm 0.007)$ | $0.864/0.864(\pm 0.007)$
inception_5b-output | 1024 | $0.865/0.865(\pm 0.007)$ | $0.856/0.856(\pm 0.008)$
All concatenated | 5488 | $\textbf{0.894/0.894}(\pm 0.006)$ | $\textbf{0.884/0.884}(\pm 0.006)$
Table 3: Performance comparison of deep features extracted from Inception-V3’s Inception modules without fine-tuning measured on KonIQ-10k. Average/median $(\pm std)$ values are reported over 100 random train-test splits. The best results are typed by bold.  Layer | Dimension | PLCC | SROCC
---|---|---|---
mixed0 | 256 | $0.843/0.843(\pm 0.006)$ | $0.839/0.839(\pm 0.006)$
mixed1 | 288 | $0.848/0.848(\pm 0.005)$ | $0.844/0.844(\pm 0.005)$
mixed2 | 288 | $0.849/0.849(\pm 0.006)$ | $0.844/0.844(\pm 0.007)$
mixed3 | 768 | $0.861/0.860(\pm 0.005)$ | $0.858/0.855(\pm 0.006)$
mixed4 | 768 | $0.897/0.897(\pm 0.005)$ | $0.889/0.889(\pm 0.006)$
mixed5 | 768 | $0.906/0.906(\pm 0.004)$ | $0.898/0.898(\pm 0.005)$
mixed6 | 768 | $0.902/0.901(\pm 0.004)$ | $0.890/0.891(\pm 0.006)$
mixed7 | 768 | $0.884/0.884(\pm 0.004)$ | $0.870/0.870(\pm 0.006)$
mixed8 | 1280 | $0.892/0.891(\pm 0.004)$ | $0.879/0.879(\pm 0.006)$
mixed9 | 2048 | $0.871/0.871(\pm 0.005)$ | $0.859/0.859(\pm 0.006)$
mixed10 | 2048 | $0.842/0.844(\pm 0.006)$ | $0.828/0.829(\pm 0.008)$
All concatenated | 10048 | $\textbf{0.910/0.911}(\pm 0.005)$ | $\textbf{0.901/0.901}(\pm 0.005)$
Table 4: Performance comparison of deep features extracted from Inception-V3’s Inception modules with fine-tuning measured on KonIQ-10k. Average/median $(\pm std)$ values are reported over 100 random train-validation-test splits. The best results are typed by bold.  Layer | Dimension | PLCC | SROCC
---|---|---|---
mixed0 | 256 | $0.848/0.848(\pm 0.008)$ | $0.848/0.848(\pm 0.007)$
mixed1 | 288 | $0.853/0.853(\pm 0.007)$ | $0.853/0.853(\pm 0.006)$
mixed2 | 288 | $0.854/0.853(\pm 0.007)$ | $0.853/0.853(\pm 0.006)$
mixed3 | 768 | $0.866/0.865(\pm 0.006)$ | $0.867/0.867(\pm 0.007)$
mixed4 | 768 | $0.902/0.902(\pm 0.007)$ | $0.898/0.897(\pm 0.006)$
mixed5 | 768 | $0.911/0.910(\pm 0.005)$ | $0.908/0.908(\pm 0.006)$
mixed6 | 768 | $0.907/0.906(\pm 0.005)$ | $0.900/0.900(\pm 0.006)$
mixed7 | 768 | $0.889/0.889(\pm 0.005)$ | $0.880/0.880(\pm 0.006)$
mixed8 | 1280 | $0.897/0.897(\pm 0.006)$ | $0.888/0.887(\pm 0.008)$
mixed9 | 2048 | $0.876/0.876(\pm 0.005)$ | $0.869/0.870(\pm 0.007)$
mixed10 | 2048 | $0.847/0.847(\pm 0.005)$ | $0.837/0.836(\pm 0.008)$
All concatenated | 10048 | $\textbf{0.915/0.914}(\pm 0.005)$ | $\textbf{0.911/0.911}(\pm 0.005)$
Table 5: Publicly available IQA databases used in this study. Publicly available IQA databases can be divided into two groups. The first one contains a smaller set of reference images and artificially distorted images are derived from them using different noise types at different intensity levels, while the second one contains images with "natural" degradation without any additional artificial distortions.  Database | Year | Reference images | Test images | Distortion type | Resolution | Subjective score
---|---|---|---|---|---|---
LIVE In the Wild | 2015 | - | 1,162 | authentic | $500\times 500$ | MOS (1-5)
KonIQ-10k | 2018 | - | 10,073 | authentic | $1024\times 768$ | MOS (1-5)
KADID-10k | 2019 | 81 | 10,125 | artificial | $512\times 384$ | MOS (1-5)

###  3.4 Comparison to the state-of-the-art

To compare our proposed method to other state-of-the-art algorithms, we
collected ten traditional learning-based NR-IQA metrics ( DIIVINE [2],
BLIINDS-II [45], BRISQUE [6], CurveletQA [4], SSEQ [46], GRAD-LOG-CP [47],
BMPRI [48], SPF-IQA [49], SCORER [50], ENIQA [51] ), and two opinion-unaware
method (NIQE [52], PIQE [53]) whose original source code are available.
Moreover, we took the results of two recently published deep learning based
NR-IQA algorithms — DeepFL-IQA [54] and MLSP [55] — from their original
publication. On the whole, we compared our proposed method — MultiGAP-NRIQA —
to 12 other state-of-the-art IQA algorithms or metrics. The results can be
seen in Table 6.

To ensure a fair comparison, these traditional and deep methods were trained,
tested, and evaluated exactly the same as our proposed method. Specifically,
$\sim 60\%$ of the images was used for training, $\sim 20\%$ for validation,
and $\sim 20\%$ for testing purposes. If a validation set is not required, the
training set contains $\sim 80\%$ of the images. Moreover, for IQA databases
containing artificial distortions the splitting of the database is carried out
with respect to the reference images, so no semantic overlapping was between
the training, validation, and test sets. To compare our method to the state-
of-the-art, we report on the average PLCC and SROCC values of 100 random
train-validation-test splits of our method and those of other algorithms. As
already mentioned, the results are summarized in Table 6. More specifically,
this table illustrates the measured average PLCC and SROCC on three large
publicly available IQA databases (Table 5 summarizes the major parameters of
the IQA databases used in this paper).

From the results, it can be seen that the proposed significantly outperforms
the state-of-the-art on KonIQ-10k database. Moreover, only the MultiGAP-NRIQA
method is able perform over 0.9 PLCC and SROCC. It can be observed that GPR
with rational quadratic kernel function performs better than SVR with Gaussian
kernel function. Similarly, the proposed method outperforms the state-of-the-
art on LIVE In the Wild IQA database [35] by a large margin. On KADID-10k,
DeepFL-IQA [54] provides the best results by a large margin. The proposed
MultiGAP-GPR gives the third best results.

Table 6: Comparison of MultiGAP-NRIQA with state-of-the-art NR-IQA and FR-IQA algorithms trained and tested on KonIQ-10k, KADID-10k, and TID2013 databases. The average PLCC and SROCC values are reported measured over 100 random train-validation-test split. The best results are shown in bold and the second best results are typed in italic. The results of DeepFL-IQA [54] and MLSP [55] was measured by the authors of [54].  | KonIQ-10k | KADID-10k | LIVE In the Wild
---|---|---|---
Method | PLCC | SROCC | PLCC | SROCC | PLCC | SROCC
DIIVINE [2] | 0.709 | 0.692 | 0.423 | 0.428 | 0.602 | 0.579
BLIINDS-II [45] | 0.571 | 0.575 | 0.548 | 0.530 | 0.450 | 0.419
BRISQUE [6] | 0.702 | 0.676 | 0.383 | 0.386 | 0.503 | 0.487
NIQE [52] | - | - | 0.273 | 0.309 | - | -
CurveletQA [4] | 0.728 | 0.716 | 0.473 | 0.450 | 0.620 | 0.611
SSEQ [46] | 0.584 | 0.573 | 0.453 | 0.433 | 0.469 | 0.429
GRAD-LOG-CP [47] | 0.705 | 0.698 | 0.585 | 0.566 | 0.579 | 0.557
PIQE [53] | 0.206 | 0.245 | 0.289 | 0.237 | 0.171 | 0.108
BMPRI [48] | 0.636 | 0.619 | 0.554 | 0.530 | 0.521 | 0.480
SPF-IQA [49] | 0.759 | 0.740 | 0.717 | 0.708 | 0.592 | 0.563
SCORER [50] | 0.772 | 0.762 | 0.855 | 0.856 | 0.599 | 0.590
ENIQA [51] | 0.758 | 0.744 | 0.634 | 0.636 | 0.578 | 0.554
DeepFL-IQA [54] | 0.887 | 0.877 | 0.938 | 0.936 | 0.842 | 0.814
MLSP [55] | 0.924 | 0.913 | - | - | 0.769 | 0.734
MultiGAP-SVR [56] | 0.915 | 0.911 | 0.799 | 0.795 | 0.841 | 0.813
MultiGAP-GPR [56] | 0.928 | 0.925 | 0.820 | 0.814 | 0.857 | 0.826

###  3.5 Cross database test

To prove the generalization capability of our proposed MultiGAP-NRIQA method,
we carry out a so-called cross database test in this subsection. This means
that our model was trained on the whole KonIQ-10k [33] database and tested on
LIVE In the Wild Image Quality Challenge Database [35]. Moreover, the other
learning-based NR-IQA methods were also tested this way. The results are
summarized in Table 7. From the results, it can be clearly seen that all
learning-based methods performed significantly poorer in the cross database
test than in the previous tests. It should be emphasized that our MultiGAP-
NRIQA method generalized better than the state-of-the-art traditional or deep
learning based algorithms even without fine-tuning. The performance drop
occurs owing to the fact that images are treated slightly differently in each
publicly available IQA database. For example, in LIVE In The Wild [35]
database the images were rescaled. In contrast, the images of KonIQ-10k [33]
were cropped from their original counterparts.

Table 7: Cross database test. The learning-based NR-IQA methods were trained on the whole KonIQ-10k database and tested on LIVE In the Wild database. The measured PLCC and SROCC values are reported. The best results are shown in bold and the second best results are typed in italic. The results of DeepFL-IQA [54] was measured by the authors of [54].  | LIVE In The Wild
---|---
Method | PLCC | SROCC
BLIINDS-II [3] | 0.439 | 0.401
BRISQUE [6] | 0.625 | 0.604
DIIVINE [2] | 0.598 | 0.592
SSEQ [4] | 0.440 | 0.412
DeepFL-IQA [54] | - | 0.704
MLSP [55] | - | -
MultiGAP-NRIQA, RBF SVR (without fine-tuning) | 0.841 | 0.812
MultiGAP-NRIQA, RBF SVR | 0.841 | 0.813
MultiGAP-NRIQA, r.q. GPR (without fine-tuning) | 0.856 | 0.855
MultiGAP-NRIQA, r.q. GPR | 0.857 | 0.856

##  4 Conclusion

In this paper, we introduced a deep framework for NR-IQA which constructs a
feature space relying on multi-level Inception features extracted from
pretrained CNNs via GAP layers. Unlike previous deep methods, the proposed
approach do not take patches from the input image, but instead treat the image
as a whole and extract image resolution independent features. As a result, the
proposed approach can be easily generalized to any input image size and CNN
base architecture. Unlike previous deep methods, we extract multi-level
features from the CNN to incorporate both mid-level and high-level deep
representations into the feature vector. Furthermore, we pointed out in a
detailed parameter study that mid-level features provide significantly more
effective descriptors for NR-IQA. Another important observation was that the
feature vector containing both mid-level and high-level representations
outperforms all feature vectors containing the representation of one level. We
also carried out a comparison to other state-of-the-art methods and our
approach outperformed the state-of-the-art on the largest available benchmark
IQA databases. Moreover, the results were also confirmed in a cross database
test. There are many directions for future research. Specifically, we would
like to improve the fine-tuning process in order to transfer quality-aware
features more effectively into the base CNN. Another direction of future
research could be the generalization of the applied feature extraction method
to other CNN architectures, such as residual networks.

## References

  * [1] Pamela Reinagel and Anthony M Zador.  Natural scene statistics at the centre of gaze.  Network: Computation in Neural Systems, 10(4):341–350, 1999.
  * [2] Anush Krishna Moorthy and Alan Conrad Bovik.  Blind image quality assessment: From natural scene statistics to perceptual quality.  IEEE transactions on Image Processing, 20(12):3350–3364, 2011.
  * [3] Michele A Saad, Alan C Bovik, and Christophe Charrier.  Dct statistics model-based blind image quality assessment.  In 2011 18th IEEE International Conference on Image Processing, pages 3093–3096. IEEE, 2011.
  * [4] Lixiong Liu, Bao Liu, Hua Huang, and Alan Conrad Bovik.  No-reference image quality assessment based on spatial and spectral entropies.  Signal Processing: Image Communication, 29(8):856–863, 2014.
  * [5] Yuming Li, Lai-Man Po, Xuyuan Xu, and Litong Feng.  No-reference image quality assessment using statistical characterization in the shearlet domain.  Signal Processing: Image Communication, 29(7):748–759, 2014.
  * [6] Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik.  No-reference image quality assessment in the spatial domain.  IEEE Transactions on Image Processing, 21(12):4695–4708, 2012.
  * [7] Lihuo He, Dacheng Tao, Xuelong Li, and Xinbo Gao.  Sparse representation for blind image quality assessment.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1146–1153. IEEE, 2012.
  * [8] Pedro Garcia Freitas, Luísa Peixoto Da Eira, Samuel Soares Santos, and Mylene Christine Queiroz de Farias.  On the application lbp texture descriptors and its variants for no-reference image quality assessment.  Journal of Imaging, 4(10):114, 2018.
  * [9] Lin Zhang, Lei Zhang, and Alan C Bovik.  A feature-enriched completely blind image quality evaluator.  IEEE Transactions on Image Processing, 24(8):2579–2591, 2015.
  * [10] Jongyoo Kim and Sanghoon Lee.  Fully deep blind image quality predictor.  IEEE Journal of selected topics in signal processing, 11(1):206–220, 2016.
  * [11] Kede Ma, Wentao Liu, Kai Zhang, Zhengfang Duanmu, Zhou Wang, and Wangmeng Zuo.  End-to-end blind image quality assessment using deep neural networks.  IEEE Transactions on Image Processing, 27(3):1202–1213, 2017.
  * [12] Qiang Li and Zhou Wang.  Reduced-reference image quality assessment using divisive normalization-based image representation.  IEEE journal of selected topics in signal processing, 3(2):202–211, 2009.
  * [13] Chunling Fan, Yun Zhang, Liangbing Feng, and Qingshan Jiang.  No reference image quality assessment based on multi-expert convolutional neural networks.  IEEE Access, 6:8934–8943, 2018.
  * [14] Le Kang, Peng Ye, Yi Li, and David Doermann.  Convolutional neural networks for no-reference image quality assessment.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1733–1740, 2014.
  * [15] Sebastian Bosse, Dominique Maniry, Thomas Wiegand, and Wojciech Samek.  A deep neural network for image quality assessment.  In 2016 IEEE International Conference on Image Processing (ICIP), pages 3773–3777. IEEE, 2016.
  * [16] Jie Li, Lian Zou, Jia Yan, Dexiang Deng, Tao Qu, and Guihui Xie.  No-reference image quality assessment using prewitt magnitude based on convolutional neural networks.  Signal, Image and Video Processing, 10(4):609–616, 2016.
  * [17] Jie Li, Jia Yan, Dexiang Deng, Wenxuan Shi, and Songfeng Deng.  No-reference image quality assessment based on hybrid model.  Signal, Image and Video Processing, 11(6):985–992, 2017.
  * [18] Simone Bianco, Luigi Celona, Paolo Napoletano, and Raimondo Schettini.  On the use of deep learning for blind image quality assessment.  Signal, Image and Video Processing, 12(2):355–362, 2018.
  * [19] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [20] Fei Gao, Jun Yu, Suguo Zhu, Qingming Huang, and Qi Tian.  Blind image quality prediction by exploiting multi-level deep representations.  Pattern Recognition, 81:432–442, 2018.
  * [21] Weixia Zhang, Kede Ma, Jia Yan, Dexiang Deng, and Zhou Wang.  Blind image quality assessment using a deep bilinear convolutional neural network.  IEEE Transactions on Circuits and Systems for Video Technology, 2018\.
  * [22] Lihuo He, Yanzhe Zhong, Wen Lu, and Xinbo Gao.  A visual residual perception optimized network for blind image quality assessment.  IEEE Access, 7:176087–176098, 2019.
  * [23] Weiping Ji, Jinjian Wu, Guangming Shi, Wenfei Wan, and Xuemei Xie.  Blind image quality assessment with semantic information.  Journal of Visual Communication and Image Representation, 58:195–204, 2019.
  * [24] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [25] Zhong Zhang, Hong Wang, Shuang Liu, and Tariq S Durrani.  Deep activation pooling for blind image quality assessment.  Applied Sciences, 8(4):478, 2018.
  * [26] Domonkos Varga.  No-reference video quality assessment based on the temporal pooling of deep features.  Neural Processing Letters, pages 1–14, 2019.
  * [27] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size.  arXiv preprint arXiv:1602.07360, 2016.
  * [28] Albert Gordo, Jon Almazán, Jerome Revaud, and Diane Larlus.  Deep image retrieval: Learning global representations for image search.  In European conference on computer vision, pages 241–257. Springer, 2016.
  * [29] Omar Alaql and Cheng-Chang Lu.  No-reference image quality metric based on multiple deep belief networks.  IET Image Processing, 2019.
  * [30] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson.  Cnn features off-the-shelf: an astounding baseline for recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806–813, 2014.
  * [31] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [32] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  * [33] Hanhe Lin, Vlad Hosu, and Dietmar Saupe.  Koniq-10k: Towards an ecologically valid and large-scale iqa database.  arXiv preprint arXiv:1803.08489, 2018.
  * [34] Hanhe Lin, Vlad Hosu, and Dietmar Saupe.  Kadid-10k: A large-scale artificially distorted iqa database.  In 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), pages 1–3. IEEE, 2019.
  * [35] Deepti Ghadiyaram and Alan C Bovik.  Massive online crowdsourced study of subjective and objective picture quality.  IEEE Transactions on Image Processing, 25(1):372–387, 2015.
  * [36] Thomas Serre, Lior Wolf, Stanley Bileschi, Maximilian Riesenhuber, and Tomaso Poggio.  Robust object recognition with cortex-like mechanisms.  IEEE Transactions on Pattern Analysis & Machine Intelligence, (3):411–426, 2007.
  * [37] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  arXiv preprint arXiv:1502.03167, 2015.
  * [38] Ke Gu, Shiqi Wang, Guangtao Zhai, Weisi Lin, Xiaokang Yang, and Wenjun Zhang.  Analysis of distortion distribution for pooling in image quality prediction.  IEEE Transactions on Broadcasting, 62(2):446–456, 2016.
  * [39] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018.
  * [40] Harris Drucker, Christopher JC Burges, Linda Kaufman, Alex J Smola, and Vladimir Vapnik.  Support vector regression machines.  In Advances in neural information processing systems, pages 155–161, 1997.
  * [41] Carl Edward Rasmussen.  Gaussian processes in machine learning.  In Summer School on Machine Learning, pages 63–71. Springer, 2003\.
  * [42] Nikolay Ponomarenko, Lina Jin, Oleg Ieremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, et al.  Image database tid2013: Peculiarities, results and perspectives.  Signal Processing: Image Communication, 30:57–77, 2015.
  * [43] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [44] Long Xu, Weisi Lin, and C-C Jay Kuo.  Visual quality assessment by machine learning.  Springer, 2015.
  * [45] Michele A Saad, Alan C Bovik, and Christophe Charrier.  Blind image quality assessment: A natural scene statistics approach in the dct domain.  IEEE transactions on Image Processing, 21(8):3339–3352, 2012.
  * [46] Lixiong Liu, Bao Liu, Hua Huang, and Alan Conrad Bovik.  No-reference image quality assessment based on spatial and spectral entropies.  Signal Processing: Image Communication, 29(8):856–863, 2014.
  * [47] Wufeng Xue, Xuanqin Mou, Lei Zhang, Alan C Bovik, and Xiangchu Feng.  Blind image quality assessment using joint statistics of gradient magnitude and laplacian features.  IEEE Transactions on Image Processing, 23(11):4850–4862, 2014.
  * [48] Xiongkuo Min, Guangtao Zhai, Ke Gu, Yutao Liu, and Xiaokang Yang.  Blind image quality estimation via distortion aggravation.  IEEE Transactions on Broadcasting, 64(2):508–517, 2018.
  * [49] Domonkos Varga.  No-reference image quality assessment based on the fusion of statistical and perceptual features.  Journal of Imaging, 6(8):75, 2020.
  * [50] Mariusz Oszust.  Local feature descriptor and derivative filters for blind image quality assessment.  IEEE Signal Processing Letters, 26(2):322–326, 2019.
  * [51] Xiaoqiao Chen, Qingyi Zhang, Manhui Lin, Guangyi Yang, and Chu He.  No-reference color image quality assessment: from entropy to perceptual quality.  EURASIP Journal on Image and Video Processing, 2019(1):77, 2019\.
  * [52] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik.  Making a “completely blind” image quality analyzer.  IEEE Signal processing letters, 20(3):209–212, 2012.
  * [53] N Venkatanath, D Praneeth, Maruthi Chandrasekhar Bh, Sumohana S Channappayya, and Swarup S Medasani.  Blind image quality evaluation using perception based features.  In 2015 Twenty First National Conference on Communications (NCC), pages 1–6. IEEE, 2015.
  * [54] Hanhe Lin, Vlad Hosu, and Dietmar Saupe.  Deepfl-iqa: Weak supervision for deep iqa feature learning.  arXiv preprint arXiv:2001.08113, 2020.
  * [55] Vlad Hosu, Bastian Goldlucke, and Dietmar Saupe.  Effective aesthetics prediction with multi-level spatially pooled features.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9375–9383, 2019.
  * [56] Domonkos Varga.  Multi-pooled inception features for no-reference image quality assessment.  Applied Sciences, 10(6):2186, 2020.


# Discovering changes in birthing narratives during COVID-19

Daphna Spira1 Noreen Mayat1 Caitlin Dreisbach2 Adam Poliak1,2
1Barnard College
2Columbia Data Science Institute
<EMAIL_ADDRESS><EMAIL_ADDRESS>
##  1 Introduction

Social media has the potential to elucidate a deeper level of understanding of
COVID-19’s impacts across wide ranging communities. Recent work leveraged
publicly available data from social media platforms to detect emerging
symptoms from COVID-19 [Santosh et al., 2020] as well as the pandemic’s
impacts on mental health [Biester et al., 2020, Tabak and Purver, 2020,
Thukral et al., 2020, Wolohan, 2020].

With tightened restrictions in hospital settings, COVID-19 has greatly
impacted expecting parents, newborns, and their families [Findeklee and
Morinello, 2020, DeYoung and Mangum, 2021, Gutschow and Davis-Floyd, 2021,
Altman et al., 2021, Vazquez-Vazquez et al., 2021, Mariño-Narvaez et al.,
2021]. However, no existing work has used natural language processing
techniques to analyze impacts of COVID-19 from long-form text written by new
parents explicitly describing their birthing experiences.

We investigate whether, and if so how, birthing narratives written by new
parents on Reddit changed during COVID-19. Our results indicate that the
presence of family members significantly decreased and themes related to
induced labor significantly increased in the narratives during COVID-19. Our
work builds upon recent research that analyze how new parents use Reddit to
describe their birthing experiences [Pilkington and Rominov, 2017, Antoniak et
al., 2019, Pilkington and Bedford-Dyer, 2021].

##  2 Data Collection

Reddit is a social media platform where users can post anonymous submissions
and comments in various subreddits. We use the Pushshift Reddit API
[Baumgartner et al., 2020] to collect all submissions posted to nine
subreddits related to the birthing experience between April 2009 and June
2021.111These nine subreddits are: r/BabyBumps, r/beyondthebump,
r/BirthStories, r/daddit, r/predaddit, r/pregnant, r/Mommit, r/NewParents, and
r/InfertilityBabies. Following ?), we remove all posts that do not include any
of the terms “birth story,” “birth stories,” or “graduat,” guaranteeing our
corpus consists of birthing narratives. We remove all posts that contain less
than $500$ tokens,222We noticed that in recent years, new parents post a
picture of the baby and then describe the birthing narrative in the first
comment. We include these examples in our filtering process. resulting in a
corpus of 4,484 birthing narratives before and 913 during COVID-19.333We
demarcate March 11th, 2020 as the start of COVID-19, the day the WHO declared
a global pandemic.

##  3 Method

### Topic Modeling.

To discover distinct topics across our collection of birthing narratives, we
apply Latent Dirichlet Allocation [Blei et al., 2003], as implemented in
Mallet [McCallum, 2002]. In initial experiments, $k$, the number of topics,
ranges from $5$ to $50$. We choose $k=50$ based on $C_{v}$ coherence [Röder et
al., 2015]. Discovered topics include induced labor, family, breastfeeding,
and the first moments between a new parent and child. To determine whether the
prevalence of these topics changed during COVID-19, we fit topic-specific
Prophet models, an additive regression approach for forecasting time series
data [Taylor and Letham, 2018], on the topic’s average monthly prevalence in
our corpus prior to March 2020. We then compare the topic’s actual average
monthly prevalence in our corpus during COVID-19 with the corresponding
model’s forecast. Following ?), we quantify how often the actual topic’s
monthly probabilities fall outside the model’s 95% CI and use one-tailed
Z-tests to determine statistical significance.

### Quantifying Personas Presence.

Determining the prevalence of types of characters, or personas, in a narrative
can illuminate information from an author’s perspective, e.g. who is most the
relevant, valued, or supportive character. Following ?), we quantify a
persona’s prevalence by counting how often they are mentioned, using a
dictionary of terms for each persona.444 Doctor: [doctor, dr, doc, ob, obgyn,
gynecologist, physician]; Partner: [partner, husband, wife]; Nurse: [nurse];
Midwife: [midwife]; Family: [mom, dad, mother, father, brother, sister];
Anesthesiologist: [anesthesiologist]; Doula: [doula] We examine the difference
in average mentions of each persona before and during COVID-19.555Since
narratives during COVID-19 were on average roughly 20% shortner, we adjust the
counts. We apply t-tests to compute statistical significance.

##  4 Results

Figure 1 shows how the forecasted prevalence’s for the family and induction
topics significantly differ with the topics’ prevalence during COVID-19. The
increase in the induction topic (Figure 1b) may reflect the increased
recommendation of planned induction, enabling COVID-19 testing of expecting
parents in advance of delivery [Goer, 2020]. The decrease in the family topic
(Figure 1a) might correlate with hospitals restricting visitors during the
pandemic.

Our experiments quantifying the personas’ presence demonstrate that the
healthcare providers were mentioned at similar rates before and during
COVID-19, indicating that from the perspective of birthing parents, providers’
roles in birthing narratives remained consistent. We notice a significant drop
in the family persona’s presence (22.8% , $p<1^{-7}$) and significant increase
in the partner persona’s presence (5%, $p<0.028$) during the pandemic. Figure
2 indicates that these differences were usually most apparent from periods 2
to 8 in the stories, which correlates with the time generally spent in the
hospital during birthing narratives [Antoniak et al., 2019]. This might
suggest that partners supplemented the support missing by families that were
unable to visit hospitals before and after delivery.

##  5 Conclusion

We presented a study demonstrating how parents’ self-described birthing
experiences significantly changed during COVID-19. Our results indicate that
hospital policies may be reflected in birthing narratives. Our work presents a
case study in how we can analyze patient experience from their own written
narratives and perspectives.

## References

  * [Altman et al., 2021]  Molly R. Altman, Amelia R. Gavin, Meghan K. Eagen-Torkko, Ira Kantrowitz-Gordon, Rue M. Khosa, and Selina A. Mohammed.  2021\.  Where the system failed: The covid-19 pandemic’s impact on pregnancy and birth care.  Global Qualitative Nursing Research, 8:23333936211006397.  PMID: 33869668.
  * [Antoniak et al., 2019]  Maria Antoniak, David Mimno, and Karen Levy.  2019\.  Narrative paths and negotiation of power in birth stories.  Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1–27.
  * [Baumgartner et al., 2020]  Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and J. Blackburn.  2020\.  The pushshift reddit dataset.  In ICWSM.
  * [Biester et al., 2020]  Laura Biester, Katie Matton, Janarthanan Rajendran, Emily Mower Provost, and Rada Mihalcea.  2020\.  Quantifying the effects of covid-19 on mental health support forums.  Computer Science & Engineering, University of Michigan, sept.
  * [Blei et al., 2003]  David M Blei, Andrew Y Ng, and Michael I Jordan.  2003\.  Latent dirichlet allocation.  the Journal of machine Learning research, 3:993–1022.
  * [DeYoung and Mangum, 2021]  Sarah E. DeYoung and Michaela Mangum.  2021\.  Pregnancy, birthing, and postpartum experiences during covid-19 in the united states.  Frontiers in Sociology, 6:12.
  * [Findeklee and Morinello, 2020]  Sebastian Findeklee and Emanuela Morinello.  2020\.  Clinical implications and economic effects of the corona virus pandemic on gynaecology, obstetrics and reproductive medicine in germany-learning from italy.  Minerva ginecologica.
  * [Goer, 2020]  Henci Goer.  2020\.  Covid-19: Should you agree to elective induction of labor?
  * [Gutschow and Davis-Floyd, 2021]  Kim Gutschow and Robbie Davis-Floyd.  2021\.  The impacts of covid-19 on us maternity care practices: A followup study.  Frontiers in Sociology, 6:108.
  * [Mariño-Narvaez et al., 2021]  Carolina Mariño-Narvaez, Jose A Puertas-Gonzalez, Borja Romero-Gonzalez, and Maria Isabel Peralta-Ramirez.  2021\.  Giving birth during the covid-19 pandemic: The impact on birth satisfaction and postpartum depression.  International Journal of Gynecology & Obstetrics, 153(1):83–88.
  * [McCallum, 2002]  Andrew Kachites McCallum.  2002\.  Mallet: A machine learning for language toolkit.  http://mallet.cs.umass.edu.
  * [Pilkington and Bedford-Dyer, 2021]  Pamela D. Pilkington and Isabella Bedford-Dyer.  2021\.  Mothers’ worries during pregnancy: A content analysis of reddit posts.  The Journal of Perinatal Education, 30(2):98–107.
  * [Pilkington and Rominov, 2017]  Pamela D. Pilkington and Holly Rominov.  2017\.  Fathers’ worries during pregnancy: A qualitative content analysis of reddit.  The Journal of Perinatal Education, 26(4):208–218.
  * [Röder et al., 2015]  Michael Röder, Andreas Both, and Alexander Hinneburg.  2015\.  Exploring the space of topic coherence measures.  In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM ’15, page 399–408, New York, NY, USA. Association for Computing Machinery.
  * [Santosh et al., 2020]  Roshan Santosh, H. Schwartz, Johannes Eichstaedt, Lyle Ungar, and Sharath Chandra Guntuku.  2020\.  Detecting emerging symptoms of COVID-19 using context-based Twitter embeddings.  In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online, December. Association for Computational Linguistics.
  * [Tabak and Purver, 2020]  Tom Tabak and Matthew Purver.  2020\.  Temporal mental health dynamics on social media.  In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online, December. Association for Computational Linguistics.
  * [Taylor and Letham, 2018]  Sean J Taylor and Benjamin Letham.  2018\.  Forecasting at scale.  The American Statistician, 72(1):37–45.
  * [Thukral et al., 2020]  Sachin Thukral, Suyash Sangwan, Arnab Chatterjee, and Lipika Dey.  2020\.  Identifying pandemic-related stress factors from social-media posts – Effects on students and young-adults.  In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online, December. Association for Computational Linguistics.
  * [Vazquez-Vazquez et al., 2021]  Adriana Vazquez-Vazquez, Sarah Dib, Emeline Rougeaux, Jonathan C Wells, and MS Fewtrell.  2021\.  The impact of the covid-19 lockdown on the experiences and feeding practices of new mothers in the uk: Preliminary data from the covid-19 new mum study.  Appetite, 156:104985.
  * [Wolohan, 2020]  JT Wolohan.  2020\.  Estimating the effect of COVID-19 on mental health: Linguistic indicators of depression during a global pandemic.  In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, Online, July. Association for Computational Linguistics.

(a) The family members topic fell significantly below the confidence interval
in the post-COVID-19 period.

(b) The induction topic rose significantly above the confidence interval in
the post-COVID-19 period.

Figure 1: Topic forecasts versus actual data for two topics over time. The
x-axis, y-axis, and red vertical line respectively indicate the date, monthly
average topic probability, and beginning of COVID-19. The blue line, shaded
region, and black line respectively represent the models’ prediction, 95% CI,
and actual data. Each figure includes the top-5 words for each topic.

Figure 2: How often the persona is mentioned on average (y-axis) during the
course of a narrative (x-axis).


# HausaMT v1.0: Towards English–Hausa Neural Machine Translation

Adewale Akinfaderin
Data Duality
<EMAIL_ADDRESS>

###### Abstract

Neural Machine Translation (NMT) for low-resource languages suffers from low
performance because of the lack of large amounts of parallel data and language
diversity. To contribute to ameliorating this problem, we built a baseline
model for English–Hausa machine translation, which is considered a task for
low–resource language. The Hausa language is the second largest Afro–Asiatic
language in the world after Arabic and it is the third largest language for
trading across a larger swath of West Africa countries, after English and
French. In this paper, we curated different datasets containing Hausa–English
parallel corpus for our translation. We trained baseline models and evaluated
the performance of our models using the Recurrent and Transformer
encoder–decoder architecture with two tokenization approaches: standard
word–level tokenization and Byte Pair Encoding (BPE) subword tokenization.

##  1 Introduction

Hausa is a language spoken in the western part of Africa. It belongs to the
Afro–Asiatic phylum and it is the second most spoken native language on the
continent, after Swahili. The language is spoken by more than 40 million
people as a first language and about 15 million people use it as a second and
third language. Most of the speakers are concentrated in Nigeria, Niger and
Chad – all resulting to both anglophone and francophone influences [Sabiu et
al., 2018, Eberhard et al., 2019]. Our work on curating datasets and creating
evaluation benchmark for English–Hausa Neural Machine Translation (NMT) is
inspired by the socio–linguistic facts of the Hausa language. Hausa has been
referred to as the largest internal political unit in Africa. There has been
extensive linguistic academic research on Hausa and the language benefits from
the existence of trans–border communication in the West African Sahel belt and
the availability of international radio stations like the BBC Hausa and Voice
of America Hausa [Odoje, 2013].

The exponential growth of social media platforms have eased communication
among users. However, the advances in technological adoption have also
informed the need to translate human languages. In low–resource countries, the
language inequality can be ameliorated by using machine translation to bridge
gaps in technological, political and socio–economic advancements [Odoje,
2016]. The recent successes in NMT over Phrased–Based Statistical Machine
Translation (PBSMT) for high–resource data conditions can be leveraged to
explore best practices, data curation and evaluation benchmark for
low–resource NMT tasks [Bentivogli et al., 2016, Isabelle et al., 2017]. Using
the JW300, Tanzil, Tatoeba and Wikimedia public datasets, we trained and
evaluated baseline NMT models for Hausa language.

00footnotetext: 4th Widening NLP Workshop, Annual Meeting of the Association
for Computational Linguistics, ACL 2020

##  2 Related Works

###  _Hausa Words Embedding_ :

Researchers have recently curated datasets and trained word embedding models
for the Hausa language. The results from this trained models have been
promising, with approximately 300% increase in prediction accuracy over other
baseline models [Abdulmumin & Galadanci, 2019].

###  _Masakhane_ :

Due to the linguistic complexity and morphological properties of languages
native to continent of Africa, using abstractions from successful
resource–rich cross–lingual machine translation tasks often fail for
low–resource NMT task. The Masakhane project was created to bridge this gap by
focusing on facilitating open–source NMT research efforts for African
languages [$\forall$, Orife et al., 2020].

##  3 Dataset Description

For the HausaMT task, we used the JW300, Tanzil, Tatoeba and Wikimedia public
datasets. The JW300 dataset is a crawl of the parallel data available on
Jehovah Witness’ website. Most of the data are from the magazines, _Awake!_
and _Watchtower_ , and they cover a diverse range of societal topics in a
religious context [Agić & Vulić, 2019]. The Tatoeba database is a collection
of parallel sentences in 330 languages [Raine, 2018]. The dataset is
crowdsourced and published under a Creative Commons Attribution 2.0 license.
The Tanzil dataset is a multilingual text aimed at producing a highly verified
multi-text of the Quran text [Zarrabi-Zadeh et al., 2007]. The Wikimedia
dataset are parallel sentence pairs extracted and filtered from noisy parallel
and comparable wikipedia copora [Wolk & Marasek, 2014]. For this work, we
trained on two datasets which are: 1) the JW300 as our baseline, and 2) All
the datasets combined. The number of tokens, number of sentences and
statistical properties of the datasets are in Table 1.

Dataset | Sentence Length (Mean $\pm$ Std) | Tokens | Sentences
---|---|---|---
| English | Hausa | English | Hausa |
JW300 | 18.11 $\pm$ 10.53 | 20.14 $\pm$ 11.57 | 4,051,322 | 4,506,787 | 223,723
All∗ | 19.71 $\pm$ 24.31 | 21.28 $\pm$ 24.60 | 6,919,805 | 7,471,256 | 351,024
Table 1: Dataset summary.∗Combination of JW300, Tanzil, Tatoeba and Wikimedia
datasets.

##  4 Experiments and Results

For our baseline model, we trained a recurrent-based model with the Long
Short-Term Memory (LSTM) network as our encoder and decoder type with the
Luong attention mechanism [Luong et al., 2015]. To achieve an improved
benchmark, we also trained a Transformer encoder–decoder model. The
Transformer is based on attention mechanism and the training time is
significantly faster than architectures based on convolutional or recurrent
networks [Vaswani et al., 2017]. For the hyperparameters used to train both
the recurrent and transformer based architecture, we used an embedding size of
256, hidden units of 256, batch size of 4096 and an encoder and decoder depth
of 6 respectively.

Dataset | Model | BPE | Word
---|---|---|---
dev | test | dev | test
JW300 | Recurrent | 20.06 | 19.39 | 25.36 | 24.75
Transformer | 21.33 | 20.38 | 28.71 | 28.06
All | Recurrent | 31.89 | 33.48 | 40.78 | 42.29
Transformer | 31.91 | 32.42 | 44.42 | 45.98
Table 2: BLEU scores for BPE and word-level tokenization. Best scores of the
Transformer model against the Recurrent are highlighted in bold.

To preprocess the parallel corpus, we used the standard word–level
tokenization and Byte Pair Encoding (BPE) [Gage, 1994]. The BPE is a subword
tokenization which has become a successful choice in translation tasks. The
model was trained based on the 4000 BPE tokens used on a recent machine
translation study for South African languages [Martinus & Abbott, 2019]. To
train our model, the Joey NMT minimalist toolkit, which is open source and
based on PyTorch was used [Kreutzer et al., 2019]. The models were trained
using a Tesla P100 GPU. The model training for the baseline and repeated tasks
(datasets & tokenization type) took between 5-9 hours for each run.

##  5 Conclusion and Future Work

Evaluating the model on the test set, we observed that the word–level
tokenization outperform the BPE by a BLEU score factor of ~1.27–1.42 times
(Table 2). The qualities of the English to Hausa translations using both
word–level and BPE subword tokenizations were rated positively by first
language speakers. Table 3 shows some of the translations example.

Source: | This is normal, because they themselves have not been anointed.
---|---
Reference: | Hakan ba abin mamaki ba ne don ba a shafa su da ruhu mai tsarki ba.
Hypothesis: | Wannan ba daidai ba ne, domin ba a shafe su ba.
Source: | A white - haired man in a frock coat appears on screen.
Reference: | Wani mutum mai furfura ya bayyana da dogon kwat a majigin.
Hypothesis: |  | Wani mutum mai suna da wani mutum mai suna da ke cikin mota yana
---
da nisa a cikin kabari
Source: | Why is that of vital importance?
Reference: | Me ya sa hakan yake da muhimmanci?
Hypothesis: | Me ya sa wannan yake da muhimmanci?
Table 3: Example Translations.

A significant portion of both the training and test datasets are from the the
JW300 parallel data, which are religious texts. We acknowledge that for us to
reach a viable state of real-world translation quality, we need to evaluate
our model on "general" Hausa data. However, parallel data for other out-of-
domain areas does not exist. High-yielding avenue for future work include
evaluating on English texts in other domains and crowd-sourcing L1 speakers to
manually evaluate the quality of the translations by editing. The post edited
translation can then be used as the reference to calculate the evaluation
metric. Other future work include carrying out an empirical study to explore
the effect of word–level and subword tokenizations. Other methods such as the
linguistically motivated vocabulary reduction (LMVR) have shown to perform
better for languages in the Afro–Asiatic family [Ataman & Federico, 2018]. The
datasets, pre–trained models, and configurations are available on
Github.111https://github.com/WalePhenomenon/Hausa-NMT

## Acknowledgements

The author would like to thank Gabriel Idakwo for the qualitative analysis of
the translations.

## References

  * [Eberhard et al., 2019]  David M. Eberhard, Gary F. Simons, and Charles D. Fennig (eds.).  2019\.  Ethnologue: Languages of the world. twenty-second edition.  URL: http://www.ethnologue.com.
  * [Sabiu et al., 2018]  Ibrahim T. Sabiu, Fakhrul A. Zainol, and Mohammed S. Abdullahi.  2018\.  Hausa People of Northern Nigeria and their Development.  Asian People Journal (APJ), eISSN : 2600-8971, Volume 1, Issue 1, Pp 179-189.
  * [Odoje, 2013]  Clement Odoje.  2013\.  Language Inequality: Machine Translation as the Bridging Bridge for African languages.  4, 01.
  * [Odoje, 2016]  Clement Odoje.  2016\.  The Peculiar Challenges of SMT to African Languages .  s. ICT, Globalisation and the Study of Languages and Linguistics in Africa, Pp 223.
  * [Bentivogli et al., 2016]  Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico.  2016\.  Neural versus Phrase-Based Machine Translation Quality: a Case Study.  Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Pp 257-267, Austin, Texas.
  * [Isabelle et al., 2017]  Pierre Isabelle, Colin Cherry,and George Foster.  2017\.  A Challenge Set Approach to Evaluating Machine Translation.  Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Pp 2486-2496, Copenhagen, Denmark.
  * [Abdulmumin & Galadanci, 2019]  Idris Abdulmumin and Bashir S. Galadanci.  2019\.  hauWE: Hausa Words Embedding for Natural Language Processing.  2019 2nd International Conference of the IEEE Nigeria Computer Chapter, Pp 1-6, Zaria, Nigeria.
  * [$\forall$, Orife et al., 2020]  $\forall$, Iroro F. O. Orife, Julia Kreutzer, Blessing Sibanda, Kathleen Siminyu, Laura Martinus, Jamiil Toure Ali, Jade Abbott, Vukosi Marivate, Salomon Kabongo, Musie Meressa, Espoir Murhabazi, Orevaoghene Ahia, Elan van Biljon, Arshath Ramkilowan, Adewale Akinfaderin, Alp Öktem, Wole Akin, Ghollah Kioko, Kevin Degila, Herman Kamper, Bonaventure Dossou, Chris Emezue, Kelechi Ogueji, and Abdallah Bashir.  2020\.  Masakhane – Machine Translation For Africa.  To appear in the Proceedings of the AfricaNLP Workshop, International Conference on Learning Representations (ICLR 2020).
  * [Agić & Vulić, 2019]  Željko Agić and Ivan Vulić.  2019\.  JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages.  Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Pp 3204-3210, Florence, Italy.
  * [Raine, 2018]  Paul Raine.  2018\.  Building Sentences with Web 2.0 and the Tatoeba Database.  Accents Asia, 10(2), Pp 2-7.
  * [Zarrabi-Zadeh et al., 2007]  Hamid Zarrabi-Zadeh, Abbas Ahmadi, Morteza Bagheri, Yousef Daneshvar, Mohammad Derakhshani, Mohammad Fakharzadeh, Ehsan Fathi, Yusof Ganji, Mojtaba Haghighi, Nasser Lashgarian, Zahra Mousavian, Mohsen Saboorian, Yaser Shanjani, Mohammad-Reza Nikseresht, and Mahdi Mousavian.  2007\.  Tanzil Project.  URL: http://tanzil.net/docs/home.
  * [Wolk & Marasek, 2014]  Krzysztof Wolk and Krzysztof Marasek.  2014\.  Building Subject-aligned Comparable Corpora and Mining it for Truly Parallel Sentence Pairs.  Procedia Technology, Volume 18, Pp 126-132.
  * [Vaswani et al., 2017]  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is All you Need.  In 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
  * [Gage, 1994]  Philip Gage.  1994\.  A New Algorithm for Data Compression.  C Users J., 12(2), Pp 23-38.
  * [Martinus & Abbott, 2019]  Laura Martinus and Jade Abbott.  2019\.  A Focus on Neural Machine Translation for African Languages.  CoRR, abs/1906.05685. URL: http://arxiv.org/abs/1906.05685.
  * [Kreutzer et al., 2019]  Julia Kreutzer, Joost Bastings, and Stefan Riezler.  2019\.  Joey NMT: A Minimalist NMT Toolkit for Novices.  Proceedings of the 2019 EMNLP and the 9th IJCNLP (System Demonstrations), Pp 109-114, Hong Kong, China.
  * [Ataman & Federico, 2018]  Duygu Ataman and Marcello Federico.  2018\.  An Evaluation of Two Vocabulary Reduction Methods for Neural Machine Translation.  Proceedings of AMTA 2018, vol. 1: MT Research Track, Pp 97-110, Boston, MA.
  * [Luong et al., 2015]  Thang Luong, Hieu Pham, and Christopher D. Manning.  2015\.  Effective Approaches to Attention-based Neural Machine Translation.  Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Pp 1412-1421, Lisbon, Portugal.