# Voice Conversion Can Improve ASR in Very Low-Resource Settings

###### Abstract

Voice conversion (VC) could be used to improve speech recognition systems in
low-resource languages by using it to augment limited training data. However,
VC has not been widely used for this purpose because of practical issues such
as compute speed and limitations when converting to and from unseen speakers.
Moreover, it is still unclear whether a VC model trained on one well-resourced
language can be applied to speech from another low-resource language for the
aim of data augmentation. In this work we assess whether a VC system can be
used cross-lingually to improve low-resource speech recognition. We combine
several recent techniques to design and train a practical VC system in
English, and then use this system to augment data for training speech
recognition models in several low-resource languages. When using a sensible
amount of VC augmented data, speech recognition performance is improved in all
four low-resource languages considered. We also show that VC-based
augmentation is superior to SpecAugment (a widely used signal processing
augmentation method) in the low-resource languages considered.

Index Terms: voice conversion, data augmentation, low-resource speech
processing, speech recognition.

##  1 Introduction

Although automatic speech recognition (ASR) systems have greatly improved over
the last few years [1, 2, 3], they still struggle in very low-resource
settings where training data is minimal and a high quality language model (LM)
is not available [4]. Concretely, even state-of-the-art ASR performance is
limited for languages where we only have a few minutes of transcribed audio
from a handful of speakers [5], e.g. when building systems for regional
dialects or code-switched speech.

One possible solution is to use data augmentation: available data is processed
so that it has different characteristics from the original while preserving
the content (words spoken), effectively giving more training data. This idea
has a long history [6, 7], with more recent methods like SpecAugment [8]
improving ASR performance and robustness. While these signal processing
approaches have proven effective, they are mostly constrained to direct
general-purpose modifications of the audio waveform – e.g. pitch shifting,
time stretching and frequency masking – ignoring some of the unique aspects of
human speech. To address this, one research direction is pursuing the idea of
using voice conversion for data augmentation.

In voice conversion (VC), an utterance spoken by one speaker is processed so
that it sounds as if it was spoken by a different target speaker, while
keeping the content unchanged [9]. VC approaches are becoming more practical
in terms of speed [10], naturalness [11, 12], and performance on unseen
speakers [13, 14]. Some newer methods even provide finer-grained ways to
control different properties of the generated speech [15, 12], and can be used
cross-lingually [16].

To use VC for data augmentation, ASR training utterances are fed into a VC
system and converted to arbitrary target speakers, thereby creating additional
synthetic training utterances with a known transcript. It has been
hypothesized that this would improve ASR performance in low-resource settings
[17] , although initial attempts (Sec. 2) have had limited success. Moreover,
no studies have considered applying a VC model trained on a well-resourced
language to speech from another unseen low-resource language for the purpose
of data augmentation – this is a requirement for using VC augmentation in
arbitrary settings.

Here we aim to address this shortfall and see if VC can indeed be used as an
effective data augmentation approach to improve ASR in unseen low-resource
settings. We first propose a VC system trained on English that is practical
for cross-lingual data augmentation: running faster than real time on unseen
speakers and unseen languages. We then apply this VC approach to create
synthetic data for several low-resource settings in English, Afrikaans,
Sepedi, isiXhosa and Setswana. To answer our research question, we compare the
performance of ASR systems trained with and without generated voice-converted
data, and also compare to SpecAugment. With limited training data (10 min) we
find that our VC approach is complementary to SpecAugment on English, while it
is superior for data augmentation on the low-resource languages. To the best
of our knowledge, we are the first to use cross-lingual VC for data
augmentation, where VC is applied to unseen languages and speakers.

##  2 Related work

Laptev et al. [18] explored using fast text-to-speech (TTS) methods to perform
data augmentation, obtaining improvements on medium-resource ASR. Since terms
about resources are contested in the literature [19], we define very low-
resource, low-resource, and medium resource as settings with roughly 10 min, 1
h, and 10 h to 50 h of labelled audio, respectively. However, for very low-
resource applications like the ones explored in this paper, a good TTS model
or LM is usually unavailable.

The idea of using VC to augment data has a long history [20]. Cui et al. [21]
made initial attempts to improve medium-resource ASR showing moderate
improvements despite limitations of VC techniques available at the time. Other
studies [22, 23] showed that using more recent deep learning-based VC methods
can improve TTS systems where the desired speaker is low-resource. However,
these studies use a target speaker from a language seen during training,
leaving the question open as to whether ASR can be improved in unseen low-
resource languages.

Zhao et al. [24] made an attempt to use cross-lingual VC to improve code-
switched TTS models. However, their VC models are trained separately for each
speaker considered, making their system unsuitable for use on new unseen low-
resource languages. Finally, [25] found that applying a deep-learning-based VC
approach failed to noticeably improve medium-resource ASR models when all
models are trained and evaluated on English. They were, however, limited by
practical problems associated with VC approaches available at the time, such
as the fast but low-quality Griffin-Lim vocoder.

##  3 Voice conversion method

Our goal is to use VC for low-resource data augmentation. This means that our
overall VC model needs to be computationally efficient, and be able to convert
input speech from a language (and speaker) that are unseen during training.

Overview: Our approach is shown in Fig. 1. We want to convert an input
waveform into a sequence of feature vectors and then separate the linguistic
content (the words being spoken) from the speaker information. We do this with
specially designed style and content encoders, denoted with dashed boxes. The
style encoder produces a single style vector $\mathbf{s}$, while the content
encoder produces a sequence of content embedding vectors. During training
(Fig. 1(a)) the style vector $\mathbf{s}=\mathbf{s}_{\textrm{src}}$ and
content embeddings are obtained from the same input utterance. The model is
trained to reconstruct the input by summing $\mathbf{s}_{\textrm{src}}$ with
each content embedding, passing the resulting vectors into a decoder module.
At test time (Fig. 1(b)) the content and speaker encoders receive speech from
different utterances. This means that the source or reference utterance (or
both) may be from languages and speakers unseen during training. If the
content and speaker information is appropriately separated, then the source
utterance will be produced in the voice of the speaker supplying the reference
utterance. Details on each component follows.

Speech encoder: Instead of using a spectrogram represent the reference or
source utterances, we use a pretrained speech encoder. Specifically, we use
the contrastive predictive coding (CPC) model from [26]. This model,
$E_{\textrm{CPC}}$ in Fig. 1, produces a 512-dimensional vector for every 10
ms of 16 kHz-sampled audio. This model transforms the input waveform into
representations that makes it easier to disentangle content and speaker
information for the downstream style and content encoders [27].

Style encoder: The style encoder consists of a reference encoder $E_{R}$ and a
hierarchical global style token (HGST) layer [15]. The reference encoder
$E_{R}$ consists of several 1-D convolutional layers followed by an LSTM [28].
The final hidden state of the LSTM is linearly transformed and passed to the
HGST layer. The HGST layer acts as a form of information bottleneck; the idea
is that the network learns to only include speaker information in
$\mathbf{s}$, since the content information can be obtained from the content
encoder output. The HGST layer is parameterized by $l$ sublayers and $h$
trainable vectors per sublayer. The first sublayer attempts to represent the
last output vector from $E_{R}$ as a convex combination of its $h$ vectors.
The difference between this combination and the input vector is then used as
the input to the next sublayer, so each layer in the hierarchy models the
residue from the previous approximation [15]. With small $h$ and $l$, the HGST
layer forces the final style vector $\mathbf{s}$ to only retain information
about the global speaking style.

(a) Training setup

(b) Inference setup

Figure 1:  Our VC approach. (a) During training, the same utterance is fed
through a CPC encoder into both the style and content encoders; the decoder
tries to reconstruct the input utterance’s spectrogram. (b) At inference,
separate input and reference utterances are fed into the style and content
encoders, respectively, before vocoding the spectrogram using HiFi-GAN [10].

Content encoder: The content encoder attempts to remove speaker information
from the CPC features so as to only retain linguistic content. The
quantization block $Q$ in Fig. 1 first performs speaker normalization using
means and variances computed for features from the pretrained
$E_{\textrm{CPC}}$ on a subset of the training data. This was shown to improve
speaker invariance [27]. During inference with unseen source speakers, we
compute the mean and variance on the features of the single source utterance.
The normalized embeddings are quantized by representing them by the nearest
centroid vector in a $K$-means clustering model with $K=100$. This quantized
sequence is then passed through a content encoder module $E_{C}$ with the same
architecture as $E_{R}$, except returning all LSTM states – one content vector
every 10 ms. The $K$-means and speaker normalization parameters are
precomputed before the rest of the network is trained.

Decoding and vocoding: Each content vector is summed with the style vector
$\mathbf{s}$. The resulting sequence is then passed into a Tacotron 2 decoder
network $D$ to produce an output mel-spectrogram [29]. We use the same
Tacotron 2 decoder architecture as in [29]. During training (Fig. 1(a)), the
loss is formulated the same as for the original Tacotron 2: an $L_{2}$ loss
between the predicted spectrogram and the spectrogram of the source utterance
with an additional loss associated with the length of the output spectrogram
[29]. During inference (Fig. 1(b)), the output spectrogram is converted to the
time-domain using We use the spectrogram parameters and pretrained HiFi-GAN
model from [10], due to its high performance and fast inference speed.

With these choices for the different modules, our overall VC approach is
faster than real time on a modern GPU.

##  4 Experimental setup

Data: We want to know whether a VC model trained on a well-resourced language
can be applied to an unseen low-resource language to improve ASR. English is
our well-resourced language; we use data from LibriSpeech [30] and Libri-Light
[31], which consists of hundreds of hours of read English. We consider four
low-resource languages: Afrikaans, Setswana, isiXhosa and Sepedi. Data for the
first three are obtained from [32] (roughly 3.2 h per language), while Sepedi
data is obtained from [33] (10.8 h).

VC model: For the speech encoder $E_{\textrm{CPC}}$, we use the pretrained
CPC-big model from [26]. The $E_{R}$ and $E_{C}$ modules use exactly the same
architecture as in [29], except that the convolutional kernel sizes are 9
instead of 5 and $E_{R}$ has an additional 512-dimensional linear output
layer. For the HGST layer, we use $l=3$ sublayers with $h=5$ vectors each,
outputting a 512-dimensional style vector $\mathbf{s}$. With the pretrained
$E_{\textrm{CPC}}$ and quantization module $Q$ fixed, the weights of $E_{R}$,
$E_{C}$, $D$ and HGST are trained on 275k utterances from the LibriSpeech
training set and validated on the remaining utterances in the set. We use Adam
optimization [34], gradient norm clip of 1, a maximum learning rate of $6\cdot
10^{-4}$ with the scheduler from [35], and train for 50 epochs with a batch
size of 32. To stabilize training, we further follow [36]: utterances in a
batch are randomly cropped to a fixed length; the base crop length is linearly
increased from 0.6 s to 10.2 s over the training iterations, with the actual
crop length uniformly sampled from $\pm 20\%$ of the current base value.

VC inference: In all experiments, the reference utterance is randomly selected
from the LibriSpeech training set. For Afrikaans, isiXhosa and Setswana, we
only consider reference utterances from female target speakers since these
datasets only contain speech from females. For English and Sepedi, we consider
reference utterances from both male and female speakers. At inference time, we
convert arbitrary length utterances into 7 s chunks, stitching the results
together to produce a final output.

SpecAugment: We use the SpecAugment settings exactly as in [8]. We also
consider whether VC and SpecAugment is complementary: in these experiments we
first perform VC and then further augment the result with SpecAugment. We
denote this chained augmentation as VC$\rightarrow$SpecAug.

Automatic speech recognition: For doing ASR in each of our low-resource
languages, we fine-tune the pretrained XLSR-53 wav2vec 2.0 model [5] with a
character-level CTC loss [37]. In validation experiments on English, we use
the pretrained wav2vec 2.0 English models from [1]. We train the ASR models
using fairseq [38] with its 10 min fine-tuning configuration unless otherwise
stated. Since we are interested in demonstrating the potential change in ASR
performance, we opt to use greedy CTC decoding without an LM to isolate the
effect of augmentation on the acoustic model. In many real low-resource
settings, it might also not be possible to obtain a good LM. However, to
confirm the trends for cases where an LM is available, we also include English
results with the official 4-gram LibriSpeech LM [30].

##  5 Experiments

###  5.1 Experiment 1: Validating voice conversion quality

We first perform validation experiments to confirm that our VC model performs
adequately both on English (its training language) and unseen languages. For
the latter, we focus here on Afrikaans and Sepedi, treating the other two
languages as completely unseen test languages for our final experiment (Sec.
5.3).

To assess intrinsic VC quality, we use two tests. The first checks whether
linguistic content is retained through a re-synthesis experiment [36]. Each
test utterance from each language is converted to a target speaker as per Sec.
4. ASR is then performed on both the input and converted utterances. If the
drop in ASR scores after VC is small in terms of word/character error rate
(W/CER), this indicates that VC did not significantly reduce intelligibility.
Table 1 shows the results. As expected, the VC data is less intelligible –
higher W/CER for the full model – compared to the original audio, with a
larger drop in scores on the languages that have not been seen in VC training
(Afrikaans and Sepedi). However, the drop is small enough that the output
utterance can still be understood.

The second intrinsic test assesses whether an output utterance matches the
target speaker. A pretrained speaker verification model
[39]111https://github.com/RF5/simple-speaker-embedding v0.1 is used to check
whether the speaker embedding associated with the converted speech is closer
to that of the reference or source utterance in terms of their cosine
similarity. We report an error rate: the proportion of converted test-set
utterances which are incorrectly closer to the source utterance rather than
the reference. Table 2 shows the results. We again see a drop in performance
from seen to unseen languages for the full model, where the converted
Afrikaans and Sepedi data lead to more errors in capturing the reference
speaker identity. Recall, though, that our goal of data augmentation is to
ensure that the converted utterance sounds sufficiently different to the input
utterance to increase speaker variety in a limited dataset while still being
intelligible so as to aid in ASR training.

Table 1: Re-synthesis results in terms of ASR performance on original and converted data. The English-trained VC system is applied to English, Sepedi and Afrikaans evaluation data. Lower W/CER (%) is better (higher intelligibility).  | English | Afrikaans | Sepedi
---|---|---|---
VC Model | WER | CER | WER | CER | WER | CER
Original data | $5.7$ | $1.9$ | $6.3$ | $4.3$ | $2.1$ | $0.9$
Full model | $20.6$ | $9.6$ | $32.5$ | $11.0$ | $20.4$ | $9.5$
Sans HGST | $21.3$ | $9.9$ | $34.0$ | $11.9$ | $21.1$ | $9.9$
Sans $Q$ | $7.1$ | $2.6$ | $17.0$ | $4.6$ | $3.7$ | $1.6$
Table 2: Speaker similarity error rates (%). Lower scores indicate that converted utterances are more often closer to the reference speaker than the source speaker (better VC).  VC Model | English | Afrikaans | Sepedi
---|---|---|---
Full model | $8.7$ | $22.0$ | $58.3$
Sans HGST | $2.3$ | $6.9$ | $34.4$
Sans $Q$ | $99.7$ | $99.8$ | $99.9$

The last two rows in Tables 1 and 2 show the effect when we remove either the
HGST or the normalization-quantization block $Q$ from the full model. Without
the HGST bottleneck, speaker conversion performance improves (Table 2), but at
the cost of intelligibility (Table 1). This is because the style vector
$\mathbf{s}$ captures more granular reference speaker information (better
conversion) but also more content information from the reference (less
intelligible). The opposite happens when we remove the bottleneck $Q$ from the
content encoder. Including both blocks therefore gives a balance between
intelligibility and conversion quality, and so we use the full model in the
remaining experiments. 222 Audio samples:
https://rf5.github.io/interspeech2022/.

###  5.2 Experiment 2: Validating data augmentation

Before applying VC cross-lingually (Sec. 5.3), we first want to test what
level of augmentation is beneficial on the seen language (English). We want to
specifically know: How much and in which low-resource settings does real vs
augmented data help? And how does it compare or synergize with SpecAugment? To
answer these questions, we fine-tune the Base wav2vec 2.0 English ASR models
on 10 min, 1 h, and 10 h of real data from Libri-Light’s fine-tuning dataset
[31]. In each setting we train several models: we compare VC, SpecAug, and
VC$\rightarrow$SpecAug augmentation (with 100% and 500% of additional
generated data) to an ASR model trained without any augmented data.

The results are given in Table 3. Looking at the 10 min setting without an LM,
we can make a few observations: First, using more augmented data appears to
improve performance, with the lowest WER (a 5.3% absolute improvement over no
augmentation) achieved with 500% additional data generated by chaining VC and
SpecAugment. Second, VC augmentation appears to give similar improvements to
SpecAugment when applied independently. Third, applying both VC and
SpecAugment improves performance more than either in isolation. This shows
that the two approaches are complementary on English.

Does this improvement from VC hold in all low-resource settings? No. With more
real data (1 or 10 h), adding more VC-augmented data appears to hinder ASR
performance. VC-augmentation gives more speaker variability but it does so at
the cost of intelligibility (Sec. 5.1); we suspect higher-resourced settings
already have sufficient speaker variability, so the additional (potentially
less intelligible) generated data contributes little. When using an LM for
decoding (last 3 columns), the benefits of any augmentation appear greatly
reduced. In very low-resource settings, however, an LM is often not available
and VC still gives a small improvement even though an LM is used.

We can now answer our main research question in part: using VC for data
augmentation does help in very low-resource contexts on a seen language. We
now turn to the question of whether this holds for languages unseen during VC
training. As a reminder, our goal is to determine whether VC can improve low-
resource ASR, not whether it is the best augmentation method.

Table 3:  WERs (%) on LibriSpeech test data for ASR models trained with increasing amounts of VC- and SpecAug-augmented data, with and without 4-gram LM decoding. |  | No LM | LM decoded
---|---|---|---
Augmentation | Amount | 10 min | 1 h | 10 h | 10 min | 1 h | 10 h
None | 0% | $47.7$ | $30.4$ | $13.4$ | $17.4$ | $10.6$ | $7.5$
VC | 100% | $43.8$ | $32.7$ | $13.5$ | $17.2$ | $11.4$ | $7.6$
VC | 500% | $43.5$ | $34.4$ | $14.4$ | $17.9$ | $11.9$ | $8.1$
SpecAug | 100% | $44.3$ | $31.8$ | $13.1$ | $18.8$ | $11.2$ | $7.6$
SpecAug | 500% | $43.1$ | $34.4$ | $13.3$ | $17.7$ | $12.1$ | $7.7$
VC$\rightarrow$SpecAug | 100% | $42.5$ | $31.3$ | $13.2$ | $18.5$ | $11.2$ | $7.6$
VC$\rightarrow$SpecAug | 500% | $42.4$ | $35.0$ | $14.2$ | $18.4$ | $12.5$ | $8.1$

###  5.3 Experiment 3: Very low-resource settings

In this final experiment we use the English-trained VC system to augment data
from four unseen low-resource languages: Afrikaans, Setswana, isiXhosa and
Sepedi. For each language we use a 10 min training set with roughly equally
sized validation and test sets covering the rest of the data in each
respective language. There is no speaker overlap between any of the sets. For
each language, we fine-tune XLSR-53 [5] on the original 10 min of data, and
compare this to training with 100% (10 min) of additional VC and
VC$\rightarrow$SpecAug generated data.

Table 4: ASR results (%) on test data of four low-resource languages when trained on 10 min of real audio data and different amounts of additional VC- and combined VC-SpecAug augmented data. Sepedi∗ uses a non-default training procedure.  Language |  Augmentation | Amount | WER | CER
---|---|---|---|---
Afrikaans |  None | 0% | 52.3 | 15.9
VC | 100% | 48.9 | 15.0
VC$\rightarrow$SpecAug | 100% | 53.5 | 16.5
Setswana |  None | 0% | 68.9 | 26.1
VC | 100% | 65.9 | 25.1
VC$\rightarrow$SpecAug | 100% | 69.3 | 26.8
isiXhosa |  None | 0% | 63.2 | 15.5
VC | 100% | 56.5 | 13.8
VC$\rightarrow$SpecAug | 100% | 69.3 | 26.8
Sepedi* |  None | 0% | 92.6 | 50.7
VC | 100% | 52.8 | 19.9
VC$\rightarrow$SpecAug | 100% | 97.8 | 69.1

For all four unseen languages, Table 4 shows an improvement in ASR performance
when 10 min of additional data is generated using the English-trained VC
system. For isiXhosa, adding the augmented data gives a 7% absolute
improvement in WER. Chaining VC with SpecAugment, on the other hand, worsens
performance. Additional experiments were done with 500% of additional VC and
VC$\rightarrow$SpecAug data to see if trends followed similar to Sec. 5.2.
However W/CER got worse in all cases compared to training with just the
original data. We also applied SpecAugment in isolation on Afrikaans, which
again gave worse performance compared to doing no augmentation.

Some of Table 4’s results therefore contradicts the results on English in Sec.
5.2: in Table 3 SpecAugment and VC gave decent gains when applied in isolation
and even more so when chained, while here on unseen languages SpecAugment
hampers performance in all cases. This requires further investigation.
Nevertheless, we see on all four low-resource languages that cross-lingual VC
augmentation improves ASR performance.

It is worth briefly commenting on the results for Sepedi in Table 4, where
performance is very poor without augmentation. We used the same 10 min
training configuration for all languages, but on Sepedi, this configuration
failed (WER $>$ 98%) regardless of whether data augmentation was used or not.
We therefore did validation experiments and made minimal changes to the
default configuration, only for Sepedi: we use 1k warmup updates and train for
a total of 4k updates. Using this setting enabled the ASR model to learn,
achieving the test performance in Table 4.

Taken together, the results in Table 4 confirm our hypothesis: a VC system
trained on one well-resourced language (English) can be applied cross-
lingually to generate additional data for an unseen low-resource language,
improving ASR performance. This seem to conflict with [25], where a similar
approach was used but the VC and ASR models were both trained and evaluated on
English (Sec. 2). Our study is different in that we do cross-lingual VC, and
also use much less data to fine-tune the ASR models – since our focus is on
very low-resource ASR.

##  6 Conclusion

We have shown that voice conversion (VC) can be used for data augmentation to
improve ASR performance in very low-resource settings. We specifically showed
that a VC system trained on one well-resourced language can be used to
generate additional training data for unseen low-resource languages. When
labelled resources are very limited (roughly 10 min), ASR performance improved
on all four low-resource languages considered. Additional experiments also
demonstrated SpecAugment’s inability to improve ASR performance in these very
low-resource settings.

Our goal was to show that ASR improvements are possible with VC-based
augmentation. We did not focus exhaustively on how different design decisions
within the VC model itself impacts downstream ASR performance, and the
interplay between VC and traditional augmentation methods remains largely
unexplored – important directions for future work.

##  7 Acknowledgements

This work is supported in part by the National Research Foundation of South
Africa (grant no. 120409). Experiments were performed on Stellenbosch
University’s HPC Cluster.

## References

  * [1] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in _NeurIPS_ , 2020\.
  * [2] Y. Zhang, J. Qin, D. S. Park, W. Han, C.-C. Chiu _et al._ , “Pushing the limits of semi-supervised learning for automatic speech recognition,” _arXiv preprint arXiv:2010.10504_ , 2020.
  * [3] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov _et al._ , “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” _arXiv preprint arXiv:2106.07447_ , 2021\.
  * [4] L. Besacier, E. Barnard, A. Karpov, and T. Schultz, “Automatic speech recognition for under-resourced languages: A survey,” _Speech Commun._ , vol. 56, pp. 85–100, 2014.
  * [5] A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsupervised cross-lingual representation learning for speech recognition,” _arXiv preprint arXiv:2006.13979_ , 2020.
  * [6] M. Hunt and C. Lefebvre, “A comparison of several acoustic representations for speech recognition with degraded and undegraded speech,” in _ICASSP_ , 1989.
  * [7] G. Lathoud, M. Magimai-Doss, B. Mesot, and H. Bourlard, “Unsupervised spectral subtraction for noise-robust ASR,” in _ASRU_ , 2005.
  * [8] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph _et al._ , “SpecAugment: A simple data augmentation method for automatic speech recognition,” in _Interspeech_ , 2019.
  * [9] S. H. Mohammadi and A. Kain, “An overview of voice conversion systems,” _Speech Commun._ , vol. 88, pp. 65–82, 2017.
  * [10] J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,” in _NeurIPS_ , 2020.
  * [11] Y. Zhao, W.-C. Huang, X. Tian, J. Yamagishi, R. K. Das _et al._ , “Voice conversion challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion,” _arXiv preprint arXiv:2006.13979_ , 2020\.
  * [12] A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia _et al._ , “Speech resynthesis from discrete disentangled self-supervised representations,” in _Interspeech_ , 2021.
  * [13] K. Qian, Y. Zhang, S. Chang, X. Yang, and M. Hasegawa-Johnson, “AutoVC: Zero-shot voice style transfer with only autoencoder loss,” in _PMLR_ , 2019\.
  * [14] M. Baas and H. Kamper, “StarGAN-ZSVC: Towards zero-shot voice conversion in low-resource contexts,” in _SACAIR_ , 2020.
  * [15] X. An, Y. Wang, S. Yang, Z. Ma, and L. Xie, “Learning hierarchical representations for expressive speaking style in end-to-end speech synthesis,” in _ASRU_ , 2019.
  * [16] Z. Yang, W. Zhang, Y. Liu, and X. Xing, “Cross-lingual voice conversion with disentangled universal linguistic representations,” in _Interspeech_ , 2021\.
  * [17] N. Jaitly and G. E. Hinton, “Vocal tract length perturbation (VTLP) improves speech recognition,” in _ICML_ , 2013.
  * [18] A. Laptev, R. Korostik, A. Svischev, A. Andrusenko, I. Medennikov _et al._ , “You do not need more data: Improving end-to-end speech recognition by text-to-speech data augmentation,” in _CISP-BMEI_ , 2020.
  * [19] M. A. Hedderich, L. Lange, H. Adel, J. Strötgen, and D. Klakow, “A survey on recent approaches for natural language processing in low-resource scenarios,” in _NAACL_ , 2021.
  * [20] J. Bellegarda, P. de Souza, A. Nadas, D. Nahamoo, M. Picheny _et al._ , “The metamorphic algorithm: a speaker mapping approach to data augmentation,” _IEEE/ACM TASLP_ , vol. 2, no. 3, pp. 413–420, 1994.
  * [21] X. Cui, V. Goel, and B. Kingsbury, “Data augmentation for deep neural network acoustic modeling,” _IEEE/ACM TASLP_ , vol. 23, no. 9, pp. 1469–1477, 2015\.
  * [22] G. Huybrechts, T. Merritt, G. Comini, B. Perz, R. Shah _et al._ , “Low-resource expressive text-to-speech using data augmentation,” in _ICASSP_ , 2021.
  * [23] S. Shahnawazuddin, N. Adiga, K. Kumar, A. Poddar, and W. Ahmad, “Voice conversion based data augmentation to improve children’s speech recognition in limited data scenario,” in _Interspeech_ , 2020.
  * [24] S. Zhao, T. H. Nguyen, H. Wang, and B. Ma, “Towards natural bilingual and code-switched speech synthesis based on mix of monolingual recordings and cross-lingual voice conversion,” in _Interspeech_ , 2020.
  * [25] G. Keskin, T. Lee, C. Stephenson, and O. H. Elibol, “Measuring the effectiveness of voice conversion on speaker identification and automatic speech recognition systems,” _arXiv preprint arXiv:1905.12531_ , 2019.
  * [26] T. A. Nguyen, M. de Seyssel, P. Rozé, M. Rivière, E. Kharitonov _et al._ , “The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling,” _arXiv preprint arXiv:2011.11588_ , 2020.
  * [27] B. van Niekerk, L. Nortje, M. Baas, and H. Kamper, “Analyzing speaker information in self-supervised models to improve zero-resource speech processing,” in _Interspeech_ , 2021.
  * [28] S. Hochreiter and J. Schmidhuber, “Long Short-term Memory,” _Neural Comput._ , vol. 9, pp. 1735–80, 1997.
  * [29] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly _et al._ , “Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions,” in _ICASSP_ , 2018.
  * [30] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an ASR corpus based on public domain audio books,” in _ICASSP_ , 2015.
  * [31] J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu _et al._ , “Libri-Light: A benchmark for ASR with limited or no supervision,” in _ICASSP_ , 2020.
  * [32] D. van Niekerk, C. van Heerden, M. Davel, N. Kleynhans, O. Kjartansson _et al._ , “Rapid development of TTS corpora for four South African languages,” in _Interspeech_ , 2017.
  * [33] T. I. Modipa, M. H. Davel, and F. de Wet, “Implications of Sepedi/English code switching for ASR systems,” in _PRASA_ , 2013.
  * [34] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _ICLR_ , 2015.
  * [35] L. N. Smith and N. Topin, “Super-convergence: Very fast training of neural networks using large learning rates,” in _SPIE AIML MDO_ , 2019.
  * [36] K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak _et al._ , “Generative spoken language modeling from raw audio,” _arXiv preprint arXiv:2102.01192_ , 2021.
  * [37] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in _ICML_ , 2006.
  * [38] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross _et al._ , “fairseq: A fast, extensible toolkit for sequence modeling,” in _NAACL-HLT_ , 2019.
  * [39] L. Wan, Q. Wang, A. Papir, and I. L. Moreno, “Generalized end-to-end loss for speaker verification,” in _ICASSP_ , 2018.


# UIF: An Objective Quality Assessment for Underwater Image Enhancement

Yannan Zheng  College of Physics and Information Engineering, Fuzhou
UniversityFujianChina<EMAIL_ADDRESS>,  Weiling Chen  College of
Physics and Information Engineering, Fuzhou UniversityFujianChina
<EMAIL_ADDRESS>,  Rongfu Lin  College of Physics and Information
Engineering, Fuzhou UniversityFujianChina<EMAIL_ADDRESS>and  Tiesong
Zhao  College of Physics and Information Engineering, Fuzhou
UniversityFujianChina<EMAIL_ADDRESS>
(2021)

###### Abstract.

Due to complex and volatile lighting environment, underwater imaging can be
readily impaired by light scattering, warping, and noises. To improve the
visual quality, Underwater Image Enhancement (UIE) techniques have been widely
studied. Recent efforts have also been contributed to evaluate and compare the
UIE performances with subjective and objective methods. However, the
subjective evaluation is time-consuming and uneconomic for all images, while
existing objective methods have limited capabilities for the newly-developed
UIE approaches based on deep learning. To fill this gap, we propose an
Underwater Image Fidelity (UIF) metric for objective evaluation of enhanced
underwater images. By exploiting the statistical features of these images, we
present to extract naturalness-related, sharpness-related, and structure-
related features. Among them, the naturalness-related and sharpness-related
features evaluate visual improvement of enhanced images; the structure-related
feature indicates structural similarity between images before and after UIE.
Then, we employ support vector regression to fuse the above three features
into a final UIF metric. In addition, we have also established a large-scale
UIE database with subjective scores, namely Underwater Image Enhancement
Database (UIED), which is utilized as a benchmark to compare all objective
metrics. Experimental results confirm that the proposed UIF outperforms a
variety of underwater and general-purpose image quality metrics.

Image Quality Assessment (IQA), Underwater Image Enhancement (UIE), underwater
image processing

††copyright: acmcopyright††journalyear: 2021††doi:
10.1145/1122445.1122456††conference: Chengdu ’29: ACM Multimedia; October
20–24, 2021,; Chengdu, China††booktitle: Chengdu ’29: ACM Multimedia, October
20–24, 2021, Chengdu, China††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††submissionid: 2657††ccs: Computing
methodologies-¿Image Processing

##  1\. Introduction

The underwater optical images bring additional information beyond sonar
imaging. However, the complex waterbody and poor light conditions impair the
visual quality of underwater images. In practice, Underwater Image Enhancement
(UIE) technique is thus necessary to transfer low-quality underwater images to
high-quality ones. As shown in Fig. 1, diversified enhancements are employed
in original underwater images, resulting in pictures with higher visual
qualities. To compare the UIE algorithms and select optimal results, it is
imperative to score these enhanced images, which is still a challenging task.

(a) Original image

(b) 0.7541/1.7745

(c) 0.6326/1.5410

(d) 0.6254/1.4213

(e) Original image

(f) 0.6321/1.6217

(g) 0.6441/1.6745

(h) 0.5865/1.5233

(i) Original image

(j) 0.6458/1.6245

(k) 0.6605/1.7614

(l) 0.6139/1.4705

Figure 1. Typical underwater images (the first left column) and their scored
enhancements (the other three columns), where the scores of each enhanced
image are respectively given by UCIQE and UIQM. How to compare the visual
quality of enhanced underwater images is still a challenging task.

Until now, the visual quality of images can be evaluated by Image Quality
Assessment (IQA), whose popular works include subjective and objective
methods. For most of images, human is the ultimate receiver, thus the
subjective evaluation is considered to be the most accurate and reliable way
of IQA. Recently, ITU has promoted several methods to subjectively evaluate
image quality (I and ITUR Recommendation, 2012). However, the subjective
evaluation also has significant drawbacks: high complexity, high cost and
unable to be embedded into real-world systems. As a result, the subjective
methods are usually utilized to set benchmark to evaluate objective metrics,
and the objective metrics with higher correlations to subjective scores are
embedded into real-world systems.

Existing objective IQA can be classified into full-reference, reduced-
reference and no-reference algorithms, subject to the accessibility of ideally
unimpaired references. In underwater imaging, an ideally unimpaired image is
unable to be obtained, thus the typical reference-based IQA approaches, such
as Structural Similarity Index (SSIM) (Wang et al., 2020) and Feature
Similarity Index (FSIM) (Zhang et al., 2011), are not applicable. On the other
hand, no-reference IQA approaches have achieved significant performances to
evaluate generic images (Mittal et al., 2012; Mittal et al., 2013; Gu et al.,
2015b; Zhang et al., 2015; Yang et al., 2016). However, these no-reference
approaches are usually designed based on Natural Scene Statistics (NSS), which
performs different in underwater environment. Thus, they also fail to evaluate
the quality of underwater images.

The Underwater Color Image Quality Evaluation (UCIQE) (Yang and Sowmya, 2015)
and Underwater Image Quality Measure (UIQM) (Panetta et al., 2016) metrics
have been widely used to qualify the enhancement performance of underwater
images (Li et al., 2016; Chang et al., 2019; Peng and Cosman, 2017; Li and
Anwar, 2019). Among them, the UCIQE metric quantifies non-uniform color casts,
blurring, and noise in underwater images, and then combines them in a linear
manner. The UIQM metric evaluates underwater images with color, sharpness and
contrast. These two methods have been proved to achieved good performances in
most underwater scenarios at that time. However, this decade has witnessed a
booming of UIE algorithms, especially deep-learning-based UIE algorithms. In
such case, the UIE may generate complicated color and structure changes which
are unable to be evaluated by conventional metrics. From Fig. 1, a higher
UCIQE/UIQM score may not represent a better visual quality. For example, some
images have obvious reddish color shifts and artifacts as shown in Fig 1. (b),
(g) and (k), but these images obtain better UCIQE and UIQM scores. Therefore,
it is highly desirable to design an effective IQA metric for objective UIE
evaluation.

To address this issue, we exploit the characteristics of underwater images to
obtain three types of features: naturalness-related, sharpness-related and
structure-related features, which are demonstrated to be effective to assess
the underwater image quality. The Support Vector Regression (SVR) are then
utilized to fuse all features into a final metric. In addition, we also
develop the largest-ever UIE quality database with subjective scores, which
helps to guarantee the generalization performance of our model. It can also
serve as a benchmark to compare subjective IQA metrics for UIE. In summary,
the main contributions of this paper include:

  1. 1)

A large-scale Underwater Image Enhancement Database (UIED), which is the
first-of-its-kind database with human-labeled quality scores. The database
includes different underwater scenes that are enhanced by 10 representative
UIE approaches. The UIED can be utilized as a benchmark to develop and
evaluate subjective methods of underwater image quality assessment.

  2. 2)

Three types of image features to evaluate the underwater image fidelity in
three dimensions. An optimal UIE should generate natural images with sharp
textures and high structure similarity to original image. Inspired by this, we
propose to extract and model the three types of features based on statistics
of underwater images. The effectiveness of features has been proved in
ablation study.

  3. 3)

An SVR-based fusion of features to obtain our Underwater Image Fidelity (UIF)
metric. We employ the popular SVR models for feature pooling and regression.
To ensure the generalization performance of our UIF metric, it is trained and
examined by k-fold validation in UIED. Experimental results reveal the
efficiency of our method.

##  2\. RELATED WORK

In this section, we review related works of our paper. Among them, the UIE
methods are utilized to construct our UIED database. The conventional IQA
methods and Underwater IQA methods are compared in our experiments.

###  2.1. Underwater Image Enhancement Methods

In recent years, many UIE methods have been proposed. The existing UIE methods
can be divided into three categories: Non-physical-model-based methods (Ancuti
et al., 2012), (Fu et al., 2014), (Fu et al., 2017), physical-model based
(Drews et al., 2016), (Peng et al., 2018), (Peng and Cosman, 2017), (Li et
al., 2016) and deep-learning-based methods(Li and Anwar, 2019), (Li et al.,
2020).

Non-physical-model-based methods aim to adjust input image pixel values to
improve visual quality. Ancuti _et al._ (Ancuti et al., 2012) proposed a UIE
algorithm with multi-scale fusion strategy. Fu _et al._ (Fu et al., 2014)
proposed a retinex-based UIE method, which decomposes the reflectance and
illumination of underwater images. In (Fu et al., 2017), they also proposed a
two-step enhancement procedure, which includes a color correction and a
contrast enhancement.

Physical-model-based methods construct physical models for underwater
degradation. Li _et al._ (Li et al., 2016) proposed a contrast enhancement
algorithm which combined with an image dehazing model. In (Drews et al.,
2016), an Underwater Dark Channel Prior (UDCP) was proposed based on the fact
that the information of red channel is undependable. Peng _et al._ (Peng et
al., 2018) proposed a Generalized Dark Channel Prior (GDCP) that incorporates
adaptive color correction into an image formation model for UIE. Based on
image blurriness and light absorption, they also (Peng and Cosman, 2017)
proposed a depth estimation method for underwater scenes.

Deep-learning-based methods have led a fast development and offered state-of-
the-art performance in many UIE tasks. Li _et al._ (Li and Anwar, 2019)
proposed an end-to-end deep network, namely UWCNN, to address the UIE problem
for diverse underwater images. In (Li et al., 2020), they also designed a
Water-Net model which is trained on paired underwater images and the
corresponding reference images.

In addition, the commercial application dive+ (Apple, 2020) has also been
released in iTunes store, with good performances of underwater enhancement.

###  2.2. Conventional IQA Methods

IQA plays an important role in many computer vision problems. In recent years,
metrics have been used to evaluate enhancement or restoration performance for
underwater images.

Most traditional no-reference IQA metrics are based on NSS regularities. Anish
_et al._ extracted effective statistical features to evaluate image quality
(Mittal et al., 2012). The Natural Image Quality Evaluator (NIQE) model
proposed by Mittal _et al._ (Mittal et al., 2013) extracted a set of local
features from an image. Gu _et al._ (Gu et al., 2015b) designed No-reference
Free Energy Robust Metric (NFERM) by adding features of Human Vision System
(HVS). Gu _et al._ (Gu et al., 2015a) solved the no-reference IQA for blur
images using a sharpness metric in autoregressive parameter space. Xue _et
al_. (Xue et al., 2014) combined gradient magnitude with laplacian of gaussian
to predict image quality. Kang _et al._ (Kang et al., 2014) investigated a
Convolutional Neural Network (CNN) to jointly learn features for IQA. However,
an underwater image is always degraded by light absorption and scattering,
which may not be captured by these NSS models effectively. Thus, these NSS-
based approaches are not effective enough for underwater images.

Meanwhile, some no-reference IQA methods have been proposed for enhanced image
quality assessment. Fang _et al._ (Fang et al., 2015) proposed a blind
contrast quality metric based on image skewness, kurtosis, and entropy. Gu _et
al._ (Gu et al., 2018) proposed a blind metric according to 17 features
through analysis of images. These methods may not perform well in underwater
IQA due to lack of consideration on the underwater imaging models and image
features. Liu _et al._ (Liu et al., 2019) used synthetic data to generate the
ideal reference image, thereby converting the evaluation of enhanced images
into a full-reference IQA problem. However, its synthetic data only cover very
limited types of underwater conditions, which may be insufficient to simulate
underwater image distortions. Thus, these methods are also not reliable to
apply for underwater image enhancement.

###  2.3. Underwater IQA Methods

Until now, there are two underwater IQA methods that are widely applied to
evaluate UIE approaches. They are the UCIQE (Yang and Sowmya, 2015) and UIQM
(Panetta et al., 2016) metrics.

Yang _et al._ designed the UCIQE metric in (Yang and Sowmya, 2015). The UCIQE
transforms an underwater image from RGB color space to CIELab color space,
which is more consistent with HVS. It quantifies non-uniform color casts,
blurring and noise in underwater monitor images and then linearly combines
these three components. A higher UCIQE score indicates the result has a better
balance among the chroma, saturation and contrast.

Panetta _et al._ proposed the UIQM inspired by HVS in (Panetta et al., 2016).
The UIQM comprises a colorfulness measure, asharpness measure, and a contrast
measure for underwater images, based on the underwater image modeling
presented in (Panetta et al., 2016). The choice of weighted coefficients
depends on the application purpose. A higher UIQM score denotes the result is
more consistent with human visual perception.

As shown in Fig. 1, the two metrics also have lower correlations to human
scores when the enhancement distortion is complex. However, due to the wide
applications of deep learning models, the UIE technique has inevitably brought
complex warping and color changes. To model the impacts of these complex
distortions, we suggest to utilize a deep learning network. This also requires
a large-scale database to train and test the deep-learning-based model.

##  3\. PROPOSED UIED DATABASE

There is a lack of publicly available large-scale UIE database with human
subjective scores. To fill this void, the UIED dataset is developed as the
largest-ever subjective database of UIE images, which can serve as a benchmark
to develop and evaluate objective approaches. In this section, we will
elaborate the preparation of images, subjective test and post-processing to
construct this database.

###  3.1. Preparation of Enhanced Underwater Images

We select 100 authentic underwater images from Google and related works (Xiao
et al., 2010), (Li et al., 2020). These underwater images are taken from real
underwater scenes, with resolutions ranged from 183 $\times$ 275 to 1350
$\times$ 1800\. Typical underwater images are presented in Fig. 2. To generate
more enhanced images for test, we utilize 10 representative UIE algorithms,
including 3 non-physical-model-based methods (_i.e._ fusion-based (Ancuti et
al., 2012), retinex-based (Fu et al., 2014), two-step-based (Fu et al.,
2017)), 4 physical-model-based methods (_i.e._ histogram prior (Li et al.,
2016), UDCP (Drews et al., 2016), UIBLA (Peng and Cosman, 2017), GDCP (Peng et
al., 2018)), 2 deep-learning-based methods (_i.e._ UWCNN (Li and Anwar, 2019),
Water-Net (Li et al., 2020) and 1 commercial application (_i.e._ dive+ (Apple,
2020)). With the 100 images and 10 UIE approaches, we have a total of 1,000
enhanced underwater images. All enhancement underwater images and the
corresponding raw images are included in the UIED.

(a) Coral

(b) Marine life

(c) Seabed rock

(d) Sculpture

(e) Wreck

(f) Diver

Figure 2. Typical underwater images in UIED, including 16 coral images, 26
marine life images, 14 seabed rock images, 12 sculpture images, 10 wreck
images and 22 diver images.

###  3.2. Subjective Test

We implement a subjective quality study to evaluate the 1,000 enhanced images.
In the subjective quality evaluation, we adopt a single-stimulus strategy and
a five-level quality scale to label images (I and ITUR Recommendation, 2012).
A lower rating score represents a worse perceptual visual quality, and vice
versa. A detailed description of the rating criteria is given in Table 1.

The subjective test consists of 10 sessions and in each session, 100 of the
1,000 images are evaluated. At the beginning of each session, a subject
watches and evaluates 5 prescreen images to get familiar to testing
environment and procedure. Then, the subject is asked to score the following
100 images in a random order. Each image is shown for 7 seconds before next.
The subject can take a 5-minute break between sessions to avoid visual
fatigue.

There are totally 16 subjects including 9 males and 7 females in the
subjective test. They all have prior knowledges of image processing. All test
images are shown in random order with a MATLAB graphical user interface. The
images are displayed in laboratory environment with normal illumination
conditions. All of the above conditions are set and calibrated according to
the recommendations of ITU-R (I and ITUR Recommendation, 2012). Table 2 lists
an overview of the test methodology and conditions.

Table 1. Rating Criteria for Subjective Test Level | Description
---|---
1 | The target is invisible, and color severely distorts.
2 | The target is invisible, and color partially distorts.
3 | The target is visible, and color slightly distorts.
4 | The target is visible, some flaws in visual quality.
5 | The target is clearly, and visual quality is great.
Table 2. Detailed Setting of Subjective Experiment Category | Item | Detail
---|---|---
Display | Monitor | AOC 24n2h LCD
Resolution | 1920 × 1080
Platform | Matlab R2016b
Methodology | Method | Single-stimulus
Quality scale | 5-level categorical
Order | Random
Test settings | Groups | 10
Subjects number | 9 males / 7 females
Time interval | 7 seconds
Environment | Laboratory

###  3.3. Data Post-Processing

To show the reliability of data, we choose to use Normalized Cross Correlation
(NCC) and Euclidean distance (EUD) to evaluate the agreement of subject
ratings (Ma et al., 2012). A higher NCC value or a lower EUD indicates higher
correlation between two subjective rating vectors. For the above 16 subjective
ratings, the average value of NCC and EUD is 0.961 and 0.055, respectively.
Therefore, the subject ratings are agreed on the perceptual qualities of these
images and the testing results are reliable.

(a) Original image

(b) MOS:23.94

(c) MOS:42.89

(d) MOS:52.76

(e) Original image

(f) MOS:28.80

(g) MOS:37.61

(h) MOS:56.40

(i) Original image

(j) MOS:33.08

(k) MOS:22.42

(l) MOS:58.77

Figure 3. The MOS values of images in Fig. 1.

We then follow the steps in (Min et al., 2017) to process the subjective
ratings. Rating for an image is considered as outlier if it is outside 2 or
$\sqrt{20}$ standard deviations of the mean rating of that image. A subject
with more than 5% outlier evaluations is rejected. Both outlier ratings and
outlier subjects are excluded from the following processing. The normalized
ratings for an image are averaged over all valid subjects to the Mean Opinion
Score (MOS). The MOS of the database is almost ranged from 20 to 80. That is,
our collected subjective scores span a wide range from low to high scores.

Fig.3 shows the MOS values corresponding to the enhancement results in Fig.1.
It can be seen that our subjective experiment rated has a high correlation to
HVS. On the other hand, the popular UCIQE and UIQM show inferior performances
that still have room to be further improved.

##  4\. Proposed UIF Metric

Based on the UIED, we are able to develop the UIF metric for a more accurate
quantitative evaluation of enhanced underwater images. The design philosophy
lies in three key aspects, including naturalness, sharpness, and structure of
an enhanced underwater image. In this section, we present to attract features
to characterize the naturalness, sharpness and structure, and further
integrate them to infer the ultimate quality score. Details of our method are
presented as follows.

###  4.1. Naturalness-Related Features

In most cases, the natural underwater images have low color richness due to
light absorption and scattering. Thus, many UIE methods aim to eliminate the
color attenuation from vision and improve color richness. However,
overemphasis on color may also result in unreasonable color or an unnatural
look. Therefore, we employ the NSS regularities to approximately capture the
attributes of naturalness of underwater images. In addition, researches have
shown a good correlation between underwater colors and human perception in
CIELab space (Yang and Sowmya, 2015). Inspired by this, we also transfer the
underwater images to CIELab space to calculate the naturalness-related
features.

_Distribution of NSS model._ To detect naturalness-related features in
enhanced underwater images, we first choose the NSS model which is frequently
used in IQA tasks (Yang et al., 2016; Li and Hu, 2017). It is defined on the
brightness of image

(1)
$\begin{split}f(x;\nu,\sigma^{2})=\dfrac{\nu}{2\delta\Gamma(\dfrac{1}{\theta})}e^{(-(\dfrac{|x|}{\theta})^{\nu})},\end{split}\
$

where $\delta$ and $\Gamma(\cdot)$ are defined as:

(2)
$\begin{split}\theta=\sigma\sqrt{\dfrac{\Gamma(1/\nu)}{\Gamma(3/\nu)}},\end{split}\
$

(3)
$\begin{split}\Gamma(\theta)=\int_{0}^{\infty}t^{\theta-1}e^{-t}dt\quad\theta>0.\end{split}\
$

The parameter $\nu$ and $\sigma^{2}$ control the shape and variance of the
distribution, respectively, which are collected to describe image naturalness.

_Contrast and variance in CIELab._ We transform the underwater image from RGB
to CIELab, which is a uniform color space. Considering the luminance contrast
is one of the most sensitive factors in poor illumination, we calculate the
luminance contrast for enhanced images in turbid underwater environment. It is
obtained by the ratio between highest and lowest luminance values

(4) $\begin{split}\sigma_{\rm
CIE}=\dfrac{1}{N}\sum_{i=1}^{N}\dfrac{\sqrt{\alpha_{i}^{2}+\beta_{i}^{2}}}{L_{i}},\end{split}\
$

where $\alpha$ and $\beta$ are the channel parameters of CIELab space, $L$ is
value in luminance channel and $N$ presents the number of image pixels.

In summary, the naturalness-related features include

(5) $\begin{split}\emph{F}_{\rm naturalness}=\\{\nu,\sigma^{2},C_{\rm
CIE},\sigma_{\rm CIE}\\}.\end{split}\ $

###  4.2. Sharpness-Related Features

A high-fidelity image is usually with rich details that are characterized by
sharpness of edges and pixels. In underwater imaging, forward scattering may
affect image details and blurs edges. As a result, an important objective of
UIE is to improve the sharpness of edges and thus present high-quality
pictures with clear objects. To characterize the sharpness of an enhanced
image, we exploit the following features including Dark Channel Prior (DCP)
index, contrast, edge contrast and entropy.

_DCP Index._ The index was utilized to capture image contrast during haze
removal (He et al., 2011). In underwater environment with low light
intensities, dark images are usually captured, which might hinder object
recognition by humans or algorithms. Therefore, the UIE approaches are
designed to enhance the lightness of underwater images, which changes the DCP
channel pixels. Inspired by this, we compute the average value of DCP channel
pixels as:

(6) $\begin{split}\mu_{\rm dark}={\rm
average}\\{{\underset{c\in{R,G,B}}{\min}I_{c}\\}},\end{split}\ $

where $c\in R,G,B$ indicates the RGB channels of enhanced image $I$.

_Contrast._ The contrast has been utilized in visual enhancement of underwater
images (Li et al., 2016). First of all, the enhanced image is divided into
non-overlapped grayscale patches with size $64\times 64$. Then, a patch is
labeled as a textured patch if its edge density, i.e. the ratio between edge
pixels and all pixels within the patch, is larger than 0.2%. Finally, the
overall contrast is obtained as the sum of standard variances of all textured
patches:

(7) $\begin{split}C=\sum_{i=1}^{M}{\rm std\\_var}(P_{i,j}),\end{split}\ $

where $P_{i,j}$ is the $i,j$-th patch of image and $M$ represents the number
of textured patches.

_Edge contrast._ The edge information has been widely utilized to assess
fidelities of images (Zhan and Zhang, 2018; Li et al., 2018). In this work, we
utilize a simple but efficient edge extraction for ease of calculation. First
of all, the Canny edge detector is applied to all channels of image to obtain
three edge maps. Then, each edge map is equally divided into $m\times n$ non-
overlapped blocks with size 5$\times$5\. Finally, the contrasts of all blocks
are calculated and averaged to obtain a final measure within edge maps
(Panetta et al., 2014):

(8) $\begin{split}C_{\rm edge}=\sum_{c=1}^{3}\lambda_{c}\cdot\
\dfrac{2}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}\log(\dfrac{{\rm max}B_{c,i,j}}{{\rm
min}B_{c,i,j}}),\end{split}\ $

where $B_{c,i,j}$ represents the $i,j$-th edge block in channel $c$, and
$\max$ and $\min$ are to calculate the extreme values within the edge block.
$\lambda_{c}$ is a coefficient for RGB color channels.

_Entropy._ As a classic image measurement, the entropy characterizes diversity
of image pixels. Therefore, it has a high correlation to image contrast. For
the enhanced image with $L$ brightness levels, the entropy is defined by

(9) $\begin{split}E=-\sum_{i=0}^{L-1}p_{i}\log p_{i},\end{split}\ $

where $p_{i}$ is the histogram probability of brightness value $i$.

In summary, the sharpness-related features include

(10) $\begin{split}F_{\rm sharpness}=\\{{\mu_{\rm dark},C,C_{\rm
edge},E}\\}.\end{split}\ $

Figure 4. An example of structural artifact introduced by UIE.

(a) original image

(b) over-enhanced image

(c) variance similarity map

Figure 5. An example of over-enhanced underwater images and variance
similarity map. (a) original image, (b) over-enhanced image, (c) variance
similarity map. Figure 6. The detailed framework of the proposed UIF method.

###  4.3. Structure-Related Features

The structural similarity between original and enhanced images is critical
because the UIE algorithms, especially the deep-learning-based UIE, may
introduce structural artifacts during the enhancement process. An example is
shown in Fig. 4, where unwanted textures are introduced due to color changes.
These artifacts may severely degrade visual quality of enhanced images.
Therefore, we need to measure the structural similarity between original and
enhanced images. It is noted that the image structural similarity was firstly
introduced by SSIM. To evaluate the similarity between an image and its
distorted version. Inspired by this index, we calculate our structure-related
features with brightness values of original and enhanced underwater images.

_Variance similarity._ In most cases, over enhancement of underwater images
will cause large variance of brightness, with an example shown in Fig. 5 (b).
In such case, we can construct a variance similarity measure to identify these
over-enhanced regions. Similar to SSIM, we define the variance similarity as:

(11)
$\begin{split}s_{\sigma}=\dfrac{2\sigma_{I}\sigma_{0}+c_{1}}{\sigma_{I}^{2}+\sigma_{0}^{2}+c_{1}},\end{split}\
$

where $\sigma_{I}$ and $\sigma_{0}$ are the variances of enhanced and original
images, respectively. They are calculated locally with a 7$\times$7 window.
$c_{1}$ is a small constant to avoid zero denominators. With this measure, we
can calculate the similarity between Fig. 5 (a) and (b) to obtain the variance
map shown in Fig. 5 (c). It can be readily seen that the over-enhanced regions
with large brightness changes are labeled with low variance similarities.

_Mean and normalized similarities._ The local mean $\mu$ and normalized image
$\bar{I}$ can describe the perceptual similarity of textured regions (Gu et
al., 2017), where the normalized image is calculated by
$(I-\mu(I))/\sigma(I)$. Therefore, we also utilize the similarities between
mean and normalized images as supplementary indexes:

(12)
$\begin{split}s_{\mu}=\dfrac{2\mu_{I}\mu_{0}+c_{2}}{\mu_{I}^{2}+\mu_{0}^{2}+c_{2}},\end{split}\
$

(13)
$\begin{split}s_{\bar{I}}=\dfrac{2\bar{I}_{I}\bar{I}_{0}+c_{3}}{\bar{I}_{I}^{2}+\bar{I}_{0}^{2}+c_{3}},\end{split}\
$

where $\mu_{I}$ and $\mu_{0}$ respectively denote the local mean values of
enhanced and original images, $\bar{I}_{I}$ and $\bar{I}_{0}$ respectively
denote the normalizations of enhanced and original images, $c_{2}$ and $c_{3}$
are small constants to avoid zero denominators.

In summary, the structure-related features include

(14) $\begin{split}F_{\rm
structure}=\\{{s_{\sigma},s_{\mu},s_{\overline{I}}}\\}.\end{split}\ $

###  4.4. Feature Pooling and Regression

Fig. 6 shows the detailed framework of proposed UIF metric, which mainly
consists of two modules: feature extraction and feature regression. In feature
extraction module, we obtain the naturalness-related, sharpness-related and
structure-related features, as discussed above. If a feature is represented by
a two-dimensional map, an average pooling is employed to reduce the
dimensions. All features are then linked into a feature vector:

(15) $\begin{split}F=[\nu,\sigma^{2},C_{\rm
CIE},...,s_{\sigma},s_{\mu},s_{\bar{I}}].\end{split}\ $

In feature regression module, we select the SVR for fusion considering its
success in regression tasks. As illustrated in Fig. 6, we use labeled
underwater enhanced image pairs to train the regressor, which can be utilized
to predict the quality of any input image. Given the features
$F=[\nu,\sigma^{2},C_{\rm CIE},...,s_{\sigma},s_{\mu},s_{\bar{I}}]$, the
corresponding quality label MOS values $S_{i}$ and the training image set, we
can train the regressor by SVR:

(16) $\begin{split}{\rm regressor}={\rm TRAIN}(F_{i},S_{i}).\end{split}\ $

After the training process, we can apply this regressor to yield quality
scores of any testing enhanced image $I$ and original image $I_{0}$ :

(17) $\begin{split}q={\rm PREDICT}(I,I_{0},\rm regressor).\end{split}\ $

LIBSVM (Chang and Lin, 2007) is adopted to implement SVR and has previously
been applied to IQA problems(Fang et al., 2015). In training process, we
choosea Radial Basis Function (RBF) kernel. The other SVR parameters are set
as: penalty coefficient=0.1, $\varepsilon$-insensitive loss function=0.01, and
RBF kernel parameters=1.

##  5\. EXPERIMENTAL RESULTS

In this section, we will evaluate proposed UIF metric with experimental
results. Popular IQA metrics, including general no-reference metrics, enhanced
image quality metrics and UIE quality metrics are examined for comparison. We
also present ablation study to show the effectiveness of all types of
features.

###  5.1. Experiment Settings

To demonstrate the efficiency of our method, we choose a variety of IQA
methods for comparison. They include 9 no-reference IQA methods for natural
images (BRISQUE (Mittal et al., 2012), NIQE (Mittal et al., 2013), NFERM (Gu
et al., 2015b), IL-NIQE(Zhang et al., 2015), SISBLIM (Gu et al., 2014),
BLIINDS-II (Xue et al., 2014), dipIq (Ma et al., 2017), og-iqa (Liu et al.,
2016), CNN-IQA (Kang et al., 2014)), 3 IQA methods for enhanced images (CPCQI
(Gu et al., 2018), BIQME (Gu et al., 2018), and NR-CDIQA (Fang et al., 2015))
and 2 IQA methods for enhanced underwater images (UCIQE (Yang and Sowmya,
2015) and UIQM (Panetta et al., 2016)). For fair comparison, we use the
publicly available codes provided by authors. If a metric is obtained by
machine learning and its training code is available, its parameters are
further tuned for fair comparison. To examine the generalization ability of
machine learning models including our UIF metric, they are examined with
$k$-fold cross-validations, where $k$ is set as 4. A fold of data is examined
and recorded only when it is used as testing test.

The performances of all IQA metrics are evaluated by two commonly used
consistency criteria, including Spearman Rankorder Correlation Coefficient
(SRCC) and Pearson Linear Correlation Coefficient (PLCC). The SRCC scores the
prediction monotonousness to MOS values, while the PLCC scores the linear
correlation between the IQA model’s predictions and MOS values. In particular,
higher PLCC or SRCC indicates an IQA metric is more consistent with subjective
quality evaluations.

###  5.2. Comparisons and Discussions

The comparison results of all IQA metrics are summarized in Table 3, where the
best results and 2nd-best results are highlighted with red bold and blue bold,
respectively. As shown in the table, some no-reference IQA methods are
designed for images in the air (_e.g._ BRISQUE, NFERM, NIQE, IL-NIQE, OG-IQA,
SISBLIM, and BLIINDS-II), thus their extracted NSS characteristics are not
applicable to all underwater images. The accuracy of these no-reference IQA
metrics is low. Even for learning-based metrics such as dipIQ, CNN IQA, the
performance in terms of SRCC and PLCC are not high. This fact implies the
particularity and complexity of underwater image characteristics, which makes
the IQA for UIE images a more challenging problem.

Table 3. Performance Comparison of Selected IQA metrics Type | Methods | SRCC | PLCC
---|---|---|---
General IQA | BRISQUE | 0.465 | 0.496
NFERM | 0.355 | 0.339
NIQE | 0.326 | 0.344
IL-NIQE | 0.393 | 0.347
OG-IQA | 0.216 | 0.233
SISBLIM | 0.277 | 0.321
BLIINDS-II | 0.341 | 0.352
dipIQ | 0.126 | 0.217
CNN IQA | 0.027 | 0.081
IQA for enhancement | CPCQI | 0.266 | 0.284
BIQME | 0.205 | 0.259
CDIQA | 0.276 | 0.292
IQA for UIE | UCIQE | 0.252 | 0.298
UIQM | 0.276 | 0.268
UIF | 0.733 | 0.757

The methods designed for enhanced images (_e.g._ CPCQI, BIQME, and CDIQA) also
show inferior performances in UIE images. This may be because they are
designed to prefer images with high contrasts without consideration to the
special distortions and artifacts in underwater images. As a result, the
features extracted by these methods are not competitive and the experimental
results are not ideal.

Besides, the performances of popular UCIQE and UIQM are still low in UIED. The
possible reasons are as follows. Firstly, these methods were proposed earlier
when few UIE algorithms were developed. The recent booming of UIE approaches
have led to a variety of image distortions and artifacts, which cannot be
considered by early methods. Secondly, they were tuned in small datasets due
to lack of large-scale dataset, which could not be constructed due to lack of
UIE algorithms. Thirdly, they incorporated fewer features to evaluate
underwater images, which limits their performance in diverse underwater
scenarios.

Finally, the proposed UIF metric achieves superior performance to other
metrics. This demonstrates the effectiveness of multiple features and SVR-
based fusion. More importantly, our metric is also benefited from the large-
scale UIED, which is the largest-ever underwater image quality database with
subjective scores.

To present an intuitive comparison between UIF and conventional metrics, we
select four typical underwater images and their enhancements, as shown in Fig.
7. We also calculate the UCIQE and UIF values of all enhanced images and
present them in the same figure. From this figure, the conventional UCIQE
metric may focus on chroma components of images and thus tend to prefer images
with rich colors. It gives high scores to the (b) (f) (l) (p), which have
either red artifacts or contrast distortions. In contrast, the UIF metric has
preferences for (c) (h) (j) (n). This is because the UIF considers
naturalness, contrast and similarity besides of underwater image colors, which
are more consistent with HVS. Therefore, our UIF metric shows the state-of-
the-art performance in quality evaluation of enhanced underwater images.

(a) 0.544/41.59

(b) 0.714/38.63

(c) 0.643/55.54

(d) 0.627/28.37

(e) 0.465/34.10

(f) 0.696/29.51

(g) 0.591/42.21

(h) 0.631/43.09

(i) 0.640/55.54

(j) 0.644/73.81

(k) 0.643/72.60

(l) 0.674/54.54

(m) 0.432/28.37

(n) 0.654/58.25

(o) 0.593/52.37

(p) 0.751/54.97

Figure 7. Typical underwater images (the first left column) and their scored
enhancements (the other three columns), where the scores of each enhanced
image are respectively given by UCIQE and proposed UIF.

###  5.3. Ablation Study

To evaluate the contribution of each type of features, we conduct a series of
ablation experiments. Specifically, the image features merged in the following
methods.

• Method1: Only naturalness-related features.
• Method2: Only sharpness-related features.
• Method3: Only structure-related features.
• Method4: Naturalness-related and sharpness related features.
• Method5: Sharpness-related and structure-related features.
• Method6: Naturalness-related and structure-related features.
• Method7: The UIF with all three types of features.

Table 4 shows the performances of all 7 methods. From the table, we can draw
several conclusions. Firstly, the model achieves acceptable performances even
with one type of feature. Compared with Table 3, the corresponding performance
is still superior to other IQA models in underwater IQA. Secondly, with more
types of features, the proposed model generally improves its prediction
performance in terms of SRCC and PLCC. Thirdly, by a fusion of all types of
features, our UIF metric achieves the best performance of all. This fact
demonstrates the effects of all types of features in our UIF metric. With an
SVR-based regression, the fused index achieves superior performance to predict
subjective evaluations with an objective approach.

Table 4. Performance of Different Feature Groups Feature Groups | Descriptions | SRCC | PLCC
---|---|---|---
Method 1 | Only naturalness | 0.663 | 0.689
Method 2 | Only sharpness | 0.588 | 0.605
Method 3 | Only structure | 0.618 | 0.645
Method 4 | Naturalness and sharpness | 0.676 | 0.691
Method 5 | Sharpness and structure | 0.707 | 0.713
Method 6 | Naturalness and structure | 0.701 | 0.727
Method 7 | All type features | 0.733 | 0.757

###  5.4. Further Applications

The proposed UIF metric can be further applied in other IQA tasks of enhanced
images, with acceptable correlations to subjective scores. In this section, we
test the UIF metric in three other databases, including DHQ (Yeganeh et al.,
2015) for dehazing quality assessment, CID2013(Gu et al., 2013) and CCID2014
(Gu et al., 2016) for contrast-enhanced quality assessments. For comparison,
we keep the same SVR parameter settings and training steps. The training to
testing ratio is set as 80:20. In Table 5, we present average SRCC and PLCC
values for the three databases. Although the existing IQA methods for general
enhanced images cannot work well in underwater IQA, as shown in Table 3, our
UIF metric can still achieve acceptable performances in for IQA of general
enhanced images. This also shows the generalization of our UIF metric to a
certain extent.

Table 5. Evaluations on Other IQA Tasks Databases | Type | SRCC | PLCC
---|---|---|---
CID2013 | Contrast-changed Image | 0.6959 | 0.6795
CCID2014 | Contrast-changed Image | 0.7167 | 0.7031
DHQ | Dehazing Image | 0.6234 | 0.6707

##  6\. CONCLUSIONS

Recent efforts of UIE have greatly promoted the visual quality of underwater
images. Nevertheless, the quality measures of UIE have not been updated to
evaluate the enhanced images by newly developed UIE models. In this paper, we
make the first attempt to build a large-scale underwater image quality
database for UIE, namely UIED, which can be utilized to train and evaluate
objective IQA approaches for enhanced underwater images. Based on this
database, we also propose a new quality metric of underwater images, namely
UIF, which shows the state-of-the-art performance in this scenario.
Experimental results also demonstrate the effectiveness of features and the
generalization ability of UIF. The proposed metric can evaluate the qualities
of enhanced underwater images and also help to select optimal UIE approaches
under different underwater environments.

###### Acknowledgements.

## References

  * (1)
  * Ancuti et al. (2012) Cosmin Ancuti, Codruta O. Ancuti, Tom Haber, and Philippe Bekaert. 2012.  Enhancing underwater images and videos by fusion. In _2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 81–88.
  * Apple (2020) Apple. 2020.  _dive+._ available:https://apps.apple.com/us/app/dive-video-color-correction/id1251506403/.
  * Chang and Lin (2007) ChihChung Chang and ChihJen Lin. 2007.  LIBSVM: A library for support vector machines.  _ACM Transactions on Intelligent Systems and Technology_ 2, 3 (2007).
  * Chang et al. (2019) HerngHua Chang, ChiaYang Cheng, and ChiaChi Sung. 2019\.  Single Underwater Image Restoration Based on Depth Estimation and Transmission Compensation.  _IEEE Journal of Oceanic Engineering_ 44, 4 (2019), 1130–1149.
  * Drews et al. (2016) Paulo L. J. Drews, Erickson R. Nascimento, Silvia S. C. Botelho, and Mario F. Montenegro Campos. 2016\.  Underwater Depth Estimation and Image Restoration Based on Single Images.  _IEEE Computer Graphics and Applications_ 36, 2 (2016), 24–35.
  * Fang et al. (2015) Yuming Fang, Kede Ma, Zhou Wang, Weisi Lin, Zhijun Fang, and Guangtao Zhai. 2015\.  No-Reference Quality Assessment of Contrast-Distorted Images Based on Natural Scene Statistics.  _IEEE Signal Processing Letters_ 22, 7 (2015), 838–842.
  * Fu et al. (2017) Xueyang Fu, Zhiwen Fan, Mei Ling, Yue Huang, and Xinghao Ding. 2017.  Two-step approach for single underwater image enhancement. In _2017 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS)_. 789–794.
  * Fu et al. (2014) Xueyang Fu, Peixian Zhuang, Yue Huang, Yinghao Liao, Xiaoping Zhang, and Xinghao Ding. 2014\.  A retinex-based enhancing approach for single underwater image. In _2014 IEEE International Conference on Image Processing (ICIP)_. 4572–4576.
  * Gu et al. (2017) Ke Gu, Leida Li, Hong Lu, Xiongkuo Min, and Weisi Lin. 2017.  A Fast Reliable Image Quality Predictor by Fusing Micro- and Macro-Structures.  _IEEE Transactions on Industrial Electronics_ 64, 5 (2017), 3903–3912.
  * Gu et al. (2018) Ke Gu, Dacheng Tao, JunFei Qiao, and Weisi Lin. 2018\.  Learning a No-Reference Quality Assessment Model of Enhanced Images With Big Data.  _IEEE Transactions on Neural Networks and Learning Systems_ 29, 4 (2018), 1301–1313.
  * Gu et al. (2016) Ke Gu, Guangtao Zhai, Weisi Lin, and Min Liu. 2016\.  The Analysis of Image Contrast: From Quality Assessment to Automatic Enhancement.  _IEEE Transactions on Cybernetics_ 46, 1 (2016), 284–297.
  * Gu et al. (2014) Ke Gu, Guangtao Zhai, Xiaokang Yang, and Wenjun Zhang. 2014\.  Hybrid No-Reference Quality Metric for Singly and Multiply Distorted Images.  _IEEE Transactions on Broadcasting_ 60, 3 (2014), 555–567.
  * Gu et al. (2015a) Ke Gu, Guangtao Zhai, Xiaokang Yang, and Wenjun Zhang. 2015a.  No-reference image sharpness assessment in autoregressive parameter space.  _IEEE Transactions on Image Processing_ 24, 10 (2015), 3218–3231.
  * Gu et al. (2015b) Ke Gu, Guangtao Zhai, Xiaokang Yang, and Wenjun Zhang. 2015b.  Using Free Energy Principle For Blind Image Quality Assessment.  _IEEE Transactions on Multimedia_ 17, 1 (2015), 50–63.
  * Gu et al. (2013) Ke Gu, Guangtao Zhai, Xiaokang Yang, Wenjun Zhang, and Min Liu. 2013.  Subjective and objective quality assessment for images with contrast change. In _2013 IEEE International Conference on Image Processing_. 383–387.
  * He et al. (2011) Kaiming He, Jian Sun, and Xiaoou Tang. 2011.  Single Image Haze Removal Using Dark Channel Prior.  _IEEE Transactions on Pattern Analysis & Machine Intelligence_ 33, 12 (2011), 2341–2353.
  * I and ITUR Recommendation (2012) Recommendation I and Bt. ITUR Recommendation. 2012.  Methodology for the Subjective Assessment of the Quality of Television Pictures BT Series Broadcasting Service.  _R Rec Bt Ebu Technical Review – April_ (2012).
  * Kang et al. (2014) Le Kang, Peng Ye, Yi Li, and David Doermann. 2014\.  Convolutional Neural Networks for No-Reference Image Quality Assessment. In _2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 1733–1740.
  * Li and Anwar (2019) Chongyi Li and Saeed Anwar. 2019.  Underwater Scene Prior Inspired Deep Underwater Image and Video Enhancement.  _Pattern Recognition_ 98 (2019), 107038.
  * Li et al. (2020) Chongyi Li, Chunle Guo, Wenqi Ren, Runmin Cong, Junhui Hou, and Dacheng Tao Tao. 2020\.  An Underwater Image Enhancement Benchmark Dataset and Beyond.  _IEEE Transactions on Image Processing_ 29 (2020), 4376–4389.
  * Li et al. (2016) ChongYi Li, JiChang Guo, RunMin Cong, YanWei Pang, and Bo Wang. 2016.  Underwater Image Enhancement by Dehazing With Minimum Information Loss and Histogram Distribution Prior.  _IEEE Transactions on Image Processing_ 25, 12 (2016), 5664–5677.
  * Li et al. (2018) Leida Li, Yu Zhou, Ke Gu, Weisi Lin, and Shiqi Wang. 2018.  Quality Assessment of DIBR-Synthesized Images by Measuring Local Geometric Distortions and Global Sharpness.  _IEEE Transactions on Multimedia_ 20, 4 (2018), 914–926.
  * Li and Hu (2017) Yanqing Li and Xinping Hu. 2017.  No-Reference Stereoscopic Image Quality Assessment Using Natural Scene Statistics. In _2017 2nd International Conference on Multimedia and Image Processing (ICMIP)_. 123–127.
  * Liu et al. (2019) Hao Liu, Dong Zhang, Yannan Zhou, and Shaoyi Du. 2019\.  Enhanced image no-reference quality assessment based on color space distribution.  _IET Image Processing_ 9 (2019).
  * Liu et al. (2016) Lixiong Liu, Yi Hua, Qingjie Zhao, and Alan C.Bovik. 2016\.  Blind image quality assessment by relative gradient statistics and adaboosting neural network.  _Signal Processing Image Communication_ 40, 2 (2016), 1–15.
  * Ma et al. (2017) Kede Ma, Wentao Liu, Tongliang Liu, Zhou Wang, and Dacheng Tao. 2017.  dipIQ: Blind Image Quality Assessment by Learning-to-Rank Discriminable Image Pairs.  _IEEE Transactions on Image Processing_ 26, 8 (2017), 3951–3964.
  * Ma et al. (2012) Lin Ma, Weisi Lin, Chenwei Deng, and King N Ngan. 2012\.  Image Retargeting Quality Assessment: A Study of Subjective Scores and Objective Metrics.  _IEEE Journal of Selected Topics in Signal Processing_ 6, 6 (2012), 626–639.
  * Min et al. (2017) Xiongkuo Min, Kede Ma, Ke Gu, Guangtao Zhai, Zhou Wang, and Weisi Lin. 2017\.  Unified blind quality assessment of compressed natural, graphic, and screen content images.  _IEEE Transactions on Image Processing_ 26, 11 (2017), 5462–5474.
  * Mittal et al. (2012) Anish Mittal, Anush K. Moorthy, and Alan C. Bovik. 2012\.  No-Reference Image Quality Assessment in the Spatial Domain.  _IEEE Transactions on Image Processing_ 21, 12 (2012), 4695–4708.
  * Mittal et al. (2013) Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. 2013\.  Making a “Completely Blind” Image Quality Analyzer.  _IEEE Signal Processing Letters_ 20, 3 (2013), 209–212.
  * Panetta et al. (2016) Karen Panetta, Chen Gao, and Sos Agaian. 2016\.  Human-Visual-System-Inspired Underwater Image Quality Measures.  _IEEE Journal of Oceanic Engineering_ 41, 3 (2016), 541–551.
  * Panetta et al. (2014) Karen Panetta, Zohrab A. Samani, and Sos Agaian. 2014\.  Choosing the Optimal Spatial Domain Measure of Enhancement for Mammogram Images.  _International Journal of Biomedical Imaging_ 20, 2 (2014), 937–949.
  * Peng et al. (2018) YanTsung Peng, Keming Cao, and Pamela C. Cosman. 2018\.  Generalization of the Dark Channel Prior for Single Image Restoration.  _IEEE Transactions on Image Processing_ 27, 6 (2018), 2856–2868.
  * Peng and Cosman (2017) YanTsung Peng and Pamela C. Cosman. 2017.  Underwater Image Restoration Based on Image Blurriness and Light Absorption.  _IEEE Transactions on Image Processing_ 26, 4 (2017), 1579–1594.
  * Wang et al. (2020) Zhihao Wang, Jian Chen, and Steven. C. H. Hoi. 2020\.  Deep Learning for Image Super-resolution: A Survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (Early Access)_ (2020).
  * Xiao et al. (2010) Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. 2010\.  SUN database: Large-scale scene recognition from abbey to zoo. In _2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 3485–3492.
  * Xue et al. (2014) Wufeng Xue, Xuanqin Mou, Lei Zhang, and Alan C. Bovik. 2014.  Blind image quality assessment using joint statistics of gradient magnitude and laplacian features.  _IEEE Transactions on Image Processing_ 23, 11 (2014), 4850–4862.
  * Yang et al. (2016) Luping Yang, Haiqing Du, Jingtao Xu, and Yong Liu. 2016\.  Blind image quality assessment on authentically distorted images with perceptual features. In _2016 IEEE International Conference on Image Processing (ICIP)_. 2042–2046.
  * Yang and Sowmya (2015) Miao Yang and Arcot Sowmya. 2015.  An Underwater Color Image Quality Evaluation Metric.  _IEEE Transactions on Image Processing_ 24, 12 (2015), 6062–6071.
  * Yeganeh et al. (2015) Hojatollah Yeganeh, Mohammad Rostami, and Zhou Wang. 2015\.  Objective Quality Assessment of Interpolated Natural Images.  _IEEE Transactions on Image Processing_ 24, 11 (2015), 4651–4663.
  * Zhan and Zhang (2018) Yibing Zhan and Rong Zhang. 2018.  No-Reference Image Sharpness Assessment Based on Maximum Gradient and Variability of Gradients.  _IEEE Transactions on Multimedia_ 20, 7 (2018), 1796–1808.
  * Zhang et al. (2015) Lin Zhang, Lei Zhang, and Alan C. Bovik. 2015.  A Feature-Enriched Completely Blind Image Quality Evaluator.  _IEEE Transactions on Image Processing_ 24, 8 (2015), 2579–2591.
  * Zhang et al. (2011) Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. 2011\.  FSIM: A Feature Similarity Index for Image Quality Assessment.  _IEEE Transactions on Image Processing_ 20, 8 (2011), 2378–2386.