Figure 1. Visual results of our GUPDM on various underwater image types. The
first row displays a series of degraded underwater images ranging from yellow
to blue and the second row shows the enhanced results with our method.

# A Generalized Physical-knowledge-guided Dynamic Model for Underwater Image
Enhancement

Pan Mu  College of Computer Science and Technology, Zhejiang University of
Technology<EMAIL_ADDRESS>,  Hanning Xu  College of Computer Science and
Technology, Zhejiang University of Technology<EMAIL_ADDRESS>,  Zheyuan
Liu  College of Computer Science and Technology, Zhejiang University of
Technology<EMAIL_ADDRESS>,  Zheng Wang  College of Computer Science
and Technology, Zhejiang University of Technology<EMAIL_ADDRESS>,
Sixian Chan  College of Computer Science and Technology, Zhejiang University
of Technology<EMAIL_ADDRESS>and  Cong Bai  College of Computer Science
and Technology, Zhejiang University of Technology<EMAIL_ADDRESS>
(2023)

###### Abstract.

Underwater images often suffer from color distortion and low contrast
resulting in various image types, due to the scattering and absorption of
light by water. While it is difficult to obtain high-quality paired training
samples with a generalized model. To tackle these challenges, we design a
Generalized Underwater image enhancement method via a Physical-knowledge-
guided Dynamic Model (short for GUPDM), consisting of three parts: Atmosphere-
based Dynamic Structure (ADS), Transmission-guided Dynamic Structure (TDS),
and Prior-based Multi-scale Structure (PMS). In particular, to cover complex
underwater scenes, this study changes the global atmosphere light and the
transmission to simulate various underwater image types (e.g., the underwater
image color ranging from yellow to blue) through the formation model. We then
design ADS and TDS that use dynamic convolutions to adaptively extract prior
information from underwater images and generate parameters for PMS. These two
modules enable the network to select appropriate parameters for various water
types adaptively. Besides, the multi-scale feature extraction module in PMS
uses convolution blocks with different kernel sizes and obtains weights for
each feature map via channel attention block and fuses them to boost the
receptive field of the network. The source code will be available at
https://github.com/shiningZZ/GUPDM.

Physical-Knowledge-Guided Model, Deep Learning, Underwater Image Enhancement,
Hyper-parameter Optimization

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
31st ACM International Conference on Multimedia; October 29–November 3, 2023;
Ottawa, ON, Canada.††booktitle: Proceedings of the 31st ACM International
Conference on Multimedia (MM ’23), October 29–November 3, 2023, Ottawa, ON,
Canada††isbn: 979-8-4007-0108-5/23/10††doi: 10.1145/3581783.3612323††ccs:
Networks Network architectures††ccs: Theory of computation Models of
computation††ccs: Computer systems organization Architectures

##  1\. Introduction

Due to the scattering and absorbing effects of the water on light, underwater
images generally suffer from low contrast, color shift and blur. The quality
of underwater images can directly or indirectly affect the accuracy of
underwater robots in performing tasks such as detection, segmentation,
tracking and classification. Therefore, Underwater Image Enhancement (UIE)
plays an important role in underwater tasks.

The existing UIE methods can be coarsely divided into two categories:
traditional and deep learning-based ones. Traditional UIE methods includes
prior-based (Galdran et al., 2015; Drews et al., 2013; Akkaynak and Treibitz,
2018; Chiang and Chen, 2011; Li et al., 2016; Akkaynak and Treibitz, 2019a, b;
Peng and Cosman, 2017; Wang et al., 2017) and model-free (Ancuti et al., 2012;
Ghani and Isa, 2015; Ancuti et al., 2019; Iqbal et al., 2010; Zhang et al.,
2017) approaches. Prior-based ones utilize rich priors and estimate the
parameters of underwater image formation model to generate the enhanced
images. For example, (Drews et al., 2013; Akkaynak and Treibitz, 2018; Chiang
and Chen, 2011; Li et al., 2016) focus on the estimation of medium
transmittance to restore underwater images. Although these methods make full
use of prior information, they are less practical in complex scenarios, which
often leads to over-enhancement of images and thus cannot be applied to real
underwater scenes. Model-free methods often rely on the spatial relationship
between pixel values of the original underwater image to improve the
brightness, contrast and saturation of the image, such as Gray World
(Buchsbaum, 1980), Max RGB (Land, 1977) and White Balance(Liu et al., 1995).
However, these methods tend to ignore the details and depth information of the
image, resulting in artifacts and poor adaptation to the complex underwater
degradations.

Deep learning has introduced new strategies for UIE task (Li et al., 2020; Fu
et al., 2022b; Li et al., 2021; Fu et al., 2022a; Mu et al., 2022; Guo et al.,
2022; Huang et al., 2023; Jiang et al., 2022a; Liu et al., 2022a; Lin et al.,
2021; Jiang et al., 2022b; Lin et al., 2021). Those methods can achieve
complex non-systematic end-to-end modeling or combine physical priors with
networks to solve the existing problems. These methods have better feature
representation capacity, benefiting from large data they utilized to train.
Hypernet (Ha et al., 2016; Lin et al., 2020; Yin et al., 2022; Liu et al.,
2020b, 2022b) is one of the great works among deep learning-based approaches,
which uses a relatively small network to generate weights for target network.
These methods have better feature representation capacity, as they leverage
large data to train. However, the main challenge is the high cost and
difficulty of acquiring large-scale underwater datasets, which forces most
methods to use small-scale datasets. Consequently, many GAN-based approaches
focus on unsupervised UIE (Li et al., 2019b; Guo et al., 2019; Islam et al.,
2020b; Fabbri et al., 2018; Jiang et al., 2022b) models, aiming to synthesize
underwater images by learning a realistic representation of underwater
conditions from unlabeled images. Nevertheless, a significant gap between the
synthesized underwater images and the real ones still remains, in terms of
plausibility and scene diversity.

Overall, we summarize the challenges existing in underwater image enhancement
as follows. 1) Underwater images often suffer from color distortion and low
contrast, due to the scattering and absorption of light by water. Moreover,
the water quality and the distance of light transmission also affect the image
clarity, making the underwater images blurry. 2) The scarcity of high-quality
paired training samples constrains the performance of learning-based models.
3) Model generalization is a critical task for UIE task, which is important
but has been neglected by many researches, as shown in Fig. 2. For example, a
model obtained through training one dataset maybe not suitable for another
water types.

Contributions: To tackle these challenges, this work designs a Generalized
Underwater image enhancement method via Physical-knowledge-guided Dynamic
Model (short for GUPDM), consisting of three parts: Atmosphere-based Dynamic
Structure (ADS), Transmission-guided Dynamic Structure (TDS) and Prior-based
Multi-scale Structure (PMS), as shown in Fig. 3. In particular, to cover
complex underwater scenes, this study varies the global atmosphere and the
transmission, reaching various underwater image types (e.g., the underwater
image color ranging form yellow to blue) through the formation model. Then, we
design a ADS that uses dynamic convolutions to adaptively extract prior
information of underwater images and generate parameters for PMS. In addition,
to force our model to pay more attention to various details in images, this
work introduces a TDS to enable our network to adaptively select suitable
parameters. Thus, the whole network can obtain appropriate parameters
according to the water type of underwater image, making the proposed model
more robust. Besides, the Multi-scale Feature Extraction (MFE) module in PMS
uses convolution blocks with different kernel sizes and obtain weights for
each feature map via channel attention block and then fuses them to boost the
receptive field of network. We summarize the main contributions as follows:

  * •

This work designs a generalized underwater image enhancement method via
physical-knowledge-guided (i.e., various atmosphere and transmission) dynamic
model to adaptively enhance the underwater images with different water types
(as shown in Fig. 1).

  * •

To cover complex underwater scene, by varying global atmosphere light, we
design a ADS that uses dynamic convolutions to adaptively extract prior
information of underwater images and generates parameters for base network
structure. This also allows the entire model to adjust the parameters
according to the degradation level of input image, improving the
generalization capacity of network.

  * •

The proposed TDS uses medium transmittance-based prior to prompt the network
to pay more attention to areas with the most quality degradation, enabling the
network to adaptively select appropriate parameters according to water
quality.

  * •

Extensive experiments demonstrate that our method achieves superior
performance and the best generalization capacity, reaching the state-of-the-
art level on multiple test datasets.

Figure 2. Schematic description of previous deep-learning methods (left) and
our model (right). Figure 3.  Overall illustration of the proposed GUPDM
framework. GUPDM is composed of three components: Atmosphere-based Dynamic
Structure (ADS), Transmission-guided Dynamic Structure (TDS) and Prior-based
Multi-scale Structure (PMS).

##  2\. Related Works

We briefly review previous related works regarding the traditional UIE models
and deep learning-based UIE approaches.

Traditional UIE Models: The Traditional UIE Models can be summarized as prior-
based approaches and model-free approaches. The prior-based approaches are
grounded on physical model intended to estimate the parameters of underwater
image formation model by using visual cues and then reconstruct a clean image
by applying these parameters in reverse. For instance, in (Galdran et al.,
2015), researchers propose a variant of DCP which uses red channel information
to estimate the transmission map of underwater images. UDCP (Drews et al.,
2013) estimates the transmission map by considering the blue and green color
channels. Moreover, Sea-thru (Akkaynak and Treibitz, 2019a) estimates the back
scatter and attenuation coefficient by using RGBD images as input.

Another type of traditional methods is model-free approach which improve the
chromatic aberration and contrast by modifying the overall pixel values of
underwater images, including histogram equalization (Pizer, 1990), white
balance (Liu et al., 1995) and Retinex (Rahman et al., 1996). To fully enhance
the image details and colors, (Ghani and Isa, 2015) proposes a method that
integrates global and local contrast stretching. To handle the artifacts
caused by the severely uneven color spectrum distribution, 3C (Ancuti et al.,
2019) reconstructs the lost channel based on opponent color. These traditional
methods can improve the visual effect to some extent. However, these methods
may result in color distortions and artifacts when encountered with
sophisticated illumination conditions.

UIE Approaches based on Deep-learning: Deep learning methods are trained on
large scale underwater images and can automatically extract relevant features
from them to improve the quality of enhanced images. For example, methods use
Generative Adversarial Network (GAN) for image enhancement includes FGAN (Li
et al., 2019b), DenseGAN (Guo et al., 2019), UGAN (Fabbri et al., 2018) and
FUnIE-GAN (Islam et al., 2020b). The main purpose of GAN strategy is to expand
the source of pairing data through generative networks, but there is still a
lack of high-quality training samples that truly match real underwater
scenarios and diverse degradation. Due to the nonavailability of ground truth
high-quality images, a novel probabilistic network PUIE-Net (Fu et al., 2022b)
is proposed to learn the enhanced distribution of degraded underwater images.
These end-to-end methods can produce visually pleasing results. However, they
usually require a specific model for each dataset and lack generalization
ability and flexibility in handling different underwater scenarios, due to the
complexity of underwater environments. The methods trained on real data can
produce visually pleasing results. However, they cannot restore the color and
structure of specific objects well and tend to produce inauthentic results
since the reference images are not the actual ground truths. To tackle these
problems, some models (Liu et al., 2019, 2018; Li et al., 2020, 2021; Qian et
al., 2022) integrate the priors like transmission map and atmosphere light to
deal with the environmental information. For instance, UWCNN (Li et al., 2020)
is trained on synthesize datasets with underwater scene prior. Instead of
estimating the parameters of underwater imaging model, UWCNN directly
reconstructs clear enhanced images. Ucolor (Li et al., 2021) uses medium
transmission-guided multi-color embedding to solve the color casts and low
contrast issues. DDNet (Qian et al., 2022) uses medium transmission maps and
global atmosphere light to form the haze and scene-adaptive convolutions.

##  3\. Our Developed Framework

This section detailed introduces the developed framework as shown in Fig. 3.
Subsection 3.1 first present the motivation and the problem formulation of
this work. Subsection 3.2 describes the detailed structure of GUPDM and
Subsection 3.3 presents the loss functions and training procedure.

###  3.1. Motivation and Problem Formulation

Following Akkaynak’s light scattering model (Akkaynak and Treibitz, 2019a),
the degraded underwater images can be expressed as

(1)
$\mathbf{I}_{c}(x)=\mathbf{J}_{c}(x)e^{-\bm{\beta}d(x)}+\left(1-e^{-\bm{\beta}d(x)}\right)\mathbf{A}_{c},c\in\\{r,g,b\\}$

where $x$ indicates the spatial location of each pixel, $\mathbf{I}_{c}$ is
the observed image, $\mathbf{J}_{c}$ is the restored haze-free image and
$\mathbf{A}_{c}$ means the global background light, $d(x)$ is the scene depth
at pixel $x$ and $\bm{\beta}$ is the channel-wise extinction coefficient
depending on the water quality. $\mathbf{T}_{c}(x):=e^{-\bm{\bm{\beta}}d(x)}$
is the medium transmission map representing the percentage of scene spoke
brightness that reaches the camera after reflection from point $x$ in the
underwater scene, which also reflect the water type.

Due to the color cast caused by varying light attenuation with different
wavelengths and haze effect caused by scattering, there exist various types of
degraded underwater images (see Fig. 4). In other words, we can roughly
simulate complex underwater scene with the global atmosphere light
$\mathbf{A}_{c}$ and the transmission map $\mathbf{T}_{c}$. Inspired by
aforementioned information, we change the original input underwater images by
adjusting the two parameters. We aim to reduce the color cast of underwater
images through making such changes so that our encoders can learn more
information about the color. This operation can also help to augment the data
to better handle the deficient data problem of underwater image. Thus, we
generate the input images as is shown in Fig. 4. We summarize the re-
degradation types as
$\mathbf{I}_{c}(x)\stackrel{{\scriptstyle(a),(b)}}{{\longrightarrow}}\mathbf{I}_{c}^{m,n}(x)$,
where

  * (a).

Varing $\mathbf{A}_{c}\to\mathbf{A}_{c}^{m}$, based on Eq. (1), we obtain
$\mathbf{I}_{c}^{m}(x)$;

  * (b).

If we further vary $\mathbf{T}_{c}(x)\to\mathbf{T}_{c}^{n}(x)$, we obtain
$\mathbf{I}_{c}^{n}(x)$.

Motivated by the above analysis, this work aims to restore various underwater
images using a generalized model. Therefore, the key is to enable the designed
network ot estimate the features of varied types. We design an Atmosphere-
based Dynamic Structure $\mathcal{A}$ (i.e., ADS) and a Transmission-based
Dynamic Structure $\mathcal{T}$ (i.e., TDS) to jointly guide the Prior-based
Multi-scale Structure $\mathcal{N}$ (i.e., PMS). Actually, the $\mathcal{N}$
acts as a base network structure for UIE while $\mathcal{A}$ and $\mathcal{T}$
are two hyper-guided modules with different level. Thus, we formulate this
problem in the following forms:

(2)
$\begin{array}[]{l}\min\limits_{\bm{\theta},\bm{\phi}}L\left(\widehat{\mathbf{J}}_{c}(x),\bar{\mathbf{J}}_{c}(x);\mathbf{I}_{c}^{m}(x),\mathbf{I}_{c}^{n}(x)\right)\\\
s.t.,\bm{\omega}\in\arg\min\limits_{\bm{\omega}}F(\widetilde{\mathbf{J}}_{c}(x);\mathbf{I}_{c}(x)),\end{array}$

where $\bm{\theta},\bm{\phi}$ and $\bm{\omega}$ are the parameters of module
ADS, TDS and PMS, respectively; $L$ and $F$ are the loss functions;
$\widetilde{\mathbf{J}}_{c}(x)$, $\widehat{\mathbf{J}}_{c}(x)$ and
$\bar{\mathbf{J}}_{c}(x)$ are respectively estimated by:

$\begin{array}[]{l}\widetilde{\mathbf{J}}_{c}(x)\leftarrow\mathcal{N}(\bm{\omega};\mathcal{A}(\bm{\theta},\mathbf{A}_{c}),\mathcal{T}(\bm{\phi},\mathbf{T}_{c});\mathbf{I}_{c}(x)),\\\
\widehat{\mathbf{J}}_{c}(x)\leftarrow\mathcal{N}(\bm{\omega};\mathcal{A}(\bm{\theta},\mathbf{A}_{c}^{m}),\mathcal{T}(\bm{\phi},\mathbf{T}_{c}(x);\mathbf{I}_{c}(x)),\\\
\bar{\mathbf{J}}_{c}(x)\leftarrow\mathcal{N}(\bm{\omega};\mathcal{A}(\bm{\theta},\mathbf{A}_{c}^{m}),\mathcal{T}(\bm{\phi},\mathbf{T}_{c}^{n}(x);\mathbf{I}_{c}(x)).\end{array}$

Figure 4. Generating various re-degraded underwater images.

###  3.2. Physical Knowledge-guided Dynamic Network

In this part, we introduce the developed model in detail, as shown in Fig. 3.
The proposed GUPDM mainly consists of three parts: atmosphere-based dynamic
structure, transmission-guided dynamic structure and prior-based multi-scale
structure. ADS and TDS use dynamic convolutions to adaptively extract prior
information from underwater images and generate parameters for PMS. We combine
the meta-learning strategy to train our physical knowledge-based networks.
This procedure can prompt our GUPDM to attend to different water types. Thus,
the whole network can adaptively select appropriate parameters according to
the water types of input images, achieving more robustness and generalization
capacity.

Atmosphere-guided Dynamic Structure. With the degraded underwater model in Eq.
(1), this work varies the atmosphere light $\mathbf{A}_{c}$ to different
values, i.e., $\mathbf{A}_{c}^{m},\ m=1,\cdots,M$, obtaining $M$ degraded
underwater types:

$\mathbf{I}_{c}^{m}=\mathcal{G}_{\mathbf{A}}(\mathbf{A}_{c})=\lambda_{c}^{m}\mathbf{A}_{c}.$

where $c\in\\{r,g,b\\}$ and $\lambda_{c}^{m}\in(0.3,0.6)$ is randomly
generated degradation level. Then, to adaptively extract atmosphere-based
prior information, we design ADS (i.e., $\mathcal{A}(\bm{\theta})$) with
dynamic convolutions. We formulate this procedure as:

$\mathbf{f}_{\text{ADS}}=\mathcal{A}(\bm{\theta},\mathbf{A}_{c}^{m};\\{\mathbf{I}_{c}^{m}\\}),\
m=1,\cdots,M.$

The ADS consists of duplicated blocks composed of two Dynamic Convolutions
(DC) (Chen et al., 2020) and one Global Average Pooling (GAP) module. The
schematic of DC is depicted in Fig.3 where DC aggregates multiple parallel
convolution kernels dynamically based upon their attentions, boosting the
model complexity without increasing the network depth or width.

Algorithm 1 GUPDM Updating Framework

0: Paired underwater data, necessary initialization parameters.

1: $t=1$;

2: while Not Converge do

3: Use a batch of images $\mathbf{I}_{c}(x)$ as input :

4:
$\widetilde{\mathbf{J}}_{c}(x)\leftarrow\mathcal{N}(\bm{\omega};\mathcal{A}(\bm{\theta},\mathbf{A}_{c}),\mathcal{T}(\bm{\phi},\mathbf{T}_{c});\mathbf{I}_{c}(x))$

5: Evaluate the total loss in
$L_{\text{total}}=L(\widehat{\mathbf{J}},\mathbf{I}_{\text{gt}};\mathbf{I}_{c}(x))$

6: $\\#$ Update the parameters $\bm{\omega}$ of PMS

7:
$\bm{\omega}\leftarrow\bm{\omega}-\rho_{0}\nabla_{\bm{\omega}}L_{\text{total}}$

8: if $(t\ Mod\ t_{0})=0$ then

9: Generating $M$ patches degraded underwater images by
$\mathcal{G}_{\mathbf{A}}$ as input $\mathbf{I}_{c}^{m}(x)$:

10:
$\widehat{\mathbf{J}}_{c}(x)\leftarrow\mathcal{N}(\bm{\omega};\mathcal{A}(\bm{\theta},\mathbf{A}_{c}^{m}),\mathcal{T}(\bm{\phi},\mathbf{T}_{c}(x);\mathbf{I}_{c}(x))$

11: Evaluate the total loss in

12:
$L_{\text{total}}^{A}=\sum_{m=1}^{M}L(\widehat{\mathbf{J}},\mathbf{I}_{\text{gt}};\mathbf{I}_{c}^{m}(x),\mathbf{I}_{c}(x))$

13: $\\#$ Update the parameters $\bm{\theta}$ of ADS

14:
$\bm{\theta}\leftarrow\bm{\theta}-\rho_{1}\nabla_{\bm{\theta}}L_{\text{total}}^{A}$

15: end if

16: if $(t\ Mod\ t_{1})=0$ then

17: Generating $N$ patches degraded underwater images by
$\mathcal{G}_{\mathbf{T}}$ as input $\mathbf{I}_{c}^{n}(x)$:

18:
$\bar{\mathbf{J}}_{c}(x)\leftarrow\mathcal{N}(\bm{\omega};\mathcal{A}(\bm{\theta},\mathbf{A}_{c}^{m}),\mathcal{T}(\bm{\phi},\mathbf{T}_{c}^{n}(x);\mathbf{I}_{c}(x))$

19: Evaluate the total loss in

20:
$L_{\text{total}}^{T}=\sum_{n=1}^{N}L(\bar{\mathbf{J}},\mathbf{I}_{\text{gt}};\mathbf{I}_{c}^{n}(x),\mathbf{I}_{c}(x))$

21: $\\#$ Update the parameters $\bm{\phi}$ of TDS

22:
$\bm{\phi}\leftarrow\bm{\phi}-\rho_{2}\nabla_{\bm{\phi}}L_{\text{total}}^{T}$

23: end if

24: $t=t+1$

25: end while

Transmission-guided Dynamic Structure. We first estimate the transmission map
$\mathbf{T}_{c}(x)$ via general dark channel prior (UDCP (Drews et al.,
2013)). Since the transmission $\mathbf{T}_{c}(x)$ can reflect water types, we
vary it through multiplying $\mathbf{T}_{c}(x)$ with randomly generated
coefficients:

$\mathbf{T}_{c}^{n}(x)=\mathcal{G}_{\mathbf{T}}(\mathbf{T}_{c}(x))=\gamma_{c}^{n}\cdot\mathbf{T}_{c}(x),n=1,\cdots,N,$

where $\gamma_{c}^{n}\in(0.5,1.1)$ is the random coefficient. We then
introduce a TDS (i.e., $\mathcal{T}(\bm{\phi})$) to enable our model to
adaptively select appropriate parameters according to water types while paying
more attention to various image details. We formulate this step as:

$\mathbf{f}_{\text{TDS}}=\mathcal{T}(\bm{\phi},\mathbf{T}_{c}^{n};\\{\mathbf{I}_{c}^{n}\\}),\
n=1,\cdots,N.$

Specifically, $\mathbf{f}_{\text{TDS}}$ and the transmission map
$\mathbf{f}_{\text{m}}$ are integrated in Transmission Guided Feature
Extraction module $\mathcal{M}$ (i.e., TGFE). The TDS has the same base
structure with ADS.

Table 1. Quantitative comparisons on datasets with reference. We employ
experiment on four datasets and three metrics, and achieve the best
performance under most settings. We highlight the best result in black bold
and underline the second ones.

Datasets | Metrics | UDCP (Drews et al., 2013) | Fusion (Ancuti et al., 2012) | Water-Net (Li et al., 2019a) | UGAN (Fabbri et al., 2018) | FUnIEGAN (Islam et al., 2020b) | Ucolor (Li et al., 2021) | USUIR (Fu et al., 2022a) | PUIE_net (Fu et al., 2022b) | Uranker-NU2Net (Guo et al., 2022) | Semi-UIR (Huang et al., 2023) | Ours
---|---|---|---|---|---|---|---|---|---|---|---|---
| PSNR$\uparrow$ | 16.38 | 17.61 | 20.14 | 21.89 | 18.43 | 22.23 | 18.2 | 17.7 | 23.92 | 18.7 | 25.48
Test-E515 | SSIM$\uparrow$ | 0.64 | 0.75 | 0.68 | 0.8 | 0.76 | 0.83 | 0.74 | 0.76 | 0.84 | 0.73 | 0.86
| MSE$\downarrow$ | 1990 | 1331 | 826 | 556 | 1115 | 492 | 1153 | 1424 | 368 | 1015 | 254
| PSNR$\uparrow$ | 13.05 | 17.6 | 19.11 | 20.51 | 16.81 | 15.52 | 20.64 | 19.21 | 20.47 | 22.42 | 21.53
Test-U90 | SSIM$\uparrow$ | 0.62 | 0.77 | 0.79 | 0.79 | 0.74 | 0.67 | 0.85 | 0.87 | 0.85 | 0.89 | 0.86
| MSE$\downarrow$ | 3779 | 1331 | 1220 | 911 | 1778 | 2217 | 716 | 966 | 879 | 522 | 721
| PSNR$\uparrow$ | 12.66 | 14.48 | 17.73 | 24.43 | 19.43 | 17.92 | 19.35 | 19.61 | 24.66 | 22.36 | 25.36
Test-L504 | SSIM$\uparrow$ | 0.62 | 0.79 | 0.82 | 0.86 | 0.81 | 0.75 | 0.85 | 0.86 | 0.92 | 0.86 | 0.93
| MSE$\downarrow$ | 4529 | 3501 | 1361 | 285 | 891 | 1596 | 933 | 840 | 296 | 522 | 261
| PSNR$\uparrow$ | 18.26 | 14.58 | 22.46 | 21.93 | 18.35 | 21.79 | 17.87 | 17.09 | 23.09 | 18.68 | 24.34
Test-U120 | SSIM$\uparrow$ | 0.72 | 0.54 | 0.79 | 0.77 | 0.73 | 0.76 | 0.72 | 0.72 | 0.77 | 0.72 | 0.79
| MSE$\downarrow$ | 1249 | 2968 | 458 | 525 | 1172 | 495 | 1222 | 1633 | 370 | 1005 | 297

Table 2. Averaged unsupervised scores (i.e., PS, UIQM, UCIQE, NIQE and
URanker) on three real-world underwater datasets without reference images
(i.e., Test-C60, Test-R300 and Test-S16).

Datasets | Test-C60 | Test-R300 | Test-S16
---|---|---|---
PS$\uparrow$ | UIQM$\downarrow$ | UCIQE$\uparrow$ | NIQE$\downarrow$ | Uranker $\uparrow$ | PS$\uparrow$ | UIQM$\downarrow$ | UCIQE$\uparrow$ | NIQE$\downarrow$ | Uranker$\uparrow$ | PS$\uparrow$ | UIQM$\downarrow$ | UCIQE$\uparrow$ | NIQE$\downarrow$ | Uranker$\uparrow$
Water-Net (Li et al., 2019a) | 6.45 | 2.86 | 27.39 | 4.97 | 0.93 | 4.45 | 2.58 | 27.49 | 4.91 | 2.17 | 6.18 | 2.11 | 24.66 | 6.41 | 1.99
FUnIEGAN (Islam et al., 2020b) | 4.58 | 4.54 | 30.05 | 6.21 | 1.67 | 4.22 | 4.00 | 28.58 | 4.89 | 2.25 | 6.51 | 3.27 | 27.81 | 6.49 | 2.02
USUIR (Fu et al., 2022a) | 3.94 | 4.71 | 32.04 | 5.74 | 1.14 | 3.66 | 3.34 | 20.34 | 4.68 | 1.83 | 5.31 | 3.30 | 28.24 | 6.37 | 0.42
Fusion (Ancuti et al., 2012) | 4.73 | 2.67 | 32.26 | 5.29 | 1.24 | 3.18 | 3.06 | 31.58 | 4.36 | 2.34 | 7.86 | 2.01 | 31.35 | 12.19 | 0.91
Ucolor (Li et al., 2021) | 6.26 | 4.31 | 24.27 | 5.34 | 0.63 | 4.40 | 3.36 | 20.48 | 4.83 | 0.77 | 8.90 | 3.11 | 27.51 | 11.85 | 0.58
UDCP (Drews et al., 2013) | 6.66 | 1.54 | 32.55 | 5.44 | 0.62 | 4.88 | 2.30 | 28.72 | 5.41 | 0.46 | 7.47 | 1.02 | 33.92 | 9.05 | 0.26
UGAN (Fabbri et al., 2018) | 4.53 | 4.69 | 31.08 | 6.81 | 1.91 | 4.03 | 4.29 | 29.44 | 5.22 | 2.61 | 6.01 | 3.29 | 28.26 | 6.98 | 2.02
PUIE_NET (Fu et al., 2022b) | 4.54 | 3.64 | 27.25 | 6.22 | 1.05 | 4.27 | 4.00 | 26.83 | 4.90 | 1.77 | 6.99 | 3.34 | 27.81 | 7.97 | 0.99
Uranker-NU2Net (Guo et al., 2022) | 4.26 | 4.57 | 28.90 | 5.79 | 1.18 | 4.65 | 4.18 | 28.63 | 4.69 | 2.30 | 5.84 | 3.26 | 30.17 | 6.66 | 2.11
Semi-UIR (Huang et al., 2023) | 3.97 | 4.74 | 30.63 | 5.77 | 1.69 | 4.52 | 4.11 | 29.23 | 4.52 | 2.45 | 5.79 | 2.63 | 30.00 | 6.43 | 1.98
Ours | 6.45 | 4.88 | 32.36 | 4.76 | 1.94 | 4.95 | 4.35 | 31.05 | 4.26 | 2.64 | 8.06 | 3.52 | 31.78 | 6.18 | 2.22
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Prior-based Multi-scale Structure. This structure is a basic part of our
underwater image enhancement. First, to mine the physical knowledge and
features of depth-texture information at different scales, we adopt the prior-
based multi-scale structure to estimate the preliminary pixel feature
$\mathbf{f}_{\text{PMS}}$ in this branch.

$\mathbf{J}_{c}(x)\leftarrow\mathcal{N}(\bm{\omega};\mathcal{A}(\bm{\theta},\mathbf{A}_{c}),\mathcal{T}(\bm{\phi},\mathbf{T}_{c});\mathbf{I}_{c}(x)).$

As for $\mathcal{N}$, it mainly consists of a series of HMH and 3H modules
with a tailored Transmission-Guided Feature Extraction (TGFE) module at its
bottleneck. HMH comprises a Hyper Residual Block (HRB), a Multi-scale Feature
Extraction module (MFE) and another HRB, while 3H consists of three HRBs. The
HRB converts input parameters (i.e. $\mathbf{f}_{\text{ADS}}$ or
$\mathbf{f}_{\text{TDS}}$) to weights of convolution block via a Fully
Connected (FC) layers. The detailed structure of the MFE module is shown in
top right of Fig. 3. To improve the receptive field of the framework, we
utilize three convolution blocks with different kernel sizes $1\times 1$,
$3\times 3$ and $5\times 5$, generating three feature maps. We then employ
global max pooling and average pooling in channel attention block to
automatically extract and fuse the main features from three feature maps at
different scales. TGFE consists of a HRB, a TC, a MFE and another HRB
sequentially, integrating the transmission feature $\mathbf{f}_{m}$ to tackle
the blur and haze.

###  3.3. Loss Functions and Training Procedure

We carefully design the training loss functions to guide the model to produce
enhanced results with minimum color artifacts, blurriness and the closest
details to reference images. Firstly, we use pretrained VGG16 network to
extract the feature in $3^{th},8^{th}$ and $15^{th}$ layers to formulate the
perceptional loss $L_{per}$:

(3) $L_{\text{per}}(\mathbf{J},\mathbf{I}_{\text{gt}})=L_{\text{MSE
}}\left(\operatorname{VGG}_{3,8,15}(\mathbf{J}),\mathrm{VGG}_{3,8,15}\left(\mathbf{I}_{\text{gt}}\right)\right),$

where $L_{\text{MSE }}$ denotes the Mean Square Error (MSE). We also impose
the smooth $L_{1}-\operatorname{loss}$ as reconstruction loss since we notice
that unnecessary interference often occurs from the background color:

(4)
$L_{1}(\mathbf{J},\mathbf{I}_{\text{gt}})=\frac{1}{n}\sum_{i=1}^{n}|\mathbf{J}(i)-\mathbf{I}_{\text{gt}}(i)|,$

where $n=H\times W$ is the overall pixel number. We furthermore apply the SSIM
loss to focus more on the structural details. Thus, the total loss function
can be summarized as follows:

(5)
$L(\mathbf{J},\mathbf{I}_{\text{gt}})=\ell_{1}+\lambda_{1}\mathcal{L}_{\text{SSIM
}}+\lambda_{2}\mathcal{L}_{\text{per }}$

where $\lambda_{1}$ and $\lambda_{2}$ stand for the weights of each loss
functions.

We illustrate the training procedure of proposed method in Algorithm 1. In
particular, we first train the base-net PMS with given atmosphere
$\mathbf{A}_{c}$ and transmission $\mathbf{T}_{c}$ and loss function
$L(\widetilde{\mathbf{J}},\mathbf{I}_{\text{gt}};\mathbf{I}_{c}(x))$. Next, we
fix the base-net PMS and update parameter of ADS (i.e., $\bm{\theta}$) which
is optimized with loss
$L(\widehat{\mathbf{J}},\mathbf{I}_{\text{gt}};\mathbf{I}_{c}^{m}(x),\mathbf{I}_{c}(x))$.
Similarly, the parameter of TDS (i.e., $\bm{\phi}$ ) is updated through
optimizing
$L(\bar{\mathbf{J}},\mathbf{I}_{\text{gt}};\mathbf{I}_{c}^{n}(x),\mathbf{I}_{c}(x))$.
The ADS, TDS and PMS are trained in an alternate manner until they converge.

##  4\. Experimental Results

This section first introduces the implementation details in Subsection 4.1.
Then, to evaluate the performance when comparing with existing state-of-the-
art approaches, a series of qualitative and quantitative assessments are
conducted in Subsection 4.2. To analyze the developed method, we conduct
various ablation studies to verify the effectiveness of different branches in
Subsection 4.3. _More experimental results are provided in our supplementary
material_.

###  4.1. Implantation Details

Datasets. Our model is trained on LSUI (Peng et al., 2023), a real underwater
dataset, which consisting 4500 training pairs and 504 test pairs (i.e.
Test-504). In order to evaluate the effectiveness and robustness of our
proposed method, we test it on six real world datasets: Test-U90 (i.e. UIEB
(Li et al., 2019a)), Test-L504 (i.e. LSUI (Peng et al., 2023)), Test-U120
(i.e. UFO (Islam et al., 2020a)), Test-S16 (i.e. SQUID (Berman et al., 2020)),
Test-R300 (i.e. RUIE (Liu et al., 2020a)) and Test-C60 (Li et al., 2019a), and
a synthetic one: Test-E515 (i.e. EUVP (Islam et al., 2020c)). There are two
categories of testing datasets: those that have reference images (either real
or generated by another method) and those that don’t. The first type includes
Test-E515, Test-U90, Test-L504 and Test-U120 datasets, where Test-E515 uses
synthetic images created by a CycleGAN-based model. The second type includes
Test-C60, Test-R300 and Test-S16 datasets. Specifically, Test-C60 contains 60
difficult images from UIEB and Test-S16 has 16 images from its original
dataset without references.

EUVP |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
UIEB |  |  |  |  |  |  |  |
LSUI |  |  |  |  |  |  |  |
UFO |  |  |  |  |  |  |  |
| FUnIEGAN | USUIR | Ucolor | PUIENet | Uranker-NU2Net | Semi-UIR | Ours | Reference
Figure 5. Visual comparison with different methods on synthesis dataset (EUVP) and real-world datasets (UIEB, LSUI and UFO). Comparing with FUnIEGAN, USUIR, Ucolor, PUIENet, URanker, Semi-UIR, our developed method (i.e., SIM-Net) achieves an natural color and preserves more details, performing the best visual quality.  Test-C60 |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
Test-R300 |  |  |  |  |  |  |  |
Test-S16 |  |  |  |  |  |  |  |
| Input | FUnIEGAN | USUIR | Ucolor | PUIENet | Uranker-NU2Net | Semi-UIR | Ours
Figure 6. Visual comparison on datasets with no references (Test-C60,
Test-R300, Test-S16) with Uranker metric value annotated. Comparing with
FUnIEGAN, USUIR, Ucolor, PUIENet, URanker, Semi-UIR, our developed method
performs the best visual quality.

Settings.  PyTorch is utilized to implement proposed network and we use NVIDIA
RTX 3090 cards to perform all the experiments. We use Adam as optimizer while
training all the models we compare. All the images are resized to $256\times
256$. During the training process, we set the total epoch to 200, the
batchsize to 8 and the initial learning rate of PMS, ADS and TDS are 1e-4,
1e-4 and 1e-6. Moreover, in Algorithm 1 the update step $t_{0}$ is 10 and
$t_{1}$ is 11. The coefficientes $\lambda_{1},\lambda_{2}$ of Eq. 5 are 0.04
and 0.02.

Evaluation Metrics. We adopt both reference-dependent and non-reference
evaluation measurements to comprehensively assess the performance of our
model. For reference-dependent metrics, we select Peak Signal to Noise Ratio
(PSNR), Structural Similarity (SSIM) and Mean Square Error (MSE). As for the
non-reference metrics, we employ Underwater Color Image Quality Evaluation
(UCIQE (Yang and Sowmya, 2015)), Underwater Image Quality Measure (UIQM
(Panetta et al., 2015)) and Perceptual Scores (PS) which reflect the visual
quality of the image from a human perspective. We also adopt Natural Image
Quality Evaluator (NIQE (Mittal et al., 2012)), which evaluates the
naturalness of generated images. Furthermore, we utilize URanker (Guo et al.,
2022), an UIQA method based on the Transformer. We apply these metrics on
real-world underwater challenge datasets.

###  4.2. Comparing with Other UIE Methods

In this section, we make comparisons with traditional methods (i.e., Fusion
(Ancuti et al., 2012) and UDCP (Drews et al., 2013) ), GAN-based methods
(i.e., UGAN (Fabbri et al., 2018), FUnIEGAN (Islam et al., 2020b)) and CNN-
based approaches (i.e, Water-Net (Li et al., 2019a), Ucolor (Li et al., 2021),
USUIR (Fu et al., 2022a), PUIE_net (Fu et al., 2022b), Uranker-NU2Net (Guo et
al., 2022) and Semi-UIR (Huang et al., 2023)). All of the competitive models
are retrained with their public codes on LSUI dataset for fair comparison.

Quantitative Comparisons. We first evaluate the capacity of our method
quantitatively and summarize the comparisons results in Tab. 1 and Tab. 2
where the best result is highlighted in black bold and the second one is
marked underline. From Tab. 1, we can observe that, our proposed method ranks
the first in Test-E515, Test-L504 and Test-U120 datasets in all three metrics.
We achieve the percentage gain of 6.5%, 2.8% and 5.4% regarding PSNR and
obtain leading 30.9%, 8.4% and 19.7% percentage improvement regarding MSE
metric in three datasets, respectively, indicating our results are more
consistent with ground-truth images and our superior performance compared to
other methods. Besides, we are only slightly behind Semi-UIR in Test-U90
dataset, with a minor gap. Moreover, Tab. 2 demonstrates that we can achieve
the best result in UIQM, NIQE and Uranker metrics, and gain the second
performance in all other evaluation cases. These indicate that our method is
equipped with superior performance when compared to other methods on challenge
dataset using unsupervised metrics.

Qualitative Comparisons.  We display the results on synthesized datasets in
Fig. 5 and the images from datasets with no references in Fig. 6 where we
manually pick one picture from each dataset. Besides, we annotate the
corresponding PSNR, MSE results of each image in the right bottom in Fig. 5
and the Uranker metric in Fig. 6. In general, our method provides images with
more natural color, abundant details, and less blurs. We have the closest
overall chroma to references and the best metrics as in all four rows in Fig.
5. In Fig. 6, FGAN, USUIR, Ucolor and PUIENet fail to predict the plausible
color, suffering from over or under enhancement results. Uranker-NU2Net and
Semi-UIR achieve good performance but obtain lower value of Uranker metric as
depicted in the right bottom. Our method attains the best results both
visually and metrically.

|  |  |  |  |  |
---|---|---|---|---|---|---
|  |  |  |  |  |
Input | w/o ADS | w/o TDS | w/ ADS, w/o $\mathcal{G}_{\mathbf{A}}$ | w/ TDS, w/o $\mathcal{G}_{\mathbf{T}}$ | Ours | Reference
Figure 7. Visual results of ablation on ADS, TDS, $\mathcal{G}_{\mathbf{A}}$ and $\mathcal{G}_{\mathbf{T}}$. The second row show the corresponding heat feature map.  |  |  |
---|---|---|---
|  |  |
Input | w/o 3H | w/o HMH | Ours
Figure 8. Ablation study on 3H and HMH module. We show the corresponding histogram distance between enhanced images and references images in the second row.  Table 3. Analyzing the effectiveness of ADS, TDS, $\mathcal{G}_{\mathbf{A}}$ and $\mathcal{G}_{\mathbf{T}}$. Baselines | PSNR$\uparrow$ | SSIM$\uparrow$ | MSE$\downarrow$
---|---|---|---
w/o ADS | 0.84 | 19.90 | 912
w/o TDS | 0.85 | 20.67 | 789
w/ ADS, w/o $\mathcal{G}_{\mathbf{A}}$ | 0.85 | 20.69 | 774
w/ TDS, w/o $\mathcal{G}_{\mathbf{T}}$ | 0.84 | 20.30 | 830
Ours | 0.86 | 21.53 | 721
|  |  |
Table 4. Analyzing the effectiveness of different Modules (including TGFE, 3H and HMH) on UIEB dataset. Modules | Ablation | Metrics
---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | MSE$\downarrow$
TGFE | w/o TGFE | 19.82 | 0.83 | 941
w/o TC | 21.01 | 0.85 | 745
w/o MFE | 20.86 | 0.85 | 780
w/o HRB | 20.92 | 0.85 | 727
3H | w/o 3H | 20.86 | 0.85 | 764
HMH | w/o HMH | 18.41 | 0.78 | 1143
Ours | - | 21.53 | 0.86 | 721
|  |  |  |
Table 5. Analyzing the effectiveness of $\mathbf{f}_{\text{ADS}}$ and $\mathbf{f}_{\text{TDS}}$. Modules | Baselines | Average Metrics
---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | MSE$\downarrow$
3H | w/o $\mathbf{f}_{\text{ADS}}$ | 0.85 | 23.87 | 404
HMH | w/o $\mathbf{f}_{\text{ADS}}$ | 0.85 | 23.85 | 411
TGFE | w/o $\mathbf{f}_{\text{TDS}}$ | 0.84 | 23.65 | 427
Full Model | w/o $\mathbf{f}_{\text{ADS}}$, w/o $\mathbf{f}_{\text{TDS}}$ | 0.84 | 22.99 | 491
w/ $\mathbf{f}_{\text{ADS}}$, w/ $\mathbf{f}_{\text{TDS}}$ | 0.86 | 24.17 | 383
|  |  |  |
Table 6. Analyzing the effectiveness of training schema. Baselines | PSNR$\uparrow$ | SSIM$\uparrow$ | MSE$\downarrow$
---|---|---|---
(a) end-to-end | 0.84 | 20.09 | 901
(b) ADS and TDS together | 0.85 | 20.95 | 726
(c) ADS, TDS and PMS together | 0.85 | 20.89 | 743
(d) ADS, TDS and PMS separate | 0.86 | 21.53 | 721
|  |  |

###  4.3. Ablation Study

Analysis the effectiveness of ADS and TDS.  We investigate the effectiveness
of physical knowledge-based dynamic structure (i.e., ADS and TDS). We show the
statistical results in Tab. 3 and visual images in Fig. 7. Without the whole
ADS branch, the model cannot capture the atmosphere information. Without TDS
branch providing ample transmission information, the performance of our model
also deteriorates, resulting in less vivid images. Besides, ablating
$\mathcal{G}_{\mathbf{A}}$ and $\mathcal{G}_{\mathbf{T}}$ that generates
various degraded priors deprives the model from learning diverse atmosphere
and transmission scenarios in ADS and TDS, respectively. Thus the produced
images lack detail as shown in the body of fish in the heat map in second row
of Fig. 7.

Analysis the components of PMS.  To verify how each module in our method plays
a irreplaceable role, we ablate: the Transmission-map Guided Feature
Extraction (TGFE), 3H and HMH. Furthermore, we exam the Transmission-based
Connection (TC) block, Multi-scale Feature extraction (MFE) module and Hyper
Residual Block (HRB) in TGFE to dig the effectiveness of each module. Note
that when ablating the 3H and HMH module, we change all four modules posed in
PMS. The corresponding results are listed in Tab. 4.

Without MFE module extracting multiple feature from distinct scales, the
performance of our model decreases which is reflected in reported scores. 3H
module, consisting of three hyper residual block, obtains weights from dynamic
prior hyper net and serves as a color optimizer. Thus removing them leads to
suboptimal scores. Furthermore, we display the visual results in Fig. 8 which
demonstrates that, without 3H and HMH integrating prior knowledge, the
generated images are more blur and dull, the histogram distance to references
is also worsen.

Influence of $\mathbf{f}_{\text{ADS}}$ and $\mathbf{f}_{\text{TDS}}$ to
generalization capacity.  We discuss the impact of $\mathbf{f}_{\text{ADS}}$
and $\mathbf{f}_{\text{TDS}}$ here. Note that removing them means that the
weights of res-blocks in each setting are directly learned form training
phrase, not obtained from hyper-nets. We perform experiments on res-blocks in
3H, HMH and TGFE. All ablations are conducted on LSUI dataset for training and
evaluated on four datasets: Test-E515, Test-U90, Test-L504 and Test-U120. We
compute the average scores regarding the PSNR, SSIM and MSE metrics on four
datasets and report them in Tab. 5.

We observe that, the full model ranks the first in all three measurements,
indicating that we achieve the best generalization ability among datasets.
Without $\mathbf{f}_{\text{ADS}}$ or $\mathbf{f}_{\text{TDS}}$ in any
position, the performance deteriorates. Therefore, we keep
$\mathbf{f}_{\text{ADS}}$ and $\mathbf{f}_{\text{TDS}}$ in those modules to
ensure a better generalization capability.

Analysis on training strategy.  We list different training methods and their
results in Tab. 6. Specifically, (a) means that we train two hypernets and PMS
in an end-to-end manner. (b) means that we optimize ADS and TDS together, and
then optimize PMS alternately. (c) means that we optimize three networks (ADS,
TDS and PMS) all together. (d) is our strategy which separately optimizes
three networks. It can be seen that the our training strategy is of great
important. We argue that this can be attributed to the inconsistent
convergence speed of the hyper-nets and the main net. Thus, we train and
optimize three nets in an alternate and separate way.

##  5\. Conclusion

In this work we developed a novel approach named GUPDM for underwater image
enhancement. We tailored ADS and TDS as hyper-nets to generate parameters for
PMS via varied global atmosphere and transmission to cover complex underwater
scenes. Experimental results illustrated our superior performance in terms of
quantitative scores, visual results and generalization ability.

###### Acknowledgements.

This work is supported by Natural Science Foundation of China (Grant No.
62202429, U20A20196) and Zhejiang Provincial Natural Science Foundation of
China under Grant No. LY23F020024, LR21F020002, LY23F020023.

## References

  * (1)
  * Akkaynak and Treibitz (2018) Derya Akkaynak and Tali Treibitz. 2018.  A revised underwater image formation model. In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_. 6723–6732.
  * Akkaynak and Treibitz (2019a) Derya Akkaynak and Tali Treibitz. 2019a.  Sea-thru: A method for removing water from underwater images. In _Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition_. 1682–1691.
  * Akkaynak and Treibitz (2019b) Derya Akkaynak and Tali Treibitz. 2019b.  Sea-thru: A method for removing water from underwater images. In _Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition_. 1682–1691.
  * Ancuti et al. (2012) Cosmin Ancuti, Codruta Orniana Ancuti, Tom Haber, and Philippe Bekaert. 2012.  Enhancing underwater images and videos by fusion. In _2012 IEEE conference on Computer Vision and Pattern Recognition_. IEEE, 81–88.
  * Ancuti et al. (2019) Codruta O Ancuti, Cosmin Ancuti, Christophe De Vleeschouwer, and Mateu Sbert. 2019.  Color channel compensation (3C): A fundamental pre-processing step for image enhancement.  _IEEE Transactions on Image Processing_ 29 (2019), 2653–2665.
  * Berman et al. (2020) Dana Berman, Deborah Levy, Shai Avidan, and Tali Treibitz. 2020\.  Underwater single image color restoration using haze-lines and a new quantitative dataset.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 43, 8 (2020), 2822–2837.
  * Buchsbaum (1980) Gershon Buchsbaum. 1980\.  A spatial processor model for object colour perception.  _Journal of the Franklin institute_ 310, 1 (1980), 1–26.
  * Chen et al. (2020) Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. 2020\.  Dynamic convolution: Attention over convolution kernels. In _Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition_. 11030–11039.
  * Chiang and Chen (2011) John Y Chiang and Ying-Ching Chen. 2011.  Underwater image enhancement by wavelength compensation and dehazing.  , 1756–1769 pages.
  * Drews et al. (2013) Paul Drews, Erickson Nascimento, Filipe Moraes, Silvia Botelho, and Mario Campos. 2013\.  Transmission estimation in underwater single images. In _Proceedings of the IEEE international conference on computer vision workshops_. 825–830.
  * Fabbri et al. (2018) Cameron Fabbri, Md Jahidul Islam, and Junaed Sattar. 2018\.  Enhancing Underwater Imagery Using Generative Adversarial Networks. In _2018 IEEE International Conference on Robotics and Automation (ICRA)_. 7159–7165.  https://doi.org/10.1109/ICRA.2018.8460552
  * Fu et al. (2022a) Zhenqi Fu, Huangxing Lin, Yan Yang, Shu Chai, Liyan Sun, Yue Huang, and Xinghao Ding. 2022a.  Unsupervised Underwater Image Restoration: From a Homology Perspective. In _Proceedings of the AAAI Conference on Artificial Intelligence_. 643–651.
  * Fu et al. (2022b) Zhenqi Fu, Wu Wang, Yue Huang, Xinghao Ding, and Kai-Kuang Ma. 2022b.  Uncertainty Inspired Underwater Image Enhancement. In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVIII_. Springer, 465–482.
  * Galdran et al. (2015) Adrian Galdran, David Pardo, Artzai Picón, and Aitor Alvarez-Gila. 2015.  Automatic red-channel underwater image restoration.  _Journal of Visual Communication and Image Representation_ 26 (2015), 132–145.
  * Ghani and Isa (2015) Ahmad Shahrizan Abdul Ghani and Nor Ashidi Mat Isa. 2015.  Enhancement of low quality underwater image through integrated global and local contrast correction.  _Applied Soft Computing_ 37 (2015), 332–344.
  * Guo et al. (2022) Chunle Guo, Ruiqi Wu, Xin Jin, Linghao Han, Zhi Chai, Weidong Zhang, and Chongyi Li. 2022.  Underwater ranker: learn which is better and how to be better.  _arXiv preprint arXiv:2208.06857_ (2022).
  * Guo et al. (2019) Yecai Guo, Hanyu Li, and Peixian Zhuang. 2019.  Underwater image enhancement using a multiscale dense generative adversarial network.  _IEEE Journal of Oceanic Engineering_ 45, 3 (2019), 862–870.
  * Ha et al. (2016) David Ha, Andrew Dai, and Quoc V Le. 2016.  Hypernetworks.  _arXiv preprint arXiv:1609.09106_ (2016).
  * Huang et al. (2023) Shirui Huang, Keyan Wang, Huan Liu, Jun Chen, and Yunsong Li. 2023.  Contrastive Semi-supervised Learning for Underwater Image Restoration via Reliable Bank.  _arXiv preprint arXiv:2303.09101_ (2023).
  * Iqbal et al. (2010) Kashif Iqbal, Michael Odetayo, Anne James, Rosalina Abdul Salam, and Abdullah Zawawi Hj Talib. 2010\.  Enhancing the low quality images using unsupervised colour correction method. In _2010 IEEE International Conference on Systems, Man and Cybernetics_. IEEE, 1703–1709.
  * Islam et al. (2020a) Md Jahidul Islam, Peigen Luo, and Junaed Sattar. 2020a.  Simultaneous enhancement and super-resolution of underwater imagery for improved visual perception.  _arXiv preprint arXiv:2002.01155_ (2020).
  * Islam et al. (2020b) Md Jahidul Islam, Youya Xia, and Junaed Sattar. 2020b.  Fast Underwater Image Enhancement for Improved Visual Perception.  _IEEE Robotics and Automation Letters_ 5, 2 (2020), 3227–3234.
  * Islam et al. (2020c) Md Jahidul Islam, Youya Xia, and Junaed Sattar. 2020c.  Fast underwater image enhancement for improved visual perception.  _IEEE Robotics and Automation Letters_ 5, 2 (2020), 3227–3234.
  * Jiang et al. (2022a) Zhiying Jiang, Zhuoxiao Li, Shuzhou Yang, Xin Fan, and Risheng Liu. 2022a.  Target Oriented Perceptual Adversarial Fusion Network for Underwater Image Enhancement.  _IEEE Transactions on Circuits and Systems for Video Technology_ (2022).
  * Jiang et al. (2022b) Zhiying Jiang, Zengxi Zhang, Yiyao Yu, and Risheng Liu. 2022b.  Bilevel modeling investigated generative adversarial framework for image restoration.  _The Visual Computer_ (2022), 1–13.
  * Land (1977) Edwin H Land. 1977\.  The retinex theory of color vision.  _Scientific american_ 237, 6 (1977), 108–129.
  * Li et al. (2021) Chongyi Li, Saeed Anwar, Junhui Hou, Runmin Cong, Chunle Guo, and Wenqi Ren. 2021\.  Underwater image enhancement via medium transmission-guided multi-color space embedding.  _IEEE Transactions on Image Processing_ 30 (2021), 4985–5000.
  * Li et al. (2020) Chongyi Li, Saeed Anwar, and Fatih Porikli. 2020.  Underwater scene prior inspired deep underwater image and video enhancement.  _Pattern Recognition_ 98 (2020), 107038.
  * Li et al. (2019a) Chongyi Li, Chunle Guo, Wenqi Ren, Runmin Cong, Junhui Hou, Sam Kwong, and Dacheng Tao. 2019a.  An underwater image enhancement benchmark dataset and beyond.  _IEEE Transactions on Image Processing_ 29 (2019), 4376–4389.
  * Li et al. (2016) Chong-Yi Li, Ji-Chang Guo, Run-Min Cong, Yan-Wei Pang, and Bo Wang. 2016.  Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior.  _IEEE Transactions on Image Processing_ 25, 12 (2016), 5664–5677.
  * Li et al. (2019b) Hanyu Li, Jingjing Li, and Wei Wang. 2019b.  A fusion adversarial underwater image enhancement network with a public test dataset.  _arXiv preprint arXiv:1906.06819_ (2019).
  * Lin et al. (2021) Runjia Lin, Jinyuan Liu, Risheng Liu, and Xin Fan. 2021\.  Global structure-guided learning framework for underwater image enhancement.  _The Visual Computer_ (2021), 1–16.
  * Lin et al. (2020) Xi Lin, Zhiyuan Yang, Qingfu Zhang, and Sam Kwong. 2020\.  Controllable pareto multi-task learning.  _arXiv preprint arXiv:2010.06313_ (2020).
  * Liu et al. (2018) Risheng Liu, Xin Fan, Minjun Hou, Zhiying Jiang, Zhongxuan Luo, and Lei Zhang. 2018\.  Learning aggregated transmission propagation networks for haze removal and beyond.  _IEEE transactions on neural networks and learning systems_ 30, 10 (2018), 2973–2986.
  * Liu et al. (2020a) Risheng Liu, Xin Fan, Ming Zhu, Minjun Hou, and Zhongxuan Luo. 2020a.  Real-world underwater enhancement: Challenges, benchmarks, and solutions under natural light.  _IEEE Transactions on Circuits and Systems for Video Technology_ 30, 12 (2020), 4861–4875.
  * Liu et al. (2019) Risheng Liu, Minjun Hou, Jinyuan Liu, Xin Fan, and Zhongxuan Luo. 2019.  Compounded layer-prior unrolling: A unified transmission-based image enhancement framework. In _2019 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 538–543.
  * Liu et al. (2022a) Risheng Liu, Zhiying Jiang, Shuzhou Yang, and Xin Fan. 2022a.  Twin adversarial contrastive learning for underwater image enhancement and beyond.  _IEEE Transactions on Image Processing_ 31 (2022), 4922–4936.
  * Liu et al. (2020b) Risheng Liu, Pan Mu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. 2020b.  A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton. In _International Conference on Machine Learning_. PMLR, 6305–6315.
  * Liu et al. (2022b) Risheng Liu, Pan Mu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. 2022b.  A general descent aggregation framework for gradient-based bi-level optimization.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 45, 1 (2022), 38–57.
  * Liu et al. (1995) Yung-Cheng Liu, Wen-Hsin Chan, and Ye-Quang Chen. 1995\.  Automatic white balance for digital still camera.  _IEEE Transactions on Consumer Electronics_ 41, 3 (1995), 460–466.
  * Mittal et al. (2012) Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. 2012\.  Making a “completely blind” image quality analyzer.  _IEEE Signal processing letters_ 20, 3 (2012), 209–212.
  * Mu et al. (2022) Pan Mu, Haotian Qian, and Cong Bai. 2022.  Structure-Inferred Bi-level Model for Underwater Image Enhancement.  _Proceedings of the 30th ACM International Conference on Multimedia (MM)_ (2022).
  * Panetta et al. (2015) Karen Panetta, Chen Gao, and Sos Agaian. 2015.  Human-visual-system-inspired underwater image quality measures.  _IEEE Journal of Oceanic Engineering_ 41, 3 (2015), 541–551.
  * Peng et al. (2023) Lintao Peng, Chunli Zhu, and Liheng Bian. 2023.  U-shape transformer for underwater image enhancement. In _Computer Vision–ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part II_. Springer, 290–307.
  * Peng and Cosman (2017) Yan-Tsung Peng and Pamela C Cosman. 2017.  Underwater image restoration based on image blurriness and light absorption.  _IEEE Transactions on Image Processing_ 26, 4 (2017), 1579–1594.
  * Pizer (1990) Stephen M Pizer. 1990\.  Contrast-limited adaptive histogram equalization: Speed and effectiveness stephen m. pizer, r. eugene johnston, james p. ericksen, bonnie c. yankaskas, keith e. muller medical image display research group. In _Proceedings of the first conference on visualization in biomedical computing, Atlanta, Georgia_ , Vol. 337. 1.
  * Qian et al. (2022) Haotian Qian, Wentao Tong, Pan Mu, Zheyuan Liu, and Hanning Xu. 2022.  Real-world Underwater Image Enhancement via Degradation-aware Dynamic Network. In _PRICAI 2022: Trends in Artificial Intelligence: 19th Pacific Rim International Conference on Artificial Intelligence, PRICAI 2022, Shanghai, China, November 10–13, 2022, Proceedings, Part III_. Springer, 530–541.
  * Rahman et al. (1996) Zia-ur Rahman, Daniel J Jobson, and Glenn A Woodell. 1996\.  Multi-scale retinex for color image enhancement. In _Proceedings of 3rd IEEE International Conference on Image Processing_ , Vol. 3. IEEE, 1003–1006.
  * Wang et al. (2017) Yi Wang, Hui Liu, and Lap-Pui Chau. 2017.  Single underwater image restoration using adaptive attenuation-curve prior.  _IEEE Transactions on Circuits and Systems I: Regular Papers_ 65, 3 (2017), 992–1002.
  * Yang and Sowmya (2015) Miao Yang and Arcot Sowmya. 2015.  An underwater color image quality evaluation metric.  _IEEE Transactions on Image Processing_ 24, 12 (2015), 6062–6071.
  * Yin et al. (2022) Guanghao Yin, Wei Wang, Zehuan Yuan, Wei Ji, Dongdong Yu, Shouqian Sun, Tat-Seng Chua, and Changhu Wang. 2022\.  Conditional hyper-network for blind super-resolution with multiple degradations.  _IEEE Transactions on Image Processing_ 31 (2022), 3949–3960.
  * Zhang et al. (2017) Shu Zhang, Ting Wang, Junyu Dong, and Hui Yu. 2017\.  Underwater image enhancement via extended multi-scale Retinex.  _Neurocomputing_ 245 (2017), 1–9.


# Evaluating Quality of Answers for Retrieval-Augmented Generation:
A Strong LLM Is All You Need

Yang Wang
Visa Data & AI Platform
Foster City, California
<EMAIL_ADDRESS>&Alberto Garcia Hernandez
Visa Consulting & Analytics
London, United Kingdom
<EMAIL_ADDRESS>Roman Kyslyi
Visa Consulting & Analytics
Kyiv, Ukraine
<EMAIL_ADDRESS>&Nicholas Kersting
Visa Data & AI Platform
Austin, Texas
<EMAIL_ADDRESS>
###### Abstract

We present a comprehensive evaluation of answer quality in Retrieval-Augmented
Generation (RAG) applications using vRAG-Eval, a novel grading system that is
designed to assess correctness, completeness, and honesty. We further map the
grading of quality aspects aforementioned into a binary score, indicating an
accept or reject decision, mirroring the intuitive "thumbs-up" or "thumbs-
down" gesture commonly used in chat applications. This approach suits factual
business settings where a clear decision opinion is essential. Our assessment
applies vRAG-Eval to two Large Language Models (LLMs), evaluating the quality
of answers generated by a vanilla RAG application. We compare these
evaluations with human expert judgments and find a substantial alignment
between GPT-4’s assessments and those of human experts, reaching 83% agreement
on accept or reject decisions. This study highlights the potential of LLMs as
reliable evaluators in closed-domain, closed-ended settings, particularly when
human evaluations require significant resources.

Evaluating Quality of Answers for Retrieval-Augmented Generation:
A Strong LLM Is All You Need

Yang Wang Visa Data & AI Platform Foster City, California<EMAIL_ADDRESS>Alberto Garcia Hernandez Visa Consulting & Analytics London, United Kingdom
<EMAIL_ADDRESS>

Roman Kyslyi Visa Consulting & Analytics Kyiv, Ukraine<EMAIL_ADDRESS>Nicholas Kersting Visa Data & AI Platform Austin, Texas<EMAIL_ADDRESS>

##  1 Introduction

Since the launch of ChatGPT in November 2022, Large Language Models (LLMs)
have become increasingly popular integrated components for organizations
seeking to enhance productivity and enrich their product portfolio offerings.
However, it is well known that GPT-4 Turbo’s training corpora cut-off date is
April 2023, rendering the model lacking in current events knowledge.
Furthermore, as LLMs are pre-trained on public domain text and do not possess
proprietary information, their capabilities are limited when it comes to our
company’s knowledge-intensive applications.

Fine-tuning (Radford et al., 2018; Dodge et al., 2020) is a technique that can
be used to inject new knowledge into pre-trained LLMs by adjusting their
gradient parameters through further, specialized training. However, OpenAI’s
fine-tuning API is currently only available through an experimental access
program, and GPT-4 fine-tuning requires more effort to achieve significant
improvements, as noted by OpenAI (2023).

Retrieval-augmented generation (RAG) was proposed initially by Lewis et al.
(2021). This method involves storing extra knowledge in a non-parametric dense
vector index and using a pre-trained neural retriever to search relevant
context, followed by generating content with a pre-trained sequence-to-
sequence (seq2seq) model. Ovadia et al. (2024) argue that RAG consistently
outperforms unsupervised fine-tuning on a wide range of knowledge-intensive
tasks.

A RAG application leveraging LLMs enhanced with the company’s proprietary
knowledge has become one of the pivotal factors advancing the adoption of
GPT-4-based applications at our enterprise, a global payment technology
company. This phenomenon highlights the need to apply viable approaches to
evaluate RAG applications, as lacking performance metrics poses a risk to the
enterprise’ business and may have negative consequences.

One approach to evaluate the quality of RAGs focuses on their unique
characteristic: that they consist of a Retriever model and a Generator model.
Therefore, studies have used metrics such as context relevance, and answer
relevance to evaluate the two components separately then assess the answer
faithfulness between them (Saad-Falcon et al., 2024; Es et al., 2023).

The other paradigm for evaluating RAG applications treats them as traditional
question answering systems. Researchers have proposed metrics such as SAS, a
semantic answer similarity estimation (Risch et al., 2021), and F1/EM scores
(Wang et al., 2024) to evaluate these applications on public question
answering (QA) datasets. Zheng et al. (2023) propose using strong LLMs such as
GPT-4 to evaluate other LLMs. They argue that traditional benchmarks cannot
effectively align quality measurement with human preferences in open-ended
tasks.

In some companies, RAG applications are mostly developed for a closed-domain,
closed-ended setting, where employees seek informative answers based on the
organizations’ proprietary knowledge base. Additionally, external-facing RAG
applications allow clients to search for information that is not necessarily
available on the Internet. Therefore, it is crucial to research and propose
quality evaluation approaches and metrics that are suitable for the business
and align with the companies’ values.

To study this, we introduce a benchmark dataset called VND-Bench, comprising
155 high-quality questions across 14 subject areas in a proprietary payment
network data domain. We also collect their corresponding answers from a
communication channel on an internal employee collaboration platform to serve
as ground truth labels. We build a vanilla RAG application, tRAG, as a test
bed and ask it to answer the 155 questions. We then request five human
experts, GPT-4, and Llama 2 to assess tRAG’s answer quality based on the
labels and vRAG-Eval, a set of grading instructions and a rubric. Our results
show that GPT-4’s evaluation agrees with human evaluators’ scores at a rate of
83% based on a binary accept/reject category.

We summarize our contributions as follows:

  * •

We design a RAG evaluation mechanism, vRAG-Eval, which includes a set of
grading instructions and a rubric that measures answer quality in three
aspects: correctness, completeness, and honesty, resulting in one single
score. It can be readily converted to a binary accept/reject decision that
suits business settings.

  * •

We build a high-quality benchmark dataset, VND-Bench, with 155
questions/answers pairs that cover 14 subject areas in data domain.

  * •

We conduct experiments using LLMs as a RAG’s quality evaluators in a closed-
domain closed-ended single-turn QA setting and find that GPT-4’s grading
agrees with human experts’ opinions at a rate of 83% in terms of answer
accepted or rejected decisions.

The rest of the paper is organized as follows: in section 2, we introduce
related work. And in section 3, we explain the motivation, problem setting,
and our method of experimentation. In section 4, we provide our experimental
results. We make concluding remarks in sections 5&6.

##  2 Related Work

Embedding and semantic similarity Zhang et al. (2020) propose the BERTScore
metric as an automatic evaluation method for text generation tasks. Unlike
traditional lexical approaches that rely on word matching, BERTScore sums the
cosine similarities between the token embeddings of both two sentences: the
reference answer and the QA system-generated answer. Risch et al. (2021)
introduce SAS, a metric designed to estimate the semantic similarity between
ground-truth annotations and answers generated by a QA model. It is found that
semantic similarity metrics, especially those based on contemporary
transformer models, align more accurately with human assessments compared to
the conventional lexical similarity metrics.

Reference-free evaluation In scenarios where human annotations are not readily
available, Es et al. (2023) present RAGAS, a framework for automating the
assessment of RAG systems. This framework considers RAG’s 2-module composition
and proposes three key metrics: Context Relevance to evaluate the Retriever,
Answer Relevance to assess the Generator, and Faithfulness to measure the
coherence between the two modules. Similarly, ARES ((Saad-Falcon et al.,
2024)) evaluates along the same 3 dimensions to assess the quality of RAG
components. While not entirely reference-free, ARES leverages a few hundred
human annotations during evaluation and employs a lightweight fine-tuned LLM
as a judge, rather than relying on a frozen LLM.

LLM as a judge Two projects closely related to our research are LLM-as-a-Judge
(Zheng et al., 2023) and A Case Study on the Databricks Documentation Bot
(Leng et al., 2023). To evaluate LLM-based chat assistants, Zheng et al.
(2023) examine the usage and limitations of LLMs as judges for open-ended
questions. This setting may not be suitable for most RAG applications in
enterprises, where users typically seek definitive answers. Leng et al. (2023)
employ LLMs to generate grades as a weighted composite score of Correctness,
Comprehensiveness, and Readability. While they report an alignment rate of 80%
with human scores on individual factors, their grading system lacks a crucial
element: one single metric that provides a decision-making-ready quality
score. In business-centric settings, this limitation become particularly
significant.

##  3 Motivation and Problem Setting

We investigate the challenges of automatically evaluating the qualities of
RAG-generated answers with respect to correctness, completeness, and honesty,
in aligning with human preferences as reflected in our experiment’s human
grading. The objective is to determine whether LLMs can reliablly assess the
quality of answers generated by RAG systems or not.

###  3.1 Preparation

We first develop a test bed RAG application, tRAG. Its knowledge base consists
of a corpus of 18 PDF documents, totaling 15M tokens, which includes payment
processing specifications, API reference guides, data standard manuals, and
others.

During the preprocessing stage, for each document $d$ in the knowledge corpus,
we split its content into chunks such that $d=\\{c_{i}\\}_{i=1}^{n}$, and
vectorize each text chunk $c_{i}$ using an embedding model $M:v_{i}=M(c_{i})$.
Then we store the key value pair $<v_{i},c_{i}>$ into a local vector store
instance. Appendix A provides configurations and technical details about the
in-memory vector store. Appendix B illustrates a high-level workflow of this
preprocessing stage.

Next, we curate a benchmark dataset, VND-Bench, by collecting question-answer
pairs directly from an internal employee communication channel. This channel
is frequently used by a diverse range of data users across the organization to
seek help in understanding specific topics related to a proprietary Payment
Network’s operations and transactional data semantics. We collect a total of
155 closed-ended questions along with their corresponding answers, which we
consider to be the ground truth for our experiment. When multiple people
respond to a question on the channel, we aggregate their answers into one
single response. The set of questions are categorized into subject areas
specified in Table 1.

1\. Acceptance | 8\. Issuing
---|---
2\. Authentication | 9\. Master Data
3\. Authorization | 10\. OCT
4\. Clearing and Settlement | 11\. Other
5\. Commercial | 12\. Processing
6\. Dispute | 13\. Product
7\. Fraud | 14\. Token
Table 1: Subject areas of the questions considered in the experiment.

###  3.2 Retrieval, Augmentation, and Generation

For each question $Q$ in the VND-Bench dataset, tRAG embeds the text into a
fixed-length dense vector using model $M:q=M(Q)$, then conducts distance
search $\Delta$ within the knowledge database, and returns the top $K$ most
relevant document chunks matching the query.

$K=\underset{i}{\mathrm{argmin}}\,\Delta(v_{i},q)$

During the Augmentation stage tRAG concatenates those top $K$ chunks as the
context $C$.

$C=\\{c_{j}|j\in K\\}$

And combines with the question to form a prompt: $Q\oplus C$ . This prompt is
then sent to the answer generator.

We utilize GPT-4 model as the tRAG’s answer generator model $G$. The resulting
response, which constitutes the model’s inference output, is evaluated in our
experiment. Appendix C illustrates tRAG’s single-turn question answering
workflow.

###  3.3 Grading Method

We compile a fielded dataset containing 155 entries with the following
columns: Subject Area, Question, Label, tRAG’s Answer. Additionally, we design
a grading rubric as illustrated in Table 2 to evaluate tRAG’s Answer.

1: |  The response is not aligned with the Label or is off-topic; includes hallucination.
---|---
2: |  The response admits it cannot provide an answer or lacks context; honest.
3: |  The response is relevant but contains notable discrepancies or inaccuracies.
4: |  The response is acceptable, sufficient but not exhaustive.
5: |  The response is fully accurate and comprehensive, based on the Label.
Table 2: Grading rubric

Each score assesses a distinct aspect of answer quality. Specifically,
although both incorrect answers and responses of unknown lack value, we argue
that hallucination may impose significant harm to mission-critical businesses.
Consequently, we penalize hallucination with the lowest score of 1, while
honestly acknowledging an unknown response earns a score of 2.

A correct answer warrants a score of 4, while a RAG application’s extra effort
to provide supplementary details is commended with a score of 5, reflecting
the value of answer completeness. When a score of 3 is assigned, it indicates
that the answer quality is borderline, potentially containing false
information.

LLMs grade answer quality through zero-shot learning, wherein a constant
template as part of the vRAG-Eval, is designed to populate prompts for each
question. Table 3 illustrates the grading template.

You are an AI assistant. In the following task, you are given a Question,
---
a RAG application’s response, and a Ground-truth Answer referred to as
’Label’.
Assess how well the RAG application’s response aligns with the Label,
using the grading rubric below:
`[Start of Grading Rubric]`
`{rubric}`
`[End of Grading Rubric]`
Treat the Label as the definitive answer. Present your final score in the
format: "[[score]]",
followed by your justification. Example:
Score: [[3]], Reason: [[The RAG’s response partially aligns with the Label
but with some discrepancies]].
`[Start of User Question]`
`{question}`
`[End of User Question]`
`[Start of Label]`
`{label}`
`[End of Label]`
`[Start of RAG’s Application Response]`
`{tRAG answer}`
`[End of RAG’s Application Response]`
Table 3:  Prompt template used for LLM grading.

In our experiment, inputs to the evaluators are a sequence of tRAG’s answers
$\hat{y}_{1},\hat{y}_{2},\cdots,\hat{y}_{k}$ together with a sequence of
labels $y_{1},y_{2},\cdots,y_{k}$. The quality scores given by evaluators can
be generally denoted as $S_{E}:=P(\hat{Y}=Y)$, where
$E\in\\{GPT-4,Llama2,Human\\}$. Appendix D depicts the workflow of the grading
experiments.

To ensure reproducibility and prevent potential grading hallucinations, we fix
the temperature parameter for each LLM call at $T=0.0$.

##  4 Experiments

We ask GPT-4, Llama 2 7B, Llama 2 13B, and human experts at our company to
evaluate the quality of the aforementioned tRAG’s answers respectively and
independently. Their assessment is guided by the vRAG-Eval instructions and
grading rubric.

###  4.1 GPT-4

vRAG-Eval instructs that grading responses should be in the format of “Score:
[[score]], Reason: [[explanation]]”. GPT-4 illustrates the capability to
strictly adhere to the instructions. Table 4 shows an example question and
it’s ground truth answer (“Label”) in the VND-Bench dataset. And Table 5
illustrates GPT-4’s grading response.

Question |  Label
---|---
I have following queries related to data in [anomaly_tranx_table_name].

There are around [number_of_records] records in this table where
[purchase_date_field_name] is greater than [post_date_field_name]. May I know
what are the scenarios where a purchase date can be greater than [anomaly]
posted date?

There are records where same [tranx_id_field_name] have more than one record. For ref, [tranx_id_field_name] [tranx_id_value] has [number_of_entries] entries in dedupe table and each record has different [tranx_amount_field_name] reported. May I know why there can be more than 1 entry for the same [tranx_id_field_name] |  [tranx_id_field_name] may be used to tie multiple clearing records to one authorisation, as is the case with split shipments (one order placed online will have original authorisation, followed by multiple shipments, each one cleared separately)
Table 4: An example question and its Label. Note that we retain spelling errors in the original label text. Exact table names, column names, and values are masked in square brackets. tRAG’s Answer |  Assessment
---|---
Based on the context provided, I can provide some general insights into your
queries. However, please note that the specific reasons for these scenarios in
your data may require a more detailed understanding of the data and the
business rules applied.

1\. Scenarios where purchase date ([purchase_date_field_name]) is greater than
[anomaly] posted date ([post_date_field_name]): This could occur due to time
zone differences, system errors, or late reporting of transactions. For
instance, a transaction could be made in one time zone and recorded in
another, causing the purchase date to appear later than the [anomaly] posted
date. It could also be due to delays in transaction posting or reporting.

2\. Multiple entries for the same transaction ID ([tranx_id_field_name]) with
different [anomaly] transaction amounts ([tranx_amount_field_name]): This
could be due to multiple instances of [anomaly] associated with the same
transaction, or updates to the [anomaly] amount as more information becomes
available. It could also be a result of data errors or duplications.

Without more specific information about the data and the business rules applied, it’s difficult to provide a more precise explanation. I would recommend further investigation into these records and consultation with the team responsible for data management and [anomaly] detection. |  Score: [[2]], Reason: [[The RAG’s response provides general insights into the user’s queries but does not align with the specific answer provided in the Label. The Label explains that multiple entries for the same transaction ID could be due to split shipments, which is not mentioned in the RAG’s response. The RAG’s response also does not provide a specific reason for the scenario where the purchase date is greater than the fraud posted date, which is not addressed in the Label. Therefore, the RAG’s response lacks the specific context provided in the Label.]]
Table 5: GPT-4 Evaluator strictly follows the grading format.

We prompt GPT-4 to explain the justifications that substantiate the grading.
It allows us to understand LLM’s thinking process. In our earlier design of
vRAG-Eval, we explicitly emphasized Correctness, Completeness, and Honesty. It
became evident that the GPT-4 evaluator partially deviated from the Label then
graded answers based on its own understanding of those quality metrics. For
instance, as illustrated in Table 6, tRAG’s answer for the same question was
graded 4 instead of 2. This example underscores the importance of developing
clear and specific grading standards, contrasting with vague guidelines that
can be applied universally in public settings but require adaptation for
company-specific purposes. Appendix F shows the initial design of grading
instruction and rubric.

Initial Assessment
---
Rating: [[4]], Reason: [[The RAG application’s answer is acceptable and
provides a general understanding of the possible reasons for the user’s
queries. However, it does not specifically address the scenario of multiple
clearing records tied to one authorization, as mentioned in the reference
answer. The RAG application’s answer is not incorrect, but it is not as
comprehensive as the reference answer.]]
Table 6: GPT-4 hallucinates the grading due to implicit quality definition
shifts.

###  4.2 Human

We collect human grading that score tRAG’s answer quality. These experts are
experienced data and machine learning practitioners.

The 155 Q&A pairs in VND-Bench are collected in two phases. The answers to the
first 52 questions provided by tRAG are graded by three human experts, while
the next set of 103 questions are graded by two others.

To obtain a robust and representative view of humans’ opinions, we choose the
median function to sample the first 52 grading scores from three human
experts, and then randomly select the remaining scores from the other two
experts.

The score distributions are illustrated in Figure 1.

Figure 1: Left: Distribution of human grading scores for tRAG’s answers to the
first 52 questions. Right: Distribution of human grading scores for tRAG’s
answers to the next 103 questions.

###  4.3 Compare GPT-4 with Human

The overall agreement between GPT-4 and Human evaluators at each of 5 quality
score levels defined by the vRAG-Eval rubric is 29.7%. Figure 2 shows
comparison at each level.

Figure 2: Score distribution comparison at each level.

A closer analysis of grading samples reveals that the two evaluators may have
subtle opinion differences at the exact level of the score assigned. Table 7
provides an example question and label pair, then tRAG’s answer and GPT-4’s
grading are shown in Table 8. Human scores tRAG’s answer at quality level 2,
with no explanations provided (we did not request any). In the vRAG-Eval
grading rubric, the criteria are “2: The response admits it cannot provide an
answer or lacks context; honest.”. Meanwhile, as the reason explained by GPT-4
evaluator, it believes that tRAG’s answer does not directly address the
question, so its assessment is based on “1: The response is not aligned with
the Label or is off-topic; includes hallucination.” We also believe that GPT-4
explanation justifies the score 1. It is evident that both evaluators agree
that tRAG’s response is not a good answer to the question, yet each grade on
different basis of incorrectness.

Question |  Label
---|---
I would like to browse transactions - invoice from [tranx_code_value]. Could you please advice if the data is availabe somewhere. |  [tranx_code_value]s are carried in [tranx_detail_table_name] [tranx_code_field_name] = [tranx_code_value]. [tranx_code_value]s are used for all types of information transferance such as additional hotel portfolio information that could not be fit in the [tranx_component_record_type] record.
Table 7: A sample question and the answer label. tRAG’s Answer |  GPT-4’s Grading
---|---
Based on the context provided, it appears that the data for [tranx_code_value] - [tranx_component_record_type] - Commercial Card - Invoice (Line Detail) is contained within specific tables. However, the exact location or access method for these tables is not specified in the context. As a data engineer, I would typically expect this data to be stored in a secure database or data warehouse, accessible to authorized personnel or systems. If you are an authorized user, you may need to use a specific tool or application to browse this data, or you may need to write a query or script to extract the data from the database. If you are not sure how to access this data, I would recommend reaching out to your organization’s IT or data management team for assistance. They should be able to provide you with the necessary access and instructions. Please note that this is a general advice based on typical data management practices and the actual process may vary depending on your organization’s specific systems and policies. |  Score: [[1]], Reason: [[The RAG’s response does not align with the Label. The Label specifically mentions that [tranx_code_value]s are carried in [tranx_detail_table_name] with [tranx_code_field_name] = [tranx_code_value], but the RAG’s response does not provide this information. Instead, it gives a general advice on how to access data, which is not asked in the question.]]
Table 8: tRAG’s answer and GPT-4’s grading justification.

Leng et al. (2023) confirms that both humans and LLMs struggle to hold the
same standard for the same score when grading on high precision. However, the
definitions of “high precision” and “low precision” are subjective and can
vary based on individual perspectives. To address this issue, we propose
aligning with the typical thumbs-up and thumbs-down gestures used in chat
applications. It is also appropriate in business settings, where a binary
quality scale effectively predicts whether an answer can be accepted or
rejected by clients.

Therefore, we convert the 5-grading scale to binary level: 1&2&3 (answer
rejected), 4&5 (answer accepted). We observe an impressive 82.6% agreement
rate between GPT-4 and human evaluators. Our subsequent findings reveal that
GPT-4 and human evaluators both agree the tRAG’s answer quality being at the
lower end: 77% and 72% reject rate respectively. Note that tRAG is a hastily
developed RAG application only to create a test bed. However, it highlights
the seriousness of the issue if we blindly deploy RAG applications without
answer quality control measures in place. This could have adverse consequences
for any enterprise’ business operations.

###  4.4 Llama 2

Llama 2 is an open-source language model (LLM) that is available in three
versions, each with a different parameter size: 7B, 13B, and 70B. Due to our
infrastructure constraints, we are only able to experiment with two of the
smaller versions, 7B and 13B.

Similar to the GPT-4 experiment, we use the VND-Bench dataset, which includes
questions, labels, and tRAG’s responses. We then ask 7B and 13B to evaluate
tRAG’s quality based on vRAG-Eval’s grading instructions and rubric.

7B gives a uniform score of 4 to all grading requests, leading us to speculate
that at this not-so-large parameter size, Llama 2 may not be intelligent
enough to discern the quality of answers. In contrast, 13B shows slightly more
variation in scores, with 127 answers receiving a quality grade of 4 and the
remaining 28 answers getting a grade of 3. We are left wondering why Llama 2
favors certain answers over others, and how confident it is in its grading. To
address this, we add an additional instruction text, as shown in Table 9, to
elicit Llama 2’s confidence level. This approach, coined as “verbalized
probability” by (Lin et al., 2022), allows the LLM to express its confidence
level in natural language without use of model logits.

Present your final score in the format: "[[score]]",
followed by your confidence level of the grading in the range of 0 to 100,
with 100 being very confident and 0 being not sure about your grading at all.
At the end, disclose your justification. Example:
Score: [[3]], Confidence: [[50]], Reason: [[The RAG’s response partially
aligns with the Label but with some discrepancies]].
---
Table 9: Addition instruction to request Llama 2 disclosing its internal
confidence level.

Figure 3 shows the correlation between the 13B evaluator’s confidence and
answer quality scores. An approximate diagonal pattern indicates that it tends
to assign a grade of 4 when highly confident (80%) and a grade of 3 when less
confident (50%).

Figure 3: The heatmap reveals a positive correlation between answer quality
scores and grading confidence levels that 13B perceives.

##  5 Discussion

While LLMs continue to grow in size and intelligence, utilizing powerful LLMs
as an automated alternative to human expert graders is a promising and
prominent area of research. This approach complements traditional lexical and
semantic measurements and demonstrates exceptional explainability and human
preference alignment capabilities. We identify several areas for potential
future research and practice direction:

Enterprise’s Values In designing the vRAG-Eval grading system, we incorporate
honesty into the grading rubric to reflect that human evaluators prefer
answers that admit uncertainty over hallucination. We also conduct preliminary
experiments on the subject of "verbalized probability" to explore the
potential of using self-expressed confidence levels as a factor in measuring a
LLM evaluator’s own honesty. Additionally, we consider integrating more
ethical aspects, such as inclusiveness and fairness, into the grading metrics
to ensure that the system is both effective and responsible.

More Language Support The VND-Bench dataset currently only includes Q&A pairs
in English, yet our enterprise is a global company that supports businesses
worldwide. We consider extending the dataset to include other languages,
reflecting the company’s value of inclusiveness and diversity.

Domain Adaptation We curate the VND-Bench dataset, focusing on transactional
data domain knowledge. We consider building benchmark datasets in other areas
and study vRAG-Eval’s applicability to explore its potential in those business
priorities.

Few-shot Evaluation. Our experiments demonstrate that the GPT-4 evaluator
exhibits high agreement rates in accept/reject decisions. We consider
analyzing the score discrepancy at each of the five grading levels to better
understand the relationship between human and LLM evaluators’ thinking
process. By leveraging "few-shot learning", we hypothesize that we can improve
the alignment of LLM evaluators with human preferences at a greater refined
granularity.

More Question Categories. The VND-Bench dataset currently focuses on factual
knowledge questions in the transactional data domain. We consider expanding to
include math and logical reasoning questions, which are essential for
evaluating business-centric RAG applications. By developing high-order grading
capabilities, we can better support the company’s growth and innovation.

Llama 2 (70B). We consider quantizing the 70B model to 4-bit integers and
evaluating its grading agreement with human experts and GPT-4. This experiment
could enable Llama 2 to become a more economical and less risky alternative to
GPT-4, suitable for deployment on commodity hardware.

##  6 Conclusion

In this paper, we thoroughly study evaluating Retrieval-Augmented Generation
(RAG) applications in a high-stakes business environment where answer quality
is paramount.

Our study uniquely address this by casting a 5-grade evaluation scale into a
binary evaluation system, categorizing responses as either accepted or
rejected. This mapping lead to a clearer, more direct comparison between human
evaluator and AI models, particularly GPT-4.

We observe a remarkably high agreement rate of 82.6% between GPT-4 and human
evaluators, which underscores the potential of advanced AI models in
understanding and aligning with human judgement criteria through exceptional
explainability.

Furthermore, our comparison between GPT-4 and Llama 2 models reveals a
significant disparity in performance, with GPT-4 demonstrating superior
results, highlighting the importance of selecting a strong LLM.

## Acknowledgments

We thank Ranjan Dutta, Toni Wang, and Ajit Patil for their thoughtful
discussions and assistance with data annotation. We acknowledge Ping Zhu, Chi
Wo Chung, Jiayin Zheng, Mohammad Rahman and their team providing robust OpenAI
API access; Gurpreet Bhasin and a team of AI volunteers for them making Llama
2 available. We thank our managers Salila Khilani, Dmitrii Krainov, and Yu Gu
for their support.

## References

  * Dodge et al. (2020) Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  _arXiv:2002.06305_.
  * Es et al. (2023) Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023.  Ragas: Automated evaluation of retrieval augmented generation.  arXiv:2309.15217.
  * Leng et al. (2023) Quinn Leng, Kasey Uhlenhuth, and Alkis Polyzotis. 2023.  Best practices for llm evaluation of rag applications.  _Databrick Engineering Blog_.
  * Lewis et al. (2021) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  arXiv:2005.11401.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  Teaching models to express their uncertainty in words.  _arXiv:2205.14334_.
  * OpenAI (2023) OpenAI. 2023.  New models and developer products announced at devday.
  * Ovadia et al. (2024) Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. 2024.  Fine-tuning or retrieval? comparing knowledge injection in llms.  arXiv:2312.05934.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  _OpenAI_.
  * Risch et al. (2021) Julian Risch, Timo Möller, Julian Gutsch, and Malte Pietsch. 2021.  Semantic answer similarity for evaluating question answering models.  arXiv:2108.06130.
  * Saad-Falcon et al. (2024) Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024.  Ares: An automated evaluation framework for retrieval-augmented generation systems.  arXiv:2311.09476.
  * Wang et al. (2024) Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, and Bryan Catanzaro. 2024.  Instructretro: Instruction tuning post retrieval-augmented pretraining.  arXiv:2310.07713.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.  In _ICLR 2020_.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.  Judging llm-as-a-judge with mt-bench and chatbot arena.  In _37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks_.

##  Appendix A Document Vector Store

Chunk size | 1000
---|---
Chunk overlap | 25%
Embedding model | text-embedding-ada-002
Indexing | Hierarchical Navigable Small World (HNSW)
Distance | Squared L2 Norm
Top-K | 3
Embeddings | 13644

##  Appendix B Preprocessing Workflow

##  Appendix C Document Retrieval and Answer Generation Workflow

##  Appendix D Answer Generation and Quality Evaluation

##  Appendix E GPT-4 Chat Configuration

Model | GPT-4
---|---
Endpoint | https://[domain]/genai-api/v1/queries/chat
Temperature | 0.0

##  Appendix F Early Design of Grading Instructions and Rubric

Please act as an impartial judge and evaluate the quality of the answer
provided by a RAG application.
---
Your evaluation should consider correctness, completeness, and honesty.
You will be given a reference answer, and the RAG application’s answer,
then grade the quality of the RAG application’s answer based on rubric below:
`[The Start of Grading Rubric]`
`{rubric}`
`[The End of Grading Rubric]`
Be as objective as possible. Output your final verdict by strictly following
this format: "[[rating]]",
then providing your explanation.
For example: "Rating: [[5]], Reason: [[The answer is correct and
comprehensive]].
1: completely incorrect, hallucination
2: admits it cannot answer or lack of context, honest
3: pertinent but contains noticeable errors or inaccuracies
4: acceptable answer, adequate but not comprehensive
5: fully accurate and exhaustive