#  ParaShoot: A Hebrew Question Answering Dataset

Omri Keren Omer Levy
The Blavatnik School of Computer Science
Tel Aviv University
<EMAIL_ADDRESS>
###### Abstract

NLP research in Hebrew has largely focused on morphology and syntax, where
rich annotated datasets in the spirit of Universal Dependencies are available.
Semantic datasets, however, are in short supply, hindering crucial advances in
the development of NLP technology in Hebrew. In this work, we present
ParaShoot, the first question answering dataset in modern Hebrew. The dataset
follows the format and crowdsourcing methodology of SQuAD, and contains
approximately 3000 annotated examples, similar to other question-answering
datasets in low-resource languages. We provide the first baseline results
using recently-released BERT-style models for Hebrew, showing that there is
significant room for improvement on this task.

##  1 Introduction

Natural language processing has seen a surge in the pretraining paradigm in
recent years with the appearance of pretrained models in a plethora of
languages, including Hebrew (Chriqui and Yahav, 2021; Seker et al., 2021).
While such models have shown to perform remarkably well on a variety of tasks,
most of the evaluation of the Hebrew models, however, has been focused on
morphology and syntax tasks in the spirit of universal dependencies (Nivre et
al., 2017), while end-user-focused evaluation has been limited to sentiment
analysis (Chriqui and Yahav, 2021) and named entity recognition (Bareket and
Tsarfaty, 2020).

In this paper, we try to remedy the scarcity of semantic datasets by
presenting ParaShoot,111A portmanteau of paragraph and t”/sw (shoot), the
Hebrew abbreviation of Q&A. the first question answering dataset in Hebrew, in
the style of SQuAD (Rajpurkar et al., 2016). We follow similar work in
constructing non-English question answering datasets (d’Hoffschmidt et al.,
2020; Mozannar et al., 2019; Lim et al., 2019, inter alia), and turn to
Hebrew-speaking crowdsource workers, asking them to write questions given
paragraphs sampled at random from Hebrew Wikipedia. Through this process, we
collect approximately 3000 annotated (paragraph, question, answer) triplets,
in a setting that may be suitable for few-shot learning, simulating the amount
of data a startup or academic group can quickly collect with a limited
annotation budget or a short deadline.

Statistical analysis of ParaShoot shows that the dataset is diverse in
question types and complexity, and that the annotations are of decent quality.
We provide baseline results based on two recently-released BERT-style models
in Hebrew, showing that there is much potential in devising better pretraining
and fine-tuning schemes to improve the performance of Hebrew language models
on this dataset. We hope that this new dataset will pave the way for
practitioners and researchers to advance natural language understanding in
Hebrew.222The dataset is publicly available at
https://github.com/omrikeren/ParaShoot

##  2 Dataset

We present ParaShoot, a question answering dataset in Hebrew, in a format that
closely follows that of SQuAD (Rajpurkar et al., 2016). Each example in the
dataset is a triplet consisting of a paragraph, a question, and a span from
the paragraph text constituting the answer to the question. We scrape
paragraphs from random Hebrew Wikipedia articles, and crowdsource questions
and answers for each one, resulting in 3038 annotated examples. While larger
datasets may facilitate better-performing models, recent work has advocated
for research on smaller labeled datasets Ram et al. (2021), which more
accurately reflect the amount of data a startup or academic lab can collect in
a short amount of time and resources.

###  2.1 Corpus

We collect random articles from Hebrew Wikipedia, covering a wide range of
domains and topics. We only sample articles containing at least two paragraphs
and 500 characters.333We filter out images, tables, etc. Finally, for each
such article, two candidate paragraphs are randomly sampled and added to the
annotation corpus. These paragraphs will eventually become the passages in the
question answering dataset.

Figure 1: The annotation user interface, containing the article’s title, the
paragraph, a slot for entering a question, and an additional slot for entering
the answer. Dragging the mouse over a span in the paragraph automatically
fills the question slot, allowing for quick and accurate annotation of answer
spans.

###  2.2 Annotation

We recruit annotators by using the Prolific crowdsourcing
platform.444www.prolific.co Being a native Hebrew speaker is the only required
qualification, allowing the participation of a few dozen annotators in the
campaign. Annotators are presented with random paragraphs from the annotation
set, and tasked to write 3-5 questions that are explicitly answered by the
given text, for each paragraph. As in the original SQuAD annotation campaign,
annotators are instructed to phrase the questions in their own words, and
highlight the minimal span of characters from the paragraph that contains the
answer to each question. Our implementation also provides automatic data
validation heuristics that alert the annotators if, for instance, the answer
span is too long or not a substring of the paragraph. Figure 1 shows a
screenshot from the annotation web page.555The platform’s code is based upon
https://github.com/cdqa-suite/cdQA-annotator.

We acknowledge the fact that this data collection technique is known to
encourage annotation artifacts Gururangan et al. (2018); Kaushik and Lipton
(2018), and several newer annotation methods, such as TyDi QA Clark et al.
(2020), have been introduced to alleviate them. Nevertheless, we follow
SQuAD’s annotation methodology, as it necessitates considerably fewer
resources. Maintaining an hourly wage of over $10,6667.50 GBP $\approx$ 10.50
USD, at the time of writing. we were able to collect our entire dataset,
including discarded data from development runs, for under $800.

###  2.3 Post-Processing

In total, we amass 3106 question-answer examples. Of those, we discard 68
examples (2.2%) that contained yes/no questions or extremely short/long
answers. The resulting dataset contains 3038 examples, which we divide to
training, validation, and test by article, preventing content overlap. Table 1
details the amount of unique articles, paragraphs, and questions of each
split.

| #Articles | #Paragraphs | #Questions
---|---|---|---
Train | 295 | 565 | 1792
Validation | 33 | 63 | 221
Test | 165 | 319 | 1025
Total | 493 | 947 | 3038
Table 1: The number of unique articles, paragraphs, and questions in each
split of ParaShoot. The dataset is partitioned by articles.

##  3 Analysis

We analyze the dataset in various ways to assess its quality and limitations
as a benchmark.

Figure 2: The length distribution of questions (left) and answers (right) in
the entire dataset.

###  3.1 Annotation Quality

To measure the quality of the annotated data, we randomly select 50 examples
from the validation set, and manually analyze them ourselves.777The authors
are native speakers of modern Hebrew. Specifically, we check whether the
annotated answer span is correct (answers the question) and minimal (contains
only the answer). Table 2 shows that the majority of the annotations are
indeed valid, answering the questions with a minimal span. Yet, a significant
minority contains additional supporting information, which makes the answer
span longer than the desired minimal span by 2.5 times on average. We can thus
expect an upper bound of 57% token F1 on those examples, setting the
performance ceiling at around 84% F1 for the entire dataset. Finally, we
present examples from the validation set that illustrate the annotation
quality (Figure 3).

Answer Span | Frequency
---|---
Minimal | 70%
Too Long | 28%
Too Short | 2%
Table 2: Distribution of annotated answer span quality, based on manual analysis of 50 examples from the validation set. Question Word | Frequency
---|---
What | …/mhw/mh | 16.29%
Which | …/’yzw/’yzh | 15.84%
Who | …/myhw/my | 14.03%
When |  mmty/mty | 13.57%
Where | …/hykn/’yph | 10.86%
How |  ky.sd/’yk | 6.79%
How much/many | …/bkmh/kmh | 5.43%
Why |  mdw‘/lmh | 4.52%
Table 3: Question word distribution, according to the first word of each
question in the validation set. Inflected words and synonyms are clustered
together to better align with English question types.

###  3.2 Question Diversity

To measure the dataset’s diversity, we cluster questions by their question
word (typically the first word in the question). Table 3 shows that what (mh)
and which (’yzh) questions account for a third of the sample, with other
answer types being distributed in a rather balanced distribution. We also
observe that about 11% of the data contains how (’yk) and why (lmh) questions,
which may reflect more complex instances.

###  3.3 Sequence Length

We measure the length in words (using whitespace tokenization) of each
question and each answer. Figure 2 shows the distributions of annotated
questions and answers. We observe that most questions use between 4-7 words,
which is typical of simple questions in Hebrew. More complicated questions
constitute 27.6% of the data, for example: ?’yk nqr’t h’wprh h’.hrwnh /sktbw
gylbr.t ws’lybn y.hdyw (What is the last opera written jointly by Gilbert and
Sullivan called?) There are even questions with only 2 words; due to Hebrew’s
rich morphology, these questions are usually translated to 3-4 words in
English, e.g. ?mhw hmnyk’yzm (What is Manichaeism?) Answer lengths, however,
can vary greatly, depending on whether the annotators wrote minimal spans
(typically 1-4 words) or included supporting information in the answer spans
(see Section 3.1).

###  3.4 Linguistic Phenomena

As a morphologically-rich language Tsarfaty et al. (2010); Seddah et al.
(2013), modern Hebrew exhibits a variety of non-trivial phenomena that are
uncommon in English and could be challenging for NLP models Tsarfaty et al.
(2020). We can identify some of these phenomena in our dataset. Consider for
example the following question-answer pair from the validation set:
[numoffset=-2em, interpartskip=3ex, samplelabel=(a),labelformat=A:,
avoidnumlabelclash] [label=Q] [everygla=, belowglpreambleskip=0pt,
aboveglftskip=0pt] ?mh hyh /s.t.hw /sl kpr /smryhw k/shwqm // ma haya shitkho
shel kfar shmaryahu kshe-hukam // what was area-of-it of Kfar Shmaryahu when-
was.established // ‘What was Kfar Shmaryahu’s area when it was established?’
//

[label=A] [everygla=, belowglpreambleskip=0pt, aboveglftskip=0pt] … hy/swb
hwqm ‘l /s.t.h /sl // ha-yeshuv hukam al shetakh shel … // the-village
was.established on area of … // ‘The village was established on an area of …’
// This example illustrates a morphological variation between the question and
the answer: the same entity appears as a morpheme in a compound word in the
question’s text: /s.t.hw (its area), k/shwqm (when it was established), but as
a standalone word (i.e. without inflection) in the answer: /s.t.h (area), hwqm
(was established). These phenomena make exact match-optimized predictions more
difficult for models aimed to solve this task.

Figure 3: Examples from the validation set. The text in bold shows crowd-
annotated answers. The underlined text represents the (expert-annotated)
minimal answer span. The first example demonstrates a non-minimal span that
has some overlap with the question’s text. The second example demonstrates a
valid minimal span selection.

##  4 Baselines

We establish baseline results for ParaShoot using BERT-style models. Results
indicate the task is challenging, leaving much room for future work in Hebrew
NLP to advance the state of the art.

###  4.1 Experiment Setup

We fine-tune three adaptations of BERT Devlin et al. (2019): mBERT, trained by
the original authors on a corpus consisting of the entire Wikipedia dumps of
100 languages; HeBERT Chriqui and Yahav (2021), trained on the OSCAR corpus
Ortiz Suárez et al. (2020) and Hebrew Wikipedia; AlephBERT Seker et al.
(2021), also trained on the OSCAR corpus, with an additional 71.5 million
tweets in Hebrew. All models are equivalent in size to BERT-base, i.e. 12
layers, 768 model dimensions, and 110M parameters in total.

We fine-tune the models using the default implementation of HuggingFace
Transformers Wolf et al. (2020). We select the best model by validation set
performance over the following hyperparameter grid: learning rate
$\in\\{3\mathrm{e}{-5},5\mathrm{e}{-5},1\mathrm{e}{-4}\\}$, batch size
$\in\\{16,32,64\\}$, and update steps $\in\\{512,800,1024\\}$. We compare the
models’ predictions to the annotated answer using token-wise F1 score and
exact match (EM), as defined by Rajpurkar et al. (2016).

###  4.2 Results

Table 4 shows the performance of each model on ParaShoot, with mBERT achieving
the highest performance (56.1 F1). We also observe significant variance across
the models, with mBERT and AlephBERT performing significantly better than
HeBERT. It is not immediately clear where this discrepancy stems from; one
possibility is that the introduction of noisy data via multilinguality (mBERT)
or tweets (AlephBERT) makes that model more robust to potential noise in the
annotated questions (e.g. typos). Comparing these results to the estimated
ceiling performance of 84 F1 (see Section 3.1), we can infer that ParaShoot
poses a genuine challenge to future Hebrew models and encourages further
analysis of the semantic capabilities of the current models.

Model | F1 | EM
---|---|---
HeBERT | 36.7 | 18.2
AlephBERT | 49.6 | 26.0
mBERT | 56.1 | 32.0
Table 4: Baseline performance on the test set.

###  4.3 Error Analysis

We analyze the error distribution by sampling 50 examples from the validation
set and comparing AlephBERT’s predictions to the annotated answers. Table 5
shows how the examples are distributed into five categories, accounting for
every type of overlap between the model’s prediction and the annotated answer.
Putting aside exact matches (which account for about a quarter of examples),
nearly half of the errors stem from zero overlap between the annotated answer
and the model’s prediction. We observe that a significant part of the sample
(22%) contains cases where the annotated answer is a substring of the model’s
prediction, which might be, to a large extent, an artifact of the long answer
annotations we observe in Section 3.1. For examples of erroneous predictions
see Appendix A.

Overlap Type | Sample | Error
---|---|---
Frequency | Frequency
Model $=$ Annotation | 26% | –
Model $\subset$ Annotation | 14% | 19%
Model $\supset$ Annotation | 22% | 30%
Model $\cap$ Annotation $\neq\emptyset$ | 4% | 5%
Model $\cap$ Annotation $=\emptyset$ | 34% | 46%
Table 5: An error analysis of 50 random examples from the validation set,
based on AlephBERT’s predictions. The first reflects exact matches, and the
last case accounts for zero overlap between model prediction and annotated
answer. The three categories in the middle refer to partially correct answers,
where the model’s prediction has some overlap with the annotated answer.

##  5 Conclusion

In this paper, we present ParaShoot, the first question answering dataset in
modern Hebrew, in a style and data collection methodology similar to that of
SQuAD. Baseline results demonstrate the potential of this dataset for
researchers and practitioners alike to develop better models and datasets for
natural language understanding in Hebrew.

## Acknowledgements

This work was supported by the Tel Aviv University Data Science Center, Len
Blavatnik and the Blavatnik Family foundation, the Alon Scholarship, Intel
Corporation, and the Yandex Initiative for Machine Learning. We thank Reut
Tsarfaty for her valuable feedback.

## References

  * Bareket and Tsarfaty (2020) Dan Bareket and Reut Tsarfaty. 2020.  Neural modeling for named entities and morphology (nemo^ 2).  _arXiv preprint arXiv:2007.15620_.
  * Chriqui and Yahav (2021) Avihay Chriqui and Inbal Yahav. 2021.  Hebert & hebemo: a hebrew bert model and a tool for polarity analysis and emotion recognition.  _arXiv preprint arXiv:2102.01909_.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.  _Transactions of the Association for Computational Linguistics_ , 8:454–470.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * d’Hoffschmidt et al. (2020) Martin d’Hoffschmidt, Wacim Belblidia, Quentin Heinrich, Tom Brendlé, and Maxime Vidal. 2020.  FQuAD: French question answering dataset.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1193–1208, Online. Association for Computational Linguistics.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Kaushik and Lipton (2018) Divyansh Kaushik and Zachary C. Lipton. 2018.  How much reading does reading comprehension require? a critical investigation of popular benchmarks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5010–5015, Brussels, Belgium. Association for Computational Linguistics.
  * Lim et al. (2019) Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019.  Korquad1. 0: Korean qa dataset for machine reading comprehension.  _arXiv preprint arXiv:1909.07005_.
  * Mozannar et al. (2019) Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019.  Neural Arabic question answering.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_ , pages 108–118, Florence, Italy. Association for Computational Linguistics.
  * Nivre et al. (2017) Joakim Nivre, Daniel Zeman, Filip Ginter, and Francis Tyers. 2017.  Universal Dependencies.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts_ , Valencia, Spain. Association for Computational Linguistics.
  * Ortiz Suárez et al. (2020) Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2020.  A monolingual approach to contextualized word embeddings for mid-resource languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1703–1714, Online. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Ram et al. (2021) Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. 2021.  Few-shot question answering by pretraining span selection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 3066–3079. Association for Computational Linguistics.
  * Seddah et al. (2013) Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Galletebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yannick Versley, Veronika Vincze, Marcin Woliński, Alina Wróblewska, and Eric Villemonte de la Clergerie. 2013.  Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages.  In _Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages_ , pages 146–182, Seattle, Washington, USA. Association for Computational Linguistics.
  * Seker et al. (2021) Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021.  Alephbert: A hebrew large pre-trained language model to start-off your hebrew nlp application with.  _arXiv preprint arXiv:2104.04052_.
  * Tsarfaty et al. (2020) Reut Tsarfaty, Dan Bareket, Stav Klein, and Amit Seker. 2020.  From SPMRL to NMRL: What did we learn (and unlearn) in a decade of parsing morphologically-rich languages (MRLs)? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7396–7408, Online. Association for Computational Linguistics.
  * Tsarfaty et al. (2010) Reut Tsarfaty, Djamé Seddah, Yoav Goldberg, Sandra Kuebler, Yannick Versley, Marie Candito, Jennifer Foster, Ines Rehbein, and Lamia Tounsi. 2010\.  Statistical parsing of morphologically rich languages (SPMRL) what, how and whither.  In _Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages_ , pages 1–12, Los Angeles, CA, USA. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.

##  Appendix A Error Examples

Figure A.1: Predictions made by fine-tuned AlephBERT vs. annotated answers. In
the first example, the prediction produced by the model is clearly an error.
In the second example, the annotated answer span is excessively long, and the
model predicts a more accurate substring of this span. In the third example,
the model predicts a full sentence, while the annotated answer span is
shorter.


# Evaluating BBRv2 on the Dropbox Edge Network

Alexey Ivanov
Dropbox, Inc
Mountain View, CA
<EMAIL_ADDRESS>

###### Abstract

Nowadays, loss-based TCP congestion controls in general and CUBIC specifically
became the de facto standard for the Internet. BBR congestion control
challenges the loss-based approach by modeling the network based on estimated
bandwidth and round-trip time. At Dropbox, we’ve been using BBRv1 since 2017
and are accustomed to its pros and cons. BBRv2 introduces a set of
improvements to network modeling (explicit loss targets and inflight limits)
and fairness (differential probing and headroom for new flows.) In this paper,
we go over experimental data gathered on the Dropbox Edge Network. We compare
BBRv2 to BBRv1 and CUBIC showing that BBRv2 is a definite improvement over
both of them. We also show that BBRv2 experimental results match its
theoretical design principles.

## Keywords

BBR, BBRv2, CUBIC, TCP, Congestion Control, AQM.

## Introduction

Since the “Bottleneck Bandwidth and Round-trip propagation time” (BBR)
congestion control paper[5] was released it became production-ready and was
added to Linux, FreeBSD, and Chrome (as a part of QUIC.) Back then, Dropbox
evaluated[15] BBRv1 congestion control on our edge network and it showed
promising results compared to CUBIC:

Figure 1: Dropbox desktop clients’ download goodput.

After BBRv1 has been fully deployed to Dropbox Edge Network we started to
identify its corner cases and shortcomings. Some of them were eventually
fixed, for example, BBRv1 being measurably slower for Wi-Fi users. Other
trade-offs were quite conceptual, for example BBRv1’s unfairness towards loss-
based congestion controls[21], RTT-unfairness between BBRv1 flows[9], and its
disregard for packet loss. Dropbox Edge Network observed packet loss rates up
to 6% on hosts using BBRv1, compared to around 0.5% on hosts using CUBIC with
pacing.

## BBRv1 shortcomings

BBR developers were aware of these problems and actively worked on solutions.
Following issues were identified[12]:

  * •

Low throughput for Reno/CUBIC flows sharing a bottleneck with bulk BBR flows.

  * •

Loss-agnostic; high loss if $bottleneck<1.5*BDP$.

  * •

ECN-agnostic.

  * •

Low throughput for paths with high degrees of aggregation (e.g. Wi-Fi.)

  * •

Throughput variation due to low cwnd in PROBE_RTT.

## BBR version 2

BBRv2 aims to solve some of the major drawbacks of the first version. Here is
the list of BBR design principles (bold means it’s new in BBRv2[13]. See Table
1.):

  * •

Leave headroom: leave space for entering flows to grab.

  * •

React quickly: using loss/ECN, adapt to delivery process now to maintain flow
balance.

  * •

Don’t overreact: don’t do a multiplicative decrease on every round trip with
loss/ECN.

  * •

Probe deferentially: probe on a time scale to allow coexistence with
Reno/CUBIC.

  * •

Probe robustly: try to probe beyond estimated max bw, max volume before we cut
estimation.

  * •

Avoid overshooting: start probing at an inflight measured to be tolerable.

  * •

Grow scalably: start probing at 1 extra packet; grow exponentially to use free
capacity.

|  Cubic |  BBRv1 |  BBRv2
---|---|---|---
Model parameters for the state machine |  N/A |  Throughput, RTT |  Throughput, RTT, max aggregation, max inflight
Loss |  Reduce cwnd by $30\%$ on window by any loss |  N/A |  Explicit loss rate target
ECN |  RFC3168 (Classic ECN) |  N/A |  DCTCP-inspired ECN (See “Explicit Congestion Notification” section.)
Startup |  Slow-start until RTT rises (Hystart) or any loss |  Slow-start until throughput plateaus |  Slow-start until throughput plateaus or ECN or Loss rate ¿ target
Table 1: What’s new in BBRv2: a summary

## Test constraints

### Non-mobile ISPs

The test was focusing on the Dropbox Desktop Client traffic and therefore
mostly excludes mobile ISPs.

### Bulk traffic focus

This experiment was aimed at high-throughput workloads. All TCP connections
and nginx logs mentioned in the paper were filtered by having at least 1Mb of
data transferred.

### Localized test

This experiment was performed in a single point of presence (PoP) in Tokyo,
Japan. Therefore, it carries over some of the biases common to traffic from
that geographic area, including internet speeds, operating systems, client
devices, etc.

### Real user traffic

This is a real-world experiment with all of its pros and cons, including ISPs
with heavy over-subscription, broken embedded TCP/IP implementations,
etc111For testing congestion controls in a lab environment
github.com/google/transperf can be used. It allows testing TCP performance
over a variety of emulated network scenarios, including RTT, bottleneck
bandwidth, and policed rate that can change over time..

## Test setup

Following combinations of kernels and congestion control algorithms were
tested:

  * •

5.3 kernel, cubic

  * •

5.3 kernel, bbr

  * •

5.3 kernel, bbr2

All of the servers are using a combination of mq and sch_fq qdiscs with
default settings. Kernel is based on Ubuntu-hwe-edge-5.3.0-18.19_18.04.2 with
all patches from the v2alpha-2019-11-17 tag.

Graphs were generated from either connection-level information from ss -neit
sampling, machine-level stats from /proc, or server-side application-level
metrics from web-server logs.

## Caveats

### Keeping the kernel up-to-date

Newer kernels usually bring quite a bit of improvement222Recent Linux kernels
also include mitigations for the newly discovered CPU vulnerabilities. We
highly discourage disabling them (especially on the edge!) so be prepared to
also take a CPU usage hit. to all subsystems including the TCP/IP stack. For
example, if we compare 4.15 performance to 5.3, we can see the latter having
around $15\%$ higher goodput:

Figure 2: 4.15 vs 5.3 kernels server-side file upload goodput.

Most likely candidates for this improvement are:

  * •

“tcp_bbr: adapt cwnd based on ack aggregation estimation,” fixing Wi-Fi
performance.

  * •

“tcp: switch to Early Departure Time model,” fixing the RTT jitter observed by
TCP when used with pacing (See “Beyond As Fast As Possible” section.)

### Keeping userspace up-to-date

Having recent versions of userspace is quite important if you are using
kernels that are newer compared to what your OS was bundled with. This is
especially true for packages like ethtool and iproute2.

In our configuration we’ve used Ubuntu 16.04 Xenial with a fairly recent 5.3.0
kernel along with iproute2-5.4.0. Here is an example of using the new version
of ss (with the new output fields in bold):

Listing 1: iproute2 5.4.0

⬇

$ ss -tie

ts sack bbr rto:220 rtt:16.139/10.041 ato:40 mss:1448 pmtu:1500 rcvmss:1269
advmss:1428 cwnd:106 ssthresh:52 bytes_sent:9070462 bytes_retrans:3375
bytes_acked:9067087 bytes_received:5775 segs_out:6327 segs_in:551
data_segs_out:6315 data_segs_in:12
bbr:(bw:99.5Mbps,mrtt:1.912,pacing_gain:1,cwnd_gain:2) send 76.1Mbps
lastsnd:9896 lastrcv:10944 lastack:9864 pacing_rate 98.5Mbps delivery_rate
27.9Mbps delivered:6316 busy:3020ms rwnd_limited:2072ms(68.6%) retrans:0/5
dsack_dups:5 rcv_rtt:16.125 rcv_space:14400 rcv_ssthresh:65535 minrtt:1.907

As you can see, the new ss version has all the new goodies from the kernel’s
struct tcp_info, plus the internal BBR state from the struct tcp_bbr_info.
This adds lots of metrics that can be used even in day-to-day TCP performance
troubleshooting. Including very useful insufficient sender buffer and
insufficient receiver window/buffer stats from the “tcp: sender chronographs
instrumentation” patchset.

## Experimental results

### Packet loss

Most notably, switching from BBRv1 to BBRv2 results in an enormous drop in
retransmits:

Figure 3: Retransmits, %. CUBIC, BBRv1, and BBRv2.

RetransSegs on these boxes is still higher than on ones using CUBIC but given
that BBR was designed to ignore non-congestion induced packet loss, we would
assume that things work as intended.

Looking deeper at the ss stats we can confirm this: BBRv2 packet loss is
multiple times lower than BBRv1 (Figure 4.) though still higher than CUBIC
(Figure 5.).

Figure 4: Retransmits, %, PDF. BBRv1 vs BBRv2. Figure 5: Retransmits, %. PDF.
CUBIC vs BBRv2.

The deeper inspection also shows that BBRv2 has connections with $>60\%$
packet loss. These types of outliers are present neither on BBRv1 nor CUBIC
machines. Looking at some of these connections closer does not reveal any
obvious patterns: connections with absurdly large packet loss come from
different OS’es (based on Timestamp/ECN support,) connections types (based on
MSS,) and locations (based on RTT.)

Aside from these outliers, BBRv2 retransmissions are lower across all RTTs,
when compared to BBRv1.

Figure 6: Round Trip Time. BBRv1 vs BBRv2.

### Throughput

On the throughput side, we looked at the nginx file upload goodput
metric333From the Traffic Team’s perspective, one of our SLIs for edge
performance is end-to-end client-reported download speed. For this test, we’ve
been using server-side file upload speed as the closest proxy for it. (Figure
7.). For lower percentiles of connection speeds, BBRv2 performance is closer
to CUBIC. For higher ones, it starts getting closer to BBRv1. This is likely
due to BBRv2 being fairer to other TCP connections on the bottleneck since the
slower a connection is the more likely it is being congested (instead of being
app limited.)

Figure 7: File upload goodput from nginx point of view.

Connection-level stats confirm that BBRv2 has lower bandwidth than BBRv1
(Figure 8.) but still higher than CUBIC (Figure 9.).

Figure 8: Bandwidth distribution. BBRv1 vs BBRv2. Figure 9: Bandwidth
distribution. CUBIC vs BBRv2

So, does that mean that BBRv2 is slower? Yes, it does, at least to some
extent. So, what are we getting in return? Based on connection stats, quite a
lot. We’ve already mentioned lower packet loss (and therefore higher
“goodput”,) but there is more.

### Packets in-flight

We’ve observed way fewer “unacked” packets which is a good proxy for bytes in-
flight. BBRv2 looks way better than in BBRv1 (Figure 10.) and even slightly
better than CUBIC (Figure 11.).

Figure 10: Unacked packets distribution. BBRv1 vs BBRv2. Figure 11: Unacked
packets distribution. CUBIC vs BBRv2.

Plotting RTT vs in-flight heatmap shows that in the BBRv1 case amount of data
in-flight tends to be dependent on the RTT. This is fixed in BBRv2 (Figure
15.).

Figure 12: Receive Window Limited. BBRv1 vs BBRv2.

As BBR co-author, Neal Cardwell, explains it:

> “In all of the cases I’ve seen with unfairness due to differing $min\\_rtt$
> values, the dominant factor is simply that with BBRv1 each flow has a cwnd
> that is basically $2*bw*min\\_rtt$, which tends to try to maintain
> $1*bw*min\\_rtt$ in the bottleneck queue, which quite directly means that
> flows with higher $min\\_rtt$ values maintain more packets in the bottleneck
> queue and therefore get a higher fraction of the bottleneck bandwidth. The
> most direct way I’m aware of to improve RTT fairness in the BBR framework is
> to get rid of that excess queue, or ensure that the amount of queue is
> independent of a flow’s $min\\_rtt$ estimate.”

### Receive Window Limited connections

We similarly observed that BBRv2 connections spend way less time being receive
window limited than both BBRv1 (Figure 12.) and CUBIC444CUBIC was using sch_fq
and pacing in this test too.(Figure 13.).

Figure 13: Receive Window Limited. CUBIC vs BBRv2. Figure 14: Retransmits vs
RTT heatmap. BBRv1 vs BBRv2.

### Round Trip Time

Based on connection stats BBRv2 also has a lower RTT than BBRv1 (Figure 6.).
This can be explained by a more graceful PROBE_RTT phase.

Figure 15: Unacked vs RTT heatmap. BBRv1 vs BBRv2.

### Correlations between Min RTT and Bandwidth

If we construct bbr_mrtt vs bbr_bw heatmap then vertical bands represent a
network distance from user to our Tokyo PoP555The 130ms RTT band represents
cross-Pacific traffic and can be attributed to GSLB failure to properly route
users to the closest PoP. This since have been fixed by RUM DNS[19][8]. while
horizontal bands represent common Internet speeds (Figure 16.)

Figure 16: bbr_mrtt vs bbr_bw heatmap.

What’s interesting here is the exponentially decaying relationship between
MinRTT and bandwidth for both BBRv1 and BBRv2. This means that there are some
cases where bandwidth is artificially limited by RTT. Since this behavior is
the same between BBRv1 and BBRv2 we did not dig too much into it. It’s likely
though that the bandwidth is being artificially limited by the user’s receive
window.

### CPU Usage

Previously, BBRv1 updated the whole model on each ACK received, which is quite
a lot of work given that our average server receives millions of them every
second. BBRv2 has an even more sophisticated network model but it also adds a
fast path for ACK processing666Along with ACK fast-path other optimizations to
the BBR were introduced, aimed at both CPU usage improvements and goodput
increase e.g. improved TSO auto-sizing, faster ACKs, and reduced standing
queuing at the bottleneck with many competing BBR flows[14]. that skips model
updates for the application-limited case. This, in theory, should greatly
reduce CPU usage on common workloads.

In our tests, though, we did not see any measurable difference in CPU usage
between BBRv1 and BBRv2. This is likely due to BBRv2 having quite a lot of
debug code enabled.

## Conclusions

In our testing, BBRv2 showed the following properties:

  * •

Bandwidth is comparable to CUBIC for users with lower percentiles of Internet
speeds.

  * •

Bandwidth is comparable to BBRv1 for users with higher percentiles of Internet
speeds.

  * •

Packet loss is 4 times lower compared to BBRv1777Minus the 0.0001% of the
outliers with a $>60\%$ packet loss.; still 2 times higher than CUBIC.

  * •

Data in-flight is 3 times lower compared to BBRv1; slightly lower than CUBIC.

  * •

RTTs are lower compared to BBRv1; still higher than CUBIC.

  * •

Higher RTT-fairness compared to BBRv1.

Overall, BBRv2 is a great improvement over the BBRv1 and indeed seems way
closer to being a drop-in replacement for Reno/CUBIC in cases where one needs
higher bandwidth with lower buffer bloat. Adding the experimental ECN support
to that and we can even see BBRv2 being used as a drop-in replacement for Data
Center TCP (DCTCP).

##  Appendix A Explicit Congestion Notification

ECN is a mechanism for the network bottleneck to proactively notify the sender
to slow down before it runs out of buffers and starts “tail dropping” packets.
Currently, though, ECN on the Internet is mostly deployed in a so-called
“passive” mode. Based on Apple’s data $74\%$ most popular websites “passively”
support ECN[20]. In our Tokyo PoP, we observe $3.68\%$ of connections being
negotiated with ECN, $88\%$ of which have the ecnseen flag.

One downside of the Classic ECN[6] is that it’s too prescriptive about the
explicit congestion signal. Some RFCs, like the “Problem Statement and
Requirements for Increased Accuracy in Explicit Congestion Notification (ECN)
Feedback[17],” call out the low granularity of classic ECN that is only able
to feed back one congestion signal per RTT. Also for a good reason: DCTCP (and
BBRv2888Pay special attention to the CPU usage if you are testing BBR with ECN
enabled since it may render GRO/GSO/TSO unusable for high packet loss
scenarios.) implementation of ECN greatly benefits from its increased
accuracy[1].

Another RFC, namely “Relaxing Restrictions on Explicit Congestion Notification
(ECN) Experimentation[3],” tries to fix it by relaxing this requirement. That
way implementations are free to choose behavior outside of the one specified
by Classic ECN.

Talking about ECN it’s hard to not mention that there is also a “congestion-
notification conflict” going over a single code point (a half a bit) of space
in the IP header between the “Low Latency, Low Loss, Scalable Throughput
(L4S)”[4] proposal and the bufferbloat folks behind the “The Some Congestion
Experienced ECN Codepoint (SCE)”[18] draft.

As Jonathan Corbet summarizes it:

> “These two proposals are clearly incompatible with each other; each places
> its own interpretation on the ECT(1) value and would be confused by the
> other. The SCE side argues that its use of that value is fully compatible
> with existing deployments, while the L4S proposal turns it over to private
> use by suitably anointed protocols that are not compatible with existing
> congestion-control algorithms. L4S proponents argue that the dual-queue
> architecture is necessary to achieve their latency objectives; SCE seems
> more focused on fixing the endpoints.”

Time will show which, if any, draft is approved by IETF, in the meantime, we
can all help the Internet by deploying AQMs[2](e.g. fq-codel[10], cake[11]) to
the network bottlenecks under our control.

##  Appendix B Beyond As Fast As Possible

### Evolving from AFAP – Teaching NICs about time

There is a great talk by Van Jacobson about the evolution of computer networks
and that sending “as fast as possible” is not an optimal strategy in today’s
Internet and even inside a datacenter[16].

This talk is a great summary of the reasons why one might consider using
pacing on the network layer and a delay-based congestion control algorithm.

### Fair Queue scheduler

All our Edge boxes are running Fair Queue qdisc. Our main goal is not the fair
queuing itself but the pacing introduced by sch_fq999Earlier fq
implementations did add some jitter to TCP’s RTT estimations which can be
problematic inside the data center since it will likely inflate p99s of RPC
requests. This was solved in “tcp: switch to Early Departure Time model.” and
should be available since Linux 4.20..

One can use bpftrace qdisc-fq.bt101010qdisc-fq.bt is a part of supplementary
material to the “BPF Performance Tools: Linux System and Application
Observability” book by Brendan Gregg[7]. to measure the time difference
between packets being enqueued into the qdisc and dequeued from it, and hence
the effect of pacing on the network.

Listing 2: qdisc-fq.bt output on a live system.

⬇

# bpftrace qdisc-fq.bt

@us:

[0] 237486 | |

[1] 8712205 |@@@@ |

[2, 4) 21855350 |@@@@@@@@@@@@ |

[4, 8) 4378933 |@@ |

[8, 16) 372762 | |

[16, 32) 178145 | |

[32, 64) 279016 | |

[64, 128) 603899 | |

[128, 256) 1115705 | |

[256, 512) 2303138 |@ |

[512, 1K) 2702993 |@@ |

[1K, 2K) 11999127 |@@@@@@@ |

[2K, 4K) 5432353 |@@@ |

[4K, 8K) 1823173 |@ |

[8K, 16K) 778955 | |

[16K, 32K) 385202 | |

[32K, 64K) 146435 | |

[64K, 128K) 31369 | |

[128K, 256K) 2967 | |

[256K, 512K) 271 | |

In our tests, deploying FQ across the Dropbox internal network essentially
eliminated the premium queue frame discards on shallow-buffered switches.

## References

  * Alizadeh et al. [2010] Alizadeh, M.; Greenberg, A.; Maltz, D. A.; Padhye, J.; Patel, P.; Prabhakar, B.; Sengupta, S.; and Sridharan, M.  2010\.  Data center TCP (DCTCP).  ACM SIGCOMM Computer Communication Review 40(4):63.
  * Baker and Fairhurst [2015] Baker, F., and Fairhurst, G.  2015\.  IETF Recommendations Regarding Active Queue Management.  RFC 7567.
  * Black [2018] Black, D. L.  2018\.  Relaxing Restrictions on Explicit Congestion Notification (ECN) Experimentation.  RFC 8311.
  * Briscoe et al. [2019] Briscoe, B.; Schepper, K. D.; Bagnulo, M.; and White, G.  2019\.  Low Latency and Low Loss and Scalable Throughput (L4S) Internet Service: Architecture.  Internet-Draft draft-ietf-tsvwg-l4s-arch-04, Internet Engineering Task Force.  Work in Progress.
  * Cardwell et al. [2017] Cardwell, N.; Cheng, Y.; Gunn, C. S.; Yeganeh, S. H.; and Jacobson, V.  2017\.  BBR.  Communications of the ACM 60(2):58–66.
  * Floyd, Ramakrishnan, and Black [2001] Floyd, S.; Ramakrishnan, D. K. K.; and Black, D. L.  2001\.  The Addition of Explicit Congestion Notification (ECN) to IP.  RFC 3168.
  * Gregg [2019] Gregg, B.  2019\.  BPF Performance Tools (Addison-Wesley Professional Computing Series).  Addison-Wesley Professional.
  * Guba and Ivanov [2018] Guba, O., and Ivanov, A.  2018\.  Dropbox traffic infrastructure: Edge network.  Available at https://blogs.dropbox.com/tech/2018/10/dropbox-traffic-infrastructure-edge-network/.
  * Hock, Bless, and Zitterbart [2017] Hock, M.; Bless, R.; and Zitterbart, M.  2017\.  Experimental evaluation of BBR congestion control.  In 2017 IEEE 25th International Conference on Network Protocols (ICNP).  IEEE.
  * Høiland-Jørgensen et al. [2018] Høiland-Jørgensen, T.; McKenney, P.; Täht, D.; Gettys, J.; and Dumazet, E.  2018\.  The Flow Queue CoDel Packet Scheduler and Active Queue Management Algorithm.  RFC 8290.
  * Høiland-Jørgensen, Täht, and Morton [2018] Høiland-Jørgensen, T.; Täht, D.; and Morton, J.  2018\.  Piece of CAKE: A comprehensive queue management solution for home gateways.  In 2018 IEEE International Symposium on Local and Metropolitan Area Networks (LANMAN).  IEEE.
  * iet [2019a] 2019a.  IETF ICCRG 104. BBR v2: A Model-based Congestion Control.
  * iet [2019b] 2019b.  IETF ICCRG 105. BBR v2: A Model-based Congestion Control.
  * iet [2019c] 2019c.  IETF ICCRG 106. BBR v2: A Model-based Congestion Control. Performance Optimizations.
  * Ivanov [2017] Ivanov, A.  2017\.  Optimizing web servers for high throughput and low latency.  Available at https://blogs.dropbox.com/tech/2017/09/optimizing-web-servers-for-high-throughput-and-low-latency/#congestion-control.
  * Jacobson [2018] Jacobson, V.  2018\.  Evolving from AFAP: Teaching NICs about time.
  * Kühlewind, Scheffenegger, and Briscoe [2015] Kühlewind, M.; Scheffenegger, R.; and Briscoe, B.  2015\.  Problem Statement and Requirements for Increased Accuracy in Explicit Congestion Notification (ECN) Feedback.  RFC 7560.
  * Morton and Grimes [2019] Morton, J., and Grimes, R.  2019\.  The Some Congestion Experienced ECN Codepoint.  Internet-Draft draft-morton-tsvwg-sce-01, Internet Engineering Task Force.  Work in Progress.
  * Shirokov [2020] Shirokov, N.  2020\.  Intelligent DNS based load balancing at Dropbox.  Available at https://blogs.dropbox.com/tech/2020/01/intelligent-dns-based-load-balancing-at-dropbox/.
  * Stuart Cheshire and David Schinazi and and Christoph Paasch [2017] Stuart Cheshire and David Schinazi and and Christoph Paasch.  2017\.  Advances in networking.  In 2017 WWDC The Apple Worldwide Developers Conference.  Apple.
  * Turkovic et al. [2019] Turkovic; Belma; Kuipers; A., F.; Uhlig; and Steve.  2019\.  Fifty Shades of Congestion Control: A Performance and Interactions Evaluation.  arXiv e-prints arXiv:1903.03852.


#  U-shape Transformer for Underwater Image Enhancement

Lintao Peng
Chunli Zhu
Beijing Institute of Technology  Liheng Bian
Corresponding author<EMAIL_ADDRESS>
###### Abstract

The light absorption and scattering of underwater impurities lead to poor
underwater imaging quality. The existing data-driven based underwater image
enhancement (UIE) techniques suffer from the lack of a large-scale dataset
containing various underwater scenes and high-fidelity reference images.
Besides, the inconsistent attenuation in different color channels and space
areas is not fully considered for boosted enhancement. In this work, we
constructed a large-scale underwater image (LSUI) dataset including 5004 image
pairs, and reported an U-shape Transformer network where the transformer model
is for the first time introduced to the UIE task. The U-shape Transformer is
integrated with a channel-wise multi-scale feature fusion transformer (CMSFFT)
module and a spatial-wise global feature modeling transformer (SGFMT) module,
which reinforce the network’s attention to the color channels and space areas
with more serious attenuation. Meanwhile, in order to further improve the
contrast and saturation, a novel loss function combining RGB, LAB and LCH
color spaces is designed following the human vision principle. The extensive
experiments on available datasets validate the state-of-the-art performance of
the reported technique with more than 2dB superiority. The dataset and demo
code are made available for non-commercial use.

##  1 Introduction

Underwater Image Enhancement (UIE) technology[46, 38] is essential for
obtaining underwater images and investigating the underwater environment,
which has wide applications in ocean exploration, biology, archaeology,
underwater robots[19] and among other fields. However, underwater images
frequently have problematic issues, such as color casts, color artifacts and
blurred details[39]. Those issues could be explained by the strong absorption
and scattering effect on light, which are caused by dissolved impurities and
suspended matter in the medium (water). Therefore, UIE-related innovations are
of great significance in improving the visual quality and merit of images in
accurately understanding the underwater world.

Figure 1: Compared with the existing UIE methods, the image produced by our
U-shape Transformer has the highest PSNR[23] score and best visual quality.

In general, the existing UIE methods can be categorized into three types,
which are physical model-based, visual prior-based and data-driven methods,
respectively. Among them, visual prior-based UIE methods [2, 29, 13, 11, 17,
18] mainly concentrated on improving the visual quality of underwater images
by modifying pixel values from the perspectives of contrast, brightness and
saturation. Nevertheless, the ignorance of the physical degradation process
limits the improvement of enhancement quality. In addition, physical-model
based UIE methods [15, 8, 7, 44, 35, 30, 5, 12, 27] mainly focus on the
accurate estimation of medium transmission. With the estimated medium
transmission and other key underwater imaging parameters such as the
homogeneous background light, a clean image can be obtained by reversing a
physical underwater imaging model. However, the performance of physical model-
based UIE is restricted to complicated and diverse real-world underwater
scenes. That is because, (1)_model hypothesis is not always plausible with
complicated and dynamic underwater environment_ ; (2)_evaluating multiple
parameters simultaneously is challenging._ More recently, as to the data-
driven methods[14, 19, 31, 24, 28, 45, 1, 10, 26, 40, 9, 41, 39], which could
be regarded as deep learning technologies in UIE domain, exhibit impressive
performance on UIE task. However, the existing underwater datasets more-or-
less have the disadvantages, such as a small number of images, few underwater
scenes, or even not real-world scenarios, which limits the performance of the
data-driven UIE method. Besides, the inconsistent attenuation of the
underwater images in different color channels and space areas have not been
unified in one framework.

In this work, we first built a large scale underwater image (LSUI) dataset,
which covers more abundant underwater scenes and better visual quality
reference images than existing underwater datasets[32, 26, 1, 31]. The dataset
contains 5004 real-world underwater images, and the corresponding clear images
are generated as comparison references. Furthermore, with the prior knowledge
that the attenuation of different color channels and space areas in underwater
images is inconsistent, we designed a channel-wise multi-scale feature fusion
transformer (CMSFFT) and a spatial-wise global feature modeling transformer
(SGFMT) based on the attention mechanism, and embedded them in our U-shape
Transformer which is designed based on [20]. Moreover, a multi-scale gradient
flow mechanism[22] is utilized to guarantee the stable training process,
whilst a multi-color space loss function is constructed with the RGB, LAB and
LCH color space in accordance with the human eye’s cognition. Fig. 7 shows the
result of our UIE method and some comparison UIE methods, and the main
contributions of this paper can be summarized as follows:

  * •

We reported a novel U-shape Transformer dealing with the UIE task, in which
the designed channel-wise and spatial-wise attention mechanism enables to
effectively remove color artifacts and casts.

  * •

We designed a novel multi-color space loss function combing the RGB, LCH and
LAB color-space features, which further improves the contrast and saturation
of output images.

  * •

We released a large-scale underwater dataset containing 5004 image pairs,
which facilitates further development of underwater imaging techniques.

##  2 Related work

###  2.1 Data-driven UIE Methods

As we mentioned the pros and cons of physical model-based and visual prior-
based UIE methods in section 1, this part concerns only data-driven UIE
methods.

Current data-driven UIE methods can be divided into two main technical routes,
(1) _designing an end-to-end module_ (2) _utilizing deep models directly to
estimate physical parameters, and then restore the clean image based on the
degradation model_. To alleviate the need for real-world underwater paired
training data, Li et al. [31] proposed a WaterGAN to generate underwater-like
images from in-air images and depth maps in an unsupervised manner, in which
the generated dataset is further used to train the WaterGAN. Moreover, [28]
exhibited a weakly supervised underwater color transmission model based on
CycleGAN[49]. Benefiting from the adversarial network architecture and
multiple loss functions, that network can be trained using unpaired underwater
images, which refines the adaptability of the network model to underwater
scenes. However, images in the training dataset used by the above methods are
not matched real underwater images, which leads to limited enhancement effects
of the above methods in diverse real-world underwater scenes. Recently, Li et
al.[26] proposed a gated fusion network named WaterNet, which uses gamma-
corrected images, contrast-improved images, and white-balanced images as the
inputs to enhance underwater images. Yang et al.[47] proposed a conditional
generative adversarial network (cGAN) to improve the perceptual quality of
underwater images.

The methods mentioned above usually use existing deep neural networks for
general purposes directly on UIE tasks and neglect the unique characteristics
of underwater imaging. For example, [28] directly used the CycleGAN [49]
network structure, and [26] adopted a simple multi-scale convolutional
network. Other models such as UGAN [9],WaterGAN[31] and cGAN[47], they still
inherited the disadvantage of GAN-based models, which produces unstable
enhancement results. In addition, Ucolor [24] combined the underwater physical
imaging model and designed a medium transmission guided model to reinforce the
network’s response to areas with more severe quality degradation, which could
improve the visual quality of the network output to a certain extent. However,
physical models sometimes failed with varied underwater environments.

From above, our proposed network aims at generating high visual quality
underwater images by properly accounting the inconsistent attenuation
characteristics of underwater images in different color channels and space
areas.

###  2.2 Underwater Image Datasets

The sophisticated and dynamic underwater environment results in extreme
difficulties in the collection of matched underwater image training data in
real-world underwater scenes. Present datasets can be classified into two
types, they are, (1) Non-reference datasets. Liu et al. [32] proposed RUIE
dataset, which encompasses varied underwater lighting, depth of field,
blurriness and color cast scenes. Akkaynak et al.[1] published a non-reference
underwater dataset with a standard color comparison chart. Those datasets,
however, cannot be used for end-to-end training for lacking matched clear
reference underwater images. (2) Full-reference datasets. Li et al.[31]
presented an unsupervised network dubbed WaterGAN to produce underwater-like
images using in-air images and depth maps. Similarly, Fabbri et al.[9] used
CycleGAN to generate distorted images from clean underwater images based on
weakly supervised distribution transfer. However, these methods rely heavily
on training samples, which is easy to produce artifacts that are out of
reality and unnatural. Li et al.[26] constructed a real UIE benchmark UIEB,
including 890 images pairs, in which reference images were hand-crafted using
the existing optimal UIE methods. Although those images are authentic and
reliable, the number, content and coverage of underwater scenes are limited.
In contrast, our LSUI dataset contains 5004 real underwater images pairs,
which has abundant underwater environments and higher visual quality
references.

##  3 Proposed dataset and method

Figure 2: The network structure of the reported U-shape Transformer. CMSFFT
and SGFMT reinforce the network’s attention to the more severely attenuated
color channels and spatial regions. The multi-scale connections of the
generator and the discriminator make the gradient flow freely between the
generator and the discriminator, therefore making the training process more
stable.

###  3.1 LSUI Dataset

Data Collection. We have collected 8018 underwater images, which are sourced
from Bubble Vision111https://www.naturefootage.com/stock-video/ocean-
underwater-video, BIOS222https://coral.bios.edu/in-water-validation/ and
existing datasets[32, 1, 31, 9, 26]. Real underwater images with rich water
scenes, water types, lighting conditions and target categories, are selected
to the extent possible, for further generating clear reference images.

Reference Image Generation. Inspired by ensemble learning [36] that multiple
weak classifiers could form a strong one, we use 17 existing optimal UIE
methods[2, 11, 35, 8, 7, 30, 5, 12, 27, 19, 31, 28, 45, 10, 40, 41] to process
the collected underwater images successively. Moreover, a commercial software
Dive+333http://dive.plus/ is used to make the alternative reference images.
Accordingly, a set with $18*8018$ images is generated for the next-step
optimal reference dataset selection.

In order to reduce the number of images that need to be selected manually,
non-reference metrics UIQM[34] and UCIQE[48] are adopted to score all
generated images with equal weights. Then, the top-three reference images of
each original one form a set with the size $3*8018$. Considering individual
differences, 10 volunteers were invited to rate images with a score from 0 to
10, where the higher score represents the more contentedness. Note here
original image would also be provided to volunteers. With subjective scores
from 10 volunteers, and objectively scores from UIQM and UCIQE, each reference
picture could obtain 12 scores. Here, the manual scores were normalized
between 0 and 1, for further summing with UIQM and UCIQE. The top-one
reference image was chosen with the highest summation. In addition, images
with the highest summation lower than 8 have been removed from the dataset.
Eventually, our LSUI dataset contains 5004 pair of images.

###  3.2 U-shape Transformer

####  3.2.1 Overall Architecture

The overall architecture of the U-shape Transformer is shown as Fig. 8, which
includes a CMSFFT & SGFMT based generator and a discriminator.

In the generator, (1) Encoding: Except being directly input to the network,
the original image will be downsampled three times respectively. Then after
1*1 convolution, the three scale feature maps are input into the corresponding
scale convolution block. The outputs of four convolutional blocks are the
inputs of the CMSFFT and SGFMT; (2) Decoding: After feature remapping, the
SGFMT output is directly sent to the first convolutional block. Meanwhile,
four convolutional blocks with varied scales will receive the four outputs
from CMSFFT.

In the discriminator, the input of the four convolutional blocks includes: the
feature map output by its own upper layer, the feature map of the
corresponding size from the decoding part and the feature map generated by
$1*1$ convolution after downsampling to the corresponding size using the
reference image. With the described multi-scale connections, the gradient flow
can flow freely on multiple scales between the generator and the
discriminator, such that a stable training process could be obtained, details
of the generated images could be enriched. The detailed structure of SGFMT and
CMSFFT in the network will be described in the following two subsections.

####  3.2.2 SGFMT

Figure 3: Data flow diagram of the SGFMT module.

The SGFMT (as shown in Fig. 9) is used to replace the original bottleneck
layer of the generator, which can assist the network to model the global
information and reinforce the network’s attention on severely degraded parts.
Assuming the size of the input feature map is
$F_{in}\in\mathbb{R}^{\frac{H}{16}*\frac{W}{16}\ *C}$. For the expected one-
dimensional sequence of the transformer, linear projection is used to stretch
the two-dimensional feature map into a feature sequence
$S_{in}\in\mathbb{R}^{\frac{HW}{256}\ast\ C}$. For preserving the valued
position information of each region, learnable position embedding is merged
directly, which can be expressed as,

$S_{in}=W\ast F_{in}+{\rm{PE}},$ (1)

where $W*F_{i}$ represents a linear projection operation, PE represents a
position embedding operation.

Then we input the feature sequence $S_{in}$ to the transformer block, which
contains 4 standard transformer layers[42]. Each transformer layer contains a
multi-head attention block (MHA) and a feed-forward network (FFN). The FFN
includes a normalization layer and a fully connected layer. The output of the
$l$-th$(l\in[1,2,\ldots.,l])$ layer in the transformer block can be calculated
by,

$S_{l}^{{}^{\prime}}={\rm MHA}({\rm LN}(S_{l-1}))+S_{l-1}$ (2) $S_{l}={\rm
FFN}({\rm LN}(S_{l}^{{}^{\prime}}))+S_{l}^{{}^{\prime}},$ (3)

where LN represents layer normalization, and $S_{l}$ represents the output
sequence of the $l$-th layer in the transformer block. The output feature
sequence of the last transformer block is
$S_{l}\in\mathbb{R}^{\frac{HW}{256}\ast\ C}$, which is restored to the feature
map of $F_{out}\in\mathbb{R}^{\frac{H}{16}\ast\ \frac{W}{16}\ \ast C}$ after
feature remapping.

####  3.2.3 CMSFFT

Figure 4: Detailed structure of the CMSFFT module.

To reinforce the network’s attention on the more serious attenuation color
channels, inspired by [43], we designed the CMSFFT block to replace the skip
connection of the original generator’s encoding-decoding architecture
(Fig.10), which consists of the following three parts.

Multi-Scale Feature Encoding. The inputs of CMSFFT are the feature maps
$F_{i}\in\mathbb{R}^{\frac{H}{2^{i}}*\frac{W}{2^{i}}*C_{i}}(i=0,1,2,3)$ with
different scales. Differs from the linear projection in Vit[6] which is
applied directly on the partitioned original image, we use convolution kernels
with related filter size $\frac{P}{2^{i}}*\frac{P}{2^{i}}(i=0,1,2,3)$ and step
size $\frac{P}{2^{i}}(i=0,1,2,3)$, to conduct linear projection on feature
maps with varied scales. In this work, $P$ is set as 32. After that, four
feature sequence $S_{i}\in\mathbb{R}^{d\ast\ C_{i}}(i=1,2,3,4)$ could be
obtained, where $d\in\frac{HW}{P^{2}}$. Those four convolution kernels divide
feature maps into the same number of blocks, while the number of channels
${C_{i}}(i=1,2,3,4)$ remains unchanged. Then, four query vectors
$Q_{i}\in\mathbb{R}^{d\ast\ C_{i}}(i=1,2,3,4)$, $K\in\mathbb{R}^{d\ast\ C}$
and $V\in\mathbb{R}^{d\ast\ C}$ can be obtained by Eq.(4).

$Q_{i}=S_{i}W_{Q_{i}}\quad K=SW_{K}\quad V=SW_{V},$ (4)

where $W_{Q_{i}}\in\mathbb{R}^{d\ast\ C_{i}}(i=1,2,3,4)$,
$W_{K}\in\mathbb{R}^{d\ast\ C}$ and $W_{V}\in\mathbb{R}^{d\ast\ C}$ stands for
learnable weight matrices; S is generated by concatenating
$S_{i}\in\mathbb{R}^{d\ast\ C_{i}}(i=1,2,3,4)$ via the channel dimension,
where $C=C_{1}+C_{2}+C_{3}+C_{4}$. In this work, $C_{1}$, $C_{2}$, $C_{3}$,
and $C_{4}$ are set as 64, 128, 256, 512, respectively.

Channel-Wise Multi-Head Attention(CMHA). The CMHA block has six inputs, which
are $K\in\mathbb{R}^{d\ast\ C}$, $V\in\mathbb{R}^{d\ast\ C}$ and
$Q_{i}\in\mathbb{R}^{d\ast\ C_{i}}(i=1,2,3,4)$. The output of channel-wise
attention ${\rm CA_{i}}\in\mathbb{R}^{C_{i}\ast\ d}(i=1,2,3,4)$ could be
obtained by,

$\displaystyle{\rm CA}_{i}$ $\displaystyle={\rm SoftMax}({\rm
IN}(\frac{Q_{i}^{T}K}{\sqrt[2]{C}}))V^{T},$ (5)

where IN represents the instance normalization operation. This attention
operation performs along the channel-axis instead of the classical patch-
axis[6], which can guide the network to pay attention to channels with more
severe image quality degradation. In addition, IN is used on the similarity
maps to assist the gradient flow spreads smoothly.

The output of the $i$-th CMHA layer can be expressed as,

${\rm
CMHA_{i}}=({\rm{CA}_{i}}^{1}+{\rm{CA}_{i}}^{2}+.......,+{\rm{CA}_{i}}^{N})/N+Q_{i},$
(6)

where $N$ is the number of heads, which is set as 4 in our implementation.

Feed-Forward Network(FFN). Similar to the forward propagation of [6], the FFN
output can be expressed as,

$O_{i}=\rm{CMHA}_{i}+\rm{MLP}(\rm{LN}(\rm{CMHA_{i}})),$ (7)

where $O_{i}\in\mathbb{R}^{d\ast\ C_{i}}(i=1,2,3,4)$; MLP stands for multi-
layer perception. Here, The operation in Eq. (7) needs to be repeated $l$
($l$=4 in this work) times in sequence to build the $l$-layer transformer.

Finally, feature remappings are performed on the four different output feature
sequences $O_{i}\in\mathbb{R}^{C_{i}\ast\ d}(i=1,2,3,4)$ to reorganize them
into four feature maps
$F_{i}\in\mathbb{R}^{\frac{H}{2^{i}}*\frac{W}{2^{i}}*C_{i}}(i=0,1,2,3)$ ,
which are the input of convolutional block in the generator’s decoding part.

###  3.3 Loss Function

To take advantage of the LAB and LCH color spaces’ wider color gamut
representation range and more accurate description of the color saturation and
brightness, we designed a multi-color space loss function combining RGB, LAB
and LCH color spaces to train our network. The image from RGB space is firstly
converted to LAB and LCH space, and reads,

$\displaystyle L^{G(x)},A^{G(x)},B^{G(x)}=\rm{RGB2LAB}(G(x))$ (8)
$\displaystyle L^{y},A^{y},B^{y}=\rm{RGB2LAB}(y),$ $\displaystyle
L^{G(x)},C^{G(x)},H^{G(x)}=\rm{RGB2LCH}(G(x)),$ (9) $\displaystyle
L^{y},C^{y},H^{y}=\rm{RGB2LCH}(y),$

where $x$, $y$ and $G(x)$ represents the original inputs, the reference image,
and the clear image output by the generator, respectively.

Loss functions in the LAB and LCH space are written as Eq.(10) and Eq.(11).

$\displaystyle Loss_{LAB}(G(x),y)=E_{x,y}[{(L^{y}-L^{G(x})}^{2}-$ (10)
$\displaystyle\sum_{i=1}^{n}{Q(A_{i}^{y})log(Q(A_{i}^{G(x)}))-\sum_{i=1}^{n}{Q(B_{i}^{y})log(Q(B_{i}^{G(x)}))}}],$
$\displaystyle
Loss_{LCH}(G(x),y)=E_{x,y}[-\sum_{i=1}^{n}{Q(L_{i}^{y})log(Q(L_{i}^{G(x)}))}$
(11) $\displaystyle+{(C^{y}-C^{G(x)})}^{2}+{(H^{y}-H^{G(x)})}^{2}],$

where $Q$ stands for the quantization operator.

${L}_{2}$ loss in the RGB color space $Loss_{RGB}$ and the perceptual loss
$Loss_{per}$[21] , as well as $Loss_{LAB}$ and $Loss_{LCH}$ are the four loss
functions for the generator.

Besides, standard GAN loss function is introduced for minimizing the loss
between generated and reference pictures, and written as,

$\displaystyle L_{GAN}(G,D)=E_{y}[logD(y)]+E_{x}[log(1-D(G(x)))],$ (12)

where $D$ represents the discriminator. D aims at maximizing $L_{GAN}(G,D)$,
to accurately distinguish the generated image from the reference image. And
the goal of generator G is to minimize the loss between generated pictures and
reference pictures.

Then, the final loss function is expressed as,

$\displaystyle
G^{*}=arg\mathop{\min}\limits_{G}\mathop{\max}\limits_{D}L_{GAN}(G,D)+\alpha{Loss}_{LAB}(G(x),y)$
(13) $\displaystyle+\beta{Loss}_{LCH}(G(x),y)+\gamma{Loss}_{RGB}(G(x),y)$
$\displaystyle+\mu{Loss}_{per}(G(x),y),$

where $\alpha,\beta,\gamma,\mu$ are hyperparameters, which are set as 0.001,
1, 0.1, 100, respectively, with numerous experiments.

##  4 Experiments

###  4.1 Implementation Details

The LSUI dataset was randomly divided as Train-L (4500 images) and Test-L504
(504 images) for training and testing, respectively. The training set was
enhanced by cropping, rotating and flipping the existing images. All images
were adjusted to a fixed size (256*256) when input to the network, and the
pixel value will be normalized to [0,1].

We use python and pytorch framework via NVIDIA RTX3090 on Ubuntu20 to
implement the U-shape Transformer. Adam optimization algorithm is utilized for
the total of 800 epochs training with batchsize set as 6. The initial learning
rate is set as 0.0005 and 0.0002 for the first 600 epochs and the last 200
epochs, respectively. Besides, the learning rate decreased 20% every 40
epochs. For $Loss_{RGB}$, $L_{2}$ loss is used for the first 600 epochs, and
$L_{1}$ loss is used for the last 200 epochs.

###  4.2 Experiment Settings

Benchmarks. Besides Train-L, the second training set Train-U contains 800
pairs of underwater images from UIEB[26] and 1,250 synthetic underwater images
from [25]; the third training set Train-E contains the paired training images
in the EUVP[19] dataset. Testing datasets are categorized into two types, (1)
full-reference testing dataset: Test-L504 and Test-U90 (remaining 90 pairs in
UIEB); (2) non-reference testing dataset: Test-U60 and SQUID. Here, Test-U60
includes 60 non-reference images in UIEB; 16 pictures from SQUID[1] forms the
second non-reference testing dataset.

Compared Methods. We compare U-shape Transformer with 10 UIE methods to verify
our performance superiority. It includes two physical-based models (UIBLA[35],
UDCP[8]), three visual prior-based methods (Fusion[2], retinex based[11],
RGHS[17]), and five data-driven methods (WaterNet[26], FUnIE[19], UGAN[9],
UIE-DAL[41], Ucolor[24]).

Evaluation Metrics. For the testing dataset with reference images, we
conducted full-reference evaluations using PSNR[23] and SSIM[16] metrics.
Those two metrics reflect the proximity to the reference, where a higher PSNR
value represents closer image content, and a higher SSIM value reflects a more
similar structure and texture.

For images in the non-reference testing dataset, non-reference evaluation
metrics UCIQE[48] and UIQM[34] are employed, in which higher UCIQE or UIQM
score suggests better human visual perception. For UCIQE and UIQM cannot
accurately measure the performance in some cases [26] [3], we also conducted
an user survey following [24], which results are stated as “perception score
(PS)”. In the user survey, 10 volunteers with image processing background were
invited to score the enhancement results. The integral score from 1 to 5
represents image quality from the worst to the best, which criterion is from
color deviation, artifacts, natural looking, and contrast & visibility. Each
image’s final PS score is the average value of all volunteers. Moreover, NIQE
[33], which lower value represents a higher visual quality, is also adopted as
the metrics.

###  4.3 Dataset Evaluation

The effectiveness of LSUI is evaluated by retraining the compared methods
(U-net[37], UGAN[9] and U-shape Transformer) on Train-L, Train-U and Train-E.
The trained network was tested on Test-L504 and Test-U90. As shown in Tab.1,
the model trained on our dataset is the best of PSNR and SSIM. It could be
explained that LSUI contains richer underwater scenes and better visual
quality reference images than existing underwater image datasets, which can
improve the enhancement and generalization ability of the tested network. The
visual comparisons can be found in the supplementary material.

Table 1: Dataset evaluation results. The highest PSNR and SSIM scores are marked in red. Methods |  Training Data | Test-U90 | TestL-504
---|---|---|---
PSNR | SSIM | PSNR | SSIM
U-net[37] | Train-U | 17.07 | 0.76 | 19.19 | 0.79
Train-E | 17.46 | 0.76 | 19.45 | 0.78
Ours | 20.14 | 0.81 | 20.89 | 0.82
UGAN[9] | Train-U | 20.71 | 0.82 | 19.89 | 0.79
Train-E | 20.72 | 0.82 | 19.82 | 0.78
Ours | 21.56 | 0.83 | 21.74 | 0.84
Ours | Train-U | 21.25 | 0.84 | 22.87 | 0.85
Train-E | 21.75 | 0.86 | 23.01 | 0.87
Ours | 22.91 | 0.91 | 24.16 | 0.93

###  4.4 Network Architecture Evaluation

Figure 5: Visual comparison of enhancement results sampled from the Test-L504
dataset. From left to right are raw images, results of Fusion[2], Retinex
based[11], FUnIE[19], UGAN[9], Ucolor[24], our U-shape Transformer and the
reference image (recognized as ground truth (GT)). More examples can be found
in the supplementary material.

Full-Reference Evaluation. The Test-L504 and Test-U90 datasets were used for
evaluation. The statistical results and visual comparisons are summarized in
Tab. 2 and Fig. 11. For the 5 deep learning based UIE methods, we used the
code and pre-training weights provided by the authors.

Table 2: Quantitative comparison among different UIE methods on the full-reference testing set. The highest scores are marked in red. Methods | Test-L504 | Test-U90
---|---|---
PSNR | SSIM | PSNR | SSIM
UIBLA[35] | 13.54 | 0.71 | 15.78 | 0.73
UDCP[8] | 11.89 | 0.59 | 13.81 | 0.69
Fusion[2] | 17.48 | 0.79 | 19.04 | 0.82
Retinex based[11] | 13.89 | 0.74 | 14.01 | 0.72
RGHS[17] | 14.21 | 0.78 | 14.57 | 0.79
WaterNet[26] | 17.73 | 0.82 | 19.81 | 0.86
FUnIE[19] | 19.37 | 0.84 | 19.45 | 0.86
UGAN[9] | 19.79 | 0.78 | 20.68 | 0.84
UIE-DAL[41] | 17.45 | 0.79 | 16.37 | 0.78
Ucolor[24] | 22.91 | 0.89 | 20.78 | 0.87
Ours | 24.16 | 0.93 | 22.91 | 0.91

As in Tab.2, our U-shape Transformer demonstrates the best performance for
both PSNR and SSIM metrics. The potential limitations of the poor performance
of the 5 data-driven methods are analyzed as follows. FUnIE[19] aims at
achieving fast, lightweight and fewer parameter models, thereby it is easy to
reach the performance bottleneck on complex and distorted testing samples.
UGAN[9] and UIE-DAL[41] ignore the inconsistent characteristics of the
underwater images. Although Ucolor[24] also introduced the concept of multi-
color space, it simply adds them to the network’s encoding part, which cannot
fully utilize the advantages of multi-color space.

The visual comparisons shown in Fig. 11 reveal that enhancement results of our
method are the closest to the reference image, which has fewer color artifacts
and high-fidelity object areas. Five selected methods tend to produce color
artifacts that deviated from the original color of the object. Among the
methods, Fusion[2] exhibits severe color artifacts. Retinex based[11] could
improve the image contrast to a certain extent, but cannot remove the color
casts and color artifacts effectively. The enhancement result of FUnLE[19] is
yellowish overall. Although UGAN[9] and Ucolor[26] could provide relatively
good color appearance, they are often affected by local over-enhancement, and
there are still some color casts in the result.

Non-reference Evaluation. The Test-U60 and SQUID datasets were utilized for
the non-reference evaluation, in which statistical results and visual
comparisons are shown in Tab. 3 and Fig. 12.

Table 3: Quantitative comparison among different UIE methods on the non-reference testing set. The highest scores are marked in red.  Methods | Test-U60 | SQUID
---|---|---
PS↑ | UIQM↑ | UCIQE↑ | NIQE↓ | PS↑ | UIQM↑ | UCIQE↑ | NIQE↓
input | 1.46 | 0.82 | 0.45 | 7.16 | 1.23 | 0.81 | 0.43 | 4.93
UIBLA[35] | 2.18 | 1.21 | 0.60 | 6.13 | 2.45 | 0.96 | 0.52 | 4.43
UDCP[8] | 2.01 | 1.03 | 0.57 | 5.94 | 2.57 | 1.13 | 0.51 | 4.47
Fusion[2] | 2.12 | 1.23 | 0.61 | 4.96 | 2.89 | 1.29 | 0.61 | 5.01
Retinex based[11] | 2.04 | 0.94 | 0.69 | 4.95 | 2.33 | 1.01 | 0.66 | 4.86
RGHS[17] | 2.45 | 0.66 | 0.71 | 4.82 | 2.67 | 0.82 | 0.73 | 4.54
WaterNet[26] | 3.23 | 0.92 | 0.51 | 6.03 | 2.72 | 0.98 | 0.51 | 4.75
FUnIE[19] | 3.12 | 1.03 | 0.54 | 6.12 | 2.65 | 0.98 | 0.51 | 4.67
UGAN[9] | 3.64 | 0.86 | 0.57 | 6.74 | 2.79 | 0.90 | 0.58 | 4.56
UIE-DAL[41] | 2.03 | 0.72 | 0.54 | 4.99 | 2.21 | 0.79 | 0.57 | 4.88
Ucolor[24] | 3.71 | 0.84 | 0.53 | 6.21 | 2.82 | 0.82 | 0.51 | 4.32
Ours | 3.91 | 0.85 | 0.73 | 4.74 | 3.23 | 0.89 | 0.67 | 4.24

As in Tab. 3, our method achieved the highest scores on PS and NIQE metrics,
which confirmed the initial idea to contemplate the human eye’s color
perception and better generalization ability to varied real-world underwater
scenes. Note that UCIQE and UIQM of all deep learning-based UIE methods are
weaker than physical model-based or visual prior-based, also reported in [24].
Those two metrics are of valuable reference, but cannot as absolute
justifications [26][3], for they are non-sensitive to color artifacts & casts
and biased to some features.

UGAN and Ucolor are selected for visual comparison according to previous
results, as in Fig. 12, where UGAN has obvious color artifacts in some local
areas and Ucolor has an obvious color cast. That is due to the architecture of
UGAN and Ucolor ignoring the inconsistent attenuation characteristics of
underwater image on different image areas and color channels, respectively. In
our method, the reported CMSFFT and SGFMT modules could reinforce the
network’s attention to the color channels and spatial regions with serious
attenuation, therefore obtaining high visual quality enhancement results
without color artifacts and color casts.

Figure 6: Visual comparison sampled from the Test-U60 dataset. UGAN[9] and
Ucolor[24] are selected for visual comparison considering their performance in
full-reference experiment. More examples can be found in the supplementary
material.

###  4.5 Ablation Study

To prove the effectiveness of each component, we conduct a series of ablation
studies on the Test-L504 and Test-U90. Four factors are considered including
the CMSFFT, the SGFMT, the multi-scale gradient flow mechanism (MSG) and the
multi-color space loss function (MCSL).

Experiments are all trained by Train-L. Statistical results are shown in Tab.
4, in which baseline model (BL) refers to [20], full models is the complete
U-shape Transformer. In Tab. 4, our full model achieves the best quantitative
performance on the two testing dataset, which reflects the effectiveness of
the combination of CMSFFT, SGFMT, MSG and MCSL modules. Compared with the BL
model, BL+CMSFFT, BL+SGMFT, BL+MSG and BL+MCSL have achieved better results,
which proves the effectiveness of each module.

Table 4: Statistical results of ablation study on the Test-L504 and the Test-U90. The highest scores are marked in red. Models | Test-L504 | Test-U90
---|---|---
PSNR | SSIM | PSNR | SSIM
BL | 19.34 | 0.79 | 19.36 | 0.81
BL+CMSFFT | 22.47 | 0.88 | 21.72 | 0.86
BL+SGFMT | 21.78 | 0.86 | 21.36 | 0.87
BL+MSG | 20.11 | 0.82 | 21.24 | 0.85
BL+MCSL | 21.51 | 0.82 | 20.16 | 0.81
Full Model | 24.16 | 0.93 | 22.91 | 0.91

##  5 Conclusions

In this work, we released the LSUI dataset which is the largest real-world
underwater dataset with high-fidelity reference images. Besides, we reported
an U-shape Transformer network for state-of-the-art enhancement. The network’s
CMSFFT and SGFMT modules could solve the inconsistent attenuation issue of
underwater images in different color channels and space regions, which has not
been considered among existing methods. Extensive experiments validate the
superior ability of the network to remove color artifacts and casts. Combined
with the multi-color space loss function, the contrast and saturation of
output images are further improved. Nevertheless, it is impossible to collect
images of all the complicated scenes such as deep-ocean low-light scenarios.
Therefore, we will introduce other general enhancement techniques such as low-
light boosting [4] for further improvement.

## References

  * [1] Derya Akkaynak and Tali Treibitz.  Sea-thru: A method for removing water from underwater images.  In CVPR, pages 1682–1691, 2019.
  * [2] Cosmin Ancuti, Codruta Orniana Ancuti, Tom Haber, and Philippe Bekaert.  Enhancing underwater images and videos by fusion.  In CVPR, pages 81–88, 2012.
  * [3] Dana Berman, Deborah Levy, Shai Avidan, and Tali Treibitz.  Underwater single image color restoration using haze-lines and a new quantitative dataset.  IEEE TPAMI, 43(8):2822–2837, 2021.
  * [4] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun.  Learning to see in the dark.  In CVPR, pages 3291–3300, 2018.
  * [5] John Y. Chiang and Ying-Ching Chen.  Underwater image enhancement by wavelength compensation and dehazing.  IEEE TIP, 21(4):1756–1769, 2012.
  * [6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  ArXiv, abs/2010.11929, 2021.
  * [7] Paulo L.J. Drews, Erickson R. Nascimento, Silvia S.C. Botelho, and Mario Fernando Montenegro Campos.  Underwater depth estimation and image restoration based on single images.  IEEE Comput. Graph. Appl., 36(2):24–35, 2016.
  * [8] P. Drews Jr, E. do Nascimento, F. Moraes, S. Botelho, and M. Campos.  Transmission estimation in underwater single images.  In ICCV workshops, pages 825–830, 2013.
  * [9] Cameron Fabbri, Md. Jahidul Islam, and Junaed Sattar.  Enhancing underwater imagery using generative adversarial networks.  ICRA, pages 7159–7165, 2018.
  * [10] Xueyang Fu, Zhiwen Fan, Mei Ling, Yue Huang, and Xinghao Ding.  Two-step approach for single underwater image enhancement.  In ISPACS, pages 789–794, 2017.
  * [11] Xueyang Fu, Peixian Zhuang, Yue Huang, Yinghao Liao, Xiao-Ping Zhang, and Xinghao Ding.  A retinex-based enhancing approach for single underwater image.  In ICIP, pages 4572–4576, 2014.
  * [12] Adrian Galdran, David Pardo, Artzai Picón, and Aitor Alvarez-Gila.  Automatic red-channel underwater image restoration.  JVCIR, 26:132–145, 2015.
  * [13] Ahmad Shahrizan Abdul Ghani and Nor Ashidi Mat Isa.  Underwater image quality enhancement through composition of dual-intensity images and rayleigh-stretching.  In ICCE, pages 219–220, 2014.
  * [14] Yecai Guo, Hanyu Li, and Peixian Zhuang.  Underwater image enhancement using a multiscale dense generative adversarial network.  IEEE J. OCEANIC. ENG., 45(3):862–870, 2019.
  * [15] Kaiming He, Jian Sun, and Xiaoou Tang.  Single image haze removal using dark channel prior.  In CVPR, pages 1956–1963, 2009.
  * [16] Alain Horé and Djemel Ziou.  Image quality metrics: Psnr vs. ssim.  In ICPR, pages 2366–2369, 2010.
  * [17] Dongmei Huang, Yan Wang, Wei Song, Jean Sequeira, and Sébastien Mavromatis.  Shallow-water image enhancement using relative global histogram stretching based on adaptive parameter acquisition.  In MMM, pages 453–465. Springer, 2018.
  * [18] Kashif Iqbal, Michael Odetayo, Anne James, Rosalina Abdul Salam, and Abdullah Zawawi Hj Talib.  Enhancing the low quality images using unsupervised colour correction method.  In IEEE Int. Conf. Syst. Man. Cybern., pages 1703–1709, 2010.
  * [19] Md Jahidul Islam, Youya Xia, and Junaed Sattar.  Fast underwater image enhancement for improved visual perception.  IEEE Robot. Autom. Lett., 5(2):3227–3234, 2020.
  * [20] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In CVPR, pages 1125–1134, 2017.
  * [21] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In ECCV, 2016.
  * [22] Animesh Karnewar and Oliver Wang.  Msg-gan: Multi-scale gradients for generative adversarial networks.  In CVPR, pages 7796–7805, 2020.
  * [23] Jari Korhonen and Junyong You.  Peak signal-to-noise ratio revisited: Is simple beautiful?  In QoMEX, pages 37–38. IEEE, 2012.
  * [24] Chongyi Li, Saeed Anwar, Junhui Hou, Runmin Cong, Chunle Guo, and Wenqi Ren.  Underwater image enhancement via medium transmission-guided multi-color space embedding.  IEEE TIP, 30:4985–5000, 2021.
  * [25] Chongyi Li, Saeed Anwar, and Fatih Porikli.  Underwater scene prior inspired deep underwater image and video enhancement.  Pattern Recognition, 98:107038, 2020.
  * [26] Chongyi Li, Chunle Guo, Wenqi Ren, Runmin Cong, Junhui Hou, Sam Kwong, and Dacheng Tao.  An underwater image enhancement benchmark dataset and beyond.  IEEE TIP, 29:4376–4389, 2020.
  * [27] Chongyi Li, Jichang Guo, Shanji Chen, Yibin Tang, Yanwei Pang, and Jian Wang.  Underwater image restoration based on minimum information loss principle and optical properties of underwater imaging.  In ICIP, pages 1993–1997, 2016.
  * [28] Chongyi Li, Jichang Guo, and Chunle Guo.  Emerging from water: Underwater image color correction based on weakly supervised color transfer.  IEEE Signal. Process. Lett., 25(3):323–327, 2018.
  * [29] Chong-Yi Li, Ji-Chang Guo, Run-Min Cong, Yan-Wei Pang, and Bo Wang.  Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior.  IEEE TIP, 25(12):5664–5677, 2016.
  * [30] Chong-Yi Li, Ji-Chang Guo, Run-Min Cong, Yan-Wei Pang, and Bo Wang.  Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior.  IEEE TIP, 25(12):5664–5677, 2016.
  * [31] Jie Li, Katherine A Skinner, Ryan M Eustice, and Matthew Johnson-Roberson.  Watergan: Unsupervised generative network to enable real-time color correction of monocular underwater images.  IEEE Robot. Autom. Lett., 3(1):387–394, 2017.
  * [32] Risheng Liu, Xin Fan, Ming Zhu, Minjun Hou, and Zhongxuan Luo.  Real-world underwater enhancement: Challenges, benchmarks, and solutions under natural light.  IEEE Trans. Circuits. Syst. Video Technol., 30:4861–4875, 2020\.
  * [33] Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik.  Making a “completely blind” image quality analyzer.  IEEE Signal Process. Lett., 20(3):209–212, 2013.
  * [34] Karen Panetta, Chen Gao, and Sos Agaian.  Human-visual-system-inspired underwater image quality measures.  IEEE J. Ocean. Eng., 41(3):541–551, 2016.
  * [35] Yan-Tsung Peng and Pamela C Cosman.  Underwater image restoration based on image blurriness and light absorption.  IEEE TIP, 26(4):1579–1594, 2017.
  * [36] Robi Polikar.  Ensemble learning.  In Ensemble machine learning, pages 1–34. Springer, 2012.
  * [37] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In MICCAI, pages 234–241. Springer, 2015.
  * [38] Pooja Sahu, Neelesh Gupta, and Neetu Sharma.  A survey on underwater image enhancement techniques.  IJCA, 87(13), 2014.
  * [39] Raimondo Schettini and Silvia Corchs.  Underwater image processing: State of the art of restoration and image enhancement methods.  EURASIP. J. Adv. Signal Process., 2010:1–14, 2010.
  * [40] Wei Song, Yan Wang, Dongmei Huang, and Dian Tjondronegoro.  A rapid scene depth estimation model based on underwater light attenuation prior for underwater image restoration.  In PCM, pages 678–688. Springer, 2018.
  * [41] Pritish M Uplavikar, Zhenyu Wu, and Zhangyang Wang.  All-in-one underwater image enhancement using domain-adversarial learning.  In CVPR Workshops, pages 1–8, 2019.
  * [42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NIPS, pages 5998–6008, 2017.
  * [43] Haonan Wang, Peng Cao, Jiaqi Wang, and Osmar R. Zaiane.  Uctransnet: Rethinking the skip connections in u-net from a channel-wise perspective with transformer, 2021.
  * [44] Yi Wang, Hui Liu, and Lap-Pui Chau.  Single underwater image restoration using adaptive attenuation-curve prior.  IEEE Trans. Circuits. Syst. I. Regul. Pap., 65(3):992–1002, 2018\.
  * [45] Hung-Yu Yang, Pei-Yin Chen, Chien-Chuan Huang, Ya-Zhu Zhuang, and Yeu-Horng Shiau.  Low complexity underwater image enhancement based on dark channel prior.  In IBICA, pages 17–20, 2011.
  * [46] Miao Yang, Jintong Hu, Chongyi Li, Gustavo Rohde, Yixiang Du, and Ke Hu.  An in-depth survey of underwater image enhancement and restoration.  IEEE Access., 7:123638–123657, 2019.
  * [47] Miao Yang, Ke Hu, Yixiang Du, Zhiqiang Wei, Zhibin Sheng, and Jintong Hu.  Underwater image enhancement based on conditional generative adversarial network.  Signal Process., Image Commun., 81:115723, 2020.
  * [48] Miao Yang and Arcot Sowmya.  An underwater color image quality evaluation metric.  IEEE TIP, 24(12):6062–6071, 2015.
  * [49] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In ICCV, pages 2242–2251, 2017.

##  Appendix A Overview

This supplementary document provides complimentary details about the LSUI
dataset (Sec.B) and comparisons (Sec.C) in the paper. We also provide the
comparison of underwater video enhancement results of different underwater
image enhancement methods in the supplementary video.

##  Appendix B LSUI Dataset

As shown in Fig .7, compared with UIEB[26], our LSUI dataset contains larger
number of images and richer underwater scenes and object categories. In
particular, our LSUI dataset includes deep-sea scenes and underwater cave
scenes that are not available in previous underwater datasets. We provide some
examples of our LSUI dataset in Fig .8, which shows that our LSUI dataset
contains a variety of underwater scenes, water types, lighting conditions and
target categories. As far as we know, LSUI is the largest real underwater
image dataset with high quality reference images until now, and we believe it
will facilitate the further development of underwater imaging techniques.

Figure 7: Statistics of our LSUI dataset and the existing largest real
underwater dataset UIEB[26] with high quality reference images . Figure 8:
Example images in the LSUI dataset. The top of each pair of images is the
reference image, and the bottom is the raw underwater image.

##  Appendix C Additional Visual Comparisons

###  C.1 Visual Examples of Dataset Evaluation

Fig. 9 is the sampled enhancement results of U-shape transformer trained on
different underwater datasets, which is a supplement of the _Data Evaluation_
part of the paper. Enhancement results training on Train-L (a portion of our
LSUI dataset) demonstrates the highest PSNR value and preferable visual
quality, while results training on other datasets show a certain degree of
color cast. For the high-quality reference images and rich underwater scenes
(lighting conditions, water types and target categories), our constructed LSUI
dataset could improve the imaging quality and generalization performance of
the UIE network.

Figure 9: Enhancement results of U-shape transformer trained on different
underwater datasets. (a): Input images sampled from Test-L504; (b): Enhanced
results using the model trained on the Train-U; (c): Enhanced results using
the model trained on the Train-E; (d): Enhanced results using the model
trained by our proposed dataset Train-L; (e): Reference images(recognized as
ground truth (GT)).

###  C.2 Visual Comparisons of Non-Reference Experiments

Fig. 10 is a supplement to the _Non-reference Evaluation_ of the paper.
Enhancement results of our method have the highest PS value, which index
reflects the visual quality. Generally, compared methods are unsatisfactory,
which includes undesirable color artifacts, over-saturation and unnatural
color casts. Among the methods, results of the UIBLA [35] and FUnIE [19] are
collectively reddish and yellowish, respectively. Retinex based[11] method
introduces artifacts and unnatural colors. UGAN and UIT-DAL have the issue of
local over-enhancement and color artifacts, which main reason is they ignore
the inconsistent attenuation characteristics of the underwater images in the
different space areas and the color channels. Although Ucolor [24] introduces
the transmission medium prior to reinforcing the network’s attention on the
spatial area with severe attenuation, it still ignores the inconsistent
attenuation characteristics of the underwater image in different color
channels, which results in the problem of overall color cast.

Figure 10: Visual comparison sampled from the Test-U60 dataset. From left to
right are raw images, results of UIBLA[35], Retinex based[11], FUnIE[19],
UGAN[9], Ucolor[24], UIE-DAL[41] and our U-shape Transformer.

###  C.3 Visual Comparisons of Full-Reference Experiments

Fig. 11 is a supplement to the _Full-reference Evaluation_ of the paper.
Results of our method are the closest with the reference images, which have
the best visual quality and the highest PSNR value. Five selected methods
exhibit a different degree of color artifacts and casts, which enhancement
results differ considerably from the object’s original color and fall far
short from the reference images. Among the methods, Fusion[2] does not remove
the color casts (bias to green) and exhibits severe color artifacts. Retinex
based[11] method could improve the image contrast to a certain extent, but the
color of the enhanced image is unnatural. The enhancement result of FUnLE[19]
is yellowish overall. Although UGAN[9] and Ucolor[26] could provide relatively
good color appearance, they are often affected by local over-enhancement, and
there are still some color casts in the results. The limitation of the UGAN
and Ucolor method is that their network structures do not consider the
inconsistent attenuation of underwater images in different color channels and
spatial regions.

Figure 11: Visual comparison of enhancement results sampled from the Test-L504
dataset. From left to right are raw images, results of Fusion[2], Retinex
based[11], FUnIE[19], UGAN[9], Ucolor[24], our U-shape Transformer and the
reference image (recognized as ground truth (GT)).

###  C.4 Visual Comparisons of Ablation Study

Figure 12: Visual comparison of the ablation study, which images are sampled
from the Test-L504 dataset. The highest PSNR score is marked in yellow.

As shown in Fig .12, the enhancement result of the full model has the highest
PSNR score and best visual quality. The results of BL+MSG have less noise and
artifacts than that of the single BL module, which is due to the mechanism of
MSG helps to reconstruct local details. The results’ overall color of BL+MCSL
are close to the reference picture for which adds a multi-color space loss
function. However, the unevenly distributed visualization and artifacts in
local areas of BL+MCSL are owing to the absence of attention mechanism
guidance. Although the enhanced results of BL+CMSFFT and BL+SGFMT are evenly
distributed, the overall color is not accurate. The investigated four modules
have their particular functionality in the enhancement process, which
integration could improve the overall performance of our network.