# The Effect of Normalization for Bi-directional Amharic-English Neural
Machine Translation

Tadesse Destaw Belay  College of Informatics
Wollo University
Kombolcha, Ethiopia
<EMAIL_ADDRESS>Atnafu Lambebo Tonja  Centro de Investigación en
Computación
Instituto Politécnico Nacional
Mexico City, Mexico
<EMAIL_ADDRESS>Olga Kolesnikova  Centro de Investigación en
Computación
Instituto Politécnico Nacional
Mexico City, Mexico
<EMAIL_ADDRESS>Seid Muhie Yimam  Dept. of Informatics
Universität Hamburg
Hamburg, Germany
<EMAIL_ADDRESS>Abinew Ali Ayele  ICT4D Research Center
Bahir Dar University
Bahir dar, Ethiopia
<EMAIL_ADDRESS>Silesh Bogale Haile  Dept. of Computer Science
Assosa University
Assosa, Ethiopia
<EMAIL_ADDRESS>Grigori Sidorov  Centro de Investigación en
Computación
Instituto Politécnico Nacional
Mexico City, Mexico
<EMAIL_ADDRESS>Alexander Gelbukh  Centro de Investigación en Computación
Instituto Politécnico Nacional
Mexico City, Mexico
<EMAIL_ADDRESS>
###### Abstract

Machine translation (MT) is one of the prominent tasks in natural language
processing whose objective is to translate texts automatically from one
natural language to another. Nowadays, using deep neural networks for MT task
has received a great deal of attention. These networks require lots of data to
learn abstract representations of the input and store it in continuous
vectors. This paper presents the first relatively large-scale Amharic-English
parallel sentence dataset. Using these compiled data, we build bi-directional
Amharic-English translation models by fine-tuning the existing Facebook M2M100
pre-trained model achieving a BLEU score of 37.79 in Amharic-English
translation and 32.74 in English-Amharic translation. Additionally, we explore
the effects of Amharic homophone normalization on the machine translation
task. The results show that normalization of Amharic homophone characters
increases the performance of Amharic-English machine translation in both
directions.

###### Index Terms:

Neural machine translation, pre-trained models, Amharic-English MT, homophone
normalization, low-resourced language

##  I Introduction

Machine translation (MT) is a sub-field of natural language processing (NLP)
that investigates how to use computer software to automatically translate text
or speech from one language to another without human involvement. MT is one of
the prominent tasks in NLP that is tackled in several ways [1]. The first MT
research began at about 1950s and in 1952 the first International Conference
on Machine Translation was organized at the Massachusetts Institute of
Technology (MIT). It has long research history and experienced four stages,
namely, Rule-based MT [2], Statistical MT (SMT) [3], hybrid MT, and Neural MT
(NMT) [4, 5].

The most severe drawback of the rule-based method is that it has ignored the
need for context information in the translation process. It is highly
dependent on hand-crafted features. Phrase-based SMT (PBSMT), the most
prevalent version of SMT, generates translation by segmenting the source
sentence into several phrases and performing phrase translation and
replacement. It may ignore the long sentence dependency and require high
computing devices [6]. Recently, using deep neural networks for MT task has
received great attention. NMT also improves training procedures due to the
end-to-end procedure without tedious feature engineering and complex setups.
NMT employs such techniques as recurrent neural network (RNN) [7],
convolutional neural network (CNN) [8], and self-attention network
(Transformer) [9].

Transformer models with the pre-training approach is a new NMT strategy
entirely based on attention mechanisms proposed in 2017 [9]. Among the
different neural network architectures, the Transformer model has emerged as
the dominant NMT paradigm [10, 11, 12]. It has become the state-of-the-art
model for many artificial intelligence tasks, including machine translation.
In terms of model, the Transformer-based pre-trained models are fast to fine-
tune, highly accurate and has been proven to outperform widely used recurrent
networks [13, 6, 14].

The focus of MT research for the Amharic language has been on rule-based and
SMT methods. In this work, we used the transformer model as a baseline
translation system to explore the applicability of Facebook M2M100 multi-
lingual pre-trained language model for Amharic-English translation in both
directions. Furthermore, this research work investigated the impact of
normalization of the Amharic homophones on Amharic-English MT tasks.

The main contributions of this work are:

  1. 1.

Exploration of the Amharic-English and English-Amharic machine translation
tasks.

  2. 2.

Introduction of the first large-scale publicly available Amharic-English
translation parallel dataset.

  3. 3.

Development and implementation of state-of-the-art Amharic-English translation
models.

  4. 4.

Investigation of the effect of Amharic homophone character normalization on
the machine translation task.

The rest of this paper is organized as follows. Section II presents a detail
description of Amharic language while Section III shows the motivation for
this research. In Section IV, we review related work. Section V describes the
existing parallel corpus and the collection of a new corpus from the news
domain. The general pre-processing steps applied to both corpora are presented
in Section VI. Section VII discusses the proposed NMT models and Section VIII
gives the experimental results. In the end, Section IX concludes the paper and
sheds some light on possible future work.

##  II Amharic language

Amharic is the second most spoken Semitic language next to Arabic which has
its own alphabet and writing scripts called ’Fidel’, that was borrowed from
Ge’ez, another Ethiopian Semitic language. Fidel is a syllable-based writing
system where the consonants and vowels co-exist within each graphic symbol.
The Amharic language is spoken by more than 57 million people with up to 32
million native speakers and 25 million non-native speakers [15]. Amharic is
the working language of the Federal Democratic Republic of Ethiopia (FDRE) and
for many regional states in the country. In Amharic, there are 34 core
characters each having seven different derivatives to represent vowels. In
addition, it has 20 labialized characters, more than 20 numerals, and 8
punctuation marks. Amharic uses a total of more than 310 characters. The
language is known for being morphologically complex and it is highly
inflectional. Unlike English, French, Spanish, Japanese, and Chinese, Amharic
is considered low-resource because the data are not well organized and
technologically less supported [16].

##  III Motivation

Nowadays advancement in technology has made the lifestyle of human beings much
easier by helping daily activities. One of the applications that solved
communication barriers between people speaking different languages is machine
translation. Many big technology companies such as Google, Microsoft, IBM,
etc. provide translation services for many languages to facilitate
communication between people without using a human translator. However, the
quality of NMT is massively dependent on quantity, quality, and relevance of
the training dataset [17]. Such companies have achieved promising results for
bilingual high-resource languages, but they are inadequate for low-resource
languages like Amharic.

Figure 1: Examples of Google Amharic to English translation

Figure 1 shows Google translation of Amharic words into English. Google
translated the Amharic input as ”stomach empty” which is a wrong translation.
The correct literal meaning will be ”he become Disappointed”. This shows that
the applications used by companies that provide translation system like Google
require improvements. One of the cause for poor performance of MT systems for
languages like Amharic is availability of limited resource in digital space
[17]. In this research work we present a newly curated English-Amharic
parallel dataset that can be used for MT research and help to solve the
performance issue of an Amharic MT system.

On the other hand, Amharic is one of morphologically rich language and
normalization of the Amharic homophone characters might have an impact on such
downstream NLP applications as MT and sentiment analysis [18]. This research
work is intended to study the effect of homophone normalization on Amharic-
English machine translation. Furthermore, expanding the translation dataset
and developing state-of-the-art bi-directional Amharic-English translation
models are our motivations to carryout this research work.

##  IV Related work

Many automatic translation works have been carried out for the major pairs of
European and Asian languages, taking advantage of large-scale parallel
corpora. However, very few research studies have been conducted on low-
resource languages like Amharic to English due to its scarcity of parallel
data. In this section, we have focused on exploring how machine translation is
conducted for the Amharic language. Among recent works, Biadgligne and Smaïli
[19] described the development of an English-Amharic Statistical Machine
Translation (SMT) and Neural Machine Translation (NMT) experiments, achieving
26.47 and 32.44 BLEU scores, respectively. They harvested and used 225,304
parallel sentences in different domains. To the best of our knowledge,
Biadgligne and Smaïli [19] work is the largest dataset used in Amharic machine
translation research work.

Gezmu et al. [20] trained neural machine translation and phrase-based
statistical machine translation models using 145,364 parallel English-Amharic
sentences, achieved a BLEU score of 20.2 and 26.6 respectively. They concluded
that neural machine translation models outperform phrase-based statistical
machine translation models. Abate et al. [21] focused on the development of
parallel corpora for English and five Ethiopian languages, including Amharic.
They used 40,726 parallel sentences for Amharic and bi-directional Statistical
Machine Translation (SMT) technique. Finally, they achieved a BLEU score 22.68
in Amharic-English translation.

Teshome et al. [22] considered the application of phone-based statistical
machine translation for English-Amharic translation. They used 18,432 parallel
sentences and achieved a BLEU score of 37.5, a gain of 2.21 BLEU point from
their own previous baseline phrase-based MT experiments. Further more, Tracey
and Strassel [23] developed a low-resource language dataset for emergent
incidents (LORELEI). Their data includes multiple languages and was published
in the Linguistic Data Consortium (LDC) where Amharic is among the languages
with 60,884 Amharic-English parallel sentences. However, this dataset is not
freely available for further experiments.

The remain and addressed related works are summarized in Table I, with dataset
and methods used. As we can see in this related work table, only a few studies
have been conducted for the translation from Amharic into English or vice
versa and most of them were conducted using traditional approaches with a
small number of parallel sentences. This is due to unavailability of enough
Amharic linguistic resources for deep learning experiments.

TABLE I: Amharic-English and English-Amharic MT studies in terms of dataset size, method(s) used, and BLEU score achieved Authors | Trans. direction | # Dataset used | Method(s) | BLEU score
---|---|---|---|---
Biadgligne and Smaïli [19] | En→Am | 225,304 | Statistical machine translation | 26.47
|  |  | Neural machine translation | 32.44
Gezmu et al. [20] | Am→En | 45,364 | Phrase-based statistical machine translation | 20.2
|  |  | Neural Machine Translation | 26.6
Abate et al. [21] | Am→En | 40,726 | Statistical machine translation | 22.68
| En→Am |  | Statistical machine translation | 13.31
Teshome et al. [22] | En→Am | 18,432 | Phoneme-based statistical machine translation | 37.5
Teshome and Besacier [24] | En→Am | 18,432 | Phrase-based statistical machine translation | 35.32
Ashengo et al. [25] | En→Am | 8,603 | Combination of context-based MT (CBMT) with RNN | 11.34
Tadesse and Mekuria. [26] | En→Am | 37,970 | Statistical machine translation | 18.74
Hadgu et al. [27] | Am→En | 977 | Google translate, Yandex translate | 23.2, 4.8
| En→Am | 1915 | Google translate, Yandex translate | 9.6, 1.3

##  V Building a Parallel Dataset

In this section, we have identified available data sources that we have used
for the research work. As machine translation requires parallel documents as
an input, Table II shows the potential Amharic-English bi-lingual resources.
It can be seen in the table that the largest parallel corpus for Amharic-
English language pairs was collected by Biadgligne and Smaïli [28]. In
addition to the available datasets in Table II, we have contributed to the MT
research field by creating a new parallel corpus with 33,955 sentence pairs
extracted text from such news platforms as Ethiopian Press
Agency111https://www.press.et/, Fana Broadcasting
Corporate222https://www.fanabc.com/, and Walta Information
Center333https://waltainfo.com/. As the data we used is from different
sources, it includes various domains such as religious (Bible and Quran),
politics, economics, sports, news, among others.

As one can see in Table II, the total number of parallel sentences is about
1.1M, while the unique parallel sentences are 888,837. This is due to
duplication in the sources we used. This unique parallel sentence is the
largest to date then.

##  VI Data pre-processing

Before performing experiments, the development of every NLP task begins with a
text pre-processing step [29]. As our data were collected from different
sources, we noticed a lot of text irregularities. Some data was too noisy so
we eliminated it from our corpus. Then we performed a series of pre-processing
steps to canonize all tokens in Amharic and English sentences. In these steps,
the following tasks were performed: data cleaning (removing emojis, URL),
abbreviation expansion, Latin character lowercase, duplicated sentence
removal, and Amharic homophone character normalization. For abbreviation
expansion, we created a list of known abbreviations for both Amharic and
English, then expanded them to their full writing form. Most of English
abbreviations were collected from GitHub
repositories444https://github.com/JRC1995/Machine-Translation-Transformers.

TABLE II: Available Amharic-English parallel data sources Data source | # Sentence pairs | Accessible
---|---|---
Am-En ELRA-W0074555http://catalog.elra.info/en-us/repository/browse/ELRA-W0074/ | 13,347 | yes
Biadgligne and Smaïli[19] | 225,304 | yes
Horn MT666https://github.com/asmelashteka/HornMT | 2,030 | yes
Am-En MT corpus777https://github.com/adtsegaye/Amharic-English-Machine-Translation-Corpus | 53,312 | yes
Gezmu et al.[20] | 145,364 | yes
Abate et al.[21] | 40,726 | yes
Open Parallel Corpus (OPUS) [30] | 562,141 | yes
LORELEI-Amharic [23] | 60,884 | no
Admasethiopia888https://github.com/admasethiopia/parallel-text | 153 | yes
MT Evaluation Dataset999https://doi.org/10.5281/zenodo.3734260 | 2,914 | yes
Newly curated (our data) | 33,955 | yes
Total | 1,140,130 | yes
Unique sentence pairs | 888,837 | yes

Homophone character normalization: In Amharic writing, there are different
characters with the same sound which are called homophones. Homophones with
different symbols in Amharic text might have different writing standards and
different meanings. However, they are also considered as redundant alphabets
by most of the users, specially by the online and social media communities.

The current trend in Amharic NLP research is to normalize the homophone
characters into a single representation [16, 21, 20, 19]. Belay et al. [18]
have studied the impact of normalization for some downstreamm NLP
applications. They showed that homophone normalization improves the
performance of some downstream applications such as information retrieval (IR)
but not recommended for sentiment analysis. However, there is no study to show
that normalization helps to build efficient machine translation models or not.
We developed our translation models in the regular (unnormalized) and the
normalized forms (representing different homophones as a single character) to
analyze the impact of normalization. For the normalized model, we used
frequency-based homophone normalizer [18] replacing the set of characters with
similar function with a single most frequently used character.

Figure 2: Amharic homophone characters with their 7 derivatives using their
International Phonetic Alphabet (IPA) notation. The normalized homophones is
the first one in boldface in cell of each row except the of 4 column of the 1
and 2 row where the 1 character was used as the normalized one.

##  VII The proposed neural machine translation models

In this section, we describe the transformer-based pre-trained models that
could be used to train Neural Machine Translation systems. Unlike other neural
networks such as RNNs, the Transformer does not necessarily process the input
data in sequential order. Instead, the self-attention mechanism identifies the
context which gives meaning to each position in the input sequence, allowing
more parallelization, and reducing the training time. The architecture of the
Transformer network follows the so called encoder-decoder paradigm, trained in
an end-to-end fashion. The encoder is used to represent the source sentence as
a semantic vector, while the decoder takes in the semantic vector and makes
prediction to a target sentence. The Transformer model, which applies a self-
attention approach to measure the strength of a relationship between two words
in a sentence (contextual information), has contributed to improving
performance in MT and various natural language processing tasks.

###  VII-A Pre-trained language models (PLMs)

Pre-trained Language Models (PLMs) are large neural networks used in a wide
variety of NLP tasks [31]. Models are first pre-trained over a large text
corpus and then fine-tuned on a downstream task. PLMs are thought of as good
language encoders, supplying basic language understanding capabilities that
can be used with ease for many downstream tasks [31, 32].

Fan et al. [33], Facebook researchers, proposed a multilingual encoder-decoder
model trained for Many-to-Many multilingual translation with 418M parameters
(M2M100 418M). This multilingual machine translation model is based on the
Transformer sequence-to-sequence architecture. The model can directly
translate between the 9,900 directions of 100 languages. We used this pre-
trained model for our bi-directional translation experiments. This pre-trained
multi-lingual model is available at Hugging
Face101010https://huggingface.co/facebook/m2m100_418M

##  VIII Experimental setup and results

###  VIII-A Experimental setup

To demonstrate the effectiveness of our approach, we built our baseline
Transformer models and fine-tuned the available pre-trained model. We used
Google colab pro+ to train our bi-directional Amharic to English translation
models. During model training, the parallel sentences for Amharic and English
were divided into 80% for training, 10% validation, and 10% for testing.
Automatic evaluation was made using Bilingual Evaluation Understudy (BLEU)
metric [34]. BLEU score is defined in the range between 0 and 1, where 1 is a
perfect match with the reference and 0 is for no words matched.

Transformer baseline: We trained Transformer sequence-to-sequence models from
scratch for bi-directional Amharic-English NMT using OpenNMT with TensorFlow
deep learning framework [35]. We tokenized the text using Byte Pair Encoding
(BPE) [36] subword tokenization, which is a simple form of data compression
algorithm in which the most common pair of consecutive bytes of data is
replaced with a byte that does not occur in that data. The parameters used to
train the model are 512 hidden units, 6 layers, a learning rate of 0.0001, and
a maximum step of 50K, a batch size of 32, and the Adam optimizer.

Pre-trained model: We used the multilingual Facebook M2M-100 pre-trained model
with 418M parameters [33] to fine-tune into bi-directional Amharic-English
NMT. For fine-tuning, we used max source & target length of 128 per device
train and validation, a batch size of 4, and 3 epochs.

###  VIII-B Results and discussion

In our experiments, we built Transformer models and fine-tuned M2M100_418M,
the multi-lingual pre-trained language model for bi-directional Amharic-
English translation. All models were built on a regular (unnormalized) and
normalized datasets to study the effect of Amharic homophone normalization in
Amharic-English translation. Table III shows the experimental results of
Transformer and PLMs of M2M100 in both directions.

TABLE III:  Experimental results Models | Regular | Normalized
---|---|---
Transformer (Amharic→English) | 14.78 | 16.26
Transformer (English→Amharic) | 10.79 | 13.06
M2M100 48M (Amharic→English) | 34.12 | 37.79
M2M100 48M (English→Amharic) | 29.65 | 32.74

As it can be observed from the results in Table III, the multilingual
translation model M2M-100 outperforms the Transformer-based models in both
directions. When we compare our results with the attempts mentioned in Section
IV, our research shows an improvement in parallel corpus size and BLEU scores
using multilingual translation model. We can evaluate our models in the
translation directions as follows.

Amharic-English Translation: For both Amharic-English and English-Amharic
translations, the Transformer models did not work well, performed even less
than some traditional methods. The pre-trained transformer-based model
performed better with 34.12 and 37.79 BLEU score for Amharic-English
translation, on the regular and normalized data, respectively. This clearly
shows that pre-trained based translation outperforms the baseline system.

English-Amharic Translation: Among the works conducted for English-Amharic
translation, the work by Teshome et al. [22] reached a BLEU score of 37.53
using phoneme-based statistical MT, while our pre-trained model showed a BLEU
score of 29.65 and 32.74 on regular and normalized data, respectively. This
phoneme-based work [22] is built by converting the Amharic syllables to
phoneme-based characters; however 1) the conversion is rule-based (unable to
handle all exceptions), 2) Amharic homophones do not have constant Latin
representations, and 3) we used new train and test sets. So, the results are
not comparable. Another insight from this work is pre-trained model still
needs improvement to translate from technologically favored languages to
languages like Amharic which is morphologically rich.

Figure 3: Reporting of our work result with other previous works

Our results also clearly show the effect of homophone character normalization
in the performance of bi-directional Amharic-English translation. For both
(Transformer and pre-trained) models, homophone character normalization
increased the performance of NMT system. Accordingly, comparing our results
with the previous attempts as shown in Figure 3, we can confirm that our
models demonstrated a 0.29% BLEU score increase. This BLEU score improvement
was made using our new training and testing sets. This is because, first, all
the mentioned MT data is not available; second, the available data is not
split into constant training and testing categories for future evaluation
purposes. For the next time, our dataset will be available in train, test, and
validation files.

##  IX Conclusion and future work

In this paper, we presented our Amharic-English parallel corpus and bi-
directional machine translation experiments. We gathered more than 888K
parallel unique sentences and applied different pre-processing techniques.
This is the first large-scale parallel data (more than 5 times bigger than the
data in the previously conducted MT works) which can be used as a good
benchmark for future machine translation research. The main gaps from the
previous work were that they used small data and were not ready for future
comparison (benchmark) by splitting the data into constant train, test, and
validation sets. Our work will solve this issue and be used as a benchmark.
According to our result, we can conclude that the pre-trained models
outperformed the baseline Transformer-based model. The morphological richness
of the Amharic language and the size of the parallel dataset has a great
impact on Amharic-English MT experiments. For the future, we will expand this
work for more languages by including other Ethiopian low-resourced languages
and also use data augmentation techniques. Additionally, we plan to explore
the applicability of other pre-trained language models for Amharic-English
translations. Our dataset of parallel Amharic-English sentence pairs, the
models, and pre-processing scripts will be released in the GitHub
repository111111https://github.com/atnafuatx/EthioNMT-datasets

## References

  * [1] O. Bojar, R. Chatterjee, C. Federmann, Y. Graham, B. Haddow, S. Huang, M. Huck, P. Koehn, Q. Liu, V. Logacheva _et al._ , “Findings of the 2017 conference on machine translation (wmt17),” in _Proceedings of the Second Conference on Machine Translation_. Copenhagen, Denmark: Association for Computational Linguistics, 2017, pp. 169–214.
  * [2] M. L. Forcada, M. Ginestí-Rosell, J. Nordfalk, J. O’Regan, S. Ortiz-Rojas, J. A. Pérez-Ortiz, F. Sánchez-Martínez, G. Ramírez-Sánchez, and F. M. Tyers, “Apertium: a free/open-source platform for rule-based machine translation,” _Machine translation_ , vol. 25, no. 2, pp. 127–144, 2011.
  * [3] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens _et al._ , “Moses: Open source toolkit for statistical machine translation,” in _Proceedings of the 45th annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions_. Association for Computational Linguistics, 2007, pp. 177–180.
  * [4] K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation: Encoder-decoder approaches,” _arXiv preprint arXiv:1409.1259_ , 2014.
  * [5] N. Kalchbrenner and P. Blunsom, “Recurrent continuous translation models,” in _Proceedings of the 2013 conference on empirical methods in natural language processing_. Seattle, Washington, USA: Association for Computational Linguistics, 2013, pp. 1700–1709.
  * [6] L. H. Baniata, I. Ampomah, S. Park _et al._ , “A transformer-based neural machine translation model for Arabic dialects that utilizes subword units,” _Sensors_ , vol. 21, no. 19, pp. 1–28, 2021.
  * [7] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” _arXiv preprint arXiv:1409.0473_ , 2014\.
  * [8] J. Gehring, M. Auli, D. Grangier, and Y. N. Dauphin, “A convolutional encoder model for neural machine translation,” _arXiv preprint arXiv:1611.02344_ , 2016.
  * [9] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, pp. 5998–6008, 2017.
  * [10] A. Raganato, J. Tiedemann _et al._ , “An analysis of encoder representations in transformer-based machine translation,” in _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_. The Association for Computational Linguistics, 2018, pp. 287–297. [Online]. Available: http://hdl.handle.net/10138/263704
  * [11] M. G. Yigezu, M. M. Woldeyohannis, and A. L. Tonja, “Multilingual neural machine translation for low resourced languages: Ometo-english,” in _2021 International Conference on Information and Communication Technology for Development for Africa (ICT4DA)_. IEEE, 2021, pp. 89–94.
  * [12] A. L. Tonja, M. M. Woldeyohannis, and M. G. Yigezu, “A parallel corpora for bi-directional neural machine translation for low resourced ethiopian languages,” in _2021 International Conference on Information and Communication Technology for Development for Africa (ICT4DA)_ , 2021, pp. 71–76.
  * [13] J. Zhang, H. Luan, M. Sun, F. Zhai, J. Xu, M. Zhang, and Y. Liu, “Improving the transformer translation model with document-level context,” _arXiv preprint arXiv:1810.03581_ , 2018.
  * [14] S. Yang, Y. Wang, and X. Chu, “A survey of deep learning techniques for neural machine translation,” _arXiv preprint arXiv:2002.07526_ , 2020.
  * [15] D. M. Eberhard, G. F. Simons, and C. D. Fennig, “Ethnologue: Languages of the world (2022),” 2022. [Online]. Available: URL:https://www.ethnologue.com/
  * [16] M. M. Woldeyohannis and M. Meshesha, “Experimenting statistical machine translation for Ethiopic Semitic languages: The case of Amharic-Tigrigna,” in _International Conference on Information and Communication Technology for Development for Africa (ICT4DA 2017)_ , vol. 244\. Springer, Cham, 2017, pp. 140–149.
  * [17] B. Ahmadnia and B. J. Dorr, “Augmenting neural machine translation through round-trip training approach,” _Open Computer Science_ , vol. 9, no. 1, pp. 268–278, 2019.
  * [18] T. D. Belay, A. A. Ayele, G. Gelaye, S. M. Yimam, and C. Biemann, “Impacts of homophone normalization on semantic models for amharic,” in _2021 International Conference on Information and Communication Technology for Development for Africa (ICT4DA 2021)_. Bahir Dar, Ethiopia: IEEE, 2021, pp. 101–106.
  * [19] Y. Biadgligne and K. Smaïli, “Parallel corpora preparation for English-Amharic machine translation,” in _International Work-Conference on Artificial Neural Networks_. Springer, Cham, 2021, pp. 443–455.
  * [20] A. M. Gezmu, A. Nürnberger, and T. B. Bati, “Extended parallel corpus for Amharic-English machine translation,” _arXiv preprint arXiv:2104.03543_ , 2021.
  * [21] S. T. Abate, M. Melese, M. Y. Tachbelie, M. Meshesha, S. Atinafu, W. Mulugeta, Y. Assabie, H. Abera, B. Ephrem, T. Abebe, W. Tsegaye, A. Lemma, T. Andargie, and S. Shifaw, “Parallel corpora for bi-lingual English-Ethiopian languages statistical machine translation,” in _Proceedings of the 27th International Conference on Computational Linguistics_. Santa Fe, New Mexico, USA: Association for Computational Linguistics, aug 2018, pp. 3102–3111. [Online]. Available: https://aclanthology.org/C18-1262
  * [22] M. G. Teshome, L. Besacier, G. Taye, and D. Teferi, “Phoneme-based English-Amharic statistical machine translation,” in _AFRICON 2015_. Addis Ababa, Ethiopia: IEEE, 2015, pp. 1–5.
  * [23] J. Tracey and S. Strassel, “Basic language resources for 31 languages (plus English): The LORELEI representative and incident language packs,” in _Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL)_. Marseille, France: European Language Resources association, 2020, pp. 277–284. [Online]. Available: https://aclanthology.org/2020.sltu-1.39
  * [24] M. G. Teshome and L. Besacier, “Preliminary experiments on English-Amharic statistical machine translation,” in _Spoken Language Technologies for Under-Resourced Languages_ , Cape Town, South Africa, 2012, pp. 36–41. [Online]. Available: https://www.isca-speech.org/archive/
  * [25] Y. A. Ashengo, R. T. Aga, and S. L. Abebe, “Context based machine translation with recurrent neural network for English–Amharic translation,” _Machine Translation_ , vol. 35, no. 1, pp. 19–36, 2021.
  * [26] T. Ambaye and M. Yared, “English to Amharic machine translation using statistical machine translation,” _Master’s thesis_ , 2000.
  * [27] A. T. Hadgu, A. Beaudoin, and A. Aregawi, “Evaluating amharic machine translation,” _arXiv preprint arXiv:2003.14386_ , 2020.
  * [28] y. Biadgligne and K. Smaïli, “Offline Corpus Augmentation for English-Amharic Machine Translation,” in _2022 The 5th International Conference on Information and Computer Technologies_ , New York, United States, Mar. 2022. [Online]. Available: https://hal.archives-ouvertes.fr/hal-03547539
  * [29] A. A. Kumar and S. Chandrasekhar, “Text data pre-processing and dimensionality reduction techniques for document clustering,” _International Journal of Engineering Research & Technology (IJERT)_, vol. 1, no. 5, pp. 1–6, 2012\.
  * [30] P. Lison and J. Tiedemann, “Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles,” pp. 923–929, 2016.
  * [31] Y. Elazar, N. Kassner, S. Ravfogel, A. Ravichander, E. Hovy, H. Schütze, and Y. Goldberg, “Measuring and improving consistency in pretrained language models,” _Transactions of the Association for Computational Linguistics_ , vol. 9, pp. 1012–1031, 2021.
  * [32] R. Weng, H. Yu, S. Huang, S. Cheng, and W. Luo, “Acquiring knowledge from pre-trained model to neural machine translation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 05, 2020, pp. 9266–9273.
  * [33] A. Fan, S. Bhosale, H. Schwenk, Z. Ma, A. El-Kishky, S. Goyal, M. Baines, O. Celebi, G. Wenzek, V. Chaudhary, N. Goyal, T. Birch, V. Liptchinsky, S. Edunov, E. Grave, M. Auli, and A. Joulin, “Beyond english-centric multilingual machine translation,” _arXiv preprint_ , 2020.
  * [34] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_. Philadelphia: Association for Computational Linguistics, 2002, pp. 311–318.
  * [35] G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush, “Opennmt: Open-source toolkit for neural machine translation,” _arXiv preprint arXiv:1701.02810_ , 2017.
  * [36] P. Gage, “A new algorithm for data compression,” _C Users Journal_ , vol. 12, no. 2, pp. 23–38, 1994.


# Seeing BDD100K in dark: Single-Stage Night-time Object Detection via
Continual Fourier Contrastive Learning

Ujjal Kr Dutta
<EMAIL_ADDRESS>
###### Abstract

Despite tremendous improvements in state-of-the-art object detectors,
addressing object detection in the night-time has been studied only sparsely,
that too, via non-uniform evaluation protocols among the limited available
papers. In addition to the lack of methods to address this problem, there was
also a lack of an adequately large benchmark dataset to study night-time
object detection. Recently, the large scale BDD100K was introduced, which, in
our opinion, should be chosen as the benchmark, to kickstart research in this
area. Now, coming to the methods, existing approaches (limited in number), are
mainly either generative image translation based, or image enhancement/
illumination based, neither of which is natural, conforming to how humans see
objects in the night time (by focusing on object contours). In this paper, we
bridge these 3 gaps: 1. Lack of an uniform evaluation protocol (using a
single-stage detector, due to its efficacy, and efficiency), 2. Choice of
dataset for benchmarking night-time object detection, and 3. A novel method to
address the limitations of current alternatives. Our method leverages a
Contrastive Learning based feature extractor, borrowing information from the
frequency domain via Fourier transformation, and trained in a continual
learning based fashion. The learned features when used for object detection
(after fine-tuning the classification and regression layers), help achieve a
new state-of-the-art empirical performance, comfortably outperforming an
extensive number of competitors.

##  1 Introduction

Figure 1: Qualitative comparison of our method against state-of-the-art image
translation (ForkGAN, CoMoGAN) and enhancement methods (KinD, Enlighten GAN,
Zero DCE). The figure is best viewed when zoomed in color. Red arrows indicate
incorrect prediction (false negative/positive), while a green arrow indicates
correct predictions in difficult cases, where other methods failed.

Object Detection:  Object Detection has witnessed tremendous developments in
terms of performance, and efficiency, over the past few years. Existing object
detection techniques can mainly be categorized as either two-stage [5, 13, 15,
31], or single-stage [23, 25, 30, 36]. While the former category with its
region proposal step tends to be more accurate, the latter is usually more
efficient, owing to the employment of predefined anchors. In time critical
real-world applications (eg, autonomous driving), the latter is naturally the
preferred choice, and thus, has gradually seen substantial improvements in
accuracy, with optimized architectures.

Multi-scale feature representation plays another important role in deciding
the accuracy of an object detector. To address this, Feature Pyramid Network
(FPN) [22] demonstrated that a top-down pathway could be leveraged to combine
multi-scale features. Recently, Neural Architecture Search (NAS) based FPN
(NAS-FPN) [12] has also been explored, for automatic network topology
generation. However, such methods not only have interpretatbility issues, but
also require large number of hours of training.

The EfficientDet [41] has been recently introduced to perform a more
principled multi-scale feature fusion, by utilizing compound scaling of
network depth, width and image resolution. Another interesting approach that
extensively studied FPN for single-stage detectors is the recently proposed
State-Of-The-Art (SOTA) You Only Look One-level Feature (YOLOF) method [8].
Surprisingly, this simple method showed that by using just one C5 feature of
an underlying ResNet backbone [16] (even without requiring feature fusion!),
it could get comparable performance against FPN. To compensate the lack of
multi-level features, it simply adds a Dilated Encoder for using the single
level feature obtained by the backbone.

Challenges in Object Detection at night-time:  Despite all the significant
developments in the object detection literature, the problem of addressing
object detection in night-time images has been studied rather sparsely. In
addition, there also does not exist an uniform evaluation protocol (among the
very few papers available) that can be used to foster research in this
direction. Arguably, one might get a large number of annotated night-time
images and train an object detector. However, this comes with significant
manual effort. Thus, one would ideally want an existing SOTA object detector
already trained on a large annotated dataset, say, from daytime, to perform
well on night-time inference images as well. However, as the images from
daytime and night-time vary largely with respect to their input domain
properties, a detector trained on source domain daytime labeled data alone
might not perform well on target domain night-time images. To bridge this
domain discrepancy, one may leverage additional unlabeled nighttime images
along with labeled daytime images, leading to an Unsupervised Domain
Adaptation (DA) setting [28, 33].

A number of DA approaches have been proposed to address similar tasks, like
semantic segmentation [50, 49, 3], detection in rain/haze [38], optical flow
[48], etc. While a few approaches tried DA for object detection in art images
[32], and other tasks [17], some others have addressed night-time for semantic
segmentation [47, 34], depth estimation [43], visibility enhancement [37], and
via thermal images [44]. However, the concepts and assumptions that hold true
for the above settings, do not translate well to detecting objects in night
images. For instance, the assumption of the presence of a prior in DA for
object detection in haze/fog [38], does not hold true for DA from day to
night. This makes object detection in night-time images even more challenging.

Figure 2: An illustration of the proposed approach. The input images belong to
the BDD100K dataset. The figure is best viewed in color.

Due to these facts, a few researchers have resorted to Generative Adversarial
Networks (GAN) based image translation techniques [29, 53, 1, 2] to transfer
the style among day and night images, and train an existing object detector on
the translated images. However, such an image translation often produces
severe artefacts in the translated images, and even cause disappearance of
objects from images, thus making generative image translation not the best
alternative. Although not directly related to domain adaptation, it might also
be tempting to leverage SOTA image enhancement techniques [19, 21, 52] that
could perhaps improve the illumination of night time images, with the hope
that an object detector trained on day images could be used on the illuminated
images. However, this also does not work well in practice (see Figure 1).

Lastly, while techniques play an important part in machine learning, data is
yet another aspect (and perhaps the more important) that is essential to
solving any real-world problem. To this end, till recently, there was also a
lack of an adequately large enough benchmark dataset that could be used to
address the object detection at night-time problem, and that captures the
essential corner cases of various real world scenarios of both day and night
time. Fortunately, the large scale BDD100K dataset [51] was recently proposed,
that provides an adequately large number of 2-D bounding box annotated images
(from both day/night time), for the evaluation of object detectors in the
night-time. [2] did make use of this dataset in the past, but only for
detecting certain large object categories (such as cars). We on the other hand
aim at detecting all the categories.

Figure 3: Comparison of GAN and Fourier Transformation based image translation
from Day to Night.

Our contributions : In this paper, we seek to address the core challenges
involved in the problem of nighttime object detection: 1. Proposal of an
uniform experimental protocol, via an uniform tool and an uniform dataset, to
conduct further research in this problem. 2. Proposal of a novel learning
based technique, to address the fundamental limitations of the (limited number
of) existing techniques that attempt at solving this problem. 3. An extensive
comparison of our proposed method against multiple approaches that could
provide suitable alternatives to address night-time object detection.

Firstly, we believe that for fostering practical, real-world applicable
research in the lesser studied and difficult night-time object detection
problem, it is essential to leverage an efficient and accurate single-stage
detector as the bedrock. Thus, we conduct all our experiments on the recently
proposed SOTA YOLOF method. While YOLOF can also include other backbones like
CSPDarkNet-53 as used in concurrent object detectors (Scaled YOLOv4 [45],
YOLOR [46]), we simply fix the ResNet50 backbone, as our focus is more on DA
for night-time object detection, and not the architecture. Following are the
main reasons for choosing YOLOF (over alternatives such as RetinaNet, YOLOv4,
and transformer based DETR [6]): It outperforms others in similar settings
[8], can be trained in much lesser epochs, requires lesser GFLOPs, and also
has higher FPS. Having said that, we are only interested in a single-stage
object detector that first obtains a latent encoding of an image, to be
further used for classification and bounding box regression. YOLOF has a
feature extractor (backbone and Dilated Encoder) and a Decoder (classification
/ regression layers) (see Figure 2).

At the same time, while there are multiple tools available to implement object
detectors, we make use of Detectron2
111https://github.com/facebookresearch/detectron2, that i) supports a number
of computer vision research projects, ii) provides a much cleaner way to
incorporate new capabilities, and iii) facilitates much faster training. As
for the dataset, we believe that the focus should be BDD100K. This is because,
while the Average Precision (AP) performance metric of SOTA object detectors
on standard benchmarks like COCO [24] is high (37-55 using various backbones),
the same is very poor in BDD100K (around 17-24). BDD100K captures real-world
day-night corner cases, is large enough, and thus provides significant room
for improvement.

Secondly, our key contribution in this paper is a novel, learning based
approach to address the domain discrepancy among day and night images. As
illustrated in Figure 2, our approach leverages Contrastive Learning [14] to
bridge the domain gap, by learning domain invariant feature representations of
images, which could be further used by the object detector to update its
classification and regression layers. The main advantage of Contrastive
Learning is that it enjoys nice convergence properties (see Figure 7), which
is absent in domain adversarial feature learning counterparts [11].

The images used for Contrastive Learning are obtained using frequency domain
information via Fourier transformation. Considering frequency domain
information has shown [50] to preserve the semantic content (eg, object
contours) of the translated images (vs generative image translation approaches
which result in dark patches) (see Figure 3). Because the (Fourier) phase
information contains high-level semantics (such as object contours), this also
conforms to how humans look at objects in nighttime. Also, Fourier based image
translation does not require any training! It should be noted that our design
choice of Contrastive Learning is justified by the fact that self-supervision
helps in domain adaptation [40].

We also noticed that while an auxiliary task based on Contrastive Learning
might be beneficial, care should be taken to ensure that there is no
catastrophic forgetting of the information learned earlier. As Contrastive
Learning training progresses, the features become more and more suitable to
bridge the domain gap. However, they also deviate from their initial
representations which were good for object detection. Thus, we add a Elastic
Weight Consolidation (EWC) [20] based penalty to the Contrastive loss,
inspired by continual learning [9], that seeks to learn new information
(plasticity), while retaining earlier ones (stability).

Lastly, for the first time, we consolidate a plethora of approaches that could
provide suitable alternatives to address the night-time object detection
problem (eg, image translation, image enhancement), and compare our method
against them. We empirically and qualitatively demonstrate our method’s
comfortable superiority against the SOTA baselines (by virtue of its domain
invariant features).

Figure 4: Qualitative Results of Fourier Transform based Night image synthesis
from Day images.

Figure 5: Fourier Transform based Night Image Synthesis with different beta
($\beta$) values.

##  2 Related Work and Background

Datasets:  The KAIST dataset [18] consists of color-thermal pairs, but has
annotations only for certain categories (person, people, cyclist). Nightowls
dataset [27] has annotated images at nighttime, but only for pedestrians. For
a day-night coupling, it also needs a separate daytime pedestrian dataset like
Caltech Pedestrian [10]. In any case, a single category does not make up a
decent evaluation protocol. The Waymo open [39] and nuScenes [4] datasets
provide 3D labels. The Dark Zurich dataset [34] only provides GPS labels for
day-twilight-night correspondence. The ACDC dataset [35] provides semantic
segmentation labels. The Alderdey dataset [26] has day-night correspondences,
but no annotations.

Generative Image Translation:  ForkGAN [53] aims at learning a general-purpose
task agnostic image translation network. It has demonstrated night-time object
detection performance by leveraging its translation network. CoMoGAN [29]
recently employed physics-inspired models for image data organization on a
functional manifold, the points on which can represent different times of day,
as indicated by an angle. However, we found that despite its good performance
in other translation tasks, the generated images had artefacts present in them
when applied in our dataset. ToDayGAN [1] was proposed as a translation model
aimed at addressing the localization task based on retrieval. [2] proposed a
translation method to only address car detection.

Image Enhancement:  EnlightenGAN [19] is a GAN based apporach for addressing
the low-light image enhancement problem, that can be trained without
low/normal-light image pairs. The Zero-DCE [21] method trains a lightweight
deep network to model light enhancement as a task of image-specific curve
estimation. KinD [52] is an image enhancement method based on Retinex theory.
It decomposes an image into an illumination component (for light adjustment),
and a reflectance component (for degradation removal).

##  3 Proposed Method

Problem Description:  Let us assume that we are given the two sets of images:
i) $\\{x_{i}^{d},\\{o^{di}_{m}\\}_{m=1}^{M(x_{i}^{d})}\\}_{i=1}^{N_{d}}$, and
ii) $\\{x_{i}^{n}\\}_{i=1}^{N_{n}}$, respectively belonging to the source
domain $\mathcal{D}_{d}$ of daytime images and target domain $\mathcal{D}_{n}$
of nighttime images. Each image $x_{i}^{d}\in\mathcal{D}_{d}$ is associated
with a set of object instances $\\{o^{di}_{m}\\}_{m=1}^{M(x_{i}^{d})}$, where
$M(x_{i}^{d})$ denotes the number of objects annotated in the image
$x_{i}^{d}$ by using 2D bounding boxes, and a category label. Here,
$o^{di}_{m}$ denotes the $m^{th}$ object instance present in the image
$x_{i}^{d}$. The set $\\{x_{i}^{n}\\}_{i=1}^{N_{n}}$ is unannotated, and the
idea is to use it along with
$\\{x_{i}^{d},\\{o^{di}_{m}\\}_{m=1}^{M(x_{i}^{d})}\\}_{i=1}^{N_{d}}$, to
correctly predict $\\{o^{nt}_{m}\\}_{m=1}^{M(x_{t}^{n})}$ for any arbitrary
unseen (during training) test image $x_{t}^{n}\in\mathcal{D}_{n}$.

Stage 1 (Burn-in):  Also, assume that we have a general-purpose off-the-shelf
single-stage object detector consisting of two major components: i) A feature
extractor and ii) Classification and Regression layers. The function of the
feature extractor is to obtain latent encoding/ representation of a raw input
image, which can be further used by the regression layers to predict the
bounding box locations of the objects, and the classification layers to
predict the categories of the detected bounding boxes. In our work, we use the
YOLOF method as the object detector, which further segregates the feature
extractor into i) a backbone, and ii) the dilated encoder (Figure 2). The
classification and regression layers have been collectively called as the
Decoder. At first, we train our object detector (with parameters $\theta_{f}$
and $\theta_{d}$) end-to-end using the available annotated set of day images
from $\mathcal{D}_{d}$. Here, $\theta_{f}$ refers to the combined collection
of parameters of the backbone and the dilated encoder, and $\theta_{d}$ refers
to the parameters of the Decoder. We refer to this as the burn-in stage
(denoted as Stage 1 in Figure 2)

Stage 2 (Continual Fourier Contrastive Learning):  Now, given a raw image
$x_{i}$ (from either domain), let $f_{\theta_{f}}(x_{i})$ denote the encoding
obtained after the dilated encoder step (and in general, by the feature
extractor). We conjecture that the drop in performance of an object detector
trained on $\mathcal{D}_{d}$ and tested on $\mathcal{D}_{n}$ could be reduced,
if, for any arbitrary day image $x_{i}^{d}\in\mathcal{D}_{d}$ and its
corresponding night-time representation $x_{i}^{d\rightarrow
n}\in\mathcal{D}_{n}$, the backbone and dilated encoder are collectively able
to learn similar representations for $f_{\theta_{f}}(x_{i}^{d})$ and
$f_{\theta_{f}}(x_{i}^{d\rightarrow n})$. However, the challenge lies in the
fact that it is impossible to physically collect $x_{i}^{d\rightarrow n}$ for
all $x_{i}^{d}$, as in any real-world street/ place, objects keep moving.

2-a) Fourier Style Transfer:  Generative image translation techniques could be
leveraged to obtain $x_{i}^{d\rightarrow n}$, but that often leads to unwanted
artefacts (and even disappearance of objects!). Hence, we follow Fourier
Domain Adaptation (FDA) [50] and use frequency domain information via Fourier
transformation to obtain $x_{i}^{d\rightarrow n}$. Compared to generative
image translation, FDA does not require any training, and also produces better
semantics preserving images (Figure 3). While FDA has been used in the context
of semantic segmentation, we are the first ones to study FDA in the context of
object detection. Precisely, to obtain $x_{i}^{d\rightarrow n}$ using
$x_{i}^{d}\in\mathcal{D}_{d}$, we need another random image
$x_{i}^{n}\in\mathcal{D}_{n}$, to swap the style (provided by amplitude) of
$x_{i}^{d}$ with that of $x_{i}^{n}$ (see Figure 4). Let, $\mathcal{F}_{a}$
and $\mathcal{F}_{p}$ respectively denote the amplitude and phase components
of the Fourier transformation $\mathcal{F}$ of an image $x_{i}$. Then, we can
obtain $x_{i}^{d\rightarrow n}$ as [50]:

$x_{i}^{d\rightarrow
n}=\mathcal{F}^{-1}([M_{\beta}\circ\mathcal{F}_{a}(x_{i}^{n})+(1-M_{\beta})\circ\mathcal{F}_{a}(x_{i}^{d}),\mathcal{F}_{p}(x_{i}^{d})]).$
(1)

Here, $\mathcal{F}^{-1}$ denotes the inverse Fourier transformation for
mapping back the signals to the image space. $M_{\beta}$ is a mask with values
zero except at a square (of size $\beta$) located at the center (0,0) of the
amplitude signals. Here, $\beta\in(0,1)$.

2-b) Contrastive Representation Learning:  Now, to learn similar
representations for $f_{\theta_{f}}(x_{i}^{d})$ and
$f_{\theta_{f}}(x_{i}^{d\rightarrow n})$, we employ a contrastive learning
module (Stage 2 of Figure 2). For this, we maintain a separate siamese network
having two branches, query and key. The latter is updated as an Exponential
Moving Average (EMA) [42] of the former to ensure stable updates, and provide
guidance/ target to the query branch. Let
$\phi_{q}=(\theta_{f-q},\theta_{pro-q})$ denote parameters of the query
branch. Then, the parameters of the key branch
$\phi_{k}=(\theta_{f-k},\theta_{pro-k})$ is obtained as:

$\phi_{k}=\mu\phi_{k}+(1-\mu)\phi_{q},\mu\in[0,1].$ (2)

Also, let $\theta_{f-q}$, $\theta_{pro-q}$, $\theta_{f-k}$, $\theta_{pro-k}$
and $\theta_{pre}$ respectively denote the parameters of the query feature
extractor (backbone + dilated encoder), query projector, key feature
extractor, key projector and the predictor. $\theta_{f-q}$ and $\theta_{f-k}$
are initialized using the learned $\theta_{f}$ from Stage 1. $\theta_{pro-q}$,
$\theta_{pro-k}$ and $\theta_{pre}$ are initialized randomly. The key idea of
this contrastive learning module is to obtain a query vector $\mathbf{q}$
corresponding to $f_{\theta_{f}}(x_{i}^{d})$ (via
$f_{\theta_{f-q}}(x_{i}^{d})$), and a target vector $\mathbf{t}$ corresponding
to $f_{\theta_{f}}(x_{i}^{d\rightarrow n})$ (via
$f_{\theta_{f-k}}(x_{i}^{d\rightarrow n})$), and align their representations
(by maximizing inner product similarity) using a contrastive loss following
Self-Supervised Learning (SSL) [14]. This is done with the hope that in turn,
$f_{\theta_{f}}(x_{i}^{d})$ and $f_{\theta_{f}}(x_{i}^{d\rightarrow n})$ would
also be aligned (thus, bridging the domain gap).

As shown in Figure 2, we apply average pooling and flattening of
$f_{\theta_{f-q}}(x_{i}^{d})$ (and $f_{\theta_{f-k}}(x_{i}^{d\rightarrow
n})$), before passing them via a projector (consisting of fully connected
layers with batch norm and ReLU), to obtain vector representations in a low-
dimensional latent space. However, naively obtaining $\mathbf{q}$ and
$\mathbf{t}$ may lead to a model-collapse where we might get a trivial
solution with the same embedding for $\mathbf{q}$ and $\mathbf{t}$. Thus,
following SSL [14], we add a predictor after the query model, to obtain
$\mathbf{q}$ in a different embedding space of the same dimensionality as
$\mathbf{t}$, and enforce an asymmetry to avoid a model collapse. It should be
noted that as the key model is maintained as an EMA of the query model,
gradients are backpropagated only through the query model.

2-c) Continual Learning with Elastic Weight Consolidation (EWC):  However,
naively optimizing a (Fourier based) Contrastive Loss $\mathcal{L}_{FCL}$ as
above, although bridges the domain gap, leads to a catastrophic forgetting of
the information learned during the burn-in stage. Thus, we add a penalty term
to $\mathcal{L}_{FCL}$ that prohibits significant updates to the parameters
important for the Task 1 (object detection/ burn-in), while learning Task 2
(Fourier Contrastive Learning in Stage 2). This penalty is based on Elastic
Weight Consolidation (EWC) [20] enforced on the parameters of the query model.
Using this penalty helps in continually learning domain invariant features
while preserving the information learned apriori during burn-in.

Stage 3 (Decoder Update):  Finally, in Stage 3, we replace the parameters
$\theta_{f}$ of the object detector backbone and the dilated encoder, with
that of the parameters $\theta_{f-q}$ of the query model, and update
$\theta_{d}$ using labeled day images.

Objective:  Our overall objective can thus be expressed as:

$\begin{split}\min_{\theta_{f},\theta_{d},\theta_{f-q},\theta_{pro-q},\theta_{pre}}&(1-\lambda_{ewc})\mathcal{L}_{FCL}+\lambda_{ewc}\mathcal{L}_{EWC}+\\\
&\mathcal{L}_{cls}+\mathcal{L}_{boxreg}.\end{split}$ (3)

Here, $\lambda_{ewc}\in(0,1)$ is a hyperparameter,
$\mathcal{L}_{FCL}=2-2\frac{\mathbf{q}^{\top}\mathbf{t}}{\left\|\mathbf{q}\right\|_{2}\left\|\mathbf{t}\right\|_{2}}$
is the Fourier translation based Contrastive Loss, $\mathcal{L}_{cls}$ and
$\mathcal{L}_{boxreg}$ are the classification and bounding box regression
losses of the YOLOF method. Also, in our case, we define the EWC based penalty
as:

$\mathcal{L}_{EWC}=\sum_{p}F_{p}({(\theta_{f-q})}_{p}-{(\theta_{f,Burn-
in})}^{*}_{p})^{2}.$ (4)

Here, $F$ is the Fisher Information matrix, ${(\theta_{f-q})}_{p}$ indicates
the $p^{th}$ parameter of $\theta_{f-q}$, and ${(\theta_{f,Burn-in})}^{*}_{p}$
is its corresponding optimal value obtained during the burn-in stage.

Setting | AP | AP50 | AP75 | APS | APM | APL | person | rider | car | bus | truck | bike | motor | traffic light | traffic sign
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
YOLOF R50_C5_1x (1-Stage Object Detector)
Day2Day | 24.32 | 46.88 | 21.88 | 6.08 | 31.62 | 52.61 | 20.82 | 16.87 | 38.95 | 37.79 | 36.00 | 15.86 | 15.18 | 13.98 | 23.47
Day2Night | 17.59 | 35.66 | 15.32 | 4.50 | 20.44 | 36.15 | 16.03 | 8.09 | 28.51 | 27.21 | 26.51 | 13.85 | 10.05 | 6.13 | 21.89
Night2Night | 19.75 | 41.39 | 16.43 | 5.33 | 22.14 | 41.33 | 15.75 | 10.58 | 33.29 | 30.30 | 30.30 | 12.44 | 12.24 | 9.15 | 23.72
Night2Day | 17.45 | 35.61 | 14.99 | 3.44 | 22.73 | 39.97 | 15.35 | 12.19 | 34.02 | 25.19 | 24.36 | 10.66 | 8.24 | 8.90 | 18.10

Table 1: Performances of SOTA Object Detectors drop when a model trained in
Domain A (eg, Day) is used for inference on images from Domain B with a huge
domain mismatch (eg, Night). Day2Day: Train on Day, Test on Day; Day2Night:
Train on Day, Test on Night; Night2Day: Train on Night, Test on Day;
Night2Night: Train on Night, Test on Night. Results are obtained using the
recent large-scale BDD100K dataset. We are interested in performing better
than Day2Night.

Method | Nature | AP | AP50 | AP75 | APS | APM | APL | person | rider | car | bus | truck | bike | motor | traffic light | traffic sign
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Day2Night | Baseline | 17.59 | 35.66 | 15.32 | 4.50 | 20.44 | 36.15 | 16.03 | 8.09 | 28.51 | 27.21 | 26.51 | 13.85 | 10.05 | 6.13 | 21.89
KinD [52] | Enhancement | 12.91 | 26.87 | 10.66 | 3.35 | 15.34 | 26.32 | 12.24 | 6.48 | 21.48 | 23.29 | 18.46 | 11.26 | 4.12 | 3.29 | 15.57
ZeroDCE [21] | Enhancement | 12.12 | 26.17 | 9.89 | 3.52 | 15.40 | 23.85 | 11.19 | 6.11 | 19.14 | 19.61 | 18.10 | 9.87 | 2.58 | 5.04 | 17.40
EnlightenGAN [19] | Enhancement | 12.50 | 25.89 | 10.44 | 3.29 | 14.15 | 24.91 | 12.34 | 6.48 | 21.98 | 21.31 | 18.43 | 11.27 | 2.76 | 2.34 | 15.58
ToDayGAN [1] | Translation | 14.95 | 32.29 | 11.73 | 3.48 | 17.15 | 33.66 | 14.03 | 8.81 | 25.67 | 22.70 | 21.78 | 13.79 | 4.77 | 5.01 | 17.95
CoMoGAN-infer [29] | Translation | 11.81 | 25.42 | 9.49 | 2.14 | 12.98 | 27.25 | 13.34 | 5.74 | 20.37 | 17.14 | 16.20 | 10.10 | 5.31 | 2.26 | 15.82
CoMoGAN [29] | Translation | 15.82 | 32.59 | 14.05 | 3.26 | 17.43 | 34.34 | 15.33 | 10.22 | 28.65 | 23.84 | 23.00 | 12.09 | 7.19 | 4.13 | 17.94
ForkGAN [53] | Translation | 18.24 | 36.63 | 15.73 | 4.78 | 20.47 | 39.53 | 16.19 | 9.81 | 29.66 | 29.23 | 27.66 | 13.39 | 9.16 | 6.55 | 22.51
FDCL EWC 2w (Ours) |  | Feature
---
Learning
19.57 | 39.32 | 17.29 | 5.19 | 21.88 | 41.79 | 17.72 | 10.52 | 30.48 | 30.59 | 27.77 | 17.17 | 11.08 | 7.36 | 23.47

Table 2: Comparison of our method against various SOTA translation and
enhancement approaches. Metrics corresponding to the best method are shown in
bold.

##  4 Experiments

For our experiments, we use the BDD100K dataset with its original class
labels, and filter out the images that belong to day/ night-time. This results
in the following data splits (along with respective number of images within
bracket): i) Training Daytime (36728), ii) Training Nighttime (27971), iii)
Validation Daytime (5258), and iv) Validation Nighttime (3929). There are 9
categories (person, rider, car, bus, truck, bike, motor, traffic light,
traffic sign) that are present in the images and are suitable for training
(the train category is omitted due to very few instances).

Figure 6: Detection Results of various methods on several images of the BDD
dataset. The figure is best viewed when zoomed in a color monitor.

Figure 7: Convergence behaviour of the contrastive loss with EWC.

For the object detector, we make use of the YOLOF_R_50_C5_1x [8] model
222https://github.com/chensnathan/YOLOF, with the PyTorch based Detectron2
tool, and fix a batch size of 16, and base learning rate of 0.025 (and initial
warmup iterations of 1500) for all the experiments 333[53] also uses a similar
data split, but with MMDetection [7] based 2-stage Faster RCNN. Different from
them, we introduce an experimental protocol using the more practical single-
stage YOLOF, and report all hyperparameters for easy reproducibility. This set
of hyperparameters is feasible for single GPU training. We do not make use of
any learning rate decay. Rest of the hyperparameters are used as default from
the Detectron2 tool.

Domain mismatch causes performance drop! To validate the hypothesis that a
mismatch in train and test image domains causes performance drop, we conduct
the following experiment: We train the YOLOF model using (all labeled) images
of Training Daytime and evaluate it on images of: i) Validation Daytime
(Referred to as Day2Day in Table 1), and ii) Validation Nighttime (Day2Night
in Table 1). Similarly, we train the YOLOF model using (all labeled) images of
Training Nighttime and evaluate it on images of: i) Validation Daytime
(Referred to as Night2Day in Table 1), and ii) Validation Nighttime
(Night2Night in Table 1). We could clearly see that using commonly used AP
metrics, compared to Day2Day results, Day2Night results are poorer, and
compared to Night2Night results, Night2Day results are poorer. Because we used
all the image annotations for the above experiments, we now have an idea of
the upper and lower bounds. It should be noted that all the above results have
been reported by training the models for 20000 iterations in Detectron2.

Realistic Day to Night Object Detection:  For a realistic evaluation, we
assume the access to labeled Training Daytime images, and, the unannotated
pool of images from Training Nighttime, using which we aim to build a model
following an Unsupervised Domain Adaptation (UDA) setting. The model should
perform better than the lower bound Day2Night (shown in red color in Table 1).

Fourier Transformation based Image Translation:  In our method, to employ
Contrastive Learning for obtaining domain invariant features for images, we
make use of Fourier Transformation. This is because compared to generative
image translation methods, it does not require any network training, and is
able to produce better semantics preserving images. For instance, from Figure
3, it can be observed that images produced by generative approaches may lead
to unwanted artefacts (eg, dark patches) while translating from day to night.
This would rather hurt the performance of an object detector. On the contrary,
the phase component obtained by Fourier transformation preserves the high-
level semantics essential to identify an object (eg, the contours, upon which
humans rely as well, to see at night).

Given a day time training image, in order to obtain its nighttime equivalent
image, we simply sample a random nighttime training image, and replace the
amplitude of the daytime image with that of the night one (Figure 4). We tried
different values of $\beta$ in (1), and inspected the generated images
visually. It is observed that neither too low, nor too high value for it is
suitable. In the case of the former, the images resemble more of the original
untranslated day images, and in the case of the latter, certain distortion may
appear (as shown in Figure 5). We found $\beta=0.01$ to be optimal in our
case. During translated image generation for training, we randomly use
$\beta=0.01$ 90% of the time, and $\beta=0.05$ 10% of the time (to make some
of the images relatively darker).

###  4.1 Comparison with state-of-the-art

To demonstrate the effectiveness of our proposed method, we compare it against
the following SOTA methods: i) ToDayGAN [1], ii) ForkGAN [53], iii) CoMoGAN
[29], iv) KinD [52], v) EnlightenGAN [19] and vi) ZeroDCE [21]. Here, the
first 3 are generative image translation methods, while the remaining 3 are
image (illumination) enhancement methods. We also compare these methods with
the lower bound of this setting, i.e, Day2Night. ToDayGAN, ForkGAN and CoMoGAN
being image translation methods, are used to obtain day to night images via
image translation and the original bounding boxes from daytime are used to
train the model. ForkGAN additionally uses a 2-stage image translation (and
refinement) during inference as well. For the image enhancement methods, the
inference/ validation nighttime images are enhanced, and inferred upon by the
model (trained with labeled daytime images). For ours, we use the final
checkpoint of the model trained on labeled daytime images, as initialization
for our method, which constitutes the Stage-1/ Burn-in of our method, and
train upon it using Stage-2 and Stage-3. We call our method as Fourier Domain
based Contrastive Learning with EWC (FDCL EWC).

Table 2 reports the performances of all the compared methods. As expected, the
enhancement based methods do not perform competitive. This is because despite
the illumination enhancement, the domain shift is not reduced. Among
(generative) translation based methods, the manifold based CoMoGAN performs
better than ToDayGAN, but the more sophisticated ForkGAN outperforms them
both. Various reasons, like the presence of a strong adversarial
regularization, and the 2-stage translation and refinement during inference
contributed for the strong performance of ForkGAN. We also tried a variant of
CoMoGAN (CoMoGAN-infer) to simply translate inference images to daytime, and
use the day time trained model, without any training, and found it to be
inferior than CoMoGAN with training. In fact, due to artefacts like patches
generated, methods like ToDayGAN and CoMoGAN perform worse than the Day2Night
baseline. From Table 2, it is clear that our method comfortably outperforms
all the other approaches. In Figure 1, we showcase a few qualitative detection
results across all the methods compared, and demonstrate the superiority of
our approach. Also, in Figure 6 we compare more qualitative results for the
top three performing methods (CoMoGAN, ForkGAN and Ours).

To obtain the results for our method reported in Table 2, Stage-2 of
Contrastive learning was trained using batch size of 32, $\mu=0.99$ in (2),
$\lambda_{ewc}=0.9$ in (3), SGD optimizer with learning rate $5.5666945e-06$
(kept same as object detector, obtained using Detectron2 framework), weight
decay of $1e-6$, momentum of $0.9$. Also, following is the architectural
configuration in Figure 2: FC_pro1(in=512,out=512), FC_pro2(in=512,out=128),
FC_pre1(in=128,out=512), and FC_pre2(in=512,out=128). For data augmentation,
we randomly used the whole image 75% of the time, and performed random
cropping the remaining 25% (the same crop was applied to both day, and the day
to night translated image). Joint Stage-2/-3 training was performed for 14000
iterations.

###  4.2 Ablations/ Detailed Analysis of our method

Iterations | Method | AP | AP50 | AP75 | APS | APM | APL
---|---|---|---|---|---|---|---
2000 | FDCL EWC | 18.21 | 37.22 | 15.68 | 4.12 | 20.38 | 39.25
| FDCL EWC 2w | 18.73 | 37.41 | 16.80 | 4.52 | 21.05 | 39.21
8000 | FDCL EWC | 18.74 | 37.61 | 16.55 | 4.36 | 21.03 | 40.95
| FDCL EWC 2w | 19.35 | 38.67 | 17.64 | 5.03 | 21.42 | 41.26
10000 | FDCL EWC | 18.52 | 37.19 | 16.24 | 4.57 | 20.74 | 38.84
| FDCL EWC 2w | 19.34 | 38.92 | 17.13 | 4.96 | 21.75 | 41.58
14000 | FDCL EWC | 18.38 | 37.33 | 15.69 | 4.34 | 20.20 | 39.78
| FDCL EWC 2w | 19.57 | 39.32 | 17.29 | 5.19 | 21.88 | 41.79

Table 3: Effect of EWC 2 way updates after different iterations

Method | AP | AP50 | AP75 | APS | APM | APL
---|---|---|---|---|---|---
Cropped, 0.85 | 15.61 | 32.76 | 13.10 | 4.54 | 19.47 | 30.76
Full, 0.85 | 16.87 | 34.59 | 14.42 | 4.24 | 19.82 | 34.63
EWC | 17.43 | 35.44 | 15.11 | 4.19 | 20.29 | 36.00

Table 4: Ablations to study the effect of not using EWC, of cropping images
during Stage-2, and not performing Decoder update.

Method | AP | AP50 | AP75 | APS | APM | APL
---|---|---|---|---|---|---
Baseline | 17.59 | 35.66 | 15.32 | 4.50 | 20.44 | 36.15
Day longer | 17.87 | 37.22 | 15.92 | 4.38 | 19.80 | 36.72
FDCL EWC 2w | 19.57 | 39.32 | 17.29 | 5.19 | 21.88 | 41.79

Table 5: Comparison with longer training of the baseline model.

Method | AP | AP50 | AP75 | APS | APM | APL
---|---|---|---|---|---|---
ToDayGAN | 14.95 | 32.29 | 11.73 | 3.48 | 17.15 | 33.66
CoMoGAN | 15.82 | 32.59 | 14.05 | 3.26 | 17.43 | 34.34
Fourier FT | 15.89 | 33.19 | 13.70 | 4.19 | 18.99 | 33.15
FDCL EWC 2w (Ours) | 19.57 | 39.32 | 17.29 | 5.19 | 21.88 | 41.79

Table 6: Benefit of Contrastive Learning over direct fine-tuning with Fourier
Translated images. Second best results are underlined.

Effect of 2-way update signals in Stage-2/-3:  After the burn-in (Stage-1),
the Stage-2 and Stage-3 interplay can be maintained in 2 different ways,
resulting in two variants of our method: i) FDCL EWC, and ii) FDCL EWC 2 way
(FDCL EWC 2 w). Table 2 reports values for the latter. For Stage-2, using a
sampled mini-batch (size 32), we backpropagate and update the query model,
followed by EMA update of the key using (2). After that, for Stage-3, we can
use the updated backbone and dilated encoder of the query, to replace the
counterparts in the object detector, and update only the Decoder (using mini-
batch sampled for object detection as earlier, with labeled day images).
Stage-2/-3 when alternated as mentioned, refers to FDCL EWC.

Alternately, for Stage-3, we can use the updated backbone and dilated encoder
of the query, to replace the counterparts in the object detector, and update
all of backbone, dilated encoder and Decoder. Now, this updated backbone and
dilated encoder (denoted together as $\theta_{f}^{*}$) can be used to further
update the counterparts $\theta_{f-q}$ of the query in EMA manner as:
$\theta_{f-q}=\mu\theta_{f-q}+(1-\mu)\theta_{f}^{*}$, with $\mu=0.99$, before
updating the query again using Stage-2 of the next iteration. This
bidirectional EMA signal refers to our FDCL EWC 2 w variant, which obtains
slightly better results (over different iterations), as shown in Table 3.
However, FDCL EWC still manages to perform competitive, even outperforming the
baselines and other SOTA methods.

Effect of EWC, cropping, and decoder update:  Despite EWC based contrastive
learning, it is important to update the decoder in Stage-3, otherwise,
although the backbone and dilated encoder have been updated, the decoder would
still carry weights from burn-in, and may not be consistent. At the same time,
EWC is also important. To prove this, we train another model, but do not use
the EWC based penalty. Rather, we simply train the query using contrastive
learning, and simply update $\theta_{f}$ in EMA manner as:
$\theta_{f}=\mu\theta_{f}+(1-\mu)\theta_{f-q}^{*}$, with $\mu=0.85$, where
$\theta_{f-q}^{*}$ is feature extractor parameters from updated query in
Stage-2. Additionally, we also see whether solely using cropping or full-
images in Stage-2 is beneficial.

In Table 4, we report results of three models (all without decoder update): i)
(Cropped, 0.85), ii) (Full, 0.85), iii) EWC. The first two are without EWC,
but updated in EMA manner as mentioned above, and third one is with EWC based
penalty. The first one uses cropped images, while the second uses full images.
We found that solely using cropped images though helpful in detecting smaller
objects, tends to get lower overall performance then solely using Full images.
Also, using EWC based penalty is more beneficial (third row has best
performance). But, in all three cases, the lack of the decoder update causes
poorer performance than those reported in Table 2, signifying the importance
of decoder update, as expected.

Additional studies:  We also train our burn-in model for an additional 14000
iterations, similar to FDCL EWC 2 w, but found that FDCL EWC 2 w performs much
better (Table 5), indicating the benefit of contrastive learning with EWC
penalty, for bridging the domain gap. In Table 6 we compare our method’s
results with Fourier FT (simple fine-tuning with Fourier translated images).
While Fourier FT outperforms generative translation methods on small and
medium objects, our contrastive learning based FDCL EWC 2 w is still superior.
It demonstrates that image level translation is not sufficient, but, rather,
feature alignment is necessary as well, to bridge the domain gap. We also
study the convergence behaviour of Stage-2 alone, by initializing with model
from burn-in, and training just the contrastive loss. Figure 7 displays the
convergence behaviour of contrastive learning, yet again demonstrating its
benefit over other alternatives such as adversarial learning [11] for aligning
domains, when it comes to enjoying better convergence.

##  5 Conclusion: Cons and Pros

Like with many other deep learning techniques, method like ours, can be used
for negative applications (surveillance/ military). Also, being empirical, and
like all other deep learning models, our method may have corner cases of
failure, vulnerability of being fooled by adversarial attacks, addressing all
of which together is beyond this single piece of work. Having said that, the
positive outcomes of our method may far outweigh its negatives. Being solely a
computer vision approach without requiring other modalities based information
(eg, LiDAR, thermal images, etc), it could find applications in navigation and
autonomous driving, which aim at making our lives better, and easier.

## References

  * [1] Asha Anoosheh, Torsten Sattler, Radu Timofte, Marc Pollefeys, and Luc Van Gool.  Night-to-day image translation for retrieval-based localization.  In 2019 International Conference on Robotics and Automation (ICRA), pages 5958–5964. IEEE, 2019.
  * [2] Vinicius F Arruda, Thiago M Paixão, Rodrigo F Berriel, Alberto F De Souza, Claudine Badue, Nicu Sebe, and Thiago Oliveira-Santos.  Cross-domain car detection using unsupervised image-to-image translation: From day to night.  In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2019.
  * [3] Francesco Barbato, Marco Toldo, Umberto Michieli, and Pietro Zanuttigh.  Latent space regularization for unsupervised domain adaptation in semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2835–2845, 2021.
  * [4] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
  * [5] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6154–6162, 2018.
  * [6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European Conference on Computer Vision, pages 213–229. Springer, 2020.
  * [7] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al.  Mmdetection: Open mmlab detection toolbox and benchmark.  arXiv preprint arXiv:1906.07155, 2019.
  * [8] Qiang Chen, Yingming Wang, Tong Yang, Xiangyu Zhang, Jian Cheng, and Jian Sun.  You only look one-level feature.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13039–13048, 2021.
  * [9] Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [10] Piotr Dollar, Christian Wojek, Bernt Schiele, and Pietro Perona.  Pedestrian detection: An evaluation of the state of the art.  IEEE transactions on pattern analysis and machine intelligence, 34(4):743–761, 2011.
  * [11] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The Journal of Machine Learning Research (JMLR), 17(1):2096–2030, 2016.
  * [12] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le.  Nas-fpn: Learning scalable feature pyramid architecture for object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7036–7045, 2019.
  * [13] Ross Girshick.  Fast r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015.
  * [14] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Pires, Zhaohan Guo, Mohammad Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  In Proc. of Neural Information Processing Systems (NeurIPS), 2020\.
  * [15] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [17] Zhenwei He and Lei Zhang.  Domain adaptive object detection via asymmetric tri-way faster-rcnn.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIV 16, pages 309–324. Springer, 2020.
  * [18] Soonmin Hwang, Jaesik Park, Namil Kim, Yukyung Choi, and In So Kweon.  Multispectral pedestrian detection: Benchmark dataset and baseline.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1037–1045, 2015.
  * [19] Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang.  Enlightengan: Deep light enhancement without paired supervision.  IEEE Transactions on Image Processing, 30:2340–2349, 2021.
  * [20] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [21] Chongyi Li, Chunle Guo, and Change Loy Chen.  Learning to enhance low-light image via zero-reference deep curve estimation.  IEEE Transactions on Pattern Analysis & Machine Intelligence, pages 1–1, 2021.
  * [22] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [23] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [24] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [25] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In European conference on computer vision, pages 21–37. Springer, 2016.
  * [26] Michael J Milford and Gordon F Wyeth.  Seqslam: Visual route-based navigation for sunny summer days and stormy winter nights.  In 2012 IEEE international conference on robotics and automation, pages 1643–1649. IEEE, 2012.
  * [27] Lukáš Neumann, Michelle Karg, Shanshan Zhang, Christian Scharfenberger, Eric Piegert, Sarah Mistr, Olga Prokofyeva, Robert Thiel, Andrea Vedaldi, Andrew Zisserman, et al.  Nightowls: A pedestrians at night dataset.  In Asian Conference on Computer Vision, pages 691–705. Springer, 2018.
  * [28] Poojan Oza, Vishwanath A Sindagi, Vibashan VS, and Vishal M Patel.  Unsupervised domain adaption of object detectors: A survey.  arXiv preprint arXiv:2105.13502, 2021.
  * [29] Fabio Pizzati, Pietro Cerri, and Raoul de Charette.  Comogan: continuous model-guided image-to-image translation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14288–14298, 2021.
  * [30] Joseph Redmon and Ali Farhadi.  Yolo9000: better, faster, stronger.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7263–7271, 2017.
  * [31] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28:91–99, 2015\.
  * [32] Adrian Lopez Rodriguez and Krystian Mikolajczyk.  Domain adaptation for object detection via style consistency.  In BMVC, 2019.
  * [33] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko.  Universal domain adaptation through self supervision.  Advances in Neural Information Processing Systems, 33, 2020.
  * [34] Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Guided curriculum model adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7374–7383, 2019.
  * [35] Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding.  arXiv preprint arXiv:2104.13395, 2021.
  * [36] Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, and Yann LeCun.  Overfeat: Integrated recognition, localization and detection using convolutional networks.  In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
  * [37] Aashish Sharma and Robby T Tan.  Nighttime visibility enhancement by increasing the dynamic range and suppression of light effects.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11977–11986, 2021.
  * [38] Vishwanath A Sindagi, Poojan Oza, Rajeev Yasarla, and Vishal M Patel.  Prior-based domain adaptive object detection for hazy and rainy conditions.  In European Conference on Computer Vision, pages 763–780. Springer, 2020.
  * [39] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2446–2454, 2020.
  * [40] Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros.  Unsupervised domain adaptation through self-supervision.  arXiv preprint arXiv:1909.11825, 2019.
  * [41] Mingxing Tan, Ruoming Pang, and Quoc V Le.  Efficientdet: Scalable and efficient object detection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10781–10790, 2020.
  * [42] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  Advances in Neural Information Processing Systems, 30, 2017.
  * [43] Madhu Vankadari, Sourav Garg, Anima Majumder, Swagat Kumar, and Ardhendu Behera.  Unsupervised monocular depth estimation for night-time images using adversarial domain feature adaptation.  In European Conference on Computer Vision, pages 443–459. Springer, 2020.
  * [44] Johan Vertens, Jannik Zürn, and Wolfram Burgard.  Heatnet: Bridging the day-night domain gap in semantic segmentation with thermal images.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8461–8468. IEEE, 2020.
  * [45] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao.  Scaled-yolov4: Scaling cross stage partial network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13029–13038, 2021.
  * [46] Chien-Yao Wang, I-Hau Yeh, and Hong-Yuan Mark Liao.  You only learn one representation: Unified network for multiple tasks.  arXiv preprint arXiv:2105.04206, 2021.
  * [47] Xinyi Wu, Zhenyao Wu, Hao Guo, Lili Ju, and Song Wang.  Dannet: A one-stage domain adaptation network for unsupervised nighttime semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15769–15778, 2021.
  * [48] Wending Yan, Aashish Sharma, and Robby T Tan.  Optical flow in dense foggy scenes using semi-supervised learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13259–13268, 2020.
  * [49] Yanchao Yang, Dong Lao, Ganesh Sundaramoorthi, and Stefano Soatto.  Phase consistent ecological domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9011–9020, 2020.
  * [50] Yanchao Yang and Stefano Soatto.  Fda: Fourier domain adaptation for semantic segmentation.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4085–4095, 2020.
  * [51] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving dataset for heterogeneous multitask learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2636–2645, 2020.
  * [52] Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo.  Kindling the darkness: A practical low-light image enhancer.  In Proceedings of the 27th ACM international conference on multimedia, pages 1632–1640, 2019.
  * [53] Ziqiang Zheng, Yang Wu, Xinran Han, and Jianbo Shi.  Forkgan: Seeing into the rainy night.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 155–170. Springer, 2020.