# Self-supervised SAR-optical Data Fusion and Land-cover Mapping using
Sentinel-1/-2 Images

Yuxing Chen, Lorenzo Bruzzone, Y. Chen and L. Bruzzone are with the Department
of Information Engineering and Computer Science, University of Trento, 38122
Trento, Italy
(e-mail:yuxing.chen@unitn.it;lorenzo.bruzzone@unitn.it).Corresponding author:
L. Bruzzone

###### Abstract

The effective combination of the complementary information provided by the
huge amount of unlabeled multi-sensor data (e.g., Synthetic Aperture Radar
(SAR) and optical images) is a critical topic in remote sensing. Recently,
contrastive learning methods have reached remarkable success in obtaining
meaningful feature representations from multi-view data. However, these
methods only focus on image-level features, which may not satisfy the
requirement for dense prediction tasks such as land-cover mapping. In this
work, we propose a self-supervised framework for SAR-optical data fusion and
land-cover mapping tasks. SAR and optical images are fused by using multi-view
contrastive loss at image-level and super-pixel level in the early,
intermediate and later fashion individually. For the land-cover mapping task,
we assign each pixel a land-cover class by the joint use of pre-trained
features and spectral information of the image itself. Experimental results
show that the proposed approach achieves a comparable accuracy and that
reduces the dimension of features with respect to the image-level contrastive
learning method. Among three fusion fashions, the intermediate fusion strategy
achieves the best performance. The combination of the pixel-level fusion
approach and spectral indices leads to further improvements on the land-cover
mapping task with respect to the image-level fusion approach, especially with
few pseudo labels.

###### Index Terms:

Feature Fusion, Self-supervised Learning, Pixel-wise, Sentinel-1/-2, Remote
Sensing.

##  I Introduction

Every year a large number of Earth Observation Satellites are operated to
monitor human activities, Earth environment, and their mutual influences
across our planet. Hundreds of terabytes of remote sensing (RS) data are
accumulated per day from various systems, which cover most bands of the
electromagnetic spectrum and include both active and passive sensors [1]. In
this context, deep learning methods, especially supervised deep learning
approaches, have been developed to process and analyze such massive amounts of
multimodal RS data for specific applications, such as land-cover mapping,
target recognition, and change detection. However, these applications are
mostly limited to the use of a single type of image and require a large amount
of labeled data for the training of the algorithm.

The most common are deep learning techniques based on single modality data,
e.g., multispectral, hyperspectral, LiDAR, and Synthetic Aperture Radar (SAR).
The fusion of various RS data from different sensors has not received
sufficient attention yet. However, it is well known that the complementary use
of multimodal RS data offers more complete information on a scene that can
result in better performance in many applications [2]. For example,
multispectral/hyperspectral images acquire spectral information provided to
interpret the land-cover categories on the basis of their spectral signatures,
while radar images provide dielectric properties, and are not affected by
cloud occlusions.

Inspired by the success of deep learning in computer vision (CV), some works
[3, 4, 5, 6, 7, 8, 5] have investigated the fusion of multimodal RS data using
deep learning methods. Their results have shown that deep learning techniques
play a significant role in multimodal RS data fusion. However, recent
successes of deep learning techniques in multimodal RS data fusion are mainly
focused on supervised methods, which are often limited from the availability
of annotated data. Labeled remote sensing data are often scarce. The limited
access to such labeled data has driven the development of unsupervised
methods, such as generative models (e.g., GAN[9], CAE[10], VAE[11]), which are
probably the most used representation learning methods. Nevertheless, some
studies have shown that such generative models overly focus on pixels rather
than on abstract feature representations.

For land-cover mapping tasks, existing supervised methods [12, 13, 8] often
require prohibitive annotation effort for labeling data. Thus, semi-supervised
techniques [14] have been developed and explored that require very little
labeled data to train the classifier. Self-training is a popular semi-
supervised technique that trains the supervised classifier in various
iterations with the help of a few labels and the large pool of unlabeled
samples [15]. Besides, Chen et al [16], find that the use of self-supervised
pretraining for fine-tuning in self-training can give a nice improvement in
computer vision.Unlike semi-supervised learning, many unsupervised land-cover
mapping works [17, 18] have used a set of spectral indices (e.g., normalized
difference water index (NDWI), normalized difference vegetation index (NDVI),
bare soil index (BI) and backscattering values (BS)) of SAR and optical images
to select training samples from the image itself. These indices have the
potential to highlight a set of land-cover classes in terms of their physical
characteristics. We state that can be used to select pseudo labels in a self-
straining paradigm.

Recent researches [19, 20, 21, 22, 23] in contrastive learning demonstrate how
these methodologies can encourage the network to learn more interpretable and
meaningful feature representations. This resulted in improvements in
classification and segmentation tasks, where contrastive methods outperformed
the generative counterparts. However, existing contrastive methods (MCL)
rarely consider the pixel-wise feature representations.

To address this limitation, in this paper we propose a new self-supervised
fusion approach to fuse the complementary information present in SAR and
optical images at the pixel level. The proposed approach can be implemented
according to three fusion strategies: I) early fusion (PixEF), II)
intermediate fusion (PixIF) and late fusion (PixLF). We also investigate the
self-trained land-cover classification in considering spectral information in
SAR-optical images and the benefits of self-supervised pre-trained features.
The proposed SAR-optical fusion approach is compared with the instance level
contrastive method under the common linear protocol in the context of a land-
cover mapping task. The evaluation of the self-supervised land-cover mapping
approach was performed on a subset of DFC2020.

The main contributions of this work are as follows:

1) We first introduce and verify the effectiveness of multi-view contrastive
loss in SAR-optical data fusion. Then we propose a self-supervised approach,
which can obtain pixels-wise feature representations from SAR and optical
image pairs without using any annotation. This is achieved by using U-Net and
contrastive loss, preserving local information at the superpixel level.

2) We investigate the impact of different fusion strategies (i.e., early
fusion, intermediate fusion and later fusion) in the proposed approach.
Concretely, late and intermediate fusion fashion learns feature
representations by comparing SAR and optical images directly, while early
fusion fashion distills the complementary information from the concatenation
of image pairs. In addition, the efficiency of SAR-optical fusion with respect
to the use of a single modality in the land-cover mapping task is analyzed
correspondingly.

3) We conduct a self-supervised land-cover mapping task by jointly using the
spectral information of SAR-optical images and the pretraining fusion
features.

The rest of this paper is organized as follows. Section II presents the
related works of SAR-optical data fusion based on supervised and unsupervised
methods. Section III introduces the proposed approach by describing the three
considered self-supervised data fusion fashion and a self-training land-cover
mapping task. The descriptions of the dataset, network setup, and experimental
settings are given in Section IV. Experimental results obtained on linear
protocol and a land-cover mapping task are illustrated in Section V. Finally,
Section VI concludes the paper.

##  II Related Works

Various feature fusion methods, including supervised learning and unsupervised
learning techniques, have been investigated to improve the performance of
combining complementary information from SAR and optical images. Early works
in both already proved the effectiveness of combining SAR and optical data
with the multi-layer perception (MLP) classifier [24, 25]. Newly Sentinel-1
and Sentinel-2 images are combined to improve land-use and land-cover (LULC)
classification accuracy on monsoon regions using a random forest model in
[26]. However, these fusion algorithms are a simple concatenation of SAR and
optical images and have no capability to learn high-level features. With the
development of deep convolution neural networks (CNNs), hand-crafting features
have become unnecessary for SAR-optical feature fusion. Task-relevant features
can be extracted from the input data automatically during the network
training. Kussul et al.[27] first explore the deep CNNs in SAR-optical fusion
for LULC classification and demonstrate its superior to traditional MLP
classifiers. In [28], Feng et al. propose a multi-branch CNNs to improve the
classification accuracy in coastal areas by fusing Sentinel-1 and Sentinel-2
images. A multi-temporal W-Net was proposed to integrate Sentinel-1 and
Sentinel-2 images in land-cover mapping [12]. An integration of fully
convolutional networks (FCN) and object-based methods are proposed for LULC
mapping and achieved an overall accuracy of 95.33% with the Sentinel SAR and
optical images [13]. Recently, Dino et al. [8] propose a deep learning
architecture, namely TWINNS, to fuse Sentinel-1 and Sentinel-2 time series
data in land-cover mapping. Besides the complete supervised fashion, Wang et
al. [29] find a correlation between the image classification accuracy and the
segmentation accuracy, and demonstrate that a model trained on image
classification performs well on a segmentation task in a weakly supervised
way.

Supervised learning methods, despite their powerful ability to feature
learning of SAR-optical images, are often limited by the amount of annotated
data. An alternative method is to use unsupervised learning techniques, such
as principal component analysis (PCA), convolutional AutoEncoder (CAE) and
canonical correlation analysis (CCA)[30]. These techniques can learn feature
representations from unlabeled multimodal data. In [31], Amarsaikhan et al.
use PCA to enhance the features extracted from SAR-optical images and improve
the urban land-cover maps. Similarly, multi-view learning methods also provide
a solution to the unsupervised combination of complementary information from
SAR-optical images. In [32], Jie et al. propose a deep bimodal autoencoder
(BDAE) to fuse SAR and multispectral images for classification. In [33], Allan
et al. jointly analyze Sentinel SAR and optical data for change detection
using CCA. Based on CCA, Andrew et al. propose the deep canonical correlation
analysis (DCCA), which learns separate representations for each modality from
a shared latent subspace using CNNs.

Figure 1: Overview of the presented self-supervised SAR-optical fusion
approach. The dash arrow line represents a contrastive loss. (a) An
illustration of pixel-wise representation learning framework for the late
fusion fashion. The two inputs have an offset but keep an overlap. The
approach follows the common contrastive learning architecture where both
branches consist of a ResUnet block and a projection. Then, a shift
transformation is included in the one branch for aligning representations
between two branches. (b) The ResUnet block follows the early fusion fashion.
(c) The ResUnet block follows the intermediate fusion where the encoder
contains two parts used for encoding SAR and optical images independently.

Despite the success of these methods, their performance is limited in feature
learning ability or overly focus on the pixel values. The recent success of
contrastive learning [20] shows the possibility of alleviating such a
dependency and obtaining useful representations from unlabeled data. Tian et
al. [21] propose an extension of the contrastive loss to self-supervised
learning at the multi-view setting, where different views are related to the
same context. Maximizing mutual information between different modalities
within the same context will force the corresponding sub-networks to generate
the same or similar features.

Even with the useful representation, the absence of labels is not enough for
the land-cover mapping task. Many works [18, 17, 14] have been dedicated to
decreasing the use of labels using the spectral information in SAR-optical
images themselves. Aswatha et al. [17] jointly use scattering and spectral
properties from SAR and multi-spectral images to distinguish land-cover
classes as training samples. Besides, they train a random forest classifier to
predict land-cover classes based on the selected samples and learned features.
Wu et al.[14] utilize self-training to gradually assign highly confident
pseudo labels to unlabeled hyperspectral images by clustering and employs
spatial constraints to regulate the self-training process. Rather than
training from scratch, Chen et al. find that the self-training based on self-
supervised pre-training can improve the performance in classification task
[34].

##  III Methodology

This section presents the proposed self-supervised approach to SAR-optical
image fusion, which aims to learn pixel-wise representations from unlabeled
SAR-optical image pairs. The proposed SAR-optical fusion approach consists of
three fusion strategies: early fusion, intermediate fusion and late fusion
(see Fig. 1 (a, b and c)). Afterward, a land-cover mapping task based on the
spectral information of the image itself and the pre-trained features is
presented.

###  III-A Network Architecture

The proposed approach has two branches (Fig. 1 (a)), where the input image of
each branch has a relative shift. Each branch contains a ResUnet block
followed by a linear layer projector. After the projection, the same shift
operation is performed on the output for feature alignment between two
branches. We adopt a similar ResUnet architecture as the [35] and only used
residual blocks in the encoder part. Like U-net, ResUnet consists of encoder,
decoder and skip connections between downsampling and upsampling path. In this
work, ResNet-18 is used as the encoder of the ResUnet block, but without the
fourth residual block. The decoder part has three blocks and consists of a
convolution layer (Conv) and batch normalization (BN), ReLU, and upsampling in
each block. All padding type in the ResUnet block was changed as the same
padding. The projector consists of a $1\times 1$ Conv for each pixel and
follows with the ResUnet block is used to reconstruct the learned
representations.

Feature alignment is achieved by using same shift operation on input images
and output features of two branches individually. Specifically, given an input
$I_{1}$ consisting of SAR and optical image pairs obtained from the same
scene, we use a shift operation to make a random offset of $I_{1}$ along the x
and y-axis direction. In this way, we can define the random shift
transformation as $T$ and obtain an augmented view $I_{2}=T(I_{1})$. During
the training, the augmented view and original input are fed into two branches,
respectively, to obtain pixel-wise representations $v_{1}$ and $v_{2}$. To
align pixel-wise representations of two branches, the same transformation is
applied to the output of the other branch $v_{1}=T(v_{1})$.

In particular, two branches of PixEF and PixIF share the same parameters, but
the encoder of PixIF is split into two groups and each group has half channels
of the counterpart of PixEF in each layer. We denote the network of PixEF as
$U_{e}$ and its encoder as $E_{e}$. Meanwhile, the network of PixIF as $U_{i}$
and its two indepent encoders as $E_{i1}$ and $E_{i2}$. Unlike these two
models, the PixLF has two independent branches with half channels of PixEF in
each layer, where the input channels were adjusted to the input images. We
denote the network of PixLF as $U_{l}$ and its two indepent encoders as
$E_{l1}$ and $E_{l2}$.

###  III-B Loss Function

The proposed approach consists of two types of contrastive loss based on image
and superpixel individually. The main idea behind a contrastive loss is to
find a feature representation that is invariant to augmentations. Given a
dataset of $S$ that consists of a collection of image pairs
$\\{s_{1}^{i},s_{2}^{i}\\}_{i=1,2,\dots,N}$, we consider each image pair
$(s_{1}^{i},s_{2}^{i})$ sampled from the joint distribution
$p(s_{1}^{i},s_{2}^{i})$, which we call positives. And $s_{2}^{j}$ is taken
from other scene, then samples $(s_{1}^{i},s_{2}^{j})$ from the product of
marginals $p(s_{1}^{i})p(s_{2}^{j})$, which we call negatives. The model
$h(\cdot)$ is expected to know which pair was drawn from the joint
distribution while the other is not exactly by computing their cosine
similarity with a hyper-parameter $\tau$. In multi-view setting, the model
$h(\cdot)$ is a neural network consits of two braches that with the
independent parameter $f_{{\theta}_{1}}$ and $f_{{\theta}_{2}}$.

$h\left(s_{1},s_{2}\right)=\exp\left(\frac{f_{\theta_{1}}\left(s_{1}\right)\cdot
f_{\theta_{2}}\left(s_{2}\right)}{\left\|f_{\theta_{1}}\left(s_{1}\right)\right\|\cdot\left\|f_{\theta_{2}}\left(s_{2}\right)\right\|}\cdot\frac{1}{\tau}\right)$
(1)

The final loss function can be writen as
$L(f_{{\theta}_{1}},f_{{\theta}_{2}},S)$ given dataset $S$.

$L(f_{{\theta}_{1}},f_{{\theta}_{2}},S)=-\underset{S}{\mathbb{E}}{\left[\log\frac{h(s_{1}^{1},s_{2}^{1})}{\sum_{j=1}^{N}h(s_{1}^{1},s_{2}^{j})}\right]}$
(2)

where $(s_{1}^{1},s_{2}^{1})$ is a positive pair sample,
$(s_{1}^{1},s_{2}^{j}|j\geq 1)$ is a negative pair sample and
$S=\\{s_{1}^{1},s_{2}^{1},s_{2}^{2},\cdots,s_{2}^{N-1}\\}$ is a set that
contains $N-1$ negative samples and one positive sample. In the training
process, the network is trained to increase the value of positive pairs and
decrease the value of negative pairs. This results in a feature representation
that is close to positive pairs whereas it is not for negative pairs.

In the pixel-level contrastive loss, we sample and average features from two
branches over superpixels that are located on the overlap between two
branches. This aims to keep the consistency of the normalized pixel-wise
representations between two branches. Here, we construct a set of pixel-wise
feature pairs $P$ where positive feature pair $(p_{1}^{i},p_{2}^{i})$ is
sampled from the same location while $p_{2}^{j}$ in negative pairs was taken
from another location. Compared with the instance-level contrastive learning,
this loss function can make the model get more detailed representations and is
more suitable for dense prediction downstream tasks. To overcome the noise
when using single pixels, we adopt the contrastive loss at the superpixel
level. Together with the pixel-level contrastive loss, an instance-level
contrastive loss is used to improve the performance. The instance-level loss
help to discriminate the similarity between the shited views. Like pixel-level
loss, we can construct a set $M$ of concatenated image pairs, where
$(m_{1}^{i},m_{2}^{i})$ is sampled from the same scene $i$ while $m_{2}^{j}$
is taken from another scene. Finally, we use the pixel-wise contrastive loss
in conjunction with the image-level contrastive loss, leading to the total
loss of three fusion approches:

$\begin{array}[]{l}\mathcal{L}_{\text{e}}=L(U_{e},U_{e},P)+L(E_{e},E_{e},M)\\\
\mathcal{L}_{\text{l}}=L(U_{l1},U_{l2},P)+L(E_{l1},E_{l2},M)\\\
\mathcal{L}_{\text{i}}=L(U_{i},U_{i},P)+L(E_{i1},E_{i2},M)+L(E_{i},E_{i},M)\end{array}$
(3)

where $\mathcal{L}_{\text{e}}$,$\mathcal{L}_{\text{l}}$ and
$\mathcal{L}_{\text{i}}$ are the loss function of PixLF, PixEF and PixIF
individually.

###  III-C Self-training Land-cover mapping

To further implement the proposed SAR-optical fusion approach in land-cover
mapping tasks that do not rely on manually annotated data, a self-training
process is introduced and used in conjunction with the spectral information of
images. We first calculate a set of spectral indices of each image and then
select reliable, representative and diverse pseudo labels according to the
characteristics of different land-cover classes. For different spectral
indices, the larger or minor index value the pixel has, the greater
possibility that pixels are associated with the related land-cover classes. In
this paper, NDVI, NDWI, BI and BS were used to produce the index images for
different classes. The generation of pseudo labels for different land-cover
classes that not only decided on the certain spectral index but also on those
indices related to other classes, which can be utilized for synergistically
labeling a pixel.

To avoid choosing a threshold value for each index, we use k-means [36] to
over cluster Sentinel-1/-2 images and calculate the average spectral indices
and average backscatter value for each class. Here, we segment sentinel-2
images to eight classes and sentinel-1 images to four classes. Then, we define
a set of generic rules (see Algorithm 1) for labeling 6 classes considering
all indices and the combination of pseudo-classes defined by k-means on
Sentinel-1 and -2 images. We denote labeled pixels of the forest, grassland,
water, urban, bare land, and sparse vegetation as $U^{f}$, $U^{g}$, $U^{w}$,
$U^{u}$, $U^{b}$ and $U^{s}$. For instance, the NDVI can indicate forest and
grassland, and backscatter values in SAR images can further discriminate
between these two classes. It is noted that just a few pixels with high
confidence (thresholds) are labeled with a given class and the mean value of
pre-defined classes is used as the threshold value.

The generated sparse pseudo labels are utilized to train a linear classifier
layer based on the learned fusion features and then the classes of unlabeled
pixels are predicted using the well-trained models. However, the generated
labels inevitably contain incorrect predictions. To overcome this problem, we
fine-tune all parameters on the inference labels. The training framework in
this task is illustrated in Fig. 2. It is noted that the pre-trained PixIF
model is used as the backbone and followed a linear classifier.

Figure 2: Overview of the presented self-training land-cover mapping approach.
The step one is training on the sparse pseudo labels first. Then, training
whole network on the predicted labels in step two. Algorithm 1 Unsupervised
Training Samples Collection.

0: $I_{1}$, $I_{2}$ images, NDVI, NDWI, BI and BS

0: $U^{f}$, $U^{g}$, $U^{w}$, $U^{u}$, $U^{b}$, $U^{s}$

1: Overclustering $I_{1}$, $I_{2}$ into $m$ and $n$ classes.

2: Selecting the classes with maximun indicies values $H_{ndvi}$, $H_{ndwi}$,
$H_{bi}$ in $I_{2}$ and $H_{bs}$ in $I_{1}$ as well as the class with minum
value $L_{bs}$ in $I_{1}$. Also selecting the classes with medium NDVI and BI
value$M_{ndvi}$ and $M_{bi}$. Then calculate their mean values $V_{ndvi}$,
$V_{ndwi}$, $V_{bi}$, $V_{bs}$.

3: Iterating each pixel $p$ in $I_{1}$ and $I_{2}$.

4: if $p$ in $H_{ndwi}$ and $L_{bs}$; NDWI$(p_{2})>V_{ndwi}$ then

5: $p\in U^{w}$

6: elif $p$ in $H_{ndvi}$ and $H_{bs}$; NDVI$(p_{2})>V_{ndvi}$ then

7: $p\in U^{f}$

8: elif $p$ in $H_{ndvi}$ and $L_{bs}$; NDVI$(p_{2})>V_{ndvi}$ then

9: $p\in U^{g}$

10: elif $p$ in $H_{bs}$; BS$(p_{1})>V_{bs}$ then

11: $p\in U^{u}$

12: elif $p$ in $H_{bi}$ and $L_{bs}$; BI$(p_{2})>V_{bi}$ then

13: $p\in U^{b}$

14: elif $p$ in $M_{bi}$ and $M_{ndvi}$; NDVI$(p_{2})<V_{ndvi}$ then

15: $p\in U^{s}$

16: else

17: $p\in None$

18: end if

19: return $U^{f}$, $U^{g}$, $U^{w}$, $U^{u}$, $U^{b}$, $U^{s}$

##  IV Design of Experiments

In this section, we present the dataset for training and validation of the
proposed approach as well as the land-cover mapping task. Besides, the details
of network setup and evaluation experiments are introduced.

###  IV-A Description of the Dataset

####  IV-A1 DFC2020

We developed our experiments on the DFC2020 dataset. The DFC2020, which has
been issued by the IEEE-GRSS 2020 Data Fusion Contest [37], is used as the
training set and the evaluation set for comparison between different methods.
This dataset consists of a total of 6114 quadruple samples, which are SAR-
optical image pairs, MODIS-derived labels, and more accurate semi-manually
derived high resolution (10m) land-cover maps [38]. SAR images were acquired
by Sentinel-1 and consist of dual-polarized (VV and VH) components and the
optical images were taken by multi-spectral Sentinel-2. Each pixel in the
DFC2020 was assigned to a land-cover class manually, which has eight fine-
grained classes (i.e., Forest, Shrub-land, Grassland, Wetlands, Croplands,
Urban/built-up, Barren and Water). We also provide the image-level label for
each scene, which is derived by the majority class of the related pixel-level
land-cover maps. The previous research [29] pointed out the effectiveness of
training a CNN on image-level labels that can guide the weakly supervised
model (WSL) to learn a powerful representation of images.

####  IV-A2 Training and Test Splits

A random split of the DFC2020 dataset into a training set (1000) and a test
set (5114) were applied in this work. Here, the training set is used to tune
the parameters of the linear protocol in the evaluating phase. The test set
was used to validate the effectiveness of the features learned using different
methods. To assess the effectiveness of these methods with limited labels, we
randomly split the training set into five groups with 10, 50, 100, 200, and
1000 samples. Each small number group is a sub-samples of the corresponding
full training set. Note that all self-supervised and unsupervised models were
trained on unlabeled SAR-optical image pairs. In addition, we selected 2000
samples from the whole data set for the self-training land-cover mapping task.
We reclassify the cropland, wetland, and grassland as grassland and reclassify
shrubland and barren as bared land. Other classes keep the same as the
original. Therefore, this data set only contains 6 classes (i.e., Forest,
Grassland, urban, bare land, water and sparse vegetation) according to the
land surface properties.

###  IV-B Network Setup

The training process of the self-supervised approach includes three parts for
PixEF, PixLF and PixIF. For PixEF, SAR-optical image pairs were concatenated
as one input. For PixIF and PixLF, SAR and optical images are in input to two
branches independently. The Adam with a learning rate of 0.0003, a weight
decay of 0.0004 and a momentum of 0.9 was adapted as our optimizer. We use a
mini-batch size of 1000; models are run for 700 epochs. We deploy a step
scheduling learning rate policy in the training process. Apart from the shift
transformation, we further apply a random flip transformation to improve the
performance of the proposed approach.

For self-training in the land-cover mapping task, we train the model with a
mini-batch size of 50, and models are run for 100 epochs. Adam optimizer is
used with an initial learning rate of $1e^{-4}$ for the pre-trained encoder
layer and $3e^{-4}$ for the other parameters. When finetuning, the initial
learning rate was set as $1e^{-4}$ for all parameters.

###  IV-C Experiment Settings

To evaluate the learned feature representation of different methods, we
provide an evaluation with a linear classifier followed by the frozen features
on the test set. In particular, the feature representation in the proposed
approach has 256 channels, while that of the comparison methods (i.e., DCCA
and MCL) is a concatenation of multi-level features with 512 channels. Note
that we decided to use a linear classifier as our main evaluation metric for
the quality of representations since it has a small number of extra
parameters. The learning rate is set to 0.05 and the SGD with a mini-batch
size of 8 was adopted as the optimizer for the linear protocol as well as the
maximum number of epochs is 50.

In this experiment, we also assess the performance of the proposed approach
applied to the Sentinel-1 (S1) image, the Sentinel-2 (S2) image, and both of
them (S1S2). This is done to validate whether the SAR-optical fusion can
obtain more discriminative representations than single image type for the
downstream land-cover mapping task. To this purpose, we train the proposed
approach only on single modality images in the proposed early fusion fashion.
Then the linear protocol on the frozen pre-trained models was used to evaluate
the effectiveness of learned representations with limited training labels.

In the land-cover mapping task, we adopt features obtained from the proposed
self-supervised SAR-optical fusion approach and instance-level contrastive
approach as pre-training. However, the instance level self-supervised method
does not have an encoder and can not perform the same strategy. To provide a
fair comparison, we also adopt the same decoder and the classifier of the
proposed approach as a readout, which is followed by encoders pre-trained by
instance self-supervised methods and used to reconstruct the concatenation
features for downstream tasks. Furthermore, the proposed approach is fine-
tuned with the predicted pseudo labels. To present a sparse label scenario, we
just keep less than 10-pixel labels for each class of each image, where aband
the class with less than 10-pixel labels.

##  V Results

###  V-A Linear Evaluation

The performance of the proposed approaches (PixEF, PixIF and PixLF) were
evaluated on the test set in comparison to the weakly supervised method (WSL)
and the other two instance-level self-supervised methods (DCCA and MCL). Also
in this case we considered different amounts of labeled data for the training
of the linear classifier (see Fig. 3). The average class accuracy (AA) and
mean intersection over union (mIoU) are common metrics used to assess the
performance in land-cover mapping and are used to evaluate the overall
precision of all land-cover classes in our work.

Fig. 3 shows the linear protocol results on mIoU. As we can see that the
proposed PixIF outperforms all other methods while the DCCA performs
significantly worse against any other method when the number of training
samples increases. However, the WSL method outperforms all other methods with
few labels. In general, the proposed PixEF and PixLF as well as the MCL have a
similar performance. And the gap between WSL and all contrastive approaches is
reduced when the number of labeled samples increases. Finally, the proposed
PixIF outperforms WSL in the case of 1000 training samples. Among contrastive
approaches, the performance of PixIF is slightly better than MCL. It is worth
noting that MCL obtained representations with 512 channels, while that of
proposed approaches only with 256 channels. This means that the proposed
approach significantly reduces the dimensionality of features while keeping or
even improving the feature representation ability.

Figure 3: Mean intersection over union metric achieved by different methods on test set versus the number of samples used for the training of the linear classifier on frozen encoders. TABLE I: Class-wise and overall accuracies achieved on the test set by a linear classifier used with the different methods considering 1000 SAR-optical training samples. Class | Average Accuracies %
---|---
WSL | DCCA | MCL | PixIF | PixLF | PixEF
Forest | 90.2 | 92.7 | 90.9 | 92.6 | 92.3 | 91.3
Shrub-land | 70.4 | 26.3 | 40.4 | 50.8 | 46.3 | 53.2
Grassland | 57.6 | 63.0 | 68.7 | 73.0 | 64.5 | 74.5
Wetlands | 70.4 | 56.9 | 64.9 | 62.7 | 57.2 | 59.9
Croplands | 81.2 | 72.8 | 82.8 | 84.7 | 85.8 | 80.0
Urban | 86.6 | 86.9 | 87.7 | 87.8 | 84.4 | 87.1
Barren | 34.5 | 6.3 | 46.3 | 30.9 | 29.5 | 37.0
Water | 99.2 | 99.0 | 99.3 | 99.3 | 99.4 | 99.2
AA | 73.8 | 63.0 | 72.7 | 72.7 | 69.9 | 72.8
mIoU | 0.490 | 0.411 | 0.487 | 0.498 | 0.476 | 0.490
Figure 4: Land-cover maps achieved on five different images by different
considered methods with a linear classifier. (see Table I for quantitative
results.)

Table I presents a detailed comparison of the class-by-class accuracy obtained
by the classifier on the test set for each method when trained with 1000
labeled samples. According to the results, the proposed PixIF and PixEF and
WSL achieve an AA higher than 72% and the mIoU over 0.49 on the test set. They
sharply outperform DCCA, which obtain an AA smaller than 65% and the mIoU less
than 0.42. Among proposed approaches, PixIF obtained a higher mIoU while PixLF
get the worst performers. In addition to the quantitative evaluations, we also
provide a qualitative visual comparison of the land-cover maps predicted by
different methods. Fig. 4 illustrates five examples of results. Each example
includes the predicted land-cover maps of different methods as well as the
ground truth in DFC2020. As one can observe, the proposed PixIF, PixEF and WSL
show better results than the rest of the methods in all cases, and the
proposed PixIF confirms to be more effective than the other two fusion
strategies (PixEF and PixLF). For each land cover class, similar conclusions
to those derived by Table I can be given.

In general, the results obtained in all comparisons confirm that the
contrastive approach is superior to DCCA in this land-cover mapping task. The
effectiveness of the proposed PixIF against other methods is due to its
ability to use three-level contrastive loss. It is interesting to note that
despite the annotations are used in WSL, the proposed approach achieves
comparable performance without any use of labels. This confirms the
effectiveness of the self-supervised methods in feature representation
learning.

We further investigate the performance of a single sensor image in the land-
cover mapping task, where we train the proposed PixEF approach on Sentinel-1
(S1) and Sentinel-2 (S2) image as well as the concatenation of Sentinel-1/-2
(S1S2) images individually. TableII shows a quantitative evaluation of the
accuracy on each class considered S1, S2 and S1S2 with the 1000 training
samples under both two evaluations. As one can see, the SAR-optical fusion
outperforms the use of any single modality data in both linear protocol and
fine-tune evaluations. The performance of PixEF on S2 is very close to the
performance on S1S2. While the performance on S1S2 has an increase of more
than 10% of AA with respect to the performance on S1 in both types of
evaluation. In addition, the use of sentinel-2 data results in higher
classification accuracy on all classes than the use of only sentinel-1 data
for both evaluation methods. For individual classes, water achieved the
highest accuracy in all conditions, whereas the barren has the lowest
accuracy. The classification accuracy of water does not show an obvious
improvement after fusion, because there is already enough information in each
single modality data. However, the SAR-optical fusion improved the performance
of shrubland, grassland and barren.

TABLE II: Class-wise and overall accuracies achieved by PixEF on Sentinel-1 images alone (S1), Sentinel-2 images alone (S2) and Sentinel-1/-2 image fusion (S1S2) with the linear protocol and the fine-tuning evaluation. class | Linear Evaluation | Fine-tuning Evaluation
---|---|---
S1 | S2 | S1S2 | S1 | S2 | S1S2
Forest | 84.2 | 90.3 | 91.3 | 86.2 | 92.2 | 92.0
Shrubland | 27.5 | 48.9 | 53.2 | 31.2 | 44.2 | 62.3
Grassland | 61.1 | 67.2 | 74.5 | 61.9 | 68.0 | 78.0
Wetlands | 35.0 | 58.8 | 59.9 | 51.5 | 62.3 | 62.3
Croplands | 66.0 | 81.4 | 80.0 | 71.1 | 79.6 | 85.7
Urban | 78.8 | 86.5 | 87.1 | 84.7 | 89.7 | 86.1
Barren | 3.5 | 30.0 | 37.0 | 7.8 | 30.7 | 38.6
Water | 98.8 | 99.2 | 99.2 | 99.2 | 99.5 | 99.4
Average | 56.9 | 70.0 | 72.8 | 61.7 | 70.8 | 75.6
mIoU | 0.362 | 0.470 | 0.490 | 0.395 | 0.474 | 0.521
Figure 5: Land-cover maps obtained by PixEF on Sentinel-1 images alone (S1),
Sentinel-2 images alone (S2) and Sentinel-1/-2 image fusion with the linear
classifier and fine-tuning evaluation for five different images. (see Table IV
for quantitative results.)

Apart from quantitative assessment, we also made a visual quantitative
comparison of both two evaluations on S1, S2 and S1S2, respectively.
Obviously, the performance of each modality and SAR-optical fusion keeps
consistent in both evaluation criteria. As shown in Fig. 5, the SAR-optical
fusion classifies various classes in a more accurate way, especially in
barren, which obtains a significant improvement with respect to the use of
single modality data. Besides, a trend can be figured out, that is, the
methods with the input of fused S1S2 data achieve more smooth parsing results
compared with the input of single modality data. Moreover, Fig. 5 also shows
the advantages of each single modality and other improvements after SAR-
optical fusion. Both Fig. 5(a) and Fig. 5(b) show that the presence of clouds
on Sentinel-2 (but of course not in Sentinel-1) and the obvious influence on
the corresponding classification map. The presence of the cloud leads to
misclassification in Sentinel-2, but not in Sentinel-1. Although the
misclassification induced by clouds is still present in the classification
maps of SAR-optical fusion, it is significantly mitigated compared to the
result of Sentinel-2 alone. A similar phenomenon is also present in Fig.5(e),
where the airport was distinguished as water in the Sentinel-1 result. This is
the results of the similar backscatter between flat runway and water.
Conversely, this was correctly distinguished as built-up in the Sentinel-2
result. After SAR-optical fusion, the misclassification error in each modality
was obviously reduced. Overall, the visual comparison is coherent with the
quantitative results presented in Table II and confirms again the
effectiveness of the presented self-supervised approach.

###  V-B Self-trained land-cover mapping task

TABLE III: Class-wise and overall accuracies achieved on the test set by a linear classifier used with the different methods considering 2000 samples. Method | Forest | Sparse Veg. | Grass. | Built. | Barren | Water | AA | mIoU
---|---|---|---|---|---|---|---|---
MCL | 87.3 | 36.9 | 53.3 | 68.8 | 42.5 | 99.0 | 64.7 | 0.408
PixIF | 89.6 | 39.3 | 55.7 | 73.0 | 47.4 | 99.2 | 67.4 | 0.438
Fine. | 91.6 | 38.2 | 57.2 | 74.4 | 47.1 | 99.3 | 68.0 | 0.449

We further validate the proposed approach on a particular self-supervised
land-cover mapping task. TableIII presents a quantitative evaluation of the
accuracy on each class considered image-level feature fusion (MCL) and the
pixel-level feature fusion (PixIF) as well as the fine-tuning results of
PixIF. As one can see, the pixel-level fusion approach achieved a higher AA of
67.4% sharply outperforming the image-level fusion approach, which obtain an
AA of 64.7%. Similarly, compared with the image-level fusion, the pixel-level
fusion outperforms it with a margin of 0.03 in mIoU. In particular, the fine-
tuning results obtain a higher AA and mIoU, with a significant improvement of
3.3 % and 0.04 with respect to the image-level fusion.

For individual classes, forest and water achieved the highest accuracy. On the
contrary, the accuracy of sparse vegetation and barren is below 50%. The
classification of the sparsely vegetated ground is a big challenge for all
methods. The possible reason is that the spectral characteristics of forest
and water are very distinctive and very easy to be distinguished. On the
contrary, barren and sparse vegetation present a high similarity of textural
and spectral characteristics. Another possible reason has reported in [39], is
that there are also many misclassifications among shrub-land, wetland, barren,
and cropland contained in the origin ground-truth label.

Figure 6: Self-trained land-cover maps obtained by MCL with decoder and pixel-
wise feature fusion as well as its fine-tuning results.

Apart from quantitative assessment, we also made a visual quantitative
comparison of the land-cover maps predicted by different methods. As shown in
Fig. 6, the pixel-level feature fusion classifies various classes in a more
accurate way, especially in small areas. A trend can be figured out, that is,
the proposed approach achieves more smooth parsing results compared with the
MCL results. And the result of fine-tuning also leads to an improvement on AA
and mIoU. Overall, the visual comparison is coherent with the quantitative
results presented in Table III and confirms again the effectiveness of the
proposed self-supervised approach on the land-cover mapping task.

TABLE IV: The effect of the use of geometric, photometric, shift augmentation and global loss in the proposed approach. Only Shift | Geometric (affine) | Photometric | Global loss | AA | mIoU
---|---|---|---|---|---
$\checkmark$ |  |  | $\checkmark$ | 72.7 | 0.498
| $\checkmark$ |  | $\checkmark$ | 68.4 | 0.460
|  | $\checkmark$ | $\checkmark$ | 68.8 | 0.464
$\checkmark$ |  |  |  | 70.6 | 0.479

###  V-C Discussion

In this section, we discuss the effects of different components of PixIF that
contribute to its performance. All results showed in Table IV are trained and
tested on the same setup of the linear protocol with 1000 training samples.
The first row refers to the proposed approach using only shift transformation
and additional global loss. The second row refers to the proposed approach
using geometric transformation (shift, rotate, resize and sheer) instead of
shift operation and additional global loss. The third row refers to the
proposed approach using photometric transformation (gaussian blur and noise)
and additional global loss. And the fourth row refers to the proposed approach
only use shift operation.

As one can see, the combination of shift operation and the use of global
contrastive loss achieves the highest accuracy. In contrast, the lack of
global loss makes the performance slightly decayed. It demonstrates the
benefits of the use of image-level contrastive loss. We also investigate a
universal geometric transformation instead of shift operation in the network
training, where the performance drops about 0.04 on mIoU. The proposed
approach can work with the affine geometric transformation but with little
performance drops. Similarly in photometric transformation, it leads to 0.03
drops on mIoU. Finally, we find that the shift operation is the most useful
and simple data augmentation approach for the proposed approach in this work.

##  VI Conclusion

In this paper, we proposed a new self-supervised SAR-optical data fusion
approach by jointly using image-level and pixel-level contrastive loss and
shift data augmentation. The proposed approach explores three fusion
strategies to distill related representations from different modalities data.
We additionally investigate the efficiency of SAR-optical fusion with respect
to the single modality use in the proposed approach.

To evaluate the performance of the proposed approach, we compared it with two
instance-level self-supervised methods (i.e., CML and DCCA) and also with a
weakly supervised method considering the linear protocol evaluation with
different numbers of training samples. The results show that the proposed
PixIF achieves the best outperforms among all self-supervised methods and
achieves comparable performance to that of a weakly supervised method. The
effectiveness of the proposed PixIF can be explained by the use of different
levels of contrastive loss for a dense prediction task. Comparisons between
the performance of proposed PixEF on SAR-optical fusion and single modality
data were also considered. The experiment verified again the benefit of SAR-
optical fusion in the land-cover mapping task.

We also propose a self-training land-cover mapping approach based on the self-
supervised pre-training and spectral information of images themselves. This
approach is assessed quantitatively and qualitatively by the selected subset
of DFC2020, which validated an average accuracy measure of 68% in classifying
land-cover to six classes. Results show that the proposed PixIF together with
the self-training approach can achieve a comparable result on land-cover
mapping without manually annotated labels. The results confirm the benefit of
the proposed approach in the land-cover mapping task with respect to the use
of the instance-level self-supervised methods. On the basis of the freely
accessed Sentinel-1/-2 data, the proposed approach demonstrates a promising
potential for automatic large-scale land-cover mapping.

The proposed approach also has some limitations in initial pseudo labels
generation. The considered training strategy just gives six classes and just
focuses on values of backscatter values of sentinel-1 images while ignores the
polarization information. Accordingly, as future development, we plan to
explore the possibility of more precise classes in the presented approach for
the land-cover mapping task.

## References

  * [1] M. Chi, A. Plaza, J. A. Benediktsson, Z. Sun, J. Shen, and Y. Zhu, “Big data for remote sensing: Challenges and opportunities,” _Proceedings of the IEEE_ , vol. 104, no. 11, pp. 2207–2219, 2016.
  * [2] L. Gómez-Chova, D. Tuia, G. Moser, and G. Camps-Valls, “Multimodal classification of remote sensing images: A review and future directions,” _Proceedings of the IEEE_ , vol. 103, no. 9, pp. 1560–1584, 2015.
  * [3] N. Audebert, B. Le Saux, and S. Lefèvre, “Beyond rgb: Very high resolution urban remote sensing with multimodal deep networks,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 140, pp. 20–32, 2018.
  * [4] X. Jiang, G. Li, Y. Liu, X.-P. Zhang, and Y. He, “Change detection in heterogeneous optical and sar remote sensing images via deep homogeneous feature fusion,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 13, pp. 1551–1566, 2020.
  * [5] J. Bermudez, P. Happ, D. Oliveira, and R. Feitosa, “Sar to optical image synthesis for cloud removal with generative adversarial networks.” _ISPRS Annals of Photogrammetry, Remote Sensing & Spatial Information Sciences_, vol. 4, no. 1, 2018.
  * [6] H. Li, P. Ghamisi, U. Soergel, and X. X. Zhu, “Hyperspectral and lidar fusion using deep three-stream convolutional neural networks,” _Remote Sensing_ , vol. 10, no. 10, p. 1649, 2018.
  * [7] S. Srivastava, J. E. Vargas-Muñoz, and D. Tuia, “Understanding urban landuse from the above and ground perspectives: A deep learning, multimodal solution,” _Remote sensing of environment_ , vol. 228, pp. 129–143, 2019\.
  * [8] D. Ienco, R. Interdonato, R. Gaetano, and D. H. T. Minh, “Combining sentinel-1 and sentinel-2 satellite image time series for land cover mapping via a multi-source deep learning architecture,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 158, pp. 11–22, 2019.
  * [9] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in neural information processing systems_ , 2014, pp. 2672–2680.
  * [10] J. Masci, U. Meier, D. Cireşan, and J. Schmidhuber, “Stacked convolutional auto-encoders for hierarchical feature extraction,” in _International conference on artificial neural networks_. Springer, 2011, pp. 52–59.
  * [11] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [12] M. Gargiulo, D. A. Dell’Aglio, A. Iodice, D. Riccio, and G. Ruello, “Integration of sentinel-1 and sentinel-2 data for land cover mapping using w-net,” _Sensors_ , vol. 20, no. 10, p. 2969, 2020.
  * [13] S. Liu, Z. Qi, X. Li, and A. G.-O. Yeh, “Integration of convolutional neural networks and object-based post-classification refinement for land use and land cover mapping with optical and sar data,” _Remote Sensing_ , vol. 11, no. 6, p. 690, 2019.
  * [14] Y. Wu, G. Mu, C. Qin, Q. Miao, W. Ma, and X. Zhang, “Semi-supervised hyperspectral image classification via spatial-regulated self-training,” _Remote Sensing_ , vol. 12, no. 1, p. 159, 2020.
  * [15] P. S. S. Aydav and S. Minz, “Self-training with neighborhood information for the classification of remote sensing images,” in _Information and Communication Technology for Competitive Strategies_. Springer, 2019, pp. 465–474.
  * [16] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton, “Big self-supervised models are strong semi-supervised learners,” in _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020.
  * [17] S. M. Aswatha, J. Mukherjee, P. K. Biswas, and S. Aikat, “Unsupervised classification of land cover using multi-modal data from multi-spectral and hybrid-polarimetric sar imageries,” _International Journal of Remote Sensing_ , vol. 41, no. 14, pp. 5277–5304, 2020.
  * [18] J. Li, X. Huang, T. Hu, X. Jia, and J. A. Benediktsson, “A novel unsupervised sample collection method for urban land-cover mapping using landsat imagery,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 57, no. 6, pp. 3933–3951, 2019.
  * [19] P. Bachman, R. D. Hjelm, and W. Buchwalter, “Learning representations by maximizing mutual information across views,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 15 535–15 545.
  * [20] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [21] Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” _arXiv preprint arXiv:1906.05849_ , 2019.
  * [22] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio, “Learning deep representations by mutual information estimation and maximization,” _arXiv preprint arXiv:1808.06670_ , 2018.
  * [23] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, “Infogan: Interpretable representation learning by information maximizing generative adversarial nets,” in _Advances in neural information processing systems_ , 2016, pp. 2172–2180.
  * [24] L. Bruzzone, C. Conese, F. Maselli, and F. Roli, “Multisource classification of complex rural areas by statistical and neural-network approaches,” _Photogrammetric Engineering and Remote Sensing_ , vol. 63, no. 5, pp. 523–532, 1997.
  * [25] L. Bruzzone, D. F. Prieto, and S. B. Serpico, “A neural-statistical approach to multitemporal and multisource remote-sensing image classification,” _IEEE Transactions on Geoscience and remote Sensing_ , vol. 37, no. 3, pp. 1350–1359, 1999.
  * [26] M. J. Steinhausen, P. D. Wagner, B. Narasimhan, and B. Waske, “Combining sentinel-1 and sentinel-2 data for improved land use and land cover mapping of monsoon regions,” _International journal of applied earth observation and geoinformation_ , vol. 73, pp. 595–604, 2018.
  * [27] N. Kussul, M. Lavreniuk, S. Skakun, and A. Shelestov, “Deep learning classification of land cover and crop types using remote sensing data,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 14, no. 5, pp. 778–782, 2017.
  * [28] Q. Feng, J. Yang, D. Zhu, J. Liu, H. Guo, B. Bayartungalag, and B. Li, “Integrating multitemporal sentinel-1/2 data for coastal land cover classification using a multibranch convolutional neural network: A case of the yellow river delta,” _Remote Sensing_ , vol. 11, no. 9, p. 1006, 2019\.
  * [29] S. Wang, W. Chen, S. M. Xie, G. Azzari, and D. B. Lobell, “Weakly supervised deep learning for segmentation of remote sensing imagery,” _Remote Sensing_ , vol. 12, no. 2, p. 207, 2020.
  * [30] H. Hotelling, “Relations between two sets of variates,” in _Breakthroughs in statistics_. Springer, 1992, pp. 162–190.
  * [31] D. Amarsaikhan, H. Blotevogel, J. Van Genderen, M. Ganzorig, R. Gantuya, and B. Nergui, “Fusing high-resolution sar and optical imagery for improved urban land cover study and classification,” _International Journal of Image and Data Fusion_ , vol. 1, no. 1, pp. 83–97, 2010.
  * [32] J. Geng, H. Wang, J. Fan, and X. Ma, “Classification of fusing sar and multispectral image via deep bimodal autoencoders,” in _2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS)_. IEEE, 2017, pp. 823–826.
  * [33] A. A. Nielsen and R. Larsen, “Canonical analysis of sentinel-1 radar and sentinel-2 optical data,” in _Scandinavian Conference on Image Analysis_. Springer, 2017, pp. 147–158.
  * [34] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton, “Big self-supervised models are strong semi-supervised learners,” _arXiv preprint arXiv:2006.10029_ , 2020.
  * [35] Z. Zhang, Q. Liu, and Y. Wang, “Road extraction by deep residual u-net,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 15, no. 5, pp. 749–753, 2018.
  * [36] K. Krishna and M. N. Murty, “Genetic k-means algorithm,” _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ , vol. 29, no. 3, pp. 433–439, 1999.
  * [37] N. Yokoya, P. Ghamisi, R. Haensch, and M. Schmitt, “2020 ieee grss data fusion contest: Global land cover mapping with weak supervision [technical committees],” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 8, no. 1, pp. 154–157, 2020.
  * [38] M. Schmitt, J. Prexl, P. Ebel, L. Liebel, and X. X. Zhu, “Weakly supervised semantic segmentation of satellite images for land cover mapping–challenges and opportunities,” _arXiv preprint arXiv:2002.08254_ , 2020.
  * [39] Q. Yu, W. Liu, and J. Li, “Spatial resolution enhancement of land cover mapping using deep convolutional nets,” _The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences_ , vol. 43, pp. 85–89, 2020.


CleanAirNowKC full map displayed on loading the website.

# CleanAirNowKC: Building Community Power by Improving Data Accessibility

Rifat Ara Proma
University of South Florida  e-mail<EMAIL_ADDRESS>Matthew Sumpter
University of South Florida  e-mail<EMAIL_ADDRESS>Humberto Lugo
CleanAirNow  e-mail<EMAIL_ADDRESS>Elizabeth Friedman
Children’s Mercy e-mail<EMAIL_ADDRESS>Kansas City  Khandaker Tasnim Huq
University of South Florida  e-mail<EMAIL_ADDRESS>Paul Rosen
University of South Florida  e-mail<EMAIL_ADDRESS>
###### Abstract

As cities continue to grow globally, air pollution is increasing at an
alarming rate, causing a significant negative impact on public health. One way
to affect the negative impact is to regulate the producers of such pollution
through policy implementation and enforcement. CleanAirNowKC (CAN-KC) is an
environmental justice organization based in Kansas City (KC), Kansas. As part
of their organizational objectives, they have to date deployed nine PurpleAir
air quality sensors in different locations about which the community has
expressed concern. In this paper, we have implemented an interactive map that
can help the community members to monitor air quality efficiently. The system
also allows for reporting and tracking industrial emissions or toxic releases,
which will further help identify major contributors to pollution. These
resources can serve an important role as evidence that will assist in
advocating for community-driven just policies to improve the air quality
regulation in Kansas City.

Introduction

Rapid urbanization is occurring globally, and with it comes an increased
threat of air pollution [23]. Urbanization leads to more centralized
industries, including transportation, waste management, and energy production
[14]. Air pollution has been shown to have significant adverse effects on
public health, including an increased risk of asthma [18], cardiovascular
disease [6], and cancer [25], leading to an estimated healthcare cost of US$21
billion globally in 2015 [22]. Air pollution is an avoidable consequence of
urbanization; however, in order to regulate the main producers of pollution,
significant policy implementations and enforcement are required [14].

Air pollution policy can only succeed with a high level of community
engagement, and those most likely to become engaged are those who are most
directly impacted by pollution [17, 9]. A significant amount of research has
shown that the main producers of urban pollution are highly concentrated in
disadvantaged, poor, and minority communities [16]. Therefore, these
communities have the most potential to drive fruitful policy development if
they are engaged in the process effectively.

Community engagement regarding air pollution has historically been a challenge
due to the high cost of air monitoring technology [6]. However, the emergence
of low-cost sensors creates opportunities for community-based organizations to
collect data and perform community science [6]. Data by itself lacks the power
to inform communities, and thus data communication becomes a foundational
challenge in driving engagement. Interactive data visualization provides a
powerful outlet to present this data to these communities, empowering those
affected to advocate for themselves. Visualization becomes a tool that can
educate the people, allowing them to inform each other, which ultimately helps
inform decision-making. This develops a combination of knowledge, skills,
values, and motivation to make a difference and allows for robust civic
engagement and participation.

CleanAirNowKC (CAN-KC) is an environmental justice organization based in
Kansas City (KC), Kansas. Their goal is to work with communities threatened by
air pollution to increase their participation in policymaking decisions and
improve the health and quality of life in those communities. To date, CAN-KC
has deployed nine PurpleAir air quality sensors in different locations
throughout the community. The sites were chosen based on locations the
community participants deemed most concerning, with plans to add more.
Creating an engaging visualization that is both effective and uncluttered is a
non-trivial task, which is compounded by real-time data processing. This paper
details the implementation of an interactive map that will soon be deployed to
the CAN-KC website. To build the visualization, a combination of web
technologies (namely D3.js and Leaflet.js) are used in conjunction with the
PurpleAir API for data collection. We aim to design an interactive map to
visualize air quality sensor data that can facilitate the following tasks:

  1. 1.

Display real-time air quality data in these communities;

  2. 2.

Convey the impact of poor air quality on locals’ health;

  3. 3.

Visualizes the unequal burden of air pollution occurring in Kansas City fence
line communities; and

  4. 4.

Provide visualizations that encourage community engagement, civic
participation, community-led solutions in policymaking, and accountability of
polluting industries.

This work seeks to create a resource that will assist CAN-KC in organizing and
reporting data and engage Kansas City decision-makers. By creating this
resource, we hope that we can contribute to the efforts being made by CAN-KC
to improve the air quality policy of Kansas City and perhaps inspire other
municipalities to take similar action.

##  1 Background

###  1.1 Air Monitoring Technology

There have been numerous significant works on air monitoring technology in
recent years. Baldauf et al. [1] presented a study on highly time-resolved
characterization of traffic activity, meteorology, and air quality at varying
distances from the road. They used several air quality parameters that
represent the complex mixture of pollutants emitted by motor vehicles. Jiao et
al. [11] developed an integrated air and weather measurement system that was
more easily deployed in outdoor community spaces, minimizes operator
maintenance, and provides real-time, quality-checked data to the public. A
system called MAAV was developed by Moore et al. [19] to perform three
tasks—measure air quality, annotate data streams and visualize real-time
PM2.5111PM2.5 is a pollution measurement of the density of particulate with a
diameter of 2.5 micrometers or smaller. levels. The system used indoor and
outdoor air quality monitors to capture PM2.5 levels. To inspect the measured
data, they built a tablet-based interactive visualization.

Historically, air quality sensors were expensive. Technological advancement
has led to affordable air quality sensors, empowering individuals and
community organizations to monitor their local air quality. In this work, we
utilize the low-cost PurpleAir sensor [7]. PurpleAir uses laser particle
counters to measure air pollutants and transmit the data via WiFi to PurpleAir
servers.

###  1.2 Air Quality Visualization Systems

Several prior visualization tools have inspired our design.

In 2017, Lu et al. [15] introduced an interactive web map application, built
using web.py, Leaflet.js, and DS.js, that focused on the air quality of the
leading cities of China. The data were retrieved from the national real-time
air quality reporting system, which was reported hourly. Their observations
detected several locations in China that had alarming air quality throughout
the year.

Chen [4] used Google Earth and Keyhole Markup Language (KML) functionality to
produce a heatmap visualization based on air quality readings. The heatmap
ranged from blue (low concentration of pollutants) to red (high concentration
of pollutants). Users could click on points of interest, which triggered a
popup that displayed a 24-hour histogram of air quality, along with current
conditions and considerations that should be taken under those conditions.

Cleary et al. [5] created a web-based visualization, called Weave, that
personalized air quality by relating the data to users’ health, specifically
targeting health hazards caused by the air pollution-related to traffic for
members of the Boston Chinatown community. The tools used previously collected
data (wind speed, wind direction, temperature, and traffic volume) and a
statistical prediction model with interactively adjustable parameters to
visualize how traffic influenced PM2.5 concentrations.

Finally, Nurgazy et al. [21] extended the idea of data personalization by
developing user profiles that informed context-aware visualizations. By using
air quality data provided by the city of Melbourne, Australia, they developed
an air pollution map that would adjust the visualization based on the user’s
reported sensitivities, included user age, color vision impairments, and
sensitivity to pollutants. The website of a Swiss air quality technology
company IQAir [10] includes a map-based visualization of air quality of
different countries and regions of the world. Although it is capable of
conveying air quality related information to anyone who is interested, a
system like this does not provide a community level focus, therefore not
necessarily meeting the goals of any specific community.

###  1.3 Community Empowered Air Quality Monitoring

There have been several initiatives around the world to engage communities in
improving air quality. Carvlin et al. [3] developed a community-engaged
research study to provide real-time particulate matter (PM) air quality
information at a high spatial resolution in Imperial County, California. The
air quality network provided information on susceptible populations, assisted
in identifying air pollution hotspots, and increased community awareness of
air pollution. Hsu et al. [9] collaborated with Allegheny County CleanAirNow
community to develop an air monitoring system. Their focus was to help the
community gather enough scientific evidence to support policymaking. The
system built in this study was able to generate animated smoke images, air
quality data, and crowd-sourced smell reports. The result of their survey
suggested that the system provided scientific evidence that could work in
favor of the community in policymaking. However, the evaluation was done based
on a small number of participants, all well-educated.

Other approaches have tried engagement techniques through social networks and
gaming to improve air quality awareness. Niemeyer et al. [20] developed a
collective networked public air pollution sensor for use within a game context
for high school students in Los Angeles. The main goal of the game is to
transform students into agents of change by making real-world measurements of
air quality, tracking down the sources of pollutions, and exploring their
impact on the environment around them. Kim and Paulos [12] developed a tool,
“inAir,” for measuring and having visualizations of indoor air quality within
a social network. The study showed how simple visualizations of information
undetectable to humans, such as indoor particle counts, can play a significant
role in increasing people’s awareness and understanding of air quality.

Finally, Hooker et al. [8] developed an electronic street sign, called a
“Pollution e-Sign,” using a form of hacking called bluejacking, which exploits
weaknesses in Bluetooth. The system communicated local air quality information
to passing devices without their permission to improve awareness.

###  1.4 Kansas City Air Pollution

Kansas City includes low-income communities of color fenceline to industrial
sources of pollution, including the second-largest freight-car classification
rail yard in the United States [2]. With very limited accountability for
environmental enforcement, Kansas City faces cumulative exposure to air
pollutants.

CAN-KC and community members from Argentine, Kansas (a neighborhood in Kansas
City) collectively gathered air pollution data using low-cost sensors in
2015-2016 to address community air pollution concerns, specifically regarding
elemental carbon emitted by diesel engines used in the rail yard [24]. One-
third of the air monitors measuring elemental carbon in November 2013 were
reported to be above a level associated with short-term health risks. The
community’s concern was enough to spark a one-year EPA study called KCTRAQS
[13]. The analyses conducted by the EPA are still being released. In
partnership with CAN-KC, local academics are using the EPA KC-Traq air quality
data to assess for a correlation between local air quality and health.

This particular project began in 2020, when members of the Kansas City-based
environmental justice organization, CAN-KC, were connected to researchers at
the University of South Florida through the Union of Concerned Scientists.

##  2 Methods

Our overall design goal was to provide Kansas City community members with
access to current air quality information (D1), as well as information about
the health impact of bad air quality (D2). Additionally, we provide access to
recent historical (D3) air quality data, including possible sources of
pollution, to help to draw the attention of the policymakers to take necessary
steps for improvement. In addition, we wanted to give community members the
opportunity to actively participate in the process by reporting first-hand
pollution information (D4), e.g., industrial emissions, toxic releases, etc.
Finally, we wanted the interactive map to enable CAN-KC members to gather
evidence of pollution to guide decision-making (D5).

###  2.1 Air Quality Data Collection

The data to be visualized is reported by numerous PurpleAir air quality
sensors that the CAN-KC, in partnership with community members, placed around
Kansas City. These sensors are connected via WiFi, and the sensor readings are
made available through the PurpleAir API. The request frequency limit is
relatively low (max $1\ req./min.$). To properly provide pseudo-live readings,
front-end requests are routed through a python-based proxy server, which
refreshes the data from PurpleAir every 10 minutes. For development and
evaluation, we captured a static dataset consisting of readings from 8 live
PurpleAir sensors over 2 days.

Figure 1: The info card is displayed when a sensor is clicked on.

###  2.2 Air Quality Map

The main feature of this data visualization is the map (see CleanAirNowKC:
Building Community Power by Improving Data Accessibility). Using Leaflet.js,
we create the map with MapBox tiles and OpenStreetMaps data as the primary
users are local residents evaluating their surroundings in Kansas City.
Therefore, a street-level map is the most helpful layer set. To summarize the
data available and provide a central location for map interaction and
interpretation, we have included a small overlay in the bottom left corner of
the map. This overlay offers several functionalities. First, it allows the
user to select which sensor data is displayed on the map (D1). Additionally, a
user may choose a specific sensor by name, which will zoom the map to the
location of that sensor. A “Find Me!” button focuses the map on the user’s
location. Finally, it provides the ability to submit a report of transient air
quality and other pollution events (D4).

###  2.3 Air Quality Glyphs

To provide current air quality information (D1), the most recent data for each
sensor is represented with a circle marker at the reported sensor latitude and
longitude. Additionally, the most recent PM2.5 reading is used to fill the
marker using a linear color scale (using a stoplight metaphor—green to yellow
to red) shown in the lower right corner of the map display. A numeric
representation is also appended to the center. This approach allows for both a
cursory visual inspection of the map and specific real-time reading. For
recent historical data (D3), different colored rings are placed around each
marker that indicates the PM2.5 reading over different intervals. With the map
fully rendered, users can zoom in and pan to explore the other parts of the
city.

###  2.4 Sensor Popups

####  2.4.1 Info Card

An information popup (see Figure 1) appears when the sensor marker is clicked.
This information card gives the user an overall summary of the selected
sensor. It shows basic information, including date, the value of the chosen
metric (PM2.5_10 minute by default), what the Air Quality Index (AQI)222Air
Quality Index is a nationally uniform index for reporting air quality
developed and promoted by the US Environmental Protection Agency (EPA). value
means for health and safety concerns (D2), and the sensor name. It also shows
the average AQI value over different time intervals (D3).

Figure 2: The secondary popup line chart view.

####  2.4.2 Line Chart

A button in the bottom right corner of the sensor popup (see Figure 1) lets
the user switch to a secondary historical line chart view (D3). This view
allows power users to access additional details. The line chart, generated
using PlotlyJS, provides an overview of the sensor readings for the selected
metric (default PM2.5_10Minutes) over time (see Figure 2). The background of
the line chart is filled with the corresponding air quality color scale to
enable visual correlation to the health risk level of that pollution (D2). The
line chart is interactive, offering the ability to adjust the scales, zoom in
with a bounding box, and tooltips that reveal specific air quality data
corresponding to their mouse position. After adjusting the view, there is also
an option to save the graph as an image, providing a method for gathering and
sharing information for community engagement (D5).

###  2.5 Hazardous Waste Facilities

Figure 3: The map with hazardous waste layer and the popup displayed for
hazardous waste sites (upper right).

Superfund and other hazardous waste sites are included as a third layer on the
map (off by default to reduce information overload, see Figure 3), sourced
from the EPA333https://enviro.epa.gov/facts/rcrainfo/search.html. There are
numerous hazardous waste sites in Kansas City; individual points are clustered
into markers to designate a larger area. Upon zooming in, the clusters are
dismissed in favor of the localized hazardous waste sites. Users may click on
individual markers (Figure 3 upper right) to see the name of the primary
contact of the waste site and its address, as well as a link to the EPA entry
on that location (which details the type of waste emitted and local contacts
regarding disposal). This information serves two purposes: first, it allows
Kansas City residents to correlate their localized air quality with its
potential contributors (D1), and second, it guides CAN-KC organizers regarding
where new sensors may be best placed (D5).

##  3 Evaluation

The visualization has been designed as a community engagement tool. The main
objective of this visualization is to stimulate community engagement and
community-led solutions in policymaking. As the primary users of this
visualization are community members and the CAN-KC organizers, we asked them
to use the tool and subsequently complete a survey to evaluate the
effectiveness of this tool. We received 20 responses from CAN-KC organizers
and members. The ages of these community members range between 18-84 years, 9
female, 11 male. Of the survey participants, 1 was involved in the website
development process and is a co-author of this paper.

The evaluation was designed to probe 3 main questions. The first section
focused on identifying how participants accessed the map to understand if the
tool works properly on different machines and browsers. The second section
asked the users to perform specific tasks to gauge the intuitive usability of
the map. The final section evaluates the overall user experience. The overall
feedback from the participants was very positive.

##### Access Method

Most participants used a computer to access the map, but a few people used
mobile phones as well. Participants using a computer did not face any
technical issues. However, on mobile devices, some user interaction issues
were identified, e.g., overlapping components, components too large for the
screen, etc.

##### Task Completion

Most users were able to access the map and find their location on the map with
ease and retrieve information regarding the air quality impact on health.
However, 5 participants found it extremely difficult to locate themselves.
This indicates that (D1) and (D2) were largely accomplished. When the
participants were asked about the different colored rings around the sensor
markers, 12 participants were unclear what it indicates. This feedback
suggests that (D3) was not fully satisfied, and we either need to redesign our
glyph or provide an explanation of it on the map.

##### Overall Experience

When the participants were asked if the map could effectively communicate air
pollution data in Kansas City, most agreed. The overview of the feedback for
this question is showed in Figure 4 left. This result is an indirect
indication that the system would be useful for (D5). Participants responded
positively when asked if they would use this map to monitor the air quality.
Since one important aspect of (D4) was that we wanted to increase community
engagement and participation, the responses we received indicated that the
system could attain that. We also asked the participants how often they would
access the map if a mobile application were available to monitor the air
quality. Although the response rate was similar to the desktop version, 2
participants reported they would never use the mobile app to check the map.
The contrast of the responses regarding the frequency of usage is shown in
Figure 4 right. When they were asked if they believed that the Kansas City
community would benefit from having this map available online, 16 participants
strongly agreed and 1 participant somewhat agreed. In the survey, 2 community
members mentioned that they had trouble understanding what different intervals
of data collection mean, suggesting either an explanation should be provided
or that this feature is too technical for a general audience.

Figure 4: Left: Feedback regarding the effectiveness of the map at
communicating air pollution data in Kansas City. Right: Feedback regarding the
frequency of using the map.

##  4 Conclusion & Future Work

In this paper, we detail an interactive map designed in partnership with the
environmental justice non-profit CAN-KC. The goal of the map is two-fold: to
interactively display local air pollution data and inform the community
members about air pollution in their community. The map displays information
regarding the current and historical air quality values read from PurpleAir
sensors and provides an interpretation of what health actions should be taken.
Additionally, users can monitor and identify industrial pollution sources. We
will make the website live in the near future. Use of this map, in unison with
educational events and attendance to city-directed meetings, can serve to
improve community engagement and involvement in local policy. Once these
resources are available to the general public, we will then be able to
evaluate how successful our visualization is at informing and engaging the
community.

One challenge of working with the community to develop a tool was that most of
them are unfamiliar with any technical aspect of the system. It made it
challenging to elicit requirements, but involving CAN-KC organizers in the
development process was critical to understanding the requirements before
making the design decisions.

Based on the feedback we have received, we plan to update the CAN-KC website
with better explanations and easier-to-understand and more user-friendly
visualizations of data collection intervals, industrial pollution sources, and
the meaning of AQI. We also plan on making the map more suitable for mobile
devices and fix the existing user interface issues. We want to improve the
functionality of pollution reporting for users. Having a mobile app version of
this map, we also believe, would make submitting reports easier.

We believe this will increase community engagement and also help to collect
evidence against major contributors to pollution that can play a significant
role in policymaking.

###### Acknowledgements.

This work was partially supported by NIH-5R34HL145442-02.

## References

  * [1] R. Baldauf, E. Thoma, M. Hays, R. Shores, J. Kinsey, B. Gullett, S. Kimbrough, V. Isakov, T. Long, R. Snow, et al.  Traffic and meteorological impacts on near-road air quality: Summary of methods and trends from the raleigh near-road study.  Journal of the Air & Waste Management Association, 58(7):865–878, 2008. doi: 10 . 3155/1047-3289 . 58 . 7 . 865
  * [2] C. Bodenner.  Orbital view: A massive rail yard.  The Atlantic, 2015.  https://www.theatlantic.com/notes/2015/12/orbital-view-a/420192/.
  * [3] G. Carvlin, H. Lugo, L. Olmedo, E. Bejarano, A. Wilkie, D. Meltzer, M. Wong, G. King, A. Northcross, M. Jerrett, et al.  Development and field validation of a community-engaged particulate matter air quality monitoring network in imperial, california, usa.  Journal of the Air & Waste Management Association, 67(12):1342–1352, 2017. doi: 10 . 1080/10962247 . 2017 . 1369471
  * [4] P. Chen.  Visualization of real-time monitoring datagraphic of urban environmental quality.  Eurasip Journal on Image and Video Processing, 2019(1):42, 2019\. doi: 10 . 1186/s13640-019-0443-6
  * [5] E. G. Cleary, A. Patton, H.-C. Wu, A. Xie, J. Stubblefield, W. Mass, G. Grinstein, S. Koch-Weser, D. Brugge, and C. Wong.  Making air pollution visible: A tool for promoting environmental health literacy.  JMIR Public Health and Surveillance, 3(2):e16, 2017. doi: 10 . 2196/publichealth . 7492
  * [6] A. Commodore, S. Wilson, O. Muhammad, E. Svendsen, and J. Pearce.  Community-based participatory research for the study of air pollution: a review of motivations, approaches, and outcomes.  Environmental Monitoring and Assessment, 189(8):1–30, 2017. doi: 10 . 1007/s10661-017-6063-7
  * [7] B. Feenstra, V. Papapostolou, S. Hasheminassab, H. Zhang, B. Der Boghossian, D. Cocker, and A. Polidori.  Performance evaluation of twelve low-cost pm2.5 sensors at an ambient air monitoring site.  Atmospheric Environment, 216:116946, 2019. doi: 10 . 1016/j . atmosenv . 2019 . 116946
  * [8] B. Hooker, W. Gaver, A. Steed, and J. Bowers.  The pollution e-sign.  In Workshop on Ubiquitous Sustainability (Ubicomp), 2007.
  * [9] Y.-C. Hsu, P. Dille, J. Cross, B. Dias, R. Sargent, and I. Nourbakhsh.  Community-empowered air quality monitoring system.  In ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 1607–1619, 2017. doi: 10 . 1145/3025453 . 3025853
  * [10] IQAir.  https://www.iqair.com.  Accessed: August 25, 2021.
  * [11] W. Jiao, G. S. Hagler, R. Williams, R. N. Sharpe, L. Weinstock, and J. Rice.  Field assessment of the village green project: An autonomous community air quality monitoring system.  Environmental Science & Technology, 49(10):6085–6092, 2015. doi: 10 . 1021/acs . est . 5b01245
  * [12] S. Kim and E. Paulos.  Inair: sharing indoor air quality measurements and visualizations.  In ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 1861–1870, 2010. doi: 10 . 1145/1753326 . 1753605
  * [13] S. Kimbrough, S. Krabbe, R. Baldauf, T. Barzyk, M. Brown, S. Brown, C. Croghan, M. Davis, P. Deshmukh, R. Duvall, et al.  The kansas city transportation and local-scale air quality study (kc-traqs): integration of low-cost sensors and reference grade monitoring in a complex metropolitan area. part 1: overview of the project.  Chemosensors, 7(2):26, 2019. doi: 10 . 3390/chemosensors7020026
  * [14] P. Landrigan.  Air pollution and health, 2017. doi: 10 . 1016/S2468-2667(16)30023-8
  * [15] W. Lu, T. Ai, X. Zhang, and Y. He.  An interactive web mapping visualization of urban air quality monitoring data of china.  Atmosphere, 8(8):148, 2017. doi: 10 . 3390/atmos8080148
  * [16] M. Lubell, A. Vedlitz, S. Zahran, and L. Alston.  Collective Action, Environmental Activism, and Air Quality Policy.  Political Research Quarterly, 59(1):149–160, 2006. doi: 10 . 1177/106591290605900113
  * [17] S. Mahajan, P. Kumar, J. A. Pinto, A. Riccetti, K. Schaaf, G. Camprodon, V. Smári, A. Passani, and G. Forino.  A citizen science approach for enhancing public understanding of air pollution.  Sustainable Cities and Society, 52:101800, 2020. doi: 10 . 1016/j . scs . 2019 . 101800
  * [18] S. Martenies and S. Batterman.  Effectiveness of using enhanced filters in schools and homes to reduce indoor exposures to pm2.5 from outdoor sources and subsequent health benefits for children with asthma.  Environmental Science & Technology, 52(18):10767–10776, 2018. doi: 10 . 1021/acs . est . 8b02053 . s001
  * [19] J. Moore, P. Goffin, M. Meyer, P. Lundrigan, N. Patwari, K. Sward, and J. Wiese.  Managing in-home environments through sensing, annotating, and visualizing air quality data.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(3):1–28, 2018. doi: 10 . 1145/3264938
  * [20] G. Niemeyer, A. Garcia, and R. Naima.  Black cloud: patterns towards da future.  In ACM International Conference on Multimedia, pp. 1073–1082, 2009\. doi: 10 . 1145/1631272 . 1631514
  * [21] M. Nurgazy, A. Zaslavsky, P. P. Jayaraman, S. Kubler, K. Mitra, and S. Saguna.  CAVisAP: Context-Aware Visualization of Outdoor Air Pollution with IoT Platforms.  In International Conference on High Performance Computing and Simulation (HPCS), pp. 84–91. Institute of Electrical and Electronics Engineers Inc., 2019. doi: 10 . 1109/HPCS48598 . 2019 . 9188167
  * [22] OECD.  The Economic Consequences of Outdoor Air Pollution.  2016\. doi: https://doi . org/10 . 1787/9789264257474-en
  * [23] D. D. Parrish and W. Stockwell.  Urbanization and air pollution: Then and now.  96, 2015. doi: 10 . 1029/2015EO021803
  * [24] Staff.  Kansas city star calls for bnsf and epa to act on citizen report of diesel exhaust air pollution from the argentine rail yard.  CleanAirNow Kansas City, 2015.  http://cankc.org/2015/07/kansas-city-star-calls-for-bnsf-and-epa-to-take-action-regarding-air-pollution-from-the-argentine-rail-yard/.
  * [25] M. Turner, D. Krewski, W. R. Diver, C. A. Pope III, R. Burnett, M. Jerrett, J. Marshall, and S. Gapstur.  Ambient air pollution and cancer mortality in the cancer prevention study ii.  Environmental Health Perspectives, 125(8):087013, 2017. doi: 10 . 1289/EHP1249