declare autowrapped toks=pic me, pic dir tree/.style=for
tree=folder,font=,grow’=0,, before typesetting nodes=for tree=edge
label+/.option=pic me,,, , pic me set/.code n args=2#1/.style=inner xsep=2pic
me=pic #2, , pic me set=directoryfolder, pic me set=filefile,

|  |
---|---|---
[5pt] |  | FLAIR: French Land cover from Aerospace ImageRy.
|  | \- Challenge FLAIR #1: semantic segmentation and domain adaptation -
|  | Anatol Garioud, Stéphane Peillet, Eva Bookjans, Sébastien Giordano, Boris Wattrelos
|  | Institut national de l’information géographique et forestière (IGN), France
|  |<EMAIL_ADDRESS>

## Dataset overview

Figures
---

➔ 20,293,091,328 pixels annotated at 0.20 m spatial resolution
---
➔ 77,412 patches (512$\times$512)
➔ 50 spatio-temporal domains and 950 areas covering 810 km²
➔ 13 semantic classes (+6 optional ones)

Structure
---

pic dir tree, where level=0directory,, for tree= s sep=0.05cm, l sep=1cm,
font= [Dataset [train\val [domain _ year [area [img [IMG_ID.tif, file]] [msk
[MSK_ID.tif, file]] ] ] ] [test [domain_year [area [img [IMG_ID.tif, file]] ]
] ] [metadata.json, file] ]
---

## Context

The French National Institute of Geographical and Forest Information (IGN)[1]
has the mission to document and measure land-cover on French territory and
provides referential geographical datasets, including high-resolution aerial
images and topographic maps. The monitoring of land-cover plays a crucial role
in land management and planning initiatives, which can have significant socio-
economic and environmental impact. Together with remote sensing technologies,
artificial intelligence (IA) promises to become a powerful tool in determining
land-cover and its evolution.

IGN is currently exploring the potential of IA in the production of high-
resolution land cover maps. Notably, deep learning methods are employed to
obtain a semantic segmentation of aerial images. However, territories as large
as France imply heterogeneous contexts: variations in landscapes and image
acquisition make it challenging to provide uniform, reliable and accurate
results across all of France.

The FLAIR-one dataset presented is part of the dataset currently used at IGN
to establish the French national land cover map reference Occupation du sol à
grande échelle (OCS-GE).

## Domain adaptation challenge

The aerial surveys covering the French territory at very high spatial
resolution raise substantial domain adaptation challenges:

Large-scale coverage: vast territories the size of a country, such as France,
include varied landscapes and climates. Consequently, a single semantic class
encompasses very heterogeneous characteristics. For example, agricultural
crops grown in Northern France are different, in terms of species and
phenology, to those in Southern France. Likewise, forest species composition
varies significantly between coastal and mountainous areas. Besides obviously
affecting vegetation land cover classes, large-scale discrepancies in land
cover characteristics affect almost all semantic classes to some extent: the
roofing depends on regional building practices and therefore can be
dissimilar; the color of bare soils varies widely depending on the soil type
and so on.

Temporal shift: images within a particular aerial survey are acquired between
April and November. Therefore a given area can include multiple seasons. This
particularity greatly affects the appearance of the vegetation land cover
classes. For instance, wheat crops are pictured at different stages of growth:
from bare soil to low and fully developed plants back to bare soil after
harvest. Significant seasonal changes also occur in deciduous trees or
forested areas.

Multi-sensor acquisition: different cameras are used for the aerial image
acquisitions. This implies different sensors and consequently, different image
characteristics. For example, even if the final product is delivered at a
specified target spatial resolution (i.e. 0.2 m), depending on the camera, the
actual spatial resolution can vary slightly. The same occurs with radiometric
information, as the spectral sensitivity (bandwidth, center value, etc.) of
the channels (blue, green, red, near-infrared) is different among the used
cameras.

Radiometric processing: within a spatial domain (see next section for a
definition), all aerial acquisitions are radiometrically corrected to reduce
disparities in sunlight and contrast. Nonetheless, this homogenization is not
applied across all the different spatial domains (Figure 1) equally. As
opposed to satellite imagery, the pixel intensity in the image channels can
therefore not be considered as a physical measure.

Figure 1: Radiometric discrepancies of the aerial images between areas.The 3
channels image displayed is a composite of Near-Infrared, Red and Green
spectral information.

In summary, spatial and temporal heterogeneity both in semantic and
radiometric terms make it challenging for generalizations of learned models on
the FLAIR-one dataset.

## Spatial and temporal domains definition

Spatial domains: a spatial domain is equivalent to a ”département” and is
therefore determined by administrative boundaries. Within each spatial domain,
areas of similar sizes have been defined. These areas are subdivided in
patches, which are the same size across the dataset (Figure 2). The total
number of patches within a spatial domain varies only slightly, ensuring a
balanced sampling per spatial domain.

Temporal domains: covering large scale territories with aerial imagery takes
time. Approximately three years are needed to cover the entire metropolitan
French territory. As previously stated, the aerial images are usually acquired
between the months of April and November depending on the availability of
IGN’s planes and on the cloud cover, to ensure that resulting images are
cloudless. Consequently, acquisitions within a single spatial domain are
sometimes taken on several dates, possibly differing by several months thus
encompassing different temporal domains. Furthermore, even within an area of a
spatial domain the acquisitions date may vary.

Figure 2: Spatial domains, areas and patches.

## Dataset extent

The dataset includes $50$ spatial domains (Figure 3) representing the
different landscapes and climates of metropolitan France. The train dataset
constitutes $\nicefrac{{4}}{{5}}$ of the spatial domains (40) while the
remaining $\nicefrac{{1}}{{5}}$ domains (10) are kept for testing. Within each
spatial domain several areas have been selected for a total of approximately
18 km² resulting in a total of $951$ areas spread across the French
metropolitan territory.

Figure 3: The 50 spatial domains of the dataset and the train/test split.

The selection of spatial domains takes into account the major landscapes
(coastal, urban, agricultural, rural, mountainous,…), the diversity of
semantic classes and the variations in radiometry across domains. As a result,
the 50 spatial domains can be considered to be representative of the land-
cover diversity found in metropolitan France. Additionally, the spatial
domains have been selected to include the different temporal domains
characteristic of large-scale aerial imagery. An effort was made to balance
the acquisitions dates in terms of months (Figure 4).

Figure 4: Aerial imagery acquisition months per spatial domain.

A perfect balance, however, is not possible given that certain months have a
greater probability to have a cloudless day than others and therefore are
naturally more heavily represented.

## Data sources

An aerial survey is a collection of individual aerial images covering an
entire spatial domain, whereby the individual aerial images overlap each other
in such a way that 3D information can be extracted. An individual image covers
only a small portion of a spatial domain; approximately 3,000 images are
needed to cover a medium-sized spatial domain. Each image is characterized by
a unique time and date of acquisition and the properties of the camera used.

Two types of aerial cameras were used in the FLAIR-one dataset containing 50
spatial domains: the Ultracam Eagle M3 camera from Vexcel Imaging and the
CAMv2 camera from IGN [2]. Both types of cameras are composed of 4
multispectral (MS) channels and a panchromatic channel (PAN). The MS channels
correspond to 3 spectral bands in the visible part of the electromagnetic
spectrum: Blue (B), Green (G), Red (R) and a spectral band in the near
infrared (NIR). The PAN channel is a unique channel covering the entire
visible spectrum. The PAN channel has a higher spatial resolution than the MS
channels. To obtain a similar spatial resolution, pan-sharpening techniques
are used combining R, G, B and PAN channels. The NIR channel is artificially
oversampled to match the PAN spatial resolution using traditional
interpolation techniques. We note that the resolution of the MS channels
varies between the two types of cameras used: for the Ultracam Eagle M3 camera
pixel, the spatial resolution of the MS channels is lower by a factor of three
compared to the PAN channel resolution, whereas for the CAMv2 camera it is
lower by a factor of two. As a result, the actual spatial resolution of the MS
channels can slightly differ depending on the camera type used.

The FLAIR-one dataset is based on 3 data sources: ORTHO HR® imagery, the
Digital Surface Model (DSM) and the RGE ALTI Digital Terrain Model (DTM). The
ORTHO HR® and DSM sources are generated from the aerial survey of the domain,
whereas the DTM is computed with auxiliary sources.

ORTHO HR®: the ORTHO HR® imagery is a mosaic of all the individual images
taken during an aerial survey and mapped onto a cartographic coordinate
reference system. The individual images are projected to the RGE ALTI DTM,
which provides solely the altitude of the ground. Since the height of the
buildings and vegetation (e.g., trees) is not taken into consideration, these
objects can appear to lean on the ORTHO HR® images depending on their position
in the image. The final ORTHO HR® product has a spatial resolution of 0.20 m
(R, G, B and NIR channels). Additionally, some radiometric processing methods
are applied to obtain the final product. First, radiometric equalization
methods are applied to the individual images. Then, a global radiometric
correction is carried out on the merged images covering an entire spatial
domain to provide a more satisfying color balance between channels.
Consequently, the shift in radiometry observable between spatial domains
(Figure 1) is due to both the date of acquisition and the specific radiometric
corrections applied. Therefore, the radiometry of R, G, B and NIR images of
the ORTHO HR® product cannot be considered as a physical measurement of
channel reflectance.

Digital Surface Model (DSM): the DSM gives the altitude, in meters, for each
pixel. Thanks to dense matching techniques, the DSM is derived from the same
aerial survey that is used to produce the ORTHO HR®. This gives the DSM the
same spatial resolution as the ORTHO HR® (0.20 m), but more importantly
prevents temporal shifts and ground cover changes between the two products
making them temporally coherent. However, there still are small geometric
differences between the two products. Buildings and vegetation objects
contained in the DSM do not lean as it is the case in the ORTHO HR® product,
possibly resulting in small shifts, which are more visible for high altitude
objects. Moreover, dense matching techniques are applied automatically,
potentially introducing noise and artifacts. In particular, radiometrically
homogeneous areas (e.g., parts of the aerial image with little texture) tend
to lead to locally false and varying 3D information. Apart from these
artifacts, the vertical accuracy of the DSM is considered to be twice the
spatial resolution (0.4 m).

RGE ALTI Digital Terrain Model (DTM): this product is a national DTM available
at a spatial resolution of 1 m. It is constructed from different sources such
as dense matching of aerial images, airborne Lidar or, for mountainous areas,
from airborne Synthetic Aperture Radar acquisitions. Depending on the source,
the vertical accuracy of the RGE ALTI DTM can vary between 0.3 m and 7 m.

Figure 5: Example of input and supervision data: true color composition, near-
infrared color composition, elevation band and supervision masks. The data
from the first three columns are retrieved from the IMG files while the last
column corresponds to the MSK files.

## Data description and naming conventions

The dataset is composed of 77,412 patches for a total of 20,319,305,628 pixels
covering approximately 810 km². Each patch is 512$\times$512 pixels
corresponding to 10,485.76 m2 on the ground. Preprocessing of the dataset was
mainly carried out with the odeon-landcover library [3].

The input patches (IMG) have 5 channels (5$\times$512$\times$512)
corresponding to: Blue, Green, Red, Near-Infrared and Elevation channels. The
first 4 channels are retrieved from the ORTHO HR® aerial imagery and are
encoded as 8bit unsigned integer datatype. The fifth channel is the difference
between the Digital Surface Model (DSM) and the Digital Terrain Model (DTM)
providing information on the above-ground height of the imaged objects. For
storage optimization reasons, this elevation information is multiplied by a
factor of 5 and encoded as a 8bit unsigned integer datatype.

The supervision patches (MSK) have 1 channel (1$\times$512$\times$512),
describing the semantic class of the pixel (values between 1 and 19) and is
encoded as a 8bit unsigned integer datatype.

Metadata (MTD): provides per patch additional information:

  * ➔

the acquisition date (YYYY-MM-DD) and time (HH:MM:SS) of the aerial image,
allowing the integration of temporal information (e.g., seasons).

  * ➔

the geographical coordinates (XY) of the centroid and the mean altitude (Z) of
the patch, providing spatial information on the relative location of the
patches to each other on a broader scale.

  * ➔

the camera type used for the aerial image acquisition, providing further
information on the data.

The data structure and naming convention is illustrated in the data overview
given above. The filename of the data starts with the type of data (IMG or
MSK) followed by ‘_ID’, whereby each pair of IMG-MSK pair has a unique ID
(e.g., 000845). IDs from 000001 to 061713 correspond to the train dataset and
those from 061714 to 077413 to the test set. Examples of IMG-MSK pairs are
provided in Figure 5. Finally, we note that the final two letters of the area
folder name indicate the two major types of land cover in the area. The letter
U indicates an urban, N a natural, A an agricultural, and F a forest area.

## Label production and semantic classes

The thematic application of the dataset consists in determining the land cover
at the pixel-level, (i.e., measuring how much part of a territory is covered
by buildings, forest and so on), enabling land management authorities to
better understand the current landscape and its evolution.

The supervision data (MSK) is based on photo-interpretation of the ORTHO HR®
aerial imagery and has been manually produced by experts following a call for
tenders from IGN. An initial spatial multi-level image segmentation approach
using PYRAM [4] was applied, simplifying the labeling at the cluster level. We
note that this segmentation was not necessarily final, but was modified
interactively when deemed appropriate. It was specified that movable objects
(e.g., cars, boats) are not to be annotated as such, but to be classified as
the underlying cover. For example, a car on an asphalt road is labeled as an
impervious surface.

Original nomenclature: The dataset has an original semantic richness of 18
common land cover classes and a class other. The class other corresponds to
pixels that could not be assigned to one of the 18 well-defined semantic
classes with certainty by the photo-interpreters, due to, for example,
radiometry issues or visual obstruction caused by the angle of incidence of
acquisition.

| Class | MSK | Pixels | %
---|---|---|---|---
| building | 1 | 1,670,300,028 | 8.23
| pervious surface | 2 | 1,636,681,162 | 8.07
| impervious surface | 3 | 2,836,695,330 | 13.98
| bare soil | 4 | 741,261,583 | 3.65
| water | 5 | 1,034,960,677 | 5.10
| coniferous | 6 | 540,883,511 | 2.67
| deciduous | 7 | 3,061,129,298 | 15.08
| brushwood | 8 | 1,408,504,416 | 6.94
| vineyard | 9 | 665,099,230 | 3.28
| herbaceous vegetation | 10 | 3,798,549,882 | 18.72
| agricultural land | 11 | 2,061,788,310 | 10.16
| plowed land | 12 | 720,090,325 | 3.55
| other | >13 | 117,147,576 | 0.58
TABLE I: Semantic classes of the main nomenclature of the flair-one dataset.

Class description Note: as previously stated, semantic classes are assigned on
the cluster level. In a given aerial image, only observable objects are
labeled, whereby temporal aspects are not taken into consideration.
Anthropized surfaces without vegetation (1, 2, 3, 13 and 18) Class 1 –
building includes not only buildings, but also other type of constructions
such as towers, agricultural silos, water towers and dams. Greenhouses (class
18) are an exception. Class 2 – pervious surface defined as man-made bare
soils covered with mineral materials (e.g. gravel, loose stones) and
considered to be pervious. It includes pervious transport networks (e.g.
gravel pathways, railways), quarries, landfills, building sites and coastal
ripraps. Class 3 – impervious surface is defined as man-made bare soils that
are impervious due to their building materials (e.g. concrete, asphalt,
cobblestones). It includes roadways, parking lots, and certain types of sports
fields. Class 13 – swimming pool is defined as man-made artificial (open-air)
swimming pools. It is not included in class 5 (water). Class 18 – greenhouse
although it can be considered as a building, is given a distinct labeled.
Greenhouses are a class of their own and are not part of class 1. Natural
areas without vegetation (4, 5 and 14) Class 4 – bare soil defined as natural
permanently bare soils. These natural soils remain without vegetation
throughout the year and generally are covered with sand, pebbles, rocks or
stones. Examples of natural bare soils are frequently found in coastal,
mountainous and forested areas. Class 5 – water is defined as areas covered by
water, such as sea, rivers, lakes and ponds. An exception are swimming pools
(class 13). Class 14 – snow refers to surfaces covered by snow. It is an
extremely rare class as the images are taken in the summertime and only very
few regions in France are covered with snow year-round. Woody natural
vegetation surfaces (6, 7, 8, 15, 16 and 17) Class 6 – coniferous, is defined
as trees identifiable as coniferous (pines, firs, cedars, cypress trees, …)
and taller than 5 m. Class 7 – deciduous is defined as trees identifiable as
deciduous (oaks, beeches, birches, chestnuts, poplars, …) and taller than 5 m.
Class 8 – brushwood refers to natural woody surfaces with a vegetation less
than 5 m high. It includes short and young trees, brushwood, shrublands,
mountain moors and abandoned agricultural lands. Class 15 – clear-cut, is
defined as forest areas, in which the trees have been cut down and harvested.
Class 16 – ligneous is an extremely rare class used to describe forest areas
with a homogeneous representation of either coniferous or deciduous trees.
Class 17 – mixed is an extremely rare class used to describe forest areas with
heterogeneous trees for which the types of trees (coniferous/ deciduous)
cannot be determined with sufficient certainty. Agricultural surfaces (9, 11
and 12) Class 9 – vineyard despite being an agricultural use of the land, are
assigned a class apart, a reason being their rather distinctive land cover
characteristics. Class 11 – agricultural land encompasses various different
agricultural classes. For example, besides major crops, it also includes
permanent and temporary grasslands with agricultural use. Vineyards (class 9)
are not included in this class. Class 12 – plowed land is defined as
agricultural land with no visible vegetation (e.g. recently plowed and freshly
harvested land). Herbaceous surfaces (10) Class 10 – herbaceous vegetation
defines herbaceous surfaces that are not intensively exploited for agriculture
purposes. This class includes ornamental lawns (e.g. gardens, public parks),
recreational fields (e.g. used for sport), natural herbaceous areas in
forested or mountainous areas, non-cultivated grass in agricultural areas or
along transportation networks.

Reduced baseline nomenclature: Among the 18 classes, some are very strongly
under-represented. We therefore propose to re-group the following 6 least
common classes together with the original class other (value 19): swimming
pool (value 13); snow (value 14); clear cut (value 15); mixed (value 16);
ligneous (value 17); greenhouse (value 18). Users of the dataset are
encouraged to use this reduced nomenclature instead of the original one, in
order to avoid being confronted with a very imbalanced segmentation task and
to instead focus on the domain adaptation challenge. The baseline code made
available includes an automatic remapping of the MSK files during dataloading.
If independent code is being developed, this reduction of classes should be
integrated.

It was nonetheless decided to share the MSK files with the full nomenclature
(19 classes) to allow a full exploration of the available information. The
proposed reduced nomenclature used for the benchmark has 13 land cover
classes, presented in Table I.

## Semantic class frequency

The class distribution of the train and test datasets of FLAIR-one are
presented in Figure 6 in percentages.

Figure 6: Class distribution of the train dataset (top) and test dataset
(bottom).

The train and test datasets have similar semantic class distributions. In both
cases, herbaceous vegetation, deciduous trees and impervious surfaces dominate
in contrast to bare soil, coniferous, vineyard and plowed land, which in total
account for less than 5% of the landcover in the datasets. It should be noted
that the class other only represents 0.65% of the train dataset and 0.29% for
the test dataset, even after regrouping the 6 least common classes of the full
nomenclature with the class other.

Figure 7: U-Net architecture used for the baseline, corresponding to the one
implemented in the segmentation-models-pytorch library. IMG = input image; MTD
= input metadata; PRED = prediction output. One potential and traditional
approach to integrate the metadata would be to add a Multi-layer Perceptron
for encoding and add the output to the output of the last layer of the encoder
or as additional band to the IMG input.

## Benchmark architecture

The PyTorch lightning [5] framework is used to implement the baseline
architecture. We choose a U-Net architecture [6] with a ResNet34 encoder
backbone for a total of $\approx$ 24.4:M parameters and rely on the
implementation available in the segmentation-models-pytorch library. The
architecture employed is illustrated in Figure 7.

The Cross Entropy loss (CE) is used as the criterion:

$\mathcal{L_{CE}}=-\sum_{i=1}^{n}t_{i}\log(p_{i})$

where $t_{i}$ is the MSK label and $p_{i}$ the Softmax probability for the ith
class. Since the class other does not contain any information relevant to the
segmentation task, we use a class weighting strategy setting its class weight
to 0 while the weights of well-defined relevant classes are set to 1.

A stochastic gradient descent optimization technique is used during training.
A learning rate of $0.02$ is used with a reduction strategy
(ReduceLROnPlateau) having a patience set to 10. We train the models with the
maximum number of epochs set to 200, but implement an early stopping strategy
with a patience of 30 epochs. A batch size of $5$ is used for the baselines.

To ensure the reproducibility of the results, we fix the global seed (random,
numpy and torch) to 2022. From the 40 domains of the train dataset, 8 are
selected for validation. The following spatial domain split of the dataset is
used for the baseline :

TRAIN: |  D006, D007, D008, D009, D013, D016, D017, D021, D023, D030, D032, D033, D034, D035, D038, D041, D044, D046, D049, D051, D052, D055, D060, D063, D070, D072, D074, D078, D080, D081, D086, D091
---|---
VALIDATION: |  D004, D014, D029, D031, D058, D066, D067, D077
TEST: |  D012, D022, D026, D064, D068, D071, D075, D076, D083, D085

Users of the dataset intending to report their results to the benchmark must
use the same test dataset.

The computational resources used for training are ten NVIDIA Tesla V100 GPUs
with 32 GB memory located on a High Performance Computing (HPC) cluster. The
distributed data parallel (ddp) strategy available in PyTorch lightning was
used to take advantage of multi-GPU data distribution and computation. The
training time with such a configuration is less than 4 hours for the full
FLAIR-one dataset.

Concerning the exploitation of metadata, simple approaches have been tested.
The strategies explored have a first step of metadata encoding: positional
encoding of spatial and temporal information and one-hot-encoding (OHE) for
the camera type. A shallow Multi-layer Perceptron (MLP) with dropout
(probability of 0.4) and ReLU activation is then defined to jointly encode the
metadata and to provide a specified output size. Subsequently, multiple
different integration strategies with the current ResNet34/U-Net segmentation
architecture are possible. For example, a commonly employed strategy (depicted
as ’bottom’ in Figure 7) consists in matching the MLP output size to the
output size of the last layer of the ResNet34 encoder. The two vectors
(encoded metadata and encoded images) can then be added and fed into the first
layer of the architecture’s decoder. Strategies following a similar approaches
that add the MLP encoded output at different positions in the architecture’s
encoder or decoder parts (e.g., after the first input convolution layer, with
the last decoder layer, or even added as a sixth channel to the input image)
are possible.

Commonly used image data augmentation techniques are also tested using the
Albumentation library. By introducing variance in the dataset, image data
augmentation helps to prevent overfitting and provides trained models enhanced
generalization capabilities. A wide range of image data augmentation exists,
mainly based on geometric (flip, crop, resize, random affine transform…) or
color space (brightness, contrast, saturation,…) transformations. For the
baseline, only geometric transformations are explored. Vertical and horizontal
flips, and random rotations of 0, 90, 180 or 270 degrees are tested.

| metadata | data augmentation | parameters (M) | epoch saved | mIoU
---|---|---|---|---|---
baseline | ✗ | ✗ | 24.4 | 23 | 0.5443$\pm$0.0014
baseline + bottom | ✔ | ✗ | 24.6 | 20 | 0.5536$\pm$0.0014
baseline + augmentation | ✗ | ✔ | 24.4 | 55 | 0.5335$\pm$0.0038
baseline + bottom \+ augmentation | ✔ | ✔ | 24.6 | 40 | 0.5570$\pm$0.0027
TABLE II: Baseline results of ResNet34/U-Net architecture with different
strategies on the FLAIR-one test set. Results are averages of 5 runs of each
configuration.

## Benchmark metric

To evaluate the semantic segmentation task, first, confusion matrices for each
test patch are calculated between the ground truth masks (MSK) and the model
predictions (PRED). The resulting test patch confusion matrices are then
summed obtaining a single confusion matrix describing the test dataset.
Subsequently, the Intersection-over-Union (IoU, or Jaccard Index) metric, of
each semantic class, is calculated from this summed confusion matrix:

$IoU=\frac{|U\cap V|}{|U\cup V|}=\frac{TP}{TP+FP+FN}$

where U denotes the intersection, V the union, TP = true positives, FP = false
positives and FN = false negatives.

To obtain a final and single metric allowing the different results to be
assessed and ranked, the mean Intersection-over-Union (mIoU) defined as the
average of the per-class IoU is calculated. However, since the class other is
ill-defined and equivalent to void and therefore should not affect the
results, its IoU is not taken into account for the mIoU calculation. The final
mIoU is thus the average of the 12 remaining IoUs obtained for the test
dataset.

The workflow of the mIoU calculation is illustrated in Figure 8.

Figure 8: Confusion matrices derived from each patch are summed before the IoU
calculation per semantic class. The final metric (mIoU) is provided by the
averaging of the IoUs.

## Benchmark results

The results obtained using $61,712$ patches for training, and testing on the
remaining $15,700$ patches of the FLAIR-one dataset are reported in Table II.
The given results are average and their standard deviation of 5 mIoU scores
obtained for 5 runs in a given configuration.

Besides the baseline result, we also report in Table II indicative results
about metadata integration and data augmentation. The ’bottom’ strategy refers
to the adding the MLP encoded metadata to the last layer of the architecture’s
encoder. The ’augmentation’ strategy uses the three geometrical augmentations
previously described with a probability of 0.5.

We note that the results are not improved with the current use of metadata or
data augmentation. Note that potential of metadata integration or image data
augmentation has not fully been explored. Opportunities for improvement can be
pursued: for example, the encoding methods of the different modalities and
their integration into the learning, or strategically applying the image data
augmentations by taking into account the semantic class frequencies.

Detailed per-class IoU results for the baseline (without metadata integration
or data augmentations) are illustrated in Figure 9.

Figure 9: Per-class IoU obtained from the baseline architecture on the FLAIR-
one dataset.

We observe highly accurate results with IoU $\approx$ 0.8 for three semantic
classes (considering the large and heterogeneous dataset used): buildings,
water and vineyards. In contrast, other classes exhibit relatively poor
results $<0.4$, such as bare soil, coniferous, brushwood and plowed land. One
explanation for these poor results could be due to the relatively low
frequency of these classes in the dataset. However, the vineyard class less
frequent in the dataset that bare soil, is detected with a significantly
higher accuracy.

Figure 10 provides the baseline (without metadata integration or data
augmentation) confusion matrix of the test dataset, normalized by rows. The
confusion matrix shows that inter-class confusions are mainly found between
semantically close classes, e.g. between herbaceous vegetation and
agricultural land, bare soil and pervious surfaces, and coniferous and
deciduous.

Figure 10: Baseline confusion matrix of the test dataset normalized by rows.

|
---|---
Figure 11: Example of a semantic segmentation result using the baseline model.

Finally, Figure 11 provides an example of a semantic segmentation of an urban
and coastal area in the D076 spatial domain, obtained with the baseline
trained model.

## References

  * [1] Institut national de l’information géographique et forestière.  https://www.ign.fr.  [Accessed: 10-October-2022].
  * [2] Jean-Philippe Souchon, Christian Thom, Christophe Meynard, and Olivier Martin.  A large format camera system for national mapping purposes.  Revue Française de Photogrammétrie et de Télédétection, (200):48–53, 2012.
  * [3] Object Delineation on Earth Observations with Neural network (ODEON).  https://github.com/IGNF/odeon-landcover.  [Accessed: 10-October-2022].
  * [4] Laurent Guigues, Jean Pierre Cocquerez, and Hervé Le Men.  Scale-sets Image Analysis.  International Journal of Computer Vision, 68(3):289–317, 2006\.
  * [5] PyTorch Lightning.  https://www.pytorchlightning.ai.  [Accessed: 10-October-2022].
  * [6] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pages 234–241. Springer International Publishing, 2015.

## Acknowledgment

This work was performed using HPC/AI resources from GENCI-IDRIS (Grant
2022-A0131013803).


11institutetext:  Institute of Informatics and Telematics (IIT)
National Research Council (CNR), Via G. Moruzzi 1, 56124, Pisa, Italy
11email: {g.cola, m.mazza<EMAIL_ADDRESS>
# From Tweet to Theft:
Tracing the Flow of Stolen Cryptocurrency

Guglielmo Cola 11 0000-0003-2890-723X Michele Mazza 11 0000-0003-1874-3753
Maurizio Tesconi 11 0000-0001-8228-7807

###### Abstract

This paper presents a case study of a cryptocurrency scam that utilized
coordinated and inauthentic behavior on Twitter. In 2020, 143 accounts sold by
an underground merchant were used to orchestrate a fake giveaway. Tweets
pointing to a fake blog post lured victims into sending Uniswap tokens (UNI)
to designated addresses on the Ethereum blockchain, with the false promise of
receiving more tokens in return. Using one of the scammer’s addresses and
leveraging the transparency and immutability of the Ethereum blockchain, we
traced the flow of stolen funds through various addresses, revealing the
tactics adopted to obfuscate traceability. The final destination of the funds
involved two deposit addresses. The first, managed by a well-known
cryptocurrency exchange, was likely associated with the scammer’s own account
on that platform and saw deposits exceeding $3.5 million. The second address
was linked to a popular cryptocurrency swap service. These findings highlight
the critical need for more stringent measures to verify the source of funds
and prevent illicit activities.

Note to Readers: This is a revised version of the paper originally presented
at the ITASEC 2023 conference and included in CEUR Workshop
Proceedings111https://ceur-ws.org/Vol-3488/paper11.pdf. The revision includes
new information disclosed by Arkham Intelligence, a leading blockchain
analytics firm. Specifically, it has been revealed that the address we
referred to as “forwarding address” is actually part of the cryptocurrency
swap service SimpleSwap. This information enhances our understanding of the
scam’s operations and the methods used to obfuscate the trail of stolen
cryptocurrency.

Keywords: Blockchain investigation, cryptocurrency scam, Ethereum, fake
giveaway, social media, Uniswap.

##  0.1 Introduction

The growing adoption of cryptocurrencies has put under the spotlight the risks
for the users of such blockchain-based technologies. Due to the decentralized
and immutable nature of distributed ledgers, users must exercise a high degree
of caution to protect themselves from potential fraud. Twitter, with its large
user base, has become a popular target for scammers seeking to reach
inexperienced cryptocurrency users. One such scam is the advance-fee scam,
where the victims are lured into sending funds to the scammer with the fake
promise of receiving a larger sum in return. Fake giveaways, a form of
advance-fee scam, are particularly prevalent on social media. They are
generally based on false content designed to convince users that a well-known
individual is giving away money, for example to celebrate a specific event or
milestone. Coordinated and inauthentic behavior from fake accounts, commonly
known as CIB, enable scammers to spread their deceptive invitation to a large
audience, making it easier for them to reach vulnerable and unsuspecting
users.

The investigation presented in this paper stems from a previous work related
to CIBs produced by fake accounts [7]. In that work, we proposed a novel
approach to spot fake accounts for sale and then monitor their activity. As a
result, over five million tweets and four coordinated campaigns were found
throughout 2020. One of these CIBs was aimed at promoting a fake giveaway
related to the Uniswap token (UNI). A group of fake accounts advertised a link
to a fraudulent blog post, which in turn described the fake giveaway and lured
users into sending their tokens to a designated address on the Ethereum
blockchain. These URLs are no longer accessible at present, still an example
of their content can be retrieved by using the Internet Archive. Also, during
our research in [7], we were able to capture a screenshot and identify one of
the Ethereum addresses used by the scammer.

In this paper, we leverage the immutable and public nature of the Ethereum
blockchain to investigate the flow of ill-gotten UNI tokens, starting from
that single address linked to the scammer. Our analysis reveals some of the
strategies used by scammers to capture funds and transfer them to
cryptocurrency exchanges, where the stolen funds cannot be traced anymore by
using plain blockchain analysis. The presented findings can foster further
research in the area, which can possibly lead to the development of automated
techniques for quickly identifying malicious addresses and mitigating the harm
to cryptocurrency and social media users.

The paper is organized as follows. In the following Section we briefly present
some of the most relevant studies in the field of cryptocurrency scams, with a
focus on the advance-fee scam form. In Section 0.3 we provide some details on
the CIB that lured victims into sending UNI tokens to the scammer. Next, in
Section 0.4 we show the blockchain-based analysis of the flow of UNI tokens,
which uncovers the technique used by the scammer to deposit the stolen funds
to a well-known cryptocurrency exchange and a popular cryptocurrency swap
service. Section 0.5 concludes the paper and suggests avenues for future
research.

##  0.2 Related work

The last few years have witnessed increased interest in blockchain-based
technologies, not only from the general public but also from institutional
investors [5]. This rise in popularity has led to a corresponding increase in
the risks associated with cryptocurrencies, as various types of scams have
been exploited by malicious actors [2]. Examples of scams that have been
discussed by the literature include advance-fee scams [9], fake exchange apps
[13], fake initial coin offerings (ICOs) [14], phishing attacks [4], Ponzi
schemes [11, 1], and smart contract honeypots [10].

Cybercriminals have exploited the “pseudonymity” feature of blockchains, which
allows them to conceal their identities while carrying out illicit activities.
However, the transparency and immutability of public blockchains also enable
the development of techniques to trace the flows produced by such illicit
activities. Various techniques have been proposed for this purpose, including
graph analysis and machine learning [12, 3], with promising results.

Social media and the use of fake accounts have greatly facilitated the spread
of misleading contents aimed at targeting unsuspecting cryptocurrency users
[7]. One particular scam that has gained attention is the advance-fee scam,
where the victims are lured into sending an amount of money as a fee with the
false promise of receiving a greater return [2]. In the field of blockchain-
based analysis of advance-fee scams, a relevant contribution was made by [9].
Using DBSCAN clustering on the content of scam websites, they found out that
the same entities are running multiple instances of similar scams, as revealed
also by their blockchain activity. These bad actors can even fabricate ad-hoc
blockchain activity to pretend their promises are genuine. The authors also
reported that cryptocurrency exchanges were the most common destination for
funds obtained through such scams, followed by gambling platforms.

While illicit transactions should not overshadow the potential benefits that
blockchain technologies can offer to various application areas [8], the
associated risks cannot be ignored. Our work can contribute to a deeper
understanding of advance-fee scams in the context of cryptocurrency, and
ultimately help devise proper measures to mitigate their impact on users.

##  0.3 Coordinated behavior from fake Twitter accounts

Figure 1: Tweet from a fake account and fake giveaway blog post Table 1: Most used hashtags in tweets advertising the fake giveaway Hashtag | Tweets | Accounts
---|---|---
#uniswap | 12,169 | 101
#defi | 9,866 | 78
#uni | 1,197 | 14
#sushiswap | 623 | 7
#yfi | 524 | 12
#crypto | 466 | 38
#DeFi | 454 | 14
#bitcoin | 278 | 33
#relax | 212 | 2
#Crypto | 187 | 30

The analysis presented in the paper stems from a broader study on coordinated
inauthentic behavior (CIB) that we described in [7]. Fake accounts, i.e.,
accounts that hide the real identities of the people running them [6], were
identified on website buyaccs.com and then monitored over the course of one
year to detect coordinated and inauthentic behavior (CIB).

One of the four CIBs that emerged consisted in the 143 accounts that
orchestrated the Uniswap-related fake giveaway. These accounts were virtually
inactive throughout 2020, except for the second part of September, during
which they shared 146,546 tweets. The activity turned out to be a scam
operation which exploited the launch of the UNI token, occurred a few days
before these tweets started. To reach potential victims, the fake accounts
used both hashtags strictly related to the UNI token and more generic hashtags
related to the decentralized finance paradigm and other cryptocurrencies, as
shown in Table 1. The strategy behind this scam operation features tweets as a
starting point. In fact, in their tweets the fake accounts claimed to have
multiplied by ten times their amount of UNI tokens. Moreover, the tweets
featured a URL (often shortened through the buffer.com service) pointing to
articles that were visually identical to an article posted on medium.com. An
example of a fake tweet and part of the blog post is shown in Figure 1. The
article was about a UNI token giveaway and included a second URL to reach the
giveaway website, which invited users to send their UNI tokens to a designated
address on the Ethereum blockchain. Furthermore, instructions were given on
how to multiply the tokens: for every token sent to the address on the
website, one would receive back ten times as many. As shown in Figure 1, there
was also a comment section with several fake positive feedback. Thus, victims
of the scam were tricked into sending their UNI tokens to the address, with
the false promise of receiving more tokens in return. Although the strategy
sounds quite simple, relying on timing and prepared content to exploit the
hype around a specific event, it has also been proven effective, as shown in
the following Section.

Twitter systems seemed to be ineffective in countering the operation on its
own platform, as despite the large volume of tweets produced in a limited time
window, 97 accounts out of 143 were still active at the end of 2020. Moreover,
48 accounts result active as of February 2023, and their tweets related to the
scam have not been removed. The links to the fake post are broken, however its
content can be retrieved on the Internet Archive.

##  0.4 Blockchain analysis

Ethereum is an open-source and decentralized blockchain network that was
designed to enable the creation of smart contracts, which in turn can be used
to build decentralized applications (DApps). Ethereum’s native cryptocurrency
is ether (ETH), which is used to pay for “gas fees”, namely the fees
associated with executing transactions and smart contracts. One key feature of
Ethereum is the ability to “mint” and transfer fungible tokens, which could be
used to represent new assets or a specific utility according to the ERC-20
standard. Non-fungible tokens, known as NFTs, can be minted as well, according
to the ERC-721 standard. Each address on the Ethereum blockchain must contain
at least a small amount of ether to interact actively with the blockchain
(i.e., send ETH/tokens to other addresses or request execution to a smart
contract). Hence, each address may contain ETH and/or tokens.

The UNI token is an example of an ERC-20 fungible token. More specifically, it
was designed as a governance token: possessors of UNI gain the ability to vote
for decisions regarding Uniswap. The latter was the first popular
decentralized exchange, i.e., a DApp that allows users to swap tokens while
retaining ownership of funds and without the need for trusting a centralized
platform like Coinbase or Binance. Starting from September 2020, the addresses
that interacted with Uniswap before September 1, 2020, were eligible to claim
a specific amount of UNI tokens. The scammer exploited this event to organize
the fake giveaway presented in the previous Section.

In the following we show how we traced the stolen funds starting from a single
address belonging to the scammer and using blockchain analysis.

###  0.4.1 Overview of address movements

Figure 2: Overview of the addresses used in the scam

Our blockchain analysis started from a single address used by the scammer to
receive the “advance-fee” in UNI tokens from victims. Using the popular block
explorer Etherscan and its APIs, we identified the amount of UNI received by
the address and the destination of such tokens. Following the flow of UNI
tokens, we uncovered the set of addresses used by the scammer, as shown by the
graph in Figure 2.

The graph displays the flow of UNI tokens, with single addresses as nodes and
edges representing transactions to another address. The initial address from
which our analysis started is labeled as G5: it belongs to a group of
“giveaway addresses” that were advertised directly to victims so as to receive
UNI tokens. It can be seen that ten addresses were used in this way: nine of
them (G1-G9) sent their tokens to another address that we named “Consolidation
address” (CA). CA then sent the tokens to a new address, which we named “Dex
Swap Address” (DS). DS also received funds directly from another giveaway
address, G10, before converting all the UNI into ETH by using a decentralized
exchange. After, it sent approximately two-thirds of the ETH to an exchange
deposit address (D1) and the remaining to a cryptocurrency swap service
deposit address (D2).

From a temporal point of view, most of the UNI tokens were received on the
giveaway addresses between September 18 and September 28, 2020. During these
days, the giveaway addresses were advertised on Twitter and thus received UNI
tokens from victims. On September 28, funds from all the giveaway addresses
except G10 were moved to CA. All the remaining transactions occurred on
November 3, 2020, when the scammer amassed funds from CA and G10 to DS, before
swapping them for ETH and making two separate deposits to centralized services
(D1 and D2).

Further details and clarifications on each step are provided in the following
subsections.

###  0.4.2 Giveaway addresses

Figure 3: Temporal distribution of UNI token deposits to giveaway addresses Table 2: Outbound UNI token transactions from giveaway addresses Date (UTC) | From | To | Amount
---|---|---|---
2020-09-28 07:51 | G1 | CA | 1,399.9
2020-09-28 07:53 | G2 | CA | 4,581.2
2020-09-28 07:55 | G3 | CA | 5,865.1
2020-09-28 07:58 | G4 | CA | 1,651.1
2020-09-28 07:59 | G5 | CA | 1,567.5
2020-09-28 08:02 | G6 | CA | 1,017.9
2020-09-28 08:04 | G7 | CA | 52.4
2020-09-28 08:05 | G8 | CA | 1,430.8
2020-09-28 08:10 | G9 | CA | 800.0
2020-11-03 13:12 | G10 | DS | 2,443.4

Most of the giveaway addresses were labeled on Etherscan as “Phish/Hack”,
meaning that users reported them as malicious. Among G1-G10, only G4 and G6
did not receive such label, however, their activity pattern strongly suggest
that they were part of the addresses used to collect funds from victims. For
each of the giveaway addresses, the pattern was the same: receiving UNI tokens
from other addresses before sending all the tokens to another address (CA for
G1-G9, DS for G10).

More detail on the timing of the deposits to giveaway addresses is revealed in
Figure 3. In the plot, each dot represents a UNI token deposit to a specific
address, with size proportional to the amount of tokens. It can be observed
that there is almost no temporal overlap between the deposits to different
addresses. This suggests that the scammer might have changed the address
advertised in the blog post so as to elude user reports of malicious activity,
which might have informed potential victims.

As shown in Figure 2, G1-G9 sent their tokens to CA, whereas G10 sent its UNI
tokens directly to DS. More information on outbound transactions from giveaway
addresses is provided in Table 2. The deposits from G1-G9 to CA occurred in a
very short timeframe, about 20 minutes in the morning of September 28.
Instead, funds from G10 were moved to DS on November 3.

On September 28, the giveaway addresses contained a total 20,809 UNI tokens.
If we consider the UNI value of $4.78 provided by crypto data aggregator
CoinGecko (www.coingecko.com) for September 28, it turns out that the scammer
managed to collect around $100,000 worth of UNI in just ten days.

###  0.4.3 Path to deposit addresses

Table 3: Remaining transactions to move the funds to the deposit addresses (D1 and D2) Date (UTC) | From | To | Crypto | Amount
---|---|---|---|---
2020-11-03 13:18 | CA | DS | UNI | 18,365.8
2020-11-03 15:24 | DS | 1inch | UNI | 20,809.3
2020-11-03 15:24 | 1inch | DS | ETH | 115.8
2020-11-03 15:26 | DS | D1 | ETH | 40.0
2020-11-03 15:28 | DS | D2 | ETH | 40.0
2020-11-03 15:43 | DS | D1 | ETH | 48.2

The list of transactions involving CA and DS is shown in Table 3. As mentioned
in the previous subsection, CA received 18,365.8 UNI from G1-G9. These tokens
remained dormant until November 3, 2020, when they were transferred to DS. As
shown in Table 2, on the same day (just six minutes earlier) DS also received
2,443.4 UNI tokens from G10. A couple of hours later, the UNI tokens in DS
were swapped for around 115.8 ETH using the popular DEX aggregator named
1inch. A DEX aggregator connects automatically to multiple decentralized
exchanges, in order to find the best “route” for a given trade. As mentioned
before, decentralized exchanges, like Uniswap, allow users to trade
cryptocurrencies without going through centralized exchanges, as the trade is
managed by a smart contract executed on the blockchain. An additional 12.4 ETH
were obtained in DS by swapping two different ERC-20 tokens, sushiToken
(SUSHI) and yearn.finance (YFI), whose origin is beyond the scope of this
investigation. The total 128.2 ETH were then split: 88.2 ETH were sent to a
first deposit address (D1) with two transactions of 40.0 and 48.2 ETH, while
the remaining 40.0 ETH were sent to another deposit address (D2).

###  0.4.4 D1 – Exchange deposit address

Table 4: Value of deposits in D1 Top 5 Tokens | Value $
---|---
1 | USDC | 1,474,369
2 | ETH | 1,169,189
3 | USDT | 380,879
4 | REN | 185,863
5 | TUSD | 122,921
Total D1 | 3,564,959
Figure 4: Daily and cumulative deposits made to D1

The scammer(s) deposited 88.2 ETH to address D1. The transactions involving
this address reveal patterns that are typical of an exchange deposit address.
A centralized cryptocurrency exchange, also known as CEX, is a platform that
acts as an intermediary between buyers and sellers, and that makes money
mainly through commissions and fees. Popular CEXes include Binance, Coinbase,
and Crypto.com. To start using a CEX, users have to deposit fiat currrency or
cryptocurrency, which can then be exchanged for other fiat/crypto. For crypto
deposits, CEXes provide users with a dedicated deposit address on the
blockchain. For instance, if the user asks to deposit ETH or ERC-20 tokens,
the CEX provides an address on the Ethereum blockchain where the user can
deposit their ETH/tokens. After the deposit is made, the CEX credits the
deposited amount to the user’s balance on the platform, which can then be
exchanged for the other fiat currencies or crypto tokens available on the CEX.

The deposit address is generated by the CEX, which controls the keys needed to
move funds out of that address. Therefore, when a user deposits crypto to such
an address, the effective ownership of the cryptocurrency is transferred to
the CEX, while the user becomes a creditor. The cryptocurrency received in a
deposit address is typically moved by the CEX to another address, where the
cryptocurrencies received from multiple users are aggregated for easier
management. More information on how deposit addresses are managed is provided
in this blog post222www.binance.com/en/blog/from-cz/transparency-on-wallets-
at-binance-4080204794657829130 from popular CEX Binance.

Address D1 received over 1,500 deposits. Most of the cryptocurrency deposited
was later transferred to Ethereum addresses that are publicly labeled as
belonging to Binance. Also, the addresses received small amounts of ETH to be
used for gas fees from a smart contract created by Binance itself. These
patterns strongly suggest that D1 is controlled by Binance and allowed the
scammer(s) to deposit their ETH and ERC-20 tokens on that popular CEX. By
using a CEX, the scammer was able to break the traceability of funds on the
blockchain. Presumably, funds were later withdrawn to a different address on
the blockchain or traded for fiat currency (e.g., US dollars or euros) and
then sent to a bank account. Indeed, differently from the transactions
happening on the blockchain, what happens within a CEX cannot be studied
publicly, as funds from all users are merged into a limited set of adddresses
belonging to the CEX itself, and there is no way to link the initial deposit
address to a withdrawal address.

To evaluate the scale of the scammer’s activity, Table 4 presents the top five
tokens deposited to D1 ranked by their aggregated USD value. The USD value of
each deposit was calculated using the exchange rate at the time of deposit.
The total USD-equivalent value deposited is also shown, considering ETH as
well as all the ERC-20 tokens deposited to D1. The total USD value is
remarkable, amounting to over $3.5 million. Top used tokens show a prevalence
of stablecoins (USDC, USDT, TUSD), i.e., tokens that aim to be pegged to the
value of the US dollar and thus protect users from the typical volatility of
the crypto market. All the tokens shown are highly popular in the context of
decentralized finance.

A temporal view of the scammer’s activity related to D1 is provided in Figure
4. Each blue dot shows the aggregated value of the deposits made on a single
day (value on the left y axis). Instead, the green line shows the cumulative
USD-equivalent value of deposits over time (value on the right y axis).
Deposits were aggregated daily for better visualization. D1 was mostly active
between November 2019 and February 2021, with the exception of a very large
USDT transaction (worth around $220,000) on August 26, 2021. Daily deposits
had an average value of about $10,450. It should be noted that the
distribution of daily deposited values is highly skewed, as there are
exceptionally high transactions on given days as well as a large number of
relatively small deposits.

Figure 5: Daily and cumulative deposits made to SimpleSwap’s deposit address
on Binance

###  0.4.5 D2 – Swap service deposit address

Address D2 received 40 ETH, which were entirely transferred to another address
within just 32 minutes. After these two events, D2 became completely inactive.
The address that received the funds is another deposit address owned by
Binance, thus indicating it is linked to a Binance customer. Initially, we
believed address D2 was used as a forwarding address to slightly obfuscate the
trail of funds. However, the blockchain analytics firm Arkham
Intelligence333https://platform.arkhamintelligence.com/ disclosed that D2 is a
deposit address of the cryptocurrency swap service SimpleSwap. SimpleSwap
allows users to swap a wide variety of cryptocurrencies without the need to
create an account. Users send their cryptocurrency to a specifically
designated address; shortly thereafter, the cryptocurrency resulting from the
swap is sent to a user-specified address, which could be on Ethereum or
another blockchain. The funds sent to a deposit address (D2 in this study) are
quickly aggregated by SimpleSwap into a deposit address controlled by a CEX,
such as Binance, to facilitate future swap operations. SimpleSwap thus uses
Binance and other popular CEXes to hold custody of their funds.

Figure 5 shows the volume of SimpleSwap’s deposit address on Binance. There
are over 79,000 deposits, presumably from multiple legitimate users. The total
deposited value in USD terms amounts to over $270 million. As all funds were
merged, it is practically impossible to understand which of these funds came
from the scammer analyzed in this study. Furthermore, it is impossible to
determine which token (and on which blockchain) the original 40 ETH sent by
the scammer were converted into.

##  0.5 Conclusions and future work

In this case study, we presented a blockchain-based investigation into a fake
giveaway of Uniswap (UNI) tokens that was promoted through coordinated and
inauthentic behavior on social media. As we have demonstrated, the scammer
lured inexperienced cryptocurrency users into sending around $100,000 worth of
UNI to a set of designated addresses on the Ethereum blockchain between
September 18 and September 28, 2020. On November 3, 2020, the UNI tokens were
aggregated into a single “Consolidation” address before being sent to a new
address, to which we have assigned the pseudonym “DEX Swap”. This address was
used to swap the UNI tokens for ETH via a decentralized finance platform.
Finally, the resulting ETH was split into two separate addresses (D1 and D2),
which we later identified as deposit addresses controlled by the popular
exchange Binance and cryptocurrency swap service SimpleSwap, respectively.
This enabled the scammers to obfuscate the trail of stolen cryptocurrency, and
they were presumably able to cash out the funds or move them to different
addresses on Ethereum or even different blockchains.

This study shows how the transparency of public blockchains can facilitate the
linkage of illicit activities starting from a single address labeled as
malicious. The UNI scam proved to be just the tip of the iceberg, as the
aggregate activity on D1 revealed a volume of deposits worth over $3.5
million. Even though we cannot infer that all of these funds were obtained
illicitly and that the scammer managed to withdraw their funds from Binance,
the potential impact revealed by our analysis is concerning. In general, it
could be difficult for exchanges to determine with certainty whether deposited
funds were obtained illicitly, especially if the scammer adopts sophisticated
techniques to obfuscate the origin of their funds. Nevertheless, these
findings emphasize the need for centralized exchanges to adopt more stringent
measures to prevent illicit activities and protect their users. This is
especially critical for deposits originating from cryptocurrency swap
services, as these platforms can obscure fund origins due to their no-account
trading feature. Automated blockchain-based analysis could be a key tool to
promptly identify suspicious activity.

In future work, we plan to extend the graph of transactions related to this
and other scams in order to identify common and detectable patterns. Also, it
would be interesting to explore automated techniques to detect suspicious
deposit addresses, which could help centralized exchanges in complying with
stricter policies.

## Acknowledgments

This work was partially supported by: the European Union – Horizon 2020
Program under the scheme “INFRAIA-01-2018-2019 – Integrating Activities for
Advanced Communities”, Grant Agreement n.871042, “SoBigData++: European
Integrated Infrastructure for Social Mining and Big Data Analytics”
(http://www.sobigdata.eu); project SERICS (PE00000014) under the NRRP MUR
program funded by the EU - NGEU; Regione Toscana, which funded this project
(Project INTERROGATE) within the framework of the “Por Fse 2014-2020
Investimenti a favore della crescita, dell’occupazione e del futuro dei
giovani Fondo Sociale Europeo”. Decreto di finanziamento DD 21607 del
29/11/2021.

## References

  * [1] Massimo Bartoletti, Salvatore Carta, Tiziana Cimoli, and Roberto Saia.  Dissecting ponzi schemes on ethereum: Identification, analysis, and impact.  Future Generation Computer Systems, 102:259–277, 2020.
  * [2] Massimo Bartoletti, Stefano Lande, Andrea Loddo, Livio Pompianu, and Sergio Serusi.  Cryptocurrency scams: analysis and perspectives.  IEEE Access, 9:148353–148373, 2021.
  * [3] Steven Farrugia, Joshua Ellul, and George Azzopardi.  Detection of illicit accounts over the ethereum blockchain.  Expert Systems with Applications, 150:113318, 2020.
  * [4] Artsiom Holub and Jeremiah O’Connor.  COINHOARDER: Tracking a ukrainian bitcoin phishing ring DNS style.  In 2018 APWG Symposium on Electronic Crime Research (eCrime), pages 1–5. IEEE, 2018.
  * [5] Xiaoran Huang, Juan Lin, and Peng Wang.  Are institutional investors marching into the crypto market?  Economics Letters, 220:110856, 2022.
  * [6] Michele Mazza, Marco Avvenuti, Stefano Cresci, and Maurizio Tesconi.  Investigating the difference between trolls, social bots, and humans on twitter.  Computer Communications, 196:23–36, 2022.
  * [7] Michele Mazza, Guglielmo Cola, and Maurizio Tesconi.  Ready-to-(ab)use: From fake account trafficking to coordinated inauthentic behavior on twitter.  Online Social Networks and Media, 31:100224, 2022.
  * [8] Ahmed Afif Monrat, Olov Schelén, and Karl Andersson.  A survey of blockchain from the perspectives of applications, challenges, and opportunities.  IEEE Access, 7:117134–117151, 2019.
  * [9] Ross Phillips and Heidi Wilder.  Tracing cryptocurrency scams: Clustering replicated advance-fee and phishing websites.  In 2020 IEEE International Conference on Blockchain and Cryptocurrency (ICBC), pages 1–8. IEEE, 2020.
  * [10] Christof Ferreira Torres, Mathis Steichen, and Radu State.  The art of the scam: Demystifying honeypots in ethereum smart contracts.  In 28th USENIX Security Symposium (USENIX Security 19), pages 1591–1607, Santa Clara, CA, August 2019. USENIX Association.
  * [11] Marie Vasek and Tyler Moore.  Analyzing the bitcoin ponzi scheme ecosystem.  In Aviv Zohar, Ittay Eyal, Vanessa Teague, Jeremy Clark, Andrea Bracciali, Federico Pintore, and Massimiliano Sala, editors, Financial Cryptography and Data Security, pages 101–112, Berlin, Heidelberg, 2019. Springer Berlin Heidelberg.
  * [12] Jiajing Wu, Qi Yuan, Dan Lin, Wei You, Weili Chen, Chuan Chen, and Zibin Zheng.  Who are the phishers? phishing scam detection on ethereum via network embedding.  IEEE Transactions on Systems, Man, and Cybernetics: Systems, 52(2):1156–1166, 2020.
  * [13] Pengcheng Xia, Haoyu Wang, Bowen Zhang, Ru Ji, Bingyu Gao, Lei Wu, Xiapu Luo, and Guoai Xu.  Characterizing cryptocurrency exchange scams.  Computers & Security, 98:101993, 2020.
  * [14] Dirk A Zetzsche, Ross P Buckley, Douglas W Arner, and Linus Fohr.  The ICO gold rush: It’s a scam, it’s a bubble, it’s a super challenge for regulators.  Harv. Int’l LJ, 60:267, 2019.