# FoMo-Bench: a multi-modal, multi-scale and multi-task Forest Monitoring
Benchmark for remote sensing foundation models

Nikolaos Ioannis Bountos1,3,4 Arthur Ouaknine1,2 David Rolnick1,2
1Mila Quebec AI Institute
2McGill University
3Orion Lab, National Observatory of Athens & National Technical University of
Athens
4Harokopio University of Athens

###### Abstract

Forests are an essential part of Earth’s ecosystems and natural systems, as
well as providing services on which humanity depends, yet they are rapidly
changing as a result of land use decisions and climate change. Understanding
and mitigating negative effects requires parsing data on forests at global
scale from a broad array of sensory modalities, and recently many such
problems have been approached using machine learning algorithms for remote
sensing. To date, forest-monitoring problems have largely been approached in
isolation. Inspired by the rise of foundation models for computer vision and
remote sensing, we here present the first unified Forest Monitoring Benchmark
(FoMo-Bench). FoMo-Bench consists of 15 diverse datasets encompassing
satellite, aerial, and inventory data, covering a variety of geographical
regions, and including multispectral, red-green-blue, synthetic aperture radar
(SAR) and LiDAR data with various temporal, spatial and spectral resolutions.
FoMo-Bench includes multiple types of forest-monitoring tasks, spanning
classification, segmentation, and object detection. To further enhance the
diversity of tasks and geographies represented in FoMo-Bench, we introduce a
novel global dataset, TalloS, combining satellite imagery with ground-based
annotations for tree species classification, spanning 1,000+ hierarchical
taxonomic levels (species, genus, family). Finally, we propose FoMo-Net, a
foundation model baseline designed for forest monitoring with the flexibility
to process any combination of commonly used sensors in remote sensing. This
work aims to inspire research collaborations between machine learning and
forest biology researchers in exploring scalable multi-modal and multi-task
models for forest monitoring. All code and data will be made publicly
available.

##  1 Introduction

Figure 1: The Forest Monitoring Benchmark. FoMo-Bench is composed of fifteen
datasets, including TalloS (ours), counting seven groups of data with distinct
physical properties and three tasks. In parallel, we introduce FoMo-Net, a
baseline forest foundation model pre-trained using a spectral band masked
autoencoder (MAE) approach.

Forests worldwide are under threat from climate change, land use change, and
invasive species [4, 9, 20, 56], leading to significant biodiversity decline
and economic losses. However, effective forest management and conservation are
hampered by a lack of information, highlighting the need for scalable forest
monitoring. This gap includes multiple specific problems, such as tree species
identification, tree size assessment, land cover classification, etc., which
also vary considerably across geographies.

Remote sensing has been explored for forest monitoring using various sensors
and spatial scales [19, 39], and increasingly, deep learning methods have been
used to address various specific forest-related problems [30]. Simultaneously,
the field of computer vision has seen a growth in foundation models, _i.e_.
models flexible to diverse modalities, scales and data regimes while
performing well across downstream tasks [31, 25, 26]. However, it has been
noted that general-purpose foundation models in computer vision may be
ineffective in specific problem domains, presenting a need for foundation
models explicitly designed for specific problem spaces, such as Earth
observation [32] and weather or climate modeling [40].

In this work, we propose the development of foundation models for forest-
monitoring tasks. Our Forest Monitoring Benchmark (FoMo-Bench) brings together
a wide range of publicly available datasets for forest monitoring,
encompassing significant spatial, temporal, and spectral diversity. The
benchmark includes a mixture of classification, segmentation and detection
tasks, from land cover classification to tree crown segmentation. To improve
the geographic diversity of FoMo-Bench, we also propose TalloS, a unique,
global, multi-modal dataset focused on tree species identification and based
on worldwide forest inventories [27]. Each sample included in TalloS combines
data acquired from the Sentinel-2 and Sentinel-1 missions, meteorological data
from the ERA-5 dataset and elevation information, and georeferenced tree
species information, in the form of a multi-classification task. Finally, we
use FoMo-Bench combined with unlabeled data archives to train the first multi-
modal foundation model for forest monitoring applications, FoMo-Net, which can
effectively address a wide range of forest-related tasks and can process most
commonly used spectral bands in remote sensing.

Overall, our contributions can be summarized as follows:

  * •

We provide the first, curated benchmark for forest monitoring, with a global
spatial coverage and a diverse set of evaluation tasks;

  * •

We improve the coverage of this benchmark by introducing a novel multi-modal
dataset for tree species multi-classification;

  * •

We create the first foundation model for forest monitoring, using a novel pre-
training approach leveraging a wide range of remote sensing products.

##  2 Related Work

Large-scale remote sensing data has been available for many years through
state-funded projects such as Sentinel and Landsat. Building on such sources,
various machine learning benchmarks have been developed, such as BigEarthNet
[53], Sen12MS [51], Sen4AgriNet [55] and CropHarvest [60]. These initiatives
have facilitated extensive research particularly in general land use and land
cover (LULC) estimation, as well as for forest-related tasks such as species
classification [13].

Recent exploration of deep learning methods for forest monitoring [30] has
been made possible by a range of specific datasets at various scales and
geographic locations [41]. High resolution image recordings from unmanned
aerial vehicles (UAV) are part of recent datasets for automated canopy mapping
[15], woody invasive species detection [28] and tree species detection [29].
In particular, ReforesTree [48] combines UAV recordings with inventories for
both detecting trees and estimate their above ground biomass. The NeonTree
dataset [64] is unique in grouping airborne recordings including light
detection and ranging (LiDAR), red-blue-green (RGB) and hyperspectral
measurements. Data recorded from satellite sensors have been used for
deforestation detection [24] and canopy height estimation [43] using Landsat,
or vegetation cover estimation using Sentinel [51]. Although inventories have
been used alongside satellite data in Russia with less than a thousand samples
[5, 63], we are not aware of any remote sensing dataset providing such fine
annotations on a large volumetric and geographical scale. Recently, manual
forest inventories, including manual measurements of tree height and
aboveground biomass, have recently been aggregated in Tallo [27], a data
source that we leverage for machine learning in the present work.

Increasingly, benchmarks across the space of computer vision have moved beyond
single tasks to encompass a diversity of related tasks. The large scale
BigEarthNet-MM dataset [53], built from Sentinel-1 and Sentinel-2 data, is
designed to assess both performance and training complexity of deep learning
methods [42]. The SustainBench [66] combines various satellite imagery and
street-level images to monitor sustainable development goals covering 15
different topics. Geo-Bench [33] proposes to group remote sensing datasets –
including SAR, multispectral, elevation and cloud probabilities data – to
tackle 6 classification and 6 segmentation tasks focused on LULC. However, the
spatial coverage of Geo-Bench is narrow, reducing the potential of the
benchmark to generalize to all continents. To the best of our knowledge,
however, there is no existing benchmark focused on forest monitoring
worldwide, combining inventories, aerial and satellite data, while proposing
classification, segmentation and detection tasks.

Generalization capacities of deep learning models are explored with foundation
models [3]. Principally driven by LULC applications, foundation models have
been investigated for multi-modal and multi-task remote sensing [54, 7, 47],
but are also focused on specific satellite products. The recently introduced
Presto algorithm [61] is capable to handle a large amount of spectral bands,
but it is restricted to single-pixel timeseries. To the best of our knowledge,
foundation models have not been yet applied to various forest monitoring tasks
and at diverse resolutions (a few centimeters to tens of meters), despite
their significant potential [41] such as for above ground biomass estimation
worldwide [57, 62].

##  3 FoMo-Bench: a multi-modal, global benchmark for forest monitoring

(a) FoMo-Bench spatial distribution without TalloS.

(b) TalloS spatial distribution.

Figure 2: FoMo-Bench spatial distribution. (a) Spatial distribution of the
datasets included in FoMo-Bench, without TalloS; satellite-based datasets are
marked with arrows, aerial-based datasets with circles. (b) Distribution of
the train, validation and test splits of the TalloS dataset worldwide.

A wide variety of different open-access forest monitoring datasets exist, with
great diversity in terms of sensors as well as spatial, spectral and temporal
resolutions [41]. This section introduces the composition of FoMo-Bench, the
first unified forest monitoring benchmark, providing a challenging evaluation
framework for future foundation models. This benchmark encompasses a diverse
set of tasks focused on forest composition spanning from LULC classification
to tree species detection.

By carefully selecting datasets composing FoMo-Bench, we have ensured that it
(1) provides a wide spatial coverage, offering evaluation sets in diverse
locations and environmental conditions, (2) includes a diverse area of task
types, and (3) includes input data stemming from a large variety of sensors
_e.g_. optical, SAR, multispectral, DEM and LiDAR, as well as ground sampling
distances spanning from a few centimeters to $60\text{m}$ per pixel. The
diversity of FoMo-Bench in terms of sensors and tasks to tackle is illustrated
in Figure 1. Below, we describe the datasets included in FoMo-Bench, based on
the sensory modalities of the input data, the tasks in question, the spatial
coverage, and other characteristics.

We have curated all the datasets in FoMo-Bench, spatially split them to train
machine learning models and created a reproducible environment for future work
with a unified set of PyTorch data loaders.

Satellite-based datasets FoMo-Bench includes five datasets involving satellite
imagery, namely BigEarthNet-MM, Sen12MS, RapidAI4EO, ForestNet,
FiveBillionPixels and TreeSatAI. The BigEarthNet-MM [53] and Sen12MS [51]
datasets are two of the most commonly used multi-modal datasets for LULC in
remote sensing. Both are based on the Copernicus Sentinel missions offering
SAR and multispectral data from Sentinel-1 and Sentinel-2 respectively. The
BigEarthNet-MM dataset also includes two nomenclatures for its annotations
based on the Corine land cover database [11] covering 10 countries in Europe.
A higher level nomenclature has been used in FoMo-Bench, containing 19
distinct classes with broad-leaved, coniferous and mixed forests. Contrary to
BigEarthNet-MM, the Sen12MS dataset has a global spatial coverage (see Figure
2(a)), offering annotations for 17 classes based on MODIS land cover maps [14]
with 5 different classes for forests cover. FoMo-Bench also includes
RapidAI4EO [2] as a third LULC dataset with a unique source of high resolution
satellite imagery offering timeseries of Planet data, at 3m per pixel
resolution, for 500,000 locations across Europe. We construct a nomenclature
for RapidAI4EO annotations with the 24 classes, 12 of them relevant to forest
monitoring. The ForestNet dataset [24] aims to identify drivers leading to
deforestation and uses Landsat-8 imagery. FiveBillionPixels dataset [58]
provides high resolution Gaofen-2 images ($4\text{m}$) for LULC of cities in
China with 24 categories including 5 forest monitoring classes. The TreeSatAI
dataset [1] offers multi-modal data (_e.g_. aerial images, SAR and
multispectral) for tree species identification, including 20 European tree
species derived from forest administration data of the federal state of Lower
Saxony, Germany. Finally, we also introduce a new dataset, TalloS, which is
detailed in Section 4.

Aerial-based datasets The limited spatial resolution offered by free satellite
data is not suited for several remote sensing tasks, such as tree canopy
segmentation or detection. FoMo-Bench includes eight datasets with high
resolution data from aerial recordings, namely NeonTree, Woody, ReforesTree,
Spekboom, Waititu, FLAIR #1, FLAIR #2 and FORinstance. The NeonTree dataset
[64] is based on tree crown detection with RGB imagery, LiDAR and
hyperspectral data recorded from airborne systems. The FoMo-Bench focuses on
the NeonTree high resolution RGB imagery (0.1m) for detection and LiDAR data
to enhance the benchmark with point cloud representations. The Woody dataset
[28] is composed of RGB images recorded from an UAV, at a few centimeters
resolution, for invasive species detection, in particular Pinus radiata, Ulex
europaeus and Acacia dealbata in Chile. Similarly, the Spekboom [15] and the
Waititu [29] datasets are composed of RGB images recorded from UAVs to map
tree species in South Africa and New Zealand respectively. In the same vein,
the ReforesTree dataset [48] has been built for tree species detection with 6
classes using high resolution RGB images recorded from an UAV, as well as
ground measurements, such as tree height and diameter at breast height. The
FORinstance dataset [44] provides a unique source of data with LiDAR point
clouds and semantic annotations collected in 5 countries (see Fig. 2(a)).
Contrary to the previous aerial datasets, FLAIR #1 [16, 18] and FLAIR #2 [17,
18] are large-scale datasets targeting distribution shifts induced by
spatiotemporal alterations. Both datasets cover wide areas in France,
providing high resolution aerial images (0.2m) along with elevation images
aiming to identify LULC using 19 categories, including coniferous and
deciduous forest classes. Diverging from FLAIR #1, FLAIR #2 introduces
Sentinel-2 timeseries along side with a new test dataset split.

##  4 TalloS: A multi-modal tree species multi-classification dataset

Split | Number of Classes | Number of samples
---|---|---
Train | 1159 | 161332
Val | 24 | 4049
Test | 223 | 25100

Table 1: TalloS splits. Distribution of classes and samples of the proposed
TalloS train, validation and test splits.

As shown in Fig. 2, the spatial coverage of pre-existenting datasets in FoMo-
Bench is limited. We improve the diversity and spatial coverage of FoMo-Bench
by designing a new dataset, TalloS. This dataset couples manual tree
inventories with multispectral and SAR timeseries imagery to bring a fine
grained multi-classification challenge for species, genus and family tree
diversity.

As described in the previous section, various tasks in forest monitoring have
required high-resolution data from UAVs but these typically have small spatial
coverage due to human and hardware limits [15, 48, 28, 29]. Satellite-based
datasets with fine grained annotations have previously been lacking for forest
monitoring applications.

We propose TalloS, a large-scale multi-modal dataset of satellite-based
imagery for tree species multi-classification. It is based on the Tallo
database [27], containing almost $500,000$ manual georeferenced recordings
spread across the globe and aggregating information from forest inventories
such as tree species, genus and family categories as well as tree height, stem
diameter or crown radius. It groups recordings from forested and non-forested
biomes, including tropical forests, temperate forests, boreal forests,
Mediterranean woodland, tropical savanna, temperate grassland and dryland.

Considering the location distribution of the inventory recordings [27], TalloS
has been built focusing on areas containing at least 50 trees. This is also
based on the relatively small ground sampling distance of publicly accessible
satellites ($\approx 10\text{m}$), as regions with lower tree populations
would not show a distinctive signature in satellite imagery.

TalloS is composed, for each location, of a time series from 2016 to 2023 with
a temporal frequency of 15 days. Each sample comprises a multispectral image
from Sentinel-2, a SAR image from Sentinel-1, the associated digital elevation
model (DEM) from Copernicus GLO-30, the meteorological information from ERA5
[23] and the corresponding detailed ground-level annotations. Each remote
sensing sample of TalloS is thus mapped to ground-validated and fine grained
measurements of trees w.r.t. its given area, including tree species, genus and
family as well as their specific traits.

In Figure 2(b), we demonstrate the wide spatial distribution of TalloS as well
as the proposed training, validation and test split. Assigning points to train
and test sets by randomly selecting geographic regions would result in highly
skewed distributions, where a great number of classes could potentially exist
solely in the test or validation set. We faced the challenge of creating
meaningful geospatial splits by maximizing the geographic spatial coverage of
each split while avoiding region overlap and ensuring that classes in
validation and test are included in train. In particular, we ensure a fair
label distribution among the training and test split by completing the missing
classes in the training set from the set of regions assigned in the test set.
The final dataset split is demonstrated in Table 1.Preliminary experiments
conducted on TalloS for multi-classification as part of the FoMo-Bench are
detailed in Section 5 and presented in Table 2.

##  5 The FoMo-Bench evaluation

In this section, we provide strong baselines to evaluate the global diversity
of FoMo-Bench in addressing three distinct types of computer vision tasks:
classification, semantic segmentation and object detection. Specifically, for
each task, we train a set of standard computer vision methods. Importantly,
each of these baselines is trained in a supervised manner on the data for that
single task. This is because the heterogeneity of the tasks in both input and
output data prevents the naive use of a single architecture trained on all
data. (It is this challenge that we address in Section 6 with the introduction
of the foundation model baseline FoMo-Net.)

#### Classification

Due to the large ground sampling distance of sensors in remote sensing,
classification tasks are usually multi-label. Considering a single sample, the
end-goal is to predict the presence of more than one class. The multi-
classification setting is thus used for BigEarthNet-MM, Sen12MS, RapidAI4EO,
TreeSat AI and TalloS for either LULC or tree species recognition. Our
baselines for the classification tasks include ResNet50 [21], ViT [10] and
ConvNext [36] models. We initialize all models with weights pretrained on
ImageNet. All models are trained using a cosine learning rate schedule for 10
epochs. We use the AdamW optimizer and set the learning rate to $0.001$. The
weight decay is set to $0.001$ and $0.0001$ depending on the dataset and the
architecture. For each model we report the micro and macro F1 score for all
datasets in 2.

Dataset | Modalities | # Classes | Method | F1 micro | F1 macro
---|---|---|---|---|---
BigEarthNet [53] |  Multispectral SAR | 19 | ResNet50 | 76.96 | 68.96
ViT | 77.36 | 70.8
ConvNext | 77.01 | 69.3
FoMo-Net (ours) | 54.78 | 33.15
Sen12MS [51] |  Multispectral SAR | 17 | ResNet50 | 52.55 | 31.05
ViT | 49.74 | 31.9
ConvNext | 55.21 | 29.38
FoMo-Net (ours) | 27.47 | 13.24
TalloS (ours) |  Multispectral SAR DEM | 1159 | ResNet50 | 19.81 | 0.86
ViT | 11.81 | 0.21
ConvNext | 10.39 | 0.37
FoMo-Net (ours) | 13.52 | 0.34
RapidAI4EO [2] |  High resolution optical Multispectral | 24 | ResNet50 | 57.3 | 33.4
ViT | 48.99 | 20.27
ConvNext | 62.91 | 43.58
FoMo-Net (ours) | 16.54 | 2.5
TreeSatAI [1] |  Multispectral | 15 | ResNet50 | 34.33 | 20.67
ViT | 13.08 | 13.88
ConvNext | 34.93 | 19.29
FoMo-Net (ours) | 39.28 | 15.38

Table 2: Evaluation of classification tasks. The multi-classification tasks
are evaluated using four methods, including finetuning FoMo-Net with a single
linear decoding layer. The F1-score at the micro and macro levels are used to
compare the baselines performances.

#### Semantic segmentation of grid-based data

Semantic segmentation occurs in applications such as pixel-wise tree species
recognition and is well suited to grid-based data, _e.g_. remotely sensed
imagery rather than point clouds. The FoMo-Bench includes 7 datasets for
semantic segmentation: Waititu, FLAIR #1, FLAIR #2, Spekboom,
FiveBillionPixels, ForestNet and Woody. Our baseline models here are UNet
[50], DeepLabv3 [6] and UPerNet [65]. We use a ResNet50 backbone for UNet and
DeepLabv3, and a Swin Transformer backbone for UPerNet. For UNet and DeepLabv3
architectures we use models pretrained on ImageNet. UPerNet is based on a
Swin-Base backbone from [8]. All models are trained using a cosine learning
rate schedule for 10 epochs. We use the AdamW optimizer for training and set
the learning rate and weight decay to either $0.001$ or $0.0001$ depending on
the dataset and architecture. The performances are reported with the mean
intersection over union (mIoU) and the F1-score at micro level. Results are
presented in Table 3.

Dataset | Modalities | # Classes | Method | mIoU | F1 micro
---|---|---|---|---|---
Waititu [29] | Aerial RGB | 3 | UNet | 70.63 | 82.79
DeepLabv3 | 70.9 | 82.97
UPerNet | 66.77 | 80.07
FoMo-Net (ours) | 55.67 | 71.52
FLAIR #1 [16, 18] | Aerial multispectral | 19 | UNet | 54.81 | 70.81
DeepLabv3 | 59.71 | 74.93
UPerNet | 61.65 | 75.22
FoMo-Net (ours) | 34.21 | 50.98
FLAIR #2 [17, 18] | Aerial multispectral | 19 | UNet | 42.2 | 57.81
DeepLabv3 | 66.15 | 79.63
UPerNet | 49.27 | 66.02
FoMo-Net (ours) | 38.11 | 34.81
Spekboom [15] | Aerial RGB | 2 | UNet | 97.67 | 95.44
DeepLabv3 | 97.72 | 95.54
UPerNet | 95.68 | 97.79
FoMo-Net (ours) | 86.47 | 92.74
FiveBillionPixels [58] | High resolution optical | 22 | UNet | 55.83 | 71.65
DeepLabv3 | 55.91 | 71.72
UPerNet | 47.3 | 64.22
FoMo-Net (ours) | 40.65 | 57.8
ForestNet [24] |  Multispectral | 5 | UNet | 91.15 | 95.37
DeepLabv3 | 91.01 | 95.29
UPerNet | 90.89 | 95.23
FoMo-Net (ours) | 90.93 | 95.25
Woody [28] | Aerial RGB | 4 | UNet | 85.87 | 92.4
DeepLabv3 | 88.71 | 94.02
UPerNet | 88.46 | 93.88
FoMo-Net (ours) | 78.16 | 87.74

Table 3: Evaluation of semantic segmentation tasks for grid-based data. The
semantic segmentation tasks focused on image-based data are evaluated using
four methods, including finetuning FoMo-Net with a single linear decoding
layer. The mean intersection over union (mIoU) and the F1-score at the micro
level are used to compare the baselines performances.

#### Object detection

The object detection task is useful in reforest monitoring to count and
precisely locate trees, with a distinction per species when applicable. As it
requires to distinguish canopies of each other, aerial RGB data are
particularly well suited for this task. The NeonTree and ReforesTree are
explored for binary and multi-class object detection respectively. They are
evaluated using Faster R-CNN [49], RetinaNet [35] and YOLOS [12] models. The
performances are reported with well known metrics for object detection [34],
_i.e_. the mean intersection over union (mIoU), the mean average precision
with an $\text{IoU}=0.5$ threshold ($\text{mAP}_{50}$) and the mean average
precision with a variable threshold of $\text{IoU}\in[0.5,0.95]$
($\text{mAP}_{50:95}$). Results are presented in Table 4. The models include
pre-trained backbones on either the COCO dataset (ResNet-50 and DeiT-S) or
following the FoMo-Net pre-training method. The Faster R-CNN are finetuned for
200 epochs, with a learning rate of $0.01$ for the NeonTree dataset and
$0.001$ for the ReforesTree dataset with an exponential decay, both using the
SGD optimizer. The RetinaNet are finetuned for 200 epochs with a learning rate
of $0.001$ with an exponential decay for both datasets, both using the AdamW
optmizer [37]. The YOLOS are finetuned for 300 epochs with learning rate of
$0.0001$ with an exponential decay for both datasets both using the AdamW
optmizer [37]. We used the released splits for NeonTree, and we propose our
own spatial splits for ReforesTree which will be released with the code.

Dataset | Modalities | # Classes | Method | Backbone | mIoU | $\text{mAP}_{50}$ | $\text{mAP}_{50:95}$
---|---|---|---|---|---|---|---
NeonTree [64] | Airborne RGB | 1 | Faster R-CNN | ResNet-50 | 61.99 | 41.51 | 15.02
RetinaNet [35] | ResNet-50 | 56.75 | 9.56 | 3.14
YOLOS | DeiT-S [59] | 53.11 | 28.56 | 13.05
Faster R-CNN | FoMo-Net (ours) | 41.35 | 0.17 | 0.03
ReforesTree [48] | Aerial RGB | 6 | Faster R-CNN | ResNet-50 | 65.12 | 7.77 | 3.44
RetinaNet [35] | ResNet-50 | 66.36 | 3.54 | 1.74
YOLOS | DeiT-S [59] | 53.97 | 0.62 | 0.39
Faster R-CNN | FoMo-Net (ours) | 45.65 | 0.08 | 0.02

Table 4: Evaluation of object detection tasks.  The object detection tasks are
evaluated using four methods, including finetuning FoMo-Net with a Faster
R-CNN head. The baseline performances are evaluated with the mean intersection
over union (mIoU) and the mean average precision (mAP) for both
$\text{IoU}=0.5$ and $\text{IoU}\in[0.5,0.95]$ thresholds.

#### Semantic segmentation of point clouds

Processing point cloud representations from LiDAR sensors aims at better
understanding the geometry of trees at large scale. Classifying each point,
_i.e_. segmenting a point cloud, can be used to distinguish trees, delineate
their components (_e.g_. trunk, branches or leaves) or evaluate their traits
(_e.g_. height, species or biomass). FoMo-Bench includes NeonTree and
FORinstance datasets for binary and multi-class point cloud segmentation
respectively. For these datasets, we use the baselines PointNet [45],
PointNet++[46] and Point Transformer [67]. The performances are reported with
the mean intersection over union (mIoU) and the F1-score at micro level.
Results are presented in Table 5. The PointNet models are trained for 300
epochs, with a learning rate of $0.001$ for the NeonTree dataset and $0.0001$
for the FORinstance dataset with an exponential decay, both using the AdamW
optimizer [37]. Both the PointNet++ and PointTransformer models are trained
for 300 epochs, with a learning rate of $0.0001$ for both datasets with an
exponential decay, both using the AdamW optimizer [37]. We used the released
splits for both NeonTree and FORinstance datasets.

Dataset | Modalities | # Classes | Method | mIoU | F1 micro
---|---|---|---|---|---
NeonTree [64] | Airborne LiDAR | 1 | PointNet | 62.35 | 76.81
PointNet++ | 33.95 | 50.69
Point Transformer | 60.49 | 75.35
FORinstance [44] | Aerial LiDAR | 7 | PointNet | 35.7 | 52.62
PointNet++ | 10.56 | 19.1
Point Transformer | 28.32 | 44.14

Table 5: Evaluation of semantic segmentation tasks for point clouds. The
semantic segmentation tasks focused on point cloud data are evaluated using
three methods. The mean intersection over union (mIoU) and the F1-score at the
micro level are used to compare the baselines performances.

##  6 FoMo-Net: A baseline forest foundation model

The development of FoMo-Bench facilitates reliable evaluation of foundation
model capacities in the diverse settings involved in forest monitoring.
Despite the rapid advancements in geo-spatial foundation models [54, 7, 47,
61], we are not aware of previous work capable to adapt in tandem to such a
diverse suite of downstream tasks with variable forms of sensory input data
(see Sec. 2). It is particularly notable in FoMo-Bench that certain datasets
related to similar tasks (_e.g_. LULC) nonetheless use different modality
combinations.

To meet this challenge, we detail the FoMo-Net algorithm, which builds upon
both FoMo-Bench and previous works, in an effort to create a more flexible
geospatial foundation model. Our approach is motivated by three core
considerations. First, the algorithm should enable maximum possible
adaptability in regards to the input space, assuming no specific set of
sensory modalities. Second it should have the capacity to process information
and generate meaningful representations for the whole globe. Third, it should
be applicable to a wide range of downstream task types.

#### Pre-training data

Based on the vast amount of data provided by FoMo-Bench (see Fig. 2), as well
as on external large-scale datasets, the FoMo-Net pre-training scheme contains
rich multi-sensor information from most parts of the world. In particular, we
use the RapidAI4EO dataset as a source of Planet and Sentinel data; TalloS
containing Sentinel-1, Sentinel-2 and DEM; SSL4EO-Landsat [52] to acquire
global information from Landsat 8 and 9; a combination of all available UAV
datasets in FoMo-Bench; and the FiveBillionPixel dataset providing a unique
source of Gaofen-2 high resolution satellite imagery. The datasets used in the
FoMo-Net pre-training framework are detailed in Table 6. The proposed
framework is able to process any combination of the 36 most common modalities
in remote sensing, stemming from Sentinel-1, Sentinel-2, Landsat 8-9, Planet,
Gaofen-2 and UAV sensors with ground sampling distance spanning from a few
centimeters to $60\text{m}$ per pixel. In this work, a distinct modality is
defined as any band with a unique combination of wavelength and spatial
resolution. Given its importance in Earth observation, digital elevation
models (DEM) are also included in our definition as a specific band.

Dataset | Sensors | Spatial resolution | Sampling weight
---|---|---|---
SSL4EO-Landsat | Landsat 8-9 | Low | 0.2
RapidAI4EO | Planet, Sentinel-2 | Medium, Low | 0.2
TalloS | Sentinel-1, Sentinel-2, DEM | Low | 0.2
FLAIR #1 | Aerial | High | 0.1
FiveBillionPixels | Gaofen-2 | Medium | 0.2
UAV-datasets | Aerial | Very high | 0.1

Table 6: FoMo-Net pre-training datasets. The proposed pre-training framework
has been experimented with four satellite-based datasets and four aerial-based
datasets, including three UAV datasets from FoMo-Bench (Waititu, Spekboom and
Woody). Their corresponding spatial resolution is defined as very high
($<5\text{cm}$), high ($>5\text{cm}$ and $<1\text{m}$), medium ($>1\text{m}$
and $<10\text{m}$) or low ($>10\text{m}$). The sampling weight is used for the
gradient accumulation during pre-training.

#### Spectral band MAE

Inspired from recent foundation models for remote sensing [54, 7, 47, 61], the
proposed framework is based on a standard masked autoencoder (MAE) [22], but
diverging in two aspects (see Fig. 3). First, we introduce a random spectral
band selection to model interactions between a wide range of band
combinations, creating a highly flexible encoder able to process any set of
spectral bands. Second, we include a spectral position embedding while
tokenizing each band separately producing $N\times(H/P)\times(W/P)$ patches,
where $N$ is the number of bands, $H$ and $W$ the height and width of the
image and $P$ the patch size. Each produced patch is linearly projected to an
embedding dimension $D$. A positional embedding is included to model spatial
relationships as well as a spectral embedding to identify the processed
modalities. Given the randomness at the input space, the spectral embedding is
of utmost importance as no other information regarding the nature of the input
is provided.

#### Pre-training pipeline

The FoMo-Net pre-training pipeline is summarized in Algorithm 1 and
illustrated in Figure 3. At each training iteration, a set of spectral bands
is randomly sampled from a single dataset. The sampling factors associated to
each dataset for the pre-training are presented in Table 6. Each dataset has
either a unique set of spectral bands or several sets of spectral bands when
data from multiple sensors are provided, _e.g_. RapidAI4EO combining high-
resolution Planet and Sentinel-2 images. This random sampling is uniform for
all samples in the batch, drastically increasing the amount of unique samples,
while creating a more efficient training pipeline. As an example, a dataset
with $15$ spectral bands (_e.g_. RapidAI4EO) will result in $32,768$ potential
combinations. The randomly selected bands are then tokenized independently
adding a positional and a spectral embedding. The next steps of the procedure
are identical to the MAE framework reconstructing the masked tokens with an
autoencoder and quantifying the reconstruction loss with the mean squared
error (MSE). The combination of the proposed spectral band sampling and the
token masking is particularly computationally efficient compared to generating
and processing tokens from the entire spectrum information. We note that
randomly choosing datasets and spectral bands at each training iteration
results in highly heterogeneous batches, which could destabilize the
optimization process. To compensate, the FoMo-Net pre-training process
exploits gradient accumulation (see Fig. 3). In this work, a vision
transformer (ViT) [10] is used with 12 layers and 6 attention heads as an
encoder. Compared to existing foundation models, FoMo-Net is quite shallow,
intended as a starting point towards a general remote sensing foundation
model, and as a lightweight model since most stakeholders implementing forest-
monitoring algorithms in practice have very limited computational resources.

#### Experimental baseline

We test the FoMo-Net pre-training framework using satellite and aerial
datasets from both the FoMo-Bench and open access satellite archives.
Overrall, the SSL4EO-Landsat contains over 500 GB of data, BigEarthNet around
60 GB, while the total size of RapidAI4EO is over 10 TB of data. In testing
FoMo-Net on FoMo-Bench, we finetune the model with a linear layer for
classification and semantic segmentation, and use it as a backbone for
detection. Results are presented in Table 2, Table 3 and Table 4.

Figure 3: FoMo-Net pre-training framework. Considering a set of potential
spectral input bands and a sub-training process of $n$ batches, each batch
includes a sub-set of bands w.r.t. to the sampled dataset and modalities it
contains. For each batch, the input band tokens, the spectral tokens and the
positional tokens are independently projected with a linear transformation.
These embeddings are summed element-wise to obtain an embedding of each
spectral band available in the batch. The spectral band masked autoencoder
(MAE), $f_{\theta}(\cdot)$ parameterized by $\theta$, tries to reconstruct the
partially masked bands, illustrated with dashed squared, of a given batch. The
gradients of the loss $\mathscr{L}(\cdot)$ are accumulated to preserve
balanced information, and backpropagated through both $f_{\theta}(\cdot)$ and
the linear transformations. For the sake of clarity, we do not include the
elevation modality which is not considered as a spectral band, but included in
the pre-training scheme. Algorithm 1 FoMo-Net pre-training pseudocode.

⬇

# update_model(index): True -> update model, False -> accumulate gradients

for idx in range(number_of_steps):

# Pick a random dataset

dataset_name, dataloader = get_random_dataloader()

# Get a random combination of modalities

selected_bands = get_random_bands(dataset_name)

batch = dataloader.__next__()

data = data[:, selected_bands, :, :]

# Create tokens in both spatial and spectral dimension

tokens = list()

data = rearrange(data, "b c h w -> c b h w")

all_patches = list()

for i, channel in enumerate(data):

channel_patches = patchify(channel)

channel_tokens = tokenize(channel_patches)

channel_tokens += pos_embedding

channel_tokens += spectral_embedding(i,selected_bands)

tokens.extend(channel_tokens)

all_patches.extend(channel_patches)

#Mask tokens

unmasked_tokens, masked_patches = mask_tokens(tokens,all_patches,
ratio=mask_ratio)

#Encoder

encodings = encoder(unmasked_tokens)

#Decoder

reconstructed = decoder(encodings)

loss = MSE(reconstructed, masked_patches)

#Optimize

if update_model(idx):

loss.backward()

optimizer.step()

optimizer.zero_grad()

##  7 Discussion

###  7.1 Analysis of TalloS

The experiments conducted on FoMo-Bench are presented in Tables 2, 3, 4 and 5.
The overall objective is to set up strong evaluation frameworks for either
problem-specific or foundation models with a particular focus on forest-
monitoring applications. We find that standard computer vision models such as
ConvNext and Vision Transformers perform relatively poorly on several
datasets, _e.g_. TalloS and Sen12MS. In particular, the TalloS training set
contains more than 1100 classes (see 1) making a single-classification task
challenging (random classification accuracy around $\approx 1/1159$), and the
multi-classification one even more. Another relevant factor in this dataset
(see Sec 4) is that tree species are not distributed uniformly around the
globe, with many classes present only in specific locations. Within a single
genus, for example, different species may be visually very similar but present
in different locations. One opportunity to improve large-scale tree species
classification tasks in FoMo-Net would thus be to incorporate tree species
occurrence maps as biological prior knowledge or to learn such information
from the data (see e.g. [38]). Introducing FoMo-Bench opens up opportunities
to better exploit individual modalities and tasks as well as combining them to
extend global understanding of forests at different scales and spectrum with
respect to model capacities.

###  7.2 FoMo-Net assessment and next steps

The FoMo-Bench offers a delineated framework to evaluate remote sensing
foundation models on forest monitoring tasks. The introduction of FoMo-Net is
a step towards the development of such models, proposing a generic method to
digest well known remote sensing products. Nonetheless, evaluating a
particularly shallow ViT pre-trained with the FoMo-Net framework showcases
efficacy greater than random guess in diverse scenarios, often providing
comparable results with specialist models e.g Spekboom, Waititu and Woody in
Table 3. Interestingly, FoMo-Net performed the best, in terms of micro F1
score for the TreeSatAI dataset.

FoMo-Net on classification tasks The results presented in Table 2 indicate
that the pre-training framework helped to learn significant features for LULC
and tree species multi-classification tasks. The FoMo-Net also show
interesting capacities in the latter with a strong foundation model baseline
compared to other specialized baselines despite the high number of classes in
TalloS.

FoMo-Net on segmentation tasks The reported performances in Table 3 highlight
the FoMo-Net pre-training providing relevant semantic information. The FoMo-
Net also offers a solid foundation model baseline compared to specialized
methods for semantic segmentation on both aerial and satellite data, even if
the gap in resolution is significant.

FoMo-Net on detection tasks As depicted in Table 4, the object detection tasks
are challenging for both datasets and the finetuning using FoMo-Net performs
less well than specialized models. The foundation model baseline seems to
partially detect trees but struggles to categorise them.

More generally, the FoMo-Net pre-training framework is capable of processing
and reconstructing randomly sampled spectral bands at varying ground sampling
distances, offering promising opportunities in remote sensing super-
resolution. Considering the results reported in FoMo-Bench, optimizing the
FoMo-Net pre-training scheme, as well as enhancing its architecture to process
non-grid like structures, _e.g_. LiDAR point clouds, is a priority for future
work. This latter perspective is achievable with an appropriate tokenizer that
generates tokens from point clouds and a respective reconstruction projection
to the point cloud space.

##  8 Conclusion

This work presents FoMo-Bench, a multi-modal, multi-scale and multi-task
benchmark, geographically distributed worldwide, providing a robust and
reliable evaluation framework for future forest monitoring foundation models.
To further expand the task and spatial coverage of FoMo-Bench, we introduce
TalloS, an extensive globally distributed dataset encompassing multi-temporal
and multi-modal samples for tree species classification. Building on these
contributions, we craft a foundation model, FoMo-Net, designed for forest-
monitoring applications. FoMo-Net leverages a pre-training framework designed
to process a flexible range of spectral bands and be applicable for a wide
range of tasks in remote sensing. Our experimental evaluation of FoMo-Net on
the FoMo-Bench tasks establishes the first baseline for general-purpose forest
monitoring, while identifying opportunities and challenges for this promising
field.

## References

  * Ahlswede et al. [2023] S. Ahlswede, C. Schulz, C. Gava, P. Helber, B. Bischke, M. Förster, F. Arias, J. Hees, B. Demir, and B. Kleinschmit.  TreeSatAI Benchmark Archive: a multi-sensor, multi-label dataset for tree species classification in remote sensing.  _Earth System Science Data_ , 2023.
  * Bhugra et al. [2022] P. Bhugra, B. Bischke, C. Werner, R. Syrnicki, C. Packbier, P. Helber, C. Senaras, A. S. Rana, T. Davis, W. De Keersmaecker, D. Zanaga, A. Wania, R. Van de Kerchove, and G. Marchisio.  Rapidai4eo: Mono-and multi-temporal deep learning models for updating the corine land cover product.  In _IGARSS_ , 2022.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R’e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang.  On the Opportunities and Risks of Foundation Models.  _ArXiv_ , 2021.
  * Bonan [2008] Gordon B. Bonan.  Forests and Climate Change: Forcings, Feedbacks, and the Climate Benefits of Forests.  _Science_ , 2008.
  * Brieger et al. [2019] Frederic Brieger, Ulrike Herzschuh, Luidmila A. Pestryakova, Bodo Bookhagen, Evgenii S. Zakharov, and Stefan Kruse.  Advances in the Derivation of Northeast Siberian Forest Metrics Using High-Resolution UAV-Based Photogrammetric Point Clouds.  _Remote Sensing_ , 2019.
  * Chen et al. [2017] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  _ArXiv_ , 2017.
  * Cong et al. [2022] Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon.  Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery.  _NeurIPS_ , 2022.
  * Contributors [2020] MMSegmentation Contributors.  MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation, 2020.
  * Curtis et al. [2018] Philip G. Curtis, Christy M. Slay, Nancy L. Harris, Alexandra Tyukavina, and Matthew C. Hansen.  Classifying drivers of global forest loss.  _Science_ , 2018.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _ArXiv_ , 2020.
  * European Environment Agency and European Environment Agency [2020] European Environment Agency and European Environment Agency.  CORINE Land Cover 2018 (vector), Europe, 6-yearly, 2020.
  * Fang et al. [2021] Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu.  You only look at one sequence: Rethinking transformer in vision through object detection.  In _NeurIPS_ , 2021.
  * Fassnacht et al. [2016] Fabian Ewald Fassnacht, Hooman Latifi, Krzysztof Stereńczak, Aneta Modzelewska, Michael Lefsky, Lars T. Waser, Christoph Straub, and Aniruddha Ghosh.  Review of studies on tree species classification from remotely sensed data.  _Remote Sensing of Environment_ , 2016.
  * Friedl et al. [2010] Mark A. Friedl, Damien Sulla-Menashe, Bin Tan, Annemarie Schneider, Navin Ramankutty, Adam Sibley, and Xiaoman Huang.  MODIS Collection 5 global land cover: Algorithm refinements and characterization of new datasets.  _Remote Sensing of Environment_ , 2010.
  * Galuszynski et al. [2022] Nicholas C Galuszynski, Robbert Duker, Alastair J Potts, and Teja Kattenborn.  Automated mapping of portulacaria afra canopies for restoration monitoring with convolutional neural networks and heterogeneous unmanned aerial vehicle imagery.  _PeerJ_ , 2022.
  * Garioud et al. [2022] Anatol Garioud, Stéphane Peillet, Eva Bookjans, Sébastien Giordano, and Boris Wattrelos.  Flair #1: semantic segmentation and domain adaptation dataset.  _ArXiv_ , 2022.
  * Garioud et al. [2023a] Anatol Garioud, Apolline De Wit, Marc Poupée, Marion Valette, Sébastien Giordano, and Boris Wattrelos.  Flair #2: textural and temporal information for semantic segmentation from multi-source optical imagery.  _ArXiv_ , 2023a.
  * Garioud et al. [2023b] Anatol Garioud, Nicolas Gonthier, Loic Landrieu, Apolline De Wit, Marion Valette, Marc Poupée, Sebastien Giordano, and Boris Wattrelos.  FLAIR : a country-scale land cover semantic segmentation dataset from multi-source optical imagery.  In _NeurIPS_ , 2023b.
  * Guimaraes et al. [2020] Nathalie Guimaraes, Luis Padua, Pedro Marques, Nuno Silva, Emanuel Peres, and Joaquim J. Sousa.  Forestry Remote Sensing from Unmanned Aerial Vehicles: A Review Focusing on the Data, Processing and Potentialities.  _Remote Sensing_ , 2020.
  * Hartmann et al. [2022] Henrik Hartmann, Ana Bastos, Adrian J Das, Adriane Esquivel-Muelbert, William M Hammond, Jordi Martínez-Vilalta, Nate G McDowell, Jennifer S Powers, Thomas AM Pugh, Katinka X Ruthrof, and others.  Climate change risks to global forest health: emergence of unexpected events of elevated tree mortality worldwide.  _Annual Review of Plant Biology_ , 2022.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _CVPR_ , 2022.
  * Hersbach et al. [2020] Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz‐Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, Adrian Simmons, Cornel Soci, Saleh Abdalla, Xavier Abellan, Gianpaolo Balsamo, Peter Bechtold, Gionata Biavati, Jean Bidlot, Massimo Bonavita, Giovanna De Chiara, Per Dahlgren, Dick Dee, Michail Diamantakis, Rossana Dragani, Johannes Flemming, Richard Forbes, Manuel Fuentes, Alan Geer, Leo Haimberger, Sean Healy, Robin J. Hogan, Elías Hólm, Marta Janisková, Sarah Keeley, Patrick Laloyaux, Philippe Lopez, Cristina Lupu, Gabor Radnoti, Patricia De Rosnay, Iryna Rozum, Freja Vamborg, Sebastien Villaume, and Jean‐Noël Thépaut.  The ERA5 global reanalysis.  _Quarterly Journal of the Royal Meteorological Society_ , 146, 2020.
  * Irvin et al. [2020] Jeremy Irvin, Hao Sheng, Neel Ramachandran, Sonja Johnson-Yu, Sharon Zhou, Kyle Story, Rose Rustowicz, Cooper Elsworth, Kemen Austin, and Andrew Y Ng.  Forestnet: Classifying drivers of deforestation in indonesia using deep learning on satellite imagery.  _NeurIPS Workshop_ , 2020.
  * Jaegle et al. [2021] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira.  Perceiver: General Perception with Iterative Attention.  _ICML_ , 2021.
  * Jaegle et al. [2022] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and Joāo Carreira.  Perceiver IO: A General Architecture for Structured Inputs & Outputs.  _ICLR_ , 2022.
  * Jucker et al. [2022] Tommaso Jucker, Fabian Jörg Fischer, Jérôme Chave, David A Coomes, John Caspersen, Arshad Ali, Grace Jopaul Loubota Panzou, Ted R Feldpausch, Daniel Falster, Vladimir A Usoltsev, et al.  Tallo: A global tree allometry and crown architecture database.  _Global change biology_ , 2022.
  * Kattenborn et al. [2019] Teja Kattenborn, Javier Lopatin, Michael Förster, Andreas Christian Braun, and Fabian Ewald Fassnacht.  Uav data as alternative to field sampling to map woody invasive species based on combined sentinel-1 and sentinel-2 data.  _Remote sensing of environment_ , 2019.
  * Kattenborn et al. [2020] Teja Kattenborn, Jana Eichel, Susan Wiser, Larry Burrows, Fabian E Fassnacht, and Sebastian Schmidtlein.  Convolutional neural networks accurately predict cover fractions of plant species and communities in unmanned aerial vehicle imagery.  _Remote Sensing in Ecology and Conservation_ , 2020.
  * Kattenborn et al. [2021] Teja Kattenborn, Jens Leitloff, Felix Schiefer, and Stefan Hinz.  Review on Convolutional Neural Networks (CNN) in vegetation remote sensing.  _ISPRS Journal of Photogrammetry and Remote Sensing_ , 2021.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _ArXiv_ , 2023.
  * Lacoste et al. [2021] Alexandre Lacoste, Evan David Sherwin, Hannah Kerner, Hamed Alemohammad, Björn Lütjens, Jeremy Irvin, David Dao, Alex Chang, Mehmet Gunturkun, Alexandre Drouin, et al.  Toward foundation models for earth monitoring: Proposal for a climate change benchmark.  _NeurIPS Workshops_ , 2021.
  * Lacoste et al. [2023] Alexandre Lacoste, Nils Lehmann, Pau Rodriguez, Evan David Sherwin, Hannah Kerner, Björn Lütjens, Jeremy Andrew Irvin, David Dao, Hamed Alemohammad, Alexandre Drouin, et al.  Geo-bench: Toward foundation models for earth monitoring.  _NeurIPS_ , 2023.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _ECCV_ , 2014.
  * Lin et al. [2020] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  _TPAMI_ , 2020.
  * Liu et al. [2022] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In _CVPR_ , 2022.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _ICLR_ , 2019.
  * Mac Aodha et al. [2019] Oisin Mac Aodha, Elijah Cole, and Pietro Perona.  Presence-only geographical priors for fine-grained image classification.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9596–9606, 2019.
  * Michalowska and Rapinski [2021] Maja Michalowska and Jacek Rapinski.  A Review of Tree Species Classification Based on Airborne LiDAR Data and Applied Classifiers.  _Remote Sensing_ , 2021.
  * Mukkavilli et al. [2023] S Karthik Mukkavilli, Daniel Salles Civitarese, Johannes Schmude, Johannes Jakubik, Anne Jones, Nam Nguyen, Christopher Phillips, Sujit Roy, Shraddha Singh, Campbell Watson, et al.  Ai foundation models for weather and climate: Applications, design, and implementation.  _ArXiv_ , 2023.
  * Ouaknine et al. [2023] Arthur Ouaknine, Teja Kattenborn, Etienne Laliberté, and David Rolnick.  Openforest: A data catalogue for machine learning in forest monitoring.  _ArXiv_ , 2023.
  * Papoutsis et al. [2023] Ioannis Papoutsis, Nikolaos Ioannis Bountos, Angelos Zavras, Dimitrios Michail, and Christos Tryfonopoulos.  Benchmarking and scaling of deep learning models for land cover image classification.  _ISPRS Journal of Photogrammetry and Remote Sensing_ , 2023.
  * Potapov et al. [2019] P. Potapov, A. Tyukavina, S. Turubanova, Y. Talero, A. Hernandez-Serna, M.C. Hansen, D. Saah, K. Tenneson, A. Poortinga, A. Aekakkararungroj, F. Chishtie, P. Towashiraporn, B. Bhandari, K.S. Aung, and Q.H. Nguyen.  Annual continuous fields of woody vegetation structure in the Lower Mekong region from 2000-2017 Landsat time-series.  _Remote Sensing of Environment_ , 2019.
  * Puliti et al. [2023] Stefano Puliti, Grant Pearse, Peter Surový, Luke Wallace, Markus Hollaus, Maciej Wielgosz, and Rasmus Astrup.  For-instance: a uav laser scanning benchmark dataset for semantic and instance segmentation of individual trees.  _ArXiv_ , 2023.
  * Qi et al. [2017a] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _CVPR_ , 2017a.
  * Qi et al. [2017b] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  _NeurIPS_ , 2017b.
  * Reed et al. [2023] Colorado J Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Kurt Keutzer, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell.  Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning.  In _ICCV_ , 2023.
  * Reiersen et al. [2022] Gyri Reiersen, David Dao, Björn Lütjens, Konstantin Klemmer, Kenza Amara, Attila Steinegger, Ce Zhang, and Xiaoxiang Zhu.  Reforestree: A dataset for estimating tropical forest carbon stock with deep learning and aerial imagery.  In _AAAI_ , 2022.
  * Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In _NeurIPS_. MIT Press, 2015.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _MICCAI_ , 2015.
  * Schmitt et al. [2019] Michael Schmitt, Lloyd Haydn Hughes, Chunping Qiu, and Xiao Xiang Zhu.  Sen12ms–a curated dataset of georeferenced multi-spectral sentinel-1/2 imagery for deep learning and data fusion.  _ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , 2019.
  * Stewart et al. [2023] Adam J Stewart, Nils Lehmann, Isaac A Corley, Yi Wang, Yi-Chia Chang, Nassim Ait Ali Braham, Shradha Sehgal, Caleb Robinson, and Arindam Banerjee.  Ssl4eo-l: Datasets and foundation models for landsat imagery.  _ArXiv_ , 2023.
  * Sumbul et al. [2021] Gencer Sumbul, Arne De Wall, Tristan Kreuziger, Filipe Marcelino, Hugo Costa, Pedro Benevides, Mario Caetano, Begüm Demir, and Volker Markl.  Bigearthnet-mm: A large-scale, multimodal, multilabel benchmark archive for remote sensing image classification and retrieval [software and data sets].  _IEEE Geoscience and Remote Sensing Magazine_ , 2021.
  * Sun et al. [2022] Xian Sun, Peijin Wang, Wanxuan Lu, Zicong Zhu, Xiaonan Lu, Qibin He, Junxi Li, Xuee Rong, Zhujun Yang, Hao Chang, Qinglin He, Guang Yang, Ruiping Wang, Jiwen Lu, and Kun Fu.  RingMo: A Remote Sensing Foundation Model with Masked Image Modeling.  _IEEE Transactions on Geoscience and Remote Sensing_ , 2022.
  * Sykas et al. [2021] Dimitris Sykas, Ioannis Papoutsis, and Dimitrios Zografakis.  Sen4agrinet: A harmonized multi-country, multi-temporal benchmark dataset for agricultural earth observation machine learning applications.  In _IGARSS_ , 2021.
  * The Food and Agriculture Organization of the United Nations [2020] The Food and Agriculture Organization of the United Nations.  _Global Forest Resources Assessment 2020_.  FAO, 2020.
  * Tolan et al. [2023] Jamie Tolan, Hung-I. Yang, Ben Nosarzewski, Guillaume Couairon, Huy Vo, John Brandt, Justine Spore, Sayantan Majumdar, Daniel Haziza, Janaki Vamaraju, Theo Moutakani, Piotr Bojanowski, Tracy Johns, Brian White, Tobias Tiecke, and Camille Couprie.  Sub-meter resolution canopy height maps using self-supervised learning and a vision transformer trained on Aerial and GEDI Lidar.  _ArXiv_ , 2023.
  * Tong et al. [2023] Xin-Yi Tong, Gui-Song Xia, and Xiao Xiang Zhu.  Enabling country-scale land cover mapping with meter-resolution satellite imagery.  _ISPRS Journal of Photogrammetry and Remote Sensing_ , 2023.
  * Touvron et al. [2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou.  Training data-efficient image transformers & distillation through attention.  In _ICML_ , 2021.
  * Tseng et al. [2021] Gabriel Tseng, Ivan Zvonkov, Catherine Lilian Nakalembe, and Hannah Kerner.  Cropharvest: A global dataset for crop-type classification.  In _NeurIPS_ , 2021.
  * Tseng et al. [2023] Gabriel Tseng, Ivan Zvonkov, Mirali Purohit, David Rolnick, and Hannah Kerner.  Lightweight, pre-trained transformers for remote sensing timeseries.  _ArXiv_ , 2023.
  * Tucker et al. [2023] Compton Tucker, Martin Brandt, Pierre Hiernaux, Ankit Kariryaa, Kjeld Rasmussen, Jennifer Small, Christian Igel, Florian Reiner, Katherine Melocik, Jesse Meyer, Scott Sinno, Eric Romero, Erin Glennie, Yasmin Fitts, August Morin, Jorge Pinzon, Devin McClain, Paul Morin, Claire Porter, Shane Loeffler, Laurent Kergoat, Bil-Assanou Issoufou, Patrice Savadogo, Jean-Pierre Wigneron, Benjamin Poulter, Philippe Ciais, Robert Kaufmann, Ranga Myneni, Sassan Saatchi, and Rasmus Fensholt.  Sub-continental-scale carbon stocks of individual trees in African drylands.  _Nature_ , 2023.
  * van Geffen et al. [2022] Femke van Geffen, Birgit Heim, Frederic Brieger, Rongwei Geng, Iuliia A. Shevtsova, Luise Schulte, Simone M. Stuenzi, Nadine Bernhardt, Elena I. Troeva, Luidmila A. Pestryakova, Evgenii S. Zakharov, Bringfried Pflug, Ulrike Herzschuh, and Stefan Kruse.  SiDroForest: a comprehensive forest inventory of Siberian boreal forest investigations including drone-based point clouds, individually labeled trees, synthetically generated tree crowns, and Sentinel-2 labeled image patches.  _Earth System Science Data_ , 2022.
  * Weinstein et al. [2020] Ben G Weinstein, Sarah J Graves, Sergio Marconi, Aditya Singh, Alina Zare, Dylan Stewart, Stephanie A Bohlman, and Ethan P White.  A benchmark dataset for individual tree crown delineation in co-registered airborne rgb, lidar and hyperspectral imagery from the national ecological observation network.  _PLoS computational biology_ , 2020.
  * Xiao et al. [2018] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In _ECCV_ , 2018.
  * Yeh et al. [2021] Christopher Yeh, Chenlin Meng, Sherrie Wang, Anne Driscoll, Erik Rozi, Patrick Liu, Jihyeon Lee, Marshall Burke, David B Lobell, and Stefano Ermon.  Sustainbench: Benchmarks for monitoring the sustainable development goals with machine learning.  _NeurIPS_ , 2021.
  * Zhao et al. [2021] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip H.S. Torr, and Vladlen Koltun.  Point transformer.  In _ICCV_ , 2021.