# How accurate are existing land cover maps for agriculture in Sub-Saharan
Africa?

Hannah Kerner  Arizona State University, School of Computing and Augmented
Intelligence, Tempe, AZ 85281, USA  corresponding author(s): Hannah Kerner
<EMAIL_ADDRESS> Catherine Nakalembe  University of Maryland, Department of
Geographical Sciences, College Park, MD 20740, USA  Adam Yang  University of
Maryland, Department of Computer Science, College Park, MD 20740, USA  Ivan
Zvonkov  Arizona State University, School of Computing and Augmented
Intelligence, Tempe, AZ 85281, USA  Ryan McWeeny  University of Maryland,
Department of Geographical Sciences, College Park, MD 20740, USA  Gabriel
Tseng  McGill University, Mila – Quebec AI Institute, Montreal, Quebec, Canada
Inbal Becker-Reshef  University of Maryland, Department of Geographical
Sciences, College Park, MD 20740, USA

###### Abstract

Satellite Earth observations (EO) can provide affordable and timely
information for assessing crop conditions and food production. Such monitoring
systems are essential in Africa, where there is high food insecurity and
sparse agricultural statistics. EO-based monitoring systems require accurate
cropland maps to provide information about croplands, but there is a lack of
data to determine which of the many available land cover maps most accurately
identify cropland in African countries. This study provides a quantitative
evaluation and intercomparison of 11 publicly available land cover maps to
assess their suitability for cropland classification and EO-based agriculture
monitoring in Africa using statistically rigorous reference datasets from 8
countries. We hope the results of this study will help users determine the
most suitable map for their needs and encourage future work to focus on
resolving inconsistencies between maps and improving accuracy in low-accuracy
regions.

## Introduction

Africa is a critical area for research on food security. Half of the low-
income and food-deficient countries (43 out of 86) are on the continent[1] and
Africa has the highest prevalence of undernourished people (20.2% of the
population in 2021).[2] Efforts to assess, monitor, and mitigate food
insecurity in Africa are hindered by a lack of information to inform such
efforts, such as agricultural production statistics or crop conditions
assessments. Satellite Earth observations (EO) provide an affordable,
reliable, and timely source of information for assessing crop conditions and
food production, but EO-based monitoring systems require accurate cropland
masks that focus observations on locations where crops are likely being grown
in a given year.[3, 4, 5] Cropland masks are typically derived from land cover
maps, which may be agriculture-specific maps indicating cropland areas or
capture broad land cover and land use classes. Map products are becoming the
go-to source for informing agricultural policies, development investments,
food and nutrition security monitoring, and climate modeling.[6] While EO-
based map assessments are more scalable and affordable than traditional
survey-based estimates, they must be used cautiously when they inform policy
or other high-stakes decisions.[7] Crop maps are directly used in EO-based
monitoring systems such as the GEOGLAM Crop Monitors,[8, 9] Global
Agricultural Monitoring (GLAM) System,[10] and ASAP,[11] among others.[5]
Inaccurate crop masks increase the risk of over- or under-estimation of crop
conditions or impacts of disasters or climate shocks on food production and
may result in dubious statistics informing aid and development efforts.

Production of global land cover maps continues to accelerate due to
advancements in satellite sensors, machine learning, and cloud computing, as
well as increased availability of open data and growing interest in
environmental monitoring and management. Users now have an overwhelming array
of options for land cover maps but lack independent quantitative information
to evaluate these maps and assess their relative advantages or disadvantages,
particularly in the context of agricultural monitoring in Africa. As more
researchers and decision-makers look to Earth observations as a solution for
agriculture and food security monitoring in Africa, there is a need for a
comprehensive evaluation and inter-comparison of available maps to help users
identify suitable data products and guide future work efforts to develop
improved maps.

Previous work showed that available land cover maps had low accuracy in Africa
and substantial inconsistencies between maps.[12, 13, 14, 15, 16, 17, 18, 19,
20] For example, Nabil et al. (2019) found that cropland accuracies for four
maps was below 65% in Africa.[12] Pérez-Hoyos et al. (2017) found full
agreement between seven land cover maps in just 2.15% of Africa.[13] Venter et
al. (2022) compared three global land cover maps released since 2021 (Google’s
Dynamic World,[21] ESA’s WorldCover,[22] and Esri’s Land Use Land Cover[23])
and found substantial inaccuracies.[14] The discrepancies between cropland
classifications by different maps have been attributed to several factors
including heterogeneous land cover, cloud cover affecting data availability in
remote sensing model inputs, and the small and fragmented nature of crop
fields in Africa.[12] Some work has sought to resolve these inconsistencies
and create improved regional maps by combining the classifications from
several individual maps.[24, 25, 26] Researchers have hypothesized that
higher-resolution satellite datasets (e.g., 10 m/px Sentinel-2) would enable
the development of models that can capture small-scale and heterogeneous
fields not captured by earlier models based on coarser datasets, but this
hypothesis has not been tested as higher-resolution map products have been
published. Most studies comparing land cover maps were published before
several recent maps became available, focused on a small subset of maps,
focused on broad land cover classes and not specifically agriculture, or used
reference datasets sampled from large areas or lacking statistically rigorous
sampling, and thus did not provide country-scale evaluations useful for
assessing the suitability of maps for national crop monitoring.

This study provides a quantitative evaluation and intercomparison of 11
publicly available land cover maps to assess their suitability for cropland
classification and EO-based agriculture monitoring in Africa. We summarize
important metadata for each map in Table 3. The maps span a range of spatial
resolutions (10 m/px to 1000 m/px) and years (2009 to 2020). In order from
finest to coarsest spatial resolution, they are Digital Earth Africa Cropland
Extent,[27] Dynamic World,[21] Esri LULC,[23] ESA WorldCover,[22] CCI Land
Cover Africa,[28] GFSAD Global Cropland Extent,[29] Nabil et al.,[24]
GLAD,[30] Copernicus Land Cover,[31] ESA GlobCover,[32] and ASAP Crop
Mask.[11] The main goals of this study are to: 1) assess the accuracy of each
map using statistically rigorous reference datasets, 2) quantify and visualize
consensus across all maps and pairwise agreement between maps, 3) understand
the influence of spatial resolution and temporal relevance in map accuracy,
and 4) demonstrate how the choice of a map can affect downstream
interpretations of agriculture conditions. We evaluated the accuracy of each
map using reference datasets for 8 countries selected to span the diverse
agriculture of Sub-Saharan Africa (Kenya, Malawi, Mali, Rwanda, Tanzania,
Togo, Uganda, and Zambia) and evaluation protocols based on best
practices.[33] We quantified the consensus between maps using all pixels
within each country boundary (after resampling all maps to a common resolution
of 10 meters per pixel). In addition to our analysis, we make the following
contributions:

  1. 1.

A high quality reference dataset of 3,386 samples from 8 countries in Sub-
Saharan Africa, which can be used as a common benchmark for assessing cropland
accuracy in future land cover maps

  2. 2.

A Google Earth Engine App to enable users to visualize and compare the maps in
this study

  3. 3.

A publicly-accessible code repository to facilitate evaluation for new
countries or datasets not included in this study

We found very low consensus across the 11 compared maps when predicting
cropland—all maps unanimously agree on a cropland prediction in fewer than
0.5% of pixels in each of the 8 countries studied, which is much smaller than
the estimated percentage of land used for agriculture in each country.[34]
There is a large disparity in the magnitude of the performance metrics between
countries for all maps, with average F1 scores across all maps ranging from as
low as $0.20\pm 0.08$ for Mali to $0.71\pm 0.01$ for Tanzania; the average F1
score is less than 0.6 for six out of eight countries. We show that a majority
vote ensemble that combines the predictions of all of the maps performs better
than most, but not all, individual maps. We show that performance is overall
weakly correlated with spatial resolution and temporal mismatch for maps with
resolution $\leq 100$ m/px and maps within 5 years of the reference data year.
We hope this analysis will help users determine the most suitable map for
their needs and encourage future work to focus on resolving inconsistencies
between maps and improving accuracy in poorly-classified regions.

## Results

### Accuracy assessment

Table 1 reports the overall accuracy, F1 score, precision (user’s accuracy),
and recall (producer’s accuracy) with associated standard errors from
evaluating each of the maps using the reference datasets summarized in Table
4. Refer to the Methods section for a definition of each metric and derivation
of standard errors. We also present the results from the majority vote
ensemble of all maps. Figure 1 shows a visual illustration of the results in
Table 1. Given results from four different metrics and 8 different countries,
how can we conclude which map is “best” overall? The map that most frequently
has the highest score across all metrics and countries is Esri LULC (7 bold
blue values in Table 1, not including the Mean), followed by Digital Earth
Africa (6 bold blue values). The map that most frequently has the highest or
second-highest score across all metrics and countries is WorldCover (19 bold
blue or bold black values in Table 1, not including the Mean). However,
summarizing the results in this way does not account for the disparities
between metrics that can indicate poor overall performance. For example, Esri
has the highest accuracy in Mali but all other metrics are 0 for the same
dataset; since only about 2% of the points have a true label of crop, accuracy
poorly describes the overall performance, evidenced by the Esri map achieving
0.98 accuracy even though it classified all of the reference points as non-
crop. F1 score is often considered a better metric than overall accuracy to
describe the overall performance because it combines two more descriptive
metrics, precision and recall. The Majority Vote ensemble achieved the highest
mean F1 score over all countries, closely followed by GLAD. WorldCover
achieved the highest mean precision (user’s accuracy) score followed by GLAD.
Digital Earth Africa achieved the highest mean recall (producer’s accuracy)
score, followed by GFSAD and Nabil et al.[24]. Considering the number of mean
metrics for which a map scores highest or second highest, GLAD comes in first
(second-highest metric for 3 out of 4 metrics), World Cover in second (highest
metric for 2 out of 4 metrics), and the Majority Vote comes in third (highest
or second-highest metric for 2 out of 4 metrics). The lowest performing maps
were Copernicus, GlobCover, ASAP, ESA-CCI, and Dynamic World.

There is a large disparity in the magnitude of the performance metrics between
countries. The last column of Table 1 reports each metric averaged across all
maps for each country evaluated. The average F1 score ranges from as low as
$0.20\pm 0.08$ for Mali to $0.71\pm 0.01$ for Tanzania. The average F1 score
falls below 0.6 for all but two of the eight countries (Rwanda and Tanzania).
The precision score is also very low for some countries, particularly Mali,
with an average precision of $0.14\pm 0.03$. This illustrates that even though
some maps may perform better than others, overall cropland classification in
these countries remains challenging.

Country | Metric | Majority Vote | Esri | Dynamic World | DEA | WorldCover | ESA-CCI | Nabil | GFSAD | GLAD | Copernicus | GlobCover | ASAP | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Kenya | Accuracy | $\bm{0.95\pm 0.01}$ | $\bm{0.96\pm 0.01}$ | $0.89\pm 0.01$ | $0.92\pm 0.01$ | $0.94\pm 0.01$ | $0.89\pm 0.01$ | $0.92\pm 0.01$ | $0.93\pm 0.01$ | $\bm{0.96\pm 0.01}$ | $0.91\pm 0.01$ | $0.78\pm 0.01$ | $0.90\pm 0.01$ | $0.91\pm 0.00$
F1 | $\bm{0.68\pm 0.19}$ | $0.66\pm 0.23$ | $0.42\pm 0.18$ | $0.54\pm 0.20$ | $0.41\pm 0.24$ | $0.48\pm 0.15$ | $0.61\pm 0.14$ | $0.62\pm 0.15$ | $\bm{0.73\pm 0.21}$ | $0.49\pm 0.19$ | $0.29\pm 0.11$ | $0.36\pm 0.20$ | $0.52\pm 0.04$
Precision (UA) | $0.58\pm 0.07$ | $\bm{0.68\pm 0.08}$ | $0.31\pm 0.06$ | $0.43\pm 0.06$ | $\bm{0.61\pm 0.12}$ | $0.34\pm 0.05$ | $0.45\pm 0.06$ | $0.47\pm 0.06$ | $\bm{0.68\pm 0.07}$ | $0.38\pm 0.06$ | $0.18\pm 0.03$ | $0.30\pm 0.06$ | $0.45\pm 0.02$
Recall (PA) | $\bm{0.83\pm 0.06}$ | $0.64\pm 0.07$ | $0.61\pm 0.08$ | $0.72\pm 0.07$ | $0.31\pm 0.06$ | $\bm{0.83\pm 0.06}$ | $\bm{0.94\pm 0.04}$ | $\bm{0.94\pm 0.04}$ | $0.78\pm 0.06$ | $0.69\pm 0.07$ | $0.72\pm 0.07$ | $0.44\pm 0.08$ | $0.70\pm 0.01$
Malawi | Accuracy | $\bm{0.86\pm 0.03}$ | $0.82\pm 0.03$ | $0.82\pm 0.03$ | $0.84\pm 0.03$ | $\bm{0.89\pm 0.02}$ | $0.82\pm 0.03$ | $0.80\pm 0.03$ | $0.80\pm 0.03$ | $\bm{0.86\pm 0.03}$ | $0.85\pm 0.03$ | $0.76\pm 0.03$ | $0.68\pm 0.03$ | $0.82\pm 0.00$
F1 | $\bm{0.68\pm 0.23}$ | $0.44\pm 0.26$ | $0.33\pm 0.24$ | $0.66\pm 0.23$ | $\bm{0.74\pm 0.22}$ | $0.66\pm 0.20$ | $0.61\pm 0.21$ | $0.62\pm 0.20$ | $0.62\pm 0.24$ | $0.66\pm 0.23$ | $0.49\pm 0.23$ | $0.33\pm 0.22$ | $0.57\pm 0.02$
Precision (UA) | $0.62\pm 0.08$ | $0.58\pm 0.12$ | $\bm{0.64\pm 0.15}$ | $0.59\pm 0.08$ | $\bm{0.71\pm 0.08}$ | $0.54\pm 0.07$ | $0.50\pm 0.07$ | $0.50\pm 0.07$ | $\bm{0.71\pm 0.09}$ | $0.61\pm 0.08$ | $0.43\pm 0.08$ | $0.29\pm 0.07$ | $0.56\pm 0.02$
Recall (PA) | $0.74\pm 0.07$ | $0.35\pm 0.07$ | $0.23\pm 0.05$ | $0.74\pm 0.07$ | $0.77\pm 0.07$ | $\bm{0.84\pm 0.06}$ | $0.77\pm 0.07$ | $\bm{0.81\pm 0.07}$ | $0.55\pm 0.07$ | $0.71\pm 0.07$ | $0.58\pm 0.08$ | $0.39\pm 0.08$ | $0.62\pm 0.01$
Mali | Accuracy | $\bm{0.95\pm 0.01}$ | $\bm{0.98\pm 0.01}$ | $0.94\pm 0.01$ | $\bm{0.95\pm 0.01}$ | $\bm{0.95\pm 0.01}$ | $0.80\pm 0.01$ | $0.91\pm 0.01$ | $0.90\pm 0.01$ | $0.94\pm 0.01$ | $0.92\pm 0.01$ | $0.67\pm 0.01$ | $0.86\pm 0.01$ | $0.90\pm 0.00$
F1 | $0.28\pm 0.34$ | $0.00\pm 0.00$ | $0.12\pm 0.24$ | $\bm{0.46\pm 0.25}$ | $\bm{0.43\pm 0.28}$ | $0.10\pm 0.11$ | $0.20\pm 0.22$ | $0.15\pm 0.19$ | $0.34\pm 0.26$ | $0.18\pm 0.23$ | $0.08\pm 0.07$ | $0.11\pm 0.14$ | $0.20\pm 0.08$
Precision (UA) | $0.21\pm 0.10$ | $0.00\pm 0.00$ | $0.09\pm 0.06$ | $\bm{0.31\pm 0.09}$ | $\bm{0.30\pm 0.09}$ | $0.05\pm 0.02$ | $0.13\pm 0.05$ | $0.10\pm 0.05$ | $0.23\pm 0.08$ | $0.12\pm 0.06$ | $0.04\pm 0.02$ | $0.06\pm 0.03$ | $0.14\pm 0.03$
Recall (PA) | $0.40\pm 0.15$ | $0.00\pm 0.00$ | $0.20\pm 0.12$ | $\bm{0.90\pm 0.09}$ | $\bm{0.80\pm 0.12}$ | $0.50\pm 0.16$ | $0.50\pm 0.15$ | $0.40\pm 0.15$ | $0.70\pm 0.14$ | $0.40\pm 0.15$ | $0.60\pm 0.15$ | $0.40\pm 0.15$ | $0.48\pm 0.04$
Rwanda | Accuracy | $0.79\pm 0.04$ | $0.81\pm 0.04$ | $0.71\pm 0.04$ | $\bm{0.86\pm 0.03}$ | $\bm{0.87\pm 0.03}$ | $0.70\pm 0.04$ | $0.75\pm 0.04$ | $0.75\pm 0.04$ | $0.76\pm 0.04$ | $0.78\pm 0.04$ | $0.63\pm 0.05$ | $0.61\pm 0.05$ | $0.75\pm 0.01$
F1 | $0.72\pm 0.19$ | $0.71\pm 0.15$ | $0.44\pm 0.06$ | $\bm{0.83\pm 0.16}$ | $\bm{0.82\pm 0.15}$ | $0.61\pm 0.20$ | $0.70\pm 0.19$ | $0.70\pm 0.19$ | $0.62\pm 0.18$ | $0.71\pm 0.19$ | $0.57\pm 0.19$ | $0.30\pm 0.20$ | $0.64\pm 0.04$
Precision (UA) | $0.78\pm 0.07$ | $\bm{0.93\pm 0.05}$ | $\bm{1.00\pm 0.00}$ | $0.80\pm 0.06$ | $0.91\pm 0.05$ | $0.64\pm 0.08$ | $0.67\pm 0.07$ | $0.67\pm 0.07$ | $0.84\pm 0.07$ | $0.73\pm 0.07$ | $0.53\pm 0.07$ | $0.53\pm 0.12$ | $0.75\pm 0.03$
Recall (PA) | $0.67\pm 0.06$ | $0.58\pm 0.05$ | $0.28\pm 0.03$ | $\bm{0.86\pm 0.05}$ | $\bm{0.74\pm 0.05}$ | $0.58\pm 0.06$ | $0.72\pm 0.06$ | $0.72\pm 0.06$ | $0.49\pm 0.05$ | $0.70\pm 0.06$ | $0.63\pm 0.06$ | $0.21\pm 0.05$ | $0.60\pm 0.01$
Tanzania | Accuracy | $\bm{0.90\pm 0.01}$ | $0.76\pm 0.01$ | $0.76\pm 0.01$ | $0.83\pm 0.01$ | $\bm{0.89\pm 0.01}$ | $0.78\pm 0.01$ | $\bm{0.89\pm 0.01}$ | $0.88\pm 0.01$ | $0.87\pm 0.01$ | $0.86\pm 0.01$ | $0.70\pm 0.01$ | $0.77\pm 0.01$ | $0.82\pm 0.00$
F1 | $\bm{0.84\pm 0.04}$ | $0.49\pm 0.03$ | $0.50\pm 0.04$ | $0.73\pm 0.06$ | $\bm{0.83\pm 0.05}$ | $0.66\pm 0.06$ | $\bm{0.83\pm 0.05}$ | $0.82\pm 0.05$ | $0.78\pm 0.05$ | $0.79\pm 0.05$ | $0.54\pm 0.07$ | $0.71\pm 0.06$ | $0.71\pm 0.01$
Precision (UA) | $\bm{0.96\pm 0.01}$ | $\bm{0.99\pm 0.01}$ | $0.93\pm 0.02$ | $0.84\pm 0.02$ | $0.94\pm 0.01$ | $0.73\pm 0.02$ | $0.90\pm 0.02$ | $0.89\pm 0.02$ | $0.95\pm 0.01$ | $0.86\pm 0.02$ | $0.60\pm 0.03$ | $0.65\pm 0.02$ | $0.85\pm 0.01$
Recall (PA) | $0.75\pm 0.02$ | $0.33\pm 0.01$ | $0.34\pm 0.01$ | $0.65\pm 0.02$ | $0.75\pm 0.02$ | $0.60\pm 0.02$ | $\bm{0.77\pm 0.02}$ | $\bm{0.77\pm 0.02}$ | $0.67\pm 0.02$ | $0.72\pm 0.02$ | $0.49\pm 0.02$ | $\bm{0.78\pm 0.02}$ | $0.64\pm 0.00$
Togo | Accuracy | $0.84\pm 0.03$ | $0.73\pm 0.03$ | $0.75\pm 0.03$ | $0.83\pm 0.03$ | $\bm{0.86\pm 0.03}$ | $0.78\pm 0.03$ | $0.79\pm 0.03$ | $0.78\pm 0.03$ | $\bm{0.86\pm 0.03}$ | $0.79\pm 0.03$ | $0.74\pm 0.03$ | $0.64\pm 0.03$ | $0.78\pm 0.00$
F1 | $0.63\pm 0.15$ | $0.07\pm 0.11$ | $0.21\pm 0.11$ | $0.62\pm 0.17$ | $0.70\pm 0.15$ | $0.58\pm 0.19$ | $0.61\pm 0.19$ | $0.62\pm 0.19$ | $\bm{0.71\pm 0.16}$ | $0.59\pm 0.19$ | $0.43\pm 0.20$ | $0.45\pm 0.18$ | $0.52\pm 0.03$
Precision (UA) | $\bm{0.89\pm 0.06}$ | $0.67\pm 0.33$ | $0.86\pm 0.14$ | $0.83\pm 0.07$ | $\bm{0.91\pm 0.05}$ | $0.62\pm 0.07$ | $0.62\pm 0.07$ | $0.60\pm 0.07$ | $0.88\pm 0.06$ | $0.64\pm 0.07$ | $0.56\pm 0.09$ | $0.39\pm 0.06$ | $0.71\pm 0.08$
Recall (PA) | $0.49\pm 0.05$ | $0.04\pm 0.02$ | $0.12\pm 0.02$ | $0.49\pm 0.05$ | $0.57\pm 0.05$ | $0.55\pm 0.06$ | $0.61\pm 0.06$ | $\bm{0.63\pm 0.06}$ | $0.59\pm 0.05$ | $0.55\pm 0.06$ | $0.35\pm 0.05$ | $0.53\pm 0.06$ | $0.46\pm 0.01$
Uganda | Accuracy | $0.86\pm 0.02$ | $0.87\pm 0.02$ | $\bm{0.88\pm 0.02}$ | $0.77\pm 0.02$ | $\bm{0.92\pm 0.02}$ | $0.76\pm 0.02$ | $0.83\pm 0.02$ | $0.83\pm 0.02$ | $\bm{0.88\pm 0.02}$ | $0.78\pm 0.02$ | $0.61\pm 0.02$ | $0.55\pm 0.02$ | $0.80\pm 0.00$
F1 | $\bm{0.51\pm 0.24}$ | $0.39\pm 0.27$ | $0.31\pm 0.28$ | $0.43\pm 0.18$ | $\bm{0.51\pm 0.26}$ | $0.40\pm 0.19$ | $0.44\pm 0.23$ | $0.46\pm 0.22$ | $\bm{0.54\pm 0.25}$ | $0.40\pm 0.20$ | $0.30\pm 0.13$ | $0.28\pm 0.12$ | $0.41\pm 0.05$
Precision (UA) | $0.41\pm 0.08$ | $0.40\pm 0.10$ | $\bm{0.46\pm 0.14}$ | $0.29\pm 0.06$ | $\bm{0.77\pm 0.12}$ | $0.28\pm 0.06$ | $0.35\pm 0.07$ | $0.35\pm 0.07$ | $\bm{0.46\pm 0.08}$ | $0.29\pm 0.06$ | $0.19\pm 0.04$ | $0.17\pm 0.03$ | $0.37\pm 0.03$
Recall (PA) | $0.65\pm 0.09$ | $0.38\pm 0.08$ | $0.23\pm 0.07$ | $\bm{0.77\pm 0.08}$ | $0.38\pm 0.07$ | $0.69\pm 0.09$ | $0.62\pm 0.09$ | $0.65\pm 0.09$ | $0.65\pm 0.08$ | $0.65\pm 0.09$ | $\bm{0.77\pm 0.08}$ | $\bm{0.77\pm 0.08}$ | $0.60\pm 0.01$
Zambia | Accuracy | $\bm{0.98\pm 0.01}$ | $\bm{0.98\pm 0.01}$ | $\bm{0.97\pm 0.01}$ | $0.94\pm 0.01$ | $\bm{0.97\pm 0.01}$ | $0.93\pm 0.01$ | $0.95\pm 0.01$ | $0.95\pm 0.01$ | $\bm{0.98\pm 0.01}$ | $0.95\pm 0.01$ | $0.91\pm 0.01$ | $0.67\pm 0.01$ | $0.93\pm 0.00$
F1 | $\bm{0.74\pm 0.27}$ | $\bm{0.74\pm 0.28}$ | $0.61\pm 0.30$ | $0.57\pm 0.18$ | $0.65\pm 0.30$ | $0.46\pm 0.24$ | $0.56\pm 0.26$ | $0.58\pm 0.25$ | $\bm{0.71\pm 0.28}$ | $0.57\pm 0.26$ | $0.19\pm 0.22$ | $0.15\pm 0.08$ | $0.54\pm 0.06$
Precision (UA) | $0.70\pm 0.10$ | $\bm{0.78\pm 0.10}$ | $\bm{0.77\pm 0.12}$ | $0.40\pm 0.07$ | $0.61\pm 0.10$ | $0.34\pm 0.07$ | $0.44\pm 0.09$ | $0.46\pm 0.09$ | $0.68\pm 0.10$ | $0.45\pm 0.09$ | $0.15\pm 0.06$ | $0.08\pm 0.02$ | $0.49\pm 0.03$
Recall (PA) | $\bm{0.80\pm 0.08}$ | $0.70\pm 0.09$ | $0.50\pm 0.09$ | $\bm{0.95\pm 0.05}$ | $0.70\pm 0.09$ | $0.70\pm 0.10$ | $0.75\pm 0.09$ | $\bm{0.80\pm 0.09}$ | $0.75\pm 0.09$ | $0.75\pm 0.09$ | $0.25\pm 0.09$ | $0.70\pm 0.10$ | $0.70\pm 0.01$
Mean | Accuracy | $\bm{0.90\pm 0.01}$ | $0.88\pm 0.01$ | $0.85\pm 0.01$ | $0.88\pm 0.01$ | $\bm{0.92\pm 0.01}$ | $0.82\pm 0.01$ | $0.87\pm 0.01$ | $0.86\pm 0.01$ | $\bm{0.90\pm 0.01}$ | $0.87\pm 0.01$ | $0.74\pm 0.01$ | $0.72\pm 0.02$ | –
F1 | $\bm{0.60\pm 0.13}$ | $0.42\pm 0.24$ | $0.34\pm 0.10$ | $0.56\pm 0.07$ | $0.58\pm 0.12$ | $0.44\pm 0.06$ | $0.53\pm 0.07$ | $0.53\pm 0.07$ | $\bm{0.59\pm 0.08}$ | $0.52\pm 0.10$ | $0.32\pm 0.07$ | $0.31\pm 0.06$ | –
Precision (UA) | $0.60\pm 0.04$ | $0.61\pm 0.16$ | $0.57\pm 0.06$ | $0.51\pm 0.02$ | $\bm{0.66\pm 0.04}$ | $0.40\pm 0.02$ | $0.47\pm 0.02$ | $0.47\pm 0.02$ | $\bm{0.62\pm 0.03}$ | $0.48\pm 0.03$ | $0.30\pm 0.03$ | $0.28\pm 0.03$ | –
Recall (PA) | $0.64\pm 0.06$ | $0.36\pm 0.05$ | $0.32\pm 0.06$ | $\bm{0.72\pm 0.06}$ | $0.58\pm 0.05$ | $0.61\pm 0.05$ | $\bm{0.70\pm 0.06}$ | $\bm{0.70\pm 0.06}$ | $0.64\pm 0.06$ | $0.64\pm 0.06$ | $0.49\pm 0.04$ | $0.56\pm 0.05$ | –

Table 1: Performance metrics and associated standard errors for each map and
reference dataset compared in this study, including a majority vote ensemble
of all 11 maps. The highest value in each row is in bold blue and the second-
highest in bold black. Figure 1: Visual depiction of results in Table 1. Some
error bars are not visible due to small errors.

### Map agreement

We quantified the consensus between cropland classifications across all of the
compared maps as well as the agreement between all pairs of maps. We
characterized map agreement in 3 ways: 1) quantified as a percentage of total
pixels with agreement across all maps, 2) illustrated visually as a map to
show spatial patterns of agreement and disagreement, and 3) quantified as a
percentage of total pixels that agree for pairwise maps as an agreement
matrix.

#### Consensus across all maps

Table 2 quantifies the agreement between crop and non-crop predictions made by
all maps as a percentage of the total pixels in the 10 m/pixel resolution
consensus map. The highest overall consensus, i.e., the highest percentage of
all locations in which all maps predict the same class, was in Mali (69.9%)
and Kenya (60.6%). The lowest overall consensus was in Rwanda (15.8%) and
Malawi (21.8%). The crop consensus, i.e., the percentage of locations in which
all maps predict cropland, was very low—less than 0.5% for all countries; the
highest crop consensus was in Kenya (0.4%) and Uganda (0.4%). The non-crop
consensus, i.e., the percentage of locations in which all maps predict non-
crop, was approximately equivalent to the overall consensus because non-crop
predictions constituted the majority of consensus pixels. This shows there is
greater disagreement among maps in predicting where cropland is compared to
where cropland is not. The “split prediction” column in Table 2 gives the
percentage of pixel locations where the consensus between all maps is split;
i.e., 5 or 6 maps predict crop while the remaining 6 or 5 maps predict non-
crop. These are the regions of the lowest consensus across all maps. The split
prediction percentage was highest in Rwanda (24.0%) and lowest in Mali (2.9%).

Country | All predict same class | All predict crop | Split prediction | None predict crop (all non-crop)
---|---|---|---|---
Kenya | 60.6 | 0.4 | 5.0 | 60.2
Malawi | 21.8 | 0.0 | 17.5 | 21.8
Mali | 69.9 | 0.0 | 2.9 | 69.9
Rwanda | 15.8 | 0.1 | 24.0 | 15.8
Tanzania | 44.4 | 0.0 | 9.0 | 44.4
Togo | 23.8 | 0.2 | 17.0 | 23.6
Uganda | 29.1 | 0.4 | 11.6 | 28.8
Zambia | 49.5 | 0.0 | 4.4 | 49.5
Table 2: Agreement between 11 compared maps as percent of total pixels. “Split
prediction” describes pixels where the prediction across all maps is
approximately evenly split (5 predict crop but 6 predict non-crop, or 6
predict crop but 5 predict non-crop).

#### Spatial visualization of map agreement

Figure 2 visualizes the spatial distribution of agreement between the 11 maps,
where colors indicate the number of maps that predict the crop class in each
10m pixel location. Blue pixels (where the map value is 11) indicate locations
where all maps unanimously or near-unanimously predict cropland (which
constitutes less than 0.5% of pixels in each country, as described in the
previous section). Red pixels (where the map value is 0) indicate locations
where all maps unanimously or near-unanimously predict the non-crop class. The
fraction of unanimous non-crop pixels is especially high in Kenya, Zambia,
Mali, and Tanzania, where the majority of the country’s land area is not used
for agriculture.[34] Yellow pixels indicate locations of high disagreement
between maps where approximately half of the maps predict crop and half
predict non-crop. While some regions in each map appear primarily yellow,
yellow-orange, or yellow-green (indicating homogeneous areas of disagreement),
most of the disagreement regions are on the edges surrounding high agreement
pixels of either class. This suggests that most zones of high disagreement lie
on the boundaries between crop and non-crop regions, where transitions between
land cover types might make discrete classification challenging. These zone
are also likely locations of cropland expansion and are important to
accurately identify for efficient resource allocation, crop rotation, pest
management, environmental conservation, land use planning, and monitoring.

Figure 2: Map showing the number of maps that predict cropland in each
10m/pixel location. Panel A shows the maps in the context of the African
continent while Panel B gives larger views of the maps (countries not to
scale). Blue and red indicate regions of high or unanimous agreement between
maps for the crop and non-crop classes, respectively. Yellow indicates regions
of high disagreement between maps.

#### Pairwise map agreement

Figure 3 quantifies the pairwise agreement between each map compared in this
study, and the majority vote ensemble map, using a symmetric agreement matrix
for each country. Each cell reports the fraction of pixels that agree (predict
the same class) for each pair of compared maps, ordered by spatial resolution
(high to low). Since maps are ordered by spatial resolution (highest to
coarsest, i.e., increasing ground sampling distance), blocky patterns about
the diagonal indicate that maps with the same or similar resolution have the
highest pairwise agreement. Diagonal cells are set to zero. Figure 4 shows the
mean agreement matrix (average agreement matrices for each country). All
matrices use the same color bar ranging from 0 to 1, so relative agreement
fractions can be compared across all countries. The pairwise agreement between
maps is low for Rwanda, Togo, and Malawi and relatively high for Zambia and
Mali. Overall, the maps that share the highest agreement are Nabil et al. and
GFSAD, Nabil et al. and majority vote, and Dynamic World and Esri. The Nabil
et al. map has high agreement with the majority vote map with GFSAD because it
combines the predictions of GFSAD and two other maps used in the study.

Figure 4 shows the order of pairwise maps ranked from lowest to the highest
mean pairwise agreement within each row. There is high agreement between Nabil
et al.[24] and the ESA-CCI, Copernicus, and GFSAD crop masks, which is
expected since the Nabil et al. mask is a combination of these maps. ASAP has
the overall lowest pairwise agreement with other maps, and ESA-CCI has
particularly low agreement with all other maps in Mali. Surprisingly,
GlobCover agrees most with Dynamic World and Esri. Digital Earth Africa,
WorldCover, GLAD, and Copernicus all agree most with the majority vote map,
meaning they are most consistent with the majority prediction across all the
maps. Excluding the majority vote, GLAD agrees most with WorldCover, and vice
versa, and Digital Earth Africa agrees most with GLAD.

Figure 3: Fraction of pixels that agree (predict same class) for each pair of
compared maps, ordered by spatial resolution (high to low). Diagonal entries
are set to 0. See online version for high resolution. Figure 4: Left matrix
shows the average fraction of pixels that agree (predict same class) for each
pair of compared maps, averaged over all countries (mean of matrices in Figure
3). Right matrix shows the ranking of lowest (1) to highest (11) pairwise
agreement between maps within each row. Diagonal entries are set to 0.

### Accuracy vs. spatial resolution

Previous work has hypothesized that land cover products based on higher
resolution satellite observations (e.g., 10 m/pixel Sentinel-2) should have
improved cropland mapping performance in Sub-Saharan Africa as a result of
small and fragmented field sizes being more clear in the higher-resolution
images,[13] though other work found that small field sizes were not the
primary influencing factor on the disagreement between maps.[12] Figure 5
shows a scatter plot of the performance metric for each map, averaged across
all countries, vs. spatial resolution (in m/px). The title of each plot
reports the Pearson product-moment correlation coefficient. Accuracy and
precision (user’s accuracy) show a strong negative correlation with spatial
resolution (R2 of -0.81 and -0.7, respectively); i.e., accuracy and precision
scores tend to be higher for higher resolution maps. However, we found no
correlation between recall (producer’s accuracy) and spatial resolution
(R2=-0.09). We found a moderate negative correlation between F1 score and
spatial resolution. This suggests that finer spatial resolution seems to help
reduce false positives (other land cover types incorrectly classified as
cropland) but not false negatives (cropland incorrectly classified as other
land cover). When interpreting the correlation between spatial resolution and
performance metrics, it’s important to consider the impact of the ASAP and
GlobCover maps, which have different spatial resolutions compared to other
maps. ASAP has a resolution of 1000 m/px and GlobCover has a resolution of 300
m/px, while the remaining maps range from 10-100 m/px. After excluding ASAP
and GlobCover, the correlation between spatial resolution and each metric
became R2 of -0.22 for accuracy, 0.26 for F1, -0.45 for precision, and 0.52
for recall. In other words, for maps with a spatial resolution of 100 m/px and
below, we find a weak to moderate positive correlation between F1 and recall
and spatial resolution, meaning coarser spatial resolution products tend to
have higher F1 and recall scores, but a weak to moderate negative correlation
between accuracy and precision, meaning finer spatial resolution products tend
to have higher accuracy and precision. The intended use case of the map should
influence which metrics should be weighted most highly in choosing the most
suitable map.

Figure 5: Scatter plots showing mean performance metrics (averaged across all
countries) vs. spatial resolution (in m/pixel).

### Accuracy vs. temporal mismatch

Previous work has stressed the importance of using a crop mask with the same
(or as close as possible to the same) year as the downstream analysis, e.g.,
when assessing in-season crop conditions.[35] Figure 6 plots each metric
against the temporal mismatch (number of years difference) between the
reference data year and the map year (calculated as the absolute difference
between the map year and the reference data year). We find a weak to moderate
negative correlation between accuracy, F1, precision, and temporal mismatch
(R2 of -0.40, -0.19, and -0.31, respectively), indicating that higher
performance somewhat correlates with closer temporal matching. We found no
correlation (R2 of 0.07) between recall (producer’s accuracy) and temporal
mismatch.

As in the case of spatial resolution, it is important to consider the effect
of outliers in this assessment. The nominal year for the GlobCover map is 2009
and is an outlier among the map years, which otherwise range from 2017 to
2020. When GlobCover is excluded, the correlation between temporal mismatch
and each metric is R2 of -0.22 for accuracy, 0.01 for F1, -0.24 for precision,
and 0.24 for recall. In other words, for maps within 5 years of the reference
data year, the correlation becomes weaker between temporal mismatch and
accuracy, F1, and precision; however, there is also a weak positive
correlation between temporal mismatch and recall. These results suggest that
maps within a few years of reference/analysis year may be sufficient, and an
exact temporal match may not be needed for all analyses, though the importance
of temporal matching between map year and reference/analysis year will depend
on the intended use of the map.

Figure 6: Scatter plots showing performance metrics vs. temporal mismatch
between the map year and the reference data year (unit is the number of
years).

### Impact of map differences in downstream use

Cropland maps are commonly used as masks to select the cropland pixels from
satellite datasets for crop conditions assessments, yield estimation, and
other downstream analyses.[35] For example, in the Global Agricultural
Monitoring (GLAM) System, a user can assess current season crop conditions by
comparing the normalized difference vegetation index (NDVI) time series of the
current year (a measure of vegetation condition throughout the year) to that
of a typical year or range of years (representing average vegetation
conditions).[10] In Figure 7, we illustrate how the choice of crop mask can
affect the NDVI time series and, thus, the end-user’s interpretation of
vegetation conditions. Each plot shows a time series of the mean NDVI from the
MODIS MOD13A1.061 Terra Vegetation Indices 16-Day Global 500m data product,
created by masking the image collection using each map and computing the mean
of all pixels within the country boundary (panel A) and an administrative
level 1 (admin1) region (panel B). Admin1 refers to the largest subnational
administrative unit within a country, e.g., states in the United States.[36]
We chose specific admin1 regions based on areas of high disagreement between
maps to highlight discrepancies that may arise due to map differences. These
plots reveal that GlobCover substantially overestimates crop conditions as
measured by NDVI compared to other maps, particularly in Malawi, Tanzania,
Rwanda, and Zambia. In some countries such as Malawi, Tanzania, and Zambia,
most masks result in similar time series, but there is a large spread in
others such as Kenya, Mali, and Togo. Notably, Kenya had the highest
percentage of pixels in agreement across all maps of the 8 countries (Table
2), but the greatest differences between the maps when used as crop masks for
vegetation indices. This suggests it is important to consider the locations
and other characteristics of discrepancies between maps and not just the
magnitude of those discrepancies in future work. We used the MODIS data
product since this is commonly used by current agriculture monitoring systems
such as GLAM.[10] Since the MODIS NDVI data product used has 500 m/pixel
resolution, the time series are aggregated using each crop mask resampled to
500 m/pixel resolution to match the NDVI data, thus most of the masks are
downscaled to a much coarser resolution than their true resolution. If a
higher-resolution NDVI data product (such as Landsat or Sentinel-2) were used,
we expect the time series resulting from each mask would diverge even more
substantially than in the downscaled MODIS time series.

Figure 7: Time series of 16-day MODIS MOD13A1.061 Terra Vegetation Indices
averaged within (A) each country and (B) an administrative level 1 region,
masked using each crop mask assessed in this study.

## Discussion

### Which map is best?

No single map can be considered optimal across all evaluations, metrics, and
countries, thereby leaving users of these maps uncertain about the most
suitable choice for a given use case or application. The GLAD map had good
performance for most countries in the reference dataset evaluation, had high
agreement with the majority vote, and had similar NDVI time series to the
majority of maps. GlobCover had low performance for most countries in the
reference dataset evaluation, moderate agreement with the majority vote, and
NDVI time series that differed substantially from the majority of maps. Thus
GLAD is likely to be a sound choice for many use cases, while GlobCover should
be used cautiously. The best map for a given use case should be chosen based
on the accuracy and relevance of that map for the specific use case, which may
require users to develop use case-specific evaluation experiments for choosing
the best map. Users should identify the right map based on the region-specific
evaluation data or their intended use case. Users should also visually inspect
candidate maps in their intended study area to qualitatively assess map
suitability, as quantitative metrics may not fully capture map performance for
a given use case.[37]

### Regional vs. global models

Researchers have been shifting toward training single, global models that are
optimized to predict land cover anywhere in the world. Global models compared
in this study are Esri, Dynamic World, and WorldCover. Global models are
appealing because they simplify computation and enable models to learn from
more diverse data, but these benefits may come at the expense of high intra-
class variance that makes learning to predict classes accurately across a wide
range of agro-ecological conditions challenging. In contrast, regional models
are trained to optimize classification performance for a regional sub-group,
such as an agro-ecological zone, a global grid tile, or a country (see Table
3). Table 1 shows that models trained to optimize regional performance tend to
outperform global models. In addition to training a global model, the Esri and
Dynamic World maps are the only models that do not include temporal features
in the input. Classification is based on segmentation of a single image (or
composite image). Thus, spatial and spectral information is available to the
model but not temporal. Temporal information is important for identifying
cropland and could be another factor contributing to improved performance.

### Ensemble models

Since ensembles often perform better than individual base classifiers, it may
be surprising that the Majority Vote ensemble does not give the best overall
performance. However, necessary conditions for an ensemble of base classifiers
to perform better than individual base classifiers are that the base
classifiers are independent of each other (make independent errors) and
perform better than random guessing,[38] which is not necessarily true for all
of the maps in this study. The Nabil et al.[24] crop mask is also an ensemble;
it combines four data products, three of which are also compared in this study
(GFSAD, ESA-CCI, and Copernicus). These maps are highly correlated (see Figure
3). Nabil et al. proposed the ensemble crop mask to provide a combined product
better than the individual maps. The Nabil et al.[24] ensemble mask performs
better overall than GFSAD, ESA-CCI, and Copernicus individually but is not
among the highest-performing maps of all those compared in this study.

### Future efforts

This study’s results show low consensus between the 11 compared maps in each
of the studied countries, particularly for the cropland class. Unanimous
agreement between maps on cropland locations is rare. In addition, the average
performance across all of the maps in each country is quite low. Most previous
efforts to create more accurate global land cover maps focus on creating new
methods and models that optimize performance over samples drawn globally. We
encourage future work focusing on targeted improvements that boost performance
for the lowest-performing sub-groups (e.g., countries or biomes) among
existing methods. This would help to resolve inconsistencies between existing
maps and reduce the disparity in performance across countries globally,
resulting in more geographically fair land cover maps and associated machine
learning models. The results from this study can be used to inform such future
work. For example, future efforts to create labeled datasets could focus on
low-performing countries (e.g., Mali) or collect samples from regions of high
disagreement between maps using random uniform sampling or stratified sampling
in which consensus levels define strata. Future work could extend this
analysis for more countries and maps using our publicly provided code. Future
work could also help direct efforts to improve map accuracy and provide useful
information to map users by identifying the types of land cover that are
commonly confused with cropland in these maps. Finally, we hope that this
study’s reference datasets and results can provide a useful benchmark for
evaluating future map products against a common baseline. We provide our
reference datasets to enable this benchmarking and for transparency but
emphasize that future inter-comparison with these results requires researchers
to use the reference datasets only for final, independent accuracy assessment
and not use them during the model training or map creation process.

## Methods

### Cropland and Land Cover Maps

We analyzed 11 global and continent-scale land cover and land use maps made
publicly available in recent years. These maps are summarized in Table 3,
which span a range of temporal availability, spatial resolutions, and
geographic coverage. Prior studies comparing available land cover maps have
not included several of these maps. Each map was created by making dense,
contiguous predictions over a geographic area using a trained machine learning
classifier (though the classifier used for each differs). We categorize these
map products as being produced using regional or global models. In the
regional model category, separate machine learning models are optimized for a
specific region (e.g., a country, continent, or agroecological zone) that
generates a predicted map for that region. These regional maps may then be
combined to cover a larger geographic area, such as a global map product. In
the global model category, a single machine learning model is optimized for
making predictions in all regions of the world, where data used for training
may be aggregated from many different regions. A single global model is then
used to make predictions everywhere to form one predicted global map.

All datasets were obtained from the Google Earth Engine asset listed in Table
3. The values of each map at the reference sample locations were obtained
using the reduceRegions() function in Google Earth Engine sampled at the
native resolution of each map. This value extraction is fully reproducible
using the code provided in the Code Availability section. All maps were
clipped to the border of each country in the study and converted to a binary
crop/non-crop map. Consensus and agreement analyses were performed after
resampling all maps to a common spatial resolution of 10 m/px. In the
subsequent sections, we summarize how each map was created and used in this
study.

Dataset | Year(s) | Res. (m/px) | Model Scale | Coverage | GEE Asset
---|---|---|---|---|---
Digital Earth Africa Cropland Extent[27, 39] | 2019 | 10 | AEZ | Continent | ee.ImageCollection("projects/sat-io/open-datasets/DEAF/CROPLAND-EXTENT/mask")
Dynamic World[21] | 2015-2023 | 10 | Global | Global | ee.ImageCollection("GOOGLE/DYNAMICWORLD/V1")
Esri LULC[23, 40] | 2017-2022 | 10 | Global | Global | ee.ImageCollection("projects/sat-io/open-datasets/landcover/ESRI_Global-LULC_10m_TS")
ESA WorldCover[22, 41] | 2020-2021 | 10 | Global | Global | ee.ImageCollection("ESA/WorldCover/v100")
CCI Land Cover Africa[28, 42] | 2016 | 20 | Continent | Continent | ee.Image("projects/sat-io/open-datasets/ESA/ESACCI-LC-L4-LC10-Map-20m-P1Y-2016-v10")
GFSAD Global Cropland Extent[29, 43] | 2015 | 30 | AEZ | Global | ee.ImageCollection("projects/sat-io/open-datasets/GFSAD/GCEP30")
Nabil et al.[24, 44] | 2016 | 30 | Mixed | Continent | ee.Image("projects/sat-io/open-datasets/landcover/AF_Cropland_mask_30m_2016_v3")
GLAD[30, 45] | 2003, 2007, 2011, 2015, 2019 | 30 | $1^{\circ}\times 1^{\circ}$ | Global | ee.ImageCollection("users/potapovpeter/Global_cropland_2019")
Copernicus Land Cover[31, 46] | 2015-2019 | 100 | Biome | Global | ee.ImageCollection("COPERNICUS/Landcover/100m/Proba-V-C3/Global")
ESA GlobCover[32, 47] | 2005, 2009 | 300 | Strata | Global | ee.Image("ESA/GLOBCOVER_L4_200901_200912_V2_3")
ASAP Crop Mask[11, 48] | 2017 | 1000 | Mixed | Global | ee.Image("users/sbaber/asap_mask_crop_v03")

Table 3: Map data products evaluated in this study.

#### Regional models

##### Digital Earth Africa.

The Digital Earth Africa Cropland Extent map[27] estimates crop extent at 10
m/pixel resolution for the year 2019 in Africa. The continent-scale map was
created by combining maps predicted for eight different agro-ecological zones
(AEZ) within the African continent covering Eastern, Western, Northern, Sahel,
Southern, Southeast, Central, and Indian Ocean regions of Africa. A separate
random forest model was trained using data sampled from each AEZ and used to
predict a complete map for that AEZ. The input feature vector contains hand-
crafted features (temporal statistics and geomedian composites) from a
12-month time series of Sentinel-2 multispectral observations and ancillary
datasets (topography and climatology). All maps and associated training and
validation datasets are for 2019 with 10 m/pixel spatial resolution. Digital
Earth Africa reported an overall accuracy of 90.3% in Eastern Africa (includes
Kenya), 83.6% in Western Africa (includes Togo), and 87.3% in Southeast Africa
(includes Malawi).

##### ESA CCI Land Cover Africa.

This 20 m/pixel land cover map of Africa for the year 2016 was produced by UC
Louvain as part of the European Space Agency (ESA) Climate Change Initiative
(CCI). The classification into 10 land cover classes (including “cropland”) is
based on input features extracted from one year (December 2015-December 2016)
of Sentinel-2A multispectral observations. The product documentation states
that classification was performed using “two classification algorithms, the
Random Forest (RF) and Machine Learning (ML)” but does not specify which
machine learning technique is used in the second algorithm.[28] Independent
evaluations of the map accuracy reported overall accuracy of 56% in Kenya[49]
and around 65% in over the continent, noting that the map should be improved
before being used for research or practical purposes.[50] We binarized the
dataset by converting pixels labeled with the cropland class to labels of 1
and 0 otherwise.

##### GFSAD Global Cropland Maps.

The goal of the Global Food Security Support Analysis Data (GFSAD) project was
to create a global map of cropland extent at 30 m/pixel resolution based on
Landsat satellite Earth observation data.[29] The GFSAD Global Cropland Extent
Product (GCEP) was created by combining maps predicted for 74 AEZs defined
globally. A predicted map was created for each AEZ by combining the outputs of
four machine learning models—random forest, support vector machine (SVM), an
object-based classifier, and recursive hierarchical segmentation—trained for
binary cropland classification. The input features were derived from Landsat-8
multispectral time series observations in addition to elevation and slope
attributes derived from the SRTM DEM for years 2013-2016, with outputs
intended to represent the nominal year 2015. The study reported overall
accuracy of 93.7% in Africa and, more specifically, 91.3% in AEZ 36 (includes
Kenya and Malawi) and 90.8% in AEZ 34 (includes Togo).

##### GLAD.

The GLAD Global Cropland Maps provide binary cropland classifications at 30
m/pixel for 2003, 2007, 2011, 2015, and 2019. Classification is performed
using bagged decision trees with features extracted from time series of
Landsat Analysis Ready Data (ARD).[30] A separate model is trained for each
$1^{\circ}\times 1^{\circ}$ ARD tile, and the predictions from each tile are
merged to form a global map. The regional accuracy reported for Africa for
2016-2019 was $96.5\pm 0.8$, and the global map was noted to underestimate the
cropland area in Africa due to the spatial resolution limitations. We
binarized the 2019 dataset by converting pixels with values $>0.5$ to a 1 and
otherwise 0 label.

##### ESA GlobCover.

The ESA GlobCover project aimed to provide global land cover maps based on
observations from the 300 m/pixel MERIS satellite sensor. The project produced
global land cover maps for 2009 (v2.3, based on observations from January-
December 2009) and 2005 (v2.2, based on observations from December 2004-June
2006); we used v2.3 in this study. A separate classifier was trained for each
of the 22 global strata designed to reduce land surface reflectance
variability in the data processed by each regional classifier. Classification
into 22 land cover classes is achieved through a four-stage process of (1a)
supervised classification of under-represented classes, (1b) unsupervised
clustering of pixels not classified in step 1a, (2) temporal characterization
of spectral clusters from step 1b, (3) aggregation of spectral clusters into
fewer spectro-temporal clusters based on similar temporal patterns, and (4)
rule-based classification of spectro-temporal clusters.[32] Defourny et al.
[51] reported an overall accuracy of 73% but did not report Africa-specific
metrics.[51] We binarized the dataset by converting pixels labeled with the
“post-flooding or irrigated croplands”, “rainfed croplands”, and “mosaic
cropland (50-70%) / vegetation (grassland, shrubland, forest) (20-50%)”
classes as 1, and all other classes as 0.

##### Copernicus Land Cover.

The Copernicus Global Land Service Land Cover (CGLS-LC) map provides global
land cover maps annually from 2015-2019 based on observations from the 100
m/pixel PROBA-V satellite sensor.[31] The land cover classification includes
23 classes based on the United Nations Food and Agriculture Organization (UN-
FAO) Land Cover Classification System and surface area statistics for 10 land
cover types. More than 100 input features, including vegetation indices and
time series statistics, were extracted from the PROBA-V multispectral
satellite observations for 141,000 training sample locations. These samples
were used to train a random forest classifier separately for each biome, where
biomes were clusters of pixels determined from multiple global ecological
datasets. Tsendbazar et al. (2020) reported a precision score (user’s
accuracy) of $62.4\pm 3.7\%$ and a recall score (producer’s accuracy) of
$57.3\pm 3.6\%$ for the cropland class in Africa and $80.1\pm 2.0\%$ overall
accuracy in Africa.[52] We used the discrete classification in the 2019 map
and created a binary cropland map by converting pixels labeled with the
“Cultivated and managed vegetation/agriculture” class to 1 and all other
classes to 0.

#### Global models

##### Esri Land Use/Land Cover.

The Esri Land Use and Land Cover (LULC) map, created by Esri, Impact
Observatory, and Microsoft, provides a global classification of 9 land cover
and land use classes (including crops) at 10 m/pixel resolution for the years
2017-2021.[23] Classification was performed using a U-Net deep convolutional
neural network model for semantic segmentation. The U-Net was trained using an
extremely large dataset of over 5 billion human-labeled pixels paired with
Sentinel-2 multispectral image composites. To create a LULC map for a given
year, the least cloudy scenes over the year are selected and predictions are
made for every Sentinel-2 tile, then combined by taking a class-weighted mode
of all predictions. Karra et al. (2021) reported an overall accuracy of 85%
but did not provide more granular metrics for countries in Africa. We
binarized the dataset by converting pixels labeled with the crop class to a
label of 1, and otherwise 0.

##### ESA WorldCover.

The ESA WorldCover global land cover product was designed to build on the
lessons learned from the ESA GlobCover and ESA CCI Land Cover products
described in the previous section. WorldCover provides a global land cover
classification map at 10 m/pixel resolution for 11 classes (including
cropland).[53] A large number of input features were extracted from Sentinel-2
multispectral, Sentinel-1 synthetic aperture radar, and several other Earth
observations datasets (similar to the features extracted for the Copernicus
Land Cover map described previously), in addition to localizing features such
as the latitude/longitude position. These features were then used to train a
CatBoost decision tree classifier, the outputs from which are post-processed
using expert rules designed to reduce classification errors. The product
validation report reported a precision score (user’s accuracy) of $71.4\pm
0.7\%$ and recall score (producer’s accuracy) of $50.8\pm 0.7\%$ for the
cropland class in Africa and an overall accuracy of $73.6\pm 0.2\%$ in
Africa.[54] WorldCover provides a v100 map for the year 2020 and a v200 map
for the year 2021; we used v100 for year 2020, which most closely matches our
reference data.

##### Dynamic World.

Unlike the previous cropland and land cover maps that represent the land cover
status of map locations based on satellite observations collected over a year
or multiple years, Dynamic World represents the land cover status at a
particular date. Dynamic World provides a new global land cover classification
at 10 m/pixel for every new cloud-free Sentinel-2 observation, which is
nominally acquired every 5 days everywhere on Earth.[21] Dynamic World uses a
fully-convolutional neural network (FCNN) to segment (i.e., classify all
pixels in an input image) the land cover classes in a given Sentinel-2
multispectral image, trained on globally-distributed samples. Dynamic World
provides land cover class probabilities and class labels (the class with the
highest probability in each pixel) for nine land cover classes, including
cropland. While Dynamic World is meant to capture near real-time land cover,
it can represent a longer time period by computing a mode composite over that
time period (e.g., one year as in the other maps in this study). We used a
mode composite of Dynamic World land cover classes corresponding to the
reference data year of each test region (e.g., 01/01/2019-12/31/2020 for
Kenya) and then binarized the map so that the crop class had a label of 1 and
all other classes had a label of 0 (representing non-crop classes). Brown et
al. (2022) reported an accuracy of 88.9% for the crop class, noting that
Dynamic World tended to identify crops more poorly than other classes.[21]

#### Ensemble maps

##### ASAP Crop Mask.

The Anomaly hot Spots of Agricultural Production (ASAP) crop mask was created
to provide a mask that could be applied to various Earth observation datasets
to provide timely information about potential crop production anomalies using
metrics computed from earth observations in crop areas.[11] The ASAP global
crop mask combines land cover and land use maps from multiple sources. For
Africa, ASAP used the 250 m/pixel land cover map from Vancutsem et al. (2013),
which combined ten different regional and global land cover maps based on
expert judgment to create a “best available” crop mask for Africa. In certain
countries for which a national crop mask was available (Afghanistan,
Argentina, Australia, Europe, Mexico, and USA), that map was used in ASAP. In
all other countries, the GlobCover[32] classification was used. All datasets
were resampled to 1 km/pixel resolution with pixel values interpreted as the
crop area fraction in each pixel (ranging from 0 to 100%). We re-classified
pixels with crop area fraction ranging from 5-95% to 1, and all other pixels
to 0, following the procedure described in user documentation
(https://glam1.gsfc.nasa.gov/api/doc/cropmask/v1/EC-JRC-ASAP-LC_v02_crops).

##### Nabil et al. (2022).

[24] Similar to the ASAP crop mask, the combined crop mask published by Nabil
et al. (2022) combines the best of four compared land cover maps within each
of 41 AEZs to create a more accurate cropland map for the African continent
than each of the individual maps.[24] The authors compared the accuracy in
each AEZ of four maps—ESA-S2-LC20,[28] GFSAD Global Cropland Extent,[29] CGLS-
LC100-2016,[31] and FROM-GLC30-2017[55] (the first three of which we included
in this study)—using a combination of labeled validation datasets. For each
AEZ, the compared map that had the highest accuracy was used in the final map.
The majority of the map uses classifications from GFSAD GCEP followed by CGLS-
LC100-2016, as these were found to have the highest accuracy in the majority
of AEZs in Africa. Using an independent random reference set to evaluate the
combined map, Nabil et al. (2022) reported a precision score (user’s accuracy)
of $93.73\%$, recall score (producer’s accuracy) of $61.93\%$, and overall
accuracy of $91.64\%$.

##### Majority Vote.

We created a majority vote ensemble to evaluate whether a combination of all
of the maps compared in this study would achieve better performance than
individual maps. If 6 or more of the 11 compared maps classified a pixel as
crop, the pixel is classified as crop in the Majority Vote map if 5 or fewer
classified a pixel as crop (i.e., 6 or more classified as non-crop) then the
pixel is classified as non-crop.

### Reference datasets

We created reference datasets for evaluating the accuracy of all maps in Togo,
Kenya, Malawi, Mali, Rwanda, Tanzania, Uganda, and Zambia. For all countries
except Mali, we sampled reference points by drawing a random uniform sample of
point locations within each country’s boundaries. For Mali, we sampled points
using a stratified random sample with strata defined by NDVI ranges (using the
mean annual NDVI from Sentinel-2). Table 4 summarizes the characteristics of
each dataset. For each country sample, trained individuals manually analyzed
high-resolution satellite images of PlanetScope (3 m/pixel resolution) monthly
composites and other auxiliary sources (Sentinel-2, and sub-meter resolution
images in Google Earth Pro) in the Collect Earth Online platform. For each
point, annotators were instructed to inspect images from each month spanning
the country’s growing season and determine whether the point contained active
cropland (i.e., patterns of sowing, growing, and/or harvesting in an
agricultural field could be observed). Interpreters were blind to the class
predicted by any maps during labeling. At least two annotators labeled every
point to maximize label confidence; the number of labelers that annotated each
point in each dataset is indicated in the “Num. labels per sample” column in
Table 4. We discarded points that did not have unanimous agreement between
labelers to ensure high-confidence labels in the final reference dataset.

| Number of samples |  |
---|---|---|---
Country | Total | Crop | Non-crop | Crop % | Label validity period | Num. labels per sample
Kenya | 573 | 36 | 537 | 6.3 | Feb 2019-Jan 2020 | 2
Malawi | 154 | 31 | 123 | 25.2 | Sep 2020-Aug 2021 | 2
Mali | 447 | 10 | 437 | 2.2 | Feb 2019-Jan 2020 | 2
Rwanda | 107 | 43 | 64 | 40.2 | Jan 2019-Dec 2019 | 3
Tanzania | 1201 | 431 | 770 | 35.9 | Jan 2019-Dec 2019 | 2
Togo | 182 | 51 | 131 | 39.0 | Jan 2019-Dec 2019 | 4
Uganda | 233 | 26 | 207 | 11.2 | Jan 2019-Dec 2019 | 2
Zambia | 489 | 20 | 269 | 4.1 | Jan 2019-Dec 2019 | 2
Total | 3386 | 740 | 2646 | 21.9 | – | –
Table 4: Reference sample datasets used for independent evaluation of cropland
maps

### Metrics and evaluation

We used the following metrics to evaluate the accuracy of each map in this
study using the reference datasets, which are commonly used for accuracy
assessment in both machine learning and remote sensing: overall accuracy and
F1 score, precision (also called user’s accuracy or UA), and recall (also
called producer’s accuracy or PA). These metrics are reported in Table 1. The
recall for the positive (crop) class is also known as sensitivity. The recall
or producer’s accuracy for the positive (crop) class can be expressed in terms
of true positives ($TP$) and false negatives ($FN$) as follows:

$Recall=\frac{TP}{TP+FN}$ (1)

The precision or user’s accuracy for the positive (crop) class can be
expressed in terms of true positives and false positives ($FP$) as follows:

$Precision=\frac{TP}{TP+FP}$ (2)

The overall accuracy can be expressed in terms of $TP$, $FP$, $FN$, and true
negatives ($TN$) using the following equation:

$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$ (3)

We used the equations from Stehman & Foody (2019)[33] and Olofsson et al.
(2014)[56] to compute standard errors for precision (UA), recall (PA), and
overall accuracy. We computed the standard error for F1 score by propagating
errors from precision ($P$) and recall ($R$):[57]

$\displaystyle F_{1}$ $\displaystyle=\frac{2PR}{P+R}\text{, which can be
simplified to }$ (4) $\displaystyle F_{1}$ $\displaystyle=\frac{X}{Y}\text{,
where }X=2PR\text{ and }Y=P+R$ (5)

Since the uncertainty of X and Y are not independent, they do not add in
quadrature. Instead their relative errors are added so that:

$\displaystyle\frac{\Delta F_{1}}{F_{1}}$ $\displaystyle=\frac{\Delta
X}{X}+\frac{\Delta Y}{Y}\text{, where }\Delta F_{1},\Delta X,\Delta Y\text{are
the errors of }F_{1},X,Y$ (6)

Since $F_{1}=X/Y$, we multiply both sides by $X/Y$ to get:

$\displaystyle\Delta F_{1}$ $\displaystyle=\frac{\Delta X}{Y}+\frac{X\Delta
Y}{YY}$ (7)

The error in $X$ and $Y$ are:

$\displaystyle\Delta X$ $\displaystyle=2(R\Delta P+P\Delta R)$ (8)
$\displaystyle\Delta Y$ $\displaystyle=\Delta P+\Delta R$ (9)

We substitute Eqns. 8 and 9 into Eqn. 7 to get the final expression for the
error of F1:

$\displaystyle\Delta F_{1}$ $\displaystyle=\frac{2(R\Delta P+P\Delta
R)}{P+R}+\frac{2PR(\Delta P+\Delta R)}{(P+R)(P+R)}$ (10)

We designed our accuracy assessment to meet the good practice criteria
described in Stehman & Foody (2019):[33]

  1. 1.

Map relevant: the accuracy estimates and error matrices reflect the
proportional area representation of the study region via uniform random
sampling and we report unnormalized error matrices in Supplementary File 1.

  2. 2.

Statistically rigorous: we implemented a probability sampling design of simple
uniform random sampling. We quantify the variability of the accuracy by
reporting standard errors.

  3. 3.

Quality assured: protocols were established to monitor and evaluate the
quality of reference data of results including assigning two or more
interpreters to label each point while blind to the map category; only points
with unanimous interpreter consensus were used in the analysis.

  4. 4.

Reliable: The variability among interpretations of reference sample labels is
very low since we only used points that had unanimous labeler agreement in
analysis. It is likely that different samples and different interpreters (with
the same training) would lead to similar results.

  5. 5.

Transparent: We have provided all relevant details to inform readers about the
quality of the results.

  6. 6.

Reproducible: We provide the reference sample datasets used in the analysis in
a public Google Earth Engine asset. The code for generating the samples and
instructions provided to interpreters is available upon request.

## Data availability

The datasets used in this analysis can be downloaded from a Zenodo repository
at https://doi.org/10.5281/zenodo.8048967. The Zenodo repository includes the
following specific datasets:

  1. 1.

Labeled reference datasets for each country used for evaluation and summarized
in Table 4, in shapefile format

  2. 2.

CSV file of the evaluation metrics computed for each map using the reference
dataset from each country

  3. 3.

Consensus maps for each country indicating in each pixel the number of maps
that predict cropland (used to create Figure 2), in GeoTIFF format with 10
m/pixel resolution

The reference datasets are also available as public feature collection assets
via Google Earth Engine (note that you must be signed in with a Google Earth
Engine account to view/access this link):
https://code.earthengine.google.com/?asset=projects/bsos-geog-
harvest1/assets/harvest-reference-datasets.

The maps and other datasets used in this study can be visualized and compared
in a Google Earth Engine app at
https://hkerner_umd.users.earthengine.app/view/intercomparison-of-public-crop-
maps-in-sub-saharan-africa.

## Code availability

The code used for processing and evaluating each of the maps in this study is
publicly accessible at https://github.com/nasaharvest/crop-
mask/blob/master/src/compare_covermaps.py.

## References

  * [1] Bjornlund, V., Bjornlund, H. & van Rooyen, A.  Why food insecurity persists in sub-Saharan Africa: A review of existing evidence.  _Food Security_ 14, 845–864 (2022).
  * [2] WFP, W., UNICEF _et al._ The state of food security and nutrition in the world 2022 (2022).
  * [3] Nakalembe, C. & Kerner, H.  Considerations for AI-EO for agriculture in Sub-Saharan Africa.  _Environmental Research Letters_ 18, 041002 (2023).
  * [4] Fritz, S. _et al._ The need for improved maps of global cropland.  _Eos, Transactions American Geophysical Union_ 94, 31–32 (2013).
  * [5] Fritz, S. _et al._ A comparison of global agricultural monitoring systems and current gaps.  _Agricultural systems_ 168, 258–272 (2019).
  * [6] Mbow, H.-O. P., Reisinger, A., Canadell, J. & O’Brien, P.  Special report on climate change, desertification, land degradation, sustainable land management, food security, and greenhouse gas fluxes in terrestrial ecosystems (sr2).  _Ginevra, IPCC_ 650 (2017).
  * [7] Kelly, E. _et al._ Sustainability indicators for improved assessment of the effects of agricultural policy across the EU: Is FADN the answer?  _Ecological indicators_ 89, 903–911 (2018).
  * [8] Becker-Reshef, I. _et al._ The GEOGLAM crop monitor for AMIS: Assessing crop conditions in the context of global markets.  _Global Food Security_ 23, 173–181 (2019).
  * [9] Becker-Reshef, I. _et al._ Strengthening agricultural decisions in countries at risk of food insecurity: The GEOGLAM Crop Monitor for Early Warning.  _Remote Sensing of Environment_ 237, 111553 (2020).
  * [10] Becker-Reshef, I. _et al._ Monitoring global croplands with coarse resolution earth observations: The Global Agriculture Monitoring (GLAM) project.  _Remote Sensing_ 2, 1589–1609 (2010).
  * [11] Rembold, F. _et al._ Asap: A new global early warning system to detect anomaly hot spots of agricultural production for food security analysis.  _Agricultural systems_ 168, 247–257 (2019).
  * [12] Nabil, M. _et al._ Assessing factors impacting the spatial discrepancy of remote sensing based cropland products: A case study in Africa.  _International Journal of Applied Earth Observation and Geoinformation_ 85, 102010 (2020).
  * [13] Pérez-Hoyos, A., Rembold, F., Kerdiles, H. & Gallego, J.  Comparison of global land cover datasets for cropland monitoring.  _Remote Sensing_ 9, 1118 (2017).
  * [14] Venter, Z. S., Barton, D. N., Chakraborty, T., Simensen, T. & Singh, G.  Global 10 m land use land cover datasets: A comparison of Dynamic World, World Cover and Esri Land Cover.  _Remote Sensing_ 14, 4101 (2022).
  * [15] Fritz, S., See, L. & Rembold, F.  Comparison of global and regional land cover maps with statistical information for the agricultural domain in Africa.  _International Journal of Remote Sensing_ 31, 2237–2256 (2010).
  * [16] Fritz, S. & See, L.  Identifying and quantifying uncertainty and spatial disagreement in the comparison of global land cover for different applications.  _Global Change Biology_ 14, 1057–1075 (2008).
  * [17] Xu, Y. _et al._ Comparisons of three recent moderate resolution African land cover datasets: CGLS-LC100, ESA-S2-LC20, and FROM-GLC-Africa30.  _International Journal of Remote Sensing_ 40, 6185–6202 (2019).
  * [18] Vancutsem, C., Marinho, E., Kayitakire, F., See, L. & Fritz, S.  Harmonizing and combining existing land cover/land use datasets for cropland area monitoring at the African continental scale.  _Remote Sensing_ 5, 19–41 (2012).
  * [19] Tchuenté, A. T. K., Roujean, J.-L. & De Jong, S. M.  Comparison and relative quality assessment of the GLC2000, GLOBCOVER, MODIS and ECOCLIMAP land cover data sets at the African continental scale.  _International Journal of Applied Earth Observation and Geoinformation_ 13, 207–219 (2011).
  * [20] Herold, M., Mayaux, P., Woodcock, C., Baccini, A. & Schmullius, C.  Some challenges in global land cover mapping: An assessment of agreement and accuracy in existing 1 km datasets.  _Remote Sensing of Environment_ 112, 2538–2556 (2008).
  * [21] Brown, C. F. _et al._ Dynamic world, near real-time global 10 m land use land cover mapping.  _Scientific Data_ 9, 251 (2022).
  * [22] Zanaga, D. _et al._ ESA WorldCover 10 m 2021 v200 (2022).
  * [23] Karra, K. _et al._ Global land use/land cover with Sentinel 2 and deep learning.  In _2021 IEEE international geoscience and remote sensing symposium IGARSS_ , 4704–4707 (IEEE, 2021).
  * [24] Nabil, M., Zhang, M., Wu, B., Bofana, J. & Elnashar, A.  Constructing a 30m African cropland layer for 2016 by integrating multiple remote sensing, crowdsourced, and auxiliary datasets.  _Big Earth Data_ 6, 54–76 (2022).
  * [25] Fritz, S. _et al._ Cropland for sub-Saharan Africa: A synergistic approach using five land cover data sets.  _Geophysical Research Letters_ 38 (2011).
  * [26] Pérez-Hoyos, A., Udías, A. & Rembold, F.  Integrating multiple land cover maps through a multi-criteria analysis to improve agricultural monitoring in Africa.  _International Journal of Applied Earth Observation and Geoinformation_ 88, 102064 (2020).
  * [27] Burton, C. _et al._ Co-production of a 10-m cropland extent map for continental Africa using Sentinel-2, cloud computing, and the open-data-cube.  _Authorea Preprints_ (2022).
  * [28] Ramoino, F., Pera, F. & Arino, O.  ‘S2 prototype LC map at 20m of Africa 2016’ users feedback compendium (2018).
  * [29] Thenkabail, P. S. _et al._ Global cropland-extent product at 30-m resolution (GCEP30) derived from Landsat satellite time-series data for the year 2015 using multiple machine-learning algorithms on Google Earth Engine cloud (2021).
  * [30] Potapov, P. _et al._ Global maps of cropland extent and change show accelerated cropland expansion in the twenty-first century.  _Nature Food_ 3, 19–28 (2022).
  * [31] Buchhorn, M. _et al._ Copernicus global land cover layers—collection 2.  _Remote Sensing_ 12, 1044 (2020).
  * [32] Team, U. _et al._ Globcover 2009 products description and validation report.
  * [33] Stehman, S. V. & Foody, G. M.  Key issues in rigorous accuracy assessment of land cover products.  _Remote Sensing of Environment_ 231, 111199 (2019).
  * [34] FAO.  Arable land and permanant crops (2022).
  * [35] Whitcraft, A. K. _et al._ No pixel left behind: Toward integrating earth observations for agriculture into the United Nations Sustainable Development Goals framework.  _Remote Sensing of Environment_ 235, 111470 (2019).
  * [36] FAO.  Codes for global administrative districts maps (2020).
  * [37] Rolf, E.  Evaluation challenges for geospatial ML.  _arXiv preprint arXiv:2303.18087_ (2023).
  * [38] Tan, P., Steinbach, M. & Kumar, V.  _Introduction to Data Mining_ (Pearson Education, 2007).
  * [39] Digital Earth Africa.  Digital Earth Africa Cropland Extent Map (2019), https://registry.opendata.aws/deafrica-crop-extent/ (2019).
  * [40] Esri.  Esri Sentinel-2 10m Land Use/Land Cover, https://livingatlas.arcgis.com/landcoverexplorer/ (2022).
  * [41] ESA WorldCover consortium.  ESA WorldCover project, https://esa-worldcover.org/en/data-access (2020).
  * [42] ESA CCI Land Cover Team.  CCI Land Cover - S2 Prototype Land Cover 20m Map of Africa 2016, https://2016africalandcover20m.esrin.esa.int/download.php (2017).
  * [43] Xiong, J. _et al._ NASA Making Earth System Data Records for Use in Research Environments (MEaSUREs) Global Food Security-support Analysis Data (GFSAD) Cropland Extent 2015 Africa 30 m V001 [Data set], 10.5067/MEaSUREs/GFSAD/GFSAD30AFCE.001 (2017).
  * [44] Nabil, M., Zhang, M., Wu, B., Bofana, J. & Elnashar, A.  A 30m African Cropland Layer for 2016 by Integrating Multiple Remote sensing, Crowdsource, and Auxiliary Datasets., 10.6084/m9.figshare.13520141.v1 (2021).
  * [45] Potapov, P. _et al._ Global cropland expansion in the 21st century, https://glad.umd.edu/dataset/croplands (2021).
  * [46] Buchhorn, M. _et al._ Copernicus Global Land Service: Land Cover 100m: collection 3: epoch 2019: Globe, 10.5281/zenodo.3939050 (2020).
  * [47] ESA GlobCover 2009 Project.  GlobCover 2009 (global land cover map), http://due.esrin.esa.int/page_globcover.php (2010).
  * [48] Perez-Hoyos, A.  Global crop and rangeland masks [data set], http://data.europa.eu/89h/jrc-10112-10005 (2010).
  * [49] LACO-Wiki.  Accuracy assessment of the 20 m land cover map of Africa (2020).
  * [50] Lesiv, M. _et al._ Evaluation of ESA CCI prototype land cover map at 20m (2017).
  * [51] Defourny, P. _et al._ Accuracy assessment of a 300 m global land cover map: the GlobCover experience.  In _33rd International Symposium on Remote Sensing of Environment (ISRSE)_ (2009).
  * [52] Tsendbazar, N.-E. _et al._ Copernicus global land service: Land cover 100m: Version 3 globe 2015-2019: Validation report (2020).
  * [53] WorldCover.  Product user manual (2020).
  * [54] WorldCover.  Product validation report (2020).
  * [55] Feng, D. _et al._ A multiple dataset approach for 30-m resolution land cover mapping: a case study of continental Africa.  _International Journal of Remote Sensing_ 39, 3926–3938 (2018).
  * [56] Olofsson, P. _et al._ Good practices for estimating area and assessing accuracy of land change.  _Remote sensing of Environment_ 148, 42–57 (2014).
  * [57] Palmer, M.  Propagation of uncertainty through mathematical operations.

## Acknowledgements

We are grateful to Dr. Samapriya Roy for adding many of the map products used
in this study to the awesome-gee-datasets catalog and to Dr. Jacob Adler for
assistance with data visualization and error estimation. This work was
supported by the NASA Harvest Consortium on Food Security and Agriculture
(Award #80NSSC18M0039). We are also grateful to the many individuals who
contributed to labeling the reference datasets used in this study.

## Author contributions statement

H.K. conceived the study and experiment(s), H.K. and A.Y. conducted the
experiments, H.K. analyzed the results. The manuscript was written by H.K. and
C.N. All authors reviewed the manuscript and provided input on results and
experiments.

## Competing interests

The authors declare no competing interests.