# ISCUTE: Instance Segmentation of Cables
Using Text Embedding

Shir Kozlovsky, Omkar Joglekar
Bosch Center for Artificial Intelligence
Haifa, Israel
{kos1tv<EMAIL_ADDRESS>
&Dotan Di Castro
Bosch Center for Artificial Intelligence
Haifa, Israel
<EMAIL_ADDRESS>
Both authors contributed equally to this work, the order was finalized by coin
toss.

###### Abstract

In the field of robotics and automation, conventional object recognition and
instance segmentation methods face a formidable challenge when it comes to
perceiving Deformable Linear Objects (DLOs) like wires, cables, and flexible
tubes. This challenge arises primarily from the lack of distinct attributes
such as shape, color, and texture, which calls for tailored solutions to
achieve precise identification. In this work, we propose a foundation model-
based DLO instance segmentation technique that is text-promptable and user-
friendly. Specifically, our approach combines the text-conditioned semantic
segmentation capabilities of CLIPSeg model with the zero-shot generalization
capabilities of Segment Anything Model (SAM). We show that our method exceeds
SOTA performance on DLO instance segmentation, achieving a mIoU of $91.21\%$.
We also introduce a rich and diverse DLO-specific dataset for instance
segmentation.

##  1 Introduction

Deformable Linear Objects (DLOs), encompassing cables, wires, ropes, and
elastic tubes, are commonly found in domestic and industrial settings (Keipour
et al., 2022b; Sanchez et al., 2018). Despite their widespread presence in
these environments, DLOs present significant challenges to automated robotic
systems, especially in perception and manipulation, as discussed by Cop et al.
(2021). In terms of perception, the difficulty arises from the absence of
distinct shapes, colors, textures, and prominent features, which are essential
factors for precise object perception and recognition.

Over the past few years, there has been a notable emergence of customized DLO
approaches. State-of-the-art (SOTA) instance segmentation methods such as RT-
DLO (Caporali et al., 2023) and mBEST (Choi et al., 2023) use novel and
remarkable approaches influenced by classic concepts from graph topology or by
bending energy to segment these challenging objects accurately. However, none
of these methods excel in handling real and complex scenarios, nor do they
incorporate prompt-based control functionality for segmentation, such as text
prompts, which could enhance user accessibility.

In the domain of purely computer vision-based approaches, the Segment Anything
Model (SAM; Kirillov et al., 2023) is one of the most notable segmentation
models in recent years. As a foundation model, it showcases remarkable
generalization capabilities across various downstream segmentation tasks,
using smart prompt engineering (Bomasani & Others, 2021). However, SAM’s
utility is limited to manual, counterintuitive prompts in the form of points,
masks, or bounding boxes, with basic, proof-of-concept text prompting. On the
other hand, in the domain of deep vision-language fusion, CLIPSeg (Lüddecke &
Ecker, 2022) presents a text-promptable semantic segmentation model. However,
this model does not extend its capabilities to instance segmentation.

In this paper, we present a novel adapter model that facilitates as a
communication channel between the CLIPSeg model and SAM, as illustrated in
Figure 1. Specifically, our model is capable of transforming semantic mask
information corresponding to a text prompt to batches of point prompt
embedding vectors for SAM, which processes them to generate instance
segmentation masks.

In our method, we exhibit the following novelties:

  1. 1.

A novel prompt encoding network that converts semantic information obtained
from text prompts to point prompts’ encoding that SAM can decode.

  2. 2.

A classifier network to filter out duplicate or low-quality masks generated by
SAM.

  3. 3.

A CAD-generated, fully annotated, DLO-specific dataset consisting of about 30k
high-resolution images of cables in industrial settings.

Our method also improves the SOTA in DLO segmentation, achieving
$mIoU=91.21\%$ on the generated dataset, compared to RT-DLO ($mIoU=50.13\%$).
Furthermore, we show our model’s strong zero-shot generalization capabilities
to datasets presented in RT-DLO (Caporali et al., 2023) and mBEST (Choi et
al., 2023). Finally, our method offers a user-friendly approach to the
problem, requiring only an image and a text prompt to obtain one-shot instance
segmentation masks of cables.

Figure 1: Overview of the full pipeline - blocks in red represent our
additions

##  2 Related Work

In this section, we review related works of DLOs instance segmentation and
methods we use in our approach.

###  2.1 DLOs Instance Segmentation

DLO detection algorithms have been introduced with distinctive methodologies
that are necessary to overcome the challenges associated with the problem.
Zanella et al. (2021) first presented a Convolutional Neural Network (CNN;
LeCun et al., 1995) based approach to perform semantic segmentation of DLOs.
Yan et al. (2019) and Keipour et al. (2022a) pioneered data-driven techniques
for DLO instance segmentation, using neural networks to reconstruct DLO
topology and fitting curvatures and distances for a continuous DLO,
respectively. However, these methods were limited to a single DLO.

_Ariadne_ (De Gregorio et al., 2018) and it is immediate successor _Ariadne+_
(Caporali et al., 2022b), segment DLOs into superpixels (Achanta et al., ),
followed by path traversal to generate instance masks. _Ariadne+_ provided a
DeepLabV3+ (Chen et al., 2018) based model to extract a semantic mask before
superpixel processing. This allowed the authors to forego _Ariadne’s_ limiting
assumption about having a uniformly colored DLO and requiring the manual
selection of endpoints. However, both Ariadne and Ariadne+ encounter a
significant challenge due to their reliance on numerous hyperparameters .

FASTDLO (Caporali et al., 2022a) and RT-DLO (Caporali et al., 2023), approach
DLOs instance segmentation using skeletonization of semantic masks, instead of
superpixel generation. This allows both algorithms to operate much faster
because the slow process of superpixel generation is non-essential. In
FASTDLO, the skeletonized semantic mask is processed using path traversal, and
a neural network is used to process intersections. Contrastively, RT-DLO
employs a graph node sampling algorithm from DLO centerlines using the
semantic mask. This is followed by topological reasoning-based edge selection
and path determination. Although both algorithms offer either near-real-time
or real-time performance (depending on the number of DLOs and intersections in
the image), they are sensitive to noise introduced by centerline sampling and
scene-dependent hyperparameter tuning.

Another recent work, mBEST (Choi et al., 2023), involves fitting a spline to
minimize bending energy while traversing skeleton pixels. This method has
shown impressive results compared to all the aforementioned works, exhibiting
a unique solution to the problem, but it is limited to fewer than two
intersections between different DLOs and relies on a manually tuned threshold
that depends on the number of DLOs in the scene.

All the aforementioned approaches, including the current SOTA baselines, RT-
DLO (Caporali et al., 2023) and mBEST (Choi et al., 2023) exhibit sensitivity
to scenarios such as occlusions, cables intersecting near image edges, and
small cables positioned at the edges of the image, which are commonly
encountered scenes in industrial settings. Furthermore, these methods face
challenges when transferring to other datasets and are limited to images with
specific operating conditions. Our approach is designed to address these
challenges through the utilization of a diverse and comprehensive generated
dataset as well as the generalization power of foundation models. Furthermore,
to the best of our knowledge, this is the first approach that combines DLO
instance segmentation with text prompts, offering a user-friendly solution.

###  2.2 Foundation Models

Introduced by Kirillov et al. (2023), SAM is a powerful foundation model that
shows remarkable performance on a myriad of downstream tasks. It shows
promising performance on datasets containing DLOs that have historically been
difficult to segment. The generalization power of this model lies in its
prompting capabilities, specifically using points, boxes, or masks to direct
the segmentation. SAM outperforms other promptable segmentation models such as
FocalClick (Chen et al., ), PhraseClick (Ding et al., ), PseudoClick (Liu et
al., 2022b) and SimpleClick (Liu et al., 2022a) in the generalizability and
quality of segmentation. The authors of SAM also present a proof-of-concept
for text-based prompting, but this method still requires manual point prompt
assistance to give the same high-quality segmentation masks. Box prompts are
not conducive to one-shot instance segmentation, as a single bounding box can
encompass multiple DLOs, as illustrated in Fig. 2(c). Single point-prompting
displays a strong potential for performance (Fig. 2(b)). However, it struggles
to perform effectively in complex scenarios (Fig. 2(a)). Manually inputting
multiple points per cable proves inefficient for process automation in an
industrial context because it necessitates human intervention. In our work, we
harness the capabilities of the SAM model for segmentation and its ability to
use point prompts. We replaced its prompt encoder, capable of converting text
prompts into point prompts’ embedding. Moreover, our model performs instance
segmentation in one forward pass without the need for sequential mask
refinement as in Kirillov et al. (2023). Additionally, we also introduce a
network to filter out duplicate and low-quality masks.

Multi-modal data processing such as text-image, text-video, and image-audio
has gained impetus in recent years following the advent of mode-fusion models
such as CLIP (Radford et al., 2021) and GLIP (Li et al., 2021). GroundingDINO
(Liu et al., 2023) uses BERT (Devlin et al., ) and either ViT (Dosovitskiy et
al., 2020) or Swin-Transformer (Liu et al., 2021) to perform text conditioned
object detection. The approach is a SOTA object detector but relies on
bounding boxes, proving less effective for DLOs (Fig. 2(c)).

A notable counter example is CLIPSeg (Lüddecke & Ecker, 2021), that leverages
a CLIP (Radford et al., 2021) backbone followed by a FiLM layer (Perez et al.,
2017) and finally a UNet (Ronneberger et al., 2015) inspired decoder to
generate semantic segmentation based on text prompts (Fig. 1). Important to
our application, the penultimate layer’s embedding space in CLIPSeg represents
a spatially oriented attention heatmap corresponding to a given text prompt.

Another recent trend in computer-vision research is _adapters_ that have been
proposed for incremental learning and domain adaptation (Chen et al., 2022;
Rebuffi et al., 2017; Selvaraj et al., 2023). Some adapters like VL-Adapter
(Sung et al., 2021), CLIP-Adapter (Gao et al., 2021), Tip-Adapter (Zhang et
al., 2021) use CLIP to transfer pre-trained knowledge to downstream tasks. The
method we propose in this work is also inspired by some ideas from research in
adapters.

(a) An example of unsuccessful segmentation with a single point.

(b) An example of successful segmentation with a single point

(c) An example of box prompt for DLO segmentation.

Figure 2: Issues with using SAM out-of-the-box

##  3 Methods

In this section, we describe the core method behind our approach.

###  3.1 The ISCUTE Model

The adapter model we propose consists of $2$ main networks (as depicted in
Fig. 1):

  1. 1.

Prompt encoder network \- This network extracts batches of point prompts from
CLIPSeg’s embedding space. It can be controlled using $2$ hyperparameters: $N$
(the number of prompt batches) and $N_{p}$ (the number of points in each
batch).

  2. 2.

Classifier network \- A binary classifier network that labels whether a
particular mask from the generated $N$ masks should appear in the final
instance segmentation.

In the following, we detail each of these networks. We perform ablation
studies to justify our choice of architecture. The results of these ablation
studies can be found in Appendix C. We note that we neither fine-tune CLIPSeg
nor fine-tune SAM, and the only trained component is the adapter. The
motivation for this is twofold. First, we would like to refrain from degrading
the original performance of both CLIPSeg and SAM. Second, it is more efficient
in many cases to train only an adapter instead of foundation models.

Figure 3: The ISCUTE adapter: on the left, the architecture of the prompt
encoder network is outlined (indicated by the purple dashed line), while on
the right, the classifier network architecture is detailed (represented by the
pink dashed line).

####  3.1.1 Prompt encoder network

The Prompt Encoder Network’s goal is to efficiently convert the embedding
space generated by CLIPSeg into batches of point prompt embedding that SAM’s
mask decoder can accurately decipher, as depicted in Figure 3. This process of
selecting the appropriate point prompts relies on the following stages:

  1. 1.

Identification of critical patches.

  2. 2.

Determining an optimal threshold based on patch importance and filtering out
the rest.

  3. 3.

Categorizing each selected point as foreground, background, or no-point to
leverage SAM’s capabilities.

CLIPSeg generates a $22\times 22\times 64$ embedding tensor, which embeds a
semantic mask that aligns with the input image spatially and is conditioned on
text. To maintain a consistent embedding size throughout the pipeline, we
employ an MLP (bottom left MLP in Fig. 3) to upscale the $64$-dimensional
embedding to $256$ dimensions, followed by a self-attention layer, which
learns inter-patch correlations to focus on the relevant patches. CLIPSeg’s
embedding output is enhanced with Dense Positional Encoding (DPE) to ensure
that the self-attention layer has access to crucial geometric information. To
this end, the DPE values are added to the embedding vector even after
participating in the self-attention layer. To generate our DPE, we use an
identical frequency matrix as SAM. This ensures that every element within each
vector of the DPE conveys consistent information, that is aligned with what
SAM’s decoder has been trained to interpret. This output is processed by an
MLP that acts like a filter and aims to retain only the most important
patches. After the important patches are extracted, we pass them into the
Sampler Attention (pink layer in Fig. 3). This process resembles standard
scoring, where the interaction of the keys and queries determines which
positional encoding vector to select from the DPE. Here, the self-attended
patches act as the keys, and the DPE tensor as the values. The output of this
layer is the positional encoding of the points we wish to prompt SAM with.
Finally, a single Linear layer is applied to this output, to categorize the
chosen point as foreground/background/no-point. This layer aims to mimic the
labeling protocol from SAM, which employs a standard affine transformation,
where a learned embedding is added to each point embedding corresponding to
its category.

It is important to highlight that the queries are reintroduced to the updated
tokens whenever they engage in an attention layer. This allows for a strong
dependence on both the queries that carry the semantic information and the
values that carry the geometric information. Moreover, layer normalization (Ba
et al., 2016) is applied after each attention layer, as recommended by Vaswani
et al. (2023). Furthermore, our DPE, like SAM, takes inspiration from Fourier
features (Tancik et al., 2020) alongside classical techniques from digital
communication (Ling, 2017), utilizing the same frequency matrix as SAM instead
of a randomly generated one.

####  3.1.2 Mask classification network

In previous works, such as DETR (Carion et al., 2020) and MaskFormer (Cheng et
al., 2021), the authors train a classifier network along with the box
regression model to classify which object is contained within the box. Both of
these works introduced an additional _no-object class_ , to filter out
duplicate or erroneous boxes/masks during prediction. Inspired by them, we
developed a binary classifier network for this purpose.

This binary classifier model is composed of an MLP, one cross-attention block,
followed by one self-attention block, and an MLP, as illustrated in Figure 3.
First, the sampled point embeddings (from our prompt encoder) are transformed
using an MLP. Second, a cross-attention block operates on these transformed
embedding generated by our model, which encode text-conditioned information,
and on the mask tokens produced by SAM, which encapsulate submask-related
details. This interaction results in an embedding that fuses both types of
information for each generated submask. Subsequently, the queries are combined
with the output to reintroduce textual information to these _classifier
tokens_. These classifier tokens then undergo a self-attention layer followed
by the MLP to yield binary classifications.

###  3.2 Training protocol

The full architectural framework of our model, illustrated in Figure 1,
accepts a single RGB image and a text prompt as input, delivering an instance
segmentation mask for each DLO in the image. Inspired by the work of Cheng et
al. (2021), we employ a bipartite matching algorithm (Kuhn, 1955) to establish
the optimal correspondence between the generated masks and ground-truth
submasks to train the prompt encoder. We use a combination of the focal loss
(Lin et al., 2017) and the DICE loss (Milletari et al., 2016) in the ratio of
$20:1$, as recommended by Kirillov et al. (2023).

The binary classification labels are extracted from the output of the
bipartite matching algorithm. If a mask successfully matches, then it is
labeled as $1$; else, it is labeled as $0$. We use the binary cross-entropy
loss to train this network. To balance the distribution of 0s and 1s in the
dataset, we introduce a weight for positive labels that we set to $3$.

The most time-intensive step within the SAM workflow involves generating the
image embedding. However, once this image embedding is created, it can be used
repeatedly to produce multiple submasks as needed. This approach significantly
speeds up the process and can be executed in a single step. To enhance
training efficiency using this method, we set the training batch size to just
$1$. This configuration enables us to form batches of prompts, with each DLO
in the scene associated with an individual prompt. We place a cap of $N=11$
prompts for each image and limit the number of points within each prompt to
$N_{p}=3$. Specific details about the training process can be found in
Appendix A.

##  4 Experiments

###  4.1 The cables dataset

The dataset used for training, testing, and validation was generated using
Blender111https://www.blender.org/, a 3D rendering software. It comprises
images depicting various cables in diverse thicknesses, shapes, sizes, and
colors within an industrial context, exemplified in Figures 1 and 4. The
dataset consists of 22k training images, along with 3k images, each designated
for validation and testing. Each image has a resolution of 1920x1080, and each
DLO within an image is accompanied by a separate mask, referred to as a
submask in this work. This pioneering DLO-specific dataset encompasses an
extensive range of unique scenarios and cable variations. We anticipate that
granting access to this comprehensive dataset will significantly advance
numerous applications in the field of DLO perception within industrial
environments. For more details and sample images from the dataset, refer to
the Appendix B. This dataset will be made publicly available on acceptance.

###  4.2 Baseline experiments

In the process of designing the model and configuring the best
hyperparameters, we carried out multiple experiments. The initial experiment
involved selecting the number of points per prompt, denoted as $N_{p}$. As
previously mentioned, the point prompt within the SAM framework can encompass
more than just a single point, with each point being labeled as either
foreground or background. In the SAM paper (Kirillov et al., 2023), the
authors explored various values for the number of points and documented the
resulting increase in Intersection over Union (IoU) for the generated masks.
They observed diminishing returns after reaching $3$ points. However, it was
necessary to reevaluate this parameter for one-shot applications. We conducted
this using the same base model but with different values for $N_{p}$,
specifically $N_{p}=2,3,4$. Our findings led us to the conclusion that
$N_{p}=3$ is indeed the optimal number of points per prompt, and we adopted
this value for all subsequent experiments. The second experiment focuses on
SAM’s capacity to generate multiple masks, specifically, three masks, per
prompt. We examined and compared this multi-mask output with the single-mask
output in terms of the mean IoU (mIoU). Our findings indicated that employing
multi-mask output led to only a minor improvement at the cost of significantly
higher memory utilization. Consequently, for all our subsequent experiments
involving SAM, we opted to disable the multi-mask functionality.

###  4.3 Quantitative experiments

In these experiments, our primary objective is to compare our model’s DLO
instance segmentation results with those of SOTA baselines while also
assessing the model’s limitations using the _Oracle_ method, inspired by
Kirillov et al. (2023), which involves removing the classifier network during
testing and employing bipartite matching to filter out duplicate and low-
quality masks. This method allows us to independently evaluate our prompt
encoder and classifier networks, providing insights into the performance of
foundation models under ideal conditions, with the important note that our
model has access to ground-truth annotations during Oracle tests.

We additionally test the following characteristics - the overall quality of
the generated instance masks, zero-shot generalization capabilities, and the
effect of image augmentation on the generalization. In Table 1, we explore the
effects of image augmentation and the oracle method. Adhering to recent
testing protocols in the domain of DLO instance segmentation, as conducted by
Choi et al. (2023) in mBEST, we use the DICE score as a metric to analyze the
performance of our model. The comparative results are portrayed in Table 2.
These tests exhibit the zero-shot transfer capabilities of our models to the
testing datasets discussed in mBEST (Choi et al., 2023).

##  5 Results

###  5.1 Quantitative results

We summarize the comparison of results of different configurations of our
model on our test dataset in Table 1. As a reference, we tested the RT-DLO
algorithm (Caporali et al., 2023) on our test dataset. It achieved an
$mIoU=50.13\%$. The Oracle tests show upper bound on the performance our
method can achieve. This upper bound is related to the limitations of CLIPSeg
to produce good semantic masks given a text prompt and the limitations of SAM
to segment a particular object in the scene given a set of points as a prompt.

| Test data performance |  |
---|---|---|---
Model Configuration | mIoU [%] | DICE [%] | Augmentation | Oracle
A (Aug only) | 90.64 | 99.78 | Y | N
A+O (Aug+Oracle) | 92.10 | 99.80 | Y | Y
B+O (Base+Oracle) | 92.51 | 99.82 | N | Y
B (Base) | 91.21 | 99.77 | N | N
Table 1: Comparison of our model configurations (values in bold signify best
performance).

Table 2 shows the DICE scores of our model compared to current SOTA baselines:
Ariadne+ (Caporali et al., 2022b), FASTDLO (Caporali et al., 2022a), RT-DLO
(Caporali et al., 2023) and mBEST (Choi et al., 2023). The datasets C1, C2, C3
are published by Caporali et al. (2023), while the datasets S1, S2, S3 are
published by Choi et al. (2023).

In the _no Oracle_ setting, we observe that the model generalizes better if it
is trained using augmentations in the dataset, as expected. ISCUTE exhibits
strong zero-shot generalization to all the datasets. Specifically for C1, C2,
and C3, our model exceeds SOTA performance even in the base configuration.
Furthermore, in the case of Oracle configuration, we see that our proposed
method outperforms all SOTA baselines on all datasets, showing that the major
bottleneck in our full model is the mask classifier network.

| DICE[%]
---|---
|  |  |  |  | ISCUTE (Ours)
Dataset | Ariadne+ | FASTDLO | RT-DLO | mBEST | A | A+O | B+O | B
C1 | 88.30 | 89.82 | 90.31 | 91.08 | 97.03 | 98.85 | 98.86 | 94.55
C2 | 91.03 | 91.45 | 91.10 | 92.17 | 96.91 | 98.69 | 98.70 | 96.12
C3 | 86.13 | 86.55 | 87.27 | 89.69 | 97.13 | 98.81 | 98.90 | 90.26
S1 | 97.24 | 87.91 | 96.72 | 98.21 | 97.36 | 98.43 | 98.60 | 93.08
S2 | 96.81 | 88.92 | 94.91 | 97.10 | 97.71 | 98.54 | 98.42 | 97.01
S3 | 96.28 | 90.24 | 94.12 | 96.98 | 96.69 | 98.79 | 98.58 | 95.83
Table 2: DICE score result from comparison for out-of-dataset, zero-shot
transfer. Values in bold signify the best performance

To assess generalizability across various text prompts, we tested our model on
prompts like ”wires” and ”cords,” in addition to our training prompt,
”cables,” which specifically refers to DLOs. Table 3 showcases the performance
of these prompts when assessed on the same model using Oracle. Remarkably, our
model demonstrates the ability to effectively generalize across different
prompts zero-shot, producing results comparable to the base prompt, ”cables.”

Text Prompt | mIoU [%] | DICE [%]
---|---|---
cables (baseline) | 92.51 | 99.82
wires | 89.6 | 99.77
cords | 89.3 | 99.64
Table 3: Generalizability Across Text Prompts: Demonstrates results comparable
to the base prompt ”cables” through zero-shot transfer.

###  5.2 Qualitative results

In this section, we will present the actual instance masks generated by our
model. We used the configuration that was trained on augmentations (A) to
generate all these results (without Oracle).

Figure 4 presents the instance masks generated by ISCUTE in various complex
scenarios, along with the results of RT-DLO on the same image set for
comparison. These scenarios encompass occlusions, DLOs with identical colors,
varying thicknesses, small DLOs in the corners, a high density of cables in a
single image, and real-world scenarios. Our model shows impressive performance
across all of these scenarios.

Figure 4: Qualitative comparison in specific scenarios. Each scenario
demonstrates the following: (a) and (b) real images, (c) identical colors, (d)
a high density of cables in a single image, and (e) and (f) small DLOs at the
edge of the image with varying thicknesses.

In Figure 5, we show the instance masks generated by ISCUTE compared to the
current SOTA baselines on an external dataset. It is evident that our model
demonstrates remarkable zero-shot transfer capabilities when applied to data
distributions different from its training data.

Figure 5: A qualitative comparison of our model vs. the SOTA baselines

Finally, we test our model on real images taken around our lab using a
smartphone camera. The final output masks are shown in Figure 6. We see that
our model generalizes to real-world, complex scenarios zero-shot. However,
because of an imperfect classifier, it misses some submasks in certain complex
situations. We report more generated instance masks in Appendix D.

Figure 6: Qualitative results on real-world, complex scenarios zero-shot

##  6 Conclusions and future work

We introduced a novel method for DLO instance segmentation, leveraging SAM, a
robust foundation model for segmentation, alongside CLIPSeg, which combines
text prompts with semantic segmentation. We presented and tested an adapter
model that converts text-conditioned semantic masks embedding into point
prompts for SAM and introduced a novel approach to automatically filter
duplicate and low-quality instance masks generated by SAM, both conditioned on
text embedding, potentially enabling adaptation to standard rigid-object
instance segmentation with text prompts. This work is an avenue for future
research.

Our results show that our model performs exceptionally on our test dataset and
exhibits exceptional zero-shot transfer capability to other DLO-specific
datasets. However, the final outcome of the segmentation is upper-bounded by
the performance of SAM and CLIPSeg, as we observe from the Oracle tests (Refer
to Appendix E for a detailed discussion). The performance with Oracle
configuration exhibits the strength of our prompting network, given an optimal
classifier network. In the current state (without Oracle), our classifier
network limits the highest performance our model can achieve without access to
the ground truth. Improving our classifier network is an ongoing line of work.

## References

  * (1) Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk.  SLIC Superpixels Compared to State-of-the-art Superpixel Methods.  Technical report.  URL http://ivrg.epfl.ch/research/superpixels.
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer Normalization.  7 2016.  URL http://arxiv.org/abs/1607.06450.
  * Bomasani & Others (2021) Rishi Bomasani and Others.  On the Opportunities and Risks of Foundation Models.  8 2021.  URL http://arxiv.org/abs/2108.07258.
  * Caporali et al. (2022a) Alessio Caporali, Kevin Galassi, Riccardo Zanella, and Gianluca Palli.  FASTDLO: Fast Deformable Linear Objects Instance Segmentation.  _IEEE Robotics and Automation Letters_ , 7(4):9075–9082, 10 2022a.  ISSN 23773766.  doi: 10.1109/LRA.2022.3189791.
  * Caporali et al. (2022b) Alessio Caporali, Riccardo Zanella, Daniele De Greogrio, and Gianluca Palli.  Ariadne+: Deep Learning - Based Augmented Framework for the Instance Segmentation of Wires.  _IEEE Transactions on Industrial Informatics_ , 18(12):8607–8617, 12 2022b.  ISSN 19410050.  doi: 10.1109/TII.2022.3154477.
  * Caporali et al. (2023) Alessio Caporali, Kevin Galassi, Bare Luka Zagar, Riccardo Zanella, Gianluca Palli, and Alois C. Knoll.  RT-DLO: Real-Time Deformable Linear Objects Instance Segmentation.  _IEEE Transactions on Industrial Informatics_ , 2023.  ISSN 19410050.  doi: 10.1109/TII.2023.3245641.
  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-End Object Detection with Transformers.  5 2020.  URL http://arxiv.org/abs/2005.12872.
  * Chen et al. (2018) Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation.  2 2018.  URL http://arxiv.org/abs/1802.02611.
  * (9) Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, and Hengshuang Zhao.  FocalClick: Towards Practical Interactive Image Segmentation.  Technical report.
  * Chen et al. (2022) Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao.  Vision Transformer Adapter for Dense Predictions.  5 2022.  URL http://arxiv.org/abs/2205.08534.
  * Cheng et al. (2021) Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov.  Per-Pixel Classification is Not All You Need for Semantic Segmentation.  7 2021.  URL http://arxiv.org/abs/2107.06278.
  * Choi et al. (2023) Andrew Choi, Dezhong Tong, Brian Park, Demetri Terzopoulos, Jungseock Joo, and Mohammad Khalid Jawed.  mBEST: Realtime Deformable Linear Object Detection Through Minimal Bending Energy Skeleton Pixel Traversals.  2 2023.  URL http://arxiv.org/abs/2302.09444.
  * Cop et al. (2021) Konrad P. Cop, Arne Peters, Bare L. Zagar, Daniel Hettegger, and Alois C. Knoll.  New Metrics for Industrial Depth Sensors Evaluation for Precise Robotic Applications.  In _IEEE International Conference on Intelligent Robots and Systems_ , pp. 5350–5356. Institute of Electrical and Electronics Engineers Inc., 2021.  ISBN 9781665417143.  doi: 10.1109/IROS51168.2021.9636322.
  * De Gregorio et al. (2018) Daniele De Gregorio, Gianluca Palli, and Luigi Di Stefano.  Let’s take a Walk on Superpixels Graphs: Deformable Linear Objects Segmentation and Model Estimation.  10 2018.  URL http://arxiv.org/abs/1810.04461.
  * (15) Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova Google, and A I Language.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  Technical report.  URL https://github.com/tensorflow/tensor2tensor.
  * (16) Henghui Ding, Scott Cohen, Brian Price, and Xudong Jiang.  PhraseClick: Toward Achieving Flexible Interactive Segmentation by Phrase and Click.  Technical report.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  10 2020.  URL http://arxiv.org/abs/2010.11929.
  * Gao et al. (2021) Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  CLIP-Adapter: Better Vision-Language Models with Feature Adapters.  10 2021.  URL http://arxiv.org/abs/2110.04544.
  * Keipour et al. (2022a) Azarakhsh Keipour, Maryam Bandari, and Stefan Schaal.  Deformable One-Dimensional Object Detection for Routing and Manipulation.  1 2022a.  doi: 10.1109/LRA.2022.3146920.  URL http://arxiv.org/abs/2201.06775http://dx.doi.org/10.1109/LRA.2022.3146920.
  * Keipour et al. (2022b) Azarakhsh Keipour, Mohammadreza Mousaei, Maryam Bandari, Stefan Schaal, and Sebastian Scherer.  Detection and physical interaction with deformable linear objects.  _arXiv preprint arXiv:2205.08041_ , 2022b.
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment Anything.  4 2023.  URL http://arxiv.org/abs/2304.02643.
  * Kuhn (1955) H W Kuhn.  THE HUNGARIAN METHOD FOR THE ASSIGNMENT PROBLEM’.  Technical report, 1955.
  * LeCun et al. (1995) Yann LeCun, Lawrence D Jackel, Léon Bottou, Corinna Cortes, John S Denker, Harris Drucker, Isabelle Guyon, Urs A Muller, Eduard Sackinger, Patrice Simard, et al.  Learning algorithms for classification: A comparison on handwritten digit recognition.  _Neural networks: the statistical mechanics perspective_ , 261(276):2, 1995.
  * Li et al. (2021) Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao.  Grounded Language-Image Pre-training.  12 2021.  URL http://arxiv.org/abs/2112.03857.
  * Lin et al. (2017) Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal Loss for Dense Object Detection.  8 2017.  URL http://arxiv.org/abs/1708.02002.
  * Ling (2017) Fuyun Ling.  Carrier Synchronization.  In _Synchronization in Digital Communication Systems_ , pp. 193–270. Cambridge University Press, 6 2017.  doi: 10.1017/9781316335444.007.  URL https://www.cambridge.org/core/product/identifier/9781316335444%23CT-bp-5/type/book_part.
  * Liu et al. (2022a) Qin Liu, Zhenlin Xu, Gedas Bertasius, and Marc Niethammer.  SimpleClick: Interactive Image Segmentation with Simple Vision Transformers.  10 2022a.  URL http://arxiv.org/abs/2210.11006.
  * Liu et al. (2022b) Qin Liu, Meng Zheng, Benjamin Planche, Srikrishna Karanam, Terrence Chen, Marc Niethammer, and Ziyan Wu.  PseudoClick: Interactive Image Segmentation with Click Imitation.  7 2022b.  URL http://arxiv.org/abs/2207.05282.
  * Liu et al. (2023) Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang.  Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection.  3 2023.  URL http://arxiv.org/abs/2303.05499.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.  3 2021.  URL http://arxiv.org/abs/2103.14030.
  * Lüddecke & Ecker (2022) Timo Lüddecke and Alexander Ecker.  Image segmentation using text and image prompts. in 2022 ieee.  In _CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 7076–7086, 2022.
  * Lüddecke & Ecker (2021) Timo Lüddecke and Alexander S. Ecker.  Image Segmentation Using Text and Image Prompts.  12 2021.  URL http://arxiv.org/abs/2112.10003.
  * Milletari et al. (2016) Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation.  6 2016.  URL http://arxiv.org/abs/1606.04797.
  * Paszke et al. (2019) Adam Paszke et al.  PyTorch: An imperative style, high-performance deep learning library.  _Advances in Neural Information Processing Systems_ , 721(NeurIPS):8026–8037, 2019.  ISSN 10495258.
  * Perez et al. (2017) Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville.  FiLM: Visual Reasoning with a General Conditioning Layer.  9 2017.  URL http://arxiv.org/abs/1709.07871.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning Transferable Visual Models From Natural Language Supervision.  2 2021.  URL http://arxiv.org/abs/2103.00020.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.  Learning multiple visual domains with residual adapters.  5 2017.  URL http://arxiv.org/abs/1705.08045.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  5 2015.  URL http://arxiv.org/abs/1505.04597.
  * Sanchez et al. (2018) Jose Sanchez, Juan Antonio Corrales, Belhassen Chedli Bouzgarrou, and Youcef Mezouar.  Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey.  _International Journal of Robotics Research_ , 37(7):688–716, 6 2018.  ISSN 17413176.  doi: 10.1177/0278364918779698.
  * Selvaraj et al. (2023) Nithish Muthuchamy Selvaraj, Xiaobao Guo, Adams Kong, Bingquan Shen, and Alex Kot.  Adapter Incremental Continual Learning of Efficient Audio Spectrogram Transformers.  2 2023.  URL http://arxiv.org/abs/2302.14314.
  * Sung et al. (2021) Yi-Lin Sung, Jaemin Cho, and Mohit Bansal.  VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks.  12 2021.  URL http://arxiv.org/abs/2112.06825.
  * Tancik et al. (2020) Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng.  Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains.  6 2020.  URL http://arxiv.org/abs/2006.10739.
  * Vaswani et al. (2023) Ashish Vaswani, Google Brain, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention Is All You Need.  Technical report, 2023.
  * Yan et al. (2019) Mengyuan Yan, Yilin Zhu, Ning Jin, and Jeannette Bohg.  Self-Supervised Learning of State Estimation for Manipulating Deformable Linear Objects.  11 2019.  URL http://arxiv.org/abs/1911.06283.
  * Zanella et al. (2021) Riccardo Zanella, Alessio Caporali, Kalyan Tadaka, Daniele De Gregorio, and Gianluca Palli.  Auto-generated Wires Dataset for Semantic Segmentation with Domain-Independence.  In _2021 International Conference on Computer, Control and Robotics, ICCCR 2021_ , pp. 292–298. Institute of Electrical and Electronics Engineers Inc., 1 2021.  ISBN 9781728190358.  doi: 10.1109/ICCCR49711.2021.9349395.
  * Zhang et al. (2021) Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li.  Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling.  11 2021.  URL http://arxiv.org/abs/2111.03930.

##  Appendix A Training details

In our method, we chose the ViT-L/16-based model of SAM (Kirillov et al.,
2023) to attempt to balance the speed-accuracy trade-off. We observed that
this model gave high-quality segmentation masks while being $\sim 2\times$
smaller compared to ViT-H/16 in terms of the number of parameters. On the
other hand, for the CLIPSeg (Lüddecke & Ecker, 2021) model, we used the
ViT-B/16 based model, with $reduce\\_dim=64$. Throughout our training process,
we freeze the weights and biases of both foundational models, SAM and CLIPSeg.

Throughout the training phase, our text prompt remains ”cables,” with the aim
of obtaining instance segmentation for _all_ the cables within the image.
During training, we conduct augmentations applied to the input images. These
augmentations encompass random grayscale conversion, color jitter, and patch-
based and global Gaussian blurring and noise. In terms of computing, our model
is trained using $2$ NVIDIA A5000 GPUs, each equipped with $32$ GB of memory.
All testing is carried out on a single NVIDIA RTX 2080 GPU with $16$ GB of
memory.

Our training procedure employs a learning rate warm-up spanning $5$ epochs,
followed by a cosine decay. The peak learning rate is set to $lr=0.0008$, in
line with recommendations from Kirillov et al. (2023). We employ the default
_AdamW_ optimizer from Paszke et al. (2019) with a weight decay of $0.01$. The
convergence graphs and learning rate profile of all the models can be seen in
Figure 7. Figure 8 displays the binary classification accuracy and mIoU
computed on the validation dataset during training. No smoothing was applied
in creating the plots.

(a) Learning rate profile

(b) Convergence graphs

Figure 7: Learning rate profile and convergence graphs

(a) mIoU[%]

(b) Binary classifier accuracy[%]

Figure 8: Binary classification accuracy and mIoU

A brief summary of all the _hyperparameters_ can be seen the Table 4

Hyperparameter | Value
---|---
Number of epochs | $50$
Max learning rate | $8\times 10^{-4}$
Learning rate warmup |  $5$ (epochs)
Optimizer | _AdamW_
Optimizer weight decay | $0.01$
Batch size | $1$
Attention dropout | $0.5$
Number of prompts per batch ($N$) | $11$
Number of points per prompt ($N_{p}$) | $3$
Number of attention heads (for all models) | $8$
Embedding dimension | $256$
SAM model type |  _ViT-L/16_ (frozen)
CLIPSeg model type |  _ViT-B/16_ (frozen)
Focal loss weight | $20$
DICE loss weight | $1$
Positive label weight (binary cross-entropy) | $3$
Classifier MLP activation | _ReLU_
Prompt encoder MLP activation | _GELU_
Train dataset size | $20038$
Validation dataset size | $3220$
Test dataset size | $3233$
Image size | $(1920\times 1080)$
Total number of parameters (including foundation models) | 466M
Trainable parameters | 3.3M
Table 4: Hyperparameters

##  Appendix B Generated dataset

The dataset we presented contains $20038$ train images, $3220$ validation
images, and $3233$ test images. Each image we provide is accompanied by its
corresponding semantic mask and binary submasks for each DLO in the image. The
images are located in {train,test,val}/RGB, and named as
train/RGB/00000_0001.png, train/RGB/00001_0001.png, and so on. In the {train,
test, val}/Masks folder, we have a sub-folder containing the binary submasks
for each corresponding RGB image. For example, train/Masks/00000 contains all
the submasks corresponding to train/RGB/00000_0001.png. Additionally, the
semantic mask for train/RGB/00000_0001.png is called
train/Masks/00000_mask.png. The folder structure can be seen in Figure 9.

Figure 9: Dataset directory tree

Each image and its corresponding masks are $1920\times 1080$ in resolution.
The number of cables, their thickness, color, and bending profile are randomly
generated for each image. There are 4 possible colors for the cables - _cyan,
white, yellow, black_. The number of cables in each image is randomly sampled
from $1$ to $10$. A sample image from the dataset, along with its
corresponding submasks and semantic mask, are portrayed in Figure 10.

Figure 10: Dataset Example

##  Appendix C Ablation study

Within the _prompt encoder network_ , each block is fundamental to the proper
sampling of point embedding vectors conditioned on text and performs a
specific function in the process, as outlined in section 3.1.1. However, the
_classifier network_ architecture requires an in-depth ablation study. The
modularity of our method permits the independent testing of the classifier and
prompt encoder networks. We train multiple configurations of the classifier
network and compare the classification accuracy that the network achieves, on
the test dataset. In the remainder of this section, we discuss various
configurations of the classifier network and the labelling accuracy they
achieve. The results are summarized in the Table 5.

The mask token that SAM outputs, contains encoded information about each of
the $N$ submasks. Using this information, we need to generate tokens that can
be used to classify each submask as _keep_ or _discard_. Each of these $N$
tokens must be correlated with each other, which will extract information
about duplicate masks. For this purpose we must use a self attention layer.
The first study (config 1 in Table 5) was conducted by passing these tokens
through a self attention layer followed by a classifier MLP. This architecture
caused the classifier network to diverge.

In the second study (config 2 in Table 5), we introduced $N$ trainable tokens
(we call these tokens _classifier tokens_ in the remainder of this section).
These tokens are first passed through a self-attention block, followed by
cross attention with the mask tokens generated by SAM and finally through a
classifier MLP. The cross attention block was added to fuse the submask
information with the classifier token information. This method converged to a
final binary classification accuracy of $76.37\%$.

The third study was conducted similar to _config 2_ except that we switched
the order of self and cross attention. Namely in config 3, we first fuse the
mask token information and classifier token information using a cross
attention block, which is then followed by a self attention block and then a
classifier MLP. This configuration achieved an accuracy of $81.18\%$. This
shows that cross attention-based fusing before self-attention improves
classification performance. However, by using this method, the classifier
network does not have direct access to the text information and cannot be
controlled using text prompts for example.

In the fourth configuration (config 4 in Table 5), we no longer instantiate
trainable tokens. We take the output of the _sampler attention_ block from the
_prompt encoder network_ (Fig. 3), which are the text-conditioned prompt
embedding vectors. These embedding vectors are then fused using cross
attention with mask tokens, followed by self attention and then a classifier
MLP. This method achieves an accuracy of $80.26\%$, but improves _config 3_ ,
by providing the classifier network with direct access to text information.

In the final configuration (config 5 in Table 5), we use the classifier
network exactly as portrayed in Fig. 3. Here we transform the text-conditioned
prompts using an MLP, before passing it through the cross and self attention
blocks. This method boosts the classifier performance significantly, with a
classification accuracy of $84.83\%$. The comparison of all the aforementioned
configurations is summarized in Table 5

Configuration | Binary Classification Accuracy [$\%$]
---|---
config 1 | NA (model diverged)
config 2 | 76.37
config 3 | 81.18
config 4 | 80.26
config 5 | 84.83
Table 5: Ablations of the classifier network (values in bold signify best
performance)

##  Appendix D More qualitative results

Figures 11 and 12 show more instance segmentation masks generated by ISCUTE,
on datasets from RT-DLO and mBEST as well as on our generated test dataset,
respectively. All the images are examples of masks generated without oracle.

Figure 11: Qualitative results on RT-DLO and mBEST images Figure 12:
Qualitative results on our test images

##  Appendix E Limitations of foundation models

The Oracle test results from Tables 1 and 2 display the maximum one-shot
performance that can be achieved with the ViT-L/16-based version of SAM on the
DLO-dataset, with up to 3 points in a prompt. The mask refinement method
presented in SAM (Kirillov et al., 2023) can potentially improve this upper
bound, but for a practical use-case, one-shot mask generation is desirable.
CLIPSeg also has some limitations on generating the perfect heatmap embedding
for a given text prompt. This is demonstrated in Figure 13, where for the same
prompt ”cables,” the model successfully locates all the cables in the bottom
image, while it fails to locate the white cable in the top one.

Figure 13: CLIPSeg generated heatmaps. In the top image, CLIPSeg is
unsuccessful in detecting the white cable, while it detects all the cables in
the bottom one. The prompt used was ”cables”. Figure 14: CLIPSeg’s
effectiveness on complicated text-prompts


blue indicating relatively low values. A dot with a high SHAP value for a
feature suggests a positive contribution to the prediction, whereas a very
negative SHAP value leads to a lower prediction. The features are ranked in
order of descending average importance and the blue, green and orange squares
indicate whether the feature is an environmental, agrological or socioeconomic
characteristic. Figure 3: Comparison of the application rates per ha per year
for various crops between our predicted model output and the data reported by
the United States Department of Agriculture (USDA) for the USA. Figure 4:
Comparison of the application rates per ha per year for various crops between
our predicted model output and the data reported by the Department for
Environment, Food & Rural Affairs (DEFRA) for the UK.

Figure 5: Spatial pattern of crop-specific fertilizer (N) consumed by each
0.05° grid cell for the following: a) average for the 1960s decade across all
13 crop classes, b) average for the 1960s decade for wheat, c) average for the
1960s decade for rice, d) average for the 1960s decade for maize, e) average
for the 1960s decade for other cereals, f) average for the 1960s decade for
all oil crops, g) average for the 1960s decade for vegetables and fruits, h)
average for the 1960s decade for roots and tubers, sugar crops, fiber crops,
and other crop classes, i) average for the 2010s decade across all 13 crop
classes, j) average for the 2010s decade for wheat, k) average for the 2010s
decade for rice, l) average for the 2010s decade for maize, m) average for the
2010s decade for other cereals, n) average for the 2010s decade for all oil
crops, o) average for the 2010s decade for vegetables and fruits, p) average
for the 2010s decade for roots and tubers, sugar crops, fiber crops, and other
crop classes. The 1960s decade includes the years 1961-1969, and the 2010s
decade includes the years 2010-2019

Figure 6: Spatial pattern of crop-specific fertilizer (P2O5) consumed by each
0.05° grid cell for the following: a) average for the 1960s decade across all
13 crop classes, b) average for the 1960s decade for wheat, c) average for the
1960s decade for rice, d) average for the 1960s decade for maize, e) average
for the 1960s decade for other cereals, f) average for the 1960s decade for
all oil crops, g) average for the 1960s decade for vegetables and fruits, h)
average for the 1960s decade for roots and tubers, sugar crops, fiber crops,
and other crop classes, i) average for the 2010s decade across all 13 crop
classes, j) average for the 2010s decade for wheat, k) average for the 2010s
decade for rice, l) average for the 2010s decade for maize, m) average for the
2010s decade for other cereals, n) average for the 2010s decade for all oil
crops, o) average for the 2010s decade for vegetables and fruits, p) average
for the 2010s decade for roots and tubers, sugar crops, fiber crops, and other
crop classes. The 1960s decade includes the years 1961-1969, and the 2010s
decade includes the years 2010-2019

Figure 7: Spatial pattern of crop-specific fertilizer (K2O) consumed by each
0.05° grid cell for the following: a) average for the 1960s decade across all
13 crop classes, b) average for the 1960s decade for wheat, c) average for the
1960s decade for rice, d) average for the 1960s decade for maize, e) average
for the 1960s decade for other cereals, f) average for the 1960s decade for
all oil crops, g) average for the 1960s decade for vegetables and fruits, h)
average for the 1960s decade for roots and tubers, sugar crops, fiber crops,
and other crop classes, i) average for the 2010s decade across all 13 crop
classes, j) average for the 2010s decade for wheat, k) average for the 2010s
decade for rice, l) average for the 2010s decade for maize, m) average for the
2010s decade for other cereals, n) average for the 2010s decade for all oil
crops, o) average for the 2010s decade for vegetables and fruits, p) average
for the 2010s decade for roots and tubers, sugar crops, fiber crops, and other
crop classes. The 1960s decade includes the years 1961-1969, and the 2010s
decade includes the years 2010-2019

## References

  * [1] Guohua Xu, Xiaorong Fan, and Anthony J. Miller.  Plant nitrogen assimilation and use efficiency.  Annual Review of Plant Biology, 63:153–182, 2012.
  * [2] Jianbo Shen, Lixing Yuan, Junling Zhang, Haigang Li, Zhaohai Bai, Xinping Chen, Weifeng Zhang, and Fusuo Zhang.  Phosphorus dynamics: From soil to plant.  Plant Physiology, 156:997–1005, 2011.
  * [3] Jordi Sardans and Josep Peñuelas.  Potassium control of plant functions: Ecological and agricultural implications.  Plants, 10, 2021.
  * [4] Jordi Sardans and Josep Peñuelas.  Potassium: A neglected nutrient in global change.  Global Ecology and Biogeography, 24:261–275, 2015.
  * [5] C. I. Ludemann, N. Wanner, P. Chivenge, A. Dobermann, R. Einarsson, P. Grassini, A. Gruere, K. Jackson, L. Lassaletta, F. Maggi, G. Obli-Laryea, M. K. van Ittersum, S. Vishwakarma, X. Zhang, and F. N. Tubiello.  A global FAOSTAT reference database of cropland nutrient budgets and nutrient use efficiency (1961–2020): nitrogen, phosphorus and potassium.  Earth System Science Data, 16(1):525–541, 2024.
  * [6] R. Xu, H. Tian, S. Pan, S. R. S. Dangal, J. Chen, J. Chang, Y. Lu, U. M. Skiba, F. N. Tubiello, and B. Zhang.  Increased nitrogen enrichment and shifted patterns in the world’s grassland: 1860-2016.  Earth System Science Data, 11(1):175–187, 2019.
  * [7] MA Sutton, Bleeker A, Bekunda M, Grizzetti B, de Vries W, van Grinsven HJM, Abrol YP, Adhya TK, Billen G, Davidson EA, Datta A, Diaz R, Erisman JW, Liu XJ, Oenema O, Palm C, Raghuram N, Reis S, Scholz RW, Sims T, Yan XY, and Zhang Y.  Our Nutrient World: The challenge to produce more food and energy with less pollution.  Centre for Ecology and Hydrology (CEH), Edinburgh UK on behalf of the Global Partnership on Nutrient Management and International Nitrogen Initiative., 2013.
  * [8] Josep Penuelas, Ivan A. Janssens, Philippe Ciais, Michael Obersteiner, and Jordi Sardans.  Anthropogenic global shifts in biospheric n and p concentrations and ratios and their impacts on biodiversity, ecosystem productivity, food security, and human health.  Global Change Biology, 26:1962–1985, 2020.
  * [9] Wulahati Adalibieke, Xiaoqing Cui, Hongwei Cai, Liangzhi You, and Feng Zhou.  Global crop-specific nitrogen fertilization dataset in 1961-2020.  Scientific data, 10(1):617, 12 2023.
  * [10] Chaoqun Lu and Hanqin Tian.  Global nitrogen and phosphorus fertilizer use for agriculture production in the past half century: Shifted hot spots and nutrient imbalance.  Earth System Science Data, 9(1):181–192, 2017.
  * [11] Kazuya Nishina, Akihiko Ito, Naota Hanasaki, and Seiji Hayashi.  Reconstruction of spatially detailed global map of $NH_{4}^{+}$ and $NO_{3}^{-}$ application in synthetic nitrogen fertilizer.  Earth System Science Data, 9:149–162, 2017.
  * [12] Graham K. MacDonald, Elena M. Bennett, Philip A. Potter, and Navin Ramankutty.  Agronomic phosphorus imbalances across the world’s croplands.  Proceedings of the National Academy of Sciences of the United States of America, 108:3086–3091, 2011.
  * [13] Peiyu Cao, Chaoqun Lu, and Zhen Yu.  Historical nitrogen fertilizer use in agricultural ecosystems of the contiguous united states during 1850-2015: Application rate, timing, and fertilizer types.  Earth System Science Data, 10, 2018.
  * [14] Zhen Yu, Jing Liu, and Giri Kattel.  Historical nitrogen fertilizer use in china from 1952 to 2018.  Earth System Science Data, 14:5179–5194, 2022.
  * [15] Richard T Conant, Aaron B Berdanier, and Peter R Grace.  Patterns and trends in nitrogen use and nitrogen recovery efficiency in world agriculture.  Global Biogeochemical Cycles, 27(2):558–566, 2013.
  * [16] Cameron I. Ludemann, Armelle Gruere, Patrick Heffer, and Achim Dobermann.  Global data on fertilizer use by crop and by country.  Scientific Data, 9(1):502, 2022.
  * [17] Adolfo Martinez and Ray B Diamond.  Fertilizer Use Statistics In Crop Production.  Technical Report T-24, International Fertilizer Developemnt Center, 9 1982.
  * [18] Adolfo Martinez.  Fertilizer use statistics and crop yields.  Technical Report T-37, International Fertilizer Development Center, Muscle Shoals, Alabama, 1 1990.
  * [19] EFMA.  Fertilizer application by crop in EU countries 2001/02.  Technical report, EFMA, unpublished.
  * [20] EFMA.  Fertilizer application by crop in EU countries 2006/07.  Technical report, EFMA, unpublished.
  * [21] Fertilizer Europe.  Fertilizer application by crop in EU countries 2011/12.  Technical report, Fertilizer Europe, unpublished.
  * [22] Fertilizer Europe.  Fertilizer application by crop in EU countries 2014/15.  Technical report, Fertilizer Europe, unpublished.
  * [23] Tianqi Chen and Carlos Guestrin.  XGBoost: A Scalable Tree Boosting System.  Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 13-17-August-2016:785–794, 3 2016.
  * [24] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay.  Scikit-learn: Machine Learning in Python.  Journal of Machine Learning Research, 12(85):2825–2830, 2011.
  * [25] Scott M Lundberg, Paul G Allen, and Su-In Lee.  A Unified Approach to Interpreting Model Predictions.  Advances in Neural Information Processing Systems, 30, 2017.
  * [26] T. Zou, X. Zhang, and E. A. Davidson.  Global trends of cropland phosphorus use and sustainability challenges.  Nature, 611:81–87, 2022.
  * [27] Luis Lassaletta, Gilles Billen, Bruna Grizzetti, Juliette Anglade, and Josette Garnier.  50 year trends in nitrogen use efficiency of world cropping systems: The relationship between yield and nitrogen input to cropland.  Environmental Research Letters, 9:105011, 2014.
  * [28] Rasmus Einarsson, Alberto Sanz-Cobena, Eduardo Aguilera, Gilles Billen, Josette Garnier, Hans J.M. van Grinsven, and Luis Lassaletta.  Crop production and nitrogen use in european cropland and grassland 1961–2019.  Scientific Data, 8:288, 2021.
  * [29] Chad Monfreda, Navin Ramankutty, and Jonathan A Foley.  Farming the planet: 2. geographic distribution of crop areas, yields, physiological types, and net primary production in the year 2000.  Global biogeochemical cycles, 22(1), 2008.
  * [30] Kees Klein Goldewijk, Arthur Beusen, Jonathan Doelman, and Elke Stehfest.  Anthropogenic land use estimates for the holocene–hyde 3.2.  Earth System Science Data, 9(2):927–953, 2017.
  * [31] FAO, IFA, and IFDC.  Fertilizer use by crop 1.  Technical Report ESS/MISC/1992/3, FAO, Viale delle Termi di Caracalla, 00100, Rome, 10 1992.
  * [32] FAO, IFA, and IFDC.  Fertilizer use by crop 2.  Technical Report ESS/MISC/1994/4, FAO, Viale delle Termi di Caracalla, 00100, Rome, 1994.
  * [33] FAO, IFA, and IFDC.  Fertilizer use by crop 3.  Technical Report ESS/MISC/1996/1, FAO, Viale delle Termi di Caracalla, 00100, Rome, 1996.
  * [34] FAO, IFA, and IFDC.  Fertilizer use by crop 4.  Technical report, FAO, Viale delle Termi di Caracalla, 00100, Rome, 1999.
  * [35] FAO, IFA, and IFDC.  Fertilizer use by crop 5.  Technical report, FAO, Viale delle Termi di Caracalla, 00100, Rome, 2002.
  * [36] Pattrick Heffer.  Asessment of Fertilizer Use by Crop at the Global Level 2006/07 - 2007/08.  Technical Report A/09/55, International Fertilizer Industry Association, rue Marbeuf - 75008, Paris, April 2009.
  * [37] Patrick Heffer, Armelle Gruère, and Terry Roberts.  Assessment of fertilizer use by crop at the global level 2014-2014/15.  Technical Report A/17/134, International Fertilizer Industry Association & International Plant Nutrition Insitute, November 2017.
  * [38] USDA.  Fertilizer use and price, 2019.  https://www.ers.usda.gov/data-products/fertilizer-use-and-price/.
  * [39] Agricultural Census Division.  All india bulletin on input survey 1986-87.  Technical report, Goverment of India, 9 1992.
  * [40] Agricultural Census Division.  All india report on input survey 1991-1992.  Technical Report AGRI/2000-1A, Goverment of India, 1 2000.
  * [41] Agricultural Census Division.  All india report on input survey 1996-1997.  Technical report, Goverment of India, 2007.
  * [42] Agriculture Census Division.  All india report on input survey 2001-02.  Technical report, Ministry of Agriculture, 8 2008.
  * [43] Agriculture Census Division.  All india report on input survey 2006-07.  Technical report, Ministry of Agriculture, 3 2012.
  * [44] Agriculture Census Division.  All india report on input survey 2011-12.  Technical report, Minister of Agriculture & Farmers Welfare, 4 2016.
  * [45] Agriculture Census Division.  All india report on input survey 2016-17.  Technical report, Minister of Agriculture, Cooperation & Farmers Welfare, 2 2021.
  * [46] FAO.  World programme for the census of agriculture 2020: Programme, concepts and definitions, 2015.
  * [47] FAOSTAT.  Crops and livestock products, 2023.  https://www.fao.org/faostat/en/##data/QCL.
  * [48] Pattrick Heffer.  Asessment of Fertilizer Use by Crop at the Global Level 2010 - 2010/11.  Technical Report A/13/111, International Fertilizer Industry Association, rue Marbeuf - 75008, Paris, August 2013.
  * [49] Maiz’Europ’.  Figures, 2024.  https://www.maizeurop.com/en/structure/cepm/figures/.
  * [50] DEFRA.  British survey of fertiliser practice dataset, 7 2023.  https://www.gov.uk/government/statistical-data-sets/british-survey-of-fertiliser-practice-dataset.
  * [51] NFDC.  Statistics national fertilizer development centre, 2023.  http://www.nfdc.gov.pk/stat.html.
  * [52] Philippine Statistics Authority.  Estimated inorganic fertilizer use by geolocation, grade and year, by area harvested and year, 2023.  https://openstat.psa.gov.ph/.
  * [53] Roehlano M Briones.  The fertilizer industry and philippine agriculture: Policies, problems, and priorities.  Philippine Journal of Development, 43, 2017.
  * [54] Official Statistics of Sweden.  Use of fertilisers and animal manure in agriculture in 2010/11.  Technical Report SM 1203, Official Statistics of Sweden, 6 2014.
  * [55] Official Statistics of Sweden.  Use of fertilisers and animal manure and cultivation measures in agriculture 2012/13.  Technical Report SM 1402, Official Statistics of Sweden, 6 2014.
  * [56] Official Statistics of Sweden.  Use of fertilisers and animal manure and cultivation measures in agriculture 2015/16.  Technical Report SM 1702, Official Statistics of Sweden, 6 2017.
  * [57] Official Statistics of Sweden.  Use of fertilisers and animal manure and cultivation measures in agriculture 2018/19.  Technical Report SM 2002, Official Statistics of Sweden, 6 2020.
  * [58] Stats NZ.  Agricultural production survey, 2021.  https://www.stats.govt.nz/indicators/fertilisers-nitrogen-and-phosphorus.
  * [59] FAOSTAT.  Fertilizers by nutrient, 2023.  https://www.fao.org/faostat/en/##data/RFN.
  * [60] CEPAL, FAO, and BID.  El uso de fertilizantes en Argentina.  Technical Report E/CN.12/741, CEPAL, 3 1966.
  * [61] Fertilizar Asociación Civil.  Consumo de fertilizantes por cultivo 2006, 2006.  https://fertilizar.org.ar/estadisticas/.
  * [62] Fertilizar Asociación Civil.  Consumo de fertilizantes por cultivo 2011, 2011.  https://fertilizar.org.ar/estadisticas/.
  * [63] Fertilizar Asociación Civil.  Consumo de fertilizantes por cultivo 2012, 2012.  https://fertilizar.org.ar/estadisticas/.
  * [64] Fertilizar Asociación Civil.  Consumo de fertilizantes por cultivo 2013, 2013.  https://fertilizar.org.ar/estadisticas/.
  * [65] Fertilizar Asociación Civil.  Consumo de fertilizantes por cultivo 2014, 2014.  https://fertilizar.org.ar/estadisticas/.
  * [66] Fertilizar Asociación Civil.  Consumo de fertilizantes por cultivo 2015, 2015.  https://fertilizar.org.ar/estadisticas/.
  * [67] Fertilizar Asociación Civil.  Consumo de fertilizantes por cultivo 2016, 2016.  https://fertilizar.org.ar/estadisticas/.
  * [68] Fertilizar Asociación Civil.  Consumo de fertilizantes por cultivo 2017, 2017.  https://fertilizar.org.ar/estadisticas/.
  * [69] Fertilizar Asociación Civil.  Consumo de fertilizantes por cultivo 2018, 2018.  https://fertilizar.org.ar/estadisticas/.
  * [70] FAO.  Fertilizer use by crop in Brazil.  Technical Report TC/D/Y5376E/1/05.04/300, FAO, Viale delle Terme di Caracalla 00100, Rome, 2004.
  * [71] Robert M. Boddey, Deise F. Xavier, Bruno J.R. Alves, and Segundo Urquiaga.  Brazilian agriculture: The transition to sustainability.  Journal of Crop Production, 9:593–621, 2003.
  * [72] Sarah J. Pogue, Nicole Bamber, Hugues Imbeault-Tétreault, Adeline Baudoin, Aaron L. McPherson, Steven Kega, Madavine Tom, Roland Kröbel, Kim H. Ominski, Karen A. Beauchemin, Brenna Grant, and Tim A. McAllister.  Regionalized life cycle inventory data collection and calculation for perennial forage production in canada: methodological best practices and limitations.  International Journal of Life Cycle Assessment, 2023.
  * [73] James D. Beaton and Joseph Berger.  Present and Potential Use of Fertilizer for Forage Production in Temperate Zones, chapter 2, pages 17–37.  American Society of Agronomy, 1974.
  * [74] CEPAL, FAO, and BID.  El uso de fertilizantes en Chile.  Technical Report E/CN.12/757, CEPAL, 6 1966.
  * [75] Rubén D. Nuñez.  Estudio sobre el mercado de fertilizantes en la República Dominicana.  Technical Report IICA/E72/9, Instituto Inteamericano de Cooperación para la Agricultural, United States Agency for International Development, Secretariado Técnico de la Presidencia Oficina Nacional de Planificación ONAPLAN, 9 1999.
  * [76] Réal Michaud, W. F. Lehman, and M. D. Rumbaugh.  World distribution and historical development, chapter 2.  Agronomy Monographs. American Society of Agronomy, 1988.
  * [77] L. B Nelson, M. H Mcvickar, R. D Munson, L. F Seatz, S. L Tisdale, and W. C. White.  Changing patterns in fertilizer use.  Soil Science Society of America, inc, 1968.
  * [78] Darrell A. Russel, W. Joe Free, and Donald L. McCune.  Potential for Fertilizer Use on Tropical Forages, chapter 3, pages 39–65.  American Society of Agronomy, 1974.
  * [79] Eduardo Casanova.  Problemática de los fertilizantes en Venezuela.  Venesuelos, 12:5–16, 2004.
  * [80] J. C.B. Dubeux, L. E. Sollenberger, B. W. Mathews, J. M. Scholberg, and H. Q. Santos.  Nutrient cycling in warm-climate grasslands.  Crop Science, 47:915–928, 2007.
  * [81] R. P. Rawnsley, A. P. Smith, K. M. Christie, M. T. Harrison, and R. J. Eckard.  Current and future direction of nitrogen fertiliser use in australian grazing systems.  Crop and Pasture Science, 70:1034–1043, 2019.
  * [82] P. M. Barrow.  Potassium fertilizer use in south australia.  Agronomy branch report, 1:1–11, 1968.
  * [83] P. H. Williams and R. J. Haynes.  Influence of improved pastures and grazing animals on nutrient cycling within new zealand soils.  New Zealand Journal of Ecology, 14:49–57, 1990.
  * [84] Karl Buchgraber, Andreas Schaumberger, and Erich M Pötsch.  Grassland farming in Austria - status quo and future prospective.  Grassland Science in Europe, 16:13–24, 2011.
  * [85] Stany Vandermoere.  Improving agricultural phosphorus use efficiency and reducing soil phosphorus losses at the field scale. PhD thesis, Ghent University, 2020.
  * [86] T. Németh.  Past, Present and Future Status of N-Fertilization Policies in Hungary, pages 243–252.  Springer Netherlands, 2000.
  * [87] Philip M. Raup.  Postwar recovery of Western German agriculture.  Journal of Farm Economics, 32:1–14, 1950.
  * [88] H. J. Styhr Petersen.  Forecasting Danish nitrogen fertilizer consumption.  Industrial Marketing Management, 6:211–221, 1977.
  * [89] P Virkajärvi, M Rinne, J Mononen, O Niskanen, K Järvenranta, and A Sairananen.  Dairy production systems in Finland.  18th Symposium of the European Grassland Federation, Grassland and forages in high output dairy farming systems, 20:51–66, 2015.
  * [90] Geniana R Edwards and Geraldine W Abbott.  The Agricultural Economy of Finland, volume 169 of ERS foreign.  US Department of Agriculture, Economic Research Service, 1966.
  * [91] J. Le Noë, G. Billen, F. Esculier, and J. Garnier.  Long-term socioecological trajectories of agro-food systems revealed by n and p flows in french regions from 1852 to 2014.  Agriculture, Ecosystems and Environment, 265:132–143, 2018.
  * [92] Ministère de l’Agriculture.  Les Praires en 1982, volume 233 of Collections de statistique agricole.  Ministère de l’Agriculture, 1984.
  * [93] Véronique Rabaud and Michelle Cesses.  Enquête sur les practiques culturales - 2001.  Technical Report 159, Ministère de l’Agriculture (SSP), 6 2004.
  * [94] Ministère de l’Agriculture (SSP).  Pratiques culturales - 2006.  https://agreste.agriculture.gouv.fr/agreste-web/disaron/Dos8/detail/, 2006.
  * [95] Ministère de l’Agriculture (SSP).  Pratiques culturales sur les grandes cultures - 2017.  https://agreste.agriculture.gouv.fr/agreste-web/disaron/Chd2009/detail/, 2020.
  * [96] H. A. Pearson, C. H Herbel, and D. T Pendleton.  A tour of East German agriculture.  Rangelands, 1:9–11, 1979.
  * [97] D.A Panos, S. Sotiriadis, and E Fikas.  Grassland’s progress in Greece.  Der Z’́uchter, 31:37–47, 1961.
  * [98] Yong Hou, Lin Ma, Katalin Sárdi, István Sisák, and Wenqi Ma.  Nitrogen flows in the food production chain of hungary over the period 1961–2010.  Nutrient Cycling in Agroecosystems, 102:335–346, 2015.
  * [99] W. E. Murphy, W. F. O’Keeffe, and Foras Taluntais.  Fertiliser use surveys, 1972, 1974 & 1975\.  Technical Report 14, Fertiliser Association of Ireland, 1978.
  * [100] W. E. Murphy and W. F. O’Keefe.  Fertiliser use survey 1981-82.  Technical Report 24, The Fertiliser Association of Ireland, 11 1983.
  * [101] W. E. Murphy and O’Keefem W. F.  Fertiliser use survey.  Technical Report 27, The Fertiliser Association of Ireland, 11 1987.
  * [102] B. S. Coulter, W. E. Murphy, N. Culleton, G. Quinlan, and L. Connolly.  A survey of fertilizer use from 2001-2003 for grassland and arable crops.  Technical report, Teagasc, 7 2005.
  * [103] S.T.J Lalor, B. S. Coulter, G. Quinlan, and L. Connolly.  A survey of fertilizer use in Ireland from 2004-2008 for grasslands and arable crops.  Technical report, Teagasc, 3 2010.
  * [104] Emma Dillon, Cathal Buckley, Brian Moran, John Lennon, and David Wall.  Teagasc national farm survey, fertiliser use survey 2005-2015.  Technical report, Teagasc, 5 2018.
  * [105] Ireland CSO.  Table ACEN2: Area used by rural district, farm land utilisation and year (1926–1980), 2007.  https://data.cso.ie/.
  * [106] Ireland CSO.  Table AQA02: Farm land utilisation in june by type of land use, year and region (1980–1999), 2020.  https://data.cso.ie/.
  * [107] Ireland CSO.  Table AQA01: Area farmed in june by type of land use, year and region (1991–2007), 2021.  https://data.cso.ie/.
  * [108] Ireland CSO.  Table AQA05: Area farmed in june by type of land use, year and region (2008–2012), 2020.  https://data.cso.ie/.
  * [109] Ireland CSO.  Table AQA06: Area farmed in june by type of land use, year and region (2013–2023), 2024.  https://data.cso.ie/.
  * [110] T. Walsh, P. F. Ryan, and J. Kilroy.  A half century of fertiliser and lime use in Ireland.  Journal of the Statistical and Social Inquiry Society of Ireland, 19:104–136, 1957.
  * [111] John. F. Heavy.  The economic optimum use of fertilisers in Ireland.  Technical Report 2, Fertiliser Association of Ireland, 10 1969.
  * [112] W. H Prins.  Limits to nitrogen fertilizer on grassland.  PhD thesis, Wageningen University & Research, 3 1983.
  * [113] Eliza Kurek.  The fertilizing of main crops in peasant farming and its effectiveness.  Zagadnienia Ekonomiki Rolnej, 3:59–71, 1971.
  * [114] Pilar García-Serrano Jiménez, Juan José Lucena Marotta, Sebastián Ruano Criado, and Mariano Nogales García.  Guía Práctica de la Fertilización Racional de los Cultivos en España. Parte 1. Ministerio de Medio Ambiente y Medio Rural y Marino, 2010.
  * [115] Ewert Åberg.  Recent changes in swedish crop production.  Advances in Agronomy, 7:39–74, 1955.
  * [116] B. M. Church and J. Webber.  Fertiliser practice in england and wales: A new series of surveys.  Journal of the Science of Food and Agriculture, 22:1–7, 1971.
  * [117] DEFRA.  United Kingdom land areas, livestock numbers and agricultural workforce on agricultural holdings on 1 June, 2024.  https://www.gov.uk/government/statistical-data-sets/structure-of-the-agricultural-industry-in-england-and-the-uk-at-june.
  * [118] DEFRA.  Crops areas and livestock numbers in England from the June Census of Agriculture: 1900-2010, 2024.  https://www.gov.uk/government/statistical-data-sets/structure-of-the-agricultural-industry-in-england-and-the-uk-at-june.
  * [119] Torfi Jóhannesson.  Agriculture in Iceland Conditions and Characteristics.  The Agricultural University of Iceland, 2010.
  * [120] A. Helgadóttir, E. Eythórsdóttir, and T. Jóhannesson.  Agriculture in iceland - a grassland based production.  Grassland Science in Europe, 18:30–43, 2013.
  * [121] Office fédéral de la statistique.  Surface agricole utile sans les alpages, 1 2024.  https://www.bfs.admin.ch/bfs/fr/home/statistiques/agriculture-sylviculture/agriculture.assetdetail.30245951.html.
  * [122] Asbjørn Nordgård.  Orientation and intensity of Norwegian agriculture.  Norsk Geografisk Tidsskrift, 29:169–220, 1975.
  * [123] H Steinshamn, L Nesheim, and A K Bakken.  Grassland production in Norway.  Grassland Science in Europe, 21:15–25, 2016.
  * [124] The Federal Administration for Plant Protection and Veterinarian Medicine.  Yugoslavia: Country report to the fao international technical conference on plan genetic resources (leipzig, 1996).  Technical report, FAO, 11 1995.
  * [125] Z. Lugić, D. Lazarević, P. Erić, V. Mihajlović, and S. Vučković.  The state of forage crops production in Serbia.  Biotechnology in Animal Husbandry, 26:29–47, 2010.
  * [126] G. Loza and I. Kurtsev.  The growth of productive forces in agriculture in the tenth five-year plan.  Problems in Economics, 19:3–22, 1977.
  * [127] W. Klatt.  Reflections on the 1975 soviet harvest.  Soviet Studies, 28:485–498, 1976.
  * [128] Jacklyn Y. Shend.  Agricultural statistics of the former USSR republics and the Baltic States.  Number 863 in Statistical bulletin (United States Department of Agriculture). U.S. Department of Agriculture, 1993.
  * [129] Mekhlis Suleimenov and Peter Oram.  Trends in feed, livestock production, and rangelands during the transition period in three Central Asian countries.  Food Policy, 25:681–700, 2000.
  * [130] F. Zhang, J. Qi, F. M. Li, C. S. Li, and C. B. Li.  Quantifying nitrous oxide emissions from chinese grasslands with a process-based model.  Biogeosciences, 7:2039–2050, 2010.
  * [131] Atushi Ushimaru, Kei Uchida, and Takeshi Suka.  Grassland Biodiversity in Japan: Threats, Management and Conservation, chapter 9, page 22.  Taylor & Francis Group, 6 2017.
  * [132] Bae Hun Lee, Ji Yung Kim, Kyung Il Sung, and Byong Wan Kim.  Investigation on the actual state of grassland in Republic of Korea.  Journal of The Korean Society of Grassland and Forage Science, 39, 2019.
  * [133] FAO.  Fertilizer use by crop in Indonesia.  Technical Report TC/D/Y7063E/1/05.05/300, FAO, Viale delle Terme di Caracalla 00100 Rome, 2005.
  * [134] P. K. Ghosh, S. K. Mahanta, and S. N. Ram.  Nitrogen dynamics in grasslands.  In Yash P. Abrol, Tapan K. Adhya, Viney P. Aneja, Nandula Raghuram, Himanshu Pathak, Umesh Kulshrestha, Chhemendra Sharma, and Bijay Singh, editors, The Indian Nitrogen Assessment: Sources of Reactive Nitrogen, Environmental and Climate Effects, Management Options, and Policies, pages 187–205. Elsevier, 2017.
  * [135] Muhammad Irfan and Nighat Hasnain.  Nitrogen emissions from agriculture sector in Pakistan: context, pathways, impacts and future projections.  In Tariq Aziz, Abdul Wakeel, Muhammad Arif Watto, Muhammad Sanaullah, Muhammad Aamer Maqsood, and Aysha Kiran, editors, Nitrogen Assessment: Pakistan as a Case-Study, chapter 6, pages 99–125. Academic Presh, 2022.
  * [136] Ahmed S. Elrys, Mohamed K. Abdel-Fattah, Sajjad Raza, Zhujun Chen, and Jianbin Zhou.  Spatial trends in the nitrogen budget of the african agro-food system over the past five decades.  Environmental Research Letters, 14, 2019.
  * [137] FAO.  Fertilizer use by crop in Egypt.  Technical Report TC/D/Y5863E/1/01.05/300, FAO, Viale delle Terme di Caracalla, 00100 Rome, Italy, 2005.
  * [138] Hadi S. Esfahani.  Aggregate trends in four main agricultural regions in Egypt, 1964–1979.  International Journal of Middle East Studies, 20:135–164, 1988.
  * [139] M Bounejmate.  The Role of Legumes in the Farming Systems of the Mediterranean Areas: proceedings of a workshop on the role of legumes in the farming systems of the Mediterranean areas, UNDPI ICARDA, volume 38 of Developments in plant and soil science, chapter The role of legumes in the farming systems of Morocco, pages 85–93.  Kluwer Academic Publishers, 1 edition, 1990.
  * [140] FAO.  Utilisation des engrais par culture au maroc.  Technical Report TC/D/A710F/1/10.06/300, FAO, Viale delle Terme di Caracalla, 00100 Rome, Italy, 2006.
  * [141] Maria Niedertscheider, Simone Gingrich, and Karl Heinz Erb.  Changes in land use in South Africa between 1961 and 2006: An integrated socio-ecological analysis based on the human appropriation of net primary production framework.  Regional Environmental Change, 12:715–727, 2012.
  * [142] A. Smith and J. M.L.C. Rhind.  Eight decades of pasture plant improvement in South Africa.  Journal of the Grassland Society of Southern Africa, 1:25–28, 1984.
  * [143] Paswel P. Marenya and Christopher B. Barrett.  Soil quality and fertilizer use rates among smallholder farmers in western Kenya.  Agricultural Economics, 40(5):561–572, 2009.
  * [144] Kaushik Bora.  Rainfall shocks and fertilizer use: A district level study of India.  Environment and Development Economics, 27:556–577, 2022.
  * [145] Christian Levers, Van Butsic, Peter H. Verburg, Daniel Müller, and Tobias Kuemmerle.  Drivers of changes in agricultural intensity in Europe.  Land Use Policy, 58:380–393, 2016.
  * [146] Ian Harris, Timothy J. Osborn, Phil Jones, and David Lister.  Version 4 of the CRU TS monthly high-resolution gridded multivariate climate dataset.  Scientific Data, 7(1):190, 2020.
  * [147] Laura Poggio, Luis M. De Sousa, Niels H. Batjes, Gerard B.M. Heuvelink, Bas Kempen, Eloi Ribeiro, and David Rossiter.  SoilGrids 2.0: Producing soil information for the globe with quantified spatial uncertainty.  SOIL, 7(1):217–240, 2021.
  * [148] N. Middleton and D. S.G. Thomas.  World atlas of desertification, 1992.
  * [149] Xiaotang Ju, Baojing Gu, Yiyun Wu, and James N. Galloway.  Reducing China’s fertilizer use by increasing farm size.  Global Environmental Change, 41:26–32, 2016.
  * [150] FAOSTAT.  Structural data from agricultural censuses, 2024.  https://www.fao.org/faostat/en/##data.
  * [151] Sarah K. Lowder, Jakob Skoet, and Terri Raney.  The number, size, and distribution of farms, smallholder farms, and family farms worldwide.  World Development, 87:16–29, 2016.
  * [152] L. Jordan-Meille, G. H. Rubæk, P. A.I. Ehlert, V. Genot, G. Hofman, K. Goulding, J. Recknagel, G. Provolo, and P. Barraclough.  An overview of fertilizer-P recommendations in europe: Soil testing, calibration and fertilizer recommendations.  Soil Use and Management, 28:419–435, 2012.
  * [153] G. Feder, R. E. Just, and D. Zilberman.  Adoption of agricultural innovations in developing countries: a survey.  Economic Development & Cultural Change, 33:255–298, 1985.
  * [154] M. Hossain and V. P. Singh.  Fertilizer use in Asian agriculture: Implications for sustaining food security and the environment.  Nutrient Cycling in Agroecosystems, 57:155–169, 2000.
  * [155] FAOSTAT.  Fertilizers Archive, 2020.  https://www.fao.org/faostat/en/##data/RA.
  * [156] World Bank.  World Bank Commodity Price Data (Pink Sheet).  Technical report, World Bank Development Group, 2024.  https://www.worldbank.org/en/research/commodity-markets.
  * [157] John W. McArthur and Gordon C. McCord.  Fertilizing growth: Agricultural inputs and their effects in economic development.  Journal of Development Economics, 127:133–152, 2017.
  * [158] Robert J Hijmans, Roger Bivand, Karl Forner, Jeroen Ooms, Edzer Pebesma, and Michael D Sumner.  Package ‘terra’.  Maintainer: Vienna, Austria, 2022.
  * [159] D. J. Weiss, A. Nelson, C. A. Vargas-Ruiz, K. Gligorić, S. Bavadekar, E. Gabrilovich, A. Bertozzi-Villa, J. Rozier, H. S. Gibson, T. Shekel, C. Kamath, A. Lieber, K. Schulman, Y. Shao, V. Qarkaxhija, A. K. Nandi, S. H. Keddie, S. Rumisha, P. Amratia, R. Arambepola, E. G. Chestnutt, J. J. Millar, T. L. Symons, E. Cameron, K. E. Battle, S. Bhatt, and P. W. Gething.  Global maps of travel time to healthcare facilities.  Nature Medicine, 26:1835–1838, 2020.
  * [160] Ulrich Kleine-Kleffmann.  The discovery of the first potash mine and the development of the potash industry since 1861.  Journal of Plant Nutrition and Soil Science, 186:615–622, 12 2023.
  * [161] Lieven Clarisse, Martin Van Damme, Cathy Clerbaux, and Pierre François Coheur.  Tracking down global NH3 point sources with wind-adjusted superresolution.  Atmospheric Measurement Techniques, 12:5457–5473, 2019.
  * [162] FAOSTAT.  Producer prices, 2023.  https://www.fao.org/faostat/en/##data/PP.
  * [163] FAOSTAT.  Producer prices (old series), 2023.  https://www.fao.org/faostat/en/##data/PA.
  * [164] World Bank.  Official exchange rate (LCU per US $, period average), 2023.  https://data.worldbank.org/indicator/PA.NUS.FCRF.
  * [165] Jongrim Ha, M. Ayhan Kose, and Franziska Ohnsorge.  One-stop source: A global database of inflation.  Journal of International Money and Finance, 137, 2023.
  * [166] D. Tilman, J. Fargione, B. Wolff, C. D’Antonio, A. Dobson, R. Howarth, D. Schindler, W. H. Schlesinger, D. Simberloff, and D. Swackhamer.  Forecasting agriculturally driven global environmental change.  Science, 292(5515):281–284, 2001.
  * [167] Tao Xiang, Tariq H. Malik, and Klaus Nielsen.  The impact of population pressure on global fertiliser use intensity, 1970–2011: An analysis of policy-induced mediation.  Technological Forecasting and Social Change, 152:1–12, 2020.
  * [168] Nils Chr Stenseth and Atle Mysterud.  Climate, changing phenology, and other life history traits: Nonlinearity and match-mismatch to the environment.  Proceedings of the National Academy of Sciences of the United States of America, 99(21):13379–13381, 10 2002.
  * [169] Jeyan Thiyagalingam, Mallikarjun Shankar, Geoffrey Fox, and Tony Hey.  Scientific machine learning benchmarks.  Nature Reviews Physics 2022 4:6, 4(6):413–420, 4 2022.
  * [170] Massimo Bertolini, Davide Mezzogori, Mattia Neroni, and Francesco Zammori.  Machine Learning for industrial applications: A comprehensive literature review.  Expert Systems with Applications, 175:114820, 8 2021.
  * [171] Anne E. Thessen.  Adoption of Machine Learning Techniques in Ecology and Earth Science.  One Ecosystem 1: e8621, 1:e8621–, 2016.
  * [172] Sylvain Christin, Eric Hervet, and Nicolas Lecomte.  Applications for deep learning in ecology.  Methods in Ecology and Evolution, 10(10):1632–1644, 10 2019.
  * [173] Devdatta A Bondre and Santosh Mahagaonkar.  Prediction of crop yield and fertilizer recommendation using machine learning algorithms.  International Journal of Engineering Applied Sciences and Technology, 4:371–376, 2019.
  * [174] Liujun Xiao, Guocheng Wang, Enli Wang, Shengli Liu, Jinfeng Chang, Ping Zhang, Hangxin Zhou, Yuchen Wei, Haoyu Zhang, Yan Zhu, Zhou Shi, and Zhongkui Luo.  Spatiotemporal co-optimization of agricultural management practices towards climate-smart crop production.  Nature Food 2024 5:1, 5(1):59–71, 1 2024.
  * [175] Max Grell, Giandrin Barandun, Tarek Asfour, Michael Kasimatis, Alex Silva Pinto Collins, Jieni Wang, and Firat Güder.  Point-of-use sensors and machine learning enable low-cost determination of soil nitrogen.  Nature Food 2022 2:12, 2(12):981–989, 12 2021.
  * [176] Carla Pacheco, Mario Guimaraes, Eduardo Bezerra, Dacy Lobosco, Jorge Soares, Pedro Henrique González, Adalberto Andrade, Cristina Gomes De Souza, and Eduardo Ogasawara.  Exploring Data Preprocessing and Machine Learning Methods for Forecasting Worldwide Fertilizers Consumption.  Proceedings of the International Joint Conference on Neural Networks, 2022-July, 2022.
  * [177] Peng Xu, Geng Li, Yi Zheng, Jimmy C.H. Fung, Anping Chen, Zhenzhong Zeng, Huizhong Shen, Min Hu, Jiafu Mao, Yan Zheng, Xiaoqing Cui, Zhilin Guo, Yilin Chen, Lian Feng, Shaokun He, Xuguo Zhang, Alexis K.H. Lau, Shu Tao, and Benjamin Z. Houlton.  Fertilizer management for global ammonia emission reduction.  Nature 2024 626:8000, 626(8000):792–798, 1 2024.
  * [178] Jerome Friedman, Trevor Hastie, and Robert Tibshirani.  Additive logistic regression: a statistical view of boosting.  https://doi.org/10.1214/aos/1016218223, 28(2):337–407, 4 2000.
  * [179] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu.  LightGBM: A Highly Efficient Gradient Boosting Decision Tree.  Advances in Neural Information Processing Systems, 30, 2017.
  * [180] Damjan Krstajic, Ljubomir J. Buturovic, David E. Leahy, and Simon Thomas.  Cross-validation pitfalls when selecting and assessing regression and classification models.  Journal of Cheminformatics, 6(1):1–15, 3 2014.
  * [181] M. Stone.  Cross-Validatory Choice and Assessment of Statistical Predictions.  Journal of the Royal Statistical Society: Series B (Methodological), 36(2):111–133, 1 1974.
  * [182] Sudhir Varma and Richard Simon.  Bias in error estimation when using cross-validation for model selection.  BMC Bioinformatics, 7(1):1–8, 2 2006.
  * [183] Fernando Coello, Steven Mortier, Thomas Decorte, Iris Janssens, Tim Verdonck, Jordi Sardans, and Josep Peñuelas.  Fertilizer application rate maps per crop and year.  _figshare_ https://doi.org/10.6084/m9.figshare.25435432, 2024.
  * [184] Yongbo Liu, Xubin Pan, and Junsheng Li.  A 1961–2010 record of fertilizer use, pesticide application and cereal yields: a review.  Agronomy for Sustainable Development, 35:83–93, 2015.
  * [185] Stefano Longo and Richard York.  Agricultural exports and the environment: A cross-national study of fertilizer and pesticide consumption.  Rural Sociology, 73:82–104, 2008.
  * [186] Mingliang Liu and Hanqin Tian.  China’s land cover and land use change from 1700 to 2005: Estimations from high-resolution satellite data and historical archives.  Global Biogeochemical Cycles, 24(3), 2010.
  * [187] Hanqin Tian, Kamaljit Banger, Tao Bo, and Vinay K Dadhwal.  History of land use in india during 1880-2010: Large-scale land transformations reconstructed from satellite data and historical archives.  Global and Planetary Change, 121:78–88, 2014.
  * [188] Zhen Yu and Chaoqun Lu.  Historical cropland expansion and abandonment in the continental US during 1850 to 2016.  Global Ecology and Biogeography, 27(3):322–333, 2018.
  * [189] Iris Janssens, Fernando Coello, Steven Mortier, Thomas Decorte, Jordi Sardans, Josep Peñuelas, and Tim Verdonck.  Code for "machine learning-driven global crop-specific fertilization dataset from 1960–2020".  _figshare_ https://doi.org/10.6084/m9.figshare.25435594, 2024.
  * [190] United Nations.  Land area, 2024.  https://data.un.org/Data.aspx?d=FAO&f=itemCode:6601&c=2,4,5,6,7&s=countryName:asc,elementCode:asc,year:desc&v=1.
  * [191] FAOSTAT.  Definitions and standards. country group 2022, 2022.  https://data.apps.fao.org/catalog/iso/457196b9-1d93-410d-8ad3-a57aadf09a1a.
  * [192] World Bank.  Agricultural irrigated land (% of total agricultural land), 2023.  https://data.worldbank.org/indicator/AG.LND.IRIG.AG.ZS.
  * [193] World Bank.  Agricultural machinery, tractors, 2023.  https://data.worldbank.org/indicator/AG.AGR.TRAC.NO.
  * [194] FAOSTAT.  Land use, 2022.  https://www.fao.org/faostat/en/##data/RL.
  * [195] World Bank.  Goverment expenditure in education, total (% of GDP), 10 2023.  https://data.worldbank.org/indicator/SE.XPD.TOTL.GD.ZS.
  * [196] United Nations.  Per capita gdp at current prices - us dollars, 2023.  https://data.un.org/Data.aspx?d=SNAAMA&f=grID:101;currID:USD;pcFlag:1&c=2,3,5,6&s=_crEngNameOrderBy:asc,yr:desc&v=1.
  * [197] FAOSTAT.  Annual population, 2023.  https://www.fao.org/faostat/en/##data/OA.
  * [198] Guido Van Rossum and Fred L Drake.  Python 3 Reference Manual.  CreateSpace, Scotts Valley, CA, 2009.
  * [199] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant.  Array programming with NumPy.  Nature, 585(7825):357–362, 2020.
  * [200] Wes McKinney.  Data structures for statistical computing in python.  In Stéfan van der Walt and Jarrod Millman, editors, Proceedings of the 9th Python in Science Conference, pages 56 – 61, 2010.
  * [201] Sean Gillies et al.  Rasterio: geospatial raster i/o for Python programmers, 2013–.  "https://github.com/mapbox/rasterio".
  * [202] R Core Team.  R: A Language and Environment for Statistical Computing.  R Foundation for Statistical Computing, Vienna, Austria, 2021.  https://www.R-project.org/.
  * [203] Edzer Pebesma and Roger Bivand.  Spatial Data Science: With applications in R.  Chapman and Hall/CRC, 2023.  10.1201/9780429459016.
  * [204] Edzer Pebesma.  Simple Features for R: Standardized Support for Spatial Vector Data.  The R Journal, 10(1):439–446, 2018.
  * [205] David Pierce and Maintainer David Pierce.  Package ‘ncdf4’, 2019.  https://www.vps.fmvz.usp.br/CRAN/web/packages/ncdf4/ncdf4.pdf.
  * [206] Daniel Baston.  exactextractr: Fast Extraction from Raster Datasets using Polygons, 2020.  https://CRAN.R-project.org/package=exactextractr.
  * [207] Hadley Wickham and Jennifer Bryan.  readxl: Read Excel Files, 2023.  https://readxl.tidyverse.org,https://github.com/tidyverse/readxl.
  * [208] Hadley Wickham.  stringr: Simple, Consistent Wrappers for Common String Operations, 2023.  https://github.com/tidyverse/stringr.
  * [209] Hadley Wickham, Romain François, Lionel Henry, Kirill Müller, and Davis Vaughan.  dplyr: A Grammar of Data Manipulation, 2023.  https://github.com/tidyverse/dplyr.
  * [210] Hadley Wickham, Jim Hester, and Jennifer Bryan.  readr: Read Rectangular Text Data, 2024.  R package version 2.1.5, https://github.com/tidyverse/readr.
  * [211] Hadley Wickham.  ggplot2: Elegant Graphics for Data Analysis.  Springer-Verlag New York, 2016.  https://ggplot2.tidyverse.org.
  * [212] Hadley Wickham, Mara Averick, Jennifer Bryan, Winston Chang, Lucy D’Agostino McGowan, Romain François, Garrett Grolemund, Alex Hayes, Lionel Henry, Jim Hester, Max Kuhn, Thomas Lin Pedersen, Evan Miller, Stephan Milton Bache, Kirill Müller, Jeroen Ooms, David Robinson, Dana Paige Seidel, Vitalie Spinu, Kohske Takahashi, Davis Vaughan, Claus Wilke, Kara Woo, and Hiroaki Yutani.  Welcome to the tidyverse.  Journal of Open Source Software, 4(43):1686, 2019.
  * [213] Nils B. Weidmann, Doreen Kuse, and Kristian Skrede Gleditsch.  The geography of the international system: The cshapes dataset.  International Interactions, 36(1):86–106, 2010.