# The self-supervised channel-spatial attention-based transformer network for
automated, accurate prediction of crop nitrogen status from UAV imagery

Xin Zhang, Liangxiu Han*, Tam Sobeih, Lewis Lappin, Mark Lee, Andew Howard and
Aron Kisdi Xin Zhang, Liangxiu Han, Tam Sobeih are with the Department of
Computing, and Mathematics, Manchester Metropolitan University, Manchester
M15GD, U.K (e-mail<EMAIL_ADDRESS>l.han@mmu.ac.uk;T.Sobeih@mmu.ac.uk)Lewis Lappin and Aron Kisdi is the GMV,
Glasgow, Scotland G431QQ, U.K (lewis.lappin@gmvnsl.com;akisdi@gmvnsl.com)Mark
Lee is with the Department of Health Studies, Royal Holloway, University of
London, Egham, Surrey, TW200EX, U.K<EMAIL_ADDRESS>Symeon
Lechareas is with the Radiology, Whiston Hospital, St Helens and Knowsley
Teaching Hospitals NHS Trust, Warrington Road, Prescot L355DR, U.K
(e-mail:symeon.lechareas@rlbuht.nhs.uk)Ascanio Tridente is with the Intensive
Care Unit, Whiston Hospital, St Helens and Knowsley Teaching Hospitals NHS
Trust, Warrington Road, Prescot L355DR, U.K
(e-mail:Ascanio.Tridente@sthk.nhs.uk)Andew Howard is with the Bockhanger Farms
Ltd, Oaklands Farm, Ashford TN261ER U.K
(e-mail:bockhanger@btconnect.com)Corresponding author*: L. Han (e-mail:
l.han@mmu.ac.uk)Manuscript received XXX; revised XXX

###### Abstract

Nitrogen (N) fertiliser is routinely applied by farmers to increase crop
yields. At present, farmers often over-apply N fertilizer in some locations or
timepoints because they do not have high-resolution crop N status data. N-use
efficiency can be low, with the remaining N lost to the environment, resulting
in high production costs and environmental pollution. Accurate and timely
estimation of N status in crops is crucial to improving cropping systems’
economic and environmental sustainability. The conventional approaches based
on tissue analysis in the laboratory for estimating N status in plants are
time consuming and destructive. Recent advances in remote sensing and machine
learning have shown promise in addressing the aforementioned challenges in a
non-destructive way. We propose a novel deep learning framework: a channel-
spatial attention-based vision transformer (CSVT) for estimating crop N status
from large images collected from a UAV in a wheat field. Unlike the existing
works, the proposed CSVT introduces a Channel Attention Block (CAB) and a
Spatial Interaction Block (SIB), which allows capturing nonlinear
characteristics of spatial-wise and channel-wise features from UAV digital
aerial imagery, for accurate N status prediction in wheat crops. Moreover,
since acquiring labeled data is time consuming and costly, local-to-global
self-supervised learning is introduced to pre-train the CSVT with extensive
unlabelled data. The proposed CSVT has been compared with the state-of-the-art
models, tested and validated on both testing and independent datasets. The
proposed approach achieved high accuracy (0.96) with good generalizability and
reproducibility for wheat N status estimation.

###### Index Terms:

Crop Nitrogen status, Wheat, Deep Learning, Transformer, Self-Supervised
Learning, UAV

##  I Introduction

Nitrogen is an essential plant nutrient and is vital for plant growth and
development. The application of N fertilizers has revolutionized farming,
increasing crop yields and food production to meet the nutritional needs of
billions of people. It is estimated that global nitrogen fertilizer demand was
110 Million Tonnes (MT) in 2015 and is projected to be 120 MT in 2020, costing
farmers over $100 billion per year [1, 2]. Optimal application of N
fertilizers enhances soil fertility and increases crop yields. On the other
hand, excessive N inputs are costly for farmers but do not deliver any
additional yield benefits, instead of resulting in pollution of natural
ecosystems, increases in emissions of the potent greenhouse gas, nitrous
oxide, and reductions in biodiversity [3, 4]. Wheat crops invariably require
fertilizer to grow optimally and are one of the worldwide staple foods and the
world’s most commonly consumed cereal grain. About 35%-40% of the global
population depend on wheat as their major food crop. In 2020, the world wheat
land area is estimated at around 199 million ha, grown on more land than any
other food crop[5]. Accurate monitoring of the N status in wheat informs
farmer decisions on nitrogen fertilizer application rates and timing and is
therefore crucial for the economic and environmental sustainability of
cropping systems to support a secure food supply chain.

There are two main techniques to estimate crop N status: destructive and non-
destructive approaches. The destructive method in determining crop N is based
on laboratory chemical analysis of plant leaves[3, 6]. This method has the
best accuracy. However, it is costly, labor-intensive, and time-consuming,
meaning that it is rarely possible to take multiple measurements throughout a
growing season. In contrast, the non-destructive methods perform estimation of
the crop’s N status without causing damage to plants in a timely fashion. Most
commonly used non-destructive methods are based on optical sensing, in which
optical sensors (e.g., RGB digital camera, multispectral/hyperspectral images)
are instrumented on the ground and aerial vehicles. Recently, ground and
aerial-based RGB digital imaging, as a non-destructive method, has been
successfully used to provide information on nitrogen status in wheat and maize
under field conditions[7, 8] and the prediction of the crop growth,
quantification of water deficiency and N nutrition status [9]. An RGB image
provides three channels of information, including red, green, and blue
wavelengths, which are used to estimate N status. The leaf color chart (LCC)
is an early stage and commonly used method to determine the N status of crops
by using the color information[10]. The LCC has five categories, ranging in
color from yellow to green. It determines the nitrogen content of crops based
on the degree of green color of rice leaves (Fig. 1).

Figure 1: Leaf Color Chart determines the Nitrogen status of paddy rice [10].

With advances in computer vision and machine learning, more computer-aided
methods are used for N status estimation based on digital images. It can be
broadly divided into two categories: linear and nonlinear analysis methods.
The linear analysis, such as linear nonparametric regression[3] principal
component analysis (PCA)[11], mainly generates an indicator (index) from the
channel information to determine the N status of the crop. These methods are
typically based on empirical and experimental data and effectiveness on the
specific crop[12, 13]. They are easy to calculate and interpret. However, the
N content of crops is considered a nonlinear problem, which is affected by
various influential factors such as ambient lighting conditions and variations
amongst the types of crops. Machine learning has shown the effectiveness of
solving nonlinear problems from multiple sources [14]. Several machine
learning-based methods such as random forest (RF) [15], support vector machine
(SVM)[16], and Artificial Neural Networks (ANNs) [17] have been applied to
estimate crop stress or N status with satisfactory results[18, 19]. However,
these aforementioned methods typically focus on color and channel information.
Over the past few years, out of the various ML techniques at our disposal,
deep learning methods, especially the convolutional neural networks (CNN),
have dominated computer vision tasks with the emergence of graphic processing
units (GPU) [20]. Unlike traditionally hand-crafted feature extraction
methods, CNN automatically extracts spatial, and channel features from images
in a local receptive field [21]. In the work [22], the authors proposed a CNN
classification model for N status prediction in coffee plants. Sethy et al.
[23] used six leading deep learning architectures to predict nitrogen
deficiency on rice crops. In the work [24], the CNN has been used to classify
and predict early N deficiencies during the growth of the tomato plant.
Recently, Vision Transformer (ViT) [25] has attracted increasing attention in
computer vision tasks due to its capability of capturing spatial interactions
as well as introducing less inductive bias, compared to widely-used
convolutional neural networks (CNNs). It has been considered to be a solid
alternative for CNNs. The essence of ViT is to use a self-attention scheme
[26] to capture long-range dependencies or global information, focusing on
spatial information. However, the direct application of ViT to N status
estimation is impractical. This is because it does not focus on channel
information, which is very important for crop N status estimation. In
addition, ViT has quadratic computational complexity since the self-attention
operation in ViT increases exponentially with the input size, which does not
scale well with the high-resolution imagery. Moreover, ViT is data-hungry in
nature, which requires large datasets (labeled data) for training models to
achieve good performance. Our study has proposed a new channel-spatial
attention-based vision transformer network for automatic and accurate crop N
status estimation. To the best of our knowledge, this is the first work that
explores transformer network combining with Self-Supervised Learning for
accurate crop N status prediction.Our contributions include:

  1. 1.

Novel channel-spatial attention-based vision transformer is proposed, in which
both the channel (color) and spatial (height, texture, shape) information are
considered. A Channel Attention Block (CAB) is proposed to replace the self-
attention in ViT. The CBA operates across the transposed input that makes the
attention module attend over channels. Meanwhile, a Spatial Interact (SIB) is
introduced after CBA to learn corresponding spatial information.

  2. 2.

Local-to-global self-supervised learning (SSL) is proposed to pretrain the
model on the unlabeled images generated from the UAV imagery to resolve the
data-hungry paradigm in transformer model training.

  3. 3.

A linear computational complexity is achieved using the cross-covariance
matrix instead of the original gram matrix operation in the attention block.
It changes the complexity of the transformer layer from quadratic to linear,
which can efficiently process images with large images.

The rest of this paper is organized as follows: Section 2 presents the related
work; Section 3 details the proposed method; In Section 4, the experimental
evaluation is described; Section 5 concludes the proposed work and highlights
the future work.

##  II Related work

###  II-A Digital Imaging for Crop N Estimation

Ground and aerial based RGB digital cameras are among the most common non-
destructive data acquisition devices used in crop nutrition monitoring [27,
28]. The cost of capturing red-green-blue (RGB) images with a digital camera
is much lower than other spectroscopic instruments. The captured images can be
at a scale ranging from leaf to canopy. RGB images captured by a charge
coupled device (CCD) or complementary metal oxide semiconductor (CMOS) sensors
contain not only color information of the subject (Red, green, and blue
broadband) but also spatial information (eg. the shape, size of the leaf, and
the crop’s height). The leaf color chart (LCC) is the easiest method to
evaluate crop N by identifying the color (Fig. 1). It relies on a manual
visual interpretation comparison between leaf color and a color chart to
assess the N status of the crop. This method has been widely used for many
crops such as maize [29], rice [30] and sugarcane [31].

With recent advances in computer vision and machine learning, more computer-
aided methods are being used for nitrogen status estimation based on digital
images. These can be broadly classified into two categories: Linear and
Nonlinear Analysis. The linear analysis generates a linear indicator index
from the channel information to determine the N status of the crop. The
greenness indicator is considered to be the most related indicator to the N
status of the crop. To measure the N status of barley leaves, Pagola et al.
devised a greenness index based on RGB components of digital colour pictures
[13]. Jia et al. found that the red–color intensity, green–color intensity,
and normalized redness intensity had significant inverse linear correlations
with the N status of rice canopy [32]. The color spaces such as HSL (for hue,
saturation, lightness), HSV (for hue, saturation, value), and LAB (Lightness,
Red/Green Value, and Blue/Yellow Value) were used to enhance the channel
information. The work [33] evaluated the relationship between RGB, HSV and LAB
with nitrogen values in maize. The Green, Blue/Yellow Values and lightness
provide good accuracy. Wiwart et al. used the color space HSL and LAB to
assess macronutrients (N, P, K, and Mg) in plants using the Euclidean distance
[34]. Gloria proved that the improved hue, luminance and saturation (IHLS)
color space is a fast and economical method to estimate Nitrogen on tomato
seedlings [35]. However, the N content of crops is a typical nonlinear
problem. The linear-based methods lack generalizability because they are
affected by ambient lighting conditions and variations amongst the types of
crops. Instead, we often need to use nonlinear analysis. During the last
decade, machine learning (ML) approaches have become widely utilised in
complicated and data-intensive domains such as health, biology, and precision
agriculture due to their capacity to identify information hidden in data.
Several machine learning-based methods have been applied to estimate crop
nutrition or N status with satisfactory results. In the research [15], simple
nonlinear regression (SNR), backpropagation neural network (BPNN), and random
forest (RF) regression were used to determine the rice N nutrition status with
RGB images. In work [16] and [17], the author used support vector machine
(SVM), multiple linear regression (SMLR), and Artificial Neural Networks
(ANNs) to estimate rice nitrogen nutrition index with UAV RGB images. The work
[36] used ANNs and RF to predict the biotic stress of winter wheat. Yao et al.
[37] applied ANN and SVM regression methods to estimate the leaf nitrogen in
winter wheat. The results showed that the SVM method was more robust in coping
with potential confounding factors for most varieties, ecological sites, and
growth stages. The RF, SVM, and ANN were employed in the study [28] to
construct quantitative correlations between the nitrogen nutrition index (NNI)
of pakchoi and image-based phenotyping variables. In both the seedling and
harvest stages, the RF based model outperformed the others. A review research
[18] indicated that machine learning approaches will result in more cost-
effective and comprehensive solutions for a better crop status assessment.
However, these aforementioned works typically focus on color and channel
information. The digital image contains not only channel information but also
spatial information of these image pixels. The spatial information such as the
shape, size of the leaf, and the crop’s height are considered essential
attributes for characterizing the N status of the crop. Currently, only a
small amount of hand-crafted and straightforward spatial information, such as
canopy cover, are used in N status estimation [38, 39, 40, 41]. Over the past
few years, the convolutional neural networks (CNN), as the latest machine
learning techniques, have dominated computer vision tasks with the emergence
of graphic processing units (GPU) [20]. Unlike standard hand-crafted features,
CNN, as a filter bank, can automatically extract spatial and channel features
from a local receptive field in images [21]. Azimi [42] proposed a 23-layered
CNN to measure the crop stress level in plants due to nitrogen deficiency and
found that CNN outperforms most machine learning methods in fast & accurate
identification of stress in plants. Lee [43] used CNN to extract spatial
features of the leaf to perform plant classification. Their results showed the
strength of detecting spatial features using CNN as compared to hand-crafted
features. Meanwhile, in this work, it was found that traditional CNN can only
extract local spatial information [44] while failing to capture long-range
global spatial information and proposed hybrid global-local feature extraction
models.

###  II-B Vision Transformer

With the proposed Attention is All You Need [26], the self-attention mechanism
has become the most famous structure on natural language processing (NLP). The
self-attention mechanism is designed to focus on spatial location (where),
which provides the ability to capture long-range dependencies/global
information. Transformer, in essence, use the concept of self-attention have
rapidly been increasing in popularity and become a major focus of modern
architecture [25]. Vision Transformer (ViT) [45] is the first major
demonstration of a pure transformer backbone applied to computer vision tasks
that do not contain convolution operations. Unlike standard CNN models, ViT
needs to divide images into patches, flatten them into a sequence, and feed
the sequence to the transformer encoder.

Figure 2: The structure of Vision Transformer [45].

There are four main parts in the transformer encoder as shown in Fig. 2:
Multi-Head Self Attention Layer (MSP), Multi-Layer Perceptrons (MLP), Layer
Norm, and Residual connections introduced in CNN evolution. The MSP is the
core of the transformer. It allows the model to integrate information globally
across the entire image. It is used to concatenate the multiple attention
outputs linearly to expected dimensions. The multiple attention heads help
learn local and global dependencies in the image. MLP contains two fully
connected layers with Gaussian Error Linear Unit (GELU). It has been proven to
be an essential part of the transformer that stops and drastically slows down
rank collapse in model training [46]. Layer Norm is the normalization method
in the NLP area instead of Batchnorm in vision tasks. It is applied before
every block as it does not introduce any new dependencies between the training
images. This helps to improve training time and generalization performance.
Residual connections are applied after every block as they allow the gradients
to flow through the network directly without passing through nonlinear
activations. However, the ViT requires large-scale training datasets (i.e.,
JFT-300M) to perform well. DeiT [47] introduces a distillation training
strategy that allows ViT to effectively train on a smaller image dataset. The
results of ViT on image classification are encouraging. However, its
architecture is still unsuitable for large images due to its low-resolution
feature maps and the quadratic increase in complexity with image size.

###  II-C Self-supervised learning (SSL)

Acquiring extensive, labeled data for training ViT is challenging. Self-
supervised learning provides an effective way to enable learning from large
amounts of unlabeled data. SSL can be broadly divided into Generalized
Adversarial Networks (GANs) and Contrastive learning [48]. GANs are
unsupervised learning tasks that involve automatically discovering and
learning the regularities or patterns in input data in such a way that the
model can be used to generate new samples [49]. Unlike generative models,
Contrastive learning [50] is used to determine which representations attract
comparable samples and which ones repel them. The representations from
contrastive self-supervised pretraining can be used in specific supervised
downstream vision tasks. Generally, contrastive SSL usually consists of three
parts: 1) image augmentation, 2) feature extraction/encoder, and 3)
contrastive loss to quantify the similarity between representations. Image
augmentation creates positive pairs by generating different augmented views of
the same image, such as color augmentation, image rotation/cropping and other
geometric transformations. Then a CNN network is used to encode the augmented
images as vector representations. The Siamese Neural Network [51] is the most
widely used neural network architecture to find the similarity between the
representations in contrastive learning. It contains two or more identical
subnetworks. Each sub-network has the same configuration with the same
parameters and weights. Parameter updating is mirrored across both sub-
networks. In general, training in Siamese Neural Network is compared against a
positive pair and a negative pair. The negative vector pair is learning in the
network, while the positive pair acts in a regularization role. The negative
pairs rely on different images, which are hard to define. An evolutional work
(BYOL) retains the Siamese architectures but eliminates the requirement of
negative samples [52]. BYOL proposed a momentum training that rolling weight
updates as a way to give contrastive signals to the training. Recent methods
such as SwAV [53], MoCo [54], and SimCLR [55] with modified approaches have
produced results comparable to the state-of-the-art supervised method on the
ImageNet public dataset. However, most SSL methods are mainly based on
standard convolutional networks. The SSL for vision transformer models are
new,. In this work, inspired by BYOL, we proposed a local to global SSL for
the vision transformer network.

##  III Method and Materials

Figure 3: The structure of the proposed channel-spatial attention vision
transformer (CSVT)

###  III-A Channel-spatial attention based vision transformer (CSVT)

Inspired by the original vision transformer [45], we have proposed a new
vision transformer called a channel-spatial attention vision transformer
(CSVT) to accurately estimate the nitrogen status of wheat, capable of
capturing both spatial and channel features from large UAV-based digital
aerial imagery. The proposed conceptual architecture is shown in Fig. 3. The
input images are firstly split into patches and flattened into vectors using a
linear projection operation. Then, each vector is regarded as a sequence and
fed into the transformer encoders. A class token is added to represent an
entire image that can be used for classification. It is actually a vector that
is learned during gradient descent. In this work, we add the class token in
the last encoder block, which only lets encoders’ attention mechanism perform
between images. Each encoder consists of two core components, including 1)
Channel-Spatial attention Block, which consists of channel-based and spatial
interaction blocks, and 2) Multi-Layer Perceptron (MLP). The design rationale
is three-fold:

  1. 1.

As shown in previous research [18], channel information plays a vital role in
determining nitrogen status at leaf and canopy scales. In this work, the
channel-based attention block is proposed to replace the self-attention
module. We transpose the input, enabling the module attending over channels to
be able to learn channel-wise features such as color information.

  2. 2.

To learn spatial information, a spatial interaction block is introduced after
the channel-based attention block.

  3. 3.

To address the quadratic computing complexity of the original ViT, the
covariance matrix is used to replace the gram matrix, which can help reduce
computational complexity from the quadratic complexity ($O(n2)$) to linear
complexity ($O(n)$). n represents a number of input patches.

####  III-A1 Channel-Spatial attention Block

In this work, to address the channel and spatial information in the
transformer encoder, we proposed a Channel-Spatial attention Block to replace
the Multi-Head Self Attention Layer. The Channel-Spatial attention Block
consists of two main parts: Channel Based attention (CBA) and Spatial
Interaction block (SIB).

##### Channel Based attention (CBA)

The CBA module adopts an attention mechanism with Query, Key and Value. In
this work, to decrease the computational complexity of the scaled dot-product
attention used by Transformer. A cross-covariance based self-attention
function is proposed. Given the packed matrix representations of queries $\
Q\in R^{n\times d}$, keys $K\in R^{n\times d}$ , and values $V\in R^{n\times
d}$, the cross-covariance attention is given by:

$\mathrm{CBA\
}\left(Q,K,V\right)=A_{XC}\left(K,Q\right)V=SoftMax{\left(\frac{{\hat{K}}^{\top}\hat{Q}}{\tau}\right)}V$
(1)

Where $n$ denotes the number of patches, $d$ denotes the dimensions of keys
(or queries) and values, which means the number of pixels in each patch.
$A_{XC}\left(K,Q\right)$ denotes as attention matrix, $SoftMax$ is applied to
a row-wise manner. Where the attention weights $A_{XC}$ are computed based on
the cross-covariance matrix. ${\hat{K}}^{\top}\hat{Q}$ is the cross-covariance
matrix size of $d\times d$. In [56], the author found that controlling the
data range in attention strongly enhances the stability of training, here
$\hat{Q}$ and $\hat{K}$ denoted the normalized matrices Q and K. The inner
products are scaled before the $Softmax$ by the $\tau$ which is a learnable
parameter that allows for a sharper or more uniform distribution of attention
weights. The new $A_{XC}\left(K,Q\right)$ operates along the dimensions of
input vector d, which denoted the channel information of the image, rather
than along the amount of the patches n. Each output embedding is a convex
combination of its corresponding embedding in V’s d features. The
computational cost is $O(d^{2}n)$ which has a linear computational computing
complexity of input size. Then, for building a deeper model, a residual
connection is employed around each module, followed by Layer Normalization
[57]. For instance, each encoder block ($H^{\prime}$) can be written as:

$H^{\prime}=\mathrm{\ LayerNorm\ }\left(\mathrm{\ CBA}\left(x\right)+x\right)$
(2)

##### Spatial Interaction block (SIB)

The SIB is not concerned with the spatial information between patches. After
each CBA block, a SIB is introduced to enable explicit communication between
patches. The SIB is build by two depth-wise 3×3 convolutional layers with
Batch Normalization and ReLU non-linearity in between [58]. The output of the
SIB can be written as:

$\begin{gathered}S=\operatorname{LayerNorm}\left(H^{\prime}+\right.\\\
\left.\operatorname{Conv}2\left(\text{ BatchNorm
}\left(\operatorname{ReLU}\left(\operatorname{Conv}1\left(\operatorname{Conv}1\left(H^{\prime}\right)\right)\right)\right)\right)\right)\end{gathered}$
(3)

####  III-A2 Multilayer Perceptron (MLP)

A multilayer perceptron is a particular case of a feedforward neural network
where every layer is a fully connected layer. As is common in transformer
models, an MLP is added at the end of each encoder block, which contains two
fully connected layers. While the CBA block restricts feature interaction
within groups and the SIB can not allow for feature interaction, the MLP
allows for interaction across all features. The output of MLP (F) can be
written as:

$F=\mathrm{LayerNorm\ }(S+Fc2(ReLU(Fc1(S)))$ (4)

###  III-B Local-to-Global Self-supervised learning

Figure 4: The flowchart of the Self-supervised learning

In this work, to solve the data-hungry issue of transformer training, self-
supervised contrastive learning is used to pretrain the proposed CSVT with
unlabeled images. Vision transformer is good at capturing long-range global
spatial information. However, it fails to capture the local spatial
information of small patches. To address this problem, we have proposed a
local-to-global SSL method in which both local and global augmentations are
performed to provide both global and local views of the input. First, random
color augmentation, including color jittering, Gaussian blur, and
solarization, are applied to all the images. Then, we perform global and local
augmentation on the same image to obtain both global and local views. The
global views use, with an image size of $224^{2}$. We assume that it contains
the global context of the image. The small crops are called local views that
use an image size of $96^{2}$. It covers less than 50% of the global views. We
assume that it contains the local context. Then two views are fed into the SSL
network. All local views are passed through the student, while only the
teacher passes the global views. It encourages the student network to
interpolate context from a small crop and the teacher network to interpolate
context from a bigger image. Fig. 4 shows the SSL framework. The SSL network
learns through a process called ‘self-distillation’ proposed by the paper ‘Be
Your Own Teacher’ [59]. There is a teacher and student network both using the
proposed model CSVT. They have the same configuration with the same parameters
and weights. The teacher is a momentum teacher, which means that the students’
weights ($\theta_{s}$) are an exponentially moving average (EMA). The update
rule for the teacher’s weights ($\theta_{t}$) is:

$\theta_{t}\leftarrow\lambda\theta_{t}+\left(1-\lambda\right)\theta_{s}$ (5)

With $\lambda$ following a cosine schedule from 0.996 to 1 during training.
The cross-entropy loss is used to make the two distributions the same, just as
in knowledge distillation.

$Loss=-P_{Teacher}log{P_{Student}}$ (6)

Centering [53] is used to prevent the model from predicting a uniform
distribution along all dimensions or dominated by one dimension regardless of
the entry. The teacher’s raw activations ($A_{t}(x)$) have their exponentially
moving average (c) subtracted from them. The center c is updated with an
exponential moving average.

$A_{t}(x)\leftarrow A_{t}(x)\ -c$ (7)

###  III-C Dataset description

In this work, we have collected the data at a controlled wheat field located
near Ashford, south-eastern UK (51.156N, 0.876E) (Fig. 5). We adopted a 4x4
factorial design in the controlled field experiment, with four randomly
allocated N treatments were replicated within four blocks, totaling 16 plots
of 16 $m^{2}$ (4m x 4m). Plots were established prior to the first fertilizer
application. The four treatments were low (80 $kgNha^{-}1yr^{-}1$), medium
(160 $kgNha^{-}1yr^{-}1$), and high (240 $kgNha^{-}1yr^{-}1$) fertilizer
rates, with unfertilized control. These values were chosen because they were
representative of application rates commonly used by arable farmers. Five
applications were used to add N fertilizer to the plots every three weeks
between February – June 2021.

Figure 5: a) Experimental layout, with plots randomly allocated into four
treatments, split into four blocks. Treatments were high (green), medium
(blue) and low fertilizer rates (red), and an unfertilized control (yellow).
b) Images collected from different plots with different treatments. TABLE I:
Monitoring schedule and Data collection summary

Growing Stage | Date | Field image collection | Drone
---|---|---|---
| 21-Mar | 160 | $\checkmark$
| 02-Apr | 163 | $\checkmark$
Tillering & Stem Extension | 09-Apr | 177 | $\checkmark$
| 06-May | 166 | $\checkmark$
| 14-May | 177 | $\checkmark$
Heading & Flowering | 24-May | 175 | $\checkmark$
| 07-Jun | 177 | $\checkmark$
| 22-Jun | 177 | $\checkmark$
Ripening & Maturity | 28-Jun | 175 | $\checkmark$

Two types of digital camera images are collected at the canopy scale, via the
near-ground sensing and UAV-based remote sensing, from these plots during all
the wheat growing stages, including Tillering & Stem Extension, Heading &
Flowering and Ripening & Maturity. A Sony Xperia 5 with a 12-megapixel Exmor
RS CMOS is used to collect images near-ground. The focal length is 26mm. A DJI
MAVIC pro with 12.35-megapixel CMOS is selected to capture images in the air.
The focal length is 26mm. The detailed monitoring schedule is shown in Table.
I. Fig. 6 shows the sample images. The UAV flight heights were from 10m to
30m. A total of 1449 field images are used in this work. The image size is
4032 x 3024.

Figure 6: The collected images. The left one is collected near the ground. The
right one is collected from the drone.

##  IV Result and discussion

###  IV-A Experimental configuration

This work aims to develop a new method for accurately estimating N status in
crops (i.e., wheat in this case) based on crop images at the canopy scale. As
described in section 4.1, since there are four types of N treatments,
including High, Medium, Low and Control, our task is to classify the images
into these four categories automatically. Fig. 5 b) shows images collected
from different plots with different treatments. We randomly cropped them into
224 x 224 patches for the drone images and generated the unlabeled images for
SSL. In this work, 4,800,000 images are generated. The detailed configuration
of the proposed CSVT is shown in Table. II. It has 12 encoder layers. For each
CBA, it runs through the cross-covariance attention four times in parallel.
The dimension of keys is 384.

TABLE II: The transformer model configuration

Model | Patch size | Layer | Dimension | Param (M)
---|---|---|---|---
Proposed CSVT | 8 | 12 | 384 | 25.87

In this work, the SSL uses the AdamW optimizer (Loshchilov and Hutter, 2017)
and a batch size of 64, distributed over 3 GPUs (GeForce RTX 2080 Ti). The
learning rate is linearly ramped up during the first ten epochs as 0.0005.
After this warmup, we decay the learning rate with a cosine schedule. The
weight decay also follows a cosine schedule from 0.04 to 0.4. Data
augmentations, including color jittering, Gaussian blur, and solarization, are
used in this work. A multi-crop [53] with a bicubic interpolation is used to
generate the global and local view of the image. The images with crop ratio
greater than 50% are defined as global view images. The images covering a
small area (crop ratio less than 50%) are defined as local view images. For
the CSVT classification model training, AdamW optimizer is used for 100 epochs
using a cosine decay learning rate scheduler and 20 epochs of linear warm-up.
A batch size of 64, an initial learning rate of 0.001, and a weight decay of
0.05 are used. We decay the learning rate with a cosine schedule. The
augmentation and regularization strategies used in this training include,
random-size cropping, random horizontal flipping, data mix-up and label-
smoothing regularization. We used five-fold cross-validation in this study.
The dataset is divided into five groups at random, with four groups (80% of
dataset) utilised for training and the remaining groups used for testing each
time. The average of the accuracies on the testing set over all folds is used
to evaluate the classification performance. All the compared models are
implemented by open-sourced project Timm [60], and ImageNet pre-trained
weights are used to fine-tune and boost the model training. For each model, we
train 50 epochs by using AdamW optimizer and a cosine decay learning rate
scheduler. A batch size of 64, an initial learning rate of 0.0001, and a
weight decay of 0.05 are used.

###  IV-B Evaluation metrics

Accuracy, Precision, Recall, F1 score and the Confusion matrix are selected
for the accuracy assessment to evaluate model performance. Accuracy is the
most intuitive performance measure, and it is simply a ratio of correctly
predicted observations to the total observations. The Precision measures the
fraction of true positive detections, and the Recall measures the fraction of
correctly identified positives. The F1-score considers both the Precision and
the Recall to compute the score. The study established the classification
matrices which Precision, Recall and F1-score calculated with the following
equations:

$\mathrm{Accuracy\ }=\frac{tp+tn}{tp+tn+fp+fn}$ (8)

$Precision=\frac{tp}{tp+fp}$ (9)

$Recall=\frac{tp}{tp+fn}$ (10)

${F1}_{score}=\frac{Recall\times Precision}{Recall+Precision}\times 2$ (11)

Where True Positives (TP) are the correctly predicted positive values. True
Negatives (TN) are the correctly predicted negative values. False positives
and false negatives occur when the actual class contradicts the predicted
class. False Positives (FP) means the predicted class is yes when the actual
class is no. False Negatives (FN) means predicted class in no when actual
class is yes.

###  IV-C Experimental evaluation

To evaluate the performance of the proposed CSVT, we have conducted three
types of experiments: 1) Performance evaluation of CSVT for automated crop N
prediction. 2) Ablation Study. 3) Evaluation of the generalizability of the
proposed model using independent datasets.

####  IV-C1 Performance evaluation of CSVT for automated crop N prediction

In this case, we first evaluate the performance of the proposed CSVT for
automated crop N prediction with the input size of 224×224. Then we evaluate
the performance of the model on a larger input of 384×384. For different model
comparisons, we have selected the four most widely used and state-of-the-art
CNNs. Two commonly used CNN-based classification models, ResNet and
EfficientNet and their state-of-the-art versions RegNet [61] and
EfficientNetV2 [62], are selected for performance comparison. The general
network structure of CNN models can be summarized as Fig. 5 a), which consists
of a stem, followed by the body part, and head classifier (average pooling
followed by a fully connected layer) that predicts output classes. The body
part is composed of 4 stages that operate at progressively reduced resolution,
and each stage consists of a sequence of identical blocks. The identical
blocks of each CNN are shown in Fig. 7 b,c,d,e.

Figure 7: General network structure and detailed block structure for CNN
models.

For direct comparisons and to isolate benefits resulting from network design,
the configuration of the models is based on the trained parameters, in which
20 million parameters are selected as the baseline in this work. Based on
this, the ResNet with 50 layers, EfficientNet_B5, RegNetY -4.0G,
EfficientNetv2_small are selected for comparison.

####  IV-C2 Ablation Study

In this case, three ablation studies are set to evaluate: 1) the performance
of the proposed CSVT with and without SSL; 2) the impact of the channel-
spatial attention block; 3) The computational cost with and without the cross-
covariance attention.

##### The performance of the proposed CSVT with and without SSL

In this work, a local-to-global SSL method is proposed to pretrain the model
on the unlabelled image generated from the drone. We evaluate the performance
of the proposed CSVT trained from SSL and trained from scratch to show the
impact of the SSL on model generalization. The performance of the trained
model from scratch and fine-tuning from the SSL weights are reported.

##### The impact of the channel-spatial attention block

This work proposes the channel-spatial attention block to replace the self-
attention in ViT, making the attention module attend over the channel and
spatial information. In this case, we evaluate the effect of the proposed
module on the vision transformer. The original vision transformer is used for
comparison. The ViT-small with a similar number of parameters is selected in
this work. The detailed configuration is shown in Table. II. In this work, to
ensure the ViT model generalizes and converges correctly. The self-learning
pretrained on unlabelled drone images is also used on the ViT model train. The
training strategy is the same as the proposed model in case 1.

##### The computational cost with and without the cross-covariance attention

In this work, to solve the computational complexity quadratic to the image
size on ViT. The cross-covariance matrix is used to replace gram matrix
operation in the attention module. In this case, we calculate and report the
GPU usage of three models with increased input image size. We start from the
commonly used size of 224×224 and then gradually and linearly increase the
size of the input image (336, 448, 560 ……). The most widely used CNN model,
Resnet with 50 layers and the original ViT, are selected for comparison.

####  IV-C3 Evaluation of the generalizability of the proposed model using
independent drone datasets

In this case, to evaluate the generalizability of the proposed CSVT model. We
have evaluated the trained model on independent datasets. The images are
captured from the drone in every growing stage, including Tillering & Stem
Extension, Heading & Flowering and Ripening & Maturity. The performance of the
trained model for each growing stage is reported. We also report the
performance of the Resnet with 50 layers CNN model for comparison. Meanwhile,
we have visualized the attention map from the last block of the model to
evaluate the decision-making area of each model on independent datasets.

###  IV-D Experiment result

####  IV-D1 Performance evaluation of CSVT for automated crop N prediction

In this case, we report the performance of the proposed CSVT for automated
crop N prediction (Table. III). With the input image size of 224×224, the
accuracy of the proposed model reaches 0.962. With the input image size of
384×384, the accuracy of the proposed model reaches 0.965, which is slightly
higher.

TABLE III: The proposed model performance comparison with different input
sizes under four N treatments

| Image size 224×224
---|---
Types of N treatments | Control | Low | Medium | High
Precision | 0.949 | 0.992 | 0.985 | 0.923
Recall | 0.936 | 0.992 | 0.98 | 0.941
F1-score | 0.943 | 0.992 | 0.982 | 0.932
Accuracy | 0.962
| Image size 384×384
Precision | 0.973 | 0.991 | 0.982 | 0.917
Recall | 0.924 | 0.989 | 0.981 | 0.964
F1-score | 0.948 | 0.99 | 0.982 | 0.94
Accuracy | 0.965

TABLE IV: Model performance comparison

Models | Param (M) | GFLOPs (GMac) | Accuracy (%)
---|---|---|---
ResNet_50 | 23.52 | 4.12 | 0.945
EfficientNet_B5 | 28.35 | 2.4 | 0.95
RegNetY-4.0G | 20.6 | 4.1 | 0.951
EfficientNetv2_small | 20.18 | 2.87 | 0.949
Our proposed CVT | 25.87 | 4.71 | 0.962

Meanwhile, we have compared our proposed model with the four most widely used
CNN models. The results are shown in Table. IV. With the lowest flops and the
most parameters, EfficientNet reaches an accuracy of 0.95. The proposed model
has intermediate parameters and the highest flops. The performance outperforms
other approaches.

####  IV-D2 Ablation Study

##### The performance of the proposed CSVT with and without SSL

In this case, we firstly pretrain the proposed model with unlabeled images.
The loss of SSL is shown in Fig. 8, which shows that the model converges well.
The classification performance of the proposed model without SSL is reported
in Table. V and Fig. 9. Without the pretrained weights, the proposed vision
transformer model cannot converge correctly. The Accuracy of the proposed
model is only 0.836. As shown in Fig. 9, the model without SSL performs well
on N status, Control. However, it performs unsatisfactorily on, High, Low and
Medium. The model trained with SSL weights performs well on all N statuses.

Figure 8: The loss trend on self-supervised learning TABLE V: Model
performance without self-supersized learning for the four N fertilizer
treatments

Model | CVT w/o SSL
---|---
Types of N treatments | Control | Low | Medium | High
Precision | 0.937 | 0.898 | 0.738 | 0.784
Recall | 0.939 | 0.844 | 0.774 | 0.789
F1-score | 0.938 | 0.87 | 0.756 | 0.787
Accuracy | 0.836

Figure 9: The Confusion matrix of the classification results achieved by the
proposed CSVT (a) without and (b) with self-supervised learning

##### The impact of the channel-spatial attention block

In this case, to evaluate the effect of the proposed channel-spatial attention
block on vision transformer, we report the model performance of the proposed
model and the original ViT. The results are shown in Table. VI. It has
demonstrated that our proposed model with channel-spatial attention block has
better classification performance for crop N status estimation than that of
ViT.

TABLE VI: Model performance comparison with the proposed model with CBA and
the original ViT ( four types of N treatment: Control, Low, Medium and High)

Model | ViT
---|---
Types of N treatments | Control | Low | Medium | High
Precision | 0.925 | 0.956 | 0.931 | 0.969
Recall | 0.994 | 0.975 | 0.944 | 0.865
F1-score | 0.959 | 0.965 | 0.938 | 0.914
Accuracy | 0.944
Model | CSVT
Precision | 0.949 | 0.992 | 0.985 | 0.923
Recall | 0.936 | 0.992 | 0.98 | 0.941
F1-score | 0.943 | 0.992 | 0.982 | 0.932
Accuracy | 0.962

##### The computational cost with and without the cross-covariance attention

In this case, we calculate and report the GPU usage of three models with
increased input image size. The inference GPU memory usage of the ResNet, the
original VIT, and the proposed CSVT are shown in Fig. 10. Our proposed CSVT
has linear computational complexity with the size of the input image, which
makes it possible to scale to much larger image size (1600×1600 with 16GB GPU
memory and 1344×1344 with 12GB GPU memory). The original VIT has quadratic
computational complexity to the image size, which can only handle images with
a size of 896×896 in 16 GB GPU memory and 784×784 in 12 GB GPU memory.
Meanwhile, the proposed model has better computational efficiency and
utilization than that of ResNet with a larger image.

Figure 10: Inference GPU memory usage of ResNet, original VIT, and proposed
CSVT

####  IV-D3 Evaluation of the generalizability of the proposed model using
independent datasets

To evaluate the generalizability of the proposed CSVT model, we evaluate the
trained model on independent drone datasets captured in every growing stage,
including Tillering & Stem Extension, Heading & Flowering and Ripening &
Maturity. The performance of the trained model on each growing stage are
reported in Fig. 11.

Figure 11: The model performance on independent drone datasets captured
throughout all growing stages Figure 12: The nitrogen status estimation result
on drone images captured at the tillering & stem extension stage

Compared to our proposed model’s accuracy based on training field data, the
accuracy on independent drone data is slightly decreased. The accuracy in
Tillering & Stem Extension, Heading & Flowering, and Ripening & Maturity is
0.939, 0.926 and 0.933, respectively. However, the performance of ResNet on
independent drones dropped significantly. Compared to its accuracy on ground
field data, it dropped by 0.07, 0.049 and 0.033 in all growing stages. Fig. 12
shows the N status estimation result on drone images captured at the early
growing stage (Tillering & Stem Extension). In the early stages of crop
growth, all characteristics are not prominent. The estimated result of ResNet
shows many misclassifications. The result of our proposed model is more
precise and more accurate. The result shows the good generalizability of our
proposed model. To investigate the reasons for the good generalizability of
our proposed CSVT on the independent drone data, we visualize the attention
map from the last block of the trained model to evaluate the decision-making
area of each model. The attention maps are shown in Fig. 13. The middle column
is the attention map of the ResNet model, and the right column is the map of
the proposed CSVT. The first line is a field image sample, and the attention
maps show that the two models highlight the wheat stem areas on the drone
images in line 2 and 3. The attention map of the proposed CSVT distinguishes
well between regions with different N statuses. However, the attention map of
ResNet also distinguishes between different regions, but not as clearly as the
proposed CSVT. This leads to the higher accuracy of our proposed CSVT on
independent drone data. Based on these experiments, it has been demonstrated
that our proposed approach has good generalizability and reproducibility for
wheat N status estimation.

Figure 13: Visualization of the attention map for ResNet and proposed CSVT.
The middle column represents the attention map of ResNet. The right column
represents the attention map of CSVT

##  V Conclusion

We have proposed a channel-spatial attention-based vision transformer (CSVT)
for accurately estimating the nitrogen status of wheat using images from UAV
imagery. This work proposes a channel-spatial attention Block to replace self-
attention in ViT, capturing both spatial and channel features. In addition,
local-to-global self-supervised learning has been introduced to train the
model with unlabeled data. The proposed model has been compared with state-of-
the-art methods as well as being tested and evaluated against both testing and
independent datasets. Meanwhile, the cross-covariance matrix is used to
replace Gram to derive an attention map that changes the complexity of the
transformer layer from quadratic to linear. The experimental results show
competitive advantages over the existing works in terms of accuracy and
computing performance, and model generalizability. Moreover, since acquiring
labeled data is time consuming and costly, a local-to-global self-supervised
learning is introduced to pre-train the proposed CSVT with unlabelled data.
The proposed CSVT has been compared with state-of-the-art models as well as
tested and validated on both testing and independent datasets. The proposed
approach achieved high accuracy (0.96) with good generalizability and
reproducibility.

## Acknowledgment

The work reported in this paper The work reported in this paper has formed
part of the N2Vision project funded by UKRI-ISCF-TFP (Grant no. 134063).

## References

  * [1] FAO, “World fertilizer trends and outlook to 2020: Summary report,” 2017.
  * [2] A. Good, “Toward nitrogen-fixing plants,” _Science_ , vol. 359, no. 6378, p. 869–870, 2018, publisher: American Association for the Advancement of Science.
  * [3] K. Berger, J. Verrelst, J.-B. Féret, Z. Wang, M. Wocher, M. Strathmann, M. Danner, W. Mauser, and T. Hank, “Crop nitrogen monitoring: Recent progress and principal developments in the context of imaging spectroscopy missions,” _Remote Sensing of Environment_ , vol. 242, p. 111758, 6 2020.
  * [4] D. Wang, Z. Xu, J. Zhao, Y. Wang, and Z. Yu, “Excessive nitrogen application decreases grain yield and increases nitrogen loss in a wheat–soil system,” _Acta Agriculturae Scandinavica, Section B-Soil & Plant Science_, vol. 61, no. 8, pp. 681–692, 2011.
  * [5] Knoema, “Wheat area harvested,” 8 2021, [Online; accessed 2021-08-08]. [Online]. Available: https://knoema.com//atlas/topics/Agriculture/Crops-Production-Area-Harvested/Wheat-area-harvested
  * [6] M. Benitez Ramirez, “Monitoring nitrogen levels in the cotton canopy using real-time active-illumination spectral sensing,” 2010.
  * [7] M. Buchaillot, A. Gracia-Romero, O. Vergara-Diaz, M. A. Zaman-Allah, A. Tarekegne, J. E. Cairns, B. M. Prasanna, J. L. Araus, and S. C. Kefauver, “Evaluating maize genotype performance under low nitrogen conditions using rgb uav phenotyping techniques,” _Sensors_ , vol. 19, no. 8, p. 1815, 2019, publisher: Multidisciplinary Digital Publishing Institute.
  * [8] L. Prey, M. Von Bloh, and U. Schmidhalter, “Evaluating rgb imaging and multispectral active and hyperspectral passive sensing for assessing early plant vigor in winter wheat,” _Sensors_ , vol. 18, no. 9, p. 2931, 9 2018, number: 9 publisher: Multidisciplinary Digital Publishing Institute.
  * [9] P. A. Moghaddam, M. H. Derafshi, and M. Shayesteh, “A new method in assessing sugar beet leaf nitrogen status through color image processing and artificial neural network,” p. 6, 2010.
  * [10] W.-H. Yang, S. Peng, J. Huang, A. L. Sanico, R. J. Buresh, and C. Witt, “Using leaf color charts to estimate leaf nitrogen status of rice,” _Agronomy Journal_ , vol. 95, no. 1, p. 212–217, 2003, publisher: Wiley Online Library.
  * [11] L. Liu, B. Song, S. Zhang, and X. Liu, “A novel principal component analysis method for the reconstruction of leaf reflectance spectra and retrieval of leaf biochemical contents,” _Remote Sensing_ , vol. 9, no. 11, p. 1113, 2017, publisher: Multidisciplinary Digital Publishing Institute.
  * [12] J. P. Baresel, P. Rischbeck, Y. Hu, S. Kipp, Y. Hu, G. Barmeier, B. Mistele, and U. Schmidhalter, “Use of a digital camera as alternative method for non-destructive detection of the leaf chlorophyll content and the nitrogen nutrition status in wheat,” _Computers and Electronics in Agriculture_ , vol. 140, pp. 25–33, 8 2017.
  * [13] M. Pagola, R. Ortiz, I. Irigoyen, H. Bustince, E. Barrenechea, P. Aparicio-Tejo, C. Lamsfus, and B. Lasa, “New method to assess barley nitrogen nutrition status based on image colour analysis: comparison with spad-502,” _Computers and electronics in agriculture_ , vol. 65, no. 2, p. 213–218, 2009, publisher: Elsevier.
  * [14] M. I. Jordan and T. M. Mitchell, “Machine learning: Trends, perspectives, and prospects,” _Science_ , vol. 349, no. 6245, p. 255–260, 2015, publisher: American Association for the Advancement of Science.
  * [15] P. Shi, Y. Wang, J. Xu, Y. Zhao, B. Yang, Z. Yuan, and Q. Sun, “Rice nitrogen nutrition estimation with rgb images and machine learning methods,” _Computers and Electronics in Agriculture_ , vol. 180, p. 105860, 1 2021.
  * [16] Z. Qiu, F. Ma, Z. Li, X. Xu, H. Ge, and C. Du, “Estimation of nitrogen nutrition index in rice from uav rgb images coupled with machine learning algorithms,” _Computers and Electronics in Agriculture_ , vol. 189, p. 106421, 10 2021.
  * [17] H. Zha, Y. Miao, T. Wang, Y. Li, J. Zhang, W. Sun, Z. Feng, and K. Kusnierek, “Improving unmanned aerial vehicle remote sensing-based rice nitrogen nutrition index prediction with machine learning,” _Remote Sensing_ , vol. 12, no. 2, p. 215, 2020, publisher: Multidisciplinary Digital Publishing Institute.
  * [18] A. Chlingaryan, S. Sukkarieh, and B. Whelan, “Machine learning approaches for crop yield prediction and nitrogen status estimation in precision agriculture: A review,” _Computers and electronics in agriculture_ , vol. 151, p. 61–69, 2018, publisher: Elsevier.
  * [19] H. Liu, H. Zhu, and P. Wang, “Quantitative modelling for leaf nitrogen content of winter wheat using uav-based hyperspectral data,” _International Journal of Remote Sensing_ , vol. 38, no. 8-10, p. 2117–2134, 2017, publisher: Taylor & Francis.
  * [20] M. Z. Alom, M. Hasan, C. Yakopcic, T. M. Taha, and V. K. Asari, “Improved inception-residual convolutional neural network for object recognition,” _Neural Computing and Applications_ , 8 2018, [Online; accessed 2019-02-20]. [Online]. Available: https://doi.org/10.1007/s00521-018-3627-6
  * [21] L. Nanni, S. Ghidoni, and S. Brahnam, “Handcrafted vs. non-handcrafted features for computer vision classification,” _Pattern Recognition_ , vol. 71, pp. 158–172, 11 2017.
  * [22] K. P. Lewis and J. D. Espineli, “Classification and detection of nutritional deficiencies in coffee plants using image processing and convolutional neural network (cnn),” vol. 9, no. 04, p. 6, 2020.
  * [23] P. K. Sethy, N. K. Barpanda, A. K. Rath, and S. K. Behera, “Nitrogen deficiency prediction of rice crop based on convolutional neural network,” _Journal of Ambient Intelligence and Humanized Computing_ , vol. 11, no. 11, pp. 5703–5711, 11 2020.
  * [24] T.-T. Tran, J.-W. Choi, T.-T. H. Le, and J.-W. Kim, “A comparative study of deep cnn in forecasting and classifying the macronutrient deficiencies on development of tomato plant,” _Applied Sciences_ , vol. 9, no. 8, p. 1601, 1 2019, number: 8 publisher: Multidisciplinary Digital Publishing Institute.
  * [25] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, “Transformers: State-of-the-art natural language processing.” Online: Association for Computational Linguistics, 10 2020, p. 38–45, [Online; accessed 2021-05-12]. [Online]. Available: https://www.aclweb.org/anthology/2020.emnlp-demos.6
  * [26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _arXiv:1706.03762 [cs]_ , 6 2017, arXiv: 1706.03762. [Online]. Available: http://arxiv.org/abs/1706.03762
  * [27] F. Ben Abdallah, W. Philippe, and J. P. Goffart, “Comparison of optical indicators for potato crop nitrogen status assessment including novel approaches based on leaf fluorescence and flavonoid content,” _Journal of Plant Nutrition_ , vol. 41, no. 20, p. 2705–2728, 2018, publisher: Taylor & Francis.
  * [28] X. Xiong, J. Zhang, D. Guo, L. Chang, and D. Huang, “Non-invasive sensing of nitrogen in plant using digital images and machine learning for brassica campestris ssp. chinensis l.” _Sensors_ , vol. 19, no. 11, p. 2448, 2019\.
  * [29] A. Nguy-Robertson, Y. Peng, T. Arkebauer, D. Scoby, J. Schepers, and A. Gitelson, “Using a simple leaf color chart to estimate leaf and canopy chlorophyll a content in maize (zea mays),” _Communications in Soil Science and Plant Analysis_ , vol. 46, no. 21, pp. 2734–2745, 2015.
  * [30] C. Witt, J. Pasuquin, R. Mutters, and R. J. Buresh, “New leaf color chart for effective nitrogen management in rice,” _Better crops_ , vol. 89, no. 1, p. 36–39, 2005.
  * [31] G. R V Hari, S. Riyanarto, and W. Tri Adhi, “Sugarcane leaf color classification in sa*b* color element composition.” Jakarta-Indonesia: 2013 International Conference on Computer, Control, Informatics and Its Application (IC3INA), 11 2013, pp. 175–178, [Online; accessed 2021-10-05]. [Online]. Available: http://repository.unitomo.ac.id/40/
  * [32] L.-L. Jia, M.-S. Fan, F.-S. Zhang, X.-P. Chen, S.-h. Lü, and Y.-M. Sun, “Nitrogen status diagnosis of rice by using a digital camera,” _Spectroscopy and Spectral Analysis_ , vol. 29, no. 8, p. 2176–2179, 2009, publisher: Editorial Office, Spectroscopy and Spectral Analysis.
  * [33] J. F. Reyes, C. Correa, and J. Zúñiga, “Reliability of different color spaces to estimate nitrogen spad values in maize,” _Computers and Electronics in Agriculture_ , vol. 143, pp. 14–22, 12 2017.
  * [34] M. Wiwart, G. Fordoński, K. Żuk Go\laszewska, and E. Suchowilska, “Early diagnostics of macronutrient deficiencies in three legume species by color image analysis,” _Computers and electronics in agriculture_ , vol. 65, no. 1, p. 125–132, 2009, publisher: Elsevier.
  * [35] G. F. Mata-Donjuan, A. Mercado-Luna, E. Rico-García, and G. Herrera-Ruiz, “Use of improved hue, luminance and saturation (ihls) color space in the estimation of nitrogen on tomato seedlings (lycopersicon esculentum),” _Scientific Research and Essays_ , vol. 7, no. 27, pp. 2343–2349, 2012.
  * [36] L. K. Mehra, C. Cowger, K. Gross, and P. S. Ojiambo, “Predicting pre-planting risk of stagonospora nodorum blotch in winter wheat using machine learning models,” _Frontiers in plant science_ , vol. 7, p. 390, 2016.
  * [37] X. Yao, Y. Huang, G. Shang, C. Zhou, T. Cheng, Y. Tian, W. Cao, and Y. Zhu, “Evaluation of six algorithms to monitor wheat leaf nitrogen concentration,” _Remote Sensing_ , vol. 7, no. 11, pp. 14 939–14 966, 2015\.
  * [38] K.-J. Lee and B.-W. Lee, “Estimating canopy cover from color digital camera image of rice field,” _Journal of Crop Science and Biotechnology_ , vol. 14, no. 2, pp. 151–155, 6 2011.
  * [39] Y. Li, D. Chen, C. N. Walker, and J. F. Angus, “Estimating the nitrogen status of crops using a digital camera,” _Field Crops Research_ , vol. 118, no. 3, pp. 221–227, 9 2010.
  * [40] L. Roth and B. Streit, “Predicting cover crop biomass by lightweight uas-based rgb and nir photography: an applied photogrammetric approach,” _Precision Agriculture_ , vol. 19, no. 1, pp. 93–114, 2 2018.
  * [41] B. Zhao, Y. Zhang, A. Duan, Z. Liu, J. Xiao, Z. Liu, A. Qin, D. Ning, S. Li, and S. T. Ata-Ul-Karim, “Estimating the growth indices and nitrogen status based on color digital image analysis during early growth period of winter wheat,” _Frontiers in Plant Science_ , vol. 12, p. 502, 2021.
  * [42] S. Azimi, T. Kaur, and T. K. Gandhi, “A deep learning approach to measure stress level in plants due to nitrogen deficiency,” _Measurement_ , vol. 173, p. 108650, 3 2021.
  * [43] S. H. Lee, C. S. Chan, S. J. Mayo, and P. Remagnino, “How deep learning extracts and learns leaf features for plant classification,” _Pattern Recognition_ , vol. 71, pp. 1–13, 11 2017.
  * [44] M. A. Islam, S. Jia, and N. D. B. Bruce, “How much position information do convolutional neural networks encode?” _arXiv:2001.08248 [cs]_ , 1 2020, arXiv: 2001.08248. [Online]. Available: http://arxiv.org/abs/2001.08248
  * [45] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv:2010.11929 [cs]_ , 10 2020, arXiv: 2010.11929. [Online]. Available: http://arxiv.org/abs/2010.11929
  * [46] Y. Dong, J.-B. Cordonnier, and A. Loukas, “Attention is not all you need: Pure attention loses rank doubly exponentially with depth,” _arXiv:2103.03404 [cs]_ , 3 2021, arXiv: 2103.03404 version: 1. [Online]. Available: http://arxiv.org/abs/2103.03404
  * [47] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” _arXiv:2012.12877 [cs]_ , 1 2021, arXiv: 2012.12877. [Online]. Available: http://arxiv.org/abs/2012.12877
  * [48] X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang, “Self-supervised learning: Generative or contrastive,” _IEEE Transactions on Knowledge and Data Engineering_ , pp. 1–1, 2021.
  * [49] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” _arXiv:1406.2661 [cs, stat]_ , 6 2014, arXiv: 1406.2661. [Online]. Available: http://arxiv.org/abs/1406.2661
  * [50] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” vol. 2. IEEE, 2006, p. 1735–1742.
  * [51] J. Bromley, J. W. Bentz, L. Bottou, I. Guyon, Y. LeCun, C. Moore, E. Säckinger, and R. Shah, “Signature verification using a “siamese” time delay neural network,” _International Journal of Pattern Recognition and Artificial Intelligence_ , vol. 7, no. 04, p. 669–688, 1993, publisher: World Scientific.
  * [52] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko, “Bootstrap your own latent: A new approach to self-supervised learning,” _arXiv:2006.07733 [cs, stat]_ , 9 2020, arXiv: 2006.07733. [Online]. Available: http://arxiv.org/abs/2006.07733
  * [53] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” _arXiv:2006.09882 [cs]_ , 1 2021, arXiv: 2006.09882. [Online]. Available: http://arxiv.org/abs/2006.09882
  * [54] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” 2020, p. 9729–9738.
  * [55] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” _arXiv:2002.05709 [cs, stat]_ , 6 2020, arXiv: 2002.05709. [Online]. Available: http://arxiv.org/abs/2002.05709
  * [56] A. El-Nouby, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, I. Laptev, N. Neverova, G. Synnaeve, J. Verbeek, and H. Jegou, “Xcit: Cross-covariance image transformers,” _arXiv:2106.09681 [cs]_ , 6 2021, arXiv: 2106.09681. [Online]. Available: http://arxiv.org/abs/2106.09681
  * [57] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv:1607.06450 [cs, stat]_ , 7 2016, arXiv: 1607.06450. [Online]. Available: http://arxiv.org/abs/1607.06450
  * [58] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv:1704.04861 [cs]_ , 4 2017, arXiv: 1704.04861. [Online]. Available: http://arxiv.org/abs/1704.04861
  * [59] L. Zhang, J. Song, A. Gao, J. Chen, C. Bao, and K. Ma, “Be your own teacher: Improve the performance of convolutional neural networks via self distillation,” 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 10 2019, pp. 3712–3721, [Online; accessed 2021-08-03]. [Online]. Available: https://ieeexplore.ieee.org/document/9008829/
  * [60] R. Wightman, A. Arora, C. Ha, Mike, contrastive, morizin, mrT23, C. Kertész, D. Mehta, G. Cucurull, K. Singh, A. Soare, A. Lavin, J. Zhuang, M. Hollemans, V. Shults, Y. Uchida, Z. Zhong, Comar, T. Kim, A. Chernov, G. D. Han, E. Uriegas, Jasha10, M. Kardas, M. Monashev, Mike, M. Chen, M. Rashad, and N. Mu, _rwightman/pytorch-image-models: v0.4.12. Vision Transformer AugReg support and more_. Zenodo, 6 2021, dOI: 10.5281/zenodo.5047297. [Online]. Available: https://zenodo.org/record/5047297
  * [61] I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Dollár, “Designing network design spaces,” _arXiv:2003.13678 [cs]_ , 3 2020, arXiv: 2003.13678. [Online]. Available: http://arxiv.org/abs/2003.13678
  * [62] M. Tan and Q. V. Le, “Efficientnetv2: Smaller models and faster training,” _arXiv:2104.00298 [cs]_ , 6 2021, arXiv: 2104.00298. [Online]. Available: http://arxiv.org/abs/2104.00298