# SQ-Swin: a Pretrained Siamese Quadratic Swin Transformer for Lettuce
Browning Prediction

Dayang Wang1 Boce Zhang2 Yongshun Xu1
Yaguang Luo3∗ Hengyong Yu${}^{1^{*}}$ IEEE Senior Member * Drs. Yaguang Luo
and Hengyong Yu serve as co-corresponding authors. 1Dayang Wang, Yongshun Xu,
and Hengyong Yu are with Department of Electrical and Computer Engineering,
University of Massachusetts, Lowell, MA, USA2Boce Zhang is with Department of
Food Science and Human Nutrition, University of Florida, Gainesville, FL,
USA3Yaguang Luo is with Food Quality Lab / Environmental Microbial and Food
Safety Lab, Agricultural Research Service, United States Department of
Agriculture, Beltsville, MD, 20705, US

###### Abstract

Enzymatic browning is a major quality defect of packaged “ready-to-eat” fresh-
cut lettuce salads. While there have been many research and breeding efforts
to counter this problem, progress is hindered by the lack of a technology to
identify and quantify browning rapidly, objectively, and reliably. Here, we
report a deep learning model for lettuce browning prediction. To the best of
our knowledge, it is the first-of-its-kind on deep learning for lettuce
browning prediction using a pretrained Siamese Quadratic Swin (SQ-Swin)
transformer with several highlights. First, our model includes quadratic
features in the transformer model which is more powerful to incorporate real
world representations than the linear transformer. Second, a multi-scale
training strategy is proposed to augment the data and explore more of the
inherent self-similarity of the lettuce images. Third, the proposed model uses
a siamese architecture which learns the inter-relations among the limited
training samples. Fourth, the model is pretrained on the ImageNet and then
trained with the reptile meta-learning algorithm to learn higher-order
gradients than a regular one. Experiment results on the fresh-cut lettuce
datasets show that the proposed SQ-Swin outperforms the traditional methods
and other deep learning-based backbones.

###### Index Terms:

lettuce, enzymatic browning, transformer, quadratic, siamese model, reptile.

##  I Introduction

Lettuce (Lactuca sativa L.) is an important vegetable crop worldwide and a
common ingredient in packaged ’ready-to-eat’ salads. However, browning
discoloration on the leaf ribs’ cut edges significantly reduces the vegetable
quality and shortens the shelf-life [1, 2]. Many studies have shown that the
browning of fresh-cut lettuce is caused by the enzymatic oxidation of phenolic
metabolites [3]. The tissue injuries sustained during cutting disrupt the
natural compartmentation of polyphenol oxidase and phenolics, thus
accelerating the browning process [4, 5].

Developing technologies to accurately identify, quantify, and predict browning
is critical, yet, achieving both reliability and efficiency is challenging.
Commonly used colorimeters are useless since the cut edges are much narrower
than the available smallest probes. Visual evaluation approaches are not only
time consuming but also subjective. Digital imaging following feature
extracting with RGB or Lab improved objectivity and speed over the traditional
visual method. However, multiple values required to depict the discoloration
made it impossible for data comparison among lettuce cultivars.

Compared to the traditional learning models, deep learning models can be
trained end-to-end to learn the high-dimensional features incrementally,
thereby eliminating the need for domain expertise and manual feature
extraction. Due to the advantages, a plethora of deep learning algorithms have
been developed in many science and engineering fields [6, 7, 8, 9, 10, 11,
12]. Very recently, the transformer has been featured as the most popular
backbone architecture [13, 14, 15, 16, 17, 18, 19, 20, 21, 22]. Dosovitskiy et
al. first introduced the transformer to the computer vision (CV) field by
mapping a 16×16 image patch into a word sequence [23]. Based on the first-of-
its-kind vision transformer, Yuan et al. introduced a Token-to-token process
to further enrich the token selection from the image. Wang et al. further
extended the Token-to-Token method to low-dose CT denoising and obtained more
perceptually pleasing CT images [24]. Touvron et al. introduced the
distillation to the transformer by taking the convnet as a teacher and the
transformer as a student, thus achieving comparable performance with convnet
on ImageNet [25]. Han et al. designed a transformer by considering the
attention inside the image patches. Liu et al. further employed hierarchical
shifted window attention on the vision transformer and achieved the state-of-
the-art performances in image classification, object detection, and semantic
segmentation [26].

The transformer models have proved their successes in various computer vision
tasks. However, their input tokens fed into the inherent self-attention module
are mapped from the image features using the simple linear neuron.
Nevertheless, the bio-neuron is far more complex and diverse in the real
world. Quadratic neuron is one of the critical neurons that features quadratic
mapping across the neural layers. Research has shown that quadratic neurons
encode more rich features than linear features [27, 28, 29, 30]. Therefore, we
are motivated for the first time to introduce the quadratic neuron to the
transformer and prototype a high-order transformer based on Swin transformer
modules. Naturally, we are curious whether the second-order SQ-Swin can
deliver competitive performances in the lettuce browning prediction field,
hence, transforming the theoretical deep learning model into proper deep
learning applications.

To the best of our knowledge, the contributions of this paper are fourfold: i)
The first deep learning model is developed for lettuce browning prediction.
ii) A quadratic transformer model is first proposed to empower the token
mapping neurons with more powerful representation ability. iii) The siamese
architecture is introduced to the quadratic transformer to learn the inter-
relations among the limited samples. Then, the reptile meta-learning algorithm
is adopted to train the proposed model. iv) With the merits of transfer
learning, the SQ-Swin is pretrained on the ImageNet, then trained and
evaluated on the fresh cut lettuce data. Comprehensive experiments demonstrate
the high efficacy and the efficiency of the SQ-Swin on the lettuce browning
prediction problem.

##  II Related works

Quadratic neuron. The mainstream deep neural network uses linear neurons for
feature inference. However, in the real world, the bio-neuron such as that in
the human brain’s nerve cell is fundamentally more diverse and complex. Fan et
al. prototyped a quadratic neural network by replacing the inner product in
the traditional artificial neuron with a quadratic operation [27]. The
quadratic neuron is the first high-order neuron. It has been explored in a
plethora of researches and shown the advantage over the traditional linear
neuron in representation and efficiency [27, 28, 29, 30].

Mathematically, given an input $\boldsymbol{x}\in\mathrm{R}^{n}$, a quadratic
neuron $\mathrm{q}$ is characterized as

$\mathrm{q}(\boldsymbol{x})=\sigma((\boldsymbol{x}^{\top}\mathbf{w}^{r}+\mathbf{b}^{r})(\boldsymbol{x}^{\top}\mathbf{w}^{g}+\mathbf{b}^{g})+(\boldsymbol{x}\odot\boldsymbol{x})^{\top}\mathbf{w}^{b}+\mathbf{b}^{b}),$
(1)

where $\mathbf{w}^{r},\mathbf{w}^{g},\mathbf{w}^{b}\in\mathrm{R}^{n}$ are
weights and $\mathbf{b}^{r},\mathbf{b}^{g},\mathbf{b}^{b}\in\mathrm{R}^{n}$
are biases. $\sigma(\cdot)$ is regular nonlinear activation function such as
ReLU. $\odot$ denotes element-wise product. We denote
$\mathbf{w^{r}},\mathbf{b^{r}}$ as linear terms, and $\mathbf{w}^{g}$,
$\mathbf{b}^{g}$, $\mathbf{w}^{b}$, and $\mathbf{b}^{b}$ as quadratic terms.
Notably, that the definition of the quadratic neuron only requires
$\mathcal{O}(3n)$ parameters, which is much less than the regular quadratic
complexity of $\mathcal{O}(\frac{n(n+1)}{2})$. Furthermore, to facilitate the
training of the quadratic parameters and guarantee the model convergence, Fan
et al. introduced a Relinear strategy which includes a special initialization
on the quadratic terms: $\mathbf{w}^{g}=0$, $\mathbf{b}^{g}=1$,
$\mathbf{w}^{b}=0$, and $\mathbf{b}^{b}=0$, assisted by a shrunk small
learning rate to prevent magnitude explosion [31].

Siamese model. The siamese model was proposed to address the few-shot learning
problem by learning the inter-relations among images [32], and it has been
explored in various applications [33, 34, 35, 36]. Bertinetto et al. adopted
the fully-convolutional siamese design in video object detection and achieved
high performances in many benchmarks [33]. Guo et al. proposed a simple yet
effective siamese fully convolutional classification and regression network
for visual tracking and achieved leading performance in real-time speed [37].
Chen et al. developed a siamese transformer to extract the non-local features
from the multitemporal image pairs and demonstrated the model’s potentials in
the multitemporal remote sensing interpretation tasks [38].

##  III Methods

The lettuce browning prediction task is a typical regression problem. Given
data $(\boldsymbol{x},\boldsymbol{y})$, a general deep regression model aims
to minimize the mean square error (MSE) between the predictions and the
labels:

$\underset{\boldsymbol{\theta}}{\min}\ \
\mathcal{J}(\boldsymbol{\theta};\boldsymbol{x})=\frac{1}{N}\left\|f(\boldsymbol{\theta};\boldsymbol{x})-\boldsymbol{y}\right\|^{2},$
(2)

where $f(\boldsymbol{\theta};\boldsymbol{x})$ is the model,
$\boldsymbol{\theta}$ is a collection of the model parameters, and $N$ is the
number of samples.

To further explore the inter-relations among the limited data, a siamese
design is introduced to the proposed SQ-Swin transformer. Given data pairs
$(\boldsymbol{x_{0}},\boldsymbol{y_{0}})$ and
$(\boldsymbol{x_{1}},\boldsymbol{y_{1}})$, suppose the two base quadratic swin
transformers generate features and predictions
($\boldsymbol{f_{0}},\boldsymbol{\hat{y}_{0}}$) and
($\boldsymbol{f_{1}},\boldsymbol{\hat{y}_{1}}$), respectively. Two losses are
used in the SQ-Swin model during optimization: prediction loss and siamese
loss. The prediction loss generalizes the margin between the predictions and
the true labels. The siamese loss is based on the prior that the linear
combination of specific features determines the lettuce browning level.
Therefore, the difference between the features should indicate the difference
between the labels. Specifically, The prediction loss is defined as:

$\mathcal{L}_{p}=\left\|\boldsymbol{y_{0}}-\boldsymbol{\hat{y}_{0}}\right\|^{2}+\left\|\boldsymbol{y_{1}}-\boldsymbol{\hat{y}_{1}}\right\|^{2}.$
(3)

The siamese loss is defined as:

$\mathcal{L}_{s}=\left\|\left\|\boldsymbol{f_{0}}-\boldsymbol{f_{1}}\right\|^{2}-\left\|\boldsymbol{y_{0}}-\boldsymbol{y_{1}}\right\|^{2}\right\|_{1}.$
(4)

The total loss is the combination of the two losses with a balance parameter
$\alpha$,

$\mathcal{L}=\mathcal{L}_{p}+\alpha\cdot\mathcal{L}_{s}.$ (5)

Figure 1: The architecture of proposed siamese quadratic swin transformer (SQ-
Swin).

###  III-A Model Architecture

This paper proposes a siamese quadratic swin transformer for lettuce browning
prediction (SQ-Swin). As shown in Fig. 1, the multiple-scale patch extraction
stage first generates scaled image patches for latter training process. Then,
image patches are randomly paired to feed into the SQ-Swin model.
Specifically, the SQ-Swin takes the siamese design as the base architecture,
which includes two routes of quadratic Swin transformer (Q-Swin) for feature
inference. The two Q-Swins share identical designs and parameters. Finally,
two multilayer perceptrons (MLPs) are attached on top of the Q-Swin for
feature extraction and regression, respectively.

Figure 2: Shift window attention.

Multi-scale patch extraction. Multiple scale visual patterns are essential in
semantic segmentation, fine-grained classification, and object detection [39,
40, 41, 42, 43]. Inspired by these works, we seek to exploit the multi-scale
information from the lettuce images. In Contrast to the common methods that
incorporate multi-scale features in the network design, we derive the multi-
scalability directly from the images by taking advantage of their self-
similarities. Specifically, the training images are extracted from multi-scale
patches of different sizes from the large raw lettuce image, then resized to a
uniformed shape when fed to the models. Thus, the training images are a
collections of mixed lettuce images with various feature scales, therefore
boosting the generalizability and robustness of the model.

Q-Swin transformer. Recently, the Swin transformer has been very popular as a
backbone architecture [26]. It introduced the hierarchical attention with
shifted windows as shown in Fig. 2. Specifically, given an input image, the
patch partition process first splits the image into non-overlapped patches.
Then linear embedding module maps these patches into tokens to feed to the
transformer blocks. Several blocks with attention are applied for the feature
inference. To reduce computational complexity and facilitate hierarchical
representation, the patch merging layers between successive transformer blocks
are applied to reduce the feature map size by half. Based on the design of the
Swin transformer, the proposed quadratic Swin transformer further employs a
quadratic transformer block for window attention rather than a regular linear
transformer.

Quadratic Swin Transformer Block: Instead of calculating the attention
globally where the relations among all the tokens from all the patches are
calculated, the swin transformer calculates attention within a certain window
as shown in Fig. 2. In a transformer block, for a given token sequence
$\mathbf{T}\in\mathbb{R}^{b\times n\times d}$ within a window, they are first
mapped to three matrices of query, key and value
$\mathbf{Q},\mathbf{K},\mathbf{V}\in\mathbb{R}^{b\times n\times d}$. Here $b$
is the batch size, $n$ is the number of tokens, and $d$ is the token embedding
dimension. The output of the multiple head attention (MHA) is calculated as

$\mathrm{MHA}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{softmax}(\frac{\mathbf{Q}\mathbf{K^{\top}}}{\sqrt{d_{k}}})\mathbf{V},$
(6)

where $\frac{1}{\sqrt{{{d}_{k}}}}$ is the scaling factor determined by the
network depth. Instead of using the traditional linear multiple layer
perceptron (MLP) to calculate $\mathbf{Q},\mathbf{K}$, and $\mathbf{V}$, the
proposed quadratic transformer adopts quadratic multiple layer perceptron
(QMLP) to obtain the three matrices,

$\begin{cases}&\mathbf{Q}=(\mathbf{T}\mathbf{W}_{q}^{r})(\mathbf{T}\mathbf{W}_{q}^{g})+(\mathbf{T}\odot\mathbf{T})\mathbf{W}_{q}^{b}\\\
&\mathbf{K}=(\mathbf{T}\mathbf{W}_{k}^{r})(\mathbf{T}\mathbf{W}_{k}^{g})+(\mathbf{T}\odot\mathbf{T})\mathbf{W}_{k}^{b}\\\
&\mathbf{V}=(\mathbf{T}\mathbf{W}_{v}^{r})(\mathbf{T}\mathbf{W}_{v}^{g})+(\mathbf{T}\odot\mathbf{T})\mathbf{W}_{v}^{b},\\\
\end{cases}$ (7)

where $\mathbf{W}_{q}^{r}$, $\mathbf{W}_{q}^{g}$, $\mathbf{W}_{q}^{b}$,
$\mathbf{W}_{k}^{r}$, $\mathbf{W}_{k}^{g}$, $\mathbf{W}_{k}^{b}$,
$\mathbf{W}_{v}^{r}$, $\mathbf{W}_{v}^{g}$,
$\mathbf{W}_{v}^{b}\in\mathbb{R}^{d\times d}$ are linear operators. The biases
are omited for simplicity. After MHA, layer normalization (LN) and shortcuts
are also applied for more feature inference. As shown in Fig. 1, the quadratic
swin transformer block is characterized as:

$\begin{cases}&\mathbf{T}^{{}^{\prime}}=\mathrm{LN}(\mathrm{MHA}(\mathrm{QMLP}(\mathbf{T})))+\mathbf{T}\\\
&\mathbf{T}_{\mathrm{q}}=\mathrm{LN}(\mathrm{MLP}(\mathbf{T}^{\prime}))+\mathbf{T}^{{}^{\prime}},\end{cases}$
(8)

where $\mathbf{T^{\prime}},\mathbf{T_{q}}\in\mathbb{R}^{b\times n\times d}$.

###  III-B Model Training

The SQ-Swin can be trained regularly like common deep learning models using
optimization algorithms such as Adam or stochastic gradient descent (SGD).
However, since limited data are available in the lettuce browning prediction
task, the model is easily susceptible to over-fitting. We adopt reptile meta-
learning in the training process to alleviate this problem, a typical first-
order gradient-based meta-learning algorithm for few-shot learning problems
[44]. Specifically, given training batch $\boldsymbol{x}_{b}$ in a step with
model parameters $\phi$. A traditional optimizing algorithm such as Adam or
SGD simply updates the parameters as

$\tilde{\phi}=\mathrm{U}(\phi),$ (9)

where $\tilde{\phi}$ denotes updated parameters and $\mathrm{U}$ is the
optimizing algorithm. As shown in Fig. 3, in reptile the model will sample
different tasks $\tau$ on $\boldsymbol{x}_{b}$, then, the parameters are
updated as follows:

$\begin{cases}&\phi^{\prime}=\mathrm{U}_{\tau}^{k}(\phi)\\\
&\tilde{\phi}=\phi+\eta(\phi^{\prime}-\phi),\end{cases}$ (10)

where $k$ stands for the steps of the optimizing operator $\mathrm{U}$ like
Adam or SGD, and $\eta$ denotes the updating step for the new gradients. If
$k=1$, the reptile corresponds to the base optimizer. However, when $k>1$, the
update would not assemble taking a gradient step in the base optimizer. It
will include the terms from second-and-higher order gradients. Thus, the
reptile will converge to a minimum that is very different from the base
optimizer and fits better on the few-shot learning problem.

Figure 3: The update of the model parameters $\phi$ using reptile.

##  IV Experiments and Results

This section presents the dataset preparation, the experiment details and
regression results. The SQ-Swin is implemented on the fresh-cut lettuce
dataset collected by our group. The proposed model is quantitatively compared
with other state-of-the-art models. Moreover, the interpretability of the
proposed SQ-Swin is derived for more reliable application.

Figure 4: Some fresh-cut lettuce examples with different browning scores.

###  IV-A Dataset Preparation

Lettuce samples were grown, harvested, and processed according to Teng et. al.
(2019) [45] and Peng et. al. (2021) [46]. Digital images of the cut lettuce
were acquired inside a shooting tent with controlled lighting and shooting
distance using a Nikon D1X cameras. Then, the browning level of each image is
determined independently by three trained evaluators. Finally, the label of
each image is an average of the three scores from the practitioners. There are
100 lettuce images of $7360\times 4912$ in total with different browning
levels, of which 80% images are used for training and 20% images for testing.
The browning scores range continuously from 0 to 3.0, corresponding to
different brown levels as indicated in Table I. Fig. 4 also shows some lettuce
images with different browning scores. In the multiple scale patch extraction
stage, the original patches are randomly extracted from the raw image
$7360\times 4912$ with size of $1024\times 1024$, $2048\times 2048$,
$3072\times 3072$, or $4096\times 4096$. Forty patches are extracted from a
single image. Then, all these extracted patches are resized into $224\times
224$ with bilinear interpolation. Two image transformations flipping (up/down,
left/right) and rotation ($90\degree$, $180\degree$, or $270\degree$) are also
applied to augment the dataset.

TABLE I: Different browning score and description. Score | Description
---|---
0 | fresh
1.0 | slight brown
2.0 | medium brown
3.0 | heavy brown

###  IV-B Experiment Settings

All experiments are conducted on Ubuntu 18.04.5 LTS, Intel(R) Core (TM)
i9-9920X CPU @ 3.50GHz using PyTorch 1.7.1 [47], CUDA 10.2.0. with four NVIDIA
2080TI 11G GPUs. Below are the details of the experiment settings:

  * •

We adopt the default version with 2, 2, 6, 2 layers in the four stages for the
numbers of layers from the base Swin transformer modules. Between each stage,
there is a downsampling and window shift operation to facilitate diverse
representations.

  * •

The window size for each Q-Swin transformer stage is 7, and the shift is
$\frac{1}{4}$ of the corresponding feature map.

  * •

After pretrained on the ImageNet, an MLP of size 100 is attached on top of the
SQ-Swin for the feature extraction. Then, a regression head is appended for
browning level prediction.

  * •

For reptile meta-learning, the mini-batch size for each sampled task is 32,
and the inner epoch is 4 with a learning rate of 0.6.

  * •

In the meta step of the reptile, the model is trained using Adam through 200
epochs with a batch size of 256. Except for the quadratic terms, all other
parameters are optimized with an initial learning rate of $1\times 10^{-4}$
and scheduled learning rate decay of 0.2 at epochs 100 and 150.

  * •

The training of the quadratic terms utilizes the Relinear strategy. The
quadratic terms are initially set untrainable and begin to train on epoch 50.
The initial learning rate of the quadratic terms are $1\times 10^{-6}$, and
then it is decreased to $2\times 10^{-7}$ and $4\times 10^{-8}$ on epoch 100
and 150, respectively.

###  IV-C Evaluation Metrics

Three metrices: mean absolute error (MAE), mean square error (MSE) and
Pearson’s correlation coefficient (PCC) [48] are employed to measure the
prediction performance quantitatively. Given the true labels $\boldsymbol{y}$
and the model predictions $\hat{\boldsymbol{y}}$, MAE is expressed as
$\mathrm{MAE}(\boldsymbol{y},\hat{\boldsymbol{y}})=\frac{1}{N}|\boldsymbol{y}-\hat{\boldsymbol{y}}|$,
where $N$ is the number of testing samples. It is a straightforward
representation of the prediction errors, thereby more intuitive. Compared with
other metrics, it provides more insight and interpretation of the result.
Furthermore, to better evaluate the performance on the outlier data, MSE is
also adopted
$\mathrm{MSE}(\boldsymbol{y},\hat{\boldsymbol{y}})=\frac{1}{N}||\boldsymbol{y}-\hat{\boldsymbol{y}}||^{2}$,
since it tends to penalize more significant prediction errors than smaller
one, thus magnifying or inflating the mean error. Next, PCC is employed to
measure the association or the statistical relationship between the prediction
and the truth to make the prediction more reliable. Using covariance in the
formation, it informs us the linear correlation between the two variables.

$\mathrm{PCC}(\boldsymbol{y},\hat{\boldsymbol{y}})=\frac{Cov(\boldsymbol{y},\hat{\boldsymbol{y}})}{\sigma_{\boldsymbol{y}}\sigma_{\hat{\boldsymbol{y}}}},$
(11)

where $Cov(\cdot)$ indicates the covariance. $\sigma_{\boldsymbol{y}}$ and
$\sigma_{\hat{\boldsymbol{y}}}$ are the standard deviations.

###  IV-D Comparative Analysis

The SQ-Swin is fairly compared with the state-of-the-art deep convolution
neural network and transformer models. The convolution models include VGG11
[49], Ghostnet [50], and Resnet18 [6]. The transformer models include Twins
[51], Deit [52], TNT [53], Visformer [54], and Swin transformer [26]. We
reimplement these models on the lettuce data according to their officially
disclosed codes. Specifically, the initial parameters of these models are
transfered from the pretrained models in Pytorch Image Models [55]. Then, we
add a feature extraction layer and a regression head to these models to match
the lettuce browning prediction task.

Figure 5: The siamese and prediction losses of the linear transformer and the quadratic transformer. TABLE II: Quantitative evaluation results of different methods on the testing data. The proposed model is marked by a star, and the best results are bold-font. Model | Type | MAE$\downarrow$ | MSE$\downarrow$ | PCC$\uparrow$
---|---|---|---|---
vgg11 | Conv | 0.1165 | 0.0261 | 0.9696
Densenet121 | Conv | 0.1177 | 0.0247 | 0.9710
Resnet18 | Conv | 0.1210 | 0.0265 | 0.9691
Res2net | Conv | 0.1349 | 0.0305 | 0.9626
Twins | Trans | 0.1384 | 0.0328 | 0.9580
Deit | Trans | 0.1231 | 0.0269 | 0.9674
TNT | Trans | 0.1260 | 0.0276 | 0.9648
Visformer | Trans | 0.1338 | 0.0300 | 0.9628
Swin-T | Trans | 0.1253 | 0.0269 | 0.9661
S-Swin* | Trans | 0.0280 | 0.0027 | 0.9968
SQ-Swin* | Trans | 0.0203 | 0.0018 | 0.9976

Quantitative Results. Table II shows that all studied models are effective in
predicting the lettuce browning level. All other convolution or transformer-
based competitors can achieve a low MAE around 0.12, an MSE around 0.02, and a
PCC around 0.96. However, with the siamese and reptile design, the proposed
model greatly surpasses other state-of-the-art deep models in terms of the
three metrics, even utilizing a traditional linear swin transformer as the
base network. When the quadratic transformer is introduced, the performance
further surpasses the traditional linear one with an MAE of 0.0203, an MSE of
0.0018, and a PCC of 0.9976.

Figure 6: Representative feature maps of the linear transformer and the
quadratic transformer.

Quadratic vs. Linear Transformer. Besides, we also illustrate the prediction
losses and the siamese losses during the training of the quadratic (SQ-Swin)
and linear (S-Swin) transformers. As shown in Fig. 5, both losses of the two
transformers drops in the same way in epoch 0-50, because quadratic terms are
frozen at the start. Then, the loss of the linear transformer drops faster
than the loss of the quadratic transformer in epoch 50-100. However, when the
learning rate decreases in epoch 100 and 150, the quadratic loss drops more
while the linear loss drops but rise again. The zoomed curves of epoch 60-100
also show that the quadratic curve is smoother than the linear one. This
further indicates a more stable training process.

After training, as shown in Fig. 6, the feature maps of the quadratic
transformer and the linear transformer are also visualized. In the early
layers, the two transformers follow very similar learning patterns. The
feature outline and texture are nearly the same. However, when the networks go
deep, the features gradually become distinct. The last layer indicates that
the quadratic transformer can learn more unified and polarized features, which
to some extent, provides more representative features for the browning
evaluation, thereby obtaining a more accurate prediction with lower MAE. In
summary, the quadratic transformer is more powerful in loss optimization,
training stability, and feature representations compared with the linear
transformer.

Inference Time and Parameters. The traditional lettuce image analysis uses
Progenesis QI software (Nonlinear Dynamics, Newcastle, U.K.) for tabular data
extraction. It takes about 0.5-16 minutes of inference time, and then it uses
principal component analysis (PCA) to extract principal components [56]. The
whole process is rather time-consuming, thus hindering its real-world
applications. Therefore, we are motivated to study the inference time of the
proposed SQ-Swin on a single image. During inference, the SQ-Swin evaluates 40
randomly selected multi-scale patches from each testing image of $7360\times
4912$, then the averaged score is used as the prediction. Table III shows that
the inference time of the two deep learning methods is within one second which
is much shorter than that of the traditional model. Moreover, the employment
of the siamese architecture further expedites the inference process since the
siamese models can process two images simultaneously in theory.

Further, The number of the parameters and multiply-accumulate operations
(MACs) of the SQ-Swin are also studied. As shown in Table IV, the SQ-Swin has
few more parameters because the quadratic mapping introduces extra
computation. But the computation cost is still comparable to that of the
linear transformer (S-Swin). Please note that the MACs are fewer in SQ-Swin
because we rewrote part of the inherent torch code.

TABLE III: Inference time of the traditional model, Densenet121, and SQ-Swin on a single lettuce image. Model | Type | Time
---|---|---
Progenesis QI | Tradition | 0.5-16min
Densenet121 | Non-Siamese (deep learning) | 0.85s
SQ-Swin | Siamese (deep learning) | 0.62s
TABLE IV: Parameters of the SQ-Swin and S-Swin. Model | #param. | MACs
---|---|---
S-Swin | 28.39M | 4.36G
SQ-Swin | 41.36M | 3.32G

###  IV-E Visual Interpretability

In real-world applications, interpretability is a desired property of any
proposed model. The higher interpretability of a model is, the more people
will comprehend how the decisions are made. Therefore, to further boost the
reliability and the applicability of the SQ-Swin, we derive some visual
interpretation of the model by leveraging the inherent attention mechanism in
the model. Specifically, we extract the attention map
$\mathrm{Att}(\mathbf{Q},\mathbf{K})=\mathrm{softmax}(\frac{\mathbf{Q}\mathbf{K^{\top}}}{\sqrt{d_{k}}})$
from the multiple head attention module and analyze the internal patterns of
the attention maps.

By visualizing the initial attention map of the pretrained SQ-Swin in Fig. 7,
it is noticed that the attention basically focuses on the desired fresh-cut
lettuce pieces. Moreover, by investigating the lettuce parts on which the
attention centers as indicated by the red dotted circle in Fig. 7, it
illustrates that the attention tends to focus on the corner or the boundary of
the lettuce pieces. Therefore, we conclude that the SQ-Swin somehow relies on
the corner or the boundary part of the lettuce pieces to make a browning
prediction decision. This corresponds to the fact that the browning process on
a fresh-cut vegetable piece usually starts from the corners and boundaries.

Figure 7: Examples of the attention maps for different lettuce patches.

###  IV-F Ablation Study

In the SQ-Swin transformer, several important methods are employed to boost
the model performance: multi-scale patch, pretraining, siamese architecture,
reptile meta-learning, and quadratic transformer. In this part, we intensively
investigate the impacts of these methods. MAE is utilized to quantitatively
evaluate the prediction performances since it informs us the intuitive gap
between the prediction and the truth. Table V indicates that all studied
methods can effectively boost the model performances. For example, adopting
the pretraining, siamese architecture, and the reptile learning method
improves the performance with an MAE above 0.4. Following up, the multi-scale
patch also decreases the prediction error by 0.0337. However, the performance
of the quadratic transformer further surpasses the excellent scores under
these four methods with an improvement of 0.0077. In conclusion, all
investigated methods in the SQ-Swin are valid and effective for the lettuce
browning prediction problem.

TABLE V: Quantitative evaluation results of different methods on the testing data. The bold-font text indicates the new method added to the model in the previous row. The bold-font numbers indicate the best results.  SQ-Swin | MAE$\downarrow$ | Boost
---|---|---
none | 0.1921 | -
multi-scale | 0.1584 | 0.0337$\downarrow$
multi-scale+pretrain | 0.1174 | 0.0410$\downarrow$
multi-scale+pretrained+siamese | 0.0712 | 0.0462$\downarrow$
multi-scale+pretrained+siamese+reptile | 0.0280 | 0.0432$\downarrow$
multi-scale+pretrained+siamese+reptile+quadratic | 0.0203 | 0.0077$\downarrow$

##  V Conclusion

In this paper, we propose a pretrained siamese quadratic swin transformer for
lettuce browning prediction. To the best of our knowledge, this is the first-
of-its-kind deep learning model in the lettuce brown detection field, and it
greatly improves the inference time for the lettuce browning prediction.
Moreover, compared with a typical linear transformer, the proposed SQ-Swin is
the first second-order transformer that is more powerful in capturing real-
world representations with comparable computation complexity. Experimental
results validate its advantages in loss optimization, training stability, and
feature representations. Furthermore, we unveil the visual interpretability of
the SQ-Swin for browning prediction, which facilitates its reliability and
applicability. In addition, comprehensive ablation studies show that all
components, including multiscale patch, pretraining, siamese architecture,
reptile meta-learning, and quadratic transformer, are effective in the lettuce
browning prediction. Finally, extensive comparative experiments indicate that
the SQ-Swin outperforms the state-of-the-art methods in terms of MAE, MSE, and
PCC. In the future, we will explore further the application of the SQ-Swin to
other food science areas.

##  VI Acknowledgement

Authors wish to thank Drs. Ivan Simko and Hui Peng at USDA-ARS in Salinas CA
for growing and harvesting lettuce, and Drs. Bin Zhou, Zi Teng, Ms. Ellen
Turner, and Mr. Daniel Pearlstein at USDA-ARS in Beltsville MD for processing
and evaluating lettuce.

## References

  * [1] Q. He and Y. Luo, “Enzymatic browning and its control in fresh-cut produce.,” 2007\.
  * [2] T. Zhou, A. Harrison, R. McKellar, J. Young, J. Odumeru, P. Piyasena, X. Lu, D. Mercer, and S. Karr, “Determination of acceptability and shelf life of ready-to-use lettuce by digital image analysis,” Food Research International, vol. 37, no. 9, pp. 875–881, 2004.
  * [3] F. Taranto, A. Pasqualone, G. Mangini, P. Tripodi, M. M. Miazzi, S. Pavan, and C. Montemurro, “Polyphenol oxidases in crops: Biochemical, physiological and genetic aspects,” International journal of molecular sciences, vol. 18, no. 2, p. 377, 2017.
  * [4] E. Cantos, J. C. Espín, and F. A. Tomás-Barberán, “Effect of wounding on phenolic enzymes in six minimally processed lettuce cultivars upon storage,” Journal of Agricultural and Food Chemistry, vol. 49, no. 1, pp. 322–330, 2001.
  * [5] M. C. Luna, J. A. Tudela, F. A. Tomas-Barberan, and M. I. Gil, “Modified atmosphere (ma) prevents browning of fresh-cut romaine lettuce through multi-target effects related to phenolic metabolism,” Postharvest Biology and Technology, vol. 119, pp. 84–93, 2016.
  * [6] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [7] F. Fan, D. Wang, H. Guo, Q. Zhu, P. Yan, G. Wang, and H. Yu, “On a sparse shortcut topology of artificial neural networks,” IEEE Transactions on Artificial Intelligence, 2021.
  * [8] Z. Wu, R. Ge, G. Shi, L. Zhang, Y. Chen, L. Luo, Y. Cao, and H. Yu, “Md-ndnet: a multi-dimensional convolutional neural network for false-positive reduction in pulmonary nodule detection,” Physics in Medicine & Biology, vol. 65, no. 23, p. 235053, 2020.
  * [9] Y. Li, C. Wang, Y. Cao, B. Liu, J. Tan, and Y. Luo, “Human pose estimation based in-home lower body rehabilitation system,” in 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8, IEEE, 2020.
  * [10] D. Wang, F.-L. Fan, B.-J. Hou, H. Zhang, R. Lai, H. Yu, and F. Wang, “Manifoldron: Direct space partition via manifold discovery,” arXiv preprint arXiv:2201.05279, 2022.
  * [11] Y. Li, C. Wang, Y. Cao, B. Liu, Y. Luo, and H. Zhang, “A-hrnet: Attention based high resolution network for human pose estimation,” in 2020 Second International Conference on Transdisciplinary AI (TransAI), pp. 75–79, IEEE, 2020.
  * [12] Z. Jia, Y. Luo, D. Wang, Q. N. Dinh, S. Lin, A. Sharma, E. M. Block, M. Yang, T. Gu, A. J. Pearlstein, et al., “Nondestructive multiplex detection of foodborne pathogens with background microflora and symbiosis using a paper chromogenic array and advanced neural network,” Biosensors and Bioelectronics, vol. 183, p. 113209, 2021.
  * [13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  * [14] J. Xin, R. Tang, J. Lee, Y. Yu, and J. Lin, “Deebert: Dynamic early exiting for accelerating bert inference,” arXiv preprint arXiv:2004.12993, 2020\.
  * [15] Y. Zhang, S. Sun, M. Galley, Y.-C. Chen, C. Brockett, X. Gao, J. Gao, J. Liu, and B. Dolan, “Dialogpt: Large-scale generative pre-training for conversational response generation,” arXiv preprint arXiv:1911.00536, 2019\.
  * [16] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, and A. Askell, “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, 2020.
  * [17] J. Chen, Y. Du, Y. He, W. P. Segars, Y. Li, and E. C. Frey, “Transmorph: Transformer for unsupervised medical image registration,” arXiv preprint arXiv:2111.10480, 2021.
  * [18] Y. Gu, Z. Piao, and S. J. Yoo, “Sthardnet: Swin transformer with hardnet for mri segmentation,” Applied Sciences, vol. 12, no. 1, p. 468, 2022.
  * [19] K. He, C. Gan, Z. Li, I. Rekik, Z. Yin, W. Ji, Y. Gao, Q. Wang, J. Zhang, and D. Shen, “Transformers in medical image analysis: A review,” arXiv preprint arXiv:2202.12165, 2022.
  * [20] F. Shamshad, S. Khan, S. W. Zamir, M. H. Khan, M. Hayat, F. S. Khan, and H. Fu, “Transformers in medical imaging: A survey,” arXiv preprint arXiv:2201.09873, 2022.
  * [21] Z. Xiong, C. Wang, Y. Li, Y. Luo, and Y. Cao, “Swin-pose: Swin transformer based human pose estimation,” arXiv preprint arXiv:2201.07384, 2022.
  * [22] Z. Wu, R. Ge, M. Wen, G. Liu, Y. Chen, P. Zhang, X. He, J. Hua, L. Luo, and S. Li, “Elnet: Automatic classification and segmentation for esophageal lesions using convolutional neural network,” Medical Image Analysis, vol. 67, p. 101838, 2021.
  * [23] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, and S. Gelly, “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
  * [24] D. Wang, F. Fan, Z. Wu, R. Liu, F. Wang, and H. Yu, “Ctformer: Convolution-free token2token dilated vision transformer for low-dose ct denoising,” arXiv preprint arXiv:2202.13517, 2022.
  * [25] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee, 2009.
  * [26] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” arXiv preprint arXiv:2103.14030, 2021.
  * [27] F. Fan, H. Shan, M. K. Kalra, R. Singh, G. Qian, M. Getzin, Y. Teng, J. Hahn, and G. Wang, “Quadratic autoencoder (q-ae) for low-dose ct denoising,” IEEE transactions on medical imaging, vol. 39, no. 6, pp. 2035–2050, 2019.
  * [28] S. Sarao Mannelli, E. Vanden-Eijnden, and L. Zdeborová, “Optimization and generalization of shallow neural networks with quadratic activation functions,” Advances in Neural Information Processing Systems, vol. 33, pp. 13445–13455, 2020.
  * [29] F. Fan, J. Xiong, and G. Wang, “Universal approximation with quadratic deep networks,” Neural Networks, vol. 124, pp. 383–392, 2020.
  * [30] K. Xu, H. Bastani, and O. Bastani, “Robust generalization of quadratic neural networks via function identification,” arXiv preprint arXiv:2109.10935, 2021.
  * [31] F.-L. Fan, M. Li, F. Wang, R. Lai, and G. Wang, “Expressivity and trainability of quadratic networks,” arXiv preprint arXiv:2110.06081, 2021.
  * [32] G. Koch, R. Zemel, R. Salakhutdinov, et al., “Siamese neural networks for one-shot image recognition,” in ICML deep learning workshop, vol. 2, p. 0, Lille, 2015.
  * [33] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr, “Fully-convolutional siamese networks for object tracking,” in European conference on computer vision, pp. 850–865, Springer, 2016.
  * [34] D. Chicco, “Siamese neural networks: An overview,” Artificial Neural Networks, pp. 73–94, 2021.
  * [35] X. Chen and K. He, “Exploring simple siamese representation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750–15758, 2021.
  * [36] A. He, C. Luo, X. Tian, and W. Zeng, “A twofold siamese network for real-time object tracking,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4834–4843, 2018.
  * [37] D. Guo, J. Wang, Y. Cui, Z. Wang, and S. Chen, “Siamcar: Siamese fully convolutional classification and regression for visual tracking,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6269–6277, 2020.
  * [38] H. Chen, E. Nemni, S. Vallecorsa, X. Li, C. Wu, and L. Bromley, “Dual-tasks siamese transformer framework for building damage assessment,” arXiv preprint arXiv:2201.10953, 2022.
  * [39] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117–2125, 2017\.
  * [40] A. Kirillov, R. Girshick, K. He, and P. Dollár, “Panoptic feature pyramid networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6399–6408, 2019.
  * [41] S.-H. Gao, M.-M. Cheng, K. Zhao, X.-Y. Zhang, M.-H. Yang, and P. Torr, “Res2net: A new multi-scale backbone architecture,” IEEE transactions on pattern analysis and machine intelligence, vol. 43, no. 2, pp. 652–662, 2019\.
  * [42] C. Guo, B. Fan, Q. Zhang, S. Xiang, and C. Pan, “Augfpn: Improving multi-scale feature learning for object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12595–12604, 2020\.
  * [43] M. Pauly, R. Keiser, and M. Gross, “Multi-scale feature extraction on point-sampled surfaces,” in Computer graphics forum, vol. 22, pp. 281–289, Wiley Online Library, 2003.
  * [44] A. Nichol, J. Achiam, and J. Schulman, “On first-order meta-learning algorithms,” arXiv preprint arXiv:1803.02999, 2018.
  * [45] Z. Teng, Y. Luo, E. R. Bornhorst, B. Zhou, I. Simko, and F. Trouth, “Identification of romaine lettuce (lactuca sativa var. longifolia) cultivars with reduced browning discoloration for fresh-cut processing,” Postharvest Biology and Technology, vol. 156, p. 110931, 2019.
  * [46] H. Peng, Y. Luo, Z. Teng, B. Zhou, E. R. Bornhorst, J. M. Fonseca, and I. Simko, “Phenotypic characterization and inheritance of enzymatic browning on cut surfaces of stems and leaf ribs of romaine lettuce,” Postharvest Biology and Technology, vol. 181, p. 111653, 2021.
  * [47] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, and L. Antiga, “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, pp. 8026–8037, 2019.
  * [48] J. Benesty, J. Chen, Y. Huang, and I. Cohen, “Pearson correlation coefficient,” in Noise reduction in speech processing, pp. 1–4, Springer, 2009.
  * [49] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  * [50] K. Han, Y. Wang, Q. Tian, J. Guo, C. Xu, and C. Xu, “Ghostnet: More features from cheap operations,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1580–1589, 2020.
  * [51] X. Chu, Z. Tian, Y. Wang, B. Zhang, H. Ren, X. Wei, H. Xia, and C. Shen, “Twins: Revisiting the design of spatial attention in vision transformers,” Advances in Neural Information Processing Systems, vol. 34, 2021.
  * [52] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in International Conference on Machine Learning, pp. 10347–10357, PMLR, 2021.
  * [53] K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang, “Transformer in transformer,” Advances in Neural Information Processing Systems, vol. 34, 2021.
  * [54] Z. Chen, L. Xie, J. Niu, X. Liu, L. Wei, and Q. Tian, “Visformer: The vision-friendly transformer,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 589–598, 2021.
  * [55] R. Wightman, “Pytorch image models.” https://github.com/rwightman/pytorch-image-models, 2019.
  * [56] Z. Liu, J. Sun, Z. Teng, Y. Luo, L. Yu, I. Simko, and P. Chen, “Identification of marker compounds for predicting browning of fresh-cut lettuce using untargeted uhplc-hrms metabolomics,” Postharvest Biology and Technology, vol. 180, p. 111626, 2021.