# An Efficient MLP-based Point-guided Segmentation Network for Ore Images with
Ambiguous Boundary

Guodong Sun  Yuting Peng  Le Cheng  Mengya Xu  An Wang  Bo Wu  Hongliang Ren
and Yang Zhang Corresponding author: Yang Zhang.G. Sun, Y. Peng, L. Cheng, and
Y. Zhang are with the School of Mechanical Engineering, Hubei University of
Technology, Wuhan 430068, China (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>). M. Xu and H. Ren
are with the Department of Biomedical Engineering, National University of
Singapore (NUS), Singapore 117575, Singapore (e-mail<EMAIL_ADDRESS>hlren@ieee.org). A. Wang, H. Ren, and Y. Zhang are also with the Department of
Electronic Engineering, The Chinese University of Hong Kong, Hong Kong 999077,
China (e-mail: wa09@link.cuhk.edu.hk).B. Wu is with the Shanghai Advanced
Research Institute, Chinese Academy of Sciences, Shanghai 201210, China
(e-mail: wubo@sari.ac.cn).

###### Abstract

The precise segmentation of ore images is critical to the successful execution
of the beneficiation process. Due to the homogeneous appearance of the ores,
which leads to low contrast and unclear boundaries, accurate segmentation
becomes challenging, and recognition becomes problematic. This paper proposes
a lightweight framework based on Multi-Layer Perceptron (MLP), which focuses
on solving the problem of edge burring. Specifically, we introduce a
lightweight backbone better suited for efficiently extracting low-level
features. Besides, we design a feature pyramid network consisting of two MLP
structures that balance local and global information thus enhancing detection
accuracy. Furthermore, we propose a novel loss function that guides the
prediction points to match the instance edge points to achieve clear object
boundaries. We have conducted extensive experiments to validate the efficacy
of our proposed method. Our approach achieves a remarkable processing speed of
over 27 frames per second (FPS) with a model size of only 73 MB. Moreover, our
method delivers a consistently high level of accuracy, with impressive
performance scores of 60.4 and 48.9 in $AP_{50}^{box}$ and $AP_{50}^{mask}$
respectively, as compared to the currently available state-of-the-art
techniques, when tested on the ore image dataset. The source code will be
released at https://github.com/MVME-HBUT/ORENEXT.

Instance segmentation, Point guidance, Edge processing, Local correlation, Ore
image.

##  1 Introduction

Ore particle size analysis is an important part of ore processing tasks.
Particle size information is an essential indicator to judge the effectiveness
of the crusher, and it can serve as a guide for adjusting the process
parameters of each process. Accurate ore segmentation is a significant
prerequisite for particle size statistical analysis. The production site for
beneficiation presents a complex environment with several challenging factors,
such as ore stacking, ore adhesion resulting from dry-wet mixing, and
variations in lighting conditions, as depicted in Fig. 1. These factors pose a
significant challenge to achieving accurate ore segmentation. Moreover, the
beneficiation scene is often arranged in the field, so the limitation of
equipment resources is also a problem faced by the ore particle size analysis
system.

Traditional image processing methods perform the segmentation of ore by
setting thresholds, cluster analysis, and edge detection. With the development
of convolutional neural networks (CNN), the algorithms based on deep learning
have shown significant advancements in automatic feature extraction and
generalization performance [1, 2, 3, 4, 5], so they have gradually gained a
leading position in industrial image processing. Most existing instance
segmentation methods are implemented on public datasets such as MS COCO [6].
These methods are not effective for ore image processing. It can be seen from
Fig. 1(c) that the baseline framework is less capable of segmenting the ore
edges. The processing of ore images is significantly challenged by complex
working environments and the presence of diverse feature information.

(a) Input

(b) GroundTruth

(c) Baseline

(d) Ours

Figure 1: Visualizations of the feature maps. The ore stacking in the input
image makes the boundary difficult to distinguish. The edge features of the
feature maps obtained by the baseline are blurred. The consistency instances
with clearer edges can be obtained using our network.

Aiming at the specific needs of ore segmentation, the U-Net was used to
segment broken stones for the first time. However, these complex CNN-based
methods have high computational costs, large model sizes, and low accuracy.
Moreover, facing the complex environment, the existing algorithmic framework
will easily ignore the problem of edge blur caused by inter-adhesive and
shadowing of ore images. In this work, we focus on solving this problem and
designing an efficient network with less computational overhead, fewer
parameters, faster inference time, and better performance.

Recently, it has been found that MLP-based architectures can achieve
comparable results to CNN and Transformer methods with less computation. The
MLP-Mixer [7] proposed token mixing and channel mixing MLP to allow
interaction between spatial locations and channels. The ResMLP [8] used cross-
patch and cross-channel sublayers as components. Inspired by these works, we
propose OreNeXt, an instance segmentation model based on the MLP framework for
the ore edge problem. A lightweight MLP backbone network for feature
extraction is introduced, followed by a feature pyramid network using a
SparseMLP module to enhance the semantic information, then we introduce a loss
function guided by the edge. Using MLP and a simple hybrid mechanism, we
obtain a lightweight model suitable for deployment.

To implement this model framework to solve the above problems, we use the two-
stage detector PointRend [9] as the baseline for the ore task. For ore tasks,
the low-level features of ore overlapping edges are extremely crucial.
Therefore, we propose a novel backbone network StoneMLP, which incorporates
shifting operations to extract local information corresponding to different
axial shifts. A sparse feature pyramid network is proposed to strengthen the
small target information that is easy to ignore and misjudge while maintaining
sufficient semantic information. Furthermore, we propose a loss, including
detection loss that predicts the foreground score of each point and mask loss
that performs edge guidance by dynamically adjusting vertex pairing. As
illustrated in Fig. 1(d), our proposed framework is more likely to obtain
clear ore segmentation results. Experiments on the ORE image dataset
demonstrate that our framework performs state-of-the-art methods with smaller
model size and faster inference speed. Our main contributions are summarized
as follows.

  1. 1)

A lightweight image segmentation network OreNeXt is designed, which solves the
problem of blurred edges in ores by guiding the boundary points.

  2. 2)

We propose a lightweight backbone StoneMLP for capturing local correlation and
introduce a semantic enhancement SparseFPN network.

  3. 3)

A loss function that matches edges by guided points is introduced,
significantly improving the quality of predicted boundary details.

  4. 4)

The experimental results validate the efficacy of our method in enhancing the
performance of ore image segmentation tasks, while simultaneously reducing the
number of parameters and improving inference speed.

The remainder of this paper is structured as follows. Section 2 introduces an
overview of existing mineral image segmentation methods and MLP architecture
development. Section 3 presents our overall framework, including two new
structures and a new loss function. To validate the effectiveness of our
method, in Section 4, we conduct integrated experiments. The paper is
concluded in Section 5.

##  2 Related works

###  2.1 Ore image segmentation

Accurate individual ore segmentation is a crucial prerequisite for granularity
statistics and an essential component in ore processing tasks. With the
development of CNN, the task of ore image processing has gradually shifted
from traditional image processing techniques to deep learning-based
segmentation algorithms. For instance, Sun et al. [10] proposed an efficient
instance segmentation algorithm to split ore. Dipti et al. [11] proposed an
image segmentation system for estimating the granularity of oil sandstones.
The ore image on the conveyor belt often suffers from ore adhesion and
occlusion caused by dust and soil. These factors result in the blurring or
even complete disappearance of multiple ore edges in the image, posing
significant challenges to achieving accurate individual ore segmentation. To
address this edge problem, numerous models have been employed to tackle the
issue of under-segmentation in ore processing [12], Li et al. [13] proposed a
model based on U-Net, which alleviated the problem of ore granularity
detection by improving the loss function and using watershed technology. Liu
et al. [14] developed the RLPNet to obtain high-precision segmentation images
by reducing the interference of complex textures and enhancing the expression
of edge features. With the effective emergence of Transformer, numerous
Transformer-based designs have been proposed to effectively divide the
adhesive ore [15], but the network structures are often complex and not
suitable for outdoor deployment. Although these methods have improved the
segmentation of ore images, there are still problems in clear edge
segmentation.

###  2.2 Instance segmentation

Currently, instance segmentation methods can be divided into two categories:
one-stage and two-stage. One-stage methods mainly involve a simple fully
convolutional networks (FCN) architecture for mask prediction without region
of interest (RoI) pooling, such as YOLACT [16] and BlendMask [17], proposal
generation and feature re-pooling, with fast speed but poor accuracy. Two-
stage instance segmentation methods first detect boundary boxes and then
perform segmentation in each RoI region. Mask RCNN [18] adds a mask branch to
the Faster R-CNN [19] architecture. Mask-Refined R-CNN [20] refines the
detailed information of the target and considers the relationship between the
edge pixels of the object. PointRend [9] samples feature points with low
confidence scores and further improves their labels using a shared MLP.
Cascade Mask R-CNN [21] improves segmentation accuracy through cascading
methods. Since most two-stage methods use RoI Align operations, there is
information loss in the spatial features of large targets, especially at the
edges. Therefore, rough target edge prediction is a common problem for two-
stage instance segmentation algorithms.

Figure 2: A schematic overview of OreNeXt. The input image is fed into the
lightweight backbone StoneMLP (Fig. 3) to produce feature maps. StoneMLP
captures local dependencies and extracts edge information through horizontal
and vertical shift operations (Fig. 4). Then, the feature maps enter the
SparseFPN to generate multi-scale information-integrated feature maps. Our
improved FPN structure adds two SparseMLP modules (Fig. 5), which are divided
into three parallel branches for feature fusion through weighted summation.
The addition of two sparsely connected SparseMLP modules allows both local and
global features to be taken into account. Next, each layer feature map is
computationally fused by the region proposal network (RPN) to obtain the fused
feature map $S_{i}$ (the i-th feature maps). Finally, The point head uses
interpolated features computed from the fine-grained feature of the CNN
feature maps ($S_{i}$) and the coarse prediction mask for subdivision
prediction.

###  2.3 MLP-Based Models

With the success of the transformer in the field of natural language
processing, some researchers have started exploring how to apply the
transformer to the field of vision. ViT [22] pioneered the approach of
segmenting images into non-overlapping blocks as tokens for operations,
enabling the use of the transformer framework for image processing. Swin
transformer [23] proposed a local group self-attention mechanism incorporating
locality. Inspired by the elegant structure of ViT, MLP with simpler network
structures has also received a lot of attention in the field of computer
vision. MLP-Mixer [7] developed by Google, replaces the self-attention
mechanism with MLPs to construct a pure MLP architecture. The ResMLP [8] used
cross-patch and cross-channel sublayers as components. However, most existing
MLP-based methods cannot adapt to image resolution, making them difficult to
apply to downstream tasks. ASMLP [24] and CycleMLP [25], both of which use the
motion of feature maps to integrate local information, enabling the pure MLP
architecture to be used for downstream tasks. These MLP-based models have a
similar overall structure but differ in the detailed design of the main
modules. After MLP-Mixer [7] was proposed, people began to explore the
architecture of MLP. With increased computational capacity and the
availability of larger datasets coupled, the ancient simple structure of MLP
can also achieve effective performance improvement in various vision tasks.
Through the design structure, these MLP-based models no longer rely on prior
knowledge of two-dimensional images, but on the input feature maps for long-
range interactions. However, this rarely makes full use of local information,
and the extraction of low-level features is also significant for many tasks.

##  3 Method

In this section, we initially introduce the overall structure of the network.
On this basis, we introduce the StoneMLP model as the backbone, then describe
a lightweight SparseFPN network and improve the detection head.

###  3.1 Overall Framework

In the ore image segmentation task, there is a problem of edge blur caused by
the adhesiveness of ore images. Therefore, we propose a lightweight instance
segmentation model with a point-based prediction strategy to address the edge
problem, which utilizes a lightweight backbone and feature pyramid network
(FPN) structure based on MLP to extract high-quality edge features. To further
enhance the accuracy of edge prediction, we introduce an Edge Guidance Loss
that dynamically aligns the predicted edge with the instance edge.

The OreNeXt is a two-stage instance segmentation model with a network
structure as mentioned in Fig. 2. The model mainly consists of a lightweight
MLP backbone, a SparseFPN, and a fine detection head. To better capture the
low-level features, we designed a structure StoneMLP for extracting features
of ore images as the backbone. By moving the feature information axially,
information flows in different directions can be obtained, which helps to
capture the local correlation between overlapping ores. The SparseFPN is used
to extract hierarchical features. The feature maps are processed in the
SparseMLP to efficiently obtain the global receptive field. Global information
can assist in better describing semantic information. Furthermore, we use a
lightweight segmentation head to generate coarse prediction features for each
detected object. For the fine-grained feature from the CNN feature
maps($S_{i}$), we perform bilinear interpolation by coordinate points on the
feature maps to compute the feature vector. Then the fine-grained features
that provide object detail information are combined with the coarse prediction
features that provide global context information. The fused feature map is fed
into the prediction head, making point-wise segmentation.

###  3.2 StoneMLP

We propose a backbone network StoneMLP based on the architecture of MLP.
Compared to convolution and attention networks, the MLP structure has a lower
inductive bias. It allows the model to learn solely from raw data, thereby
making the network more concise. We propose a spatial shift method that
enables the network to obtain different directions of information flow,
achieving the same field of view as CNN and completing feature extraction.
Instance segmentation of ore images necessitates the utilization of low-level
features, including contours, textures, colors, and shapes. Our proposed
spatial movement method effectively captures local correlations by
prioritizing the extraction of these low-level features, thereby yielding more
comprehensive edge information.

In Fig. 3, it shows the architecture of our StoneMLP model. For example,
ResMLP [8] has shown that self-attention is not the key factor of transformers
for achieving excellent performance. With further research, many methods prove
that self-attention modules can be replaced by MLP, Convolution, or other
layers, indicating that the success of the transformer may come from the
entire architecture design. Therefore, we refer to Transformer [26] for the
design of the overall structural framework to expect good results. It takes
RGB images $I\in\mathbb{R}^{3\times H\times W}$ as input and then slices them
into patches of $4\times 4$ size. All tokens obtained at this stage are
$48\times\frac{H}{4}\times\frac{W}{4}$. StoneMLP Block contains four fully
connected layers, each corresponding to four channel projections for
communicating specific location information. Channel projections, vertical
movement, and horizontal movement are used to extract features in the Pixel
Shift operation. The temporal and computational expenses associated with the
axial movement are exceedingly minimal. The computational complexity is
$\mathrm{\Omega(StoneMLP)}=4HWC^{2}$.

As illustrated in Fig. 4, we will explain the horizontal movement. The
following channel projection will use the features that were extracted for the
dashed box. The vertical movement operation is very similar to the horizontal
movement. Combining the horizontal and vertical movements, the features in
different positions are realigned on a channel. During the next channel
projection operation, information from both directions is integrated to obtain
the result after local communication. When the input $x$ and the displacement
size are given, the output $Y_{S}$ is obtained.

$Y_{S}=X^{h}\ W_{c}^{h}+X^{v}\ W_{c}^{v},$ (1)

where $W_{c}^{h}$ and $W_{c}^{v}$ denote the channel projection’s learnable
weights in the vertical and horizontal directions,
$X^{h}=Concat(X_{1}^{h},....,X_{s}^{h})$ and
$X^{v}=Concat(X_{1}^{v},....,X_{s}^{v})$ represent the axial displacement.

Figure 3: Architecture of StoneMLP. The proposed StoneMLP block mainly
includes Norm, Pixel Shift operation, MLP, channel projection, and residual
connection.

(a) Horizontal shift

(b) Vertical shift

Figure 4: The horizontal shift and vertical shift, where the arrows indicate
the steps, and the number in each box is the index of the feature.

###  3.3 SparseFPN

We propose a new SparseFPN for improving inference speed while maintaining
accuracy. Traditional FPN structures generate multi-scale integrated features,
due to the multi-level feature fusion, small targets are easily missed or
misidentified. Therefore, we append two SparseMLP modules after traditional
FPN and use a $1\times 1$ convolution to output features. MLP can deal with
more complex nonlinear relationships and learn more abstract feature
representations.

The structure of SparseMLP is demonstrated in Fig. 5(a), which is a module
based on MLP with a parallel structure composed of three parts: W channel
mapping, H channel mapping, and identity mapping. In the horizontal mixing
path, features are reshaped and mixed information for each row. In the
vertical mixing path, similar operations are applied. The three parallel
branches are connected by channels, and feature fusion is achieved by weighted
summation and $1\times 1$ convolution, producing an output tensor
$X^{out}=FC(concat(X_{H}\ ,\ X_{W}\ ,\ X))$ with the same dimensions as the
input tensor. SparseMLP modules avoid the common overfitting problem affecting
MLP-based models’ performance. Each token only interacts directly with tokens
in the same row or column, and each row and column can share the same
projection weight. As shown in Fig. 5(b), a layer of SparseMLP will form a
cross-shaped receptive field, and after two SparseMLPs, a global receptive
field can be formed. That is to say, if this module is repeated twice, each
token can accumulate information throughout the two-dimensional space,
improving accuracy while reducing computational complexity
$\mathrm{\Omega(SparseMLP)}=HWC(H+W)+3HWC^{2}$.

(a) SparseMLP

(b) Receptive Field

Figure 5: The SparseMLP block consists of three branches: two are responsible
for mixing information along the horizontal and vertical directions,
respectively, and the other path is a constant mapping. (b) shows the
receptive fields generated by two consecutive SparseMLPs.

###  3.4 Edge Guidance Loss

To address the uncertainty resulting from the blurriness of edges in ore
images, we use PointRend [9] as the baseline and perform a recomputation of
boundary hard pixels. We adopt the point selection and point-wise
representation strategies to prioritize uncertain edge points for optimization
of image segmentation for object edges. Additionally, we propose the Edge
Guidance Loss function to improve the model performance. The loss can be
divided into the following four parts:

$L_{EG}={\underbrace{L_{cls}^{b}\mathrm{+}\ \alpha
L_{ploc}^{b}}_{L_{det}}}\mathrm{+}{\underbrace{\beta L_{coarse}^{m}\mathrm{+}\
L_{pmat}^{m}}_{L_{mask}}}.$ (2)

where $\alpha$ and $\beta$ represent the weight of $L_{ploc}^{b}$ and
$L_{coarse}^{m}$, which is generally set to 0.5 and 1.0 respectively in the
experiments. To improve the success rate of edge detection and segmentation,
in addition to the classification loss and the coarse mask loss, we also
introduce an Edge point Guidance match Loss and improve the target box offset
loss to improve the positioning accuracy.

In our method, $L_{cls}^{b}$ and $L_{coarse}^{m}$ are defined as a standard
cross entropy loss function with softmax activation. We treat the foreground
points as positive samples and the other points as negative samples. The new
regression localization loss $L_{ploc}^{b}$ is responsible for predicting the
foreground score of the prediction point:

$L_{ploc}^{b}(R,R^{\prime})=\frac{1}{n}\sum\limits_{k=1}^{n}\parallel(x_{i},y_{i})-(x_{i}^{\prime},y_{i}^{\prime})\parallel_{2}.$
(3)

It uses the point-to-point distance prediction, input predicted point, and
real border box. Moreover, it sorts the coordinates of test points in the x or
y direction and obtains new coordinates. The distance between each point and
four borders is calculated, and the boundary loss of each border point is
obtained, which is finally obtained by weighted addition.

Figure 6: Edge point Guidance match Loss. The yellow points represent
predicted vertices, while the blue points indicate labeled vertices. The
arrows depict the path, representing the pairing relationship. Each prediction
point is adjusted to the nearest point on the truth boundary.

The edge guidance match loss $L_{pmat}^{m}$ is the average of the traditional
cross-entropy loss $L_{pcls}$ function and the edge point guidance match loss
$L_{pg}$:

$L_{pmat}^{m}=\frac{1}{2}(L_{pcls}+L_{pg}).$ (4)

In Fig. 6, the $L_{pg}$ first calculates the distance between the prediction
point $Pred_{i}^{in}$ and the real border point, finds the nearest target
point $gt_{X}^{ipt}$ to the prediction point according to the distance, and
interpolates the target points to a specified number of points using an
interpolation function to obtain the offset difference $X_{i}^{*}$. It also
increases the classification accuracy of the target and makes the target
smoother.

$X_{i}^{*}=arg\min_{X}\parallel{Pred}_{i}^{in}-{gt}_{X}^{ipt}\parallel_{2}.$
(5)

Then, the predicted points ${Pred}_{i}^{out}$ and the matched nearest
interpolation label points ${gt}_{X_{i}^{*}}^{ipt}$ are dynamically matched
using the loss function smoothL1 to supervise the quality of boundary
prediction. The edge point guidance match loss $L_{pg}$ can consider the
smoothness and robustness of large error and small error, which is not
affected by outliers.

$L_{pg}=\frac{1}{N}\sum_{k=1}^{n}\parallel{Pred}_{i}^{out}-{gt}_{X_{i}^{*}}^{ipt}\parallel_{1}.$
(6)

##  4 Experiments

In this section, we introduce the hardware environment for ore processing
tasks and the datasets, evaluation metrics, and implementation details. Next,
we conducted ablation research to evaluate the effectiveness of the design
decisions. Eventually, we compare OreNeXt with the state-of-the-art methods
for instance segmentation on the ore dataset.

###  4.1 Hardware Environment

The actual scene of the ore processing tasks is depicted in Fig. 7, revealing
significant adhesion of the ore and blurred edges. The detection system is
responsible for identifying the size indexes of ores on the conveyor belt,
providing insights into the effectiveness of the upstream processes, and
facilitating the identification of over-specification ores. Ore detection on
the production line is performed through sampling inspection. The traditional
manual screening involves the random selection of a set of ores as a sample to
assess their compliance with required standards. The online detection method
based on deep learning considers the complete ores present on the surface of
the conveyor belt, as captured by an industrial camera, as a representative
sample, so we only expect to segment independent and complete ore individuals
to obtain their particle size information. Therefore, precise segmentation of
the surface ores is crucial, particularly in accurately identifying the edges
of overlapping ores. It can be seen from Fig. 7 that ore processing tasks are
typically conducted outdoors, where hardware resources are limited, making the
use of high-performance computing equipment impractical. Consequently, the
model size becomes a critical factor in ore processing scenarios with
constrained computing resources. We verify on the ORE dataset that compared
with the current segmentation methods, our method achieves optimal accuracy
while ensuring minimal memory usage.

(a) Production site

(b) Conveyor belt

Figure 7: Conveyor belt at the mine production site. The site environment is
simple and the computing resources are poor. There is serious adhesion between
the ores and the edges are blurred.

###  4.2 Experiments Setup

####  4.2.1 Datasets

In the experiment, we apply ImageNet [27] to pre-train the backbone model and
train instance segmentation models using our created rock data set. For the
ORE image dataset, we use 4,060 images for training and 1,060 images for
validation. We supplement the data set in the following ways: first, we
collect different scales of rock images using our experimental platform. Then,
we change the positioning of rocks at different scales, such as sparse, thick,
and so on. In addition, We break up large images into smaller ones that can be
utilized to train networks. We segment images using a sliding window to
further expand the datasets.

####  4.2.2 Evaluation Metrics

To confirm the efficiency of the suggested method, seven evaluation indicators
are used such as $AP^{box}$, $AP_{50}^{box}$, $AP^{mask}$, $AP_{50}^{mask}$,
inference time (FPS), inference memory consumption, and model size. The first
four indicators are used to evaluate the accuracy. AP stands for Average
Precision, and the number after the AP represents the IoU(Intersection over
Union) threshold, which refers to the degree of overlap between the predicted
box and the ground truth box. In general, a higher value of $AP_{50}$
indicates a better performance of the algorithm. The latter three metrics
evaluate whether a model can be deployed on low-cost hardware. The
computational cost of the model is quantified by its inference time, and we
aim to maximize this value. The utilization of memory during training and the
model’s size demonstrate its reliance on the hardware system, and we strive to
minimize these factors while maintaining accuracy.

####  4.2.3 Implementation Details

OreNeXt is trained via back-propagation and the SGD (stochastic gradient
descent) optimizer. For the ORE image dataset, all models shown in Table 1 are
trained for 12k iterations on an NVIDIA RTX2080Ti GPU. We set the initial
learning rate to 0.001 cut it by 1/10 at 8k iterations and use a batch size of
8 for all ablation studies. The input images are changed during training to
have short sides between 160 and 320 pixels and long sides of 320 pixels. The
experiments on the MS COCO dataset, the pre-train of the StoneMLP, and SAM-
related experiments are all trained on the NVIDIA RTX3090Ti GPU. To ensure
fairness and reproducibility in the experiment, a fixed random seed is used,
and the hyperparameters of all models are set according to the best values
provided in their respective papers. The parameters that achieve the highest
accuracy on the validation set are preserved for prediction.

###  4.3 Comparison with State-of-the-Art Methods

####  4.3.1 Classic Segmentation

To further demonstrate the supremacy of OreNeXt, we conduct a comparative
analysis with state-of-the-art methods on the ORE dataset. Specifically, we
compare the segmentation performance of a typical two-stage segmentation
framework utilizing various CNN, Transformer, and MLP backbone networks. The
results can be found in Table 1, which compares the segmentation accuracy,
inference time, model size, and GPU memory usage.

Table 1:  Comparison of accuracy with the state-of-the-art methods on ore image datasets. T-based: Transformer-based. Method | Backbone | $AP^{box}\uparrow$ | $AP^{box}_{50}\uparrow$ | $AP^{mask}\uparrow$ | $AP^{mask}_{50}\uparrow$ | FPS$\uparrow$ |  Model Size (MB) $\downarrow$ |  Inf.Memory (MB) $\downarrow$
---|---|---|---|---|---|---|---|---
_CNN-based_ | Mask RCNN[18] | ResNet101 | 43.0 | 51.7 | 12.1 | 22.1 | 13 | 480 | 1876
PointRend[9] | ResNet101 | 37.4 | 54.0 | 21.0 | 43.8 | 17 | 489 | 3361
YOLACT[16] | ResNet101 | 40.3 | 50.6 | 7.1 | 11.9 | 24 | 410 | 7758
Mask RCNN[18] | ResNet50 | 42.8 | 51.7 | 12.4 | 22.6 | 17 | 334 | 1769
PointRend[9] | ResNet50 | 36.3 | 54.0 | 25.8 | 47.7 | 21 | 344 | 2677
Cascade Mask RCNN[21] | ResNet50 | 41.6 | 51.7 | 36.4 | 48.3 | 16 | 587 | 1947
CondInst[28] | ResNet50 | 43.1 | 52.2 | 39.0 | 48.7 | 23 | 259 | 1713
MS RCNN[29] | ResNet50 | 42.7 | 51.7 | 14.7 | 14.7 | 16 | 428 | 1749
Blend Mask[17] | ResNet50 | 44.1 | 58.1 | 38.9 | 48.7 | 23 | 274 | 1641
BoxInst[30] | ResNet50 | 44.3 | 56.9 | 34.8 | 48.7 | 23 | 261 | 1681
SparseInst[31] | ResNet50 | - | - | 24.8 | 37.9 | 21 | 380 | 1711
RTMDet[32] | CSPNeXt | 38.7 | 49.7 | 35.1 | 43.4 | 22 | 109 | 2013
_T-based_ | Mask RCNN[18] | Swin-T | 38.3 | 52.0 | 27.7 | 46.5 | 19 | 542 | 1995
PointRend[9] | Swin-T | 30.6 | 55.4 | 32.8 | 44.7 | 16 | 371 | 2631
Cascade Mask RCNN[21] | Swin-T | 42.0 | 50.7 | 38.5 | 48.5 | 14 | 920 | 2063
Mask2Former[33] | Swin-T | 55.7 | 71.3 | 35.3 | 44.3 | 20 | 572 | 6098
Mask RCNN[18] | PVTv2 | 39.6 | 51.4 | 27.3 | 44.8 | 24 | 263 | 1943
PointRend[9] | PVTv2 | 39.8 | 51.8 | 16.1 | 43.0 | 19 | 185 | 2501
| Cascade Mask RCNN[21] | PVTv2 | 42.7 | 50.9 | 39.0 | 48.7 | 17 | 641 | 1907
_MLP-based_ | Mask RCNN[18] | ASMLP | 35.9 | 49.2 | 21.1 | 38.5 | 21 | 542 | 1811
PointRend[9] | ASMLP | 36.6 | 52.6 | 27.0 | 41.1 | 20 | 371 | 2389
Cascade Mask RCNN[21] | ASMLP | 42.4 | 50.7 | 38.7 | 48.5 | 19 | 613 | 2043
Mask RCNN[18] | CycleMLP-B1 | 36.9 | 51.7 | 29.1 | 45.2 | 26 | 104 | 1673
PointRend[9] | CycleMLP-B1 | 25.0 | 50.2 | 20.0 | 39.8 | 26 | 109 | 1843
OreNeXt(Ours) | StoneMLP | 38.2 | 60.4 | 33.8 | 48.9 | 27 | 73 | 1931

Firstly, the networks based on CNN have higher accuracy but are slower and
have larger model sizes than ours. Since the transformer structure has the
largest model capacity, the transformer-based networks have a larger model
size, while the MLP-based networks have a smaller model size but worse
accuracy. Secondly, Our proposed OreNext network exceeds the best CNN and
Transformer algorithms both by 0.2 on $AP_{50}^{mask}$ metrics, while also
achieving high accuracy on $AP_{50}^{box}$, due to our novel approach of using
a Sparse FPN that balances local and global information with two MLP
structures, as well as an Edge Guidance Loss based on a point processing
strategy to improve accuracy. Recent methods like Mask2Former [33] based on
SwinTransformer [23] have achieved impressive results on the $AP^{box}$
metric, surpassing the average performance, while its corresponding model size
is $6\times$ larger than our OreNeXt. A small model size is crucial in ore
processing tasks with limited computational resources. Therefore, models that
prioritize box accuracy at the expense of model size are not suitable for
field ore tasks. Furthermore, OreNeXt also outperforms other lightweight
frameworks significantly in inference time, model size, and GPU memory usage,
thanks to our lightweight backbone StoneMLP, which is even smaller than other
lightweight MLP-based backbone networks such as CycleMLP. OreNeXt strikes a
balance between accuracy and computational cost and is more suitable for ore
segmentation tasks. Finally, the performance of Swin-T and ASMLP used as the
backbone network of PointRend is similar to that of OreNext in terms of
accuracy and model size, which supports the idea that the token mixer module
of the transformer model is not the key to its excellent performance but the
entire transformer architecture. Therefore, replacing the most time-consuming
attention of the transformer with the StoneMLP network can also achieve
excellent performance.

####  4.3.2 Segment Anything Model

Table 2: The comparison of OreNeXt with SAM-based instance segmentation
methods, $\star$ means only validation

Method | ${AP^{box}_{50}}$ | ${AP^{mask}_{50}}$ | FPS |  Model Size (MB)
---|---|---|---|---
SAM[34] | 42.2 | 44.7 | 3 | 609
RSPrompter[35] | 42.7 | 41.6 | 2 | 631
$FastSAM^{\star}$[36] | 38.0 | 37.2 | 7 | 150
OreNeXt(Ours) | 60.4 | 48.9 | 27 | 73

By finding foundational models that exhibit high performance in the fields of
NLP and CV, the foundation Segment Anything Model (SAM) [34] proposed by Meta
AI exhibits remarkable generalization capabilities. SAM is becoming a
foundation step for many visual tasks, like image editing and remote sensing
image segmentation. In the ore processing task, we compare the existing
instance segmentation methods based on the SAM foundation model with our
method, as shown in Table 2. As a category-agnostic instance segmentation
method, SAM does not show its superiority in single-class ore image
segmentation. Its segmentation accuracy is average, but the model size exceeds
most traditional segmentation methods. In single-task industrial application
scenarios, SAM not only fails to play its advantages of strong generalization
ability, but its huge computation costs prevent it from wider applications in
industry scenarios. At present, the traditional instance segmentation method
is still the best choice after balancing accuracy and computational complexity
in industrial applications. In the future, the goal is to achieve high-
performance compression of large models and develop comprehensive industrial
foundation modes capable of addressing multiple scenarios and tasks.

The visualization of various results from the instance segmentation method in
ore images is depicted in Fig. 8. It can be observed that our method is
significantly better than other methods in mask quality as well as edge
sharpness, and the missed detection rate is also at a low level. Compared to
the foundation large model, our method can detect as many complete ore images
as possible. It can obtain the granularity information required for industrial
production and meet the needs of sampling inspection.

(a) Input

(b) GroundTruth

(c) SagmentAnything

(d) PointRend

(e) YOLACT

(f) OreNeXt

Figure 8: The visual results of ore images processed by different methods. It
can be seen that our method (f) has the best mask quality, and the foundation
model (c) gets too much information about incomplete ores, which is unsuitable
for actual production.

###  4.4 Ablation Study

####  4.4.1 StoneMLP

To demonstrate the effectiveness of our proposed backbone network, we verified
the accuracy of this backbone network by comparing various backbone changes.
In Table 3, StoneMLP outperforms the other backbone networks in three
indicators of $AP^{box}$, $AP^{mask}$, $AP_{50}^{mask}$. The index
$AP_{50}^{box}$ of evaluating detection bounding box is 60.4, which is
slightly lower than the highest value of ResNet101 in 61.9, while the model
size of 73M is 6.5$\times$ smaller than ResNet101. The small model size is
also vital in ore processing tasks with limited computational resources. Since
StoneMLP is designed for feature extraction of ore low-level edge information,
it greatly outperforms the lightweight backbone CycleMLP with 99M, which is
also based on MLP in terms of accuracy across the board. In order to verify
the general effectiveness of StoneMLP, we conduct experiments on the MS COCO
[6] dataset, and the results in Table 4 show that the accuracy is
significantly improved compared with the baseline.

Table 3: The segmentation results of different backbones in OreNeXt

Backbone | ${AP^{box}}$ | ${AP^{box}_{50}}$ | ${AP^{mask}}$ | ${AP^{mask}_{50}}$ |  Model Size (MB)
---|---|---|---|---|---
ResNet50[37] | 36.8 | 60.8 | 31.1 | 47.4 | 329
ResNet101[37] | 34.0 | 61.9 | 29.9 | 48.3 | 474
RegNet[38] | 36.1 | 57.1 | 29.2 | 47.4 | 255
HRNet[39] | 36.7 | 61.8 | 25.7 | 25.1 | 229
PVTv2[40] | 36.1 | 61.3 | 22.5 | 47.7 | 178
Swin-T[23] | 30.3 | 61.2 | 11.8 | 25.8 | 220
ASMLP[24] | 36.4 | 58.9 | 30.5 | 46.8 | 214
CycleMLP-B1[25] | 26.6 | 59.3 | 22.4 | 43.8 | 99
StoneMLP(ours) | 38.2 | 60.4 | 33.8 | 48.9 | 73

Table 4: Comparison crucial components on MS COCO datasets

| ${AP^{box}}$ | ${AP^{box}_{50}}$ | ${AP^{mask}}$ | ${AP^{mask}_{50}}$
---|---|---|---|---
Baseline | 21.6 | 37.4 | 20.3 | 34.5
EGLoss | $19.9_{~{}{\color[rgb]{0,0,1}\tiny-1.7}}$ | $39.3_{~{}{\color[rgb]{1,0,0}\tiny+1.9}}$ | $21.1_{~{}{\color[rgb]{1,0,0}\tiny+0.8}}$ | $36.8_{~{}{\color[rgb]{1,0,0}\tiny+2.3}}$
StoneMLP | $25.1_{~{}{\color[rgb]{1,0,0}\tiny+3.5}}$ | $41.2_{~{}{\color[rgb]{1,0,0}\tiny+3.8}}$ | $23.6_{~{}{\color[rgb]{1,0,0}\tiny+3.3}}$ | $38.8_{~{}{\color[rgb]{1,0,0}\tiny+4.3}}$

Table 5: Comparison of using different MLP in SparseFPN

Method | ${AP^{box}_{50}}$ | ${AP^{mask}_{50}}$ | FPS |  Model Size (MB)
---|---|---|---|---
MLP[23] | 51.6 | 39.5 | 25 | 84
CycleMLP[25] | 50.5 | 37.7 | 23 | 83
SparseMLP(ours) | 53.0 | 48.5 | 27 | 77

####  4.4.2 SparseFPN

To improve detection accuracy while maintaining light weight, we designed an
MLP module SparseMLP to enhance the receptive field by mixing information and
fusing features through channel mapping. The module can be implemented using a
simple MLP structure. In Table 5, we compare the effectiveness of different
MLP structures. Experimental results show that connecting two SparseMLP after
FPN can improve the receptive field of images, reduce the parameter count to
avoid overfitting, and promote multi-stage processing in the pyramid
architecture. The SparseFPN can strengthen the small object information that
is easy to misjudge while maintaining sufficient semantic information.

####  4.4.3 Loss Function

To verify the robustness of EGLoss to the network, we compared the baseline
loss with the newly proposed edge guidance loss and found that the
$AP_{50}^{box}$ and $AP_{50}^{mask}$ values were significantly improved while
the model size remained constant. Additionally, we compared the accuracy
stability before and after replacing the loss and observed that the
$AP_{50}^{mask}$ value became relatively stable after replacing the loss. In
Table 4, experiments on the public dataset MS COCO[6] show that EGLoss
achieves good results on $AP_{50}^{mask}$, which is an indicator for
evaluating mask quality.

Table 6: Effects of the components of the proposed method. SM: StoneMLP, SF: SparseFPN, EL:EGLoss Baseline | SM | SF | EL | ${AP^{box}_{50}}$ | ${AP^{mask}_{50}}$ |  Model Size (MB)
---|---|---|---|---|---|---
✓ |  |  |  | 54 | 47.7 | 344
✓ | ✓ |  |  | 51.7 | 47.8 | 85
✓ | ✓ | ✓ |  | 53.9 | 48.4 | 77
✓ | ✓ | ✓ | ✓ | 60.4 | 48.9 | 73

####  4.4.4 Module Breakdown Analysis

We conducted ablation studies, as presented in Table 6, to investigate the
specific contributions of each module in OreNeXt. StoneMLP incorporates
shifting operations to capture local information, resulting in a slight
performance improvement while significantly reducing the number of parameters.
The model size decreases from 344M to 95M. Next, the SparseMLP blocks are
introduced into FPN, helping extract small features and improve accuracy.
Finally, the Loss function is replaced with the EGLoss, leading to a
significant improvement in accuracy from 53.9 to 60.4 in the $AP^{box}_{50}$
and from 48.4 to 48.9 in the $AP^{mask}_{50}$.

####  4.4.5 Hyperparameter

To explore the influence of hyperparameter settings on network performance, we
conducted ablation experiments on the weight of the loss function and the
number of channels in the neck. We have reduced the model size in SpraseFPN by
changing the convolution size and reducing the number of channels. The results
are depicted in Table 7, and the model size gradually lowers as the number of
channels reduces from 256 to 32. The highest value of accuracy is achieved
when the number of channels is 64, which are 54.7 in $AP_{50}^{box}$ and 47.6
in $AP_{50}^{mask}$ respectively. The channels of the feature map contain the
feature information, and the number of channels should be reasonably designed
according to the feature complexity of the detected object. Reducing the
number of channels can improve accuracy due to the ORE dataset having fewer
shots and features than the public dataset. When the number of channels is
reduced from 64 to 32, the accuracy does not increase, which proves that the
feature information will also be lost when the number of channels is too
small. A channel number of 64 increases the model size almost negligible
compared to a channel number of 32. These results indicate that a channel of
64 and $1\times 1$ convolution is the optimal choice for ore segmentation.

Parameter $\beta$ in (2) reflects the significance of $L_{coarse}^{m}$ in
training. We perform ablation experiments on the MS COCO dataset and ORE
dataset, respectively, and the results are reflected in Table 8. It can be
seen that a smaller $\beta$ value can achieve the highest detection accuracy
due to the single category of the ore dataset, while for the MS COCO dataset
with rich target categories, a larger weight is needed to make the model not
ignore the sample features of some categories.

Table 7: Comparisons of different channels in SparseFPN

Channels | Convs. | ${AP^{box}_{50}}$ | ${AP^{mask}_{50}}$ | FPS |  Model Size (MB)
---|---|---|---|---|---
32 | 1x1 | 52.2 | 46.1 | 25 | 77
| 3x3 | 53.6 | 42.6 | 24 | 79
64 | 1x1 | 54.7 | 47.6 | 26 | 80
| 3x3 | 52.6 | 46.7 | 24 | 88
96 | 1x1 | 53.2 | 45.6 | 25 | 90
| 3x3 | 52.9 | 44.3 | 24 | 98
128 | 1x1 | 52.2 | 46.4 | 26 | 98
| 3x3 | 54.6 | 46.5 | 23 | 109
256 | 1x1 | 52.0 | 44.9 | 23 | 130
| 3x3 | 52.7 | 44.3 | 22 | 161

Table 8: Validation of $L_{coarse}^{m}$ in the different datasets. NC means cannot converge | ORE | MS COCO
---|---|---
$\beta$ | 0.5 | 1 | 1.5 | 1.5 | 2.5 | 3
${AP^{mask}_{50}}$ | 45.5 | 48.9 | 46.4 | 30.8 | 36.8 | NC

##  5 Conclusion

This study presents OreNeXt, a lightweight and efficient instance segmentation
network designed for ore image segmentation. The network specifically
addresses the challenging issue of edge blurring that results from ore
overlap. OreNeXt is an MLP-based architecture that leverages two essential
components to enhance low-level edge features. Firstly, it incorporates a
StoneMLP backbone network structure that facilitates spatial information
interaction through shift operations. Secondly, it incorporates a SparseFPN
that effectively balances global and local information to improve accuracy.
Additionally, a point-guided loss function is employed to enhance the clarity
of edge segmentation. When assessed using the ore dataset, our model
demonstrated superior performance in ore segmentation compared to other
existing methods while also exhibiting faster inference time, decreased
computational complexity, and a reduction in the number of parameters
required. For future work, we will focus on further improving edge accuracy
and increasing the model’s speed. We will also explore edge segmentation
algorithms that can be extended to other industrial areas.

## References

  * [1] Y. Zhang, Y. Zhou, H. Pan, B. Wu, and G. Sun, “Visual fault detection of multiscale key components in freight trains,” IEEE Trans. Ind. Informat., vol. 19, no. 8, pp. 9082–9090, 2023.
  * [2] S. Xia, L. Chu, L. Pei, W. Yu, and R. C. Qiu, “A boundary consistency-aware multitask learning framework for joint activity segmentation and recognition with wearable sensors,” IEEE Trans. Ind. Informat., vol. 19, no. 3, pp. 2984–2996, 2023.
  * [3] Y. Zhang, M. Liu, Y. Yang, Y. Guo, and H. Zhang, “A unified light framework for real-time fault detection of freight train images,” IEEE Trans. Ind. Informat., vol. 17, no. 11, pp. 7423–7432, 2021.
  * [4] J. Chu, Z. Guo, and L. Leng, “Object detection based on multi-layer convolution feature fusion and online hard example mining,” IEEE Access, vol. 6, pp. 19959–19967, 2018.
  * [5] W. Lin, J. Chu, L. Leng, J. Miao, and L. Wang, “Feature disentanglement in one-stage object detection,” Pattern Recognition, vol. 145, p. 109878, 2024\.
  * [6] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Eur. Conf. Comput. Vis., pp. 740–755, 2014.
  * [7] I. O. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit, M. Lucic, and A. Dosovitskiy, “Mlp-mixer: An all-mlp architecture for vision,” in Neural Information Processing Systems, vol. 34, pp. 24261–24272, 2021.
  * [8] H. Touvron, P. Bojanowski, M. Caron, M. Cord, A. El-Nouby, E. Grave, G. Izacard, A. Joulin, G. Synnaeve, J. Verbeek, and H. Jegou, “Resmlp: Feedforward networks for image classification with data-efficient training,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 4, pp. 5314–5321, 2023.
  * [9] A. Kirillov, Y. Wu, K. He, and R. Girshick, “Pointrend: Image segmentation as rendering,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 9796–9805, 2020.
  * [10] G. Sun, D. Huang, L. Cheng, J. Jia, C. Xiong, and Y. Zhang, “Efficient and lightweight framework for real-time ore image segmentation based on deep learning,” Minerals, vol. 12, no. 5, pp. 1–18, 2022.
  * [11] D. P. Mukherjee, Y. Potapovich, I. Levner, and H. Zhang, “Ore image segmentation by learning image and shape features,” Pattern Recognition Letters, vol. 30, no. 6, pp. 615–622, 2009.
  * [12] Y. Zhang, L. Cheng, Y. Peng, C. Xu, Y. Fu, B. Wu, and G. Sun, “Faster orefsdet: A lightweight and effective few-shot object detector for ore images,” Pattern Recognition, vol. 141, p. 109664, 2023.
  * [13] H. Li, C. Pan, Z. Chen, A. Wulamu, and A. Yang, “Ore image segmentation method based on u-net and watershed,” Computers, Materials and Continua, vol. 65, no. 1, pp. 563–578, 2020.
  * [14] J. Liu, Z. Jiang, W. Gui, and Z. Chen, “A novel particle size detection system based on rgb-laser fusion segmentation with feature dual-recalibration for blast furnace materials,” IEEE Transactions on Industrial Electronics, vol. 70, no. 10, pp. 10690–10699, 2023.
  * [15] Y. Liu, Z. Zhang, X. Liu, W. Lei, and X. Xia, “Deep learning based mineral image classification combined with visual attention mechanism,” IEEE Access, vol. 9, pp. 98091–98109, 2021.
  * [16] D. Bolya, C. Zhou, F. Xiao, and Y. J. Lee, “YOLACT: real-time instance segmentation,” in IEEE/CVF Int. Conf. Comput. Vis., pp. 9156–9165, 2019\.
  * [17] H. Chen, K. Sun, Z. Tian, C. Shen, Y. Huang, and Y. Yan, “Blendmask: Top-down meets bottom-up for instance segmentation,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 8570–8578, 2020.
  * [18] K. He, G. Gkioxari, P. DollÃ¡r, and R. Girshick, “Mask r-cnn,” in IEEE/CVF Int. Conf. Comput. Vis., pp. 2980–2988, 2017.
  * [19] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1137–1149, 2017.
  * [20] Y. Zhang, J. Chu, L. Leng, and J. Miao, “Mask-refined r-cnn: A network for refining object details in instance segmentation,” Sensors, vol. 20, no. 4, 2020.
  * [21] Z. Cai and N. Vasconcelos, “Cascade R-CNN: high quality object detection and instance segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 5, pp. 1483–1498, 2021.
  * [22] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations, pp. 1–21, 2021.
  * [23] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in IEEE/CVF Int. Conf. Comput. Vis., pp. 9992–10002, 2021.
  * [24] D. Lian, Z. Yu, X. Sun, and S. Gao, “AS-MLP: An axial shifted MLP architecture for vision,” in International Conference on Learning Representations, pp. 1–19, 2022.
  * [25] S. Chen, E. Xie, C. Ge, R. Chen, D. Liang, and P. Luo, “Cyclemlp: A mlp-like architecture for dense visual predictions,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 12, pp. 14284–14300, 2023.
  * [26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Neural Information Processing Systems, pp. 6000–6010, 2017.
  * [27] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 248–255, 2009.
  * [28] Z. Tian, C. Shen, and H. Chen, “Conditional convolutions for instance segmentation,” in Eur. Conf. Comput. Vis., pp. 282–298, 2020.
  * [29] Z. Huang, L. Huang, Y. Gong, C. Huang, and X. Wang, “Mask scoring R-CNN,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 6402 – 6411, 2019\.
  * [30] Z. Tian, C. Shen, X. Wang, and H. Chen, “Boxinst: High-performance instance segmentation with box annotations,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 5439–5448, 2021.
  * [31] T. Cheng, X. Wang, S. Chen, W. Zhang, Q. Zhang, C. Huang, Z. Zhang, and W. Liu, “Sparse instance activation for real-time instance segmentation,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 4423–4432, 2022.
  * [32] C. Lyu, W. Zhang, H. Huang, Y. Zhou, Y. Wang, Y. Liu, S. Zhang, and K. Chen, “Rtmdet: An empirical study of designing real-time object detectors,” arXiv:2212.0778, 2022.
  * [33] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar, “Masked-attention mask transformer for universal image segmentation,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 1280–1289, 2022.
  * [34] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollar, and R. Girshick, “Segment anything,” in IEEE/CVF Int. Conf. Comput. Vis., pp. 4015–4026, October 2023.
  * [35] K. Chen, C. Liu, H. Chen, H. Zhang, W. Li, Z. Zou, and Z. X. Shi, “Rsprompter: Learning to prompt for remote sensing instance segmentation based on visual foundation model,” ArXiv:2306.12156, 2023.
  * [36] X. Zhao, W. Ding, Y. An, Y. Du, T. Yu, M. Li, M. Tang, and J. Wang, “Fast segment anything,” arXiv:2306.12156, 2023.
  * [37] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 770–778, 2016.
  * [38] I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. DollÃ¡r, “Designing Network Design Spaces,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 10425–10433, 2020.
  * [39] K. Sun, B. Xiao, D. Liu, and J. Wang, “Deep high-resolution representation learning for human pose estimation,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 5686–5696, 2019.
  * [40] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in IEEE/CVF Int. Conf. Comput. Vis., pp. 548–558, 2021.


# All-optical control of excitons in semiconductor quantum wells

V. M. Kovalev  M. V. Boev  O. V. Kibis<EMAIL_ADDRESS>Department of
Applied and Theoretical Physics, Novosibirsk State Technical University, Karl
Marx Avenue 20, Novosibirsk 630073, Russia

###### Abstract

Applying the Floquet theory, we developed the method to control excitonic
properties of semiconductor quantum wells by a high-frequency electromagnetic
field. It is demonstrated, particularly, that the field induces the blue shift
of exciton emission from the quantum wells and narrows width of the
corresponding spectral line. As a consequence, the field strongly modifies
optical properties of the quantum wells and, therefore, can be used to tune
characteristics of the optoelectronic devices based on them.

##  I Introduction

The all-optical control of electronic properties of solids by a high-frequency
off-resonant electromagnetic field, based ideologically on the Floquet theory
of periodically driven quantum systems (Floquet engineering), remains an
exciting field of modern physics during long time since it results in many
light-induced effect in various nanostructures Oka_2019 ; Basov_2017 ;
Eckardt_2015 ; Goldman_2014 ; Bukov_2015 ; Casas_2001 ; Kibis_2020_1 ;
Koshelev_2015 ; Rechtsman_2013 ; Wang_2013 ; Torres_2014 ; Calvo_2015 ;
Mikami_2016 ; Oka_2009 ; Iurov_2017 ; Iurov_2013 ; Syzranov_2013 ; Usaj_2014 ;
Perez_2014 ; Glazov_2014 ; Sentef_2015 ; Sie_2015 ; Kibis_2017 ; Iurov_2019 ;
Iurov_2020 ; Cavalleri_2020 . Among the actively studied nanostructures, the
semiconductor quantum wells (QWs) Nag_2002book  take deserved place as
building blocks of modern nanoelectronics. In previous studies, the Floquet
theory was applied to irradiated QWs to describe the electron transport
Morina_2015 , the spin effects Pervishko_2015 , the magnetic properties of
electrons Dini_2016 , polaritons Kyriienko_2017  and the topological states
Lindner_2011 . However, the Floquet theory of excitons in QWs still wait for
detailed analysis. This problem is of current importance since the excitons —
together with the electron scattering and capture processes Zakhleniuk_1999 ;
Stavrou_2001  — form the main mechanism for light emission from semiconductor
structures, which dominates for low temperatures Ivchenko_2007book .
Correspondingly, the Floquet engineering of excitons will open the novel way
to control radiative characteristics of various light-emitting optoelectronic
devices based in the QWs Nag_2002book ; Zory_1993book . The present article is
aimed to fill partially this gap in the theory.

##  II Model

Figure 1: Sketch of the system under consideration: (a) The semiconductor
quantum well (QW) irradiated by the circularly polarized electromagnetic wave
(EMW) with the frequency $\omega_{0}$ and the electric field amplitude
$E_{0}$; (b) The energy band structure of the QW containing the electron-hole
pair (exciton), where $\varepsilon_{c(v)}$ is the band edge of the conduction
(valence) band of the semiconductor.

Let us consider an exciton in a semiconductor QW irradiated by a circularly
polarized electromagnetic wave (EMW) with the frequency $\omega_{0}$ and the
electric field amplitude $E_{0}$, which propagates perpendicularly to the QW
plane $x,y$ (see Fig. 1). In the following, the field frequency $\omega_{0}$
will be assumed to be below the optical absorption threshold of the QW.
Therefore, the EMW can be treated as an off-resonant field which cannot be
absorbed by charge carriers and only “dresses” them (the dressing field),
driving periodically electrons and holes. Correspondingly, the exciton
Hamiltonian for the irradiated QW reads

$\hat{\cal
H}_{X}=\sum_{j=e,h}\frac{[\hat{\mathbf{p}}_{j}-e\xi_{j}\mathbf{A}(t)/c]^{2}}{2m_{j}}+U_{C}(\mathbf{r}_{e},\mathbf{r}_{h}),$
(1)

where $\hat{p}_{e(h)}$ is the in-plane electron (hole) momentum operator,
$m_{e(h)}$ is the electron (hole) effective mass,

$\mathbf{A}(t)=(A_{x},A_{y})=[cE_{0}/\omega_{0}](\sin\omega_{0}t,\cos\omega_{0}t)$
(2)

is the vector potential of the dressing field, $\xi_{e}=1$ and $\xi_{h}=-1$,
$\mathbf{r}_{e(h)}$ is the in-plane radius-vector of electron (hole),

$U_{C}(\mathbf{r}_{e},\mathbf{r}_{h})=-\frac{e^{2}}{\epsilon|\mathbf{r}_{e}-\mathbf{r}_{h}|},$
(3)

is the potential energy of the Coulomb interaction between electron and hole,
and $\epsilon$ is the permittivity. The Schrödinger equation with the
periodically time-dependent Hamiltonian (1) corresponds to the Floquet problem
for an exciton dressed by the field (2). To simplify the problem, let us apply
the the Kramers-Henneberger unitary transformation Kramers ;
henneberger1968perturbation ,

$\hat{\mathcal{\cal U}}(t)=\exp{\left[\frac{i}{\hbar
c}\sum_{j=e,h}\frac{e}{m_{j}}\int^{t}dt^{\prime}\left(\mathbf{A}(t^{\prime})\xi_{j}\hat{\mathbf{p}}_{j}-\frac{eA^{2}(t^{\prime})}{2c}\right)\right]},$
(4)

which removes the coupling of the momentums $\hat{\mathbf{p}}_{e,h}$ to the
vector potential $\mathbf{A}(t)$ in the Hamiltonian (1) and transfers the time
dependence from the kinetic energy of electron and hole to their potential
energy (3) [see Appendix A for details]. Then the transformed Hamiltonian (1)
reads

$\displaystyle\hat{\cal H}$ $\displaystyle=$ $\displaystyle\hat{\cal
U}^{\dagger}(t)\hat{\cal H}_{X}\hat{\cal U}(t)-i\hbar\hat{\cal
U}^{\dagger}(t)\partial_{t}\hat{\cal U}(t)$ (5) $\displaystyle=$
$\displaystyle\sum_{j=e,h}\frac{\hat{\mathbf{p}}_{j}^{2}}{2m_{j}}+U_{C}(\mathbf{r}_{e}-\mathbf{r}_{e}^{\prime}(t),\mathbf{r}_{h}-\mathbf{r}_{h}^{\prime}(t)),$

where

$\mathbf{r}_{e(h)}^{\prime}(t)=\xi_{e(h)}\bar{r}_{e(h)}(\cos\omega_{0}t,-\sin\omega_{0}t)$
(6)

is the radius-vector describing the classical circular trajectory of electron
(hole) in the field (2), and $\bar{r}_{e(h)}=|e|E_{0}/m_{e(h)}\omega_{0}^{2}$
is the radius of the trajectory.

The Hamiltonian (5) is still physically equal to the exact Hamiltonian of
irradiated exciton (1). To proceed, we have to make the approximation
described in Appendix B. Namely, considering the Floquet problem with the
Hamiltonian (5), one can apply the $1/\omega_{0}$ expansion (the Floquet-
Magnus expansion Eckardt_2015 ; Goldman_2014 ; Bukov_2015 ; Casas_2001 ) in
order to turn the periodically time-dependent Hamiltonian (5) into the
effective stationary Hamiltonian

$\hat{\cal H}_{\mathrm{eff}}=\hat{\cal
H}_{0}+\sum_{n=1}^{\infty}\frac{[\hat{\cal H}_{n},\hat{\cal
H}_{-n}]}{n\hbar\omega_{0}}+{\it O}\left[\left(\frac{\hat{\cal
H}_{n}}{\hbar\omega_{0}}\right)^{2}\right],$ (7)

where $\hat{\cal H}_{n}$ are the components of the Fourier expansion of the
time-dependent Hamiltonian (5), $\hat{\cal
H}=\sum_{n=-\infty}^{\infty}\hat{\cal H}_{n}e^{in\omega_{0}t}$. Assuming the
field frequency $\omega_{0}$ to be high enough, one can restrict the Floquet-
Magnus expansion (7) by the term $\hat{\cal H}_{0}$. Then the effective
Hamiltonian (7) reads

$\hat{\cal
H}_{\mathrm{eff}}=\sum_{j=e,h}\frac{\hat{\mathbf{p}}_{j}^{2}}{2m_{j}}+U_{0}(\mathbf{r}_{e},\mathbf{r}_{h}),$
(8)

where

$U_{0}(\mathbf{r}_{e},\mathbf{r}_{h})=\frac{1}{2\pi}\int_{-\pi}^{\pi}U_{C}(\mathbf{r}_{e}-\mathbf{r}_{e}^{\prime}(t),\mathbf{r}_{h}-\mathbf{r}_{h}^{\prime}(t))d(\omega
t)$ (9)

is the Coulomb potential (3) modified by the dressing field (2) (the dressed
Coulomb potential). Introducing the new coordinates
$\mathbf{R}=(m_{e}\mathbf{r}_{e}+m_{h}\mathbf{r}_{h})/(m_{e}+m_{h})$ and
${\mathbf{r}}=\mathbf{r}_{e}-\mathbf{r}_{h}$, the effective Hamiltonian (8)
can be rewritten as

$\hat{\cal
H}_{\mathrm{eff}}=\frac{\hat{\mathbf{P}}^{2}}{2M}+\frac{\hat{\mathbf{p}}^{2}}{2\mu}+U_{0}({r}),$
(10)

where $\mathbf{R}$ is the radius-vector of the exciton center of mass,
${\mathbf{r}}$ is the radius-vector of relative position of electron and hole,
$M=m_{e}+m_{h}$ is the total exciton effective mass,
$\mu=m_{e}m_{h}/(m_{e}+m_{h})$ is the reduced exciton mass,
$\hat{\mathbf{P}}=-i\hbar\nabla_{\mathbf{R}}$ and
$\hat{\mathbf{p}}=-i\hbar\nabla_{\mathbf{r}}$ are the momentum operators in
the new coordinates,

$\displaystyle U_{0}({r})$ $\displaystyle=$
$\displaystyle\left\\{\begin{array}[]{rl}-({2e^{2}}/{\pi
r_{0}\epsilon})K\left({r}/{r_{0}}\right),&{r}/{r_{0}}\leq 1\\\ \\\
-({2e^{2}}/{\pi r\epsilon})K\left({r_{0}}/{r}\right),&{r}/{r_{0}}\geq
1\end{array}\right.$ (14)

is the dressed Coulomb potential (9) written in an explicit form (see Fig. 2),
the function $\mathrm{K}(\zeta)$ is the elliptical integral of the first kind,
and

$r_{0}=\frac{|e|E_{0}}{\mu\omega_{0}^{2}}$ (15)

is the radius of the classical circular trajectory in the circularly polarized
field (2) for a particle with the mass $\mu$ and the charge $e$. Thus, the
Floquet problem for excitons in the QW irradiated by the dressing field (2) is
reduced to the Schrödinger problem with the stationary Hamiltonian (10), which
is under consideration below.

Figure 2: (a) The bare Coulomb potential $U_{C}(r)$ and (b) the dressed
Coulomb potential $U_{0}(r)$ plotted as functions of the planar coordinates
$x,y$.

##  III Results and discussion

The eigenfunction of the Hamiltonian (10) reads

$\Psi(\mathbf{R},\mathbf{r})=\frac{1}{\sqrt{2\pi
S}}e^{i\mathbf{K}\mathbf{R}}e^{im\phi}\psi_{X}(r),$ (16)

where $\mathbf{K}$ is the wave vector of the exciton center of mass, $m$ is
the exciton angular momentum, $\varphi$ is the polar angle in the $x,y$ plane,
$S$ is the area of the QW, and the wave function $\psi_{X}(r)$ defines the
radiative broadening of the exciton spectral line Fang_2019

$\Gamma_{X}=\frac{2\pi
e^{2}|p_{cv}|^{2}}{cm_{0}^{2}\omega_{X}\sqrt{\epsilon}}|\psi_{X}(0)|^{2},$
(17)

where $p_{cv}$ is the interband matrix element of the momentum for the
semiconductor, $m_{0}$ is the mass of electron in vacuum,
$\hbar\omega_{X}=\varepsilon_{g}-\varepsilon_{X}$ is the photon energy of the
exciton emission, $\varepsilon_{g}$ is the semiconductor band gap, and
$\varepsilon_{X}$ is the exciton binding energy. In the following, we will
restrict the analysis by the exciton states with the angular momentum $m=0$,
which make the main contribution to the exciton emission. Indeed, it follows
from the Hamiltonian (10) that $\psi_{X}(r)\propto r^{|m|}$ near $r=0$ and,
therefore, the radiative lifetime of exciton is
$\tau_{X}=\hbar/2\Gamma_{X}\rightarrow\infty$ for $m\neq 0$. Then we arrive
from the Hamiltonian (10) to the Schrödinger equation,

$\left[-\frac{\hbar^{2}}{2\mu}\frac{1}{r}\frac{\partial}{\partial
r}\left(r\frac{\partial}{\partial r}\right)+U_{0}(r)\right]\psi_{X}(r)={\cal
E}_{X}\psi_{X}(r),$ (18)

which defines both the exciton wave function $\psi_{X}(r)$ and the exciton
binding energy $\varepsilon_{X}=-{\cal E}_{X}$.

The solutions of the Schrödinger equation (18) depend on the two
characteristic scales: The exciton Bohr radius

$a_{X}=\frac{\hbar^{2}\epsilon}{\mu e^{2}}$ (19)

and the field-induced trajectory radius $r_{0}$ described by Eq. (15). In what
follows, we will consider the case of weak dressing field (2), when the
condition $a_{X}\gg r_{0}$ meets. Under this condition, the dressed potential
(14) can be considered as a weakly perturbed Coulomb potential (3) and,
therefore, the conventional perturbation theory can be applied to solve the
Schrödinger equation (18). In the first order of the perturbation theory, the
exciton wave function, $\psi_{X}(r)$, and the corresponding exciton binding
energy, $\varepsilon_{X}$, are described by the equations

$\psi_{X}^{(n)}(r)=\psi_{X0}^{(n)}(r)-\sum_{l\neq
n}\frac{\langle\psi_{X0}^{(l)}(r)|\Delta
U(r)|\psi_{X0}^{(n)}(r)\rangle}{\varepsilon_{X0}^{(l)}-\varepsilon_{X0}^{(n)}}\psi_{X0}^{(l)}(r)$
(20)

and

$\varepsilon_{X}^{(n)}=\varepsilon_{X0}^{(n)}-\langle\psi_{X0}^{(n)}(r)|\Delta
U(r)|\langle\psi_{X0}^{(n)}(r)\rangle,$ (21)

respectively, where $\Delta U(r)=U_{0}(r)-U_{C}(r)$ is the perturbation
potential and the index $n$ labels different eigenstates of the Schrödinger
problem (18). Here $\psi_{X0}^{(n)}(r)$ and $\varepsilon_{X0}^{(n)}$ are the
solutions of the problem in the absence of irradiation $(E_{0}=0)$, when the
dressed potential (9) turns into the usual two-dimensional Coulomb potential
(3). In what follows, we will consider only the ground exciton state, omitting
the index $n$ in both the wave function (20) and the binding energy (21). We
will also calculate the wave function (20) only for the zero coordinate,
$r=0$, since its value at this point, $\psi_{X}(0)$, defines the broadening
(17). Substituting the known solutions of the Schrödinger problem for the two-
dimensional Coulomb potential Yang_1991  into Eqs. (20) and (21), after
straightforward but cumbersome calculations we arrive at the simple equations

$\displaystyle\psi_{X}(0)$ $\displaystyle=$
$\displaystyle\frac{4}{a_{X}}\left[1-8\left(1-\frac{\pi^{2}}{16}-\ln{\sqrt{2}}\right)\left(\frac{r_{0}}{a_{X}}\right)^{2}\right]$
(22) $\displaystyle+$ $\displaystyle{\it
O}\left[\left(\frac{r_{0}}{a_{X}}\right)^{4}\right]$

and

$\varepsilon_{X}=\frac{2e^{2}}{a_{X}}\left[1-8\left(\frac{r_{0}}{a_{X}}\right)^{2}\right]+{\it
O}\left[\left(\frac{r_{0}}{a_{X}}\right)^{4}\right].$ (23)

For definiteness, let us restrict the following analysis by a GaAs-based QW
with the band gap $\varepsilon_{g}=1.51~{}\mathrm{eV}$, the permittivity
$\epsilon=12$, and the effective masses of electrons and holes
$m_{e}=0.067m_{0}$ and $m_{h}=0.47m_{0}$, respectively. The calculation
results are presented in Fig. 3 for the different irradiation intensities
$I=cE_{0}^{2}/4\pi$ and field frequencies $\omega_{0}$. Comparing the dressed
Coulomb potential (9) and the “bare” Coulomb potential (3) plotted in Fig. 2,
one can conclude that the dressing field (2) induces the repulsive area in the
dressed potential $U_{0}(r)$ near $r=0$ with the local maximum at $r=0$,
shifting the attractive potential well from the point $r=0$ to the ring of the
radius $r=r_{0}$. As a result, this repulsive area decreases the exciton
binding energy, $\varepsilon_{X}$. Correspondingly, the field-induced shift of
photon energy of the exciton emission is
$\hbar\Delta\omega_{X}=\hbar(\omega_{X}-\omega_{X0})=-(\varepsilon_{X}-\varepsilon_{X0})>0$,
where $\omega_{X0}$ and $\varepsilon_{X0}$ are the frequency of exciton
emission and the exciton binding energy in the absence of irradiation,
respectively. Thus, the irradiation results in the blue shift of the exciton
emission, which increases with increasing the irradiation intensity $I$ (see
Fig. 3a). The same field-induced repulsive area of the dressed potential
$U_{0}(r)$ increases the distance between electron and hole and, therefore,
decreases the exciton wave function value (22). As a consequence, the
irradiation decreases the radiative broadening (17) plotted in Fig. 3b, what
means the field-induced narrowing of the exciton spectral line. It should be
noted that the decreasing of the broadening $\Gamma_{X}$ leads also to
increasing the exciton lifetime $\tau_{X}=\hbar/2\Gamma_{X}$, i.e. the field-
induced stabilization of exciton (dynamical stabilization) appears. This looks
promising to control the quantum effects originated from the Bose nature of
excitons (particularly, the Bose-Einstein condensation of them Butov_1994 ;
Butov_2002 ) which strongly depend on the lifetime.

Figure 3: (a) The irradiation-induced shift of the light emission frequency,
$\Delta\omega_{X}$, from the ground exciton state in the GaAs-based QW as a
function of the irradiation intensity, $I$, and the irradiation frequency,
$\omega_{0}$; (b) The radiative broadening of the exciton spectral line,
$\Gamma_{X}$, as a function of the irradiation intensity, $I$, and the
irradiation frequency, $\omega_{0}$ for the same QW, where $\Gamma_{X0}$ is
the broadening in the absence of irradiation.

To discuss the coordinate dependence of the eigenfunctions of the Hamiltonian
(10), it should be noted that the dressed potential (14) turns into the usual
Coulomb potential (3) in the absence of irradiation. In this case, the
discussed exciton eigenfunction coincides with the well-known eigenfunction of
two-dimensional hydrogen atom (which is the product of the exponent and the
confluent hypergeometric function) Yang_1991 . Particularly, the eigenfunction
of the ground exciton state in the absence of irradiation reads
$\psi_{X0}(r)=(4/a_{X})e^{-2r/a_{X}}$. It follows from the aforesaid that the
irradiation perturbs the Coulomb potential $U_{C}(r)$ substantially only near
its origin point $r=0$, inducing the local potential maximum there (see Fig.
2). Therefore, the eigenfunctions of the Hamiltonian (10) in the presence of
the irradiation are the above-mentioned eigenfunctions of two-dimensional
hydrogen atom Yang_1991 , which are slightly perturbed (reduced) by the
irradiation near $r=0$.

To discuss the polarization dependence of the excitonic effects, it should be
noted that the dressed Coulomb potential $U_{0}(\mathbf{r})$ crucially depends
on the light polarization Kibis_2019 . Namely, the circular polarization leads
to the local maximum of the potential at $r=0$ (see Fig. 2b), whereas the
linear polarization induces only the saddle point at $r=0$ (see Fig. 2a in
Ref. Kibis_2019, ). Since the discussed excitonic effects originate physically
from the light-induced electron-hole repulsion near $r=0$, they are more
pronounced for the dressed potential with the local maximum which repulses an
electron and a hole from each other more effectively than the saddle-like
potential. Therefore, the circular polarization considered in the present
article is preferred to observe the discussed excitonic effects
experimentally.

The present theory is developed under the assumption of stable exciton states.
Therefore, it describes the problem accurately if the exciton lifetime,
$\tau_{X}$, much exceeds the dressing field period, $T$, i.e. the condition
$\omega_{0}\tau_{X}\gg 1$ should be satisfied. In semiconductor QWs fabricated
with using the modern nanotechnologies, the lifetime $\tau_{X}$ is of tens
picosecond scale and, therefore, the developed theory is applicable to
dressing field frequencies $\omega_{0}$ starting from the upper microwave
range.

Finally, one has to stress again that the discussed excitonic effects appear
due to the feature of the dressed Coulomb potential near $r=0$, where the
dressing field (2) induces the repulsive area (see Fig. 2b). This differs
crucially from the case of the repulsive Coulomb interaction between charge
particles. Indeed, the circularly polarized field (2) applied to the repulsive
Coulomb potential induces the attractive area in the core of the potential,
which leads to the electron states bound at the repulsive potential and,
particularly, to the field-induced electron pairing Kibis_2019 ; Kibis_2020_2
; Kibis_2021_2 ; Kibis_2021_3 ; Iorsh_2021 .

##  IV Conclusion

We showed theoretically that a high-frequency off-resonant electromagnetic
field can serve as a tool to control excitons in semiconductor quantum wells.
Since the excitons form the main mechanism for light emission from
semiconductor structures, the developed theory can find its application to
tune various characteristics of the optoelectronic devices based on quantum
wells, including the emission frequency and the radiative broadening of the
spectral line. Moreover, it can be used also to control various effects
depending on the lifetime of excitons, which rises under the irradiation.
Since the onset of field-induced excitonic effects happens at the timescale of
the field period, the discussed method to control excitonic properties of
quantum wells by a high-frequency electromagnetic field is very fast as
compared to the relatively slow electrostatic control of them by the gate
voltage. This is the doubtless advantage from viewpoint of its possible
applications to the fast optoelectronic devices based on quantum wells. The
present analysis is performed for conventional GaAs-based semiconductor
quantum wells and demonstrates that the discussed effects can be observable
for relatively weak irradiation with the intensity of tens mW/cm2.

##  V Acknowledgements

The reported study was funded by the Russian Science Foundation (project
20-12-00001).

##  Appendix A The Kramers-Henneberger unitary transformation

The Kramers-Henneberger unitary transformation Kramers ;
henneberger1968perturbation ,

$\hat{\cal
U}(t)=\exp\left\\{\frac{i}{\hbar}\int^{\,t}\left[\frac{e}{mc}\mathbf{A}(\tau)\hat{\mathbf{p}}-\frac{e^{2}}{2mc^{2}}A^{2}(\tau)\right]d\tau\right\\},$
(24)

was introduced into the quantum mechanics in order to remove the coupling of
the momentum $\hat{\mathbf{p}}$ to the vector potential $\mathbf{A}(t)$ in the
conventional Hamiltonian,

$\hat{\cal
H}=\frac{[\hat{\mathbf{p}}-e\mathbf{A}(t)/c]^{2}}{2m}+U(\mathbf{r}),$ (25)

describing the behaviour of a particle with the charge $e$ and the mass $m$ in
the electromagnetic field with the vector potential $\mathbf{A}(t)$ and the
potential energy $U(\mathbf{r})$. Taking into account that

$\hat{T}=\exp\left[-\frac{ie}{\hbar
mc}\int^{\,t}\mathbf{A}(\tau)\hat{\mathbf{p}}d\tau\right]$ (26)

is the translation operator, we arrive at the equality
henneberger1968perturbation

$\hat{T}f(\mathbf{r})=f\left(\mathbf{r}-\frac{e}{mc}\int^{\,t}\mathbf{A}(\tau)d\tau\right),$
(27)

where $f(\mathbf{r})$ is any function depending on the particle coordinate
$\mathbf{r}$. It follows from Eq. (27) that the applying of the unitary
transformation (24) to the Hamiltonian (25) results in the Hamiltonian

$\displaystyle\hat{\cal H}^{\prime}$ $\displaystyle=$ $\displaystyle\hat{\cal
U}^{\dagger}(t)\hat{\cal H}\hat{\cal U}(t)-i\hbar\hat{\cal
U}^{\dagger}(t)\partial_{t}\hat{\cal U}(t)$ (28) $\displaystyle=$
$\displaystyle\frac{\hat{\mathbf{p}}^{2}}{2m}+U\left(\mathbf{r}-\frac{e}{mc}\int^{\,t}\mathbf{A}(\tau)d\tau\right),$

which is physically equal to the initial Hamiltonian (25). However, in
contrast to the Hamiltonian (25), the terms $\propto\hat{\mathbf{p}}$ are
absent in the Hamiltonian (28), what simplifies solving many Schrödinger
problems. Taking into account the different masses and charges of electrons
and holes, we arrive easily from Eq. (24) to the unitary transformation (4)
corresponding to the considered exciton problem.

##  Appendix B The Floquet-Magnus expansion

In the most general form, the nonstationary Schrödinger equation for an
electron in a periodically time-dependent field with the frequency
$\omega_{0}$ can be written as $i\hbar\partial_{t}\psi(t)=\hat{\cal
H}(t)\psi(t)$, where $\hat{\cal H}(t+T)=\hat{\cal H}(t)$ is the periodically
time-dependent Hamiltonian and $T=2\pi/\omega_{0}$ is the field period. It
follows from the well-known Floquet theorem that solution of the Schrödinger
equation is the Floquet function, $\psi(t)=e^{-i\varepsilon
t/\hbar}\varphi(t)$, where $\varphi(t+T)=\varphi(t)$ is the periodically time-
dependent function and $\varepsilon$ is the electron (quasi)energy describing
behavior of the electron in the periodical field. The Floquet problem is aimed
to find the electron energy spectrum, $\varepsilon$. To solve the problem, let
us introduce the unitary transformation, $\hat{\cal U}=e^{iS}$, which
transfers the time dependence from the Hamiltonian $\hat{\cal H}(t)$ to its
basis states. Then we arrive from the time-dependent Hamiltonian $\hat{\cal
H}(t)$ to the effective time-independent Hamiltonian

$\hat{\cal H}_{\mathrm{eff}}=\hat{\cal U}^{\dagger}\hat{\cal H}\hat{\cal
U}-i\hbar\hat{\cal U}^{\dagger}\partial_{t}\hat{\cal U}.$ (29)

Solving the stationary Schrödinger problem with the Hamiltonian (29),
$\hat{\cal H}_{\mathrm{eff}}\Psi=\varepsilon\Psi$, one can find the sought
electron energy spectrum, $\varepsilon$.

There is the regular method to find the transformation matrix $S$ in the
particular case of high field frequency, $\omega_{0}$, which satisfies the
condition ${\hat{\cal H}_{n}}/{\hbar\omega_{0}}\ll 1$, where $\hat{\cal
H}_{n}$ are the components of the Fourier expansion of the time-dependent
Hamiltonian, $\hat{\cal H}(t)=\sum_{n=-\infty}^{\infty}\hat{\cal
H}_{n}e^{in\omega_{0}t}$. Namely, both the transformation matrix $S$ and the
effective Hamiltonians (29) can be found as the $1/\omega_{0}$ expansion (the
Floquet-Magnus expansion) Eckardt_2015 ; Goldman_2014 ; Bukov_2015 ;
Casas_2001 . As a result, we arrive at the effective stationary Hamiltonian
(8). Thus, the Floquet-Magnus expansion reduces the time-dependent Floquet
problem to the conventional stationary Schrödinger problem with the time-
independent Hamiltonian (8).

## References

  * (1) Oka T and Kitamura S 2019 Floquet Engineering of Quantum Materials Annu. Rev. Condens. Matter. Phys. 10 387
  * (2) Basov D N, Averitt R D and Hsieh D 2017 Towards properties on demand in quantum materials Nat. Mater. 16 1077
  * (3) Eckardt A and Anisimovas E 2015 High-frequency approximation for periodically driven quantum systems from a Floquet-space perspective New J. Phys. 17 093039
  * (4) Goldman N and Dalibard J 2014 Periodically driven quantum systems: effective hamiltonians and engineered gauge fields Phys. Rev. X 4, 031027
  * (5) Bukov M, D’Alessio L and Polkovnikov A 2015 Universal high-frequency behavior of periodically driven systems: from dynamical stabilization to Floquet engineering Adv. Phys. 64, 139
  * (6) Casas F, Oteo J A and Ros J 2001 Floquet theory: exponential perturbative treatment J. Phys. A 34, 3379
  * (7) Kibis O V, Boev M V, Kovalev V M and Shelykh I A 2020 Floquet engineering of the Luttinger Hamiltonian Phys. Rev. B 102 035301
  * (8) Koshelev K, Kachorovskii V Y and Titov M 2015 Resonant inverse Faraday effect in nanorings Phys. Rev. B 92, 235426
  * (9) Rechtsman M C, Zeuner J M, Plotnik Y, Lumer Y, Podolsky D, Dreisow F, Nolte S, Segev M and Szameit A 2013 Photonic Floquet topological insulator Nature 496, 196
  * (10) Wang Y H, Steinberg H, Jarillo-Herrero P and Gedik N 2013 Observation of Floquet-Bloch states on the surface of a topological insulator Science 342, 453
  * (11) Foa Torres L E F, Perez-Piskunow P M, Balseiro C A and Usaj G 2014 Multiterminal Conductance of a Floquet Topological Insulator Phys. Rev. Lett. 113, 266801
  * (12) Calvo H L, Foa Torres L E F, Perez-Piskunow P M, Balseiro C A and Usaj G 2015 Floquet interface states in illuminated three-dimensional topological insulators Phys. Rev. B 91, 241404(R)
  * (13) Mikami T, Kitamura S, Yasuda K, Tsuji N, Oka T and Aoki H 2016 Brillouin-Wigner theory for high-frequency expansion in periodically driven systems: Application to Floquet topological insulators Phys. Rev. B 93, 144307
  * (14) Oka T and Aoki H 2009 Photovoltaic Hall effect in graphene Phys. Rev. B 79, 081406(R)
  * (15) Iurov A, Gumbs G and Huang D 2017 Exchange and correlation energies in silicene illuminated by circularly polarized light J. Mod. Opt. 64, 913
  * (16) Iurov A, Gumbs G, Roslyak O and Huang D 2013 Photon dressed electronic states in topological insulators: tunneling and conductance J. Phys.: Condens. Matter 25, 135502
  * (17) Syzranov S V, Rodionov Ya I, Kugel K I and Nori F 2013 Strongly anisotropic Dirac quasiparticles in irradiated graphene Phys. Rev. B 88, 241112(R)
  * (18) Usaj G, Perez-Piskunow P M, Foa Torres L E F and Balseiro C A 2014 Irradiated graphene as a tunable Floquet topological insulator Phys. Rev. B 90, 115423
  * (19) Perez-Piskunow P M, Usaj G, Balseiro C A and Foa Torres L E F 2014 Floquet chiral edge states in graphene Phys. Rev. B 89, 121401(R)
  * (20) Glazov M M and Ganichev S D 2014 High frequency electric field induced nonlinear effects in graphene Phys. Rep. 535, 101
  * (21) Sentef M A, Claassen M, Kemper A F, Moritz B, Oka T, Freericks J K and Devereaux T P 2015 Theory of Floquet band formation and local pseudospin textures in pump-probe photoemission of graphene Nat. Commun. 6, 7047
  * (22) Sie E J, McIver J W, Lee Y H, Fu L, Kong J and Gedik N 2015 Valley-selective optical Stark effect in monolayer WS2 Nat. Mater. 14, 290
  * (23) Kibis O V, Dini K, Iorsh I V and Shelykh I A 2017 All-optical band engineering of gapped Dirac materials Phys. Rev. B 95, 125401
  * (24) Iurov A, Gumbs G and Huang D H 2019 Peculiar electronic states, symmetries, and Berry phases in irradiated alpha-T(3)materials Phys. Rev. B 99, 205135
  * (25) Iurov A, Zhemchuzhna L, Dahal D, Gumbs G and Huang D 2020 Quantum-statistical theory for laser-tuned transport and optical conductivities of dressed electrons in alpha-T(3)materials Phys. Rev. B 101, 035129
  * (26) McIver J W, Schulte B, Stein F U, Matsuyama T, Jotzu G, Meier G and Cavalleri A 2020 Light-induced anomalous Hall effect in graphene Nat. Phys. 16, 38
  * (27) Nag B R 2002 Physics of Quantum Well Devices (New York: Kluwer Academic Press)
  * (28) Morina S, Kibis O V, Pervishko A A and Shelykh I A 2015 Transport properties of a two-dimensional electron gas dressed by light Phys. Rev. B 91, 155312
  * (29) Pervishko A A, Kibis O V, Morina S and Shelykh I A 2015 Control of spin dynamics in a two-dimensional electron gas by electromagnetic dressing Phys. Rev. B 92, 205403
  * (30) Dini K, Kibis O V and Shelykh I A 2016 Magnetic properties of a two-dimensional electron gas strongly coupled to light Phys. Rev. B 93, 235411
  * (31) Kyriienko O, Kibis O V and Shelykh I A 2017 Floquet control of dipolaritons in quantum wells Opt. Lett. 42, 2398
  * (32) Lindner N H, Refael G and Galitski V 2011 Floquet topological insulator in semiconductor quantum wells Nat. Phys. 7, 490
  * (33) Zakhleniuk N A, Bennett C R, Stavrou V N, Babiker M and Ridley B K 1999 Quantum Capture of Injected Electrons in GaN-Based Laser Heterostructures Phys. Status Solidi (a) 176, 79
  * (34) Stavrou V N, Bennett C R, Al-Dossary O M M M and Babiker M 2001 Electron scattering and capture rates in quantum wells by emission of hybrid optical phonons Phys. Rev. B 63, 205304
  * (35) Ivchenko E L 2007 Optical Spectroscopy of Semiconductor Nanostructures (Berlin: Springer)
  * (36) Zory P S 1993 Quantum Well Lasers (London: Academic Press)
  * (37) Kramers H A 1952 Collected Scientific Papers (Amsterdam: North-Holland)
  * (38) Henneberger W C 1968 Perturbation method for atoms in intense light beams Phys. Rev. Lett. 21, 838
  * (39) Yang X L, Guo S H and Chan F T 1993 Analytic solution of a two-dimensional hydrogen atom. I. Nonrelativistic theory Phys. Rev. A 43, 1186
  * (40) Fang H H, Han B, Robert C, Semina M A, Lagarde D, Courtade E, Taniguchi T, Watanabe K, Amand T, Urbaszek B, Glazov M M and Marie X 2019 Control of the Exciton Radiative Lifetime in van der Waals Heterostructures Phys. Rev. Lett. 123, 067401
  * (41) Butov L V 2017 Excitonic Devices Superlattice. Microst. 108, 2
  * (42) Butov L V, Zrenner A, Abstreiter G, Bohm G and Weimann G 1994 Condensation of Inderect Excitons in Coupled AlAs/GaAs Quantum Wells Phys. Rev. Lett. 73, 304
  * (43) Butov L V, Lai C W, Ivanov A L, Gossard A C and Chemla D S 2002 Towards Bose-Einstein condensation of excitons in potential traps Nature 417, 47
  * (44) Kibis O V 2019 Electron pairing in nanostructures driven by an oscillating field Phys. Rev. B 99, 235416
  * (45) Kibis O V, Boev M V and Kovalev V M 2020 Light-induced bound electron states in two-dimensional systems: Contribution to electron transport Phys. Rev. B 102, 075412
  * (46) Kibis O V, Kolodny S A and Iorsh I V 2021 Fano resonances in optical spectra of semiconductor quantum wells dressed by circularly polarized light Opt. Lett. 46, 50
  * (47) Kibis O V, Boev M V and Kovalev V M 2021 Optically induced hybrid Bose–Fermi system in quantum wells with different charge carriers Opt. Lett. 46, 5316
  * (48) Iorsh I V and Kibis O V 2021 Optically induced Kondo effect in semiconductor quantum wells J. Phys.: Condens. Matter 33, 495302