# mBEST: Realtime Deformable Linear Object Detection
Through Minimal Bending Energy Skeleton Pixel Traversals

Andrew Choi1, Dezhong Tong2, Brian Park1,
Demetri Terzopoulos1, Jungseock Joo3, and Mohammad Khalid Jawed†,2 The authors
are with the University of California, Los Angeles (UCLA), CA 90095,
USA1Andrew Choi, Brian Park, and Demetri Terzopoulos are with the UCLA
Computer Science Department (email<EMAIL_ADDRESS><EMAIL_ADDRESS>dt@cs.ucla.edu)2Dezhong Tong and Mohammad Khalid Jawed are with the UCLA
Department of Mechanical & Aerospace Engineering (email:
<EMAIL_ADDRESS>khalidjm@seas.ucla.edu)3Jungseock Joo is with the UCLA
Department of Communication and is currently working at NVIDIA Corporation
(email: jjoo@comm.ucla.edu)† Corresponding author.This research was funded in
part by the National Science Foundation under award numbers OAC-2209782,
CMMI-2101751, CAREER-2047663, and IIS-1925360.

###### Abstract

Robotic manipulation of deformable materials is a challenging task that often
requires realtime visual feedback. This is especially true for deformable
linear objects (DLOs) or “rods”, whose slender and flexible structures make
proper tracking and detection nontrivial. To address this challenge, we
present mBEST, a robust algorithm for the realtime detection of DLOs that is
capable of producing an ordered pixel sequence of each DLO’s centerline along
with segmentation masks. Our algorithm obtains a binary mask of the DLOs and
then thins it to produce a skeleton pixel representation. After refining the
skeleton to ensure topological correctness, the pixels are traversed to
generate paths along each unique DLO. At the core of our algorithm, we
postulate that intersections can be robustly handled by choosing the
combination of paths that minimizes the cumulative bending energy of the
DLO(s). We show that this simple and intuitive formulation outperforms the
state-of-the-art methods for detecting DLOs with large numbers of sporadic
crossings and curvatures with high variance. Furthermore, our method achieves
a significant performance improvement of approximately 40 FPS compared to the
15 FPS of prior algorithms, which enables realtime applications.

##  I INTRODUCTION

As robots become increasingly more intelligent and capable, developing robust
and effective deformable material manipulation skills has started to receive
large research attention [1]. Among various deformable objects, deformable
linear objects (DLOs) — typically referred to as “rods” by the mechanics
community — are a special group, including everyday objects such as cables,
ropes, tubes, and threads. Due to their unique geometric characteristic (width
$\sim$ height $\ll$ length), DLOs are widely used in various domestic and
industrial applications, including surgical suturing [2], knot fastening [3,
4], cable manipulation [5, 6], food manipulation [7], mechanics analysis [8],
and more. However, because of their flexibility, DLOs are often prone to
complex tangling, which complicates manipulation. In addition, the complicated
structures made by DLOs usually have unique topology-induced mechanical
properties [9, 10, 11, 12, 13] and are, therefore, used to tie knots for
sailing, fishing, climbing, and various other engineering applications. Given
all of the above, a robust, efficient, and accurate perception algorithm for
DLOs is crucial to both deformable material manipulation and soft robotics.

In this paper, we present an algorithm named mBEST (Minimal Bending Energy
Skeleton pixel Traversals) for robust, accurate, and fast instance
segmentation of DLOs. Without any prior knowledge regarding the geometries,
colors, and total number of DLOs, mBEST takes a raw RGB image as input and
provides a series of ordered pixels expressing the centerline of each unique
DLO, thus allowing for the configurations of different DLOs to be easily
incorporated into motion planning and manipulation schemes.

We implement the following sequence of processing procedures to achieve
instance segmentation of DLOs. Like previous work [14], we first apply
semantic image segmentation to achieve a binary mask of the DLOs against the
background. We try two options for semantic segmentation. The first involves
using a Deep Convolutional Neural Network (DCNN) segmentation model, resulting
in binary masks of varying quality. The second is color filtering, which can
achieve hyper-accurate binary masks but requires adequate color contrast
between the DLOs and background. These two options are discussed in further
detail in Sec. IV.

After a binary mask is achieved, we apply a thinning algorithm to the mask to
obtain a skeleton pixel representation of the DLOs. The resulting
representation preserves the connectivity and centerlines of the binary mask
while being only a single pixel in width. Therefore, key points such as ends
and intersections are easily detected. After a series of refinement steps to
ensure topological correctness, the skeleton is then traversed, one end at a
time, in a way that minimizes the cumulative bending energy of the DLOs until
another end is encountered. Each traversal results in a unique DLO’s
centerline pixel coordinates, which can then be used to optionally produce
segmentation masks. Fig. 1 presents a high-level overview of the mBEST
processing pipeline.

Overall, our main contributions in this article are to

  1. 1.

develop a robust end-to-end pipeline for obtaining ordered centerline
coordinates and segmentation masks of DLOs from images;

  2. 2.

showcase that the relatively simple (and physically insightful) optimization
objective of minimizing cumulative bending energy outperforms the state of the
art (SOTA); and

  3. 3.

achieve realtime performance that more than doubles the speed of the previous
SOTA.

In addition to the above, we release all the source code and datasets (with
ground truth) used at https://github.com/StructuresComp/mBEST.

The remainder of the article is organized as follows: We proceed with a review
of related work in Sec. II. The algorithmic formulation of mBEST is then
discussed extensively in Sec. III. In Sec. IV, we report our experimental
results comparing mBEST with SOTA approaches. Finally, we offer concluding
remarks and discuss potential future research directions in Sec. V.

Figure 1: Pipeline overview of mBEST. An input image (a) is converted to a
binary mask (b) using a segmentation method. This binary mask is then
converted to a skeleton pixel representation (c), where the connectivity and
centerlines of the DLOs are preserved as a single-pixel-width structure, and
keypoints such as intersections and ends are then detected. This is followed
by a series of refinement steps to maintain the topological correctness of the
skeleton: split ends (d1) are pruned (d2) and pixels representing a single
topological intersection (e1) are clustered and replaced with a more intuitive
intersection (e2). Finally, paths for each DLO are generated (f) by traversing
skeleton pixels and choosing the path that minimizes cumulative bending
energy.

##  II Related Work

Although research into developing manipulation skills for DLOs has been highly
prevalent, the perception algorithms used in these efforts are usually
undeveloped. For example, in the work of Tong et al. [8], attached markers are
required for extracting the configuration of the manipulated DLO. Zhu et al.
[5] carefully adjusted the workspace to increase the contrast between the
manipulated DLO (cables) and their background. Although these prior efforts
successfully completed their target manipulation tasks, the simplistic
perception algorithms restrict real world applicability.

Consequently, DLO detection algorithms featuring various methodologies have
been proposed. For example, Keipour et al. [15] evaluate both curvatures and
distances to fit a continuous DLO. Using data-driven methods, Yan et al. [16]
train a neural network to reconstruct the topology of a DLO based on a coarse-
to-fine nodal representation. Though these methods achieve good results for
some datasets, they work under the strict assumption that only one DLO exists
within the scene, thus dramatically restricting their applicability.

One of the first perception algorithms capable of detecting multiple DLOs,
Ariadne [17], segmented images into superpixels and traversed the superpixels
belonging to DLOs in order to produce paths. The ambiguity of intersections is
handled using a multi-faceted cost function that takes into consideration
color, distance, and curvature. Despite its satisfactory performance, this
early approach suffered from a large number of hyperparameters, an
overreliance on DLOs being a uniform color, and the tedious requirement on the
user to manually select the ends of DLOs. Furthermore, the processing speed of
Ariadne was on the order of seconds, precluding realtime operation.

In recent years, data-driven methods have attracted increasing attention in
instance segmentation. In particular, researchers have shown that general
instance segmentation problems can be tackled efficiently and accurately using
Deep Convolutional Neural Networks (DCNNs) [18, 19, 20, 21]. Furthermore,
several tools have been introduced to help synthetically generate large
quantities of photorealistic data in order to adequately train such models
[22, 23]. Using DCNNs, Zanella et al. [24] created segmentations of DLOs such
as wires; however, the segmentations did not distinguish between each DLO.

Improving upon Ariadne, Ariadne+ [25], like [24], utilizes a DCNN model to
extract an initial binary mask of the DLOs. This allows the algorithm to then
apply superpixel segmentation purely on the binary mask itself, significantly
reducing the computation time. Paths are then generated in a similar fashion
to the original Ariadne algorithm by traversing superpixels while
intersections are handled using a neural network to predict the most probable
paths. The neural network uses the same three inputs as Ariadne: color,
distance, and curvature. Despite these improvements, Ariadne+ is sub-realtime;
i.e., less than 3 FPS.

The state-of-the-art FASTDLO [14] improves upon Ariadne+’s speed by forgoing
superpixel segementation altogether. Instead, it uses a skeleton pixel
representation of the DLO binary mask for path traversals. Intersections are
then also handled by a neural network. By replacing superpixel segmentation
with skeletonization, FASTDLO is reportedly able to achieve a realtime
performance of 20 FPS for images of size $640\times 360$ pixels.

Though Ariadne+ and FASTDLO are considered state-of-the-art DLO perception
algorithms, both algorithms have been evaluated only on scenes with DLOs with
relatively smooth curvatures and minimal self-loops. Our experiments will show
that both algorithms struggle to solve complicated configurations (e.g., DLOs
with highly variable curvatures resulting in many crossings and tangles)
correctly.

By comparison, our mBEST robustly solves complex scenes using the simple
notion that the most probable path is that which minimizes cumulative bending
energy. Not only does it outperform the SOTA on complex scenes, but we also
achieve realtime performance more than double that of FASTDLO. The key
algorithmic differences between mBEST and the SOTA are summarized in Table I.
Overall, we argue that the use of neural networks to solve intersections
yields unsatisfactory results once input scenes stray away from the training
data distributions, and we show that using a more principled formulation with
physical insight can outperform black box neural network approaches for a wide
variety of scenes containing DLOs.

TABLE I: Algorithmic Comparison Algorithm |  | Intersection
---
Rule
| DLO
---
Representation
Realtime
Ariadne [17] |  | color, distance,
---
curvature
superpixels | ✗
Ariadne+ [25] | DNN prediction | superpixels | ✓-
FASTDLO [14] | DNN prediction | skeleton pixels | ✓
mBEST | curvature | skeleton pixels | ✓

##  III METHODOLOGY

Our algorithm may be divided into the following steps:

  1. 1.

DLO Segmentation

  2. 2.

Skeletonization

  3. 3.

Keypoint Detection

  4. 4.

Pruning Split Ends

  5. 5.

Intersection Clustering and Replacement

  6. 6.

Minimal Bending Energy Path Generation

  7. 7.

Computing DLO Radii and Crossing Order

The following sections will describe each step in detail.

###  III-A DLO Segmentation

The first step in detecting the DLOs is to obtain a binary mask
$\mathbf{M}_{\textup{dlo}}$ of the image that distinguishes all DLO-related
pixels from the background. As mentioned previously, we use two semantic
segmentation methods: a DCNN segmentation model and color filtering. In
particular, we use FASTDLO’s pretrained DCNN model [14] in our experiments. To
eliminate noise, morphological closing and opening operations are performed on
the binary mask to remove any hollow areas.

The initial semantic segmentation method is not a key contribution of mBEST.
Rather, it is a modular component of the pipeline, allowing for different
methods to be plugged in depending on the use case.

###  III-B Skeletonization

The next step of the algorithm is to convert $\mathbf{M}_{\textup{dlo}}$ to a
skeleton mask $\mathbf{M}_{\textup{sk}}$ as shown in Fig. 1(b-c).
$\mathbf{M}_{\textup{sk}}$ is useful as both the connectivity and general
topology of the DLOs are maintained. Furthermore, as segments are only 1 pixel
in width, traversals along segments are not susceptible to path ambiguity. To
achieve skeletonization, we use an efficient thinning algorithm designed
specifically for 2D images; refer to [26] for the details.

###  III-C Keypoint Detection

Once a skeleton pixel representation is obtained, we can then detect two types
of key points: ends and intersections. Locating ends is crucial as they serve
as the start and finish points for skeleton pixel traversals. They also
indirectly tell us the number of DLO(s) in the image as
$n_{\textrm{dlo}}=n_{\textrm{ends}}/2$ (assuming the initial binary mask
maintained connectivity). Locating intersections is crucial as these represent
the only points at which a pixel traversal will have multiple possible routes.
Therefore, care must be given in choosing the correct path when passing
through an intersection.

To detect ends and intersections, a skeleton pixel classification kernel,

$\mathbf{K}=\begin{bmatrix}1&1&1\\\ 1&10&1\\\ 1&1&1\end{bmatrix},$

can be used to apply a convolution
$\mathbf{M}_{\textup{sk}}\circledast\mathbf{K}$ along the skeleton mask. We
can then obtain all end pixels $\mathbf{E}$ as pixels that have a value of 11
(1 neighbor) and all intersection pixels $\mathbf{I}$ as pixels having a value
greater than 12 (3 or more neighbors).

After obtaining both $\mathbf{E}$ and $\mathbf{I}$, additional work must be
done to obtain the correct representative sets. For example, end pixels that
are unindicative of a topological end may be produced from a noisy binary
mask. These “split ends” will then inadvertently produce intersection pixels
themselves as shown in Fig. 2. Additionally, a single topological intersection
will result in either two Y-shaped divides or a single X-shaped divide as
shown in Fig. 3(a). Such pixels must be clustered together accordingly and a
single point for the intersection must be determined. In the case of a
skeleton possessing two Y-shaped divides in regards to a single intersection,
the intersection must also be replaced with an X-shaped divide that more
accurately represents the true centerlines of the DLOs. The next two sections
will cover the pruning and clustering operations in detail.

###  III-D Pruning Split Ends

Figure 2: Examples of split ends that may occur during the skeletonization
process. Row (a) showcases split ends that may occur at an actual topological
end, while row (b) showcases a split end along a segment produced by a jagged
mask. For both examples, the first column shows the binary mask; the second
column shows the split end after skeletonization, and the third column
showcases the topologically correct structure after pruning.

When the boundary of the binary mask $\mathbf{M}_{\textup{dlo}}$ is jagged,
the skeleton mask $\mathbf{M}_{\textup{sk}}$ may contain several types of
split ends as shown in Fig. 2. Such split ends must be identified and pruned
as they do not accurately represent the topology of the DLO(s) and will result
in incorrect start points as well as cause path ambiguity during pixel
traversals.

Note that the length of a split end can be at most the radius of the DLO it is
sprouting from. Therefore, suffice it to say that the length of all split ends
should be within a threshold $\delta\ll$ length of the DLO. Given this, for
every end in $\mathbf{E}$, we can traverse along its segment until 1 of 3
things may occur:

  1. 1.

an intersection is encountered before traversing $\delta$ pixels,

  2. 2.

an end is encountered before traversing $\delta$ pixels,

  3. 3.

or neither was encountered after traversing $\delta$ pixels.

For conditions (1) and (2), we remove the segment that was just traversed from
$\mathbf{M}_{\textup{sk}}$ as well as the corresponding end from $\mathbf{E}$.
For condition (1) specifically, we must also remove all intersection pixels
that were produced from the pruned split end from $\mathbf{I}$. For any
endpoint that satisfies condition (3), we simply do nothing. This concludes
eliminating any noise-induced end and intersection pixels.

###  III-E Intersection Clustering and Replacement

Figure 3: The intersection clustering, replacement, and optimal path
generation pipeline. Two sample intersections are shown where skeletonization
results in a (a1) 2Y-shaped crossing and an (a2) X-shaped crossing. As
2Y-shaped crossings are topologically incorrect, we replace them by (b)
clustering the intersection pixels in two stages. The first stage involves
grouping adjacent pixels, while the second involves grouping nearby clusters.
Using the centroid location of the cluster, we can then (c) replace the
intersection by creating new ends and having new segments sprout and connect
to the centroid. Finally, the new generated ends and segments can be used to
(d) discover the combination of paths that minimizes the cumulative bending
energy of the DLO.

As mentioned previously in Sec. III-C, a single topological intersection can
result in either a 2Y or X-shaped branching as shown in Fig. 3(a).
Furthermore, each of these branches may have several intersection pixels,
i.e., pixels with 3 or more neighbors. Our goal then is to group each pixel in
$\mathbf{I}$ to a single branch and then group each branch to its true
topological intersection. With all the intersection pixels properly grouped,
we can then define a single intersection pixel that represents the true center
of a crossing for all crossings.

We perform clustering using Density-based Spatial Clustering of Applications
with Noise (DBSCAN) [27], a clustering algorithm that clusters data points
that are within a distance threshold $\epsilon$ of eachother. In our case,
$\epsilon$ will refer to Euclidean pixel distance. This algorithm is highly
convenient to use as it does not necessitate prior knowledge of the number of
clusters and thus, intersections.

To properly group all the intersection pixels in $\mathbf{I}$, two phases of
clustering are performed:

  1. 1.

($\epsilon=2$) All adjacent pixels in $\mathbf{I}$ are clustered together.
Each cluster is then averaged create a new $\mathbf{I}$.

  2. 2.

($\epsilon>2$) All pixels in the new $\mathbf{I}$ are clustered together with
a higher $\epsilon$. This is to account for intersections that produced
multiple Y-shaped branches. A new $\mathbf{I}$ is then created using the
averages of the clusters like step 1.

This two step clustering process (shown in Fig. 3(b)) is crucial as skipping
the first clustering step can result in the final center intersection pixel
being heavily biased towards a particular branch. After the correct set of
intersection pixels $\mathbf{I}$ is obtained, we can then replace all
clustered Y-shaped branches with an X-shaped branch as shown in Fig. 3(c).
This is done by simply removing all segments within a square window of the new
intersection pixel and then creating new segments that sprout from the new
intersection. Note that Fig. 3(c) shows that we record new “ends” when
replacing the intersection. These ends are recorded so that we know that an
intersection is upcoming during a pixel traversal. This allows us to then take
the correct precomputed path, which is discussed in the next section.

Input: $\mathbf{M}_{\textup{dlo}},\delta,\epsilon$

Output: $\mathbf{P}$

1 Func  _mBEST(_$\mathbf{M}_{\textup{dlo}},\delta,\epsilon$_)_:

2 $\mathbf{P}\leftarrow[\ ]$

3
$\mathbf{M}_{\textup{sk}}\leftarrow\textnormal{{Skeletonize}}(\mathbf{M}_{\textup{dlo}})$

4
$\mathbf{E},\mathbf{I}\leftarrow\textnormal{{DetectKeyPoints}}(\mathbf{M}_{\textup{sk}}\circledast\mathbf{K}$)

5
$\mathbf{E},\mathbf{I}\leftarrow\textnormal{{PruneSplitEnds}}(\mathbf{E},\mathbf{I},\mathbf{M}_{\textup{sk}},\delta)$

6
$\mathbf{I}\leftarrow\textnormal{{ClusterIntersections}}(\mathbf{I},\epsilon)$

7
$\mathbf{P}_{\textup{inter}}\leftarrow\textnormal{{GenIntersectionPaths}}(\mathbf{I},\mathbf{M}_{\textup{sk}})$

8

9 while  _$\mathbf{E}$ is not empty_ do

10 $\mathbf{x}\leftarrow\mathbf{E}$.pop()

11 while  _True_ do

12 $\boldsymbol{\tau}\leftarrow$ traverse along $\mathbf{M}_{\textup{sk}}$
from $\mathbf{x}$ until reaching an end $\mathbf{e}$

13 if  _$\mathbf{e}\in\mathbf{P}_{\textup{inter}}$_ then

14
$\boldsymbol{\tau}\leftarrow\boldsymbol{\tau}+\mathbf{P}_{\textup{inter}}^{i}$

15 $\mathbf{x}\leftarrow$ last pixel of $\mathbf{P}_{\textup{inter}}^{i}$

16

17 else

18 $\mathbf{E}$.remove$(\mathbf{e})$

19 break

20

21 $\mathbf{P}$.append($\boldsymbol{\tau}$)

22

23 return $\mathbf{P}$

24

Algorithm 1 mBEST Pseudocode

###  III-F Minimal Bending Energy Path Generation

For rods that have nonuniform curvatures, bending energy must be computed in a
discretized fashion. Given this, if we discretize a rod into $N$ nodes and
$N-1$ edges, then the total bending energy becomes

$\displaystyle E_{b}$
$\displaystyle=\frac{1}{2}\frac{EI}{dl}\sum^{N-2}_{k=1}(\kappa_{k}-\kappa^{0}_{k})^{2},$
(1)

where $EI$ is the bending stiffness, $\kappa_{k}$ and $\kappa_{k}^{0}$ are the
deformed and undeformed discrete dimensionless curvatures at node
$k\in[1,N-2]$, and $dl$ is the Voronoi length. For our DLOs, we make the
assumption that the undeformed curvature is always a straight configuration
($\kappa^{0}=0$). It is then trivial to see that minimizing the bending energy
of an elastic rod is the same as minimizing the discrete curvatures.

The norm of the discrete dimensionless curvature for a node $k$ can easily be
computed using the unit tangent vectors of the adjacent edges [28]:

$\bar{\kappa}_{k}=\left\lVert\frac{2\mathbf{t}^{k-1}\times\mathbf{t}^{k}}{1+\mathbf{t}^{k-1}\cdot\mathbf{t}^{k}}\right\rVert,$
(2)

where $\mathbf{t}^{k-1}$ and $\mathbf{t}^{k}$ are the unit tangent vectors of
the $k-1$-th and $k$-th edges, respectively.

Note that the only time we have to choose between multiple paths is at an
intersection as traversals through segments are unambiguous. Also recall that
all intersections have been replaced with new segments in an X-shaped pattern.
Using the new ends shown in Fig. 3(c), we can then compute the combination of
paths that minimizes the cumulative bending energy of the DLOs using the
optimization scheme

$\displaystyle(\mathbf{p}_{1}^{1},$
$\displaystyle\mathbf{p}_{1}^{2}),(\mathbf{p}_{2}^{1},\mathbf{p}_{2}^{2})=\operatorname*{arg\,min}_{\mathbf{a},\mathbf{b},\mathbf{c},\mathbf{d}}\lVert\kappa_{1}\rVert+\lVert\kappa_{2}\rVert,$
(3) $\displaystyle\mathbf{t}_{1}^{1}$
$\displaystyle=\frac{\mathbf{i}-\mathbf{p}_{1}^{1}}{\lVert\mathbf{i}-\mathbf{p}_{1}^{1}\rVert},\
\mathbf{t}_{1}^{2}=\frac{\mathbf{p}_{1}^{2}-\mathbf{i}}{\lVert\mathbf{p}_{1}^{2}-\mathbf{i}\rVert},$
$\displaystyle\mathbf{t}_{2}^{1}$
$\displaystyle=\frac{\mathbf{i}-\mathbf{p}_{2}^{1}}{\lVert\mathbf{i}-\mathbf{p}_{2}^{1}\rVert},\
\mathbf{t}_{2}^{2}=\frac{\mathbf{p}_{2}^{2}-\mathbf{i}}{\lVert\mathbf{p}_{2}^{2}-\mathbf{i}\rVert},$
$\displaystyle\kappa_{1}$
$\displaystyle=\frac{2\mathbf{t}_{1}^{1}\times\mathbf{t}_{1}^{2}}{1+\mathbf{t}_{1}^{1}\cdot\mathbf{t}_{1}^{2}},\
\kappa_{2}=\frac{2\mathbf{t}_{2}^{1}\times\mathbf{t}_{2}^{2}}{1+\mathbf{t}_{2}^{1}\cdot\mathbf{t}_{2}^{2}},$

where $\mathbf{i}$ is the intersection pixel and
($\mathbf{a},\mathbf{b},\mathbf{c},\mathbf{d}$) are the new recorded ends from
the intersection replacement. A visual example of this optimization can be
seen in Fig. 3(d) where out of the 3 possible combinations of paths, the one
that minimizes total curvature is selected. With the paths through
intersections properly precomputed, the skeleton pixel traversals to obtain
each DLO’s centerline can now take place. Pseudocode pertaining to the full
pipeline can be seen in Alg. 1.

###  III-G Computing DLO Radii and Crossing Order

The final step of the pipeline concerns computing the pixel radii of the DLOs
(to create the optional segmentation masks) as well as ascertaining which DLO
is resting on top of the other at intersections. To solve both problems, we
use modified versions of FASTDLO’s solutions [14].

Similar to FASTDLO, to compute the radii of the DLOs, we first perform a
distance transform on the binary mask $\mathbf{M}_{\textrm{dlo}}$, which
results in a new mask $\mathbf{M}_{\textrm{dist}}$ containing the closest
distance to a 0-value pixel. Diverting from FASTDLO, we then use the average
distance value of $\mathbf{M}_{\textrm{dist}}$ along a DLO’s centerline as the
radius of that DLO when creating segmentation masks. Using the average in this
manner allows for smooth segmentations even if the initial binary mask itself
is noisy. For segments of the DLO near the edges of the image, we use the
pixel specific values of $\mathbf{M}_{\textrm{dist}}$ themselves.

To compute crossing order at intersections, we use the precomputed optimal
paths created in Fig. 3(d). Crossing order can then be determined by computing
the sum of the standard deviations of the RGB channels of the pixels along
each path. The path that contains the lower sum is then assumed to be the one
on top. Though this solution from FASTDLO works fairly well, we noticed that
failures could occur due to glare along the centerline. This glare issue could
even cause failures for intersections with two completely different colored
DLOs. Therefore, to eliminate the influence of glare, we compute the standard
deviations of the intersection path pixels on a blurred image, rather than the
original input image.

##  IV EXPERIMENTAL RESULTS

Figure 4: Sample segmentations for the complex background dataset. Each row showcases segmentation results for a different image. From left to right, the first column showcases the original image. Columns 2-4 showcase Ariadne+, FASTDLO, and mBEST results, respectively. Finally, the fifth column showcases ground truth. TABLE II: Experimental results with DICE metric and average runtime Dataset | Segmentation | Image Size | DICE (%) | Average Runtime (FPS)
---|---|---|---|---
Type | Ariadne+ | FASTDLO | mBEST | Ariadne+ | FASTDLO | mBEST
Complex background | DCNN | $1000\times 750$ | 85.39 | 83.93 | 85.66 | 0.96 | 6.88 | 11.48
Simple background | DCNN | $672\times 896$ | 80.53 | 69.00 | 81.67 | 0.76 | 8.23 | 15.00
Simple background | Color | $672\times 896$ | 93.87 | 79.84 | 95.61 | 0.73 | 14.76 | 39.32
Figure 5: Sample segmentations for the simple background dataset. Each row
showcases segmentation results for a different image. From left to right, the
first column showcases the original image. Columns 2-4 showcase results for
Ariadne+, FASTDLO, and mBEST using DCNN segmentation, respectively. Columns
5-7 do the same except with color filter segmentation. Finally, ground truth
can be seen in the last column. Figure 6: Examples of mBEST with DCNN
segmentation working for scenes with complex DLO configurations and
backgrounds.

###  IV-A Experiments and Datasets

We conduct a total of 3 experiments using two datasets. The first experiment
consists of segmenting a dataset of 10 images from [14] where a series of
wires can be seen against complex backgrounds. Though the background is
complex, the wires themselves possess relatively smooth curvatures and no
self-loops. For this “complex background” dataset, the DLO segmentation method
used is FASTDLO’s DCNN model. Ground truth images for the complex background
dataset are created using the photo editting software Photopea [29].

The second and third experiments consists of segmenting a dataset of 50 images
consisting of up to two uniquely colored elastic rods against a simple black
background using DCNN and color filter segmentation, respectively. Using both
methods lets us observe the effects of the initial DLO segmentation on overall
performance. This “simple background” dataset consists of DLOs with highly
varying curvatures and numerous self-loops. Ground truth images for the simple
background dataset are created simply using the color filters themselves. We
focus mostly on images with a simple black background since the initial binary
mask segmentation is not a key focus of our algorithm. Having said this, we
show that mBEST still works for complex backgrounds in Fig. 6. All experiments
were run using an Intel i9-9900KF CPU and an NVIDIA RTX 2080 Ti GPU.

###  IV-B Baselines and Parameters

We test our algorithm against two state-of-the-art baselines: Ariadne+ [25]
and FASTDLO [14]. For complex background images, the number of superpixels for
Ariadne+ was set to 75, while simple background images used 200 superpixels.
Both these values were chosen as optimal after performing a parameter sweep on
each dataset.

For all experiments involving using the DCNN model, a pixel segmentation
threshold of 50 and 77 (0-255) were used for the simple and complex
backgrounds, respectively. Furthermore, though Ariadne+ has its own neural
network for initial segmentation of the DLOs, we exchange this for FASTDLO’s
DCNN model for consistency as well as the latter’s model having better
performance. Parameters for mBEST involve an intersection clustering threshold
of $\epsilon=75$ and minimum pixel length threshold of $\delta=40$ for complex
background images. For simple background images, ($\epsilon=45,\delta=25$) and
($\epsilon=40,\delta=25$) were used for segmentations with DCNN and color
filtering, respectively.

###  IV-C Results

For experiments, we report two key metrics. First, we look at segmentation
accuracy using the popular metric DICE. As all methods share the same initial
segmentation method for creating the binary mask, we do not treat the
background as one of the classes when computing the DICE scores. In addition
to this, we also report the average run times for each algorithm in terms of
frames per second (FPS). Both metrics can be seen listed in Table II for all
experiments.

Fig. 4 showcases results for the complex background dataset. For this dataset,
all three algorithms have relatively similar performance with mBEST slightly
edging out the others. As this dataset was originally used to evaluate
FASTDLO, it comes as no surprise that all three algorithms perform well,
especially since the configurations of the DLOs themselves are quite simple.
Though segmentation accuracy between the algorithms are similar, mBEST
achieves roughly a 12$\times$ runtime improvement over Ariadne+ and
1.67$\times$ runtime improvement over FASTDLO.

Where we start to see a large improvement in performance is the images
containing complex configurations of DLOs as shown in Fig. 5. For these
images, mBEST and Ariadne+ significantly outperform FASTDLO when using both
DCNN and color filtering approaches, with mBEST slightly beating the latter.
We suspect that Ariadne+’s use of curvature as one of its neural network
inputs helps it keep up with mBEST. Despite this, we can see that Ariadne+ is
still susceptible to choosing incorrect paths when encountering intersections
with high curvatures as shown in various examples in Fig. 5. In terms of
runtime, we see that mBEST has up to 2.66$\times$ speed boost over FASTDLO and
up to a staggering 53.86$\times$ speed boost over Ariadne+ when using color
filtering. These improvements decrease when using the DCNN model which
indicates that the DCNN passthrough takes a significant amount of computation
time on its own. Overall, we show that our physically insightful intersection
handling scheme can lead to more robust results when compared to blackbox
neural networks.

Figure 7: Example of a failure caused by inadequate selection of the
intersection clustering threshold $\epsilon$. (a) showcases the original image
where the orange dashed box encloses a (b1) set of three closely packed
intersections. (b2) showcases the skeleton pixel structure of the
intersections. (c1-c2) showcases the intersections improperly being replaced
when $\epsilon$ is set too high, while (d1-d2) showcases the intersections
properly handled.

###  IV-D Failure Cases

Though mBEST performs quite well for detecting complex tangles, it is still
reliant on correctly setting $\epsilon$. Improperly setting $\epsilon$ by even
a few pixels can result in improper handling of intersections as shown in Fig.
7 where $\epsilon=40$ results in several topological intersections being
incorrectly clustered as one. Removing this hyperparameter reliance will be a
focus in future works.

##  V CONCLUSION

In this work, we have introduced mBEST, an end-to-end pipeline for DLO
segmentation that improves upon SOTA in both accuracy and computational speed.
Through a wide variety of experiments, we have shown that mBEST can robustly
handle scenes with highly tangled DLOs by simply generating paths that
minimize cumulative bending energy. For future work, we would like to explore
solutions that take into consideration occlusions, strands touching in
parallel, and removing the hyperparameter dependency of $\epsilon$.

## References

  * [1] J. Sanchez, J.-A. Corrales, B.-C. Bouzgarrou, and Y. Mezouar, “Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey,” The International Journal of Robotics Research, vol. 37, no. 7, pp. 688–716, 2018.
  * [2] J. Schulman, A. Gupta, S. Venkatesan, M. Tayson-Frederick, and P. Abbeel, “A case study of trajectory transfer through non-rigid registration for a simplified suturing scenario,” in 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4111–4117, IEEE, 2013.
  * [3] H. Mayer, F. Gomez, D. Wierstra, I. Nagy, A. Knoll, and J. Schmidhuber, “A system for robotic heart surgery that learns to tie knots using recurrent neural networks,” Advanced Robotics, vol. 22, no. 13-14, pp. 1521–1537, 2008.
  * [4] A. Choi, D. Tong, M. K. Jawed, and J. Joo, “Implicit contact model for discrete elastic rods in knot tying,” Journal of Applied Mechanics, vol. 88, no. 5, 2021.
  * [5] J. Zhu, B. Navarro, P. Fraisse, A. Crosnier, and A. Cherubini, “Dual-arm robotic manipulation of flexible cables,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 479–484, IEEE, 2018.
  * [6] M. Yu, H. Zhong, and X. Li, “Shape control of deformable linear objects with offline and online learning of local linear deformation models,” in 2022 International Conference on Robotics and Automation (ICRA), pp. 1337–1343, IEEE, 2022.
  * [7] J. Iqbal, Z. H. Khan, and A. Khalid, “Prospects of robotics in food industry,” Food Science and Technology, vol. 37, pp. 159–165, 2017.
  * [8] D. Tong, A. Borum, and M. K. Jawed, “Automated stability testing of elastic rods with helical centerlines using a robotic system,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 1126–1133, 2021.
  * [9] B. Audoly, N. Clauvelin, and S. Neukirch, “Elastic knots,” Physical Review Letters, vol. 99, no. 16, p. 164301, 2007.
  * [10] M. K. Jawed, P. Dieleman, B. Audoly, and P. M. Reis, “Untangling the mechanics and topology in the frictional response of long overhand elastic knots,” Physical review letters, vol. 115, no. 11, p. 118302, 2015.
  * [11] V. P. Patil, J. D. Sandt, M. Kolle, and J. Dunkel, “Topological mechanics of knots and tangles,” Science, vol. 367, no. 6473, pp. 71–75, 2020.
  * [12] P. Johanns, P. Grandgeorge, C. Baek, T. G. Sano, J. H. Maddocks, and P. M. Reis, “The shapes of physical trefoil knots,” Extreme Mechanics Letters, vol. 43, p. 101172, 2021.
  * [13] T. G. Sano, P. Johanns, P. Grandgeorge, C. Baek, and P. M. Reis, “Exploring the inner workings of the clove hitch knot,” Extreme Mechanics Letters, p. 101788, 2022.
  * [14] A. Caporali, K. Galassi, R. Zanella, and G. Palli, “Fastdlo: Fast deformable linear objects instance segmentation,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 9075–9082, 2022.
  * [15] A. Keipour, M. Bandari, and S. Schaal, “Deformable one-dimensional object detection for routing and manipulation,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 4329–4336, 2022.
  * [16] M. Yan, Y. Zhu, N. Jin, and J. Bohg, “Self-supervised learning of state estimation for manipulating deformable linear objects,” IEEE robotics and automation letters, vol. 5, no. 2, pp. 2372–2379, 2020.
  * [17] D. De Gregorio, G. Palli, and L. Di Stefano, “Let’s take a walk on superpixels graphs: Deformable linear objects segmentation and model estimation,” 2018.
  * [18] D. Bolya, C. Zhou, F. Xiao, and Y. J. Lee, “Yolact: Real-time instance segmentation,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 9157–9166, 2019.
  * [19] D. Bolya, C. Zhou, F. Xiao, and Y. J. Lee, “Yolact++: Better real-time instance segmentation,” IEEE transactions on pattern analysis and machine intelligence, 2020.
  * [20] H. Chen, K. Sun, Z. Tian, C. Shen, Y. Huang, and Y. Yan, “Blendmask: Top-down meets bottom-up for instance segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8573–8581, 2020.
  * [21] Z. Tian, C. Shen, and H. Chen, “Conditional convolutions for instance segmentation,” in European conference on computer vision, pp. 282–298, Springer, 2020.
  * [22] M. Denninger, M. Sundermeyer, D. Winkelbauer, Y. Zidan, D. Olefir, M. Elbadrawy, A. Lodhi, and H. Katam, “Blenderproc,” arXiv preprint arXiv:1911.01911, 2019.
  * [23] W. Qiu and A. Yuille, “Unrealcv: Connecting computer vision to unreal engine,” in European Conference on Computer Vision, pp. 909–916, Springer, 2016.
  * [24] R. Zanella, A. Caporali, K. Tadaka, D. De Gregorio, and G. Palli, “Auto-generated wires dataset for semantic segmentation with domain-independence,” in 2021 International Conference on Computer, Control and Robotics (ICCCR), pp. 292–298, IEEE, 2021.
  * [25] A. Caporali, R. Zanella, D. D. Greogrio, and G. Palli, “Ariadne+: Deep learning–based augmented framework for the instance segmentation of wires,” IEEE Transactions on Industrial Informatics, vol. 18, no. 12, pp. 8607–8617, 2022.
  * [26] T. Y. Zhang and C. Y. Suen, “A fast parallel algorithm for thinning digital patterns,” Commun. ACM, vol. 27, p. 236â€“239, mar 1984.
  * [27] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “A density-based algorithm for discovering clusters in large spatial databases with noise,” in Knowledge Discovery and Data Mining, 1996.
  * [28] M. Bergou, M. Wardetzky, S. Robinson, B. Audoly, and E. Grinspun, “Discrete elastic rods,” in ACM SIGGRAPH 2008 Papers, SIGGRAPH ’08, (New York, NY, USA), Association for Computing Machinery, 2008.
  * [29] “Photopea.” https://www.photopea.com/.


††thanks: Z. X. and Z. C. contributed equally to this work.††thanks: Z. X. and
Z. C. contributed equally to this work.

#  Maximal quantum interaction between free electrons and photons

Zetao Xie  Department of Physics and HK Institute of Quantum Science and
Technology, The University of Hong Kong, Pokfulam, Hong Kong, China Zeling
Chen  Department of Physics and HK Institute of Quantum Science and
Technology, The University of Hong Kong, Pokfulam, Hong Kong, China Hao Li
Department of Applied Physics & Energy Sciences Institute, Yale University,
New Haven, CT 06520, USA Qinghui Yan  Interdisciplinary Center for Quantum
Information, State Key Laboratory of Modern Optical Instrumentation, College
of Information Science and Electronic Engineering, Zhejiang University,
Hangzhou 310027, China Solid State Institute and Faculty of Electrical and
Computer Engineering, Technion – Israel Institute of Technology, 32000 Haifa,
Israel Hongsheng Chen  Interdisciplinary Center for Quantum Information, State
Key Laboratory of Modern Optical Instrumentation, College of Information
Science and Electronic Engineering, Zhejiang University, Hangzhou 310027,
China Xiao Lin  Interdisciplinary Center for Quantum Information, State Key
Laboratory of Modern Optical Instrumentation, College of Information Science
and Electronic Engineering, Zhejiang University, Hangzhou 310027, China Ido
Kaminer  Solid State Institute and Faculty of Electrical and Computer
Engineering, Technion – Israel Institute of Technology, 32000 Haifa, Israel
Owen D. Miller<EMAIL_ADDRESS>Department of Applied Physics & Energy
Sciences Institute, Yale University, New Haven, CT 06520, USA Yi Yang
<EMAIL_ADDRESS>Department of Physics and HK Institute of Quantum Science and
Technology, The University of Hong Kong, Pokfulam, Hong Kong, China

###### Abstract

The emerging field of free-electron quantum optics enables electron-photon
entanglement and holds the potential for generating nontrivial photon states
for quantum information processing. Although recent experimental studies have
entered the quantum regime, rapid theoretical developments predict that
qualitatively unique phenomena only emerge beyond a certain interaction
strength. It is thus pertinent to identify the maximal electron-photon
interaction strength and the materials, geometries, and particle energies that
enable one to approach it. We derive an upper limit to the quantum vacuum
interaction strength between free electrons and single-mode photons, which
illuminates the conditions for the strongest interaction. Crucially, we obtain
an explicit energy selection recipe for electrons and photons to achieve
maximal interaction at arbitrary separations and identify two optimal regimes
favoring either fast or slow electrons over those with intermediate
velocities. We validate the limit by analytical and numerical calculations on
canonical geometries and provide near-optimal designs indicating the
feasibility of strong quantum interactions. Our findings offer fundamental
intuition for maximizing the quantum interaction between free electrons and
photons and provide practical design rules for future experiments on electron-
photon and electron-mediated photon-photon entanglement. They should also
enable the evaluation of key metrics for applications such as the maximum
power of free-electron radiation sources and the maximum acceleration gradient
of dielectric laser accelerators.

Free–electron–light interaction with structured optical environments leads to
photon emission through diverse radiation mechanisms, including Cherenkov
radiation, Smith-Purcell radiation, transition radiation, and incoherent
cathodoluminescence De Abajo (2010); Polman  _et al._ (2019); Garcia de Abajo
and Di Giulio (2021); Talebi (2018); Rivera and Kaminer (2020); Shiloh  _et
al._ (2022a); Roques-Carmes  _et al._ (2023). These interaction processes lie
at the heart of modern electron microscopy De Abajo (2010); Polman  _et al._
(2019); Garcia de Abajo and Di Giulio (2021), spectroscopy Varkentina  _et
al._ (2022), radiation sources Massuda  _et al._ (2018); Roques-Carmes  _et
al._ (2019); Yang  _et al._ (2023); Gong  _et al._ (2023), particle detection
Lin  _et al._ (2018, 2021), free–electron laser Pellegrini  _et al._ (2016),
accelerators Sapra  _et al._ (2020); Shiloh  _et al._ (2021); Fishman  _et
al._ (2023); Chlouba  _et al._ (2023), and biomedical imaging Roques-Carmes
_et al._ (2022); Kurman  _et al._ (2023). Recent theoretical advances have
shown that the quantum-photonic nature of light could become critical for
describing the interaction Kfir (2019); Di Giulio  _et al._ (2019). It was
further predicted that the interaction could shape and entangle electron and
photon wavefunctions Bendana  _et al._ (2011); Kfir (2019); Di Giulio and de
Abajo (2020); Ben Hayun  _et al._ (2021); Di Giulio  _et al._ (2021); Kfir
_et al._ (2021); Karnieli  _et al._ (2021); Konečná  _et al._ (2022), and
induce entanglement among electrons Kfir (2019) and photons Baranes  _et al._
(2022) themselves. The interaction has also been proposed for free-electronic
topological probes and quantum simulators Peng  _et al._ (2019); Pan  _et al._
(2022); Li  _et al._ (2024). Such a wide range of predicted quantum-optical
interactions now form the basis of free-electron quantum optics.

With the rapid developments of photon-induced near-field electron microscopy
(PINEM) Barwick  _et al._ (2009); Park  _et al._ (2010); García de Abajo  _et
al._ (2010) on photonic platforms Kfir  _et al._ (2020); Wang  _et al._
(2020); Henke  _et al._ (2021); Yang  _et al._ (2024), the predicted quantum
free-electron-light interaction has been realized in a few experiments Dahan
_et al._ (2021); Adiv  _et al._ (2023); Feist  _et al._ (2022). The crucial
parameters for describing these interactions are the dimensionless spontaneous
coupling strength $g_{\mathrm{Q}}$ with vacuum and its stimulated counterpart
$g$ with external optical fields ($g_{\mathrm{Q}}$ and $g$ are sometimes
respectively denoted as $\beta_{0}$ and $\beta$ in the literature, e.g. Ref.
Vanacore  _et al._ (2018); Polman  _et al._ (2019); Di Giulio  _et al._
(2019)). $g_{\mathrm{Q}}$ can be calculated via the spectral integration of
spontaneous electron energy loss Kfir (2019); Di Giulio  _et al._ (2019); Kfir
_et al._ (2021); Di Giulio  _et al._ (2021); Huang  _et al._ (2023a); Baranes
_et al._ (2023), and the two parameters relate to each other via
$g=\sqrt{N}g_{\mathrm{Q}}$ where $N$ is the number of photons injected into
the optical mode from a pump laser Kfir (2019); Di Giulio  _et al._ (2019). In
particular, the vacuum spontaneous processes exhibit intriguing prospects for
universal and ultrafast quantum computation and information processing because
they can create a variety of highly non-trivial quantum optical states under
designed interaction sequences Dahan  _et al._ (2023); Baranes  _et al._
(2023) and they are essential for the proposed hybrid free-electron-polariton
blockade Karnieli  _et al._ (2024a).

The key to these predictions is a strong quantum interaction with
$\absolutevalue{g_{\mathrm{Q}}}>1$. Intuitively, this regime indicates that
each emitted photon can cycle back to the electron and cause a cascade of
multi-photon emission and absorption. Crucially, this strong interaction
regime involves higher-order quantum electrodynamics processes and can be
understood as inducing a unique type of optical nonlinearity. Strong optical
nonlinearities are being pursued in all fields of photonics, with the goal of
reaching single-photon nonlinearities. Thus, a question of fundamental
importance is whether electron-photon interactions can reach this regime.
Answering this question requires an upper bound to the quantum interaction
strength $g_{\mathrm{Q}}$ in arbitrary photonic environments.

In this work, we derive universal limits to the quantum interaction strength
$g_{\mathrm{Q}}$ between free electrons and single-mode photons and provide
analytical and numerical validations. We find that the limits allow for the
strong quantum interaction condition $\absolutevalue{g_{\mathrm{Q}}}>1$, which
is feasible in near-optimal designs under realistic conditions. Limits for
both line and point electrons are obtained for arbitrary geometries, relying
only on material electrostatic constants, electron velocity, and electron-
structure separation. Intriguingly, the limits produce an explicit criterion
for selecting electron and photon energy under certain electron-structure
separations and give rise to two optimal interaction regimes, one favoring
fast electrons and the other favoring slow electrons. We evaluate the limits
across the electromagnetic spectrum and reveal the dichotomy of fast and slow
electrons: the former is advantageous for high-energy photons, while the
latter predominates towards the long-wavelength regime.

Figure 1: Illustration of the theoretical framework. Schematic of an electron
beam with velocity $\beta=v/c$ traveling close to an arbitrary structure at a
separated distance $d$. The structure is defined by material permittivity
$\epsilon(\mathbf{r},\omega)$ within a volume $V$. The electron (e-) can
become an entangled state with its emitted photons (ph) through spontaneous or
stimulated processes. These quantized emission events are able to be
identified via EELS and PINEM, respectively. The inset Feynman diagrams show
the corresponding physical processes and represent the interaction between
electrons and photons can be described by a spontaneous quantum strength
$g_{\mathrm{Q}}$ in EELS and a stimulated quantum strength $\mathit{g}$ in
PINEM. The spontaneous quantum interaction strength $g_{\mathrm{Q}}$ relates
to the electron-energy-loss spectral density by $\lvert
g_{\mathrm{Q}}\lvert^{2}$ = $\int\Gamma(\omega)\,\mathrm{d}\omega$, where the
integration window $\Delta\omega$ relates to the intended mode(s) that free
electrons couple to. The aforementioned interaction can be described by a
special scattering matrix “$\mathbbm{T}$”, which can be decomposed into a set
of Drude–Lorentz oscillators with matrix-valued coefficients
$\mathbbm{T}_{\mathit{i}}$.

We begin the analysis by considering an electron beam at normalized velocity
$\beta=v/c$, interacting with an arbitrary photonic structure characterized by
volume $V$ and material permittivity $\epsilon(\mathbf{r},\omega)$ in its near
field with a separation $d$, as shown in Fig. 1. Photon exchanges between the
electron and the structure can be strongly amplified when the photonic
structure supports modes with large spatial and spectral overlap with the
electron’s excitation field. In the framework of quantum optics, the joint
electron (e-)-photon (ph) state
$\ket{\Psi}=\ket{\psi^{\mathrm{e}}}\otimes\ket{\psi^{\mathrm{ph}}}=\sum_{n=0}^{\infty}\sum_{k=-\infty}^{+\infty}c_{n,k}\ket{\mathit{E}_{k},n}$
is a superposition state with electron energy
$\mathit{E}_{0}+\mathit{k}\hbar\omega$ and $\mathit{n}$ photons where $n$
denotes the photonic Fock states and $k$ denotes the electronic energy ladder.
For spontaneous processes with an electron of energy $\textit{E}_{0}$ in an
empty cavity, the initial state is
$\ket{\Psi_{\mathrm{i}}}=\ket{\textit{E}_{0},0}$, and the final state
$\ket{\Psi_{\mathrm{f}}}$ satisfies $n=-k$ by virtue of energy conservation.
The initial and final states are related by the scattering matrix, represented
as $\ket{\Psi_{\rm{i}}}=\hat{\mathit{S}}\ket{\Psi_{\rm{f}}}$, which is given
by Di Giulio  _et al._ (2019); Kfir (2019)

$\hat{\mathit{S}}=\exp\left(g_{\mathrm{Q}}\hat{\mathit{b}}^{\dagger}\hat{\mathit{a}}-g_{\mathrm{Q}}^{*}\hat{\mathit{b}}\hat{\mathit{a}}^{\dagger}\right),$
(1)

where $\hat{\mathit{a}}$, $\hat{\mathit{a}}^{\dagger}$ are the annihilation
and creation operators of the optical mode; $\hat{\mathit{b}}$,
$\hat{\mathit{b}}^{\dagger}$ denote the ladder operators for electron energy,
describing energy exchanges between the electron and the optical mode; and
$g_{\mathrm{Q}}$ is the dimensionless electron-photon interaction strength.
The strong-coupling regime of $\absolutevalue{g_{\mathrm{Q}}}>1$ indicates
that one free electron becomes more likely to emit more than one photon and
that this electron becomes quantum entangled with these photons. In the limit
of vacuum zero field Garcia-Vidal  _et al._ (2021), the quantum-optical theory
of PINEM coincides with the theory of electron energy loss spectroscopy
(EELS), which enables the calculation of $g_{\mathrm{Q}}$ from the spectrally-
integrated electron energy loss probability (see SI Sec. S1): $\lvert
g_{\mathrm{Q}}\lvert^{2}=\int\Gamma(\omega)\,\mathrm{d}\omega$ without
competing loss mechanisms Kfir  _et al._ (2021); Di Giulio  _et al._ (2021);
Huang  _et al._ (2023a); Baranes  _et al._ (2023). In stimulated processes
like PINEM, the general quantum framework still holds, whereas one should
replace $g_{\mathrm{Q}}$ with the stimulated strength $g$ amplified by the
input coherent state amplitude.

To impose a bound on $g_{\mathrm{Q}}$ and $g$, we treat free electrons
interacting with an arbitrary photonic environment as an electromagnetic
scattering problem (Fig. 1). Although this scattering perspective has
previously led to a single-frequency (zero-bandwidth) limit Yang  _et al._
(2018), it cannot translate to a $g_{\mathrm{Q}}$ limit straightforwardly
because of a divergent material metric for lossless materials. Alternatively,
a sum-rule approach can bound near-field interactions (e.g. power loss from a
stationary point dipole) Shim  _et al._ (2019); Miller (2023), but its
applicability to free electrons is hindered by the non-analytic behaviors of
their external fields near the complex-plane origin. Recently, a
$\mathbbm{T}$-matrix–based oscillator representation framework Zhang  _et al._
(2023) (where the $\mathbbm{T}$ operator relates the polarization fields with
incident fields Carminati and Schotland (2021); Chao  _et al._ (2022)) appears
ideal for analyzing free-electron–light scattering.

In the polarization-response representation, the frequency-dependent
extinction equals the work done by the incident field on the induced
polarization field in the scatterer,
$\textit{P}_{\rm{ext}}(\omega)=\left({\omega}/{2}\right)\mathrm{Im}\int_{\textit{V}}\mathbf{E}_{\rm{inc}}^{*}\cdot\mathbf{P}\,\mathrm{d}\mathit{V}$.
The polarization response is related to the incident field through a linear
operator, the “$\mathbbm{T}$ matrix” Carminati and Schotland (2021):
$\mathbf{P}(\mathbf{r})=\int_{V}\mathbbm{T}(\mathbf{r},\mathbf{r^{\prime}})\cdot\mathbf{E}_{\rm{inc}}(\mathbf{r^{\prime}})\,\mathrm{d}\mathbf{r^{\prime}}$,
or $\mathbf{p}=\mathbbm{T}\mathbf{e}_{\rm{inc}}$ in vector-matrix notation
that we use throughout. Causality and passivity produce a Kramers–Kronig
relation and sum-rule constraints for the $\mathbbm{T}$ matrix. These
properties enable the decomposition of the $\mathbbm{T}$ matrix into an
infinite sum of lossless Drude–Lorentz oscillators with resonance frequency
$\omega_{i}$ and highly constrained matrix-valued oscillator strengths
$\mathbbm{T}_{\mathit{i}}$ representing the only degrees of freedom Zhang  _et
al._ (2023) (see Sec. S2). Contributions from $\mathbbm{T}_{\mathit{i}}$ at
well-separated $\omega_{i}$ originate from different resonances. For
reciprocal systems, combining the zero-frequency sum rule, the oscillator
representation, and the domain monotonicity of polarizability Shim  _et al._
(2019) along the positive-frequency axis yields
$\omega\mathrm{Im}\mathbbm{T}(\omega)\leq\left({\pi\tau}/{2}\right)\sum_{i=1}^{\infty}\omega^{2}\mathbbm{T}_{i}\delta(\omega-\omega_{i})$,
which can reformulate the extinction as

$\displaystyle\textit{P}_{\rm{ext}}(\omega)$
$\displaystyle\leq\frac{\omega^{2}\pi\epsilon_{0}\tau}{4}\sum_{\textit{i}=1}^{\infty}\mathbf{e}_{\rm{inc}}^{\dagger}\mathbbm{T}_{\textit{i}}\mathbf{e}_{\rm{inc}}\delta(\omega-\omega_{\textit{i}}).$
(2)

The constant $\tau$ is a dimensionless electrostatic permittivity coefficient
relating the incident and total fields at zero frequency via
$\mathbf{P}(0)=\epsilon_{0}\tau\mathbf{E}_{\rm{inc}}(0)$ ($\epsilon_{0}$ is
the vacuum permittivity).

Based on this foundation, we next derive the upper limit to the vacuum
strength $g_{\mathrm{Q}}$ of free-electron-photon interaction. Under certain
interaction bandwidth described by a dimensionless window function
$\Theta(\omega;\omega_{0},\Delta\omega)\leq 1$ (peaked at $\omega_{0}$ with
bandwidth $\Delta\omega$), $g_{\mathrm{Q}}$ can be evaluated by integrating
the loss probability
$\Gamma(\omega)=\textit{P}_{\rm{ext}}(\omega)/\hbar\omega$:
$\absolutevalue{g_{\mathrm{Q}}}^{2}=\int\Gamma(\omega)\Theta(\omega;\omega_{0},\Delta\omega)\,\mathrm{d}\omega$.
By combining Eq. (2), $\absolutevalue{g_{\mathrm{Q}}}^{2}$ becomes

$\absolutevalue{g_{\mathrm{Q}}}^{2}\leq\frac{\pi\epsilon_{0}\tau}{4\hbar}\mathrm{Tr}\sum_{i=1}^{\infty}\mathbbm{T}_{i}\mathbbm{E}_{\textit{i}},$
(3)

a $\mathbbm{T}_{\mathit{i}}$-weighted sum of $\mathbbm{E}_{\textit{i}}\equiv$
$\omega_{i}\mathbf{e}_{\rm{inc}}(\omega_{i})\mathbf{e}_{\rm{inc}}^{\dagger}(\omega_{i})\Theta(\omega_{i};{\omega_{0},\Delta\omega})$
that incorporates the frequency of the emitted photons, the incident field,
and the window function. Because
$\omega\mathbf{e}_{\rm{inc}}(\omega)\mathbf{e}_{\rm{inc}}^{\dagger}(\omega)$
varies much slower than $\Theta(\omega;\omega_{0},\Delta\omega)$ under a
high-Q interaction scenario ($\omega_{0}\gg\Delta\omega$),
$\mathbbm{E}_{\textit{i}}$ also spectrally peaks at $\omega_{0}$ with
negligible detuning. Therefore, setting
$\mathbbm{T}_{\mathit{i}}=\mathbb{I}_{V}$ for the oscillator at $\omega_{0}$
and zero elsewhere will maximize Eq. (3), which simultaneously satisfies the
sum-rule constraint $\sum_{i=1}^{\infty}\mathbbm{T}_{i}=\mathbb{I}_{V}$, where
$\mathbb{I}_{V}$ is an identity matrix within the scatterer volume $V$. This
mathematically optimal choice corresponds to the physical condition of all of
the sum-rule-constrained polarization responses concentrating within the
frequency window of interest.

Applying the single-mode condition at $\omega_{0}$ to Eq. (3), we get

$\absolutevalue{g_{\mathrm{Q}}}^{2}\leq\frac{\pi\epsilon_{0}\omega_{0}\tau}{4\hbar}\int_{V}\absolutevalue{\mathbf{E}_{\rm{inc}}(\omega_{0})}^{2}\,\mathrm{d}V.$
(4)

This limit to $g_{\mathrm{Q}}$ thus becomes a simple product among physical
constants, photon frequency, electrostatic polarizability, and the integration
of the incident fields within the scatterer. We can obtain $\tau$ analytically
for free electrons with canonical geometries (e.g.
$\tau=2\epsilon_{1}(\epsilon_{2}-1)/(\epsilon_{2}+\epsilon_{1})$ for a half-
space, and $\tau=\epsilon_{1}(\epsilon_{2}-1)/{\epsilon_{2}}$ for a concentric
cylinder, both made of materials $\epsilon_{1}$ and $\epsilon_{2}$, where
$\epsilon_{1}$ and $\epsilon_{2}$ are electrostatic permittivity of materials
1 and 2, respectively; see Sec. S2.D). Evidently, $\tau$ is finite and bounded
for both dielectrics and perfect conductors (see Sec. S2.D).

Figure 2: Universal behaviors of the quantum interaction strength
$\absolutevalue{g_{\mathrm{Q}}}^{2}$. a. Line-electron upper limit Eq. (5) as
a function of electron energy and photon energy under a separation distance
$d=$100\text{\,}\mathrm{nm}$$. The quantum interaction strength is normalized
between 0 and 1 by the maximum
$\absolutevalue{g_{\mathrm{Q}}}^{2}_{\mathrm{2D}}$, and the white dashed line
denotes the cut-off photon energy of the optimal region. b. Photon energy cut-
off of line-electron limit Eq. (7) across various separation $d$. c. Point-
electron limit Eq. (6) for $d=\leavevmode\nobreak\ $100\text{\,}\mathrm{nm}$$.
The white dashed line highlights optimal choices of electron and photon
energies. d. The optimal electron and photon energies under different
separation $d$ [Eqs.(8) and (9)].  Figure 3:
$\absolutevalue{g_{\mathrm{Q}}}^{2}$ upper limit at typical photon frequencies
across the electromagnetic spectrum. a. Silicon at extreme ultraviolet (EUV)
$13.5\text{\,}\mathrm{nm}$. b. Silicon at telecommunication
$1.55\text{\,}\mathrm{\SIUnitSymbolMicro m}$. c. Perfect electric conductor
(PEC) at $1\text{\,}\mathrm{THz}$. d. PEC at microwave
$100\text{\,}\mathrm{GHz}$. We vary electron velocities $\beta$ and fix the
interaction length at $100\text{\,}\mathrm{\SIUnitSymbolMicro m}$. The black
dashed line represents ultra-relativistic
$6.8\text{\,}\mathrm{T}\mathrm{e}\mathrm{V}$ electrons. The light gray shading
indicates the $\absolutevalue{g_{\mathrm{Q}}}^{2}>1$ regime.

We further obtain shape-independent upper limits for line and point electrons
by enclosing arbitrary scatterers with canonical geometries. Assuming
translational invariance along one of its transverse directions and the
spatial coherence of the source current Remez  _et al._ (2019), a line
electron beam (sometimes also denoted as “sheet electron beam”) is a
simplified mathematical treatment used in many free-electron theory analyses
Luo  _et al._ (2003); Andrews and Brau (2004); Kumar and Kim (2006); Wang  _et
al._ (2016); Pan and Xu (2023).

Applying Eq. (4) to a line electron beam moving above a half-space (which can
enclose an arbitrary scatterer) at a separation d yields a shape-independent
limit for 2D line electrons (see Sec. S3):

$\absolutevalue{g_{\mathrm{Q}}}^{2}_{\mathrm{2D}}\leq\frac{\pi\mathit{q}^{2}\tau
L}{32\hbar\epsilon_{0}}\frac{\left(\textit{k}_{v,0}^{2}+\kappa_{\rho,0}^{2}\right)}{\omega_{0}\kappa_{\rho,0}}\mathrm{e}^{-2\kappa_{\rho,0}d},$
(5)

where $\mathit{q}$ is the charge density per unit transverse length, $L$ is
the interaction length, and $\textit{k}_{0}=\omega_{0}/c$,
$\textit{k}_{v,0}=\omega_{0}/v$,
$\kappa_{\rho,0}=\sqrt{\textit{k}_{v,0}^{2}-\textit{k}_{0}^{2}}=\textit{k}_{0}/\beta\gamma$
are the free-space, electron longitudinal and transverse wavevectors of the
target photon frequency $\omega_{0}$, respectively and
$\gamma=1/\sqrt{1-\beta^{2}}$ is the Lorentz factor.

Applying Eq. (4) to a point electron moving at the center of a concentric
hollow cylinder sector (which can enclose an arbitrary scatterer) of azimuthal
opening angle $\psi$, inner radius $d$, and outer radius $\infty$ leads to a
shape-independent limit for point electrons (see Sec. S4):

$\absolutevalue{g_{\mathrm{Q}}}^{2}_{\mathrm{3D}}\leq\frac{\alpha\tau\psi
L}{4c}\frac{\omega_{0}}{\beta^{2}}\kappa_{\rho,0}d\textit{K}_{0}(\kappa_{\rho,0}d)\textit{K}_{1}(\kappa_{\rho,0}d),$
(6)

where $\alpha$ is the fine-structure constant, $K_{\mathrm{n}}$ is the
modified Bessel function of the second kind. The limits Eqs. (4)-(6) are the
key analytical findings of the paper. They show that the maximum
$\absolutevalue{g_{\mathrm{Q}}}$ for a target emission center frequency
$\omega_{0}$ is solely determined by the electron velocity (via $\beta$ and
$\kappa_{\rho,0}$), the separation $d$, and the electrostatic polarizability
$\tau$ of the optical environment.

We illustrate the universal behaviors of the quantum upper limits in Fig. 2 to
depict the key features of Eqs. (5) and (6), ignoring the fractional pre-
factors comprising physical constants and structural coefficients. This way,
we can obtain a general energy selection recipe under arbitrary separations.

For line electrons (Fig. 2a), a photon energy cut-off condition (dashed line
in Fig. 2a; Sec. S5) appears:

$\kappa_{\rho,0}d=\omega_{0}d/c\beta\gamma=0.5,$ (7)

meaning that large $\absolutevalue{g_{\mathrm{Q}}}$ is possible and forbidden
below and above the cut-off photon energies, respectively, as indicated by the
dichotomy of the brightness and darkness to the left and right of the cut-off
line in Fig. 2a. As the separation distance reduces, the cut-off line
blueshifts in photon energy (Fig. 2b).

For point electrons (Fig. 2c), an optimal condition (dashed line in Fig. 2c
and see Sec. S6) illuminates the choice of electron and photon energies to
maximize their interaction:

$\textit{K}_{0}(\kappa_{\rho,0}d)\textit{K}_{1}(\kappa_{\rho,0}d)=\kappa_{\rho,0}d\left[\textit{K}_{0}(\kappa_{\rho,0}d)^{2}+\textit{K}_{1}(\kappa_{\rho,0}d)^{2}\right].$
(8)

Such an optimal condition is satisfied when

$\kappa_{\rho,0}d\approx 0.4064,$ (9)

which differs from line electrons’ cut-off condition Eq. (7). In particular,
two optimal regimes emerge and favor relativistic (top-right brightness in
Fig. 2c, ultraviolet $10\text{\,}\mathrm{eV}$ photons coupled to
$\gtrapprox$6\text{\,}\mathrm{MeV}$$ electrons when
$d=$100\text{\,}\mathrm{nm}$$) and non-relativistic (bottom-left brightness in
Fig. 2c, long-infrared $0.1\text{\,}\mathrm{eV}$ photons coupled to
$\lessapprox$4\text{\,}\mathrm{keV}$$ electrons when
$d=$100\text{\,}\mathrm{nm}$$) electrons, respectively. The former optimal
regime aligns with traditional free-electron physics, where high-energy
photons are generated with relativistic electrons, whereas the latter optimal
regime indicates the huge potential for quantum electron-light interaction
with slow electrons, which has been recognized in an early study on multi-
plasmon generation in graphene de Abajo (2013) and rapid development is being
made in more recent years Liu  _et al._ (2017); Yang  _et al._ (2018); Talebi
(2020); Liebtrau  _et al._ (2021); Huang  _et al._ (2023b); Karnieli and Fan
(2023); Chen  _et al._ (2023); Pan and Xu (2023); Eldar  _et al._ (2024);
Shiloh  _et al._ (2022b); Rasmussen  _et al._ (2024); Karnieli  _et al._
(2024b); Di Giulio  _et al._ (2024). For intermediate electron energies, the
optimal photon energies can be analogously defined using Eq. (8), although the
associated limit is lower than those for the two optima. The two optimal
regimes prevail under various separation distances (Fig. 2d).

We next contextualize the discussion by applying the universal feature above
to evaluate the feasibility of $\absolutevalue{g_{\mathrm{Q}}}>1$ at concrete
frequency windows. Under various electron velocities $\beta$ and separation
$d$, Fig. 3 shows the $\absolutevalue{g_{\mathrm{Q}}}^{2}$ upper limit for
point electrons [Eq. (6)] in four technologically relevant regimes, namely,
extreme ultraviolet (EUV) (Fig. 3a), near-infrared (Fig. 3b), Terahertz (Fig.
3c), and microwave (Fig. 3d). At the EUV $13.5\text{\,}\mathrm{nm}$ (Fig. 3a),
despite the high limits near the atomic-scale separations (under which quantum
tunneling and surface effects can become pronounced and modify the bounds here
Yang  _et al._ (2019); Gonçalves and García de Abajo (2023); Zhu  _et al._
(2016)), $\absolutevalue{g_{\mathrm{Q}}}>1$ is almost impossible for
$d\gtrapprox$10\text{\,}\mathrm{nm}$$ except with $\beta\gtrapprox 0.8$
relativistic electrons. At the telecom
$1.55\text{\,}\mathrm{\SIUnitSymbolMicro m}$ (Fig. 3b), whereas slow
electrons’ upper limits are higher at tens-of-nanometer or smaller separation,
fast electrons with $\beta\gtrapprox 0.3$ are superior at
$d\gtrapprox$100\text{\,}\mathrm{nm}$$, a typical separation in modern
grazing-interaction experiments. Towards the longer wavelengths, the
advantages of slow electrons are predominant.
$\absolutevalue{g_{\mathrm{Q}}}>1$ is only possible for $\beta\lessapprox 0.3$
and $\beta\lessapprox 0.1$ at $1\text{\,}\mathrm{THz}$ (Fig. 3c) and
$100\text{\,}\mathrm{GHz}$ (Fig. 3d), respectively. Despite the $\ln(\gamma)$
divergence of the limit Eq. (6) in electron energy, the slow-electron
advantage remains evident when comparing with electrons of
$6.8\text{\,}\mathrm{TeV}$ (dashed lines in Fig. 3c and d), the current
highest acceleration energy Ciccotelli  _et al._ (2023). Taken together, the
$\absolutevalue{g_{\mathrm{Q}}}>1$ goal is indeed challenging for EUV and
X-ray photons; nevertheless, slow electrons are promising from electrostatics
to the far-infrared, and fast electrons are advantageous from mid-infrared to
far ultraviolet; the evaluation is based on the current levels of electron
beam collimation and focusing.

Figure 4: Validations (a-c) of the theoretical upper limits and near-optimal
designs (d-f). a-b. Analytical calculations compared to the upper limits. A
line electron (a) with charge density $\mathit{q}$ = $1.6\times
10^{-19}\leavevmode\nobreak\ \rm{C/nm}$ and a point electron (b) interacting
with a half space made of air and a Drude metal
($\epsilon_{\rm{m}}=1-\omega_{\rm{p}}^{2}/\omega(\omega+\mathrm{i}\gamma_{\rm{m}})$,
$\omega_{\rm{p}}=$9.06\text{\,}\mathrm{eV}$$,
$\gamma_{\rm{m}}=$0.071\text{\,}\mathrm{eV}$$ Echarri  _et al._ (2021) for
gold). c. Numerical simulations compared to the upper limits. A point electron
moves above a Si-SiO2 waveguide (thickness $220\text{\,}\mathrm{nm}$ and width
$500\text{\,}\mathrm{nm}$) at a distance $d$. d. Near-optimal electron-photon
coupling design for slow electrons. An air-GaP (Gallium Phosphide) core-shell
structure with a $40\text{\,}\mathrm{nm}$ GaP shell is designed for non-
relativistic electrons. The property of polar dielectric GaP is described by a
Lorentz oscillator model,
$\epsilon_{\mathrm{d}}=\epsilon_{\infty}+\epsilon_{\infty}(\omega_{\mathrm{LO}}^{2}-\omega_{\mathrm{TO}}^{2})/(\omega_{\mathrm{TO}}^{2}-\omega^{2}-\mathrm{i}\omega\gamma_{\mathrm{d}})$,
where background permittivity $\epsilon_{\infty}=\leavevmode\nobreak\
$9.1\text{\,}$$ Van Vechten (1969), $\gamma_{\mathrm{d}}=\leavevmode\nobreak\
$1.25\text{\,}\mathrm{meV}$$, longitudinal optical $\omega_{\mathrm{LO}}$ and
transverse optical $\omega_{\mathrm{TO}}$ phonon frequencies are
$67.8\text{\,}\mathrm{meV}$ and $47.1\text{\,}\mathrm{meV}$, respectively
Caldwell  _et al._ (2015). e. Near-optimal electron-photon coupling design for
fast electrons. An air-aluminum core-shell structure with a
$13\text{\,}\mathrm{nm}$ aluminum (Al) Rakić (1995) shell is used for
relativistic electrons. The light gray shows the
$\absolutevalue{g_{\mathrm{Q}}}>1$ regime (circles: exact calculations; lines:
theoretical upper limit). f. The coupling ideality as a function of separation
(orange squares: GaP design; green circles: Al design). The interaction length
is fixed at $100\text{\,}\mathrm{\SIUnitSymbolMicro m}$ for all calculations.

We analytically and numerically validate the derived upper limits in Fig.
4a-c. We first consider two analytical scenarios: line (Fig. 4a) and point
(Fig. 4b) electron beams pass above a Drude-metal half-space in vacuum and
launch surface plasmon polaritons (SPPs). Both cases permit analytical
treatment of energy loss (see derivations in Sec. S7) to validate the line-
electron [Eq. (5)] and point-electron [Eq. (6)] upper limits, respectively. We
compute $g_{\mathrm{Q}}$ when electrons excite SPPs at frequency
$\omega_{\mathrm{SPP}}$ that vary under different electron velocities. The
definite integral of energy loss is calculated between ($\omega_{\rm{SPP}}$ \-
$\gamma_{\mathrm{m}}$) and ($\omega_{\rm{SPP}}$ \+ $\gamma_{\mathrm{m}}$) for
an interval of two full-width-half-maximum, where $\gamma_{\mathrm{m}}$ is
simultaneously the intrinsic material loss and the surface plasmon decay rate
of the Drude metal (see Sec. S8). In both the 2D and 3D cases, the analytical
results closely trail the upper limits at the considered separation. We also
numerically validate the upper limit using full-wave simulations of an
integrated waveguide setup in Fig. 4c, where an electron passes above a
$\rm{Si}$ waveguide embedded in $\rm{SiO_{2}}$ substrate; the numerical
coupling strength $g_{\mathrm{Q}}$ of the fundamental mode of the waveguide
falls short of the upper limit for about one order of magnitude.

Finally, we design two near-optimal hollow-core structures (Fig. 4d-f) to
verify the two predicted optimal regimes (two bright areas in Fig. 2b)
associated with slow and fast electrons, respectively. The core-shell
configuration is motivated by the sum-rule constraint constant $\tau$, which
is maximum when the largest possible near-field area of a source is occupied
by the polarizable material. For the slow-electron optimum, we utilize a
Lorentz polar dielectric gallium phosphide (GaP) with low optical phonon
frequencies to allow the coupling between $0.058\text{\,}\mathrm{eV}$ infrared
photons and non-relativistic $\beta=0.05$ electrons; for the fast-electron
optimum, we adopt Aluminium that features a high plasma frequency to achieve
an efficient interaction between $8.01\text{\,}\mathrm{eV}$ ultraviolet
photons and $\beta=0.995$ relativistic electrons. In both cases,
$\absolutevalue{g_{\mathrm{Q}}}>1$ is predicted possible by the upper limits
and is validated by the slightly trailing concrete designs at the separation
$100\text{\,}\mathrm{nm}$ (dashed vertical lines in Fig. 4d-e) and interaction
length $100\text{\,}\mathrm{\SIUnitSymbolMicro m}$ that are relevant to
current electron beam focusing and collimation techniques. In Fig. 4d and e,
GaP and Aluminium, two lossy materials, are used primarily to verify the two
optimal regimes (Fig. 2c), as their material properties facilitate the
coupling of electrons and photons at the desired energy. Alternatively,
lossless platforms (e.g. Fig. 4c) can be employed to maintain the entanglement
of the interaction.

In addition, Fig. 4f depicts the coupling ideality of the two near-optimal
designs as the function of separation. The ideality of the target mode m is
defined as
$I_{m}=\absolutevalue{\mathit{g}_{\rm{Qu,}\mathit{m}}}^{2}/\sum_{\textit{i}}\absolutevalue{\mathit{g}_{\rm{Qu,}\mathit{i}}}^{2}$,
which describes the fraction of coupling into the designated mode among
exciations of all modes Di Giulio  _et al._ (2019); Huang  _et al._ (2023a).
The ideality of the slow-electron design (Fig. 4d; orange squares in Fig. 4f)
grows monotonically with the considered separation, reaching more than 95%
under $\textit{d}>\leavevmode\nobreak\ $500\text{\,}\mathrm{nm}$$ due to the
dominance of the fundamental mode. Meanwhile, the fast-electron design (Fig.
4e; green circles in Fig. 4f) exhibits a good ideality $>80\%$ for a wide
range of separations, including both near- and far-field of the
$8.01\text{\,}\mathrm{eV}$ photon.

In summary, we have theoretically derived a universal upper limit to the
quantum interaction strength $g_{\mathrm{Q}}$ between free electrons and
single-mode photons. The limit allows us to evaluate the feasibility of
achieving the $\absolutevalue{g_{\mathrm{Q}}}>1$ strong interaction condition
over all possible designs across the electromagnetic spectrum without
exhaustive computational optimization. Under arbitrary separations, the limits
identify two optimal interaction regimes, showing how to select electron and
photon energy to maximize their interaction. Notably, the $g_{\mathrm{Q}}$
limit developed here can directly translate to a limit for the stimulated
strength $g$, scaled by the coherent state amplitude. There still exists some
gap between the limits and the performance of prevalent structures, indicating
the potential for further tightening the limits and improving designs to
approach them.

The limits derived here hinge on reciprocity, under which the skew-symmetric
part of the $\mathbbm{T}$-matrix vanishes, and the sum rule of the real-
symmetric part of $\mathbbm{T}$ can be applied; it is thus pertinent to seek
the generalization of the current limits to nonreciprocal free-electron
systems Yu  _et al._ (2021); Prudêncio and Silveirinha (2018); Fallah  _et
al._ (2021) and explore the possibility for higher $g_{\mathrm{Q}}$ therein.
As the electron-structure separation increases into the far field, electron
energy loss undergoes exponential decay, and its spectral composition can
become multimodal. This scenario may violate the single-mode condition,
decrease the coupling ideality, and cause an apparent breakdown of the limits
developed here because the spectral integration of energy loss no longer
corresponds to the $\absolutevalue{g_{\mathrm{Q}}}^{2}$ of a particular mode
of interest (e.g. when the tail of the zero-loss peak dominates; see Sec. S9).
Therefore, it would also be of interest to generalize the present work into
the multimodal condition. The findings here offer an intuitive understanding
of how to maximize the quantum interaction between free electrons and photons.
They also provide practical design guidelines for future experiments that aim
to achieve strong entanglement between electrons and photons, as well as
photon entanglement mediated by free electrons.

_Acknowledgments._ We thank J. García de Abajo, F. J. García-Vidal, Yixin Sha,
and Zhexin Zhao for fruitful discussions.

Note added. During the completion of this manuscript, we became aware of
related work Zhao (2024).

## References

  * De Abajo (2010) F. G. De Abajo, Reviews of Modern Physics 82, 209 (2010).
  * Polman  _et al._ (2019) A. Polman, M. Kociak, and F. J. García de Abajo, Nature Materials 18, 1158 (2019).
  * Garcia de Abajo and Di Giulio (2021) F. J. Garcia de Abajo and V. Di Giulio, ACS Photonics 8, 945 (2021).
  * Talebi (2018) N. Talebi, Advances in Physics: X 3, 1499438 (2018).
  * Rivera and Kaminer (2020) N. Rivera and I. Kaminer, Nature Reviews Physics 2, 538 (2020).
  * Shiloh  _et al._ (2022a) R. Shiloh, N. Schönenberger, Y. Adiv, R. Ruimy, A. Karnieli, T. Hughes, R. J. England, K. J. Leedle, D. S. Black, Z. Zhao, _et al._ , Advances in Optics and Photonics 14, 862 (2022a).
  * Roques-Carmes  _et al._ (2023) C. Roques-Carmes, S. E. Kooi, Y. Yang, N. Rivera, P. D. Keathley, J. D. Joannopoulos, S. G. Johnson, I. Kaminer, K. K. Berggren, and M. Soljačić, Applied Physics Reviews 10 (2023).
  * Varkentina  _et al._ (2022) N. Varkentina, Y. Auad, S. Y. Woo, A. Zobelli, L. Bocher, J.-D. Blazit, X. Li, M. Tencé, K. Watanabe, T. Taniguchi, _et al._ , Science Advances 8, eabq4947 (2022).
  * Massuda  _et al._ (2018) A. Massuda, C. Roques-Carmes, Y. Yang, S. E. Kooi, Y. Yang, C. Murdia, K. K. Berggren, I. Kaminer, and M. Soljačić, ACS Photonics 5, 3513 (2018).
  * Roques-Carmes  _et al._ (2019) C. Roques-Carmes, S. E. Kooi, Y. Yang, A. Massuda, P. D. Keathley, A. Zaidi, Y. Yang, J. D. Joannopoulos, K. K. Berggren, I. Kaminer, _et al._ , Nature Communications 10, 3176 (2019).
  * Yang  _et al._ (2023) Y. Yang, C. Roques-Carmes, S. E. Kooi, H. Tang, J. Beroz, E. Mazur, I. Kaminer, J. D. Joannopoulos, and M. Soljačić, Nature 613, 42 (2023).
  * Gong  _et al._ (2023) Z. Gong, J. Chen, R. Chen, X. Zhu, C. Wang, X. Zhang, H. Hu, Y. Yang, B. Zhang, H. Chen, _et al._ , Proceedings of the National Academy of Sciences 120, e2306601120 (2023).
  * Lin  _et al._ (2018) X. Lin, S. Easo, Y. Shen, H. Chen, B. Zhang, J. D. Joannopoulos, M. Soljačić, and I. Kaminer, Nature Physics 14, 816 (2018).
  * Lin  _et al._ (2021) X. Lin, H. Hu, S. Easo, Y. Yang, Y. Shen, K. Yin, M. P. Blago, I. Kaminer, B. Zhang, H. Chen, _et al._ , Nature Communications 12, 5554 (2021).
  * Pellegrini  _et al._ (2016) C. Pellegrini, A. Marinelli, and S. Reiche, Reviews of Modern Physics 88, 015006 (2016).
  * Sapra  _et al._ (2020) N. V. Sapra, K. Y. Yang, D. Vercruysse, K. J. Leedle, D. S. Black, R. J. England, L. Su, R. Trivedi, Y. Miao, O. Solgaard, _et al._ , Science 367, 79 (2020).
  * Shiloh  _et al._ (2021) R. Shiloh, J. Illmer, T. Chlouba, P. Yousefi, N. Schönenberger, U. Niedermayer, A. Mittelbach, and P. Hommelhoff, Nature 597, 498 (2021).
  * Fishman  _et al._ (2023) T. Fishman, U. Haeusler, R. Dahan, M. Yannai, Y. Adiv, T. L. Abudi, R. Shiloh, O. Eyal, P. Yousefi, G. Eisenstein, _et al._ , Nature Communications 14, 3687 (2023).
  * Chlouba  _et al._ (2023) T. Chlouba, R. Shiloh, S. Kraus, L. Brückner, J. Litzel, and P. Hommelhoff, Nature 622, 476 (2023).
  * Roques-Carmes  _et al._ (2022) C. Roques-Carmes, N. Rivera, A. Ghorashi, S. E. Kooi, Y. Yang, Z. Lin, J. Beroz, A. Massuda, J. Sloan, N. Romeo, _et al._ , Science 375, eabm9293 (2022).
  * Kurman  _et al._ (2023) Y. Kurman, N. Lahav, R. Schuetz, A. Shultzman, C. Roques-Carmes, A. Lifshits, S. Zaken, T. Lenkiewicz, R. Strassberg, O. Beer, _et al._ , arXiv preprint arXiv:2302.01300 (2023).
  * Kfir (2019) O. Kfir, Physical Review Letters 123, 103602 (2019).
  * Di Giulio  _et al._ (2019) V. Di Giulio, M. Kociak, and F. J. G. de Abajo, Optica 6, 1524 (2019).
  * Bendana  _et al._ (2011) X. Bendana, A. Polman, and F. J. García de Abajo, Nano Letters 11, 5099 (2011).
  * Di Giulio and de Abajo (2020) V. Di Giulio and F. J. G. de Abajo, Optica 7, 1820 (2020).
  * Ben Hayun  _et al._ (2021) A. Ben Hayun, O. Reinhardt, J. Nemirovsky, A. Karnieli, N. Rivera, and I. Kaminer, Science Advances 7, eabe4270 (2021).
  * Di Giulio  _et al._ (2021) V. Di Giulio, O. Kfir, C. Ropers, and F. J. Garcia de Abajo, ACS Nano 15, 7290 (2021).
  * Kfir  _et al._ (2021) O. Kfir, V. Di Giulio, F. J. G. de Abajo, and C. Ropers, Science Advances 7, eabf6380 (2021).
  * Karnieli  _et al._ (2021) A. Karnieli, N. Rivera, A. Arie, and I. Kaminer, Science Advances 7, eabf8096 (2021).
  * Konečná  _et al._ (2022) A. Konečná, F. Iyikanat, and F. J. García de Abajo, Science Advances 8, eabo7853 (2022).
  * Baranes  _et al._ (2022) G. Baranes, R. Ruimy, A. Gorlach, and I. Kaminer, npj Quantum Information 8, 32 (2022).
  * Peng  _et al._ (2019) S. Peng, N. J. Schilder, X. Ni, J. Van De Groep, M. L. Brongersma, A. Alù, A. B. Khanikaev, H. A. Atwater, A. Polman, _et al._ , Physical Review Letters 122, 117401 (2019).
  * Pan  _et al._ (2022) Y. Pan, B. Zhang, and D. Podolsky, arXiv preprint arXiv:2207.07010 (2022).
  * Li  _et al._ (2024) J. Li, Y. Fang, Y. Liu, _et al._ , Physical Review Letters 132, 073801 (2024).
  * Barwick  _et al._ (2009) B. Barwick, D. J. Flannigan, and A. H. Zewail, Nature 462, 902 (2009).
  * Park  _et al._ (2010) S. T. Park, M. Lin, and A. H. Zewail, New Journal of Physics 12, 123028 (2010).
  * García de Abajo  _et al._ (2010) F. J. García de Abajo, A. Asenjo-Garcia, and M. Kociak, Nano Letters 10, 1859 (2010).
  * Kfir  _et al._ (2020) O. Kfir, H. Lourenço-Martins, G. Storeck, M. Sivis, T. R. Harvey, T. J. Kippenberg, A. Feist, and C. Ropers, Nature 582, 46 (2020).
  * Wang  _et al._ (2020) K. Wang, R. Dahan, M. Shentcis, Y. Kauffmann, A. Ben Hayun, O. Reinhardt, S. Tsesses, and I. Kaminer, Nature 582, 50 (2020).
  * Henke  _et al._ (2021) J.-W. Henke, A. S. Raja, A. Feist, G. Huang, G. Arend, Y. Yang, F. J. Kappert, R. N. Wang, M. Möller, J. Pan, _et al._ , Nature 600, 653 (2021).
  * Yang  _et al._ (2024) Y. Yang, J.-W. Henke, A. S. Raja, F. J. Kappert, G. Huang, G. Arend, Z. Qiu, A. Feist, R. N. Wang, A. Tusnin, _et al._ , Science 383, 168 (2024).
  * Dahan  _et al._ (2021) R. Dahan, A. Gorlach, U. Haeusler, A. Karnieli, O. Eyal, P. Yousefi, M. Segev, A. Arie, G. Eisenstein, P. Hommelhoff, _et al._ , Science 373, eabj7128 (2021).
  * Adiv  _et al._ (2023) Y. Adiv, H. Hu, S. Tsesses, R. Dahan, K. Wang, Y. Kurman, A. Gorlach, H. Chen, X. Lin, G. Bartal, _et al._ , Physical Review X 13, 011002 (2023).
  * Feist  _et al._ (2022) A. Feist, G. Huang, G. Arend, Y. Yang, J.-W. Henke, A. S. Raja, F. J. Kappert, R. N. Wang, H. Lourenço-Martins, Z. Qiu, _et al._ , Science 377, 777 (2022).
  * Vanacore  _et al._ (2018) G. M. Vanacore, I. Madan, G. Berruto, K. Wang, E. Pomarico, R. Lamb, D. McGrouther, I. Kaminer, B. Barwick, F. J. García de Abajo, _et al._ , Nature Communications 9, 2694 (2018).
  * Huang  _et al._ (2023a) G. Huang, N. J. Engelsen, O. Kfir, C. Ropers, and T. J. Kippenberg, PRX Quantum 4, 020351 (2023a).
  * Baranes  _et al._ (2023) G. Baranes, S. Even-Haim, R. Ruimy, A. Gorlach, R. Dahan, A. A. Diringer, S. Hacohen-Gourgy, and I. Kaminer, Physical Review Research 5, 043271 (2023).
  * Dahan  _et al._ (2023) R. Dahan, G. Baranes, A. Gorlach, R. Ruimy, N. Rivera, and I. Kaminer, Physical Review X 13, 031001 (2023).
  * Karnieli  _et al._ (2024a) A. Karnieli, S. Tsesses, R. Yu, N. Rivera, A. Arie, I. Kaminer, and S. Fan, PRX Quantum 5, 010339 (2024a).
  * Garcia-Vidal  _et al._ (2021) F. J. Garcia-Vidal, C. Ciuti, and T. W. Ebbesen, Science 373, eabd0336 (2021).
  * Yang  _et al._ (2018) Y. Yang, A. Massuda, C. Roques-Carmes, S. E. Kooi, T. Christensen, S. G. Johnson, J. D. Joannopoulos, O. D. Miller, I. Kaminer, and M. Soljačić, Nature Physics 14, 894 (2018).
  * Shim  _et al._ (2019) H. Shim, L. Fan, S. G. Johnson, and O. D. Miller, Physical Review X 9, 011043 (2019).
  * Miller (2023) O. D. Miller, Fundamental limits to near-field optical response, in  _Advances in Near-Field Optics_, edited by R. Gordon (Springer International Publishing, Cham, 2023) pp. 25–85.
  * Zhang  _et al._ (2023) L. Zhang, F. Monticone, and O. D. Miller, Nature Communications 14, 7724 (2023).
  * Carminati and Schotland (2021) R. Carminati and J. C. Schotland,  _Principles of Scattering and Transport of Light_ (Cambridge University Press, 2021).
  * Chao  _et al._ (2022) P. Chao, B. Strekha, R. Kuate Defo, S. Molesky, and A. W. Rodriguez, Nature Reviews Physics 4, 543 (2022).
  * Remez  _et al._ (2019) R. Remez, A. Karnieli, S. Trajtenberg-Mills, N. Shapira, I. Kaminer, Y. Lereah, and A. Arie, Physical Review Letters 123, 060401 (2019).
  * Luo  _et al._ (2003) C. Luo, M. Ibanescu, S. G. Johnson, and J. Joannopoulos, Science 299, 368 (2003).
  * Andrews and Brau (2004) H. Andrews and C. Brau, Physical Review Special Topics-Accelerators and Beams 7, 070701 (2004).
  * Kumar and Kim (2006) V. Kumar and K.-J. Kim, Physical Review E 73, 026501 (2006).
  * Wang  _et al._ (2016) Z. Wang, K. Yao, M. Chen, H. Chen, and Y. Liu, Physical Review Letters 117, 157401 (2016).
  * Pan and Xu (2023) D. Pan and H. Xu, Physical Review Letters 130, 186901 (2023).
  * de Abajo (2013) F. J. G. de Abajo, ACS Nano 7, 11409 (2013).
  * Liu  _et al._ (2017) F. Liu, L. Xiao, Y. Ye, M. Wang, K. Cui, X. Feng, W. Zhang, and Y. Huang, Nature Photonics 11, 289 (2017).
  * Talebi (2020) N. Talebi, Physical Review Letters 125, 080401 (2020).
  * Liebtrau  _et al._ (2021) M. Liebtrau, M. Sivis, A. Feist, H. Lourenço-Martins, N. Pazos-Pérez, R. A. Alvarez-Puebla, F. J. G. de Abajo, A. Polman, and C. Ropers, Light: Science & Applications 10, 82 (2021).
  * Huang  _et al._ (2023b) S. Huang, R. Duan, N. Pramanik, J. S. Herrin, C. Boothroyd, Z. Liu, and L. J. Wong, Nature Photonics 17, 224 (2023b).
  * Karnieli and Fan (2023) A. Karnieli and S. Fan, Science Advances 9, eadh2425 (2023).
  * Chen  _et al._ (2023) J. Chen, R. Chen, F. Tay, Z. Gong, H. Hu, Y. Yang, X. Zhang, C. Wang, I. Kaminer, H. Chen, _et al._ , Physical Review Letters 131, 113002 (2023).
  * Eldar  _et al._ (2024) M. Eldar, Z. Chen, Y. Pan, and M. Krüger, Physical Review Letters 132, 035001 (2024).
  * Shiloh  _et al._ (2022b) R. Shiloh, T. Chlouba, and P. Hommelhoff, Physical Review Letters 128, 235301 (2022b).
  * Rasmussen  _et al._ (2024) T. P. Rasmussen, Á. R. Echarri, J. D. Cox, and F. J. G. de Abajo, Science Advances 10, eadn6312 (2024).
  * Karnieli  _et al._ (2024b) A. Karnieli, C. Roques-Carmes, N. Rivera, and S. Fan, arXiv preprint arXiv:2403.13071 (2024b).
  * Di Giulio  _et al._ (2024) V. Di Giulio, E. Akerboom, A. Polman, and F. de Abajo, arXiv preprint arXiv:2403.15823 (2024).
  * Yang  _et al._ (2019) Y. Yang, D. Zhu, W. Yan, A. Agarwal, M. Zheng, J. D. Joannopoulos, P. Lalanne, T. Christensen, K. K. Berggren, and M. Soljačić, Nature 576, 248 (2019).
  * Gonçalves and García de Abajo (2023) P. Gonçalves and F. J. García de Abajo, Nano Letters (2023).
  * Zhu  _et al._ (2016) W. Zhu, R. Esteban, A. G. Borisov, J. J. Baumberg, P. Nordlander, H. J. Lezec, J. Aizpurua, and K. B. Crozier, Nature Communications 7, 11495 (2016).
  * Ciccotelli  _et al._ (2023) A. Ciccotelli, R. B. Appleby, F. Cerutti, K. Bilko, L. S. Esposito, R. G. Alia, A. Lechner, and A. Tsinganis, Physical Review Accelerators and Beams 26, 061002 (2023).
  * Echarri  _et al._ (2021) A. R. Echarri, P. Gonçalves, C. Tserkezis, F. J. G. de Abajo, N. A. Mortensen, and J. D. Cox, Optica 8, 710 (2021).
  * Van Vechten (1969) J. A. Van Vechten, Physical Review 182, 891 (1969).
  * Caldwell  _et al._ (2015) J. D. Caldwell, L. Lindsay, V. Giannini, I. Vurgaftman, T. L. Reinecke, S. A. Maier, and O. J. Glembocki, Nanophotonics 4, 44 (2015).
  * Rakić (1995) A. D. Rakić, Applied optics 34, 4755 (1995).
  * Yu  _et al._ (2021) R. Yu, A. Konečná, and F. J. G. de Abajo, Physical Review Letters 127, 157404 (2021).
  * Prudêncio and Silveirinha (2018) F. R. Prudêncio and M. G. Silveirinha, Physical Review B 98, 115136 (2018).
  * Fallah  _et al._ (2021) A. Fallah, Y. Kiasat, M. G. Silveirinha, and N. Engheta, Physical Review B 103, 214303 (2021).
  * Zhao (2024) Z. Zhao, arXiv preprint arXiv:2404.01221 (2024).