# Text Perceptron: Towards End-to-End Arbitrary-Shaped Text Spotting

Liang Qiao1 Sanli Tang1 Zhanzhan Cheng21 Yunlu Xu1 Yi Niu1 Shiliang Pu1 Fei
Wu2
1Hikvision Research Institute, China; 2Zhejiang University, China
{qiaoliang6, tangsanli, chengzhanzhan, xuyunlu, niuyi,
<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author.

###### Abstract

Many approaches have recently been proposed to detect irregular scene text and
achieved promising results. However, their localization results may not well
satisfy the following text recognition part mainly because of two reasons: 1)
recognizing arbitrary shaped text is still a challenging task, and 2)
prevalent non-trainable pipeline strategies between text detection and text
recognition will lead to suboptimal performances. To handle this
_incompatibility_ problem, in this paper we propose an end-to-end trainable
text spotting approach named Text Perceptron. Concretely, Text Perceptron
first employs an efficient segmentation-based text detector that learns the
latent text reading order and boundary information. Then a novel Shape
Transform Module (_abbr._ STM) is designed to transform the detected feature
regions into regular morphologies without extra parameters. It unites text
detection and the following recognition part into a whole framework, and helps
the whole network achieve global optimization. Experiments show that our
method achieves competitive performance on two standard text benchmarks, i.e.,
ICDAR 2013 and ICDAR 2015, and also obviously outperforms existing methods on
irregular text benchmarks SCUT-CTW1500 and Total-Text.

##  1 Introduction

Spotting scene text is a hot research topic due to its various applications
such as invoice recognition and road sign reading in advanced driver
assistance systems. With the advances of deep learning, many deep neural-
network-based methods (?; ?; ?; ?; ?) have been proposed for spotting text
from a natural image, and have achieved promising results.

Figure 1:  Illustration of the traditional pipelined text spotting process and
Text Perceptron. Sub-figure (a) is a traditional pipeline strategy by
combining text detection, rectification and recognition into a framework. Sub-
figure (b) is an end-to-end trainable text spotting approach by applying the
proposed STM. The black and red arrows mean the forward and backward
processing, respectively. The red points denote generated fiducial points
generated.

However, in the real-world, many texts appear in arbitrary layouts (_e.g._
multi-oriented or curved), which make quadrangle-based methods (?; ?; ?)
cannot be well adapted in many situations. Some works (?; ?; ?) began to focus
on irregular text localization by segmenting text masks as detection results
and achieved relatively good performance in terms of Intersection-over-Union
(IoU) evaluation. However, they still leave many challenges to the following
recognizing task. For example, a common pipeline of text spotting is to crop
the masked texts within bounding-box regions, and then adopt a recognition
model with rectification functions to generate final character sequences.
Unfortunately, such strategy decreases the robustness of text spotting mainly
in two aspects: 1) one needs to design extra rectification network, like
methods in (?) and (?), to transform irregular texts into regular ones. In
practice, it is hard to be optimized without human-labeled geometric ground
truth, and also introduces extra computational cost. 2) Pipelined text
spotting methods are not end-to-end trainable and result in suboptimal
performance because the errors from the recognition model cannot be utilized
for optimizing the text detector. In Figure 1(a), although the text detector
provides true positive results, the clipped text masks still lead to wrong
recognition results. We denote above problem _incompatibility_ between text
detection and recognition.

Recently, two methods were proposed for spotting irregular text in the end-to-
end manners. (?) proposed an end-to-end trainable network inspired by Mask-
RCNN (?), aiming at reading irregular text character-by-character. However,
this approach loses the context information among characters, and also
requires amounts of expenditure on character-level annotations. (?) attempted
to transform irregular text with a perspective ROI module, but this operation
has difficulty in handling some complicated distortions such as curved shapes.

These limitations motivate us to explore new and more effective method to spot
irregular scene text. Inspired by (?), thin-plate splines (_abbr._ TPS) (?)
may be a feasible approach to rectify various-shaped text into regular form
using a group of fiducial points. Although these points can be implicitly
learned from cropped rectangular text by a deep spatial transform network (?),
the learning process of fiducial points is hard to be optimized. As a result,
such methods are not robust especially for texts in some complex distortions.

In a more achievable way, we attempt to solve this problem as follows: 1)
explicitly finding out a group of reliable fiducial points over text regions
so that irregular text can be directly rectified by TPS, and 2) dynamically
tuning fiducial points by back-propagating errors from recognition to
detection. Specifically, we develop a Shape Transform Module (_abbr._ STM) to
build a robust irregular text spotter and eliminate the _incompatibility_
problem. STM integrates irregular text detection and recognition into an end-
to-end trainable model, and iteratively adjusts fiducial points to satisfy the
following recognition module. As shown in Figure 1(b), in the early training
stage, despite high IoU in detection evaluation, the transformed text regions
may not satisfy the recognition module. With end-to-end training, fiducial
points will be gradually adjusted to obtain better recognition results.

In this paper, we propose an end-to-end trainable irregular text spotter named
Text Perceptron which consists of three parts: 1) A segmentation-based
detection module which orderly describes a text region as four subregions: the
_center_ region, _head, tail_ and _top &bottom_ _boundary_ regions, detailed
in Section 3. Here, boundary information not only helps separate text regions
that are very close to each other, but also contributes to capture latent
reading-orders. 2) STM for iteratively generating potential fiducial points
and dynamically tuning their positions, which alleviates _incompatibility_
between text detection and recognition. 3) A sequence-based recognition module
for generating final character sequences.

Major contributions of this paper are listed as follows: 1) We design an
efficient order-aware text detector to extract arbitrary-shaped text. 2) We
develop the differentiable STM devoting to optimizing both detection and
recognition in an end-to-end trainable manner. 3) Extensive experiments show
that our method achieves competitive results on two regular text benchmarks,
and also significantly surpasses previous methods on two irregular text
benchmarks.

##  2 Related Works

Here, we briefly review the recent advances in text detection and end-to-end
text spotting.

###  2.1 Text Detection

Methods of text detection can usually be divided into two categories: anchor-
based methods and segmentation-based methods.

_Anchor-based methods_. These methods usually follows the technique of Faster
R-CNN (?) or SSD (?) that uses anchors to provide rectangular region
proposals. To overcome the significantly varying aspect ratios of texts, (?)
designed long default boxes and filters to enhance text detection, and then
(?) extended this work by generating quadrilateral boxes to fit the texts with
perspective distortions. (?) proposed a rotated regional proposal network to
enhance multi-oriented text detection. To detect arbitrary-shaped text, many
Mask RCNN (?)-based methods, _e.g._ , CSE (?), LOMO (?) and SPCNet (?), were
developed to capture irregular texts and achieved good performance.

_Segmentation-based methods_. These methods usually learn a global semantic
segmentation without region proposals, which is more efficient compared to
anchor-based methods. Segmentation can easily be used to describe text in
arbitrary shapes but highly relies on complicated post-processes to separate
different text instances. To solve this problem, (?) introduced boundary
semantic segmentation to reduce the efforts in post-proposing. EAST (?)
learned a shrink text region and directly regressed the multi-oriented
quadrilateral boxes from text pixels. (?) designed a series of overlapping
disks with different radii and orientations to describe arbitrary-shaped text
regions. (?) proposed a method that first generates text region masks with
various shrinkage ratios and then uses a progressive expansion algorithm to
produce the final text region masks. (?) predicted each text pixel and
assigned them with a regression value denoting the direction to its nearest
boundary to help separate different texts.

###  2.2 Text Spotting

Most of existing text-spotting methods (?; ?; ?) generally first localize each
text with a trained detector such as (?) and then recognize the cropped text
region with a sequence decoder (?). For sufficiently exploiting the
complementarity between detection and recognition, some works (?; ?; ?) were
proposed to jointly detect and recognize text instances in an end-to-end
trainable manner, which utilized the recognition information to optimize the
localization task. However, these methods are incapable of spotting arbitrary-
shaped text due to the irrationality of rectangles or quadrangles. To address
these problems, (?) adopted a perspective ROI transforming module to rectify
perspective text, but this operation still has difficulty in handling serious
curved text. (?) proposed an end-to-end text spotter inspired by Mask-RCNN for
detecting arbitrary-shaped text character-by-character, but this method loses
the context information among characters and also requires character-level
location annotations.

##  3 Methodology

###  3.1 Overview

Figure 2: The workflow of Text Perceptron. The black and red arrows separately
mean the forward and backward process.

We propose a text spotter named Text Perceptron whose overall architecture is
shown in Figure 2, which consists of three parts:

(1) The text detector adopts ResNet (?) and Feature Pyramid Network (_abbr_.
FPN) (?) as backbone, and is implemented by simultaneously learning three
tasks: an order-aware multiple-class semantic segmentation, a corner
regression, and a boundary offset regression. In this way, the text detector
can localize arbitrary-shaped text and achieve state of the art on text
detection.

(2) STM is responsible for uniting text detection and recognition into an end-
to-end trainable framework. This module iteratively generates fiducial points
on text boundaries based on the predicted score and geometry maps, and then
applies the differentiable TPS to rectify irregular text into regular form.

(3) The text recognizer is used to generate the predicted character sequences,
which can be any traditional sequence-based method, such as CRNN (?),
attention-based method (?).

###  3.2 Text Detection Module

#### Order-aware Semantic Segmentation

The text detector learns a global multi-class semantic segmentation, which is
much more efficient than those Mask-RCNN-based methods. Inspired by (?), we
introduce text boundary segmentation to separate different text instances.
Considering text with arbitrary shapes, we further category boundaries into
_head_ , _tail_ , and _top &bottom boundary_ types, respectively. In Figure 3,
the green, yellow, blue and pink regions separately denote the _head_ , _tail_
, _top &bottom_ boundaries and the _center_ text region. Here, _head_ and
_tail_ also capture potential information about text reading order (_e.g._ top
to bottom for vertical text). Therefore, we learn the text detector by
conducting the multi-class semantic segmentation task using several binary
Dices Coefficient Loss (?) (denoted by $\mathcal{L}_{cls}$).

#### Corner and Boundary Regressions

To boost the arbitrary-shaped segmentation performance as well as provide
position information for fiducial points, we integrate two other regression
tasks into the learning process, as shown in Figure 3 (c) and (d),

  * •

_Corner Regression._ For pixels in _head_ and _tail_ regions, we regress the
offsets (_e.g._ the $\Delta dx_{1},\Delta dy_{1},\Delta dx_{2}$ and $\Delta
dy_{2}$) to their corresponding two corner points, which is denoted by
$\mathcal{L}_{corner}$.

  * •

_Boundary Offset Regression._ For pixels in _center_ region, we regress the
vertical and horizontal offsets to their nearest boundaries (_e.g._ the
$\Delta dx_{1}^{\prime},\Delta dy_{1}^{\prime},\Delta dx_{2}^{\prime}$ and
$\Delta dy_{2}^{\prime}$), which is denoted by $\mathcal{L}_{boundary}$.

Here, we adopt a proximity regression strategy to solve the inaccurate large-
offset regression problem like in EAST (?). That is, the Corner Regressions
only regress their neighboring corresponding corners. In the Boundary Offset
Regression, we can simply ignore or lower the loss weights of regression value
generated from the larger side (_e.g._ $\Delta dx_{1}^{\prime},\Delta
dx_{2}^{\prime}$ for a horizontal text). In this way, our detector can well
describe the texts with very large width-height ratios. Both of two
regressions are trained with Smooth-L1 loss:

$\mathcal{L}_{corner}~{}or~{}\mathcal{L}_{boundary}=\left\\{\begin{array}[]{lr}0.5(\sigma
z)^{2}\qquad|z|<1/\sigma^{2}\\\ |z|-0.5/\sigma^{2}\quad\text{otherwise}\\\
\end{array}\right.,$ (1)

where $z$ is the geometry offset value, and $\sigma$ is a tunable parameter
(default by 3).

#### The Detection Inference

In the forward process, we generate predicted segmentation maps by orderly
overlaying the segmented _center_ , _head_ , _tail_ , and _top &bottom_
boundary feature maps. Subsequently, text instances can be found as connected-
regions of _center_ pixels. We see that all text instances are easily
separated by boundaries, and different _head_ (or _tail_) regions will also be
separated by _up &bottom_ boundary region. Therefore, each _center_ region can
be matched with a neighboring pair of _head_ and _tail_ region during the
pixel traversal process. Specifically, for text with more than 1 _head_ (or
_tail_) regions, we choose the one with the maximum area as its _head_ (or
_tail_). While for predicted _center_ text regions without corresponding
_head_ or _tail_ region, we just treat them as false positives and filter them
out.

#### Ground-Truth Generation

The process of ground-truth of segmentation and geometry map can be divided
into three steps, as shown in Figure 3.

Figure 3: The label generation process.

_(1) Identifying four corners._ We denote the 1st and 4th corners as the two
corners in the _head_ region, while the 2nd and 3rd corners are corresponding
to the _tail_ region, as shown in Figure 3(a). This weak-supervised
information is not provided by most of the datasets, but we found that in
general, polygon points $\\{P_{1}^{\prime},...,P_{M}^{\prime}\\}$ are usually
annotated from the left-top corner to the left-bottom corner in a clockwise
manner for text instances. Differently, for polygon annotations with a fixed
number of points like SCUT-CTW1500 (?), we can directly identify the four
corner points by their indexes. However, for annotations with varying number
of points like Total-Text (?), we can only obtain the 1st corner
($P_{1}^{\prime}$) and 4th corner ($P_{M}^{\prime}$). To search the 2nd and
3rd corners, we design a heuristic corner estimating strategy based on the
assumptions that 1) two boundaries neighboring _tail_ are nearly parallel, and
2) two neighbor interior angles of _tail_ are closed to $\frac{\pi}{2}$.
Therefore, the probable 2nd corner can be estimated as:

$\arg\min_{P_{i}^{\prime}}[\gamma(|\angle
P_{i}^{\prime}-\frac{\pi}{2}|+|\angle P_{i+1}^{\prime}-\frac{\pi}{2}|)+|\angle
P_{i}^{\prime}+\angle P_{i+1}^{\prime}-\pi|]$ (2)

where $\angle P_{i}^{\prime}$ is the degree of interior angle for polygon
point $P_{i}^{\prime}$, and $\gamma$ is a weighting parameter (default by
0.5). Then the point $P_{i+1}^{\prime}$ following $P_{i}^{\prime}$ is treated
as the 3-rd corner point. Specifically, for vertical text annotated from the
top-left corner, we reassign its top-right corner as the 1st key corner.

_(2) Generating score maps._ Figure 3(b) shows the generated score maps. We
firstly generate the _center_ text regions follows by their annotations and
then generate boundaries by referring to the shrink and expansion mechanism
used in (?). Differently, the _head_ and _tail_ score maps are generated by
only applying the shrink operation, which submerges part of the _center_
region. And _top &bottom_ boundary region is then generated by applying both
the expansion and shrink operations, which will partly submerge all of the
other regions. In this way, we need less effort on post-processing to separate
different text instances and it is easy to match their relative _head_ (or
_tail_) region with a _center_ region. Boundary widths are constrained as
$\delta$$\times$$minLen$, where $minLen$ is the minimum length of edges in the
text polygon and $\delta$ is a ratio parameter. Here, we set $\delta$=$0.2$
for _top &bottom_ boundaries and $\delta$=$0.3$ for _head_ and _tail_.

_(3) Generating geometry maps._ As mentioned in Corner and Boundary
Regression, pixels belonging to the _head_ region are assigned geometry offset
values in 4 channels ($\Delta dx_{1},\Delta dy_{1},\Delta dx_{2}$ and $\Delta
dy_{2}$) corresponding the 1st and 4th key corner, as shown in Figure 3(c).
Similarly, the geometry map of the _tail_ region is also formed in 4 channels.
The geometry values of the _center_ text region are computed as the horizontal
and vertical offsets to the nearest boundaries, shown as $\Delta
dx^{\prime}_{1},\Delta dy^{\prime}_{1},\Delta dx^{\prime}_{2}$ and $\Delta
dy^{\prime}_{2}$ in Figure 3(d).

###  3.3 Shape Transform Module

STM is designed to iteratively generate initial fiducial points around text
instances and transform text feature regions into regular shapes with the
supervision of following recognition.

#### Fiducial Points Generation

With the learned segmentation maps and geometry maps, we propose to generate
preset $2$$\times$$N$ potential fiducial points ($N$$\geq$$2$) for each text
instance, denoted as $\\{P_{1},...,P_{N},P_{N+1},...,P_{2\times N}\\}$, which
can be divided into two stages.

Figure 4: The fiducial points generation process.

_(1) Generating four corner points._ We first obtain the positions of four
corner fiducial points for each text feature region by averaging the
coordinate of pixels with their predicted offsets in corresponding boundaries.
Taking the 1-st corner point ($P_{1}$) as an example, it is computed based on
all pixels in the _head_ region $\mathcal{R}_{\mathcal{H}}$, and formalized by

$P_{1}=\left(\frac{\sum_{(x,y)\in\mathcal{R}_{\mathcal{H}}}(x+\Delta
dx)}{||\mathcal{R}_{\mathcal{H}}||},\frac{\sum_{(x,y)\in\mathcal{R}_{\mathcal{H}}}(y+\Delta
dy)}{||\mathcal{R}_{\mathcal{H}}||}\right)$ (3)

where $||.||$ means the number of pixels in $\mathcal{R}_{\mathcal{H}}$, and
$\Delta dx,\Delta dy$ mean the predicted corner offsets corresponding to
${P}_{1}$. The other three corner points ($P_{N}$ in
$\mathcal{R}_{\mathcal{H}}$, $P_{N+1}$, $P_{2\times N}$ in tail region
$\mathcal{R}_{\mathcal{T}}$) can be calculated similarly.

_(2) Generating other fiducial points._ After obtaining four corner fiducial
points, the other fiducial points can be located using a dichotomous method.
This strategy is suitable for any arbitrary shaped text even serious curved or
in different reading orders.

An example of the generation process is shown in Figure 4. We firstly connect
$P_{1}$ and $P_{N}$, and judge whether the connected line has a longer span in
horizontal direction or vertical direction. Without loss generality, if it has
a longer span in horizontal direction as shown, we calculate a middle point
$P_{\lfloor(N+1)/2\rfloor}$ between $P_{1}$ and $P_{N}$ whose x-coordinate
formed as:

$x_{mid}=\frac{\lceil(N-1)/2\rceil}{N-1}\times
P_{1,x}+\frac{\lfloor(N-1)/2\rfloor}{N-1}\times P_{N,x}$ (4)

Then we use the learned boundary offsets from detector to predict the
y-coordinate of $P_{\lfloor(N+1)/2\rfloor}$. Concretely, we define the _band_
region $\mathcal{B}_{i}$ as the part of the _center_ region $\mathcal{R}_{C}$:

$\mathcal{B}_{\lfloor\frac{1+N}{2}\rfloor}=\\{(x,y)\in\mathcal{R}_{\mathcal{C}}|x\in[x_{mid}-\Delta
ep,x_{mid}+\Delta ep]\\}$ (5)

where $\Delta ep$ defines the range of the band region (default by 3). Similar
to the generation of four corner fiducial points, we can use all pixels in the
corresponding band region to predict an average y-coordinate for this fiducial
point. Then, the coordinate of $P_{\lfloor(N+1)/2\rfloor}$ can be formed as:

$P_{\lfloor\frac{1+N}{2}\rfloor}=\left(x_{mid},\frac{\sum_{(x_{t},y_{t})\in\mathcal{B}_{\lfloor\frac{1+N}{2}\rfloor}}y_{t}+\Delta
dy^{\prime}_{t}}{||\mathcal{B}_{\lfloor\frac{1+N}{2}\rfloor}||}\right)$ (6)

where $\Delta dy^{\prime}_{t}$ is the learned boundary offset value to the
top-boundary ($\Delta dy^{\prime}_{1}$). This process can be iteratively
conducted using corresponding $\Delta dx^{\prime}_{t}$ or $\Delta
dy^{\prime}_{t}$ until all of the fiducial points be calculated. Similarly,
the fiducial points on the bottom boundary can be calculated by connecting
$P_{N+1}$ and $P_{2\times N}$ and using the same strategy.

#### Shape Transformation

With the generated potential fiducial points on text boundaries, we can
explicitly transform an irregular feature region $\mathcal{R}$ into a regular
form $\mathcal{R}^{*}$. Here, fiducial points are mapped into some preset
positions of the transformed feature map by directly applying TPS to the
original feature regions. Specifically, we transform all feature regions into
a region with width $W$ and height $H$:

$\mathcal{R}^{*}=TPS^{-1}(\mathcal{P},\mathcal{R}),$ (7)

where the fiducial point $P_{i}\in\mathcal{P}$ will be mapped into:

$P^{*}_{i}=\left\\{\begin{array}[]{lr}\left((i-1)\times\frac{H-2\times\Delta
w}{N-1}+\Delta w,~{}\Delta h\right),~{}1\leq i\leq N\\\ \left((2\times
N-i)\times\frac{H-2\times\Delta w}{N-1}+\Delta w,~{}H-\Delta h\right),\\\
~{}~{}~{}\qquad\qquad\qquad\qquad\qquad\qquad N<i\leq 2\times N\\\
\end{array}\right.$ (8)

where $\Delta w$ and $\Delta h$ are preset offsets (default by 0.1$\times W$
and 0.1$\times H$) to preserve space for fiducial points tuning.

Then, all text feature regions are packed into a batch and sent to the
following recognition part. Here, we assume that the final predicted character
strings $Y$ are generated as:

$Y=Recog(\mathcal{R}^{*}),$ (9)

where ‘$Recog$’ is the sequence recognition process.

#### Dynamically Finetuning Fiducial Points

The assumption here is that although text detector supervised by polygon
annotations can generate satisfying polygon masks, the results may not always
suitable for the following recognition. To avoid the suboptimal problem and
improve overall performance, Text Perceptron will back-propagate differences
from ‘$Recog$’ to each pixel value in $\mathcal{R}$ via STM, i.e.

$\Delta\mathcal{R}=\frac{\partial
Y}{\partial\mathcal{R}^{*}}\frac{\partial\mathcal{R}^{*}}{\partial\mathcal{R}}.$
(10)

Then we can calculate the adjustment values of $\mathcal{P}$ by

$\Delta\mathcal{P}=\frac{\partial
Y}{\partial\mathcal{R}^{*}}\frac{\partial\mathcal{R}^{*}}{\partial\mathcal{R}}\frac{\partial\mathcal{R}}{\partial\mathcal{P}}.$
(11)

Furthermore, we back-propagate $\Delta\mathcal{P}$ to the corresponding
geometry maps in _head_ , _tail_ and _band_ regions. Formally, for each pixel
$pi$, we have

$\Delta
pi=\Delta\hat{pi}+\frac{\Delta\mathcal{P}}{||\mathcal{R}_{\mathcal{R}^{*}}||},$
(12)

where
$\mathcal{R}_{\mathcal{R}^{*}}\in\\{\mathcal{R}_{\mathcal{H}},\mathcal{R}_{\mathcal{T}},\mathcal{B}\\}$
and $\Delta\hat{pi}$ is calculated from $\mathcal{L}_{corner}$ or
$\mathcal{L}_{boundary}$.

###  3.4 End-to-End Training

Our recognition part can be implemented by any sequence-based recognition
network, such as CRNN (?) or (?).

The loss of the whole framework contains the following parts: the order-aware
multi-class semantic segmentation, the corner regressions for pixels in _head_
and _tail_ , the boundary offset regression for pixels in the _center_ region
and the word recognition, that is,

$\mathcal{L}=\mathcal{L}_{cls}+\lambda_{b}\mathcal{L}_{corner}+\lambda_{c}\mathcal{L}_{boundary}+\lambda_{r}\mathcal{L}_{recog},$
(13)

where $\lambda_{b}$, $\lambda_{c}$ and $\lambda_{r}$ are auto-tunable
parameters, and $\mathcal{L}_{recog}$ is the loss from recognition.

Since learning fiducial points highly depends on the segmentation map
learning, we use a soft loss weight strategy to automatically tune
$\lambda_{b}$, $\lambda_{c}$ and $\lambda_{r}$. In other words, in the first
few epochs, fiducial points are mainly adjusted by regression tasks; while at
the last few epochs, points are mainly restricted to recognition. Formally,

$\lambda_{b}=\lambda_{c}=\lambda^{*}-\max{(0.02\times E,0.5)},$ (14)
$\lambda_{r}=\min{(\max{(-0.1+0.02\times E,0)},\lambda_{r}^{*})},$ (15)

where $E$ is the number of training epochs, and $\lambda^{*}$ and
$\lambda_{r}^{*}$ separately control the maximum loss weight of regression and
recognition. In our experiments, we set $\lambda^{*}=0.6$ and
$\lambda_{r}^{*}=0.8$.

##  4 Experiments

Dataset | Method | Detection | End-to-End | Word Spotting
---|---|---|---|---
P | R | F | FPS | S | W | G | S | W | G
IC13 | Textboxes (?) | 88.0 | 83.0 | 85.0 | 1.37 | 91.6 | 89.7 | 83.9 | 93.9 | 92.0 | 85.9
Li et al. (?) | 91.4 | 80.5 | 85.6 | - | 91.1 | 89.8 | 84.6 | 94.2 | 92.4 | 88.2
TextSpotter (?) | - | - | - | - | 89.0 | 86.0 | 77.0 | 92.0 | 89.0 | 81.0
He et al. (?) | 91.0 | 88.0 | 90.0 | - | 91.0 | 89.0 | 86.0 | 93.0 | 92.0 | 87.0
FOTS (?) | - | - | 88.2 | 23.9 | 88.8 | 87.1 | 80.8 | 92.7 | 90.7 | 83.5
TextNet* (?) | 93.3 | 89.4 | 91.3 | - | 89.8 | 88.9 | 83.0 | 94.6 | 94.5 | 87.0
Mask TextSpotter* (?) | 95.0 | 88.6 | 91.7 | 4.6 | 92.2 | 91.1 | 86.5 | 92.5 | 92.0 | 88.2
Ours (2-stage) | 92.7 | 88.7 | 90.7 | 10.3 | 90.8 | 90.0 | 84.4 | 93.7 | 93.1 | 86.2
Ours (End-to-end) | 94.7 | 88.9 | 91.7 | 10.3 | 91.4 | 90.7 | 85.8 | 94.9 | 94.0 | 88.5
IC15 | EAST (?) | 83.6 | 73.5 | 78.2 | 13.2 | - | - | - | - | - | -
TextSnake* (?) | 84.9 | 80.4 | 82.6 | 1.1 | - | - | - | - | - | -
SPCNet* (?) | 88.7 | 85.8 | 87.2 | - | - | - | - | - | - | -
PSENet-1s* (?) | 86.9 | 84.5 | 85.7 | 1.6 | - | - | - | - | - | -
TextSpotter (?) | - | - | - | - | 54.0 | 51.0 | 47.0 | 58.0 | 53.0 | 51.0
He et al. (?) | 87.0 | 86.0 | 87.0 | - | 82.0 | 77.0 | 63.0 | 85.0 | 80.0 | 65.0
FOTS (?) | 91.0 | 85.2 | 88.0 | 7.8 | 81.1 | 75.9 | 60.8 | 84.7 | 79.3 | 63.3
TextNet* (?) | 89.4 | 85.4 | 87.4 | - | 78.7 | 74.9 | 60.5 | 82.4 | 78.4 | 62.4
Mask TextSpotter* (?) | 91.6 | 81.0 | 86.0 | 4.8 | 79.3 | 73.0 | 62.4 | 79.3 | 74.5 | 64.2
| Ours (2-stage) | 91.6 | 81.8 | 86.4 | 8.8 | 78.2 | 74.5 | 63.0 | 80.6 | 76.6 | 65.5
| Ours (End-to-end) | 92.3 | 82.5 | 87.1 | 8.8 | 80.5 | 76.6 | 65.1 | 84.1 | 79.4 | 67.9

Table 1: Results on IC13 and IC15. ‘P’, ‘R’ and ‘F’ separately mean the
‘Precision’, ‘Recall’ and ‘F-Measure’. ‘S’, ‘W’ and ‘G’ mean recognition with
strong, weak and generic lexicon, respectively. Superscript ‘*’ means that the
method considered the detection of irregular text.

###  4.1 Datasets

The datasets used in this work are listed as follows:

_SynthText 800k_ (?) contains 800k synthetic images that are generated by
rendering synthetic text with natural images, and it is used as the pre-
training dataset.

_ICDAR2013_ (?) (_abbr._ IC13) is collected as the focused scene text, which
is mainly horizontal text containing 229 training images and 233 testing
images.

_ICDAR2015_ (?) (_abbr._ IC15) is collected as incidental scene text
consisting of many perspective text. It contains 1000 training and 500 testing
images.

_Total-Text_ (?) consists of multi-oriented and curve text and is therefore
one of the important benchmarks in evaluating shape-robust text spotting
tasks. It contains 1255 training and 300 testing images, and each text is
annotated by a word-level polygon with transcription.

_SCUT-CTW1500_ (?) (_abbr._ CTW1500) is a curved text benchmark consists of
1000 training and 500 testing images. In contrast to Total-Text, all text
instances are annotated with 14-point polygons in the line-level.

###  4.2 Implementation Details

The detector uses ResNet-50 as the backbone and further be modified following
the suggestions from (?) for obtaining dense features. We remove the fifth
stage, modify $conv4\\_1$ layer with stride=1 instead of 2, and apply atrous
convolution for all subsequent layers to maintain enough receptive field.
Training loss is calculated from the outputs of three stages: the fourth stage
($8$$\times$), the third stage ($8$$\times$), and the second stage
($4$$\times$) feature maps of FPN, and testing is only conducted on
$4$$\times$ feature map. We directly adopt the attention-based network
described in (?) as the recognition model. All experiments are implemented in
Caffe with 8 32GB-Tesla-V100 GPUs. The code will be published soon.

Data augmentation. We conduct data augmentation by simultaneously 1) randomly
scaling the longer side of input images with length in range of [$720$,
$1600$], 2) randomly rotating the images with the degree in range of
[$-15^{\circ},15^{\circ}]$, and 3) applying random brightness, jitters, and
contrast on input images.

Training details. The networks are trained by SGD with batch-size=8,
momentum=0.9 and weight-decay=$5\times 10^{-4}$. For both detection and
recognition part, we separately pre-train them on SynthText for 5 epochs with
initial learning rate $2\times 10^{-3}$. Then, we jointly fine-tune the whole
network using the soft loss weight strategy mention previously on each dataset
for other 80 epochs. The initial learning rate is $1\times 10^{-3}$. The
learning rate will be divided by 10 for every 20 epochs. Online hard example
mining (OHEM) (?) strategy is also applied for balancing the foreground and
background samples.

Testing details. We resize input images with the longer side 1440 for IC13,
2000 for IC15, 1350 for Total-text and 1250 for CTW1500. We set the number of
fiducial points as 4 for two standard text datasets and 14 for two irregular
text datasets. The detection results are given by connecting the predicted
fiducial points. Note that, all images are tested in the single-scale.

###  4.3 Results on Standard Text Benchmarks

_Evaluation on horizontal text_. We first evaluate our method on IC13 mainly
consisting of horizontal texts. Table 1 shows the results, and represents that
our method achieve competitive performance compared to previous methods on the
‘Detection’, ‘End-to-End’ and ‘Word Spotting’ evaluation items. Besides, our
method is also very efficient and achieves ‘10.3’ of Frame Per Second (_abbr._
FPS).

_Evaluation on perspective text_. We evaluate our method on IC15 containing
many perspective texts, and the results are shown in Table 1. In the detection
stage, our method achieves comparable performance with the irregular text
spotting methods such as TextNet and Mask TextSpotter. In the ‘End-to-End’ and
‘Word Spotting’ tasks, our method significantly outperforms previous
irregular-text-based methods and achieves the remarkable state-of-the-art
performance on general lexicon cases, which demonstrates the effectiveness of
our method.

###  4.4 Results on Irregular Text Benchmarks

Method | Detection | End-to-End
---|---|---
P | R | F | None | Full
TextSnake (?) | 82.7 | 74.5 | 78.4 | - | -
FTSN (?) | 84.7 | 78.0 | 81.3 |  |
TextField (?) | 81.2 | 79.9 | 80.6 | - | -
SPCNet (?) | 83.0 | 82.8 | 82.9 | - | -
CSE (?) | 81.4 | 79.1 | 80.2 | - | -
PSENet-1s (?) | 84.0 | 78.0 | 80.9 | - | -
LOMO (?) | 75.7 | 88.6 | 81.6 | - | -
Mask TextSpotter (?) | 69.0 | 55.0 | 61.3 | 52.9 | 71.8
TextNet (?) | 68.2 | 59.5 | 63.5 | 54.0 | -
Ours (2-stage) | 88.1 | 78.9 | 83.3 | 63.3 | 73.9
Ours (End-to-end) | 88.8 | 81.8 | 85.2 | 69.7 | 78.3

Table 2: Result on Total-Text. “Full” indicates lexicons of all images are
combined. “None” means lexicon-free.

We test our method on two irregular text benchmarks: Total-Text and CTW1500,
as shown in Table 2 and 3. In the detection stage, our method outperforms all
previous methods and surpasses the best result 2.3% on Total-Text and 2.4% on
CTW1500 on F-measure evaluation.

Moreover, our method significantly outperforms previous methods on the
precision item, which attributes to the false-positive filtering strategy. In
the end-to-end case, our method significantly surpasses the best-reported
results (?) by 15.7% on ‘None’ and the best of results (?) by 6.5% on ‘Full’,
which mainly attributes to STM achieving the end-to-end training strategies.
Since CTW1500 releases the recognition annotation recently, there is no
reported result on the end-to-end evaluation. Here, we report the end-to-end
results lexicon-freely, and believe our method will significantly outperform
previous methods.

Method | Detection | End-to-End
---|---|---
P | R | F | None
TextSnake (?) | 69.7 | 85.3 | 75.6 | -
TextField (?) | 83.0 | 79.8 | 81.4 | -
CSE (?) | 81.1 | 76.0 | 78.4 | -
PSENet-1s (?) | 84.8 | 79.7 | 82.2 | -
LOMO (?) | 69.6 | 89.2 | 78.4 | -
Ours (2-stage) | 88.7 | 78.2 | 83.1 | 48.6
Ours (End-to-end) | 87.5 | 81.9 | 84.6 | 57.0

Table 3: Result on CTW1500. ‘None’ means lexicon-free.

In summary, the results on Total-Text and CTW1500 demonstrate the
effectiveness of our method for arbitrary-shaped text spotting. Moreover,
compared with 2-staged results, the end-to-end trainable strategy markedly
boosts text spotting performance, especially for the recognition part.

###  4.5 Ablation Results of Fiducial Points

The number of fiducial points directly influences the detection and end-to-end
results when texts are displayed in the curve or even waved shapes. Table 4
shows the result that how the number of fiducial points affects the detection
and end-to-end evaluations on different benchmarks. It is clear that 4 points
annotation is enough for regular benchmark such as IC15, and there is almost
no influence on the result when the number of fiducial points increases. On
the other hand, for two irregular benchmarks, the detection F-score as well as
end-to-end F-score raises along with the increasing number of fiducial points,
and the performance becomes stable when $2\times N\geq 10$.

Dataset | Number of fiducial points
---|---
4 | 6 | 8 | 10 | 12 | 14 | 16 | 18
IC15 | 87.1 | 87.0 | 87.0 | 86.9 | 87.0 | 86.9 | 86.8 | 86.8
Total-Text | 71.5 | 82.8 | 84.5 | 85.0 | 85.2 | 85.2 | 85.2 | 85.3
CTW1500 | 68.7 | 81.9 | 84.1 | 84.3 | 84.4 | 84.6 | 84.4 | 84.5
Total-Text | 55.9 | 68.5 | 69.8 | 69.6 | 69.8 | 69.7 | 69.5 | 69.9
CTW1500 | 40.2 | 52.2 | 56.2 | 57.0 | 57.1 | 57.0 | 56.5 | 56.4

Table 4: Detection (top part) and end-to-end (bottom part) evaluation
(F-measure) under varied number of fiducial points for different benchmarks.

Figure 5 shows an example of end-to-end evaluation under different number of
fiducial points. We see that the generated text masks by few fiducial points
are hard to cover the entire curve texts. As the growing number of fiducial
points, STM has more power to catch and rectify irregular text instances,
which yields higher recognition accuracy.

Figure 5: Results of Text Perceptron with different number of fiducial points
(4,6,10,12).

In contrast to previous works, our method can generate any fixed number of
fiducial points on text boundaries. The fiducial points generation method can
also be used to annotate arbitrary-shaped text.

###  4.6 Visualization Results

Figure 6: Visualization results on origin images.  Figure 7: Visualization
result on Total-Text and CTW1500. The first row displays the segmented results
and the second row shows the end-to-end results. Fiducial points are also
visualized as colored points on text boundaries.

Figure 6 and Figure 7 demonstrate some visualization results in Total-Text and
CTW1500 datasets. Text Perceptron shows its powerful ability in catching the
reading order of irregular scene text (including curved, long perspective,
vertical, etc.), and with the help of fiducial points which can further
recognize text in a much simpler way. From the segmentation results, we find
that many of text-like false positives have been filtered out due to the
missing of _head_ or _tail_ boundary. This means the features of _head_ or
_tail_ boundaries contain the different semantic information with that of the
_center_ region. Figure 6 also shows the visualization of some rectified
irregular text instances, in which vertical texts can be well transformed into
the “lying-down” shapes.

#### Failure Samples

Figure 8: Visualization of some failure samples.

We illustrate some failure samples that are difficult for Text Perceptron, as
shown in Figure 8.

_Overlapped text_. It is a common tough task for segmentation-based detection
methods. Pixels belong to the _center_ text region for one text instance may
also become the boundary region for another one. Even though our orderly
overlaying strategy allows pixels to have multiple classes and makes boundary
pixels have higher priority than _center_ text pixels, which encourages inner
instance to be separated from the outer instance. But experiments found that
many times, the boundaries of inner instance cannot be fully recalled to
embrace such instance, and connecting between _center_ text pixels will result
in the failure of detecting such inner an instance.

_Recognition of vertical instance_. On the one hand, vertical texts appear in
little frequency in the common datasets. One the other hand, although Text
Perceptron can read vertical instances from left to right, it is still a
challenge for recognition algorithm to distinguish whether the instance is a
horizontal text or a ’lying-down’ vertical one. Therefore, there are some
correctly detected instances cannot be recognized right. It is also a common
difficult problem for all existing recognition algorithms.

##  5 Conclusion

In this paper, we propose an end-to-end trainable text spotter named Text
Perceptron aiming at spotting text with arbitrary-shapes. To achieve global
optimization, a Shape Transform Module is proposed to unite the text detection
and recognition into a whole framework. A segmentation-based detector is
carefully designed to distinguish text instances and capture the latent
information of text reading orders. Extensive experiments show that our method
achieves competitive result in standard text benchmarks and the state-of-the-
art in both detection and end-to-end evaluations on popular irregular text
benchmarks.

## References

  * [Bookstein 1989] Bookstein, F. L.  1989\.  Principal Warps: Thin-Plate Splines and the Decomposition of Deformations.  IEEE TPAMI 11(6):567–585.
  * [Bušta, Neumann, and Matas 2017] Bušta, M.; Neumann, L.; and Matas, J.  2017\.  Deep Textspotter: An End-to-End Trainable Scene Text Localization and Recognition Framework.  In ICCV, 2223–2231.
  * [Cheng et al. 2017] Cheng, Z.; Bai, F.; Xu, Y.; Zheng, G.; Pu, S.; and Zhou, S.  2017\.  Focusing Attention: Towards Accurate Text Recognition in Natural Images.  In ICCV, 5076–5084.
  * [Ch’ng and Chan 2017] Ch’ng, C. K., and Chan, C. S.  2017\.  Total-text: A Comprehensive Dataset for Scene Text Detection and Recognition.  In ICDAR, volume 1, 935–942.
  * [Dai et al. 2018] Dai, Y.; Huang, Z.; Gao, Y.; Xu, Y.; Chen, K.; Guo, J.; and Qiu, W.  2018\.  Fused Text Segmentation Networks for Multi-oriented Scene Text Detection.  In ICPR, 3604–3609.
  * [Gupta, Vedaldi, and Zisserman 2016] Gupta, A.; Vedaldi, A.; and Zisserman, A.  2016\.  Synthetic Data for Text Localisation in Natural Images.  In CVPR, 2315–2324.
  * [He et al. 2016] He, K.; Zhang, X.; Ren, S.; and Sun, J.  2016\.  Deep Residual Learning for Image Recognition.  In CVPR, 770–778.
  * [He et al. 2017] He, K.; Gkioxari, G.; Dollar, P.; and Girshick, R.  2017\.  Mask R-CNN.  In ICCV, 2980–2988.
  * [He et al. 2018] He, T.; Tian, Z.; Huang, W.; Shen, C.; Qiao, Y.; and Sun, C.  2018\.  An End-to-End TextSpotter with Explicit Alignment and Attention.  In CVPR, 5020–5029.
  * [Huang et al. 2017] Huang, J.; Rathod, V.; Sun, C.; Zhu, M.; Korattikara, A.; Fathi, A.; Fischer, I.; Wojna, Z.; Song, Y.; Guadarrama, S.; et al.  2017\.  Speed Accuracy Trade-offs for Modern Convolutional Object Detectors.  In CVPR, 7310–7311.
  * [Jaderberg et al. 2015] Jaderberg, M.; Simonyan, K.; Zisserman, A.; et al.  2015\.  Spatial Transformer Networks.  In NeurIPS, 2017–2025.
  * [Jaderberg, Vedaldi, and Zisserman 2014] Jaderberg, M.; Vedaldi, A.; and Zisserman, A.  2014\.  Deep Features for Text Spotting.  In ECCV, 512–528.
  * [Karatzas et al. 2013] Karatzas, D.; Shafait, F.; Uchida, S.; Iwamura, M.; i Bigorda, L. G.; Mestre, S. R.; Mas, J.; Mota, D. F.; Almazan, J. A.; and De Las Heras, L. P.  2013\.  ICDAR 2013 Robust Reading Competition.  In ICDAR, 1484–1493.
  * [Karatzas et al. 2015] Karatzas, D.; Gomez-Bigorda, L.; Nicolaou, A.; Ghosh, S.; Bagdanov, A.; Iwamura, M.; Matas, J.; Neumann, L.; Chandrasekhar, V. R.; Lu, S.; et al.  2015\.  ICDAR 2015 Competition on Robust Reading.  In ICDAR, 1156–1160.
  * [Li, Wang, and Shen 2017] Li, H.; Wang, P.; and Shen, C.  2017\.  Towards End-to-end Text Spotting with Convolutional Recurrent Neural Networks.  In ICCV, 5248–5256.
  * [Liao et al. 2017] Liao, M.; Shi, B.; Bai, X.; Wang, X.; and Liu, W.  2017\.  TextBoxes: A Fast Text Detector with a Single Deep Neural Network.  In AAAI, 4161–4167.
  * [Liao, Shi, and Bai 2018] Liao, M.; Shi, B.; and Bai, X.  2018\.  TextBoxes++: A Single-Shot Oriented Scene Text Detector.  IEEE TIP 27(8):3676–3690.
  * [Lin et al. 2017] Lin, T.-Y.; Dollár, P.; Girshick, R.; He, K.; Hariharan, B.; and Belongie, S.  2017\.  Feature Pyramid Networks for Object Detection.  In CVPR, 2117–2125.
  * [Liu et al. 2016] Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; and Berg, A. C.  2016\.  SSD: Single Shot Multibox Detector.  In ECCV, 21–37.  Springer.
  * [Liu et al. 2018] Liu, X.; Liang, D.; Yan, S.; Chen, D.; Qiao, Y.; and Yan, J.  2018\.  FOTS: Fast Oriented Text Spotting with a Unified Network.  In CVPR, 5676–5685.
  * [Liu et al. 2019a] Liu, Y.; Jin, L.; Zhang, S.; Luo, C.; and Zhang, S.  2019a.  Curved Scene Text Detection via Transverse and Longitudinal Sequence Connection.  PR 90:337–345.
  * [Liu et al. 2019b] Liu, Z.; Lin, G.; Yang, S.; Liu, F.; Lin, W.; and Goh, W. L.  2019b.  Towards Robust Curve Text Detection With Conditional Spatial Expansion.  In CVPR.
  * [Long et al. 2018] Long, S.; Ruan, J.; Zhang, W.; He, X.; Wu, W.; and Yao, C.  2018\.  Textsnake: A Flexible Representation for Detecting Text of Arbitrary Shapes.  In ECCV, 19–35.
  * [Luo, Jin, and Sun 2019] Luo, C.; Jin, L.; and Sun, Z.  2019\.  MORAN: A Multi-Object Rectified Attention Network for Scene Text Recognition.  PR.
  * [Lyu et al. 2018] Lyu, P.; Liao, M.; Yao, C.; Wu, W.; and Bai, X.  2018\.  Mask Textspotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes.  In ECCV, 71–88.
  * [Ma et al. 2018] Ma, J.; Shao, W.; Ye, H.; Wang, L.; Wang, H.; Zheng, Y.; and Xue, X.  2018\.  Arbitrary-Oriented Scene Text Detection via Rotation Proposals.  IEEE TMM 20(11):3111–3122.
  * [Milletari, Navab, and Ahmadi 2016] Milletari, F.; Navab, N.; and Ahmadi, S.-A.  2016\.  V-net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation.  In 3DV, 565–571.
  * [Ren et al. 2015] Ren, S.; He, K.; Girshick, R.; and Sun, J.  2015\.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  In NeurIPS, 91–99.
  * [Shi, Bai, and Yao 2017] Shi, B.; Bai, X.; and Yao, C.  2017\.  An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition.  IEEE TPAMI 39(11):2298–2304.
  * [Shi et al. 2016] Shi, B.; Wang, X.; Lyu, P.; Yao, C.; and Bai, X.  2016\.  Robust Scene Text Recognition with Automatic Rectification.  In CVPR, 4168–4176.
  * [Shrivastava, Gupta, and Girshick 2016] Shrivastava, A.; Gupta, A.; and Girshick, R.  2016\.  Training Region-based Object Detectors with Online Hard Example Mining.  In CVPR, 761–769.
  * [Sun et al. 2018] Sun, Y.; Zhang, C.; Huang, Z.; Liu, J.; Han, J.; and Ding, E.  2018\.  TextNet: Irregular Text Reading from Images with an End-to-End Trainable Network.  In ACCV.
  * [Wang et al. 2012] Wang, T.; Wu, D. J.; Coates, A.; and Ng, A. Y.  2012\.  End-to-End Text Recognition with Convolutional Neural Networks.  In ICPR, 3304–3308.
  * [Wang et al. 2019] Wang, W.; Xie, E.; Li, X.; Hou, W.; Lu, T.; Yu, G.; and Shao, S.  2019\.  Shape Robust Text Detection With Progressive Scale Expansion Network.  In CVPR.
  * [Wu and Natarajan 2017] Wu, Y., and Natarajan, P.  2017\.  Self-organized Text Detection with Minimal Post-processing via Border Learning.  In ICCV, 5010–5019.
  * [Xie et al. 2019] Xie, E.; Zang, Y.; Shao, S.; Yu, G.; Yao, C.; and Li, G.  2019\.  Scene Text Detection with Supervised Pyramid Context Network.  In AAAI.
  * [Xu et al. 2019] Xu, Y.; Wang, Y.; Zhou, W.; Wang, Y.; Yang, Z.; and Bai, X.  2019\.  Textfield: Learning a deep direction field for irregular scene text detection.  IEEE TIP.
  * [Xue, Lu, and Zhan 2018] Xue, C.; Lu, S.; and Zhan, F.  2018\.  Accurate Scene Text Detection through Border Semantics Awareness and Bootstrapping.  In ECCV, 370–387.
  * [Zhan and Lu 2019] Zhan, F., and Lu, S.  2019\.  Esir: End-to-end scene text recognition via iterative image rectification.  In CVPR, 2059–2068.
  * [Zhang et al. 2018] Zhang, S.; Liu, Y.; Jin, L.; and Luo, C.  2018\.  Feature Enhancement Network: A Refined Scene Text Detector.  In AAAI.
  * [Zhang et al. 2019] Zhang, C.; Liang, B.; Huang, Z.; En, M.; Han, J.; Ding, E.; and Ding, X.  2019\.  Look More Than Once: An Accurate Detector for Text of Arbitrary Shapes.  In CVPR.
  * [Zhou et al. 2017] Zhou, X.; Yao, C.; Wen, H.; Wang, Y.; Zhou, S.; He, W.; and Liang, J.  2017\.  EAST: An Efficient and Accurate Scene Text Detector.  In CVPR, 2642–2651.


# Real-time observation of Cooper pair splitting showing strong non-local
correlations

Antti Ranni<EMAIL_ADDRESS>NanoLund and Solid State Physics, Lund
University, Box 118, 22100 Lund, Sweden  Fredrik Brange  Elsa T. Mannila
Christian Flindt  Department of Applied Physics, Aalto University, 00076
Aalto, Finland  Ville F. Maisi<EMAIL_ADDRESS>NanoLund and Solid
State Physics, Lund University, Box 118, 22100 Lund, Sweden

Controlled generation and detection of quantum entanglement between spatially
separated particles constitute an essential prerequisite both for testing the
foundations of quantum mechanics and for realizing future quantum
technologies. Splitting of Cooper pairs from a superconductor provides
entangled electrons at separate locations. However, experimentally accessing
the individual split Cooper pairs constitutes a major unresolved issue as they
mix together with electrons from competing processes. Here, we overcome this
challenge with the first real-time observation of the splitting of individual
Cooper pairs, enabling direct access to the time-resolved statistics of Cooper
pair splitting. We determine the correlation statistics arising from two-
electron processes and find a pronounced peak that is two orders of magnitude
larger than the background. Our experiment thereby allows to unambiguously
pinpoint and select split Cooper pairs with 99 % fidelity. These results open
up an avenue for performing experiments that tap into the spin-entanglement of
split Cooper pairs.

Cooper pair splitters are promising solid-state devices for generating non-
local entanglement of electronic spins.Recher  _et al._ (2001); Lesovik  _et
al._ (2001) The basic operating principle is based on the tunneling of spin-
entangled electrons from a superconductor into spatially separated normal-
state structures, whereby entanglement between different physical locations is
obtained. The controlled generation of entangled particles is not only of
fundamental interest to test the foundations of quantum mechanics.Brunner  _et
al._ (2014) It is also a critical prerequisite for future quantum
technologies,Zagoskin (2011); Shevchenko (2019) such as quantum information
processors and other quantum devices, which will be ‘fuelled’ by entanglement.

Until now, the splitting of Cooper pairs has been indirectly inferred from
measurements of the currents in the outputs of a Cooper pair splitterBeckmann
_et al._ (2004); Russo  _et al._ (2005); Yeyati  _et al._ (2007); Hofstetter
_et al._ (2009); Cadden-Zimansky  _et al._ (2009); Herrmann  _et al._ (2010);
Wei and Chandrasekhar (2010); Hofstetter  _et al._ (2011); Herrmann  _et al._
; Das  _et al._ (2012); Fülöp  _et al._ (2014); Schindele  _et al._ (2014);
Tan  _et al._ (2015); Fülöp  _et al._ (2015); Borzenets  _et al._ (2016);
Schindele  _et al._ (2012); Deacon  _et al._ (2015); Ueda  _et al._ (2019);
Tan  _et al._ (2021); Bruhat  _et al._ (2018); Pandey  _et al._ (2021) or
their low-frequency cross-correlations.Wei and Chandrasekhar (2010); Das  _et
al._ (2012) These approaches rely on measuring the average currents, or small
fluctuations around them, due to a large number of splitting events. The
currents consist of contributions from Cooper pair splitting as well as other
competing transport processes, which not only makes these experiments highly
challenging, but also hinders direct access to the correlated electron pairs.
In particular, a considerable fraction of the electrical signal is due to
unwanted processes, and those electrons that actually are entangled have
already passed through the device and are lost, once the currents have been
measured.

Here we take a fundamentally different route and use charge detectors to
observe the splitting of individual Cooper pairs in real time as it happens.
Unlike earlier experiments, we use isolated islands that are not connected to
external drain electrodes. Hence, once a Cooper pair is split, the two
correlated electrons remain stored on the islands and can be detected in real-
time.

## Results

Device architecture. Figure 1a shows a scanning electron microscope image of
one of our Cooper pair splitters made of a superconducting aluminium electrode
coupled to two normal-state copper islands. The grounded superconducting
electrode and the islands are connected via insulating aluminium oxide layers
acting as tunnel junctions. The superconductor functions as a source of Cooper
pairs, which are split into two separate electrons that tunnel into the
islands, one in each. We detect the number of electrons on each island using
single-electron transistors whose conductance depends sensitively on the
charge state of the islands.Schoelkopf  _et al._ (1998); Gustavsson  _et al._
(2009); Pekola  _et al._ (2013) Each detector is biased by a voltage
$V_{\mathrm{D\alpha}}$ ($\alpha=L$ or $R$) and the currents
$I_{\mathrm{D\alpha}}$ through them are measured in the grounded contacts. The
detectors are tuned to charge sensitive operation points by the gate voltages
$V_{\mathrm{DG\alpha}}$. We can thus monitor the tunneling of electrons in and
out of the islands, including simultaneous processesMaisi  _et al._ (2011,
2014), such as Cooper pair splitting, where one electron tunnels into each of
the two islands at the same time. The electronic population of each island is
feedback-controlled by the gate voltages $V_{\mathrm{G\alpha}}$, which are
adjusted to maintain the two lowest-lying charge states at equal
energies.Koski  _et al._ (2014); Singh  _et al._ (2019) Hence, there is no
energy cost for tunneling processes that change the electron number on each
island by one.

Figure 1: Cooper pair splitter and correlation measurements. a, Scanning
electron micrograph of the grounded superconducting reservoir (colored blue)
coupled via tunnel junctions to two normal-state metallic islands (in orange).
The voltages $V_{\mathrm{GL}}$ and $V_{\mathrm{GR}}$ are connected to gate
electrodes and tuned to control the electronic populations on the islands.
Each island is capacitively coupled via a metal strip (in green) to a single-
electron transistor, which serves as a real-time charge detector to read out
the number of electrons on each island from the measured signals
$I_{\mathrm{DL}}$ and $I_{\mathrm{DR}}$. The detectors are biased by the
voltages $V_{\mathrm{D}\alpha}$ ($\alpha=L$ or $R$) and gated by the voltages
$V_{\mathrm{DG}\alpha}$. The inset in the upper right corner shows the tunnel
junctions that connect the superconductor to the two metallic islands. b,
Measurements of the auto-correlation function $g^{(2)}$ for single-electron
tunneling into the right island with a time delay $\tau$ between tunneling
events. The correlation function is suppressed at short times due to the
strong Coulomb interactions on the island, which lead to anti-bunching. c,
Cross-correlation function $g^{(2)}_{x}$ for transitions between the
superconductor and the left island at the time $t=0$, followed by transitions
between the superconductor and the right island at the time $t=\tau$. The
cross-correlations are strongly enhanced for short times due to crossed two-
electron processes such as Cooper pair splitting and elastic cotunneling
between the islands via the superconductor. The error bars in b and c are
given by the standard deviations. Details of the device fabrication and the
measurements are provided in the Methods section.

Correlation measurements. Figures 1b and c show correlation measurements using
our charge detectors. The $g^{(2)}$-function describes correlations between
tunneling processes with a time delay $\tau$ between them and are for example
used extensively in quantum optics to characterize light sources.Loudon (2000)
A value of $g^{(2)}(\tau)=1$ at all times implies that the particles are
uncorrelated. On the other hand, a $g^{(2)}$-function that peaks at short
times indicates that the particles tend to bunch as for instance for thermal
light, which has $g^{(2)}(0)=2$ at $\tau=0$. By contrast, a coherent single-
photon source is characterized by a $g^{(2)}$-function with a dip at short
times, indicating that the particles are anti-bunched, ideally with
$g^{(2)}(0)=0$.

Here we measure the auto-correlations for tunneling events from the
superconductor to the right island as well as the cross-correlations for
tunneling between the superconductor and each of the two islands. The auto-
correlations in Fig. 1b are fully suppressed at short times. The suppression
arises from the strong Coulomb interactions, which prevent more than one
electron at a time to tunnel into the right island, leading to anti-bunching
of the tunneling events. A theoretical analysis captures the experimental
findings in Fig. 1b by the expression

$g^{(2)}(\tau)=1-e^{-\gamma|\tau|},$ (1)

where $\gamma=4.5$ s-1 is the inverse correlation time, which may be
determined from the tunneling rates (see Supplementary Note 2 for details).
Equation (1) is similar to the auto-correlation function of a simple two-level
system, however, in our case, it follows from an elaborate model of the Cooper
pair splitter, which involves several charge states of each island, and the
inverse correlation time is a complicated function of all tunneling rates.

Figure 2: Real-time observation of Cooper pair splitting. a, Typical time
traces of the currents $I_{\mathrm{DL}}$ and $I_{\mathrm{DR}}$ in the left and
the right single-electron transistors correspondingly, which switch between
two distinct levels corresponding to having $n=0$ or 1 (excess) electrons on
each island. b, The points marked with \raisebox{-.9pt} {1}⃝ and
\raisebox{-.9pt} {3}⃝ in the left panel correspond to processes, where a
single electron tunnels between the superconductor and one of the islands. The
point marked with \raisebox{-.9pt} {2}⃝ is a Cooper pair splitting event,
where two electrons simultaneously tunnel from the superconductor into the two
islands, one in each, while the point marked with \raisebox{-.9pt} {4}⃝ is an
elastic cotunneling event, where an electron is transferred between the two
islands via the superconductor.

Turning next to the cross-correlations in Fig. 1c, a completely different
picture emerges. We now consider the conditional probability of observing a
tunneling event between the superconductor and the right island at the time
delay $\tau$ after a tunneling event between the superconductor and the left
island has occurred. In this case, we observe a large peak at short times
which is two orders of magnitude larger than the uncorrelated background of
$g^{(2)}_{x}(\tau)=1$. These correlations are a direct manifestation of nearly
instantaneous two-electron processes involving both islands. The two-electron
processes lead to correlated single-electron events in the two islands
occurring on a microsecond timescale, which is much faster than the
correlation time of the single-electron processes for each island as seen in
Fig. 1b. Theoretically, we can describe the cross-correlations as

$g^{(2)}_{x}(\tau)=1+\alpha_{2}\frac{e^{-\frac{1}{2}(\tau/\sigma_{D})^{2}}}{\sqrt{2\pi}\sigma_{D}},$
(2)

where $\sigma_{D}=460$ $\mu$s is the broadening due to timing jitter of the
detectors, and $\alpha_{2}=210$ ms is the time-integrated contribution from
two-electron processes. Figure 1c shows that this expression agrees well with
the experiment, explaining the strong non-local correlations. Figure 1 also
illustrates how the cross-correlations, unlike the auto-correlations, can be
measured on a time-scale, which is shorter than the rise time of each detector
of about 4 ms: The cross-correlations concern tunneling events in different
islands and are not limited by the dead time of the detectors (see
Supplementary Note 1).

Cooper pair splitting in real-time. We now address the identification of the
individual tunneling events. Typical time-traces of the electrical currents in
the two single-electron transistors are shown in Fig. 2a, illustrating how we
can detect the tunneling events in each island. At the time marked by
\raisebox{-.9pt} {1}⃝, the current in the left detector (red curve) switches
suddenly from 60 pA to 80 pA as an electron tunnels out of the left island as
depicted in panel b. At the time marked by \raisebox{-.9pt} {2}⃝, the current
in the left detector switches again as an electron tunnels into the left
island. However, this time, the current in the right detector (blue curve)
also changes as an electron tunnels into the right island. As we discuss
below, statistical arguments allow us to conclude with near unity probability
that these simultaneous tunneling events occur due to the splitting of a
Cooper pair as indicated in panel b, rather than being two uncorrelated
single-electron processes. Continuing further in time, the point marked by
\raisebox{-.9pt} {3}⃝ again corresponds to the tunneling of a single electron
as in \raisebox{-.9pt} {1}⃝, except that now an electron tunnels out of the
right island. Finally, at the time marked by \raisebox{-.9pt} {4}⃝, the
detector currents switch in opposite directions due to elastic cotunneling in
which an electron tunnels from the left to the right island via the
superconductor.

Figure 3: Waiting time distributions $\mathcal{W}$. a, Waiting times $\tau$
between single-electron tunneling out of the right island. The theory curve
given by Eq. (3) agrees well with the measurements. b, Waiting times $\tau$
between single-electron tunneling out of the left island followed by single-
electron tunneling out of the right island. Note that the peak at the short
times is more than two orders of magnitude higher than the long time data. The
experimental data is well captured by Eq. (4). The dashed line serves as a
guide to the eye. The inset illustrates the waiting times between tunneling
events of the same or different types. The blue lines correspond to tunneling
out of the right island, while the red ones correspond to the left island. The
error bars mark $1\sigma$ confidence intervals.

We now return to the alleged Cooper pair splitting at the time marked by
\raisebox{-.9pt} {2}⃝. Strictly speaking, the two simultaneous tunneling
events could in principle be completely uncorrelated. However, to rule out
that scenario, we again consider our cross-correlation measurements in Fig.
1c. Uncorrelated single-electron processes give rise to a flat background with
$g_{x}^{(2)}(\tau)=1$, while correlated two-electron processes, such as Cooper
pair splitting, lead to the pronounced peak at short delay times, and they
essentially all take place within a time window of width $6\sigma_{D}\simeq 3\
\mathrm{ms}$. Within this time window, the fraction of uncorrelated single-
electron processes is $6\sigma_{D}/(\alpha_{2}+6\sigma_{D})$, while the
fraction of correlated two-electron processes is
$\alpha_{2}/(\alpha_{2}+6\sigma_{D})$. Hence, with fidelity
$\mathcal{F}=\alpha_{2}/(\alpha_{2}+6\sigma_{D})\simeq 99\%$ we can say that
the point marked by \raisebox{-.9pt} {2}⃝ represents the splitting of a Cooper
pair.

Waiting time distributions. Based on the identification of the involved
tunneling processes, we can determine all relevant tunneling rates in our
experiment. Furthermore, additional information about the tunneling processes
can be obtained from measurements of the electron waiting times.Walldorf  _et
al._ (2018, 2020); Wrześniewski and Weymann (2020) Figure 3a shows the
distribution of waiting times between consecutive electrons tunneling out of
the right island. Unlike the correlation functions, this is an exclusive
probability density, since no tunneling events of the same type are allowed
during the waiting time. Electron waiting times are typically hard to measure,
since they require nearly perfect detectors, unlike the $g^{(2)}$-functions.
The probability to observe a short waiting time is low, since only one
electron at the time can tunnel out of the island. After having reached its
maximum, the distribution falls off as it becomes exponentially unlikely to
wait a long time for the island to be refilled and the next electron can
tunnel out. These measurements agree well with the theoretical expectation

$\mathcal{W}(\tau)=\frac{1}{\langle\tau\rangle
u}\left(e^{-\gamma(1-u)\tau/2}-e^{-\gamma(1+u)\tau/2}\right),$ (3)

which in addition to the inverse correlation time from Eq. (1) also contains
the average waiting time between tunneling events, $\langle\tau\rangle=1.2$ s,
which enters through the parameter $u=\sqrt{1-4/(\gamma\langle\tau\rangle)}$.
At short times, $\gamma\tau\ll 1$, the waiting time distribution is linear in
time, $\mathcal{W}(\tau)\simeq\gamma\tau/\langle\tau\rangle$, just as the
$g^{(2)}$-function. By contrast, at longer times, where the $g^{(2)}$-function
flattens out, it describes the small probabilities to observe long waiting
times.

Turning to the cross-waiting time distribution in Fig. 3b, we consider here
the waiting time between an electron tunneling out of the left island followed
by the tunneling of an electron out of the right island. These experimental
results are well-captured by a detailed theoretical analysis, which yields the
expression

$\mathcal{W}_{x}(\tau)\\!=\\!\frac{\eta_{0}}{2}\left[\sqrt{\frac{2}{\pi}}\frac{e^{-\frac{\tau^{2}}{2\sigma_{D}^{2}}}}{\sigma_{D}}\\!+\\!\mathcal{W}(\tau)\right]+(1-\eta_{0})\mathcal{W}_{0}(\tau),$
(4)

where the first term in the brackets arises from the correlated two-electron
processes, happening with the weight $\eta_{0}/2$, where $\eta_{0}\simeq
0.36$, such as processes where an electron from each island tunnels into the
superconductor to form a Cooper pair. The bracket also contains the
distribution $\mathcal{W}(\tau)$ from Equation (3), corresponding to the
waiting time between a two-electron process and a subsequent transition on the
right island. The last term in Equation (4) originates from tunneling
processes from the right island into the superconductor, which are weakly
correlated with the left island and obey a Poissonian waiting time
distribution $\mathcal{W}_{0}(\tau)\propto e^{-\gamma(1-u)\tau/2}$ for long
times. Importantly, the exponential dependence at long times allows us to
characterize the storage time of the split electron pairs on the islands,
i.e., the typical duration from \raisebox{-.9pt} {2}⃝ to \raisebox{-.9pt} {3}⃝
in Fig. 2a. We note that the storage time ($\sim\gamma^{-1}$) is more than two
orders of magnitude longer than the detection time of the splitting process
($\sim\sigma_{D}$).

## Discussion

We have observed the splitting of individual Cooper pairs in real-time and
thereby enabled the identification and storage of single split Cooper pairs, a
procedure which is not possible with conventional current and noise
measurements. As such, our work paves the way for a wide range of future
experimental developments. Specifically, by embedding the single-electron
detectors in a radio-frequency circuit, it should be possible to read out the
charge states in a few microseconds,Schoelkopf  _et al._ (1998); Bäuerle  _et
al._ (2018) i.e., on a timescale that is much shorter than the spin lifetime
and coherence time of semiconductor quantum dots, which typically exceed
hundred microseconds.Hanson  _et al._ (2007); Zwanenburg  _et al._ (2013) In
addition to the fast detection, the other key ingredient to probe the
entanglement of the electrons is to detect the spin states of the split
electrons. The readout methods of spin qubits are based on either the energy
splitting of the spin states or spin-blockade, which would readily provide the
required techniques.Hanson  _et al._ (2007); Zwanenburg  _et al._ (2013)
Alternatively, ferromagnetic leads may be used to determine the spin state of
the electrons.Cottet  _et al._ (2004a, b) Thus, by combining the fast charge
detection with the spin read-out, one may witness and certify the quantum
entanglement of the split Cooper pairs.Kłobus  _et al._ (2014); Brange  _et
al._ (2017) Ultimately, several Cooper pair splitters may be combined to
realize the first rudimentary quantum information algorithms using split
Cooper pairs. Interestingly, our real-time detection scheme of crossed events
would also enable to discriminate different types of processes based on their
statistics. For example, the instantaneous Cooper pair splitting events
generate a sharp detection-limited peak as presented here, whereas the spin
selective tunneling discussed by in Refs. Cottet  _et al._ , 2004a, b would
yield tunneling time broadened peak that could be probed with the correlation
function measurements. In conclusion, our work opens up avenues for
experiments using entangled spins, and it thereby enables future quantum
technologies based on entangled electrons in on-chip solid-state devices.

## Methods

Device fabrication. The device was fabricated on a silicon substrate with a
300 nm thermal silicon oxide layer on top. The fabrication started with a
2/30/2 nm thick patterned Ti/Au/Ti layer (yellow in Fig. 1a) acting as a
ground plane to filter out stray high-frequency noise. Then, a 10 nm thick
chromium strip (green line running in 45∘ angles in the figure) was patterned
and deposited to obtain a capacitive coupling between the islands and the
detectors. Next, a 40 nm thick AlOx layer was grown with atomic layer
deposition to electrically insulate the ground plane and the coupler from the
rest of the structures. Finally, electron beam lithography and shadow mask
deposition were used to make the Cooper pair splitter and the charge
detectors. In the last step, four metal layers were deposited at different
angles with two oxidation steps in between. First, the superconducting lead
(in blue), 20 nm thick Al, was formed. Next, the aluminum surface was oxidised
to obtain tunnel barriers for the splitter. After that, the normal-state
islands of 25 nm thick copper (in orange) were deposited, completing the
splitter structure. As a third layer, a second 30 nm thick aluminum film was
deposited followed by oxidation to produce the tunnel barriers for the
detectors. The fabrication was then completed with the deposition of the 80 nm
thick copper leads of the detectors. The four layer processing allows for
independent tuning of the barrier transparencies for the splitter and the
detectors. Fabricating the splitter first ensured that the aluminum reservoir
is not directly connected to any of the extra non-operational shadow parts
(seen e.g. in the insets showing the detectors), thus preventing any proximity
effect. Also, the normal-state islands of the splitter are designed so that
the second aluminum layer does not overlap with them until several micrometers
away from the tunnel junctions. The detectors use the inverse proximity effect
to suppress the superconductivity of the small aluminum patches connected
directly to the first, 25 nm thick Cu layer to obtain nearly normal-metallic
charge detectors.

The tunnel junctions in the Cooper pair splitter were made with a distance of
$l=100\ \mathrm{nm}$. This distance was chosen to be shorter than the
coherence length $\xi=200\ \mathrm{nm}$ of the superconducting aluminum, hence
allowing for a finite rate of Cooper pair splitting, which otherwise would be
exponentially suppressed as $\exp(-l/\xi)$. By contrast, single-electron
tunneling between the superconductor and the islands is suppressed as
$\exp(-\Delta/k_{B}T)$, where
$\Delta=$200\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{e}\mathrm{V}$$ is the
superconducting gap and $T=50\,\mathrm{mK}\ll\Delta/k_{B}$ is the electronic
temperature. Two-electron processes were thus the dominant charge transfer
mechanism between the superconductor and the islands. The filtering with the
ground plane was paramount for obtaining the suppression as stray radiation
causes excess single-electron tunneling. We note that measurements with the
superconducting electrode in the normal-state were not performed, since the
superconducting gap suppresses the sequential tunneling rates, which would
otherwise exceed the detector bandwidth.

Experiments. All experimental results presented here were obtained at the base
temperature of a dry dilution refrigerator with an electronic temperature of
50 mK. The detectors were biased with
$200\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{V}$, and the currents were
measured with a digitizer after amplifying the signals with standard room
temperature current preamplifiers with 1 kHz bandwidth. Voltages were applied
to the gates to tune the charge detectors to a charge sensitive operation
point and to tune the populations of the normal-state islands so that the two
lowest-lying states in Fig. 2a had equal probabilities. Time traces of the two
detector currents were simultaneously recorded with a multichannel analog-to-
digital converter at a sampling rate of 20 kHz, while adjusting the gate
voltages in-between to compensate for slow drifts in the detector operation
point or in the occupations of the two lowest charge states. The adjustment
was done in a feedback loop by measuring a 60 s long time trace and extracting
population probabilities from it. The electronics and measurement
configuration were made identical on both detector sides to minimize timing
differences.

Data analysis. The 60 s long time traces were digitally filtered through a
low-pass filter with a cut-off frequency of 200 Hz which sets the detector
rise times to about 4 ms. All the results presented in our paper were obtained
by analysing these filtered data. Time traces with the populations at the two
lowest-lying charge states deviating considerably from each other or with the
detector currents drifting so that it became difficult to identify charge
states in the detector signal were excluded. The exact procedure and criteria
are given in Supplementary Note 1. We then identified all tunneling events and
the instances when they happened. These instances directly yield the
experimental correlation functions in Figs. 1b and c and the waiting time
distributions in Figs. 3a and b. In addition to the instances the theory
curves require the detector broadening as input parameter which was obtained
from the current noise of the detectors and their slew rate.

Theory. The system is described by a rate equation,
$\frac{d}{dt}|p(t)\rangle\\!\rangle=\mathbf{L}|p(t)\rangle\\!\rangle,$ where
$|p(t)\rangle\\!\rangle$ is a column vector containing the probabilities of
being in the different charge states of the islands, and the rate matrix
$\mathbf{L}$ contains the tunneling rates. The off-diagonal elements of
$\mathbf{L}$ are given by the transition rates between the different charge
states, while the diagonal elements contain the total escape rate (with a
minus sign) from each charge state. In addition, we introduce a jump operator,
$\mathbf{J}_{\alpha}$, describing transitions of type $\alpha$. The
$g^{(2)}$-function for processes of type $\alpha$ and $\beta$ can then be
obtained as

$\displaystyle g^{(2)}(\tau)$ $\displaystyle=$
$\displaystyle\frac{\langle\\!\langle\mathbf{J}_{\beta}e^{\mathbf{L}\tau}\mathbf{J}_{\alpha}\rangle\\!\rangle}{\langle\\!\langle\mathbf{J}_{\beta}\rangle\\!\rangle\langle\\!\langle\mathbf{J}_{\alpha}\rangle\\!\rangle}\theta(\tau)+\frac{\langle\\!\langle\mathbf{J}_{\alpha}e^{-\mathbf{L}\tau}\mathbf{J}_{\beta}\rangle\\!\rangle}{\langle\\!\langle\mathbf{J}_{\beta}\rangle\\!\rangle\langle\\!\langle\mathbf{J}_{\alpha}\rangle\\!\rangle}\theta(-\tau)$
(5)
$\displaystyle+\frac{\langle\\!\langle\mathbf{J}_{\alpha\beta}\rangle\\!\rangle}{\langle\\!\langle\mathbf{J}_{\beta}\rangle\\!\rangle\langle\\!\langle\mathbf{J}_{\alpha}\rangle\\!\rangle}\delta(\tau),$

where
$\langle\\!\langle\mathbf{A}\rangle\\!\rangle\equiv\langle\\!\langle\tilde{0}|\mathbf{A}|p_{s}\rangle\\!\rangle$
denotes the expectation value with respect to the steady-state fulfilling
$\mathbf{L}|p_{s}\rangle\\!\rangle=0$, the vector representation of the trace
operation is denoted as $\langle\\!\langle\tilde{0}|$, and $\theta(\tau)$ is
the Heaviside step function. The $g^{(2)}$-function yields the normalized
joint probability of detecting a transition of type $\alpha$ at some time and
a transition of type $\beta$ at a time $\tau$ later. The last term in Equation
(5) accounts for the instantaneous correlations between electrons belonging to
the same two-electron process, described by a jump operator
$\mathbf{J}_{\alpha\beta}$. These correlations are convolved with a Gaussian
distribution to take into account the timing jitter of the detectors. The
$g^{(2)}$-function approaches unity on timescales over which the correlations
vanish.

The waiting time distribution can be obtained as

$\mathcal{W}(\tau)=\frac{\langle\\!\langle\mathbf{J}_{\beta}e^{(\mathbf{L}-\mathbf{J}_{\beta})\tau}(\mathbf{J}_{\alpha}-\mathbf{J}_{\alpha\beta})\rangle\\!\rangle}{\langle\\!\langle\mathbf{J}_{\alpha}\rangle\\!\rangle}+\eta_{0}\delta(\tau),$
(6)

with
$\eta_{0}\equiv\frac{\langle\\!\langle\mathbf{J}_{\alpha\beta}\rangle\\!\rangle}{\langle\\!\langle\mathbf{J}_{\alpha}\rangle\\!\rangle}$,
which, just as for the $g^{(2)}$-function, includes the instantaneous
correlations of two-electron processes. The waiting time distribution yields
the probability density to observe a waiting time $\tau$ from a transition of
type $\alpha$ has occurred until the first subsequent transition of type
$\beta$ takes place. Similar to the $g^{(2)}$ function, we use a convolution
with a Gaussian distribution to describe the timing jitter of the detectors.
As a probability distribution, the waiting time distribution is normalized
such that $\int_{0}^{\infty}d\tau\mathcal{W}(\tau)=1$.

Data availability. The data that support the findings of this study are
available from the corresponding authors upon reasonable request.

Code availability. The code to analyse the experimental data and to compute
the theoretical results is available from the corresponding authors upon
reasonable request.

## References

  * Recher  _et al._ (2001) P. Recher, E. V. Sukhorukov, and D. Loss, “Andreev tunneling, Coulomb blockade, and resonant transport of nonlocal spin-entangled electrons,” Phys. Rev. B 63, 165314 (2001).
  * Lesovik  _et al._ (2001) G. B. Lesovik, T. Martin, and G. Blatter, “Electronic entanglement in the vicinity of a superconductor,” Eur. Phys. J. B 24, 287 (2001).
  * Brunner  _et al._ (2014) N. Brunner, D. Cavalcanti, S. Pironio, V. Scarani, and S. Wehner, “Bell nonlocality,” Rev. Mod. Phys. 86, 419 (2014).
  * Zagoskin (2011) A. Zagoskin,  _Quantum Engineering: Theory and Design of Quantum Coherent Structures_ (Cambridge University Press, 2011).
  * Shevchenko (2019) S. N. Shevchenko,  _Mesoscopic Physics meets Quantum Engineering_ (World Scientific Press, 2019).
  * Beckmann  _et al._ (2004) D. Beckmann, H. B. Weber, and H. v. Löhneysen, “Evidence for Crossed Andreev Reflection in Superconductor-Ferromagnet Hybrid Structures,” Phys. Rev. Lett. 93, 197003 (2004).
  * Russo  _et al._ (2005) S. Russo, M. Kroug, T. M. Klapwijk, and A. F. Morpurgo, “Experimental Observation of Bias-Dependent Nonlocal Andreev Reflection,” Phys. Rev. Lett. 95, 027002 (2005).
  * Yeyati  _et al._ (2007) A. Levi Yeyati, F. S. Bergeret, A. Martin-Rodero, and T. M. Klapwijk, “Entangled Andreev pairs and collective excitations in nanoscale superconductors,” Nature Phys. 3, 455 (2007).
  * Hofstetter  _et al._ (2009) L. Hofstetter, S. Csonka, J. Nygård, and C. Schönenberger, “Cooper pair splitter realized in a two-quantum-dot Y-junction,” Nature 461, 960 (2009).
  * Cadden-Zimansky  _et al._ (2009) P. Cadden-Zimansky, J. Wei, and V. Chandrasekhar, “Cooper-pair-mediated coherence between two normal metals,” Nature Phys. 5, 393 (2009).
  * Herrmann  _et al._ (2010) L. G. Herrmann, F. Portier, P. Roche, A. Levy Yeyati, T. Kontos, and C. Strunk, “Carbon Nanotubes as Cooper-Pair Beam Splitters,” Phys. Rev. Lett. 104, 026801 (2010).
  * Wei and Chandrasekhar (2010) J. Wei and V. Chandrasekhar, “Positive noise cross-correlation in hybrid superconducting and normal-metal three-terminal devices,” Nature Phys. 6, 494 (2010).
  * Hofstetter  _et al._ (2011) L. Hofstetter, S. Csonka, A. Baumgartner, G. Fülöp, S. d’Hollosy, J. Nygård, and C. Schönenberger, “Finite-Bias Cooper Pair Splitting,” Phys. Rev. Lett. 107, 136801 (2011).
  * (14) L. G. Herrmann, P. Burset, W. J. Herrera, F. Portier, P. Roche, C. Strunk, A. Levy Yeyati, and T. Kontos, “Spectroscopy of non-local superconducting correlations in a double quantum dot,” arXiv:1205.1972 .
  * Das  _et al._ (2012) A. Das, R. Ronen, M. Heiblum, D. Mahalu, A. V. Kretinin, and H. Shtrikman, “High-efficiency Cooper pair splitting demonstrated by two-particle conductance resonance and positive noise cross-correlation,” Nature Commun. 3, 1165 (2012).
  * Fülöp  _et al._ (2014) G. Fülöp, S. d’Hollosy, A. Baumgartner, P. Makk, V. A. Guzenko, M. H. Madsen, J. Nygård, C. Schönenberger, and S. Csonka, “Local electrical tuning of the nonlocal signals in a Cooper pair splitter,” Phys. Rev. B 90, 235412 (2014).
  * Schindele  _et al._ (2014) J. Schindele, A. Baumgartner, R. Maurand, M. Weiss, and C. Schönenberger, “Nonlocal spectroscopy of Andreev bound states,” Phys. Rev. B 89, 045422 (2014).
  * Tan  _et al._ (2015) Z. B. Tan, D. Cox, T. Nieminen, P. Lahteenmaki, D. Golubev, G. B. Lesovik, and P. J. Hakonen, “Cooper Pair Splitting by Means of Graphene Quantum Dots,” Phys. Rev. Lett. 114, 096602 (2015).
  * Fülöp  _et al._ (2015) G. Fülöp, F. Domínguez, S. d’Hollosy, A. Baumgartner, P. Makk, M. H. Madsen, V. A. Guzenko, J. Nygård, C. Schönenberger, A. Levy Yeyati, and S. Csonka, “Magnetic Field Tuning and Quantum Interference in a Cooper Pair Splitter,” Phys. Rev. Lett. 115, 227003 (2015).
  * Borzenets  _et al._ (2016) I. V. Borzenets, Y. Shimazaki, G. F. Jones, M. F. Craciun, S. Russo, M. Yamamoto, and S. Tarucha, “High Efficiency CVD Graphene-lead (Pb) Cooper Pair Splitter,” Sci. Rep. 6, 23051 (2016).
  * Schindele  _et al._ (2012) J. Schindele, A. Baumgartner, and C. Schönenberger, “Near-Unity Cooper Pair Splitting Efficiency,” Phys. Rev. Lett. 109, 157002 (2012).
  * Deacon  _et al._ (2015) R. S. Deacon, A. Oiwa, J. Sailer, S. Baba, Y. Kanai, K. Shibata, K. Hirakawa, and S. Tarucha, “Cooper pair splitting in parallel quantum dot Josephson junctions,” Nature Commun. 6, 7446 (2015).
  * Ueda  _et al._ (2019) K. Ueda, S. Matsuo, H. Kamata, S. Baba, Y. Sato, Y. Takeshige, K. Li, S. Jeppesen, L. Samuelson, H. Xu, and S. Tarucha, “Dominant nonlocal superconducting proximity effect due to electron-electron interaction in a ballistic double nanowire,” Science Adv. 5, eaaw2194 (2019).
  * Tan  _et al._ (2021) Z. B. Tan, A. Laitinen, N. S. Kirsanov, A. Galda, V. M. Vinokur, M. Haque, A. Savin, D. S. Golubev, G. B. Lesovik, and P. J. Hakonen, “Thermoelectric current in a graphene Cooper pair splitter,” Nature Commun. 12, 138 (2021).
  * Bruhat  _et al._ (2018) L. E. Bruhat, T. Cubaynes, J. J. Viennot, M. C. Dartiailh, M. M. Desjardins, A. Cottet, and T. Kontos, “Circuit QED with a quantum-dot charge qubit dressed by Cooper pairs,” Phys. Rev. B 98, 155313 (2018).
  * Pandey  _et al._ (2021) P. Pandey, R. Danneau, and D. Beckmann, “Ballistic Graphene Cooper Pair Splitter,” Phys. Rev. Lett. 126, 147701 (2021).
  * Schoelkopf  _et al._ (1998) R. J. Schoelkopf, P. Wahlgren, A. A. Kozhevnikov, P. Delsing, and D. E. Prober, “The Radio-Frequency Single-Electron Transistor (RF-SET): A Fast and Ultrasensitive Electrometer,” Science 280, 1238 (1998).
  * Gustavsson  _et al._ (2009) S. Gustavsson, R. Leturcq, M. Studer, I. Shorubalko, T. Ihn, K. Ensslin, D. C. Driscoll, and A. C. Gossard, “Electron counting in quantum dots,” Surf. Sci. Rep. 64, 191 (2009).
  * Pekola  _et al._ (2013) J. P. Pekola, O.-P. Saira, V. F. Maisi, A. Kemppinen, M. Möttönen, Yu. A. Pashkin, and D. V. Averin, “Single-electron current sources: Toward a refined definition of the ampere,” Rev. Mod. Phys. 85, 1421 (2013).
  * Maisi  _et al._ (2011) V. F. Maisi, O.-P. Saira, Yu. A. Pashkin, J. S. Tsai, D. V. Averin, and J. P. Pekola, “Real-Time Observation of Discrete Andreev Tunneling Events,” Phys. Rev. Lett. 106, 217003 (2011).
  * Maisi  _et al._ (2014) V. F. Maisi, D. Kambly, C. Flindt, and J. P. Pekola, “Full Counting Statistics of Andreev Tunneling,” Phys. Rev. Lett. 112, 036801 (2014).
  * Koski  _et al._ (2014) J. V. Koski, V. F. Maisi, J. P. Pekola, and D. V. Averin, “Experimental realization of a szilard engine with a single electron,” Proceedings of the National Academy of Sciences 111, 13786–13789 (2014).
  * Singh  _et al._ (2019) S. Singh, É. Roldán, I. Neri, I. M. Khaymovich, D. S. Golubev, V. F. Maisi, J. T. Peltonen, F. Jülicher, and J. P. Pekola, “Extreme reductions of entropy in an electronic double dot,” Phys. Rev. B 99, 115422 (2019).
  * Loudon (2000) R. Loudon,  _The Quantum Theory of Light_ (Oxford University Press, 2000).
  * Walldorf  _et al._ (2018) N. Walldorf, C. Padurariu, A.-P. Jauho, and C. Flindt, “Electron Waiting Times of a Cooper Pair Splitter,” Phys. Rev. Lett. 120, 087701 (2018).
  * Walldorf  _et al._ (2020) N. Walldorf, F. Brange, C. Padurariu, and C. Flindt, “Noise and full counting statistics of a Cooper pair splitter,” Phys. Rev. B 101, 205422 (2020).
  * Wrześniewski and Weymann (2020) K. Wrześniewski and I. Weymann, “Current cross-correlations and waiting time distributions in Andreev transport through Cooper pair splitters based on a triple quantum dot system,” Phys. Rev. B 101, 155409 (2020).
  * Bäuerle  _et al._ (2018) C. Bäuerle, D. C. Glattli, T. Meunier, F. Portier, P. Roche, P. Roulleau, S. Takada, and X. Waintal, “Coherent control of single electrons: a review of current progress,” Rep. Prog. Phys. 81, 056503 (2018).
  * Hanson  _et al._ (2007) R. Hanson, L. P. Kouwenhoven, J. R. Petta, S. Tarucha, and L. M. K. Vandersypen, “Spins in few-electron quantum dots,” Rev. Mod. Phys. 79, 1217 (2007).
  * Zwanenburg  _et al._ (2013) Floris A. Zwanenburg, Andrew S. Dzurak, Andrea Morello, Michelle Y. Simmons, Lloyd C. L. Hollenberg, Gerhard Klimeck, Sven Rogge, Susan N. Coppersmith, and Mark A. Eriksson, “Silicon quantum electronics,” Rev. Mod. Phys. 85, 961–1019 (2013).
  * Cottet  _et al._ (2004a) A. Cottet, W. Belzig, and C. Bruder, “Positive Cross Correlations in a Three-Terminal Quantum Dot with Ferromagnetic Contacts,” Phys. Rev. Lett. 92, 206801 (2004a).
  * Cottet  _et al._ (2004b) A. Cottet, W. Belzig, and C. Bruder, “Positive cross-correlations due to dynamical channel blockade in a three-terminal quantum dot,” Phys. Rev. B 70, 115315 (2004b).
  * Kłobus  _et al._ (2014) W. Kłobus, A. Grudka, A. Baumgartner, D. Tomaszewski, C. Schönenberger, and J. Martinek, “Entanglement witnessing and quantum cryptography with nonideal ferromagnetic detectors,” Phys. Rev. B 89, 125404 (2014).
  * Brange  _et al._ (2017) F. Brange, O. Malkoc, and P. Samuelsson, “Minimal Entanglement Witness from Electrical Current Correlations,” Phys. Rev. Lett. 118, 036804 (2017).

Acknowledgements. We thank J. P. Pekola and P. Samuelsson for fruitful
discussions. The work was supported financially by the QuantERA project “2D
hybrid materials as a platform for topological quantum computing”, Swedish
National Science Foundation, NanoLund and the Academy of Finland (project
numbers 308515 and 331737). F.B. acknowledges support from the European
Union’s Horizon 2020 research and innovation programme under the Marie
Skłodowska-Curie grant agreement number 892956. F.B., E.T.M., and C.F.
acknowledge the support by the Academy of Finland through the Finnish Centre
of Excellence in Quantum Technology (project numbers 312057 and 312299). We
acknowledge the provision of facilities by Aalto University at OtaNano -
Micronova Nanofabrication Centre.

Author contributions. The experiment was carried out by A.R. and V.F.M. The
devices were fabricated by E.T.M. The theory was developed by F.B. and C.F.
All authors contributed to the discussion and analysis of the results and the
writing of the manuscript.

Competing interests. The authors declare no competing interests.