# Model-Agnostic Multi-Agent Perception Framework

Runsheng Xu1∗, Weizhe Chen2∗, Hao Xiang1∗, Xin Xia1, Lantao Liu2, Jiaqi Ma1
*The first three authors contribute equally.1Runsheng Xu, Hao Xiang, Xin Xia,
and Jiaqi Ma are with the University of California, Los Angeles, CA, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>Chen and Lantao
Liu are with Indiana University, Bloomington, IN, USA {chenweiz,
<EMAIL_ADDRESS>
###### Abstract

Existing multi-agent perception systems assume that every agent utilizes the
same model with identical parameters and architecture. The performance can be
degraded with different perception models due to the mismatch in their
confidence scores. In this work, we propose a model-agnostic multi-agent
perception framework to reduce the negative effect caused by the model
discrepancies without sharing the model information. Specifically, we propose
a confidence calibrator that can eliminate the prediction confidence score
bias. Each agent performs such calibration independently on a standard public
database to protect intellectual property. We also propose a corresponding
bounding box aggregation algorithm that considers the confidence scores and
the spatial agreement of neighboring boxes. Our experiments shed light on the
necessity of model calibration across different agents, and the results show
that the proposed framework improves the baseline 3D object detection
performance of heterogeneous agents.

##  I Introduction

Recent advancements in deep learning have improved the performance of modern
perception systems on many tasks, such as object detection [1, 2, 3], semantic
segmentation [4, 5], and visual navigation [6, 7]. Despite the remarkable
progress, single-agent perception systems still have many limitations due to
single-view constraints. For instance, autonomous vehicles (AVs) usually
suffer from occlusion [8], and such situations are difficult to handle because
of the lack of sensory observations of the occluded area. To address this
issue, recent studies [9, 10, 11, 12, 13, 14, 15] have explored wireless
communication technology to enable nearby agents to share the sensory
information and collaboratively perceive the surrounding environment.

Although existing fusion frameworks have obtained a significant 3D object
detection performance boost, they assume that all the collaborating agents
share an identical model with the same parameters. This assumption is hard to
satisfy in practice, particularly in autonomous driving. Distributing the
model parameters among AVs might raise privacy and confidentiality concerns,
especially for vehicles from different automotive companies. Even for AVs from
the same company, the detection models can have various versions, depending on
the vehicle type and model updating frequency. Without adequately handling the
inconsistency, the shared sensory information can have a large domain gap, and
the advantage brought by multi-agent perception will be diminished rapidly.

$0.81$$0.79$$0.78$$0.79$…$0.81$$0.78$$0.71$$0.68$…(a)(b)

(a)  Figure 1: Ground truth (green) and bounding box candidates (red) produced
by three connected autonomous vehicles. (a) Some agents have confidence scores
that are systematically larger than others, e.g., the blue scores versus the
orange scores. However, they might be confidently wrong, which mislead the
fusion process. (b) Candidates with slightly lower confidence scores (orange)
but higher spatial agreement with neighboring boxes can be better than a
singleton with a higher confidence score (blue).

To this end, we propose a model-agnostic multi-agent perception framework to
handle the model heterogeneity while maintaining confidentiality. The
perception outputs (i.e., detected bounding boxes and confidence scores) are
shared to bypass the dependency on the underlying model’s detailed
information. Due to the distinct models used by the agents, the confidence
scores provided by different agents can be systematically misaligned. Some
agents may be over-confident, whereas others tend to be under-confident.
Directly fusing bounding box proposals from neighboring agents using, for
example, Non-Maximum Suppression (NMS)[16] can result in poor detection
accuracy due to the presence of over-confident and low-quality candidates.

We propose a simple yet flexible confidence calibrator, called Doubly Bounded
Scaling (DBS), to mitigate the misalignment. We also propose a corresponding
bounding box aggregation algorithm, named Promote-Suppress Aggregation (PSA),
that considers the confidence scores and the spatial agreement of neighboring
boxes. Fig. 1 illustrates the importance of these two components. This
framework does not reveal model design and parameters, ensuring
confidentiality. We evaluate our approach on an open-source large-scale multi-
agent perception dataset – OPV2V [12]. Experiments show that in the presence
of model discrepancies among agents, our framework significantly improves
multi-agent LiDAR-based 3D object detection performance, outperforming the
baselines by at least 6% in terms of Average Precision (AP).

##  II Related Work

Multi-Agent Perception. Multi-agent perception investigates how to leverage
visual cues from neighboring agents through the communication system to
enhance the perception capability. There are three categories of existing work
according to the information sharing schema: 1) early fusion [8], where raw
point clouds are transmitted directly and projected into the same coordinate
frame, 2) late fusion [17], where detected bounding boxes and confidence
scores are shared, and 3) intermediate fusion [11, 10, 12, 9, 14], where
compressed latent neural features extracted from point clouds are propagated.
Though early fusion has no information loss, it usually requires large
bandwidth. Intermediate fusion can achieve a good balance between accuracy and
transmission data size, but it requires the complete knowledge of each agent’s
model, which is non-trivial to satisfy in reality due to intellectual property
concerns. On the contrary, late fusion only needs the outputs of the detector
without demanding access to the underlying neural networks, which are
typically confidential for automotive companies. Therefore, our approach
adopts the late fusion strategy but further designs customized new components
to address the model discrepancy issue in vanilla late fusion.

3D LiDAR Detection. To tackle the irregular or unstructured data format of
point clouds, researchers have come up with point-based, voxel-based, and
point-voxel-based methods. Frustum PointNet [18] uses 2D image detection
bounding boxes to generate frustums on raw point clouds. Then, we can directly
operate the point clouds in the frustums to obtain the final bounding box
positions. PointRCNN [19] develops a two-stage framework for 3D detection,
which first produces rough bounding box proposals and then fine-tunes them in
the second stage. McCraith et al. [20] combines outlier detection [21, 22, 23]
and PoinNet to make precise predictions. In [1, 24, 25], point clouds are
aggregated into voxels and generate latent features per voxel. Such an
approach usually follows a one-stage fashion, with less accuracy but lower
inference latency than the two-stage methods. [26, 27] integrate both voxel-
based network and PointNet-based [28] set abstraction to produce more robust
point cloud features, which can keep high learning efficiency while enjoying
flexible receptive fields of the PointNet-based networks.

Confidence Calibration. For a probabilistic classifier, the probability
associated with the predicted class label should reflect its correctness
likelihood. However, many modern neural networks do not have such property
[29]. Confidence calibration aims to endow a classifier with such property.
Calibration methods can be tightly coupled with the neural networks, such as
Bayesian neural networks and regularization techniques [30, 31, 32], or serve
as a post-processing step. Post-processing methods include histogram binning
methods [33], scaling methods [34, 35], and mixtures[36] that combine the
first two branches. Due to the popularity of the Temperature Scaling method
[29] which is a single-parameter version of Platt Scaling [34], scaling
methods are widely adopted for calibrating neural networks. Our proposed
method follows the same fashion.

Bounding Box Aggregation. Object detection models typically require bounding
box aggregation to lump the proposals corresponding to the same object. The de
facto standard post-processing method is Non-Maximum Suppression (NMS) [16,
37], which sequentially selects the proposals with the highest confidence
score and then suppresses other overlapped boxes. NMS does not fully exploit
information in the proposals because it only uses the relative order of
confidence, ignoring the absolute confidence scores and the spatial
information hidden in the bounding box coordinates. Several works have been
proposed to refine the box aggregation strategies. Soft-NMS [38] softly decays
the confidence scores of the proposals proportional to the degree of overlap.
In [37] NMS can be learned by a neural network to achieve better occlusion
handling and bounding box localization. Adaptive NMS [39] applies a dynamic
suppression threshold to an instance according to the target object density.
Rothe et al. [40] formulate NMS as a clustering problem and use Affinity
Propagation Clustering to solve the problem. The idea of message passing
between proposals is related to the aggregation algorithm introduced in
Section III-C, but our update rules are simpler and more efficient.

Public DatasetDetectorClassificationConfidenceConfidence Calibrator(Doubly
Bounded Scaling)Promote-SuppressAggregationOffline TrainingOnline Inference
(a)  Figure 2: Overview of the proposed framework. Each agent trains its
confidence calibrator (i.e., Doubly Bounded Scaling) on the same public
dataset offline (orange arrows). Promote-Suppress Aggregation yields the final
detection result, considering the spatial information and calibrated
confidence of bounding boxes given by connected autonomous vehicles.

##  III Methodology

In this paper, we consider the cooperative perception of a heterogeneous
multi-agent system, where agents communicate to share sensing information from
different perception models without revealing model information, i.e., model-
agnostic collaboration. We focus on a 3D LiDAR detection task in autonomous
driving, but the methodology can also be customized and used in other
cooperative perception applications. Our goal is to develop a robust framework
to handle the heterogeneity among agents while preserving confidentiality. The
proposed model-agnostic collaborative perception framework is shown in Fig. 2,
which can be divided into two stages. In the offline stage, we train a model-
specific calibrator. During the online phase, real-time on-road sensing
information is calibrated and aggregated.

###  III-A Model-Agnostic Fusion Pipeline

Agents with distinct perception models usually generate systematically
different confidence. The mismatch in confidence distributions can affect the
fusion performance. For instance, an inferior model may be over-confident and
dominate the aggregation process, decreasing the accuracy of the final
results.

To alleviate this issue, we train a calibrator offline for each model to align
its confidence score to its empirical accuracy on a calibration dataset.
Concretely, each agent first runs its well-trained detector on the same public
dataset to produce a model-specific calibration dataset containing labels and
confidence scores. The calibration dataset is then fed to a calibrator for
training (see Section III-B for more details). After training, the calibrator
is saved locally in each agent.

When the vehicle is driving on-road and making predictions from the sensor
measurements, the calibrator will align the predicted confidence score towards
the same standard, thus alleviating the aforementioned mismatch. Then the
bounding box coordinates and calibrated confidence scores are packed together
and transmitted to neighboring agents. The receiving agent (i.e., ego vehicle)
will fuse the shared information via the Promote-Suppress Aggregation
algorithm (see Section III-C for details) to output the final results. Since
each agent learns its calibrator independently in the offline stage and only
shares the detection outputs during the online phase, the detector
architecture and parameters are invisible to other agents, protecting the
intellectual property.

###  III-B Classification Confidence Calibration

To eliminate the bias brought by the system heterogeneity, the models need to
be _well-calibrated_. If the confidence scores can imply the likelihood of
correct prediction, for example, $80\%$ confidence leads to $80\%$ accurate
predictions, this model is _well-calibrated_. Formally, let $\tilde{s}$ be the
confidence score produced by the model and $y\in\\{0,1\\}$ be the label
indicating vehicle or background111We discuss binary classification here for
simplicity but the proposed framework can be generalized to the multi-class
case.. A model is well-calibrated if its confidence score $\tilde{s}$ matches
the expectation of correctly predicting the label:

$\mathbb{E}[y=1\mid\tilde{s}]=\tilde{s}.$ (1)

Scaling-Based Confidence Calibration. The goal of scaling-based confidence
calibration is to learn a parametric scaling function (i.e., calibrator)
$\mathtt{c}_{\bm{\theta}}(\tilde{s}):[0,1]\mapsto[0,1]$ on a calibration
dataset to transform the uncalibrated confidence scores $\tilde{s}$ into well-
calibrated ones $s$. Given a calibration set
$\mathcal{D}\triangleq\\{(\tilde{s}_{n},y_{n})\\}_{n=1}^{N}$ containing the
model-dependent confidence scores $\tilde{s}$ and ground-truth labels $y$, we
optimize the parameters $\bm{\theta}$ of the calibrator
$\mathtt{c}_{\bm{\theta}}(\tilde{s})$ by gradient descent on the binary cross
entropy loss

$\ell_{CE}=-y_{n}\log(s_{n})-(1-y_{n})\log\left(1-s_{n}\right),$ (2)

where $s_{n}=\mathtt{c}_{\bm{\theta}}(\tilde{s}_{n})$. Training a parametric
function by optimizing Eq. 2 is similar to standard binary classification,
however, extra constraints are required on the scaling function for confidence
calibration . Designing a suitable calibrator for our application requires
satisfying three conditions: (a) The scaling function needs to be
monotonically non-decreasing as a higher confidence score is supposed to
indicate a higher expected accuracy; (b) The scaling function should be
relatively smooth to avoid over-fitting to the calibration set; (c) The
scaling function is supposed to be doubly bounded, meaning that it maps a
confidence interval $[0,1]$ to the same $[0,1]$ range. In the following sub-
sections, we will explain why the commonly used calibration methods do not
meet all these conditions, which motivates the development of our proposed
calibrator.

(a) Logistic

(b) Kumaraswamy

Figure 3: Scaling functions with various parameters that follow (a) the
logistic form and (b) the Kumaraswamy CDF. Note that, in (b), the “inverse-
sigmoid” shape (green curve, $a=0.4,b=0.4$) and the identity map (orange
curve, $a=1,b=1$) are not in the logistic family.

Platt Scaling and Temperature Scaling. The most popular scaling methods are
arguably Platt Scaling [34] and Temperature Scaling [29]. Platt Scaling uses
the logistic family as the calibrator:

$\mathtt{c}_{\text{Platt}}(\tilde{s};a,b)=\frac{1}{1+\exp\left(-(a\times{\tilde{s}}+b)\right)},$
(3)

where $a,b$ are parameters with $a\geq{0}$ to ensure that the calibration map
is monotonically non-decreasing. Temperature Scaling is a special case of Eq.
3 where $b$ is fixed to $0$. Fig. 3 shows several scaling functions from this
family. Platt Scaling can fail if its parametric assumptions are not met [41].
For example, we cannot learn an “inverse-sigmoid” (see the green curve in Fig.
3b) scaling function within this family. Furthermore, the identity function is
also not a member of the logistic family. In addition to the aforementioned
limitations, the logistic family is also not a function family that can
naturally map $[0,1]$ to $[0,1]$ as its input domain is $\mathbb{R}$,
therfore, these popular choices are not our ideal candidates.

Doubly Bounded Scaling (DBS). We propose to use the Kumaraswamy Cumulative
Density Function (CDF) [42] that meets all the three constraints while being
sufficiently flexible. To the best of our knowledge, this is the first time
that this function family has been adopted in confidence calibration.
Specifically, we learn a scaling function with the following form

$\mathtt{c}(\tilde{s};a,b)=1-\left(1-\tilde{s}^{a}\right)^{b},$ (4)

where $a>0$ and $b>0$ are the parameters. Scaling functions that follow Eq. 4
are monotonically non-decreasing, smooth, and doubly bounded, hence the name.
we can see that DBS is more flexible than the logistic form by comparing Fig.
3a and Fig. 3b. For each detector, we optimize the $a$ and $b$ on a
calibration dataset by minimizing Eq. 2.

PromoteSuppressIntersectionoverUnionUpdatedConfidenceScore

(a)  Figure 4: Illustration of Promote-Suppress Aggregation. The size of a
node indicates the confidence score of the bounding box and the edge width
represents the Intersection-over-Union of two boxes.

###  III-C Promote-Suppress Aggregation (PSA)

Detection models typically output a bunch of overlapped bounding box
candidates for the same detected object, thus we need a post-processing step
to select from these candidates. In most of the detection algorithms, the
optimization objective function is a summation of a bounding box regression
loss and a classification loss. The detector can express its “confidence” by
assigning high classification scores to the promising bounding boxes or
allocating more bounding boxes to the region that it finds relevant features.
To select the high-score bounding boxes with many confident neighbors, we
propose Promote-Suppress Aggregation (PSA), which takes into account both the
regression and classification confidences.

Fig. 4 illustrates the idea of PSA. We first construct a spatial graph of
bounding box candidates based on Intersection-over-Union (IoU) values and the
confidence scores. In the promotion step, the IoU weighted confidence scores
are propagated to the neighboring nodes. We design the propagation rule to
meet the following desiderata:

  * •

A candidate should be promoted if many other candidate boxes have large
intersections with it;

  * •

A candidate with many high-score neighbors should be promoted;

  * •

If possible, the update rules should be parallelizable and permutation-
invariant. Namely, the propagation order does not change the result.

In the suppression step, the candidate with the highest updated score will
softly suppress the scores of other candidates. Finally, we select one or more
bounding boxes that rank in the first (few) places. The idea of soft
suppression and selecting more than one candidate is akin to soft-NMS [38],
which is beneficial when the bounding box of a small object is within the box
of a large object. Below we formally describe the PSA algorithm.

######  Definition 1 (Bounding Box Graph).

Let $\mathcal{G}$ be a weighted graph with a set of edges $\mathcal{E}$ and a
set of nodes/vertices $\mathcal{V}$, where each vertex $v\in\mathcal{V}$
represents a bounding box candidate $b$ with an associated confidence score
$s$ after calibration. The edge weigh $w_{ij}$ between vertex $v_{i}$ and
$v_{j}$ is defined as the Intersection-over-Union value
$\mathtt{IoU}(b_{i},b_{j})\triangleq\nicefrac{{\cap(b_{i},b_{j})}}{{\cup(b_{i},b_{j})}}$.
An edge connects vertex $v_{i}$ and $v_{j}$ if the edge weight is non-zero.

######  Definition 2 (Connected Components).

The graph consists of a number of connected components in which every pair of
nodes is connected via a sequence of edges.

######  Problem 1 (Bounding Box Aggregation).

Given the Intersection-over-Union matrix $\mathbf{U}\in[0,1]^{N\times{N}}$
among $N$ bounding box candidates
$\mathbf{B}=[\mathbf{b}_{1},\dots,\mathbf{b}_{N}]^{{\mathchoice{\raisebox{0.0pt}{$\displaystyle\intercal$}}{\raisebox{0.0pt}{$\textstyle\intercal$}}{\raisebox{0.0pt}{$\scriptstyle\intercal$}}{\raisebox{0.0pt}{$\scriptscriptstyle\intercal$}}}}$
and their confidence scores
$\mathbf{s}=[s_{1},\dots,s_{N}]^{{\mathchoice{\raisebox{0.0pt}{$\displaystyle\intercal$}}{\raisebox{0.0pt}{$\textstyle\intercal$}}{\raisebox{0.0pt}{$\scriptstyle\intercal$}}{\raisebox{0.0pt}{$\scriptscriptstyle\intercal$}}}}$,
our goal is to compute an index set $\mathcal{I}$ to select/filter candidates
that best match the ground-truth bounding boxes.

Algorithm 1 Promote-Suppress Aggregation

Arguments: bounding boxes
$\mathbf{B}=[\mathbf{b}_{1},\dots,\mathbf{b}_{N}]^{{\mathchoice{\raisebox{0.0pt}{$\displaystyle\intercal$}}{\raisebox{0.0pt}{$\textstyle\intercal$}}{\raisebox{0.0pt}{$\scriptstyle\intercal$}}{\raisebox{0.0pt}{$\scriptscriptstyle\intercal$}}}}$,
confidence score vector
$\mathbf{s}=[s_{1},\dots,s_{N}]^{{\mathchoice{\raisebox{0.0pt}{$\displaystyle\intercal$}}{\raisebox{0.0pt}{$\textstyle\intercal$}}{\raisebox{0.0pt}{$\scriptstyle\intercal$}}{\raisebox{0.0pt}{$\scriptscriptstyle\intercal$}}}}$,
soft selection parameters $\varepsilon$, and threshold $\phi$

1:Initialize selected box indices to an empty set $\mathcal{I}=\emptyset$

2:Compute IoU matrix $\mathbf{U}\in[0,1]^{N\times{N}}$ using $\mathbf{B}$

3:Find vertex indices of connected components
$\mathcal{C}\triangleq\\{\mathbf{c}_{m}\\}_{m=1}^{M}$

4:for each $\mathbf{c}_{m}\in\mathcal{C}$ do

5: Extract IoU sub-matrix $\mathbf{U}_{m}\in[0,1]^{N_{m}\times{N_{m}}}$ via
$\mathbf{c}_{m}$

6: Extract score sub-vector $\mathbf{s}_{m}\in[0,1]^{N_{m}}$ via
$\mathbf{c}_{m}$

7: $\mathbf{\hat{s}}_{m}=\mathbf{U}_{m}\mathbf{s}_{m}$$\triangleright$ Promote

8:
$\mathbf{\bar{s}}_{m}=\mathtt{softmax}(\nicefrac{{\mathbf{\hat{s}}_{m}}}{{\varepsilon}})$$\triangleright$
Suppress

9:
$\mathcal{I}=\mathcal{I}\cup\\{c_{m}^{(n)}\mid\bar{s}_{m}^{(n)}>\phi,n=1,\dots,N_{m}\\}$$\triangleright$
Select

10:return selected candidate indices $\mathcal{I}$

Algorithm 1 shows how PSA computes the index set. Given the IoU adjacency
matrix, we can find out the indices of each component and put them into a
component set $\mathcal{C}=\\{\mathbf{c}_{m}\\}_{m=1}^{M}$, where $M$ is the
number of components and $\mathbf{c}_{m}$ contains the indices of $N_{m}$
vertices (line 3). For each component, we extract the IoU matrix
$\mathbf{U}_{m}\in[0,1]^{N_{m}\times{N_{m}}}$ and confidence score vector
$\mathbf{s}_{m}\in[0,1]^{N_{m}}$ corresponding to this component (line 5-6).
Then, we perform the promotion step
$\mathbf{\hat{s}}_{m}=\mathbf{U}_{m}\mathbf{s}_{m}$ where each vertex updates
its score to be the IoU-weighted sum of scores from other vertices in the
component (line 7). In the suppression step, we normalize the updated scores
back to $[0,1]$ and distill the winning candidate via
$\mathbf{\bar{s}}_{m}=\mathtt{softmax}(\nicefrac{{\mathbf{\hat{s}}_{m}}}{{\varepsilon}})$
(line 8). In the end, indices with updated scores larger than a threshold are
added to the set $\mathcal{I}$ (line 9). We can select multiple candidates if
$\varepsilon\in(0,1]$ is large and $\phi$ is small. In our application,
however, one component typically contains only one object/vehicle, so we use a
small $\varepsilon$ and $\phi=0.5$. Overall, PSA is highly parallelizable as
each component operates independently and each step only requires simple
linear search or small matrix-vector multiplication.

##  IV Experiments

###  IV-A Dataset

We evaluate the proposed framework on a large-scale open-source multi-agent
perception dataset OPV2V [12], which is simulated using the high-fidelity
simulator CARLA [43] and a cooperative driving automation simulation framework
OpenCDA [44]. It includes $73$ scenarios with an average of $25$ seconds
duration. In each scene, various numbers ($2$ to $7$) of Autonomous Vehicles
(AVs) provide LiDAR point clouds from their viewpoints. The
train/validation/test splits are $6764/1981/2169$ frames, respectively. For
details of the dataset, please refer to [12].

TABLE I: Object detection performance. Average Precision (AP) at IoU=0.7 on Homo, Hetero1, and Hetero2 setting. Methods |  | Homo
---
<EMAIL_ADDRESS>
| Hetero1
---
<EMAIL_ADDRESS>
| Hetero2
---
<EMAIL_ADDRESS>
No fusion | 0.602 | 0.602 | 0.602
Intermediate w/o calibration | 0.815 | 0.677 | 0.571
Late fusion w/o calibration | 0.781 | 0.691 | 0.723
Our method | 0.813 | 0.750 | 0.784

###  IV-B Experiment Setup

Evaluation metric. Following [44], we evaluate the detection accuracy in the
range of $x\in[-140,140]\text{m}$ and $y\in[-40,40]\text{m}$, centered at the
ego-vehicle coordinate frame. The detection performance is measured with
Average Precision (AP) at $IoU=0.7$.

Evaluation setting. We evaluate our method under three different settings: 1)
Homo Setting, where the detectors of agents are homogeneous with the same
architecture and trained parameters. This setting has no confidence
distribution gap and is used to demonstrate the performance drop when taking
heterogeneity into account; 2) Hetero Setting 1, where the agents have the
same model architecture but different parameters; 3) Hetero Setting 2, where
the detector architectures are disparate. For Homo Setting, we select pre-
trained Pointpillar [45] as the backbone for all the AVs. For Hetero Setting
1, the ego vehicle employs the same pre-trained Pointpillar model as in Homo
Setting, whereas other AVs pick the parameters of Pointpillar from a different
epoch during training. Likewise, in the Hetero Setting 2, the ego vehicle
utilizes Pointpillar while other AVs use SECOND [25] for detection. As
intermediate fusion requires equal feature map resolution, we apply simple bi-
linear interpolation under this setting. The ego vehicle uses the identical
model with the same parameters across all settings for the No Fusion and Late
Fusion. To compare with existing calibrators, we use the same calibration
method for all agents, but the parameters are agent-specific. The proposed
framework should also work even when the calibration methods across agents are
heterogeneous, as long as the prediction bias is effectively reduced.

Compared methods. We regard No Fusion as the baseline, which only takes the
ego vehicle’s LiDAR data as input and omits any collaboration. Ideally, the
multi-agent system should at least outperform this baseline. To validate the
necessity of the calibration, we compare our method with naive late fusion and
intermediate fusion that ignore calibrations. The naive late fusion gathers
all detected bounding box positions and confidence scores together and simply
applies NMS to produce the final results. The intermediate fusion method is
the same as the one in [12]. We exclude the early fusion in the comparison as
it requires large bandwidth, which leads to high communication delay thus is
impractical to be deployed in the real world. Moreover, we also compare the
proposed Doubly Bounded Scaling (DBS) with two other commonly used scaling-
based calibrators: Temperature Scaling (TS) [29] and Platt Scaling (PS) [34].

TABLE II: Component ablation study. Components | Hetero1 | Hetero2
---|---|---
DBS | PSA |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
|  | 0.691 | 0.723
✓ |  | 0.734 | 0.776
✓ | ✓ | 0.750 | 0.784

###  IV-C Quantitative Evaluation

Main performance analysis. Table I describes the performance comparisons of
different methods under Homo, Hetero1, and Hetero2 Setting. In the unrealistic
Homo setting, all methods exceed the baseline remarkably while intermediate
fusion and our method have very close performance (0.2% difference). However,
when we consider the realistic model discrepancy factor, our method
outperforms the classic late fusion and intermediate fusion significantly by
5.9%, 7.3% under Hetero1 Setting, and by 6.1%, 21.3% under Hetero2 Setting,
respectively. The classic late fusion and intermediate fusion suffer from the
model discrepancy, leading to clear accuracy decreases. In the Hetero2
Setting, the intermediate fusion even becomes lower than the baseline. On the
contrary, our method only drops around 6% and 3% under the two realistic
settings, indicating the effectiveness of the proposed calibration for the
heterogeneity of the multi-agent perception system. Note that although the
design essence of our framework aims to handle the heterogeneous situations,
we also obtain performance boost under the Homo Setting compared with the
standard late fusion that shares detection proposals. We attribute this gain
to PSA and the filtering operation of low-confidence proposals after
confidence calibration that removes some potential false positives.

Major component analysis. Here we investigate the contribution from each
component by incrementally adding DBS and PSA. Table II reveals that both
modules are beneficial for the performance boost, while the calibration
exhibits more contributions – increasing the AP by 4.3% and 5.3% .

Perfect Reliability CurveGap to be calibratedCurrent Reliability
Curve$0$$1$Empirical Accuracy$0.2$$1$Confidence Scores

(a) Reliability Diagram before Calibration

Reliability CurveAfter CalibrationPerfect Reliability Curve$0$$1$Empirical
Accuracy$0.2$$1$Confidence Scores

(b) Calibrated by Doubly Bounded Scaling

Temperature ScalingPlatt ScalingDBS (Ours)Average Precision ($\text{IoU}>0.7$)

(c) Performance of Different Calibrators

Figure 5: The reliability diagrams in (a) and (b) reveal that Doubly Bounded
Scaling method can effectively calibrate the classification confidence scores.
In (c), the proposed Doubly Bounded Scaling outperforms Temperature Scaling
and Platt Scaling under various experiment setups and aggregation algorithms.

(a) Intermediate Fusion in Hetero1

(b) Late Fusion in Hetero1

(c) Ours in Hetero1

(d) Intermediate Fusion in Hetero2

(e) Late Fusion in Hetero2

(f) Ours in Hetero2

Figure 6: Qualitative comparison in a busy freeway and a congested
intersection. Green and red 3D bounding boxes represent the groun truth and
prediction respectively. Our method yields more accurate detection results.

Confidence calibration evaluation. Fig. 5a show the reliability diagram of
Pointpillar used by the ego vehicle, in which a perfect calibration will
produce a diagonal reliability curve, indicating the real accuracy matches the
predictive confidence score. Reliability curves under or above the diagonal
line represent over-confident or under-confident models, respectively.
Pointpillar has much higher empirical accuracy than its reported confidence
score. When using NMS to fuse the predictions of Pointpillar with that of
another inaccurate but over-confident detector, the under-estimated confidence
will result in the removal of Pointpillar’s good predictions. After being
calibrated by DBS, in Fig. 5b, the reliability curve of Pointpillar lies on
the diagonal line.

Comparison with other calibration methods. Fig. 5c describes the comparison
between our DBS calibration and other calibration methods, including TS and
PS. Our DBS achieves better performance than others under both heterogeneous
settings. Moreover, PSA can also improve the accuracy of different calibrators
and experimental settings, showing the generalized capability to refine the
prediction results.

###  IV-D Qualitative Results

Fig. 6 shows the detection results of intermediate fusion, classic late
fusion, and our method under Hetero1 and Hetero2 Setting. Our method can
identify more objects while keeping very few false positives. The zoom-in
examples show that our method can regress the bounding box positions more
accurately, indicating the robustness against the model discrepancy in multi-
agent perception systems.

##  V Conclusions

In the context of cooperative perception, agents from different stakeholders
have heterogeneous models. For the sake of confidentially, information related
to the models and parameters should not be revealed to other agents. In this
work, we present a model-agnostic collaboration framework that addresses two
critical challenges of the vanilla late fusion strategy. First, we propose a
confidence calibrator to align the classification confidence distributions of
different agents. Second, we present a bounding box aggregation algorithm that
takes into account both the calibrated classification confidence and the
spatial congruence information given by bounding box regression. Experiments
on a large-scale cooperative perception dataset shed light on the necessity of
model calibration across heterogeneous agents. The results show that combining
the two proposed techniques can improve the state-of-the-art for cooperative
3D object detection when different agents use distinct perception models.

## References

  * Zhou and Tuzel [2018] Yin Zhou and Oncel Tuzel.  VoxelNet: End-to-end learning for point cloud based 3d object detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4490–4499, 2018.
  * Lu et al. [2021] Yantao Lu, Xuetao Hao, Shiqi Sun, Weiheng Chai, Muchenxuan Tong, and Senem Velipasalar.  Raanet: Range-aware attention network for lidar-based 3d object detection with auxiliary density level estimation, 2021.
  * Fan et al. [2021] Zhaoxin Fan, Yazhi Zhu, Yulin He, Qi Sun, Hongyan Liu, and Jun He.  Deep learning on monocular object pose detection and tracking: A comprehensive overview.  _arXiv preprint arXiv:2105.14291_ , 2021.
  * Pan et al. [2020] Bowen Pan, Jiankai Sun, Ho Yin Tiga Leung, Alex Andonian, and Bolei Zhou.  Cross-view semantic segmentation for sensing surroundings.  _IEEE Robotics and Automation Letters_ , 5(3):4867–4873, 2020.
  * Xiong et al. [2019] Peixi Xiong, Xuetao Hao, Yunming Shao, and Jerry Yu.  Adaptive attention model for lidar instance segmentation.  In _International Symposium on Visual Computing_ , pages 141–155. Springer, 2019.
  * Du et al. [2020] Heming Du, Xin Yu, and Liang Zheng.  Learning object relation graph and tentative policy for visual navigation.  In _European Conference on Computer Vision_ , pages 19–34. Springer, 2020.
  * Pal et al. [2021] Anwesan Pal, Yiding Qiu, and Henrik Christensen.  Learning hierarchical relationships for object-goal navigation.  In _Conference on Robot Learning_ , pages 517–528. PMLR, 2021.
  * Chen et al. [2019a] Qi Chen, Sihai Tang, Qing Yang, and Song Fu.  Cooper: Cooperative perception for connected autonomous vehicles based on 3D point clouds.  In _2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)_ , pages 514–524, 2019a.
  * Wang et al. [2020] Tsun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel Urtasun.  V2VNet: Vehicle-to-vehicle communication for joint perception and prediction.  In _European Conference on Computer Vision (ECCV)_ , pages 605–621. Springer, 2020.
  * Chen et al. [2019b] Qi Chen, Xu Ma, Sihai Tang, Jingda Guo, Qing Yang, and Song Fu.  F-Cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3d point clouds.  In _Proceedings of the 4th ACM/IEEE Symposium on Edge Computing_ , page 88–100, 2019b.
  * Li et al. [2021a] Yiming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang.  Learning distilled collaboration graph for multi-agent perception.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 34, 2021a.
  * Xu et al. [2022a] Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma.  OPV2V: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication.  In _2022 IEEE International Conference on Robotics and Automation (ICRA)_ , 2022a.
  * Li et al. [2021b] Yiming Li, Ziyan An, Zixun Wang, Yiqi Zhong, Siheng Chen, and Chen Feng.  V2x-sim: A virtual collaborative perception dataset for autonomous driving.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)_ , 2021b.
  * Xu et al. [2022b] Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, and Jiaqi Ma.  V2x-vit: Vehicle-to-everything cooperative perception with vision transformer.  _arXiv preprint arXiv:2203.10638_ , 2022b.
  * Song and Festag [2021] R. Song and A. Festag.  Analysis of existing approaches for information sharing in cooperative intelligent transport systems – V2X messaging and SENSORIS.  In _2021 38th FISITA World Congress_ , 2021.
  * Neubeck and Van Gool [2006] Alexander Neubeck and Luc Van Gool.  Efficient non-maximum suppression.  In _18th International Conference on Pattern Recognition (ICPR’06)_ , volume 3, pages 850–855. IEEE, 2006.
  * Rawashdeh and Wang [2018] Zaydoun Yahya Rawashdeh and Zheng Wang.  Collaborative automated driving: A machine learning-based method to enhance the accuracy of shared information.  In _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_ , pages 3961–3966. IEEE, 2018.
  * Qi et al. [2018] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas.  Frustum pointnets for 3d object detection from rgb-d data.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 918–927, 2018.
  * Shi et al. [2019] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li.  Pointrcnn: 3d object proposal generation and detection from point cloud.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 770–779, 2019.
  * McCraith et al. [2021] Robert McCraith, Eldar Insafutdinov, Lukas Neumann, and Andrea Vedaldi.  Lifting 2d object locations to 3d by discounting lidar outliers across objects and views.  _arXiv preprint arXiv:2109.07945_ , 2021.
  * Li et al. [2022] Zheng Li, Yue Zhao, Xiyang Hu, Nicola Botta, Cezar Ionescu, and George H Chen.  Ecod: Unsupervised outlier detection using empirical cumulative distribution functions.  _arXiv preprint arXiv:2201.00382_ , 2022.
  * Zhao et al. [2021] Yue Zhao, Ryan Rossi, and Leman Akoglu.  Automatic unsupervised outlier model selection.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Chen and Liu [2022] Weizhe Chen and Lantao Liu.  Informative planning in the presence of outliers.  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2022.
  * Lang et al. [2019a] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12689–12697, 2019a.
  * Yan et al. [2018] Yan Yan, Yuxing Mao, and Bo Li.  Second: Sparsely embedded convolutional detection.  _Sensors (Basel, Switzerland)_ , 18, 2018.
  * Shi et al. [2020] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  Pv-rcnn: Point-voxel feature set abstraction for 3d object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10529–10538, 2020.
  * Shi et al. [2021] Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  Pv-rcnn++: Point-voxel feature set abstraction with local vector representation for 3d object detection.  _arXiv preprint arXiv:2102.00463_ , 2021.
  * Qi et al. [2017] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 652–660, 2017.
  * Guo et al. [2017] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In _International Conference on Machine Learning_ , pages 1321–1330. PMLR, 2017.
  * Maddox et al. [2019] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson.  A simple baseline for bayesian uncertainty in deep learning.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , pages 1050–1059. PMLR, 2016.
  * Thulasidasan et al. [2019] Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak.  On mixup training: Improved calibration and predictive uncertainty for deep neural networks.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Zadrozny and Elkan [2001] Bianca Zadrozny and Charles Elkan.  Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers.  In _Icml_ , volume 1, pages 609–616. Citeseer, 2001.
  * Platt et al. [1999] John Platt et al.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  _Advances in large margin classifiers_ , 10(3):61–74, 1999.
  * Zadrozny and Elkan [2002] Bianca Zadrozny and Charles Elkan.  Transforming classifier scores into accurate multiclass probability estimates.  In _Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 694–699, 2002.
  * Kumar et al. [2019] Ananya Kumar, Percy S Liang, and Tengyu Ma.  Verified uncertainty calibration.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Hosang et al. [2017] Jan Hosang, Rodrigo Benenson, and Bernt Schiele.  Learning non-maximum suppression.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4507–4515, 2017.
  * Bodla et al. [2017] Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis.  Soft-nms–improving object detection with one line of code.  In _Proceedings of the IEEE international conference on computer vision_ , pages 5561–5569, 2017.
  * Liu et al. [2019] Songtao Liu, Di Huang, and Yunhong Wang.  Adaptive nms: Refining pedestrian detection in a crowd.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6459–6468, 2019.
  * Rothe et al. [2014] Rasmus Rothe, Matthieu Guillaumin, and Luc Van Gool.  Non-maximum suppression for object detection by passing messages between windows.  In _Asian conference on computer vision_ , pages 290–306. Springer, 2014.
  * Kull et al. [2017] Meelis Kull, Telmo Silva Filho, and Peter Flach.  Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers.  In _Artificial Intelligence and Statistics_ , pages 623–631. PMLR, 2017.
  * Kumaraswamy [1980] Ponnambalam Kumaraswamy.  A generalized probability density function for double-bounded random processes.  _Journal of hydrology_ , 46(1-2):79–88, 1980\.
  * Dosovitskiy et al. [2017] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.  Carla: An open urban driving simulator.  In _Conference on robot learning_ , pages 1–16. PMLR, 2017.
  * Xu et al. [2021] Runsheng Xu, Yi Guo, Xu Han, Xin Xia, Hao Xiang, and Jiaqi Ma.  Opencda: an open cooperative driving automation framework integrated with co-simulation.  In _2021 IEEE International Intelligent Transportation Systems Conference (ITSC)_ , pages 1155–1162. IEEE, 2021.
  * Lang et al. [2019b] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12697–12705, 2019b.


# Non-adiabatic Strong Field Ionization of Atomic Hydrogen

D. Trabert1 N. Anders1 S. Brennecke2<EMAIL_ADDRESS>M. S.
Schöffler1 T. Jahnke1 L. Ph. H. Schmidt1 M. Kunitski1 M. Lein2 R. Dörner1 S.
Eckart1<EMAIL_ADDRESS>1 Institut für Kernphysik, Goethe-
Universität, Max-von-Laue-Str. 1, 60438 Frankfurt am Main, Germany
2 Institut für Theoretische Physik, Leibniz Universität Hannover, Appelstr. 2,
30167 Hannover, Germany

###### Abstract

We present experimental data on the non-adiabatic strong field ionization of
atomic hydrogen using elliptically polarized femtosecond laser pulses at a
central wavelength of 390 nm. Our measured results are in very good agreement
with a numerical solution of the time-dependent Schrödinger equation (TDSE).
Experiment and TDSE show four above-threshold ionization (ATI) peaks in the
electron’s energy spectrum. The most probable emission angle (also known as
’attoclock-offset angle’ or ’streaking angle’) is found to increase with
energy, a trend that is opposite to standard predictions based on Coulomb
interaction with the ion. We show that this increase of deflection-angle can
be explained by a model that includes non-adiabatic corrections of the initial
momentum distribution at the tunnel exit and non-adiabatic corrections of the
tunnel exit position itself.

Atomic hydrogen is the simplest atomic system and thus it is often used to
benchmark theoretical models making it the drosophila of theories of light-
matter interaction. Its key advantage is the absence of multi-electron effects
and the well-defined electrostatic potential of the proton, eliminating the
need for approximations. While atomic hydrogen is very frequently used in
theoretical studies and textbook examples, due to the experimental challenges
connected with producing hydrogen atoms in an ultra high vacuum environment
and separating events from those resulting from non-dissociated $H_{2}$, only
a single experiment has been reported for strong field ionization [1]. In this
pioneering work, Sainadh et al. have applied the technique of angular
streaking [2, 3] to adiabatic tunnel ionization of atomic hydrogen. In angular
streaking experiments, the most probable electron emission angle is analyzed
and the experimentally obtained results are used to benchmark theoretical
models to better understand strong field ionization [4, 5, 6, 7, 8, 9, 10].

In this letter, we report on a similar experiment to the one by Sainadh et al.
[1] but target non-adiabatic strong field ionization [11, 12, 13, 14] instead
of the much better understood adiabatic process [15]. Technically this is done
by choosing a different intensity and wavelength regime as in [1]. For
adiabatic tunnel ionization, the experimentally measured electron energy
spectrum usually shows only one broad peak. We observe four above-threshold
ionization (ATI) peaks [16, 17] in the electron’s energy spectrum that are
spaced by the photon energy, which allows for the investigation of angular
streaking for each energy peak separately [18, 19]. To this end, atomic
hydrogen is irradiated with an elliptically polarized femtosecond laser pulse
at a central wavelength of 390 nm and a peak intensity of $1.4\cdot
10^{14}$W/cm2. This corresponds to a Keldysh parameter of 3, indicating that
the temporal evolution of the tunnel barrier cannot be neglected under these
conditions and that field-driven dynamics before and during tunneling will
contribute to the ionization dynamics [11].

In our experiment, atomic hydrogen was generated using a commercially
available source (H-flux, Tectra GmbH). Hydrogen gas is thermally dissociated
in a tungsten capillary heated by electron bombardment. The hydrogen beam is
collimated to a half opening angle of about $1^{\circ}$ using an aperture and
reaches the laser focus after propagating in vacuum for $260\,$mm. Our optical
setup is based on a commercial Ti:sapphire femtosecond laser system (100 kHz,
Wyvern-500, KMLabs). We used a 200$\,\mu$m $\beta$-barium-borate (BBO) crystal
to frequency double the laser pulses. The power was adjusted by a $\lambda/2$
waveplate and a subsequent thin-film polarizer. The polarization of the laser
pulses was controlled using a $\lambda/4$ waveplate followed by a $\lambda/2$
waveplate. We use elliptically polarized light with an ellipticity of
$\epsilon=0.85$. The duration of the laser pulses at a central wavelength of
390 nm was $50\,$fs, and the pulses were focused by a spherical mirror ($f=60$
mm) onto the jet of hydrogen atoms. A COLTRIMS reaction microscope [20] was
used to measure the three-dimensional momenta of electron and ion in
coincidence. In the spectrometer, a homogeneous electric field of 27 V/cm and
a parallel homogeneous magnetic field of $14.2\,$G separated and accelerated
the charged particles towards two position- and time-sensitive detectors [21].
The length of the ion (electron) spectrometer was 17 cm (31 cm). Each detector
comprises a stack of two micro-channel plates with a diameter of 80 mm
followed by a hexagonal delay line anode. The time-of-flight (TOF) and
position-of-impact information were used to calculate the momenta of electron
and ion, which were measured in coincidence. The absolute orientation of the
polarization ellipse in the experiment has been determined by the most
probable emission angle of $H^{+}$ ions resulting from the reaction
$H_{2}\rightarrow H+H^{+}+e^{-}$ undergone by $H_{2}$ molecules [22] that were
not dissociated in the atomic hydrogen source. The $H^{+}$ ions from
ionization of atomic hydrogen and those from ionization of $H_{2}$ with
subsequent dissociation are unambiguously distinguishable, as the latter carry
a momentum of about $9$ a.u. The average intensity in the focus was $9\cdot
10^{13}$ W/cm2 (taking volume averaging into account as in Ref. [23]). This
corresponds to a peak intensity of $1.4\cdot 10^{14}$ W/cm2 (peak electric
field of $E_{\mathrm{peak}}=0.046\,$a.u. at an ellipticity of
$\epsilon=0.85$). The calibration of the laser intensity was done by comparing
the measured electron energy distribution to numerical solutions of the TDSE.
The numerical solutions of the TDSE are found in three dimensions in the
length gauge for a laser pulse with a sin2-envelope and a duration of 20
optical cycles. The numerical propagation uses the pseudospectral method
described in Refs. [24, 25, 26]. In order to obtain photoelectron momentum
distributions, we project the final electronic wave function on the exact
scattering states of the Coulomb potential. Afterwards, the distributions are
averaged over four carrier-envelope phase values from 0 to 2$\pi$ and over the
focal volume intensity distribution, assuming a Gaussian beam profile in the
focus. We have confirmed that the observables presented in this paper do not
significantly depend on the pulse length that is used in the TDSE simulation
by varying the pulse length in the range between 10 and 30 optical cycles.

Fig. 1(a) shows the electron momentum distribution projected onto the laser’s
polarization plane ($p_{x}p_{y}$-plane). Four ATI rings are visible that are
spaced by the photon energy of 3.18 eV. The orientation of the major axis of
the polarization ellipse is aligned along the $p_{x}$-direction. Fig. 1(b)
shows the corresponding result that is obtained from a numerical solution of
the time-dependent Schrödinger equation (TDSE). Experiment and TDSE result
show excellent agreement. This is underlined by Fig. 1(c) which shows the
corresponding electron energy distributions that are also in good agreement.

Figure 1:  (a) Measured electron momentum distribution projected onto the
laser’s polarization plane for the ionization of atomic hydrogen by
femtosecond laser pulses at a central wavelength of 390 nm, an ellipticity of
$\epsilon=0.85$ and a peak intensity of $1.4\cdot 10^{14}$ W/cm2. The light’s
helicity and the orientation of the polarization ellipse are indicated by the
inset in the lower right corner. (b) shows a focal-averaged numerical solution
of the time-dependent Schrödinger equation (TDSE) for the parameters that were
used in (a). $\alpha_{\mathrm{off}}$ indicates the ’angular offset’ with
respect to the minor axis of the ellipse of the laser electric field and the
negative vector potential in (a) and (b). The intensity in (a) [(b)] has been
normalized to a maximum of 1 [0.8]. (c) shows a comparison of the electron
energy distribution for the data shown in (a) and (b). The error bars show
statistical errors only.

In order to compare the angular offset angles $\alpha_{\mathrm{off}}$, the
momentum distributions from Fig. 1(a) and (b) are shown in Fig. 2(a) and (b)
in cylindrical coordinates. $\alpha_{\mathrm{off}}$ is the angle between the
observed angle of maximum electron yield and the minor axis of the
polarization ellipse (see Fig. 1). Row-wise normalization is performed to
improve the visibility of the angular distributions for less probable $p_{r}$.
Besides the good agreement of experiment and TDSE, it is evident that
$\alpha_{\mathrm{off}}$ increases as a function of the radial momentum in the
laser’s polarization plane $p_{r}=\sqrt{p_{x}^{2}+p_{y}^{2}}$. What is the
microscopic reason for this dependence of $\alpha_{\mathrm{off}}$ on $p_{r}$?

Figure 2:  (a) [(b)] shows the electron momentum distribution from Fig. 1(a)
[1(b)] in cylindrical coordinates ($p_{r}=\sqrt{p_{x}^{2}+p_{y}^{2}}$ and
$\phi$ is the angle of $\vec{p}_{elec}$ in the laser polarization plane) after
row-wise normalization. $\alpha_{\mathrm{off}}$ indicates the ’angular
offset’, the white contours indicate the ATI peak positions. (c) shows the
row-wise normalized electron momentum distribution that is obtained from the
classical two-step (CTS) model with initial momenta distributed according to
ADK theory, taking Coulomb interaction after tunneling into account and using
the tunnel exit position from the TIPIS model in analogy to (a) and (b). (d)
shows the result from our NACTS model. Within the NACTS model, the ionization
probability and the initial conditions are taken from SFA. The black line
guides the eye and is the same in all panels. The data in (a)-(d) has been
symmetrized making use of the two-fold symmetry. Every row has been normalized
independently. The intensity in (a) [(b)-(d)] has been normalized to a maximum
of 1 [0.8].

A common way to model strong field ionization in a time-dependent electric
field $\vec{E}(t)$ is to split the ionization dynamics into two steps [27].
First, at a given instant $t_{0}$ the electron tunnels through the time-
dependent potential barrier that is formed by the ionic potential and the
laser electric field. The electron is released with an initial momentum
$\vec{p}_{init}$ at a position $\vec{r}_{0}$ that is anti-parallel to the
electric field vector $\vec{E}(t_{0})$ at the time $t_{0}$. It is typically
assumed that $\vec{p}_{init}$ is perpendicular to $\vec{E}(t_{0})$ [18, 28].
In the second step, the classical forces, that act upon the electron and which
are due to the laser field and Coulomb interaction with the ion, determine the
electron’s dynamics. For elliptically polarized light, the probability for
tunneling maximizes at the two instants per laser cycle when the electric
field vector points along the major axis of the polarization ellipse. Those
instants in time are $t_{1}$ and $t_{2}=t_{1}+T/2$, where $T$ is the duration
of one cycle of the light field. $\vec{p}_{init}$ is usually small and is
typically modeled by a nearly Gaussian distribution [29]. Therefore, if
Coulomb interaction after tunneling is neglected, this two-step model yields
maximum probability for the occurrence of momenta of
$\vec{p}_{elec}=-\vec{A}(t_{1})$ and $\vec{p}_{elec}=-\vec{A}(t_{2})$ giving
rise to a two-fold symmetry of the electron momentum distribution in the plane
of polarization. The orientation of this two-fold symmetric distribution in
the plane of polarization is the basis of ’attoclock-experiments’ [3, 4, 5, 6,
7, 8, 9]. The deviation of the orientation of the two-fold symmetric
distribution with respect to the direction that is defined by
$-\vec{A}(t_{1})$ and $-\vec{A}(t_{2})$ is referred to as ’angular offset’
[30]. A state-of-the-art classical model is the semiclassical two-step (SCTS)
model that includes Coulomb interaction after tunneling and even allows for
the modeling of interference [27, 31, 28, 32].

In the remainder of this letter, we build on the classical two-step (CTS)
model [14], which is equivalent to the SCTS model [28] but neglects
interference. As a reference we first perform a CTS simulation in which the
tunneling probability is given by the Ammosov–Delone–Krainov (ADK) theory [15,
28]. This is the CTS model, which is often used to model attoclock experiments
[33, 34, 9]. It takes Coulomb interaction after tunneling into account and
uses the tunnel exit position from the TIPIS model (tunnel ionization in
parabolic coordinates with induced dipole and Stark shift). In the following,
we neglect the polarizability of the hydrogen atom. Inclusion of the
polarizability would lead to a Stark shift of the ionization potential by
about 0.13 eV. Within the TIPIS model, the tunnel exit position is given by:

$r_{\mathrm{TIPIS}}(\vec{E}(t))=\frac{I_{p}+\sqrt{I_{p}^{2}-(4-\sqrt{8I_{p}})|\vec{E}(t)|}}{2|\vec{E}(t)|}$
(1)

Here, $I_{p}=0.5$ a.u. is the ionization potential of atomic hydrogen. The
tunnel exit position $r_{\mathrm{TIPIS}}$ depends solely on the magnitude of
the instantaneous laser electric field $|\vec{E}(t)|$ which makes TIPIS an
adiabatic model. The result from this CTS simulation is shown in Fig. 2(c). As
expected, the Coulomb potential leads to a non-zero offset angle, but the
effect decreases with increasing $p_{r}$; a trend just opposite to the
findings obtained from experiment and TDSE.

In a next step, we employ a hybrid model that combines the strength of strong-
field approximation (SFA), which is the inclusion of non-adiabatic dynamics in
the classically forbidden region (also referred to as tunnel), and the
strength of the CTS model, which is the inclusion of Coulomb interaction after
the electron is released from the classically forbidden region. We refer to
this improved CTS model as non-adiabatic, classical two-step model (NACTS).
The NACTS model and the CTS model are both based on classical trajectories. In
contrast to the CTS model, the NACTS model uses the initial probability
distribution from SFA in order to launch a swarm of classical trajectories.
This implies that the momenta along the direction of the electric field at the
instant of ionization can be nonzero, which is an important difference with
respect to ADK theory. The same holds true for the initial position of the
classical trajectories within the NACTS model, which are not simply calculated
using the TIPIS model but they are also obtained from an SFA simulation [35,
36]. In summary, our NACTS simulations are equivalent to the scheme presented
in Ref. [37], except that interference of trajectories is neglected. The NACTS
model results in the distribution that is shown in Fig. 2(d). Its exact shape
is affected by the correlated initial momentum and position distribution (see
supplemental material for details). The result from the NACTS model shows the
same trend as observed in the experiment and the TDSE simulation. Comparing
the two different classical models (Figs. 2(c) and 2(d)) it is found that the
result from the NACTS model shows superior agreement with the experiment and
with the result from the TDSE simulation. It should be noted, that the
conclusions that are drawn from the CTS and the NACTS model are limited by the
validity of the models (see supplemental material for details). Within the CTS
model there is an adiabatic tunneling step that is modeled by ADK theory. This
is in contrast to SFA, which is the basis for the NACTS model. SFA also works
for $\gamma>1$ and makes no assumption on the existence of tunneling in the
sense of a slowly-varying potential barrier.

Figure 3:  (a) The angular offset $\alpha_{\mathrm{off}}$ is plotted as a
function of the radial momentum $p_{r}$ for the data from Fig. 2(a)-(d). Error
bars show statistical errors only. The gray dashed line guides the eye and
shows the emission angle of electrons that tunnel at the peak electric field
without taking Coulomb interaction into account. The magnitude of the negative
vector potential at the peak of the laser pulse is 0.33 a.u. (b) illustrates
that the tunnel exit position within the TIPIS model does not depend on the
initial momentum at the tunnel exit $p_{\perp}$. SFA predicts decreasing
values for the initial position $\vec{r}_{\mathrm{SFA}}$ as a function of
$p_{\perp}$ for $0$ a.u.$<p_{\perp}<0.7$ a.u. The most probable value of
$p_{\perp}$ from SFA is 0.15 a.u. (c) Visualization of a potential microscopic
explanation for the dependence of $\vec{r}_{\mathrm{SFA}}$ on $p_{\perp}$ (see
text).

To quantitatively compare the results shown in Fig. 2, we extract the angular
offset for each ATI peak in Fig. 2(a) and 2(b). To this end, we integrate the
signal corresponding to each ATI peak separately and thereby obtain an angular
distribution for each ATI peak. Then, we determine the offset angle
$\alpha_{\mathrm{off}}$ from the second-order Fourier coefficients. For the
distributions in Fig. 2(c) and 2(d), $\alpha_{\mathrm{off}}$ is determined for
each row separately. The results are shown in Fig. 3(a) as a function of
$p_{r}$. This quantitative comparison underlines the excellent agreement of
the TDSE simulation with the experiment. Further, it is evident that the NACTS
model qualitatively agrees with the experiment and the result from the TDSE
simulation. To summarize the importance of the dependence of the initial
position on the initial momentum, Fig. 3(b) compares
$\vec{r}_{\mathrm{TIPIS}}$ and $\vec{r}_{\mathrm{SFA}}$ for tunneling at the
peak electric field of an elliptically polarized light field at 390 nm with a
peak electric field of 0.046 a.u. and an ellipticity of 0.85.

The physics behind the dependence of the initial position on the initial
momentum is intriguingly simple: For non-adiabatic tunneling, the energy at
the tunnel exit can be higher than the ground state energy because energy can
be absorbed from the light field in the classically forbidden region [11].
Klaiber et al. have shown that non-adiabatic tunneling can be modeled as non-
resonant multi-photon excitation and subsequent adiabatic tunneling [38, 13].
Within this picture, the absorption of energy from the light field leads to an
inward shift of the tunnel exit position as illustrated in Fig. 3(c) since it
effectively reduces $I_{p}$ in Eq. (1) by the energy that is absorbed from the
light field. For almost circularly polarized light, the absorption of energy
from the light field also induces a well-defined change in angular momentum.
This results in an increased initial momentum at the tunnel exit [14] and
explains the overall decrease of $\vec{r}_{\mathrm{SFA}}$ as a function of
$p_{\perp}$. Interestingly, for very high $p_{\perp}$ the energy
$\frac{1}{2}p_{\perp}^{2}$ is not negligible compared to $I_{p}$, which
manifests as an effectively increased $I_{p}$, which explains the increasing
values of $\vec{r}_{\mathrm{SFA}}$ for $p_{\perp}>0.7$ a.u.

Fig. 3(c) visualizes that SFA does not include any long-range potential of the
ion. Neglecting Coulomb effects on the under-the-barrier dynamics is a
possible reason why the result from our NACTS model (see Fig. 2(d)) does not
show complete agreement with the experiment. Further, we observe that the TDSE
simulation shows a dependence of $\alpha_{\mathrm{off}}$ on $p_{r}$ even
within single ATI peaks as can be seen in Fig. 2(b). Interestingly such a
feature is also seen in the experiment at $p_{r}\approx 0.4$ a.u. (see Fig.
2(a)). Currently we cannot explain this observation, which warrants further
research. We speculate that this might be related to focal averaging, as we
observe an offset angle for the lowest energy peak of only $50^{\circ}$ in the
TDSE simulation without volume averaging for an intensity of $9\cdot 10^{13}$
W/cm2.

In conclusion, we have presented a benchmark experiment on the non-adiabatic
strong field ionization of atomic hydrogen. The result is in excellent
agreement with our ab initio TDSE simulation. The simplicity of the structure
of atomic hydrogen allows for the exclusion of multi-electron effects as well
as initial states with atomic orbitals carrying angular momentum [39]. We find
that the CTS model, which is typically used to interpret attoclock
experiments, is not able to describe the angular offset
$\alpha_{\mathrm{off}}$ as a function of the electron momentum in the plane of
polarization $p_{r}$. We find that our NACTS model, a semiclassical model with
non-adiabatic initial conditions and ionization probabilities from SFA,
reproduces the experimental findings qualitatively. Within the NACTS model,
the initial momenta and the initial positions of the wave packet at the tunnel
exit are correlated, leading to a clearly improved agreement with the
experiment. TDSE calculations over a wide range of Keldysh parameters (see
supplemental material) suggest that the non-adiabatic effects discussed in
this paper significantly alter the offset angle for $\gamma>0.5$. The
inclusion of non-adiabatic effects, as in our NACTS model, can extend the
validity range of two-step models to $\gamma\approx 3$, which is considered
outside the tunneling regime in the usual strong-field terminology. This
allows for a new class of attoclock experiments that are resolved with respect
to the electron’s radial momentum. We expect that the NACTS model will help to
interpret future attoclock experiments and will thereby lead to a better
understanding of the role of electron positions and momenta in the ionization
process. The NACTS model is an important step towards probing the wave
packet’s tunnel exit position with sub-Ångstrom precision [40] as proposed by
Kheifets [41] who has referred to this approach as ’nano-ruler’. We expect
that our findings will serve as a benchmark for the development of models for
other atoms and small (chiral) molecules.

##  I Acknowledgments

The experimental work was supported by the DFG (German Research Foundation).
S.B. acknowledges funding of the German Academic Exchange Service. M.L. and
S.E. acknowledge funding of the DFG through Priority Programme SPP 1840 QUTIF.

## References

  * Sainadh  _et al._ [2019] U. S. Sainadh, H. Xu, X. Wang, A. Atia-Tul-Noor, W. C. Wallace, N. Douguet, A. Bray, I. Ivanov, K. Bartschat, A. Kheifets, R. T. Sang, and I. V. Litvinyuk, Attosecond angular streaking and tunnelling time in atomic hydrogen, Nature 568, 75 (2019).
  * Smolarski  _et al._ [2010] M. Smolarski, P. Eckle, U. Keller, and R. Dörner, Semiclassical model for attosecond angular streaking, Opt. Express 18, 17640 (2010).
  * Eckle  _et al._ [2008a] P. Eckle, A. N. Pfeiffer, C. Cirelli, A. Staudte, R. Dörner, H. G. Muller, M. Büttiker, and U. Keller, Attosecond ionization and tunneling delay time measurements in helium, Science 322, 1525 (2008a).
  * Torlina  _et al._ [2015] L. Torlina, F. Morales, J. Kaushal, I. Ivanov, A. Kheifets, A. Zielinski, A. Scrinzi, H. G. Muller, S. Sukiasyan, M. Ivanov, and O. Smirnova, Interpreting attoclock measurements of tunnelling times, Nat. Phys. 11, 503 (2015).
  * Shafir  _et al._ [2012] D. Shafir, H. Soifer, B. D. Bruner, M. Dagan, Y. Mairesse, S. Patchkovskii, M. Ivanov, O. Smirnova, and N. Dudovich, Resolving the time when an electron exits a tunnelling barrier, Nature 485, 343 (2012).
  * Bray  _et al._ [2018] A. W. Bray, S. Eckart, and A. S. Kheifets, Keldysh-Rutherford model for the attoclock, Phys. Rev. Lett. 121, 123201 (2018).
  * Ni  _et al._ [2016] H. Ni, U. Saalmann, and J.-M. Rost, Tunneling ionization time resolved by backpropagation, Phys. Rev. Lett. 117, 023002 (2016).
  * Camus  _et al._ [2017] N. Camus, E. Yakaboylu, L. Fechner, M. Klaiber, M. Laux, Y. Mi, K. Z. Hatsagortsyan, T. Pfeifer, C. H. Keitel, and R. Moshammer, Experimental evidence for quantum tunneling time, Phys. Rev. Lett. 119, 023201 (2017).
  * Ni  _et al._ [2018] H. Ni, U. Saalmann, and J.-M. Rost, Tunneling exit characteristics from classical backpropagation of an ionized electron wave packet, Phys. Rev. A 97, 013426 (2018).
  * Hofmann  _et al._ [2019] C. Hofmann, A. S. Landsman, and U. Keller, Attoclock revisited on electron tunnelling time, Journal of Modern Optics 66, 1052 (2019).
  * M. Yu. Ivanov  _et al._ [2005] M. Yu. Ivanov, M. Spanner, and O. Smirnova, Anatomy of strong field ionization, J. Mod. Opt. 52, 165 (2005).
  * Barth and Smirnova [2011] I. Barth and O. Smirnova, Nonadiabatic tunneling in circularly polarized laser fields: Physical picture and calculations, Phys. Rev. A 84, 063415 (2011).
  * Klaiber and Briggs [2016] M. Klaiber and J. S. Briggs, Crossover from tunneling to multiphoton ionization of atoms, Phys. Rev. A 94, 053405 (2016).
  * Eckart  _et al._ [2018a] S. Eckart, K. Fehre, N. Eicke, A. Hartung, J. Rist, D. Trabert, N. Strenger, A. Pier, L. Ph. H. Schmidt, T. Jahnke, M. S. Schöffler, M. Lein, M. Kunitski, and R. Dörner, Direct experimental access to the nonadiabatic initial momentum offset upon tunnel ionization, Phys. Rev. Lett. 121, 163202 (2018a).
  * Ammosov  _et al._ [1986] M. V. Ammosov, N. B. Delone, and V. P. Krainov, Tunnel ionization of complex atoms and of atomic ions in an alternating electromagnetic field, Sov. Phys. JETP 64, 1191 (1986).
  * Voronov and Delone [1966] G. S. Voronov and N. B. Delone, Many-photon ionization of the xenon atom by ruby laser radiation, Sov. Phys. JETP 23, 54 (1966).
  * Agostini  _et al._ [1979] P. Agostini, F. Fabre, G. Mainfray, G. Petite, and N. K. Rahman, Free-free transitions following six-photon ionization of xenon atoms, Phys. Rev. Lett. 42, 1127 (1979).
  * Goreslavski  _et al._ [2004] S. P. Goreslavski, G. G. Paulus, S. V. Popruzhenko, and N. I. Shvetsov-Shilovski, Coulomb asymmetry in above-threshold ionization, Phys. Rev. Lett. 93, 233002 (2004).
  * Wang  _et al._ [2021] C. Wang, X. Li, X. Liu, J. Li, S. Zhou, Y. Yang, X. Song, J. Chen, W. Yang, and D. Ding, Decoding electron tunnelling delay time by embracing wave-particle duality, arXiv preprint, arXiv:2104.06144 (2021).
  * Ullrich  _et al._ [2003] J. Ullrich, R. Moshammer, A. Dorn, R. Dörner, L. Ph. H. Schmidt, and H. Schmidt-Böcking, Recoil-ion and electron momentum spectroscopy: reaction-microscopes, Rep. Prog. Phys. 66, 1463 (2003).
  * Jagutzki  _et al._ [2002] O. Jagutzki, A. Cerezo, A. Czasch, R. Dörner, M. Hattas, M. Huang, V. Mergel, U. Spillmann, K. Ullmann-Pfleger, T. Weber, H. Schmidt-Böcking, and G. D. W. Smith, Multiple hit readout of a microchannel plate detector with a three-layer delay-line anode, IEEE Trans. Nucl. Sci. 49, 2477 (2002).
  * Staudte  _et al._ [2009] A. Staudte, S. Patchkovskii, D. Pavičić, H. Akagi, O. Smirnova, D. Zeidler, M. Meckel, D. M. Villeneuve, R. Dörner, M. Y. Ivanov, and P. B. Corkum, Angular tunneling ionization probability of fixed-in-space H2 molecules in intense laser pulses, Phys. Rev. Lett. 102, 033004 (2009).
  * Wang  _et al._ [2005] P. Wang, A. M. Sayler, K. D. Carnes, B. D. Esry, and I. Ben-Itzhak, Disentangling the volume effect through intensity-difference spectra: application to laser-induced dissociation of H${}_{2}^{+}$, Opt. Lett. 30, 664 (2005).
  * Tong and Chu [1997] X.-M. Tong and S.-I. Chu, Theoretical study of multiple high-order harmonic generation by intense ultrashort pulsed laser fields: A new generalized pseudospectral time-dependent method, Chemical Physics 217, 119 (1997), dynamics of Driven Quantum Systems.
  * Murakami  _et al._ [2013] M. Murakami, O. Korobkin, and M. Horbatsch, High-harmonic generation from hydrogen atoms driven by two-color mutually orthogonal laser fields, Phys. Rev. A 88, 063419 (2013).
  * Baykusheva  _et al._ [2017] D. Baykusheva, S. Brennecke, M. Lein, and H. J. Wörner, Signatures of electronic structure in bicircular high-harmonic spectroscopy, Phys. Rev. Lett. 119, 203201 (2017).
  * Gallagher [1988] T. F. Gallagher, Above-threshold ionization in low-frequency limit, Phys. Rev. Lett. 61, 2304 (1988).
  * Shvetsov-Shilovski  _et al._ [2016] N. I. Shvetsov-Shilovski, M. Lein, L. B. Madsen, E. Räsänen, C. Lemell, J. Burgdörfer, D. G. Arbó, and K. Tőkési, Semiclassical two-step model for strong-field ionization, Phys. Rev. A 94, 013415 (2016).
  * Arissian  _et al._ [2010] L. Arissian, C. Smeenk, F. Turner, C. Trallero, A. V. Sokolov, D. M. Villeneuve, A. Staudte, and P. B. Corkum, Direct test of laser tunneling with electron momentum imaging, Phys. Rev. Lett. 105, 133002 (2010).
  * Eckle  _et al._ [2008b] P. Eckle, M. Smolarski, P. Schlup, J. Biegert, A. Staudte, M. Schöffler, H. G. Muller, R. Dörner, and U. Keller, Attosecond angular streaking, Nature Physics 4, 565 (2008b).
  * Arbó  _et al._ [2010] D. G. Arbó, K. L. Ishikawa, K. Schiessl, E. Persson, and J. Burgdörfer, Intracycle and intercycle interferences in above-threshold ionization: The time grating, Phys. Rev. A 81, 021403(R) (2010).
  * Eckart  _et al._ [2018b] S. Eckart, M. Kunitski, I. Ivanov, M. Richter, K. Fehre, A. Hartung, J. Rist, K. Henrichs, D. Trabert, N. Schlott, L. Ph. H. Schmidt, T. Jahnke, M. S. Schöffler, A. Kheifets, and R. Dörner, Subcycle interference upon tunnel ionization by counter-rotating two-color fields, Phys. Rev. A 97, 041402(R) (2018b).
  * Bisgaard and Madsen [2004] C. Z. Bisgaard and L. B. Madsen, Tunneling ionization of atoms, American Journal of Physics 72, 249 (2004).
  * Pfeiffer  _et al._ [2012] A. N. Pfeiffer, C. Cirelli, M. Smolarski, D. Dimitrovski, M. Abu-Samha, L. B. Madsen, and U. Keller, Attoclock reveals natural coordinates of the laser-induced tunnelling current flow in atoms, Nat. Phys. 8, 76 (2012).
  * Popruzhenko and Bauer [2008] S. Popruzhenko and D. Bauer, Strong field approximation for systems with Coulomb interaction, J. Mod. Opt. 55, 2573 (2008).
  * Popruzhenko  _et al._ [2008] S. V. Popruzhenko, G. G. Paulus, and D. Bauer, Coulomb-corrected quantum trajectories in strong-field ionization, Phys. Rev. A 77, 053409 (2008).
  * Brennecke  _et al._ [2020] S. Brennecke, N. Eicke, and M. Lein, Gouy’s phase anomaly in electron waves produced by strong-field ionization, Phys. Rev. Lett. 124, 153202 (2020).
  * Klaiber  _et al._ [2015] M. Klaiber, K. Z. Hatsagortsyan, and C. H. Keitel, Tunneling dynamics in multiphoton ionization and attoclock calibration, Phys. Rev. Lett. 114, 083001 (2015).
  * Eckart  _et al._ [2018c] S. Eckart, M. Kunitski, M. Richter, A. Hartung, J. Rist, F. Trinter, K. Fehre, N. Schlott, K. Henrichs, L. Ph. H. Schmidt, T. Jahnke, M. Schöffler, K. Liu, I. Barth, J. Kaushal, F. Morales, M. Ivanov, O. Smirnova, and R. Dörner, Ultrafast preparation and detection of ring currents in single atoms, Nat. Phys. 14, 701 (2018c).
  * Brennecke  _et al._ [2021] S. Brennecke, S. Eckart, and M. Lein, Attoclock with bicircular laser fields as a probe of velocity-dependent tunnel-exit positions, submitted (2021).
  * Kheifets [2020] A. S. Kheifets, The attoclock and the tunneling time debate, Journal of Physics B: Atomic, Molecular and Optical Physics 53, 072001 (2020).