# PMI Sampler: Patch Similarity Guided Frame Selection For Aerial Action
Recognition

Ruiqi Xian, Xijun Wang, Divya Kothandaraman, Dinesh Manocha
University of Maryland - College Park
{rxian, xijun, dkr<EMAIL_ADDRESS>
###### Abstract

We present a new algorithm for the selection of informative frames in video
action recognition. Our approach is designed for aerial videos captured using
a moving camera where human actors occupy a small spatial resolution of video
frames. Our algorithm utilizes the motion bias within aerial videos, which
enables the selection of motion-salient frames. We introduce the concept of
patch mutual information (PMI) score to quantify the motion bias between
adjacent frames, by measuring the similarity of patches. We use this score to
assess the amount of discriminative motion information contained in one frame
relative to another. We present an adaptive frame selection strategy using
shifted leaky ReLu and cumulative distribution function, which ensures that
the sampled frames comprehensively cover all the essential segments with high
motion salience. Our approach can be integrated with any action recognition
model to enhance its accuracy. In practice, our method achieves a relative
improvement of 2.2 - 13.8% in top-1 accuracy on UAV-Human, 6.8% on NEC Drone,
and 9.0% on Diving48 datasets. The code is available at
https://github.com/Ricky-Xian/PMI-Sampler.

Figure 1: Sample eight frames from typical videos in Diving48 and UAV-Human.
Compared with the state-of-the-art, MG Sampler [49], our approach can better
represent the motion distribution and provides an easier way to distinguish
motion salient frames in videos. PMI Sampler is more robust to noises and
outliers and can better handle the background changes caused by the moving
camera.

##  1 Introduction

The applications of unmanned aerial vehicles (UAVs) have been expanding
rapidly into search and rescue, agriculture, security, surveillance, etc. This
is giving rise to many challenging problems including detection, re-
identification, tracking, and recognition. Among these tasks, aerial video
action recognition is regarded as one of the most difficult problems. Despite
the success of deep learning-based methods in video action recognition on
ground camera footage [10, 20, 31], current methods do not result in high
accuracy on aerial videos.

Aerial video data are often captured by a camera mounted on a moving UAV in
oblique or overhead angles. The resulting footage features human actors that
appear significantly smaller (typically less than 10% pixels), due to the high
camera altitude, with a large portion of the video frame dominated by the
background information. The size and scale of the human actor may vary
considerably, owing to changes in flying altitude during video data
collection. Furthermore, the continuous movement of the UAV can cause the
camera viewing angle shifting, resulting in motion blur and occlusion. These
factors collectively make the development of accurate algorithms for aerial
action recognition a challenging task.

The decrease in performance of generic video recognition models on aerial data
can be reduced through the use of motion-guided frame sampling during training
[23, 47]. However, current deep learning-based action recognition methods
mostly use fixed hand-crafted sampling techniques for video analysis [39, 42,
44]. Typically, frames are randomly sampled in a uniform manner or
successively with a fixed stride from the original video. This fixed sampling
strategy can be sub-optimal for several reasons. First, the motion duration
varies for different videos and actions, and fixed sampling may not capture
the entire motion duration, potentially overlooking some useful information.
Second, the sampling approach should prioritize discriminative frames over
redundant or uninformative background frames, as not all frames are equally
useful in terms of recognition.

Recently, some techniques [46, 21, 15, 27, 13] have been proposed for frame
selection by modeling it as a decision making task. Typically, these methods
employ a learning-based module to sequentially select more informative frames
or to conditionally exit early. While these methods have shown promising
results, their performance heavily relies on the training data and may not
easily transfer to unseen domains. Unfortunately, the scarcity of labeled
aerial videos, coupled with the challenges of collection and annotation, makes
the task of training such modules more difficult. It turns out that the number
and size of UAV video datasets is far fewer and smaller than those available
for ground video datasets. For instance, the Kinetics dataset contains 650k
videos, while the UAV-Human dataset has only 20k videos. Additionally, these
learning-based methods are primarily designed for untrimmed videos, and
adapting them to trimmed videos poses additional challenges. Another
alternative [50] is to use a statistical model to represent the motion bias
between frames, and devise an adaptive sampling strategy for frame selection
based on motion information distribution along the temporal domain. However,
this formulation fails to account for the distinct features of aerial video
data, such as small resolution, multi-scale, and moving camera.

#### Main Contribution:

We present a novel frame selection scheme for aerial action recognition. Our
approach is general and is designed to address some of the challenges in
aerial data. We utilize the similarity between patches to assess the amount of
discriminative motion information contained in the aerial videos, and ensure
that more informative frames can be selected for video representation. Our
method can be combined with any recognition model to obtain improved accuracy
in terms of aerial action recognition. The novel components of our work
includes:

  1. 1.

We introduce patch mutual information (PMI) score to leverage the motion bias
in the aerial videos. PMI score quantifies how much discriminative motion
information is contained in one frame given another by measuring the
similarity of frame patches via mutual information calculation.

  2. 2.

We propose an adaptive frame selection strategy based on shifted Leaky ReLu
and cumulative distribution function. Our formulation enhances the motion bias
between adjacent frames, making it easier to distinguish the motion-salient
frames in videos. Furthermore, our method is designed in a direct plug-in
manner to avoid complex training.

We evaluate the effectiveness of our method in three aerial datasets and
experimental results show that our approach consistently outperforms current
state-of-the-art by large margins in terms of top-1 accuracy. Practically, we
demonstrate a relative improvement of 2.2 - 13.8% on UAV-Human [25], 6.8% on
NEC Drone[4], and 9.0% on Diving48 datasets[26].

Figure 2: Patch Mutual Information formulation: Given a pair of adjacent
frames $I_{t-1}$ and $I_{t}$ in size $H\times W\times C$, we first divide them
into patches with size $r\times r\times C$ and map each patch to a
$d$-dimensional vector, where $d=r^{2}C$. We concatenate the two
$d$-dimensional vectors representing the corresponding patches from two
frames, the resulted vector is in $2d$ space. We perform the same operations
for each patches and further concatenate these $2d$ vectors to get the
multivariate distribution of size $2d\times\frac{HW}{r^{2}}$ for this image
pair. Then, we calculate the covariance matrix of the multivariate
distribution, compute the approximated entropies using Eq 5 and eventually get
the patch mutual information following the Eq 3

.

##  2 Related Work

###  2.1 Action Recognition for Aerial Videos

The accuracy of action recognition on ground-camera video datasets has
increased due to recent advancements in deep learning techniques. However, the
current method cannot demonstrate a similar level of accuracy on videos
captured using UAV cameras [33]. For aerial videos,
[14],[30],[29],[32],[1],[12],[32] utilize 2D convolutional neural networks
(such as ResNet and MobileNet) as the foundational models for single-frame
classification, and subsequently merge the outcomes of all video frames. Other
methods [2],[35],[36] utilize two-stream convolutional neural networks (CNNs)
to leverage both human motion and appearance attributes for improved action
recognition. [4],[6],[25],[32],[40] employ the I3D network [3] to capture the
spatial-temporal features of both human agents and their surroundings. [45]
proposes a framework leveraging CNNs and attention mechanisms for aerial
action recognition on both edge devices and decent GPUs. [22, 24] introduced
an attention mechanism based on the Fourier transform to emphasize motion
salience. Our proposed PMI Sampler is complimentary and could incorporate with
aforementioned method to improve the overall recognition accuracy.

###  2.2 Frame Sampling

For some deep learning-based methods [3, 9, 41], the frame sequence used for
training is obtained by randomly picking a fixed number of consecutive frames
in the video. Other methods [10, 44] use a uniform sampling strategy, where
frames are evenly sampled along the video’s temporal domain. These two
sampling techniques are commonly used for action recognition models. However,
they do not exploit the motion bias between frames and do not consider the
video characteristics corresponding to different human actions. There are also
some learning-based frame selection methods [46, 21, 13, 48, 8]. FastForward
[8] employs reinforcement learning for planning frame skipping and early stop
decisions. Adaframe [46] utilizes a policy gradient-trained LSTM, enhanced
with a global memory for frame selection. SCSampler [21] and Listen to Look
[13] utilize audio as an additional modality to exploit the natural semantic
correlation between audios and frames. However, those methods mainly focus on
untrimmed videos. They require large amounts of training samples and involve
complex training procedures.

MGSampler [50] leverages the temporal variations and use RGB difference
between two adjacent frames to estimate the motion salience for each frame.
However, such an approach may not be robust. Considering the challenges of
aerial data (e.g. small resolution for human actors), their formulation may
not be accurate due to the large number of outliers and noises belonging to
the background information. In order to address these issues, we present a
motion information representation technique using patch mutual information
between frames and introduce a new sampling strategy based on the PMI score
and shifted leaky ReLu.

###  2.3 Similarity Measure between Images

Numerous similarity metrics have been suggested for image analysis, with
Euclidean distance being a common choice [49]. Nevertheless, for UAV videos,
Euclidean distance struggles due to pronounced background noise and frame
instability. Cosine similarity is used for high-dimensional data but overlooks
pixel value magnitude [16]. Mutual information is a well-adopted similarity
measure [43, 28], especially in medical registration [37, 19]. It boasts
resistance to outliers, yielding smooth cost functions for optimization [5].
However, it lacks geometry consideration, solely focusing on pixel values, and
neglecting spatial pixel relationships. NMI, which is a variation of Mutual
Information, overlooks the spatial correlation between pixels and incurs
higher computational costs. PSNR, derived from Mean Square Error (MSE),
primarily focuses on pixel-level comparisons, is sensitive to dominant
background changes encountered in aerial videos. Similarly, SSIM, a widely
employed similarity measure, assesses the luminance, contrast, and structure
of images but proves to be highly sensitive to structural variations such as
rotations and shifts, which are frequently observed in aerial videos.
Russakoff et al. [38] propose RMI, integrating spatial data with mutual
information, but it’s computationally expensive. Inspired by these, we
introduce patch mutual information, an RMI extension, simpler to implement yet
provides enhanced accuracy.

Figure 3:  PMI Sampler: Given a sequence of video frames, we compute the patch
mutual information score for each pair of adjacent frames. We further remap
the PMI scores using shifted Leaky ReLu to enhance the motion distribution.
Finally, we accumulated the PMI scores using cumulative distribution function
and segment the video into $N$ parts where $N$ is the number of samplings
required for training. We randomly select one frame from each segments,
constitute them into a sequence and feed them to the recognition backbones for
action classification.

##  3 Our Approach: PMI Sampler

In this section, we present the details of our proposed patch similarity
guided frame selection strategy. Specifically, we first introduce the concept
of mutual information in Section 3.1. Next, we present the key component of
our approach, including patch mutual information (PMI) score in Section 3.2.
Finally, we present the overall pipeline of our proposed PMI Sampler based on
shifted leaky ReLu and cumulative distribution function in Section 3.3.

###  3.1 Mutual Information

Mutual information (MI) is a concept in information theory that essentially
measures the amount of information given by one variable when observing
another variable. Mutual information is highly correlated with entropy and
joint entropy. The entropy is a measure of the uncertainty of a random
variable and the joint entropy examines the overall complexity of all possible
outcomes given both random variables. Specifically, given two discrete random
variables $X$ and $Y$ with alphabet $\mathcal{X}$ and $\mathcal{Y}$. Their
probability mass functions (PMFs) are denoted as $p_{X}(x)$ and $p_{Y}(y)$,
the entropy of $X$,$H(X)$, and the joint entropy of $X$ and $Y$, $H(X,Y)$, can
be calculated as:

$H(X)=-\sum_{x\in\mathcal{X}}p_{X}(x)\log p_{X}(x).$ (1)
$H(X,Y)=-\sum_{x\in\mathcal{X},y\in\mathcal{Y}}p_{XY}(x,y)\log p_{XY}(x,y).$
(2)

The mutual information between $X$ and $Y$ is defined as:

$\begin{split}I(X;Y)&=H(X)+H(Y)-H(X,Y),\\\ \end{split}$ (3)

If we plug Eq 1 and Eq 2 into Eq 3, we can get the calculation of mutual
information as follows:

$I(X;Y)=\sum_{x\in\mathcal{X},y\in\mathcal{Y}}p_{XY}(x,y)\log\frac{p_{XY}(x,y)}{p_{X}(x)p_{Y}(y)}.$
(4)

The equation 4 further suggests that mutual information essentially measures
the distance between the real joint distribution $p_{XY}(x,y)$ and the
distribution under the assumption of complete independence of
$p_{X}(x)p_{Y}(y)$, which makes it a very nature measure of dependence [18].

Mutual information can be used to measure the similarity between images [43,
28]. The probability distributions (in Eq 4) associated with images are
normally approximated using respective marginal and joint histograms. One set
of pixel co-occurrence in the joint distribution is represented by one entry
in a two-dimensional joint histogram.

Regional mutual information [38] is an extension of mutual information. The
intuition of RMI is to embed the spatial information of the images into mutual
information calculations. Instead of standard pixel-wise mutual information,
RMI represents a pixel as a multi-dimensional vector that consists of not only
the pixel itself but also its neighboring pixels. However, the way that
representing pixels with their neighborhood essentially makes extra focus on
the local details. It may be beneficial for generic camera datasets, but
considering that most of the pixels belong to the background in aerial data
and the background details are redundant for action recognition, RMI is not a
good choice to measure the motion information in aerial videos.

###  3.2 Patch Mutual Information (PMI) score

Inspired by regional mutual information (RMI), we propose patch mutual
information (PMI). Instead of mapping pixels into multi-dimensional space, we
divide the image into patches and map each patch into a multi-dimensional
points and then calculate the mutual information between corresponding patches
in adjacent frames. In this manner, we can encode the spatial relationships of
patches in the mutual information calculation without giving too much focus on
the redundant background noises, which yields to better motion information
representation for aerial videos.

Entropy approximation We can exactly calculate the patch mutual information in
the traditional manner that mentioned in Section 3.1, but now with
$d$-dimensional histograms for marginal probability distribution and
$2d$-dimensional histograms for joint distribution. However, such operation
seems reasonable but is hard to implement in practice. In fact, we can make
the calculation easier by assuming that the multivariate distribution of the
image is normally distributed [38] and it is well supported by the
$m$-dependence variable concept proposed in [17].

For a normally distributed set of points $P=[p_{1},p_{2},\cdots,p_{N}]^{T}$,
$p_{1},p_{2},\cdots,p_{N}\in R^{d}$, the entropy can be calculated as:

$H(P)=\frac{1}{2}log((2\pi e)^{d}det(var(P))).$ (5)

where $var(P)$ is the covariance matrix of $P$. Let $E(P)$ be the expected
value of $P$, i.e, the mean of $P$, covariance matrix of $P$ is defined as:

$var(P)=E[(P-E[P])(P-E[P])^{T}],$ (6)

Patch mutual information formulation We here describe the detailed procedures
to get the multivariate distribution for a pair of images and calculate the
patch mutual information between these two images.

As shown in Figure 2, given frame $I_{t-1}$ and $I_{t}$, we divide each frame
into patches with square length $r$. For each patch pair, we flatten the
patches into vectors in $d$-dimensional space where $d=Cr^{2}$ and concatenate
them to a vector in $R^{2d}$. For each pair of frames in $R^{H\times W\times
C}$, we will have $N=\frac{HW}{r^{2}}$ patches (we simply ignore the pixels
along the edges as they will not have a significant effect on the final
entropy). Since each pair of patches has been mapped into a $2d$ vector, we
then have a multivariate distribution of those $N$ points by further
concatenating the $2d$ vectors into a $2d\times N$ matrix,
$P_{t}=[p_{1},p_{2},\cdots,p_{N}]$.

We calculate the covariance matrix of $P_{t}$ by first centering all the $N$
points to their means and then compute the covariance of $P_{t}$ following Eq.
6. The covariance matrix of $P_{t}$ is denoted as $\Omega$:

$\Omega=\frac{1}{N}(P_{t}-\frac{1}{N}\sum^{N}p_{i})(P_{t}-\frac{1}{N}\sum^{N}p_{i})^{T}.$
(7)

We estimate the entropy of these points $H(P_{t})$ using Eq 5. Since we
concatenate the patches from two frames to form $P_{t}\in R^{2d\times N}$,
$H(P_{t})$ is essentially the approximation of the joint entropy between frame
$I_{t-1}$ and $I_{t}$. However, we can easily obtain the covariance matrices
of each frame. Given that $P_{t}$ is a combination of two $d$-dimensional
vectors, the computation of the covariance matrix of $P_{t}$ also results in
the covariance matrices of these 2 subsets of points. In practice, the
covariance of frame $I_{t-1}$, denoted as $\Omega_{t-1}$, corresponds to the
$d\times d$ matrix in the top left of $\Omega$ and $\Omega_{t}$ is the
$d\times d$ matrix in the bottom right representing the covariance of frame
$I_{t}$. Moreover, the marginal entropies, $H(\Omega_{t-1})$ and
$H(\Omega_{t})$, can be computed using the same formulation in Eq 5.

Finally, we obtain the patch mutual information in a pair of frames $I_{t-1}$
and $I_{t}$, denoting as $M_{t}$, following the Eq 3:

$M_{t}=PMI(I_{t-1},I_{t})=H(\Omega_{t-1})+H(\Omega_{t})-H(\Omega).$ (8)

Patch mutual information score Note that, PMI holds the properties of standard
mutual information and is always greater than or equal to 0. In case, the two
frames are complete independent from each other, then $PMI=0$. However, since
video contents are always consistent and have some coherence between the
frames, the two successive frames are always correlated and dependent.
Therefore, PMI between adjacent frames will always be greater than zero,
$M_{t}>0$ for $t\in{1,2,\cdots,T}$. For frame at time 0, we simply define
$M_{0}=0$.

When the two frames are more discriminative, the dependence relationship
weakens and PMI gets smaller, indicating there is potentially more motion
information contained in the current frame. The scale and range of the PMI in
different videos also varies. To better analyze the motion information
distribution, we inversely remap the PMI $M_{t}$ and normalize it with the
$l_{1}$ norm:

$M^{\prime}_{t}=\max_{i\in T}(M_{i})-M_{t},$ (9)
$\hat{M}_{t}=\frac{M^{\prime}_{t}}{\sum_{T}M^{\prime}_{t}}.$ (10)

Eventually, we obtain the PMI score $\hat{M}_{t}$. PMI score represents the
discriminative motion information contained in current frame given the
previous frame. The higher PMI score indicates there existing salient motion
in current frame.

###  3.3 PMI Sampler: Frame Selection

We present a novel frame selection strategy based on shifted leaky ReLu and
cumulative distribution function (CDF). The intuition behind our method is to
sample a sequence of frames that contains as much discriminative motion
information as possible. Similar to MG Sampler [49], our method also utilize a
temporal segmentation scheme and adaptively selects frames according to the
motion information distribution over the entire video.

As shown in Figure 3, given a video contain $T$ frames, we calculate PMI
between adjacent frames and generate a set of PMI scores,
$S=\\{\hat{M_{0}},\hat{M_{1}},\cdots,\hat{M_{T}}\\}$ with mean $\mu$. For each
element $\hat{M_{t}}\in[0,1],t\in\\{0,1,\cdots,T\\}$, we map it based on the
following equation:

$M^{*}_{t}=\left\\{\begin{array}[]{lcl}\alpha\hat{M_{t}}&&{\hat{M_{t}}\leq\mu},\\\
\frac{1-\alpha\mu}{1-\mu}(\hat{M_{t}}-1)+1&&{\hat{M_{t}}>\mu}.\end{array}\right.$
(11)

where $\alpha$ is a hyper parameter to control the smoothness. This function
is very much similar to Leaky ReLu, but with the origin shifts to
$(\mu,\alpha\mu)$ and the function is constrained by the origin and $(1,1)$.
Therefore, we call it as shifted Leaky ReLu. As shown in Figure 4, it
polarizes the PMI scores and makes the motion-salient frames more distinct. We
further examine the impact of $\alpha$ in Section 4.3.

Then, we normalize $M^{*}_{t}$ again using $l_{1}$ norm and accumulate the the
remapped PMI score to get the cumulative motion distribution of the video:

$F_{T}(t)=\sum_{i\leq t}M^{*}_{i}.$ (12)

The constructed cumulative distribution of motion information is shown in Fig
3, where the $X$-axis represents the frame index and $Y$-axis stands for the
motion information accumulation up to the current frame.

Finally, we divide the video frames into N parts with interval $1/N$ on the
$Y$-axis. The frames with index in the corresponding interval on the $X$-axis
will be clustered as one segment. Considering that the $X$-axis value may not
be an integer, we choose the closest integer value instead. Then, we will
randomly select one frame from each segment and constitute a frame sequence to
represent the video for the recognition. As shown in Fig 3 and Fig 1, our
approach can select more frames during periods with higher motion salience and
less frames in motion static segments, enabling more discriminative motion
information to be conveyed to the recognition model.

Figure 4: The PMI score and cumulative distribution before or after the
shifted Leaky ReLu mapping under different value of $\alpha$. After mapping,
motion salient frames become more distinct.

##  4 Results

In this section, we present the experimental results of our approach along
with the state-of-the-art methods on three aerial datasets: UAV-Human [25],
NEC Drone [4] and Diving48 [26]. We also conduct ablation studies on the
effects of patch mutual information, shifted leaky ReLu, and patch size. Our
approach can incorporate with any existing recognition backbones, we use X3D
[10] as our default recognition backbone model unless otherwise specified. The
implementation details are included in supplementary material.

###  4.1 Results on UAV-Human

UAV Human [25] is regarded as the largest and most comprehensive dataset of
UAV-based human behavior understanding data to date. The collection includes
22,476 high-definition videos captured in various indoor and outdoor settings,
encompassing a broad range of lighting and weather conditions. The videos
showcase dynamic backgrounds and feature diverse UAV motions and flying
altitudes, making this dataset highly challenging. A total of 155 unique
actions have been annotated, with some being difficult to differentiate, such
as squeeze and yawn.

We evaluate the performance of our proposed PMI Sampler on UAV-Human along
with current state-of-the-art methods. The results are shown in Table 1. All
the videos are pre-processed with the same procedures in MITFAS [47], and
followed by data augmentation that consistent with X3D [10]. Our method
demonstrates a relative improvement in top-1 accuracy over the current state-
of-the-art methods by $2.2-13.8\%$, with regards to different settings in
number of frames, input frame size, and model initialization.

Method |  Frames Number | Input Size | Init. |  Top-1 (%) Acc. $\uparrow$
---|---|---|---|---
FAR [22] | $8$ | $540\times 540$ | None | $28.8$
MITFAS [47] | $8$ | $540\times 540$ | None | 38.4
Ours | $8$ | $540\times 540$ | None | 39.7
X3D-M [10] | $8$ | $540\times 540$ | Kinetics | $36.6$
FAR [22] | $8$ | $540\times 540$ | Kinetics | $38.6$
DiffFAR [23] | $8$ | $540\times 540$ | Kinetics | $41.9$
Ours | $8$ | $540\times 540$ | Kinetics | 47.7
FAR [22] | $8$ | $620\times 620$ | Kinetics | $39.1$
MITFAS [47] | $8$ | $620\times 620$ | Kinetics | $46.6$
Ours | $8$ | $620\times 620$ | Kinetics | 52.0
X3D-M [10] | $16$ | $224\times 224$ | Kinetics | $30.6$
FAR [22] | $16$ | $224\times 224$ | Kinetics | $31.9$
AZTR [45] | $16$ | $224\times 224$ | Kinetics | $47.4$
MITFAS [47] | $16$ | $224\times 224$ | Kinetics | 50.8
MG Sampler [49] | $16$ | $224\times 224$ | Kinetics | 53.8
Ours | $16$ | $224\times 224$ | Kinetics | 55.0

Table 1: Results on UAV-Human. We demonstrate relative improvements in the
top-1 accuracy by $2.2-13.8\%$ over previous state-of-the-art methods. Our
method outperforms the state-of-the-art methods under different settings,
which further indicates the benefits of our proposed PMI Sampler.

Method | Frames | Backbone |  Diving48 Top-1 (%) |  NEC Drone Top-1 (%)
---|---|---|---|---
Random [10] | $16$ | X3D-M | $71.1$ | $52.0$
Uniform [10] | $16$ | X3D-M | $73.5$ | $55.4$
MG Sampler [49] | $16$ | X3D-M | $74.6$ | $58.5$
K-centered [34] | $16$ |  ViT [7] | $72.5$ | $36.3$
Ours | $16$ | X3D-M | 81.3 | 62.5

Table 2: Results on Diving48 and NEC-Drone. Our method relatively improves the
top-1 accuracy by 9.0% on Diving48, by 6.8% on NEC Drone.

###  4.2 Results on Diving48 and NEC-Drone

NEC Drone [4] is an indoor video dataset that features 5,250 videos depicting
16 distinct actions performed by 19 actors. The videos were captured using a
UAV flying at low altitude over a basketball court. In contrast to the UAV
Human dataset, the lighting conditions in NEC Drone are more consistent;
however, the dataset is plagued by noise due to light reflections. Diving48
[26] is a comprehensive video dataset that offers a fine-grained analysis of
competitive diving, free from any significant biases towards static or short-
term motion representations. It comprises approximately 18,000 trimmed video
clips, each depicting one of 48 unambiguous dive sequences. Although it is not
a UAV-captured dataset, the majority of the videos are captured by cameras at
high altitudes and oblique angles with dynamic movement.

We compare our method with other state-of-the-art methods on Diving48 and NEC-
Drone. The frames are extracted from raw videos and augmented as in X3D [10].
The baseline methods are uniform and random samplings. As shown in Table 2, on
diving48, our method achieves 10.6 - 14.3% relative improvement in top-1
accuracy over the baseline methods and relative 9.0% over the SOTA. On NEC
Drone, PMI Sampler outperforms the baseline methods by 12.8 - 20.2% and
improves by 6.8% over the SOTA, relatively.

Similarity measure |  UAV-Human subset Top-1 Acc (%) |  Time cost per frame (ms)
---|---|---
Euclidean Distance | $56.4$ | $1.7$
Cosine Similarity | $55.0$ | $2.5$
Mutual Information | $58.4$ | $10.6$
Regional Mutual Information (RMI) | $59.2$ | $20.2$
Normalized Mutual Information (NMI) | 58.8 | 32.8
Peak Signal-to-Noise Ratio (PSNR) | 57.0 | 3.7
Structural Similarity Index Measure (SSIM) | 57.7 | 79.9
Patch Mutual Information | 59.8 | 4.5

Table 3: Comparison between different similarity measures. Our proposed Patch
Mutual Information(PMI) outperforms other measures on UAV-Human.

Mapping function |  NEC Drone Top-1 Acc (%) |  Diving48 Subset Top-1 Acc (%)
---|---|---
without Mapping | $59.3$ | $58.8$
Quadratic | $60.3$ | $63.4$
Sigmoid | $61.4$ | $65.7$
Softmax | $58.4$ | $53.9$
Tanh | $60.9$ | $56.6$
ReLu | $60.5$ | $60.6$
Shifted Leaky ReLu | 62.5 | 66.3

Table 4: Comparison between different mapping functions. Shifted Leaky ReLu
can better map the motion information distribution and make it easier to
distinguish frames containing more motion information.

###  4.3 Ablation Studies

We conduct ablation studies on the effectiveness of patch mutual information
and shifted leaky ReLu. We also explore the impact of the hyper-parameter
$\alpha$ in the shifted leaky ReLu functions as well as the size of the
patches. We generate subsets of UAV-Human and Diving48 by randomly choosing
30% videos per class from the original datasets and denote them as UAV-Human
subset ($\sim$ 6k videos) and Diving48 subset ($\sim$ 5.8k videos).

Effectiveness of patch mutual information: As mentioned in Section 2.3, there
are many other similarity measurements like Euclidean distance, cosine
similarity, . We conduct experiments to test the results generated based on
different similarity measures. As shown in table 3, our proposed PMI
outperforms other similarity measures.

Effectiveness of shifted Leaky ReLu: After getting the PMI score for the
video, we remap it using shifted Leaky ReLu. It polarizes the motion
information distribution and makes it easier to distinguish the motion-salient
frames in aerial videos, see Figure 4. However, there are many other mapping
functions we can use, like sigmoid, softmax, tanh functions. We conduct
ablation experiments on different mapping functions, and the results are shown
in Table 4, which demonstrate that shifted Leaky ReLu is the better choice for
aerial videos.

Impact of the hyper-parameter in shifted leaky ReLu: We also conduct
experiments to explore the impact of the hyper-parameter $\alpha$ in the
shifted leaky ReLu function. Results are shown in Table 6. When $\alpha$ is
smaller, the distance between small PMI scores and large PMI scores will be
enlarged, yielding to steeper cumulative motion information distribution, see
Figure 4. With $\alpha$ larger, such distance will be reduced and the
cumulative distribution will be smoother.

Method | Frames | Input Resolution |  Training Time per Epoch (s) $\downarrow$ |  Training Time per Video (ms) $\downarrow$ |  Diving48 Top-1(%) $\uparrow$ |  Diving48 Top-5(%) $\uparrow$
---|---|---|---|---|---|---
Random | $16$ | $224\times 224$ | 196.6 | 13.1 | $71.1$ | $94.8$
Uniform | $16$ | $224\times 224$ | 191.4 | 12.7 | $73.5$ | $95.1$
MG Sampler | $16$ | $224\times 224$ | 287.4 | 19.2 | $74.6$ | $95.0$
PMI Sampler (Ours) | 8 | $224\times 224$ |  248.3 (-39.1) |  16.5 (-2.7) |  75.0 (+0.4) |  95.6 (+0.6)
PMI Sampler (Ours) | 12 | $224\times 224$ |  276.8 (-10.6) |  18.4 (-0.8) |  77.7 (+3.1) |  97.5 (+2.5)
PMI Sampler (Ours) | $16$ | 172 $\times$ 172 |  294.8 (+7.4) |  19.6 (+0.4) |  79.0 (+4.4) |  97.7 (+2.7)
PMI Sampler (Ours) | $16$ | $224\times 224$ |  295.1 (+7.7) |  19.6 (+0.4) |  81.3 (+6.7) |  97.7 (+2.7)

Table 5: Training efficiency. PMI Sampler achieves higher accuracy with less
spatial/temporal information, leading to better speed-accuracy tradeoffs. We
show results for fewer frames (8 and 12) and smaller input sizes
(172$\times$172).

| $\alpha$ | 0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7
---|---|---|---|---|---|---|---|---|---
Diving48 subset | Top-1 (%) | 60.6 | 65.4 | 64.7 | 66.3 | 66.2 | 65.4 | 65.6 | 65.4
Top-5 (%) | 92.6 | 92.8 | 94.4 | 94.9 | 94.6 | 93.3 | 94.5 | 94.6
UAV-Human subset | Top-1 (%) $\uparrow$ | 53.7 | 58.1 | 59.8 | 59.8 | 59.7 | 58.7 | 59.5 | 59.3
Top-5 (%) $\uparrow$ | 83.8 | 85.3 | 86.5 | 86.3 | 85.3 | 86.2 | 85.5 | 86.0
NEC-Drone | Top-1 (%) $\uparrow$ | 54.1 | 57.9 | 58.0 | 58.2 | 60.7 | 61.2 | 62.5 | 61.7
Top-5 (%) $\uparrow$ | 85.4 | 89.5 | 88.6 | 89.3 | 89.5 | 90.0 | 89.6 | 89.5

Table 6: Impact of the hyper-parameter $\alpha$. The dynamic camera movement
in UAV-Human and Diving48 videos necessitates a lower $\alpha(\sim 0.3)$ to
differentiate real motion from background noise. In contrast, NEC-Drone
videos, captured by a hovering UAV, require a higher $\alpha(\sim 0.6)$ to
preserve the motion info distribution.

Impact of the patch size: We compare the results from PMI scores that were
generated using different sizes of patches. As we mentioned in Section 3.2,
the entropy approximation is based on the assumption that the multivariate
distribution of the image is normally distributed, and it is well supported by
the $m$-dependence variable concept. Such approximation is closer to the real
entropy when the patch size is larger. However, if the patch size is larger,
the time for the calculation will be more expensive. As shown in Table 7, we
find out that when patch size is $7\times 7$, we achieve the best trade-off
between accuracy and time cost. Therefore, we divide the images into $7\times
7$ patches for all the other experiments in this paper unless further
specified.

Size of Patch ($r\times r$) |  UAV-Human Top-1 Accuracy (%) |  Time cost per frame (ms)
---|---|---
$3\times 3$ | $53.8$ | $5.4$
$5\times 5$ | $54.6$ | $6.8$
$7\times 7$ | 55.0 | 9.2
$11\times 11$ | $52.3$ | $17.9$
$21\times 21$ | $51.9$ | $67.8$

Table 7: Impact of the patch size. A better tradeoff between accuracy and time
cost is achieved with patch size $7\times 7$.

Dataset | SthSthV2 | UCF101 | HMDB51
---|---|---|---
Top-1(%) | Top-5(%) | Top-1(%) | Top-5(%) | Top-1(%) | Top-5(%)
Uniform | 57.5 | 84.3 | 94.3 | 99.3 | 64.6 | 88.1
MG Sampler | 59.2 | 85.2 | 94.6 | 99.2 | 64.9 | 87.9
Ours (Patch 7$\times$7) | 59.7 (+0.5) | 85.2 (+0) | 94.4 (-0.2) | 99.6 (+0.4) | 64.5 (-0.4) | 87.2 (-0.7)
Ours (Patch 3$\times$3) | 60.3 (+1.1) | 85.8 (+0.6) | 95.1 (+0.5) | 99.5 (+0.3) | 65.5 (+0.6) | 88.1 (+0.2)

Table 8: Results on SthSthV2, UCF101, HMDB51. Given that the majority of
videos in these datasets feature fixed cameras capturing close-range scenes
with prominent human actors and minimal background changes, utilizing smaller
patches (3$\times$3) can effectively capture finer details of the actions,
leading to improved accuracy.

Training efficiency: The training time on the Diving48 dataset is presented in
Table 5. Similar to the MG Sampler, our approach computes the PMI in the pre-
processing phase to avoid redundant computations for the same set of frames
during training. PMI Sampler achieves improved accuracy without significantly
increasing the training time. Also, compared to MG Sampler, our proposed PMI
Sampler achieves better accuracy with less spatial or temporal information
(fewer frames and smaller input size).

Results on ground datasets: We evaluate our proposed PMI Sampler on general
ground camera datasets like SomethingSomething V2, UCF101 and HMDB51. UCF101
and HMDB51 both have 3 train-test splits, we report the average results here
with X3D as the backbone. As shown in Table 8, PMI Sampler also outperforms
the current state-of-the-art on these datasets.

More ablation studies are shown in Appendix. B and C. We also include a
detailed analysis in Appendix. D and more visualization results in Appendix.
E.

##  5 Conclusion, Limitations and Future Work

In this paper, we present a novel frame selection method for aerial video
action recognition. We first introduce patch mutual information (PMI) score to
represent the motion information between adjacent frames by measuring the
similarity of frame patches via mutual information calculation. Then, we
propose an adaptive frame selection strategy based on shifted Leaky ReLu and
cumulative distribution function, which ensures that the sampled frames
comprehensively cover all the essential segments with high motion salience.
Our evaluations on multiple datasets illustrate the effectiveness of our
method. Even though we improved the SOTA methods, the absolute accuracy on
UAV-Human is not very high, and we will further explore the backbone
architecture design to improve the accuracy.

Acknowledgement This work was supported in part by ARO Grants W911NF2110026,
W911NF2310046, W911NF2310352 and Army Cooperative Agreement W911NF2120076

## References

  * [1] Oscar L Barbed, Pablo Azagra, Lucas Teixeira, Margarita Chli, Javier Civera, and Ana C Murillo.  Fine-grained pointing recognition for natural drone guidance.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1040–1041, 2020.
  * [2] Mohammadamin Barekatain, Miquel Martí, Hsueh-Fu Shih, Samuel Murray, Kotaro Nakayama, Yutaka Matsuo, and Helmut Prendinger.  Okutama-action: An aerial view video dataset for concurrent human action detection.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 28–35, 2017.
  * [3] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6299–6308, 2017.
  * [4] Jinwoo Choi, Gaurav Sharma, Manmohan Chandraker, and Jia-Bin Huang.  Unsupervised and semi-supervised domain adaptation for action recognition from drones.  In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1717–1726, 2020.
  * [5] Thomas M. Cover and Joy A. Thomas.  Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing).  Wiley-Interscience, USA, 2006.
  * [6] Ugur Demir, Yogesh S Rawat, and Mubarak Shah.  Tinyvirat: Low-resolution video action recognition.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 7387–7394. IEEE, 2021.
  * [7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [8] Hehe Fan, Zhongwen Xu, Linchao Zhu, Chenggang Yan, Jianjun Ge, and Yi Yang.  Watching a small portion could be as good as watching all: Towards efficient video classification.  In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 705–711. International Joint Conferences on Artificial Intelligence Organization, 7 2018\.
  * [9] Hehe Fan, Zhongwen Xu, Linchao Zhu, Chenggang Clarence Yan, Jianjun Ge, and Yi Yang.  Watching a small portion could be as good as watching all: Towards efficient video classification.  In International Joint Conference on Artificial Intelligence, 2018\.
  * [10] Christoph Feichtenhofer.  X3d: Expanding architectures for efficient video recognition.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 203–213, 2020.
  * [11] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019.
  * [12] Harshala Gammulle, Simon Denman, Sridha Sridharan, and Clinton Fookes.  Predicting the future: A jointly learnt model for action anticipation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5562–5571, 2019.
  * [13] Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani.  Listen to look: Action recognition by previewing audio.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10457–10467, 2020.
  * [14] Ruben Geraldes, Artur Goncalves, Tin Lai, Mathias Villerabel, Wenlong Deng, Ana Salta, Kotaro Nakayama, Yutaka Matsuo, and Helmut Prendinger.  Uav-based situational awareness system using deep learning.  IEEE Access, 7:122583–122594, 2019.
  * [15] Amir Ghodrati, Babak Ehteshami Bejnordi, and Amirhossein Habibian.  Frameexit: Conditional early exiting for efficient video recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15608–15618, 2021.
  * [16] Jiun Tian Hoe, Kam Woh Ng, Tianyu Zhang, Chee Seng Chan, Yi-Zhe Song, and Tao Xiang.  One loss for all: Deep hashing with a single cosine similarity based learning objective.  Advances in Neural Information Processing Systems, 34:24286–24298, 2021.
  * [17] Wassily Hoeffding and Herbert Robbins.  The central limit theorem for dependent random variables.  Duke Mathematical Journal, 15(3):773 – 780, 1948.
  * [18] Aapo Hyvärinen and Erkki Oja.  Independent component analysis: algorithms and applications.  Neural networks, 13(4-5):411–430, 2000.
  * [19] Stefan Klein, Marius Staring, and Josien P. W. Pluim.  Evaluation of optimization methods for nonrigid medical image registration using mutual information and b-splines.  IEEE Transactions on Image Processing, 16:2879–2890, 2007.
  * [20] Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong.  Movinets: Mobile video networks for efficient video recognition.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16020–16030, 2021.
  * [21] Bruno Korbar, Du Tran, and Lorenzo Torresani.  Scsampler: Sampling salient clips from video for efficient action recognition.  2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6231–6241, 2019.
  * [22] Divya Kothandaraman, Tianrui Guan, Xijun Wang, Shuowen Hu, Ming Lin, and Dinesh Manocha.  Far: Fourier aerial video recognition.  In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision – ECCV 2022, pages 657–676, Cham, 2022. Springer Nature Switzerland.
  * [23] Divya Kothandaraman, Ming Lin, and Dinesh Manocha.  Differentiable frequency-based disentanglement for aerial video action recognition.  arXiv preprint arXiv:2209.09194, 2022.
  * [24] Divya Kothandaraman, Xijun Wang, Tianrui Guan, Sean Hu, Ming Lin, and Dinesh Manocha.  Frequency-based aerial video recognition.  In Open Architecture/Open Business Model Net-Centric Systems and Defense Transformation 2023, volume 12544, pages 127–129. SPIE, 2023.
  * [25] Tianjiao Li, Jun Liu, Wei Zhang, Yun Ni, Wenqian Wang, and Zhiheng Li.  Uav-human: A large benchmark for human behavior understanding with unmanned aerial vehicles.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16266–16275, 2021.
  * [26] Yingwei Li, Yi Li, and Nuno Vasconcelos.  Resound: Towards action recognition without representation bias.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 513–528, 2018.
  * [27] Jintao Lin, Haodong Duan, Kai Chen, Dahua Lin, and Limin Wang.  Ocsampler: Compressing videos to one clip with single-step sampling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13894–13903, 2022.
  * [28] Frederik Maes, André M. F. Collignon, Dirk Vandermeulen, Guy Marchal, and Paul Suetens.  Multimodality image registration by maximization of mutual information.  IEEE Transactions on Medical Imaging, 16:187–198, 1997.
  * [29] Balmukund Mishra, Deepak Garg, Pratik Narang, and Vipul Mishra.  Drone-surveillance for search and rescue in natural disaster.  Computer Communications, 156:1–10, 2020.
  * [30] Hazar Mliki, Fatma Bouhlel, and Mohamed Hammami.  Human activity recognition from uav-captured video sequences.  Pattern Recognition (PR), 100:107140, 2020.
  * [31] Mathew Monfort, Bolei Zhou, Sarah Adel Bargal, Alex Andonian, Tom Yan, Kandan Ramakrishnan, Lisa M. Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, and Aude Oliva.  Moments in time dataset: One million videos for event understanding.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 42:502–508, 2020.
  * [32] Lichao Mou, Yuansheng Hua, Pu Jin, and Xiao Xiang Zhu.  Event and activity recognition in aerial videos using deep neural networks and a new dataset.  In IGARSS 2020-2020 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pages 952–955. IEEE, 2020.
  * [33] Kien Nguyen, Clinton Fookes, Sridha Sridharan, Yingli Tian, Xiaoming Liu, Feng Liu, and Arun Ross.  The state of aerial surveillance: A survey.  arXiv preprint arXiv:2201.03080, 2022.
  * [34] Seong Hyeon Park, Jihoon Tack, Byeongho Heo, Jung-Woo Ha, and Jinwoo Shin.  K-centered patch sampling for efficient video recognition.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV, pages 160–176. Springer, 2022.
  * [35] Asanka G Perera, Yee Wei Law, and Javaan Chahl.  Drone-action: An outdoor recorded drone video dataset for action recognition.  Drones, 3(4):82, 2019.
  * [36] Asanka G Perera, Yee Wei Law, Titilayo T Ogunwa, and Javaan Chahl.  A multiviewpoint outdoor dataset for human action recognition.  IEEE Transactions on Human-Machine Systems, 50(5):405–413, 2020\.
  * [37] Josien P. W. Pluim, J. B. Antoine Maintz, and Max A. Viergever.  Mutual-information-based registration of medical images: a survey.  IEEE Transactions on Medical Imaging, 22:986–1004, 2003.
  * [38] Daniel B. Russakoff, Carlo Tomasi, Torsten Rohlfing, and Calvin R. Maurer.  Image similarity using mutual information of regions.  In European Conference on Computer Vision, 2004.
  * [39] Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  In NIPS, 2014.
  * [40] Waqas Sultani and Mubarak Shah.  Human action recognition in drone videos using a few aerial training examples.  Computer Vision and Image Understanding, 206:103186, 2021.
  * [41] Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  2015 IEEE International Conference on Computer Vision (ICCV), pages 4489–4497, 2014.
  * [42] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6450–6459, 2017.
  * [43] Paul A. Viola and William M. Wells.  Alignment by maximization of mutual information.  International Journal of Computer Vision, 24:137–154, 1995.
  * [44] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In European Conference on Computer Vision, 2016.
  * [45] Xijun Wang, Ruiqi Xian, Tianrui Guan, Celso M. de Melo, Stephen M. Nogar, Aniket Bera, and Dinesh Manocha.  Aztr: Aerial video action recognition with auto zoom and temporal reasoning.  arXiv preprint arXiv:2303.01589, 2023.
  * [46] Zuxuan Wu, Caiming Xiong, Chih-Yao Ma, Richard Socher, and Larry S. Davis.  Adaframe: Adaptive frame selection for fast video recognition.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1278–1287, 2018.
  * [47] Ruiqi Xian, Xijun Wang, and Dinesh Manocha.  Mitfas: Mutual information based temporal feature alignment and sampling for aerial video action recognition.  arXiv preprint arXiv:2303.02575, 2023.
  * [48] Yin-Dong Zheng, Zhaoyang Liu, Tong Lu, and Limin Wang.  Dynamic sampling networks for efficient action recognition in videos.  IEEE transactions on image processing, 29:7970–7983, 2020.
  * [49] Yuan Zhi, Zhan Tong, Limin Wang, and Gangshan Wu.  Mgsampler: An explainable sampling strategy for video action recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1513–1522, 2021.
  * [50] Yuan Zhi, Zhan Tong, Limin Wang, and Gangshan Wu.  Mgsampler: An explainable sampling strategy for video action recognition.  2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1493–1502, 2021.

##  Appendix A Implementation Details

Evaluation metrics We evaluate our method and other state-of-the-art methods
using Top-1 and Top-5 accuracy scores, where the predictions are considered to
be correct if the top 1 or top 5 highest probability answers match the actual
label.

Implementation Details All models in this paper are trained using NVIDIA
GeForce 2080Ti GPUs and NVIDIA RTX A5000 GPUs. The initial learning rate is
set at $0.1$ for training from scratch and $0.05$ for initializing with
Kinetics pretrained weights. Stochastic Gradient Descent (SGD) is used as the
optimizer with 0.0005 weight decay and 0.9 momentum. We use cosine/poly
annealing for learning rate decay and multi-class cross entropy loss to
constrain the final predictions. Unless further specified, the videos are
decoded as a single clip and all the frames are randomly scaled and center
cropped to the size $224\times 224$ during training. During testing, we scale
the shorter spatial side to $256$ and take 3 crops of $224\times 224$ to cover
the longer spatial axis. We average the scores for all individual predictions.

##  Appendix B Incorporate with different recognition backbone models

We further demonstrate that our method could be used with different
recognition backbone models to improve the accuracy. We compare the results
using our PMI Sampler as well as uniform sampling and MG Sampler on 3
different backbones: SlowOnly-R50 [11], I3D [3] and X3D [10]. Results on Table
9 show that our method consistently outperforms other methods and brings
accuracy improvement across different backbone models.

Method | Frames | Backbone | Top-1 Acc (%) | Top-5 Acc (%)
---|---|---|---|---
Uniform | $8$ |  I3D [3] | $59.2$ | $89.9$
MG Sampler [49] | $8$ |  I3D [3] | $55.2$ | $87.6$
Ours | $8$ |  I3D [3] | 61.8 | 91.7
Uniform | $8$ |  SlowOnly-R50 [11] | $60.0$ | $90.3$
MG Sampler [49] | $8$ |  SlowOnly-R50 [11] | $57.1$ | $88.4$
Ours | $8$ |  SlowOnly-R50 [11] | 63.1 | 93.5
Uniform | $16$ |  X3D [10] | $73.5$ | $95.1$
MG Sampler [49] | $16$ |  X3D [10] | $74.6$ | $95.0$
Ours | $16$ |  X3D [10] | 81.3 | 97.7

Table 9: Evaluate our method with different recognition backbones on Diving48.
PMI Sampler can be incorporated with any recognition backbone models to
improve the accuracy.

Method | Frames | Backbone | Top-1 (%) | Top-5 (%)
---|---|---|---|---
Uniform | $8$ |  X3D-M [10] | $74.0$ | $95.3$
MG Sampler [49] | $8$ |  X3D-M [10] | $74.6$ | $95.9$
Ours | $8$ |  X3D-M [10] | 75.5 | 96.0

Table 10: Our propposed PMI Sampler can be used in dense clip sampling for
improved accuracy. We demonstrate an relative improvement in top-1 accuracy
over baseline method by $2\%$ and $1.2\%$ over SOTA.

##  Appendix C Use in dense clip sampling

Our proposed PMI Sampler could also be used in dense clip sampling during
training. We uniformly sample the videos into 10 clips and for each clip, we
adaptively select 8 frames. The baseline method is to uniformly select those
frames in each clip. We evaluate the performance of our proposed method along
with the current state-of-the-art MG Sampler in dense clip sampling scenario.
As shown in Table 10, our method achieves a relative improvement over the
baseline method by $2\%$ and $1.2\%$ over SOTA.

##  Appendix D Analysis

In aerial videos, the human actor occupies less than $10\%$ resolutions and
the rest pixels belong to the background. When the camera is moving, the
overall background deviations are much larger than the actual motion changes.
Therefore, pixel-wise RGB difference used in MG Sampler [50] will be dominated
by background noises and fails to map the motion distribution for both videos
in Figure 1. However, our proposed patch mutual information is more robust
because of the inherent advantage of mutual information. Mutual information
measures the image similarity only by considering the overall pixel value
distribution in the two images, see Eq 4. Thus, it is more robust to outliers
and noises. However, it ignores spatial information between pixels and that is
very important for action recognition. Patch mutual information avoids such
issues by dividing the frames into patches and measuring the mutual
information of small patches. In this way, the spatial information within the
patches can be conserved. Because of the robustness of PMI, we can further
employ the shifted Leaky ReLu to make the motion-salient frames easier to
distinguish. As shown in Figure 1, PMI Sampler ensures that the sampled frame
comprehensively covers all the essential segments with high motion salience,
so that key information about the somersault in Diving48 may not be missed.
Also, PMI Sampler is robust to background noises. It can identify the motion
static period even when the camera is shaking in UAV videos, see Figure 1,
selecting more frames from the motion salient period and fewer frames from the
motion static period.

##  Appendix E More Visualization Results

We generate more visualization results between our method and current state-
of-the-art method, MG Sampler [49], on the three datasets: UAV-Human [25],
NEC-Drone [4] and Diving48 [26] in Figure 5,7,6,8.

As mentioned in Section 3.3, our proposed method quantifies the motion
information contained in adjacent frames based on the similarity measure
between corresponding frame patches. As for trimmed videos, human actors are
performing scripted actions in the same scene and the backgrounds are always
similar in the same video. Therefore, our patch similarity guided frame
selection strategy is more robust to background noises. Moreover, since the
backgrounds are similar, the unsimilarity is dominated by human actions, thus
our method yields to better motion information representations. As shown in
Figure 5, 6, when the camera is moving and no salient motion exists, MG
Sampler [49] suffers from the pixel value changes corresponding to the
backgrounds. However, our proposed PMI Sampler can accurately identify the
motion static periods.

Our method also gives more accurate motion information distribution for aerial
videos and makes it much easier to distinguish the motion salient frames, see
Figure 7. As mentioned in Section D, this is attributed to our proposed patch
mutual information score, which considers both the pixel distribution and
spatial relationships inside the patches. Our method can select more frames
from the motion salient periods and fewer frames from the motion static
periods.

##  Appendix F Limitations

Our proposed method may have two limitations. First, as shown in Figure 8,
when the motion is consistent and smooth (such as swing the racket, drink, rub
hands) during the whole video, our method will perform just like uniform
sampling. Second, in the case where the action label is highly associated with
the gesture during motion static periods, our method may be less effective due
to the reduced number of sampled frames during these periods. For instance, as
depicted in the second video in Figure 8 with label ”all clear”, the action is
primarily determined by the static gesture between frames 12 and 28. However,
our approach samples fewer frames during this period, which might hinder the
performance. To mitigate this issue, we can adjust the value of $\alpha$ in
the shifted Leaky ReLu to achieve a smoother motion information distribution,
enabling the selection of more frames during such periods. Nonetheless,
further investigation is necessary to comprehensively address this concern.

Figure 5: Comparison between our method and MG Sampler [49] on typical videos
from Diving48 [26]. As shown above, MG Sampler fails to measure the motion
information between frames and cannot reflect the motion distribution of the
video because of the background changes caused by the camera moving. However,
our method is more robust to background noises and could accurately identify
the motion static periods in the start, middle and end of the videos. Figure
6: Comparisons between our method and MG Sampler [49] on typical videos from
UAV-Human [25] and NEC-Drone [4]. Compare to Diving48 [26], UAV videos are
more shaky and most pixels are corresponding to backgrounds(frames in the
figure are cropped for better visualization). Therefore, they contain more
background noises. MG Sampler fails to handle such challenges from UAV videos.
However, due to the robustness of our proposed patch mutual information, our
method could accurately distinguish the motion static period.  Figure 7: More
comparisons between our method and MG Sampler [49] on typical videos from UAV-
Human [25] and NEC-Drone [4]. Our method makes it easier to distinguish the
motion salient frames. Our method selects more frames from the motion salient
periods and less frames from motion static period, so that sampled frames
contain more useful motion information.  Figure 8: There are two limitations
of our method. First, as shown in the first video (with label: swing the
racket), when motion is consistent and smooth during the whole video, our
method will perform just like uniform sampling. Second, in instances where the
action label is highly associated with the gesture during motion static
periods, our method may be less effective due to the reduced number of sampled
frames during these periods. As shown in the second video above (with label:
all clear), the action is primarily determined by the static gesture between
frame 12 to frame 28. However, our method samples fewer frames from such
period. To mitigate this issue, we can adjust the value of $\alpha$ in the
shifted Leaky ReLu to achieve a smoother motion information distribution,
enabling the selection of more frames during such periods. Nonetheless,
further investigation is necessary to comprehensively address this concern.


11institutetext: Department of Physics, Lund University, P.O. Box 118, 22100
Lund, Sweden
22institutetext: Physikalisches Institut, Albert-Ludwigs-Universität, Stefan-
Meier-Strasse 19, 79104 Freiburg, Germany
33institutetext: Department of Physics, Stockholm University, AlbaNova
University Center, SE-106 91 Stockholm, Sweden
${}^{\text{\textdagger}}$ Current address: Department of Physics, Gothenburg
University, Box 100, 405 30, Gothenburg, Sweden
${}^{\text{\textdaggerdbl}}$ Current address: Department of Physics and
Astronomy, Uppsala University, Box 516, 751 20, Uppsala, Sweden
${}^{\text{*}}$ These authors contributed equally to this work

# Complete characterization of multi-channel single photon ionization

Jasper Peschel${}^{\text{,*}}$ 11 David Busto${}^{\text{,*}}$ 1122 Marius
Plach 11 Mattias Bertolino 11 Maria Hoflund 11 Sylvain
Maclot${}^{\text{,\textdagger}}$ 11 Jimmy Vinbladh 1133 Hampus
Wikmark${}^{\text{,\textdaggerdbl}}$ 11 Felipe Zapata 11 Eva Lindroth 33
Mathieu Gisselbrecht 11 Jan Marcus Dahlström 11 Anne L’Huillier 11 Per Eng-
Johnsson 11

###### Abstract

Ionization of atoms and molecules by absorption of a light pulse results in
electron wavepackets carrying information on the atomic or molecular structure
as well as on the dynamics of the ionization process. These wavepackets can be
described as a coherent sum of waves of given angular momentum, called partial
waves, each characterized by an amplitude and a phase. The complete
characterization of the individual angular momentum components is
experimentally challenging, requiring the analysis of the interference between
partial waves both in energy and angle. Using a two-photon interferometry
technique based on extreme ultraviolet attosecond and infrared femtosecond
pulses, we characterize the individual partial wave components in the
photoionization of the $2p^{6}$ shell in neon. The study of the phases of the
angular momentum channels allows us to unravel the influence of short-range,
correlation and centrifugal effects. This approach enables the complete
reconstruction of photoionization electron wavepackets in time and space,
providing insight into the photoionization dynamics.

Photoionization is a fundamental process that happens when electromagnetic
radiation of high enough frequency is absorbed by matter. Since the 70s,
synchrotron radiation has been used for photoionization studies [1, 2, 3],
playing an important role for our understanding of the quantum nature of
matter and its interaction with light. In general, an electron is released
into the continuum via various ionization channels corresponding to different
initial or final angular momenta. In these channels, the probability to find
the released electron in space is proportional to the square of a spherical
harmonic function, as exemplified in Figure 1 for the case of the ejection of
an electron from the $2p^{6}$ shell of neon. The characterization of the
photoionization process requires the determination of the amplitudes of the
different channels and, when they add coherently, their relative phases.
Experimentally this requires angular detection, to analyse the interference
between different outgoing angular momentum channels [4, 5]. Additionally,
energy resolution is needed to disentangle ionization pathways leading to
different ionic states or distinguish between different excitation schemes.

In order to perform “complete” experiments, where the outgoing electron
wavepacket is reconstructed, additional experimental information is usually
needed [3, 6, 7]. Advances in this direction have been made by preparing atoms
in aligned or oriented excited states, using light with several frequency and
polarization components [8, 9, 10]. Recent experiments have extended such
studies to ground state atoms using two-photon schemes combining either laser-
generated attosecond extreme ultraviolet (XUV) pulses [11, 12] or free-
electron laser femtosecond pulses [13] with infrared (IR) laser pulses. These
have been limited, however, to the characterization of the final two-photon
wavepacket [14, 12, 13].

Figure 1: Angular momenta and energy level diagram: Angular channels involved
in the one- and two-photon transitions addressed in the present work (left
panel) and principle of the interferometric RABBIT technique (right panel).
The photoionization from the $2p^{6}$-ground state of neon towards $s$\- and
$d$\- continuum states (blue arrows) is followed by the absorption or
stimulated emission of an additional IR photon (red arrows, solid and dashed
lines, right panel), leading to final $p$\- and $f$-states. The left panel
presents the quantum paths corresponding to the initial state $m=0$. For
$m=\pm 1$, the path going via $\lambda=0$ is forbidden.

In this article, we completely characterize one-photon ionization from the
$2p^{6}$-ground state of neon, obtaining the amplitude and relative phase of
the electric dipole transition matrix elements towards $s$\- and $d$-continuum
states. To extract channel-resolved one-photon ionization (scattering) phases,
we apply the reconstruction of attosecond beating by interference of two-
photon transitions (RABBIT) technique [11] while detecting electrons with full
angular resolution [15, 16, 17, 18]. We use XUV pulses from high-order
harmonic generation (HHG) [19] in argon in the 20-50 eV range and delayed weak
IR pulses from the laser driving the HHG process. The phase extraction relies
on the fact that transitions between continuum states are universal, i.e.
independent of the atom. We then determine the channel-resolved one-photon
amplitudes from measurements with only the XUV field. The retrieved values for
the scattering phases and channel amplitudes are in excellent agreement with
calculations using angular-channel-resolved many-body perturbation theory [20,
21, 22].

Figure 2: Angle and delay-dependence of photoelectron spectra: (a) Angle-
integrated spectrum as a function of the delay; (b) Angle-resolved two-photon
photoelectron spectrum integrated over all delays; (c) Energy-integrated
sideband 16 as a function of angle and delay. The red dots show the extracted
phase, and the black line shows the result of a simulation (see Methods
section).

Figure 1 describes the principle of our experiment. Photoionization from the
$2p^{6}$-ground state of neon leads to photoelectrons of different angular
momenta $s$ or $d$. Additional absorption or emission of IR photons, through
transitions between continuum states, called “continuum-continuum” (cc)
transitions, leads to sidebands in the photoelectron spectrum. These sidebands
can be reached by two interfering quantum paths (solid and dashed lines in
Figure 1) [11, 23]. Our experiment consists in recording the photoelectron
spectra as a function of delay between the XUV and IR pulses and emission
angle (see experimental details in the Methods section).

Figure 2 presents angular-integrated delay-dependent (a) as well as angular-
resolved delay-integrated photoelectron spectra (b). It shows contributions
from absorption of harmonics 15 to 25 as well as sidebands 16 to 24 (the
number indicates the total energy absorbed in units of the IR photon energy),
as a result of the interaction between released electrons and the infrared
field. The sidebands in Figure 2(a) oscillate at frequency $2\omega$, where
$\omega$ is the angular frequency of the driving laser. As shown in [14], the
phase of the oscillation depends on the intrinsic group delay of the
attosecond pulses as well as on a delay due to the photoionization process.
The photoelectron peaks corresponding to absorption of harmonics oscillate
with opposite phase, due to redistribution of the electrons from the main
absorption peaks to the sidebands [24]. Figure 2(b) presents the angular
dependence of the delay-integrated photoelectron signal. The sidebands are
maximized at the angles of 0° and 180°, corresponding to emission along the
common polarization axis of the XUV and IR fields. In contrast the main
absorption peaks present a maximum at 90°.

Finally, Figure 2(c) shows the delay and angle dependence of sideband 16. The
oscillation phase is extracted by fitting a cosine to the temporal evolution
for each angle. The red dots in Figure 2(c) show the measured phase as a
function of the emission angle. The phase variation around 80° is close to 2
rad. The experimental data is compared to simulations (black line) based on
calculations presented in the Methods section.

The observed angular structure of the sidebands depends not only on the
interference between the different partial waves of the angular momentum
channels reached via single photon ionization, but is also strongly influenced
by the cc-transitions [16, 18]. The additional interaction with the IR field
leads to an increase in the number of angular channels (see Figure 1, where
only the $m=0$ angular path is indicated), and modifies the radial amplitude
and phase of the outgoing photoionization wavepacket [25, 26, 27]. In the
following, we present a general method to retrieve the amplitude and phase of
each photoionization angular channel from experimental data.

Figure 3: Continuum-continuum transitions: Calculated amplitude ratios
$\alpha_{\lambda\ell}^{\pm}$ (a,d) and continuum-continuum phases
$\phi_{L\lambda}^{\pm}$ (b-c and e-f) for the absorption (a-c) and emission
(d-f) processes. For the amplitude ratios, the curves correspond to different
intermediate states ($\lambda$=1,2,3 in blue, red, green, respectively) and
different atoms/initial states (square, He, 1s initial state); (cross, Kr,
3d); (triangle, Ne, 2p); (circle, Ar, 3p). (b,e) refer to transitions with
increasing angular momentum, $L=\lambda+1$, while in (c,f), $L=\lambda-1$ .

The angle- and delay-dependent sideband intensity can be written as

$\displaystyle
I_{SB}(\theta,\tau)\\!\\!\propto\\!\\!\int\\!\\!d\phi\\!\\!\sum_{m=0,\pm
1}\\!\\!\Big{|}\sum_{L=1,3\atop{\lambda=0,2}}\\!\\!\mathcal{M}_{L\lambda\ell
m}^{+}Y_{Lm}(\theta,\phi)\text{e}^{\text{i}\omega\tau}\\!\\!+\\!\\!\mathcal{M}_{L\lambda\ell
m}^{-}Y_{Lm}(\theta,\phi)\text{e}^{-\text{i}\omega\tau}\Big{|}^{2}\\!\\!,$ (1)

where ($\pm$) refers to the pathways with IR absorption ($+$) or emission
($-$), $\mathcal{M}_{L\lambda\ell m}^{\pm}$ is the two-photon transition
matrix element with final state angular momentum $L$, intermediate state
orbital angular momentum $\lambda$, initial state orbital angular momentum
$\ell$ ($\ell=1$ in the case studied here) and initial magnetic quantum number
$m$, which is kept constant in the two-photon transition, since the XUV and IR
fields are linearly polarized in the same direction. $Y_{Lm}(\theta,\phi)$ are
spherical harmonics and $\text{e}^{\pm\text{i}\omega\tau}$ is the phase term
introduced by absorption and emission of the IR photon with angular frequency
$\omega$, which depends on the delay $\tau$ between the XUV and IR fields. The
transition matrix element can be decomposed into amplitude and phase terms as

$\mathcal{M}_{L\lambda\ell
m}^{\pm}=C_{L\lambda}^{m}C_{\lambda\ell}^{m}~{}\sigma_{L\lambda\ell}^{\pm}~{}\text{e}^{\text{i}(\phi_{L\lambda}^{\pm}+\varphi_{\lambda\ell}^{\pm}+\Phi_{2q\mp
1})}.$ (2)

Here, $C_{L\lambda}^{m},C_{\lambda\ell}^{m}$ are known angular coefficients
(See Eq. (S1) of the Supplementary Material, SM) and
$\sigma_{L\lambda\ell}^{\pm}$ is the radial amplitude. The phase term includes
three different contributions: $\varphi_{\lambda\ell}^{\pm}$ is the phase
associated to the one-photon ionization channel $\ell\rightarrow\lambda$,
$\phi_{L\lambda}^{\pm}$ is the cc-phase, and $\Phi_{2q\mp 1}$ the phase of the
$(2q\mp 1)^{\mathrm{th}}$ harmonic field. The characterization of multi-
channel one-photon ionization using two-photon interferometry requires the
determination of all these quantities, from either theoretical arguments or
experimental measurements.

Applying and extending the results of previous work [25, 26, 28], we use the
universal behavior of the cc-transitions to determine some of the terms in Eq.
(2), thus reducing the number of unknown quantities. Figure 3 (a,d) presents
the ratios between the two-photon transition radial amplitudes, calculated as
described in the methods section and defined as

$\alpha_{\lambda\ell}^{\pm}=\frac{\sigma_{(\lambda-1)\lambda\ell}^{\pm}}{\sigma_{(\lambda+1)\lambda\ell}^{\pm}},$
(3)

for increasing or decreasing angular momentum from the same intermediate state
as the function of the kinetic energy of the electron. The different curves
correspond to different intermediate states ($\lambda$=1,2,3 in blue, red and
green) and different atoms and initial states. The ratio shows an universal
behavior [28], independent of the atom. Only the orbital angular momentum of
the intermediate state, $\lambda$, is of importance.

We present $\phi_{L\lambda}^{\pm}$ in Figure 3 for increasing (b,e) and
decreasing (c,f) angular momenta, in the absorption (b,c) and emission cases
(e,f). The colors and symbols, corresponding to different intermediate angular
momenta and atoms/initial states are indicated in the figure caption. These
results show that the variation of the continuum-continuum phase is universal,
depending mainly on whether the IR photon is absorbed or emitted. For the
absorption process, the cc-phase decreases as a function of kinetic energy and
is positive, while for the emission, it increases and is negative. Note that
the phases are not mirror image of each other, i.e.
$\phi_{L\lambda}^{+}\neq-\phi_{L\lambda}^{-}$. The cc-phases depend on whether
the angular momentum increases or decreases, especially at low kinetic energy,
as observed by comparing (b) and (c), or (e) and (f). Finally, the cc-phases
depend only slightly on the intermediate angular momentum (compare blue and
red curves) and not at all on the atomic system (e.g. compare circle and
triangle) in the range of energies studied here.

Our channel-resolved amplitude and phase retrieval is based on the knowledge
of the cc-transitions. The unknown and known quantities in Eq. (1), after
having expressed the transition matrix elements as in Eq. (2), and used the
available information in Fig. 3, are indicated in Table 1 for an initial
$\ell=1$ state. We note that only the phase difference
$\Delta\Phi_{2q}=\Phi_{2q+1}-\Phi_{2q-1}$, and not the individual high-order
harmonic phases, plays a role in Eq. (1). The number of unknown quantities is
therefore nine. While here the case of $\ell=1$ is shown, this is also true
for higher initial angular momenta.

Table 1:  Unknown and known quantities involved in Eq. (1) |  Unknown |  Known
---|---|---
|  quantities |  quantities
Atomic phases |  $\varphi_{01}^{\pm}$, $\varphi_{21}^{\pm}$ |  $\phi_{10}^{\pm}$, $\phi_{12}^{\pm}$, $\phi_{32}^{\pm}$, from Fig. 3(b,c,e,f)
2-photon amplitudes |  $\sigma_{101}^{\pm}$, $\sigma_{321}^{\pm}$ |  $\alpha_{21}^{\pm}$, from Fig. 3(a,d)
|  |  $\rightarrow~{}\sigma_{121}^{\pm}=\alpha_{21}^{\pm}\sigma_{321}^{\pm}$
Harmonic phase |  $\Delta\Phi_{2q}=\Phi_{2q-1}-\Phi_{2q+1}$ |

We determine these nine unknown quantities using a global fit to our
experimental measurements. In general, multiphoton electron angular
distributions can be written as an expansion in Legendre polynomials [26, 27,
29, 30]. For a two-photon transition, without parity mixing, the expansion
needs only three polynomials, $P_{0}(x)=1$, $P_{2}(x)=(3x^{2}-1)/2$, and
$P_{4}(x)=(35x^{4}-30x^{2}+3)/8$, reading as

$I_{\text{SB}}(\theta,\tau)=h_{0}(\tau)+h_{2}(\tau)P_{2}(\text{cos}~{}\theta)+h_{4}(\tau)P_{4}(\text{cos}~{}\theta).$
(4)

The theoretical expressions for the coefficients $h_{i}(\tau)$, $i=0,2,4$, can
be obtained by expanding Eq. (1) and replacing the products of spherical
harmonics by Legendre polynomials, leading to Eqs. (S2-S4) of the SM. Figure 4
shows the variation of $h_{i}(\tau)$ extracted from the experimental data for
each delay (black points).

Figure 4: Global fit to experimental data:  Delay dependence of the
coefficients $h_{i}(\tau)$, $i=0,2,4$ for sideband 18. The black dots are
obtained from the angular distributions (Eq. 4) for each delay and the error
bars correspond to one standard deviation. The red curves are the result of a
simultaneous fit of the delay dependent $h_{i}$ functions using Eqs. (S2-S4)
in the SM.

As is clear from Eqs. (S2-S4) in the SM and from Figure 4, each $h_{i}(\tau)$
oscillates with the delay $\tau$ at the frequency $2\omega$ and is therefore
fully determined by three quantities: mean value, amplitude and phase. Thus, a
total of nine parameters describe the angle and delay dependence of the
sideband signal $I_{\text{SB}}(\theta,\tau)$. This implies that the nine
unknown quantities in table 1 can be determined through a global fit of the
three analytical expressions of $h_{i}(\tau)$ given in the SM, to the
experimentally measured coefficients in Figure 4. The result of such a global
fit is shown by the red lines in Figure 4.

The nine unknown quantities in table 1 are not completely independent, since
the one-photon ionization phases only appear as differences in Eqs. (S2-S4)
(SM). We therefore lock $\varphi_{01}^{-}$ for the first sideband and
determine all of the others ($\varphi_{01}^{+}$,$\varphi_{21}^{\pm}$). To map
out the energy dependence of the one-photon phases, the global fit is repeated
for each sideband. Using the fact that $\varphi_{\lambda
1}^{-}(SB_{n})=\varphi_{\lambda 1}^{+}(SB_{n+2})$, we iteratively retrieve the
one-photon phases, as a function of energy, for the two possible angular
momenta. From now on, we drop the $\pm$ superscript to describe the one-photon
phase, as it only refers to the path used for the determination. The
experimental results for $\varphi_{01}$ and $\varphi_{21}$ are shown in Fig.
5(a) in green and blue symbols respectively, together with the phases
calculated using many-body perturbation theory, as described in the methods
section (solid lines).

The phase $\varphi_{\lambda\ell}$ can be written as the sum of the scattering
phase $\eta_{\lambda}$ and a contribution from the centrifugal barrier
$-\pi\lambda/2$ [25]. The scattering phase is itself the sum of the Coulomb
phase $\varsigma_{\lambda}=\arg\Gamma(\lambda+1-iZ/k)$ and a contribution
$\delta_{\lambda}$ from the short range potential. (Here $Z$ is the atomic
number, $k=\sqrt{2mE}/\hbar$ the wavenumber and $\Gamma$, the gamma function).
In order to emphasize the influence of the short range potential, we also show
in Figure 5(a) $\varsigma_{0}$ and $\varsigma_{2}$ (green and blue dashed
lines), as well as $\varsigma_{0}+\delta_{0}$ and
$\varsigma_{2}+\delta_{2}-\pi$ (green and blue dot-dashed lines), where
$\delta_{\lambda}$ are taken from Kennedy and Manson [1]. The calculated
phases are very close to $\varsigma_{\lambda}+\delta_{\lambda}-\pi\lambda/2$
(compare solid and dot-dashed lines). The difference between $\varphi_{01}$
and $\varsigma_{0}$ is due to the short range potential contributing by $\sim
1.2\pi$, while that between $\varphi_{02}$ and $\varsigma_{2}$ is mainly due
to the effect of the centrifugal barrier, leading to a $\pi$ phase shift. For
$\lambda=2$, short range effects are small due to the centrifugal barrier that
prevents the electron to come close to the core. The measured phases are also
in excellent agreement with the predictions from quantum defect theory at
threshold, leading to $\delta_{\lambda}=\pi\mu_{\lambda}\simeq 1.3\pi$ for
$\lambda=2$ and $\delta_{\lambda}\simeq 0$ for $\lambda=0$, ($\mu_{\lambda}$
is the quantum defect extracted from experimental energy values) [31]. Since
the short range effects for the $s$-electron are comparable to the effect of
the centrifugal barrier for the $d$-electron, the difference between
$\varphi_{01}$ and $\varphi_{21}$ therefore mostly reflects the difference in
Coulomb phases. The increase of the phases with energy, which is similar for
the two angular momenta also follows the behavior of the Coulomb phases.

As explained above, our procedure allows us to determine the phases as
function of energy and angular momentum up to a global phase offset. After
adjustment of this phase offset, the theoretical and experimental results are
found to be in excellent agreement. Both the energy dependence and the
difference between the angular channels is well reproduced by the experimental
results. The observed deviation for the highest energy might be due to low
statistics originating from the low intensity of the 25th harmonic.

Figure 5: Extracted one-photon amplitudes and phases as a function of the
kinetic energy. (a) Scattering phases $\varphi_{01}$ (green) and
$\varphi_{21}$ (blue) for experimental (dots) and simulated data (solid
lines). The dashed lines correspond to the contributions from the Coulomb
phase $\varsigma_{0}$ (green) and $\varsigma_{2}$ (blue), where for the dotted
lines the effect of the short-range potential and the centrifugal effect is
added: $\varsigma_{\lambda}+\delta_{\lambda}-\pi\lambda/2$. For $\lambda=0$
this shift is dominated by the short range potential ($\delta_{0}\approx
1.2\pi$) and for $\lambda=2$ by the centrifugal effect ($\pi$), as indicated
by the arrows. (b) Ratio between amplitudes $\sigma_{21}$ and $\sigma_{01}$
extracted from XUV only data (dots). The black line shows the result of
angular-channel-resolved many-body perturbation simulation.

Finally, in order to fully characterize single photon ionization, it is
necessary to also determine the one-photon amplitudes $\sigma_{\lambda\ell}$.
However, it is not possible to extract $\sigma_{\lambda\ell}$ from the two-
photon ionization data. Instead, we perform additional measurements in the
absence of the IR field, in order to access directly the one-photon transition
amplitudes. The one-photon angle-dependent photoelectron signal is written as:

$I_{\text{H}}(\theta)\propto\int d\phi~{}\sum_{m=0,\pm
1}\Big{|}\sum_{\lambda=0,2}M_{\lambda\ell m}Y_{\lambda
m}(\theta,\phi)\Big{|}^{2}.$ (5)

Here, $M_{\lambda\ell m}$ is the one-photon matrix element from the ground
state with angular momentum $\ell$ and magnetic quantum number $m$ to the
continuum state with angular momentum $\lambda$, which can be written as

$M_{\lambda\ell m}\approx
C_{\lambda\ell}^{m}~{}\sigma_{\lambda\ell}~{}\text{e}^{\text{i}(\varphi_{\lambda\ell}+\Phi_{2q+1})},$
(6)

where $2q+1$ is the order of the harmonic used for the photoionization.
$I_{\text{H}}(\theta)$ can be written as an expansion of Legendre polynomials
$P_{0}$ and $P_{2}$ [see Eqs. (S5) and (S6) in the SM]. The coefficients of
the expansion $h_{0}$ and $h_{2}$ are extracted from the experimental data.
Using the one-photon phases obtained previously, we determine the relative
radial amplitudes of the $\lambda=0$ and $\lambda=2$ channels. The ratio
between these amplitudes is shown in Figure 5(b), together with the calculated
one. The ratio is above one for all energies, thus in agreement with Fano’s
propensity rule for one-photon absorption.

The retrieval of the amplitudes and phases of angular momentum channels in
photoionization has been a major challenge during the last decades, and has
only been realized for specially prepared atoms or for single angular
channels. Here, we present a method using laser-assisted photoionization,
based on the universality of the continuum-continuum transitions to retrieve
the one-photon photoionization amplitudes and phases for the different angular
channels. We apply the method both experimentally and numerically to
characterize photoionization from the 2p6 shell of neon. Our method is general
and can be applied to other shells (e.g. $d$) and thus to more complex atomic
systems. In particular, the study of the phases of the different angular
momentum channels, unravels the interplay between short-range, correlation
and/or centrifugal effects.

##  1 Acknowledgments

The authors acknowledge support from the Swedish Research Council, the
European Research Council (advanced grant QPAP), the Knut and Alice Wallenberg
Foundation, and the Crafoord Foundation.

## Methods

Figure 6: Schematic drawing of the XUV-infrared interferometer. Two holey
mirrors split and recombine the XUV and infrared components of the beam and a
linear translation stage introduces a time delay between them.

The XUV attosecond pulse trains (ATPs) used in the experiments are synthesized
using high-order harmonic generation (HHG) by focusing 40 fs, 806 nm, $\sim$45
mJ pulses from a 10 Hz Ti:sapphire laser into a pulsed gas jet of Argon in a
loose focusing geometry ($\sim$8.7 m focal length) [32]. The obtained XUV
spectrum spans from $\sim$20 to 45 eV. The beamline is designed to generate
high-flux high-order harmonics, hence the loose focusing geometry [33].

The APTs are separated from the fundamental infrared field by taking advantage
of the lower divergence of the XUV beam. Figure 6 shows a scheme of the newly
developed interferometer, which consists of a set of holey mirrors creating
two interferometric arms. The hole diameter is chosen in a way such that the
entire XUV beam propagates through the hole, whereas the reflected part only
consists of fundamental infrared radiation. A translation stage in the
infrared arm varies the path difference between both arms and hence introduces
a time delay. A 200 nm thick aluminum filter in the XUV arm blocks the
remaining infrared and the intense low-order harmonics. Due to the long
beamline design, traditional interferometers add the risk of pointing
instabilities and temporal jitters, while propagating over such long
distances. Hence, we designed this compact in-line interferometer, such that
the path difference is kept as short as possible, which results in excellent
temporal and spatial stability.

Both beams, after being collinearly overlapped, are focused tightly using two
toroidal mirrors in a Wolter configuration [34]. In focus they interact with
the neon target gas introduced using a pulsed Even-Lavie valve [35] and a set
of two skimmers. The resulting photoelectrons are detected by a velocity map
imaging spectrometer (VMIS) with the ability to record angle-resolved momentum
distribution [36, 37]. The 2D projections of the momentum distribution of the
photoelectrons is recorded by a CCD camera imaging a phosphor screen coupled
to a set of multi-channel plates. Since the VMIS is recording the 2D
projection of the 3D momentum distribution, an inverse Abel transform has to
be applied, which is done using an iterative method [38].

Our calculations are based on a one-electron Hamiltonian, with a Dirac-Fock
potential plus a correction that ensures the correct long-range potential for
ionized photoelectrons [20]. The absorption of one ionizing photon is treated
within the Relativistic Random Phase Approximation with Exchange (RPAE)
resulting in a so-called perturbed wave function describing the ionized
electron. The method accounts for important many-body effects such as inter-
channel coupling and ground-state correlation. Exterior complex scaling is
used in order to be able to use a finite numerical box.

The complex-valued two-photon matrix elements, expressed in Eq. (2), are then
calculated as the transition from the perturbed wave function to the final
continuum state in each angular momentum channel, following the procedure
described in [20, 21, 22] for the non-relativistic case. The integration is
performed numerically out to a distance far outside the atomic core, but
within the unscaled region, while the last part of the integral is carried out
using analytical Coulomb waves along the imaginary radial axis. The amplitude
and phase shift of these Coulomb waves are determined from the numerical
solutions for the perturbed wave function and for the final state describing a
free electron within the potential of the remaining ion. The numerical
stability is monitored by comparison of different “break points” between the
numerical and analytical descriptions.

## References

  * [1] D. J. Kennedy and S. T. Manson.  Photoionization of the Noble Gases: Cross Sections and Angular Distributions.  Phys. Rev. A, 5:227, 1972.
  * [2] Volker Schmidt.  Photoionization of atoms using synchrotron radiation.  Rep. Prog. Phys., 55:1483–1659, 1992.
  * [3] Uwe Becker.  Complete photoionisation experiments.  Journal of Electron Spectroscopy and Related Phenomena, 96(1):105–115, 1998.
  * [4] J. Cooper and R. N. Zare.  Angular Distribution of Photoelectrons.  J. Chem. Phys, 48:942, 1968.
  * [5] J. W. Cooper.  Photoelectron-angular-distribution parameters for rare-gas subshells.  Phys. Rev. A, 47:1841–1851, Mar 1993.
  * [6] Uwe Becker.  Comments At. Mol. Phys., 1(1):41–44, 1969.
  * [7] M. Eminyan, K. B. MacAdam, J. Slevin, and H. Kleinpoppen.  Measurements of complex excitation amplitudes in electron-helium collisions by angular correlations using a coincidence method.  Phys. Rev. Lett., 31:576–579, Aug 1973.
  * [8] K. Godehusen, P. Zimmermann, A. Verweyen, A. von dem Borne, Ph. Wernet, and B. Sonntag.  A complete photoionization experiment with polarized atoms using magnetic dichroism and phase tilt measurements.  Phys. Rev. A, 58:R3371–R3374, Nov 1998.
  * [9] N.A. Cherepkov.  Complete experiments in photoionization of atoms and molecules.  Journal of Electron Spectroscopy and Related Phenomena, 144-147:1197–1201, 2005.  Proceeding of the Fourteenth International Conference on Vacuum Ultraviolet Radiation Physics.
  * [10] F J Wuilleumier and M Meyer.  Pump–probe experiments in atoms involving laser and synchrotron radiation: an overview.  Journal of Physics B: Atomic, Molecular and Optical Physics, 39(23):R425–R477, nov 2006.
  * [11] P.M. Paul, E.S. Toma, P. Breger, G. Mullot, F. Augé, Ph. Balcou, H.G. Muller, and P. Agostini.  Observation of a train of attosecond pulses from high harmonic generation.  Science, 292:1689, 2001.
  * [12] Louis H. Haber, Benjamin Doughty, and Stephen R. Leone.  Photoelectron angular distributions and cross section ratios of two-color two-photon above threshold ionization of argon.  The Journal of Physical Chemistry A, 113(47):13152–13158, Nov 2009\.
  * [13] T. Mazza, M. Ilchen, A. J. Rafipoor, C. Callegari, P. Finetti, O. Plekan, K. C. Prince, R. Richter, M. B. Danailov, A. Demidovich, G. De Ninno, C. Grazioli, R. Ivanov, N. Mahne, L. Raimondi, C. Svetina, L. Avaldi, P. Bolognesi, M. Coreno, P. O’Keeffe, M. Di Fraia, M. Devetta, Y. Ovcharenko, Th. Möller, V. Lyamayev, F. Stienkemeier, S. Düsterer, K. Ueda, J. T. Costello, A. K. Kazansky, N. M. Kabachnik, and M. Meyer.  Determining the polarization state of an extreme ultraviolet free-electron laser beam using atomic circular dichroism.  Nature Communications, 5(1):3648, Apr 2014.
  * [14] K. Klünder, J. M. Dahlström, M. Gisselbrecht, T. Fordell, M. Swoboda, D. Guénot, P. Johnsson, J. Caillat, J. Mauritsson, A. Maquet, R. Taïeb, and A. L’Huillier.  Probing single-photon ionization on the attosecond time scale.  Phys. Rev. Lett., 106:143002, 2011.
  * [15] S. A. Aseyev, Y. Ni, L. J. Frasinski, H. G. Muller, and M. J. J. Vrakking.  Attosecond Angle-Resolved Photoelectron Spectroscopy.  Phys. Rev. Lett., 91:223902, 2003.
  * [16] Sebastian Heuser, Álvaro Jiménez Galán, Claudio Cirelli, Carlos Marante, Mazyar Sabbar, Robert Boge, Matteo Lucchini, Lukas Gallmann, Igor Ivanov, Anatoli S. Kheifets, J. Marcus Dahlström, Eva Lindroth, Luca Argenti, Fernando Martín, and Ursula Keller.  Angular dependence of photoemission time delay in helium.  Phys. Rev. A, 94:063409, Dec 2016.
  * [17] D. M. Villeneuve, Paul Hockett, M. J. J. Vrakking, and Hiromichi Niikura.  Coherent imaging of an attosecond electron wave packet.  Science, 356(6343):1150–1153, 2017.
  * [18] Claudio Cirelli, Carlos Marante, Sebastian Heuser, C. L. M. Petersson, Álvaro Jiménez Galán, Luca Argenti, Shiyang Zhong, David Busto, Marcus Isinger, Saikat Nandi, Sylvain Maclot, Linnea Rading, Per Johnsson, Mathieu Gisselbrecht, Matteo Lucchini, Lukas Gallmann, J. Marcus Dahlström, Eva Lindroth, Anne L’Huillier, Fernando Martín, and Ursula Keller.  Anisotropic photoemission time delays close to a fano resonance.  Nature Communications, 9(1):955, Mar 2018.
  * [19] M. Ferray, A. L’Huillier, X.F. Li, L.A. Lompre, G. Mainfray, and C. Manus.  Multiple-harmonic conversion of 1064 nm radiation in rare gases.  J. Phys. B, 21:L31, 1988.
  * [20] JM Dahlström, Thomas Carette, and Eva Lindroth.  Diagrammatic approach to attosecond delays in photoionization.  Phys. Rev. A, 86(6):061402, 2012.
  * [21] J M Dahlström and E Lindroth.  Study of attosecond delays using perturbation diagrams and exterior complex scaling.  J. Phys. B, 47(12):124012, 2014.
  * [22] Jimmy Vinbladh, Jan Marcus Dahlström, and Eva Lindroth.  Many-body calculations of two-photon, two-color matrix elements for attosecond delays.  Phys. Rev. A, 100:043424, Oct 2019.
  * [23] V. Véniard, R. Taïeb, and A. Maquet.  Phase dependence of (N+1) - color (N$>$1) ir-uv photoionization of atoms with higher harmonics.  Phys. Rev. A, 54:721, 1996.
  * [24] Mattias Bertolino and Jan Marcus Dahlström.  Multiphoton interaction phase shifts in attosecond science.  Phys. Rev. Research, 3:013270, Mar 2021.
  * [25] J Marcus Dahlström, Diego Guénot, Kathrin Klünder, Matthieu Gisselbrecht, Johan Mauritsson, Anne L’Huillier, Alfred Maquet, and Richard Taïeb.  Theory of attosecond delays in laser-assisted photoionization.  Chem. Phys., 414:53–64, 2013.
  * [26] Jaco Fuchs, Nicolas Douguet, Stefan Donsa, Fernando Martin, Joachim Burgdörfer, Luca Argenti, Laura Cattaneo, and Ursula Keller.  Time delays from one-photon transitions in the continuum.  Optica, 7(2):154–161, Feb 2020.
  * [27] J Joseph, F Holzmeier, D Bresteau, C Spezzani, T Ruchon, J F Hergott, O Tcherbakoff, P D’Oliveira, J C Houver, and D Dowek.  Angle-resolved studies of XUV–IR two-photon ionization in the RABBITT scheme.  Journal of Physics B: Atomic, Molecular and Optical Physics, 53(18):184007, aug 2020.
  * [28] David Busto, Jimmy Vinbladh, Shiyang Zhong, Marcus Isinger, Saikat Nandi, Sylvain Maclot, Per Johnsson, Mathieu Gisselbrecht, Anne L’Huillier, Eva Lindroth, and Jan Marcus Dahlström.  Fano’s propensity rule in angle-resolved attosecond pump-probe photoionization.  Phys. Rev. Lett., 123:133201, 2019.
  * [29] Katharine L Reid.  Photoelectron angular distributions.  Annu. Rev. Phys. Chem, 54(1):397–424, 2003.
  * [30] E. Arnous, S. Klarsfeld, and S. Wane.  Angular distribution in the two-quantum atomic photoeffect.  Phys. Rev. A, 7:1559–1568, May 1973.
  * [31] H. Friedrich.  Theoretical atomic physics.  Springer Verlag, 1994.
  * [32] B. Manschwetus, L. Rading, F. Campi, S. Maclot, H. Coudert-Alteirac, J. Lahl, H. Wikmark, P. Rudawski, C. M. Heyl, B. Farkas, T. Mohamed, A. L’Huillier, and P. Johnsson.  Two-photon double ionization of neon using an intense attosecond pulse train.  Phys. Rev. A, 93:061402, Jun 2016.
  * [33] P. Rudawski, C. M. Heyl, F. Brizuela, J. Schwenke, A. Persson, E. Mansten, R. Rakowski, L. Rading, F. Campi, B. Kim, P. Johnsson, and A. L’Huillier.  A high-flux high-order harmonic source.  Review of Scientific Instruments, 84(7):–, 2013.
  * [34] H. Coudert-Alteirac, H. Dacasa, E. Campi, F.and Kueny, B. Farkas, F. Brunner, S. Maclot, B. Manschwetus, H. Wikmark, J. Lahl, L. Rading, J. Peschel, B. Major, K. Varjú, G. Dovillaire, P. Zeitoun, P. Johnsson, A. L’Huillier, and P. Rudawski.  Micro-focusing of broadband high-order harmonic radiation by a double toroidal mirror.  Appl. Sci., 7:1159, 2017.
  * [35] U. Even.  Pulsed supersonic beams from high pressure source: Simulation results and experimental measurements.  Advances in Chemistry, 2014:636042, Aug 2014.
  * [36] A. T. J. B. Eppink and D. H. Parker.  Velocity map imaging of ions and electrons using electrostatic lenses: Application in photoelectron and photofragment ion imaging of molecular oxygen.  Rev. Sci. Instr., 68:3477, 1997.
  * [37] Linnea Rading, Jan Lahl, Sylvain Maclot, Filippo Campi, Hélène Coudert-Alteirac, Bart Oostenrijk, Jasper Peschel, Hampus Wikmark, Piotr Rudawski, Mathieu Gisselbrecht, and Per Johnsson.  A versatile velocity map ion-electron covariance imaging spectrometer for high-intensity xuv experiments.  Applied Sciences, 8(6), 2018.
  * [38] M. J. J. Vrakking.  An iterative procedure for the inversion of two-dimensional ion/photoelectron imaging experiments.  Rev. Sci. Instr., 72:4084, 2001.