# Adapting Short-Term Transformers for Action Detection in Untrimmed Videos

Min Yang1 Huan Gao2 Ping Guo3 Limin Wang1, ✉
1State Key Laboratory for Novel Software Technology, Nanjing University
2Inchitech 3Intel Labs China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Vision transformer (ViT) has shown high potential in video recognition, owing
to its flexible design, adaptable self-attention mechanisms, and the efficacy
of masked pre-training. Yet, it still remains unclear how to adapt these pre-
trained short-term ViTs for temporal action detection (TAD) in untrimmed
videos. The existing works treat them as off-the-shelf feature extractors for
each short trimmed snippet without capturing the fine-grained relation among
different snippets in a broader temporal context. To mitigate this issue, this
paper focuses on designing a new mechanism for adapting these pre-trained ViT
models as a unified long-form video transformer to fully unleash its modeling
power in capturing inter-snippet relation, while still keeping low computation
overhead and memory consumption for efficient TAD. To this end, we design
effective cross-snippet propagation modules to gradually exchange short-term
video information among different snippets from two levels. For inner-backbone
information propagation, we introduce a cross-snippet propagation strategy to
enable multi-snippet temporal feature interaction inside the backbone. For
post-backbone information propagation, we propose temporal transformer layers
for further clip-level modeling. With the plain ViT-B pre-trained with
VideoMAE, our end-to-end temporal action detector (ViT-TAD) yields a very
competitive performance to previous temporal action detectors, riching up to
69.0 average mAP on THUMOS14, 37.12 average mAP on ActivityNet-1.3 and 17.20
average mAP on FineAction.

##  1 Introduction

As an important task in video understanding, temporal action detection (TAD)
[13, 12, 21] aims to localize all action instances and recognize their
categories in a long untrimmed video. Most TAD methods [42, 17, 43, 16, 39,
44] rely on the pre-trained action recognition networks (backbones) to extract
short-term features for each snippet and then apply the TAD heads on top of
feature sequence for action detection in long-form videos. In this pipeline,
the feature extracted from the backbone is of great importance in the final
TAD performance. To obtain powerful features, the existing TAD methods [43,
42, 39, 44] have tried different kinds of backbones, from CNN-based backbones
[37, 7] to Transformer-based ones [23, 11]. Recently, the transformer [34] has
become a promising alternative to CNN in modern TAD pipeline design, thanks to
its effective self-attention operations. Furthermore, the flexible vision
transformer (ViT) [10] enables self-supervised masked pre-training on
extensive video datasets [32], resulting in a more robust video representation
learner.

Figure 1:  Different input processing between baseline and our ViT-TAD. The
dashed box illustrates the feature modeling within the backbone. In contrast
to the baseline approach which models each snippet individually, our ViT-TAD
allows snippets to collaboratively interact with each other during the
modeling process within the backbone.

Several TAD methods [39, 9, 44, 8] have tried to apply Transformer-based
backbones, such as VideoSwin [23] and MViT [11], to the task of TAD. Unlike
CNN, these transformer backbones [23, 11, 10] present challenges when applied
directly to untrimmed video modeling due to the quadratic computational cost
associated with self-attention operations [34]. Therefore, the existing
methods often treat the transformer network as the off-the-shelf feature
extractor for each snippet independently, as shown in Fig 1(a), thereby
neglecting the intricate temporal relationships across snippets and failing to
fully harness the end-to-end representation learning power of Transformers for
TAD. Meanwhile, the hierarchical transformer architecture of VideoSwin or MViT
they choose which is hard to benefit from the powerful masked video pre-
training [32] on large-scale unlabeled videos. Therefore, it still remains
unclear how to effectively adapt the short-term plain ViT to untrimmed video
action detection with the ability of capturing cross-snippet temporal
structure in an end-to-end manner.

To this end, we focus on building a simple and general temporal action
detector based on the plain ViT backbone [10]. Rather than employing the pre-
trained short-term ViTs as the snippet feature extractors, we design an
efficient mechanism of adapting them to model longer video consisting of
multiple snippets within a unified transformer backbone, as shown in Fig 1(b).
Following the success of ViT in object detection [14], we first divide video
into non-overlapping snippets and apply both intra-snippet and inter-snippet
self-attention operations to keep a trade-off between representation power and
computational cost. In this way, our unified transformer backbone can model
multiple snippets as a whole and capture the fine-grained and holistic
temporal structure for the TAD task.

Specifically, we apply dense self-attention on all tokens within each snippet
to capture their spatiotemporal relations in intra-snippet blocks and propose
a cross-snippet propagation module to aggregate global temporal information in
a position-wise manner in inter-snippet blocks. These two kinds of blocks are
stacked alternatively to gradually exchange temporal information within long
videos. In addition, inspired by the design of DETR [6], we devise the post-
backbone information propagation module after our Transformer-based backbone,
which is composed of several temporal transformer layers to aggregate global
temporal information. This post-backbone propagation module can effectively
enlarge the temporal receptive field and capture global context. Equipped with
simple TAD head such as BasicTAD [42] and AFSD [16], our final temporal action
detector, termed as ViT-TAD, enjoys a simple yet effective design with end-to-
end training. In particular, our ViT-TAD can embrace the powerful self-
supervised masked pre-training [32] and yield state-of-the-art performance on
the challenging datasets THUMOS14 and ActivityNet-1.3. In summary, our
contributions are as follows:

  * •

We introduce ViT-TAD, the first end-to-end TAD framework that utilizes the
plain ViT backbone. Through the incorporation of a straightforward inner-
backbone information propagation module, ViT-TAD can effectively treat
multiple video snippets as a unified entity, facilitating the exchange of
temporal global information.

  * •

With a simple TAD head and careful implementation, we can train our ViT-TAD in
an end-to-end manner under the limited GPU memory. This simple design fully
unleashes the modeling power of the transformer and embraces the strong pre-
training of VideoMAE [32].

  * •

The extensive experiments on THUMOS14 [13], ActivityNet-1.3 [12] and
FineAction [21] demonstrate that our simple ViT-TAD outperforms the previous
state-of-the-art end-to-end TAD methods.

##  2 Related Work

#### Transformer in Action Recognition.

Action recognition is an important task in video understanding. With the
success of the self-attention mechanism in computer vision, several works
tried to apply the transformer to their structures. Specifically, VTN [25] and
STAM [28] introduced temporal transformers to encode frame-level relationships
between features. ViViT [2] and TimeSformer [4] factorized along spatial and
temporal dimensions on the granularity of the encoder. SMAVT [5] aggregated
information from tokens at the same spatial location within a local temporal
window. Some works [11, 23] tried to re-introduce hierarchical designs into
transformer inspired by ConvNet. Among them, MViT [11] presented a
hierarchical transformer structure to progressively shrink the spatiotemporal
resolution of feature maps and increase channels as the network structure goes
deeper. VideoSwin [23] used shifted window attention to enable information
propagation inspired by Swin Transformer [22]. Furthermore, with the
development of self-supervised learning [32], Transformer-based methods
benefit from larger training data and achieve better results than CNN-based
methods on the action recognition task.

#### Transformer in Temporal Action Detection.

With the development of transformer, more and more TAD approaches began to
apply it or its variants into TAD head [20, 31, 43, 30] or backbone [39, 9,
44, 8]. Different from increasingly successful applications of transformer in
TAD head, few works seriously explored the application of transformer in the
backbone. Existing works [8, 9, 44] treated Transformer-based backbones as
“black box” and applied them as short-term feature extractors. STPT [39]
attempted to explore the inner modeling of a hierarchical Transformer-based
backbone by inserting new blocks in it. However, it needs additional pre-
training on the action recognition dataset which fails to embrace the benefits
of pre-trained big models [38, 26]. In this work, we try to design a new
mechanism for adapting a pre-trained short-term snippet-level non-hierarchical
ViT-based backbone as a unified clip-level video transformer.

Figure 2:  Overview of ViT-TAD. We suppose the ViT-based backbone has $n$
blocks and divide them into several subsets. Each subset has $i$ blocks. We
divide a video clip into several snippets and send each into the backbone for
feature extraction. We perform temporal feature interaction among all snippets
through the inner-backbone information propagation strategy. We further
conduct clip-level modeling to refine clip-level features through the post-
backbone information propagation strategy. $*$ means the last layer is
initialized as zero.

#### End-to-End Temporal Action Detection.

Due to GPU memory constraints, most existing methods treated TAD [17, 18, 41,
45] as a two-step approach separating feature modeling and action detection.
Since temporal features are extracted using backbones that are pre-trained on
the action recognition dataset, there is a gap between these two different
tasks, leading to sub-optimal performance. In order to address this gap,
incorporating the backbone into the network’s training process forms an end-
to-end TAD framework [16, 9, 19, 39, 44]. Simultaneously updating the network
weights of both parts in the two-step approach will easily exceed the GPU
memory limit. To address this issue, some methods [16, 19, 39] tried to
downscale the frame resolution, while others tried to keep the resolution of
the original video frame through improvements in training mechanisms. Among
them, TALLFormer [9] attempted to build an offline memory bank to store
features that cannot be updated synchronously and only update a portion of
features during end-to-end training. Re2TAL [44] built a backbone with
reversible modules to save memory consumption. Due to the existence of
positional encoding, how to fine-tune the TAD model when spatiotemporal
resolution differs from the pre-trained Transformer-based model is a problem.
TALLFormer [9] and Re2TAL [44] circumvented this problem by keeping the same
spatiotemporal resolution as the pre-trained backbone, while STPT [39] pre-
trained its own backbone. We adopt the downscaling frame resolution strategy
and use the operation from DeiT [33] to fix the positional encoding across
resolutions to better use the pre-trained weight of the ViT-based backbone.

##  3 ViT-TAD

Overview. We describe the details of our ViT-TAD pipeline shown in Fig 2.
Formally, for each input video $V\in R^{T\times H\times W\times 3}$, where
$T$, $H$ and $W$ represent the number, height and width of RGB frames
respectively. We divide it into $N_{s}$ non-overlapping snippets
$s=\\{s_{j}\\}^{N_{s}}_{j=1}$, where $s_{j}\in R^{(T/N_{s})\times H\times
W\times 3}$. Then we feed each non-overlapping snippet $s_{j}$ into
consecutive blocks of the ViT-based backbone. In the baseline approach (shown
in Fig 1(a)) which is adopted by the existing methods [9, 44], each snippet
has no interaction with each other during fine-tuning, so we propose _inner-
backbone information propagation_ module to enable cross-snippet interaction
between all snippets. Given that the inner-snippet and cross-snippet modeling
process within the backbone is gradual and sparse, resulting in relatively
small receptive fields, it becomes necessary to enhance the final snippet-
level features by effectively expanding their temporal receptive field. So we
propose the _post-backbone information propagation_ module composed of several
temporal transformer layers to refine snippet-level temporal features with the
global context. Finally, the enhanced features will be sent to the TAD head
for action detection, producing $K$ predicted actions
$\\{(s_{i},e_{i},c_{i},p_{i})\\}_{i=1}^{K}$ as final detection results, where
$s_{i},e_{i},c_{i},p_{i}$ represent the start timestamp, end timestamp,
category and confidence score of $i^{th}$ action instance.

###  3.1 Inner-backbone Information Propagation

We design a cross-snippet propagation module for inner-backbone information
exchange. Unlike hierarchical backbone structures like VideoSwin [23], we do
not shift the temporal windows across layers. To allow information
propagation, we use very few blocks that can go across snippets. Inspired by
ViT-Det [14], we evenly split a pre-trained ViT-based backbone into several
subsets, and then insert a cross-snippet propagation module after the last
block of each subset to enable information propagation. For the simplicity of
the method, we adopt two simple but useful blocks as our cross-snippet
propagation modules, coined as _Local Propagation Block_ and _Global
Propagation Block_.

#### Local Propagation Block.

We choose the bottleneck architecture, comprising three 3D convolutions and an
identity shortcut, as our local block. The last normalization layer within
this block is initialized to zero, establishing an initial identity state for
the block. This design permits integration into any place without breaking the
pre-trained model. Although the cross-snippet modeling capacity of
convolutions is limited, it can still exchange cross-snippet information
gradually via the stacking of multiple such modules.

Figure 3:  Temporal positional encoding for all snippets. Original input
consists of a snippet and its snippet-level PE. We add a learnable additional
temporal PE, called clip-level PE here. PE is short for positional encoding.

#### Global Propagation Block.

Self-attention mechanism is naturally suitable for long-term modeling, so we
choose it to build a global block shown in Fig 2. To keep low computation
overhead and memory consumption, we perform temporal self-attention among all
video snippets after the last block of each stage. It is noteworthy that
spatial dimensions do not participate in self-attention where width($W$) and
height($H$) are directly flattened to batch dimension. Specifically, we obtain
the sequence of clip-level spatiotemporal features $X\in R^{(W\times H)\times
T\times C}$ and split them into $W\times H$ temporal features $x\in R^{T\times
C}$, where $X=\mathrm{concat}(x_{1},x_{2},...,x_{W\times H})$ and each $x$
represents the clip-level temporal feature of a certain spatial location. For
$x$ in each spatial location, we formulate our self-attention mechanism as:

$\begin{split}q=xw_{q},k=xw_{k},v=xw_{v},\\\
y=\mathrm{softmax}(qk^{T}/\sqrt{C_{q}})v\end{split}$ (1)

where $w_{q}\in R^{C\times C_{q}}$, $w_{k}\in R^{C\times C_{k}}$, $w_{v}\in
R^{C\times C_{v}}$, $C_{q}=C_{k}=C/m$ and $m$ is the number of attention head.
Our ViT-TAD complexity is reduced from $O(H^{2}W^{2}L_{s}^{2}N_{s}C)$ to
$O(HWT^{2}C)$ where $L_{s}=T/N_{s}$. Finally, we obtain
$Y=\mathrm{concat}(y_{1},y_{2},...,y_{W\times H})\in R^{(W\times H)\times
T\times C}$. After the application of multi-head dot-product attention, a
linear layer is required to attend to information from distinct representation
sub-spaces and positions jointly. Similar to the local propagation block, the
last linear layer in the global block is initialized as zero and an identity
shortcut is employed. Since each spliced short-term snippet in the clip shares
the same temporal positional encoding, concatenating snippet-level positional
encoding cannot capture the long-term temporal order of the time series.
Therefore, we propose additional clip-level positional encoding to depict the
temporal sequence’s overall order within the video. As shown in Fig 3, we add
a learnable clip-level positional encoding to each temporal position of the
snippet.

###  3.2 Post-backbone Information Propagation

Given that modeling snippet-level features in the backbone with an inner-
backbone information propagation strategy is a sparse and gradual process. we
have designed a post-backbone information propagation module to ensure
adequate feature interaction among snippets and incorporate clip-level
modeling. Inspired by the encoder designed by DETR [6], we build $L$ temporal
transformer encoder layers to allow snippets to interact with each other, as
shown in Fig 2. It is worth noting that the spatial dimension of each snippet
feature has been squeezed with average pooling before this propagation module.
Formally, given the video features $F$ concatenated by several snippets of
feature provided by backbone, several temporal transformer encoder layers are
adopted to enhance these features: $F^{i}=\mathrm{encoder}(F^{i-1})$, where
$i\in[1,L]$ and L is the number of encoder layers, $F^{0}=F$ and $L=3$ in our
experiment.

###  3.3 TAD Head

Our proposed ViT-TAD framework is a simple and general temporal action
detection pipeline. In principle, it is compatible with any TAD head for
converting features into predictions. Following the previous practice in TAD,
we choose a one-stage anchor-free method from BasicTAD [42] on THUMOS14 and
FineAction, a two-stage anchor-free method from AFSD [16] on ActivityNet-1.3.

Discussion with ViT-Det. In spirit, our work is similar to ViT-Det [14] which
also has similar findings in the field of object detection where restricted
window attention is not enough to capture object information in a global view.
However, dynamic motion scenes are different from static object scenes, so our
ViT-TAD behaves differently in several ways. First, temporal continuity makes
action content highly correlated with long-range context, making the global
block perform better than the local block which is different from ViT-Det.
Second, we design post-backbone propagation to enhance further temporal
context modeling based on the first finding, which is missing in ViT-Det.
Finally, unlike spatial downsampling in ViT-Det, our ViT-TAD exhibits the same
temporal downsampling rate and pattern as the previous CNN-based TAD method
(e.g., BasicTAD [42]). This is the reason why we directly use the same FPN
with BasicTAD without specific exploration, but ViT-Det needs further study.
In addition, the TAD community often tend to avoid designing end-to-end
detection pipeline, but instead devise complicated temporal modules on
outdated pre-extracted features (e.g., I3D of 6 years ago). Building a neat
end-to-end TAD pipeline and embracing more powerful models is more urgent and
we make a meaningful attempt to facilitate TAD research in this direction.

##  4 Experiments

#### Datasets and Evaluation Metric.

We perform extensive experiments on THUMOS14 [13], ActivityNet-1.3 [12] and
FineAction [21] to demonstrate the effectiveness of ViT-TAD. THUMOS14 is a
commonly-used dataset in TAD, containing 200 validation videos and 213 test
videos with labeled temporal annotations from 20 categories. ActivityNet-1.3
is a large-scale dataset containing 10,024 training videos and 4,926
validation videos belonging to 200 activities. FineAction is a newly collected
large-scale fine-grained TAD dataset containing 57,752 training instances from
8,440 videos and 24,236 validation instances from 4,174 videos and 21,336
testing instances from 4,118 videos. Following previous work, we report the
mean average precision (mAP) with tIoU thresholds [0.3:0.1:0.7] for THUMOS14
and [0.5:0.05:0.95] for ActivityNet-1.3 and FineAction. Avg is average mAP on
these thresholds.

#### Implementation Details.

For THUMOS14 and FineAction, we sample each video clip with temporal windows
of 32 seconds covering 99.7% of action instances for THUMOS14 and 48 seconds
covering 97.0% of action instances for FineAction. We sample each video clip
with 256 frames for THUMOS14 and 384 frames for FineAction at 8 FPS and then
divide each clip into 16 and 24 snippets each, while each snippet consists of
16 frames. Due to the limited GPU memory, we resize frame’s original size into
short-128 (the short side of the frame is set to 128) and set the crop size
(the size of the cropped images) to $112\times 112$ for THUMOS14 in ablation
experiments and report final results in short-180 original size and $160\times
160$ cropping size. Inspired by DeiT [33], we need to adapt the positional
embeddings to smaller spatial resolution with bicubic interpolation. For TAD
head, we choose a one-stage anchor-free method from BasicTAD [42] for its
astonishing detection performance. We train the model using SGD with a
momentum of 0.9 and weight decay of 0.0001 on 8 TITAN Xp GPUs. The batch size
is set to 2 for each GPU. For ActivityNet-1.3, we resize all videos to 768
frames and then treat these frames as a video clip. Similarly, we divide each
clip into 48 snippets and each snippet consists of 16 frames. For TAD head, we
choose a two-stage anchor-free method from AFSD [16] for its more accurate
boundary regression. We resize the frame’s original size into short-180 and
set the crop size to $160\times 160$. We adopt positional embeddings to
smaller spatial resolution with bicubic interpolation. We train the model
using AdamW [24] with a learning rate of 0.0002 and weight decay of 0.0001 on
8 TITAN Xp GPUs. The batch size is set to 1 for each GPU.

prop. strat. | blk. num | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | Avg
---|---|---|---|---|---|---|---
none | - | 74.6 | 70.2 | 62.6 | 51.3 | 38.4 | 59.4
global | 4 | 78.4 | 74.5 | 66.6 | 53.4 | 38.7 | 62.3
12 | 76.5 | 71.6 | 63.3 | 51.9 | 38.4 | 60.3
local | 4 | 75.9 | 71.6 | 64.7 | 53.3 | 38.3 | 60.8
12 | 77.1 | 72.8 | 65.4 | 54.2 | 40.6 | 62.0

Table 1: Study on inner-backbone information propagation. Both propagation
strategies and different numbers of blocks are explored. “none” is our
baseline without propagation.

temporal position. enc. | 0.3 | 0.5 | 0.7 | Avg
---|---|---|---|---
✓ | 78.4 | 66.6 | 38.7 | 62.3
| 77.9 | 64.4 | 38.8 | 61.3
(a) Effect of clip-level temporal positional encoding

prop. strategy | 0.3 | 0.5 | 0.7 | Avg | Mem.
---|---|---|---|---|---
none | 74.6 | 62.6 | 38.4 | 59.4 | 8.3GB
1D | 78.4 | 66.6 | 38.7 | 62.3 | 9.3GB
3D | 78.0 | 65.5 | 39.7 | 62.2 | 27GB
(b) Comparison between 1D strategy and 3D strategy. Both strategies are shown
in Fig 4.

propagation location | 0.3 | 0.5 | 0.7 | Avg
---|---|---|---|---
evenly 4 blocks | 78.4 | 66.6 | 38.7 | 62.3
first 4 blocks | 75.2 | 64.1 | 38.5 | 60.1
last 4 blocks | 77.6 | 66.0 | 39.6 | 61.8
(c) Study on the locations of global blocks. Here we choose 4 global blocks
strategy.

Table 2: Study on global propagation strategy. “none” is our baseline without
propagation.

###  4.1 Ablation Study

In this section, we perform ablation experiments for ViT-TAD on THUMOS14 with
the ViT-S backbone to discuss the architecture’s design in ViT-TAD.

#### Study on inner-backbone information propagation.

Table 1 ablates our inner-backbone information propagation strategy. These
propagation blocks are placed in the backbone evenly and kernel size is set to
$(3\times 3\times 3)$ for the local propagation block. We compare our global
and local propagation blocks with baseline, both of them performs better than
the baseline in 4-block settings. We further explore the number of propagation
blocks and find that the local strategy benefits from having more blocks, but
the global strategy does the opposite. There are two plausible explanations
for this. First, the model’s ability to learn valuable information is hampered
by frequent global interactions. Second, the self-attention operation cannot
use prior information like the convolutional operator. To enhance temporal
interaction, a more in-depth exploration is warranted.

#### Study on global propagation strategy.

The design of global block is more flexible than local block and requires
further discussion. As shown in Table 2a, additional clip-level positional
encoding can enable the model to recognize clip-level feature sequences after
snippet splicing, leading to better detection results. To save computational
consumption, we disassemble the complete cross-snippet spatiotemporal
relationship modeling into cross-snippet temporal modeling and inner-snippet
spatiotemporal modeling. For vivid comparison, our global propagation block is
referred to as 1D (temporal only) strategy shown in Fig 4(b), and applying
clip-level spatiotemporal modeling on the last block of each subset in the
backbone is referred to as 3D (temporal+spatial) strategy (Fig 4(a)). Both 1D
and 3D strategies obtain similar detection results while 3D strategy causes
more memory consumption shown in Table 2b, inferring that concentrating on
clip-level temporal modeling is sufficient. We further study where global
blocks should be located in the backbone. By default 4 global blocks are
placed evenly. We compare by placing them in the first or last 4 blocks
instead. As is shown in Table 2c, evenly 4 blocks configuration performs best,
and last 4 blocks configuration follows. This is in line with the observation
in ViT [10] that ViT has a longer attention distance in later blocks and is
more localized in earlier ones. Premature temporal interaction of individual
snippet features yields limited benefit. Finally, we adopt 4 evenly placed
global blocks as the final configuration of the global propagation block.

Figure 4:  Comparison between 1D and 3D Propagation Strategy. (a) The 3D
setting: the ($i+1$)th block takes all snippets as input and directly applies
spatiotemporal self-attention to the whole video clip. (b)The 1D setting:
global propagation block is inserted between consecutive backbone blocks.

prop. strategy | use p-b | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | Avg
---|---|---|---|---|---|---|---
none |  | 74.6 | 70.2 | 62.6 | 51.3 | 38.4 | 59.4
✓ | 77.3 | 73.5 | 65.1 | 52.6 | 37.7 | 61.2

Table 3: Effectiveness of post-backbone information propagation. p-b is short
for post-backbone information propagation.

prop. strategy | use p-b | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | Avg
---|---|---|---|---|---|---|---
train | infer
4 global blocks |  |  | 78.4 | 74.5 | 66.6 | 53.4 | 38.7 | 62.3
✓ |  | 78.1 | 73.8 | 65.7 | 54.6 | 38.5 | 62.2
✓ | ✓ | 78.7 | 74.1 | 66.2 | 55.5 | 40.3 | 63.0
12 local blocks |  |  | 77.1 | 72.8 | 65.4 | 54.2 | 40.6 | 62.0
✓ |  | 75.0 | 70.8 | 63.9 | 50.9 | 36.3 | 59.4
✓ | ✓ | 77.5 | 73.3 | 65.9 | 52.7 | 38.2 | 61.5

Table 4: Study on different strategies using post-backbone information
propagation. p-b is short for post-backbone information propagation. Here we
explore how applying p-b or not during the training and inference phases
affects the result.

#### Study on post-backbone information propagation.

Post-backbone information propagation strategy is used for sufficient feature
interaction among snippets. As shown in Table 3, only applying such strategy
is also effective. Equipped with post-backbone information propagation, we
then compare the detection results for both local and global blocks with the
best configuration each in inner-backbone strategy. As is shown in Table 4, we
could see that global propagation strategy obtains better detection results
(from 62.3% to 63.0%), while the detection results of local propagation
strategy drops (from 62.0% to 61.5%). This interesting finding prompts us to
further investigate the impact on the modeling ability of post-backbone
information propagation and backbone after introducing post-backbone
information propagation. When we keep post-backbone information propagation
during training but remove it during inference, detection results are worse
compared with keeping it during inference for both global (from 63.0% to
62.2%) and local (from 61.5% to 59.4%) blocks. This confirms that post-
backbone information propagation can improve the effect of global modeling.
When both strategies discard it during inference and only compare the modeling
capabilities of backbone, we find that backbone’s modeling ability of global
propagation strategy is almost the same (compare 62.3% with 62.2%) but local
propagation strategy drops a lot (from 62.0% to 59.4%). We propose that the
varying modeling approaches employed by the convolutional operator and the
self-attention mechanism contribute to this phenomenon. The former relies on
priors to learn global features with translation invariance, while the latter
disrupts such internal modeling, resulting in a diminished modeling capability
of the backbone. Ultimately, we employ a global propagation strategy for both
internal-backbone information propagation and post-backbone information
propagation in the construction of our ViT-TAD.

#### Comparison between our ViT-TAD and other TAD pipelines.

Method | Backb. | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | Avg
---|---|---|---|---|---|---|---
ActionFormer [43] | ViT-S | 75.2 | 70.0 | 62.0 | 48.7 | 33.4 | 57.9
ViT-S* | 75.6 | 70.3 | 63.4 | 52.4 | 39.6 | 60.3
BasicTAD [42](baseline) | ViT-S | 74.6 | 70.2 | 62.6 | 51.3 | 38.4 | 59.4
ViT-TAD | ViT-S | 78.7 | 74.1 | 66.2 | 55.5 | 40.3 | 63.0

Table 5: Comparison between our ViT-TAD and other TAD pipelines. ViT-S* means
that backbone feature is fine-tuned by BasicTAD in an end-to-end manner.

Method | Backb. | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | Avg
---|---|---|---|---|---|---|---
TALLFormer | Swin-B | 76.0 | - | 63.2 | - | 34.5 | 59.2
TALLFormer (w.o. proposed module) | ViT-B | 78.9 | 75.0 | 67.6 | 56.1 | 37.9 | 63.1
TALLFormer (w. proposed module) | ViT-B | 81.0 | 77.0 | 70.7 | 58.9 | 42.7 | 66.1

Table 6: Effectiveness of proposed module in ViT-TAD to TALLFormer.  Proposed
module means both inner-backbone and post-backbone propagation blocks.

In this section, we compare ViT-TAD with three convincing methods of TAD,
namely ActionFormer [43], BasicTAD [42] and TALLFormer [9]. ActionFormer is
the state-of-the-art TAD pipeline that inputs pre-extracted temporal features
for detection. BasicTAD is a simple end-to-end TAD pipeline whose head is
adopted in ViT-TAD. Here we migrate the feature modeling capabilities of ViT-
TAD to these methods to further explore the reasons why ViT-TAD achieves good
detection results. Shown in Table 5, we use the feature extracted by ViT-S for
ActionFormer, obtaining an average mAP of 57.9% on THUMOS14. While BasicTAD
achieves an average mAP of 59.4%. To eliminate the benefit of end-to-end
training manner, we treat the backbone fine-tuned by BasicTAD as a feature
extractor for ActionFormer, making the detection accuracy of ActionFormer
reach an average mAP of 60.3%. When we further adopt our proposed ViT-TAD, it
can achieve an average mAP of 63.0%, even higher than ActionFormer with fine-
tuned features from BasicTAD. Therefore we can draw several conclusions.
First, the Actionformer’s detector is stronger than BasicTAD. Second, BasicTAD
achieves better results after using the proposed modules in ViT-TAD, which
further proves that ViT-TAD is a better TAD pipeline. In order to further
verify the effectiveness of the module proposed in ViT-TAD, we migrate it to
another end-to-end TAD method TAllFormer that handles clip-level temporal
modeling. Shown in Table 6, our proposed modules improve TALLFormer’s
detection results when both are applied with ViT-B (from 63.1% to 66.1%). This
further proves the effectiveness of both inner-backbone and post-backbone
information propagation strategies proposed in our ViT-TAD.

###  4.2 Analysis

In this section, we further analyze the efficiency and limitations of ViT-TAD.

#### Efficiency analysis of ViT-TAD.

prop. strategy | details | Avg | params | train mem. | FLOPs | FPS
---|---|---|---|---|---|---
none | ViT-S${}_{112\times 112}^{256frames}$ | 59.4 | 25.62M | 8.39G | 104.59G | 2204
4 global blocks+p-b | ViT-S112×112 | 63.0 | 33.35M | 9.33G | 108.80G | 2135
ViT-S160×160 | 64.3 | 33.35M | 23.56G(4.35G*) | 220.59G | 1349
ViT-B160×160 | 69.0 | 131.25M | 8.97G* | 866.73G | 845

Table 7: Practical performance of backbone adaptation strategies. p-b is short
for post-backbone information propagation. $*$ means checkpoint training
strategy is used.

We compare the differences between baseline and variants of ViT-TAD. As is
shown in Table 7, ViT-TAD introduces additional training parameters and
consumes more memory due to the introduction of multiple blocks in the
backbone. Thanks to checkpoint training strategy (marked by $*$ in Table 7),
it can still run successfully with limited computing resources.

Figure 5:  Error analysis of our ViT-TAD. There are error rates of 5 types on
top-10G predictions, where G denotes the number of ground truths.

#### Error analysis.

To analyze the limitations of our model, we provide false positive error chart
[1] of our ViT-TAD on THUMOS14 dataset shown in Fig 5. We obtain quite high
true positive rate on Top-1G predictions. As is mentioned by BasicTAD [42],
one-stage anchor-free methods suffer from “Background Error” due to limited
anchors causing more predictions failing to match ground truth. A more precise
regression loss design is needed.

Figure 6:  Heatmap visualization. (a): Original input video frames. (b):
Heatmap from baseline method. (c): Heatmap from ViT-TAD.

###  4.3 Visualization

We provide heatmap of action “TennisSwing” in THUMOS14 for both baseline and
ViT-TAD detectors to visualize inner-backbone information propagation
strategy’s impact on frame-level modeling. We use Grad-CAM [27] to generate
corresponding heatmap in backbone’s last layer. As is shown in Fig 6, due to
the integration of global temporal information, the model prioritizes the
regions of the image that are relevant for action recognition, encompassing
the athlete’s body and the pertinent background during the action. When the
model is restricted to short-term observations, it may lead to the model
diverting its attention to irrelevant or imprecise areas.

###  4.4 Comparison with the State of the Art

Method | Backb. | Flow | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | Avg
---|---|---|---|---|---|---|---|---
BSN [18] | TSN | ✔ | 53.5 | 45.0 | 36.9 | 28.4 | 20.0 | 36.8
BMN [17] | TSN | ✔ | 56.0 | 47.4 | 38.8 | 29.7 | 20.5 | 38.5
DBG [15] | TSN | ✔ | 57.8 | 49.4 | 39.8 | 30.2 | 21.7 | 39.8
BC-GNN [3] | TSN | ✔ | 57.1 | 49.1 | 40.4 | 31.2 | 23.1 | 40.2
G-TAD [41] | TSN | ✔ | 66.4 | 60.4 | 51.6 | 37.6 | 22.9 | 47.8
RTD-Net [31] | I3D | ✔ | 58.5 | 53.1 | 45.1 | 36.4 | 25.0 | 43.6
VSGN [45] | TSN | ✔ | 66.7 | 60.4 | 52.4 | 41.0 | 30.4 | 50.2
ReAct [30] | TSN | ✔ | 69.2 | 65.0 | 57.1 | 47.8 | 35.6 | 55.0
ActionFormer [43] | I3D | ✔ | 82.1 | 77.8 | 71.0 | 59.4 | 43.9 | 66.8
ActionFormer [43] | ViT-B | ✗ | 80.5 | 75.7 | 68.6 | 57.9 | 42.6 | 65.1
TriDet [29] | I3D | ✔ | 83.6 | 80.1 | 72.9 | 62.4 | 47.4 | 69.3
PBRNet [19]96×96 | I3D | ✔ | 58.5 | 54.6 | 51.3 | 41.8 | 29.5 | 47.1
AFSD [16]96×96 | I3D | ✔ | 67.3 | 62.4 | 55.5 | 43.7 | 31.1 | 52.0
STPT [39]96×96 | MViT | ✗ | 70.6 | 65.7 | 56.4 | 44.6 | 30.5 | 53.6
DaoTAD [35]112×112 | res50-I3D | ✗ | 62.8 | 59.5 | 53.8 | 43.6 | 30.1 | 50.0
e2e-tadtr [20]112×112 | SlowFast-R50 | ✗ | 69.4 | 64.3 | 56.0 | 46.4 | 34.9 | 54.2
TALLFormer [9]224×224 | Swin-B | ✗ | 76.0 | - | 63.2 | - | 34.5 | 59.2
TALLFormer [9]224×224 | ViT-B | ✗ | 78.9 | 75.0 | 67.6 | 56.1 | 37.9 | 63.1
BasicTAD [42]160×160 | SlowOnly-R50 | ✗ | 75.5 | 70.8 | 63.5 | 50.9 | 37.4 | 59.6
Re2TAL [44]224×224 | Swin-T | ✗ | 77.0 | 71.5 | 62.4 | 49.7 | 36.3 | 59.4
Re2TAL [44]224×224 | SlowFast-R101 | ✗ | 77.4 | 72.6 | 64.9 | 53.7 | 39.0 | 61.5
ViT-TAD112×112 | ViT-S | ✗ | 78.7 | 74.1 | 66.2 | 55.5 | 40.3 | 63.0
ViT-TAD160×160 | ViT-S | ✗ | 79.8 | 75.2 | 68.4 | 56.4 | 41.7 | 64.3
ViT-TAD160×160 | ViT-B | ✗ | 84.9 | 80.8 | 72.8 | 61.0 | 45.6 | 69.0

Table 8: Comparison with state-of-the-art methods on THUMOS14. The subscript
indicates the spatial resolution for end-to-end methods. Flow denotes whether
each method uses optical flow as input.

Method | Backb. | Flow | 0.5 | 0.75 | 0.95 | Avg
---|---|---|---|---|---|---
BSN [18] | TSN | ✔ | 46.45 | 29.96 | 8.02 | 30.03
ReAct [30] | TSN | ✔ | 49.60 | 33.00 | 8.60 | 32.60
BMN [17] | TSN | ✔ | 50.07 | 34.78 | 8.29 | 33.85
BC-GNN [3] | TSN | ✔ | 50.56 | 34.75 | 9.37 | 34.26
G-TAD [41] | TSN | ✔ | 50.36 | 34.60 | 9.02 | 34.09
RTD-Net [31] | I3D | ✔ | 47.21 | 30.68 | 8.61 | 30.83
VSGN [45] | TSN | ✔ | 52.38 | 36.01 | 8.37 | 35.07
ActionFormer [43] | I3D | ✔ | 53.50 | 36.20 | 8.20 | 35.60
TriDet [29] | I3D | ✔ | 54.50 | 36.80 | 11.50 | 36.80
TriDet [29] | SlowFast | ✗ | 56.70 | 39.30 | 11.70 | 38.60
STPT [39]96×96 | MViT | ✗ | 51.40 | 33.70 | 6.80 | 33.40
AFSD [16]96×96 | I3D | ✔ | 52.40 | 35.30 | 6.50 | 34.40
PBRNet [19]96×96 | I3D | ✔ | 53.96 | 34.97 | 8.98 | 35.01
e2e-tadtr [20]112×112 | SlowFast-R50 | ✗ | 50.47 | 35.99 | 10.83 | 35.10
TALLFormer [9]224×224 | Swin-B | ✗ | 54.10 | 36.20 | 7.90 | 35.60
Re2TAL [44]224×224 | Swin-T | ✗ | 54.75 | 37.81 | 9.03 | 36.80
Re2TAL [44]224×224 | SlowFast-R101 | ✗ | 55.25 | 37.86 | 9.05 | 37.01
ViT-TAD160×160 | ViT-S | ✗ | 55.09 | 37.81 | 8.75 | 36.69
ViT-TAD160×160 | ViT-B | ✗ | 55.44 | 37.91 | 8.99 | 37.12

Table 9: Comparison with state-of-the-art methods on ActivityNet-1.3. The
subscript indicates the spatial resolution for end-to-end methods. Flow
denotes whether each method uses optical flow as input.

Method | Backbone | Flow | 0.5 | 0.75 | 0.95 | Avg
---|---|---|---|---|---|---
BMN [17] | I3D | ✔ | 14.44 | 8.92 | 3.12 | 9.25
DBG [15] | I3D | ✔ | 10.65 | 6.43 | 2.50 | 6.75
G-TAD [41] | I3D | ✔ | 13.74 | 8.83 | 3.06 | 9.06
ActionFormer [43] | I3D | ✗ | - | - | - | 13.20
VideoMAE V2 [36] | ViT-G | ✗ | - | - | - | 18.20
BasicTAD [42]160×160 | SlowOnly-R50 | ✗ | 24.34 | 10.57 | 0.43 | 12.15
ViT-TAD160×160 | ViT-B | ✗ | 32.61 | 15.85 | 2.68 | 17.20

Table 10: Comparison with state-of-the-art methods on FineAction. The
subscript indicates the spatial resolution for end-to-end methods. Flow
denotes whether each method uses optical flow as input.

We compare our ViT-TAD with the previous state-of-the-art methods on THUMOS14
[13], ActivityNet-1.3 [12] and FineAction [21]. We classify these methods into
end-to-end methods (lower part of tables) and non-end-to-end methods (upper
part of tables). For THUMOS14, the results are shown in Table 8. ViT-TAD with
ViT-S backbone achieves mAP of 63.0% and obtains about 3.6% higher accuracy
compared with baseline. We can also afford video frames with higher resolution
and obtain average mAP of 64.3%. When we adopt ViT-B pre-trained from VideoMAE
[32], our ViT-TAD can obtain an average mAP of 69.0% which outperforms the
state-of-the-art end-to-end TAD methods by a large margin. For
ActivityNet-1.3, the results are shown in Table 9. We attempt to adapt our
model to predict binary action proposals and obtain the detection results by
applying video-level action classifiers [40]. We get competitive results with
ViT-S (36.69%) and state-of-the-art results with ViT-B (37.12%) among all end-
to-end TAD methods. For FineAction, the results are shown in Table 10. we get
good results with ViT-B (17.20%), only slightly lower than VideoMAE V2 [36]
using ViT-G.

##  5 Conclusion

In this paper, we have presented a simple TAD framework (ViT-TAD) based on the
plain ViT backbone. Our ViT-TAD incorporates the inner-backbone propagation
module and post-backbone propagation module to capture more fine-grained
temporal information across different snippets and global contexts. We perform
in-depth ablation studies on the design of different components in ViT-TAD.
With the simple TAD head and powerful masked video pre-training, our ViT-TAD
yields a state-of-the-art performance compared with other end-to-end methods
on the challenging datasets THUMOS14, ActivityNet-1.3 and FineAction. We hope
it will serve as a new TAD baseline for future research.

## References

  * [1] Humam Alwassel, Fabian Caba Heilbron, Victor Escorcia, and Bernard Ghanem.  Diagnosing error in temporal action detectors.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part III, volume 11207 of Lecture Notes in Computer Science, pages 264–280. Springer, 2018.
  * [2] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid.  Vivit: A video vision transformer.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 6816–6826. IEEE, 2021.
  * [3] Yueran Bai, Yingying Wang, Yunhai Tong, Yang Yang, Qiyue Liu, and Junhui Liu.  Boundary content graph neural network for temporal action proposal generation.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV, volume 12373, pages 121–137, 2020.
  * [4] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 813–824. PMLR, 2021.
  * [5] Adrian Bulat, Juan-Manuel Pérez-Rúa, Swathikiran Sudhakaran, Brais Martínez, and Georgios Tzimiropoulos.  Space-time mixing attention for video transformer.  In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 19594–19607, 2021.
  * [6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, volume 12346 of Lecture Notes in Computer Science, pages 213–229. Springer, 2020.
  * [7] João Carreira and Andrew Zisserman.  Quo vadis, action recognition? A new model and the kinetics dataset.  In CVPR, pages 4724–4733, 2017.
  * [8] Shimin Chen, Chen Chen, Wei Li, Xunqiang Tao, and Yandong Guo.  Faster-tad: Towards temporal action detection with proposal generation and classification in a unified network.  CoRR, abs/2204.02674, 2022.
  * [9] Feng Cheng and Gedas Bertasius.  Tallformer: Temporal action localization with a long-memory transformer.  In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV, volume 13694 of Lecture Notes in Computer Science, pages 503–521. Springer, 2022.
  * [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * [11] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer.  Multiscale vision transformers.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 6804–6815. IEEE, 2021.
  * [12] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In CVPR, pages 961–970, 2015.
  * [13] Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar.  THUMOS challenge: Action recognition with a large number of classes.  http://crcv.ucf.edu/THUMOS14/, 2014.
  * [14] Yanghao Li, Hanzi Mao, Ross B. Girshick, and Kaiming He.  Exploring plain vision transformer backbones for object detection.  In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, volume 13669 of Lecture Notes in Computer Science, pages 280–296. Springer, 2022.
  * [15] Chuming Lin, Jian Li, Yabiao Wang, Ying Tai, Donghao Luo, Zhipeng Cui, Chengjie Wang, Jilin Li, Feiyue Huang, and Rongrong Ji.  Fast learning of temporal action proposal via dense boundary generator.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11499–11506, 2020.
  * [16] Chuming Lin, Chengming Xu, Donghao Luo, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu.  Learning salient boundary feature for anchor-free temporal action localization.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 3320–3329. Computer Vision Foundation / IEEE, 2021.
  * [17] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen.  BMN: boundary-matching network for temporal action proposal generation.  In ICCV, pages 3888–3897, 2019.
  * [18] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang.  BSN: boundary sensitive network for temporal action proposal generation.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, ECCV, volume 11208, pages 3–21, 2018.
  * [19] Qinying Liu and Zilei Wang.  Progressive boundary refinement network for temporal action detection.  In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 11612–11619. AAAI Press, 2020.
  * [20] Xiaolong Liu, Song Bai, and Xiang Bai.  An empirical study of end-to-end temporal action detection.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 19978–19987. IEEE, 2022.
  * [21] Yi Liu, Limin Wang, Xiao Ma, Yali Wang, and Yu Qiao.  Fineaction: A fine-grained video dataset for temporal action localization.  CoRR, abs/2105.11107, 2021.
  * [22] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9992–10002. IEEE, 2021.
  * [23] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  Journal of Foo, 14(1):234–778, 2004.
  * [24] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  * [25] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann.  Video transformer network.  CoRR, abs/2102.00719, 2021.
  * [26] AJ Piergiovanni, Weicheng Kuo, and Anelia Angelova.  Rethinking video vits: Sparse video tubes for joint image and video learning.  CoRR, abs/2212.03229, 2022.
  * [27] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  Int. J. Comput. Vis., 128(2):336–359, 2020.
  * [28] Gilad Sharir, Asaf Noy, and Lihi Zelnik-Manor.  An image is worth 16x16 words, what is a video worth?  CoRR, abs/2103.13915, 2021.
  * [29] Dingfeng Shi, Yujie Zhong, Qiong Cao, Lin Ma, Jia Li, and Dacheng Tao.  Tridet: Temporal action detection with relative boundary modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18857–18866, 2023.
  * [30] Dingfeng Shi, Yujie Zhong, Qiong Cao, Jing Zhang, Lin Ma, Jia Li, and Dacheng Tao.  React: Temporal action detection with relational queries.  In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part X, volume 13670 of Lecture Notes in Computer Science, pages 105–121. Springer, 2022.
  * [31] Jing Tan, Jiaqi Tang, Limin Wang, and Gangshan Wu.  Relaxed transformer decoders for direct action proposal generation.  In ICCV, pages 13526–13535, 2021.
  * [32] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  CoRR, abs/2203.12602, 2022.
  * [33] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10347–10357. PMLR, 2021.
  * [34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008, 2017.
  * [35] Chenhao Wang, Hongxiang Cai, Yuxin Zou, and Yichao Xiong.  RGB stream is enough for temporal action detection.  CoRR, abs/2107.04362, 2021.
  * [36] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao.  Videomae V2: scaling video masked autoencoders with dual masking.  CoRR, abs/2303.16727, 2023.
  * [37] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In ECCV, pages 20–36, 2016.
  * [38] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao.  Internvideo: General video foundation models via generative and discriminative learning.  CoRR, abs/2212.03191, 2022.
  * [39] Yuetian Weng, Zizheng Pan, Mingfei Han, Xiaojun Chang, and Bohan Zhuang.  An efficient spatio-temporal pyramid transformer for action detection.  In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV, volume 13694 of Lecture Notes in Computer Science, pages 358–375. Springer, 2022.
  * [40] Yuanjun Xiong, Limin Wang, Zhe Wang, Bowen Zhang, Hang Song, Wei Li, Dahua Lin, Yu Qiao, Luc Van Gool, and Xiaoou Tang.  CUHK & ETHZ & SIAT submission to activitynet challenge 2016.  CoRR, abs/1608.00797, 2016.
  * [41] Mengmeng Xu, Chen Zhao, David S. Rojas, Ali K. Thabet, and Bernard Ghanem.  G-TAD: sub-graph localization for temporal action detection.  In CVPR, pages 10153–10162. Computer Vision Foundation / IEEE, 2020.
  * [42] Min Yang, Guo Chen, Yin-Dong Zheng, Tong Lu, and Limin Wang.  Basictad: an astounding rgb-only baseline for temporal action detection.  CoRR, abs/2205.02717, 2022.
  * [43] Chen-Lin Zhang, Jianxin Wu, and Yin Li.  Actionformer: Localizing moments of actions with transformers.  In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IV, volume 13664 of Lecture Notes in Computer Science, pages 492–510. Springer, 2022.
  * [44] Chen Zhao, Shuming Liu, Karttikeya Mangalam, and Bernard Ghanem.  Re^2tal: Rewiring pretrained video backbones for reversible temporal action localization.  CoRR, abs/2211.14053, 2022.
  * [45] Chen Zhao, Ali K. Thabet, and Bernard Ghanem.  Video self-stitching graph network for temporal action localization.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 13638–13647. IEEE, 2021.


# Eigenstates of linear combinations of phase operators

C. V. Sukumar
Wadham College,
University of Oxford, Oxford OX1 3PN, U.K.

###### Abstract

The eigenstates of linear combinations of the Susskind and Glogower phase
operators for the harmonic oscillator are constructed. It is shown that such
eigenstates are squeezed states.

PACS INDICES: 03.65, 42.50Dv

It is well known that the coherent states of the harmonic oscillator [1, 2]
are eigenstates of the annihilation operator and that the gaussian squeezed
states of the oscillator [3, 4] are eigenstates of particular linear
combinations of the creation and annihilation operators. It has recently been
shown that the phase operators of the harmonic oscillator first considered by
Susskind and Glogower [5, 6] may be used to construct non-Gaussian squeezed
states [7, 8] of the oscillator. In this paper it is shown that the
eigenstates of linear combinations of the Susskind and Glogower phase
operators are squeezed states. For simplicity of notation, Planck’s constant
$\hbar$ and the frequency of the oscillator are set equal to 1.

Consider a harmonic oscillator mode described by the creation and annihilation
operators $a^{{\dagger}}$ and $a$ which satisfy $[a,a^{{\dagger}}]=1$. The
Hamiltonian for the oscillator is $H=N+1/2$ in which $N=a^{{\dagger}}a$ is the
number operator. The oscillator energy eigenstates are denoted by $|n\rangle,\
n=0,1,2,...$ where $|n\rangle$ is an eigenstate of $N$ with eigenvalue $n$.
The coherent state defined by

$\Psi_{c}=e^{\eta a^{{\dagger}}-\eta^{\star}a}\ |0\rangle$ (1)

is generated from the vacuum by the displacement operator and is an eigenstate
of $a$ with eigenvalue $\eta$. The squeezed vacuum defined by

$\Psi_{s}=e^{(\xi a^{{{\dagger}}^{2}}-\xi^{\star}a^{2})/2}\ |0\rangle$ (2)

is generated from the vacuum by the squeeze operator. $\Psi_{s}$ is an
eigenstate with eigenvalue zero of the operator linear combination defined by
[4]

$A=a-\nu a^{{\dagger}}$ (3)

where

$\nu=e^{i\chi}\tanh r\quad,\quad\ \xi=re^{i\chi}$ (4)

It is clear that the magnitude of $\nu$ is restricted to be less than 1. The
eigenstates of A with non-zero eigenvalues are the squeezed and displaced
states obtained from the squeezed vacuum state by the application of the
displacement operator [4].

Susskind and Glogower [5] considered operators of the form

$P^{-}=(N+1)^{-1/2}a\quad,\quad\ P^{+}=a^{{\dagger}}(N+1)^{-1/2}$ (5)

which have the following properties:

$\displaystyle P^{-}|n\rangle$ $\displaystyle=|n-1\rangle,\ n=1,2,\cdots,$
$\displaystyle P^{-}|0\rangle$ $\displaystyle=0,$ $\displaystyle
P^{+}|n\rangle$ $\displaystyle=|n+1\rangle,\ n=0,1,2,\cdots,$ $\displaystyle
P^{-}P^{+}|n\rangle$ $\displaystyle=|n\rangle,\ n=0,1,2,\cdots,$
$\displaystyle P^{+}P^{-}|n\rangle$ $\displaystyle=|n\rangle,\ n=1,2,\cdots,$
$\displaystyle P^{+}P^{-}|0\rangle$ $\displaystyle=0$ (6)

The operators $P^{\pm}$ whose classical analogues are $\exp{\mp i\theta}$,
where $\theta$ is the phase of the classical oscillator, may then be used to
construct the Hermitian linear combinations

$C=(P^{-}+P^{+})/2\quad,\quad\ S=(P^{-}-P^{+})/(2i)$ (7)

in terms of which uncertainty relations for phase and number may be
constructed [6]. The expectation values of C and S for a coherent state
correspond to $\cos\theta$ and $\sin\theta$ respectively when the mean number
of photons in the state is large.

It has recently been shown [7] that operators of the form

$R=P^{-}N\quad,\quad\ R^{{\dagger}}=NP^{+}$ (8)

may be used to construct the state

$\Psi_{R}=e^{(\beta R^{{\dagger}}-\beta^{\star}R)}\ |0\rangle$ (9)

$\Psi_{R}$ may also be written in the form

$\Psi_{R}=[1-{|\alpha|}^{2}]\ \sum_{n=0}^{\infty}\alpha^{n}\ |n\rangle$ (10)

where

$\alpha=e^{i\zeta}\tanh\sigma\quad,\quad\beta=\sigma e^{i\zeta}.$ (11)

It has been shown that $\Psi_{R}$ is a squeezed state. Using equations (6) and
(10) it can be seen that $\Psi_{R}$ is an eigenstate of $P^{-}$ with
eigenvalue $\alpha$.

Similarly operators of the form

$S=P^{-^{2}}N\quad,\quad\ S^{{\dagger}}=NP^{+^{2}}$ (12)

may be used to construct the state

$\Psi_{S}=e^{(\gamma S^{{\dagger}}-\gamma^{\star}S)}\ |0\rangle$ (13)

which may also be written in the form

$\Psi_{S}=[1-{|\delta|}^{2}]\ \sum_{n=0}^{\infty}\delta^{n}\ |2n\rangle$ (14)

where

$\delta=e^{i\phi}\tanh\rho\ \quad,\ \ \gamma=(\rho/2)e^{i\phi}$ (15)

$\Psi_{S}$ has also been shown [8] to be a squeezed state. Using equations (6)
and (14) it is readily established that $\Psi_{S}$ is an eigenstate of
$P^{-^{2}}$ with eigenvalue $\delta$. It is simple to verify that $\Psi_{S}$
is also an eigenstate of the operator

$B=(P^{-}-\delta P^{+})$ (16)

with eigenvalue zero. The squeezed states $\Psi_{R}$, $\Psi_{S}$ and the
operator $B$ are in a sense the phase analogues of the states $\Psi_{c}$,
$\Psi_{s}$ and the operator $A$. Such an analogy leads us to consider the
question of eigenstates of $B$ with non-zero eigenvalue which would be
analogous to the squeezed and displaced states which are eigenstates of $A$
with non-zero eigenvalue.

Let

$\Psi=\sum_{n=0}^{\infty}C_{n}\ |n\rangle$ (17)

be a solution of

$[P^{-}-\delta P^{+}]\ \Psi=\mu\ \Psi\ .$ (18)

Then $C_{n}$ must satisfy

$C_{1}=\mu C_{0}\ ,\ \ C_{n+1}-\delta C_{n-1}=\mu C_{n},\ n=1,2,\cdots\\\ $
(19)

For the special case when $\delta=\mu^{2}$ a solution to the above recursion
relations is given by $C_{n}=\mu^{n}F_{n}$ where $F_{n}$ is the $n^{th}$
Fibonacci number. A procedure analogous to that used for finding closed
analytic expression for the Fibonacci sequence may be used to solve the
recursion relations given by equation (19) in the general case when $\delta$
and $\mu$ are free parameters. The solution for $C_{n}$ may be given in the
form

$C_{n}=C_{0}\ \delta^{n/2}\ [p_{1}^{n+1}-p_{2}^{n+1}]/[p_{1}-p_{2}]$ (20)

where

$\displaystyle p_{1}$ $\displaystyle=[\lambda+\sqrt{\lambda^{2}+4}]/2,$
$\displaystyle p_{2}$ $\displaystyle=[\lambda-\sqrt{\lambda^{2}+4}]/2$
$\displaystyle\lambda$ $\displaystyle=\mu/\sqrt{\delta}$ (21)

In deriving the above solution it has been assumed that $\lambda\not=2i$. In
the limiting case that $\lambda\rightarrow 2i$, it is clear that
$p_{1}\rightarrow p_{2}$ and L’Hospital’s rule may be used to simplify
equation (20) to the form

$C_{n}\ =\ C_{0}\ (n+1)\ i^{n}\ \delta^{n/2}\ ,\ \mu=2i\sqrt{\delta}$ (22)

In rest of this paper we will consider the solution given by equations (20)
and (21) and restrict the analysis to real values of $\delta$ and $\mu$.

The free time evolution of a state which at time t=0 has the form given by
equation (17) is

$\Psi(t)\ =\ \sum_{n=0}^{\infty}\ e^{-i(n+1/2)t}\ C_{n}\ |n\rangle.$ (23)

The variances of the Hermitian linear combinations

$q=(a+a^{{\dagger}})/\sqrt{2}\ \ ,\ \ p=-i(a-a^{{\dagger}})/\sqrt{2}$ (24)

for the state $\Psi(t)$ when $C_{n}$ are chosen to be real can be shown to be
given by

$\displaystyle Var(q)$ $\displaystyle=1/2\
-(F_{1}-F_{2})+2(F_{1}-F_{3}^{2})\cos^{2}t$ (25) $\displaystyle Var(p)$
$\displaystyle=1/2\ -(F_{1}-F_{2})+2(F_{1}-F_{3}^{2})\sin^{2}t$ (26)

where

$\displaystyle F_{1}$
$\displaystyle=\sum_{n=0}^{\infty}C_{n}C_{n+2}\sqrt{(n+1)(n+2)}$ (27)
$\displaystyle F_{2}$ $\displaystyle=\sum_{n=0}^{\infty}C_{n}^{2}\ n$ (28)
$\displaystyle F_{3}$
$\displaystyle=\sum_{n=0}^{\infty}C_{n}C_{n+1}\sqrt{n+1}$ (29)

A state $\Psi$ is said to be squeezed when Var(p) or Var(q) can dip below the
value 1/2. A sufficient condition for squeezing is that $F_{1}\ >\ F_{2}$.

It is easy to show that when $C_{n}$ is given by equations (20) and (21) then

$C_{n}C_{n+2}\ -\ C_{n+1}^{2}\ =\ C_{0}^{2}\ \delta^{n+1}\ (-)^{n}\ \ ,\ \
C_{n}C_{n+2}\ >\ 0\ $ (30)

The above conditions together with the property that $\sqrt{(n+1)(n+2)}>(n+1)$
for any $n$ and the expression

$\sum_{n=0}^{\infty}(-)^{n}\delta^{n+1}\ (n+1)\ =\ (1+\delta)^{-2}$ (31)

may be used to establish that for the state defined by equations (17),(20) and
(21), $(F_{1}\ -\ F_{2})\ >\ 0$ and therefore it is a squeezed state. We have
shown that the eigenstates of $B$ defined by equation (16) are given by
equations (17),(18),(20) and (21) and that these states are squeezed states.
It is interesting to note that in particular a linear superposition of number
states of the form given by equation (17) with the coefficients $C_{n}$ given
by a Fibonacci sequence is a squeezed state.

The state $\Psi_{R}$ defined by equation (10) has the form of a thermal
superposition. Hence we may interpret the squeezed state $\Psi$ defined by
equations (17) and (20) as a linear of superposition of two thermal states
each of which is a squeezed state. Multiplication of both sides of equation
(18) from the left by $P^{-}$ and use of the property that $P^{-}P^{+}=1$
leads to

$[P^{-^{2}}\ -\ \mu P^{-}]\ \Psi\ =\ \delta\ \Psi$ (32)

which shows that $\Psi$ is also an eigenstate of the operator $[P^{-^{2}}-\mu
P^{-}]$ with eigenvalue $\delta$.

The structure of equation (30) suggests that a linear superposition of the
form

$\Phi=\sum_{n=0}^{\infty}\ C_{n}/\sqrt{n!}\ \ |n\rangle$ (33)

with $C_{n}$ given by equations (20) and (21) will lead to

$F\equiv(F_{1}-F_{2})\ =\ C_{0}^{2}\ \delta\ e^{-\delta}$ (34)

The positivity of $F$ for positive $\delta$ guarantees that $\Phi$ is also a
squeezed state. It can be established that $\Phi$ is a solution of the
eigenvalue equation

$(a^{2}\ -\ \mu a)\ \Phi\ =\ \delta\ \Phi\ .$ (35)

Hence the squeezed state defined by equations (20),(21) and (33) is an
eigenstate of $(a^{2}-\mu a)$ with positive eigenvalue $\delta$. When written
in the form

$\Phi=C_{0}p_{1}/(p_{1}-p_{2})\sum_{n=0}^{\infty}{(p_{1}\sqrt{\delta})}^{n}/\sqrt{n!}\
\ |n\rangle\ -\
C_{0}p_{2}/(p_{1}-p_{2})\sum_{n=0}^{\infty}{(p_{2}\sqrt{\delta})}^{n}/\sqrt{n!}\
\ |n\rangle$ (36)

it is clear that $\Phi$ is a linear superposition of of two coherent states
with displacement parameters $(p_{1}\sqrt{\delta})$ and $(p_{2}\sqrt{\delta})$
respectively. The construction of squeezed states by the superposition of
coherent states and thermal states has been the subject of some recent
discussion [9]. It is clear that equations (33) and (36) for the coherent
state superposition $\Phi$ are analogous to equations (17) and (20) for the
thermal state superposition $\Psi$.

In this paper we have shown that the eigenstates of linear combinations of the
Susskind-Glogower phase operators may be readily constructed and that such
states are squeezed states. We have shown that there is a great deal of
similarity between the properties of the eigenstates of operator linear
combinations generated from the operator sets $(P^{-},P^{+})$ and
$(a,a^{{\dagger}})$.

## References

  * [1] E. Schrödinger, Naturwiss. 14, 664 (1926).
  * [2] R.J. Glauber, Phys. Rev. 131, 2766 (1963).
  * [3] H.P. Yuen, Phys. Rev. A 13, 2226 (1975).
  * [4] R.W. Henry and S.C. Glotzer, Am. J. Phys. 56, 318 (1988).
  * [5] L. Susskind and J. Glogower, Physics 1, 49 (1964).
  * [6] P. Carruthers and M.M. Nieto, Phys. Rev.Lett. 14, 387 (1965).
  * [7] C.V. Sukumar, J. Phys. A 21, L1065 (1988).
  * [8] C.V. Sukumar, J. Mod. Opt. 36, 1591 (1989).
  * [9] V. Buzek and P.L. Knight, Opt. Commun. (Netherlands) 81, 331 (1991).


* [92] Sergei Gukov, James Halverson, and Fabian Ruehle.  Rigor with machine learning from field theory to the poincaréconjecture.  Nature Reviews Physics, 2024.
  * [93] Shumao Zhang, Pengchuan Zhang, and Thomas Y Hou.  Multiscale invertible generative networks for high-dimensional bayesian inference.  In International Conference on Machine Learning, pages 12632–12641. PMLR, 2021.
  * [94] Jinchao Xu and Ludmil Zikatanov.  Algebraic multigrid methods.  Acta Numerica, 26:591–721, 2017.
  * [95] Yifan Chen, Thomas Y Hou, and Yixuan Wang.  Exponentially convergent multiscale finite element method.  Communications on Applied Mathematics and Computation, pages 1–17, 2023.
  * [96] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  Advances in neural information processing systems, 33:7462–7473, 2020.

Appendix

##  Appendix A KAN Functionalities

Table 7 includes common functionalities that users may find useful.

Functionality | Descriptions
---|---
model.train(dataset) | training model on dataset
model.plot() | plotting
model.prune() | pruning
model.fix_symbolic(l,i,j,fun) |  fix the activation function $\phi_{l,i,j}$ to be the symbolic function fun
model.suggest_symbolic(l,i,j) |  suggest symbolic functions that match the numerical value of $\phi_{l,i,j}$
model.auto_symbolic() |  use top 1 symbolic suggestions from suggest_symbolic to replace all activation functions
model.symbolic_formula() | return the symbolic formula
Table 7: KAN functionalities

##  Appendix B Learnable activation networks (LANs)

###  B.1 Architecture

Besides KAN, we also proposed another type of learnable activation networks
(LAN), which are almost MLPs but with learnable activation functions
parametrized as splines. KANs have two main changes to standard MLPs: (1) the
activation functions become learnable rather than being fixed; (2) the
activation functions are placed on edges rather than nodes. To disentangle
these two factors, we also propose learnable activation networks (LAN) which
only has learnable activations but still on nodes, illustrated in Figure B.1.

For a LAN with width $N$, depth $L$, and grid point number $G$, the number of
parameters is $N^{2}L+NLG$ where $N^{2}L$ is the number of parameters for
weight matrices and $NLG$ is the number of parameters for spline activations,
which causes little overhead in addition to MLP since usually $G\ll N$ so
$NLG\ll N^{2}L$. LANs are similar to MLPs so they can be initialized from
pretrained MLPs and fine-tuned by allowing learnable activation functions. An
example is to use LAN to improve SIREN, presented in Section B.3.

Comparison of LAN and KAN. Pros of LANs:

  1. (1)

LANs are conceptually simpler than KANs. They are closer to standard MLPs (the
only change is that activation functions become learnable).

  2. (2)

LANs scale better than KANs. LANs/KANs have learnable activation functions on
nodes/edges, respectively. So activation parameters in LANs/KANs scale as
$N$/$N^{2}$, where $N$ is model width.

Cons of LANs:

  1. (1)

LANs seem to be less interpretable (weight matrices are hard to interpret,
just like in MLPs);

  2. (2)

LANs also seem to be less accurate than KANs, but still more accurate than
MLPs. Like KANs, LANs also admit grid extension if theLANs’ activation
functions are parametrized by splines.

###  B.2 LAN interpretability results

Figure B.1: Training of a learnable activation network (LAN) on the toy
example $f(x,y)={\rm exp}({\rm sin}(\pi x)+y^{2})$. Figure B.2: LANs on
synthetic examples. LANs do not appear to be very interpretable. We conjecture
that the weight matrices leave too many degree of freedoms.

We present preliminary interpretabilty results of LANs in Figure B.2. With the
same examples in Figure 4.1 for which KANs are perfectly interpretable, LANs
seem much less interpretable due to the existence of weight matrices. First,
weight matrices are less readily interpretable than learnable activation
functions. Second, weight matrices bring in too many degrees of freedom,
making learnable activation functions too unconstrained. Our preliminary
results with LANs seem to imply that getting rid of linear weight matrices (by
having learnable activations on edges, like KANs) is necessary for
interpretability.

###  B.3 Fitting Images (LAN)

Figure B.3: A SIREN network (fixed sine activations) can be adapted to LANs
(learnable activations) to improve image representations.

Implicit neural representations view images as 2D functions $f(x,y)$, where
the pixel value $f$ is a function of two coordinates of the pixel $x$ and $y$.
To compress an image, such an implicit neural representation ($f$ is a neural
network) can achieve impressive compression of parameters while maintaining
almost original image quality. SIREN [96] proposed to use MLPs with periodic
activation functions to fit the function $f$. It is natural to consider other
activation functions, which are allowed in LANs. However, since we initialize
LAN activations to be smooth but SIREN requires high-frequency features, LAN
does not work immediately. Note that each activation function in LANs is a sum
of the base function and the spline function, i.e., $\phi(x)=b(x)+{\rm
spline}(x)$, we set $b(x)$ to sine functions, the same setup as in SIREN but
let ${\rm spline}(x)$ be trainable. For both MLP and LAN, the shape is
[2,128,128,128,128,128,1]. We train them with the Adam optimizer, batch size
4096, for 5000 steps with learning rate $10^{-3}$ and 5000 steps with learning
rate $10^{-4}$. As shown in Figure B.3, the LAN (orange) can achieve higher
PSNR than the MLP (blue) due to the LAN’s flexibility to fine tune activation
functions. We show that it is also possible to initialize a LAN from an MLP
and further fine tune the LAN (green) for better PSNR. We have chosen $G=5$ in
our experiments, so the additional parameter increase is roughly
$G/N=5/128\approx 4\%$ over the original parameters.

##  Appendix C Dependence on hyperparameters

We show the effects of hyperparamters on the $f(x,y)={\rm exp}({\rm sin}(\pi
x)+y^{2})$ case in Figure C.1. To get an interpretable graph, we want the
number of active activation functions to be as small (ideally 3) as possible.

  1. (1)

We need entropy penalty to reduce the number of active activation functions.
Without entropy penalty, there are many duplicate functions.

  2. (2)

Results can depend on random seeds. With some unlucky seed, the pruned network
could be larger than needed.

  3. (3)

The overall penalty strength $\lambda$ effectively controls the sparsity.

  4. (4)

The grid number $G$ also has a subtle effect on interpretability. When $G$ is
too small, because each one of activation function is not very expressive, the
network tends to use the ensembling strategy, making interpretation harder.

  5. (5)

The piecewise polynomial order $k$ only has a subtle effect on
interpretability. However, it behaves a bit like the random seeds which do not
display any visible pattern in this toy example.

Figure C.1: Effects of hyperparameters on interpretability results.

##  Appendix D Feynman KANs

We include more results on the Feynman dataset (Section 3.3). Figure D.1 shows
the pareto frontiers of KANs and MLPs for each Feynman dataset. Figure D.3 and
D.2 visualize minimal KANs (under the constraint test RMSE $<10^{-2}$) and
best KANs (with the lowest test RMSE loss) for each Feynman equation fitting
task.

Figure D.1: The Pareto Frontiers of KANs and MLPs for Feynman datasets. Figure
D.2: Best Feynman KANs Figure D.3: Minimal Feynman KANs

##  Appendix E Remark on grid size

For both PDE and regression tasks, when we choose the training data on uniform
grids, we witness a sudden increase in training loss (i.e., sudden drop in
performance) when the grid size is updated to a large level, comparable to the
different training points in one spatial direction. This could be due to
implementation of B-spline in higher dimensions and needs further
investigation.

##  Appendix F KANs for special functions

We include more results on the special function dataset (Section 3.2). Figure
F.2 and F.1 visualize minimal KANs (under the constraint test RMSE $<10^{-2}$)
and best KANs (with the lowest test RMSE loss) for each special function
fitting task.

Figure F.1: Best special KANs Figure F.2: Minimal special KANs