Université de Tunis El-Manar, Département de Mathématiques, 1091 Tunis,
Tunisia, & Shaqra University, Saudi Arabia<EMAIL_ADDRESS>Université de
Tunis El-Manar, Département de Mathématiques, 1091 Tunis, Tunisia, & Northern
Borders University, Saudi Arabia<EMAIL_ADDRESS>M.ROULEUX Aix
Marseille Univ, Université de Toulon, CNRS, CPT, Marseille, France
<EMAIL_ADDRESS>
# On semi-classical spectral series for an atom in a periodic polarized
electric field

A.IFA  H.LOUATI

###### Abstract

In this report we present preliminary results about the tunneling problem for
a magnetic Schrödinger operator. As a motivation we consider the 3-D time-
dependent Schrödinger operator $H(t)=-h^{2}\Delta+V+E(t)\cdot x$ where $V$ is
a radial potential and $E(t)$ a circularly polarized field with uniform
frequency $\omega$. The quantum monodromy operator (QMO) that takes the system
through a complete period $T=2\pi/\omega$, turns out to be unitarily
equivalent to $e^{iTP_{A}(x,hD_{x})/h}$, where $P_{A}(x,hD_{x}))$ identifies
with a magnetic Schrödinger operator. When $V$ is sufficiently confining,
$P_{A}(x,hD_{x}))$ presents a double magnetic well. Then we construct its
semi-classical ground state and examine the splitting between its two first
eigenvalues.

##  1 Introduction

The general motivation is to describe some (semi-classical) spectrum for the
quantum monodromy operator (QMO) associated with a periodic time dependent
self-adjoint $N$-body Schrödinger operator $H(t)$ on $L^{2}({\bf R}^{3N})$.
Recall QMO is the unitary operator that evolves the physical system through
one period. This spectrum is the same as for Floquet Hamiltonian
$\widehat{H}=D_{t}+H(t)$ on $L^{2}({\bf R}^{3N}_{x}\times{\bf R}_{t})$, see
e.g. [21]. Scattering theory for Floquet Hamiltonian leads to important
properties, such as local decay (in time) for the probability density. Bound
states are also of interest, e.g. regarding tunneling properties. A.Tip [23]
has investigated the case where $H(t)$ is the Hamiltonian for an atom with $N$
particles of charge $e_{j}$ and mass $m_{j}$ ($e_{0}=N$ being the charge of
the nucleus of mass $m_{0}$) subject to the circularly polarized periodic
electric field $E_{1}(t)=(\sin\omega t,-\cos\omega t,0)\in{\bf R}^{3}$. Let
also $V$ (rotationally invariant in ${\bf R}^{3N}$) be the sum of all Coulomb
potentials between the particles of charge $e_{j}$. Neglecting spin effects,
we have

$H_{1}(t)=\sum_{j=0}^{N-1}(2m_{j})^{-1}p_{j}^{2}+V(x)-\sum_{j=0}^{N-1}e_{j}x_{j}\cdot
E_{1}(t)$

Time-evolution $U_{1}(t,s)$ corresponding to $H_{1}(t)$ is well defined, and
QMO is $U_{1}(T+s,s)$ (independent of $s\in{\bf R}$). An equivalent
formulation can be given in terms of

$H_{2}(t)=\sum_{j=0}^{N-1}(2m_{j})^{-1}\bigl{(}p_{j}-e_{j}A_{2}(t)\bigr{)}+V(x)$

with $E_{1}(t)=-\partial_{t}A_{2}(t)$, and the corresponding time-evolutions
$U_{j}(t,s)$, $j=1,2$ are related by a unitary operator $X(t)$. One of the
main observations in [23] is that the group of unitary operators
$R(t)=e^{i\omega tL_{3}}$, $L_{3}$ being the vertical component of the angular
momentum (tensor product over the $N$ particles), takes $A_{2}(t)$ into a
constant magnetic potential $A_{0}=({1\over\omega},0,0)$, i.e.

$R(t)H_{2}(t)R(t)^{-1}=\sum_{j}(2m_{j})^{-1}(p_{j}-e_{j}A_{0})^{2}+V(x)$

and

$U_{2}(t,s)=e^{-i\omega tL_{3}}e^{i(t-s)P_{A}}e^{i\omega sL_{3}}$

where $P_{A}$ is the time-independent self-adjoint operator

$P_{A}=\sum_{j=0}^{N-1}(2m_{j})^{-1}(p_{j}-e_{j}A_{0})^{2}+V(x)-\omega L_{3}$

Quantum monodromy operator $U_{2}(s+T,s)$ over the period $T=2\pi/\omega$, is
of the form

$U_{2}(s+T,s)=e^{-i\omega sL_{3}}e^{iTP_{A}}e^{i\omega sL_{3}}$

This relates the (discrete) spectra of $U_{2}(s+T,s)$ and $P_{A}$ (either real
spectrum or resonances), in the sense that if $E$ is an eigenvalue of $P_{A}$,
then $F=e^{iTE}$ is an eigenvalue of $U_{2}(s+T,s)$. For simplicity we assume
henceforth there is only one heavy nucleus and one electron $x\in{\bf R}^{3}$,
and introduce a semi-classical parameter $h$ so that
$L_{3}=x_{1}hD_{x_{2}}-x_{2}hD_{x_{1}}$, and $P_{A}$ on $L^{2}({\bf R}^{3})$
takes the form $P_{A}(x,hD_{x})=(hD_{x}-\nu A_{0})^{2}+V(x)-\omega L_{3}$,
where $\nu$ is a coupling constant. In case $V$ is Coulomb potential, the
complex dilation theory of $P_{A}$ set up in [23] gives an insight into the
resonant spectrum of $P_{A}$. Here we still assume $V$ to be rotation
invariant, but allow for non Coulomb (confining) potentials, so that $P_{A}$
has discrete, real spectrum.

##  2 Hamiltonians

The classical Hamiltonian is

$p_{A}(x,\xi)=(\xi_{1}-{\nu\over\omega})^{2}+\xi_{2}^{2}+\xi_{3}^{2}+V(|x|)-\omega(x_{1}\xi_{2}-x_{2}\xi_{1})$
(1)

and we may rewrite $P_{A}$ as a magnetic Schrödinger operator. Namely, after
the affine change of coordinates
$x={2\nu\over\omega^{2}}y+(0,{2\nu\over\omega^{2}},0)$, and the substitution
$W(y)={4\nu^{2}\over\omega^{4}}\bigl{(}V(|x(y)|)-{\lambda^{2}\over\omega^{2}}(y_{1}^{2}+y_{2}^{2})\bigr{)}$,
we get $p_{A}(x,\xi)={\omega^{4}\over
4\nu^{2}}p^{\prime}_{A}(y,\eta)+{\nu^{2}\over\omega^{2}}$, where

$\displaystyle p^{\prime}_{A}$
$\displaystyle(y,\eta)=(\eta_{1}+{2\nu^{2}\over\omega^{3}}y_{2})^{2}+(\eta_{2}-{2\nu^{2}\over\omega^{3}}y_{1})^{2}+\eta_{3}^{2}$
(2) $\displaystyle+W(y)=(\eta-\omega^{\prime}A(y))^{2}+W(y)$

Note that the corresponding operator $P^{\prime}_{A}(y,hD_{y})$ commutes with
symmetry $y_{3}\mapsto-y_{3}$. General spectral properties of magnetic
Schrödinger operators are well-known, see e.g. [7], in particular when
$W(y)\to+\infty$ as $y\to\infty$, $P^{\prime}_{A}(y,hD_{y})$ has only discrete
spectrum.

Discrete semi-classical spectrum of $P_{A}$ near some energy level $E$ is
generally associated with local minima of the Hamiltonian. Among these minima
it is easy to find those which are the critical points of $r\mapsto V(r)$,
namely

$\rho_{0}^{\pm}=(x_{0},\xi_{0})^{\pm}=\bigl{(}(0,2\nu\omega^{-2},\pm\sqrt{r_{0}^{2}-4\nu^{2}\omega^{-4}}),0\bigr{)}$

with $V(r_{0})=E-\omega^{2}\nu^{-2}$. We will write as well
$\rho_{0}^{\pm}=(y_{0}^{\pm},\eta_{0}^{\pm})=(y_{0}^{\pm},0)$ in the new
coordinates $y$.

So the $y$-projection of the critical points $y_{0}^{\pm}$ of
$p^{\prime}_{A}(y,\eta)$ are located on the $y_{3}$-axis, symmetric with
respect to the $(y_{1},y_{2})$ plane. At such a critical point,
$\eta_{0}^{\pm}=0$.

For a suitable choice of parameters
$(r_{0},V^{\prime\prime}(r_{0}),\omega,\nu)$, it may happen that
$\rho_{0}^{\pm}$ is an elliptic point even if $r_{0}$ is not a minimum of $V$.
However, to meet hypotheses of [19], we assume that $E$ is the bottom of the
spectrum of $P_{A}$, and thus $r_{0}$ is a global minimum of $V$. To avoid
multiple tunneling, we assume also that there is no other critical points of
$p_{A}$ than $\rho_{0}^{\pm}$ at energy $E$.

Such elliptic fixed points $\rho^{\pm}_{0}$ of Hamiltonian $p_{A}$, will be
called the magnetic microlocal wells. We consider Dirichlet realization
$P_{A}^{M^{\pm}}(x,hD_{x})$ localized in some (large enough) neighborhoods
$M^{\pm}\subset{\bf R}^{3}$ of $x_{0}^{\pm}$ (by symmetry
$P_{A}^{M^{\pm}}(x,hD_{x})$ are unitarily equivalent) and study their semi-
classical spectrum in a $h$-neighborhood of $E$, see [9]. Note that the
spectral parameters for $P_{A}^{M^{\pm}}$ and ${P^{\prime}_{A}}^{M^{\pm}}$ are
related as follows:

$\displaystyle
P_{A}^{M^{\pm}}(x,hD_{x})u_{\pm}(x)=(E+\lambda_{A}(h))u_{\pm}(x)$ (3)
$\displaystyle\Longleftrightarrow\
{P^{\prime}_{A}}^{M^{\pm}}(y,hD_{y})u^{\prime}_{\pm}(y)=(E^{\prime}+\lambda^{\prime}_{A}(h))u^{\prime}_{\pm}(y)$

where $E^{\prime}={4\nu^{2}\over\omega^{4}}V(r_{0})$,
$\lambda^{\prime}_{A}(h)={4\nu^{2}\over\omega^{4}}\lambda_{A}(h)$, and
$u^{\prime}_{\pm}(y)=u_{\pm}(x)$ we will denote for short again by
$u_{\pm}(y)$. We know that the first eigenvalue $\lambda_{A}(h)$ of
$P_{A}^{M^{\pm}}(x,hD_{x})-E$ is non degenerate. By tunneling, this will gives
raise to splitted eigenvalues $E_{0}(h)<E_{1}(h)$ for $P_{A}(x,hD_{x})$
exponentially close to $\lambda_{A}(h)$. Thus our main goal consists in
solving the rather standard problems: (1) Find semi-classical asymptotics of
$\lambda_{A}(h)$, $h\to 0$. (2) Estimate the splitting $E_{1}(h)-E_{0}(h)$
(tunneling between $\rho_{0}^{\pm}$). Point (1) results directly from the
general microlocal arguments of [19] near $\rho_{0}^{\pm}$ applied to $P_{A}$,
that we recall here. Point (2) relies more specifically on the fact that
$P_{A}$ (or $P^{\prime}_{A}$) is a magnetic Schrödinger operator. Both use
complexification of time $t\mapsto it$ for Hamilton equations in the
“classically forbidden region”.

##  3 Microlocal approach: WKB expansions near the microlocal magnetic well

We work with $P_{A}$ rather than the rescaled operator $P^{\prime}_{A}$, and
since our constructions will hold in a small neighborhood of
$x_{0}=x_{0}^{+}$, $x_{0}\in M=M^{+}$ say, we shall write sometimes $P_{A}$
instead of $P_{A}^{M^{+}}(x,hD_{x})$. At an elliptic fixed point Floquet
exponents, i.e. eigenvalues of the fundamental matrix $F_{p_{A}}={\cal
J}\mathop{\rm Hess}\nolimits p_{A}(\rho_{0}^{\pm})$, are purely imaginary.
Their square $\mu^{\prime 2}$ are the roots of a 3:rd degree polynomial with
real coefficients. It turns out that we can find an open set of parameters
$(r_{0},V^{\prime\prime}(r_{0}),\omega,\nu)$ such that
$r_{0}\omega^{2}\nu^{-1}\geq 2$ and
${\mu^{\prime}}_{1}^{2}<{\mu^{\prime}}_{2}^{2}<{\mu^{\prime}}_{3}^{2}<0$. When
these conditions are met, applying a linear canonical transformation
$\kappa_{A}$, and setting the reference energy to $E^{\prime}=0$ to simplify
notations, which amounts to set $V(r_{0})=0$, $p_{A}$ takes the form, with
$\mu^{\prime}_{j}=i\mu_{j}$, $\mu_{j}>0$

$p_{A}(x,\xi)=\sum_{j=1}^{3}\mu_{j}(x_{j}^{2}+\xi_{j}^{2})/2+{\cal
O}(|x,\xi|^{3})$ (4)

in local coordinates $(x,\xi)$ vanishing at $\rho_{0}^{\pm}$ (but not the same
as the original coordinates of the problem). It is important to note that,
although $\kappa_{A}$ changes drastically the geometry of the phase-space, we
still get informations on $u^{\prime}_{\pm}(y)$, see [19], Remark 5.2 and
below. Namely, $\kappa_{A}$ does not generate additional caustics in the
region of interest.

Examine first WKB constructions and decay properties of the normalized quasi-
mode associated with $\lambda_{A}(h)$ by microlocal technics and the method of
complex deformations [9], [10], [7], [17]. We follow here [19].

Consider the global FBI transform (with $c(h)$ a normalization constant)

$Tu(x,\xi;h)=c(h)\int_{{\bf R}^{3}}e^{i(x-y)\xi/h-(x-y)^{2}/2h}u(y)\,dy$ (5)

that maps isometrically $L^{2}({\bf R}^{3})$ into $L^{2}({\bf R}^{6})$. It is
associated with the transformation $\kappa_{T}:T^{*}{\bf
R}^{3}\to\Lambda=\\{\zeta=i\mathop{\rm Im}\nolimits z\\}$,
$(x,\xi)\mapsto(z,\zeta)=(x-i\xi,-i\xi)$, such that $-\mathop{\rm
Im}\nolimits(dz\wedge d\zeta)=dx\wedge\,d\xi$, $\mathop{\rm
Re}\nolimits(dz\wedge d\zeta)=0$ on $\Lambda$. The pluri-subharmonic (pl.s.h)
weight defining $\Lambda$, is $-\phi_{0}(z,\overline{z})=(\mathop{\rm
Im}\nolimits z)^{2}$, namely $\Lambda=\\{\zeta={\partial\phi_{0}\over\partial
z}\\}$. Transformation $T$ intertwines $P_{A}$ with
$\widetilde{P}_{A}(x,\xi,hD_{x},hD_{\xi})$ whose Weyl symbol is given by
$\widetilde{p}_{A}(x,\xi;x^{*},\xi^{*})=p_{A}^{w}(x-\xi^{*},x^{*})$, so that
the eigenvalue equation becomes, with $u_{T}=Tu$ and $u=u_{+}$ as in (3)

$(\widetilde{P}_{A}(x,\xi,hD_{x},hD_{\xi})-\lambda_{A}(h))u_{T}(x,\xi;h)\sim
0$ (6)

which we will solve by WKB method, by looking for $u_{T}$ of the form
$u_{T}(z;h)\sim e^{-\xi^{2}/2h-\varphi(z)/h}a(z;h)$. The reduced operator
$\Psi(x,\xi,hD_{x},hD_{\xi})=e^{\xi^{2}/2h+\varphi(z)/h}\widetilde{P}_{A}(x,\xi,hD_{x},hD_{\xi})e^{-\xi^{2}/2h-\varphi(z)/h}$
acting (formally) on symbols $a(z;h)$ is indeed a $h$-PDO $\Psi(z,hD_{z};h)$
with symbol
$\sigma_{\Psi}(z,\zeta;h)\sim\sigma_{0}(z,\zeta)+h\sigma_{1}(z,\zeta)+\cdots$,
and $\sigma_{0}(z,\zeta)=-p_{A}(z-\zeta,i\zeta)$.

###### Remark 1

We could also use the standard unitary Bargman transform [22]
$T_{0}:L^{2}({\bf R}^{3})\to H_{\Phi}({\bf C}^{3})$, space of holomorphic
functions square integrable with respect to the weight $\Phi(z)=|x|^{2}/4$. It
is somewhat more natural in this context, but we found it convenient to stick
to notations of [19], [18].

Performing another “absolute” metaplectic complex transformation $T_{2}$
(independent of all parameters), associated with a complex canonical
transformation $\kappa_{2}$, we can arrange so that $\sigma_{0}$ has the
quadratic approximation
$\sigma_{02}(z,\zeta)=\sum_{j=1}^{3}2\mu_{j}z_{j}\zeta_{j}$ near the magnetic
microlocal well. We still denote by $u_{T}$ the corresponding quasi-mode. In
fact $T_{2}$ could be avoided if we replace $T$ by $T_{0}$.

We solve Hamilton-Jacobi (HJ) equation
$\sigma_{02}(z,\partial_{z}\varphi_{0}(z)=0$ by $\varphi_{0}(z)={1\over
4}z^{2}$. This gives the complex “outgoing” Lagrangian manifold
$\Lambda^{0}_{+}=\\{\zeta=\partial_{z}\varphi_{0}(z)\\}$, and also the
“incoming” one $\Lambda^{0}_{-}=\\{\zeta=-\partial_{z}\varphi_{0}(z)\\}$. By
the stable-unstable manifold Theorem this carries to $\sigma_{0}(z,\zeta)$,
thus we can solve HJ equation with holomorphic phase function
$\varphi(z)=\varphi_{0}(z)+{\cal O}(|z|^{3})$ defining $\Lambda_{\pm}$.
Continuing this way, we can also solve the transport equations with an
analytic symbol $a(z;h)$. Altogether we get a WKB solution of
$(\Psi(z,hD_{z};h)-\lambda_{A}(h))a(z;h)\sim 0$ microlocally near
$\rho_{0}=\rho_{0}^{+}$, such that

$u_{T}(z,h)\sim e^{-\xi^{2}/2h-\varphi(z)/h}a(z;h)$ (7)

Here $\sim$ means modulo a remainder term ${\cal O}(e^{-1/Ch})$. Applying
classical variational principles, this is justified by Agmon type estimates,
which say how close the microlocal solution approaches the actual
eigenfunction. To this end, we use the method of non characteristic
deformations. Namely in some neighborhood $\Omega$ of (0,0) where microlocal
solution (7) holds (i.e. before the occurence of focal points on
$\Lambda_{\pm}$) there is a deformation of $-\phi_{0}$ to a family of pl.s.h.
weights $-\phi_{t}(z,\overline{z})$, such that the R-Lagrangian manifold
$\Lambda_{t}=\exp tH_{q}(\Lambda)$ is transverse to $\\{z=0\\}$ at (0,0), and
hence of the form $\zeta={\partial\phi_{t}\over\partial z}$. We check that
$\phi_{t}(z,\overline{z})\to 2\mathop{\rm Re}\nolimits\varphi(z)$ as
$t\to\infty$. Moreover $\sigma_{0}$ remains elliptic outside (0,0) along these
deformations, in the sense $-\sigma_{0}(z,\zeta)|_{\Lambda_{t}}\geq\mathop{\rm
Const.}\nolimits(|z|^{2}+|\zeta|^{2})$ for all $t$, and $\phi_{t}$ satisfies
the eikonal equation
$\partial_{t}\phi_{t}(z,\overline{z})=-2\sigma_{0}(z,\partial_{t}\phi_{t})$.
We then modify $\phi_{t}$ to another weight $\psi_{t}$ such that
$\psi_{t}<2\mathop{\rm Re}\nolimits\varphi$ near $\partial\Omega$, where WKB
constructions of type (7) fail to exist, and get the following Agmon estimate:
If $u$ is the 1:st normalized eigenfunction of $P_{A}^{M}$ then for any
compact set $K\subset\Omega$, there exists
$\mathop{\rm\varepsilon}\nolimits>0$ and
$h_{\mathop{\rm\varepsilon}\nolimits}>0$, such that uniformly for $z=x-i\xi\in
K$, $0<h<h_{\mathop{\rm\varepsilon}\nolimits}$ we have

$\displaystyle e$
$\displaystyle{}^{\xi^{2}/2h+\varphi(x-i\xi)/h}(Tu_{+}(x,\xi;h)-u_{T}(x,\xi;h))=$
(8) $\displaystyle{\cal O}(e^{-\mathop{\rm\varepsilon}\nolimits/h})$

Altogether, this gives the asymptotics for the first eigenvalue
$\lambda_{A}(h)$ for $P_{A}^{M^{+}}(x,hD_{x})$

$\lambda_{A}(h)=h\mathop{\rm Tr}\nolimits^{+}(F_{p_{A}})+{\cal O}(h^{2})$ (9)

where $\mathop{\rm Tr}\nolimits^{+}(F_{p_{A}})=\sum_{j=1}^{3}\mu_{j}$. This
gives also the asymptotics of $Tu_{+}$ in $\Omega\subset M$.

The main drawback is the poor control of the neighborhood of $x_{0}$ where
this asymptotics holds for $u=T^{-1}u_{T}$ (left inverse), because
$\kappa_{A}$ is not merely implied by a change of variables in the $x$-space.
Nevertheless since $\kappa_{T}^{-1}(\widetilde{\Lambda}_{+})$,
$\widetilde{\Lambda}_{+}=\\{\zeta=i\partial_{z}\varphi(z)\\}$ is a strictly
positive Lagrangian manifold at (0,0) (also after applying $\kappa_{2}$), the
same holds for $\kappa_{A}^{-1}\circ\kappa_{T}^{-1}(\widetilde{\Lambda}_{+})$
(in the initial canonical variables), and so $\Lambda_{+}$ has no focal points
near $\rho_{0}$. So Agmon estimate (8) again justifies the asymptotics of $u$
of $P_{A}^{M}(x,hD_{x})$ in some (real) neighborhood $\Omega_{R}$ of $x_{0}$.

Of course by symmetry $y_{3}\mapsto-y_{3}$ everything carries to $M^{-}$. To
proceed we need to extend WKB constructions, when $\omega^{\prime}$ is small
enough, near minimal geodesics for Agmon metric associated with the
Hamiltonian without a magnetic field, .

##  4 Geometry and quantization of magnetic Hamiltonians

We work here with the rescaled operator
$P^{\prime}_{A}(y,hD_{y})=(P^{\prime}_{A})^{M}(y,hD_{y})$ localized in
$M=M^{+}$. In Classical Mechanics the generalized momentum $\eta$ of a
particle is related to its velocity by $v=\eta-A(y)$. The magnetic symplectic
2-form is

$\sigma_{A}=dv\wedge dy=d\eta\wedge dy+dA(y)=\sigma+dA(y)$

Consider first the kinetic term $K_{0}(y,\eta)=|v|^{2}=(\eta-A(y))^{2}$. We
know [12], [20] that if $dA(y)\neq 0$, then
$\Sigma=\\{K_{0}=0\\}=\\{dK_{0}=0\\}$ is a 3-D submanifold of $T^{*}{\bf
R}^{3}$, and $\Sigma\cap\Sigma^{\sigma}$ ($\Sigma^{\sigma}$ denotes orthogonal
complement with respect to the symplectic 2-form $\sigma$) is the Hamilton
vector field for $\sum_{j=1}^{3}F_{j}(y)(\eta_{j}-A_{j}(y)$ where $F_{j}(y)$
are the components of the vector intensity of the magnetic field $B(y)$. In
the present case $\Sigma\cap\Sigma^{\sigma}={\bf R}{\partial\over\partial
y_{3}}$. (Note that in the 2-D case, $\Sigma$ is just a 2-D symplectic
manifold). Many examples of integrable magnetic Hamiltonians are provided in
[13].

As in the case of Schrödinger operator without a magnetic field, we may define
complex branches of the energy surface by considering imaginary times. They
glue along $\Sigma$. For Hamiltonian (1) with a potential $V$ sufficiently
confining, the “magnetic classically allowed region” (MCAR) is a domain in
${\bf R}^{3}$ with 2 connected components in $\pm y_{3}>0$ bounded by the
closed 2-D manifolds defined by
$\pi_{y}\bigl{(}\Sigma\cap\\{p^{\prime}_{A}(y,\eta)=E^{\prime}\\}\bigr{)}$,
where $\pi_{y}:T^{*}{\bf R}^{3}\to{\bf R}^{3}$ is the natural projection, and
the “magnetic classically forbidden region” (MCFR) its complement. For small
$\omega^{\prime}$, the MCAR can be viewed as a deformation of the “classically
allowed region” (CAR) for the Hamiltonian
$P^{\prime}_{0}(y,hD_{y})=(hD_{y})^{2}+W(y)$ without a magnetic field.

We stress that in the MCFR we make the substitution $v\mapsto iv$ instead of
$\xi\mapsto i\xi$ as in [10]. By this transformation the Hamiltonian remains
real and gauge invariant. This is consistent with the following example of
operator on $L^{2}({\bf R}^{2})$

$H(y,hD_{y})=(hD_{y_{1}})^{2}+(hD_{y_{2}}-by_{1})^{2}+V_{1}(y_{1})+\omega^{2}y_{2}^{2}$
(10)

which is unitarily equivalent (after partial Fourier transform with respect to
$x_{2}$) to the Schrödinger operator without a magnetic field
$H^{\prime}(x,hD_{x})=(hD_{x_{1}})^{2}+(hD_{x_{2}})^{2}+(\omega
x_{2}-bx_{1})^{2}+V_{1}(x_{1})$ (see [2], [6]). It turns out that our
computations lead (up to the present accuracy) to the same quantities as [10],
but without factor $i$ at some places, see e.g. Proposition 1 below.

###### Remark 2

Quantization of the kinetic term with a linear $A(y)=(-y_{1},0,0)$ (Landau
gauge) was considered first in [14], p.496, leading to Landau levels with
infinite degeneracy. In this case, the $y$-projection of classical
trajectories consist in a family of helices, with arbitrary center. The
generalized wave-functions are expressed in term of a family of Hermite
functions oscillating in the $y_{1}$-direction around the center. This gives a
hint at understanding the case of magnetic Schrödinger operator (MSO)
$P^{\prime}_{A}(y,hD_{y})$ : degeneracy is lifted by the potential, that fixes
somehow the center of the helix.

Notice that the potential well $U_{E^{\prime}(h)}$ for
$p^{\prime}_{0}(y,\eta)=\eta^{2}+W(y)$ at energy
$E^{\prime}(h)={4\nu^{2}\over\omega^{4}}(V(r_{0})+\lambda_{A}(h))$ with 2
connected components $U\pm_{E^{\prime}(h)}$ is given by

$\displaystyle U_{E^{\prime}(h)}$
$\displaystyle{}^{\pm}=\\{y:V(|x(y)|)-{\nu^{2}\over\omega^{2}}(y_{1}^{2}+y_{2}^{2})\leq$
(11) $\displaystyle V(r_{0})+\lambda_{A}(h),\ \pm y_{3}>0\\}$

Thus $y_{0}^{\pm}$ (the “center” of each well) verifies $y_{0}^{\pm}\in
U_{E^{\prime}}^{\pm}$, and $\partial U_{E^{\prime}(h)}^{\pm}\subset M^{\pm}$
if $r\mapsto V(r)$ grows fast enough at infinity and near 0, so that in
particular $W|_{y_{3}=0}>0$. Under these conditions, by the hyperplane
symmetry, $p^{\prime}_{0}$ has two potential wells $U^{\pm}_{E^{\prime}(h)}$
localized near $y_{0}=y_{0}^{\pm}$. Conditions on Floquet exponents (see
Sect.3) ensure that $W$ has nondegenerate minima at $y_{0}^{\pm}$.

Taking Weyl quantization of
$p^{\prime}_{A}(x,\xi)=(\eta-\omega^{\prime}A(y))^{2}+W(y)$, where
$A(y)=(-y_{2},y_{1},0)$ (symmetric gauge) and
$\omega^{\prime}={2\nu^{2}\over\omega^{3}}$, we get the magnetic Schrödinger
Hamiltonian

$\displaystyle P^{\prime}_{A}$ $\displaystyle(y,hD_{y})u(y;h)=(2\pi
h)^{-3}\int_{{\bf R}^{6}}$ (12) $\displaystyle
e^{i(y-z)\eta/h}p^{\prime}_{A}({y+z\over 2},\eta)u(z)\,dz\,d\eta$

and by a change of variables

$\displaystyle P^{\prime}_{A}$ $\displaystyle(y,hD_{y})u(y;h)=(2\pi
h)^{-3}\int_{{\bf R}^{6}}$ (13) $\displaystyle
e^{i(y-z)\bigl{(}\eta+A\bigl{(}{y+z\over
2}\bigr{)}\bigr{)}/h}p^{\prime}_{0}({y+z\over 2},\xi)u(z)\,dz\,d\eta$

###### Remark 3

For non linear magnetic potentials $A(y)$ we need to modify this quantization,
since it is no longer gauge invariant, see [15]. There are far-reaching
generalisation of such $h$-PDO’s, in particular Berezin-Toeplitz operators,
see [3], [4], [11].

##  5 WKB constructions for $P^{\prime}_{0}$ and $P^{\prime}_{A}$

We will somehow relate the quasi-modes of $P^{\prime}_{0}$ and
$P^{\prime}_{A}$ in $M^{\pm}$, outside $U^{\pm}_{E^{\prime}(h)}$. Since we are
looking for rather rough estimates on the eigenvalue splitting, it is
sufficient here to search for WKB solutions of
$\bigl{(}P^{\prime}_{0}(y,hD_{y})-\lambda^{\prime}_{A}(h)\bigr{)}w(y;h)\sim 0$
in the classically forbidden region (CFR) $W(y)\geq E^{\prime}(h)$, although
$P^{\prime}_{0}$ and $P^{\prime}_{A}$ do not have the same ground state, see
e.g. the discussion in [7], Sect.7.2.

First we recall some known facts on tunneling for Schrödinger operators
without a magnetic field. Let $S_{E^{\prime}(h)}^{+}(y)$ be Agmon distance
from $U^{+}_{E^{\prime}(h)}$ to $y$ for the Riemannian conformal metric
$\bigl{(}W(y)-E^{\prime}(h))\bigr{)}_{+}\,dy^{2}$,
$E^{\prime}(h)={4\nu^{2}\over\omega^{4}}(V(r_{0})+\lambda_{A}(h))$ which
vanishes on $U_{E^{\prime}}$. We know that the exponential decay of $w$ from
the well $U^{+}_{E^{\prime}}$, say, is given roughly by
$e^{-S_{E^{\prime}}^{+}(y)/h}$, see [9], [17]. So we have to compute $w$ near
minimal geodesics $\gamma_{E^{\prime}(h)}$ between $\partial
U^{\pm}_{E^{\prime}(h)}$. Such (finitely many) minimal geodesics are also
called librations [2], [6], [1]. Within the required accuracy on tunneling
rates, we could again replace $E^{\prime}(h)$ by $E^{\prime}$, which amounts
to replace the librations by the instanton between
$U^{\pm}_{E^{\prime}}=\\{y_{0}^{\pm}\\}$.

Changing $t$ to $it$ we may assume that in $W(y)\geq E^{\prime}(h)$ (CFR)
$P^{\prime}_{0}-E^{\prime}(h)$ takes the form
$Q^{\prime}_{0}(y,hD_{y})+E^{\prime}(h)=(hD_{y})^{2}-W(y)+E^{\prime}(h)$. Then
the WKB solutions of
$(Q^{\prime}_{0}(y,hD_{y})+E^{\prime})w(y,h)=\lambda_{A}(h)w(y,h)$ are given
by

$w(y;h)=e^{-\phi(y,E^{\prime})/h}b(y,E^{\prime};h)$ (14)

In particular the phase function $\phi$ solves (locally) Hamilton-Jacobi
equation $(\nabla\phi(y,E^{\prime}))^{2}-W(y)+E^{\prime}=0$ and the amplitude
$b(y,E^{\prime};h)=b_{0}(y,E^{\prime};h)+hb_{1}(y,E^{\prime};h)+\cdots$ the
transport equations. These WKB expansions can be justified by Agmon estimates
near the minimal geodesics $\gamma_{E^{\prime}}$ as in [17].

Let now
$Q^{\prime}_{A}(y,hD_{y})+E^{\prime}(h)=(hD_{y}-\omega^{\prime}A(y))^{2}-W(y)+E^{\prime}(h)$
be the corresponding Hamiltonian in MCFR. Recall that MCFR (with the magnetic
field of strength $\omega^{\prime}$) is a deformation ${\cal
O}(\omega^{\prime})$ of the CFR (without the magnetic field).

To solve $(Q^{\prime}_{A}(y,hD_{y})+E^{\prime}(h))u(y;h)\sim 0$, we try

$\displaystyle
u(y;h)=e^{-\psi(y,E^{\prime})/h}b(y,E^{\prime};h)w(y,E^{\prime};h)=$ (15)
$\displaystyle
e^{-(\phi(y,E^{\prime})+\psi(y;E^{\prime}))/h}b(y,E^{\prime};h)c(y,E^{\prime};h)$

and replace $E^{\prime}(h)$ by $E^{\prime}$ as stated above. We find (omitting
parameter $E^{\prime}(h)$), using (12) and (13)

$\displaystyle(Q^{\prime}_{A}(y,hD_{y})+E^{\prime})u(y;h)=(2\pi
h)^{-3}\int\int$ (16)
$\displaystyle\exp\bigl{[}i\bigl{(}(y-z)(\eta+\omega^{\prime}A({y+z\over
2}))+\phi(z)+\psi(z)\bigr{)}/h\bigr{]}$
$\displaystyle\bigl{(}q^{\prime}_{0}({y+z\over
2},\eta)+E^{\prime}\bigr{)}b(z,E^{\prime};h)c(z,E^{\prime};h)\,dz\,d\eta$

(we have identified the operator with its Weyl symbol). Applying asymptotic
stationary phase yields eikonal equation

$(-\omega^{\prime}A(y)+\phi^{\prime}(y)+\psi^{\prime}(y))^{2}-W(y)+E^{\prime}=0$
(17)

which is Hamilton-Jacobi equation for Hamiltonian
$(\eta-\omega^{\prime}A(y)+\phi^{\prime}(y))^{2}+W(y)+E^{\prime}$. By a
stability argument (17) has again a unique solution near
$\gamma_{E^{\prime}}$, and can be solved perturbatively in $\omega^{\prime}>0$
small enough.

Namely, look for $\psi(y)=\omega^{\prime}\psi_{0}(y)+\omega^{\prime
2}\widetilde{\psi}_{1}(y,\omega^{\prime})$. Substituting into (17) we get at
first order in $\omega^{\prime}$:
$2\nabla\phi(y)\nabla\psi_{0}(y)=2\langle\nabla\phi(y),A(y)\rangle$ which is a
transport equation along the integral curves of Hamilton vector field
$H_{Q^{\prime}_{0}}$. Using Hamilton Eq. for $Q^{\prime}_{0}$ it can be
written as ${d\over
dt}\psi_{0}(y(t))=2\bigl{(}-y_{2}{\partial\phi\over\partial
y_{1}}+y_{1}{\partial\phi\over\partial y_{2}}\bigr{)}$ or in cylindrical
coordinates $y_{1}=r\cos\theta,y_{2}=r\sin\theta,y_{3}=y_{3}$, ${d\over
dt}\psi_{0}(x(t))=2{\partial\phi\over\partial\theta}(x(t))$ which can be
integrated as (see also [10], Eq.$(2.27)_{1}$) $\psi_{0}(t)=\psi_{0}|_{{\cal
H}_{*}}+2\int_{0}^{t}{\partial\phi\over\partial\theta}(x(s))\,ds$. Higher
order approximations in $\omega^{\prime}$ are obtained similarly. In
particular setting $\widetilde{\psi}_{1}=\psi_{1}+\omega\widetilde{\psi}_{2}$,
we find

$\displaystyle 2$
$\displaystyle\nabla\phi(y)\nabla\psi^{\prime}_{1}(y)=-(\omega^{\prime}y)^{2}-{\psi^{\prime}_{0}}^{2}+$
$\displaystyle 2\langle\
A(y),\psi^{\prime}_{0}\rangle=-|\psi^{\prime}_{0}-A(y)|^{2}$

or (see also Eq. $(2.27)_{2}$ and Lemma 3.9 of [10]), ${d\over
dt}\psi_{1}(x(t))=-\int|\psi^{\prime}_{0}(x(s))-A(x(s))|^{2}\,ds$ The same
type of arguments holds for transport equations defining $c(y,E^{\prime};h)$.

Note that in dimension 1, the leading order term of the symbol
$b(y,E^{\prime};h)c(y,E^{\prime};h)$ would assume the familiar form
$b(y,E^{\prime};h)c(y,E^{\prime};h)=\mathop{\rm
Const.}\nolimits\exp\bigl{[}-{1\over
2}\int^{y}(W(z)-E^{\prime})^{-1/2}\,dz\bigr{]}$, and we expect this formula to
hold in the 3-D case as well.

###### Remark 4

The computation above is based on representation (13) in the real domain, can
actually be carried over MCFR where the phase becomes complex (possibly purely
imaginary), if we choose suitable branches of solutions with positive
imaginary part. This can e.g. be achieved after performing a FBI transform of
type (8) which allows for complex phases.

This eventually gives $u=u_{\mathop{\rm WKB}\nolimits}$ outside some
$\omega^{\prime}$-neighborhood of CAR $\\{y:W(y)\leq E^{\prime}(h)\\}$ of the
form
$\widetilde{\Omega}^{+}_{\omega^{\prime}}\cup\widetilde{\Omega}^{-}_{\omega^{\prime}}$.

In fact CFR and MCFR may differ by ${\cal O}(\omega^{\prime})$. To ensure that
our WKB solutions overlap with the microlocal solutions obtained in Sect.3, we
need to assume that representation (7) of the actual eigenfunction holds in a
sufficiently large neighborhood (of size ${\cal O}(\omega^{\prime})$) of the
CAR $\\{y:W(y)\leq E^{\prime}\\}$ (without the magnetic field). This requires
that we have a good control, sufficiently far away the microlocal wells, of
the pl.s.h. weights $-\phi_{t}$ constructed in Sect.3. It should be
instructive to consider the model (10) from this viewpoint.

Let us summarize our constructions in the following:

###### Proposition 1

: Assume
$\widetilde{\Omega}^{+}_{\omega^{\prime}}\cup\widetilde{\Omega}^{-}_{\omega^{\prime}}\subset\Omega^{+}_{R}\cup\Omega^{-}_{R}$,
where $\Omega^{\pm}_{R}$ have been defined at the end of Sect.2. Then there
are WKB solutions $u_{\mathop{\rm WKB}\nolimits}$ as in (15) of
$(P_{A}(x,hD_{x})-E-\lambda_{A}(h))u_{\mathop{\rm WKB}\nolimits}\sim 0$
extending the quasi-modes $u^{\pm}$ of $P_{A}^{M^{\pm}}$ constructed in (7)
along the (finite set) of minimal geodesics $\gamma_{E^{\prime}}$ between
$\partial U^{\pm}_{E^{\prime}(h)}$.

As in the case of Schrödinger operator without a magnetic field, we can
control the decay of eigenfunctions outside $U^{\pm}_{E^{\prime}}$ by Agmon
estimates. We start with a rough weighted energy estimate. Let ${\cal
E}_{a}=\\{y:d_{E^{\prime}}(U_{E^{\prime}}^{+},y)+d_{E^{\prime}}(U_{E^{\prime}}^{-},y)\leq
S_{E^{\prime}}+a\\}\subset M^{+}\cup M^{-}$ for suitable $a>0$, and
$\widetilde{\Omega}\subset{\bf R}^{3}$ large enough with $C^{2}$ boundary,
such that $U^{+}_{E^{\prime}}\subset\widetilde{\Omega}$,
$U^{-}_{E^{\prime}}\cap\widetilde{\Omega}=\emptyset$,
$\Gamma=\partial\Omega\cap{\cal E}_{a}\subset M^{+}\cap M^{-}$, see [7],p.43.
Let also $\Phi$ be a Lipschitz function on $\partial\widetilde{\Omega}$. Then
for all $u\in C^{2}(\overline{\Omega};{\bf C})$,
$u|_{\partial\widetilde{\Omega}}=0$, we recall from [7],Prop.7.2.25

$\displaystyle\int_{\widetilde{\Omega}}|(hD_{x}-A(x))(e^{\Phi/h}u)|^{2}\,dx+$
(18)
$\displaystyle\int_{\widetilde{\Omega}}(W(x)-E^{\prime}-|\Phi^{\prime}(x)|^{2})e^{2\Phi/h}|u|^{2}\,dx$
$\displaystyle=\mathop{\rm
Re}\nolimits\int_{\widetilde{\Omega}}e^{2\Phi/h}(P^{\prime}_{A}-E^{\prime})u(x)\overline{u}(x)\,dx$

The weight $\Phi$ should of course be related with $\phi+\psi$, but the term
$(W-E^{\prime}-|\Phi^{\prime}(x)|^{2})$ in the second integral refers rather
to the eikonal equation verified by $\phi$ alone, so instead of the decay
observed in the WKB solution, we end up with this we would get in case
$A(x)=0$. So sharper estimates would require either to assume analyticity of
the potential as in [10], or to change formula (18), to a formula verified by
the ratio $\phi_{A}=u_{A}/u_{0}$ of ground states with and without a magnetic
field. Note that in case $\lambda_{A}(h)=\lambda_{0}(h)$, which is achieved in
particular (Bohm-Aharonov vector potential) when $\mathop{\rm rot}\nolimits
A=0$ in $\widetilde{\Omega}$, necessarily non simply connected, and
$\int_{\gamma}A(x)\,dx\in 2\pi h{\bf Z}$ for all closed path
$\gamma\subset\widetilde{\Omega}$, then $(hD_{x}-A(x))\phi_{A}=0$. This is due
to an identity by R.Lavine and M.O’Carroll, see [7],p.93 and references
therein, and leads in particular that $\phi_{A}$ is just a ($x$-dependent)
phase factor. So in this case there would be no additionnal decay due to the
magnetic field.

##  6 The gap formula and estimate of the splitting

.

As in the case of Schrödinger operator without a magnetic field, we relate the
splitting to an integral of $u_{+}{\partial u_{-}\over\partial y_{3}}$ over
$\Gamma\subset\\{y_{3}=0\\}$ that bissects the classically forbidden region
near the minimal geodesics. The gap formula is given by the interaction matrix
using Stokes formula with the magnetic potential

$\displaystyle w_{+-}=h^{2}\int_{\Gamma}\bigl{(}\overline{u_{+}}{\partial
u_{-}\over\partial y_{3}}-u_{-}{\partial\overline{u_{+}}\over\partial
y_{3}}\bigr{)}\,dS(y)+$ (19) $\displaystyle
h\int_{\Gamma}\bigl{(}\overline{u_{+}}(y)\langle A(y),{\partial\over\partial
y_{3}}\rangle u_{-}(y)$ $\displaystyle-u_{-}(y)\langle
A(y),{\partial\over\partial y_{3}}\rangle u_{+}(y)\bigr{)}\,dS(y)$

The second term vanish since $A_{3}(y)=0$, hence (19) is identical to the gap
formula without a magnetic field. This altogether with (18) yields an estimate
for the splitting of the form

$e^{-\bigl{(}S_{E^{\prime}}+{\cal O}(\omega^{\prime})\bigr{)}/h}\leq
E_{1}(h)-E_{0}(h)\leq e^{-\bigl{(}S_{E^{\prime}}-{\cal
O}(\omega^{\prime})\bigr{)}/h}$ (20)

but getting a sharper exponent requires to modify (18) as suggested above.
Precise Agmon estimates also depend on the number of these geodesics, and on
the structure of the eigenfunction $u_{A}$ of $P_{A}(y,hD_{y})$ near $\partial
U^{\pm}_{E^{\prime}(h)}$, obtained in Sect.3.

## Acknowledgements

We thank B.Helffer for interesting discussions on Agmon estimates in the
presence of a magnetic field, and a referee for some useful references.

## References

  * [1] Anikin, A. Rouleux, M. Multidimensional tunneling between potential wells at non degenerate minima. Proceedings Days of Diffraction 2014, p. 17-22. Saint-Petersburg.
  * [2] Brüning, J., Dobrokhotov, S., Nekrasov, R. 2013, Splitting of lower energy levels in a quantum double well in a magnetic field and tunneling of wave packets in Nanowires. Theor. Math Phys 175, p.620–636.
  * [3] Charles, L. 2000, Aspects semi-classiques de la quantification géometrique. Thèse, Université Paris IX-Dauphine.
  * [4] Charles, L. 2006, Symbolic calculus for Toeplitz operators with half-form. J. Symplectic Geometry, Vol. 4(2), pp. 171-198.
  * [5] Cycon, H. Froese, R., Kirsch, W, Simon, B. 1987, Schrödinger Operators. Springer.
  * [6] Dobrokhotov, S., Anikin, A. 2014, Tunneling, Librations and Normal Forms in a Quantum Double Well with a Magnetic Field. in Nonlinear Physical Problems, Spectral Analysis, Stability and Bifurcations, Oleg N.Kirillov, Dmitry E. Pelinovsky, Eds. Series Editor Noel Challamel, John Wiley & Sons, pp.85–108.
  * [7] Helffer, B. 1988, Semi-classical Analysis for the Schrödinger and applications. Lecture Notes in Math, Vol. 1336, Springer.
  * [8] Helffer, B. 1994, On Spectral Theory for Schrödinger Operators with Magnetic Potentials. Adv. Studies in Pure Math. 23, 1994, Spectral and Scattering Theory and Applications, pp.113-141
  * [9] Helffer, B., Sjöstrand, J., 1984, Multiple wells in the semi-classical limit I, Comm. Part. Diff. Eqn., Vol. 9(4) pp. 337-408.
  * [10] Helffer, B., Sjöstrand, J., 1987, Effet tunnel pour l’équation de Schrödinger avec champ magnétique. Ann. Scuola Norm. Sup. Pisa Cl. Sci. 4(14), No.4, pp. 625–657. 1987\.
  * [11] Hitrik, M., Asymptotic Bergman kernels in the analytic case, this Conference.
  * [12] Ivrii, V., 2019, Microlocal Analysis, Sharp Spectral Asymptotics and Applications, Vol. I-V, Springer.
  * [13] Ivrii, V., Magnetic Schrodinger operators: Geometry, Classical and Quantum Dynamics and Spectral Asymptotics. Séminaire EDP (Polytechnique) (2006-2007), Exposé No. 16, 23 p.
  * [14] Landau, L. Lifschitz, E. 1967, Mécanique Quantique. Mir, Moscou.
  * [15] Mantoiu, M. Purice, R. 2004, The Magnetic Weyl Calculus. J. Math. Phys, Vol. 1336, pp.1394–1417.
  * [16] Martinez, A., 1987, Estimations de l’effet tunnel pour le double puits I. J. Math Pures et Appli., Vol. 66, pp.195–215
  * [17] Martinez, A., 1988, Estimations de l’effet tunnel pour le double puits II. Etats hautement excités.Bull. Soc. Math. France, Vol. 116(2) pp.199–219.
  * [18] Martinez, A., 2002, An Introduction to Semiclassical and Microlocal Analysis. Springer.
  * [19] Martinez, A. Sordoni, V., 1999. Microlocal WKB expansions. J.Funct.Anal. Vol. 168, pp.380–402.
  * [20] Raymond, N. Vu Ngoc, S., 2015, Geometry and Spectrum in 2D Magnetic Wells. Annales Institut Fourier Vol. 65(1), pp.137–169.
  * [21] Schach-Möller, J., 2000, 2-body short-range systems in a time-periodic electric field. Duke Math.J., Vol. 105(1).
  * [22] Sjöstrand, J., 1982, Singularités analytiques microlocales, Astérisque Vol. 95, Soc. Math. France.
  * [23] Tip, A, 1983, Atoms in circularly polarized fields: the dilation-analytic approach. J.Phys. A Math. Gen. Phys. Vol. 16, pp.3237–3259.


# Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation

Shuangrui Ding1,3 Peisen Zhao2 Xiaopeng Zhang2 Rui Qian3 Hongkai Xiong1 Qi
Tian2
1Shanghai Jiao Tong University 2Huawei Cloud 3The Chinese University of Hong
Kong
{dsr1212<EMAIL_ADDRESS><EMAIL_ADDRESS>
{pszhao93<EMAIL_ADDRESS><EMAIL_ADDRESS>Work done during an
internship at Huawei Cloud. Corresponding author. Email<EMAIL_ADDRESS>
###### Abstract

Transformers have become the primary backbone of the computer vision community
due to their impressive performance. However, the unfriendly computation cost
impedes their potential in the video recognition domain. To optimize the
speed-accuracy trade-off, we propose Semantic-aware Temporal Accumulation
score (STA) to prune spatio-temporal tokens integrally. STA score considers
two critical factors: temporal redundancy and semantic importance. The former
depicts a specific region based on whether it is a new occurrence or a seen
entity by aggregating token-to-token similarity in consecutive frames while
the latter evaluates each token based on its contribution to the overall
prediction. As a result, tokens with higher scores of STA carry more temporal
redundancy as well as lower semantics thus being pruned. Based on the STA
score, we are able to progressively prune the tokens without introducing any
additional parameters or requiring further re-training. We directly apply the
STA module to off-the-shelf ViT and VideoSwin backbones, and the empirical
results on Kinetics-400 and Something-Something V2 achieve over $30\%$
computation reduction with a negligible $\sim 0.2\%$ accuracy drop. The code
is released at https://github.com/Mark12Ding/STA.

##  1 Introduction

Recently, there has been an unstoppable shift in the general backbone design
from Convolutional Neural Networks (ConvNets) to Transformers, which are
originally employed in natural language processing, and has shown promising
potential for various vision tasks [9, 55, 54, 25, 1, 36, 5]. The key
component of Transformers is the self-attention mechanism, which is apt to
capture long-range dependencies and empowers ViT to perceive the global
reception field. The seminal work, Vision Transformer (ViT) [9] closely
follows the original Transformer architecture [42]. Equipped with a large-
scale model and dataset, ViT outperforms ConvNets in image classification by a
considerable margin. Inspired by this superior scaling behavior, Transformers
have gained popularity as a backbone choice and are widely adopted for image
recognition [25, 41], action recognition [1, 26], semantic segmentation [56,
6], action detection [48, 14], temporal perception [37, 38], _etc._

Figure 1: Kinectics-400 result for ViT and VideoSwin. The bubble’s area is
proportional to FLOPs of a variant in a model family. ViT/VideoSwin here takes
$16/32\times 224^{2}$ video. The proposed STA saves over $30\%$ FLOPs for all
model variants with a negligible drop in performance.

Despite the promising potential of Transformers in spatio-temporal vision
tasks, such as action recognition, the quadratic increase in complexity caused
by the temporal dimension makes the video Transformers computationally
unfriendly compared to images. For instance, earlier work TimeSformer [2],
which applies Transformer backbone for video, required $7.14$ Tera FLOPs to
achieve $80.7\%$ accuracy on the Kinetics-400 action recognition benchmark.
The excessive computational cost makes it impractical for deployment in real-
world scenarios. Therefore, there is an urgent need to explore ways to profit
from the performance gains of Transformers while maintaining an affordable
computation burden.

Fortunately, Transformers can handle a flexible number of tokens as input.
Recent attempts [34, 33, 47], that dynamically prune tokens for images, have
remarkably reduced computation overhead. These pruning approaches have
inspired us to explore token pruning in the video domain, so as to balance
accuracy and computation costs accordingly. However, performing frame-wise
pruning alone seems to be not optimal since it ignores temporal context and
disrupts the dynamic structure of the video. To address this issue, recent
work STTS [44] decouples token pruning into temporal and spatial dimensions.
Specifically, STTS first drops meaningless frames and then filters out detail-
rich regions from the remained frames. However, this spatio-temporal
decoupling strategy lacks contextual modeling of continuous temporal
information, leading to limited performance.

In this paper, we argue that pruning spatio-temporal tokens integrally can
lead to further computation reduction at an acceptable cost of accuracy
degradation. To this end, we propose the Semantic-aware Temporal Accumulation
(STA) score to depict the importance of each token. We take two factors into
consideration, temporal redundancy and semantic importance. Our motivation is
to discard tokens that have similar counterparts appearing earlier in the
sequence while retaining only semantically significant tokens. As an example,
static backgrounds across all timestamps contain highly repetitive information
that is unnecessary to be included. Therefore, keeping only a few
representative background patches is sufficient for reasoning. Specifically,
we evaluate the temporal redundancy of a region by determining whether it is a
novel or previously observed entity. In practice, we aggregate repetitive
information on a per-frame basis and assign each token a score of temporal
repetition degree. Nevertheless, there are cases where a repetitive region
reveals a crucial action and should be retained. For example, if the sequence
of tokens describes human-body motion, it is necessary to keep all the tokens
for better understanding, even if there are only slight differences over time.
Thus, we also take semantic importance into account during the pruning
procedure. To depict each token’s semantic contribution to video recognition,
we take the summation of the feature activation map and then integrate this
summation with the score of temporal aggregation to enhance the awareness of
semantics. Based on the STA score, we progressively prune the tokens of the
video Transformers three times. The whole pruning process does not introduce
any tuning parameter and directly accelerates the off-the-shelf video
Transformers without the need to re-train.

We apply our pruning strategy to two mainstream video Transformers, ViT [9]
and VideoSwin [26], and evaluate $10$ off-the-shelf backbones on two action
recognition benchmarks, Kinetics-400 [17] and Something-Something V2 [13], to
demonstrate the effectiveness of our method. As shown in Figure 1, we achieve
significant computation reduction with a negligible accuracy drop on
Kinetics-400. For instance, using ViT-H as the backbone, by hierarchically
pruning $57\%$ of the input tokens, STA reduces $49\%$ FLOPs while the
accuracy drop is only $0.2\%$. Besides, with STA, FLOPs of VideoSwin-B are
decreased by $38\%$ while maintaining $82.5\%$ accuracy with only a minimal
drop of $0.2\%$. A similar trend can also be observed in the Something-
Something V2 dataset. Notably, we surpass STTS [44] by a $0.4\%$ accuracy gain
with $40\%$ fewer FLOPs on Kinetics-400 and by a $0.5\%$ accuracy increase
with $20\%$ fewer FLOPs on Something-Something V2 when using the same
backbone.

##  2 Related Work

#### Video Transformers.

Designing Transformer-based architectures for vision tasks has emerged as a
general trend in the computer vision community, as evidenced by several recent
works [9, 25, 57, 31, 56, 27, 43]. With an unprecedented number of parameters
and millions of training data, Transformers significantly outperform prior
arts spanning a variety of tasks, not only in image but also in video
understanding tasks. Various variants of self-attention have been introduced
in prior works [2, 29, 54, 1, 4, 31, 10, 26, 52, 22] to capture the
spatiotemporal relationship. However, using pure patch-based Transformers
incurs prohibitive costs on memory and computation when extracting global-
range features from the whole video. To deal with it, Motionformer [31]
introduces trajectory attention that focuses on implicitly determined motion
paths and optimizes the quadratic calculation via efficient decomposition.
MeMViT [48] proposes caching ‘memory’ of past frames and attending to the
summarized prior context in an online manner. In this paper, we propose an
orthogonal approach to make Transformers lighter by pruning the spatio-
temporal tokens with high temporal redundancy.

Figure 2: An overview of our STA token-pruning algorithm for video
Transformers. (a) STA module is a simple plug-in and can be inserted at the
beginning or end of the Transformer block. In practice, we conduct a three-
stage progressive strategy to prune the token with STA. (b) Our semantic-aware
temporal accumulation algorithm. The wider arrows connecting two adjacent
frames represent higher weight to transport the STA score.

#### Token pruning for Transformers.

Several works [34, 33, 47, 23, 28, 3, 20] have focused on reducing the number
of tokens involved in the calculation to accelerate image Transformer models.
In specific, DynamicViT [33] trains a lightweight decision module to rate the
importance of each token and prune low-score tokens progressively. EViT [23]
preserves the attentive tokens guided by the class token attention and fuses
inattentive ones without the help of any extra parameter. ToMe [3] combines
similar tokens to directly expedite off-the-shelf ViT without needing to
train. While similar to ToMe [3], our method functions as a simple plug-in to
enhance off-the-shelf video Transformers without requiring additional
training. However, ToMe is an image-based pruning technique that does not
model any temporal relations. In contrast, our method specifically devises a
temporal aggregation mechanism tailored for video data. Although STTS [44]
trains a score network to choose pre-defined anchors from filtered frames
within a video, the empirical speed-accuracy trade-off remains limited. This
is because decoupled anchor-level selection still retains unwanted spatial
redundancy. On the contrary, we prune the video at the token level and
eliminate the meaningless content by a large margin.

#### Efficient video recognition.

Due to the nature of the extra dimension, video understanding is
computationally intensive. Thus, there have been attempts [58, 11, 40, 49, 18,
46, 8, 21, 19, 24, 16, 32] to design lightweight modules that enjoy both high
efficiency and high accuracy. ECO [58] introduces a network architecture to
sample a small subset of frames and learns the temporal context between these
frames. Besides, AdaFocus [46] improves the computational efficiency by
adopting a light-weighted ConvNet to localize the most salient spatiotemporal
regions. While previous works have mainly concentrated on accelerating ConvNet
models, efforts toward accelerating video Transformers have been relatively
sparse and open to exploration. A recent approach [30] devises a novel token-
based sampling using k-centered search before feeding tokens into video
Transformers. Although we also select semantically meaningful tokens for video
Transformers, our dynamic model processes whole tokens at the early stage and
prunes them based on model-dependent features.

##  3 Approach

The goal of this paper is to develop a principal token-pruning algorithm for
video Transformers that achieves an optimal balance between cost, speed, and
performance without requiring model re-training. We start by analyzing the
video Transformers at hand and observe two interesting phenomena, detailed in
Sec. 3.3. First, we find the high temporal redundancy when comparing the
inter-frame similarity. Second, the area that contributes to the final
prediction usually takes up a small portion. Motivated by these two findings,
we carefully develop two principles to prune the tokens with high temporal
redundancy and retain the meaningful tokens. The overall framework is shown in
Figure 2(a). Our method is mainly built on the standard columnar Transformer
[9], which we briefly go through in Sec. 3.1. Later, in Sec. 3.2, we elaborate
on the proposed metric to help prune unnecessary tokens. Finally, Sec. 3.3
discusses why STA works in video Transformer.

###  3.1 Revisit of Video Transformer

Video Transformers generally process video data as a 1D sequence of tokens and
directly model the relationship between them. Initially, video Transformer
linearly projects 3D data tubes into high-dimensional embeddings. Assuming the
dimensions of video clips as $\\{T,H,W\\}$ and the size of 3D tubes as
$\\{t,h,w\\}$, then the number of token embeddings is $n=n_{t}\times
n_{s}=|\frac{T}{t}|\times|\frac{HW}{hw}|$. Additionally, positional embeddings
are added to each token to break the permutation invariance. After the patch
embedding layer, an $n$-token sequence $\mathbf{X}\in\mathbb{R}^{n\times d}$
is passed into the self-attention layer, which computes a weighted sum of the
values based on the affinity of other tokens. Mathematically, the self-
attention is formulated as:

$\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{softmax}(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d}})\mathbf{V},$
(1)

where
$\mathbf{Q},\mathbf{K},\mathbf{V}=f_{Q}(\mathbf{X}),f_{K}(\mathbf{X}),f_{V}(\mathbf{X})\in\mathbb{R}^{d}$
are typically linear transformations of $\mathbf{X}$. After spatial-temporal
interaction, the tokens are sent into a feedforward network $f_{\text{FFN}}$,
which consists of a three-layer MLP to exchange inter-channel information.

###  3.2 Semantic-aware Temporal Accumulative Score

Our intuitive criterion is to drop a token if similar tokens exist before it
while reserving semantically meaningful tokens. To this end, our approach
considers two factors when determining whether to retain or discard a token in
the Transformer. The first is its similarity to the other tokens along the
temporal axis, and the second is its contribution to the class attribute. We
discuss these two principles in order.

#### Temporal redundancy.

Intuitively, a token should be removed if similar tokens have already existed
in previous frames. Therefore, we remove tokens frame-by-frame by comparing
whether similar tokens have been retained. For simplicity, we reduce a
constant number of tokens in each frame to ensure parallel computing. We
introduce the accumulative temporal score $\mathbf{A}\in[0,1]^{n_{t}\times
n_{s}}$ to model the probability of dropping a token conditioned on the
specific frame $t$. Specifically, we define:

$\mathbf{A}_{t,s}:=\mathds{P}_{\text{drop}}(\mathbf{X}_{t,s})\in[0,1]\quad\mathrm{s.t.}\sum_{s=1}^{n_{s}}\mathbf{A}_{t,s}=1,$
(2)

where tokens with higher temporal accumulative scores are more likely to be
pruned because they carry a high degree of temporal redundancy. Next, we
eliminate $r$ tokens with the highest scores from $\mathbf{A}$ at $t$-th frame
and transfer the remaining probability distribution
$\mathbf{A}_{t}^{{}^{\prime}}\in\mathbb{R}^{(n_{s}-r)\times 1}$ to the next
frame via the transition probability
$\mathds{P}_{\text{drop}}(\mathbf{X}_{t}|\mathbf{X}_{t-1}^{{}^{\prime}})\in\mathbb{R}^{n_{s}\times(n_{s}-r)}$.
By excluding the dropped tokens of the last frame, we effectively restart the
repetition aggregation. This prevents the high scores from being concentrated
on specific tokens and allows for the global identification of redundancy.
Mathematically,

$\displaystyle\begin{split}&\mathbf{A}_{t+1}:=\mathds{P}_{\text{drop}}(\mathbf{X}_{t+1}|\mathbf{X}_{t}^{{}^{\prime}})\mathbf{A}_{t}^{{}^{\prime}},\\\
&\mathds{P}_{\text{drop}}(\mathbf{X}_{t+1}|\mathbf{X}_{t}^{{}^{\prime}}):=\text{softmax}(f(\mathbf{X}_{t+1})f(\mathbf{X}_{t}^{{}^{\prime}})^{T}),\\\
\end{split}$ (3)

where $f$ is the projection head to measure the similarity, and we construct
transition probability by softmax-based affinity matrix. Note that we do not
need to train a new projection head $f$ because the self-attention provides
the necessary functionality, and the key function $f_{K}$ extracts most
relevant knowledge for affinity estimation, as shown in Table 6c. This
formulation allows us to connect all temporally distinct tokens through a
simple Markov chain and aggregate potential redundancy from the first frame to
every subsequent frame.

Algorithm 1 Pseudocode of STA in a PyTorch-like style.

⬇

# x: token embedding, n_t x n_s x d

# I: image-based token pruning method

# r: drop number per frame

# sim: token-to-token affinity function

# min-max norm, Eqn.(4)

aam = norm(x.abs().sum(-1)) # size: (n_t, n_s, 1)

# token removal at 1-st frame

x_0 = I(x[0]) # size: (n_s-r, d)

# initialization

x_list, x_old = [x_0], x_0

for t in range(1, n_t):

# token-to-token affinity matrix

A_t = sim(x[t], x_old) # size: (n_s, n_s-r)

# accumulative temporal score

s_acc = mm(A_t, s_acc) # size: (n_s, 1)

# class-aware accumulative temporal score, Eqn.(5)

s = s_acc * (1-aam[t])

s = s.squeeze(dim=-1) # size: (n_s)

# keep tokens with the minimal score

i_t = s.topk(k=N-r, largest=False) # size: (n_s-r)

x_old = x[t, i_t] # size: (n_s-r, d)

x_list = x_list.append(x_old)

# cut off the dropped tokens’ score

s_acc = s_acc[i_t] # size: (n_s-r, 1)

# first-order norm

s_acc = s_acc / s_acc.sum()

return stack(x_list, dim=0) # size: (n_t, n_s-r, d)

mm: matrix multiplication.

#### Semantic importance.

Up until this point, our approach has focused on capturing temporally
repetitive information. However, we have neglected the influence of semantic
attributes. In other words, we have treated each token equally, regardless of
its contribution to the semantics of the class. To integrate semantics
importance with our approach, we use the activation-based attention map
$\mathcal{F}$ [51], which takes the feature matrix
$\mathbf{X}\in\mathbb{R}^{n_{t}\times n_{s}\times d}$ as input and produces a
score for each token in the matrix. Specifically, we define the semantic score
for token $\mathbf{X}_{t,s}$ as:

$\mathcal{F}(\mathbf{X}_{t,s})=\sum_{i=1}^{d}|\mathbf{X}_{t,s,i}|\in\mathbb{R^{+}}.$
(4)

Intuitively, through the summation of absolute activation values over channel
dimension, a high absolute activation suggests a significant contribution to
next layers. Moreover, we apply STA on off-the-shelf Transformers supervised
by semantic labels, where high activation areas tend to represent
discriminative category information. Thus, activation-based attention maps
could effectively capture the importance or relevance of the token to the
overall semantics. We then use this score to re-weigh the temporal
accumulative scores $\mathbf{A}$, giving tokens with high semantic
contributions less weight in the pruning process. This ensures that tokens
with high semantics are more likely to be retained, even if they have a high
degree of temporal redundancy.

Finally, we compute the semantic-aware temporal accumulative score
$\widetilde{\mathbf{A}}_{t,s}$ by integrating the semantic score
$\mathcal{F}(\mathbf{X}_{t,s})$ with the accumulative temporal score
$\mathbf{A}$, i.e.,

$\widetilde{\mathbf{A}}_{t,s}=(1-\mathcal{F}(\mathbf{X}_{t,s}))\mathbf{A}_{t,s},$
(5)

where $\mathcal{F}(\mathbf{X}_{t,s})$ is min-max normalized to the range
[0,1]. We utilize the semantic-aware accumulative temporal scores to guide
token removal for all subsequent frames, except for the first frame. Thus, we
adopt an image-based token pruning method on the first frame to kick off our
algorithm. Once tokens are discarded through our strategy, they are never
employed in subsequent layers, thus accelerating the inference of the
Transformer.

We summarize the pseudocode of STA in Alg. 1. The algorithm takes token
embedding $\mathbf{X}\in\mathbb{R}^{n_{t}\times n_{s}\times d}$, an image-
based token pruning method $I$, the number of tokens to drop per frame $r$,
and a token-to-token affinity function as inputs. The algorithm calculates the
STA score for each frame, selects the tokens with the minimal STA score, and
retains them for the next frame. This process is repeated for all frames and
returns the resulting token embedding matrix with the retained tokens
$\mathbf{X^{\prime}}\in\mathbb{R}^{n_{t}\times(n_{s}-r)\times d}$.

#### Summary of the superiority of STA.

Compared to the previous token-pruning methods, our approach, STA, offers
three significant merits when applied to video data:

  * •

STA fully considers the potential repetition of tokens along the temporal axis
and eliminates the genuine redundancy with insignificant semantics. The
temporal aggregation design makes the scoring mechanism more motion-aware and
suitable for video data;

  * •

STA works as a plug-in module without the introduction of additional
parameters and it does not require the retraining of the video Transformer;

  * •

STA achieves a complexity of $O\left(n_{t}n_{s}(n_{s}-r)\right)$, resulting in
negligible additional FLOPs that only take up a small percentage of the total
forward pass. Moreover, our algorithm allows for the bulk of computation to be
done in parallel, making it friendly to modern GPU devices.

Overall, our approach is efficient and easily deployable, making it an ideal
solution for pruning video Transformers.

###  3.3 Discussion

In this section, we present a two-part practical analysis to shed the light on
the intuition behind STA.

#### Does STA effectively reduce the temporal redundancy?

To answer this question, we first define temporal redundancy as the frequency
with which similar tokens appear at different timestamps. We then assess this
phenomenon in a video by probing the last frame, denoted as
$\mathbf{X_{-1}}\in\mathbb{R}^{n_{s}\times d}$, and aggregating the cosine
similarity between each token and the most similar tokens in previous frames.
We term this aggregation as trajectory sum $S\in\mathbb{R}$. Mathematically,

$S=\frac{1}{n_{s}}\sum_{i=1}^{n_{s}}\sum_{t=1}^{n_{t}-1}\max_{j\in\\{1,\cdots,n_{s}\\}}\text{cos-
sim}(\mathbf{X}_{-1},\mathbf{X}_{t})_{ij},$ (6)

where $\text{cos-
sim}(\mathbf{X},\mathbf{Y})_{ij}=\frac{\mathbf{X}_{i}\cdot\mathbf{Y}_{j}}{|\mathbf{X}_{i}||\mathbf{Y}_{j}|}$.
A higher trajectory sum indicates greater temporal redundancy and lower
diversity along the temporal axis. When all the frames are the same, the score
would reach its theoretical maximum, which is $N_{t}-1$. Then, we compare our
method with the standard Transformer and random-prune counterparts in terms of
the proposed trajectory sum. Table 1 shows that the video Transformer exhibits
heavy temporal redundancy and random pruning fails to alleviate them
considerably. In contrast, STA achieves a far lower trajectory sum, indicating
that it effectively eliminates temporal redundancy.

Model |  Small |  Base |  Large |  Huge
---|---|---|---|---
ViT |  5.10 |  5.38 |  5.07 |  5.55
Rand-ViT |  5.00 |  5.32 |  4.95 |  5.46
STA-ViT |  4.43 |  4.74 |  4.28 |  4.78
Table 1: Trajectory sum for ViT family on the Kinetics-400 validation set.
Compared to random pruning, STA-ViT decreases the temporal redundancy
significantly. Figure 3: Gradient visualization for ViT-Large on the
Kinetics-400 validation set. We stack original RGB frames, gradient norm
heatmap, and our pruning result from top to bottom. Our pruning algorithm
preserves the area of rich semantics well.

#### Does STA retain semantics-rich tokens?

To manifest it, we calculate the gradient norm for each token when back-
propagating the training loss $\mathcal{L}$. We then aggregated the first-
order gradient norm of $\mathbf{X}^{l}$ at each layer $l$ to obtain the
GradNorm, which reflects the contribution of each token to the final
prediction. Mathematically,

$\text{GradNorm}(\mathbf{X})=\sum_{l=1}^{L}\sum_{i=1}^{d}\left|\frac{\partial\mathcal{L}}{\partial\mathbf{X}^{l}_{\cdot,\cdot,i}}\right|\in\mathbb{R}^{n_{t}\times
n_{s}},$ (7)

where $L$ is the total number of the Transformer block, e.g., $L=24$ for ViT-
Large. Figure 3 shows the GradNorm distribution in the form of a heatmap. The
heatmap reveals sparse patterns across the board, indicating that most tokens
do not contribute significantly to the final prediction. Instead, the key
regions responsible for the final prediction are usually motion-centric
entities. This agrees with our intuition of highlighting semantically
meaningful regions. With the help of activation-based attention map
$\mathcal{F}$ in Eqn. (4), STA retains almost all areas of high-activation
GradNorm as is evidenced in the last row of Figure 3.

Base Model |  Metrics | Drop Number $r_{1}$
---|---|---
0 |  32 |  48 |  64
ViT-S |  K400 Acc. (%) |  78.8 |  78.8 (-0.0) |  78.6 (-0.2) |  78.1 (-0.7)
SSV2 Acc. (%) |  66.8 |  66.6 (-0.2) |  66.4 (-0.4) |  65.8 (-1.0)
GFLOPs |  57 |  42 (-26%) |  35 (-39%) |  29 (-49%)
ViT-B |  K400 Acc. (%) |  81.2 |  81.2 (-0.0) |  81.1 (-0.1) |  80.8 (-0.4)
SSV2 Acc. (%) |  70.6 |  70.4 (-0.2) |  70.3 (-0.3) |  69.9 (-0.7)
GFLOPs |  180 |  136 (-24%) |  116 (-36%) |  96 (-47%)
ViT-L |  K400 Acc. (%) |  85.1 |  85.2 (+0.1) |  85.1 (-0.0) |  85.0 (-0.1)
GFLOPs |  597 |  446 (-25%) |  376 (-37%) |  308 (-48%)
ViT-H |  K400 Acc. (%) |  86.3 |  86.3 (-0.0) |  86.2 (-0.1) |  86.1 (-0.2)
GFLOPs |  1192 |  890 (-25%) |  748 (-37%) |  611 (-49%)
Table 2: Main Results for STA-ViT family on Kinetics-400 [17] (K400) and Something-Something V2 [13] (SSV2). All input resolution is $16\times 224^{2}$.  Base Model |  Metrics | Drop Number $r_{1}$
---|---|---
0 |  192 |  256 |  320
VideoSwin-T |  K400 Acc. (%) |  78.8 |  78.7 (-0.1) |  78.7 (-0.1) |  78.6 (-0.2)
GFLOPs |  88 |  68 (-23%) |  61 (-31%) |  54 (-39%)
VideoSwin-S |  K400 Acc. (%) |  80.5 |  80.3 (-0.2) |  80.2 (-0.3) |  80.1 (-0.4)
GFLOPs |  166 |  121 (-27%) |  106 (-36%) |  91 (-45%)
VideoSwin-B |  K400 Acc. (%) |  82.7 |  82.5 (-0.2) |  82.5 (-0.2) |  82.3 (-0.4)
K400 GFLOPs |  282 |  202 (-28%) |  176 (-38%) |  149 (-47%)
SSV2 Acc. (%) |  69.6 |  69.6 (-0.0) |  69.5 (-0.1) |  69.2 (-0.4)
SSV2 GFLOPs |  321 |  241 (-25%) |  215 (-33%) |  188 (-41%)
Table 3: Main Results for STA-VideoSwin family on Kinetics-400 [17] (K400) and
Something-Something V2 [13] (SSV2). All input resolution is $32\times
224^{2}$. VideoSwin-B employs varying window sizes for K400 and SSV2, leading
to a discrepancy in FLOPs.

##  4 Experiments

###  4.1 Experimental Setup

#### Datasets and backbones.

We evaluate our algorithm on two standard video action recognition datasets.
Kinetics-400 [17] (K400) and Something-Something V2 [13] (SSV2). Kinetics-400
is a large-scale action recognition dataset sourced from YouTube, which
consists of around 10-second clips with $400$ human action classes. The
training and validation set has approximately $240$K and $20$K videos,
respectively. Something-Something V2 is a motion-heavy benchmark with $174$
labels, where the object and background are shared across the different action
categories. Around $170$K and $25$K videos exist in the training and
validation set of SSV2, respectively. We implement our STA strategy with two
mainstream Transformer-based backbones, namely ViT [9] and VideoSwin [26]. For
ViT, we sample $16$ frames with $224^{2}$ pixels as input and the size of 3D
tube is $\\{2,16,16\\}$. Therefore, the number of tokens for all layers is
$n=\frac{16}{2}\times\frac{224^{2}}{16^{2}}=8\times 14^{2}$. We load the open-
sourced checkpoints from VideoMAE [39] due to its superior performance. For
VideoSwin, the input resolution is $32\times 224^{2}$ and the size of 3D tube
is $\\{2,4,4\\}$. The number of tokens at four hierarchical stages would be
$\\{16\times 56^{2},16\times 28^{2},16\times 14^{2},16\times 7^{2}\\}$. Though
our algorithm is not intended for window-based Transformers that require
structural integrity, we still find a simple solution. We compensate for the
discarded token with the nearest kept tokens when computing the window
attention and speed up the rest of the process, such as the linear projection.
In total, we test $10$ off-the-shelf backbones with different pre-trained
weights on two benchmarks.

#### Implementation details.

To progressively remove inattentive tokens, we apply our STA module three
times. For ViT, we insert our STA module at the end of the $1^{st}$,
$(1+L/3)^{th}$, $(1+2L/3)^{th}$ block, where $L$ is the total number of
Transformer blocks. For VideoSwin, there are four hierarchical stages with
varied resolutions and the STA module is located at the end of the first three
stages. We denote different variants of STA as STA${}^{r_{1}}$-Model, where
$r_{1}$ is the number of spatial tokens dropped per frame at the first stage.
We adopt a decreasing schedule, which reduces the dropped number by half at
each stage. For instance, STA64-ViT-L indicates pruning $\\{8\times 64,8\times
32,8\times 16\\}$ tokens on ViT-Large. We weigh the computation via two
metrics, FLOPs (floating-point operations) and throughput. FLOPs are reported
with the help of fvcore library111https://github.com/facebookresearch/fvcore
and throughput (clips/s) is measured at a batch size of $32$ on a single 32G
Tesla V100. Besides, we closely follow the inference metric for ViT and
VideoSwin in [39, 26].

  * •

ViT [39]: To evaluate on K400, we sample 5 × 3 views by uniformly selecting 5
16-frame clips from a full-length video in the temporal dimension with the
frame stride of 4. For each clip, we resize the shorter spatial side to 224
pixels and extract 3 crops of 224 × 224 resolution that cover the longer
spatial axis. The final score is the average score computed over all views.
For SSV2, we follow a similar procedure by sampling 2 clips × 3 crops and the
frames stride is $2$.

  * •

VideoSwin [26]: For K400, we extract 4 32-frame clips from each full-length
video using a temporal stride of 2 and a spatial size of 224 × 224. Similarly,
for SSV2, we extract 1 set of clips using a spatial size of 224 × 224 and 3
spatial crops, with a frame stride of 2. Besides, we prune VideoSwin three
times as $\\{r_{1},1.5r_{1}/4,2r_{1}/16\\}$. For instance, STA256-VideoSwin-S
indicates pruning $\\{16\times 256,16\times 96,16\times 32\\}$ on VideoSwin-S.

###  4.2 Main Results

We conduct a thorough investigation of two off-the-shelf model families, ViT
[9] and VideoSwin [26], on the Kinetics-400 and Something-Something V2
datasets. The results presented in Table 2 demonstrate that our proposed
method can significantly reduce the computational costs of ViT models by
$25\%$ $\sim$ $49\%$, with negligible impacts on performance (-$0.2\%$ $\sim$
-$1.0\%$). It is worth noting our method shows a favorable trade-off between
complexity and performance for larger models. For instance, our method reduces
the FLOPs of ViT-Huge by half to just $611$ GFLOPs, with only a $0.2\%$ drop
in accuracy.

To demonstrate the potential of our method to generalize well on various
transformer backbones, we are conducting further experiments on VideoSwin
[26], a modern architecture that uses a window shuffling operation to
interchange information. As VideoSwin is naturally unsuitable for unstructured
tokens, we are filling the dropped locations during the window attention
operation with the nearest tokens and then discarding the replicated tokens
after the attention operation. Empirical results in Table 3 indicate that the
performance of VideoSwin holds until FLOPs fall by roughly $40\%$. This
observation verifies that both columnar ViT and hierarchical VideoSwin have
heavy and unnecessary computations that can be significantly optimized.

Model |  GFLOPs$\times$views |  Top-1
---|---|---
TimeSformer-L [2] | $8353\times 1\times 3$ |  80.7
Motionformer-L [31] | $1185\times 10\times 3$ |  80.2
ViViT [1] | $3981\times 4\times 3$ |  84.9
Swin-L [26] | $2107\times 10\times 5$ |  84.9
MViTv2-L [22] | $2828\times 5\times 3$ |  86.1
ViT-H [39] | $1192\times 5\times 3$ |  86.3
STTS-VideoSwin-B [44] | $253\times 4\times 3$ |  81.9
ToMe-ViT-L [3] | $281\times 10\times 1$ |  84.5
STA320-VideoSwin-B (ours) | $149\times 4\times 3$ |  82.3
STA64-ViT-L (ours) | $308\times 5\times 3$ |  85.0
STA64-ViT-H (ours) | $611\times 5\times 3$ |  86.1
Table 4: Comparisons with the-state-of-the-arts method on Kinetics-400. We report the computational cost with a single view (temporal clip with spatial crop) × the number of views (FLOPs$\times$ view${}_{\text{time}}$ $\times$ view${}_{\text{space}}$). Gray represents that this method leverages the dynamic token pooling to optimize existing backbones. Model |  GFLOPs$\times$views |  Top-1
---|---|---
TimeSformer-L [2] | $5549\times 1\times 3$ |  62.4
Motionformer-L [31] | $1185\times 1\times 3$ |  68.1
MViTv2-B [22] | $225\times 1\times 3$ |  70.5
VideoSwin-B [26] | $321\times 1\times 3$ |  69.6
ViT-B [39] | $180\times 2\times 3$ |  70.6
STTS-VideoSwin-B [44] | $237\times 1\times 3$ |  68.7
STA320-VideoSwin-B (ours) | $188\times 1\times 3$ |  69.2
STA48-ViT-B (ours) | $116\times 2\times 3$ |  70.3
Table 5: Comparisons with the-state-of-the-arts method on Something-Something
V2. We report the computational cost with a single view (temporal clip with
spatial crop) × the number of views (FLOPs$\times$ view${}_{\text{time}}$
$\times$ view${}_{\text{space}}$). Gray represents that this method leverages
the dynamic token pooling to optimize existing backbones.

#### Comparison with the state of the art.

Firstly, we tabulate a comparison of our proposed method on K400 in Table 4.
Our model performs favorably in terms of both accuracy and computation cost.
For example, ViT-L equipped with our STA achieves the same accuracy as
MViTv2-L [22] but with less than a quarter of the computational cost.
Moreover, STTS [44] proposes a scorer network to conduct dynamic token
selection separately in space and time, requiring to be trained in an end-to-
end fashion. Our result surpasses STTS by $0.4\%$ accuracy using the same
backbone VideoSwin-B but with only $60\%$ GFLOPs. This result verifies that
leveraging the model itself to weigh the redundancy of tokens is sufficient to
reduce complexity. We also report the result on SSV2 in Table 5. The superior
performance of our proposed method verifies that STA prunes the
inconsequential tokens via temporal cues, as it is known that understanding
SSV2 mainly relies on temporal information. Specifically, ViT-B equipped with
our STA surpasses most of the prior arts with a considerably minor complexity
of $116$ GFLOPs. For VideoSwin, our strategy outperforms STTS-VideoSwin by
$0.5\%$ accuracy with $80\%$ of the computation cost.

#### Visualization of STA.

Figure 4 shows image patches corresponding to kept tokens after three stages.
The results align with our objective of resisting temporal redundancy and
retaining informative tokens. In a tennis sequence, STA preserves the most
meaningful patches, including a human at the far end of the court, and filters
out dull backgrounds like the blue ground. The temporal aggregation design
ensures that the kept tokens are not just the most salient ones but also a
variety of regions, preserving diversity within videos for better reasoning.

Method |  Top-1 |  Top-5
---|---|---
Random |  84.78 |  96.46
Grid |  84.85 |  96.50
ToMe |  84.96 |  96.50

(a)  Ablation on token removal methods at the first frame.

Order |  Top-1 |  Top-5
---|---|---
F-B-F |  84.96 |  96.50
B-F-B |  84.97 |  96.43
F-F-F |  84.26 |  96.41
B-B-B |  84.35 |  96.37
(b)  Ablation on temporal accumulation order.

Similarity |  Top-1 |  Top-5
---|---|---
$f_{Q}$ |  84.83 |  96.52
$f_{K}$ |  84.96 |  96.50
$f_{V}$ |  84.84 |  96.50
$f_{\text{FFN}}$ |  84.94 |  96.50
(c)  Ablation on different similarity function $f$.

Schedule |  Top-1 |  Top-5 |  clips/s
---|---|---|---
constant |  84.68 |  96.36 |  47
decreasing |  84.96 |  96.50 |  47
increasing |  77.68 |  93.72 |  44
(d)  Ablation on dropping schedule among three stages.

Score | ViT-S | ViT-B | ViT-L
---|---|---|---
$1-\mathcal{F}(\mathbf{X}_{t,s})$ | 77.33 | 80.52 | 84.87
$\mathbf{A}_{t,s}$ | 77.78 | 80.43 | 84.69
$(1-\mathcal{F}(\mathbf{X}_{t,s}))\mathbf{A}_{t,s}$ | 78.12 | 80.82 | 84.96
(e)  Ablation on scoring mechanism. Top-1 is reported.

Table 6: Results of STA ablation experiments. F and B in (b) mean forward and
backward order, respectively. The baseline ViT-L without STA obtains $85.05\%$
Top-1 and $96.55\%$ Top-5 accuracy on K400 at $19.5$ clips/s. Gray is our
default setting. Figure 4: Visualization of the proposed STA strategy. We
masked out the discarded tokens with white boxes. STA not only retains
informative tokens but also ensures diverse regions for improved video
reasoning.

###  4.3 Ablation Study

To find the optimal strategy, we conduct a series of ablation studies. We
evaluate off-the-shelf ViT-Large on K400 by default and report accuracy,
FLOPs, and throughput for reference unless otherwise stated.

#### Token removal at the first frame.

To investigate how the first-frame removal affects performance, we conduct
experiments on three candidates. (1) Random Prune: we randomly select $r$
tokens to discard. (2) Grid Prune: we split the first frame into
$\sqrt{r}\times\sqrt{r}$ grids spatially and drop one random token per grid.
(3) ToMe Prune: inspired by recent image pruning method ToMe [3], we get rid
of the most similar tokens by the simple bipartite soft matching. The
difference here is that we just drop tokens rather than ‘merge’ tokens. Note
that all three pruning ways are negligible in the terms of computation and
lead to similar speed-up. It actually echoes the main insight of STA,
leveraging temporal aggregation to reduce spatio-temporal redundancy. Even
given a random initialization state, the sequential STA strategy could still
decrease the total temporal redundancy and optimize the video Transformers
effectively.

#### Pruning schedule.

We explore how to assign the number of dropping tokens among three stages.
When maintaining a similar throughput, we devise three types of schedules:

$\displaystyle\begin{split}\text{Constant Schedule:}&\\{8\times 48,8\times
48,8\times 48\\};\\\ \text{Decreasing Schedule:}&\\{8\times 64,8\times
32,8\times 16\\};\\\ \text{Increasing Schedule:}&\\{8\times 27,8\times
54,8\times 108\\}.\\\ \end{split}$

As shown in Table 6d, the decreasing schedule owns the least accuracy
reduction with similar throughput. It verifies that standard video
Transformers process a great number of uninformative tokens that can be
dropped at the beginning.

#### Temporal accumulation order.

Besides starting the accumulation flow from the beginning of the input video,
we could also kick off at the ending frames. In Table 6b, we empirically find
alternating order at different dropping stages outperforms the consistent
counterparts. We speculate that the same accumulation direction would amplify
intrinsic propagation error but alternating the order counteracts it, leading
to more reasonable pruning.

#### Similarity function choice.

We ablate four similarity project heads
$\\{f_{Q},f_{V},f_{K},f_{\text{FFN}}\\}$. Table 6c shows that key function
$f_{k}$ captures the most correct affinity with minimal noise. The observation
coincides with previous work [3].

#### Scoring mechanism.

To explore how accumulation score and semantic identification boost each
other, we conduct the experiment with different scoring formulas. Table 6e
demonstrates that considering both temporal redundancy and semantics helps in
discovering informative tokens. The results on ViT-S show that temporal
aggregation modeled by the Markov Chain plays an important role in the pruning
process, while semantic importance functions effectively for ViT-B and ViT-L.

#### Performance vs. prune number $r$.

To seek the sweet spot of our algorithm, we vary the prune number $r$ at the
first stage ranging from [$16$, $96$] and evaluate the Top-1 accuracy. In
addition, we compare our STA with the Random pruning baseline. As displayed in
Figure. 5, STA behaves fairly robust to the token reduction and consistently
surpasses the result of the random pruning. Specifically, $r=64$ doubles the
throughput but just drops $0.1\%$ accuracy. This confirms that our algorithm
retains the semantics-rich tokens with the lowest redundancy.

Figure 5: Top-1 accuracy and throughput under two pruning methods with various
prune numbers $r$.

##  5 Conclusion

In conclusion, we propose a new token pruning strategy, Semantic-aware
Temporal Accumulation (STA), for video Transformers that can significantly
reduce computation overhead with a subtle accuracy drop. Specifically, we
consider temporal redundancy and semantic importance when deciding to keep or
drop the token. Our approach does not introduce any parameter and can directly
accelerate the off-the-shelf video Transformers without training. The
extensive experiments demonstrate that our method empowers video Transformers
to obtain a competitive speed-accuracy trade-off compared to the prior arts.

## Acknowledgement

This work was supported in part by the National Natural Science Foundation of
China under Grant 62250055, Grant 61932022, Grant 62120106007, and in part by
the Program of Shanghai Science and Technology Innovation Project under Grant
20511100100.

## References

  * [1] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6836–6846, October 2021.
  * [2] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In ICML, volume 2, page 4, 2021.
  * [3] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman.  Token merging: Your vit but faster.  In The Eleventh International Conference on Learning Representations, 2023.
  * [4] Adrian Bulat, Juan Manuel Perez Rua, Swathikiran Sudhakaran, Brais Martinez, and Georgios Tzimiropoulos.  Space-time mixing attention for video transformer.  Advances in Neural Information Processing Systems, 34:19594–19607, 2021.
  * [5] Jiaqi Chen, Jiachen Lu, Xiatian Zhu, and Li Zhang.  Generative semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7111–7120, 2023.
  * [6] Bowen Cheng, Alex Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  Advances in Neural Information Processing Systems, 34:17864–17875, 2021.
  * [7] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703, 2020.
  * [8] Shuangrui Ding, Maomao Li, Tianyu Yang, Rui Qian, Haohang Xu, Qingyi Chen, Jue Wang, and Hongkai Xiong.  Motion-aware contrastive video representation learning via foreground-background merging.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9716–9726, 2022.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [10] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer.  Multiscale vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6824–6835, 2021.
  * [11] Christoph Feichtenhofer.  X3d: Expanding architectures for efficient video recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 203–213, 2020.
  * [12] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019.
  * [13] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al.  The” something something” video database for learning and evaluating visual common sense.  In Proceedings of the IEEE international conference on computer vision, pages 5842–5850, 2017.
  * [14] Roei Herzig, Elad Ben-Avraham, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson.  Object-region video transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3148–3159, June 2022.
  * [15] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry.  Augment your batch: Improving generalization through instance repetition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8129–8138, 2020.
  * [16] Boyuan Jiang, MengMeng Wang, Weihao Gan, Wei Wu, and Junjie Yan.  Stm: Spatiotemporal and motion encoding for action recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2000–2009, 2019.
  * [17] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950, 2017.
  * [18] Bruno Korbar, Du Tran, and Lorenzo Torresani.  Scsampler: Sampling salient clips from video for efficient action recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6232–6242, 2019.
  * [19] Myunggi Lee, Seungeui Lee, Sungjoon Son, Gyutae Park, and Nojun Kwak.  Motion feature network: Fixed motion filter for action recognition.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 387–403, 2018.
  * [20] Jin Li, Yaoming Wang, Xiaopeng Zhang, Yabo Chen, Dongsheng Jiang, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Qi Tian.  Progressively compressed auto-encoder for self-supervised representation learning.  In The Eleventh International Conference on Learning Representations, 2023.
  * [21] Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang.  Tea: Temporal excitation and aggregation for action recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [22] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer.  Mvitv2: Improved multiscale vision transformers for classification and detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4804–4814, 2022.
  * [23] Youwei Liang, Chongjian GE, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie.  EVit: Expediting vision transformers via token reorganizations.  In International Conference on Learning Representations, 2022.
  * [24] Ji Lin, Chuang Gan, and Song Han.  Tsm: Temporal shift module for efficient video understanding.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 7083–7093, 2019.
  * [25] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * [26] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3202–3211, 2022.
  * [27] Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang.  Soft: Softmax-free transformer with linear complexity.  Advances in Neural Information Processing Systems, 34:21297–21309, 2021.
  * [28] Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim.  Adavit: Adaptive vision transformers for efficient image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12309–12318, 2022.
  * [29] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann.  Video transformer network.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3163–3172, 2021.
  * [30] Seong Hyeon Park, Jihoon Tack, Byeongho Heo, Jung-Woo Ha, and Jinwoo Shin.  K-centered patch sampling for efficient video recognition.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV, pages 160–176. Springer, 2022.
  * [31] Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and João F Henriques.  Keeping your eye on the ball: Trajectory attention in video transformers.  Advances in neural information processing systems, 34:12493–12506, 2021.
  * [32] Rui Qian, Shuangrui Ding, Xian Liu, and Dahua Lin.  Static and dynamic concepts for self-supervised video representation learning.  In European Conference on Computer Vision, pages 145–164. Springer, 2022.
  * [33] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh.  Dynamicvit: Efficient vision transformers with dynamic token sparsification.  Advances in neural information processing systems, 34:13937–13949, 2021.
  * [34] Michael Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova.  Tokenlearner: Adaptive space-time tokenization for videos.  Advances in Neural Information Processing Systems, 34:12786–12797, 2021.
  * [35] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  * [36] Jing Tan, Jiaqi Tang, Limin Wang, and Gangshan Wu.  Relaxed transformer decoders for direct action proposal generation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 13526–13535, 2021.
  * [37] Jing Tan, Yuhong Wang, Gangshan Wu, and Limin Wang.  Temporal perceiver: A general architecture for arbitrary boundary detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023\.
  * [38] Jing Tan, Xiaotong Zhao, Xintian Shi, Bin Kang, and Limin Wang.  Pointtad: Multi-label temporal action detection with learnable query points.  Advances in Neural Information Processing Systems, 35:15268–15280, 2022.
  * [39] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  In Advances in Neural Information Processing Systems, 2022.
  * [40] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450–6459, 2018.
  * [41] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li.  Maxvit: Multi-axis vision transformer.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIV, pages 459–479. Springer, 2022.
  * [42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [43] Qiang Wan, Zilong Huang, Jiachen Lu, Gang Yu, and Li Zhang.  Seaformer: Squeeze-enhanced axial transformer for mobile semantic segmentation.  arXiv preprint arXiv:2301.13156, 2023.
  * [44] Junke Wang, Xitong Yang, Hengduo Li, Li Liu, Zuxuan Wu, and Yu-Gang Jiang.  Efficient video transformers with spatial-temporal token selection.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV, pages 69–86. Springer, 2022.
  * [45] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803, 2018.
  * [46] Yulin Wang, Zhaoxi Chen, Haojun Jiang, Shiji Song, Yizeng Han, and Gao Huang.  Adaptive focus for efficient video recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16249–16258, 2021.
  * [47] Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang.  Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition.  Advances in Neural Information Processing Systems, 34:11960–11973, 2021.
  * [48] Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer.  Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13587–13597, 2022.
  * [49] Zuxuan Wu, Caiming Xiong, Chih-Yao Ma, Richard Socher, and Larry S Davis.  Adaframe: Adaptive frame selection for fast video recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1278–1287, 2019.
  * [50] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023–6032, 2019.
  * [51] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  In International Conference on Learning Representations, 2017.
  * [52] Xuefan Zha, Wentao Zhu, Lv Xun, Sen Yang, and Ji Liu.  Shifted chunk transformer for spatio-temporal representational learning.  Advances in Neural Information Processing Systems, 34:11384–11396, 2021.
  * [53] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In International Conference on Learning Representations, 2018.
  * [54] Yanyi Zhang, Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe.  Vidtr: Video transformer without convolutions.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 13577–13587, 2021.
  * [55] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun.  Point transformer.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 16259–16268, 2021.
  * [56] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6881–6890, 2021.
  * [57] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable {detr}: Deformable transformers for end-to-end object detection.  In International Conference on Learning Representations, 2021.
  * [58] Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox.  Eco: Efficient convolutional network for online video understanding.  In Proceedings of the European conference on computer vision (ECCV), pages 695–712, 2018.

##  Appendix A More Experimental Results

#### Training details.

We also train the ViT-B with our proposed STA. We adopt dense sampling [45,
12] on K400. We sample $16$ consecutive frames with the stride of $4$. The
resolution is $224\times 224$. We perform RandAug augmentation (9, 0.5) [7],
label smoothing (0.1) [35], mixup (0.8) [53], cutmix (1.0) [50], and random
horizontal flip (0.5). In addition, we adopt the repeated augmentation [15].
With DeepSpeed 222https://github.com/microsoft/DeepSpeed, We use the linearly
scale scheme to ensure effective parameter updates across different batch
sizes during training, i.e., $lr$ = base learning rate $\times$ batch size /
$256$. Specifically, we use the AdamW optimizer with a base learning rate of
$1$e$-3$ and weight decay of $0.05$. Beside, using a cosine decay learning
rate scheduler and 5 epochs of linear warm-up, we finetune the model for 100
epochs with a total batch size of 128 on 4 nodes of 8 Tesla V100 GPUs.

#### Training results.

Besides speeding up the inference of off-the-shelf backbones, our algorithm
also has the potential to expedite training. We report the training hours for
ViT-Base in Table 7. STA cuts the training time in half. Without modifying the
training recipe, the trained model only drops 0.6 % in Top-1 accuracy. We
believe that STA would be more effective to maintain the performance when
training deeper backbones. We leave it as the future work.

#### Number of views.

To analyze the impact of the number of test clips on our method, we conduct an
experiment by varying the number of clips and comparing the results with the
baseline ViT-L model. In Table. 8, we show that the relative performance drop
remains constant at approximately 0.1% regardless of the number of views, when
the drop number is set to $r_{1}=64$. Furthermore, when using a lower value of
$r_{1}=48$, there is no significant decrease in performance compared to the
baseline.

#### Number of STA blocks and insert location

We devise two extra ablation studies shown in Table 9. Our experiments
demonstrate that incorporating 3 progressive blocks at the very first
beginning achieves an optimal trade-off. This approach allows for preferable
computation while delivering maximal performance.

Model | clips/s | Training time | Top-1
---|---|---|---
ViT-B | 53 | 28 hrs | 81.2
STA48-ViT-B | 96 | 15 hrs | 80.6
Table 7: Comparison on training time on Kinetics-400. We measure training time on 4 nodes of 8 V100. Views | Drop Number $r_{1}$
---|---
0 | 48 | 64 | 80
2 $\times$ 3 | 83.36 | 83.21 | 83.09 | 82.56
4 $\times$ 3 | 85.10 | 85.00 | 84.85 | 84.35
6 $\times$ 3 | 85.05 | 85.07 | 84.84 | 84.59
8 $\times$ 3 | 84.91 | 84.93 | 84.80 | 84.43
16 $\times$ 3 | 84.91 | 84.97 | 84.89 | 84.48
Table 8: Ablation on the temporal views of test clips. # of STA | GFLOPs | Top-1 | Location | GFLOPs | Top-1
---|---|---|---|---|---
2 | 302 | 84.5 | 1,9,17 | 308 | 85.0
3 | 308 | 85.0 | 3,11,19 | 339 | 85.0
4 | 305 | 84.8 | 5,13,21 | 370 | 85.1
Table 9: Ablation on the number of STA blocks and insert location.

##  Appendix B More Visualization

We provide more visualization for our STA on K400 in Figure 6 and SSV2 in
Figure 7, which display image patches that correspond to the tokens retained
after three stages of pruning. We observe that the pruning results align well
with our objective of preserving detail-rich tokens and resisting temporal
redundancy. Specifically, upon examining the guitar-playing sequence in Figure
6, STA accurately preserves two partially visible guitars on the wall.
Additionally, the dropped tokens shown in Figure 7 at different timestamps are
distributed unevenly, preserving the diversity of the video content.

(a)

(b)

(c)

(d)

Figure 6: Visualization of our STA strategy on K400.

(a)

(b)

(c)

(d)

Figure 7: Visualization of our STA strategy on SSV2.