# Geometrically thick equilibrium tori around a Schwarzschild black hole in
swirling universes

Chengjia Chen1, Qiyuan<EMAIL_ADDRESS>and Jiliang
<EMAIL_ADDRESS>1 Department of Physics, Key Laboratory of Low
Dimensional Quantum Structures and Quantum Control of Ministry of Education,
Institute of Interdisciplinary Studies, and Synergetic Innovation Center for
Quantum Effects and Applications, Hunan Normal University, Changsha, Hunan
410081, China
2 Center for Gravitation and Cosmology, College of Physical Science and
Technology, Yangzhou University, Yangzhou 225009, China

###### Abstract

Abstract

We study geometrically thick non-self gravitating equilibrium tori orbiting a
Schwarzschild black hole immersed in swirling universes. This solution is
axially symmetric and non-asymptotically flat, and its north and south
hemispheres spin in opposite directions. Due to repulsive effects arising from
the swirl of the background spacetime, the equilibrium torus exists only in
the case with the small swirling parameter. With the increase of the swirling
parameter, the disk structure becomes small and the excretion of matter near
the black hole becomes strong. Moreover, the odd $Z_{2}$ symmetry of
spacetimes originating from the swirling parameter yields that the orientation
of closed equipotential surfaces deviates away from the horizontal axis and
the corresponding disk does not longer possess the symmetry with respect to
the equatorial plane. These significant features could help to further
understand the equilibrium tori and geometrically thick accretion disks around
black holes in swirling universes.

###### pacs:

04.70.-s, 98.62.Mw, 97.60.Lf

##  I Introduction

The real astrophysical black holes in the galaxies at the center of many
galaxies are surrounded by an accretion disk. The matter flows in disks spiral
into the central celestial body and drop off their initial angular momentums
outwards together with releasing their gravitational potential energy into
heat. The radiation emitted from accretion disks carries many characteristic
information on black holes so that the corresponding observations and analysis
can help to detect black holes and to examine theories of gravity in the
strong field regime. The matter accretion near black holes is a highly
complicated dynamic process and its full description can be completed only by
resorting to high precise numerical simulations [1, 2, 3, 4, 5, 6]. In the
past few decades, a simple and analytical model of geometrically thick and
stationary tori orbiting black holes has been attracted a lot of attention. In
this theoretical model, the matter motion in thick disks is simplified to be
stationarily rotating and there is no actual accretion by the black hole [7,
8, 9, 10, 11, 12, 13, 14, 15], and the self-gravity of the fluid body is also
neglected. However, it is interesting that the stationarily rotating perfect
fluid tori known as Polish doughnuts satisfy the relativistic Euler equation.
Therefore, Polish doughnuts are always treated as a feasible initial condition
to numerically simulate accretion flows near black holes. Moreover, this
simplified model could capture key features of accretion disk configurations
around black holes. Thus, the geometrically thick equilibrium tori without
self-gravity have been extensively studied in many black hole spacetimes in or
beyond general relativity [16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28,
29, 30].

Recently, the geometrically thick equilibrium tori have been investigated in
the spacetime of a static and spherically symmetric black hole in the frame of
Born-Infeld teleparallel gravity, and it is found that there exists only a
single torus and the teleparallel gravity parameter makes the size of the
torus become small [31]. For the parameterised Rezzolla-Zhidenko black hole,
there are standard single-torus and non-standard double equilibrium tori
solutions [32, 33], and the transitions between them can occur through
regulating the specific angular momentum of the fluid. The non-standard double
equilibrium tori solutions are also found in the background of a dyonic black
hole with quasi-topological electromagnetism [34]. The magnetized equilibrium
tori around Kerr black hole with scalar hair have been respectively studied
within the constant angular momentum model [35] and the nonconstant angular
momentum model [36]. The features of geometrically thick equilibrium tori
deviating from those in the usual Kerr case [24] could be used to examine the
no-hair hypothesis by combining with future astronomical observations. The
stationarily and geometrically thick tori with the constant angular momentum
have been studied in the background of a static black hole in $f(R)$-gravity
with a Yukawa-like modification to the Newtonian potential [37], which shows
that the equilibrium tori have the notable configurations differed from those
in the usual static black holes in the general relativity. Moreover, the
magnetized equilibrium tori around a Kerr black hole are further studied with
the nonconstant specific angular momentum distribution model [38]. The
configurations of the geometrically thick tori have been also studied in the
background of a binary black hole system [39] and of compact object with a
quadrupole moment [40].

We here focus on a Schwarzschild black hole solution in swirling universes
[41] satisfied the vacuum Einstein field equations. It is constructed
analytically by exploiting the Ernst formalism [42, 43] with a transformation
embedding a Schwarzschild seed spacetime into a rotating background. The
rotating background can be interpreted as a gravitational whirlpool and its
frame dragging yields that a static seed solution turns into a stationary
metric [41]. This black hole spacetime is axially symmetric and non-
asymptotically flat, and its north and south hemispheres spin in opposite
directions. Especially, there is no closed time-like curves in such spacetimes
embedding in swirling universes, which means that it is less exotic than the
Gödel or the Taub-NUT spacetime [41]. The Schwarzschild black hole solution in
swirling universes has an event horizon at $r=2M$, which is the same as that
of a usual Schwarzschild black hole and does not depend on the swirl of the
background. However, the presence of the rotating background deforms the
horizon geometry so that it becomes more oblate, while the horizon area does
not change. Moreover, the rotating background yields that this black hole also
possesses an ergoregion, but its ergosurface is disconnected and consists of
two non-compact branches above and below the black hole. The study of
particles’ geodesics [44] shows that the geodesic equations can no longer be
decoupled because the black hole is immersed in the swirling universe and the
spacetime is no longer of Petrov type D, and a small deviation of the swirling
parameter from zero can change the qualitative features of the orbits
significantly as compared to those in the Schwarzschild spacetime. Moreover,
it is found that the swirling parameter drives the light rings outside the
equatorial plane and the contour of the shadow becomes a tilted oblate shape
[45]. These studies shed new light on understanding the Schwarzschild black
hole solution in swirling universes. The main motivation of this work is to
study the geometrically thick equilibrium tori around the Schwarzschild black
hole solution in swirling universes and to see what new properties of the
equilibrium tori in this case.

The work is organized as follows. In Sec. II, we briefly introduce the
Schwarzschild black hole solution in swirling universes [41] and Polish
doughnut model of thick accretion disks [7, 8, 9, 10, 11, 12, 13, 14, 15]. In
Sec. III, we investigate properties of equilibrium tori around the
Schwarzschild black hole in the swirling background. Finally, we will include
our conclusions in the last section.

##  II Schwarzschild black hole in swirling universes and Polish doughnut
model of thick accretion disks

Lets us first briefly review the Schwarzschild black hole solution in swirling
universes [41], which is a stationary, axially symmetric and non-
asymptotically flat black hole solution of the vacuum Einstein equations and
can be obtained through an Ehlers transformation on the Schwarzschild
solution. The metric has a form as follows [41]

${\rm d}s^{2}=F(r,\theta)\bigg{(}-f{\rm d}t^{2}+\frac{1}{f}{\rm
d}r^{2}+r^{2}{\rm
d}\theta^{2}\bigg{)}+\frac{r^{2}\sin^{2}\theta}{F(r,\theta)}\bigg{(}{\rm
d}\phi+4jrf\cos\theta{\rm d}t\bigg{)}^{2},$ (1)

with

$f=1-\frac{2M}{r},\quad\quad\quad\quad
F(r,\theta)=1+j^{2}r^{4}\sin^{4}\theta.$ (2)

where $M$ is the black hole mass and $j$ is the parameter related to the
spacetime rotation. The black hole’s event horizon lies at $r=2M$ and the
intrinsic singularity is located at $r=0$, which are the same as those of a
Schwarzschild black hole. The presence of the swirling background deforms the
horizon geometry and makes it more oblate, while the horizon area still remain
$A=16\pi M^{2}$, which is independent of the swirling parameter. Moreover, due
to the spacetime rotation, the black hole spacetime (1) also possesses an
ergoregion outside the event horizon $r=2M$, which is defined by $g_{tt}=0$.
The ergoregions are located not only in the vicinity of the event horizon, but
also close to the $z$-axis, for large values of $z$ [41]. Unlike in the Kerr
black hole spacetime, the ergosurface in the swirling background (1) is not a
closed surface and two disconnected non-compact patches distribute above and
below the black hole. With the increase of $j$, the ergosurface gets closer to
the horizon surface. The angular velocity $\Omega$ due to the frame dragging
is [41, 45]

$\Omega=\frac{d\phi}{dt}=-\frac{g_{t\phi}}{g_{\phi\phi}}=-4j(r-2M)\cos\theta,$
(3)

which means that the angular velocity $\Omega$ is in opposite directions in
the two hemispheres. In the spacetime (1), since the angular coordinate $\phi$
is periodic, an azimuthal curve $\gamma=\\{t=const;r=const;\theta=const\\}$ is
a closed curve with the invariant length
$s^{2}_{\gamma}=(2\pi)^{2}g_{\phi\phi}$. Thus, there are no closed timelike
curves since $g_{\phi\phi}$ is always non-negative, which means that the
Schwarzschild black hole in swirling universes is not plagued with geometric
pathologies including conical singularities and causality violation
originating from closed timelike curves. When the parameter $j=0$, the metric
(1) recovers the Schwarzschild solution, and when $M=0$, it reduces to the
swirling universe.

Let us now to study geometrically thick equilibrium tori around the
Schwarzschild black hole in swirling universes (1). With the test-fluid
approximation [7, 8, 9, 10, 11, 12, 13, 14, 15], the accretion flow in the
disk is assumed as a barotropic perfect fluid with the positive pressure and
the disk self-gravity is negligible. The fluid is assumed to be axisymmetric
and stationary in the spacetime and the corresponding physical variables
depend on only the coordinated $r$ and $\theta$. Moreover, the rotation of the
perfect fluid is restricted to be in the azimuthal direction. Under these
assumptions, the four-velocity of the perfect fluid can be expressed as
$u^{\mu}=(u^{t},0,0,u^{\phi})$ [7, 8, 9, 10, 11, 12, 13, 14, 15], and the
corresponding stress-energy tensor is

$\displaystyle T_{\mu\nu}=wu_{\mu}u_{\nu}+pg_{\mu\nu},$ (4)

where $w$ and $p$ are the enthalpy density and the pressure of the fluid,
respectively. Making use of the conservation for the perfect fluid
$\nabla_{\mu}T_{\nu}^{\mu}=0$, one can obtain [7, 8, 9, 10, 11, 12, 13, 14,
15]

$-\nabla_{i}\ln|u_{t}|+\frac{\Omega\nabla_{i}l}{1-l\Omega}=\frac{1}{w}\nabla_{i}p,\quad\quad\quad
i=r,\;\theta,$ (5)

where $l$ is the specific angular momentum of the fluid particle, and $u_{t}$
is the redshift factor and its form in the background spacetime (1) can be
expressed as

$u_{t}=\sqrt{\frac{g^{2}_{t\phi}-g_{tt}g_{\phi\phi}}{l^{2}g_{tt}+2g_{t\phi}l+g_{\phi\phi}}}.$
(6)

The quantity $\Omega$ is the angular velocity of the fluid particle

$\Omega=\frac{d\phi}{dt}=-\frac{g_{t\phi}+lg_{\phi\phi}}{g_{\phi\phi}+lg_{t\phi}}.$
(7)

According to the von Zeipel theorem [7, 8, 9, 10, 11, 12, 13, 14, 15], one has
$\Omega=\Omega(l)$ and can further obtain

$W-W_{in}=\ln\left|u_{t}\right|-\ln\left|\left(u_{t}\right)_{\mathrm{in}}\right|-\int_{l_{\mathrm{in}}}^{l}\left(\frac{\Omega}{1-\Omega
l^{\prime}}\right)dl^{\prime},$ (8)

where the subscript “in” denotes that the quantity is evaluated at the inner
edge of the disk. The potential $W$ is given by

$W-W_{in}=\int_{0}^{p}\frac{dp}{w},$ (9)

which determines the shapes of equipotential surfaces in the disk. Assuming
that a barotropic equation of state takes a form $P=Kw^{\kappa}$ with two
constants $K$ and $\kappa$ [46], from Eq. (9), one can easily obtain

$W-W_{in}+\frac{\kappa}{\kappa-1}\frac{p}{w}=0.$ (10)

Thus, once the potenial $W$ and the specific enthalpy $h=w/\rho$ are given,
one can get the enthalpy density $w$ and the rest-mass density $\rho$
distribution in the disk. For simplicity, one can set $h=1$ as in Ref. [46].
Moreover, we here focus on only a simple case where the fluid has a constant
specific angular momentum $l$, which means the fluid angular velocity $\Omega$
in Eq. (7) becomes a function only related to the spacetime metric. In this
simple model, the potential $W$ can be further simplified as $W=\ln|u_{t}|$.
Such stationarily rotating perfect-fluid tori are known as “Polish doughnuts”.

##  III Geometrically thick equilibrium tori around the Schwarzschild black
hole in swirling universes

Thick equilibrium tori are allowed only in the spacetimes admitting closed
equipotential $W$ surfaces. The outermost closed surface crosses itself in the
cusp(s). These cusps enable the outflow of matter from the torus because of
the breaking of hydrostatic equilibrium. In the usual Schwarzschild spacetime,
there is only an inner cusp enabling an accretion onto the central black hole.
In the spacetimes with repulsive parameters including cosmological constant
[19, 20, 21], there also exists the outer cusp on the equipotential surfaces
at which the gravitational attraction is just balanced by the corresponding
repulsion. The outer cusp enables the excretion where matter outflows from the
torus into the outer space. In the case of a Schwarzschild black hole in
swirling universes, the potential $W$ has a form

$W(r,\theta)=\frac{1}{2}\ln\frac{(r-2M)r^{2}\sin^{2}\theta(1+j^{2}r^{4}\sin^{4}\theta)}{r^{3}[1+4jl(r-2M)\cos\theta]^{2}\sin^{2}\theta-(r-2M)(1+j^{2}r^{4}\sin^{4}\theta)^{2}l^{2}}.$
(11)

The features of the equipotential surfaces of fluid in the Schwarzschild black
hole spacetime with the swirling parameter $j$ are fully determined by the
number of the local extrema of the function $W(r,\theta)$ and their mutual
behavior. The local extrema of $W(r,\theta)$ can be obtained by the conditions

$\displaystyle\frac{\partial W(r,\theta)}{\partial
r}=0,\quad\quad\quad\frac{\partial W(r,\theta)}{\partial\theta}=0.$ (12)

Unlike in the usual Schwarzschild case, the local extrema of $W(r,\theta)$ is
not located in the equatorial plane except the case $j=0$, which is
understandable because the spacetime (1) possesses only the odd $Z_{2}$
(north-south) symmetry. Moreover, the potential $W$ also has the symmetry
under the transformation $(j,\theta)\leftrightarrow(-j,\;\pi-\theta)$, so we
here consider only the case $j\geq 0$ without loss of generality. The number
of the local extrema of the function $W(r,\theta)$ is determined by the
parameter $j$ and the specific angular momentum $l$. Solving numerically Eq.
(12), we can obtain the specific angular momentum $l$ for the particles moving
along circular orbits in the spacetime (1). Fig. 1 exhibits the effects of the
swirling parameter $j$ on the changes of the specific angular momentum $l$
with circular orbit radii $r$. As $j=0$, one can find that there is only a
minimum $l_{\rm ms}=\frac{3}{2}\sqrt{6}$ at $r=r_{\rm ms}$, which recovers the
results obtained in the usual Schwarzschild case. With the increase of $j$, we
find that the function $l$ has two local extrema, i.e., the minimum $l_{\rm
msi}$ and the maximum $l_{\rm mso}$, which respectively correspond to the
inner and outer marginally stable circular geodesics. The rising (descending)
part of $l$ corresponds to stable (unstable) circular geodesics. As the
swirling parameter increases up to $j>j_{c1}=0.000378$, the function $l$
becomes monotonically decreasing, which means that there exist only unstable
circular geodesics in this case. Therefore, as in the Schwarzschild-de Sitter
spacetime, a toroidal configuration of finite size can be obtained only for
values of the specific angular momentum $l_{0}$ that satisfy

Figure 1: Effects of the swirling parameter $j$ on the specific angular
momentum $l$ for the particles moving along circular orbits in the spacetime
of a Schwarzschild black hole in swirling universes. Here we set $M=1$.

$\displaystyle l_{\rm msi}<l_{0}<l_{\rm mso},$ (13)

and there is no the toroidal configuration around the black hole (1) as the
swirling parameter $j>0.000378$. Therefore, the swirling parameter $j$ plays a
crucial role in the existence of the disk solutions. The slight increase of
$j$ could yield that the possibility of having solutions decreases
dramatically. This means that this disk’s model exists only for the relatively
small swirling parameter.

Figure 2: Effects of the swirling parameter $j$ on the effective potential $W$
for the particles moving along circular orbits in the spacetime of a
Schwarzschild black hole in swirling universes for fixed specific angular
momentum $l=4$. Here we set $M=1$.

Fig. 2 presents the effects of $j$ on numbers of the local extrema of
$W(r,\theta)$ for $l=4$ as the fluid particles move along circular orbits. In
the non-zero $j$ cases, we find that $W(r,\theta)$ has three local extrema as
$j<j_{c2}=0.000132$. Two local maximum points respectively correspond to the
positions of the inner cusp $r_{\rm ci}$ and the outer cusp $r_{\rm co}$,
where the equipotential surface has a self-crossing point in the $(r,\theta)$
plane. The outer cusp disappears in the Schwarzschild spacetime but emerges in
the de Sitter spacetimes with the non-zero cosmological constant. The presence
of the outer cusp represents that the swirling parameter yields some repulsive
effects similar to the cosmological constant. Fig. 2 shows the casse $W(r_{\rm
ci})>W(r_{\rm co})$, which means that the fluid particles with $W(r_{\rm
co})<W<W(r_{\rm ci})$ moving along the circular orbit can be excreted from the
torus to the spatial infinity rather than be accreted into the black hole.
This does not appear in the usual Schwarzschild black hole case. The local
minimum point is located at the “centre” of the torus $r_{\rm max}$ where the
internal pressure has its local maximum in the torus. As $j\geq
j_{c2}=0.000132$, we find that the function $W$ has no local extremum. These
are also shown in Fig. 3, in which the equipotential surfaces of the effective
potential $W$ are given in Cartesian coordinates $\tilde{x}[M]$ and
$\tilde{z}[M]$.

Figure 3: Equipotential surfaces of the effective potential $W$ shown with
Cartesian coordinates $\tilde{x}=r\sin\theta\cos\phi$ and
$\tilde{z}=r\cos\theta$ for different swirling parameters $j$. Here we set
$M=1$ and $l=4$.

As $j=0$, the equipotential surfaces possess the equatorial symmetry. However,
the presence of the swirling parameter $j$ leads to that the equipotential
surfaces do not have the equatorial symmetry and the closed equipotential
surfaces orient away from the horizontal axis. This can be understood by the
asymmetry with respect to the equatorial plane originating from the swirl of
the background spacetime. The disk deviation from the horizontal axis
noticeably increases with the parameter $j$. The orientation of equipotential
surfaces deviating from the horizontal axis also appears in the case of
accelerating black holes [22], which can be regarded as a common feature of
equipotential surfaces in the spacetimes only with the odd $Z_{2}$ symmetry.
Moreover, the distance between the center and outer cusp points changes as a
monotonic decreasing function of $j$. Thus, the increase of $j$ results in the
smaller disk structure. As $j$ increases up to a threshold that the center and
the cusp point overlaps, there is no closed equipotential surfaces and then
the disk structure vanishes. This means that the larger disk structure is only
for the small swirl of the spacetime.

Figure 4: Contour map of the rest-mass density of the disk for different
swirling parameters $j$. Here we set $M=1$ and $l=4$.

In Fig. 4, we also present the distributions of the rest-mass density in
equilibrium tori by solving Eq. (10) with the equation of state
$p=Kw^{\kappa}$ through setting $\kappa=4/3$ and $w_{r_{\rm max}}=1$. The
coefficient $K$ is calculated by $K=W_{r_{\rm max}}-W_{\rm in}$, where
$W_{r_{max}}$ and $W_{\rm in}$ are respectively the values of $W$ at the
centre $r=r_{\rm max}$ of the equilibrium torus and at the inner cusp.
Comparing Fig. 4 with Fig. 3, it is easy to observe that the distributions of
mass density in equilibrium tori are similar to the distribution of the
effective potential $W$. With the increase of $j$, one can find that the
excretion of matter near the black hole becomes strong. These results could
help to further understand equilibrium tori and geometrically thick accretion
disks around the Schwarzschild black hole in swirling universes.

##  IV Summary

We have studied geometrically thick non-self gravitating equilibrium tori
orbiting a Schwarzschild black hole in swirling universes. This solution is
non-asymptotically flat, and its north and south hemispheres spin in opposite
directions. Due to repulsive effects arising from the swirl of the background
spacetime, the equilibrium torus exists only in the case with the small
swirling parameter. In the case of existing equilibrium tori, there are the
inner cusp $r_{\rm ci}$ and the outer cusp $r_{\rm co}$ for equipotential
surfaces and one has $W(r_{\rm ci})>W(r_{\rm co})$, meaning that the fluid
particles with $W(r_{\rm co})<W<W(r_{\rm ci})$ moving along the circular orbit
can be excreted from the torus to the spatial infinity rather than be accreted
into the black hole, which does not appear in the usual Schwarzschild black
hole case. With the increase of the swirling parameter $j$, one can find that
the disk structure becomes small and the excretion of matter near the black
hole becomes more strong. Moreover, the odd $Z_{2}$ symmetry of spacetimes
originating from the swirling parameter yields that the orientation of closed
equipotential surfaces deviates away from the horizontal axis so that the disk
does not longer possess the symmetry with respect to the equatorial plane.
Thus, the swirling parameter $j$ plays a crucial role in the existence and
behavior of the equilibrium tori around the black hole. These results could
help to understand the equilibrium tori and geometrically thick accretion
disks around black holes in swirling universes, and to further distinguish
such black holes from others by analyzing the observing features of accretion
disks.

###### Acknowledgements.

This work was supported by the National Key Research and Development Program
of China (Grant No. 2020YFC2201400) and National Natural Science Foundation of
China (Grant Nos. 12275079 and 12035005).

## References

  * [1] O. Porth et al. [Event Horizon Telescope], The Event Horizon General Relativistic Magnetohydrodynamic Code Comparison Project, Astrophys. J. Suppl. 243, 26 (2019).
  * [2] L.G. Fishbone and V. Moncrief, Relativistic fluid disks in orbit around Kerr black holes, Astrophys. J. 207, 962 (1976).
  * [3] J.A. Font, Numerical Hydrodynamics and Magnetohydrodynamics in General Relativity, Living Rev. Rel. 11, 7 (2008).
  * [4] V. Roy, S. Pu, L. Rezzolla, and D. Rischke, Analytic Bjorken flow in one-dimensional relativistic magnetohydrodynamics, Phys. Lett. B 750, 45-52 (2015).
  * [5] O. Porth, H. Olivares, Y. Mizuno, Z. Younsi, L. Rezzolla, M. Moscibrodzka, H. Falcke, and M. Kramer, The Black Hole Accretion Code, Comput. Astrophys. 4, 1 (2017); arXiv:1611.09720 [gr-qc].
  * [6] L. Anton, O. Zanotti, J.A. Miralles, J.M. Marti, J.M. Ibanez, J.A. Font, and J.A. Pons, Numerical 3+1 General Relativistic Magnetohydrodynamics: A Local Characteristic Approach, Astrophys. J. 637, 296-312 (2006).
  * [7] M.A. Abramowicz, The Relativistic von Zeipel’s Theorem, Acta Astron. 21, 81 (1971).
  * [8] M.A. Abramowicz, Theory of Level Surfaces Inside Relativistic Rotating Stars II, Acta Astron. 24, 45 (1971).
  * [9] M. Kozlowski, M. Jaroszynski, and M.A. Abramowicz, The analytic theory of fluid disks orbiting the Kerr black hole, Astron. Astrophys. 63, 209 (1978).
  * [10] M. Abramowicz, M. Jaroszynski, and M. Sikora, Relativistic Accreting Disks, Astron. Astrophys. 63, 221 (1978).
  * [11] M. Jaroszynski, M.A. Abramowicz, and B. Paczynski, Supercritical Accretion Disks, Acta Astron. 30, 1 (1980).
  * [12] M.A. Abramowicz, M. Calvani, and L. Nobili, Thick Accretion Disks with Supercritical Luminosities, Astrophys. J. 242, 772 (1980).
  * [13] B. Paczyńsky and P.J. Wiita, Thick Accretion Disks and Supercritical Luminosities, Astron. Astrophys. 88, 23 (1980).
  * [14] B. Paczynski and M.A. Abramowicz, A model of a thick disk with equatorial accretion, Astrophys. J. 253, 897 (1982).
  * [15] B. Paczynski, Thick accretion disks around black holes (Karl-Schwarzschild-Vorlesung 1981), Mitteilungen der Astronomischen Gesellschaft Hamburg, 57, 27 (1982).
  * [16] S. Chakraborty, Equilibrium configuration of perfect fluid orbiting around black holes in some classes of alternative gravity theories, Class. Quant. Grav. 32, 075007 (2015).
  * [17] I. Banerjee, S. Chakraborty, and S. SenGupta, Decoding signatures of extra dimensions and estimating spin of quasars from the continuum spectrum, Phys. Rev. D 100, 044045 (2019).
  * [18] X. Liu, S. Chen, and J. Jing, Polarization distribution in the image of a synchrotron emitting ring around a regular black hole, Sci. China Phys. Mech. Astron. 65, 120411 (2022).
  * [19] Z. Stuchlik, P. Slany, and J. Kovar, Pseudo-Newtonian and general relativistic barotropic tori in Schwarzschild-de Sitter spacetimes, Class. Quant. Grav. 26, 215013 (2009).
  * [20] L. Rezzolla, O. Zanotti, and J.A. Font, Dynamics of thick discs around Schwarzschild-de Sitter black holes, Astron. Astrophys. 412, 603 (2003).
  * [21] P. Slaný and Z. Stuchlík, Relativistic thick discs in the Kerr-de Sitter backgrounds, Class. Quantum Grav. 22 3623-3651 (2005).
  * [22] S. Faraji, A. Trova, and V. Karas, Magnetized relativistic accretion disk around a spinning, electrically charged, accelerating black hole: Case of the C metric, Phys. Rev. D 105, 103017 (2022).
  * [23] S.S. Komissarov, General relativistic MHD simulations of monopole magnetospheres of black holes, Mon. Not. Roy. Astron. Soc. 350, 1431 (2004).
  * [24] S.S. Komissarov, Magnetized Tori around Kerr Black Holes: Analytic Solutions with a Toroidal Magnetic Field, Mon. Not. Roy. Astron. Soc. 368, 993 (2006).
  * [25] J.A. Font and F. Daigne,  On the stability of thick accretion disks around black holes, Astrophys. J. Lett. 581, L23-L26 (2002).
  * [26] F. Daigne and J.A. Font, The runaway instability of thick discs around black holes. II. non constant angular momentum discs, Mon. Not. Roy. Astron. Soc. 349, 841 (2004).
  * [27] J.A. Font and F. Daigne, The Runaway instability of thick discs around black holes. I. The Constant angular momentum case, Mon. Not. Roy. Astron. Soc. 334, 383 (2002).
  * [28] M.A. Abramowicz and P.C. Fragile, Foundations of Black Hole Accretion Disk Theory, Living Rev. Rel. 16, 1 (2013).
  * [29] S. Kazempour, Y. Zou, and A. Akbarieh, Analysis of accretion disk around a black hole in dRGT massive gravity, Eur. Phys. J. C 82, 190 (2022).
  * [30] M.M. Armijo, Review: Accretion Disk Theory, arXiv:1203.6851 [astro-ph.HE].
  * [31] S. Bahamonde, S. Faraji, E. Hackmann, and C. Pfeifer, Thick accretion disk configurations in the Born-Infeld teleparallel gravity, Phys. Rev. D 106, 084046 (2022).
  * [32] M. Cassing and L. Rezzolla,  Equilibrium non-self-gravitating tori around black holes in parametrized spherically symmetric space-times, Mon. Not. Roy. Astron. Soc. 522, 2415 (2023).
  * [33] L. Rezzolla and A. Zhidenko, Accurate mapping of spherically symmetric black holes in a parameterised framework, Phys. Rev. D 90, 084009 (2014).
  * [34] X. Zhou, S. Chen, and J. Jing, Geometrically thick equilibrium tori around a dyonic black hole with quasi-topological electromagnetism, arXiv:2307.01996 [gr-qc].
  * [35] S. Gimeno-Soler, J. Font, C. Herdeiro, and E. Radu, Magnetized accretion disks around Kerr black holes with scalar hair: Constant angular momentum disks, Phys. Rev. D 99, 043002 (2019).
  * [36] S. Gimeno-Soler, J. Font, C. Herdeiro, and E. Radu, Magnetized accretion disks around Kerr black holes with scalar hair: Nonconstant angular momentum disks, Phys. Rev. D 104, 103008 (2021).
  * [37] A. Cruz-Osorio, S. Gimeno-Soler, J. Font, M. De Laurentis, and S. Mendoza, Magnetized discs and photon rings around Yukawa-like black holes, Phys. Rev. D 103, 124009 (2021).
  * [38] S. Gimeno-Soler, O.M. Pimentel, F.D. Lora-Clavijo, A. Cruz-Osorio, and J.A. Font, Magnetised tori with magnetic polarisation around Kerr black holes: variable angular momentum discs, arXiv:2303.15867 [astro-ph.HE].
  * [39] S.V. Chernov, Thick Fluid Disks Around Binary Black Holes, Grav. Cosmol. 29, 437-442 (2023).
  * [40] J. Memmen and V. Perlick, Geometrically thick tori around compact objects with a quadrupole moment, Class. Quantum Grav. 38, 135002 (2021).
  * [41] M. Astorino, R. Martelli, and A. Viganò, Black holes in a swirling universe, Phys. Rev. D 106, 064014 (2022); arXiv:2205.13548 [gr-qc].
  * [42] F.J. Ernst, New formulation of the axially symmetric gravitational field problem, Phys. Rev. 167, 1175 (1968).
  * [43] F.J. Ernst, New Formulation of the Axially Symmetric Gravitational Field Problem. II, Phys. Rev. 168, 1415 (1968).
  * [44] R. Capobianco, B. Hartmann, and J. Kunz, Geodesic Motion in a Swirling Universe: The complete set of solutions, arXiv:2312.17347 [gr-qc].
  * [45] Z.S. Moreira, C.A.R. Herdeiro, and L.C.B. Crispino, Twisting shadows: light rings, lensing and shadows of black holes in swirling universes, arXiv:2401.05658 [gr-qc].
  * [46] S. Gimeno-Soler and J.A. Font, Magnetised Polish doughnuts revisited, Astron. Astrophys. 607, A68 (2017).


# Prompt Learning for Action Recognition

Xijun Wang*, Ruiqi Xian*, Tianrui Guan, Dinesh Manocha
University of Maryland, College Park, MD, USA
{xijun, rxian, rayguan<EMAIL_ADDRESS>

###### Abstract

We present a new general learning approach for action recognition, Prompt
Learning for Action Recognition (PLAR), which leverages the strengths of
prompt learning to guide the learning process. Our approach is designed to
predict the action label by helping the models focus on the descriptions or
instructions associated with actions in the input videos. Our formulation uses
various prompts, including optical flow, large vision models, and learnable
prompts to improve the recognition performance. Moreover, we propose a
learnable prompt method that learns to dynamically generate prompts from a
pool of prompt experts under different inputs. By sharing the same objective,
our proposed PLAR can optimize prompts that guide the model’s predictions
while explicitly learning input-invariant (prompt experts pool) and input-
specific (data-dependent) prompt knowledge. We evaluate our approach on
datasets consisting of both ground camera videos and aerial videos, and scenes
with single-agent and multi-agent actions. In practice, we observe a
$3.17-10.2\%$ accuracy improvement on the aerial multi-agent dataset, Okutamam
and $0.8-2.6\%$ improvement on the ground camera single-agent dataset,
Something Something V2. We plan to release our code on the WWW.

**footnotetext: These authors contributed equally to this work

##  1 Introduction

Action recognition, the task of understanding human activities from video
sequences, is a fundamental problem in computer vision. This problem arises in
many applications: video surveillance, human-computer interaction, sports
analysis, human-robot interaction, etc. There has been considerable work on
these problems in recent years, driven by the availability of large-scale
video datasets and advancements in deep learning techniques including two-
stream Convolutional Neural Network (CNN) (Simonyan and Zisserman,, 2014),
Recurrent Neural Network (RNN) (Sun et al.,, 2017), and Transformer-based
methods (Vaswani et al.,, 2017; Li et al., 2022b, ). These methods have
achieved considerable success in extracting discriminative features from video
sequences, leading to significant improvements in action recognition accuracy
for ground videos and aerial videos.

Despite the recent progress in action recognition algorithms, the success of
most existing approaches relies on extensive labeled training data followed by
a purely supervised learning paradigm that mainly focuses on a backbone
architecture design. In this paper, our goal is to design new methods for
video action recognition using prompt learning. Prompt-based techniques (Liu
et al.,, 2023) have been proposed for natural language processing tasks to
circumvent the issue of lack of labeled data. These learning methods use
language models that estimate the probability of the text and use this
probability to predict the label thereby reducing or obviating the need for
large labeled datasets. In the context of action recognition, prompt learning
offers the potential to design better optimization strategies by providing
high-level texture descriptions or instructions associated with actions. These
prompts can guide the learning process, and enable the model to capture
discriminative spatio-temporal patterns effectively, resulting in better
performance.

Many prompt learning-based techniques have been proposed for few-shot action
recognition (Shi et al.,, 2022), zero-shot action recognition (Sato et al.,,
2023; Wang et al., 2021a, ), and ordinal action understanding. (Shi et al.,,
2022) proposes knowledge prompting, which leverages commonsense knowledge of
actions from external resources to prompt a powerful pre-trained vision-
language model for few-shot classification. (Sato et al.,, 2023) presents a
unified, user prompt-guided zero-shot learning framework using a target
domain-independent skeleton feature extractor, which is pretrained on a large-
scale action recognition dataset. Bridge-Prompt (Li et al., 2022a, ) proposes
a prompt-based framework to model the semantics across adjacent actions from a
series of ordinal actions in instructional videos. Our goal is to apply these
techniques for video action recognition.

##### Main Contributions:

We present a general prompt learning approach that alleviates the burden of
objective optimization by integrating prompt-based learning into the action
recognition pipeline. Our approach is designed to enhance the model’s ability
to process customized inputs by utilizing prompt tokens. These prompt tokens
can be either predefined templates or learnable tokens that include
information specific to video action recognition. Our formulation leverages
prompts, which are easier to focus on the interest targets, and enables the
learning of complex visual concepts.

In our prompt learning paradigm, we explore and discuss various types of
prompt, including optical flow and large vision models. In addition, we
present a learnable prompt, which dynamically generate prompts from a pool of
prompt experts under different inputs. Our goal is to optimize prompts that
guide the model’s predictions while explicitly learning input-invariant
(prompt experts) and input-specific (data-dependent) prompt knowledge. We
validate this generalization by performing extensive evaluations on datasets
comprising of ground camera videos and aerial videos, on scenarios involving
single-agent and multi-agent actions. We demonstrate that our technique can
improve the performance and enhance the generalization capabilities of video
action recognition models in different scenarios. The novel components of our
work include:

Figure 1: Task: We use prompt learning for action recognition. Our method
leverages the strengths of prompt learning to guide the learning process by
helping models better focus on the descriptions or instructions associated
with actions in the input videos. We explore various prompts, including
optical flow, large vision models, and proposed learnable prompts to improve
recognition performance. The recognition models can be CNNs or Transformers.

  1. 1.

Present a general learning approach to use prompt learning and auto-regressive
techniques for action recognition.

  2. 2.

Propose a new learnable prompt method, which can guide the model’s predictions
while explicitly learning input-invariant (prompt experts) and input-specific
(data-dependent) prompt knowledge.

  3. 3.

To the best of our knowledge, ours is the first approach to explore the
possibility of using large vision models as the prompt to instruct the action
recognition task.

  4. 4.

Through empirical evaluations, we demonstrate the potential and effectiveness
of prompt learning techniques for action recognition tasks. Specifically, we
observed 3.17-10.2% accuracy improvement on the aerial multi-agent dataset,
Okutama. Moreover, we observe a 0.8-2.6% accuracy improvement on the ground
camera single-agent video dataset, Something Something V2 (Goyal et al.,,
2017).

##  2 Related Works

###  2.1 Action Recognition

Human action recognition, i.e., recognizing and understanding human actions,
is crucial for a number of real-world applications. Recently, many deep
learning architectures have been proposed to improve the performance. At a
broad level, they can be classified into three categories:

Two-stream 2D Convolutional Neural Network: The use of two-stream 2D
Convolutional Neural Networks (CNNs) has been widely explored in the field of
action recognition (Simonyan and Zisserman,, 2014; Karpathy et al.,, 2014;
Wang et al.,, 2015; Sánchez et al.,, 2013; Chéron et al.,, 2015; Girdhar et
al.,, 2017). They capture spatial and temporal information separately to
process video data. (Zong et al.,, 2021) extended the two-stream CNN to a
three-stream CNN by incorporating a motion saliency stream to enhance the
representation of salient motion information. (Piergiovanni and Ryoo,, 2019)
proposed a trainable flow layer that eliminates the need for optical flow
computation while capturing motion information.

3D CNN-based methods: Several methods in the literature have utilized 3D
Convolutional Neural Networks (CNNs) for action recognition (Tran et al.,,
2015; Ji et al.,, 2012; Zhang et al.,, 2020; Li et al., 2020a, ; Carreira and
Zisserman,, 2017). These approaches focus on leveraging the spatio-temporal
characteristics of video data at the same time through 3D convolutions. (Zhou
et al.,, 2020) analyzed the spatio-temporal fusion in 3D CNN from a
probabilistic view. (Yang et al.,, 2020) introduced a generic Temporal Pyramid
Network (TPN) to effectively model speed variations in actions at the feature
level. (Piergiovanni and Ryoo,, 2021) investigated the impact of viewpoint
variations on recognition performance. (Hussein et al.,, 2019) proposed multi-
scale temporal-only convolutions to handle large variations in temporal
extents within complex actions. (Fayyaz et al.,, 2021) addressed the
computational cost by dynamically adapting the temporal feature resolution
within 3D CNNs.

Transformer-based approaches: Many techniques have been proposed or
transformer-based action recognition (Arnab et al.,, 2021; Bertasius et al.,,
2021; Wang et al., 2021b, ).(Tong et al.,, 2022) introduced a Video Masked
Autoencoder (Video MAE) with an attention mechanism to address the challenges
of information redundancy and temporal correlation in video data, improving
the efficiency of action recognition. (Yang et al.,, 2022) proposed an
attention gate that facilitates interactions between frame inputs and hidden
states, enabling the aggregation of global inter-frame features across the
temporal domain through recurrent execution. (Mazzia et al.,, 2022) leveraged
2D pose representations within a fully self-attentional architecture over
short temporal periods, enabling low latency and high throughput in
recognition tasks. (Li et al., 2022b, ) implemented the seminal idea of
multiscale feature hierarchies with transformer models for video and image
recognition.

Although these methods have had good success on the ground data and YouTube
videos, they cannot achieve a similar level of accuracy on videos captured
using Unmanned Aerial Vehicles (UAVs) Wang et al., (2023); Xian et al., 2023a
. Compared to ground or YouTube videos, UAV videos have their unique
characteristics like small resolution, scale and size variations, and moving
cameras. (Xian et al., 2023b, ) proposed a mutual information-based feature
alignment and sampling method to extract spatial-temporal features
corresponding to human actors for better recognition accuracy. (Kothandaraman
et al.,, 2022) introduced Fourier transformation into attention modules to
aggregate the motion salience.

###  2.2 Prompt Learning

The concept of prompt learning was introduced by (Petroni et al.,, 2019) and
has since gained significant attention in the field of Natural Language
Processing (NLP) (Brown et al.,, 2020; Jiang et al.,, 2020; Li and Liang,,
2021; Liu et al.,, 2023; Tian et al.,, 2020). The fundamental idea behind
prompt learning is to treat pre-trained language models like BERT or GPT as
knowledge bases, enabling their application in downstream tasks. Early
studies, such as those by (Petroni et al.,, 2019; Poerner et al.,, 2019),
focused on designing manually crafted prompts to enhance the performance of
language models. Subsequently, researchers like (Shin et al.,, 2020; Jiang et
al.,, 2020) aimed to automate this process using cost-effective, data-driven
approaches. More recently, some works (Han et al.,, 2022; Lester et al.,,
2021; Zhong et al.,, 2021) have attempted to learn continuous prompts instead
of searching for discrete prompts.

While prompt learning has garnered significant attention in the field of
Natural Language Processing (NLP), its application in computer vision is still
a relatively new and emerging research direction. Only recently have
researchers started exploring prompt learning techniques in computer vision
tasks (Rao et al.,, 2022; Ju et al.,, 2022; Zhou et al.,, 2021). Pretrained
VLMs (Jia et al.,, 2021; Radford et al.,, 2021) utilize manual prompts for
zero-shot inference on the downstream tasks. (Zhou et al.,, 2021) proposed
Context Optimization (CoOp) to extend continuous prompt representations to the
vision domain, enabling the automatic learning of task-relevant prompts. (Wang
et al.,, 2022) focused on prompt optimization for continual learning,
explicitly managing task-specific knowledge while maintaining plasticity.
(Zhou et al.,, 2022) introduced conditional CoOp, which involves learning a
lightweight neural network to generate input-conditional tokens for each
image. (Lu et al.,, 2022) explored the generalization capabilities of prompt
learning by learning the distribution of diverse prompts. (Ge et al.,, 2022)
addressed the challenge of distribution shifts in unsupervised domain
adaptation by learning both domain-agnostic and domain-specific prompts.
However, the majority of these works have primarily focused on image-level
tasks, leaving the exploration of prompt learning for video tasks relatively
unexplored. In this paper, we propose a general learning paradigm to
investigate the effectiveness of prompt learning specifically for video
understanding, with a specific focus on action recognition in both
ground/YouTube and aerial videos. Our goal is to bridge the gap and extend the
benefits of prompt learning to the domain of video understanding tasks.

##  3 Our Approach

The problem of video action recognition can be broadly classified into single-
agent and multi-agent action recognition. And according to different data
types, there are aerial video action recognition and ground camera action
recognition. Typically, they all involve several steps. Taking the
transformer-based methods for example, first, the input video or image
sequence is processed to extract relevant features, such as movement patterns,
appearance, or spatial-temporal information. These features are then fed into
a reasoning model inference action label. Prompt learning can help the first
step better handle feature extraction.

We denote the input as $X_{i}=\\{x_{1},x_{2},...,x_{m}\\},i\in[1,N]$, where
$x_{j}$ is the $j_{th}$ frame in the $i_{th}$ video, $m$ is the total frame
number, and $N$ is the total number of videos. The overall approach predicts
the action categories by using model $f(X_{i})$, which can be CNNs or
Transformers. As shown in Figure 2, taking transformer-based methods as an
example, we follow the same scheme to extract the features, followed by using
the reasoning process to predict the action labels. We also present a prompt-
learning-based encoder to help better extract the feature and then propose an
auto-regressive temporal reasoning algorithm for recognition models for
enhanced inference ability.

Specifically, given an action model:

$f=f_{a}\circ f_{e}([X,P]),$ (1)

where $f_{e}$ is the _prompt-learning-based input encoder_ , $P$ is the
prompt, and $f_{a}$ is the _auto-regressive-based temporal reasoning_ model,
which is used for the temporal dimension.

Figure 2: Overview action recognition framework: We use transformer-based
action recognition methods as an example. We designed a prompt-learning-based
encoder to help better extract the feature and use our auto-regressive
temporal reasoning algorithm for recognition models for enhanced inference
ability.

###  3.1 Prompt Learning based Input Encoder

For the first part of the input encoder, inspired by these prompt-based
techniques in NLP, we present a new general prompt-learning-based input
encoder for action recognition. Our formulation leverages the strengths of
prompt learning to guide the learning process by providing high-level texture
descriptions or instructions associated with actions in the inputs. We use
this to alleviate the burden of models’ optimization by helping models better
focus on the active region.

Prompts can enhance the model’s ability to process customized inputs by
utilizing prompt tokens. By leveraging prompts, models are easier to focus on
the interest targets, and prompt learning enables the model to learn complex
visual concepts and capture discriminative spatio-temporal patterns
effectively. Specifically, our prompts can be either predefined templates
(non-learnable prompt: optical flow, large vision models) or learnable tokens
(learnable prompt) that include task-specific information. They can be used
either alone or in combination.

####  3.1.1 Non-Learnable Prompt

Non-Learnable prompts make use of statistical methods (e.g. optical flow) or
existing powerful large vision models, which can offer reliable prompts
without training.

##### Optical Flow Prompt

Optical flow is a fundamental concept in computer vision that involves
estimating the motion of objects within a video sequence. It represents the
apparent motion of pixels between consecutive frames, providing valuable
information about the movement of objects and their relative velocities.

For frame $x_{i}$ and frame $x_{j}$, the optical flow is:

$o_{ij}=O(x_{i},x_{j}),x_{i}\in clip_{i},x_{j}\in clip_{j},$ (2)

$clip_{i}$ and $clip_{j}$ are two adjacent video clips, and each clip contains
several frames. This formulation is more efficient because it avoids many
calculations for every frame. Therefore, the input with optical flow prompt
becomes:

$[X,P]=\\{x_{i}*o_{ij}|\>i,j\in[1,m]\\}$ (3)

We use $[X,P]$ to replace the original $X$ in video action recognition.

##### Large Vision Model Prompt

Recently, large models have been attracting more attention for NLP and other
applications. These large models are considered powerful since they are
trained on huge amounts of data and don’t need to be finetuned on new tasks as
an auxiliary input (i.e. prompt). Our goal is to use these large models to
generate prompts (e.g. mask, bbox) for video action recognition.

One popular work is the Segment Anything Model (SAM (Kirillov et al.,, 2023)),
which can segment any object in an image given only some prompts like a single
click or box. SAM is trained on a dataset of 11 million images and 1.1 billion
masks. SAM can segment objects with high accuracy, even when they are new or
have been modified from the training data. SAM generalizes to new objects and
images without the need for additional training, so we don’t need to finetune
the model on our dataset. For some frames in a video clip, we generate a
segmentation mask using a large vision model, SAMKirillov et al., (2023).
Next, these masks are used as prompts and fused with input frames to optimize
the recognition model. Specifically, for frame $x_{i}$, the output from SAM
is:

$p_{i}=SAM(x_{i},box/point),x_{i}\in clip_{i}$ (4)

$clip_{i}$ is a video clip containing a few frames,

$[X,P]=\\{x_{i}*p_{i}|\>i\in[1,m]\\}$ (5)

We use $[X,P]$ to replace the original $X$ in video action recognition.

####  3.1.2 Learnable Prompt

To better adapt to the input data, we also propose a learnable prompt, which
learns to dynamically generate prompts from a pool of prompt experts under
different inputs. Prompt experts are learnable parameters that can be updated
from the training process. As shown in Figure 3, in our design, we use input-
invariant (prompt experts) and input-specific (data dependent) prompts. The
input-invariant prompts contain task information, and we use a dynamic
mechanism to generate input-specific prompt for different inputs.

Figure 3: Learnable prompt: learning input-invariant (prompt experts) and
input-specific (data dependent) prompt knowledge. The input-invariant prompts
will be updated from all the inputs, which contain task information, and we
use a dynamic mechanism to generate input-specific prompts for different
inputs. Add/Mul means element-wise operations.

There are different actions and domains (different video sources) for
different videos, so it’s challenging to learn a single general prompt for all
videos. Therefore, we design an input-invariant prompt experts pool, which
contains $l$ learnable prompt.

$P=\\{P_{1},...,P_{l}\\},$ (6)

which is learnable and will be updated from all the inputs. For a specific
input $X^{*}$,

$P^{*}=Matmul(\sigma(FC(X^{*})),P),$ (7)

We first use an FC layer and sigmoid function to get dynamic weights. Then we
apply these dynamic weights to the input-invariant prompt pool to get a
customized prompt $P^{*}$ for $X^{*}$.

$x_{i}^{p}=f_{e}([x_{i},p_{i}]),x_{i}\in X^{*},p_{i}\in P^{*},$ (8)

$x_{i}^{p}$ is the prompt-based feature.

###  3.2 Auto-regressive Temporal Reasoning

Temporal reasoning is important for sequence data. Therefore, we propose an
Auto-regressive Temporal Reasoning algorithm to better model the time-varying
data. Auto-regressive models are statistical models that make predictions
based on previous observations. They assume that the future values of a
variable can be estimated by considering its past values. For temporal
reasoning, this concept is extended to capture dependencies between different
frames in a video.

After getting the prompt-based feature
$X^{p}=\\{x_{1}^{p},x_{2}^{p},...,x_{m}^{p}\\}$, where $x_{i}^{p}$ represents
the observation at time step $i$, the goal is to predict the future values,

$\hat{x}_{i+1}^{p}=f_{a}(\prod_{j}^{j<(i+1)}f_{a}(x_{j}^{p})+x_{i+1}^{p})$ (9)

where $f_{a}$ denotes the auto-regressive model that maintains an internal
state and updates according to the sequential input. $\prod$ means a series of
functions here. The auto-regressive temporal reasoning model considers the
past observations of the sequence and the corresponding future observations to
learn the underlying temporal dependencies.

###  3.3 Single-agent and Multi-agent Objective

The supervision formats used for single-agent and multi-agent action
recognitions are different. As a result, choose different loss functions. In
particular, we choose the classical cross-entropy loss for single-agent action
recognition,

$L_{n}=-\sum_{c=1}^{C}\log\frac{\exp\left(\hat{x}_{n,c}^{p}\right)}{\sum_{i=1}^{C}\exp\left(\hat{x}_{n,i}^{p}\right)}y_{n,c},$
(10)

where $C$ is the class number, $n$ is the video number, and
$\hat{x}_{n,c}^{p}$ is the PLAR’s output feature. $y$ is the label. For multi-
agent action recognition on Okutama, we use the BCEWithLogitsLoss,

$L_{n,c}=-\left[y_{n,c}\cdot\log\sigma\left(\hat{x}_{n,c}^{p}\right)+\left(1-y_{n,c}\right)\cdot\log\left(1-\sigma\left(\hat{x}_{n,c}^{p}\right)\right)\right]$
(11)

where $\hat{x}_{n,c}^{p}$ is the PLAR’s output feature. $\sigma$ is a sigmoid
function. This loss combines a sigmoid function and the BCELoss, which is more
numerically stable than using a plain sigmoid followed by a BCELoss because by
combining the operations into one layer, it takes advantage of the log-sum-exp
for numerical stability. For both single-agent and multi-agent videos, by
sharing the same objective, our learning approach can optimize prompts that
guide the model’s predictions while explicitly learning input-invariant
(prompt experts pool) and input-specific (data-dependent) prompt knowledge.

##  4 Datasets and Results

To verify the effectiveness of PLAR, empirical evaluations were conducted on
Okutama (Barekatain et al.,, 2017) and Something-something V2(Goyal et al.,,
2017) datasets comprising both aerial videos and ground camera videos, and
scenarios involving single-agent and multi-agent actions.

Method | Frame size | Accuracy
---|---|---
AARN (Yang et al.,, 2019; Algamdi et al.,, 2020) | crops | $33.75\%$
Lite ECO (Zolfaghari et al.,, 2018; Algamdi et al.,, 2020) | crops | $36.25\%$
I3D(RGB)(Carreira and Zisserman,, 2017; Algamdi et al.,, 2020) | crops | $38.12\%$
3DCapsNet-DR(ZHang et al.,, 2020; Algamdi et al.,, 2020) | crops | $39.37\%$
3DCapsNet-EM(ZHang et al.,, 2020; Algamdi et al.,, 2020) | crops | $41.87\%$
DroneCaps(Algamdi et al.,, 2020) | crops | $47.50\%$
DroneAttention without bbox(Yadav et al.,, 2023) | 720x420 | $61.34\%$
PLAR without bbox (Ours) | 224x224 | $71.54\%$
DroneAttention with bbox (Yadav et al.,, 2023) | 720x420 | $72.76\%$
PLAR with bbox (Ours) | 224x224 | $75.93\%$
Table 1: Comparison with the state-of-the-art results on the Okutama dataset.
With bbox information, we achieved 10.20% improvement over the SOTA method.
Without bbox information, we outperformed the SOTA by 3.17%. crops: from
detection.

###  4.1 Datasets

##### Okutama (Barekatain et al.,, 2017)

The dataset contains diverse lighting, weather, and background conditions,
resembling real-world situations. It consists of 43 minute-long sequences with
12 action classes, providing a challenge with dynamic action transitions,
changing scales and aspect ratios, camera movement, and multi-labeled actors.
It features human-to-human interactions (hugging, handshaking), human-to-
object interactions (reading, drinking, pushing, pulling, carrying, calling),
and none-interactions (running, walking, lying, sitting, standing). The
dataset showcases up to 9 actors sequentially performing a wide range of
actions, making it highly challenging and engaging.

##### Something-something v2 (SSV2 (Goyal et al.,, 2017))

The SSV2 dataset is regarded as a substantial and comprehensive benchmark for
action recognition, encompassing a vast collection of 220k action clips. This
dataset places particular emphasis on action classes that revolve around the
dynamics of motion, showcasing various scenarios like "putting something into
something, covering something with something." In this context, the ability to
capture and comprehend fine-grained motion details becomes important, as it
plays a vital role in attaining superior performance results.

###  4.2 Settings

All experiments are conducted on a desktop equipped with 8 Nvidia A5000 GPUs.

Okutama: For the multi-agent experiments, all the frames extracted from the
video datasets were scaled to 224 × 224. The backbone is Swin-T (Liu et al.,,
2021). Following (Yadav et al.,, 2023), the feature maps obtained were
processed in the ROIAlign function (crop size of 5 × 5) to get the desired
ROIs. Then we used the fully connected layer to encode those features to
classifier features. The one-hot encoded targets and classifier features were
fed into Binary Cross Entropy Logits Loss. Other training settings follow (Liu
et al.,, 2021).

Something-something v2: Following (Li et al., 2022b, ), we fine-tune the pre-
trained Kinetics models. Specifically, we train for 100 epochs using 8 GPUs
with a batch size of 64 and a base learning rate of 5e-5 with a cosine
learning rate schedule. We use Adamw and use a weight decay of 1e-4 and a drop
path rate of 0.4. For other training and testing settings, we follow (Li et
al., 2022b, ). And the backbone is MViTv2-S (Li et al., 2022b, ).

###  4.3 Results on Okutama

Okutama is an aerial multi-agent action recognition dataset in which multiple
actors sequentially perform a diverse set of actions, which makes it very
challenging. In the real world, it’s difficult to ensure that only a single
agent is in the scene for action recognition. Therefore, multi-agent action
recognition is a very practical and important research direction. We compare
our PLAR with state-of-the-art (SOTA) works.

As shown in Table 1, if there is no bbox information, we achieved 10.20%
improvement over the SOTA method. If there is bbox information, we outperform
the SOTA by 3.17%. This demonstrates the effectiveness of our method.

Method | pretrain | Top-1 Acc. | Top-5 Acc.
---|---|---|---
TEA (Li et al., 2020b, ) | ImageNet 1k | $65.1\%$ | $89.9\%$
MoViNet-A3 (Kondratyuk et al.,, 2021) | N/A | $64.1\%$ | $88.8\%$
ViT-B-TimeSformer (Bertasius et al.,, 2021) | ImageNet 21k | $62.5\%$ | /
SlowFast R101, 8×8 (Feichtenhofer et al.,, 2019) | Kinetics400 | $63.1\%$ | $87.6\%$
MViTv1-B, 16×4 (Fan et al.,, 2021) | Kinetics400 | $64.7\%$ | $89.2\%$
MViTv2-S, 16×4 (Li et al., 2022b, ) (train by us) | Kinetics400 | $66.5\%$ | $90.6\%$
PLAR (Ours) | Kinetics400 | $67.3\%$ | $91.0\%$
Table 2: Comparison with the state-of-the-art results on the Something
Something V2. Our PLAR improves 2.6% over MViTv1 and 0.8% over strong SOTA
MViTv2.

###  4.4 Results on Something-something V2

Something-something V2 is a challenging ground camera dataset for visual
common sense because it requires models to understand the relationships
between objects and actions. For example, to predict the category of a video,
a model must understand that "something bounces a ball" is different from
"something rolls a ball". In addition, the model must simultaneously pay
attention to temporal modeling. We evaluate our PLAR’s reasoning and temporal
modeling ability on Something-somethingV2.

As shown in Table 2, our PLAR improves 2.6% over MViTv1 and 0.8% over MViTv2,
which illustrates the effectiveness of our proposed prompt learning and Auto-
regressive temporal modeling.

Method | Frame size | Accuracy
---|---|---
Baseline | 224x224 | $71.54\%$
Baseline + Optical Flow | 224x224 | $72.13\%$
Baseline + Large Vision Model (SAM) | 224x224 | $74.68\%$
Baseline + Learnable Prompt | 224x224 | $75.93\%$
Table 3: Ablation study in terms of different prompts on the Okutama dataset.
We evaluated various prompts including optical flow, a large vision model(SAM
(Kirillov et al.,, 2023)), and learnable prompts. From our experiment, the
large vision model and learnable prompt achieved better accuracy. Figure 4:
Large Vision Model: Prompts from the large vision model, no supervision
needed. We visualize the outputs in terms of different prompts, including
bbox, line, and different points. bbox and line have more stable outputs,
which means better prompts result in better outputs.

###  4.5 Ablation Study

First, we conducted ablation studies on various prompts, including optical
flow, large vision models, and learnable prompts to verify their
effectiveness. And then we further visualize the output of a large vision
model in terms of different inputs to verify if they could focus on the right
target.

Different Prompts To evaluate the effectiveness of different prompts, various
prompts, including optical flow, large vision model(SAM (Kirillov et al.,,
2023)), and learnable prompts, are examined in this work. As shown in Table 3,
the large vision model and learnable prompt achieved better accuracy.

Visualization For the large vision model, we visualize the outputs in terms of
different prompts, including bbox, line, and points. As shown in Figure 4,
bbox and line have more stable outputs, which means better prompts result in
better outputs.

##  5 Conclusion, Limitations and Future work

We present a general prompt learning approach that uses various prompts,
including optical flow, a large vision model (SAM), and learnable prompts.
Prompt learning alleviates the optimization burden by providing high-level
texture descriptions or instructions associated with actions. Our proposed
learnable prompt learns to dynamically generate prompts from a pool of prompt
experts under different inputs. Our objective is to optimize prompts that
guide the model’s predictions while explicitly learning input-invariant
(prompt experts) and input-specific (data-dependent) prompt knowledge. We
observe good accuracy improvements on some challenging datasets. Overall, ours
is among the first method that has explored the use of large vision model as
the prompt to instruct the action recognition task. Our results are very
promising and there is room to further improve the performance. Our approach
has some limitations. The formulation does not use a unified prompt form for
both CNN and Transformer. We have only considered a few prompt formats, and
the overall performance depends on this choice. In the future, We would like
to design a unified prompt form for different architectures and we will
integrate text prompts with our approach. We need to evaluate the performance
on more datasets.

## References

  * Algamdi et al., (2020) Algamdi, A. M., Sanchez, V., and Li, C.-T. (2020).  Dronecaps: Recognition of human actions in drone videos using capsule networks with binary volume comparisons.  In 2020 IEEE International Conference on Image Processing (ICIP), pages 3174–3178. IEEE.
  * Arnab et al., (2021) Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., and Schmid, C. (2021).  Vivit: A video vision transformer.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836–6846.
  * Barekatain et al., (2017) Barekatain, M., Martí, M., Shih, H.-F., Murray, S., Nakayama, K., Matsuo, Y., and Prendinger, H. (2017).  Okutama-action: An aerial view video dataset for concurrent human action detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 28–35.
  * Bertasius et al., (2021) Bertasius, G., Wang, H., and Torresani, L. (2021).  Is space-time attention all you need for video understanding?  In ICML, volume 2, page 4.
  * Brown et al., (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020).  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901.
  * Carreira and Zisserman, (2017) Carreira, J. and Zisserman, A. (2017).  Quo vadis, action recognition? a new model and the kinetics dataset.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308.
  * Chéron et al., (2015) Chéron, G., Laptev, I., and Schmid, C. (2015).  P-cnn: Pose-based cnn features for action recognition.  In Proceedings of the IEEE international conference on computer vision, pages 3218–3226.
  * Fan et al., (2021) Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., and Feichtenhofer, C. (2021).  Multiscale vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6824–6835.
  * Fayyaz et al., (2021) Fayyaz, M., Bahrami, E., Diba, A., Noroozi, M., Adeli, E., Van Gool, L., and Gall, J. (2021).  3d cnns with adaptive temporal feature resolutions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4731–4740.
  * Feichtenhofer et al., (2019) Feichtenhofer, C., Fan, H., Malik, J., and He, K. (2019).  Slowfast networks for video recognition.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211.
  * Ge et al., (2022) Ge, C., Huang, R., Xie, M., Lai, Z., Song, S., Li, S., and Huang, G. (2022).  Domain adaptation via prompt learning.  arXiv preprint arXiv:2202.06687.
  * Girdhar et al., (2017) Girdhar, R., Ramanan, D., Gupta, A., Sivic, J., and Russell, B. (2017).  Actionvlad: Learning spatio-temporal aggregation for action classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 971–980.
  * Goyal et al., (2017) Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. (2017).  The" something something" video database for learning and evaluating visual common sense.  In Proceedings of the IEEE international conference on computer vision, pages 5842–5850.
  * Han et al., (2022) Han, X., Zhao, W., Ding, N., Liu, Z., and Sun, M. (2022).  Ptr: Prompt tuning with rules for text classification.  AI Open, 3:182–192.
  * Hussein et al., (2019) Hussein, N., Gavves, E., and Smeulders, A. W. (2019).  Timeception for complex action recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 254–263.
  * Ji et al., (2012) Ji, S., Xu, W., Yang, M., and Yu, K. (2012).  3d convolutional neural networks for human action recognition.  IEEE transactions on pattern analysis and machine intelligence, 35(1):221–231.
  * Jia et al., (2021) Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. (2021).  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning, pages 4904–4916. PMLR.
  * Jiang et al., (2020) Jiang, Z., Xu, F. F., Araki, J., and Neubig, G. (2020).  How can we know what language models know?  Transactions of the Association for Computational Linguistics, 8:423–438.
  * Ju et al., (2022) Ju, C., Han, T., Zheng, K., Zhang, Y., and Xie, W. (2022).  Prompting visual-language models for efficient video understanding.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV, pages 105–124. Springer.
  * Karpathy et al., (2014) Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Fei-Fei, L. (2014).  Large-scale video classification with convolutional neural networks.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725–1732.
  * Kirillov et al., (2023) Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023).  Segment anything.  arXiv preprint arXiv:2304.02643.
  * Kondratyuk et al., (2021) Kondratyuk, D., Yuan, L., Li, Y., Zhang, L., Tan, M., Brown, M., and Gong, B. (2021).  Movinets: Mobile video networks for efficient video recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16020–16030.
  * Kothandaraman et al., (2022) Kothandaraman, D., Guan, T., Wang, X., Hu, S., Lin, M.-S., and Manocha, D. (2022).  Far: Fourier aerial video recognition.  In European Conference on Computer Vision.
  * Lester et al., (2021) Lester, B., Al-Rfou, R., and Constant, N. (2021).  The power of scale for parameter-efficient prompt tuning.  arXiv preprint arXiv:2104.08691.
  * (25) Li, M., Chen, L., Duan, Y., Hu, Z., Feng, J., Zhou, J., and Lu, J. (2022a).  Bridge-prompt: Towards ordinal action understanding in instructional videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19880–19889.
  * (26) Li, X., Shuai, B., and Tighe, J. (2020a).  Directional temporal modeling for action recognition.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16, pages 275–291. Springer.
  * Li and Liang, (2021) Li, X. L. and Liang, P. (2021).  Prefix-tuning: Optimizing continuous prompts for generation.  Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), abs/2101.00190.
  * (28) Li, Y., Ji, B., Shi, X., Zhang, J., Kang, B., and Wang, L. (2020b).  Tea: Temporal excitation and aggregation for action recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 909–918.
  * (29) Li, Y., Wu, C.-Y., Fan, H., Mangalam, K., Xiong, B., Malik, J., and Feichtenhofer, C. (2022b).  Mvitv2: Improved multiscale vision transformers for classification and detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4804–4814.
  * Liu et al., (2023) Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. (2023).  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  ACM Computing Surveys, 55(9):1–35.
  * Liu et al., (2021) Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021).  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022.
  * Lu et al., (2022) Lu, Y., Liu, J., Zhang, Y., Liu, Y., and Tian, X. (2022).  Prompt distribution learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5206–5215.
  * Mazzia et al., (2022) Mazzia, V., Angarano, S., Salvetti, F., Angelini, F., and Chiaberge, M. (2022).  Action transformer: A self-attention model for short-time pose-based human action recognition.  Pattern Recognition, 124:108487.
  * Petroni et al., (2019) Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., and Riedel, S. (2019).  Language models as knowledge bases?  arXiv preprint arXiv:1909.01066.
  * Piergiovanni and Ryoo, (2019) Piergiovanni, A. and Ryoo, M. S. (2019).  Representation flow for action recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9945–9953.
  * Piergiovanni and Ryoo, (2021) Piergiovanni, A. and Ryoo, M. S. (2021).  Recognizing actions in videos from unseen viewpoints.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4124–4132.
  * Poerner et al., (2019) Poerner, N., Waltinger, U., and Schütze, H. (2019).  E-bert: Efficient-yet-effective entity embeddings for bert.  In Findings.
  * Radford et al., (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. (2021).  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning.
  * Rao et al., (2022) Rao, Y., Zhao, W., Chen, G., Tang, Y., Zhu, Z., Huang, G., Zhou, J., and Lu, J. (2022).  Denseclip: Language-guided dense prediction with context-aware prompting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18082–18091.
  * Sánchez et al., (2013) Sánchez, J., Perronnin, F., Mensink, T., and Verbeek, J. (2013).  Image classification with the fisher vector: Theory and practice.  International journal of computer vision, 105:222–245.
  * Sato et al., (2023) Sato, F., Hachiuma, R., and Sekii, T. (2023).  Prompt-guided zero-shot anomaly action recognition using pretrained deep skeleton features.  arXiv preprint arXiv:2303.15167.
  * Shi et al., (2022) Shi, Y., Wu, X., and Lin, H. (2022).  Knowledge prompting for few-shot action recognition.  arXiv preprint arXiv:2211.12030.
  * Shin et al., (2020) Shin, T., Razeghi, Y., IV, R. L. L., Wallace, E., and Singh, S. (2020).  Eliciting knowledge from language models using automatically generated prompts.  ArXiv, abs/2010.15980.
  * Simonyan and Zisserman, (2014) Simonyan, K. and Zisserman, A. (2014).  Two-stream convolutional networks for action recognition in videos.  Advances in neural information processing systems, 27.
  * Sun et al., (2017) Sun, L., Jia, K., Chen, K., Yeung, D.-Y., Shi, B. E., and Savarese, S. (2017).  Lattice long short-term memory for human action recognition.  In Proceedings of the IEEE international conference on computer vision, pages 2147–2156.
  * Tian et al., (2020) Tian, Y., Wang, Y., Krishnan, D., Tenenbaum, J. B., and Isola, P. (2020).  Rethinking few-shot image classification: a good embedding is all you need?  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pages 266–282. Springer.
  * Tong et al., (2022) Tong, Z., Song, Y., Wang, J., and Wang, L. (2022).  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  arXiv preprint arXiv:2203.12602.
  * Tran et al., (2015) Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015).  Learning spatiotemporal features with 3d convolutional networks.  In Proceedings of the IEEE international conference on computer vision, pages 4489–4497.
  * Vaswani et al., (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017).  Attention is all you need.  Advances in neural information processing systems, 30.
  * Wang et al., (2015) Wang, L., Qiao, Y., and Tang, X. (2015).  Action recognition with trajectory-pooled deep-convolutional descriptors.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4305–4314.
  * (51) Wang, M., Xing, J., and Liu, Y. (2021a).  Actionclip: A new paradigm for video action recognition.  arXiv preprint arXiv:2109.08472.
  * Wang et al., (2023) Wang, X., Xian, R., Guan, T., de Melo, C. M., Nogar, S. M., Bera, A., and Manocha, D. (2023).  Aztr: Aerial video action recognition with auto zoom and temporal reasoning.  arXiv preprint arXiv:2303.01589.
  * (53) Wang, X., Zhang, S., Qing, Z., Shao, Y., Zuo, Z., Gao, C., and Sang, N. (2021b).  Oadtr: Online action detection with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7565–7575.
  * Wang et al., (2022) Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. (2022).  Learning to prompt for continual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139–149.
  * (55) Xian, R., Wang, X., Kothandaraman, D., and Manocha, D. (2023a).  Pmi sampler: Patch similarity guided frame selection for aerial action recognition.  arXiv preprint arXiv:2304.06866.
  * (56) Xian, R., Wang, X., and Manocha, D. (2023b).  Mitfas: Mutual information based temporal feature alignment and sampling for aerial video action recognition.  arXiv preprint arXiv:2303.02575.
  * Yadav et al., (2023) Yadav, S. K., Luthra, A., Pahwa, E., Tiwari, K., Rathore, H., Pandey, H. M., and Corcoran, P. (2023).  Droneattention: Sparse weighted temporal attention for drone-camera based activity recognition.  Neural Networks, 159:57–69.
  * Yang et al., (2020) Yang, C., Xu, Y., Shi, J., Dai, B., and Zhou, B. (2020).  Temporal pyramid network for action recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 591–600.
  * Yang et al., (2019) Yang, F., Sakti, S., Wu, Y., and Nakamura, S. (2019).  A framework for knowing who is doing what in aerial surveillance videos.  IEEE Access, 7:93315–93325.
  * Yang et al., (2022) Yang, J., Dong, X., Liu, L., Zhang, C., Shen, J., and Yu, D. (2022).  Recurring the transformer for video action recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14063–14073.
  * Zhang et al., (2020) Zhang, H., Zhang, L., Qi, X., Li, H., Torr, P. H., and Koniusz, P. (2020).  Few-shot action recognition with permutation-invariant attention.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 525–542. Springer.
  * ZHang et al., (2020) ZHang, P., Wei, P., and Han, S. (2020).  Capsnets algorithm.  In Journal of Physics: Conference Series, volume 1544, page 012030\. IOP Publishing.
  * Zhong et al., (2021) Zhong, Z., Friedman, D., and Chen, D. (2021).  Factual probing is [mask]: Learning vs. learning to recall.  arXiv preprint arXiv:2104.05240.
  * Zhou et al., (2021) Zhou, K., Yang, J., Loy, C. C., and Liu, Z. (2021).  Learning to prompt for vision-language models.  International Journal of Computer Vision, 130:2337 – 2348.
  * Zhou et al., (2022) Zhou, K., Yang, J., Loy, C. C., and Liu, Z. (2022).  Conditional prompt learning for vision-language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816–16825.
  * Zhou et al., (2020) Zhou, Y., Sun, X., Luo, C., Zha, Z.-J., and Zeng, W. (2020).  Spatiotemporal fusion in 3d cnns: A probabilistic view.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9829–9838.
  * Zolfaghari et al., (2018) Zolfaghari, M., Singh, K., and Brox, T. (2018).  Eco: Efficient convolutional network for online video understanding.  In Proceedings of the European conference on computer vision (ECCV), pages 695–712.
  * Zong et al., (2021) Zong, M., Wang, R., Chen, X., Chen, Z., and Gong, Y. (2021).  Motion saliency based multi-stream multiplier resnets for action recognition.  Image and Vision Computing, 107:104108.