# Axion Detection Experiments Meet the Majoron

Qiuyue Liang<EMAIL_ADDRESS>Kavli IPMU (WPI), UTIAS, University of
Tokyo, Kashiwa, 277-8583, Japan  Xavier Ponce Díaz
<EMAIL_ADDRESS>Istituto Nazionale di Fisica Nucleare (INFN),
Sezione di Padova, and Dipartimento di Fisica e Astronomia “G. Galilei”,
Università di Padova, Via F. Marzolo 8, 35131 Padova, Italy  Tsutomu T.
Yanagida<EMAIL_ADDRESS>Kavli IPMU (WPI), UTIAS, University of
Tokyo, Kashiwa, 277-8583, Japan  Tsung-Dao Lee Institute & School of Physics
and Astronomy, Shanghai Jiao Tong University, China

###### Abstract

The majoron is a well-motivated light (pseudo-Nambu-Goldstone) boson
associated with the spontaneous breaking of a global lepton-number symmetry.
In this letter, we relate the spontaneous breaking scale and its soft-breaking
mass by requiring that the majoron is the main component of the dark matter.
An electromagnetic-anomalous coupling can be induced by minimally modifying
the original majoron model, surprisingly, predicting a parameter region that
largely overlaps with the QCD-axion dark matter band. Thus, we expect that
axion search experiments meet the majoron.

##  I Introduction

The heavy Majorana (right-handed) neutrinos are key ingredients for generating
small neutrino masses via the seesaw mechanism [1, 2, 3, 4, 5] 111After the
discovery of the seesaw mechanism many intense discussions on the origin of
neutrino masses followed [6, 7, 8, 9, 10]. and for producing the baryon-number
asymmetry in the universe through the leptogenesis [11, 12]. However, the
origin of the heavy Majorana masses remains unknown. The most popular
mechanism to generate heavy Majorana mass terms of the right-handed neutrinos
is given by spontaneous breaking of the $B-L$ gauge symmetry[7, 8], where the
presence of the three right-handed neutrinos is motivated to cancel the
possible gauge anomalies.

There is, however, an alternative theory where the spontaneous breaking of a
lepton number $U(1)_{L}$ [13, 14] induces the heavy Majorana masses222See [15]
for a low scale lepton-number breaking.. This symmetry must be a global
symmetry, since $U(1)_{L}$ is anomalous, therefore, cannot be gauged. Although
this model is less attractive, there is a big advantage: we have a light
pseudo Nambu-Goldstone boson called “majoron $J$” which can be an interesting
candidate for Dark Matter (DM) [16, 17].

It is believed that global symmetries cannot be exact, and the majoron could
gain a mass by introducing a soft symmetry breaking term [18, 19, 20]. In this
letter, to address the dark matter problem, we will consider the majoron mass
fixed by the misalignment mechanism [21, 22, 23] to get the right DM relic
abundance, but not specify the UV origin of this mass.

Existing literature has explored the DM majoron mass primarily above MeV
region [24, 25, 26], and at keV region[27]. This focus is driven by the
possibility of searching for such a particle: majorons heavier than the MeV
could be detected in neutrino experiments [24, 25], whereas below this
threshold the detection becomes challenging, see [28] for discussion in keV
band. However, to serve the purpose of seesaw mechanism and leptogenesis, the
decay constant of the majoron model is given by, $F_{J}\sim
10^{13}\,\textrm{GeV}$ [12]. On top of this, to explain the dark matter
abundance, the desired majoron mass is, $m_{J}\sim\mu$eV. This significantly
deviates from the typical majoron detectable region, and therefore, should be
searched by different experiments. In this letter, we introduce a second Higgs
boson to enable opposite $U(1)_{L}$ charges to the leptons, making the model
anomalous under QED, and therefore, can be detected by various axion
experiments. Moreover, the majoron preferred region overlaps with the QCD-
axion dark matter band, and will meet in the way of future axion detection
experiments.

In the rest of this paper, we will introduce the relevant part of the model in
Section II, and discuss the majoron dark matter in Section III. The
conclusions and discussions are given in Section IV, with an overlook on the
possible connections of this model with other phenomena. Finally, this letter
is accompanied by an Appendix A with some details on the computations.

##  II Majoron model

In this section we briefly describe our majoron model based on a modified
lepton-number $U(1)_{L}$ symmetry.

The key aspect is that the global $U(1)_{L}$ symmetry-breaking induces the
large Majorana masses for the right-handed neutrinos $N_{R}$. Thus, we
consider the right-handed neutrinos $N_{R}$ carry the $U(1)_{L}$ charge,
fixing it to be $\mathcal{X}_{N}=+1/2$. Accordingly, we introduce a complex
scalar $\phi$, whose vacuum-expectation value (vev)
$\langle\phi\rangle=v_{\phi}/\sqrt{2}$ breaks the global $U(1)_{L}$ symmetry
inducing the Majorana masses, and the phase of the scalar field becomes a
massless pseudo Nambu-Goldstone boson, which we refer to as majoron in the
remaining context. We assign the charge of $\phi$ to be $\mathcal{X}_{\phi}=1$
so that we have a coupling

$\mathcal{L}\supset\bar{\ell}_{L}Y_{e}e_{R}H+\bar{\ell}_{L}Y_{D}N_{R}\widetilde{H}+\frac{1}{2}\bar{N}^{c}_{R}Y_{N}N_{R}\phi^{*}\
,$ (1)

where $H$ denotes the Higgs field, and $Y_{i},i\in\\{e,D,N\\}$ are the Yukawa
coupling constants. Here, we see the Majorana mass for the right-handed
neutrinos is $M_{N}=Y_{N}v_{\phi}/\sqrt{2}$. For typical right-handed neutrino
mass that can gives rise to correct neutrino mass in standard model,
$M_{N}\sim 10^{14}$GeV, the favoured vev is set to be the same magnitude
assuming a $\mathcal{O}(1)$ coupling constant. The above coupling also fixed
the lepton charge of right-handed electron $e_{R}$ to be $\mathcal{X}_{e}=1/2$
333Notice we assign the charge of the right-handed electrons to be the same as
the right-handed neutrinos. This is a reason we call the global symmetry as
the lepton-number symmetry $U(1)_{L}$., and that of the left-handed lepton
doublets $\ell_{L}$ to be $\mathcal{X}_{\ell}=1/2$. This is the common lepton
number assignment for majoron models [13]. However, one can check that this
model does not has QED anomaly, and therefore cannot be detected through axion
experiments.

In this letter, we introduce the second Higgs field, $H_{2}$, to assign
different lepton charges to $\ell_{L}$ and $e_{R}$ to obtain the QED anomaly.
The relevant part of the Lagrangian in the lepton sector is given by

$\displaystyle\mathcal{L}=$
$\displaystyle\bar{\ell}_{L}Y_{e}e_{R}H_{2}+\bar{\ell}_{L}Y_{D}N_{R}\widetilde{H}_{1}+\frac{1}{2}\bar{N}_{R}^{c}Y_{N}N_{R}\phi^{*}$
$\displaystyle+\mu_{\phi}H_{2}^{\dagger}H_{1}\phi+V(H_{1},\,H_{2},\,\phi)+\text{h.c.}\,,$
(2)

where the two Higgs fields carry different lepton charges,
$\mathcal{X}_{2}-\mathcal{X}_{1}=\mathcal{X}_{\phi}=1$. In this model, the
lepton charge of the right-handed electrons is fixed to be
$\mathcal{X}_{e}=-1/2$, while that of the lepton doublet is chosen to be
$\mathcal{X}_{\ell}=1/2$. One can then compute the anomaly coefficients as

$\displaystyle\mathcal{L}_{\textrm{anom.}}$ $\displaystyle=$ $\displaystyle
n_{f}(\mathcal{X}_{\ell}-\mathcal{X}_{e})\frac{\alpha_{\textrm{em}}}{4\pi}\frac{J}{F_{J}}F_{\mu\nu}\widetilde{F}^{\mu\nu}$
(3) $\displaystyle=$ $\displaystyle
3\frac{\alpha_{\textrm{em}}}{4\pi}\frac{J}{F_{J}}F_{\mu\nu}\widetilde{F}^{\mu\nu}=\frac{g_{J\gamma\gamma}}{4}JF_{\mu\nu}\widetilde{F}^{\mu\nu}\
,$

where $n_{f}$ denotes the lepton family number, and $F_{J}$ is the decay
constant of majoron model. To obtain the normalised kinetic term for majoron,
$F_{J}$ is set to be the same as the scalar vev, $v_{\phi}$. This anomalous
coupling can be detected through axion experiments that we will further
discuss in the next section. It is worth pointing out that this coupling
constant, $g_{J\gamma\gamma}$, coincides with the prediction of cosmological
birefringence, see [29].

Finally, before treating the majoron DM in the next section, we want to
briefly comment on the model Eq.(II). Although for simplicity we chose the
charges such that the quark sector remains uncharged, they are generally
charged and can be included in the Lagrangian as discussed in the Appendix A.
We also note that in the literature several variations of this model exist,
for instance, where the majoron becomes an axion, solving the strong-CP
problem [30, 31, 32], as an ALP [33] without including the RH neutrinos, or in
a 2 Higgs Doublet model with no scalar field $\phi$ [34]. However, to the best
of our knowledge, this majoron variation had not been presented yet.

##  III Majoron dark matter

One of the most pressing problems of particle cosmology, if not the most, is
the explanation of the Dark Matter component of the universe. In this respect,
light particles have a very appealing mechanism to saturate the relic
abundance of DM, firstly proposed for the axion, known as the misalignment
mechanism [22, 21, 23]. Since we do not specify the UV origin of the majoron
mass, we apply the misalignment mechanism directly and estimate the fractional
energy density of DM as [35],

$\displaystyle\Omega_{J}h^{2}$ $\displaystyle\simeq
0.12\left(\frac{m_{J}}{\mu\textrm{eV}}\right)^{1/2}\left(\frac{F_{J}\theta^{i}_{J}}{1.9\times
10^{13}\,\textrm{GeV}}\right)^{2}\ ,$ (4)

Figure 1: In purple we see the prediction from misalignment for our majoron.
The preferred (black) region is determined by varying the initial misalignment
angle $\theta_{J}^{i}\in\left[0.5,\,2\right]$, and of
$F_{J}\in\left[10^{13},\,10^{14}\right]\,\textrm{GeV}$.

where $\theta_{J}^{i}\sim\mathcal{O}(1)$ is the initial angle at the start of
the oscillations 444We assume that the lepton-number breaking occurs during or
before the inflation. The scale of inflation $H_{inf}$ needs to be
sufficiently small to avoid large isocurvature fluctuations [36, 37].. This
sets the scale-mass relation of the majoron DM. Considering the natural scale
for the seesaw and leptogenesis $F_{J}\sim 10^{13}-10^{14}\,$GeV [12], the
majoron mass are natural light. Hence, traditional methods to detect majoron
would fail. However, the model in Eq. (II) presents an anomalous coupling to
QED, whose coupling constant can be related to the majoron mass through,

$\displaystyle
m_{J}\simeq\left(\frac{\pi}{3\alpha_{\text{em}}\theta^{i}_{J}}\frac{g_{J\gamma\gamma}}{1.9\times
10^{-13}\text{GeV}}\right)^{4}\mu\textrm{eV}\ ,$ (5)

which is optimal for detection as we will discuss now.

In Fig. 1 we plot the current constraints on the anomalous coupling, including
both astrophysical, cosmological, and experimental bounds [38]. We further
plot the the prediction of $g_{J\gamma\gamma}$ in our model with mass
$m_{J}\in\left[10^{-12},\,10^{6}\right]$ eV in purple, with a variation of
initial angle $\theta_{J}^{i}\in\left[0.5,\,2\right]$. The favoured region by
seesaw mechanism and leptogenesis are coloured in black. Surprisingly, axion
experiments meet the majoron band! In particular, ADMX experiment [39, 40, 41]
has already reached the preferred band for this model.

In the future, there exists several proposals for axion experiments that will
explore further the preferred region for this majoron. In Fig. 1 we see
several projections that can test both particles, the QCD axion and the
majoron. Firstly, we have the future searches of ADMX 555See, talk at TAUP23.,
sweeping the region of $\mathcal{O}(1-10)\mu$eV. Then, for lower masses, the
proposals of implementing a haloscope inside BabyIAXO’s magnet, RADES [42],
and the proposal of using FINUDA’s magnet, FLASH [43], could investigate the
majoron region of $\mathcal{O}(0.1)\mu$eV. Below these masses, the method of
testing the $JF\tilde{F}$-term using resonant cavities becomes unfeasible, as
the length of the cavity needs to be comparable to the Compton wavelength of
the majoron. However, one can still test this low regions using
Superconducting Radio Frequency (SRF) cavities [44], or the DM-radio
experiment [45] which can detect the magnetic field generated by the majoron.
All in all we see, this particle is testable for the preferred region of the
model with the current proposals of axion detection.

##  IV Conclusions and discussion

The majoron is a well-motivated light boson which is generated by spontaneous
breaking of a global lepton-number $U(1)_{L}$ symmetry. The symmetry breaking
induces large Majorana masses for the right-handed neutrinos, and at the same
time a massless Nambu-Goldstone boson which we call majoron $J$. By
introducing a second Higgs field in the model, we can assign different lepton
charges to obtain an anomalous coupling $g_{J\gamma\gamma}$ under QED.

On the other hand, the majoron can also serve as dark matter. The soft-
breaking mass $m_{J}$ is introduced so that the majoron coherent oscillation
explains the observed DM density which depends on the lepton-number breaking
scale $v_{\phi}$. Combining our prediction of the $g_{J\gamma\gamma}$, we have
plotted the prediction of the majoron DM hypothesis in Fig. 1. Surprisingly,
the region of the majoron DM overlaps largely the prediction of the axion DM,
seeing how a large set of experimental proposals searching for the QCD axion
will also meet the majoron in the future!

It might be interesting to assume the global symmetry is exact beside gauge
anomaly terms. In fact, our $U(1)_{L}$ global current has $SU(2)\times SU(2)$
anomaly. Then, the majoron mass can be induced by the electroweak instantons.
However, we immediately realize the obtained mass is too small to be the dark
matter due to the small gauge coupling constant at small scales. Nonetheless,
there is an intriguing possibility to identify the majoron with the
quintessence axion [46, 47] or the electroweak axion [29] responsible for the
cosmic birefrigence [48, 49].

We would, finally, remark that the two heavy Majorana right-handed neutrinos
are sufficient [50] for generating the baryon-number asymmetry in the
universe. If it is the case, the lightest neutrino is exactly massless in our
framework. The presence of the two right-handed neutrinos is consistent with
our global lepton-number $U(1)_{L}$ symmetry.

## Ackowledgments

This work is supported by JSPS Grant-in-Aid for Scientific Research Grants No.
24H02244, the National Natural Science Foundation of China (12175134) and
World Premier International Research Center Initiative (WPI Initiative), MEXT,
Japan. This project has received funding from the European Union’s Horizon
2020 research and innovation programme under the Marie Skłodowska-Curie grant
agreement N. 860881-HIDDeN. XPD thanks Kavli IPMU for its hospitality during
the development of this work. We thank Ciaran O’Hare for useful comments on
experimental projections, and Luca Di Luzio for useful comments on the draft.

##  Appendix A Details

There are two possible lepton number charge assignments, given by the exchange
of the two Higgs doublets in the lepton sector. The first one is anomalous
under QED

$\displaystyle\mathcal{L}$
$\displaystyle=\bar{q}_{L}Y_{u}u_{R}\tilde{H}_{1}+\bar{q}_{L}Y_{d}d_{R}H_{1}+\bar{\ell}_{L}Y_{e}e_{R}H_{2}$
$\displaystyle+\bar{\ell}_{L}Y_{e}N_{R}\tilde{H}_{1}+\bar{N}_{R}^{c}N_{R}\phi^{*}+H_{2}^{\dagger}H_{1}\phi$
$\displaystyle+V(H_{1},\,H_{2},\,\phi)+\textrm{h.c.}\,,$ (6)

the second is not anomalous under QED,

$\displaystyle\mathcal{L}$
$\displaystyle=\bar{q}_{L}Y_{u}u_{R}\tilde{H}_{1}+\bar{q}_{L}Y_{d}d_{R}H_{1}+\bar{\ell}_{L}Y_{e}e_{R}H_{1}$
$\displaystyle+\bar{\ell}_{L}Y_{e}N_{R}\tilde{H}_{2}+\bar{N}_{R}^{c}N_{R}\phi^{*}+H_{2}^{\dagger}H_{1}\phi$
$\displaystyle+V(H_{1},\,H_{2},\,\phi)+\textrm{h.c.}\,.$ (7)

Here $H_{1}$, $H_{2}$ are $SU(2)_{L}$ Higgs doublets and $\phi$ is the complex
scalar that breaks spontaneously the LN symmetry. There is a global symmetry
defined by the interactions of each Lagrangian, with the charges $\mathcal{X}$
given in Tab. 1.

The majoron can be defined in this case by the current associated, following
Goldstone’s theorem $\bra{0}J_{\mu}\ket{J(k)}=-iF_{J}k_{\mu}$,

$J=\frac{1}{F_{J}}\sum_{i}\mathcal{X}_{i}v_{i}a_{i}\,,\textrm{ with
}F_{J}=\sum_{i}\mathcal{X}_{i}^{2}v_{i}^{2}\,,$ (8)

where $a_{i}$ are the phases of the scalars $H_{i}\supset\exp(ia_{i}/v_{i})$,
$\phi\supset\exp(ia_{\phi}/v_{\phi})$, and $v_{i}$ the different vevs. In this
letter, we take $v_{\phi}\gg v_{i}$, so that the majoron can be, in practice,
associated to $J\simeq a_{\phi}$ with $F_{J}\simeq v_{\phi}$.

Fields | $SU(3)_{c}$ | $SU(2)_{L}$ | $U(1)_{Y}$ | LN 1 | LN 2
---|---|---|---|---|---
$q_{L}$ | 3 | 2 | 1/6 | $\mathcal{X}_{q}$ | $\mathcal{X}_{q}$
$u_{R}$ | 3 | 1 | 2/3 | $\mathcal{X}_{q}+\mathcal{X}_{1}$ | $\mathcal{X}_{q}+\mathcal{X}_{1}$
$d_{R}$ | 3 | 1 | -1/3 | $\mathcal{X}_{q}-\mathcal{X}_{1}$ | $\mathcal{X}_{q}-\mathcal{X}_{1}$
$\ell_{L}$ | 1 | 2 | -1/2 | $1/2-\mathcal{X}_{1}$ | $-3/2-\mathcal{X}_{1}$
$e_{R}$ | 1 | 1 | -1 | $-1/2-2\mathcal{X}_{1}$ | $-3/2-2\mathcal{X}_{1}$
$N_{R}$ | 1 | 1 | 0 | 1/2 | 1/2
$H_{1}$ | 1 | 2 | 1/2 | $\mathcal{X}_{1}$ | $\mathcal{X}_{1}$
$H_{2}$ | 1 | 2 | 1/2 | $1+\mathcal{X}_{1}$ | $1+\mathcal{X}_{1}$
$\phi$ | 1 | 1 | 0 | 1 | 1
Table 1: Particle content and charges of the particles with respect to the
gauge group of the SM and Lepton Number of the two models, LN1 is the model
presented in this work, while LN2 is a possible different non-anomalous set of
chages.

From the table we can compute the anomalous coefficient to these charge
assignments. The anomalous terms are defined by the Lagrangian

$\mathcal{L}_{\textrm{Anom.}}=\frac{\alpha_{\textrm{em}}}{4\pi}E\frac{J}{F_{J}}F_{\mu\nu}\widetilde{F}^{\mu\nu}+\frac{\alpha_{s}}{4\pi}N\frac{J}{F_{J}}G^{a}_{\mu\nu}\widetilde{G}^{a\,\mu\nu}\,,$
(9)

where for the first model they can be computed as

$\displaystyle N$
$\displaystyle=\frac{n_{f}}{2}\left(2\mathcal{X}_{q}-\mathcal{X}_{u}-\mathcal{X}_{d}\right)=\frac{n_{f}}{2}\left(\mathcal{X}_{1}-\mathcal{X}_{1}\right)=0\,,$
$\displaystyle E$
$\displaystyle=n_{f}\sum_{i}\mathcal{X}_{i}Q^{2}_{i}=n_{f}\left(\frac{4}{3}\mathcal{X}_{q}+\frac{1}{3}\mathcal{X}_{q}-\frac{4}{3}\mathcal{X}_{u}\right.$
$\displaystyle\left.-\frac{1}{3}\mathcal{X}_{d}+(\mathcal{X}_{\ell}-\mathcal{X}_{e})\right)=n_{f}\left(-\mathcal{X}_{1}+\mathcal{X}_{2}\right)$
$\displaystyle=n_{f}\times\mathcal{X}_{\phi}=3\,,$ (10)

where $Q_{i}$ are the QED charges of the SM fermions and
$\mathcal{X}_{\phi}=1$.

Noting that there is no QCD anomaly and that the electromagnetic anomaly is
independent of the choice of $\mathcal{X}_{1,2}$, in the main text we have
taken $\mathcal{X}_{1}=0$ and $\mathcal{X}_{q}=0$, for simplicity. However, it
is important to note that this choice generates a mixing with the $Z$-boson
coming from the kinetic term

$\displaystyle\mathcal{L}_{kin.}$
$\displaystyle\supset(D_{\mu}H_{1})^{\dagger}D_{\mu}H_{1}+(D_{\mu}H_{2})^{\dagger}D_{\mu}H_{2}$
$\displaystyle\supset\frac{\partial_{\mu}J}{F_{J}}Z^{\mu}\frac{g^{\prime}}{2}\left(\mathcal{X}_{1}v_{1}^{2}+\mathcal{X}_{2}v_{2}^{2}\right)\,.$
(11)

It is convenient to avoid such a mixing, defining the charges as
$\mathcal{X}_{1}=-\sin^{2}\beta$ and $\mathcal{X}_{2}=\cos^{2}\beta$, with
$\tan\beta=v_{2}/v_{1}$. This choice, inevitably leads to charging the quark
sector, having the interactions

$\displaystyle\mathcal{L}_{J}^{\textrm{SM}}=$
$\displaystyle\frac{\partial^{\mu}J}{2F_{J}}\left(-\sin^{2}\beta\bar{u}\gamma_{\mu}\gamma_{5}u+\sin^{2}\beta\bar{d}\gamma_{\mu}\gamma_{5}d\right.$
$\displaystyle\left.-\cos^{2}\beta\bar{e}\gamma_{\mu}\gamma_{5}e\right)+3\frac{\alpha_{em}}{4\pi}\frac{J}{F_{J}}F\tilde{F}\,.$
(12)

The reason why no emphasis has been given in the main text to this couplings
is that for such small masses of the majoron and large scale, these couplings
are rather difficult to test and no bounds are applied.

The neutrino sector is then given by

$\mathcal{L}_{J}^{\nu}\simeq\frac{\partial^{\mu}J}{2F_{J}}\left(-\frac{1}{2}\bar{\nu}\gamma_{\mu}\gamma_{5}\nu+\frac{1}{2}\bar{N}\gamma_{\mu}\gamma_{5}N\right)$
(13)

with $\nu=\nu_{L}+\nu_{L}^{c}$ and similar for $N$. This formula is
approximate, as the diagonalization of the neutrino mass matrix induces
$\sim\mathcal{O}(v/v_{\phi})$ corrections, that we can safely neglect.

In Eq. (A), we have only mentioned the coupling $H_{2}^{\dagger}H_{1}\phi$ of
the potential, which is enough for setting the charges. Other options exist,
such as $H_{2}^{\dagger}H_{1}\phi^{2}$, which would give an equally valid
model with $E=6$. Finally, it is worth noting that the extra scalar degrees of
freedom can be decoupled from the theory. This discussion is analogous to
axion models, see for instance Appendix A of Ref. [51] for details.

## References

  * Minkowski [1977] P. Minkowski, $\mu\to e\gamma$ at a Rate of One Out of $10^{9}$ Muon Decays?, Phys. Lett. B 67, 421 (1977).
  * Yanagida [1979a] T. Yanagida, Horizontal gauge symmetry and masses of neutrinos, Conf. Proc. C 7902131, 95 (1979a).
  * Yanagida [1979b] T. Yanagida, Horizontal Symmetry and Mass of the Top Quark, Phys. Rev. D 20, 2986 (1979b).
  * Glashow [1980] S. L. Glashow, The Future of Elementary Particle Physics, NATO Sci. Ser. B 61, 687 (1980).
  * Gell-Mann  _et al._ [1979] M. Gell-Mann, P. Ramond, and R. Slansky, Complex Spinors and Unified Theories, Conf. Proc. C 790927, 315 (1979), arXiv:1306.4669 [hep-th] .
  * Weinberg [1979] S. Weinberg, Baryon and Lepton Nonconserving Processes, Phys. Rev. Lett. 43, 1566 (1979).
  * Wilczeck [1979] F. Wilczeck, Proceedings: Lepton-Photon Conference (Fermilab, Aug 1979) Conf. Proc. C790885 (1979).
  * Barbieri  _et al._ [1980] R. Barbieri, D. V. Nanopoulos, G. Morchio, and F. Strocchi, Neutrino Masses in Grand Unified Theories, Phys. Lett. B 90, 91 (1980).
  * Witten [1980] E. Witten, Neutrino Masses in the Minimal O(10) Theory, Phys. Lett. B 91, 81 (1980).
  * Mohapatra and Senjanovic [1980] R. N. Mohapatra and G. Senjanovic, Neutrino Mass and Spontaneous Parity Nonconservation, Phys. Rev. Lett. 44, 912 (1980).
  * Fukugita and Yanagida [1986] M. Fukugita and T. Yanagida, Baryogenesis Without Grand Unification, Phys. Lett. B 174, 45 (1986).
  * Buchmuller  _et al._ [2005] W. Buchmuller, R. D. Peccei, and T. Yanagida, Leptogenesis as the origin of matter, Ann. Rev. Nucl. Part. Sci. 55, 311 (2005), arXiv:hep-ph/0502169 .
  * Chikashige  _et al._ [1980] Y. Chikashige, R. N. Mohapatra, and R. D. Peccei, Spontaneously Broken Lepton Number and Cosmological Constraints on the Neutrino Mass Spectrum, Phys. Rev. Lett. 45, 1926 (1980).
  * Chikashige  _et al._ [1981] Y. Chikashige, R. N. Mohapatra, and R. D. Peccei, Are There Real Goldstone Bosons Associated with Broken Lepton Number?, Phys. Lett. B 98, 265 (1981).
  * Gelmini and Roncadelli [1981] G. B. Gelmini and M. Roncadelli, Left-Handed Neutrino Mass Scale and Spontaneously Broken Lepton Number, Phys. Lett. B 99, 411 (1981).
  * Gu  _et al._ [2010] P.-H. Gu, E. Ma, and U. Sarkar, Pseudo-Majoron as Dark Matter, Phys. Lett. B 690, 145 (2010), arXiv:1004.1919 [hep-ph] .
  * Frigerio  _et al._ [2011] M. Frigerio, T. Hambye, and E. Masso, Sub-GeV dark matter as pseudo-Goldstone from the seesaw scale, Phys. Rev. X 1, 021026 (2011), arXiv:1107.4564 [hep-ph] .
  * Giddings and Strominger [1988] S. B. Giddings and A. Strominger, Axion Induced Topology Change in Quantum Gravity and String Theory, Nucl. Phys. B 306, 890 (1988).
  * Alonso and Urbano [2019] R. Alonso and A. Urbano, Wormholes and masses for Goldstone bosons, JHEP 02, 136, arXiv:1706.07415 [hep-ph] .
  * Coleman [1985] S. Coleman,  _Aspects of Symmetry: Selected Erice Lectures_ (Cambridge University Press, Cambridge, U.K., 1985).
  * Abbott and Sikivie [1983] L. F. Abbott and P. Sikivie, A Cosmological Bound on the Invisible Axion, Phys. Lett. B 120, 133 (1983).
  * Dine  _et al._ [1981] M. Dine, W. Fischler, and M. Srednicki, A Simple Solution to the Strong CP Problem with a Harmless Axion, Phys. Lett. B 104, 199 (1981).
  * Preskill  _et al._ [1983] J. Preskill, M. B. Wise, and F. Wilczek, Cosmology of the Invisible Axion, Phys. Lett. B 120, 127 (1983).
  * Garcia-Cely and Heeck [2017] C. Garcia-Cely and J. Heeck, Neutrino Lines from Majoron Dark Matter, JHEP 05, 102, arXiv:1701.07209 [hep-ph] .
  * Akita and Niibo [2023] K. Akita and M. Niibo, Updated constraints and future prospects on majoron dark matter, JHEP 07, 132, arXiv:2304.04430 [hep-ph] .
  * de Giorgi  _et al._ [2024] A. de Giorgi, L. Merlo, X. Ponce Díaz, and S. Rigolin, The minimal massive Majoron Seesaw Model, JHEP 03, 094, arXiv:2312.13417 [hep-ph] .
  * Heeck and Teresi [2017] J. Heeck and D. Teresi, Cold keV dark matter from decays and scatterings, Phys. Rev. D 96, 035018 (2017), arXiv:1706.09909 [hep-ph] .
  * Bloch  _et al._ [2021] I. M. Bloch, A. Caputo, R. Essig, D. Redigolo, M. Sholapurkar, and T. Volansky, Exploring new physics with O(keV) electron recoils in direct detection experiments, JHEP 01, 178, arXiv:2006.14521 [hep-ph] .
  * Lin and Yanagida [2023] W. Lin and T. T. Yanagida, Consistency of the string inspired electroweak axion with cosmic birefringence, Phys. Rev. D 107, L021302 (2023), arXiv:2208.06843 [hep-ph] .
  * Langacker  _et al._ [1986] P. Langacker, R. D. Peccei, and T. Yanagida, Invisible Axions and Light Neutrinos: Are They Connected?, Mod. Phys. Lett. A 1, 541 (1986).
  * Clarke and Volkas [2016] J. D. Clarke and R. R. Volkas, Technically natural nonsupersymmetric model of neutrino masses, baryogenesis, the strong CP problem, and dark matter, Phys. Rev. D 93, 035001 (2016), arXiv:1509.07243 [hep-ph] .
  * Sopov and Volkas [2023] A. H. Sopov and R. R. Volkas, VISH$\nu$: solving five Standard Model shortcomings with a Poincaré-protected electroweak scale, Phys. Dark Univ. 42, 101381 (2023), arXiv:2206.11598 [hep-ph] .
  * Di Luzio  _et al._ [2024] L. Di Luzio, A. W. M. Guerrera, X. Ponce Díaz, and S. Rigolin, Axion-Like Particles in Radiative Quarkonia Decays (2024), arXiv:2402.12454 [hep-ph] .
  * Clarke  _et al._ [2015] J. D. Clarke, R. Foot, and R. R. Volkas, Natural leptogenesis and neutrino masses with two Higgs doublets, Phys. Rev. D 92, 033006 (2015), arXiv:1505.05744 [hep-ph] .
  * Blinov  _et al._ [2019] N. Blinov, M. J. Dolan, P. Draper, and J. Kozaczuk, Dark matter targets for axionlike particle searches, Phys. Rev. D 100, 015049 (2019), arXiv:1905.06952 [hep-ph] .
  * Bardeen  _et al._ [1983] J. M. Bardeen, P. J. Steinhardt, and M. S. Turner, Spontaneous Creation of Almost Scale - Free Density Perturbations in an Inflationary Universe, Phys. Rev. D 28, 679 (1983).
  * Seckel and Turner [1985] D. Seckel and M. S. Turner, Isothermal Density Perturbations in an Axion Dominated Inflationary Universe, Phys. Rev. D 32, 3178 (1985).
  * O’Hare [2020] C. O’Hare, cajohare/axionlimits: Axionlimits, https://cajohare.github.io/AxionLimits/ (2020).
  * Du  _et al._ [2018] N. Du _et al._ (ADMX), A Search for Invisible Axion Dark Matter with the Axion Dark Matter Experiment, Phys. Rev. Lett. 120, 151301 (2018), arXiv:1804.05750 [hep-ex] .
  * Braine  _et al._ [2020] T. Braine _et al._ (ADMX), Extended Search for the Invisible Axion with the Axion Dark Matter Experiment, Phys. Rev. Lett. 124, 101303 (2020), arXiv:1910.08638 [hep-ex] .
  * Bartram  _et al._ [2021] C. Bartram _et al._ (ADMX), Search for Invisible Axion Dark Matter in the 3.3–4.2 $\mu$eV Mass Range, Phys. Rev. Lett. 127, 261803 (2021), arXiv:2110.06096 [hep-ex] .
  * Ahyoune  _et al._ [2023] S. Ahyoune _et al._ , A Proposal for a Low-Frequency Axion Search in the 1–2 $\mu$eV Range and Below with the BabyIAXO Magnet, Annalen Phys. 535, 2300326 (2023), arXiv:2306.17243 [physics.ins-det] .
  * Alesini  _et al._ [2023] D. Alesini _et al._ , The future search for low-frequency axions and new physics with the FLASH resonant cavity experiment at Frascati National Laboratories, Phys. Dark Univ. 42, 101370 (2023), arXiv:2309.00351 [physics.ins-det] .
  * Berlin  _et al._ [2021] A. Berlin, R. T. D’Agnolo, S. A. R. Ellis, and K. Zhou, Heterodyne broadband detection of axion dark matter, Phys. Rev. D 104, L111701 (2021), arXiv:2007.15656 [hep-ph] .
  * Brouwer  _et al._ [2022] L. Brouwer _et al._ (DMRadio), Projected sensitivity of DMRadio-m3: A search for the QCD axion below 1 $\mu$eV, Phys. Rev. D 106, 103008 (2022), arXiv:2204.13781 [hep-ex] .
  * Nomura  _et al._ [2000] Y. Nomura, T. Watari, and T. Yanagida, Quintessence axion potential induced by electroweak instanton effects, Phys. Lett. B 484, 103 (2000), arXiv:hep-ph/0004182 .
  * Choi  _et al._ [2021] G. Choi, W. Lin, L. Visinelli, and T. T. Yanagida, Cosmic birefringence and electroweak axion dark energy, Phys. Rev. D 104, L101302 (2021), arXiv:2106.12602 [hep-ph] .
  * Minami and Komatsu [2020] Y. Minami and E. Komatsu, New Extraction of the Cosmic Birefringence from the Planck 2018 Polarization Data, Phys. Rev. Lett. 125, 221301 (2020), arXiv:2011.11254 [astro-ph.CO] .
  * Diego-Palazuelos  _et al._ [2022] P. Diego-Palazuelos _et al._ , Cosmic Birefringence from the Planck Data Release 4, Phys. Rev. Lett. 128, 091302 (2022), arXiv:2201.07682 [astro-ph.CO] .
  * Frampton  _et al._ [2002] P. H. Frampton, S. L. Glashow, and T. Yanagida, Cosmological sign of neutrino CP violation, Phys. Lett. B 548, 119 (2002), arXiv:hep-ph/0208157 .
  * Di Luzio  _et al._ [2023] L. Di Luzio, A. W. M. Guerrera, X. P. Díaz, and S. Rigolin, On the IR/UV flavour connection in non-universal axion models, JHEP 06, 046, arXiv:2304.04643 [hep-ph] .


# Secure Transformer Inference

Mu Yuan, Lan Zhang, Xiang-Yang Li
University of Science and Technology of China
Hefei, China
<EMAIL_ADDRESS>{zhanglan<EMAIL_ADDRESS>

###### Abstract

We present a three-party protocol that can protect both Transformer parameters
and user data during the inference phase. For each feedforward inference
process, our protocol only introduces permutation computation of input and
output data on the user side. Our protocol, Secure Transformer Inference
Protocol (STIP), can be applied to real-world services like ChatGPT.

_K_ eywords inference $\cdot$ large language model $\cdot$ permutation $\cdot$
secure protocol $\cdot$ three-party $\cdot$ Transformer

##  1 Introduction

Applications of Transformer models are exploding, e.g., ChatGPT [1]. Security
is critical to Transformer-based services, which determines whether
applications can be scaled to privacy-sensitive areas like cloud copilot for
proprietary code and documents [2].

Existing work [3, 4] studied this problem under the classic secure multi-party
computing framework. Using encryption and decryption methods requires
approximation of complex nonlinear layers and introduces heavy computational
overhead. In this work, we propose a three-party protocol using permutation to
protect both model parameters and user data without any approximation of
Transformer models.

##  2 Formalization

Let $x\in\mathbb{R}^{n\times d}$ denote the input where $n$ is the sequence
length (e.g., the number of tokens) and $d$ is the model dimension. We define
a Transformer block [5] as a function $f_{\theta}:\mathbb{R}^{n\times
d}\mapsto\mathbb{R}^{n\times d}$ with trainable parameters $\theta$. Then the
Transformer inference, i.e., $f_{\theta}(x)=y$, is computed as follows:

$\displaystyle Q=xW_{q},~{}~{}K=xW_{k},~{}~{}V=xW_{v},$ $\displaystyle
W_{q},W_{k},W_{v}\in\mathbb{R}^{d\times d},$ (1) $\displaystyle
u=\text{softmax}\left(\frac{QK^{T}}{\sqrt{k}}+M\right)VW_{o},$ $\displaystyle
M\in\mathbb{R}^{n\times n},W_{o}\in\mathbb{R}^{d\times d},$ (2) $\displaystyle
v=\text{LayerNorm}(u+x;\gamma_{1},\beta_{1}),$
$\displaystyle\gamma_{1},\beta_{1}\in\mathbb{R}^{d},$ (3) $\displaystyle
z=\text{ReLU}(vW_{1})W_{2},$ $\displaystyle W_{1}\in\mathbb{R}^{d\times
m},W_{2}\in\mathbb{R}^{m\times d},$ (4) $\displaystyle
y=\text{LayerNorm}(z+v;\gamma_{2},\beta_{2}),$
$\displaystyle\gamma_{1},\beta_{1}\in\mathbb{R}^{d},$ (5)

where $k$ is a constant equal to $d$ divided by the number of attention heads,
$M$ denotes the mask which is an all-zero matrix in the encoder and a matrix
whose upper right corner (not including the diagonal) is negative infinity in
the decoder. The parameter $\theta$ consists of attention weights
($W_{q},W_{k},W_{v},W_{o}$), feedforward weights ($W_{1},W_{2}$) and LayerNorm
weights ($\gamma,\beta$).

##  3 Protocol

Let $\pi\in\\{0,1\\}^{d\times d}$ denote a permutation matrix. We transform
the parameters $\theta$ as follows:

$\displaystyle
W_{q}^{\prime}=\pi^{T}W_{q},~{}~{}W_{k}^{\prime}=\pi^{T}W_{k},~{}~{}W_{v}^{\prime}=\pi^{T}W_{v},~{}~{}W_{1}^{\prime}=\pi^{T}W_{1}$
$\displaystyle
W_{o}^{\prime}=W_{o}\pi,~{}~{}W_{2}^{\prime}=W_{2}\pi,~{}~{}\gamma_{1}^{\prime}=\gamma_{1}\pi,~{}~{}\beta_{1}^{\prime}=\beta_{1}\pi,~{}~{}\gamma_{2}^{\prime}=\gamma_{2}\pi,~{}~{}\beta_{2}^{\prime}=\beta_{2}\pi.$

Let $\theta^{\prime}$ denote the transformed parameters, we have:

######  Theorem 1.

$f_{\theta^{\prime}}(x\pi)=f_{\theta}(x)\pi$.

###### Proof.

First, we prove that
LayerNorm$(x\pi;\gamma\pi,\beta\pi)=$LayerNorm$(x;\gamma,\beta)\pi$. The
LayerNorm function is defined for $x\in\mathbb{R}^{n\times d}$ by

$\displaystyle\text{LayerNorm}(x;\gamma,\beta)=\gamma\circ\frac{x-\mu_{x}}{\sigma_{x}}+\beta,$
$\displaystyle\gamma,\beta\in\mathbb{R}^{d},$

where $\circ$ denotes the Hadamard (element-wise) product operator. Since
$\mu_{x}$ and $\sigma_{x}$ are computed by rows, $\mu_{x\pi}=\mu_{x}$ and
$\sigma_{x\pi}=\sigma_{x}$. Therefore,

$\displaystyle\text{LayerNorm}(x\pi;\gamma\pi,\beta\pi)=\gamma\pi\circ\frac{x\pi-\mu_{x}}{\sigma_{x}}+\beta\pi=\left(\gamma\circ\frac{x-\mu_{x}}{\sigma_{x}}+\beta\right)\pi=\text{LayerNorm}(x;\gamma,\beta)\pi.$

Then, since $\forall\pi,\pi\pi^{T}=I$:

$\displaystyle Q^{\prime}=x\pi\pi^{T}W_{q}=xW_{q}=Q,$ $\displaystyle
K^{\prime}=x\pi\pi^{T}W_{k}=xW_{k}=K,$ $\displaystyle
V^{\prime}=x\pi\pi^{T}W_{v}=xW_{v}=V,$ $\displaystyle
u^{\prime}=\text{softmax}\left(\frac{Q^{\prime}K^{\prime
T}}{\sqrt{k}}+M\right)V^{\prime}W_{o}\pi=\text{softmax}\left(\frac{QK^{T}}{\sqrt{k}}+M\right)VW_{o}\pi=u\pi,$
$\displaystyle
v^{\prime}=\text{LayerNorm}(u^{\prime}+x\pi;\gamma_{1},\beta_{1})=\text{LayerNorm}(u\pi+x\pi;\gamma_{1},\beta_{1})=\text{LayerNorm}((u+x)\pi;\gamma_{1},\beta_{1})=v\pi,$
$\displaystyle z^{\prime}=\text{ReLU}(v^{\prime}\pi
W_{1})W_{2}\pi=\text{ReLU}(v\pi\pi^{T}W_{1})W_{2}\pi=\text{ReLU}(vW_{1})W_{2}\pi=z\pi,$
$\displaystyle
y^{\prime}=\text{LayerNorm}(z^{\prime}+v^{\prime};\gamma_{2},\beta_{2})=\text{LayerNorm}(z\pi+v\pi;\gamma_{2},\beta_{2})=\text{LayerNorm}((z+v)\pi;\gamma_{2},\beta_{2})=y\pi,$

i.e., $f_{\theta}^{\prime}(x\pi)=y^{\prime}=y\pi=f_{\theta}(x)\pi$. ∎

Leveraging theorem 1, we present a three-party protocol, named Secure
Transformer Inference Protocol (STIP):

• Party-1 ($P_{1}$): Model developer (e.g., OpenAI) that owns the original
Transformer model $f_{\theta}$.

• Party-2 ($P_{2}$): Cloud computing platform (e.g., Azure) that owns the
computing hardware.

• Party-3 ($P_{3}$): Users that own private input (e.g., prompt token
embedding) and output (e.g., response token logits).

1 Initialization phase:

2 $P_{1}$ randomly generate $\pi\in\mathbb{R}^{d\times d}$;

3 $P_{1}$ transform $f_{\theta}$ to $f_{\theta^{\prime}}$ using $\pi$;

4 $P_{1}$ send $f_{\theta^{\prime}}$ to $P_{2}$ and send $\pi$ to $P_{3}$;

5 Inference phase:

6 $P_{3}$ transform $x$ to $x^{\prime}=x\pi$ and send $x^{\prime}$ to $P_{2}$;

7 $P_{2}$ compute $f_{\theta^{\prime}}(x^{\prime})=y^{\prime}$ and send
$y^{\prime}$ to $P_{3}$;

$P_{3}$ de-transform $y^{\prime}$ by computing $y^{\prime}\pi^{T}$ and get
$y\pi\pi^{T}=y$.

Algorithm 1 Secure Transformer Inference Protocol

Security analysis. Consider $P_{1}$ as the attacker against user data $x,y$,
since $P_{1}$ cannot get access to $x\pi$ and $y\pi$, $P_{1}$ cannot recover
$x,y$ although it has $\pi$. Consider $P_{2}$ as the attacker against model
parameters $\theta$ and user data $x,y$, since $P_{2}$ has $W\pi$ and $x\pi$,
the possibility it guess the correct $\pi$ is $1/(d!)$. In practice, $d$ is
typically larger than 512, e.g., $d=4096$ in llama [6], so the probability of
a successful attack is negligible. Consider $P_{3}$ as the attacker against
model parameters $\theta$, since $P_{3}$ cannot get access to
$\theta^{\prime}$, $P_{3}$ cannot recover $\theta$ although it has $\pi$.

##  4 Discussion

Row-wise permutation. Our protocol permutes $x$ in the column dimension, so a
natural question is: What about doing row-wise permutation? In fact, the
permutation equivariance property ($f(\pi x)=\pi f(x)$) in the sequence length
dimension (row-wise) has been proved for Transformer encoder [7]. For the
encoder attention layer:

$\displaystyle\text{EncAttn}(\pi x)=\text{softmax}\left(\frac{\pi
xW_{q}W_{k}^{T}x^{T}\pi^{T}}{\sqrt{k}}\right)\pi
xW_{v}W_{o}=\pi\text{softmax}\left(\frac{xW_{q}W_{k}^{T}x^{T}}{\sqrt{k}}\right)\pi^{T}\pi
xW_{v}W_{o}=\pi\text{Attn}(x).$

However, due to the mask inside the decoder, attention computation on row-wise
permuted data cannot return recoverable output:

$\displaystyle\text{DecAttn}(\pi x)=\text{softmax}\left(\frac{\pi
xW_{q}W_{k}^{T}x^{T}\pi^{T}}{\sqrt{k}}+M\right)\pi
xW_{v}W_{o}\neq\pi\text{DecAttn}(x).$

A quick fix is to send a transformed $M^{\prime}=\pi M\pi^{T}$ to the cloud
computing platform party. However, since the value of $M$ is fixed (the upper
right corner is negative infinity, and the rest are 0), the cloud computing
platform can easily recover the permutation $\pi$, which will result in loss
of protection.

RMSNorm. Llama [6] uses RMSNorm [8] instead of LayerNorm. Now we prove that
$\text{RMSNorm}(x\pi;\gamma\pi)=\text{RMSNorm}(x;\gamma)\pi$. The RMSNorm
function is defined for $x\in\mathbb{R}^{n\times d}$ by

$\displaystyle\text{LayerNorm}(x;\gamma)=\gamma\circ\frac{x}{\sqrt{\frac{1}{n}\sum_{i}x_{i}^{2}}},$
$\displaystyle\gamma\in\mathbb{R}^{d},$

where $\circ$ denotes the Hadamard (element-wise) product operator. Since
$\sum_{i}x_{i}^{2}$ is computed by rows,
$\sum_{i}(x\pi)_{i}^{2}=\sum_{i}x_{i}^{2}$. Therefore,

$\displaystyle\text{RMSNorm}(x\pi;\gamma\pi)=\gamma\pi\circ\frac{x\pi}{\sqrt{\frac{1}{n}\sum_{i}(x\pi)_{i}^{2}}}=\left(\gamma\circ\frac{x}{\sqrt{\frac{1}{n}\sum_{i}x_{i}^{2}}}\right)\pi=\text{RMSNorm}(x;\gamma)\pi.$

SwiGLU feedforward. Llama [6] uses SwiGLU [9] instead of ReLU in feedforward
layers. Let FFN${}_{\text{SwiGLU}}$ denote the feedforward layers using
SwiGLU, which is defined by:

$\displaystyle\text{FFN}_{\text{SwiGLU}}(x)=((xW_{1})\text{sigmoid}(xW_{1})xW_{3})W_{2},$
$\displaystyle W_{1},W_{3}\in\mathbb{R}^{d\times
m},W_{2}\in\mathbb{R}^{m\times d}.$

We transform parameters as follows:

$\displaystyle
W_{1}^{\prime}=\pi^{T}W_{1},~{}~{}W_{3}^{\prime}=\pi^{T}W_{3},~{}~{}W_{2}^{\prime}=W_{2}\pi,$

and let FFN${}^{\prime}_{\text{SwiGLU}}$ denote the transformed function. Now
we prove that
FFN${}^{\prime}_{\text{SwiGLU}}(x\pi)=$FFN${}_{\text{SwiGLU}}(x)\pi$:

$\displaystyle\text{FFN}^{\prime}_{\text{SwiGLU}}(x\pi)$
$\displaystyle=((x\pi\pi^{T}W_{1})\text{sigmoid}(x\pi\pi^{T}W_{1})x\pi\pi^{T}W_{3})W_{2}\pi$
$\displaystyle=((xW_{1})\text{sigmoid}(xW_{1})xW_{3})W_{2}\pi=\text{FFN}_{\text{SwiGLU}}(x)\pi.$

Applicable scope. In fact, STIP is applicable to models that are built with
any global matrix multiplication-based (e.g., attention and feedforward)
layers and row-wise (e.g., LayerNorm) layers. To give some counterexamples,
STIP cannot be applied to convolutional layers.

Our test code of STIP for the original Transformer [5] and llama [6] can be
found in https://github.com/yuanmu97/secure-transformer-inference.

##  5 Conclusion

In this paper, we present a secure protocol (STIP) for serving Transformer
models in a three-party setting.

## Acknowledgments

We would like to thank Yihang Cheng, Miao-Hui Song, Ning-Kang Zhang, Puhan
Luo, and Junyang Zhang for their contributions to the protocol design.

## References

  * [1] OpenAI.  Chatgpt.  https://openai.com/blog/chatgpt, 2022.
  * [2] Microsoft.  Microsoft 365 copilot.  https://blogs.microsoft.com/blog/2023/03/16/introducing-microsoft-365-copilot-your-copilot-for-work/, 2023.
  * [3] Xiaoyang Hou, Jian Liu, Jingyu Li, Yuhan Li, Wen-jie Lu, Cheng Hong, and Kui Ren.  Ciphergpt: Secure two-party gpt inference.  Cryptology ePrint Archive, 2023.
  * [4] Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei.  The-x: Privacy-preserving transformer inference with homomorphic encryption.  arXiv preprint arXiv:2206.00216, 2022.
  * [5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [6] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [7] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh.  Set transformer: A framework for attention-based permutation-invariant neural networks.  In International conference on machine learning, pages 3744–3753. PMLR, 2019.
  * [8] Biao Zhang and Rico Sennrich.  Root mean square layer normalization.  Advances in Neural Information Processing Systems, 32, 2019.
  * [9] Noam Shazeer.  Glu variants improve transformer.  arXiv preprint arXiv:2002.05202, 2020.


# PAT: Position-Aware Transformer for Dense Multi-Label Action Detection

Faegheh Sardari1 Armin Mustafa1 Philip J. B. Jackson1 Adrian Hilton1 1 Centre
for Vision, Speech and Signal Processing (CVSSP)
University of Surrey, UK
<EMAIL_ADDRESS>
###### Abstract

We present PAT, a transformer-based network that learns complex temporal co-
occurrence action dependencies in a video by exploiting multi-scale temporal
features. In existing methods, the self-attention mechanism in transformers
loses the temporal positional information, which is essential for robust
action detection. To address this issue, we (i) embed relative positional
encoding in the self-attention mechanism and (ii) exploit multi-scale temporal
relationships by designing a novel non-hierarchical network, in contrast to
the recent transformer-based approaches that use a hierarchical structure. We
argue that joining the self-attention mechanism with multiple sub-sampling
processes in the hierarchical approaches results in increased loss of
positional information. We evaluate the performance of our proposed approach
on two challenging dense multi-label benchmark datasets, and show that PAT
improves the current state-of-the-art result by $1.1\%$ and $0.6\%$ mAP on the
Charades and MultiTHUMOS datasets, respectively, thereby achieving the new
state-of-the-art mAP at $26.5\%$ and $44.6\%$, respectively. We also perform
extensive ablation studies to examine the impact of the different components
of our proposed network.

##  1 Introduction

Action or event detection aims to determine the boundaries of different
actions/events occurring in an untrimmed video, and plays a crucial role in
various important computer vision applications, such as video summarization,
highlighting, and captioning. Despite the recent advances in different areas
of video understanding, dense multi-label action detection is still an
unsolved problem and considered as one of the most challenging video analysis
tasks since the videos are untrimmed, and include several actions with
different time durations that can have overlap (See Fig. 1). To carry out this
task, we require to learn complex short and long term temporal relationships
amongst different actions in a video which is a challenging problem [7, 15].

Figure 1: A sample video and its corresponding action annotations from the
Charades dataset [31] where the video includes several action types with
different time spans, from short to long, and in each time step, multiple
actions can occur at the same time.

Most previous dense multi-label action detection approaches capture temporal
dependencies through temporal convolutional networks [26, 12, 15]. However,
with the success of transformer networks over the convolutional networks for
modeling complex and sequential relationships [35, 9, 10, 24, 36], recently, a
few methods, such as [33, 5, 7], leverage the self-attention mechanism and
propose transformer-based approaches where they achieve state-of-the-art
performance. Authors in [33, 5] design their network by modeling explicitly
temporal cross-class relations. In [33], there are two transformer modules
such that one of them investigates action relationships for each temporal
moment, and another one learns temporal dependencies for each action type.
However, these approaches are not computationally efficient and their
complexity grows with the number of action classes. To overcome this, Dai et
al. [7] design a hierarchical network that learns temporal-action dependencies
from multi-scale temporal features. Their network contains several transformer
layers such that the output of each layer is down-sampled and given as input
into its subsequent layer. As stated in [30, 18, 11], the self-attention
mechanism in the transformer is order-invariant and loses positional
information, and when the self-attention is embedded in a hierarchical
structure, the issue becomes worse as using multiple down-sampling processes
results in increased loss of positional information, especially in top layers.
In this paper, we tackle these issues by introducing PAT, a position-aware
transformer network for dense action detection. PAT consists of three main
modules: fine detection, coarse detection, and classification. The fine
detection module learns fine-grained action dependencies from the full
temporal resolution of the video sequence for coarse detection and
classification modules. The coarse detection module captures various ranges of
coarse action dependencies from the fine-grained features using a non-
hierarchical structure, which preserves the positional information. To further
leverage the positional information, PAT incorporates a learnable relative
positional encoding [29] in the transformer layers of both fine and coarse
detection models. Finally, the classification module estimates the
probabilities of different action classes for every timestamp in the input
video using both fine and coarse-grained action dependencies. Our key
contributions can be summarized as follows:

  * •

For the first time, we introduce the idea of leveraging positional information
in transformers for action detection

  * •

We design a novel non-hierarchical transformer-based network that preserves
positional information when learning multi-scale temporal action dependencies

  * •

We evaluate the proposed method’s performance on two challenging benchmark
dense action detection datasets where we outperform the current state-of-the-
art result by $1.1\%$ and $0.6\%$ per-frame mean average precision (mAP) on
Charades and MultiTHUMOS respectively, thereby achieving the new state-of-the-
art mAP at $26.5\%$ and $44.6\%$, respectively

  * •

We perform extensive ablation studies to evaluate our network design

##  2 Related Works

Although action detection [4, 22, 20, 22, 25, 40, 34, 38, 3] has been studied
significantly in computer vision, few works [27, 8, 26, 6, 15] have explored
it in a dense multi-labelled setup where instances of different actions or
events can overlap in different parts of a video. In this section, we review
the action detection approaches by focusing on a dense-labelled setting.

To detect the boundaries of different actions, the authors in [4, 21, 23, 19]
propose anchor-based methods where they first generate several proposals for
each frame of video by using multi-scale anchor boxes, and then refine them to
achieve the final action boundaries. However, these approaches are not usually
applied for a dense multi-label scenario, as to model effectively the dense
action distributions, they need a large amount of anchors [7]. To overcome
this, some works, such as [27, 8, 26, 6, 15], design anchor-free approaches
for dense action detection. Piergiovanni and Ryoo [27] propose a network that
represents an untrimmed video into multi-activity events. They design multiple
temporal Gaussian filters which are applied separately on the video frame
features while a soft-attention mechanism is employed to combine the output of
the filters to generate a global representation. Later in [26], they improve
their work by proposing a temporal convolutional network using Gaussian
filters as kernels to perform the temporal representation in a more efficient
and effective way. Although they design networks to address complex multi-
label action detection, the proposed models are not able to encode long-term
dependencies and mostly focus on local relationships, while our proposed
network is able to capture different ranges of temporal features from short to
long. Kahatapitiya and Ryoo [15] propose a two-stream network to capture long
term information such that one of the streams learns the most informative
frame of a long video through a dynamic sub-sampling with a ratio of 4, and
the other one learns the fine-grained contexts of the video from the full
resolution. Although their results are promising, it cannot be adapted easily
to use more temporal resolutions as it requires a dedicated Convolutional
Neural Network (CNN), _i.e_. X3D [12], for each resolution, whereas in our
proposed method, a different resolution can be processed easily by adding an
extra branch containing a few transformer blocks in the coarse detection
module.

Transformer-based Approaches –  With the success of transformer networks in
modeling complex relationships and capturing short and long term dependencies
[35, 9, 10, 24, 36], some works, such as [6, 33, 7], develop transformer-based
approaches for dense action detection task. Tirupattur et al. [33] design a
model with two transformer branches, one branch applies self-attention across
all action classes for each time step to learn the relationships amongst
actions, and another branch uses self-attention across time frames to model
the temporal dependencies, and the output of two branches are combined for
action classification. Although this method outperforms state-of-the-art
results, the method’s computational complexity increases with the number of
action classes. Similar to [15] that benefits different temporal resolutions,
Dai et al. [7] extract multi-scale features. They design a transformer-based
hierarchical structure and provide multi-resolution temporal features through
several sub-sampling processes. However, as the self-attention mechanism does
not preserve the temporal position information [30, 18, 11], joining it with
multiple sub-sampling processes makes the network lose more positional
information while preserving this information is essential for action
detection. In contrast, our position-aware transformer network PAT has been
designed to retain such temporal cues.

Figure 2: The overall schema of the proposed network PAT including (i) video
encoder E, (ii) fine detection module FDM, (iii) Coarse detection module CDM,
and (iv) classification module CLASM.

##  3 Position-Aware Transformer (PAT)

Problem Definition –  Our aim is to detect different actions/events in a
densely-labelled untrimmed video. We define the action detection problem under
this setting as [15, 33, 7]. For an untrimmed video sequence with a length of
$T$, each timestamp $t$ has a ground truth action label
${G}_{t}=\\{{{g}_{t,c}}\in\\{0,1\\}\\}^{C}_{c=1}$, where $C$ is the maximum
number of action classes in the dataset, and the network requires to estimate
action class probabilities $Y_{t}=\\{y_{t,c}\in[0,1]\\}^{C}_{c=1}$ for each
timestamp.

###  3.1 Proposed Network

Our proposed method PAT is a transformer-based network designed to exploit
different granularities of complex temporal dependencies for action detection.
The PAT network includes a video encoder E that encodes an input video
sequence into a sequence of input tokens, and three main components: fine
detection module FDM, coarse detection module CDM, and classification module
CLASM arranged as shown in Fig. 2. FDM processes an input sequence in its
original temporal resolution to obtain a fine-grained action representation
for both CDM and CLASM modules. The CDM module learns different ranges of
temporal action dependencies amongst the fine-grained features through
extracting and combining multi-scale temporal features. CLASM estimates class
probabilities from the output of both FDM and CDM modules.

Video Encoder (E) – To process an input video, PAT needs to convert it into a
sequence of tokens. To perform this, similar to the previous action detection
approaches [33, 7, 40], we first divide the L-frame input video
$V\in\rm{I\\!R}^{L\times Ch\times W\times H}$ into T non-overlapped segments
$S=\\{S_{t}\\}^{T}_{t=1}$, where ${S_{t}}\in\rm{I\\!R}^{Z\times Ch\times
W\times H}$, $Z=L/T$, and $Ch$, $W$, and $H$ define number of channels, width,
and height of each video frame respectively. Then, the video encoder E that is
a pre-trained convolutional network is employed on each segment to generate
its corresponding token $I_{t}={E}(S_{t})$, where $I_{t}\in\rm{I\\!R}^{D}$.

Relative Positional Transformer (RPT) Block – To design FDM, and CDM, we
employ our proposed transformer block RPT (see Fig. 3). The RPT block
comprises a transformer layer with relative positional embedding followed by a
local relational LR component containing two linear layers and one 1D temporal
convolutional layer as in [7] to enhance the output of the transformer layer.

As already pointed out in Section 1, the transformer self-attention mechanism
loses the order of temporal information while preserving this information is
essential for action detection, where we need to localise events precisely in
a video sequence. To solve this issue, Vaswani et al. [35] propose to add the
absolute positional embedding to the input tokens. However, in our
experiments, we observed that using the absolute positional embedding
decreases the method’s performance significantly (see Section 4.1). This has
also been observed in [7, 40]. The decrease in performance may be attributed
to breaking the translation-invariant property of the method. In action
detection, we expect the proposed method to be translation-invariant, _i.e_.
the network learns the same representation for the same video frames in two
temporally shifted videos, regardless of how much they are shifted, while the
absolute encoding can break this property as it adds different positional
encodings to the same frames in the shifted video inputs. To overcome this, we
propose to use relative positional encoding [29] in the transformer layers of
our RPT block. The relative positional encoding employs a relative pairwise
distance between every two tokens and is translation-invariant. In addition,
as the embedding is performed in each transformer layer and is passed into the
subsequent layer, the positional information can flow to the classification
module where the final estimations are provided.

We briefly formulate the transformer layer in the RPT block. In the H-head
self-attention layer of RPT, for each head $h\in\\{1,2,...,H\\}$, the input
sequence $X\in\rm{I\\!R}^{N\times D^{\diamond}}$ is first transferred into
query ${Q_{h}}$, key ${K_{h}}$, and value ${V_{h}}$ through linear operations

${Q}_{h}=XW_{h}^{q},~{}~{}{K}_{h}=XW^{k}_{h},~{}~{}{V}_{h}=XW^{v}_{h},$ (1)

where ${Q_{h}}$, ${K_{h}}$, ${V_{h}}\in\rm{I\\!R}^{N\times D_{h}}$,
$W_{h}^{q},~{}W_{h}^{k},~{}W_{h}^{v}\in\rm{I\\!R}^{D^{\diamond}\times D_{h}}$
refer the weights of linear operations, and $D_{h}=\frac{D^{\diamond}}{H}$.
Then, the self-attention with relative positional embedding is computed for
each head as

${A}_{h}=softmax(\frac{{Q}_{h}{K}^{T}_{h}+{P}^{\triangleright}_{h}}{\sqrt{D_{h}}}){V}_{h},$
(2)
${P}^{\triangleright}_{h}(n,m)=\sum_{d=1}^{D_{h}}{Q}_{h}(n,d)\Omega_{d}(n-m),$
(3)

where ${{P}^{\triangleright}_{h}}\in\rm{I\\!R}^{N\times N}$,
$n,m\in\\{1,2,...,N\\}$, and $\Omega_{d}$ operates as $D_{h}$ different
embeddings for time intervals based on the queries [29]. To compute
${{P}^{\triangleright}_{h}}$, we use the memory-efficient method proposed by
Huang et al. [13].

Finally, the self-attention of all heads are concatenated and fed into a
linear layer to output sequence ${O}$

${A}=concat(A_{1},A_{2},...,A_{m}),$ (4) ${O}={A}W^{o}+X,$ (5)

where ${A}\in\rm{I\\!R}^{N\times D^{\diamond}}$ and
$W^{o}\in\rm{I\\!R}^{D^{\diamond}\times D^{\diamond}}$.

Figure 3: Architecture of the proposed RPT block. For brevity, the computation
of the heads are not shown separately.

Fine Detection Module (FDM) – The FDM module aims to obtain a fine-grained
temporal action dependency representation of the video from the input video
sequence for the CDM and CLASM modules. FDM includes a 1D temporal
convolutional layer followed by $B$ RPT blocks. The convolution layer
${\Lambda^{\bullet}}$ has a kernel size of three and a stride of one to map
all the input tokens $I\in\rm{I\\!R}^{T\times D}$ into a lower dimension
$D^{*}$, and then the RPT blocks are applied to learn the fine-grained
dependencies $I^{\odot}$.

${I}^{\odot}=RPT^{FDM}_{1:B}({\Lambda^{\bullet}}(I)),$ (6)

where $I^{\odot}\in\rm{I\\!R}^{T\times D^{*}}$ and $D^{*}<D$.

Coarse Detection Module (CDM) – In the CMD module, we aim to learn a coarse
temporal action dependency representation of the video. To achieve this, one
solution is to extract and combine multi-scale temporal features through a
hierarchical structure, such as the proposed method in [7, 40] (see Fig. 4.
a). However, as we already explained in Section 1, using multiple sub-sampling
processes in the hierarchical structure results in losing more positional
information in the top layers of the network. Our CMD module has been designed
to overcome this issue by extracting different scales of features from the
same full-scale fine-grained information and through only one sub-sampling
process, (see Fig. 4. b). In Section 4.1, we show that our novel non-
hierarchical design to extract multi-scale features outperforms significantly
a hierarchical structure.

Figure 4: The proposed hierarchical structure in [7, 40] vs. our proposed non-
hierarchical design in fine and coarse detection modules to extract multi-
scale features for action detection.

The CMD module has $F$ granularity branches such that each branch learns a
different scale of temporal features. In the $i^{th}$ branch, first a 1D
temporal convolutional layer $\Lambda^{\circ}_{i}$ with a kernel size of three
and a stride of $2^{i}$ is applied on the fine-grained inputs received from
the preceding module FCM as

${I}^{g_{i}}=\Lambda^{\circ}_{i}({I}^{\odot}),$ (7)

where ${I}^{g_{i}}\in\rm{I\\!R}^{T^{i}\times D^{*}}$, $i\in\\{1,2,...,F\\}$,
and $T^{i}=\frac{T}{2^{i}}$. Then, the down-sampled features are given into
$B$ RPT transformer blocks to exploit the temporal dependencies amongst them

$\bar{I}^{g_{i}}=RPT^{CDM_{i}}_{1:B}({I}^{g_{i}}),$ (8)

where ${\bar{I}}^{g_{i}}\in\rm{I\\!R}^{T^{i}\times D^{*}}$. Note, to extract
all the scales of features, the striding process (sub-sampling) is used only
once and as the relative positional information has been already embedded in
the fine-grained information, after the striding process, the sub-sample
features keep the temporal positional cues.

In the CLASM module, action class probabilities are estimated for each input
token generated by the video encoder E. Therefore, the CMD module requires
obtaining a coarse dependency representation at the original temporal length.
To do this, we up-sample and combine different scales of coarse features to
provide a final coarse representation $I^{\circledcirc}$ as

$\hat{{I}}^{g_{i}}=UpSample(\bar{I}^{g_{i}}),$ (9)
$I^{\circledcirc}=\sum_{i=1}^{F}\hat{{I}}^{g_{i}},$ (10)

where $\hat{{I}}^{g_{i}},I^{\circledcirc}\in\rm{I\\!R}^{T\times D^{*}}$ and
linear interpolation is employed for up-sampling.

Classification Module (CLASM) – This module obtains the action class
probabilities for action detection from the fine and coarse contexts. To this,
two convolution blocks $CLAS^{\odot}$ and $CLAS^{\circledcirc}$ that include
two 1D convolution filters with kernel size one and stride one are applied on
the fine and coarse features separately to predict $C$ action class
probabilities for each temporal moment

$Y^{\phi}=Sig(CLAS^{\phi}(I^{\phi})),$ (11)

where $Y^{\phi}\in\rm{I\\!R}^{T\times C}$, $\phi\in\\{\odot,\circledcirc\\}$,
and $Sig$ refers to sigmoid activation function. Then, at inference, the final
estimation is computed by combining them as

$\hat{Y}={\sum_{\phi}\alpha_{\phi}Y^{\phi}},$ (12)

where $\hat{Y}\in\rm{I\\!R}^{T\times C}$ and
$\alpha_{\odot}+~{}\alpha_{\circledcirc}=1$.

###  3.2 Network Optimization

To optimize action detection models, binary cross entropy (BCE) is usually
used as in [33, 7, 40, 34]. However, in the multi-label setting, the number of
positive labels may become more than the number of negative ones. This
unbalanced number of positive and negative labels can result in poor
performance in the action detection task if we employ BCE for training, since
it does not have any control on the contribution of positive and negative
samples. To overcome this, we propose to adapt Asymmetric loss
$\mathcal{L}_{asl}$ [28] for multi-label action detection. Therfore, the total
loss $\mathcal{L}_{total}$ is computed as

$\mathcal{L}_{total}=\frac{1}{T}{\sum_{\phi}}\sum_{t=1}^{T}\sum_{c=1}^{C}\alpha_{\phi}\mathcal{L}_{asl}({g}_{t,c},{y}^{\phi}_{t,c}),$
(13)
$\mathcal{L}_{asl}({g}_{t,c},{y}^{\phi}_{t,c})=-{g}_{t,c}\mathcal{L}_{+}-(1-{g}_{t,c})\mathcal{L}_{-},$
(14) $\mathcal{L}_{+}=(1-{y}^{\phi}_{t,c})^{\gamma_{+}}log({y}^{\phi}_{t,c}),$
(15)
$\mathcal{L}_{-}={{(\acute{y}^{\phi}_{t,c}}})^{\gamma_{-}}log(1-\acute{y}_{t,c}),$
(16) $\acute{y}^{\phi}_{t,c}=max({y}^{\phi}_{t,c}-\delta,0),$ (17)

where ${g}_{t,c}$ indicates the ground truth label of action class $c$ in
temporal step $t$, and ${y}^{\phi}_{t,c}$ is its corresponding class
probability estimated by Eq. 11. $\gamma_{+}$ and $\gamma_{-}$ are focusing
parameters for positive and negative labels respectively and if we choose
$\gamma_{+}<\gamma_{-}$, we are able to increase the contribution of positive
samples. Furthermore, Eq. 17 applies another asymmetric mechanism by
discarding the very easy negative samples through setting the threshold
parameter $\delta$. In Section 4.1, we show that optimizing the proposed
network through Asymmetric loss instead of BCE improves the method’s
performance.

##  4 Experimental Results

Datasets – There are several benchmark datasets for action detection, but only
a few of them provide dense multi-label annotations. For instance, videos in
ActivityNet [1] have only one action type per timestamp. We present the
results of PAT on two challenging dense multi-label benchmark datasets,
Charades [31] and MultiTHUMOS [39].

Charades [31] is a large dataset including $9,848$ videos of daily activities
of 267 persons. It contains $66,500$ temporal interval annotations for 157
action classes while there is a high overlap amongst the action instances of
different action categories. To evaluate our method on Charades, we follow
previous methods [15, 33, 7] and use the same training and testing set as in
[31].

MultiTHUMOS contains the same set of 413 videos as in THUMOS’14 dataset [14].
However, MultiTHUMOS is more challenging than THUMOS’14 since (i) the
annotations have been extended from 20 action classes to 65, and (ii) in
contrast to sparse-lable frame-level annotations in THUMOS’14, MultiTHUMOS has
dense multi-label action annotations. To obtain the results on this dataset,
we use the same standard training and testing splits applied by previous
methods [33, 7]. Following state-of-the-art methods [27, 26, 15, 33, 7], we
evaluate our method on these datasets by standard per-frame mAP metric.

Implementation Details – Similar to the proposed method in [7], during both
training and inference, PAT uses a fixed number of $T=256$ input tokens. For
training, we randomly sample a clip containing $T$ consecutive tokens from a
video sequence. At inference, we follow previous work [33, 15] and make the
predictions for a full video sequence. Each input token is provided by
applying the video encoder E on an 8-frame segment to extract a feature vector
with dimension $D=1024$. The video encoder E is implemented by using a pre-
trained I3D [2]111Video encoder E is pre-trained on Kinetic-400 [16] and
training set of Charades for MultiTHUMOS and Charades respectively. while its
fully connected layers are replaced with a global average pooling layer and
its parameters are frozen. In the convolutional layer of FDM, the input
features are mapped into $D^{*}=512$ dimensional feature vectors. Note, the
feature dimension $D^{*}=512$ is fixed for the rest of the network. FDM and
each granularity branch of CDM have $B=3$ RPT blocks with $H=8$ multi-head
attention heads, and the number of granularity branches in CMD is set to $F=3$
as we found that with these parameters, PAT obtains the best performance. The
contributing factors for fine-grained ($\alpha_{\odot}$) and coarse-grained
($\alpha_{\circledcirc}$) features in the CLASM module are set empirically to
$\\{\alpha_{\odot}=0.1,\alpha_{\circledcirc}=0.9\\}$ and
$\\{\alpha_{\odot}=0.7,\alpha_{\circledcirc}=0.3\\}$ for Charades and
MultiTHUMOS respectively. In Asymmetric loss, we use factors of $\gamma_{+}=1$
and $\gamma_{-}=3$ for the impact of positive and negative samples
respectively, and threshold parameter $\delta=0.1$, which are determined
through trial and error.

Our experiments were performed under Pytorch on an NVIDIA GeForce RTX 3090
GPU, and we trained our model using the Adam optimiser [17] with an initial
learning rate of 0.0001 and batch size 3 for 25 and 300 epochs for Charades
and MultiTHUMOS datasets respectively. The learning rate was decreased by a
factor of 10 every 7 and 130 epochs for Charades and MultiTHUMOS respectively.
Note, using different training settings for Charades and MultiTHUMOS is due to
their different size.

###  4.1 Ablation Studies

In this section, we examine our design decisions for the proposed network and
learning paradigm.

Effect of FDM and CDM Modules – Here, we aim to evaluate the impact of the
fine and coarse detection modules (FDM and CDM) in the final results of PAT.
Table 1 shows per-frame mAP on the Charades and MultiTHUMOS datasets as we
remove each or both of FDM and CDM modules. To obtain the results of the
network when both modules are dropped, we use directly the sequence of input
tokens generated by the video encoder (I3D) for action detection. Table 1
shows that using only input tokens generated by I3D network is not enough for
effective action detection and employing fine and coarse-grained temporal
features obtained by FDM and CDM improves the performance by $9.7\%$ and
$7.9\%$ per-frame mAP on Charades and MultiTHUMOS respectively. It also shows
that both FDM and CDM modules have an important contribution to the final
results as by removing FDM and CDM, our results deteriorate by $2.4\%$ and
$3.4\%$ per-frame mAP on average on both datasets respectively. Table 1 also
shows that for different datasets that have different action types, the
contribution of fine and coarse features might be different which is the
reason we use the contribution factors
$\\{\alpha_{\odot},\alpha_{\circledcirc}\\}$ to combine the prediction results
of FDM and CDM in the CLASM module at the inference.

Module | mAP(%)
---|---
Charades | MultiTHUMOS
CLASM | 16.8 | 36.7
FDM, CLASM | 23.8 | 40.5
CDM, CLASM | 26.2 | 40.1
FDM, CDM, CLASM | 26.5 | 44.6

Table 1: Ablation studies on FDM and CDM modules of PAT on the Charades and
MultiTHUMOS dataset using RGB videos in terms of per-frame mAP metric.

Effect of Structure Design to Extract Multi-Scale Features – In this section,
we examine the design of PAT with two other variants to capture fine-grained
and coarse-grained features. In the first variant PAT-$v_{1}$, the CDM module
uses the hierarchical structure to extract the multi-scale features while the
rest of its architecture is the same as PAT. In the second variant
PAT-$v_{2}$, the CDM module has a non-hierarchical structure, the same as PAT,
but the FDM module and all granularity branches in CDM learn their features
from input tokens.

Design | mAP(%)
---|---
Charades | MultiTHUMOS
PAT-$v_{1}$ (Hierarchical) | 25.1 | 44.0
PAT-$v_{2}$ | 26.1 | 44.2
PAT | 26.5 | 44.6
Table 2: Ablation studies on structure design of the proposed method on the
Charades and MultiTHUMOS datasets using RGB videos in terms of per-frame mAP
metric.

Table 2 shows that when CMD applies a hierarchical structure to learn the
coarse-grained features, _i.e_. PAT-$v_{1}$, the method’s performance drops
$1.4\%$ and $0.6\%$ on Charades and MultiTHUMOS respectively. This proves the
contribution of our novel non-hierarchical transformer-based design which
preserves positional information when exploiting the multi-scale features.
Furthermore, in case we apply a non-hierarchical CMD, _i.e_. PAT and
PAT-$v_{2}$, if the CMD module extracts the multi-scale features from the
fine-grained context instead of the input tokens as in PAT, we achieve the
best performance at $26.5\%$ and $44.6\%$ per-frame mAP on Charades and
MultiTHUMOS respectively.

Impact of Relative Positional Encoding – Table 3 shows the performance of PAT
when different positional encodings are applied. It can be observed that
employing the relative positional encoding [29, 13] embedded in the RPT block
improves the method’s performance by $0.3\%$ per-frame mAP on both datasets,
while adding absolute positional encoding [35] into the input tokens
deteriorates the method’s performance significantly.

Positional Encoding | mAP(%)
---|---
Charades | MultiTHUMOS
No encoding | 26.2 | 44.3
Absolute | 25.3 | 43.5
Relative | 26.5 | 44.6

Table 3: Ablation studies on positional encoding used in PAT on the Charades
and MultiTHUMOS dataset using RGB videos in terms of per-frame mAP metric.

Impact of Loss Function – Here, we examine the effect of BCE and Asymmetric
[28] losses for training. As shown in Table 4, applying the Asymmetric loss
[28] to optimize PAT improves the performance by $0.5\%$ and $0.2\%$ per-frame
mAP on Charades and MultiTHUMOS respectively.

Loss | mAP(%)
---|---
Charades | MultiTHUMOS
BCE | 26.0 | 44.4
Asymmetric [28] | 26.5 | 44.6
Table 4: Ablation studies on the loss function applied for training PAT on the
Charades and MultiTHUMOS datasets using RGB videos in terms of per-frame mAP
metric.

Discussion and Analysis – The ablation studies show that leveraging positional
information in the transformer layers has an important contribution in the
final results of the network where extracting the multi-scale temporal
features through our proposed non-hierarchical design in CMD outperforms a
hierarchical structure by $1.0\%$ mAP on average on both datasets (PAT vs
PAT-v1), and embedding the relative position encoding in the RPT block
improves the performance by $0.3\%$ mAP on both datasets. Our further
ablations also reveal the effect of the Asymmetric loss in optimizing of PAT
where it increases the performance by $0.3\%$ mAP on average on both datasets.

###  4.2 State-of-the-Art Comparison

In this section, we compare the performance of the proposed method with the
state-of-the-art action detection approaches including both transformer-based
methods and the methods that do not use self-attention. Both quantitative and
qualitative results are obtained for this section.

Table 5 provides comparative results on the benchmark datasets Charades and
MultiTHUMOS based on the standard per-frame mAP metric. Table 5 shows that our
proposed method outperforms the current state-of-the-art result by $1.1\%$ and
$0.6\%$ on Charades and MultiTHUMOS respectively and achieves a new state-of-
the-art per-frame mAP results at $26.5\%$ and $44.6\%$ on Charades and
MultiTHUMOS respectively.

Method | GFLOPs | Backbone | mAP(%)
---|---|---|---
Charades | MultiTHUMOS
R-C3D [37] | ICCV 2017 | - | C3D | 12.7 | -
SuperEvent [27] | CVPR 2018 | 0.8 | I3D | 18.6 | 36.4
TGM [26] | ICML 2019 | 1.2 | I3D | 20.6 | 37.2
PDAN [6]✓$*$ | WACV 2021 | 3.2 | I3D | 23.7 | 40.2
CoarseFine [15] | CVPR 2021 | - | X3D | 25.1 | -
MLAD [33]✓ | CVPR 2021 | 44.8 | I3D | 18.4 | 42.2
CTRN [5]✓ | BMVC 2021 | - | I3D | 25.3 | 44.0
PointTAD [32] | NeurIPS 2022 | - | I3D | 21.0 | 39.8
MS-TCT [7]✓ | CVPR 2022 | 6.6 | I3D | 25.4 | 43.1
PAT✓ |  | 8.5 | I3D | 26.5 | 44.6
Table 5: Action detection results on Charades and MultiTHUMOS datasets using RGB videos in terms of per-frame mAP. The ✓ symbol highlights the transformer-based approaches, and $*$ indicates the results are taken from [7]. Method | $\tau=0$ |  | $\tau=20$ |  | $\tau=40$
---|---|---|---|---|---
$P_{AC}$ | $R_{AC}$ | ${F1}_{AC}$ | ${mAP}_{AC}$ |  | $P_{AC}$ | $R_{AC}$ | ${F1}_{AC}$ | ${mAP}_{AC}$ |  | $P_{AC}$ | $R_{AC}$ | ${F1}_{AC}$ | ${mAP}_{AC}$
I3D[2]$*$ | 14.3 | 1.3 | 2.1 | 15.2 |  | 12.7 | 1.9 | 2.9 | 21.4 |  | 14.9 | 2.0 | 3.1 | 20.3
CF [33]$*$ | 10.3 | 1.0 | 1.6 | 15.8 |  | 9.0 | 1.5 | 2.2 | 22.2 |  | 10.7 | 1.6 | 2.4 | 21.0
MLAD [33]✓ | 19.3 | 7.2 | 8.9 | 28.9 |  | 18.9 | 8.9 | 10.5 | 35.7 |  | 19.6 | 9.0 | 10.8 | 34.8
MS-TCT [7]✓ | 26.3 | 15.5 | 19.5 | 30.7 |  | 27.6 | 18.4 | 22.1 | 37.6 |  | 27.9 | 18.3 | 22.1 | 36.4
PAT✓ | 28.3 | 26.1 | 27.2 | 32.0 |  | 30.0 | 29.2 | 29.6 | 37.8 |  | 30.0 | 29.1 | 29.4 | 36.7
Table 6: Action detection results on Charades dataset based on the action-
conditional metrics [33], $P_{AC}$, $R_{AC}$, ${F1}_{AC}$, and ${mAP}_{AC}$.
$\tau$ refers the temporal window size. The same as [7, 33], both RGB and
optical flow are used for obtaining the results. The ✓ symbol highlights the
transformer-based approaches, and $*$ indicates the results are taken from
[33].

We also evaluate the performance of our proposed method by action-conditional
metrics including Action-Conditional Precision $P_{AC}$, Action-Conditional
Recall $R_{AC}$, Action-Conditional F1-Score ${F1}_{AC}$, and Action-
Conditional Mean Average Precision ${mAP}_{AC}$, as introduced in [33]. The
aim of these metrics is to measure the ability of the network to learn both
co-occurrence and temporal dependencies of different action classes. The
metrics are measured throughout a temporal window with a size of $\tau$. As
shown by the results on Charades in Table 6, the proposed method PAT achieves
state-of-the-art results on all action-conditional metrics, specifically, it
improves the state-of-the-art results significantly on $R_{AC}$ and
${F1}_{AC}$ by $10.6\%$ and $7.7\%$, $10.8\%$ and $7.5\%$, and $10.8\%$ and
$7.3\%$ where $\tau$ is 0, 20, and 40 respectively.

Fig. 5 displays qualitative results of PAT on a test video sample of Charades
and compares them with the outputs of MS-TCT [7]. Amongst the state-of-the-art
methods, we applied MS-TCT [7] and MLAD [33] on the video sample, since their
code is available, useable and compatible with our hardware. However, as the
MLAD could not predict any of the actions, we reported only the results of MS-
TCT. The results in Fig. 5 show that our proposed method’s action predictions
have a better overlap with the ground-truth labels, and our method detected
more action instances in the video than MS-TCT, _i.e_. PAT predicted all
action types except “Taking a bag” while MS-TCT could not detect “Taking a
picture”, “Taking a bag”, and “Walking”.

Figure 5: Visualization of action predictions by our proposed method PAT and
MS-TCT [7] on a test video sample of Charades including 7 different action
types.

##  5 Conclusion

In this work, we introduced a novel transformer-based network PAT that
exploits different ranges of temporal dependencies for action detection. The
proposed method has been designed to benefit from preserving temporal
positional information in learning multi-granularity features by (i) embedding
the relative positional encoding in its transformer layers and (ii) a non-
hierarchical design. We evaluated PAT on two densely-labelled challenging
benchmark action detection datasets, on which we achieved new state-of-the-art
results, and our ablation studies demonstrated the effectiveness of different
components of our proposed network. For future work, we will investigate
adapting our network to learn spatial and temporal dependencies from raw
pixels and also use audio information to improve the performance of action
detection.

## Acknowledgement

This research is supported by UKRI EPSRC Platform Grant EP/P022529/1, and
EPSRC BBC Prosperity Partnership AI4ME: Future Personalised Object-Based Media
Experiences Delivered at Scale Anywhere EP/V038087/1.

## References

  * Caba Heilbron et al. [2015] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  ActivityNet: A large-Scale Video Benchmark for Human Activity Understanding.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 961–970, 2015.
  * Carreira and Zisserman [2017] Joao Carreira and Andrew Zisserman.  Quo vadis, Action Recognition? a New Model and the Kinetics Dataset.  In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6299–6308, 2017.
  * Chang et al. [2021] Shuning Chang, Pichao Wang, Fan Wang, Hao Li, and Jiashi Feng.  Augmented Transformer with Adaptive Graph for Temporal Action Proposal Generation.  _arXiv preprint arXiv:2103.16024_ , 2021.
  * Chao et al. [2018] Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar.  Rethinking the Faster R-CNN Architecture for Temporal Action Localization.  In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1130–1139, 2018.
  * Dai et al. [2021a] Rui Dai, Srijan Das, and Francois Bremond.  Ctrn: Class-temporal relational network for action detection.  _British Machine Vision Conference_ , 2021a.
  * Dai et al. [2021b] Rui Dai, Srijan Das, Luca Minciullo, Lorenzo Garattoni, Gianpiero Francesca, and Francois Bremond.  PDAN: Pyramid Dilated Attention Network for Action Detection.  In _Proceedings of the IEEE Winter Conference on Applications of Computer Vision_ , pages 2970–2979, 2021b.
  * Dai et al. [2022] Rui Dai, Srijan Das, Kumara Kahatapitiya, Michael S Ryoo, and Francois Bremond.  MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 20041–20051, 2022.
  * Dai et al. [2019] Xiyang Dai, Bharat Singh, Joe Yue-Hei Ng, and Larry Davis.  TAN: Temporal Aggregation Network for Dense Multi-Label Action Recognition.  In _2019 IEEE Winter Conference on Applications of Computer Vision_ , pages 151–160. IEEE, 2019.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.  _International Conference on Learning Representations_ , 2020.
  * Dufter et al. [2022] Philipp Dufter, Martin Schmitt, and Hinrich Schütze.  Position Information in Transformers: An Overview.  _Computational Linguistics_ , 48(3):733–763, 2022\.
  * Feichtenhofer [2020] Christoph Feichtenhofer.  X3D: Expanding Architectures for Efficient Video Recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 203–213, 2020.
  * Huang et al. [2019] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck.  Music Transformer.  _International Conference on Learning Representations_ , 2019.
  * Jiang et al. [2014] Yu-Gang Jiang, Jingen Liu, A Roshan Zamir, George Toderici, Ivan Laptev, Mubarak Shah, and Rahul Sukthankar.  THUMOS Challenge: Action Recognition with a Large Number of Classes, 2014.
  * Kahatapitiya and Ryoo [2021] Kumara Kahatapitiya and Michael S Ryoo.  Coarse-Fine Networks for Temporal Activity Detection in Videos.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8385–8394, 2021.
  * Kay et al. [2017] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The Kinetics Human Action Video Dataset.  _arXiv preprint arXiv:1705.06950_ , 2017.
  * Kingma and Ba [2015] Diederik P Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  _International Conference on Learning Representations_ , 2015.
  * Li et al. [2021] Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio.  Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding.  _Advances in Neural Information Processing Systems_ , 34:15816–15829, 2021.
  * Li and Yao [2021] Zhihui Li and Lina Yao.  Three Birds with One Stone: Multi-Task Temporal Action Detection via Recycling Temporal Annotations.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4751–4760, 2021.
  * Lin et al. [2021] Chuming Lin, Chengming Xu, Donghao Luo, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu.  Learning Salient Boundary Feature for Anchor-Free Temporal Action Localization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3320–3329, 2021.
  * Lin et al. [2018] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang.  BSN: Boundary Sensitive Network for Temporal Action Proposal Generation.  In _Proceedings of the European Conference on Computer Vision_ , pages 3–19, 2018.
  * Lin et al. [2019] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen.  BMN: Boundary-Matching Network for Temporal Action Proposal Generation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 3889–3898, 2019.
  * Liu et al. [2019] Yuan Liu, Lin Ma, Yifeng Zhang, Wei Liu, and Shih-Fu Chang.  Multi-Granularity Generator for Temporal Action Proposal.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3604–3613, 2019.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 10012–10022, 2021.
  * Ma et al. [2020] Fan Ma, Linchao Zhu, Yi Yang, Shengxin Zha, Gourab Kundu, Matt Feiszli, and Zheng Shou.  SF-Net: Single-Frame Supervision for Temporal Action Localization.  In _Proceedings of the European Conference on Computer Vision_ , pages 420–437, 2020.
  * Piergiovanni and Ryoo [2019] AJ Piergiovanni and Michael Ryoo.  Temporal Gaussian Mixture Layer for Videos.  In _International Conference on Machine learning_ , pages 5152–5161. PMLR, 2019.
  * Piergiovanni and Ryoo [2018] AJ Piergiovanni and Michael S Ryoo.  Learning Latent Super-Events to Detect Multiple Activities in Videos.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5304–5313, 2018.
  * Ridnik et al. [2021] Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor.  Asymmetric Loss For Multi-Label Classification.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 82–91. IEEE Computer Society, 2021.
  * Shaw et al. [2018] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani.  Self-Attention with Relative Position Representations.  _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2, 2018.
  * Shen et al. [2018] Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang.  DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Sigurdsson et al. [2016] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.  Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding.  In _Proceedings of the European Conference on Computer Vision_ , pages 510–526, 2016.
  * Tan et al. [2022] Jing Tan, Xiaotong Zhao, Xintian Shi, Bin Kang, and Limin Wang.  PointTAD: Multi-Label Temporal Action Detection with Learnable Query Points.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Tirupattur et al. [2021] Praveen Tirupattur, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah.  Modeling Multi-Label Action Dependencies for Temporal Action Localization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1460–1470, 2021.
  * Vahdani and Tian [2022] Elahe Vahdani and Yingli Tian.  Deep Learning-based Action Detection in Untrimmed Videos: A Survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention Is All You Need.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Wang et al. [2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 568–578, 2021.
  * Xu et al. [2017] Huijuan Xu, Abir Das, and Kate Saenko.  R-C3D: Region Convolutional 3D Network for Temporal Activity Detection.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 5783–5792, 2017.
  * Xu et al. [2020] Mengmeng Xu, Chen Zhao, David S Rojas, Ali Thabet, and Bernard Ghanem.  G-TAD: Sub-Graph Localization for Temporal Action Detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 10156–10165, 2020.
  * Yeung et al. [2018] Serena Yeung, Olga Russakovsky, Ning Jin, Mykhaylo Andriluka, Greg Mori, and Li Fei-Fei.  Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos.  _International Journal of Computer Vision_ , 126(2):375–389, 2018.
  * Zhang et al. [2022] Chenlin Zhang, Jianxin Wu, and Yin Li.  ActionFormer: Localizing Moments of Actions with Transformers.  _Proceedings of the European Conference on Computer Vision_ , 2022\.