# Topological interaction of neutrino with photon
in the magnetic field – Electroweak Hall effect–

Kenzo Ishikawa (1,2) and Yutaka Tobita (3)

###### Abstract

Neutrino interaction with photons induced from a strong magnetic field
approximation in the electroweak theory is presented. Contrary to the weak
filed approximation, the interaction in the magnetized plasma has non-
negligible strength and a topological form expressed by the Chern-Simons form
of the neutrino current and the electromagnetic vector potential. The
derivation of the interaction Lagrangian and its properties are presented.

(1) Department of Physics, Faculty of Science,

Hokkaido University, Sapporo 060-0810, Japan

(2) Natural Science Center, Keio University, Yokohama 223-8521, Japan

(3) Department of Electric Engineering,

Hokkaido University of Science, Sapporo , Japan

##  1 Neutrino electromagnetic interaction

Neutrino is neutral and has no direct electromagnetic coupling in the lowest
order of the interaction. The interaction is induced from higher order
corrections. Neutrino magnetic moments are derived from higher order
corrections of the weak and electromagnetic interactions through virtual
fluctuation of the charged fields. These fields are massive and fluctuations
occur in a short distance region. Accordingly, the correction in the vacuum
has been studied in a weak field approximation, which incorporates short
distance correlations in Lorentz invariant way. Physical quantities at a
finite $B$ are expressed by a series of positive power $B^{n};n=0,1,\cdots$.
The coefficient of $B$ is the magnetic moment of the neutrino, and has been
investigated from broad prospects. The magnetic moments are extremely small in
the vacuum due partly to tiny neutrino mass. These effects derived from the
weak field approximation are almost negligible in the standard theory. That
may be modified generally by matter and external electromagnetic field,
especially in a strong field region.

Particle’s static energy due to electric and magnetic fields is expressed by a
vector potential $eA_{0}$ or $e{\vec{v}}\cdot{\vec{A}}$, where $A_{0}=zE$ and
$(A_{x},A_{y})=\frac{B}{2}(-y,x)$ for constant fields in $z-$ direction. $e$
is the unit of electric charge, $|\vec{v}|$ is at most the light velocity.
These energies are small in short distance, but may become large in a long
distance. That may be modified generally by matter and external
electromagnetic field.

Now, quantum effects in a strong field may be different, especially at a long
distance, since the vector potentials are proportional to the spatial
coordinates. That becomes large at large $x$ and $y$ even for a small $B$, and
coherent quantum phenomena of the system which are not expressed by the weak
field approximation may arise. These would break Lorentz invariance in the
rest system of the given electric or magnetic field, even though the whole
system is invariant under Lorentz transformation. Quantum phenomena specific
to the high magnetic field, such as the quantum Hall effects, have been found
in semiconductor chips of around millimeter length in the magnetic field of
order Tesla and the temperature 1K[1]. These reveal topological phenomena and
are expressed by a strong field approximation. Similar phenomena may arise at
high energy in a system of large size. The size $1$ mm and the energy $1$meV
becomes by multiplying a factor $10^{9}$ to $10^{9}\text{mm}$ and
$10^{9}\text{meV}$, i.e., $10^{6}$ Meters and $1$ MeV at the same magnetic
field. A strong field phenomenon will appear even though the magnetic field is
much lower than the critical field $10^{9}$ Tesla equivalent to the rest
energy of an electron $m_{e}c^{2}$. This situation is possible in the solar
system. We study a phenomenon of the neutrino similar to the quantum Hall
effect, which is represented by the strong field approximation.

Reasons why the matter effect in the neutrino reactions has been considered
small are; 1\. A dominant interaction in matter is the electromagnetic
interaction and the typical energy scale is of atomic binding energy. Beyond
this energy, matter effect will disappear. 2. The effective coupling strength
of the weak interaction, $G_{F}$, has the dimension $\frac{1}{M_{W}^{2}}$ and
the cross section for the typical energy $E$ is proportional to
$G_{F}^{2}E^{2}$. For $E^{2}=2M_{atom}E_{\nu}$, this is much smaller than the
electromagnetic ones. 3. Lorentz invariant quantities in the neutrino decays
are expressed by the neutrino masses. The matter effects may not be negligible
if these are not applied.

An effect due to charged matter in magnetic field can be different and may
become large. 1.If the whole system has a uniform magnetic field, the Lorentz
non-invariant quantity may emerge and show different behavior. 2\. The one
body Hamiltonian of a charged particle, in the strong field approximation, is
equivalent to a harmonic oscillator of no upper bound in the energy. There is
no threshold energy beyond which the effect of the magnetic field disappears,
and a large system of low magnetic field reveals same characteristic property
of a small system of high magnetic field. The specific phenomenon of the
strong magnetic field can appear in large systems even in a weak field not
only in the low energy region but also in the intermediate (or high) energy
region.

We study a system of the electrons, photons, and neutrino in a uniform
magnetic field and a finite electron density in which the quantum coherence is
fully preserved. This condition is satisfied in dilute system where the
charged particle has a large mean free path. The electrons in the uniform
magnetic field are split to Landau levels, and are expressed by the field of
many components. These are classified by a topological invariant [2]. The von-
Neumann representation reflects these properties manifestly, and is convenient
to study the many body quantum effects. Using these, the relevant effective
interaction is derived. The Lorentz non-invariant interaction Lagrangian of
the neutrino with a photon of lowest dimension, which has been overlooked in
the previous calculations, is found. This has topological origins, intriguing
properties, and sizable magnitude. This interaction in the strong field
approximation is very different from the weak field approximation and is
proportional to $\frac{\rho_{e}}{B}$ ,where $\rho_{e}$ is the electron
density. The coupling in the weak field approximation derived from the short
distance fluctuations is proportional to $B$. The coefficient is the magnetic
moment [3, 4], and is extremely small, and is irrelevant to physical phenomena
generally [5, 6, 7, 8] . The present coupling comes from a long distance
fluctuation which reveals topological properties, and is much larger. [9, 10,
11, 12] A coupling strength of this topological interaction of the neutrino
with the photon becomes sizable owing to combined quantum effects of the
magnetic field and finite electron density. This is expressed by the Chern-
Simons form of the neutrino current and the vector potential. This neutrino-
photon interaction is equivalent to that in the quantum Hall effect and is
relevant to a neutrino radiative decay in the long distance.

##  2  Electroweak Hall interaction of the massive neutrino

###  2.1 Induced electroweak Chern-Simons term

Classical motion of a charged particle in the external fields was summarized
in [13]. All the trajectories are periodic in the case $B^{2}>E^{2}$. The
radius $R$ for $B\neq 0,E=0$ is given by $R=\frac{m}{eB}v$. The radius is
proportional to the velocity $v$ and inversely proportional to the magnetic
field, and becomes large for weak field and high velocity. The quantum effects
relevant in these regions are studied by quantizing the system with the
periodic orbits. The particular effects to the magnetic field are obtained
easily. The effect appears only when this condition is met, i.e., in a large
system for weak magnetic field or large velocity. To study an electromagnetic
interaction of a neutrino induced by long distance many-body effects, the
momentum representation for electrons in Landau levels is suitable. The von
Neumann representation is such representation and is applied in the present
work.

The weak interaction is described originally by the charged current
interaction between the neutrino and the electrons, and the neutral current
interaction. Due to the magnetic field, a symmetry on space time inversion is
partly broken.

An induced interaction of the neutrino with the photon caused by a loop
correction of the electrons is studied hereafter. The charged current weak
interaction between the neutrino and the electron and the electromagnetic
interaction is described by Lagrangian density

$\displaystyle\mathcal{L}=\mathcal{L}_{0}+\frac{G_{F}}{\sqrt{2}}\bar{\psi}_{e}(x)\gamma_{\mu}(1-\gamma_{5})\psi_{e}(x)\bar{\psi}_{\nu_{e}}(x)\gamma_{\mu}(1-\gamma_{5})\psi_{\nu_{e}}(x)+ej_{\mu}(A^{\mu}_{\text{ext}}+A^{\mu}),$
$\displaystyle\mathcal{L}_{0}=\sum_{i}\bar{\psi}_{\nu_{i}}(x)(p^{\mu}\gamma_{\mu}(1-\gamma_{5})-m_{\nu_{i}})\psi_{\nu_{i}}(x)+\bar{\psi}_{e}(x)(p^{\mu}\gamma_{\mu}-m_{e})\psi_{e}(x)-\frac{1}{4}F_{\mu\nu}F^{\mu\nu},$
(1)

where $e$ and $G_{F}$ are the electric charge of the electron and Fermi
coupling constant, $A^{\mu}(x)$ and $A^{\mu}_{\text{ext}}(x)$ are the vector
potential and that of expressing the magnetic field and the chemical
potential, and $\psi_{e}(x)$ and $\psi_{\nu_{e}}(x)$ are the electron field
and the electron neutrino field. The neutral current interaction is symmetric
in all flavors and does not contribute to the neutrino radiative transitions
and was ignored in Eq. $(\ref{lagrangian})$.

Integrating the electron fields in the pure QED, the effective Lagrangian in
the vacuum was obtained first by Schwinger with the propertime formulation as,

$\displaystyle
L=-\frac{1}{4}F_{\mu\nu}F^{\mu\nu}+\frac{2\alpha^{2}}{45}\frac{1}{(m_{e}c^{2})^{4}}[({\bf
E}^{2}-{\bf B}^{2})^{2}+7(\bf E\cdot\bf B)^{2}+\cdots]$ (2)

[14]. This is the contribution from the short distance fluctuation and
manifestly Lorentz invariant and is the sum of positive powers of field
strengths. This is equivalent to the one obtained from the interaction
Hamiltonian, $e^{-\epsilon t}H_{int}$ perturbatively. The magnetic moment of
the neutrinos was obtained in the electroweak theory as

$\displaystyle-i\mu_{ij}\bar{\psi}_{\nu_{i}}(x)\sigma_{\mu\nu}\partial^{\nu}\psi_{\nu_{j}}(x)$
(3)
$\displaystyle\mu_{ij}=\frac{eG_{F}}{8{\sqrt{2}}\pi^{2}}(m_{\nu_{i}}+m_{\nu_{j}})\sum_{l=e,\mu,\tau}f(a_{l})U_{li}^{*}U_{lj},$
(4)

where $a_{l}=\frac{m_{l}^{2}}{m_{W}^{2}}\ll
1,f(a_{l})\approx\frac{3}{2}(1-\frac{a_{l}}{2})$ , $U_{li}$ is the lepton
mixing matrices [3, 4] . The magnitudes are smaller than $10^{-23}\mu_{B}$,
where $\mu_{B}=\frac{e\hbar}{2m_{e}}$.

The fluctuation in the vacuum comes from a pair of an electron and a positron.
These have more energy than the twice of the rest mass, $2m_{e}c^{2}$.
Accordingly, the energy of the vacuum and the neutrino’s magnetic moment in
the vacuum are proportional to $\frac{eB}{m_{e}c^{2}}$. $\frac{1}{B}$ term
does not appear .

####  2.1.1 Finite electron density

The system of finite electron density and magnetic field parallel to
$z$-direction is studied. The magnetic field and the chemical potential $\mu$
is described by $(A_{\text{ext}}^{x},A_{\text{ext}}^{y})=\frac{B}{2}(-y,x)$
and $A^{0}_{\text{ext}}(x)=\frac{\mu}{e}$. In this system, the electrons of
the negative energies and of the energy $0\leq E\leq\mu$ are occupied. The
effective Lagrangian from the former ones is the one of the vacuum and was
obtained in [15]. The latter one is studied in the present paper.

The electrons are occupied in the energy $0\leq E\leq\mu$, and there is no
energy gap in these fluctuations. Moreover, the fluctuation extends spatially
up to the infinity generally unless the boundary condition forbids. The
classical circular motion of the radius proportional to $\frac{1}{B}$ may lead
the physical quantities of the infinite system to have $\frac{1}{B}$ term. The
effective Lagrangian may possess a long distance component that comes from
large circular motion. This may break the Lorentz invariance and has different
magnetic field dependence from the vacuum one. We focus on the effect of these
non-relativistic electrons.

In the following we present the case of the low energy electrons for
simplicity. It is straightforward to cover the high energy region by using the
relativistic Landau levels, which are given in the appendix. The free part is

$\displaystyle\mathcal{L}_{0}^{ext}=\int
d^{3}x\psi(x)^{\dagger}\left(i\hbar\frac{\partial}{\partial
t}-\frac{({p_{x}}+e{A}_{x}^{\text{ext}})^{2}}{2m}-\frac{({p_{y}}+e{A}_{y}^{\text{ext}})^{2}}{2m}-\frac{p_{z}^{2}}{2m}\right)\psi(x)$
(5)

The system is not invariant under the space inversion $(x,y)\rightarrow-(x,y)$
due to the external vector potential $\vec{A}^{ext}$ and under the time
inversion $t\rightarrow-t$ due the magnetic field $B$. To elucidate the
interactions of the neutrino with photons, we decompose the electron field in
terms of the free wave in the $z$-direction and those of the Landau levels in
the $(x,y)$ space.

As the electron’s motion is circular around an arbitrary center, it is
transparent to express the electron field with center coordinates $(X,Y)$ and
relative coordinates $(\xi,\eta)$. One-body Hamiltonian in $(x,y)$ space is
described by the relative coordinates $(\xi,\eta)$, and commutes with center
coordinates $(X,Y)$ as,

$\displaystyle\xi=(eB)^{-1}(p_{x}+eA_{x}),\eta=(eB)^{-1}(p_{y}+eA_{y}),~{}~{}X=x-\xi,Y=y-\eta,$
(6) $\displaystyle[\xi,\eta]=-[X,Y]=\frac{a^{2}}{2\pi
i},a=\sqrt{\frac{2\pi\hbar}{eB}},~{}~{}{H^{0}}_{(xy)}=\frac{1}{2}m{\omega_{c}}^{2}(\xi^{2}+\eta^{2}),\omega_{c}=\frac{eB}{m}.$
(7)

The eigenstates of this Hamiltonian and those of the center coordinates,

$\displaystyle{H^{0}}_{(xy)}|l\rangle=E_{l}|l\rangle,E_{l}=\frac{1}{2}\hbar\omega_{c},l=0,1,2\cdots$
(8)
$\displaystyle(X+iY)|\alpha_{mn}\rangle=z_{mn}|\alpha_{mn}\rangle,z_{mn}=(m\omega_{x}+in{\omega_{y}})$
$\displaystyle|\alpha_{mn}\rangle=e^{i\pi(m+n+mn)+\sqrt{\pi}(A^{\dagger}\frac{z_{mn}}{a}-A\frac{z_{mn}^{*}}{a})}|\alpha_{00}\rangle,A=\frac{\sqrt{\pi}}{a}(X+iY),[A,A^{\dagger}]=1.$
(9)

where $\omega_{x},\omega_{y}$ are complex numbers satisfying

$\displaystyle\text{Im}[\omega_{x}^{*}\omega_{y}]=1.$ (10)

As the vector potential becomes infinite at
$\sqrt{x^{2}+y^{2}}\rightarrow\infty$, all the states in $(x,y)$ space are
bound states. The quantum effect of the magnetic field is not cut in the short
and long distance region and appears even in high energy region. The many body
effects due to their fluctuations are studied easily with the momentum states
of the guiding centers

$\displaystyle|\alpha_{\boldsymbol{p}}\rangle=\sum_{mn}e^{ip_{x}m+ip_{y}n}|\alpha_{mn}\rangle,\langle\alpha_{\boldsymbol{p}}|\alpha_{\boldsymbol{p}^{\prime}}\rangle=\alpha({\boldsymbol{p}})(2\pi)^{2}\delta({\boldsymbol{p}}-{\boldsymbol{p}}^{\prime}-2\pi\boldsymbol{N})$
$\displaystyle|\beta_{\boldsymbol{p}}\rangle=\frac{1}{\beta(\boldsymbol{p})}|\alpha_{\boldsymbol{p}}\rangle,\alpha(\boldsymbol{p})=\beta^{*}(\boldsymbol{p})\beta(\boldsymbol{p}),$
(11)

where $\boldsymbol{p}$ is the two dimensional momentum in the magnetic
Brillouin zone, and $\alpha(\boldsymbol{p})$ is expressed by Jacobi $\theta$
-function. This basis, $|l,\beta_{\boldsymbol{p}}\rangle$, the von Neumann
basis, is convenient to investigate the many-body dynamics of the electrons in
the external magnetic field [16, 17].

The $z$-direction is treated with the plane wave of the momentum eigenstate
$|p_{z}\rangle$.

Expanding the electron field with the basis as

$\displaystyle\psi({\vec{x}},t)=\int\frac{dp_{z}}{2\pi}\int_{BZ}\frac{d^{2}p}{(2\pi)^{2}}\sum_{l=0}^{\infty}b_{l}({\boldsymbol{p}},p_{z},t)\langle{\boldsymbol{x}}|l,\beta_{\boldsymbol{p}},p_{z}\rangle,\
E_{l}(p_{z})=E_{l}+\frac{p_{z}^{2}}{2m},$ (12)
$\displaystyle\langle{\vec{x}}|l,\beta_{\boldsymbol{p}},p_{z}\rangle=\langle{\boldsymbol{x}}|l,\beta_{\boldsymbol{p}}\rangle\cdot\langle
z|p_{z}\rangle$

where $E_{l}$ is the energy of Landau level and the BZ stands for the magnetic
Brillouin zone

$\displaystyle p_{i}\leq\frac{\pi}{d},\ d=\sqrt{\frac{2\pi\hbar}{{eB}}}.$ (13)

The action including the fluctuations of the effective electromagnetic field
$\tilde{A}_{\mu}=eA_{\mu}+\frac{G_{F}}{\sqrt{2}}\bar{\nu}(x)\gamma_{\mu}(1-\gamma_{5})\nu(x)$
is

$\displaystyle S=S_{0}+\int
d^{4}x[j_{0}(x){\tilde{A}}_{0}(x)-{\vec{j}}(x)\cdot{\vec{\tilde{A}}}(x)+O({\tilde{A}}_{\mu}^{2})],$
(14) $\displaystyle S_{0}=\int
dt\frac{dp_{z}}{2\pi}\int_{BZ}\frac{d^{2}p}{(2\pi)^{2}}b_{l}^{\dagger}({\bf
p},p_{z}\,)\left[i\hbar\frac{\partial}{\partial
t}-E_{l}(p_{z})\right]b_{l}({\bf p},p_{z})$

where the current operators $j_{0}(x)$ and ${\vec{j}}(x)$ are given below. The
creation and annihilation operators satisfy

$\displaystyle\\{b_{l}({\bf p},p_{z}\,),b_{l^{\prime}}^{\dagger}({\bf
p}^{\prime},p_{z}^{\prime})\\}=\delta_{ll^{\prime}}(2\pi)\delta(p_{z}-p_{z}^{\prime})(2\pi)^{2}({\boldsymbol{p}}-{\boldsymbol{p}}\,^{\prime}-2\pi{\boldsymbol{N}})e^{i\phi({\boldsymbol{p}},{\boldsymbol{N}})},$
(15)
$\displaystyle\phi({\boldsymbol{p}},{\boldsymbol{N}})=\pi(N_{x}+N_{y})-N_{y}p_{x},\
N_{i}:\text{integer}.$

The one body Hamiltonian is diagonal but the density and current in the
$(t,x,y)$ space are expressed as,

$\displaystyle j_{0}({\bf k},k_{z})=\int d^{3}xe^{-i({\bf k}{\bf
x}+k_{z}z)}\psi^{\dagger}(x)\psi(x)=\sum_{l,l^{\prime}}\int\frac{dp_{z}}{2\pi}\int_{BZ}\frac{d^{2}p}{(2\pi)^{2}}{\hat{b}}_{l}^{\dagger}({\bf
p},p_{z})\Gamma_{0}({\bf k})_{l,l^{\prime}}b_{l^{\prime}}({\bf
p}+a\hat{k},p_{z}+k_{z})$ (16) $\displaystyle{\boldsymbol{j}}({\bf
k},k_{z})=\int d^{3}xe^{-i({\bf k}{\bf
x}+k_{z}z)}\psi^{\dagger}(x){\vec{v}}\psi(x)=\sum_{l,l^{\prime}}\int\frac{dp_{z}}{2\pi}\int_{BZ}\frac{d^{2}p}{(2\pi)^{2}}b_{l}^{\dagger}({\bf
p}p_{z}){\vec{\Gamma}({\bf k},k_{z})}_{l,l^{\prime}}b_{l^{\prime}}({\bf
p}+a\hat{k})$ (17)
$W=\left(\begin{array}[]{cc}\text{Re}[{\omega}_{x}]&\text{Im}[{\omega}_{x}]\\\
\text{Re}[{\omega}_{y}]&\text{Im}[{\omega}_{y}]\end{array}\right)$
$\displaystyle\hat{k}_{i}=W_{ij}k_{j}$ (18)

where $\boldsymbol{\zeta}$, ${\boldsymbol{v}}$, and $\hat{\boldsymbol{k}}$ are
the relative coordinate, the velocity operator, and the dual momentum of
${\boldsymbol{k}}$, , and

$\displaystyle\Gamma_{0}({\bf k})_{l,l^{\prime}}=\langle f_{l}|e^{-i{\bf
k}\cdot{\bf\zeta}}|f_{l^{\prime}}\rangle,{\boldsymbol{\Gamma}({\bf
k})}_{l,l^{\prime}}=\langle f_{l}|\frac{1}{2}\\{{\boldsymbol{v}},e^{-i{\bf
k}\cdot{\bf\zeta}}\\}|f_{l^{\prime}}\rangle.$ (19)

For the third component, $v_{z}=\frac{p_{z}}{m}$ is used. The form factor
$\Gamma_{0}(\boldsymbol{k})$ in Eq.$(\ref{form-factor_1})$ is expressed by a
polynomials in $\boldsymbol{k}$. The constant term is proportional to the
electron charge and the linear term is proportional to the dipole moment,
which has non-vanishing matrix element between nearest neighbor Landau levels,
i.e., $l^{\prime}=l\pm 1$.

It is convenient to define new electron operators transformed as

$\displaystyle\hat{b}_{l}(\boldsymbol{p})^{\dagger}=\sum_{l^{\prime}}U_{ll^{\prime}}(\boldsymbol{p})b_{l^{\prime}}(\boldsymbol{p})$
(20)

with the unitary matrix

$\displaystyle U_{ll^{\prime}}(\boldsymbol{p})=\langle
f_{l}|e^{ip\cdot{\hat{\xi}}/a-\frac{i}{4\pi}p_{x}p_{y}}|f_{l^{\prime}}\rangle.$
(21)

These operators satisfy

$\displaystyle\\{{\hat{b}}_{l}({\bf
p},p_{z}\,),{\hat{b}}_{l^{\prime}}^{\dagger}({\bf
p}^{\prime},p_{z}^{\prime})\\}=\delta_{ll^{\prime}}(2\pi)\delta(p_{z}-p_{z}^{\prime})(2\pi)^{2}({\boldsymbol{p}}-{\boldsymbol{p}}\,^{\prime}-2\pi{\boldsymbol{N}})\Lambda_{ll^{\prime}}(\boldsymbol{N}),$
(22)
$\displaystyle{\hat{b}}_{l}({\boldsymbol{p}}+2\pi{\boldsymbol{N}},p_{z})=\Lambda_{ll^{\prime}}(\boldsymbol{N}){\hat{b}}_{l^{\prime}}({\boldsymbol{p}},p_{z})\Lambda_{ll^{\prime}}(\boldsymbol{N})=(-1)^{N_{x}+N_{y}}U_{ll^{\prime}}(2\pi\boldsymbol{N}),$

and lead the simple expression of the charge and current operators

$\displaystyle j_{0}({\bf
k},k_{z})=\sum_{l}\int\frac{dp_{z}}{2\pi}\int_{BZ}\frac{d^{2}p}{(2\pi)^{2}}{\hat{b}}_{l}^{\dagger}({\bf
p},p_{z}){\hat{b}}_{l}({\bf p}+a\hat{k},p_{z}+k_{z})$ (23)
$\displaystyle{\boldsymbol{j}}({\bf
k},k_{z})=\sum_{l,l^{\prime}}\int\frac{dp_{z}}{2\pi}\int_{BZ}\frac{d^{2}p}{(2\pi)^{2}}{\hat{b}}_{l}^{\dagger}({\bf
p},p_{z})\\{\boldsymbol{v}+\frac{a\omega_{c}}{2\pi}(W^{-1}\boldsymbol{p}+\frac{a}{2}\boldsymbol{k})\\}_{l,l^{\prime}}{\hat{b}}_{l^{\prime}}({\bf
p}+a\hat{k}).$ (24)

As the form factor dose not appear in the matrix element of the charge
operator, the commutation relation between the charge density and the electron
operators becomes straightforward.

Figure 1: The diagram of the current correlation function is shown. The real
line represents the electrons and the wave line represents the photon or the
neutrino current. Figure 2: The one particle pole has a positive imaginary
part in the region $E\leq\mu$. The integration path along the real axis is
deformed to the path $E=ip_{4}+\mu$ by the Wick rotation.

The electron’s propagator is defined as a matrix in the magnetic Brillouin
zone and Landau level index,

$\displaystyle\langle
0|{{\hat{b}}_{m}(\boldsymbol{p},p_{z}){\hat{b}}_{n}^{\dagger}(\boldsymbol{p}^{\prime},p_{z}^{\prime})}|0\rangle=(2\pi)^{4}\delta(\boldsymbol{p}-\boldsymbol{p}^{\prime})\delta(p_{z}-p_{z}^{\prime}){\hat{S}}_{mn}(\boldsymbol{p},p_{z}).$
(25)

The vertex part is written as

$\displaystyle\langle
j^{\mu}(q){\hat{b}}_{m}(p){\hat{b}}_{n}(p^{\prime})\rangle=(2\pi)^{4}\delta(p+Q-p^{\prime}){\hat{S}}_{mm^{\prime}}(p,p+Q){\hat{\Gamma}}^{m^{\prime}u^{\prime}}(p,p+Q){\hat{S}}_{n^{\prime}n}(p+Q)$
(26)

where $Q^{\mu}=(q_{0},a\hat{q}_{x},a\hat{q}_{y},q_{z})=t^{\mu}_{\nu}q^{\nu}$
The vertex part satisfied the magnetic Ward-Takahashi identity,

$\displaystyle{\hat{\Gamma}}_{\mu}(p,p)=t_{\mu}^{\nu}\frac{\partial{\hat{S}}^{-1}(p)}{\partial
p^{\nu}}.$ (27)

The effective potential for the neutrino and photon is obtained by applying
the derivative expansion on the fields $\tilde{A}_{\mu}(x)$ on integrating
over the electrons as

$\displaystyle S_{eff}=\frac{1}{2}\int
d^{4}xd^{4}y\tilde{A}_{\mu}(x)\pi^{\mu\nu}(x-y)\tilde{A}_{\nu}(y),$ (28)

where the current correlation function $\pi^{\mu\nu}(x-y)$ is expressed in
Fig.1 and its Fourier transform is given as

$\displaystyle\pi_{\mu\nu}({\vec{k}})=\int\frac{d^{4}p}{(2\pi)^{4}}\text{Tr}[{\hat{\Gamma}}_{\mu}{\hat{S}}(p+k){\hat{\Gamma}}_{\nu}{\hat{S}}(p)].$
(29)

Expansion coefficients of $\pi_{\mu\nu}({\vec{k}})$ in a power series on
${\vec{k}}$, gives the strengths of the effective interaction. Among the
coefficients, the linear term is most important in the long distance region,
and its coefficient, i.e., the slope at the origin, is finite now for the
space $(\mu,\nu,\rho=0,x,y)$ due to the inversion symmetry breaking. This is
expressed as a topological number of the propagator, using the Ward-Takahashi
identity ${\hat{\Gamma}}_{\mu}(p,p)=\frac{\partial{\hat{S}}^{-1}(p)}{\partial
p_{\mu}}$ as

$\displaystyle\frac{\partial}{\partial
k_{\rho}}\pi_{\mu\nu}(k)|_{k=0}=\epsilon_{\mu\nu\rho}\int\frac{dp_{z}}{2\pi}\frac{1}{2\pi}N_{w}(p_{z}),$
(30) $\displaystyle
N_{w}(p_{z})=\frac{\epsilon^{\mu\nu\rho}}{3!}\int\frac{d^{3}p}{(2\pi)^{3}}\text{Tr}\left[{\hat{S}}(p,p_{z})\frac{\partial{\hat{S}}^{-1}(p,p_{z})}{\partial
p_{\mu}}{\hat{S}}(p,p_{z})\frac{\partial{\hat{S}}^{-1}(p,p_{z})}{\partial
p_{\nu}}{\hat{S}}(p,p_{z})\frac{\partial{\hat{S}}^{-1}(p,p_{z})}{\partial
p_{\rho}}\right],$

where $N_{w}$ is the winding number of the propagator which agrees with the
filling factor in 2-dimensional space obtained by dividing $\rho(p_{z})$ with
the density of the Landau levels $\rho_{0}=\frac{eB}{2\pi\hbar}$. This form is
equivalent to the quantum Hall effect [19]. Fig.2 shows the poles of the
propagator $S(p)$.

The effective action $(\ref{effective-action})$ is decomposed to the lowest
dimensional term and higher dimensional terms,

$\displaystyle S_{eff}=S_{\text{CS}}+S_{high},$ (31) $\displaystyle
S_{\text{CS}}=\frac{\sigma_{xy}^{(4)}}{2}\int dz\int
d^{3}x\epsilon^{\mu\nu\rho}{\tilde{A}}_{\mu}\partial_{\nu}{\tilde{A}}_{\rho},$
(32) $\displaystyle\sigma_{xy}^{(4)}=\frac{1}{2\pi}\nu^{(4)},\
\nu^{(4)}=\int\frac{dp_{z}}{2\pi}\nu(p_{z})=\frac{2\pi n_{e}}{eB},$ (33)

where $S_{high}$ is the higher dimensional term, $n_{e}$ is the electron
density, in the natural unit of $c=\hbar=1$. The Lagrangian in $S_{CS}$ is
proportional to the momentum and gauge invariant, but not invariant under
three dimensional rotations and inversions. This gives a dominant contribution
in the long distance region.

###  2.2 Anomalous neutrino-photon interaction

The effective Lagrangian in $S_{CS}$, the Chern-Simons (CS) term [16, 17, 19],
is

$\displaystyle\mathcal{L}_{\text{int}}=\frac{1}{2}\frac{\nu^{(4)}}{2\pi}\epsilon^{\alpha\beta\gamma}\tilde{A}_{\alpha}\partial_{\beta}\tilde{A}_{\gamma}+O({\tilde{F}_{\alpha\beta}}^{2});\
\alpha,\beta,\gamma=(0,1,2),$ (34)

where the magnetic field is along the 3rd axis. The vector potential includes
the electromagnetic potential and the neutrino current

$\displaystyle\tilde{A}_{\alpha}=eA_{\alpha}(x)+\frac{G_{F}}{\sqrt{2}}\bar{\psi}_{\nu_{e}}(x)\gamma_{\alpha}(1-\gamma_{5})\psi_{\nu_{e}}(x),$
(35)

and
$\tilde{F}_{\alpha\beta}=\partial_{\alpha}\tilde{A}_{\beta}-\partial_{\beta}\tilde{A}_{\alpha}$.
Due to this interaction term, the Maxwell equation is modified and the
electron neutrino interacts with the photon. The electromagnetic system in the
$2+1$ dimensional space-time is that of topologically massive gauge theory
[20]. The electron neutrino is the superposition of the mass eigenstaes
$\psi_{\nu_{e}}(x)=\sum_{i}U_{ei}\psi_{\nu_{i}}(x)$, with mass eigenstaes
$\psi_{\nu_{i}}(x);\ i=1-3$ and a mixing matrix $U$. The neutrino current is
written as

$\displaystyle
J_{\alpha}(x)=g_{ij}\bar{\psi}_{\nu_{i}}(x)\gamma_{\alpha}(1-\gamma_{5})\psi_{\nu_{j}},$
(36) $\displaystyle g_{ij}=U_{ei}^{*}U_{ej}.$

The neutrino-photon interaction is described by the action

$\displaystyle S_{\nu\gamma}=G_{\nu,\gamma}\int dz\int
d^{3}\epsilon^{\mu\nu\rho}J_{\mu}(x)\partial_{\nu}A_{\rho},$ (37)
$\displaystyle G_{\nu,\gamma}=\frac{eG_{F}}{\sqrt{2}}\frac{\nu^{(4)}}{2\pi}.$

The coefficient $G_{\nu,\gamma}$, similarly to the normal Hall effect and the
quantum Hall effect in two-dimensional semi-conductors [1, 19], is a
topological invariant satisfying a low energy theorem, and remains the same in
systems of disorders at finite-temperature. The filling-factor $\nu^{(4)}$ is
written in a manifestly Lorentz invariant expression

$\displaystyle\nu^{(4)}=\frac{j^{0}}{B}\rightarrow\frac{j_{\mu}n^{\mu}}{\sqrt{F_{\mu\nu}F^{\mu\nu}}},$
(38)

where $n^{\mu}=(1,0,0,0)$ in the rest system.

###  2.3 Physical effects of $S_{\text{CS}}$

####  2.3.1 Electromagnetic waves

Farady rotation, topological mass in magnetic plasma, ionosphere reflection,
$\cdots$

The action $S_{\text{CS}}$ is written with the vector potential and gauge
invariant. Maxwell equation is modified by,

$\displaystyle\frac{\partial S_{CS}}{\partial
A_{\mu}(x)}=\sigma_{xy}^{(4)}\epsilon_{\mu\nu\rho}\partial_{\nu}A_{\rho}.$
(39)

This term causes parity violating shift of the polarization direction of the
photon, Farady rotation. In the quantum Hall system, such phenomenon has been
studied[21, 22].

####  2.3.2  Neutrino decays

The action $S_{\text{CS}}$ is linear in the momentum and is the lowest
dimensional term. Other terms of higher power in the momentum are suppressed
in the small momentum and in the long distance region. The low energy
phenomena are studied with $S_{\text{CS}}$. In the neutrino radiative decays,
Lorentz-invariant combinations of the momenta are converted to the products of
masses. As neutrinos have tiny masses, these terms are also suppressed even in
high energy region. In the process

$\displaystyle\nu_{i}({p}_{\nu_{i}})\rightarrow\nu_{f}({p}_{\nu_{f}})+\gamma(p_{\gamma}),$
(40)

three Lorentz invariant quantities , $p_{\nu_{i}}\cdot
p_{\nu_{f}},p_{\nu_{i}}\cdot p_{\gamma},p_{\nu_{f}}\cdot p_{\gamma}$, are not
independent variables. From the energy momentum conservation

$\displaystyle p_{\nu_{i}}=p_{\nu_{f}}+p_{\gamma},$ (41)

and the mass shell conditions, these are expressed as

$\displaystyle p_{\nu_{i}}\cdot
p_{\gamma}=\frac{1}{2}(m_{\nu_{i}}^{2}-m_{\nu_{f}}^{2}+m_{\gamma}^{2})$ (42)
$\displaystyle p_{\nu_{f}}\cdot
p_{\gamma}=\frac{1}{2}(m_{\nu_{i}}^{2}-m_{\gamma}^{2}-m_{\nu_{f}}^{2})$
$\displaystyle p_{\nu_{i}}\cdot
p_{\nu_{f}}=\frac{1}{2}(-m_{\gamma}^{2}+m_{\nu_{i}}^{2}+m_{\nu_{f}}^{2}),$
(43)

where $m_{\gamma}$ is the effective mass of the photon in the matter. The
Lorentz invariant bi-linear forms of the momenta are of order $m_{\nu}^{2}$
and $m_{\gamma}^{2}$ regardless of their energies. Even for the energy
$E_{\nu_{i}}=1$ MeV, the square of the neutrino mass squared are
$\text{meV}^{2}$, which is $10^{-18}$ times smaller. Accordingly, higher power
terms of possessing Lorentz invariance are neglected. $S_{\text{CS}}$ is most
relevant in high energy region also.

The transitions of a massive neutrino to another neutrino and a photon is
derived from the $S_{\nu,\gamma}$.

####  2.3.3  Comparison: $S_{\text{CS}}$ vs the magnetic moment

Lorentz invariant vs non-invariant, coupling strength, helicity properties,
vacuum or in matter, wave packet

The transition moment of the neutrino is described by Eq. $(\ref{transition
moment})$ where $\mu_{\nu}$ is due to the short distance fluctuation. This is
Lorentz invariant and the strength is extremely small. By this term, the
neutrino loses the energy in the vacuum, and also in star and other systems.
The current experimental upper bound

$\displaystyle\mu_{\nu}\leq 0.022\times 10^{-10}\mu_{B},$ (44)

was derived from observations of the tip of the red giant in the globular
cluster $\omega$-Centauri [23]. $S_{\text{CS}}$ is relevant in the dilute
plasma in uniform magnetic field such as the solar corona and the earth
ionosphere, but is not so in the inside of stars. That is not constrained from
existing ground neutrino experiments also from the same reason.

$S_{\text{CS}}$ is not Lorentz invariant and reveals different properties from
the magnetic moment. The magnetic moment is Lorentz invariant and appears in
the vacuum and in the matter, whereas $G_{\nu\gamma}$ appears in the system of
the conduction electrons and the uniform magnetic field, but not in the
vacuum. The coupling strength depends on both of the magnetic field and the
electron density. The magnitude is proportional to $\frac{\rho_{e}}{B}$, which
is not extremely weak even in the weak magnetic field if the electron
coherence and uniformity of the magnetic field are kept in large area.

####  2.3.4  wave packet decay : comparison of $P^{(d)}$ with $\Gamma$

The wave of the finite coherence length is not a stationary state. A
transition of non-stationary states is neither described by the stationary
states nor by the plane waves. These are described by the wave packets, which
are characterized by a wave length $l_{w}$ and a coherence length $l_{c}$
generally. $l_{w}$ is the de Bloglie wave length $\frac{h}{p}=\frac{hc}{E}$
for the plane wave of momentum $p$ and the energy $E=pc$. The overlapping area
of the initial and final states is also stationary for plane waves, but for
moving wave packets the overlapping area is finite and moves. The integration
over the time is made in the moving area and results to a modified angular
velocity. The angular velocity shifts by the velocity of the interaction area
from the normal energy difference to $\omega=\Delta
E-{\boldsymbol{V}}^{0}\cdot\Delta{\boldsymbol{P}}$, where $\Delta E$ is the
energy difference, $\boldsymbol{V}_{0}$ is the velocity of the interaction
area, and $\Delta\boldsymbol{P}$ is the difference of the momentum. For a
particle of tiny mass, $m_{l}$,
$E_{l}-{\boldsymbol{V}_{l}}\cdot{\boldsymbol{p}}_{l}=\frac{m_{l}^{2}}{E_{l}}$
and is much smaller than $E_{l}$. Accordingly, the low energy electron states
in the intermediate states are able to contribute to the transition of much
higher energy.

(1) $l_{ph}=\frac{h}{p}$

It has been considered that de Bloglie wave length is the typical length
$l_{ph}=l_{w}$ in the scattering processes. The scattering cross section is
governed by the length, $l_{w}=\frac{\hbar}{p}$, in fact. This is of the
atomic length for $p\geq 1$ keV, whereas the magnetic length
$R_{mag}=\frac{mv}{eB}$ , which is the size that the electrons is trapped to
the magnetic flux, for $B=10^{-4}$ Tesla and $1$ keV is $R\approx 0.1$ M , and
is much longer than the atomic distance. This does not affect the transition
amplitude and probability in the short distance region. The rates in this
situation are the same as those in the vacuum, and are not modified by the
topological interaction Eq.$(\ref{chern-simon})$. As far as the phenomena
caused by the rate are concerned, the topological interaction can be ignored.

(2) $l_{ph}\gg\frac{h}{p}$

A realistic wave is not plane but a wave packet. A wave packet is normalized
and has a finite extension. Their physical quantities are expressed by the
transition probability in a finite time interval $T$, $P(T)$. $P(T)$ is
decomposed to $P(T)=\Gamma T+P^{(d)}$, where $\Gamma$ is characterized by de
Bloglie wave length, and $P^{(d)}$ is characterized by the coherent length
$l_{c}$ of the wave function. The wave packets propagate with a group velocity
$v_{g}=\frac{\partial E(p)}{\partial p}$ and vary the phase with the phase
velocity $v_{phase}=\frac{E}{p}$. The difference of two velocities determines
a coherence length of the wave packet,
$l_{c}=c\frac{h}{p(v_{g}-v_{phase})}=ch\frac{E}{m^{2}c^{4}}$ for the particle
of the mass $m$. For the light particles such as the neutrinos, the difference
of the two velocities is extremely small. This length is much longer than the
de Bloglie wave length.

The effects specific to these states appear in the physical length satisfying
$l_{ph}\gg R_{mag}$ and are described by the action $S_{\text{CS}}$, but these
disappear in the short distance of $l_{ph}\ll R_{mag}$. $S_{\text{CS}}$ is the
action of the lowest dimension reveals the phenomena of physical length
$l_{ph}\gg R_{mag}$.

The long range property of $P^{(d)}$ is similar to the one of the EPR
correlation. The coherence length $l_{c}=\frac{c\hbar E}{m^{2}c^{4}}$ for the
neutrino, the mass is the order meV and the energy is the order MeV, can
become huge of order $10^{5}$ km for $p\geq 1$ MeV for instance. In previous
papers [10, 11, 12] it was shown that an angular frequency of the interference
term of two nearby masses are determined by the difference $\frac{\delta
m^{2}c^{2}}{p}$. For the mass range of meV, and the momentum of MeV, this
length satisfied $l_{ph}\gg R_{mag}$ and the system is governed by the
topological interaction Eq.$(\ref{chern-simon})$.

The interaction is applicable to relativistic electrons and in the system of a
finite temperature as well. One place where the neutrino, the radiation, and
the electrons in the magnetic field interact is the solar corona. In the quiet
corona, $B=10^{-4}\text{ T}$ and $n_{e}=10^{11}\text{ m}^{-3}$, the number of
the cyclotron oscillation within a mean free path is,
$\omega_{B}\times\frac{l_{\text{mfp}}^{e}}{v}\gg 1$, where
$\omega_{B}=\frac{eB}{m_{e}}$, $l_{\text{mfp}}^{e}$ and $v$ are the mean free
path and the velocity of the electrons. Thus the electron motion is determined
by the electrons in the external magnetic field, and the long distance
fluctuation is described by the eigenmodes in the external magnetic field. The
sum of the all terms of $B^{n}$ from $n=0$ to $n=\infty$ including the term
proportional to $B$ [18] is made. We take the unit $c=\hbar=1$, but write $c$
and $\hbar$ explicity when that is better.

##  3 Summary and prospects

The effective interaction between the neutrinos and photon in the dilute
plasma in the uniform magnetic field is obtained. The interaction Lagrangian
has the topological form and the coupling strength is proportional to
$eG_{F}$. In the rest system of the magnetic field, that depends on the
direction of the magnetic field and breaks manifest Lorentz invariance. The
interaction causes the radiative transition of the neutrino in this specific
system. Although the transition rate given from the Fermi’s golden rule is
expressed by the masses alone, the correction to the golden rule expressed by
$P^{(d)}$ is expressed differently and becomes sizable. This leads the
intriguing quantum phenomenon in the uniform magnetic field at large distance
regions. The physical effect in the solar corona and the earth ionosphere will
be presented in a next paper [24].

### Acknowledgments

This work was partially supported by a Grant-in-Aid for Scientific Research (
Grant Number JP21H01107).

##  Appendix A  Landau levels of Dirac equation

$\displaystyle(i\frac{\partial}{\partial
t}-m)\phi=\boldsymbol{\sigma}\cdot(\boldsymbol{p}+e\boldsymbol{A})\xi,$ (45)
$\displaystyle(i\frac{\partial}{\partial
t}+m)\xi=\boldsymbol{\sigma}\cdot(\boldsymbol{p}+e\boldsymbol{A})\phi,$

The stationary solutions $\phi=e^{iEt}\phi,\xi=e^{iEt}\xi$

$\displaystyle(E-m)\phi=\boldsymbol{\sigma}\cdot(\boldsymbol{p}+e\boldsymbol{A})\xi,$
(46)
$\displaystyle(E+m)\xi=\boldsymbol{\sigma}\cdot(\boldsymbol{p}+e\boldsymbol{A})\phi,$
$\displaystyle(E^{2}-m^{2})\phi=[{\boldsymbol{p}}^{2}+e^{2}{\boldsymbol{A}}^{2}+2e\boldsymbol{A}\cdot\boldsymbol{p}+e\boldsymbol{\sigma}\cdot\boldsymbol{H}]\phi$
(47) $\displaystyle E=\pm\sqrt{m^{2}+p_{z}^{2}+(2l+1+\sigma)eB}$

## References

  * [1] K. V. Klitzing, G. Dorda, and M. Pepper, Phys. Rev. Lett. 45, 494, (1980).
  * [2] K. Ishikawa and T. Matsuyama, Zeitschrift. fur Physics,C,33,41-45(1986)
  * [3] G. G. Raffelt, Physics. Reports., 320, 319-327, (1999).
  * [4] K. Fujikawa, and R. E. Shrock, Phys. Rev. Lett., 45, 963, (1980).
  * [5] L. Landau, Sov. Phys. Doclady 60, 207, (1948).
  * [6] C. N. Yang, Phys. Rev. 77. 242, (1950).
  * [7] M. Gell-Mann, Phys. Rev. Lett. 6, 70, (1961).
  * [8] P.A. Zyla et al. [Particle Data Group], Prog.Theor.Exp.Phys, 083c01, 10.1093/ptep/ptaa104 (2020).
  * [9] A. Einstein, B.Podlsky, and N. Rosen, Phys.Rev.,47, 777 (1935)
  * [10] K. Ishikawa and Y. Tobita, Prog. Theor. Exp. Phys. 073B02, doi:10.1093/ptep/ptt049 (2013).
  * [11] K. Ishikawa and Y. Tobita, Ann. Phys.. 344, 118-178 (2014).
  * [12] K. Ishikawa,T.Tajima, and Y. Tobita, Prog. Theor. Exp. Phys. 013B02, doi:10.1093/ptep/ptu168 (2015).
  * [13] A.H.Taub, Phys. Rev. 73. 786, (1948).
  * [14] J. Schwinger, Phys. Rev. 82. 664, (1951).
  * [15] A.Erdas Aand G. Feldman, Nuclear Physics. B343. 597, (1990).
  * [16] K. Ishikawa, N. Maeda, T. Ochiai, and H. Suzuki, Physica. 4E, 37(1999).
  * [17] K. Ishikawa, T. Aoyama, Y. Ishizuka, and N. Maeda, Int.Jour.Mod. Physics. B17, 4765(2003).
  * [18] K. Battacharya and P. B. Pal; arxiv: hep-ph/021211.
  * [19] K. Ishikawa, Phys. Rev. Lett.53, 1615, (1984); N. Imai, K. Ishikawa, T. Matsuyama, and I. Tanaka, Phys. Rev. B42, 10610, (1990).
  * [20] S. Deser, R. Jackiw, and S.Templeton, Annals of Physics, (140), 372-411,(1982).
  * [21] K. Ishikawa, Phys. Rev.D .31, 1432, (1985).
  * [22] R. Shimano, et al., Nature Communications 4,1841 (2013).
  * [23] S. Arceo-Diaz et al., ASP 70 1
  * [24] K. Ishikawa and Y.Tobita, in preparation . See also arXiv::1503.07285[hep-ph]


#  Assessing Cross-dataset Generalization of Pedestrian Crossing Predictors

Joseph Gesnouin1,2, Steve Pechberti1, Bogdan Stanciulescu2 and Fabien
Moutarde2
1 Institut VEDECOM, 78000 Versailles, France 2 Centre de Robotique, MINES
ParisTech, Université PSL, 75006 Paris, France

###### Abstract

Pedestrian crossing prediction has been a topic of active research, resulting
in many new algorithmic solutions. While measuring the overall progress of
those solutions over time tends to be more and more established due to the new
publicly available benchmark and standardized evaluation procedures, knowing
how well existing predictors react to unseen data remains an unanswered
question. This evaluation is imperative as serviceable crossing behavior
predictors should be set to work in various scenarii without compromising
pedestrian safety due to misprediction. To this end, we conduct a study based
on direct cross-dataset evaluation. Our experiments show that current state-
of-the-art pedestrian behavior predictors generalize poorly in cross-dataset
evaluation scenarii, regardless of their robustness during a direct training-
test set evaluation setting. In the light of what we observe, we argue that
the future of pedestrian crossing prediction, e.g. reliable and generalizable
implementations, should not be about tailoring models, trained with very
little available data, and tested in a classical train-test scenario with the
will to infer anything about their behavior in real life. It should be about
evaluating models in a cross-dataset setting while considering their
uncertainty estimates under domain shift.

###### Index Terms:

Pedestrian Intention Prediction, Uncertainty Estimation, Cross-dataset
Evaluation

##  I Introduction

The topic of pedestrian discrete behavior prediction is deemed essential for
robust and reliable planning leading to the deployment of autonomous vehicles.
While the domain has attracted significant interest in computer vision and
robotics communities for the past decade, the field of research has suffered
for a long time from the lack of common evaluation protocols and standardized
benchmarks, making the task of comparing performance between approaches
complex if not impossible to achieve. To compensate for such problems, a
standardized benchmark [1] to evaluate pedestrian behavior prediction for
three datasets was recently proposed to advance research further. While this
brought a breath of fresh air to the field of pedestrian behavior prediction,
we believe that current evaluation protocols do not adequately represent the
applicability of existing pedestrian prediction models for real-world
scenarii. Comparable studies have previously been conducted in computer
vision, questioning whether recent progress on the ImageNet [2] benchmark
continues to represent meaningful generalization [3] and identifying various
sources of bias and noise [4, 5]. However, going beyond accuracy to evaluate a
model for a high-risk application with limited amount of training data, such
as pedestrian crossing prediction, has never been properly investigated. In
this work, we assess how pedestrian intention prediction approaches react to
small domain shifts and evaluate their generalization capability outside a
standard train-test evaluation protocol. We show that all the current
pedestrian behavior predictors show signs of over-fitting when evaluated
during a direct training-test sets evaluation setting on those standardized
benchmarks.

This problem leads to two major drawbacks for the field:

  * •

The training source being generally not dense in variety of scenarii nor in
the number of examples, the results of state-of-the-art approaches on each
dataset might just come from noise: this noise effect should probably be
further aggravated since the existing approaches are based on deep learning,
depending heavily on the quantity and quality of data where the performance of
approaches scales up with the amount of training data.

  * •

It prevents pedestrian behavior predictors from scaling up to real-world
applications, as they are not applicable in various scenarii with small
domains shifts.

Figure 1: Examples of crossing and non-crossing pedestrians from $JAAD$ and
$PIE$ datasets. The conditions under which pedestrians act from one scenario
to another can differ drastically concerning input format and domain shift:
pedestrian size, pedestrian positioning in the scene, illumination conditions,
occlusion…

Figure 2: Pedestrian crossing prediction performance for $PIE$,
$JAAD_{behavior}$ and $JAAD_{all}$. We show a comparison between traditional
single-dataset train and test evaluation on each dataset compared to cross-
dataset evaluation for eleven methods representing the diversity of
architectures and modalities usually used for pedestrian crossing prediction.
Ensembling denotes the average prediction given by the three models trained on
each dataset for one given test set.

The above examples recap the general motivation of this work, encouraging us
to rethink the evaluation methodology to rank current top-scoring behavior
predictors from the perspective of uncertainty evaluation to small domain
shifts. We argue that:

  * •

The only empirical evaluation of models in a direct train-test sets evaluation
offered by the original work introducing the method is not sufficient to
effectively conclude anything about its applicability in a real-world
scenario. The result is often statistically non-significant during a cross-
dataset evaluation scenario and leads to an ever-changing state-of-the-art.

  * •

It would be more interesting to compare each method by evaluating how
trustworthy are their uncertainty estimates under different domain shifts.

##  II Related Work

###  II-A Pedestrian Crossing Prediction

Pedestrian crossing prediction formulates the prediction task as a binary
classification problem where the objective is to determine if a pedestrian $i$
will start crossing the street given the context observed up to some time $t$.
The prediction can rely on multiple sources of information, including visual
features of the pedestrians and their surroundings, pedestrian kinematics,
spatial positioning of the pedestrian based on 2D bounding box locations,
optical flow and ego-vehicle speed. Early works [6, 7], formulated the problem
as a static image classification problem with 2D Convolutions [8, 9], using
only the last frame in the observation sequence to predict crossing behaviors.
More successful approaches were designed to take into account temporal
coherence in short-term motions of visual features of the pedestrians by using
ConvLSTMs [10, 11], 3D Convolutions [12, 13, 14], or Spatio-Temporal DenseNet
[15]. Approaches trying to minimize the inference time of their models by
avoiding the usage of RGB images were explored: [16] proposes a transformer
using only spatial positioning of the pedestrian based on 2D bounding box
locations. Crossing prediction based on kinematics only was also explored with
various available learning architectures to monitor temporal evolution of
skeletal joints such as convolutions [17, 18, 19], recurrent cells [20, 21] or
graph-based models [22]. More recently, approaches combining multiple sources
of information emerged. Those approaches usually differ by the way they merge
the available sources, e.g. scenes, trajectories, poses and ego-vehicle speed,
and the learning architecture used to infer a crossing prediction, e.g. RNN-
based models [23, 24, 25, 1, 26] or Transformer-based models [27, 28].

###  II-B Cross-dataset evaluation

In its first year of existence, proposed approaches evaluated on the
benchmarks [1] constantly report higher classification scores [19, 16, 26, 29,
30, 31], giving the impression of clear improvements in pedestrian intention
prediction. Usually, a new algorithm is proposed and the implicit hypothesis
towards the proposed contribution is made such that it yields an improved
performance over the existing state-of-the-art. To confirm such hypothesis, an
empirical evaluation of the given contribution is realized in a direct train-
test sets evaluation and the quality of the model is evaluated by regular
classification metrics: newly proposed methods are then claimed as the new
state-of-the-art as soon as they outperform previous ones even by a small
margin. However, the ranking of the methods for a given task is currently only
as good as the quality of the data used for comparison purposes, and the
results obtained by one method on a given dataset do not always reflect its
robustness in real-world applications.

In this work, we evaluate how pedestrian intention prediction approaches react
to small domain shifts by interchanging the training set of dataset $A$ by the
training set of dataset $B$ and test it on the testing set of $A$. The given
training routine is consistent across all experiments for all three datasets.
This is referred throughout the paper as cross-dataset evaluation [32, 33,
34]. By adopting cross-dataset evaluation, we test the generalization
abilities of several state-of-the-art pedestrian crossing predictors to
distributional shift such as pedestrian size, as shown in Fig 3, pedestrian
positioning in the scene, illumination conditions or occlusion as shown in Fig
1.

Figure 3: Distribution of pedestrian bounding box height in pixel for $PIE$,
$JAAD_{behavior}$ and $JAAD_{all}$.

###  II-C Uncertainty Metrics

In real-world scenarii, quantifying uncertainty is crucial as the input
distributions are frequently shifted from the training distribution due to a
number of causes such as sampling bias. Evaluating the generalization
abilities of models by using cross-dataset evaluation and classification
metrics only is not sufficient. In high-risk applications such as pedestrian
behavior prediction, the idea that a model’s predicted probabilities of
outcomes reflect true probabilities of those outcomes is mandatory for high-
level decisions (i.e., vehicle planning module in crowded urban traffic
environments). Expected Calibration Error (ECE) and Maximum Calibration Error
(MCE) are standard uncertainty111Because confidence is the additive inverse of
uncertainty with regard to 1, the terms are often interchanged. metrics in
this context [35, 36, 37, 38]. Predictions are divided into $M$ interval bins,
we then calculate the accuracy of each bin to estimate the predicted accuracy
from finite data. Let $B_{m}$ denote the set of sample indices for which
prediction confidence is inside one interval bin. The accuracy of $B_{m}$ is
defined as

$\operatorname{acc}\left(B_{m}\right)=\frac{1}{\left|B_{m}\right|}\sum_{i\in
B_{m}}\mathbf{1}\left(\hat{y}_{i}=y_{i}\right)$ (1)

where $\hat{y}_{i}$ and $y_{i}$ are respectively the predicted and true class
labels for sample $i$. The average confidence within one interval bin $B_{m}$
is defined as:

$\operatorname{conf}\left(B_{m}\right)=\frac{1}{\left|B_{m}\right|}\sum_{i\in
B_{m}}\hat{p}_{i}$ (2)

where $\hat{p}_{i}$ is the model confidence for sample $i$. Throughout our
experiments, the maximum softmax probability [39] is used as the confidence
score. We therefore compare each model output pseudo-probabilities to its
accuracy. We obtain the following metrics to rank methods based on their
calibration:

Expected Calibration Error (ECE): takes a weighted average of the absolute
difference in accuracy and confidence.

$ECE=\sum_{m=1}^{M}\frac{\left|B_{m}\right|}{n}\left|\operatorname{acc}\left(B_{m}\right)-\operatorname{conf}\left(B_{m}\right)\right|$
(3)

Maximum Calibration Error (MCE): measures the maximum discrepancy between
accuracy and confidence.

$MCE=\max_{m}\left|\operatorname{acc}\left(B_{m}\right)-\operatorname{conf}\left(B_{m}\right)\right|$
(4)

Since the underlying binning approach has a significant impact on the accuracy
and reliability of ECE and MCE, we use an adaptive binning strategy [40]
instead of a uniform partition222https://github.com/yding5/AdaptiveBinning.

##  III Generalization Capabilities

###  III-A Datasets and Implementation Details

For this evaluation, we use two large public naturalistic datasets for
studying pedestrian behavior prediction: $JAAD$ [41] and $PIE$ [42]. These
datasets are typically obtained by a vehicle-mounted camera as it navigates
through crowded urban traffic environments: $JAAD$ contains 346 clips and
focuses on pedestrians intending to cross, $PIE$ contains 6 hours of
continuous footage and provides annotations for all pedestrians sufficiently
close to the road regardless of their intent to cross in front of the ego-
vehicle and provides more diverse behaviors of pedestrians. The $JAAD$ dataset
is split into $JAAD_{behavior}$ and $JAAD_{all}$. $JAAD_{behavior}$ is biased
towards pedestrians attempting to cross the street (402 crossing out of 648)
and the smallest dataset available. $JAAD_{all}$ adds all visible pedestrians
in $JAAD$, regardless of their position in the scene and contains more non-
crossing pedestrians (490 crossing out of 2580). Similarly, $PIE$ contains
more non-crossing pedestrians (512 crossing out of 1842). All three datasets
are heavily skewed towards one class. To compensate for such significant
datasets shifts label-wise, we train all our models using class weights
inversely proportional to the percentage of samples for each class. Following
the existing evaluation procedures [1], we use the same data sampling method,
the same splits and the same inputs sets for our
experiments333https://github.com/ykotseruba/PedestrianActionBenchmark.
However, we disregard the ego-vehicle speed input for all our models as the
sensor data used for the ego-vehicle speed is only available for $PIE$ and
could not be used for cross-dataset evaluation purposes. The observation
length for all models is fixed at 16 frames. The sample overlap is set to 0.8
for both $PIE$ and $JAAD$. We report the results using standard binary
classification metrics: AUC and F1 Score and standard confidence calibration
metrics: adaptive ECE and MCE.

###  III-B Baselines and state-of-the-art models

We select a subset of methods from the pedestrian crossing prediction
literature, and more broadly, action recognition literature for their
prevalence, practical applicability and diversity in terms of architectures
and input modalities. These include:

  * •

VGG16 [8] and Resnet50 [9] : two baseline static models that use only the last
frame in the observation sequence to predict the crossing behavior of a
pedestrian.

Figure 4: Distribution of the performance of the eleven selected approaches
when evaluated in a direct train-test scenario and when evaluated under cross-
dataset scenario.

  * •

ConvLSTM [10]: A model using a stack of images as input, pre-process those
images with pre-trained CNN and apply ConvLSTM on those features.

  * •

Convolutional-3D (C3D) [12] and Inflated-3D (I3D) [13]: two models pretrained
on Sports1M [43] using a stack of images as input and applying 3D convolutions
to extract features.

  * •

SPI-net [18] and TrouSPI-net [19]: two multi-modal models relying on
pedestrians’ pose kinematics extracted by OpenPose [44], relative euclidean
distance of key-points and evolution of the pedestrian spatial positioning.
Poses sequences are converted into 2D image-like spatio-temporal
representations and self-spatio-temporal attention is applied via CNN-based
models for multiple time resolutions. Each remaining feature is independently
processed via either U-GRUs [45] or feed forward neural network and fused by
either applying temporal and modality attention or sent to a fc layer to
predict crossing behaviors.

  * •

SingleRNN [23], Multi-stream RNN (MultiRNN) [24] and Stacked with multilevel
Fusion RNN (SFRNN) [46]: Three multi-modal models relying on RGB Images
extracted by VGG16 [8], pose kinematics extracted by OpenPose [44] and
evolution of the pedestrian spatial positioning. Input features are either
concatenated into a single vector and sent to a recurrent network followed by
a fc layer for crossing prediction, either processed independently by GRUs
[47] and the hidden state of GRUs are then concatenated and sent into a fc
layer for crossing prediction or either processed by GRUs [47] and fused
gradually at different levels of processing and complexity.

  * •

Pedestrian Crossing Prediction with Attention (PCPA) [1]: A multi-modal model
relying on RGB images extracted by C3D [12], pose kinematics extracted by
OpenPose [44] and evolution of the pedestrian spatial positioning. Non-images
features are independently encoded by GRUs [47] and each is fed to a temporal
attention block. 3D Convoluted features are flattened and fed into a fc layer.
Modality attention is then applied to all the branches to fuse them into a
single representation by weighted summation of the information from individual
modalities.

| AUC ($\uparrow$) | F1($\uparrow$) | ECE($\downarrow$) | MCE ($\downarrow$)
---|---|---|---|---
Method | pie ($\pm$0.02) | beh ($\pm$0.02) | all ($\pm$0.01) | pie ($\pm$0.03) | beh ($\pm$0.01) | all ($\pm$0.02) | pie ($\pm$0.01) | beh ($\pm$0.02) | all ($\pm$0.02) | pie ($\pm$0.02) | beh ($\pm$0.03) | all ($\pm$0.03)
VGG16 [8] | 0.52 | 0.62 | 0.76 | 0.28 | 0.67 | 0.60 | 0.07 | 0.06 | 0.20 | 0.24 | 0.13 | 0.25
Resnet [9] | 0.58 | 0.60 | 0.63 | 0.45 | 0.68 | 0.37 | 0.09 | 0.05 | 0.04 | 0.37 | 0.15 | 0.44
ConvLSTM [10] | 0.50 | 0.51 | 0.57 | 0.21 | 0.43 | 0.30 | 0.09 | 0.14 | 0.10 | 0.22 | 0.25 | 0.41
C3D [12] | 0.59 | 0.56 | 0.70 | 0.46 | 0.73 | 0.43 | 0.17 | 0.08 | 0.03 | 0.43 | 0.12 | 0.11
I3D [13] | 0.64 | 0.62 | 0.74 | 0.48 | 0.71 | 0.51 | 0.05 | 0.08 | 0.09 | 0.13 | 0.15 | 0.16
PCPA [1] | 0.69 | 0.57 | 0.64 | 0.56 | 0.67 | 0.38 | 0.12 | 0.04 | 0.12 | 0.36 | 0.13 | 0.28
SingleRNN [23] | 0.78 | 0.58 | 0.66 | 0.65 | 0.69 | 0.40 | 0.09 | 0.02 | 0.09 | 0.16 | 0.15 | 0.14
MultiRNN [24] | 0.76 | 0.54 | 0.60 | 0.64 | 0.74 | 0.35 | 0.06 | 0.08 | 0.19 | 0.13 | 0.17 | 0.378
SFRNN [46] | 0.77 | 0.63 | 0.62 | 0.67 | 0.58 | 0.36 | 0.07 | 0.08 | 0.11 | 0.11 | 0.29 | 0.16
Spi-Net [18] | 0.54 | 0.59 | 0.71 | 0.35 | 0.61 | 0.50 | 0.10 | 0.07 | 0.22 | 0.30 | 0.15 | 0.33
TrouSPI-net [19] | 0.73 | 0.59 | 0.56 | 0.61 | 0.76 | 0.32 | 0.07 | 0.05 | 0.24 | 0.12 | 0.13 | 0.41

TABLE I: Average prediction given by the three models trained on each training
sets for one given test-set (Ensembling). In addition to classification
metrics (we use arrows to indicate which direction is better), we compare
models with predictive uncertainty metrics such as Expected Calibration Error
(ECE) and Maximum Calibration Error (MCE). Dashed lines separate different
types of architectures

Figure 5: Critical Difference Diagram [48]: first a Friedman test is performed
to reject the null hypothesis, we then proceed with a post-hoc analysis based
on the Wilcoxon-Holm method. We compare the robustness of classifiers over
multiple training and testing sets shifts. We can see how each method ranks on
average. A thick horizontal line groups a set of classifiers that are not
significantly different ($\alpha=0.1$).

###  III-C Cross-dataset Evaluation Results

We present the coarse results of our cross-dataset evaluation on Fig 2. For
readability purposes, the corresponding critical difference diagram is
reported on Fig 5 and the average distribution of performance of the selected
approaches is reported on Fig 4. The results of the average prediction given
by the three models trained on each training set for one given test set is
reported on Table I.

As expected, all methods, regardless of their architecture or input
modalities, suffer a consequent performance drop when trained on $PIE$ and
tested on $JAAD$ and vice versa. Fig 4 shows that however robust the
individual classifier is, there is a general trend for classifiers to decline
when exposed to a different test set than the expected one. This is consistent
towards all our experiments with the exception of $JAAD_{behavior}$.
$JAAD_{all}$ being an extension to the set of samples with behavioral
annotations, $JAAD_{all}$ ”generalizes” well on $JAAD_{behavior}$ but
unsurprisingly, the converse is far from true. Even when trained on a
relatively diverse dataset ($PIE$) and inferred on a smaller one in comparison
($JAAD_{behavior}$), selected methods barely show signs of generalization.
More alarming, some methods even under-performed a random binary guess based
on class distribution when exposed to a different testing set than the
expected one. While the task, standardized inputs and observation length are
the same across all three datasets, none of the tested models reaches a
satisfactory level of generalization across any other testing set.

When it comes to compare performance towards small domain-shift at the
granular level of individuals methods, the critical difference diagram
reported on Fig 5, shows that none of the selected methods arise as a clear
winner when it comes to cross-dataset ranking. More importantly, the obtained
ranks of each method when evaluated under cross-dataset evaluation is far from
the one we usually consider when developing pedestrian crossing behavior
predictors: some general methods such as I3D or C3D are on par with multi-
modal methods specifically designed to tackle the problem of pedestrian
crossing prediction. This confirms the importance of rethinking the evaluation
methodology of our approaches.

The ensembling provided in Table I, is the closest plausible approximation of
the selected models’ robustness for real-world application as it integrates
all available conditions and training instances while removing the sampling
biases of each specific training set. It shows that the only empirical
evaluation of models in a direct train-test sets evaluation is not sufficient
to effectively conclude anything about its applicability in a real-world
scenario. This also demonstrates that the use of classification metrics alone
is not representative of the overall capacity of the models. For two given
models which are equivalent with respect to classification metrics (AUC or F1
score), their calibration (ECE and MCE) can differ drastically. This supports
our argument that the usage of uncertainty metrics should complement the
metrics conventionally used in order to obtain a comprehensive view of the
robustness of existing approaches.

Figure 6: Reliability Diagrams between I3D [13] randomly initialized (left)
and pre-trained on Sports1M [43](right) on $PIE$, $JAAD_{all}$ and
$JAAD_{behavior}$ datasets. If the model is perfectly calibrated, then the
diagram plots the identity function. Any deviation from a perfect diagonal
represents miscalibration: the model is either overconfident (orange) or
subconfident (green).

|  AUC ($\uparrow$) |  F1 ($\uparrow$) |  ECE ($\downarrow$) |  MCE ($\downarrow$)
---|---|---|---|---
Method | pie | beh | all | pie | beh | all | pie | beh | all | pie | beh | all
Non-pretrained | 0.55 ($\pm$0.04) | 0.50 ($\pm$0.02) | 0.69 ($\pm$ 0.05) | 0.34 ($\pm$0.09) | 0.65 ($\pm$0.08) | 0.54 ($\pm$0.06) | 0.205 ($\pm$0.062) | 0.184 ($\pm$ 0.089) | 0.111 ($\pm$ 0.069) | 0.290 ($\pm$0.039) | 0.338 ($\pm$0.191) | 0.162 ($\pm$0.075)
Ens-Nonpretrained | 0.59 ($\pm$0.06) | 0.58 ($\pm$0.05) | 0.64 ($\pm$0.04) | 0.37 ($\pm$0.10) | 0.55 ($\pm$0.10) | 0.39 ($\pm$0.05) | 0.065 ($\pm$0.022) | 0.131 ($\pm$0.058) | 0.091 ($\pm$0.026) | 0.248 ($\pm$0.122) | 0.644 ($\pm$0.152) | 0.280 ($\pm$0.172)
Deterministic | 0.72 ($\pm$0.01) | 0.56 ($\pm$0.03) | 0.76 ($\pm$0.03) | 0.60 ($\pm$0.02) | 0.74 ($\pm$0.02) | 0.61 ($\pm$0.02) | 0.026 ($\pm$0.007) | 0.143 ($\pm$ 0.020) | 0.054 ($\pm$0.010) | 0.063 ($\pm$0.007) | 0.239 ($\pm$ 0.030) | 0.118 ($\pm$0.020)
Ens-Deterministic | 0.64 ($\pm$0.01) | 0.62 ($\pm$0.01) | 0.73 ($\pm$ 0.1) | 0.49 ($\pm$0.02) | 0.70 ($\pm$0.01) | 0.51 ($\pm$0.01) | 0.053 ($\pm$0.003) | 0.080 ($\pm$0.001) | 0.097 ($\pm$0.016) | 0.120 ($\pm$0.013) | 0.138 ($\pm$0.024) | 0.172 ($\pm$0.022)
MC Dropout | 0.73 ($\pm$0.01) | 0.55 ($\pm$0.01) | 0.78 ($\pm$0.01) | 0.61 ($\pm$0.01) | 0.67 ($\pm$0.01) | 0.60 ($\pm$ 0.01) | 0.064 ($\pm$ 0.003) | 0.063 ($\pm$0.005) | 0.040 ($\pm$0.002) | 0.106 ($\pm$0.004) | 0.134 ($\pm$0.012) | 0.059 ($\pm$0.009)
Ens-MC Dropout | 0.61 ($\pm$0.01) | 0.61 ($\pm$0.01) | 0.73 ($\pm$0.01) | 0.42 ($\pm$0.01) | 0.49 ($\pm$0.02) | 0.53 ($\pm$0.01) | 0.053 ($\pm$0.002) | 0.053 ($\pm$0.003) | 0.129 ($\pm$0.002) | 0.096 ($\pm$0.005) | 0.120 ($\pm$0.013) | 0.181 ($\pm$0.007)
TempScaling | 0.70 ($\pm$0.02) | 0.58 ($\pm$0.02) | 0.76 ($\pm$0.01) | 0.57 ($\pm$0.03) | 0.72 ($\pm$0.01) | 0.61 ($\pm$ 0.02) | 0.020 ($\pm$0.005) | 0.070 ($\pm$ 0.020) | 0.037 ($\pm$0.08) | 0.050 ($\pm$0.010) | 0.300 ($\pm$0.130) | 0.146 ($\pm$0.035)
Ens-TempScaling | 0.61 ($\pm$ 0.01) | 0.66 ($\pm$0.01) | 0.75 ($\pm$0.01) | 0.41 ($\pm$0.02) | 0.69 ($\pm$0.01) | 0.56 ($\pm$0.01) | 0.058 ($\pm$0.004) | 0.054 ($\pm$0.008) | 0.142 ($\pm$0.015) | 0.127 ($\pm$0.012) | 0.237 ($\pm$0.068) | 0.215 ($\pm$0.016)
LL Dropout | 0.71 ($\pm$0.01) | 0.54 ($\pm$0.003) | 0.78 ($\pm$0.001) | 0.59 ($\pm$0.01) | 0.74 ($\pm$ 0.001) | 0.62 ($\pm$0.003) | 0.020 ($\pm$ 0.003) | 0.155 ($\pm$0.003) | 0.061 ($\pm$ 0.001) | 0.063 ($\pm$0.012) | 0.247 ($\pm$0.01) | 0.107 ($\pm$0.016)
Ens-LL Dropout | 0.65 ($\pm$ 0.002) | 0.62 ($\pm$0.003) | 0.73 ($\pm$0.002) | 0.49 ($\pm$0.003) | 0.70 ($\pm$0.001) | 0.50 ($\pm$0.003) | 0.052 ($\pm$0.003) | 0.081 ($\pm$0.002) | 0.098 ($\pm$0.002) | 0.105 ($\pm$0.002) | 0.145 ($\pm$0.023) | 0.176 ($\pm$0.005)
LL SVI | 0.74 ($\pm$0.01) | 0.53 ($\pm$0.01) | 0.77 ($\pm$ 0.003) | 0.62 ($\pm$0.01) | 0.76 ($\pm$0.01) | 0.57 ($\pm$0.004) | 0.021 ($\pm$0.002) | 0.162 ($\pm$0.004) | 0.026 ($\pm$0.003) | 0.059 ($\pm$0.009) | 0.214 ($\pm$0.006) | 0.054 ($\pm$0.009)
Ens-LL SVI | 0.68 ($\pm$0.003) | 0.61 ($\pm$0.003) | 0.69 ($\pm$0.003) | 0.55 ($\pm$0.003) | 0.73 ($\pm$0.002) | 0.43 ($\pm$0.004) | 0.045 ($\pm$0.003) | 0.036 ($\pm$0.003) | 0.046 ($\pm$0.005) | 0.146 ($\pm$0.027) | 0.075 ($\pm$0.009) | 0.133 ($\pm$0.016)

TABLE II: Average Pedestrian Crossing Prediction performance for $PIE$,
$JAAD_{behavior}$ and $JAAD_{all}$ (5 runs). Dashed lines separate each
probabilistic deep learning baseline. Each baseline is tested twice: first, in
a classical train-test evaluation protocol and then tested by ensembling all
three models trained on each training set to evaluate its robustness to small
domain shift. We highlight the highest scores for each metric and for both
evaluation protocols: train-test or ensembling.

###  III-D Role of pre-training in uncertainty calibration

Table I illustrates that generic baseline methods (i.e. VGG16, C3D, I3D) pre-
trained on well diverse and dense datasets further away from the target
domain, benefit in terms of generalization and uncertainty calibration as they
are on par with the methods specifically designed to tackle the problem of
pedestrian crossing prediction, which was not the case in a simple train-test
evaluation setting.

To better isolate the effects of pre-training with larger datasets we consider
two I3D [13] but trained with different configuration: the first one being
randomly initialized and the second one being pre-trained on Sports1M [43]. We
assess their performance on the same datasets and report our findings in Fig
6. We show that pre-trained models significantly outperform randomly
initialized models across all three datasets in terms of calibration. As far
as robustness aspects towards small domain shifts are concerned, this may
become an important factor to consider when designing pedestrian crossing
behavior approaches for real-world scenarii. The training source being
generally not dense in variety of conditions nor in the number of examples,
the results provided on each dataset might just come from noise or over-
fitting models on testing sets. Pre-training well-established models on
diverse and dense datasets further away from the target domain before fine-
tunning to our target task might prove efficient and mandatory for the next
step of pedestrian crossing behavior prediction: generalization and vehicle
implementation.

##  IV Improving Uncertainty Calibration

For the very same approach, there is a significant discrepancy between
traditional train-test and cross-dataset evaluation results. This calls into
question the reliability of current methods in regard to their capacity to
generalize. In addition, we have shown that the standard classification
metrics are not sufficient to reliably evaluate an approach since the use of
uncertainty metrics raises additional issues that are not reflected otherwise.
We are confident that the future breakthroughs in the area will not occur by
outperforming current state-of-the-art by a small margin on conventionally
used evaluation protocols as they currently fail to provide the big picture of
pedestrian crossing behavior prediction.

As we encourage the community to change the direction in which we are taking
the research field, we investigate how additional baselines from the
probabilistic deep learning literature improve the generalization ability of
pedestrian behavior predictors towards small domain shifts. We believe that
those methods could prove useful for the next generation of predictors and
present our results with the intention that they will serve as a baseline for
future work addressing our prescriptions.

###  IV-A Baselines from the probabilistic deep learning literature

Below, we present the selected methods from the probabilistic deep learning
literature applied on top of an I3D [13] model:

  * •

Non-pretrained and Deterministic: Maximum softmax probability [39] of $N$
networks trained independently on each dataset using either random
initialization or pre-trained weights from Sports1M [43]. (We set $N=5$ for
each method below.)

  * •

Monte-Carlo Dropout (MC Dropout): Dropout activated at test time as an
approximate bayesian inference in deep Gaussian processes [49].

  * •

Temperature Scaling444https://github.com/gpleiss/temperature_scaling
(TempScaling): Post-hoc calibration of softmax probability by temperature
scaling using a validation set [36].

  * •

Last Layer Dropout (LL Dropout): Bayesian inference for the parameters of the
last layer only: Dropout activated at test time on the activations before the
last layer.

  * •

Last Layer Stochastic Variational Bayesian Inference (LL SVI): Mean field
stochastic variational inference on the last layer using Flipout [50].

  * •

Ensembling (Ens): Average prediction of three networks trained independently
on each training set using pre-trained weights [51]. Similarly to Table I, we
use ensembling as a plausible approximation of one model’s robustness for
real-world scenarii.

###  IV-B Discussion

We present the results obtained by probabilistic methods for both evaluation
protocols: train-test and ensembling on Table II. This allows us to report the
effect of dataset shift on accuracy and calibration for the probabilistic deep
learning methods. Naturally, we would like to obtain a model, that is well-
calibrated on the training and testing distributions of each dataset and
remains calibrated with ensembling. We observe that, similarly to the
deterministic methods, the quality of predictions consistently degrades with
dataset shift regardless of the selected probabilistic method for both $PIE$
and $JAAD_{all}$. However, overall robustness degrades more significantly for
some methods. For instance, TempScaling, e.g. post-hoc calibration of softmax
probability, seems to be one of the best train-test probabilistic methods in
regards to expected calibration error (ECE) when evaluated in a standard
train-test procedure but falls behind when evaluated under dataset shift. In
fact, when evaluated under dataset shift, all the methods except Non-
pretrained ones outperform TempScaling in regards to ECE. Similarly, we report
that better calibration and accuracy on each test set does not correlate with
better calibration under ensembling: the average ECE of the methods when
evaluated with classical train-test scenario is [0.166, 0.074, 0.056, 0.042,
0.079, 0.070] and the average ECE of the same methods under dataset shift are
[0.096, 0.077, 0.078, 0.085, 0.077, 0.042]. Interestingly, most of the
selected probabilistic methods perform better on average than the
deterministic I3D under train-test evaluation protocols but fail to generalize
when exposed to dataset shift. The exception to the rule is LL SVI, which
looks very promising in terms of generalization to small domain shift. As our
experiments required pre-trained weights from I3D, we could not replace each
convolutional layer with mean-field variational Flipout layers, we only
changed the last layer of the given model to obtain a variational bayesian
inference for a quick baseline. Nevertheless, we believe that this could be a
future research to consider. We should explore the effects of transferring
initially learned features on large bases further away from the target task
and explore how probabilistic methods react to transfer-learning and domain-
shift.

##  V Conclusion

In this paper, we show that the classical train-test sets evaluation for
pedestrian crossing prediction, i.e., models being trained and tested on the
same dataset, is not sufficient to efficiently compare nor conclude anything
about their applicability in a real-world scenario: the benchmarks being
either too small or too loose in variety of scenarii, it is easy for a given
model to over-fit on a specific target dataset. In order to evaluate the
generalization capacity of the approaches, we conduct a study based on direct
cross-dataset evaluation for eleven methods representing the diversity of
architectures and modalities used for pedestrian crossing prediction. We found
a huge lack of generalization and robustness for all selected approaches. This
led us to a ranking of existing approaches that is much more complex and less
absolute than the standard one. We secondly discuss the importance of
quantifying a model’s uncertainty. Although this is currently completely
disregarded, it is common sense to use it in our field of application. We
discover two interesting properties: pre-training well-established models on
diverse and dense datasets further away from the target domain before fine-
tuning to our target task improves calibration and, two models with equivalent
classification scores do not necessarily have equivalent calibration scores.
This may prove interesting to consider when comparing their usefulness in
real-world scenarii with inputs distribution frequently shifted from the
training distribution. Finally, we enforce the importance of evaluating the
robustness of pedestrian crossing behavior models by evaluating how
trustworthy are their uncertainty estimates under domain shifts with cross-
dataset evaluation. We encourage the community to consider those new protocols
and metrics in order to reach the end-goal of pedestrian crossing behavior
predictors: vehicle implementation. In order to build the foundation on which
future work should be based on, and, in addition to the eleven deterministic
baselines evaluated under domain shift, we report the results of multiple
baselines from the probabilistic deep learning literature, designed to tackle
the problem of improving model calibration. Given all of the above, we advise
the community to change the direction in which we are taking the research
field: with so little existing data, non-existent generalization of models,
and inconclusive ranking of them, we need to agree to properly evaluate our
approaches in order to minimize the noise of our productions and thus, make
the research field more sustainable and representative of the real advances to
come.

## References

  * [1] I. Kotseruba, A. Rasouli, and J. K. Tsotsos, “Benchmark for evaluating pedestrian action prediction,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021, pp. 1258–1268.
  * [2] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, _et al._ , “Imagenet large scale visual recognition challenge,” _International journal of computer vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [3] L. Beyer, O. J. Hénaff, A. Kolesnikov, X. Zhai, and A. v. d. Oord, “Are we done with imagenet?” _arXiv preprint arXiv:2006.07159_ , 2020.
  * [4] C. Northcutt, L. Jiang, and I. Chuang, “Confident learning: Estimating uncertainty in dataset labels,” _Journal of Artificial Intelligence Research_ , vol. 70, pp. 1373–1411, 2021.
  * [5] P. Stock and M. Cisse, “Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 498–512.
  * [6] A. Rasouli, I. Kotseruba, and J. K. Tsotsos, “Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2017, pp. 206–213.
  * [7] D. Varytimidis, F. Alonso-Fernandez, B. Duran, and C. Englund, “Action and intention recognition of pedestrians in urban traffic,” in _2018 14th International conference on signal-image technology & internet-based systems (SITIS)_. IEEE, 2018, pp. 676–682.
  * [8] K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” _arXiv e-prints_ , p. arXiv:1409.1556, Sept. 2014.
  * [9] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [10] X. Shi, Z. Chen, H. Wang, D. Y. Yeung, W. K. Wong, and W. C. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” _Advances in neural information processing systems_ , vol. 2015, pp. 802–810, 2015.
  * [11] P. Gujjar and R. Vaughan, “Classifying pedestrian actions in advance using predicted video of urban driving scenes,” in _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 2097–2103.
  * [12] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 4489–4497.
  * [13] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 4724–4733.
  * [14] M. Chaabane, A. Trabelsi, N. Blanchard, and R. Beveridge, “Looking ahead: Anticipating pedestrians crossing with future frames prediction,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 2297–2306.
  * [15] K. Saleh, M. Hossny, and S. Nahavandi, “Real-time intent prediction of pedestrians for autonomous ground vehicles via spatio-temporal densenet,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 9704–9710.
  * [16] L. Achaji, J. Moreau, T. Fouqueray, F. Aioun, and F. Charpillet, “Is attention to bounding boxes all you need for pedestrian action prediction?” _arXiv preprint arXiv:2107.08031_ , 2021.
  * [17] A. Ranga, F. Giruzzi, J. Bhanushali, E. Wirbel, P. Pérez, T.-H. Vu, and X. Perotton, “Vrunet: Multi-task learning model for intent prediction of vulnerable road users,” _Electronic Imaging_ , vol. 2020, no. 16, pp. 109–1, 2020.
  * [18] J. Gesnouin, S. Pechberti, G. Bresson, B. Stanciulescu, and F. Moutarde, “Predicting intentions of pedestrians from 2d skeletal pose sequences with a representation-focused multi-branch deep learning network,” _Algorithms_ , vol. 13, no. 12, p. 331, 2020.
  * [19] J. Gesnouin, S. Pechberti, B. Stanciulcscu, and F. Moutarde, “Trouspi-net: Spatio-temporal attention on parallel atrous convolutions and u-grus for skeletal pedestrian crossing prediction,” in _2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)_. IEEE, 2021, pp. 01–07.
  * [20] A. Marginean, R. Brehar, and M. Negru, “Understanding pedestrian behaviour with pose estimation and recurrent networks,” in _2019 6th International Symposium on Electrical and Electronics Engineering (ISEEE)_ , 2019, pp. 1–6.
  * [21] O. Ghori, R. Mackowiak, M. Bautista, N. Beuter, L. Drumond, F. Diego, and B. Ommer, “Learning to forecast pedestrian intention from pose dynamics,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_ , 2018, pp. 1277–1284.
  * [22] P. R. G. Cadena, M. Yang, Y. Qian, and C. Wang, “Pedestrian graph: Pedestrian crossing prediction based on 2d pose estimation and graph convolutional networks,” in _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_ , 2019, pp. 2000–2005.
  * [23] I. Kotseruba, A. Rasouli, and J. K. Tsotsos, “Do they want to cross? understanding pedestrian intention for behavior prediction,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_ , 2020, pp. 1688–1693.
  * [24] A. Bhattacharyya, M. Fritz, and B. Schiele, “Long-term on-board prediction of people in traffic scenes under uncertainty,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4194–4202.
  * [25] J. Yue-Hei Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici, “Beyond short snippets: Deep networks for video classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 4694–4702.
  * [26] D. Yang, H. Zhang, E. Yurtsever, K. Redmill, and Ü. Özgüner, “Predicting pedestrian crossing intention with feature fusion and spatio-temporal attention,” _arXiv preprint arXiv:2104.05485_ , 2021.
  * [27] J. Lorenzo, I. P. Alonso, R. Izquierdo, A. L. Ballardini, Á. H. Saz, D. F. Llorca, and M. Á. Sotelo, “Capformer: Pedestrian crossing action prediction using transformer,” _Sensors_ , vol. 21, no. 17, p. 5694, 2021\.
  * [28] J. Lorenzo, I. Parra, and M. Sotelo, “Intformer: Predicting pedestrian intention with the aid of the transformer architecture,” _arXiv preprint arXiv:2105.08647_ , 2021.
  * [29] S. Bapu Sridhar and A. Moosakhanian, “Pedestrian intent prediction using deep machine learning,” 2020.
  * [30] A. Singh and U. Suddamalla, “Multi-input fusion for practical pedestrian intention prediction,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2304–2311.
  * [31] D. Zhang, F. Shi, Y. Meng, Y. Xu, X. Xiao, and W. Li, “Pedestrian intention prediction via depth augmented scene restoration,” in _2021 5th CAA International Conference on Vehicular Control and Intelligence (CVCI)_. IEEE, 2021, pp. 1–6.
  * [32] I. Hasan, S. Liao, J. Li, S. U. Akram, and L. Shao, “Generalizable pedestrian detection: The elephant in the room,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 11 328–11 337.
  * [33] Y. Chen, P. Liu, M. Zhong, Z.-Y. Dou, D. Wang, X. Qiu, and X. Huang, “Cdevalsumm: An empirical study of cross-dataset evaluation for neural summarization systems,” _arXiv preprint arXiv:2010.05139_ , 2020.
  * [34] P. Guo, Z. Xue, L. R. Long, and S. Antani, “Cross-dataset evaluation of deep learning networks for uterine cervix segmentation,” _Diagnostics_ , vol. 10, no. 1, p. 44, 2020.
  * [35] M. P. Naeini, G. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using bayesian binning,” in _Twenty-Ninth AAAI Conference on Artificial Intelligence_ , 2015.
  * [36] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 1321–1330.
  * [37] J. Heo, H. B. Lee, S. Kim, J. Lee, K. J. Kim, E. Yang, and S. J. Hwang, “Uncertainty-aware attention for reliable interpretation and prediction,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , 2018, pp. 917–926.
  * [38] Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snoek, “Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift,” _Advances in Neural Information Processing Systems_ , vol. 32, pp. 13 991–14 002, 2019.
  * [39] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” _arXiv preprint arXiv:1610.02136_ , 2016.
  * [40] Y. Ding, J. Liu, J. Xiong, and Y. Shi, “Revisiting the evaluation of uncertainty estimation and its application to explore model complexity-uncertainty trade-off,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 4–5.
  * [41] A. Rasouli, I. Kotseruba, and J. K. Tsotsos, “Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior,” in _The IEEE International Conference on Computer Vision (ICCV) Workshops_ , Oct 2017.
  * [42] A. Rasouli, I. Kotseruba, T. Kunic, and J. K. Tsotsos, “Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction,” in _ICCV_ , 2019.
  * [43] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2014, pp. 1725–1732.
  * [44] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, “Realtime multi-person 2d pose estimation using part affinity fields,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 7291–7299.
  * [45] R. Rozenberg, J. Gesnouin, and F. Moutarde, “Asymmetrical bi-rnn for pedestrian trajectory encoding,” _arXiv preprint arXiv:2106.04419_ , 2021\.
  * [46] A. Rasouli, I. Kotseruba, and J. K. Tsotsos, “Pedestrian action anticipation using contextual feature fusion in stacked rnns,” in _BMVC_ , 2019.
  * [47] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” _arXiv preprint arXiv:1412.3555_ , 2014.
  * [48] J. Demšar, “Statistical comparisons of classifiers over multiple data sets,” _The Journal of Machine Learning Research_ , vol. 7, pp. 1–30, 2006\.
  * [49] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _international conference on machine learning_. PMLR, 2016, pp. 1050–1059.
  * [50] Y. Wen, P. Vicol, J. Ba, D. Tran, and R. Grosse, “Flipout: Efficient pseudo-independent weight perturbations on mini-batches,” _arXiv preprint arXiv:1803.04386_ , 2018.
  * [51] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.