example by observing that the central $10\subseteq 5\times 2S_{4}$ acts on
$11^{2}$ through the isomorphism $10\cong(\mathbf{Z}/11)^{\times}$ and
applying Lemma 2.3.

The $3$-Sylow is contained in a maximal subgroup of shape
$2^{11}:\mathrm{M}_{24}$. There are two conjugacy classes of elements of order
$3$ in $\mathrm{M}_{24}$; the restriction map
$\operatorname{H}^{4}(\mathrm{M}_{24})_{(3)}\cong\mathbf{Z}/3\to\operatorname{H}^{4}(\langle
3\mathrm{a}\rangle)$ is zero, whereas
$\operatorname{H}^{4}(\mathrm{M}_{24})_{(3)}\cong\mathbf{Z}/3\to\operatorname{H}^{4}(\langle
3\mathrm{b}\rangle)$ is an isomorphism [16]. But $\mathrm{J}_{4}$ has only one
conjugacy class of order $3$. It follows that
$\operatorname{H}_{4}(\mathrm{J}_{4})_{(3)}=0$.

The $2$-Sylow is contained in $2^{11}:\mathrm{M}_{24}$, and also in a maximal
subgroup of shape $2^{1+22}.3\mathrm{M}_{22}.2$ centralizing conjugacy class
$2\mathrm{a}\in\mathrm{J}_{4}$. This latter subgroup turns out to be more
useful. Using Cohomolo and Proposition 5.3, we find that the $E_{2}$ page for
the LHS spectral sequence reads

$\displaystyle\begin{array}[]{ccccc}\leq 4\\\ 0&0&2\\\ 0&0&0\\\ 0&0&0&0&0\\\
\mathbf{Z}&0&2&4&2^{2}\times 3\end{array}$

The entry “$\leq 4$” in bidegree $(0,4)$ comes from the LES for the extension

$\displaystyle\operatorname{H}^{4}\big{(}2^{1+12}\big{)}=2^{12}.\operatorname{Alt}^{2}\big{(}2^{12}\big{)}.\big{(}\operatorname{Alt}^{3}\big{(}2^{12}\big{)}/2^{12}\big{)}.2$

from Section 3.3 and the calculations

$\displaystyle\operatorname{H}^{0}\big{(}3\mathrm{M}_{22}.2;2^{12}\big{)}=\operatorname{H}^{1}\big{(}3\mathrm{M}_{22}.2;2^{12}\big{)}=\operatorname{H}^{0}\big{(}3\mathrm{M}_{22}.2;\operatorname{Alt}^{3}\big{(}2^{12}\big{)}\big{)}=0,$
$\displaystyle\operatorname{H}^{0}\big{(}3\mathrm{M}_{22}.2;\operatorname{Alt}^{2}\big{(}2^{12}\big{)}\big{)}=2.$

We showed during the proof of Theorem 6.3 that
$\operatorname{H}^{4}(\mathrm{M}_{22}:2)_{(2)}=2^{2}$ is detected by
restricting to the three conjugacy classes of order $2$ in
$\mathrm{M}_{22}:2$. All three of these classes have order-$2$ preimages in
$2^{1+22}.3\mathrm{M}_{22}.2$. But both conjugacy classes of order $2$ in
$\mathrm{J}_{4}$ meet $2^{1+22}$. It follows that the images of
$\operatorname{H}^{4}(\mathrm{M}_{22}:2)_{(2)}\to\operatorname{H}^{4}\big{(}2^{1+22}.3\mathrm{M}_{22}.2\big{)}_{(2)}$
and
$\operatorname{H}^{4}(\mathrm{J}_{4})_{(2)}\to\operatorname{H}^{4}\big{(}2^{1+22}.3\mathrm{M}_{22}.2\big{)}_{(2)}$
have trivial intersection.

In particular, if $\operatorname{H}^{4}(\mathrm{J}_{4})\neq 0$, then it
contains an order-$2$ class $\alpha$ whose restriction to $2^{1+12}$ is the
unique element $\tilde{q}\in\operatorname{H}^{4}\big{(}2^{1+12}\big{)}$ which
is twice some other element. That unique element is pulled back from
$\operatorname{H}^{4}(2^{12})$, where it corresponds to the quadratic form
$q\in\operatorname{Sym}^{2}(2^{12})$ defining the extension $2^{1+12}$. Choose
any pair of vectors $v_{1},v_{2}\in 2^{12}$ such that $q(v_{1})=q(v_{2})\neq
0$. Their lifts generate a quaternion group $Q_{8}=2^{1+2}\subseteq 2^{1+12}$,
and $\tilde{q}\in\operatorname{H}^{4}(2^{1+12})$ restricts nontrivially to
that quaternion group. (These lifts of $v_{1}$, $v_{2}$ have order $4$ in
$2^{1+12}$, and so are in conjugacy class $4\mathrm{a}$ in
$2^{1+12}.3\mathrm{M}_{22}.2$.)

Choose $\tilde{g}\in 2^{1+12}.3\mathrm{M}_{22}.2$ in conjugacy class
$4\mathrm{b}$. The character table libraries confirm that this $\tilde{g}$ has
the following properties: $\tilde{g}^{2}$ is the nontrivial central element in
$2^{1+12}\subseteq 2^{1+12}.3\mathrm{M}_{12}.2$; the image $g$ in
$3\mathrm{M}_{22}.2$ of $\tilde{g}$ is in conjugacy class $2\mathrm{a}$. In
particular, $g$ acts on $2^{12}$ fixing $8$ dimensions. Regardless of the Arf
invariant of $q$, one can find a basis such that $q$ vanishes on at most one
basis vector; thus we can find a vector $v_{1}\in 2^{12}$ with $q(v)\neq 0$
and $vg\neq v$. (Following GAP’s conventions, we write the action of
$3\mathrm{M}_{22}.2$ on $2^{12}$ from the right.) Set $v_{2}=vg$; then
$q(v_{2})\neq 0$, and so the lifts of $v_{1},v_{2}$ generate a $Q_{8}$ as in
the previous paragraph.

Furthermore, we have arranged for the lifts of $v_{1}$, $v_{2}$ together with
$\tilde{g}$ to generate a binary dihedral group $2D_{8}\subseteq
2^{1+12}.3\mathrm{M}_{22}.2$ extending this $Q_{8}$. Suppose that
$\operatorname{H}^{4}(\mathrm{J}_{4})\neq 0$, and let $\alpha$ denote its
order-$2$ class. Then $\alpha|_{2D_{8}}$ has order $2$, and so is divisible by
$8$. But then $\alpha|_{Q_{8}}=0$. This contradicts the fact that $Q_{8}$
detected $\tilde{q}$, and so $\operatorname{H}^{4}(\mathrm{J}_{4})$ must
vanish. ∎

######  Theorem 7.4.

$\operatorname{H}^{4}(\mathrm{Ly})$ is trivial.

###### Proof.

For $p>5$, the Sylow $p$-subgroup of $\mathrm{Ly}$ is cyclic, so Lemma 2.2
applies.

$\mathrm{G}_{2}(5)$ and $3\mathrm{McL}:2$ are subgroups (in fact, maximal
subgroups) of the $\mathrm{Ly}$ [29, Propositions 2.5 and 5.4]. The $5$-Sylow
is contained in $\mathrm{G}_{2}(5)$, so the
$\operatorname{H}^{4}(\mathrm{Ly})_{(5)}$ vanishes by Lemma 4.3.

The $2$\- and $3$-Sylows are contained in $3\mathrm{McL}:2$. By Theorem 6.5,
the only cohomology of the latter is pulled back from the quotient
$\mathbf{Z}/2$, and so is detected on a conjugacy class of order $2$
(specifically, class $2\mathrm{b}\in 3\mathrm{McL}:2$). But $\mathrm{Ly}$ has
only one conjugacy class of order $2$, and it meets $3\mathrm{McL}\subseteq
3\mathrm{McL}:2$ (since $3\mathrm{McL}$ has a class of order $2$). It follows
that the pullback
$\operatorname{H}^{4}(2)\overset{\sim}{\to}\operatorname{H}^{4}(3\mathrm{McL}:2)$
and the restriction
$\operatorname{H}^{4}(\mathrm{Ly})_{(2)}\to\operatorname{H}^{4}(3\mathrm{McL}:2)$
have nonintersecting images. ∎

##  8 Monster sections

###  8.1 Held group

The Held group is small enough to be accessible by the methods of Sections
2.2–2.3.

######  Theorem 8.1.

$\operatorname{H}^{4}(\mathrm{He})\cong\mathbf{Z}/12$. It is spanned by
fractional Pontryagin classes.

###### Proof.

The primes not covered by Lemma 2.2 are $p=2$, $3$, and $7$.

The normalizer of a $7$-Sylow in $\mathrm{He}$ has shape $7^{1+2}:(3\times
S_{3})$. There are no $7$s in its low cohomology:
$\operatorname{H}^{4}\big{(}7^{1+2}:(3\times
S_{3})\big{)}=\operatorname{H}^{4}(3\times S_{3})=2\times 3^{2}$.

The $3$-Sylow in $\mathrm{He}$ is inside a maximal subgroup of shape $3S_{7}$,
with $\operatorname{H}^{4}(3S_{7})=2^{2}\times 4\times 3^{2}$. We claim that
the inclusions
$\operatorname{H}^{4}(\mathrm{He})_{(3)}\to\operatorname{H}^{4}(3S_{7})_{(3)}$
and $\operatorname{H}^{4}(S_{7})_{(3)}=3\to\operatorname{H}^{4}(3S_{7})_{(3)}$
have nonintersecting images, giving an upper bound
$\operatorname{H}^{4}(\mathrm{He})_{(3)}\leq 3$. To show this, we first
observe that if $C_{3}$ is a cyclic group of order $3$ then the two nonzero
classes in $\operatorname{H}^{4}(C_{3})\cong\mathbf{Z}/3$ are canonically
distinguished: one, which we will call $t^{2}\in\operatorname{H}^{4}(C_{3})$,
is the cup-square of both nonzero classes in
$\operatorname{H}^{2}(C_{3})\cong\mathbf{Z}/3$, and the other is not a cup-
square. Now consider the class
$c_{2}(\mathrm{Perm})\in\operatorname{H}^{4}(S_{7})$, where $\mathrm{Perm}$
denotes the defining permutation representation. There are three conjugacy
classes of order $3$ in $3S_{7}$: the “central” one (not actually central – it
is inverted by the odd elements of $S_{7}$), and two that act on
$\mathrm{Perm}$ with cycle structures $1^{4}3^{1}$ and $1^{1}3^{2}$,
respectively. It follows that $c_{2}(\mathrm{Perm})|_{\langle
1^{4}3^{1}\rangle}=-t^{2}$ whereas $c_{2}(\mathrm{Perm})|_{\langle
1^{1}3^{2}\rangle}=+t^{2}$, meaning that $c_{2}(\mathrm{Perm})$ takes
different values on these two classes. However, these two classes merge to the
same class in $\mathrm{He}$, and so $c_{2}(\mathrm{Perm})$ cannot be the
restriction of a cohomology class on $\mathrm{He}$.

To establish the lower bound $\operatorname{H}^{4}(\mathrm{He})_{(3)}\geq 3$,
we observe that the smallest irrep of $\mathrm{He}$ has dimension $51$, and
conjugacy class $3\mathrm{b}\in\mathrm{He}$ acts with trace $0$, and so
$c_{2}$ of this irrep, when restricted to $\langle 3\mathrm{b}\rangle$, does
not vanish.

For the prime $p=2$, we use the $2$-Sylow-containing maximal subgroup of shape
$2^{6}:3S_{6}$. The $E_{2}$ page (localized at $p=2$) of the LHS spectral
sequence for this extension reads

$\displaystyle\begin{array}[]{ccccc}E_{2}^{04}\\\ 0&2&2^{2}\\\ 0&0&0\\\
0&0&0&0&0\\\ \mathbf{Z}&0&2&2&2^{2}\times 4\end{array}$

with

$\displaystyle
E_{2}^{04}=\operatorname{H}^{0}\big{(}3S_{6};V.\operatorname{Alt}^{2}(V).\operatorname{Alt}^{3}(V)\big{)},\qquad
V=\big{(}2^{6}\big{)}^{*}.$

Since

$\displaystyle\operatorname{H}^{0}(3S_{6};V)=\operatorname{H}^{0}\big{(}3S_{6};\operatorname{Alt}^{2}(V)\big{)}=0,\qquad\operatorname{H}^{0}\big{(}3S_{6};\operatorname{Alt}^{3}(V)\big{)}\cong\mathbf{Z}/2,$

we find that

$\displaystyle E_{2}^{04}\leq 2.$

We claim that the inclusions
$\operatorname{H}^{4}(S_{6})_{(2)}\cong\operatorname{H}^{4}(3S_{6})_{(2)}\to\operatorname{H}^{4}\big{(}2^{6}:3S_{6}\big{)}$
and
$\operatorname{H}^{4}(\mathrm{He})_{(2)}\to\operatorname{H}^{4}\big{(}2^{6}:3S_{6}\big{)}$
have trivial intersection. To establish this claim, we first study
$\operatorname{H}^{4}(S_{6})_{(2)}$. There are four 5-dimensional complex
irreps of $S_{6}$, corresponding to the characters $\chi_{3}$, $\chi_{4}$,
$\chi_{5}$, and $\chi_{6}$. Their second Chern classes, when restricted to the
conjugacy classes $2\mathrm{b}$, $4\mathrm{a}$, and $4\mathrm{b}$ in $S_{6}$,
are

$\displaystyle\begin{array}[]{c|ccc}&2\mathrm{b}&4\mathrm{a}&4\mathrm{b}\\\
\hline\cr&&&\\\\[-10.0pt] c_{2}(\chi_{3})&0&-t^{2}&+t^{2}\\\
c_{2}(\chi_{4})&0&-t^{2}&-t^{2}\\\ c_{2}(\chi_{5})&1&-t^{2}&-t^{2}\\\
c_{2}(\chi_{6})&1&-t^{2}&+t^{2}\end{array}$

Our notation is the following. The two classes in
$\operatorname{H}^{4}(\langle 2\mathrm{b}\rangle)\cong\mathbf{Z}/2$ are “$0$”
and “$1$”. If $C_{4}$ is a cyclic group of order $4$, the two generators of
$\operatorname{H}^{2}(C_{4})$ have the same cup-square in
$\operatorname{H}^{4}(C_{4})$, which we call “$+t^{2}$”; the generator of
$\operatorname{H}^{4}(C_{4})$ which is not a cup-square is called “$-t^{2}$”.

The images of
$\\{c_{2}(\chi_{3}),c_{2}(\chi_{4}),c_{2}(\chi_{5}),c_{2}(\chi_{6})\\}$ in
$\operatorname{H}^{4}(\langle
2\mathrm{b}\rangle)\times\operatorname{H}^{4}(\langle
4\mathrm{a}\rangle)\times\operatorname{H}^{4}(\langle
4\mathrm{b}\rangle)\cong\mathbf{Z}/2\times(\mathbf{Z}/4)^{2}$ together span a
group isomorphic to $(\mathbf{Z}/2)^{2}\times\mathbf{Z}/4$. It follows that
these four classes span
$\operatorname{H}^{4}(S_{6})_{(2)}\cong(\mathbf{Z}/2)^{2}\times\mathbf{Z}/4$
and that the restriction map
$\operatorname{H}^{4}(S_{6})_{(2)}\to\operatorname{H}^{4}(\langle
2\mathrm{b}\rangle)\times\operatorname{H}^{4}(\langle
4\mathrm{a}\rangle)\times\operatorname{H}^{4}(\langle 4\mathrm{b}\rangle)$ is
an injection.

On the other hand, the subgroup $2^{6}\subseteq
2^{6}:3S_{6}\subseteq\mathrm{He}$ meets both conjugacy classes of order $2$,
and the order-$4$ preimages in $2^{6}\subseteq 2^{6}:3S_{6}$ of the classes
$4\mathrm{a},4\mathrm{b}\in S_{6}$ are $\mathrm{He}$-conjugate to preimages of
order-$2$ elements in $S_{6}$. It follows that the image of
$\operatorname{H}^{4}(S_{6})_{(2)}\cong\operatorname{H}^{4}(3S_{6})_{(2)}\to\operatorname{H}^{4}\big{(}2^{6}:3S_{6}\big{)}$
does not intersect $\operatorname{H}^{4}(\mathrm{He})$.

We have so far established the following upper bound on
$\operatorname{H}^{4}(\mathrm{He})_{(2)}$: it is a group of order at most $4$.
The last ingredient needed is a cohomology class of order divisible by $4$.
Let $V$ denote the irreducible $\mathrm{He}$-module with character
$\chi_{19}$: it is real and of dimension $7650$. Consider the conjugacy class
$4\mathrm{a}\in\mathrm{He}$. It squares to $2\mathrm{a}$, and

$\displaystyle\chi_{19}(4\mathrm{a})=6,\qquad\chi_{19}(2\mathrm{a})=90.$

Therefore $4\mathrm{a}$ acts with spectrum
$1^{1938}i^{1890}(-1)^{1932}(-i)^{1890}$, and so the total Chern class of $V$,
when restricted to $\langle 4\mathrm{a}\rangle$, is

$\displaystyle
c(V)=1^{1938}(1-t)^{1890}(1-2t)^{1932}(1+t)^{1890}=1-2t^{2}+\cdots\pmod{4t}.$

In particular, $c_{2}(V)|_{\langle 4\mathrm{a}\rangle}\neq 0$. But $V$ is a
real representation, and therefore Spin (since $\mathrm{He}$ has trivial Schur
multiplier). So it has a fractional Pontryagin class, twice of which is
$c_{2}(V)$. It follows that $\frac{p_{1}}{2}(V)$ has order divisible by $4$,
and so $\operatorname{H}^{4}(\mathrm{He})_{(2)}\cong\mathbf{Z}/4$. ∎

###  8.2 Harada–Norton and Thompson groups

We were able to obtain partial results about the Harada–Norton and Thompson
groups $\mathrm{HN}$ and $\mathrm{Th}$.

######  Theorem 8.2.

$\operatorname{H}^{4}\big{(}\mathrm{HN};\mathbf{Z}\big{[}\frac{1}{2}\big{]}\big{)}\cong\mathbf{Z}/3$.
At the prime $2$, $\operatorname{H}^{4}(\mathrm{HN})_{(2)}$ has order at most
$16$ and exponent at most $8$.

###### Proof.

Lemma 2.2 handles the primes $p\geq 7$. The $5$-Sylow in $\mathrm{HN}$ is
contained in a maximal subgroup of shape $5^{1+4}.2^{1+4}.5.4$; the LHS
spectral sequence gives
$\operatorname{H}^{4}\big{(}5^{1+4}.2^{1+4}.5.4\big{)}_{(5)}=0$.

The $3$-Sylow is inside a maximal subgroup of shape $3^{1+4}:4A_{5}$, where by
“$4A_{5}$” we mean the “diagonal” central extension $2.(A_{5}\times 2)$. HAP
can directly compute

$\displaystyle\operatorname{H}^{4}\big{(}3^{1+4}:4A_{5}\big{)}_{(3)}=3^{2}.$

We claim that the generator of $\operatorname{H}^{4}(4A_{5})_{(3)}=3$, when
pulled back to $3^{1+4}:4A_{5}$, is not the restriction of a class on
$\mathrm{HN}$. Indeed, that generator has nontrivial restriction to the
elements of order $3$ in $4A_{5}$, and so its pullback has nonzero restriction
to some elements of order $3$. But both conjugacy classes of order $3$ in
$\mathrm{HN}$ meet $3^{1+4}\subseteq 3^{1+4}:4A_{5}$. Finally, we claim that
$\operatorname{H}^{4}(\mathrm{HN})_{(3)}\neq 0$. There is a unique conjugacy
class of order $9$ in $\mathrm{HN}$, and its traces on either
$133$-dimensional representation, which characters $\chi_{2}$ and $\chi_{3}$,
are $\chi_{2}(9\mathrm{a})=1$ and
$\chi_{2}\big{(}9\mathrm{a}^{3}\big{)}=\chi_{2}(3\mathrm{b})=-2$. From this
one can compute that $c_{2}(\chi_{2})|_{\langle 9\mathrm{a}\rangle}\neq 0$.

The $2$-Sylow in $\mathrm{HN}$ is contained in the centralizer of conjugacy
class $2\mathrm{b}$, which has shape $2^{1+8}_{+}.(A_{5}\wr 2)$. The $E_{2}$
page of the corresponding LHS spectral sequence provides an upper bound of
$\big{|}\operatorname{H}^{4}(\mathrm{HN})_{(2)}\big{|}\leq 2^{6}$. Let
$E=2^{8}\cong\big{(}2^{8}\big{)}^{*}$ and $J=(A_{5}\wr 2)$; then

$\displaystyle\operatorname{H}^{0}(J;E)=\operatorname{H}^{1}(J;E)=\operatorname{H}^{0}\big{(}J;\operatorname{Alt}^{3}(E)\big{)}=0,\qquad\operatorname{H}^{2}(J;E)\cong\mathbf{Z}/2,$

while

$\displaystyle\operatorname{H}^{0}\big{(}J;\operatorname{Alt}^{2}(E)\big{)}\cong\mathbf{Z}/2,\qquad\operatorname{H}^{0}\big{(}J;\operatorname{Alt}^{2}(E)/2\big{)}\cong\mathbf{Z}/2;\qquad\operatorname{H}^{1}\big{(}J;\operatorname{Alt}^{2}(E)/2\big{)}\cong(\mathbf{Z}/2)^{2}.$

Therefore the $E_{2}$ page of the LHS spectral sequence, after localizing at
$2$, reads

$\displaystyle\begin{array}[]{ccccc}2\\\ 2&2^{2}\\\ 0&0&2\\\ 0&0&0&0&0\\\
\mathbf{Z}&0&2&2&2^{2}\end{array}$

As usual, the image of $\operatorname{H}^{4}(A_{5}\wr
2)_{(2)}\to\operatorname{H}^{4}\big{(}2^{1+8}_{+}.(A_{5}\wr 2)\big{)}_{(2)}$
does not intersect $\operatorname{H}^{4}(\mathrm{HN})$: the former is detected
on elements of order $2$, but both conjugacy classes of order $2$ in
$\mathrm{HN}$ meet $2^{1+8}_{+}\subseteq 2^{1+8}_{+}.(A_{5}\wr 2)$. ∎

######  Theorem 8.3.

$\operatorname{H}^{4}\big{(}\mathrm{Th};\mathbf{Z}\big{[}\frac{1}{3}\big{]}\big{)}\cong\mathbf{Z}/8$.

###### Proof.

Lemma 2.2 handles the primes $p\geq 7$. The $5$-Sylow is inside
$5^{1+5}:4S_{4}$, and Lemma 2.3 implies
$\operatorname{H}^{4}\big{(}5^{1+5}:4S_{4}\big{)}_{(5)}=0$. The $3$-Sylow does
not live in any nice maximal subgroups, and so we cannot compute
$\operatorname{H}^{4}(\mathrm{Th})_{(3)}$.

The $2$-Sylow in $\mathrm{Th}$ is contained in the Dempwolff group of shape
$2^{5}\cdot\mathrm{GL}_{5}(2)$. According to Lemma 4.1(3),
$\operatorname{H}^{4}\big{(}2^{5}\cdot\mathrm{GL}_{5}(2)\big{)}_{(2)}=8$, and
the proof of that lemma established that $c_{2}(V)\neq 0$, where $V$ denotes
the $248$-dimensional irrep of $2^{5}\cdot\mathrm{GL}_{5}(2)$. This irrep $V$
extends to the defining $248$-dimensional irrep of $\mathrm{Th}$, and so the
restriction map
$\operatorname{H}^{4}(\mathrm{Th})_{(2)}\to\operatorname{H}^{4}\big{(}2^{5}\cdot\mathrm{GL}_{5}(2)\big{)}_{(2)}$
is nonzero. Since the image of that restriction map is a direct summand, we
must have
$\operatorname{H}^{4}(\mathrm{Th})_{(2)}\cong\operatorname{H}^{4}\big{(}2^{5}\cdot\mathrm{GL}_{5}(2)\big{)}_{(2)}\cong\mathbf{Z}/8$.
∎

###  8.3 Fischer groups

The Fischer groups $\mathrm{Fi}_{22}$, $\mathrm{Fi}_{23}$, and
$\mathrm{Fi}_{24}$ are a “third generation” version of the Mathieu groups
$\mathrm{M}_{22}$, $\mathrm{M}_{23}$, and $\mathrm{M}_{24}$. Specifically, the
$2$-Sylow in $\mathrm{Fi}_{N}$, for $N\in\\{22,23,24\\}$, lives in a subgroup
of shape $2^{\lceil N/2\rceil-1}.\mathrm{M}_{N}$, making the calculation of
$\operatorname{H}^{4}(\mathrm{Fi}_{N})_{(2)}$ systematic. The extension
$2^{\lceil N/2\rceil-1}.\mathrm{M}_{N}$ splits for $N=22$ and does not split
for $N=23$ and $24$.

The $3$-Sylows in all cases are contained in orthogonal groups over
$\mathbf{F}_{3}$. We were able to handle the $3$-parts of
$\operatorname{H}^{4}(G)$ for $G=\mathrm{Fi}_{22}$ and $3\mathrm{Fi}_{22}$,
but not for the larger Fischer groups.

######  Theorem 8.4.

The Fischer groups have the following cohomologies away from the prime $3$:

  1. $1)$

$\operatorname{H}^{4}\big{(}\mathrm{Fi}_{22};\mathbf{Z}\big{[}\frac{1}{3}\big{]}\big{)}=0$;

  2. $2)$

$\operatorname{H}^{4}\big{(}2\mathrm{Fi}_{22};\mathbf{Z}\big{[}\frac{1}{3}\big{]}\big{)}$
has order $2$ or $4$;

  3. $3)$

$\operatorname{H}^{4}\big{(}\mathrm{Fi}_{23};\mathbf{Z}\big{[}\frac{1}{3}\big{]}\big{)}\cong\mathbf{Z}/2$;

  4. $4)$

$\operatorname{H}^{4}\big{(}\mathrm{Fi}_{24}^{\prime};\mathbf{Z}\big{[}\frac{1}{3}\big{]}\big{)}$
has order $2$ or $4$.

###### Proof.

Lemma 2.2 handles all primes $p\geq 5$ except for
$\operatorname{H}^{4}(\mathrm{Fi}_{24}^{\prime})_{(7)}$. But the $7$-Sylow in
$\mathrm{Fi}_{24}^{\prime}$ is inside a copy of Held’s group $\mathrm{He}$,
and $\operatorname{H}^{4}(\mathrm{He})_{(7)}=0$ by Theorem 8.1.

We now inspect the LHS spectral sequences for the extensions $2^{\lceil
N/2\rceil-1}.\mathrm{M}_{N}\subseteq\mathrm{Fi}_{N}$. The three cases are:

$\displaystyle\begin{array}[]{ccccc}\begin{array}[]{ccccc}0\\\ 0&0&2\\\
0&2&0\\\ 0&0&0&0&0\\\ \mathbf{Z}&0&0&12&0\\\
\end{array}&&\begin{array}[]{ccccc}0\\\ 0&2&2\\\ 0&0&0\\\ 0&0&0&0&0\\\
\mathbf{Z}&0&0&0&0\end{array}&&\begin{array}[]{ccccc}0\\\ 0&2&\\\ 0&0&0\\\
0&0&0&0&0\\\ \mathbf{Z}&0&0&0&12\end{array}\\\ \\\
2^{10}:\mathrm{M}_{22}\subseteq\mathrm{Fi}_{22}&&2^{11}.\mathrm{M}_{23}\subseteq\mathrm{Fi}_{23}&&2^{11}.\mathrm{M}_{24}\subseteq\mathrm{Fi}_{24}\end{array}$

The first proves claim (1) and provides the upper bound for claim (2). The
second provides the upper bound for claim (3).

Let $\alpha\in\operatorname{H}^{4}(\mathrm{M}_{24})_{(2)}\cong\mathbf{Z}/4$
denote a generator, and let $\tilde{\alpha}$ denote its pullback to
$2^{11}.\mathrm{M}_{24}$. Consider the conjugacy classes $4\mathrm{b}$ and
$4\mathrm{c}$ in $\mathrm{M}_{24}$. The formula given in [16, Section 3.3]
provides $\alpha|_{\langle 4\mathrm{b}\rangle}=0$ whereas $\alpha|_{\langle
4\mathrm{c}\rangle}$ has order $4$ in $\operatorname{H}^{4}(\langle
4\mathrm{c}\rangle)$. These classes admit preimages in
$2^{11}.\mathrm{M}_{24}$ living in conjugacy classes $8\mathrm{e}$ and
$8\mathrm{a}$ respectively, and so $\tilde{\alpha}|_{8\mathrm{e}}=0$ whereas
$\tilde{\alpha}|_{8\mathrm{a}}$ has order $2$. (The pullback map
$\operatorname{H}^{4}(C_{4})\to\operatorname{H}^{4}(C_{8})$ along a surjection
of cyclic groups $C_{8}\to C_{4}$ has image of order $2$.) Both $8\mathrm{e}$
and $8\mathrm{a}$ fuse in $\mathrm{Fi}_{24}^{\prime}$ to class $8\mathrm{a}$,
and so $\tilde{\alpha}$ cannot be the restriction of a cohomology class on
$\mathrm{Fi}_{24}^{\prime}$. Together with the above spectral sequence, we
find the upper bound in claim (4).

All that remain are the lower bounds. Let $\omega^{\natural}$ denote the
“gauge anomaly” of the Monster CFT, studied in [27]; cf. Section 8.4. The
McKay–Thompson series for class $4\mathrm{b}$ in the Monster group
$\mathrm{M}$ has a nontrivial multiplier (of order $2$), and so
$\omega^{\natural}|_{\langle 4\mathrm{b}\rangle}$ is nonzero. But
$4\mathrm{b}$ meets $2\mathrm{Fi}_{22}\subseteq\mathrm{Fi}_{23}\subseteq
3\mathrm{Fi}_{24}^{\prime}$, and so $\omega^{\natural}$ restricts with order
at least $2$ to all of these groups. ∎

######  Theorem 8.5.

$\operatorname{H}^{4}(\mathrm{Fi}_{22})=\\{0\\}$ and
$\operatorname{H}^{4}(3\mathrm{Fi}_{22})\cong\mathbf{Z}/3$.

Combined with Theorem 8.4(2), we find that
$\operatorname{H}^{4}(6\mathrm{Fi}_{22})$ has order either $6$ or $12$.

###### Proof.

Given Theorem 8.4(1), we must only compute
$\operatorname{H}^{4}(\mathrm{Fi}_{22})_{(3)}$ and
$\operatorname{H}^{4}(3\mathrm{Fi}_{22})_{(3)}$. But the $3$-Sylow in
$\mathrm{Fi}_{22}$ is contained in a maximal subgroup isomorphic to
$\Omega_{7}(3)$, and $\operatorname{H}^{4}(\Omega_{7}(3))_{(3)}=0$ by Lemma
4.4. By Lemma 2.5, the inclusions
$\operatorname{H}^{4}(\Omega_{7}(3))_{(3)}\to\operatorname{H}^{4}(3\Omega_{7}(3))_{(3)}$
and
$\operatorname{H}^{4}(\mathrm{Fi}_{22})_{(3)}\to\operatorname{H}^{4}(3\mathrm{Fi}_{22})_{(3)}$
have the same cokernel; this cokernel is $\mathbf{Z}/3$ by Corollary 4.5. ∎

###  8.4 Monster

The main result of [27] is that $\operatorname{H}^{4}(\mathrm{M})$ contains a
class $\omega^{\natural}$, arising as the gauge anomaly of the Moonshine CFT,
of exact order $24$. It is reasonable to conjecture that $\omega^{\natural}$
generates $\operatorname{H}^{4}(\mathrm{M})$. The calculations in this paper
allow us to come close to proving that conjecture:

######  Theorem 8.6.

$\operatorname{H}^{4}(\mathrm{M})\cong\mathbf{Z}/24\oplus X$, where the
$\mathbf{Z}/24$ summand is generated by $\omega^{\natural}$ and where the
order of $X$ divides $4$.

###### Proof.

The primes $p=11$ and $p\geq 17$ are covered by Lemma 2.2. To calculate
$\operatorname{H}^{4}(\mathrm{M})$, we must study the $p$-Sylows for
$p=2,3,5,7,13$. For these $p$, the $p$-Sylow is contained in the normalizer
$N(p\mathrm{b})$ of an element of conjugacy class $p\mathrm{b}$. These
normalizers are all of shape

$\displaystyle N(p\mathrm{b})=p^{1+m}.J,$

where $m=24/(p-1)$ and $J\subseteq\mathrm{Co}_{1}$. Specifically

$\displaystyle\begin{array}[]{c|c}p&J\\\ \hline\cr 2&\mathrm{Co}_{1}\\\
3&2\mathrm{Suz}.2\\\ 5&2\mathrm{J}_{2}.4\\\ 7&3\times 2S_{7}\\\ 13&3\times
4S_{4}\end{array}$

The extension $p^{1+m}.J$ splits for $p\geq 5$. When $p=3$,
$3^{1+12}.2.\mathrm{Suz}.2$ does not split, but the quotient
$3^{12}:2\mathrm{Suz}.2$ does split. When $p=2$, the quotient
$2^{24}\cdot\mathrm{Co}_{1}$ does not split.

The center of the group $J$ in all cases has order $p-1$, and acts by a
faithful central character on $p^{m}$. Applying Lemma 2.3, we find that
$\operatorname{H}^{i}\big{(}J;\operatorname{H}^{j}\big{(}p^{1+m}\big{)}\big{)}=0$
for $0<j<p$. Combined with the HAP calculation of
$\operatorname{H}^{4}(2\mathrm{J}_{2})$ from Section 6.2, the known value
$\operatorname{H}^{4}(3\times 2S_{7})_{(7)}=0$ (which follows by the methods
of Lemma 2.2), and the trivial result $\operatorname{H}^{4}(3\times
4S_{4})_{(13)}$, we find that
$\operatorname{H}^{4}\big{(}\mathrm{M};\mathbf{Z}\big{[}\frac{1}{6}\big{]}\big{)}=0$.

At the prime $3$, central character considerations imply that the only
potentially nonzero entries of total degree $4$ on the LHS spectral sequence
for $3^{1+12}.2\mathrm{Suz}.2$ are, in the notation of Lemma 3.3:

$\displaystyle\operatorname{H}^{0}\big{(}2\mathrm{Suz}.2;\operatorname{Sym}^{2}\big{(}3^{12}\big{)}\big{)},\qquad\operatorname{H}^{1}\big{(}2\mathrm{Suz}.2;\operatorname{Alt}^{2}\big{(}3^{12}\big{)}_{\omega}\big{)},\qquad\operatorname{H}^{4}(2\mathrm{Suz}.2;\mathbf{Z})_{(3)}.$

The first vanishes because $3^{12}$ is antisymmetrically but not symmetrically
self-dual as a $2.\mathrm{Suz}$ module. Actually, as a $2\mathrm{Suz}.2$
module, $3^{12}$ is not self-dual at all: the symplectic pairing changes by a
sign via the surjection $2\mathrm{Suz}.2\to 2$. The last vanishes by Theorem
6.8.

Therefore
$\operatorname{H}^{1}\big{(}2\mathrm{Suz}.2;\operatorname{Alt}^{2}\big{(}3^{12}\big{)}_{\omega}\big{)}$
gives an upper bound for $\operatorname{H}^{4}(\mathrm{M})_{(3)}$. The group
$2\mathrm{Suz}.2$ is too large for Cohomolo to handle directly, but its
$3$-Sylow-containing maximal subgroup of shape $3^{5}:(\mathrm{M}_{11}\times
2)$ is not, and Cohomolo computes

$\displaystyle\operatorname{H}^{1}\big{(}3^{5}:(\mathrm{M}_{11}\times
2);\operatorname{Alt}^{2}\big{(}3^{12}\big{)}_{\omega}\big{)}\cong 3^{1}.$

This provides an upper bound for
$\operatorname{H}^{1}\big{(}2\mathrm{Suz}.2;\operatorname{Alt}^{2}\big{(}3^{12}\big{)}_{\omega}\big{)}$,
and so

$\displaystyle\operatorname{H}^{4}(\mathrm{M})_{(3)}\leq 3.$

On the other hand, [27, Lemma 3.2.4] shows that
$\operatorname{H}^{4}(\mathrm{M})_{(3)}\geq 3$.

The $p=2$ part of $\operatorname{H}^{4}(\mathrm{M})$ is more subtle.

We first claim that the
$\mathbf{Z}/8\subseteq\operatorname{H}^{4}(\mathrm{M})_{(2)}$ generated by
$\omega^{\natural}$ is a direct summand. Equivalently, we claim that
$\omega^{\natural}$ is not divisible by $2$. Consider the subgroup
$N(3\mathrm{b})=3^{1+12}.2.\mathrm{Suz}.2\subseteq\mathrm{M}$. Since its
quotient $3^{12}:2\mathrm{Suz}.2$ splits, we can find a copy of
$6\mathrm{Suz}\subseteq 6\mathrm{Suz}.2\subseteq\mathrm{M}$. The central
$3\subseteq 6\mathrm{Suz}$ is generated by class $3\mathrm{b}$ by
construction, and the central $2\subseteq 6\mathrm{Suz}\subseteq\mathrm{M}$ is
of class $2\mathrm{b}$. It follows that this $6\mathrm{Suz}$ is (conjugate to)
a subgroup of the normalizer $N(2\mathrm{b})=2^{1+12}\cdot\mathrm{Co}_{1}$,
where it lives over a copy of $3\mathrm{Suz}\subseteq\mathrm{Co}_{1}$. As
observed in the proof of Theorem 6.8, the corresponding
$6\mathrm{Suz}\subseteq 2\mathrm{Co}_{1}$ contains the group $2D_{8}$ used in
[28]; thus the $6\mathrm{Suz}\subseteq\mathrm{M}$ contains a $2D_{8}$ such
that the central element is of class $2\mathrm{b}\in\mathrm{M}$. This is the
$2D_{8}\subseteq\mathrm{M}$ used in [27, Section 3.3] to show that the order
of $\omega^{\natural}$ is divisible by $8$. It follows in particular that the
order of $\omega^{\natural}|_{6\mathrm{Suz}}$ is divisible by $8$. But
$\operatorname{H}^{4}(6\mathrm{Suz})_{(2)}=8$ by Theorem 6.8, and so
$\omega^{\natural}|_{6\mathrm{Suz}}$ is not divisible by $2$, proving the
claim.

The last step of the proof is to study the LHS spectral sequence for the
extension $2^{1+24}\cdot\mathrm{Co}_{1}$:

$\displaystyle\begin{array}[]{ccccc}\operatorname{H}^{0}\big{(}\mathrm{Co}_{1};\operatorname{H}^{4}\big{(}2^{1+24}\big{)}\big{)}\\\
\operatorname{H}^{0}\big{(}\mathrm{Co}_{1};\operatorname{H}^{3}\big{(}2^{1+24}\big{)}\big{)}&\operatorname{H}^{1}\big{(}\mathrm{Co}_{1};\operatorname{H}^{3}\big{(}2^{1+24}\big{)}\big{)}\\\
\operatorname{H}^{0}\big{(}\mathrm{Co}_{1};\operatorname{H}^{2}\big{(}2^{1+24}\big{)}\big{)}&\operatorname{H}^{1}\big{(}\mathrm{Co}_{1};\operatorname{H}^{2}\big{(}2^{1+24}\big{)}\big{)}&\operatorname{H}^{2}\big{(}\mathrm{Co}_{1};\operatorname{H}^{2}\big{(}2^{1+24}\big{)}\big{)}\\\
0&0&0&0&0\\\ \mathbf{Z}&0&0&2&4\end{array}$

The $2$ in the bottom row is $\operatorname{H}^{3}(\mathrm{Co}_{1})$, and the
$4$ in the bottom row is $\operatorname{H}^{4}(\mathrm{Co}_{1})_{(2)}$; the
latter result is due to [28].

From Section 3.3, we know

$\displaystyle\operatorname{H}^{2}\big{(}2^{1+24}\big{)}\cong
2^{24},\qquad\operatorname{H}^{3}\big{(}2^{1+24}\big{)}\cong\operatorname{Alt}^{2}\big{(}2^{24}\big{)}/2\cong
2^{275},$

while

$\displaystyle\operatorname{H}^{4}\big{(}2^{1+24}\big{)}\cong
2^{24}.\operatorname{Alt}^{2}\big{(}2^{24}\big{)}.\big{(}\operatorname{Alt}^{3}\big{(}2^{24}\big{)}/2^{24}\big{)}.2\cong
2^{2300}.2$

has exponent $4$.

Without much difficulty, GAP can compute

$\displaystyle\operatorname{H}^{0}\big{(}\mathrm{Co}_{1};2^{24}\big{)}=\operatorname{H}^{0}\big{(}\mathrm{Co}_{1};\operatorname{Alt}^{2}\big{(}2^{24}\big{)}/2\big{)}=\operatorname{H}^{0}\big{(}\mathrm{Co}_{1};\operatorname{Alt}^{3}\big{(}2^{24}\big{)}\big{)}=0.$

and

$\displaystyle\operatorname{H}^{0}\big{(}\mathrm{Co}_{1};\operatorname{Alt}^{2}\big{(}2^{24}\big{)}\big{)}\cong\mathbf{Z}/2.$

The groups $\operatorname{H}^{i}\big{(}\mathrm{Co}_{1};2^{24}\big{)}$ for
$i=1,2$ were calculated by Derek Holt and reported in [26, Lemma 1.8.8]. They
are

$\displaystyle\operatorname{H}^{1}\big{(}\mathrm{Co}_{1};2^{24}\big{)}=0,\qquad\operatorname{H}^{2}\big{(}\mathrm{Co}_{1};2^{24}\big{)}\cong\mathbf{Z}/2.$

A presentation for $\mathrm{Co}_{1}$ is given in [35]. Using it, [27, Section
3.5] calculates

$\displaystyle\operatorname{H}^{1}\big{(}\mathrm{Co}_{1};\operatorname{Alt}^{2}\big{(}2^{24}\big{)}/2\big{)}\cong\mathbf{Z}/2.$

These calculations almost fill in the $E_{2}$ page. The remaining missing
ingredient is
$E_{2}^{04}=\operatorname{H}^{0}\big{(}\mathrm{Co}_{1};\operatorname{H}^{4}\big{(}2^{1+24}\big{)}\big{)}$.
Using the above calculations together with the long exact sequence for
cohomology with values in an extension, one finds:

$\displaystyle\operatorname{H}^{0}\big{(}\mathrm{Co}_{1};2^{24}.\operatorname{Alt}^{2}\big{(}2^{24}\big{)}.\big{(}\operatorname{Alt}^{3}\big{(}2^{24}\big{)}/2^{24}\big{)}\big{)}=\operatorname{H}^{0}\big{(}\mathrm{Co}_{1};2^{2300}\big{)}\cong\mathbf{Z}/2.$

It follows that
$\operatorname{H}^{0}\big{(}\mathrm{Co}_{1};\operatorname{H}^{4}\big{(}2^{1+24}\big{)}\big{)}$
is isomorphic to either $\mathbf{Z}/2$ or $\mathbf{Z}/4$. We suspect the
former, but were unable to compute it.

Although we do not know whether $E_{2}^{04}=2$ or $4$, we claim that
$E_{\infty}^{04}=2$. To prove this we quote two facts. First, according to
[27, Section 3.5], $\omega^{\natural}|_{2^{1+24}}$ has exact order $2$, and so
provides an element of order $2$ in $E_{\infty}^{04}$. Second, we showed above
that $\omega^{\natural}$ is not divisible by $2$ in
$\operatorname{H}^{4}(\mathrm{M})$. Since the map
$\operatorname{H}^{4}(\mathrm{M})\to\operatorname{H}^{4}\big{(}2^{1+24}.\mathrm{Co}_{1}\big{)}$
is an inclusion onto a direct summand, it follows that
$\omega^{\natural}|_{2^{1+24}.\mathrm{Co}_{1}}$ is not divisible by $2$. But
$\operatorname{H}^{4}\big{(}2^{1+24}.\mathrm{Co}_{1}\big{)}$ surjects onto
$E_{\infty}^{04}$, and sends $\omega^{\natural}$ to a nonzero value. So the
image of $\omega^{\natural}$ in $E_{\infty}^{04}$ cannot be divisible by $2$,
proving that $E_{\infty}^{04}\neq 4$.

All together, we find an $E_{\infty}$ page of the following form

$\displaystyle\begin{array}[]{ccccc}2\\\ 0&\leq 2\\\ 0&0&\leq 2\\\
0&0&0&0&0\\\ \mathbf{Z}&0&0&2&4\end{array}$

It follows that $\operatorname{H}^{4}(\mathrm{M})_{(2)}$ has order at most
$32$, completing the proof. ∎

### Acknowledgements

This project began when John Duncan asked us to compute
$\operatorname{H}^{4}(\mathrm{O^{\prime}N};\mathbf{Z})$. The referees provided
detailed and valuable corrections. We also thank Graham Ellis, Jesper Grodal,
Geoff Mason, and Robert Wilson for numerous comments and hints about finite
groups and cohomology. This work was supported by the US NSF grant DMS-1510444
and by the Government of Canada through the Department of Innovation, Science
and Economic Development Canada and by the Province of Ontario through the
Ministry of Research, Innovation and Science.

## References

  * [1] Adem A., Milgram R.J., The subgroup structure and mod $2$ cohomology of O’Nan’s sporadic simple group, J. Algebra 176 (1995), 288–315.
  * [2] Adem A., Milgram R.J., Cohomology of finite groups, 2nd ed., Grundlehren der Mathematischen Wissenschaften, Vol. 309, Springer-Verlag, Berlin, 2004\.
  * [3] Cartan H., Eilenberg S., Homological algebra, Princeton University Press, Princeton, N. J., 1956.
  * [4] Cheng M.C.N., de Lange P., Whalen D.P.Z., Generalised umbral moonshine, SIGMA 15 (2019), 014, 27 pages, arXiv:1608.07835.
  * [5] Cohen A.M., Wales D.B., Finite subgroups of $F_{4}({\bf C})$ and $E_{6}({\bf C})$, Proc. London Math. Soc. 74 (1997), 105–150.
  * [6] Conway J.H., Curtis R.T., Norton S.P., Parker R.A., Wilson R.A., Atlas of finite groups: maximal subgroups and ordinary characters for simple groups (with computational assistance from J.G. Thackray), Oxford University Press, Eynsham, 1985.
  * [7] Conway J.H., Norton S.P., Monstrous moonshine, Bull. London Math. Soc. 11 (1979), 308–339.
  * [8] Curtis M., Wiederhold A., Williams B., Normalizers of maximal tori, in Localization in Group Theory and Homotopy Theory, and Related Topics (Sympos., Battelle Seattle Res. Center, Seattle, Wash., 1974), Lecture Notes in Math., Vol. 418, Springer, Berlin, 1974, 31–47.
  * [9] Dempwolff U., On extensions of an elementary abelian group of order $2^{5}$ by ${\rm GL}(5,2)$, Rend. Sem. Mat. Univ. Padova 48 (1972), 359–364.
  * [10] Dempwolff U., On the second cohomology of ${\rm GL}(n,2)$, J. Austral. Math. Soc. 16 (1973), 207–209.
  * [11] Duncan J.F., Super-moonshine for Conway’s largest sporadic group, Duke Math. J. 139 (2007), 255–315, arXiv:math.RT/0502267.
  * [12] Duncan J.F.R., Vertex operators, and three sporadic groups, Ph.D. Thesis, Yale University, 2006.
  * [13] Duncan J.F.R., Mertens M.H., Ono K., O’Nan moonshine and arithmetic, arXiv:1702.03516.
  * [14] Dutour Sikirić M., Ellis G., Wythoff polytopes and low-dimensional homology of Mathieu groups, J. Algebra 322 (2009), 4143–4150, arXiv:0812.4291.
  * [15] Frame J.S., The characters of the Weyl group $E_{8}$, in Computational Problems in Abstract Algebra (Proc. Conf., Oxford, 1967), Pergamon, Oxford, 1970, 111–130.
  * [16] Gaberdiel M.R., Persson D., Ronellenfitsch H., Volpato R., Generalized Mathieu Moonshine, Commun. Number Theory Phys. 7 (2013), 145–223, arXiv:1211.7074.
  * [17] Gannon T., Much ado about Mathieu, Adv. Math. 301 (2016), 322–358, arXiv:1211.5531.
  * [18] Ganter N., Hecke operators in equivariant elliptic cohomology and generalized Moonshine, in Groups and Symmetries, CRM Proc. Lecture Notes, Vol. 47, Amer. Math. Soc., Providence, RI, 2009, 173–209, arXiv:0706.2898.
  * [19] Green D.J., On the cohomology of the sporadic simple group $J_{4}$, Math. Proc. Cambridge Philos. Soc. 113 (1993), 253–266.
  * [20] Griess Jr R.L., Schur multipliers of the known finite simple groups, Ph.D. Thesis, The University of Chicago, 1971.
  * [21] Griess Jr. R.L., Automorphisms of extra special groups and nonvanishing degree $2$ cohomology, Pacific J. Math. 48 (1973), 403–422.
  * [22] Griess Jr. R.L., On a subgroup of order $2^{15}\mid GL(5,2)\mid$ in $E_{8}({\bf C}),$ the Dempwolff group and ${\rm Aut}(D_{8}\circ D_{8}\circ D_{8})$, J. Algebra 40 (1976), 271–279.
  * [23] Grodal J., Low dimensional homology of finite groups of Lie type, away from the characteristic, unpublished.
  * [24] Henriques A., The classification of chiral WZW models by $H^{4}_{+}(BG,{\mathbb{Z}})$, in Lie Algebras, Vertex Operator Algebras, and Related Topics, Contemp. Math., Vol. 695, Amer. Math. Soc., Providence, RI, 2017, 99–121, arXiv:1602.02968.
  * [25] Hochschild G., Serre J.-P., Cohomology of group extensions, Trans. Amer. Math. Soc. 74 (1953), 110–134.
  * [26] Ivanov A.A., The Monster group and Majorana involutions, Cambridge Tracts in Mathematics, Vol. 176, Cambridge University Press, Cambridge, 2009\.
  * [27] Johnson-Freyd T., The moonshine anomaly, Comm. Math. Phys. 365 (2019), 943–970, arXiv:1707.08388.
  * [28] Johnson-Freyd T., Treumann D., $\mathrm{H}^{4}(\mathrm{Co}_{0};\mathbf{Z})=\mathbf{Z}/24$, Int. Math. Res. Not., to appear, arXiv:1707.07587.
  * [29] Lyons R., Evidence for a new finite simple group, J. Algebra 20 (1972), 540–569.
  * [30] Mason G., Reed–Muller codes, the fourth cohomology group of a finite group, and the $\beta$-invariant, J. Algebra 312 (2007), 218–227.
  * [31] Milgram R.J., On the geometry and cohomology of the simple groups $G_{2}(q)$ and ${}^{3}\\!D_{4}(q)$. II, in Group Representations: Cohomology, Group Actions and Topology (Seattle, WA, 1996), Proc. Sympos. Pure Math., Vol. 63, Amer. Math. Soc., Providence, RI, 1998, 397–418.
  * [32] Milgram R.J., The cohomology of the Mathieu group $M_{23}$, J. Group Theory 3 (2000), 7–26.
  * [33] Serre J.-P., Homologie singulière des espaces fibrés. Applications, Ann. of Math. 54 (1951), 425–505.
  * [34] Smith P.E., A simple subgroup of $M?$ and $E_{8}(3)$, Bull. London Math. Soc. 8 (1976), 161–165.
  * [35] Soicher L.H., Presentations for Conway’s group ${\rm Co}_{1}$, Math. Proc. Cambridge Philos. Soc. 102 (1987), 1–3.
  * [36] Thomas C.B., Characteristic classes and the cohomology of finite groups, Cambridge Studies in Advanced Mathematics, Vol. 9, Cambridge University Press, Cambridge, 1987.
  * [37] Thomas C.B., Characteristic classes and $2$-modular representations of some sporadic simple groups, in Algebraic Topology (Evanston, IL, 1988), Contemp. Math., Vol. 96, Amer. Math. Soc., Providence, RI, 1989, 303–318.
  * [38] Thomas C.B., Characteristic classes and $2$-modular representations for some sporadic simple groups. II, in Algebraic Topology Poznań 1989, Lecture Notes in Math., Vol. 1474, Springer, Berlin, 1991, 371–381.
  * [39] Thompson J.G., A conjugacy theorem for $E_{8}$, J. Algebra 38 (1976), 525–530.
  * [40] Tits J., Normalisateurs de tores. I. Groupes de Coxeter étendus, J. Algebra 4 (1966), 96–116.
  * [41] Weibel C.A., An introduction to homological algebra, Cambridge Studies in Advanced Mathematics, Vol. 38, Cambridge University Press, Cambridge, 1994\.
  * [42] Wilson R., Walsh P., Tripp J., Suleiman I., Parker R., Norton S., Nickerson S., Linton S., Bray J., Abbott R., ATLAS of finite group representations, Version 3, available at http://brauer.maths.qmul.ac.uk/Atlas/v3/.
  * [43] Wilson R.A., The complex Leech lattice and maximal subgroups of the Suzuki group, J. Algebra 84 (1983), 151–188.
  * [44] Wilson R.A., Maximal subgroups of sporadic groups, in Finite Simple Groups: Thirty Years of the Atlas and Beyond,Contemp. Math., Vol. 694, Amer. Math. Soc., Providence, RI, 2017, 57–72, arXiv:1701.02095.


# Object Importance Estimation using Counterfactual Reasoning for Intelligent
Driving

Pranay Gupta1, Abhijat Biswas1, Henny Admoni1, David Held1 1Pranay Gupta,
Abhijat Biswas, Henny Admoni and David Held are with the Robotics Institute,
Carnegie Mellon University, Pittsburgh, PA, USA<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The ability to identify important objects in a complex and dynamic driving
environment is essential for autonomous driving agents to make safe and
efficient driving decisions. It also helps assistive driving systems decide
when to alert drivers. We tackle object importance estimation in a data-driven
fashion and introduce HOIST - Human-annotated Object Importance in Simulated
Traffic. HOIST contains driving scenarios with human-annotated importance
labels for vehicles and pedestrians. We additionally propose a novel approach
that relies on counterfactual reasoning to estimate an object’s importance. We
generate counterfactual scenarios by modifying the motion of objects and
ascribe importance based on how the modifications affect the ego vehicle’s
driving. Our approach outperforms strong baselines for the task of object
importance estimation on HOIST. We also perform ablation studies to justify
our design choices and show the significance of the different components of
our proposed approach.

##  I Introduction

Assistive and autonomous driving agents need to reason about objects within
the driving scene to make safe driving decisions. However, not all objects are
equally important. The ability to identify important objects would allow
autonomous driving systems to perform efficient path planning and resource
management by reducing the number of objects that require tracking and
modeling. Moreover, prior work has also shown that importance estimation can
help reduce false positive alerts in assistive driving systems that detect
driver inattention and preemptively warn the driver (e.g., detecting a
jaywalking pedestrian and highlighting them on a HUD) [1]. Recent work has
also shown improvement in autonomous driving agents’ performance when
contrastively trained to de-prioritize unimportant objects [2]. Thus, in this
work, we focus on the task of important object identification in the context
of driving.

In prior works [3, 4, 5, 6, 7] researchers have tried to identify salient
regions in a driving scene by modeling driver attention through eye-gaze
fixations. However, in the context of driving, the correlation between gaze
and attention is questionable [8] (see Sec. 2) as the driver may not always
look at important objects due to distractions [9], inexperience [10], or
paying attention via peripheral vision [11].

(a) Red boxes indicate vehicles important for making safe driving decisions by
the ego vehicle.

(b) The blue car is important since its removal would cause the ego vehicle to
speed up.

(c) The red vehicle is important since its changing lanes would risk a
collision with the ego vehicle.

Figure 1: We perform counterfactual reasoning to identify important vehicles
in a driving scenario. We modify the motion of vehicles, and ascribe
importance based on how the modification affects the ego vehicle’s driving.

Another line of work [12, 13, 14] tries to identify the importance of objects
within the scene directly. However, the definition of “important objects” used
in that work is limited to objects which influence the driving behavior of the
ego vehicle. We extend the definition of “important objects” to include all
the objects that a driver should be aware of, in order to drive the ego
vehicle safely (Sec. III-B) i.e., all objects necessary to maintain sufficient
situational awareness [15]. This includes objects that both currently
influence and _can potentially influence_ the driving decisions of the ego
vehicle. In addition to enhancing autonomous and assistive driving systems,
identifying objects important according to our definition allows for the
development of situational awareness support systems [1].

Moreover, the object-importance datasets proposed in prior works [12, 13] are
unimodal (only ego-view RGB videos) and do not benefit from multiple sensor
modalities (multi-view RGB cameras, LiDAR) for object detection and importance
estimation; further, these datasets are not publicly accessible. To address
these challenges, we introduce HOIST: Human-annotated Object Importance in
Simulated Traffic. HOIST consists of driving scenarios from representative
routes of the Longest6 benchmark [16] with human-annotated object importance.
While prior works [12, 13, 14] obtain annotations using ego-view driving
videos, we collect annotations using bird’s-eye-view RGB videos of the driving
scene, as the bird’s-eye-view offers more context regarding the driving
scenario as compared to the front-view video. In addition to the RGB videos,
HOIST also contains data from all sensor modalities and configurations
supported by CARLA [17] (e.g., RGB Camera, LIDAR, GPS).

Our new dataset enables our second contribution: a novel approach for object
importance estimation. We propose to determine an object’s importance by
answering counterfactual questions about those objects. Specifically, we
modify the motion of objects in the scene and generate counterfactual
scenarios. We gauge the change in the ego vehicle’s driving under the
counterfactual (estimated using an autonomous driving model [18]) to estimate
each object’s importance.

We evaluate our novel approach on the task of object importance estimation,
using the HOIST dataset . The results highlight the effectiveness of our
approach in identifying important objects. We also perform extensive ablation
studies, to validate our design choices and analyze the significance of the
different components of our approach. Furthermore, since our approach relies
on autonomous driving agents to predict the ego vehicle’s behavior, analysing
our failure cases can help identify driving scenarios where the autonomous
driving agents behave unexpectedly.

The main contributions of our work are:

  * •

HOIST, a novel dataset containing simulated driving scenarios with human-
labeled important objects, which influence or can potentially influence the
ego vehicle’s driving. In addition to RGB videos, HOIST provides access to
data from all the sensors configurations offered by CARLA [17].

  * •

A novel approach that employs counterfactual reasoning to estimate object
importance in driving scenarios. We modify the motion of objects to generate
counterfactual scenes and gauge the change in the ego vehicle’s behavior to
ascribe importance.

  * •

Benchmarking and ablation experiments on HOIST that substantiate the
effectiveness of our approach in detecting important objects in a driving
scene. Our code and dataset will be available on our project
page111https://vehicle-importance.github.io.

##  II Related Works

Object Importance Estimation in on-road driving: The task of identifying
important objects in the context of driving has garnered increasing attention
from researchers. Prior approaches [12, 13, 19] learn to predict object
importance using end-to-end models, leveraging the ego vehicle’s goal [12],
interactions between objects [13], or the ego vehicle’s predicted behavior
[14] to identify important objects. Since manual annotation is time intensive,
Li et al. also performs semi-supervised learning by generating ranking based
pseudo-labels for the unlabeled data. Instead of learning end-to-end models
for estimating object importance, our approach relies on counterfactual
reasoning.

Another line of works [19, 20], performs causal inference to identify risky
objects. Similar to our approach, they remove objects and verify whether the
removal produces a change in the ego vehicle’s driving decision. While these
works only consider removal, we estimate importance by perturbing other
dynamic attributes of the objects. A third category of works attributes
importance to objects using attention weights obtained from attention-based
models trained for a different task, such as trajectory forecasting [21, 22,
23], or end-to-end driving [24, 25].

Driver’s Attention Prediction Methods: While driving, humans can selectively
attend to relevant objects in the scene [26, 27]. Hence, prior works [3, 4, 5]
tried to identify salient regions within a dynamic traffic scene by studying
the driver’s attention. Here, the driver’s eye gaze serves as a proxy for the
driver’s attention. For example, Palazzi et al. [3] proposes to use a multi-
stream 3D convolutional network that leverages RGB frames, optical flow, and
segmentation maps to predict the driver attention heatmap. Baee et al. [5]
models eye-gaze fixations as a sequence of states and actions and proposes
maximum entropy deep inverse reinforcement learning [28] to predict maximally
rewarding fixation locations. In our work, we directly estimate the importance
of objects within the scene, rather than detecting the driver’s attention
heatmaps.

Driver’s Attention Prediction Datasets: Typically, methods for driver
attention prediction leverage large-scale driver attention datasets comprising
ego-centric driving videos with the corresponding driver gaze [4]. DReyeVE [3]
was the first large-scale driver attention dataset recorded using eye-tracking
glasses in a car. Recent driver attention datasets [7, 5] have focused on
accident scenarios. Pal et al. [6] argues that driver gaze is insufficient in
describing everything a driver should attend to, and proposes to use ground
truth gaze map combined with segmentation masks of all the objects within the
scene. Unlike these datasets, HOIST consists of data from a host of sensor
observations (RGB cameras, LIDAR, GPS) along with explicitly annotated object
importance, rather than using the driver gaze as a proxy for importance.

##  III HOIST: Human-annotated Object Importance in Simulated Traffic

We consider the problem of object importance estimation in the context of
driving. We define “important objects” as the objects that the driver should
watch out for in order to make safe driving decisions. Prior object importance
estimation datasets [12, 14] limit important objects to only those which
currently influence the ego vehicle’s driving decisions. Moreover, they only
contain single-view RGB videos and are not publicly accessible. Transcending
these challenges, we curate HOIST: Human-annotated Object Importance in
Simulated Traffic. HOIST is a new dataset of driving scenarios with human
annotations of object importance. In addition to RGB videos, HOIST also
provides access to data from a host of different types of sensors, including
multi-view cameras and LIDAR. These different sensors will allow users to use
this dataset to evaluate different methods of object importance estimation, as
discussed in Sec. IV.

###  III-A Dataset Curation

We obtain the driving videos for HOIST from the Longest6 benchmark [16], which
consists of routes from the CARLA [17] driving simulator. The Longest6
benchmark was introduced as a substitute for the official CARLA evaluation
routes to overcome the compute restrictions set by CARLA. It consists of the
six longest routes from each of the six towns in CARLA, resulting in a total
of 36 routes. It preserves the most challenging aspects of the CARLA
evaluation routes in terms of traffic density, weather conditions, and
adversarial traffic scenarios. For our dataset, we select six representative
routes (one from each town) of the Longest6 benchmark. We drive the ego
vehicle through the six selected routes using a rule-based-expert [16],
recording simulation data along each route.

We extract the bird’s-eye-view videos from the simulation recording of each
route and split them into smaller 13 second videos. Unfortunately, the routes
in the Longest6 benchmark majorly involve driving in a straight lane. Driving
in straight lanes leads to trivial cases for object importance estimation as
the objects closely leading and following the ego vehicle are generally deemed
important. To ensure that we include more diverse driving scenarios, we limit
the number of videos with straight lanes and include videos where the ego
vehicle is undergoing complex driving maneuvers like crossing intersections,
turning, changing lanes, or stopping for pedestrians. We end up with a total
of 409 videos.

###  III-B Human Annotations

We hire annotators on Prolific222https://www.prolific.co/ to collect
annotations for the videos obtained in the previous step (III-A). The
annotators were first shown the bird’s-eye-view RGB videos and were then asked
to draw bounding boxes around important objects in the last frame of the
video. In our instructions, we defined important objects as the ones the
annotators would watch out for in order to drive the ego vehicle safely if
they were to take control of the ego vehicle from the point at which the video
ended. We instructed the annotators to focus on only two types of objects:
vehicles and pedestrians 333Details about the interface and the prompts used
to obtain the annotations are available on our project page : https://vehicle-
importance.github.io.

Our annotators reported that they are more than 18 years of age and have a
valid US state-issued driving license. Each annotator did three practice
rounds, two at the beginning and one at the end. These practice rounds were
straightforward and were used to exclude annotators whose annotations
demonstrated a lack of comprehension of the task. We hired 188 annotators and
each annotated 15 videos. To mitigate the bias due to the driving habits of
individual annotators, each video was annotated by at least 5 different
annotators.

###  III-C Dataset Statistics

The HOIST dataset consists of a total of 409 videos with 642 unique human-
annotated important objects. Out of these 409 videos, 8.8% of the videos have
$0$ important objects, 44.9% have $1$ object, 32.2% have $2$ important
objects, and 13.9% have more than $2$ important objects. In this work, we
focus on identifying important vehicles and pedestrians. Out of the 642 total
important objects, 597 are vehicles and 45 are pedestrians.

##  IV Estimating Object Importance

In this section, we describe our novel method for estimating the importance of
objects for making safe control decisions for the ego vehicle. Because the
movement patterns of vehicles and pedestrians are different from each other,
we propose to use different approaches to estimate their importance. For
estimating the importance of vehicles, we perform counterfactual reasoning
about the vehicles (Sec IV-A). For pedestrians, we find their proximity to the
ego vehicle to be a very strong cue for importance estimation (Sec IV-B). We
explain both of these approaches in detail in the following sections.

(a) Original

(b) Counterfactual

Figure 3: The removal score is the L2 distance between the corresponding
waypoints of the true trajectory (shown in a) and the predicted trajectory
after removing Car 0 (shown in b).

###  IV-A Vehicle Importance Estimation

For estimating vehicle importance, our proposed approach performs
counterfactual reasoning. We generate counterfactual scenarios by modifying
the motion of the non-ego vehicles; we ascribe importance based on how the
modifications affect the ego vehicle’s driving. We modify the ego vehicle’s
sensor observations to simulate modifications in the motion of the non-ego
vehicles. We then compare the changes in the ego vehicle’s trajectory under
the counterfactual. If the ego vehicle’s trajectory has significantly changed
under the counterfactual, then the vehicle is deemed important (details
below). We also perform collision estimation under the counterfactuals to
ascribe importance.

We leverage autonomous driving systems [18] to predict the ego vehicle’s
trajectory. Typical autonomous driving systems [29, 30, 31, 18, 16] perceive a
scene through various sensor modalities (e.g. multiview RGB cameras and LIDAR)
and predict the future trajectory for the ego vehicle. A trajectory is
represented by a set of waypoints. We query such a system with the true and
the counterfactual sensor observations and compare the change in the predicted
waypoints. We base our importance predictions on two types of counterfactual
scenarios:

Original

Counterfactual

(a) Hard-Stop perturbation.

Original

Counterfactual

(b) Speed-Up perturbation.

Original

Counterfactual

(c) Lane-Change perturbation.

Figure 5: The velocity perturbation score assesses potential collision, when
the non-ego vehicle undergoes a velocity perturbation. A collision occurs when
the corresponding waypoints of the ego vehicle and the non-ego vehicle
trajectories are within a threshold. The collisions occur at the $k^{th}$
waypoint. The trajectories of the ego vehicle and the non-ego vehicle are
shown in in orange and green, respectively.

Removal: To check if a vehicle affects the ego vehicle’s driving decisions, we
verify whether removing that vehicle from the scene would cause the ego
vehicle to change its trajectory. We remove vehicles by modifying the sensor
observations input to the autonomous driving system (See Section IV-C for
details). Consider the example shown in Fig. 3. The importance of an object is
calculated as the L2 distance between the waypoints predicted using the true
sensor observations (Fig. 3(a)) and the counterfactual observations (Fig.
3(b)). We call this the Removal Score ($\mathcal{RS}$). Let $w_{ego,k}$ be the
$k$th predicted waypoint of the ego vehicle with the true sensor observations,
and let $\hat{w}_{ego,k}^{i}$ be the $k$th predicted waypoint of the ego
vehicle under the counterfactual observation generated by removing the $i$th
vehicle. The Removal Score $\mathcal{RS}_{i}$ for the $i$th vehicle is

$\mathcal{RS}_{i}=\sum_{k\in\\{1,2,\ldots
K\\}}||w_{ego,k}-\hat{w}_{ego,k}^{i}||^{2}$ (1)

where $K$ is the total number of predicted waypoints.

Velocity Perturbations: The Removal Score identifies vehicles that affect the
ego vehicle’s trajectory in the current state. However, a vehicle can become a
safety hazard if it suddenly changes its speed or direction. Thus, we check
whether changing the velocity of a particular vehicle would make it
susceptible to collisions with the ego vehicle (see Fig. 5). We simulate
velocity changes by perturbing the trajectories of the vehicles. Similar to
the ego vehicle, we obtain the trajectories for non-ego vehicles using an
autonomous driving agent. Recent methods for autonomous driving predict the
trajectories of non-ego vehicles in the scene either by using deep models [18]
or by projecting it using a constant velocity model [30], where the constant
velocity is calculated using a moving average over the past timesteps. We
consider three possible velocity perturbations. To identify vehicles that
would become safety critical if they suddenly stopped, we generate a
counterfactual scenario in which the vehicle makes a hard stop. To simulate
hard stopping, we collapse all the waypoints of the vehicle into the first
waypoint (See Fig. 5(a)). Next, we identify the vehicles that would become
safety critical if they suddenly sped up. For this, we generate a
counterfactual scenario where we increase the vehicle’s velocity. We simulate
this by increasing the distance between consecutive waypoints of the vehicle’s
trajectory (See Fig. 5(b)). Finally, to identify vehicles that would become
safety critical if they suddenly changed lanes, we simulate lane change
maneuvers for each object. We perturb the trajectories of the vehicles such
that they first perform a 45-degree lane shift into an adjacent lane (left or
right) and then follow a straight line (See Fig. 5(c)).

In addition to the vehicles in the scene, the ego vehicle might also have to
deviate from its predicted trajectory under emergencies. Therefore, as a
defensive driving measure, we also perturb the ego vehicle’s trajectory
according to these three velocity perturbations and similarly measure
collisions with other vehicles under this counterfactual to estimate vehicle
importance.

We consider a vehicle important due to velocity-based perturbations if it is
predicted to collide with the ego vehicle. A collision is predicted if a
waypoint from any of the non-ego vehicle’s trajectories (predicted or
perturbed) is within the safety threshold distance of a waypoint from the ego
vehicle’s trajectories (predicted or perturbed). Moreover, a predicted
collision that happens sooner is considered more important than a predicted
collision that happens later. Hence, we score vehicle importance based on how
soon they are predicted to collide with the ego vehicle. Thus, the Velocity
Perturbation Score ($\mathcal{VS}_{i}$) for the $i$th object is given by

$d_{i,k}=||w_{i,k}-w_{ego,k}||^{2}$
$k^{*}_{i}=\operatorname*{arg\,min}\limits_{k\in[0,\cdots,K-1]}d_{i,k}$

$\mathcal{VS}_{i}=\begin{cases}-k^{*}_{i}&\text{if $d_{i,k^{*}_{i}}<\tau$}\\\
-K&\text{otherwise}\end{cases}$ (2)

Here, $w_{ego,k}$ is the $k$th waypoint of the predicted (or perturbed) ego
vehicle’s trajectory. Similarly, $w_{i,k}$ is the $k$th waypoint of the $i$th
non ego vehicle’s trajectory. $K$ is the total number of waypoints;
$k_{i}^{*}$ is the index of the waypoint at which the ego vehicle’s trajectory
is predicted to be closest to the $i$th vehicle. If this distance is less than
the safety threshold $\tau$, then we assign to the vehicle a score of
$-k_{i}^{*}$, i.e. a closer index of collision corresponds to a higher score.

Combining the scores: The final importance score $\mathcal{IS}_{i}$ for a
vehicle $i$ is given by the maximum of the two scores:

$\mathcal{IS}_{i}=\max(\mathcal{RS}_{i},\mathcal{VS}_{i})$ (3)

Here, $\mathcal{RS}_{i}$ is the removal score for the $i$th vehicle, and
$\mathcal{VS}_{i}$ is the velocity perturbation score for the $i$th vehicle.
Both scores are normalized between $0$ and $1$ using their maximum and minimum
values across the dataset.

###  IV-B Pedestrian Importance Estimation

In the HOIST dataset, the movement patterns of pedestrians are significantly
different from vehicles. Vehicle movement is confined to the roads, and they
typically adhere to traffic regulations. Pedestrians have more freedom in
their movement. They sometimes violate traffic regulations and can even dart
out onto the roads. Hence, the mechanisms for estimating importance of
vehicles do not necessarily apply to pedestrians. Since pedestrian movement is
less constrained, their distance to the ego vehicle is a strong indication for
importance. Hence, we ascribe importance to pedestrians based on their
L2-distance from the ego vehicle. The importance score ($\mathcal{PS}_{p}$)
for a pedestrian $p$ if given by:

$\mathcal{PS}_{p}=-||x_{p}-x_{ego}||^{2}$ (4)

Here, $x_{p}$ and $x_{ego}$ are the locations of the pedestrian and the ego
vehicle, respectively.

###  IV-C Implementation Details

We use LAV [18] to predict the ego vehicle trajectory in our experiments. LAV
uses LIDAR point clouds as input for predicting the trajectories of the
vehicles. Thus, the counterfactual observations for the Removal Score
($\mathcal{RS}$) are created by removing the points within the 3D bounding box
of the concerned objects. We obtain ground-truth bounding boxes from the
simulator, although a 3D bounding box detection module [32] can also be used.
In addition to the ego vehicle trajectory, LAV can also predict the
trajectories of the non-ego vehicles in the scene. However, LAV ignores the
objects behind the ego vehicle. We use a constant velocity model for
predicting the trajectories of the vehicles ignored by LAV. We average the
speed and direction of the vehicles over $5$ previous timesteps. Each
trajectory predicted by LAV consists of $K$ waypoints; we set $K=20$.

##  V Experiments

We evaluate our approach for important object identification against three
baselines using the HOIST dataset. Human annotations are used as ground truth.
An object is considered important in the ground truth data if it is marked
important by at least $\theta_{1}$ annotators. Objects marked important by
fewer than $\theta_{2}$ annotators are considered unimportant. Objects marked
important by fewer than $\theta_{1}$ but no less than $\theta_{2}$ annotators
are ignored, i.e., approaches are not penalized for identifying them
incorrectly. We do so to make our evaluation robust towards outliers in the
dataset. For our experiments, we set $\theta_{1}=3$ and $\theta_{2}=2$.

(a) Ground-truth (human annotation)

(b) Inverse-Distance

(c) PlanT

(d) Our approach

Figure 7: Qualitative comparison of our approach with the baselines. The
objects are marked as important based on the optimal threshold for the F1
Score for each approach.

###  V-A Comparison with Baselines

We compare our method with the following baselines for estimating object
importance:

Everything Important: All objects within the scene are considered equally
important. We compare our approach to this baseline to highlight the need for
the important object identification task.

Inverse Distance: Objects are deemed important based on their proximity to the
ego vehicle. Specifically, the importance score is the negative L2 distance
between the object and the ego vehicle.

PlanT [24]: PlanT is a transformer-based approach for planning in autonomous
driving. However, PlanT also outputs relevance scores for vehicles in the form
of learned attention weights. We use PlanT’s predicted relevance score as the
importance score for the objects in our dataset.

Approach | AP $\uparrow$ | OT-F1 score $\uparrow$ | OT-Accuracy $\uparrow$
---|---|---|---
Everything Important | 0.159 | 0.277 | 90.5 %
PlanT | 0.572 | 0.594 | 92.7 %
Inverse Distance | 0.630 | 0.599 | 93.1 %
Ours | 0.710 | 0.708 |  94.8 %
TABLE I:  Our approach outperforms the baseline approaches with a considerable
margin on all metrics.

Table I quantitatively compares our approach with the baseline approaches
discussed above. We report the Average Precision (AP), the Optimal Threshold
F1 Score (OT-F1), and the Optimal threshold Accuracy (OT-Acc) values. The
optimal threshold F1 score and accuracy are the highest F1 score and accuracy
values across different thresholds.

The results in Table I show that our method outperforms all the baseline
approaches across the three metrics. Compared to our approach, the “Everything
Important” approach performs significantly worse ($0.551$ lower AP, $0.431$
lower OT-F1 score, and $4.3$% worse OT-Accuracy). This shows that attending to
every object in the scene is not necessary in order to make safe driving
decisions for the ego vehicle.

Our approach also outperforms PlanT on all the metrics ($0.138$ lower AP,
$0.114$ lower OT-F1 score, and $2.1$% worse OT-Accuracy). PlanT considers a
very short horizon when reasoning about the state of the ego vehicle and other
objects. It only predicts four waypoints into the future for the ego vehicles
and reasons about the states of the other objects only for the next timestep.
Due to this, PlanT usually assigns a high relevance score to only a few
objects nearest to the ego vehicle. Further, it does not consider
perturbations in the trajectories of other objects, which can cause it to miss
some objects that would be important under perturbations.

Our approach also outperforms the “Inverse Distance” baseline ($0.080$ lower
AP, $0.109$ lower OT-F1 score, and $1.7\%$ lower OT-Accuracy). This
demonstrates that proximity does not always indicate importance. Importance
can be better estimated by reasoning about the trajectories of the non-ego
vehicles and their influence on the ego vehicle’s driving. The qualitative
examples in Fig. 7 shed more light on the reasons behind the superior
performance of our approach.

Approach | AP $\uparrow$ | OT-F1 score $\uparrow$ | OT-Accuracy $\uparrow$
---|---|---|---
Object Removal Score | $0.466$ | $0.444$ |  $91.7$ %
Velocity Perturbations | $0.621$ | $0.670$ |  $94.2$ %
Ours | 0.710 | 0.708 |  94.8 %
Ours w/o Hard Stop | $0.652$ | $0.628$ |  $93.9$ %
Ours w/o Speed Up | $0.701$ | $0.695$ |  $94.5$ %
Ours w/o Lane Change | $0.565$ | $0.512$ |  $93.1$ %
Ours w/o Ego vehicle Perturbations | $0.635$ | $0.600$ |  $93.7$ %
Ours w/o Index based Weighting | $0.435$ | $0.601$ |  $90.5$ %
TABLE II: Ablation experiments demonstrate the impact of the various
components and design choices of our method.

###  V-B Ablations

We perform extensive ablation experiments to highlight the importance of the
various components of our approach and justify our design choices. We first
ablate over the two counterfactual scenarios, i.e., the removal score
($\mathcal{RS}$) and the velocity perturbation scores ($\mathcal{VS}$). The
results on the top half of Table II show that using these scores alone perform
worse on all three metrics compared to our method. This demonstrates the
complementary nature of the two scores and the benefit of combining them, as
in our method. Next, we ablate over the three velocity perturbations. The
bottom half of Table II shows that removing any of the three velocity
perturbations worsens performance. Removing the “lane change” perturbation
results in an especially large drop in performance ($0.145$ lower AP). This
might be because the “lane change” perturbation assigns a high importance
score to objects moving in lanes adjacent to the ego vehicle.

Further, we try to understand the benefits of perturbing the ego vehicle
trajectory (a defensive driving measure, in case the ego vehicle needs to make
an emergency maneuver). We observe a decrease in performance (drop of $0.075$
in AP) if we do not perform velocity perturbations for the ego vehicle.
Finally, we perform an ablation experiment to assess the significance of index
based weighting in the velocity perturbation score. We score vehicle
importance due to the Velocity Perturbations based on how soon they are
predicted to collide with the ego vehicle (Eq. 2). Without index based
weighting (treating all vehicles identified as important due to velocity
perturbations with equal importance), the performance drops drastically across
all metrics (AP drops by $0.357$, OT-F1 score drops by $0.107$, and OT-
Accuracy drops by $4.3\%$). Removing index based weighting results in a higher
false positive rate as vehicles that influence the ego vehicle’s decisions
immediately are considered equally important as those which affect the ego
vehicle’s decisions later.

###  V-C Importance Prediction for Vehicles and Pedestrians

Approach | AP $\uparrow$ | OT-F1 score $\uparrow$ | OT-Accuracy $\uparrow$
---|---|---|---
(Only Vehicles) |  |  |
Everything Important | 0.151 | 0.265 | 91.0 %
PlanT | 0.577 | 0.598 | 93.0 %
Inverse Distance | 0.614 | 0.593 | 93.2 %
$IS$ (Eq 3, Ours - Vehicles) | 0.688 | 0.697 |  94.8 %
TABLE III: Comparison of our approach with the baseline approaches for
_vehicle_ importance estimation. Our approach outperforms the baseline
approaches across all $3$ metrics

In a dynamic traffic scene, vehicles and pedestrians have different movement
characteristics, and hence we use different approaches for estimating the
importance of vehicles and pedestrians (see Sec IV-B). To understand the
performance of our approach on vehicles versus pedestrians, we also analyzed
each of these categories separately.

Table III compares our approach for vehicles only (Sec IV-A) with the baseline
approaches (Sec V-A). We observe similar trends as for the full dataset
(vehicles & pedestrians). Our method outperforms the other baselines on all
metrics.

Table IV compares our approach for pedestrians only (Sec. IV-B) against the
baseline approaches (Sec. V-A). As described in Sec. IV-B, we use inverse
distance for pedestrian importance estimation. Using inverse distance
outperforms all the other approaches across the three metrics. Its near-
perfect AP ($0.996$) shows that pedestrians are deemed important based on
their distance from the ego vehicle. PlanT only reasons over a short horizon
(Sec V-A), hence it frequently identifies pedestrians closer to the ego
vehicle as important, resulting in a good performance. We also experiment with
using our proposed approach for vehicles ($\mathcal{IS}$, Eqn. 3) to estimate
pedestrian importance; this approach also does not perform as well as using
inverse distance, and hence we use inverse distance for estimating pedestrian
importance in our method. While $IS$ is able to identify pedestrians as
important when they venture out on the road due to the removal score
($\mathcal{RS}$), it fails to identify the pedestrians on the sidewalks as
important.

Approach | AP $\uparrow$ | OT-F1 score $\uparrow$ | OT-Accuracy $\uparrow$
---|---|---|---
(Only Pedestrians) |  |  |
Everything Important | 0.600 | 0.749 | 60.0 %
PlanT | 0.974 | 0.926 | 90.0 %
$IS$ (Eq. 3) | 0.963 | 0.960 | 95.0 %
Inverse Dist. (Ours - Ped) | 0.996 | 0.978 |  97.5 %
TABLE IV: Comparison of the baseline approaches for _pedestrian_ importance
estimation. $\mathcal{IS}$ refers to our approach for _vehicle_ importance
estimation (Sec. IV-A, Eqn. 3). Our approach outperforms all other approaches
across all $3$ metrics.

(a) Removing the black truck (red box) incorrectly results in a predicted slow
down for the ego vehicle (blue trajectory to red trajectory).

(b) The white car (red box) is incorrectly identified as important. The
perturbed trajectories of the ego vehicle and non-ego vehicle are shown in
orange and dark green, respectively.

Figure 9: Failure cases of our method.

###  V-D Failure Cases

Occasionally, our approach failed to correctly identify important objects. We
analyzed the reasons for these failures.

Errors in trajectory prediction: Our method leverages an autonomous driving
agent [18] to predict trajectories of the vehicles in the scene. Hence, our
method is prone to errors made by the autonomous driving agent in trajectory
prediction. Fig. 9(a) shows a driving scenario under which an incorrect
removal score is predicted for vehicles due to an error in the ego vehicle’s
trajectory prediction. Since the calculation of the Removal Score
($\mathcal{RS}$) involves probing an autonomous driving agent with true and
counterfactual driving scenarios, analysing the error cases that arise due to
an incorrect removal score can help gain insights about the driving scenarios
where autonomous driving agents fail.

Vehicles in far-away lanes: As a defensive driving measure, we perturb the ego
vehicle trajectory along with the other vehicles in the scene. While
perturbing the ego vehicle’s trajectory overall improves performance (Table
II), sometimes this approach marks vehicles with remote chances of collision
as important. For example, in Fig. 9(b), the white vehicle is identified as
important, even though the chances of it colliding with the ego vehicle are
slim.

##  VI Conclusion

We focus on the task of important object identification in the context of
driving. We define important objects as the ones a driver would watch out for
in order to drive the ego vehicle safely. This definition of importance is
better suited for the development of situational awareness support systems
than prior works which only consider vehicles currently influencing the ego-
vehicles trajectory. We put forth a new public dataset named HOIST, which
contains human-annotated importance labels for vehicles and pedestrians in
simulated driving scenarios. Unlike prior works, HOIST consists of multi-modal
sensor data used by state-of-the-art autonomous driving agents. We further
contribute a novel approach that relies on counterfactual reasoning to
identify object importance within dynamic traffic scenes. Our approach
outperforms various baselines for the task of important object identification
on HOIST. We hope that our work leads the way toward efficient, robust, and
explainable algorithms for the task of object importance estimation for
driving scenarios.

## References

  * [1] T. Wu, E. Sachdeva, K. Akash, X. Wu, T. Misu, and J. Ortiz, “Toward an adaptive situational awareness support system for urban driving,” in 2022 IEEE Intelligent Vehicles Symposium (IV), pp. 1073–1080, IEEE, 2022.
  * [2] A. Biswas, B. A. Pardhi, C. Chuck, J. Holtz, S. Niekum, H. Admoni, and A. Allievi, “Mitigating causal confusion in driving agents via gaze supervision,” in Aligning Robot Representations with Humans workshop @ Conference on Robot Learning, 2022.
  * [3] A. Palazzi, D. Abati, F. Solera, R. Cucchiara, et al., “Predicting the driver’s focus of attention: the dr (eye) ve project,” IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 7, pp. 1720–1733, 2018.
  * [4] Y. Xia, D. Zhang, J. Kim, K. Nakayama, K. Zipser, and D. Whitney, “Predicting driver attention in critical situations,” in Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part V 14, pp. 658–674, Springer, 2019.
  * [5] S. Baee, E. Pakdamanian, I. Kim, L. Feng, V. Ordonez, and L. Barnes, “Medirl: Predicting the visual attention of drivers via maximum entropy deep inverse reinforcement learning,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 13178–13188, 2021.
  * [6] A. Pal, S. Mondal, and H. I. Christensen, “” looking at the right stuff”-guided semantic-gaze for autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11883–11892, 2020.
  * [7] J. Fang, D. Yan, J. Qiao, J. Xue, H. Wang, and S. Li, “Dada-2000: Can driving accident be predicted by driver attentionf analyzed by a benchmark,” in 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 4303–4309, IEEE, 2019.
  * [8] I. Kotseruba and J. K. Tsotsos, “Behavioral research and practical models of drivers’ attention,” arXiv preprint arXiv:2104.05677, 2021.
  * [9] C. B. White and J. K. Caird, “The blind date: The effects of change blindness, passenger conversation and gender on looked-but-failed-to-see (lbfts) errors,” Accident Analysis & Prevention, vol. 42, no. 6, pp. 1822–1830, 2010.
  * [10] D. Crundall, P. Chapman, S. Trawley, L. Collins, E. Van Loon, B. Andrews, and G. Underwood, “Some hazards are more attractive than others: Drivers of varying experience respond differently to different types of hazard,” Accident Analysis & Prevention, vol. 45, pp. 600–609, 2012.
  * [11] A. Biswas and H. Admoni, “Characterizing drivers’ peripheral vision via the functional field of view for intelligent driving assistance,” in 2023 IEEE Intelligent Vehicles Symposium (IV), pp. 1–8, IEEE, 2023.
  * [12] M. Gao, A. Tawari, and S. Martin, “Goal-oriented object importance estimation in on-road driving videos,” in 2019 International Conference on Robotics and Automation (ICRA), pp. 5509–5515, IEEE, 2019.
  * [13] Z. Zhang, A. Tawari, S. Martin, and D. Crandall, “Interaction graphs for object importance estimation in on-road driving videos,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 8920–8927, IEEE, 2020.
  * [14] J. Li, H. Gang, H. Ma, M. Tomizuka, and C. Choi, “Important object identification with semi-supervised learning for autonomous driving,” in 2022 International Conference on Robotics and Automation (ICRA), pp. 2913–2919, IEEE, 2022.
  * [15] M. R. Endsley, “Toward a theory of situation awareness in dynamic systems,” Human factors, vol. 37, no. 1, pp. 32–64, 1995.
  * [16] K. Chitta, A. Prakash, B. Jaeger, Z. Yu, K. Renz, and A. Geiger, “Transfuser: Imitation with transformer-based sensor fusion for autonomous driving,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  * [17] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in Conference on robot learning, pp. 1–16, PMLR, 2017.
  * [18] D. Chen and P. Krähenbühl, “Learning from all vehicles,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17222–17231, 2022.
  * [19] C. Li, S. H. Chan, and Y.-T. Chen, “Who make drivers stop? towards driver-centric risk assessment: Risk object identification via causal inference,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10711–10718, IEEE, 2020.
  * [20] C. Li, S. H. Chan, and Y.-T. Chen, “Droid: Driver-centric risk object identification,” IEEE transactions on pattern analysis and machine intelligence, 2023.
  * [21] A. Vemula, K. Muelling, and J. Oh, “Social attention: Modeling attention in human crowds,” in 2018 IEEE international Conference on Robotics and Automation (ICRA), pp. 4601–4607, IEEE, 2018.
  * [22] J. Li, F. Yang, H. Ma, S. Malla, M. Tomizuka, and C. Choi, “Rain: Reinforced hybrid attention inference network for motion forecasting,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16096–16106, 2021.
  * [23] J. Li, H. Ma, Z. Zhang, J. Li, and M. Tomizuka, “Spatio-temporal graph dual-attention network for multi-agent prediction and tracking,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 8, pp. 10556–10569, 2021.
  * [24] K. Renz, K. Chitta, O.-B. Mercea, A. S. Koepke, Z. Akata, and A. Geiger, “Plant: Explainable planning transformers via object-level representations,” in Conference on Robotic Learning (CoRL), 2022.
  * [25] E. Aksoy, A. Yazıcı, and M. Kasap, “See, attend and brake: An attention-based saliency map prediction model for end-to-end driving,” arXiv preprint arXiv:2002.11020, 2020.
  * [26] Y. Xia, Driver eye movements and the application in autonomous driving.  University of California, Berkeley, 2019.
  * [27] Y. Xia, J. Kim, J. Canny, K. Zipser, T. Canas-Bajo, and D. Whitney, “Periphery-fovea multi-resolution driving model guided by human attention,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1767–1775, 2020.
  * [28] B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al., “Maximum entropy inverse reinforcement learning.,” in Aaai, vol. 8, pp. 1433–1438, Chicago, IL, USA, 2008.
  * [29] H. Shao, L. Wang, R. Chen, S. L. Waslander, H. Li, and Y. Liu, “Reasonnet: End-to-end driving with temporal and global reasoning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13723–13733, 2023.
  * [30] H. Shao, L. Wang, R. Chen, H. Li, and Y. Liu, “Safety-enhanced autonomous driving using interpretable sensor fusion transformer,” arXiv preprint arXiv:2207.14024, 2022.
  * [31] P. Wu, X. Jia, L. Chen, J. Yan, H. Li, and Y. Qiao, “Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong baseline,” Advances in Neural Information Processing Systems, vol. 35, pp. 6119–6132, 2022.
  * [32] S. Vora, A. H. Lang, B. Helou, and O. Beijbom, “Pointpainting: Sequential fusion for 3d object detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4604–4612, 2020.