[a]Kevin Flanagan

# Identifying muon rings in VERITAS data using convolutional neural networks
trained on images classified with Muon Hunters 2

and John Quinn  for the VERITAS Collaboration222a complete list of authors can
be found at the end of the proceedings Darryl Wright  Hugh Dickinson  Patrick
D. Wilcox  Michael Laraia  Stephen Serjeant

###### Abstract

Muons from extensive air showers appear as rings in images taken with imaging
atmospheric Cherenkov telescopes, such as VERITAS. These muon-ring images are
used for the calibration of the VERITAS telescopes, however the calibration
accuracy can be improved with a more efficient muon-identification algorithm.
Convolutional neural networks (CNNs) are used in many state-of-the-art image-
recognition systems and are ideal for muon image identification, once trained
on a suitable dataset with labels for muon images. However, by training a CNN
on a dataset labelled by existing algorithms, the performance of the CNN would
be limited by the suboptimal muon-identification efficiency of the original
algorithms. Muon Hunters 2 is a citizen science project that asks users to
label grids of VERITAS telescope images, stating which images contain muon
rings. Each image is labelled 10 times by independent volunteers, and the
votes are aggregated and used to assign a ‘muon’ or ‘non-muon’ label to the
corresponding image. An analysis was performed using an expert-labelled
dataset in order to determine the optimal vote percentage cut-offs for
assigning labels to each image for CNN training. This was optimised so as to
identify as many muon images as possible while avoiding false positives. The
performance of this model greatly improves on existing muon identification
algorithms, identifying approximately 30 times the number of muon images
identified by the current algorithm implemented in VEGAS (VERITAS Gamma-ray
Analysis Suite), and roughly 2.5 times the number identified by the Hough
transform method, along with significantly outperforming a CNN trained on
VEGAS-labelled data.

##  1 Introduction

Imaging atmospheric Cherenkov telescopes (IACTs) detect high-energy gamma
radiation from astrophysical sources by imaging the Cherenkov light from
extensive air showers (EAS) produced when the gamma rays interact with the
Earth’s atmosphere. As well as gamma rays, IACTs are also able to detect high-
energy cosmic ray particles, which form a large source of background for the
telescope. While most of this background is treated as noise, signal from muon
particles, which are by-products of the EAS of cosmic ray particles, may be
used for calibration purposes. The cone of Cherenkov light from an individual
muon particle produces a ring-shaped image in an IACT. The Cherenkov opening
angle (ring radius) and number of photons emitted per unit path are dependent
on the energy of the muon. Therefore by determining the radius of the ring and
measuring the amount of signal in that ring in the telescope, this can be
compared with the amount of signal expected based on the ring size. By
performing this comparison with many different muon rings, an average
calibration factor for each telescope may be determined, which is used to
correct for the light which is lost on the way to each telescope camera. To
carry out this procedure, a large number of muon images must be identified.
With advances in machine learning and particularly in image classification
through the use of convolutional neural networks (CNNs), it is possible for
the rate of automatic detection of muon images to be improved dramatically
over existing algorithms.

##  2 VERITAS Array

The Very Energetic Radiation Imaging Telescope Array System (VERITAS) is an
array of 4 IACTs based at the Fred Lawrence Whipple Observatory in Arizona and
operates in the energy range of 85 GeV to 30 TeV. The array consists of four
12 m diameter telescopes arranged in a roughly rectangular pattern, each a
distance of about 100 m apart. The telescopes have Davies-Cotton segmented
mirrors, each consisting of 350 individual hexagonal mirrors, with a total
mirror area of 110 m2. Each telescope’s PMT camera has 499 pixels and a field
of view of 3.5∘. VERITAS became operational in 2007 and is still operational
currently. The array has a trigger system which requires that at least 2
telescopes simultaneously pass their own telescope-level trigger in order for
each telescope in the array to record an image. This is largely to reject
background events, including muon events, which typically only trigger one
telescope. Many muon images contain additional Cherenkov light from the EAS
from which they were produced in order to trigger the array.

##  3 Methods for the Identification of Muon Images

VEGAS is one of the two analysis packages used by VERITAS in analysing data
from the telescopes [1], and it includes a muon identification algorithm. It
works based on the distribution of signal in each image, determining the
centroid of the signal and placing an annulus at the average radius (known as
muon radius) of signal out from the centroid. The annulus has a width of
3$\sigma$ of the signal distance to the mean radius. If more than 70% of the
signal in the image is within the annulus it is classified as a muon image.
While this method can identify full muon rings, there are many rings which it
doesn’t identify, such as partial muon rings, which occur when the muon’s
impact location is outside or near the edge of the telescope. This makes some
photons miss the telescope and thus causes part of the ring to not show in the
camera. VEGAS also doesn’t identify truncated rings, where the ring is cut off
by the edge of the camera due to the muon’s incident angle with respect to the
telescope’s optical axis being greater than 1.75∘, nor ring images which
contain signal other than the ring. These last two are currently not usable
for calibration, however.

An algorithm called the Hough transform may also be used for muon
identification [9]. This algorithm is used to parameterise assumed shapes in
digital images. This has been used to parameterise assumed circles in VERITAS
images and then have cuts on these parameters applied to determine the
presence or absence of a muon ring. This has been shown to achieve better
results than the previously described VEGAS algorithm.

Convolutional neural networks (CNNs) [2] may also be used to identify muon-
ring images. CNNs are a type of machine learning algorithm commonly used in
image classification tasks, where they have shown very high performance. They
take an image as input and pass it through convolutional layers, pooling
layers and finally fully-connected layers to produce a final output. In a
binary classification problem such as muon identification, there is a single
output, taking a value between 0 and 1. This number indicates the model’s
prediction of how likely an image is to belong to one of these classes as
opposed to the other. CNNs learn to classify images by being trained on a
large dataset of relevant labelled images. Each image in a training set is
labelled either with a 0 or a 1, depending on its true label. The CNN predicts
a value between 0 and 1 for each image, and based on how close its predictions
are to the true labels, the parameters of the CNN are adjusted using an
algorithm known as gradient descent. Following training, a cut-off is chosen
in order to turn the outputs into classifications, typically taking a value of
0.5. Provided with a large labelled dataset of muon and non-muon images, a CNN
may be trained to perform muon ring identification.

CNN classification of VERITAS muon images has been done previously, first
using a CNN trained on VEGAS-labelled data [3], and then using data from the
original Muon Hunter [4] project to train a CNN [5]. However, the first of
these used VEGAS-labelled data to both train and test the model, which didn’t
test the generalisability of the model. The second chose a single vote-
percentage cut-off (discussed further in section 4) for labelling Muon Hunter
images as ‘muon’ or ‘non-muon’, without an analysis of what the best cut-offs
would be. In this paper we both analyse model performances to determine
optimised vote-percentage cut-offs for the task and test the models on a more
independent dataset to better understand the generalisability of the model.

##  4 Training and Validation Dataset: Muon Hunters 2

Images labelled using the VEGAS algorithm may be used to create a labelled
dataset for training a CNN. However, in order to ensure that the CNN is able
to identify muon images which the VEGAS algorithm cannot identify, it is
better to have a labelled dataset which includes these types of images also in
the muon data. One way of obtaining such a dataset with more accurate labels
is via human volunteers. For this reason the Muon Hunters 2 (MH2) [6] project
was created.

Muon Hunters 2 is a citizen science project for gathering muon labels for
VERITAS images, hosted on Zooniverse.org. As a follow-on from the original
Muon Hunter project [4], Muon Hunters 2 was launched in March of 2019 with the
aim of making the identification of muon rings more efficient. In this
project, a 6 $\times$ 6 grid of images is presented, and volunteers are asked
to classify all of these images simultaneously by clicking on all images
belonging to the minority class in that grid (muon or non-muon). This works to
reduce the number of clicks required from volunteers. Figure 1 displays all
possible scenarios. Volunteers are asked to identify images which contain muon
rings of any variety, be they full rings, partial rings or truncated rings,
and also regardless of whether there is any signal other than the ring present
in the image.

Figure 1: Schematic of the possible scenarios a volunteer will encounter in
the Muon Hunters 2 project.

Each grid of images is retired once it has been classified by 10 independent
volunteers. 2.9 million images from real VERITAS data were included as part of
the MH2 project, along with 95,000 images generated from CORSIKA simulations.
For this study, a total of 605,057 labelled Muon Hunters 2 images were
available and were used. The volunteer votes were used to divide the data into
muon and non-muon categories for the purposes of training a CNN. Vote
percentage cut-offs were used to assign images muon and non-muon labels, such
that an image was assigned a muon label if voted a muon by at least X$\%$ of
volunteers and non-muon if voted a muon by at most Y$\%$. These cut-off values
(X$\%$ and Y$\%$) were determined through testing on a different,
independently labelled dataset (process explained in section 6). The optimal
cut-offs which were used to generate the labelled dataset for this study were
a 20$\%$ muon and 0$\%$ non-muon cut-off. A dataset of 10,000 muon images and
10,000 non-muon images was selected and split into training, validation and
test sets in a 65$\%$ : 17.5$\%$ : 17.5$\%$ split.

A VEGAS-labelled dataset was also produced, containing 10,000 muon images and
10,000 non-muon images. The same 65$\%$ : 17.5$\%$ : 17.5$\%$ split was also
applied to this data. VEGAS-labelled images had an extra condition that the
muon radius $r$ must be greater than 0.5∘ in order to increase the purity of
the images, as it’s found that the algorithm does not perform well on images
with smaller muon radii than this.

Each image used as input to the CNN was a jpeg generated from the charge
produced in the telescope camera. This was different to the oversampling
approach used by Feng et al. [3, 5] as that method led to a stretching of the
image which we wished to avoid.

##  5 CNN Training

Both the VEGAS-labelled and MH2 datasets were used to train a CNN for the
identification of muon images. The Python deep learning library
keras111https://keras.io with a Tensorflow222https://www.tensorflow.org
backend was used to implement the CNN model. A VGG16 [7] architecture was used
for the CNN. This was pretrained on the ImageNet [8] dataset, which is a large
dataset of labelled images often used to benchmark and pre-train image
classification models. The same training parameters were used for each
dataset. Mini-batch stochastic gradient descent with momentum was used as the
optimisation algorithm. A learning rate of 0.005 and a momentum value of 0.9
were chosen, with a batch size of 64. A binary-cross-entropy cost function was
used. Training was carried out for 50 epochs (an epoch being one full cycle of
training on the entire dataset). The training and validation set cost values
were tracked during training. As the validation cost measures the model’s
ability to generalise to unseen data, the model weights which minimised this
value were saved as the optimised weights.

##  6 Determination of Optimal Vote-Percentage Cut-Offs

As there are many ways to divide the MH2 data based on volunteer votes, it was
necessary to determine the best way to do this for training a CNN for muon
identification. A large number of different datasets were created using
different pairs of vote-percentage cut-offs and models were trained on each
dataset. Each dataset consisted of 17,200 images, split evenly between images
with muon labels and non-muon labels. This dataset size was chosen in order to
make each dataset the same size, as this was the largest possible for all
datasets produced via the vote-percentage cut-offs. These datasets were
further divided into training (10,320 images), validation (3,440 images) and
test sets (3,440 images) each with equal numbers of muon and non-muon images.
The performance of each model was tested using an independent test dataset
with expert labels produced by one of the authors of this study. This test set
contained 2,000 muon-labelled images and 2,000 non-muon-labelled images. It
was found that overall accuracy (fraction of images classified correctly) on
the independent test set was maximised by training the CNN on a dataset
generated with (20,0) vote-percentage cut-offs. This means that a cut-off of
20$\%$ was applied for labelling muon images (images were labelled as muon
images if their muon vote-percentage was 20$\%$ or higher) and a cut-off of
0$\%$ was applied for labelling non-muon images (images were labelled as non-
muon if their muon vote-percentage was 0$\%$). This model achieved a test set
accuracy of 97.1%, with 43 false positives and 73 false negatives. It was also
found that by adjusting the classification boundary on the (20,0) model output
(from the default 0.5, where an output $\geq$ 0.5 gives a non-muon
classification and an output $<$ 0.5 gives a muon classification) that this
model could achieve 0 false positives while retaining a higher overall
accuracy than other models which achieved 0 false positives. This is an
important factor in choosing this model as we particularly wished to avoid
false positives.

The (20,0) model achieving the best performance on the test data can likely be
explained by aspects of volunteer labelling ability and the presence of
ambiguous muon images. It is generally easier for volunteers to label non-muon
images correctly rather than muon images due to the presence of more
difficult-to-identify muon images which volunteers may miss. These can contain
partial and truncated rings, sometimes partially obscured by other signal in
the image. This leads to many muon images only being labelled as containing a
muon ring by volunteers a small number of times. This means that higher vote-
percentage cut-offs for non-muons would lead to non-muon datasets that are
contaminated by some of these more ambiguous muon images, which would confuse
any model. Therefore a vote-percentage cut-off of 0% produces what is closest
to a pure set of non-muon images obtainable in this manner. The high
performance of the low (20$\%$) muon cut-off likely comes down to increasing
how representative the dataset is. The higher the muon vote-percentage used to
generate the muon dataset, the less representative the dataset will be of all
of the types of muon images that exist, as the muon images that are identified
correctly by volunteers a large amount of the time are likely to be very
clear-cut muon images. Therefore the model will only learn to identify these
more obvious images. If a smaller vote-percentage

Figure 2: Fraction of images at each muon vote-percentage which contain
expert-classified muon rings, based on 1,000 random images at each muon vote-
percentage. Note the large increase at 20$\%$.

is used to generate the muon dataset, the images become more representative of
all muon image types, so the model will be better able to identify all muon
images. While the level of contamination from non-muon images in the muon set
also increases, it appears that the benefits of the increase in
representativity of the data outweigh this. Figure 2 highlights how even at
the low vote-percentage of 20% roughly 7 out of 10 images contain muon rings
according to expert labels, while the datasets become fully composed of
expert-labelled muons from 60% and above. This displays the prevalence of more
ambiguous muon images which volunteers found harder to identify at low vote-
percentages.

##  7 Final Model Performances

The performances of the models trained on VEGAS-labelled and MH2-labelled data
were evaluated on a final 10,000 image (50% muon, 50% non-muon) test dataset
which again had expert labels produced by one of the authors. The results are
displayed in Table 1.

Model | Accuracy (%) | Maximum number of pure muons
---|---|---
VEGAS | 69.9 | 551 (11%) (boundary = 0.024)
MH2 | 95.7 | 3892 (78%) (boundary = 0.064)
Table 1: Summary of VEGAS-trained and MH2-trained model performances on the
expert-labelled test set.

The VEGAS-trained model achieves a moderate accuracy score, while the
MH2-trained model achieves a significantly higher score, correctly classifying
over 95% of images. The most important metric to consider for these models is
the ‘maximum number of pure muons’ which the model is able to retrieve. The
number is obtained by adjusting the decision boundary of the output
(increasing the confidence which the model must have to make a muon
classification) until the model produces no false positive muon
classifications on the test dataset. This represents the approximate portion
of muon images which the model would be able to reliably detect without
producing false positives. If one of these models was implemented in the
VERITAS telescope, the decision boundary on the model would be adjusted to at
least this minimum confidence level in order to obtain pure sets of muon
detections. The MH2-trained model was able to detect a significantly larger
pure set of muon images than the VEGAS-trained model, with $\sim$78$\%$
detected vs $\sim$11$\%$.

Although a direct application and comparison was not made with the Hough
Transform algorithm, a comparison with the results reported in a previous
study by Tyler et al. [9] was made. In that paper, a similar analysis was
performed on the percentage of muon images identified by the algorithm (muon
efficiency). The Hough Transform study also used a set of images examined by
eye and estimated that about 6.5% of images in a typical observation run were
muon images, which agrees very closely with this study’s evaluation when
creating labelled test data (also 6.5%). This indicates that the test datasets
used in each study likely have a similar make-up. Upon application of the
Hough transform however, they achieved 29% muon efficiency with no false
positives, whereas the MH2 CNN in this study achieves 78% muon efficiency with
no false positives, indicating a substantial increase in ability to detect
muon images over the Hough Transform method also.

The performance of the MH2-trained model was also compared with the original
VEGAS algorithm. A set of 481,819 images from a VERITAS observation run were
taken and each muon detection method was applied to these images. The CNN
models were each applied with the muon/non-muon output cut-off value which
resulted in the largest pure set of muons being isolated from the expert-
labelled test dataset. The number of muon images identified by these models
was compared with the number detected by the VEGAS algorithm (with a $r$
$\geq$ 0.5∘ cut-off applied). The numbers are displayed in Table 2.

Identification method | Number of muon images identified
---|---
VEGAS algorithm | 728
VEGAS-trained CNN | 3071
MH2-trained CNN | 23748
Table 2: Number of muon images identified by each method out of the 481,819
images in the dataset.

The VEGAS-trained CNN does show an improvement over the original algorithm,
showing generalisation from the CNN training, however the MH2-trained CNN far
exceeds this, identifying $\sim$32 times more muon images than the VEGAS
algorithm.

##  8 Conclusion

This study has investigated the use of CNNs for muon identification in VERITAS
images. Tests were carried out to determine the best way of using the
volunteer votes from the Muon Hunters 2 project to generate labels, and a CNN
trained on these labelled images outperformed a CNN trained on VEGAS-labelled
data. Both CNNs were compared with the current VEGAS muon detection algorithm,
and both achieved a higher rate of detection of muon images. This was
especially true for the MH2 CNN, identifying approximately 32 times the number
of muon images that the VEGAS algorithm did. This was achieved with the output
value cut-off of the CNN adjusted so as to eliminate false positives from the
muon images identified. Certain muon images may be used for calibration of the
VERITAS telescopes. However, as the MH2-trained CNN is trained to identify all
types of muon images, including those which are not usable for calibration
such as those with truncated rings and with signal other than the ring, some
extra work is needed to extract only images which are suitable for
calibration. This may be similar to what is currently done with the VEGAS
algorithm where certain extra quality cuts are applied. As a significantly
larger number of muon images are identified by the MH2-trained CNN overall, it
should also be the case that a larger number of images suitable for
calibration are identified as well.

## Acknowledgements

This research is supported by grants from the U.S. Department of Energy Office
of Science, the U.S. National Science Foundation and the Smithsonian
Institution, by NSERC in Canada, and by the Helmholtz Association in Germany.
This research used resources provided by the Open Science Grid, which is
supported by the National Science Foundation and the U.S. Department of
Energy’s Office of Science, and resources of the National Energy Research
Scientific Computing Center (NERSC), a U.S. Department of Energy Office of
Science User Facility operated under Contract No. DE-AC02-05CH11231. We
acknowledge the excellent work of the technical support staff at the Fred
Lawrence Whipple Observatory and at the collaborating institutions in the
construction and operation of the instrument.

This publication uses data generated via the Zooniverse.org platform,
development of which is funded by generous support, including a Global Impact
Award from Google, and by a grant from the Alfred P. Sloan Foundation.

This project has been partly supported by ESCAPE - The European Science
Cluster of Astronomy $\&$ Particle Physics ESFRI Research Infrastructures,
which has received funding from the European Union’s Horizon 2020 research and
innovation programme under Grant Agreement no. 824064.

## References

  * [1] P. Cogan, VEGAS, the VERITAS Gamma-ray Analysis Suite, International Cosmic Ray Conference, Vol. 3, pp. 1385-1388, 2008
  * [2] A. Krizhevsky et al. ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems 25, pp. 1097–1105, 2012.
  * [3] Q. Feng and T. T. Y. Lin for the VERITAS Collaboration, The analysis of VERITAS muon images using convolutional neural networks. IAU Symposium 325 on Astroinformatics, 2016.
  * [4] R. Bird et al. Muon Hunter: a Zooniverse project, arxiv preprint arXiv:1802.08907 [astro-ph.IM], 2018
  * [5] Q. Feng et al. A citizen-science approach to muon events in imaging atmospheric Cherenkov telescope data: the Muon Hunter. 35th International Cosmic Ray Conference, PoS(ICRC2017)826, 2017
  * [6] M. Laraia, for the VERITAS Collaboration et al. Results from Muon Hunter 2. 36th International Cosmic Ray Conference, PoS(ICRC2019)678, 2019
  * [7] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015
  * [8] J. Deng et al. Imagenet: A large-scale hierarchical image database. 2019 IEEE conference on computer vision and pattern recognition pp. 248–255, 2019
  * [9] J. Tyler for the VERITAS Collaboration, Muon Identification with VERITAS using the Hough Transform, 33rd International Cosmic Ray Conference, PoS(ICRC2013)931, 2013

## Full Authors List: Collaboration

C. B. Adams1, A. Archer2, W. Benbow3, A. Brill1, J. H. Buckley4, M. Capasso5,
J. L. Christiansen6, A. J. Chromey7, M. Errando4, A. Falcone8, K. A. Farrell9,
Q. Feng5, G. M. Foote10, L. Fortson11, A. Furniss12, A. Gent13, G. H.
Gillanders14, C. Giuri15, O. Gueta15, D. Hanna16, O. Hervet17, J. Holder10, B.
Hona18, T. B. Humensky1, W. Jin19, P. Kaaret20, M. Kertzman2, T. K. Kleiner15,
S. Kumar16, M. J. Lang14, M. Lundy16, G. Maier15, C. E McGrath9, P.
Moriarty14, R. Mukherjee5, D. Nieto21, M. Nievas-Rosillo15, S. O’Brien16, R.
A. Ong22, A. N. Otte13, S. R. Patel15, S. Patel20, K. Pfrang15, M. Pohl23,15,
R. R. Prado15, E. Pueschel15, J. Quinn9, K. Ragan16, P. T. Reynolds24, D.
Ribeiro1, E. Roache3, J. L. Ryan22, I. Sadeh15, M. Santander19, G. H.
Sembroski25, R. Shang22, D. Tak15, V. V. Vassiliev22, A. Weinstein7, D. A.
Williams17, and T. J. Williamson10
1Physics Department, Columbia University, New York, NY 10027, USA 2Department
of Physics and Astronomy, DePauw University, Greencastle, IN 46135-0037, USA
3Center for Astrophysics $|$ Harvard & Smithsonian, Cambridge, MA 02138, USA
4Department of Physics, Washington University, St. Louis, MO 63130, USA
5Department of Physics and Astronomy, Barnard College, Columbia University, NY
10027, USA 6Physics Department, California Polytechnic State University, San
Luis Obispo, CA 94307, USA 7Department of Physics and Astronomy, Iowa State
University, Ames, IA 50011, USA 8Department of Astronomy and Astrophysics, 525
Davey Lab, Pennsylvania State University, University Park, PA 16802, USA
9School of Physics, University College Dublin, Belfield, Dublin 4, Ireland
10Department of Physics and Astronomy and the Bartol Research Institute,
University of Delaware, Newark, DE 19716, USA 11School of Physics and
Astronomy, University of Minnesota, Minneapolis, MN 55455, USA 12Department of
Physics, California State University - East Bay, Hayward, CA 94542, USA
13School of Physics and Center for Relativistic Astrophysics, Georgia
Institute of Technology, 837 State Street NW, Atlanta, GA 30332-0430 14School
of Physics, National University of Ireland Galway, University Road, Galway,
Ireland 15DESY, Platanenallee 6, 15738 Zeuthen, Germany 16Physics Department,
McGill University, Montreal, QC H3A 2T8, Canada 17Santa Cruz Institute for
Particle Physics and Department of Physics, University of California, Santa
Cruz, CA 95064, USA 18Department of Physics and Astronomy, University of Utah,
Salt Lake City, UT 84112, USA 19Department of Physics and Astronomy,
University of Alabama, Tuscaloosa, AL 35487, USA 20Department of Physics and
Astronomy, University of Iowa, Van Allen Hall, Iowa City, IA 52242, USA
21Institute of Particle and Cosmos Physics, Universidad Complutense de Madrid,
28040 Madrid, Spain 22Department of Physics and Astronomy, University of
California, Los Angeles, CA 90095, USA 23Institute of Physics and Astronomy,
University of Potsdam, 14476 Potsdam-Golm, Germany 24Department of Physical
Sciences, Munster Technological University, Bishopstown, Cork, T12 P928,
Ireland 25Department of Physics and Astronomy, Purdue University, West
Lafayette, IN 47907, USA


# Learnable Irrelevant Modality Dropout for Multimodal Action Recognition on
Modality-Specific Annotated Videos

Saghir Alfasly1,2 Jian Lu1,3, Chen Xu1,2 Yuru Zou1
1Shenzhen Key Laboratory of Advanced Machine Learning and Applications,
Shenzhen University, China
2Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen,
China 3Pazhou Lab, Guangzhou, China
{saghiralfasly, jianlu<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding
author

###### Abstract

With the assumption that a video dataset is multimodality annotated in which
auditory and visual modalities both are labeled or class-relevant, current
multimodal methods apply modality fusion or cross-modality attention. However,
effectively leveraging the audio modality in vision-specific annotated videos
for action recognition is of particular challenge.

To tackle this challenge, we propose a novel audio-visual framework that
effectively leverages the audio modality in any solely vision-specific
annotated dataset. We adopt the language models (e.g., BERT) to build a
semantic audio-video label dictionary (SAVLD) that maps each video label to
its most K-relevant audio labels in which SAVLD serves as a bridge between
audio and video datasets. Then, SAVLD along with a pretrained audio multi-
label model are used to estimate the audio-visual modality relevance during
the training phase. Accordingly, a novel learnable irrelevant modality dropout
(IMD) is proposed to completely drop out the irrelevant audio modality and
fuse only the relevant modalities. Moreover, we present a new two-stream video
Transformer for efficiently modeling the visual modalities. Results on several
vision-specific annotated datasets including Kinetics400 and UCF-101 validated
our framework as it outperforms most relevant action recognition methods.

Figure 1: Conceptual Overview: Vision-Audio Label Mapping. In an overlapped
manner, our method performs a cross-dataset textual labels mapping in which
audio labels are mapped to their most closely relevant video-based human
activity labels by using language models, _e.g_., BERT. The resulting clusters
compose a SAVLD dictionary that serves as a bridge between video and audio
datasets. Since our framework trains human activity multimodal models on
vision-specific datasets, we use SAVLD to leverage the auditory modality in
videos.

##  1 Introduction

One of the deep neural network (DNN) learning schemes to improve video
understanding is to leverage as many input modalities as available, such as
audio, RGB frames, motion, textual data, the visible text on videos, and human
skeleton joints. Therefore, multimodal learning has shown a remarkable
improvement in video-based action recognition. These methods process either
each modality with an independent DNN or all with a single shared DNN. The
modalities or their feature representations are fused either in an early
fusion, middle fusion, or late fusion manner [1, 2, 3]. Another fusion scheme
is proposed by adopting cross-modality attention or gated units [4, 5, 6, 7].

Motivation. Most of the attention and fusion methods [2] boost the audio-
visual models’ performance on audio-vision correspondence-based videos in
which auditory and visual modalities are corresponding or at least relevant as
in Kinetics-Sounds [8] and EPIC-Kitchen dataset [9]. However, these fusion
methods do not provide similar performance boost on the vision-specific
annotated datasets, such as Kinetics400 [10] and UCF-101 [11]. This condition
is due to the high irrelevance and non-correspondence between visual and
auditory modalities as concluded in [1]. In this case, multimodal methods do
not effectively gain maximum benefit from unlabeld audio modality with noisy
relevance on vision-specific datasets. For example, music can largely indicate
that the human activity is dancing. However, music audio can be associated
with several other human activities, such as car driving, or simply can be
found as background audio in any edited video. This interprets the findings in
an interesting study [7], which showed that unimodal models consistently
outperform multimodal DNN in modality-specific datasets because multimodal
networks are more prone to overfitting with their increase in capacity. This
finding motivated us to look for a method that leverages the relevant audio
modality while completely dropping out the irrelevant modality in training and
inference phases.

Contributions. To tackle the aforementioned challenge, we present a novel
multimodal training framework that trains action recognition networks with the
best audio-visual modality combination on visual modality-specific datasets.
In our method, we automatically estimate the audio-visual modalities relevance
by leveraging the pretrained audio classification models on large audio-
specific datasets such as AudioSet [12] and VGGSound [13]. This task is
effectively achieved in two phases. In the first phase, we map audio labels of
an audio-specific dataset to their most semantically relevant labels in the
vision-specific dataset. This process is achieved using the language
pretrained models, _e.g_., BERT [14] or GloVe [15]. After obtaining the
semantic sentence-based embedding of each label, we perform cross-dataset
label matching and generate overlapped label clusters each of which contains a
single label from the video dataset along with a set of its most semantically
similar labels in the audio dataset as shown in Fig. 1. For example, Dancing
label in Kinetics400 and Music, Singing, and Clapping labels in AudioSet are
in the same cluster. Overall, the resulting overlapped clusters compose a
semantic audio-video label dictionary (SAVLD) that serves as a bridge between
video and audio datasets for the training supervision in our next phase. In
the second phase, as a transfer learning task, our framework adopts a
pretrained audio Transformer to generate highly semantic audio features and
multi-label audio predictions. The audio dataset used for building the SAVLD
should be the same dataset on which the audio Transformer has been trained.
Overall, the outputs of the audio Transformer and the SAVLD are then used by
our proposed framework to annotate audio, as depicted in Fig.2.

After obtaining the audio modality predictions using any audio pretrained
model (_e.g_., AST), our framework guided by the SAVLD drops the irrelevant
audio modality. This is performed by using a novel trainable irrelevant
modality dropout (IMD) that consists of two main modules: The first module is
a neural network termed relevance network (RN) that receives auditory and
visual modalities and decides whether they are relevant. The output of this
network is sigmoidal predictions representing the relevance level that is used
by the second module to decide whether to fuse the two modalities for the
final video classification or to drop the audio modality. We further improve
our framework learning by proposing a new intra-class cross-modality
augmentation in which it randomly pairs auditory and visual modalities of the
same class from different videos. This augmentation method may show a negative
effect on some applications of audio-visual networks when the audio and video
are required to be corresponding as the case of speech recognition. However,
in our case, the audio modality is required to be relevant but is not
necessarily to be aligned or accurately correspond to the visual modality
because we tackle the problem of human activity recognition in this work. This
motivates us to propose this augmentation method that empirically provides a
reasonable performance boost.

Generally, our framework is an entirely convolution-free Transformer-based
network. It leverages three video modalities: RGB frames, optical flow, and
audio. The RGB and optical flow modalities are processed by using our
efficient two-stream video Transformer, whereas the audio modality is
processed by any audio pretrained Transformer.

The key contributions of this work are as follows:

  * •

A novel multimodal human activity recognition framework that leverages the
power of the NLP BERT model and the pretrained audio classification models is
proposed to automatically annotate audio modality. It effectively trains
audio-visual action recognition models on any vision-specific annotated
dataset.

  * •

A novel learnable IMD network is proposed to completely drop out the
irrelevant audio modality, whereas the relevant modalities are fused on the
basis of their relevance level.

  * •

An efficient two-stream video Transformer is designed to learn the visual
modality with few parameters as compared to the relevant video Transformers.

  * •

An intra-class cross-modality augmentation method is proposed for generating
more training samples by allowing each audio modality sample to be paired with
any visual modality sample among the same class.

##  2 Related Work

Multimodal Action Recognition. Multimodal action recognition aims to exploit
multimodality input for better human activity recognition. DNNs have empowered
this learning scheme, which enriches the global feature learning [3, 7, 6, 16,
1, 5]. On the basis of the modality combination, some methods use RGB frames
along with motion modality as in the two-stream networks [17, 18, 4], with the
skeleton modality [19], with detected objects [2], with textual modality [20],
or with the audio modality [1, 19, 8, 21, 22, 23] which is the most common
multimodality approach to learn global representation for video understanding.
More than two modalities are exploited by some methods [2, 7, 16, 24, 5] to
improve action recognition. Another approach is followed for obtaining video
modalities as in [5, 25], which uses the compressed videos as a source of four
modalities, I-frames, motion vectors, residuals, and the audio modalities.
Using these modalities, the 2D/3D convolution networks have shown substantial
progress in global feature modeling [19, 8, 21, 2, 26, 22, 23, 4]. However,
multimodal video Transformers [16, 1, 3, 5] have shown a competitive
performance as a normal reflection of the remarkable Vision Transformer (ViT)
success witnessed on image recognition [27]. Although, supervised multimodal
learning provides good performance [19, 8, 21, 2, 3, 7, 6, 1, 5], some methods
tend to leverage the huge unlabeled video data to learn multimodal networks in
self-/weakly supervised scheme [16, 28, 26, 29]

Modality Fusion. One of the main components of any multimodal algorithm is its
modality fusion module that is used to fuse and derive the cross-modality
representations for the final prediction. Several fusion methods have been
proposed and can be categorized into early, mid, and late fusion. These fusion
approaches are adopted and empirically studied in more detail in [18, 1, 2, 3,
7, 6]. The fusion methods vary between a simply modality feature aggregation,
concatenation, and cross-modality attention [1, 26]. Recently, several
multimodal methods [1, 4, 5] leverage the attention mechanism to perform
cross-modality feature modeling resulting in a remarkable improvement in
action recognition.

Modality Dropout and Gating. Several methods, including the well-known dropout
[30], have been proposed to improve the training process of the deep networks,
which prevents DNNs from overfitting. Model overfitting becomes more
challenging when it comes to multimodal training particularly for video
understanding, as concluded in [7]. This is due to the small video dataset
size as compared to image-based datasets and due to the bias towards majority
classes resulting by highly imbalanced datasets. Therefore, several methods
use random modality dropout, which is first used in [31], for improving the
multimodal gesture recognition training. By contrast, a drop input token is
proposed in [16] for reducing the training time at the cost of the prediction
quality. Other studies are presented in [6, 32], which propose a gated
multimodal unit for text-vision fusion in case two modalities are relevant.

Our work differs from previous methods in the following aspects. First, our
vision model is a convolution-free two-stream Transformer that derives
spatiotemporal features with few parameters as compared to [1, 5, 16]. Second,
unlike [16, 6, 32], our proposed IMD is learnable that learns to completely
drop the irrelevant modality whereas the relevant modalities are passed for
the fusion step in both training and inference phase. IMD aims to learn
multimodality representations from modality-specific annotated datasets. Thus,
the proposed IMD is different in its structure, purpose, and training
approach. Third, to the best of our knowledge, our work is the first attempt
to exploit the power of both NLP and audio Transformers to automatically
annotate audio modality that significantly boosted the action recognition
performance. Fourth, the audio-visual correspondence (AVC) [8], which
considers the dataset as audio-visual corresponding, learns to align auditory
and visual modalities and decides whether they are extracted from the same
time track of the same video, on the basis of only positive and negative
labels. However, our relevance network (RN) does not learn the modality
correspondence or alignment. Instead, it learns to simply decide whether the
audio modality is class-relevant to the visual modality or not based on the
generated label dictionary SAVLD regardless of which videos are used to sample
the input modalities. Finally, our proposed intra-class cross-modality
augmentation is different from [33], which is an inter-class sample mixing. It
is also different from [26], which exchanges audio and video tracks for one
time as a preprocessing step to ensure each input sample involves auditory and
visual modalities.

Figure 2: Proposed Multimodal Learning Framework. It uses three modalities: RGB, flow, and audio. Audio modality features are obtained by an off-the-shelf Transformer trained on the audio dataset AudioSet. Here, audio Transformer parameters are not trainable except the added AB which works as a bridge between our trainable network and the pre-trained frozen audio model. The trainable IMD is the main part of this framework, it estimates the auditory-visual modality relevance of the input video using the RN, and then it decides whether to fuse the audio modality or to completely drop it out using thresholding and masking layers (Fig. 4). IMD is optimized with binary cross-entropy. The IOU is computed between the audio Transformer multi-label predictions and the corresponding $k$ relevant audio labels generated by SAVLD. The IOU is normalized into the [0–1] range to match the relevant network (RN) output range. Table 1: Samples of video labels in Kinetics400 dataset and their most relevant labels in AudioSet dataset selected by KNN after a semantic label embedding mapping with BERT, when $k=5$. Label | Relevant AudioSet Labels
---|---
applauding | speech;applause;whistling;chime;clapping
clean and jerk | fill with liquid;pump liquid;filing rasp;rumble;rustle
feeding birds | wild animals;insect;mosquito;bird;patter
sniffing | whimper;growling;cheering;whispering;rattle
sneezing | gurgling;snoring;babbling;gargling;rapping
tickling | whispering;rustle;cheering;growling;screaming
yawning | babbling;rapping;frying food;gurgling;snoring
writing | writing;speech;typing;chatter;mechanisms

##  3 Proposed Multimodal Framework

###  3.1 Label Dictionary SAVLD

SAVLD is a cross-dataset label dictionary in which each label in an audio-
specific dataset is semantically mapped into one or more labels on the vision-
specific dataset. We achieved this preprocessing step by using the well-known
NLP model BERT [14]. We obtain the sentence-based semantic embeddings of each
textual label in audio and video datasets because most human activity video
and audio labels involve more than one word. Each video label embedding is
matched with all audio label embeddings. The most $k$ relevant audio labels
are picked by using the k nearest neighbor algorithm. However, the matching
process does not require much time because the number of labels in two
datasets is small, that is, Kinetics400 has 400 class labels and AudioSet has
527 class labels. Thus, finding the closest $k$ similar labels can be done
simply by using any distance metric. We obtain a dictionary in which each
video label has $k$ audio labels of which they all are considered semantically
similar. This dictionary can be considered $U_{v}$ overlapped clusters, where
$U_{v}$ denotes the number of video dataset classes. The cluster overlap is
represented here in terms of audio labels only. A conceptual overview of
building SAVLD is illustrated in Fig.1, whereas a set of real
Kinetics400-AudioSet mapped labels is shown in Table 1.

###  3.2 Visual-Modality Two-Stream Transformer

To leverage the pre-trained knowledge on large image datasets [34], we adopt
several parts from the image-based ViT [27] and the video Transformer [35] to
build our two-stream video Transformer.

Tokenization. Similar tokenization part of [35] is used in which the input
visual-modality sample $\textbf{V}\in\mathbb{R}^{T\times C\times W\times H}$,
of $T$ frames and $C=3$ for RGB modality and $C=2$ for optical flow modality,
is tokenized into $T\times N$ patches in which each frame is projected to
$N=HW/P^{2}$ patches each of which has a spatial dimension of $P^{2}$. After
organizing the obtained $T\times N$ patches, each patch is mapped to
$\mathbf{x}_{j,i}\in\mathbb{R}^{D}$, where $j=1,\cdots,T$ , $i=1,\cdots,N$,
and $d$ denotes the embedding size to be used throughout the visual two-stream
transformer.

$\mathbf{z}_{j,i}^{0}=\mathbf{x}_{j,i}\mathbf{E}+\mathbf{E}^{pos}_{j,i},$ (1)

where for each patch $\mathbf{x}_{j,i}$, a spatiotemporal positional embedding
$\mathbf{E}^{pos}_{j,i}\in\mathbb{R}^{D}$ is added. Additionally, a learnable
classification embedding $\mathbf{z}_{0,0}^{0}\in\mathbb{R}^{D}$ is added to
learn the visual-modality semantic video representation.

Figure 3: Visual-Modality Transformer Encoder. It consists of $L-2$ Spatial S
blocks and $2$ Spatio-Temporal ST blocks. STs are spatial–temporal factorized-
based attention blocks to preserve the Transformer efficiency

Visual-Modality Transformer Encoder. Our visual two-stream Transformer encoder
involves $L$ stacked blocks, each of which is a spatial-based module except
the last two blocks $\\{L,L-1\\}$ which involve a spatiotemporal factorized
self-attention to better model the spatiotemporal knowledge. Using this
approach, our Transformer provides competitive performance with few parameters
as compared to [35] which involves spatiotemporal attention in each block.
Each Transformer block consists of a multiheaded self-attention (MSA) [36],
LayerNorm (LN) [37], and multilayer perception (MLP) modules. The first $L-2$
blocks of the proposed Transformer encoder can be formulated as:

$\displaystyle\mathbf{z}_{j,i}^{\ell}=\mathrm{\large
MSA}\left(\mathrm{LN}(\mathbf{z}_{j,i}^{\ell-1})\right)+\mathbf{z}_{j,i}^{\ell-1},$
(2)

where $\ell\in\\{0,...,L-2\\}$. The last two blocks can be formulated as:

$\displaystyle\grave{\mathbf{z}}_{j,i}^{\ell}=\mathrm{\large
LR}\left(\mathrm{\large MSA}^{time}(\mathrm{\large
LN}(\mathbf{z}_{j,i}^{\ell-1}))\right)+\mathbf{z}_{j,i}^{\ell-1},$ (3)
$\displaystyle\mathbf{z}_{j,i}^{\ell}=\mathrm{\large
MSA}^{space}\left(\mathrm{\large
LN}(\grave{\mathbf{z}}_{j,i}^{\ell}\right)+\grave{\mathbf{z}}_{j,i}^{\ell},$

where $\mathrm{\large LR}$ refers to a linear layer that added after each
temporal module. The classification embeddings of RGB and optical flow streams
are fused with two fully-connected layers each of which is followed by ReLU
activation.

###  3.3 Audio Transformer

We choose the pretrained AST [38] because it is the most closely similar
Transformer to our video Transformer. We adopt AST to extract high
semantically audio features and to perform a multi-label prediction. AST has
been built on the basis of ViT [27] in which it treats audio modality
processing as a vision task because it converts input audio modality into
$128\times 100t$ spectrogram. AST is mostly identical to ViT except for a few
changed Transformer parameters, where AST uses patch size $P=16$ with a stride
of $10$. Overall, we follow the audio input normalization and inference
settings in [38] because we adopt AST as a downstream task with frozen
parameters. We obtain the output predictions of AST on its audio dataset
(_i.e_., AudioSet) to obtain the video label using the label dictionary SAVLD.
Additionally, we obtain the learned class embedding which is then passed to a
learnable audio bottleneck.

Audio Bottleneck (AB). We add a learnable audio bottleneck as a semantic
representation bridge between audio and video datasets, such as AudioSet-
Kinetics400. This condition is because all audio model parameters are frozen
while training our video-based action recognition model. This bottleneck plays
a significant role by transforming the audio features from audio-dataset-
oriented predictions to video-oriented predictions in the training phase.
Simply, it consists of one $\mathrm{\large LN}$ and two $\mathrm{\large LR}$
each of which is followed by ReLU activation.

###  3.4 Trainable Irrelevant Modality Dropout (IMD)

IMD is the main part of our framework that automatically drops out the
irrelevant audio modality. IMD consists of two main modules: The first module
is a simple relevance network that receives the concatenated modalities of
auditory and visual Transformers and decides whether they are relevant. The
output of this network is sigmoidal predictions representing the relevance
level. The sigmoidal predictions are forwarded to a binary thresholding layer
to decide whether audio-modality embedding can be fused with visual-modality
embedding for the final classification or to drop it. The thresholding and dot
product layers compose a masking layer in which the irrelevant modality is
multiplied with $0$. The proposed IMD can be formulated as follows:

$\displaystyle rev=\sigma(\mathrm{\large
MLP}(\mathrm{LN}(\mathbf{z}_{0,0}^{L}))+\mathbf{z}_{0,0}^{L}),$ (4)
$\displaystyle\delta=\begin{cases}0,&\text{if }z<\alpha\\\
rev,&\text{otherwise}\end{cases},$ (5)
$\displaystyle{z}_{0,0}^{(a)L},{z}_{0,0}^{(v)L}=split({z}_{0,0}^{L}),$ (6)
$\displaystyle{z}_{0,0}^{L}=Concate(({z}_{0,0}^{(a)L}\cdot\delta),{z}_{0,0}^{(v)L}),$
(7) $\displaystyle z_{av}=\mathrm{\large
MLP}(\mathrm{LN}(\mathbf{z}_{0,0}^{L})),$ (8)

where $\alpha$ denotes the defined threshold for the audio-visual modality
relevance and $z_{av}$ is the fused output. Figure 4 illustrates the IMD
network and its main layers. Although it is designed for audio-modality
dropout, it may be adopted for other modalities in the case of textual-vision
applications whenever possible to easily build a cross-dataset label
dictionary as the SAVLD we built for audio-visual training.

Figure 4: Proposed Irrelevant Modality Dropout (IMD). It is composed of two
main networks. Relevance network (RN) which estimates the class-based audio-
vision relevance of the input clip. The IMD network performs two steps: the
output of the RN is thresholded and masked as in Eqs. 5–7. Consequently, the
audio modality is completely dropped or fused as in Eq. 8.

###  3.5 Framework Optimization

Our framework has two learning targets; optimizing the modality Transformers
to learn the final human activity and optimizing the relevance network to
accurately estimate the audio-video relevance. Therefore, we adopt two cross-
entropy losses. The first loss is a binary cross-entropy for optimizing the RN
by using the label dictionary SAVLD. The second loss is the default category
cross-entropy for the classification learning, which is weighted with the RN
loss to form the final loss. Before the category cross-entropy is applied to
the output fused class embedding, it is first fed into the final
classification MLP as follows:

$\displaystyle Y_{v}=Softmax(\mathrm{\large MLP}(\mathrm{LN}(z_{av})))$ (9)

###  3.6 Intra-Class Cross-Modality Augmentation

We propose an intra-class cross-modality augmentation for human activity
recognition, which has shown a positive impact in the Transformer training
because human activity recognition on video does not require audio-video
correspondence as in the case of speech-visual recognition. Thus, in our work,
we focus on relevance rather than correspondence in which the audio modality
should be relevant to the visual class label regardless of its video source.
Therefore, we run a cross-modality intra-class augmentation by pairing audio
and visual modalities from different videos but belong to the same class.

##  4 Experimental Results

###  4.1 Experiments Setup

Datasets and Evaluation Metrics. Since our aim is to tackle the problem of
multimodal video understanding in vision-specific datasets, we have conducted
an extensive experiments on two well-known video datasets. Kinetics400 [10] is
downloadable from YouTube, where we used in this work $241,722$ videos of
$400$ classes for training, and $19,877$ videos for validation. The videos of
Kinetics400 are trimmed in $10$ seconds. UCF-101 [11] contains $13,320$ videos
of $101$ classes. Its video length about $7.2$ seconds. It is split into three
training/validation splits. We adopted AST audio model pretrained on AudioSet
[12], which is considered one of the largest available audio dataset. It
contains $1.8$M sound clips of $10$-second length. We followed the most used
evaluation metric Top-$1$ and Top-$5$. We sampled $3$ spatial crops and $4$
temporal clips from each video and averaged their obtained accuracy. We
compared our multimodal video Transformer with state-of-the-art methods in
terms of computational cost floating-point operation and the number of
parameters

Implementation Details. The entire proposed framework is built in Pytorch
[39]. The training and evaluation phases are conducted on two machines
(_i.e_., $8$ $\times$ TITAN RTX and $2$ $\times$ RTX $2080$ Ti). Unless
otherwise specified, we ensure to run each set of the framework training
instances by using the same settings on the same machine. The GPU’s memory is
saved for the visual RGB and flow stream because AST is frozen during training
and inference, except its bottleneck AB. The input RGB modality is sampled as
a clip with spatial resolution of $3\times 224\times 224$ and temporal length
of $8$-frames. Similarly, flow modality is sampled with the same RGB spatial
and temporal settings except for the channel of $2$ to represent the $x$ and
$y$ flow components. We trained each instance with a global input batch of
$64$ for $15$ epochs. SGD optimizer is utilized with an initial learning rate
of $0.005$ and weight decay of $1e-4$. Our visual video Transformer uses a
patch of $16\times 16$ size, block depth $12$, embedding size $D=768$, and
self-attention of $12$ heads. The label dictionary SAVLD is saved as a text
file, where each row represents the video dataset label and its $k$ relevant
audio labels.

###  4.2 Ablation Study

In this section, we conduct a set of extensive ablation studies on the
proposed framework variants to explore their optimal settings. We start with
SAVLD building techniques and then, the modality combination on the visual-
modality specific dataset performance. Finally, the effect of IMD elements is
studied.

####  4.2.1 Semantic Audio-Video Dictionary Quality

Embedding Extractor. We chose the well-known NLP model termed BERT to extract
the best sentence-based semantic features for each textual label due to two
main reasons: first, BERT has been trained in an unsupervised manner and has
undergone millions of unorganized text from real-world sources. Thus, it can
provide better semantic features that can detect the most relevant text.
Secondly, unlike word2vec [40] and GloVe [15], BERT as a text feature
extractor is a context-aware text analyzer that provides different embedding
for the same word depending on its position in the sentence. The performance
of the proposed framework on Kinetics400 using SAVLD constructed with the
three embedding extractors is reported in Table 10.

Table 2: Impact of embedding extractor and similarity distance methods on the AudioSet-Kinetics400 SAVLD quality. Embed. Extractor | Similarity | Top-1 (%) | Top-5 (%)
---|---|---|---
word2vec | Euclidean | $80.9$ | $94.3$
GloVe | Euclidean | $81.2$ | $94.5$
BERT | Euclidean | $82.8$ | $95.7$
BERT | Manhattan | $82.6$ | $95.5$
BERT | Cosine | $82.8$ | $95.9$

Embedding Matching. Three vector similarity matching methods, namely,
Euclidean, Manhattan, and cosine similarity distance are evaluated. The lower
part of Table 10 reports their performance on BERT embedding matching.
Notably, the similarity distance function does not show large effect on the
framework performance.

####  4.2.2 Modality Combination

Modality combination on human activity recognition with the proposed framework
is evaluated. Table 3 reports three modality combination schemes, namely,
unimodal, bimodal, and multimodal on all the possible three modalities, audio,
RGB, and optical flow. The use of more than one modality can remarkably
improve video-based action recognition However, RGB remains the modality with
the largest effect on recognition accuracy.

Table 3: Performance of the proposed framework variants in terms of modality combination on Kinetics400. Scheme | Modality | Top-1 (%) | Top-5 (%)
---|---|---|---
Uni-modal | Audio | $26.8$ | $45.1$
RGB | $79.1$ | $93.9$
Bi-modal | Audio-Flow | $56.7$ | $72.8$
Audio-RGB | $81.6$ | $94.8$
RGB-Flow | $81.1$ | $94.3$
Multi-modal | Audio-RGB-Flow | $82.8$ | $95.7$

####  4.2.3 Trainable Irrelevant Modality Dropout (IMD)

Overlapping Loss. The overlap between the output predictions of the pre-
trained audio model AST and the corresponding $K$ relevant textual audio
labels in the SAVLD is computed as ground truth for the RN optimization. The
overlap is then normalized between $0$ and $1$ to match the sigmoidal
prediction of RN. We tested two overlap loss functions IOU and Dice overlap
[41]. The upper part of Table 4 shows the performance of two functions. The
overlap function does not show a large effect on the framework performance
because this loss is performed only to build the ground truth for the binary
cross-entropy rather than to optimize the framework itself.

Most $k$ Audio Relevant Labels. The overlap between AST predictions and the
relevant textual label vectors is computed with same size of two vectors.
However, we tested the impact of using different sizes of $k$ against $Y_{a}$.
Table 4 shows that $10-20$ is the best $k-Y_{a}$ ratio for ground truth
computation.

Relevance Thresholding ($\alpha$). The sigmoidal predictions of the RN
indicate the audio relevance to the visual modality on the input video sample.
Thus, these sigmoidal predictions range between 0 and 1, which are going to be
used by the thresholding layer in the dropout module. The effect of various
threshold values on the dropout rate affecting the framework performance is
reported in Table 4. Notably, when $\alpha=0.25$, the framework shows better
results.

Table 4: Impact of different IMD settings on the proposed framework performance on Kinetics400. Loss |  $K$-$Y_{a}$ | $\alpha$ | M/W | Top-1 (%) | Top-5 (%)
---|---|---|---|---|---
IOU |  $10$-$10$ | $0.50$ | M | $81.7$ | $95.4$
Dice |  $10$-$10$ | $0.50$ | M | $81.6$ | $95.5$
IOU | 20-10 | $0.50$ | M | $81.7$ | $94.8$
IOU | 10-20 | $0.50$ | M | $82.3$ | $95.5$
IOU | 20-20 | $0.50$ | M | $81.9$ | $95.1$
IOU |  $10$-$20$ | 0.25 | M | 82.8 | 95.7
IOU |  $10$-$20$ | 0.75 | M | $80.9$ | $94.4$
IOU |  $10$-$20$ | $0.25$ | W | $80.4$ | $94.3$

Irrelevant Modality Masking Scheme. Two types of masking schemes in IMD are
evaluated. The first scheme is the default explained masking (M) method in
this work in which the relevant sigmoidal score larger than $\alpha$ is simply
multiplied with the audio embeddings received from the audio Transformer,
whereas the relevance is less than $\alpha$ is multiplied by $0$ to completely
mask the audio embeddings from the fusion step. The second scheme is applying
the same weighting (W) method in two cases when relevant sigmoidal scores are
larger or less than $\alpha$. The lower part of Table 4 shows that the
weighting relevance method provides less performance compared with the masking
method.

###  4.3 Comparison with State-of-the-art

Modality Fusion Methods. We compare our proposed IMD to other fusion methods
in the literature. When we adopt the relevant fusion methods, we disable the
RN learning to ensure the fusion process is performed on each audio-visual
input sample. Several fusion methods, including SE Gate [7], NL Gate [7],
late-concat [7], AVC [8], and GMU [6], are evaluated. Table 5 reports the
performance of our framework when using each method of these fusion methods.

Table 5: Performance comparison of the proposed IMD against other relevant fusion methods on Kinetics400. IMD∗ refers to apply the proposed intra-class cross-modality augmentation. Fusion Method | Top-1 (%) | Top-5 (%)
---|---|---
SE Gate [7] | $79.2$ | $94.1$
NL Gate [7] | $80.8$ | $94.8$
late-concat [7] | $78.6$ | $93.9$
AVC [8] | $79.5$ | $94.2$
GMU [6] | $80.9$ | $94.8$
IMD (ours) | 82.3 | 95.1
IMD∗(ours) | 82.8 | 95.7
Table 6: Comparison to prior uni and multimodal action recognition methods on Kinetics400 dataset. Most compared methods are Transformer-based except the CNN-based methods: MoViNet-A5, SlowFast, and X3D. All methods are organized by their size in which the upper part involves the Transformers of base (B) size, whereas the lower part involves the ones with large (L) size. Model | P-train | Top-1 | Top-5 | GFLOPs
---|---|---|---|---
MoViNet-A5 [42] | IN-1K | $78.2$ | N/A | $29$
SlowFast 16x8 [43] | IN-1K | $79.8$ | $93.9$ | $7,020$
VATT-B [16] | IN-1K | $79.6$ | $94.9$ | $9,090$
X3D-XXL [44] | IN-1K | $80.4$ | $94.6$ | $5,823$
TimeSFormer-B [35] | IN-21K | $78.0$ | $93.7$ | $590$
Swin-B [45] | IN-1K | $80.6$ | $94.6$ | $3,384$
Swin-B [45] | IN-21K | $82.7$ | $95.5$ | $3,384$
X-ViT-16x [46] | IN-21K | $80.2$ | $94.7$ | $850$
IMD-B (ours) | IN-21K | 82.8 | 95.7 | $4,464$
AVSlowFast [47] | - | $78.8$ | $93.6$ | $7,020$
G-Blend [7] | Sport1M | $80.4$ | $94.8$ | $3,303$
MBT (AV) [1] | IN-21K | $80.8$ | $94.6$ | N/A
TimeSFormer-L [35] | IN-21K | $80.7$ | $94.7$ | $7,140$
Mformer-L [48] | IN-1K | $80.2$ | $94.8$ | $35,553$
ViViT-L [49] | IN-21K | $80.6$ | $94.7$ | $17,352$
VATT-L [16] | IN-1K | $82.1$ | $95.5$ | $29,800$
Swin-L [45] | IN-21K | $83.1$ | $95.9$ | $7,248$
IMD-L (ours) | IN-21K | 84.2 | 96.3 | $8,232$
Table 7: Comparison to to prior multimodal action recognition methods on UCF-101. Methods are organized by pretrained data. Model | Pretrained Weights | Top-1 (%)
---|---|---
CoViAR [25] | IN-1K | $90.4$
TSN [50] | IN-1K | $94.2$
CoViAR + OF [25] | IN-1K | $94.9$
Two-stream fusion [18] | IN-1K | $92.5$
MM-ViT [5] | IN-21K | $95.4$
IMD-B (ours) | IN-21K | 95.9
I3D [10] | Kinetics | $95.4$
MM-ViT [5] | Kinetics | $98.9$
IMD-B (ours) | Kinetics | 99.1

Action Recognition. In this part, we evaluate our overall framework against
the state-of-the-art multimodal methods for video-based action recognition on
two datasets Kinetics400 [10] and UCF-101 [11]. Several uni/multimodal action
recognition methods have shown good performance including the CNN-based
models: TSN [50], I3D [10], AVSlowFast [47], SlowFast [43], X3D [44],
MoViNet-A5 [42], and G-Blend [7] along with the Transformer-based models: VATT
[16], TimeSFormer [35], Swin Video Transformer [45], X-ViT [46], MBT [1],
Mformer [48], ViViT-L [49], MM-ViT [5]. However, our multimodal Transformer-
based framework outperforms all aforementioned methods on the two datasets
with considerable margins. Table 6 reports the action recognition performance
on Kinetics400 dataset, and Table 9 reports the results on UCF-101 dataset.
Unlike our framework, most multimodal video-based deep models are usually
evaluated on multimodality datasets. However, we compared our method with the
best available models on these modality-specific datasets in which our IMD
models provides a Top-1 performance boost of the most similar multimodal
methods (highlighted in LightBlue color) G-Blend and MBT with $\sim$$3.8\%$
and $\sim$$3.4\%$, respectively.

###  4.4 Discussion and Limitations

On the basis of the reported results and overall framework building, few
worth-mentioning points need to be declared. First, our framework is a
convolution-free Transformer-based neural network. It shows a larger
performance enhancement on large datasets compared with its performance on
small datasets, as shown in Tables 6 and 9. Second, our designed visual-
modality two-stream Transformer attempts to cut the number of Transformer
parameters while preserving the best spatiotemporal representation learning.
Third, audio-modality masking based on the sigmoidal scores provided by the RN
is relayed on the audio Transformer accuracy on its audio dataset (_e.g_.,
AudioSet). For example, as reported in Table 4, the best $\alpha=0.25$. This
finding is because the AST and most audio recognition models in the literature
still do not show as high recognition accuracy as visual-based models because
audio modality is more challenging compared with visual data. AST [38]
provides mAP of $0.485$. In this manner, we think that threshold $\alpha$ can
be higher when a better pretrained audio model is used. Fourth, the proposed
intra-class cross-modality augmentation method shows a positive effect on the
framework training. However, its main effectiveness is related to the
framework design, which considers that the audio modality is noisy and
unlabeled. Thus, intra-class cross-modality augmentation improves the
performance by pairing visual and auditory modalities. Finally, even though we
evaluate the quality of the SAVDL dictionary indirectly using its impact on
the action recognition performance. However, it is of future interest to build
a new evaluation method that can effectively evaluate the quality of text-to-
text similarity matching.

##  5 Conclusion

In this work, we introduced a novel audio-visual action recognition framework.
It leverages the power of NLP and audio recognition models to automatically
annotate audio modality in the input unlabeled audio for video-based action
recognition in an online training manner. The proposed framework features a
novel IMD that completely drops the irrelevant audio modality based on a
recommendation provided by an RN, which in turn learns to estimate the audio
modality relevance to the input video category. Our framework aims to create
cross-modality high semantic features that can help to bridge the relevance
gap between the visual and auditory modalities in videos. As long as a strong
pre-trained model on a large dataset is available to be used as a transfer
learning task, this training scheme can be adopted by other bimodality and
multimodality models. Thus, the same idea can be further investigated for
other modalities, such as textual-auditory and textual-visual model training,
which is part of our future interest.

Acknowledgments This work was supported in part by the National Natural
Science Foundation of China under grants U21A20455, 61972265, 61872429 and
11871348, the Natural Science Foundation of Guangdong Province of China under
grant 2020B1515310008, Educational Commission of Guangdong Province of China
under grant 2019KZDZX1007, and the Shenzhen Basis Research Project under grant
JCYJ20180305125521534.

## References

  * [1] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun.  Attention bottlenecks for multimodal fusion.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * [2] Olga Zatsarynna, Yazan Abu Farha, and Juergen Gall.  Multi-modal temporal convolutional network for anticipating actions in egocentric videos.  In CVPR Workshops, pages 2249–2258, June 2021.
  * [3] Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, and Yale Song.  Parameter efficient multimodal transformers for video representation learning.  In International Conference on Learning Representations, 2021.
  * [4] Lu Chi, Guiyu Tian, Yadong Mu, and Qi Tian.  Two-Stream Video Classification with Cross-Modality Attention.  In ICCVW, pages 4511–4520. IEEE, oct 2019.
  * [5] Jiawei Chen and Chiu Man Ho.  MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition.  arXiv, aug 2021.
  * [6] John Arevalo, Thamar Solorio, Manuel Montes-Y-Gómez, and Fabio A. González.  Gated multimodal units for information fusion.  5th International Conference on Learning Representations, ICLR 2017 - Workshop Track Proceedings, 2019.
  * [7] Weiyao Wang, Du Tran, and Matt Feiszli.  What Makes Training Multi-Modal Classification Networks Hard?  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 12692–12702, 2020.
  * [8] Relja Arandjelovic and Andrew Zisserman.  Look, Listen and Learn.  In ICCV, volume Octob, pages 609–617, oct 2017.
  * [9] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Scaling egocentric vision: The epic-kitchens dataset.  In European Conference on Computer Vision (ECCV), 2018.
  * [10] Joao Carreira and Andrew Zisserman.  Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset.  In CVPR, pages 4724–4733. IEEE, jul 2017.
  * [11] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild.  arXiv, dec 2012.
  * [12] Jort F. Gemmeke, Daniel P.W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio Set: An ontology and human-labeled dataset for audio events.  ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, pages 776–780, 2017.
  * [13] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman.  VGGSOUND : A large-scale audio-visual dataset.  In ICASSP, pages 721–725, 2020.
  * [14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova Google, and A I Language.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  Naacl-Hlt 2019, 2018.
  * [15] Paul M. Brennan, James J.M. Loan, Neil Watson, Pragnesh M. Bhatt, and Peter Alwyn Bodkin.  Pre-operative obesity does not predict poorer symptom control and quality of life after lumbar disc surgery.  British Journal of Neurosurgery, 31(6):682–687, 2017.
  * [16] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong.  VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text.  In NeurIPS, 2021.
  * [17] Karen Simonyan and Andrew Zisserman.  Two-Stream Convolutional Networks for Action Recognition in Videos.  Advances in Neural Information Processing Systems, jun 2014.
  * [18] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.  Convolutional Two-Stream Network Fusion for Video Action Recognition.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Decem(i):1933–1941, 2016.
  * [19] Hamid Reza Vaezi Joze, Amirreza Shaban, Michael L. Iuzzolino, and Kazuhito Koishida.  MMTM: Multimodal transfer module for CNN fusion.  CVPR, pages 13286–13296, 2020.
  * [20] Hu Xu, Gargi Ghosh, Po-yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, and Luke Zettlemoyer.  VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding.  In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4227–4239, Stroudsburg, PA, USA, 2021. Association for Computational Linguistics.
  * [21] Relja Arandjelović and Andrew Zisserman.  Objects that Sound.  In European Conference on Computer Vision (ECCV), 2018.
  * [22] Ahmed Hussen Abdelaziz, Barry John Theobald, Paul Dixon, Reinhard Knothe, Nicholas Apostoloff, and Sachin Kajareker.  Modality Dropout for Improved Performance-driven Talking Faces.  ICMI 2020 - Proceedings of the 2020 International Conference on Multimodal Interaction, pages 378–386, 2020.
  * [23] Alejandro Cartas, Jordi Luque, Petia Radeva, Carlos Segura, and Mariella DImiccoli.  Seeing and hearing egocentric actions: How much can we learn?  ICCVW, pages 4470–4480, 2019.
  * [24] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and DIma Damen.  EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition.  In ICCV, volume 2019-Octob, pages 5491–5500. IEEE, oct 2019.
  * [25] Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, R. Manmatha, Alexander J. Smola, and Philipp Krahenbuhl.  Compressed Video Action Recognition.  In CVPR, pages 6026–6035. IEEE, jun 2018.
  * [26] Yu Wu and Yi Yang.  Exploring heterogeneous clues for weakly-supervised audio-visual video parsing.  In CVPR, 2021.
  * [27] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [28] Jean Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey de Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman.  Self-supervised multimodal versatile networks.  Advances in Neural Information Processing Systems, 2020-Decem(ii):1–19, 2020.
  * [29] Andrew Owens and Alexei A. Efros.  Audio-Visual Scene Analysis with Self-Supervised Multisensory Features.  In ECCV, volume 11210 LNCS, pages 639–658, 2018.
  * [30] Srivastava Nitish, Hinton Geoffrey, Krizhevsky Alex, Sutskever Ilya, and Salakhutdinov Ruslan.  Dropout: A Simple Way to Prevent Neural Networks from Overfitting.  Journal of Machine Learning Research, 2014.
  * [31] Natalia Neverova, Christian Wolf, Graham Taylor, and Florian Nebout.  ModDrop: Adaptive Multi-Modal Gesture Recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(8):1692–1706, aug 2016.
  * [32] Feiran Huang, Kaimin Wei, Jian Weng, and Zhoujun Li.  Attention-Based Modality-Gated Networks for Image-Text Sentiment Analysis.  ACM Transactions on Multimedia Computing, Communications, and Applications, 16(3):1–19, sep 2020.
  * [33] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  MixUp: Beyond empirical risk minimization.  In International Conference on Learning Representations(ICLR), 2018\.
  * [34] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  IJCV, 115:211–252, 2015.
  * [35] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In ICML, volume 139, pages 813–824, Jul 2021.
  * [36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6000–6010, California, USA, 2017. ,.
  * [37] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer Normalization.  arXiv, 2016.
  * [38] Yuan Gong, Yu-An Chung, and James Glass.  AST: Audio Spectrogram Transformer.  arXiv, 2021.
  * [39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An imperative style, high-performance deep learning library.  In Advances in Neural Information Processing Systems, 2019.
  * [40] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean.  Distributed Representations of Words and Phrases and their Compositionality.  In NIPS, oct 2013.
  * [41] W.R. Crum, Oscar Camara, and D.L.G. Hill.  Generalized Overlap Measures for Evaluation and Validation in Medical Image Analysis.  IEEE Transactions on Medical Imaging, 25(11):1451–1461, nov 2006\.
  * [42] Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong.  MoViNets: Mobile Video Networks for Efficient Video Recognition.  pages 16020–16030, 2021.
  * [43] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In ICCV, 2019.
  * [44] Christoph Feichtenhofer.  X3D: Expanding Architectures for Efficient Video Recognition.  CVPR, pages 200–210, 2020.
  * [45] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video Swin Transformer.  arXiv, pages 1–12, 2021.
  * [46] Adrian Bulat, Juan-Manuel Perez-Rua, Swathikiran Sudhakaran, Brais Martinez, and Georgios Tzimiropoulos.  Space-time mixing attention for video transformer.  In NeurIPS, 2021.
  * [47] Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer.  Audiovisual SlowFast Networks for Video Recognition.  arXiv, 2020.
  * [48] Mandela Patrick, Dylan Campbell, Yuki M. Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and João F. Henriques.  Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers.  In Advances in Neural Information Processing Systems, jun 2021.
  * [49] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  ViViT: A Video Vision Transformer.  arXiv, 2021.
  * [50] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal Segment Networks: Towards Good Practices for Deep Action Recognition.  In ECCV 2016, pages 20–36. 2016.
  * [51] Hilde Kuehne, Hueihan Jhuang, E. Garrote, T. Poggio, and T. Serre.  HMDB: A large video database for human motion recognition.  In ICCV, pages 2556–2563. IEEE, nov 2011.

Supplementary File

Semantic Audio-Video Label Dictionary (SAVLD)

For building the SAVLD, we use BERT-base uncased in which the label text is
lowercased before tokenizing it. Therefore, as a preprocessing step, we first
normalized all textual labels into lowercase form. Notably, the resulting
label dictionary SAVLD is not very accurate since the semantic gap between
video and audio datasets is still large. It is also because the number of
classes on video/audio datasets is still considered small (_i.e_. Kinetics400
has $400$ labels and AudioSet has $527$ label). However, using SAVLD
dictionaries, our framework narrows the input noise by using the audio
predictions. Then, it matches the audio scene to its closest similar visual
class regardless of the fact that many audio classes do not have large
relevance. In the training phase, the labeling process is guided by applying
the IOU function between the AST predictions and the corresponding audio
labels to the video label in the SAVLD dictionary. Table 10 shows a small part
of the Kinetics400-AudioSet label dictionary when $k=5$. Additionally, we
visualize the AudioSet labels’ relevance to Kinetics400 and UCF-101 labels in
Fig. 6 and Fig. 8, respectively. The most frequent mapped audio labels to the
vision-specific dataset labels are shown in Fig. 5 and Fig. 7 for Kinetics400
and UCF-101 datasets, respectively.111We implemented our framework in Pytorch
in which we borrowed several parts from the following codebases:
https://github.com/huggingface/transformers https://github.com/YuanGongND/ast
https://github.com/facebookresearch/SlowFast
https://github.com/facebookresearch/TimeSformer
https://github.com/rwightman/pytorch-image-models
https://github.com/unixpickle/audioset
https://github.com/ekazakos/temporal-binding-network
https://github.com/marl/l3embedding
https://github.com/johnarevalo/gmu-mmimdb

Experiments on HMDB51 and Kinetics-Sounds

We further evaluate our method against relevant methods for video-based action
recognition in two more datasets HMDB51 [51] and Kinetics-Sounds [11]. HMDB51
[51] is split into three overlapped splits. It contains $6,766$ videos of $51$
classes with an average length of $3$ seconds. Kinetics-Sounds [8] is a subset
from the original Kinetics400 [10] dataset. It contains $34$ classes in which
each class videos have a remarkable sound signature. Since Kinetics dataset
editions are downloadable from YouTube, its size may vary by time as some
videos may get removed. Herein, we use $19,627$ videos for training and
$1,344$ videos for evaluation. length of $3$ seconds. The average video length
in this dataset is $10$ seconds.

Table 8: Performance comparison to relevant visual-based methods (RGB + Optical Flow) on HMDB51 dataset. Model | Top-1 (%)
---|---
CoViAR [25] | $59.1$
Two-stream fusion [18] | $65.4$
TSN [50] | $69.4$
I3D [10] | $66.4$
CoViAR + OF [25] | $70.2$
IMD-B (ours) | 71.3
Table 9: Performance comparison to relevant methods on Kinetics-Sounds dataset. Model | P-train | Top-1 (%)
---|---|---
L3-Net [8] | IN-1K | $74.0$
SlowFast R101 [43] | IN-1K | $77.9$
AVSlowFast, R101 [47] | IN-1K | $85.0$
MBT (AV) [1] | IN-21K | $85.0$
IMD-B (ours) | IN-21K | 90.48
IMD-B⋆ (ours) | IN-21K | 91.44
Table 10: Samples of the most relevant AudioSet labels to video labels retrieved by semantic sentence-based embeddings mapping by BERT when $K=5$. Dataset | Label | Relevant AudioSet Labels
---|---|---
Kinetics400 | playing guitar | bass guitar;acoustic guitar;guitar;chopping food;electric guitar
applauding | speech;applause;whistling;chime;clapping
belly dancing | rapping;yodeling;synthetic singing;child singing;frying food
canoeing or kayaking | rowboat, canoe, kayak;motorboat, speedboat;skateboard;folk music;sailboat, sailing ship
clean and jerk | fill with liquid;pump liquid;filing rasp;rumble;rustle
country line dancing | female singing;dance music;male singing;salsa music;drum roll
driving car | emergency vehicle;motor vehicle road;filing rasp;car;engine starting
feeding birds | wild animals;insect;mosquito;bird;patter
gargling | gargling;gurgling;snoring;reversing beeps;yodeling
kissing | whispering;typing;cheering;laughter;breathing
pumping gas | frying food;pump liquid;sawing;sanding;filing rasp
recording music | vocal music;music;soundtrack music;wedding music;jingle music
scrambling eggs | singing bowl;spray;thunder;wheeze;tools
sniffing | whimper;growling;cheering;whispering;rattle
sneezing | gurgling;snoring;babbling;gargling;rapping
tickling | whispering;rustle;cheering;growling;screaming
yawning | babbling;rapping;frying food;gurgling;snoring
writing | writing;speech;typing;chatter;mechanisms
whistling | whistling;humming;whistle;whip;siren
| welding | gears;scissors;drill;boiling;bicycle

Table 8 reports the performance of several visual action recognition methods
including CoViAR [25], Two-stream fusion [18], TSN [50], I3D [10], and CoViAR
+ OF [25]. Notably, our method provides a slight performance boost on HMDB51
because our method is a Transformer-based framework that shows better
improvement on large datasets in terms of number of videos per class. However,
our framework provides top-1 of $71.3\%$ which is better compared with CoViAR
+ OF [25] with a performance boost of $\sim$$1.1\%$. We compare our method
with several methods on the visual-audio annotated dataset Kinetics-Sounds.
This dataset was first used in [8] as a subset of the main Kinetics400 dataset
[10]. Our framework provides a significant boost in this dataset, where it
provides top-1 of $90.48\%$ and $91.44\%$ with and without intra-class cross-
modality augmentation, respectively. Since this dataset is an audio-video
annotated in which audio and video are mostly relevant in each video, the
cross-modality augmentation does not improve the performance. This interprets
our finding regarding this augmentation method, where it provides most
performance boosts on datasets with low audio-video relevance. Therefore, in
our method, cross-modality augmentation is particularly applied for improving
the video-based human activity recognition on vision-specific videos.

The Visual Two-Stream Transformer

In this part, we report the performance of three Transformer variants of the
proposed two-stream visual Transformer. In order to leverage the pretrained
ImageNet knowledge, we have adopted several parts from ViT [27] in terms of
number of Transformer encoder blocks, embedding size, number of self-attention
heads, input dimensions. Our Transformer scales properties are almost similar
to the [27], _i.e_. small, base, and large, except the spatiotemporal encoder
blocks. We used one spatiotemporal block on the small encoder instance as it
involves $8$ blocks, whereas we add $2$ and $4$ spatiotemporal blocks for base
and large instances as they involve $12$ and $24$ blocks, respectively. Table
11 reports the recognition performance on Kinetics400 dataset as well as the
Transformer instances’ costs in terms of number of parameters and GFLOPs.

Table 11: Performance of the proposed visual two-stream Transformer with different size. Three Transformer instances are trained, where each of which is initialized with its ViT corresponding ImageNet-21K weights. The performance is reported on Kinetics400. Number of parameters is reported in million. T. Size | Top-1 (%) | Top-5 (%) | Params | GFLOPs
---|---|---|---|---
Small | $78.8$ | $92.8$ |  $47.9$$\times$$2$ | $2,871$
Base | $81.1$ | $94.3$ |  $88.6$$\times$$2$ | $4,464$
Large | $82.6$ | $95.2$ |  $173.7$$\times$$2$ | $8,232$
Figure 5: The most 50 frequent audio labels in AudioSet mapped to Kinetics400
labels when $k=50$. Figure 6: The heatmap of the semantic relevance estimated
by BERT between Kinetics400 labels and AudioSet labels. The darker the more
relevant. Figure 7: The most 50 frequent audio labels in AudioSet mapped to
UCF-101 labels when $k=50$. Figure 8: The heatmap of the semantic relevance
estimated by BERT between UCF-101 labels and AudioSet labels. The darker the
more relevant.