# AMA-GCN: Adaptive Multi-layer Aggregation Graph Convolutional Network for
Disease Prediction

Hao Chen1,2 Fuzhen Zhuang3,6111corresponding authors Li Xiao1,2,5∗ Ling Ma1
Haiyan Liu4 Ruifang Zhang4 Huiqin Jiang1∗&Qing He1,2
1Zhengzhou University, Zhengzhou, China
2Key Lab of Intelligent Information Processing of Chinese Academy of Sciences
(CAS), Institute of Computing Technology, CAS, Beijing 100190, China
3Institute of Artificial Intelligence, Beihang University, Beijing 100191,
China
4The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
5Ningbo Huamei Hospital, University of the Chinese Academy of Sciences,
Ningbo, China
6Xiamen Data Intelligence Academy of ICT, CAS, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recently, Graph Convolutional Networks (GCNs) have proven to be a powerful
mean for Computer Aided Diagnosis (CADx). This approach requires building a
population graph to aggregate structural information, where the graph
adjacency matrix represents the relationship between nodes. Until now, this
adjacency matrix is usually defined manually based on phenotypic information.
In this paper, we propose an encoder that automatically selects the
appropriate phenotypic measures according to their spatial distribution, and
uses the text similarity awareness mechanism to calculate the edge weights
between nodes. The encoder can automatically construct the population graph
using phenotypic measures which have a positive impact on the final results,
and further realizes the fusion of multimodal information. In addition, a
novel graph convolution network architecture using multi-layer aggregation
mechanism is proposed. The structure can obtain deep structure information
while suppressing over-smooth, and increase the similarity between the same
type of nodes. Experimental results on two databases show that our method can
significantly improve the diagnostic accuracy for Autism spectrum disorder and
breast cancer, indicating its universality in leveraging multimodal data for
disease prediction.

##  1 Introduction

There is a growing body of researchers that have realized the potential for
graph convolutional networks in medical-related fields. Recently, GCNs have
been widely used to solve a variety of medical problems already, including
localization of landmarks in craniomaxillofacial Lang et al. (2020), bone age
assessment Gong et al. (2020), representation learning for medical fMRI images
Gadgil et al. (2020) and so on. In this work, we focus on Computer Aided
Diagnosis (CADx) Kazi et al. (2019b), which uses computer technology to assist
physicians in disease prediction. At present, deep learning methods have been
widely used in disease prediction tasks and achieve great results, but in
order to further improve diagnostic accuracy, it is necessary to make full use
of complex medical multimodal data to extract the effective information hidden
in it. These data usually include medical imaging and corresponding non-
imaging phenotypic measures (e.g. subject’s age, height, or acquisition site),
which are usually non-Euclidean and difficult to be processed by traditional
deep learning methods. Moreover, not every phenotypic measure is helpful for
disease prediction and it is still a tedious and time-consuming task for
people to select phenotypic measures that can have a positive effect on
classification results.

Inspired by the success of GCNs in social network analysis and recommendation
systems Bronstein et al. (2017), graph-based methods are usually used in
multimodal data processing. At present, a large number of researchers have
made great contributions to apply GCNs in CADx. At first, the population graph
constructed by the features extracted from the multimodal data is used as the
input of GCNs Sarah et al. (2018). However, the phenotypic measures selected
in this method contribute the same to the edge weights, but should actually be
different. Recently, there are two methods to solve this problem, which are
multi-graph fusion methods Vivar et al. (2018); Kazi et al. (2019a, b) and
single comprehensive graph methods Huang and Chung (2020). Although these
methods deal with the phenotypic measures differently, they both assign
appropriate weights to it. For example, Huang et al. propose that an encoder
can be built to calculate the connection between nodes directly using
multimodal data, and the only graph constructed in the end can be used as the
input of GCNs.

Although the above two methods can achieve high accuracy for disease
prediction, their common limitation is that there is still no effective method
to screen out phenotypic measures that have a negative impact on
classification results. Moreover, the parameter quantity of multi-graph fusion
methods will become very large with the increase in phenotypic measures, which
seriously affects the real-time performance of the model. In addition,
although existing methods, such as the JK-GCN Xu et al. (2018) or EV-GCN Huang
and Chung (2020), have tried to apply layer aggregation mechanism to GCNs to
learn more structural information, due to the actual medical situation being
too complex, it is still challenging to determine an appropriate aggregation
strategy.

In order to address the above challenges, we present a new similarity-aware
adaptive calibrated multi-layer aggregation GCN structure called Adaptive
Multi-layer Aggregation GCN(AMA-GCN). As shown in Figure 1, this structure
contains a specially designed encoder to select effective phenotypic measures
and calculate the edge weights, namely phenotypic measure selection and weight
encoder (PSWE). Besides, we propose two separately designed GCN models to
aggregate the deep structural information and increase the similarity between
different objects of the same type, respectively. This structure can improve
the accuracy of the model and its robustness. The main contributions of our
work can be summarized as follows:

  * •

We design PSWE to automatically select the best combination of phenotypic
measures to interpret the similarity between subjects and calculate their
scores.

  * •

A multi-layer aggregation graph convolutional network with multiple
aggregation modes is introduced to consider more appropriate structural
information for each node. And a dynamic updating mechanism is devised to
increase the similarity between nodes of the same type.

  * •

We test AMA-GCN on real-world dataset. The results show that our method is
superior to other models known at present in terms of validation set accuracy.

Figure 1: Overall framework of the proposed method. PSWE: phenotypic measure
selection and weight encoder. GC: graph convolution. LA: information
aggregation layer. Colors in the graphs: green and orange - labeled diagnostic
values (e.g., healthy or diseased), grey: unlabeled.

##  2 Related Work

In the past, disease classification based on deep learning was usually
achieved using medical imaging. Recently, in order to further improve the
classification accuracy, non-imaging phenotypic data has also been included in
the research scope, that is, medical multimodal data.

###  2.1 Medical Imaging Based Approach

This method analyzes the medical imaging of patients from different
perspectives, so as to obtain as many image features as possible to improve
diagnostic accuracy. For example, MSE-GCN Yu et al. (2020) is proposed to
extract temporal and spatial information respectively from fMRI and DTI to
comprehensively analyze medical imaging. Zhang et al. have achieved great
result performance for nodule-level malignancy prediction by using the
transfer learning method Zhang et al. (2020). This method requires that the
input data must be specific types of medical imaging, while the imaging
methods of different diseases are usually diverse (e.g., ultrasound imaging
for the breast and fMRI for the brain), which makes this method difficult to
apply to the diagnosis of other diseases. Moreover, model learning only relies
on medical imaging, and does not take into account the rich non-image
phenotypic data, which does not conform to the diagnostic habits of
professional doctors in the actual situation.

###  2.2 Multimodal Data Based Approach

In order to further improve the classification accuracy, researchers study
patients’ non-imaging phenotypic data while studying medical imaging. This
complex data is called multimodal data, which is usually non-Euclidean and
difficult to be processed by traditional deep learning methods. At present,
GCNs are usually used to process multimodal data in the medical field. This
method needs to define the population graph where nodes represent patients,
edges represent connections between patients, and edge weights represent
similarity between patients. Specifically, phenotypic measures are usually
used to calculate similarity between patients, and image features extracted
from medical imaging are stored in the corresponding nodes. Furthermore,
according to the different processing methods of phenotypic measures, GCNs
methods can be divided into multi-graph fusion methods and single
comprehensive graph methods. Multi-graph fusion methods usually construct
graphs for each phenotypic measure and process them separately, and design
different multi-graph fusion methods to assign weights to each phenotypic
measure, such as RNN Vivar et al. (2018), self-attention Kazi et al. (2019a),
or LSTM Kazi et al. (2019b). Single comprehensive graph methods directly
extract features from the multimodal data to build a comprehensive graph as
input. For example, Huang et al. Huang and Chung (2020) propose to use an
encoder to process multimodal data. The above methods have achieved great
results, but there are still some limitations: not every phenotypic measure
has a positive effect on classification results. In fact, although our
proposed method is also to build a single comprehensive graph to predict
disease, it does not need to determine the appropriate phenotypic measures
through a large number of experiments as in the past, but is automatically
completed by the proposed encoder.

##  3 Methodology

###  3.1 Preliminaries

In our study, the population graph is defined as an undirected graph
$G=(V,E,A)$, where $V$ is the set of vertices and each vertex represents a
patient; $E$ denotes the set of edges; $A$ denotes the adjacency matrix of
population graph $G$, whose elements are the edge weights. As shown in Figure
1, the adjacency matrix $A$ is obtained by using the proposed PSWE, which can
automatically find the appropriate phenotypic measures and calculate the
corresponding phenotypic measure selection scores (PMS-scores), and then
calculate the edge weights. Note that traditional GCNs usually artificially
select phenotypic measures.

The node feature matrix $X$ is also the input of our model. We define $X$ as
$X=(x_{1},x_{2},...,x_{n})\in\mathbb{R}^{n\times m}$, where $n$ denotes the
number of samples, $m$ represents the dimension of the features. We select
feature extracted from medical imaging of the two datasets involved in our
experiment as the node information. For the ABIDE database, we use functional
connectivity derived from resting-state functional Magnetic Resonance Imaging
(rs-fMRI) for Autism Spectrum Disorder (ASD) classification. Rudie et al.
Rudie et al. (2013) proposed that ASD is linked to disruptions in the
functional and structural organization of the brain, Abraham et al. Abraham et
al. (2016) further demonstrated this assertion. More accurately, we use the
vectorized functional connectivity matrices as feature vectors. For our
collection of breast cancer mass ultrasound image data, we use ResNet-50 to
extract features of medical imaging and directly verify the performance of
these features on ResNet-50 and Ridge classifier.

###  3.2 Phenotypic Measure Selection and Weight Encoder

Edges are the channels through which the node filters information from its
neighbors and represent relationships between nodes. Our hypothesis is that
some appropriate non-imaging phenotypic data can provide critical information
as complement to explain the associations between subjects. Therefore,
selecting the best combination of phenotypic measures to interpret the
similarity between subjects and assigning appropriate weights to them are the
keys to our experiment. In our studies, it will be done by PSWE, and the
implementation process of PSWE as shown in Figure 2.

Considering a set of $H$ non-imaging phenotypic measures $K=\\{K_{h}\\}$,
including quantitative (e.g. subject’s age, height, or BMI) and non-
quantitative (e.g. subject’s calcification or capillary distribution)
phenotypic measures. The population graph’s adjacency matrix $A$ is defined as
follows:

$A(v,w)=\sum_{h=1}^{H}\alpha_{h}*\gamma(K_{h}(v),K_{h}(w))$ (1)

where $\alpha_{h}$ is the PMS-scores of phenotypic measure $K_{h}$; $\gamma$
is a measure of the distance between the value of phenotypic measure $K_{h}$
of two graph nodes. $\alpha_{h}$ is defined as follows:

$\alpha_{h}=\begin{cases}H*\frac{n^{K_{h}}}{\sum_{h=1}^{H}n^{K_{h}}},&{H*n^{K_{h}}\geq\sum\limits_{h=1}^{H}n^{K_{h}},}\\\
0,&{otherwise.}\end{cases}$ (2)

where $n^{K_{h}}$ is a measure of the number of samples in which phenotypic
measure $K_{h}$ meets the requirements, and it is defined differently
depending on the type of phenotypic measures. When $K_{h}$ is a non-
quantitative phenotypic measure (e.g. calcification or edema), we define
$n^{K_{h}}$ as a function with respect to a threshold $\theta$:

$n^{K_{h}}=\begin{cases}\frac{1}{U}\sum\limits_{u=1}^{U}\sum\limits_{p=1}^{P}n_{p\\_u}^{K_{h}},&{\frac{n_{u}^{K_{h}}-n_{p\\_u}^{K_{h}}}{n_{p\\_u}^{K_{h}}}<\theta,}\\\
0,&{otherwise.}\end{cases}$ (3)

where $n_{p\\_u}^{K_{h}}$ is a measure of the number of samples with the value
$u$ and category $p$ in the phenotypic measure $K_{h}$; $n_{u}^{K_{h}}$ is a
measure of the number of samples with the value $u$ in the phenotypic measure
$K_{h}$. Meanwhile, when $K_{h}$ is a quantitative phenotypic measure (e.g.
subject age or BMI), we define $n^{K_{h}}$ as a function with respect to a
threshold $\delta$:

$n^{K_{h}}=\begin{cases}\sum\limits_{p=1}^{P}n_{p\\_s}^{K_{h}},&{\frac{n_{p}^{K_{h}}-n_{p\\_s}^{K_{h}}}{n_{p\\_s}^{K_{h}}}<\delta,}\\\
0,&{otherwise.}\end{cases}$ (4)

where $n_{p}^{K_{h}}$ is a measure of the number of samples with the category
$p$ in the phenotypic measure $K_{h}$; We define a closed interval
$\mathcal{D}$ from $\alpha$ to $\beta$, where $\mathcal{D}\in value(K_{h})$,
and $n_{p\\_s}^{K_{h}}$ is a measure of the number of samples with the
category $p$ and value $s\notin\mathcal{D}$ in the phenotypic measure $K_{h}$.

Figure 2: Overview of the proposed PSWE. $u_{i}$: values of non-quantitative
phenotypic measures corresponding to subject $i$. $s_{i}$: values of
quantitative phenotypic measures corresponding to subject $i$. $p_{i}$: the
category corresponding to subject $i$.

$\gamma$ is also defined differently depending on the type of phenotypic
measure. For non-quantitative phenotypic measures such as family history, we
define $\gamma$ as the Kronecker delta function, meaning that the similarity
highly between two subjects if their values of phenotypic measure $K_{h}$ are
the same. For quantitative phenotypic measures such as subject’s age, we
define $\gamma$ as a function with respect to a closed interval $\mathcal{D}$
from $\alpha$ to $\beta$, where $\mathcal{D}\in value(K_{h})$:

$\displaystyle\gamma(K_{h}(v),K_{h}(w))=$ (5)
$\displaystyle\begin{cases}1,&{K_{h}(v),K_{h}(w)\notin\mathcal{D},}\\\
\frac{1}{e^{\sqrt[3]{\lvert K_{h}(v)-K_{h}(w)\rvert}}},&{\lvert
K_{h}(v)-K_{h}(w)\rvert<\beta-\alpha,}\\\ 0,&{otherwise.}\end{cases}$

The influence of effective phenotypic measures selected by PSWE on the
classification results will be investigated in our experiments, so as to
visually demonstrate the performance of PSWE.

###  3.3 Model Structure Design

In order to make nodes from different dense blocks to obtain sufficient
effective information on each layer while suppressing over-smooth, we
reconstruct the architecture of the network with multi-layer aggregation
mechanism so that the information from different layers can be adaptively
fused into the final expression of the node. As shown in Figure 1, in order to
get the key features of structural information, the first two aggregation
layers that directly integrate the information from graph convolution layers
are aggregated in the way of maxPooling. Besides, the final aggregation layer
uses the way of concating to fully summary the information aggregated by each
layer. The rebuilt model enables each node to automatically integrate the
appropriate information.

The input graph data $h^{(0)}$ is processed by the above model, and the output
$h^{(final)}$ is then calculated by the softmax activation function. The final
output $Z\in\mathbb{R}^{n\times P}$denotes the label prediction for all data
in which each row $Z_{i}$ denotes the label prediction for the $i-th$ node.
The optimal weight matrices trained by minimizing the cross-entropy loss
function as:

$\mathcal{L}_{semi}=-\sum_{i\in L}\sum_{j=1}^{P}Y_{ij}\ln{Z_{ij}}$ (6)

where $L$ indicates the set of labeled nodes. $Y_{ij}$ represents the label
information of the data.

In addition, we introduce an auxiliary dynamic update GCN to increase the
similarity between nodes of the same types while encouraging competition. As
shown in Figure 1, this model’s output is also calculated by the softmax
activation function. The final output $T\in\mathbb{R}^{n\times P}$ denotes the
classification score for all data in which each row $T_{i}$ denotes the
classification score for the $i-th$ node. Note that $T$ is the output of the
auxiliary model, which is different from the output $Z$ of the semi-supervised
classification model. The similarity loss function as:

$\mathcal{L}_{sim}=\tanh{\frac{{\sum_{i\in
L}\sum_{j=1}^{P}(Y_{ij}-T_{ij})^{2}+\xi}}{2\sigma^{2}}}$ (7)

where $\xi$ is a minimal constant; $\sigma$ determines the width of the
kernel.

Then, the joint representation is used to compute a fusion loss. It poses
extra regularization that can help generalization Liu et al. (2020). The final
loss function is defined as:

$\mathcal{L}=\mathcal{L}_{semi}+\lambda\mathcal{L}_{sim}$ (8)

where $\mathcal{L}_{semi}$ and $\mathcal{L}_{sim}$ are defined in Eq.(6) and
Eq.(7), respectively. Parameter $\lambda\geq 0$ is a tradeoff parameter with a
default value of 1.

##  4 Experiments and Results

###  4.1 Dataset

To verify the effectiveness of our model, we evaluate it on the Autism Brain
Imaging Data Exchange (ABIDE) database Martino et al. (2014). The ABIDE
publicly shares fMRI and the corresponding phenotypic data (e.g., age and
gender) of 1112 subjects, and notes whether these subjects have Autism
Spectrum Disorder (ASD). In order to compare fairly with state-of-the-art
Huang and Chung (2020) on the ABIDE, we select the same 871 subjects
consisting of 403 normal and 468 ASD individuals, and perform the same data
preprocessing steps Huang and Chung (2020); Sarah et al. (2018). Then we
delete the subjects with empty values, and finally select the 759 subjects
consisting of 367 normal and 392 ASD individuals. Besides, approved by the
local Institutional Review Board, the local hospital provides ultrasound
images of breast nodules and the corresponding phenotypic data (e.g., age,
gender, and calcification) of 572 sets, and follows the diagnostic results
given by the radiologist for each subject to note whether these subjects have
Breast Cancer. These data are acquired from 121 different patients consisting
of 55 adenopathy and 66 breast cancer individuals, and each set contains a
total of six images (two ultrasound static imaging and four ultrasound
elastography). We remove cases without complete phenotypic measures and adjust
all images to 256×256, and then organize the collected data into the breast
cancer detection dataset (BCD) and verify the universality of proposed model
on it.

Hyperparameter description | Value
---|---
Layer number of the MLA-GCN | 5
Layer number of the ADU-GCN | 2
Chebyshev polynomial | 3
Number of node features | 2000
Graph convolution kernel | 16
Learning rate of MLA-GCN | 0.005
Learning rate of ADU-GCN | 0.05
Regularization parameter | 0.0005
Dropout probability | 0.3
Number of training epoch | 300
Tradeoff parameter $\lambda$ | 1
Optimizer | Adam
Table 1: Experiment hyperparameter setting. MLA-GCN: multi-layer aggregation
GCN. ADU-GCN: auxiliary dynamic update GCN.

###  4.2 Baseline Methods and Settings

We compare the AMA-GCN with the following baselines:
ResNet-50 He et al. (2016): A single modality classification approach using
only images.
Ridge classifier Abraham et al. (2016): A single modality classification
approach using only features extracted from medical imaging data.
GCN Sarah et al. (2018): A model extracting features contained in medical
multimodal data, which is usually used to deal with non-Euclidean data.
JK-GCN Xu et al. (2018): A model using an aggregation layer before the output
layer to integrate information.
GLCN Jiang et al. (2020): A model using graph learning mechanism to constantly
optimize graph structure to improve the classification effect.
EV-GCN Huang and Chung (2020): A model using a custom encoder to obtain the
association between nodes from non-imaging phenotypic data, and JK-GCN is used
to aggregate information.
EV-GCN+PS: A model using the phenotypic measures selected by proposed PSWE as
the basis for constructing the population graph, and using EV-GCN to extract
structural information.

####  4.2.1 Ablation Study

To investigate how the PSWE, the multi-layer aggregation mechanism and
similarity loss function improve the performance of the proposed model, we
conduct the ablation study on the following variants of AMA-GCN:
AMA-GCN(noP) is a model that uses the same phenotypic measures with baseline
methods to build the population graph as input and uses proposed model for
training.
AMA-GCN(noW) is a model that only uses PSWE to select effective phenotypic
measures, and uses a two-layer GCN for training. Note that the weight ratio
calculated by PSWE is not used.
AMA-GCN(noA) is a model that uses PSWE to build the population graph, but uses
a two-layer GCN for training.
AMA-GCN(noS) is a model without the similarity loss function.

In order to ensure a fair comparison, when we do not use PSWE to select the
appropriate phenotypic measures, we choose gender and acquisition sites as the
basis for constructing the population graph of ABIDE database, and choose age
as the basis for constructing the population graph of BCD dataset. This
setting applies to all baseline models. The hyperparameters of the experiment
are shown in Table 1. We employ 10-fold cross-validation to evaluate the
performance of the model and implement our model using TensorFlow. In order to
evaluate the performance of models, we choose overall accuracy (ACC) and area
under the curve (AUC) as the evaluation indicators.

###  4.3 Results and Analysis

| ABIDE | BCD
---|---|---
ACC | AUC | ACC | AUC
ResNet-50 | 0.626 | 0.679 | 0.897 | 0.955
Ridge classifier | 0.636 | 0.688 | 0.901 | 0.961
GCN | 0.705 | 0.731 | 0.916 | 0.950
JK-GCN | 0.722 | 0.736 | 0.947 | 0.972
GLCN | 0.707 | 0.725 | 0.932 | 0.965
EV-GCN | 0.829 | 0.876 | 0.967 | 0.987
EV-GCN+PS | 0.936 | 0.949 | 0.971 | 0.989
AMA-GCN | 0.984 | 0.983 | 0.994 | 0.998
AMA-GCN(noP) | 0.724 | 0.747 | 0.956 | 0.979
AMA-GCN(noW) | 0.955 | 0.974 | 0.974 | 0.987
AMA-GCN(noA) | 0.972 | 0.986 | 0.976 | 0.983
AMA-GCN(noS) | 0.981 | 0.981 | 0.990 | 0.992
Table 2:  Quantitative comparisons between different methods on ABIDE and BCD.

We compare our AMA-GCN with the six baseline methods for predictive
classification of disease on the ABIDE database and BCD dataset, as shown in
Table 2. It can be observed that single-mode models (i.e. ResNet-50 and Ridge
classifier) only use medical imaging data as the basis for classification, and
their overall performance is poor. Comparatively, graph-based methods (GCN,
JK-GCN, GLCN, EV-GCN and ours) yield larger performance gains, benefiting from
analyzing associations between nodes in the population graphs. The proposed
method, AMA-GCN, obtains an average accuracy of 98.4$\%$ and 99.4$\%$ on ABIDE
and BCD datasets, respectively, outperforming the recent SoTA method EV-GCN,
which employs an adaptive population graph with variational edges and uses JK-
GCN to aggregate structure information. We notice that the performance of
graph-based methods is highly sensitive to the phenotypic measures used to
construct graphs, where the phenotypic measures $K_{h}=\\{$gender, acquisition
sites$\\}$ on ABIDE database used by EV-GCN yields an average accuracy of
82.9$\%$. To prove the effectiveness of the phenotypic measures selected by
our PSWE, we train EV-GCN using the same phenotypic measures as those in our
model. As depicted in Table 2, it results in 10.7$\%$ accuracy and 7.3$\%$ AUC
improvement on ABIDE database respectively, indicating that the appropriate
phenotypic measures are indeed the key to improving the performance of disease
prediction. Meanwhile, the universality in leveraging multimodal data for
disease prediction of the proposed AMA-GCN architecture is relatively
validated according to the comparison results on BCD dataset in Table 2.

(a) ACC of the ABIDE dataset (b) AUC of the ABIDE dataset (c) ACC of the BCD
dataset (d) AUC of the BCD dataset

Figure 3: Effects of effective phenotypic measures selected by PSWE on
results, the evaluation indicators are Accuracy (ACC) and Area Under
Curve(AUC).

####  4.3.1 Ablation Experiments

As an ablation study, we test whether removing the PSWE, the multi-layer
aggregation mechanism or similarity loss function affect model performance,
and the results as shown in Table 2. It can be observed that the phenotypic
measures selected by PSWE can significantly improve the classification
accuracy. Using only these phenotypic measures without considering PMS-scores,
the average accuracy of the constructed graphs on the ABIDE and BCD datasets
is improved by 25.0$\%$ and 5.8$\%$, respectively. The classification accuracy
is further improved by 1.7$\%$ and 0.2$\%$ on ABIDE and BCD datasets
respectively after calculating PMS-scores to the selected phenotypic measures.
Although the latter can also improve the classification accuracy, the
improvement is far less than the former, which indicates that finding the
appropriate phenotypic measures is the key to disease prediction. In addition,
the effectiveness of multi-layer aggregation mechanism in improving
classification accuracy is also validated according to the comparison results
in Table 2.

###  4.4 Phenotypic Measures Analysis

In order to demonstrate our conclusion that finding the appropriate phenotypic
measures is the key to solving the problem of disease classification, and also
to prove the effectiveness of proposed PSWE, we further explore the impact of
each phenotypic measure on the classification results. As shown in Figure 3,
using the single phenotypic measure selected by PSWE to construct the
population graph as input, their performances on ABIDE and BCD datasets are
respectively more than 3$\%$ and 1$\%$ better than that of randomly
constructed graph. Comparatively, as shown in Figure 4, using phenotypic
measures not selected by PSWE as the basis for constructing graphs, the final
performance is basically the same as that of randomly generated graphs, and
even worse than the latter, which indicates that these phenotypic measures
selected by PSWE are reasonable. Meanwhile, PSWE achieves the best performance
by combining multiple effective phenotypic measures, and its accuracy on ABIDE
database is even 33.8$\%$ higher than that of randomly constructed graph. This
indicates that there is hidden complementary information among phenotypic
measures, and how to learn these complementary information is the key to
disease prediction.

(a) ACC of the ABIDE dataset (b) ACC of the BCD dataset

Figure 4: Effects of phenotypic measures not selected by PSWE on results, the
evaluation indicators is Accuracy (ACC).

##  5 Conclusion

In this paper, we have proposed a generalizable graph-convolutional framework
that combines multimodal data to predict disease. We designed the population
graph structure according to the spatial distribution and text similarity of
phenotypic measures, while allowing each effective phenotypic measure to
contribute to the final prediction. We reconstructed the graph convolution
model by using the multi-layer aggregation mechanism to automatically find the
optimal feature information from each layer while suppressing over-smooth, and
introduce another channel to increase the similarity between different objects
in the same type. Experimental results show that the proposed method achieves
superior performance on brain analysis and breast cancer prediction. We
believe that such an extensible method can have a better use of helping people
with medical multimodal data for clinical computer-aided diagnosis.

## Acknowledgments

This work was supported by the National Natural Science Foundation of China
(31900979,U1604262), CCF-Tencent Open Fund, the National Key Research and
Development Program of China under Grant (2017YFB1002104), the National
Natural Science Foundation of China under Grant (U1811461), and Zhengzhou
Collaborative Innovation Major Project (20XTZX11020).

## References

  * Abraham et al. (2016) Alexandre Abraham, Michael Milham, Adriana Di Martino, Cameron Craddock, Dimitris Samaras, Bertrand Thirion, and Gael Varoquaux.  Deriving reproducible biomarkers from multi-site resting-state data: An autism-based example.  NeuroImage, 147:736, 2016.
  * Bronstein et al. (2017) Michael Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst.  Geometric deep learning: Going beyond euclidean data.  IEEE Signal Processing Magazine, 34(4):18–42, 2017.
  * Gadgil et al. (2020) Soham Gadgil, Qingyu Zhao, Adolf Pfefferbaum, Edith Sullivan, Ehsan Adeli, and Kilian Pohl.  Spatio-temporal graph convolution for functional mri analysis.  arXiv, 2020.
  * Gong et al. (2020) Ping Gong, Zihao Yin, Yizhou Wang, and Yizhou Yu.  Towards Robust Bone Age Assessment: Rethinking Label Noise and Ambiguity.  Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, 2020.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  Springer, Cham, 2016.
  * Huang and Chung (2020) Yongxiang Huang and Albert Chung.  Edge-variational graph convolutional networks for uncertainty-aware disease prediction.  In Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, 2020.
  * Jiang et al. (2020) Bo Jiang, Ziyan Zhang, Doudou Lin, Jin Tang, and Bin Luo.  Semi-supervised learning with graph learning-convolutional networks.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * Kazi et al. (2019a) Anees Kazi, Arvind Krishna, Shayan Shekarforoush, Karsten Kortuem, and Nassir Navab.  Self-attention equipped graph convolutions for disease prediction.  In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI), 2019.
  * Kazi et al. (2019b) Anees Kazi, Shayan Shekarforoush, Arvind Krishna, Hendrik Burwinkel, and Nassir Navab.  Graph Convolution Based Attention Model for Personalized Disease Prediction.  Springer, Cham, 2019.
  * Lang et al. (2020) Yankun Lang, Chunfeng Lian, Deqiang Xiao, Hannah Deng, and Dinggang Shen.  Automatic Localization of Landmarks in Craniomaxillofacial CBCT Images Using a Local Attention-Based Graph Convolution Network.  Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, 2020.
  * Liu et al. (2020) Shaoteng Liu, Lijun Gong, Kai Ma, and Yefeng Zheng.  Green: a graph residual re-ranking network for grading diabetic retinopathy.  arXiv, 2020.
  * Martino et al. (2014) Adriana Di Martino, Chao-Gan Yan, Qingyang Li, Erin Denio, Francisco Xavier Castellanos, Kaat Alaerts, Jason Anderson, Michal Assaf, Susan Bookheimer, and Mirella Dapretto.  The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism.  Molecular Psychiatry, 2014.
  * Rudie et al. (2013) Jeffrey David Rudie, Jesse Brown, Devi Beck-Pancer, Leanna Hernandez, Emily Dennis, Paul Thompson, Susan Bookheimer, and Mirella Dapretto.  Altered functional and structural brain network organization in autism.  NeuroImage : Clinical, 2, 2013.
  * Sarah et al. (2018) Parisot Sarah, Ktena Sofia Ira, Ferrante Enzo, Lee Matthew, Guerrero Ricardo, Glocker Ben, and Rueckert Daniel.  Disease prediction using graph convolutional networks: Application to autism spectrum disorder and alzheimer’s disease.  Medical Image Analysis, pages S1361841518303554–, 2018.
  * Vivar et al. (2018) Gerome Vivar, Andreas Zwergal, Nassir Navab, and Seyed-Ahmad Ahmadi.  Multi-modal Disease Classification in Incomplete Datasets Using Geometric Matrix Completion.  Springer, Cham, 2018.
  * Xu et al. (2018) Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken Ichi Kawarabayashi, and Stefanie Jegelka.  Representation learning on graphs with jumping knowledge networks.  arXiv, 2018.
  * Yu et al. (2020) Shuangzhi Yu, Shuqiang Wang, Xiaohua Xiao, Jiuwen Cao, Guanghui Yue, Dongdong Liu, Tianfu Wang, Yanwu Xu, and Baiying Lei.  Multi-scale enhanced graph convolutional network for early mild cognitive impairment detection.  Springer, Cham, 2020.
  * Zhang et al. (2020) Hanxiao Zhang, Yun Gu, Yulei Qin, Feng Yao, and Guang-Zhong Yang.  Learning with sure data for nodule-level lung cancer prediction.  In Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, 2020.


# Transfer-LMR: Heavy-Tail Driving Behavior Recognition in Diverse Traffic
Scenarios

Chirag Parikh1, Ravi Shankar Mishra1, Rohan Chandra2, Ravi Kiran
Sarvadevabhatla1 1Centre for Visual Information Technology, IIIT Hyderabad,
India<EMAIL_ADDRESS>of Texas at Austin, USA
<EMAIL_ADDRESS>
###### Abstract

Recognizing driving behaviors is important for downstream tasks such as
reasoning, planning, and navigation. Existing video recognition approaches
work well for common behaviors (e.g. “drive straight”, “brake”, “turn
left/right”). However, the performance is sub-par for underrepresented/rare
behaviors typically found in tail of the behavior class distribution. To
address this shortcoming, we propose Transfer-LMR, a modular training routine
for improving the recognition performance across all driving behavior classes.
We extensively evaluate our approach on METEOR and HDD datasets that contain
rich yet heavy-tailed distribution of driving behaviors and span diverse
traffic scenarios. The experimental results demonstrate the efficacy of our
approach, especially for recognizing underrepresented/rare driving behaviors.

##  I Introduction

Driving behavior of a vehicle is a combination of driving micro-actions
(accelerate, brake, turn, etc.) accompanied by unique long-horizon maneuvers
executed in specific road and traffic conditions. In real-world traffic
scenarios, certain behaviors are rarely observed as compared to others. Due to
this, the naturally collected driving datasets contain a heavily tailed
distribution of unique driving situations [1, 2, 3]. Tackling the heavy-tail
problem by effectively recognizing all unique driving behaviors in different
traffic situations is crucial for achieving a high level of safety with
autonomous vehicles [1, 4, 5]. This enables robust path planning and
navigation in diverse traffic scenarios [6, 3, 7, 8, 9, 10, 11].

While multiple approaches exist for recognizing driving behaviors from dashcam
videos [12] [13], they have primarily been designed to work well for balanced
dataset distributions. Due to the natural heavy-tail distribution of driving
behavior datasets, they exhibit subpar class-wise average recognition
performance. Specifically, the recognition performance of the tail class
behaviors is overshadowed by similar head class behaviors. While multiple
methods exist for modeling the imbalanced dataset distribution, they are
primarily based on large-scale video-recognition datasets that contain a huge
variety of human-action class categories (100+) [14, 15, 16, 17]. Driving
behavior datasets are comparatively low-resource and contain very few unique
class types ($<12$) [2, 3]. Due to the limited variety of unique classes,
existing long-tailed methods are not directly applicable to driving datasets
with heavy-tailed distribution.

Figure 1: Heay-Tail Distributed ego-vehicle driving behavior recognition in
diverse traffic environments. Our proposed approach, Transfer-LMR,
significantly outperforms existing behavior recognition methods by 9.2% and
2.9% in average class accuracy on real-world driving datasets METEOR[2] and
HDD[3].

To improve the representation of the tail classes in low-resource driving
datasets, we introduce an approach, Transfer-LMR, for heavy-tail driving
behavior recognition in diverse traffic environments. Transfer-LMR is a multi-
stage training routine with a video feature refinement technique for
mitigating the class imbalance problem by utilizing the similarity of features
between the head and the tail behavior classes. An overview of our approach is
shown in Figure 1.

###  I-A Main Contributions

  1. 1.

We contribute Transfer-LMR, the first approach for tackling heavy-tailed
driving behavior recognition using low-resource dashcam-video datasets
captured in diverse traffic environments.

  2. 2.

We establish state-of-the-art performance on the METEOR [2] and HDD [3]
datasets using our approach, boosting average class accuracy by 9.2% and 2.9%
respectively without sacrificing the overall performance (mAP) compared to
existing baselines.

  3. 3.

Through Transfer-LMR, we provide a modular plug-and-play training recipe
showcasing its applicability to diverse video backbones (e.g. C3D, I3D,
MotionFormer) while also being fast operating at $2-3$ milliseconds per video
($64$ frames).

We showcase the efficacy and generalizability of our approach by benchmarking
on two publicly-available driving video datasets [2] [3] with tactical and
atypical driving behaviors. We establish the benchmarks using different video
backbones [18] [19] [20] and show significant improvements over the baselines
in average class accuracy (Avg. C/A) while also improving the overall mean
average precision (mAP). A video overview of our approach can be found in the
Supplementary material.

##  II Related Work

In this section, we provide an overview of the existing literature on driving
behavior recognition and long-tailed video activity recognition.

###  II-A Ego Vehicle Driving Behavior Recognition

Multiple research works [21, 22, 23, 24] have explored the use of sensor data
for recognizing normal [25] [26], aggressive [27], and anomalous [28] [29] ego
vehicle driving behaviors. However, it is often infeasible to differentiate
between certain behaviors using sensor data when the same driving action is
performed in different situations. For example, ego vehicle deviating on valid
driving lanes vs. when it deviates towards an invalid or wrong lane has the
same deviating driving action but is performed in different road conditions.

Robust recognition of such driving behaviors (atypical [2] and tactical [3])
requires strong visual cues. Some of the proposed approaches use front-view
dashcam videos along with sensor data [30] [3]. While recently proposed
approaches solely rely on dashcam videos [12] [13], Li et al. [12] proposed
the use of Graph convolution networks (GCN) for modeling egocentric spatial-
temporal interaction with road agents and infrastructure from input videos.
Noguchi et al. [13] incorporated a semi-supervised contrastive learning
technique and utilized unlabelled video data for improving the overall
performance of driving behavior recognition. However, due to the heavy-tailed
distribution of behavior labels in driving datasets, existing approaches get
biased towards accurately predicting the head classes while overshadowing the
tail classes and degrading the class-wise average performance across the
entire dataset. Our proposed approach primarily focuses on solving this
problem without sacrificing the model’s overall performance on the dataset.

###  II-B Long-Tailed Recognition

Unlike driving datasets, large-scale human-action video and image datasets
have a long-tailed distribution with many head and tail classes. Recently,
there have been increasing efforts for long-tailed video recognition [14, 15,
16, 17] following the image-based techniques [31, 32, 33, 34] for such large-
scale datasets. Zhang et al. [14] propose FrameStack which adaptively samples
different numbers of frames from each video based on running average precision
for each class. Li et al. [16] consider frame-level imbalance with frame-level
feature learning. Moon et al. [17] simultaneously handle data imbalance at the
video level through a combination of frame-wise learnable feature aggregators
and vicinity expansion based on class frequency alleviating majority-biased
training. [15] jointly reconstructs the features of all samples in the batch
weighted by the class count. However, these methods are not designed for low-
resource driving video datasets that have a limited variety of head and tail
classes. Our proposed Transfer-LMR approach is specifically adapted to
alleviate the problems due to the heavy-tailed distribution and the limited
variety of classes in driving datasets.

##  III Background

###  III-A Problem Formulation

Approaches for video-based driving behavior recognition combine a backbone for
extracting features $Z$ and a classifier network $g$ for class prediction.
During training, the model receives the data samples in mini-batches, denoted
as $X$, comprising $B$ videos, where $X=\\{x_{i}:i=1\ldots B\\}$, and their
corresponding labels $Y=\\{y_{i}:i=1\ldots B\\}$. Every video sample $x_{i}$
is augmented using standard methods (eg. random-resized cropping, temporal
jittering, etc.) before feeding it as input to the backbone network. Standard
cross-entropy loss function is used for training the entire model and is given
by:

$L_{CE}(Y,\hat{Y})=-\frac{1}{B}\sum_{i=1}^{B}\sum_{j=1}^{C}y_{ij}\cdot\log(\hat{y}_{ij})$
(1)

where $\hat{Y}$ corresponds to the predicted labels and $C$ is the number of
unique classes in the dataset.

Figure 2: TransferLMR: A Two-Stage Training Method for Heavy-Tail Driving
Behavior Recognition. Instance-balanced (IB) and class-balanced (CB) are
sampling strategies. DA is our customized data augmentation method. Averaging
(Avg.) is used for Temporal Feature Aggregation. Long-tailed Mixed
Reconstruction (LMR) is used for Feature Refinement. Driving Behavior Class
prediction in the Testing Stage does not require LMR.

Sampling Strategies. To form the mini-batch, data points $(x_{i},y_{i})$ are
randomly sampled from the training set with different strategies, namely
instance-balanced (IB), and class-balanced (CB). With the IB method, all data
points get equal opportunity of being picked up during training. Formally its
data sampling probability, $p_{j}^{IB}$ for class $j$, is directly
proportional to the total number of instances $n_{j}$ from that class and is
given by:

$p_{j}^{IB}=\frac{n_{j}}{\sum_{i=1}^{C}n_{i}}$ (2)

This is the most common sampling method and is used by existing behavior
recognition approaches [12, 13].

The CB method samples data points from every behavior class in equal
proportions to form the mini-batch. It’s data sampling probability
$p_{j}^{CB}$ for every class $j$ is $1/C$. The CB method is more suitable for
imbalanced or heavy-tail distributed datasets to enable learning a balanced
representation towards all classes during training.

###  III-B Long-Tailed Mixed Reconstruction (LMR)

During training the data points from tail class are oversampled when using the
class-balanced sampling strategy. This causes the model to overfit the limited
diversity of the tail class samples in the training dataset. To alleviate this
issue, Perrett et al. [15] proposed the LMR approach that creates hybrid
features of training data points using feature reconstruction and pairwise
mixing to increase the tail class diversity.

Sample reconstruction. Features $Z_{i}$ of each video sample $x_{i}$ is
reconstructed using a combination of similar features in the mini-batch. The
similarity between features is calculated using the cosine metric given by
$S_{ij}=s(Z_{i},Z_{j})$, where $j\neq i$. The feature combinations are
weighted by $W_{ij}$ that is calculated using the softmax operation applied
over $S_{ij}$ and is given by:

$W_{ij}=\frac{\exp(S_{ij})}{\sum_{k=1}^{B}\exp(S_{ik})}\text{, where }j,k\neq
i$ (3)

The reconstructed features $R_{i}$ are then obtained by:

$R_{i}=\sum_{j=1}^{B}W_{ij}Z_{j}\text{, where }j\neq i$ (4)

Finally, the original features $Z_{i}$ are fused with the reconstructed
features $R_{i}$ using a class contribution function $\mathbf{c}(y_{i})$, as
defined in [15], to obtain the features $M_{i}$ given by:

$M_{i}=\mathbf{c}(y_{i})R_{i}+(1-\mathbf{c}(y_{i}))Z_{i}$ (5)

These features $M_{i}$ and their labels $y_{i}$ are then updated through
stochastic pairwise mixing as defined in [15] to obtain mixed features and
mixed labels that helps in learning robust decision boundaries. The entire LMR
approach focuses on improving the tail behavior class representation without
losing the quality of the head class features.

##  IV Transfer-LMR

The training practices followed by the LMR approach are suitable for large-
scale and long-tail distributed human-action datasets. They are not directly
applicable to low-resource and heavy-tail distributed driving video datasets.
This is because its effectiveness relies on the quality of feature
representations learned in the pre-training stage. These learned
representations are of substandard quality for driving datasets due to their
inferior scale and limited label variety as compared to human-action video
datasets. We propose Transfer-LMR, an approach for systematically obtaining
high-quality feature representations followed by their successive refinement
using LMR to increase the tail behavior class diversity for low-resource
driving datasets. This is achieved through a two-stage training pipeline that
includes different data sampling strategies, a customized data augmentation
method to preserve the peripheral image context, temporal aggregation of video
features, and the unique application of LMR for its refinement. Our approach
improves the representation of the tail behavior classes and boosts the class-
wise average recognition performance on driving datasets.

In the following subsections, we first describe the customized data
augmentation method incorporated during the training stages. Then we elaborate
on the two-stage training pipeline, LMR-based feature refinement, and the
temporal feature aggregation methods employed in our approach.

###  IV-A Data Augmentation

Standard random resized cropping augmentation, used for human-action video
datasets, arbitrarily chooses a cropping region from anywhere inside the image
boundary. But, front-view dashcam videos often contain important road and
traffic objects around the image periphery that influence the ego vehicle’s
driving behavior. These objects may get cropped out after applying the
standard augmentation method. Thus to preserve such peripheral context, we
modify the standard augmentation method to restrict cropping around the left
and right image boundaries. We implement this by empirically defining the
minimum cropping width and height ratios $r_{w}$ and $r_{h}$ as 0.9 and 0.6
respectively of the original image. Based on this the aspect ratio range for
the crop bounding box is given by $A_{min}$ and $A_{max}$ as:

$A_{min}=\frac{r_{w}\cdot w}{h}$ (6) $A_{max}=\frac{w}{r_{h}\cdot h}$ (7)

where $w$ and $h$ are image width and height respectively. The minimum
cropping area ratio using our modified method is given by
$AR_{min}=r_{w}r_{h}$, i.e., 0.54 while that for the standard augmentation
method is 0.08. A demonstration of the minimum cropping regions chosen using
our modified method and the standard method is shown in Figure 3 with orange
and blue dotted lines respectively. We employ this modified augmentation
method on the input dashcam videos during both the training stages of our
approach described in the following subsection.

Figure 3: Modified random resized cropping augmentation method tailored for
dashcam-based driving videos. The dotted orange line represents the position
and relative dimensions of the minimum cropping region obtained from this
method, while the dotted blue line corresponds to that obtained from the
original unaltered augmentation method.

###  IV-B Two-Stage Training Routine

The two-stage training methodology of Transfer-LMR is demonstrated in Figure
2.

First Stage. The first training stage comprises a video backbone pre-trained
on a large-scale video recognition dataset and a classifier network
initialized from scratch for driving behavior class prediction. During this
stage, the entire model is trained end-to-end with instance-balanced data
sampling (Section III-A) and the sampled driving videos are augmented with our
customized boundary-constrained cropping method (Section IV-A). A standard
cross-entropy loss function is used to optimize for the driving behavior
recognition task. This stage ensures the transfer of high-quality video
features to the driving domain yielding superior overall behavior recognition
performance.

Second Stage. Due to the heavy tail distribution of ego vehicle’s driving
behavior categories, the model obtained from the first training stage
underfits the tail behavior classes thereby giving subpar class-wise average
recognition performance[12] [13]. The second training stage is primarily meant
to alleviate this issue by refining the high-quality video features obtained
from the first training stage. In this stage, we first sample the training
data points using the CB sampling strategy for rebalancing the data
distribution (Section III-A). Secondly, we employ the modified data
augmentation method, which was also used in the first training stage, to
preserve the peripheral context from the input video’s RGB frames. Next, we
feed the augmented RGB frames to the I3D backbone [19] and extract the video
features for its successive refinement using the LMR approach (Section III-B).

The LMR approach is suited for refining video features that are in condensed
form having no temporal dimension. However, the video features, from the I3D
backbone, have an additional temporal dimension and thus require aggregation.
The temporal feature aggregation is accomplished by simple averaging across
the temporal dimension and is given by the following equation:

$Z_{i}=\frac{\sum_{t=1}^{T}Z_{it}}{T}$ (8)

where $Z_{it}$ is the feature at temporal index $t$, and $T$ is the total
number of temporal features. Alternative to this approach, we also adopted
learnable feature aggregation methods including Bi-directional GRU and
Transformer but they were found to yield subpar performance in comparison to
simple averaging. This indicated that learnable aggregation methods are not
well-suited for low-resource driving video datasets.

The LMR module consumes the aggregated features and their corresponding
ground-truth behavior labels $Y$ and produces refined features and their
corresponding ground-truth mixed behavior labels $Y^{*}$ using stochastic
pairwise mixing as defined in [15]. The refined features are then fed to the
classifier network to yield the predicted labels $\hat{Y}$ as the model’s
output. The mixed behavior labels $Y^{*}$ and the predicted labels $\hat{Y}$
are optimized using the soft cross-entropy loss function
$L_{SCE}(\cdot,\cdot)$ given by:

$L_{SCE}(Y^{*},\hat{Y})=-\frac{1}{B}\sum_{i=1}^{B}\sum_{j=1}^{C}{y}_{ij}^{*}\cdot\log(\hat{y}_{ij})$
(9)

It is to be noted that we reuse the classifier network weights that were
obtained from the first training stage as opposed to the original LMR approach
and also aggregate the temporal features instead of its independent
application across the time dimension.

The entire network is fine-tuned with a reduced learning rate and a tuned set
of hyperparameters, as defined in [15], for the LMR feature refinement module.
During the testing or inference stage, the LMR-based feature refinement
technique is disabled and the temporally aggregated and refined video features
extracted from an input video are directly fed to the classifier network for
driving behavior prediction. This two-stage training procedure thus increases
the diversity of the under-represented behavior classes and helps in obtaining
better class-wise average recognition performance as compared to the baselines
while also maintaining a better overall performance.

Table I: Driving Behavior Recognition on METEOR dataset [2]. Column TFA
corresponds to Temporal Feature Aggregation methods. Abbreviations for driving
behavior category: Overtaking (OT), Deviating on Unmarked Lanes (DT), Rule
Breaking by Wrong Lane driving (WL), Cutting (CT), Yielding (YD). The
frequency of each behavior class in the training set is specified in round
brackets alongside its abbreviation. Color shades from yellow to orange
indicate the heavy-tailed distribution of driving behavior categories in the
dataset.

Video Backbone | Training Method | TFA |  OT (5107) |  DT (1526) |  WL (297) |  CT (206) |  YD (62) |  Overall mAP |  Avg. C/A |  Overall Acc.
---|---|---|---|---|---|---|---|---|---|---
| CE | - |  98.2 |  91.1 |  88.0 |  56.3 |  78.0 |  82.3 |  71.8 |  90.5
|  cRT [31] | - |  98.2 |  91.1 |  87.5 |  59.8 |  78.7 |  83.1 |  78.0 |  89.0
I3D* [19] | Transfer-LMR (Ours) | Transformer |  98.6 |  91.3 |  86.0 |  53.9 |  77.1 |  81.4 |  79.4 |  90.2
| Transfer-LMR (Ours) | BiGRU |  98.6 |  92.2 |  86.4 |  59.2 |  72.8 |  81.9 |  80.6 |  90.4
| Transfer-LMR (Ours) | Average |  98.7 |  92.2 |  88.5 |  62.3 |  75.9 |  83.5 |  81.0 |  91.0
| CE | - |  97.9 |  86.5 |  80.1 |  32.1 |  56.9 |  70.7 |  58.8 |  88.2
C3D‡ [18] |  cRT [31] | - |  97.8 |  86.5 |  80.2 |  36.3 |  48.1 |  69.8 |  64.1 |  87.3
| Transfer-LMR (Ours) | - |  97.4 |  86.6 |  82.2 |  38.7 |  59.3 |  72.8 |  65.1 |  89.0
| CE | - |  95.8 |  83.6 |  82.0 |  11.2 |  84.3 |  61.4 |  54.8 |  85.7
MotionFormer† [20] |  cRT [31] | - |  97.3 |  89.5 |  82.6 |  23.2 |  56.8 |  69.8 |  59.5 |  89.3
| Transfer-LMR (Ours) | - |  97.4 |  89.5 |  80.8 |  30.3 |  61.3 |  71.9 |  63.4 |  89.4
I3D** | CE | - |  98.5 |  90.4 |  88.4 |  48.2 |  73.9 |  79.9 |  71.6 |  90.4
| Transfer-LMR (Ours) | Average |  98.7 |  92.2 |  89.7 |  60.3 |  74.9 |  83.2 |  81.0 |  91.0

  * †

Pre-Trained on EpickKitchens-100 [35]

  * ‡

Pre-Trained on UCF101 [36]

  * *

Pre-Trained on Kinteics-400 [19]

  * **

Pre-Trained on HDD [3]

##  V Experiments

###  V-A Datasets

We perform extensive experiments on two largest publicly available egocentric
driving video datasets: METEOR [2] and HDD [3]. These datasets include
diverse, atypical, and unique driving behaviors captured in different road and
traffic conditions.

METEOR: The METEOR dataset [2] contains 1250 one-minute front-view driving
videos having 1920$\times$1080 pixels resolution captured at 30fps in dense,
heterogeneous, and unstructured traffic situations of Hyderabad City, India.
It includes five unique atypical ego-driving behaviors with a heavy tail
distribution. We follow [2] and split the METEOR dataset into 1000 videos for
training and 231 videos for testing. Given the frame-level annotations in the
dataset, we extracted 7198 trimmed intervals of driving behavior video clips
from the training set and 1808 such videos from the test set. The number of
such videos per class ranges from 62 to 5107 in the training set and the
average duration of these videos is 1.8 seconds.

HDD: This dataset includes 104 hours of 137 driving video sessions with frame-
level annotations of 11 different goal-oriented tactical driving behaviors
which are observed in sparse and structured traffic environments. The videos
have a resolution of 1280$\times$720 captured at 30fps. We followed prior work
[3][12] and used 100 sessions for training and 37 sessions for testing and
extracted 9730 trimmed driving behavior videos from the training set and 2647
such videos from the test set. The number of such videos per class ranges from
68 to 4859 in the training set and the average duration of these videos is 3.7
seconds.

###  V-B Evaluation Metrics

We follow most previous works [3, 12] and use overall mean average precision
(Overall mAP) as an evaluation metric for estimating the model’s performance
on the entire dataset. We also report average precision (AP) values for each
behavior category to monitor class-wise performance variations. However, to
validate the effectiveness of our approach in improving the class-wise average
performance, we report the average class accuracy (Avg. C/A) as our main
evaluation metric. Given the imbalanced test sets of both datasets, the
overall (weighted) class accuracy (Overall Acc.) is also reported for
reference. All values are reported in percentages.

###  V-C Baselines

CE. A standard training method with instance-balanced sampling and cross-
entropy loss function. This baseline is equivalent to the first training stage
of our proposed approach which is described in Section IV-B.

cRT. Classifier Re-Training [31]. Originally proposed for long-tailed image
recognition, it involves training with instance-balanced sampling followed by
a classifier reset and class-balanced sampling. This method was modified for
its application on driving video datasets by obtaining the CE baseline first
followed by classifier re-training using class-balanced sampling without
resetting its weights. This method alleviates the class imbalance problem and
serves as a strong baseline for comparison with our approach.

Table II: Driving Behavior Recognition on HDD dataset [3]. Abbreviations for
driving behavior category: Intersection Passing (IP), Left turn (LT), Right
turn (RT), Crosswalk passing (CW), Left lane change (LLC), Right lane change
(RLC), Left lane branch (LLB), Merge (MG), Right lane branch (RLB), Railroad
passing (RP), U-turn (UT). The frequency of each behavior class in the
training set is specified in round brackets alongside its abbreviation. The
column color shades from yellow to orange indicate the heavy-tailed
distribution of driving behavior categories in the dataset.

Video Backbone | Training Method |  IP (4859) |  LT (1368) |  RT (1363) |  CW (569) |  LLC (496) |  RLC (463) |  LLB (245) |  MG (131) |  RLB (97) |  RP (71) |  UT (68) |  Overall mAP |  Avg. C/A |  Overall Acc.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
|  CE† [12] |  85.6 |  79.1 |  78.9 |  29.8 |  74 |  62.4 |  59 |  20.1 |  14.3 |  0.1 |  41.4 |  49.5 |  - |  -
I3D* [19] | CE |  99.3 |  99.0 |  99.6 |  87.4 |  92.3 |  93.8 |  91.0 |  74.0 |  65.1 |  19.5 |  71.7 |  81.1 |  77.8 |  95.0
| Transfer-LMR Avg. (Ours) |  99.3 |  98.2 |  99.6 |  89.1 |  93.9 |  92.9 |  90.8 |  76.6 |  65.4 |  19.5 |  86.4 |  82.8 |  80.7 |  94.4
|  CE† [12] |  82.4 |  77.4 |  80.7 |  17.4 |  67.9 |  56.9 |  59.7 |  20.1 |  5.2 |  3.9 |  29.5 |  45.5 |  - |  -
C3D‡ [18] | CE |  99.4 |  98.2 |  99.4 |  84.5 |  97.9 |  89.2 |  93.5 |  81.4 |  43.7 |  28.8 |  75.1 |  81.0 |  74.5 |  92.9
| Transfer-LMR (Ours) |  99.0 |  98.0 |  99.5 |  88.3 |  97.6 |  88.7 |  93.7 |  80.4 |  47.5 |  29.4 |  68.8 |  81.0 |  76.7 |  93.1
I3D** | CE |  99.5 |  99.1 |  99.5 |  87.3 |  94.0 |  93.7 |  86.1 |  68.9 |  93.7 |  28.6 |  63.1 |  80.4 |  76.2 |  94.7
| Transfer-LMR Avg. (Ours) |  99.5 |  99.0 |  99.4 |  88.3 |  94.8 |  94.8 |  86.2 |  65.3 |  94.8 |  16.2 |  72.9 |  80.5 |  79.7 |  94.0

  * ‡

Pre-Trained on UCF101 [36]

  * *

Pre-Trained on Kinteics-400 [19]

  * **

Pre-Trained on METEOR [2]

  * †

Trained with Standard Random-Cropping Augmentation

###  V-D Implementation Details

Backbones. Our proposed training method is compared with the baselines for
different video backbones such as C3D [18], I3D [19], and MotionFormer [20]
which have shown substantial success in general human action recognition
tasks. We establish new baselines and benchmark our method using all the
aforementioned video backbones on METEOR dataset. Following prior works [12],
we benchmark C3D and I3D-based models for comparison on HDD dataset. The C3D
model was pre-trained on the Sports-1M and UCF101 datasets [37][36], while the
I3D model with ResNet-50 backbone [38] was pre-trained on the Kinectics-400
human action videos dataset [19]. The MotionFormer model with ViT backbone
[39] was pre-trained on the EpicKitchens-100 dataset [35]. These pre-trained
models were then fine-tuned on respective driving datasets. Through these
experiments we showcase the generalizability of our approach across video
backbones and datasets for heavy-tailed driving behavior recognition.

Model Input. The C3D backbone intakes 16-frame clip with 112$\times$112 pixels
resolution, MotionFormer takes 16-frame clip with 224$\times$224 resolution,
and I3D takes 64-frame clip with 224$\times$224 resolution as default model
inputs. The clip frames are randomly sampled from the trimmed behavior videos
at intervals of 3fps for HDD dataset based on the prior work [12] and 30fps
for METEOR dataset. A higher frame sampling rate was required to obtain a more
granular context in the case of METEOR. This helped in capturing the short-
lived duration of the unexpected and atypical driving behaviors. The input to
LMR-based feature refinement module is the feature vector from the last layer
of the video backbone with dimension 4096$\times$1 in C3D and 768$\times$1 in
MotionFormer, where the temporal dimension is 1. The I3D model gives output
features of dimension 2048$\times$7 from the 3D average polled Mixed_5c layer.
These features are temporally aggregated to yield 2048$\times$1 dimensional
vector before feeding it as input to the LMR module.

Training Routine. We perform all experiments on a system with 4$\times$3080Ti
GPUs using PyTorch. In the first training stage, the MotionFormer-based model
is trained for 30 epochs using Adam optimizer with a learning rate of 0.0001.
The C3D and I3D-based models are trained for 40 epochs using SGD optimizer
with learning rates of 0.000125 and 0.1 respectively. In the second stage for
cRT and Transfer-LMR methods, the MotionFormer-based model is fine-tuned with
a reduced learning rate of 0.00001 while the C3D and I3D-based models are
finetuned with learning rates of 0.0000125 and 0.001 respectively. The
hyperparameters for the LMR module were set to 0.4 and 1.0 for reconstruction
weight and decay values respectively. For a fair comparison, we keep these
hyperparameters the same for all the experiments conducted on both METEOR and
HDD datasets. They can otherwise be fine-tuned for each dataset separately to
obtain better results.

##  VI Results

Table I presents the results of our newly established baselines and
experiments using Transfer-LMR for heavy-tailed driving behavior recognition
on the validation set of METEOR dataset. Our approach consistently outperforms
baseline training methods (CE and cRT) on all dataset-wise metrics: average
class accuracy, overall mAP, and overall accuracy, across all video backbones
(MotionFormer, C3D, and I3D). As compared to the CE baseline, our method shows
significant improvement in average class accuracy by 8.6%, 6.3%, and 9.2%
using MotionFormer, C3D, and I3D backbones respectively. These gains indicate
that our approach mitigates the class imbalance problem by improving the per-
class accuracy without sacrificing the model’s overall accuracy and mAP.
Specifically, our approach benefits the highly underperforming Cutting (CT)
tailed class by significantly boosting its performance across baselines.

Figure 4: Qualitative Results for Driving Behavior Recognition on METEOR
dataset [2]. The key frame of sample driving videos from the validation set of
METEOR dataset is shown. Their Bird’s Eye View (BEV) is also displayed
demonstrating the vehicle trajectories that define the ego’s driving behavior
when observed in the video’s full duration. The behavior predictions of CE and
cRT baselines are compared with Transfer-LMR (TLMR) method when using the
(Kinetics-400 pre-trained) I3D video backbone. Figure 5: Qualitative Results
for Driving Behavior Recognition on HDD dataset [3]. The key frame of sample
driving videos from the validation set of HDD dataset is shown. Their Bird’s
Eye View (BEV) is also displayed demonstrating the vehicle trajectories that
define the ego’s driving behavior when observed in the video’s full duration.
The behavior predictions of the CE baseline are compared with the Transfer-LMR
(TLMR) method when using the (Kinetics-400 pre-trained) I3D video backbone.

Figure 4 shows the selected examples of driving behaviors from the METEOR
dataset. Qualitatively, the CE method fails to correctly identify the tail
classes (CT, YD, WL, and DT). And, it gets confused among similar driving
behaviors, such as, it predicts CT as OT and WL as DT class. The similarity
between these behaviors is attributed to the similar trajectories followed by
the ego-vehicle which can be observed in their BEVs. CT and OT differ only
regarding the ego-relative orientation of the highlighted road agents in
relation to which the behaviors are defined. While WL and DT behaviors differ
only regarding the illegal and legal lane changes undertaken by them
respectively. Also, CT being a tail class has limited diversity as compared to
the head class OT, thereby it gets falsely predicted as OT with the CE method.
Similarly, WL gets falsely predicted as DT due to insufficient diversity being
a tail class. Transfer-LMR specifically addresses this issue by increasing the
diversity among such correlated examples. This results in reduced confusion
among such classes (CT, WL, DT, and OT) and substantial improvements in their
average precision values over the CE baseline, as evident from Table I for the
I3D-based backbone. Figure 4 also shows that our approach (TLMR) consistently
gives correct predictions for the ambiguous tail classes. While a failure case
shows that it incorrectly predicts the head class OT.

In Table II we benchmark and compare our proposed method with the existing
method [12] for driving behavior recognition on the HDD dataset. Since the
modified random cropping data augmentation method, described in Section IV-A,
is employed throughout the Transfer-LMR training stages, we reproduce the CE
baselines of C3D and I3D backbones for a fair comparison. We also include the
existing benchmarks, reported in [12], to showcase the gap in performance as
compared to our established baselines. Transfer-LMR outperforms in average
class accuracy by a good margin on both the video backbones (C3D and I3D),
while being comparable or better concerning overall mAP and overall accuracy.
Figure 4 shows selected examples from the HDD dataset, where the CE method
confuses tail classes with similar head classes (eg. LT and UT, LLC and LLB,
etc. having similar ego-vehicle trajectories). Consistently, Transfer-LMR
predicts the tail classes correctly while the head LT class is incorrectly
predicted. Please view the video in Supplementary for additional details.

Based on the quantitative results reported in Tables I and II, we observe that
I3D is a better alternative than C3D and MotionFormer for attaining superior
driving behavior recognition performance. Specifically using the Kinetics-400
pre-trained I3D, Transfer-LMR gives the best results for both driving datasets
METEOR and HDD. This thereby demonstrates that our training method (Transfer-
LMR) helps in disambiguation features between similar driving behaviors and
addresses the heavy tail problem across different driving datasets by boosting
the average class accuracy and overall mAP on the entire dataset.

As an ablation, we perform experiments of pre-training the I3D backbone with a
driving dataset, instead of any large-scale human-action videos dataset,
before applying the Transfer-LMR method. This was done to observe any effects
on performance if pre-training and fine-tuning dataset domains are kept
similar. The results in the last two rows of Tables I and II indicate that
Transfer-LMR outperforms the CE baseline significantly in average class
accuracy with METEOR or HDD driving datasets being used for pre-training or
fine-tuning. However, these performances were comparable but slightly inferior
to the best results obtained with Kinetics-400 as the pre-training dataset.
Thus it was concluded that pre-training on large-scale video datasets always
helps in obtaining better performance as compared to using a similar domain
and resource-constrained driving dataset for pre-training.

##  VII CONCLUSIONS

We introduce Transfer-LMR, a novel training method for heavy-tail distributed
driving behaviors recognition from front-view dashcam videos. Our method
alleviates the class imbalance problem in resource-constrained driving
datasets by improving the class-wise average performance while maintaining
better overall performance. To accomplish this we incorporate different data
sampling strategies, boundary-constrained random cropping, temporal feature
aggregation, and LMR-based feature refinement adapted toward modeling driving
behaviors. Specifically, LMR increases the diversity of the tail classes by
amalgamating similar features followed by random mixing among the training
data samples. We demonstrate the effectiveness of our approach and establish
state-of-the-art results for heavy-tail driving behavior recognition on METEOR
and HDD datasets captured in structured and unstructured traffic environments.

## References

  * [1] P. Koopman, “The heavy tail safety ceiling,” in _Automated and Connected Vehicle Systems Testing Symposium_ , vol. 1145. SAE, 2018, pp. 8950–8961.
  * [2] R. Chandra, X. Wang, M. Mahajan, R. Kala, R. Palugulla, C. Naidu, A. Jain, and D. Manocha, “Meteor: A dense, heterogeneous, and unstructured traffic dataset with rare behaviors,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 9169–9175.
  * [3] V. Ramanishka, Y.-T. Chen, T. Misu, and K. Saenko, “Toward driving scene understanding: A dataset for learning driver behavior and causal reasoning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7699–7707.
  * [4] R. Chandra, U. Bhattacharya, A. Bera, and D. Manocha, “Densepeds: Pedestrian tracking in dense crowds using front-rvo and sparse features,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 468–475.
  * [5] R. Chandra, U. Bhattacharya, T. Randhavane, A. Bera, and D. Manocha, “Roadtrack: Realtime tracking of road agents in dense and heterogeneous environments,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 1270–1277.
  * [6] R. Chandra, “Towards autonomous driving in dense, heterogeneous, and unstructured traffic,” Ph.D. dissertation, University of Maryland, College Park, 2022.
  * [7] M. Xu, M. Gao, Y.-T. Chen, L. S. Davis, and D. J. Crandall, “Temporal recurrent networks for online action detection,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 5532–5541.
  * [8] M. Shimosaka, T. Kaneko, and K. Nishi, “Modeling risk anticipation and defensive driving on residential roads with inverse reinforcement learning,” in _17th International IEEE Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2014, pp. 1694–1700.
  * [9] R. Chandra, U. Bhattacharya, A. Bera, and D. Manocha, “Traphic: Trajectory prediction in dense and heterogeneous traffic using weighted interactions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 8483–8492.
  * [10] R. Chandra, U. Bhattacharya, C. Roncal, A. Bera, and D. Manocha, “Robusttp: End-to-end trajectory prediction for heterogeneous road-agents in dense traffic with noisy sensor inputs,” in _Proceedings of the 3rd ACM Computer Science in Cars Symposium_ , 2019, pp. 1–9.
  * [11] R. Chandra, T. Guan, S. Panuganti, T. Mittal, U. Bhattacharya, A. Bera, and D. Manocha, “Forecasting trajectory and behavior of road-agents using spectral clustering in graph-lstms,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 3, pp. 4882–4890, 2020.
  * [12] C. Li, Y. Meng, S. H. Chan, and Y.-T. Chen, “Learning 3d-aware egocentric spatial-temporal interaction via graph convolutional networks,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 8418–8424.
  * [13] C. Noguchi and T. Tanizawa, “Ego-vehicle action recognition based on semi-supervised contrastive learning,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023, pp. 5988–5998.
  * [14] X. Zhang, Z. Wu, Z. Weng, H. Fu, J. Chen, Y.-G. Jiang, and L. S. Davis, “Videolt: Large-scale long-tailed video recognition,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 7960–7969.
  * [15] T. Perrett, S. Sinha, T. Burghardt, M. Mirmehdi, and D. Damen, “Use your head: Improving long-tail video recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 2415–2425.
  * [16] X. Li and H. Xu, “Meid: mixture-of-experts with internal distillation for long-tailed video recognition,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 2, 2023, pp. 1451–1459.
  * [17] W. Moon, H. S. Seong, and J.-P. Heo, “Minority-oriented vicinity expansion with attentive aggregation for video long-tailed recognition,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 2, 2023, pp. 1931–1939.
  * [18] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 4489–4497.
  * [19] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6299–6308.
  * [20] M. Patrick, D. Campbell, Y. Asano, I. Misra, F. Metze, C. Feichtenhofer, A. Vedaldi, and J. F. Henriques, “Keeping your eye on the ball: Trajectory attention in video transformers,” _Advances in neural information processing systems_ , vol. 34, pp. 12 493–12 506, 2021.
  * [21] R. Chandra, A. Bera, and D. Manocha, “Using graph-theoretic machine learning to predict human driver behavior,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 3, pp. 2572–2585, 2021.
  * [22] R. Chandra, U. Bhattacharya, T. Mittal, A. Bera, and D. Manocha, “Cmetric: A driving behavior measure using centrality functions,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 2035–2042.
  * [23] R. Chandra, U. Bhattacharya, T. Mittal, X. Li, A. Bera, and D. Manocha, “Graphrqi: Classifying driver behaviors using graph spectrums,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 4350–4357.
  * [24] A. Mavrogiannis, R. Chandra, and D. Manocha, “B-gap: Behavior-rich simulation and navigation for autonomous driving,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 4718–4725, 2022.
  * [25] N. Oliver and A. P. Pentland, “Graphical models for driver behavior recognition in a smartcar,” in _Proceedings of the IEEE Intelligent Vehicles Symposium 2000 (Cat. No. 00TH8511)_. IEEE, 2000, pp. 7–12.
  * [26] D. Mitrovic, “Reliable method for driving events recognition,” _IEEE transactions on intelligent transportation systems_ , vol. 6, no. 2, pp. 198–205, 2005.
  * [27] Y. Ma, Z. Zhang, S. Chen, Y. Yu, and K. Tang, “A comparative study of aggressive driving behavior recognition algorithms based on vehicle motion data,” _IEEE Access_ , vol. 7, pp. 8028–8038, 2018.
  * [28] M. Matousek, M. Yassin, R. van der Heijden, F. Kargl, _et al._ , “Robust detection of anomalous driving behavior,” in _2018 IEEE 87th Vehicular Technology Conference (VTC Spring)_. IEEE, 2018, pp. 1–5.
  * [29] M. Matousek, E.-Z. Mohamed, F. Kargl, C. Bösch, _et al._ , “Detecting anomalous driving behavior using neural networks,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2019, pp. 2229–2235.
  * [30] H. Xu, Y. Gao, F. Yu, and T. Darrell, “End-to-end learning of driving models from large-scale video datasets,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2174–2182.
  * [31] B. Kang, S. Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis, “Decoupling representation and classifier for long-tailed recognition,” _arXiv preprint arXiv:1910.09217_ , 2019.
  * [32] S. Zhang, Z. Li, S. Yan, X. He, and J. Sun, “Distribution alignment: A unified framework for long-tail visual recognition,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 2361–2370.
  * [33] S. Alshammari, Y.-X. Wang, D. Ramanan, and S. Kong, “Long-tailed recognition via weight balancing,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 6897–6907.
  * [34] K. Tang, M. Tao, J. Qi, Z. Liu, and H. Zhang, “Invariant feature learning for generalized long-tailed classification,” in _European Conference on Computer Vision_. Springer, 2022, pp. 709–726.
  * [35] D. Damen, H. Doughty, G. M. Farinella, A. Furnari, E. Kazakos, J. Ma, D. Moltisanti, J. Munro, T. Perrett, W. Price, _et al._ , “Rescaling egocentric vision,” _arXiv preprint arXiv:2006.13256_ , 2020.
  * [36] K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,” _arXiv preprint arXiv:1212.0402_ , 2012.
  * [37] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2014, pp. 1725–1732.
  * [38] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [39] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.