# Towards Robust Multimodal Prompting with Missing Modalities

###### Abstract

Recently, multimodal prompting, which introduces learnable missing-aware
prompts for all missing modality cases, has exhibited impressive performance.
However, it encounters two critical issues: 1) The number of prompts grows
exponentially as the number of modalities increases; and 2) It lacks
robustness in scenarios with different missing modality settings between
training and inference. In this paper, we propose a simple yet effective
prompt design to address these challenges. Instead of using missing-aware
prompts, we utilize prompts as modality-specific tokens, enabling them to
capture the unique characteristics of each modality. Furthermore, our prompt
design leverages orthogonality between prompts as a key element to learn
distinct information across different modalities and promote diversity in the
learned representations. Extensive experiments demonstrate that our prompt
design enhances both performance and robustness while reducing the number of
prompts.

Index Terms—  Multimodal, Missing modality, Prompt learning

##  1 Introduction

Advances in sensing technology allow us to acquire a variety of signals
including images, text, heat map, depth map, and more. Various methods have
emerged for multimodal downstream tasks, such as classification [1], action
recognition [2, 3, 4, 5], and emotion recognition [6, 7, 8]. Recently, several
methods [1, 9] have been proposed based on transformer [10, 11] to deal with
the missing modality scenario during the inference phase. Nevertheless, these
works assume that the training dataset is full-modality. In real-world
scenarios, users may acquire modality-incomplete data due to device
constraints regardless of the training and inference phase. Moreover,
expanding the number of modalities results in high computational and memory
costs, making it more challenging to train the entire model.

Inspired by prompt learning [12, 13, 14, 15], Lee et al. [16] propose
multimodal prompting, which adopts learnable tokens, missing-aware prompts
(MAPs), to transfer knowledge from pretrained models on large-scale datasets
to downstream tasks. This method focuses solely on training MAPs and
parameters associated with downstream tasks while keeping the backbone network
frozen. By assigning prompts to missing modality cases, prompts are updated
while learning how to deal with each missing modality case. As a result, it
can handle general scenarios where modality-incomplete data exist during the
training phase, requiring less than 1$\%$ of the total parameters to finetune
the pretrained model.

Fig. 1:  Illustrating our motivation, consider a scenario with a total of $M$
modalities. In the previous work [16], addressing all missing modality cases
requires $2^{M}-1$ missing-aware prompts. In contrast, our approach involves
only allocating $M$ prompts, each designed to capture the unique
characteristics of each individual modality.

While MAPs are a promising solution for addressing the missing modality
scenario during both training and inference, there are two notable challenges.
First, the number of prompts exhibits exponential growth as the number of
modalities increases. For a $M$ modality task, the requisite number of MAPs is
$2^{M}-1$ to treat all missing modality scenarios, as depicted in Figure 1.
This predicament is critical as the advancement of artificial intelligence
ensures the inexorable integration of an increasing number of modalities [17].
Second, it lacks robustness in unseen missing modality settings at the
training phase. Unless an ample number of samples are available for all
missing modality cases, the model remains incapable of addressing unseen
missing modality cases that were not encountered during the training phase.
For example, if we can leverage only modality-complete cases and text-only
cases in a training phase, the model can not deal with image-only cases in an
inference phase.

In this paper, we propose a simple yet effective prompt design to treat these
issues. Note that we are focusing on the image-text multimodal classification
task. In contrast to the previous study [16], which creates MAPs for each
missing modality case (image-only, text-only, and modality-complete), we are
developing modality-specific prompts (MSPs) for each individual modality. MSPs
acquire knowledge specific to modality from the backbone pretrained on large-
scale datasets. We utilize image-specific prompt and text-specific prompt as
input of multimodal transformer for image-only and text-only cases,
respectively. We adopt a combination of image-specific prompt and text-
specific prompt for modality-complete cases so that both prompts can be
updated simultaneously. In this way, we train the prompts assigned to the
modalities present in samples, disregarding the missing modality case.
Furthermore, we incorporate an additional loss term to enforce orthogonality
among MSPs, ensuring the diversity of information learned across different
modalities. Extensive experiments show that the proposed straightforward
prompt design boosts the performance and significantly enhances the robustness
for unseen missing modality settings in the training phase. We expect that our
prompt design will serve as a valuable guiding principle for a wider range of
modality tasks, paving the way for further advancements in this field.

Fig. 2: Illustration of our modality-specific prompts design and overall
framework.

##  2 Method

###  2.1 Problem Definition

We utilize a multimodal dataset labeled as $D$, comprising two types of
modality (i.e., text and image) denoted as $m_{1}$ and $m_{2}$. The multimodal
dataset is symbolized as $D=\\{D_{c},D_{m_{1}},D_{m_{2}}\\}$. Here,
$D_{c}=\\{x_{m_{1}}^{i},x_{m_{2}}^{i},y^{i}\\}$ signifies the complete set
containing both modality input and label, while
$D_{m_{1}}=\\{x_{m_{1}}^{j},y^{j}\\}$ and
$D_{m_{2}}=\\{x_{m_{2}}^{k},y^{k}\\}$ represent subsets with a single
modality. In accordance with prior work [16], we assign meaningless values to
missing-modality instances, like empty text or an image with a pixel value of
one. To retain the multimodal input structure, we introduce placeholder
inputs, $\tilde{x}_{m_{1}}$ and $\tilde{x}_{m_{2}}$ (e.g., empty strings or
pixels for text/images), for the absent modality data. This leads to
$\tilde{D}_{m_{1}}=\\{x_{m_{1}}^{j},\tilde{x}_{m_{2}}^{j},y^{j}\\}$ and
$\tilde{D}_{m_{2}}=\\{\tilde{x}_{m_{1}}^{k},x_{m_{2}}^{k},y^{k}\\}$.
Consequently, the multimodal dataset with missing modalities is denoted as
$\tilde{D}=\\{D_{c},\tilde{D}_{m_{1}},\tilde{D}_{m_{2}}\\}$. This results in
datasets covering different scenarios of missing data, including complete data
$D_{c}$, text-only data $D_{m_{1}}$, and image-only data $D_{m_{2}}$. The
overall dataset utilized in the experiments is represented as $\tilde{D}$. The
objective of the task is to enhance the ability to generalize in cases where
modalities are missing. To tackle this, we concentrate on MAPs, as proposed in
the previous study [16], to strengthen the resilience of prompts when dealing
with missing modalities.

###  2.2 Revisiting multimodal prompting

For the image-text multi-modality, there are three missing modality cases
(i.e., image-only, text-only, and modality-complete). Lee et al. [16]
introduce MAPs to categorize input data. Depending on the missing modality
case of the input data, the corresponding prompt is selected. For example,
$P_{i}$ for the image-only sample, $P_{t}$ for the text-only sample, and
$P_{c}$ for the modality-complete sample as depicted in Fig. 2. The prompt is
concatenated to the feature in the multimodal transformer and updated through
backpropagation in the training phase. In this way, the prompt distinguishes
between all missing modality cases, and the pretrained backbone transfers
knowledge to the prompt accordingly.

###  2.3 Prompt design

Instead of using prompts for missing awareness, we design MSPs that capture
the distinct characteristics of each modality. Note that $P_{is}$ and $P_{ts}$
are our image-specific prompts and text-specific prompts, respectively. Each
prompt learns knowledge from the pretrained backbone specific to its
corresponding modality. As shown in Fig. 2, we select $P_{is}$ and $P_{ts}$
for the image-only case and text-only case, respectively. Instead of $P_{c}$
in MAPs, we do not have a modality-complete case-specific prompt. Rather, we
simply combine $P_{is}$ and $P_{ts}$ to handle the modality-complete case. By
utilizing the element-wise summation of two prompts, both prompts $P_{is}$ and
$P_{ts}$ are updated when the input is a modality-complete case. Generating
differentiable complete case-specific prompts based on the given text and
image modalities, rather than random generalization as used in MAPs, allows
each prompt to learn more frequently while reducing the number of prompts.

Furthermore, we incorporate an additional loss term $L_{ortho}$ to enforce
orthogonality among MSPs. By reducing the similarity between $P_{is}$ and
$P_{ts}$, we maximize the gap between the two prompts to ensure the diversity
of information learned from image and text. Therefore, we minimize the cosine
similarity between $P_{ts}$ and $P_{is}$ as follows:

$L_{ortho}=\frac{|f(P_{is})\cdot
f(P_{ts})|}{\max\left(\left\|f(P_{is})\right\|_{2}\cdot\left\|f(P_{ts})\right\|_{2},\epsilon\right)},$
(1)

where $f$ is the flatten function, $|.|$ is the absolute value function,
$\left\|.\right\|_{2}$ is the L2-norm, and $\epsilon$ is a small value to
avoid division by zero.

###  2.4 Training objectives

For finetuning, we freeze all parameters in the text encoder, image encoder,
and multimodal transformer and train only MSPs and task-specific layers
(pooler and classifier). We note that $L_{cls}$ is a task-specific objective
function (e.g., binary cross-entropy loss for movie genre classification and
cross-entropy loss for food classification). Finally, the overall objective
function $L_{total}$ for finetuning our model can be represented as follows:

$L_{total}=L_{cls}+\lambda L_{ortho},$ (2)

where $\lambda$ is the balancing hyperparameter.

##  3 Experiments

###  3.1 Datasets and evaluation metrics

MM-IMDb [18] is a comprehensive movie genre multi-label classification dataset
that incorporates both image and text modalities. The dataset offers various
multi-label movie categories, such as action, comedy, drama, thriller, and
more. UPMC Food-101 [19] is a food category multi-class classification dataset
including image and text. All images are collected from the Google Image
Search engine and not post-filtered by human labor so that each category
contains about $5\%$ noisy images. In the case of MM-IMDb, the multi-label
classification performance is evaluated using F1-Macro. For UPMC Food-101, we
employ classification accuracy as the evaluation metric.

###  3.2 Implementation details

For a fair comparison, we followed the experiment settings with previous work
[16]. We adopted a multimodal transformer ViLT [20] pretrained on large-scale
vision-language datasets as our backbone network. For MAPs and our MSPs, we
used input-level prompting, which attaches prompts to the input of the multi-
head self-attention layer. We finetuned the entire pretrained ViLT with a
target dataset for comparison. In contrast, we froze the pretrained ViLT and
solely trained prompts and task-specific layers for MAPs and our MSPs as
depicted in Fig 2. We set the missing rate to 70% in all our experiments,
which is the total percentage of modalities removed among image or text
modalities. To consider various missing modality scenarios, we set up a total
of three scenarios, including two scenarios in which 70% of image or text were
removed and a scenario in which both modalities were equally missing by 35%.
Three scenarios were used for both training and inference as shown in Table 1
and 2. We set the balancing hyperparameter $\lambda$ to 0.15 and 0.1 for MM-
IMDb and UPMC Food-101 datasets, respectively. We conducted all experiments,
including ViLT, MAPs, and our method, with the same initial learning rate
$1\times 10^{-2}$, weight decay $2\times 10^{-2}$, AdamW optimizer [21], and
batch size of 6 by utilizing an NVIDIA RTX 3090 GPU.

###  3.3 Comparisons with the previous method

We show quantitative results of ViLT, MAPs, and our MSPs on MM-IMDb and UPMC
Food-101 datasets in Table 1. Compared to ViLT, MAPs demonstrate superior
performance when the missing modality cases encountered during inference align
with those seen during training. However, MAPs can not cope with the unseen
case where the modality composition ratio of the training and inference is
unbalanced. For example, consider a training set consisting of 100$\%$ image-
modality and 30$\%$ text-modality samples. In this scenario, MAPs successfully
train the modality-complete prompt $P_{c}$ and the image-only prompt $P_{i}$
while the text-only prompt $P_{t}$ remains unlearned. Consequently, the
prediction of a text-only sample is likely inaccurate due to the absence of
information in $P_{t}$. Due to the reasons mentioned above, MAPs lack
robustness when faced with missing modality cases that were not encountered
during the training phase, and they even demonstrate inferior performance
compared to ViLT. On the other hand, it shows that proposed MSPs achieve
robustly uniform performance improvement across various missing modality
scenarios. Regardless of the missing modality case, our approach trains prompt
for learning information from each modality in the input sample, as mentioned
in Section 2. In summary, MSPs require fewer prompts than MAPs while
outperforming both ViLT and MAPs in terms of both performance and robustness.

Datasets | Missing Rate | Training | Inference | ViLT [20] | MAPs [16] |  MSPs (Ours)
---|---|---|---|---|---|---
Image | Text | Image | Text
MM-IMDb [18] (F1-Macro) | 70% | 100% | 30% | 100% | 30% | 34.26 |  36.89 | 38.34
30% | 100% |  27.25 | 20.17 | 34.11
65% | 65% |  32.51 | 29.79 | 36.97
30% | 100% | 100% | 30% |  30.00 | 24.31 | 34.32
30% | 100% | 37.63 |  47.18 | 47.45
65% | 65% | 34.47 |  37.25 | 42.17
65% | 65% | 100% | 30% | 33.76 | 37.80 |  37.30
30% | 100% | 36.48 |  42.72 | 44.81
65% | 65% | 36.54 |  40.84 | 42.03
Food-101 [19] (Accuracy) | 100% | 30% | 100% | 30% | 66.01 |  73.71 | 73.77
30% | 100% |  43.67 | 27.46 | 57.22
65% | 65% |  54.75 | 50.79 | 65.13
30% | 100% | 100% | 30% | 41.94 | 29.71 | 56.87
30% | 100% |  76.40 | 86.34 | 86.34
65% | 65% |  59.03 | 57.92 | 71.88
65% | 65% | 100% | 30% | 64.25 |  71.17 | 71.58
30% | 100% | 73.62 |  85.55 | 85.91
65% | 65% | 68.60 |  78.49 | 78.89

Table 1: Comparison on the MM-IMDb [18] and UPMC Food-101 [19] datasets with
missing rate=70$\%$ under various training and inference missing-modality
cases. We reproduced previous work using official code with our experiment
setting, so the results of previous work are different from the original
paper. The best value is in bold, and the second best is underlined.

Training | Inference | ViLT [20] | +MSPs | +$L_{ortho}$
---|---|---|---|---
Image | Text | Image | Text
100% | 30% | 100% | 30% | 34.26 |  37.83 | 38.34
30% | 100% | 27.25 |  32.11 | 34.11
65% | 65% | 32.51 |  34.90 | 36.97
30% | 100% | 100% | 30% | 30.00 |  33.22 | 34.32
30% | 100% | 37.63 |  46.95 | 47.45
65% | 65% | 34.47 |  41.24 | 42.17
65% | 65% | 100% | 30% | 33.76 | 38.07 |  37.30
30% | 100% | 36.48 |  44.14 | 44.81
65% | 65% | 36.54 |  41.38 | 42.03

Table 2: Ablation study of our modality-specific prompts and orthogonal loss
$L_{ortho}$ on MM-IMDb [18] with missing rate=70$\%$. The evaluation metric is
F1-Macro. The best value is in bold, and the second best is underlined.

###  3.4 Ablation study

As shown in Table 2, we conducted an ablation study on MM-IMDb to verify that
our MSPs and orthogonal loss $L_{ortho}$ are effective. MSPs not only yield
improvements in performance but also enhance the robustness for unseen missing
modality cases. Furthermore, $L_{ortho}$ guides the prompts $P_{ts}$ and
$P_{is}$ to be orthogonal and acquire distinct information from each modality.
This strategy significantly contributes to enhancing performance. In
conclusion, our study demonstrates that leveraging our novel prompt design and
orthogonal loss for extracting unique information from each modality improves
both performance and robustness.

##  4 Conclusion

In this paper, we have proposed a simple yet effective prompt design for
robust real-world missing modality scenarios. We introduce modality-specific
prompts that learn distinct information from each modality distribution so
that they are missing-modality-case-agnostic. Prompt-based learning with
orthogonal loss function enforces orthogonality among prompts and enlarges the
diversities between different modalities. Evaluations conducted on two
benchmark datasets show that our method outperforms in terms of performance
and robustness compared to previous work. As the integration of increasingly
diverse multimodal signals, we firmly believe that our prompt design will open
the way for multimodal finetuning with missing modalities. In the future, we
will apply our technique to the task with more modality settings, e.g., action
recognition with RGB, depth, and IR as inputs.

##  5 Acknowledgement

This work was supported by Institute of Information communications Technology
Planning Evaluation (IITP) grant funded by the Korea government(MSIT) (No.
2021-0-00193, Development of photorealistic digital human creation and 30fps
realistic rendering technology).

## References

  * [1] M. Ma, J. Ren, L. Zhao, D. Testuggine, X. Peng, Are multimodal transformers robust to missing modality?, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18177–18186.
  * [2] Y. Li, Z. Yu, S. Xiang, T. Liu, Y. Fu, Av-tad: Audio-visual temporal action detection with transformer, in: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2023, pp. 1–5.
  * [3] J. Chen, C. M. Ho, Mm-vit: Multi-modal video transformer for compressed video action recognition, in: Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2022, pp. 1910–1921.
  * [4] A. Shahroudy, T.-T. Ng, Y. Gong, G. Wang, Deep multimodal feature analysis for action recognition in rgb+ d videos, IEEE transactions on pattern analysis and machine intelligence 40 (5) (2017) 1045–1058.
  * [5] I. Rodomagoulakis, N. Kardaris, V. Pitsikalis, E. Mavroudi, A. Katsamanis, A. Tsiami, P. Maragos, Multimodal human action recognition in assistive human-robot interaction, in: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, 2016, pp. 2702–2706.
  * [6] J. Huang, J. Tao, B. Liu, Z. Lian, M. Niu, Multimodal transformer fusion for continuous emotion recognition, in: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2020, pp. 3507–3511.
  * [7] S. Dutta, S. Ganapathy, Multimodal transformer with learnable frontend and self attention for emotion recognition, in: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2022, pp. 6917–6921.
  * [8] D. Yang, S. Huang, H. Kuang, Y. Du, L. Zhang, Disentangled representation learning for multimodal emotion recognition, in: Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 1642–1651.
  * [9] S. Woo, S. Lee, Y. Park, M. A. Nugroho, C. Kim, Towards good practices for missing modality robust action recognition, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, 2023, pp. 2776–2784.
  * [10] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017).
  * [11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, in: International Conference on Learning Representations, 2021.
  * [12] M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, S.-N. Lim, Visual prompt tuning, in: European Conference on Computer Vision, Springer, 2022, pp. 709–727.
  * [13] Z. Wang, Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, T. Pfister, Learning to prompt for continual learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 139–149.
  * [14] M. Tsimpoukelli, J. L. Menick, S. Cabi, S. Eslami, O. Vinyals, F. Hill, Multimodal few-shot learning with frozen language models, Advances in Neural Information Processing Systems 34 (2021) 200–212.
  * [15] K. Zhou, J. Yang, C. C. Loy, Z. Liu, Learning to prompt for vision-language models, International Journal of Computer Vision 130 (9) (2022) 2337–2348.
  * [16] Y.-L. Lee, Y.-H. Tsai, W.-C. Chiu, C.-Y. Lee, Multimodal prompting with missing modalities for visual recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 14943–14952.
  * [17] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, I. Misra, Imagebind: One embedding space to bind them all, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15180–15190.
  * [18] J. Arevalo, T. Solorio, M. Montes-y Gómez, F. A. González, Gated multimodal units for information fusion, in: International Conference on Learning Representations Workshops, 2017.
  * [19] X. Wang, D. Kumar, N. Thome, M. Cord, F. Precioso, Recipe recognition with large multimodal food dataset, in: 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), IEEE, 2015, pp. 1–6.
  * [20] W. Kim, B. Son, I. Kim, Vilt: Vision-and-language transformer without convolution or region supervision, in: International Conference on Machinee Learning, PMLR, 2021, pp. 5583–5594.
  * [21] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, in: International Conference on Learning Representations, 2019.


# Cluster-based pruning techniques for audio data

###### Abstract

Deep learning models have become widely adopted in various domains, but their
performance heavily relies on a vast amount of data. Datasets often contain a
large number of irrelevant or redundant samples, which can lead to
computational inefficiencies during the training. In this work, we introduce,
for the first time in the context of the audio domain, the $k$-means
clustering as a method for efficient data pruning. $K$-means clustering
provides a way to group similar samples together, allowing the reduction of
the size of the dataset while preserving its representative characteristics.
As an example, we perform clustering analysis on the keyword spotting (KWS)
dataset. We discuss how $k$-means clustering can significantly reduce the size
of audio datasets while maintaining the classification performance across
neural networks (NNs) with different architectures. We further comment on the
role of scaling analysis in identifying the optimal pruning strategies for a
large number of samples. Our studies serve as a proof-of-principle,
demonstrating the potential of data selection with distance-based clustering
algorithms for the audio domain and highlighting promising research avenues.

Index Terms—  $k$-means clustering, data pruning, keyword spotting

##  1 Introduction

With the advent of deep learning and the exponential growth in the amount of
data available, the demand for efficient storage and processing has become
crucial. For instance, a recently released large language model from OpenAI –
GPT-4 – is reportedly about six times larger than its predecessor, with one
trillion hyperparameters. Such a vast number of trainable parameters not only
slows down the inference time and increases the computational cost of the
model, but also requires an enormous amount of samples to train and
generalize. Identifying the redundancies in datasets or models is therefore
essential to reduce the computational demands. In particular, discarding – or
pruning – the irrelevant information before the training is the most general,
model-independent approach to optimize NNs. Formally, pruning can be viewed as
a discrete optimization problem, with the objective of recognizing the largest
subset of irrelevant data from the training dataset while minimizing the
change in network parameters resulting from removing the selected subset [1].
Data pruning techniques have been used so far to improve predictions for movie
ratings [2] or image classification [3], but not for audio data. Within the
audio domain, significant efforts have been rather devoted to model pruning
using, for example, sparse architecture search and weight pruning [4, 5, 6, 7,
8]. Relative to image datasets, audio datasets are generally smaller and can
benefit from optimal data selection techniques for small datasets [9, 10] or
various cross-validation schemes implemented in deep learning frameworks such
as Keras [11].

Here, we explore a model-agnostic pruning technique based on unsupervised
clustering analysis for audio data. This approach allows us to reduce the size
of training sets by exploiting similarities in the multidimensional feature
space of a pre-trained large audio model. Reduced datasets can then be
effectively used with significantly smaller neural networks without
compromising accuracy. The goal of this paper is therefore twofold: (i) to
discuss audio data pruning with $k$-means algorithm on a KWS dataset (Google
Speech Commands Dataset V3 containing one-second samples with 36 classes
[12]), and (ii) to provide guidance on extending this approach to more complex
datasets and different tasks.

The paper is structured as follows: Section 2 introduces related work on data
and model pruning, Section 3 outlines the proposed audio data pruning method,
and Section 4 describes the experimental setup with obtained results. Finally,
Section 5 concludes the paper and outlines the future work.

##  2 Related work

###  2.1 Parameter pruning in (deep) neural networks

Reducing the time and computational complexity of a neural network can be
achieved by removing unnecessary weights, layers, or channels. Neural network
pruning can be performed at initialization, for example, with single-shot
network pruning [13], dynamically during the training [14], or iteratively to
find the optimal subnetwork, such as with the lottery ticket hypothesis [15].
Another approach towards optimal data selection is to construct simpler proxy
models from the target network and investigate their performance [16].

###  2.2 Score-based data pruning methods

EL2N score Normed error-L2 distance can be used early in the training process
to determine a subset of data points that carry the most information [17].
Removing easy examples (characterized by the smallest EL2N scores) reduces the
computational cost while maintaining test accuracy. This data selection
approach can be further improved by combining it with Selective-Backprop [18]
or distillation [19].
Forgetting score Forgetting score tracks the number of times when a neural
network misclassifies an example that it was previously classified correctly
[20, 21]. Samples that are rarely forgotten have a smaller impact on training
accuracy and, therefore, can be pruned. Calculating the score is done at later
stages of the training process as it requires collecting the statistics during
the training.
Memorization and influence estimations Memorization is often viewed as a
negative trait since it implies that a neural network cannot effectively
generalize [22, 23]. The memorization score quantifies the increase in the
probability of correctly predicting the label when an example is present in
the training set compared to when it is absent. Examples with high
memorization scores are atypical and cannot be learned from the remaining
data. In addition, an influence score measures the effect of adding or
removing an example from the training dataset on the test set predictions of a
learned neural network. As a result, the samples with high memorization and
high influence are considered relevant.

These scoring metrics have been primarily used in image classification tasks
that involve large datasets and are rather model-specific. They necessitate
iterative training to calculate the score, prune the data accordingly, and
subsequently train the model from the beginning. For example, a Q-score metric
determines if a given sample will likely be misclassified based on the data’s
self-supervised feature representation and subsets with strong cross-
correlation [24].

##  3 Methodology

Standard validation techniques (such as $k$-fold cross-validation or
bootstrap) often used for small datasets rely on repeated random splits of the
dataset. Inspired by a computer vision method [3], we propose to perform more
informative audio data subsampling, as described in the following sections.

###  3.1 High-dimensional audio embeddings and clustering

In image classification, the unsupervised $k$-means clustering algorithm was
employed in the embedding space [3] In our work, we adopt a similar clustering
technique, but given the different structure of audio data, the results may
not generalize in the same way.

The proposed analysis starts with a high-dimensional audio characterization –
for example, using wav2vec2 embeddings [25] – with the training data as input.
The wav2vec2 model was trained to optimize a combination of two losses, namely
contrastive loss and diversity loss. It is expected that the samples with
similar characteristics (like the same word spoken by different persons or the
same high tone of different musical instruments) will be close in feature
space representation. The similarities between the samples can then be
quantified using the distance-based clustering algorithm.

In the $k$-means clustering step, each example is represented as a point in
high-dimensional space, and its proximity to a centroid is determined by the
Euclidean distance metric. Typical (hereafter denoted as simple) samples are
close to the cluster centers, whereas distinct (hereafter denoted as hard)
examples are located away from the centroids. Therefore, we can perform simple
(hard) pruning by removing the closest (farthest away) points, depending on
the distance and the specified fraction of data to be kept. To illustrate the
proposed data selection approach, in Fig. 1, we demonstrate the $k$-means
clusters obtained by projecting the data onto a 2D plane with principal
component analysis (PCA).

Fig. 1: Cluster-based data pruning with $k$ = 15 on a 2D projected dataset
(with PCA), where we remove 50% of (a) hard and (b) simple examples. We
emphasize that our subsequent analysis solely focuses on the high-dimensional
representation of the embeddings.

The original formulation of $k$-means relies on the pairwise Euclidean
distance. Recently, $k$-means has been adapted to work in hyperbolic space,
which is better suited for hierarchical data [26, 27]. Other clustering
algorithms, such as $k$-medians (based on Manhattan distance) or $k$-medoids
(which minimizes arbitrary distance functions), may be more appropriate for
audio data. Therefore, a systematic analysis of clustering-based pruning with
various distance metrics is needed.

###  3.2 Class selection and dataset imbalance

The number of classes in the $k$-means algorithm is a hyperparameter that is
not known a priori. To estimate the optimal $k$, we can use PCA and compute
the proportion of variance explained. In our example, we found that a
substantial number (over 150) of principal components are needed to account
for around 80% of the variance. Importantly, performing PCA before clustering
reduces the dimensionality of the data, thus accelerating the clustering
analysis and data selection. As the average complexity of the $k$-means
algorithm is $\mathcal{O}(kNT)$, where $N$ stands for the number of samples
and $T$ is the number of iterations, we suggest using state-of-the-art
libraries (such as Faiss [28]) for large datasets. The obtained optimal number
of $k$-means classes might differ from the number of label classes, suggesting
that each class might contain additional information beyond the specific
keyword it represents. We speculate that these latent features may be related
to factors such as the recording equipment used, background noise, or general
characteristics of the speaker (for example, gender, pitch, or speech rate).
To confirm the intuition behind these results, we propose investigating the
cross-correlations within the clusters with Spearman’s rank correlation.

Clustering algorithms often reinforce existing class imbalances, which can
lead to a degradation in NN performance. Based on the Shannon entropy, we
define $balance=-\sum_{i}^{c}p_{i}\log(p_{i})/\log(c)$, where $c$ is the
number of classes and $p_{i}$ is the ratio of the number of elements in class
$i$ to the total number of samples in the dataset. The effect of imbalance can
be countered by data augmentation. However, augmentation creates adversarial
examples, which can interfere with cluster-based data pruning. To understand
the interplay between augmentation and pruning, it would be necessary to
examine the relationship between hard and simple examples. This can be done,
for instance, by quantifying the information content of the samples (see [29,
30]).

##  4 Experiments and results

The purpose of this section is to demonstrate the practical applications of
cluster-based data selection described in the previous section.

###  4.1 Data preparation and training

We prepare four training sets by gradually removing simple or hard examples
from the original training set (from 10% to 40% of data points excluded). We
then use these subsets to train three KWS systems of different sizes and
measure their performance as we manipulate the number of training samples $N$.
This allows us to investigate the asymptotic, large $N$ limit, as well as the
small $N$ regime, which is relevant for devices with limited memory. The audio
classification is performed by extracting Mel Frequency Cepstral Coefficients
(MFCCs) and passing them through a simple Convolutional Neural Network (CNN)
classifier. We constructed tiny, small, and large NNs, with 3.5k, 29k and 270k
parameters respectively, based on the original LeNet architecture [31].

We obtained the best performance of the trained models for $k$ = 155; we
emphasize that the optimal value of $k$ is dataset-specific. Table 1 presents
the dataset imbalance for the simple and hard pruning methods at exemplary
pruning fractions. Notably, there is a class within our dataset that contains
only the background noise, with no keywords present. As the samples in this
class have a clearly distinct structure from those with other labels, over
half of them are removed by simple pruning.

pruning | None | 20 % hard | 40% hard | 20% simple | 40% simple
---|---|---|---|---|---
balance | 0.982 | 0.975 | 0.966 | 0.984 | 0.984

Table 1: Dataset class imbalance for different pruning methods. A dataset with
equally distributed samples is characterized by $balance=1$. Simple pruning
slightly improves the balance of the original dataset, while hard pruning
leads to a more significant imbalance.

###  4.2 Scaling analysis

It has been observed across various domains [32, 33] that the loss should
scale as a power law with dataset size, loss $\sim 1/N^{\nu}$, with $\nu<1$.
While accuracy is an intuitive metric for classification tasks, it does not
follow any scaling law. To quantify the asymptotic performance of an NN, we
extract the scaling exponent $\nu$ from the dependence of the loss on the
number of training samples. $\nu$ can be straightforwardly obtained from a
linear regression of this dependence in a log-log scale.

###  4.3 Results

Fig. 2, shows the test loss and test accuracy against training set size curves
for the most effective pruning strategies across tiny and large architectures.
The exponents $\nu$ from all experiments are collected in Table 2. When the
subset of training samples $N$ is smaller than the size of the entire training
set, and a pruning fraction is small, it is necessary to reduce the pruned
dataset further. In such cases, we randomly remove examples from the reduced
dataset. We compare the results for various pruning strategies with random,
unstructured pruning and the Q-score, informed pruning baseline [24]. As our
datasets are relatively small, obtaining reliable performance estimates
requires repeated cross-validation. For each value of $N$, we randomly
partitioned the pruned datasets into a training set 100 times and calculated
the averaged scores. This additional step is needed to demonstrate the effect
of pruning clearly and does not significantly increase computational overhead.

Fig. 2: Performance comparison of tiny and large models. For both
architectures, we observe that simple pruning outperforms random pruning. This
statement is valid for both loss and accuracy. Since the KWS dataset is rather
limited, we add some extrapolation to show the potential of our method for
larger datasets. The baseline Q-score pruning method performs as the random
pruning and thus is not shown here.

Our analysis reveals that the scaling behavior of large and small models is
similar, while the tiny network performs distinctly. Specifically, we observe
that as $N$ increases, there may be either a breakdown of power-law scaling or
a crossover between two regimes with different scaling exponents $\nu$. We
decided to exclude these points for the exponents computations. In fact, the
scaling with respect to the dataset size or model parameters may be a more
complicated, not necessarily monotonic function.

Pruning method | $\nu_{tiny}$ | $\nu_{small}$ | $\nu_{large}$
---|---|---|---
Random pruning | 0.244 | 0.396 | 0.391
10 % hard | 0.249 | 0.390 | 0.386
20 % hard | 0.240 | 0.386 | 0.389
30 % hard | 0.239 | 0.386 | 0.388
40 % hard | 0.248 | 0.377 | 0.387
10 % simple | 0.242 | 0.397 | 0.393
20 % simple | 0.250 | 0.406 | 0.401
30 % simple | 0.243 | 0.408 | 0.408
40 % simple | 0.260 | 0.413 | 0.421
Table 2: Scaling exponents $\nu$ ($\uparrow$) obtained from $loss=f(N)$
dependencies. Larger $\nu$ indicates faster loss decay as $N$ increases. The
estimated exponent for the tiny model has an error of $0.01$, while for the
small and large models, the error becomes smaller and is $0.005$. We find that
small and large NNs are affected similarly by pruning; tiny model is
characterized by a smaller $\nu$. Simple pruning generally leads to more
favorable scaling, while hard pruning may only benefit specific, fine-tuned
cases.

##  5 Discussion

While the differences in the exponents $\nu$ are not pronounced (see Table 2),
our data selection method has an apparent effect on the scaling behavior:

Simple pruning, where we discard typical samples (close to the centroids),
results in a drop in accuracy compared to random pruning when $N$ is small. As
$N$ increases, the accuracy becomes comparable between these two pruning
strategies. At around $N=5\cdot 10^{4}$, the accuracy for the pruned datasets
surpasses that of the randomly pruned case. Additionally, we have observed
that increasing the fraction of samples removed has a more noticeable impact
on accuracy for small and large models. For the tiny model, the 40% pruning
gives rise to the largest exponent. The obtained results of the proposed
simple audio data pruning differ from the original image data pruning method
[3], where the authors claim that retaining easy examples (or, equivalently,
removing hard) is more important for limited datasets. However, they used
massive image databases such as ImageNet to confirm their analytical findings
on the knowledge distillation setting.

Hard pruning, where we remove most informative samples (located far from the
centroids), leads to a slight overall decrease in accuracy. At the same time,
the extracted scaling exponents $\nu$ are smaller. For the tiny model, the
values of $\nu$ are characterized by significant uncertainty. We verified that
averaging over more training does not decrease the variance. Only the small
model exhibits a monotonic decrease in performance as we remove more samples.

Generally, we found that large fractions of simple/hard pruning do not offer
advantages over random pruning when $N$ is small. This is not unexpected, as
the networks may have limited exposure to diverse training data and,
therefore, not generalize well. Above $N=5\cdot 10^{4}$, we achieve an
accuracy of over 80% for almost all pruning techniques, except for hard
pruning at 40% samples removed. Remarkably, some classes (such as the ones
corresponding to the keywords ’on’ and ’no’) are strongly impacted by simple
pruning while remaining almost unaffected by hard pruning. This finding
suggests that a notion of similarity captured by $k$-means clustering may be
close to human auditory perception. As the $k$-means appears to identify
hidden patterns, clustering may also help in improving existing data labeling.
The question of to what extent the results from the scaling analysis are
transferable to other datasets remains open.

Our work is a first step towards informed cluster-based audio data pruning,
and it would be of great interest to apply the proposed workflow to a larger
KWS dataset such as SiDi [34], different audio processing tasks, and
eventually with various audio embeddings. The implementation of the proposed
method and experiments is open-sourced111https://github.com/Boris-
Bergsma/Audio_pruning.

## References

  * [1] Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li,  “Dataset pruning: Reducing training data by examining generalization influence,”  in The Eleventh International Conference on Learning Representations, 2023.
  * [2] Arun Thundyill Saseendran, Lovish Setia, Viren Chhabria, Debrup Chakraborty, and Aneek Barman Roy,  “Impact of data pruning on machine learning algorithm performance,”  arXiv:1901.10539, 2019.
  * [3] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos,  “Beyond neural scaling laws: beating power law scaling via data pruning,”  in Advances in Neural Information Processing Systems, 2022.
  * [4] Harsh Sinha, Vinayak Awasthi, and Pawan K. Ajmera,  “Audio classification using braided convolutional neural networks,”  IET Signal Processing, vol. 14, no. 7, pp. 448–454, 2020.
  * [5] Fady Medhat, David Chesmore, and John Robinson,  “Masked conditional neural networks for sound classification,”  Applied Soft Computing, vol. 90, pp. 106073, 2020.
  * [6] Jingyi Wang and Shengchen Li,  “Keyword spotting system and evaluation of pruning and quantization methods on low-power edge microcontrollers,”  arXiv:2208.02765, 2022.
  * [7] Flavio Martinelli, Giorgia Dellaferrera, Pablo Mainar, and Milos Cernak,  “Spiking neural networks trained with backpropagation for low power neuromorphic implementation of voice activity detection,”  in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 8544–8548.
  * [8] David Südholt, Alec Wright, Cumhur Erkut, and Vesa Välimäki,  “Pruning deep neural network models of guitar distortion effects,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 256–264, 2023.
  * [9] “Sample-size determination methodologies for machine learning in medical imaging research: A systematic review,”  Canadian Association of Radiologists Journal, vol. 70, no. 4, pp. 344–353, 2019.
  * [10] Jörg Bornschein, Francesco Visin, and Simon Osindero,  “Small data, big decisions: Model selection in the small-data regime,”  in Proceedings of the 37th International Conference on Machine Learning. 2020, ICML’20, JMLR.org.
  * [11] François Chollet et al.,  “Keras,” https://keras.io, 2015.
  * [12] Pete Warden,  “Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition,”  arXiv:1804.03209, 2018.
  * [13] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr,  “SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY,”  in International Conference on Learning Representations, 2019.
  * [14] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu,  “Sparse training via boosting pruning plasticity with neuroregeneration,”  in Advances in Neural Information Processing Systems, 2021, vol. 34.
  * [15] Jonathan Frankle and Michael Carbin,  “The lottery ticket hypothesis: Finding sparse, trainable neural networks,”  in International Conference on Learning Representations, 2019.
  * [16] Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia,  “Selection via proxy: Efficient data selection for deep learning,”  in International Conference on Learning Representations, 2020.
  * [17] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite,  “Deep learning on a data diet: Finding important examples early in training,”  in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, Eds. 2021, vol. 34, pp. 20596–20607, Curran Associates, Inc.
  * [18] Yu Shen Lu, Daniel Zamoshchin, and Zachary Chen,  “Diet selective-backprop: Accelerating training in deep learning by pruning examples,”  http://cs231n.stanford.edu/reports/2022/pdfs/93.pdf.
  * [19] Anirudh Sundar, Gokce Keskin, Chander Chandak, I-Fan Chen, Pegah Ghahremani, and Shalini Ghosh,  “Prune then distill: Dataset distillation with importance sampling,”  in ICASSP 2023, 2023.
  * [20] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon,  “An empirical study of example forgetting during deep neural network learning,”  CoRR, vol. abs/1812.05159, 2018.
  * [21] Abdul Hameed Azeemi, Ihsan Ayyub Qazi, and Agha Ali Raza,  “Dataset Pruning for Resource-constrained Spoofed Audio Detection,”  in Proc. Interspeech 2022, 2022, pp. 416–420.
  * [22] Vitaly Feldman,  “Does Learning Require Memorization? A Short Tale about a Long Tail,”  in Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, New York, NY, USA, 2020, STOC 2020, p. 954–959, Association for Computing Machinery.
  * [23] Vitaly Feldman and Chiyuan Zhang,  “What neural networks memorize and why: Discovering the long tail via influence estimation,”  in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, Eds. 2020, vol. 33, pp. 2881–2891, Curran Associates, Inc.
  * [24] Neha Kalibhat, Kanika Narang, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi,  “Measuring self-supervised representation quality for downstream classification using discriminative features,” 2023.
  * [25] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  arXiv:2006.11477, 2020.
  * [26] Ines Chami, Albert Gu, Vaggos Chatziafratis, and Christopher Ré,  “From trees to continuous embeddings and back: Hyperbolic hierarchical clustering,”  Advances in Neural Information Processing Systems, vol. 33, pp. 15065–15076, 2020.
  * [27] Darius Petermann, Gordon Wichern, Aswin Subramanian, and Jonathan Le Roux,  “Hyperbolic audio source separation,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [28] Jeff Johnson, Matthijs Douze, and Hervé Jégou,  “Billion-scale similarity search with GPUs,”  IEEE Transactions on Big Data, vol. 7, no. 3, pp. 535–547, 2019.
  * [29] Boris Bergsma, Minhao Yang, and Milos Cernak,  “PEAF: Learnable Power Efficient Analog Acoustic Features for Audio Recognition,”  in Proc. Interspeech 2022, 2022, pp. 381–385.
  * [30] Devavrat Tomar, Guillaume Vray, Behzad Bozorgtabar, and Jean-Philippe Thiran,  “Tesla: Test-time self-learning with automatic adversarial augmentation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 20341–20350.
  * [31] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner,  “Gradient-based learning applied to document recognition,”  Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
  * [32] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou,  “Deep learning scaling is predictable, empirically,”  CoRR, vol. abs/1712.00409, 2017.
  * [33] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei,  “Scaling laws for neural language models,”  arXiv:2001.08361, 2020.
  * [34] Michel Cardoso Meneses, Rafael Bérgamo Holanda, Luis Vasconcelos Peres, and Gabriela Dantas Rocha,  “SiDi KWS: A Large-Scale Multilingual Dataset for Keyword Spotting,”  in Proc. Interspeech 2022, 2022, pp. 4616–4620.


# A Modular Multi-stage Lightweight Graph Transformer Network for Human Pose
and Shape Estimation from 2D Human Pose

Ayman Ali, Ekkasit Pinyoanuntapong, Pu Wang, Mohsen Dorodchi  College of
Computing and Informatics
University of North Carolina at Charlotte
Charlotte, United States
aali26, epinyoan, pwang13<EMAIL_ADDRESS>
###### Abstract

In this research, we address the challenge faced by existing deep learning-
based human mesh reconstruction methods in balancing accuracy and
computational efficiency. These methods typically prioritize accuracy,
resulting in large network sizes and excessive computational complexity, which
may hinder their practical application in real-world scenarios, such as
virtual reality systems. To address this issue, we introduce a modular multi-
stage lightweight graph-based transformer network for human pose and shape
estimation from 2D human pose, a pose-based human mesh reconstruction approach
that prioritizes computational efficiency without sacrificing reconstruction
accuracy. Our method consists of a 2D-to-3D lifter module that utilizes graph
transformers to analyze structured and implicit joint correlations in 2D human
poses, and a mesh regression module that combines the extracted pose features
with a mesh template to produce the final human mesh parameters.

##  I Introduction

The field of computer vision has witnessed significant advancements with the
advent of deep learning models in human pose estimation from monocular images.
Specifically, 2D pose estimation involves determining the locations of human
joint coordinates within a single image. In recent years, the research
community has shown great interest in reconstructing 3D human mesh
representations due to their practical applications in human-computer
interaction, gaming, and virtual systems. Despite the progress in both 2D and
3D pose estimation and human mesh reconstruction, reconstructing human mesh
from a single image remains challenging due to factors such as depth
ambiguity, complex backgrounds, challenging postures, and a lack of annotated
in-the-wild datasets.

The fundamental process of reconstructing a 3D human mesh involves estimating
the pose and shape parameters based on statistical body models, such as SMPL
[1] and [2]. The pose parameters $\theta$ define the global root orientation
and the relative rotations of human body joints represented in an axis-angle
sequence. Moreover, the shape parameters $\beta$ represent the learned linear
gender-based body shape derived from the CAESAR dataset [3] using principal
component analysis.

The literature on human mesh reconstruction primarily consists of two
paradigms: optimization-based and regression-based approaches. Optimization-
based methods minimize the difference between the re-projected 2D pose
regressed from the human mesh and the estimated 2D pose. In contrast,
regression-based methods directly predict the pose and shape parameters.

While deep learning models have progressed in human mesh reconstruction, the
choice of input data has a significant impact on the system’s pipeline. Recent
advances have adopted an end-to-end pipeline where the input is the scene
image, and the output is the predicted mesh. However, this approach can be
computationally intensive and does not meet the efficiency requirements of
various real-world applications, such as gaming and real-time mesh recovery.

To address the computational overhead, some researchers have proposed using
skeleton data as the input, as it is sparse [4]. However, solely relying on
skeleton data is insufficient to address the computational intensity. Zheng et
al. [5] proposed a lightweight graph-based transformer architecture to focus
on efficiency.

Nevertheless, reconstructing mesh parameters remains complex, as it is
influenced by a variety of factors, including 1) Dataset Characteristics where
the performance of a model is not significantly impacted by the indoor/outdoor
setting or the number of data points, but rather by human pose and shape,
camera characteristics, and backbone features [6]. Furthermore, a diverse
range of these attributes can improve the performance, while occlusion and
SMPL fittings can enhance recovery accuracy. 2) Dataset Mix is another factor
that affects the complexity of the task. The choice of datasets is critical
for the generalizability and accuracy of the model [6]. It is imperative to
use the same combination of datasets when evaluating and comparing the impact
of other factors, such as training algorithms. Comparing two network
architectures on different dataset combinations is considered unfair. To
establish a robust baseline model, it is recommended to use more challenging
datasets and increase their contribution during training. Finally, Noisy
annotations, which in case of having high proportions of noisy data samples,
can negatively impact model performance, particularly when both SMPL
annotations and keypoints are affected. However, slightly noisy SMPL data can
still have a positive effect on training.

Our objective in this research is to design a modular and end-to-end capable
graph-based transformer network that is capable of estimating the human shape
and pose, demonstrating comparable performance with state-of-the-art methods.

Specifically, we propose the following strategies to achieve this goal:

  * •

Construct a modular, multi-stage pipeline that is capable of end-to-end
estimation of human parameters.

  * •

separate the learning of human pose, shape, and camera parameters to improve
accuracy and robustness of the model.

##  II System Description

###  II-A Preliminary

The task of Human Mesh Recovery (HMR) from images without the utilization of
auxiliary devices, such as depth sensors, poses a challenge due to various
factors such as depth ambiguity, complex backgrounds, and diverse human poses.
Typically, deep learning networks are trained to estimate the human pose
parameters using a skinned vertex-based parametric human model such as SMPL
[1].

Kanazawa et al. proposed a method that minimizes the 2D reprojection loss of
joints without relying on paired 2D-to-3D supervision to estimate the pose and
shape parameters from an image [7]. Kolotouros et al. proposed a method that
iteratively optimizes the estimated parameters to produce a pixel-level
alignment with the original image [8]. Further, Kocabas et al. introduced
adversarial training to enhance the quality of the estimated mesh by utilizing
the large-scale motion capture dataset AMASS [9].

Recent advancements in 2D pose estimation have prompted researchers to explore
the potential of using off-the-shelf 2D pose estimation networks, such as
HRNet [10], to estimate human mesh parameters. The 2D skeleton estimated from
an image is fed to the human mesh parameters estimation network for further
processing and analysis.

Graph Convolution Networks (GCN) have received considerable attention in
recent years due to their ability to intuitively model data, such as
articulated models. In particular, many studies have adopted GCN to model 3D
human pose estimation [11, 12]. In the context of human skeletons, joints and
bones are mapped to vertices and edges, respectively, thus forming a graph
represented mathematically as $\mathcal{G}={v,\epsilon}$. Inspired by [5], we
use GCN to model the 2D pose features, where the output of the GCN layer is
expressed as:

$X^{\prime}=\sigma(A\times W)$ (1)

Where $\sigma(.)$ represents the Gaussian Error Linear Unit (GELU), $A$
represents the adjacency matrix, and $W$ represents the learnable weight
matrix.

Transformer Transformer [13] has remodeled many deep learning models in
computer vision. The ability to capture the global context understanding in
the features space led to significant advancements. Self-attention can be
described as a function to compute the attention matrix given a query matrix
$Q\in\mathbb{R}^{N\times d}$, key matrix $K\in\mathbb{R}^{N\times d}$ and
value matrix $V\in\mathbb{R}^{N\times d}$ where $N$ represents the number of
vectors in the sequence, and $d$ is the dimension. A scaling factor
($\frac{1}{\sqrt{d}}$) is utilized to alleviate the growth of the softmax
function’s magnitude. The scaled-dot product attention can be expressed as:

$Attention(Q,K,V)=Softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V$ (2)

Multi-headed self-attention utilized parallel scaled-dot attentions to project
the queries, keys, and values $h$ times with different linear projections to
$dk$, $dk$, and $DV$ dimensions. A concatenation of the $h$ head attention
output can be expressed as:

$\displaystyle MSA(Q,K,V)=Concat(H_{1},H_{2},....,H_{h})W_{out}$ (3)
$\displaystyle\text{where}\;H_{i}=Attention(Q_{i},K_{i},V_{i}),i\in[1,2,...,h]$

Figure 1: Modular Multi-stage Lightweight Graph Transformer Network for Human
Pose and Shape Estimation from 2D Human Pose

###  II-B System Architecture

At first, a projection layer encodes the 2D pose estimation to create latent
space embedding to be consumed by the 3D lifter. Supervised training is
adopted to train a 3D lifter module where we minimize the distance between the
predicted 3D and the ground truth. In addition to using L1 regularization, we
use the weak perspective camera model to reproject 3D to the 2D pose to ensure
better accuracy and alignment. Furthermore, the shape parameters are applied
to the mean template mesh. We experimented with using the mean mesh, which is
represented by $M\in\mathbb{R}^{6890}$, and the rest pose regressed from the
mean template mesh. We found that using mean mesh provided more gain than
using the rest pose. The pose angles are predicated by the iterative regressor
that consumes the features generated by the pose and shape estimator. Finally,
forward kinematics are employed to calculate the human mesh.

####  II-B1 3D-lifter Module

The 3D-lifter module leverages the graph transformer as described in [5] to
extract both global and local features from human pose data. First, a
projection layer encodes the pose into a latent space. The resulting features
are then fed into multiple parallel graph transformer blocks, which have small
embedding dimensions for improved efficiency. The outputs from these parallel
branches are fused to generate features that are used to predict 3D pose
estimation, human shape, and camera parameters.

Given a 2D skeleton denoted as $S\in\mathbb{R}^{j\times 2}$, where $j$
represents the number of joints and $2$ denotes the joint coordinates in the
$X,Y$ plane, the 2D-to-3D lifting task is performed by the Pose Analysis
Module (PAM). PAM returns: 1) features embedding $F\in\mathbb{R}^{J\times D}$,
where $J$ and $D$ denote the number of joints and the dimensions of the
features latent space, respectively; 2) 3D joints $P\in\mathbb{R}^{J\times
3}$, where $J$ denotes the number of joints and $3$ denotes the coordinates of
the joints in $[X,Y,Z]$; 3) human shape $\beta\in\mathbb{R}^{10}$, where $10$
represents the first $10$ coefficients of the PCA shape space; and 4) camera
parameters $C\in\mathbb{R}^{3}$, where $3$ denotes the global rotation,
translation, and scale.

####  II-B2 Pose and Shape Estimator

The Pose and Shape Estimator module aims to estimate human mesh parameters,
specifically joint angles. System modularity is a crucial goal in this work,
and to that end, the module is designed to consume either the 3D-lifter
features in an end-to-end training setup or the 3D pose in individual module
training. However, due to its lightweight design, the module may not have
enough capacity to accurately estimate the pose angle with its small embedding
size. To address this, the module is augmented with a mean mesh template as
introduced in [5] to enrich the feature representation through an additional
branch. The features produced by the pose feature and mesh template branches
are then fused to generate the final features space, which is subsequently
consumed by the iterative 3D regression as described in [14] to infer the
final 3D pose angle.

Given the pose features on the first branch, expressed as
$F_{pose}\in\mathbb{R}^{J\times D}$ where $J$ denotes the number of joints and
$D$ denotes the dimensions, and the mesh template features on the second
branch, expressed as $F_{template}\in\mathbb{R}^{T\times D}$. Both branches
are processed by the same transformer architecture to model the features. The
resulting features from both branches are combined into a single final
embedding, represented as $F_{PSE}\in\mathbb{R}^{(J+T)\times D}$. Finally, the
iterative 3D regression consumes the fused features to generate the pose
angles, expressed as $\theta\in\mathbb{R}^{72}$.

| Methods | Humans.6M
---|---|---
MPJPE | PA-MPJPE
Image Based | HMR | CVPR 2018 | 88.0 | 56.8
GraphCMR | CVPR 2019 | - | 50.1
SPIN | ICCV 2019 | - | 41.1
METRO | CVPR 2021 | 54.0 | 36.7
MeshGraphormer | ICCV 2021 | 51.2 | 34.5
PyMAF | ICCV 2021 | 57.7 | 40.5
Video Based | VIBE | CVPR 2020 | 65.6 | 41.4
TCMR | CVPR 2021 | 62.3 | 41.1
Pose Based | Pose2Mesh | ECCV 2020 | 64.9 | 47.0
Baseline | - | 68.3 | 50.0
GTRS | - | 64.3 | 45.4
| Our | - | 65.9 | 47.13
TABLE I: Our extensive experiments showed our pipeline score comparable
results to various contributions

##  III Experimental Evaluation

###  III-A Datasets

The Human3.6M dataset [15] is a widely adopted, video-based, and large-scale
indoor dataset that was captured using an accurate marker-based motion-
capturing system. The dataset comprises 11 professional actors performing 17
predefined actions. In accordance with previous studies, such as [4] and [14],
we selected 5 subjects for training and utilized 2 subjects’ samples for
evaluation.

The 3DPW dataset [16] is an outdoor dataset consisting of 60 video sequences
that total 51,000 frames captured in an uncontrolled environment. The dataset
features ground-truth 3D pose and mesh annotations. In accordance with the
evaluation protocol established in previous studies, such as [4] and [14], we
adopted the test dataset only.

The MuCo-3DHP dataset [17] is a large-scale dataset for 3D human pose
estimation and mesh reconstruction in natural environments. The dataset
includes over 13 million frames of video data recorded by 5 actors performing
7 actions in various outdoor settings. The dataset comprises ground-truth 3D
pose and mesh annotations and has been utilized in numerous research studies
to evaluate the performance of different algorithms for 3D human pose
estimation and mesh reconstruction. Similar to [4] and [14], we utilized this
dataset for mixed training.

The MSCOCO keypoints dataset [18] is a subset of the larger MSCOCO dataset
that was created for the purpose of human pose estimation in images. The
dataset includes over 200,000 images with ground-truth keypoint annotations
for 17 different body joints. The dataset has been widely used in research to
evaluate the performance of different algorithms for human pose estimation and
has served as a benchmark for comparing the performance of various approaches.
In accordance with previous studies, such as [4] and [14], we adopted the
MSCOCO dataset for mixed training.

###  III-B Metrics for Evaluation

The mean per-joint position error (MPJPE) is a commonly used metric in the
field of 3D human pose estimation to assess the performance of algorithms. It
quantifies the average Euclidean distance between the predicted and ground-
truth 3D joint positions in a dataset, after aligning them through a rigid
transformation that minimizes the distance between the two sets of points. The
MPJPE is expressed in millimeters, with lower values indicating higher
accuracy in pose estimation.

The Procrustes aligned mean per-joint position error (PA-MPJPE) is a variant
of the MPJPE metric that also evaluates the accuracy of 3D human pose
estimation algorithms. Similar to MPJPE, it measures the average Euclidean
distance between the predicted and ground-truth joint positions. However, PA-
MPJPE incorporates an additional alignment step using the Procrustes analysis
method, which aligns the predicted and ground-truth positions by applying a
non-rigid transformation that minimizes the overall distance between the two
sets of points. The PA-MPJPE is also reported in millimeters, with lower
values indicating more accurate pose estimation.

The mean per-vertex error (MPVE) is a metric for evaluating the performance of
algorithms for 3D human mesh reconstruction. It quantifies the average
Euclidean distance between the predicted and ground-truth 3D mesh vertices in
a dataset, after aligning them through a rigid transformation that minimizes
the distance between the two sets of points. The MPVE is expressed in
millimeters, with lower values indicating higher accuracy in mesh
reconstruction.

##  IV Conclusion

In this work, we present a pose-based framework for reconstructing human mesh
parameters from 2D poses. This framework utilizes a 3D-lifter that leverages
structured and implicit joint correlations through the employment of
paralleled graph transformer blocks. As a result, the model is able to
efficiently integrate the extracted pose features with the mesh template to
produce human mesh parameters. Although the proposed framework demonstrates
competitive performance, it may not be capable of reconstructing all human
body shapes solely from 2D pose inputs. While image-based methods may achieve
more precise reconstructions, pose-based approaches maintain their
significance owing to their versatility and lightweight design. Additionally,
the modular design of the proposed framework allows for training to be
conducted in individual or end-to-end modules.

## References

  * [1] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  ACM transactions on graphics (TOG), 34(6):1–16, 2015.
  * [2] Javier Romero, Dimitrios Tzionas, and Michael J Black.  Embodied hands: Modeling and capturing hands and bodies together.  arXiv preprint arXiv:2201.02610, 2022.
  * [3] Kathleen M Robinette, Sherri Blackwell, Hein Daanen, Mark Boehmer, and Scott Fleming.  Civilian american and european surface anthropometry resource (caesar), final report. volume 1. summary.  Technical report, Sytronics Inc Dayton Oh, 2002.
  * [4] Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee.  Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose.  In European Conference on Computer Vision, pages 769–787. Springer, 2020.
  * [5] Ce Zheng, Matias Mendieta, Pu Wang, Aidong Lu, and Chen Chen.  A lightweight graph transformer network for human mesh reconstruction from 2d human pose.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 5496–5507, 2022.
  * [6] Hui En Pang, Zhongang Cai, Lei Yang, Tianwei Zhang, and Ziwei Liu.  Benchmarking and analyzing 3d human pose and shape estimation beyond algorithms.  In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  * [7] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7122–7131, 2018.
  * [8] Muhammed Kocabas, Nikos Athanasiou, and Michael J Black.  Vibe: Video inference for human body pose and shape estimation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5253–5263, 2020.
  * [9] Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black.  Amass: Archive of motion capture as surface shapes.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 5442–5451, 2019.
  * [10] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5693–5703, 2019.
  * [11] Zhiming Zou and Wei Tang.  Modulated graph convolutional network for 3d human pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11477–11487, 2021.
  * [12] Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N Metaxas.  Semantic graph convolutional networks for 3d human pose regression.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3425–3435, 2019.
  * [13] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [14] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis.  Learning to reconstruct 3d human pose and shape via model-fitting in the loop.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2252–2261, 2019.
  * [15] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  IEEE transactions on pattern analysis and machine intelligence, 36(7):1325–1339, 2013.
  * [16] Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll.  Recovering accurate 3d human pose in the wild using imus and a moving camera.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 601–617, 2018.
  * [17] Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt.  Monocular 3d human pose estimation in the wild using improved cnn supervision.  In 2017 international conference on 3D vision (3DV), pages 506–516. IEEE, 2017.
  * [18] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.