2024 Full Paper – MIDL 2024 – nnn Julie Mordacq1,2,3<EMAIL_ADDRESS>
1 Inria Saclay, France
2 Ecole Polytechnique, France
3 LIX, CNRS, IP Paris, France
Leo Milecki1,4<EMAIL_ADDRESS>
4 MICS, CentraleSupelec, Paris-Saclay University, France
Maria Vakalopoulou1,4<EMAIL_ADDRESS>
Steve OudotEqual supervision1,2<EMAIL_ADDRESS>
Vicky Kalogeiton2,3<EMAIL_ADDRESS>

# ADAPT: Multimodal Learning for Detecting Physiological Changes under Missing
Modalities

###### Abstract

Multimodality has recently gained attention in the medical domain, where
imaging or video modalities may be integrated with biomedical signals or
health records. Yet, two challenges remain: balancing the contributions of
modalities, especially in cases with a limited amount of data available, and
tackling missing modalities. To address both issues, in this paper, we
introduce the AnchoreD multimodAl Physiological Transformer (ADAPT), a
multimodal, scalable framework with two key components: (i) aligning all
modalities in the space of the strongest, richest modality (called _anchor_)
to learn a joint embedding space, and (ii) a Masked Multimodal Transformer,
leveraging both inter- and intra-modality correlations while handling missing
modalities. We focus on detecting physiological changes in two real-life
scenarios: stress in individuals induced by specific triggers and fighter
pilots’ loss of consciousness induced by $g$-forces. We validate the
generalizability of ADAPT through extensive experiments on two datasets for
these tasks, where we set the new state of the art while demonstrating its
robustness across various modality scenarios and its high potential for real-
life applications. Our code is available at https://github.com/jumdc/ADAPT.

###### keywords:

Multimodality, Missing Modalities, Contrastive Learning, Biomedical signals

††editors: Accepted for publication at MIDL 2024

##  1 Introduction

Monitoring physiological changes to external stimuli is crucial for assessing
individuals’ well-being, particularly in contexts with medical and safety
implications. Examples include stress, a response to emotional, mental, and
physical challenges [Schneiderman et al. (2005)], and a triggering or
aggravating factor for various pathological conditions [Dimsdale (2008)].
High-performance environments, such as exposure to $g$-forces in aircraft, can
lead to alterations in consciousness [Morrissette and McGowan (2000)]. At the
same time, drowsiness during driving poses a critical physiological response
with safety implications, contributing to road accidents and fatalities
[Stewart (2023)]. Various sensors report physiological changes that may be
detected visually (videos), acoustically (audio), or from biomedical signals
(e.g., electrocardiograms). However, specific modalities may be missing during
training and testing. Therefore, developing methods capable of handling
missing modalities during both stages while balancing modalities’
contributions is crucial to ensure robustness, notably when modalities with
strong unimodal performances are severely missing.

Various methods address the challenge of missing modalities, each with notable
limitations, including (1) bias towards the most available modalities leading
to sub-optimal performance [Konwer et al. (2023)], (2) dependence on complete
modalities during training [Mallya and Hamarneh (2022), Chen et al. (2021a)] ,
(3) limited generalizability to more than two modalities [Ma et al. (2021), Ma
et al. (2022)], and (4) utilization of a shared encoder tailored for
modalities with inputs of the same dimensions which complicates extension to
heterogeneous modalities like imaging and biomedical signals [Konwer et al.
(2023)].

To address the above issues, we introduce the AnchoreD multimodAl
Physiological Transformer (ADAPT) that is designed to operate effectively
under missing modalities both during training and inference enabling robust
real-life applicability. ADAPT consists of two key components. First, our goal
is to embed all modalities in the same feature space. Instead of optimizing
one loss per modality pair, which would result in quadratic growth of training
time, we align each modality to one frozen modality, called _anchor_. It
allows learning a joint embedding space with linear scalability and balancing
each modality’s contribution. We call this step the ‘anchoring’. Second, it
comprises a Masked Multimodal Transformer that leverages inter- and intra-
modality correlations to concatenate features from different modalities into a
unified representation. Additionally, we leverage masked attention from the
transformers [Vaswani et al. (2017)] to ensure flexibility in handling missing
modalities similarly to Ma et al. (2022); Milecki et al. (2022). When a
modality is unavailable, its corresponding feature representation is masked.
The transformer is trained using two objectives: self-supervised learning and
the objective of the downstream task.

ADAPT is applied to the challenging task of detecting physiological changes
using multimodal medical data with missing modalities during training and
inference. Specifically, we focus on detecting alterations in pilots’
consciousness induced by $g$-forces in fighter jets and stress triggered in
individuals by specific stimuli Chaptoukaev et al. (2023). We show that ADAPT
outperforms the previous state of the art on both tasks and datasets while
handling missing modalities. Extensive experiments demonstrate its robustness
against missing modalities across various scenarios, highlighting its
effectiveness for real-life applications.

Our contributions are: (i) ADAPT, a modular framework that aligns multimodal
representations to a common rich feature space; (ii) a modality-fusion
strategy to handle missing modalities both at training and inference time;
(iii) we set the new state of the art on two tasks and datasets and provide
extensive evaluations highlighting ADAPT’s superiority.

##  2 Related Work

Handling missing modalities. Missing modalities pose a persistent challenge in
Multimodal Learning, particularly in medical imaging, due to privacy concerns
or impractical data acquisition Liang et al. (2021); Azad et al. (2022).
Various strategies have been explored to address this challenge. Knowledge
Distillation Hu et al. (2020); Mallya and Hamarneh (2022); Wang et al. (2023b)
involves learning from a teacher network trained on complete modality data.
Generative modeling aims to impute missing inputs by generating synthetic data
Yoon et al. (2018); Sharma and Hamarneh (2019). Both approaches rely on
complete modality at training, which can be insufficient for robust training.
Another line of work is _common space modeling_ , which learns a shared latent
space from partially available modalities Ma et al. (2021); Konwer et al.
(2023); Wang et al. (2023a). SMIL Ma et al. (2021) perturbs the latent feature
space to approximate the embedding of missing modalities but is limited to bi-
modal datasets, limiting its generalizability. ShaSpec Wang et al. (2023a)
addresses more than two modalities by learning shared and specific features,
but its use of a shared encoder complicates generalization to heterogeneous
modalities of different dimensions. Additionally, shared latent space modeling
may introduce biases toward the most available modalities Konwer et al.
(2023). Simultaneously, cross-modal contrastive learning has shown impressive
results Zhang et al. (2022); Milecki et al. (2023); Li et al. (2023) by
aligning multimodal data in a joint embedding space. Recently, ImageBind
Girdhar et al. (2023) aligned six modalities by relying on image-paired data
and emphasized that aligning all pair combinations is unnecessary to bind more
than two modalities together. Our proposed ADAPT advances this by training
unimodal encoders solely with supervision from one modality, aligning them in
a joint embedding space. It ensures that every modality contributes to the
final representation, even if it is severely missing during training.

#### Multimodal transformer.

Transformers Vaswani et al. (2017) have become the de facto approach for
multimodal tasks Recasens et al. (2023); Srivastava and Sharma (2024). They
rely on the attention mechanism to model long-range dependencies with the
flexibility to account for incomplete samples. Milecki et al. (2022); Ma et
al. (2022) efficiently handle missing data in sequences and bimodal datasets
through masked attention. ADAPT extends this to more than two modalities by
leveraging attention to fuse them and exploring their inter- and intra-modal
correlations while masking missing ones. We also perform a systematic study of
missing modalities during training and testing, showing the versatility and
potential of ADAPT for real-life scenarios.

##  3 Method

Figure 1:  Overview of ADAPT. In each minibatch, ADAPT takes up to $M$
modalities, including video, audio, and biosignals, as input to produce a
modality-agnostic representation for downstream tasks. It is trained in two
steps. (i) Anchoring. We align the representations of all modalities via
contrastive learning to the one of an _anchor_ modality, i.e., the strongest
and richest modality; here the video. (ii) Fusion. The encoders’ features are
concatenated and fed into the Masked Multimodal Transformer. When a modality
is unavailable, the transformer masks its corresponding feature
representations. The final representation (i.e., [CLS] token output) is used
for downstream tasks.

This study addresses the detection of physiological changes using multimodal
data, including video, audio, and biomedical signals. Real-world scenarios
often involve missing modalities, motivating our goal to develop a modality-
agnostic representation with broad applicability and to propose ADAPT –
AnchoreD multimodAl Physiological Transformer. An overview of ADAPT is
presented in Figure 1. Notations. Let
$\mathcal{D}=\\{(x_{m}^{i})_{m=1}^{M},y^{i}\\}_{i=1}^{N}$ denote our training
dataset, with $M$ modalities and $N$ labeled observations and
$x^{i}=(x_{m}^{i})_{m=1}^{M}$ the $i$-th observation (i.e., a family of $m$
modality values) with $y^{i}\in\mathcal{Y}=\\{0,..,J\\}$ its corresponding
label (i.e., a physiological state). Given this input, we seek to train a
neural network $\mathcal{F}$, that associates to any observation, with any
missing modality, a target label $y\in\mathcal{Y}$.

###  3.1 Anchoring modality-specific encoders

We train modality-specific encoders with a contrastive learning objective to
align their representations to the one of the _anchor_. In this work, anchor
is the video, as it can capture visually distinguishable physiological
changes; however, any modality can be the anchor.

#### Modality-specific encoders.

Each modality is encoded using a dedicated encoder. For video, we use the pre-
trained Hiera Ryali et al. (2023) encoder. For audio, each sample is encoded
into a mel-spectrogram (a 2D acoustic time-frequency representation of sound),
fed to BYOL-A Niizumi et al. (2021) to obtain a 1-d feature. Biomedical
signals are processed using 1D CNNs Wang et al. (2023c); Ismail Fawaz et al.
(2019). We add a modality-specific linear projection head to each encoder to
obtain a fixed size $d$ dimensional embedding. ADAPT can be extended to other
modalities by adding their respective encoders.

#### Anchoring.

We consider a pair of modalities with aligned observations
$(\mathcal{A},\mathcal{M}_{m})$, where $\mathcal{A}$ represents the anchor
(video) and $\mathcal{M}_{m}$ another modality. The anchor video $x^{i}_{a}$
and its corresponding observation $x^{i}_{m}$ are encoded using
$z_{a}^{i}{=}E_{a}(x^{i}_{a})$ and $z^{i}_{m}{=}E_{m}(x^{i}_{m})$,
respectively, where $E_{a}$ is a pre-trained and frozen video encoder and
$E_{m}$ a DNN. Projection heads map the embeddings to
$f_{a}^{i},f_{m}^{i}{\in}\mathbb{R}^{d}$. The loss is computed on $f_{a}^{i}$
and $f_{m}^{i}$ Girdhar et al. (2023):

$\mathcal{L}_{\mathcal{A},\mathcal{M}_{m}}=-\sum_{i=1}^{B}\log\frac{\exp(\cos(f^{i}_{a},f^{i}_{m})/\tau)}{\sum_{k=1}^{B}\exp(\cos(f^{i}_{a},f^{k}_{m})/\tau)}\quad,$
(1)

where $\tau$ is a temperature parameter $\tau\in\mathbb{R}^{+}$, $\cos(.,.)$
the cosine similarity, and $B$ the batch size. In practice, we use a symmetric
loss:
$\mathcal{L}_{\mathcal{A},\mathcal{M}_{m}}+\mathcal{L}_{\mathcal{M}_{m},\mathcal{A}}$.
To alleviate the modality gap Liang et al. (2022), we add Gaussian noise to
the modality $m$ representation Gu et al. (2023). We use a cosine schedule for
the temperature parameter Kukleva et al. (2023). Given $M$ modalities, we
define the anchoring loss as
$\mathcal{L}_{\text{anchoring}}=\sum_{m=1,\mathcal{M}_{m}\neq\mathcal{A}}^{M}(\mathcal{L}_{\mathcal{A},\mathcal{M}_{m}}+\mathcal{L}_{\mathcal{M}_{m},\mathcal{A}})$.

###  3.2 Masked Multimodal Transformer

To effectively build modality-agnostic representations, we use the transformer
Vaswani et al. (2017) with $N_{L}$ attention blocks. For each sample, we stack
the modality-specific representations, $f_{m}^{i}\in\mathbb{R}^{d},\forall
m\in[1,M]$, into a single matrix and prepend a special token [CLS], yielding a
matrix $F\in\mathbb{R}^{(M+1)\times d}$. Similarly to Liu et al. (2022);
Nagrani et al. (2021), the query, key and value are derived from $F$ via:
$Q=W^{Q}F$, $K=W^{K}F$ and $V=W^{V}F$ where $Q,K\in\mathbb{R}^{(M+1)\times
d_{k}}$ and $V\in\mathbb{R}^{(M+1)\times d_{v}}$. Our modelization of inter-
modal interactions differs from the usual cross-attention Chen et al. (2021b);
Jaegle et al. (2021), which asymmetrically combines two separate embedding
sequences of the same dimension. Using stacked features $F$ allows
generalization to any number of modalities, with linear scalability in the
number of modalities instead of quadratic.

#### Handling missing modalities.

Inspired by Milecki et al. (2022) for missing follow-up patient examinations,
we apply our strategy to deal with missing modalities to the scaled dot-
product, core of each multi-head self-attention sub-layer. We consider one
sub-layer with one head ($h=1$) for simplicity. We use a masking binary matrix
$Z\in\mathbb{R}^{(M+1)\times(M+1)}$ that specifies which modalities are
missing: $z_{ij}=1$ if $i$ and $j$ are available, else $z_{ij}=0$. The output
$O\in\mathbb{R}^{(M+1)\times d_{v}}$ of the attention mechanism is, for
$O_{i}$ each line of $O$:

$O_{i}=\sum_{j}z_{ij}\frac{\exp(Q_{i}^{T}K_{j}/\sqrt{d_{k}})}{\sum_{\\{j^{\prime},z_{ij^{\prime}}=1\\}}\exp(Q_{i}^{T}K_{j^{\prime}}/\sqrt{d_{k}})}V_{j}\quad.$
(2)

When $h>1$, queries, keys, and values are linearly projected $h$ times with
different, learned linear projections, concatenated, and once again projected
after the scaled-dot product.

#### Modality dropout.

We train the Masked Multimodal Transformer with a multi-view contrastive
objective Chen et al. (2020). Drawing inspiration from Shi et al. (2022), we
mitigate the model’s over-reliance on a single modality while enhancing its
robustness in the absence of modalities through an augmentation technique
called modality dropout. We leverage the masking scheme at the attention level
to randomly mask input modalities. Given a batch $\mathcal{Z}$, we create two
simultaneous view $\mathcal{Z}^{\prime}$ and $\mathcal{Z}^{\prime\prime}$. For
each observation within $\mathcal{Z^{\prime}}$, we hide up to $M-1$ modalities
following a uniform probability, $M$ the number of modalities. Additionally,
motivated by Han et al. (2020), who recently showed the effect of additive
noise on electrocardiograms, we add $\epsilon\sim\mathcal{N}(0,\sigma)$ on the
biomedical signals to each view independently. We chose $\sigma$ based on the
amplitude of the signal. We use the infoNCE loss to enforce the similarity
between the two views Chen et al. (2020) on the final representation output
given by the [CLS] token. Since the two representations are already mapped to
the same dimension, following Jing et al. (2022), we directly optimize the
representations to enforce scalability and mitigate dimension collapse:
$\mathcal{L}_{\text{fusion}}=-\sum_{i=1}^{B}\log\frac{\exp(\cos(\text{CLS}^{i},\text{CLS}^{i^{\prime}})/\tau)}{\sum_{k=1}^{B}\exp(\cos(\text{CLS}^{i},\text{CLS}^{k^{\prime}})/\tau)}$.

##  4 Experiments & Results

###  4.1 Experiment Setting

Datasets. StressID Chaptoukaev et al. (2023) for stress identification
contains physiological responses via electrocardiogram (ECG), electrodermal
activity, respiration, audio, and videos. We use the training, val, and test
splits provided by Chaptoukaev et al. (2023).
LOC. We present the Loss Of Consciousness dataset, collected during
aeromedical training of flight personnel by the French Ministry of Armed
Forces (Appendix A). It includes videos and biomedical sensor data: ECG,
quadriceps electromyograms, acoustic breathing, pedal pressure, and self-
reported visual field. It comprises 1666 launches with 416 subjects, split
into train, val, and test sets in a 6:2:2 ratio based on patient ID. We employ
5-fold cross-validation and report the average. Each launch is labeled for
consciousness alteration. The dataset exhibits a high imbalance ratio of
{1:50}. In real life, videos are impractical due to pilots’ equipment (helmets
& $O_{2}$ masks), despite being the primary modality used by doctors to
monitor launches during aeromedical training (see Appendix A).

Missing modalities. StressID has 18% and 46% of missing video and audio
recordings, respectively. For LOC, videos are absent in 90% of observations.
We denote the entire training and testing sets as $X_{\text{train}}$,
$X_{\text{test}}$ (considering samples with and without missing modalities);
and $X_{\text{train}}^{*}$, $X_{\text{test}}^{*}$ for the train and test sets
where all modalities are available.

#### Metrics.

We use the balanced accuracy (ACC) and weighted F1 score (F1). For LOC, to
ensure robustness to class imbalance Huang et al. (2016); Luque et al. (2019),
we also report the true positive rate (TPR) as it ensures not missing out on
pilots fainting; and the true negative rate (TNR) for completeness. We report
metrics in the format mean(std) in %.

#### Implementation details.

The Anchoring and Masked Multimodal Transformer are trained on
$X_{\text{train}}^{*}$ and $X_{\text{train}}$, respectively. A linear
classifier is trained using the [CLS] token for the final task. We train for
70 epochs using AdamW optimizer, a starting learning rate of $1e^{-4}$,
followed by a cosine schedule and a linear warm-up of 4 epochs. Given their
size difference, we set the batch size to 256 for LOC and 128 for StressID. To
tackle LOC class imbalance, we use the Balanced Cross Entropy loss Huang et
al. (2016) (more in Appendix B.1).

###  4.2 Results

Comparison to the state of the art (Table 1) in the presence of all
modalities. We compare ADAPT against unimodal baselines for video, audio, and
biomedical signals concatenated (rows 1, 2 & 3), ‘feature fusion’ and
‘decision-level fusion’ (rows 4 & 5)Chaptoukaev et al. (2023), ShaSpec+ Wang
et al. (2023a) (row 6) (more in Appendices B.2 and C).

For StressID, we observe that ADAPT outperforms all methods from Chaptoukaev
et al. (2023) by a notable margin; for instance, it outperforms ‘decision-
level fusion’ by 4% in ACC and 6% in F1. Additionally, it remains highly
competitive with ShaSpec+.

For LOC, using only video (row 1) results in the best performance; however,
video is unavailable in real-life scenarios. Instead, ADAPT handles missing
modalities by leveraging representations from all modalities during training.
Surprisingly, ‘feature fusion’, ‘decision-level fusion’, and ShaSpec+ lead to
unbalanced metrics, i.e., they result in a high TNR while significantly
sacrificing TPR (respectively 29.5%, 20.4% and 7.3%), showing they predict
most samples as negative. This reveals their unsuitability for real-life cases
with highly imbalanced classes where both TPR and TNR matter. Note that in our
target scenarios, TPR is more important, as it is critical to detect pilots
losing consciousness. By contrast, ADAPT results in a TPR of 69.5% (+40% vs.
fusion methods) while maintaining a balanced TNR of 65.3%. This is further
shown in Figure 2, where ADAPT (blue crosses) strikes the best balance between
TPR and TNR vs. other methods (red crosses). This testifies to ADAPT not being
misled by the high class imbalance.

|  LOC |  | StressID
---|---|---|---
| ACC | F1 | TNR | TPR |  | ACC | F1 |  |
Video | 87.1(1.2) | 98.0 (0.2) | 98.0(0.4) | 75.6(2.5) |  |  62(4)* |  67(3)* |  |
Biomedical signals | 72.0(2.0) | 50.0(2.3) | 94.0(2.0) | 42.1(1.3) |  |  58(4)* |  66(5)* |  |
Audio | 57.6(1.5) | 96.0(0.2) | 95.5(4) | 19.7(2.9) |  |  62(4)* |  67(4)* |  |
Feature Fusion Chaptoukaev et al. (2023) | 79.3(2.5) | 63.9(9.6) | 97.0(0.9) | 29.5(20) |  |  61(3)* |  66(4)* |  |
Decision-level fusion Chaptoukaev et al. (2023) | 60.2(2.2) | 99.0(0.0) | 100.0(0.0) | 20.5(4.5) |  |  65(5)* |  72(5)* |  |
ShaSpec+ Wang et al. (2023a) | 53.4(5.1) | 97.3(1.0) | 99.4(1.0) | 7.3(10.4) |  | 70.2(3.7) | 75.7(5.3) |  |
ADAPT | 67.4(1.2) | 76.9(2.5) | 65.3(1.6) | 69.5(2.0) |  | 69.5(3.7) | 75.9(4.3) |  |

Table 1: Comparison of ADAPT to SOTA on $X_{\text{test}}^{*}$. Gray-out
denotes the video modality, which is impossible to gather in real-life. Bold,
underlined indicate the top 1, 2 performing, respectively.
*Results from Chaptoukaev et al. (2023).

#### Robustness to missing modalities

(Table 4). We first report baseline results (row 1) on the default test set
$X_{\text{test}}$, i.e., no modality removed in StressID and 90% of videos
missing for LOC. Then, we completely remove one or two modalities from
$X_{\text{test}}$ and compare the results ($\Delta$) to the ones obtained on
$X_{\text{test}}$.

Figure 2: TPR vs TNR for LOC. †Methods from Chaptoukaev et al. (2023)

For LOC, ADAPT shows robustness in all scenarios, with a $|\Delta|{<}8\%$ and
average $|\Delta|{=}2.6$ compared to the baseline. This is further shown in
Figure 2, where the balance between TNR and TPR (blue circles) remains
consistent across all scenarios. Interestingly, for no-video, even though
video-only provides strong unimodal performance (Table 8), ADAPT maintains
high performances, indicating its capability of aligning representations in
the video (anchor) space. Furthermore, for the real-life scenario where we
remove both video and visual field (row 2), the results remain competitive
with an average $|\Delta|{=}2.72\%$, even though these modalities individually
perform the best (Table 8). Additionally, no-audio demonstrates consistent
results, keeping the TNR and TPR balanced (see Appendix C).

For StressID, we remove audio and/or video, the most cumbersome modalities to
acquire and examine the no-audio, no-video and real-life (i.e., no audio, no
video) scenarios. The variation remains consistent for both no-audio and no-
video: $|\Delta|{<}8.3\%$. However, it is more consequent for real-life, with
a significant drop in TPR for an equivalent TNR, as expected as we remove the
richest modalities.

Overall, even by removing modalities, ADAPT successfully detects stress or
loss of consciousness with more than 60% ACC and more than 50% TPR,
highlighting its ability to handle missing modalities, in contrast to all
other methods unable to address this.

|  LOC |  | StressID
---|---|---|---
| ACC | $+\Delta$ |  | TNR | $+\Delta$ |  | TPR | $+\Delta$ |  |  | ACC | $+\Delta$ |  | TNR | $+\Delta$ |  | TPR | $+\Delta$ |
| 66.2(3.3) |  |  | 64.4(5.9) |  |  | 68.0(4.2) |  |  |  | 69.5(2.9) |  |  | 61.9(6.9) |  |  | 77.1(6.3) |  |
real-life | 62.0(3.2) | 4.2 |  | 60.8(1.0) | 3.6 |  | 68.5(4.3) | -0.5 |  |  | 60.0(4.8) | 9.5 |  | 66.9(9.5) | -5.0 |  | 53.1(7.9) | 24 |
no video | 67.1(2.2) | -1.1 |  | 66.2(3.9) | -1.8 |  | 68.0(2.0) | 0.0 |  |  | 61.2(4.6) | 8.3 |  | 53.7(10.1) | 8.5 |  | 68.8(7.1) | 8.3 |
no audio | 69.5(5.4) | -3.3 |  | 63.0(12.0) | 1.4 |  | 76.1(4.9) | -8.1 |  |  | 68.3(2.9) | 1.2 |  | 65.8(6.8) | -4.0 |  | 70.7(7.4) | 6.4 |

Table 2: Evaluation of ADAPT on three modality scenarios on $X_{\text{test}}$.
For each scenario, we remove one or two modalities from the test samples. We
report mean and standard deviation for ACC, TNR, and TPR, and the differences
($\Delta$) compared to tests without removed modality (first row).

Anchor |  |  LOC |  | StressID
---|---|---|---|---
Audio | Video |  | ACC | F1 |  | ACC | F1
✗ | ✗ |  | 54.6(1.0) | 78.4(4.1) |  | 65.8 (1.8) | 65.9 (1.7)
✓ | ✗ |  | 54.0(2.1) | 78.5 (1.4) |  | 63.6(4.8) | 63.3(5.0)
✗ | ✓ |  | 66.2(3.3) | 78.0(4.3) |  | 69.5(2.9) | 69.6(3.1)

Table 3: Ablation study of _anchoring_ on $X_{\text{test}}$. We report the
results with anchoring prior fusion (considering the audio or the video as the
anchor) and without. Bold indicates the top 1 performing.

Features | Anchoring | Fusion | ACC | F1
---|---|---|---|---
‡handcrafted features | ✗ |  †Feature level | 61(3)* | 66(4)*
†Decision level | 65(5)* | 72(5)*
✓ | ADAPT | 51.5(2.3) | 60.2(4.7)
ADAPT | ✓ |  †Feature level | 65.2(7.2) | 71.2(10)
†Decision level | 70.7(3.3) | 78.8(2.9)
ADAPT | 69.5(3.7) | 75.9(4.3)

Table 4: Study of ADAPT components with SOTA for StressID on
$X_{\text{test}}^{*}$. ‡Handcrafted features, †Methods, *Results from
Chaptoukaev et al. (2023). Bold and underlined indicates the top 1, 2.

#### Ablations.

1\. Impact of the anchoring before fusion and choice of anchor (Table 4).
Anchoring with video shows significant benefits, particularly in LOC with an
11.6% increase in ACC alongside consistent F1 scores. Similarly, for StressID,
anchoring improves both ACC and F1 by 3.7%. Any anchor may be considered; we
explore using the audio (row 3), but it leads to suboptimal performances.
Overall, the anchor selection is driven by its robust unimodal performance,
which remains effective despite high missing modalities.
2\. Impact of feature configurations and fusion methods (Table 4). Compared to
the ‘feature fusion’ and ‘decision-level fusion’ (rows 1,2, Chaptoukaev et al.
(2023)), our features and fusion method (last row) significantly increase ACC
and F1 by 5.7% and 6.8%, respectively, further highlighting the advantages of
anchoring. We also investigate applying anchoring to features from Chaptoukaev
et al. (2023) (row 3) by solely training the projection head, as opposed to
both the encoder and projection head. Although this yields decent results, the
inability to learn features optimally is a drawback. Finally, the ADAPT entire
pipeline (row 6) delivers competitive results while accommodating missing
modalities.

##  5 Conclusions

In this paper, we propose ADAPT, a modality-agnostic representation framework
designed to operate effectively under missing modalities during both training
and testing. Our framework has been challenged on two different tasks
targeting the detection of physiological changes, outperforming the current
state of the art while showcasing its superiority for handling missing
modalities. Extensive ablations indicate the robustness of our method on
different scenarios and strategies. Future work includes applications to other
medical tasks.

## Acknowledgments

This work was partially supported by Inria Action Exploratoire PreMediT
(Precision Medicine using Topology) and the ANR-22-CE39-0016 APATE.
Additionally, it was partly performed using HPC resources from GENCI-IDRIS
(Grant 2023-AD011014747).

## References

  * Azad et al. (2022) Reza Azad, Nika Khosravi, Mohammad Dehghanmanshadi, Julien Cohen-Adad, and Dorit Merhof.  Medical image segmentation on mri images with missing modalities: A review.  _arXiv preprint arXiv:2203.06217_ , 2022.
  * Chaptoukaev et al. (2023) Hava Chaptoukaev, Valeriya Strizhkova, Michele Panariello, Bianca D’alpaos, Aglind Reka, Valeria Manera, Susanne Thümmler, Esma Ismailova, Nicholas Evans, Francois F Bremond, et al.  Stressid: a multimodal dataset for stress identification.  In _Advances in Neural Information Processing Systems_ , 2023.
  * Chen et al. (2021a) Cheng Chen, Qi Dou, Yueming Jin, Quande Liu, and Pheng Ann Heng.  Learning with privileged multimodal knowledge for unimodal segmentation.  _IEEE transactions on medical imaging_ , 2021a.
  * Chen et al. (2021b) Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda.  Crossvit: Cross-attention multi-scale vision transformer for image classification.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021b.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International Conference on Machine Learning_ , 2020.
  * Dimsdale (2008) Joel E Dimsdale.  Psychological stress and cardiovascular disease.  _Journal of the American College of Cardiology_ , 2008.
  * Girdhar et al. (2023) Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  Imagebind: One embedding space to bind them all.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Gu et al. (2023) Sophia Gu, Christopher Clark, and Aniruddha Kembhavi.  I can’t believe there’s no images! learning visual tasks using only language supervision.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023.
  * Han et al. (2020) Xintian Han, Yuxuan Hu, Luca Foschini, Larry Chinitz, Lior Jankelson, and Rajesh Ranganath.  Deep learning models for electrocardiograms are susceptible to adversarial attack.  _Nature medicine_ , 2020.
  * Hu et al. (2020) Minhao Hu, Matthis Maillard, Ya Zhang, Tommaso Ciceri, Giammarco La Barbera, Isabelle Bloch, and Pietro Gori.  Knowledge distillation from multi-modal to mono-modal segmentation networks.  In _Internation Conference on Medical Image Computing and Computer Assisted Intervention_ , 2020.
  * Huang et al. (2016) Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang.  Learning deep representation for imbalanced classification.  In _Conference on Computer Vision and Pattern Recognition_ , 2016.
  * Ismail Fawaz et al. (2019) Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller.  Deep learning for time series classification: a review.  _Data mining and knowledge discovery_ , 2019.
  * Ismail Fawaz et al. (2020) Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey I Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petitjean.  Inceptiontime: Finding alexnet for time series classification.  _Data Mining and Knowledge Discovery_ , 2020.
  * Jaegle et al. (2021) Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira.  Perceiver: General perception with iterative attention.  In _International Conference on Machine Learning_ , 2021.
  * Jing et al. (2022) Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian.  Understanding dimensional collapse in contrastive self-supervised learning.  In _International Conference on Learning Representations_ , 2022.
  * Konwer et al. (2023) Aishik Konwer, Xiaoling Hu, Joseph Bae, Xuan Xu, Chao Chen, and Prateek Prasanna.  Enhancing modality-agnostic representations via meta-learning for brain tumor segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 21415–21425, 2023.
  * Kukleva et al. (2023) Anna Kukleva, Moritz Böhle, Bernt Schiele, Hilde Kuehne, and Christian Rupprecht.  Temperature schedules for self-supervised contrastive methods on long-tail data.  In _International Conference on Learning Representations_ , 2023.
  * Li et al. (2023) Jun Li, Che Liu, Sibo Cheng, Rossella Arcucci, and Shenda Hong.  Frozen language model helps ECG zero-shot learning.  In _Medical Imaging with Deep Learning_ , 2023.
  * Liang et al. (2021) Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al.  Multibench: Multiscale benchmarks for multimodal representation learning.  _arXiv preprint arXiv:2107.07502_ , 2021.
  * Liang et al. (2022) Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou.  Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Liu et al. (2022) Zhisong Liu, Robin Courant, and Vicky Kalogeiton.  Funnynet: Audiovisual learning of funny moments in videos.  In _Proceedings of the Asian Conference on Computer Vision_ , pages 3308–3325, 2022.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2019.
  * Luque et al. (2019) Amalia Luque, Alejandro Carrasco, Alejandro Martín, and Ana de Las Heras.  The impact of class imbalance in classification performance metrics based on the binary confusion matrix.  _Pattern Recognition_ , 2019.
  * Ma et al. (2021) Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng.  Smil: Multimodal learning with severely missing modality.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2021.
  * Ma et al. (2022) Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng.  Are multimodal transformers robust to missing modality?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Mallya and Hamarneh (2022) Mayur Mallya and Ghassan Hamarneh.  Deep multimodal guidance for medical image classification.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2022.
  * Milecki et al. (2022) Leo Milecki, Vicky Kalogeiton, Sylvain Bodard, Dany Anglicheau, Jean-Michel Correas, Marc-Olivier Timsit, and Maria Vakalopoulou.  Contrastive masked transformers for forecasting renal transplant function.  In _International Conference on Medical Image Computing and Computer Assisted Intervention_ , 2022.
  * Milecki et al. (2023) Leo Milecki, Vicky Kalogeiton, Sylvain Bodard, Dany Anglicheau, Jean-Michel Correas, Marc-Olivier Timsit, and Maria Vakalopoulou.  Medimp: 3d medical images with clinical prompts from limited tabular data for renal transplantation.  In _Medical Imaging with Deep Learning_ , 2023.
  * Morrissette and McGowan (2000) Karen L Morrissette and David G McGowan.  Further support for the concept of a g-loc syndrome: a survey of military high-performance aviators.  _Aviation, space, and environmental medicine_ , 2000.
  * Nagrani et al. (2021) Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun.  Attention bottlenecks for multimodal fusion.  _Advances in Neural Information Processing Systems_ , 34:14200–14213, 2021.
  * Niizumi et al. (2021) Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino.  Byol for audio: Self-supervised learning for general-purpose audio representation.  In _International Joint Conference on Neural Networks_ , 2021.
  * Nunneley and Stribley (1979) S.A. Nunneley and R.F. Stribley.  Heat and acute dehydration effects on acceleration response in man.  _J. Appl. Physiol._ , 1979.
  * Park et al. (2015) Myunghwan Park, Seunghoon Yoo, Hyeongju Seol, Cheonyoung Kim, and Youngseok Hong.  Unpredictability of fighter pilots’ g duration tolerance by anthropometric and physiological characteristics.  _Aerospace Medicine and Human Performance_ , 86(4):397–401, 2015.
  * Paszke et al. (2019) A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Pollock et al. (2021) Ross D Pollock, Peter D Hodkinson, and Thomas G Smith.  Oh g: The x, y and z of human physiological responses to acceleration.  _Experimental Physiology_ , 2021.
  * Recasens et al. (2023) Adrià Recasens, Jason Lin, Joāo Carreira, Drew Jaegle, Luyu Wang, Jean-baptiste Alayrac, Pauline Luc, Antoine Miech, Lucas Smaira, Ross Hemsley, et al.  Zorro: the masked multimodal transformer.  _arXiv preprint arXiv:2301.09595_ , 2023.
  * Ryali et al. (2023) Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer.  Hiera: A hierarchical vision transformer without the bells-and-whistles.  In _International Conference on Machine Learning_ , 2023.
  * Schneiderman et al. (2005) Neil Schneiderman, Gail Ironson, and Scott D Siegel.  Stress and health: psychological, behavioral, and biological determinants.  _Annual Review of Clinical Psychology_ , 2005.
  * Sharma and Hamarneh (2019) Anmol Sharma and Ghassan Hamarneh.  Missing mri pulse sequence synthesis using multi-modal generative adversarial network.  _IEEE Transactions on Medical Imaging_ , 2019.
  * Shi et al. (2022) Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed.  Learning audio-visual speech representation by masked multimodal cluster prediction.  In _International Conference on Learning Representations_ , 2022.
  * Srivastava and Sharma (2024) Siddharth Srivastava and Gaurav Sharma.  Omnivec: Learning robust representations with cross modal sharing.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1236–1248, 2024.
  * Stewart (2023) Timothy Stewart.  Overview of motor vehicle traffic crashes in 2021.  Technical report, National Highway Traffic Safety Administration, 2023.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in Neural Information Processing Systems_ , 2017.
  * Wang et al. (2023a) Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carneiro.  Multi-modal learning with missing modality via shared-specific feature modelling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15878–15887, 2023a.
  * Wang et al. (2023b) Hu Wang, Congbo Ma, Jianpeng Zhang, Yuan Zhang, Jodie Avery, Louise Hull, and Gustavo Carneiro.  Learnable cross-modal knowledge distillation for multi-modal learning with missing modality.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 2023b.
  * Wang et al. (2023c) Yihe Wang, Yu Han, Haishuai Wang, and Xiang Zhang.  Contrast everything: A hierarchical contrastive framework for medical time-series.  In _Advances in Neural Information Processing Systems_ , 2023c.
  * Yoon et al. (2018) Jinsung Yoon, James Jordon, and Mihaela Schaar.  Gain: Missing data imputation using generative adversarial nets.  In _International Conference on Machine Learning_ , 2018.
  * Zhang et al. (2022) Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz.  Contrastive learning of medical visual representations from paired images and text.  In _Machine Learning for Healthcare Conference_ , pages 2–25, 2022.

##  Appendix A LOC dataset

####  $g$-forces and fighter pilots.

Earth’s gravity, commonly referred to as $1g$, induces a constant
acceleration. Fighter pilots, however, encounter much higher accelerations,
reaching up to $9g$ along the z-axis. This force induces a blood shift towards
the lower body, reducing blood perfusion in the brain. Without mitigation,
pilots can experience altered consciousness, encompassing diminished
peripheral vision, central vision loss, Almost Loss Of Consciousness (ALOC),
and G-Induced Loss of Consciousness (GLOC) Morrissette and McGowan (2000),
sometimes with fatal outcomes. Due to the eye’s heightened sensitivity to
hypoxia, initial symptoms are often visual. As retinal blood pressure drops
below Intraocular pressure (usually 10–21 mm Hg), blood flow diminishes,
impacting the retina initially in areas farthest from the optic disc and
central retinal artery before progressing toward central vision. Fighter
pilots train in centrifuges111 Equipment capable of reproducing the intensity
and jolt of the $g$-forces accelerations experienced in high-performance
fighter jet. to apprehend the effects of $+g_{z}$ accelerations and perform
the Anti-$g$ Straining Maneuvers (AGSM). AGSM consist of muscle contractions
in the lower limbs, synchronized with breathing exercices. These maneuvers aim
to elevate arterial blood pressure to sustain blood volume in the brain
Pollock et al. (2021). Each training is monitored by doctors through
physiological data acquired in real time: electrocardiograms (ECG),
electromyograms of the quadriceps (EMG), the acoustic breathing (AUDIO), the
pressure on the pedals (PAL), the visual field (VF)222Self-filled in by
pilots, and through video monitoring. Several factors Pollock et al. (2021);
Park et al. (2015); Nunneley and Stribley (1979), may influence pilots’
tolerance to $g$-forces, thus posing challenges for doctors in detecting
alterations of consciousness, both in centrifuge simulations and in-flight.
Moreover, expanding detection to real scenarios is not trivial. The modalities
differ: the video and visual field data are impractical due to pilots’
equipment (helmets and full-face $O_{2}$ masks) and technical constraints,
respectively.

#### Case study with medical doctors.

Figure 3: Example of a _Loss of consciousness_ launch.

Figure 4: Example of a _No Loss of consciousness_ launch.

Given the rare availability of videos, we conducted a case study with a sample
of doctors in charge of the aeromedical training. The study aimed to assess
the efficacy of relying solely on biomedical signals for identifying loss of
consciousness.

_Protocol._ Each doctor was presented with 20 complete launches accompanied by
biomedical signals and tasked with labeling them as ”Loss of consciousness”
(Example in Figure 4) or ”No loss of consciousness” (Example in Figure 4).

_Results._ Surprisingly, doctors accurately classified 55% of launches with a
variance of 6.53% using only biomedical signals. However, when provided with
videos, doctors achieved 100% accuracy in launch classification. This
underscores the crucial role of video data and emphasizes the necessity, from
a medical standpoint, to incorporate its robust representation.

#### Positive and negative samples.

To detect alteration of consciousness, we train our models for binary
classification (distinguishing between altered and unaltered) on $n$-seconds
windows. Such a window $w_{t}{=}[t{-}n,t]$ is considered positive if there is
some consciousness alteration period $(t_{s},t_{e})$ such that
$t{-}n{\geq}t_{s}$ and $t{\leq}t_{e}$. Otherwise, $w_{t}$ is considered
negative. We fix $n=3.1sec$.

##  Appendix B Implementations details.

###  B.1 Architectural and training details.

|  | LOC | StressID
---|---|---|---
Unimodal encoders | Audio |  BYOL-A Niizumi et al. (2021) |  BYOL-A Niizumi et al. (2021)
Video |  Hiera Ryali et al. (2023) |  Hiera Ryali et al. (2023)
Biomedical Signals |  InceptionTime Ismail Fawaz et al. (2020) |  FCN Ismail Fawaz et al. (2019)
Anchoring | Anchor | Video | Video
$d$ | 128 | 64
$\tau$ |  Temperature schedule Kukleva et al. (2023) |  Temperature schedule Kukleva et al. (2023)
| $\gamma$ | 0.08 | 0.08
| projection head | 2 fully-connected layers | 1 fully-connected layer
Multimodal Transformer | $d_{model}$ | 64 | 64
$n_{heads}$ | 4 | 4
$N_{L}$ | 2 | 1
$d_{ffn}$ | 256 | 256
$d_{v}$ | 64 | 64
| $d_{k}$ | 64 | 64
| activation function | GeLU | GeLU
| $\sigma$ | 0.02 | 0.1
| Normalization | LayerNorm | LayerNorm
Linear Classifier | loss | Balanced Cross Entropy | Cross Entropy

Table 5: Architectural details of ADAPT.

Table 5 lists the architectural details used for each dataset. The seed is
fixed to 1999. In our experiments, for both stages we use the AdamW Loshchilov
and Hutter (2019) optimizer, with a weight decay of 0.05, a starting learning
rate of $1e^{-4}$ following a cosine schedule, and preceded by a linear warm-
up of 4 epochs on 4 NVIDIA Tesla V100 GPU using Pytorch Paszke et al. (2019).
The gradients’ norms are clipped to 1, to ensure stability during training.

_Augmentations used for multi-view contrastive learning._ We used data
augmentation with the sequential application, with each a 0.5 probability of
Gaussian noise and modality dropout.

###  B.2 Comparison to the state of the art.

We compare our approach directly with the methods utilized in Chaptoukaev et
al. (2023), which established the state-of-the-art for StressID:

  1. 1.

‘feature-level fusion’: unimodal features features are combined into a single
high-dimensional feature vector, used as input to a MLP trained with Cross-
Entropy loss for StressID and Cost-Sensitive Cross Entropy Huang et al. (2016)
for LOC to tackle class imbalance and fair comparison with ADAPT.

  2. 2.

‘decision-level fusion’: independent SVMs are trained for each modality using
the unimodal features as input, and integrate the results of the individual
classifiers at the decision level, i.e. the results are combined into a single
decision using ensemble rules. Four decision rules are proposed in Chaptoukaev
et al. (2023): sum rule fusion, average rule fusion, product rule fusion and
maximum rule fusion. The best out of the four decision rules is reported.

Additionally, we implemented ShaSpec Wang et al. (2023a). ShaSpec maximizes
the utilization of all available input modalities during training and
evaluation by learning shared and specific features for better data
representation. However, due to the varying dimensions of our input modalities
(e.g., 3D video, 1D biomedical signals), employing a shared encoder is
nontrivial. Hence, we adopt an adapted version, _ShaSpec+_ , where encoded
inputs are fed to the shared encoder instead of raw inputs. To ensure fair
comparison with ADAPT, we use identical settings, including the same encoders:
Hiera Ryali et al. (2023) for video, Byol-a Niizumi et al. (2021) for audio,
and 1D CNN Wang et al. (2023c) for biomedical signals. Additionally, to
address class imbalance for the LOC dataset we substitute the cross-entropy
loss with cost-sensitive cross-entropy Huang et al. (2016).

##  Appendix C Complementary results

###  C.1 Unimodal performances

Performances of unimodal encoders are provided in Table 8 for LOC and Table 8
for StressID.

|  LOC
---|---
| $X_{\text{test}}^{*}$ | $X_{\text{test}}$
| ACC | F1 | TNR | TPR | ACC | F1 | TNR | TPR
Video | 87.1(1.2) | 98.0 (0.2) | 98.0(0.4) | 75.6(2.5) | - | - | - | -
Audio | 57.6(1.5) | 96.0(0.2) | 95.5(4) | 19.7(2.9) | 55.6 (1.8) | 83.2(3.2) | 71.6(5.1) | 39.6(6.3)
Visual Field | 66.2(3.4) | 92.8(2.5) | 89.5(4.4) | 43.0(9.4) | 72.9(4.0) | 92.1(3.0) | 85.8(5.2) | 60.1(7.2)
Pedals | 69.2(2.5) | 96.6(0.2) | 95.8(0.5) | 42.6(5.4) | 68.5(1.5) | 89.5(0.3) | 81.2(1.0) | 55.7(3.4)
Electrocardiograms | 60.3(3.4) | 96.1(0.8) | 95.0(1.5) | 25.5(6.8) | 55.9(3.0) | 93.9(1.7) | 88.9(3.0) | 23.0(7.8)
Electromyograms | 58.5(12.9) | 26.8(11.7) | 22.8(10.2) | 94.2(9.5) | 54.9(4.2) | 27.5(10.8) | 21.6(30.6) | 88.1(22.5)

Table 6: Performance of unimodal encoders for LOC. Audio, VF, PAL, ECG, EMG
performances are evaluated after the anchoring (i.e after the alignment to the
anchor, the video). We report the results on $X_{\text{test}}$ and
$X_{\text{test}}^{\text{*}}$ in form: mean(std).

|  StressID
---|---
| $X_{\text{test}}^{*}$ | $X_{\text{test}}$
| ACC | F1 | ACC | F1
EDA | 58.0 (2.8) | 65.4 (4) | 64.0(2.2) | 64.1(2.1)
RR | 57.1(4.1) | 58.0(4.8) | 58.4 (3.0) | 58.0(3.4)
ECG | 55.6(3.6) | 39.8(7.3) | 55.5(2.2) | 48.7(4.0)
Audio | 59.9(6.2) | 66.9(9.1) | - | -

Table 7: Performance of unimodal encoders for StressID. Audio, EDA, ECG and RR
performances are evaluated after the anchoring (i.e after the alignment to the
anchor, the video). We report the results on $X_{\text{test}}$ and
$X_{\text{test}}^{\text{*}}$ in form: mean(std).

|  LOC
---|---
| ACC | $+\Delta$ |  | TNR | $+\Delta$ |  | TPR | $+\Delta$ |
| 67.4(1.3) |  |  | 65.3(1.6) |  |  | 69.5 (1.5) |  |
real-life | 61.9(7.2) | 5.5 |  | 70.1(2.1) | -4.8 |  | 61.4(10.4) | 8.1 |
no video | 64.9(8.3) | 2.5 |  | 63.0(15.2) | 2.3 |  | 66.8(6.5) | 2.7 |
no audio | 64.9(8.3) | 2.5 |  | 63.0(15.2) | 2.3 |  | 66.8(6.4) | 2.7 |

Table 8: Evaluation of ADAPT on three modality scenarios on
$X_{\text{test}}^{\text{*}}$ for LOC. For each scenario, we remove one or two
modalities from the test samples. We report mean and standard deviation for
ACC, TNR, and TPR, and calculate the differences ($\Delta$) compared to tests
without removed modality.

###  C.2 ADAPT’s robustness to missing modalities

#### Evaluation of ADAPT on $X_{\text{test}}^{\text{*}}$.

Table 8 assesses ADAPT across three modality scenarios, evaluating its
robustness by removing one or two modalities from $X_{\text{test}}^{\text{*}}$
(i.e., samples where all modalities are available) and comparing the results
to the baseline ($X_{\text{test}}^{\text{*}}$ without any modality removed).
We calculate the differences ($\Delta$) for comparison. Overall,
$|\Delta|<3.2$, further highlighting ADAPT’s robustness with full modality
availability. Importantly, even though video-only performs better individually
by a large margin, ADAPT maintains robust results when it is removed (row 3).


28

# General Mneimneh-type Binomial Sum involving Harmonic Numbers

Ende Pana, and Ce Xub,
a. College of Teacher Education, Quzhou University,
Quzhou 324022, P.R. China
b. School of Mathematics and Statistics, Anhui Normal University,
Wuhu 241002, P.R. China  Email<EMAIL_ADDRESS><EMAIL_ADDRESS>
Abstract. Recently, Mneimneh proved the remarkable identity

$\displaystyle\displaystyle\sum_{k=0}^{n}H_{k}\binom{n}{k}p^{k}(1-p)^{n-k}=\displaystyle\sum_{i=1}^{n}\displaystyle\frac{1-(1-p)^{i}}{i}\quad(p\in[0,1])$

as the main result of a 2023 _Discrete Mathematics_ paper, where
$H_{k}:=\displaystyle\sum\nolimits_{i=1}^{k}1/i$ is the classical $k$-th
harmonic number. Thereafter, Campbell provided several other proofs of
Mneimneh’s formula as above in a note published in _Discrete Mathematics_ in
2023. Moreover, Campbell also considered how Mneimneh’s identity may be proved
and generalized using the _Mathematica package Sigma_. In particular, he found
the generalized Mneimneh’s identity

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}H_{k}=(x+y)^{n}\left(H_{n}-\displaystyle\sum_{i=1}^{n}\displaystyle\frac{y^{i}(x+y)^{-i}}{i}\right).$

In this paper, we will prove a more generalization of Mneimneh’s identity
involving Bell numbers and some Mneimneh-type identities involving
(alternating) harmonic numbers by using a few results of our previous papers.

Keywords: Mneimneh’s identity; harmonic numbers; Binomial coefficients and
sums; (unsigned) Stirling numbers; Bell polynomials; Bell numbers.

AMS Subject Classifications (2020): 11M32, 11M99.

##  1 Introduction

Let $\mathbb{N}=\\{1,2,3,\ldots\\}$ be the set of natural numbers, and
$\mathbb{N}_{0}:=\mathbb{N}\cup\\{0\\}$. In his recent paper [7], Mneimneh
used the method of probabilistic analysis to establish the following binomial
sum identity involving harmonic numbers

$\displaystyle\displaystyle\sum_{k=0}^{n}H_{k}\binom{n}{k}p^{k}(1-p)^{n-k}=\displaystyle\sum_{i=1}^{n}\displaystyle\frac{1-(1-p)^{i}}{i}\quad(p\in[0,1]),$
(1.1)

where $H_{k}$ is the _classical $k$-th harmonic number_ defined by

$H_{k}:=\displaystyle\sum_{i=1}^{k}\displaystyle\frac{1}{i}\quad\text{and}\quad
H_{0}:=0.$

Quite recently, Campbell [1] gave two new proofs of (1.1) by using
Zeilberger’s algorithm and beta-type integral formula. Further, using the
Sigma package for the Mathematica Computer Algebra System, Campbell gave the
more general Mneimneh’s identity

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}H_{k}=(x+y)^{n}\left(H_{n}-\displaystyle\sum_{i=1}^{n}\displaystyle\frac{y^{i}(x+y)^{-i}}{i}\right).$
(1.2)

Clearly, setting $(x,y)=(p,1-p)$ in (1.2) gives (1.1). Moreover, Campbell also
emphasized that this approach may also be used to derive identities for
expressions such as

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}H_{k}^{2}.$

In a 2024 _Discrete Mathematics_ article [6], Komatsu and Wang extended
Mneimneh’s formula to the generalized hyperharmonic numbers. Gen$\check{\rm
c}$ev [4] studied of the binomial sum

$\displaystyle\displaystyle\sum_{k=0}^{n}\binom{n}{k}\left(\displaystyle\sum_{j=1}^{k}\displaystyle\frac{z^{j}}{j}\right)p^{k}(1-p)^{k}\quad(r\in\mathbb{N},\
p,z\in\mathbb{R},\ p\neq 1)$

and established explicit formula, see [4, Thm. 2.1].

In this paper, using the method of integrals of natural logarithms, we will
establish the explicit formulas of the following general Mneimneh-type
binomial sums involving Bell numbers and some Mneimneh-type binomial sums
involving (alternating) harmonic numbers.

######  Theorem 1.1.

For any reals $x,y$ with $x/(x+y)\geq 0$ and $n,p\in\mathbb{N}$, we have

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}Y_{p}(k)=(-1)^{p-1}p!(x+y)^{n}\displaystyle\sum_{j=1}^{n}\left(1-\Big{(}\displaystyle\frac{y}{x+y}\Big{)}^{j}\right)\displaystyle\sum_{i=0}^{p-1}(-1)^{i}\displaystyle\frac{Y_{i}(n)}{i!}\displaystyle\frac{s(j,p-i)}{j!},$
(1.3)

where $s(n,k)$ and $Y_{k}(n)$ stand for the (unsigned) Stirling numbers of the
first kind and Bell numbers (see Section 2), respectively.

######  Remark 1.2.

All Stirling numbers $s(n,k)$ and Bell numbers $Y_{k}(n)$ can be expressed in
terms of a linear combination of products of harmonic numbers.

In particular, letting $p=1,2$ in Theorem 1.1 and noting the facts that
$s({n,1})=({n-1})!,s({n,2})=({n-1})!{H_{n-1}},Y_{0}(n)=1,{Y_{1}}(n)={H_{n}},{Y_{2}}(n)=H_{n}^{2}+{H^{(2)}_{n}}$,
we also obtain (1.2) and

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}\big{(}H_{k}^{2}+H_{k}^{(2)}\big{)}=2(x+y)^{n}\displaystyle\sum_{j=1}^{n}\left(1-\Big{(}\displaystyle\frac{y}{x+y}\Big{)}^{j}\right)\displaystyle\frac{H_{n}-H_{j-1}}{j}.$
(1.4)

######  Theorem 1.3.

For any reals $x,y$ and $z\in(-\infty,1)$ with $x/(x+y)\geq 0$ and
$n\in\mathbb{N}$, we have

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}\left(\displaystyle\sum_{j=1}^{k}\displaystyle\frac{z^{j}}{j}\right)=\displaystyle\sum_{j=1}^{n}\displaystyle\frac{(y+zx)^{j}-y^{j}}{j}(x+y)^{n-j}.$
(1.5)

Obviously, letting $z\rightarrow 1$ in (1.5) gives (1.2). Setting $z=-1$ in
Theorem 1.3 yields the following corollary.

######  Corollary 1.4.

For any reals $x,y$ with $x/(x+y)\geq 0$ and $n\in\mathbb{N}$, we have

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}{\bar{H}}_{k}=\displaystyle\sum_{j=1}^{n}\displaystyle\frac{y^{j}-(y-x)^{j}}{j}(x+y)^{n-j},$
(1.6)

where ${\bar{H}}_{k}$ is the _alternating $k$-th harmonic number_ defined by

$\displaystyle{\bar{H}}_{k}:=\displaystyle\sum_{j=1}^{k}\displaystyle\frac{(-1)^{j-1}}{j}\quad\text{and}\quad{\bar{H}}_{0}:=0.$
(1.7)

Specially, setting $(x,y)=(p,1-p)\ (p\in[0,1])$ in (1.6) yields the following
Mneimneh-type identity

$\displaystyle\displaystyle\sum_{k=0}^{n}{\bar{H}}_{k}\binom{n}{k}p^{k}(1-p)^{n-k}=\displaystyle\sum_{j=1}^{n}\displaystyle\frac{(1-p)^{j}-(1-2p)^{j}}{j}.$
(1.8)

From equation (1.4) and Theorem 1.3, we can also obtain the following
corollary.

######  Corollary 1.5.

For any reals $x,y$ with $x/(x+y)\geq 0$ and $n\in\mathbb{N}$, we have

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}H_{k}^{(2)}=(x+y)^{n}\left\\{\displaystyle\sum_{j=1}^{n}\displaystyle\frac{y^{j}(x+y)^{-j}}{j}\displaystyle\sum_{i=1}^{j}\displaystyle\frac{y^{-i}(x+y)^{i}}{i}-\displaystyle\sum_{j=1}^{n}\displaystyle\frac{y^{j}(x+y)^{-j}}{j}H_{j}\right\\},$
(1.9)
$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}H^{2}_{k}=(x+y)^{n}\left\\{H_{n}^{2}+H_{n}^{(2)}-2\displaystyle\sum_{j=1}^{n}\displaystyle\frac{y^{j}(x+y)^{-j}}{j^{2}}-2H_{n}\displaystyle\sum_{j=1}^{n}\displaystyle\frac{y^{j}(x+y)^{-j}}{j}\atop+3\displaystyle\sum_{j=1}^{n}\displaystyle\frac{y^{j}(x+y)^{-j}}{j}H_{j}-\displaystyle\sum_{j=1}^{n}\displaystyle\frac{y^{j}(x+y)^{-j}}{j}\displaystyle\sum_{i=1}^{j}\displaystyle\frac{y^{-i}(x+y)^{i}}{i}\right\\}.$
(1.10)

We will prove Theorem 1.1 in Section 3, and Theorem 1.3 and Corollary 1.5 in
Section 4.

######  Conjecture 1.6.

The Theorems 1.1 and 1.3 hold for any reals $x,y$ and $z$.

##  2 Preliminaries and lemmas

###  2.1 (unsigned) Stirling number of the first kind

We recall the definition of _(unsigned) Stirling number of the first kind_.
Let $s(n,k)$ denote the (unsigned) Stirling number of the first kind, which is
defined by [3, 2]

$\displaystyle
n!x\left({1+x}\right)\left({1+\displaystyle\frac{x}{2}}\right)\cdots\left({1+\displaystyle\frac{x}{n}}\right)=\displaystyle\sum\limits_{k=0}^{n}s(n+1,k+1)x^{k+1},$
(2.11)

with $s(n,k):=0$ if $n<k$, and $s(n,0)=s(0,k):=0,\ s(0,0):=1$, or
equivalently, by the generating function:

$\displaystyle{\log^{k}}({1-x})={\left({-1}\right)^{k}}k!\displaystyle\sum\limits_{n=1}^{\infty}{s(n,k)\displaystyle\frac{{{x^{n}}}}{{n!}}}\quad(x\in[-1,1)).$

The Stirling numbers ${s\left({n,k}\right)}$ of the first kind satisfy a
recurrence relation in the form

$s({n,k})=s({n-1,k-1})+\left({n-1}\right)s({n-1,k})\quad(n,k\in\mathbb{N}).$

Obviously, ${s\left({n,k}\right)}$ can be expressed in terms of a linear
combinations of products of harmonic numbers (see (2.15)). In particular,

$\displaystyle s({n,1})=\left({n-1}\right)!,$ $\displaystyle
s\left({n,2}\right)=\left({n-1}\right)!{H_{n-1}},$ $\displaystyle
s({n,3})=\displaystyle\frac{{\left({n-1}\right)!}}{2}\left[{H_{n-1}^{2}-{H^{(2)}_{n-1}}}\right],$
$\displaystyle
s({n,4})=\displaystyle\frac{{\left({n-1}\right)!}}{6}\left[{H_{n-1}^{3}-3{H_{n-1}}{H^{(2)}_{n-1}}+2{H^{(3)}_{n-1}}}\right],$
$\displaystyle
s({n,5})=\displaystyle\frac{{\left({n-1}\right)!}}{{24}}\left[{H_{n-1}^{4}-6{H^{(4)}_{n-1}}-6H_{n-1}^{2}{H^{(2)}_{n-1}}+3(H^{(2)}_{n-1})^{2}+8H_{n-1}{H^{(3)}_{n-1}}}\right].$

In [9, Thm 2.5], we proved

$\displaystyle
s\left({n,k}\right)=\left({n-1}\right)!{\zeta_{n-1}}({{{\left\\{1\right\\}}_{k-1}}})\quad(k,n\in\mathbb{N}),$
(2.12)

where $\zeta_{n}(\\{1\\}_{k})$ is a special multiple harmonic sum defined by
($\\{l\\}_{r}$ denotes the sequence obtained by repeating $l$ exactly $r$
times)

$\displaystyle\zeta_{n}(\\{1\\}_{r}):=\displaystyle\sum_{n\geq
n_{1}>n_{2}>\cdots>n_{r}>0}\displaystyle\frac{1}{n_{1}n_{2}\cdots n_{r}}.$
(2.13)

For ${\boldsymbol{\sl{k}}}=(k_{1},\ldots,k_{r})\in\mathbb{N}^{r}$ and positive
integer $n$, the generalized _multiple harmonic sums_ (MHSs) are defined by

$\displaystyle\zeta_{n}({\boldsymbol{\sl{k}}})\equiv\zeta_{n}(k_{1},\ldots,k_{r}):=\displaystyle\sum\limits_{n\geq
n_{1}>\cdots>n_{r}>0}\displaystyle\frac{1}{n_{1}^{k_{1}}\cdots
n_{r}^{k_{r}}}.$ (2.14)

We set ${\zeta_{n}}(\emptyset):=1$ if ${\boldsymbol{\sl{k}}}=\emptyset$ and
${\zeta_{n}}({\boldsymbol{\sl{k}}}):=0$ if $n<r$. For
${\boldsymbol{\sl{k}}}=(k)\in\mathbb{N}$,

$\displaystyle\zeta_{n}(k)\equiv
H_{n}^{(k)}=\displaystyle\sum_{j=1}^{n}\displaystyle\frac{1}{j^{k}}$ (2.15)

is the $n$-th generalized _harmonic number_ of order $k$, and furthermore, if
$k=1$ then $H_{n}\equiv H_{n}^{(1)}$ is the classical $n$-th harmonic number.
When taking the limit $n\rightarrow\infty$ in (2.14) we get the so-called
classical _multiple zeta values_ (MZVs) (see [5, 12]),

$\displaystyle{\zeta}({\boldsymbol{\sl{k}}}):=\displaystyle\lim_{n\rightarrow\infty}{\zeta_{n}}({\boldsymbol{\sl{k}}}),$

defined for ${\boldsymbol{\sl{k}}}\in\mathbb{N}^{r}$ and $k_{1}>1$ to ensure
convergence of the series.

###  2.2 Bell polynomials

Define the _exponential partial Bell polynomials_ $B_{n,k}$ by

$\displaystyle\frac{1}{k!}\left(\displaystyle\sum_{n=1}^{\infty}x_{n}\displaystyle\frac{t^{n}}{n!}\right)^{k}=\displaystyle\sum_{n=k}^{\infty}B_{n,k}(x_{1},x_{2},\ldots,x_{n})\displaystyle\frac{t^{n}}{n!}\,,\quad
k=0,1,2,\ldots\,,$

and the _exponential complete Bell polynomials_ $Y_{n}$ by

$Y_{n}(x_{1},x_{2},\ldots,x_{n}):=\displaystyle\sum_{k=0}^{n}B_{n,k}(x_{1},x_{2},\ldots,x_{n})$

(see [2, Section 3.3]). According to [8, Eq. (2.44)], the complete Bell
polynomials $Y_{n}$ satisfy the recurrence

$Y_{0}=1\,,\quad
Y_{n}(x_{1},x_{2},\ldots,x_{n})=\displaystyle\sum_{j=0}^{n-1}\binom{n-1}{j}x_{n-j}Y_{j}(x_{1},x_{2},\ldots,x_{j})\,,\quad
n\geq 1\,,$

from which, the first few polynomials can be obtained immediately:

$\displaystyle Y_{0}=1\,,\quad Y_{1}(x_{1})=x_{1}\,,\quad
Y_{2}(x_{1},x_{2})=x^{2}_{1}+x_{2}\,,\quad
Y_{3}(x_{1},x_{2},x_{3})=x^{3}_{1}+3x_{1}x_{2}+x_{3}\,,$ $\displaystyle
Y_{4}(x_{1},x_{2},x_{3},x_{4})=x^{4}_{1}+6x^{2}_{1}x_{2}+4x_{1}x_{3}+3x^{2}_{2}+x_{4}\,.$

Define the _Bell number_ ${Y_{k}}(n)$ by

$\displaystyle{Y_{k}}(n):={Y_{k}}({{H_{n}},1!{H_{n}^{(2)}},2!{H_{n}^{(3)}},\cdots,({k-1})!{H_{n}^{(k)}}}).$
(2.16)

Clearly, the Bell number ${Y_{k}}(n)$ is a rational linear combination of
products of harmonic numbers. We have

$\displaystyle{Y_{1}}(n)={H_{n}},$
$\displaystyle{Y_{2}}(n)=H_{n}^{2}+{H^{(2)}_{n}},$
$\displaystyle{Y_{3}}(n)=H_{n}^{3}+3{H_{n}}{H^{(2)}_{n}}+2{H^{(3)}_{n}},$
$\displaystyle{Y_{4}}(n)=H_{n}^{4}+8{H_{n}}{H^{(3)}_{n}}+6H_{n}^{2}{H^{(2)}_{n}}+3(H^{(2)}_{n})^{2}+6{H^{(4)}_{n}},$
$\displaystyle{Y_{5}}(n)=H_{n}^{5}+10H_{n}^{3}{H^{(2)}_{n}}+20H_{n}^{2}{H^{(3)}_{n}}+15{H_{n}}({H^{(2)}_{n}})^{2}+30{H_{n}}{H^{(4)}_{n}}+20{H^{(2)}_{n}}{H^{(3)}_{n}}+24{H^{(5)}_{n}}.$

In [9, Eq. (2.9)], we showed

$\displaystyle\zeta_{n}^{\star}\left({{{\\{1\\}}_{m}}}\right)=\displaystyle\frac{1}{{m!}}{Y_{m}}\left(n\right)\quad(n,m\in\mathbb{N}_{0}),$
(2.17)

where $\zeta_{n}^{\star}(\\{1\\}_{r})$ is a special multiple harmonic star sum
defined by

$\displaystyle\zeta_{n}^{\star}(\\{1\\}_{r}):=\displaystyle\sum_{n\geq
n_{1}\geq n_{2}\geq\cdots\geq n_{r}>0}\displaystyle\frac{1}{n_{1}n_{2}\cdots
n_{r}}.$

For ${\boldsymbol{\sl{k}}}=(k_{1},\ldots,k_{r})\in\mathbb{N}^{r}$ and positive
integer $n$, the generalized _multiple harmonic star sums_ (MHSSs) are defined
by

$\displaystyle\zeta^{\star}_{n}({\boldsymbol{\sl{k}}})\equiv\zeta^{\star}_{n}(k_{1},\ldots,k_{r}):=\displaystyle\sum\limits_{n\geq
n_{1}\geq\cdots\geq n_{r}>0}\displaystyle\frac{1}{n_{1}^{k_{1}}\cdots
n_{r}^{k_{r}}}.$ (2.18)

Similarly, we set ${\zeta^{\star}_{n}}(\emptyset):=1$ if
${\boldsymbol{\sl{k}}}=\emptyset$ .

###  2.3 Some Lemmas

Next, we present some lemmas, which are useful in the development of our main
theorems.

######  Lemma 2.1.

([10, Thm. 2.9] ) For $n\in\mathbb{N}$ and $p\in\mathbb{N}_{0}$, we have

$\displaystyle\displaystyle\int\limits_{0}^{1}{{t^{n-1}}{{\log}^{p}}(1-t)}dt={({-1})^{p}}\displaystyle\frac{{{Y_{p}}\left(n\right)}}{n}.$
(2.19)

######  Lemma 2.2.

([9, Thm. 2.2]) For $n,m\in\mathbb{N}$ and $x\in(-\infty,1)$, we have

$\displaystyle\displaystyle\int\limits_{0}^{x}{{t^{n-1}}{{\log}^{m}}\left({1-t}\right)}dt=m!\displaystyle\frac{{{{\left({-1}\right)}^{m}}}}{n}\zeta_{n}^{\star}({{{\\{1\\}}_{m}};x})$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad+\displaystyle\frac{1}{n}\displaystyle\sum\limits_{j=0}^{m-1}{{{\left({-1}\right)}^{j}}j!\binom{m}{j}{{\log}^{m-j}}\left({1-x}\right)}\left({\zeta_{n}^{\star}({{{\\{1\\}}_{j}};x})-\zeta_{n}^{\star}({{{\\{1\\}}_{j}}})}\right),$
(2.20)

where $\zeta_{n}^{\star}(\\{1\\}_{r})\ (r\in\mathbb{N}_{0})$ is defined by

$\displaystyle\zeta_{n}^{\star}(\\{1\\}_{r};x):=\displaystyle\sum_{n\geq
n_{1}\geq n_{2}\geq\cdots\geq
n_{r}>0}\displaystyle\frac{x^{n_{r}}}{n_{1}n_{2}\cdots
n_{r}}\quad\text{and}\quad\zeta_{n}^{\star}(\emptyset;x):=x^{n}.$

(Note that in [9, Thm. 2.2], the range of values for $x$ is $x\in[-1,1)$, but
in fact, the above equation also holds for $x\in(-\infty,-1]$.)

######  Lemma 2.3.

([11, Thm. 4.3]) For
${{\boldsymbol{\sl{k}}}}=(k_{1},\ldots,k_{r})\in\mathbb{N}^{r}$ and
$l\in\mathbb{N}_{0},n\in\mathbb{N}$, we have

$\displaystyle\displaystyle\sum_{n\geq n_{1}\geq n_{2}\geq\cdots\geq
n_{r}>0}\prod_{j=1}^{r}\displaystyle\frac{1}{(n_{j}+l)^{k_{j}}}=(-1)^{r}\displaystyle\sum_{j=0}^{r}(-1)^{j}\zeta^{\star}_{n+l}(\overrightarrow{{\boldsymbol{\sl{k}}}}_{\hskip-2.0ptj})\zeta_{l}(\overleftarrow{{\boldsymbol{\sl{k}}}}_{\hskip-2.0ptj+1}),$
(2.21)

where
$\overrightarrow{{\boldsymbol{\sl{k}}}}_{\hskip-2.0ptj}:=(k_{1},\ldots,k_{j})$
and
$\overleftarrow{{\boldsymbol{\sl{k}}}}_{\hskip-2.0ptj}:=(k_{r},\ldots,k_{j})$
for all $1\leq j\leq r$.

##  3 Proof of Theorem 1.1

For $x/(x+y)=0$, namely $x=0$, (1.3) is obviously holds. For $x/(x+y)>0$,
applying Lemma 2.1, the left hand side of (1.3) can be rewritten as

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}Y_{p}(k)$
$\displaystyle=(-1)^{p}\displaystyle\sum_{k=1}^{n}x^{k}y^{n-k}\binom{n}{k}k\displaystyle\int_{0}^{1}t^{k-1}\log^{p}(1-t)dt$
$\displaystyle=(-1)^{p}\displaystyle\int_{0}^{1}\log^{p}(1-t)\displaystyle\sum_{k=1}^{n}x^{k}y^{n-k}\binom{n}{k}kt^{k-1}dt$
$\displaystyle=(-1)^{p}\displaystyle\int_{0}^{1}\log^{p}(1-t)\displaystyle\sum_{k=0}^{n-1}x^{k+1}y^{n-k-1}\binom{n}{k+1}(k+1)t^{k}dt$
$\displaystyle=(-1)^{p}\displaystyle\int_{0}^{1}\log^{p}(1-t)\displaystyle\sum_{k=0}^{n-1}x^{k+1}y^{n-k-1}n\binom{n-1}{k}t^{k}dt$
$\displaystyle=(-1)^{p}nx\displaystyle\int_{0}^{1}\log^{p}(1-t)\displaystyle\sum_{k=0}^{n-1}(xt)^{k}y^{n-k-1}\binom{n-1}{k}dt$
$\displaystyle=(-1)^{p}nx\displaystyle\int_{0}^{1}\log^{p}(1-t)(tx+y)^{n-1}dt\quad\quad\quad(\text{letting}\
u=tx+y)$
$\displaystyle=(-1)^{p}n\displaystyle\int_{y}^{x+y}u^{n-1}\log^{p}\Big{(}\displaystyle\frac{x+y-u}{x}\Big{)}du$
$\displaystyle=(-1)^{p}n\displaystyle\int_{y}^{x+y}u^{n-1}\left\\{\log\Big{(}\displaystyle\frac{x+y}{x}\Big{)}+\log\Big{(}1-\displaystyle\frac{u}{x+y}\Big{)}\right\\}^{p}du$
$\displaystyle=(-1)^{p}n\displaystyle\sum_{l=0}^{p}\binom{p}{l}\log^{p-l}\Big{(}\displaystyle\frac{x+y}{x}\Big{)}\displaystyle\int_{y}^{x+y}u^{n-1}\log^{l}\Big{(}1-\displaystyle\frac{u}{x+y}\Big{)}du$
$\displaystyle=(-1)^{p}n\displaystyle\sum_{l=1}^{p}\binom{p}{l}\log^{p-l}\Big{(}\displaystyle\frac{x+y}{x}\Big{)}\displaystyle\int_{y}^{x+y}u^{n-1}\log^{l}\Big{(}1-\displaystyle\frac{u}{x+y}\Big{)}du$
$\displaystyle\quad+(-1)^{p}\Big{(}(x+y)^{n}-y^{n}\Big{)}\log^{p}\Big{(}\displaystyle\frac{x+y}{x}\Big{)}\quad\quad\quad(\text{letting}\
u=v(x+y))$
$\displaystyle=(-1)^{p}n(x+y)^{n}\displaystyle\sum_{l=1}^{p}\binom{p}{l}\log^{p-l}\Big{(}\displaystyle\frac{x+y}{x}\Big{)}\displaystyle\int_{y(x+y)^{-1}}^{1}v^{n-1}\log^{l}\Big{(}1-v\Big{)}dv$
$\displaystyle\quad+(-1)^{p}\Big{(}(x+y)^{n}-y^{n}\Big{)}\log^{p}\Big{(}\displaystyle\frac{x+y}{x}\Big{)}.$
(3.22)

(In order to utilize Lemmas 2.1 and 2.2, the integration in the second to last
row of the above equation needs to satisfy $y(x+y)^{-1}<1$, namely
$x/(x+y)>0$) Using Lemmas 2.1 and 2.2, by an elementary calculation, the (3)
is equal to

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}Y_{p}(k)$
$\displaystyle=(x+y)^{n}\displaystyle\sum_{1\leq i\leq l\leq
p}(-1)^{l-i}i!\binom{l}{i}\binom{p}{l}\log^{p-i}\Big{(}\displaystyle\frac{x}{x+y}\Big{)}$
$\displaystyle\quad\quad\quad\quad\quad\quad\times\left({\zeta_{n}^{\star}({{{\\{1\\}}_{i}}})-\zeta_{n}^{\star}\Big{(}{{{\\{1\\}}_{i}}};y(x+y)^{-1}\Big{)}}\right)$
$\displaystyle=(x+y)^{n}\displaystyle\sum_{1\leq i\leq l\leq
p}(-1)^{l-i}i!\binom{l}{i}\binom{p}{l}\log^{p-i}\Big{(}\displaystyle\frac{x}{x+y}\Big{)}$
$\displaystyle\quad\quad\quad\quad\quad\quad\times\displaystyle\sum_{n\geq
k_{1}\geq\cdots\geq k_{i}\geq
1}\displaystyle\frac{1-\Big{(}\displaystyle\frac{y}{x+y}\Big{)}^{k_{i}}}{k_{1}\cdots
k_{i}}.$ (3.23)

Noting the fact that the (3) can be rewritten as

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}Y_{p}(k)$
$\displaystyle=(x+y)^{n}\displaystyle\sum_{i=1}^{p}(-1)^{i}i!\log^{p-i}\Big{(}\displaystyle\frac{x}{x+y}\Big{)}\displaystyle\sum_{n\geq
k_{1}\geq\cdots\geq k_{i}\geq
1}\displaystyle\frac{1-\Big{(}\displaystyle\frac{y}{x+y}\Big{)}^{k_{i}}}{k_{1}\cdots
k_{i}}$
$\displaystyle\quad\quad\quad\quad\quad\quad\times\displaystyle\sum_{l=i}^{p}(-1)^{l}\binom{l}{i}\binom{p}{l}$
(3.24)

and

$\displaystyle\displaystyle\sum_{l=i}^{p}(-1)^{l}\binom{l}{i}\binom{p}{l}$
$\displaystyle=\left\\{{\begin{array}[]{*{20}{c}}(-1)^{p},{\ \ (p=i),}\\\
\quad\quad 0,\;\;\ \ \ \ {(\text{otherwise}),}\\\ \end{array}}\right.$

we obtain

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}Y_{p}(k)=p!(x+y)^{n}\displaystyle\sum_{n\geq
k_{1}\geq\cdots\geq k_{p}\geq
1}\displaystyle\frac{1-\Big{(}\displaystyle\frac{y}{x+y}\Big{)}^{k_{p}}}{k_{1}\cdots
k_{p}}.$ (3.25)

In Lemma 2.3, replacing $n$ by $n-j+1$ and letting $r=p-1$,
$(k_{1},\ldots,k_{r})=(\\{1\\}_{p-1})$ and $l=j-1$, we easily obtain

$\displaystyle\displaystyle\sum_{n-j+1\geq i_{1}\geq\cdots\geq i_{p-1}\geq
1}\displaystyle\frac{1}{(i_{1}+j-1)\cdots(i_{p-1}+j-1)}$
$\displaystyle\quad\quad\quad=(-1)^{p-1}\displaystyle\sum_{i=0}^{p-1}(-1)^{i}\zeta^{\star}_{n}(\\{1\\}_{i})\zeta_{j-1}(\\{1\\}_{p-1-i}).$
(3.26)

Hence,

$\displaystyle\displaystyle\sum_{n\geq k_{1}\geq\cdots\geq k_{p}\geq
1}\displaystyle\frac{1-\Big{(}\displaystyle\frac{y}{x+y}\Big{)}^{k_{p}}}{k_{1}\cdots
k_{p}}$
$\displaystyle=\displaystyle\sum_{j=1}^{n}\displaystyle\frac{1-\Big{(}\displaystyle\frac{y}{x+y}\Big{)}^{j}}{j}\displaystyle\sum_{n\geq
k_{1}\geq\cdots\geq k_{p-1}\geq j}\displaystyle\frac{1}{k_{1}\cdots k_{p-1}}$
$\displaystyle=\displaystyle\sum_{j=1}^{n}\displaystyle\frac{1-\Big{(}\displaystyle\frac{y}{x+y}\Big{)}^{j}}{j}\displaystyle\sum_{n-j+1\geq
i_{1}\geq\cdots\geq i_{p-1}\geq
1}\displaystyle\frac{1}{(i_{1}+j-1)\cdots(i_{p-1}+j-1)}$
$\displaystyle=(-1)^{p-1}\displaystyle\sum_{j=1}^{n}\displaystyle\frac{1-\Big{(}\displaystyle\frac{y}{x+y}\Big{)}^{j}}{j}\displaystyle\sum_{i=0}^{p-1}(-1)^{i}\zeta^{\star}_{n}(\\{1\\}_{i})\zeta_{j-1}(\\{1\\}_{p-1-i})$
$\displaystyle=(-1)^{p-1}\displaystyle\sum_{j=1}^{n}\left(1-\Big{(}\displaystyle\frac{y}{x+y}\Big{)}^{j}\right)\displaystyle\sum_{i=0}^{p-1}(-1)^{i}\displaystyle\frac{Y_{i}(n)}{i!}\displaystyle\frac{s(j,p-i)}{j!},$
(3.27)

where we used the equations (2.12) and (2.17).

Finally, substituting (3) into (3.25) yields the desired evaluation (1.3).
Thus, this concludes the proof of Theorem 1.1.

##  4 Proofs of Theorem 1.3 and Corollary 1.5

For $x/(x+y)=0$, namely $x=0$, Theorem 1.3 and Corollary 1.5 are obviously
hold. For $x/(x+y)>0$, in (2.2), setting $m=1$ and replacing $x$ by $z$ gives

$\displaystyle\displaystyle\int_{0}^{z}{{t^{n-1}}\log({1-t})}dt=\displaystyle\frac{1}{n}\left\\{{{x^{n}}\log({1-z})-\displaystyle\sum\limits_{j=1}^{n}{\displaystyle\frac{{{z^{j}}}}{j}}-\log({1-z})}\right\\}.$
(4.28)

Hence,

$\displaystyle\displaystyle\sum\limits_{j=1}^{n}\displaystyle\frac{z^{j}}{j}=(z^{n}-1)\log(1-z)-n\displaystyle\int_{0}^{z}t^{n-1}\log(1-t)dt.$

Hence, by a similar argument as in the proof of (3) gives

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}\left(\displaystyle\sum_{j=1}^{k}\displaystyle\frac{z^{j}}{j}\right)$
$\displaystyle=-nx\displaystyle\int_{0}^{z}\log(1-t)(xt+y)^{n-1}dt+\log(1-z)\left((xz+y)^{n}-(x+y)^{n}\right)$
$\displaystyle=-n\displaystyle\int_{y}^{xz+y}u^{n-1}\log\left(\displaystyle\frac{x+y-u}{x}\right)du+\log(1-z)\left((xz+y)^{n}-(x+y)^{n}\right)$
$\displaystyle=-n\displaystyle\int_{y}^{xz+y}u^{n-1}\left\\{\log\left(\displaystyle\frac{x+y}{x}\right)+\log\left(1-\displaystyle\frac{u}{x+y}\right)\right\\}du$
$\displaystyle\quad+\log(1-z)\left((xz+y)^{n}-(x+y)^{n}\right)$
$\displaystyle=-\log\left(\displaystyle\frac{x+y}{x}\right)\left((xz+y)^{n}-y^{n}\right)+\log(1-z)\left((xz+y)^{n}-(x+y)^{n}\right)$
$\displaystyle\quad-n\displaystyle\int_{y}^{xz+y}u^{n-1}\log\left(1-\displaystyle\frac{u}{x+y}\right)du$
$\displaystyle=-\log\left(\displaystyle\frac{x+y}{x}\right)\left((xz+y)^{n}-y^{n}\right)+\log(1-z)\left((xz+y)^{n}-(x+y)^{n}\right)$
$\displaystyle\quad-n(x+y)^{n}\displaystyle\int_{y(x+y)^{-1}}^{(xz+y)(y+x)^{-1}}v^{n-1}\log(1-v)dv.$
(4.29)

Therefore, applying (4.28), by a direct calculation, we deduce

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}\left(\displaystyle\sum_{j=1}^{k}\displaystyle\frac{z^{j}}{j}\right)=(x+y)^{n}\displaystyle\sum_{j=1}^{n}\displaystyle\frac{\left(\displaystyle\frac{xz+y}{y+x}\right)^{j}-\left(\displaystyle\frac{y}{y+x}\right)^{j}}{j}.$
(4.30)

This completes the proof of Theorem 1.3 (Noting that from $x/(x+y)>0$ and
$z\in(-\infty,1)$ gives $(xz+y)(x+y)^{-1}<1$).

Multiplying (1.5) by $1/z$ and integrating over the interval (0,1), we have

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}H_{k}^{(2)}$
$\displaystyle=\displaystyle\sum_{j=1}^{n}\displaystyle\frac{(x+y)^{n-j}}{j}\displaystyle\int_{0}^{1}\displaystyle\frac{(xz+y)^{j}-y^{j}}{z}dz$
$\displaystyle=\displaystyle\sum_{j=1}^{n}\displaystyle\frac{(x+y)^{n-j}}{j}y^{j}\displaystyle\sum_{i=1}^{j}\displaystyle\int_{0}^{1}\Big{(}\displaystyle\frac{xz}{y}+1\Big{)}^{i-1}d\displaystyle\frac{xz}{y}$
$\displaystyle=(x+y)^{n}\left\\{\displaystyle\sum_{j=1}^{n}\displaystyle\frac{y^{j}(x+y)^{-j}}{j}\displaystyle\sum_{i=1}^{j}\displaystyle\frac{y^{-i}(x+y)^{i}}{i}-\displaystyle\sum_{j=1}^{n}\displaystyle\frac{y^{j}(x+y)^{-j}}{j}H_{j}\right\\}.$
(4.31)

Then, applying the well-known identity

$\displaystyle\displaystyle\sum_{j=1}^{n}\displaystyle\frac{H_{j}}{j}=\zeta^{\star}_{n}(1,1)=\displaystyle\frac{1}{2}Y_{2}(n)=\displaystyle\frac{H_{n}^{2}+H_{n}^{(2)}}{2}$

and substituting (4) into (1.4) yields (1.10). Thus, we complete the proof of
Corollary 1.5.

##  5 Conclusion

We presented the explicit formulas of the following Mneimneh-type binomial sum
of (alternating) harmonic numbers

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}Y_{p}(k),\quad\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}{\bar{H}}_{k},\quad\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}H_{k}^{(2)}\quad\text{and}\quad\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}H^{2}_{k}.$

It is possible that some of other Mneimneh-type binomial sums can be obtained
using techniques of the present paper. For example, multiplying (1.5) by
$\log^{p-1}(z)/z\ (p\in\mathbb{N})$ and integrating over the interval (0,1),
we have

$\displaystyle\displaystyle\sum_{k=0}^{n}x^{k}y^{n-k}\binom{n}{k}H_{k}^{(p+1)}$
$\displaystyle=\displaystyle\frac{(-1)^{p-1}}{(p-1)!}\displaystyle\sum_{j=1}^{n}\displaystyle\frac{(x+y)^{n-j}}{j}\displaystyle\int_{0}^{1}\log^{p-1}(z)\displaystyle\frac{(xz+y)^{j}-y^{j}}{z}dz.$
(5.32)

Hence, if the evaluation of the integral on the right hand of above can be
established, we can obtain the explicit formula of Mneimneh-type binomial sums
on the left hand of above. We leave the detail to the interested reader. It
should be emphasized that Komatsu-Wang [6, Eq. (4)] gave an evaluation of
(5.32) with $x=1-q$ and $y=q$, where $q$ is a real number. Gen$\check{\rm
c}$ev [4, Thm. 2.1] established a generalization of Mneimneh summation formula

$\displaystyle\displaystyle\sum_{k=0}^{n}\binom{n}{k}\left(\displaystyle\sum_{j=1}^{k}\displaystyle\frac{z^{j}}{j^{r}}\right)p^{k}(1-p)^{k}=\displaystyle\sum_{n\geq
n_{1}\geq\cdots n_{r}\geq
1}\displaystyle\frac{(1-p)^{n_{1}}\left(\left(1+\displaystyle\frac{zp}{1-p}\right)^{n_{r}}-1\right)}{n_{1}\cdots
n_{r}},$ (5.33)

where $r\in\mathbb{N},p,z\in\mathbb{R}$ and $p\neq 1$.

Declaration of competing interest. The authors declares that they has no known
competing financial interests or personal relationships that could have
appeared to influence the work reported in this paper.

Data availability. No data was used for the research described in the article.

Acknowledgments. Ce Xu expresses his deep gratitude to Profs. J. Zhao and J.M.
Campbell for valuable discussions and comments. Ce Xu is supported by the
National Natural Science Foundation of China (Grant No. 12101008), the Natural
Science Foundation of Anhui Province (Grant No. 2108085QA01) and the
University Natural Science Research Project of Anhui Province (Grant No.
KJ2020A0057).

## References

  * [1] J.M. Campbell, On Mneimneh’s binomial sum involving harmonic numbers, _Discrete Math._ 346(2023), 113549.
  * [2] L. Comtet, _Advanced Combinatorics_ , D. Reidel Publishing Co., Dordrecht, 1974.
  * [3] J.H. Conway and R.K. Guy, _The book of numbers_. Springer, New York (1996).
  * [4] M. Gen$\check{\rm c}$ev, Generalized Mneimneh sums and their application to multiple polylogarithms via Toeplitz theorem, private email dated March 9, 2024.
  * [5] M.E. Hoffman, Multiple harmonic series, _Pacific J. Math._ 152(1992), 275–290.
  * [6] T. Komatsu and P. Wang, A generalization of Mneimneh’s binomial sum of harmonic numbers, _Discrete Math._ 347(2024), 113549.
  * [7] S. Mneimneh, A binomial sum of harmonic numbers, _Discrete Math._ 346(2023), 113075.
  * [8] J. Riordan, _An Introduction to Combinatorial Analysis_ , Reprint of the 1958 original, Dover Publications, Inc., Mineola, NY, 2002.
  * [9] C. Xu, Multiple zeta values and Euler sums, _J. Number Theory_ 177(2017), 443–478.
  * [10] C. Xu, Y. Yan and Z. Shi, Euler sums and integrals of polylogarithm functions, _J. Number Theory_ 165(2016), 84-108.
  * [11] C. Xu and J. Zhao, Explicit relations of some variants of convoluted multiple zeta values. arXiv:2103.01377.
  * [12] D. Zagier, _Values of zeta functions and their applications_ , in: First European Congress of Mathematics, Vol. II, pp. 497–512, Birkhauser, Boston, 1994.