# PhysMLE: Generalizable and Priors-Inclusive Multi-task Remote Physiological
Measurement

Jiyao Wang, Hao Lu, Ange Wang, Xiao Yang, Yingcong Chen, Dengbo He, and
Kaishun Wu This work was supported by the National Natural Science Foundation
of China (No. 52202425), Guangzhou Municipal Science and Technology Project
(No. 2023A03J0011), and Guangzhou Science and Technology Program City-
University Joint Funding Project (No. 2023A03J0001) (Corresponding author:
Dengbo He). Jiyao Wang, Ange Wang, and Dengbo He is with the Systems Hub, the
Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China,
(e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>dengbohe@ust.hk). Hao Lu, Yingcong Chen, and Kaishun Wu are with the
Information Hub, the Hong Kong University of Science and Technology
(Guangzhou), Guangzhou, China, (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>wuks@hkust-gz.edu.cn). Xiao Yang is with Sichuan
Agricultural University, Yaan, China, (e-mail:
202005537@stu.sicau.edu.cn).Manuscript received May 10, 2024.

###### Abstract

Remote photoplethysmography (rPPG) has been widely applied to measure heart
rate from face videos. To increase the generalizability of the algorithms,
domain generalization (DG) attracted increasing attention in rPPG. However,
when rPPG is extended to simultaneously measure more vital signs (e.g.,
respiration and blood oxygen saturation), achieving generalizability brings
new challenges. Although partial features shared among different physiological
signals can benefit multi-task learning, the sparse and imbalanced target
label space brings the seesaw effect over task-specific feature learning. To
resolve this problem, we designed an end-to-end Mixture of Low-rank Experts
for multi-task remote Physiological measurement (PhysMLE), which is based on
multiple low-rank experts with a novel router mechanism, thereby enabling the
model to adeptly handle both specifications and correlations within tasks.
Additionally, we introduced prior knowledge from physiology among tasks to
overcome the imbalance of label space under real-world multi-task
physiological measurement. For fair and comprehensive evaluations, this paper
proposed a large-scale multi-task generalization benchmark, named Multi-Source
Synsemantic Domain Generalization (MSSDG) protocol. Extensive experiments with
MSSDG and intra-dataset have shown the effectiveness and efficiency of
PhysMLE. In addition, a new dataset was collected and made publicly available
to meet the needs of the MSSDG.

###### Index Terms:

rPPG, multi-task learning, mixture of experts, low-rank adaptation, domain
generalization.

##  1 Introduction

Physiological monitoring plays a vital role in assessing well-being and
performance in various sectors, including healthcare, emotional analysis, and
driver monitoring systems [1, 2]. Key indicators including heart rate (HR),
heart rate variability (HRV), respiration rate (RR), and blood oxygen
saturation (SpO2) are crucial for assessing well-being and performance of
human subjects. Traditional methods like electrocardiography (ECG) and
photoplethysmography (PPG) require direct contact of sensors and skin, causing
inconvenience and discomfort for long-term monitoring [3]. To address this
problem, non-contact video-based physiological measurement techniques have
gained increasing interest. Remote photoplethysmography (rPPG) is highlighted
as a promising approach, which can be realized through ubiquitous imaging
devices like RGB webcams and smartphone cameras. rPPG methods analyze subtle
facial skin periodic light absorption caused by cardiac activities [4] to
obtain blood volume pulse (BVP) signals, which can further be used to extract
HR and HRV [5]. Previous studies have also attempted to estimate RR [6, 7, 8]
and SpO2 [9, 10, 11] from RGB facial videos by detecting color changes and
movements associated with the blood flow during the breathing cycle and the
varying absorption of light by oxygenated and deoxygenated hemoglobin in the
blood vessels of the face, respectively. Recently, advancements in deep
learning (DL) have shown superior performance in complex scenarios [9, 7, 12],
making DL-based methods highly promising for non-contact physiological
measurement.

Conventional methods of remote physiological measurement are typically
concentrated on one predictive task (e.g., BVP or RR), within a single domain
[13, 14]. These methods are developed by training on datasets gathered from
one specific scenario and are tailored to predict one physiological indicator,
while is difficult to meet the needs of applications in real scenarios [15].
For example, patient monitoring systems that rely on a single vital sign may
lead to ’alarm fatigue’ and further make clinical staff desensitized [16]. At
the same time, training separate models for each task incurs high deployment
costs and reduces iterative efficiency [17]. In remote physiological
measurement, different targets are functionally related and there are
dependencies between multiple tasks. For instance, the hemodynamic changes
brought by respiration can influence the blood flow and blood volume in the
peripheral vasculature, leading to variations in the BVP signal [18]. Some
studies tried to develop one united model to simutanously estimate multiple
vital signs [19, 20]. However, although these methods utilized the
dependencies between different tasks, they neglected the multi-source domain
generalization (MSDG) [21].

Limited by the laboratory dataset acquisition process, DL methods trained on
multi-source datasets suffer from severe performance degradation in the
unknown target domain due to variations between different multiple datasets,
which was usually defined as the domain shift [22]. The domain shift can be
caused by different data acquisition equipment, individual differences, and
environmental changes. In recent years, to enhance the generalizability of DL-
based remote physiological measurement models in the presence of domain shift,
MSDG methods for rPPG continuously gained attention in the rPPG field [22, 23,
21]. However, previous multi-task methods failed to leverage the full
potential of the available data and overlook the semantic differences in the
feature space under multi-domain settings, which can result in suboptimal
performance [17]. Further, past MSDG methods focused on aligning the feature
semantics under different domains but ignored target dependencies in the label
space under multi-task settings. Particularly, in the field of remote
physiological measurement, different public datasets only provided partial
vital signs labels (see Table I). Thus, we cannot simply and directly reuse
MSDG for rPPG to multi-task settings.

Figure 1: Illustration of the difference between Multi-source Synsemantic
Domain Generalization (MSSDG) and classic multi-task learning or multi-source
domain generalization.

Given this, we raise a new challenge of Multi-Source Synsemantic Domain
Generalization (MSSDG), which extends beyond the conventional MSDG by
incorporating the complexity encountered in real-world applications (as shown
in Figure 1). Unlike isolated multi-task learning paradigms or traditional
MSDG approaches, MSSDG aims to address the simultaneous occurrence of multiple
tasks and domains, which represents a more intricate challenge. Specifically,
MSSDG must reconcile disparities in feature semantics and their relative
significance that arise both across different domains of a singular task and
within a single domain of disparate tasks. Additionally, it must account for
the varying degrees of target sparsity and the interdependencies that are
present among different tasks within the same domain as well as for the same
task across various domains.

To address this challenge, inspired by the standout performance in multi-task
learning of Mixture-of-Experts (MoE) [24], we propose an end-to-end Mixture of
Low-rank Experts for multi-task remote Physiological measurement (PhysMLE),
which fully exploits the dependencies between tasks via constructing multiple
low-rank trainable experts. Different from classic MoE architecture, the
introduced low-rank expert can significantly reduce the high tuning costs of
MoE, and implement plug-and-play. Besides, the proposed PhysMLE can also
address the limitations of previous Parameter Efficient Fine-Tuning (PEFT)
methods (e.g., P-tuning [25], LoRA [26]) when faced with multi-task setting.
Specifically, PEFT methods need to be fine-tuned for multiple sets of
parameters for each task separately, or one set of parameters for all tasks.
In brief, PhysMLE takes the primary decoded feature from the original first
block of the substitutable backbone network as input. Then, we replaced the
core feature learning module of the backbone with our PhysMLE layer, which
consists of multiple learnable Low-rank Adaptation (LoRA) experts, multiple
expert-specific feature routers, and the original frozen weights. During
training, a regularization based on weight orthogonality is proposed to force
each expert to learn distinct information to avoid overlapping of different
physiological features. The expert-specific feature routers are designed to
perform element-wise selection on the generated feature from each LoRA expert.
After processing through multiple-layer PhysMLE basic blocks, the features are
fed to each task-specific router separately for gating to obtain a sparse
task-specific representation, which is further used for task-specific head for
sign estimation.

Moreover, to cope with domain shift in MSSDG with unbalanced target task
space, we propose the following solution. In the MSSDG setting, the challenge
of generalizability exists not only in the conditional domain shift [27] due
to environmental changes, but also in identifying the invariance between the
inputs and the labeling of the varying tasks in different domains. In the
field of remote physiological monitoring based on face videos, inspired by
[28], we argue that the physiological signals reflected from different parts
of the face should remain consistent. Also, the changes in a person’s
physiological characteristics over a short time (5 seconds) should be
temporally smooth and gradual. Therefore, we constrain the aggregated
representation of each layer of the PhysMLE basic block to remain semantically
spatio-temporally consistent through two new regularizations. Furthermore, to
resolve the imbalance of target tasks in MSSDG, considering the heterogeneous
characteristics and correlations among different physiological signals, we
strengthen the learning capacity of PhysMLE through three regularizations
based on prior knowledge from biomedical fields [29, 10]. We summarize the
contributions of this work as follows:

  * •

We proposed a novel end-to-end framework PhysMLE, which adapts the strengths
of MoE and LoRA to efficiently learn multiple low-rank expert weights to avoid
gradient contradiction in multi-task settings. A weight orthogonality
constraint was used to ensure that the widest possible gradient space is
learned, and an element-wise router was tailored to a flexible combination of
expert features for maximizing the use of positive associations between tasks.

  * •

To resolve domain shift and unbalanced target task space under MSSDG, we not
only enhance the semantic spatio-temporal consistency in each layer of
PhysMLE, but also propose three task-specific priors to augment the imbalance
label space, which is introduced in rPPG modeling for the first time.

  * •

As the first study to tackle domain shift in multi-task remote physiological
measurement, we proposed a new Multi-Source Synsemantic Domain Generalization
(MSSDG) protocol. Considering the shortcomings of the previously public
dataset, we collected and publicized a new dataset (i.e., High Cognitive Load
Workers (HCW)). Jointly with five previously public datasets, we built a new
large MSSDG benchmark. Extensive experiments demonstrated the superiority of
our method.

TABLE I: Summary of Public Non-synthetic Datasets.

Dataset | Date | Frames | Subject | Camera Setting | HR | BVP | SpO2 | RR | Special Human State | Collection Environment
---|---|---|---|---|---|---|---|---|---|---
MAHNOB-HCI [30] | 2012 | _ | 27 | Allied Vision Stingray F-046C | ✓ | ✕ | ✕ | ✓ | Emotion | $\mid$
PURE [31] | 2014 | 168,120 | 10 | eco274CVGE | ✓ | ✓ | ✓ | ✕ | _ | $\mid$
COHFACE [32] | 2016 | _ | 40 | Logitech HD C525 | ✓ | ✕ | ✕ | ✓ | _ | $\mid$
MMSE-HR [33] | 2016 | _ | 40 | Di3D | ✓ | ✕ | ✕ | ✕ | Emotion | $\mid$
OBF [34] | 2018 | _ | 100 | Blackmagic URFA mini | ✓ | ✕ | ✕ | ✓ | Atrial fibrillation | Laboratory environment
UBFC-rPPG [35] | 2019 | 57,420 | 42 | Logitech C920 HD Pro | ✓ | ✓ | ✕ | ✕ | _ | $\mid$
BUAA [36] | 2020 | 129,549 | 15 | Logitech C930E HD pro | ✓ | ✓ | ✕ | ✕ | _ | $\mid$
VIPL-HR [37] | 2020 | 1,834,785 | 107 | Three cameras | ✓ | ✓ | ✓ | ✕ | _ | $\mid$
V4V [38] | 2021 | 799,113 | 140 | Di3D | ✓ | ✓ | ✕ | ✓ | Emotion | $\mid$
MMPD [39] | 2023 | 1,188,000 | 33 | Galaxy S22 Ultra | ✓ | ✕ | ✕ | ✕ | _ | $\mid$
HCW | 2024 | 1,565,230 | 41 | 32 laptop webcams | ✓ | ✓ | ✕ | ✓ | Cognitive workload | Real-world workspace

Notes: The datasets we used in this work are bolded in black.

##  2 Related Work

###  2.1 Camera-based Physiological Measurement

Since 2008, among the non-contact techniques, remote photoplethysmography
(rPPG) has emerged as a particularly promising approach for physiological
measurement [40]. Verkruysse et al. [40] pointed out that the green channel
contains the strongest plethysmography signal because hemoglobin absorbs green
light the most, and the red and blue channels also contain plethysmography
information. The basic theory is built on optical frameworks such as the
Lambert-Beer law (LBL) and Shafer’s dichromatic reflection model (DRM) to
model how light interacts with the skin [10]. Based on this, traditional
methods rely on statistical information about heartbeats and often require
manual adjustments to filter out inaccurate data [40, 41, 42]. However, these
methods often required manual adjustments and struggled with variations in
skin tones and lighting conditions. Recently, DL technologies have been
developed to improve accuracy in complex environments. Chen and McDuff [43]
first developed a deep model called DeepPhys that extracts BVP signals with
convolutional neural networks (CNN) by measuring the difference between
frames. In recent years, other advanced methods based on different feature
extractors (e.g., 3DCNN [44], Transformer [45]) were proposed as well.

Besides HR, RR is also a critical vital sign that is critical for health
monitoring. Some early studies remotely measured RR by detecting respiratory
motion through pixel movements [46] or pixel intensity variations [47, 48].
Some rPPG-based studies adopted traditional methods [49] or DL [19, 20] to
measure RR. At the same time, existing rPPG works also estimated the frequency
domain attributes of predicted BVP signals for RR estimation [34, 50, 45].
However, existing methods are still troubled by motion artifacts and when the
respiration is non-periodic [14], there are errors in estimated BVP, making it
challenging for indirect RR inferencing.

Another vital sign, SpO2, has traditionally been measured by analyzing direct
current (DC) and alternating current (AC) components at wavelengths of 520 nm
and 660 nm, with specialized cameras sensitive to specific wavelengths of
light that differentiate oxygenated and deoxygenated hemoglobin [51]. However,
recent efforts aim to estimate SpO2 using readily available RGB cameras by
analyzing subtle changes in facial skin color [52, 10]. Emerging DL-based
approaches demonstrate promising results in contactless SpO2 estimation [53,
54], although further development is required to achieve clinical-grade
accuracy and robustness.

###  2.2 Remote Multi-task Physiological Measurement

The complex multi-task issue was often handled by decomposing them into
multiple simple and mutually independent single tasks. However, training
models for each task separately incur high deployment costs, and such an
approach often ignores the correlation information enriched among tasks. To
address these problems, multi-task learning that puts related tasks together
has been proposed [55]. Multi-task learning was widely used in deep learning
[56, 57, 58, 59, 60]. The MoE model [59] turns the parameters shared by all
samples in each stratum into multiple sets of parameters to increase model
capacity. However, previous MoE-based methods [61] directly build multiple
complete feature encoders as experts, which introduces more computational
parameters. Recently, many works attempted to adapt LoRA [26] to MoE structure
[62, 63]. However, they focus mainly on LLM fine-tuning rather than
multitasking learning.

In traditional rPPG tasks, the output is usually performed only for a single
task (i.e., HR or BVP) [13, 14]. Due to different vital signs measured by rPPG
are often correlated, recently, MTTS-CAN [19] performed cardiovascular and
respiratory measurements in real-time, and BigSmall [20] outputs facial
movements, respiratory and HR characteristics simultaneously. However, these
studies ignored SpO2, which also correlates with HR, RR [64]. Furthermore,
previous multi-task rPPG works were usually trained on a single domain with
complete multi-task labels, and since current publicly available datasets
often only provide partial physiological signal labels (referring to Table I).
Thus, previous MoE or LoRA alone may not resolve the multi-task physiological
measurement problem.

###  2.3 Domain Generalization in Remote Photoplethysmography

There are also differences between different domains in rPPG tasks, to address
this problem, many DG methods [65, 21, 22] have been applied to rPPG tasks.
The main goal of the DG technique is to generalize the model trained in source
domain data to the unseen target domain. The mainstream methods can be divided
into three categories: data manipulation, representation learning, and meta-
learning. Data manipulation is mainly done through data augmentation methods
to increase the number of samples or using data generation methods to produce
different samples [66, 67]. Representation learning [68, 69, 70] adapts models
to different domains by learning domain-invariant features. Finally, meta-
learning approaches [71, 72, 73] induces domain shifts in virtual data during
training, allowing the model to handle unknown domains better.

For example, NEST [21] improved the generalization ability of the model by
maximizing the feature space during training. However, ignoring instance-
specific variations tends to affect the results of the model when confronted
with specific samples that lack instances during inference. HSRD [22] was
proposed to measure BVP signals by separating domain-invariant and instance-
specific feature space, but it still relied on explicit domain labels for
feature learning. Since existing DG methods for remote physiological
monitoring are limited to HR and BVP, when DG is applied to the multi-task
setting, semantic differences in the feature space and imbalance target tasks
might restrict the performance of previous approaches.

TABLE II: A summary of symbols and descriptions Symbol | Description
---|---
$L,W,C$ | Length, width, and the number of channels of a STMaps.
$B$ | Batch size.
$N$ | Number of tasks.
$K$ | Number of low-rank experts.
$T$ | Number of PhysMLE layers in the backbone network.
$M$ | Number of basic blocks in the backbone network.
$E$ | Low-rank expert.
$G$ | EFRouter.
$\textbf{W},\textbf{W}_{\Delta}$ | The frozen and updated weights in the expert.
$\textbf{B},\textbf{A}$ | The low-rank weight matrix in the expert.
$d_{in},d_{out}$ | Input and output dimension of weights.
$r$ | The number of ranks.
$\alpha$ | Hyper-parameter to scale the weights.
$\gamma$ | Hyper-parameter to enlarge the selected feature.
$s$ | Feature encoded by each PhysMLE basic layer.
$s^{{}^{\prime}}$ | Task-agnostic feature outputted by the PhysMLE.
$s^{{}^{\prime\prime}}$ | Task-specific feature after element-wise selection.
$X^{{}^{\prime}},Y^{{}^{\prime}}$ | Spatial-temporal augmented STMaps and labels.
$\hat{Y}$ | Estimated label by PhysMLE.
$tau,\mathrm{T}$ | Augmentation ratios for SpO2 prior.
$\delta$ | Soft boundary of the difference between estimated RR
| and RR extracted from ground-truth BVP.
$p_{i}$ | Hyper-parameter for joint training.
$\mathrm{I}$ | Matrix that all elements are 1.
$diag$ | Identity matrix.
$\epsilon$ | Random noise sampled from Gaussian distribution.
$\mathcal{L}$ | The loss fuction of this work.

##  3 Methodology

This section formally introduces the Mixture of Low-rank Experts for multi-
task remote Physiological measurement (PhysMLE). We begin with the problem
formulation and challenge of multi-source synsemantic domain generalization in
Subsection A. Subsequently, the framework of PhysMLE, including the overall
structure of how PhysMLE can be integrated into a substitutable backbone
network, the internal structure of the PhysMLE layer, and the router is
provided in Subsection B, where we also verify the idea of learning unique
low-rank parameters for each task. We then elaborate on the regularizations to
maintain semantically spatio-temporally consistent and eliminate the cross-
domain environmental variation in Subsection C. Lastly, subsection D
introduces the optimization and inference processes. Important notations and
descriptions are in Table II.

Figure 2: The overall architecture of proposed PhysMLE. Our method achieves
flexibility over two types of backbone network structures (subfigure (a)(b)).
Briefly, PhysMLE takes STMaps that are compressed from facial videos from
multiple domains as input. The core feature learning layers in each basic
block of the backbone are replaced by the PhysMLE layer (subfigure (c)). To
adapt different backbone structures, the proposed EFRouter can also be
instantiated as two types (subfigure (d)).

###  3.1 Problem Formulation

Here we define the notations and problem settings of our study. As shown in
Figure 2, suppose we have a batch of $B$ raw facial videos, which are from
multiple datasets. We first compress videos into the spatial-temporal map
(STMap) format [37]. Then we input STMaps
$X=\\{x_{i}\\}_{i=1}^{B},x_{i}\in\mathbb{R}^{L\times W\times C}$ into PhysMLE,
which is formulated as $f(X;\theta)$. The $L,W,C$ is the length, width, and
channel size of STMap, and $\theta$ is all parameters in the model. There are
$N=4$ predicted vital signs of the subject in the target domain
$Y=\\{y_{hr},y_{bvp},y_{spo},y_{rr}\\}$ in this work, where $y_{hr}$ indicates
HR, $y_{bvp}$ is the BVP, $y_{spo}$ and $y_{rr}$ corresponds to SpO2 and RR,
respectively.

In the field of remote physiological measurement, particularly in the context
of multi-task video-based assessment in a cross-domain setting, one of the
notable challenges stems from the domain shift. While the generalized
principle of rPPG-based physiological measurements [40] provides opportunities
for leveraging data from diverse contexts, it introduces a challenge: the
differing skin properties, lighting conditions, camera quality, and
environmental factors lead to changes in color intensity captured by an RGB
camera in different scenarios. Additionally, physiological signals related to
HR, BVP, RR, and SpO2 often exhibit overlapping frequency components. For
instance, HR and BVP share frequency bands, as both are influenced by cardiac
activity. Thus, the cross-contamination between physiological signals adds
another layer of complexity. Specifically, changes in one physiological
parameter can affect or mask the variations in another parameter. For example,
changes in SpO2 can impact the BVP signal, as oxygenation levels influence the
light absorption properties of tissues [74]. Therefore, the goal of our model
$f$ is to maximize the effective changes in color intensity while minimizing
interference from other signals and domain variations simultaneously.

###  3.2 PhysMLE

Mixture of Low-rank Expert In this section, we introduce the concept of
Mixture-of-Experts (MoE) [24], proposing a unified PhysMLE basic block
adaptive to the backbone network with an element-wise gate routing strategy.
To learn the underlying relationships between different physiological signals,
PhysMLE basic blocks are shared for the further estimation of each task.

In a typical MoE structure, multiple expert blocks are injected into one layer
of the model. Although this method accommodates a greater number of parameters
and overcomes the limitation of a singular set of parameters across all tasks,
the parameter number to be tuned will increase exponentially. Recently, LoRA
has demonstrated both its effectiveness and efficiency in fine-tuning, with
its integral parameters being fine-tuned for all tasks.

Thus, as shown in Figure 2(c), the proposed PhysMLE basic block seamlessly
integrates the advantages of both LoRA and MoE. Firstly, similar to LoRA for
large language models (LLM) [26], we reformulate the parameter update with the
equation $\textbf{W}+\textbf{W}_{\Delta}=\textbf{W}+\textbf{BA}$, where
$\textbf{W},\textbf{W}_{\Delta}\in\mathbb{R}^{d_{in}\times d_{out}}$ denote
the frozen weight matrix and the weight matrix that is updated during fine-
tuning, respectively. $\textbf{B}\in\mathbb{R}^{d_{in}\times r}$ and
$\textbf{A}\in\mathbb{R}^{r\times d_{out}}$ are low-rank trainable matrix with
$r\ll d_{in}/d_{out}$. Here, $d_{in},d_{out}$ are input and output vector
dimensions, $r$ is the rank. Particularly, when LoRA adapted to convolutional
network [75], $d_{in},d_{out}$ are replaced by $C_{in},C_{out}$, which present
the number of input and output channels. In our PhysMLE, we introduce a set of
low-rank experts ${E_{i}}^{K}_{i=1}$, each $E_{i}$ consists of a pair of BA,
and both two types of LoRA were implemented for different backbone networks.

Besides, being different from recent MoE-based LoRA frameworks [75, 63, 62],
PhysMLE introduces an Element-wise Feature Router (EFRouter) group mechanism
(see Figure 2(d)), which aims to leverage low-level physiological features to
dynamically aggregate element-level feature outputted by each expert. The
previous gate mechanism in MoE is mostly based on applying the softmax
function over the linearly transformed high-level representation of experts
[62, 26]. However, given the strong association between physiological signals
and facial low-level features (e.g., movement [76], expression [77]), the gate
mechanism of previous MoE is difficult to be applied to remote multitasking
physiological measurement. Thus, inspired by [17], we designed the EFRouter
groups $\\{G_{i}\\}_{i=1}^{K}$, where each $G_{i}$ corresponds to one expert
$E_{i}$, to adaptively control the importance of low-level information, and
used the hyperparameter $\gamma=2$ to further squash and expand the effective
feature elements of the corresponding expert $E_{i}$. Each $G_{i}$ takes the
low-level feature $s$ outputted by the first layer of the backbone network
(i.e., Conv1 in ResNet [78], Patch Projection & Position Embedding layers in
ViT [79]) as the input. The $G_{i}$ consists of two neural layers, which are
with one normalization layer, and one non-linear activation
($\operatorname{ReLU}$). After passing through the two layers,
$\operatorname{Sigmoid}$ function was used to generate gated vectors. In all,
the process of generating the output $s^{{}^{\prime}}$ of each PhysMLE layer
is customized as follows:

$\displaystyle s^{{}^{\prime}}$
$\displaystyle=\textbf{W}s+\frac{\alpha}{r}\cdot\textbf{W}_{\Delta}s,$ (1)
$\displaystyle=\textbf{W}s+\frac{\alpha}{r}\cdot\sum_{i=1}^{K}\gamma\cdot
G_{i}(s)\cdot E_{i}(s),$
$\displaystyle=\textbf{W}s+\frac{\alpha}{r}\cdot\sum_{i=1}^{K}\gamma\cdot
G_{i}(s)\cdot\textbf{B}_{i}\textbf{A}_{i}s.$

where $\alpha$ is the constant hyperparameter facilitating the tuning of rank
$r$, and $\cdot$ is element-wise dot product. Note that, as the convolution
layer has the stride of 2 in ResNet [78], we performed the downsampling layer
of each block.

The primary objective of PhysMLE is to estimate $N=4$ physiological signals
(i.e., HR, BVP, SpO2, and RR) simultaneously. After the STMaps, $X$ traverses
through the multiple PhysMLE basic blocks and acquires the task-agnostic high-
level feature $s^{{}^{\prime}}$. Then, we inputted this feature into $N=4$
task-specific EFRouters. These routers are responsible for generating task-
specific features
$s^{{}^{\prime\prime}}=\\{s^{{}^{\prime\prime}}_{hr},s^{{}^{\prime\prime}}_{bvp},s^{{}^{\prime\prime}}_{spo},s^{{}^{\prime\prime}}_{rr}\\}$,
wherein each task head is dedicated to the regression of the corresponding
physiological signal.

Experts Orthogonalization Furthermore, the pivotal consideration in resolving
multi-task remote physiological measurement is to identify specific distinct
features associated with each physiological signal. Thus, motivated by [80],
we noticed that directly adding the non-orthogonal columns to each other would
lead to superimposition of their information about the individual
physiological signal, making it challenging to disentangle features for each
task. Being different from the multi-scale MoE [75], we designed an Orthogonal
Experts Merge (OEM) regularization over each expert’s weights in the same
layer to further minimize computational overhead, where the Spectral
Restricted Isometry Property Regularization (SRIP) [81] was instantiated. For
instance, given $M$ basic blocks in the backbone network and $T$ PhysMLE
layers in each basic block, if there is $K=2$ expert weights
$\mathbf{W_{1}},\mathbf{W_{2}}\in\mathbb{R}^{d_{in}\times d_{out}}$
initialized in each PhysMLE layer, the $\mathcal{L}_{OEM}$ was given by:

$\displaystyle
u=(\mathbf{W_{1}}\mathbf{W_{2}}^{T}-(I-\text{{diag}}))\epsilon,$ (2)
$\displaystyle v=(\mathbf{W_{1}}\mathbf{W_{2}}^{T}-(I-\text{{diag}}))u,$
$\displaystyle\mathcal{L}_{OEM}=\frac{1}{M\times T}\sum_{i=1}^{M\times
T}\frac{{\left\|v_{i}\right\|_{2}}}{{\left\|u_{i}\right\|_{2}+1e^{-12}}}.$

The $I\in\mathbb{R}^{d_{in}\times d_{in}}$ is the a matrix of ones,
$\text{{diag}}\in\mathbb{R}^{d_{in}\times d_{in}}$ is an identity matrix, and
$\epsilon\in\mathbb{R}^{d_{in}\times 1}$ is a random noise vector sampled from
Gaussian distribution.

Figure 3: Illustration of how STMap generation and data augmentation
performed. After the 256-frame facial video (highlighted with the red box) is
aligned and compressed to an STMap, we shifted the sliding window to the next
50 frames to get the augmented STMap (highlighted with the orange box).
Meanwhile, for the augmented STMap, the row-wise shuffle was applied.

###  3.3 Shared Semantic Feature Consistent

Previous studies have highlighted the presence of analogous waveforms in
short-term rPPG signals captured from different facial regions [82, 33, 83].
This observation serves as a crucial piece of prior knowledge in the realm of
remote physiological measurement. While these analogous waveforms have been
extensively utilized for heart rate measurements [28, 84], we propose that
this knowledge can be leveraged to enhance the generalizability of multi-task
physiological measurement, specifically, through regularization and data
augmentation.

First, inspired by [85], we applied the spatio-temporal augmentation following
our prior work [21]. As shown in Figure 3, after shuffling each row of STMap
generated from RGB facial video [37] and sliding the time window with the size
of 30 frames, we obtained the original input $X$ and augmented one
$X^{{}^{\prime}}$ corresponding to specific physiological states $Y$ and
$Y^{{}^{\prime}}$, respectively. Then, being different from previous attempts
[28, 84], we maintained the spatial consistency over the aggregated features
outputted by each layer of PhysMLE to encourage consistent predictions across
different facial regions. Specifically, take the MLE Conv layer as an example,
for the feature matrix $s\in\mathbb{R}^{B\times C_{j}\times W_{j}\times
L_{j}}$ decoded by the j-th PhysMLE basic block, we regularized it with the
Semantically Spatial Consistent loss as follows:

$\mathcal{L}_{Spatial}=-\frac{1}{B\times C_{j}\times
W_{j}}\sum_{i=1}^{B}\sum_{c=1}^{C_{j}}\sum_{w=1}^{W_{j}}\operatorname{sim}(s_{icw},s_{icw^{*}}).$
(3)

where $w^{*}\in random\\{1,2,...,W_{j}\\}$, and $\operatorname{sim}$ indicates
the semantic similarity measurement, which is instantiated with cosine
similarity. Then, we took the sum $\mathcal{L}_{Spatial}$ of all PhysMLE
layers as the final $\mathcal{L}_{Spatial}$. Moreover, after getting estimated
$\hat{Y}$ and $\hat{Y}^{{}^{\prime}}$ based on $X$ and $X^{{}^{\prime}}$, we
applied a $\mathcal{L}_{Temporal}$ to penalize outputs, where
$\hat{Y}^{{}^{\prime}}$ was deviated from $\hat{Y}$ by more than the hyper-
parameter $\delta$. This loss function aims to regularize the model following
the temporal smoothness of physiological features within a short time
interval.

$\mathcal{L}_{Temporal}=\frac{1}{B}\sum_{i=1}^{B}\sum_{j=1}^{N}\left|\hat{Y}^{{}^{\prime}}_{ij}-\hat{Y}_{ij}\right|,where\
\left|\hat{Y}^{{}^{\prime}}_{ij}-\hat{Y}_{ij}\right|\geq\delta.$ (4)

###  3.4 Physiological Priors for Imbalance Target

Referring to Table I, we can notice that only ground-truth HR is widely
provided, particularly under our MSSDG setting, while other vital signs (i.e.,
HR, SpO2, and RR) are partially offered. To overcome the imbalance
optimization goal when jointly training from multiple datasets, we leverage
the prior knowledge from the physiology field to provide valid gradients from
tasks that lack sufficient labels.

Priors for SpO2 Lack of supervisable SpO2 labeling is present in large-scale
multi-task remote physiological monitoring based on publicly available
datasets. Under our setting, only PURE and VIPL-HR provided SpO2 labels.
Particularly when conducting MSSDG protocol, there will be one dataset with
SpO2 labels that can be leveraged to supervise the training of the SpO2
estimation head. Further, despite the existence of some correlation between
SpO2 and HR, RR, and BVP, no direct causal relationship is there. Therefore,
we proposed a novel data augmentation strategy based on prior knowledge of
remote SpO2 measurement.

According to [10, 86], the principle of remote SpO2 measurement with RGB
camera relies on Eq. (5). Where $AC_{RED},AC_{BLUE}$ represent the standard
deviations of the blue and red color channels, respectively, while
$DC_{RED},DC_{BLUE}$ represent the mean of the blue and red color channels.
The coefficients $\mathbf{W_{a}}$ and $\mathbf{W_{b}}$ were determined by
learning the best-fit linear correlation between the ratios of the red and
blue channels and the ground-truth SpO2.

$y_{spo}=\mathbf{W_{a}}-\mathbf{W_{b}}\frac{(AC_{RED}/DC_{RED})}{(AC_{BLUE}/DC_{BLUE})}.$
(5)

Back to our work, given the forward process of SpO2 estimation, we can argue
that: (1) the ratio $\frac{AC_{RED}/DC_{RED}}{AC_{BLUE}/DC_{BLUE}}$ can be
extracted by some of the experts in PhysMLE basic blocks, and it can be
formulated in $s^{{}^{\prime\prime}}_{spo}$ after the SpO2-specific EFRouter;
(2) through training, the $\mathbf{W_{spo}}$ of the SpO2 estimation head
should be able to represent the $\mathbf{W_{a}}$ and $\mathbf{W_{b}}$, and
reformulate Eq. (5) to $y_{spo}=\mathbf{W_{spo}}s^{{}^{\prime\prime}}_{spo}$.
Based on this, we constructed augmentation samples for SpO2 estimation through
transformation in the ratio part. Specifically, given a
$s^{{}^{\prime\prime}}_{spo}$ and its corresponding ground-truth $y_{spo}$, we
defined a set of augmentation rate $\tau\in\mathrm{T}$, which is the set of
transformations that change $s^{{}^{\prime\prime}}_{spo}$ with an arbitrary
rate that is feasible under the Nyquist sampling theorem. We dot product
$s^{{}^{\prime\prime}}_{spo}$ by $\tau$ to obtain a augmentation feature.
Theoretically, the well-trained $\mathbf{W_{spo}}$ ought to map $\tau\cdot
s^{{}^{\prime\prime}}_{spo}$ to the estimation that is also $\tau$ times to
$y_{spo}$. Such augmentation strategy effectively changes the underlying
color-channel ratios, thus creating different samples from the same individual
but with different SpO2. In practice, we limit the augmented SpO2 range to be
within [80%, 100%], ensuring the augmented SpO2 is not out of the normal range
[87]. Given this, we designed the $\mathcal{L}_{ASp}$ as follows:

$\mathcal{L}_{ASp}=\frac{1}{B}\sum^{B}\sum^{\mathrm{T}}_{\tau=1}\left|\mathbf{W_{spo}}\tau\cdot
s^{{}^{\prime\prime}}_{spo}-\tau\cdot y_{spo}\right|.$ (6)

Figure 4: Example elaboration of $\mathcal{L}_{BR}$ calculation. For those
samples with ground-truth BVP signals but without RR, we first detect R-R
intervals and conduct frequency and PSD analysis to extract $y^{bvp}_{rr}$
from the high-frequency part. Then, we use $y^{bvp}_{rr}$ to supervise the
learning of RR estimation according to Eq. (7).

Priors for RR As the respiratory process affects the autonomic nervous system,
it further leads to periodic changes in the heart rate, which is defined as
respiratory sinus arrhythmia (RSA) [88]. Meanwhile, it is believed that the
High Frequency (HF) component in HRV (i.e., 0.15-0.4Hz) reflects the
parasympathetic influence on the heart and is significantly associated with
RSA [29]. Given this, previous works [34, 50] assessed the RR by using the
peak location of the HF component from the power spectrum densities (PSD) of
the R-R intervals. Similar to $\mathcal{L}_{HB}$, we propose the
$\mathcal{L}_{BR}$ to augment the training of unsupervised RR samples with
labels calculated from ground-truth BVP. However, most of the current rPPG
works do not continuously output long-period BVP signals during training due
to computational efficiency and instantaneous HR estimation, while the precise
HRV calculation usually requires over 300s [89]. Thus, refer to Figure 4,
different from $\mathcal{L}_{HB}$, we softly constraint the difference between
the $\hat{y}_{rr}$ and the extracted $y_{rr}^{bvp}$ from ground-truth BVP
should be within $\delta$.

$\mathcal{L}_{BR}=\frac{1}{B}\sum^{B}\left|\hat{y}_{rr}-y_{rr}^{bvp}\right|,where\
\left|\hat{y}_{rr}-y_{rr}^{bvp}\right|\geq\delta.$ (7)

Priors for HR and BVP Specifically, the basic principle of rPPG is that when
pulse waves triggered by cardiac activity reach the surface of the skin, they
cause small color changes that can be captured by an RGB camera [40], and BVP
signals, which reflect the shape and intensity of the pulse waves, contain
rich information about cardiac activity. Thus, HR can be directly calculated
by the number of peaks in the BVP signal and can be calculated by peak
detection or spectral analysis. Therefore, given the HR label, when there is
no BVP label provided for some samples, we reinforce the learning of the BVP
head by constraining the HR calculated from the estimated BVP signal to be
consistent with the ground-truth HR. For the design of the optimization
objective $\mathcal{L}_{HB}$, we deliberately used the peak detection to
compute HR from estimated BVP, which is beneficial to the computational
efficiency on the one hand and on the other hand, such sensitivity-to-noise
method can force the network to inference the BVP signals that are as free of
noise as possible during the training process.

$\mathcal{L}_{HB}=\frac{1}{B}\sum^{B}\left|\hat{y}_{hr}-FindPeak(\hat{y}_{bvp})\right|.$
(8)

###  3.5 Optimization and Inference

In this section, we provide a detailed explanation of the optimization process
of PhysMLE. To enhance clarity, we present the entire procedure in Algorithm
1.

To facilitate the estimation of HR, SpO2, and RR, we formulate three loss
functions, denoted as $\mathcal{L}_{HR}$, $\mathcal{L}_{SpO2}$, and
$\mathcal{L}_{RR}$, respectively. These loss functions employed the L1 loss to
predict the mean values of the respective signal distributions. To learn the
BVP signal feature, we utilized the negative Pearson’s correlation coefficient
as the basis for the $\mathcal{L}_{BVP}$. During the joint training process,
we incorporated an adaptation factor $\lambda$ [21] to suppress meaningless
regularizations during the early iterations. By combining all the individual
loss functions, we formulated the overall loss function with trade-off
parameters $p_{i}$ for each specific regularization as:

$\displaystyle\mathcal{L}(X,Y)$
$\displaystyle=\mathcal{L}_{BVP}+\lambda(p_{1}\mathcal{L}_{HR}+p_{2}\mathcal{L}_{SpO2}+p_{3}\mathcal{L}_{RR}$
(9)
$\displaystyle+p_{4}\mathcal{L}_{OEM}+p_{5}\mathcal{L}_{Spatial}+p_{6}\mathcal{L}_{Temporal}$
$\displaystyle+p_{7}\mathcal{L}_{HB}+p_{8}\mathcal{L}_{BR}+p_{9}\mathcal{L}_{ASp}).$

Algorithm 1 Train Process of PhysMLE

1:

2:Initialize backbone network;

3:Replace basic blocks with PhysMLE with $\alpha,r,K$;

4:freeze part of parameters $\theta_{f}$ and activate $\theta_{t}$;

5:Generated STMaps $X$ from videos;

6:The ground-truth signals $Y$;

7:The number of iteration $N_{iter}$;

8:Hyper-parameters $\tau,\delta,\gamma,p$.

9:

10:for i = 1 to $N_{iter}$ do

11: Conduct data augmentation $X^{{}^{\prime}}$ from to $X$ as shown in Figure
3;

12: Encode $X,X^{{}^{\prime}}$ to $s_{0}$ by the first non-PhysMLE layer;

13: Forward $s_{0}$ in $M$ PhysMLE basic blocks;

14: Compute $\mathcal{L}_{OEM}$ based on $K$ expert weights in each PhysMLE
layers by Eq. (2);

15: Compute $\mathcal{L}_{Spatial}$ based on the outputted feature of each
PhysMLE basic block by Eq. (3);

16: Input the task-agnostic high-level feature $s^{{}^{\prime}}$ into each
task head and corresponding EFRouter to predict
$\hat{Y},\hat{Y}^{{}^{\prime}}$;

17: Compute $\mathcal{L}_{Temporal}$ according to Eq.(4);

18: Compute $\mathcal{L}_{HB},\mathcal{L}_{RB},\mathcal{L}_{ASp}$ according to
Eq.(6)(7)(8) respectively;

19: Jointly update $\theta_{t}$ with the gradient from Eq.(9);

20:end for.

Figure 5: Sample video frames captured by different webcams in HCW. Figure 6:
BVP waveform examples and statistical distribution of HR and RR in HCW. TABLE
III: Comparison Experiment Results on MSSDG Protocol I.

|  | PURE | VIPL-HR
---|---|---|---
|  | HR | SpO2 | HR | SpO2
Method Type | Method | MAE↓ | RMSE↓ |  p↑ | MAE↓ | RMSE↓ |  p↑ | MAE↓ | RMSE↓ |  p↑ | MAE↓ | RMSE↓ |  p↑
Single-task | GREEN [40] | 10.32 | 14.27 | 0.52 | — | — | — | 12.18 | 18.23 | 0.25 | — | — | —
Traditional | CHROM [41] | 9.79 | 12.76 | 0.37 | — | — | — | 11.44 | 16.97 | 0.28 | — | — | —
| POS [42] | 9.82 | 13.44 | 0.34 | — | — | — | 14.59 | 21.26 | 0.19 | — | — | —
| ARM-SpO2 [10] | — | — | — | 3.25 | 3.51 | 0.20 | — | — | — | 18.11 | 20.23 | 0.01
Single-task | RhythmNet [37] | 7.89 | 10.66 | 0.62 | — | — | — | 10.97 | 14.86 | 0.31 | — | — | —
Deep | BVPNet [13] | 7.44 | 10.51 | 0.64 | — | — | — | 10.62 | 14.1 | 0.32 | — | — | —
| Dual-GAN [90] | 7.65 | 10.86 | 0.63 | — | — | — | 10.77 | 14.81 | 0.31 | — | — | —
| PhysFormer++ [45] | 7.16 | 10.14 | 0.65 | — | — | — | 9.91 | 14.05 | 0.37 | — | — | —
| EfficientPhys [91] | 8.21 | 11.30 | 0.57 | — | — | — | 11.16 | 14.95 | 0.30 | — | — | —
| rSPO [54] | — | — | — | 3.41 | 3.60 | 0.18 | — | — | — | 14.36 | 15.54 | 0.02
Multi-task | AD-R [68] | 7.52 | 10.79 | 0.6 | 3.15 | 4.26 | 0.19 | 9.81 | 13.99 | 0.37 | 14.78 | 17.98 | 0.04
DG | VREx-R [92] | 8.00 | 12.87 | 0.66 | 2.76 | 3.42 | 0.20 | 10.79 | 15.13 | 0.32 | 14.83 | 18.40 | 0.03
| GroupDRO-R [93] | 8.98 | 10.52 | 0.62 | 3.20 | 4.22 | 0.19 | 15.65 | 17.8 | 0.22 | 16.30 | 21.48 | 0.01
| NEST-R [21] | 6.57 | 10.03 | 0.67 | 3.57 | 4.17 | 0.19 | 10.44 | 14.69 | 0.33 | 15.61 | 18.78 | 0.03
| HSRD-R [22] | 5.91 | 8.39 | 0.71 | 2.80 | 3.98 | 0.20 | 10.04 | 14.33 | 0.36 | 14.83 | 19.41 | 0.06
| AD-T [68] | 8.52 | 10.36 | 0.54 | 2.01 | 2.34 | 0.20 | 11.10 | 15.35 | 0.26 | 11.34 | 12.00 | 0.09
| VREx-T [92] | 9.29 | 11.22 | 0.48 | 2.48 | 2.99 | 0.19 | 11.72 | 16.11 | 0.26 | 14.68 | 15.12 | 0.03
| GroupDRO-T [93] | 9.46 | 11.36 | 0.49 | 2.71 | 2.74 | 0.18 | 10.74 | 14.78 | 0.29 | 13.45 | 13.92 | 0.05
| NEST-T [21] | 7.52 | 9.66 | 0.55 | 2.13 | 2.25 | 0.19 | 10.62 | 14.23 | 0.30 | 13.28 | 13.12 | 0.06
| HSRD-T [22] | 7.31 | 9.25 | 0.58 | 2.26 | 2.39 | 0.19 | 10.36 | 14.19 | 0.33 | 12.25 | 12.66 | 0.08
Ours | Base-R [78] | 8.74 | 10.11 | 0.56 | 3.93 | 4.60 | 0.18 | 9.96 | 14.24 | 0.36 | 19.05 | 19.25 | 0.01
| Base-T [94] | 8.81 | 10.21 | 0.55 | 2.45 | 2.85 | 0.19 | 11.36 | 15.40 | 0.25 | 14.61 | 15.76 | 0.03
| PhysMLE-R | 5.43 | 7.88 | 0.77 | 2.19 | 3.23 | 0.22 | 7.53 | 10.29 | 0.62 | 10.94 | 14.81 | 0.09
| PhysMLE-T | 6.62 | 8.72 | 0.67 | 1.86 | 2.34 | 0.25 | 9.59 | 12.74 | 0.37 | 9.88 | 10.77 | 0.10

Notes: In this and following tables, the bold texts show the best result
within each column.

TABLE IV: Comparison Experiment Results on MSSDG Protocol I. |  | V4V | HCW
---|---|---|---
|  | HR | RR | HR | RR
Method Type | Method | MAE↓ | RMSE↓ |  p↑ | MAE↓ | RMSE↓ |  p↑ | MAE↓ | RMSE↓ |  p↑ | MAE↓ | RMSE↓ |  p↑
Single-task | GREEN [40] | 15.64 | 21.43 | 0.06 | — | — | — | 14.36 | 20.01 | 0.04 | — | — | —
Traditional | CHROM [41] | 14.92 | 19.22 | 0.08 | — | — | — | 14.66 | 19.81 | 0.05 | — | — | —
| POS [42] | 12.67 | 18.11 | 0.06 | — | — | — | 12.37 | 17.61 | 0.11 | — | — | —
| ARM-RR [10] | — | — | — | 6.33 | 6.90 | 0.01 | — | — | — | 8.02 | 8.66 | 0.01
Single-task | RhythmNet [37] | 11.31 | 15.37 | 0.30 | — | — | — | 10.10 | 11.73 | 0.35 | — | — | —
Deep | BVPNet [13] | 11.16 | 15.10 | 0.31 | — | — | — | 9.52 | 12.01 | 0.57 | — | — | —
| Dual-GAN [90] | 10.02 | 14.23 | 0.32 | — | — | — | 9.44 | 11.61 | 0.58 | — | — | —
| PhysFormer++ [45] | 9.53 | 12.96 | 0.37 | — | — | — | 8.12 | 10.05 | 0.51 | — | — | —
| EfficientPhys [91] | 11.87 | 16.02 | 0.27 | — | — | — | 10.02 | 11.79 | 0.34 | — | — | —
Multi-task | MTTS-CAN [19] | 12.06 | 15.01 | 0.11 | 6.44 | 7.80 | 0.01 | 13.38 | 15.21 | 0.21 | 9.33 | 15.86 | 0.01
Deep | BigSmall [20] | 11.88 | 14.60 | 0.29 | 6.36 | 7.68 | 0.02 | 12.05 | 14.11 | 0.25 | 8.42 | 14.30 | 0.01
Multi-task | AD-R [68] | 10.91 | 13.72 | 0.31 | 3.06 | 3.83 | 0.14 | 9.86 | 11.14 | 0.41 | 7.54 | 13.84 | 0.04
DG | VREx-R [92] | 10.96 | 13.78 | 0.31 | 3.23 | 4.23 | 0.13 | 8.23 | 10.93 | 0.49 | 7.44 | 12.56 | 0.06
| GroupDRO-R [93] | 10.58 | 12.92 | 0.32 | 3.98 | 4.47 | 0.12 | 9.11 | 11.39 | 0.44 | 8.28 | 13.17 | 0.03
| NEST-R [21] | 9.62 | 11.05 | 0.38 | 3.46 | 4.50 | 0.12 | 9.76 | 11.35 | 0.41 | 7.31 | 12.59 | 0.04
| HSRD-R [22] | 9.18 | 10.83 | 0.42 | 3.11 | 4.01 | 0.13 | 9.35 | 10.02 | 0.47 | 7.28 | 12.92 | 0.04
| AD-T [68] | 10.95 | 13.47 | 0.31 | 5.49 | 6.04 | 0.05 | 8.75 | 10.71 | 0.46 | 8.25 | 9.95 | 0.03
| VREx-T [92] | 11.21 | 13.61 | 0.25 | 6.68 | 8.22 | 0.02 | 8.41 | 10.32 | 0.49 | 10.43 | 14.68 | 0.02
| GroupDRO-T [93] | 11.17 | 13.57 | 0.30 | 5.59 | 6.10 | 0.05 | 9.10 | 11.18 | 0.39 | 8.05 | 11.92 | 0.03
| NEST-T [21] | 10.42 | 12.79 | 0.31 | 4.75 | 5.29 | 0.07 | 8.39 | 10.30 | 0.50 | 9.23 | 10.42 | 0.03
| HSRD-T [22] | 10.32 | 13.40 | 0.33 | 4.98 | 5.20 | 0.07 | 8.30 | 10.27 | 0.49 | 8.44 | 10.21 | 0.03
Ours | Base-R [78] | 11.64 | 14.05 | 0.28 | 4.15 | 4.87 | 0.08 | 10.18 | 11.30 | 0.36 | 8.22 | 12.25 | 0.04
| Base-T [94] | 11.30 | 14.04 | 0.26 | 4.81 | 6.32 | 0.07 | 10.24 | 12.96 | 0.35 | 9.74 | 12.94 | 0.05
| PhysMLE-R | 8.12 | 10.56 | 0.51 | 3.02 | 3.66 | 0.14 | 7.13 | 10.20 | 0.61 | 5.90 | 8.84 | 0.10
| PhysMLE-T | 8.96 | 10.90 | 0.50 | 3.41 | 4.91 | 0.12 | 7.34 | 9.46 | 0.60 | 6.55 | 9.48 | 0.06

##  4 Experiment

###  4.1 Dataset

####  4.1.1 Our Collected Dataset

Previous public rPPG datasets have played a significant role in advancing the
field of remote physiological measurement. However, these datasets have
certain limitations that restrict their application in real-world scenarios.
One key limitation is the lack of consideration for the influence of human
states on physiological signs. As summarized in Table I, many existing
datasets ignore the dynamic nature of human physiological signs under varying
cognitive or emotional states [95, 96]. Although a few datasets satisfied the
demand of special human states (e.g., the stress in UBFC-Phys [97], emotions
in MAHNOB-HCI [30]), they were recorded using specialist imaging devices. This
restricts the generalizability of the models trained on these datasets to
real-world situations where a wider range of webcam types are commonly
involved. Therefore, we built a dataset collected in the real workspace, named
HCW, to promote the remote physiological measurement under the high cognitive
workload scenario. In total, 41 participants with 27 males and 14 females
(aged between 22 and 31) engaged in the collection of HCW, where subjects were
induced to the high cognitive workload status.

Specifically, as previous datasets already cover diverse environmental
illumination and subject pose, we designed our data to have variations in
cognitive workload levels. The data collection was conducted in a quite room
with stable lighting conditions. In order to replicate daily uncontrollable
application scenarios, we first asked subjects to sit casually as they do in
their own place, then encouraged them to perform daily work activities (e.g.,
literature review and paper reading) with their own laptop. Then, during the
100-minute observation, we assessed their cognitive workload using the NASA-
TLX with 25-minute constant intervals. Whenever they reported a NASA-TLX over
70 [95], we deployed physiological signal sensors and opened the in-front
webcam on their laptop to record the video of next 25 minutes at 30 fps. Some
examples captured by different webcams can be found in Figure 5. The
integrated platform PhysioLAB was utilized for simultaneous physiological
signal collection. The BVP signals were recorded with 1000Hz. Golden HR and RR
were calculated from ECG and respiration signals collected by the SMD
electrocardiograph and respiratory belt transducers of PhysioLAB.

Then, we performed noise removal and downsampled the original BVP signals to
30HZ to match the frame rate of the videos. For the purpose of enhancing the
accessibility and usability of our dataset, we preprocessed the raw data and
converted it into a MAT file format, which is compatible with both Matlab and
Python. Some BVP examples and distributions of HR and RR are in Figure 6. The
videos were filmed with a resolution of $2160\times 1080$ pixels. Using the
face detection and alignment approaches in [37], each frame of aligned facial
videos is with the size of $128\times 128$. The experiment was approved by the
Hong Kong University of Science and Technology’s Human and Artefacts Research
Ethics Committee (protocol number: HREP-2023-0159), and We will share both the
raw data (i.e., original video and raw physiological signals) and processed
data with the research community.

####  4.1.2 Other Datasets

In addition to HCW, we carefully selected five other datasets to evaluate the
performance of our PhysMLE within the context of large-scale MSSDG protocol.
The chosen datasets encompassed a range of motion, camera, and lighting
conditions, presenting a comprehensive evaluation scenario.

  * •

UBFC-rPPG [35] consists of 42 facial videos captured under both sunlight and
indoor illumination conditions. Ground-truth BVP signals and HR values in this
dataset were collected using CMS50E, providing reliable reference
measurements.

  * •

BUAA [36] aims to evaluate performance under different illumination
conditions. Only data with illumination levels of 10 lux or greater are
included, as underexposed images require specialized algorithms beyond the
scope of this study.

  * •

PURE [31] comprises 60 RGB videos featuring 10 subjects engaged in six
different activities. To align the BVP signals with the videos, the signals
are down-sampled from 60 to 30 fps using cubic spline interpolation.

  * •

VIPL-HR [37] offers nine distinct scenarios captured using three RGB cameras,
reflecting various illumination conditions and movement levels. To address the
issue of unstable video frame rates, we normalize the STMap to 30 fps using
cubic spline interpolation, incorporating insights from prior works [90, 45].

  * •

V4V [38] is designed to capture significant changes in physiological
indicators. It includes data from ten tasks.

Notably, as V4V only had HR values but without BVP signal labels and the BVP
signals and videos in VIPL-HR are not temporally aligned, the
$\mathcal{L}_{BVP}$ are inapplicable for these two datasets.

###  4.2 Implementation Details

####  4.2.1 Training Settings

The whole work was implemented by the Pytorch framework. For the data
preparation, we followed the procedures of STMap generation in [21]. The STMap
of each dataset was sampled every 10 steps with a time window of 256. To
comprehensively evaluate the effectiveness of PhysMLE adapted to different
backbone networks, we created two versions of PhysMLE: PhysMLE-R adopted from
ResNet-18 [78] and PhysMLE-T using Vision Transformer pretrained with ImageNet
[94], and their core feature extraction layers were replaced by the PhysMLE
layer. For PhysMLE-ResNet, the original STMap $\mathbb{R}^{25\times 256\times
3}$ was resized to $\mathbb{R}^{64\times 256\times 3}$, and
$\mathbb{R}^{224\times 224\times 3}$ for PhysMLE-T. For HR, SpO2, and RR
regression, one full connection layer was initialized for each task. At the
same time, the BVP head consists of four blocks, which followed the design in
[90]. The whole experiments were conducted on an RTX A6000. The batch size $B$
and iteration number $N_{iter}$ were 60 and 20000. Hyper-parameter $\tau$ and
$\delta$ were set to 0.1 and 5. The number of experts $K$ was 3. The other two
hyper-parameters $\alpha$ and $r$ controlling the size of LoRA weights were
selected from [8, 16, 32] depending on performance. Adam optimizer was used
for training, with a learning rate of 0.00001.

####  4.2.2 Evaluation Settings

In testing, following existing works [21, 22], we used mean absolute error
(MAE), root mean square error (RMSE), and Pearson’s correlation coefficient
(p) to assess estimated physiological indicators. Furthermore, for the HRV
measurement, we used normalized low frequency (LFnu), normalized high
frequency (HFnu), and LF/HF (i.e., the ratio between low-frequency power and
high-frequency power).

The MSSDG protocol was designed into two parts: (1) Protocol I, this protocol
is implemented in a cross-domain evaluation scenario by dividing six datasets
into two groups: five datasets for training (source domain) and then testing
on the other dataset (target domain); (2) Protocol II, this protocol is
implemented in an intra-dataset multi-task evaluation scenario based on large
challengable datasets. Specifically, two datasets (i.e., VIPL-HR and HCW) with
labels not limited to BVP and HR were used as training and testing sets
simultaneously. It is worth noting that, to avoid unfair comparison, we
removed the action units prediction head of BigSmall [20] and the GRU
mechanism from Rhythmnet [37].

TABLE V: HRV and HR estimation results on MSSDG protocol I. |  | LFnu | HFnu | LF/HF | HR-(bpm)
---|---|---|---|---|---
Target | Method | MAE↓ | RMSE↓ |  p↑ | MAE↓ | RMSE↓ |  p↑ | MAE↓ | RMSE↓ |  p↑ | MAE↓ | RMSE↓ |  p↑
UBFC | GREEN [40] | 0.236 | 0.284 | 0.092 | 0.236 | 0.284 | 0.092 | 0.700 | 0.951 | 0.047 | 8.018 | 9.178 | 0.363
| CHROM [41] | 0.222 | 0.281 | 0.070 | 0.222 | 0.281 | 0.070 | 0.671 | 1.054 | 0.105 | 7.229 | 8.922 | 0.512
| POS [42] | 0.236 | 0.286 | 0.136 | 0.236 | 0.286 | 0.136 | 0.652 | 0.954 | 0.135 | 7.354 | 8.040 | 0.492
| Rhythmnet [37] | 0.163 | 0.185 | 0.166 | 0.163 | 0.185 | 0.166 | 0.242 | 0.329 | 0.278 | 6.023 | 7.131 | 0.707
| Dual-GAN [90] | 0.189 | 0.092 | 0.157 | 0.189 | 0.092 | 0.157 | 0.411 | 0.366 | 0.243 | 6.552 | 7.621 | 0.668
| PhysFormer++ [45] | 0.084 | 0.113 | 0.182 | 0.084 | 0.113 | 0.182 | 0.323 | 0.365 | 0.306 | 5.635 | 7.258 | 0.769
| HSRD-R [22] | 0.062 | 0.078 | 0.179 | 0.062 | 0.078 | 0.179 | 0.238 | 0.312 | 0.310 | 5.113 | 6.211 | 0.776
| HSRD-T [22] | 0.173 | 0.267 | 0.166 | 0.173 | 0.267 | 0.166 | 0.321 | 0.558 | 0.264 | 6.176 | 7.344 | 0.672
| Base-R [78] | 0.106 | 0.280 | 0.211 | 0.106 | 0.280 | 0.211 | 0.297 | 0.843 | 0.281 | 5.790 | 7.906 | 0.733
| Base-T [94] | 0.179 | 0.283 | 0.132 | 0.179 | 0.283 | 0.132 | 0.403 | 0.989 | 0.262 | 6.553 | 8.007 | 0.652
| PhysMLE-R | 0.056 | 0.073 | 0.248 | 0.056 | 0.073 | 0.248 | 0.223 | 0.296 | 0.345 | 4.353 | 5.081 | 0.850
| PhysMLE-T | 0.063 | 0.082 | 0.201 | 0.063 | 0.082 | 0.201 | 0.233 | 0.301 | 0.315 | 4.927 | 5.746 | 0.831
BUAA | GREEN [40] | 0.347 | 0.395 | 0.087 | 0.347 | 0.395 | 0.087 | 0.645 | 0.863 | 0.092 | 5.823 | 7.988 | 0.562
| CHROM [41] | 0.379 | 0.324 | 0.068 | 0.379 | 0.324 | 0.068 | 0.681 | 0.884 | 0.072 | 6.093 | 8.294 | 0.517
| POS [42] | 0.320 | 0.376 | 0.096 | 0.320 | 0.376 | 0.096 | 0.628 | 0.842 | 0.113 | 5.041 | 7.120 | 0.637
| Rhythmnet [37] | 0.242 | 0.389 | 0.277 | 0.242 | 0.273 | 0.277 | 0.634 | 0.891 | 0.274 | 4.630 | 8.102 | 0.774
| Dual-GAN [90] | 0.218 | 0.385 | 0.281 | 0.218 | 0.385 | 0.281 | 0.626 | 0.860 | 0.263 | 4.309 | 7.871 | 0.784
| PhysFormer++ [45] | 0.201 | 0.394 | 0.289 | 0.201 | 0.394 | 0.289 | 0.599 | 0.926 | 0.282 | 4.021 | 8.711 | 0.790
| HSRD-R [22] | 0.207 | 0.371 | 0.289 | 0.207 | 0.371 | 0.289 | 0.612 | 0.879 | 0.286 | 4.176 | 7.502 | 0.814
| HSRD-T [22] | 0.189 | 0.172 | 0.313 | 0.189 | 0.172 | 0.313 | 0.583 | 0.697 | 0.320 | 3.073 | 6.912 | 0.866
| Base-R [78] | 0.324 | 0.411 | 0.269 | 0.324 | 0.411 | 0.269 | 0.642 | 0.917 | 0.269 | 4.914 | 8.614 | 0.758
| Base-T [94] | 0.385 | 0.498 | 0.195 | 0.385 | 0.498 | 0.195 | 0.723 | 0.989 | 0.143 | 5.672 | 10.490 | 0.679
| PhysMLE-R | 0.131 | 0.156 | 0.362 | 0.131 | 0.156 | 0.362 | 0.501 | 0.635 | 0.437 | 2.162 | 3.580 | 0.922
| PhysMLE-T | 0.148 | 0.169 | 0.291 | 0.148 | 0.169 | 0.291 | 0.562 | 0.698 | 0.299 | 3.456 | 7.038 | 0.834

###  4.3 Comparison Experiment over Protocol I

In this section, we present and discuss the evaluation results based on six
datasets following Protocol I. Notes that, for single-task baselines, only one
type of target label was used for training. The Base-R/T is a multi-task
variant that is built by one backbone network with four task estimation heads
like PhysMLE, and all labels were used for training regardless of the types of
labels of the domains. Those compared DG methods were also constructed based
on the Base structures.

####  4.3.1 Multi-task evaluation

Firstly, we present results when PURE and VIPL-HR are target domains (as shown
in Table III), and compared them against multiple baseline methods, including
single-task methods and DG methods. It indicates that although traditional
SpO2 methods [10] can compete with some single-task DL-based method [54] and
DG methods [68, 92, 93] integrated with ResNet-18 on relatively simple
scenarios (i.e., PURE), DG methods mostly outperformed all traditional
approaches in both HR and SpO2 estimation. This suggests traditional single-
task approaches struggle to handle intricate motions and scenarios compared to
deep learning. Referring to Table IV that contains results of V4V and HCW,
state-of-the-art (SOTA) DL-based multi-task methods [19, 20] did not
outperform single-task traditional method [10] in RR estimation notably.

Nevertheless, as a multi-task model, PhysMLE still outperformed all baselines
across all tasks. It shows that PhysMLE can achieve a promising balance among
different tasks. Besides, some interesting findings were identified when we
integrated different backbones into DG methods and our proposals. We notice
that, different backbone networks led to performance disparity even when
combined with the same method. Particularly, despite PhysMLE-R performing
better than PhysMLE-T in most cases, PhysMLE-T led to higher accuracy in the
SpO2 estimation compared to PhysMLE-R. A similar phenomenon was observed for
those DG approaches. Thus, we assume that the ViT architecture is better at
handling complex transformations between facial color signals and SpO2 values
than CNN. Additionally, we found not all DG methods can be comparable to the
SOTA single-task method [45] in HR estimation. It suggests that previous DG
methods, even for those designed for rPPG [21, 22], are still not good at
imbalance learning of multiple tasks. Lastly, soft-parameter sharing PhysMLE
performed notably better than the multi-task base models that followed the
hard-parameter sharing paradigm and performed comparatively to multi-task
baselines [19, 20]. The computational cost analysis is provided in the
following sections.

####  4.3.2 HRV evaluation

For two single-task datasets (i.e., UBFC-rPPG and BUAA), we used the HRV index
(LFnu, HFnu, and LF/HF) and HR evaluation to compare the quality of the
predicted BVP signal. As shown in Table V, similar findings from the Multi-
task evaluation section are also identified in this part. Particularly,
compared to single-task baselines, getting trained in multi-task labels did
not bring negative effects to PhysMLE, which still notably outperformed
traditional methods and the previous single-task baseline [37, 90, 45]. This
suggests that PhysMLE can explore shared information across multiple task
labels effectively to enhance generalizability in a single-task target domain.
Additionally, the best results of PhysMLE in all HRV metrics underscore the
capability of PhysMLE to yield high-quality LF and HF (Hz) prediction. This
can be beneficial to applications in emotion recognition, and healthcare [1,
2].

Figure 7: MAE of applying different regularizations over PhysMLE-R.

###  4.4 Ablation Study over Protocol I

####  4.4.1 Impact of regularizations

In Figure 7, the MAE of PhysMLE-R and its variants on three datasets (i.e.,
PURE, HCW, and UBFC-rPPG) over Protocol I are presented to illustrate the
effect of each tailored regularization in the MSSDG challenge. Overall,
missing any of the regularizations results in performance degradation compared
to the full PhysMLE. Specifically, we first notice that the variant without
$\mathcal{L}_{OEM}$ presents a notable performance decline compared to
PhysMLE-R. We argue that the lack of constraints on the distinct parameters
learned by experts led to the tendency of experts under the MoE architecture
to learn approximate and easy-to-learn features, which degraded the model into
a multi-head-like structure. Referring to Table III to V, it was found that
the performance of PhysMLE without $\mathcal{L}_{OEM}$ is only slightly better
than that of Base-R. Besides, we found that the $\mathcal{L}_{HB}$ mainly
contributed to the HR prediction tasks, despite it still helping the learning
of other tasks slightly. Similarly, the positive effects of $\mathcal{L}_{BR}$
and $\mathcal{L}_{ASp}$ were identified in RR and SpO2 estimation,
respectively. It illustrates the effectiveness of our prior-based
regularization design, and the enhanced task-specific feature can help
alleviate the task imbalance problem under MSSDG.

TABLE VI: Results of Variants with Different Hyper-parameter $\alpha$ and $r$ on MSSDG Protocol I. |  | PURE | HCW
---|---|---|---
Hyper | HR | SpO2 | HR | RR
$\mathbf{\alpha}$ | $\mathbf{r}$ | MAE↓ |  p↑ | MAE↓ |  p↑ | MAE↓ |  p↑ | MAE↓ |  p↑
8 | 8 | 10.40 | 0.46 | 2.89 | 0.20 | 12.97 | 0.23 | 6.13 | 0.06
| 16 | 9.00 | 0.52 | 4.17 | 0.17 | 9.66 | 0.44 | 7.41 | 0.03
| 32 | 8.03 | 0.56 | 6.14 | 0.11 | 7.79 | 0.59 | 8.21 | 0.01
16 | 8 | 7.13 | 0.63 | 3.49 | 0.19 | 9.61 | 0.41 | 6.10 | 0.07
| 16 | 6.69 | 0.67 | 5.91 | 0.13 | 10.12 | 0.34 | 5.99 | 0.09
| 32 | 6.80 | 0.66 | 5.77 | 0.13 | 10.89 | 0.31 | 5.96 | 0.09
32 | 8 | 9.48 | 0.50 | 3.21 | 0.19 | 8.10 | 0.48 | 5.93 | 0.10
| 16 | 5.43 | 0.77 | 2.19 | 0.22 | 7.13 | 0.61 | 5.90 | 0.10
| 32 | 6.67 | 0.67 | 3.74 | 0.18 | 7.49 | 0.54 | 6.15 | 0.06
Figure 8: MAE of PhysMLE-R with different numbers of experts. TABLE VII:
Computational cost of variants with different structures on MSSDG Protocol I.

| PURE | HCW |  |
---|---|---|---|---
| HR | SpO2 | HR | RR | Trainable Params(M) | FLOPs(G)
Base-R∗ | 8.74 | 3.93 | 10.18 | 8.22 | 13.81 | 7.32
MoE-R∗ | 5.45 | 2.77 | 7.02 | 6.08 | 44.79 | 58.92
Softmax-R+ | 6.95 | 3.14 | 8.16 | 7.60 | 10.61 | 22.16
PhysMLE-R+ | 5.43 | 2.19 | 7.13 | 5.90 | 13.85 | 34.57
Base-T∗ | 8.81 | 2.45 | 10.24 | 9.74 | 82.57 | 15.70
MoE-T∗ | 6.84 | 2.01 | 7.50 | 6.62 | 214.42 | 90.07
Softmax-T+ | 7.20 | 3.36 | 8.13 | 8.33 | 69.66 | 46.89
PhysMLE-T+ | 6.62 | 1.86 | 7.34 | 6.55 | 82.90 | 69.00

Notes: In this table, the MAE of each task is provided. The + means that its
experts are based on LoRA, and the ∗ means full fine-tuning.

####  4.4.2 Impact of hyperparameters

As shown in Table VI, given various combinations of $\alpha$ and $r$, we
present the MAE and $p$ of the multi-task evaluation over PURE and HCW.
Generally, we notice that the $r$ was positively associated with the capacity
of HR estimation and negatively related to other tasks (i.e., SpO2 and RR). We
assume it is because $r$ determines the rank of each expert’s parameters,
which is crucial to the number of learned features. Compared to HR, the range
of SpO2 and RR is narrower, which means that there are fewer and simpler
mapping relationships between the inputted STMaps and the predicted values.
Thus, for multi-task physiological measurement, too high or too low $r$ can
lead to a severe see-saw effect between tasks in the model. Additionally, for
the effect of $\alpha$ solely, although there is no clear pattern to its
effects on the model performance, some interesting findings are elaborated for
the effect of $\frac{\alpha}{r}$. As stated in [26], $\alpha$ is used to scale
the weights of LoRAs. Therefore, we find that when $r$ was 8 or 32, too large
$\frac{\alpha}{r}$ can amplify the seesaw effect; when $\frac{\alpha}{r}$ was
too small, even though $r$ was chosen to be a more appropriate value of 16,
the performance of the model can be restricted by the fact that the features
learned by LoRA experts were overly narrowed. Overall, the choice of
appropriate $r$ and $\alpha$ has a crucial impact on the performance of a
LoRA-based model.

TABLE VIII: Comparison Experiment Results on MSSDG Protocol II.

|  | VIPL-HR | HCW
---|---|---|---
|  | HR | SpO2 | HR | RR
Method Type | Method | MAE↓ | RMSE↓ |  p↑ | MAE↓ | RMSE↓ |  p↑ | MAE↓ | RMSE↓ |  p↑ | MAE↓ | RMSE↓ |  p↑
Single-task | GREEN [40] | 12.18 | 18.23 | 0.25 | — | — | — | 14.36 | 20.01 | 0.04 | — | — | —
Traditional | CHROM [41] | 11.44 | 16.97 | 0.28 | — | — | — | 14.66 | 19.81 | 0.05 | — | — | —
| POS [42] | 14.59 | 21.26 | 0.19 | — | — | — | 12.37 | 17.61 | 0.11 | — | — | —
| ARM-SpO2 [10] | — | — | — | 8.11 | 12.23 | 0.08 | — | — | — | — | — | —
| ARM-RR [10] | — | — | — | — | — | — | — | — | — | 8.08 | 8.92 | 0.01
Single-task | RhythmNet [37] | 5.30 | 8.14 | 0.76 | — | — | — | 6.27 | 10.06 | 0.65 | — | — | —
Deep | BVPNet [13] | 5.34 | 7.85 | 0.70 | — | — | — | 6.51 | 10.22 | 0.64 | — | — | —
| Dual-GAN [90] | 4.93 | 7.68 | 0.81 | — | — | — | 5.33 | 9.87 | 0.69 | — | — | —
| PhysFormer++ [45] | 4.88 | 7.62 | 0.80 | — | — | — | 4.88 | 9.01 | 0.72 | — | — | —
| EfficientPhys [91] | 5.84 | 8.91 | 0.72 | — | — | — | 6.90 | 11.13 | 0.62 | — | — | —
| rPPG-MAE∗ [98] | 4.52 | 7.56 | 0.81 | — | — | — | 4.11 | 8.20 | 0.78 | — | — | —
| Contrast-Phys+∗ [28] | 4.06 | 7.10 | 0.83 | — | — | — | 3.83 | 7.57 | 0.82 | — | — | —
| rSPO [54] | — | — | — | 2.36 | 6.54 | 0.16 | — | — | — | — | — | —
Multi-task | MTTS-CAN [19] | — | — | — | — | — | — | 8.22 | 13.31 | 0.51 | 7.66 | 11.17 | 0.04
Deep | BigSmall [20] | — | — | — | — | — | — | 7.10 | 11.64 | 0.58 | 5.42 | 9.53 | 0.06
Ours | Base-R [78] | 4.64 | 7.98 | 0.80 | 2.59 | 6.77 | 0.12 | 4.94 | 9.07 | 0.71 | 4.65 | 7.81 | 0.08
| Base-T [94] | 4.91 | 8.57 | 0.76 | 1.64 | 5.78 | 0.19 | 5.39 | 9.87 | 0.68 | 5.03 | 8.60 | 0.07
| PhysMLE-R | 4.29 | 7.37 | 0.81 | 1.85 | 5.92 | 0.17 | 3.88 | 7.63 | 0.81 | 2.23 | 4.11 | 0.14
| PhysMLE-T | 4.79 | 8.06 | 0.79 | 1.55 | 5.40 | 0.21 | 4.77 | 8.90 | 0.75 | 3.14 | 5.28 | 0.12

Notes: In this table, ∗ means it has been self-supervised pre-trained over the
target dataset before training.

####  4.4.3 Impact of the number of experts

We further conducted ablation experiments for PhysMLE-R with different numbers
$K$ of experts over PURE and HCW. The $K$ was adjusted from 1 to 6, and the
MAE was presented in Figure 8.

As shown by the blue curves in Figure 8, we notice that the best HR estimation
performance was acquired by the variant with 4 experts. The performance curve
of HR was also overall steady when $K$ was between 3 and 5. This is because
the four physiological estimation tasks can be implicitly aligned with the
experts by the expert orthogonality $\mathcal{L}_{OEM}$. Since HR estimation
is highly correlated with the BVP estimation task and most of the datasets in
MSSDG provide HR labels computed from the BVP signals, a model with 4 experts
had more experts to learn HR-related features compared to a model with 3
experts. We call this task as strong task, which means that they share more
features internally and have a relatively balanced number of labels available
for training in cross-domain scenarios.

However, when we look at the orange curves for physiological signals such as
SpO2 and RR, which were relatively weak tasks (i.e., less internal correlation
and training labels), we noticed that more experts do not bring significant
benefit to such tasks except for the fact that they bring more computational
complexity (FLOPs(G) of 53.43 for K=4, much higher than 34.57 for K=3).
Especially, for SpO2, it is essentially not directly accessible from the BVP
signal and the curve of SpO2 showed a significant increase when $K$ exceeded
3. This means that the model appears to have a more obvious seesaw effect. In
summary, we chose $K=3$ experts as the best parameter for PhysMLE in our
experiment.

####  4.4.4 Impact of key component

In this section, we validate the effectiveness and computational costs of each
key component of the PhysMLE. For comparison, we set up three types of
baselines: (1) the Base model that solely consists of one backbone network and
multi-task heads with full fine-tuning; (2) the MoE model that replaces the
$K=3$ LoRA experts in PhysMLE with the two full parameter experts and the
original weights of the backbone network; (3) the Softmax model that remains
the LoRA experts but replaces EFRouter in PhysMLE with softmax gating [59].
Except for the MAE of each estimation task, we provide the number of trainable
parameters and FLOPs as well.

Figure 9: Visualization of predicted physiological signals on HCW. The BLUE
line indicates the ground-truth signal, the RED line indicates the predicted
signal by PhysMLE, and the ORANGE line is the predicted signal by Base.

From the results of Table VII, we find that the PhysMLE generally performed
well on most tasks. Besides, in terms of computational cost, only the Softmax
models needed a smaller number of trainable parameters and thus lower
arithmetic consumption. However, the performance of Softmax was only better
than the Base models and much worse than PhysMLE. Thus, although our proposed
EFRouter introduces additional parameters and computational effort, the
significant performance improvement shows that it is still a relatively
promising option for the gating mechanism. Compared to the MoE methods, the
full fine-tuning brought a notable increase in the number of training
parameters and FLOPs. However, on the more complex dataset of HCW, the
performance of full fine-tuning methods in HR measurement was slightly better
than PhysMLE, and in other cases, the performance was close to our proposal as
well; it indicates that the HR estimation task in complex scenarios still
requires a certain number of parameters and model complexity to realize.
Nevertheless, considering the overall better performance of PhysMLE and the
relatively acceptable training cost (training parameters number is very close
to Base), we believe that PhysMLE is competitive.

###  4.5 Comparison Experiment over Protocol II

####  4.5.1 Intra-dataset test

In this section, considering the diverse illumination, motion scenes, and
acquisition devices in VIPL-HR and HCW, we provide the results of the five-
fold cross-validation experiment over these two challengable datasets in Table
VIII. We found that only the SOTA self-supervised pre-train single-task model
[98] outperformed our proposal in HR estimation. Nevertheless, PhysMLE showed
a comparatively better performance than other single-task methods, including
one pre-trained method [98] in the single-task indicator. At the same time,
PhysMLE enabled simultaneous multiple human vital signs measurements with
comparable or better performance than baseline models. Compared to multi-task
methods [19, 20], our proposal still kept the leading performance. In all, the
results further supported the effectiveness of our approach in addressing the
challenges posed by these two complex datasets.

Figure 10: Visualization of predicted physiological signals on VIPL-HR. The
BLUE line indicates the ground-truth signal, the RED line indicates the
predicted signal by PhysMLE, and the ORANGE line is the predicted signal by
Base. Figure 11: Visualization of the extracted features on HCW under Protocol
II. The subfigure (a)(d) is the feature produced by Base-R, and the others are
given by PhysMLE-R. Among them,(b)(e) reflects the task-agnostic feature
$s^{{}^{\prime}}$, and (c)(f) is the task-specific feature space after the
EFRouter. The HR and RR value is represented as the color lightens. Figure 12:
Visualization of the extracted features on VIPL-HR under Protocol II. The
subfigure (a)(d) is the feature produced by Base-R, and the others are given
by PhysMLE-R. Among them, (b)(e) reflects the task-agnostic feature
$s^{{}^{\prime}}$, and (c)(f) is the task-specific feature space after the
EFRouter. The HR and SpO2 value is represented as the color lightens.

####  4.5.2 Visualization of predicted vital signs

We present and visualize some example predictions (about 1 minute) from both
HCW and VIPL-HR. Firstly, as shown in Figure 9(a)(b), we found that the output
HR of the PhysMLE-R was overall smoother and closer to ground-truth compared
to Base-R. We believe this was benefited from our temporal-consistent
constraints $\mathcal{L}_{Temporal}$. By contrast, for VIPL-HR, there are more
HR mutations in the provided ground-truth HR (e.g., HR at position 28s in
Figure 10(a) mutates from around 90 to 100). As a result, the HR curve
inference by PhysMLE trained on VIPL-HR also showed more fluctuation when
compared to that on HCW, while numerically and in terms of the overall trend,
it was still closer to ground truth compared to Base-R.

As for other tasks, for RR estimation in HCW, we found the output RR curve of
our proposal is smoother and basically closer to the mean value of ground
truth compared to Base-R. However, we also found that individually occurring
RR fluctuations (e.g., in Figure 9(c), RR should rise to 17 Hz overall between
10s and 40s) were not captured by our PhysMLE-R. From Figure 9(d), we also
note that the overall RR rise that should occur around the 20th second was not
perceived by PhysMLE-R until the 30th second. These indicate that our model,
as a multi-task model, still has room for improvement in capturing RR
transformations more accurately. We expect future work to further address this
issue. Nevertheless, on the SpO2 estimation task, our model showed a
competitive performance. Specifically, refering to Figure 10(c)(d), although
the curve of Base-T and PhysMLE-T was close, our proposal presented higher
stability, and its output was more in line with the characteristics of SpO2.
For instance, we noticed that the output SpO2 value of Base-T sometimes
exceeded 100 (%), which should be the hard range boundary of SpO2. By
contrast, the output of PhysMLE-T always remained in the reasonable numeric
range, which we assume benefited from our $\mathcal{L}_{Asp}$. At the same
time, when there were abnormal values in the ground truth (e.g., SpO2 should
not change suddenly from around 95 to below 50 in Figure 10(c)), PhysMLE-T
could still stabilize its output. This suggests that PhysMLE is capable of
performing the SpO2 estimation task in a stable and robust manner.

####  4.5.3 Visualization of feature space

Based on protocol II, we randomly select 1000 samples from the test set and
visualize their feature spaces by t-SNE [99]. Firstly, see subfigures (a)(d)
in Figure 11, 12, which represent the feature space of Base-R after training.
As one type of hard-parameter sharing structure, the features encoded by the
backbone network are shared among tasks and directly forwarded to each
estimation head. Thus, we identify no clear distribution pattern or
classification plane between samples with different physiological signs. Due
to the diversity of physiological metrics and the specificity between
physiological metrics, constructing the same feature space for different tasks
for the same sample may exacerbate the negative effect on task-wise
performance. This finding can also be identified from subfigures (b)(e) in
Figure 11, 12. These spaces are the task-agnostic feature $s^{{}^{\prime}}$
encoded by the PhysMLE-R before feeding to the task-specific heads and
routers. We notice that, for the task with abundant training labels (i.e.,
HR), there is a clear classification plane (Figure 12(b)) or hierarchical
distribution with increasing HR (Figure 11(b)). By contrast, for those weak
tasks that are with limited labels, there are no very clear identifiable
distributional differences between samples with different levels of RR or
SpO2. It suggests that, a single and uniform feature space is not appropriate
for multi-task physiological measurement, especially in the MSSDG setting,
where weak tasks tend to be compromised by learning from strong tasks.

Refer to the subfigures (c)(f) in Figure 11, 12, we find that, with task-
specific EFRouter feature selection for $s^{{}^{\prime}}$, there is a clearer
pattern of sample distribution as the level of physiological indicators
changes, and there are disparities between the feature spaces of the same
batch of samples under different tasks. Therefore, constructing new feature
spaces for different tasks is important for multi-task learning.

##  5 Limitation

Through our extensive experiments, although it is demonstrated that our method
remains quite competitive on both MSSDG and classic intra-dataset multi-task
evaluations, there are still some limitations and pending issues to be
addressed. The first one is on the number of parameters. Due to our proposed
EFRouter, as an indispensable component in the forward process of the model,
it is not possible to merge the weights directly on the frozen weights after
fine-tuning by utilizing LoRA as in previous works [26]. Therefore, although
our model reduces the training cost of experts significantly by LoRA during
the training process, our model always needs to load both the experts and the
full backbone parameters. While we believe this is an unavoidable problem due
to the MoE architecture, we still hope that future work can address it.

In addition, the accuracy of the RR detection remains somewhat flawed even in
the intra-dataset setting (e.g., not being able to sense changes in RR in
time). We believe that this is a common problem for rPPG-based RR detection.
The basic principle of extracting respiration rate from rPPG is based on HRV,
as we mentioned before in Priors for RR, and this method is, in fact, still
interfered with by irregular respiration patterns, individual variability, and
illumination, etc. In this paper, as a multi-task model, we only rely on the
rPPG-based approach in order to extract more shared features among tasks.
Future multi-task remote physiological monitoring work should consider more
human movement features during respiration for RR estimation.

##  6 Conclusion

In this paper, we propose a low-rank MoE-based multi-task remote physiological
measurement method PhysMLE. In terms of structural design, we innovatively
fuse MoE and LoRA to address multi-task settings to learn shared and unique
features in a more efficient way. Our proposed EFRouter fuses and selects
features to serve different estimation tasks more accurately than the gating
mechanism used in previous MoE approaches. At the same time, considering the
challenge of domain shift between datasets and imbalance label space between
tasks, which are often encountered in the real world, we define the MSSDG
protocol and provide a larger scale benchmark. To fulfill our protocol, in
addition to five public datasets that have been widely used in previous work,
we collected a new dataset HCW to consider physiological monitoring in high
cognitive workload states. In terms of algorithmic innovations, in addition to
spatio-temporal semantic consistency being used to learn shared features, we
innovatively introduce and instantiate three prior knowledge to solve the
issue of unbalanced task labeling under MSSDG. Through experiments on various
datasets, extensive baseline comparisons, structure combination, and
hyperparameter selection, we demonstrate the effectiveness of our proposed
method on multiple evaluation protocols and multiple physiological monitoring
tasks. Future works are expected to make efforts on more efficient multi-task
modeling structures and more accurate RR estimation.

## References

  * [1] C. Gouveia, C. Loss, P. Pinho, J. Vieira, and D. Albuquerque, “Low-profile textile antenna for bioradar integration into car seat upholstery: Wireless acquisition of vital signs while on the road.” _IEEE Antennas and Propagation Magazine_ , 2023.
  * [2] Y.-C. Wu, L.-W. Chiu, C.-C. Lai, B.-F. Wu, and S. S. Lin, “Recognizing, fast and slow: Complex emotion recognition with facial expression detection and remote physiological measurement,” _IEEE Transactions on Affective Computing_ , 2023.
  * [3] Y. Akamatsu, T. Umematsu, and H. Imaoka, “Calibrationphys: Self-supervised video-based heart and respiratory rate measurements by calibrating between multiple cameras,” _IEEE Journal of Biomedical and Health Informatics_ , 2023.
  * [4] Z. Yu, X. Li, and G. Zhao, “Facial-video-based physiological signal measurement: Recent advances and affective applications,” _IEEE Signal Processing Magazine_ , vol. 38, no. 6, pp. 50–58, 2021.
  * [5] X. Niu, Z. Yu, H. Han, X. Li, S. Shan, and G. Zhao, “Video-based remote physiological measurement via cross-verified feature disentangling,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_. Springer, 2020, pp. 295–310.
  * [6] M.-A. Fiedler, M. Rapczyński, and A. Al-Hamadi, “Fusion-based approach for respiratory rate recognition from facial video images,” _IEEE Access_ , vol. 8, pp. 130 036–130 047, 2020.
  * [7] J. Pourbemany, A. Essa, and Y. Zhu, “Real-time video-based heart and respiration rate monitoring,” in _NAECON 2021-IEEE National Aerospace and Electronics Conference_. IEEE, 2021, pp. 332–336.
  * [8] A. I. Siam, N. A. El-Bahnasawy, G. M. El Banby, A. Abou Elazm, and F. E. Abd El-Samie, “Efficient video-based breathing pattern and respiration rate monitoring for remote health monitoring,” _JOSA A_ , vol. 37, no. 11, pp. C118–C124, 2020.
  * [9] B.-J. Wu, B.-F. Wu, Y.-C. Dong, H.-C. Lin, and P.-H. Li, “Peripheral oxygen saturation measurement using an rgb camera,” _IEEE Sensors Journal_ , 2023.
  * [10] L. Tarassenko, M. Villarroel, A. Guazzi, J. Jorge, D. Clifton, and C. Pugh, “Non-contact video-based vital sign monitoring using ambient light and auto-regressive models,” _Physiological measurement_ , vol. 35, no. 5, p. 807, 2014.
  * [11] N. H. Kim, S.-G. Yu, S.-E. Kim, and E. C. Lee, “Non-contact oxygen saturation measurement using ycgcr color space with an rgb camera,” _Sensors_ , vol. 21, no. 18, p. 6120, 2021.
  * [12] H. Shao, L. Luo, J. Qian, S. Chen, C. Hu, and J. Yang, “Tranphys: Spatiotemporal masked transformer steered remote photoplethysmography estimation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2023.
  * [13] A. Das, H. Lu, H. Han, A. Dantcheva, S. Shan, and X. Chen, “Bvpnet: Video-to-bvp signal prediction for remote heart rate estimation,” in _2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)_. IEEE, 2021, pp. 01–08.
  * [14] J. Du, S.-Q. Liu, B. Zhang, and P. C. Yuen, “Weakly supervised rppg estimation for respiratory rate estimation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2391–2397.
  * [15] C. Orphanidou, “A review of big data applications of physiological signal data,” _Biophysical reviews_ , vol. 11, no. 1, pp. 83–87, 2019.
  * [16] C. Orphanidou, T. Bonnici, P. H. Charlton, D. A. Clifton, D. Vallance, and L. Tarassenko, “Signal-quality indices for the electrocardiogram and photoplethysmogram: Derivation and applications to wireless monitoring,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 19, pp. 832–838, 2015. [Online]. Available: https://api.semanticscholar.org/CorpusID:2709335
  * [17] J. Chang, C. Zhang, Y. Hui, D. Leng, Y. Niu, Y. Song, and K. Gai, “Pepnet: Parameter and embedding personalized network for infusing with personalized prior information,” in _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2023, pp. 3795–3804.
  * [18] E. Peper, R. Harvey, I.-M. Lin, H. Tylova, and D. Moss, “Is there more to blood volume pulse than heart rate variability, respiratory sinus arrhythmia, and cardiorespiratory synchrony?” _Biofeedback_ , vol. 35, no. 2, 2007.
  * [19] X. Liu, J. Fromm, S. Patel, and D. McDuff, “Multi-task temporal shift attention networks for on-device contactless vitals measurement,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 19 400–19 411, 2020.
  * [20] G. Narayanswamy, Y. Liu, Y. Yang, C. Ma, X. Liu, D. McDuff, and S. Patel, “Bigsmall: Efficient multi-task learning for disparate spatial and temporal physiological measurements,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2024, pp. 7914–7924.
  * [21] H. Lu, Z. Yu, X. Niu, and Y.-C. Chen, “Neuron structure modeling for generalizable remote physiological measurement,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 18 589–18 599.
  * [22] J. Wang, H. Lu, A. Wang, Y. Chen, and D. He, “Hierarchical style-aware domain generalization for remote physiological measurement,” _IEEE Journal of Biomedical and Health Informatics_ , 2023.
  * [23] W. Sun, X. Zhang, H. Lu, Y. Chen, Y. Ge, X. Huang, J. Yuan, and Y. Chen, “Resolve domain conflicts for generalizable remote physiological measurement,” in _Proceedings of the 31st ACM International Conference on Multimedia_ , ser. MM ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 8214–8224. [Online]. Available: https://doi.org/10.1145/3581783.3612265
  * [24] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” in _International Conference on Learning Representations_ , 2016.
  * [25] X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang, “Gpt understands, too,” _AI Open_ , 2023.
  * [26] E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen _et al._ , “Lora: Low-rank adaptation of large language models,” in _International Conference on Learning Representations_ , 2021.
  * [27] K. Zhang, B. Schölkopf, K. Muandet, and Z. Wang, “Domain adaptation under target and conditional shift,” in _International Conference on Machine Learning_ , 2013.
  * [28] Z. Sun and X. Li, “Contrast-phys+: Unsupervised and weakly-supervised video-based remote physiological measurement via spatiotemporal contrast,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2024.
  * [29] G. G. Berntson, J. Thomas Bigger Jr, D. L. Eckberg, P. Grossman, P. G. Kaufmann, M. Malik, H. N. Nagaraja, S. W. Porges, J. P. Saul, P. H. Stone _et al._ , “Heart rate variability: origins, methods, and interpretive caveats,” _Psychophysiology_ , vol. 34, no. 6, pp. 623–648, 1997.
  * [30] M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A multimodal database for affect recognition and implicit tagging,” _IEEE transactions on affective computing_ , vol. 3, no. 1, pp. 42–55, 2011.
  * [31] R. Stricker, S. Müller, and H.-M. Gross, “Non-contact video-based pulse rate measurement on a mobile service robot,” in _The 23rd IEEE International Symposium on Robot and Human Interactive Communication_. IEEE, 2014, pp. 1056–1062.
  * [32] G. Heusch, A. Anjos, and S. Marcel, “A reproducible study on remote heart rate measurement,” _arXiv preprint arXiv:1709.00962_ , 2017.
  * [33] S. Tulyakov, X. Alameda-Pineda, E. Ricci, L. Yin, J. F. Cohn, and N. Sebe, “Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2396–2404.
  * [34] X. Li, I. Alikhani, J. Shi, T. Seppanen, J. Junttila, K. Majamaa-Voltti, M. Tulppo, and G. Zhao, “The obf database: A large face video database for remote physiological signal measurement and atrial fibrillation detection,” in _2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018)_. IEEE, 2018, pp. 242–249.
  * [35] S. Bobbia, R. Macwan, Y. Benezeth, A. Mansouri, and J. Dubois, “Unsupervised skin tissue segmentation for remote photoplethysmography,” _Pattern Recognition Letters_ , vol. 124, pp. 82–90, 2019.
  * [36] L. Xi, W. Chen, C. Zhao, X. Wu, and J. Wang, “Image enhancement for remote photoplethysmography in a low-light environment,” in _2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)_. IEEE, 2020, pp. 1–7.
  * [37] X. Niu, S. Shan, H. Han, and X. Chen, “Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 2409–2423, 2019.
  * [38] A. Revanur, Z. Li, U. A. Ciftci, L. Yin, and L. A. Jeni, “The first vision for vitals (v4v) challenge for non-contact video-based physiological estimation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2760–2767.
  * [39] J. Tang, K. Chen, Y. Wang, Y. Shi, S. Patel, D. McDuff, and X. Liu, “Mmpd: multi-domain mobile video physiology dataset,” in _2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_. IEEE, 2023, pp. 1–5.
  * [40] W. Verkruysse, L. O. Svaasand, and J. S. Nelson, “Remote plethysmographic imaging using ambient light.” _Optics express_ , vol. 16, no. 26, pp. 21 434–21 445, 2008.
  * [41] G. De Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” _IEEE Transactions on Biomedical Engineering_ , vol. 60, no. 10, pp. 2878–2886, 2013.
  * [42] W. Wang, A. C. Den Brinker, S. Stuijk, and G. De Haan, “Algorithmic principles of remote ppg,” _IEEE Transactions on Biomedical Engineering_ , vol. 64, no. 7, pp. 1479–1491, 2016.
  * [43] W. Chen and D. McDuff, “Deepphys: Video-based physiological measurement using convolutional attention networks,” in _Proceedings of the european conference on computer vision (ECCV)_ , 2018, pp. 349–365.
  * [44] Z. Yu, W. Peng, X. Li, X. Hong, and G. Zhao, “Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 151–160.
  * [45] Z. Yu, Y. Shen, J. Shi, H. Zhao, Y. Cui, J. Zhang, P. Torr, and G. Zhao, “Physformer++: Facial video-based physiological measurement with slowfast temporal difference transformer,” _International Journal of Computer Vision_ , vol. 131, no. 6, pp. 1307–1330, 2023.
  * [46] J. Cheng, R. Liu, J. Li, R. Song, Y. Liu, and X. Chen, “Motion-robust respiratory rate estimation from camera videos via fusing pixel movement and pixel intensity information,” _IEEE Transactions on Instrumentation and Measurement_ , 2023.
  * [47] R. Janssen, W. Wang, A. Moço, and G. De Haan, “Video-based respiration monitoring with automatic region of interest detection,” _Physiological measurement_ , vol. 37, no. 1, p. 100, 2015.
  * [48] F. Schrumpf, C. Mönch, G. Bausch, and M. Fuchs, “Exploiting weak head movements for camera-based respiration detection,” in _2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 2019, pp. 6059–6062.
  * [49] M. Ghodratigohar, H. Ghanadian, and H. Al Osman, “A remote respiration rate measurement method for non-stationary subjects using ceemdan and machine learning,” _IEEE Sensors Journal_ , vol. 20, no. 3, pp. 1400–1410, 2019.
  * [50] X. Niu, Z. Yu, H. Han, X. Li, S. Shan, and G. Zhao, “Video-based remote physiological measurement via cross-verified feature disentangling.” in _European Conference on Computer Vision (ECCV)_ , 2020.
  * [51] L. Kong, Y. Zhao, L. Dong, Y. Jian, X. Jin, B. Li, Y. Feng, M. Liu, X. Liu, and H. Wu, “Non-contact detection of oxygen saturation based on visible light imaging device using ambient light,” _Optics express_ , vol. 21, no. 15, pp. 17 464–17 471, 2013.
  * [52] U. Bal, “Non-contact estimation of heart rate and oxygen saturation using ambient light,” _Biomedical optics express_ , vol. 6, no. 1, pp. 86–97, 2015.
  * [53] M. Hu, X. Wu, X. Wang, Y. Xing, N. An, and P. Shi, “Contactless blood oxygen estimation from face videos: A multi-model fusion method based on deep learning,” _Biomedical Signal Processing and Control_ , vol. 81, p. 104487, 2023.
  * [54] Y. Akamatsu, Y. Onishi, and H. Imaoka, “Blood oxygen saturation estimation from facial video via dc and ac components of spatio-temporal map,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [55] R. Caruana, “Multitask learning,” _Machine learning_ , vol. 28, pp. 41–75, 1997.
  * [56] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, “Cross-stitch networks for multi-task learning,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3994–4003.
  * [57] S. Liu, E. Johns, and A. J. Davison, “End-to-end multi-task learning with attention,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 1871–1880.
  * [58] Y. Yang, P.-T. Jiang, Q. Hou, H. Zhang, J. Chen, and B. Li, “Multi-task dense prediction via mixture of low-rank experts,” _arXiv preprint arXiv:2403.17749_ , 2024.
  * [59] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” _arXiv preprint arXiv:1701.06538_ , 2017.
  * [60] H. Lu, X. Niu, J. Wang, Y. Wang, Q. Hu, J. Tang, Y. Zhang, K. Yuan, B. Huang, Z. Yu _et al._ , “Gpt as psychologist? preliminary evaluations for gpt-4v on visual affective computing,” _2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) workshop_ , 2024.
  * [61] J. Ma, Z. Zhao, X. Yi, J. Chen, L. Hong, and E. H. Chi, “Modeling task relationships in multi-task learning with multi-gate mixture-of-experts,” in _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_, 2018, pp. 1930–1939.
  * [62] Q. Liu, X. Wu, X. Zhao, Y. Zhu, D. Xu, F. Tian, and Y. Zheng, “Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications,” _arXiv preprint arXiv:2310.18339_ , 2023.
  * [63] Z. Chen, Z. Wang, Z. Wang, H. Liu, Z. Yin, S. Liu, L. Sheng, W. Ouyang, Y. Qiao, and J. Shao, “Octavius: Mitigating task interference in mllms via moe,” _arXiv preprint arXiv:2311.02684_ , 2023.
  * [64] R. Yousefi, M. Nourani, S. Ostadabbas, and I. Panahi, “A motion-tolerant adaptive algorithm for wearable photoplethysmographic biosensors,” _IEEE journal of biomedical and health informatics_ , vol. 18, no. 2, pp. 670–681, 2013.
  * [65] J. Du, S.-Q. Liu, B. Zhang, and P. C. Yuen, “Dual-bridging with adversarial noise generation for domain adaptive rppg estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 10 355–10 364.
  * [66] S. Shankar, V. Piratla, S. Chakrabarti, S. Chaudhuri, P. Jyothi, and S. Sarawagi, “Generalizing across domains via cross-gradient training,” _arXiv preprint arXiv:1804.10745_ , 2018.
  * [67] Y. Shi, X. Yu, K. Sohn, M. Chandraker, and A. K. Jain, “Towards universal representation learning for deep face recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 6817–6826.
  * [68] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in _International conference on machine learning_. PMLR, 2015, pp. 1180–1189.
  * [69] B. Sun, J. Feng, and K. Saenko, “Return of frustratingly easy domain adaptation,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 30, no. 1, 2016.
  * [70] W. Jiyao, L. Hao, H. Hu, C. Yingcong, H. Dengbo, and W. Kaishun, “Generalizable remote physiological measurement via semantic-sheltered alignment and plausible style randomization,” _in progress_ , 2024.
  * [71] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International conference on machine learning_. PMLR, 2017, pp. 1126–1135.
  * [72] D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales, “Learning to generalize: Meta-learning for domain generalization,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 32, no. 1, 2018.
  * [73] F. Lv, J. Liang, S. Li, B. Zang, C. H. Liu, Z. Wang, and D. Liu, “Causality inspired representation learning for domain generalization,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 8046–8056.
  * [74] P. D. Mannheimer, “The light–tissue interaction of pulse oximetry,” _Anesthesia & Analgesia_, vol. 105, no. 6, pp. S10–S17, 2007.
  * [75] Z. Zhong, Z. Tang, T. He, H. Fang, and C. Yuan, “Convolution meets lora: Parameter efficient finetuning for segment anything model,” in _The Twelfth International Conference on Learning Representations_ , 2024.
  * [76] J. Li, Z. Yu, and J. Shi, “Learning motion-robust remote photoplethysmography through arbitrary resolution videos,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 1, 2023, pp. 1334–1342.
  * [77] C. Á. Casado, M. L. Cañellas, and M. B. López, “Depression recognition using remote photoplethysmography from facial videos,” _IEEE Transactions on Affective Computing_ , 2023.
  * [78] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [79] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2020.
  * [80] V. Shah, N. Ruiz, F. Cole, E. Lu, S. Lazebnik, Y. Li, and V. Jampani, “Ziplora: Any subject in any style by effectively merging loras,” _arXiv preprint arXiv:2311.13600_ , 2023.
  * [81] N. Bansal, X. Chen, and Z. Wang, “Can we gain more from orthogonality regularizations in training deep networks?” _Advances in Neural Information Processing Systems_ , vol. 31, 2018.
  * [82] A. Lam and Y. Kuno, “Robust heart rate measurement from video using select random patches,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 3640–3648.
  * [83] M. Kumar, A. Veeraraghavan, and A. Sabharwal, “Distanceppg: Robust non-contact vital signs monitoring using a camera,” _Biomedical optics express_ , vol. 6, no. 5, pp. 1565–1588, 2015.
  * [84] Z. Sun and X. Li, “Contrast-phys: Unsupervised video-based remote physiological measurement via spatiotemporal contrast,” in _European Conference on Computer Vision_. Springer, 2022, pp. 492–510.
  * [85] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [86] C.-H. Cheng, Z. Yuen, S. Chen, K.-L. Wong, J.-W. Chin, T.-T. Chan, and R. H. So, “Contactless blood oxygen saturation estimation from facial videos using deep learning,” _Bioengineering_ , vol. 11, no. 3, p. 251, 2024.
  * [87] E. Suprayitno, M. Marlianto, and M. Mauliana, “Measurement device for detecting oxygen saturation in blood, heart rate, and temperature of human body,” in _Journal of Physics: Conference Series_ , vol. 1402, no. 3. IOP Publishing, 2019, p. 033110.
  * [88] H. M. Stauss, “Heart rate variability,” _American Journal of Physiology-Regulatory, Integrative and Comparative Physiology_ , vol. 285, no. 5, pp. R927–R931, 2003.
  * [89] A. Bernardes, R. Couceiro, J. Medeiros, J. Henriques, C. Teixeira, M. Simões, J. Durães, R. Barbosa, H. Madeira, and P. Carvalho, “How reliable are ultra-short-term hrv measurements during cognitively demanding tasks?” _Sensors_ , vol. 22, no. 17, p. 6528, 2022.
  * [90] H. Lu, H. Han, and S. K. Zhou, “Dual-gan: Joint bvp and noise modeling for remote physiological measurement,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 12 404–12 413.
  * [91] X. Liu, B. Hill, Z. Jiang, S. Patel, and D. McDuff, “Efficientphys: Enabling simple, fast and accurate camera-based cardiac measurement,” in _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , 2023, pp. 5008–5017.
  * [92] D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville, “Out-of-distribution generalization via risk extrapolation (rex),” in _International Conference on Machine Learning_. PMLR, 2021, pp. 5815–5826.
  * [93] G. Parascandolo, A. Neitz, A. Orvieto, L. Gresele, and B. Schölkopf, “Learning explanations that are hard to vary,” _arXiv preprint arXiv:2009.00329_ , 2020.
  * [94] B. Wu, C. Xu, X. Dai, A. Wan, P. Zhang, Z. Yan, M. Tomizuka, J. Gonzalez, K. Keutzer, and P. Vajda, “Visual transformers: Token-based image representation and processing for computer vision,” 2020.
  * [95] H. Wang, N. Jiang, T. Pan, H. Si, Y. Li, and W. Zou, “Cognitive load identification of pilots based on physiological-psychological characteristics in complex environments,” _Journal of Advanced Transportation_ , vol. 2020, pp. 1–16, 2020.
  * [96] S. Basu, A. Bag, M. Aftabuddin, M. Mahadevappa, J. Mukherjee, and R. Guha, “Effects of emotion on physiological signals,” in _2016 IEEE Annual India Conference (INDICON)_. IEEE, 2016, pp. 1–6.
  * [97] R. M. Sabour, Y. Benezeth, P. De Oliveira, J. Chappe, and F. Yang, “Ubfc-phys: A multimodal database for psychophysiological studies of social stress,” _IEEE Transactions on Affective Computing_ , vol. 14, no. 1, pp. 622–636, 2021.
  * [98] X. Liu, Y. Zhang, Z. Yu, H. Lu, H. Yue, and J. Yang, “rppg-mae: Self-supervised pretraining with masked autoencoders for remote physiological measurements,” _IEEE Transactions on Multimedia_ , 2024.
  * [99] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” _Journal of machine learning research_ , vol. 9, no. 11, 2008.