# Gated Domain-Invariant Feature Disentanglement for Domain Generalizable
Object Detection

Haozhuo Zhang
<EMAIL_ADDRESS>Huimin Yu
<EMAIL_ADDRESS>Yuming Yan
<EMAIL_ADDRESS>Runfa Wang<EMAIL_ADDRESS>
###### Abstract

For Domain Generalizable Object Detection (DGOD), Disentangled Representation
Learning (DRL) helps a lot by explicitly disentangling Domain-Invariant
Representations (DIR) from Domain-Specific Representations (DSR). Considering
the domain category is an attribute of input data, it should be feasible for
networks to fit a specific mapping which projects DSR into feature channels
exclusive to domain-specific information, and thus much cleaner
disentanglement of DIR from DSR can be achieved simply on channel dimension.
Inspired by this idea, we propose a novel DRL method for DGOD, which is termed
Gated Domain-Invariant Feature Disentanglement (GDIFD). In GDIFD, a Channel
Gate Module (CGM) learns to output channel gate signals close to either $0$ or
$1$, which can mask out the channels exclusive to domain-specific information
helpful for domain recognition. With the proposed GDIFD, the backbone in our
framework can fit the desired mapping easily, which enables the channel-wise
disentanglement. In experiments, we demonstrate that our approach is highly
effective and achieves state-of-the-art DGOD performance.

###### Index Terms:

Object detection, Domain generalization, Domain adaptation, Representation
disentanglement

##  I Introduction

As a method for locating objects and identifying target categories, object
detection plays a fundamental and vital role in computer vision. The past
decade has witnessed the booming development of supervised object detection,
many great works have demonstrated excellent performance when the training and
testing data are collected from the same domain [1, 2, 3, 4, 5, 6, 7, 8, 9].
However, labels of the target domain are probably difficult to obtain, and in
some cases target domain data is even unavailable for training. Due to the
domain gap between target domain and source domain, the absence of target
labels or even target data for training will lead to performance degradation
of conventional detectors on target domain. To address this problem, an
emerging Domain Generalizable Object Detection (DGOD) method [10] and many
Unsupervised Domain Adaptive Object Detection (UDAOD) methods [11, 12, 13, 14,
15, 16, 17, 18] have been proposed to enhance the generalizability of object
detectors. The DGOD method only requires labeled data from several source
domains to align the data distributions and achieves better detector
generalizability on the unseen target domain. UDAOD methods, as less powerful
ways, utilize both labeled source data and unlabeled target data to reduce the
domain gap. In this paper, we focus on the DGOD task that is more challenging.

Figure 1: (a) Using a shared domain-invariant backbone $E_{di}$ across domains
and one domain-specific backbone $E_{ds}$ for each source domain to extract
DIR $F_{di}$ and DSR $F_{ds}$ respectively. A decoder $G_{rec}$ is used to
reconstruct the input image $I$. (b) Both $E_{di}$ and $E_{ds}$ are shared
shallow encoders following the backbone, specific losses can be utilized to
make $F_{di}$ and $F_{ds}$ independent. (c) Only a shared shallow encoder
$E_{di}$ is used to extract $F_{di}$ from the backbone feature $F_{b}$,
$F_{ds}$ is simply the difference between $F_{b}$ and $F_{di}$. $F_{di}$ and
$F_{ds}$ are constrained to be orthogonal. (d) Proposed gated DRL module based
on the channel-wise gate signal $S_{di}$ output by Channel Gate Module (CGM)

The first and currently the only DGOD method DIDN [10] employs Disentangled
Representation Learning (DRL) to address DGOD, which attempts to disentangle
Domain-Invariant Representations (DIR) from Domain-Specific Representations
(DSR) explicitly and the feature alignment across domains is performed on the
disentangled DIR without the interference from DSR. DGOD can be tackled by
detecting objects based on the DIR which has little domain-specific
information detrimental to detector generalizability. Specifically, DIDN uses
multiple non-shared domain-specific backbones $E_{ds}$ and one shared domain-
invariant backbone $E_{di}$ as shown in Fig. 1(a). Besides this DRL structure
with a huge number of parameters for DGOD, there are two types of lightweight
DRL modules in existing UDAOD researches: one is adopted by the powerful UDAOD
method DDF [18] (Fig. 1(b)), the other is the Vector-Decomposed
Disentanglement (VDD) proposed in [17] (Fig. 1(c)). These two DRL methods use
very shallow encoders $E_{di}$ to extract DIR $F_{di}$ from the backbone
feature $F_{b}$, and VDD even directly takes the difference between $F_{b}$
and $F_{di}$ as $F_{ds}$, which ensures no loss of information in $F_{b}$
without the reconstruction operation used in DIDN. However, a helpful a priori
knowledge is not leveraged in designing DRL modules for DGOD and UDAOD.
Considering that the domain category is an attribute of input data, it is
feasible to make specific feature channels responsible for encoding only the
domain-specific information related to the domain category, and cleaner
disentanglement of DIR from DSR can be achieved by simply filtering out the
channels exclusive to domain-specific information. Although domain-specific
information can be exclusively projected into specific feature channels by
stacked convolutional layers used in existing DRL methods, these vanilla
networks do not guarantee to fit the special mapping we described.

Instead of expecting stacked convolutional layers to fit the desired mapping,
we propose a novel DRL method Gated Domain-Invariant Feature Disentanglement
(GDIFD) to explicitly let the backbone fit the mapping and achieve domain-
invariant feature disentanglement simply along the channel dimension.
Specifically, a Channel Gate Module (CGM) is designed to output channel-wise
domain-invariant gate signal $S_{di}$ which is expected to mask out the
dedicated channels for domain-specific information and maintains the other
domain-invariant ones for object detection. As shown in Fig. 1(d), the
disentanglement of DIR from DSR will be achieved easily by performing element-
wise product of $S_{di}$ and the backbone feature $F_{b}$. To explicitly
filter out the domain-specific information on channel dimension, DSR is used
to perform domain recognition following our idea. Additionally, a gate loss is
proposed to make CGM output $S_{di}$ with values very close to either $0$ or
$1$ to enable much cleaner channel-wise disentanglement, which is also the key
difference between our channel-wise gate signal and the conventional channel
attention [19] with unconstrained values ranged from $0$ to $1$. Domain-
specific channels are facilitated to be exclusive to DSR by the gate loss,
which enables the desired cleaner channel-wise disentanglement. Furthermore,
the commonly used adversarial alignment of DIR is adopted to promote the
independence between DIR and DSR. With the help of this novel DRL method, the
backbone in our DGOD framework learns to make some channels exclusively encode
all the domain information and thus enable simple channel-wise
disentanglement.

Our main contributions are summarized as follows: (1) We propose a novel Gated
Domain-Invariant Feature Disentanglement (GDIFD) for DGOD, which enables much
cleaner disentanglement on channel dimension. (2) A Channel Gate Module (CGM)
is designed for better channel-wise gate signal prediction. In addition, a
special initialization of CGM is proposed to further improve performance of
our DGOD framework. (3) Compared to the baseline, significant DGOD performance
gain of our DRL method is demonstrated in extensive experiments. Additionally,
the superiority of GDIFD over other DRL methods and the effectiveness of our
module design are validated.

##  II Related Work

###  II-A DGOD and UDAOD based on Feature Alignment

Domain gap can be reduced by minimizing the discrepancy of the feature spaces
of multiple domains. Ganin et al. [20] first propose the widely used Gradient
Reversal Layer (GRL) to enable feature alignment via adversarial learning. The
model equipped with GRL can learn to extract domain-invariant features by
deceiving an additional domain classifier in training. Lin et al. [10] propose
the first and currently the only DGOD method DIDN, they resort to feature
disentanglement on both image and instance level to obtain the final domain-
invariant instance-level feature for detection. The disentanglement is
promoted by adversarially aligning the domain-invariant features. Due to the
separation of domain-specific features, the alignment becomes easier and thus
DIDN achieves good performance. As for the UDAOD task, Chen et al. [11] first
propose an end-to-end UDAOD method by applying adversarial feature alignment
on image and instance level. Saito et al. [21] introduce adaptive weights for
domain classification losses to pay more attention on the locations which are
easier for domain confusion. By this way, the difficulty of adversarial
feature alignment is reduced. Zheng et al. [22] use attention maps to make
adversarial alignment focus on the foreground areas. Additionally, they build
a prototype for each category in different domains and achieve alignment by
explicitly drawing prototypes of the same category closer to each other.
Considering the noise from background, Hsu et al. [14] conduct center-aware
feature alignment by using the output centerness and classification maps of
FCOS [9] detector as weights for the features to align. Guan et al. [23]
further introduce an uncertainty metric to focus on the pool-aligned samples
to achieve more refined and robust feature alignment.

###  II-B DRL in Domain Generalization and Domain Adaptation

Besides the aforementioned DGOD method DIDN, DRL is widely used for DG and DA
tasks because of the ability to ease feature alignment by explicitly
disentangling DIR from DSR.

Peng et al. [24] extend the adversarial training to the classification task
and propose a novel three-branch DRL structure which can output class-
irrelevant features, domain-specific features and desired domain-invariant
features which are also task-relevant. In this way, more noise is separated.
Nonetheless, this method requires additional reconstruction like DIDN does.
Chen et al. [25] use a simpler DRL structure as depicted in Fig. 1(b) for DG
classification. A novel domain-specific style memory is built to constrain the
style in a single domain to be consistent, which facilitate the DRL. Liu et
al. [18] use the same DRL method as the former algorithm and achieves
remarkable UDAOD results by increasing the distances between DIR and DSR on
both image and instance level. In addition, the performance can be further
enhanced by reducing the similarity between instance features from different
domains. Wu et al. [17] propose a novel vector-decomposed DRL structure which
only has one extractor for DIR and uses simple orthogonal constraint instead
of reconstruction (Fig. 1(c)).

Different from the existing DRL ways, we note that it is possible to
disentangle DIR from DSR on channel dimension. We propose a gated DRL method
to explicitly make the backbone project information private for domains into
some dedicated channels. By this way, channel-wise domain-invariant feature
disentanglement can be achieved simply by only maintaining the complementary
domain-invariant channels of a backbone feature.

##  III Proposed Method

###  III-A Algorithm Overview

There are sufficient available labeled data from multiple source domains
during training for DGOD, and we aim to use these data to train an object
detector that generalizes well on the unseen target domain. To achieve this
goal, we employ Disentangled Representation Learning (DRL) to explicitly
disentangle the domain-invariant feature $F_{di}$ from the domain-specific
feature $F_{ds}$ that has negative effects on model generalization. As shown
in Fig. 2, we design a novel Gated Domain-Invariant Feature Disentanglement
(GDIFD) module which is shared for the five-scale features extracted by the
backbone $E_{b}$ with Feature Pyramid Network (FPN) [26]. For a deep feature
$F_{b}$, the proposed Channel Gate Module (CGM) outputs a channel-wise gate
signal $S_{di}$ which can mask out the channels exclusive to domain-specific
information. The DIR $F_{di}$ we desire can be obtained by doing element-wise
product for $F_{b}$ and $S_{di}$, and the complementary DSR $F_{ds}$ is the
element-wise product of $F_{b}$ and $\textbf{1}-S_{di}$. The explicit
extraction of DSR on channel dimension is achieved by domain recognition via
the domain classification loss $\mathcal{L}_{D-cls}$. In addition, a gate loss
$\mathcal{L}_{gate}$ calculated using $S_{di}$ and $\textbf{1}-S_{di}$
facilitates the CGM to output gate signal with values close to $0$ or $1$ and
enables much cleaner disentanglement on channel dimension. To further promote
the independence between DIR and DSR, the adversarial domain classification
loss $\mathcal{L}_{D-adv}$ is imposed on DIR, which can largely remove the
domain-specific information in DIR. Finally, the detection results are output
by the FCOS head $H$ fed with the domain-invariant feature $F_{di}$.

###  III-B Gated Domain-Invariant Feature Disentanglement

To enable the novel channel-wise disentanglement, we first design the CGM
(described in Section III-C) for GDIFD to output the channel gate signal
$S_{di}\in\mathbb{R}^{C}$ whose values ranged from $0$ to $1$. With the help
of this channel gate signal, a backbone feature $F_{b}\in\mathbb{R}^{C\times
H\times W}$ of $C$ channels and $H\times W$ spatial resolution can be
separated into two complementary parts denoted as
$F_{di}\in\mathbb{R}^{C\times H\times W}$ and $F_{ds}\in\mathbb{R}^{C\times
H\times W}$ by performing element-wise product of $F_{b}$ with $S_{di}$ and
$\textbf{1}-S_{di}$ respectively as shown in Fig. 2.

Figure 2: A baseline model equipped with the proposed Gated Domain-Invariant
Feature Disentanglement (GDIFD) for DGOD. Images from source domains are fed
to a shared backbone network $E_{b}$ which outputs deep features of 5 scales.
The proposed GDIFD module and FCOS heads $H$ are shared across the 5 scales.
Adversarial feature alignment using loss $\mathcal{L}_{D-adv}$ and domain
classification using loss $\mathcal{L}_{D-cls}$ are imposed on DIR $F_{di}$
and DSR $F_{ds}$, respectively. $D_{di}$ and $C_{ds}$ are the discriminator
with GRL and the domain classifier, respectively. The proposed gate losses
$\mathcal{L}_{gate}$ are computed using channel gate signal $S_{di}$.
Detection losses $\mathcal{L}_{det}$ are computed using outputs of $H$

As depicted in Section I, we propose to achieve the disentanglement of DIR
from DSR on channel dimension by utilizing the a priori knowledge that domain
category is an attribute of the input data. Specifically, $F_{ds}$ desired to
be DSR is fed to a five-layer fully convolutional domain classifier $C_{ds}$
to conduct domain classification for each spatial location on $F_{ds}$. The
domain category label is set as $T_{s}\in\mathbb{R}^{M}$ for a source domain
$S_{s}\in\\{S_{1},S_{2},\dots,S_{M}\\}$, where $M$ is the number of the source
domains can be seen. The value $T_{s}^{(s)}$ is $1$ and $T_{s}^{(d)}$ is $0$
for $d\neq s$. For an input image $I_{s}$ from the source domain $S_{s}$, the
domain classification loss is defined as:

$\displaystyle\mathcal{L}_{D-cls}(I_{s})$ $\displaystyle=-\dfrac{1}{H\times
W}\sum_{u=1,v=1}^{H,W}\sum_{d=1}^{M}(T_{s}^{(d)}\log(C_{ds}(F_{ds})^{(u,v,d)}))\
.$ (1)

To achieve much cleaner disentanglement on channel dimension, we further try
to let the backbone fit a special mapping that projects the encoded domain-
specific information into channels exclusive to DSR. A gate loss which
constrains the values of $S_{di}$ very close to either $0$ or $1$ is designed
to enable the cleaner channel-wise disentanglement. The gate loss is
calculated as:

$\displaystyle\mathcal{L}_{gate}$ $\displaystyle={\rm
max}(S_{di}^{T}(\textbf{1}-S_{di})-m,0)\ .$ (2)

$m$ is a margin we add to ease the training, since the gate signal output by
sigmoid activation are in the range of $(0,1)$ and the inner product of
$S_{di}$ and $\textbf{1}-S_{di}$ can never be $0$. We simply set $m$ as
$0.01$. With the help of $\mathcal{L}_{gate}$, the orthogonality desired for
DRL between $F_{di}$ and $F_{ds}$ can be met easily in GDIFD, since the inner
product of $F_{di}^{(u,v)}$ and $F_{ds}^{(u,v)}$ for a spatial location
$(u,v)$ will be a positive scalar near $0$ with all the values of $S_{di}$
very close to either $0$ or $1$.

At this point, the unwanted DSR is extracted explicitly and the DIR mainly
encodes the information for the object detection task. However, it should be
noted that the orthogonality does not guarantee $F_{di}$ and $F_{ds}$ are
independent. To promote the independence between DIR and DSR, the commonly
used adversarial feature alignment method [11] is imposed on $F_{di}$ to
ensure the domain-specific information is largely removed from DIR.
Specifically, $F_{di}$ is fed to a discriminator $D_{di}$ with the same
structure of $C_{ds}$ to perform domain classification. For an input image
$I_{s}$ from the source domain $S_{s}$, the adversarial loss function for an
extracted $F_{di}$ is defined as:

$\displaystyle\mathcal{L}_{D-adv}(I_{s})$ $\displaystyle=-\dfrac{1}{H\times
W}\sum_{u=1,v=1}^{H,W}\sum_{d=1}^{M}(T_{s}^{(d)}\log(D_{di}(F_{di})^{(u,v,d)}))\
.$ (3)

A Gradient Reversal Layer (GRL) [20] which reverses the sign of the
backpropagating gradient is inserted between the extractor of $F_{di}$ and the
discriminator $D_{di}$ to make the domain-invariant part of our model learn to
adversarially deceive $D_{di}$ by outputting the desired $F_{di}$ with little
domain-specific information for domain recognition.

###  III-C Channel Gate Module

As described in Section III-B, the CGM is designed to output channel-wise
signal $S_{di}\in\mathbb{R}^{C}$. The things that come to mind firstly when we
think of channel-wise predictions are likely to be the widely used channel
attention which can be output by SE block [19] and CBAM [27]. With these
channel attention modules, $S_{di}$ of the same shape as the channel attention
can certainly be obtained. However, these modules are subject to efficiency
constraints. Specifically, the global pooling operation before encoding is so
rough that the details of domain-level information may be lost, which will
increase the difficulty and instability of the subsequent gate signal
prediction. Instead, we insert the global average pooling layer after per-
pixel gate signal prediction in the proposed CGM to let every feature location
“vote” for the final global channel signal $S_{di}$.

Figure 3: Network design of the proposed Channel Gate Module (CGM). The bias
of the last convolutional layer is specially initialized in training

Still different from usual methods, we use four $3\times 3$ convolutional
layers of stronger representational capacity followed by Instance
Normalization [28] and ReLU activation [29] to predict the channel signal for
each feature location (Fig. 3). Note that the last convolutional layer is
followed by sigmoid for outputting signal values ranged from $0$ to $1$.

In addition, the task-relevant DIR should be much more informative than the
DSR encoding only domain-specific information for domain recognition and thus
the optimal backbone feature $F_{b}$ may have fewer channels dedicated to DSR
encoding. Based on this thought, we try to make all the values of $S_{di}$
very close to $1$ at the start of training, which is probably close to the
optimal case and can help our DGOD framework achieve better performance.
Specifically, the special initialization is achieved by setting the bias of
the last convolutional layer as large values to get $s_{init}$ close to $1$,
since the weights of convolutional layers are usually initialized to values
close to $0$ and $s_{init}$ is close to $Sigmoid(b_{init})$. $b_{init}$
denotes the initial value filling the bias of the last convolutional layer and
$s_{init}$ denotes the expected initial values filling $S_{di}$. Then we can
calculate $b_{init}$ as:

$\displaystyle b_{init}$ $\displaystyle=-\log(\dfrac{1-s_{init}}{s_{init}})\
.$ (4)

###  III-D Overall Objective for Proposed DGOD Framework

Following the UDAOD method CFA [14], our DGOD framework is based on the FCOS
detector [9]. As shown in Fig. 2, the backbone with FPN outputs features of
five scales, which can be denoted as $F_{b}^{l}\in\mathbb{R}^{C\times H\times
W}$ with $l\in\\{3,4,5,6,7\\}$ and $C=256$. Our disentangled module and the
FCOS head are shared across the five scale levels, but the domain
discriminators and domain classifiers depicted in Section III-B for feature
alignment are not shared.

As for the detection task, for a backbone feature $F_{b}^{l}$ of an input
image $I_{s}$, the fully convolutional FCOS head will output an object
classification score map $M_{cls}$, a centerness map $M_{ctr}$ predicting the
distances of each pixel to the four sides of the corresponding bounding box
and a regression map $B$ containing the encoded predicted bounding box
information. Given the ground-truth bounding boxes $B_{gt}$, loss functions
$\mathcal{L}_{cls},\mathcal{L}_{ctr}$ and $\mathcal{L}_{reg}$ will be
calculated for the three FCOS head outputs respectively. $\mathcal{L}_{cls}$
is the focal loss [6], $\mathcal{L}_{ctr}$ is the binary cross-entropy loss
and $\mathcal{L}_{reg}$ is the IoU loss [30]. The loss function $\mathcal{L}$
for the detection task of scale level $l$ can be written as:

$\displaystyle\mathcal{L}_{det}^{l}(I_{s},B_{gt})$
$\displaystyle=\mathcal{L}_{cls}^{l}+\mathcal{L}_{ctr}^{l}+\mathcal{L}_{reg}^{l}\
.$ (5)

Denoting the previously depicted losses of scale level $l$ for the GDIFD
module as $\mathcal{L}_{D-adv}^{l},\mathcal{L}_{D-cls}^{l}$ and
$\mathcal{L}_{gate}^{l}$, the overall loss function can be expressed as:

$\displaystyle\mathcal{L}(I_{s},B_{gt})$
$\displaystyle=\sum_{l=3}^{7}(\mathcal{L}_{det}^{l}+\lambda_{D-adv}\mathcal{L}_{D-adv}^{l}+\lambda_{D-cls}\mathcal{L}_{D-cls}^{l}+\lambda_{gate}\mathcal{L}_{gate}^{l})\
.$ (6)

##  IV Experiment

###  IV-A Datasets and DGOD settings

The dataset settings described in the first and currently the only DGOD method
DIDN [10] are followed and there are five datasets used to simulate DGOD in
the real world.

####  IV-A1 Datasets.

The following five large-scale object detection datasets are used: BDD100k
[31], Cityscapes [32], Foggy Cityscapes [32], KITTI [33] and SIM 10k [34].

BDD100k is a large-scale dataset which is collected by a driving platform on
streets. This dataset is very diversified but only the validation set
including $10000$ images is used for DGOD simulation. Cityscapes is a dataset
of urban scenes for driving scenarios and Foggy Cityscapes is the synthetic
foggy version of Cityscapes. These two dataset both have $2975$ images in the
training set and $500$ images in the validation set. KITTI is a database for
autonomous driving, which includes $7481$ images. SIM 10k is a virtual dataset
containing $10000$ images rendered by the GTA gaming engine. For simpler
representations of different DGOD settings, we will denote the five datasets
as B, C, F, K, and S, respectively.

####  IV-A2 DGOD settings.

There are eight categories with instance labels shared across BDD100k,
Cityscapes and Foggy Cityscapes. But KITTI and SIM 10k only have the car
category. Following DIDN [10], we first conduct DGOD experiments based on the
shared eight categories using BDD100k, Cityscapes and Foggy Cityscapes.
Specially, the “train” category is not evaluated for BDD100k because there are
few train objects in this dataset. The second setting is to utilize all the
datasets and only considering the shared “car” category. Following DIDN,
BDD100k and KITTI is used as the validation dataset to evaluate a DGOD method
respectively in this setting. The ablation study is conducted in the multi-
category DGOD settings following DIDN.

###  IV-B Implementation Details

Our method is implemented with PyTorch [35]. We use FCOS [9] without
generalization operations as our baseline, which is denoted as “Base” for
simplicity. We empirically set $\lambda_{D-adv}$, $\lambda_{D-cls}$,
$\lambda_{gate}$ and $s_{init}$ as $0.1$, $0.01$, $0.1$ and $0.9999$
respectively in experiments, unless specified. The weight for the reversed
gradient of GRL is set as $0.01$. ResNet-50 [36] and ResNet-101 [36] pre-
trained on ImageNet [37] with FPN [26] is used as the backbone of the FCOS
detector. Following CFA, the input images are resized with their shorter side
as $800$ and longer side less or equal to $1333$. We use SGD optimizer with
momentum of $0.9$ and weight decay of $1\times 10^{-4}$.

TABLE I: Results (%) of three DGOD settings using B, C and F | Method | person | car | train | rider | truck | motor | bike | bus | mAP
---|---|---|---|---|---|---|---|---|---|---
B&C | DIDN-50 | 31.8 | 49.3 | 26.5 | 38.4 | 27.7 | 24.8 | 33.1 | 35.7 | 33.4
toF | Base-50 | 39.8 | 58.2 | 4.7 | 39.5 | 26.7 | 28.5 | 36.7 | 37.2 | 33.9
Ours-50 | 41.1 | 59.5 | 19.7 | 40.2 | 28.3 | 28.2 | 35.9 | 41.3 | 36.8
Base-101 | 40.1 | 59.5 | 7.2 | 39.5 | 29.8 | 27.9 | 37.3 | 38.2 | 35.0
Ours-101 | 41.5 | 59.7 | 25.4 | 42.2 | 31.8 | 27.8 | 38.6 | 39.4 | 38.3
Oracle-101 | 49.0 | 68.0 | 42.9 | 46.2 | 35.0 | 35.4 | 41.7 | 56.0 | 46.8
B&F | DIDN-50 | 43.6 | 63.2 | 51.1 | 46.2 | 41.9 | 36.0 | 41.3 | 60.9 | 47.9
toC | Base-50 | 48.7 | 70.6 | 17.2 | 43.3 | 36.2 | 36.0 | 38.1 | 49.5 | 42.5
Ours-50 | 50.4 | 71.2 | 46.8 | 45.5 | 40.3 | 38.8 | 39.8 | 56.8 | 48.7
Base-101 | 49.2 | 70.3 | 24.1 | 43.4 | 43.5 | 34.2 | 38.2 | 51.8 | 44.3
Ours-101 | 50.9 | 70.7 | 52.1 | 47.6 | 46.0 | 37.2 | 39.5 | 60.4 | 50.6
Oracle-101 | 52.1 | 70.6 | 53.4 | 50.6 | 40.5 | 39.8 | 42.6 | 57.6 | 50.9
C&F | DIDN-50 | 34.5 | 44.2 | - | 30.4 | 21.2 | 9.2 | 22.8 | 19.0 | 22.7
toB | Base-50 | 39.0 | 49.3 | - | 21.8 | 15.1 | 9.4 | 21.6 | 14.0 | 21.3
Ours-50 | 40.1 | 51.1 | - | 25.7 | 16.9 | 7.5 | 22.2 | 12.8 | 22.0
Base-101 | 36.6 | 51.1 | - | 23.7 | 16.6 | 10.7 | 20.2 | 13.2 | 21.6
Ours-101 | 40.3 | 51.1 | - | 23.6 | 19.8 | 6.5 | 22.1 | 19.4 | 22.9
Oracle-101 | 55.0 | 73.9 | - | 29.1 | 51.4 | 23.0 | 35.0 | 45.3 | 39.1

There are two training schedules in our experiments. We note that the model
performance can converge fast in the DGOD settings “C&F to B”, “B&C&F&S to K”
and “C&F&K&S to B”. Therefore, we first train with learning rate $0.003$ for
$4K$ iterations and then train with learning rate $0.0003$ for another $4K$
iterations to achieve fast training. For all the experiments including the
“Oracle” ones (train and test on the same target domain) in other DGOD
settings, the model is trained with learning rate $0.005$ for $14K$ iterations
and then trained with learning rate $0.0005$ for another $6K$ iterations. In
addition, there is a 500-iteration warm-up stage with very small learning rate
in the beginning of training for all experiments. One GPU is used for one
session of training and test. In one iteration, we iterate over the source
domains to train the model. For one domain in an iteration, a batch composed
of four images from this domain is fed to the model for one step of
optimization. Following the first DGOD method DIDN, mean average precision
(mAP) with a threshold of 0.5 is employed to evaluate the results of all the
categories.

###  IV-C Domain Generalization Results

####  IV-C1 Domain Generalization Results.

We report the quantitative results of the state-of-the-art algorithm DIDN, the
non-disentangled baseline and our method in Table I and Table II. The Oracle
results of the vanilla FCOS detector with ResNet-101 on the multi-category
DGOD settings are also reported. We have the following observations based on
these results.

(1) Compared with the Oracle results in Table I, the DGOD performance drops
considerably on all the settings except for the one with the simpler
validation set from Cityscapes. The negative effect of the domain shift
problem can be demonstrated by comparing the DGOD results with the Oracles.

TABLE II: Results (%) of two Domain Generalization settings using B, C, F, K and S Method | DIDN-50 | Base-50 | Ours-50 | Base-101 | Ours-101
---|---|---|---|---|---
B&C&F&S to K | 76.8 | 77.6 | 79.4 | 78.2 | 80.1
C&F&K&S to B | 52.3 | 54.1 | 55.3 | 55.9 | 57.0

(2) The state-of-the-art DGOD algorithm DIDN is even inferior to the FCOS
baseline by $0.5\%$, $0.8\%$, $1.8\%$ in the settings of B&C to F, B&C&F&S to
K and C&F&K&S to B. However, it should be noted that the remarkable
performance of our DGOD framework is largely attributed to the proposed gated
DRL method rather than the simple baseline. Our method achieves $2.9\%$,
$6.2\%$, $0.7\%$, $1.8\%$ and $1.2\%$ improvement against the baseline with
ResNet-50 in Foggy Cityscapes, Cityscapes, multi-category BDD100k, KITTI and
single-category BDD100k, respectively. Compared to the baseline equipped with
ResNet-101 on these five test set, our method achieves $3.3\%$, $6.3\%$,
$1.3\%$, $1.9\%$ and $1.1\%$ improvement. The power of GDIFD for the DGOD task
is shown.

(3) Compared to the state-of-the-art DGOD method DIDN, our DGOD framework with
the same backbone achieves $3.4\%$, $0.8\%$, $2.6\%$ and $3.0\%$ in Foggy
Cityscapes, Cityscapes, KITTI and BDD100k (C&F&K&S to B), respectively. Our
framework is inferior to DIDN by $0.7\%$ in the challenging setting of C&F to
B. It should be noted that our framework is much more lightweight compared to
DIDN with multiple backbones in training, which is a notable advantage brought
by our simple but effective DRL method. Results of DIDN with ResNet-101 are
not provided in [10]. We additionally provide the quantitative results of our
framework with ResNet-101, which achieves the state-of-the-art performance in
all the DGOD settings.

####  IV-C2 Superiority of GDIFD to Other Lightweight DRL Methods for DGOD.

Results of baselines equipped with different lightweight DRL methods are shown
in Table III to further prove the effectiveness of GDIFD. All the models with
a DRL module are optimized via the same adversarial feature alignment losses,
domain classification losses and detection losses. The orthogonal loss
proposed in [17] is imposed on the other two DRL modules, which has a similar
function to our gate loss to promote the orthogonality between DIR and DSR.

The vanilla lightweight DRL methods proposed for UDAOD negatively affect the
performance of the baseline model in DGOD settings, which is probably due to
the fact that the absence of target training data make clean feature
disentanglement mapping on the target domain very difficult to fit for the
simple stacked convolutional layers in these methods. The model with VDD is
even inferior to the model with the DRL method used in [18] according to our
experiments, possibly because it becomes much more difficult to train a single
encoder $E_{di}$ of weak representational capacity to directly extract high-
quality DSR and DIR simultaneously in DGOD settings. Instead of directly
extracting DIR from the input features, GDIFD only aims to output simple
channel-wise gate signals, which also explicitly makes the backbone project
DIR and DSR into different channels and enables much cleaner disentanglement
of DIR from DSR on channel dimension. As shown in Table III, only our
lightweight DRL method outperforms the baseline model in these DGOD settings.

TABLE III: Results (%) of the baseline model with different lightweight DRL methods  Lightweight DGOD Method | Use DRL | B&C to F | B&F to C | C&F to B
---|---|---|---|---
Baseline-101 |  | 35.0 | 44.3 | 21.6
\+ DRL in [18] (Fig. 1(b)) | ✓ | 34.6 | 43.5 | 19.2
\+ VDD (Fig. 1(c)) | ✓ | 30.8 | 32.8 | 16.1
\+ GDIFD (Ours, Fig. 1(d)) | ✓ | 38.3 | 50.6 | 22.9

###  IV-D Ablation Study

####  IV-D1 Effectiveness of the Proposed DGOD Method.

We provide the results of different DGOD strategies in Table IV to validate
the effectiveness of the proposed DRL method. It is shown that only using the
DSR learning part of GDIFD can enhance the model generalizability, since the
domain-specific information can be largely separated out by DSR learning and
the feature for object detection mainly encodes the task-relevant information.
The additional adversarial learning can remove the domain-specific information
in DIR and promote the independence between DIR and DSR as analyzed in Section
III-B, which further improves the DGOD performance of our framework.
Additionally, our DRL method outperforms the non-disentangled method with only
DIR learning, which can be mainly attributed to the fact that the alignment of
DIR performs much better without interference from DSR due to the explicit
domain-invariant feature disentanglement.

TABLE IV: Results (%) of different DGOD schedules in the settings using B, C and F. DSRL: domain-specific representation learning in GDIFD with $\mathcal{L}_{D-cls}$ and $\mathcal{L}_{gate}$; DIRL: conventional adversarial domain-invariant representation learning with $\mathcal{L}_{D-adv}$ Setting | B&C to F | B&F to C | C&F to B
---|---|---|---
| Base |  |  | Ours | Base |  |  | Ours | Base |  |  | Ours
DSRL |  | ✓ |  | ✓ |  | ✓ |  | ✓ |  | ✓ |  | ✓
DIRL |  |  | ✓ | ✓ |  |  | ✓ | ✓ |  |  | ✓ | ✓
mAP | 35.0 | 36.6 | 36.0 | 38.3 | 44.3 | 48.9 | 47.9 | 50.6 | 21.6 | 21.7 | 22.1 | 22.9

####  IV-D2 Effectiveness of the Configurations for CGM.

We compare CGM to other modules of different configurations to demonstrate its
effectiveness via the results in Table V. The corresponding channel signals
are visualized in Fig. 4. As analyzed in Section III-C, the efficient channel
attention module in CBAM [27] performs so rough global pooling before the gate
signal estimation that it can not guarantee comparable results to CGM.
Additionally, the visualization shows that the shared simple channel attention
module of CBAM is not able to output consistent signals across the scale
levels, which indicates that the backbone has to fit a more difficult mapping
to project DSR of different levels to different channels and affects the DGOD
performance negatively.

TABLE V: Results (%) of our framework equipped with CGM of different configurations. $\mathcal{L}_{gate}$ and SI indicate using the gate loss and using special initialization, respectively Channel Gate Modules | $\mathcal{L}_{gate}$ |  SI | B&C to F | B&F to C | C&F to B
---|---|---|---|---|---
CBAM style CGM | ✓ | ✓ | 36.8 | 49.3 | 21.3
CGM (Ours) | ✓ | ✓ | 38.3 | 50.6 | 22.9
CGM | ✓ |  | 36.8 | 49.8 | 22.3
CGM |  | ✓ | 37.3 | 47.9 | 22.1
CGM |  |  | 36.0 | 49.4 | 22.1

The proposed CGM that uses both special initialization and the gate loss shows
the best performance. On one hand, as shown in Fig. 4, due to the gate signals
initialized close to $1$ (channels are exclusive to DIR initially), the model
will tend to use as few channels as possible for the domain classification
task based on DSR. Specifically, our model learns to squeeze the task-
irrelevant DSR to only one channel for the binary source domain classification
in the setting of B&C to F. Without the special initialization, the DGOD
performance goes down, since too many channels are dedicated to encoding the
task-irrelevant DSR and the extraction of the task-relevant DIR may be
negatively affected. On the other hand, without the gate loss, the gate signal
values are not constrained to either $0$ or $1$ and correspond to gray instead
of black or white in the visualization. The gray gate signal values indicate
that the corresponding channels are not exclusive to either DSR or DIR, which
means that these two types of information that we want to disentangle on
channel dimension are actually entangled in these channels. In a word, cleaner
channel-wise disentanglement is not feasible without the gate loss, which
leads to performance degradation.

Figure 4:  $S_{di}$ for five scale levels from different channel gate modules
(B&C to F). $\mathcal{L}_{gate}$ and SI indicate the gate loss and the special
initialization, respectively. Black for low domain-specific gate signals, and
white for high domain-invariant gate signals

###  IV-E Extension to Domain Adaptation

We also adapt our DGOD framework for the UDAOD task to further validate the
effectiveness of our method. As shown in Table VI, previous state-of-the-art
UDAOD methods use different kinds of backbones, base detectors, base
adversarial alignments and their own novel UDAOD approaches, which makes a
fair comparison almost impossible. However, the effectiveness of our simple
method can be validated by comparing the performance of our approach to the
one of CFA, which has the same FCOS baseline as us. By using the proposed
gated DRL method GDIFD instead of the additional center-aware feature
alignment fine-tuning of CFA, our framework obtains $1.2\%$ improvement and
shows good UDAOD performance.

TABLE VI: Results (%) of different methods in the UDAOD setting of C to F. Details of these methods are also described. Backbone VGG-16, ResNet-50, ResNet-101 are denoted by $v$, $r$ and $\hat{r}$, respectively. One-stage and two-stage object detection are denoted by $o$ and $\hat{o}$, respectively. Feature alignment on image-level and the one on both image-level and instance-level are denoted by $a$ and $\hat{a}$, respectively. Feature disentanglement on image-level and the one on both image-level and instance-level are denoted by $d$ and $\hat{d}$, respectively. $\hat{g}$ denotes the graph-induced prototype alignment. $\hat{f}$ denotes the additional model fine-tuning method. $\hat{c}$ denotes the categorical regularization method of VDD’s backbone model. $\hat{u}$ denotes the uncertainty-aware method imposed on the adversarial alignment of image-level and instance-level. $\hat{m}$ denotes the metric learning methods on image-level and instance-level  Method | Details | person | car | train | rider | truck | motor | bike | bus | mAP
---|---|---|---|---|---|---|---|---|---|---
GPA[38] | $r\hat{o}\hat{a}\hat{g}$ | 32.9 | 54.1 | 41.1 | 46.7 | 24.7 | 32.4 | 38.7 | 45.7 | 39.5
CFA[14] | $\hat{r}oa\hat{f}$ | 41.5 | 57.1 | 39.7 | 43.6 | 29.4 | 29.0 | 36.1 | 44.9 | 40.2
VDD[17] | $v\hat{o}\hat{a}d\hat{c}$ | 33.4 | 51.7 | 34.7 | 44.0 | 33.9 | 34.2 | 36.8 | 52.0 | 40.0
DIDN[10] | $r\hat{o}\hat{a}\hat{d}$ | 38.3 | 51.8 | 34.7 | 44.4 | 28.7 | 32.4 | 40.4 | 53.3 | 40.5
UaDAN[23] | $r\hat{o}\hat{a}\hat{u}$ | 36.5 | 53.6 | 42.7 | 46.1 | 28.9 | 32.3 | 38.9 | 49.4 | 41.1
DDF[18] | $r\hat{o}a\hat{d}\hat{m}$ | 37.6 | 56.1 | 47.0 | 45.5 | 30.7 | 31.1 | 39.8 | 50.4 | 42.3
Ours | $\hat{r}oad$ | 43.4 | 58.9 | 39.6 | 45.4 | 28.8 | 28.7 | 39.0 | 47.1 | 41.4

##  V Conclusions

In this paper, we propose Gated Domain-Invariant Feature Disentanglement
(GDIFD) for the DGOD task. To enhance the object detector generalizability on
the unseen target domain, we disentangle domain-invariant feature more cleanly
on channel dimension, which is used for the object detection task. Extensive
experiments have been conducted to validate the effectiveness of GDIFD and the
state-of-the-art performance of our framework. In the future, we will explore
the power of GDIFD on other DG and UDA tasks.

## References

  * [1] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 580–587.
  * [2] R. Girshick, “Fast r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1440–1448.
  * [3] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _Advances in neural information processing systems_ , vol. 28, pp. 91–99, 2015.
  * [4] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788.
  * [5] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _European conference on computer vision_. Springer, 2016, pp. 21–37.
  * [6] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [7] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [8] H. Hu, J. Gu, Z. Zhang, J. Dai, and Y. Wei, “Relation networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3588–3597.
  * [9] Z. Tian, C. Shen, H. Chen, and T. He, “Fcos: Fully convolutional one-stage object detection,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 9627–9636.
  * [10] C. Lin, Z. Yuan, S. Zhao, P. Sun, C. Wang, and J. Cai, “Domain-invariant disentangled network for generalizable object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 8771–8780.
  * [11] Y. Chen, W. Li, C. Sakaridis, D. Dai, and L. Van Gool, “Domain adaptive faster r-cnn for object detection in the wild,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3339–3348.
  * [12] Z. He and L. Zhang, “Multi-adversarial faster-rcnn for unrestricted object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 6668–6677.
  * [13] C. Zhuang, X. Han, W. Huang, and M. Scott, “ifan: Image-instance full alignment networks for adaptive object detection,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 13 122–13 129.
  * [14] C.-C. Hsu, Y.-H. Tsai, Y.-Y. Lin, and M.-H. Yang, “Every pixel matters: Center-aware feature alignment for domain adaptive object detector,” in _European Conference on Computer Vision_. Springer, 2020, pp. 733–748.
  * [15] Y. Wang, R. Zhang, S. Zhang, M. Li, Y. Xia, X. Zhang, and S. Liu, “Domain-specific suppression for adaptive object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 9603–9612.
  * [16] F. Rezaeianaran, R. Shetty, R. Aljundi, D. O. Reino, S. Zhang, and B. Schiele, “Seeking similarities over differences: Similarity-based domain alignment for adaptive object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9204–9213.
  * [17] A. Wu, R. Liu, Y. Han, L. Zhu, and Y. Yang, “Vector-decomposed disentanglement for domain-invariant object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9342–9351.
  * [18] D. Liu, C. Zhang, Y. Song, H. Huang, C. Wang, M. Barnett, and W. Cai, “Decompose to adapt: Cross-domain object detection via feature disentanglement,” _IEEE Transactions on Multimedia_ , 2022.
  * [19] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [20] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in _International conference on machine learning_. PMLR, 2015, pp. 1180–1189.
  * [21] K. Saito, Y. Ushiku, T. Harada, and K. Saenko, “Strong-weak distribution alignment for adaptive object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 6956–6965.
  * [22] Y. Zheng, D. Huang, S. Liu, and Y. Wang, “Cross-domain object detection through coarse-to-fine feature adaptation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 13 766–13 775.
  * [23] D. Guan, J. Huang, A. Xiao, S. Lu, and Y. Cao, “Uncertainty-aware unsupervised domain adaptation in object detection,” _IEEE Transactions on Multimedia_ , 2021.
  * [24] X. Peng, Z. Huang, X. Sun, and K. Saenko, “Domain agnostic learning with disentangled representations,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 5102–5112.
  * [25] Y. Chen, Y. Wang, Y. Pan, T. Yao, X. Tian, and T. Mei, “A style and semantic memory mechanism for domain generalization,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9164–9173.
  * [26] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [27] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 3–19.
  * [28] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” _arXiv preprint arXiv:1607.08022_ , 2016\.
  * [29] A. L. Maas, A. Y. Hannun, A. Y. Ng _et al._ , “Rectifier nonlinearities improve neural network acoustic models,” in _Proc. icml_ , vol. 30, no. 1. Citeseer, 2013, p. 3.
  * [30] J. Yu, Y. Jiang, Z. Wang, Z. Cao, and T. Huang, “Unitbox: An advanced object detection network,” in _Proceedings of the 24th ACM international conference on Multimedia_ , 2016, pp. 516–520.
  * [31] F. Yu, W. Xian, Y. Chen, F. Liu, M. Liao, V. Madhavan, and T. Darrell, “Bdd100k: A diverse driving video database with scalable annotation tooling,” _arXiv preprint arXiv:1805.04687_ , vol. 2, no. 5, p. 6, 2018.
  * [32] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3213–3223.
  * [33] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” _The International Journal of Robotics Research_ , vol. 32, no. 11, pp. 1231–1237, 2013.
  * [34] M. Johnson-Roberson, C. Barto, R. Mehta, S. N. Sridhar, K. Rosaen, and R. Vasudevan, “Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks?” _arXiv preprint arXiv:1610.01983_ , 2016.
  * [35] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, pp. 8026–8037, 2019.
  * [36] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [37] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [38] M. Xu, H. Wang, B. Ni, Q. Tian, and W. Zhang, “Cross-domain detection via graph-induced prototype alignment,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 355–12 364.


[1]This document is the results of the research project funded by the National
Science Foundation of China under Grant 62372190 and Industry University
Cooperation Project of Fujian Province under Grant 2021H6030.

Conceptualization of this study, Methodology, Software

[orcid=0000-0003-3115-6959] [1] Supervision, Writing and editing

Writing and reviewing

Writing and reviewing

Writing and reviewing

Writing and reviewing

[cor1]Corresponding author

# Multi-Task Learning for Fatigue Detection and Face Recognition of Drivers
via Tree-Style Space-Channel Attention Fusion Network

Shulei Qu  Zhenguo Gao<EMAIL_ADDRESS>Xiaowei Chen  Na Li  Yakai Wang
Xiaoxiao Wu  Department of Computer Science and Technology, Huaqiao
University, Xiamen 361021, Fujian, China  Key Laboratory of Computer Vision
and Machine Learning of Fujian Provincial Universities, Xiamen 361021, Fujian,
China  Department of Mechanical Engineering and Automation, Huaqiao
University, Xiamen 361021, Fujian, China

###### Abstract

In driving scenarios, automobile active safety systems are increasingly
incorporating deep learning technology. These systems typically need to handle
multiple tasks simultaneously, such as detecting fatigue driving and
recognizing the driver’s identity. However, the traditional parallel-style
approach of combining multiple single-task models tends to waste resources
when dealing with similar tasks. Therefore, we propose a novel tree-style
multi-task modeling approach for multi-task learning, which rooted at a shared
backbone, more dedicated separate module branches are appended as the model
pipeline goes deeper. Following the tree-style approach, we propose a multi-
task learning model for simultaneously performing driver fatigue detection and
face recognition for identifying a driver. This model shares a common feature
extraction backbone module, with further separated feature extraction and
classification module branches. The dedicated branches exploit and combine
spatial and channel attention mechanisms to generate space-channel fused-
attention enhanced features, leading to improved detection performance. As
only single-task datasets are available, we introduce techniques including
alternating updation and gradient accumulation for training our multi-task
model using only the single-task datasets. The effectiveness of our tree-style
multi-task learning model is verified through extensive validations.

###### keywords:

Multi-task learningFatigue Driving DetectionFace RecognitionAlternating
UpdatesGradient Accumulation

##  1 Introduction

In the driving scenario, as automobile active safety systems increasingly
embrace deep learning technology, more deep learning techniques are being
applied in automobiles. Mature deep learning technologies are typically
designed for targeted tasks, necessitating the deployment of numerous models
within an automotive system to accomplish various tasks. Previous research has
focused on optimizing the performance of individual tasks and has proposed
many deep learning-based methods.

When facing multiple tasks, as the approach of parallelly combining multiple
single-task models is simple and straight forward, so this approach is usually
adopted in the initial of multi-task learning paradigm, which is referred to
as parallel-style approach. However, tasks in driving scenarios often exhibit
some strong similarities. For example, commercial vehicles such as buses need
to monitor both the driver’s fatigue status and their identity, both of which
require extracting features from the driver’s facial images.

Although excellent algorithms exist for both fatigue detection and face
recognition, the parallel-style approach for the joint task of fatigue
detection and face recognition would lead to significant redundant
computations and resource wastage due to the similarity between these tasks.
As both tasks rely on the facial images of the driver for assessment, there
must be shared underlying features between these two tasks. Therefore, more
efficient multi-task learning model is required.

Furthermore, most datasets in the literature are initially constructed and
annotated for a targeted task, whereas multi-task learning requires datasets
with multiple annotations for various tasks, making them usually unable to be
trained using single-task datasets.

In this paper, we deal with the two closely related tasks of fatigue driving
detection and face recognition of drivers. Considering that both fatigue
driving detection and face recognition require the same input image, we
propose a novel tree-style approach for multi-task modeling. In this modeling
approach, all tasks share a common feature extraction network backbone as the
root module, and as the model pipeline goes deeper, dedicated module branches
are appended, with the following branches being more dedicated to the final
tasks. Finally, for each task, a dedicated task head is appended. Thus, the
model pipeline forms a tree structure. The entire tree is referred to as a
tree-style multi-task model, and the model part corresponding to a node in the
tree is called a module. The model is sometimes called a tree-model for
emphasizing its tree-structure. Currently, there isn’t a tree-style multi-task
model targeted for fatigue detection and face recognition of drivers.

In addition, existing datasets are initially constructed and labeled for one
dedicated task. To train our multi-task model using existing single-task
datasets, we adopted two training techniques including alternately updating
and gradient accumulation. Alternately updating was used to update the weights
of the generator and the discriminator in GAN models[1]. Gradient accumulation
was initially used to address out-of-memory issues by imitating the training
process with large batches through the accumulation of gradients over multiple
small batches, thus reducing memory consumption.

Following a tree-style approach, we propose a multi-task learning model to
simultaneously perform driver fatigue detection and face recognition for
driver identification. A Convolutional Neural Network (CNN) based backbone
serves as the root module, shared by both tasks. Then, following the root
module are two branch modules, which are dedicated to fatigue driving
detection and face recognition of drivers, respectively. The two branch
modules have similar structures but with different parameters. Each branch
module contains a LANet, a SENet, and a residual connection as three parallel
branches, thus it is named LASE-Net. In a LASE-Net module, the LANet module is
dedicated to emphasizing channel attention, while the SENet module tries to
exploit spatial attention. Thus, both channel attention and spatial attention
are employed to obtain better features. The residual connection facilitates
the training process for better convergence. Our entire tree-style model is
named a Tree-style Space-Channel Attention Fusion (T-SCAF) model. Extensive
experiments were conducted, and the results show that our T-SCAF model
achieves state-of-the-art performance on average.

The contributions in this paper are summarized as follows:

  * •

We propose a tree-style multi-task modeling approach for multi-task learning,
where all tasks share a common feature extraction network backbone as the root
module, and as the model pipeline goes deeper, more dedicated separate modules
are branched further, with the following branches being more dedicated to the
tasks.

  * •

We propose a Tree-style Space-Channel Attention Fusion (T-SCAF) model for
simultaneously conducting the two tasks of driver fatigue detection and driver
identification.

  * •

We introduce alternating updation and gradient accumulation techniques to
train our T-SCAF model, enabling it to be trained using single-task datasets.

The remaining sections are organized as follows. Section 2 reviews related
work. Section 3 provides our T-SCAF model; Section 5 introduces the
alternating updation and gradient accumulation techniques for training T-SCAF.
Section 5 provides the experiment results. Finally, Section 6 presents the
conclusion.

##  2 Related Work

###  2.1 Fatigue Detection

Fatigue detection methods based on driver behavior characteristics aim to
discern whether a driver is experiencing fatigue by analyzing the driver’s
facial images. This is primarily achieved by identifying features such as eye
movements, mouth movements, head posture, and facial expressions. Several
studies have proposed deep learning models specifically for fatigue detection,
achieving promising performance.

Ref. [2] introduced a dual-stream network, where each stream extracts two
feature steams from respectively of the left eye and mouth images parts. They
fused the features extracted from these four streams to generate detection
results, achieving an accuracy of 91.3% on their self-collected dataset. Ref.
[3] utilized ResNext101 as the backbone network and added a Weibull Pooling
layer afterward. This model extracts features from input facial images and
performs classification, achieving an accuracy of 84.21%.

###  2.2 Face Recognition

Face recognition is a biometric authentication technology aimed at confirming
an individual’s identity by analyzing and identifying biological features in
the facial image. Face recognition models extract features from facial images
by employing convolutional neural networks, then map these features into high-
dimensional vectors. Typically, the similarity between different facial images
is determined by calculating the distance (e.g., Euclidean distance, cosine
distance) between their feature vectors, allowing for the identification of
whether two facial images belong to the same person.

FaceNet[4] introduced a novel task which adopts three images as input: two
images of the same person from different angles, while the remaining image
contains a different person. The objective is to minimize the feature distance
between images of the same person and maximize the distance between features
of different individuals. ArcFace[5] improves upon softmax loss by introducing
the ArcFace loss, which expands the inter-class margin during classification
training, which enhances recognition accuracy.

###  2.3 Multi-Task Learning

Multi-task learning models typically consist of multiple branches, with each
dedicated to a specific task. The former (shallower) layers of a model are
expected to extract shared feature representations, while a task-dedicated
branch is responsible for outputting predictions for its targeted task. This
allows the shared representations at the former layers to be further-exploited
at later (deeper) layers through the optimization process of targeted tasks,
enhancing the model’s ability to adapt to each task. Based on how the model
parameters are shared, multi-task models are categorized into soft parameter
sharing type and hard parameter sharing type.

In soft parameter sharing type multi-task model, each task typically has its
independent module. However, the modules of different tasks share parameters
of certain layers through various mechanisms, as demonstrated in works[6, 7].
This type of multi-task model corresponds to what we refer to as the parallel-
style modeling approach.

In hard parameter sharing type multi-task model, the former layers of the
model (typically the first few layers) are shared, while each task has its
independent later-layer branch module. This means that each task shares the
same feature extractor, but there are separate branch modules containing task-
specific layers for generating task-specific outputs, as seen in works[8, 9].
This type of multi-task model corresponds to what we refer to as the tree-
style modeling approach, yet their tree structures are restricted to 2-layer
depth.

Models such as Mask R-CNN[10] simultaneously perform tasks like image
segmentation, object recognition, and classification. Similarly, MTCNN[11] and
RetinaFace[12] can detect faces and facial landmarks concurrently.

##  3 The T-SCAF Model for Fatigue Detection and Face Recognition of Drivers

###  3.1 Tree-style Multi-Task Modeling Approach

We propose a conceptual tree-style multi-task modeling approach to multi-task
learning. In this approach, all tasks share a common feature extraction
network backbone as the root module. As the model pipeline progresses, more
dedicated separate modules branch out further, with subsequent branches being
more specialized for the final tasks. Finally, each task has a dedicated task
head. Thus, the model pipeline forms a tree.

In the tree-style multi-task modeling approach, the branching points depend on
the similarity levels between related tasks as well as the model structures
for the tasks. The tasks may be grouped hierarchically, which can be directly
mapped to the tree structure of a tree-style multi-task model.

Our tree-style multi-task modeling approach contains the so-called hard
parameter sharing type multi-task learning as a special case, where the depth
of the tree-style model equals two.

###  3.2 Tree-style Space-Channel Attention Fusion (T-SCAF) model

Focusing on the two closely related tasks of fatigue detection and face
recognition of drivers, adopting the tree-style multi-task modeling approach,
we propose our Tree-style Space-Channel Attention Fusion (T-SCAF) model, which
exploits and fuses both space and channel attention mechanisms to produce more
representative features for promoting the detection performance. The structure
of the T-SCAF model is illustrated in Fig. 1.

Figure 1: Structure of our tree-style space-channel attention fusion (T-SCAF)
model for fatigue detection and face recognition of drivers.

We use the same backbone network as the root module of the tree-model, which
is used to extract feature matrices from images for both fatigue detection and
face recognition tasks. Thus, both tasks share the same feature matrix
generated by the backbone module. The root module is followed by two branch
modules, each of which contains a LANet module, a SENet module, and a residual
connection, and hence such a branch module is called a LASE-Net module. The
two LASE-Net branch modules are respectively dedicated to fatigue detection
and face recognition. In a LASE-Net module, the LANet module is dedicated to
emphasizing space attention, while the SENet module tries to exploit channel
attention, thus both space attention and channel attention are employed to
obtain more representative features. The residual connection facilitates the
training process for better convergence.

###  3.3 The LANet Module

The LANet module is proposed in [13], here it is adopted to achieve space
attention. LANet is a convolutional structure composed of 2 layers of
$1{\times}1$ convolutions followed by ReLU and Sigmoid activation functions,
respectively. The pipeline of the SENet is illustrated in the lower part of of
the lower LASE-Net box in Figure 1.

The input feature matrix $\textbf{x}_{1}{\in}\mathbb{R}^{H{\times}W{\times}C}$
first passes through the a convolutional layer, reducing the channel dimension
of the feature matrix to $C/r$, where $r$ is a hyper-parameter defining the
feature channel compression ratio. Then the reduced features are passed to
ReLU activation functions. Next, the features pass through the second
convolutional layer to resize to $H{\times}W{\times}1$, thus reducing the
number of feature channels to 1. The feature matrix passes through the Sigmoid
layer to obtain the spatial attention weight matrix
$\textbf{Att}_{\text{space}}{\in}\mathbb{R}^{H{\times}W{\times}1}$. Till now,
all the channels of a pixel in the input feature matrix is compressed into a
scale value in the final matrix, thus the scale value reveals the relative
importance of each pixel in some sense. As a result, the final
$\textbf{Att}_{\text{space}}{\in}\mathbb{R}^{H{\times}W{\times}1}$ encloses
the space attention information.

The obtained spatial attention weight matrix $\textbf{Att}_{\text{space}}$ is
duplicated along the channel dimension to change shape to
$H{\times}W{\times}C$, which is also the shape of $\textbf{x}_{1}$. Weighting
the elements in $\textbf{x}_{1}$ using $\textbf{Att}_{\text{space}}$, i.e., by
element-wise multiplication of $\textbf{x}_{1}$ and
$\textbf{Att}_{\text{space}}$, we obtain the space-attention adjusted feature
matrix $\textbf{x}_{\text{LANet}}$.

In summary, the LANet module generates $\textbf{x}_{\text{LANet}}$ from
$\textbf{x}_{1}$ following Eq. (1).

$\textbf{x}_{\text{LANet}}{=}\text{dup}\left(\textbf{Att}_{\text{space}},[3],[C]\right){\odot}\textbf{x}_{1}.$
(1a)
$\textbf{Att}_{\text{space}}{=}\sigma\left(\text{CNN}\left(\text{ReLU}\left(\text{CNN}\left(\textbf{x}_{1}\right)\right)\right)\right).$
(1b)

In Eq. (1),
$\text{dup}(\cdot,[i_{1},i_{2},\ldots,i_{n}],[j_{1},j_{2},\ldots,j_{n}])$
represents the replication function to enlarge a matrix by enlarging the
$i_{k}$-th dimention by $j_{k}$ times, $k{\in}\\{1,2,\ldots,n\\}$. Here in Eq.
(1), the matrix is enlarged from size $H{\times}W{\times}1$ to
$H{\times}W{\times}C$. $\sigma(\cdot)$ and $\mathrm{ReLU}(\cdot)$ represent
the corresponding activation functions, $\text{CNN}(\cdot)$ denotes the
function of a convolution layer, $\mathrm{avg}(\cdot)$ represents the global
average pooling function, and $\odot$ represents the Hadamard product.

###  3.4 The SENet Module

The SENet module is proposed in [14], here it is adopted to achieve channel
attention. SENet is a fully connected network consisting of one global average
pooling layer, two fully connected layers, and ReLU and Sigmoid activation
function layers. The pipeline of the SENet is illustrated in the upper part of
the lower LASE-Net box in Figure 1.

The input feature matrix $\textbf{x}_{1}{\in}\mathbb{R}^{H{\times}W{\times}C}$
first passes through the global average pooling layer, resulting in a new
feature matrix with size
$\textbf{x}_{2}{\in}\mathbb{R}^{1{\times}1{\times}C}$. The $m$-th channel of
$\textbf{x}_{2}$ is calculated as Eq. (2).

$\textbf{x}_{2,m}=\text{avg}(\textbf{x}_{1}){=}\frac{1}{H{\times}W}\left(\sum_{i=0}^{H}\sum_{j=0}^{W}x{i,j,m}\right)$
(2)

The feature matrix $x_{2}$ then passes through the first fully connected
layer, resulting in a new feature matrix $x_{3}$ with a reduced channel
dimension of $C/16$. $x_{3}$ then passes through the ReLU activation layer.
Subsequently, the feature matrix passes through the second fully connected
layer and the Sigmoid activation layer, resulting in a channel attention
weight vector $\textbf{Att}_{channel}$ of size $1{\times}1{\times}C$. Till
now, each feature slice in the feature matrix $x_{1}$ is compressed into a
scale value in $\textbf{Att}_{\text{channel}}$, thus the scale value reveals
the relative importance of each channel in some sense. As a result,
$\textbf{Att}_{\text{channel}}{\in}\mathbb{R}^{1{\times}1{\times}C}$ encloses
the channel attention information.

$\textbf{x}_{\text{channel}}$ is duplicated to size of $H{\times}W{\times}C$
to obtain a channel attention matrix $\textbf{x}_{4}$. $\textbf{x}_{4}$ and
$\textbf{x}_{1}$ have the same shape, so they are combined into a channel-
attention adjusted feature matrix $\textbf{x}_{\text{SENet}}$ by element-wise
multiplication. In summary, the SENet module generates
$\textbf{x}_{\text{SENet}}$ from $x_{1}$ following Eq. (3).

$\textbf{x}_{\text{SENet}}{=}\text{dup}\left(\textbf{Att}_{\text{channel}},[1,2],[H,W]\right){\odot}\textbf{x}_{1}$
(3a)
$\textbf{Att}_{\text{channel}}{=}\sigma\left(\text{FC}\left(\text{ReLU}\left(\text{FC}\left(\text{avg}(\textbf{x}_{1})\right)\right)\right)\right).$
(3b)

In Eq. (3), the replication function $\text{dup}_{(}\cdot)$ enlarges the shape
size of the input matrix from $1{\times}1{\times}C$ to $H{\times}W{\times}C$.
$\mathrm{FC}(\cdot)$ denotes the function of the full connection layer,
$\mathrm{avg}(\cdot)$ represents the global average pooling function.

###  3.5 Space-Channel Attention Fusion Module LASE-Net

The original feather matrix $x_{1}$ not only passes the parallel LANet and
SENet branches respectively to generate two attended feature matrix
$\textbf{x}_{\text{LANet}}$ and $\textbf{x}_{\text{SENet}}$, but also directly
connects to the end of the two branches via a residual connection.
$\textbf{x}_{\text{LANet}}$, $\textbf{x}_{\text{SENet}}$, and $\textbf{x}_{1}$
are element-wisely added together to obtain a final space-channel attention
fused feature matrix $\textbf{x}_{\text{AttFused}}$ with shape
$H{\times}W{\times}C$, as expressed in Eq. (4).

$\textbf{x}_{\text{AttFused}}{=}\textbf{x}_{1}{+}\textbf{x}_{\text{LANet}}{+}\textbf{x}_{\text{SENet}}$
(4)

The combined module consisting of a LANet, a SENet, and a residual connection
as three parallel branches is named as a LASE-Net. In a LASE-Net module, the
LANet devotes to channel attention, while SENet devotes to spatial attention,
thus both channel attention and spatial attention are employed, resulting into
a better feature matrix. The residual connection facilitates the training
process for better convergence.

###  3.6 Dedicated Branches for Fatigue Detection and Face Recognition

The fatigue detection branch consists of one LASE-Net module followed by one
fully connected layer. The shared feature matrix outputted by the root module
undergoes feature extraction through the LASE-Net, then passes the fully
connected layer to obtain the fatigue detection result of yes or no.

Similarly, the face recognition branch also consists of one LASE-Net module
and one fully connected layer. The shared feature matrix outputted by the root
module undergoes feature extraction through the LASE-Net, then is mapped to a
512-dimensional feature vector by the fully connected layer. The cosine
distance is calculated between this feature vector and the feature vectors in
the face database to determine its identity.

##  4 Training Techniques for T-SCAF

Existing datasets are initially constructed and labeled for one dedicated
task. To train our T-SCAF multi-task model using existing single-task
datasets, we adopted two techniques including alternating updation and
gradient accumulation.

###  4.1 Alternating Updation

This technique is inspired by GAN. A GAN model contains two components: a
generator and a discriminator. The generator generates a batch of fake
samples, which are fed into the discriminator along with real samples. The
discriminator’s objective is to distinguish between real and fake samples and
compute the loss to update its parameters. The generator’s objective is to
make the discriminator misclassify the samples it generated as real one, and
based on this to compute the loss to update the generator’s parameters. These
two steps alternate until the model converges.

In our multi-task model for fatigue detection and face recognition, we also
employ alternating parameter updation. When updating the fatigue detection
branch, fatigue detection data is sampled and fed into the network for forward
and backward propagation. The loss and gradients are computed based on the
output of the fatigue detection branch, and the parameters of the root module
and the fatigue detection branch are updated whereas the face recognition
branch is frozen. When updating the face recognition branch, face recognition
data is sampled and fed into the network. The loss and gradients are computed
based on the output of the face recognition branch, and the parameters of the
root module and the face recognition branch are updated whereas the fatigue
detection branch is frozen. These two steps alternate until the model
converges.

###  4.2 Gradient Accumulation

Gradient accumulation is initially devised to train models with large batch
size under limited hardware resources. The main idea is to partition a large
batch of data into smaller batches. During each iteration, a small batch of
data is processed for forward and backward propagation to compute the
gradients. However, instead of immediately updating the model parameters, the
gradients are accumulated. After processing all the small batches, the
accumulated gradients are then used to update the model parameters, thereby
mimicking the training with a large batch size.

We apply this concept to the training of a multi-task model. Currently, we
only have labeled datasets for single tasks: fatigue detection and facial
recognition. In each step, we take one batch of data for fatigue detection and
one batch of data for facial recognition. We then perform forward and backward
propagation for each task independently, accumulating gradients without
updating model parameters. After completing forward and backward propagation
for both tasks, we compute the accumulated gradients and update the model
parameters together. This approach enables us to train a multi-task model
using single-task datasets.

We use cross-entropy loss in the fatigue detection branch, as shown in Eq.
(5), where $N$ is the total number of samples, $y_{i}$ is the label value, and
$p_{i}$ is the probability predicted by the network.

$L_{\text{Drowsy}}{=}\frac{1}{N}\sum_{i}{-}[y_{i}{\cdot}\log(p_{i}){+}(1{-}y_{i}){\cdot}\log(1{-}p_{i})].$
(5)

In the face recognition branch, ArcFace loss[5] is employed, which is defined
in Eq. (6).

$L_{\text{Face}}={-}\log\frac{e^{s\cdot\cos\left(\theta_{y_{i}}{+}m\right)}}{e^{s\cdot\cos\left(\theta_{y_{i}}{+}m\right)}{+}\sum_{j{=}1,j{\neq}y_{i}}^{N}e^{s\cdot\cos\theta_{j}}}.$
(6a)
$\theta_{j}{=}\arccos\bigl{(}\max_{k}\bigl{(}\textbf{W}_{i_{k}}^{\text{T}}\mathbf{x}_{i}\bigr{)}\bigr{)},k{\in}\\{1,\cdots,K\\},W{\in}\mathbb{R}^{512{\times}N{\times}K}$
(6b)

In Eq. (6), $\theta_{y_{i}}$ denotes the angle between the extracted feature
vector and the true label; $m$ is the penalty hyper-parameter set to 0.5; $s$
is the scaling parameter set to 20; $N$ is the number of classes for training,
and $K$ is a hyper-parameter set to 3. $\textbf{W}^{\text{T}}$ represents the
transpose of matrix W. $\textbf{W}_{i}$ represents the $i$-th column vector of
matrix W. We set the weight of the fatigue detection loss to $w$ and the face
recognition loss to $1{-}w$. Therefore, our overall optimization objective is
expressed in Eq. (7).

$L{=}w{\cdot}L_{\text{Drowsy}}{+}(1{-}w){\cdot}L_{\text{Face}}.$ (7)

##  5 Performance Evaluation

In this section, we first introduce the dataset used, and then provide
experiment details. To validate the effectiveness of our method, we present
the results of ablation experiments. As there are no publicly available works
specifically addressing the joint tasks of fatigue detection and face
recognition, we compare our method with other single-task methods for fatigue
detection and face recognition.

###  5.1 Datasets

We trained our model on a self-built fatigue detection dataset and the CASIA-
Webface dataset [15]. We validated the fatigue detection branch and face
recognition branch respectively on our self-built fatigue detection dataset
and the Labeled Faces in the Wild (LFW) dataset [16].

Self-built Dataset: The self-built dataset includes two types of video
recordings: (1) surveillance footage from buses operating in Xiamen, China and
(2) simulated driving videos recorded with RGB cameras in stationary vehicles.
Surveillance footage contains the driving experiences of 32 drivers, including
videos in a diverse range of conditions. Videos from the bus surveillance
systems, taken with grayscale cameras, span daytime and nighttime driving,
present a realistic spectrum of lighting conditions. The simulated driving
videos, recorded with RGB cameras, represent daytime driving scenarios. Our
dataset further includes challenging situations, such as low illumination,
vital facial features such as the mouth or eyes concealed.

We extracted images from the video data at intervals of 10 frames and labeled
each frame as either ’Drowsy’ or ’No Drowsy’. Subsequently, these labeled
images were systematically segregated into a training set and a test set at a
9:1 ratio to facilitate the development and evaluation of our analytical
model.

Fig. 2 shows some sample pictures of drivers in our dataset in both drowsy and
no-drowsy status.

Figure 2: Images in drowsy and no-drowsy status in the self-built dataset

CASIA-WebFace [15]: CASIA-WebFace is a large-scale face recognition dataset
created and maintained by the Institute of Automation, Chinese Academy of
Sciences. One of its notable features is the inclusion of 494,414 face images
sourced from the Internet, covering over 10,000 unique identities. This
dataset exhibits diversity in facial expressions, poses, and lighting
conditions, making it a challenging benchmark for testing. Each image comes
with corresponding identity labels, providing information about the
individuals depicted, thus making the dataset suitable for training and
evaluating face recognition methods.

Labeled Faces in the Wild [16]: Labeled Faces in the Wild (LFW) is a public
dataset frequently used for face recognition research. It comprises over
13,000 real-world face images collected from the Internet, each accompanied by
corresponding identity labels. Widely employed for evaluating the performance
of face recognition methods, LFW presents challenges such as diverse facial
expressions, occlusions, and varying lighting conditions, reflecting real-
world complexity. The common evaluation protocol for this dataset is one-to-
one mode, which involves determining whether a pair of face images depict the
same individual.

###  5.2 Implementation Details

All models were implemented using the PyTorch deep learning framework in
Python. Throughout our experiments, training and testing were conducted on a
computer equipped with an RTX 3090 GPU. The computer features an AMD Ryzen
Threadipper Pro 3995WX CPU and 64GB of RAM, running Ubuntu 22.04.

For a 2-class classification method, we use True Positive (TP) to denote the
number of cases correctly predicted as positive, False Positive (FP) to
indicate the number of cases wrongly predicted as positive, True Negative (TN)
to represent the number of cases correctly predicted as negative, and False
Negative (FN) to signifie the number of cases wrongly predicted as negative.
Fig. 3 shows the relationships among them.

Figure 3: Confusion Matrix

We utilized accuracy (ACC) to evaluate the fatigue detection performance of
the T-SCAF model. ACC can be calculated as Eq. (8).

$\text{ACC}=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{TN}+\text{FP}+\text{FN}}$
(8)

As for experiment details, we utilized RetinaFace[12] to extract facial
regions from the original images and manually adjusted them for calibration.
All extracted facial images were resized to $112{\times}112$ pixels. For the
root module of our T-SCAF model, we used IResNet50[17], which is the backbone
network of ArcFace[5]. We utilized the Adaptive Moment Estimation optimizer
(ADAM) to optimize the parameters, with an initial learning rate of 1e-2 and
momentum of 0.9. The model was trained for 20 epochs with a batch size of 512,
and the learning rate was adjusted every 20 epochs using CosineAnnealingLR.
For both gradient accumulation and alternating updation, we set the data
sampling method to sample a batch of data alternately from the fatigue
detection dataset and the face recognition dataset.

###  5.3 Comparative Experiment

We compared our model with single-task models for fatigue detection and face
recognition, and the quantitative results are shown in Table 1 and Table 2.

Compared to the methods in [2, 18, 19, 20], our model achieved an accuracy
improvement of 3.2%, 0.61%, 0.96%, and 1.18% respectively in the fatigue
detection task. Compared to ArcFace, our model’s accuracy decreased by 0.77%
in the face recognition task. Compared to combination methods combining each
of [2, 19, 20] and the ArcFace, our model achieved an average accuracy
improvement of 1.18%, 0.1%, 0.21% respectively in both tasks. It is noteworthy
that our model achieved the results with fewer parameters and computations
than the three aforementioned single-task model combination methods.

Compared to a method combining the one in [18] and ArcFace, our model’s
average accuracy decreased by 0.08%, but our model has only 1/7 parameters and
requires only 1/5 computations. Compared to the combination method of our
previously proposed fatigue detection model MAF [21] and ArcFace, our T-SCAF
model’s average accuracy decreased by 0.77%, but nearly half parameters and
computations are saved. Additionally, we replaced the root backbone module of
T-SCAF model with the more lightweight MobileNetV2 [22], resulting into a mode
with 1/3 parameters and 1/50 computations of the previous model using
IResNet50[17], whereas with a trivial penalty of accuracy reduction of 4.77% .

|  Fatigue Detection Model |  Face Recognition Model |  Model Size(M)$\downarrow$ |  GFLOPs$\downarrow$
---|---|---|---|---
Parallel-style multi-task model | Method[2] | ArcFace[5] | 53.48 | 17.00
Method[18] | ArcFace[5] | 315.39 | 64.73
Method[19] | ArcFace[5] | 157.61 | 13.89
Method[20] | ArcFace[5] | 85.72 | 16.81
MAF[21] | ArcFace[5] | 110.09 | 25.49
Tree-style multi-task model | Ours(MobileNetV2) | 14.77 | 0.25
Ours(IResNet50) | 43.92 | 12.72
Table 1: Model size and FLOPs of different models |  Fatigue Detection Model |  Face Recognition Model |  ACC$\uparrow$ (Fatigue Detection) |  ACC$\uparrow$ (Face Recognition) |  ACC$\uparrow$ (Avg)
---|---|---|---|---|---
Parallel-style multi-task model | Method[2] | ArcFace[5] | 0.9297 | 0.9620 | 0.9459
Method[18] | ArcFace[5] | 0.9549 | 0.9620 | 0.9585
Method[19] | ArcFace[5] | 0.9514 | 0.9620 | 0.9567
Method[20] | ArcFace[5] | 0.9492 | 0.9620 | 0.9556
MAF[21] | ArcFace[5] | 0.9688 | 0.9620 | 0.9654
Tree-style multi-task model | Ours(MobileNetV2) | 0.9091 | 0.9108 | 0.9100
Ours(IResNet50) | 0.9610 | 0.9543 | 0.9577
Table 2: ACC of various models

###  5.4 Ablation Experiments

Backbone | SENet | LANet | ACC(Fatigue Detection)$\uparrow$ | ACC(Face Recognition)$\uparrow$ | ACC(Avg)$\uparrow$
---|---|---|---|---|---
$\checkmark$ | $\checkmark$ |  | 0.9318 | 0.9033 | 0.9176
$\checkmark$ |  | $\checkmark$ | 0.9578 | 0.9465 | 0.9522
$\checkmark$ | $\checkmark$ | $\checkmark$ | 0.9610 | 0.9543 | 0.9577
Table 3: Results of ablation experiments

We conducted ablation experiments by progressively removing modules from the
model to validate the effectiveness of each module. The quantitative results
of the ablation experiments are shown in Table 3. We added attention modules
to both branches to separate the features that are more useful for each task
extracted from the root backbone module. We found that When only adding the
LANet module, the accuracy of fatigue detection and face recognition tasks
increased by 1.95% and 0.25%, respectively. When adding both SENet and LANet
simultaneously, the inclusion of SENet lead to futher performance improvement,
with the accuracy of the fatigue detection and face recognition tasks
increased by 2.27% and 1.03%, respectively. However, when only adding the
SENet module, both the accuracy of fatigue detection and face recognition
tasks decreased with varying degrees, by 0.65% and 4.13%, respectively. This
may because that, channel attention ability of SENet reduces the feature
weights across the entire channel dimension, which may impair the
effectiveness of the features.

We also compared the two training methods of alternating updation and gradient
accumulation. The quantitative results are shown in Table 4. The alternating
updation method achieved the highest accuracy when the two tasks are combined.
The gradient accumulation training method allows controlling the influences of
the two tasks on model updating by adjusting the loss weight. With an increase
in $w$, the overall accuracy of fatigue detection tends to improve, while the
accuracy of face recognition first increases and then decreases. The
comprehensive accuracy of fatigue detection and face recognition reaches its
maximum when $w$ is set to 0.5. Additionally, we found that gradient
accumulation requires larger GPU memory resources compared to alternating
updation, as it involves storing the results of two backward passes during
training.

Training techniques | ACC(Fatigue Detection)$\uparrow$ | ACC(Face Recognition)$\uparrow$ | ACC(Avg)$\uparrow$
---|---|---|---
Alternating updation | 0.9610 | 0.9543 | 0.9577
Gradient accumulation | $w$ |
0.1 | 0.8994 | 0.9318 | 0.9156
0.2 | 0.8506 | 0.9085 | 0.8796
0.3 | 0.8474 | 0.9107 | 0.8791
0.4 | 0.9286 | 0.9412 | 0.9359
0.5 | 0.9545 | 0.9473 | 0.9509
0.6 | 0.9513 | 0.9243 | 0.9378
0.7 | 0.9513 | 0.9117 | 0.9315
0.8 | 0.9513 | 0.9270 | 0.9392
0.9 | 0.9610 | 0.9183 | 0.9397
Table 4: Impacts of loss weights of alternating updation and gradient
accumulation training techniques.

Furthermore, to verify that the root backbone module extracts shared universal
features that can be used for both fatigue detection and face recognition
tasks, we split the two tasks in the multi-task model into two separate
modules, each composed of the same copy of the backbone module and its
corresponding LASE-Net branch module. The quantitative results are shown in
Table 5. Compared with parallel-style multi task model, our T-SCAF multi-task
model achieves improvements in accuracy respectively by 0.4% and 0.33% in the
fatigue detection and face recognition tasks. The results indicates that, the
T-SCAF multi-task model enables both tasks to share underlying feature
representations, allowing the model to learn universal features. Additionally,
each task in the multi-task model can be considered as a regularization term
for the other task. As the model needs to find a balance between multiple
tasks, it tends to mitigate over-fitting and improve generalization
performance on unseen data, thus achieving better results on the test dataset.

Model | ACC(Fatigue Detection)$\uparrow$ | ACC(Face Recognition)$\uparrow$
---|---|---
Backbone(root module) + Fatigue detection branch | 0.9570 |
Backbone(root module) + Face recognition branch |  | 0.9510
Ours | 0.9610 | 0.9543
Table 5: Results of splitting our T-SCAF multi-task model into parallel-style
multi-task model.

###  5.5 Result Visualization

To further validate the effectiveness of the T-SCAF multi-task model, we
visualized the features of some intermediate layers.

The feature matrices returned by the LASE-Net module in the fatigue detection
branch and face recognition branch using the Grad-CAM method[23] are shown in
Fig. 4. The first row depicts the visualization of the feature matrix returned
by the LASE-Net module in the fatigue detection branch, whereas the second row
shows those of the face recognition branch. It can be observed that, for
different detection tasks, the two branches focus on different facial regions
from the shared features, which are extracted by the root backbone module. The
features emphasized by the fatigue detection branch are concentrated around
the eyes and mouth regions, whereas the features emphasized by the face
recognition branch are dispersed across different parts of the face.

Figure 4: Visualization of LASE-Net output feature maps

We visualized the feature vectors before the last fully connected layer in the
fatigue detection branch of two model configurations, one with only the root
backbone module, the other combines the root backbone module and an LASE-Net
module. These configurations are respectively referred as Backbone multi-task
model and Backbone+LASE-Net multi-task model. We used the t-SNE[24] technique
to embed high-dimensional features in a two-dimensional plane for better
interpretation. The visualization results are shown in Fig. 5, with the left
image representing the Backbone multi-task model and the right image
representing the Backbone+LASE-Net multi-task model. The results show that,
compared to the Backbone multi-task model, the Backbone+LASE-Net multi-task
model exhibits clearer boundaries between different categories and tighter
aggregation within the same category. This indicates that the LASE-Net module
enhances the feature extraction capability of the fatigue detection branch.

Figure 5: t-SNE visualization results.

The ROC curves for fatigue detection and face recognition tasks of our T-SCAF
multi-task model and the other comparison models are shown in Fig. 6. The Area
Under Curve (AUC) values for each model in different tasks are annotated along
the curve. In the fatigue detection task, T-SCAF’s AUC value is 0.022 higher
than that of the Basic model (which only includes the shared root backbone
module without the LASE-Net). It shares the same AUC value with the single-
task model MAF[21] and shows varying degrees of improvement compared to other
single-task models. In the face recognition task, T-SCAF’s AUC value is 0.004
higher than that of the Basic model. Additionally, even with a lighter model
size and faster computation speed, T-SCAF’s AUC is only 0.001 lower than the
single-task models. This demonstrates the effectiveness of our T-SCAF multi-
task model.

Figure 6: ROC curve comparison of different models

##  6 Conclusion

This paper proposes a novel tree-style multi-task modeling approach for multi-
task learning. which rooted at a shared backbone, more dedicated separate
module branches are appended as the model pipeline goes deeper. Following the
tree-style approach, we propose a tree-style space-channel attention fusion
(T-SCAF) network model for simultaneously performing driver fatigue detection
and face recognition for identifying a driver. This model shares a common
feature extraction backbone module, with further separated feature extraction
and classification module branches. The dedicated branches exploit and combine
spatial and channel attention mechanisms to generate space-channel fused-
attention enhanced features, leading to improved detection performance. we
also introduce alternating updation and gradient accumulation techniques for
training our T-SCAF model using only the single-task datasets. The experiments
on our self-built dataset, CASIA-WebFace, and LFW dataset validates the
effectiveness of our T-SCAF model.

## References

  * [1] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” Advances in Neural Information Processing Systems, vol. 27, 2014.
  * [2] B. Reddy, Y.-H. Kim, S. Yun, C. Seo, and J. Jang, “Real-time driver drowsiness detection for embedded system using model compression of deep neural networks,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 438–445, IEEE, 2017.
  * [3] S. S. Husain, J. Mir, S. M. Anwar, W. Rafique, and M. O. Ullah, “Development and validation of a deep learning-based algorithm for drowsiness detection in facial photographs,” Multimedia Tools and Applications, vol. 81, no. 15, pp. 20425–20441, 2022.
  * [4] F. Schroff, D. Kalenichenko, and J. Philbin, “FaceNet: A unified embedding for face recognition and clustering,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 815–823, IEEE, 2015.
  * [5] J. Deng, J. Guo, J. Yang, N. Xue, I. Kotsia, and S. Zafeiriou, “ArcFace: Additive angular margin loss for deep face recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 5962–5979, 2022.
  * [6] J. Ma, Z. Zhao, J. Chen, A. Li, L. Hong, and E. H. Chi, “SNR: Sub-network routing for flexible parameter sharing in multi-task learning,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 216–223, Jul. 2019.
  * [7] J. Ma, Z. Zhao, X. Yi, J. Chen, L. Hong, and E. H. Chi, “Modeling task relationships in multi-task learning with multi-gate mixture-of-experts,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1930–1939, Association for Computing Machinery, 2018.
  * [8] R. Cipolla, Y. Gal, and A. Kendall, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7482–7491, IEEE, 2018\.
  * [9] Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich, “Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,” in International Conference on Machine Learning, pp. 794–803, PMLR, PMLR, 2018\.
  * [10] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” in 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988, IEEE, 2017.
  * [11] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” IEEE Signal Processing Letters, vol. 23, no. 10, pp. 1499–1503, 2016.
  * [12] J. Deng, J. Guo, E. Ververas, I. Kotsia, and S. Zafeiriou, “RetinaFace: Single-shot multi-level face localisation in the wild,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5202–5211, IEEE, 2020.
  * [13] Q. Wang and G. Guo, “LS-CNN: Characterizing local patches at multiple scales for face recognition,” IEEE Transactions on Information Forensics and Security, vol. 15, pp. 1640–1653, 2020.
  * [14] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132–7141, IEEE, 2018.
  * [15] D. Yi, Z. Lei, S. Liao, and S. Z. Li, “Learning face representation from scratch,” arXiv preprint arXiv:1411.7923, 2014.
  * [16] G. B. Huang, M. A. Mattar, T. L. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” 2008.
  * [17] I. C. Duta, L. Liu, F. Zhu, and L. Shao, “Improved residual networks for image and video recognition,” in 2020 25th International Conference on Pattern Recognition (ICPR), pp. 9415–9422, IEEE, 2021.
  * [18] M. Dua, Shakshi, R. Singla, S. Raj, and A. Jangra, “Deep CNN models-based ensemble approach to driver drowsiness detection,” Neural Computing and Applications, vol. 33, no. 8, pp. 3155–3168, 2021.
  * [19] S. Park, F. Pan, S. Kang, and C. D. Yoo, “Driver drowsiness detection system based on feature representation learning using various deep networks,” in Proceedings of the Asian Conference on Computer Vision (ACCV), pp. 154–164, Springer, 2017.
  * [20] S. S. Husain, J. Mir, S. M. Anwar, W. Rafique, and M. O. Ullah, “Development and validation of a deep learning-based algorithm for drowsiness detection in facial photographs,” Multimedia Tools and Applications, vol. 81, no. 15, pp. 20425–20441, 2022.
  * [21] S. QU, Z. Gao, X. Wu, and Y. Qiu, “Multi-attention fusion drowsy driving detection model,” arXiv preprint arXiv:2312.17052, 2023.
  * [22] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “MobileNetV2: Inverted residuals and linear bottlenecks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510–4520, IEEE, 2018.
  * [23] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 618–626, IEEE, 2017.
  * [24] L. van der Maaten and G. Hinton, “Visualizing data using t-SNE,” Journal of Machine Learning Research, vol. 9, no. 86, pp. 2579–2605, 2008.