#  Activity-Biometrics: Person Identification from Daily Activities

Shehreen Azad Yogesh Singh Rawat
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Center for Research in Computer Vision, University of Central Florida

###### Abstract

In this work, we study a novel problem which focuses on person identification
while performing daily activities. Learning biometric features from RGB videos
is challenging due to spatio-temporal complexity and presence of appearance
biases such as clothing color and background. We propose ABNet, a novel
framework which leverages disentanglement of biometric and non-biometric
features to perform effective person identification from daily activities.
ABNet relies on a bias-less teacher to learn biometric features from RGB
videos and explicitly disentangle non-biometric features with the help of
biometric distortion. In addition, ABNet also exploits activity prior for
biometrics which is enabled by joint biometric and activity learning. We
perform comprehensive evaluation of the proposed approach across five
different datasets which are derived from existing activity recognition
benchmarks. Furthermore, we extensively compare ABNet with existing works in
person identification and demonstrate its effectiveness for activity-based
biometrics across all five datasets. The code and dataset can be accessed at:
https://github.com/sacrcv/Activity-Biometrics/

##  1 Introduction

Person identification is an important task with a wide range of applications
in security, surveillance, and various domains where recognizing individuals
across different locations or time frames is essential [44]. We have seen a
great progress in face recognition [2, 33], however scenarios exist where
faces may not be visible, such as at long distances, with uncooperative
subjects, under occlusion, or due to mask-wearing. This limitation prompts the
exploration of whole-body-based person identification methods where most of
the existing works are often restricted to image-based approaches [15, 5, 46],
overlooking crucial motion patterns. Video-based methods for person
identification is comparatively recent area where most of the work is focused
on gait recognition; mostly silhouette-based [28, 13, 14] with some recent
works on RGB frames [27, 47]. However these works are mainly focused on
walking style of individuals (see Figure 2).

Figure 1:  Different approaches for person identification: (left) samples for
existing person identification problems such as face recognition (top:
Celeb-A[31]), whole body recognition (middle: Market-1501[48]), and gait
recognition (bottom: CASIA-B[45]). (right) we focus on person identification
from daily activities which presents more challenges beyond learning walking
or facial patterns. We show some samples from datasets we used to study this
problem; (top: NTU RGB-AB, middle: Charades-AB, bottom: ACC-
MM1-Activities222The subjects consented to publication).

In this work, we study a novel problem which focuses on face-restricted person
identification during routine activities. The current landscape of image-based
and video-based whole-body person identification methods predominantly centers
around analyzing human walking patterns from images or videos [16, 43, 35,
21]. However, in real-world scenarios, the individual requiring identification
might not always be engaged in walking; instead, they could be involved in
various daily activities. It is crucial to acknowledge the significance of
capturing and understanding motion cues that extend beyond simple walking
patterns to ensure accurate and reliable identification in diverse and complex
situations. These activities may offer unique cues that can prove instrumental
in identifying individuals even without explicit facial information, paving
the way for diverse applications in real-world scenarios, like increased
surveillance in public spaces, workplace security and productivity, assistance
for people requiring special needs, and smart home automation.

Learning biometrics from videos of daily activities presents several inherent
challenges. Learning from such diverse activities amplifies the difficulty in
capturing essential biometrics features. Among the crucial challenges lies the
necessity to prioritize biometrics features while mitigating appearance biases
present in RGB video frames, including background variations, clothing color,
and other external factors. Striking a balance between extracting pertinent
biometrics cues and disregarding irrelevant appearance-related biases is
essential in developing robust and accurate video-based biometrics
identification methods.

We propose a novel framework ABNet, which addresses some of these challenges
and provides effective biometrics representation for person identification
from videos of daily activities. It relies on two main components; 1) feature
disentanglement, and 2) joint activity-biometrics learning. Feature
disentanglement aims at avoiding appearance biases while learning the
biometrics features. It explicitly learns biometrics and non-biometrics
features with the help of, a) distillation from bias-less teacher, and b) bias
learning using biometrics distortion. Joint activity-biometrics learning
provides activity prior for biometrics where the knowledge of performed
activity helps in person identification.

We present extensive evaluations on five different benchmarks using several
metrics comparing the proposed approach with several state-of-the-art person
identification methods including both image-based and video-based approaches.
This comprehensive evaluation demonstrates the effectiveness and superiority
of our proposed method in handling diverse datasets and scenarios for
activity-based biometrics. Our main contributions can be summarized as,

  * •

We study a novel problem of person identification from daily activities using
RGB videos.

  * •

We propose a simple and novel strategy to disentangle biometrics and non-
biometrics features from videos for person identification.

  * •

We show the benefits of activity-prior for biometrics.

  * •

We present several benchmarks to study this problem; these datasets are
dervied from existing activity recognition datasets specifically curated for
person identification.

##  2 Related work

Image-based identification: Most of the existing person identification methods
use image-based approach [7, 20, 22, 42, 5, 43]. Moreover, most of these
methods are designed towards learning better features in-terms of body shape,
clothes, appearance etc. In recent years, learning cloth invariant features is
found to be a promising direction in person identification with several works
trying to address this issue. For example, one of the most popular person
identification approach, CAL [16] uses advarsarial loss to learn cloth
invariant features. On the other hand, SCNet[17] uses a tri-stream network to
learn semantically invariant features. Some works also attempt to use multiple
modalities (e.g., silhouettes [24], skeletons [36], 3D shape [7]) etc. for
better feature representation. Even though the image based methods can have
better performance than some video-based methods, this performance is measured
on very specific datasets, which might or might not generalize to more complex
datasets where the person in consideration is performing some other activities
rather than walking.

Video-based identification: The key for video-based person identification is
to extract representations robust to spatial and temporal distractors. These
methods incorporate temporal information in their learned features and
generally have better performance than image based methods. Several previous
works [11, 46] have exploited temporal cues by aggregating frames features via
LSTM network. However, instead of using aggregated features extracted by RNNs,
3D CNNs perform better in terms of directly extracting spatio-temporal
features that are more robust for person identification [6, 29]. Following
current research direction [23, 35, 21, 40, 4], our work is also based on 3D
CNN.

Gait recognition: Gait recognition is a very active area of research where the
goal is to identify individuals using their walking style. Existing methods
mostly utilize silhouettes to avoid interference of appearance [14, 13, 28]
which limits their applicability on real-world RGB videos. There are some
approaches making use of RGB for gait recognition [27, 47], but they do
require silhouette in addition to RGB data. In our proposed method we only use
silhouette during training and it is not required for inference.

Figure 2: Overview of our proposed method ABNet. RGB video is passed to a
video encoder $S_{\varphi}(\cdot)$ for spatio-temporal feature $F_{AB}$
extraction which is passed to the activity head $C^{A}$ and the actor head
$C^{B}$. $C^{B}$ captures both biometrics (in red) and appearance (in green)
features in $F_{BT}$. To disentangle features, bias-less teacher encoder
$T_{\theta}(\cdot)$ distills biometrics knowledge from corresponding
silhouettes. The appearance feature bias is learned via a distortion network
using encoder $A_{\varphi}(\cdot)$ on the distorted video input. Similar to
$C^{B}$, $C^{DB}$ also captures both distorted biometrics (in red) and
distorted appearance (in green) features in $F^{D}_{BT}$. Here, green and red
denote positive and negative feature. Joint training is performed using both
$C^{A}$ and $C^{B}$. During inference, only the dashed box highlighted branch
is utilized.

Knowledge distillation: It is one of the most common techniques to transfer
knowledge from a large model (teacher) to a smaller model (student) for
compression and efficient learning [19]. It has also been found very effective
for semi-supervised learning where the models can learn from unlabelled
samples under student teacher setup [39]. In some recent efforts it was also
explored for person identification too for effective cross-view [35] and
cross-scene [41] representation learning. It has been mostly explored within
same modality, whereas we perform a cross-modal distillation to leverage the
teacher’s knowledge of a different data modality to improve the performance of
the student.

##  3 Method

Our goal is to identify an individual given an RGB video of that individual
performing some activity. We are using a face restricted setting to perform
this task, where the face of the individual is blurred so as to avoid learning
any of the facial features. Avoiding the explicit learning of facial features
is motivated by acknowledging potential issues like wearing accessory (masks,
sunglasses), privacy concerns, and individuals’ unwillingness to reveal their
faces.

Problem formulation: Given a dataset $D$ containing elements of
${v,y^{A},y^{B}}$ with $N$ samples, we want to train a person identification
model $M$ which can provide a latent feature $F_{AB}$ for each video $v$ which
can be used for matching it with the person id $y^{B}$. Here
$v\in\mathbb{R}^{nXCXHXW}$ represents an RGB video, where $n$ is the number of
frames, $C,H,W$ are the number of channels, height and width of the video, and
$y^{B}$ is its ground truth actor label that is performing some activity
$y^{A}$. Once trained, the model $M$ will be evaluated on a gallery
$G\in{v,y^{b}}$ and probe $P\in{v,y^{b}}$. The goal is to match the id of the
person $y^{b}$ in probe video $v$ with the correct id in videos from gallery.

Overview: We propose ABNet, Activity Biometrics Network, denoted as $M$ to
solve this problem. ABNet performs biometrics-bias disentanglement and make
use of activity prior to learn a discriminative identity feature for person
identification. Given a video $v$, the model $M$ first extracts spatio-
temporal features $F_{AB}$ with the help of a video encoder
$S_{\varphi}(\cdot)$. The spatio-temporal feature $F_{AB}$ is split into two
segments and are passed to the actor head $C^{B}$ for person identification as
well as the activity head $C^{A}$ for activity recognition. Joint biometrics
and activity learning enables the use of activity-prior for biometrics. We get
actor features $F_{BT}$ from $C^{B}$ that contains both biometrics and
appearance feature entangled with each other. Now to make the model robust to
appearance bias while learning accurate biometrics features, we introduce two
different components - 1) distillation from a bias-less teacher and learning
the bias using biometrics distortion. The actor feature $F_{BT}$ are
disentangled into biometrics feature $f_{bb}$ and appearance feature $f_{ba}$.
This disentanglement for biometrics feature $f_{bb}$ is performed using
distillation from a bias-less teacher $T$. On the contrary, the
disentanglement for appearance feature $f_{ba}$ is done by constraining it
using a distortion network $A$. An overview of the proposed method is shown in
Figure 2.

###  3.1 Biometrics bias disentanglement

Appearance bias in biometrics arises when the models overly rely on
superficial visual cues, such as clothing or specific accessories for
identification. This leads to challenges such as limited generalization across
appearances, vulnerability to adversarial attacks, and reduced robustness to
environmental variations. This bias can result in biased matching decisions,
and inconsistent performance across cameras. There has been extensive research
done to avoid clothing features for person reidentification [16, 17, 43],
however, appearance bias can come from features other than clothes as well. To
deal with this issue of appearance bias, we introduce two different aspects;
1) bias-less distillation from a teacher network, and 2) learning the bias
using negative mining through biometrics distortion.

Bias-less distillation: One split segment of the extracted feature $F_{AB}$ is
fed to the actor head $C^{B}$, which contains $D^{B}_{\omega}$ that is a
standard transformer decoder. We get actor feature $F_{BT}$ from
$D^{B}_{\omega}$, which contains biometrics feature $f_{bb}$ and appearance
feature $f_{ba}$. $D^{B}_{\omega}$ uses self-attention to process the input
sequence and then projects the attention output into $f_{bb}$ and $f_{ba}$
using separate linear layers. Now to disentangle the biometrics features from
the appearance features, we propose the use of silhouette features to perform
bias-less distillation using teacher network $T$. $T$ is termed as bias-less
because it is trained on binary silhouette video
$b_{s}\in\mathbb{R}^{nXCXHXW}$ that corresponds to RGB video $v$, and thus
have no knowledge of appearance based features. $T$ contains a silhouette
encoder $T_{\theta}(\cdot)$ that takes $b_{s}$ as input and extracts $F_{S}$
features. Following [19] we use the standard Kullback-Leibler (KL) divergence
loss to minimize the discrepancy between the probability distributions of the
teacher $T$ and our model $M$. The distillation loss $\mathcal{L}_{KD}$ is
formulated as below:

$\begin{split}\mathcal{L}_{KD}=\tau{{}^{2}}KL(y_{T}||y_{S}),\end{split}$ (1)

where, $y_{T}$ and $y_{S}$ are the probability distribution of the teacher $T$
and our model $M$. $\tau$ is the temperature parameter that controls the
softness of the teacher’s output. Along with this distillation loss
$\mathcal{L}_{KD}$, $C^{B}$ has its own biometrics loss $\mathcal{L}_{Bio}$
formulated as below:

$\mathcal{L}_{Bio}=\mathcal{L}_{ce}+\mathcal{L}_{tri},$ (2)

where, $\mathcal{L}_{ce}$ and $\mathcal{L}_{tri}$ are standard triplet and
cross-entropy losses for person identification formulated as below:

$\mathcal{L}_{ce}=-y\log\hat{y},$ (3)
$\mathcal{L}_{tri}=\max((D(f_{a},f_{p})-D(f_{a},f_{n})+m),0),$ (4)

where, $y$ and $\hat{y}$ are the ground truth and predicted label, $f_{p}$ and
$f_{n}$ are the positive and negative features for an anchor feature $f_{a}$
within the same batch, $D(\cdot)$ is the Euclidean distance function, and $m$
is the margin of triplet loss.

Bias learning: To make the model robust to appearance bias, we introduce the
distortion network $A$, which is identical to $M$ and shares weights. It
contains video encoder $A_{\varphi}(\cdot)$ that takes distorted video
$\hat{v}\in\mathbb{R}^{nXCXHXW}$ that corresponds to the original video $v$.
The key idea is to distort the identity of the person while preserving the
appearance. We rely on elastic transform [1] which randomly transforms the
morphology of objects in images and produces a see-through-water-like effect
in the image still preserving the appearance. It is used to generate
“negative” or “distractor” samples in the training dataset where the distorted
samples will have the same appearance while changing the identity. Some sample
distorted images are shown in Fig. 3.

Figure 3: Biometrics distortion: here original samples are shown in the top
row and their corresponding distorted samples in the bottom row. From left to
right, every two columns contain samples from NTU RGB-AB, PKU MMD-AB,
Charades-AB, ACC-MM1-Activities and BRIAR-BGC3 dataset respectively. The
subjects from BRIAR-BGC3 and ACC-MM1-Activities consented to publication.

Similar to $M$, this distortion network $A$ also extracts spatio-temporal
feature $F^{D}_{AB}$ using encoder $A_{\varphi}(\cdot)$. Since this branch is
designed for bias-learning, thus the activity head $C^{DA}$ of $A$ is not
utilized. On the contrary, $A$’s actor head $C^{DB}$ extracts distorted
biometrics feature $f^{D}_{bb}$ and distorted appearance feature $f^{D}_{ba}$.
Due to the distortion, $f_{ba}$ and $f^{D}_{ba}$ are treated as positive
samples, whereas, $f_{bb}$ and $f^{D}_{bb}$ as hard negative samples. The goal
is to pull together positive pairs (i.e. similar features) and push apart
negative pairs (i.e. dissimilar features). We use this distorted augmentation
loss $\mathcal{L}_{Dis}$ for bias learning and it is described as,

$\mathcal{L}_{Dis}=max((D(f_{ba},f^{D}_{ba})-D(f_{bb},f^{D}_{bb})+m),0),$ (5)

where $D(\cdot)$ is the Euclidean distance function and $m$ is the margin for
the contrastive loss.

###  3.2 Joint biometrics and activity learning

Jointly training a network for both activity recognition and person
identification can benefit person identification when the training data
includes activities by enabling the model to learn shared representations. By
learning to understand contextual cues from activities alongside actor
features, the network can develop richer embeddings, thereby enhancing the
model’s ability to accurately identify individuals across varying activity
contexts. Thus we perform joint learning of the activity and actor branch of
ABNet. One segment of feature $F_{AB}$ is fed to activity head $C^{A}$ that
contains decoder $D_{\Omega}^{A}$ that learns features $F_{Ac}$. $C^{A}$ is
trained using $\mathcal{L}_{Ac}$ which is a standard cross-entropy loss for
the activity labels regardless of the actor labels. This joint training also
enables ABNet to utilize activity priors for biometrics, where we use
knowledge of activity for person identification. This is accomplished by
concatenating the activity features $F_{AC}$ with biometrics features $f_{bb}$
during testing.

###  3.3 Overall learning objective

Finally the model $M$ is optimized by combining all the losses which include,
biometrics loss $\mathcal{L}_{Bio}$, distillation loss $\mathcal{L}_{KD}$,
distortion loss $\mathcal{L}_{Dis}$ and activity loss $\mathcal{L}_{Ac}$ and
we get the total loss $\mathcal{L}$ formulated as,

$\mathcal{L}=\mathcal{L}_{Bio}+\lambda_{1}\mathcal{L}_{Ac}+\lambda_{2}\mathcal{L}_{KD}+\lambda_{3}\mathcal{L}_{Dis}$
(6)

where $\lambda_{i}$, $i\in[1,2,3]$ are the weights for each of the losses.

##  4 Experiments and results

Datasets: We perform our experiments on five different datasets which are
derived from existing activity recognition benchmarks. 1) NTU RGB-AB is
derived from NTU RGB+D [30] which is a large-scale benchmark for activity
recognition. We ignore mutual activities and consider 94 activity classes with
88692 samples fro NTU RGB-AB. The activity classes are divided into daily
activities and medical conditions performed by a total of 106 subjects across
32 different setups, 155 different views which are shown with 3 cameras. We
use the official cross-subject split for the train test separation. 2) PKU
MMD-AB is derived from PKU-MMD [9] which is another large scale benchmark for
activity recognition. Similar to NTU RGB-AB, we ignore mutual activities from
PKU-MMD and PKU MMD-AB has 41 activity categories with almost 17,000 labeled
activity instances. These activities are performed by 66 actors in 3 different
camera views and we use the official cross-subject split for our experiments.
3) Charades-AB contains all the 9,848 annotated videos from Charades [38] with
approximately 6.8 activities per video performed by 267 actors across 157
activity classes from a single viewpoint. We use the official train-test split
for our experiments. 4) ACC-MM1-Activities [34] is a recently curated daily
activities dataset which contains 1378 annotated videos where 7 daily
activities are being performed by 200 subjects from a single view-point. These
activities are - enter/exit car, pull/push door, walk upstairs/downstairs, and
texting. We use the official train-test split for our experiments. 5) BRIAR-
BGC3 [10] is a large-scale, in-the-wild person identification dataset
containing samples across varying distances, environment conditions. It is
mainly focused on walking/standing scenario and consists of 3 different
walking conditions (structured walk, random walk and standing) performed by
1055 subjects in outdoor settings from different ranges and angle of
elevation. BRIAR-BGC3 contains over 1300 hours of labeled training videos from
1055 subjects in indoor/outdoor settings. We use a 20K subset of this dataset
for training with official face-restricted testing set for evaluation.
The videos from all five datasets undergo an arbitrarily chosen value of hue
shifting. Training a model on hue-shifted data, even when appearance features
are not explicitly utilized, serves to enhance the model’s robustness and
generalization capabilities. To facilitate face restricted person
identification the faces are blurred using Gaussian blur for both the test and
train split of all datasets.

Table 1: Comparison with state-of-the-art person identification methods: Evaluation shown on NTU RGB-AB, PKU MMD-AB, Charades-AB, and ACC-MM1-Activities on same-activity, View+ evaluation protocol . †: this model was trained on silhouettes.  | Methods | Venue | NTU RGB-AB | PKU MMD-AB | Charades-AB | ACC-MM1-Activities
---|---|---|---|---|---|---
Rank 1 | mAP | Rank 1 | mAP | Rank 1 | mAP | Rank 1 | mAP
Image |  CAL [16] | CVPR22 | 73.79 | 28.40 | 81.31 | 49.45 | 43.84 | 25.81 | 69.83 | 42.81
PSTR [5] | CVPR22 | 69.14 | 34.14 | 84.33 | 47.52 | 37.15 | 24.69 | 57.41 | 34.48
SCNet [17] | ACM MM23 | 69.89 | 31.47 | 79.53 | 43.55 | 31.73 | 21.89 | 64.68 | 39.79
AIM [43] | CVPR23 | 71.37 | 35.41 | 82.52 | 48.89 | 40.13 | 28.31 | 74.79 | 49.14
Video |  TSF [23] | AAAI20 | 71.79 | 31.80 | 76.43 | 37.50 | 35.38 | 21.89 | 49.41 | 29.73
VKD [35] | ECCV20 | 67.41 | 35.63 | 78.35 | 38.54 | 36.31 | 20.71 | 55.38 | 29.57
BiCnet-TKS [21] | CVPR21 | 72.71 | 34.45 | 80.79 | 38.52 | 40.31 | 27.34 | 60.44 | 32.79
STMN [12] | ICCV21 | 72.98 | 35.08 | 76.55 | 47.92 | 38.72 | 24.49 | 59.44 | 39.68
PSTA [40] | ICCV21 | 67.41 | 34.78 | 77.44 | 50.42 | 42.89 | 28.32 | 71.41 | 50.31
SINet [4] | CVPR22 | 69.41 | 30.68 | 79.58 | 40.80 | 40.31 | 26.90 | 65.39 | 45.41
Video-CAL [16] | CVPR22 | 75.49 | 39.86 | 79.59 | 49.42 | 43.91 | 28.51 | 77.48 | 50.08
Baselines |  GaitGL [28] † | - | 61.51 | 28.89 | 65.38 | 33.78 | 18.43 | 6.81 | 39.41 | 18.51
ResNet3D-50 [18] | - | 64.23 | 26.89 | 69.70 | 32.64 | 32.25 | 17.42 | 44.31 | 22.54
MViTv2 [26] | - | 63.87 | 26.41 | 68.37 | 28.52 | 28.51 | 15.39 | 40.59 | 21.52
| ABNet (ours) | - | 78.76 | 40.31 | 86.83 | 57.31 | 45.84 | 31.58 | 80.43 | 52.71

Implementation and training details: The proposed method is implemented using
Pytorch. We use ResNet3D-50 [18] as the backbone of the video encoder
$S_{\varphi}(\cdot)$ and GaitGL [28] for the teacher’s silhouette encoder
$T_{\theta}(\cdot)$. The silhouettes of the RGB videos are extracted using
Mask2Former [8] to use as input to $T_{\theta}(\cdot)$. We create RGB video
clips from each original video by randomly selecting 8 frames with a stride of
4. Every input frame undergoes resizing to dimensions of $256X128$. We train
the model with a batch size of 32 with each batch containing 8 person and 4
clips for each person. Adam [25] is used as the optimizer with weight decay of
$5x10^{-4}$ and learning rate of $3.5X10^{-4}$. The model is trained for 150
epochs with a decay factor $0.1$ after every 40 epochs. The triplet loss
margin $m$ is set to $0.3$ and $\lambda_{i}$, $i\in[1,2,3]$ in Eq. 6 is set to
$0.01$. During inference the activity feature $F_{Ac}$ is concatenated with
the biometrics feature $f_{bb}$ that acts as the activity prior.

Evaluation protocol: For all datasets except BRIAR-BGC3, we randomly split the
test set into gallery and probe (more details in supplementary). We use two
different evaluation protocols; 1) same activity inclusive, and 2) cross-
activity. For the first one, we use all the activities in the gallery whereas
in cross-activity we exclude the activity in the probe while retrieval.
Similarly, we also evaluate for same-view (View+) and cross-view (View-) for
NTU RGB-AB and PKU MM-AB where view information is available. For BRIAR-BGC3,
we use the official protocol for face-restricted evaluation.

Evaluation metrics: For a thorough assessment of the model’s performance, we
employ rank 1 accuracy, rank 5 accuracy, mean average precision (mAP), and TAR
@ 0.1% FAR. While the first three evaluation metrics are more popular to
evaluate a person identification model, the latter metric is also crucial to
check the model’s ability to minimize the false acceptance rate.

Baseline methods: We consider ResNet3D-50 [18], MViTv2 [26] and GaitGL [28] as
baselines. To further demonstrate the effectiveness of our model, we compare
it against several state-of-the-art image based (CAL [16], PSTR [5], SCNet
[17] and AIM [43]) and video based (TSF [23], VKD [35], BiCnet-TKS [21], STMN
[12], PSTA [40], SINet [4],Video-CAL[16]) person identification methods.

###  4.1 Results

In Table 1, we present rank 1 accuracy and mAP metrics for different baselines
and state-of-the-art person identification methods across NTU RGB-AB, PKU MMD-
AB, Charades-AB, and ACC-MM1-Activities datasets, using the same activity
View+ evaluation protocol. ABNet consistently outperforms both the best SOTA
models and baselines across all four datasets. Table 3 compares ABNet with
top-performing identification methods and baselines on the BRIAR-BGC3 dataset.

For a detailed evaluation, Table 2 shows ABNet’s performance across NTU RGB-
AB, PKU MMD-AB, Charades-AB, and ACC-MM1-Activities datasets. This includes
both same activity and cross activity evaluation protocols, featuring View+
and View- settings for NTU RGB-AB and PKU MMD-AB. As view information is
unavailable for Charades-AB and ACC-MM1-Activities datasets, the evaluation
focuses solely on same and cross activity protocols.

Table 2: Comprehensive performance evaluation of ABNet: results shown on NTU RGB-AB, PKU MMD-AB, Charades and ACC-MM1-Activities. We observe that cross-view and cross-activity setup is the most challenging with some performance drop when compared with same activity and same view setup. Dataset | Evaluation Protocol | R@1 | R@5 | mAP | TAR @ 0.1% FAR
---|---|---|---|---|---
View+ |  View- |  View+ |  View- |  View+ |  View- |  View+ |  View-
NTU RGB-AB | Same activity | 78.76 | 77.81 | 85.31 | 82.41 | 40.31 | 38.80 | 39.83 | 35.68
Cross activity | 77.01 | 76.43 | 81.37 | 80.37 | 37.64 | 36.14 | 34.92 | 33.79
PKU MMD-AB | Same activity | 86.83 | 81.41 | 91.37 | 87.73 | 57.31 | 51.74 | 42.79 | 40.31
Cross activity | 81.44 | 79.41 | 89.31 | 84.83 | 51.79 | 46.30 | 37.31 | 34.38
Charades | Same activity | 45.84 | - | 51.04 | - | 31.58 | - | 25.39 | -
Cross activity | 44.82 | - | 52.01 | - | 28.78 | - | 22.61 | -
ACC-MM1-Activities | Same activity | 80.43 | - | 89.31 | - | 52.71 | - | 43.72 | -
Cross activity | 68.31 | - | 76.39 | - | 38.83 | - | 35.32 | -
Table 3: Performance comparison on BRIAR-BGC3 against best state-of-the-art person identification and baselines. Model | R@1 | mAP | TAR@ 0.1%FAR
---|---|---|---
Image-CAL [15] | 30.57 | 17.44 | 25.38
Video-CAL [15] | 28.32 | 15.43 | 24.16
PSTA [40] | 27.75 | 13.78 | 21.54
GaitGL[28] | 12.61 | 9.51 | 6.44
ResNet3D-50[18] | 22.50 | 12.83 | 19.71
MViTv2[26] | 11.78 | 10.21 | 8.44
ABNet (ours) | 34.38 | 18.78 | 26.42

Comparisons: From Tables 1 and 3, it’s clear that existing methods are
primarily focused on identifying individuals based on walking patterns in
various settings, lacking optimization for diverse activities. Our proposed
ABNet consistently outperforms existing models across all datasets. ABNet
demonstrates approximately $2\%$ to $4\%$ higher rank 1 accuracy compared to
the best existing method. This consistent superiority highlights ABNet’s
effectiveness in person identification across diverse activity scenarios.

In Table 2, ABNet shows relatively stable performance across different
evaluation protocols, except for ACC-MM1-Activities, which has fewer activity
classes leading to larger performance gaps. The presence of overlapping
activities in Charades-AB video samples reduces its performance compared to
other datasets. Despite these challenges, ABNet consistently delivers strong
results. Even on the predominantly walking-focused BRIAR-BGC3 dataset, ABNet
outperforms the best SOTA model by $4\%$ in rank 1 accuracy. Overall, ABNet
demonstrates robust performance, particularly on datasets with diverse
activity classes.

###  4.2 Ablations

To verify the effectiveness of ABNet and each of its components, we perform
ablation study on the NTU RGB-AB dataset in Table 4 on the same activity
evaluation protocol. Refer to the supplementary for ablation study on the
cross activity evaluation protocol. Here, B/L stands for the baseline which is
just the backbone model taking RGB video as input. K/D stands for bias-less
distillation, A/P stands for activity prior, and lastly F/D stands for the
bias learning.

Effect of bias-less distillation: Introducing bias-less distillation, either
independently (row 2) or with an activity prior (row 4), leads to notable
performance improvements over the baseline. However, combining bias-less
distillation and activity prior demonstrates superior performance over
independent use of distillation, showcasing their synergistic effect on model
enhancement.

Effect of bias learning: Incorporating bias learning through a distorted video
encoder branch boosts model performance even more (row 5). Similar to bias-
less distillation, combining bias learning with an activity prior yields the
best overall performance (row 6), highlighting the importance of their synergy
in enhancing model robustness and disentangling biometrics and appearance
information.

Effect of activity prior: Incorporating activity and biometrics features
during inference significantly enhances performance compared to using only the
baseline model (row 3). This integration consistently improves model efficacy
across various model configurations demonstrating the role of activity
recognition for biometrics.

Table 4: Ablation studies of each component of ABNet on NTU RGB-AB on same activity evaluation protocol. B/L |  K/D |  A/P |  F/D |  View+ |  View-
---|---|---|---|---|---
R@1 | mAP | R@1 | mAP
✓ |  |  |  | 64.23 | 26.89 | 62.10 | 22.45
✓ |  ✓ |  |  | 69.31 | 28.01 | 66.57 | 24.29
✓ |  |  ✓ |  | 69.43 | 27.97 | 67.37 | 24.77
✓ |  ✓ |  ✓ |  | 72.89 | 32.38 | 70.17 | 30.68
✓ |  ✓ |  |  ✓ | 76.70 | 36.21 | 73.82 | 33.18
✓ |  ✓ |  ✓ |  ✓ | 78.76 | 40.31 | 77.81 | 38.80
Figure 4: Effect of distortion on feature space: The t-SNE plots illustrate
the impact of varying distortion amount $\alpha\in[0,50,100,150,250,300,350]$
on biometrics (top) and appearance (bottom) features of ABNet for ten random
NTU RGB-AB identities. As $\alpha$ increases from left to right, the optimal
results occur at $\alpha=250$ (shown in square) where biometrics changes while
appearance remains consistent. Beyond $\alpha=250$, appearance gets distorted
too, making it unsuitable for disentanglement.

###  4.3 Discussion and analysis

Effect of distortion: Figure 4 presents t-SNE plots of the biometrics and
appearance feature space for ten NTU RGB-AB individuals at
$\alpha\in[0,50,100,150,200,250,300,350]$, where $\alpha$ represents the
amount of distortion. For optimal $\alpha$ we want to find such a value where
biometrics feature clusters are overlapped due to being negative, but
appearance feature clusters still remain relatively same due to being
positive. Increasing $\alpha$ causes more overlap in biometrics feature
clusters; whereas up to $\alpha=250$, appearance feature clusters remain
relatively stable. However, beyond this point, excessive distortion causes
overlapping appearance clusters. Thus $\alpha=250$ is selected as the optimal
value. From the quantitative results presented in Table 5 similar effect of
$\alpha$ is observed on the model’s performance.

Table 5: Effect of distortion on model performance for NTU RGB-AB on the same activity evaluation protocol Distortion amount |  View+ |  View-
---|---|---
R@1 | mAP | R@1 | mAP
$\alpha=200$ | 78.23 | 38.31 | 76.81 | 37.91
$\alpha=250$ | 78.76 | 40.31 | 77.81 | 38.80
$\alpha=300$ | 75.24 | 31.42 | 73.17 | 29.84

Performance analysis across activities: Figure 5 illustrates the comparison
between our method and the baseline across selected activities, encompassing
the top five best and bottom five worst instances in person identification
performance. Notably, activities posing challenges for person identification,
resulting in lower performance, also exhibit reduced accuracy in activity
recognition, except for a few exceptional activity classes. This correlation
underscores the consistent relationship between the difficulty of identifying
individuals within activities and the corresponding accuracy of recognizing
those activities.

Figure 5: Performance analysis across activities: The bar plot on left axis
shows rank 1 identification accuracy for given activity of ABNet against
baseline on 10 activity (5 best and 5 worst) classes of NTU RGB-AB. The
scatter plot with markers on right axis shows activity recognition accuracy
for corresponding classes.

Effect of face restriction: Table 6 illustrates the model’s performance on the
same activity evaluation protocol, indicating a minimal increase in
performance despite the presence of facial features. This suggests the model’s
resilience to facial variations, showcasing its capability to identify
individuals based on non-facial cues. ABNet demonstrates stability in
performance even after the removal of facial appearance cues, highlighting its
reliance on other distinguishing features, such as activity-related cues.

Table 6: Effect of face restriction on model performance for NTU RGB-AB on same activity evaluation protocol Face Restricted |  View+ |  View-
---|---|---
R@1 | mAP | R@1 | mAP
Yes | 78.76 | 40.31 | 77.81 | 38.80
No | 79.24 | 41.64 | 78.87 | 40.04

Qualitative results: In addition to the quantitative results, we show top 4
rank retrieval results in Figure 6. Each row in this figure corresponds to a
probe (left) and the identities retrieved (right) by ABNet. The retrieval list
shows accurate person identification across a variety of activities and
appearance, effectively highlighting ABNet’s ability to learn from activity
cues rather than appearance.

Figure 6: Top 4 rank retrieval samples for ABNet on NTU RGB-AB, Charades-AB,
PKU MMD-AB and ACC-MM1-Activities on row $1,2,3,4$ respectively. The left most
column shows the probe and rest of the columns are the retrieved list.
Accurate retrieval is shown with green box and inaccurate with red. The
subjects from ACC-MM1-Activities consented to publication.

##  5 Conclusion

In this work we study a novel problem of person identification from videos of
daily activities. We propose ABNet, a simple approach to solve this problem
which relies on feature disentanglement and activity prior for person
identification. This approach incorporates feature disentanglement at both
biometric and appearance levels, leveraging distinct strategies to enhance
accuracy and mitigate biases. By distilling biometric knowledge from a bias-
free silhouette-trained model and learning appearance biases via elastic
distortion-based transformations, our framework ensures a comprehensive
understanding of individuals’ inherent biometric traits while accounting for
appearance variations. Moreover, the integration of an activity prior during
inference further enriches the model’s capabilities. Through extensive
evaluations on five benchmark datasets derived from large-scale activity
recognition datasets, our approach consistently surpasses several state-of-
the-art methods.

##  6 Acknowledgement

This research is based upon work supported in part by the Office of the
Director of National Intelligence (IARPA) via 2022-21102100001. The views and
conclusions contained herein are those of the authors and should not be
interpreted as necessarily representing the official policies, either
expressed or implied, of ODNI, IARPA, or the US Government. The US Government
is authorized to reproduce and distribute reprints for governmental purposes
notwithstanding any copyright annotation therein.

## References

  * [1] Elastic transform.  https://pytorch.org/vision/main/generated/torchvision.transforms.ElasticTransform.html.  [Online; accessed 08-November-2023].
  * Adjabi et al. [2020] Insaf Adjabi, Abdeldjalil Ouahabi, Amir Benzaoui, and Abdelmalik Taleb-Ahmed.  Past, present, and future of face recognition: A review.  _Electronics_ , 9(8):1188, 2020.
  * Arnab et al. [2021] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In _ICCV_ , pages 6836–6846, 2021.
  * Bai et al. [2022] Shutao Bai, Bingpeng Ma, Hong Chang, Rui Huang, and Xilin Chen.  Salient-to-broad transition for video person re-identification.  In _CVPR_ , pages 7339–7348, 2022.
  * Cao et al. [2022] Jiale Cao, Yanwei Pang, Rao Muhammad Anwer, Hisham Cholakkal, Jin Xie, Mubarak Shah, and Fahad Shahbaz Khan.  Pstr: End-to-end one-step person search with transformers.  In _CVPR_ , pages 9458–9467, 2022.
  * Chen et al. [2020] Guangyi Chen, Jiwen Lu, Ming Yang, and Jie Zhou.  Learning recurrent 3d attention for video-based person re-identification.  _IEEE TIP_ , 29:6963–6976, 2020.
  * Chen et al. [2021] Jiaxing Chen, Xinyang Jiang, Fudong Wang, Jun Zhang, Feng Zheng, Xing Sun, and Wei-Shi Zheng.  Learning 3d shape feature for texture-insensitive person re-identification.  In _CVPR_ , pages 8146–8155, 2021.
  * Cheng et al. [2022] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In _CVPR_ , 2022.
  * Chunhui et al. [2017] Liu Chunhui, Hu Yueyu, Li Yanghao, Song Sijie, and Liu Jiaying.  Pku-mmd: A large scale benchmark for continuous multi-modal human action understanding.  _arXiv preprint arXiv:1703.07475_ , 2017.
  * Cornett et al. [2023] David Cornett, Joel Brogan, Nell Barber, Deniz Aykac, Seth Baird, Nicholas Burchfield, Carl Dukes, Andrew Duncan, Regina Ferrell, Jim Goddard, et al.  Expanding accurate person recognition to new altitudes and ranges: The briar dataset.  In _WACV_ , pages 593–602, 2023.
  * Dai et al. [2018] Ju Dai, Pingping Zhang, Dong Wang, Huchuan Lu, and Hongyu Wang.  Video person re-identification by temporal residual learning.  _IEEE TIP_ , 28(3):1366–1377, 2018.
  * Eom et al. [2021] Chanho Eom, Geon Lee, Junghyup Lee, and Bumsub Ham.  Video-based person re-identification with spatial and temporal memory networks.  In _ICCV_ , pages 12036–12045, 2021.
  * Fan et al. [2020] Chao Fan, Yunjie Peng, Chunshui Cao, Xu Liu, Saihui Hou, Jiannan Chi, Yongzhen Huang, Qing Li, and Zhiqiang He.  Gaitpart: Temporal part-based model for gait recognition.  In _CVPR_ , pages 14225–14233, 2020.
  * Fan et al. [2023] Chao Fan, Junhao Liang, Chuanfu Shen, Saihui Hou, Yongzhen Huang, and Shiqi Yu.  Opengait: Revisiting gait recognition towards better practicality.  In _CVPR_ , pages 9707–9716, 2023.
  * Gu et al. [2020] Xinqian Gu, Hong Chang, Bingpeng Ma, Hongkai Zhang, and Xilin Chen.  Appearance-preserving 3d convolution for video-based person re-identification.  In _ECCV_ , pages 228–243. Springer, 2020.
  * Gu et al. [2022] Xinqian Gu, Hong Chang, Bingpeng Ma, Shutao Bai, Shiguang Shan, and Xilin Chen.  Clothes-changing person re-identification with rgb modality only.  In _CVPR_ , pages 1060–1069, 2022.
  * Guo et al. [2023] Peini Guo, Hong Liu, Jianbing Wu, Guoquan Wang, and Tao Wang.  Semantic-aware consistency network for cloth-changing person re-identification.  In _ACM MM_ , 2023.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , pages 770–778, 2016.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Hong et al. [2021] Peixian Hong, Tao Wu, Ancong Wu, Xintong Han, and Wei-Shi Zheng.  Fine-grained shape-appearance mutual learning for cloth-changing person re-identification.  In _CVPR_ , pages 10513–10522, 2021.
  * Hou et al. [2021] Ruibing Hou, Hong Chang, Bingpeng Ma, Rui Huang, and Shiguang Shan.  Bicnet-tks: Learning efficient spatial-temporal representation for video person re-identification.  In _CVPR_ , pages 2014–2023, 2021.
  * Huang et al. [2019] Yan Huang, Qiang Wu, Jingsong Xu, and Yi Zhong.  Celebrities-reid: A benchmark for clothes variation in long-term person re-identification.  In _IJCNN_ , pages 1–8. IEEE, 2019.
  * Jiang et al. [2020] Xinyang Jiang, Yifei Gong, Xiaowei Guo, Qize Yang, Feiyue Huang, Wei-Shi Zheng, Feng Zheng, and Xing Sun.  Rethinking temporal fusion for video-based person re-identification on semantic and time aspect.  In _AAAI_ , pages 11133–11140, 2020.
  * Jin et al. [2022] Xin Jin, Tianyu He, Kecheng Zheng, Zhiheng Yin, Xu Shen, Zhen Huang, Ruoyu Feng, Jianqiang Huang, Zhibo Chen, and Xian-Sheng Hua.  Cloth-changing person re-identification from a single image with gait prediction and regularization.  In _CVPR_ , pages 14278–14287, 2022.
  * Kingma and Ba [2015] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _ICLR_ , 2015.
  * Li et al. [2022] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer.  Mvitv2: Improved multiscale vision transformers for classification and detection.  In _CVPR_ , pages 4804–4814, 2022.
  * Liang et al. [2022] Junhao Liang, Chao Fan, Saihui Hou, Chuanfu Shen, Yongzhen Huang, and Shiqi Yu.  Gaitedge: Beyond plain end-to-end gait recognition for better practicality.  In _ECCV_ , pages 375–390. Springer, 2022.
  * Lin et al. [2021] Beibei Lin, Shunli Zhang, and Xin Yu.  Gait recognition via effective global-local feature representation and local temporal aggregation.  In _ICCV_ , pages 14648–14656, 2021.
  * Liu et al. [2019a] Chih-Ting Liu, Chih-Wei Wu, Yu-Chiang Frank Wang, and Shao-Yi Chien.  Spatially and temporally efficient non-local attention network for video-based person re-identification.  _arXiv preprint arXiv:1908.01683_ , 2019a.
  * Liu et al. [2019b] Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C Kot.  Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding.  _IEEE TPAMI_ , 42(10):2684–2701, 2019b.
  * Liu et al. [2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _ICCV_ , 2015.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _ICCV_ , pages 10012–10022, 2021.
  * Meng et al. [2021] Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou.  Magface: A universal representation for face recognition and quality assessment.  In _CVPR_ , pages 14225–14234, 2021.
  * O’Brien et al. [2024] K. O’Brien, M. Rybak, J. Huang, A. Stevens, M. Fredriksz, M. Chaberski, D. Russell, L. Castin, M. Jou, N. Gurrapadi, and M. Bosch.  Accenture-mm1: A multimodal person recognition dataset.  In _WACVW_ , 2024.
  * Porrello et al. [2020] Angelo Porrello, Luca Bergamini, and Simone Calderara.  Robust re-identification by multiple views knowledge distillation.  In _ECCV_ , pages 93–110. Springer, 2020.
  * Qian et al. [2020] Xuelin Qian, Wenxuan Wang, Li Zhang, Fangrui Zhu, Yanwei Fu, Tao Xiang, Yu-Gang Jiang, and Xiangyang Xue.  Long-term cloth-changing person re-identification.  In _ACCV_ , 2020.
  * Ren et al. [2024] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang.  Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
  * Sigurdsson et al. [2016] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.  Hollywood in homes: Crowdsourcing data collection for activity understanding.  In _ECCV_ , pages 510–526, 2016.
  * Tarvainen and Valpola [2017] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  _NeurIPS_ , 30, 2017.
  * Wang et al. [2021] Yingquan Wang, Pingping Zhang, Shang Gao, Xia Geng, Hu Lu, and Dong Wang.  Pyramid spatial-temporal aggregation for video-based person re-identification.  In _ICCV_ , pages 12026–12035, 2021.
  * Wu et al. [2019] Ancong Wu, Wei-Shi Zheng, Xiaowei Guo, and Jian-Huang Lai.  Distilled person re-identification: Towards a more scalable system.  In _CVPR_ , pages 1187–1196, 2019.
  * Yang et al. [2019] Qize Yang, Ancong Wu, and Wei-Shi Zheng.  Person re-identification by contour sketch under moderate clothing change.  _IEEE TPAMI_ , 43(6):2029–2046, 2019.
  * Yang et al. [2023] Zhengwei Yang, Meng Lin, Xian Zhong, Yu Wu, and Zheng Wang.  Good is bad: Causality inspired cloth-debiasing for cloth-changing person re-identification.  In _CVPR_ , pages 1472–1481, 2023.
  * Ye et al. [2021] Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi.  Deep learning for person re-identification: A survey and outlook.  _IEEE TPAMI_ , 44(6):2872–2893, 2021.
  * Yu et al. [2006] Shiqi Yu, Daoliang Tan, and Tieniu Tan.  A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition.  In _ICPR_ , pages 441–444, 2006.
  * Zhang et al. [2017] Dongyu Zhang, Wenxi Wu, Hui Cheng, Ruimao Zhang, Zhenjiang Dong, and Zhaoquan Cai.  Image-to-video person re-identification with temporally memorized similarity learning.  _IEEE TCSVT_ , 28(10):2622–2632, 2017.
  * Zhang et al. [2019] Ziyuan Zhang, Luan Tran, Xi Yin, Yousef Atoum, Xiaoming Liu, Jian Wan, and Nanxin Wang.  Gait recognition via disentangled representation learning.  In _CVPR_ , pages 4710–4719, 2019.
  * Zheng et al. [2015] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian.  Scalable person re-identification: A benchmark.  In _ICCV_ , pages 1116–1124, 2015.

Supplementary Material

The supplementary is organized as follows:

  * •

Section A provides detailed description of the gallery probe setup of the
datasets.

  * •

Section B provides comparison of ABNet with state-of-the-art methods on the
cross-activity evaluation protocol.

  * •

Section C provides ablations and some more discussion and analysis

  * •

Section D provides some more qualitative samples of retrieval results of
ABNet.

The code and all the datasets used for this work will be made publicly
available.

##  Appendix A Gallery probe setup

We evaluate the performance in terms of same activity and cross activity. In
the same activity evaluation protocol, probe and gallery contains all the
activities, however, probe contains a smaller subset of samples and the rest
are placed in gallery. In the cross activity evaluation protocol, probe and
gallery contains mutually exclusive activities, where probe contains a smaller
subset of samples and rest of the samples from those activities are discarded;
on the contrary the gallery contains all samples from a certain activity. Here
for each actor there are multiple activity samples, and each activity again
has different view-point or setup variation (for NTU RGB-AB and PKU MMD-AB).
The samples are randomly selected for gallery and probe sets. For NTU RGB-AB
and PKU MMD-AB two variations are checked - probe view included in gallery
(View+) and probe view excluded from gallery (View-) in case of both same
activity and cross activity protocol. However, since Charades and ACC-
MM1-Activities does not contain multiple view points, the evaluation protocol
with inclusion/exclusion of probe view from gallery is not relevant in these
case. Table 1 illustrates a detailed description of all the datasets.

Table 1: Dataset statistics Dataset | Split | #actors | #activities | #samples
---|---|---|---|---
NTU RGB-AB | train | 85 | 94 | 70952
gallery | 21 | 14192
probe | 3548
PKU MMD-AB | train | 53 | 41 | 13634
gallery | 13 | 2727
probe | 681
Charades-AB | train | 214 | 157 | 45111
gallery | 53 | 9022
probe | 2256
ACC-MM1-Activities | train | 182 | 7 | 7717
gallery | 45 | 1543
probe | 386
BRIAR-BGC3 | train | 870 | 3 | 20000
gallery | 130 | 4171
probe | 922

##  Appendix B Comparison with state-of-the-art methods

We present the comparison of different state-of-the-art methods against our
proposed ABNet to show its effectiveness across NTU RGB-AB, PKU MMD-AB,
Charades-AB and ACM-MM1-Activities datasets on the cross-activity View+
evaluation protocol in Table 2 which corresponds to Table 1 of the main paper.
Similar to the quantitative comparisons presented in the main paper, in case
of cross-activity evaluation protocol as well, ABNet outperforms all the
existing methods and baselines by a competitive margin in terms of both
evaluation metrics. This shows the robustness of our method against same or
cross activity evaluation.

##  Appendix C More analysis and discussion

Ablations on cross-activity evaluation protocol. Table 3 illustrates the
effect of each component of our proposed ABNet on NTU RGB-AB dataset on the
cross-activity evaluation protocol. This table is an extension of Table 4 of
the main paper and similar to the same-activity evaluation protocol, the
performance of the model remains stable in case of cross-activity and also
each modification component gives a performance boost to the model, which
finally contributes to the overall model’s performance. Now, some activities
might be easier to recognize and hence, we perform an experiment on top 5 best
and top 5 worst performing activities with and without the activity prior (AP)
to see whether the easily recognizable activities introduce any bias through
the activity information. In Figure 1 we see that the performance pattern
remains consistent across activities with or without AP which indicates that
AP consistently helps and the difficulty level of activities do not introduce
any bias.

Figure 1: Performance analysis w/ and w/o activity prior; bars represent biometrics rank 1 and dots represent activity accuracy. Figure 2: Effect of distortion amount Original sample zoomed in to show effect of $\alpha=50,100,150$ (top) and $\alpha=200,250,300$ (bottom). As $\alpha$ increases, the distortion keeps increasing. Table 2: Comparison with state-of-the-art person identification methods: Evaluation shown on NTU RGB-AB, PKU MMD-AB, Charades-AB, and ACC-MM1-Activities on cross-activity, View+ evaluation protocol . †: this model was trained on silhouettes.  | Methods | Venue | NTU RGB-AB | PKU MMD-AB | Charades-AB | ACC-MM1-Activities
---|---|---|---|---|---|---
Rank 1 | mAP | Rank 1 | mAP | Rank 1 | mAP | Rank 1 | mAP
Image |  CAL [16] | CVPR22 | 70.31 | 24.08 | 78.31 | 43.43 | 40.13 | 21.23 | 67.33 | 38.21
PSTR [5] | CVPR22 | 68.34 | 32.54 | 77.98 | 41.23 | 35.12 | 20.32 | 53.46 | 30.18
SCNet [17] | ACM MM23 | 68.82 | 26.31 | 73.91 | 39.65 | 27.42 | 17.61 | 55.38 | 32.42
AIM [43] | CVPR23 | 72.79 | 30.21 | 79.22 | 44.90 | 35.56 | 26.36 | 66.81 | 38.14
Video |  TSF [23] | AAAI20 | 67.81 | 26.88 | 71.61 | 33.22 | 30.21 | 18.29 | 41.31 | 21.43
VKD [35] | ECCV20 | 66.33 | 31.46 | 72.19 | 34.34 | 31.89 | 18.81 | 51.26 | 22.16
BiCnet-TKS [21] | CVPR21 | 69.13 | 30.21 | 77.13 | 33.32 | 38.33 | 23.34 | 58.41 | 30.21
STMN [12] | ICCV21 | 70.21 | 30.13 | 71.53 | 42.21 | 33.89 | 20.81 | 57.61 | 37.61
PSTA [40] | ICCV21 | 65.13 | 31.42 | 72.43 | 47.42 | 38.72 | 24.84 | 67.31 | 37.33
SINet [4] | CVPR22 | 66.21 | 27.81 | 74.11 | 26.21 | 37.31 | 21.90 | 61.32 | 36.41
Video-CAL [16] | CVPR22 | 73.31 | 31.73 | 77.34 | 45.72 | 41.50 | 25.81 | 67.48 | 38.23
Baselines |  GaitGL [28] † | - | 57.04 | 27.13 | 61.22 | 27.84 | 14.51 | 4.85 | 35.13 | 16.31
ResNet3D-50 [18] | - | 62.80 | 23.52 | 65.12 | 29.41 | 27.35 | 14.89 | 39.89 | 19.83
MViTv2 [26] | - | 59.27 | 21.38 | 61.40 | 25.31 | 21.89 | 12.79 | 37.31 | 17.80
| ABNet (ours) | - | 77.01 | 37.64 | 81.44 | 51.79 | 44.82 | 28.78 | 68.31 | 38.83
Table 3: Ablation studies of each component of ABNet on NTU RGB-AB on cross activity evaluation protocol B/L |  K/D |  A/P |  F/D |  View+ |  View-
---|---|---|---|---|---
R@1 | mAP | R@1 | mAP
✓ |  |  |  | 62.80 | 23.52 | 61.71 | 21.41
✓ |  ✓ |  |  | 66.90 | 23.94 | 63.03 | 22.01
✓ |  |  ✓ |  | 66.24 | 23.81 | 64.61 | 22.48
✓ |  ✓ |  ✓ |  | 69.21 | 31.01 | 66.41 | 30.43
✓ |  ✓ |  |  ✓ | 74.33 | 33.79 | 72.85 | 31.68
✓ |  ✓ |  ✓ |  ✓ | 77.01 | 37.64 | 76.43 | 36.14

Effect of distortion: Table 4 reports the effect of distortion on cross-
activity evaluation protocol on the NTU RGB-AB dataset which is an extension
of Table 5 of the main paper. Figure 3 illustrates the t-SNE plots of the
biometrics and appearance feature space for ten individuals from two of the
challenging datasets; Charades-AB and BRIAR-BGC3. It is observed from this
figure that the effect of $\alpha$ is consistent and our choice of $\alpha$ is
applicable for even these challenging datasets as well. Figure 2 illustrates
the qualitative samples of the effect of distortion. In Figure 2, we zoomed in
the face portion of the actor for better visualization and it is observed that
as $\alpha$ increases gradually, the distortion amount increases and by
$\alpha=300$, the sample is so distorted that it becomes unsuitable for our
purpose.

Figure 3: Effect of distortion on feature space. The t-SNE plots illustrate the impact of varying distortion amount $\alpha\in[200,225,250,275,300,325,350]$ on biometrics (top) and appearance (bottom) features of ABNet for ten random identities for Charades-AB (top) and BRIAR-BGC3 (bottom). As $\alpha$ increases from left to right, the optimal results occur at $\alpha=250$ (shown in square) where biometrics changes while appearance remains consistent. Beyond $\alpha=250$, appearance gets distorted too, making it unsuitable for disentanglement Table 4: Effect of distortion on model performance for NTU RGB-AB on the cross activity evaluation protocol Distortion amount |  View+ |  View-
---|---|---
R@1 | mAP | R@1 | mAP
$\alpha=200$ | 75.91 | 37.04 | 75.12 | 35.83
$\alpha=250$ | 77.01 | 37.64 | 76.43 | 36.14
$\alpha=300$ | 72.70 | 29.01 | 71.03 | 28.94
Table 5: Effect of face restriction on model performance for NTU RGB-AB on cross activity evaluation protocol Face Restricted |  View+ |  View-
---|---|---
R@1 | mAP | R@1 | mAP
Yes | 77.01 | 37.64 | 76.43 | 36.14
No | 77.70 | 39.01 | 76.98 | 38.84
Table 6: Activity recognition performance of different datasets on ABNet. x-sub and x-view respectively denote cross-subject and cross-view evaluation protocols for its corresponding dataset, if applicable. Dataset | x-sub | x-view
---|---|---
NTU RGB-AB | 88.71 | 89.50
PKU MMD-AB | 91.42 | 94.21
Charades-AB | 41.31 | -
ACC-MM1-Activities | 71.08 | -
BRIAR-BGC3 | 79.31 | -
Table 7: Choice of Backbone. Performance comparison of different backbones on NTU RGB-AB Network | Backbone | Same activity | Cross activity
---|---|---|---
View+ |  View- |  View+ |  View-
R@1 | mAP | R@1 | mAP | R@1 | mAP | R@1 | mAP
Teacher | GaitGL[28] | 61.51 | 28.89 | 57.78 | 26.78 | 57.04 | 27.13 | 55.80 | 26.41
GaitPart[13] | 54.79 | 16.73 | 53.93 | 15.91 | 52.18 | 15.01 | 46.89 | 13.84
GaitBase[14] | 60.21 | 28.02 | 59.04 | 26.76 | 59.90 | 26.31 | 57.91 | 25.96
Student |  MViT v2[26] | 63.87 | 26.41 | 61.01 | 23.81 | 59.27 | 21.38 | 59.16 | 20.01
ViViT[3] | 58.81 | 20.41 | 57.10 | 16.42 | 57.30 | 12.41 | 52.01 | 9.68
Swin[32] | 59.20 | 21.68 | 58.41 | 19.41 | 58.70 | 16.91 | 54.31 | 11.47
ResNet3D-50[18] | 64.23 | 26.89 | 62.10 | 22.45 | 62.80 | 23.52 | 61.71 | 21.41
ResNet3D-34[18] | 63.90 | 25.93 | 60.45 | 21.87 | 60.21 | 22.74 | 59.79 | 20.47
Figure 4: Dataset samples. Here each two samples show different values of hue
shifting for the same video of NTU RGB-AB (top-left), PKU MMD-AB (top-right),
Charades-AB (bottom-left) and ACC-MM1-AB (bottom-right). All the samples have
their faces blurred.

Figure 5: Performance analysis across activities. The bar plot on left axis
shows rank 1 identification accuracy for given activity of ABNet against
baseline PKU MMD-AB (top-left), Charades AB (top-right), ACC-MM1-Activities
(bottom-left) and BRIAR-BGC3 (bottom-right) datasets. The scatter plot with
markers on right axis shows activity recognition accuracy for corresponding
classes.

Figure 6: Top 4 rank retrieval samples for ABNet on NTU RGB-AB (top-left),
Charades-AB (top-right), PKU MMD-AB (bottom-left) and BRIAR-BGC3 (bottom-
right). The left most columns for each dataset samples hold the probe samples
and the following four columns to that probe are its retrieval list. Accurate
retrieval is shown with green box and inaccurate with red.

Effect of face restriction on cross-activity evaluation protocol is reported
in Table 5 on the NTU RGB-AB dataset. Similar to the results reported in the
main paper, even in case of the cross-activity evaluation protocol, the model
performance remains stable even when faces are restricted showing the learning
of non-facial cues across cross-activity evaluation protocol.

Choice of backbone. The performance comparison of different backbone networks
is shown in Table 7, where the backbone model takes the silhouette/RGB video
frames as input respectively for the teacher/student network for the task of
person identification. Here this experiment is run only on the baseline where
none of the modification components are present. This selection of backbones
ensures that the teacher network contributes its expertise to the specific
task it is designed for in the student network. Moreover, similar to existing
recent work [16, 21, 5, 43] in person identification, in our case also CNN
based backbones outperform transformer based ones. From this experiment, we
pick the best performing backbone for both networks.

Performance of action recognition: Table 6 reports the performance of ABNet on
activity recognition results for different datasets. Here the reported
evaluation metric is accuracy on cross-subject and cross-view evaluation
protocol. NTU RGB-AB and PKU MMD-AB are evaluated on these two protocols,
however, since there is no explicit view information for rest of the three
datasets, the accuracies are reported in terms of cross-subject because the
test and train split contains mutually exclusive actors/subjects.
Figure 5 compare ABNet and the baseline across the top five best and bottom
five worst performing activities in person identification for PKU MMD-AB and
Charades (top row). The bottom row shows person identification performance
across all 7 activities of the ACC-MM1-Activities dataset and all 3 activities
of BRIAR-BGC3 dataset. It is observed that activities with minimal overall
body movement pose greater challenges for individual identification, whereas
more overall body movement contribute to higher person identification
accuracy. This highlights the significance of incorporating activity prior in
our model. Moreover, it also emphasizes the importance of activity cues
demonstrating the efficacy of our joint training approach in effectively
learning such cues.

Accuracy of silhouette extractor and effectiveness of silhouettes: The
accuracy of the silhouette extraction process will indeed affect model’s
performance and to explore that we perform an experiment using Grounded-SAM
[37] which is an open-world segmentation model. The results are reported on a
small subset (10 action classes) of the NTU-RGB-AB dataset on the same
activity View+ setting in Table 8. It is observed that with Grounded-SAM as
silhouette extractor the performance does go up, which can be attributed to it
being an open-world model and thus being more robust. Similarly, a
$\mathbf{3\%}$ rank 1 accuracy gain is seen in case of a small subset of the
Charades-AB dataset when using Grounded-SAM as opposed to Mask2Former.
Nevertheless, even with a weaker silhouette extractor our model still performs
well and since this extraction process is not part of the inference stage,
training the model with a better silhouette extractor will provide some
benefits.

Table 8: Performance with varying silhouette extractors Silhouette extractor | Rank 1 | mAP
---|---|---
Mask2Former [8] | 85.2 | 87.3
Grounded-SAM [37] | 87.8 | 88.5

The main motivation behind using silhouette features is to distill appearance-
less knowledge, e.g. purely biometrics information that not only contains
gait; but also pose, body shape, structure etc information to aid
disentanglement. The recognition performance of the two decoupled features is
reported in Table 9 for the Charades-AB dataset. The huge performance gap
between the biometrics and non-biometrics features shows that the non-
biometrics features do not have meaningful information to perform person
identification; essentially proving the effectiveness of the disentanglement
process. To demonstrate the effectiveness of using silhouette features in our
method, we distort the silhouettes and distill that knowledge to the
biometrics features, which resulted in a huge performance drop (about
$\mathbf{24\%}$) (row 3 of Table 9). This shows that even in case of
activities beyond walking, the silhouette-based biometrics features contribute
to a great extent in accurate recognition. We specifically select the
Charades-AB dataset for this experiment as it is a real-world dataset
encompassing a diverse range of appearance variations.

Table 9: Performance of disentangled features Feature | Rank 1 | mAP
---|---|---
Biometrics | 45.8 | 31.6
Non-biometrics | 2.8 | 0.4
Biometrics w/ distorted sils | 21.4 | 10.5

##  Appendix D Qualitative analysis

Figure 4 illustrates examples of different values of hue-shifting, from which
it can be observed that the color profile for each frame is distinct from the
other. Figure 6 illustrates the top 4 rank retrieval results for a given probe
for NTU RGB-AB, PKU MMD-AB and Charades-AB datasets. Some of the failure cases
is seen for having difficulty performing accurate retrieval due to the absence
of a lot of overall body movement (e.g. probe activity is sitting in first
sample of Charades-AB and second sample of PKU MMD-AB). Moreover, another
failure case is seen in case of the second sample of Charades-AB which shows
the inherent challenges present in the dataset, e.g. data quality, no standard
way of performing an activity etc. Despite these challenges, from the figure
it is observed that accurate retrieval is done in most cases irrespective of
view-point, activity and appearance, which shows the effectiveness of ABNet.