# DFS: A Diverse Feature Synthesis Model for Generalized Zero-Shot Learning

Bonan Li
University of Chinese Academy of Sciences
<EMAIL_ADDRESS>Xuecheng Nie 111This work is supported by Yitu
Technology
Yitu Technology
<EMAIL_ADDRESS>Congying Han 222Corresponding author
University of Chinese Academy of Sciences
<EMAIL_ADDRESS>
###### Abstract

Generative based strategy has shown great potential in the Generalized Zero-
Shot Learning task. However, it suffers severe generalization problem due to
lacking of feature diversity for unseen classes to train a good classifier. In
this paper, we propose to enhance the generalizability of GZSL models via
improving feature diversity of unseen classes. For this purpose, we present a
novel Diverse Feature Synthesis (DFS) model. Different from prior works that
solely utilize semantic knowledge in the generation process, DFS leverages
visual knowledge with semantic one in a unified way, thus deriving class-
specific diverse feature samples and leading to robust classifier for
recognizing both seen and unseen classes in the testing phase. To simplify the
learning, DFS represents visual and semantic knowledge in the aligned space,
making it able to produce good feature samples with a low-complexity
implementation. Accordingly, DFS is composed of two consecutive generators: an
aligned feature generator, transferring semantic and visual representations
into aligned features; a synthesized feature generator, producing diverse
feature samples of unseen classes in the aligned space. We conduct
comprehensive experiments to verify the efficacy of DFS. Results demonstrate
its effectiveness to generate diverse features for unseen classes, leading to
superior performance on multiple benchmarks. Code will be released upon
acceptance.

##  1 Introduction

Generalized Zero-Shot Learning (GZSL) is an important yet challenging problem
in computer vision, aiming to recognize object categories unseen in the
training phase. It is widely utilized in various applications, _e.g_., image
recognition [22, 17, 6, 42], object detection [29, 46, 21] and super-
resolution [35], due to the powerfulness of facilitating a model to reason
class information of new objects with only their semantic correlations to
known ones.

Figure 1: Motivation of our Diverse Feature Synthesis model for the
Generalized Zero-Shot Learning task. (a) Prior works fail to generate diverse
features of unseen classes, causing classifier trained with these samples
poorly generalize to testing scenarios. (b) Our DFS model can enhance feature
diversity for unseen classes, thus improving the generalizability of
classifiers for the GZSL task.

Existing methods to tackle the GZSL task mainly follow two strategies: one is
embedding based [19, 26, 34, 44, 7, 16], that learns an embedding model
between visual and semantic spaces, following with nearest neighbor search for
deriving class cues; the other is generation based [40, 10, 25, 14, 33, 23],
that applies generative model to produce features for unseen classes,
converting the problem to a traditional classification problem. Recently,
generative based strategy has shown dominated performance over embedding based
one, regarding to its capability of alleviating the seen-bias problem. Despite
its successful, generative based strategy still faces two major problems, that
heavily hurt its generalizability while are ignored by prior works. The first
problem is lack of diversity for generated features of unseen classes, leading
to inaccurate classification boundary. As shown in Figure 1(a), generated
samples from prior works are platitudinous and cannot describe the true
distribution of unseen classes, causing classifier trained with these samples
performs poorly in the testing phase. The second problem is high-complexity of
feature generator, due to employing GAN with complex settings, which results
in unstable training procedure [40, 47, 13, 25]. Motivated by this, we propose
to boost the generalizability of GZSL models via enhancing the feature
diversity with low-complexity generators.

To achieve this goal, we present a novel Diverse Feature Synthesis (DFS) model
for the GZSL task in this paper. Generally, semantic knowledge has
specificness, _e.g_., one class label only maps to a particular semantic
feature; Whereas, visual knowledge has diversity, _e.g_., one class label maps
to multiple visual features. Prior works [14, 33, 23] generate feature samples
for unseen class solely relied on their semantic knowledge. Differently, our
DFS model proposes to incorporate visual and semantic knowledge, together, for
inheriting both of their properties. Thus, DFS can generate diverse features
for a specific unseen class. Besides, instead of visual space, DFS performs
feature generation and category classification in the aligned space due to its
simplicity and descriptiveness, therefore, alleviating the stress for
capturing feature diversity and making a low-complexity model competent for
the task.

In particular, DFS is implemented with two generators: an Aligned Feature
Generator (AFG) and a Synthetic Feature Generator (SFG). DFS first utilizes
AFG to transfer features from semantic and visual spaces into an aligned
space, which is derived from the cross model learning [33]. Then, DFS learns
SFG, designed as a Conditional Variational AutoEncoder (CVAE), with two steps:
In the first step, SFG feeds aligned semantic and visual features of _seen
classes_ as input to the encoder, which produces latent features embedding
with both semantic and visual knowledge; In the second step, SFG feeds the
latent features into its decoder to reconstruct aligned visual features of
_seen classes_ , given the semantic knowledge as condition. After SFG learned,
in the inference phase, DFS removes its encoder and only uses the decoder to
generate feature samples for _unseen classes_. Here, latent features for
unseen classes are synthesized via randomly sampling from standard Gaussian
distribution. In this way, DFS introduces visual knowledge into the feature
generation process, it significantly increases the diversity of generated
samples, overcoming the drawbacks of prior works and leading to improved
classifier with better generalizability, as shown in Figure 1(b). In addition,
usage of aligned features simplifies the feature generation process, enabling
a low-complexity VAE to satisfy the requirement. The overall framework of DFS
is shown in Figure 2.

Comprehensive experiments verify the efficacy of DFS to generate diverse
feature samples for unseen classes as well as to improve the generalizability
of GZSL models. Our contributions are in two folds: (1) We propose a novel
model for effectively and efficiently generating diverse features for unseen
classes for the GZSL task; (2) With feature samples generated from our model,
we set new state-of-the-arts on multiple benchmarks for the GZSL task.

##  2 Related Work

In literature, Zero-Shot Learning (ZSL) has been well studied. It can be
categorized into Conventional ZSL (CZSL) and Generalized ZSL (GZSL) depending
on the classes contained in the testing dataset. For CZSL, the testing dataset
only contains unseen classes samples. However, for GZSL, both seen and unseen
classes samples are included in the testing dataset. Compared with CZSL, GZSL
is more practical, and most of current researches in ZSL area aim at solving
this problem.

Early ZSL approaches are mainly based on embedded models and can be divided
into three groups. The first group [19, 31] learns a projection function from
visual feature space to a semantic space. The second group [43, 3] of approach
map semantic features to visual space. The third group [7, 31, 45, 14] adopts
latent space to establish mapping between semantic and visual domains.
Although the above methods have achieved remarkable results in CZSL setting,
these models will produce obvious bias to the visible classes in GZSL setting.
This is demonstrated by the fact that the classification accuracy of seen
classes is much higher than that of unseen classes.

Recently, the powerful generative methods, _e.g_. Generative Adversarial
Network(GAN) [11] and Variational Autoencoder(VAE) [2], are utilized to
synthesize massive features of unseen classes from prototype vector [37, 40,
18, 28, 15, 24, 10, 47]. These synthesized features of unseen classes will be
used together with features of seen classes to train a fully supervised
classifier. This way can promote generalization of the classifier, thus
reducing the bias to seen classes, resulting in a higher harmonic mean.
f-CLSWGAN [40] applies GAN to generate visual features conditioned on semantic
features, but it suffers from mode collapse issues and unstable training phase
[4]. VAE based algorithm [24, 15] can train stably, but it fails to capture
the complex distribution [5], leading to unsatisfied results. In order to
overcome the above shortcomings, [41, 10] combines two generative models,
_i.e_., VAE/GAN, to generate samples for unseen classes. Despite of achieving
performance improvement, their complex parameter setting and tedious training
process can not be ignored. In contrast to the above methods, the works [33,
23] train the cross model to encode and decode features from semantic and
visual modalities by matching their parameterized distributions and joining a
cross-modal reconstruction term. Although [33, 23] exhibits superior results
with stable training, they are still unable to effectively generate massive
significantly different features with the same semantic information, which
would lead to the classifier inevitably biasing seen classes. To alleviate
these two problems at the same time, we take a stable model to synthesize more
diverse features of unseen classes. Details are depicted in next section.

##  3 Method

###  3.1 Problem Definition

We first depict the mathematical formulation for the Generalized Zero-Shot
Learning (GZSL) problem. Let
$\mathcal{S}=\\{(v_{s},a_{s},l_{s})|v\in{V}_{s},a_{s}\in{A}_{s},l_{s}\in{L}_{s}\\}$
denote the training set for seen classes, where $v_{s}$ is the visual feature
of an image, $l_{s}$ is the corresponding class label, and $a_{s}$ is the
semantic embedding for class $l_{s}$. Let
$\mathcal{U}=\\{(v_{u},a_{u},l_{u})|v_{u}\in{V}_{u},a_{u}\in{A}_{u},l_{u}\in{L}_{u}\\}$
denote the testing set for unseen classes, where $v_{u}$, $a_{u}$, and $l_{u}$
are similarly defined as $v_{s}$, $a_{s}$, and $l_{s}$, but
${L}_{s}\cap{L}_{u}=\emptyset$, meaning that seen classes and unseen ones are
disjoint. Given $\mathcal{S}$, $A_{u}$ and $L_{u}$, GZSL targets at learning a
function $f$ that can recognise both seen and unseen classes,

$f:v\rightarrow l$

where $v\in V_{s}\cup V_{u}$ and $l\in L_{s}\cup L_{u}$.

To solve the GZSL task, generative based strategy models $f$ as a classifier
through converting the original problem to the traditional classification
problem. Accordingly, its core is to generate visual features $\hat{V_{u}}$
for unseen classes, thus forming the training set $\mathcal{T}=\\{(v,l)|v\in
V_{s}\cup\hat{V_{u}},l\in L_{s}\cup L_{u}\\}$ to learn the classifier $f$. For
getting $\hat{V_{u}}$, prior works mainly follow two ways: (1) performing
feature generation in the _original_ visual space with a Generative
Adversarial Network based generator $g_{1}$, formulated as

$g_{1}:A_{u},z\rightarrow\hat{V_{u}}$

where $z$ is a noise sampled from standard Gaussian distribution; (2)
performing feature generation in the _aligned_ space with a sampler $g_{2}$:

$g_{2}:H(A_{u})\rightarrow H(\hat{V_{u}})$

where $H(\cdot)$ is a function for transferring features from original space
to aligned space. However, the first way is always built on a complex model to
mitigate the large gap between semantic and visual knowledge, and the second
way often suffers from samples lacking of diversity.

Differently, in this paper, we propose a Diverse Feature Synthesis (DFS)
model, defined by $\hat{g}$, to generate diverse feature samples for unseen
classes in the aligned space, as

$\hat{g}:H(A_{u}),z\rightarrow H(\hat{V_{u}})$

In this way, DFS is able to effectively generate diverse feature samples for
unseen class via low-complexity model. Thus, DFS overcomes drawbacks of
previous and leads to a more accurate classifier $f$ for the GZSL task. In
next subsection, we will illustrate the implementation details of the proposed
DFS model.

Figure 2: The overview of our proposed Diverse Feature Synthesis (DFS) model
for the Generalized Zero-Shot Learning (GZSL) tasks. Top: the _training_
phase. DFS first utilizes an aligned feature generator to transfer semantic
and visual features of seen classes, $a_{s}$ and $v_{s}$ into aligned space,
_i.e_., $z_{l_{s}}^{1}$ and $z_{l_{s}}^{2}$. Then, it employs a synthesised
feature generator to produce synthesised features. Specifically, it feeds the
aligned features into the decoder $E^{3}$ to generate the latent feature
$z_{l_{s}}^{3}$ via sampling from $\mu_{l_{s}}^{3}$ and $\sigma_{l_{s}}^{3}$
and reconstructs the visual representation by the decoder $D^{3}$ with
$\tilde{z}_{l_{s}}^{2}$. Bottom: the _inference_ phase. For seen classes, the
aligned feature $z_{l_{s}}^{2}$ are directly produced by the aligned feature
generator, given $v_{s}$ as input. While for unseen classes, DFS removes the
encoder of the synthesised feature generator. After deriving aligned feature
$z_{l_{u}}^{1}$ from $a_{u}$, it synthesises the latent feature $z^{noise}$
via random sampling from standard Gaussian distribution for input to $D^{3}$
to generate the feature sample $z_{l_{u}}^{2}$ of unseen class.

###  3.2 Diverse Feature Synthesis Model

In this section, we will explain the implementation of our DFS model in
details, including its network architecture as well as the training and
inference phases.

####  3.2.1 Network Architecture

Our DFS model is composed of two modules: an Aligned Feature Generator and a
Synthetic Feature Generator. Their details will be illustrated in the
following, respectively.

##### Aligned Feature Generator

We implement the aligned feature generator based on the recently proposed
model CADA-VAE [33], which achieves impressive results for generalized zero-
shot learning by using a stable cross VAE model. Nevertheless, we are not
limited to choosing CADA-VAE model as the basis and our DFS network can still
be effective in improving performance when other aligned feature generators
are selected as the baseline. More concretely, two encoders, $E^{1}$ and
$E^{2}$, first encode semantic and visual vectors as aligned features,
$z^{1}_{l_{s}}$ and $z^{2}_{l_{s}}$, respectively. After obtaining the aligned
features, we take them passed through the decoders, $D^{1}$and $D^{2}$, to
generate reconstruction features which have the same dimensions with original
input vectors. According to classical VAE model, we can formulate the loss as
follows:

$\begin{split}\mathcal{L}_{VAE}&=\sum^{M}_{k=1}\mathbb{E}_{q_{\phi}(z^{k}_{l_{s}}|x^{k}_{l_{s}})[\log
p_{\theta}(x^{k}_{l_{s}}|z^{k}_{l_{s}})]}\\\
&-\beta_{1}D_{KL}(q_{\phi}(z^{k}_{l_{s}}|x^{k}_{l_{s}})||p_{\theta}(z^{k}_{l_{s}}))\end{split}$
(1)

where $q_{\phi}(z^{k}_{l_{s}}|x^{k}_{l_{s}})$ is modeled as $E^{k}$,
$p_{\theta}(z^{k}_{l_{s}})$ is assumed to be $\mathcal{N}(0,1)$,
$p_{\theta}(x^{k}_{l_{s}}|z^{k}_{l_{s}})$ is equal to $D^{k}$ and
$D_{KL}(\cdot)$ denotes KL-Divergence. $\beta_{1}$ is the hyper-parameter to
weight the loss of KL-Divergence and restruction loss. $z$ is the feature in
aligned space. For ZSL, the features are usually draw from visual and semantic
domain, so we set $M=2$, $x^{1}_{l_{s}}\in A_{s}$ and $x^{2}_{l_{s}}\in
V_{s}$.

Here, in order to learn representations within an aligned space, two extra
loss terms which named Distribution-Alignment loss ($\mathcal{L}_{DA}$) and
Cross-Reconstruction loss ($\mathcal{L}_{CA}$) are introduced into model .

$\mathcal{L}_{DA}$ is mainly used to minimize the Wasserstein distance between
the latent multivariate Gaussian distributions to ensure the consistency of
different modalities. The specific forms are as follows:

$\mathcal{L}_{DA}=\sum^{M}_{k=1}\sum^{M}_{t=1\atop t\neq
k}(||\mu^{k}_{l_{s}}-\mu^{t}_{l_{s}}||^{2}_{2}+||(\sigma^{k}_{l_{s}})^{\frac{1}{2}}-(\sigma^{t}_{l_{s}})^{\frac{1}{2}}||^{2}_{Frobenius})^{\frac{1}{2}}$
(2)

where $\mu^{k}_{l_{s}}$ is mean and $\sigma^{k}_{l_{s}}$ is variance
parameters for multivariate Gaussian distributions from $k^{th}$ modality.

Due to the aligned space contains the domain shared and transferable
information, the modality-specific features should also can be reconstructed
with the aligned feature of the same sample but from distinct modalities.
Thus, $\mathcal{L}_{CA}$ is defined as follows:

$\mathcal{L}_{CA}=\sum^{M}_{k=1}\sum^{M}_{t=1\atop t\neq
k}Dis(x^{k}_{l_{s}},D^{k}_{l_{s}}(z^{t}_{l_{s}}))$ (3)

where $z^{k}_{l_{s}}$ is the latent feature from $k^{th}$ modality and
$z^{k}_{l_{s}}=E^{k}(x^{k}_{l_{s}})$. $Dis(\cdot)$ is the Manhattan distance
function.

Thus at the end of this part, we encode the features of different modalities
to an aligned space. However, the model lack the ability to generate
significantly diverse features for unseen classes.

##### Synthetic Feature Generator

For one class $l\in L_{s}\cup L_{u}$, aligned embeddings from visual space
usually sampled from multiple distributions, that is, $z^{2}_{l}$ can be
sampled from
$\\{\mathcal{N}(\mu^{2}_{l,1},\sigma^{2}_{l,1}),\mathcal{N}(\mu^{2}_{l,2},\sigma^{2}_{l,2}),...,\mathcal{N}(\mu^{2}_{l,r},\sigma^{2}_{l,r})\\}$,
where $r$ denotes the $r^{th}$ sample in class $l$. Nevertheless, aligned
embeddings $z^{1}_{l}$ from semantic space only can be sampled from unitary
distributions $\mathcal{N}(\mu^{1}_{l},\sigma^{1}_{l})$. Since the variance of
specific information in visual space is difficult to be captured directly by
semantic features, these information is contained in $\mu^{2}_{l,r}$ as the
unique feature of each sample.

In order to make the instances generated from semantic feature more diverse,
in this part, a module named SFG is designed to explicitly capture the
distribution of visual-specific information in aligned space. The framework of
SFG is shown in Figure 2. By given the paired features
$(z^{1}_{l_{s}},z^{2}_{l_{s}})$ encoded in aligned space, an encoder $E^{3}$
is used to compute the latent parameters $\mu^{3}_{l_{s}}$ and
$\sigma^{3}_{l_{s}}$. After that, the feature $z^{3}_{l_{s}}$ is obtained by
sampling from $\mathcal{N}(\mu^{3}_{l_{s}},\sigma^{3}_{l_{s}})$ with
reparameterization trick. The decoder $D^{3}$ reconstructs $z^{2}_{l_{s}}$
with $z^{3}_{l_{s}}$ and $z^{1}_{l_{s}}$ as input. Therefore, both $E^{3}$ and
$D^{3}$ are conditioned on the latent feature $z^{1}_{l_{s}}$ and then we can
learn it with the follow loss:

$\begin{split}\mathcal{L}_{CVAE}&=\sum^{L_{s}}_{l_{s}=1}\mathbb{E}_{E^{3}(z^{1}_{l_{s}},z^{2}_{l_{s}})}[\log
D^{3}(z^{3}_{l_{s}},z^{1}_{l_{s}})]\\\
&-\beta_{2}D_{KL}(E^{3}(z^{1}_{l_{s}},z^{2}_{l_{s}})||p(z^{3}_{l_{s}}|z^{1}_{l_{s}}))\end{split}$
(4)

where $D_{KL}(\cdot)$ is also denote KL-Divergence like Equation 1,
$p(z^{3}_{l_{s}}|z^{1}_{l_{s}})$ is a prior distribution assumed to be
$\mathcal{N}(0,1)$ and $\log D^{3}(z^{3}_{l_{s}},z^{1}_{l_{s}})$ is the loss
of the reconstruction. $\beta_{2}$ is the hyper-parameter to weight the loss
of these two items. In contrast to [24], using $z^{1}_{l_{s}}$ instead of
original semantic feature as condition can benefit from the following two
points. First, the learning process of the network would not be disturbed by
domain-specific information in the semantic space. Second, $z^{1}_{l_{s}}$ and
target distribution are on the same manifold, which will further reduce the
training difficulty of the network. In particular, we let
$z^{1}_{l_{s}}=\mu^{1}_{l_{s}}$ to stabilize the training process of the
model.

After training SFG, We can provide diverse samples in aligned space for each
unseen classes with the condition from aligned semantic embedding.

####  3.2.2 Training and Inference

##### Training

In training stage, we firstly learn the encoder ($E^{1},E^{2}$) and decoder
($D^{1},D^{2}$) of different modalities simultaneously by minimizing the
combination of the three loss function terms. The objtctive can be formulated
as follows:

$\mathcal{L}_{AFG}=\mathcal{L}_{VAE}+\eta\mathcal{L}_{DA}+\delta\mathcal{L}_{CA}$
(5)

where $\eta$ and $\delta$ are the penalty regularization coefficients for the
loss of two regularization, respectively.

Then, the parameters of alignment feature generator are fixed and only the
parameters of $E^{3}$ and $D^{3}$ are optimized by minimizing:

$\mathcal{L}_{SFG}=\mathcal{L}_{CVAE}$ (6)

##### Inference

For each seen class $l_{s}\in{L}_{s}$, we generate the instances in the
aligned space by sampling from
$\\{\mathcal{N}(\mu^{2}_{l_{s},1},\sigma^{2}_{l_{s},1}),\mathcal{N}(\mu^{2}_{l_{s},2},\sigma^{2}_{l_{s},2}),...,\mathcal{N}(\mu^{2}_{l_{s},r},\sigma^{2}_{l_{s},r})\\}$
where $\mu^{2}_{l_{s},r},\sigma^{2}_{l_{s},r}$ are computed by $E^{2}$ with
$r^{th}$ visual feature of class $l_{s}$ as input. For each class
$l_{u}\in{L}_{u}$, we first obtain the conditional feature
$z^{1}_{l_{u}}=\mu^{1}_{l_{u}}$ by $E^{1}$ with the semantic feature of class
$l_{u}$. Subsequently, we take a set of noises from Gaussian noises
$\mathcal{N}(0,1)$ with the same dimension as $z^{3}$ and connect them with
$z^{1}_{l_{u}}$ respectively. These connected features are input to $D^{3}$,
thus generating multiple diverse samples for $l_{u}$ class. The specific
process can be seen the inference phase in Figure 2.

##  4 Experiments

Table 1: Illustration on datasets used in our experiments

Dataset | Detail | Seen/Unseen Classes | Images | Visaul | Att
---|---|---|---|---|---
AWA2 | coarse | 40/10 | 37322 | 2048 | 85
CUB | fine | 150/50 | 11788 | 2048 | 312
SUN | fine | 645/72 | 14340 | 2048 | 102
FLO | fine | 82/20 | 8189 | 2048 | 1024
APY | coarse | 20/12 | 15339 | 2048 | 64

###  4.1 Experiment setup

##### Datasets

We evaluate our framework on five benchmarking datasets: AWA2 [39], CUB [38],
SUN [27], FLO [30] and APY [8]. They contain 50, 200, 717, 102 and 32
categories, respectively. Other details on these five datasets are listed in
Table 1 for reference.

##### Visual Space and Dataset Split

The visual features with 2048 dimensions we use in all experiments are
extracted by powerful deep Convolutional Neural Networks (CNN), ResNet [12],
which is pre-trained with ImageNet [32]. In this work, we apply Proposed
Splitting (PS) proposed by Xian _et al_. [39] to all datasets.

##### Training Details

For aligned feature generator, we set the aligned space dimension of coarse-
grained dataset (AWA2, APY) to 64 and fine-grained datasets (CUB, SUN, FLO) to
256, because fine-grained datasets often need more information to train an
effective classifier. For coarse-grained datasets, all the network setting
comes from [33]. Since a higher aligned space dimension is set for fine-
grained datasets, we appropriately increase the dimensions of encoder and
decoder. Specifically, 6240 hidden units are used for $E^{1}$ and 4980 hidden
units are used for $D^{1}$. The $E^{2}$ and $D^{2}$ for semantic domain have
3600 and 1330 hidden units, respectively. In addition, all the hyper-
parameters in this part are also follow the settings in [33]. For synthetic
feature generator, $E^{3}$ and $D^{3}$ network are set with one hidden layer
and have 1990 and 1560 hidden units respectively. $\beta_{2}$ is set to 0.6.
The dimension of SFG can be fine tuned by the accuracy on the validation
dataset, but it is worth noting that the training data of our final model come
from the training dataset and validation dataset. Learning rate is set to
0.00015 and training epoch is 100 across all the datasets. After training, a
linear classifier is used to classify in aligned space.

##### Evaluation Metric

We average the classification accuracy of each test class and report the
average top accuracy as following:

$Acc_{u}=\frac{1}{|\mathcal{U}|}\sum_{{u}\in\mathcal{U}}\frac{acc_{u}}{|{u}|}$
(7)

where $acc_{u}$ denotes the number of correctly classified samples for unseen
classes $l_{u}$.

In the GZSL setting, we report the harmonic mean of the accuracy over seen and
unseen classes which is defined as:

$Acc_{H}=\frac{2*Acc_{s}*Acc_{u}}{Acc_{s}+Acc_{u}}$ (8)

where $Acc_{s}$ denotes the mean class accuracy on seen classes, and $Acc_{u}$
indicates the mean class accuracy on unseen class.

Table 2: Comparison between our DFS model with SOTAs using the generalized
zero-shot learnig settings. Here, $Acc_{s}$ is TOP-1 accuracy on seen data,
$Acc_{u}$ is TOP-1 accuracy on unseen data, and $H$ is Harmonic Mean. Baseline
reports the results of aligned feature generator. $\wr$ denotes methods based
on embedded model, $\dagger$ denotes the methods based on GAN and $\ddagger$
denotes the methods based on VAE, respectively. The best results are in red,
and the second best results are in blue. (Best viewed in colour)

| Method | AWA2 | CUB | SUN | FLO | APY
---|---|---|---|---|---|---
| $Acc_{s}$ | $Acc_{u}$ | $Acc_{H}$ | $Acc_{s}$ | $Acc_{u}$ | $Acc_{H}$ | $Acc_{s}$ | $Acc_{u}$ | $Acc_{H}$ | $Acc_{s}$ | $Acc_{u}$ | $Acc_{H}$ | $Acc_{s}$ | $Acc_{u}$ | $Acc_{H}$
$\wr$ | DAP [19] | 84.7 | 0.0 | 0.0 | 67.9 | 1.7 | 3.3 | 25.1 | 4.2 | 7.2 | - | - | - | 78.3 | 4.8 | 9.0
IAP [19] | 87.6 | 0.9 | 1.8 | 72.8 | 0.2 | 0.4 | 37.8 | 1.0 | 1.8 | - | - | - | 65.6 | 5.7 | 10.4
CONSE [26] | 90.6 | 0.5 | 1.0 | 72.2 | 1.6 | 3.1 | 39.9 | 6.8 | 11.6 | - | - | - | 91.2 | 0.0 | 0.0
CMT [34] | 90 | 0.5 | 1 | 49.8 | 7.2 | 12.6 | 21.8 | 8.1 | 11.8 | - | - | - | 85.2 | 1.4 | 2.8
SSE [44] | 82.5 | 8.1 | 14.8 | 46.9 | 8.5 | 14.4 | 36.4 | 2.1 | 4 | - | - | - | 78.9 | 0.2 | 0.4
ALE [1] | 81.8 | 14 | 23.9 | 62 | 23.7 | 34.4 | 33.1 | 21.8 | 26.3 | 61.6 | 13.3 | 21.9 | 73.7 | 4.6 | 8.7
SAE [16] | 82.2 | 1.1 | 2.2 | 54.0 | 7.8 | 13.6 | 18.0 | 8.8 | 11.8 | - | - | - | 80.9 | 0.4 | 0.9
EZSL [31] | 77.8 | 5.9 | 11.0 | 63.8 | 12.6 | 21.0 | 27.9 | 11.0 | 15.8 | - | - | - | 70.1 | 2.4 | 4.6
PSR [3] | 73.8 | 20.7 | 32.3 | 54.3 | 24.6 | 33.9 | 37.2 | 20.8 | 26.7 | - | - | - | 51.4 | 13.5 | 21.4
$\dagger$ | f-CLSWGAN [40] | 68.9 | 52.1 | 59.4 | 57.7 | 43.7 | 49.7 | 36.6 | 42.6 | 39.4 | 73.8 | 59.0 | 65.6 | 61.7 | 32.9 | 42.9
Cycle-WGAN [9] | 63.4 | 59.6 | 59.8 | 59.3 | 47.9 | 53.0 | 33.8 | 47.2 | 39.4 | 69.2 | 61.6 | 65.2 | - | - | -
SABR [28] | 93.9 | 30.3 | 46.9 | 58.7 | 55.0 | 56.8 | 35.1 | 50.7 | 41.5 | - | - | - | - | - | -
f-VAEGAN-D2 [41] | 70.6 | 57.6 | 63.5 | 60.1 | 48.4 | 53.6 | 38.0 | 45.1 | 41.3 | 74.9 | 56.8 | 64.6 | - | - | -
LisGAN [20] | 76.3 | 52.6 | 62.3 | 57.9 | 46.5 | 51.6 | 37.8 | 42.9 | 40.2 | 83.8 | 57.7 | 68.3 | - | - | -
Zero-VAE-GAN [10] | 70.9 | 57.1 | 62.5 | 47.9 | 43.6 | 45.5 | 30.2 | 45.2 | 36.3 | - | - | - | 52.2 | 32.0 | 39.7
TF-VAEGAN [25] | 75.1 | 59.8 | 66.6 | 64.7 | 52.8 | 58.1 | 40.7 | 45.6 | 43.0 | 84.1 | 62.5 | 71.7 | 57.4 | 35.9 | 44.2
$\ddagger$ | CVAE [24] | - | - | 51.2 | - | - | 34.5 | - | - | 26.7 | - | - | - | - | - | -
OCD-CVAE [15] | 73.4 | 59.5 | 65.7 | 59.9 | 44.8 | 51.3 | 42.9 | 44.8 | 43.8 | - | - | - | - | - | -
CADA-VAE [33] | 75.0 | 55.8 | 63.9 | 53.5 | 51.6 | 52.4 | 35.7 | 47.2 | 40.6 | 80.7 | 54.0 | 64.7 | 53.2 | 34.8 | 42.1
DE-VAE [23] | 78.9 | 58.8 | 67.4 | 56.3 | 52.5 | 54.3 | 36.9 | 45.9 | 40.9 | - | - | - | - | - | -
Baseline | 75.0 | 55.8 | 63.9 | 57.3 | 49.7 | 53.3 | 35.5 | 49.0 | 41.2 | 77.1 | 54.2 | 63.7 | 53.2 | 34.8 | 42.1
DFS (Ours) | 78.6 | 58.4 | 67.2 | 59.2 | 57.4 | 58.3 | 39.2 | 53.8 | 45.4 | 84.3 | 60.1 | 70.2 | 60.7 | 37.1 | 46.0

###  4.2 Comparison with SOTAs for GZSL

We compare our model with recent state-of-the-art methods on generalized zero-
shot learning, and the results are shown in Table 2. Since our whole model did
not use any data from the testing dataset for training, only comparisons with
inductive ZSL are made in all experiments. For a fair comparison, we also
trained a classifier using the features obtained from aligned feature
generator and calculated the accuracy of its seen and unseen classes and the
results can be seen on the model named Baseline.

Compared with baseline, significant improvement can be observed on all
benchmarks. The accuracy difference between our model and Baseline is as
follows: 67.2% vs 63.9% on AWA2, 58.3% vs 53.3% on CUB, 45.4% vs 41.2% on SUN,
70.2% vs 63.7% on FLO and 46.0% vs 42.1% on APY. At the same time, both
$Acc_{s}$ and $Acc_{u}$ are improved, we attribute this high performance gain
to the using of diverse samples for training classifier. Compared with CVAE
[24], we use the aligned space as classification space while using the aligned
semantic information as the condition. These changes resulted in 16%, 23.8%
and 18.7% improvement in our network over CVAE for AWA2, CUB and SUN,
respectively. The model f-VAEGAN-D2 [41] which based on VAE-GAN reports
classification accuracies of 63.5%, 53.6%, 41.3% and 64.6% on AWA2, CUB, SUN,
and FLO, respectively. The improved model of f-VAEGAN-D2, TF-VAEGAN [25],
obtains state-of-the-art classification scores of 66.6%, 58.1%, 43.0% and
71.7% on the same datasets. But it is worth noting that it introduces GAN and
applies more complex training process, which often makes the model unable to
learn stably. In contrast, all of our modules are built on VAE. Hence, our
method is simple and can be trained stably without using any training skills.

In addition, DFS outperforms TF-VAEGAN 0.6%, 0.2%, 2.4% and 1.8% on four
datasets and set new state-of-the-art. Similarly to our work, DE-VAE [23] is
also an improved model based on CADA-VAE. However, except for AWA2, the
performance of our model is far better than it. Since the motivation of DFS
and DE-VAE do not conflict with each other, we speculate that integrating the
two methods together will yield superior results. Nevertheless, this
experiment cannot be performed in this paper because the code of DE-VAE is not
available. In the conventional zero-shot learning, DFS also provides
favourable performance, 69.1% on AWA2, 64.7% on CUB, 64.4% SUN, 68.9% on FLO
and 43.6% on APY. Nevertheless, we focus on the more practical and challenging
GZSL setting in this work.

###  4.3 Ablation Study

##### Generalization Capabilities

As mentioned in Section 3.2.1, our innovation point is not limited to a
specific cross model, but can be directly introduced into most of the
previously proposed cross models. To verify the generalizability of our idea,
we perform an experiment by integrating the contributions proposed in this
work in ReVISE [14]. The results in Figure 3 show that DFS-ReVISE outperforms
ReVISE on all three datasets. These performance gains entirely benefit from
the fact that we synthetise more diverse samples for unseen classes in latent
space, which is helpful to learn an effective classifier. Since CADA-VAE is
the powerful model to learn aligned space in GZSL, we implement AFG with CADA-
VAE in the following experiments for discussion.

Figure 3: Verification on the generalization capability of our DFS model.
Here, we replace CADA-VAE with ReVISE to generate the aligned features and
exploit the harmonic mean accuracy ($\%$) as the metric. Results on AWA2, CUB
and SUN datasets are shown. (Best viewed in color)

##### Dimension of Aligned Space

Figure 4 presents the summary statistics of accuracy under different
dimensionality of the aligned space on three datasets. It can be observed that
with the increasing dimensionality, the accuracy of DFS increases initially
until $d=256$, $d=256$ and $d=320$ for CUB, SUN and FLO, respectively.
Intuitively, higher dimensions tend to contain more complex information, such
as information specific to visual space. It is our assertion that part of the
visual-specific information is beneficial for the classification task because
similar categories may not be classified by domain shared information. The
visual space features usually come from the powerful model trained on large-
scale datasets and thus will contain more information that can effectively
distinguish between different categories. Although these information is not
directly derived from the semantic features, we believe that some of it can be
interpreted by semantic features. In our work, SFG aids to capture the
distribution of visual-specific information that can be reasonably inferred by
class embeddings. However, the distribution will become extremely complex
while the number of dimension is too large, which will result in SFG unable to
learn the real distribution, so the performance begins to decline. In order to
make the model not lose generality, we set the $d=256$ for all fine-grained
datasets.

Figure 4: Ablation study on the effect of the dimentionality of the aligned
features on CUB, SUN and FLO datasets. Harmonic mean accuracy is used as the
metric. (Best viewed in color)

##### Feature Visualization

(a)

(b)

(c)

Figure 5: Visualization of samples in aligned space with t-SNE on CUB (1st
row), SUN (2nd row) and APY (3rd row). (a) Samples generated for seen classes.
(b) Samples generated for unseen classes by Baseline. (c) Samples generated
for unseen classes by DFS. (Best viewed in color)

To verify that the samples generated by DFS for unseen classes are more
diverse, we visualized the features in the aligned space by using t-SNE [36].
For each dataset, we randomly selected 10 seen classes and 10 unseen classes,
respectively. Each seen class takes 400 samples in aligned space using
multiple visual features. And for each unseen class, 400 samples are generated
by semantic vectors. It is not difficult to find by Figure 5(a) and Figure
5(b), compared to the seen classes, the diversity of the unseen classes
instances generated by Baseline is far from adequate. When the two classes are
closer to each other, the classifier would tend to predict the test sample as
the class with greater diversity. Figure 5(b) and Figure 5(c) show a
comparison between Baseline and our methods. Obviously, in all three datasets,
the instances synthesised by DFS are more diverse. This will help the
classifier learn the decision boundary with better generalization performance.

Table 3: Ablation study on the influence of different features as conditions on performance. $a$ denotes the original semantic feature, $\mu^{1}$ and $\sigma^{1}$ are the parameters computed by $E^{1}$ with $a$ as input. Harmonic mean accuracy is used as the metric. Condition | CUB | FLO | APY
---|---|---|---
$a$ | 56.4 | 68.6 | 40.9
$z^{1}\sim\mathcal{N}(\mu^{1},\sigma^{1})$ | 55.0 | 69.2 | 42.7
$z^{1}=\mu^{1}$ | 58.3 | 70.2 | 46.0

##### Choice of Condition

To further justify the influence of different supervision signals on the
performance of the model, we use different features as condition to train SFG
module. It is apparent from Table 3 that using the mean of semantic features
in aligned space as supervised information provides the best results on all
three datasets. We analyze that this is mainly attributed to the fact that
$z^{1}$ is on the same manifold as the target distribution to be learned,
which significantly reduces the training difficulty of the generator. But at
the same time, if we let $z^{1}\sim\mathcal{N}(\mu^{1},\sigma^{1})$, it will
make an unstable condition and therefore interfere with the learning process
of the generator.

##  5 Conclusion

In this paper, we present a novel Diverse Feature Synthesis (DFS) model for
enhancing the generalizability of generative based strategy for the
generalized zero-shot learning task. In particular, DFS effectively improves
the feature diversity of unseen classes with a low-complexity implementation.
For this purpose, DFS first utilizes an aligned feature generator to transfer
features from semantic and visual spaces into the aligned space, offering a
way for simplifying the feature generation process. Then, DFS exploits a
synthesised feature generator to produce aligned features for unseen classes
via incorporating visual knowledge, thus leading to feature diversity
improvement. In this way, DFS overcomes drawbacks of prior works, and helps to
learn a more accurate and robust classifier for the GZSL task. Comprehensive
experiments on multiple benchmarks verify the effectiveness of our proposed
DFS model for improving the performance in GZSL settings.

## References

  * [1] Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid.  Label-embedding for attribute-based classification.  In CVPR, 2013.
  * [2] D. P. K. andMax Welling.  Auto-encoding variational bayes.  In ICLR, 2014.
  * [3] Y. Annadani and S. Biswas.  Preserving semantic relations for zero-shot learning.  In CVPR, 2018.
  * [4] M. Arjovsky and L. Bottou.  Towards principled methods for training generative adversarial networks.  In ICLR, 2017.
  * [5] J. Bao, D. Chen, F. Wen, H. Li, and G. Hua.  Cvae-gan: fine-grained image generation through asymmetric training.  In ICCV, pages 2745–2754, 2017.
  * [6] B. Brattoli, J. Tighe, F. Zhdanov, P. Perona, and K. Chalupka.  Rethinking zero-shot video classification: End-to-end training for realistic applications.  In CVPR, 2020.
  * [7] S. Changpinyo, W.-L. Chao, B. Gong, and F. Sha.  Synthesized classifiers for zero-shot learning.  In CVPR, 2016.
  * [8] A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth.  Describing objects by their attributes.  In CVPR, 2009.
  * [9] R. Felix, I. Reid, G. Carneiro, et al.  Multi-modal cycle-consistent generalized zero-shot learning.  In ECCV, 2018.
  * [10] R. Gao, X. Hou, J. Qin, J. Chen, L. Liu, F. Zhu, Z. Zhang, and L. Shao.  Zero-vae-gan: Generating unseen features for generalized and transductive zero-shot learning.  IEEE Transactions on Image Processing, 29:3665–3680, 2020.
  * [11] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In NeurIPS, 2014.
  * [12] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [13] H. Huang, C. Wang, P. S. Yu, and C.-D. Wang.  Generative dual adversarial network for generalized zero-shot learning.  In CVPR, 2019.
  * [14] Y.-H. Hubert Tsai, L.-K. Huang, and R. Salakhutdinov.  Learning robust visual-semantic embeddings.  In ICCV, 2017.
  * [15] R. Keshari, R. Singh, and M. Vatsa.  Generalized zero-shot learning via over-complete distribution.  In CVPR, 2020.
  * [16] E. Kodirov, T. Xiang, and S. Gong.  Semantic autoencoder for zero-shot learning.  In CVPR, 2017.
  * [17] Y. Kumar, D. Sahrawat, S. Maheshwari, D. Mahata, A. Stent, Y. Yin, R. R. Shah, and R. Zimmermann.  Harnessing gans for zero-shot learning of new classes in visual speech recognition.  In AAAI, 2020.
  * [18] V. Kumar Verma, G. Arora, A. Mishra, and P. Rai.  Generalized zero-shot learning via synthesized examples.  In CVPR, 2018.
  * [19] C. H. Lampert, H. Nickisch, and S. Harmeling.  Attribute-based classification for zero-shot visual object categorization.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(3):453–465, 2013.
  * [20] J. Li, M. Jing, K. Lu, Z. Ding, L. Zhu, and Z. Huang.  Leveraging the invariant side of generative zero-shot learning.  In CVPR, 2019.
  * [21] Y. Li, Peike andWei and Y. Yang.  Consistent structural relation learning for zero-shot segmentation.  In NeurIPS, 2020.
  * [22] R. Luo, N. Zhang, B. Han, and L. Yang.  Context-aware zero-shot recognition.  In AAAI, 2020.
  * [23] P. Ma and X. Hu.  A variational autoencoder with deep embedding model for generalized zero-shot learning.  In AAAI, 2020.
  * [24] A. Mishra, S. Krishna Reddy, A. Mittal, and H. A. Murthy.  A generative model for zero shot learning using conditional variational autoencoders.  In CVPR, 2018.
  * [25] S. Narayan, A. Gupta, F. S. Khan, C. G. Snoek, and L. Shao.  Latent embedding feedback and discriminative features for zero-shot classification.  arXiv, 2020.
  * [26] M. Norouzi, T. Mikolov, S. Bengio, Y. Singer, J. Shlens, A. Frome, G. S. Corrado, and J. Dean.  Zero-shot learning by convex combination of semantic embeddings.  In ICLR, 2014.
  * [27] G. Patterson and J. Hays.  Sun attribute database: Discovering, annotating, and recognizing scene attributes.  In CVPR, 2012.
  * [28] A. Paul, N. C. Krishnan, and P. Munjal.  Semantically aligned bias reducing zero shot learning.  In CVPR, 2019.
  * [29] S. Rahman, S. Khan, and N. Barnes.  Improved visual-semantic alignment for zero-shot object detection.  In AAAI, 2020.
  * [30] S. Reed, Z. Akata, H. Lee, and B. Schiele.  Learning deep representations of fine-grained visual descriptions.  In CVPR, 2016.
  * [31] B. Romera-Paredes and P. Torr.  An embarrassingly simple approach to zero-shot learning.  In ICML, 2015.
  * [32] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision, 115(3):211–252, 2015\.
  * [33] E. Schonfeld, S. Ebrahimi, S. Sinha, T. Darrell, and Z. Akata.  Generalized zero-and few-shot learning via aligned variational autoencoders.  In CVPR, 2019.
  * [34] R. Socher, M. Ganjoo, C. D. Manning, and A. Ng.  Zero-shot learning through cross-modal transfer.  In NeurIPS, 2013.
  * [35] J. W. Soh, S. Cho, and N. I. Cho.  Meta-transfer learning for zero-shot super-resolution.  In CVPR, 2020.
  * [36] L. Van der Maaten and G. Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [37] V. K. Verma and P. Rai.  A simple exponential family framework for zero-shot learning.  In ECML, 2017.
  * [38] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [39] Y. Xian, C. H. Lampert, B. Schiele, and Z. Akata.  Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(9):2251–2265, 2018.
  * [40] Y. Xian, T. Lorenz, B. Schiele, and Z. Akata.  Feature generating networks for zero-shot learning.  In CVPR, 2018.
  * [41] Y. Xian, S. Sharma, B. Schiele, and Z. Akata.  f-vaegan-d2: A feature generating framework for any-shot learning.  In CVPR, 2019.
  * [42] C. Zhan, D. She, S. Zhao, M.-M. Cheng, and J. Yang.  Zero-shot emotion recognition via affective structural embedding.  In ICCV, 2019.
  * [43] L. Zhang, T. Xiang, and S. Gong.  Learning a deep embedding model for zero-shot learning.  In CVPR, 2017.
  * [44] Z. Zhang and V. Saligrama.  Zero-shot learning via semantic similarity embedding.  In ICCV, 2015.
  * [45] Z. Zhang and V. Saligrama.  Zero-shot learning via joint latent similarity embedding.  In CVPR, 2016.
  * [46] T. Zhou, S. Wang, Y. Zhou, Y. Yao, J. Li, and L. Shao.  Motion-attentive transition for zero-shot video object segmentation.  In AAAI, 2020.
  * [47] Y. Zhu, M. Elhoseiny, B. Liu, X. Peng, and A. Elgammal.  A generative adversarial approach for zero-shot learning from noisy texts.  In CVPR, 2018.


Title: Detecting Gait Abnormalities in Foot-Floor Contacts During Walking
Through Footstep-Induced Structural Vibrations

$\begin{array}[]{ll}\text{Authors}:&\text{Yiwen Dong}\\\ &\text{Yuyan Wu}\\\
&\text{Haeyoung Noh}\end{array}$

ABSTRACT

This study aims to detect abnormal human gait patterns through the dynamic
response of floor structures during foot-floor interactions. Gait abnormality
detection is critical for the early discovery and progressive tracking of
musculoskeletal and neurological disorders, such as Parkinson’s and Cerebral
Palsy. Especially, analyzing the foot-floor contacts during walking provides
important insights into gait patterns, such as contact area, contact force,
and contact time, enabling gait abnormality detection through these
measurements. Existing studies use various sensing devices to capture such
information, including cameras, wearables, and force plates. However, the
former two lack force-related information, making it difficult to identify the
causes of gait health issues, while the latter has limited coverage of the
walking path.

In this study, we leverage footstep-induced structural vibrations to infer
foot-floor contact profiles, which allows force-informed and more wide-ranged
gait abnormality detection. The main challenge lies in modeling the complex
force transfer mechanism between the foot and the floor surfaces, leading to
difficulty in reconstructing the force and contact profile during foot-floor
interaction using structural vibrations. To overcome the challenge, we first
characterize the floor vibration for each contact type (e.g., heel, midfoot,
and toe contact) to understand how contact forces and areas affect the induced
floor vibration. Then, we leverage the time-frequency response spectrum
resulting from those contacts to develop features that are representative of
each contact type. Finally, gait abnormalities are detected by comparing the
predicted foot-floor contact force and motion with the healthy gait. To
evaluate our approach, we conducted a real-world walking experiment with 8
subjects. Our approach achieves 91.6% and 96.7% accuracy in predicting contact
type and time, respectively, leading to 91.9% accuracy in detecting various
types of gait abnormalities, including asymmetry, dragging, and midfoot/toe
contacts.

000 Yiwen Dong, Ph.D. Candidate, Email<EMAIL_ADDRESS>Structures as
Sensors Lab, Department of Civil and Environmental Engineering, Stanford
University, Stanford, CA, USA.

INTRODUCTION

Structural vibrations induced by human footsteps during walking contain
important gait pattern information, allowing ubiquitous gait health monitoring
and abnormality detection in daily life [1, 2]. Detecting gait abnormalities
is critical for the early discovery and progressive tracking of
musculoskeletal and neurological disorders, such as Parkinson’s and Cerebral
Palsy [3, 4, 5]. Gait abnormalities are typically reflected in deviations of
posture, balance, and speed of walking from normal walking patterns, which
indicates underlying conditions that affect the muscle groups or nervous
systems [6]. Among them, analyzing the dynamic interaction between the foot
and the floor during walking is a critical aspect because it helps to
understand how the ground reaction forces act and transmit through the body
[7]. Modeling the foot-floor contact can inform the gait health status to
provide better design of medical interventions in neurological/musculoskeletal
disorders, manage fall risks, and prevent further injuries or complications.

There are existing studies that detect gait abnormalities during walking,
including force plates, wearable devices, and cameras [8, 9, 10, 11, 12].
Force plates are the most commonly used technique for measuring foot-floor
contact forces during walking. However, the coverage of a force plate is
typically limited to one footstep only. Cameras and wearables measure the
movement of the foot. They are more portable and wide-ranged than force plates
and are more practical for everyday use. However, the camera measurements are
limited to kinematic aspects of the gait, making it difficult to infer the
joint forces and identify abnormal loading patterns. The wearables require a
person to carry devices, which may cause discomfort and inconvenience.

In this study, we leverage footstep-induced floor vibrations to detect gait
abnormalities in foot-floor contacts during walking. The primary intuition is,
as a person’s foot contacts the floor and generates structural vibrations
during walking, we capture those vibrations through floor-mounted geophone
sensors. By analyzing the vibration signals, we infer the foot-floor contact
profiles, including contact type and duration. Our approach is non-invasive,
wide-ranged (up to 20 m range [13]), and produces more comprehensive gait
information that informs both force and movement.

The main challenge of this study is the complex mechanism during the foot-
floor contact - not only the characteristics of the contact (i.e., force
magnitude, direction, contact area) are changing when walking, they are also
entangled with the dynamic property of the structure in the observed
vibrations. To overcome this challenge, we first identify foot-floor contact
types that are commonly described in medical settings and characterize their
influence on the resultant vibrations. Then, we leverage the time-frequency
response spectrum resulting from those contacts to develop features that are
representative of each contact type. Finally, we detect abnormal gait patterns
through machine learning, which translates the model estimations into clinical
insights about gait health.

The contributions of the study are that we:

  1. 1.

Develop a novel approach to detect abnormal gait using footstep-induced floor
vibrations;

  2. 2.

Characterize structural vibrations induced by foot-floor contacts to develop
representative features; and

  3. 3.

Evaluate our approach through a real-world human experiment to demonstrate its
effectiveness

We conducted a real-world walking experiment with 8 human subjects. Our
approach achieves 91.6% and 96.7% accuracy in predicting contact type and
duration, respectively, and achieves a 91.9% accuracy in detecting various
types of gait abnormalities. (including toe-walking, dragging, and asymmetry),
which establishes its efficacy in gait health monitoring.

CHARACTERIZING FOOTSTEP-INDUCED STRUCTURAL VIBRATIONS FOR ABNORMAL CONTACT
DETECTION

In this section, we characterize the complex mechanism during foot-floor
contact when walking. Specifically, we first discuss how various contact types
affect the resultant vibrations and then conduct time-frequency response
analysis to quantify and interpret such effects.

Background of Foot-Floor Contact Types

To characterize various foot-floor contacts during walking, we first group
them into three types that are commonly described in medical settings [14,
15], including 1) heel contact, 2) midfoot contact, and 3) toe contact. As
shown in Figure 1, each type of contact has a distinct form in terms of
contact area and force distribution. To understand the force transfer
mechanism during the foot-floor contact, we also measure the footstep forces
(ground reaction force (GRF) in clinical terms) during these contacts.

Figure 1: Illustration of three types of foot-floor contacts and vertical
footstep force variations during the contacts.

  * •

Heel Contact: A heel contact is when the heel of the foot strikes the floor
first, which is typical in healthy adults. The force has both frictional and
normal components and the normal component magnitude varies around the body
weight.

  * •

Midfoot Contact: A midfoot means the entire foot contacts the floor first,
which is an indication of abnormal gait with insufficient ankle dorsiflexion.
The duration of the contact is longer with fewer force variations.

  * •

Toe Contact: A toe contact means the toe strikes the floor whereas the heel
remains lifted. This is typically seen in individuals with later-stage gait
disorders where excessive muscle contraction occurs. The force is
significantly larger due to the impulse of the toe strike.

Modeling Foot-Floor Contacts using Dynamic Floor Response Analysis

To examine how various foot-floor contacts and structural properties affect
the resultant floor vibration, we formulate the problem by simplifying the
complex situation. We first assume that each footstep exerts a force
$F_{l}(t)$ at a simply supported beam within the linear elastic range, as
described in Figure 2.

Figure 2: Structural dynamics model for foot-floor contact.

Based on the equation of motion [16], we have:

$M\ddot{u}(t)+C\dot{u}(t)+Ku(t)=F_{l}(t)=-S(t)\ddot{u}_{f}(t)$ (1)

where $S(t)$ is the equivalent spatial loading matrix, which can be further
expressed as the multiplication of body weight $M_{bw}$ and the spatial
distribution of the force $\iota(t)$ (i.e., $S(t)=M_{bw}\iota(t)$).
$\ddot{u}_{f}(t)$ is the foot acceleration, which is proportional to the force
described in Figure 1.

In order to simplify the complex temporal and spatial dependency, we focus on
the initial contact and assume the contact surface and force are constant
during that time (i.e., $S(t)=S$). This is because the main difference between
the three contact types is the initial contact force and surface, which
results in distinct vibration patterns. Assuming temporal and spatial
independence, we leverage modal decomposition and Fourier transform, resulting
in the modal displacement response and the time-frequency response as follows:

$u_{i}(t)=P_{i}\phi_{i}D_{i}(t)\Rightarrow\mathcal{F}{u_{i}(t)}=P_{i}\mathcal{F}{\phi_{i}D_{i}(t)}$
(2)

where $P_{i}=\frac{1}{m^{*}_{i}}\phi^{T}_{i}S$ is the modal participation
factor of $i$-th mode; $\phi_{i}$ describes the mode shape; $D_{i}(t)$
describes the generalized modal displacement resulting from the foot
acceleration $\ddot{u}_{f}(t)$.

In summary, the above derivation suggests that the time-frequency response of
the floor is determined by 1) the body weight, 2) the spatial distribution of
the footstep force, 3) the foot acceleration, as in amplitude and direction
during the contact. This formulation provides a theoretical foundation for
modeling and predicting various foot-floor contact types.

Time-Frequency Response Analysis for Typical Foot-Floor Contacts

After establishing the theoretical basis, we conduct a time-frequency response
analysis of the observed floor vibration to discuss its relation to typical
contact characteristics. Figure 3 demonstrates the time domain, frequency
domain, and wavelet (time-frequency) domain data plots of the floor vibration
under typical foot-floor contacts from the same person. The peaks and hotspots
in the wavelet plot indicate the activated modes under a specific contact. We
observe that midfoot contact induces lower modes (11, 53 Hz) than heel/toe
contacts (11, 53, 80, 112, 165, 200 Hz).

Figure 3: Floor vibration induced by typical foot-floor contacts from time,
frequency, and wavelet domain.

To understand how the above observation is related to the foot-floor contact
formulation, a qualitative analysis is summarized below:

  * •

Effect of the Body Weight: The amplitude in the frequency spectrum increases
proportionally to the body weight because the modal participation factor is
$S=M_{bw}\iota$. For the same person, the body weight does not affect the
variation of the frequency spectrum across contact types as the body weight is
typically constant.

  * •

Effect of the Contact Area: A larger contact area by midfoot contact typically
induces lower modes of vibration than heel/toe contacts. This is because the
modal participation factor is an inner product of the spatial loading matrix
$S$ and mode shape $\phi_{i}$, which can be regarded as a weighted sum of the
mode shape amplitudes within the contact area. For a wider spread load
distribution $S$ for the midfoot contact, a higher mode produces a smaller sum
because it is more likely to have oscillations in shape within the contact
area that cancel out in the inner product. A lower mode is less likely to have
oscillation within the contact area. On the other hand, a more concentrated
load $S$ results in a higher inner product even for higher mode because it is
less likely to have oscillations within that narrow contact area.

  * •

Effect of the Force Direction: A force applied in the horizontal direction
along the floor surface typically induces higher modes than the same force
applied in the vertical direction. This is because the horizontal force
affects the vertical vibration mainly through the floor thickness with waves
that have shorter wavelengths. As a result, the modal participation factor for
higher modes is larger for horizontal forces.

Since the area of midfoot contact is larger than the heel/toe contact and the
force direction is mostly vertical, it tends to induce lower modes than the
heel/toe contact. In addition, the toe contact has a less proportion of
horizontal force than the heel contact, so it has a lower proportion of the
high-frequency components ($\geq$100 Hz) in the spectrum.

Based on the analysis above, we develop features using the frequency spectrum
of the vibration signals to represent the characteristics of the various foot-
floor contact types. The features include the amplitudes at the union of
dominant frequencies induced by these three contacts.

ABNORMAL GAIT DETECTION USING FOOTSTEP-INDUCED FLOOR VIBRATIONS

Based on clinical studies, abnormal foot-floor contacts typically manifest
distinct contact types and durations. For example, toe-walking is a
pathological gait with toe contacts, and limping has an asymmetrical contact
duration [14, 5, 15].

Our framework for abnormal gait detection incorporates both contact type and
duration, shown in Figure 4. First, we pre-process the input floor vibrations
generated from footsteps through noise filters and footstep detection
algorithms - for noise filtering, we use the lowpass and wiener filters; for
footstep detection, we set thresholds on cumulative wavelet coefficients [1].
Then, we detect the initial contact time when the higher frequencies are first
activated and the foot-off time when the free vibration starts to dominate the
spectrum, as developed in our prior work [2]. Meanwhile, we model the contact
types using the frequency response features discussed in the previous section.
Finally, we compare the similarity between these features and the pre-
collected data from the typical contacts (both normal and abnormal) through a
similarity-aware machine learning model to predict the probability of the
incoming data being abnormal.

Figure 4: The abnormal gait detection framework.

In our approach, a support vector machine with a radial basis function kernel
is used to maximize the cluster margins between normal contact and each of the
abnormal contact types. This is because the model has a good balance between
bias and variance, and can learn from a limited amount of training data.

To further empower the model to be aware of similarity, we apply Platt scaling
to fit the margins for each sample, which provides a unified measurement of
how similar the observed footsteps are to the abnormal ones [17], described as
follows:

$P(y=abnormal|f(x_{i}))=\frac{1}{1+exp(wf(x_{i})+b)}$ (3)

where $f(x_{i})$ is the margin of sample $x_{i}$, $w$, $b$ are the parameters
of the probablistic scaling model. The larger the probability is, the more
likely the observed footstep is abnormal.

EVALUATION THROUGH REAL-WORLD WALKING EXPERIMENTS

To evaluate the effectiveness of our approach, we conducted a real-world
walking experiment with 8 human subjects (ages 18-40 years old). All
experiments were conducted in accordance with the approved IRBs.

Experiment Setup

The experiment setup involves four geophones for data collection and a Vicon
Motion Capture system for ground truth. The geophones are placed at the side
of a wooden walkway as shown in Figure 5.

During the experiment, 5 test subjects walk across the walkway 30-40 times
using their natural gait (which may or may not have abnormal contacts). 3
control subjects walk according to four types of abnormal gaits to simulate
the abnormal contacts, including midfoot strike, toe-walking, dragging, and
walking asymmetry.

Figure 5: Experiment setup with geophones (data) and Vicon cameras (ground
truth). Figure 6: Contact types visualized in time-frequency response features
using t-SNE (left). Accuracy of contact type prediction for three subjects
(right).

Results and Discussion

Contact Type Prediction. Our approach has an average of 91.6% and 96.7%
accuracy in predicting contact types and duration using the features extracted
from the time-frequency response analysis. Figure 6 (left) shows that these
features produce distinct clusters for different contact types through t-SNE
visualization. Figure 6 (right) shows consistent accuracy of the 3 control
subjects.

Abnormal Gait Detection. Our approach has an average of 91.9% accuracy in
detecting four types of abnormal contacts (see Figure 8 (left)). Figure 7
shows the features from time-frequency analysis form distinct clusters for
dragging and asymmetry gait, respectively. Through our similarity-aware model,
we compute the probability of having abnormal contacts for each test subject.
Figure 8 (right) shows a sample subject’s probability profile among the four
abnormal types as an example. For this sample subject, the median values are
relatively low despite the variation among footsteps, which means his/her gait
is unlikely to be abnormal.

Figure 7: Dragging and asymmetry visualized in time-frequency features using
t-SNE. Figure 8: Mean accuracy of abnormal gait detection among 8 participants
(left). Probability of having abnormal contacts for a sample subject (right).

CONCLUDING REMARKS

In this study, we leverage footstep-induced floor vibrations to detect gait
abnormalities in foot-floor contacts during walking. To overcome the challenge
of the complex force transfer mechanism during the contacts, we characterize
the vibration signals induced by three typical foot-floor contacts through
dynamics formulation and controlled experiments. We introduce an abnormal gait
detection framework that models contact time and type to make predictions on
the abnormal contact probability. We evaluate our approach through a real-
world experiment and results show promising accuracy in predicting contact
types and detecting abnormal gait patterns.

ACKNOWLEDGMENT

This work was funded by the U.S. National Science Foundation (under grant
number NSF-CMMI-2026699). The views and conclusions contained here are those
of the authors and should not be interpreted as necessarily representing the
official policies or endorsements, either express or implied, of any
University, the National Science Foundation, the United States Government or
any of its agencies.

## References

  * 1. Dong, Y., J. J. Zou, J. Liu, J. Fagert, M. Mirshekari, L. Lowes, M. Iammarino, P. Zhang, and H. Y. Noh. 2020. “MD-Vibe: physics-informed analysis of patient-induced structural vibration data for monitoring gait health in individuals with muscular dystrophy,” in _Adjunct proceedings of the 2020 ACM international joint conference on pervasive and ubiquitous computing and proceedings of the 2020 ACM international symposium on wearable computers_ , pp. 525–531.
  * 2. Dong, Y. and H. Y. Noh. “Structure-Agnostic Gait Cycle Segmentation for In-Home Gait Health Monitoring through Footstep-Induced Structural Vibrations,” .
  * 3. Pistacchi, M., M. Gioulis, F. Sanson, M. Ennio De Giovannini, G. Filippi, F. Rossetto, and S. Z. Marsala. 2017. “Gait analysis and clinical correlations in early Parkinson’s disease,” _Functional neurology_ , 32(1):28.
  * 4. Martin, C. L., B. A. Phillips, T. Kilpatrick, H. Butzkueven, N. Tubridy, E. McDonald, and M. Galea. 2006. “Gait and balance impairment in early multiple sclerosis in the absence of clinical disability,” _Multiple Sclerosis Journal_ , 12(5):620–628.
  * 5. DELUCA, P. A. 1991. “Gait analysis in the treatment of the ambulatory child with cerebral palsy,” _Clinical Orthopaedics and Related Research®_ , 264:65–75.
  * 6. Agostini, V., G. Balestra, and M. Knaflitz. 2013. “Segmentation and classification of gait cycles,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , 22(5):946–952.
  * 7. Mickelborough, J., M. Van Der Linden, J. Richards, and A. Ennos. 2000. “Validity and reliability of a kinematic protocol for determining foot contact events,” _Gait & Posture_, 11(1):32–37.
  * 8. Potluri, S., S. Ravuri, C. Diedrich, and L. Schega. 2019. “Deep learning based gait abnormality detection using wearable sensor system,” in _2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_ , IEEE, pp. 3613–3619.
  * 9. Liu, T., Y. Inoue, K. Shibata, and K. Shiojima. 2011. “A mobile force plate and three-dimensional motion analysis system for three-dimensional gait assessment,” _IEEE Sensors Journal_ , 12(5):1461–1467.
  * 10. Purcell, B., J. Channells, D. James, and R. Barrett. 2006. “Use of accelerometers for detecting foot-ground contact time during running,” in _BioMEMS and nanotechnology II_ , SPIE, vol. 6036, pp. 292–299.
  * 11. Nguyen, T.-N., H.-H. Huynh, and J. Meunier. 2016. “Skeleton-based abnormal gait detection,” _Sensors_ , 16(11):1792.
  * 12. Chen, M., B. Huang, and Y. Xu. 2008. “Intelligent shoes for abnormal gait detection,” in _2008 IEEE international conference on robotics and automation_ , IEEE, pp. 2019–2024.
  * 13. Pan, S., M. Mirshekari, P. Zhang, and H. Y. Noh. 2016. “Occupant traffic estimation through structural vibration sensing,” in _Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems 2016_ , SPIE, vol. 9803, pp. 41–52.
  * 14. Buldt, A. K., J. J. Allan, K. B. Landorf, and H. B. Menz. 2018. “The relationship between foot posture and plantar pressure during walking in adults: A systematic review,” _Gait & posture_, 62:56–67.
  * 15. Wyers, L., R. Di Marco, S. Zambelli, S. Masiero, A. Hallemans, P. Van de Walle, K. Desloovere, and A. Del Felice. 2021. “Foot-floor contact pattern in children and adults with Dravet Syndrome,” _Gait & Posture_, 84:315–320.
  * 16. Chopra, A. K. 2007. _Dynamics of structures_ , Pearson Education India.
  * 17. Platt, J. et al. 1999. “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,” _Advances in large margin classifiers_ , 10(3):61–74.