# Bootstrapping Large-Scale Fine-Grained Contextual Advertising Classifier
from Wikipedia

Yiping Jin1,2, Vishakha Kadam1, Dittaya Wanvarie2
1Knorex, 140 Robinson Road, 14-16 Crown @ Robinson, Singapore
2Department of Mathematics & Computer Science, Chulalongkorn University,
Thailand
{jinyiping, vishakha.kadam<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Contextual advertising provides advertisers with the opportunity to target the
context which is most relevant to their ads. The large variety of potential
topics makes it very challenging to collect training documents to build a
supervised classification model or compose expert-written rules in a rule-
based classification system. Besides, in fine-grained classification,
different categories often overlap or co-occur, making it harder to classify
accurately.

In this work, we propose wiki2cat, a method to tackle large-scaled fine-
grained text classification by tapping on the Wikipedia category graph. The
categories in the IAB taxonomy are first mapped to category nodes in the
graph. Then the label is propagated across the graph to obtain a list of
labeled Wikipedia documents to induce text classifiers. The method is ideal
for large-scale classification problems since it does not require any
manually-labeled document or hand-curated rules or keywords. The proposed
method is benchmarked with various learning-based and keyword-based baselines
and yields competitive performance on publicly available datasets and a new
dataset containing more than 300 fine-grained categories.

##  1 Introduction

Despite the fast advancement of text classification technologies, most text
classification models are trained and applied to a relatively small number of
categories. Popular benchmark datasets contain from two up to tens of
categories, such as SST2 dataset for sentiment classification (2 categories)
Socher et al. (2013), AG news dataset (4 categories) Zhang et al. (2015) and
20 Newsgroups dataset Lang (1995) for topic classification.

In the meantime, industrial applications often involve fine-grained
classification with a large number of categories. For example, Walmart built a
hybrid classifier to categorize products into 5000+ product categories Sun et
al. (2014), and Yahoo built a contextual advertising classifier with a
taxonomy of around 6000 categories Broder et al. (2007). Unfortunately, both
systems require a huge human effort in composing and maintaining rules and
keywords. Readers can neither reproduce their system nor is the system or data
available for comparison.

In this work, we focus on the application of contextual advertising Jin et al.
(2017), which allows advertisers to target the context most relevant to their
ads. However, we cannot fully utilize its power unless we can target the page
content using fine-grained categories, e.g., “coupé”’ vs. “hatchback” instead
of “automotive” vs. “sport”. This motivates a classification taxonomy with
both high coverage and high granularity. The commonly used contextual taxonomy
introduced by Interactive Advertising Bureau (IAB) contains 23 coarse-grained
categories and 355 fine-grained categories
111https://www.iab.com/guidelines/taxonomy/. Figure 1 shows a snippet of the
taxonomy.

Figure 1: Snippet of IAB Content Categorization Taxonomy.

Large online encyclopedias, such as Wikipedia, contain an updated account of
almost all topics. Therefore, we ask an essential question: can we bootstrap a
text classifier with hundreds of categories from Wikipedia without any manual
labeling?

We tap on and extend previous work on Wikipedia content analysis Kittur et al.
(2009) to automatically label Wikipedia articles related to each category in
our taxonomy by Wikipedia category graph traversal. We then train
classification models with the labeled Wikipedia articles. We compare our
method with various learning-based and keyword-based baselines and obtain a
competitive performance.

##  2 Related Work

###  2.1 Text Classification Using Knowledge Base

Large knowledge bases like Wikipedia or DMOZ content directory cover a wide
range of topics. They also have a category hierarchy in either tree or graph
structure, which provides a useful resource for building text classification
models. Text classification using knowledge bases can be broadly categorized
into two main approaches: vector space model and semantic model.

Vector space model aims to learn a category vector by aggregating the
descendant pages and perform nearest neighbor search during classification. A
pruning is usually performed first based on the depth from the root node or
the number of child pages to reduce the number of categories. Subsequently,
each document forms a document vector, which is aggregated to form the
category vector. Lee et al. (2013) used tf-idf representation of the document,
while Kim et al. (2018) combined word embeddings and tf-idf representations to
obtain a better performance.

In semantic models, the input document is mapped explicitly to concepts in the
knowledge base. The concepts are used either in conjunction with bag-of-words
representation Gabrilovich and Markovitch (2006) or stand-alone Chang et al.
(2008) to assign categories to the document.

Gabrilovich and Markovitch (2006) used a feature generator to predict relevant
Wikipedia concepts (articles) related to the input document. These concepts
are orthogonal to the labels in specific text classification tasks and are
used to enrich the representation of the input document. Experiments on
multiple datasets demonstrated that the additional concepts helped improve the
performance. Similarly, Zhang et al. (2013) enriched the document
representation with both concepts and categories from Wikipedia.

Chang et al. (2008) proposed Dataless classification that maps both input
documents and category names into Wikipedia concepts using Explicit Semantic
Analysis Gabrilovich et al. (2007). The idea is similar to Gabrilovich and
Markovitch Gabrilovich and Markovitch (2006), except (1) the input is mapped
to a real-valued concept vector instead of a discrete list of related
categories, and (2) the category name is mapped into the same semantic space,
which removes the need for labeled documents.

Most recently, Chu et al. (2020) improved text classification by utilizing
naturally labeled documents such as Wikipedia, Stack Exchange subareas, and
Reddit subreddits. Instead of training a traditional supervised classifier,
they concatenate the category name and the document and train a binary
classifier, determining whether the document is related to the category. They
benchmarked their proposed method extensively on 11 datasets covering topical
and sentiment classification.

Our work is most similar to Lee et al. (2013). However, they only evaluated on
random-split Wikipedia documents, while we apply the model to a real-world
large-scale text classification problem. We also employed a graph traversal
algorithm to label the documents instead of labeling all descendant documents.

###  2.2 Wikipedia Content Analysis

Some previous work tried to understand the distribution of topics in Wikipedia
for data analysis and visualization Mesgari et al. (2015). Kittur et al.
(2009) calculated the distance between each page to top-level category nodes.
They then assigned the category with the shortest distance to the page. With
this approach, they provided the first quantitative analysis of the
distribution of topics in Wikipedia.

Farina et al. (2011) extended the method by allowing traversing upward in the
category graph and assigning categories proportional to the distance instead
of assigning the category with the shortest-path only. More recently,
Bekkerman and Donin (2017) visualized Wikipedia by building a two-level
coarse-grained/fine-grained graph representation. The edges between categories
capture the co-occurrence of categories on the same page. They further pruned
edges between categories that rarely appear together. The resulting graph
contains 441 largest categories and 4815 edges connecting them.

##  3 Method

Figure 2: Overview of wiki2cat, a framework to bootstrap large-scale text
classifiers from Wikipedia. We first map user-defined categories to category
nodes in the Wikipedia category graph. Then, we traverse the category graph to
label documents automatically. Lastly, we use the labeled documents to train a
supervised classifier.

We propose wiki2cat, a simple framework using Wikipedia to bootstrap text
categorizers. We first map the target taxonomy to corresponding Wikipedia
categories (briefed in Section 3.1). We then traverse the Wikipedia category
graph to automatically label Wikipedia articles (Section 3.2). Finally, we
induce a classifier from the labeled Wikipedia articles (Section 3.3). Figure
2 overviews the end-to-end process of building classifiers under the wiki2cat
framework.

###  3.1 Mapping the Target Taxonomy to Wikipedia Categories

Wikipedia contains 2 million categories, which is 4 orders of magnitude larger
than IAB taxonomy. We index all Wikipedia category names in Apache Lucene
222https://lucene.apache.org and use the IAB category names to query the
closest matches. We perform the following: 1) lemmatize the category names in
both taxonomies, 2) index both Wikipedia category names and their alternative
names from redirect links (e.g., “A.D.D.” and “Attention deficit disorder”),
3) split conjunction category names and query separately (e.g., “Arts &
Entertainment” $\rightarrow$ “Arts”, “Entertainment”), and 4) capture small
spelling variations with string similarity 333We use Jaro-Winkler string
similarity with a threshold of 0.9 to automatically map IAB categories to
Wikipedia categories..

Out of all 23 coarse-grained and 355 fine-grained categories in IAB taxonomy,
311 categories (82%) can be mapped trivially. Their category names either
match exactly or contain only small variations. E.g., the IAB category
“Pagan/Wiccan” is matched to three Wikipedia categories “Paganism”, “Pagans”,
and “Wiccans”. One author of this paper took roughly 2 hours to curate the
remaining 67 categories manually and provided the mapping to Wikipedia
categories. Out of the 67 categories, 23 are categories that cannot be matched
automatically because the category names look very different, e.g., “Road-Side
Assistance” and “Emergency road services”. The rest are categories where the
system can find a match, but the string similarity is below the threshold
(e.g., correct: “Chronic Pain” and “Chronic Pain Syndromes”; incorrect:
“College Administration” and “Court Administration”). We use the curated
mapping in subsequent sections.

###  3.2 Labeling Wikipedia Articles by Category Graph Traversal

With the mapping between IAB and Wikipedia categories, we can anchor each IAB
category as nodes in the Wikipedia category graph 444We construct the category
graph using the “subcat” (subcategory) relation in the Wikipedia dump. The
graph contains both category nodes and page nodes. Pages all appear as leaf
nodes while category nodes can be either internal or leaf nodes., referred to
as the root category nodes. Our task then becomes to obtain a set of labeled
Wikipedia articles by performing graph traversal from the root category nodes.
From each root category node, the category graph can be traversed using the
breadth-first search algorithm to obtain a list of all descendant categories
and pages.

One may argue that we can take all descendant pages of a Wikipedia category to
form the labeled set. However, in Wikipedia page $A$ belongs to category $B$
does not imply a hypernym relation. In fact, some pages have a long list of
categories, most of which are at their best remotely related to the main
content of the page. E.g., the page “Truck Stop Women”
555https://en.wikipedia.org/wiki/Truck_Stop_Women is a descendant page of the
category “Trucks”. However, it is a 1974 film, and the main content of the
page is about the plot and the cast.

Figure 3: Intuition of the pruning for the category “Trucks”. The page “Ford
F-Max” belongs to four categories. Three of which can be traversed from
“Trucks” and one cannot (marked in red and italic).

We label Wikipedia pages using a competition-based algorithm following Kittur
et al. (2009) and Farina et al. (2011). We treat each category node from which
a page can be traversed as a candidate category and evaluate across all
candidate categories to determine the final category(s) for the page.

Firstly, all pages are pruned based on the percentage of their parent
categories that can be traversed from the root category. Figure 3 shows two
Wikipedia pages with a snippet of their ancestor categories. Both pages have a
shortest distance of 2 to the category “Trucks”. However, the page “Ford
F-Max” is likely more related to “Trucks” than the page “Camping and
Caravanning Club” because most of its parent categories can be traversed from
“Trucks”. We empirically set the threshold that we will prune a page with
respect to a root category if less than 30% of its parent categories can be
traversed from the root category.

While the categories in IAB taxonomy occur in parallel, the corresponding
categories in Wikipedia may occur in a hierarchy. For example, the category
“SUVs” and “Trucks” are in parallel in IAB taxonomy but “SUVs” is a descendant
category of “Trucks” in Wikipedia (Trucks ›Trucks by type ›Light trucks ›Sport
utility vehicles). While traversing from the root category node, we prune all
the branches corresponding to a competing category.

Pruning alone will not altogether remove the irrelevant content, because the
degree of semantic relatedness is not considered. We measure the semantic
relatedness between a page and a category based on two factors, namely the
shortest path distance and the number of unique paths between them. Previous
work depends only on the shortest path distance Kittur et al. (2009); Farina
et al. (2011). We observe that if a page is densely connected to a category
via many unique paths, it is often an indication of a strong association. We
calculate the weight $w$ of a page with respect to a category as follows:

$w=\displaystyle\sum_{i=0}^{k}\frac{1}{2^{d_{i}}}$ (1)

where $k$ is the number of unique paths between the page and the category
node, and $d_{i}$ is the distance between the two in the $i$th path. To
calculate the final list of categories, the weights for all competing
categories are normalized to 1 by summing over each candidate category $j$ and
the categories which have a weight higher than 0.3 are returned as the final
assigned categories.

$w_{j}=\displaystyle\sum_{i=0}^{k_{j}}\frac{1}{2^{d_{ij}}}/(\displaystyle\sum_{j}\displaystyle\sum_{i=0}^{k_{j}}\frac{1}{2^{d_{ij}}})$
(2)

The labeling process labeled in total 1.16 million Wikipedia articles. The
blue scattered plot in Figure 4 plots the number of labeled training articles
per fine-grained category in log-10 scale. We can see that the majority of the
categories have between 100 to 10k articles.

Figure 4: Blue: # of automatically labeled Wikipedia articles per fine-grained
category in log-10 scale. (mean=2.95, std=0.86). Orange: # of articles per
fine-grained category in the full test set in log-10 scale (mean=1.94,
std=0.78).

###  3.3 Training Contextual Classifiers

The output of the algorithm described in Section 3.2 is a set of labeled
Wikipedia pages. In theory, we can apply any supervised learning method to
induce classifiers from the labeled dataset. The focus of this work is not to
introduce a novel model architecture, but to demonstrate the effectiveness of
the framework to bootstrap classifiers without manual labeling. We experiment
with three simple and representative classification models. The first model is
a linear SVM with tf-idf features, which is a competitive baseline for many
NLP tasks Wang and Manning (2012). The second model is a centroid classifier,
which is commonly used in large-scale text classification Lee et al. (2013).
It averages the tf-idf vectors of all documents belonging to each category and
classifies by searching for the nearest category vector. The third model uses
BERT Devlin et al. (2019) to generate the semantic representation from the
text and uses a single-layer feed-forward classification head on top. We
freeze the pre-trained BERT model and train only the classification head for
efficient training.

The number of labeled Wikipedia documents for each category is highly
imbalanced. Minority categories contain only a handful of pages, while some
categories have hundreds of thousands of pages. We perform random over- and
downsampling to keep 1k documents for each fine-grained category and 20k
documents for each coarse-grained category to form the training set. 666We use
the original dataset without sampling for the centroid classifier since it is
not affected by label imbalance.

##  4 Experiments

###  4.1 Evaluation Datasets

We evaluated our method using three contextual classification datasets. The
first two are coarse-grained evaluation datasets published by Jin et al.
(2020) covering all IAB tier-1 categories except for “News” (totaling 22
categories). The datasets are collected using different methods (news-crawl-v2
dataset (nc-v2) by mapping from news categories; browsing dataset by manual
labelling) and contain 2,127 and 1,501 documents separately
777https://github.com/YipingNUS/nle-supplementary-dataset.

We compiled another dataset for fine-grained classification comprising of
documents labeled with one of the IAB tier-2 categories. The full dataset
consists of 134k documents and took an effort of multiple person-year to
collect. The sources of the dataset are news websites, URLs occurring in the
online advertising traffic and URLs crawled with keywords using Google Custom
Search 888https://developers.google.com/custom-search/.

The number of documents per category can be overviewed in Figure 4 (the orange
scatter plot). 23 out of 355 IAB tier-2 categories are not included in the
dataset because they are too rare and are not present in our data source. So
there are in total 332 fine-grained categories in the datasets. Due to company
policy, we can publish only a random sample of the dataset with ten documents
per category 999https://github.com/YipingNUS/contextual-eval-dataset. We
report the performance on both datasets for future work to reproduce our
result. To our best knowledge, this dataset will be the only publicly
available dataset for fine-grained contextual classification.

We focus on classifying among fine-grained categories under the same parent
category. Figure 5 shows the number of fine-grained categories under each
coarse category. While the median number of categories is 10, the
classification is challenging because categories are similar to each other.

Figure 5: Number of fine-grained categories per coarse-grained category in our
fine-grained contextual classification evaluation dataset.

###  4.2 Experimental Settings

Throughout this paper, we use the Wikipedia dump downloaded on 10 December
2019. After removing hidden categories and list pages, the final category
graph contains 14.9 million articles, 1.9 million categories and 37.9 million
links. The graph is stored in Neo4J database 101010https://neo4j.com and
occupies 4.7GB disk space (not including the page content).

We use the SGD classifier implementation in scikit-learn 111111https://scikit-
learn.org with default hyperparameters for linear SVM. Words are weighted
using tf-idf with a minimum term frequency cutoff of 3. We implement the
centroid classifier using TfidfVectorizer in scikit-learn and use numpy to
implement the nearest neighbor classification.

For BERT, we use DistilBERT implementation by HuggingFace
121212https://huggingface.co/transformers/model_doc/distilbert.html, a model
which is both smaller and faster than the original BERT-base model. We use a
single hidden layer with 256 units for the feed-forward classification head.
The model is implemented in PyTorch and optimized with Adam optimizer with a
learning rate of 0.01.

We compare wiki2cat with the following baselines:

  * •

Keyword voting (kw voting): predicts the category whose name occurs most
frequently in the input document. If none of the category names is present,
the model predicts a random label.

  * •

Dataless Chang et al. (2008): maps the input document and the category name
into the same semantic space representing Wikipedia concepts using Explicit
Semantic Analysis (ESA) Gabrilovich et al. (2007).

  * •

Doc2vec Le and Mikolov (2014): similar to the Dataless model. Instead of using
ESA, it uses doc2vec to generate the document and category vector.

  * •

STM Li et al. (2018): seed-guided topic model. The state-of-the-art model on
coarse-grained contextual classification. Underlying, STM calculates each
word’s co-occurrence and uses it to “expand” the knowledge beyond the given
seed words. For coarse-grained classification, STM used hand-curated seed
words while STM,Slabel used category names as seed words. Both were trained by
Jin et al. (2020) on a private in-domain dataset. We also trained STM using
our Wikipedia dataset, referred to as STM,Dwiki. For fine-grained
classification, we report only the result of STM,Slabel since no previously
published seed words are available.

Keyword voting and Dataless do not require any training document. Both Doc2vec
and STM require unlabeled training corpus. We copy the coarse-grained
classification result for Doc2vec, STM, and STM,Slabel from Jin et al. (2020).
For fine-grained classification, we train Doc2vec and STM,Slabel using the
same set of Wikipedia documents as in wiki2cat.

###  4.3 Result of Coarse-Grained Contextual Classification

We present the performance of various models on nc-v2 and browsing dataset in
Table 1.

Model | nc-v2 | browsing
---|---|---
acc |  ma$F_{1}$ |  acc+ |  ma$F_{1}$
kw voting |  .196 |  .180 |  .251 |  .189
Dataless |  .412 |  .377 |  .536 |  .392
Doc2vec |  .480 |  .461 |  .557 |  .424
STM |  .623 |  .607 |  .794 |  .625
STM,Slabel |  .332 |  .259 |  .405 |  .340
STM,Dwiki |  .556 |  .533 |  .780 |  .595
w2csvm |  .563 |  .539 |  .659 |  .523
w2ccentroid |  .471 |  .426 |  .675 |  .523
w2cbert |  .440 |  .403 |  .621 |  .482
Table 1: Performance of various models on IAB coarse-grained classification
datasets. The best performance is highlighted in bold.

We can observe that wiki2cat using SVM as the learning algorithm outperformed
Dataless and Doc2vec baseline. However, it did not perform as well as STM. The
STM model was trained using a list of around 30 carefully chosen keywords for
each category. It also used in-domain unlabeled documents during training,
which we do not use. Jin et al. (2020) demonstrated that the choice of seed
keywords has a significant impact on the model’s accuracy. STM,Slabel is the
result of STM using only unigrams in the category name as seed keywords.
Despite using the same learning algorithm as STM, its performance was much
worse than using hand-picked seed words.

To investigate the contribution of the in-domain unlabeled document to STM’s
superior performance, we trained an STM model with the manually-curated
keywords in Jin et al. (2020) and the Wikipedia dataset we used to train
wiki2cat (denoted as STM,Dwiki). There is a noticeable decrease in performance
in STM,Dwiki without in-domain unlabeled documents. It underperformed w2csvm
on nc-v2 dataset and outperformed it on browsing dataset.

w2ccentroid performed slightly better than w2csvm on the browsing dataset but
worse on the nc-v2 dataset. Surprisingly, BERT did not perform as well as the
other two much simpler models. We conjecture there are two possible causes.
Firstly, BERT has a limitation of sequence length (maximum 512 words). The
average sequence length of news-crawl-v2 and browsing datasets are 1,470 and
350 words. Incidentally, there was a more substantial performance gap between
BERT and SVM on the news-crawl-v2 dataset. Secondly, our training corpus
consists of only Wikipedia articles, while the model was applied to another
domain. Therefore, the contextual information that BERT captured may be
irrelevant or even counterproductive. We leave a more in-depth analysis to
future work and adhere to the SVM and Centroid model hereafter.

###  4.4 Impact of Graph Labeling Algorithms

Model | nc-v2 | browsing
---|---|---
acc |  ma$F_{1}$ |  acc+ |  ma$F_{1}$
w2c |  .563 |  .539 |  .659 |  .523
w2cchild |  .325 |  .289 |  .340 |  .322
w2cdescendant |  .539 |  .503 |  .607 |  .481
w2cmin-dist |  .533 |  .498 |  .612 |  .489
w2cno-pruning |  .488 |  .466 |  .608 |  .491
Table 2: Performance of the SVM model trained with datasets labeled using
different labeling algorithms.

We now turn our attention to the impact of different graph labeling algorithms
on the final classification accuracy. We compare our graph labeling method
introduced in Section 3.2 with three methods mentioned in previous work,
namely labeling only immediate child pages (child), labeling all descendant
pages (descendant), assigning the label with shortest distance (min-dist) as
well as another baseline removing the pruning step from our method (no-
pruning). We use an SVM model with the same hyperparameters as w2csvm. Their
performance is shown in Table 2.

Using only the immediate child pages led to poor performance. Firstly, it
limited the number of training documents. Some categories have only a dozen of
immediate child pages. Secondly, the authors of Wikipedia often prefer to
assign pages to specific categories instead of general categories. They assign
a page to a general category only when it is ambiguous. Despite previous work
in Wikipedia content analysis advocated using shortest distance to assign the
topic to articles Kittur et al. (2009); Farina et al. (2011), we did not
observe a substantial improvement using shortest distance over using all
descendant pages. Our graph labeling method outperformed all baselines,
including its modified version without pruning.

###  4.5 Result of Fine-Grained Contextual Classification

Table 3 presents the result on fine-grained classification. We notice a
performance difference on the full and sample dataset. However, the relative
performance of various models on the two datasets remains consistent.

Model | Full dataset | Sample dataset
---|---|---
acc |  ma$F_{1}$ |  acc |  ma$F_{1}$
kw voting |  .108 |  .018 |  .075 |  .025
Dataless |  .428 |  .376 |  .477 |  .462
Doc2vec |  .246 |  .152 |  .253 |  .211
STM,Slabel |  .493 |  .370 |  .533 |  .464
w2csvm |  .542∗ |  .464∗ |  .646∗ |  .627∗
w2ccentroid |  .548∗ |  .451∗ |  .595∗ |  .566∗
Table 3: Performance of various models on IAB fine-grained classification
datasets. * indicates a statistically significant improvement from baselines
with p-value<0.05 using single-sided sample T-test.

A first observation is that the keyword voting baseline performed very poorly,
having 7.5-10.8% accuracy. It shows that the category name itself is not
enough to capture the semantics. E.g., the category “Travel $>$ South America”
does not match a document about traveling in Rio de Janeiro or Buenos Aires
but will falsely match content about “South Korea” or “United States of
America”.

Dataless and STM outperformed the keyword voting baseline by a large margin.
However, wiki2cat is clearly the winner, outperforming these baselines by
5-10%. It demonstrated that the automatically labeled documents are helpful
for the more challenging fine-grained classification task where categories are
more semantically similar and harder to be specified with a handful of
keywords.

##  5 Conclusions and Future Work

We introduced wiki2cat, a simple framework to bootstrap large-scale fine-
grained text classifiers from Wikipedia without having to label any document
manually. The method was benchmarked on both coarse-grained and fine-grained
contextual advertising datasets and achieved competitive performance against
various baselines. It performed especially well on fine-grained
classification, which both is more challenging and requires more manual
labeling in a fully-supervised setting. As an ongoing effort, we are exploring
using unlabeled in-domain documents for domain adaptation to achieve better
accuracy.

## Acknowledgement

YJ was supported by the scholarship from ‘The 100th Anniversary Chulalongkorn
University Fund for Doctoral Scholarship’. We thank anonymous reviewers for
their valuable feedback.

## References

  * Bekkerman and Donin (2017) Ron Bekkerman and Olga Donin. 2017.  Visualizing wikipedia for interactive exploration.  In _Proceedings of KDD 2017 Workshop on Interactive Data Exploration and Analytics (IDEA17)_ , Halifax, Nova Scotia, Canada.
  * Broder et al. (2007) Andrei Broder, Marcus Fontoura, Vanja Josifovski, and Lance Riedel. 2007.  A semantic approach to contextual advertising.  In _Proceedings of the 30th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 559–566.
  * Chang et al. (2008) Ming-Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. 2008.  Importance of semantic representation: Dataless classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 2, pages 830–835.
  * Chu et al. (2020) Zewei Chu, Karl Stratos, and Kevin Gimpel. 2020.  Natcat: Weakly supervised text classification with naturally annotated datasets.  _arXiv preprint arXiv:2009.14335_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4171–4186, Minneapolis, Minnesota.
  * Farina et al. (2011) Jacopo Farina, Riccardo Tasso, and David Laniado. 2011.  Automatically assigning wikipedia articles to macrocategories.  In _Proceedings of Hypertext_.
  * Gabrilovich and Markovitch (2006) Evgeniy Gabrilovich and Shaul Markovitch. 2006.  Overcoming the brittleness bottleneck using wikipedia: Enhancing text categorization with encyclopedic knowledge.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 6, pages 1301–1306.
  * Gabrilovich et al. (2007) Evgeniy Gabrilovich, Shaul Markovitch, et al. 2007.  Computing semantic relatedness using wikipedia-based explicit semantic analysis.  In _Proceedings of the Twentieth International Joint Conference on Artificial Intelligence_ , volume 7, pages 1606–1611.
  * Jin et al. (2017) Yiping Jin, Dittaya Wanvarie, and Phu Le. 2017.  Combining lightly-supervised text classification models for accurate contextual advertising.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 545–554.
  * Jin et al. (2020) Yiping Jin, Dittaya Wanvarie, and Phu T. V. Le. 2020.  Learning from noisy out-of-domain corpus using dataless classification.  _Natural Language Engineering_.
  * Kim et al. (2018) Kang-Min Kim, Aliyeva Dinara, Byung-Ju Choi, and SangKeun Lee. 2018.  Incorporating word embeddings into open directory project based large-scale classification.  In _Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining_ , pages 376–388. Springer.
  * Kittur et al. (2009) Aniket Kittur, Ed H Chi, and Bongwon Suh. 2009.  What’s in wikipedia? mapping topics and conflict using socially annotated category structure.  In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , pages 1509–1512.
  * Lang (1995) Ken Lang. 1995.  Newsweeder: Learning to filter netnews.  In _Proceedings of the Twelfth International Conference on Machine Learning_ , pages 331–339. Elsevier.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents.  In _Proceedings of the International Conference on Machine Learning_ , pages 1188–1196.
  * Lee et al. (2013) Jung-Hyun Lee, Jongwoo Ha, Jin-Yong Jung, and Sangkeun Lee. 2013.  Semantic contextual advertising based on the open directory project.  _ACM Transactions on the Web (TWEB)_ , 7(4):1–22.
  * Li et al. (2018) Chenliang Li, Shiqian Chen, Jian Xing, Aixin Sun, and Zongyang Ma. 2018.  Seed-guided topic model for document filtering and classification.  _ACM Transactions on Information Systems_ , 37(1):1–37.
  * Mesgari et al. (2015) Mostafa Mesgari, Chitu Okoli, Mohamad Mehdi, Finn Årup Nielsen, and Arto Lanamäki. 2015.  The sum of all human knowledge: A systematic review of scholarly research on the content of wikipedia.  _Journal of the Association for Information Science and Technology_ , 66(2):219–245.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642.
  * Sun et al. (2014) Chong Sun, Narasimhan Rampalli, Frank Yang, and AnHai Doan. 2014.  Chimera: Large-scale classification using machine learning, rules, and crowdsourcing.  _Proceedings of the VLDB Endowment_ , 7(13):1529–1540.
  * Wang and Manning (2012) Sida Wang and Christopher D Manning. 2012.  Baselines and bigrams: Simple, good sentiment and topic classification.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2_ , pages 90–94. Association for Computational Linguistics.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Proceedings of Advances in Neural Information Processing Systems_ , pages 649–657.
  * Zhang et al. (2013) Zhilin Zhang, Huaizhong Lin, Pengfei Li, Huazhong Wang, and Dongming Lu. 2013.  Improving semi-supervised text classification by using wikipedia knowledge.  In _Proceedings of the International Conference on Web-Age Information Management_ , pages 25–36. Springer.


# Unsupervised Domain Adaptation for Device-free Gesture Recognition

Bin-Bin Zhang, Dongheng Zhang, Yadong Li, Yang Hu, and Yan Chen B. Zhang, D.
Zhang, Y. Li, Y. Chen are with the School of Cyber Science and Technology,
University of Science and Technology of China, Hefei 230026, China.
(E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>eecyan@ustc.edu.cn). Y. Hu is with the School of
Information Science and Technology, University of Science and Technology of
China, Hefei 230026, China.
(E-mail: eeyhu@ustc.edu.cn).

###### Abstract

Device-free human gesture recognition with Radio Frequency (RF) signals has
attained acclaim due to the omnipresence, privacy protection, and broad
coverage nature of RF signals. However, neural network models trained for
recognition with data collected from a specific domain suffer from significant
performance degradation when applied to a new domain. To tackle this
challenge, we propose an unsupervised domain adaptation framework for RF-based
gesture recognition by making effective use of the unlabeled target domain
data. Specifically, we apply pseudo-labeling and consistency regularization
with elaborated design on target domain data to produce pseudo-label and align
instance’s feature of the target domain. Then, we design two data augmentation
methods by randomly erasing the input data to enhance the robustness of the
model. Furthermore, we apply a confidence control constraint to tackle the
overconfidence problem. We conduct extensive experiments on a public Wi-Fi
dataset and a public millimeter wave (mmWave) radar dataset. The experimental
results demonstrate the superior effectiveness of the proposed framework.

###### Index Terms:

Gesture Recognition, Cross Domain, Unsupervised Domain Adaptation, Device
Free.

##  1 Introduction

Human gesture recognition plays an important role in human-computer
interaction, enabling applications including smart home, smart driving and
virtual reality, etc. Traditional approaches adopt cameras [1, 2], wearable
devices and phones[3, 4] as the sensing module, which however suffer from
inherent drawbacks including privacy leakage, inconvenience, and limited
sensing range. To address this challenge, significant efforts [5, 6, 7, 8, 9,
10, 11, 12, 13, 14, 15, 16, 17, 18, 19] have recently been made to explore
Radio Frequency (RF)-based human gesture recognition techniques without
letting the monitored subject carry a dedicated device. These techniques
recognize gestures by perceiving RF signals variations caused by gestures in
space. However, the signals arriving at the receiver not only contain
information of the performed gestures, but also carry substantial information
of the environment, subject, location, and orientation. We usually use the
term ”domain” to summarize these gesture irrelevant factors. As a result, the
classifiers trained with data in one domain usually undergo drastically
performance drop in another domain.

To tackle this problem, researchers have explored how to improve cross domain
RF-based device-free gesture recognition performance. These RF-based methods
can be roughly divided into two categories: Wi-Fi-based and mmWave-based
methods. Among the Wi-Fi-based methods, WiAG [20] and Widar3.0 [21] construct
mathematical models to derive features that are independent of users’ location
and orientation. However, WiAG doesn’t consider cases across environments and
subjects, Widar3.0 still can not remove all the influence of domain.
CrossSense [22] proposes an offline trained ANN-based roaming model mapping
features from one environment to another, which however is not applicable to
the case with multiple source domains. Then EI [23] and CACDGR [24] are
proposed by using an adversarial training to improve the cross domain
performance of recognition model. However, EI [23] doesn’t consider location
and orientation’s influence, CACDGR [24] needs an amount of extra training
efforts. Though the above methods improve the performance of the recognition
models, they need extra neural network modules and an amount of extra training
efforts. For the mmWave-based methods, Liu et al. [25] extract dynamic
variation of gestures from mmWave signals and design a lightweight CNN to
recognize gestures. MmASL [26] designs a multi-task deep neural network to
achieve American sign language recognition. RadarNet [27] designs an efficient
neural network and collects a large scale dataset to train a robust model.
However, for cross domain situation, these mmWave-based recognition models can
not perform as well as the in-domain situation.

Due to the fact that the signal propagation in practical environment is very
complicated, the data distributions collected from different domains may be
very different. Thus, the model trained with source domain data cannot be
directly generated to unlabeled target domain. To tackle this challenge, a
common solution is to collect labeled target domain data and transfer the
trained model into target domain. However, such data collection is costly and
not practical in deployment. Compared with labeled data, unlabeled data is
more accessible which also contains the information of the target domain.

In this paper, we propose an unsupervised domain adaptation framework for FR-
based device-free human gesture recognition to increase the recognition
model’s ability of cross-domain. Inspired by the semi-supervised learning
(SSL) techniques [28, 29, 30, 31, 32, 33], we propose the pseudo-labeling
technique and consistency regularization strategy with elaborated design on
target domain data to make effective use of the unlabeled target domain data.
Among the two techniques, pseudo-labeling can produce labels for unlabeled
target data which can further be utilized to train recognition model. With the
aid of pseudo label, the generalization ability of the model could be
enhanced. Consistency regularization would align instance’s feature of the
target domain data by decreasing the gap between original feature and
augmented version of original feature, which can make the model pay more
attention to the intrinsic feature associated with gesture category. More
importantly, we combine these two techniques with elaborated design rather
than employing them independently, which make the training process simple and
efficient. Specifically, the pseudo labels produced based on unlabeled target
domain data are adopted as targets of augmented version of the same data. In
addition, we propose two data augmentation methods for RF signals by randomly
erasing the feature of input data, which can enhance the effectiveness of the
consistency regularization. And we adopt a confidence control constraint to
tackle the overconfidence problem, which would lead to the model predicting
wrong high confident classification.

Our main contributions are summarized as follows:

  * •

We propose an unsupervised domain adaptation framework for RF-based device-
free gesture recognition to improve cross domain performance of the neural
network.

  * •

We propose the pseudo-labeling and consistency regularization techniques with
elaborated design to make effective use of the unlabeled RF data in target
domain. In addition, we design a confidence control constraint to tackle the
overconfidence problem. According to the characteristics of RF signals, we
design two data augmentation methods to enhance the effectiveness of the
consistency regularization.

  * •

We conduct comprehensive experiments on a public Wi-Fi dataset and a public
mmWave radar dataset. Our framework achieves better performance than state-of-
the-art models, which demonstrates the superiority and generalization ability
of our framework.

The rest of this paper is organized as follows. Section 2 introduces the
related work. Section 3 formulates the problem. Section 4 introduces the
system overview, recognition model and the proposed unsupervised domain
adaptation framework. Section 5 presents the experiments on the two RF-based
signals, the performance of the cross domain evaluations and the ablation
study. Section 6 concludes this paper.

##  2 Related Work

Inspired by the semi-supervised learning (SSL) techniques, we propose an
unsupervised domain adaptation framework for device-free gesture recognition.
The proposed framework is related to these three techniques: _unsupervised
domain adaptation_ , _semi-supervised learning_ and _cross domain gesture
recognition_.

###  2.1 Unsupervised Domain Adaptation

The UDA models assume that the source domain has sufficient labels, while the
unlabeled target domain participates the model training in an unsupervised
manner [34]. Several approaches are adopted to learn domain-invariant features
through different metrics, e.g., Maximum Mean Discrepancy (MMD) [35, 36].
Contrastive Adaptation Network (CAN) [37] optimizes the metric for minimizing
the domain discrepancy, which explicitly models the intra-class domain
discrepancy and the inter-class domain discrepancy. Deep Adaption Network
(DAN) [35] adapts the high-layer features with the multi-kernel MMD criterion.
Adversarial discriminative domain adaptation (ADDA) [38] learns a
discriminative representation using the source labels, and then, a separate
encoding that maps the target data to the same space based on a domain-
adversarial loss is used. General to Adapt (GTA) [39] introduces a symbiotic
relation between the embedding network and the generative adversarial network.
Some other methods aim to learn a feature extractor to extract the domain-
invariant features, and maintain the representation consistency through
minimizing the reconstruction error between domains. For example, Domain
Adversarial Neural Network (DANN) [40] proposes a domain-adversarial training
method to promote the emergence of features that are discriminative for the
main learning task on the source domain. Unsupervised Image-to-Image
Translation (UNIT) [41] makes a shared-latent space assumption and proposes an
unsupervised image-to-image translation framework based on Coupled GANs.
Different from the existing UDA methods, our framework adopts the pseudo-
labeling and consistency regularization on target domain to obtain pseudo
labels and align instance’s feature of the target domain data, which can make
effective use of the unlabeled target domain data.

Figure 1: Overview of our framework. After an unlabeled original feature being
fed into the model, we obtain the predictions. When the model assigns a
probability to any class which is above a threshold, we obtain pseudo-label.
Then we obtain the model’s predictions for augmentation version of the same
feature. We train the model by computing and optimizing the cross-entropy loss
between prediction on the augmented version and the pseudo-label.

###  2.2 Semi-Supervised Learning

Semi-supervised learning (SSL) [42, 43, 33, 29, 44, 31] leverages unlabeled
data to improve a model’s performance when limited labeled data is provided,
which alleviates the expensive labeling process efficiently. Some recently
proposed semi-supervised learning methods, such as MixMatch [43], FixMatch
[31], and ReMixMatch [42], are based on augmentation viewpoints. MixMatch [43]
uses low-entropy labels for data-augmented unlabeled instances and mixes
labeled and unlabeled data for semi-supervised learning. FixMatch [31]
generates pseudo labels using the model’s predictions on weakly augmented
unlabeled images. ReMixMatch [42] uses a weakly-augmented example to generate
an artificial label and enforce consistency against strongly-augmented
examples. Semi-supervised domain adaptation has more information about some
target labels compared with UDA, and some related works [45, 46, 47, 48, 49]
have been proposed leveraging semi-supervised signals. Specifically, in [46],
a minimax entropy approach is proposed that adversarially optimizes an
adaptive few-shot model. In [47], the learning of opposite structures is
unified whereby it consists of a generator and two classifiers trained with
opposite forms of losses for a unified object. Inspired by the semi-supervised
learning (SSL) techniques, we propose an UDA framework for RF-based gesture
recognition. Specially, we combine the pseudo-labeling and consistency
regularization with elaborated design and use the model’s predictions on
original unlabeled samples to obtain the pseudo labels.

###  2.3 Cross Domain Gesture Recognition

Wireless human sensing techniques [50, 51, 52, 53, 54, 55] have drawn
considerable attention and made great progress in recent years. Due to the
fact that human gesture recognition is a core part of human-computer
interaction, researchers explore how to improve the performance of the RF-
based gesture recognition. There are many prior works focusing on cross domain
gesture recognition to reduce data collection and labeling efforts to
generalize the recognition model, which can be roughly divided into two
categories: Wi-Fi-based and mmWave-based methods.

Among the Wi-Fi-based methods, WiAG [20] and Widar3.0 [21] construct
mathematical models to derive features that are independent of users’ location
and orientation. CrossSense [22] proposes an offline trained ANN-based roaming
model mapping features from one environment to another. EI [23] uses an
adversarial training scheme together with several constraints to generalize
the model to new environments and new subjects. WiHARAN [56] aligns the joint
distributions of features and activity classes of samples from multiple
domains by adversarial learning. CACDGR [24] presents a model using an
adversarial learning scheme together with feature disentanglement modules that
can remove the influence of the domain. Though the above methods improve the
performance of the recognition model, they need an amount of extra training
efforts. For the mmWave-based models, Liu et al. [25] extract dynamic
variation of gestures from mmWave signals and design a lightweight CNN to
recognize gestures. MmASL [26] designs a multi-task deep neural network to
achieve American sign language recognition. RadarNet [27] designs an efficient
neural network and collects a large scale dataset to train a robust model.
However, for cross domain situation, these mmWave-based recognition models can
not perform as well as the in-domain situation. In this paper, we introduce an
unsupervised domain adaptation framework for device-free gesture recognition
adopting the pseudo-labeling and consistency regularization with elaborated
design.

##  3 Problem Formulation

The input data contains both labeled and unlabeled ones. We refer to the data
with and without gesture labels as source and target domain, respectively. All
the data are attached with domain labels including four dimensions, i.e.,
environment, subject, location and orientation, indicating under what
configurations the data are collected. Suppose there are $N$ source domains
corresponding to $N$ different underlying distributions
$\left\\{p_{s_{i}}\left(x,y\right)\right\\}^{N}_{i=1}$. Source domains’ data
are sampled from $N$ distributions respectively with gesture labels, denoted
as
$\left(X_{s_{i}},Y_{s_{i}}\right)=\left\\{x^{j}_{s_{i}},y^{j}_{s_{i}}\right\\}^{n_{s_{i}}}_{j=1}$,
where $i\in\left\\{1,...,N\right\\}$. We regard the target domain as a whole
set without label, denoted as
$X_{t}=\left\\{x^{j}_{t}\right\\}^{n_{t}}_{j=1}$. All the source domains and
target domain have the same gesture categories. In this paper, we aim to
improve the performance of the recognition model trained using labeled source
domain data and unlabeled target domain data on target domain.

##  4 Methodology

In this section, we present our unsupervised domain adaptation framework for
RF-based gesture recognition. First, we provide an overview of our framework.
Next, we introduce the input of the recognition model. Then, we illustrate
each module of the unsupervised domain adaptation framework. Finally, we give
the objective function and training strategy.

###  4.1 Overview

An overview of the proposed framework is shown in Fig. 1. The framework aims
to train a gesture recognition model using the unlabeled target data and
labeled source domain data. Since the collected gesture data contain temporal
information and exist multiple links, the recognition model can be built up by
CNN and GRUs to extract discriminative features and classify the gesture
categories. To mitigate domain discrepancies between target and each source
domain, we adopt the two techniques: pseudo-labeling and consistency
regularization in our framework. Pseudo-labeling produces pseudo labels of
unlabeled target data which can be utilized to increase the generalization
ability of the recognition model. Consistency regularization strategy can
align instance’s feature of the target domain data, which can make the model
pay more attention to the intrinsic feature associated with gesture category.
We combine these two techniques with elaborated design rather than employing
them independently, which make the training process simple and efficient. In
addition, we propose two data augmentation methods for RF signals by randomly
erasing the feature of input data, which can enhance the effectiveness of the
consistency regularization. And we adopt a confidence control constraint to
tackle the overconfidence problem, which would lead to the model predicting
wrong high confident classification. The details of our framework will be
elaborated in the rest of this section.

###  4.2 Model Inputs

Our framework can be used for different RF signals. In this paper, we consider
two types of RF signals: Wi-Fi signal and mmWave radar signal. We choose the
Body-coordinate Vlocity Profile (BVP) feature and Dynamic Range Angle Image
(DRAI) sequences as the input of our model, respectively. BVP feature is
obtained by body-coordinate velocity profile generation technique in Widar 3.0
[21]. It describes power distribution over different velocities, at which body
parts involved in the gesture movements. BVP feature is a matrix with
dimension as $N\times N\times T$, where $N$ is the number of possible values
of velocity components decomposed, $T$ is the number of BVP snapshots. mmWave
radar signal can be processed to DRAI sequences in [57] to obtain doppler
power distribution over spatial positions while reducing environmental noise
and multi-path effects. DRAI sequences is a matrix with dimension as $M\times
M\times T$, where $M$ is the number of possible values of range and angle, and
$T$ denotes the time dimension.

###  4.3 Recognition Model

As shown in Fig. 2, our model consists of two network components, which are
feature extractor and gesture recognizer.

Feature Extractor: We employ one common extractor $G$ to extract gesture
features from all the source domains and target domain. We expect that the
extractor $G$ can extract the features which are related to gestures. In the
feature extractor, we use single-layer CNN with 2D kernels to extract
features. After applying a max-pooling layer to down-sample the features and a
dropout layer to reduce overfitting, we flatten the 2D features into a feature
vector. Then the vector is fed into two dense layers with Relu activation to
get a higher level representation Ut. To extract temporal relationships within
feature series, $\mathbf{U}=\left\\{\mathbf{U}_{t},t=1,2,...,T\right\\}$ are
fed into single-layer GRUs to generate the feature vector $\mathbf{Z}$. Given
the input series X, we can obtain feature vector $\mathbf{Z}$ as follows:

$\mathbf{Z}=G(\mathbf{X};\mathbf{\Theta}).$ (1)

Figure 2: Structure of gesture recognition module.

Gesture Recognizer: Based on feature vector $\mathbf{Z}$, a dropout layer and
a dense layer with Relu activation are adopted to learn the representation
$\mathbf{H}_{i}$ as follows:

$\mathbf{H}_{i}=\mathrm{Softplus}\left(\mathbf{W}_{z}\mathbf{Z}_{i}+\mathbf{b}_{z}\right),$
(2)

where $\mathbf{W}_{z}$ and $\mathbf{b}_{z}$ are the parameters to be learned,
the softplus function is an activation function. Then a softmax layer is used
to obtain the probability vector $\mathbf{\hat{y}}$. For the labeled source
domain data, cross-entropy function is used to calculate the loss between the
predictions and the ground truths as follows:

$L_{a}=-\frac{1}{\left|\mathbf{X}^{l}\right|}\sum_{i=1}^{\left|\mathbf{X}^{l}\right|}\sum_{c=1}^{C}\mathbf{y}^{l}_{ic}\log\left(\mathbf{\hat{y}}^{l}_{ic}\right),$
(3)

where $|\mathbf{X}^{l}|$ is the number of labeled data, $C$ is the number of
categories of human gestures. By optimizing Eq. (3), we can learn model
parameters and make predictions on target domain data.

###  4.4 Unsupervised Domain Adaptation Framework

To improve the cross domain generalization ability of the recognition model,
we propose an unsupervised domain adaptation framework for the RF-based
gesture recognition. As shown in Fig. 1, we propose the two techniques:
pseudo-labeling and consistency regularization with elaborated design to make
effective use of the unlabeled target domain data.

We first obtain $\mu B$ unlabeled data $\mathbf{X}^{u}$ from target domain
data pool $X_{t}$, where $B$ is a batch of labeled data and $\mu$ is a
hyperparameter that determines the relative sizes of $B$ and $\mu B$. Then we
augment $\mathbf{X}^{u}$ with two kinds of data augmentation strategies, which
is denoted by $\mathcal{A}\left(.\right)$. After that, we concatenate the
source domain labeled data $\mathbf{X}^{l}$, unlabeled target domain data
$\mathbf{X}^{u}$, and augmented version data $\mathbf{X}^{u}_{aug}$ as
follows:

$\displaystyle\mathbf{X}^{u}_{aug}$
$\displaystyle=\mathcal{A}\left(\mathbf{X}^{u}\right),$ (4)
$\displaystyle\mathbf{X}_{in}=\mathbf{X}^{l}$
$\displaystyle+\mathbf{X}^{u}+\mathbf{X}^{u}_{aug}.$

When we input the $\mathbf{X}_{in}$ into the model, the output can be obtained
by
$\mathbf{\hat{y}}=\left[\mathbf{\hat{y}}^{l},\mathbf{\hat{y}}^{u},\mathbf{\hat{y}}^{u}_{aug}\right]$,
where $\mathbf{\hat{y}}^{u}$ denotes the predictions of unlabeled data,
$\mathbf{\hat{y}}^{u}_{aug}$ denotes the predictions of augmented version.

####  4.4.1 Pseudo-Labeling

Making effective use of the target domain data can enhance the recognition
model’s ability of domain adaptation. Due to the fact that the target domain
data have no labels, we propose to utilize the pseudo-labeling technique to
make effective use of the unlabeled target domain data. Pseudo-labeling
technique can produce pseudo labels of unlabeled target data that are used to
train recognition model. Training the model with target data can increase the
ability of adaptation on target domain. During training, right pseudo labels
can speed up the convergence of the model and improve the performance of the
model, while the incorrect one will cause the negative impact on the training
of model [33]. To address this problem, we expect to obtain the ”hard” labels.
In specific, when the largest class probability of $\mathbf{\hat{y}}^{u}$
surpasses the threshold $\tau$, the pseudo label $\mathbf{y}^{u}$ can be
obtained as follows:

$\mathbf{y}^{u}_{ij}=\left\\{\begin{aligned} 1&,&\
\mathbf{\hat{y}}^{u}_{ij}\geqslant\tau,\\\ 0&,&\
\mathrm{otherwise}.\end{aligned}\ \ j=1,2,...,C\ \mathrm{and}\
i=1,2,...,|X^{u}|,\right.$ (5)

where $|X^{u}|$ is the number of the unlabeled data. With model training, the
confidence of output also increases. In order to ensure the effectiveness of
the pseudo labels, we use the dynamic threshold, which increases with the
number of iterations as follows:

$\tau_{d}=\tau_{0}+\eta\ epoch$ (6)

where $\tau_{d}$ is the dynamic threshold, $\tau_{0}$ is the initial value of
threshold, $\eta$ denotes the updated step size and $epoch$ is the number of
iterations.

After obtaining pseudo labels of unlabeled examples, we add the data with
pseudo labels to the training set. Different from the other traditional
methods, we adopt the pseudo labels as the target of the augmented version of
the unlabeled data $\mathbf{X}^{u}_{aug}$ rather than $\mathbf{X}^{u}$ to
combine the pseudo-labeling technique and consistency regularization with
elaborated design, which can make our framework simple and effective.

####  4.4.2 Consistency Regularization

Consistency regularization is an important component of recent state-of-the-
art SSL algorithms [29, 32, 29]. Inspired by SSL algorithms, we propose to
utilize consistency regularization to align instance’s feature of the target
domain data by decreasing the gap between original feature and augmented
version of original feature, which can force the recognition model to learn
the essential characteristics of gestures, which can weaken the influence of
the domain. Specifically, we expect the model to output similar predictions
when fed different versions of the same data. We first augment the unlabeled
target domain data to gain the version of the target domain data. Once the
original data and the augmented data have been fed into the recognition model,
we obtain the two predictions of unlabeled data $\mathbf{\hat{y}}^{u}$ and the
predictions of augmented version $\mathbf{\hat{y}}^{u}_{aug}$. Then we measure
the discrepancies between the predictions of unlabeled data
$\mathbf{\hat{y}}^{u}$ and the predictions of augmented version
$\mathbf{\hat{y}}^{u}_{aug}$.

After obtaining the predicted probabilities of unlabeled data
$\mathbf{\hat{y}}^{u}$ and the predicted probabilities of augmented same
unlabeled data $\mathbf{\hat{y}}^{u}_{aug}$, we expect the two predictions be
similar as much as possible so that they belong to the same classification
category. That is equivalent to the two predictions can produce the same
pseudo labels. To achieve this goal, we train the recognition model by
adopting the pseudo label $\mathbf{y}^{u}$ as target of
$\mathbf{\hat{y}}^{u}_{aug}$, computing and reducing the cross-entropy loss
between the target $\mathbf{y}^{u}$ and $\mathbf{\hat{y}}^{u}_{aug}$. The loss
function $L_{u}$ can be defined as follow:

$L_{u}=-\frac{1}{\left|\mathbf{y}^{u}\right|}\sum_{i=1}^{\left|\mathbf{y}^{u}\right|}\sum_{c=1}^{C}\mathbf{y}^{u}_{ic}\log\left(\mathbf{\hat{y}}^{u}_{aug\
ic}\right),$ (7)

where $\left|\mathbf{y}^{u}\right|$ is the number of pseudo labels. The goal
of unsupervised domain adaptation framework is to minimize the loss $L_{u}$.

####  4.4.3 Confidence Control Constraint

In our framework, whether it can be beneficial to our framework or not mainly
depends on the accuracy of the pseudo-labeling. The right pseudo labels can
speed the convergence of the model up and improve the performance of the
model, but the incorrect pseudo labels will cause the negative impact on the
training of model. As the model with only source domain data for training is
prone to overfitting on the target domain data, the predictions on target
domain $\mathbf{\hat{y}}^{u}$ tend to be overconfident on incorrect categories
and produce incorrect pseudo labels. Such phenomenon will make the model
undergo drastically drop in accuracy. To tackle this problem, we expect the
probability distribution $\mathbf{\hat{y}}^{u}$ not too overconfident. So we
adopt an effective constraint $L_{c}$ to achieve this goal. In specific, we
compute and reduce the cross-entropy loss between the predictions of unlabeled
data $\mathbf{\hat{y}}^{u}$ and the vector $\mathbf{I}$, where $\mathbf{I}$ is
a vector whose every element equals to one. The loss is defined as follows:

$L_{c}=-\frac{1}{\left|\mathbf{y}^{u}\right|}\sum_{i=1}^{\left|\mathbf{y}^{u}\right|}\sum_{c=1}^{C}\log\left(\mathbf{\hat{y}}^{u}_{ic}\right).$
(8)

In particular, the loss is equivalent to Kullback–Leibler (KL) divergence
between the predictions of unlabeled data $\mathbf{\hat{y}}^{u}$ and the
vector $\mathbf{I}$, which measures the discrepancy between the vector
$\mathbf{I}$ and the unlabeled target probabilities $\mathbf{\hat{y}}^{u}$.
Reducing the loss can make the $\mathbf{\hat{y}}^{u}$ be similar to the vector
$\mathbf{I}$ so that the $\mathbf{\hat{y}}^{u}$ is not overconfident.

####  4.4.4 Data augmentation

Reducing the gap between original feature and augmented version of original
features is the key procedure of the consistency regularization. When
transforming the original features, we can not change the semantics of the
feature. Thus, the augmentation methods for images cannot be directly applied
to RF signals (i.e., WiFi signal and mmWave signal). To tackle this challenge,
we design two data augmentation methods: _Feature Erasing_ and _Time Erasing_
as below.

_(1) Feature Erasing_ : Due to the fact that the local features contain unique
domain information, paying too much attention to the local feature will lead
to the performance degradation. To solve the problem, we expect the model pays
more attention to the overall feature associated with the gesture category.
Inspired by [58], we select some local features randomly and erase them by
making their value equal to zero. To ensure that the semantics remain
unchanged, we do not focus on erasing a certain area. The details can be
described as follows:

$\mathbb{F}_{(i,j)}^{(t)}=0,\ t=1,2,...,T,\
(i,j)\in\mathrm{Rand}\left(\mathrm{m}\right),$ (9)

where $\mathbb{F}^{(t)}$ is the feature of $t$-th time dimension, whose
dimensions are $N\times N$, $\mathrm{Rand}(\mathrm{m})$ denotes local feature
coordinates to be erased and m is the number of erasing feature.

_(2) Time Erasing_ : Deep neural networks are prone to overfitting without
sufficient data, which usually leads to disappointing performance. Actually,
it is usually difficult to collect sufficient gesture data in device-free
gesture recognition scenarios. Therefore, increasing the number and diversity
of gestures is helpful to boost the model performance. Erasing the feature on
time dimension within the certain number can increase the number and diversity
of data. Inspired by [59], we augment the data in the time dimension with time
erasing, i.e., we choose a certain number of frames of the sample randomly,
then make the frames’ feature value be zero. This process can be expressed as:

$\mathbb{F}^{(i)}=0,\ i\in\mathrm{Rand}\left(\mathrm{q}\right),$ (10)

where $\mathbb{F}^{(i)}$ is the feature of $i$-th time dimension, whose
dimensions is $N\times N$, $\mathrm{Rand}(\mathrm{q})$ denotes time dimensions
to be erased and q is the number of erasing time feature.

###  4.5 Objective and Training

The final objective is composed of classification loss, unsupervised domain
adaptation loss, and constraint loss. The objective function is defined as
follows

$L=L_{a}+\lambda L_{u}+\eta L_{c},$ (11)

where $L_{a}$ is the classification loss, $L_{u}$ is the unsupervised domain
adaptation loss and $L_{c}$ is the confidence control constraint loss,
$\lambda$ represents the weight of the loss $L_{u}$, $\eta$ is the weight of
the loss $L_{c}$. In the training process, we iteratively update the
parameters according to SGD [60].

##  5 Experiments

In this section, we conduct experiments on two different RF-based gesture
recognition testbeds, i.e., Wi-Fi and mmWave, to evaluate the performance of
the proposed framework. We also conduct the ablation experiments to analyze
the modules in our framework.

###  5.1 Experiment with Wi-Fi Signals

####  5.1.1 Dataset

We adopt the public Widar 3.0 [21] dataset. The dataset is collected from 3
rooms, 16 users, 5 locations and 5 orientations in total. Fig. 3(a) shows the
layouts and sensing areas of the three experimental environments containing
various types, i.e., classroom, hall and office. Fig. 3(b) shows a typical
examples of the deployment of devices and domain configurations in the sensing
area, which is a $2m\times 2m$ square. In this study, we classify 6 commonly
used gestures as shown in Fig. 3(c). The dataset contains 11250 data samples
(15 users $\times$ 5 positions $\times$ 5 orientations $\times$ 6 gestures
$\times$ 5 instances).

(a) Layouts of three evaluation environments.

(b) A typocal setup of devices and domains in one environment.

(c) Sketches of gestures evaluated in the experiment.

Figure 3: Experimental setting with Wi-Fi signals, figure from [21].

####  5.1.2 Data Preprocessing

In our experiments, we use BVP as our model input. BVP is proposed by Widar
3.0 [21] to remove the influence of the domain by describing power
distribution over different velocities. With the noises and offsets filtered
out, only prominent multipath components with non-zero DFS are retained.
Further applying short-term Fourier transform yields power distribution over
the time and Doppler frequency domains, the DFS can be obtained. After that,
we establish the local body coordinates whose origin is the location of the
person and positive $x$-axis aligns with the orientation of the person. Then
we exploit existing passive tracking systems to obtain the location and
orientation of the person. All locations and orientations used in the
following derivation are in the local body coordinates. Any velocity
components around the human body will contribute its signal power to some
frequency component. Finally, we adopt the idea of compressed sensing to
estimate BVP feature.

####  5.1.3 Baseline Methods

We compare our approach with three deep learning models CrossSense [22], EI
[23] and Widar3.0 [21]. Specifically, CrossSense proposes an ANN-based roaming
model to translate signal features from source domains to target domains, and
employs multiple expert models for gesture recognition. EI is a state-of-the-
art adversarial learning scheme for unsupervised domain adaptation, which can
be applied for multiple source domains and target domains. It is composed of
one feature extractor, one domain discriminator and one gesture classifier in
EI. In Widar 3.0 [21], the authors extract the domain invariant feature BVP
based on DFS data and feed them into a model combining CNN’s and GRUs.

####  5.1.4 Cross Domain Evaluation

We evaluate the performance of our framework on cases crossing different
domain factors, including orientation, location, subject, and environment.
When we evaluate on each domain factor, we keep the other domain factors
unchanged. For each case, all of the target domain data without labels are
used to train with the source data and all the target domain data are used to
test. The numbers or letters behind the arrow $\rightarrow$ represents the
target domain data. We conduct experiments using all the baseline methods to
validate our framework’s superiority.

_(1) Cross Orientation:_ In this experiment, we select data of one orientation
in all rooms as the target domain and the other 4 orientations as the source
domains in the three rooms, 15 Users’s data. Table I shows that the proposed
method has an overall accuracy of 87.21%, which is the highest average
accuracy comparing to other methods. Even in this case, our method has an
improvement of 4.35% over Widar 3.0. Orientation 2 and 3 have an accuracy
above 90% while the accuracy declines over 10% for orientation 5. This is
reasonable since orientation 5 is too faraway and the receiver may not capture
the full information of gestures.

TABLE I: Cross Orientation Accuracy (%)

Methods |  $\rightarrow$ 1 |  $\rightarrow$ 2 |  $\rightarrow$ 3 |  $\rightarrow$ 4 |  $\rightarrow$ 5 | Avg
---|---|---|---|---|---|---
CrossSense | 69.41 | 80.23 | 80.45 | 83.18 | 70.07 | 76.67
EI | 70.42 | 81.15 | 88.16 | 90.83 | 70.74 | 80.26
Widar 3.0 | 80.24 | 89.64 | 89.05 | 82.75 | 72.64 | 82.86
Ours | 87.44 | 91.64 | 91.98 | 85.46 | 79.53 | 87.21

_(2) Cross Location:_ Similar to the cross orientation settings, to evaluate
the performance of crossing location, we use data of 4 locations in all rooms
as the source domains, and the data of the last location as the target domain.
The result comparison is shown in Table II. Compared to other methods, our
framework achieves the highest average accuracy of 91.87%. Our framework has
an accuracy above 90% on all location domains. Location E has the highest
accuracy of 95.86%, which is reasonable since location E is the center of
other four locations. In addition, the average accuracy of our method is 1.2%
higher than Widar 3.0.

TABLE II: Cross Location Accuracy (%)

Methods |  $\rightarrow$ A |  $\rightarrow$ B |  $\rightarrow$ C |  $\rightarrow$ D |  $\rightarrow$ E | Avg
---|---|---|---|---|---|---
CrossSense | 88.61 | 85.23 | 82.89 | 78.94 | 85.16 | 84.16
EI | 87.91 | 90.13 | 86.85 | 85.82 | 93.14 | 88.77
Widar 3.0 | 89.14 | 89.10 | 90.50 | 89.50 | 95.15 | 90.67
Ours | 90.47 | 90.30 | 92.30 | 90.44 | 95.86 | 91.87

_(3) Cross Subject:_ Data collected from different persons may have
discrepancies due to their various behavior patterns and their body shapes. To
evaluate on different subjects, we use data from $User\ 9-17$ in Room 1. Each
time, we use the data from one user as the target domain and the other users’
data as the source domains. The results in Table III show our framework
achieves the highest average accuracy of 88.83%. In addition, the average
accuracy of our method is 0.58% higher than Widar 3.0.

TABLE III: Cross Subject Accuracy (%)

Methods |  $\rightarrow$U10 |  $\rightarrow$ U11 |  $\rightarrow$ U12 |  $\rightarrow$ U13 |  $\rightarrow$ U14 |  $\rightarrow$ U15 |  $\rightarrow$ U16 |  $\rightarrow$ U17 | Avg
---|---|---|---|---|---|---|---|---|---
CrossSense | 77.67 | 81.35 | 80.24 | 83.73 | 85.97 | 82.43 | 80.19 | 82.59 | 81.77
EI | 80.11 | 83.56 | 85.91 | 82.05 | 84.36 | 85.44 | 81.23 | 86.21 | 83.61
Widar 3.0 | 86.26 | 90.13 | 83.33 | 89.86 | 87.73 | 89.97 | 91.46 | 88.34 | 88.38
Ours | 86.79 | 89.73 | 80.93 | 91.60 | 88.13 | 92.51 | 91.33 | 90.67 | 88.96

_(4) Cross Environment:_ To evaluate the impact of the environment, we use the
gesture samples collected in two rooms as the source domains data and the
other one as the target domain. The dataset contains 9 persons’ gesture
samples in Room 1, 3 persons’ gesture samples in Room 2, and 4 persons’
gesture samples in Room 3, respectively. The results in Table IV show that we
have an overall accuracy of 90.55%, which is the highest average accuracy.
Moreover, our framework has an improvement of 0.68% over the Widar 3.0.

TABLE IV: Cross Environment Accuracy (%)

Methods |  $\rightarrow$ Room 1 |  $\rightarrow$ Room 2 |  $\rightarrow$ Room 3 | Avg
---|---|---|---|---
CrossSense | 72.78 | 81.75 | 82.76 | 79.09
EI | 75.73 | 84.54 | 83.98 | 81.42
Widar 3.0 | 81.58 | 93.47 | 91.08 | 88.87
Ours | 83.93 | 94.88 | 92.85 | 90.55

_(5) Comparison with CACDGR:_ Due to the fact that the CACDGR model [24] only
uses very small part of the Widar 3.0 dataset to evaluate their model, we also
use the same part of dataset similar with CACDGR to compare our framework with
their model. We evaluate the performance on the environment domain. Same as
CACDGR, we keep each room’s person number as the same. We choose 3 persons
from each room and obtain a subset of the whole dataset which is $User\ 14-16$
in Room 1, $User1,User2,User6$ in Room 2 and $User\ 7-9$ in Room 3. The
experiments take one room’s data as the source domains. From Table V, we find
that our framework has the highest accuracy of 88.80%.

TABLE V: Accuracy (%) Comparison with CACDGR

Methods | Room 1 | Room 2 | Room 3 | Avg
---|---|---|---|---
Widar 3.0 | 87.50 | 86.36 | 85.16 | 86.34
CACDGR | 87.30 | 87.40 | 88.80 | 87.83
Ours | 89.36 | 89.10 | 87.94 | 88.80

In general, our framework has the highest average accuracy in four domains
(locations, orientations, subjects and environments), which shows that our
framework indeed improves performance in the unlabeled target domain on Wi-Fi
signal dataset.

###  5.2 Experiment with mmWave Radar Signals

####  5.2.1 Dataset

We use the public dataset [57], which is collected from 25 volunteers, 6
environments and 5 locations to evaluate our framework. Fig. 4 shows one of
the evaluation environments and Fig. 5 illustrates the setup of 5 locations in
each environment. The dataset contains six common gestures including push
(PH), pull (PL), left swipe (LS) right swipe (RS), clockwise turning (CT) and
anticlockwise turning (AT). Each volunteer is asked to perform each kind of
gesture 5 times at each location. In total, the dataset has collected 10800
samples.

Figure 4: Experiment setting with mmWave radar signals. Figure 5: A typical
setup of devices in one environment.

####  5.2.2 Data Preprocessing

As shown in Fig. 6, To obtain the Dynamic Range Angle Image (DRAI) sequences
for gesture recognition, we first employ 3D-FFT on raw signals to derive the
ranges, velocities and angles of hands. Then, we adopt noise elimination to
filter environmental interference and improve the robustness of the
recognition model.

Figure 6: The calculation process of DRAI sequences.

In specific, the radar continuously transmits FMCW signals and mixes the
received chirp to obtain an intermediate frequency (IF) signal. The range of
the detected object can be computed using FFT. After that, the radar transmits
a frame consisting of N chirps to obtain the velocity of the targeted object.
The velocity of the object can be derived from the phase difference caused by
the doppler effect between two adjacent chirps. We employ FFT to extract
doppler information and obtain Range Doppler Image (RDI). Moreover, the Angle
of Arrival (AoA) can be computed by cascading multiple RDIs which are
different from each other because there are phase changes between adjacent
receiving antennas. The FFT of angle is performed on the direction of
receiving antennas to acquire the range-doppler-angle matrix. Finally, we make
doppler frequency lower than a velocity threshold as zero to remove static
clutter. To obtain the signal intensity of each doppler bin, we sum the
averaged range doppler matrix along the range dimension. Only doppler bins
whose signal intensity is higher than the threshold will be counted to obtain
DRAI sequences.

####  5.2.3 Baseline Methods

We compare our approach with two deep learning models RadarNet [27] and
Widar3.0 [21]. Specifically, RadarNet designs an efficient neural network and
trains a robust model. We first process the mmWave radar signals to Range
Doppler Image, then use the Range Doppler Image as the RadarNet model’s input.
In Widar 3.0 paper, the authors extract the domain invariant feature BVP based
on Wi-Fi signals and feed them into a gesture recognition model combining
CNN’s and GRUs. In this experiment, we use Widar 3.0’s gesture recognition
model as the backbone model and use the DRAI sequences as the model’s input.

####  5.2.4 Cross Domain Evaluation

We evaluate the performance of the proposed framework on cases crossing
different domain factors, including location, subject, and environment. When
evaluating on each domain factor, we keep the other domain factors unchanged.
For each case, all of the target domain data without labels are used to train
with the source data and all the target domain data are used to test. We
conduct experiments using all the baseline methods to validate our framework’s
superiority and generalization ability.

_(1) Cross Location:_ To evaluate our framework’s performance of crossing
location, we use 4 locations of all data as the source domains, and the data
of the last location as the target domain. In our experiment, $L1,L2,L3,L4,L5$
represent the five locations of subject. The result comparison is shown in
Table VI. Compared to other methods, our framework achieves the highest
average accuracy of 95.65% and has significant improvement of 15.23% over the
Widar 3.0 and 10.75% over RadarNet. Location L2 has the highest accuracy of
98.84%, which is reasonable since location P2 is the center of other four
locations.

TABLE VI: Cross Location Accuracy (%)

Methods |  $\rightarrow$ L1 |  $\rightarrow$ L2 |  $\rightarrow$ L3 |  $\rightarrow$ L4 |  $\rightarrow$ L5 | Avg
---|---|---|---|---|---|---
RadarNet | 67.69 | 96.81 | 69.77 | 95.19 | 95.05 | 84.90
Widar 3.0 | 80.13 | 97.45 | 82.59 | 75.92 | 65.65 | 80.32
Ours | 91.20 | 98.84 | 94.30 | 96.94 | 96.95 | 95.65

_(2) Cross Subject:_ Different subjects’ data may show different discrepancies
due to their various behavior patterns and their body shapes. To evaluate on
the impact of different subjects, we use data from $User\ 1-6$ in all rooms to
evaluate the model. Each time, we use the data from one user as the target
domain and the other users’ data as the source domains. The results in Table
VII show our framework achieves the highest average accuracy of 97.97%. All
subjects have the accuracy above 93%, $User\ 1-3$ have the accuracy above 99%,
which almost 100%. In addition, the average accuracy of our method is 1.52%
higher than Widar 3.0.

TABLE VII: Cross Subject Accuracy (%)

Methods |  $\rightarrow$U1 |  $\rightarrow$ U2 |  $\rightarrow$ U3 |  $\rightarrow$ U4 |  $\rightarrow$ U5 |  $\rightarrow$ U6 | Avg
---|---|---|---|---|---|---|---
RadarNet | 95.06 | 92.93 | 83.06 | 85.20 | 90.26 | 90.40 | 89.48
Widar 3.0 | 98.80 | 98.13 | 91.85 | 98.33 | 98.83 | 92.78 | 96.45
Ours | 99.42 | 99.02 | 93.28 | 99.33 | 98.83 | 96.90 | 97.97

_(3) Cross Environment:_ Different rooms have different layouts, which have an
influence on the signal propagation path. To evaluate the impact of the
environment, we use the gesture samples collected in five rooms as the source
domains data and the one as the target domain. $E\ 1-6$ represent the six
environments. The results in Table VIII show that we achieve an overall
accuracy of 99.04%, which is the highest average accuracy. All environments
have the accuracy above 98%. Moreover, our method has a improvement of 2.25%
over the Widar 3.0 and 2.83% over the RadarNet.

TABLE VIII: Cross Environment Accuracy (%)

Methods |  $\rightarrow$E1 |  $\rightarrow$ E2 |  $\rightarrow$ E3 |  $\rightarrow$ E4 |  $\rightarrow$ E5 |  $\rightarrow$ E6 | Avg
---|---|---|---|---|---|---|---
RadarNet | 99.09 | 97.11 | 92.47 | 98.66 | 98.00 | 91.93 | 96.21
Widar 3.0 | 97.33 | 98.53 | 94.80 | 95.06 | 98.13 | 96.93 | 96.79
Ours | 99.19 | 98.93 | 98.66 | 99.46 | 98.93 | 99.06 | 99.04

In conclusion, similar to the experiment on Wi-Fi signals, our framework
achieves considerable improvement of performance in the unlabeled target
domain on mmWave radar signal dataset, which shows the superiority of our
framework.

###  5.3 Ablation Study

In this section, we perform extensive ablation study on the two RF signals to
understand the impact of confidence control constraint, data augmentation
methods and pseudo labeling.

####  5.3.1 Impact of Confidence Control Constraint

In this sub-section, we study the impact of the confidence control constraint,
then evaluate the impact of different weights $\eta$ of confidence control
constraint in the final objective. We evaluate the impact of the constraint
and different weights with the Wi-Fi signal and mmWave radar signal,
respectively. Furthermore, we conduct experiments on five orientation domains
with Wi-Fi signal and the five location domains with mmWave radar signal. We
use one domain data as the target domain and the remaining domain data as the
source domain. We train the model with source domain data and test it on
target domain data. The accuracy corresponding to each threshold is the
average result of the five target domains.

To understand whether the proposed framework always can achieve the effective
performance without constraint, we first compare the full model with base
model (without confidence control constraint). On Wi-Fi dataset, Fig. 7(a)
shows that the performance of the model without the confidence control
constraint is worse than the performance of the full model. On mmWave radar
dataset, Fig. 7(b) shows that the full model achieves the the accuracy above
the 90%, while the base model is difficult to converge. This indicates that
the confidence control constraint is necessary and effective for our framework
on Wi-Fi dataset and mmWave radar dataset.

(a) Impact of Confidence Control Constraint on Wi-Fi Dataset

(b) Impact of Confidence Control Constraint on mmWave Radar Dataset

Figure 7: Impact of Confidence Control Constraint.

(a) Impact of Constraint’s Weight on Wi-Fi Dataset

(b) Impact of Constraint’s Weight on mmWave Radar Dataset

Figure 8: Impact of Constraint’s Weight.

Then, we employ the analysis experiment to evaluate the impact of different
constraint weights $\eta$. In specific, we compare the performance of our
framework with different constraint weights. The weight values $\eta$ in this
experiment ranges from 0 to 1. From Fig. 8(a), we find that the weight value
of 0.92 shows the highest accuracy on Wi-Fi dataset. Fig. 8(b) shows that the
model achieve the best performance on mmWave radar dataset when the weight
value is 0.95.

####  5.3.2 Impact of Data Augmentation

To study the impact of the data augmentation methods, we employ experiments
with the two RF-based signals: Wi-Fi signal and mmWave radar signal. In
specific, we conduct the ablation experiments with five orientations data on
Wi-Fi signals and five locations data on mmWave radar signals. We use four
domain data as source domains and the remaining domain data as the target
domain. We compare the full model with bare model (without data augmentation).
From Table IX, we can observe that the performance of the model without data
augmentation drops 1.25% on Wi-Fi dataset. The Table X shows that the the
classification accuracy of the base model is not good as the model adopting
the data augmentation. Thus, we can summarize that the data augmentation plays
important roles in our framework.

TABLE IX: Impact of Data Augmentation with Wi-Fi Signal

Methods |  $\rightarrow$ 1 |  $\rightarrow$ 2 |  $\rightarrow$ 3 |  $\rightarrow$ 4 |  $\rightarrow$ 5 | Avg
---|---|---|---|---|---|---
Bare Model | 85.66 | 90.88 | 90.74 | 84.94 | 77.62 | 85.96
Full Model | 87.44 | 91.64 | 91.98 | 85.46 | 79.53 | 87.21

TABLE X: Impact of Data Augmentation with mmWave Radar Signal

Methods |  $\rightarrow$ 1 |  $\rightarrow$ 2 |  $\rightarrow$ 3 |  $\rightarrow$ 4 |  $\rightarrow$ 5 | Avg
---|---|---|---|---|---|---
Bare Model | 90.01 | 95.30 | 92.45 | 93.21 | 92.37 | 92.66
Full Model | 91.20 | 98.84 | 94.30 | 96.94 | 96.95 | 95.65

####  5.3.3 Impact of Pseudo Label Threshold

In our framework, we use pseudo labeling technique to obtain the pseudo labels
of the unlabeled target domain data. The accuracy of the pseudo labeling have
a significant impact on the performance of our framework. To reduce the number
of incorrect pseudo labels, we utilize the threshold to select the highly
confident data. In the sub-section, we study the impact of different pseudo
label initial threshold. In specific, we employ the analysis on the five
location domains with mmWave radar signal and the five orientation domains
with Wi-Fi signal. The initial threshold values in this experiment ranges from
0 to 0.99. In the Fig. 9, the accuracy corresponding to each threshold is the
average result of the five domains. The Fig. 9(a) shows the impact of the
confidence threshold on Wi-Fi dataset. The performance of the recognition
model improves with the threshold increasing and achieves the highest accuracy
of 91.98% when the initial threshold value is 0.92. The Fig. 9(b) shows that
the accuracy of model training with mmWave radar signal drops when using a
small initial threshold value and achieves the highest accuracy of 95.65% when
the initial threshold value is 0.95.

(a) Impact of Confidence Threshold on Wi-Fi Dataset.

(b) Impact of Confidence Threshold on mmWave Radar Dataset.

Figure 9: Impact of Confidence Threshold.

##  6 Conclusion

In this paper, we propose an unsupervised domain adaptation framework for RF-
based gesture recognition. Our framework can increase the capability of cross
domain recognition by adopting pseudo-labeling and consistency regularization
with elaborated design. Utilizing two data augmentation methods and a
confidence control constraint, our framework achieves state-of-the-art
prediction performance. We conduct experiments on two different device-free
gesture recognition testbeds, i.e., Wi-Fi and mmWave, to evaluate the
performance of the proposed framework. The experimental results demonstrated
the superiority of our framework over existing ones. The proposed framework
can be extended to other wireless signals, enabling applications in future
smart home environments.

## References

  * [1] G. Gkioxari, R. Girshick, P. Dollár, and K. He, “Detecting and recognizing human-object interactions,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8359–8367.
  * [2] M. Wang, B. Ni, and X. Yang, “Recurrent modeling of interaction context for collective activity recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3048–3056.
  * [3] A. Bulling, U. Blanke, and B. Schiele, “A tutorial on human activity recognition using body-worn inertial sensors,” _ACM Computing Surveys (CSUR)_ , vol. 46, no. 3, pp. 1–33, 2014.
  * [4] Y. Guan and T. Plötz, “Ensembles of deep lstm learners for activity recognition using wearables,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 1, no. 2, pp. 1–28, 2017\.
  * [5] X. Huang and M. Dai, “Indoor device-free activity recognition based on radio signal,” _IEEE Transactions on Vehicular Technology_ , vol. 66, no. 6, pp. 5316–5329, 2016.
  * [6] Y. Wang, J. Liu, Y. Chen, M. Gruteser, J. Yang, and H. Liu, “E-eyes: device-free location-oriented activity identification using fine-grained wifi signatures,” in _Proceedings of the 20th annual international conference on Mobile computing and networking_ , 2014, pp. 617–628.
  * [7] S. Shi, S. Sigg, W. Zhao, and Y. Ji, “Monitoring attention using ambient fm radio signals,” _IEEE Pervasive Computing_ , vol. 13, no. 1, pp. 30–36, 2014\.
  * [8] W. Wang, A. X. Liu, M. Shahzad, K. Ling, and S. Lu, “Device-free human activity recognition using commercial wifi devices,” _IEEE Journal on Selected Areas in Communications_ , vol. 35, no. 5, pp. 1118–1131, 2017.
  * [9] Y. Ma, G. Zhou, S. Wang, H. Zhao, and W. Jung, “Signfi: Sign language recognition using wifi,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 2, no. 1, pp. 1–21, 2018\.
  * [10] Y. He, Y. Chen, Y. Hu, and B. Zeng, “Wifi vision: sensing, recognition, and detection with commodity mimo-ofdm wifi,” _IEEE Internet of Things Journal_ , vol. 7, no. 9, pp. 8296–8317, 2020.
  * [11] D. Zhang, Y. Hu, Y. Chen, and B. Zeng, “Breathtrack: Tracking indoor human breath status via commodity wifi,” _IEEE Internet of Things Journal_ , vol. 6, no. 2, pp. 3899–3911, 2019.
  * [12] D. Zhang, Y. He, X. Gong, Y. Hu, Y. Chen, and B. Zeng, “Multitarget aoa estimation using wideband lfmcw signal and two receiver antennas,” _IEEE Transactions on Vehicular Technology_ , vol. 67, no. 8, pp. 7101–7112, 2018.
  * [13] Y. He, D. Zhang, Y. Hu, and Y. Chen, “Non-line-of-sight imaging with radio signals,” in _2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2020, pp. 11–16.
  * [14] J. Lien, N. Gillian, M. E. Karagozler, P. Amihood, C. Schwesig, E. Olson, H. Raja, and I. Poupyrev, “Soli: Ubiquitous gesture sensing with millimeter wave radar,” _ACM Transactions on Graphics (TOG)_ , vol. 35, no. 4, pp. 1–19, 2016.
  * [15] S. Zhu, J. Xu, H. Guo, Q. Liu, S. Wu, and H. Wang, “Indoor human activity recognition based on ambient radar with signal processing and machine learning,” in _2018 IEEE international conference on communications (ICC)_. IEEE, 2018, pp. 1–6.
  * [16] T. Fan, C. Ma, Z. Gu, Q. Lv, J. Chen, D. Ye, J. Huangfu, Y. Sun, C. Li, and L. Ran, “Wireless hand gesture recognition based on continuous-wave doppler radar sensors,” _IEEE Transactions on Microwave Theory and Techniques_ , vol. 64, no. 11, pp. 4012–4020, 2016.
  * [17] C. L. Li, M. Liu, and Z. Cao, “Wihf: Gesture and user recognition with wifi,” _IEEE Transactions on Mobile Computing_ , 2020.
  * [18] Y. Zou, J. Xiao, J. Han, K. Wu, Y. Li, and L. M. Ni, “Grfid: A device-free rfid-based gesture recognition system,” _IEEE Transactions on Mobile Computing_ , vol. 16, no. 2, pp. 381–393, 2016.
  * [19] H. Abdelnasser, K. Harras, and M. Youssef, “A ubiquitous wifi-based fine-grained gesture recognition system,” _IEEE Transactions on Mobile Computing_ , vol. 18, no. 11, pp. 2474–2487, 2018.
  * [20] A. Virmani and M. Shahzad, “Position and orientation agnostic gesture recognition using wifi,” in _Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services_ , 2017, pp. 252–264.
  * [21] Y. Zheng, Y. Zhang, K. Qian, G. Zhang, Y. Liu, C. Wu, and Z. Yang, “Zero-effort cross-domain gesture recognition with wi-fi,” in _Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services_ , 2019, pp. 313–325.
  * [22] J. Zhang, Z. Tang, M. Li, D. Fang, P. Nurmi, and Z. Wang, “Crosssense: Towards cross-site and large-scale wifi sensing,” in _Proceedings of the 24th Annual International Conference on Mobile Computing and Networking_ , 2018, pp. 305–320.
  * [23] W. Jiang, C. Miao, F. Ma, S. Yao, Y. Wang, Y. Yuan, H. Xue, C. Song, X. Ma, D. Koutsonikolas _et al._ , “Towards environment independent device free human activity recognition,” in _Proceedings of the 24th Annual International Conference on Mobile Computing and Networking_ , 2018, pp. 289–304.
  * [24] H. Kang, Q. Zhang, and Q. Huang, “Context-aware wireless based cross domain gesture recognition,” _IEEE Internet of Things Journal_ , 2021.
  * [25] H. Liu, Y. Wang, A. Zhou, H. He, W. Wang, K. Wang, P. Pan, Y. Lu, L. Liu, and H. Ma, “Real-time arm gesture recognition in smart home scenarios via millimeter wave sensing,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 4, no. 4, pp. 1–28, 2020\.
  * [26] P. S. Santhalingam, A. A. Hosain, D. Zhang, P. Pathak, H. Rangwala, and R. Kushalnagar, “mmasl: Environment-independent asl gesture recognition using 60 ghz millimeter-wave signals,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 4, no. 1, pp. 1–30, 2020.
  * [27] E. Hayashi, J. Lien, N. Gillian, L. Giusti, D. Weber, J. Yamanaka, L. Bedal, and I. Poupyrev, “Radarnet: Efficient gesture recognition technique utilizing a miniature radar sensor,” in _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , 2021, pp. 1–14.
  * [28] X. J. Zhu, “Semi-supervised learning literature survey,” 2005.
  * [29] M. Sajjadi, M. Javanmardi, and T. Tasdizen, “Regularization with stochastic transformations and perturbations for deep semi-supervised learning,” _Advances in neural information processing systems_ , vol. 29, pp. 1163–1171, 2016.
  * [30] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” _arXiv preprint arXiv:1703.01780_ , 2017.
  * [31] K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” _arXiv preprint arXiv:2001.07685_ , 2020\.
  * [32] Q. Xie, Z. Dai, E. Hovy, M.-T. Luong, and Q. V. Le, “Unsupervised data augmentation for consistency training,” _arXiv preprint arXiv:1904.12848_ , 2019.
  * [33] D.-H. Lee _et al._ , “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,” in _Workshop on challenges in representation learning, ICML_ , vol. 3, no. 2, 2013, p. 896.
  * [34] S. Ben-David, J. Blitzer, K. Crammer, F. Pereira _et al._ , “Analysis of representations for domain adaptation,” _Advances in neural information processing systems_ , vol. 19, p. 137, 2007.
  * [35] M. Long, Y. Cao, J. Wang, and M. Jordan, “Learning transferable features with deep adaptation networks,” in _International conference on machine learning_. PMLR, 2015, pp. 97–105.
  * [36] M. Long, H. Zhu, J. Wang, and M. I. Jordan, “Deep transfer learning with joint adaptation networks,” in _International conference on machine learning_. PMLR, 2017, pp. 2208–2217.
  * [37] G. Kang, L. Jiang, Y. Yang, and A. G. Hauptmann, “Contrastive adaptation network for unsupervised domain adaptation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4893–4902.
  * [38] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, “Adversarial discriminative domain adaptation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7167–7176.
  * [39] S. Sankaranarayanan, Y. Balaji, C. D. Castillo, and R. Chellappa, “Generate to adapt: Aligning domains using generative adversarial networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8503–8512.
  * [40] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” _The journal of machine learning research_ , vol. 17, no. 1, pp. 2096–2030, 2016.
  * [41] M.-Y. Liu, T. Breuel, and J. Kautz, “Unsupervised image-to-image translation networks,” in _Advances in neural information processing systems_ , 2017, pp. 700–708.
  * [42] D. Berthelot, N. Carlini, E. D. Cubuk, A. Kurakin, K. Sohn, H. Zhang, and C. Raffel, “Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring,” _arXiv preprint arXiv:1911.09785_ , 2019.
  * [43] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” _arXiv preprint arXiv:1905.02249_ , 2019.
  * [44] X. Zhai, A. Oliver, A. Kolesnikov, and L. Beyer, “S4l: Self-supervised semi-supervised learning,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1476–1485.
  * [45] S. Ao, X. Li, and C. Ling, “Fast generalized distillation for semi-supervised domain adaptation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 31, no. 1, 2017.
  * [46] K. Saito, D. Kim, S. Sclaroff, T. Darrell, and K. Saenko, “Semi-supervised domain adaptation via minimax entropy,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 8050–8058.
  * [47] C. Qin, L. Wang, Q. Ma, Y. Yin, H. Wang, and Y. Fu, “Contradictory structure learning for semi-supervised domain adaptation,” in _Proceedings of the 2021 SIAM International Conference on Data Mining (SDM)_. SIAM, 2021, pp. 576–584.
  * [48] D. Li and T. Hospedales, “Online meta-learning for multi-source and semi-supervised domain adaptation,” in _European Conference on Computer Vision_. Springer, 2020, pp. 382–403.
  * [49] L. Yang, Y. Wang, M. Gao, A. Shrivastava, K. Q. Weinberger, W.-L. Chao, and S.-N. Lim, “Mico: Mixup co-training for semi-supervised domain adaptation,” _arXiv e-prints_ , pp. arXiv–2007, 2020.
  * [50] Y. Chen, H. Deng, D. Zhang, and Y. Hu, “Speednet: Indoor speed estimation with radio signals,” _IEEE Internet of Things Journal_ , vol. 8, no. 4, pp. 2762–2774, 2020.
  * [51] D. Zhang, Y. Hu, Y. Chen, and B. Zeng, “Calibrating phase offsets for commodity wifi,” _IEEE Systems Journal_ , vol. 14, no. 1, pp. 661–664, 2019\.
  * [52] Q. Xu, Y. Chen, B. Wang, and K. R. Liu, “Trieds: Wireless events detection through the wall,” _IEEE Internet of Things Journal_ , vol. 4, no. 3, pp. 723–735, 2017.
  * [53] Y. Han, Y. Chen, B. Wang, and K. R. Liu, “Enabling heterogeneous connectivity in internet of things: A time-reversal approach,” _IEEE Internet of Things Journal_ , vol. 3, no. 6, pp. 1036–1047, 2016.
  * [54] D. Zhang, Y. Hu, and Y. Chen, “Mtrack: Tracking multiperson moving trajectories and vital signs with radio signals,” _IEEE Internet of Things Journal_ , vol. 8, no. 5, pp. 3904–3914, 2020.
  * [55] Y. Chen, X. Su, Y. Hu, and B. Zeng, “Residual carrier frequency offset estimation and compensation for commodity wifi,” _IEEE Transactions on Mobile Computing_ , vol. 19, no. 12, pp. 2891–2902, 2019.
  * [56] Z. Wang, S. Chen, W. Yang, and Y. Xu, “Environment-independent wi-fi human activity recognition with adversarial network,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 3330–3334.
  * [57] Y. Li, D. Zhang, J. Chen, J. Wan, D. Zhang, Y. Hu, Q. Sun, and Y. Chen, “Towards domain-independent and real-time gesture recognition using mmwave signal,” _arXiv preprint arXiv:2111.06195_ , 2021.
  * [58] J. Li, Z. Wang, and X. Hu, “Learning intact features by erasing-inpainting for few-shot classification,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 9, 2021, pp. 8401–8409.
  * [59] J. Zhang, F. Wu, B. Wei, Q. Zhang, H. Huang, S. W. Shah, and J. Cheng, “Data augmentation and dense-lstm for human activity recognition using wifi signal,” _IEEE Internet of Things Journal_ , vol. 8, no. 6, pp. 4628–4641, 2020.
  * [60] N. Qian, “On the momentum term in gradient descent learning algorithms,” _Neural networks_ , vol. 12, no. 1, pp. 145–151, 1999.