# Information-Preserving Contrastive Learning for Self-Supervised
Representations

Tianhong Li Lijie Fan∗ Yuan Yuan Hao He Yonglong Tian Dina Katabi
MIT CSAIL Indicates equal contribution.

###### Abstract

Contrastive learning is very effective at learning useful representations
without supervision. Yet contrastive learning has its limitations. It can
learn a shortcut that is irrelevant to the downstream task, and discard
relevant information. Past work has addressed this limitation via custom data
augmentations that eliminate the shortcut. This solution however does not work
for data modalities that are not interpretable by humans, e.g., radio signals.
For such modalities, it is hard for a human to guess which shortcuts may exist
in the signal, or how to alter the radio signals to eliminate the shortcuts.
Even for visual data, sometimes eliminating the shortcut may be undesirable.
The shortcut may be irrelevant to one downstream task but important to
another. In this case, it is desirable to learn a representation that captures
both the shortcut information and the information relevant to the other
downstream task. This paper presents information-preserving contrastive
learning (IPCL), a new framework for unsupervised representation learning that
preserves relevant information even in the presence of shortcuts. We
empirically show that IPCL addresses the above problems and outperforms
contrastive learning on radio signals and learning RGB data representation
with different features that support different downstream tasks.

##  1 Introduction

The area of unsupervised or self-supervised representation learning is growing
rapidly [19, 21, 9, 20, 5, 11, 3, 4, 13, 10, 16, 22, 27, 28]. It refers to
learning useful representations of data that capture potential labels of
interest, and doing so without human supervision. Further, contrastive
learning is increasingly considered as a standard and highly competitive
method for unsupervised representation learning. Features learned with this
method have been shown to generalize well to downstream tasks, and in some
cases surpass the performance of supervised models [20, 5, 11, 3, 4, 19, 21,
9, 12].

Contrastive learning learns representations by contrasting positive samples
against negative samples. When learning without supervision, typically, a data
sample is chosen as an anchor (e.g., an image); positive samples are chosen as
different augmented versions of the anchor (e.g., randomly cropped and color
distorted version of of the anchor image), whereas negative samples come from
other samples in the dataset.

Despite its effectiveness, contrastive learning is vulnerable to shortcuts –
i.e., if simple features are contrastive enough to separate positive samples
from negative samples, contrastive learning might learn such simple (or
simpler) features though they may be irrelevant to downstream tasks, and
ignore other more relevant features. For example, the authors of [3] show that
color distribution can be a shortcut to distinguish two patches cropped from
the same image, from two patches from different images, but this feature is
not very useful for object classification. Past work addresses this problem by
adding data augmentations to break the shortcut semantics, so that the network
may learn the desired semantics [3, 11, 4, 5].

However, it is not always possible to address shortcuts via data augmentation.
First, machine learning tasks are expanding to new modalities, many of which
are not directly interpretable by humans such radio signals [31, 33, 32, 18,
7, 8], or acceleration data [15, 14, 7]. Consider for example the task of
through-wall pose estimation [31]. This task leverages that radio signals
traverse walls and occlusions and reflect off the human body; it designs
neural network models that analyze the radio frequency (RF) signals and infer
the human pose, even when people are behind a wall, or the scene has poor
lighting conditions. Such task would benefit significantly from unsupervised
representation learning given the limited amount of labeled RF signals, and
the difficulties of labeling such modality which is not interpretable by human
workers [31, 33, 32, 18]. However, like RGB data, RF data can have shortcuts
that make contrastive learning ineffective (see the results section for
empirical validation). But unlike RGB data, it is hard for humans to guess
what the shortcuts may be, or how to alter the RF signal to create data
augmentations that eliminate the shortcuts while preserving useful
information.

Figure 1: Illustrative example, where the input has two types of information:
digit and background object, but existing unsupervised contrastive learning
methods generate a representation that captures mostly the background object
and ignores the digit.

Second, eliminating shortcut features is not always desirable, even for RGB-
based tasks. Sometimes the shortcut features are irrelevant to one downstream
task but important to another task. For example, the authors of [25] highlight
the scenario in Figure 1, where each image shows a digit (from MNIST) on a
randomly chosen background object (from STL-10). They show that features
related to the background object can create a shortcut that prevents
contrastive learning from learning features related to the digit. However,
eliminating features about the background object is not desirable since
ideally one would like a representation that works for both downstream tasks:
digit classification, and background object classification.

To address the above problems, we propose a new framework for contrastive
learning that we call information-preserving contrastive learning (IPCL). The
key idea underlying IPCL is to ensure that even when the data has potential
shortcuts, the representation will learn features about all types of semantic
information in the data (digit and background object), and contrastive
learning will ensure that all of those features (not just the shortcut
features) are closely mapped in the embedding space for positive samples and
stay distant for negative samples. To ensure that these properties are
satisfied, IPCL augments contrastive learning with an auxiliary task that uses
the learned representation to restore randomly cropped patches in the anchor
sample. By doing so, IPCL ensures that the learned representation summarizes
all useful information in the input, but also enjoys contrastive properties.

We apply the IPCL framework to both RGB-based and RF-based tasks, designing
neural network models appropriate for those modalities. We then evaluate IPCL
on both RGB and RF datasets. Specifically, we evaluate IPCL on two image
classification datasets. The first is the Colorful-MNIST, a synthetic dataset
proposed in [25] and illustrated in Figure 1. The results show that IPCL
outperforms by a high margin state-of-the-art contrastive learning methods
like SimCLR, MoCo, and Contrastive Predictive Coding in its ability to learn a
representation that works for both digit classification and background object
classification. Our method can achieve 11.7% error rate on digit
classification and 54.0% error rate on background object classification, while
all other contrastive learning baselines have near-random error rate on digit
classification ($\sim$90%) and have comparable error rates to IPCL on
background object classification (51% to 64%). The second dataset is FairFace
[17], where we show that IPCL is more effective than state-of-the-art
contrastive learning at learning a single representation that works for
multiple downstream classification tasks based on race, gender and age. IPCL
outperforms SimCLR on age and gender classification by 5.7% and 8.6% in error
rate, and achieves comparable results as SimCLR on ethnicity classification
(38.8% vs. 37.7% error rate).

We also evaluate IPCL on three real-world RF-based through-wall human sensing
tasks: 3D pose estimation, human action detection, and person re-
identification. Our empirical results show that IPCL can consistently improve
the performance of RF-based models on each of these tasks, and outperform the
contrastive learning baselines. Specifically, IPCL achieves 8.6% relative
improvement in keypoint-localization error of 3D pose estimation, 2.7%
relative improvement on action recognition and 3.6% relative improvement on
person ReID at home.

To summarize, our paper has the following contributions:

  * •

We introduce IPCL, a new framework that allows contrastive learning to be
effective in the presence of shortcuts. It is particularly suitable for data
modalities that are hard to interpret by humans (e.g., RF signals), and for
learning representations that preserve information about multiple aspects of
the input data, particularly when these aspects are independent (e.g., digit
and background object) or share low mutual information.

  * •

We are the first to study unsupervised representation learning from radio
signals, demonstrating the possibility of boosting the performance of RF-based
human sensing tasks by leveraging unlabeled RF signals.

  * •

We conduct extensive experiments on a synthetic RGB image dataset, a real-
world RGB dataset, and RF datasets demonstrating the impact of shortcuts on
contrastive learning, and the ability of IPCL to address those problems.

##  2 Related Work

(a) Unsupervised Contrastive Representation Learning:  Early work on
unsupervised representation learning has focused on designing pretext tasks
and training the network to predict the pseudo labels for those tasks. Such
tasks include solving Jigsaw Puzzles [19], filling a missing patch in the
input [21], or predicting image rotation [9]. However, pretext tasks have to
be handcrafted, and the generality of their representations is typically
limited [3].

Hence, recently researchers have been focusing on contrastive unsupervised
learning, which emerged as a competitive and systematic method for learning
effective representations without human supervision. The learned features
generalize well to downstream tasks, outperform representations learned
through handcrafted pretext tasks, and even surpass the performance of
supervised models on some tasks [5, 11, 3, 4]. Multiple successful contrastive
learning frameworks have been proposed, which typically differ in the way they
sample negative pairs. To name a few, SimCLR [3] uses a large batch-size, and
samples negative pairs within each batch. The momentum-contrastive approach
[11] leverages a moving-average encoder and a queue to generate negative
samples on-the-fly during training. Contrastive-Multiview-Coding [23]
maintains a memory-bank to store features, and generates negative samples from
this memory bank.

Past work has also reported limitations in using contrastive learning for
unsupervised representations. For example, the authors of [3] showed that
color distribution can be a shortcut, and proposed to break this shortcut
through color-distortion data augmentation. Further, the authors of [25] noted
that when the data includes multiple types of semantics, one type may produce
a shortcut that prevents contrastive learning from learning effective features
of other semantics (as is the case in Figure 1, where the background object
information can be a shortcut that prevents learning information related to
the digit in the image). They proposed a solution that learns contrastive
views suitable for the desired downstream task. While they share our goal of
supporting different downstream tasks, their method requires supervision since
they learn their contrastive views from labeled data. In contrast, our
representation learning is completely unsupervised.

Another related work is contrastive-predictive-coding (CPC) [20, 12]. CPC has
some similarity to IPCL in that it has a predictive task that aims to restore
missing information. However, the key difference between CPC and our method is
that the prediction target of CPC is the features of a future frame, while the
prediction target of our method is the raw input data. As a result, the
representation learned by CPC is not forced to contain necessary information
to reconstruct the input, making it susceptible to shortcuts, just like other
contrastive learning methods.

(b) Learning from Radio Signals:  There is a growing body of literature that
leverages radio signals to learn pose estimation, action recognition,
captioning, and re-identifications through walls and occlusions [31, 33, 18,
6, 29, 8, 7, 32, 15, 26, 24]. All of this work however relies on supervised
learning from annotated data. Given the difficulty of annotating RF datasets
with ground truth labels for pose or actions, past work is limited to
relatively small datasets, which hampers its performance. Our work allows
these models to leverage large amounts of unannotated RF signals to improve
the performance of their downstream tasks.

##  3 Information-Preserving Contrastive Learning

Previous work has shown that contrastive learning can be easily biased to
‘shortcut’ semantics, and addressed this bias through data augmentations that
break the ‘shortcut’ semantics so that the network may learn the desired
semantics. Such a design however suffers from two problems:

  1. 1.

It requires knowing which types of semantics can be shortcut and designing
specific augmentations to remove that shortcut. However for modalities like
RF, which are not directly interpretable by humans, it is unclear to a human
which semantics in the RF signal create a shortcut. As a result, it is hard to
design augmentation techniques to remove the shortcut.

  2. 2.

As shown in [25], there might be multiple types of semantics that are useful
for downstream tasks, and one type of them can be a shortcut with respect to
one task but a key useful feature for another task. In this case, removing a
shortcut might help one downstream task but hurt the other, making it hard to
learn a representation that generalizes well to multiple downstream tasks.

To address the above problems, we need an unsupervised representation learning
method that does not require prior knowledge of the potential shortcut
semantics in the data, and can learn a representation that preserves the
different semantics in the input, even when one type of semantic information
can be a shortcut for contrastive learning.

Figure 2: Illustration of the IPCL framework on RGB data. The framework has
two branches: 1) a reconstructive branch, illustrated in the blue box, which
ensures that the representation has enough information to restore the missing
patches in the input, and 2) a contrastive branch, illustrated in the orange
box, which ensures that the representation keeps positive samples close to
each other and away from negative samples.

Our idea is to use the task of reconstructing missing patches in the input as
a regularization of contrastive learning to force it to learn multiple types
of semantics, even in the presence of a shortcut, and without requiring the
designer to know which information in the input may create a shortcut. By
reconstructing missing patches, we mean reconstructing randomly cropped
regions in the input, (e.g., image inpainting). This task has traditionally
been used as a pretext task in self-supervised learning. Here, however, we use
it as a regularization of contrastive learning.

We propose Information-Preserving Contrastive Learning (IPCL), which combines
the task of reconstructing missing patches with contrastive learning. In
Figure 2, we show how we apply IPCL to RGB data. Later in Section 5, we show
how we apply it to RF data. As clear from Figure 2, a IPCL model has two
branches, one is a common contrastive learning branch which uses random
augmentation on the same image to generate positive pairs, and maximizes the
similarity between their features (in comparison to the negative pairs).
Another is a patch reconstruction branch, which masks out random patches in
the input and forces the network to reconstruct the missing patches. The goal
of IPCLis to train a good encoder network that generates a representation that
simultaneously minimizes the contrastive loss and the patch reconstruction
loss.

(a) Reconstructive Branch: The reconstructive branch is illustrated in the
blue box in Figure 2. Given an input image $x$, we first randomly mask several
patches to get the masked input $M(x)$. Then the masked input is passed
through an encoder network $E$ with parameter $\theta$, and a decoder network
$D$, with parameter $\delta$, to obtain the reconstruction result
$D_{\delta}(E_{\theta}(M(x)))$. The reconstruction loss $\mathcal{L}_{r}$ is
defined as the reconstruction error between the original input $x$ and the
reconstructed one $D_{\delta}(E_{\theta}(M(x)))$:

$\mathcal{L}_{r}=||D_{\delta}(E_{\theta}(M(x)))-x||_{2}.$

The reconstruction branch forces the representation to preserve enough
information to reconstruct the missing patches.

(b) Contrastive Branch: The contrastive branch is illustrated in the orange
box in Figure 2. For each image, we first generate a pair of positive samples
by using two random augmentations $\tau_{1}$ and $\tau_{2}$, then we forward
the two augmented inputs separately into the encoder $E$ parameterized by
$\theta$ and a multi-layer nonlinear projection head $H$ parameterized by $h$
to get the latent representations $z_{1}$ and $z_{2}$ for these two positive
samples. We use the commonly used InfoNCE loss [3] as the contrastive loss
$\mathcal{L}_{c}$. Namely, for a batch of $N$ different input images
$x_{i},i=1,...,N$,

$\mathcal{L}_{c}=-\sum_{i=1}^{N}\log\sum\frac{\exp\big{(}\text{sim}(z_{2i},z_{2i+1})/t\big{)}}{\sum_{k=1}^{2N}\mathbbm{1}_{k\neq
2i}\exp\big{(}\text{sim}(z_{2i},z_{k})/t\big{)}},$

where $\text{sim}(u,v)=u^{T}v/(\|u\|_{2}\|v\|_{2})$ denotes the dot product
between the normalized $u$ and $v$ (i.e., cosine similarity),
$t\in\mathbb{R}^{+}$ is a scalar temperature parameter, and $z_{2i},z_{2i+1}$
are the encoded features of positive pairs generated from $x_{i}$, i.e.,
$z_{2i}=H_{h}(E_{\theta}(\tau_{1}(x_{i})))$ and
$z_{2i+1}=H_{h}(E_{\theta}(\tau_{2}(x_{i})))$.

(c) Training Procedure: We have empirically found that it is better to train
the model in two phases. In the first phase, only the reconstructive branch is
trained. In the second phase both branches are trained together. In this
latter case, the overall training loss is the combination of the
reconstruction loss and the contrastive loss, i.e.:

$\mathcal{L}=\lambda_{1}\cdot\mathcal{L}_{r}+\lambda_{2}\cdot\mathcal{L}_{c}$

This forces the network to learn fine-grained high-level semantics while
keeping the reconstruction loss as a regularization so that contrastive
learning will not focus only on some shortcut semantics and discard other
semantics.

##  4 Experiments on RGB datasets

We evaluate IPCL on two RGB datasets, and compare it to state-of-the-art
contrastive learning methods.

###  4.1 Digit & Background Object Classification

To better understand the impact of shortcuts on contrastive learning and IPCL,
we experiment with the Colorful-MNIST dataset proposed in [25]. The dataset is
constructed by assigning each digit from MNIST a background object image
selected randomly from the STL-10 dataset. Hence, each image in the Colorful-
MNIST dataset has two types of semantic information that correspond to the
digit class and the background object class. Figure 1 shows an anchor image
from this dataset and corresponding positive and negative samples.

Reconstructive Setting. For each input image with a size of $64$ by $64$
pixels, we randomly mask out $3$ to $5$ rectangle patches at random locations
in the image. The size of each square is chosen by setting its side randomly
between $10$ and $16$ pixels. These masked patches are filled with the average
pixel value of the dataset.

Contrastive Setting. For the contrastive branch in IPCL, similar to [3], we
apply random cropping and random color jittering on an image twice to generate
positive pairs. The negative pairs are sampled within a batch.

Network Structure. We use a 6-layer ConvNet for the encoder network $E$. The
encoder weights for the predictive and contrastive branches are shared. The
decoder network $D$ is a 6-layer deconvolutional network which is symmetric to
$E$. The projection head $H$ for contrastive learning is a 2-layer non-linear
head which embed the feature into a 64-dim normalized space.

Baselines. We use state-of-the-art contrastive learning methods as baselines,
including SimCLR [3], MoCo [11] and CPC [12] for unsupervised representation
learning from the Colorful-MNIST dataset. The same network structure is used
for all baselines and for IPCL’s contrastive branch. The hyper-parameters for
each baseline are fine-tuned to achieve its best performance. Other
implementation details are provided in Appendix.

Evaluation Metric. We evaluate the representations learned by IPCL and the
baselines by using them to train two linear classifiers for two downstream
tasks: digit classification and background object classification. We report
the error rate for each method.

####  4.1.1 Results

Table 1 reports the performance of the learned representation on two
downstream tasks: digit and background object classification. The table
compares various contrastive learning methods with IPCL. It also compares the
performance of these unsupervised representation learning methods with
supervised methods for learning representation, where the supervision is
applied on the digit, background object or both. All methods including the
supervised ones use the same encoder network architecture described above.

Method | digit cls. | bkgd cls.
---|---|---
| error rate (%) | error rate (%)
SimCLR | 85.1 | 51.5
MoCo | 84.3 | 53.7
CPC | 84.2 | 64.8
IPCL (ours) | 11.7 | 54.0
Supervised on digit | 3.9 | 88.6
Supervised on bkgd | 87.1 | 43.3
Supervised on digit & bkgd | 7.0 | 45.5
Table 1: We study the representation quality learned from Colorful-MNIST using
different unsupervised methods. We evaluate the learned representation on the
downstream tasks of digit classification and background object classification.

Table 1 reveals the following findings:

  * •

All contrastive learning methods fail to learn features that capture
information relevant to digit classification. In fact, their error rate on
digit classification (84% to 85%) is similar to the error rate of a supervised
method that was trained on background object classification with no
information about digits ($\sim$87%). In contrast, contrastive learning
methods perform reasonably well on background object classification with an
error rate about 10% higher than a supervised method trained on background
object classification. This indicates that the background object features
played the role of a shortcut for contrastive learning, i.e., those features
were sufficient to contrast positive and negative samples, and hence there was
no need for contrastive learning to learn digit-related features.

  * •

IPCL on the other hand learned a representation that is useful for both digit
and background object classifications. Its error rate on background object
classification is as good as the contrastive learning baselines. However, its
error rate on digit classification is $\sim$70% lower than the contrastive
baselines. IPCL was able to learn features related to digit classification
because of its reconstructive branch, which forced the representation to keep
enough information to reconstruct missing patches in the input.

###  4.2 Face Attribute Classification

In this section, we evaluate IPCL on FairFace [17], a real world human face
attribute dataset, where each image contains multiple semantics including
gender, age, and ethnicity, as shown in Figure 3.

Figure 3: Illustration of FairFace dataset. The positive pairs are from random
cropping and random color jittering on an image twice. The negative pairs are
sampled within batch.

####  4.2.1 Experiment Setup

Reconstructive Setting. We use the strategy in [21] to mask out $3$ to $5$
rectangles with a random sidesize between $7$ to $12$ pixels at random
locations in the image. We fill the masked rectangles with the mean pixel
value.

Contrastive Setting. For contrastive learning, we follow the strategy in [3]
to sample positive and negative pairs. We apply random cropping and random
color jittering on an image twice to generate positive pairs. The negative
pairs are sampled within batch.

Network Structure. We use a standard ResNet-18 for the encoder network $E$.
The decoder network $D$ is a symmetric deconvolutional network. The projection
head $H$ for contrastive learning is a 2-layer non-linear head which embed the
feature into a 128-dimensional unit sphere.

Evaluation Metric. We evaluate the learned representations on three downstream
tasks: age, gender and ethnicity classification. For the training of the
downstream tasks, we freeze the encoder and train a linear layer for each task
to classify the features. For the supervised baselines, supervised (a), (g)
and (e) are trained on the single task of age, gender and ethnicity,
respectively, while supervised (all) is jointly trained on all the three
tasks.

Baselines. We compare IPCL with SimCLR. For SimCLR, we use the same sampling
strategy and network structure as our IPCL. We also fine-tune the hyper-
parameters of SimCLR to achieve its best performance.

####  4.2.2 Results

Table 2 compares the performance of state-of-the-art contrastive learning
(SimCLR) and IPCL on the FairFace dataset. As shown by the supervised
baselines, the three tasks share some common features since network trained on
one of the task can also be used to classify other tasks with reasonable
accuracy. However, each task also has its unique features since there are
still big performance gaps between network trained on the specific task of
interest and networks trained on the other two tasks. The table also shows
that IPCL’s performances on age and gender are significantly better than
SimCLR. In fact, SimCLR’s performance on age and gender are similar to that of
supervised learning on ethnicity alone ($\sim$56% and $\sim$21% error rate,
respectively). IPCL improves over SimCLR’s performance on age and gender
classification by 5.7% and 8.6% respectively, while keeping comparable
performance to SimCLR on ethnicity classification (38.8% vs. 37.7%). This
further demonstrates IPCL’s ability to preserve useful information that could
be ignored by contrastive learning.

Method |  | age cls.
---
error rate (%)
| gender cls.
---
error rate (%)
| ethnicity cls.
---
error rate (%)
SimCLR | 55.7 | 21.4 | 37.7
IPCL (ours) | 50.0 | 12.8 | 38.8
Supervised (a) | 44.5 | 21.3 | 54.9
Supervised (g) | 56.7 | 7.5 | 54.6
Supervised (e) | 57.9 | 23.2 | 28.6
Supervised (all) | 45.2 | 8.1 | 29.2
Table 2: We study the representation quality learnt using different methods.
We evaluate the learned representation on downstream tasks of age, gender and
ethnicity classification.

##  5 Learning Unsupervised Representations from RF Signals

Radio Frequency (RF) signals have been studied a lot recently for pose
estimation, action recognition, and person re-identification [31, 33, 18, 6,
29, 8, 32, 1, 26, 24, 30, 34, 15, 7], mainly due to its good properties that
it can traverse occlusions and are insensitive to visible light. Therefore,
these tasks can be performed through walls and in dark settings where RGB-
based cameras fail. However, past works on these tasks are based on supervised
learning using labeled RF datasets. Since RF signals are not interpretable by
a human, the annotation is collected by deploying a camera system and
synchronizing it with the RF device, which is labor-intensive and hard to
scale. If we can learn useful representation from unlabeled RF signals, we can
reduce overhead and improve the accuracy of the downstream tasks.

Before delving into the details of our unsupervised representation learning,
we first explain the typical neural models used in these tasks. For all of
these tasks, the input is a time sequence of RF frames. Each RF frame is a
pair of 2D heatmaps: one horizontal heatmap and one vertical heatmap, as
illustrated in Figure 4. The horizontal heatmap is a projection of the radio
signals on a plane parallel to the ground, and the vertical heatmap is a
projection of the radio signals on a plane perpendicular to the ground. There
are typically 30 pairs of heatmaps per second.

Figure 4: Illustration of RF signals as a pair of horizontal and vertical
heatmaps, and an RGB image recorded at the same time. Red color refers to high
signal power and blue refers to low power. Figure 5: RF-based human sensing
framework. The RF heatmaps are first fed into a Feature Network to extract
spatio-temporal features, and ROI Align with the given bounding box is used to
crop out the regions with humans. Each downstream task has a unique sub-
network to extract task-specific features and predictions. Figure 6: IPCL on
RF signals. It has two branches: Reconstructive Branch and Contrastive Branch.
For the reconstructive branch, we randomly mask some frames across time for
reconstruction. For the contrastive branch, different transformation
$\tau_{1}$ and $\tau_{2}$ to the bounding box are adopted to generate positive
samples. Both branches share the same feature extraction network (Green Box),
and then use ROI Align with the given bounding box to crop out regions
containing humans. The reconstructive branch has a decoder to reconstruct
inputs, while contrastive branch has a non-linear projection head to generate
normalized features. The combination of reconstruction loss and contrastive
loss is used to train the model.

The neural models used in all the three tasks (i.e., pose estimation, action
recognition, and person re-identification) have the general structure in
Figure 5. The model first uses a spatio-temporal convolutional feature network
to extract features from the input RF frames. It then crops out a bounding box
around the person in the feature map. Finally, the cropped features are fed to
a task-specific sub-network to generate frame-level features for downstream
tasks. The general structure is the same for all tasks, but different tasks
can have different sub-networks specifically designed for that task. The goal
of RF-based unsupervised representation learning is to learn a feature
extraction network that can extract both informative and fine-grained features
from input RF frames for different downstream tasks.

###  5.1 Adapting IPCL to RF Representations

Reconstructive Setting. Each sample is a sequence of 100 RF frames. We
randomly mask out 25 frames within this sequence and fill them with zero for
later reconstruction.

Contrastive Settings. As shown in Figure 6, the anchor and positive sample
share the same input RF frames with different random jitter of the bounding
boxes. The jitter value is randomly chosen in $[-30cm,30cm]$. Negative samples
of an anchor are all other samples in the batch except for itself and its
positive sample. All samples in a batch are randomly sampled from the same
environment.

Training Pipeline. During the unsupervised learning phase, we employ a spatio-
temporal RF feature extraction network to encode the input RF frames. Then an
ROI Align is applied on top of the extracted features to crop out the region
around the person, where the region of interest bounding boxes come from the
WiTrack localization algorithm [2]. For contrastive learning, a non-linear
projection head consisting of 3 convolutional residual block and 2 fully-
connected layers is added on top of the cropped feature maps to project the
features and apply the contrastive loss $l_{P}$ between features from positive
and negative samples. For the reconstruction task, a decoder is added to
reconstruct the masked RF frames, and a per-pixel reconstruction loss $l_{C}$
is added between the reconstructed RF and the ground truth RF frames. For
IPCL, we combine these two losses together as $\lambda_{1}\cdot
l_{P}+\lambda_{2}\cdot l_{C}$. We set $\lambda_{1}=100$ and $\lambda_{2}=1$.

###  5.2 Experimental Setup

Datasets. We pre-train IPCL on two RF datasets: RF-MMD [18] and RRD [7]. RF-
MMD is a RF-based pose estimation and action recognition dataset consisting of
25 hours of RF data with 30 volunteers performing 35 actions in 10 different
indoor environments. RRD is an RF-based person re-identification dataset. It
contains 863 RF sequences from 100 different identities at 5 different
locations on a university campus, and 6305 RF sequences from 38 different
identities in 19 homes.

Evaluation Metric. After pre-training using IPCL, we evaluate the quality of
the learned representation by applying the learned RF feature extraction
network to three downstream RF-based tasks: 3D pose estimation, action
recognition, and person re-identification. For 3D pose estimation, we use the
average $l_{2}$ distance between 14 predicted keypoints and their ground-truth
values, as evaluation metric. For action recognition, we use mean average
precision (mAP) at different intersection-over-union (IoU) thresholds $\theta$
to evaluate the accuracy of the model in detecting and classifying an action
event. For person re-identification, we first separate the dataset into query
set and gallery set randomly. The query samples and gallery samples are then
encoded to feature vectors using our model. Then we calculate the cosine
distance between each query sample’s features and each gallery sample’s
features and rank the distance to retrieve the top-N closest gallery samples
for each query sample. We compute the standard evaluation metrics for person
ReID based on the ranking results: mean average precision score (mAP) and the
cumulative matching curve (CMC) at rank-1 and rank-5.

Fine-tuning. In the downstream training phase, we use the pre-trained RF
feature extraction network as initialization of the RF feature extraction
network for each downstream task, and fine-tune the model for each task on
downstream datasets. We use the pre-trained model on RF-MMD for downstream
tasks of pose estimation and action recognition, and use the pre-trained model
on RRD for person re-identification. We also train a supervised baseline from
scratch on the same data for each task to compare with the results of
unsupervised learning.

###  5.3 Downstream Evaluation Results

We evaluate the RF feature extraction network learned by IPCL on multiple
downstream RF-based tasks. The results are shown in Table 9, 4, 5, 6. As shown
in the results, pre-training the feature network using IPCL and fine-tuning it
on the downstream tasks generates the best results in all tasks. Overall,
compared to its supervised counterparts, IPCL has 8.6% relative improvement in
3D pose estimation error, 3.7% relative improvement in action recognition mAP
at $\theta=0.1$, 2.2% relative improvement in person ReID mAP on RRD-Campus
and 3.6% relative improvement on RRD-Home.

Method | Skeleton Err. (cm)
---|---
Train from Scratch | 3.84
Contrastive+finetune | 3.88
Reconstructive+finetune | 3.62
IPCL+finetune | 3.51
Table 3: 3D pose estimation performance of different models on RF-MMD in skeleton error (the lower the better). Methods | mAP
---|---
$\theta=0.1$ | $\theta=0.5$
Train from Scratch | 90.1 | 87.8
Contrastive+finetune | 89.8 | 87.4
Reconstructive+finetune | 91.7 | 88.7
IPCL+finetune | 92.5 | 89.6
Table 4: Action recognition performance of different models on RF-MMD in mAP at different IoU thresholds (the higher the better). Methods | mAP | CMC-1 | CMC-5
---|---|---|---
Train from Scratch | 59.5 | 82.1 | 95.5
Contrastive+finetune | 59.0 | 81.7 | 94.1
Reconstructive+finetune | 60.1 | 83.3 | 95.5
IPCL+finetune | 60.8 | 84.2 | 96.0
Table 5: Person re-identification performance of different models on RRD-Campus in mAP and CMC (the higher the better). Methods | mAP | CMC-1 | CMC-5
---|---|---|---
Train from Scratch | 46.4 | 74.6 | 89.5
Contrastive+finetune | 45.9 | 73.8 | 88.5
Reconstructive+finetune | 47.5 | 75.3 | 90.3
IPCL+finetune | 48.1 | 76.1 | 91.0
Table 6: Person re-identification performance of different models on RRD-Home
in mAP and CMC (the higher the better).

The tables also show that the feature network pre-trained using contrastive
learning alone generates worse results than random initialization. This is
likely due to that contrastive learning learns shortcut semantics irrelevant
to human sensing, and thus is harmful to human-related downstream tasks. Using
the reconstruction task alone without the contrastive loss can generate better
results than training from scratch, but the performance is worse than that of
IPCL. This is likely because inptaining alone tend to learn low-level features
too close to the pixels in the RF heatmaps, and fail to learn abstract
features about the human body. In contrast, IPCL is able to utilize the
combination of contrastive learning and inpainting loss to learn abstract and
information-preserving features that are useful for downstream tasks. This
demonstrates that the RF feature extraction network learned by IPCL can
extract representations that contain multiple types of semantics related to
various downstream tasks.

##  6 Conclusion

In this paper, we introduce information-preserving contrastive learning, a
novel framework to keep unsupervised contrastive learning effective and
preserve useful information in the presence of shortcuts. We apply our
framework to RGB datasets to show its ability to solve the problem of
shortcuts. We also adapt our framework to RF data to demonstrate the
possibility of boosting performance of RF-based human sensing tasks with
unsupervised learning. Extensive experiments on both RGB and RF datasets
demonstrate the effectiveness of our proposed framework.

## References

  * [1] Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, and Frédo Durand.  Capturing the human figure through a wall.  ACM Transactions on Graphics (TOG), 34(6):1–13, 2015.
  * [2] Fadel Adib, Zach Kabelac, Dina Katabi, and Robert C Miller.  3d tracking via body radio reflections.  In 11th $\\{$USENIX$\\}$ Symposium on Networked Systems Design and Implementation ($\\{$NSDI$\\}$ 14), pages 317–329, 2014.
  * [3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  arXiv preprint arXiv:2002.05709, 2020.
  * [4] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton.  Big self-supervised models are strong semi-supervised learners.  Advances in Neural Information Processing Systems, 33, 2020.
  * [5] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [6] Kevin Chetty, Qingchao Chen, Matthew Ritchie, and Karl Woodbridge.  A low-cost through-the-wall fmcw radar for stand-off operation and activity detection.  In Radar Sensor Technology XXI, volume 10188, page 1018808. International Society for Optics and Photonics, 2017.
  * [7] Lijie Fan, Tianhong Li, Rongyao Fang, Rumen Hristov, Yuan Yuan, and Dina Katabi.  Learning longterm representations for person re-identification using radio signals.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10699–10709, 2020.
  * [8] Lijie Fan, Tianhong Li, Yuan Yuan, and Dina Katabi.  In-home daily-life captioning using radio signals.  arXiv preprint arXiv:2008.10966, 2020.
  * [9] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  arXiv preprint arXiv:1803.07728, 2018.
  * [10] Michael Gutmann and Aapo Hyvärinen.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297–304, 2010.
  * [11] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [12] Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord.  Data-efficient image recognition with contrastive predictive coding.  arXiv preprint arXiv:1905.09272, 2019.
  * [13] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio.  Learning deep representations by mutual information estimation and maximization.  arXiv preprint arXiv:1808.06670, 2018.
  * [14] Chen-Yu Hsu, Aayush Ahuja, Shichao Yue, Rumen Hristov, Zachary Kabelac, and Dina Katabi.  Zero-effort in-home sleep and insomnia monitoring using radio signals.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(3):1–18, 2017.
  * [15] Chen-Yu Hsu, Rumen Hristov, Guang-He Lee, Mingmin Zhao, and Dina Katabi.  Enabling identification and behavioral sensing in homes using radio reflections.  In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–13, 2019.
  * [16] Xu Ji, João F Henriques, and Andrea Vedaldi.  Invariant information clustering for unsupervised image classification and segmentation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 9865–9874, 2019.
  * [17] Kimmo Kärkkäinen and Jungseock Joo.  Fairface: Face attribute dataset for balanced race, gender, and age.  arXiv preprint arXiv:1908.04913, 2019.
  * [18] Tianhong Li, Lijie Fan, Mingmin Zhao, Yingcheng Liu, and Dina Katabi.  Making the invisible visible: Action recognition through walls and occlusions.  In Proceedings of the IEEE International Conference on Computer Vision, pages 872–881, 2019.
  * [19] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In European Conference on Computer Vision, pages 69–84. Springer, 2016.
  * [20] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [21] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [22] Kihyuk Sohn.  Improved deep metric learning with multi-class n-pair loss objective.  In Advances in neural information processing systems, pages 1857–1865, 2016.
  * [23] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  arXiv preprint arXiv:1906.05849, 2019.
  * [24] Yonglong Tian, Guang-He Lee, Hao He, Chen-Yu Hsu, and Dina Katabi.  Rf-based fall monitoring using convolutional neural networks.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(3):1–24, 2018.
  * [25] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.  What makes for good views for contrastive learning.  arXiv preprint arXiv:2005.10243, 2020.
  * [26] Deepak Vasisht, Anubhav Jain, Chen-Yu Hsu, Zachary Kabelac, and Dina Katabi.  Duet: Estimating user position and identity in smart homes using intermittent and incomplete rf-data.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(2):1–21, 2018.
  * [27] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733–3742, 2018.
  * [28] Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang.  Unsupervised embedding learning via invariant and spreading instance feature.  In Proceedings of the IEEE Conference on computer vision and pattern recognition, pages 6210–6219, 2019.
  * [29] Zhenyuan Zhang, Zengshan Tian, and Mu Zhou.  Latern: Dynamic continuous hand gesture recognition using fmcw radar sensor.  IEEE Sensors Journal, 18(8):3278–3289, 2018.
  * [30] Mingmin Zhao, Fadel Adib, and Dina Katabi.  Emotion recognition using wireless signals.  In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking, pages 95–108, 2016.
  * [31] Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, and Dina Katabi.  Through-wall human pose estimation using radio signals.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7356–7365, 2018.
  * [32] Mingmin Zhao, Yingcheng Liu, Aniruddh Raghu, Tianhong Li, Hang Zhao, Antonio Torralba, and Dina Katabi.  Through-wall human mesh recovery using radio signals.  In Proceedings of the IEEE International Conference on Computer Vision, pages 10113–10122, 2019.
  * [33] Mingmin Zhao, Yonglong Tian, Hang Zhao, Mohammad Abu Alsheikh, Tianhong Li, Rumen Hristov, Zachary Kabelac, Dina Katabi, and Antonio Torralba.  Rf-based 3d skeletons.  In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, pages 267–281, 2018.
  * [34] Mingmin Zhao, Shichao Yue, Dina Katabi, Tommi S Jaakkola, and Matt T Bianchi.  Learning sleep stages from radio signals: A conditional adversarial architecture.  In International Conference on Machine Learning, pages 4100–4109, 2017.

## Appendix A: Implementation Details

In this section, we provide the implementation details of the models used in
our experiments. All experiments are performed on 8 NVIDIA Titan X Pascal
GPUs. We note that the parameters for each baseline are chosen to optimize for
its best performance.

Digit & Background Object Classification: For the baseline approaches (SimCLR,
MoCo and CPC), we use the SGD optimizer with 0.1 learning rate, 1e-4 weight
decay, and 0.9 momentum to train the model for 200 epochs. We note that we
tried both Adam and SGD, and picked SGD since it performed better, which is
consistent with prior work [3,11]. The learning rate is scaled with a factor
of 0.1 at epoch 150 and 175. The batch size is set to 512. The temperature for
contrastive loss is set to 0.1. For IPCL, the network is trained for 200
epochs with the Adam optimizer. We also tried both SGD and Adam and found that
Adam is better for training IPCL. This may be because the Adam optimizer is
better for optimizing the reconstructive loss as reported in [21]. The
learning rate is set to 1e-3. The first 50 epochs are warm-up epochs, where we
only train the network with the reconstruction loss $\mathcal{L}_{r}$.

Face Attribute Classification: For the SimCLR baseline, we train the network
for 1000 epochs using the SGD optimizer with 0.1 learning rate, 1e-4 weight
decay and 0.9 momentum. The learning rate is scaled with a factor of 0.2 at
epoch 450, 600 and 750. The batch size is set to 512. The temperature for
contrastive loss is set to 0.1. For IPCL, the network is trained for 500
epochs, using the Adam optimizer with 1e-3 learning rate. The first 100 epochs
are warm-up epochs, where we only train the network with the reconstruction
loss $\mathcal{L}_{r}$.

RF Experiments: For the baseline contrastive approach, we train the network
for 50 epochs, using the SGD optimizer with 0.01 learning rate, 1e-5 weight
decay and 0.9 momentum. For IPCL, we train the network using the Adam
optimizer with 1e-3 learning rate and 1e-5 weight decay. The network is
trained for 50 epochs with the first 10 epochs as warm-up epochs trained with
only the reconstruction loss.

## Appendix B: Ablation Study on RGB data

We perform extensive ablation studies to better understand the good practices
for applying IPCL to RGB data. The RGB ablation studies are performed on
Colorful-MNIST dataset with the downstream task of digit and background object
classification.

Cropping Strategy: We first compare the performance of IPCL with different
cropping/masking strategies (recall that cropping is used in the
reconstruction task in IPCL). We compare 3 different strategies: center region
(25x25 patch in the center of the input image), random region (25x25 patch at
a random position in the input image), and random patches (3 to 5 rectangle
patches at random locations in the image. The size of each rectangle is chosen
by setting its side randomly between 10 and 16 pixels.). As shown in the
results below, the random-patches strategy performs the best.

Cropping Strategy | Digit cls. err. (%) | Bkgd cls. err. (%)
---|---|---
Center Region | 25.5 | 58.1
Random Region | 20.1 | 56.2
Random Patches | 11.7 | 54.0
Table 7: Classification error rate with different image cropping strategies.

Size of Cropped Patches: Next, We study the influence of the sidesize of each
masked patch. Note that when the size is equal to 0, no patches are cropped,
and the reconstruction branch becomes a normal auto-encoder. In all of these
experiments, we follow the strategy of randomly selecting 3 to 5 rectangle
patches of fixed sidesize. As shown in Figure 7, the best performance on digit
classification is to apply patches with sidesize of 10 pixels and the best
performance on background object classification is to apply patches with
sidesize of 16 pixels. Therefore, we finally choose the size of each rectangle
by setting its side randomly between 10 and 16 pixels to optimize for both
classification task.

Figure 7: Classification error rate with different sizes of masked patches.

Warm-up Training: As explained in the paper, IPCL has a warm-up training phase
in which it uses only the reconstruction loss. To show the effectiveness of
the proposed warm-up training strategy, we compare the results of warm-up
training with the results of directly using the combined loss $\mathcal{L}$
(i.e., combining the reconstruction loss and the contrastive loss) from the
beginning:

Warm-up Training | Digit cls. err. (%) | Bkgd cls. err. (%)
---|---|---
Yes | 11.7 | 54.0
No | 75.1 | 52.8
Table 8: Classification error rate with and without warm-up training.

As shown in Table 8, without the warm-up training phase, IPCL largely
degenerates to become similar to the contrastive learning baselines and cannot
learn good features related to digit classification. This indicates that
without the warm-up phase, the contrastive loss can dominate the network
causing it to only learn the shortcut of background object classification at
the beginning, and that the network cannot later jump out of the local minimum
associated with the shortcut. On the other hand, with warm-up training, the
network first learns a coarse representation useful for both digit and
background classification; then the contrastive loss helps the network learn
more fine-grained representations.

## Appendix C: Ablation Study on RF data

We also perform ablation studies on applying IPCL to real-world RF datasets.
All ablation studies are performed on RF-MMD [18] dataset with the downstream
task of pose estimation, where we report the average error across all
keypoints.

Masking Strategy: When applied to RF signals, IPCL’s reconstruction task is
performed by masking some frames in a short window RF signals, and
reconstructing them from the remaining frames. We compare the performance of
IPCL with different masking strategies. The input RF signal is a 100-frames RF
sequence. We compare 3 different masking strategies: center segment (25 frames
in the center of the input), random segments (five 5-frames segments at random
positions of input), and random (each frame is masked with 0.25 probability).
As shown in the results below, the random segments masking strategy performs
the best.

Mask Strategy | Skeleton Error (cm)
---|---
Center Segment | 3.58
Random Segments | 3.51
Random | 3.53
Table 9: Skeleton error (the lower the better) with different masking
strategies.

Size of Masked Segment: Next, We study the influence of the size of each
masked segment. Note that when the mask size equals to 0, no frame is masked,
and the reconstruction branch becomes a normal auto-encoder. In all of these
experiments, we mask 5 segments and follow the strategy of randomly locating
the masked segments in the input. As shown in Table 10, the best performance
is for masking 5 consecutive frames, i.e., a segment of size 5.

Masking Size | 0 | 1 | 3 | 5 | 10
---|---|---|---|---|---
Skeleton Err. (cm) | 3.68 | 3.60 | 3.52 | 3.51 | 3.60
Table 10: Skeleton error with different sizes of masked segments.

Warm-up Training: To show the effectiveness of the proposed warm-up training
strategy on RF signals, we compare the results of warm-up training with the
results of directly using the combined loss $\mathcal{L}$ (i.e., combining the
reconstruction loss and the contrastive loss) from the beginning:

Warm-up Training | Skeleton Error (cm)
---|---
Yes | 3.51
No | 3.83
Table 11: Skeleton error (the lower the better) with and without warm-up
training.

As shown in Table 11, without the warm-up training, IPCL performs similar to
just applying the contrastive loss without adding the reconstructive loss
(3.88 cm skeleton error as reported in Table 3 of the paper). This indicates
that without the warm-up phase, the contrastive loss can dominate the network
causing it to learn the shortcut at the beginning, and that the network cannot
later jump out of the local minimum associated with the shortcut. On the other
hand, with warm-up training, the network first learns a coarse representation
of the RF signals; then the contrastive loss helps the network learn more
fine-grained representations.

Benefits of Unlabeled Data: To show the effectiveness of the proposed method
in utilizing large unlabeled RF dataset, we simulate the scenario where only a
handful of labeled RF data is available and a large amount of RF data is
unlabeled data. Specifically, we randomly select 10% of the training set of
RF-MMD to be RF-MMD-L to serves as the small labeled dataset. We compare the
performance of IPCL when it is pretrained on RF-MMD-L and RF-MMD and finetuned
on RF-MMD-L.

Methods | Skeleton Error (cm)
---|---
Training from scratch (RF-MMD-L) | 4.87
IPCL on RF-MMD-L+finetune | 4.50
IPCL on RF-MMD+finetune | 4.21
Table 12: Skeleton error (the lower the better) with different IPCL training
dataset.

As shown in 12, IPCL can improve the performance of RF-based pose estimation
without using any additional unlabeled data. This is because IPCL can learn a
general representation of RF signals and thus provide better generalization
ability. Indeed, IPCL improves the skeleton error by 0.29 cm by leveraging the
unlabeled data. This demonstrates the potential of using IPCL to leverage
large-scale unlabeled RF data to improve the performance of RF-based human
sensing methods.