# ViFiCon: Vision and Wireless Association Via Self-Supervised Contrastive
Learning

Nicholas Meegan
Rutgers University
<EMAIL_ADDRESS>Hansi Liu
Rutgers University
<EMAIL_ADDRESS>Bryan Bo Cao
Stony Brook University
<EMAIL_ADDRESS>Abrar Alali
Old Dominion University
<EMAIL_ADDRESS>Kristin Dana
Rutgers University
<EMAIL_ADDRESS>Marco Gruteser
Rutgers University
<EMAIL_ADDRESS>Shubham Jain
Stony Brook University
<EMAIL_ADDRESS>Ashwin Ashok
Georgia State University
<EMAIL_ADDRESS>
###### Abstract

We introduce ViFiCon, a self-supervised contrastive learning scheme which uses
synchronized information across vision and wireless modalities to perform
cross-modal association. Specifically, the system uses pedestrian data
collected from RGB-D camera footage as well as WiFi Fine Time Measurements
(FTM) from a user’s smartphone device. We represent the temporal sequence by
stacking multi-person depth data spatially within a banded image. Depth data
from RGB-D (vision domain) is inherently linked with an observable pedestrian,
but FTM data (wireless domain) is associated only to a smartphone on the
network. To formulate the cross-modal association problem as self-supervised,
the network learns a scene-wide synchronization of the two modalities as a
pretext task, and then uses that learned representation for the downstream
task of associating individual bounding boxes to specific smartphones, i.e.
associating vision and wireless information. We use a pre-trained region
proposal model on the camera footage and then feed the extrapolated bounding
box information into a dual-branch convolutional neural network along with the
FTM data. We show that compared to fully supervised SoTA models, ViFiCon
achieves high performance vision-to-wireless association, finding which
bounding box corresponds to which smartphone device, without hand-labeled
association examples for training data.

##  1 Introduction

Figure 1:  Associating visual observations to Wi-Fi signals enables novel
applications such as emergency messaging since observed users can be linked to
their smart phones. Our approach to learning vision-to-wireless associations
does not require annotated ground-truth matches. Instead, a pretext task of
temporal synchronization guides multimodal representation learning for the
downstream task of associating individual bounding boxes to specific
smartphones.

Cross-domain associations are becoming more feasible in real-world
environments with the addition of sensors from multiple sensing modalities.
Applications that observe and communicate to participating pedestrians can
provide emergency alerts and other messaging tuned to a pedestrian’s position
in the scene. Consequently, two domains which greatly benefit from joint use
is the vision domain (e.g. RGB-D images), and the wireless domain, (e.g.
estimated distance from a WiFi access point through FTM, Fine Time
Measurements). By identifying and associating individuals in the camera, even
in scenarios where the person’s face is occluded, it is possible to send alert
messages to the user’s device when an emergency occurs. Connecting a
pedestrian’s vision data to a wireless identifier such as a smartphone ID
necessitates an opt-in requirement for the messaging application.

Existing approaches associating information across modalities use information
such as camera and laser ranging [26, 23], or clothing color and motion
patterns [28]. Among multimodal methods for vision and wireless, [4] fuses
camera and received signal strength (RSS) data, and [29, 9] fuse camera and
WiFi channel state information (CSI). These prior vision and wireless methods
have limitations such as requiring multiple WiFi access points (AP). Most
similar to our work are Vi-Fi [17] and ViTag [5], which use the vision
modality, (camera and depth data) along with the WiFi modality (FTM and
inertial measurement unit IMU), to perform cross-modal association using a
single AP. However, in our work we compare and use fewer features, using only
depth data and FTM information for creating the multimodal association, while
using the camera data along with off-the-shelf detected bounding boxes to
extract depth. Moreover, we make no use of hand-labeled data unlike Vi-Fi and
ViTag.

In this paper, we present ViFiCon, a self-supervised contrastive learning
model to make associations between camera and wireless modalities without
hand-crafted labeled or ground truth data. Hand-labelling datasets (providing
ground truth associations between vision and wireless) is expensive and time-
consuming, and so instead, we create multimodal associations without using
hand-labeled data. We leverage bounding boxes for person detection using an
off-the-shelf object detection model [1] to obtain combinations of pedestrian
depths from the vision modality, without knowledge of which pedestrians are
contributing FTM data. We then construct positive and negative pairings of
these depth combinations from the vision domain and the known FTM distances
from the wireless domain based on passively collected timestamp information,
as inspired by the time contrastive audio and video self-supervised
synchronization task proposed in [14]. To bring the two modalities into a
joint representation, we create a novel band image representation that maps
the signals into a sequence of gray-scale bands. Not only can we represent a
single signal for each modality, but we can combine multiple of these signals
in a singular image to learn both a scene-wide temporal synchronization of the
data as well as a downstream signal-to-signal association for the depth and
FTM data. As the vision domain uses unlabeled bounding boxes, this scene-wide
synchronization allows the representation to know how many signals should be
considered in a single image representation by considering the fixed sized FTM
image representation. That is, the number of smartphones is known and
constrains the number of relevant bounding box depth signals. We then train a
siamese convolutional neural network model on the scene-wide synchronization
task to embed the positive and negative pairings into a joint latent space
with a Euclidean distance-based contrastive loss. When trained on the pseudo-
labeled data, we can then apply the task downstream without any more training
on the individual association task. We show the motivation of ViFiCon in
Figure 1.

Summary of Contributions We summarize our contributions as follows:

• We generate a novel representation of signal data, which represents a group
of signals from two modalities as a set of gray-scale bands.

• We devise a self-supervised learning framework to learn a multimodal latent
space representation of signal data without the use of hand-labelling.

• We demonstrate the strength of the convolutional neural network in
generalizing a global scene view of signal data in a pretext synchronization
task to a one-to-one individual association downstream task without further
training, yielding an 84.77% Identity Precision on a one-to-one association
with a 10-frame temporal window view.

##  2 Background and Related Work

Multimodal Association Multimodal association attempts to enhance a deep
learning model’s performance by introducing a more robust context of a scene
through shared knowledge between domains. Traditionally, these modalities are
fused together by mapping the data from the two into a shared latent space
representation, where vector representations of the data can be directly
compared with one another. Multimodal association has been applied to learning
information between audio and video domains to learn an association between
the two, such as whether or not an input video of an instrument corresponds
audio samples [14] or matching a video of lip movements to mel-spectrogram
audio representations [7]. Other work utilize the notion of multimodal deep
learning for fusing audio and text for sentiment recognition [12] and video
and wireless sensors for human activity recognition [29] or tracking [21, 22].

We specifically build off of work associating vision domain information (such
as RGB or depth data) with wireless domain sensor data such as WiFi Fine Time
Measurements (FTM) or inertial measurement unit (IMU). Because such signals
are able to overcome obstructions such as walls [3], we can make use of the
WiFi modality along with the vision domain to re-identify or track individuals
behind obstructions, not possible with vision alone. RGB-W [4] leverages
captured video data and received signal strength (RSS) data from cell phones’
WiFi or Bluetooth in the scene to associate detected bounding boxes with cell
phone MAC addresses. Despite working indoors, an improvement over previous
work which use GPS [24], the setup requires multiple WiFi antenna responses to
track individuals. WiVi [29] uses WiFi channel state information (CSI) along
with a vision sensing module to first extract and classify features from each
modality and then fuse the outputs together using a deep neural network to
classify activity. However, significant performance degradation occurs under
poor lighting conditions on the vision side, and objects interfering with WiFi
bands yields a decreased performance on the wireless side. Eye-Fi also makes
use of CSI but instead calculates angle of arrival for associating vision and
wireless data by matching the modalities, though also requires multiple APs
for use [9]. Other techniques use a fusion of FTM, IMU, and vision information
to perform association and tracking between domains while using only one WiFi
access point and an affinity matrix [17, 18] or X-translator network [5]. We
simplify the multimodal association task by utilizing two similar signals in
the vision and wireless domains (depth from RGB-D camera and estimated
distance from FTM) along with simplify the model architecture, only requiring
a set of convolutional neural networks (CNNs).

Self-Supervised Contrastive Learning Due to the expensive nature of hand
labeling datasets, self-supervised learning is introduced to learn on
unlabeled datasets with a set of pseudo-labels to cue in on similarities
between pairs of training samples. In self-supervised contrastive learning,
the pseudo-labels are made up of positive and negative pairing samples, where
positive correspond to the same class and negative correspond to different
classes, and are embedded into a shared latent space. The vector
representation of the classes in the space will embed positives close to one
another distance-wise, and negatives far away. A pioneering method in
contrastive learning is SimCLR [6], which defines the structure that inspired
our model design.

Prior methods of wireless and vision multimodal matching such as Vi-Fi and
ViTag have been fully-supervised, making use of labeled datasets such as hand-
crafted bounding boxes or ground truth labels, and as far as we know a self-
supervised contrastive learning has not been applied between extracted depth
information and FTM data. Prior work has explored a self-supervised
contrasting of signal data, such as through the use of RF signal
representations from video data to perform activity recognition [16] or
determining the similarity between two audio sources through passing in raw
audio data [20]. Unlike prior approaches, we represent multiple signals in an
image representation to perform contrastive learning and create a latent space
representation for both modalities.

WiFi Fine Time Measurements (FTM) The WiFi Fine Time Measurements (FTM)
protocol introduced in the IEEE 02.11-2016 Standard (802.11REVmc) [2, 19]
allows for distance with a margin of error to be collected indoor and outdoor
by performing wireless ranging from a WiFi connected device and a WiFi AP. To
obtain the distance, round trip time is calculated between the AP and the
connected device, with the range then estimated using propagation speed to
bring meter-level precision [13].

##  3 Data Pre-Processing and Representation

###  3.1 Dataset Selection and Pre-Processing

We make use of the Vi-Fi multimodal dataset [8] to create a self-supervised
representation of the data. The Vi-Fi dataset consists of collected and
labeled walking data across 98 different sequences over 5 scenes with
pedestrians (people not actively participating in the data collection process)
and participants (people holding smartphone devices active in the collection
process). The dataset contains information across the vision and wireless
modalities, including RGB-D data, 9-axis IMU data, and FTM data. Vision data
is collected through a ZED [1] RGB-D camera, whereas the IMU and FTM data is
collected by users holding Google Pixel-3A smartphones [10] communicating with
a Wi-Fi access point located beneath the camera such that the collected
distance information between modalities can be aligned. We use a subset of
scenes from the dataset, specifically from Dataset B in outdoor scenarios
including 3 participants and a number of background pedestrians. As the
representation in this work only requires person object detection for
extracting depth, the results generalize to indoor scenarios.

We use only the collected RGB-D data and FTM data for our work, unlike
previous state-of-the-art work which uses RGB-D data, FTM, and IMU [17, 5]. In
pre-processing, we remove any invalid frames, corresponding to timestamps at
which FTM data was not collected in the scenes with respect to the camera view
based on a file included with the dataset outlining the start and end of the
multimodal data collection. We sub-sample the RGB-D data and use every 4th
frame to better capture the movement of pedestrians and participants in the
scene. To extract depth information for the band image representation, an off-
the-shelf ZED SDK [1] object detector is used to collect bounding boxes of the
people in the sequences. No labeled identifiers or ground truth data are used
in the representation, and so the model cannot distinguish between who is
collecting wireless data and who is not.

Cross-Modal Temporal Alignment Between Camera and WiFi-FTM Due to the varying
sampling rates between the RGB-D and FTM data, we pre-process the data such
that a cross-modal temporal alignment can be achieved. The RGB-D data is
recorded at 10 frames per second, and the FTM data is collected at a rate of
3Hz. We extract the $t$ nearest timestamps larger and smaller timestamps from
the FTM data corresponding to valid RGB-D timestamps and perform a linear
interpolation to align the signals temporally using only timestamp data, as
depicted in Figure 2.

Figure 2: FTM-Depth Data Temporal Alignment. By temporally aligning the FTM
signal and RGB-D data in this manually matched example as a pre-processing
step, we can see the similarity and hence the benefits a cross-modal
association between the signals yields on manually unannotated data. In the
example shown, while we know the depth signal of the pedestrian is obtained
from the bounding box in the scene, we do not know which individual is
associated with the FTM signal. We use ViFiCon to perform this multimodal
association as a downstream task.

###  3.2 Image Band Representation

Both RGB-D camera depth information and FTM information for each individual in
a scene act as a 1-dimensional signal, with the depth from the camera or
distance collected from FTM varying over each timestamp. Because of this, we
can map the signal data to a unique set of images we call the image band
representation, which we can then use image-based networks and algorithms to
process the data. This simple band image representation allows for multiple
signals to be processed in the same image, allowing for a more robust and
accurate view of the activity in the scene. This allows us to take advantage
of the spatially invariant property of CNNs in the setup of our
representation. Because there exists a cross-modal relationship between the
RGB-D and FTM data, where the number of FTM signals in a scene is fixed in the
experiment set-up while the number of incoming signals from the RGB-D data
varies depending on the number of background pedestrians, we also immediately
know exactly how many signals at a time should be considered for the depth
samples. Figure 3 provides more details regarding the setup.

(a) Pretext Synchronization Task Band Image Representation

(b) Downstream Association Task Band Image Representation

Figure 3:  Band image representation. In the pretext synchronization task,
obtain a scene-wide alignment of frames. We use the number of incoming FTM
signals as guidance for the number of depth signals that should be present in
the depth image. We then create combinations of the depth data for comparison
with the FTM data. Positive pairings correspond to temporally synchronized
depth and FTM image bands. In the downstream association task, we compare
individual bands. We avoid ever needing to provide ground truth association
data (vision-to-wireless matches) to the model.

Across all sampled timestamps for each data sample sequence in both the vision
and wireless domain, we determine the minimum and maximum distance and map
these values between 0 and 255. Each value then corresponds to a color to be
mapped in the image band representation. Each band representing a distance
value is one pixel wide by ten pixels tall, and the temporal nature of the
signal is represented by horizontally-stacked bands, with time increasing left
to right. Each band image also contains multiple signal sequences. Each signal
sequence is separated by an equally sized, solid colored line with the color
value being determined by the average of the two signals. This ensures a
gradual separation between the two sequences instead of a sharp decline
between them, and provides more context to the deep learning model through
separation.

The image band representations of the RGB-D depth data and FTM distance data
used in the self-supervised dataset consist of slices of the three minute
sequences provided in the Vi-Fi dataset. A hyperparameter controls how many
frames should be considered for each image band image, which then is used to
split the image band representation.

###  3.3 Self-Supervised Dataset

Our self-supervised dataset then consists of sets of positive and negative
image band matches. We consider positive pairs as temporally synchronized
RGB-D and FTM band images, whereas negative pairs are made up of temporally
unsynchronized RGB-D and FTM band images. Hence, we use no human-labeled data
for training using the self-supervised dataset, only passively collected
timestamp data. Positive pairs are given a synchronization label of ”1” and
negative pairs receive a synchronization label of ”0” to be used in the
contrastive loss.

The workflow for generating positive and negative pairs is outlined in Figure
4. We first split the multimodal data from each 3 minute sequence into two
separate processing steps, one for the RGB-D camera data and one for the FTM
ranging data. Using the off-the-shelf ZED person detection model, we extract
bounding boxes with ID values corresponding to individuals in the scene on the
vision side. The object detector is not aware of which detected people are
holding phones. We extract depth signals by taking the average depth value
across the bounding box output for an identified individual for each frame.
The number of returned signals varies depending on the number of detected
individuals in the frames in the temporal window. On the FTM or wireless
domain side, we extract the fixed number of ranging data for each individual
in the scene holding a phone.

Upon extracting information from both modalities, the timestamp information is
aligned across both domains. We then, from a provided hyperparameter, split
the two modalities up into the image band representations. On the FTM side,
each signal is mapped into the band image for the global, scene-wide pretext
task of scene synchronization. In the downstream association task, the signals
are split individually into their own band images. The order of the signals in
the band image with respect to which participant is holding a phone does not
matter, as the model will pick up on spatial cues to perform matching between
the modalities.

In the RGB-D image band representation, a variable number of depth signals may
exist in the $k$ frame temporal window. The wireless modality gives an
indication at how many users in the scene hold phones, and we use this
information in the vision modality to create combinatorics of depth walking
data. However, one such issue that arises is the fact that some combinatorics
of depth data may not strictly match the data collected from the wireless
domain, including scenarios where no participants are selected for the depth
representation. To account for this, we threshold for depth values that are
not constant for a long amount of time within the window, signifying that an
individual has yet to enter or has exited the scene. When multiple
combinatorics exist for a certain frame window, we choose randomly between the
images. We then generate the positive and negative pairs based on whether or
not the FTM and depth data are temporally synchronized.

Figure 4: Self-Supervised Dataset Creation and Pre-Processing. For each
sequence, we use an off-the-shelf object detector to obtain bounding box
information for pedestrians in the scene. We then take the average depth of
the bounding box to extract depth signals for each pedestrian, and create
combinations of depth bands with respect to the fixed number of FTM signals in
the scene if possible.

##  4 Approach

We create our self-supervised contrastive learning model as a set of two
identical subnetworks to process the depth and FTM band image representations.
The result is siamese CNNs responsible for fusing the vision and wireless data
together in a joint latent space, shown in Figure 5. We first pass the band
image into a set of convolutional layers to obtain feature maps from the
original representation. After being passed through a set of convolutional
layers, a fully connected layer maps the representation into a 256-dimensional
feature vector to be embedded into the shared latent space. The fully
connected layer is dependent on the input size of the band image
representation, hence re-training may be necessary for different frame window
sizes and number of detected FTM bands. The representation is normalized so
that the two vectors are mapped to a shared latent space.

Determining the embedding distance of each of the vectors in the latent space
is performed using the Euclidean distance (ED), given by equation 1:

${ED}(T_{v},T_{w})=||T_{v}-T_{w}||_{2}$ (1)

Where $T_{v}$ represents the 256-dimensional feature vector embedding for the
vision depth data, and $T_{w}$ represents the 256-dimensional feature vector
embedding for the wireless FTM data. The loss which is back-propagated through
the model is a standard contrastive loss function using the calculated
Euclidean distances of the feature vector embeddings. The loss function uses
the implicit synchronization label determined based on timestamp data in order
to minimize the distance between positive examples and maximize the distance
between negative examples. The contrastive loss (CL) is defined by equation 2:

$\footnotesize{CL}={\frac{1}{B}}\times(Y\times||T_{v}-T_{w}||_{2}^{2}+(1-Y)\times\max(M-||T_{v}-T_{w}||_{2},0)^{2})$
(2)

Where $Y$ is the synchronization label for the given pair, $M$ is a margin
hyperparameter, and $B$ is the batch size, the number of examples passed
through in a singular batch. For positive pairs, or when $Y=1$, a small
Euclidean distance value will yield to a small loss, whereas for negative
pairs, when $Y=0$, so long as the Euclidean distance is larger than the margin
hyperparameter $M$, the loss will be minimized. After training, we expect
positive pairs to be embedded close to one another in the latent space, that
is, a small Euclidean distance value, whereas negative pairs should have a
large Euclidean distance as defined by the contrastive loss function.

Figure 5: Network Architecture. We create a siamese convolutional neural
network with shared model weights structure to process the image band
representations of the vision and wireless modalities. The output of each
subnetwork is a normalized 256-dimensional vector representation of the
respective modality, used to map into a shared latent space. We then compute
the distance between the two vectors using the Euclidean distance, with
network updates performed using a traditional contrastive loss.

##  5 Experiments and Evaluation Setup

We train the ViFiCon multimodal self-supervised contrastive learning model on
a subset of sequences from Dataset B of the Vi-Fi dataset. Specifically, we
train on a set of 30, 3 minute outdoor sequences using the band image
representation for global scene synchronization. Each band image consists of a
maximum of 3 signal bands on both the vision side and wireless side. On the
vision side, training is performed using only the raw bounding box data
extracted from the object detector with no hand tweaking and no ground truth
information. We then test the model on a set of 5 different 3 minute outdoor
sequences as defined in ViTag for comparison with state-of-the-art fully
supervised baseline models [17, 5]. We report results on the global scene
pretext synchronization task on the test set and the generalization to
downstream individual person association between depth and FTM without any
further training to the model. We also vary the number of frames to be used in
the band image representation, including $k=10,25,$ and $50$ and compare the
results. We use the contrastive loss as our loss function and the stochastic
gradient descent (SGD) as our optimizer. ViFiCon is trained for a total of 400
epochs at a learning rate of 0.003 and a standard margin hyperparameter for
the loss function of 0.2. Depending on the number of frames used, training the
ViFiCon model takes only 10 minutes on an NVIDIA 3090 GPU.

Evaluation Metrics Like Vi-Fi [17] and ViTag [5], our primary evaluation
metric is the IDentification Precision [25], or IDP. IDP is defined in
equation 3:

$IDP=\frac{IDTP}{IDTP+IDFP}$ (3)

Where IDTP corresponds to the identified true positive pairs IDFP corresponds
to the false positive pairs in the latent space. We also provide insight into
the accuracy and F1 score of the ViFiCon model on both the pretext
synchronization task and downstream association task, but we focus our
attention on IDP as these correspond to synchronization and association for
the bounding boxes in the temporal window.

Determining which pairs in the latent space correspond to true positives,
false positives, true negatives, and false negatives is obtained using a
margin line. We plot each of the depth-FTM pairing Euclidean Distances as a
latent space representation, and then determine the best separation of
positive and negative pairings by looping across all of the pairs. True
positives are defined to be any positive pairings to the left of the margin
line as these pairs have a small Euclidean Distance and any positive pairings
to the right of the margin line as a false positive pair. Similarly, any
negative pairs to the right of the margin line are considered true negatives
and any negative pairs to the left of the margin line are false negatives. A
sample view of the latent embedding space including margin line separation is
shown in Figure 6.

Figure 6: Sample Latent Space Embedding on downstream association task on test
set. We use the true and false positive and negatives to evaluate the model
performance. Observe that positive pairings of depth and FTM data
(corresponding to a correct association) are embedded close to one another
based on the Euclidean distance, whereas negative pairings (corresponding to
an incorrect association) have a larger distance separating the two.

##  6 Results

(a) Manually Selected Ground Truth Participants on Test Set

(b) Manually Corrected Pedestrian Bounding Boxes on Test Set

(c) Off-the-Shelf Detector [1] on Test Set

Figure 7:  Metrics on manually selected and annotated ground truth, manually
corrected pedestrian bounding boxes, and off-the-shelf detector variants for
depth data of the ViFiCon model. Notice as the temporal window size increases,
model performance increases. In general the manually annotated and selected
ground truth has the best performance, though the manually corrected
pedestrian bounding boxes and off-the-shelf detector perform comparably due to
our thresholding technique.

(a) Metrics for a Consistent Margin Line with Train Latent Space
Representation

(b) Metrics for a Variable Margin Line

Figure 8:  Effects of keeping a consistent margin line with respect to the
margin line from the train set embedding and varying the margin line across
each set. In general, we see model performance improvement when varying the
margin line with respect to the embeddings of the current set.

Performance on Pretext Synchronization Task We record the results of ViFiCon’s
performance of the pretext synchronization task on the test set in Table 1. We
see that in general, ViFiCon’s performance on the pretext synchronization task
improves for all three metrics as the temporal window size increases, with the
best results at a large number of frames as more context is provided.

Performance on Downstream Association Task We record the results of ViFiCon’s
performance of the downstream association task on the test set in Table 2. We
see that though the model’s performance progressively improves in terms of
accuracy and F1 score as the temporal window size increases, the best IDP
score is obtained at a temporal window size of 25 frames.

Window Size ($k$) | 10 Frame | 25 Frame | 50 Frame
---|---|---|---
IDP | 88.26 % | 82.29 % | 95.83%
Acc | 71.4 % | 78.65 % | 85.42%
F1 | 75.53 % | 79.4 % | 86.79%
Table 1: Summary of SSL ViFiCon performance on Pretext Synchronization Task on Test Set. Window Size ($k$) | 10 Frame | 25 Frame | 50 Frame
---|---|---|---
IDP | 84.77 % | 92.63 % | 85.33%
Acc | 58.33 % | 72.65 % | 78.23%
F1 | 67.1 % | 77.38 % | 80%
Table 2: Summary of SSL ViFiCon performance on Downstream Association Task on
Test Set.

We additionally compare our IDP score with the Vi-Fi and ViTag fully
supervised models on the association task, which each use a frame temporal
window size of 10, as well as a technique that uses pedestrian dead reckoning
(PDR) and procrustes analysis (PA) from the ViTag paper. Unlike Vi-Fi and
Vitag our approach simplifies the vision domain representation, using only
bands of depth data whereas the fully-supervised models uses the RGB camera
data. We show that by extracting information from the vision domain to obtain
the depth signals, we are able to obtain comparable results without the need
for any hand labeling of the dataset. We compare our results from temporal
window size 10 with the fully supervised models on the test dataset in Table
3.

Model | PDR+PA | Vi-Fi | ViTag | ViFiCon
---|---|---|---|---
|  [27, 15] |  [17] |  [5] | (Ours)
| [11, 5] |  |  |
Features | - | IMU, | IMU, | Depth
Used in |  | Bbox | BBox | FTM
Training |  | Vectors, FTM | Vectors, FTM |
% Train | - | 100 % | 100 % | 33%
Set Used |  |  |  |
IDP | 41.79 % | 84.97 % | 87.85 % | 84.77%
Table 3: Comparison of SSL ViFiCon performance with state-of-the-art on Downstream Association Task on Test Set. We note that overall ViFiCon uses fewer features, only requiring depth and FTM data whereas Vi-Fi and Vitag uses IMU, FTM, and bounding boxes (bbox) vectors from the camera data. Method | Manually | Manually | Off-the-Shelf
---|---|---|---
(50 Frames) | Selected GT | Corrected | BBoxes
|  | Pedestrian Bboxes | [1]
IDP | 94 % | 91.33 % | 85.33%
Acc | 91.5 % | 90.48 % | 78.23%
F1 | 91.86 % | 90.73 % | 80%
Table 4: Summary of ViFiCon performance on Downstream Association Task on Test Set across manually selected and annotated ground truth (GT) participants, manually corrected pedestrian bounding boxes, and off-the-shelf detector variants for depth data with temporal window size $k=50$. Method | Consistent | Variable
---|---|---
(Window 50 Frames) | Training Margin Line | Margin Line
IDP | 96 % | 94 %
Acc | 88.78 % | 91.5 %
F1 | 89.72 % | 91.86 %
Table 5: Summary of SSL ViFiCon performance on Downstream Association Task on
Test Set across fixed training margin line and variable margin line variants
with temporal window size $k=50$.

Case Study: Comparing Off-The-Shelf and Manually Corrected Bounding Boxes We
compare our self-supervised approach with off-the-shelf bounding box
detections on the depth data to two different approaches. In the first, we use
the ground truth participant bounding boxes for the band image representation
on the depth side. In the second approach, we manually correct bounding boxes
produced by the off-the-shelf model such that the ID values of participants
and pedestrians are kept consistent if the person exits the scene for a long
while by hand-tweaking the bounding box IDs. This approach ensures that a band
image representation containing all participants holding phones does exist in
the set of possible combinatorics, however does not guarantee this image will
be selected in the generation of the dataset, and train using this band image.
We provide tabular results of the study on the downstream association task in
Table 4 for $k=50$ frames, and provide a chart of the IDP for all scenarios
across all temporal window sizes in Figure 7.

Case Study: Effect of Margin Line Tweaking We additionally study the effect
that the margin line has on the performance of the self-supervised model. We
consider two cases: using the margin line location that is returned when
running the model on the train set and applying the margin line across the
test set pretext and downstream latent spaces, and allowing the margin line to
vary with respect to the current embedded representations. No learning is
involved with the margin line itself - we only iterate over the positive and
negative embeddings to determine the best fit. Table 5 provides insight on the
50 frame temporal window size band image version of ViFiCon with a fixed and
variable margin line, and Figure 8 provides results across all temporal window
sizes considered. As expected, choosing the best fit margin line results in
the highest model performance.

##  7 Conclusion

In this work, we explore the impact of self-supervised contrastive learning in
learning both a global scene synchronization between vision and wireless
modalities and a downstream individual association without further training
using depth and FTM signals. We propose the image band representation as a
means of representing a set of signals in a shared image and to bring the two
modalities closer together in a comparable space. To learn the pretext
synchronization task and apply the knowledge to the downstream synchronization
task, we design the ViFiCon model, made up of a dual-stream siamese CNN which
is fed both the vision (depth data) and wireless (FTM ranging data) modality
band images and outputs a set of normalized vector representations using the
contrastive loss. Our system achieves a comparable IDP to online fully-
supervised association models using information from the vision modality and
wireless modality, receiving an 84.77% IDP, with state-of-the-art receiving an
IDP of 87.85%. Though our model does simplify the task by associating between
depth and FTM signals and without original input camera frames as in the
fully-supervised task, we show that by simply extracting information from the
camera modality to map into a similar representation, we are able to achieve
comparable performance while not requiring any hand-labeled or ground truth
data.

##  8 Acknowledgements

This research has been supported by the National Science Foundation (NSF)
under Grant No. CNS-1901355, CNS-1910170, CNS-1901133, CNS-2055520.

## References

  * [1] Zed camera website (accessed: 04.09.2021).
  * [2] ”IEEE Standard for Information technology–Telecommunications and information exchange between systems Local and metropolitan area networks–Specific requirements - Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications”.  ”IEEE Std 802.11-2016 (Revision of IEEE Std 802.11-2012)”, pages 1–3534, Dec 2016.
  * [3] Fadel Adib and Dina Katabi.  See through walls with wifi!  In Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM, pages 75–86, 2013.
  * [4] Alexandre Alahi, Albert Haque, and Li Fei-Fei.  Rgb-w: When vision meets wireless.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3289–3297, 2015.
  * [5] Bryan Bo Cao, Abrar Alali, Hansi Liu, Nicholas Meegan, Marco Gruteser, Kristin Dana, Ashwin Ashok, and Shubham Jain.  Vitag: Online wifi fine time measurements aided vision-motion identity association in multi-person environments.
  * [6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [7] Joon Son Chung and Andrew Zisserman.  Out of time: automated lip sync in the wild.  In Asian conference on computer vision, pages 251–263. Springer, 2016.
  * [8] Liu et al.  Vi-fi multi-modal dataset.
  * [9] Shiwei Fang, Tamzeed Islam, Sirajum Munir, and Shahriar Nirjon.  Eyefi: Fast human identification through vision and wifi-based trajectory matching.  In 2020 16th International Conference on Distributed Computing in Sensor Systems (DCOSS), pages 59–68. IEEE, 2020.
  * [10] Google.  Google pixel website.
  * [11] John C Gower.  Generalized procrustes analysis.  Psychometrika, 40(1):33–51, 1975.
  * [12] Yue Gu, Xinyu Li, Shuhong Chen, Jianyu Zhang, and Ivan Marsic.  Speech intention classification with multimodal deep learning.  In Canadian conference on artificial intelligence, pages 260–271. Springer, 2017.
  * [13] Mohamed Ibrahim, Hansi Liu, Minitha Jawahar, Viet Nguyen, Marco Gruteser, Richard Howard, Bo Yu, and Fan Bai.  Verification: Accuracy evaluation of wifi fine time measurements on an open platform.  In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking, pages 417–427, 2018.
  * [14] Bruno Korbar, Du Tran, and Lorenzo Torresani.  Cooperative learning of audio and video models from self-supervised synchronization.  Advances in Neural Information Processing Systems, 31, 2018.
  * [15] Wojtek Krzanowski.  Principles of multivariate analysis, volume 23.  OUP Oxford, 2000.
  * [16] Tianhong Li, Lijie Fan, Yuan Yuan, and Dina Katabi.  Unsupervised learning for human sensing using radio signals.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3288–3297, 2022.
  * [17] Hansi Liu, Abrar Alali, Mohamed Ibrahim, Bryan Bo Cao, Nicholas Meegan, Hongyu Li, Marco Gruteser, Shubham Jain, Kristin Dana, Ashwin Ashok, et al.  Vi-fi: Associating moving subjects across vision and wireless sensors.
  * [18] Hansi Liu, Abrar Alali, Mohamed Ibrahim, Hongyu Li, Marco Gruteser, Shubham Jain, Kristin Dana, Ashwin Ashok, Bin Cheng, and Hongsheng Lu.  Lost and found! associating target persons in camera surveillance footage with smartphone identifiers.  In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services, pages 499–500, 2021.
  * [19] Alliance W Wi-Fi CERTIFIED Location.  Brings wi-fi indoor positioning capabilities.  Wi-Fi Alliance.[Online]. Avail-able: https://wi-fi. org/news-events/newsroom/wi-fi-certified-locationbrings-wi-fi-indoor-positioning-capabilities, 2017\.
  * [20] Pranay Manocha, Zeyu Jin, Richard Zhang, and Adam Finkelstein.  Cdpam: Contrastive learning for perceptual audio similarity.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 196–200. IEEE, 2021.
  * [21] Alessandro Masullo, Tilo Burghardt, Dima Damen, Toby Perrett, and Majid Mirmehdi.  Who goes there? exploiting silhouettes and wearable signals for subject identification in multi-person environments.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [22] Alessandro Masullo, Tilo Burghardt, Dima Damen, Toby Perrett, and Majid Mirmehdi.  Person re-id by fusion of video silhouettes and wearable signals for home monitoring applications.  Sensors, 20(9):2576, 2020.
  * [23] Gledson Melotti, Cristiano Premebida, Nuno MM da S Gonçalves, Urbano JC Nunes, and Diego R Faria.  Multimodal cnn pedestrian classification: a study on combining lidar and camera data.  In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pages 3138–3143. IEEE, 2018.
  * [24] Takashi Miyaki, Toshihiko Yamasaki, and Kiyoharu Aizawa.  Tracking persons using particle filter fusing visual and wi-fi localizations for widely distributed camera.  In 2007 IEEE International Conference on Image Processing, volume 3, pages III–225. IEEE, 2007.
  * [25] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi.  Performance measures and a data set for multi-target, multi-camera tracking.  In European conference on computer vision, pages 17–35. Springer, 2016.
  * [26] Luciano Spinello, Rudolph Triebel, and Roland Siegwart.  Multimodal detection and tracking of pedestrians in urban environments with explicit ground plane extraction.  In 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1823–1829. IEEE, 2008.
  * [27] Boyuan Wang, Xuelin Liu, Baoguo Yu, Ruicai Jia, and Xingli Gan.  Pedestrian dead reckoning based on motion mode recognition using a smartphone.  Sensors, 18(6):1811, 2018.
  * [28] He Wang, Xuan Bao, Romit Roy Choudhury, and Srihari Nelakuditi.  Visually fingerprinting humans without face recognition.  In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services, pages 345–358, 2015.
  * [29] Han Zou, Jianfei Yang, Hari Prasanna Das, Huihan Liu, Yuxun Zhou, and Costas J Spanos.  Wifi and vision multimodal learning for accurate and robust device-free human activity recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0–0, 2019.