# ConfLab: A Data Collection Concept, Dataset, and Benchmark for Machine
Analysis of Free-Standing Social Interactions in the Wild

Chirag Raman1 &Jose Vargas-Quiros111footnotemark: 1 &Stephanie
Tan111footnotemark: 1 &Ashraful Islam2
Ekin Gedik1 Hayley Hung1
1Delft University of Technology, Delft, The Netherlands
{c.a.raman, j.d.vargasquiros, s.tan-1, e.gedik<EMAIL_ADDRESS>
2Rensselaer Polytechnic Institute, New York, USA
<EMAIL_ADDRESS>
Equal contribution

###### Abstract

Recording the dynamics of unscripted human interactions in the wild is
challenging due to the delicate trade-offs between several factors:
participant privacy, ecological validity, data fidelity, and logistical
overheads. To address these, following a datasets for the community by the
community ethos, we propose the Conference Living Lab (ConfLab): a new concept
for multimodal multisensor data collection of in-the-wild free-standing social
conversations. For the first instantiation of ConfLab described here, we
organized a real-life professional networking event at a major international
conference. Involving $48$ conference attendees, the dataset captures a
diverse mix of status, acquaintance, and networking motivations. Our capture
setup improves upon the data fidelity of prior in-the-wild datasets while
retaining privacy sensitivity: $8$ videos ($1920\times 1080,60$ fps) from a
non-invasive overhead view, and custom wearable sensors with onboard recording
of body motion (full $9$-axis IMU), privacy-preserving low-frequency audio
($1250$ Hz), and Bluetooth-based proximity. Additionally, we developed custom
solutions for distributed hardware synchronization at acquisition, and time-
efficient continuous annotation of body keypoints and actions at high sampling
rates. Our benchmarks showcase some of the open research tasks related to in-
the-wild privacy-preserving social data analysis: keypoints detection from
overhead camera views, skeleton-based no-audio speaker detection, and
F-formation detection.

##  1 Introduction

A crucial challenge towards developing artificial socially intelligent systems
is understanding how real-life situational contexts affect social human
behavior [1]. Social-science findings indeed show that the dynamics of how we
conduct daily interactions vary significantly depending on the social
situation [2, 3, 4]. Unfortunately, such dynamics are not adequately captured
by many data collection setups where role-played or scripted scenarios are
typical [5].

In this paper we address the problem of collecting a privacy-sensitive dataset
of unscripted social dynamics of real-life relationships where encounters can
influence someone’s daily life. We argue that doing so requires recording
these exchanges in the natural ecology, requiring an approach different from
the typical setup of locally-organized studies. Specifically, we focus on
free-standing interactions within the setting of an international conference
(see Figure 1).

Recording an international community in its natural habitat is characterized
by several intersecting challenges: an intrinsic trade-off exists between data
fidelity, ecological validity, and privacy preservation. For ecological
validity, a non-invasive capture setup is essential for mitigating any
influence on behavior naturalness [6, 7, 8].The most common solution involves
mounting cameras from aerial perspectives such as top-down [9, 10] and
elevated-side views [11, 12, 13]. Now elevated-side views make it easy to
capture sensitive personal information such as faces, which leads to several
ethical concerns. For instance, capturing faces has been related to harmful
downstream surveillance applications [14]. Besides, state-of-the-art (SOTA)
body-keypoint estimation techniques perform poorly on aerial perspectives [15,
9], making the extraction of automatic pose annotations challenging (Figure
3). To avoid such issues, some researchers have turned to more privacy-
preserving wearable sensors shown to benefit many behavior analysis tasks [8,
16, 17].

In all, the closest related datasets (see Table 1) suffer from several
technical limitations precluding the analysis and modeling of fine-grained
social behavior: (i) lack of articulated pose annotations; (ii) a limited
number of people in the scene, preventing complex interactions such as group
splitting/merging behaviors, and (iii) an inadequate data sampling-rate and
synchronization-latency to study time-sensitive social phenomena [18, Sec.
3.3].

Figure 1: Snapshot of the interaction area from our cameras. We annotated only
cameras highlighted with red borders (high scene overlap). For a clearer
visual impression of the scene, we omit cameras 1 (few people recorded) and 5
(failed early in the event). Faces blurred to preserve privacy.

To address all these limitations, we propose the Conference Living Lab
(ConfLab): a new concept for multimodal multisensor data collection of
ecologically-valid social settings. From the first instantiation of ConfLab,
we provide a high-fidelity dataset of $48$ participants at a professional
networking event.

Figure 2: Frequency of newcomer/veteran participants (left) and reported
research interests (right).

Figure 3: Keypoint detection using pretrained RSN [19]. Additional SOTA
results are in Appendix F.1

#### Methodological Contributions:

We describe a data collection design that captures a diverse mix of real
levels of seniority, acquaintance, affiliation, and motivation to network (see
Figure 13). This was achieved by organizing ConfLab as part of a major
international scientific conference. ConfLab had these goals: (i) a data
collection effort follwing a by the community for the community ethos: the
more volunteers, the more data, (ii) volunteers who potentially use the data
can experience first-hand potential privacy and ethical considerations related
to sharing their own data, (iii) in light of recent data sourcing issues [20,
14], we incorporated privacy and invasiveness considerations directly into the
decision-making process regarding sensor type, positioning, and sample-rates.

#### Technical Contributions:

(i) aerial-view articulated pose: our annotations of $17$ full-body keypoints
enable improvements in (a) pose estimation and tracking, (b) pose-based
recognition of social actions (under-explored in the top-down perspective),
(c) pose-based F-formation estimation (has not been possible from prior work
[10, 21, 22, 23]), and (d) the direct study of interaction dynamics using full
body poses (previously limited to lab settings [24]). (ii) subtle body
dynamics: we are the first to use a full 9-axis Inertial Measurement Unit
(IMU) enabling a richer representation of behaviour at higher sample rates;
previous rates were found to be insufficient for downstream tasks [17]. (iii)
enabling finer temporal-scale research questions: a sub-second crossmodal
latency of $\sim 13$ ms along with higher sampling rate of features ($60$ fps
video, $56$ Hz IMU) opens the gateway for the in-the-wild study of nuanced
time-sensitive social behaviors like mimicry and synchrony.

Table 1: Comparison of ConfLab with prior datasets of free-standing
conversation groups in in-the-wild social interaction settings. Conflab is the
first and only social interaction dataset that offers skeletal keypoints and
speaking status at high annotation resolution, as well as hardware
synchronized camera and multimodal wearable signals at high resolution.

Dataset |  People/ Scene | Video | Manual Annotations | Wearable Signals | Synchronization
---|---|---|---|---|---
Cocktail [13]$\dagger$ | $7$ | $512\times 384$ |  F-formations ($20$ and $30$ min, $1/5$ Hz) | None | Unknown
CoffeeBreak [12] | $14$ | $1440\times 1080$ |  F-formations (130 frames in two sequences) | None | None
IDIAP [10] | $>50$ |  $180$ min; $654\times 439$ $20$ fps |  F-formations ($82$ independent frames) | None | None
SALSA [11]$\dagger$ | $18$ |  $60$ min; $1024\times 768$ $15$ fps |  Bounding boxes ($30$ min) Head & body ori. ($30$ min) F-formations ($60$ min) (all $1/3$ Hz) |  Audio MFCCs ($30$ Hz) Acceleration ($20$ Hz) IR proximity ($1$ Hz) |  Post-hoc infra-red event-based (no-drift assumption)
MnM [9]$\dagger$ | $32$ |  $30$ min; $1920\times 1080$ $30$ fps |  Bounding boxes ($30$ min, $1$ Hz $\ddagger$ ) F-formations ($10$ min, $1$ Hz ) Actions ($45$ min, $1$ Hz$\ddagger$) |  Accelerometer ($20$ Hz) Radio proximity ($1$ Hz) |  Intra-wearable sync via gossiping protocol; Inter-modal sync using manual inspection @$1$ Hz
ConfLab | $48$ |  $\sim 45$ min; $1920\times 1080$ $\mathbf{60}$ fps |  $\mathbf{17}$ keypoints ($\mathbf{16}$ min, $\mathbf{60}$ Hz) F-formations ($16$ min, $1$ Hz) Speaking status ($\mathbf{16}$ min, $\mathbf{60}$ Hz) |  Low-freq. audio ($\mathbf{1250}$ Hz) BT proximity ($\mathbf{5}$ Hz) $\mathbf{9}$-axis IMU ($\mathbf{56}$ Hz) |  Wireless hardware sync at acquisition, max latency of $\sim\mathbf{13}$ ms [18]

  * •

$\dagger$ Includes self-assessed personality ratings

  * •

$\ddagger$ Upsampled to $20$ Hz using Vatic [25]

  * •

BT: Bluetooth IMU: Inertial Measurement Unit

##  2 Related Work

Early datasets of in-the-wild social events either spanned only a few minutes
(e.g. Coffee Break [12]), or were recorded at such a large distance from the
participants that performing robust, automated person detection or tracking
with SOTA approaches was non-trivial (e.g. Idiap Poster Data [10]). More
recently, two different strategies have emerged to circumvent such issues.

One approach involves fully instrumented labs with a high resolution multi-
camera setup for video and audio data. Here automatic detectors [26, 27, 24]
could be applied to obtain poses. This circumvents the cost- and labor-
intensive process of manually labeling head poses, at the cost of less
portable sensing setups. Notable examples of such in-the-lab studies include
seated scenarios, such as the AMI meeting corpus [28], and more recently
standing scenarios like the Panoptic Dataset [24]. Both enable the learning of
multimodal behavioral dynamics. However, the dynamics of seated, scripted, or
role-playing scenarios are different from that of an unconstrained social
setting such as ours. In contrast, ConfLab moves out of the lab with a more
modular and portable multimodal, multisensor solution that scales easily in
the wild.

Another approach exploited wearable sensor data to allow for multimodal
processing—sensors included 3 or 6 DOF inertial measurement units (IMU);
infrared, bluetooth, or radio sensors to measure proximity; or microphones for
speech behavior [9, 11]. While proximity has been used as a proxy of face-to-
face interaction [11, 29, 30, 31, 32], recent findings highlight significant
problems with such an assumption [33]. Such errors can have a significant
impact on the machine-perceived experience of an individual, precluding the
development of personalized technology. Chalcedony badges used by [9] show
more promising results with a radio-based proximity sensor and accelerometer
[34], but such data remains insufficient for more downstream tasks due to the
relatively low sample (20Hz) and annotation (1Hz) frequency [17]. In light of
these challenges in wearable sensing, ConfLab features custom-developed Midge
sensors that enable more flexible and fine-grained on-device recording. At the
same time, ConfLab enables researchers in the wearable and ubiquitous
computing communities to investigate the benefit of exploiting wearable and
multimodal data.

Furthermore, while both SALSA [11] and MatchNMingle [9] capture a multimodal
dataset of a large group of individuals involved in mingling behavior, the
inter-modal synchronization is only guaranteed at 1/3 Hz and 1 Hz,
respectively. Prior works coped with lower tolerances by computing summary
statistics over input windows [17, 35, 36]. While 1 Hz is able to capture some
conversation dynamics [37], it is insufficient to study fine-grained social
phenomena such as back-channeling or mimicry that involve far lower latencies
[18, Sec. 3.3]. ConfLab provides data streams with higher sampling rates,
synchronized at acquisition with our method shown to yield a $13$ ms latency
at worst [18] (see Sec. 3). Table 1 summarizes the differences between ConfLab
and other related datasets.

##  3 Data Acquisition

Figure 4: Screenshots from the ConfLab: Meet the Chairs! event website

Figure 5: The Midge

In this section we describe the considerations, design, and supporting
community engagement activities for the first instantiation of ConfLab at ACM
Multimedia 2019 (MM’19), to serve as a template and case study for other
similar efforts.

#### Ecological Validity and Recruitment

An often-overlooked but crucial aspect of in-the-wild data collection is the
design and ecological validity of the interaction setting [6, 7, 8]. To
capture natural interactions in a professional setting and encourage mixed
levels of status, acquaintance, and motivations to network, we co-designed a
networking event with the MM’19 organizers called Meet the Chairs! Our event
website (https://conflab.ewi.tudelft.nl/) served to inform participants about
the goals of a community created dataset, and transparently describe the data
collection process (Figure 5). During the conference, participants were
recruited via word-of-mouth marketing, social media, conference announcements,
and the event website. As an additional incentive beyond interacting with the
Chairs and participating in a community-driven data endeavor, we provided
attendees with post-hoc insights into their networking behavior from the
collected wearable-sensors data. See Supplementary material for a sample
participant report.

#### Privacy and Ethics

The collection and sharing of ConfLab is GDPR compliant. The dataset design
and process was approved by both, the Human Research Ethics Committee (HREC)
at our institution (TUDelft) and the conference location’s national
authorities (France). All participants gave consent for the recording and
sharing of their data at registration.(See the Datasheet in the Appendix for
the consent form.) Given the involvement of private human data, ConfLab is
only available for academic research purposes under an End User License
Agreement. Such an as open as possible and as closed as necessary ethos for
open science acknowledges the limitation that personal data places on open
sharing [38, 39].

Figure 6: Comparing the top-down (top-left, camera 4) and elevated-side camera
views (rest). Note how the top-down view is better at mitigating the capture
of faces and suffers from fewer occlusions. This allows for a clearer capture
of gestures and lower extremities for the most number of people while also
preserving privacy.

#### Data Capture Setup

Our goal while designing the capture setup was to find the best trade-off
between maximizing data fidelity and interfering with the naturalness of the
interaction (ecological validity) or violating participant privacy (ethical
considerations). Through discussions with the HREC and General Chairs of MM’19
we decided to mitigate the capture of faces, which constitute one of the most
sensitive personally-identifiable features. Avoiding the inclusion of faces
serves two purposes. First, it safeguards against misuse in downstream tasks
with potential negative societal impacts such as harmful surveillance. Such
issues have led to the retraction of some person re-identification datasets
[14]. Second, it protects the participants who are part of a real research
community; since the dataset does not involve role-playing or scripted
conversations, the dataset contains their actual behavior. Consequently, we
chose an aerial perspective for the video modality (see Figure 6). The $10$ m
$\times$ $5$ m interaction area was recorded by $14$ GoPro Hero 7 Black
cameras ($60$fps, $1080$p, Linear, NTSC) [40]. $10$ of these were placed
directly overhead at a height of $\sim 3.5$ m at $1$ m intervals, with $4$
cameras at the corners providing an elevated-side-view perspective. (The HREC
has suggested not sharing the elevated-side-view videos due to the presence of
faces.) For capturing multimodal data streams, we designed a custom wearable
multi-sensor pack called the Midge111Documentation and schematics:
https://github.com/TUDelft-SPC-Lab/spcl_midge_hardware (see Figure 5 for a
design render), based on the open-source Rhythm Badge designed for office
environments [41]. We improved upon the Rhythm Badge to achieve more fine-
grained and flexible data capture (see Appendix D). We designed the Midge in a
conference badge form-factor for seamless integration. Unlike smartphones,
wearable badges allow for a simple grab-and-go setup and do not suffer from
sensor/firmware differences across models. Popular human behavior datasets are
synchronized by maximizing similarity scores around manually identified common
events, such as infrared camera detections [11], or speech plosives [42].
While recordings in lab settings can allow for fully wired recording setups,
recording in-the-wild requires a distributed wireless solution. We developed a
solution to synchronize the cameras and wearable sensors directly at
acquisition while significantly lowering the cost of the recording setup [18],
making it easier for others to replicate our capture setup. See Appendix D for
synchronization and calibration details, and Appendix B for images of the
setup.

#### Data Association and Participant Protocol

One consideration for multimodal data recording is the data association
problem—how can pixels corresponding to an individual be linked to their other
data streams? To this end, we designed a participant registration protocol.
Arriving participants were greeted and fitted with a Midge. The ID of the
Midge acted as the participant’s identifier. One team member took a picture of
the participant while ensuring both the face of the participant and the ID on
the Midge were visible. In practice, it is preferable to avoid this step by
using a fully automated multimodal association approach. However this remains
an open research challenge [43, 44]. During the event, participants mingled
freely—they were allowed to carry bags or use mobile phones. Conference
volunteers helped to fetch drinks for participants. Participants could leave
before the end of the one hour session.

#### Replicating Data Collection Setup and Community Engagement

After the event, we gave a tutorial at MM’19 [45] to demonstrate how our
collection setup could be replicated, and to invite conference attendees and
event participants to reflect on the broader considerations surrounding
privacy-preserving data capture, sharing, and future directions such
initiatives could take.

##  4 Data Annotation

#### Continuous Keypoints Annotation

Existing datasets of in-the-wild social interactions have mainly focused on
localizing subjects via bounding boxes [11, 9]. However, richer information
about the social dynamics such as gestures and changes in orientation cannot
be retrieved from bounding boxes alone, and necessitates the labeling of
multiple skeletal keypoints. The typical approach to keypoint annotation
involves using tools such as Vatic [25] or CVAT [46] to manually label every
$N$ frames followed by interpolating over the rest of the frames. This one-
frame-at-a-time annotation procedure makes obtaining keypoint annotations a
labor- and cost-intensive process. Moreover, interpolation fails to capture
the finer temporal dynamics of the underlying behavior, and reduces the
benefits of higher-framerate video capture. Limited by existing tools, no
related dataset of in-the-wild human behavior has included time-continuous
pose or speaking status annotations.

(a) Keypoint annotation interface in covfee [47] (b) Gallery of identities (c)
Occlusion

Figure 7: Illustration of the body keypoints annotation procedure: (a): our
custom time continuous annotation interface; (b): the gallery of person
identities used by annotators to identify people in the scene (faces blurred);
and (c): the skeleton template with the fraction of occluded frames.

In contrast, to overcome these issues we collected fine-grained time-
continuous annotations of keypoints via a web-based interface implemented as
part of the Covfee framework [48]. Here, annotators follow individual joints
using their mouse or trackpad while playing the video in their web browser.
The playback speed of the video is automatically adjusted using an optical-
flow-based technique to enable annotators to follow keypoints continuously
without pausing the video. This design enables easy keypoint labeling in every
frame of the video ($60$ Hz). We also incorporated a binary occlusion flag for
every body keypoint. Annotators simultaneously controlled this flag to
indicate when a body joint was not directly visible. Note that the flag is
only an additional confidence indicator; we asked the annotators to label the
occluded keypoint using their best estimate if it was deemed to be within the
frame. Our pilot study on the efficacy of Covfee compared to non-continuous
annotation via CVAT [46] is presented in [48]. For the pilot annotators, the
continuous annotation methodology resulted in a 3$\times$ speedup with
statistically indifferent error rates.

We chose the top-down camera views for annotation since they suffer from fewer
occlusions than the elevated-side views, enabling improved capture of gestures
and lower extremities for more number of people (see Figure 6). Given the
overlap in the camera views, we annotated keypoints in five of the ten
overhead cameras (see Figure 1). Note that the same subject could be annotated
in multiple cameras due to the overlap in even the five annotated cameras.
Videos were split into two-minute segments to ease the annotation procedure.
Each segment was annotated by one annotator by tracking the joints of all the
people in the scene.

#### Continuous Speaking Status Annotations

Speaking status is a key non-verbal cue for many social interaction analysis
tasks [49]. We annotated the binary speaking status of every subject due to
its importance as a key feature of social interaction [50, 16, 51, 52, 53] and
to contribute the existing community who are working on this task [54, 55,
17]. Action annotations have traditionally been carried out using frame-wise
techniques [9], where annotators find the start and end frame of the action of
interest using a graphical interface. Given the speed enhancement of
continuous annotation, we also annotated speaking status via a continuous
technique. We implemented a binary annotation interface as part of Covfee
[48]. We asked annotators to press a key when they perceived speaking starting
or ending. In a pilot study with two annotators, we measured a frame-level
agreement (Fleiss’ $\kappa$) of 0.552, comparable to previous work [35].
Similar to [9], the annotations were made by watching the video. We provided
the annotators with all overhead views to best capture visual behavior.

#### F-formation Annotations

Identifying who is likely to have social influence on whom is another
important feature for analyzing social behavior. This is operationalised via
the theory of F-formations, which are groups of people arranging themselves to
converse or socially interact. Similar to prior datasets [9, 11, 13],
F-formations group membership were annotated using an approximation of
Kendon’s definition [56]. F-formation stands for Facing formation, which is a
socio-spatial arrangement where people have direct, easy and equal access
while excluding the space from others in the surroundings. The arrangement
commonly maintains a convex space in the middle of all the participants
(determined by the location and orientation of their lower body), although
other spatial arrangements (e.g., side-by-side, L-shaped) are possible,
especially for smaller-sized groups of people. Annotations were labeled by one
annotator at 1 Hz, following this definition. Since this is a largely
objective and common framework for defining F-formations, we deemed it
sufficient to obtain one set of annotations. Further, since F-formations may
span camera views, we always used the camera that captured each F-formation in
its entirety for annotation.

##  5 Dataset Statistics

(a) speaking turn lengths (b) group size (c) group duration (d) fraction of
newcomers in groups

Figure 8: Data distributions for speaking status and conversation groups

#### Individual-Level Statistics

7(c) shows the average occlusion values we obtained from annotators for each
of the $17$ keypoints. In 8(a) we show the distribution of turn lengths in our
speaking status annotations, for both newcomers and veterans, as per their
self-reported newcomer status to the conference. We defined a turn to be a
contiguous segment of positively-labeled speaking status, which resulted in a
total of $4096$ turns annotated.

#### Group-Level Statistics

We found $119$ distinct F-formations of size greater than or equal to two, and
$38$ instances of singletons. Of these, there are $14$ F-formations and $2$
singletons that include member(s) using the mobile phone. The distributions
for group size and duration per group size are shown in 8(b) and 8(c),
respectively. Mean group duration doesn’t seem to be influenced by group size
although higher variations are seen at smaller group sizes. The fraction of
community newcomers (first-time attending the conference) in groups is
summarized in histogram in 8(d). The figure demonstrates two peaks on both
sides of the spectrum (i.e., no newcomers vs. all newcomers in the same
group). This spread over mixed and non-mixed seniority presents opportunities
to study how acquaintance and seniority influence conversation dynamics.

##  6 Research Tasks

We report experimental results on three baseline benchmark tasks: person and
keypoints detection, speaking status detection, and F-formation detection. The
first task is a fundamental building block for automatically analyzing human
social behaviors. The other two demonstrate how learned body keypoints can be
used in the behavior analysis pipeline. We chose these benchmarking tasks
since they have been commonly studied on other in-the-wild behavior datasets.
Code for all benchmark tasks is available at: https://github.com/TUDelft-SPC-
Lab/conflab. See the Uses section of the Datasheet in the Appendix for a
discussion of the broader range of tasks ConfLab enables.

###  6.1 Person and Keypoints Detection

This benchmark involves the tasks of person detection (identifying bounding
boxes) and pose estimation (localizing skeletal keypoints). Since pre-trained
SOTA methods struggle with a privacy-sensitive top-down perspective [15] (also
see Figure 3 and Appendix F.1 for ConfLab results), we finetune COCO-
pretrained models on our dataset. We used Mask-RCNN [57] (Detectron2 framework
[58] implementation) with a ResNet-50 backbone for both tasks for
benchmarking. Since keypoint annotations were made per camera, we used four of
the overhead cameras for training (Cameras 2, 4, 8, 10) and one for testing
(Camera 6). Implementation details are available in Appendix E.1.

#### Evaluation Metrics

We evaluated person-detection performance using the standard metrics in the
MS-COCO dataset paper [59]. We report average precision (AP) for intersection
over union (IoU) thresholds of $0.50$ and $0.75$, and the mean AP from an IoU
range from $0.50$ to $0.95$ in $0.05$ increments. For keypoint detection, we
use object keypoint similarity (OKS) [59]. AP${}^{\text{OKS}}$ is a mean
average precision for different OKS thresholds from $0.5$ to $0.95$.

#### Results and Analyses

Figure 10 summarizes our person detection and joint estimation results. Our
baseline achieves 73.9 AP${}_{\text{50}}$ in detection and 45.3
AP${}^{\text{OKS}}_{\text{50}}$ in keypoint estimation. Figure 10 shows
qualitative results from our fine-tuned network.

Figure 9: Mask-RCNN results for person bounding box detection and keypoint
estimation.

Model Person Detection Keypoint Estimation AP${}_{\text{50}}$ AP
AP${}_{\text{75}}$ AP${}^{\text{OKS}}_{\text{50}}$ AP${}^{\text{OKS}}$
AP${}^{\text{OKS}}_{\text{75}}$ R50-FPN 73.9 38.9 38.4 45.3 13.5 3.3

Figure 10: Predictions from the Mask-RCNN model; COCO pretrained (left), and
ConfLab finetuned (right).

For further insight we performed several analyses and ablations. In Appendix
Table 21, we depict the effect of varying the number of training samples on
performance. For training, we use the same four cameras and only vary the
number of frames for each camera. We evaluate on the same testing images from
camera 6. We find that performance saturates at $16$% training samples. We
next investigated the effect of increasing training data size by adding
specific cameras one at a time. We report results in Appendix Table 21. There
is a $260$% performance gain when first doubling the training samples to $69$
k with the addition of camera 4, and a $46$% gain when adding another $43$ k
samples from camera 8. Finally, since the lower body regions suffer from
higher occlusion, we experiment with different sections of body for further
insight and report results in Appendix Table 23.

###  6.2 Speaking Status Detection

In data collected from real-life social settings, individual audio recordings
can be hard to obtain due to privacy concerns [60]. This has led to the
exploration of other modalities to capture some of the motion characteristics
of speaking-related gestures [35, 36]. In this task we explore the use of body
pose and wearable acceleration data for detecting the speaking status of a
person in the scene.

#### Setup

We use the SOTA MS-G3D graph neural network for skeleton action recognition
[61], pre-trained on Kinetics Skeleton 400. For the acceleration modality, we
evaluated three time series classifiers, each of which we trained from
scratch: 1D Resnet [62], InceptionTime [63], and Minirocket [64]. We performed
late fusion by averaging the scores from both modalities. Like prior work [36,
17], the task was set up as a binary classification problem. We divided our
pose (skeleton) tracks into $3$-second windows with $1.5$ s overlap. A window
was labeled positive if more than $50$% of the continuous speaking status
labels within it are positive. This resulted in an imbalanced dataset of
$42882$ windows with $29.2$% positive labels. Poses were pre-processed for
training following [61]. Three of the keypoints (head, and feet tips) were
discarded due to not being present in Kinetics. We adapted the network by
freezing all layers except for the last fully connected layer and training for
five extra epochs. Acceleration readings were not pre-processed, other than by
interpolating the original variable-sampling-rate signals to a fixed $50$ Hz.

#### Evaluation

Evaluation was carried out via $10$-fold cross-validation at the subject
level, ensuring that no examples from the test subjects were used in training.
We used the area under the ROC curve (AUC) as main evaluation metric to
account for the imbalance in the labels.

#### Results

The results in Table 3 indicate a better performance from the acceleration-
based methods. One possible reason for the lower performance of the pose-based
methods is the significant domain shift between Kinetics and Conflab,
especially in camera viewpoint (frontal vs top-down). The acceleration
performance is in line with previous work [17]. Multimodal results were
slightly higher than acceleration-only results, despite our naive fusion
approach, a possible point to improve in future work [65]. Experiments with
the rest of the IMU modalities are presented in Appendix F.2.

Table 2: ROC AUC and accuracy of skeleton-based, acceleration-based and
multimodal speaking status detection (10-fold cross-validation).

Modality Model AUC Acc. Pose MS-G3D [66] 0.676 0.677 Acceleration
InceptionTime [63] 0.798 0.768 Resnet 1D [62] 0.801 0.767 Minirocket [64]
0.813 0.768 Multimodal MS-G3D + Minirocket 0.823 0.775

Table 3: Average F1 scores for F-formation detection comparing GTCG [23] and
GCFF [67] with the effect of different threshold and orientations (standard
deviation in parenthesis).

GTCG GCFF T=2/3 T=1 T=2/3 T=1 Head 0.51 (0.09) 0.40 (0.12) 0.47 (0.07) 0.31
(0.23) Shoulder 0.46 (0.11) 0.38 (0.11) 0.56 (0.25) 0.36 (0.16) Hip 0.45
(0.10) 0.37 (0.12) 0.39 (0.06) 0.25 (0.11)

###  6.3 F-formation Detection

#### Setup

Like prior work [10, 22, 21, 23], we operationalize interaction groups using
the framework of F-formations [56]. We provide performance results for
F-formation detection using GTCG [23] and GCFF [67] as a baseline. Recent deep
learning methods such as DANTE [22] are not directly applicable since they
depend on knowing the number of people in the scene, which is variable for
ConfLab. We use pre-trained model parameters (reported in the original GTCG
and GCFF papers on the Cocktail Party dataset [13]) and tuned a subset of
parameters more relevant to ConfLab attributes on camera 6. More details can
be found in Appendix E.2. We derive three different sets of orientation
features from (i) head, (ii) shoulder and (iii) hip keypoints.

#### Evaluation Metrics

We use the standard F1 score as evaluation metric for group detection [23,
67]. A group is correctly estimated (true positive) if at least
$\left\lceil\textit{T}*|\textit{G}|\right\rceil$ of the members of group G are
correctly identified, and no more than
$1-\left\lceil\textit{T}*|\textit{G}|\right\rceil$ is incorrectly identified,
where T is the tolerance threshold. We report results for $T=\frac{2}{3}$ and
$T=1$ (more strict threshold) in Table 3.

#### Results

We show that different results are obtained using different sources of
orientations. Different occlusion levels in keypoints due to camera viewpoint
may have affected performance. Another factor influencing model performance is
that F-formations (which are driven by lower-body orientations [56]) may have
multiple conversations floors [51]. Floors are indicated by coordinated
speaker turn taking patterns and influence coordinated head orientations of
the group.

##  7 Conclusion and Discussion

ConfLab contributes a new concept for real-life data collection in the wild
and captures a high-fidelity dataset of mixed levels of acquaintance,
seniority, and personal motivations.

#### ConfLab: the Dataset

We improved upon prior work by providing higher-resolution, fidelity, and
synchronization across sensor networks. We also carefully designed our social
interaction setup to enable a diverse mix of seniority, acquaintanceship, and
motivations for mingling. The result is a rich set of $17$ body-keypoint
annotations of $48$ people at $60$ Hz from overhead cameras for developing
more robust estimation of keypoints, speaking status and F-formations for
further analyses of more complex socio-relational phenomena. Our benchmark
results for these tasks highlight how the improved fidelity of ConfLab can
assist in the development of more robust methods for these key tasks. We hope
that models trained on ConfLab for localizing keypoints would fill the gap in
the cue extraction pipeline, enabling past datasets [10, 9] without
articulated pose data to be reinvigorated; this would open the floodgates for
more robust analysis of the social phenomena labeled in these other datasets.
Finally, our baseline social tasks form the basis for further explorations
into downstream prediction tasks of socially-related constructs such as
conversation quality [68] , dominance [53], rapport [50], influence [69] etc.

#### ConfLab: the Data-Collection Concept

To relate an individual’s behaviors to trends within their social network,
further iterations of ConfLab are needed. These iterations would enable the
study of behavioral patterns at different timescales, including multiple
interactions in one day, multiple days at a conference, or across distinct
conferences. This paper serves as a template for such future ventures. We hope
that if the idea of a conference as a living lab gains traction, the effort
and cost of data collection can be amortized across different research groups,
even involving support from the conference organizers. This data by the
community for the community ethos can enable the generation of a corpus of
related datasets enabling new research questions.

#### Societal Impact

ConfLab’s long-term vision is towards developing technology to assist
individuals in navigating social interactions. In this work we have identified
choices that maximize data fidelity while upholding ethical best practices: an
overhead camera perspective that mitigates identifying faces, recording audio
at a low-frequency, and using non-intrusive wearable sensors matching a
conference badge form-factor. We argue this is an essential step towards a
long-term goal of developing personalized and socially aware technologies that
enhance social experiences. At the same time, such interventions could also
affect a community in unintended ways: worsened social satisfaction, lack of
agency, stereotyping; or benefit only those members of the community who make
use of resulting applications at the expense of the rest. More nefarious uses
involve exploiting the data for developing methods that harmfully surveil or
profile people. Researchers must consider such inadvertent effects while
developing downstream applications. Finally, since we recorded the dataset at
a scientific conference and required voluntary participation, there is an
implicit selection bias in the population represented in the data. Researchers
should be aware that insights resulting from the data may not generalize to
the general population.

#### Empowering Users Through an Agentist Rather Than Structurist Approach

The analysis of human behavior in social settings has classically taken a more
top-down perspective. For instance, the analysis of situated interactions (via
only proximity networks) has provided insight into the process of making
science in the field of Meta Science [70]. However, while social network
science is a well-populated domain, it lacks a more individualized measurement
of social behavior: see more discussion of the structure vs. agency debate
[71]. Relying on the network science approach jeopardizes an individual’s
right to technologies that enable free will. We consider the agency in
choosing such technologies to be a form of individual harm avoidance. ConfLab
provides access to more than just proximity data about social interactions,
enabling the study of context-specific social dynamics. These dynamics are a
uniquely dependent not only on the individual, but also the group they are
interacting with [72]. We hope our highlighting of participatory design
practices and these value-sensitive design principles promote social safety in
developing socially assistive technologies.

## Acknowledgements

The authors would like to thank: the ACM Multimedia 2019 General Chairs Martha
Larson, Benoit Huet, and Laurent Amsaleg for their support in making the data
collection at a major international conference a reality; Bernd Dudzik,
Yeshwanth Napolean, and Ruud de Jong for their help in setting up the
recording on-site; the participants and student volunteers for the Meet the
Chairs! event; the Amazon Mechanical Turk workers for their efforts in
annotating the dataset; Rich Radke, Martin Atzmueller, Laura Cabrera Quiros,
Alan Hanjalic, and Xucong Zhang for the insightful discussions; Santosh
Ilamparuthi for the innumerable discussions and support towards strengthening
the ethical soundness of recording and sharing ConfLab; Jan van der Heul for
the incredibly responsive support in setting up the 4TU Data repository for
ConfLab; and Bart Vastenhouw, Myrthe Tielman, and Catharine Oertel for help
with the data sharing.

ConfLab was partially funded by Netherlands Organization for Scientific
Research (NWO) under project number 639.022.606 with associated Aspasia Grant,
and also by the ACM Multimedia 2019 conference via student helpers, and crane
hiring for camera mounting.

## References

  * Dudzik et al. [2021] Bernd Dudzik, Simon Columbus, Tiffany Matej Hrkalovic, Daniel Balliet, and Hayley Hung.  Recognizing perceived interdependence in face-to-face negotiations through multimodal analysis of nonverbal behavior.  In _Proceedings of the 2021 International Conference on Multimodal Interaction_ , pages 121–130, 2021.
  * Fleeson [2007] William Fleeson.  Situation-based contingencies underlying trait-content manifestation in behavior.  _Journal of personality_ , 75:825–862, 8 2007.  ISSN 0022-3506.  doi: 10.1111/J.1467-6494.2007.00458.X.  URL https://pubmed.ncbi.nlm.nih.gov/17576360/.
  * Guardia and Ryan [2007] Jennifer G. La Guardia and Richard M. Ryan.  Why identities fluctuate: Variability in traits as a function of situational variations in autonomy support.  _Journal of Personality_ , 75:1205–1228, 12 2007.  ISSN 00223506.  doi: 10.1111/j.1467-6494.2007.00473.x.
  * Hall et al. [2019] Judith A. Hall, Terrence G. Horgan, and Nora A. Murphy.  Nonverbal communication.  _https://doi.org/10.1146/annurev-psych-010418-103145_ , 70:271–294, 1 2019.  ISSN 15452085.  doi: 10.1146/ANNUREV-PSYCH-010418-103145.  URL https://www.annualreviews.org/doi/abs/10.1146/annurev-psych-010418-103145.
  * Osborne-Crowley [2020] Katherine Osborne-Crowley.  Social cognition in the real world: reconnecting the study of social cognition with social reality.  _Review of general psychology_ , 24(2):144–158, 2020.
  * Andrade [2018] Chittaranjan Andrade.  Internal, external, and ecological validity in research design, conduct, and evaluation.  _Indian journal of psychological medicine_ , 40(5):498–499, 2018.
  * Labonte-LeMoyne et al. [2018] Élise Labonte-LeMoyne, François Courtemanche, Marc Fredette, and Pierre-Majorique Léger.  How wild is too wild: Lessons learned and recommendations for ecological validity in physiological computing research.  In _PhyCS_ , pages 123–130, 2018.
  * Hung et al. [2019a] Hayley Hung, Ekin Gedik, and Laura Cabrera Quiros.  Complex conversational scene analysis using wearable sensors.  In _Multimodal Behavior Analysis in the Wild_ , pages 225–245. Elsevier, 2019a.
  * Cabrera-Quiros et al. [2021] Laura Cabrera-Quiros, Andrew Demetriou, Ekin Gedik, Leander van der Meij, and Hayley Hung.  The matchnmingle dataset: A novel multi-sensor resource for the analysis of social interactions and group dynamics in-the-wild during free-standing conversations and speed dates.  _IEEE Transactions on Affective Computing_ , 12(1):113–130, 2021.
  * Hung and Kröse [2011] Hayley Hung and Ben Kröse.  Detecting f-formations as dominant sets.  In _Proceedings of the 13th international conference on multimodal interfaces_ , pages 231–238, 2011.
  * Alameda-Pineda et al. [2015] Xavier Alameda-Pineda, Jacopo Staiano, Ramanathan Subramanian, Ligia Batrinca, Elisa Ricci, Bruno Lepri, Oswald Lanz, and Nicu Sebe.  Salsa: A novel dataset for multimodal group behavior analysis.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 38(8):1707–1720, 2015.
  * Cristani et al. [2011] Marco Cristani, Loris Bazzani, Giulia Paggetti, Andrea Fossati, Diego Tosato, Alessio Del Bue, Gloria Menegaz, and Vittorio Murino.  Social interaction discovery by statistical analysis of f-formations.  In Jesse Hoey, Stephen J. McKenna, and Emanuele Trucco, editors, _British Machine Vision Conference, BMVC 2011, Dundee, UK, August 29 - September 2, 2011. Proceedings_ , pages 1–12. BMVA Press, 2011.  doi: 10.5244/C.25.23.  URL https://doi.org/10.5244/C.25.23.
  * Zen et al. [2010] Gloria Zen, Bruno Lepri, Elisa Ricci, and Oswald Lanz.  Space speaks: towards socially and personality aware visual surveillance.  In _Proceedings of the 1st ACM international workshop on Multimodal pervasive video analysis_ , pages 37–42, 2010.
  * Murgia [2019] Madhumita Murgia.  Who’s using your face? the ugly truth about facial recognition.  _Financial Times_ , 2019.
  * Carissimi et al. [2018] Nicolo Carissimi, Paolo Rota, Cigdem Beyan, and Vittorio Murino.  Filling the gaps: Predicting missing joints of human poses using denoising autoencoders.  In _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_ , pages 0–0, 2018.
  * Gedik and Hung [2018] Ekin Gedik and Hayley Hung.  Detecting conversing groups using social dynamics from wearable acceleration: Group size awareness.  _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ , 2(4), dec 2018.  doi: 10.1145/3287041.  URL https://doi.org/10.1145/3287041.
  * Gedik and Hung [2017] Ekin Gedik and Hayley Hung.  Personalised models for speech detection from body movements using transductive parameter transfer.  _Personal and Ubiquitous Computing_ , 21(4):723–737, August 2017.  ISSN 1617-4909.  doi: 10.1007/s00779-017-1006-4.
  * Raman et al. [2020] Chirag Raman, Stephanie Tan, and Hayley Hung.  A modular approach for synchronized wireless multimodal multisensor data acquisition in highly dynamic social settings.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , MM ’20, page 3586–3594, New York, NY, USA, 2020. Association for Computing Machinery.  ISBN 9781450379885.  doi: 10.1145/3394171.3413697.  URL https://doi.org/10.1145/3394171.3413697.
  * Cai et al. [2020] Yuanhao Cai, Zhicheng Wang, Zhengxiong Luo, Binyi Yin, Angang Du, Haoqian Wang, Xiangyu Zhang, Xinyu Zhou, Erjin Zhou, and Jian Sun.  Learning delicate local representations for multi-person pose estimation.  In _European Conference on Computer Vision_ , pages 455–472. Springer, 2020.
  * Birhane and Prabhu [2021] Abeba Birhane and Vinay Uday Prabhu.  Large image datasets: A pyrrhic win for computer vision?  In _IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021_ , pages 1536–1546. IEEE, 2021\.  doi: 10.1109/WACV48630.2021.00158.  URL https://doi.org/10.1109/WACV48630.2021.00158.
  * Setti et al. [2015] Francesco Setti, Chris Russell, Chiara Bassetti, and Marco Cristani.  F-formation detection: Individuating free-standing conversational groups in images.  _PloS one_ , 10(5):e0123783, 2015.
  * Swofford et al. [2020] Mason Swofford, John Peruzzi, Nathan Tsoi, Sydney Thompson, Roberto Martín-Martín, Silvio Savarese, and Marynel Vázquez.  Improving social awareness through dante: Deep affinity network for clustering conversational interactants.  _Proceedings of the ACM on Human-Computer Interaction_ , 4(CSCW1):1–23, 2020.
  * Vascon et al. [2014] Sebastiano Vascon, Eyasu Zemene Mequanint, Marco Cristani, Hayley Hung, Marcello Pelillo, and Vittorio Murino.  A game-theoretic probabilistic approach for detecting conversational groups.  In _Asian conference on computer vision_ , pages 658–675. Springer, 2014.
  * Joo et al. [2017] Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Scott Godisart, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh.  Panoptic studio: A massively multiview system for social interaction capture.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2017.
  * Vondrick et al. [2013] Carl Vondrick, Donald Patterson, and Deva Ramanan.  Efficiently Scaling up Crowdsourced Video Annotation.  _International Journal of Computer Vision_ , 101(1):184–204, 2013.  ISSN 0920-5691.  doi: 10.1007/s11263-012-0564-1.
  * Ricci et al. [2015] Elisa Ricci, Jagannadan Varadarajan, Ramanathan Subramanian, Samuel Rota Bulò, Narendra Ahuja, and Oswald Lanz.  Uncovering interactions and interactors: Joint estimation of head, body orientation and f-formations from surveillance videos.  In _2015 IEEE International Conference on Computer Vision (ICCV)_ , pages 4660–4668, 2015.
  * Bazzani et al. [2013] Loris Bazzani, Marco Cristani, Diego Tosato, Michela Farenzena, Giulia Paggetti, Gloria Menegaz, and Vittorio Murino.  Social interactions by visual focus of attention in a three-dimensional environment.  _Expert Systems_ , 30(2):115–127, 2013.
  * Carletta et al. [2006] Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, Iain McCowan, Wilfried Post, Dennis Reidsma, and Pierre Wellner.  The ami meeting corpus: A pre-announcement.  In Steve Renals and Samy Bengio, editors, _Machine Learning for Multimodal Interaction_ , pages 28–39, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
  * Cattuto et al. [2010] C. Cattuto, W. V. D. Broeck, A. Barrat, V. Colizza, J. Pinton, and Alessandro Vespignani.  Dynamics of person-to-person interactions from distributed rfid sensor networks.  _PLoS ONE_ , 5, 2010.
  * Hoffman et al. [2020] Marion Hoffman, Per Block, Timon Elmer, and Christoph Stadtfeld.  A model for the dynamics of face-to-face interactions in social groups.  _Network Science_ , 8(S1):S4–S25, 2020.  doi: 10.1017/nws.2020.3.
  * Atzmueller and Lemmerich [2018] Martin Atzmueller and Florian Lemmerich.  Homophily at academic conferences.  In _Companion Proceedings of the The Web Conference 2018_ , WWW ’18, page 109–110, Republic and Canton of Geneva, CHE, 2018. International World Wide Web Conferences Steering Committee.
  * Olguín et al. [2008] Daniel Olguín Olguín, Benjamin N Waber, Taemie Kim, Akshay Mohan, Koji Ara, and Alex Pentland.  Sensible organizations: Technology and methodology for automatically measuring organizational behavior.  _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ , 39(1):43–55, 2008.
  * Chaffin et al. [2017] Daniel Chaffin, Ralph Heidl, John R Hollenbeck, Michael Howe, Andrew Yu, Clay Voorhees, and Roger Calantone.  The promise and perils of wearable sensors in organizational research.  _Organizational Research Methods_ , 20(1):3–31, 2017.
  * Rosatelli et al. [2019] Alessio Rosatelli, Ekin Gedik, and Hayley Hung.  Detecting f-formations roles in crowded social scenes with wearables: Combining proxemics dynamics using lstms.  In _2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)_ , pages 147–153, 2019.  doi: 10.1109/ACIIW.2019.8925179.
  * Cabrera-Quiros et al. [2018] Laura Cabrera-Quiros, David M.J. Tax, and Hayley Hung.  Gestures in-the-wild : Detecting conversational hand gestures in crowded scenes using a multimodal fusion of bags of video trajectories and body worn acceleration.  pages 1–10, 2018.
  * Quiros and Hung [2019] J. V. Quiros and H. Hung.  CNNs and Fisher Vectors for No-Audio Multimodal Speech Detection.  In _MediaEval_ , 2019.
  * Tan et al. [2021a] Stephanie Tan, David M. J. Tax, and Hayley Hung.  Multimodal joint head orientation estimation in interacting groups via proxemics and interaction dynamics.  _Proc. ACM Interactive, Mobile, Wearable, and Ubiquitous Technology_ , 5(1), March 2021a.
  * [38] University of york research data management.  https://www.york.ac.uk/library/info-for/researchers/data/sharing/access/.
  * [39] Utrecht university research data management.  https://www.uu.nl/en/research/research-data-management/guides/handling-personal-data.
  * [40] Go pro hero 7 black.  https://gopro.com/en/nl/shop/cameras/hero7-black/CHDHX-701-master.html.
  * Lederman et al. [2018] Oren Lederman, Akshay Mohan, Dan Calacci, and Alex Sandy Pentland.  Rhythm: A unified measurement platform for human organizations.  _IEEE MultiMedia_ , 25(1):26–38, 2018.
  * Ringeval et al. [2013] F. Ringeval, A. Sonderegger, J. Sauer, and D. Lalanne.  Introducing the recola multimodal corpus of remote collaborative and affective interactions.  In _2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)_ , pages 1–8, 2013.
  * Cabrera-Quiros and Hung [2016] Laura Cabrera-Quiros and Hayley Hung.  Who is where? matching people in video to wearable acceleration during crowded mingling events.  In _Proceedings of the 24th ACM international conference on Multimedia_ , pages 267–271, 2016.
  * Cabrera-Quiros and Hung [2018] Laura Cabrera-Quiros and Hayley Hung.  A hierarchical approach for associating body-worn sensors to video regions in crowded mingling scenarios.  _IEEE Transactions on Multimedia_ , 21(7):1867–1879, 2018.
  * Hung et al. [2019b] Hayley Hung, Chirag Raman, Ekin Gedik, Stephanie Tan, and Jose Vargas Quiros.  Multimodal data collection for social interaction analysis in-the-wild.  In _Proceedings of the 27th ACM International Conference on Multimedia_ , pages 2714–2715, 2019b.
  * [46] Computer Vision Annotation Tool (CVAT).
  * [47] Covfee: Continuous Video Feedback Tool.  Jose Vargas.
  * Vargas Quiros et al. [2022] Jose Vargas Quiros, Stephanie Tan, Chirag Raman, Laura Cabrera-Quiros, and Hayley Hung.  Covfee: an extensible web framework for continuous-time annotation of human behavior.  In Cristina Palmero, Julio C. S. Jacques Junior, Albert Clapés, Isabelle Guyon, Wei-Wei Tu, Thomas B. Moeslund, and Sergio Escalera, editors, _Understanding Social Behavior in Dyadic and Small Group Interactions_ , volume 173 of _Proceedings of Machine Learning Research_ , pages 265–293. PMLR, 16 Oct 2022.  URL https://proceedings.mlr.press/v173/vargas-quiros22a.html.
  * Gatica-Perez [2006] Daniel Gatica-Perez.  Analyzing group interactions in conversations: a review.  In _2006 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems_ , pages 41–46, 2006.  doi: 10.1109/MFI.2006.265658.
  * Müller et al. [2018] Philipp Müller, Michael Xuelin Huang, and Andreas Bulling.  Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behaviour.  In _23rd International Conference on Intelligent User Interfaces_. ACM, 2018.  ISBN 978-1-4503-4945-1.  doi: 10.1145/3172944.3172969.
  * Raman and Hung [2019] Chirag Raman and Hayley Hung.  Towards automatic estimation of conversation floors within F-formations.  _arXiv:1907.10384 [cs]_ , July 2019.
  * Hung and Gatica-Perez [2010] Hayley Hung and Daniel Gatica-Perez.  Estimating cohesion in small groups using audio-visual nonverbal behavior.  _IEEE Transactions on Multimedia_ , 12(6):563–575, 2010.
  * Hung et al. [2011] Hayley Hung, Yan Huang, Gerald Friedland, and Daniel Gatica-Perez.  Estimating Dominance in Multi-Party Meetings Using Speaker Diarization.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 19(4):847–860, may 2011.
  * Beyan et al. [2020] Cigdem Beyan, Muhammad Shahid, and Vittorio Murino.  RealVAD: A Real-world Dataset and A Method for Voice Activity Detection by Body Motion Analysis.  _x_ , 9210(c):1–16, 2020.  doi: 10.1109/tmm.2020.3007350.
  * Shahid et al. [2019] Muhammad Shahid, Cigdem Beyan, and Vittorio Murino.  Voice activity detection by upper body motion analysis and unsupervised domain adaptation.  _Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019_ , pages 1260–1269, 2019.  doi: 10.1109/ICCVW.2019.00159.
  * Kendon [1990] Adam Kendon.  _Conducting interaction: Patterns of behavior in focused encounters_ , volume 7.  CUP Archive, 1990.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * Wu et al. [2019] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer, 2014.
  * Shen et al. [2018] Jiaxing Shen, Oren Lederman, Jiannong Cao, Florian Berg, Shaojie Tang, and Alex Sandy Pentland.  GINA: Group Gender Identification Using Privacy-Sensitive Audio Data.  _Proceedings - IEEE International Conference on Data Mining, ICDM_ , 2018-Novem:457–466, 2018.  ISSN 15504786.  doi: 10.1109/ICDM.2018.00061.
  * Gupta et al. [2020] Pranay Gupta, Anirudh Thatipelli, Aditya Aggarwal, Shubh Maheshwari, Neel Trivedi, Sourav Das, and Ravi Kiran Sarvadevabhatla.  Quo Vadis, Skeleton Action Recognition ?  _arXiv:2007.02072 [cs]_ , July 2020.
  * Wang et al. [2016] Zhiguang Wang, Weizhong Yan, and Tim Oates.  Time series classification from scratch with deep neural networks: A strong baseline, 2016.  URL https://arxiv.org/abs/1611.06455.
  * Wang et al. [2017] Zhiguang Wang, Weizhong Yan, and Tim Oates.  Time series classification from scratch with deep neural networks: A strong baseline.  _Proceedings of the International Joint Conference on Neural Networks_ , 2017-May:1578–1585, 2017.  doi: 10.1109/IJCNN.2017.7966039.
  * Tan et al. [2021b] Chang Wei Tan, Angus Dempster, Christoph Bergmeir, and Geoffrey I. Webb.  Multirocket: Multiple pooling operators and transformations for fast and effective time series classification, 2021b.  URL https://arxiv.org/abs/2102.00457.
  * Baltrusaitis et al. [2019] Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency.  Multimodal machine learning: A survey and taxonomy.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 41(2):423–443, February 2019.
  * Liu et al. [2020] Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C. Kot.  NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 42(10):2684–2701, October 2020.  ISSN 0162-8828, 2160-9292, 1939-3539.  doi: 10.1109/TPAMI.2019.2916873.
  * Cristani et al. [2013] Marco Cristani, Ramachandra Raghavendra, Alessio Del Bue, and Vittorio Murino.  Human behavior analysis in video surveillance: A social signal processing perspective.  _Neurocomputing_ , 100:86–97, 2013.
  * Raman et al. [2022a] Chirag Raman, Navin Raj Prabhu, and Hayley Hung.  Perceived conversation quality in spontaneous interactions, 2022a.  URL https://arxiv.org/abs/2207.05791.
  * Dong et al. [2007] Wen Dong, Bruno Lepri, Alessandro Cappelletti, Alex Sandy Pentland, Fabio Pianesi, and Massimo Zancanaro.  Using the influence model to recognize functional roles in meetings.  In _Proceedings of the 9th international conference on Multimodal interfaces_ , pages 271–278, 2007.
  * Eberle et al. [2021] Julia Eberle, Karsten Stegmann, Alain Barrat, Frank Fischer, and Kristine Lund.  Initiating scientific collaborations across career levels and disciplines–a network analysis on behavioral data.  _International Journal of Computer-Supported Collaborative Learning_ , 16(2):151–184, 2021.
  * Pleasants [2019] Nigel Pleasants.  Free will, determinism and the “problem” of structure and agency in the social sciences.  _Philosophy of the Social Sciences_ , 49(1):3–30, 2019.
  * Raman et al. [2021] Chirag Raman, Hayley Hung, and Marco Loog.  Social Processes: Self-Supervised Meta-Learning over Conversational Groups for Forecasting Nonverbal Social Cues.  _arXiv:2107.13576 [cs]_ , July 2021.
  * Gebru et al. [2021] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford.  Datasheets for datasets.  _Communications of the ACM_ , 64(12):86–92, 2021\.
  * Pineau et al. [2021] Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle.  Improving reproducibility in machine learning research: a report from the neurips 2019 reproducibility program.  _Journal of Machine Learning Research_ , 22, 2021.
  * Barquero et al. [2022] German Barquero, Johnny Núnez, Sergio Escalera, Zhen Xu, Wei-Wei Tu, Isabelle Guyon, and Cristina Palmero.  Didn’t see that coming: a survey on non-verbal social human behavior forecasting.  In _Understanding Social Behavior in Dyadic and Small Group Interactions_ , pages 139–178. PMLR, 2022.
  * Raman et al. [2022b] Chirag Raman, Hayley Hung, and Marco Loog.  Why did this model forecast this future? closed-form temporal saliency towards causal explanations of probabilistic forecasts.  _arXiv preprint arXiv:2206.00679_ , 2022b.
  * Hung et al. [2013] Hayley Hung, Gwenn Englebienne, and Jeroen Kools.  Classifying social actions with a single accelerometer.  In _Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing_ , pages 207–210, 2013.
  * Raj Prabhu et al. [2020] Navin Raj Prabhu, Chirag Raman, and Hayley Hung.  Defining and quantifying conversation quality in spontaneous interactions.  In _Companion Publication of the 2020 International Conference on Multimodal Interaction_ , pages 196–205, 2020.
  * Quiros et al. [2021] Jose David Vargas Quiros, Oyku Kapcak, Hayley Hung, and Laura Cabrera-Quiros.  Individual and joint body movement assessed by wearable sensing as a predictor of attraction in speed dates.  _IEEE Transactions on Affective Computing_ , 2021.
  * OpenCV [2015] OpenCV.  Open source computer vision library.  https://github.com/opencv/opencv, 2015.
  * [81] Idiap multi camera calibration suite.  https://github.com/idiap/multicamera-calibration.
  * [82] Tdkicm20948.  https://invensense.tdk.com/products/motion-tracking/9-axis/icm-20948/.  Accessed: 2021-10-15.
  * Ba and Odobez [2010] Sileye O Ba and Jean-Marc Odobez.  Multiperson visual focus of attention from head pose and meeting contextual cues.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 33(1):101–116, 2010.
  * Li et al. [2019] Wenbo Li, Zhicheng Wang, Binyi Yin, Qixiang Peng, Yuming Du, Tianzi Xiao, Gang Yu, Hongtao Lu, Yichen Wei, and Jian Sun.  Rethinking on multi-stage networks for human pose estimation.  _arXiv preprint arXiv:1901.00148_ , 2019.
  * Cheng et al. [2020] Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S. Huang, and Lei Zhang.  Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Newell et al. [2017] Alejandro Newell, Zhiao Huang, and Jia Deng.  Associative embedding: End-to-end learning for joint detection and grouping.  _Advances in neural information processing systems_ , 30, 2017.
  * Oguiza [2022] Ignacio Oguiza.  tsai - a state-of-the-art deep learning library for time series and sequential data.  Github, 2022.  URL https://github.com/timeseriesAI/tsai.

ConfLab: A Data Collection Concept, Dataset, and
Benchmark for Machine Analysis of Free-Standing
Social Interactions in the Wild

Appendices

The Appendices are organized as follows:

  * •

Section A: Hosting, licensing, and organization information for ConfLab

  * •

Section B: Documentation for ConfLab, following Datasheets for Datasets [73]

  * •

Section C: Sample post-hoc behavioral analysis report sent to each ConfLab
participant

  * •

Section D: Details about out data-capture setup

  * •

Section E: Implementation details for models used in our benchmark research
tasks

  * •

Section F: Additional experimental results and ablations

  * •

Section G: Details for reproducibility following the ML Reproducibility
Checklist [74]

##  Appendix A Hosting, Licensing, and Organization

The dataset is hosted by 4TU.ResearchData, available at
https://doi.org/10.4121/c.6034313.

The dataset itself is available under restricted access defined by an End-User
License Agreement (EULA). The EULA itself is available under a CC0 license.
The code (https://github.com/TUDelft-SPC-Lab/conflab) for the benchmark
baseline tasks, and the schematics and data associated with the design of our
custom wearable sensor called the Midge (https://github.com/TUDelft-SPC-
Lab/spcl_midge_hardware) are available under the MIT License.

Figure 11 on the next page illustrates the organization of the ConfLab dataset
on 4TU.ResearchData. The components are as follows:

  * •

Annotations (restricted, https://doi.org/10.4121/20017664):
annotations of pose, speaking status, and F-formations

  * •

Datasheet for ConfLab (public, https://doi.org/10.4121/20017559):
documentation of the dataset following Datasheets for Datasets [73] (see
Appendix B)

  * •

EULA (public, https://doi.org/10.4121/20016194):
End User License Agreement to be signed for requesting access to the
restricted components

  * •

Processed-Data (restricted, https://doi.org/10.4121/20017805):
processed video and wearable sensor used for annotations

  * •

Raw-Data (restricted, https://doi.org/10.4121/20017748):
raw video and wearable sensor data

  * •

Data Samples (restricted, https://doi.org/10.4121/20017682):
samples of the sensor, audio, and video data

Figure 11: File structure of the ConfLab dataset

##  Appendix B Datasheet For ConfLab

This document is based on Datasheets for Datasets by Gebru et al. [73]. Please
see the most updated version here.

MOTIVATION

Q. For what purpose was the dataset created? Was there a specific task in
mind? Was there a specific gap that needed to be filled? Please provide a
description.

There are two broad motivations for creating this dataset: first, to enable
the privacy-preserving, multimodal study of real-life social conversation
dynamics; second, to bring the higher fidelity of wired in-the-lab recording
setups to in-the-wild scenarios, enabling the study of fine time-scale social
dynamics in-the-wild.

We propose the Conference Living Lab (ConfLab) with the following goals: (i) a
data collection effort that follows a by the community for the community
ethos: the more volunteers, the more data, (ii) volunteers who potentially use
the data can experience first-hand potential privacy and ethical
considerations related to sharing their own data, (iii) in light of recent
data sourcing issues [20], we incorporated privacy and invasiveness
considerations directly into the decision-making process regarding sensor
type, positioning, and sample-rates.

From a technical perspective, closest related datasets (see Table 1 in the
main paper) suffer from several technical limitations precluding the analysis
and modeling of fine-grained social behavior: (i) lack of articulated pose
annotations; (ii) a limited number of people in the scene, preventing complex
interactions such as group splitting/merging behaviors, and (iii) an
inadequate data sampling-rate and synchronization-latency to study time-
sensitive social phenomena [18, Sec. 3.3]. This often requires modeling
simplifications such as the summarizing of features over rolling windows [17,
35, 36]. On the other hand, past high-fidelity datasets have largely involved
role-played or scripted interactions in lab settings, with often a single-
group in the scene.

This dataset wasn’t created with a specific task in mind, but intends to
support a wide variety of multimodal modeling and analysis tasks across
research domains (see the Uses section).

Q. Who created this dataset (e.g., which team, research group) and on behalf
of which entity (e.g., company, institution, organization)?

ConfLab was initiated by the Socially Perceptive Computing Lab, Delft
University of Technology in cooperation and support from the general chairs of
ACM Multimedia 2019 (Martha Larson, Benoit Huet, and Laurent Amsaleg), Nice,
France. Since this dataset was by the community, for the community, members of
the Multimedia community contributed as subjects in the dataset.

Q. What support was needed to make this dataset?  (e.g.who funded the creation
of the dataset? If there is an associated grant, provide the name of the
grantor and the grant name and number, or if it was supported by a company or
government agency, give those details.)

ConfLab was partially funded by Netherlands Organization for Scientific
Research (NWO) under project number 639.022.606 with associated Aspasia Grant,
and also by the ACM Multimedia 2019 conference via student helpers, and crane
hiring for camera mounting.

Q. Any other comments?
None.

COMPOSITION

Q. What do the instances that comprise the dataset represent (e.g., documents,
photos, people, countries)? Are there multiple types of instances (e.g.,
movies, users, and ratings; people and interactions between them; nodes and
edges)? Please provide a description.

The dataset contains multimodal recordings of people interacting during a
networking event embedded in an international multimodal machine learning
conference.

Overall, the interaction scene contained conversation groups (operationalized
as f-formations), composed of individual subjects, each of which had
individual data associated to their wearable sensors. The complete interaction
scene was additionally captured by overhead cameras. Figure 12 shows the
structure of these instances and their relationships.

Figure 12: Structure of some of the instances in the dataset and their
relationships. The interaction space was captured via overhead videos, in
which f-formations (conversation groups) were annotated. An F-formation
consists of set of people interacting for a variable period of time, and
identified via a subject ID. Each person in the F-formation can be associated
to their pose (annotated in the videos), their wearable sensor (IMU) data, and
their action (speaking status) labels.

Note however that the precise notion of what constitutes an instance in the
dataset is very much task-specific. In our baseline tasks we considered the
following instances:

Person and Keypoints Detection

Frames, containing pose annotations ($17$ body keypoints per person per frame
@$60$ Hz) from $5$ overhead videos ($1920\times 1080,60$ fps) for $16$ minutes
of interaction.

Speaking Status Detection

Windows ($3$ seconds) of wearable sensor data and speaking status annotations
($60$ Hz) extracted from each subject’s data.

F-formations

Operationalized conversation groups, annotated at $1$ Hz from the $16$ minutes
of annotated data, and the pose data associated to the people in the
F-formation.

Q. How many instances are there in total (of each type, if appropriate)?
The notion of instance is very much dependent on how a user intends to use the
data. Regarding the instances in Figure 12, our full dataset consist of 45
minutes of:

Video recordings

from $10$ overhead cameras placed over the interaction area. Five of these
videos, enough to cover the complete interaction area, were used in
annotation.

Individual wearable sensor data

For the $48$ subjects in the interaction area, a chest-worn conference-type
badge recorded: audio ($1250$ Hz), and Inertial Measurement Unit (IMU)
readings (accelerometer @ $56$ Hz, gyroscope @$56$ Hz, magnetometer @$56$ Hz
and Bluetooth RSSI-based proximity @$5$ Hz)

Conference experience label

For each of the $48$ subjects, an associated self-report label indicating
whether it was their first time in the conference.

The instances in the annotated $16$ minutes segment out of the $45$ minutes of
interaction contain:

2D body poses

For each of the $48$ subjects, full body pose tracks annotated at $60$Hz ($17$
keypoints per person). These were annotated using $5$ of the $10$ overhead
cameras due to the significant overlap in views (cameras $2$, $4$, $6$, $8$,
and $10$). Annotations were done separately for each camera by annotating all
of the people visible in each video, for each of the $5$ cameras, and tagged
with a participant ID. We made use of a novel continuous technique for
annotation of keypoints. We chose this approach via a pilot study with 3
annotators, comparing our technique to annotations done using the non-
continuous CVAT tool. We found no statistically significant differences in
errors per-frame (as measured using Mean Squared Error across annotators),
despite a 3x speed-up in annotation time in the continuous condition. The
details of the technique and this pilot study can be found in [48].

Speaking status annotations

For each of the $48$ subjects, these include a) a binary signal ($60$ Hz)
indicating whether the person is perceived to be speaking or not; b)
continuous confidence value ($60$ Hz) indicating the degree of confidence of
the annotator in their speaking status assessment. These annotations were done
without access to audio due to issues with the synchronization of the audio
recordings at the time of annotation. The confidence assessment is therefore
largely based on the visibility of the target person and their speaking-
associated gestures (eg. occlusion, orientation w.r.t. camera, visibility of
the face)? We measured inter-annotator agreement for speaking status in a
pilot where two annotators labeled three data subjects for 2 minutes each. We
measured a frame-level agreement (Fleiss’ $\kappa$) of 0.552, comparable to
previous work [35].

F-formation annotations

These annotations label the conversing groups in the scene following previous
work. Each individual belongs to one F-formation at a time or is a singleton
in the interaction scene. The membership is binary. The annotations were done
by one of the authors at $1$ Hz by watching the video. The time-stamped usage
of mobile phones are available as auxiliary annotations, which are useful for
the study of the role of mobile phone users as associates of F-formations.
Since Kendon’s theories date back to before the widespread use of mobile
phones, their influence on F-formation membership remains an open question.

In our baseline tasks, which made use of the complete annotated section of the
dataset, the instance numbers were the following:

Person and Keypoints Detection

$119$k frames ($60$fps) containing $1967$k person instances (poses) in total,
from $48$ subjects recorded in $5$ cameras ($16$ minutes of annotated
segment).

Speaking Status Detection

$42884$ $3$-second windows, extracted from the $48$ participants’ wearable
data and speaking status annotations.

F-formations

$119$ conversation groups. Details are in Section 5.

Q. Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set? If the dataset is a
sample, then what is the larger set? Is the sample representative of the
larger set (e.g., geographic coverage)? If so, please describe how this
representativeness was validated/verified. If it is not representative of the
larger set, please describe why not (e.g., to cover a more diverse range of
instances, because instances were withheld or unavailable).

The participants in our data collection are a sample of the conference
attendees. Participants were recruited via the conference website, social
media posting, and approaching them in person during the conference. Because
participation in such a data collection can only be voluntary, the sample was
not pre-designed and may not be representative of the larger set.

Additionally, $16$ minutes of sensor data has been annotated for keypoints,
speaking status and F-formations out of the total of $45$ minutes recorded.
The remaining part (across all modalities) is provided with no labels. For
privacy reasons, the elevated cameras (distinct from the previously mentioned
$8$ overhead cameras) and also individual frontal headshots that were used for
manually associating the video data to the wearable sensor data is not being
shared.

Q. Is any information missing from individual instances?  If so, please
provide a description, explaining why this information is missing (e.g.,
because it was unavailable). This does not include intentionally removed
information, but might include, e.g., redacted text.

Camera $5$ failed early during the recording, but the space underneath it was
captured by the adjacent cameras due to the high overlap in the camera field-
of-views. Nevertheless we share what was recorded before the failure from
camera $5$, bringing the total number of cameras to $9$.

Q. Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)?  If so, please describe how these
relationships are made explicit.

The F-formations, subjects, and their associated data relate as shown in
Figure 12. These associations are made explicit in the dataset via anonymous
subject IDs, associated to pose tracks, speaking status annotations, and
wearable sensor data. These same IDs were used to annotate the F-formations.

Pre-existing personal relationships between the subjects were not requested
for privacy reasons.

Q. Are there recommended data splits (e.g., training, development/validation,
testing)?

Since the dataset can be used to study a variety of tasks, the answer to this
question is task dependent. Please refer to our reproducibility details
(Appendix G of our associated paper) for information about the splits that we
used in out baselines.

Q. Are there any errors, sources of noise, or redundancies in the dataset?  If
so, please provide a description.

Individual audio

Because audio was recorded by a front-facing wearable device worn on the
chest, it contains a significant amount of cocktail party noise and cross-
contamination from other people in the scene. In our experience this means
that automatic speaking status detection is challenging with existing
algorithms but manual annotation is possible.

Videos and 2D body poses

It is important to consider that the same person may appear in multiple videos
at the same time if the person was in view of multiple cameras. Because 2D
poses were annotated per video, the same is true of pose annotations. Each
skeleton was tagged with a person ID, which should serve to identify such
cases when necessary.

Q. Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)?

The dataset is self-contained.

Q. Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor-patient
confidentiality, data that includes the content of individuals’ non-public
communications)?

The data contains personal data under GDPR in the form of video and audio
recordings of subjects. The dataset is shared under an End User License
Agreement for research purposes, to ensure that the data is not made public,
and to protect the privacy of data subjects.

Q. Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety?

No.

Q. Does the dataset relate to people?

Yes, the dataset contains recordings of human subjects.

Q. Does the dataset identify any subpopulations (e.g., by age, gender)?  If
so, please describe how these subpopulations are identified and provide a
description of their respective distributions within the dataset.

Data subjects answered the following questions before the start of the data
collection event, after filling in their consent form:

  * •

Is this your first time attending ACM MM?

  * •

Select the area(s) that describes best your research interest(s) in recent
years. Descriptions of each theme are listed here: https://acmmm.org/call-for-
papers/

Figure 13 shows the distribution of the responses / populations.

Figure 13: Distribution of participant seniority (left) and research interests
(right) in percentage.

Q. Is it possible to identify individuals (i.e., one or more natural persons),
either directly or indirectly (i.e., in combination with other data) from the
dataset?

We do not share any directly identifiable information as part of the dataset.
However, individuals may be identified in the video recordings if the observer
knows the participants in the recordings personally. Otherwise, individuals in
the dataset may potentially be identified in combination with publicly
available pictures or videos (from conference attendees or conference official
photographer) from other media from the conference the dataset was recorded
at. In any case, re-identifying the subjects is strictly against the End User
License Agreement under which we share the dataset.

Q. Does the dataset contain data that might be considered sensitive in any way
(e.g., data that reveals racial or ethnic origins, sexual orientations,
religious beliefs, political opinions or union memberships, or locations;
financial or health data; biometric or genetic data; forms of government
identification, such as social security numbers; criminal history)?

We did not request any such information from data participants. Here, the ACM
Multimedia ’19 General Chair Martha Larson also helped advocate on behalf of
the attendees during the survey-design stage. As a result of these
discussions, information such as participant gender, ethnicity, or country of
origin was not asked.

Q. Any other comments?
None.

COLLECTION

Q. How was the data associated with each instance acquired?  Was the data
directly observable (e.g., raw text, movie ratings), reported by subjects
(e.g., survey responses), or indirectly inferred/derived from other data
(e.g., part-of-speech tags, model-based guesses for age or language)? If data
was reported by subjects or indirectly inferred/derived from other data, was
the data validated/verified? If so, please describe how.

The collected data is directly observable, containing video recordings, low-
frequency audio recordings and wearable sensing signals (inertial motion unit
(IMU) and Bluetooth proximity sensors) of individuals in the interaction
scenes. Accompanying data includes self-reported binary categorization of
experience level which is available upon request from the authors. The self-
reported interests categories are not shared because of privacy concerns.

Video recordings capture the whole interaction floor where the association
from multi-modal data to individual is done manually by annotators by
referring to frontal (not-shared) and overhead views. The rest of the data was
acquired from the wearable sensing badges, which is person-specific (i.e., no
participant shared the device). Video and audio data were verified in
playback. Wearable sensing data was verified through plots after parsing.

Q. Over what timeframe was the data collected?  Does this timeframe match the
creation timeframe of the data associated with the instances (e.g., recent
crawl of old news articles)? If not, please describe the timeframe in which
the data associated with the s was created. Finally, list when the dataset was
first published.

All data was collected on October 24, 2019, except the self-reported
experience level and research interest topics which are either obtained on the
same day or not more than one week before the data collection day. This time
frame matches the creation time frame of the data association for wearable
sensing data. Video data was associated with individual during annotation
stage (2020-2021), but all information used for association was obtained on
the data collection day.

Q. What mechanisms or procedures were used to collect the data (e.g., hardware
apparatus or sensor, manual human curation, software program, software API)?

To record videos, we used 14 GoPro Hero 7 Black cameras. The wearable sensor
hardware has been documented and open-sourced at https://github.com/TUDelft-
SPC-Lab/spcl_midge_hardware. The validation of the sensors was completed
through an external contractor engineer. The data collection software was
documented and published in [48], which includes validation of the system.
These hardwares and mechanisms have been open-sourced along with their
respective publication.

The synchronization setup for data collection (intramodal and intermodal) was
documented and published in [18], which includes validation of the system.

To lend the reader further insight into the process of setting up the
recording of such datasets in-the-wild, we share images of our process in
Figure 14.

(a) Aligning cameras

(b) Affixing the mounting beam

(c) Aligning floor markers

(d) Marking the floor grid

(e) Interaction area

(f) Verifying camera sync.

(g) Assembling Midges

(h) Midges

(i) Verifying crossmodal sync.

Figure 14: Illustrating the process of setting up the data recording.

Q. What was the resource cost of collecting the data?

Table 4: Itemized costs associated with recording ConfLab Item |  | Cost (USD)
---|---|---
Travel (total for 6 people) |  |
Flights |  | 1800
Accommodation |  | 1500
Equipment (one time) |  |
Mounting scaffold |  | 2000
14 $\times$ GoPro Hero 7 Black |  | 4900
Designing the Midge (custom wearable, now made open source) |  | 26000
110 $\times$ Midges (boards, batteries, 4 GB sd cards, cases) |  | 3660
Multimodal synchronization setup |  | 730
Annotations |  | 8000
Computational cost for experiments |  | 500

The resources required to run this first edition of ConfLab include equipment,
logistics, and travel costs. Table 4 shows the full breakdown of the costs.
The equipment expenses are fixed one-time costs since the same equipment can
be used for future iterations of ConfLab. The on-site costs at the conference
venue were toward renting a crane for a day to mount the cameras on a scaffold
on the ceiling. We have open-sourced the Midge (our custom wearable)
schematics so that others don’t need to spend on the design and development.

No additional energy consumption was incurred for collecting the data.
However, the ancillary activities (e.g., flights, accommodation) resulted in
energy consumption. Flights from the Netherlands to France round-trip for six
passengers results in 1020 kg carbon emissions. Accommodation for six members
resulted in 22 kWh energy consumption.

Q. If the dataset is a sample from a larger set, what was the sampling
strategy (e.g., deterministic, probabilistic with specific sampling
probabilities)?

ConfLab contains both annotated and unannotated segments of multi-modal data.
The segment where the articulated pose and speaking status were annotated is
selected to maximize crowd density in the scenes. The annotated segment is
$16$ minutes; the whole set is roughly $1$ hour of recordings.

Q. Who was involved in the data collection process (e.g., students,
crowdworkers, contractors) and how were they compensated (e.g., how much were
crowdworkers paid)?

The Conflab dataset was captured during a special social event called Meet the
Chairs! at an international conference on signal processing and machine
learning. Newcomers and old-timers to the conference freely donated their
social behaviour data as part of a by the community, for the community data
collection effort. Aside from the chance to meet the chairs and create a
community dataset, the attendees also received a personalised report of their
social behaviour from the wearable sensors (see Appendix C) Conference student
volunteers were involved in assisting the set-up of the event. Conference
organizers (mentioned in the Motivation section) assisted in connecting us
with conference venue contacts to mount our technical set-ups in the room.
Volunteers and conference organizers were not paid by us. Conference venue
contacts were paid by the conference organizers.

Data annotations were completed by crowdsourced workers. The crowdsourced
workers were paid $0.20 for qualification assignment (note that typically
requesters do not pay for qualification tasks). Depending on the submitted
results, workers earn qualification to access of the actual tasks. The
annotation tasks were categorized into low-effort ($150), medium-effort
($300), and high-effort ($450), corresponding to the amount of estimated time
each would take. The duration of the tasks was determined by the crowd density
and through timing of the pilot studies. The average hourly payment to workers
is around $8.

Q. Were any ethical review processes conducted (e.g., by an institutional
review board)?  If so, please provide a description of these review processes,
including the outcomes, as well as a link or other access point to any
supporting documentation.

The data collection was approved by the Human Research Ethics Committee (HREC)
of our university (Delft University of Technology), which reviews all research
involving human subjects. The data collection protocol is also compliant to
the conference location’s national authorities (France). The review process
included addressing privacy concerns to ensure compliance with GDPR and
university guidelines, review of our informed consent form, data management
plan, and end user license agreement for the dataset and a safety check of our
custom wearable devices.

Q. Does the dataset relate to people?

Yes.

|
---|---
Figure 15: Screenshots of the ConfLab web-page used for participant
recruitment and registration.

Q. Did you collect the data from the individuals in question directly, or
obtain it via third parties or other sources (e.g., websites)?

We collected the data from individuals directly.

Q. Were the individuals in question notified about the data collection?  If
so, please describe (or show with screenshots or other information) how notice
was provided, and provide a link or other access point to, or otherwise
reproduce, the exact language of the notification itself.

The individuals were notified about the data collection and their
participation is voluntary. The data collection was staged at an event called
Meet the Chairs at ACM MM 2019. The ConfLab web page
(https://conflab.ewi.tudelft.nl/) served to communicate the aim of the event,
what was being recorded, and how participants could sign up. This allowed us
to embed the informed consent into this framework so we could keep track of
sign ups. See Figure 15 for screenshots. This event website was also shared by
the conference organizers and chairs (https://2019.acmmm.org/conflab-meet-the-
chairs/index.html).

Q. Did the individuals in question consent to the collection and use of their
data?  If so, please describe (or show with screenshots or other information)
how consent was requested and provided, and provide a link or other access
point to, or otherwise reproduce, the exact language to which the individuals
consented.

All the individuals who participated in the data collection gave their consent
by signing a consent form. A copy of the form is attached below in Figure 16.

Q. If consent was obtained, were the consenting individuals provided with a
mechanism to revoke their consent in the future or for certain uses?  If so,
please provide a description, as well as a link or other access point to the
mechanism (if appropriate)

Yes, the consenting individuals were informed about the possibility of
revoking access to their data within a period of $3$ months after the data
collection experiment, and not after that. The description is included in the
consent form.

Q. Has an analysis of the potential impact of the dataset and its use on data
subjects (e.g., a data protection impact analysis) been conducted?
No.

Q. Any other comments?
None.

PREPROCESSING / CLEANING / LABELING

Figure 16: Consent form signed by each participant in the data collection.

Q. Was any preprocessing/cleaning/labeling of the data
done(e.g.,discretization or bucketing, tokenization, part-of-speech tagging,
SIFT feature extraction, removal of instances, processing of missing values)?
If so, please provide a description. If not, you may skip the remainder of the
questions in this section.

We did not pre-process the signals obtained from the wearable devices or
cameras. The only exception is the audio data. Due to a hardware malfunction
(this is resolved for the Midges by using different SD cards), the audio
needed to be post-processed in order to synchronize it with the other
modalities. The synchronization against other modalities was manually checked.

Labeling of the dataset was done as explained in the Composition section.

Q. Was the “raw” data saved in addition to the preprocessed/cleaned/labeled
data (e.g., to support unanticipated future uses)?

The dataset is separated into raw data and the post processed data. For the
audio, the original raw data is not suitable for most use cases due to the
mentioned synchronization issue. So we share the synchronized version in the
raw part of the repository.

Q. Is the software used to preprocess/clean/label the instances available?  If
so, please provide a link or other access point.

The processing / fixing of the audio files did not require special software.

The annotation of keypoints and speaking status was done by making use of the
Covfee framework: https://josedvq.github.io/covfee/

Q. Any other comments?

None.

USES

Q. Has the dataset been used for any tasks already?  If so, please provide a
description.

In the main paper, we have benchmarked three baseline tasks: person and
keypoints detection, speaking status detection, and F-formation detection. The
first task is a fundamental building block for automatically analyzing human
social behaviors. The other two demonstrate how learned body keypoints can be
used in the behavior analysis pipeline for inferring more socially related
phenomena. We chose these benchmarking tasks since they have been studied on
other in-the-wild behavior datasets.

Q. Is there a repository that links to any or all papers or systems that use
the dataset?

None at the time of writing of the paper.

Q. What (other) tasks could the dataset be used for?

Given the richness and the unscripted open-ended nature of the social
interactions, ConfLab can be used for many other tasks.

#### Forecasting, causal relationship discovery

Recently, tasks pertaining to the forecasting low-level social cues in
conversations have been receiving increased attention from the community [72,
75]. The real-life nature of ConfLab along with the increased data and
annotation fidelity can prove a valuable resource for such tasks. Similarly,
ConfLab can also be used for efforts towards discovering causal relationships
between social behaviors [76].

#### Data Association.

A crucial assumption made in many former multimodal datasets[9, 11, 24] is
that the association of video data to the wearable modality can be manually
performed. Few works [43, 44] have tried to address this issue but using
movement cues alone to associate the modalities is challenging as conversing
individuals are mostly stationary. This remains a significant and open
question for future large scale deployable multimodal systems. One solution
may be to annotate more social actions as a form of top-down supervision.
However, detecting pose and actions robustly from overhead cameras remains to
be solved.

#### Conversation floor and F-formation estimation

Prior analysis on the MatchNMingle dataset has demonstrated that F-formations
can contain multiple simultaneous conversations when the F-formations contain
a least 4 people [51]. If this is the case for the ConfLab dataset, this may
drastically change how F-formations should be labelled (e.g. returning to
being a more subjective task [10]) as more time-precise labelling could enable
a more nuanced take on F-formation and conversation floor membership over
time.

#### Multi-class social action estimation

More annotations resources were focused on speaker status, F-formation, and
keypoint estimation. However, there are a wealth of other social actions in
the data that could be interesting to combine into a more complex multi-class
social action estimation task. Example social actions include drinking, mobile
phone use, hand and head gesture types [9, 77].

#### Estimation and analysis of socially-related phenomena

Beyond the modeling of human behavior which is of interest to the Computer
Vision and Machine Learning communities, our benchmarked tasks form the basis
for further explorations into downstream prediction of socially-related
constructs which is of interest to the Social Science and Social Psychology
communities. Such constructs include conversation quality [78, 68], dominance
[53], rapport [50], and influence [69].

#### Investigation of novel crossmodal fusion strategies

The baseline tasks in our paper rely only on a late fusion strategy. However,
ConfLab’s sub-second expected cross modal latency of $\sim 13$ ms along with
higher sampling rate of features ($60$ fps video, $56$ Hz IMU) opens the
gateway for the in-the-wild study of nuanced time-sensitive social behaviors
like mimicry and synchrony (for predicting e.g. attraction [79]) which need
tolerances as low as $40$ ms [18, Sec.3.2]. Prior works coped with lower
tolerances by computing summary statistics over input windows [17, 35, 36].
ConfLab enables for the first time, the exploration of Multimodal machine
learning approaches for social behaviour analysis in these highly dynamic in-
the-wild settings [65]. Through the provided annotations Conflab also enables
research in the topic of usage of mobile phones in small-group social
interactions in-the-wild.

#### Person attribute estimation

Estimating individuals that are newcomers/old timers from the dataset may be
possible based on their networking strategies.

Q. Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses?  For
example, is there anything that a future user might need to know to avoid uses
that could result in unfair treatment of individuals or groups (e.g.,
stereotyping, quality of service issues) or other undesirable harms (e.g.,
financial harms, legal risks) If so, please provide a description. Is there
anything a future user could do to mitigate these undesirable harms?

Although ConfLab’s long-term vision is towards developing technology to assist
individuals in navigating social interactions, the data could also affect a
community in unintended ways: for instance, cause worsened social
satisfaction, a lack of agency, stereotype newcomers and veterans, or benefit
only those members of the community who make use of resulting applications at
the expense of the rest. More nefarious uses involve exploiting the data for
developing methods that harmfully surveil or profile people. Researchers must
consider such inadvertent effects must while developing downstream
applications. Finally, since we recorded the dataset at a scientific
conference and required voluntary participation, there is an implicit
selection bias in the population represented in the data. Consequently,
researchers using the data should be aware that resulting insights may not
generalize to the general population.

Q. Are there tasks for which the dataset should not be used?  If so, please
provide a description.

Beyond the cautionary discussion in the previous question, tasks involving the
re-identifying the subjects is strictly against the End User License Agreement
under which we share the dataset.

Q. Any other comments?

None.

DISTRIBUTION

Q. Will the dataset be distributed to third parties outside of the entity
(e.g., company, institution, organization) on behalf of which the dataset was
created?  If so, please provide a description.

The dataset is available for third parties outside of Delft University of
Technology to use for academic research purposes subject signing and approval
of our End User License Agreement. The dataset will be hosted by
4TU.ResearchData (see the Maintenance section for description of the 4TU
entity).

Q. How will the dataset will be distributed (e.g., tarball on website, API,
GitHub)?  Does the dataset have a digital object identifier (DOI)?

The dataset will be distributed via the 4TU.ResearchData user interface where
the data can be downloaded. The dataset has a DOI:
https://doi.org/10.4121/c.6034313

Q. When will the dataset be distributed?

The dataset has been available since June 9, 2022.

Q. Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)?  If so,
please describe this license and/or ToU, and provide a link or other access
point to, or otherwise reproduce, any relevant licensing terms or ToU, as well
as any fees associated with these restrictions.

The dataset will be distributed under a restricted copyleft license, specified
within our End User License Agreement, accessible through the 4TU.ResearchData
dataset website. No fees are associated with the license.

Q. Have any third parties imposed IP-based or other restrictions on the data
associated with the instances?

No.

Q. Do any export controls or other regulatory restrictions apply to the
dataset or to individual instances?  If so, please describe these
restrictions, and provide a link or other access point to, or otherwise
reproduce, any supporting documentation.

The terms of our EULA and the European General Data Protection Regulations
(GDPR) apply.

Any other comments?
None.

MAINTENANCE

Q. Who is supporting/hosting/maintaining the dataset?

The dataset is hosted by 4TU.ResearchData (https://www.4tu.nl/en/about_4tu/),
and supported and maintained by The Socially Perceptive Computing Lab at
TUDelft.

Q. How can the owner/curator/manager of the dataset be contacted (e.g., email
address)?

Via email<EMAIL_ADDRESS>
Q. Is there an erratum?

No.

Q. Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)?  If so, please describe how often, by whom, and
how updates will be communicated to users (e.g., mailing list, GitHub)?

Updates will be done as needed as opposed to periodically. Instances could be
deleted, added, or corrected. The updates will be posted on the
4TU.ResearchData dataset website.

Q. If the dataset relates to people, are there applicable limits on the
retention of the data associated with the instances (e.g., were individuals in
question told that their data would be retained for a fixed period of time and
then deleted)?

No limits were communicated to our data participants.

Q. Will older versions of the dataset continue to be
supported/hosted/maintained?  If so, please describe how. If not, please
describe how its obsolescence will be communicated to users.

Only the latest version of the dataset will be maintained. If applicable, we
will also host older versions of the data, accessible through the
4TU.ResearchData website.

Q. If others want to extend/augment/build on/contribute to the dataset, is
there a mechanism for them to do so?  If so, please provide a description.
Will these contributions be validated/verified? If so, please describe how. If
not, why not? Is there a process for communicating/distributing these
contributions to other users? If so, please provide a description.

We are open to contributions to the dataset. In accordance with our End User
License Agreement, contributions should be made available, indicating if there
are any restrictions on their contribution. We encourage the potential
contributors to contact us to discuss how they wish to be attributed (e.g.
citation of a paper or repository related to code/annotations). After
finalizing the attribution discussion, we can add the attribution as an update
following the same process explained above.

##  Appendix C Sample Participant Report

Figure 17: Sample post-hoc report sent to each participant of ConfLab. The
report contains insights into the participant’s networking behavior from the
collected wearable-sensors data. This insight served as an additional
incentive to participate in ConfLab, beyond interacting with the Chairs and
contributing to a community-driven data endeavor (see main paper Section 3).

##  Appendix D Data Capture Setup Details

#### The Midge

We improved upon the Rhythm Badge in three ways towards enabling more fine-
grained and flexible data capture: (i) enabling full audio recording with a
frequency up to $48$ KHz, with an on-board switch to allow physical selection
between high and low frequency capture directly at acquisition; (ii) adding a
9-axis Inertial Measurement Unit (IMU) with an on-board Digital Motion
Processor (DMP) to record orientation; and (iii) an on-board SD card to
directly store raw data, avoiding issues related to packet loss during
wireless data transfer required by the Rhythm Badge. IMUs combine three tri-
axial sensors: an accelerometer, a gyroscope, and a magnetometer. These
measure acceleration, orientation, and angular rates respectively. These
sensor measurements are combined on-chip by a Digital Motion Processor. Rough
proximity estimation is performed by measuring the Received Signal Strength
Indicator (RSSI) for Bluetooth packets broadcast every second ($1$ Hz) by
every Midge. During the event, IMUs were set to record at $50$ Hz. We recorded
audio at $1250$ Hz to mitigate extraction of verbal content while still
ensuring robustness to cocktail-party noise.

#### Wireless Synchronization at Acquisition

The central idea for our syncrhonization approach involves using a common
Network Time Protocol (NTP) signal as reference for the camera and wearables
sub-networks. The set-up achieved a cross-modal latency of 13 ms at worst,
which is well below the 40 ms latency tolerance suitable for behavior research
in our setting [18, Sec. 3.3]. Additionally, our synchronization approach
allowed for dynamic addition of sensors to the network while still obtaining
synchronized data streams. This is crucial in extreme in-the-wild events where
some participants might arrive late.

#### Sensor Calibration

For computing the camera extrinsics, we marked a grid of $1$ m $\times$ $1$ m
squares in tape across the interaction area floor. We ensured line alignment
and right angles using a laser level tool (STANLEY Cross90). For computing the
camera intrinsics, we used the OpenCV asymmetric circles grid pattern [80].
The calibration was performed using the Idiap multi camera calibration suite
[81]. All wearable sensors include one TDK InvenSense ICM-20948 IMU [82] unit
that provides run time calibration. To establish a correspondence with the
camera frame of reference, the sensors were lined up against a common
reference-line visible in the cameras to acquire an alignment so that the
camera data can offer drift and bias correction for the wearable sensors.

##  Appendix E Implementation Details

###  E.1 Person and Keypoint Detection Models

#### Data Cleaning

A few frames contained some incorrectly labeled keypoints, a product of
annotation errors like mis-assignment of participant IDs. We removed these
using a threshold on the proximity to other keypoints of the same person.
Further, in some cases, a person might be partially outside a camera’s field
of view. For the person detection task, we compute the bounding box from the
keypoint ground-truth annotations. If more than half the body (50% keypoints)
is missing in the frame so that e.g. only their legs are visible (see top of
7(a)), we don’t consider the person for that frame in the person detection
experiments. Note that due to the significant overlap between the camera
views, the person would be considered for the corresponding frame in the next
camera. If they move back into the original view, we again take them into
consideration for the original camera for the corresponding frame. Moreover,
if there are more than 10% missing keypoints across all people in an image, we
also discard that image from the experiment. This preprocessing resulted in a
training set with 112k frames (1809k person instances) and a test set with 7k
frames (158k person instances).

#### Training

We resized the images to $960\times 540$, and augmented the data by
randomizing brightness and horizontal flips. The learning rate was set to
$0.02$ and batch size to $4$. We trained the models for $50$ k iterations,
using the COCO-pretrained weights for initialization. All hyper-parameters
were chosen based on the performance on a separate hold-out camera chosen as
validation set. During training, any missing ground-truth keypoints (resulting
from the person being partially outside the camera’s view for instance) are
ignored during back-propagation.

###  E.2 F-formation Detection

#### Data Cleaning

Because keypoint annotations of the subjects are based on camera view and that
the F-formation clustering methods cannot group subjects that do not exist
under one camera view (e.g., when there are more identities than in associated
ground truths), we processed the ground truth also based on camera number.
This filtering pre-processing was decided based on the best camera view of the
F-formations.

#### Feature Extraction

The required features of GCFF and GTCG include location and orientation of the
subjects. We used the X and Y position of subjects’ head (as it is the most
visible from the top-down view) for location, and extracted orientations for
head, shoulders and hips. The orientations are calculated based on
corresponding vectors determined by head and nose keypoints, left and right
shoulder keypoints, and left and right hip keypoints, respectively.

#### Training

We used pre-trained parameters for field of view (FoV) and frustum aperture
(GTCG) and minimum description length (GCFF), provided in these models trained
on the Cocktail Party. FOV and aperture are related to human eye gaze and head
anatomical constraints reported by [83], and hence not dataset specific. The
minimum description length is an initialized prior dictated by the same form
of the Akaike Information Criterion, and becomes part of the optimization
formulation. We tuned parameters such as frustum length (GTCG) and stride
(GCFF) to account for average interpersonal distance in ConfLab based on
Camera 6, as they vary across different datasets.

##  Appendix F Additional Results

###  F.1 Person and Keypoints Detection

#### Predictions from pretrained SOTA models

Figure 18 shows predictions from SOTA human keypoint estimation models,
namely, RSN [19], MSPN[84], HigherHRNet [85], and HourglassAENet [86], for the
testing images of the Conflab dataset. Note that RSN and MSPN are top-down
networks, i.e., they require person bounding boxes to predict the keypoints in
each bounding box. We use COCO pretrained faster-RCNN network for bounding box
estimation. HigherHRNet and HourglassAENet are bottom-up models, i.e., they
directly predict keypoints from the full image. We use publicly available COCO
pretrained checkpoints for prediction. The results show that the _state-of-
the-arts 2D body keypoint detection models fail to capture the body keypoints
in the Conflab dataset_. We infer that training on the dataset (e.g., COCO)
that contains mostly side-view images does not work well in top-view images,
for which Conflab dataset is important to the community.

Figure 18: Results from Pretrained keypoint detection models. From top to
bottom - predictions from RSN [19], MSPN[84], HigherHRNet [85], and
HourglassAENet [86]. Results show that _SOTA 2D body keypoint detection models
fail to capture the body keypoints in the ConfLab dataset_.

Figure 19: Results from (top) COCO pretrained Mask-RCNN model, (bottom) our
ConfLab finetuned Mask-RCNN model.

Figure 20: Effect of varying % frames from each camera at training on keypoint estimation. % of training samples | AP${}^{\text{OKS}}_{\text{50}}$
---|---
1.6% | 29.0
3.2% | 35.9
8% | 39.0
16% | 44.5
100% | 45.3

Figure 21: Effect of adding all frames from individual cameras to the training set on keypoint estimation. Train Camera | #(training samples) | AP${}^{\text{OKS}}_{\text{50}}$
---|---|---
cam 2 | 34k | 8.6
cam 2 + cam 4 | 69k | 31.1
cam 2 + cam 4 + cam 8 | 112k | 45.3

Figure 22: Keypoint estimation ablation with keypoints from different body
sections: head and shoulders (5), + torso (9), + hips (11), + knees and feet
(full 17).

#Keypoints AP${}^{\text{OKS}}_{\text{50}}$ AP${}^{\text{OKS}}$
AP${}^{\text{OKS}}_{\text{75}}$ 5 26.6 7.1 1.4 9 26.5 6.9 2.0 11 35.8 9.5 2.2
17 45.3 13.5 3.3

Figure 23: ROC AUC and accuracy for different sensor modalities from out 9-dof IMU in speaking status detection using the Minirocket classifier [64]. The number of channels in the corresponding modality is indicated in parentheses. Input Modality | AUC | Accuracy
---|---|---
Acceleration (3) | 0.813 | 0.768
Gyroscope (3) | 0.765 | 0.716
Magnetometer (3) | 0.610 | 0.656
Rotation vector (4) | 0.726 | 0.696
All (13) | 0.774 | 0.739

#### Qualitative Results from ResNet-50 Finetuning

Figure 19 illustrates more qualitative results from our finetuning
experiments. We find that finetuning on our non-invasive top-down camera
perspective significantly improves the keypoint estimation performance.

#### Ablations

Tables 21 and 21 include the results of our experiments investigating the
effect of varying the training data size on keypoint detection performance
(see main paper Section 6.1). In Table 23, we show keypoint detection scores
for experiments with different number of keypoints. We first focus on the five
upper body keypoints: {head, nose, neck, rightShoulder, leftShoulder}. We then
additionally considered the torso region keypoints for a total of nine:
{rightElbow, rightWrist, leftElbow, leftWrist}. Finally, we add the hip
keypoints {rightHip, leftHip} to the set. The experiments in the main paper
are performed with all 17 keypoints. The results show that performance drops
slightly when adding the arms keypoints ($5\to 9$,
AP${}^{\text{OKS}}_{\text{50}}$ and AP${}^{\text{OKS}}$), and that the
relative gain when adding the hip keypoints ($9\to 11$) is lower than when
adding the lower body keypoints ($11\to 17$, especially
AP${}^{\text{OKS}}_{\text{75}}$). We believe this is largely due to the lower
body being more static relative to the arms that move a lot to execute
gestures during conversations.

###  F.2 Speaking Status Detection

#### Experiments with different sensor modalities

Table 23 displays the results from experiments using specific modalities from
our IMUs for the task of speaking status detection. We used the best
performing classifier (Minirocket [64]) among the ones tested in Table 3. The
experiment setup is the same as detailed in Section 6.2, and the model is not
changed between runs, except for the fact that different modalities may have a
different number of input channels.

##  Appendix G Reproducibility Checklist

###  G.1 Person and Keypoints Detection

  * •

Source code link: https://github.com/TUDelft-SPC-Lab/conflab

  * •

Data used for training: $112$k frames ($1809$k person instances).

  * •

Pre-processing: See Section 4, Appendix E.1.

  * •

How samples were allocated for train/val/test: cameras $2$, $4$, and $8$ are
selected for training. For hyperparameter tuning, camera $8$ are held out for
validation.

  * •

Hyperpatameter consideration: We considered learning rates
$(0.001/0.005/0.05/0.01)$, number of epochs $(10/20/50/100)$, detection
backbone (R50-FPN/R50-C4). Also see Appendix E.1

  * •

Number of evaluation runs: $5$

  * •

How experiments were ran: See Section 6.1.

  * •

Evaluation metrics: Average precision at different thresholds.

  * •

Results: See Section 6.1 and Appendix F.1.

  * •

Computing infrastructure used: All baseline experiments were ran on Nvidia
V100 GPU ($16$GB) with IBM POWER9 Processor.

###  G.2 Speaking Status Detection

  * •

Source code link: https://github.com/TUDelft-SPC-Lab/conflab

  * •

Data used for training: $42884$ windows ($3$ seconds), extracted from $48$
participants’ wearable data and speaking status annotations

  * •

Pre-processing: Data was windowed into $3$-second segments (see Section 6.2).
The source code includes this pre-processing step.

  * •

How samples were allocated for train/val/test: 10-fold cross-validation at the
subject level ($48$ subjects) to test generalization to unseen data subjects.
The splits can be reproduced exactly using the source code.

  * •

Hyperparameter considerations: For acceleration-based methods, we used default
network hyper-parameters and architectures from their tsai implementation
[87]. For the MS-G3D baseline [61], we used default hyperparameters from the
authors’ implementation. For both, we determined the early stoppage point
using a small subset ($10\%$) of the training set.

  * •

Number of evaluation runs: 1 run of 10-fold cross-validation

  * •

How experiments were ran: For each fold, the early stoppage point was first
determined using $10\%$ of the training data as validation set and AUC as
performance metric. The model at this stoppage point was then applied to the
test set for evaluation.

  * •

Evaluation metrics: Area under the ROC curve (AUC)

  * •

Results: See Section 6.2

  * •

Computing infrastructure used: Experiments were ran on a personal computer
with GPU acceleration (NVidia RTX3080).

###  G.3 F-formation Detection

  * •

Source code link: https://github.com/TUDelft-SPC-Lab/conflab

  * •

Data used for training: Camera $6$

  * •

Pre-processing: See Section E.2 for data cleaning and feature extraction.

  * •

How samples were allocated for train/val/test: samples from Camera $6$ were
used to select the best model parameters. The rest are for test (evaluation).
However, we note that Table 3 shows averaged performance on all cameras to
provide a holistic view of the F-formation detection performance on ConfLab.

  * •

(Hyper)parameter considerations: Both baseline methods are not deep-learning
based and model parameters are interpretable. For GTCG, the parameters are
frustum length ($275$), frustum aperture ($160$), frustum samples ($2000$),
and sigma for affinity matrix ($0.6$). For GCFF, the parameters are minimum
description length ($30000$) and stride ($70$).

  * •

Number of evaluation runs: 1

  * •

How experiments were ran: A total of eight experiments were run for choosing
the best parameters, and three for evaluation (for camera $2$, $4$, and $8$).
The parameters were chosen based on grid-search. For optimizing frustum length
in GTCG, we searched over $[170,195,220,245,275]$ with $275$ being averaged
interpersonal distance based on Camera $6$. For optimizing stride $D$ in GCFF,
we searched over $[30,50,70]$.

  * •

Evaluation metrics: F1

  * •

Results: See Section 6.3

  * •

Computing infrastructure used: The experiments were run on Linux-based cluster
instances on CPU with Matlab 2018a.