# RTMO: Towards High-Performance One-Stage Real-Time Multi-Person Pose
Estimation

Peng Lu1,2, Tao Jiang2, Yining Li2, Xiangtai Li3, Kai Chen2*, Wenming Yang1

1Tsinghua Shenzhen International Graduate School
2Shanghai AI Laboratory 3Nanyang Technological University Corresponding
author.

###### Abstract

Real-time multi-person pose estimation presents significant challenges in
balancing speed and precision. While two-stage top-down methods slow down as
the number of people in the image increases, existing one-stage methods often
fail to simultaneously deliver high accuracy and real-time performance. This
paper introduces RTMO, a one-stage pose estimation framework that seamlessly
integrates coordinate classification by representing keypoints using dual 1-D
heatmaps within the YOLO architecture, achieving accuracy comparable to top-
down methods while maintaining high speed. We propose a dynamic coordinate
classifier and a tailored loss function for heatmap learning, specifically
designed to address the incompatibilities between coordinate classification
and dense prediction models. RTMO outperforms state-of-the-art one-stage pose
estimators, achieving 1.1% higher AP on COCO while operating about 9 times
faster with the same backbone. Our largest model, RTMO-l, attains 74.8% AP on
COCO val2017 and 141 FPS on a single V100 GPU, demonstrating its efficiency
and accuracy. The code and models are available at https://github.com/open-
mmlab/mmpose/tree/dev-1.x/projects/rtmo.

##  1 Introduction

Figure 1: Efficiency and efficacy comparison among real-time pose estimation
methods across different inference backends and devices. The radial axes
indicate inference speed in Frames Per Second (FPS). The outer circular axis
shows Average Precision (AP) on the COCO val2017 dataset. Models marked with
$\dagger$ were trained with additional data beyond the COCO train2017.

Multi-person pose estimation (MPPE) is essential in the field of computer
vision, with applications ranging from augmented reality to sports analytic.
Real-time processing is particularly crucial for applications requiring
instant feedback, such as coaching for athlete positioning. Although numerous
real-time pose estimation techniques have emerged [15, 3, 30, 16], achieving a
balance between speed and accuracy remains challenging.

Current real-time pose estimation methods fall into two categories: top-down
[15, 3] and one-stage [30, 16]. Top-down methods employ pre-trained detectors
to create bounding boxes around subjects, followed by pose estimation for each
individual. A key limitation is that their inference time scales with the
number of people in the image (see Figure 1). On the other hand, one-stage
methods directly predict the locations of keypoints for all individuals in the
image. However, current real-time one-stage methods [30, 16, 33] lag in
accuracy compared to top-down approaches (see Figure 1). These methods,
relying on the YOLO architecture, directly regress the keypoint coordinates,
which hinders performance since this technique resembles using a Dirac delta
distribution for each keypoint, neglecting the inherent ambiguity and
uncertainty [20].

Alternatively, the coordinate classification methods employ dual 1-D heatmaps
that increase spatial resolution by spreading the probability of keypoint
locations over two sets of bins spanning the entire image. This offers more
accurate predictions with minimal extra computational cost [22, 15]. However,
directly applying coordinate classification to dense prediction scenarios like
one-stage pose estimation leads to inefficient bin utilization due to global
bin distribution across the image and each person occupying only a minor
region. Additionally, conventional Kullback–Leibler divergence (KLD) losses
treat all samples equally, which is suboptimal for one-stage pose estimation
where instance difficulty varies significantly across grids.

In this work, we overcome the above challenges and incorporate the coordinate
classification approach within the YOLO-based framework, leading to the
development of Real-Time Multi-person One-stage (RTMO) pose estimation models.
RTMO introduces a Dynamic Coordinate Classifier (DCC) that includes dynamic
bin allocation localized to bounding boxes and learnable bin representations.
Furthermore, we propose a novel loss function based on Maximum Likelihood
Estimation (MLE) to effectively train the coordinate heatmaps. This new loss
allows learning of per-sample uncertainty, automatically adjusting task
difficulty and balancing optimization between hard and easy samples for more
effective and harmonized training.

Consequently, RTMO achieves accuracy comparable to real-time top-down methods
and exceeds that of other lightweight one-stage methods, as shown in Figure 1.
Moreover, RTMO demonstrates superior speed when processing multiple instances
in an image, outpacing top-down methods with similar accuracy. Notably, the
RTMO-l model attains a 74.8% Average Precision (AP) on the COCO val2017
dataset [23] and exhibited 141 frames per second (FPS) on the NVIDIA V100 GPU.
On the CrowdPose benchmark [18], RTMO-l achieves 73.2% AP, a new state of the
art for one-stage methods. The key contributions of this work include:

  * •

An innovative coordinate classification technique tailored for dense
prediction scenarios, utilizing coordinate bins for precise keypoint
localization while addressing the challenges posed by the varying sizes and
complexities of instances.

  * •

A new real-time one-stage MPPE approach that seamlessly integrates coordinate
classification with the YOLO architecture, achieving an optimal balance of
performance and speed among existing top-down and one-stage MPPE methods.

##  2 Related Works

###  2.1 One-Stage Pose Estimator

Inspired by advancements in one-stage object detection algorithms [40, 24, 9,
51, 7], a series of one-stage pose estimation methods have emerged [39, 34,
51, 10, 30]. These methods execute MPPE within a single forward pass and
directly regress instance-specific keypoints from predetermined root
locations. Alternative approaches such as PETR [37] and ED-Pose [46] treat
pose estimation as a set prediction problem, establishing a fully end-to-end
framework for keypoint regression. Beyond regression-based solutions,
techniques like FCPose [31], InsPose [35] and CID [42] utilize dynamic
convolution or attention mechanisms to generate instance-specific heatmaps for
keypoint localization.

Compared to two-stage pose estimation methods, one-stage approaches eliminate
the need for pre-processing (e.g., human detection for top-down methods) and
post-processing (e.g., keypoint grouping for bottom-up methods). This results
in two benefits: 1) consistent inference time, irrespective of the number of
instances in the image; and 2) a simplified pipeline that facilitates
deployment and practical use. Despite these advantages, the existing one-stage
methods struggle to balance high accuracy with real-time inference. High
accuracy models [42, 46] often depend on resource-intensive backbones like
HRNet [38] or Swin [25], making real-time estimation challenging. Conversely,
real-time models [33, 30] compromise on performance. Our model addresses this
trade-off, delivering both high accuracy and fast real-time inference.

###  2.2 Coordinate Classification

SimCC [22] and RTMPose [15] have adopted coordinate classification for pose
estimation. These methods classify keypoints based on sub-pixel bins along
horizontal and vertical axes, achieving spatial discrimination without relying
on high-resolution feature maps. This balances accuracy and speed effectively.
However, spanning bins across the entire image for dense prediction methods is
impractical due to the need for a vast number of bins to diminish quantization
error, which leads to inefficiency from many bins being superfluous for
individual instances. DFL [20] sets bins around a predefined range near each
anchor, which may not encompass the keypoints of large instances and could
induce significant quantization errors for small instances. Our approach
assigns bins based on each instance’s size, placing them within localized
regions, which optimizes bin utilization, ensures coverage of keypoints and
minimizes quantization error.

Figure 2: Overview of the RTMO Network Architecture. Its head outputs
predictions for the score, bounding box, keypoint coordinates and visibility
for each grid cell. The Dynamic Coordinate Classifier translates pose features
into K pairs of 1-D heatmaps for both the horizontal and vertical axes,
encompassing an expanded region 1.25 times the size of the predicted bounding
boxes. From these heatmaps, keypoint coordinates are precisely extracted. K
denotes the total number of keypoints.

###  2.3 Transformer-Enhanced Pose Estimation

Transformer-based architectures have become ubiquitous in pose estimation,
leveraging state-of-the-art transformer backbones to improve accuracy as in
ViTPose [45], or combining transformer encoders with CNNs to capture spatial
relationships [47]. TokenPose [21] and Poseur [32] demonstrate the efficacy of
token-based keypoint embedding in both heatmap and regression-based methods,
leveraging visual cues and anatomical constraints. Frameworks like PETR [37]
and ED-Pose [46] introduce transformers in end-to-end multi-person pose
estimation, and RTMPose [15] integrates self-attention with a SimCC-based [22]
framework for keypoint dependency analysis, an approach also adopted by RTMO.
While positional encoding is standard in attention to inform query and key
positions, we innovatively employ it to form representation vectors for each
spatial bin, enabling computation of bin-keypoint similarity that facilitates
accurate localization predictions.

##  3 Methodology

In our model, we adopt a YOLO-like architecture, illustrated in Figure 2. The
backbone is CSPDarknet [9], and we process the last three feature maps using a
Hybrid Encoder [28], yielding spatial features $P_{4}$ and $P_{5}$ with
respective downsampling rates of 16 and 32. Each pixel in these features maps
to a grid cell uniformly distributed on the original image plane. The network
head, utilizing dual convolution blocks at each spatial level, generates a
score and corresponding pose features for every grid cell. These pose features
are utilized to predict bounding boxes, keypoint coordinates, and visibility.
The generation of 1-D heatmap predictions through the Dynamic Coordinate
Classifier is detailed in Sec. 3.1 while the proposed heatmap loss based on
MLE is presented in Sec. 3.2. The complete training and inference procedures
are outlined in Sec. Sec. 3.3.

###  3.1 Dynamic Coordinate Classifier

The pose features associated with each grid cell encapsulate the keypoint
displacement from the grid. Previous works [34, 10, 30] directly regress these
displacement, and thus fall short in performance. Our study explores the
integration of coordinate classification with a one-stage pose estimation
framework to improve keypoint localization accuracy. A notable limitation of
existing coordinate classification methods is their static strategy for bin
assignment. To address this problem, we introduce the Dynamic Coordinate
Classifier (DCC), which dynamically assigns ranges and forms representations
for bins in two 1-D heatmaps, effectively resolving the incompatibilities of
coordinate classification in dense prediction contexts.

#### Dynamic Bin Allocation.

Coordinate classification technique employed in top-down pose estimators
allocates bins across the entire input image [22, 15]. This approach leads to
bin wastage in one-stage methods, as each subject occupies only a small
portion of the image. DFL [20] sets bins within a predefined range near each
anchor, which may miss keypoints in larger instances and cause significant
quantization errors in smaller ones. DCC addresses these limitations by
dynamically assigning bins to align with each instance’s bounding box,
ensuring localized coverage. The bounding boxes are initially regressed using
a pointwise convolution layer and then expanded by a factor of 1.25 to cover
all keypoints, even in cases of inaccurate predictions. These expanded
bounding boxes are uniformly divided into $B_{x},B_{y}$ bins along horizontal
and vertical axes. The x-coordinate for each horizontal bin is calculated
using:

$x_{i}=x_{l}+\left(x_{r}-x_{l}\right)\frac{i-1}{B_{x}-1},$

where $x_{r},x_{l}$ are the left and right sides of the bounding box, and
index $i$ varies from 1 to $B_{x}$. The y-axis bins are computed similarly.

#### Dynamic Bin Encoding.

In the context of DCC, the position of each bin varies across grids since
their predicted bounding boxes differ. This differs from previous methods [22,
15] where bin coordinates are fixed. Rather than a shared representation for
bins across grids used in these methods, DCC generates tailored
representations on-the-fly. Specifically, we encode each bin’s coordinates
into positional encodings to create bin-specific representations. We utilize
sine positional encoding [41] defined as:

$\left[\boldsymbol{PE}\left(x_{i}\right)\right]_{c}=\left\\{\begin{array}[]{lc}\sin\left(\frac{x_{i}}{t^{c/C}}\right),&\text{for
even}\ c\\\ \cos\left(\frac{x_{i}}{t^{(c-1)/C}}\right),&\text{for odd}\
c\end{array}\right.,$

where $t$ denotes the temperature, $c$ is the index, and $C$ represents the
total number of dimension. We refine the positional encoding’s adaptability
for our task using a fully connected layer, which applies a learnable linear
transformation $\boldsymbol{\phi}$, thereby optimizing its effectiveness in
DCC.

The primary objective of DCC is to accurately predict keypoint occurrence
probabilities at each bin, informed by bin coordinates and keypoint features.
Keypoint features are extracted from the pose feature and refined via a Gated
Attention Unit (GAU) module [13] following RTMPose [15], to enhance inter-
keypoint consistency. The probability heatmap is generated by multiplying the
keypoint features $\boldsymbol{f}_{k}$ with the positional encodings of each
bin $\boldsymbol{PE}\left(x_{i}\right)$, followed by a softmax:

$\hat{p}_{k}\left(x_{i}\right)=\frac{e^{\boldsymbol{f}_{k}\cdot\boldsymbol{\phi}\left(\boldsymbol{PE}\left({x_{i}}\right)\right)}}{\sum_{j=1}^{B_{x}}e^{\boldsymbol{f}_{k}\cdot\boldsymbol{\phi}\left(\boldsymbol{PE}\left({x_{j}}\right)\right)}},$

where $\boldsymbol{f}_{k}$ is the $k$-th keypoint’s feature vector.

###  3.2 MLE for Coordinate Classification

In classification tasks, one-hot targets and cross-entropy loss are commonly
utilized. Label smoothing, like Gaussian label smoothing used in SimCC [22]
and RTMPose [15], along with KLD, can improve performance. The Gaussian mean
$\mu_{x},\mu_{y}$ and variance $\sigma^{2}$ are set to the annotated
coordinates and a predefined parameter. The target distribution is defined as:

$p_{k}\left(x_{i}\mid\mu_{x}\right)=\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{\left(x_{i}-\mu_{x}\right)^{2}}{2\sigma^{2}}}\sim\mathcal{N}\left(x_{i};\mu_{x},\sigma^{2}\right).$

Importantly, we note that $p_{k}(x_{i}\mid\mu_{x})$ is mathematically
identical to the likelihood $p_{k}(\mu_{x}\mid x_{i})$ of the annotation
$\mu_{x}$ under a Gaussian error model with true value $x_{i}$. This
symmetrical property arises because the Gaussian distribution is symmetric
with respect to its mean. Treating the predicted $\hat{p}_{k}(x_{i})$ as the
prior of $x_{i}$, the annotation likelihood for the $k$-th keypoint is:

$\displaystyle P\left(\mu_{x}\right)$
$\displaystyle=\sum_{i=1}^{B_{x}}P\left(\mu_{x}\mid
x_{i}\right)P\left(x_{i}\right)$
$\displaystyle=\sum_{i=1}^{B_{x}}\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{\left(x_{i}-\mu_{x}\right)^{2}}{2\sigma^{2}}}\hat{p}_{k}\left(x_{i}\right).$

Maximizing this likelihood models the true distribution of the annotations.

In practice, we employ a Laplace distribution for $P\left(\mu_{x}\mid
x_{i}\right)$ and a negative log-likelihood loss:

$\mathcal{L}_{\textit{mle}}^{\left(x\right)}=-\log\left[\sum_{i=1}^{B_{x}}\frac{1}{\hat{\sigma}}e^{-\frac{\left|x_{i}-\mu_{x}\right|}{2\hat{\sigma}s}}\hat{p}_{k}\left(x_{i}\right)\right],$

where the instance size $s$ normalizes the error and $\hat{\sigma}$ is the
predicted variance. The constant factor is omitted as it does not affect the
gradient. The total Maximum Likelihood Estimation (MLE) loss is
$\mathcal{L}_{\textit{mle}}=\mathcal{L}_{\textit{mle}}^{\left(x\right)}+\mathcal{L}_{\textit{mle}}^{\left(y\right)}$.

Unlike KLD, our MLE loss allows for learnable variance, representing
uncertainty. This uncertainty learning framework automatically adjusts the
difficulty of various samples [12, 5]. For hard samples, the model predicts a
large variance to ease optimization. For simple samples, it predicts a small
variance, aiding in accuracy improvement. With KLD, adopting a learnable
variance is problematic - the model leans to predict a large variance to
flatten the target distribution as this simplifies learning.

###  3.3 Training and Inference

|  |
---|---|---
Figure 3: Comparison of RTMO with other real-time multi-person pose
estimators. The latency for top-down methods varies depending on the number of
instances in the image, as indicated by numerical values in the figures. All
models are evaluated without test-time augmentation. $\dagger$ indicates that
the model was trained using additional data beyond the COCO train2017 dataset.

#### Training.

Our model, adhering to a YOLO-like structure, employs dense grid prediction
for human detection and pose estimation. It is crucial for the model to
differentiate between positive and negative grids. We extend SimOTA [9] for
training, assigning positive grids based on grid scores, bounding box
regression and pose estimation accuracy. The head’s score branch classifies
these grids, supervised by varifocal loss [50] $\mathcal{L}_{\textit{cls}}$,
with target scores being the Object Keypoint Similarity (OKS) between the
predicted pose and the assigned ground truth for each grid.

Positive grid tokens yield bounding box, keypoint coordinates, and visibility
predictions. Keypoint coordinates are derived via the DCC, while other
predictions come from pointwise convolution layers. The losses applied are IoU
loss for bounding boxes $\mathcal{L}_{\textit{bbox}}$, MLE loss for keypoints
$\mathcal{L}_{\textit{mle}}$, and BCE loss for visibility
$\mathcal{L}_{\textit{vis}}$.

Given the DCC’s computational demands, we implement a pointwise convolution
layer for preliminary coordinate regression, similar to YOLO-Pose [30], to
mitigate out-of-memory issues. This regressed keypoints
$\mathrm{kpt}_{\textit{reg}}$ serves as a proxy in SimOTA for positive grid
selection, with the decoded keypoints $\mathrm{kpt}_{\textit{dec}}$ later used
to calculate OKS. The regression branch’s loss is OKS loss [30]:

$\mathcal{L}_{\textit{proxy}}=1-\mathrm{OKS}\left(\mathrm{kpt}_{\textit{reg}},\mathrm{kpt}_{\textit{dec}}\right).$

The total loss for the proposed model is

$\mathcal{L}=\lambda_{1}\mathcal{L}_{\textit{bbox}}+\lambda_{2}\mathcal{L}_{\textit{mle}}+\lambda_{3}\mathcal{L}_{\textit{proxy}}+\lambda_{4}\mathcal{L}_{\textit{cls}}+\mathcal{L}_{\textit{vis}},$

with hyperparameters $\lambda_{1},\lambda_{2},\lambda_{3}$, and $\lambda_{4}$
set at $\lambda_{1}=\lambda_{2}=5,\lambda_{3}=10$, and $\lambda_{4}=2$.

#### Inference.

During inference, our model employs a score threshold of 0.1 and non-maximum
suppression for grid filtering. It then decodes pose features from selected
grids into heatmaps, using integral over the heatmaps to derive keypoint
coordinates. This selective decoding approach minimizes the number of features
for processing, reducing computational cost.

##  4 Experiments

###  4.1 Settings

#### Datasets.

Experiments were primarily conducted on the COCO2017 Keypoint Detection
benchmark [23], comprising approximately 250K person instances with 17
keypoints. Performance comparisons were made with state-of-the-art methods on
both the val2017 and test-dev sets. To explore our model’s performance
ceiling, training was also extended to include additional datasets: CrowdPose
[18], AIC [43], MPII [1], JHMDB [14], Halpe [8], and PoseTrack18 [2]. These
annotations were converted to COCO format. RTMO was further evaluated on the
CrowdPose benchmark [18] which is known for its high complexity due to crowded
and occluded scenes, comprising 20K images and approximately 80K persons with
14 keypoints. OKS-based Average Precision (AP) served as the evaluation metric
for both datasets.

#### Implementation Details.

During training, we adopt the image augmentation pipeline from YOLOX [9],
incorporating mosaic augmentation, random color adjustments, geometric
transformations, and MixUp [49]. Training images are resized to dimensions
[480, 800]. Epoch counts are set to 600 and 700 for the COCO and CrowdPose
datasets, respectively. The training process is divided into two stages: the
first involves training both the proxy branch and DCC using pose annotations,
and the second shifts the target of the proxy branch to the decoded pose from
DCC. The AdamW optimizer [26] is used with a weight decay of 0.05, and
training is performed on Nvidia GeForce RTX 3090 GPUs with batch size 256.
Initial learning rates are set to $4\times 10^{-3}$ and $5\times 10^{-4}$ for
the two training phases, decaying to $2\times 10^{-4}$ via cosine annealing.
For inference, images are resized to 640. CPU latency is measured on an Intel
Xeon Gold CPU using ONNXRuntime. GPU latency is tested on an NVIDIA V100 GPU
with ONNXRuntime and with TensorRT using half-precision floating-point (FP16)
format.

###  4.2 Benchmark Results

Method | Backbone | #Params | Time (ms) | AP | $\textrm{AP}_{50}$ | $\textrm{AP}_{75}$ | $\textrm{AP}_{M}$ | $\textrm{AP}_{L}$ | AR
---|---|---|---|---|---|---|---|---|---
DirectPose [39] | ResNet-50 | - | 74 | 62.2 | 86.4 | 68.2 | 56.7 | 69.8 | -
DirectPose [39] | ResNet-101 | - | - | 63.3 | 86.7 | 69.4 | 57.8 | 71.2 | -
FCPose [31] | ResNet-50 | 41.7M | 68 | 64.3 | 87.3 | 71.0 | 61.6 | 70.5 | -
FCPose [31] | ResNet-101 | 60.5M | 93 | 65.6 | 87.9 | 72.6 | 62.1 | 72.3 | -
InsPose [35] | ResNet-50 | 50.2M | 80 | 65.4 | 88.9 | 71.7 | 60.2 | 72.7 | -
InsPose [35] | ResNet-101 | - | 100 | 66.3 | 89.2 | 73.0 | 61.2 | 73.9 | -
CenterNet [7] | Hourglass | 194.9M | 160 | 63.0 | 86.8 | 69.6 | 58.9 | 70.4 | -
PETR [36] | ResNet-50 | 43.7M | 89 | 67.6 | 89.8 | 75.3 | 61.6 | 76.0 | -
PETR [36] | Swin-L | 213.8M | 133 | 70.5 | 91.5 | 78.7 | 65.2 | 78.0 | -
CID [42] | HRNet-w32 | 29.4M | 84.0 | 68.9 | 89.9 | 76.9 | 63.2 | 77.7 | 74.6
CID [42] | HRNet-w48 | 65.4M | 94.8 | 70.7 | 90.4 | 77.9 | 66.3 | 77.8 | 76.4
ED-Pose [46] | ResNet-50 | 50.6M | 135.2 | 69.8 | 90.2 | 77.2 | 64.3 | 77.4 | -
ED-Pose [46] | Swin-L | 218.0M | 265.6 | 72.7 | 92.3 | 80.9 | 67.6 | 80.0 | -
KAPAO-s [33] | CSPNet | 12.6M | 26.9 | 63.8 | 88.4 | 70.4 | 58.6 | 71.7 | 71.2
KAPAO-m [33] | CSPNet | 35.8M | 37.0 | 68.8 | 90.5 | 76.5 | 64.3 | 76.0 | 76.3
KAPAO-l [33] | CSPNet | 77.0M | 50.2 | 70.3 | 91.2 | 77.8 | 66.3 | 76.8 | 77.7
YOLO-Pose-s [30] | CSPDarknet | 10.8M | 7.9 | 63.2 | 87.8 | 69.5 | 57.6 | 72.6 | 67.6
YOLO-Pose-m [30] | CSPDarknet | 29.3M | 12.5 | 68.6 | 90.7 | 75.8 | 63.4 | 77.1 | 72.8
YOLO-Pose-l [30] | CSPDarknet | 61.3M | 20.5 | 70.2 | 91.1 | 77.8 | 65.3 | 78.2 | 74.3
RTMO-r50 | ResNet-50 | 41.7M | 15.5 | 70.9 | 91.0 | 78.2 | 65.8 | 79.1 | 75.0
RTMO-s | CSPDarknet | 9.9M | 8.9 | 66.9 | 88.8 | 73.6 | 61.1 | 75.7 | 70.9
RTMO-s$\dagger$ | CSPDarknet | 9.9M | 8.9 | 67.7 | 89.4 | 74.5 | 61.5 | 77.2 | 71.9
RTMO-m | CSPDarknet | 22.6M | 12.4 | 70.1 | 90.6 | 77.1 | 65.1 | 78.1 | 74.2
RTMO-m$\dagger$ | CSPDarknet | 22.6M | 12.4 | 71.5 | 91.0 | 78.6 | 66.1 | 79.9 | 75.6
RTMO-l | CSPDarknet | 44.8M | 19.1 | 71.6 | 91.1 | 79.0 | 66.8 | 79.1 | 75.6
RTMO-l$\dagger$ | CSPDarknet | 44.8M | 19.1 | 73.3 | 91.9 | 80.8 | 68.3 | 81.1 | 77.4
Table 1: Performance comparison of state-of-the-art one-stage methods on the
COCO test-dev dataset. The symbol $\dagger$ denotes models trained with
additional data beyond the COCO train2017 dataset. Inference time in italic
are obtained using a single NVIDIA Tesla V100 GPU, while times without this
emphasis are from PETR’s paper [36] and evaluated using the same device. Times
underlined were measured using PyTorch due to ONNX exportation
incompatibilities with those models.

#### COCO

To assess RTMO against other real-time pose estimators, we measured AP and
inference latency on the COCO val2017 dataset. For one-stage methods, we
considered KAPAO [33], YOLOv8-Pose [16], and YOLOX-Pose—an adaptation of YOLO-
Pose [30] on YOLOX [9]. For top-down approaches, RLE [19], SimCC [22] and
RTMPose [15] were selected for comparison. RTMDet-nano [29], a highly
efficient object detection model, served as the human detector for top-down
models. Since top-down models slow down as more people appear in the image, we
partitioned the COCO val2017 set based on person counts and assessed top-down
model speeds accordingly. As shown in Fig. 3, the RTMO series outperform
comparable lightweight one-stage methods in both performance and speed.
Against top-down models, RTMO-m and RTMO-l are as accurate as RTMPose-m and
RTMPose-l, and faster when more people are in the image. With ONNXRuntime,
RTMO matches RTMPose in speed with around four people, and with TensorRT FP16,
RTMO is quicker with two or more people. This demonstrates RTMO’s advantage in
multi-person scenarios. Importantly, although the number of tokens processed
varies with the number of people in the image, the difference in inference
latency is marginal. For example, the latency of RTMO-l on a GPU in a subset
with more than 10 persons is only about 0.1 ms higher than in a subset with a
single person, accounting for roughly 0.5% of the total latency.

Figure 4: Visualization of estimated human pose (top) and corresponding
heatmaps (bottom).

Our evaluation of RTMO against leading one-stage pose estimators on the COCO
test-dev is presented in Table 1. RTMO showcases significant advancements in
speed and precision. Specifically, RTMO-s outperforms PETR [36] using a
ResNet-50 [11] backbone, being ten times quicker yet maintaining similar
accuracy. Compared to lightweight models like KAPAO and YOLO-Pose, RTMO
consistently outperforms in accuracy across different model sizes. When
trained on COCO train2017, RTMO-l has the second-best performance among all
tested models. The highest-performing model, ED-Pose [46] with a Swin-L [25]
backbone, is quite heavy and not deployment-friendly. RTMO, using the same
ResNet-50 backbone, surpassed ED-Pose by 1.1% in AP and was faster.
Additionally, transferring ED-Pose to the ONNX format resulted in a slower
latency than its PyTorch model, about 1.5 seconds per frame. By contrast, the
ONNX model of RTMO-l processes an image in just 19.1ms. With further training
on additional human pose datasets, RTMO-l performs best among one-stage pose
estimators in terms of accuracy.

#### CrowdPose

Method |  #Params |  AP |  $\textrm{AP}_{E}$ |  $\textrm{AP}_{M}$ |  $\textrm{AP}_{H}$
---|---|---|---|---|---
Top-down methods
Sim.Base. [44] |  34.0M |  60.8 |  71.4 |  61.2 |  51.2
HRNet [38] |  28.5M |  71.3 |  80.5 |  71.4 |  62.5
TransPose-H [47] |  - |  71.8 |  79.5 |  72.9 |  62.2
HRFormer-B [48] |  43.2M |  72.4 |  80.0 |  73.5 |  62.4
RTMPose-m [15] |  13.5M |  70.6 |  79.9 |  71.9 |  58.2
Bottom-up methods
OpenPose [4] |  - |  - |  62.7 |  48.7 |  32.3
HrHRNet [6] |  63.8M |  65.9 |  73.3 |  66.5 |  57.9
DEKR [10] |  65.7M |  67.3 |  74.6 |  68.1 |  58.7
SWAHR [27] |  63.8M |  71.6 |  78.9 |  72.4 |  63.0
One-stage methods
PETR [36] |  220.5M |  71.6 |  77.3 |  72.0 |  65.8
CID [42] |  65.4M |  72.3 |  78.7 |  72.1 |  64.8
KAPAO-l [33] |  77.0M |  68.9 |  76.6 |  69.9 |  59.5
ED-Pose [46] |  218.0M |  73.1 |  80.5 |  73.8 |  63.8
ED-Pose$\dagger$ [46] |  218.0M |  76.6 |  83.3 |  77.3 |  68.3
RTMO-s |  9.9M |  67.3 |  73.7 |  68.2 |  59.1
RTMO-m |  22.6M |  71.1 |  77.4 |  71.9 |  63.4
RTMO-l |  44.8M |  73.2 |  79.2 |  74.1 |  65.3
RTMO-l$\dagger$ |  44.8M |  81.7 |  86.9 |  82.5 |  74.7
Table 2: Performance comparisons with state-of-the-art methods on CrowdPose.
The highest and second-highest performances are highlighted in bold and
underlined, respectively. $\dagger$ indicates that the model was trained using
additional data beyond CrowdPose.

To evaluate RTMO under challenging scenarios, we test it on the CrowdPose [18]
benchmark, characterized by images with dense crowds, significant person
overlap, and occlusion. The results are summarized in Table 2. Among bottom-up
and single-stage approaches, RTMO-s has accuracy comparable to DEKR [10], yet
it uses only 15% of the parameters. When trained on the CrowdPose dataset,
RTMO-l surpasses ED-Pose [46] which uses a Swin-L [25] backbone, despite
having a smaller model size. Notably, RTMO-l exceeds ED-Pose primarily on
medium and hard samples, demonstrating its effectiveness in challenging
situations. Moreover, with additional training data, RTMO-l achieves a state-
of-the-art 81.7% AP, highlighting the model’s capacity.

###  4.3 Quantitative Results

RTMO employs the coordinate classification method and shows strong performance
in challenging multi-person scenarios, where individuals are relatively small
and frequent occlusions occur. Figure 4 reveals that RTMO generates spatially
accurate heatmaps even under these difficult conditions, facilitating robust
and context-aware predictions for each keypoint.

###  4.4 Ablation Study

Decoding | loss | COCO | CrowdPose
---|---|---|---
AP | AR | AP | AR
Regression | OKS | 65.6 | 69.9 | 66.1 | 70.9
CC+DBA+DBE | KLD | 64.4 | 68.2 | 62.5 | 67.5
CC | MLE | 66.7 | 70.6 | 65.8 | 70.7
CC+DBA | MLE | 65.6 | 70.0 | 65.2 | 70.2
CC+DBA+DBE | MLE | 67.7 | 71.5 | 67.3 | 72.3
Table 3: Comparison of decoding and supervision methods on COCO val2017 and
CrowdPose. The base model is RTMO-s. The term CC denotes Coordinate
Classification; DBA and DBE denote Dynamic Bin Allocation and Dynamic Bin
Encoding.

#### Classification v.s. Regression.

To assess the effectiveness of coordinate classification against regression,
we replaced the model’s 1-D heatmap generation with a fully-connected layer
for regression, supervised by the OKS loss [30]. Table 3 compares the
performances. Using the DCC module and MLE loss, coordinate classification
outperforms regression with 2.1% AP on the COCO.

#### Losses for Coordinate Classification.

Contrary to other pose estimation methods with coordinate classification that
use KLD as the loss function, our research indicates its inadequacy for RTMO.
Evidence from Table 3 demonstrates lower accuracy with KLD compared to our MLE
loss. We attribute this improvement to the inclusion of learnable variance in
the MLE loss function, which helps to balance the learning between hard and
easy samples. Specifically, in a one-stage pose estimator framework, the
difficulty level varies for each grid and is influenced by several factors,
including the assigned instance’s pose and size, as well as the relative
position between the grid and instance. KLD fails to account for this
variability, making it less effective in this context.

#### Dynamic Strategy in Coordinate Classification.

|
---|---
Figure 5: Heatmaps decoded without (left) and with (right) DBE.

We firstly adopt a static coordinate classification strategy similar to DFL
[20] where bins are distributed in a fixed range around each grid. This
approach outperformed the regression method on the COCO dataset but
underperformed on CrowdPose. Introducing the Dynamic Bin Allocation (DBA)
strategy to this baseline resulted in decreased performance on both datasets.
This is reasonable, as the semantic of each bin vary across different samples
without corresponding representation adjustments. This issue was rectified by
incorporating Dynamic Bin Encoding (DBE). With DBE, our DCC methods exceeded
the efficacy of the static strategy on both datasets. Furthermore, Without
dynamic bin encoding (DBE), the probabilities of nearby bins can vary
significantly, as shown in Figure 5, contradicting the expectation that
adjacent spatial locations should have similar probabilities. In contrast,
incorporating DBE leads to smoother output heatmaps, indicating improved
decoder training by enabling representation vectors that better capture
similarities between nearby locations.

#### Feature Maps Selection.

Model | features | Latency (ms) | Accuracy
---|---|---|---
CPU | GPU | AP | AR
RTMO-s | {P3, P4, P5} | 65.3 | 8.96 | 67.6 | 71.8
{P4, P5} | 48.7 | 8.91 | 67.7 | 71.5
RTMO-m | {P3, P4, P5} | 108.6 | 16.87 | 71.4 | 75.1
{P4, P5} | 80.2 | 12.40 | 70.9 | 74.7
RTMO-l | {P3, P4, P5} | 186.3 | 22.01 | 72.7 | 76.9
{P4, P5} | 125.4 | 19.09 | 72.4 | 76.2
Table 4: Comparison of performance and latency for models using 2 or 3
features. Accuracy metrics are based on the COCO val2017 dataset. Latency
measurements for both CPU and GPU are taken using ONNXRuntime.

Feature pyramids [17] leverage multi-scale features for detecting instances of
varying sizes; deeper features typically detect larger objects. Our initial
model, inspired by YOLOX, used P3, P4, P5 features with strides of 8, 16, and
32 pixels. However, P3 contributed 78.5% of the FLOPs in the model head while
accounting for 10.7% of correct detections. To improve efficiency, we focused
on P4, P5. As shown in Table 4, omitting P3 led to significant speed gains
with minimal accuracy loss, indicating that P4 and P5 alone are effective for
multi-person pose estimation. This demonstrates that the role of P3 in
detecting smaller instances can be sufficiently compensated by the remaining
features.

##  5 Conclusion

In conclusion, our RTMO model significantly improves the speed-accuracy
tradeoff in one-stage multi-person pose estimation. By integrating coordinate
classification within a YOLO-based framework, we achieve both real-time
processing and high precision. Our approach, featuring a dynamic coordinate
classifier and a loss function based on maximum likelihood estimation,
effectively improves the location precision in dense prediction models. This
breakthrough not only enhances pose estimation, but also establishes a robust
foundation for future advancements in the scope of dense prediction for visual
detection tasks.

## References

  * Andriluka et al. [2014] Mykhaylo Andriluka, Leonid Pishchulin, Peter V. Gehler, and Bernt Schiele.  2D human pose estimation: New benchmark and state of the art analysis.  In _CVPR_ , 2014.
  * Andriluka et al. [2018] Mykhaylo Andriluka, Umar Iqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele.  Posetrack: A benchmark for human pose estimation and tracking.  In _CVPR_ , 2018.
  * [3] PaddlePaddle Authors.  Paddledetection, object detection and instance segmentation toolkit based on paddlepaddle.  https://github.com/PaddlePaddle/PaddleDetection.
  * Cao et al. [2019] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields.  _TPAMI_ , 2019.
  * Chang et al. [2020] Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei.  Data uncertainty learning in face recognition.  In _CVPR_ , 2020.
  * Cheng et al. [2020] Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S Huang, and Lei Zhang.  Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation.  In _CVPR_ , 2020.
  * Duan et al. [2019] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian.  Centernet: Keypoint triplets for object detection.  In _ICCV_ , 2019.
  * Fang et al. [2022] Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu.  Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time.  _TPAMI_ , 2022.
  * Ge et al. [2021] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun.  YOLOX: Exceeding yolo series in 2021.  _arXiv preprint arXiv:2107.08430_ , 2021.
  * Geng et al. [2021] Zigang Geng, Ke Sun, Bin Xiao, Zhaoxiang Zhang, and Jingdong Wang.  Bottom-up human pose estimation via disentangled keypoint regression.  In _CVPR_ , 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * He et al. [2019] Yihui He, Chenchen Zhu, Jianren Wang, Marios Savvides, and Xiangyu Zhang.  Bounding box regression with uncertainty for accurate object detection.  In _CVPR_ , 2019.
  * Hua et al. [2022] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le.  Transformer quality in linear time.  In _ICML_ , 2022.
  * Jhuang et al. [2013] Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael J Black.  Towards understanding action recognition.  In _ICCV_ , 2013.
  * Jiang et al. [2023] Tao Jiang, Peng Lu, Li Zhang, Ningsheng Ma, Rui Han, Chengqi Lyu, Yining Li, and Kai Chen.  RTMPose: Real-time multi-person pose estimation based on mmpose.  _arXiv preprint arXiv:2303.07399_ , 2023.
  * Jocher et al. [2023] Glenn Jocher, Ayush Chaurasia, and Jing Qiu.  YOLO by Ultralytics.  https://github.com/ultralytics/ultralytics, 2023.  Accessed: February 30, 2023.
  * Kim et al. [2018] Seung-Wook Kim, Hyong-Keun Kook, Jee-Young Sun, Mun-Cheon Kang, and Sung-Jea Ko.  Parallel feature pyramid network for object detection.  In _ECCV_ , 2018.
  * Li et al. [2019] Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu.  Crowdpose: Efficient crowded scenes pose estimation and a new benchmark.  In _CVPR_ , 2019.
  * Li et al. [2021a] Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, and Cewu Lu.  Human pose regression with residual log-likelihood estimation.  In _ICCV_ , 2021a.
  * Li et al. [2022a] Xiang Li, Chengqi Lv, Wenhai Wang, Gang Li, Lingfeng Yang, and Jian Yang.  Generalized focal loss: Towards efficient representation learning for dense object detection.  _TPAMI_ , 2022a.
  * Li et al. [2021b] Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, and Erjin Zhou.  Tokenpose: Learning keypoint tokens for human pose estimation.  In _ICCV_ , 2021b.
  * Li et al. [2022b] Yanjie Li, Sen Yang, Peidong Liu, Shoukui Zhang, Yunxiao Wang, Zhicheng Wang, Wankou Yang, and Shu-Tao Xia.  Simcc: A simple coordinate classification perspective for human pose estimation.  In _ECCV_ , 2022b.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In _ECCV_ , 2014.
  * Liu et al. [2016] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In _ECCV_ , 2016.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _ICCV_ , 2021.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Luo et al. [2021] Zhengxiong Luo, Zhicheng Wang, Yan Huang, Liang Wang, Tieniu Tan, and Erjin Zhou.  Rethinking the heatmap regression for bottom-up human pose estimation.  In _CVPR_ , 2021.
  * Lv et al. [2023] Wenyu Lv, Shangliang Xu, Yian Zhao, Guanzhong Wang, Jinman Wei, Cheng Cui, Yuning Du, Qingqing Dang, and Yi Liu.  Detrs beat yolos on real-time object detection.  _arXiv preprint arXiv:2304.08069_ , 2023.
  * Lyu et al. [2022] Chengqi Lyu, Wenwei Zhang, Haian Huang, Yue Zhou, Yudong Wang, Yanyi Liu, Shilong Zhang, and Kai Chen.  RTMDet: An empirical study of designing real-time object detectors.  _arXiv preprint arXiv:2212.07784_ , 2022.
  * Maji et al. [2022] Debapriya Maji, Soyeb Nagori, Manu Mathew, and Deepak Poddar.  Yolo-pose: Enhancing yolo for multi person pose estimation using object keypoint similarity loss.  In _CVPR Workshop_ , 2022.
  * Mao et al. [2021] Weian Mao, Zhi Tian, Xinlong Wang, and Chunhua Shen.  Fcpose: Fully convolutional multi-person pose estimation with dynamic instance-aware convolutions.  In _CVPR_ , 2021.
  * Mao et al. [2022] Weian Mao, Yongtao Ge, Chunhua Shen, Zhi Tian, Xinlong Wang, Zhibin Wang, and Anton van den Hengel.  Poseur: Direct human pose regression with transformers.  In _ECCV_ , 2022.
  * McNally et al. [2022] William McNally, Kanav Vats, Alexander Wong, and John McPhee.  Rethinking keypoint representations: Modeling keypoints and poses as objects for multi-person human pose estimation.  In _ECCV_ , 2022.
  * Nie et al. [2019] Xuecheng Nie, Jiashi Feng, Jianfeng Zhang, and Shuicheng Yan.  Single-stage multi-person pose machines.  In _ICCV_ , 2019.
  * Shi et al. [2021] Dahu Shi, Xing Wei, Xiaodong Yu, Wenming Tan, Ye Ren, and Shiliang Pu.  Inspose: instance-aware networks for single-stage multi-person pose estimation.  In _ACMMM_ , 2021.
  * Shi et al. [2022a] Dahu Shi, Xing Wei, Liangqi Li, Ye Ren, and Wenming Tan.  End-to-end multi-person pose estimation with transformers.  In _CVPR_ , 2022a.
  * Shi et al. [2022b] Dahu Shi, Xing Wei, Liangqi Li, Ye Ren, and Wenming Tan.  End-to-end multi-person pose estimation with transformers.  In _CVPR_ , 2022b.
  * Sun et al. [2019] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In _CVPR_ , 2019.
  * Tian et al. [2019a] Zhi Tian, Hao Chen, and Chunhua Shen.  Directpose: Direct end-to-end multi-person pose estimation.  _arXiv preprint arXiv:1911.07451_ , 2019a.
  * Tian et al. [2019b] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In _CVPR_ , 2019b.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NeurIPS_ , 2017.
  * Wang and Zhang [2022] Dongkai Wang and Shiliang Zhang.  Contextual instance decoupling for robust multi-person pose estimation.  In _CVPR_ , 2022.
  * Wu et al. [2017] Jiahong Wu, He Zheng, Bo Zhao, Yixin Li, Baoming Yan, Rui Liang, Wenjia Wang, Shipei Zhou, Guosen Lin, Yanwei Fu, et al.  Ai challenger: A large-scale dataset for going deeper in image understanding.  _arXiv preprint arXiv:1711.06475_ , 2017.
  * Xiao et al. [2018] Bin Xiao, Haiping Wu, and Yichen Wei.  Simple baselines for human pose estimation and tracking.  In _ECCV_ , 2018.
  * Xu et al. [2022] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao.  Vitpose: Simple vision transformer baselines for human pose estimation.  In _NeurIPS_ , 2022.
  * Yang et al. [2023] Jie Yang, Ailing Zeng, Shilong Liu, Feng Li, Ruimao Zhang, and Lei Zhang.  Explicit box detection unifies end-to-end multi-person pose estimation.  In _ICLR_ , 2023.
  * Yang et al. [2021] Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang.  Transpose: Keypoint localization via transformer.  In _ICCV_ , 2021.
  * Yuan et al. [2021] Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang.  HRFormer: High-resolution vision transformer for dense predict.  In _NeurIPS_ , 2021.
  * Zhang et al. [2018] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  Mixup: Beyond empirical risk minimization.  In _ICLR_ , 2018.
  * Zhang et al. [2021] Haoyang Zhang, Ying Wang, Feras Dayoub, and Niko Sunderhauf.  Varifocalnet: An iou-aware dense object detector.  In _CVPR_ , 2021.
  * Zhou et al. [2019] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  _arXiv preprint arXiv:1904.07850_ , 2019.


Jay Kejriwal1,2, Štefan Beňuš1,3, Lina M. Rojas-Barahona4

# Unsupervised Auditory and Semantic Entrainment Models with Deep Neural
Networks

###### Abstract

Speakers tend to engage in adaptive behavior, known as entrainment, when they
become similar to their interlocutor in various aspects of speaking. We
present an unsupervised deep learning framework that derives meaningful
representation from textual features for developing semantic entrainment. We
investigate the model's performance by extracting features using different
variations of the BERT model (DistilBERT and XLM-RoBERTa) and Google's
universal sentence encoder (USE) embeddings on two human-human (HH) corpora
(The Fisher Corpus English Part 1, Columbia games corpus) and one human-
machine (HM) corpus (Voice Assistant Conversation Corpus (VACC)). In addition
to semantic features we also trained DNN-based models utilizing two auditory
embeddings (TRIpLet Loss network (TRILL) vectors, Low-level descriptors (LLD)
features) and two units of analysis (Inter pausal unit and Turn). The results
show that semantic entrainment can be assessed with our model, that models can
distinguish between HH and HM interactions and that the two units of analysis
for extracting acoustic features provide comparable findings.

Index Terms: entrainment, deep learning, unsupervised, DNN embeddings,
behavioral signal processing, conversations, interaction.

##  1 Introduction

Entrainment in spoken interaction is the tendency of speakers to adjust some
properties of their speech to match the characteristics of their
interlocutors. It affects several linguistic dimensions, such as lexical
choice [1], syntactic structure [2], acoustic prosodic features [3], or
semantic similarity [4]. In addition, it correlates with different social
aspects of the conversation, such as task success [5], liking [6], cooperation
[7], or naturalness and rapport [8].

Assessing and formally modelling entrainment in both human-human (HH) and
human-machine (HM) spoken interactions is widely researched particularly due
to the assumption that implementing natural entrainment functionalities
reliably detected in HH dialogues would increase the naturalness and efficacy
of applications using HM spoken interactions. To this end, several methods for
measuring entrainment have been used ranging from time-series analyses,
Pearson’s correlations, recurrence analyses, and spectral methods [9]. Most of
these approaches, however, assume a linear relationship between features of
adjacent speaker turns. Yet, current understanding of entrainment is that it
is a complex multi-layered phenomenon. For example, in [10], the authors
reported that also dis-entrainment could sometimes enhance conversation
development. It seems that linear functions cannot capture this diverse and
complex nature of entrainment.

A potential approach addressing this issue was suggested in [11] who developed
a DNN-based Neural entrainment distance (NED) that uses a non-linear function
to measure auditory entrainment. They developed a novel unsupervised model,
which relies on an auto-encoder architecture to produce the next turn's
features. Instead of compressing and reconstructing the original embeddings,
the main goal of this architecture was to learn the representation of the next
turn based on the previous turn. Using these bottleneck features, NED
represents the degree of auditory entrainment. With low-level description
(LLD) auditory features extracted from inter-pausal units (IPUs) they reported
classification accuracy of 98.87% on Fisher corpus English Part 1 [12]. In
their extended work, [13] proposed a triplet network-based approach where they
used i-vectors for training the DNN model measuring auditory entrainment. They
compared the performance of both approaches on different corpora and reported
that the NED-based approach with LLD features provides the highest accuracy of
98.87% in the Fisher corpus but the triplet network-based approach with
i-vectors trained on Fisher corpus was best (94.63%) in the Suicide Risk
Assessment corpus [14]. Finally, [15], proposed a DNN based entrainment model
that isolates the effect of consistency; i.e. the tendency to adhere to one's
own vocal style. The authors extracted LLD features from each adjacent IPU of
both speakers and trained the DNN model using the deconfounding measures
proposed in [16]. They reported that their model performs slightly worse than
the NED measure, with an accuracy of 92.3%.

In this paper we have two main goals in extending this promising line of work.
While the mentioned studies explore the auditory modality of entrainment, to
fully understand HH entrainment and design applications using it in HM
interactions, also the textual modality has to be included. Hence, our first
goal is to propose augmenting the original auto-encoder NED-based DNN
architecture so that it could be used to assess also semantic entrainment. We
test our approach and the performance of the models using variations of BERT
and Google's Universal Sentence Encoder (USE).

The second goal is to test some aspects of the auditory NED-based approach in
the effort to improve our understanding of its usability and reliability.
First, it is not clear how different features, embeddings and units of
analysis affect the performance of these models. Several types of auditory
features have been identified for detecting auditory entrainment, including
LLDs comprising temporal, spectral, and acoustic-prosodic features or DNN
embeddings such as i-vectors. Also, TRILL vectors have shown state-of-the-art
performance in a classification task related to detecting stress in speech
outperforming LLDs and also the above-mentioned i-vectors [17]. Therefore, we
test these types of features with the NED-based architecture on three
different spoken corpora. Regarding the unit of analysis for extracting the
features, IPUs are more common in this research but the entire turns might
provide richer representations of speaker's characteristics. Hence, we
designed experiments testing the performance of LLD and TRILL features on NED-
based auditory entrainment models on three datasets using two different units
of analysis.

Second, in the original papers describing the development of the NED-based
model, cross-validation is done using the hold-out method, with data split
into 80:10:10 for training, validation, and testing respectively. However,
k-fold cross-validation, i.e. randomly splitting the dataset into ‘k’ groups,
have been shown to provide more consistent results on smaller and larger
datasets with quality classification than the hold-out [18]. Hence, it is
warranted to evaluate the performance of the NED-based models using 10-fold
cross-validation.

Finally, given the broader goal of developing entrainment functionalities in
spoken HM interactions, it is critical to have robust and reliable measures of
entrainment in both HH and HM scenarios of various domains and genres. We thus
employ two HH corpora (The Fisher Corpus English Part 1, Columbia games
corpus) and one HM corpus (Voice Assistant Conversation Corpus, VACC)) to test
if the NED-based entrainment measure can safely distinguish between HH and HM
interactions.

In sum, we propose adjustments to the auto-encoder architecture for using NED-
based approach also for semantic entrainment, and we compare the performance
of DNN-auditory entrainment models by training them on different features and
units of analysis using three different corpora to understand if entrainment
can be captured in HH and HM interactions. We also compare the performance of
DNN-auditory and semantic entrainment models using different auditory and
semantic embeddings by splitting the dataset with 10-fold cross-validation,
which reduces the variance. The experimental results show that the auditory
NED model provides better accuracy with TRILL vectors, and XLM-RoBERTa
embeddings provide better accuracy on the semantic entrainment model. This
implies that non-semantic speech representation (DNN embeddings) for speech
and textual embeddings can be useful for speech and semantic entrainment
detection in a uni-model architecture.

##  2 Data and features

###  2.1 Datasets

The Fisher Corpus English Part 1 [12] consists of 5850 spontaneous telephone
dyadic conversations between native English speakers. During each session, two
previously unacquainted subjects discuss a topic for 10 min. The dataset has
984 hours of speech, contains manual transcripts with time stamps for the
speakers' turn and pauses enabling us to extract turns and IPUs.

Columbia Games Corpus [19] contains 12 dyadic conversations between native
speakers of Standard American English. Six females and seven males
participated, 11 participated in two sessions on different days. The dyads
played four computer games, requiring verbal collaboration. A curtain ensured
only verbal communication. Over 9h of recordings were made.

Voice Assistant Conversation Corpus (VACC) [20] includes recordings of 27
native German speakers. An Amazon Echo Dot 2nd generation was used to explore
the interactions between human-computer (solo condition) and human-human-
computer (confederate condition). In the interaction, Calendar (formal) and
Quiz (informal) tasks were used. In the Calendar task, the participant
scheduled several appointments with the confederate during predefined weeks by
interacting with Alexa. In the Quiz task, the participant answered trivia
questions, such as “When was Albert Einstein born?” In total, approximately
13,500 utterances in over 17h was recorded (31 minutes average per
interaction). Information about speakers and turn times was manually
annotated. In the current study, we used the solo condition only.

###  2.2 Feature extraction

We extracted LLDs and TRILL vectors from the three datasets. For LLDs we used
the OpenSMILE toolkit [21]. LLDs involve 38 features comprising 4 prosody ones
(pitch, energy, their deltas), 31 spectral ones (15 MFCCs, 8 MFBs, 8 LSFs),
and 3 voice quality ones (shimmer, two variants of jitter). Furthermore, we
extracted 6 functionals of each acoustic feature (mean, median, standard
deviation, 1st percentile, 99th percentile, and range (99th percentile - 1st
percentile). We extracted 38 $\times$ 6 = 228 LLD features for each unit (IPU
or turn). Step size and time frame were set at 25ms and 10ms, respectively. In
addition, we z-score normalized the features. We used the TRILL vector [22] on
all three datasets to extract each unit's auditory embeddings representing 512
one-dimensional vectors.

Furthermore, we extracted semantic features using three different neural
network-trained models. For the two English corpora we used the fine-tuned
DistilBERT [23], where each turn is encoded into 768 one-dimensional semantic
features. For the German dataset (VACC), we utilized XLM-RoBERTa [24], where
each unit is encoded into 768 one-dimensional vectors. For comparison, we also
extracted semantic features from all three datasets using Google's neural
network-trained model Universal Sentence Encoder (USE) [25], representing 512
one-dimensional semantic features for each unit.

###  2.3 Modelling with DNN

####  2.3.1 Auditory modelling

We utilized the neural architecture as proposed in [11] illustrated in Figure
1 (a). As a first step, ${x_{1}}$ is the input to the encoder network. By
restricting the dimensionality of z to be lower than that of ${x_{1}}$, the
output of the encoder network that z represents is under complete
representation of ${x_{1}}$. Secondly, a feed-forward network is used as a
decoder to predict $\tilde{x_{1}}$ from z. Lastly, the loss function compares
$\tilde{x_{1}}$ with its reference ${x_{2}}$. After training the model,
bottleneck embedding z can be obtained from the encoder network of the trained
model to measure neural entrainment distance. Unlike classical VAE, which
reconstructs the same vector in compressed form, the bottleneck embedding z
contains relevant entrainment information of ${x_{1}}$ and ${x_{2}}$.

We trained two DNN models with different auditory features for measuring
auditory entrainment: LLD and TRILL. Both models have the same architecture.
The architecture comprises two fully connected (FC) hidden layers in both the
encoder and decoder networks. Both networks use batch normalization layers and
Rectified Linear Unit (ReLU) activation layers between fully connected layers.
The bottleneck embedding dimension is 30. For the LLD model, Hidden layers
contain the following neuron units: [228 $\rightarrow$128 $\rightarrow$30
$\rightarrow$128 $\rightarrow$228]. We used the smooth L1 norm as the loss
function and the Adam optimizer. For the TRILL model, the number of neuron
units in the hidden layers is: [512 $\rightarrow$128 $\rightarrow$30
$\rightarrow$128 $\rightarrow$512]. We used the Kullback-Leibler (KL)
divergence equation (1) as the loss function, and the Adam optimizer was
applied. The number of epochs was set to 10. A 10-fold cross-validation
technique was used to evaluate the models with a batch size of 128.

$\displaystyle
KL({\tilde{x_{1}}},{x_{2}})={x_{2}}\cdot\log(\frac{{x_{2}}}{\tilde{x_{1}}})={x_{2}}\cdot(\log{x_{2}}-\log\tilde{x_{1}})$
(1)

Figure 1: An overview of (a) Auditory and (b) Semantic entrainment models

####  2.3.2 Semantic modelling

We propose a different architecture for measuring semantic entrainment.
Instead of learning representation from $\tilde{x_{1}}$ to ${x_{2}}$, we used
auto-encoder architecture where loss from ${x_{1}}$ to $\tilde{x_{1}}$ is
calculated as reconstruction loss. Secondly, we also measure the loss on
$\tilde{x_{1}}$ to ${x_{2}}$ as mapping loss as shown in Figure 1 (b). Summing
the two losses and backpropagation allows the model to learn semantic
entrainment of the next turn based on the previous turn. These bottleneck
features determine the Neutral Entrainment Distance (NED), a semantic
entrainment measure.

For semantic entrainment, we trained two DNN models: BERT and USE, for each
dataset. The DNN models share the same auto-encoder architecture. Two fully
connected (FC) hidden layers in both encoder and decoder networks. Between
fully connected layers, both networks use batch normalization layers and
Rectified Linear Units (ReLU). The bottleneck embedding size for the BERT
model is 384, whereas the USE model is 30. For the BERT model, Hidden layers
contain the following neuron units: [768 $\rightarrow$512 $\rightarrow$384
$\rightarrow$512 $\rightarrow$768]. Similarly, the USE model the size is: [512
$\rightarrow$128 $\rightarrow$30 $\rightarrow$128 $\rightarrow$512]. We used
the smooth L1 norm as the loss function and the Adam optimizer in both models.
A 10-fold cross-validation technique was used to evaluate the models with a
batch size of 128.

###  2.4 Neural entrainment distance (NED) measure

After training the model, inputting the features into the encoder of the
trained model will provide the bottleneck embeddings of the DNN model. The NED
between these bottleneck embeddings is measured using different methods.

Auditory model: For the LLD model, we measure NED using the absolute
difference between two bottleneck embeddings (say x1 and x2), as shown in
equation (2). For the TRILL model, we measure NED using cosine distance
between two bottleneck embeddings, as shown in equation (3).

Semantic model: For both BERT and USE models, we used cosine distance Eq.(3)
as NED.

$\displaystyle NED_{lld}=\mid x_{1}-x_{2}\mid$ (2)

$\displaystyle NED_{cosine\ distance}=\cos(x_{1},x_{2})=\frac{x_{1}\cdot
x_{2}}{|x_{1}||x_{2}|}$ (3)

##  3 Results and Discussion

In order to validate NED as a valid proximity metric for entrainment, we ran
three different classification experiments. The steps to measure the
performance of each model are as follows: We measure two sets of distances for
the entire session, namely consecutive NED and non-consecutive NED.
Consecutive NED is the distance between two consecutive turns of speakers A
and B. Non-consecutive NED is the distance between a speaker's turn and
another random non-consecutive turn of another speaker. For DNN embeddings, if
the consecutive NED is greater than the non-consecutive NED, we can infer that
people are getting closer to each other and model is classifying cases
correctly. Similarly, if consecutive NED distance is lower for LLD features,
we can infer that people are entraining to each other, and these cases are
categorized correctly. Later, the model accuracy is traditionally calculated
as the proportion of correct cases.

| Columbia games corpus | VAC corpus | Fisher corpus
---|---|---|---
Feature | IPU | Turn | IPU | Turn | IPU | Turn
LLD | 74.69($\pm 3.81)$ | 74.98 ($\pm 3.83)$ |  77.94 ($\pm 3.84)$ | 77.26 ($\pm 3.97)$ | 83.94 ($\pm 0.13)$ | 84.14 ($\pm 0.03)$
a) Classification accuracy for IPU vs Turn in three datasets
| A to B | B to A | Spkr to Alexa | Alexa to Spkr | A to B | B to A
LLD | 72.29 ($\pm 3.25)$ | 74.29 ($\pm 3.23)$ | 58.25 ($\pm 4.98)$ | 78.87 ($\pm 2.12)$ | 84.13 ($\pm 0.12)$ | 80.23 ($\pm 0.07)$
b) Classification accuracy by splitting into groups
| One RT | Ten RT | One RT | Ten RT | One RT | Ten RT
LLD |  74.98 ($\pm 3.83)$ | 71.23 ($\pm 3.45)$ |  77.26 ($\pm 3.97)$ | 76.83 ($\pm 2.11)$ | 84.14 ($\pm 0.03)$ | 80.16 ($\pm 0.08)$
TRILL | 53.98 ($\pm$3.76) | 56.17 ($\pm$5.82) | 31.17 ($\pm 3.67)$ | 29.79 ($\pm 3.16)$ | 94.14 ($\pm 0.01)$ | 93.75 ($\pm 0.02)$
BERT | 55.05 ($\pm 4.67)$ | 57.12 ($\pm 4.67)$ | 66.64 ($\pm 6.28)$ | 67.55 ($\pm 5.21)$ | 60.31 ($\pm 0.04)$ | 61.64 ($\pm 0.04)$
USE | 47.03 ($\pm 3.35)$ | 47.85 ($\pm 3.35)$ | 42.10 ($\pm 9.18)$ | 44.45 ($\pm 8.91)$ | 62.03 ($\pm 0.06)$ | 63.05 ($\pm 0.06)$
c) Classification accuracy for selecting different random turns (RT)

Table 1: Summary of Classification accuracy for different auditory and
semantic features on Columbia games corpus, Voice Assistant Conversation
Corpus (VACC), and The Fisher Corpus English Part 1 (standard deviation shown
in parentheses)

###  3.1 Experimnent 1: IPU vs Turn

To analyze the effect of units of analysis on model performance, we trained
six DNN entrainment models on three datasets using two units of analysis by
extracting LLD features. Table 1 (a) shows the accuracy of different units of
analysis on three different datasets. In the Columbia games, VACC, and Fisher
corpus, we found that models trained on the turn as a unit of analysis provide
an accuracy of 74.98%, 77.26%, and 84.14%, whereas models trained on IPU
provide an accuracy of 74.69%, 77.94%, and 83.94%. Hence, no significant
difference in model performance trained on IPU and Turn in three datasets was
found.

###  3.2 Experiment 2: Comparison in human-human and human-machine interaction

To analyze if DNN auditory entrainment models can distinguish between HH and
HM interaction we split each dataset into two groups: one has turns spoken by
Speaker A first, followed by Speaker B, and the other has turns spoken by
Speaker B, followed by Speaker A. We extracted LLD features from each group
and trained the neural network models separately. Table 1 (b) shows the
accuracy of different groups on three different datasets. We found that the
accuracy of the VACC corpus in the Speaker to Alexa group drops to 58.25% from
78.82% in Alexa to Speaker. In the two HH corpora we did not find such a
robust change between the groups. The drop in model performance of the VACC
corpus can be explained as Alexa not entraining on auditory features with the
speaker. Comparing consecutive and non-consecutive NED provides similar NED
distance resulting in poor performance. We found DNN auditory models can
distinguish between HH and HM interactions.

###  3.3 Experiment 3: Comparison using different auditory and semantic
embeddings

Table 1 (c) shows the accuracy of different auditory and semantic entrainment
models trained on different embeddings. To assess the robustness of the model,
we measured non-consecutive distance in two ways. First, we used NED with one
random turn (RT), and then we randomly selected ten non-consecutive turns of
another speaker and calculated the mean NED.

####  3.3.1 Experiment with Auditory features

We compared the performance of LLDs and TRILL vectors on three datasets; rows
1-2 of Table 1 (c). In the Columbia games and VACC corpora, LLD features
outperformed TRILL vectors and provided higher accuracy of 74.98% and 77.26%
when consecutive NED was compared to one random non-consecutive NED. Further,
the performance dropped slightly when consecutive NED is compared to the mean
of ten random non-consecutive NED on all three datasets. In contrast, TRILL
vectors outperformed LLD features and provided higher accuracy of 94.14% when
consecutive NED was compared to one random non-consecutive NED in the Fisher
corpus and 93.75% when compared to the mean of ten random non-consecutive NED.
We also notice a drop in the performance of the LLD model by 4% when
consecutive NED was compared to the mean of 10 random NED, whereas there was a
negligible drop of 0.39% in the performance of TRILL vectors suggesting more
robustness of the TRILL vector model.

The poor performance of TRILL vectors on two datasets (CGC and VACC) might be
explained by the scarcity of data. TRILL vectors require a large amount of
training data for learning representations in a meaningful way. Both datasets
are smaller in size when compared to the Fisher corpus. The results reported
in the current study on the Fisher Corpus are slightly worse than those
reported in [11] where accuracy was reported at 98.87%; we believe this is
because we utilized a 10-fold cross-validation method instead of the hold-out
method, which affects the performance of the model.

####  3.3.2 Experiment with Semantic features

We train and evaluate the performance of variations of BERT (DistilBERT and
XLM-RoBERTa) and USE embeddings using DNN-based semantic models on three
datasets. Rows 3-4 in Table 1 (c) show that the Columbia Games corpus shows
DistilBERT outperforms USE with an accuracy of 56.12 % when compared to the
mean of ten non-consecutive NED. Similarly, in the VACC corpus, we found that
XLM-RoBERTa outperforms USE model with an accuracy of 67.55% when consecutive
NED is compared to the mean of ten random non-consecutive NED. In contrast, in
the Fisher corpus, we found that the USE model provides slightly better
accuracy of 63.05% when compared to DistilBERT embeddings (62.03%) when
consecutive NED was compared to ten random non-consecutive turns. We also
noticed an improvement in performance by 2% in CGC and VACC corpora and 1% in
Fisher corpus when consecutive NED was compared to ten random turns.

Importantly, while training the models, we measured two losses: mapping loss
and reconstruction loss. Without calculating the reconstruction loss, our
novel contribution, accuracy dropped by 10% and the model was not learning
much. We can infer that semantic embeddings reduced by auto-encoders affect
the model's performance. Secondly, we achieved the highest accuracy of 63.05%.
The language model for extracting semantic features can lead to this poor
performance. Finally, the bottleneck embedding size affects the performance of
the model. In auditory models, the bottleneck size was 30, whereas, in
semantic models, the size is nearly half of the total embedding size. Reducing
the size of bottleneck embeddings has a significant effect on the performance
of the model.

We also conducted a qualitative error analysis to understand the poor
performance of DNN semantic entrainment models on all three datasets. Table 2
shows few instances on all three datasets where the model measured non-
consecutive NED is smaller than consecutive NED. We observed that it is
difficult for human readers to predict the next consecutive turn based on the
choices shown in the table. We analyzed the Fisher corpus and the Columbia
games corpus and found one probable cause of the errors is when backchannels
and short utterances like okay, yeah, right are compared to measure NED. Both
datasets comprise many affirmative cue words resulting in poor performance. In
the VACC corpus, we observed the errors made by the model are caused when the
model predicts consecutive speaker turn preceded by Alexa's turn. Since Alexa
only responds to the questions asked by the speaker, the model fails to
predict the next consecutive turn of the speaker.

Sr. No | Speaker A | Speaker B (non-consecutive) | Speaker B (consecutive)
---|---|---|---
1 | and nail on the right okay |  | and on the lower right a bottle of
---
wine and a glass half full of wine
| right alien on the top yellow lion
---
on the bottom left and money on
the bottom right
2 | on the top | I don't have anything like that | mmhm
3 | eh I think it's worth taking | oh okay yeah |  | okay so I got outline of a airplane
---
on the top um ball of yarn on the bottom
left and oreo cookie on the bottom right
| (a) Columbia games corpus
1 |  | i don't know i just recovered
---
from a cold right now and uh
oh | okay
2 | and so my strategy is do nothing | right | okay
3 | yeah that's true | oh wow | so how did you get into this study
| (b) Fisher corpus English Part 1
1 |  | Alexa book my appointment
---
on Tuesday the fifth
| Booking an appointment for
---
brainstorming IT at 2 p.m
| on Tuesday 5th December
---
no slots are available
2 |  | On Monday fourth of
---
December there are four dates
Thanks Alexa |  | Alexa what appointments do
---
I have on Tuesday the fifth
3 |  | On Wednesday December 6th
---
there are four appointments
at nine in the morning
| Alexa, what slots are
---
available on wednesday
| Alexa, book my appointments
---
on Wednesday
| (c) VAC corpus

Table 2: Error analysis for semantic entrainment on a) Columbia games corpus,
b) The Fisher Corpus English Part 1, and c) Voice Assistant Conversation
Corpus (VACC)

##  4 Conclusion

In this paper, we measured the performance of auditory and semantic features
and trained DNN-based entrainment models using 10-fold cross-validation in HH
and HMI corpora. We find that the unit of analysis (turn vs IPU) doesn't
affect the performance of the DNN auditory entrainment model. Also, DNN
entrainment models can distinguish between auditory entrainment in HH and HM
interactions. Finally, we found TRILL vectors provide higher accuracy in the
auditory entrainment model, and variation of BERT (XLM-RoBERTa) provides
higher accuracy in DNN based semantic entrainment model.

##  5 Acknowledgements

## References

  * [1] S. E. Brennan and H. H. Clark, ``Conceptual pacts and lexical choice in conversation,'' _Journal of Experimental Psychology: Learning, Memory, and Cognition_ , vol. 22, no. 6, p. 1482–1493, 1996.
  * [2] D. Reitter, J. Moore, and F. Keller, ``Priming of syntactic rules in task-oriented dialogue and spontaneous conversation,'' in _Proceedings of the 28th Annual Conference of the Cognitive Science Society_ , R. Sun, Ed., Vancouver, 2006, p. 685–690.
  * [3] R. Levitan, A. Gravano, and J. Hirschberg, ``Entrainment in speech preceding backchannels.'' in _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_. Portland, Oregon, USA: Association for Computational Linguistics, Jun. 2011, pp. 113–117. [Online]. Available: https://aclanthology.org/P11-2020
  * [4] V. P. Ta, M. J. Babcock, and W. Ickes, ``Developing latent semantic similarity in initial, unstructured interactions: The words may be all you need,'' _Journal of Language and Social Psychology_ , vol. 36, no. 2, pp. 143–166, 2017. [Online]. Available: https://doi.org/10.1177/0261927X16638386
  * [5] D. Reitter and J. D. Moore, ``Predicting success in dialogue,'' in _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_. Prague, Czech Republic: Association for Computational Linguistics, Jun. 2007, pp. 808–815. [Online]. Available: https://aclanthology.org/P07-1102
  * [6] M. E. Ireland, R. B. Slatcher, P. W. Eastwick, L. E. Scissors, E. J. Finkel, and J. W. Pennebaker, ``Language style matching predicts relationship initiation and stability,'' _Psychological Science_ , vol. 22, no. 1, p. 39–44, 2011.
  * [7] J. H. Manson, G. A. Bryant, M. M. Gervais, and M. A. Kline, ``Convergence of speech rate in conversation predicts cooperation,'' _Evolution and Human Behavior_ , vol. 34, p. 419–426, 2013.
  * [8] N. Lubold, H. Pon-Barry, and E. Walker, ``Naturalness and rapport in a pitch adaptive learning companion,'' _2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU_ , p. 103–110, 2015.
  * [9] E. Delaherche, M. Chetouani, A. Mahdhaoui, C. Saint-georges, S. Viaux, and D. Cohen, ``Interpersonal synchrony: A survey of evaluation methods across disciplines,'' _IEEE Transactions on Affective Computing_ , vol. 3, p. 349–365, 2012.
  * [10] J. M. Pérez, R. H. Gálvez, and A. Gravano, ``Disentrainment may be a Positive Thing: A Novel Measure of Unsigned Acoustic-Prosodic Synchrony, and its Relation to Speaker Engagement,'' in _Proc. Interspeech 2016_ , 2016, pp. 1270–1274.
  * [11] M. Nasir, B. Baucom, S. Narayanan, and P. Georgiou, ``Towards an Unsupervised Entrainment Distance in Conversational Speech Using Deep Neural Networks,'' in _Proc. Interspeech 2018_ , 2018, pp. 3423–3427.
  * [12] C. Cieri, D. Miller, and K. Walker, ``The fisher corpus: a resource for the next generations of speech-to-text,'' in _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal. European Language Resources Association (ELRA_ , 2004.
  * [13] M. Nasir, B. Baucom, C. Bryan, S. Narayanan, and P. Georgiou, ``Modeling vocal entrainment in conversational speech using deep unsupervised learning,'' _IEEE Transactions on Affective Computing_ , vol. 13, no. 3, p. 1651–1663, 2022.
  * [14] C. J. Bryan, B. R. Baucom, A. O. Crenshaw, Z. Imel, D. C. Atkins, T. A. Clemans, B. Leeson, T. S. Burch, J. Mintz, and M. D. Rudd, ``Associations of patient-rated emotional bond and vocally encoded emotional arousal among clinicians and acutely suicidal military personnel,'' _Journal of Consulting and Clinical Psychology_ , vol. 86, pp. 372–383, 2018, place: US Publisher: American Psychological Association.
  * [15] A. Weise and R. Levitan, ``Decoupling entrainment from consistency using deep neural networks,'' Nov. 2020, arXiv:2011.01860 [cs]. [Online]. Available: http://arxiv.org/abs/2011.01860
  * [16] R. Pryzant, K. Shen, D. Jurafsky, and S. Wagner, ``Deconfounded lexicon induction for interpretable social science,'' in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 1615–1625. [Online]. Available: https://aclanthology.org/N18-1146
  * [17] J. Kejriwal, S. Benus, and M. Trnka, ``Stress detection using non-semantic speech representation,'' in _2022 32nd International Conference Radioelektronika (RADIOELEKTRONIKA)_ , 2022, pp. 1–5.
  * [18] S. Yadav and S. Shukla, ``Analysis of k-Fold Cross-Validation over Hold-Out Validation on Colossal Datasets for Quality Classification,'' in _2016 IEEE 6th International Conference on Advanced Computing (IACC)_ , Feb. 2016, pp. 78–83.
  * [19] J. Hirschberg, A. Gravano, S. Benus, G. Ward, and E. S. German, _Columbia Games Corpus LDC2021S02. Web Download_. Philadelphia: Linguistic Data Consortium, 2021.
  * [20] I. Siegert, J. Krüger, O. Egorow, J. Nietzold, R. Heinemann, and A. Requardt, ``Voice assistant conversation corpus (vacc): A multi-scenario dataset for addressee detection in human-computer-interaction using amazon's alexa,'' in _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , may 2018.
  * [21] F. Eyben, M. Wöllmer, and B. Schuller, ``Opensmile: the munich versatile and fast open-source audio feature extractor,'' in _Proceedings of the 18th ACM international conference on Multimedia_. New York, NY, USA: Association for Computing Machinery, 2010, pp. 1459–1462,.
  * [22] E. Hoffer and N. Ailon, ``Deep metric learning using triplet network,'' 2018, arXiv:1412.6622 [cs, stat]. arXiv: 1412.6622.
  * [23] N. Reimers and I. Gurevych, ``Sentence-bert: Sentence embeddings using siamese bert-networks,'' 2019, arXiv:1908.10084 [cs]. arXiv:.
  * [24] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, ``Unsupervised cross-lingual representation learning at scale,'' _CoRR_ , vol. abs/1911.02116, 2019. [Online]. Available: http://arxiv.org/abs/1911.02116
  * [25] D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. St. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, B. Strope, and R. Kurzweil, ``Universal sentence encoder for English,'' in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 169–174. [Online]. Available: https://aclanthology.org/D18-2029