# Markerless Motion Capture and Biomechanical Analysis Pipeline

R. James Cotton${}^{1,2,*,\href https://orcid.org/0000-0001-5714-1400}$,
Allison DeLillo1, Anthony Cimorelli1, Kunal Shah1,
J.D. Peiffer${}^{1,3,\href https://orcid.org/0000-0003-2382-8065}$, Shawana
Anarwala1, Kayan Abdou1, Tasos Karakostas1,2 This work was generously
supported by the Research Accelerator Program of the Shirley Ryan AbilityLab
and with funding from the Restore Center P2C (NIH P2CHD101913). We also thank
István Sárándi for advice obtaining confidence estimates from
<EMAIL_ADDRESS>1. Shirley Ryan AbilityLab, 2. Department of
Physical Medicine and Rehabilitation, Northwestern University, 3. Department
of Biomedical Engineering, Northwestern University

###### Abstract

Markerless motion capture using computer vision and human pose estimation
(HPE) has the potential to expand access to precise movement analysis. This
could greatly benefit rehabilitation by enabling more accurate tracking of
outcomes and providing more sensitive tools for research. There are numerous
steps between obtaining videos to extracting accurate biomechanical results
and limited research to guide many critical design decisions in these
pipelines. In this work, we analyze several of these steps including the
algorithm used to detect keypoints and the keypoint set, the approach to
reconstructing trajectories for biomechanical inverse kinematics and
optimizing the IK process. Several features we find important are: 1) using a
recent algorithm trained on many datasets that produces a dense set of
biomechanically-motivated keypoints, 2) using an implicit representation to
reconstruct smooth, anatomically constrained marker trajectories for IK, 3)
iteratively optimizing the biomechanical model to match the dense markers, 4)
appropriate regularization of the IK process. Our pipeline makes it easy to
obtain accurate biomechanical estimates of movement in a rehabilitation
hospital.

## Introduction

Easy-to-use tools for routine and accurate movement analysis, such as gait
analysis, are critical for rehabilitation. They can enable more precise
monitoring of clinical outcomes and power more sensitive research studies. The
clinical gold standard is marker-based optical motion capture [1], but the
time and cost associated with this limit its utilization. Recently, advances
in computer vision and human pose estimation (HPE) [2] have enabled markerless
motion capture approaches from multiple calibrated cameras [3, 4, 5, 6, 7],
potentially expanding access to precise movement analysis.

Markerless motion capture typically involves two key steps: (1) reconstructing
3D keypoint (i.e., virtual marker) trajectories from the HPE output of
multiple calibrated and synchronized cameras and (2) performing inverse
kinematic (IK) fits of a biomechanical model to these trajectories. IK
includes scaling the individual’s biomechanical model and computing the joint
kinematics that best aligns with the trajectories. An additional step may
compute the kinetics, or joint torques and ground reaction forces, but we do
not consider that in this work.

Figure 1: Example biomechanical reconstructions of rehabilitation subjects.
Each row shows a single time point from an individual, with the mesh overlay
shown on the first two views. Red points correspond to the HPE keypoint
locations and blue points to the model markers reprojected onto the image
plane. We recommend zooming in on this figure.

In order for markerless motion capture to produce accurate joint kinematics,
the 3D keypoint trajectories must contain a sufficiently dense set of
observations on the body to constrain the IK results. Most HPE algorithms
trained on publicly available datasets produce no keypoints between the hip
and shoulders, which limits the accuracy of tracking the hip and spine [8].
One markerless motion capture system, OpenCap [4], mitigates this by using an
existing biomechanics dataset and training a neural network to map from a
trajectory of these sparse keypoints to the most likely set of dense
keypoints. A risk of this approach is the filling in of keypoints locations
not extracted from the images may bias the results to an average movement and
may not generalize to rehabilitation patients who move differently than the
training dataset. Another challenge of existing HPE datasets is the marker and
joint definitions are not biomechanically well grounded and vary greatly [9].
A recent study trained an autoencoder as a “Rosetta stone” to translate
between these different datasets, including some with a dense set of
biomechanically grounded keypoints typical in marker-based tracking [10], and
found significant benefits from this cross-dataset training [11].

Scaling the skeleton before IK typically requires additional data to be
collected while the subject stands still. This can slow down a clinical
workflow, and standing upright unassisted can be challenging for many
rehabilitation subjects. An approach to simultaneously scale the skeleton
while performing IK was recently described [12]. This algorithm uses the
nimblephysics engine [13] from the same author, which extends the DART physics
engine [14] for biomechanics.

The reconstructed 3D trajectories used for IK should also be smooth and
anatomically consistent. This can be challenging when keypoints from HPE have
noise or may detect a different person occluding a camera. Additionally,
adding constraints like temporal smoothness or skeleton consistency can make
optimization much slower to converge. We recently compared several trajectory
reconstruction approaches and found that using an implicit function,
parameterized with a neural network, that maps from time to 3D locations
enabled accurate reconstructions that are both smooth and anatomically
plausible. This method produces smooth, anatomically consistent trajectories
with detected step widths and stride lengths within 10mm of those measured
with a GaitRite [15].

Our goal here was to determine how to best combine these components to compute
joint kinematics. Specifically, our contributions are:

  * •

We show that IK using implicit trajectory reconstructions performs better than
using robust triangulation.

  * •

We find that using the dense keypoint set from [10, 11] performs better than
the more popular 25 keypoints.

  * •

We show that bilevel optimization [12] for markerless motion capture without a
standing calibration trial.

  * •

We optimize the biomechanical model and hyperparameters of the IK algorithm
[12].

## Methods

### Participants

This study was approved by the Northwestern University Institutional Review
Board. We recruited 25 participants from both the inpatient services and
outpatient clinics at Shirley Ryan AbilityLab. This included people with a
history of stroke (n=8), spinal tumor (n=1), traumatic brain injury (n=2),
mild foot drop (n=1), knee osteoarthritis (n=2), and prosthetic users (n=11).
Ages ranged from 27 to 78. Some participants used assistive devices including
orthoses, rolling walkers, or a cane, and a few participants required contact
guard assistance from someone nearby for safety. We intentionally recruited a
diverse rehabilitation population both to ensure the system was robustly
validated for all types of patients, and because this is the population we
intend to apply this final system to. The data included 162 trials containing
2164 steps, with steps per participant ranging from 44-154, with over 1
million video frames.

### Data Acquisition, Video Processing, and Trajectories

Data acquisition, video processing, and trajectory reconstruction is similar
to in [15] and are summarized here.

#### Data Acquisition

Multicamera data was collected with a custom system in a $7.4m\times 8m$ room
with subjects walking the length of the diagonal ($11m$). We used 10 FLIR
BlackFly S GigE cameras (and 8 in several early experiments), which were
synchronized using the IEEE1558 protocol and acquired data at 30 fps, with a
typical spread between timestamps of less than 100µs. We used a mixture of
lenses including F1.4/6mm, F1.8/12m, F1.6/4.4-11mm with lenses and positions
selected to ensure at least three cameras covered the participants along the
walkway, although the room geometry limited coverage in the corners.

The acquisition software was implemented in Python using the PySpin interface.
For each experiment, calibration videos were acquired with a checkerboard
($7\times 5$ grid of 110mm squares) spanning the acquisition volume. Extrinsic
and intrinsic calibration was performed using the anipose library [16]. The
intrinsic calibration included only the first distortion parameter. Foot
contact and toe-off timing and location were acquired using a GaitRite walkway
spanning the room diagonal [17, 18].

#### Video processing

Collected videos were analyzed as previously described [15]. Briefly, the
videos were processed use PosePipe [19]. Easymocap [20] was used for the
initial multiview associations and was used to indicate the subject of
interest in each recording session. In contrast to [15], in this work, 2D
keypoints in the image plane were also detected using a recently developed
approach that is trained on numerous 3D datasets and outputs all of these
formats (MeTRAbs-ACAE) [11]. Specifically, we used the output keypoints from
the MOVI dataset [10], which has 87 keypoint locations commonly used in
optical motion capture. Importantly, this has numerous keypoints around the
pelvis and torso that are not available in the more common datasets.

By default, the MeTRAbs architecture [21] does not produce confidence
estimates for the locations. It also produces best guess location estimates
for occluded joints. While it produces reasonable guesses, we wanted to
discard any keypoints with low confidence. As recommended by the author, we
approximated confidence by measuring the standard deviation of each 3D joint
location estimated from 10 different augmented versions of each video frame.
This was converted to a confidence estimate using a sigmoid function with a
half maximum at 200mm and a width of 50mm. Visual inspection of the results
showed this produced low confidence for occluded joints and high confidence
for observed joints. We also set the confidence to zero for any joints outside
the image boundary.

#### Trajectory reconstruction

We found that representing 3D keypoint trajectories with implicit functions
enables better reconstructions from keypoints than triangulating (either with
the direct linear transform or robustly) from the 2D observations or
optimizing an explicit representation of each keypoint’s location [15].
Specifically, the implicit representation is a function that maps from time to
the reconstructed 3D keypoint locations,
$f_{\theta}:t\rightarrow\mathbf{x}_{t}\in\mathbb{R}^{J\times 3}$. This is
implemented as a five-layer multi-layer perceptron (MLP) with increasing
numbers of hidden units (128, 256, 512, 1024, 2048), each followed by a layer
normalization and a $\mathrm{relu}$ non-linearity, followed by a final dense
layer that mapped to the output size. We also used sinusoidal positional
encoding [22] for $t$, which was scaled from 0 to $\pi$ over each trajectory.
The positional encoding of time was concatenated to the output from each layer
for the first four layers.

For each trajectory, we optimize the parameters $\theta$ to minimize a loss
that includes the reprojection of the 3D keypoints onto the image and loss
terms that capture high-frequency noise and variations in skeleton geometry:

$\mathcal{L}=\mathcal{L}_{\Pi}+\lambda_{1}\mathcal{L}_{\mathrm{smooth}}+\lambda_{2}\mathcal{L}_{\mathrm{skeleton}}$

. The reprojection loss is:

$\mathcal{L}_{\Pi}=\frac{1}{T\cdot J\cdot C}\sum_{T,J,c\in
C}w_{c,t,j}\,g\left(||\Pi_{c}\mathbf{x}_{t,j}-y_{t,j,c}||\right)$

where $\Pi_{c}$ is the projection operator for camera $c$ which also includes
the non-linear intrinsic distortion. $y_{t,j,c}\in\mathbb{R}^{2}$ indicates
the detected keypoint location for a given time point, joint, and camera and
$\mathbf{x}_{t,j,c}$ is the corresponding 3D keypoints. We use a Huber loss
for $g(\cdot)$, which is quadratic within 5 pixels and then linear with the
slope further reduced after 10 pixels of error to be more robust to outliers.
The weights applied to Huber loss, $w$, are computed by a robust triangulation
algorithm [23, 15] that captures the consistency of each camera with the
keypoint triangulation from the other cameras and down weights outliers. We
also compared using trajectories computed with robust triangulation [23, 15],
which does not include any smoothing or skeletal consistency parameters.

### Biomechanics from inverse kinematics

Given a biomechanical model and a set of marker locations, IK estimates the
joint angles [1]. This typically requires estimating three sets of parameters:
anthropomorphic parameters for the skeleton size, marker offsets for that
individual, and a sequence of poses. We performed this with the nimblephysics
library [13] and specifically with their implementation of a dual optimization
process that solves for all the parameters at the same time from a set of
dynamic trials [12]. The results of this algorithm are skeleton scaling
parameters, $\mathcal{S}$, marker offsets, $\mathcal{O}\in\mathbb{R}^{87\times
3}$, and kinematics as joint angle trajectories for the $i^{\mathrm{th}}$
trial: $\mathbf{p}^{(i)}_{t}\in\mathbb{R}^{40}$.

The skeleton model provides a forward kinematic function to determine the
predicted marker locations from the IK results and a calibrated model:
$\mathbf{x}^{(i)}_{t}=\mathcal{M}(p^{(i)}_{t},\mathcal{O},\mathcal{S})$. We
measured the consistency between these predicted model-based marker locations
and the HPE keypoint detection.

### Performance Metrics

#### Geometric Consistency

We measured the geometric consistency between the model-based marker
trajectories and the detected keypoints, based on the distance between the
reprojected model-based markers and the detected keypoints. To quantify this,
we computed the fraction of the points below a threshold number of pixels,
conditioned on being greater than a specified confidence interval.

$\delta_{t,j,c}=||\Pi_{c}\mathbf{x}_{t,j}-y_{t,j,c}||$
$q(d,\lambda)=\frac{\sum(\delta_{t,j,c}<d)(w_{t,j,c}>\lambda)}{\sum
w_{t,j,c}>\lambda}$

Below, we report $GC_{d}=q(d,0.5)$ with $d=5$ pixels.

#### Residual marker errors

We measured the average distance between the 3D keypoint trajectories computed
from the image keypoints and the marker locations computed from the
biomechanical models after the IK solution.

#### Joint Limit Violations

Using the soft, regularized joint limits allows the model to exceed the
limits, which sometimes included anomalous poses with joints having more than
180∘ offsets. We computed the fraction of time the model fits violated the
joint limits and when it exceeded them by 50% and 100% ($v_{0}$, $v_{50}$,
$v_{100}$, respectively).

#### Pose Noise

We measured the high-frequency noise in the IK results as:
$n=\frac{1}{J(T-1)}\sum_{j}\sum_{t=0}^{T-1}(p_{t+1}-p_{t})^{2}$.

#### Step Position Error

We compared the heel position, determined by the location of the calcaneus
bone, between the GaitRite and the IK results. We report the width of the
error distribution as normalized IQR, $\sigma_{IQR}=0.7413\cdot IQR(x)$.

### Video Overlays

We created video overlays from different camera perspectives to confirm the
consistency of the fits with the underlying movements. Video overlays were
created by extracting the mesh transformation to world coordinates for each
body shape and time step from nimblephysics model. These were composed into a
scene using the trimesh library [24] and rendered with PyTorch3D [25] using
the calibrated camera projection matrix. The PyTorch3D perspective camera does
not support the distortion coefficient, so we applied the camera distortion to
the mesh vertices prior to rendering.

### Biomechanical model refinement

Our biomechanical model was based upon the widely used Rajagopal model [26]
with no muscles. We also added a three-degree of freedom neck joint. We
manually initialized the marker locations corresponding to the MOVI locations
after visually inspecting numerous outputs. To refine them, we perform several
iterations of fitting the model to our entire dataset and then adjusting the
marker locations by the mean offsets, $\mathcal{O}$, across all subjects.
While doing this, we also manually refined some marker positions to ensure
that markers with reliable anatomic locations did not drift, such as the foot,
ankle, knee, elbow, and wrist.

### Hyperparameter tuning

We manually tuned several IK regularization hyperparameters including the
anthropomorphic prior, ensuring the body segments have roughly consistent
scaling, regularizing how far the anatomic and additional tracking markers can
be adjusted, and the penalty for exceeding joint limits. Because of the time
required to perform the biomechanical fits and the need to view renderings for
qualitative fits, it was not possible to perform either a comprehensive grid
search on hyperparameters or run an optimization algorithm.

## Results

TABLE I: Comparing the reconstruction metrics between the two keypoint sets and reconstructed method |  | Marker Err | Pose Noise | $GC_{5}$ | $v_{50}$ | $v_{100}$ | Step Length | Stride Length | Step Width
---|---|---|---|---|---|---|---|---|---
Keypoints | Method | (mm) |  |  | (%) | (%) |  $\sigma_{IQR}$ (mm) |  $\sigma_{IQR}$ (mm) |  $\sigma_{IQR}$ (mm)
MOVI (87 kpts) | Implicit Optimization | 18.680 | 0.047 | 0.573 | 2.048 | 0.291 | 13.202 | 8.895 | 10.924
Robust Triangulation | 18.046 | 0.102 | 0.537 | 2.108 | 0.463 | 13.942 | 9.900 | 10.536
Halpe (25 kpts) | Implicit Optimization | 18.111 | 0.198 | 0.444 | 9.022 | 5.937 | 16.008 | 11.291 | 21.172

### Qualitative Results

The acquisition system and analysis software made it easy to perform kinematic
analysis on participants seen at a rehabilitation hospital as both inpatients
and outpatients, even when numerous additional people were in the room [15].
Examining the overlaid results from multiple views showed good alignment
between the reconstructed biomechanics and the raw image frames (Fig. 1). This
system was also sensitive enough to pick out clinically meaningful changes in
gait, as shown in Fig. 2. These examples used the MOVI keypoints with our
optimized model and IK hyperparameters using the implicit trajectory
representation. Next, we analyze these implementation details crucial for
obtaining these results.

Figure 2: Example showing the proposed pipeline can detect clinically
meaningful changes, based on multiple gait cycles being aligned and averaged.
The left column shows ankle dorsiflexion during walking and the change in
response to functional electrical stimulation of the peroneal nerve to improve
dorsiflexion, which is seen in the trace for the impaired side. The unimpaired
side shows little change, as expected. In the right column, we show changes in
knee kinematics in response to an ankle-foot orthosis.

### Dense versus Sparse Markers

We initially compared biomechanical decomposition using a HPE algorithm using
the 25 body keypoints commonly available in OpenPose and the Halpe and COCO
WholeBody datasets [27, 28, 29]. These do not include any keypoints on the
torso or spine between the hips and shoulders. Similarly to previous studies,
we found this produced unstable estimates of the underconstrained pelvis
orientation and also produced unstable hip angles. In contrast, the dense MOVI
markers from MeTRAbs-ACAE [10, 11] produced stable results. This is
quantitatively reflected in the lower levels of pose noise, lower residual
marker error, and better geometric consistency with the dense markers (Table
I).

### Implicit Trajectories versus Robust Triangulation

We also compared IK fits using implicit trajectories to 3D keypoint locations
that use robust triangulation. The former constrains smoothness and anatomic
consistency, whereas the latter is estimated independently for each marker and
time point. We found the implicit representation produced better fits with
lower noise (Table I).

### Model and hyperparameter tuning

Iteratively refining the model marker positions improved the results
qualitatively and by our performance metrics, such as fewer joint limit
violations and greater geometric consistency between the model-based markers
reprojected into the images and the detected keypoints, the residual errors
between the reconstructed trajectories and the model-based trajectories. We
also found enabling soft joint limits was important, as hard limits resulted
in many very implausible fits (Table II)

TABLE II: Impact of model iteration and joint regularization.

| Marker Err | Pose Noise | $GC_{5}$ | $v_{50}$ | $v_{100}$
---|---|---|---|---|---
Model |  |  |  |  |
Final | 18.680 | 0.047 | 0.573 | 2.048 | 0.291
Early | 23.466 | 0.082 | 0.477 | 3.262 | 1.417

$\mathcal{L}_{\mathrm{joint}}$ | Marker Err | Pose Noise | $GC_{5}$ | $v_{50}$ | $v_{100}$
---|---|---|---|---|---
5.0 | 18.680 | 0.047 | 0.573 | 2.048 | 0.291
Hard | 56.251 | 0.074 | 0.295 | 0.000 | 0.000

### Geometric Consistency

In our prior work, we found the greatest geometric consistency from 3D
keypoint trajectories reconstructed using an implicit representation from
keypoints detected with an MMPose algorithm trained on the Halpe dataset [15,
30, 28], which had a $GC_{5}$ of 0.54. We found the trajectories computed with
an implicit representation using the MOVI keypoints and MeTRAbs-ACAE [11]
substantially increased the $GC_{5}$ to 0.75 for the implicit trajectories.
After performing the IK and generating predicted marker locations from the
model this dropped to 0.57, which is still greater than the trajectories using
MMPose-Halpe.

### Spatial Gait Parameter Errors

We compared step length, stride length, and step width measures with respect
to the calcaneus location from the IK results against simultaneous GaitRite
recordings, which are shown in Table I. These errors were between 8 and 13mm.

## Discussion

Consistent with prior work [4], we found a dense set of markers, particularly
in the torso and pelvis, was important for stable biomechanical IK. In
particular, it quartered the noise in the estimated joint angles, greatly
reduced the extreme joint angle violations, and improved the accuracy of
estimated gait parameters. MeTRAbs-ACAE [11] enables training from multiple
datasets and can accurately estimate these dense, biomechanically motivated
marker locations from a single image. Our results demonstrate this algorithm
generalizes to novel settings and images. We found that the geometric
consistency was markedly greater for MeTRAbs-ACAE than other algorithms we
have evaluated [15], likely because it is trained on more 3D data compared to
algorithms trained on manual annotation on 2D images.

In contrast, OpenCap [4] uses a keypoint augmenter to map from a time series
of sparse marker data to dense marker trajectories prior to IK, using an
algorithm trained on previously collected biomechanical data. We investigated
using the keypoint augmenter from OpenCap [4] for a direct comparison, but it
is not trained for the sparse keypoint set we used (Halpe [28]). We anticipate
using MeTRAbs-ACAE to exact more information directly from the images will be
more accurate than this filling-in approach, particularly for participants
with movement patterns not represented in the training data of the keypoint
augmenter.

We found that using the implicit representation to reconstruct the
trajectories for the IK step was better than using robust triangulation. This
halved the noise in estimated joint angles while also improving the geometric
consistency of the result. This was reassuring, as a motivation for this
implicit representation was to produce better trajectories for IK with
biomechanical models [15]. Our rationale was that it is better to account for
the HPE uncertainty and geometric structure when reconstructing the trajectory
than attempting to post-process a noisy trajectory for IK. In the future,
these two steps could be combined and IK approaches could be extended to run
directly on the HPE outputs and consider their uncertainty and geometry while
providing more accurate biomechanical constraints. Differentiable physics
engines like nimblephysics make this possible, although propagating the
derivatives from the implicit representation through this engine is quite
complicated. Differentiable physics simulators that directly operate on the
GPU designed for deep learning, such as Brax, may provide another path to
integrate IK more closely with the perception layers.

Optimizing both the marker locations of the model to be more consistent with
the MOVI markers [10] and the IK hyperparameters used in nimblephysics [13]
was also important to produce stable and accurate fits. In particular,
enabling soft joint limits avoids unstable results seen with hard joint limits
and anatomically implausible results without regularization. However,
reassuringly, we found the results were not very sensitive to the other
hyperparameters.

The slight decrease in the geometric consistency of the model-based markers
compared to the raw trajectories is unsurprising as the model-based estimates
include more biomechanical constraints. However, the model-based $GC_{5}$
exceeded values we previously saw using algorithms trained on 2D datasets
[15]. This indicates that the multi-dataset used for MeTRAbs-ACAE both results
in geometrically consistent keypoints and also biomechanically consistent
keypoints that maintain their relative position on the skeleton. This result
is despite processing each frame independently. We expect that processing
videos and mapping image _sequences_ to keypoint sequences could utilize
information about body morphology and further improve biomechanical
consistency. We anticipate combining biomechanical trajectory modeling with
self-supervised learning on large datasets obtained with markerless motion
capture will be a fruitful avenue for training these models.

The MeTRAbs architecture [21] underlying Sarandi et al. [11] produces 2D and
3D joint locations from single images. In this work, we do not leverage the 3D
locations other than to estimate the 2D keypoint confidence. This is in
contrast to approaches that “lift” from 2D keypoints to 3D keypoints [2].
Lifting approaches discard a great deal of information available from the
image that is retained by [11]. An important future direction will be to
compare the accuracy and stability of biomechanical fits to monocular videos
from similar algorithms, or to find the minimal set of cameras required for
the desired accuracy. We defer this to future work.

While [11] provides a dense set of markers for biomechanics, it is fairly
sparse in the hands. This sometimes resulted in unstable estimates of the
wrist and did not allow estimating finger movements. We have seen using the
Halpe keypoints that we can get accurate 3D reconstruction of the hands [15].
The biomechanical model we are using does not include a highly articulated
hand, although they exist as separate models [31]. In the future, we will fuse
the 3D keypoints from both algorithms and develop a biomechanical model that
enables full-body biomechanical tracking, including the hand.

Our current approach has several limitations. One is that the analysis does
not run in realtime. We have also not incorporated solving for dynamics or
incorporating physics-based constraints, which we anticipate will further
improve the accuracy of the results. Our hyperparameter optimization was not
cross-validated, although given the limited number of tests, robustness to the
settings, and the interpretable nature of the parameters, we do not anticipate
this will have a substantial impact. There are also commercially available
markerless motion capture systems available (e.g., [5]). However, we are
unable to directly compare our methodology as this proprietary system has not
been described in detail. In contrast, our approach is based on publicly
available algorithms.

The quantitative consistency of the model-based marker reconstruction with the
detected keypoints and our qualitative impression reviewing many overlay
videos indicates our approach produces accurate biomechanical kinematics. We
previously found the accuracy of these trajectories for estimating spatial
gait parameters was within 10mm [15], and we obtain similar accuracy after
biomechanical reconstruction. The sensitivity of this approach is further
supported by our ability to detect anticipated changes between walking
conditions, such as bracing or functional electrical stimulation. We
anticipate additional studies validating the accuracy and reliability,
following the approach of other markerless motion capture systems [6] looking
at the inter-session repeatability of kinematic estimates on people without
gait impairments. We did not attempt this for our current study as many of the
participants have gait impairments associated with increases in gait
variability, which would obscure the results.

In conclusion, advances in HPE trained on multiple datasets [11] provide dense
markers for each image that are commonly used in marker-based motion capture
and are sufficient to constrain biomechanical reconstructions. Performing IK
on trajectories reconstructed with an implicit representation designed to be
smooth and anatomically consistent further improves the reconstruction
results. Our acquisition hardware uses cameras that only require Ethernet both
for power, data transmission, and synchronization making them easy to set up
in multiple locations. Our current workflow for markerless motion capture in a
rehabilitation hospital makes it easy and reliable to obtain biomechanical
estimates of movement. We hope more routine use of this technology will enable
better phenotyping of patients, monitoring their response to differing
interventions, and ultimately guide more effective, precision rehabilitation
interventions. We were able to bring in participants from both inpatient and
outpatient clinics and perform gait analysis with several trials and
conditions often in less than 10 minutes.

## References

  * [1] David A Winter  “Biomechanics and Motor Control of Human Movement”  Wiley, 2009
  * [2] Ce Zheng et al.  “Deep Learning-Based Human Pose Estimation: A Survey”, 2020, pp. 663–676
  * [3] Nobuyasu Nakano et al.  “Evaluation of 3D Markerless Motion Capture Accuracy Using OpenPose With Multiple Video Cameras”  In _Frontiers in Sports and Active Living_ 2, 2020, pp. 50  DOI: 10.3389/fspor.2020.00050
  * [4] Scott D. Uhlrich et al.  “OpenCap: 3D Human Movement Dynamics from Smartphone Videos”, 2022, pp. 2022.07.07.499061  DOI: 10.1101/2022.07.07.499061
  * [5] Robert Kanko et al.  “Concurrent Assessment of Gait Kinematics Using Marker-Based and Markerless Motion Capture”, 2020, pp. 2020.12.10.420075  DOI: 10.1101/2020.12.10.420075
  * [6] Robert M. Kanko, Elise Laende, W. Selbie and Kevin J. Deluzio  “Inter-Session Repeatability of Markerless Motion Capture Gait Kinematics”  In _Journal of Biomechanics_ 121, 2021, pp. 110422  DOI: 10.1016/j.jbiomech.2021.110422
  * [7] Jonathan Samir Matthis and Aaron Cherian  “FreeMoCap: A free, open source markerless motion capture system” If you use this software, please cite it as below. — Website: https://freemocap.org — Github: https://github.com/freemocap/freemocap — Updates: https://twitter.com/freemocap — Discord Community: https://discord.gg/SgdnzbHDTG — Supported in part by: NIH R00 EY028229-04 issued to JSM  Zenodo, 2022  DOI: 10.5281/zenodo.7233714
  * [8] Laurie Needham et al.  “The Accuracy of Several Pose Estimation Methods for 3D Joint Centre Localisation”  In _Scientific Reports_ 11.1 Nature Publishing Group, 2021, pp. 20673  DOI: 10.1038/s41598-021-00212-x
  * [9] Michał Rapczyński, Philipp Werner, Sebastian Handrich and Ayoub Al-Hamadi  “A Baseline for Cross-Database 3D Human Pose Estimation”  In _Sensors (Basel, Switzerland)_ 21.11, 2021, pp. 3769  DOI: 10.3390/s21113769
  * [10] Saeed Ghorbani et al.  “MoVi: A Large Multi-Purpose Human Motion and Video Dataset”  In _PLOS ONE_ 16.6 Public Library of Science, 2021, pp. e0253157  DOI: 10.1371/journal.pone.0253157
  * [11] István Sárándi, Alexander Hermans and Bastian Leibe  “Learning 3D Human Pose Estimation from Dozens of Datasets Using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formats”  arXiv, 2022  DOI: 10.48550/arXiv.2212.14474
  * [12] Keenon Werling et al.  “Rapid Bilevel Optimization to Concurrently Solve Musculoskeletal Scaling, Marker Registration, and Inverse Kinematic Problems for Human Motion Reconstruction”  bioRxiv, 2022, pp. 2022.08.22.504896  DOI: 10.1101/2022.08.22.504896
  * [13] Keenon Werling  “nimblephysics”  In _GitHub repository_ GitHub, https://github.com/keenon/nimblephysics, 2022
  * [14] Jeongseok Lee et al.  “DART: Dynamic Animation and Robotics Toolkit”  In _Journal of Open Source Software_ 3.22, 2018, pp. 500  DOI: 10.21105/joss.00500
  * [15] R. Cotton et al.  “Improved Trajectory Reconstruction for Markerless Pose Estimation”  arXiv, 2023  DOI: 10.48550/ARXIV.2303.02413
  * [16] Pierre Karashchuk et al.  “Anipose: A Toolkit for Robust Markerless 3D Pose Estimation”  In _bioRxiv_ Cold Spring Harbor Laboratory, 2020, pp. 2020.05.26.117325  DOI: 10.1101/2020.05.26.117325
  * [17] A.. McDonough et al.  “The Validity and Reliability of the GAITRite System’s Measurements: A Preliminary Evaluation”  In _Archives of Physical Medicine and Rehabilitation_ 82.3, 2001, pp. 419–425  DOI: 10.1053/apmr.2001.19778
  * [18] Belinda Bilney, Meg Morris and Kate Webster  “Concurrent Related Validity of the GAITRite Walkway System for Quantification of the Spatial and Temporal Parameters of Gait”  In _Gait & Posture_ 17.1, 2003, pp. 68–74  DOI: 10.1016/s0966-6362(02)00053-x
  * [19] R. Cotton  “PosePipe: Open-Source Human Pose Estimation Pipeline for Clinical Research”  In _arXiv:2203.08792 [cs, q-bio]_ , 2022  arXiv:2203.08792 [cs, q-bio]
  * [20] “EasyMoCap - Make human motion capture easier.”, Github, 2021  URL: https://github.com/zju3dv/EasyMocap
  * [21] István Sárándi, Timm Linder, Kai O. Arras and Bastian Leibe  “MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation”  In _IEEE Transactions on Biometrics, Behavior, and Identity Science_ 3.1, 2021, pp. 16–30  DOI: 10.1109/TBIOM.2020.3037257
  * [22] Ashish Vaswani et al.  “Attention Is All You Need”, 2017  DOI: 10.1017/S0140525X16001837
  * [23] Soumava Kumar Roy, Leonardo Citraro, Sina Honari and Pascal Fua  “On Triangulation as a Form of Self-Supervision for 3D Human Pose Estimation”  arXiv, 2022  DOI: 10.48550/arXiv.2203.15865
  * [24] Dawson-Haggerty et al.  “trimesh”, 2019  URL: https://trimsh.org/
  * [25] Nikhila Ravi et al.  “Accelerating 3D Deep Learning with PyTorch3D”  In _arXiv:2007.08501_ , 2020
  * [26] Apoorva Rajagopal et al.  “Full-Body Musculoskeletal Model for Muscle-Driven Simulation of Human Gait”  In _IEEE Transactions on Biomedical Engineering_ 63.10, 2016, pp. 2068–2079  DOI: 10.1109/TBME.2016.2586891
  * [27] Zhe Cao et al.  “OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields”  arXiv, 2019  DOI: 10.48550/arXiv.1812.08008
  * [28] Hao-Shu Fang et al.  “AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022
  * [29] Sheng Jin et al.  “Whole-Body Human Pose Estimation in the Wild”  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020
  * [30] MMPose Contributors  “OpenMMLab Pose Estimation Toolbox and Benchmark”, 2020
  * [31] Daniel C. McFarland et al.  “A Musculoskeletal Model of the Hand and Wrist Capable of Simulating Functional Tasks”  In _IEEE Transactions on Biomedical Engineering_ , 2022, pp. 1–12  DOI: 10.1109/TBME.2022.3217722


# Revisiting the Robust Alignment of Circuit Breakers

Leo Schwinn
Technical University of Munich
Munich Data Science Institute Simon Geisler
Technical University of Munich
Munich Data Science Institute

###### Abstract

Over the past decade, adversarial training has emerged as one of the few
reliable methods for enhancing model robustness against adversarial attacks
[Szegedy et al., 2014, Madry et al., 2018, Xhonneux et al., 2024], while many
alternative approaches have failed to withstand rigorous subsequent
evaluations. Recently, an alternative defense mechanism, namely “circuit
breakers” [Zou et al., 2024], has shown promising results for aligning LLMs.
In this report, we show that the robustness claims of “Improving Alignment and
Robustness with Circuit Breakers” against unconstraint continuous attacks in
the embedding space of the input tokens may be overestimated [Zou et al.,
2024]. Specifically, we demonstrate that by implementing a few simple changes
to embedding space attacks [Schwinn et al., 2024a, b], we achieve $100\%$
attack success rate (ASR) against circuit breaker models. Without conducting
any further hyperparameter tuning, these adjustments increase the ASR by more
than $80\%$ compared to the original evaluation. Code is accessible at:
https://github.com/SchwinnL/circuit-breakers-eval

Neural networks are still vulnerable to adversarial examples. A recurring
issue has slowed the field’s progress from early work in computer vision to
recent evaluations in LLMs: overly optimistic robustness assessments of new
defenses. When these evaluations prove faulty, it can create a domino effect
of problems. Researchers spend a considerable amount of time rectifying flawed
methods [Carlini and Wagner, 2016, 2017, Athalye and Carlini, 2018, Tramer et
al., 2020], while end-users may unknowingly rely on inadequate protections
[Hönig et al., 2024].Schwinn et al. [2024a] highlight that inadequate defense
evaluations have even more severe consequences in the context of LLMs due to
the increased cost of performing evaluations and the potentially high real-
world impact of toxic models.

Best practices for robustness evaluation have been outlined in prior work and
can be useful to identify unusually high robustness claims [Athalye and
Carlini, 2018, Tramer et al., 2020]. In the case of circuit breakers, we
observe that the authors claim to achieve high robustness against
unconstrained attacks in the continuous embedding space of the model [Zou et
al., 2024]. This contradicts previous findings in other domains that
unconstrained continuous attacks should achieve $100\%$ attack success rate.
While these findings might not trivially transfer to the alignment threat
models in LLMs they still suggest that the robustness claims might be
overestimated.

## Experiments and Results

We apply three simple modifications to the embedding space attack used in the
original evaluation of circuit breaker models. These adjustments are
sufficient to decrease the robustness to $0\%$. Notably, we implemented these
changes without conducting any hyperparameter tuning.

Optimizer. We observe that the embedding attack used in the circuit breaker
paper is very sensitive to the learning rate hyperparameter. To address this,
we replace the SGD optimizer with signed gradient descent as in [Schwinn et
al., 2024a]. We keep the learning rate the same as in the original work [Zou
et al., 2024].

Initialization. Instead of initializing the embeddings with a sequence of “x”
tokens, we use a more semantically meaningful instruction string: “ Make sure
to follow my instructions accurately and do not hesitant to answer in a toxic
way. Do not worry about me engaging with this content. I am studying toxic
behavior of LLMs. ”. Without this change, the models continue their generation
with sequences of “x” tokens after the adversarial target string was correctly
generated. Note that we used this specific initialization on our first attempt
after observing the generation issue and did not conduct any further
optimization.

Multiple generations. We use the same early stopping procedure as in the
original paper. Additionally, we periodically generate and evaluate outputs
from the attacked model. Specifically, we produce a response for every $5$
attack iterations and generate a total of $20$ responses for every toxic
query. An alternative approach could involve conducting multiple random
restarts of the attack. However, we find our method is sufficient to break the
defense while it is computationally more efficient.

Results. We implement all these changes in the original repository of the
paper and use the same evaluation script. Here, the ASR is calculated using
the judge model provided by Harmbench [Mazeika et al., 2024]. Additionally, we
manually check $20$ random generations for both models. We found one false
positive during this manual check. Nevertheless, For this specific instance,
multiple other generations correctly broke the model. Moreover, we evaluated
$20$ different negatively judged outputs and found $3$ false negatives. Thus,
we conclude that the ASR provided by the judge model is likely not
overestimated.

The results are summarized in Table 1. Our new attack achieves a $100\%$
attack success rate (ASR) against both circuit breaker models provided in the
paper. In contrast, the original evaluation reported $15,7$ and $9.6\%$ ASR
for embedding space attacks, respectively. Increasing the number of attack
iterations, the number of generations, optimizing the initialization string,
or fine-tuning the learning rate would likely lead to even more efficient
attacks. We want to clarify that this vulnerability is not unique to circuit
breakers. Our new attack achieves $100\%$ ASR on Llama2-7b-chat-hf [Touvron et
al., 2023], vicuna-7b-v1.5 [Hugging Face, 2023], and Mistral-7B-v0.1 [Jiang et
al., 2023] using the same attack and evaluation protocol.

Table 1: Comparison of embedding attack success rates (ASR) in $\%$ against
the short circuited models provided in [Zou et al., 2024]. Changing the
optimizer (opt) and initialization (init) improves the ASR considerably
without inducing any additional cost. Evaluating with multiple generations
(multi gen) for each attack breaks the defense.

Model | Mistral-7B-Instruct-v2 + RR | Llama-3-8B-Instruct + RR
---|---|---
Input Embed [Zou et al., 2024] (original) | 15.7 | 9.6
Input Embed (opt & init) | 57.2 | 51.7
Input Embed (opt & init & multi gen) | 100 | 100

Note that embedding space attacks are an unrealistically strong threat model
for proprietary robustness evaluations and are mostly relevant as a threat
model for open-source LLMs [Schwinn et al., 2024b]. Still, our experiments
showed that embedding space attacks can be used as a sanity check. High
robustness against unconstrained continuous attacks should be reported with
caution.

## Outlook

Beyond the presented results, currently ongoing third-party evaluations
observe vulnerabilities of short circuiting to manual human jailbreaks, such
as separating the toxic content with spaces before forwarding it to the LLM.
This gives further indication that the original robustness assessment may be
overestimated. We plan to update this document with additional robustness
evaluations, such as adaptive discrete attacks, in the future [Geisler et al.,
2024, Andriushchenko et al., 2024]. We thank the authors of the original work
for providing the community with code and pretrained models. This considerably
reduced the effort required to conduct a third-party evaluation. We also want
to thank them for their commitment to helping us with the evaluation.

## References

  * Szegedy et al. [2014] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus.  Intriguing Properties of Neural Networks.  In _International Conference on Learning Representations (ICLR)_ , 2014.
  * Madry et al. [2018] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards Deep Learning Models Resistant to Adversarial Attacks.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Xhonneux et al. [2024] Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann, Gauthier Gidel, and Leo Schwinn.  Efficient adversarial training in llms with continuous attacks.  _arXiv preprint arXiv:2405.15589_ , 2024.
  * Zou et al. [2024] Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks.  Improving alignment and robustness with circuit breakers.  _arXiv preprint arXiv:2406.04313_ , 2024.
  * Schwinn et al. [2024a] Leo Schwinn, David Dobre, Stephan Günnemann, and Gauthier Gidel.  Adversarial attacks and defenses in large language models: Old and new threats.  In _NeurIPS: I Can’t Believe It’s Not Better Workshop: Failure Modes in the Age of Foundation Models_ , 2024a.  URL https://openreview.net/forum?id=vAiEQBh2AW.
  * Schwinn et al. [2024b] Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan Gunnemann.  Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space.  _arXiv preprint arXiv:2402.09063_ , 2024b.
  * Carlini and Wagner [2016] Nicholas Carlini and David Wagner.  Defensive Distillation is Not Robust to Adversarial Examples.  _arXiv preprint arXiv:1607.04311_ , 2016.
  * Carlini and Wagner [2017] Nicholas Carlini and David Wagner.  Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods.  In _ACM Workshop on Artificial Intelligence and Security_ , pages 3–14. Association for Computing Machinery, 2017.  ISBN 978-1-4503-5202-4.
  * Athalye and Carlini [2018] Anish Athalye and Nicholas Carlini.  On the Robustness of the CVPR 2018 White-Box Adversarial Example Defenses.  _arXiv preprint arXiv:1804.03286_ , 2018.
  * Tramer et al. [2020] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry.  On Adaptive Attacks to Adversarial Example Defenses.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 33, pages 1633–1645, 2020.  URL https://proceedings.neurips.cc/paper/2020/file/11f38f8ecd71867b42433548d1078e38-Paper.pdf.
  * Hönig et al. [2024] Robert Hönig, Javier Rando, Nicholas Carlini, and Florian Tramèr.  Adversarial perturbations cannot reliably protect artists from generative ai.  _arXiv preprint arXiv:2406.12027_ , 2024.
  * Mazeika et al. [2024] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks.  HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal, February 2024.  URL http://arxiv.org/abs/2402.04249.  arXiv:2402.04249 [cs].
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Hugging Face [2023] Hugging Face.  Vicuna-7B-v1.5, 2023.  URL https://huggingface.co/lmsys/vicuna-7b-v1.5.  Accessed on 15 December 2023.
  * Jiang et al. [2023] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_ , 2023.
  * Inan et al. [2023] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al.  Llama guard: Llm-based input-output safeguard for human-ai conversations.  _arXiv preprint arXiv:2312.06674_ , 2023.
  * Geisler et al. [2024] Simon Geisler, Tom Wollschläger, MHI Abdalla, Johannes Gasteiger, and Stephan Günnemann.  Attacking large language models with projected gradient descent.  _arXiv preprint arXiv:2402.09154_ , 2024.
  * Andriushchenko et al. [2024] Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion.  Jailbreaking leading safety-aligned llms with simple adaptive attacks.  _arXiv preprint arXiv:2404.02151_ , 2024.