# HandyPriors: Physically Consistent Perception
of Hand-Object Interactions with Differentiable Priors

Shutong Zhang1,∗, Yi-Ling Qiao2,∗, Guanglei Zhu1,∗, Eric Heiden3, Dylan
Turpin1,3, Jingzhou Liu1,
Ming Lin2, Miles Macklin3, Animesh Garg1,3 1University of Toronto & Vector
Institute, 2University of Maryland College Park, 3Nvidia, ∗ Equal
contributionMore experiments and supplementary materials can be found at:
https://handypriors.github.io/

###### Abstract

Various heuristic objectives for modeling hand-object interaction have been
proposed in past work. However, due to the lack of a cohesive framework, these
objectives often possess a narrow scope of applicability and are limited by
their efficiency or accuracy. In this paper, we propose HandyPriors, a unified
and general pipeline for pose estimation in human-object interaction scenes by
leveraging recent advances in differentiable physics and rendering. Our
approach employs rendering priors to align with input images and segmentation
masks along with physics priors to mitigate penetration and relative-sliding
across frames. Furthermore, we present two alternatives for hand and object
pose estimation. The optimization-based pose estimation achieves higher
accuracy, while the filtering-based tracking, which utilizes the
differentiable priors as dynamics and observation models, executes faster. We
demonstrate that HandyPriors attains comparable or superior results in the
pose estimation task, and that the differentiable physics module can predict
contact information for pose refinement. We also show that our approach
generalizes to perception tasks, including robotic hand manipulation and
human-object pose estimation in the wild.

##  I Introduction

Hands serve as the primary means by which humans interact with the physical
world as they are responsible for most of the intricate and dexterous object
manipulation tasks. A good understanding of hand-object interaction helps with
numerous learning-based tasks such as action recognition [1], robotic
manipulation [2], user interaction [3], etc.

Estimating hand and object interaction presents several challenges. Firstly,
the complex, deformable geometry and dynamics of hands present challenges in
learning and optimization. Unlike rigid objects, hands are articulated with
joints and possess high degrees of freedom. Their non-linearity and non-
convexity further exacerbate the complexity of the task. Secondly, the
presence of both self-occlusion and hand-object occlusion result in
uncertainty when predicting hand poses during manipulation. Finally, the rich
contacts between hands and objects in interaction scenes necessitate the need
for a consistent estimation method for both the hand and the object to avoid
penetration.

There are generally two categories of approaches for estimating hand and
object poses. The first approach is purely learning-based, where the network
architectures, loss functions, and training strategies are carefully designed
to train a generalizable mapping from images to hand and object configurations
[4, 5]. Although neural networks and large training datasets can mitigate the
difficulties posed by high degrees of freedom and non-convexity, they cannot
guarantee interpretability nor correctness. The second category of approaches
is optimization-based [6, 7], which involves defining and optimizing heuristic
target functions to improve physical feasibility and reduce penetrations.
Although the optimization process may be slower, per-frame iterations can
improve the accuracy of the results.

Figure 1: HandyPriors is a modular block application in multiple tasks. The
estimation of hand-object interaction can be achieved by (a) optimization-
based refinement or (e) filtering with Extended Kalman Filter (EKF). Moreover,
the differentiable contact module can be used to perform (b) pose refinement
given contact status. Our differentiable priors can also be used in pose
estimation for scenes with (c) robot hands or (d) human bodies.

In this paper, we leverage both learning and optimization techniques. Our
method utilizes pre-trained networks to provide an initial estimate of poses,
and then uses differentiable self-supervised terms to further refine the
estimation by correcting errors from pixel-level misalignment and non-physical
penetration/oscillation. Inspired by the success of self-supervised learning
in large-scale vision and language models [8, 9], we adopt similar techniques
to estimate hand-object interactions. Specifically, we propose supervising the
pose prediction by comparing it with subsequent video frames, similar to the
way in which text generation can be trained using sequence completion.

To enable backpropagation of gradients from image space, we develop fully-
differentiable operators that simulate and render predicted poses. This
closed-loop optimization approach improves the accuracy by making the
estimation resemble more of the input image. We also incorporate several
regularization terms to minimize motion oscillation and collision in predicted
sequences.

While the optimization process improves the accuracy, it can be time-
consuming. To address this issue, we present an alternative filtering
technique based on the Kalman filter [10] to further utilize our
differentiable dynamics and observation modules. This approach allows the user
to make a trade-off between efficiency and accuracy depending on whether they
choose to perform online tracking or offline optimization. Employing this
integrated differentiable rendering and simulation pipeline, we also apply our
pose estimation method to human and synthetic robot hand scenes, demonstrating
the generalizability and robustness of our method. Figure 1 summarizes several
applications of this approach.

In summary, the key contributions of this work are:

  * •

An integrated differentiable rendering (Sec. III-B) and simulation (Sec.
III-C) pipeline to estimate hand-object interaction.

  * •

An offline optimization (Sec. III-D) and an online tracking (Sec. III-E)
process that can predict poses and contact information, providing the user
with options to balance between accuracy and efficiency.

  * •

Experiments that generalize our priors to applications, e.g. robotic
manipulation (Sec. IV-A), contact refinement (Sec. IV-D), and human bodies
(Sec. IV-E).

In our experiments, our method achieves comparable or better pose estimation
results in hand-object-interaction datasets [11, 12]. Additionally, we show
that the contact information predicted by our method effectively refines pose
estimation. Our ablation study compares the differences between the
optimization and tracking results, further validating the contributions of
different losses.

##  II Related Works

Figure 2: An overview of our optimization and filtering pipelines. We provide
two alternatives for utilizing the differentiable priors. Given the image and
the estimation from previous frames, (a) the optimization pipeline first
initializes the poses with pre-trained networks and then minimizes the
rendering and physics losses from the differentiable operators; (b) the
filtering pipeline can take some simple observations and use Extended Kalman
Filter (EKF) to update the state estimation. EKF requires differentiable
physics and rendering to model the system and runs much faster than the
optimization pipeline.

Hand and object pose estimation from monocular images [13, 14, 15, 16] is a
fundamental problem in computer vision. Many recent works predict 6 DoF object
poses assuming known rigid mesh models [17, 18, 19]. For human hands, MANO
[20] proposes a parametric model to represent the hand geometry using
different shape and pose parameters. Numerous learning-based approaches focus
on predicting 3D joint locations from RGB(D) inputs [21, 22, 23, 24, 25, 26,
27]. Others [28, 29, 7, 30] then regress to the MANO pose and shape
parameters. There are also approaches directly learning a mapping between
visual inputs and hand poses from annotated datasets [31, 32, 33, 34]. These
methods achieved remarkable success on hand-only datasets but had poor
performance when hands were holding objects. There are also optimization-based
approaches that optimize hand parameters based on 2D key points [6, 7, 35] or
segmentations [11]. They tend to have better accuracy but are much slower due
to the iterative optimization process. We propose a tracking-based pipeline
with higher efficiency. Our differentiable priors are more general and can be
applied to a wider range of applications.

Hand object interaction has been garnering progressively more attention.
Treating hand and object pose estimation separately results in inaccurate
estimation for both. With an increasing number of hand-object datasets being
created [36, 37, 38, 11, 4, 39], many recent works have started to focus on
jointly tracking hands and objects. However, simultaneously estimating both is
challenging due to occlusions and depth ambiguity. [5] designed a joint
learning framework based on contextual reasoning between hand and object
representations. [40] proposed an ordinal relation loss to correct depth
misalignment between hand and object. [41] incorporated a feature injection
mechanism that better handles occlusion by integrating hand information into
the obscured areas. These methods improve the accuracy of pose estimation but
pay little attention to the physical interaction between hand and objects.
Other works aim to predict physically plausible hand-object poses. To achieve
this goal, interaction constraints are applied to encourage contact and
minimize interpenetration. [7, 6] proposed to use signed distance functions
(SDF) to model contact between hand and object, [42] design a deep network to
estimate contact areas and a virtual capsule technique to simulate soft hand
tissue deformation. [43] apply motion and force vectors to reconstruct
interactions. Compared to these methods, our approach is more general and can
be used in contact-based refinement and pose estimation in both human-hand and
non-human-hand scenarios.

Differentiable Priors. Differentiable rendering [44, 45, 46] and physics [47,
48, 49] have been used to solve inverse problems for their efficiency in
optimization. For hand grasping, [50] leverages a physics simulator to
evaluate dynamic interactions between the hand and the object. [51], [52] uses
differentiable physics to generate physically realistic and stable grasps.
There are also recent works that use differentiable rendering to optimize
human [53] or object [54] poses. However, they do not consider their physical
interactions. [55] utilize a differentiable articulated body simulator to
learn and smoothen human-pose estimation. In contrast, our method leverages
rendering priors for each frame, thereby significantly accelerating pose
estimation in a video.

##  III Methods: HandyPriors

###  III-A Problem Setting

Given a sequence of RGB hand-object-interaction images and the object’s 3D
model, we propose two pipelines to estimate the configuration of the hand and
object in each frame. Figure 2 shows an overview of our method.

In particular, we represent the hand using the MANO [20] parameterization,
which describes a hand by its pose $\theta\in\mathbb{R}^{45+6}$ (representing
the rotations of 15 hand joints and a 6 DoF global transformation) and shape
$\beta\in\mathbb{R}^{10}$ (a statistical model) parameters. The 778 vertices
$\mathbf{V}_{hand}\in\mathbb{R}^{778\times 3}$ of the hand mesh can be
computed from $\theta,\beta$ using a learned linear model
$\mathbf{V}_{hand}=m(\theta,\beta)$.

The object pose has 6 degrees of freedom, which contains its translation
$\mathbf{t}\in R^{3}$ and rotation (represented by XYZ Euler angles)
$\mathbf{r}\in R^{3}$.

###  III-B Differentiable Rendering Priors

The first prior of our approach is differentiable rendering, which helps align
the estimated 3D model with the 2D image. Instance segmentation is the key
information we rely on to optimize hand and object poses. We use a
segmentation detector [56] trained on the COCO dataset [57] to extract hand
and object masks. Our experiment shows that although the image segmentation
detector, PointRend [56], was not trained on the YCB objects [58], it can
correctly predict object masks. However, it fails to adequately segment hands
as fingertips are harder to detect. Thus, we further trained PointRend to
obtain better segmentations for the hand and hand-held objects.

For each frame, we utilize a differentiable renderer SoftRas [44] that
rasterizes and projects 3D hand and object models to 2D, then we optimize hand
parameters $\theta$ and object parameters $\mathbf{r},\mathbf{t}$ by fitting
the projected vertices onto the hand and object masks. Next, we describe loss
terms related to the rendering part.

RGB image loss. Since hand texture varies greatly from person to person, we
only render the RGB images for objects. We then apply L2 loss between the
rendered image $I_{r}$ and the input RGB image $I_{in}$ cropped by the
detected object mask $M_{obj}$. The image loss term can be expressed as
follow:

$L_{img}(\mathbf{r},\mathbf{t})=\lVert(1-M_{hand})I_{r}-M_{obj}I_{in}\rVert_{2}^{2}$
(1)

Mask loss. Compared to the RGB image loss, masks have fewer features but are
not sensitive to lighting and texture, hence the loss based on masks is more
robust and general. We use a differentiable renderer to render the silhouette
of hands and objects together. We then apply L2 loss between the rendered mask
$M_{r}$ and the detected mask $M_{d}$,

$L_{mask}(\theta,\mathbf{r},\mathbf{t})=\lVert M_{r}-M_{d}\rVert_{2}^{2}$ (2)

###  III-C Differentiable Physics Priors

Besides the rendering priors defined on the 2D image spaces, we also add more
physics-related priors in the 3D and temporal domain to help us regularize the
hand-object understanding.

Contact loss. During manipulation, we observe that most of the contacting
vertices between hand and objects do not have sudden relative sliding. Based
on this observation, we first utilize a GPU-based differentiable contact
module to infer the signed distance function $sdf(\mathbf{V}_{hand})$ from
hand vertices to the object. Then, we find the contact points $\mathbf{v}_{t}$
where the SDF values are smaller than 0.1. The relative sliding between $t$
and $t-1$ is,

$L_{sliding}(\theta,\mathbf{r},\mathbf{t})=\lVert(\mathbf{v}_{t}-\mathbf{r}_{t})(\mathbf{r}_{t}^{-1})^{\top}-(\mathbf{v}_{t-1}-\mathbf{t}_{t-1})(\mathbf{r}_{t-1}^{-1})^{\top}\rVert$

Moreover, given the SDF results, we reduce penetration by penalizing the
negative SDF values.

$L_{penetration}(\theta,\mathbf{r},\mathbf{t})=\min(0,-sdf(\mathbf{V}_{hand}))$
(3)

Smoothness loss. To enhance the continuity of hand and object across time, we
penalize sudden changes of object and hand in two consecutive frames. For
objects, we apply L2 loss between object vertices in the current frame
$\mathbf{V}_{t}$ and previous frame $\mathbf{V}_{t-1}$. For hands, we compute
the L2 loss between hand poses in two frames $\theta_{t},\theta_{t-1}$,

$L_{con}=\lVert\mathbf{V}_{t}-\mathbf{V}_{t-1}\rVert_{2}^{2}+\lVert\theta_{t}-\theta_{t-1}\rVert_{2}^{2}.$
(4)

During the experiment, we also observed that objects usually have jittering
for several frames. Thus, we add a smoothness term to punish the oscillation
in object velocity,

$L_{smo}=\lVert(\mathbf{V}_{t}-\mathbf{V}_{t-1})-(\mathbf{V}_{t-1}-\mathbf{V}_{t-2})\rVert_{2}^{2}.$
(5)

###  III-D Optimization-based Refinement

In summary, the losses of rendering priors are

$L_{render}=\lambda_{1}L_{image}+\lambda_{2}L_{mask},$ (6)

and the physics-inspired losses are

$L_{physics}=\lambda_{3}L_{sliding}+\lambda_{4}L_{penetration}+\lambda_{5}L_{con}+\lambda_{6}L_{smo},$

where $\lambda$’s are the weights for the loss terms. With these
differentiable loss terms, we optimize the pose of the hand $\theta$ and
object $\mathbf{r},\mathbf{t}$ using a gradient-based optimization method such
as Adam [59].

Figure 3: Diagram of the EKF filtering process. With (1) the differentiable
physics $f$ and (2) rendering $h$ modules to compute the Jacobians, (3) the
extended Kalman Filter (EKF) can perform state estimation given the control
signal, observation, and previously estimated states.  TABLE I: Ablation
study. We run ablation studies for both the optimization (left column) and
filtering pipeline (right column). For optimization, 2D and 3D errors ($cm$)
decrease with gradually added rendering $\mathcal{L}_{render}$ and physics
loss $\mathcal{L}_{physics}$. Filtering is much faster than the optimization
pipeline but also scarifies accuracy.

Optimization | 2D Error | 3D Error | Runtime (s) | Filtering | 2D Error | 3D Error | Runtime (s)
---|---|---|---|---|---|---|---
Initialization | 1.03 | 52.5 | 0.072 | $\mathcal{O}_{hand}$ | 44.59 | 5.57 | 0.024
+$\mathcal{L}_{render}$ | 0.77 | 1.54 | 2.29 | $\mathcal{O}_{object}$ | 19.00 | 11.24 | 0.027
+$\mathcal{L}_{physics}$ | 0.43 | 1.10 | 5.83 | $\mathcal{O}_{hand+object}$ | 8.84 | 7.41 | 0.029

###  III-E Filtering-based Tracking

Besides the optimization-based process, we can also use the differentiable
priors in a more light-weight pipeline (Fig. 3). Multiple filtering algorithms
can be integrated with this method for fast tracking; in this work, we employ
the Extended Kalman Filter (EKF) [10] as an example. EKF can give a state
estimation in a dynamic system

$\mathbf{x}_{t}=f(\mathbf{x}_{t-1},\mathbf{u}_{t})+\epsilon_{s},\mathbf{z}_{t}=h(\mathbf{x}_{t})+\epsilon_{o},$
(7)

where $\mathbf{x}$ is the state, $\mathbf{u}$ is the control, $\mathbf{z}$ is
the observation, $\epsilon_{s}$ and $\epsilon_{o}$ are noises,
$f(\cdot),h(\cdot)$ are differentiable functions for states and observation,
respectively. Using the EKF, we can compute an estimation
$\hat{\mathbf{x}}_{t}$ for the hidden state $\mathbf{x}_{t}$ given the
observation $\mathbf{z}_{t}$ and control $\mathbf{u}_{t}$.

The states of the hand-object system are
$\mathbf{x}_{t}=(\theta_{t},\mathbf{r}_{t},\mathbf{t}_{t})$. Using those pose
parameters (along with the known object model and shape parameters $\beta$),
we can reconstruct the entire scene. For most of the cases in real life and
nearly all of the cases in existing datasets, hands can be modelled as
actuators whereas objects are primarily passive. Therefore, the control input
for the system is the hand pose $\mathbf{u}_{t}=\theta$.

Since the differentiable renderer can output various quantities (2D keypoints,
bounding box, full image set), the observations we can choose are also
flexible. For instance, the 2D positions of the fingertips can serve as the
only feedback supervision. Then the observation function is,

$h(\mathbf{x}_{t})=S_{tips}\cdot\mathbf{V}_{hand}(\theta_{t}),$ (8)

where $S_{tips}\in\mathbb{R}^{5\times 778}$ is a one-hot encoding matrix that
selects fingertips out of the hand.

The more challenging part is to construct the dynamics function $f(\cdot)$. A
direct solution is to use a differentiable full dynamics model to describe the
system. However, the actuation signals of the hand (like muscle contraction)
are unavailable from the MANO parameter. We thus choose to use simpler and
more robust quasistatic dynamics. If the hand makes contact with an object on
vertices $\mathbf{v}_{t-1}=S_{contact}\cdot\mathbf{V}_{hand}(\theta_{t-1})$ at
time step $t-1$, we assume the relative sliding on the vertices from
$\mathbf{v}_{t-1}$ to $\mathbf{v}_{t}\in\mathbb{R}^{n_{contact}\times 3}$ is
small. The rotation $\mathbf{R}$ and translation $\mathbf{T}$ are estimated
as,

$\displaystyle\mathbf{U}\Sigma\mathbf{V}^{\dagger}$
$\displaystyle=svd(\tilde{\mathbf{v}}_{t-1}^{\top}\tilde{\mathbf{v}}_{t})$ (9)
$\displaystyle\mathbf{R}_{t}$
$\displaystyle=(\mathbf{U}\mathbf{V}^{\dagger})^{\top}\mathbf{R}_{t-1}$
$\displaystyle\mathbf{T}_{t}$
$\displaystyle=\bar{\mathbf{v}}_{t}-\bar{\mathbf{v}}_{t-1}(\mathbf{U}\mathbf{V}^{\dagger})+\mathbf{T}_{t-1}$

where $\tilde{\mathbf{v}}=\mathbf{v}-\bar{\mathbf{v}}$ is the contact vertices
subtracting their centroids $\bar{\mathbf{v}}$, $svd(\cdot)$ is the singular
value decomposition.

With the differentiable observation $h(\cdot)$ (Eq. 8) and dynamics $f(\cdot)$
(Eq. 9), we can then use the EKF to track and estimate the states
$\mathbf{x}_{t}=(\theta_{t},\mathbf{r}_{t},\mathbf{t}_{t})$.

##  IV Experiments

In this section, we apply our differentiable priors to five scenarios: (a)
object pose estimation for robotic manipulation, (b) pose estimation, (c)
filtering-based tracking, (d) contact refinement, and (e) human-object pose
estimation. Our method can achieve comparable or better results in a wide
range of tasks.

|  |
---|---|---
(a) Input | (b) Semi [5] | (c) HandyPriors
Figure 4: Real-world pose estimation. (a) We record a robotic hand
manipulating a cracker box in the real world using a calibrated camera. With
(b) initialization using [5]. We observe that (c) our method can recover the
transformation of the box.

###  IV-A Real-world Robotic Hand and Object Iteration

Our method can generalize to a wide range of scenarios, including robotic
hands and object interaction. To demonstrate this, we designed a robotic hand
manipulation task. We first record sequences of RGB images where an Allegro
robotic hand interacts with a YCB object [58] using the calibrated Intel
RealSense camera and record the corresponding hand pose parameters. We then
predict the initial object pose using only the RGB image with [5]. Given the
robotic hand configuration calculated using the hand pose parameter and
segmentation masks predicted by Pointrend [56]. We then use our pipeline to
optimize the object pose $\mathbf{r},\mathbf{t}$. Figure 4 shows the
optimization result of our method. The results demonstrate that our method is
capable of generating accurate object poses even with robotic hand interaction
scenes.

###  IV-B Pose Estimation

HandyPriors
---
Semi [5]
Figure 5: Pose Estimation Results. We compare the hand and object pose
estimation results with other methods and visualize the 3D meshes projected on
the original image. Our method yields reasonable joint estimation for the
interaction scenes. TABLE II: Evaluation of hand pose estimation. We measure
the joint error ($cm$), mesh error ($cm$), and joint AUC on the DexYCB dataset
[38]. PA is Procrustes Alignment.

Methods | Joint (PA) | Joint | Mesh (PA) | Mesh | Joints AUC
---|---|---|---|---|---
Artiboost [40] | 0.65 | 1.12 | 0.69 | 1.21 | 0.870
Homan [7] | 1.84 | 4.41 | 1.80 | 4.34 | 0.644
HandOccNet [41] | 0.62 | 1.16 | 0.66 | 1.20 | 0.877
HandyPriors (ours) | 0.52 | 0.81 | 0.51 | 0.89 | 0.896

####  IV-B1 Datasets and Metrics

Datasets. HO3D dataset [11] is a real hand-object interaction dataset that
contains approximately 80K images from more than 65 sequences. It uses 10
models from the YCB dataset [58]. The evaluation of the HO3D dataset is done
via an online server. We report the performance of all of the methods on the
evaluation split of the dataset. Dex-YCB dataset [38] consists of 582K images
over 1,000 sequences of 10 subjects grasping 20 different YCB objects [58]
from 8 camera views. It is currently the largest real hand-object interaction
dataset. In this work, we report the performance of our method on the $S_{0}$
test split of the dataset. Following [40], we filter out frames that contain
the left hand or where the hand is not in contact with the object. Metrics. In
Table III, we measure the 2D error by projecting the object vertices onto the
image plane and computing the Chamfer distance between the ground truth
vertices and predicted vertices. The 3D error is the Chamfer distance in the
original 3D space. We also report the average interpenetration volume between
the hand and the object to measure how well the methods handle interactions.
This metric is denoted as Collision. In Table II, we compute the joint error,
mesh error, and joint AUC (Area Under the ROC Curve). For joint and mesh
error, we report results before and after Procrustes alignment, where
Procrustes alignment changes the global rotation, translation, and scale of
the estimated pose to match the ground truth.

TABLE III: Quantitative evaluation of object pose estimation. Our approach is
compared with Semi [5], Artiboost [40], Homan [7] on the HO3D [11] and DexYCB
Dataset [38]. We report the 2D ($cm$) and 3D ($cm$) errors.

Method | 2D Error (HO3D) | 3D Error (HO3D) | 2D Error (DexYCB) | 3D Error (DexYCB)
---|---|---|---|---
Artiboost [40] | 6.41 | 10.83 | 3.01 | 6.78
Homan [7] | 4.60 | 41.33 | 15.21 | 95.07
Semi [5] | 2.06 | 26.35 | 9.20 | 47.77
HandyPriors (ours) | 1.67 | 3.22 | 0.96 | 3.79

####  IV-B2 Optimization-based Refinement

We compare our optimization-based method described in Section III-D with Semi
[5], Artiboost [40], Homan [7] for the pose estimation task. In the HO3D
dataset, we test three objects from the evaluation split: Mustard Bottle
(SM11), Meat Can (MPM10-15), and Bleach Cleanser (SB11, 13). Table III shows
that our method outperforms alternative methods for estimating the pose of the
objects. Our method also achieves comparable results in terms of collision
metrics. Table II shows the hand prediction results compared to Artiboost
[40], Homan [7], and HandOccNet [41]. Our method also improves over the
initialization from [41] and achieves the best accuracy in all metrics. Figure
5 shows some qualitative results of the predictions. Compared to other
methods, our optimized results are better aligned with the image and have more
reasonable relative positions between hands and objects.

####  IV-B3 Ablation Study

In this part, we study the influence of different loss terms on the estimation
results. All the experiments are run in the evaluation set SM1 from the HO3D
dataset [11]. Table I presents the quantitative results. Starting from the
initialization by the pre-trained network [5], adding rendering loss
$\mathcal{L}_{rendering}$ drastically improves the performance, especially
with regards to the 3D error as it can adjust the translation errors in the
depth direction. Moreover, adding the physics loss term
$\mathcal{L}_{physics}$ improves the results further by reducing penetration
and oscillation in the predicted sequences.

|  |  |
---|---|---|---
(a) Semi [5] | (b) Filtering | (c) Semi [5] | (d) Filtering
Figure 6: Filtering results. We show that the filtering pipeline can estimate
the object pose with just fingertips (red points) as input. It can give robust
and continuous estimation even when the purely learning-based method (a, c)
fails.

###  IV-C Filtering-based tracking

We apply the tracking algorithm described in Section III-E to perform pose
estimation. We run on the SM1 sequence and qualitative results are shown in
Figure 6. Our filtering pipeline (b) yields adequate estimation results of the
objects given only the fingertips as inputs (red dots in the figures).
Meanwhile, the (a) learning-based method [5] would have huge errors in object
pose estimation.

We also conduct quantitative experiments in Table I where different
observations are fed into the filter. $\mathcal{O}_{hand}$ use the predicted
3D positions of fingertips from as observation; $\mathcal{O}_{object}$ takes
the predicted object center as observations; $\mathcal{O}_{hand+object}$ uses
both the aforementioned hand and object information as an observation. The
results show that with more observation available, the tracking results tend
to be more accurate. The tracking method is much faster than the optimization
pipeline; however, the accuracy decreases since less information is provided
to it.

|  |  |
---|---|---|---
(a) Initialized | (b) Optimized | (c) Initialized | (d) Optimized
Figure 7: Qualitative results of contact-based pose refinement. Given
randomized initial poses (a,c) and desired contact regions, our method can
refine poses using a differentiable contact model & gradient-based
optimization (b,d).

###  IV-D Contact Refinement

The physics module can output contact information during prediction. Given the
pose of the hand and the object, our GPU-based differentiable Warp kernel [60]
can output the signed distance function $s$ and the normal direction
$\mathbf{n}$ for each hand vertex
$\\{\mathbf{n}_{i},s_{i}\\}_{i<n_{hand}}=contact(\theta,\mathbf{r},\mathbf{t})$.
Following ContactOpt [42], we define a contact value
$CO_{i}=\min(\frac{1}{s_{i}},1)$ for each vertex. When a vertex is within 1mm
($s_{i}<1$) of the object, $CO_{i}=1$ then means this point is in-contact.

Given randomly initialized hand poses, our method refer to the ground truth
contact information provided by ContactPose datasets [61] as the target. Since
our $contact(\cdot)$ module is also differentiable, we can then optimize the
poses to match the target contact status $\\{CO_{i},\mathbf{n}_{i}\\}$ with
gradient-based optimization.

Figure 7 shows examples of the optimization process. (a) and (c) are the input
randomized hand, (b) and (d) are the optimized hand-object interaction. The
results are similar to how people handle those objects in daily life.

###  IV-E Generalization to Human

Our proposed differentiable priors can also be applied to human object pose
estimation. With RGB images from the COCO2017 dataset [57], we use [54] to
initialize both human and object poses, and then use [56] to predict the
instance segmentation mask for both the human and objects. Subsequently, we
replaced the MANO hand model [20] in our pipeline by SMPL human body model
[62] which describes the human body also by pose $\theta\in\mathbb{R}^{72}$
and shape $\beta\in\mathbb{R}^{10}$ parameters. Following the same set up as
hand-object optimization, we refine human parameters
$(\theta,\beta,T_{human})$ and object parameters$(\mathbf{r},\mathbf{t})$
through our pipeline using both differentiable rendering and physics priors.
Our experiment shows that our method can generate accurate human-object pose
estimation. As ground truth poses for human and object are unavailable, we
report the intersection of union(IoU) and the interpenetration volume between
human and object in Table IV. Figure 8 shows the qualitative result of our
method and the contribution of each component of the pipeline. The rendering
loss in (b) helps to better align the 3D models with the RGB images but there
are still non-physical collisions. The physics loss in (c) finetunes the poses
and reduces such penetrations.

TABLE IV: Quantitative evaluation on human object pose estimation Our method can achieve higher 2D IoU and lower collision error ($cm^{3}$) compared to PHOSA [54]. Method | human IoU | object IoU | Collision
---|---|---|---
PHOSA | 0.45 | 0.51 | 6818.18
+$\mathcal{L}_{render}$ | 0.64 | 0.59 | 8742.63
+$\mathcal{L}_{physics}$ | 0.68 | 0.64 | 666.67

|  |
---|---|---
(a) Initialization | (b) $\mathcal{L}_{render}$ | (c) $\mathcal{L}_{render+physics}$

Figure 8: Human Object Pose Refinement. Qualitative comparison between (a)
initialization using [54], (b) refinement using only differentiable rendering,
and (c) refinement using both differentiable rendering and physics simulator.
The rendering loss helps to align the image, and the physics loss can mitigate
penetrations.

##  V Conclusion

In this paper, we present HandyPriors, a framework of differentiable rendering
and physics priors for pose estimation in hand-object-interaction scenes. Two
alternatives are presented to match different use cases: the optimization-
based pose refinement which focuses on the prediction accuracy, and the
filtering-based tracking which focuses on the runtime speed. Our proposed
rendering and physics-related operators enable closed-loop optimization and
tracking with self-supervision from the input RGB image sequences. Experiments
show that our method yields comparable or better results in the pose
estimation and contact refinement task due to our unified and general priors.
We also demonstrate that our method can be applied to robotic hand
manipulation and human pose estimation tasks.

We also note some limitations in our implementation. First, the optimization
pipeline can fail if the initialization is poor. This problem could be
alleviated by adding bounding boxes or key-point related terms to the
differentiable rendering module. Second, the performance of the optimization
could be improved through a faster differentiable rendering module, such as
Nvdiffrast [63]. Third, the current filtering pipeline based on EKF could
potentially output the uncertainty of the prediction. It would be promising to
incorporate our pipeline with probabilistic prediction.

## References

  * [1] G. Garcia-Hernando, S. Yuan, S. Baek, and T.-K. Kim, “First-person hand action benchmark with rgb-d videos and 3d hand pose annotations,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 409–419.
  * [2] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, _et al._ , “Learning dexterous in-hand manipulation,” _The International Journal of Robotics Research_ , vol. 39, no. 1, pp. 3–20, 2020.
  * [3] R. P. Sharma and G. K. Verma, “Human computer interaction using hand gesture,” _Procedia Computer Science_ , vol. 54, pp. 721–727, 2015.
  * [4] Y. Hasson, G. Varol, D. Tzionas, I. Kalevatykh, M. J. Black, I. Laptev, and C. Schmid, “Learning joint reconstruction of hands and manipulated objects,” in _CVPR_ , 2019.
  * [5] S. Liu, H. Jiang, J. Xu, S. Liu, and X. Wang, “Semi-supervised 3d hand-object poses estimation with interactions in time,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2021.
  * [6] Z. Cao, I. Radosavovic, A. Kanazawa, and J. Malik, “Reconstructing hand-object interactions in the wild,” _ICCV_ , 2021.
  * [7] Y. Hasson, G. Varol, C. Schmid, and I. Laptev, “Towards unconstrained joint hand-object reconstruction from rgb videos,” in _2021 International Conference on 3D Vision (3DV)_ , 2021.
  * [8] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16 000–16 009.
  * [9] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [10] M. I. Ribeiro, “Kalman and extended kalman filters: Concept, derivation and properties,” _Institute for Systems and Robotics_ , vol. 43, no. 46, pp. 3736–3741, 2004.
  * [11] S. Hampali, M. Rad, M. Oberweger, and V. Lepetit, “Honnotate: A method for 3d annotation of hand and object poses,” in _CVPR_ , 2020.
  * [12] S. Hampali, S. D. Sarkar, M. Rad, and V. Lepetit, “Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation,” in _CVPR_ , 2022.
  * [13] G. Billings and M. Johnson-Roberson, “Silhonet: An rgb method for 6d object pose estimation,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 3727–3734, 2019.
  * [14] E. Brachmann, A. Krull, F. Michel, S. Gumhold, J. Shotton, and C. Rother, “Learning 6d object pose estimation using 3d object coordinates,” in _Computer Vision – ECCV 2014_ , D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 2014, pp. 536–551.
  * [15] Y. Hu, J. Hugonot, P. Fua, and M. Salzmann, “Segmentation-driven 6d object pose estimation,” in _CVPR_ , 2019.
  * [16] B. Tekin, S. N. Sinha, and P. Fua, “Real-Time Seamless Single Shot 6D Object Pose Prediction,” in _CVPR_ , 2018.
  * [17] Y. He, H. Huang, H. Fan, Q. Chen, and J. Sun, “Ffb6d: A full flow bidirectional fusion network for 6d pose estimation,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021.
  * [18] C. Wang, D. Xu, Y. Zhu, R. Martín-Martín, C. Lu, L. Fei-Fei, and S. Savarese, “Densefusion: 6d object pose estimation by iterative dense fusion,” in _Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [19] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox, “Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes,” in _Robotics: Science and Systems (RSS)_ , 2018.
  * [20] J. Romero, D. Tzionas, and M. J. Black, “Embodied hands: Modeling and capturing hands and bodies together,” _ACM Transactions on Graphics, (Proc. SIGGRAPH Asia)_ , 2017.
  * [21] Y. Cai, L. Ge, J. Cai, and J. Yuan, “Weakly-supervised 3d hand pose estimation from monocular rgb images,” in _European Conference on Computer Vision_ , 2018.
  * [22] L. Ge, Y. Cai, J. Weng, and J. Yuan, “Hand pointnet: 3d hand pose estimation using point sets,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018.
  * [23] F. Mueller, F. Bernard, O. Sotnychenko, D. Mehta, S. Sridhar, D. Casas, and C. Theobalt, “Ganerated hands for real-time 3d hand tracking from monocular rgb,” in _Proceedings of Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [24] A. Spurr, J. Song, S. Park, and O. Hilliges, “Cross-modal deep variational hand pose estimation,” in _CVPR_ , 2018.
  * [25] C. Zimmermann and T. Brox, “Learning to estimate 3d hand pose from single rgb images,” in _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017.
  * [26] L. Yang and A. Yao, “Disentangling latent hands for image synthesis and pose estimation,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [27] S. Yuan, G. Garcia-Hernando, B. Stenger, G. Moon, J. Chang, K. Lee, P. Molchanov, J. Kautz, S. Honari, L. Ge, J. Yuan, X. Chen, G. Wang, F. Yang, K. Akiyama, Y. Wu, Q. Wan, M. Madadi, S. Escalera, S. Li, D. Lee, I. Oikonomidis, A. Argyros, and T. Kim, “Depth-based 3d hand pose estimation: From current achievements to future goals,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [28] S. Baek, K. Kim, and T. Kim, “Pushing the envelope for rgb-based dense 3d hand pose estimation via neural rendering,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [29] A. Boukhayma, R. d. Bem, and P. H. Torr, “3d hand shape and pose from images in the wild,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 843–10 852.
  * [30] Y. Rong, T. Shiratori, and H. Joo, “Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration,” in _IEEE International Conference on Computer Vision Workshops_ , 2021.
  * [31] C. Zimmermann, D. Ceylan, J. Yang, B. Russel, M. Argus, and T. Brox, “Freihand: A dataset for markerless capture of hand pose and shape from single rgb images,” in _IEEE International Conference on Computer Vision (ICCV)_ , 2019. [Online]. Available: ”https://lmb.informatik.uni-freiburg.de/projects/freihand/”
  * [32] D. Kulon, R. A. Guler, I. Kokkinos, M. M. Bronstein, and S. Zafeiriou, “Weakly-supervised mesh-convolutional hand reconstruction in the wild,” in _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [33] G. Moon, S.-I. Yu, H. Wen, T. Shiratori, and K. M. Lee, “Interhand2.6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image,” in _European Conference on Computer Vision (ECCV)_ , 2020.
  * [34] F. Mueller, D. Mehta, O. Sotnychenko, S. Sridhar, D. Casas, and C. Theobalt, “Real-time hand tracking under occlusion from an egocentric rgb-d sensor,” in _Proceedings of International Conference on Computer Vision (ICCV)_ , 2017. [Online]. Available: http://handtracker.mpi-inf.mpg.de/projects/OccludedHands/
  * [35] P. Panteleris, I. Oikonomidis, and A. Argyros, “Using a single rgb frame for real time 3d hand pose estimation in the wild,” in _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2018.
  * [36] Y. Liu, Y. Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and L. Yi, “Hoi4d: A 4d egocentric dataset for category-level human-object interaction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 21 013–21 022.
  * [37] S. Brahmbhatt, C. Tang, C. D. Twigg, C. C. Kemp, and J. Hays, “ContactPose: A dataset of grasps with object contact and hand pose,” in _The European Conference on Computer Vision (ECCV)_ , August 2020.
  * [38] Y.-W. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield, J. Kautz, and D. Fox, “DexYCB: A benchmark for capturing hand grasping of objects,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [39] G. Garcia-Hernando, S. Yuan, S. Baek, and T.-K. Kim, “First-person hand action benchmark with rgb-d videos and 3d hand pose annotations,” in _Proceedings of Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [40] L. Yang, K. Li, X. Zhan, J. Lv, W. Xu, J. Li, and C. Lu, “ArtiBoost: Boosting articulated 3d hand-object pose estimation via online exploration and synthesis,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [41] J. Park, Y. Oh, G. Moon, H. Choi, and K. M. Lee, “Handoccnet: Occlusion-robust 3d hand mesh estimation network,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [42] P. Grady, C. Tang, C. D. Twigg, M. Vo, S. Brahmbhatt, and C. C. Kemp, “Contactopt: Optimizing contact to improve grasps,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1471–1481.
  * [43] H. Hu, X. Yi, H. Zhang, J.-H. Yong, and F. Xu, “Physical interaction: Reconstructing hand-object interactions with physics,” in _SIGGRAPH Asia 2022 Conference Papers_ , 2022.
  * [44] S. Liu, T. Li, W. Chen, and H. Li, “Soft rasterizer: A differentiable renderer for image-based 3d reasoning,” _The IEEE International Conference on Computer Vision (ICCV)_ , Oct 2019.
  * [45] S. Laine, J. Hellsten, T. Karras, Y. Seol, J. Lehtinen, and T. Aila, “Modular primitives for high-performance differentiable rendering,” _ACM Trans. Graph._ , vol. 39, no. 6, 2020.
  * [46] M. Nimier-David, D. Vicini, T. Zeltner, and W. Jakob, “Mitsuba 2: A retargetable forward and inverse renderer,” _ACM Trans. Graph._ , vol. 38, no. 6, 2019.
  * [47] E. Heiden, D. Millard, E. Coumans, Y. Sheng, and G. S. Sukhatme, “NeuralSim: Augmenting differentiable simulators with neural networks,” _arXiv:2011.04217_ , 2020.
  * [48] J. Xu, T. Chen, L. Zlokapa, M. Foshey, W. Matusik, S. Sueda, and P. Agrawal, “An end-to-end differentiable framework for contact-aware robot design,” _arXiv preprint arXiv:2107.07501_ , 2021.
  * [49] E. Heiden, C. E. Denniston, D. Millard, F. Ramos, and G. S. Sukhatme, “Probabilistic inference of simulation parameters via parallel differentiable simulation,” _International Conference on Robotics and Automation (ICRA)_ , 2022.
  * [50] S. Christen, M. Kocabas, E. Aksan, J. Hwangbo, J. Song, and O. Hilliges, “D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [51] D. Turpin, L. Wang, E. Heiden, Y.-C. Chen, M. Macklin, S. Tsogkas, S. Dickinson, and A. Garg, “Grasp’d: Differentiable contact-rich grasp synthesis for multi-fingered hands,” in _European Conference on Computer Vision (ECCV)_ , 2022.
  * [52] D. Turpin, T. Zhong, S. Zhang, G. Zhu, E. Heiden, M. Macklin, S. Tsogkas, S. Dickinson, and A. Garg, “Fast-grasp’d: Dexterous multi-finger grasp generation through differentiable simulation,” in _ICRA_ , 2023.
  * [53] S. K. Dwivedi, N. Athanasiou, M. Kocabas, and M. J. Black, “Learning to regress bodies from images using differentiable semantic rendering,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 11 250–11 259.
  * [54] J. Y. Zhang, S. Pepose, H. Joo, D. Ramanan, J. Malik, and A. Kanazawa, “Perceiving 3d human-object spatial arrangements from a single image in the wild,” in _European Conference on Computer Vision (ECCV)_ , 2020.
  * [55] E. Gärtner, M. Andriluka, E. Coumans, and C. Sminchisescu, “Differentiable dynamics for articulated 3d human motion reconstruction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 13 190–13 200.
  * [56] A. Kirillov, Y. Wu, K. He, and R. Girshick, “Pointrend: Image segmentation as rendering,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [57] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European Conference on Computer Vision (ECCV)_ , D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds., 2014.
  * [58] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar, “The ycb object and model set: Towards common benchmarks for manipulation research,” in _2015 International Conference on Advanced Robotics (ICAR)_ , 2015.
  * [59] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _International Conference on Learning Representations_ , 2014.
  * [60] M. Macklin, “Warp: A high-performance python framework for gpu simulation and graphics,” https://github.com/nvidia/warp, March 2022, nVIDIA GPU Technology Conference (GTC).
  * [61] S. Brahmbhatt, C. Tang, C. D. Twigg, C. C. Kemp, and J. Hays, “Contactpose: A dataset of grasps with object contact and hand pose,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16_. Springer, 2020, pp. 361–378.
  * [62] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, “SMPL: A skinned multi-person linear model,” _ACM Trans. Graphics (Proc. SIGGRAPH Asia)_ , 2015.
  * [63] S. Laine, J. Hellsten, T. Karras, Y. Seol, J. Lehtinen, and T. Aila, “Modular primitives for high-performance differentiable rendering,” _ACM Transactions on Graphics_ , vol. 39, no. 6, 2020.


# Length-scale Dependence of Stokes-Einstien Breakdown in Active Glass-forming
Liquids

Anoop Mutneja 1, and Smarajit Karmakar 1 1 Tata Institute of Fundamental
Research, 36/P, Gopanpally Village, Serilingampally Mandal,Ranga Reddy
District, Hyderabad, Telangana 500107, India

###### Abstract

Stokes-Einstein (SE) relation, which relates diffusion constant with the
viscosity of a liquid at high temperatures in equilibrium, is violated in the
supercooled temperature regime. Whether this relation is obeyed in
nonequilibrium active liquids is a question of significant current interest to
the statistical physics community trying to develop the theoretical framework
of nonequilibrium statistical mechanics. Via extensive computer simulations of
model active glass-forming liquids in three dimensions, we show that SE is
obeyed at a high temperature similar to the equilibrium behaviour, and it gets
violated in the supercooled temperature regimes. The degree of violation
increases systematically with the increasing activity which quantifies the
amount the system is driven out of equilibrium. First passage-time (FPT)
distributions helped us to gain insights into this enhanced breakdown from the
increased short-time peak, depicting hoppers. Subsequently, we study the wave
vector dependence of SE relation and show that it gets restored at a wave
vector that decreases with increasing activity, and the cross-over wave vector
is found to be proportional to the inverse of the dynamical heterogeneity
length scale in the system. Our work showed how SE violation in active
supercooled liquids could be rationalized using the growth of dynamic length
scale, which is found to grow enormously with increasing activity in these
systems.

##  I Introduction

The theory of Brownian motion captures the random erratic motion of a probe
suspended in a host, and at its heart is the assumption of the separation of
time scales. Put simply, the Brownian particle is hit by smaller particles in
the medium (smaller time scale), which creates a non-zero force causing it to
move (larger time scale). However, for the particle to move, the host
particles must displace through viscous dynamics, facilitating the Brownian
dynamics of a probe particle. Einstein [1] linked the probe and host dynamics
by equating the two-time scales, namely the viscous time scale, $\eta
R^{3}/T$, and the diffusive time scale of the probe $R^{2}/D_{P}$, via
fluctuation-dissipation relation, up to a constant. It is known as the Stokes-
Einstein (SE) relation [2] and is obeyed by a probe if $\eta D_{p}/k_{B}T$ is
independent of temperature. Perrin [3] used the SE relation to determine
Avogadro’s number for the first time. This probe can be any external particle,
or the particle of the medium itself, leading to the self version of the SE
relation, $\eta D_{s}/k_{B}T=constant$.

Supercooled liquids, which show a considerable viscosity increase with a
slight decrease in temperature, challenge the SE relation [4, 5, 6, 7, 8, 9,
10, 11, 12]. The decrease in diffusion constant on reducing temperature does
not compensate for the massive increase in the system’s viscosity, resulting
in Stokes-Einstein breakdown (SEB). Both experiments and numerical studies
have demonstrated that the SE parameter ($\eta D_{s}/k_{B}T$) increases as
temperature decreases. The SE breakdown is also a hallmark of dynamic
heterogeneity (DH), the existence of spatial regions with different
mobilities. The broadening of relaxation time distribution would result in the
bifurcation of the two-time scales; the slower set would control the viscous
time scale, while the faster set would control the diffusive time scale; hence
the violation [13]. Another set of reasoning involves the change of inherent
particle dynamics; with increasing supercooling, the dynamics of the host
particles change from normal viscous to collective, activated, and hence
slower dynamics [14, 15, 16]. SE relation works if the host follows the normal
viscous dynamics; however, the collective, or the “hopping dynamics” in the
supercooled regime, would enhance the diffusion of the probe [14, 15, 16].

In [9, 10], the distribution of diffusion constants were explicitly shown to
have a bimodal structure with increasing supercooling. The slower group of
particles obeys the SE relation even in the supercooled regime if one compares
their respective relaxation time and diffusion constant. In contrast, for the
bulk measurement, relaxation time is controlled by the slower particles, and
diffusion is controlled by the faster set, leading to the violation of SE.
Alternatively, the system tends to follow a fractional SE relation, viz.
$D_{s}\sim\tau_{\alpha}^{-(1-\omega)}$, with $\omega\in[0,1]$, note that the
relaxation time $\tau_{\alpha}$ is usually used as a proxy for $\eta/T$. The
deviation of $\omega$ from zero characterizes the degree of deviation and
depends on the dynamical features of the system. In Ref. [10], the SE relation
was tested with increasing random pinning in the system; the power $\omega$
was shown to increase with increasing pinning concentration. In Ref.[5], SE
relation is tested across the spatial dimensions ($d$) of the system, and it
was noted that SE violation gets systematically reduced with increasing
spatial dimensions and SE is completely recovered at the upper critical
dimension $d_{u}\simeq 8$. Above this dimension, the system becomes more mean-
field-like, and one expects diminished or negligible dynamic heterogeneity. In
this study, our goal is to test the SE relation of a model active glass-
forming liquid with increasing activity, which is introduced by giving a self-
propulsion force to a fraction of particles in the system, as discussed in the
model and methods section.

The intensity of SEB varies depending on the type of probe used [6, 4]. For
example, in a molecular liquid like othro-terphenyl (OTP) [4], the SE relation
holds for larger probes (viz. rubrene), but SEB intensity increases as the
probe size decreases. This suggests that there may be a specific length scale
that controls SE breakdown. This idea has been tested through molecular
dynamics simulations in Ref. [17, 18], where probes of varying sizes were used
in supercooled liquids. The growing, dynamic length scale, $\xi_{d}$
(introduced later), was found to control the SE breakdown. Additionally, it is
well-established that the SE breakdown is not observed when probed at a
smaller wavevector (large length scale) [8]. In this study, the authors
checked the SE relation at different length scales by using the relaxation
times obtained from the self-part of the intermediate scattering function
($F_{s}(k,t)$) (defined later) for different wavevectors $k$. A method was
then proposed to obtain the SEB temperature for different wavevectors, which
followed the dynamic length scale of the system.

The growing length scale seems to be an essential requisite for the
hypothesized glass transition, and it was found that at least two length
scales are required to explain the hallmarks of glass-forming liquids [19, 20,
21, 22]. Although there are alternative theories that considers glass
transition to be purely dynamic in nature and does not necessarily require a
growing static correlation length [23, 24, 25], but existence of a dynamical
length scale is largely accepted across these diverse theories of glass
transition. This first scale is associated with DH patches, which control the
SE breakdown, Non-Gaussian dynamics, and non-exponential relaxation. However,
this scale does not causally account for the growing relaxation time of the
system. The second length scale, known as the static length scale, was
proposed in the RFOT framework [26, 27] to relate the slow dynamics causally.
A few procedures are used to obtain these length scales numerically, such as
the finite size scaling analysis [19, 20, 21] and the Block analysis [28].

The computer glass forming systems only fall short of the order of growing
length scale; they only show a slowdown of 4-5 decades, with the length scale
growing by a few particle diameters. Active supercooled systems have recently
gained much attention because of the enormous growing length scales [29] and
ubiquitous presence. Researchers in the past decade have focused on the
emerging nonequilibrium class of soft matter systems, namely active glasses
[29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41] to develop an
understanding of the nonequilibrium steady states and their statistical
properties. The constituent particles, along with the glassy disordered
behavior, are energy-driven (internal or supplied externally). The
intercellular dynamics, wound healing, progressing cancer cells, and driven
granular matter [42, 43, 44] are a few examples of active glassy systems. As
already mentioned, such systems show enhanced DH, which is also experimentally
sound; the cancer cells (more active) show greater DH than the healthy cells,
which may lead to metastatic cell clustering [45, 46]. The dynamic length
scale in these systems can grow by as much as $30$ particle diameters (for the
largest studied activity), providing a perfect playground to test various
tools developed for equilibrium systems to work in such superlative
nonequilibrium systems.

This work addresses the length-scale variation of SE violation in the active
system. The reason behind its breakdown in a passive supercooled system is
debated; one stream of thought links it with the DH, while the other thinks it
to be an effect of change in inherent dynamics, from viscous to activated. A
natural question that arises in active systems is whether SE relation is
obeyed or not. As active systems are in nonequilibrium states, the SE
breakdown need not require any reason. Nevertheless, the SE relation at high
temperatures seems to be obeyed perfectly like any other liquid. Thus validity
or violation of SE relation in active supercooled liquid is a pertinent
question, and if it is violated, whether the degree of violation increases
with increasing activity and whether dynamic heterogeneity plays any role in
this context or not are questions worth further studies. These questions are
non-trivial as we are asking these questions for a system that is at a
nonequilibrium steady state, and it is apriori not known whether concepts of
equilibrium statistical mechanics can be applied to these systems. In this
work, we show that SE violation in active glass-forming liquids increases with
increasing activity, and the physics of SE violation in active glasses can be
understood in a unified manner using the growing dynamic heterogeneity length
scale in the same way as in equilibrium passive glass-forming liquids. In
fact, we show that a systematic wave vector dependence of SE relation can
itself yield the dynamic heterogeneity length scale.

The rest of the article is organized as follows. We first give the details of
the model active system we studied using extensive molecular dynamics
simulations. Then we explain how various dynamical quantities are extracted
using appropriate correlation functions. In the results section, we discuss
the violation of the SE relation and how the SE relation gets restored once we
look at it at a different probing length or wave vector with changing activity
and temperatures in the system. We then show how the dynamic heterogeneity
length scale intimately relates to the systems’ SE violation. Finally, we
conclude with implications of our findings on the physics of glass transition
in nonequilibrium systems and their similarity and difference with the
equilibrium counterparts.

Figure 1: (a) Time evolution plot of mean square displacement (MSD) for an
active system with $f_{0}=2.0$. (b) The $MSD/t$ plot shows the eventual
reaching of the diffusive behavior, with the diffusion constant proportional
to the large time limit of the y-axis. (c) The extracted diffusion constant
$D$ is plotted against the temperature for the systems with different
activities.

##  II Models and Methods

This work consists of large-scale molecular dynamics simulations performed on
the binary system (particle tag A and B) in the ratio of $80:20$, known in the
literature as the Kob-Andersen model [47], in three dimensions with the system
size of $N=50000$ particles. The system is simulated in an NVT ensemble using
a 3-chain Nose-Hoover thermostat. Activity is introduced as an eight-states
clock model with the active force assigned as

$F^{A}_{i}=f_{0}\left(k^{i}_{x}\hat{x}+k^{i}_{y}\hat{y}+k^{i}_{z}\hat{z}\right),$
(1)

on $c=0.1$ concentration of randomly tagged active particles. In Eq.1,
$k^{i}_{\alpha}\in\pm 1$ are chosen randomly after the persistence time
$\tau_{P}=1.0$. $\sum_{i=1}^{N_{A}}k_{\alpha}^{i}=0$ is ensured for each
spatial dimension to maintain the system’s zero center-of-mass velocity. We
averaged our results with $16$ statistically independent simulation runs for
each state point, and the random particle choices are different for each run.

The obtained trajectory of the system is then analyzed to get the average mean
squared displacement (MSD) (Fig.1(a)) and hence the diffusion constant as
$D\propto MSD/t$ (Fig.1(b, c)) in the long time limit. The intermediate
scattering function calculated as the Fourier transform of the density-density
correlation function, calculated for different wavevectors
$k_{\alpha}=2n_{\alpha}\pi/L$, gives the relaxation profile at different
length scales. Here $\alpha$ is for dimensions, and $n_{\alpha}$ is a positive
integer. $L$ is the system size. The self part of the intermediate scatter
function, $F_{s}(k,t)$ is defined as

$F_{s}(k,t)=\frac{1}{N}\left\langle\sum_{i=1}^{N}e^{-i\textbf{k}.(\textbf{r}_{i}(t)-\textbf{r}_{i}(0)}\right\rangle,$
(2)

where $\textbf{r}_{i}(t)$ is the instantaneous position of particle $i$ at
time $t$ and the $\langle\cdots\rangle$ refers to thermal averaging.

Figure 2: (a) $F_{s}(k,t)$ plots for active system ($f_{0}=2.0$) subjected to
different temperatures, showing the typical supercooled two-step non-
exponential profile for low temperatures. These are the plots for $k=7.20$,
where the structure factor peaks. (b) Wavevector variation of the $F_{s}(k,t)$
is shown for the same active system at temperature $T=0.480$. The relaxation
profile becomes more and more exponential with decreasing $k$. (c) The
extracted $\tau_{\alpha}^{\prime}(k,T)$ from
$Fs(k,\tau_{\alpha}^{\prime})=0.01$ cut is plotted against $k$ for different
wavevectors. (d) The system’s temperature variation of
$\tau_{\alpha}^{\prime}$ ($k=7.20$) is shown for different activities.

Fig.2(a) shows the $Fs(k,t)$ for an active system with $f_{0}=2.0$ for
different temperatures, while the wavevector dependence can be seen in
Fig.2(b) for the same system with $f_{0}=2.0$, and $T=0.480$. The relaxation
time $\tau_{\alpha}(k)$ for a wavevector $k$ is calculated as
$F_{s}(k,\tau_{\alpha}(k))=1/e$. It has been shown in Ref. [8] that with
$\tau^{\prime}_{\alpha}$ calculated using
$F_{s}(k,\tau^{\prime}_{\alpha}(k))=0.01$, does not show SEB for all $k$ in
the high-temperature regime, in contrast of the $\tau_{\alpha}(k)$. Thus, we
would use $\tau_{\alpha}^{\prime}$ in the rest of the article. The relaxation
time of the system $\tau_{\alpha}^{\prime}(T)$ is calculated for wavevector
$k=7.20$ at which the structure factor, $S(k)$, peaks. The static structure
factor is defined as

$S(k)=\frac{1}{N}\left\langle\sum_{i,j=1}^{N}e^{-i\textbf{k}.(\textbf{r}_{j}-\textbf{r}_{i})}\right\rangle.$
(3)

The $k$ dependence of $\tau_{\alpha}^{\prime}$ can be seen in Fig.2(c), while
its temperature dependence for systems with different activities is shown in
panel (d) of Fig.2. Also, we only use the dynamic information of particles of
type A.

Figure 3: (a) The relaxation times ($\tau_{\alpha}^{\prime}$) are shown for
different temperatures as a function of active forcing $f_{0}$. The lines are
the VFT lines and not fits. The equation used is
$\tau_{\alpha}^{\prime}(T)=\tau_{0}exp\left(\frac{A}{T_{eff}-T_{VFT}}\right)$,
where $T_{eff}=T+Kf_{0}^{2}$, is the effective temperature with $K=0.015$ as a
constant. The value of $T_{VFT}$ is obtained from the VFT fit of the passive
case, along with the other unknown constants ($\tau_{0}$, $A$). Changing $T$
is what creates all the lines on the graph. (b) The graph also shows the
inverse diffusion constant, further supporting the idea that active systems
behave similarly to passive systems at higher effective temperatures. (c) We
plot the SE parameter using VFT lines for different levels of activity in
systems. The SE breakdown for $f_{0}=0$ is accurately detected, but it shows a
weaker breakdown for higher activity levels. This is in contrast to what is
numerically observed.

##  III Results

In Ref. [29, 37, 48], it was shown that relaxation behaviour as quantified by
the relaxation time of an active system with changing activity could be
completely understood using a suitably defined effective temperature,
$T_{eff}$. Thus, it is natural to expect the diffusion constant to follow the
same behaviour. In Fig.3(a), we plot $\tau_{\alpha}^{\prime}$ as a function of
$f_{0}$ for all the temperatures and the lines are the theoretical predictions
within an effective temperature description as shown in Ref.[37]. In our case,
we fit the relaxation time vs. temperature for the passive system ($f_{0}=0$)
with Vogel-Fulcher-Tamman (VFT) formula as

$\tau_{\alpha}^{\prime}(T)=\tau_{0}exp\left(\frac{A}{T-T_{VFT}}\right)$ (4)

and extract the unknown constants $\tau_{0}$, $A$, and $T_{VFT}$ and then used
the following definition of effective temperature $T_{eff}=T+Kf_{0}^{2}$ and
plotted the function
$\tau_{\alpha}^{\prime}(T)=\tau_{0}exp\left(\frac{A}{T_{eff}-T_{VFT}}\right)$
for all the value of $f_{0}$ studied in this work. The predictions (the solid
lines) with $K=0.015$ [29] are in perfect agreement with the simulation
results (symbols). A similar analysis for inverse diffusion constant is done
in Fig.3(b), where we kept $T_{VFT}$ and $K$ to be the same and obtained the
two free parameters $D_{0}$ and $B$ of the following equation

$\frac{1}{D(T)}=\frac{1}{D_{0}}exp\left(\frac{B}{T-T_{VFT}}\right)$ (5)

using the diffusion constant data for the passive system only. The comparison
with effective temperature description is reasonably good at higher
temperatures but not so good at lower temperatures and higher activity,
indicating a possible decoupling between relaxation time, which is related to
multiparticle correlation, and the diffusion constant, which is the average
single-particle property of the system.

When we use the estimated diffusion constant and relaxation time within the
effective temperature description (lines in Figs.3(a) and (b)) and plot the SE
parameter against relaxation time, we observe a violation of SE relation.
However, the dependence on activity is not accurately represented in Fig.3(c).
It suggests a weaker violation in active systems with increasing activity,
while stronger violations are observed in more active systems. Details about
this will be discussed further below.

Figure 4: The scaled SE parameter ($D\tau^{\prime}_{\alpha}$) is plotted
against the relaxation time for systems with different activity strengths. The
breakdown intensifies with increasing activity for a system with similar
relaxation time, with the inference of increased DH. Also, the SEB happens at
lower relaxation times for the systems of a larger activity.  Figure 5: (a)
The cross plot of Log-log plot of the diffusion constant $D$ and
$\tau^{\prime}_{\alpha}$, showing the SE relation for high temperatures, and
fractional SE relation $D\propto\tau^{\prime-(1-\omega)}_{\alpha}$ in the low-
temperature regime. With increasing activity, the power $1-\omega$ decreases
further and further from one and can be seen in panel (b). (c) Wavevector
variation of power $1-\omega$ for systems of different activities. $\omega$
approaches zero in the limit of small wavevectors and low temperatures,
implying the soundness of SE relation.

In Fig.4, we have plotted the SE parameter $D\tau^{\prime}_{\alpha}$ for
systems with different active forcing $f_{0}$; all these systems obey the SE
relation in the high-temperature limit and show the violation in the
supercooling regime. The breakdown parameter escalates with decreasing
temperature, while the rate of increase magnifies with increasing activity.
The relaxation time where this breakdown happens also decreases with
increasing activity. It is important to highlight that a similar effect was
also seen with increasing random pinning concentration in the system [10]. It
is clear from Fig.4 that for systems with similar relaxation times but
different activities, SE violation is much stronger for more active systems,
which would have larger DH [29] leading to a strong link between DH and the SE
breakdown. A substantial increase of DH with increasing activity has also been
reported in Keta et al. [49] and in some biological systems [45, 46].

Another way to characterize the observed intensified SE violation in an active
system is to study the fractional SE relation,
$D\tau^{\prime(1-\omega)}_{\alpha}$. The power $\omega=0$ corresponds to the
SE relation, while for the supercooled liquids at low temperatures, $\omega>0$
is observed. The microscopic understanding of the fractional SE exponent is
still unknown, but it depends on the microscopic interactions’ details. In
Fig.5(a), we have shown a log-log plot of the diffusion constant and
relaxation time; at the high-temperature regime, all systems with different
activities follow the SE relation ($\omega=0$), even with the same
proportionality constant. However, the curves get separated from each other in
the low-temperature limit; more active systems have a more significant
deviation from $\omega=0$. Fig.5(b) shows the values of $\omega$ with
increasing activity. To study the effect of probing wavevector, we also look
at the SE violation with wavevector dependent relaxation time,
$\tau_{\alpha}^{\prime}(k)$ using the following SE relation

$D\sim\tau_{\alpha}^{\prime}(k)^{-(1-\omega(k))}.$ (6)

Fig.5(c) shows the wavevector dependence of the power $\omega$ for systems of
different activities in the limit of low temperatures. The power converges to
$\omega=0$ in the limit of small $k$ for all activities; a systematic trend of
decreasing $\omega$ with activity can also be seen.

Figure 6: (a) The Brownian particle starting from the origin in a space with
an absorbing boundary at $|r|=1$; the time it gets absorbed would be the first
passage time ($t_{FPT}$). (b) The distribution of FPT for such a Brownian
particle with different diffusion constants $D$. (c) Simple scaling of
$t_{FPT}*D$ brings all the curves to a master plot. (d) The FPT distributions
for 3dKA active system with $f_{0}=2.0$ subjected to different temperatures.
The appearance of a bimodal character and small time peak with increasing
supercooling is proof of the DH and a reason for SE breakdown. Panel (e,f)
shows the scaling of $t_{FPT}$, by $\tau_{\alpha}^{\prime}$, and $1/D$,
respectively. The first one tends to give better scaling of large-time
distribution than the latter, proving the involvement of different particles
in calculating $\tau_{\alpha}^{\prime}$ and $D$.

To unearth the intricate role of activity on SE relation, we further look at
the first-passage-time (FPT) of the particles. For a Brownian particle in 3d,
the $FPT(r_{c})$ is defined as the time when a particle crosses a sphere of
radius $r_{c}$ for the first time starting from origin (see Fig.6(a)). The
Brownian particle in such an enclosed boundary would result in exponentially
decaying FPT distributions. A typical FPT distribution would look like
Fig.6(b) for different choices of diffusion constant of the Brownian particle.
All these curves can be scaled by the diffusion constant $D$ itself, as shown
in Fig.6(c), suggesting only one timescale in the Brownian problem with a
homogeneous medium. Fig.6(a-c) are the FPT plots for non-interacting Brownian
particles, while Fig.6(d-f) are the FPT plots of the supercooled system at
different temperatures with active forcing of $f_{0}=2.0$. With increasing
supercooling, the FPT distribution shows a bimodal behavior (Fig.6(d)), with
the emergent peak at short times being the contribution from the hoppers in
the systems. This short-time peak systematically increases with increasing
supercooling. If we try to scale the FPT with the relaxation time of the
system, Fig.6(e) (assuming the SE relation), the large-time part scales to
some extent to a master plot; however, the hoppers fall out. On the other
hand, the scaling with the diffusion constant, Fig.6(f), does not show such
collapse. The broadening of FPT distribution, especially in the short time
regime, indicates the existence of a population of particles having diffusion
constants that are significantly different from each other and is one of the
primary causes for the SE breakdown. This suggests that distinct sets of
particles with different diffusion constants contribute to estimating the
relaxation time $\tau_{\alpha}^{\prime}$ and mean diffusion constant of the
system, $D$. The slower set of particles appears to control
$\tau_{\alpha}^{\prime}$ as the right-hand part of the FPT distribution can be
scaled to a large extent by the average relaxation time of the system, while
$D$ gets contributions from all the sets.

Figure 7: (a)Relaxation profile of similarly relaxing active systems with
different active forcing. The temperature of the system is so adjusted to
match the relaxation times. (b) FPT distributions of the above mentioned
systems; the increase in the fraction of hoppers with increasing activity is
clear, thus the enhanced DH reported in [29, 50], and the increase in the
power $\omega$.

Next, in Fig.7(b), we show the FPT distribution of systems with different
activities and bath temperature but having similar relaxation times as shown
by the decay of $F_{s}(k,t)$ in Fig.7(a). The increased amplitude of the first
peak in Fig.7(b) shows that the population of fast-moving hoppers increases
with increasing activity, which is also supported by Fig.7(c) and Fig.7(d)
where the same FPT distribution is computed by considering only the passive
particles and only the active particles respectively. As argued earlier, these
hoppers lead to the SE violation; thus, the increased population of hoppers
with increasing activity will naturally mean stronger SE violation with a
significant decrease in fractional SE power, $1-\omega$. This is also
consistent with the observed increase in DH in a similar system by Paul et al.
[29]. The FPT plots in supercooled regimes also support the fact of change in
inherent dynamics of the constituents, from normal viscous to activated
(dynamics of cages and hoppers) [14, 15, 16].

Figure 8: (a) The cross plot of diffusion constant, $D$ against the relaxation
time, $\tau_{\alpha}^{\prime}(k)$ for different wavevectors, $k$. The dotted
line corresponds to $y=1/x$ and is obeyed for high temperatures and smaller
$k$. (b) The SE parameter, $D\tau_{\alpha}^{\prime}(k)$, is shown for
different $k$. (c) The SE parameter scaled with the higher temperature values
is shown. The breakdown happens at higher and higher temperatures for larger
$k$. The lines in the plot are a fit to a function,
$\ln{y}=x+c_{0}(x/x_{0})^{n}+c_{1}(x/x_{0})^{2n}$, where
$x=log(\tau_{\alpha}^{\prime})$, and the SEB temperature is calculated from
the relaxation time for $y=1.1$ cut. (d) The length scale obtained from SE
breakdown compares well with the $\xi_{d}$ obtained by conventional methods in
Ref.[29].

We will now focus on examining the connection between the DH and the SE
breakdown through the dynamic length scale, $\xi_{d}$. In equilibrium liquids,
$\xi_{d}$ is responsible for regulating the finite size effects observed in
the four-point dynamic susceptibility ($\chi_{4}^{P}(T,N)$), which measures
the fluctuations in the two-point correlation function and serves as an
indicator of the correlated relaxation process in a system [20]. Similarly, it
is found to be the same as the coarse-graining scale over which the vanHove
function crosses over from non-Gaussian to Gaussian form [51]. $\xi_{d}$ also
dictates the dependence of non-normal parameters for a probe particle in
supercooled liquids on the size of the probe [17, 18]. The relaxation times
obtained from the dynamics at length scales larger than the dynamic length
scale of the system were also shown to be coupled to the diffusion constant in
Ref.[8]. We establish a similar finding in nonequilibrium active systems too.
Fig.8(a) shows the log-log plot of diffusion constant ($D$) and relaxation
time ($\tau^{\prime}_{\alpha}(k)$) for different wavevectors ($k$). Fig.8 is
for a system with active forcing of magnitude $f_{0}=2.0$, while the results
for all other active forcing are presented in Appendix. The relaxation times
for wavevector $k$ can be thought of as the relaxation time probed at the
coarse grain length scale of $\sim\frac{2\pi}{k}$; thus, the larger $k$ values
refer to the relaxation times probed at smaller length scales and vice versa.
The relaxation times obtained for smaller $k$ values likely follow the SE
relation, as seen from Fig. 8 (b), where we plotted the SE parameter for
different wavevectors. We scaled the SE parameter with its large temperature
values 8(c) to observe the SE breakdown at different length scales
systematically. The SE breakdown temperatures for different wavevectors are
extracted by putting a threshold of $Scaled~{}D\tau^{\prime}_{\alpha}=1.1$.
The SEB length scale compares well with the temperature dependence of the
$\xi_{d}$ of the active system [29] as shown in Fig.6(d). Filled circles refer
to the data obtained in [29] via various methods of extracting the dynamical
length, whereas the square symbols refer to the length scale obtained from the
cross-over length scale above which SE is obeyed at that temperature and
activity. This backs the hypothesis of the DH controlling the SEB in the
active systems as well. Note that the dynamic length scale proliferates with
supercooling in active glass-forming liquids and can grow by as much as
$20-25$ particle diameter with increasing activity. A similar analysis for
systems with different activities is provided in Appendix to highlight the
generic nature of our results across changing activity.

##  IV Discussion

To summarise, we have studied various aspects of SE relation in the active
system with different activities. These systems are found to exhibit enhanced
dynamic heterogeneity effects and large growing dynamic length scales. The
active systems follow the SE relation in the limit of higher temperatures.
However, the SE relation is violated at lower temperatures, and instead, the
fractional SE relation is obeyed in complete accordance with passive
supercooled liquids. The exponent of fractional SE relation however decreases
systematically with increasing activity, and the exponent is found to decrease
as much as $1-\omega=0.612$ when the relaxation time is computed at the
probing wavevector of $k\sim 7.25$ (near the peak of the structure factor) for
the highest activity $f_{0}=2.5$. The exponent $\omega$ has significant
wavevector dependence; it increases with increasing $k$ and reaches
$1-\omega=0.573$ for the highest activity and largest $k$ considered in this
study. The first passage time (FPT) distributions of particles in these active
supercooled liquids provided significant insight into enhanced SE breakdown in
these systems. The FPT distributions developed a small time peak signifying
the change in the inherent viscous dynamics to hopping-dominated dynamics with
increasing activity. The fraction of hoppers were found to be larger in
similarly relaxing but more active systems, which we conclude to be the main
driving factor for the observed enhanced DH and stronger SE violation. The
large time tail of FPT distributions was found to be scaled reasonably well by
the relaxation time of the system, while the mean follows the diffusion
coefficient. We showed that the breakdown temperature of the SE relation
increase with increasing $k$, providing information on the intricate role
played by growing dynamic length scales in the active systems. As the dynamic
length scale in these systems can grow by nearly an order of magnitude
compared to passive glass-forming liquids, the role of dynamic heterogeneity
length scale in SE violation became even more prominent than their equilibrium
counterparts. Finally, this work shows that the SE relation (an equilibrium
concept) can be extended to inherently nonequilibrium systems to gain insights
into the dynamics and to extract information on growing correlation lengths in
the system. Although Stokes-Einstein breakdown in active liquids is found to
be similar to the equilibrium supercooled liquids in the range of the studied
parameters, it is not immediately clear whether these concepts can be extended
for active systems in their extreme limit, for example, with very large
persistence time $\tau_{p}$, at which the system can show intermittent
dynamical behaviour [52].

##  V Acknowledgements

We acknowledge funding by intramural funds at TIFR Hyderabad from the
Department of Atomic Energy (DAE) under Project Identification No. RTI 4007.
Core Research Grant CRG/2019/005373 from Science and Engineering Research
Board (SERB) is acknowledged for generous funding. Most of the computations
are done using the HPC clusters procured using CRG/2019/005373 grant and
Swarna Jayanti Fellowship, grants DST/SJF/PSA01/2018-19, and
SB/SFJ/2019-20/05.

##  VI Appendix

The SE breakdown analysis for different activities is presented in Fig.9. The
length scale extracted compares well with the dynamic length scale of the
system.

Figure 9: The SE analysis to obtain the breakdown temperature for systems with
different activity. See Results for details.

## References

  * Einstein [1905] A. Einstein, Über die von der molekularkinetischen theorie der wärme geforderte bewegung von in ruhenden flüssigkeiten suspendierten teilchen, Annalen der Physik 322, 549 (1905).
  * Hansen and McDonald [2013] J.-P. Hansen and I. R. McDonald, Chapter 7 - time-dependent correlation and response functions, in  _Theory of Simple Liquids (Fourth Edition)_, edited by J.-P. Hansen and I. R. McDonald (Academic Press, Oxford, 2013) fourth edition ed., pp. 265–310.
  * Soddy [1910] F. Soddy, Brownian movement and molecular reality (1910).
  * Liu  _et al._ [2015] Q. Liu, S. Huang, and Z. Suo, Brownian motion of molecular probes in supercooled liquids, Physical Review Letters 114, 10.1103/physrevlett.114.224301 (2015).
  * Adhikari  _et al._ [2021] M. Adhikari, S. Karmakar, and S. Sastry, Spatial dimensionality dependence of heterogeneity, breakdown of the stokes–einstein relation, and fragility of a model glass-forming liquid, The Journal of Physical Chemistry B 125, 10232 (2021).
  * Rajian and Quitevis [2007] J. R. Rajian and E. L. Quitevis, Translational diffusion in sucrose benzoate near the glass transition: Probe size dependence in the breakdown of the stokes-einstein equation, The Journal of Chemical Physics 126, 224506 (2007).
  * Swallen  _et al._ [2003] S. F. Swallen, P. A. Bonvallet, R. J. McMahon, and M. D. Ediger, Self-diffusion of tris-naphthylbenzene near the glass transition temperature, Phys. Rev. Lett. 90, 015901 (2003).
  * Parmar  _et al._ [2017] A. D. Parmar, S. Sengupta, and S. Sastry, Length-scale dependence of the stokes-einstein and adam-gibbs relations in model glass formers, Physical Review Letters 119, 10.1103/physrevlett.119.056001 (2017).
  * Sengupta and Karmakar [2014] S. Sengupta and S. Karmakar, Distribution of diffusion constants and stokes-einstein violation in supercooled liquids, The Journal of Chemical Physics 140, 224505 (2014).
  * Bhowmik  _et al._ [2016] B. P. Bhowmik, R. Das, and S. Karmakar, Understanding the stokes–einstein relation in supercooled liquids using random pinning, Journal of Statistical Mechanics: Theory and Experiment 2016, 074003 (2016).
  * Ediger and Harrowell [2012] M. D. Ediger and P. Harrowell, Perspective: Supercooled liquids and glasses, The Journal of Chemical Physics 137, 080901 (2012).
  * Berthier and Biroli [2011] L. Berthier and G. Biroli, Theoretical perspective on the glass transition and amorphous materials, Reviews of Modern Physics 83, 587 (2011).
  * Ediger [2000] M. D. Ediger, Spatially heterogeneous dynamics in supercooled liquids, Annual Review of Physical Chemistry 51, 99 (2000).
  * Faupel  _et al._ [2003] F. Faupel, W. Frank, M.-P. Macht, H. Mehrer, V. Naundorf, K. Rätzke, H. R. Schober, S. K. Sharma, and H. Teichler, Diffusion in metallic glasses and supercooled melts, Reviews of Modern Physics 75, 237 (2003).
  * Zhang and Schweizer [2017] R. Zhang and K. S. Schweizer, Correlated matrix-fluctuation-mediated activated transport of dilute penetrants in glass-forming liquids and suspensions, The Journal of Chemical Physics 146, 194906 (2017).
  * Mei and Schweizer [2021] B. Mei and K. S. Schweizer, Activated penetrant dynamics in glass forming liquids: size effects, decoupling, slaving, collective elasticity and correlation with matrix compressibility, Soft Matter 17, 2624 (2021).
  * Mutneja and Karmakar [2021a] A. Mutneja and S. Karmakar, Probing dynamic heterogeneity and amorphous order using rotational dynamics of rodlike particles in supercooled liquids, Physical Review Applied 16, 10.1103/physrevapplied.16.034022 (2021a).
  * Mutneja and Karmakar [2021b] A. Mutneja and S. Karmakar, Translational dynamics of a rod-like probe in supercooled liquids: an experimentally realizable method to study stokes–einstein breakdown, dynamic heterogeneity, and amorphous order, Soft Matter 17, 5738 (2021b).
  * Karmakar  _et al._ [2014] S. Karmakar, C. Dasgupta, and S. Sastry, Growing length scales and their relation to timescales in glass-forming liquids, Annual Review of Condensed Matter Physics 5, 255 (2014).
  * Karmakar  _et al._ [2009] S. Karmakar, C. Dasgupta, and S. Sastry, Growing length and time scales in glass-forming liquids, Proceedings of the National Academy of Sciences 106, 3675 (2009), https://www.pnas.org/content/106/10/3675.full.pdf .
  * Karmakar  _et al._ [2015] S. Karmakar, C. Dasgupta, and S. Sastry, Length scales in glass-forming liquids and related systems: a review, Reports on Progress in Physics 79, 016601 (2015).
  * Tah  _et al._ [2021] I. Tah, A. Mutneja, and S. Karmakar, Understanding slow and heterogeneous dynamics in model supercooled glass-forming liquids, ACS Omega 6, 7229 (2021).
  * Schweizer [2005] K. S. Schweizer, Derivation of a microscopic theory of barriers and activated hopping transport in glassy liquids and suspensions, The Journal of Chemical Physics 123, 244501 (2005).
  * Reichman and Charbonneau [2005] D. R. Reichman and P. Charbonneau, Mode-coupling theory, Journal of Statistical Mechanics: Theory and Experiment 2005, P05013 (2005).
  * Berthier and Tarjus [2010] L. Berthier and G. Tarjus, Critical test of the mode-coupling theory of the glass transition, Phys. Rev. E 82, 031502 (2010).
  * Kirkpatrick  _et al._ [1989] T. R. Kirkpatrick, D. Thirumalai, and P. G. Wolynes, Scaling concepts for the dynamics of viscous liquids near an ideal glassy state, Phys. Rev. A 40, 1045 (1989).
  * Lubchenko and Wolynes [2007] V. Lubchenko and P. G. Wolynes, Theory of structural glasses and supercooled liquids, Annual Review of Physical Chemistry 58, 235 (2007).
  * Chakrabarty  _et al._ [2017] S. Chakrabarty, I. Tah, S. Karmakar, and C. Dasgupta, Block analysis for the calculation of dynamic and static length scales in glass-forming liquids, Phys. Rev. Lett. 119, 205502 (2017).
  * Paul  _et al._ [2021a] K. Paul, S. K. Nandi, and S. Karmakar, Dynamic heterogeneity in active glass-forming liquids is qualitatively different compared to its equilibrium behaviour (2021a).
  * Paul  _et al._ [2021b] K. Paul, S. K. Nandi, and S. Karmakar, Non-trivial activity dependence of static length scale and critical tests of active random first-order transition theory (2021b).
  * Janssen [2019] L. M. C. Janssen, Active glasses, J. Phys.: Condens. Matter 31, 503002 (2019).
  * Narayan  _et al._ [2007] V. Narayan, S. Ramaswamy, and N. Menon, Long-lived giant number fluctuations in a swarming granular nematic, Science 317, 105 (2007).
  * Cugliandolo  _et al._ [2019] L. F. Cugliandolo, G. Gonnella, and I. Petrelli, Effective temperature in active brownian particles, Fluctuation and Noise Lett. 18, 1940008 (2019).
  * Caprini  _et al._ [2020] L. Caprini, U. M. B. Marconi, C. Maggi, M. Paoluzzi, and A. Puglisi, Hidden velocity ordering in dense suspensions of self-propelled disks, Phys. Rev. Res. 2, 023321 (2020).
  * Merrigan  _et al._ [2020] C. Merrigan, K. Ramola, R. Chatterjee, N. Segall, Y. Shokef, and B. Chakraborty, Arrested states in persistent active matter: Gelation without attraction, Phys. Rev. Res. 2, 013260 (2020).
  * Chaki and Chakrabarty [2020] S. Chaki and R. Chakrabarty, Escape of a passive particle from an activity-induced energy landscape: emergence of slow and fast effective diffusion, Soft Matter 16, 7103 (2020).
  * Nandi  _et al._ [2018] S. K. Nandi, R. Mandal, P. J. Bhuyan, C. Dasgupta, M. Rao, and N. S. Gov, A random first-order transition theory for an active glass, Proc. Natl. Acad. Sci. (USA) 115, 7688 (2018).
  * Szamel [2016] G. Szamel, Theory for the dynamics of dense systems of athermal self-propelled particles, Phys. Rev. E 93, 012603 (2016).
  * Nandi [2018] S. K. Nandi, Activity-dependent self-regulation of viscous length scales in biological systems, Phys. Rev. E 97, 052404 (2018).
  * Nandi and Gov [2017] S. K. Nandi and N. S. Gov, Nonequilibrium mode-coupling theory for dense active systems of self-propelled particles, Soft Matter 13, 7609 (2017).
  * Berthier and Kurchan [2013] L. Berthier and J. Kurchan, Non-equilibrium glass transitions in driven and active matter, Nat. Phys. 9, 310 (2013).
  * Malinverno  _et al._ [2017] C. Malinverno, S. Corallino, F. Giavazzi, M. Bergert, Q. Li, M. Leoni, A. Disanza, E. Frittoli, A. Oldani, E. Martini, T. Lendenmann, G. Deflorian, G. V. Beznoussenko, D. Poulikakos, K. H. Ong, M. Uroz, X. Trepat, D. Parazzoli, P. Maiuri, W. Yu, A. Ferrari, R. Cerbino, and G. Scita, Endocytic reawakening of motility in jammed epithelia, Nature Materials 16, 587 (2017).
  * Cerbino  _et al._ [2021] R. Cerbino, S. Villa, A. Palamidessi, E. Frittoli, G. Scita, and F. Giavazzi, Disentangling collective motion and local rearrangements in 2d and 3d cell assemblies, Soft Matter 17, 3550 (2021).
  * Avila  _et al._ [2014] K. E. Avila, H. E. Castillo, A. Fiege, K. Vollmayr-Lee, and A. Zippelius, Strong dynamical heterogeneity and universal scaling in driven granular fluids, Phys. Rev. Lett. 113, 025701 (2014).
  * Giuliano  _et al._ [2018] M. Giuliano, A. Shaikh, H. C. Lo, G. Arpino, S. D. Placido, X. H. Zhang, M. Cristofanilli, R. Schiff, and M. V. Trivedi, Perspective on circulating tumor cell clusters: Why it takes a village to metastasize, Cancer Research 78, 845 (2018).
  * Palamidessi  _et al._ [2019] A. Palamidessi, C. Malinverno, E. Frittoli, S. Corallino, E. Barbieri, S. Sigismund, G. V. Beznoussenko, E. Martini, M. Garre, I. Ferrara, C. Tripodo, F. Ascione, E. A. Cavalcanti-Adam, Q. Li, P. P. D. Fiore, D. Parazzoli, F. Giavazzi, R. Cerbino, and G. Scita, Unjamming overcomes kinetic and proliferation arrest in terminally differentiated cells and promotes collective motility of carcinoma, Nature Materials 18, 1252 (2019).
  * Kob and Andersen [1995] W. Kob and H. C. Andersen, Testing mode-coupling theory for a supercooled binary lennard-jones mixture i: The van hove correlation function, Phys. Rev. E 51, 4626 (1995).
  * Mandal  _et al._ [2016] R. Mandal, P. J. Bhuyan, M. Rao, and C. Dasgupta, Active fluidization in dense glassy systems, Soft Matter 12, 6268 (2016).
  * Keta  _et al._ [2022] Y.-E. Keta, R. L. Jack, and L. Berthier, Disordered collective motion in dense assemblies of persistent particles, Phys. Rev. Lett. 129, 048002 (2022).
  * Mutneja and Karmakar [2022] A. Mutneja and S. Karmakar, A novel method to probe the pronounced growth of correlation lengths in an active glass-forming liquids using elongated probe (2022).
  * Bhowmik  _et al._ [2018] B. P. Bhowmik, I. Tah, and S. Karmakar, Non-gaussianity of the van hove function and dynamic-heterogeneity length scale, Physical Review E 98, 10.1103/physreve.98.022122 (2018).
  * Mandal  _et al._ [2020] R. Mandal, P. J. Bhuyan, P. Chaudhuri, C. Dasgupta, and M. Rao, Extreme active matter at high densities, Nature Communications 11, 10.1038/s41467-020-16130-x (2020).