# Privacy-Preserving Pose Estimation for Human-Robot Interaction

Youya Xia${}^{1^{*}}$, Yifan Tang${}^{1^{*}}$, Yuhan Hu1 and Guy Hoffman1
*These authors contribute equally to the work.1Youya Xia, Yifan Tang, Yuhan Hu
and Guy Hoffman are with Cornell University

###### Abstract

Pose estimation is an important technique for nonverbal human-robot
interaction. That said, the presence of a camera in a person’s space raises
privacy concerns and could lead to distrust of the robot. In this paper, we
propose a privacy-preserving camera-based pose estimation method. The proposed
system consists of a user-controlled translucent filter that covers the camera
and an image enhancement module designed to facilitate pose estimation from
the filtered (shadow) images, while never capturing clear images of the user.
We evaluate the system’s performance on a new filtered image dataset,
considering the effects of distance from the camera, background clutter, and
film thickness. Based on our findings, we conclude that our system can protect
humans’ privacy while detecting humans’ pose information effectively.

##  I Introduction

In this paper, we describe a method for privacy-preserving visual human-robot
interaction (HRI). Our approach rests on effective human pose inference from a
user-degraded camera. This would allow a user to cover the robot’s camera with
a translucent film but still interact with the robot using nonverbal
behaviors.

Visual observation of humans plays a vital role in understanding humans’
states and intentions for HRI [12]. However, being monitored by cameras,
especially at home, leads to privacy concerns [3]. Prior work on protecting
humans’ privacy in captured images focuses on using depth cameras or specially
designed low-resolution cameras [32, 23]. However, these methods require
specialized equipment. In contrast, we propose to improve pose inference in
images captured from a standard RGB camera which has been covered by a
translucent film. The main contribution of this work is an image enhancement
method that maintains privacy but enables robust pose estimation.

Pose estimation on clear images has been studied extensively [14, 29], but
conducting pose estimation directly on degraded images is challenging.
Researchers have proposed a number of image enhancement methods to facilitate
pose estimation for degraded images [17, 19]. However, these methods are
generally based on a specific image formation model, such as low light or
haze. We found that the privacy-preserving images in our case do not comply
with existing models. Therefore, the existing models cannot solve the pose
estimation problems for our purpose (see Section VI-D for a detailed
evaluation).

Figure 1: Left to right: the pose estimation results on the ground-truth clear
image, the corresponding shadow image and enhanced image output by our
enhancement module.

Our method is building on the recommendation put forth in Hu et al. [15],
which proposed privacy-preserving HRI with cameras covered up by filtering
materials. To detect humans’ poses from the degraded images, we propose a
neural network based architecture for image enhancement. The proposed network
is trained to produce an enhanced version of the input image. Afterwards, the
enhanced image is passed into OpenPose [4] to obtain the final estimated poses
(Figure 1). Experimental results indicate that our system can obtain human
pose information effectively.

The paper thus makes the following contributions:

  * •

A design for camera-based human-robot interaction, which can protect humans’
privacy while obtaining their pose information effectively;

  * •

A neural network architecture designed for image enhancement specifically
aimed at pose estimation from translucent-film-filtered images111The training
and testing code is released at:
https://github.com/xiaxx244/shadow_pose_estimation;

  * •

An extensive quantitative evaluation of the proposed system providing insights
on the effects of distance, background clutter, and film thickness.

##  II Related Work

In this section, we review prior works on privacy-enhanced HRI and on image
enhancement for pose estimation.

###  II-A Privacy Protection in HRI

The camera used in many robots as an input to monitor a user’s state or
intentions carries with it privacy concerns, especially in personal spaces
such as the home. Researchers have proposed privacy-protecting strategies,
such as blur filtration of all or part of the image [8, 24]. As these methods
aim to protect humans’ privacy through processing already-taken video data,
they still allow robots to capture high-fidelity visual data.

Users often take control of their camera privacy by physically obstructing the
lens of a camera placed in a private space. Building on this practice, Hu et
al. propose a privacy-maintaining possibility for camera-based interaction
with social robots [15]. The authors suggest that through physically covering
the robot’s cameras with a translucent material, a robot can still exploit
some interaction data in the form of users’ shadows instead of high-fidelity
images. The paper, however, only described a concept sketch of this idea,
without proposing algorithms for inferring users’ poses from full-body shadow
images. Establishing an effective design for such an algorithm is the aim of
the current work, and at its core lies an appropriate image enhancement
module.

###  II-B Image Enhancement for Pose Estimation

Image enhancement is an active field of computer vision, with existing methods
broadly falling into two categories: single-image dehazing and low-light image
enhancement.

#### Single-image Dehazing

Haze is a common type of image degradation, leading to two kinds of work
related to single-image dehazing. The first line of work focuses on estimating
transmission maps and atmospheric light. It then uses this atmospheric
scattering model to remove haze. For example, Li et al. (2017) provides an
end-to-end network that directly estimates the combination of atmospheric
light and transmission map [17]. The other line of work focuses on learning a
mapping from a hazy image to a haze-free image. For instance, Li et al. (2020)
proposes a convolutional neural network (CNN) to learn a residual between hazy
underwater images and haze-free images [19]. But, both lines of work do not
have robust performances when images contain non-uniform light or heavy haze
[27, 28].

#### Low-Light Image Enhancement

Image generated from low-light circumstances is another type of image
degradation. The early literature about low-light image enhancement focuses on
contrast enhancement [25, 6, 1, 5]. However, the early literature’s enhanced
images suffer from problems such as image distortion and artificial
illumination. State-of-the-art methodologies [33, 10, 22, 11] have improved
the naturalness of illumination and the realism of images significantly.
Nevertheless, they still encounter failures of enhancement in regions with
complex textures.

Both lines of state-of-the-art image enhancement methodologies focus on
degraded images with explicit formation models. Since there is no such
formation model for images generated with a film-covered camera, implementing
the state-of-the-art methodologies in our method is not a promising avenue. In
Section V we explicitly evaluate the use of existing image enhancement methods
in comparison with the method propose in this paper.

##  III System

This section describes the full privacy-preserving pose estimation pipeline,
which includes the privacy-preserving camera setup, the proposed network for
image enhancement module, and the pose estimation module.

###  III-A Privacy-preserving camera setups

Based on the prior work of ShadowSense [15], in the proposed system, the
robots’ eyes are covered with a translucent material which allows it to
capture users’ full-body visual data in the form of their shadows. This is a
solution that can easily be implemented by the end-user and leaves them in
full control of the image captured by the robot.

While adding privacy, conducting pose estimation directly on the defined
shadow images is challenging (see Section V for detailed performance
evaluation of pose estimation on these images). To counter this issue, we
present an image enhancement module that can produce an enhanced version of a
shadow image tailored to human pose estimation.

Figure 2: The general structure of the MiniRes-based architecture. It is
composed of an input layer with size $256\times 256\times 3$ and three
Enhancement Modules (EM), each of which has a respective optimization
objective (i.e., structural loss, perceptual loss and edge loss). The second
and the third modules are connected with the input layer via a short-cut
respectively. Figure 3: The architecture of each Enhancement Module (EM). It
consists of 3 MiniRes blocks with a short-cut from the first MiniRes block to
the third MiniRes block.

###  III-B MiniRes-based architecture for Image Enhancement

We propose a design for deep neural network (Figure 2), composed of an input
layer with size $256\times 256\times 3$ and three separate “Enhancement
Modules” (EM). Each EM is tuned to a specific optimization objective which is
important to pose estimation.

The first enhancement module aims to restore the shadow image’s structural
information so that the enhanced image has higher structural similarity to the
ground-truth clear image. In the second enhancement module, the optimization
objective is to restore the shadow image’s perceptual features so that the
enhanced image has a smaller distance to the ground-truth image in visual
space. The final enhancement module is supposed to restore the edge
information of the shadow image so that the enhanced image preserves the edge
information of the ground-truth image. The three EMs are connected in
sequential order, and the output of our network is the output from the third
enhancement module. Moreover, to avoid classical degradation and vanishing
gradient problems encountered by deep neural networks, we add a shortcut from
the input layer to the second and the third enhancement modules, respectively.

Inside each of the enhancement modules (Figure 3), the input is connected to a
3D Convolution layer whose dimension is $3\times 3\times 32$ and a MaxPooling
layer whose filter size is $3\times 3$. We then use three sequential copies of
a custom-designed component which is based on the ResBlock structure in ResNet
[13], an efficient network structure for various challenging vision tasks. We
call this comoponent MiniRes.

MiniRes blocks are different from original ResBlocks in two ways. First, to
allow for the variance of shadow images created by different filters, we
remove the two batch normalization layers from the original ResBlock
architecture. Second, to speed up the inference process, we reduce the
dimension of two 3D convolution layers from $3\times 3\times 64$ to $3\times
3\times 32$.

In addition, in each EM, a shortcut is added from the first to the third
MiniRes Block to preserve information about the image’s content during
optimization process. Finally, we connect the third MiniRes block to a 3D
convolution layer with dimension $3\times 3\times 32$ to produce the output of
size $256\times 256\times 3$ for each enhancement module.

###  III-C Loss Function Formulation

Each EM uses a module-specific loss function, with the total loss function
defined as

$\mathcal{L}=\mathcal{L}_{SL}+\mathcal{L}_{PL}+\mathcal{L}_{EL}$ (1)

Here, $\mathcal{L}_{SL}$ represents the structural loss, $\mathcal{L}_{PL}$
represents the perceptual loss and $\mathcal{L}_{EL}$ represents the edge
loss, mapping to the three optimization objectives listed above.

  * •

The structural loss $\mathcal{L}_{SL}$ aims to restore the structural
information of the shadow image’s content. To do so, we use the Structural
Similarity Index Measure (SSIM) [34], which has shown success for repairing an
image’s structural degradation [22, 20]. For each pixel $x$, SSIM is
calculated within a window size $11\times 11$ around $x$ as follows:

$\text{SSIM}(x):=\frac{(2\mu_{e}(x)\mu_{c}(x)+d_{1})(2\sigma_{e*c}(x)+d_{2})}{(\mu_{e}^{2}(x)+\mu_{c}^{2}(x)+d_{1})(\sigma_{e}^{2}(x)+\sigma_{c}^{2}(x)+d_{2})}$
(2)

Here, $e$ and $c$ correspond to the network’s latent image and the paired
ground-truth clear image. $\mu,\sigma^{2}$ correspond to expectation and
variance of pixel values lying in each image window, and $\sigma_{e*c}^{2}$
corresponds to the the pixels’ covariance between paired windows in $e$ and
$c$. $d_{1}$ and $d_{2}$ are regulating constants which are set to $0.0001$
and $0.0009$ respectively. Given this definition, $\mathcal{L}_{SL}$ is
defined as:

$\mathcal{L}_{SL}:=1-\frac{1}{3M}\sum_{i=1}^{M}(\text{SSIM}_{r}(e_{i})+\text{SSIM}_{g}(e_{i})+\text{SSIM}_{b}(e_{i}))$
(3)

where $M$ is the number of pixels in the network’s latent image $e$ and
$\text{SSIM}_{h}(e_{i})$ means for each image channel $h\in\\{r,g,b\\}$, the
SSIM value at pixel $i$ inside $e$.

  * •

The perceptual Loss $\mathcal{L}_{PL}$ ensures that the network’s output image
is similar to the clear image in visual space.

$\mathcal{L}_{PL}:=\mathcal{L}_{\text{MSE}}+2\mathcal{L}_{\text{MAE}}+\mathcal{L}_{\textit{ResNet}}$
(4)

We use a mixture of the Mean Absolute Error (MAE) (the $L_{1}$ norm) and the
Mean Squared Error (MSE) (the $L_{2}$ norm), as the mixture has been proved to
be a more robust metric for comparing global similarity in pixel space than
using either norm alone [9]. In addition, because ResNet has already been
shown to be a robust feature extractor for high-level vision tasks, we use a
feature extractor $\phi$ based on the top 6 layers of ResNet-50 [13]
pretrained on ImageNet [7]. Then we compute the $L_{2}$ distance between the
network’s latent image $e$ and the ground-truth image $c$ in the ResNet
feature space:

$\mathcal{L}_{\textit{ResNet}}:=||\phi(e)-\phi(c)||_{2}$ (5)

  * •

Finally, the edge loss $\mathcal{L}_{EL}$ aims to restore the edge information
of the ground-truth image. Specifically, we use an edge map extractor $\Omega$
which extracts the Sobel Edge [30] maps of the network’s latent image $e$ and
the corresponding ground-truth image $c$. Then, we computed the $L_{2}$
distance of those extracted maps as follows:

$\mathcal{L}_{EL}:=||\Omega(e)-\Omega(c)||_{2}$ (6)

###  III-D Network Training

There exists no large-scale image dataset with film-filtered camera images.
Based on prior literature of image enhancement [17, 19], training only on the
limited number of shadow images that we would be able to collect does not
produce robust performances. We therefore chose to train the proposed image
enhancement module on the Outdoot Training Set (OTS) [18]. The OTS is a large-
scale synthetic dataset consisting of degraded images with different degrees
of degradation. We train our proposed MiniRes-based network on the OTS using
two NVIDIATM RTX 2080 Ti cards.

###  III-E Pose Estimation Module

After the image enhancement module, the proposed system uses a pose estimation
module to produce the final estimated poses in the enhanced image. Although
various solutions for human body-pose detection have been proposed such as
FastPose [31] and other skeleton trackers [16], OpenPose is established as the
state-of-the-art [4]. In the last part of the proposed pipeline, we use
OpenPose, as is, on the enhanced images as its inputs, to extract points lying
inside humans’ bodies.

Figure 4: The figure shows the system when we put $1$ layer of the plastic
mask (i.e., one of our chosen filter layers) on top of the left camera.

##  IV Data Collection

To test our image enhancement method, we collected a “shadow image” dataset
using a binocular camera system (Figure 4), covering up one of the two cameras
when capturing an image. Using a binocular camera ensured that the ground-
truth image and the paired shadow image include the same content. To reduce
the impact of binocular disparity [26], we ensure that the distance between
humans and the camera during the data collection process is at least $2$m.

Figure 5: The shadow images created by our chosen filter layers. Left to
right: shadow images created by $1$, $2$ and $3$ filter layers respectively.

Then, to create the shadow effect in one of the two cameras in the system, we
used a freezer bag as the “base” filter. To evaluate the proposed system with
different levels of privacy protection, we use $1$, $2$, & $3$ layers of the
freezer bag. Figure 5 shows the shadow effects created by selected layers. To
test our proposed system’s robustness, we conducted the experiments under
different background complexities (i.e., plain backgrounds, and complex
backgrounds). Environments containing less than 4 objects are defined as plain
backgrounds, denoted “private” spaces below. Similarly, we define environments
containing more than 3 objects as complex backgrounds and we call them
“public” spaces. In addition, since OpenPose uses a bottom-up approach for
pose estimation, the farther distance between humans and cameras causes more
challenges for the body part detection branch in its architecture. Aiming to
understand the effectiveness of the MiniRes-based network for enhancing
details (i.e., smaller body parts under farther distance) in the shadow image,
we also conduct experiments under different proximity (i.e., close & far).
Finally, to make sure there are no gender biases in our dataset, we collected
images with both a female and a male actor.

We asked the two actors to conduct 3 activities (reading, exercising, and
drinking) in both public and private spaces. The actors were also requested to
repeat the activities with different distances ($2$ & $4$ meters) to the
camera system. In total, we have collected $72$ paired datasets in public and
private spaces which are composed of $99,450$ paired images. In each type of
spaces, we have collected $6$ paired datasets for each activity under close
and far distances respectively.

##  V Experiments and Metrics

This section conducts a quantitative evaluation of the proposed system, using
the collected test shadow image dataset. We first evaluate the effectiveness
of chosen filter layers, the robustness of the proposed image enhancement
module quantitatively. Based on the proposed system’s performances, we then
offer future researchers potential directions about how to choose suitable
filters and proximity for setting up a privacy-preserving camera system for a
social robot.

###  V-A Filter Layers: Effectiveness Evaluation of Privacy Protection

To provide a quantitative measure of the filter layers’ effectiveness for
privacy protection, we use an established measure of image quality and
estimate the subsequent degradation. Spatial–Spectral Entropy-based Quality
(SSEQ) [21] is a comprehensive metric for measuring image quality based on
images’ visual and structural information without using ground-truth images as
references.

| $\text{SSEQ}_{\text{Clear}}$ | $\text{SSEQ}_{\text{Shadow}}$ | SR
---|---|---|---
1 Layer | 42.7710 | 52.6125 | 0.2301
2 Layers | 42.7596 | 58.4768 | 0.3676
3 Layers | 39.0658 | 57.7562 | 0.4784
TABLE I: The SSEQ scores for our chosen filter layers in both private and
public spaces. We also include the Shadow Ratio scores in the table to
demonstrate the each filter layers’ ability to protect humans’ identities.

Table I shows the mean SSEQ score for clear and shadow dataset, as well as a
Shadow Ratio (SR) defined as

$\text{SR}:=\frac{\text{SSEQ}_{\text{Shadow}}-\text{SSEQ}_{\text{Clear}}}{\text{SSEQ}_{\text{Clear}}}$
(7)

Table I suggests that a single layer of film has the lowest image degradation,
and this the lowest potential for privacy protection, whereas three layers of
film has the strongest degradation and thus the most potential for privacy
protection. This can also be seen qualitatively in the illustrative images in
Figure 5.

###  V-B Image Enhancement Metrics

The proposed image enhancement module aims to facilitate the precision of pose
estimation. So to evaluate the robustness of the trained MiniRes-based
network, we propose two evaluation metrics based on the pose estimation
results we obtain from the OpenPose module.

#### Detection

The first metric we propose is Detection Rate,
$\text{DR}:=\frac{N_{e}}{N_{c}}$. Here, $N_{e}$ stands for the detected
skeleton key points in the shadow images or enhanced images, while $N_{c}$
stands for the detected key points in the paired ground-truth clear images.
This metric only measures the number of detected features, without taking into
account their accuracy.

#### Precision

Inspired by the mean Average Precision metric on pose estimation [2], we also
use a second metric, the Shadow mean Average Precision,
$\text{SmAP}:=\frac{N_{te}}{N_{e}}$ Here, $N_{te}$ stands for the number of
the precisely detected key points in the shadow or enhanced images.
Specifically, for each detected keypoint $k$ in a shadow or enhanced image, we
compare its location with the keypoint’s location associated with the same
body part in the paired clear image and calculate their $L_{2}$ distance as
$L_{ec}$. To count for the precision errors arising from binocular disparity
which, in our dataset, is around $10$ pixels, we regard $k\in N_{te}$ if
$L_{ec}\leq 10$ pixels.

We will compare both metrics across filter layers, distance from camera, and
background clutter.

##  VI Results

We evaluate the robustness of the trained network on the test dataset
collected in public and private spaces using the proposed metrics (See our
attached video for more experiments’ examples).

###  VI-A Private Space

We evaluate the effectiveness of our method for two different proximity
settings and three layer settings. Figure 6 shows the mean DR and SmAP scores
for each chosen filter layers on the private space dataset (low background
clutter). Comparing scores for the original filtered (Shadow) and the enhanced
images, we can see that the detection rate for shadow images degrades with
increasing filter layers. SmAP (precision) is very low even with a single
filter layer. In comparison, the enhanced images maintain high detection and
accuracy scores even with increasing layers. Figure 7 shows an example image,
comparing the pose estimation on the clear, filtered, and enhanced versions of
the same image.

Figure 6: The Evaluation histograms in private spaces. Top to Bottom:
Detection Rate and precision for different filter layers on shadow and
enhanced images when the distance between the participants and the camera is
$2$ and $4$ meters respectively. Figure 7: Private space evaluation: the
female participant sits $4$ meters away from the camera system covered with
$2$ filter layers while reading in her bedroom. The images from left to right
show the pose estimation results on the ground-truth clear image, the paired
shadow image and enhanced image respectively.

Besides, from the histograms in Figure 6, we find that proximity does not have
a consistent effect on the shadow images’ pose estimation results. Both
detection rates and precision are similar for people standing closer and
farther from the camera.

###  VI-B Public Space

Figure 8 shows an example of an image taken in the public (cluttered)
environment, demonstrating the pose estimation performance on the clear,
filtered, and enhanced versions of the same image.

Figure 8: Public space evaluation: the male participant sits $2$ meters away
from the camera system covered with $3$ filter layers while drinking in a
cafe. The images from left to right show the pose estimation results on the
ground-truth clear image, the paired shadow image and enhanced image
respectively.

Figure 9: The Evaluation histograms in public spaces. Top to Bottom: Detection
Rate and precision for different filter layers on shadow and enhanced images
when the distance between the participants and the camera is $2$ and $4$
meters respectively.

The results, seen in Figure 9, indicate a similar trend as in the private
spaces. That said, un-enhanced shadow images degrade faster in public spaces,
with loss of detection rates happening with $2$ layers of filtering. In
addition, compared with private spaces, for the same filter layers and
proximity, the percentage of precisely detected key points on enhanced images
in public spaces is smaller than the percentage in private spaces. Therefore,
the public spaces are more challenging for image enhancement than private
spaces.

###  VI-C Ablation Study

To evaluate each component’s importance in our defined loss function, we
conduct an ablation study for our proposed network’s loss function. During the
ablation study, to make more obvious distinctions of our network’s
performances between original and modified loss functions, we choose datasets
created from $2$ filter layers and $3$ filter layers. Specifically, for each
type of spaces, we choose $3610$ paired images generated from $3$ filter
layers and $3956$ paired images generated from $2$ filter layers. Then by
changing the optimization objective in the enhancement module previously
associated with $\mathcal{L}_{SL}$ to $\mathcal{L}_{PL}+\mathcal{L}_{EL}$, we
remove the structural loss from our optimization objective and retrain the
network from scratch. Similarly, we train our network by removing the
perceptual loss and edge loss respectively. In the ablation study table we
list below, we use $i_{o}$ to represent the original enhanced image output by
the proposed network for $i$ filter layers with $i\in\\{1,2,3\\}$, $i_{g}$,
$i_{p}$ and $i_{t}$ to represent the enhanced image we obtain by removing one
of the edge loss, perceptual loss, structural loss respectively.

$\text{Filler}_{\text{Loss}}$/Metric $2_{o}$ $2_{t}$ $2_{g}$ $2_{p}$ $3_{o}$
$3_{t}$ $3_{g}$ $3_{p}$ DR 0.959 0.841 0.944 0.557 0.954 0.846 0.920 0.693
SmAP 0.901 0.748 0.895 0.633 0.887 0.758 0.838 0.597

TABLE II: The ablation study table we obtain by removing one of the perceptual
loss, structural loss and edge loss.

We can infer two implications from the ablation study table (Table II).  1.
The perceptual loss is more important than edge loss and structural loss for
the performance of our proposed network since $i_{p}$ suffers from the
greatest decline in DR and SmAP scores compared with $i_{o}$, with edge loss
seemingly being the least contributing factor;  2. Since scores for the
original enhanced images are higher than the enhanced images we obtain by
removing structural loss, perceptual loss, and edge loss, respectively, the
three components in our loss functions contribute to the network’s
optimization process.

###  VI-D Comparison to Existing Image Enhancement Methods

There is no existing method proposed for boosting pose estimation’s
performances on the type of shadow images we are interested in this work. That
said, given the similarity of shadow images to hazy and low-light images in
feature space, we evaluate the performances of two state-of-the-art methods
focused on enhancing hazy and low-light images. Specifically, we choose AODNet
[17], which focuses on hazy image enhancement, and MBLLEN [22], which focuses
on low-light image enhancement for comparison study. Then we use the same
paired images as the ablation study. Finally, we evaluate the pose estimation
results (i.e., DR, and SmAP scores) of the enhanced images generated from the
two state-of-the-art methods using OpenPose.

Table III shows the performance of our method ($i_{e}$) for $i$ layers of
filters, $i\in\\{1,2,3\\}$. $i_{m}$ indicated the output image that MBLLEN
produces and $i_{a}$ the output image that AODNet obtains. We can conclude
that our proposed network boosts pose estimation on collected shadow images
more effectively than AODNet and MBLLEN.

$\text{Layer}_{\text{Methods}}$/Metric $2_{e}$ $2_{m}$ $2_{a}$ $3_{e}$ $3_{m}$
$3_{a}$ DR 0.959 0.605 0.730 0.954 0.531 0.582 SmAP 0.901 0.069 0.049 0.887
0.044 0.062

TABLE III: The evaluation table which shows the performances of AODNet and
MBLLEN on the selected shadow image dataset.

##  VII Discussion

Based on our evaluation results, we offer future researchers potential
directions about configuring the privacy-protected camera system.

  * •

Proximity: From the four evaluation histograms in Figure 6 and 9, we conclude
that the system can protect humans’ privacy and have robust performances on
pose estimation when the distance between humans and camera system is between
$2$ and $4$ meters. Therefore, we suggest that future designers can choose the
proximity within range $[2,4]$. Also, they are encouraged to explore a more
suitable distance between humans and the system to balance the performances on
privacy protection and pose estimation for their purposes;

  * •

Filters: According to Table I and the four histograms, we can conclude that
$1$ filter layer is more optimal for pose estimation, while $3$ filter layers
are more optimal for privacy protection. $2$ filter layers, on the other hand,
has the medium score of shadow ratio and can be potentially chosen for
balancing between privacy protection and pose estimation. Therefore, based on
the SR value of Table I, we propose that designers should choose filters with
$\text{SR}\in[0.36,0.45]$ which is the SR value between the $2$ and $3$ filter
layers.

##  VIII Conclusion

This paper proposes a camera system that can protect humans’ privacy while
obtaining their pose information effectively. It is based on an image
enhancement neural network that uses structural, perceptual, and edge
information and is trained on an off-context dataset. We evaluate the filter
layers’ privacy protection ability and our proposed network’s robustness and
find that our method regains the possibility for pose estimation even with a
privacy filter. This can provide a useful solution for users wanting to cover
a robot’s camera with a physical filter, but still interact with it. Finally,
we offer future researchers potential directions about suitable filter choices
and proximity between the camera system and humans. In future, we hope to
incorporate social factors into our system design consideration by conducting
human studies.

## References

  * [1] Mohammad Abdullah-Al-Wadud, Md Hasanul Kabir, M Ali Akber Dewan, and Oksam Chae.  A dynamic histogram equalization for image contrast enhancement.  IEEE Transactions on Consumer Electronics, 53(2):593–600, 2007\.
  * [2] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele.  2d human pose estimation: New benchmark and state of the art analysis.  In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pages 3686–3693, 2014.
  * [3] Kelly Caine, Selma Šabanovic, and Mary Carter.  The effect of monitoring by cameras and robots on the privacy enhancing behaviors of older adults.  In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction, pages 343–350, 2012.
  * [4] Zhe Cao, Gines Hidalgo Martinez, Tomas Simon, Shih-En Wei, and Yaser A Sheikh.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [5] Turgay Celik and Tardi Tjahjadi.  Contextual and variational contrast enhancement.  IEEE Transactions on Image Processing, 20(12):3431–3441, 2011.
  * [6] HD Cheng and XJ Shi.  A simple and effective histogram equalization approach to image enhancement.  Digital signal processing, 14(2):158–170, 2004.
  * [7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [8] Jianping Fan, Hangzai Luo, Mohand-Said Hacid, and Elisa Bertino.  A novel approach for privacy-preserving video sharing.  In Proceedings of the 14th ACM international conference on Information and knowledge management, pages 609–616, 2005.
  * [9] Haoying Fu, Michael K Ng, Mila Nikolova, Jesse Barlow, and Wai-ki Ching.  Fast algorithms for l1 norm/mixed l1 and l2 norms for image restoration.  In International Conference on Computational Science and Its Applications, pages 843–851. Springer, 2005.
  * [10] Xueyang Fu, Delu Zeng, Yue Huang, Yinghao Liao, Xinghao Ding, and John Paisley.  A fusion-based enhancing method for weakly illuminated images.  Signal Processing, 129:82–96, 2016.
  * [11] Xueyang Fu, Delu Zeng, Yue Huang, Xiao-Ping Zhang, and Xinghao Ding.  A weighted variational model for simultaneous reflectance and illumination estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2782–2790, 2016.
  * [12] Michael A Goodrich and Alan C Schultz.  Human-robot interaction: a survey.  Now Publishers Inc, 2008.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [14] Samitha Herath, Mehrtash Harandi, and Fatih Porikli.  Going deeper into action recognition: A survey.  Image and vision computing, 60:4–21, 2017.
  * [15] Yuhan Hu, Sara Bejarano, and Guy Hoffman.  Shadowsense: Detecting human touch in a social robot through shadow image processing.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(1), 2020.
  * [16] Pushmeet Kohli and Jamie Shotton.  Key developments in human pose estimation for kinect.  In consumer depth cameras for computer vision, pages 63–70. Springer, 2013.
  * [17] Boyi Li, Xiulian Peng, Zhangyang Wang, Jizheng Xu, and Dan Feng.  AOD-net: All-in-one dehazing network.  In Proceedings of the IEEE international conference on computer vision, pages 4770–4778, 2017.
  * [18] Boyi Li, Wenqi Ren, Dengpan Fu, Dacheng Tao, Dan Feng, Wenjun Zeng, and Zhangyang Wang.  Benchmarking single-image dehazing and beyond.  IEEE Transactions on Image Processing, 28(1):492–505, 2018.
  * [19] Chongyi Li, Saeed Anwar, and Fatih Porikli.  Underwater scene prior inspired deep underwater image and video enhancement.  Pattern Recognition, 98:107038, 2020.
  * [20] Ruoteng Li, Loong-Fah Cheong, and Robby T Tan.  Heavy rain image restoration: Integrating physics model and conditional adversarial learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1633–1642, 2019.
  * [21] Lixiong Liu, Bao Liu, Hua Huang, and Alan Conrad Bovik.  No-reference image quality assessment based on spatial and spectral entropies.  Signal Processing: Image Communication, 29(8):856–863, 2014.
  * [22] Feifan Lv, Feng Lu, Jianhua Wu, and Chongsoon Lim.  Mbllen: Low-light image/video enhancement using cnns.  In BMVC, page 220, 2018.
  * [23] Lingjuan Lyu, Xuanli He, Yee Wei Law, and Marimuthu Palaniswami.  Privacy-preserving collaborative deep learning with application to human activity recognition.  In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 1219–1228, 2017.
  * [24] Carman Neustaedter, Saul Greenberg, and Michael Boyle.  Blur filtration fails to preserve privacy for home-based video conferencing.  ACM Transactions on Computer-Human Interaction (TOCHI), 13(1):1–36, 2006.
  * [25] Etta D Pisano, Shuquan Zong, Bradley M Hemminger, Marla DeLuca, R Eugene Johnston, Keith Muller, M Patricia Braeuning, and Stephen M Pizer.  Contrast limited adaptive histogram equalization image processing to improve the detection of simulated spiculations in dense mammograms.  Journal of Digital imaging, 11(4):193, 1998.
  * [26] Ning Qian.  Binocular disparity and the perception of depth.  Neuron, 18(3):359–368, 1997.
  * [27] Wenqi Ren, Si Liu, Hua Zhang, Jinshan Pan, Xiaochun Cao, and Ming-Hsuan Yang.  Single image dehazing via multi-scale convolutional neural networks.  In Computer Vision – ECCV 2016, pages 154–169. Springer International Publishing, 2016.
  * [28] Wenqi Ren, Jinshan Pan, Hua Zhang, Xiaochun Cao, and Ming-Hsuan Yang.  Single image dehazing via multi-scale convolutional neural networks with holistic edges.  International Journal of Computer Vision, 128(1):240–259, 2020\.
  * [29] Nikolaos Sarafianos, Bogdan Boteanu, Bogdan Ionescu, and Ioannis A Kakadiaris.  3d human pose estimation: A review of the literature and analysis of covariates.  Computer Vision and Image Understanding, 152:1–20, 2016.
  * [30] Robert J Schalkoff.  Digital image processing and computer vision, volume 286.  Wiley New York, 1989.
  * [31] Gregory Shakhnarovich, Paul Viola, and Trevor Darrell.  Fast pose estimation with parameter-sensitive hashing.  In null, page 750. IEEE, 2003.
  * [32] Vinkle Srivastav, Afshin Gangi, and Nicolas Padoy.  Human pose estimation on privacy-preserving low-resolution depth images.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 583–591. Springer, 2019.
  * [33] Shuhang Wang, Jin Zheng, Hai-Miao Hu, and Bo Li.  Naturalness preserved enhancement algorithm for non-uniform illumination images.  IEEE Transactions on Image Processing, 22(9):3538–3548, 2013.
  * [34] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.


# Cooperation in a fluid swarm of fuel-free micro-swimmers

Matan Yah Ben Zion<EMAIL_ADDRESS>Center for Soft Matter Research,
Department of Physics, New York University, 726 Broadway Avenue, New York, NY
10003, USA  UMR Gulliver 7083 CNRS, ESPCI Paris, PSL Research University, 10
rue Vauquelin, Paris 75005, France  Yaelin Caba  Center for Soft Matter
Research, Department of Physics, New York University, 726 Broadway Avenue, New
York, NY 10003, USA  Alvin Modin  Center for Soft Matter Research, Department
of Physics, New York University, 726 Broadway Avenue, New York, NY 10003, USA
Paul M Chaikin  Center for Soft Matter Research, Department of Physics, New
York University, 726 Broadway Avenue, New York, NY 10003, USA

Cooperation is vital for the survival of a swarm 1. Large scale cooperation
allows murmuring starlings to outmaneuver preying falcons 2, shoaling sardines
to outsmart sea lions 3, and homosapiens to outlive their Pleistocene peers 4.
On the micron-scale, bacterial colonies show excellent resilience thanks to
the individuals’ ability to cooperate even when densely packed, mitigating
their internal flow pattern to mix nutrients, fence the immune system, and
resist antibiotics 5, 6, 7, 8, 9, 10, 11, 12, 13, 14. Production of an
artificial swarm on the micro-scale faces a serious challenge — while an
individual bacterium has an evolutionary-forged internal machinery to produce
propulsion, until now, artificial micro-swimmers relied on the precise
chemical composition of their environment to directly fuel their drive 14, 15,
16, 17, 18, 19, 20, 21, 22, 23. When crowded, artificial micro-swimmers
compete locally for a finite fuel supply, quenching each other’s activity at
their greatest propensity for cooperation. Here we introduce an artificial
micro-swimmer that consumes no chemical fuel and is driven solely by light. We
couple a light absorbing particle to a fluid droplet, forming a colloidal
chimera that transforms light energy into propulsive thermo-capillary action.
The swimmers’ internal drive allows them to operate and remain active for a
long duration (days) and their effective repulsive interaction allows for a
high density fluid phase. We find that above a critical concentration,
swimmers form a long lived crowded state that displays internal dynamics. When
passive particles are introduced, the dense swimmer phase can re-arrange and
spontaneously corral the passive particles. We derive a geometrical,
depletion-like condition for corralling by identifying the role the passive
particles play in controlling the effective concentration of the micro-
swimmers.

Figure 1: The light driven fuel-free micro-swimmers display distinct
behaviours at different densities. a a single micro-swimmer is composed of a
light absorbing particle (dark sphere), and a fluid droplet (white sphere).
Individual swimmers transition from diffusive dynamics when light is off (b)
to more ballistic when light is turned on (c) (see Supplementary Video 3). d
at an intermediate concentration, swimmers collide, align, and create small
co-moving groups. d-g an individual swimmer (orange) maintains its activity
even when passing through an oppositely moving group (cyan), penetrating, and
emerging from the other side (see Supplementary Video 4). h-i long exposure (1
min) images of swimmers at high concentrations reveal internal flow structures
(see Supplementary Video 5). Scale bars: a: $5\;\mu\rm{m}$; b, c:
$20\;\mu\rm{m}$; d-g: $25\;\mu\rm{m}$; h - i: $50\;\mu\rm{m}$. Figure 2: The
internal drive of the fluid micro-swimmers relies on light activated thermo-
capillary action, independent from a chemical fuel in the surrounding bath. a
the mean square displacement (MSD) shows an individual swimmer’s motility can
be continuously tuned from diffusive ($\propto t^{1}$) to ballistic ($\propto
t^{2}$). b the speed of a swimmer scales linearly with the power flux of the
light source reaching more than one body length per second. c with increasing
power flux, trajectories of free swimmers show transition from random (bottom)
to more ballistic (top) motion. Using an external magnetic field the
orientation of a swimmer can be controlled to move on a straight line (d) or
in circles (e) allowing a direct measurement of the nominal swimming speed. f
increasing the internal viscosity of the fluid droplet directly reduces the
speed of the swimmer. Solid line is the theoretically predicted swimming speed
using no fitting parameters (full characterisation of optical absorption and
thermo-capillary measurements are found in Supplementary Information section
1). Scale bars: $20\;\mu\rm{m}$.

In order to explore the dynamics of a dense swarm of micro-swimmers, we
synthesized a colloidal dimer by coupling a light absorbing particle (M280) to
a liquid droplet (n-dodecane), forming a $5\;\mu\rm{m}$ long, peanut shaped
swimmer (see Fig.1a and in Methods for synthesis steps24). When exposed to
light, dimers commence a persistent swimming motion; when light is turned off,
the dimers revert to random Brownian motion (see Fig.1b,c). As a light source,
we used a wide beam (diameter $d=310\;\mu\rm{m}$) infrared (wavelength
$\lambda=1064\;\rm{nm}$) laser (see section 1.3 in Supplementary Information
for details). Typical experiments were performed in deionized water, yet
swimmers are also active in saline phosphate buffer (PBS), or when exposed to
a visible, non-coherent, light sources (see Methods for experimental setup and
sample preparation). We find drastically different collective dynamics of the
active particles when their area fraction, $\phi_{A}$, is increased,
transitioning from quickly dispersing when dilute ($\phi_{A}<0.14$, Fig.1b-c
and Supplementary Video 3); to forming transient, aligned aggregates at
intermediate concentration ($0.14<\phi_{A}<0.28$, Fig.1d-g, and Supplementary
Video 4); Finally, at high concentrations, swimmers crowd in a dense, active
colony with internal flows ($0.48\leq\phi_{A}$, Fig.1h-i, and Supplementary
Video 5).

In both the intermediate and high concentration regimes individual swimmers
are often observed to swim within, and sometimes against, a group of other
active particles (see Fig.1d-f, Fig.3a, and Supplementary Videos 1 and 4).
This high degree of autonomy has been observed in living bacteria 12, 11, 10,
9, 13, 14, but was not reported in previously studied systems of artificial
micro-swimmers fuelled by either external ionic currents 15, 16, consumption
of micelles 20, hydrolysis of hydrogen peroxide 23, or critical de-mixing of
their surrounding buffer 17. Instead, those systems exhibit interrupted
internal dynamics, with the crowded phase being frozen, and sometimes
crystalline.

The propulsion mechanism of a single active particle is quantitatively
captured using a thermo-capillary drive for a fluid droplet at a local
temperature gradient25, 26. Since surface tension, $\gamma$, is a powerful
agent on the micro-scale and is generally temperature dependent27, 28,
$\gamma=\gamma\left(T\right)$, a small local temperature-gradient suffices to
generate a considerable propulsion. Using light, we heat up the light
absorbing particle to establish a local temperature gradient, $\vec{\nabla}T$,
at the vicinity of the fluid droplet. The temperature gradient generates a
surface-tension gradient proportional to the thermo-capillary coefficient,
$\beta\equiv\frac{\partial\gamma}{\partial T}$. The velocity, $\vec{v}$, of a
fluid droplet in a temperature gradient is given by25:

$\vec{v}=-\left[\frac{1}{2\eta_{f}+3\eta_{w}}\right]\frac{\rm{D}}{2+\kappa_{w}/\kappa_{f}}\beta\vec{\nabla}T,$
(1)

with $\rm{D}$ being the droplet diameter, $\eta_{i}$ is linear viscosity, and
$\kappa_{i}$ the thermal conductivity (index $i$ indicates $w$-ater or
$f$-luid droplet). Note that for high droplet viscosity,
$\eta_{f}\gg\eta_{w}$, the swimming speed is dictated by the internal flow
inside the fluid droplet, $v\propto\beta\nabla T/\eta_{f}$. We independently
measured both the heating of the light absorbing particle and the thermo-
capillary coefficient of the fluid droplet, finding numerical values
consistent with existing literature 29, 30, 31 (see Figs. 5, 6, and section 1
in Supplementary Information for additional informatoin). By synthesising a
family of swimmers made from a homologous series of alkanes, we observe a
decrease in swimming speed, $v$, with increasing internal viscosity,
$\eta_{f}$, as required by an internal drive (Fig.2f). When accounting for the
reduced mobility of particles near a solid surface 32, we find quantitative
agreement between measured swimming speed, and the light driven thermo-
capillary model for swimmers of different compositions (see Fig.2f) with no
fitting parameters.

Figure 3: At sufficiently high average area fraction, $\phi_{0}$, the swimmer
population segregates into a long lived state of dilute and dense fluids. a
the instantaneous local density of each swimmer is measured from the inverse
size of the corresponding Voronoi cell. Both the dilute and the dense phases
are fluid as can be seen from the trace of an individual swimmer crossing the
different phases, moving slower (darker trace) in the denser region (see
Supplementary Video 1). b the probability for finding an individual swimmer at
specific local density shows two characteristic phases. Particle displacement
over a fixed interval (5 seconds) shows a sharp decrease above local
concentration of $\phi^{*}=0.38$ (black squares). Inset shows the displacement
probability distribution function for an individual swimmer below (yellow
squares) and above (red circles) $\phi^{*}$. The low density region deviates
from a Gaussian distribution (yellow curve) while the displacement PDF in the
dense phase fits a Gaussian distribution (red curve). c velocity correlation
function within the dense phase indicates flows correlate on a length scale
much larger than the swimmer size. Inset shows streamlines measured using
particle image velocimetry, displaying an internal flow structure (see section
5 in Supplementary Information). d at lower initial concentration,
$\phi_{0}\leq 0.28$, most of the swimmers will disperse, $\Delta N\gg N_{0}$,
within a few seconds up to minutes. At higher initial concentrations,
$\phi_{0}\geq 0.49$, the swarm forms a persistent crowded state, which remains
throughout the measurement time (hours). Inset shows temporal evolution. Scale
bars are $20\;\mu\rm{m}$.

An individual swimmer can be oriented using an external magnetic field
(through the small remanent moment of the superparamagnetic light absorbing
particle) as shown in Fig.2d,e. Fixing the orientation allows direct
measurement of the swimming speed. Decoupling propulsion from random motion we
find a persistent time of $\tau\approx 15\;\rm{s}$ for the free swimmer
(Fig.2a and section 2 in Supplementary Information). With increasing light
intensity, individual swimmers transition from Brownian to ballistic (Fig.2a)
and can move up to 3 body lengths per second ($15\;\mu\rm{m}/s$). Here we
focus on the slower swimming speeds, $v\leq 5\;\mu\rm{m}/s$, corresponding to
a Péclet Number, $P_{e}$, of up to $200$. In the presence of other swimmers,
the individual swimmer remains active, but with dramatically different
dynamics.

Figure 4: A swarm of active swimmers cooperatively corrals passive particles.
a a ternary phase diagram of active ($\phi_{A}$) passive ($\phi_{P}$) and the
surrounding water bath ($\phi_{W}$) depicts the resulting dynamics of a
mixture of passive particles and active swimmers, given their initial area
fraction. Green triangles show that at low overall-particle area fraction
(large $\phi_{W}$), the active swimmers quickly disperse. Red circles show a
region where the active particles crowd once activated and form a persistent
dense phase. Between the red and the green regions,
$1-4\phi_{P}<\phi_{A}/\phi^{*}<1-\frac{\sqrt{12}}{\pi}\phi_{P}$, a yellow
region emerges, where orange hexagons signify experiments where active
particles spontaneously corral the passive particles ($\phi^{*}$ is denoted by
a star). Red and orange curves are theoretical predictions for the onset of
the corralling state extrapolated for the small $\phi_{P}$ limit. Thumbnails
show snapshots from corresponding points on the diagram (orange and cyan label
active and passive particles respectively). Blue region indicates an overall
particle area fraction greater than 0.91 (hexagonal packing). b-d snapshots
taken at time 0 mins (b) 5 mins (c) and 30 minutes (d) show the global
dynamics of the active particles (orange) as they corall the passive particles
(cyan) (see Supplementary Video 6). Insets are the structure factor,
$S\left(\vec{q}\right)$, of the passive particles, showing the evolution
towards hexagonal order in their packed arrangement. e the pair correlation
function has peaks at 1, 2, 3, and 4 diameter, with $1-\sqrt{3}$ splitting
around 2 indicative of ordered hexagonal packing. f snapshots at 20 seconds
intervals show the corralling process which proceeds through entrainment
events, where an active particle (orange) deposits a passive particle (cyan)
at the dense region. After deposition, the active particle leaves (see
Supplementary Video 2). Scale bars: b-d $30\;\mu\rm{m}$; f: $10\;\mu\rm{m}$.

The individual swimmer motility is characterized by a significant slow down
once entering a region of high local concentration (see Fig.3a and
Supplementary Video 1). It is known that the speed-concentration dependence of
the individual swimmer, $v\left(\phi\right)$, can be used to predict the
critical concentration, $\phi^{*}$, at which the whole swarm tends to form a
persistent dense region33. The trajectory of a single swimmer as it samples a
heterogeneous density field is shown in Fig.3a and Supplementary Video 1. The
long lived crowded state is characterized by the co-existence of dense and
dilute regions, with corresponding peaks in the density probability
distribution (Fig.3b). By measuring the local, instantaneous area fraction,
$\phi$ (using Voronoi tesselation of surrounding swimmers, see section 3 in
Supplementary Information), we find that a swimmer maintains its nominal speed
at low concentrations, and experiences a sharp decrease in motility inside the
dense region (Fig.3b). The swimmer regains its nominal speed once emerging
from the other side of the dense region. It is theoretically predicted that
when the decrease in speed with concentration is sufficiently sharp,
$\frac{\partial v}{\partial\phi}|_{\phi={\phi^{*}}}\leq-\frac{v}{\phi}$,
concentration gradients appear at steady state in a process known as Motility
Induced Phase Separation 33. The critical concentration we anticipate from
$v\left(\phi\right)$ if Fig.3b is $\phi^{*}\approx 0.38$, consistent with
numerical predictions found in the literature 33, 34, 35, 36, 37, 38. Indeed,
the swarm’s dynamics changes dramatically above this critical concentration.
Below $\phi^{*}$ the initial population, $N_{0}$, quickly disperses, with the
lion’s share of the swimmers, $\Delta N$, leaving the observed field of view
$\Delta N\approx N_{0}$. By contrast, at concentrations above $\phi^{*}$ the
swarm maintains a long lived crowded huddle, with only a small fraction of the
swimmers leaving, $\Delta N\ll N_{0}$ (see Fig.3d). As in dense bacterial
colonies 10, 7, 13, the crowded swimmer phase is liquid, characterized by flow
correlations much larger than the individual, yet displays a turbulent
structure (Fig.3c), allowing internal re-arrangement, a key ingredient for
their cooperativity.

As the swimmers remain active even at high concentrations, they cooperate to
corral passive particles (Supplementary Video 6). Figures 4b-d show the time
evolution of a heterogeneous mixture of active and passive particles, where
the passive particles are compressed by active particles into a dense phase
with hexagonal order, as seen in the pair correlation function,
$g\left(r\right)$, and the structure factor, $S\left(\vec{q}\right)$ in
Fig.4b-e (see section 6 in Suuplementary Information). Microscopically, the
corralling builds-up through series of entrainment events, similar to those
observed with microalga 39 where active particles chaperon passive particles
to the dense region. The passive particles are then deposited while the active
particles depart (Fig.4f and Supplementary Video 2). A ternary phase diagram
(Fig.4a) summarizes the different dynamics observed given the initial area
fractions of the active particles $\phi_{A}$, passive particles, $\phi_{P}$,
and the surrounding water phase, $\phi_{W}$. When the initial concentration is
less than dense packing (blue region $\phi_{A}+\phi_{P}\leq 0.907$), we
identify three characteristic behaviours: active particles quickly disperse at
low particle area fraction (green region); a long lived dense active state is
formed at higher concentrations (red region); at intermediate particle
concentrations, the active particles corral the passive particles (yellow
region).

A simple geometrical construction captures the corralling phase boundaries in
the phase diagram. Introducing passive particles reduces the total area
available for the active particles, increasing their effective area fraction,
$\phi^{\rm{eff}}_{A}$. The increased effective area fraction may surpass the
critical concentration, $\phi^{*}$, and trigger the crowding dynamics. The
extent of this effect however depends on the spatial arrangement of the
passive particles. When randomly dispersed each passive particle occupies much
more room as it carries a corona of excluded area. When the passive particles
are packed in a lattice (as in Figs.4d,e), their excluded areas overlap and
they effectively occupy a smaller area. Therefore we expect the effective area
fraction of the active particles to reduce when the passive particles are
hexagonally packed (HP), $\phi_{A}^{P,\rm{rand}}>\phi_{A}^{P,\rm{HP}}$. For
circular passive particles in 2D we can estimate the effective area fraction
of the active particles for the two cases as,

$\phi^{\rm{eff}}_{A}\approx\begin{cases}\phi_{A}/\left(1-4\phi_{P}\right)&\rm{passive\;are\;random}\\\
\phi_{A}/\left(1-\frac{\sqrt{12}}{\pi}\phi_{P}\right)&\rm{passive\;are\;HP}\end{cases}$
(2)

(see Supplementary Information section 7 for full derivation). The red and
orange curves in Fig.4a respectively represent the point at which the
effective area fraction of the active particles is equal to their critical
area fraction for crowding, $\phi_{A}^{P,\rm{rand}}=\phi^{*}$ and
$\phi_{A}^{P,\rm{HP}}=\phi^{*}$. The corralling region is found between those
two curves.

The following picture arises: when active and passive particles are
homogeneously mixed, the passive particles can trigger the crowding of the
active particles at concentrations lower than the critical concentration of
the pure active system, $\phi^{*}$. Once the active particles complete the
corralling task, they disperse, as the area gained by the overlapping excluded
areas makes them sub-critical. A similar equilibrium process is the depletion
interaction 27, 40, where the diffusion of many small particles (depletants)
can drive fewer, larger particles to crystallize. There is one important
distinction: even in the absence of passive particles, active particles have
an intrinsic critical clustering concentration because of their activity. Our
analysis shows qualitatively similar results to previous theoretical work on
mixed active/passive populations 41, 42, 36, 35, but may prove more general as
it relies on a generic geometrical argument.

In this work we have introduced a synthetic route and a driving strategy for a
new family of micro-swimmers that are propelled solely by light — a colloidal
chimera of a light absorbing particle bound to a fluid droplet. An individual
swimmer’s propulsion mechanism is quantitatively captured by a thermo-
capillary drive. The swimmers remain active at high area fractions,
$\phi_{A}>0.5$, and can be used as an experimentally tuneable model system to
study a host of phenomena in dense active matter such as 2D turbulence in a
compressible fluid, previously seen in bacterial colonies13, 9, 10, 11, 12. We
found that at low concentrations swimmers tend to quickly disperse, but above
a critical concentration form a long lived dense phase. We identify a
depletion-like interaction where the excluded volume of passive particles in
random or close packed configurations can change $\phi^{*}$ of the active
particles resulting in passive particle crystallization. As the mechanism is
geometric, it may be applicable generally from living organisms to robotic
swarms. As our optically powered Marangoni swimmers are largely agnostic to
their precise chemical environment, we were able to graft DNA on their
surface. Micro-swimmers augmented by DNA nanotechnology 43, 44 can prove
useful in the self-assembly of active tissue, mimicking biofilms, in the
making of future, functional active matter.

## References

  * 1 Hamann, H.  _Swarm Robotics: A Formal Approach_ (Springer International Publishing, Cham, 2018).  URL http://link.springer.com/10.1007/978-3-319-74528-2.
  * 2 Procaccini, A. _et al._ Propagating waves in starling, Sturnus vulgaris, flocks under predation.  _Animal Behaviour_ 82, 759–765 (2011).  URL http://dx.doi.org/10.1016/j.anbehav.2011.07.006.
  * 3 BBC.  Hunger at Sea (Oceans) (2015).
  * 4 Harari, Y. N.  _Sapiens : a brief history of humankind_ (Harper, New York, 2015).
  * 5 Costerton, J. W., Stewart, P. S. & Greenberg, E. P.  Bacterial biofilms: A common cause of persistent infections.  _Science_ 284, 1318–1322 (1999).
  * 6 Saintillan, D. & Shelley, M. J.  Hydrodynamic fluctuations and instabilities in ordered suspensions of self-propelled particles.  _Physics of Fluids_ 20, 1–17 (2008).
  * 7 Secchi, E. _et al._ Intermittent turbulence in flowing bacterial suspensions.  _Journal of Royal Society Interfaces_ 13 (2016).
  * 8 Marchetti, C. M. _et al._ Hydrodynamics of soft active matter.  _Reviews of Modern Physics_ 85, 1143–1189 (2013).  abs/1207.2929.
  * 9 Dunkel, J. _et al._ Fluid dynamics of bacterial turbulence.  _Physical Review Letters_ 110, 1–5 (2013).  abs/1302.5277.
  * 10 Wensink, H. H. _et al._ Meso-scale turbulence in living fluids.  _Proceedings of the National Academy of Sciences_ 109, 14308–14313 (2012).  abs/1302.5277.
  * 11 Zhang, H. P., Be’er, A., Florin, E. L. & Swinney, H. L.  Collective motion and density fluctuations in bacterial colonies.  _Proceedings of the National Academy of Sciences of the United States of America_ 107, 13626–13630 (2010).
  * 12 Dombrowski, C., Cisneros, L., Chatkaew, S., Goldstein, R. E. & Kessler, J. O.  Self-Concentration and Large-Scale Coherence in Bacterial Dynamics.  _Physical Review Letters_ 93, 098103 (2004).
  * 13 Xu, H., Dauparas, J., Das, D., Lauga, E. & Wu, Y.  Self-organization of swimmers drives long-range fluid transport in bacterial colonies.  _Nature Communications_ 10, 1–12 (2019).  URL http://dx.doi.org/10.1038/s41467-019-09818-2.
  * 14 Gompper, G. _et al._ The 2020 motile active matter roadmap.  _Journal of Physics: Condensed Matter_ 32, 193001 (2020).  abs/1912.06710.
  * 15 Van Der Linden, M. N., Alexander, L. C., Aarts, D. G. & Dauchot, O.  Interrupted Motility Induced Phase Separation in Aligning Active Colloids.  _Physical Review Letters_ 123, 98001 (2019).  URL https://doi.org/10.1103/PhysRevLett.123.098001.  1902.08094.
  * 16 Geyer, D., Martin, D., Tailleur, J. & Bartolo, D.  Freezing a Flock: Motility-Induced Phase Separation in Polar Active Liquids.  _Physical Review X_ 9, 31043 (2019).  URL https://doi.org/10.1103/PhysRevX.9.031043.  1903.01134.
  * 17 Bäuerle, T., Fischer, A., Speck, T. & Bechinger, C.  Self-organization of active particles by quorum sensing rules.  _Nature Communications_ 9, 1–8 (2018).  URL http://dx.doi.org/10.1038/s41467-018-05675-7.
  * 18 Yan, J. _et al._ Reconfiguring active particles by electrostatic imbalance.  _Nature Materials_ 15, 1095–1099 (2016).
  * 19 Lozano, C., tan Hagen, B., Lowen, H. & Bechinger, C.  Phototaxis of synthetic microswimmers in optical landscapes.  _Nature_ 7, 12828 (2016).
  * 20 Maass, C. C., Krüger, C., Herminghaus, S. & Bahr, C.  Swimming Droplets.  _Annual Review of Condensed Matter Physics_ 7, 171–193 (2016).  URL http://www.annualreviews.org/doi/10.1146/annurev-conmatphys-031115-011517.
  * 21 Izri, Z., van der Linden, M. N., Michelin, S. & Dauchot, O.  Self-Propulsion of Pure Water Droplets by Spontaneous Marangoni-Stress-Driven Motion.  _Physical Review Letters_ 113, 248302 (2014).  URL https://link.aps.org/doi/10.1103/PhysRevLett.113.248302.
  * 22 Howse, J. R. _et al._ Self-Motile Colloidal Particles: From Directed Propulsion to Random Walk.  _Physical Review Letters_ 99, 8–11 (2007).  0706.4406.
  * 23 Palacci, J., Sacanna, S., Steinberg, A. P., Pine, D. J. & Chaikin, P. M.  Living Crystals of Light-Activated Colloidal Surfers.  _Science_ 339, 936–940 (2013).  URL http://www.sciencemag.org/cgi/doi/10.1126/science.1230020.
  * 24 Ben Zion, M. Y., Caba, Y., Sha, R., Seeman, N. C. & Chaikin, P. M.  Mix and match - A versatile equilibrium approach for hybrid colloidal synthesis.  _Soft Matter_ 16, 4358–4365 (2020).
  * 25 Young, B. N., Goldstein, J. S. & Blocks, M. J.  The motion of bubbles in a vertical temperature gradient.  _Journal of Fluid Mechanics_ 6, 350–356 (1959).
  * 26 Barton, K. D. & Shankar Subramanian, R.  The migration of liquid drops in a vertical temperature gradient.  _Journal of Colloid And Interface Science_ 133, 211–222 (1989).
  * 27 Doi, M.  _Soft Matter Physics_ (Oxford University Press, Oxford, 2013).
  * 28 de Gennes, P.-G., Brochard-Wyart, F. & Quere, D.  _Capillarity and Wetting Phenomena_ (Springer, 2002).
  * 29 Sloutskin, E., Bain, C. D., Ocko, M. & Deutsch, M.  Surface freezing of chain molecules at the liquid – liquid and liquid – air interfaces.  _Faraday Discussions_ 339–352 (2005).
  * 30 Baroud, C. N., Delville, J.-p., Gallaire, F. & Wunenburger, R.  Thermocapillary valve for droplet production and sorting.  _Physical Review E_ 75, 046302 (2007).
  * 31 Robert de Saint Vincent, M. _et al._ Laser switching and sorting for high speed digital microfluidics Laser switching and sorting for high speed digital microfluidics.  _Applied Physics Letters_ 92, 154105 (2008).
  * 32 Happel, J. & Brenner, H.  _Low Reynolds Number Hydrodynamics_ (Springer, Englewood Cliffs, 1983).
  * 33 Tailleur, J. & Cates, M. E.  Statistical mechanics of interacting run-and-tumble bacteria.  _Physical Review Letters_ 100, 3–6 (2008).  abs/0803.1069.
  * 34 Fily, Y. & Marchetti, C. M.  Athermal Phase Separation of Self-Propelled Particles with No Alignment.  _Physical Review Letters_ 235702, 1–5 (2012).
  * 35 Redner, G. S., Hagan, M. F. & Baskaran, A.  Structure and dynamics of a phase-separating active colloidal fluid.  _Physical Review Letters_ 110, 1–5 (2013).
  * 36 Stenhammar, J., Wittkowski, R., Marenduzzo, D. & Cates, M. E.  Activity-induced phase separation and self-assembly in mixtures of active and passive particles.  _Physical Review Letters_ 114, 1–5 (2015).  1408.5175.
  * 37 Cates, M. E. & Tailleur, J.  Motility-Induced Phase Separation.  _Annual Review of Condensed Matter Physics_ 6, 219–244 (2015).
  * 38 Solon, A. P. _et al._ Pressure is not a state function for generic active fluids.  _Nature Physics_ 11, 673–678 (2015).  abs/1412.3952.
  * 39 Jeanneret, R., Pushkin, D. O., Kantsler, V. & Polin, M.  Entrainment dominates the interaction of microalgae with micron-sized objects.  _Nature Communications_ 7, 1–7 (2016).  abs/1602.01666.
  * 40 Kardar, M.  _Statistical Physics of Particles_ (Cambridge University Press, Cambridge, 2007).
  * 41 Weber, S. N., Weber, C. A. & Frey, E.  Binary Mixtures of Particles with Different Diffusivities Demix.  _Physical Review Letters_ 116, 1–5 (2016).  abs/1505.00525.
  * 42 Grosberg, A. Y. & Joanny, J. F.  Nonequilibrium statistical mechanics of mixtures of particles in contact with different thermostats.  _Physical Review E - Statistical, Nonlinear, and Soft Matter Physics_ 92, 1–10 (2015).
  * 43 Seeman, N. C.  _Structural DNA Nanotechnology_ (Cambridge University Press, 2015).
  * 44 Ben Zion, M. Y. _et al._ Self-assembled three-dimensional chiral colloidal architecture.  _Science_ 358, 633–636 (2017).
  * 45 Mustafaev, R. A.  Thermal conductivity of higher saturated n-hydrocarbons over wide ranges of temperature and pressure.  _Journal of Engineering Physics_ 24, 465–469 (1973).
  * 46 Calado, J. C., Fareleira, J. M., Nieto de Castro, C. A. & Wakeham, W. A.  Thermal conductivity of five hydrocarbons along the saturation line.  _International Journal of Thermophysics_ 4, 193–208 (1983).
  * 47 Tanaka, Y., Itani, Y., Kubota, H. & Makita, T.  Thermal conductivity of five normal alkanes in the temperature range 283-373 k at pressures up to 250 MPa.  _International Journal of Thermophysics_ 9, 331–350 (1988).
  * 48 Schmidt, R. _et al._ Hydrocarbons.  In _Ullmann’s Encyclopedia of Industrial Chemistry_ , 1–74 (Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim, Germany, 2014).  URL http://doi.wiley.com/10.1002/14356007.a13{_}227.pub3.
  * 49 Rueden, C. T. _et al._ ImageJ2: ImageJ for the next generation of scientific image data.  _BMC Bioinformatics_ 18, 529 (2017).
  * 50 Allan, D. _et al._ soft-matter/trackpy: Trackpy v0.4.2 (2019).  URL https://doi.org/10.5281/zenodo.3492186.
  * 51 Liberzon, A. _et al._ OpenPIV/openpiv-python: OpenPIV - Python (v0.22.2) with a new extended search PIV grid option (2020).  URL https://doi.org/10.5281/zenodo.3930343.
  * 52 Harper, M.  python-ternary: Ternary Plots in Python.  _Zenodo 10.5281/zenodo.594435_ (2019).  URL https://github.com/marcharper/python-ternary.

### Acknowledgements

We thank N. Oppenheimer, O. Dauchot, A. M. Leshansky, T. Goldfriend and C. P.
Kelleher for insights. This research was primarily supported by the Center for
Bio-Inspired Energy Sciences, an Energy Frontier Research Center funded by the
DOE, Office of Sciences, Basic Energy Sciences, under award no. DE-SC0000989,
and ), and the Diversity Undergraduate Research Initiative (DURI) New York
University.

## Methods

## Swimmer Synthesis and Sample Preparation

Hybrid dimers were synthesized by binding a $3\;\mu\rm{m}$ light absorbing
particle (M280 BANGS LAB) to a fluid oil particle (see list in section 1)
using the previously introduced Mix-And-Match strategy24. Briefly, fluid
droplets were made through membrane emulsification where oil with $0.1\%$ v/v
SPAN80 (SIGMA) is pressurized through a porous membrane forming a stable oil
in water emulsion (stable for months at room temperature). Typical particle
size was $3\;\mu\rm{m}$. M280 particle and fluid droplets are
stoichiometrically mixed, destabilized, quenched, and purified using density
gradient step centrifugation. Dimers were found to be stable for months when
stored in a refrigerator at $4^{o}\rm{C}$. Samples were prepared for imaging
by loading a suspension to a pre-treated glass capillary (channel height
$0.10$ mm, width $2.00$ mm Vitrotubes W5010-050), that were plasma cleaned
(SPI supplies Plasma Prep II), and passivated through vapour deposition of
hexamethyldisilazane (SIGMA). Loaded capillaries were placed on a clean
microscope glass slide, and sealed on their ends with UV curable resine (LOON
OUTDOORS UV CLEAR FLY FINISH).

## Experimental Setup

Imaging was done using bright field on a home built microscope coupled to a
laser source. A commercial light emitting diode ($\lambda=505$ nm THORLABS)
with a diffuser (ground glass N-BK7 600 grit, THORLABS), a condenser and an
iris, to achieve Köhler illumination. Scattered light was picked up by the
microscope objective (HCX PL APO 40x NA = 0.85, LEICA), and a tube lens (B&H),
and detected by a digital camera (DCC1545M, IMAGING SOURCE) and acquired using
commercial video recording software (IC CAPTURE, IMAGING SOURCE). A heating
beam was introduced on a separate optical path (see Fig.7 in Supplementary
Information). A $1064$ nm laser beam (YLR-10-1064-LP, IPG Photonics) was
passed through a zero order half plate (WPH05M-1064 Thorlabs) and contracted
using a custom Galilean telescope to achieve a $\sim 300\;\mu$m beam (see
Supplementary Information). The laser beam was introduced into the sample
using a polarizing beam splitter (PBS CM1-PBS253 THORLABS) and its intensity
at the sample was controlled by a combination of the electronic laser head
controller and by adjusting the half-plate, and was measured using an optical
power meter (PM100D power meter, with S175C sensor, THORLABS). In order to
eliminate laser intensity before the camera, stained glasses (FGS900S,
THORLABS) were stacked after the objective.

### Supplementary Videos

S1 - Swimmer’s activity inside the dense phase.

S2 - Entrainment mediated corralling.

S3 - Quick dispersing at low density.

S4 - Schooling and anti-schooler at intermediate density.

S5 - Internal flow inside the dense phase.

S6 - Corralling of passive particles.

##  Light Driven Thermo-Capillary Swimmer Model

###  Fluid Droplet in a Temperature Gradient

For the thermo-capillary swimmers presented, the temperature gradient is given
by the temperature profile around the hotter light absorbing particle when it
is exposed to radiation
$\vec{\nabla}T=-\frac{P_{0}}{4\pi\kappa_{w}\rm{R}^{2}}{\bf\hat{e}_{r}}$, where
$P_{0}$ is the heating power at the particle, and $R$ the distance from its
center, along the radial direction ${\bf\hat{e}_{r}}$. The power absorbed by
the particle is given by $P_{0}=\sigma J$, where $J$ is the flux and $\sigma$
the particle’s absorption cross section (see section 1.3 for measurement of
$\sigma$). As on the length scale of the swimmers, heat transport is dominated
by high thermal diffusivity, the temperature profile is assumed to move with
the swimmer. Since the droplet is intimately bound to the particle, we
approximate the temperature gradient to be constant with its value at the
center of the droplet, allowing us to use Eq.1. To quantitatively verify that
indeed this is the correct swimming mechanism, we directly measured the
thermo-capillary coefficient, $\beta$ (see section 1.2), and use literature
data for the thermal conductivies and viscosities of the different swimmers
species (see Table 1). The swimmer’s proximity to the wall, $\delta\approx
0.2$ nm (estimated from its gravitational height), leads to enhanced drag,
estimated to increase by a factor of
$\frac{8}{15}\rm{log}\left(\frac{\rm{D}}{2\delta}\right)$ for a sphere moving
along a solid wall in Stokes flow 32.

Material | $\kappa_{f}\left(\frac{\rm{W}}{\rm{K\;m}}\right)$ | $\eta_{\rm{f}}\left(\rm{mPas}\right)$
---|---|---
n-decane | 0.15 | 0.93
n-dodecane | 0.14 | 1.5
n-tetradecane | 0.14 | 2.3
n-hexadecane | 0.14 | 3.5
mineral oil | 0.17 * | 13 *
Table 1:  Material properties of different fluid droplets were taken from the
literature 45, 46, 47, 48. According to manufacturer.

###  Thermo-Capillary Measurement

Surface tension measurements were carried out using a commercial optical
tensiometer (THETA ONE ATTENSION) by hanging a deionized water drop
($18\;\rm{M}\Omega\rm{cm}$, TOC = $8$ ppb, MILLIPORE MILLI-Q) in a n-dodecane
(ALPHA AESAR) bath with $0.1\%$v/v SPAN-80 (SIGMA) inside a glass cuvette
(THERMOFISHER) and fitting the profile of the pendant drop to the Young-
Laplace model 28. The cuvette was wrapped with resistive heating wire to
ensure a homogeneous temperature, monitored using a thermometer (BS4 59-7569
Monitoring Thermometer) with a thermo-couple sensor (IT-21 microprobe) placed
in close proximity to the droplet. Surface tension readings were taken after
pendant drop was allowed to equilibrate at each temperature. The surface
tension was then measured for multiple droplets, and their average reading was
taken. Temperature was varied between $21$-$27^{o}$C and with a surface
tension increase from $5.2\;\frac{\rm{mN}}{\rm{m}}$ at room temperature, to
$8\;\frac{\rm{mN}}{\rm{m}}$. The linear thermo-capillary coefficient is found
from linear regression (see Fig.5) to be $\beta=0.58\pm
0.02\;\frac{\rm{mN}}{\rm{{}^{o}Cm}}$, quantitatively consistent with known
values in the literature 29, 30, 31.

Figure 5: Linear thermo-capillary coefficient measurement.

###  Laser Heating Power Balance

Heating measurements were carried out by tracking the temperature increase
rate, $\frac{\partial T}{\partial t}$, (BS4 59-7569 Monitoring Thermometer)
with a thermo-couple sensor (IT-21 microprobe) placed inside a standard glass
cuvette (THERMOFISHER length $L=1$ cm) filled with a suspension of $D=3\;\mu$m
light absorbing particles (M280 Bangs Lab) in deionized water
($18\rm{M}\Omega\rm{cm}$, TOC=$8$ ppb, MILLIPORE MILLI-Q), when irradiated
with a collimated $1064$ nm laser beam (YLR-10-1064-LP, IPG Photonics) while
stirring using a magnetic stir bar to ensure rapid thermal homogenization. The
total light intensity of the beam, $P=1.3$ W was measured using an optical
power meter (PM100D power meter, with S175C sensor, THORLABS), and temperature
increase rate was found from the temperature vs. time slope at early times. In
order to extract the particle’s absorption cross section,
$\sigma=\pi\left(\frac{D}{2}\right)^{2}\sigma_{0}$ (where $\sigma_{0}$ is the
differential cross section for absorption), the measurements were taken at
increasing volume fraction, $\Phi$ ($0-0.4\%$v/v), eliminating the large but
constant contribution to the heating, $W$, by the water bulk. Assuming dilute
suspension ($\Phi\ll 1$), small temperature gradients from ambient ($t\ll 1$
min), and fast homogenization (stirring), the light absorbing particles
contribution to the heating is given by the slope of

$\frac{\partial T}{\partial
t}=W+\frac{\frac{3}{2}\frac{L}{D}P}{C_{P}}\sigma_{0}\Phi,$ (3)

where the heat capacity of the fluid, $C_{P}=c_{p}m_{\rm{H_{2}O}}$ is
dominated by the water, and the differential cross-section is found to be
$\sigma_{0}=5\%$, (see Fig.6).

Figure 6: Thermal heating due to light absorption

###  Beam Size Calibration

The beam profile was measured by exposure to infrared visualizing shavings
(980nm-1064nm IR Infrared Laser Visualizer) fixed inside a UV curable resin
(LOON OUTDOORS UV CLEAR FLY FINISH) between a microscope slide and a cover-
slip. Laser beam intensity, $\rm{I}$, was measured at the objective using an
optical power-meter (PM100D power meter, with S175C sensor, THORLABS). The
visible light emitted from the IR visualizer was imaged using a lower
magnification objective (Leica HC Pl Fluotar 10X), to create the profile seen
in Fig.8). The spatial distribution of the beam $J\left(R\right)$ was then
fitted to a two dimensional Gaussian $J=J_{0}e^{-\rm{R}^{2}/\sigma^{2}}$, with
$R$ the radial distance from the Gaussian center,
$J_{0}=\frac{\rm{I}}{\pi\sigma^{2}}$ being the flux at the center of the beam,
and $2\sigma=310\;\mu\rm{m}$ is the fitted beam diameter (see Fig.8).

Figure 7: Schematics of experimental setup. Figure 8: Broad field image of
laser beam (contours show relative power given by Gaussian fit). Experiments
were performed within a field of view of $\sim 100\;\mu\rm{m}$ at the center
of the beam where power spatial variation is smaller than 10%.

##  Persistent Swimmer Model

Individual swimmer’s dynamics were modelled as a persistent random walk with
speed $v$, persistent time $\tau$, and a diffusion constant $D_{0}$. The mean
square displacement, $\left<\Delta L^{2}\right>$, of such a swimmer can be
found in 22:

$\left<\Delta L^{2}\right>=4D_{0}\Delta
t+\frac{v^{2}\tau^{2}}{2}\left[\frac{2\Delta t}{\tau}+e^{-2\Delta
t/\tau}-1\right].$ (4)

Which for the short time limit ($\Delta t\ll\tau$) is ballistic, $\left<\Delta
L^{2}\right>\approx v^{2}\Delta t^{2}\propto\Delta t^{2}$, and for longer lag
times ($\Delta t\gg\tau$) behaves diffusively with an enhanced diffusion
constant, $\left<\Delta L^{2}\right>\approx\left(4D_{0}+v^{2}\tau\right)\Delta
t\propto\Delta t^{1}$. Swimmer videos were preprocessed using ImageJ 49, and
the subsequent particle motion tracked and analyzed using Trackpy package on
Python 50. Fig.2a in the main text shows the Brownian ($\propto\Delta t^{1}$)
to ballistic ($\propto\Delta t^{2}$) transition for increased flux. Swimming
speed was independently measured by imposing their orientation using a custom
built vector magnet made of 3 Helmholtz pairs, generating a constant
homogenous magnetic field ($\sim 1$mT) in the imaging plane. In the presence
of a magnetic field, the swimmer maintains a fixed direction (see Fig.2d),
allowing extraction of the swimmers speed, $v$, by measuring displacement over
time, which is found to be linear with flux (as shown in Fig.2b) and is
expected by the model (Eq.1). Note that the swimming direction is not
necessarily along the magnetic field as the dimmer orientation is decoupled
from the magnetic dipole (for example see Fig.2d). We find that the
reorientation time of the swimmer is then $\tau\approx 15$ s, corresponding to
a persistence length of up to $70\;\mu$m with a Peclet number $Pe\approx 200$
using the definition found in the literature 8. As our microscope imaging is
done from the top, with the laser introduced from the bottom (see Fig.7),
swimmers are subjected to radiation pressure and swim on the ceiling of the
capillary, explaining the small deviation from linearity at low fluxes in
Fig.2b. We did however perform similar experiment on an inverted version of
our apparatus (that is inverted imaging and laser introduced from above),
finding similar swimmers dynamics.

##  Local Density Measurement

Local instantaneous density measurements were done by locating the two
dimensional coordinates of all the particles in each frame
$\left(x_{i},y_{i},t_{i}\right)$ 50, around which a Voronoi tessellation was
constructed using standard Python routines. The local area fraction of each
particle $\phi_{i}$ is found by dividing the area of each particle, $a_{i}$ by
the area, $A_{i}$ of its Voronoi cell $\phi_{i}\equiv\frac{a_{i}}{A_{i}}$.
Histograming these values over 2600 frames gives the two population densities
found in Fig.3b. The area fraction field, $\phi\left(x,y,t\right)$ is then
found piecewise from the local area fraction of each particle
$\phi\left(x,y,t\right)=\sum_{i}\delta\left(x-x_{i}\right)\delta\left(y-y_{i}\right)\delta\left(t-t_{i}\right)\phi_{i}$
(where $\delta$ is the Dirac delta function). The single swimmer speed as a
function of concentration $v\left(\phi\right)$ is obtained by tracking a
swimmer’s displacement, $\left|\Delta r\right|$, moving in the dynamic
environment. Average displacement as a function of concentration
$\left<\left|\Delta r\right|\right>\left(\phi\right)$ is then found by
averaging displacements within density bins of $6.7\%$ area fraction within
time intervals of $5$ seconds from which points in Fig.3b are measured. Error
bars are the standard deviation of the displacement. A prediction for critical
crowding area fraction, $\phi^{*}$, is the found by fitting the measured
displacement-concentration curve $\Delta\left|r\right|\left(\phi\right)$ to a
sigmoid
$f\left(\phi\right)=\frac{B}{1+e^{\left(\phi-\phi^{*}\right)/\alpha}}+C$
($\sigma$, $B$, and $C$ are constants) and the critical crowding concentration
can be found from when satisfying the theoretically expected relation 33 of
the derivative: $\frac{\partial f}{\partial\phi}=-\frac{f}{\phi}$.

##  Critical Crowding Concentrations

In order to find the critical area fraction, $\phi^{*}$, above which swimmers
form a crowded phase, a series of experiments were carried out initiating
swimming in regions with a homogeneous swimmer distribution with an initial
mean area fraction, $\phi_{0}$ calculated by the total area of the observed
particles, $\sum^{N_{0}}_{i=1}a_{i}$, over the area of the field of view
($A_{\rm{FOV}}$ kept constant throughout the experiment)
$\phi_{0}\equiv\frac{1}{A_{\rm{FOV}}}\sum^{N_{0}}_{i=1}a_{i}$, where $N_{0}$
the initial number of particles. As the collective dynamics evolve, with the
number of swimmers inside the FOV is monitored as a function of time
$N\left(t\right)$, given the evolution of the relative number
$\frac{N\left(t\right)}{N_{0}}$ is found in Fig.3d.

##  Velocity Correlation Function from Particle Image Velocimetry

The normalized velocity correlation function,

$\left<\vec{V}\left(0\right)\cdot\vec{V}\left(\vec{R}\right)\right>_{N}=\frac{1}{\left<\vec{V}\left(\vec{r}\right)^{2}\right>_{r}}\left<\vec{V}\left(\vec{r}\right)\cdot\vec{V}\left(\vec{r}+\vec{R}\right)\right>_{r},$
(5)

was found from the auto correlation of the velocity field,
$\vec{V}\left(\vec{R}\right)$ ($\left<\right>_{r}$ denotes spatial average).
The velocity field $\vec{V}\left(\vec{R}\right)$ was found using OpenPIV
python library 51 from pairs of frames $1$ second apart taken from within the
dense phase for the duration of the experiment. The VCF is computed component-
wise in Fourier space $\tilde{V_{i}}\left(\vec{k_{i}}\right)$ (index $i$ is
for each coordinate $i\in\left[x,y\right]$) then using convolution theorem,
converted to real space from the inverse Fourier transform of
$\tilde{VCF}\left(\vec{k}\right)=\left|\tilde{V_{x}}\right|^{2}\left(\vec{k}\right)+\left|\tilde{V_{y}}\right|^{2}\left(\vec{k}\right)$,
with angular averaging to extract the radial dependence.

##  Spatial Correlations of Passive Particles

Structure factor for the passive particles, $S\left(\vec{q}\right)$, was
defined by

$S\left(\vec{q}\right)=\frac{1}{N}\left|\tilde{\rho}\left(\vec{q}\right)\right|^{2},$
(6)

where $\vec{q}$ is the wave vector, and $\tilde{\rho}\left(\vec{q}\right)$ is
the Fourier transform of the particle density, $\rho\left(\vec{r}\right)$,
which we compute by plotting
$\rho\left(\vec{r}\right)=\sum_{i}^{N}F\left(\vec{R_{i}}\right)$, with
$\\{\vec{R_{i}}\\}$ being the set of experimentally located particle
coordinates, and $F$ being a Gaussian form factor that reduces short wave
length noise. The radial pair correlation function, $g\left(r\right)$ in the
same figure was calculated by directly histogramming the distances, $r$,
measured any two pairs of passive particles, with constant binning of $200$ nm
($0.067$ particle diameter).

##  Corralling Through Excluded Area

The collective dynamics were measured by choosing a field of view with a
rather homogenous particle distribution, turning the light on, and tracking
the dynamics. We define the three states as follows: dispersing state where
after turning on the light source, most of the particles leave the field of
view (FOV); the crowding state where after turning on the light still most of
the swimmers are still inside the FOV; and the corralling state where the
major entity left in the FOV is the passive particles. Points on Ternary phase
space are defined as described in section 4, by measuring the initial area
fraction of passive $\phi_{P}$ and active $\phi_{A}$, and plotted using
python-ternary package 52.

To derive the criterion for corralling we note that the area available for the
active particles, $A$, is the total area, $A_{T}$, after accounting for the
effective area taken by the passive particles $A^{eff}_{P}$

$A=A_{T}-A^{eff}_{P}.$ (7)

Generally the area occupied by the passive particles depends on their spatial
arrangement. Given that there are $N_{A}$ active ($N_{P}$ passive) particles,
each of which occupies an area $a_{A}$ ($a_{P}$), their area fraction is
defined as $\phi_{A}\equiv\frac{a_{A}N_{A}}{A_{T}}$
($\phi_{P}\equiv\frac{a_{P}N_{P}}{A_{T}}$). The effective area fraction of the
active particles is then defined as $\phi^{\rm{eff}}_{A}\equiv\frac{N_{A}}{A}$
giving

$\phi^{\rm{eff}}_{A}=\phi_{A}\Big{/}\left(1-\frac{A^{eff}_{P}}{A_{T}}\right),$
(8)

which in general also depends on the spatial arrangement of the passive
particles. The question thus becomes the following: for a given spatial
arrangement of the passive particles, is the effective area fraction of the
active particles super critical: $\phi^{\rm{eff}}_{A}>\phi^{*}$? We shall
treat two cases of different spatial arrangement of the passive particles,
Random, and Hexgonally Packed (HP). When the passive particles are randomly
distributed, additionally to their own area, $a_{P}$, each particle occupies a
corona of excluded region, inaccessible to other particles (see Figure 9a).
For the simple case of equisized particles, this excluded area is 3 times that
of the particle itself, $a^{e}_{P}=3a_{P}$, making the effective area occupied
by the passive particles

$A^{\rm{rand}}_{P}=4a_{P}N_{P}.$ (9)

If however the passive particles are arranged in a HP crystal, their effective
area is considerably smaller,
$A^{\rm{HP}}_{P}=\frac{\sqrt{12}}{\pi}a_{P}N_{P}+O\left(\sqrt{N_{P}}\right)$
(as can be seen in Fig.9b). Here the excluded area becomes sub-extensive as it
grows like the perimeter of the crystal, and for large enough crystals, we can
approximate

$A^{\rm{HP}}_{P}\approx\frac{\sqrt{12}}{\pi}a_{P}N_{P}.$ (10)

Plugging the two cases (Eqs. 9 and 10) into Eq. 8 we recover the result in the
main text (Eq.2). This shows that there are combinations of concentrations of
the active and passive particles, where the effective area fraction of the
active particles, $\phi_{A}^{eff}$, can either be above or below the critical
crowding concentration, $\phi^{*}$, depending on the arrangement of the
passive particles. The system is expected to be in the crowding state when
even if the passive particles were hexagonally packed, the effective area
fraction of the active particles is still above critical,
$\phi^{eff}=\phi_{A}\Big{/}\left(1-\frac{\sqrt{12}}{\pi}\phi_{P}\right)>\phi^{*}$
(red region in phase diagram in Fig.4a in the main text). Correspondingly, if
the passive particles were randomly distributed, yet the effective area
fraction of the active particles was still below critical,
$\phi^{eff}_{A}\phi_{A}\Big{/}\left(1-4\phi_{P}\right)<\phi^{*}$, the system
is said to be in the dispersing regime (green shade in phase diagram in
Fig.4a). Between these two regimes, the corralling state is defined (yellow
region in Fig.4a).

Figure 9: Geometry of effective area occupied by 18 passive particles when a
randomly distributed, or b hexagonally packed. Light shaded areas are the
regions excluded for the test particle (red).