# Randomly Distributed Delayed Communication and Coherent Swarm Patterns

Brandon Lindley and Luis Mier-y-Teran-Romero and Ira B. Schwartz This work was
supported by the Office of Naval Research and the National institutes of
Health.Brandon Lindley is an NRC postodctoral fellow at the US Naval Research
Labooratory, Code 6792, Washington, DC 20375 USA,
<EMAIL_ADDRESS>Mier-y-Teran-Romero is an NIH post doctoral
fellow at the Naval Research Laboratory<EMAIL_ADDRESS>B. Schwartz
is at the US Naval Research Labooratory, Code 6792, Washington, DC 20375 USA
<EMAIL_ADDRESS>
###### Abstract

Previously we showed how delay communication between globally coupled self-
propelled agents causes new spatio-temporal patterns to arise when the delay
coupling is fixed among all agents [1]. In this paper, we show how discrete,
randomly distributed delays affect the dynamical patterns. In particular, we
investigate how the standard deviation of the time delay distribution affects
the stability of the different patterns as well as the switching probability
between coherent states.

##  I INTRODUCTION

Numerous recent investigations have been devoted to the study of interacting
multi-agent or swarming systems in various natural and engineering fields of
study. Investigations of interacting systems have revealed the emergence of
highly complex dynamic behaviors in space and time which arise even though the
dynamics of a single agent is quite simple [2]. In particular, these multi-
agent swarms can self-organize in complicated spatio-temporal patterns that
depend on the details of the inter-agent interactions. These investigations
have been motivated by and had an impact on many diverse biological systems
such as bacterial colonies, schooling fish, flocking birds, swarming locusts,
ants, and pedestrians [3, 4, 5, 6, 7]. In this paper, we are interested in the
application that biological analogies have on the design of systems of
autonomous, inter-communicating robotic systems [8, 9, 10, 11] and mobile
sensor networks [12].

There is great interest to design agent-interaction protocols to carry out
robotic motion planning, consensus and cooperative control, and spatio-
temporal formation. One methodology is to combine inter-agent potentials with
external ones in order to achieve multi-agent cooperative motion in a manner
that is not too sensitive with respect the number of agents. Some important
applications making use of scalable numbers of agents are: obstacle avoidance
[10], boundary tracking [13, 14], environmental sensing [12, 15],
decentralized target tracking [16], environmental consensus estimation [12,
17] and task allocation [18].

Authors have employed very diverse approaches in the study of multi-agent
systems. Some authors have described the swarms at the individual level,
writing their models in terms of ordinary differential equations (ODEs) or
delay differential equations (DDEs) to describe their trajectories [19, 20,
9]. The addition of noise on the swarm’s dynamics introduces even richer
behavior, such as noise-induced transitions between different coherent
patterns [2, 1]. The study of noisy swarm dynamics has benefited from tools
from statistical physics applied to both first and second order phase
transitions that have been found in the formation of coherent states [21].

One important aspect of the understanding and design of space-time behavior in
communicating robotic systems is that of time delay. Time delay arises in
latent communication between agents, as well as actuation lag times due to
inertia. Time delays can have interesting and surprising dynamical
consequences in a system, such as large-scale synchronization [22, 23, 24],
and have been used successfully for control purposes [25, 26]. Many of the
initial time-delay studies focused on the case of one or a few discrete time
delays. Recently, more complex situations have been considered such as the
case of having several [27] and random time-delays [28, 29]. Another
interesting case is that of distributed time delays, i.e. when the dynamics of
the system depends on a continuous interval in its past instead of on a
discrete instant [30].

In the case of swarming systems in stochastic environments, it has been
observed that the introduction of a discrete communication time delay induces
a transition from one spatio-temporal pattern to another as the time delay
passes a certain threshold [1]. It was shown in [1] how the complex interplay
exists between the attractive coupling and the time delay in the transitions
between different spatio-temporal patterns [31, 32]. Time delays in robotic
systems have been also studied in the contexts of consensus estimation [17]
and task allocation [18]; in the latter, the time delays originate from the
period of time required to switch between different tasks.

In this paper, we consider a swarming model with discrete, randomly
distributed time delays. We explicitly show how a distribution of delay times
perturb the dynamics from the single discrete case delay case analytically. We
illustrate the dynamical effects of delay distributions with varying width and
show that the system is bistable, and very sensitive to choice of initial
starting conditions.

##  II Swarm Model

We investigate the dynamics of a two-dimensional system of $N$ identical self-
propelling agents that are attracted to each other in a symmetric manner. We
consider the attraction between agents to occur in a time-delayed fashion, due
to the finite communication speeds and information-processing times.
Specifically, we focus on the situation in which the time-delay is nonuniform
across agents: there is one time delay for every pair of agents
$\tau_{ij}(=\tau_{ji})$, for particles $i$ and $j$. The time delays
$\tau_{ij}$’s are time-independent and are drawn independently from a random
distribution $\rho_{\tau}(\tau)$. The swarm dynamics are described by the
following governing equations:

$\displaystyle\dot{\mathbf{r}}_{i}=$ $\displaystyle\mathbf{v}_{i},$ (1a)
$\displaystyle\dot{\mathbf{v}}_{i}=$
$\displaystyle\left(1-|\mathbf{v}_{i}|^{2}\right)\mathbf{v}_{i}-\frac{a}{N}\mathop{\sum_{j=1}^{N}}_{i\neq
j}(\mathbf{r}_{i}(t)-\mathbf{r}_{j}(t-\tau_{ij})),$ (1b)

for $i=1,2\ldots,N$. The position and velocity of the $i$th agent at time $t$
are denoted by $\mathbf{r}_{i}$ and $\mathbf{v}_{i}$, respectively. Each agent
has self-propulsion and frictional drag forces given by the expression term
$\left(1-|\mathbf{v}_{i}|^{2}\right)\mathbf{v}_{i}$. The coupling constant $a$
measures the strength of the attraction between agents and the communication
time delay between particles $i$ and $j$ is given by $\tau_{ij}$. Note that in
the absence of coupling agents tend to move in a straight line with unit speed
as time tends to infinity.

##  III Mean Field Approximation

We carry out a mean field approximation of the swarming system by switching to
particle coordinates relative to the center of mass and disregarding the noise
terms. The center of mass of the swarming system is given by

$\displaystyle\mathbf{R}(t)=\frac{1}{N}\sum_{i=1}^{N}\mathbf{r}_{i}(t).$ (2)

We can decompose the position of each particle into

$\displaystyle\mathbf{r}_{i}=\mathbf{R}+\delta\mathbf{r}_{i},\qquad
i=1,2\ldots,N,$ (3)

where we’ll have

$\displaystyle\sum_{i=1}^{N}\delta\mathbf{r}_{i}(t)=0.$ (4)

Inserting Eq. (3) into the second order system equivalent to Eq. (1) and
simplifying we get

$\displaystyle\ddot{\mathbf{R}}+\delta\ddot{\mathbf{r}}_{i}=$
$\displaystyle\left(1-|\dot{\mathbf{R}}|^{2}-2\dot{\mathbf{R}}\cdot\delta\dot{\mathbf{r}}_{i}-|\delta\dot{\mathbf{r}}_{i}|^{2}\right)(\dot{\mathbf{R}}+\delta\dot{\mathbf{r}_{i}})$
$\displaystyle-\frac{a(N-1)}{N}\bigg{(}\mathbf{R}(t)+\delta\mathbf{r}_{i}(t)\bigg{)}$
$\displaystyle+\frac{a}{N}\mathop{\sum_{j=1}^{N}}_{i\neq
j}\left(\mathbf{R}(t-\tau_{ij})+\delta\mathbf{r}_{j}(t-\tau_{ij})\right),$ (5)

Summing Eq. (III) over $i$ and using Eq. (4), we get

$\displaystyle\ddot{\mathbf{R}}=$
$\displaystyle\left(1-|\dot{\mathbf{R}}|^{2}-\frac{1}{N}\sum_{i=1}^{N}|\delta\dot{\mathbf{r}}_{i}|^{2}\right)\dot{\mathbf{R}}$
$\displaystyle-\frac{1}{N}\sum_{i=\
1}^{N}\left(2\dot{\mathbf{R}}\cdot\delta\dot{\mathbf{r}}_{i}+|\delta\dot{\mathbf{r}}_{i}|^{2}\right)\delta\dot{\mathbf{r}_{i}}$
$\displaystyle-a\frac{N-1}{N}\mathbf{R}(t)+\frac{a}{N^{2}}\sum_{i=1}^{N}\mathop{\sum_{j=1}^{N}}_{i\neq
j}\left(\mathbf{R}(t-\tau_{ij})+\delta\mathbf{r}_{j}(t-\tau_{ij})\right).$ (6)

We now make some approximations on the terms with the double sums. For the
displacements from the center of mass, we have

$\displaystyle\frac{a}{N^{2}}$
$\displaystyle\sum_{i=1}^{N}\mathop{\sum_{j=1}^{N}}_{i\neq
j}\delta\mathbf{r}_{j}(t-\tau_{ij})=\frac{a(N-1)}{N^{2}}\sum_{j=1}^{N}\frac{1}{N-1}\mathop{\sum_{i=1}^{N}}_{i\neq
j}\delta\mathbf{r}_{j}(t-\tau_{ij})$
$\displaystyle\approx\frac{a(N-1)}{N^{2}}\int_{0}^{\infty}\sum_{j=1}^{N}\delta\mathbf{r}_{j}(t-\tau)\rho_{\tau}(\tau)d\tau=0,$
(7)

since $\sum_{j=1}^{N}\delta\mathbf{r}_{j}(t-\tau)=0$ by Eq. (4). In passing
from the discrete to the continuous averaging above, we argue as follows. The
expression $\frac{1}{N-1}\mathop{\sum_{i=1}^{N}}_{i\neq
j}\delta\mathbf{r}_{j}(t-\tau_{ij})$ is the average of
$\delta\mathbf{r}_{j}(t)$ at the $N-1$ times $t-\tau_{ij}$. Since $N\gg 1$ and
the times $\tau_{ij}$ are distributed with density $\rho_{\tau}(\tau)$, this
is approximately equal to
$\int_{0}^{\infty}\delta\mathbf{r}_{j}(t-\tau)\rho_{\tau}(\tau)d\tau$.

Similarly,

$\displaystyle\frac{a}{N^{2}}\sum_{i=1}^{N}\mathop{\sum_{j=1}^{N}}_{i\neq
j}\mathbf{R}(t-\tau_{ij})\approx\frac{a(N-1)}{N}\int_{0}^{\infty}\mathbf{R}(t-\tau)\rho_{\tau}(\tau)d\tau.$
(8)

In a purely heuristic manner, we neglect all fluctuation terms
$\delta\mathbf{r}_{j}(t)$ in the dynamics of the center of mass and obtain the
mean field approximation:

$\displaystyle\ddot{\mathbf{R}}=$
$\displaystyle\left(1-|\dot{\mathbf{R}}|^{2}\right)\dot{\mathbf{R}}-a\left(\mathbf{\
R}(t)-\int_{0}^{\infty}\mathbf{R}(t-\tau)\rho_{\tau}(\tau)d\tau\right).$ (9)

where we approximated $\frac{N-1}{N}\approx 1$, since we are considering large
numbers of agents.

##  IV Bifurcations in the Mean Field Equation

The behavior of the system in the mean field approximation in different
regions of parameter space may be better understood by using bifurcation
analysis. This mathematical technique will allow us to show how the parameter
plane of coupling constant $a$ and mean time delay $\mu_{\tau}$ is divided
into regions with different dynamical behaviors.

First we show that Eq. (9) has a uniformly translating solution
$\mathbf{R}(t)=\mathbf{R}_{0}+\mathbf{V}_{0}\cdot t$, where $\mathbf{R}_{0}$
and $\mathbf{V}_{0}$ are constant, two-dimensional vectors. Inserting the
uniformly translating state into Eq. (9), we get

$\displaystyle 0=$
$\displaystyle\left(1-|\mathbf{V}_{0}|^{2}\right)\mathbf{V}_{0}-a\int_{0}^{\infty}\tau\rho_{\tau}(\tau)d\tau\mathbf{\
V}_{0},$ (10)

since $\int_{0}^{\infty}\rho_{\tau}(\tau)d\tau=1$. Hence, the speed
$|\mathbf{V}_{0}|$ of the uniformly translating state must satisfy

$\displaystyle|\mathbf{V}_{0}|^{2}=1-a\int_{0}^{\infty}\tau\rho_{\tau}(\tau)d\tau=1-a\mu_{\tau},$
(11)

where $\mu_{\tau}$ is the mean of the $\rho_{\tau}$ distribution. We note that
the direction of motion and starting point $\mathbf{R}_{0}$ are arbitrary.

The other state of interest is the stationary state
$\mathbf{R}(t)=\mathbf{R}_{0}$, for an arbitrary constant vector
$\mathbf{R}_{0}$. In the two-parameter space $(a,\mu_{\tau})$, the hyperbola
$a\mu_{\tau}=1$ is in fact a pitchfork bifurcation line on which the uniformly
translating states are born from the stationary state.

The linear stability of the stationary state is determined by the solutions to
the characteristic equation of Eq. (9):

$\displaystyle\mathcal{D}(\lambda)=a\left(1-\int_{0}^{\infty}\rho_{\tau}(\tau)e^{-\lambda\tau}d\tau\right)-\lambda+\lambda^{2}=0,$
(12)

and so involves the Laplace transform of the distribution $\rho_{\tau}$.

In our numerical simulations of system (1), we considered a truncated Gaussian
distribution:

$\displaystyle\rho_{\tau}=\begin{cases}{\cal
N}e^{\frac{(\tau-\tau_{0})^{2}}{2\tau_{1}^{2}}}&\text{if }\tau\geq 0\\\
0&\text{if }\tau<0,\end{cases}$ (13)

where ${\cal N}$ is the normalization constant. Note that because of the
truncation, $\tau_{0}$ and $\tau_{1}$ are only approximately equal to the mean
and standard deviation of $\rho_{\tau}$ and ${\cal N}$ is only approximately
$1/\sqrt{2\pi\tau_{1}^{2}}$.

We approximate the Laplace transform of the truncated Gaussian distribution by
extending the integration range to the whole real line and taking ${\cal
N}\approx\sqrt{2\pi\tau_{1}^{2}}$. In addition, we approximate the mean and
standard deviation of $\rho_{\tau}$ as $\mu_{\tau}\approx\tau_{0}$ and
$\sigma_{\tau}\approx\tau_{1}$, respectively. The result is

$\displaystyle\int_{0}^{\infty}\rho_{\tau}(\tau)e^{-\lambda\tau}d\tau\approx
e^{\lambda\mu_{\tau}+\lambda^{2}\sigma_{\tau}^{2}/2}.$ (14)

We use the above approximation to the Laplace transform of $\rho_{\tau}$ to
search for Hopf bifurcation curves in the $(a,\ \mu_{\tau})$ plane, by taking
$\lambda=i\omega$ in the characteristic equation (12). The equation
$\mathcal{D}(i\omega)=0$ is equivalent to:

$\displaystyle
a-\omega^{2}-i\omega=ae^{i\omega\mu_{\tau}-\omega^{2}\sigma_{\tau}^{2}/2},$
(15)

from which we obtain the Hopf curves parameterized by $\omega$:

$\displaystyle a_{H}(\omega)=$
$\displaystyle\frac{\omega\left(\omega\pm\sqrt{e^{-\sigma_{\tau}^{2}\omega^{2}}(1+\omega^{2})-1}\right)}{1-e^{-\sigma_{\tau}^{2}\omega^{2}}},$
(16a) $\displaystyle\mu_{\tau H}(\omega)=$
$\displaystyle\frac{1}{\omega}\left(\arctan\left(\frac{\omega}{a_{H}(\omega)-\omega^{2}}\right)+2n\pi\right),\
n=0,1,\ldots.$ (16b)

In the above expression for $\mu_{\tau H}(\omega)$, the branch of tan in
$(0,\pi)$ should be used, since the complex number on the left hand side of
Eq. (15) is always on the top half plane. This family of Hopf curves labeled
by $n$, together with the pitchfork bifurcation curve $a\mu_{\tau}=1$ are
shown in Figure 1, for various values of $\sigma_{\tau}$.

Figure 1: Hopf (blue) and pitchfork (red) branches in $a$ and $\mu_{\tau}$
space. The standard deviations of the time-delay distribution $\rho_{\tau}$
for the panels (a) through (d) are 0, 0.2, 0.4 and 0.6, respectively. Note the
change of scale in the abscissae.

When $\sigma_{\tau}=0$, the system exhibits a degenerate point at $a=1/2$,
$\mu_{\tau}=2$ (Fig. 1), where the Hopf bifurcation frequency becomes zero.
This is similar, but not equivalent to a Bogdanov-Takens bifurcation as is
known from previous work [31, 32]. Since the point on the Hopf curve in a two-
parameter bifurcation plane occurs when the Hopf frequency becomes zero, we
define this point as a Zero Frequency Hopf (ZFH) point.

For $\sigma_{\tau}>0$, this ZFH point shifts and a second ZFH point appears at
$a\rightarrow\infty$ and $\mu_{\tau}\rightarrow 0$. The location of the two
ZFH points in the $(a,\ \mu_{\tau})$ plane is given by
$(a_{{ZFH}}^{(\pm)},1/a_{{ZFH}}^{(\pm)})$, where
$a_{{ZFH}}^{(\pm)}=\frac{1}{\sigma_{\tau}^{2}}(1\pm\sqrt{1-\sigma_{\tau}^{2}})$.

When $\sigma_{\tau}=0$, the behavior of the mean field in the vicinity of the
ZFH point is relatively well understood [32, 31] and is as follows (see Fig.
1). In the region between the pitchfork and the first member of the Hopf
family, the stationary state is stable. A simulation of the full system (1)
with parameters in this area reveals that indeed the center of mass of the
agents comes to rest as time progresses and the particles spread themselves
along a ring with radius $1/\sqrt{a}$. Roughly half of the particles move
clockwise and the other half counterclockwise. Along the first Hopf curve, a
stable limit cycle is born and the center of mass begins to oscillate
periodically on a circular orbit. Below the pitchfork bifurcation curve
$a\mu_{\tau}=1$, the translating state is stable. Finally, we mention that
there is a region of bistability in the parameter region above the ZFH point
$(1/2,\ 2)$ between the pitchfork curve $a\mu_{\tau}=1$ and the curve
$a\mu_{\tau}^{2}=2$ (not shown), where the center of mass can either translate
or rotate. On the curve $a\mu_{\tau}^{2}=2$ there is a global bifurcation
where the radius of the orbit diverges and the limit cycle disappears.

The above discussion helps us understand the bifurcation planes in Figs. 1
through 1. Most significantly, we see that the parameter region where the
stationary state is stable decreases in size as the width of the time delay
distribution widens. Hence the system has a higher tendency to behave in an
oscillatory manner for wider time delay distributions. This effect has been
corroborated in numerical simulations (results not shown).

##  V Numerical Simulations

We analyze the dynamics of system (1) by solving the system of DDEs
numerically. We use Heun’s method together with quadratic Lagrange
interpolation to evaluate the time-delayed terms of Eqs. (1). Overall, the
numerical method is second order with respect to the step-size $\Delta t$. For
all simulations we take the agents to be uniformly distributed in a random
fashion within the unit box $0\leq x\leq 1$ and $0\leq y\leq 1$, and each
particle is initially at rest $\mathbf{v}_{j}=0$. Moreover, since we are
interested in investigating the time-asymptotic behavior, for all numerical
experiments the time of integration is long enough to allow transients to
decay.

In [31, 32] it was shown that for the parameter set $a=2$, $\tau=2$ (fixed
delay) that the system exhibited a bistable set of solutions. In the rotating
state solution, all particles collapse to a point and that cluster of
particles rotates around a fixed center in a circular orbit. The other
possible stable solution is a ring state, in which all particles distribute
themselves uniformly along a circle and orbit around its center at unit speed.
Interestingly, not all particles traverse the ring in the same direction;
roughly half move clockwise and half move anti-clockwise. We will now examine
these two states, but with random delays given by the truncated Gaussian
distribution in Eq. (13).

Figure 2: Two stable attractors for the swarm dynamics. Here $a=2$,
$\mu_{\tau}=2$, and $\sigma_{\tau}=0.15$. The number of particles is set to be
$N=150$. The final state shown for both simulations is $t=300$. Panel (a)
depicts the rotating state at three snapshots at times $t=$297.6, 298.8, 300,
in red, green and blue, respectively. Panel (b) depicts the ring state.

Figure 2 shows the two final particle distributions after transients in a
simulation with an initial state of $N=150$ randomly placed particles, and
where $\mu_{\tau}=2$ and $\sigma_{\tau}=0.15$. In this case, depending upon
the random selection of delays, either stable solution (ring or rotating) is
possible. To understand the effects of increasing the standard deviation of
the random delays, we use a Monte Carlo method. At 100 different values of
$\sigma_{\tau}$ in the range $0\leq\sigma_{\tau}\leq 0.5$, we generate random
time delays from the distribution in Eq. (13), we then simulate the system
starting from the same initial condition and we determine what state is
acquired by the swarm in the long-time limit. To determine this, we first
measure the time-averaged distance of particle $j$ to the center of mass over
the interval $(t_{1},\ t_{2})$:

$\langle\delta\mathbf{r}_{j}\rangle_{(t_{1},t_{2})}=\frac{1}{t_{2}-t_{1}}\int_{t_{1}}^{t_{2}}|\delta\mathbf{r}_{j}(t)|dt$
(17)

where the size of the interval $(t_{1},\ t_{2})$ is long enough to include
several periods of oscillation. The ensemble average of Eq. (17) is then

$\langle\delta\mathbf{r}\rangle_{(t_{1},t_{2})}=\frac{1}{N}\sum_{j=1}^{N}\langle\delta\mathbf{r}_{j}\rangle_{(t_{1},t_{2})}.$
(18)

A value $\langle\delta\mathbf{r}\rangle_{(t_{1},t_{2})}\sim 1/\sqrt{a}$ will
indicate111When the delays are uniform, the ring state has a radius of
$1/\sqrt{a}$ [32]. that the system has converged to the ring state, while
$\langle\delta\mathbf{r}\rangle_{(t_{1},t_{2})}\ll 1/\sqrt{a}$ shows that the
rotating state has been adopted instead222This is true for the range of values
of $\sigma_{\tau}$ considered here..

Figure 3 demonstrates the effect of increasing $\sigma_{\tau}$ on the final
state. The blue circles show that for $\sigma_{\tau}$ small, this initial
condition converges to the rotating state. However, for $\sigma_{\tau}\gtrsim
0.2$ the same initial collection of particles will converge to the ring state
with high probability. In between, there is a transition region where both
states are commonly observed; the state that occurs depends on the random
choice of time delays. The black dashed lines of 3 show two simulations, one
which starts near the rotating state (the lower curve), and one which starts
near the ring state (the upper curve) as $\sigma_{\tau}$ is increased. These
curves demonstrate the stability of these steady states, and the effect of
random delays near these states.

Figure 3 shows the conditional probability of ending up in the ring state as a
function of $\sigma_{\tau}$. As expected, for this choice of initial
conditions, for $\sigma_{\tau}$ small enough, there is zero probability of
leaving the rotating state; however, as $\sigma_{\tau}$ is increased, the
probability increases to one.

The results of these numerical studies strongly suggest that even though there
is bi-stability between the ring and the rotating states, the size of their
respective basins of attraction is changing dramatically as the standard
deviation $\sigma_{\tau}$ increases.

Figure 3: As $\sigma_{\tau}$ is increased, we see a bifurcation from the
stable rotating state. Panel (a) captures the transition from the rotating
state to the ring state as the standard deviation of the random delay
increases. Panel (b) shows the probability of converging to the ring state for
a given $\sigma_{\tau}$ of the delays. These results were compiled using a
Monte Carlo simulation with 100 random distributions of delays for 100
uniformly-spaced values of $\sigma_{\tau}$ and for $N=50$ particles. See
accompanying online movie and Appendix to see the agents converge to each
stable pattern.

##  VI Discussion

In this paper we studied the dynamics of a self-propelling swarm with time-
delayed inter-agent attraction. In contrast to the previously considered case
of uniform time delay across agents, we considered the situation in which the
time delay between every pair of agents is drawn randomly from a distribution
$\rho_{\tau}$.

Using a mean-field model of the swarm, we showed how the two parameter
bifurcation plane of coupling strength and mean time delay changes with
respect to the case in which all time delays are equal. The full implications
of these bifurcation results are the subject of our ongoing work. In
particular, it is unclear what the stable solutions are. Nevertheless, the
dramatic changes seen in the two parameter bifurcation plane as the standard
deviation $\sigma_{\tau}$ increases suggest that the basins of attraction of
each attractor undergo big changes as well.

Our numerical experiments show that the swarm displays bi-stable behavior
between the ring and rotating states, at the parameters considered.
Interestingly, however, our work suggests that the basin of attraction of the
ring state greatly expands as the distribution of time-delays $\rho_{\tau}$
widens. Thus, in a sense, widening the distribution of time-delays stabilizes
the stationary state of the swarm center of mass.

Even though in our model the attractive force among agents is linear, we
believe this work is useful since it represents a first approximation for
other, more general forms of attractive interaction. Here, we have limited our
focus to the case where the delays between agents are symmetric and constant.
However, one important generalization of this system involves incorporating
time dependent delays, including those which vary as a function of the
distance between the two agents. This particular refinement of our model is
the subject of ongoing work and beyond the scope of the current paper.

Finally, although we did not consider repulsion between agents, preliminary
research leads us to believe that the patterns observed in this investigation
persist when the characteristic repulsion strength between robots is small
compared to global attraction parameters. For these reasons, our results
indicate how to exploit time-delayed actuation when designing swarm robotic
systems with desired tasks and functionalities.

##  VII Appendix-Video Description

The purpose of this research is to investigate the effects of randomized
communication delay on emerging patterns in swarming dynamics. This short
video captures the transition between two different stable patterns for a
swarm as a function of the standard distribution of the delays.

The two coordinate axes in the video show a scatter plot of the positions of
the particles animated in time. The initial positions are identically randomly
distributed particles in the unit box. The temporal state of the swarm is
updated in time using a numerical scheme called Heun’s Method, and a snapshot
is captured at every discrete time interval. Here, the left coordinate axis
uses a standard deviation of the delays $\sigma_{\tau}=0.1$ while the right
axis uses a standard deviation of $\sigma_{\tau}=0.3$. The mean delay for each
simulation is set to be $\mu_{\tau}=2$, and the number of particles for both
is $N=50$. The vectors at each particle give the velocity associated with that
particle. The two simulations are run side by side to demonstrate the dynamics
involved in converging to the “rotating” final state on the left, and the
“ring” final state on the right. The video demonstrates the dynamics over the
time interval from $t=0$ to $t=45$, and so includes transients.

##  VIII ACKNOWLEDGMENTS

The authors gratefully acknowledge the Office of Naval Research for their
support. LMR and IBS are supported by Award Number R01GM090204 from the
National Institute Of General Medical Sciences. The content is solely the
responsibility of the authors and does not necessarily represent the official
views of the National Institute Of General Medical Sciences or the National
Institutes of Health. E.F. is supported by the Naval Research Laboratory
(Award No. N0017310-2-C007).

## References

  * [1] E. Forgoston and I. Schwartz, “Delay-induced instabilities in self-propelling swarms,” Phy. Rev. E, vol. 77, 2008.
  * [2] U. Erdmann and W. Ebeling, “Noise-induced transition from translational to rotational motion of swarms,” Phys. Rev. E, vol. 71, 2005.
  * [3] E. Budrene and H. Berg, “Dynamics of formation of symmetrical patterns by chemotactic bacteria,” Nature, vol. 376, no. 6535, pp. 49–53, 1995.
  * [4] J. Toner and Y. Tu, “Long-range order in a two-dimensional dynamical xy model: How birds fly together,” Phys. Rev. Lett., vol. 75, no. 23, pp. 4326–4329, 1995.
  * [5] J. K. Parrish, “Complexity, pattern, and evolutionary trade-offs in animal aggregation,” Science, vol. 284, pp. 99–101, Apr 1999.
  * [6] C. Topaz and A. Bertozzi, “Swarming patterns in a two-dimensional kinematic model for biological groups,” SIAM Journal on Applied Mathematics, vol. 65, no. 1, pp. 152–174, 2004.
  * [7] D. Hebling and P. Molnar, “Social force model for pedestrian dynamics,” Phys. Rev. E., vol. 51, pp. 4282–4286, 1995.
  * [8] N. Leonard and E. Fiorelli, “Virtual leaders, artificial potentials and coordinated control of groups,” in Proc. of the 40th IEEE Conference on Decision and Control., vol. 3, pp. 2968–2973, 2002.
  * [9] E. Justh and P. Krishnaprasad, “Steering laws and continuum models for planar formations,” in Proc. of the 42nd IEEE Conference on Decision and Control., vol. 4, pp. 3609–3614, 2004.
  * [10] D. Morgan and I. Schwartz, “Dynamic coordinated control laws in multiple agent models,” Phys. Lett. A, vol. 340, no. 1-4, pp. 121–131, 2005.
  * [11] Y.-L. Chuang, Y. R. Huang, M. R. D’Orsogna, and A. L. Bertozzi, “Multi-vehicle flocking: Scalability of cooperative control algorithms using pairwise potentials,” in Proc. of the 2007 IEEE International Conference on Robotics and Automation., pp. 2292–2299, 2007.
  * [12] K. M. Lynch, P. Schwartz, I. B. Yang, and R. A. Freeman, “Decentralized environmental modeling by mobile sensor networks,” IEEE Trans. Robotics, vol. 24, no. 3, pp. 710–724, 2008.
  * [13] C. Hsieh, Z. Jin, D. Marthaler, B. Nguyen, D. Tung, A. Bertozzi, and R. Murray, “Experimental validation of an algorithm for cooperative boundary tracking,” in Proc. of the 2005 American Control Conference., pp. 1078–1083, 2005.
  * [14] I. Triandaf and I. B. Schwartz, “A collective motion algorithm for tracking time-dependent boundaries,” Mathematics and Computers in Simulation, vol. 70, pp. 187–202, 2005.
  * [15] B. Lu, J. Oyekan, D. Gu, H. Hu, and H. F. G. Nia, “Mobile sensor networks for modelling environmental pollutant distribution,” Int. J. Sys. Sci., vol. 42, no. 9, SI, pp. 1491–1505, 2011.
  * [16] T. Chung, J. Burdick, and R. Murray, “A decentralized motion coordination strategy for dynamic target tracking,” in Proc. 2006 Conference on International Robotics and Automation, pp. 2416–22, 2006.
  * [17] A. Papachristodoulou and A. Jadbabaie, “Synchronization in oscillator networks with heterogeneous delays, switching topologies and nonlinear dynamics,” in Proc. of the 45th IEEE Conference on Decision and Control, pp. 4307 – 4312, 2007.
  * [18] T. W. Mather and M. A. Hsieh, “Macroscopic modeling of stochastic deployment policies with time delays for robot ensembles,” Int. J. Robot Res., vol. 30, pp. 590–600, APR 2011.
  * [19] T. Vicsek, A. Czirók, E. Ben-Jacob, I. Cohen, and O. Shochet, “Novel type of phase transition in a system of self-driven particles,” Phys. Rev. Lett., vol. 75, no. 6, pp. 1226–1229, 1995.
  * [20] G. Flierl, D. Grünbaum, S. Levins, and D. Olson, “From individuals to aggregations: the interplay between behavior and physics,” J. Theor. Biol., vol. 196, no. 4, pp. 397–454, 1999.
  * [21] M. Aldana, V. Dossetti, C. Huepe, V. Kenkre, and H. Larralde, “Phase transitions in systems of self-propelled agents and related network models,” Phys. Rev. Lett., vol. 98, 2007.
  * [22] A. Englert, S. Heiligenthal, W. Kinzel, and I. Kanter, “Synchronization of chaotic networks with time-delayed couplings: An analytic study,” Phys. Rev. E, vol. 83, APR 26 2011.
  * [23] Z. Zuo, C. Yang, and Y. Wang, “A unified framework of exponential synchronization for complex networks with time-varying delays,” Phys. Lett. A, vol. 374, pp. 1989–1999, APR 19 2010.
  * [24] A. Hache, “Synchronization with sound propagation delays,” EPL, vol. 90, APR 2010.
  * [25] F. M. Atay, “Networks with delays,” in Networks: From Biology to Theory (Feng, J and Jost, J and Qian, MP, ed.), pp. 23–33, 2007.  Sino, German International Conference on Networks - From Biology to Theory, Beijing, Peoples Rep. China, APR 04-08, 2005.
  * [26] K. Konishi, H. Kokame, and N. Hara, “Stabilization of a steady state in network oscillators by using diffusive connections with two long time delays,” Phys. Rev. E, vol. 81, JAN 2010.
  * [27] A. Ahlborn and U. Parlitz, “Controlling spatiotemporal chaos using multiple delays,” Phys. Rev. E, vol. 75, JUN 2007.
  * [28] D. Wu, S. Zhu, and X. Luo, “Cooperative effects of random time delays and small-world topologies on diversity-induced resonance,” EPL, vol. 86, JUN 2009.
  * [29] A. C. Marti, M. Ponce C, and C. Masoller, “Chaotic maps coupled with random delays: Connectivity, topology, and network propensity for synchronization,” Physica A, vol. 371, pp. 104–107, NOV 1 2006\.  9th Latin American Workshop on Nonlinear Phenomena, San Carlos de Bariloche, Argentina, Oct 23-28, 2005.
  * [30] T. Omi and S. Shinomoto, “Can distributed delays perfectly stabilize dynamical networks?,” Phys. Rev. E, vol. 77, APR 2008.
  * [31] L. Mier-y Teran-Romero, E. Forgoston, and I. B. Schwartz, “Noise, bifurcations, and modeling of interacting particle systems.” IROS 2011 conference papers, accepted, 2011.
  * [32] L. Mier-y Teran-Romero, E. Forgoston, and I. B. Schwartz, “Coherent pattern prediction in swarms of delay-coupled agents.” IEEE TRO, submitted, 2011.
  * [33] J. Guckenheimer and P. Holmes, Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields.  Berlin: Springer-Verlag, first ed., 1986.


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext:  ${}^{1\;\;}$State Key Laboratory for Novel Software
Technology, Nanjing University
${}^{2\;\;}$Tencent Inc.
11email: https://stabledrag.github.io/

#  _StableDrag_ : Stable Dragging for Point-based Image Editing

Yutao Cui 1122 Xiaotong Zhao 22 Guozhen Zhang 11 Shengming Cao 22 Kai Ma 22
Limin Wang Corresponding author, E-mail<EMAIL_ADDRESS>
###### Abstract

Point-based image editing has attracted remarkable attention since the
emergence of DragGAN. Recently, DragDiffusion further pushes forward the
generative quality via adapting this dragging technique to diffusion models.
Despite these great success, this dragging scheme exhibits two major
drawbacks, namely inaccurate point tracking and incomplete motion supervision,
which may result in unsatisfactory dragging outcomes. To tackle these issues,
we build a stable and precise drag-based editing framework, coined as
_StableDrag_ , by designing a discirminative point tracking method and a
confidence-based latent enhancement strategy for motion supervision. The
former allows us to precisely locate the updated handle points, thereby
boosting the stability of long-range manipulation, while the latter is
responsible for guaranteeing the optimized latent as high-quality as possible
across all the manipulation steps. Thanks to these unique designs, we
instantiate two types of image editing models including StableDrag-GAN and
StableDrag-Diff, which attains more stable dragging performance, through
extensive qualitative experiments and quantitative assessment on DragBench.

###### Keywords:

Stable dragging Image editing Drscriminative tracking Confident motion
supervision

Figure 1: The comparison between DragGAN/DragDiffusion [50] and our proposed
StableDrag. StableDrag-GAN and StableDrag-Diff are our proposed methods
constructed upon GAN and Diffusion models respectively. Given an image input
(synthetic image by GAN/Diffusion model, or real image), users can assign
handle points (red points) and target points (blue points) to drive the
semantic positions of the handle points to reach corresponding target points.
The example of the Mona Lisa portrait and examples in the last row are the
real-image inputs, while the others are synthetic from StyleGAN2 or Stable
Diffusion-V1.5 [47] models. The examples demonstrate that our method achieves
more precise point-level manipulation and generates higher-quality editing
image than DragGAN and DragDiffusion.

##  1 Introduction

Controllable image editing with generative models [30, 40, 19, 25, 27, 36] has
achieved remarkable achievements in the past few years, which can customize
the generative results for further refinement purposes. Recently the
pioneering DragGAN [43] has largely pushed forward accurate image editing with
interactive point-based manipulation, that is, driving semantic objects based
on user-input handle points toward the corresponding target points. DragGAN
formulates a novel dragging technique, primarily contains motion supervision
and point tracking, where the former supervises the local patches around the
handle points to move towards the target points step by step, while the latter
is responsible for locating the updated handle points at each step.

Despite the great success of DragGAN, its editing ability is still constrained
by the inherent model capacity and generality of generative adversarial
networks. Therefore, recent works [50, 41] resort to diffusion models [23, 13,
48, 46, 16, 20, 42, 49, 17, 31, 28] for high-quality drag-style image editing.
A representative work DragDiffusion [50] explores to adapt the dragging scheme
to diffusion models, i.e., first fine-tuning a LoRA, then optimizing the
latent at a single diffusion step, finally denoising the optimized latent
based on MasaCtrl [7]. For the key component of diffusion latent optimization,
it directly follows the DragGAN’s convention of iteratively conducting motion
supervision and point tracking. We analyze that the current dragging scheme
still suffers from the following issues. i) _Inaccurate point tracking._ These
methods leverage the feature difference as the similarity measurement to track
the updated handle points, which is insufficient to precisely locate the right
ones from the distractors (i.e., the around misleading points with similar
content). Especially in diffusion models, since the features are sampled from
the intermediate diffusion process with much noise injection, the updated
points become increasingly challenging to be distinguished from their local
surroundings. This may lead to unsatisfactory dragging outcomes, as showcased
by the examples of the Mona Lisa portrait and the vase in Fig. 1. ii)
_Incomplete motion supervision._ During the motion supervision process, the
latent may not be adequately optimized at certain steps, resulting in a
deterioration of the manipulation quality (see examples of the elephant and
the woman in Fig. 1) as well as the point tracking drift. In diffusion models,
the latent is more stable and harder to manipulate than GAN’s [50], especially
when fine-tuning the LoRA on a specific image, which may aggravate the
problem.

Considering the aforementioned issues, we argue that there are two primary
principles for designing a more stable dragging framework. First, _a robust
yet efficient point tracking_ method is required, to avoid locating the
incorrect points and increasing much latency, thus enabling the point-based
drag to be precise. Second, we should guarantee _the motion supervision to be
complete_ at each optimization step, so as to keep the editing content as
high-quality as possible across all the manipulation process, and fully
unleash the strong restoring power of generative models. In addition, complete
motion supervision can enhance the similarity between the content of the given
handle points and the updated points, preventing the accumulation of tracking
errors.

Driven by the above analysis, we re-formulate the dragging scheme of point
tracking and motion supervision in DragGAN and DragDiffusion, and present a
more stable dragging framework for point-based image editing, coined as
StableDrag. Specifically, inspired by the success in visual object tracking
[4, 12, 10], we try to derive a simple yet powerful point tracking model, in
the form of a convolution filter, from a discriminative learning loss. This
model is capable of suppressing the tracking confidence score of the
distractor points as well as enhancing that of the handle points. At the
beginning of the manipulation steps, we update the tracking model weights
under the supervision of a tailored similarity learning function. Once the
tracking model is prepared, we employ it, in conjunction with the original
feature difference method for robust and precise point tracking. Notably, this
approach scarcely increases inference latency, since we only need to optimize
the simple tracking model (i.e., a single convolution filter) at the initial
manipulation step. Furthermore, we design a confidence-based latent
enhancement strategy, to make motion supervision complete enough at each step.
In detail, we utilize the tracking confidence score of the handle points to
assess the quality of the current manipulation process. Normally, we use the
same manner of motion supervision as DragDiffusion. Nevertheless, when the
quality score falls below an acceptable threshold, we employ the template
features (i.e., the initial features of the given start handle points) to
supervise that of the current handle points’ content, until its confidence
score is satisfactory. Thanks to the unique designs for dragging scheme, we
instantiate two types of image editing models including StableDrag-GAN and
StableDrag-Diff, built on GAN and Diffusion models respectively, which attains
more stable and precise drag performance. Our contributions are summarized as
follows:

  * $\bullet$

We propose a discriminative point tracking method, which allows the model to
accurately distinguish the updated handle points from the distractor ones,
hence promoting the stability of dragging.

  * $\bullet$

We devise a confidence-based latent enhancement strategy for motion
supervision, which can improve the optimization quality at each manipulation
step.

  * $\bullet$

Under these designs, we build StableDrag, a point-based image editing
framework, upon different generative models including GAN and Stable
Diffusion. Through extensive qualitative experiments on a variety of examples
and quantitative assessment on DragBench [50], we demonstrate the
effectiveness of our StableDrag-GAN and StableDrag-Diff.

##  2 Related Work

###  2.1 Image Editing

Image editing is a hot topic with a wide range of applications. Generative
Adversarial Networks (GANs) have made significant strides in the field of
image generation[18, 26], leading to numerous prior image editing
techniques[14, 43, 2, 32, 45, 53] being founded upon the GAN framework.
Nonetheless, the model capacity of GANs remains somewhat constrained, as well
as the challenge of effectively transforming real images into GAN latent
spaces[1, 9, 37], the practicality of these approaches was inevitably
constrained. Recently, large-scale text-to-image diffusion models have
produced remarkably realistic generation results [23, 13, 48, 46, 16, 20, 42,
49, 17, 31, 28], which have given rise to numerous diffusion-based image
editing methods[22, 7, 38, 29, 44, 35, 41, 51, 6, 39, 3, 15]. These techniques
primarily strive to edit images by adjusting the prompts associated with the
image. Nevertheless, as many editing endeavors prove challenging to convey
through text, the prompt-based strategy frequently modifies the image’s high-
level semantics or styles, thereby lacking the capability to achieve precise
pixel-level spatial manipulation.

In order to facilitate fine-grained editing, a number of studies have been
proposed to execute point-based modifications, such as [43, 14, 52]. In
particular, DragGAN has exhibited remarkable dragging-based manipulation
through two straightforward components: the optimization of latent codes to
shift the handle points towards their desired destination points and a point
tracking mechanism to locate the updated handle points. However, its
generality is constrained due to the limited capacity of GAN. DragDiffusion
[50] and DragonDiffusion [41] further extend the dragging scheme to diffusion
models to leverage its excellent generative capacity. FreeDrag [36] has
proposed to improve DragGAN by introducing a point-tracking-free paradigm. In
this work, we explore a new dragging scheme with re-formulating a confident
motion supervision module and a discriminative point tracking module, enabling
stable point-based image editing.

###  2.2 Visual Tracking

Since the proposed discriminative point tracking takes inspiration from the
visual tracking research, we give a brief overview for these methods. We
divide the works into three categories. First, correlation-filter-based
trackers [5, 21, 12] learned an online target-dependent discriminative model
for tracking. [5, 21] employed online correlation filters to distinguish
targets from background and obtains good performance with a high running
speed, which is very practical until now. Second, Siamese-based trackers [4,
34] attract a lot of attention due to its simplicity and efficiency. These
methods combined a correlation operation with the Siamese network, modeling
the appearance similarity and correlation between the target and search.
SiamFC [4] employed a Siamese network to measure the similarity between the
template and the search area with a high tracking speed. SiamRPN++ [33]
improved cross correlation to depth-wise cross correlation, which can increase
both the performance and efficiency. Finally, some recent trackers [10, 8, 11]
introduced a transformer-based integration module to capture the similarity
between the target and search region. Inspired by these findings, we devise a
robust point tracking model via discriminative learning. Different from these
works, we build the tracking model on top of the intermediate feature of GAN
or diffusion models to leverage their discriminativeness and only optimize the
tracking model, which is effective yet efficient.

##  3 Method

###  3.1 Preliminary on Point-based Dragging

Firstly, we briefly review the recent literature on the point-based dragging
framework behind GAN and diffusion models, which are the basics of our work.

#### DragGAN.

Given an image generated by GAN models [27, 24], in conjunction with the user-
input handle points $\\{{p_{i}=(x_{pi},y_{pi}),i=1,2,...,n}\\}$ and the target
points $\\{{t_{i}=(x_{ti},y_{ti}),i=1,2,...,n}\\}$, DragGAN aims to drive the
content at every handle point $p_{i}$ move towards their corresponding target
point ${t_{i}}$. In this sense, the primary concern lies in how to precisely
control the point-level editing while maintaining high image fidelity. To
achieve the goal, DragGAN tailors a novel paradigm, which involves repeated
motion supervision and point tracking. Considering the generator’s
characteristic that the intermediate features are very discriminative, they
leverage a simple online motion supervision loss to optimize the latent code.
When denoting the local region around $p_{i}$ as $\Theta(p_{i})$, i.e, the
pixels whose distance to $p_{i}$ is less than the radius $r_{i}$, the loss can
be defined as:

$\mathcal{L}_{1}=\sum_{i=0}^{n}\|\mathbf{F}(\Theta(p_{i}))-\mathbf{F}(\Theta(p_{i}+d_{i})\|_{1}+\eta\|(\mathbf{F}-\mathbf{F}^{0})\cdot(1-\mathbf{M})\|_{1},$
(1)

where $\mathbf{F}$ represents for the intermediate feature at current
optimization step, $\mathbf{F}^{0}$ is the feature at initial step, $n$ is the
number of handle points, $d_{i}=\frac{t_{i}-p_{i}}{\|t_{i}-p_{i}\|_{2}}$ is a
deviation vector and $\mathbf{M}$ is the pre-defined mask to control the
changing area. Particularly, since the $\mathbf{F}(\Theta(p_{i}))$ gets
detached, the content of current $p_{i}$ will be motivated to $t_{i}$ by a
small step. However, due to the inherent indeterminacy of optimization, it is
hard to guarantee $p_{i}$ to approach $p_{i}+d_{i}$. Consequently, they
utilize a simple feature difference method as point tracking to determine the
updated state of $pi$. The above optimization process iterates until each of
the handle points $p_{i}$ converges to their respective target points $t_{i}$.

#### DragDiffusion.

DragDiffusion [50] extends the point-based editing framework to diffusion
models, such as Stable Diffusion (SD-V1.5 [47]), so as to unleash its strong
power of high stability and superior generation quality. This editing method
involves three sub-processes, i.e., finetuning a LoRA on the real image,
optimizing the latent on a certain diffusion step and denoising the updated
latent to generate the edited image. Specifically, they adopt the same
dragging formulation of repeated motion supervision and point tracking on a
single intermediate diffusion step to manipulate the latent. Besides, a LoRA
finetuing strategy is employed to preserve the image identity through the
whole manipulation process. Finally, a self-attention control mechanism
MasaCtrl [7] is used to enhance the consistency between the original image and
the edited image.

Figure 2: Illustration of our dragging scheme for an intermediate single-step
optimization. The core of the dragging pipeline illustrated herein is based on
GAN, whereas the one based on diffusion models remains the same.
‘Discriminative PT.’ denotes for discriminative point tracking module and
‘Confident MS.’ represents for confident motion supervision process. $P_{i}$
means the current handle point at $i^{th}$ step optimization. Notably, the
tracking model, in the form of a convolution filter, is only learned at the
first optimization step and can be just employed in the subsequent steps.
Details about its learning process at the first step are described in Fig. 3.
The latent code $w$ is supposed to be optimized via the backward updating
across all steps.

###  3.2 Overview

As illustrated in Fig. 1, DragGAN and DragDiffusion may result in deteriorated
editing images due to the imprecise point tracking and incomplete motion
supervision. Therefore, in this work, we cast attention on the current
dragging technique to achieve more stable and precise image manipulation. The
developed dragging pipeline is illustrated in Fig. 2, which comprises a
discriminative point tracking module and a confident motion supervision
module. Specifically, we design a new point tracking approach that integrates
the original feature difference with the tracking score yielded from a learned
discriminative tracking model, thereby boosting the point tracking accuracy as
well as the drag precision. Based on the tracking score, we then explore a
confidence-based latent enhancement strategy to achieve complete enough motion
supervision. We also observe that DragGAN masters large deformation and
creative content (e.g., transforming a lion with its mouth closed into a
roaring state) within a short run-time. While DragDiffusion is good at
generating superior-quality and higher-fidelity editing outcomes. To enable
the dragging model to accommodate a wide range of scenarios, we build
StableDrag upon both DragGAN and DragDiffusion with the designed dragging
scheme. In this section, we will introduce the proposed dragging method in
details.

###  3.3 Discriminative Point Tracking

Point tracking serves as a pivotal function in identifying the updated handle
points $p_{i}$, to circumvent dragging erroneous points and produce
unsatisfactory editing results. The prevalent approach employed in DragGAN and
DragDiffusion is straightforward, that is, conducting nearest neighbor search
by identifying the position with minimal feature difference to the initial
feature template of $p_{0}$. However, this entirely ignores background
appearance information, which is crucial for discriminating the handle points
from the similar ones in the complex scene. Particularly, in diffusion models,
since the supervision features are extracted from the intermediate diffusion
stage, which incorporates substantial noise, it becomes progressively
difficult to discern the updated points. For instance, as shown in the case of
the Mona Lisa portrait of Fig. 1, the handle point of the nose possesses
similar appearance with the adjacent points, which causes the misleading
location in DragDiffusion. Therefore, in this work, we explore an alternative
method for accomplishing more discriminative yet simple point tracking.

Distinguishing the given handle points from the distractors can be addressed
using a learnable discriminative tracking model. In our design, the point
tracking model constitutes the weights of a convolutional layer, providing the
point classification scores as output. In detail, we propose to learn a
function $g(\mathbf{F}(\Theta_{2}),z_{i})$, where $g$ denotes a convolution
function, $\Theta_{2}$ is the local patch around the current handle point $pi$
and $z_{i}$ is the learned tracking model, which returns a high score if the
tracking model $z_{i}$ matches the content at a certain position and discerns
it as the updated handle point $p_{i}$, and a low score otherwise. In
particular, the tracking model $z_{i}$ is learned before the latent
optimization and keep unchanged across all the manipulation steps. In this
sense, this approach scarcely increases the editing runtime. Finally, we merge
the classification score yielded by the tracking model with the original
feature difference score, so as to achieve both discriminative and precise
point location. The detailed procedure of the discriminative point tracking is
illustrated in Fig. 2.

Figure 3: Learning process of our point tracking model. It is only performed
before the manipulation process. The initial feature of the local patch gets
detached, indicating that only the tracking model is supposed to be optimized.
The tracking model weight is initialized with the the template feature
$f_{i}$.

Formally, given the local patch
$\Theta_{2}(p_{i},r_{2})=\\{(x,y)~{}\big{|}~{}\lvert
x-x_{pi}<r_{2}\rvert,\lvert y-y_{pi}<r_{2}\rvert\\}$, the tracked point
$p_{i}$ is updated as:

$\displaystyle
S(\Theta_{2})=\lambda*e^{-\|\mathbf{F}(\Theta_{2})-f_{i}\|_{1}}+(1-\lambda)*g(\mathbf{F}(\Theta_{2}),z_{i}),$
(2) $\displaystyle
p_{i}:=\mathop{\arg\max}_{q_{i}\in\Theta_{2}(p_{i},r_{2})}S(\Theta_{2}(p_{i},r_{2})),$
$\displaystyle
s_{i}=\mathop{\max}_{q_{i}\in\Theta_{2}(p_{i},r_{2})}S(\Theta_{2}(p_{i},r_{2})),$

where $S(\Theta_{2})$ represents for the tracking confidence score map of the
local patch $\Theta_{2}$, $\lambda$ is the weighting factor,
$f_{i}=\mathbf{F}^{0}(p_{i}^{0})$ is the original feature of the initial
handle point $p_{i}^{0}$ at the step-0, and $s_{i}$ is the maximal tracking
confidence score at the current step, which is used to guide the motion
supervision. In the terms of $S(\Theta_{2})$, the former one measures the
feature difference the template and the search region. Although it can provide
accurate point localization in the majority of instances, it may be misled by
the distractor points. Therefore, the second term is responsible to improve
the tracking robustness with the discriminative learning, i.e., suppressing
the score of surrounding points during the initial optimization process for
$z_{i}$. Unlike the plain feature difference method, this tracking model is
capable of leveraging background information and harnessing distinguishing
characteristics of the intermediate feature, thus providing a valuable
enhancement to the original approach.

#### Learning for tracking model $\mathbf{z_{i}}$.

The learning of the point tracking model $z_{i}$, which is a convolutional
filter with the size of $1\times C\times 1\times 1$, is performed before the
manipulation process. Overview of the learning process is shown in Fig. 3. We
use $f_{i}$ to initialize $z_{i}$ and update the weights under the supervision
of the following loss:

$\mathcal{L}_{track}=\|g(\mathbf{F_{0}}(\Theta_{2}(p_{i},r_{2})),z_{i})-y_{i}\|^{2}.$
(3)

Here, $\mathbf{F_{0}}$ denotes the initial feature at step-0, $y_{i}$
represents for the ground-truth label, which is the desired confidence scores
at each position, generally set to a Gaussian function centered at $p_{i}$.
During the learning process, the gradient is not back-propagated through
$\mathbf{F_{0}}(\Theta_{2}(p_{i},r_{2}))$. In other words, we only need to
optimize the tracking model $z_{i}$, allowing for rapid convergence. Through
the optimization, we highlight the handle points while simultaneously
suppressing the confidence score of the background points. Then in the
subsequent manipulation steps, the tracking model $z_{i}$ keeps unchanged for
efficiency.

###  3.4 Confident Motion Supervision

Motion supervision is the core to progressively encourage the points to move
towards their intended destination. DragGAN employs an online loss in equation
(1) to achieve the goal, however may yielding unsatisfactory results in long-
range drag. Alternatively, we devise a confident motion supervision component
based on the tenet that, _not only ensuring high-quality and comprehensive
supervision at each step but also allowing for suitable modifications to
accommodate the novel content creation for the updated states_. For example,
the case of a woman wearing a skirt in Fig. 1 demonstrates the significance of
complete supervision in maintaining visual coherence.

To attain the above goal, we propose a confidence-based latent enhancement
strategy as shown in Fig. 2. Firstly, we introduce the maximal value of the
tracking score, i.e. $s_{i}$, to represent the current supervision confidence,
and the confidence score $s_{1}$ at the step-1 to produce the threshold for
enhancement strategy. Normally, the original motion supervision as in equation
(1) is employed when we discern the current state being confident enough. If
the current confidence score falls below the pre-defined threshold, we resort
to the initial template for supervision. The concrete enhancement supervision
is defined as:

$\displaystyle\mathcal{L}_{2}=$
$\displaystyle\sum_{i=0}^{n}\|\mathbf{F}^{0}(\Theta(p_{i}^{0}))-\mathbf{F}(\Theta(p_{i}+d_{i})\|_{1}$
(4)
$\displaystyle+\eta\|(\mathbf{F}-\mathbf{F}^{0})\cdot(1-\mathbf{M})\|_{1},$

where $\mathbf{F}^{0}(\Theta(p_{i}^{0}))$ is the fixed template with no
gradient back-propagating, which can enforce the content of updated points to
mimic the initial state. Moreover, the choice of whether to use this latent
enhancement supervision is determined according to the following guidelines:

$\mathcal{L}_{motion}=\left\\{\begin{array}[]{lr}\mathcal{L}_{1},&s_{i}>\tau*s_{1},\\\
\mathcal{L}_{2},&s_{i}<=\tau*s_{1},\end{array}\right.$ (5)

where $\tau$ is a threshold rate to control the enhancement strength. In this
way, we can prevent the current content of handle points from significantly
deviate from the original template, thus achieving confident motion
supervision. On the other hand, when the confidence score surpasses the
threshold, we rely on the dynamic motion supervision $\mathcal{L}_{1}$ to
sustain a high editability.

Figure 4: Comparison between FreeDrag [36] and our StableDrag. For the example
in the top left, handle points at each optimization step are visualized to
show the difference of the optimization path of FreeDrag and our StableDrag-
GAN. The example in the bottom left is to demonstrate our method’s strength in
creating novel content. And the others are to show that StableDrag can
generate more precise dragging outcomes.

#### Discussion.

To better expound the insight of the confident motion supervision, we make a
comparison with the method proposed in FreeDrag [36], which employs an
adaptive template and a linear search to set free the point tracking module.
First, The preset linear search in FreeDrag may impose restrictions on the
flexibility of the latent optimization, thereby significantly increasing the
difficulty of dragging. As shown in the top-left example of Fig. 4, The handle
points of FreeDrag frequently oscillate along the predefined path and
necessitate 320 steps of optimization. However, our method allows the handle
points to move towards the destination along _a more optimal path_ , which is
not linear, in only 46 steps. Besides, FreeDrag struggles in generating
creative and out-of-distribution content, as demonstrated by the bottom-left
example in Fig.4, since it primarily relies on a template feature for
supervision, even though an updating strategy is employed. In contrast, our
StableDrag-GAN can generate satisfactory creative content given a long-range
dragging path, demonstrating better editability.

##  4 Experiments

###  4.1 Implementation Details

We implement the approach, including StableDrag-GAN and StableDrag-Diff, based
on PyTorch. During the process of optimizing the latent code $w_{i}$, we use
Adam optimizer with learning rate of 0.01 for StableDrag-Diff and the 0.001
for StableDrag-GAN, which follows their default settings. In most cases, the
hyper-parameters of $\lambda$ and $\tau$ are set to 0.3 and 0.4, respectively.
For other parameters and model settings, we follow the default ones in DragGAN
and DragDiffusion. The experiments are conducted on an NVIDIA V100 GPU.

###  4.2 Qualitative Comparison

Fig. 5 shows the qualitative results between DragGAN and StableDrag-GAN,
DragDiffusion and StableDrag-Diff, FreeDrag-Diff and StableDrag-Diff for fair
comparison. To evaluate the method’s generality, for the GAN-based models, the
input images are generated from StyleGAN2 [27]. While for the Diffusion-based
models, we input real images and use DDIM inversion to reconstruct them. It
can be seen that our method can more precisely move the handle points to the
target points, such as the mountain peak, the lion’s chin, the deer’s forehead
and the little lamp. Besides, our StableDrag can generate higher-quality and
higher-fidelity editing results, for example, maintaining the appearance of
the bag, the glasses, the horse and the Terra Cotta Warriors sculpture. We
also compare our StableDrag-Diff with the FreeDrag [36] based on Diffusion
model. We can see that ours-Diff produces more precise results and maintains
the details of the initial images. This demonstrates the effectiveness of the
proposed discriminative point tracking and confident motion supervision, which
can achieve more stable dragging performance.

Figure 5: Comparison between DragGAN [43]/DragDiffusion [50]/FreeDrag [36] and our StableDrag. As in DragGAN, users can optionally draw a mask of the flexible region (brighter area), keeping the rest of the image fixed. The green dashed box in the examples of the Terra Cotta Warriors Sculpture and the Panda is to show the differences in detail. Best viewed with zooming in. Table 1: Quantitative comparison on DragBench. ‘MD’ denotes Mean Distance $\downarrow$ and ‘IF’ is the Image Fidelity (1-LIPIPS) $\uparrow$. Optimization Steps | 60 | 80 | 100
---|---|---|---
Metric | MD/IF | MD/IF | MD/IF
DragDiffusion | 39.58/0.876 | 37.98/0.868 | 38.86/0.863
StableDrag-Diff |  36.36/0.893 |  36.98/0.884 |  35.92/0.869

###  4.3 Quantitative Results

We quantitatively evaluate our method on DragBench [50], comprising 205
samples with pre-defined drag points and mask. We notice that, in DragBench,
there are many examples that are not compatible with proper StyleGAN2 models,
so we only conduct the experiments on DragDiffusion and ours-Diff. We compare
our StableDrag-Diff to DragDiffusion and use the same LoRA weights and the
common hyper-parameters for fair comparison. As shown in Table 1, under three
different setting of the optimization steps, StableDrag-Diff consistently
outperforms the DragDiffusion, especially surpassing the baseline by 3.22 of
Mean Distance score and 0.017 of Image Fidelity score with 60-step
optimization. This further indicates that our StableDrag can achieve promising
results in editing accuracy and content consistency via the proposed confident
motion supervision and discriminative point tracking.

Figure 6: Effects of the latent enhancement strategy employed in confident
motion supervision component. ‘W/O’ denotes only using the original motion
supervision method as in DragGAN.

Figure 7: Effects of the discriminative point tracking. ‘W/O’ denotes only
using the plain feature difference method.

###  4.4 Exploration Study

To verify the effectiveness and give a thorough analysis on our proposed
method, we perform a detailed ablation study through qualitative visualization
based on both GAN and diffusion models, and quantitative evaluation on
DragBench based on diffusion models.

Table 2: Effects of our discriminative point tracking and confident motion supervision. ‘DPT’ denotes the discirminative point tracking and ‘CMS’ is the confident motion supervision. Metric | Mean Distance $\downarrow$ | Image Fidelity $\uparrow$
---|---|---
DragDiffusion | 39.58 | 0.876
StableDrag-Diff W/O DPT. | 38.63 | 0.895
StableDrag-Diff W/O CMS. | 37.87 | 0.875
StableDrag-Diff | 36.36 | 0.893

#### Confident motion supervision.

Here we study the effect of our confident motion supervision component.
Firstly, we conduct experiments of the horse editing based on StableDrag-GAN.
It can be seen from Fig. 7 that, as the confidence score gradually decreases,
StableDrag without the confident motion supervision module produces low-
quality editing image. This indicates the importance of performing confident
supervision at each step, and also demonstrates that the tracking score can
reflect the quality of motion supervision. As shown in Table 2 ,the image
fidelity decrease by 0.018 when substituting the confident motion supervision
with original supervision method in DragGAN, which further substantiates the
above conclusion.

Table 3: Analysis on time consuming of training tracker and the drag process. Evaluation is performed on StableDrag-Diff. Model | Tracker iters. | Drag steps | Tracker time(s) | Drag time(s)
---|---|---|---|---
StableDrag-Diff | 1000 | 60 | 1.17 | 29.06
StableDrag-Diff | 1000 | 80 | 1.08 | 38.80

#### Discriminative point tracking.

In Fig. 7 and Table 2, we evaluate our StableDrag and the one without the
discriminative tracking model. We can see that StableDrag without the
discriminative tracking model may suffer from misleading by the background
distractor points, causing inaccurate results. Especially, StableDrag-Diff
without our discriminative tracking model increases the StableDrag-Diff by
2.27 of Mean Distance. From the results, we can derive that the proposed
discriminative tracking model helps the dragging model to achieve more
accurate point-based image editing.

#### Practicality of the tracking module.

The proposed point tracker is concise in both formulation and implementation.
As shown in Table. 3, the training process of the tracker (about only 1
second) costs far less time than the drag process. As for the point tracking
before each supervision step, it runs very fast since only a convolution
operation should be performed. It is worth noting that, during the point
tracking process, we use a local search strategy to avoid discerning two
completely similar objects (e.g., two almost identical dog) in global area.
Besides, the core code implementation is simple and easy to adapt to other
related methods, since only around 60-rows code is added to the baseline. And
we will release the code.

#### Sensitivity analysis on $\tau$ and $\lambda$.

To better understand the robustness of the proposed method, we have conducted
sensitivity analysis on $\tau$ and $\lambda$ as in Table 4 and Table 5.
Through the results, we can arrive that, i) the confident motion supervision
is critical for stable dragging and a proper threshold is important, ii)
merging the proposed tracker with the original feature difference can obtain
optimal dragging performance.

Table 4: Sensitivity analysis on $\tau$, where $\lambda$ is fixed to 0.0. $\tau$ | 0.0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0
---|---|---|---|---|---|---
MD/IF | 42.1/0.868 | 41.6/0.874 |  39.8/0.891 | 43.3/0.913 | 47.4/0.939 | 51.2/0.955
Table 5: Sensitivity analysis on $\lambda$, where $\tau$ is fixed to 0.0. $\lambda$ | 0.0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0
---|---|---|---|---|---|---
MD/IF | 42.1/0.868 | 41.6/0.869 | 41.6/0.87 |  37.9/0.875 | 40.7/0.874 | 39.0/0.875
Figure 8: Visualization of the learning process for the tracking model
$z_{i}$.

###  4.5 Visualization of learning process for $z_{i}$

To give a more comprehensive understanding for the discriminative tracking
model, in Fig. 8, we visualize the prediction results of the tracking model
during the learning process. It can be seen that, with the training iterations
increasing, background points (i.e., points away from the center) are
gradually suppressed, resulting in a more robust and discriminative point
tracking model, which can help the dragging model to generate more accurate
editing results.

##  5 Conclusion

We have built a stable drag-based editing framework, coined as StableDrag, by
designing a discirminative point tracking method and a confidence-based latent
enhancement strategy for motion supervision. With the proposed point tracking
method, we can precisely locate the updated handle points, thereby boosting
the stability of long-range manipulation. While the latter can guarantee the
optimized latent as high-quality as possible across all the manipulation
steps. Thanks to the unique designs, we have instantiated two types of models
including StableDrag-GAN and StableDrag-Diff to demonstrate the generality.
Through extensive qualitative and quantitative experiments on a variety of
examples, StableDrag has attained stable and precise drag performance. We
expect our findings and analysis can facilitate the development of precise
image editing.

Figure 9:  More results of our StableDrag-GAN and StableDrag-Diff.

## Appendix

We provide more visualization results of our StableDrag, including StableDrag-
GAN and StableDrag-Diff, which are built upon DragGAN [43] and DragDiffusion
[50] respectively. It can be seen from the Fig. 9, our method can produce
precise and stable dragging performance on a majority of scenarios.
Furthermore, we provide more visualization results and give detailed
comparison between the StableDrag and FreeDrag [36] dragging process in
https://stabledrag.github.io/. Code will be released upon acceptance.

## References

  * [1] Abdal, R., Qin, Y., Wonka, P.: Image2stylegan: How to embed images into the stylegan latent space? In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4432–4441 (2019)
  * [2] Abdal, R., Zhu, P., Mitra, N.J., Wonka, P.: Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows. ACM Transactions on Graphics (ToG) 40(3), 1–21 (2021)
  * [3] Bar-Tal, O., Ofri-Amar, D., Fridman, R., Kasten, Y., Dekel, T.: Text2live: Text-driven layered image and video editing. In: European Conference on Computer Vision. pp. 707–723. Springer (2022)
  * [4] Bertinetto, L., Valmadre, J., Henriques, J.F., Vedaldi, A., Torr, P.H.S.: Fully-convolutional siamese networks for object tracking. In: ECCV Workshops (2016)
  * [5] Bolme, D.S., Beveridge, J.R., Draper, B.A., Lui, Y.M.: Visual object tracking using adaptive correlation filters. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2010)
  * [6] Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18392–18402 (2023)
  * [7] Cao, M., Wang, X., Qi, Z., Shan, Y., Qie, X., Zheng, Y.: Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.08465 (2023)
  * [8] Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., Lu, H.: Transformer tracking. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2021)
  * [9] Creswell, A., Bharath, A.A.: Inverting the generator of a generative adversarial network. IEEE transactions on neural networks and learning systems 30(7), 1967–1974 (2018)
  * [10] Cui, Y., Jiang, C., Wang, L., Wu, G.: Mixformer: End-to-end tracking with iterative mixed attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)
  * [11] Cui, Y., Song, T., Wu, G., Wang, L.: Mixformerv2: Efficient fully transformer tracking. In: Advances in Neural Information Processing Systems (2023)
  * [12] Danelljan, M., Bhat, G., Khan, F.S., Felsberg, M.: ATOM: accurate tracking by overlap maximization. In: CVPR (2019)
  * [13] Dhariwal, P., Nichol, A.: Diffusion models beat GANs on image synthesis. In: NeurIPS (2021)
  * [14] Endo, Y.: User-controllable latent transformer for stylegan image layout editing. arXiv preprint arXiv:2208.12408 (2022)
  * [15] Epstein, D., Jabri, A., Poole, B., Efros, A.A., Holynski, A.: Diffusion self-guidance for controllable image generation. arXiv preprint arXiv:2306.00986 (2023)
  * [16] Esser, P., Rombach, R., Blattmann, A., Ommer, B.: ImageBART: Bidirectional context with multinomial diffusion for autoregressive image synthesis. In: NeurIPS (2021)
  * [17] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-Or, D.: An image is worth one word: Personalizing text-to-image generation using textual inversion. In: ICLR (2023)
  * [18] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K. (eds.) Advances in Neural Information Processing Systems. vol. 27. Curran Associates, Inc. (2014), https://proceedings.neurips.cc/paper_files/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf
  * [19] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A.C., Bengio, Y.: Generative adversarial nets. In: NeurIPS (2014)
  * [20] Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., Guo, B.: Vector quantized diffusion model for text-to-image synthesis. In: CVPR (2022)
  * [21] Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 583–596 (2015)
  * [22] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 (2022)
  * [23] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS (2020)
  * [24] Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., Aila, T.: Alias-free generative adversarial networks. In: NeurIPS (2021)
  * [25] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: CVPR (2019)
  * [26] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4401–4410 (2019)
  * [27] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: CVPR (2020)
  * [28] Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., Irani, M.: Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276 (2022)
  * [29] Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., Irani, M.: Imagic: Text-based real image editing with diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6007–6017 (2023)
  * [30] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
  * [31] Kumari, N., Zhang, B., Zhang, R., Shechtman, E., Zhu, J.Y.: Multi-concept customization of text-to-image diffusion. arXiv preprint arXiv:2212.04488 (2022)
  * [32] Leimkühler, T., Drettakis, G.: Freestylegan: Free-view editable portrait rendering with the camera manifold. arXiv preprint arXiv:2109.09378 (2021)
  * [33] Li, B., Wu, W., Wang, Q., Zhang, F., Xing, J., Yan, J.: Siamrpn++: Evolution of siamese visual tracking with very deep networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2019)
  * [34] Li, B., Yan, J., Wu, W., Zhu, Z., Hu, X.: High performance visual tracking with siamese region proposal network. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2018)
  * [35] Liew, J.H., Yan, H., Zhou, D., Feng, J.: Magicmix: Semantic mixing with diffusion models. arXiv preprint arXiv:2210.16056 (2022)
  * [36] Ling, P., Chen, L., Zhang, P., Chen, H., Jin, Y.: Freedrag: Point tracking is not you need for interactive point-based image editing. arXiv preprint arXiv:2307.04684 (2023)
  * [37] Lipton, Z.C., Tripathi, S.: Precise recovery of latent vectors from generative adversarial networks. arXiv preprint arXiv:1702.04782 (2017)
  * [38] Mao, J., Wang, X., Aizawa, K.: Guided image synthesis via initial image editing in diffusion model. arXiv preprint arXiv:2305.03382 (2023)
  * [39] Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.Y., Ermon, S.: Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 (2021)
  * [40] Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  * [41] Mou, C., Wang, X., Song, J., Shan, Y., Zhang, J.: Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.02421 (2023)
  * [42] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021)
  * [43] Pan, X., Tewari, A., Leimkühler, T., Liu, L., Meka, A., Theobalt, C.: Drag your GAN: Interactive point-based manipulation on the generative image manifold. arXiv preprint arXiv:2305.10973 (2023)
  * [44] Parmar, G., Singh, K.K., Zhang, R., Li, Y., Lu, J., Zhu, J.Y.: Zero-shot image-to-image translation. arXiv preprint arXiv:2302.03027 (2023)
  * [45] Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., Lischinski, D.: Styleclip: Text-driven manipulation of stylegan imagery. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2085–2094 (2021)
  * [46] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125 (2022)
  * [47] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10684–10695 (2022)
  * [48] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022)
  * [49] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S.K.S., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., et al.: Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487 (2022)
  * [50] Shi, Y., Xue, C., Pan, J., Zhang, W., Tan, V.Y., Bai, S.: Dragdiffusion: Harnessing diffusion models for interactive point-based image editing. arXiv preprint arXiv:2306.14435 (2023)
  * [51] Tumanyan, N., Geyer, M., Bagon, S., Dekel, T.: Plug-and-play diffusion features for text-driven image-to-image translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1921–1930 (2023)
  * [52] Wang, S.Y., Bau, D., Zhu, J.Y.: Rewriting geometric rules of a gan. ACM Transactions on Graphics (TOG) 41(4), 1–16 (2022)
  * [53] Zhu, J.Y., Krähenbühl, P., Shechtman, E., Efros, A.A.: Generative visual manipulation on the natural image manifold. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14. pp. 597–613. Springer (2016)