# Statistical properties and lensing effect on the repeating fast radio burst
FRB 180916.J0158+65

Yu-Bin Wang1,2, Abdusattar Kurban1,3,4, Xia Zhou1,3,4 , Yun-Wei Yu5 and Na
Wang1,3,4
1Xinjiang Astronomical Observatory, Chinese Academy of Sciences, Urumqi
830011, China
2University of Chinese Academy of Sciences, 19A Yuquan Road, Beijing 100049,
China
3Key Laboratory of Radio Astronomy, Chinese Academy of Sciences, Nanjing
210008, China
4Xinjiang Key Laboratory of Radio Astrophysics, 150 Science1-Street, Urumqi
830011, China
5Institute of Astrophysics, Central China Normal University, Wuhan 430079,
China E-mail<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

FRB 180916.J0158+65 is a well-known repeating fast radio burst with a period
($16.35~{}\rm days$) and an active window ($5.0~{}\rm days$). We give out the
statistical results of the dispersion measures and waiting times of bursts of
FRB 180916.J0158+65. We find the dispersion measures at the different
frequencies show a bimodal distribution. The peaking dispersion measures of
the left mode of the bimodal distributions increase with frequency, but the
right one is inverse. The waiting times also present the bimodal distribution,
peaking at 0.05622s and 1612.91266s. The peaking time is irrelevant to the
properties of bursts, either for the preceding or subsequent burst. By
comparing the statistical results with possible theoretical models, we suggest
that FRB 180916.J0158+65 suffered from the plasma lensing effects in the
propagation path. Moreover, this source may be originated from a highly
magnetized neutron star in a high-mass X-ray binary.

###### keywords:

pulsars: general $-$ stars: individual (FRB 180916.J0158+65) $-$ transients:
fast radio bursts $-$ interstellar medium $-$ scattering

††pubyear: 2022††pagerange: Statistical properties and lensing effect on the
repeating fast radio burst FRB 180916.J0158+65–Statistical properties and
lensing effect on the repeating fast radio burst FRB 180916.J0158+65

##  1 Introduction

Fast radio bursts (FRBs) are mysterious bright (emitting $\rm\sim Jy$) radio
sources that can release extremely high energy within milliseconds of time
scale. Up to now, more than 600 FRBs have been detected, and 24 of them are
repeaters (Petroff et al., 2016; Luo et al., 2020; Amiri et al., 2021). Their
unusually high dispersion measures (DMs) indicate that they are extragalactic
or cosmic origin rather than Galactic (Thornton et al., 2013). Recently, FRB
20200428 from the magnetar SGR 1935+2154 has been detected the similar
properties as the repeaters (Andersen et al., 2019), and the bursts of FRB
180301 have the analogous polarization angle swings to pulsars (Luo et al.,
2020). These suggest that the repeating sources could be the possibility of
luminous coherent emission processes around pulsars or magnetars (Kumar et
al., 2017; Andersen et al., 2019, 2020; Li et al., 2021a). However, FRBs are
still topical events and keep mysterious in its physical nature.

The repeating FRBs may have two origins, i.e., the isolated neutron stars
(NSs) and the NS binary systems (Platts et al., 2019; Kurban et al., 2022). An
isolated NS requires the accumulation of enough energy to release the next
burst randomly, which presents the propriety of aperiodicity. Thus, the
investigation for waiting time ($\Delta t$), the time interval between two
adjacent bursts within an observational campaign, is pretty important for
understanding the physical nature of FRBs. For example, the waiting time with
unimodal distribution has been found in the bursts activities from the
magnetars in the Milky Way (Cheng et al., 2020; Younes et al., 2020) or the
giant pulses from the isolated pulsar (Abbate et al., 2020; Kuiack et al.,
2020; Geyer et al., 2021). However, the waiting times of FRB 121102 presented
a bimodal distribution, which is uncorrelated with the fluences, peak flux
densities, pulse widths, and high energy components of bursts (Li et al.,
2019; Li et al., 2021b). The waiting times of FRB 20201124A also has a bimodal
distribution (Xu et al., 2022). These indicate that some repeaters, like FRB
121102 and FRB 20201124A, may originate from the activity in the binary
systems (Du et al., 2021; Geng et al., 2021; Wang et al., 2022a) rather than
the isolated NS.

FRB 180916.J0158+65 exhibited similar observational properties to FRB 121102
and FRB 20201124A. For example, some bursts among the three repeaters present
shorter delay times at low frequency than that at high frequency (Chamma et
al., 2021; Platts et al., 2021; Kumar et al., 2022). In addition, they are
located behind the clump of plasma (Tendulkar et al., 2017; Marcote et al.,
2020; Xu et al., 2022); they occupy an optical counterpart (Tendulkar et al.,
2017; Li et al., 2022; Ravi et al., 2022); they have the periodicity and an
active window (Amiri et al., 2020; Rajwade et al., 2020; Mao et al., 2022);
and they produce a delay time of about tens of milliseconds between bursts
(Amiri et al., 2020; Li et al., 2021b; Xu et al., 2022). These indicate FRB
180916.J0158+65 may have similar statistical properties to FRB 121102. Thus,
we give out the statistic of the DM and waiting time of FRB 180916.J0158+65,
which is detected at frequencies from 110 to 5.4 GHz. Coupled with the
statistical results for the repeater, the effects in the propagation path
(gravitational lensing and plasma lens) will be discussed, which may help us
to reveal the physical nature of this source.

This paper is organized as follows. In Section 2, we give out the statistical
properties of FRB 180916.J0158+65, especially for waiting time and DMs. The
lensing effects of the propagation path are discussed in Section 3. Finally, a
summary and discussion are given in Section 4.

##  2 Statistical properties of FRB 180916.J0158+65

FRB 180916.J0158+65 with Galactic longitude $l=129.7^{\circ}$ and latitude
$b=3.7^{\circ}$ is reported a possible $P_{\rm orb}=16.35$ day periodicity and
a 5.0-day phase window approximately. The Milky Way DM contribution is
$200\,\rm pc\,cm^{-3}$ (NE2001, Cordes & Lazio 2002) or $325.23\,\rm
pc\,cm^{-3}$ (YMW16, Yao et al. 2017; Amiri et al. 2020). It is located in a
nearby spiral galaxy with redshift $z_{d}=0.0337\pm 0.0002$ and luminosity
distance $d_{\rm os}=149.0\pm 0.9\,\rm Mpc$ (Marcote et al., 2020). The
projected distance of the repeating source (roughly $4.7\,\rm kpc$) is far
from the core of the host galaxy. There is no other comparable large and
bright galaxy in its observing field of view, but a young stellar clump with
size $\sim\rm 380\,pc$ around the repeater (the source environment $\sim\rm
30-60\,pc$) (Marcote et al., 2020; Tendulkar et al., 2021). Over 195 bursts
have been detected from this source by different instruments at frequencies
ranging from 110 MHz to 5.4 GHz (Amiri et al., 2020; Chawla et al., 2020;
Marcote et al., 2020; Marthi et al., 2020; Pearlman et al., 2020; Pilia et
al., 2020; Pastor-Marazuela et al., 2021; Pleunis et al., 2021; Bethapudi et
al., 2022; Mckinven et al., 2023). These instruments are including the 100-m
Effelsberg Radio Telescope (Effelsberg; 4-5.4 GHz), the European VLBI Network
(EVN; 1636-1764 MHz), the Westerbork Synthesis Radio Telescope (WSRT) with the
Apertif Radio Transient System (Apertif; 1220-1520 MHz), the upgraded Giant
Metrewave Radio Telescope (uGMRT; 550-750 MHz and 250-500 MHz), the Canadian
Hydrogen Intensity Mapping Experiment Fast Radio Burst Project (CHIME/FRB;
400-600 MHz), the Green Bank Telescope (GBT; 300-400 MHz), the Sardinia Radio
Telescope (SRT; 296-360 MHz) and the Low-Frequency Array (LOFAR; 110$-$188
MHz). The pulse widths of these bursts are ranging from a few milliseconds to
about one hundred milliseconds (Amiri et al., 2020; Marthi et al., 2020;
Pleunis et al., 2021).

Table 1: Observational MJD and DMs of FRB 180916.J0158+65.

MJD | Telescope | DM | Ref | MJD | Telescope | DM | Ref | MJD | Telescope | DM | Ref
---|---|---|---|---|---|---|---|---|---|---|---
|  | $\rm pc\,\,cm^{-3}$ |  |  |  | $\rm pc\,\,cm^{-3}$ |  |  |  | $\rm pc\,\,cm^{-3}$ |
58477.16185 | CHIME | $348.732\pm 0.01$ | [1] | 58950.58347838 | LOFAR | $349.09\pm 0.04$ | [6] | 59095.01647024 | Apertif | $348.70\pm 0.40$ | [6]
58478.15521 | CHIME | $348.791\pm 0.023$ | [1] | 58951.54162736 | LOFAR | $349.03\pm 0.05$ | [6] | 59095.03083630 | Apertif | $348.71\pm 0.27$ | [6]
58638.71643 | CHIME | $348.744\pm 0.032$ | [1] | 58951.55801455 | LOFAR | $348.98\pm 0.02$ | [6] | 59095.03119917 | Apertif | $348.73\pm 0.11$ | [6]
58639.70561 | CHIME | $349.89\pm 0.15$ | [1] | 58951.58470795 | LOFAR | $348.99\pm 0.03$ | [6] | 59095.04813576 | Apertif | $348.74\pm 0.16$ | [6]
58639.71008 | CHIME | $348.813\pm 0.072$ | [1] | 58951.59135120 | LOFAR | $348.86\pm 0.08$ | [6] | 59095.06242878 | Apertif | $348.64\pm 0.26$ | [6]
58653.09613665 | EVN | $348.76\pm 0.1$ | [2] | 58978.59561357 | Apertif | $348.63\pm 0.14$ | [6] | 59095.07525644 | Apertif | $348.59\pm 0.34$ | [6]
58653.11125735 | EVN | $348.76\pm 0.1$ | [2] | 58979.50785914 | Apertif | $348.35\pm 0.38$ | [6] | 59095.07913932 | Apertif | $348.10\pm 0.23$ | [6]
58653.14659694 | EVN | $348.76\pm 0.1$ | [2] | 58980.35572077 | Apertif | $348.75\pm 0.26$ | [6] | 59095.10211216 | Apertif | $349.06\pm 0.23$ | [6]
58653.27850789 | EVN | $348.76\pm 0.1$ | [2] | 58980.38590828 | Apertif | $349.44\pm 0.26$ | [6] | 59095.11289895 | Apertif | $348.50\pm 0.28$ | [6]
58786.32075 | CHIME | $348.68\pm 0.46$ | [1] | 58980.44318898 | Apertif | $350.09\pm 0.37$ | [6] | 59095.11989684 | Apertif | $348.78\pm 0.14$ | [6]
58802.25840267 | GBT | $349.3\pm 0.2$ | [3] | 58980.46375074 | Apertif | $347.86\pm 0.54$ | [6] | 59095.12368446 | Apertif | $348.93\pm 0.23$ | [6]
58835.17324 | CHIME | $349.11\pm 0.2$ | [1] | 58980.46995949 | Apertif | $347.28\pm 0.32$ | [6] | 59095.14075045 | Apertif | $349.14\pm 0.51$ | [6]
58836.16929624 | GBT | $348.8\pm 0.4$ | [3] | 58980.47426015 | Apertif | $349.06\pm 0.33$ | [6] | 59095.16236684 | Apertif | $349.29\pm 0.26$ | [6]
58836.16929695 | GBT | $348.8\pm 0.4$ | [3] | 58980.52593337 | Apertif | $348.70\pm 0.56$ | [6] | 59095.19030365 | Apertif | $349.68\pm 0.52$ | [6]
58836.16929720 | GBT | $348.8\pm 0.4$ | [3] | 58980.54629988 | Apertif | $348.07\pm 0.44$ | [6] | 59096.20840871 | Apertif | $348.43\pm 0.54$ | [6]
58836.17198 | CHIME | $350.0527\pm 0.0075$ | [1] | 58980.54684270 | Apertif | $349.52\pm 0.76$ | [6] | 59111.40643 | CHIME | $350.12\pm 0.34$ | [1]
58836.17591788 | GBT | $350.1\pm 0.4$ | [3] | 58980.62542392 | Apertif | $348.78\pm 0.48$ | [6] | 58477.16185 | CHIME | $348.732\pm 0.01$ | [1]
58852.13628 | CHIME | $349.847\pm 0.083$ | [1] | 58980.62998094 | Apertif | $348.68\pm 0.13$ | [6] | 59143.81778929 | Apertif | $348.89\pm 0.19$ | [6]
58852.13773 | CHIME | $349.097\pm 0.061$ | [1] | 58980.65889322 | Apertif | $348.87\pm 0.16$ | [6] | 59190.73164688 | Apertif | $348.76\pm 0.19$ | [6]
58868.07586 | CHIME | $348.98\pm 0.12$ | [1] | 58981.38138907 | Apertif | $350.68\pm 0.32$ | [6] | 59191.74125466 | Apertif | $348.87\pm 0.19$ | [6]
58868.08221442 | GBT | $348.9\pm 0.1$ | [3] | 58981.77662 | CHIME | $349.507\pm 0.084$ | [1] | 59225.10428 | CHIME | $348.99\pm 0.13$ | [1]
58868.08461892 | GBT | $348.7\pm 0.2$ | [3] | 58982.76846813 | CHIME | $352.6\pm 3.2$ | [4] | 59241.06071 | CHIME | $348.8435\pm 0.0098$ | [1]
58868.08679636 | GBT | $348.8\pm 0.2$ | [3] | 58982.77157 | CHIME | $349.46\pm 0.57$ | [1] | 59244.04400 | CHIME | $348.838\pm 0.071$ | [1]
58882.04681 | CHIME | $349.27\pm 0.27$ | [1] | 58996.15501128 | Apertif | $348.81\pm 0.21$ | [6] | 59244.06229 | CHIME | $348.6885\pm 0.002$ | [1]
58883.02020163 | CHIME | $370.4\pm 1.6$ | [4] | 58996.19203445 | Apertif | $348.79\pm 0.21$ | [6] | 59245.05833 | CHIME | $349.47\pm 0.31$ | [1]
58883.03995 | CHIME | $349.37\pm 0.20$ | [1] | 58996.23898191 | Apertif | $348.68\pm 0.14$ | [6] | 59275.96877 | CHIME | $349.542\pm 0.051$ | [1]
58883.04405 | CHIME | $349.725\pm 0.48$ | [1] | 58996.27129126 | Apertif | $348.68\pm 0.23$ | [6] | 59276.96257 | CHIME | $348.83\pm 0.45$ | [1]
58883.05372 | CHIME | $348.840\pm 0.5$ | [1] | 58996.34499129 | Apertif | $350.23\pm 0.84$ | [6] | 59277.96034 | CHIME | $348.988\pm 0.02$ | [1]
58899.00706 | CHIME | $348.73\pm 0.025$ | [1] | 58996.36224320 | Apertif | $348.78\pm 0.44$ | [6] | 59306.89010 | CHIME | $349.60\pm 0.17$ | [1]
58477.16185 | CHIME | $348.732\pm 0.01$ | [1] | 58996.42810299 | Apertif | $349.47\pm 0.29$ | [6] | 59355.74759 | CHIME | $348.93\pm 0.37$ | [1]
58899.56141184 | SRT | $349.8\pm 0.1$ | [5] | 58996.48015176 | Apertif | $348.63\pm 0.25$ | [6] | 59357.74680 | CHIME | $348.95\pm 0.34$ | [1]
58899.56781756 | SRT | $349.4\pm 0.1$ | [5] | 58996.60480633 | Apertif | $348.97\pm 0.28$ | [6] | 59390.65203 | CHIME | $348.97\pm 0.21$ | [1]
58899.57561573 | SRT | $350.1\pm 0.1$ | [5] | 58996.61583838 | Apertif | $348.81\pm 0.43$ | [6] | 59406.60025 | CHIME | $349.38\pm 0.14$ | [1]
58930.47097294 | Apertif | $348.70\pm 0.20$ | [6] | 58997.15492630 | Apertif | $348.87\pm 0.22$ | [6] | 59407.59936 | CHIME | $348.8165\pm 0.0093$ | [1]
58931.51122577 | Apertif | $348.88\pm 0.18$ | [6] | 58997.23883623 | Apertif | $348.24\pm 0.25$ | [6] | 59440.50726 | CHIME | $348.731\pm 0.043$ | [1]
58931.54877968 | Apertif | $349.02\pm 0.59$ | [6] | 58997.26968437 | Apertif | $348.76\pm 0.25$ | [6] | 59440.51585 | CHIME | $349.062\pm 0.012$ | [1]
58931.56964778 | Apertif | $348.70\pm 0.97$ | [6] | 58997.35800780 | Apertif | $348.87\pm 0.17$ | [6] | 59486.38963 | CHIME | $349.79\pm 0.21$ | [1]
58932.550594 | uGMRT | $349.06\pm 0.32$ | [7] | 58997.38837259 | Apertif | $348.75\pm 0.18$ | [6] | 59519.30410 | CHIME | $349.9\pm 0.19$ | [1]
58932.604069 | uGMRT | $348.82\pm 0.10$ | [7] | 58998.15708057 | Apertif | $348.69\pm 0.26$ | [6] | 59519.30470 | CHIME | $348.967\pm 0.023$ | [1]
58932.612024 | uGMRT | $348.90\pm 0.86$ | [7] | 59013.69287 | CHIME | $348.955\pm 0.031$ | [1] | 59519.30909 | CHIME | $348.862\pm 0.014$ | [1]
58949.53491816 | LOFAR | $349.03\pm 0.11$ | [6] | 59014.68533 | CHIME | $348.7917\pm 0.0055$ | [1] | 59521.28871 | CHIME | $348.94\pm 0.19$ | [1]
58949.63987585 | LOFAR | $348.94\pm 0.08$ | [6] | 59031.13727 | uGMRT | $349.5\pm 0.1$ | [7] | 59569.16266 | CHIME | $349.05\pm 0.16$ | [1]
58950.52919335 | LOFAR | $349.02\pm 0.08$ | [6] | 59095.01258701 | Apertif | $348.21\pm 0.42$ | [6] | 59570.15498 | CHIME | $348.901\pm 0.027$ | [1]
58950.54130169 | LOFAR | $349.41\pm 0.03$ | [6] |  |  |  |  |  |  |  |

Ref:[1] Mckinven et al. (2023); [2] Marcote et al. (2020); [3] Chawla et al.
(2020); [4] Pearlman et al. (2020); [5] Pilia et al. (2020); [6] Pastor-
Marazuela et al. (2021); [7] Marthi et al. (2020); Pleunis et al. (2021).

###  2.1 The statistical properties of DM

Figure 1: MJD-DM relation (upper) and histogram distribution of DM (bottom)
with the 0.16 bin size. In the upper panel, the black, cyan, blue, green, red,
brown and purple dots are the observational data from CHIME, EVN, Apertif,
GBT, SRT, uGMRT (650 MHz) and LOFAR, respectively. And the red dashed line in
the upper panel is the linear fitting result.

In this subsection, we give out the statistical properties of DMs of FRB
180916 observed by different telescopes. Their arrival times, telescope names,
DMs, and references are listed in Table 1.

Since all bursts observed by LOFAR with the given DMs are around 160 MHz
(Pastor-Marazuela et al., 2021; Pleunis et al., 2021), we take the 160 MHz as
a center frequency of bursts observed by LOFAR in the following discussions.
The DM variation of the burst ($\Delta\rm DM\sim 21\,pc\,cm^{-3}$) at MJD
58883.02020163 is significantly higher than that of other bursts detected by
CHIME/FRB (Mckinven et al., 2023). However, the DM variations of other
repeaters are within the range of $\rm 20\,pc\,cm^{-3}$ (Li et al., 2021b; Xu
et al., 2022). This means that the abnormal bursts reported by Pearlman et al.
2020 may have a distinctly different origin or de-dispersion algorithm, we
will ignore these bursts here.

Figure 1 shows the MJD-DM relationship as well as a histogram of DMs. We also
give out a linear fitting of the DMs and the slope is $0.028(0.12)\,\rm
pc\,cm^{-3}\,yr^{-1}$, indicating that the DM increases over time. The
Kolmogorov-Smirnov (K-S) test from scipy.stats.kstest is used to examine the
Gaussian property of DMs; their p-value (0.0012) is less than the significant
level of 0.05. To perform a K-S test on the random sampling DMs of all data,
we take the error bar of each DM observation as the standard deviation and
sample 1000 points using a Gaussian random distribution. We discover that the
p-value is also less than the 0.05 level of significance. The total DMs are
not Gaussian distribution but may have multiple components.

To reveal the real distributions, the distribution density of DM with weighted
error and different frequencies have been used to estimate the distributions
of DMs (Yusifov & Küçük, 2004). Using this method, we find a relatively large
difference between the estimated and actual DM, which may be due to a small
number of DM samples. As a result, the upper panel of Figure 2 depicts the
kernel density estimation (KDE) of randomly sampled DMs at different
frequencies. The sub-plot demonstrates the KDEs of DMs between $350.3\,\rm
pc\,cm^{-3}$ and $350.9\,\rm pc\,cm^{-3}$. According to Figure 2, DMs for
three frequencies have two main peaks, but the correct distribution for 600
MHz may have multiple peaks. The histogram of DMs for 600 MHz is shown in the
bottom panel of Figure 2. It also shows that the DMs have two main peaks, thus
we take the peak DM around $349.734\,\rm pc\,cm^{-3}$ at this frequency as the
right peak. We use the KDE approach for DMs to evaluate two DM peaks, which
are repeated 1000 times for each frequency. The mean value of peaking DMs is
approximated for each peak. Their standard deviation is set as an error. Table
2 finally shows the peaking DMs at four frequencies. The right peaking DM at
1370 MHz appears more than 250 times but exists, causing a significantly
higher inaccuracy in Table 2, whereas the other peaking DMs appear 1000 times.
Thus, the DM of the left peaks decreases with the frequency increases, but the
reverse is true for the right ones.

Figure 2:  (Upper panel) KDEs of these DMs at different frequencies. Each DM is sampled 1000 points by using a Gaussian random distribution with the error bar of each DM observation as the standard deviation. (Bottom panel) Histogram of DMs at 600 MHz. The red solid line is fitted by two-component Gaussians, and the Adjusted R-Square is 0.93, the peaks of the Gaussians are at $348.826\,\rm pc\,cm^{-3}$ and $349.734\,\rm pc\,cm^{-3}$ respectively. Table 2: Two peaking DMs at different frequencies, which are based on one thousand times of KDEs. The subscript ‘L’ denotes the DMs at the left peaks, ‘R’ denotes the right one. Center frequency | 160 MHz | 350 MHz | 600 MHz | 1370 MHz
---|---|---|---|---
$\rm DM_{\rm obs,L}$ ($\rm pc\,cm^{-3}$) | 348.988(1.9) | 348.854(10.2) | 348.820(6.9) | 348.755(12.6)
$\rm DM_{\rm obs,R}$ ($\rm pc\,cm^{-3}$) | 349.410(2.0) | 349.453(12.3) | 349.742(8.1) | 350.436(110.5)
$\rm\Delta DM_{\rm obs}$ ($\rm pc\,cm^{-3}$) | 0.422(2.8) | 0.599(16.0) | 0.992(10.6) | 1.774(111.2)

###  2.2 The properties of waiting time

The waiting time is calculated by two adjacent bursts within an observational
campaign, thus the bursts are taken into account for data from EVN (Marcote et
al., 2020), Apertif (Pastor-Marazuela et al., 2021), GBT (Chawla et al.,
2020), SRT (Pilia et al., 2020), uGMRT (Marthi et al., 2020; Pleunis et al.,
2021), LOFAR (Pastor-Marazuela et al., 2021; Pleunis et al., 2021), and CHIME.
Considering the lack of pulse width, fluence, and peak flux density in
Mckinven et al. 2023, we will use the data given by Amiri et al. 2020 to
maintain consistency in the following discussions.

Since the bursts of the repeater appear random at observable frequency
bandwidth and an occasional time in the phase window, the histogram of waiting
times in the logarithm coordinate is given in Figure 3, which shows a typical
bimodal distribution. The waiting times of most bursts are in the range of
$31.352\rm\,s$ to $11397.10608\rm\,s$. In the right part, 89 waiting times are
well fitted by the Gaussian function with a reduced Chi-Square of $\sim 2.737$
and an Adjusted R-Square of $\sim 0.918$, their peaking value is at $\Delta
t=1612.91266\rm\,s$. We used the K-S test to examine the right part of the
waiting time and found that the p-value (0.974) is greater than the 0.05
significance level. Thus the waiting times on the right part cannot reject a
Gaussian distribution. We are also aware that CHIME/FRB only has an exposure
of 12 min or $\sim$40 min each time (Amiri et al., 2020; Chawla et al., 2020),
and other telescopes usually have few-hour exposures each time, so it is
difficult to see pairs longer than the CHIME/FRB telescope observation length,
which may lead to the decrease in counts greater than $10^{3}$ s (17 min).
However, our result can be consistent with the burst rate ($\sim$1.8 bursts
hr-1) of the repeater (Amiri et al., 2020; Chawla et al., 2020). In the left
part, seven waiting times vary around tens of milliseconds, appearing in the
300$\sim$800 MHz and frequency-dependent. The mean waiting times at different
frequencies are $37.4\,\rm ms$ ($350\,\rm MHz$), $52.7\,\rm ms$ ($600\,\rm
MHz$), and $86.4\,\rm ms$ ($650\,\rm MHz$), respectively. We also use the
Gaussian function to fit the left part and obtain the peaking value at $\Delta
t=0.05622\,\rm s$. The Anderson-Darling (A-D) test gives out their statistic
value (0.421) and critical value (0.742). These results suggest that waiting
time follows a Log-normal distribution. We thought the decrease in the waiting
time less than 10 ms may be related to the width of the bursts and the
definition of individual bursts, which may be caused by the different physical
properties discussed in the following.

Figure 3: Histogram for the distribution of the waiting time in units of the
second. The bin size is 0.3, and the red solid and black dash lines are the
fitting curves of the right and left parts of Gaussian functions,
respectively.

The accumulated energy of an FRB burst for the next one depends on the
duration of two consecutive bursts. The possible correlation between the
waiting time and the burst intensity could appear in the preceding or
subsequent bursts, which is presented in Figure 4. The DMs of subsequent
bursts have a higher mean value than that in previous bursts. The mean fluence
and peak flux density for the subsequent bursts are provided with relatively
lower values, and the mean pulse width of previous bursts is narrow, they
suggest that the width may have opposite properties with the mean fluence and
peak flux density. It is easily found that all plots are scattered data
points, so these parameters have no obvious correlation with the waiting time.
We also take the observational data in Figure 4 to examine the correlations
between these parameters and the frequency. In Figure 5, the pulse width,
fluence, and peak flux density show anti-correlative with the frequency in the
Log-Log coordinate. The variations of mean values for the three parameters of
the previous and subsequent bursts are irrelevant to the frequency. Coupled
with the results of previous statistical analysis on the DMs and waiting time,
some external mechanisms and the effects in the propagation path are a more
probable contribution to the repeater.

Figure 4: Waiting time vs. pulse width (the left upper panel), fluence (the
right upper panel), peak flux density (the left bottom panels) and DM (the
right bottom panel) of FRB bursts. The black dots represent the data of the
previous bursts, the red dots are for these from the subsequent bursts, and
the dashed lines are the mean values. Figure 5: Observational center frequency
vs. median for pulse width (the upper panel), fluency (the middle panel) and
peak flux density (the bottom panel). The black, cyan, blue, green, red,
brown, orange and purple dots are the data from CHIME, EVN, Apertif, GBT, SRT,
uGMRT ($650\rm\,MHz$), uGMRT ($350\,\rm MHz$) and LOFAR, respectively. The
error bars are the minimum and maximum values of parameters, the red dashed
lines are the linear fitting in the log-log coordinate.

##  3 The lensing effects on FRB 180916.J0158+65

In this section, we will discuss multiple images caused by the lens.
Furthermore, the delay time difference of multiple images within the lens
model will be calculated through the statistical results of DMs.

For the axially symmetric gravitational lensing in the vacuum, the light rays
propagate along the null geodesic curve and are converged by the gravitational
field. Since the light rays from the two sides of the lens could have a
propagation path difference, an observer may detect two frequency-independent
images with different arrival times and magnifications, e.g., the review by
Bisnovatyi-Kogan & Tsupko 2017. The delay time difference between two images
should be smaller than $15\rm\,ms$ for primordial black holes (Muñoz et al.,
2016). The two images with different propagation paths could also undergo
different non-magnetized cold plasma. Thus the dispersive delay difference
depends on the contribution of their DM difference ($\Delta\rm DM(\nu)$) and
is simplified as

$\displaystyle\Delta t_{\rm dis}=4.15{\rm\,ms}\,\frac{\Delta\rm
DM(\nu)}{\nu^{2}_{\rm GHz}},$ (1)

where $\nu_{\rm GHz}=\nu/\rm GHz$ is the frequency of the image in the unit of
$\rm GHz$. From Table 2, $\Delta t_{\rm dis}$ at $600\,\rm MHz$, $350\,\rm
MHz$ and $160\,\rm MHz$ are $11.44\,\rm ms$, $20.29\,\rm ms$ and $68.41\,\rm
ms$, respectively. Coupled with the gravitational lensing effects, they are
still incompatible with the results of Figure 3. Some other mechanisms may
exist in the propagation path of the repeater.

The radio signals passing through the plasma lens can diverge as multiple
images with different propagation paths. The DMs of images are the frequency-
dependence and can present the multiple-components DM distributions in the
observation (our previous paper, i.e., Wang et al. 2022b). Moreover, the delay
time difference between two images depends on the variations of their DMs and
propagation paths.

Thus, the higher-order effects of DM variations in the theoretical prediction
should be introduced to the delay time differences, which are roughly
expressed as

$\displaystyle\Delta t_{\rm PL}=\chi+4.15{\rm\,ms}\,\frac{\Delta\rm
DM}{\nu^{2}_{\rm GHz}}-b\frac{\Delta\rm DM^{2}}{\nu^{4}_{\rm GHz}},$ (2)

where $\chi$ is the geometric delay time difference contributed by the
repeater itself (Wang et al., 2019) and some plasma clouds with relation
$\overset{n}{\underset{i=1}{\sum}}{\rm\Delta DM}_{i}\propto\nu^{2}$ (Tuntsov
et al., 2021), the second term in the right side of equation is the classical
dispersion relation, the third term is the delay time difference due to the
geometric effects, and $b$ is a free parameter. From the results at 350 MHz
and 600 MHz in Section 2, we can get $\chi=52.51\,\rm ms$ and $b=1.48\,\rm ms$
through Equation (2). Consequently, the delay time differences at the 160 MHz
and 1370 MHz are $-281.38\rm\,ms$ and $55.11\rm\,ms$, respectively.

Table 3: Two peaking DMs, DM differences ($\Delta\rm DM_{\rm G}$) and the delay time differences ($\Delta t_{\rm G}$) at different center frequencies calculated from Equation (2). The subscript ‘G’ is for the results from the two-component Gaussian function. Center frequency | 160 MHz | 350 MHz | 600 MHz | 650 MHz | 1370 MHz
---|---|---|---|---|---
$\rm DM_{\rm G,L}$ ($\rm pc\,cm^{-3}$) | 348.919 | 348.847 | 348.805 | 348.799 | 348.742
$\rm DM_{\rm G,R}$ ($\rm pc\,cm^{-3}$) | 349.252 | 349.505 | 349.724 | 349.762 | 350.406
$\rm\Delta DM_{\rm G}$ ($\rm pc\,cm^{-3}$) | 0.333 | 0.658 | 0.919 | 0.963 | 1.664
$\Delta t_{\rm G}$ ($\rm ms$) | -144.01 | 32.08 | 53.45 | 54.27 | 55.02

Due to the physical properties of the plasma lens, the frequencies of
corresponding DMs should be taken as positive values. These bursts produced by
a single source cannot occupy two peaking DMs while $\nu\rightarrow\infty$. We
found the two-component Gaussian function is the best fit for the peaking DMs
in the Table 2 (an Adjusted R-square of $\sim$ 0.967). For the DM
distribution, the two peaking DMs at five frequencies, the DM differences
($\Delta\rm DM_{\rm G}$) for each frequency and the delay time differences are
given in Table 3. The delay times differences are decreasing with the
decreasing frequency except for the value at 160 MHz. We found that $\rm
DM_{\rm G,R}$ at 160 MHz is relatively lower than $\rm DM_{\rm obs,R}$. The
DMs listed in Table 1, except for the burst at MJD 58950.54130169, are around
$348.98\rm\,pc\,cm^{-3}$ at 160 MHz, and their scattering timescales are
around $46\,\rm ms$ at 150 MHz (Pastor-Marazuela et al., 2021). All these
results suggested that the scattering effect may contribute $\sim 0.249\,\rm
pc\,cm^{-3}$ to the DM of burst at MJD 58950.54130169. Furthermore, it can
cause a more significant delay time difference than a single burst of $\rm
Width\gtrsim 40\,\rm ms$. Therefore, the delay time differences and the
frequency-dependent DMs are consistent with the plasma lensing model (Wang et
al., 2022b). FRB 180916.J0158+65 may be affected by the plasma lens with the
two-component Gaussian model describing DM distributions.

##  4 Summary and Discussions

We investigate the repeating features of FRB 180916.J0158+65 in terms of
statistics, especially waiting times and DMs. The DMs have a precise bimodal
distribution at different frequencies. The peaking values of DMs increase with
frequency drop for the left distribution but decrease with frequency decrease
for the right distribution, demonstrating that repeating bursts at different
frequencies could come from different propagation paths (Wang et al., 2022b).
This viewpoint is supported by the degree of linear polarization with
frequency-dependent characteristics in the repeated FRBs (Feng et al., 2022).
We fit these DM peaks and find that the two-component Gaussian function is
more appropriate for the DM peaks at different frequencies. The waiting time
is also a discontinuous bimodal distribution peaking at 56.22 ms and
1612.91266 s, the mean waiting time in the left distribution decreases as the
frequency decrease. The parameters of the preceding or subsequent bursts,
including pulse width, fluence, peak flux density and DM, do not correlate
with the waiting times. Considering the preceding and subsequent burst
parameters, external effects may contribute to the repeater.

Based on the statistics for observed bursts of FRB 180916.J0158+65, the
repeater suffered from lensing effects in the propagation path has been
examined. The delay time difference between two images produced by the
gravitational lensing is inconsistent with the waiting time. The higher-order
term from the plasma lensing effects can be consistent with the left peak
distribution of the waiting times and the variations of DMs, except at 160
MHz. As a relatively clean line-of-sight path at 160 MHz (Pleunis et al.,
2021), the scattering effect may highly contribute to the burst at the right
distribution of DMs. The frequency-dependent DM may contribute to the
properties of near-source plasma and the intervening galaxy, such as the
supernova remnants around the source, the pulsar wind nebulae produced by the
source, HII regions, and galactic halo (Yang & Zhang, 2017; Prochaska et al.,
2019; Er & Mao, 2022). The multi-frequency observations of some repeaters are
an important way to reveal their propagating effects and the plasma lensing
model.

FRB 180916.J0158+65 may originate from the binary system. For the NS$-$white
dwarf (WD) scenario (Gu et al., 2016, 2020), the waiting times vary from
$100\,$ to $\rm 1.59392\times 10^{4}\,\rm s$ when the Eddington limit are
considered ($\dot{M}\lesssim 10^{18}\,\rm g~{}s^{-1}$). But the energy of a
burst from the gravitational potential energy of the accreted material is
incompatible with the observation (Frank et al., 2002). In the NS$-$asteroid
belt collision scenario (Geng & Huang, 2015; Dai et al., 2016; Dai & Zhong,
2020), the typical energy of a burst ($10^{36}-10^{38}\,\rm erg$) depends on
the gravitational energy of the asteroid (Dai et al., 2016; Smallwood et al.,
2019), but the long time to the next collision ($\sim 0.8\,\rm h$) (Dai et
al., 2016) is also inconsistent with the observations of the repeater.

For the “cosmic comb” model (Zhang, 2017), the stellar wind in the NS$-$NS
binary scenario or NS$-$black hole binary scenario cannot be strong enough to
explain the properties of the repeater FRB 180916.J0158+65 (Du et al., 2021).
According to the maximal energy ($\sim 6\times 10^{38}\,\rm erg$) of the
repeater, the dipole magnetic field of NS at $1\,\rm GHz$ requires $B_{\rm
d}\gtrsim 4.18\times 10^{13}\rm\,G$ (Kumar et al., 2017; Metzger et al.,
2019). For an high mass X-ray binary (HMXB) system, the model requires the
pulsar with the spin period of $P\gtrsim 1.09\,\rm s\,B_{\rm
d,12}^{1/3}$(Walter et al., 2015). That means the highly magnetized NS in HMXB
may be of origin of FRB 180916.J0158+65. In addition, the multiple bursts in
this model can be produced in the pulsar’s magnetosphere (Wang et al., 2019;
Levkov et al., 2022). The interval time between two bursts is $\lesssim
18.70\,{\rm ms}$ when taken $P=23.5~{}\rm{s}$ and the radial radius within the
light cylinder radius, which suggests that the zero-order term of FRB
180916.J0158+65 given by Equation (2) may be mainly contributed by other
effects in the propagation path (Tuntsov et al., 2021). Till now, 3 in 24
repeaters, including FRB 180916.J0158+65, FRB 121102 and FRB 20201124A, have
some similar statistical properties (Li et al., 2021b; Xu et al., 2022), FRB
20201124A may reside in a magnetar/Be star binary (Wang et al., 2022a). This
type of repeater may be related to a highly magnetized NS in the HMXB.

## Acknowledgements

This work is supported in part by the Opening Foundation of Xinjiang Key
Laboratory (No. 2021D04016), the National Natural Science Foundation of China
(Nos. 12033001, 12288102, 11833003, 12273028, 12041304), the Natural Science
Foundation of Xinjiang Uygur Autonomous Region (No. 2022D01A363), the Major
Science and Technology Program of Xinjiang Uygur Autonomous Region (No.
2022A03013-1, 2022A03013-3), the Chinese Academy of Sciences (CAS) “Light of
West China” Program (No. 2018-XBQNXZ-B-025), and the special research
assistance project of the CAS. X. Z. would like to thank Qing-min Li for her
fruitful suggestions.

## Data availability

Observational data used in this paper are quoted from the cited works.
Additional data generated from computations can be made available upon
reasonable request.

## References

  * Abbate et al. (2020) Abbate F., et al., 2020, MNRAS, 498, 875
  * Amiri et al. (2020) Amiri M., et al., 2020, Nature, 582, 351
  * Amiri et al. (2021) Amiri M., et al., 2021, ApJS, 257, 59
  * Andersen et al. (2019) Andersen B. C., et al., 2019, ApJ, 885, L24
  * Andersen et al. (2020) Andersen B. C., et al., 2020, Nature, 587, 54
  * Bethapudi et al. (2022) Bethapudi S., Spitler L. G., Main R. A., Li D. Z., Wharton R. S., 2022, arXiv e-prints, p. arXiv:2207.13669
  * Bisnovatyi-Kogan & Tsupko (2017) Bisnovatyi-Kogan G., Tsupko O., 2017, Universe, 3, 57
  * Chamma et al. (2021) Chamma M. A., Rajabi F., Wyenberg C. M., Mathews A., Houde M., 2021, MNRAS, 507, 246
  * Chawla et al. (2020) Chawla P., et al., 2020, ApJ, 896, L41
  * Cheng et al. (2020) Cheng Y., Zhang G. Q., Wang F. Y., 2020, MNRAS, 491, 1498
  * Cordes & Lazio (2002) Cordes J. M., Lazio T. J. W., 2002, arXiv e-prints, pp astro–ph/0207156
  * Dai & Zhong (2020) Dai Z. G., Zhong S. Q., 2020, ApJ, 895, L1
  * Dai et al. (2016) Dai Z. G., Wang J. S., Wu X. F., Huang Y. F., 2016, ApJ, 829, 27
  * Du et al. (2021) Du S., Wang W., Wu X., Xu R., 2021, MNRAS, 500, 4678
  * Er & Mao (2022) Er X., Mao S., 2022, MNRAS, 516, 2218
  * Feng et al. (2022) Feng Y., et al., 2022, Science, 375, 1266
  * Frank et al. (2002) Frank J., King A., Raine D. J., 2002, Accretion Power in Astrophysics: Third Edition
  * Geng & Huang (2015) Geng J. J., Huang Y. F., 2015, ApJ, 809, 24
  * Geng et al. (2021) Geng J., Li B., Huang Y., 2021, The Innovation, 2, 100152
  * Geyer et al. (2021) Geyer M., et al., 2021, MNRAS, 505, 4468
  * Gu et al. (2016) Gu W.-M., Dong Y.-Z., Liu T., Ma R., Wang J., 2016, ApJ, 823, L28
  * Gu et al. (2020) Gu W.-M., Yi T., Liu T., 2020, MNRAS, 497, 1543
  * Kuiack et al. (2020) Kuiack M., Wijers R. A. M. J., Rowlinson A., Shulevski A., Huizinga F., Molenaar G., Prasad P., 2020, MNRAS, 497, 846
  * Kumar et al. (2017) Kumar P., Lu W., Bhattacharya M., 2017, MNRAS, 468, 2726
  * Kumar et al. (2022) Kumar P., Shannon R. M., Lower M. E., Bhandari S., Deller A. T., Flynn C., Keane E. F., 2022, MNRAS, 512, 3400
  * Kurban et al. (2022) Kurban A., et al., 2022, ApJ, 928, 94
  * Levkov et al. (2022) Levkov D. G., Panin A. G., Tkachev I. I., 2022, ApJ, 925, 109
  * Li et al. (2019) Li B., Li L.-B., Zhang Z.-B., Geng J.-J., Song L.-M., Huang Y.-F., Yang Y.-P., 2019, arXiv e-prints, p. arXiv:1901.03484
  * Li et al. (2021a) Li C. K., et al., 2021a, Nature Astronomy, 5, 378
  * Li et al. (2021b) Li D., et al., 2021b, Nature, 598, 267
  * Li et al. (2022) Li L., Li Q.-C., Zhong S.-Q., Xia J., Xie L., Wang F.-Y., Dai Z.-G., 2022, ApJ, 929, 139
  * Luo et al. (2020) Luo R., et al., 2020, Nature, 586, 693
  * Mao et al. (2022) Mao J.-W., et al., 2022, Research in Astronomy and Astrophysics, 22, 065006
  * Marcote et al. (2020) Marcote B., et al., 2020, Nature, 577, 190
  * Marthi et al. (2020) Marthi V. R., Gautam T., Li D. Z., Lin H. H., Main R. A., Naidu A., Pen U. L., Wharton R. S., 2020, MNRAS, 499, L16
  * Mckinven et al. (2023) Mckinven R., et al., 2023, ApJ, 950, 12
  * Metzger et al. (2019) Metzger B. D., Margalit B., Sironi L., 2019, MNRAS, 485, 4091
  * Muñoz et al. (2016) Muñoz J. B., Kovetz E. D., Dai L., Kamionkowski M., 2016, Phys. Rev. Lett., 117, 091301
  * Pastor-Marazuela et al. (2021) Pastor-Marazuela I., et al., 2021, Nature, 596, 505
  * Pearlman et al. (2020) Pearlman A. B., Majid W. A., Prince T. A., Nimmo K., Hessels J. W. T., Naudet C. J., Kocz J., 2020, ApJ, 905, L27
  * Petroff et al. (2016) Petroff E., et al., 2016, Publ. Astron. Soc. Australia, 33, e045
  * Pilia et al. (2020) Pilia M., et al., 2020, ApJ, 896, L40
  * Platts et al. (2019) Platts E., Weltman A., Walters A., Tendulkar S. P., Gordin J. E. B., Kandhai S., 2019, Phys. Rep., 821, 1
  * Platts et al. (2021) Platts E., et al., 2021, MNRAS, 505, 3041
  * Pleunis et al. (2021) Pleunis Z., et al., 2021, ApJ, 911, L3
  * Prochaska et al. (2019) Prochaska J. X., et al., 2019, Science, 366, 231
  * Rajwade et al. (2020) Rajwade K. M., et al., 2020, MNRAS, 495, 3551
  * Ravi et al. (2022) Ravi V., et al., 2022, MNRAS, 513, 982
  * Smallwood et al. (2019) Smallwood J. L., Martin R. G., Zhang B., 2019, MNRAS, 485, 1367
  * Tendulkar et al. (2017) Tendulkar S. P., et al., 2017, ApJ, 834, L7
  * Tendulkar et al. (2021) Tendulkar S. P., et al., 2021, ApJ, 908, L12
  * Thornton et al. (2013) Thornton D., et al., 2013, Science, 341, 53
  * Tuntsov et al. (2021) Tuntsov A., Pen U.-L., Walker M., 2021, arXiv e-prints, p. arXiv:2107.13549
  * Walter et al. (2015) Walter R., Lutovinov A. A., Bozzo E., Tsygankov S. S., 2015, A&ARv, 23, 2
  * Wang et al. (2019) Wang W., Zhang B., Chen X., Xu R., 2019, ApJ, 876, L15
  * Wang et al. (2022a) Wang F. Y., Zhang G. Q., Dai Z. G., Cheng K. S., 2022a, Nature Communications, 13, 4382
  * Wang et al. (2022b) Wang Y.-B., Wen Z.-G., Yuen R., Wang N., Yuan J.-P., Zhou X., 2022b, Research in Astronomy and Astrophysics, 22, 065017
  * Xu et al. (2022) Xu H., et al., 2022, Nature, 609, 685
  * Yang & Zhang (2017) Yang Y.-P., Zhang B., 2017, ApJ, 847, 22
  * Yao et al. (2017) Yao J. M., Manchester R. N., Wang N., 2017, ApJ, 835, 29
  * Younes et al. (2020) Younes G., et al., 2020, ApJ, 904, L21
  * Yusifov & Küçük (2004) Yusifov I., Küçük I., 2004, A&A, 422, 545
  * Zhang (2017) Zhang B., 2017, ApJ, 836, L32


# Improving Diffusion-based Image Translation using Asymmetric Gradient
Guidance

Gihyun Kwon
Department of Bio and Brain Engineering
KAIST
<EMAIL_ADDRESS>
&Jong Chul Ye
Kim Jaechul Graduate School of AI
KAIST
<EMAIL_ADDRESS>

###### Abstract

Diffusion models have shown significant progress in image translation tasks
recently. However, due to their stochastic nature, there’s often a trade-off
between style transformation and content preservation. Current strategies aim
to disentangle style and content, preserving the source image’s structure
while successfully transitioning from a source to a target domain under text
or one-shot image conditions. Yet, these methods often require computationally
intense fine-tuning of diffusion models or additional neural networks. To
address these challenges, here we present an approach that guides the reverse
process of diffusion sampling by applying asymmetric gradient guidance. This
results in quicker and more stable image manipulation for both text-guided and
image-guided image translation. Our model’s adaptability allows it to be
implemented with both image- and latent-diffusion models. Experiments show
that our method outperforms various state-of-the-art models in image
translation tasks.

Figure 1: Image translation examples by our proposed method. Our model can
generate high-quality translation and editing outputs using both text and
image conditions.

##  1 Introduction

Image translation is a task that aims to convert an image from a source domain
to a target domain while preserving the structural attributes of the source
image. In the past, image translation methods based on Generative Adversarial
Networks(GAN) [1] were mainly used, ranging from unimodal image-to-image
translation (I2I) [2, 3, 4, 5] to multimodal I2I models [6, 7]. While these
methods showed good performance, they had the disadvantage of being unable to
perform image-to-image translation for arbitrary domains and conditions.

Recently, various methods have been proposed to solve this problem.
Particularly notable are the methods that manipulate the generation process of
a pre-trained generative model by applying modifications to generate images
with desired conditions. This approach has the advantage of using the superior
generation performance of the pre-trained model while being able to generate
the desired images. Specifically, previous methods mainly used pre-trained GAN
models to enable image manipulation [8, 9]. Recently, the use of large
foundation models, especially text-to-image embedding models such as CLIP
[10], has been introduced for image editing based on text conditions [11, 12].
However, these methods show limited performance due to the limitations of GAN
models, such as limited domains and the cumbersome process of embedding real
images into the latent space.

With the development of diffusion models and the combination of text encoder
models, text-to-image diffusion models [13, 14] have been developed, and
research on image translation and image editing using diffusion models has
become more active [15, 16]. In particular, various methods have been proposed
based on cross-attention with text conditions [17, 18], allowing for image-to-
image translation. However, the text-to-image model requires an inversion
process that accurately embeds the source image to latent space, which
consumes additional time for optimization. Moreover, they have difficulty
applying to image translation cases where the source image is provided as
condition.

To address this, a recent approach of DiffuseIT [19] uses gradient guidance
during the generation process to preserve the structural attributes of the
source image while disentangling and changing the target style. While these
models have shown good performance, they still requires extensive use of
slicing ViT network [20] and its computationally expensive gradient
computation.

To address the limitations of existing methods, here we present a novel
sampling approach using so-called Asymmetric Gradient Guidance (AGG), which
amalgamates the strengths of various models. Specifically, the gradient update
by AGG is composed of two step: a single MCG framework [21] to compute the
initial update, which is followed by computationally efficient Decomposed
Diffusion Sampling (DDS) update from the denoised signal by Tweedie’ fomula,
which can be efficiently computed by Adam optimizer. Furthermore, in contrast
to DiffuseIT that relies on slicing ViT based structural preservation loss, we
use much simpler structural regularization term using the intermediate product
stored during the forward DDIM step. Combined with simpler regularization loss
and efficient gradient update, our method can provide much faster and
effective image translation under text- and one-shot image conditions.
Moreover, this integration provides a significant advantage as it enables our
sampling approach to operate in both traditional image diffusion models and
latent diffusion models [14].

Our contribution can be summarized as follows:

  * •

We propose Asymmetric Gradient Guidance method, which is composed of two step:
a single MCG framework [21] followed by computationally efficient Adam
optimization from the denoised signal by Tweedie’ fomula.

  * •

We introduce a simpler structural loss function which utilizes the
intermediate products saved during the DDIM forward step, allowing for faster
and efficient gradient guidance in both image diffusion models and latent
diffusion models.

  * •

Our method is so efficient and flexible that it can be integrated with
arbitrary loss functions, making it suitable for a wide range of applications
such as text-guided image translation, image-guided appearance transfer,
artistic style transfer, image editing, etc.

##  2 Related Works

Recently, the combination of pre-trained GAN models and the CLIP model has
been proposed for various applications such as naive generation [22], style
transfer with fine-tuning the models [12], and text-guided editing [11].
However, generating images through GAN has limitations such as restricted
domains and poor performance. With the development of diffusion-based models,
recent attempts combined the CLIP and the diffusion model for various tasks
such as generating text-conditioned images [15, 16] or editing images using
mask regions [23]. Some methods involve tuning the entire or part of the model
to transform images [24, 25]. Methods that can transform images based on text
conditions without training but only through the sampling process [19, 26]
have also demonstrated impressive performance.

Independent from text-conditioned image manipulation, several researchs have
been conducted on image manipulation with target image conditions. Initially,
image style transfer models using feature tensor modulation [27, 28, 29] were
developed. Subsequently methods adapting GAN models [30, 31, 32] showed good
performance. Recent models can transfer the appearance of one-shot images
based on ViT feature [20], and subsequent studies inspired by this have
successfully combined diffusion models to transfer appearance [19].

##  3 Background

Denoising Diffusion Probabilistic Models (DDPM) DDPM[33] is trained to
estimate the added noise in the noisy image ${\boldsymbol{x}}_{t}$ produced by
Markovian forward process :

$q({\boldsymbol{x}}_{T}|{\boldsymbol{x}}_{0}):=\prod_{t=1}^{T}q({\boldsymbol{x}}_{t}|{\boldsymbol{x}}_{t-1}),\quad\mbox{where}\quad
q({\boldsymbol{x}}_{t}|{\boldsymbol{x}}_{t-1}):={\mathcal{N}}({\boldsymbol{x}}_{t};\sqrt{1-\beta_{t}}{\boldsymbol{x}}_{t-1},\beta_{t}{\boldsymbol{I}}),$
(1)

Here, the noise schedule $\beta_{t}$ is an increasing sequence of $t$, with
$\bar{\alpha}_{t}:=\prod_{i=1}^{t}\alpha_{t},\,\alpha_{t}:=1-\beta_{t}$. For
the reverse process, with the learned noise prediction network
$\boldsymbol{\epsilon}_{\theta}$, we can predict the noisy sample of previous
timestep ${\boldsymbol{x}}_{t-1}$:

$\displaystyle{\boldsymbol{x}}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\Big{(}{\boldsymbol{x}}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\boldsymbol{\epsilon}_{\theta}({\boldsymbol{x}}_{t},t)\Big{)}+\sigma_{t}\boldsymbol{z},$
(2)

where ${\boldsymbol{z}}\sim{\mathcal{N}}(0,{\boldsymbol{I}})$ and $\sigma^{2}$
is variance which is set as $\beta$ in DDPM. With iterative process, we can
sample the image ${\boldsymbol{x}}_{0}$ from initial sample
${\boldsymbol{x}}_{T}\sim{\mathcal{N}}(0,{\boldsymbol{I}})$.

Denoising Diffusion Implicit Models (DDIM) Since the diffusion models requires
large number of steps, the full sampling process is slow. To address the
problem, DDIM [34] proposes another method by re-defining the DDPM formulation
as non-Markovian process. Specifically, the reverse step process is described
as :

$\displaystyle{\boldsymbol{x}}_{t-1}=\underbrace{\sqrt{\bar{\alpha}_{t-1}}\hat{\boldsymbol{x}}_{0,t}({\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},t))}_{\text{Denoise}}+\underbrace{D_{t}({\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},t))}_{\text{Noise}}+\sigma_{t}{\boldsymbol{z}}_{t},\quad$
(3)

where

$\displaystyle\hat{\boldsymbol{x}}_{0,t}({\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},t)):=\frac{{\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},t)}{\sqrt{\bar{\alpha}_{t}}},$
$\displaystyle\quad
D_{t}({\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},t)):=\sqrt{1-\bar{\alpha}_{t-1}-\sigma_{t}^{2}}\epsilon_{\theta}({\boldsymbol{x}}_{t},t)$
(4)

If $\sigma_{t}=0$, the reverse process is fully deterministic, leading to a
faster sampling.

##  4 Main Contribution

The recent focus of the conditional diffusion researches is how to incorporate
the conditioning gradient during the reverse sampling using Eq. (2) or Eq.
(3). This is because for a given loss function $\ell({\boldsymbol{x}})$, a
direct injection of the gradient of the loss computed at
${\boldsymbol{x}}_{t}$ produces inaccurate gradient guidance. To address this,
various methods have been developed. In this following, we first provide a
brief review of them and present a novel Asymmetric Gradient guidance (AGG) by
taking the best of the existing methods.

###  4.1 Gradient guidance: a review

MCG. The manifold constraint gradient (MCG) [21] begins by projecting the
noisy image ${\boldsymbol{x}}_{t}$ onto the estimated clean image
$\hat{\boldsymbol{x}}_{0,t}$ using the Tweedie’s formula in Eq. (4).
Specifically, the sampling involves guiding the noisy image
${\boldsymbol{x}}_{t-1}$ as follows:

$\displaystyle{\boldsymbol{x}}^{\prime}_{t-1}\leftarrow{\boldsymbol{x}}_{t-1}-\nabla_{{\boldsymbol{x}}_{t}}\ell(\hat{\boldsymbol{x}}_{0,t}({\boldsymbol{x}}_{t})),$
(5)

where ${\boldsymbol{x}}_{t}$ is the initially obtained next-step sample as per
Eq. (3), and
$\nabla{{\boldsymbol{x}}_{t}}\ell(\hat{\boldsymbol{x}}_{0,t}({\boldsymbol{x}}_{t}))$
is the gradient of the loss $\ell$, calculated with the denoised image
$\hat{\boldsymbol{x}}_{0,t}({\boldsymbol{x}}_{t})$ with respect to
${\boldsymbol{x}}_{t}$. It was shown in [21] that the MGC gradient enforces
the update to stay on the original noisy manifold (see Fig. 2 (a)).

Asyrp. Asymmetric Reverse Sampling (Asyrp) [25] converts the reverse sampling
step in Eq. (3) to the following:

$\displaystyle{\boldsymbol{x}}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\hat{\boldsymbol{x}}_{0,t}(\tilde{\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},t))+D_{t}({\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},t))+\sigma_{t}{\boldsymbol{z}}_{t},$
(6)

where

$\displaystyle\tilde{{\boldsymbol{\epsilon}}}_{\theta}({\boldsymbol{x}}_{t},t)={{\boldsymbol{\epsilon}}}_{\theta}({\boldsymbol{x}}_{t},t)+\Delta{\boldsymbol{\epsilon}}_{t}$
(7)

To compute the update of the diffusion model
$\Delta{\boldsymbol{\epsilon}}_{t}$, they introduced a concept of $h$-space,
which is to manipulate the bottleneck feature of score network
${\boldsymbol{\epsilon}}_{\theta}$. Therefore, they set
$\hat{\boldsymbol{x}}_{0,t}(\tilde{{\boldsymbol{\epsilon}}}_{\theta})$ as
$\hat{\boldsymbol{x}}_{0,t}({{\boldsymbol{\epsilon}}}_{\theta}({\boldsymbol{x}}_{t}|\Delta
h_{t},t))$ in Eq. (6), which is an estimated clean image from score network
with the manipulated $h$-space (see Fig. 2(b)).

DDS. In Decomposed Diffusion Sampling (DDS) [35], the following optimization
problem is solved after projecting the sample on the clean manifold through
Tweedie’s formula:

$\displaystyle\Delta{\boldsymbol{x}}_{0}=\arg\min_{\Delta}\ell(\hat{\boldsymbol{x}}_{0,t}+\Delta),$
(8)

which results in the following modified reverse sampling step:

$\displaystyle{\boldsymbol{x}}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\left(\hat{\boldsymbol{x}}_{0,t}({\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},t))+\Delta{\boldsymbol{x}}_{0}\right)+D_{t}({\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},t))+\sigma_{t}{\boldsymbol{z}}_{t},$
(9)

It was shown [35] that under some regularity condition the conjugate gradient
(CG)-based update $\Delta{\boldsymbol{x}}_{0}$ still ensures the updated
sample $\hat{\boldsymbol{x}}_{0,t}+\Delta{\boldsymbol{x}}_{0}$ to stay on the
clean manifold, so that subsequent noising process transfers the sample to a
correct noisy manifold (see Fig. 2 (c)).

Figure 2: Various gradient guidance scheme. (a) MCG utilizes manifold gradient
that ensures the update sample to stay on the correct noisy manifold. (b)
Asyrp updates the diffusion model for denoising formula with h-space
manipulation. (c) DDS use optimization based update to guide the denoised
image. (d) Our proposed method is composed of one-step MCG and Tweedie’s
formula followed by additional DDS optimization.

###  4.2 Asymmetric Gradient Guidance

Key observation. Although aforementioned approaches appear seemingly
different, there are fundamental connection between them. Specifically, by
translating $\Delta{\boldsymbol{x}}$ of Eq. (8) back to the noisy data space
of ${\boldsymbol{x}}_{t}$, the authors in [36] observed that the original
forward model
${\boldsymbol{x}}_{t}=\sqrt{\bar{\alpha}_{t}}{\boldsymbol{x}}_{0}+\sqrt{1-\bar{\alpha}_{t}}{\boldsymbol{\epsilon}}_{t}$
can be converted to

$\displaystyle{\boldsymbol{x}}_{t}=\sqrt{\bar{\alpha}_{t}}(\hat{\boldsymbol{x}}_{0,t}+\Delta{\boldsymbol{x}}_{0})+\sqrt{1-\bar{\alpha}_{t}}\tilde{\boldsymbol{\epsilon}}_{t},\quad$
(10)

where the perturbed diffusion model is given by

$\displaystyle\tilde{\boldsymbol{\epsilon}}_{t}:={\boldsymbol{\epsilon}}_{t}-\sqrt{\bar{\alpha}_{t}/(1-\bar{\alpha}_{t})}\Delta{\boldsymbol{x}}_{0}$
(11)

Therefore, the diffusion model update by Asyrp and the sample domain update by
DDS are equivalent up to a scaling factor. Furthermore, using the method of
the variation, $\Delta{\boldsymbol{x}}_{0}$ in Eq. (8) can be represented by
$\Delta{\boldsymbol{x}}_{0}=-\eta\nabla_{{\boldsymbol{x}}_{t}}\ell(\hat{\boldsymbol{x}}_{0,t}({\boldsymbol{x}}_{t}))$,
leading to

$\displaystyle\tilde{\boldsymbol{\epsilon}}_{t}:={\boldsymbol{\epsilon}}_{t}+s_{t}\nabla_{{\boldsymbol{x}}_{t}}\ell(\hat{\boldsymbol{x}}_{0,t}({\boldsymbol{x}}_{t}))$
(12)

for some $s_{t}$, suggesting the condition based guidance of the diffusion
model. The previous work [36] sets $s_{t}=\sqrt{1-\bar{\alpha}_{t}}$.

Proposed method. Given the equivalence between the sample update and diffusion
model update, we can derive a general update formula for the conditional
diffusion:

$\displaystyle{\boldsymbol{x}}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\hat{{\boldsymbol{x}}}^{\prime}_{t}+D_{t}({\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{x}}_{t},t))+\sigma_{t}{\boldsymbol{z}}_{t},$
(13)

where the following two forms are interchangely used for
$\hat{\boldsymbol{x}}_{t}^{\prime}$:

$\displaystyle\hat{{\boldsymbol{x}}}^{\prime}_{t}=\begin{cases}\hat{\boldsymbol{x}}_{0,t}({{\boldsymbol{\epsilon}}}_{t})-\nabla_{{\boldsymbol{x}}_{t}}\ell(\hat{\boldsymbol{x}}_{0,t}({\boldsymbol{x}}_{t}))\\\
\hat{\boldsymbol{x}}_{0,t}(\tilde{{\boldsymbol{\epsilon}}}_{t}),\quad\mbox{where}\quad\tilde{{\boldsymbol{\epsilon}}}_{t}:={{\boldsymbol{\epsilon}}}_{t}-s_{t}\nabla_{{\boldsymbol{x}}_{t}}\ell(\hat{\boldsymbol{x}}_{0,t}({\boldsymbol{\epsilon}}_{t}))\end{cases}$
(14)

Therefore, the remaining question is the computation of the MCG gradient
$\nabla_{{\boldsymbol{x}}_{t}}\ell(\hat{\boldsymbol{x}}_{0,t}({\boldsymbol{x}}_{t}))$,
which requires a backpropagation. The key idea of the DDS is that for general
linear inverse problems, when the underlying solution space lies in the Kyrov
subspace, a few step of standard CG update starting from a denoised sample
still ensures the update samples to live in the clean manifold. Therefore, the
computationally expensive backpropagation for MGC can be bypassed [35].

Unfortunately, we found that DDS approaches does not provide a stable solution
when the loss function is originated from text-driven and image-driven image
translation of our interest, as the associated inverse problem cannot be
represented as a linear inverse problem. However, applying the MCG at each
iteration is computationally expensive. To address this problem, we propose
the following hybrid update of $\hat{{\boldsymbol{x}}}_{t}^{\prime}$ for Eq.
(13):

$\displaystyle\hat{{\boldsymbol{x}}}_{t}^{\prime}=\bar{\boldsymbol{x}}_{t}^{\prime}+\arg\min_{\Delta}\ell(\bar{\boldsymbol{x}}_{t}^{\prime}+\Delta),$
$\displaystyle\quad\mbox{where}\quad\bar{\boldsymbol{x}}_{t}^{\prime}:=\hat{\boldsymbol{x}}_{0,t}(\tilde{{\boldsymbol{\epsilon}}}_{t})$

where the loss minimization in the first term is done without MGC gradient
using Adam optimizer, and the perturbed model output
$\tilde{\boldsymbol{\epsilon}}_{t}$ is computed by Eq. (14). This implies that
for image translation tasks one step MGC gradient is necessary for the
successive DDS type optimization to ensures the updated sample to live on the
correct manifold. We call this as Asymmetric Gradient Guidance (AGG).

AGG can be applied not only to image diffusion models but also to latent
diffusion models (LDM) [14]. In this case, instead of using
${\boldsymbol{x}}_{t}$ in the image diffusion, we utilize an embedded latent
${\boldsymbol{z}}_{t}$ after the encoder.

Figure 3: Detailed sampling strategy of our proposed method. We start our
sampling process with encoding the real image ${\boldsymbol{x}}_{src}$ with
forward DDIM to ${\boldsymbol{x}}_{T}$. During forward step, we save the
intermediate estimated clean images
$\hat{{\boldsymbol{x}}}^{*}_{0,t}=\hat{{\boldsymbol{x}}}^{*}_{0,t}({\boldsymbol{\epsilon}}_{t})$
to use them as target for regularization. With initial ${\boldsymbol{x}}_{T}$,
we guide the reverse sampling through loss function calculated with
$\hat{\boldsymbol{x}}_{0,t}=\hat{\boldsymbol{x}}_{0,t}({\boldsymbol{\epsilon}}_{t})$.
We manipulate $\hat{{\boldsymbol{x}}}_{t}$ with the calculated gradient and
additional optimization step.

###  4.3 Loss function

For image translation, our loss function is composed of two parts: style loss
$\ell_{sty}$ for image domain change, and the regularization loss $\ell_{reg}$
for preserving the source image structure.

$\displaystyle\ell_{total}=\lambda_{sty}\ell_{sty}+\lambda_{reg}\ell_{reg},$
(15)

where we can control the content preservation strength with controlling
parameter $\lambda_{reg}$.

As for $\ell_{sty}$, we leveraged the style loss functions used in DiffuseIT
[19]. Specifically, for text-guided I2I, we use embedding similarity loss
using pre-trained CLIP model, whereas for image-guided I2I, we use [CLS] token
matching loss using DINO ViT[37].

For structural preservation, DiffuseIT utilize a powerful content preservation
loss using ViT. However, we propose a simpler yet more efficient
regularization approach. Specifically, we propose a regularization scheme of
the reverse step using the pre-calculated intermediate samples from DDIM
forward process. As shown in Fig. 3, in the forward sampling of DDIM, we can
deterministically calculate the intermediate samples by modifying the reverse
sampling in Eq. (3). With changing the timestep $t$ and setting $\sigma_{t}=0$
such as:

$\displaystyle{{\boldsymbol{x}}}^{*}_{t+1}={\sqrt{\bar{\alpha}_{t+1}}\hat{{\boldsymbol{x}}}^{*}_{0,t}({\boldsymbol{\epsilon}}_{t})}+{\sqrt{1-\bar{\alpha}_{t+1}}{\boldsymbol{\epsilon}}_{t}},$
(16)

where ${{\boldsymbol{x}}}^{*}_{t}$ is calculated noisy image from source image
${{\boldsymbol{x}}}_{src}$, and
$\hat{{\boldsymbol{x}}}^{*}_{0,t}({\boldsymbol{\epsilon}}_{t})$ is estimated
denoised image calculated using Tweedie’s formula in Eq. (4). Furthermore,
rather than saving for all $t$, we save
$\hat{{\boldsymbol{x}}}^{*}_{0,t}({\boldsymbol{\epsilon}}_{t})$ for
$T-t_{edit}<t$, in which $t_{edit}$ is point where we stop the gradient
guidance. Using the saved intermediate denoised samples, our regularization
loss is defined as:

$\ell_{reg}=d(\hat{{\boldsymbol{x}}}^{*}_{0,t}({\epsilon}^{\theta*}_{t}),\hat{\boldsymbol{x}}_{0,t}({\epsilon}^{\theta}_{t}))$

in where we use $l_{1}$ loss for the distance metric $d(\cdot,\cdot)$.

Our regularization has two main effects. Firstly, it provides a general effect
of guiding the output to follow the attributes of the source image by
preventing significant deviations. Secondly, since the guidance target is also
a denoised image at a specific timestep $t$, it helps to prevent the
manipulated clean image $\hat{{\boldsymbol{x}}}^{\prime}_{t}$ from deviating
from original manifold. This constraint leads to an improvement in the quality
of the generation output, as demonstrated in the ablation study included in
our appendix.

##  5 Experimental Results

#### Experimental Details

For implementation, we refer to the official source code of DiffuseIT [19].
For experiment, we use unconditional score model pre-trained with Imagenet
256$\times$256 resolution datasets [38]. In all the experiments, we used
diffusion step of $T=60$. We only used guided reverse step until
$t>T-t_{edit}$. We set $t_{edit}=20$ for image diffusion model. The generation
process takes 15 seconds per image in single RTX 3090 including forward and
reverse DDIM in text-guided I2I. Our detailed experimental settings are
elaborated in Appendix.

Figure 4: Qualitative comparison of text-guided translation on Animals and
Landscape dataset. Our model shows better perceptual quality than the
baselines.

Method | Animals | Landscapes | Time
---|---|---|---
SFID$\downarrow$ | CSFID$\downarrow$ | LPIPS$\downarrow$ | SFID$\downarrow$ | CSFID$\downarrow$ | LPIPS$\downarrow$ | sec
VQGAN-CLIP | 30.01 | 65.51 | 0.462 | 33.31 | 82.92 | 0.571 | 8s
CLIP-GD | 12.50 | 53.05 | 0.468 | 18.13 | 62.19 | 0.458 | 25s
DiffusionCLIP | 25.09 | 66.50 | 0.379 | 29.85 | 76.29 | 0.568 | 70s
FlexIT | 32.71 | 57.87 | 0.215 | 18.04 | 60.04 | 0.243 | 26s
Asyrp | 31.41 | 89.60 | 0.338 | 23.65 | 74.32 | 0.457 | 65s
DiffuseIT | 9.98 | 41.07 | 0.372 | 16.86 | 54.48 | 0.417 | 40s
Ours | 9.64 | 38.12 | 0.336 | 14.33 | 54.11 | 0.414 | 15s

Table 1:  Quantitative comparison for the text-guided image translation. Our
model outperforms baselines in overall scores for both of Animals and
Landscapes datasets with faster sampling.

#### Text-guided Image Translation

In order to assess the effectiveness of our text-guided image translation
approach, we performed comparisons against the state-of-the-art baseline
models which use image diffusion models. The chosen baseline methods were
recently introduced models that utilize pre-trained CLIP for text-guided image
manipulation : VQGAN-CLIP [22], CLIP-guided diffusion (CGD) [15],
DiffusionCLIP [24], FlexIT [39], Asyrp [25], and DiffuseIT [19]. To ensure
consistency, we referred to the official source codes of all the baseline
methods.

Following the same experimental protocol used in DiffuseIT, we show
quantitative and qualitative evaluation on natural image translation. For
testing our translation performance, we use two different datasets: animal
faces [40] and landscapes [41]. The animal face dataset contains 14 classes of
animal face images, and the landscapes dataset consists of 7 classes of
various natural landscape images.

For quantitative metrics, we used three different metrics following the
baseline DiffuseIT [19] : simplified FID (SFID) [42], class-wise SFID (CSFID),
and LPIPS. For overall image quality measurement, we used SFID score. For
caterogy-wise quality check, we show CSFID. To measure the input-output
structural consistency, we calculated LPIPS distance between source and
outputs. Further experimental settings can be found in our Supplementary
Materials.

In Table 1, we show the quantitative comparison results. Our model
demonstrated superior performance in image quality measurement using SFID and
CSFID compared to all baseline methods. Regarding content preservation
measured by the LPIPS score, our approach achieved the second-best
performance. The FlexIT model obtained the highest score in LPIPS since it was
directly trained with LPIPS loss. However, an excessively low LPIPS value is
undesirable as it indicates a failure in achieving proper semantic changes.
With respect to the translation time, training-based methods Asyrp and
DiffusionCLIP require longer time. Comparing with other sampling-based
methods, our model showed relatively faster sampling speed. Considering the
output quality, our model is the most efficient method for I2I.

In qualitative comparison in Fig. 4, our results accurately capture the
semantic features of the target texts while preserving the content. On the
other hand, baseline models suffered from poor sample quality or failure to
change the domain. To further evaluate the perceptual quality of generated
samples, we show user study results in our Appendix.

Figure 5: Qualitative comparison of image-guided image translation. Our
results have better perceptual quality than the baseline outputs.

#### Image-guided Image Translation

To assess the image translation guided by target images, we conducted
comparison experiments to evaluate its performance. Specifically, we compared
our model against appearance transfer models such as DiffuseIT [19], Splicing
ViT [20], STROTSS [29], and style transfer methods WCT2 [28].

Figure 5 presents the qualitative comparison results of the image-guided
translation task. Our model successfully generated outputs that accurately
captured the semantic styles of the target images while preserving the content
from the source images. Conversely, the other models exhibited severe content
deformations or inadequately reflected the semantic styles. Additionally, we
show a user study to evaluate the overall perceptual quality in Supplementary
Materials.

Figure 6: Qualitative comparison of text-guided translation with latent
diffusion model. Our model generates realistic samples that reflects the both
of source image attribute and target text condition, with better perceptual
quality than the baselines.

#### Experiment on Latent Diffusion Models

As baseline models, we used stable diffusion I2I (S-I2I), prompt-to-prompt
(P2P) [17], plug-and-play diffusion model (PnP) [18], and Diffedit [43], which
are real image manipulation methods. Except for the simple forward-reverse
process of S-I2I, all other methods required real image inversion, which took
more than 2 minutes. In contrast, our method only utilized the sampling
process, which took less than 20 seconds. Detailed time comparisons and
experimental details on latent diffusion model can be found in Appendix.

Fig. 6 presents the results obtained using our latent diffusion model. Our
model successfully achieved semantic changes in images within a short time. In
contrast, the baselines exhibited limitations such as the inability to achieve
proper semantic changes or failure to preserve the shape of the source image.
Recent models, particularly PnP diffusion, showed pleasing quality in natural
image translation but failed when applied to artificial images.

Figure 7: Various applications of our proposed method. Our method can be used
for overall style transfer and attribute editing.

#### Other applications

By adjusting the regularization loss $\lambda_{reg}$, we can expect various
effects. Reducing the loss strength allows for overall style changes,
resembling style transfer, while increasing the loss strength and making all
steps deterministic enables editing of the source image attributes. In Fig. 7,
we show the results of style transfer and image editing. More samples and
experimental details are provided in our Supplementary Material.

##  6 Societal Impacts, Limitations, and Reproducibility

Figure 8: Example of failure case.

Societal impacts. Our method can contribute to society by empowering fields
such as entertainment and the arts through conditional generation. However,
there are also negative aspects, such as the malicious use of the method to
create deepfakes. These concerns can be addressed through regulatory measures.

Limitations. Despite the good performance in image translation, it has
limitations. Since we use pre-trained CLIP model, there is a problem when a
text condition has large distance from source image in CLIP space (e.g., lion
-> building) as shown in Fig. 8. This limitation can be addressed in the
future with the development of better text embedding models.

Reproducibility. In supplementary materials, we provide an anonymous link
containing codes, algorithms and hyper-parameters.

##  7 Conclusion

In this paper, we propose a novel method that utilizes asymmetric gradient
guidance during the reverse sampling process to transform the source image to
the target domain. Our proposed method enables more stable and faster image
translation compared to existing reverse sampling approaches. Our experimental
results demonstrate improved performance compared to recently proposed state-
of-the-art image translation models, achieving successful image-to-image (I2I)
translation in both image- and latent-diffusion models. Furthermore, our model
is not confined to semantic I2I, with showing variety of applications such as
style transfer and image editing.

## References

  * [1] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  NIPS’14, page 2672–2680, Cambridge, MA, USA, 2014. MIT Press.
  * [2] Sagie Benaim and Lior Wolf.  One-sided unsupervised domain mapping.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [3] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, Kun Zhang, and Dacheng Tao.  Geometry-consistent generative adversarial networks for one-sided unsupervised domain mapping.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [4] Taesung Park, Alexei A. Efros, Richard Zhang, and Jun-Yan Zhu.  Contrastive learning for unpaired image-to-image translation.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 319–345, Cham, 2020. Springer International Publishing.
  * [5] Chanyong Jung, Gihyun Kwon, and Jong Chul Ye.  Exploring patch-wise semantic relation for contrastive learning in image-to-image translation tasks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18260–18269, June 2022.
  * [6] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz.  Multimodal unsupervised image-to-image translation.  In ECCV, 2018.
  * [7] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  StarGAN v2: Diverse image synthesis for multiple domains.  pages 8188–8197, 2020.
  * [8] Y. Shen, J. Gu, X. Tang, and B. Zhou.  Interpreting the latent space of gans for semantic face editing.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9240–9249, Los Alamitos, CA, USA, jun 2020. IEEE Computer Society.
  * [9] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  Ganspace: Discovering interpretable gan controls.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9841–9850. Curran Associates, Inc., 2020.
  * [10] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020, 2021.
  * [11] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085–2094, 2021.
  * [12] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  arXiv preprint arXiv:2108.00946, 2021.
  * [13] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [14] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * [15] Katherine Crowson.  Clip-guided diffusion.  2022\.
  * [16] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022.
  * [17] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or.  Prompt-to-prompt image editing with cross-attention control.  In The Eleventh International Conference on Learning Representations, 2023.
  * [18] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  arXiv preprint arXiv:2211.12572, 2022.
  * [19] Gihyun Kwon and Jong Chul Ye.  Diffusion-based image translation using disentangled style and content representation.  In The Eleventh International Conference on Learning Representations, 2023.
  * [20] Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel.  Splicing vit features for semantic appearance transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10748–10757, 2022.
  * [21] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye.  Improving diffusion models for inverse problems using manifold constraints.  arXiv preprint arXiv:2206.00941, 2022.
  * [22] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff.  Vqgan-clip: Open domain image generation and editing with natural language guidance.  arXiv preprint arXiv:2204.08583, 2022.
  * [23] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208–18218, 2022.
  * [24] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426–2435, 2022.
  * [25] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh.  Diffusion models already have a semantic latent space.  In The Eleventh International Conference on Learning Representations, 2023.
  * [26] Serin Yang, Hyunmin Hwang, and Jong Chul Ye.  Zero-shot contrastive loss for text-guided diffusion image style transfer, 2023.
  * [27] Dae Young Park and Kwang Hee Lee.  Arbitrary style transfer with style-attentional networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5880–5888, 2019.
  * [28] Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, and Jung-Woo Ha.  Photorealistic style transfer via wavelet transforms.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9036–9045, 2019.
  * [29] Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich.  Style transfer by relaxed optimal transport and self-similarity.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10051–10060, 2019.
  * [30] Gihyun Kwon and Jong Chul Ye.  One-shot adaptation of gan in just one clip.  arXiv preprint arXiv:2203.09301, 2022.
  * [31] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang.  Few-shot image generation via cross-domain correspondence.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10743–10752, 2021.
  * [32] Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka.  Mind the gap: Domain gap control for single shot domain adaptation for generative adversarial networks.  arXiv preprint arXiv:2110.08398, 2021.
  * [33] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [34] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In International Conference on Learning Representations, 2021.
  * [35] Hyungjin Chung, Suhyeon Lee, and Jong Chul Ye.  Fast diffusion sampler for inverse problems by geometric decomposition, 2023.
  * [36] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Universal guidance for diffusion models, 2023.
  * [37] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660, 2021.
  * [38] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  * [39] Guillaume Couairon, Asya Grechka, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Flexit: Towards flexible semantic image translation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18270–18279, 2022.
  * [40] Zhangzhang Si and Song-Chun Zhu.  Learning hybrid image templates (hit) by information projection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(7):1354–1367, 2012.
  * [41] Yang Chen, Yu-Kun Lai, and Yong-Jin Liu.  Cartoongan: Generative adversarial networks for photo cartoonization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9465–9474, 2018.
  * [42] Chung-Il Kim, Meejoung Kim, Seungwon Jung, and Eenjun Hwang.  Simplified fréchet distance for generative adversarial nets.  Sensors, 20(6), 2020.
  * [43] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Diffedit: Diffusion-based semantic image editing with mask guidance.  In The Eleventh International Conference on Learning Representations, 2023.
  * [44] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models, 2022.
  * [45] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam.  Searching for mobilenetv3, 2019.
  * [46] Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick.  Benchmarking detection transfer learning with vision transformers, 2021\.

##  Appendix A Experiment on Latent Diffusion Models

As described in our main paper, out method can be easily adapted to the recent
Latent Diffusion Model (LDM) [14]. With encoded latent ${\boldsymbol{z}}$, we
can obtain noisy latent ${\boldsymbol{z}}_{t}$ with the same process of the
forward DDIM process of image diffusion models. With re-iterating our sampling
process, we can define our update formula for latent diffusion model:

$\displaystyle{\boldsymbol{z}}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\hat{{\boldsymbol{z}}}^{\prime}_{t}+D_{t}({\boldsymbol{\epsilon}}_{\theta}({\boldsymbol{z}}_{t},t,c_{trg}))+\sigma_{t}\epsilon,$
(17)

where the following two forms are interchangely used for
$\hat{\boldsymbol{z}}_{t}^{\prime}$:

$\displaystyle\hat{{\boldsymbol{z}}}^{\prime}_{t}=\begin{cases}\hat{\boldsymbol{z}}_{0,t}({{\boldsymbol{\epsilon}}}_{t})-\nabla_{{\boldsymbol{z}}_{t}}\ell(\hat{\boldsymbol{z}}_{0,t}({\boldsymbol{z}}_{t}))\\\
\hat{\boldsymbol{z}}_{0,t}(\tilde{{\boldsymbol{\epsilon}}}_{t}),\quad\mbox{where}\quad\tilde{{\boldsymbol{\epsilon}}}_{t}:={{\boldsymbol{\epsilon}}}_{t}-s_{t}\nabla_{{\boldsymbol{z}}_{t}}\ell(\hat{\boldsymbol{z}}_{0,t}({\boldsymbol{\epsilon}}_{t}))\end{cases}$
(18)

where $c_{trg}$ is the text embedding of target text $t_{trg}$, and
$\epsilon\sim{\mathcal{N}}(0,{\boldsymbol{I}})$,
${\boldsymbol{\epsilon}}_{t}={\boldsymbol{\epsilon}}(z_{t},t,c_{trg})$, and
$\nabla_{{\boldsymbol{z}}_{t}}\ell(\hat{\boldsymbol{z}}_{0,t}({\boldsymbol{\epsilon}}_{t}))$
refers the MCG gradient.

Similar to the image diffusion case, we apply additional DDS process for
hybrid update of $\hat{{\boldsymbol{z}}}^{\prime}_{t}$ for Eq. Eq. (17) with
the following formula:

$\displaystyle\hat{{\boldsymbol{z}}}_{t}^{\prime}=\bar{\boldsymbol{z}}_{t}^{\prime}+\arg\min_{\Delta}\ell(\bar{\boldsymbol{z}}_{t}^{\prime}+\Delta),$
$\displaystyle\quad\mbox{where}\quad\bar{\boldsymbol{z}}_{t}^{\prime}:=\hat{\boldsymbol{z}}_{0,t}(\tilde{{\boldsymbol{\epsilon}}}_{t}).$

The argmin operation in the right hand side is done without MGC gradient using
Adam optimizer, and the perturbed model output
$\tilde{\boldsymbol{\epsilon}}_{t}$ is computed by Eq. (18).

In latent diffusion case, we can optionally leverage the cross attention map
of text conditions. Especially, we obtain the semantic mask for the specific
word in sentence. With following the recent model [17], we can obtain the
saliency map by aggregating the cross attenton map $A$ of specific word. With
applying threshold to saliency map, we can obtain semantic mask $M$, which
cover the targe object we want to translate. More specifcally, with obtained
mask, we can use differentiated guided sampling strategy, in which we apply
guided sampling until different timesteps with respect to masks:

$\displaystyle\hat{{\boldsymbol{z}}}^{\prime}_{t}=\begin{cases}\hat{{\boldsymbol{z}}}^{\prime}_{t},\quad\mbox{where}\quad
T>t>T-t_{edit1}\\\
(1-M)*\hat{{\boldsymbol{z}}}^{\prime}_{t}+M*\hat{{\boldsymbol{z}}}_{0,t}({{\boldsymbol{\epsilon}}}_{t}),\quad\mbox{where}\quad
T-t_{edit1}\geq t>T-t_{edit2}\\\
\hat{{\boldsymbol{z}}}_{0,t}({{\boldsymbol{\epsilon}}}_{t}),\quad\mbox{where}\quad
T-t_{edit2}\geq t\end{cases}$ (19)

With the manipulated denoised image, we can predict next sample output with
the formula Eq. (17).

For the loss function, we only use regularization loss term in our latent
diffusion case, as the model already incorporates text condition.
Specifically, in the forward sampling of DDIM, we can deterministically
calculate the intermediate samples such as:

$\displaystyle{{\boldsymbol{z}}}^{*}_{t+1}={\sqrt{\bar{\alpha}_{t+1}}\hat{{\boldsymbol{z}}}^{*}_{0,t}({\boldsymbol{\epsilon}}(z_{t},t,c_{src}))}+{\sqrt{1-\bar{\alpha}_{t+1}}{\boldsymbol{\epsilon}}(z_{t},t,c_{src})},$
(20)

where ${{\boldsymbol{z}}}^{*}_{t}$ is calculated noisy image from source
latent obtained from encoder
${{\boldsymbol{z}}}_{src}=Enc({\boldsymbol{x}}_{src})$, and
$\hat{{\boldsymbol{z}}}^{*}_{0,t}({\boldsymbol{\epsilon}}(z_{t},t,c_{src}))$
is estimated denoised image calculated using Tweedie’s formula. Using the
saved intermediate denoised samples, our regularization loss is defined as:

$\ell_{reg}=d(\hat{{\boldsymbol{z}}}^{*}_{0,t}({{\boldsymbol{\epsilon}}}(z^{*}_{t},t,c_{src})),\hat{\boldsymbol{z}}_{0,t}({{\boldsymbol{\epsilon}}}(z_{t},t,c_{trg})))$

in where we use $l_{1}$ loss for the distance metric $d(\cdot,\cdot)$.

##  Appendix B Experimental Details

For experiment hyperparameters, we set $\lambda_{reg}=200$ and
$\lambda_{sty}=200$ in MCG gradient calculation for image diffusion models.
For additional DDS optimization part, we used the same loss function as MCG,
using Adam optimizer with learning rate of 0.02. In image diffusion model
case, we used only one or two steps of DDS optimization, as it showed marginal
improvement with more steps. For all of the experiments, we give moderate
random stochastic noise with setting $\sigma_{t}=0.8$, as it showed better
perceptual quality.

For style loss, we used the loss function proposed in DiffuseIT [19]. More
specifically, for text-guided image translation, our style loss is defined as:

$\displaystyle\ell_{CLIP}({\boldsymbol{x}};{\boldsymbol{d}}_{trg},{\boldsymbol{x}}_{src},{\boldsymbol{d}}_{src}):=-\mathrm{sim}({\boldsymbol{v}}_{trg},{\boldsymbol{v}}_{src})$
(21)

where

$\displaystyle{\boldsymbol{v}}_{trg}:=E_{T}({\boldsymbol{d}}_{trg})+\lambda_{i}E_{I}({\boldsymbol{x}}_{src})-\lambda_{s}E_{T}({\boldsymbol{d}}_{src}),$
$\displaystyle\quad{\boldsymbol{v}}_{src}:=E_{I}(\mathrm{aug}({\boldsymbol{x}}))$
(22)

where $E_{T}$ and $E_{I}$ is text and image encoder, respectively, and
${\boldsymbol{d}}_{src}$,${\boldsymbol{d}}_{trg}$,${\boldsymbol{x}}_{src}$
represents source text, target text and source image, respectively. In this
case, ${\boldsymbol{x}}$ is estimated denoised image; $\mathrm{aug(\cdot)}$
denotes the augmentation for preventing adversarial artifacts from CLIP. For
parameters, we use exactly same values proposed in DiffuseIT.

For image-guided image translation, we also use proposed loss in DiffuseIT,
which is defined as:

$\displaystyle\ell_{sty}({\boldsymbol{x}}_{trg},{\boldsymbol{x}})=||{\boldsymbol{e}}_{[CLS]}^{L}({\boldsymbol{x}}_{trg})-{\boldsymbol{e}}_{[CLS]}^{L}({{\boldsymbol{x}}})||_{2}+\lambda_{mse}||{\boldsymbol{x}}_{trg}-{{\boldsymbol{x}}}||_{2}.$
(23)

where ${\boldsymbol{e}}_{[CLS]}^{L}$ denotes the last layer [CLS] token of
DINO ViT model.

For experiments of latent diffusion models, we used public stable diffusion
model with version v1.5. We used total timesteps $T=50$ for all experiments.
For hyperparameters, we set $\lambda_{reg}=0.1$ in MCG gradient calculation.
For additional DDS optimization, we used Adam with weight decay (AdamW)
optimizer with learning rate of 0.01. We used 10 DDS optimization steps per
each timestep. For editing timesteps, we set $t_{edit2}=40$, and $t_{edit1}$
is varied between $[10:20]$. We provide the source code in our official
repository111https://github.com/submissionanon18/AGG.

##  Appendix C User Study

Method |  Text$\uparrow$ |  Realism$\uparrow$ |  Content$\uparrow$
---|---|---|---
VQGAN-CLIP | 2.20 | 1.61 | 1.77
CLIP-GD | 2.47 | 1.47 | 1.38
DiffusionCLIP | 1.73 | 1.93 | 2.43
FlexIT | 2.12 | 3.36 | 3.69
Asyrp | 2.79 | 2.69 | 2.77
DiffuseIT | 2.81 | 3.71 | 3.54
Ours | 4.07 | 4.14 | 4.23

Table 2:  User study for the text-guided image translation on image diffusion
models. Our model outperforms baselines in overall scores.

Method |  Style$\uparrow$ |  Realism$\uparrow$ |  Content$\uparrow$
---|---|---|---
WCT2 | 2.28 | 4.46 | 4.28
STROTSS | 3.28 | 3.58 | 3.72
SpliceViT | 2.85 | 1.92 | 2.39
DiffuseIT | 3.07 | 2.07 | 2.70
Ours | 4.21 | 3.79 | 3.85

Table 3:  User study for the image-guided image translation on image diffusion
models. Our model outperforms baselines in overall scores.

Method |  Text$\uparrow$ |  Realism$\uparrow$ |  Content$\uparrow$
---|---|---|---
S-I2I | 2.84 | 3.47 | 3.30
P2P | 4.19 | 3.82 | 3.59
DiffEdit | 2.74 | 2.61 | 3.35
PnP | 3.94 | 2.83 | 2.43
Ours | 3.82 | 3.89 | 4.05

Table 4:  User study for the text-guided image translation on latent diffusion
models.

Method | Time
---|---
WCT2 | 0.18s
STROTSS | 53s
SpliceViT | 25m 30s
DiffuseIT | 37s
Ours | 15s

Method | Time
---|---
S-I2I | 5s
P2P | 2m 8s
DiffEdit | 12s
PnP | 2m 36s
Ours | 17s

Table 5: Time comparison results for evaluating the efficiency of our model.

For evaluation of perceptual quality of our proposed model, we conducted user
study with following similar protocol to baseline model [19].

First, we conducted a user study to evaluate the performance of our proposed
and baseline models in text-guided image translation. The study included 49
images generated from 7 different text conditions. Participants were asked
questions regarding three aspects: 1) whether the outputs accurately conveyed
the semantic meaning of the target text (Text-match), 2) the realism of the
generated images (Realism), and 3) the preservation of the content information
from the source images (Content). A total of 20 users from different age
groups (20s and 50s) were randomly recruited for the study. The participants
were provided with the questions using Google Form and were asked to rate the
answers on a scale from 1 to 5, where 1 represented "Very Unlikely" and 5
represented "Very Likely." In the user study on image-guided translation, we
generated 20 distinct images. We then followed a similar procedure as the
text-guided image translation study, with the exception of the question
content. Participants were asked to provide feedback in three categories: 1)
whether the outputs accurately conveyed the semantic meaning of the target
style image (Style-match), 2) the realism of the generated images (Realism),
and 3) the preservation of content information from the source images
(Content). For the user study involving the Latent diffusion model, we
presented 20 different images and gathered user opinions. In this case, we
employed the exact same set of questions used in our text-guided translation
study on image diffusion models.

Based on the criterion, we first evaluated the performance of text-guided
image translation applied to image diffusion model. In Table 2, we show the
user study results. In overall scores including text-matching, realism, and
content preservation, our model scored the best among other baseline models.
Also we show the user study result of image-guided image translation in Table
3. Our model scored the best in style-matching score and the second best in
realism and content preservation score. In WCT2, the model shows the best
score in realism and content preservation, but severely low score in style
matching score. This indicates that the model could not properly change the
source images into the target domain. Finally, we show the user study results
on latent-diffusion based models. In Text-matching semantic score, baseline
P2P, PnP and our model all showed decent scores. However, ours showed the best
score in realism and content-preservation score. As already shown in our main
script, baseline of PnP and P2P shows good performance in generating images
which has proper semantic of target texts, but they show degraded performance
in preserving the attribute of original images.

##  Appendix D Computation Time Comparison

To compare the computation times taken for image translation, we show the
comparison results in Table 5. In image-guided translation, our model shows
the second-best performance in translation time. For baseline models of Stable
diffusion I2I and DiffEdit, the consumed times are relatively low as they are
only applied to sampling steps. However, they suffer from degraded performance
for image translation. In advanced method of P2P and PnP, they show comparable
performance in image translation, but they require longer time (>2min) since
they require additional process of inversion. For P2P, we incorporated the
standard inversion process of null-text inversion [44]. For PnP, we used the
DDIM-inversion process with following the official source code.

##  Appendix E Ablation Study

Figure 9: Qualitative ablation study results. (a) Our best setting. (b) Outputs without proposed $\ell_{reg}$. (c) Generated images only using DDS. (d) Outputs with Vit-based regularization (e) Outputs without our proposed asymmetric update rule.  Settings | Animals
---|---
SFID$\downarrow$ |  CSFID$\downarrow$ |  LPIPS$\downarrow$
Symmetric update (e) | 4.04 | 60.87 | 0.260
ViT Loss instead of $\ell_{reg}$ (d) | 21.24 | 50.04 | 0.312
Only DDS (c) | 29.66 | 71.01 | 0.460
No $\ell_{reg}$ (b) | 14.65 | 41.17 | 0.448
Ours (a) | 9.64 | 38.12 | 0.336
Table 6: Quantitative comparison of ablation studies.

To verify the effectiveness of our proposed components, we show the ablation
study results. In Fig. 9, we show the results using different experimental
settings. (a) Firstly, we show the generated outputs with our best setting.
(b) To evaluate out proposed regularization loss, we generated results without
using our proposed $\ell_{reg}$. (c) We generated results using only DDS
optimization. (d) We show the generated results using ViT regularization loss
proposed in DiffuseIT [19], instead of our proposed $\ell_{reg}$. (e) We show
the generated images using symmetric update rule in which we apply MCG
gradient to both of denoised and noise components.

In case of (b), we can see the content structure is severely degraded. In (c)
and (d), we can observe that the results maintain the structure of source
images, but they suffer from unwanted artifacts. In (e), the model could not
change the semantic of source image to target texts. These pattern are also
exhibited in our quantitative ablation study in Table 6. Without asymmetric
update in (e), SFID and LPIPS scores are extremely low, while class-wise FID
shows relatively high score. This indicates that the model hardly changed the
semantic of source images. With using ViT loss instead of our proposed
regularization loss, we can see the LPIPS score is lower than ours, but the
image quality is degraded due to artifacts. For (c) and (b), the models failed
in image translation with degraded scores compared to our model. In overall,
our proposed setting shows the best quality.

##  Appendix F Algorithms

For detailed explanation, we show algorithms in Algorithm 1.

Algorithm 1 Image translation: given a diffusion score model
$\epsilon_{\theta}({\boldsymbol{x}}_{t},t)$, CLIP model, and VIT model

Input: source image ${\boldsymbol{x}}_{src}$, diffusion steps $T$, editing
timestep $t_{edit}$ target text ${\boldsymbol{d}}_{trg}$, source text
${\boldsymbol{d}}_{src}$ or target image ${\boldsymbol{x}}_{trg}$
Output: translated image $\hat{{\boldsymbol{x}}}$ which has semantic of
${\boldsymbol{d}}_{trg}$ (or ${\boldsymbol{x}}_{trg}$) and content of
${\boldsymbol{x}}_{src}$
${\boldsymbol{x}}_{src}={\boldsymbol{x}}^{*}_{0}$

1:for all $t$ from 0 to $T$ do

2:
$\epsilon^{*}\leftarrow\boldsymbol{\epsilon}_{\theta}({\boldsymbol{x}}^{*}_{t},t)$

3:
$\hat{{\boldsymbol{x}}}^{*}_{0,t}\leftarrow\frac{{\boldsymbol{x}}^{*}_{t}-\sqrt{1-\bar{\alpha}_{t}}\epsilon^{*}}{\sqrt{\bar{\alpha}_{t}}}$

4: save $\hat{{\boldsymbol{x}}}^{*}_{0,t}$

5:
${\boldsymbol{x}}^{*}_{t+1}=\sqrt{\bar{\alpha}_{t+1}}\hat{{\boldsymbol{x}}}^{*}_{0,t}+\sqrt{1-\bar{\alpha}_{t+1}-\sigma_{t}^{2}}\epsilon^{*}+\sigma_{t}{\boldsymbol{z}}_{t}$

6:end for

7:${\boldsymbol{x}}_{T}\leftarrow{\boldsymbol{x}}^{*}_{T}$

8:for all $t$ from $T$ to 0 do

9: $\epsilon\leftarrow\boldsymbol{\epsilon}_{\theta}({\boldsymbol{x}}_{t},t)$

10:
$\hat{{\boldsymbol{x}}}_{0,t}\leftarrow\frac{{\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\epsilon}{\sqrt{\bar{\alpha}_{t}}}$

11: if $T$> $t$ >$T-t_{edit}$ then

12: if text-guided then

13:
$\ell_{total}(\hat{{\boldsymbol{x}}}_{0,t})\leftarrow[\ell_{CLIP}(\hat{{\boldsymbol{x}}}_{0,t};{\boldsymbol{d}}_{trg},{\boldsymbol{x}}_{src},{\boldsymbol{d}}_{src})+d(\hat{{\boldsymbol{x}}}_{0,t},\hat{{\boldsymbol{x}}}^{*}_{0,t})]$

14: else if image-guided then

15:
$\ell_{total}(\hat{{\boldsymbol{x}}}_{0,t})\leftarrow[\ell_{sty}(\hat{{\boldsymbol{x}}}_{0,t};{\boldsymbol{x}}_{trg})+d(\hat{{\boldsymbol{x}}}_{0,t},\hat{{\boldsymbol{x}}}^{*}_{0,t})]$

16: end if

17: $\tilde{\epsilon}\leftarrow\epsilon-
s_{t}\nabla_{{\boldsymbol{x}}_{t}}\ell_{total}(\hat{{\boldsymbol{x}}}_{0,t})$

18:
$\bar{{\boldsymbol{x}}}_{0,t}\leftarrow\hat{{\boldsymbol{x}}}_{0,t}(\tilde{\epsilon})$

19: if use-DDS then

20:
$\hat{{\boldsymbol{x}}}^{\prime}_{0,t}\leftarrow\bar{{\boldsymbol{x}}}_{0,t}+\text{arg}\min_{\Delta}\ell_{total}(\bar{{\boldsymbol{x}}}_{0,t}+\Delta)$

21: else

22:
$\hat{{\boldsymbol{x}}}^{\prime}_{0,t}\leftarrow\bar{{\boldsymbol{x}}}_{0,t}$

23: end if

24: ${\boldsymbol{z}}\sim\mathcal{N}(0,\mathbf{I})$

25:
${\boldsymbol{x}}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\hat{{\boldsymbol{x}}}^{\prime}_{0,t}+\sqrt{1-\bar{\alpha}_{t-1}-\sigma_{t}^{2}}\epsilon+\sigma_{t}{\boldsymbol{z}}_{t}$

26: else

27: ${\boldsymbol{z}}\sim\mathcal{N}(0,\mathbf{I})$

28:
${\boldsymbol{x}}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\hat{{\boldsymbol{x}}}_{0,t}+\sqrt{1-\bar{\alpha}_{t-1}-\sigma_{t}^{2}}\epsilon+\sigma_{t}{\boldsymbol{z}}_{t}$

29: end if

30:end for

31:return ${\boldsymbol{x}}_{-1}$

##  Appendix G Additional Experiments

In this section, we show additional generated samples. First, we show the
image-translation results using image diffusion models pre-trained on human
face dataset. In Fig. 10, the results show that our model can be easily
applied to human face editing. We can change the face attribute using target
text conditions, while preserting the original facial features.

For the additional applications, we can replace our regularization loss into
other losses such as cross-entropy of semantic segmentation model or object
detection model. In this case, we use large number of DDS steps more than
2,000. During DDS step, we used resampling strategy in which we apply one
reverse and forward DDIM step after each 100 DDS steps with following the
strategy of [36]. We applied our guided sampling to initial 3 to 5 timesteps,
as the initial stage of diffusion sampling plays major role in the overall
generated contents. For semantic segmantation, we combined pre-trained model
from previous work [45]. In object detection model, we also used pre-trained
model [46].

In Fig. 11, we show the generated output using segmentation model. We can
generate text-conditioned images in which the image components are aligned
with semantic segmentation maps. In Fig. 12, we also show the outputs using
pre-trained object detection model. We can generate images in which the
objects positions are aligned with given bounding box information.

To show more image translation examples using our method, we show additional
samples generated from latent diffusion models in Fig. 13.

Figure 10: Qualitative results on human face image translation.  Figure 11:
Qualitative results of generated images using pre-trained semantic
segmentation models.  Figure 12: Qualitative results of generated images using
pre-trained object detection model.

Figure 13: Qualitative results of generated images using latent diffusion
model.