# Missed opportunities: GRB 211211A and the case for continual gravitational-
wave coverage with a single observatory.

Nikhil Sarin1,2, Paul D. Lasky3,4, Rowina S. Nathan3,4
1Nordita, Stockholm University and KTH Royal Institute of Technology Hannes
Alfvéns väg 12, SE-106 91 Stockholm, Sweden
2The Oskar Klein Centre, Department of Physics, Stockholm University,
AlbaNova, SE-106 91 Stockholm, Sweden
3School of Physics and Astronomy, Monash University, Clayton, VIC 3800,
Australia
4OzGrav: The ARC Centre of Excellence for Gravitational-wave Discovery,
Clayton, VIC 3800, Australia

###### Abstract

Gamma-ray burst GRB 211211A may have been the result of a neutron star merger
at $\approx 350$ Mpc. However, none of the LIGO-Virgo detectors were operating
at the time. We show that the gravitational-wave signal from a GRB
211211A-like binary neutron star inspiral in the next LIGO-Virgo-KAGRA
observing run (O4) would be below the conventional detection threshold,
however a coincident gamma-ray burst observation would provide necessary
information to claim a statistically-significant multimessenger observation.
We calculate that with O4 sensitivity, approximately $11\%$ of gamma-ray
bursts within 600 Mpc will produce a confident association between the
gravitational-wave binary neutron star inspiral signature and the prompt
gamma-ray signature. This corresponds to a coincident detection rate of
$0.22^{+8.3}_{-0.22}\,\mathrm{yr^{-1}}$, where the uncertainties are the 90%
confidence intervals arising from uncertainties in the absolute merger rate,
beaming and jet-launching fractions. These increase to approximately $34\%$
and $0.71^{+26.8}_{-0.70}\,\mathrm{yr^{-1}}$ with proposed O5 sensitivity. We
show that the above numbers do not depend significantly on the number of
gravitational-wave observatories operating with the specific sensitivity. That
is, the number of confident joint gamma-ray burst and gravitational-wave
detections is only marginally improved with two or three detectors operating
compared to a single detector. It is therefore worth considering whether one
detector with sufficient sensitivity (post O4) should remain in sky-watch mode
at all times to elucidate the true nature of GRB 211211A-like events, a
proposal we discuss in detail.

###### keywords:

transients: neutron star mergers – transients: gamma-ray bursts

##  1 Introduction

GRB 211211A was a long-duration gamma-ray burst associated with an optical and
infrared excess consistent with a kilonova (D’Ai et al., 2021; Fermi GBM Team,
2021). One possible scenario is that GRB 211211A was the result of a binary
neutron star merger, where the prolonged high-energy gamma-ray emission was
due to a long-lived millisecond magnetar merger remnant or magnetic activity
in an accretion disk (Rastinejad et al., 2022; Gompertz et al., 2022; Troja et
al., 2022; Gao et al., 2022; Suvorov et al., 2022). GRB 211211A observations
are also consistent with the merger of a neutron star and a low-mass black
hole, with the kilonova arising from both disk and dynamical ejecta
(Rastinejad et al., 2022; Gompertz et al., 2022). The infrared excess could
have also been the thermal emission from dust and not a signature of a
kilonova (Waxman et al., 2022), in which case GRB 211211A is a somewhat
typical long gamma-ray burst. Additional constraints from another messenger,
had they existed, would have provided the necessary information to understand
the physics of GRB 211211A.

At the distance of GRB 211211A ($\approx 350$ Mpc), a
$1.4+1.4\,\mathrm{M_{\odot}}$ binary neutron star merger would have a three-
detector network optimal signal-to-noise ratio (SNR) in the LIGO-Virgo
observatories (Aasi et al., 2015; Acernese et al., 2015) of 6.3, 10.3, and
17.6 with sensitivity of the third, fourth, and fifth observing runs,
respectively111Note that we cannot reproduce the signal-to-noise ratios
presented in Rastinejad et al. (2022), despite following their outlined
description.. A neutron star-black hole merger would be louder, with signal-
to-noise ratio depending on the mass of the black hole. The conventional
threshold for a confident single (network) detector gravitational-wave signal
is $8$ ($12$) (e.g., Abbott et al., 2020, and references therein). However,
these thresholds are calculated assuming no electromagnetic counterpart; it is
a subtle question whether signals in association of another messenger require
the same conventional threshold for being called a detection. In particular,
this threshold should be smaller given both the sky location and merger time
of GRB 211211A are well known, implying the trials factors associated with
gravitational-wave searches are significantly reduced. We note that some
signals in the LIGO-Virgo-KAGRA gravitational-wave catalog (Abbott et al.,
2021a) are classified as detections despite being lower than the conventional
threshold for a network detection as they satisfy false-alarm-rate and
astrophysical-significance estimate thresholds.

In this work, we show that a $1.4+1.4\,\mathrm{M_{\odot}}$ binary neutron star
merger at $350$ Mpc with GRB 211211A-like electromagnetic observations is
detectable in a LIGO-Virgo network operating at the proposed fourth observing
run sensitivity. We emphasise that such a gravitational-wave signal is below
the threshold for traditional gravitational-wave discovery, however the
coincident electromagnetic observations provide the necessary confidence to
claim a detection222We emphasise that none of our results hinge on GRB 211211A
actually being a neutron star merger, but instead we use this event as
motivation for a potential future merger. . An otherwise equivalent
$1.4+4.5\,\mathrm{M_{\odot}}$ neutron star-black hole merger with
dimensionless spin, $a=0.85$ (sufficient to disrupt the neutron star outside
the event horizon) is detectable in the network with sensitivity of the third-
observing run (O3), despite only having a network optimal signal-to-noise
ratio of 10.0.

This case study provides an opportunity to analyse potential observing
scenarios as the sensitivity of gravitational-wave instruments continues to
improve. For example, consider the current network of two LIGO, one Virgo, and
one KAGRA (Akutsu et al., 2019) observatory. As the sensitivity of these
instruments is being upgraded, is it best practice to upgrade only three
detectors at one time with one detector always online, such as the “astro-
watch” mode of GEO600 (Willke et al., 2002)? Is it optimal to have two
observatories in astro-watch mode, or should we simply minimise instrument
downtime by upgrading all instruments at once? While these questions also come
with fiduciary and societal implications, we can somewhat answer this question
from a purely scientific perspective by considering only the multi-messenger
detectability of future gravitational-wave signals.

Figure 1: Posterior distributions for GRB 211211A-like injections of a binary
neutron star (left) and neutron star-black hole merger into gravitational-wave
networks operating at the sensitivity of the third observing run. The top
panel shows the recovered posterior distributions in a three-detector network
in red, and the GRB 211211A sky localisation from XRT on board Swift. The one
dimensional marginalised posterior distributions (bottom panels) show the
coalescence time $t_{c}$ and luminosity distance $d_{L}$ assuming a three-
detector network (red) and a two-detector LIGO Livingston and Virgo network
(green). The blue histogram for $t_{c}$ assumes a four-second symmetric window
around the GRB prompt emission, and for $d_{L}$ is derived from the host-
galaxy redshift.

Searches for gravitational waves using only single observatories are in their
infancy. Statements about detection confidence require a thorough
understanding of detector noise properties because transient noise artifacts
known as glitches can mimic gravitational-wave signals in single detectors
(see e.g., Callister et al., 2017; Nitz et al., 2020; Cannon et al., 2021;
Cabourn Davies & Harry, 2022). Detection confidence can therefore be increased
by either using two or more gravitational-wave observatories searching for
nearly-simultaneous and coherent signals, or by using coincident
electromagnetic and/or neutrino observations to increase significance (e.g.,
Clark et al., 2015; Nitz et al., 2019; Magee et al., 2019; Bartos et al.,
2019; Stachie et al., 2020; Sarin & Lasky, 2022; Abbott et al., 2022b). The
latter approach increases the detection confidence of marginal gravitational-
wave signals by calculating the probability that the signals originated from
the same source.

In this work, using the current best estimate for the beaming and jet-
launching fractions of neutron star mergers, we predict that $\approx 11\%$ of
binary neutron star mergers out to $600\,\mathrm{Mpc}$ will produce a joint
gravitational-wave and gamma-ray burst observation with the proposed O4
sensitivity. This joint detection fraction rises to $\approx 34\%$ with the
proposed O5 sensitivity. This suggests a
$0.22^{+8.3}_{-0.22}\,\mathrm{yr^{-1}}$ and
$0.71^{+26.8}_{-0.70}\,\mathrm{yr^{-1}}$ (90% confidence) coincident detection
rate, respectively. We show that the rate of joint gamma-ray burst and
gravitational-wave detections is practically the same whether one, two, or
three detectors remain online, providing motivation to leave a single detector
online in astro-watch mode. We discuss the potential implications of such a
proposal in detail. We note that our approach is different to other
predictions of the rates of joint multi-messenger events in future
gravitational-wave observing runs (e.g., Petrov et al., 2022; Patricelli et
al., 2022; Colombo et al., 2022) as we consider how confidently two
independently identified events can be associated with each other, rather than
a consideration of how likely an event is able to successfully detect a
counterpart in follow-up to a gravitational-wave trigger. This approach relies
on the conventional pipelines for serendipitous electromagnetic and
gravitational-wave detection, instead of the ability to follow up a
gravitational-wave sky map.

##  2 GRB 211211A: The missed opportunity

Could gravitational waves from GRB 211211A have been detected if the LIGO-
Virgo observatories were operating at the time of the electromagnetic
observations? To answer this, we inject gravitational-wave signals from a
$1.4+1.4\,\mathrm{M_{\odot}}$ binary neutron star merger, and a
$1.4+4.5\,\mathrm{M_{\odot}}$ neutron star-black hole merger, at 350 Mpc into
a network of two LIGO observatories, as well as a network of two LIGO and one
Virgo observatories, operating at O3 sensitivity into simulated Gaussian
noise. We inject these signals at the sky location and time of GRB 211211A,
and perform Bayesian astrophysical inference to determine how well the binary
parameters, including sky localisation, coalescence time, and luminosity
distance, can be determined from these mock observations.

If one observes GRB prompt emission, the system’s orbital angular momentum is
pointing close to the line of sight ($\theta_{JN}\lesssim 10^{\circ}$; Fong et
al., 2015). We perform signal injections with inclination angle
$\theta_{JN}=0$; choosing a larger inclination angle consistent with
electromagnetic observations will not change our results substantially. We
assume zero spins for the neutron star progenitors. However, if the system was
a neutron star-black hole binary, a $\approx 4.5\,M_{\odot}$ black hole would
need to be spinning with dimensionless angular momentum $a\gtrsim 0.8$ to
ensure tidal disruption of the neutron star (Foucart et al., 2018), and hence
the observed electromagnetic emission. We therefore assume an aligned spin for
the black hole of $a=0.85$.

We perform our injections using the Bilby software suite (Ashton et al., 2019;
Romero-Shaw et al., 2020) with the Dynesty nested sampler (Speagle, 2020),
using standard priors on all quantities (see Romero-Shaw et al., 2020). In
Fig. 1, we plot the recovered sky localisation posterior distributions (top
rows) and $90\%$ credible interval of the coalescence time and luminosity-
distance posteriors (bottom rows) for the binary neutron star merger (left)
and neutron star-black hole merger (right). The sky maps show the sky-
localisation posteriors for the three-detector network in red, with the blue
reticle showing the sky localisation from the XRT monitor on board Swift. The
one-dimensional marginalised coalescence time ($t_{c}$) and luminosity-
distance ($d_{L}$) posteriors are shown for a three-detector network in red
and a two-detector Livingston-Virgo network in green. The blue posterior
distributions for the luminosity distance are from taking the host-galaxy
redshift $z=0.0763\pm 0.0002$ (Rastinejad et al., 2022), and converting to
luminosity distance using Planck-18 cosmology (Aghanim et al., 2020).

The one-dimensional marginalised posterior distributions for $t_{c}$ and
$d_{L}$ are largely uninformative for the binary neutron star injection into
the O3 network (left-hand panels of Fig. 1; note that the standard prior width
for $t_{c}$ is 0.3 s). On the other hand, the posteriors for the neutron star-
black hole injection (right panels) are informative owing to the relatively
bigger signal-to-noise ratio. For example, the posterior width on $t_{c}$ is
approximately $0.015\,\mathrm{s}$. On the other hand, the merger time as
measured from the gamma/x-ray observations is less clear. For example, the
gravitational-wave coalescence time for GW170817 was $1.7\,\mathrm{s}$ before
the prompt emission (Abbott et al., 2017b). GRB 211211A had x-ray precursor
emission approximately one second before the prompt (Xiao et al., 2022).
Taking an agnostic view towards the origin of this pre-cursor emission, we
estimate the posterior for the time of coalescence (i.e., merger time) from
the electromagnetic observations as being uniform over a four-second range
centred around the time of onset of the prompt emission. These estimated
posterior distributions are shown in blue in Fig. 1.

Determining the detectability of the gravitational-wave signal given the
electromagnetic observation requires calculating the probability that the
electromagnetic and gravitational-wave signals came from the same source. For
this we employ the bAyesian Coincident Detection Criterion (AC/DC; Ashton et
al., 2018). We calculate overlap integrals of the posterior distributions
using each of the separate data sets (i.e., gravitational-wave, prompt, and
host galaxy identification; for details of the calculation, see Ashton et al.
(2018) and Sarin & Lasky (2022)). We then calculate the odds
$\mathcal{O}_{C/R}$, where $C$ refers to the hypothesis that the independent
gravitational-wave and electromagnetic data come from a common origin, and $R$
refers to the hypothesis that they are random (i.e., not associated with one
another).

The odds is the multiplication of the prior odds $\pi_{C/R}$ with each common
parameter’s overlap integral (see Ashton et al., 2018; Sarin & Lasky, 2022,
for details). By definition, if signals come from a common event, parameters
such as the masses and spins of the binary components must be the same.
Unfortunately, models of short gamma-ray bursts and kilonovae that connect to
the intrinsic binary parameters are not robust. Instead, we only assume the
merger time, luminosity distance, and sky location are common across the
gravitational-wave and electromagnetic signal. The increase in odds obtained
by including any other parameters is proportional to how informed the
posteriors for that parameter are with electromagnetic and gravitational-wave
data relative to their prior.

Estimating the prior odds requires understanding both the rate of
gravitational-wave triggers and the rate of gamma-ray triggers (for details,
see Ashton et al., 2018)

$\pi_{C/R}=\frac{R_{\rm{GW,EM}}}{R_{\rm GW}R_{\rm EM}T}.$ (1)

Here $R_{\rm{GW,EM}}$ is the rate of astrophysical gravitational-wave and
electromagnetic signals, i.e., the rate at which neutron star binaries that
can produce an electromagnetic counterpart occur, $R_{\rm{GW}}$ and $R_{\rm
EM}$ are the rate of gravitational-wave and electromagnetic triggers
respectively, and $T$ is the co-observing time. We take $T=4\,\mathrm{s}$ but
emphasise that this factor cancels out when multiplied by the overlap integral
of the time of coalescence (see Ashton et al., 2018; Sarin & Lasky, 2022, for
details).

As a rule of thumb, the false-alarm rate (i.e., the rate of triggers) in a
single gravitational-wave detector goes down two orders of magnitude for every
one unit of signal-to-noise ratio $\rho$; i.e.,

$\displaystyle{\rm FAR}\propto 10^{-\xi\rho},$ (2)

with $\xi\approx{2}$ (Kelley et al., 2013; Cabourn Davies & Harry, 2022). We
use the above relation, calibrating it to the rate of single detector searches
for gravitational waves from binary black holes (Callister et al., 2017)333See
their Figure 1. We note that this is likely an overestimate for the rate of
gravitational-wave triggers, as both neutron star black hole and binary
neutron star mergers are longer duration signals, and therefore less likely to
be mistaken for instrumental glitches. Furthermore, having two or more
detectors also allows for coherence tests making the above choice a
conservative estimate for the prior odds in that scenario.

In principle, prompt gamma-ray observations also have a similar relationship
to that described above between the rate of triggers and the signal-to-noise
ratio. However, as most of the gamma-ray bursts observed in the visible volume
of gravitational-wave detectors will be electromagnetically bright, we can
continue to use the estimate of $2\,\mathrm{day^{-1}}$ (von Kienlin et al.,
2020). We note that GRB 211211A was also observed by Swift and identification
of the afterglow, kilonova and host-galaxy would increase the significance of
the trigger. However, Fermi triggers more often, implying that our prior odds
calculation is conservative.

Given we know the signal-to-noise ratio of our injected signals, we can
calculate the likelihood that a trigger of that signal-to-noise ratio or
greater is a false alarm in our detector. We then use this number, combined
with $R_{\rm EM}$ and $R_{\rm{GW,EM}}$ in Eq. 1 to determine the prior odds.
To estimate $R_{\rm{GW,EM}}$, we use absolute binary neutron star and neutron
star-black hole merger rates of $10-1700\,\mathrm{Gpc^{3}\,yr^{-1}}$ and
$8-140\,\mathrm{Gpc^{-3}\,yr^{-1}}$ respectively (90% credible intervals
Abbott et al., 2021b)444We use the rate constraints marginalised over the
different mass models from Abbott et al. (2021b), with our median value being
the median value from the binned Gaussian method in that work., and beaming
and jet-launching fractions from Sarin et al. (2022). For example, for the
binary neutron star merger injection into a three-detector network at O3
sensitivity, the network SNR is $7.2$, corresponding to a prior odds $0.01-26$
(90% credible interval). That is, a priori it is between $0.01-26$ times more
likely that a coincident gravitational-wave trigger of SNR=$7.2$ with a gamma-
ray burst trigger is of astrophysical origin and from the same source.

We set a Bayesian odds $\ln\mathcal{O}_{C/R}\geq 10$ to be the threshold for a
confident association, although we emphasise that defining a threshold for
confidence is subjective. A $\ln\mathcal{O}_{C/R}=10$ implies that a common
origin is $\approx 22,000$ times more likely than a random coincidence555We
note the odds as calculated by Ashton et al. (2018) between GRB170817A and
GW170817 is $\ln\mathcal{O}_{C/R}\gtrsim 12$, however this is not directly
comparable to our results as they do not include the SNR dependence in the
gravitational-wave trigger rate when calculating the prior odds.. Our
calculated Bayesian odds for a GRB 211211A-like neutron star binary in O3
sensitivity are summarised in Table. 1.

For the binary neutron star merger injected into the two (three) detector
network at O3 sensitivity, the $\ln\mathcal{O}_{C/R}$ is $7.7^{+3.7}_{-3.9}$
($8.3^{+3.7}_{-3.9}$), indicating that had two LIGO detectors been online at
the time of GRB211211A, we would likely not have confidently associated the
electromagnetic counterpart to the gravitational-wave signal (although this
depends on the choice of threshold and prior odds). With O4 sensitivity, the
binary neutron star $\ln\mathcal{O}_{C/R}=29.1^{+3.7}_{-3.9}$, implying this
would be a confident multi-messenger detection, despite the gravitational-wave
signal being below the conventional threshold. Not only would such a detection
have enabled the plethora of analyses similar to GW170817, but also questions
unique to this event, perhaps unravelling the mystery of gamma-ray burst
extended emission—we discuss this further in Section 4.

By contrast to the binary neutron star mergers, the odds for a neutron star-
black hole merger with the two (three) detector network, is
$\ln\mathcal{O}_{C/R}=33.3^{+3.7}_{-3.9}$ ($33.6^{+3.7}_{-3.9}$) at O3
sensitivity, implying we would confidently associate the electromagnetic
counterpart to the gravitational-wave signal. We note that the neutron star-
black hole gravitational-wave signal would also be below the conventional
threshold for a network detection but would have a SNR$\gtrsim 8$ in the
Livingston detector.

| BNS | NSBH
---|---|---
| $\ln{\mathcal{O}_{C/R}}$ | $\ln{\mathcal{O}_{C/R}}$
HL O3 | $7.7^{+3.7}_{-3.9}$ | $33.3^{+3.7}_{-3.9}$
HLV O3 | $8.3^{+3.7}_{-3.9}$ | $33.6^{+3.7}_{-3.9}$
Table 1: Natural log of the common-to-random odds $\ln{\mathcal{O_{C/R}}}$ for
a GRB 211211A-like binary neutron star merger (left column) and neutron star-
black hole merger (right column) with a two and three gravitational-wave
detector network at O3 sensitivity. The odds for both BNS and NSBH at O4
sensitivity are well above the threshold of detection.

##  3 Multi-messenger observation rates

GRB 211211A may have been a missed opportunity—how do we derive optimal
observing strategies to ensure we do not miss another multimessenger compact
binary merger? To wit, we calculate the expected rate of coincident
gravitational-wave and electromagnetic observations of mergers using the
coincident detection criteria.

We simulate binary neutron star and neutron star-black hole mergers
isotropically distributed in the sky with inclination angles uniformly drawn
from $\theta_{JN}\in\left[0^{\circ},10^{\circ}\right]$, and intrinsic
parameters $m_{\textrm{NS}}=1.4M_{\odot}$, $m_{\textrm{BH}}=4.5M_{\odot}$,
$\chi_{\textrm{BH}}=0.85$, $\chi_{\textrm{NS}}=0.01$,
$R_{\textrm{NS}}=12\,\mathrm{km}$. Here, the subscripts NS and BH denote
neutron star and black hole, respectively, and $M$, $R$, and $\chi$ are the
objects’ mass, radius, and in-plane dimensionless spin. These parameters are
specifically chosen as they are more likely to produce detectable prompt
gamma-ray burst emission and kilonova than, say, mergers with higher
inclination angle or more massive black holes. In reality, only some fraction
of neutron star mergers will be able to launch a jet or disrupt sufficient
matter to produce a detectable kilonova (e.g., Sarin et al., 2022). We expect
this to have a small effect on the rates of detectable emission from binary
neutron star mergers. However, the effect may be significantly larger for
neutron star-black hole mergers, although the size of the effect is largely
unknown given such system’s unknown black hole mass and spin distribution.

We calculate the coincident odds for both binary neutron star and neutron
star-black hole mergers with prompt gamma-ray and host galaxy observations as
a function of distance for O4 and O5 (i.e., A+) projected
sensitivities666Sensitivity curves are taken from
https://dcc.ligo.org/LIGO-T2000012/public (Abbott et al., 2020) with simulated
Gaussian noise. To investigate optimal observing strategies, we inject our
signals into two different detector networks: 1) a network of Livingston and
Virgo, and 2) only Livingston. Note that these are proxy networks; our
quantitative result would be largely unchanged if we, for example, used a
single Virgo or KAGRA detector—the important aspect here is the sensitivity of
the instruments, not their location. We note that at O5 sensitivity, Virgo is
projected to be $\approx 60\%$ as sensitive as the Livingston detector. As in
the previous section, we calculate the prior odds using Eq. 1, with a gamma-
ray trigger rate of $2\,\mathrm{day^{-1}}$ and a gravitational-wave trigger
rate which is proportional to the signal-to-noise ratio.

Figure 2: $\ln\mathrm{BF}_{C/R}$ ($\ln\mathcal{O}_{C/R}$) as a function of
gravitational-wave SNR in the top (bottom) row for the binary neutron star
(left panel) and neutron star-black hole (right panel) injections for a single
Livingston detector operating at O4 (blue crosses) and O5 (green crosses)
sensitivity. The black shaded region indicates $\ln\mathcal{O}_{C/R}<=10$
i.e., the region where one cannot confidently claim a multi-messenger
detection. The faint circles illustrate injections where the $t_{c}$ posterior
is not more informative than the prior, we consider these injections to be
undetectable regardless of the $\ln\mathcal{O}_{C/R}$.

In Fig. 2, we plot the $\ln\mathrm{BF}_{C/R}$ as a function of gravitational-
wave SNR in the top row for the binary neutron star (left panel) and neutron
star-black hole (right panel) injections for a single Livingston detector
operating at O4 (blue crosses) and O5 (green crosses) sensitivity. The faint
circles illustrate injections where the $t_{c}$ posterior $90\%$ credible
interval is larger than $0.2\,\mathrm{s}$ and therefore the gravitational-wave
analysis is not more informative than the prior. We do not consider these
events coincident detections regardless of whether their
$\ln\mathcal{O}_{C/R}\geq 10$. In the bottom row, we show the
$\ln\mathcal{O}_{C/R}$ as a function of gravitational-wave SNR created by
fitting a curve through the $\ln\mathrm{BF}_{C/R}$ shown in the top row and
using Eq. 1 for the prior odds. We note that we fit a curve through
$\ln\mathrm{BF}_{C/R}$ as creating a smooth $\ln\mathcal{O}_{C/R}$ vs. SNR
curve would require many injections to account for the stochasticity of
$\ln\mathrm{BF}_{C/R}$, which is computationally expensive and beyond the
scope for this paper. In the bottom panel we also shade the region below
$\ln\mathcal{O}_{C/R}=10$, i.e., the region where we would not claim a
confident multi-messenger detection.

We find that across both networks, sensitivities and binary neutron star or
neutron star-black hole mergers, the effective SNR where the gravitational-
wave signal can be confidently associated to the gamma-ray counterpart is
$\approx 7$ in a single gravitational-wave detector. This is lower than the
conventional single detector threshold of $\rm{SNR}=8$. That is, even though
the gravitational-wave signal alone is not significant enough to claim a
detection, the gamma-ray counterpart provides the necessary confidence for a
multi-messenger detection.

The above analysis assumes fixed masses, spins, and inclination angles. In
reality, these properties dictate whether a neutron star binary can launch a
jet that can produce gamma-ray emission and whether that jet is beamed at the
observer. Refining this analysis, we take a realistic mass and spin
distribution of neutron star binaries from an isolated binary evolution
simulation (the so-called “no pulsational pair instability supernova model” in
Broekgaarden et al. (2021)) and calculate the rate of joint gravitational-wave
signals and gamma-ray bursts out to a distance of $600\,\mathrm{Mpc}$ for
binary neutron star mergers and $1000\,\mathrm{Mpc}$ for neutron star-black
hole mergers for the same networks and sensitivities as above. We also account
for whether a given binary can launch a jet and whether the gamma-ray burst is
detectable using beaming and jet-launching fractions outlined in Sarin et al.
(2022).

| BNS | BNS | NSBH | NSBH
---|---|---|---|---
| $\mathcal{F}$ | $\mathcal{R}\,\mathrm{[yr^{-1}]}$ | $\mathcal{F}$ | $\mathcal{R}\,\mathrm{[yr^{-1}]}$
L O4 | 0.11 | $0.22^{+8.3}_{-0.22}$ | $0.09$ | $0.08^{+1.7}_{-0.08}$
LV O4 | 0.10 | $0.21^{+7.8}_{-0.2}$ | 0.1 | $0.09^{+1.98}_{-0.09}$
L O5 | 0.34 | $0.71^{+26.8}_{-0.70}$ | $0.35$ | $0.32^{+6.7}_{-0.32}$
LV O5 | 0.37 | $0.78^{+29.1}_{-0.76}$ | 0.34 | $0.31^{+6.5}_{-0.31}$
Table 2: Predicted fraction $\mathcal{F}$ and rate $\mathcal{R}$ of binary
neutron star (BNS) and neutron star-black hole (NSBH) binaries with confident
coincident gravitational-wave and electromagnetic observations. We consider
networks of two detectors consisting of LIGO-Livingston and Virgo (LV) and
just Livingston (L) operating at O4 (i.e., design) and O5 (i.e., A+)
sensitivities. The fraction of events is calculated assuming a luminosity
distance $d_{L}\leq 600\,\mathrm{Mpc}$ for BNS and $d_{L}\leq
1000\,\mathrm{Mpc}$ for NSBH.

Our key results are summarised in Tab. 2. The first column shows the fraction
$\mathcal{F}$ of detected short gamma-ray bursts caused by binary neutron star
mergers at luminosity distance $d_{L}\leq 600\,\mathrm{Mpc}$ that would be
confident ($\ln\mathcal{O}_{C/R}\geq 10$) multi-messenger detections for our
different network configurations. For example, for a single LIGO-Livingston
observatory operating at O4 sensitivity (L O4), approximately 11% of short
gamma-ray bursts from binary neutron stars within $600\,\mathrm{Mpc}$ would be
confident multi-messenger detections. This number increases to 34% using O5
(i.e., A+) sensitivity. Interestingly, this number does not change appreciably
when adding another gravitational-wave detector, a direct consequence of the
odds being predominantly informed by the measurement of the merger time, which
is largely insensitive to the number of detectors in a network.

This fraction translates into a rate of joint gravitational-wave and gamma-ray
burst detections of $0.22^{+8.3}_{-0.22}\,\mathrm{yr^{-1}}$ and
$0.71^{+26.8}_{-0.70}\,\mathrm{yr^{-1}}$ at O4 and O5 sensitivity, out to a
horizon distance of $600\,\mathrm{Mpc}$ with the uncertainties coming from the
uncertainty on the merger rate, beaming, and jet-launching fractions.

We also calculate the fraction and rate for neutron star-black hole mergers of
$0.09$ ($0.35$) and $0.08^{+1.7}_{-0.08}\,\mathrm{yr^{-1}}$
($0.32^{+6.7}_{-0.32}\,\mathrm{yr^{-1}}$) at O4 (O5) sensitivity,
respectively. These rates are much lower, a consequence of the smaller jet-
launching fraction of neutron star-black hole mergers (e.g., Sarin et al.,
2022).

##  4 Conclusion

GRB 211211A may have been caused by a binary neutron star or neutron star-
black hole merger at $\approx 350\,\mathrm{Mpc}$. If the latter, ground-based
gravitational-wave observatories with their most recent (i.e., third-observing
run; O3) sensitivity would have detected this multimessenger event if a single
observatory were operating. This would have been a remarkable outcome: the
gravitational-wave observations would have given us both the masses of the
progenitors and the spin of the black hole. We note that, as with all
measurements with gravitational waves, the chirp mass would have been best
measured, and constraints on the component masses and mass ratio would have
been weak. This information, the kilonova observations, and numerical-
relativity modelling of neutron star-black hole mergers, would have provided
stringent constraints on the neutron star equation of state (e.g., Coughlin et
al., 2019), and informed our understanding of r-process nucleosynthesis (e.g.,
Metzger, 2019; Foucart et al., 2018). Alternatively, if GRB 211211A was
produced in the merger of a binary neutron star, a single gravitational-wave
observatory operating at the proposed O4 sensitivity, will be able to make a
confident multi-messenger detection. This is despite the gravitational-wave
signal itself being below the conventional threshold for detection, with the
gamma-ray burst providing the necessary confidence. Such a discovery would
have shed light on the nature of the central engine in GRB 211211A.

With the imminent sensitivity improvement of gravitational-wave observatories,
binary neutron star mergers with electromagnetic counterparts will be
observable in gravitational waves at $\lesssim 350\,\mathrm{Mpc}$, even with
only a single observatory operating. We show this will be possible with the
proposed O4 sensitivity, due to begin observing in early 2023 (Abbott et al.,
2020). We find that the rate of joint gamma-ray and gravitational wave
detections do not change significantly whether one gravitational-wave detector
remains online or more. This is a direct consequence of the odds
$\mathcal{O}_{C/R}$ being predominantly informed by the measurement of the
time of coalescence. This motivates the question whether one single
gravitational-wave detector should remain online at all times to not miss
opportunities like GRB 211211A.

We advocate that once gravitational-wave detectors reach O5 sensitivity, at
least one detector remain online at all times. This will ensure that at least
$0.71^{+26.8}_{-0.70}$ ($0.32^{+6.7}_{-0.32}$) per year of gamma-ray bursts
produced by binary neutron star (neutron star-black hole mergers) out to a
distance of $600\,\mathrm{Mpc}$ ($1000\,\mathrm{Mpc}$) can be associated to
their often sub-threshold gravitational-wave signal enabling the rich promise
of multi-messenger astronomy, while other detectors in the network perform
maintenance or upgrades. Ignoring uncertainties from the beaming and jet-
launching fraction, this would mean that $\approx 34\%$ ($35\%$) of short
gamma-ray bursts within $600\,\mathrm{Mpc}$ ($1000\,\mathrm{Mpc}$) produced by
a binary neutron star (neutron star-black hole) would yield a confident
gravitational-wave detection with a single gravitational-wave detector. We
emphasize that this calculation assumes that the gamma-ray burst redshift is
measured, which therefore requires gamma-ray burst triggers continue to be
followed up independently and produce a successful identification of the host
galaxy and or optical counterpart.

Even though the gravitational-wave signals may be sub-threshold, the potential
science output of multimessenger observations as described here is immense.
For example:

  * •

Hubble constant measurements. The 90% credible-interval posterior width on the
luminosity distance measured with gravitational waves from a single-detector
binary neutron star merger is $200\,\mathrm{Mpc}$ (assuming conservatively,
that the observation of a gamma-ray burst does not provide any measurement of
the inclination angle). This implies that the sub-threshold gravitational-wave
signals with host-galaxy redshifts can be used for accurate, distance-ladder-
free inferences of the Hubble constant (Schutz, 1986; Abbott et al., 2017a).
For example, assuming GRB 211211A was a binary neutron star merger, a single
gravitational-wave detector operating at O4 sensitivity would have produced a
Hubble constant measurement of
$H_{0}={74.72}_{-11.64}^{+21.20}\,\mathrm{km\,s^{-1}\,Mpc^{-1}}$ without
including any constraint on the viewing angle from the gamma-ray burst. If the
gamma-ray burst measurement of the viewing angle was included, the Hubble
constant measurement would have been
$H_{0}={62.12}_{-3.96}^{+5.09}\,\mathrm{{km\,s^{-1}\,Mpc^{-1}}}$.

  * •

Tests of gravity. The gravitational-wave uncertainty on the time of
coalescence, even in the case of a single-detector observation with matched
filter signal-to-noise ratio $\approx 7$, is $\lesssim 0.1\,\mathrm{s}$.
Comparing this time with the prompt emission will therefore allow constraints
on the speed of gravity, akin to what was achieved with multimessenger
observations of GW170817 (Abbott et al., 2017b)

  * •

Equation of state. While the tidal deformability of neutron star mergers
cannot be accurately measured from such sub-threshold events (the posterior
distribution is similar to the prior in our simulations), knowledge of the
progenitor masses does allow for modelled calculations of the ejecta mass that
can be compared to e.g., the kilonova observations to provide constraints on
the equation of state (e.g., Coughlin et al., 2019; Nicholl et al., 2021).
Electromagnetic observations also provide constraints on the fate of binary
neutron star mergers (e.g., Sarin & Lasky, 2021), which can be combined with
measurements of the progenitor masses to constrain the maximum neutron star
mass (e.g., Margalit & Metzger, 2017). The $90\%$ credible-interval posterior
width of the progenitor chirp mass and mass ratio in our single-detector O4
Livingston observation for a GRB 211211A-like binary neutron star is $\lesssim
0.0023\,\mathrm{M_{\odot}}$ and $\lesssim 0.47$, respectively.

  * •

Gamma-ray burst physics. Comparing the time of coalescence with the prompt
gamma-ray emission also allows for studies into the physics of the the delay
time between the merger, and the launch and propagation of the jet.
Furthermore, given an approximate distance to the source is known from the
host galaxy, an approximate viewing angle can be determined from the
gravitational-wave posterior distribution of the inclination angle. In the
case of a single-detector Livingston observation in O4, this inclination angle
can be determined with an uncertainty of $20^{\circ}$ from the gravitational-
wave data alone, implying the physics of jet opening angles and angular energy
distributions can be studied (e.g., Mandel, 2018; Lamb et al., 2022; Nativi et
al., 2022).

The above laundry list provides a somewhat narrow view of the potential
science for gravitational-wave multimessenger astronomy by only focusing on
neutron star mergers. Coincident gravitational-wave and electromagnetic
observations from nearby (i.e., galactic) supernovae would provide a rich
cornucopia of science, however the event rate of such events is currently
believed to be $\lesssim 1\,\mathrm{/(50\,yr)}$ (Tammann et al., 1994),
implying one should minimise the downtime of gravitational-wave observatories
and continue to improve the sensitivity of the instruments. Of course, there
are other potentially rare sources of gravitational waves that could be
accompanied by electromagnetic observations such as giant magnetar flares,
pulsar glitches, and fast-radio bursts (Macquet et al., 2021; Abbott et al.,
2022a; Yang et al., 2022).

While for the above reasons it is worth considering whether one detector with
sufficient sensitivity should always remain in sky-watch mode, such a proposal
needs to be tempered by commissioning considerations. Furthermore, this
ignores the consideration for following up gravitational-wave observations to
find kilonovae, which rely on multiple detectors for a small sky map.
Therefore, any consideration also needs to account for the difficulty of
finding kilonovae in follow-up to a short gamma-ray burst trigger as opposed
to a gravitational-wave trigger. This is perhaps simple to answer currently
as, due to the size of the visible volume in gravitational waves and beaming
effects, the detectable kilonova rate will be larger than the detectable
gamma-ray burst rate (e.g., Metzger & Berger, 2012; Kelley et al., 2013).
However, as gravitational-wave interferometers near third-generation
sensitivities, this will quickly change, at which point it will also become
important to consider the sensitivity of gamma-ray observatories, which will
become the limiting factor for joint detections. Such a proposal must also
weigh the affect of this sky-watch mode on science from binary black hole
mergers. Although several studies have looked at the aspect of low-latency
detection for binary black holes with a single detector (e.g., Callister et
al., 2017; Cabourn Davies & Harry, 2022), parameter estimation and
astrophysical inference (e.g., how measurements of spin precession,
eccentricity, higher order modes are affected by only having a single
detector, or the implications for population-level analyses) have not been
studied in detail.

## Acknowledgements

We are grateful to Gareth Cabourn Davies, Steve Fairhurst, Ilya Mandel, David
Ottaway, Fiona Panther, Simon Stevenson, Eric Thrane, Ariel Goobar, and
Michael Coughlin for valuable discussions. This work was supported through
Australian Research Council (ARC) Centre of Excellence CE170100004, and ARC
Discovery Project DP220101610. N.S is supported by a Nordita Fellowship,
Nordita is supported in part by NordForsk. This work was performed in part at
Aspen Center for Physics, which is supported by National Science Foundation
grant PHY-1607611. This work was also performed on the OzSTAR national
facility at Swinburne University of Technology. The OzSTAR program receives
funding in part from the Astronomy National Collaborative Research
Infrastructure Strategy (NCRIS) allocation provided by the Australian
Government. This material is based upon work supported by NSF’s LIGO
Laboratory which is a major facility fully funded by the National Science
Foundation.

## Data Availability

No data was used in this manuscript.

## References

  * Aasi et al. (2015) Aasi J., et al., 2015, CQG, 32, 074001
  * Abbott et al. (2017a) Abbott B. P., et al., 2017a, Nature, 551, 85
  * Abbott et al. (2017b) Abbott B. P., et al., 2017b, ApJ, 848, L12
  * Abbott et al. (2020) Abbott B. P., et al., 2020, Living Reviews in Relativity, 23, 3
  * Abbott et al. (2021a) Abbott R., Abbott T. D., Acernese F., Ackley K., et al. 2021a, arXiv e-prints, p. arXiv:2111.03606
  * Abbott et al. (2021b) Abbott R., et al., 2021b, arXiv e-prints, p. arXiv:2111.03634
  * Abbott et al. (2022a) Abbott B. P., et al., 2022a, arXiv e-prints, p. arXiv:2203.12038
  * Abbott et al. (2022b) Abbott R., Abbott T. D., Acernese F., Ackley K., et al. 2022b, ApJ, 928, 186
  * Acernese et al. (2015) Acernese F., et al., 2015, doi:10.1088/0264-9381/32/2/024001, 32, 024001
  * Aghanim et al. (2020) Aghanim N., et al., 2020, A&A, 641, A6
  * Akutsu et al. (2019) Akutsu T., Ando M., Arai K., et al. 2019, Nature Astronomy, 3, 35
  * Ashton et al. (2018) Ashton G., et al., 2018, ApJ, 860, 6
  * Ashton et al. (2019) Ashton G., et al., 2019, ApJS, 241, 27
  * Bartos et al. (2019) Bartos I., Veske D., Keivani A., Márka Z., Countryman S., Blaufuss E., Finley C., Márka S., 2019, Phys. Rev. D, 100, 083017
  * Broekgaarden et al. (2021) Broekgaarden F. S., Berger E., Neijssel C. J., Vigna-Gómez A., et al. 2021, MNRAS, 508, 5028
  * Cabourn Davies & Harry (2022) Cabourn Davies G. S., Harry I. W., 2022, Class. Quant. Grav., 39, 215012
  * Callister et al. (2017) Callister T. A., Kanner J. B., Massinger T. J., Dhurandhar S., Weinstein A. J., 2017, Classical and Quantum Gravity, 34, 155007
  * Cannon et al. (2021) Cannon K., Caudill S., Chan C., Cousins B., et al. 2021, SoftwareX, 14, 100680
  * Clark et al. (2015) Clark J., Evans H., Fairhurst S., Harry I. W., et al. 2015, ApJ, 809, 53
  * Colombo et al. (2022) Colombo A., Salafia O. S., Gabrielli F., Ghirlanda G., et al. 2022, ApJ, 937, 79
  * Coughlin et al. (2019) Coughlin M. W., Dietrich T., Margalit B., Metzger B. D., 2019, MNRAS, 489, L91
  * D’Ai et al. (2021) D’Ai A., Ambrosi E., D’Elia V., Gropp J. D., et al. 2021, GRB Coordinates Network, 31202, 1
  * Fermi GBM Team (2021) Fermi GBM Team 2021, GRB Coordinates Network, 31201, 1
  * Fong et al. (2015) Fong W., Berger E., Margutti R., Zauderer B. A., 2015, ApJ, 815, 102
  * Foucart et al. (2018) Foucart F., Hinderer T., Nissanke S., 2018, Phys. Rev. D, 98, 081501
  * Gao et al. (2022) Gao H., Lei W.-H., Zhu Z.-P., 2022, arXiv e-prints, p. arXiv:2205.05031
  * Gompertz et al. (2022) Gompertz B. P., Ravasio M. E., Nicholl M., Levan A. J., et al. 2022, arXiv e-prints, p. arXiv:2205.05008
  * Kelley et al. (2013) Kelley L. Z., Mandel I., Ramirez-Ruiz E., 2013, Phys. Rev. D, 87, 123004
  * Lamb et al. (2022) Lamb G. P., Nativi L., Rosswog S., Kann D. A., et al. 2022, arXiv e-prints, p. arXiv:2201.09796
  * Macquet et al. (2021) Macquet A., Bizouard M. A., Burns E., Christensen N., et al. 2021, ApJ, 918, 80
  * Magee et al. (2019) Magee R., et al., 2019, ApJ, 878, L17
  * Mandel (2018) Mandel I., 2018, ApJ, 853, L12
  * Margalit & Metzger (2017) Margalit B., Metzger B. D., 2017, ApJ, 850, L19
  * Metzger (2019) Metzger B. D., 2019, Living Reviews in Relativity, 23, 1
  * Metzger & Berger (2012) Metzger B. D., Berger E., 2012, ApJ, 746, 48
  * Nativi et al. (2022) Nativi L., Lamb G. P., Rosswog S., Lundman C., et al. 2022, MNRAS, 509, 903
  * Nicholl et al. (2021) Nicholl M., Margalit B., Schmidt P., Smith G. P., Ridley E. J., Nuttall J., 2021, MNRAS, 505, 3016
  * Nitz et al. (2019) Nitz A. H., Nielsen A. B., Capano C. D., 2019, ApJ, 876, L4
  * Nitz et al. (2020) Nitz A. H., Dent T., Davies G. S., Harry I., 2020, ApJ, 897, 169
  * Patricelli et al. (2022) Patricelli B., Bernardini M. G., Mapelli M., D’Avanzo P., et al. 2022, MNRAS, 513, 4159
  * Petrov et al. (2022) Petrov P., Singer L. P., Coughlin M. W., Kumar V., et al. 2022, ApJ, 924, 54
  * Rastinejad et al. (2022) Rastinejad J. C., et al., 2022, arXiv e-prints, p. arXiv:2204.10864
  * Romero-Shaw et al. (2020) Romero-Shaw I. M., et al., 2020, MNRAS, 499, 3295
  * Sarin & Lasky (2021) Sarin N., Lasky P. D., 2021, General Relativity and Gravitation, 53, 59
  * Sarin & Lasky (2022) Sarin N., Lasky P. D., 2022, Publ. Astron. Soc. Australia, 39, e007
  * Sarin et al. (2022) Sarin N., Lasky P. D., Vivanco F. H., Stevenson S. P., Chattopadhyay D., Smith R., Thrane E., 2022, Phys. Rev. D, 105, 083004
  * Schutz (1986) Schutz B. F., 1986, Nature, 323, 310
  * Speagle (2020) Speagle J. S., 2020, MNRAS, 493, 3132
  * Stachie et al. (2020) Stachie C., et al., 2020, Classical and Quantum Gravity, 37, 175001
  * Suvorov et al. (2022) Suvorov A. G., Kuan H.-J., Kokkotas K. D., 2022, arXiv e-prints, p. arXiv:2205.11112
  * Tammann et al. (1994) Tammann G. A., Loeffler W., Schroeder A., 1994, ApJS, 92, 487
  * Troja et al. (2022) Troja E., Fryer C. L., O’Connor B., Ryan G., et al. 2022, arXiv e-prints, p. arXiv:2209.03363
  * Waxman et al. (2022) Waxman E., Ofek E. O., Kushnir D., 2022, arXiv e-prints, p. arXiv:2206.10710
  * Willke et al. (2002) Willke B., Aufmuth P., Aulbert C., Babak S., et al. 2002, Classical and Quantum Gravity, 19, 1377
  * Xiao et al. (2022) Xiao S., et al., 2022, arXiv e-prints, p. arXiv:2205.02186
  * Yang et al. (2022) Yang J., Zhang B.-B., Ai S., Liu Z.-K., Wang X., Lü H.-J., Zhang B., 2022, arXiv e-prints, p. arXiv:2204.12771
  * von Kienlin et al. (2020) von Kienlin A., Meegan C. A., Paciesas W. S., Bhat P. N., et al. 2020, ApJ, 893, 46


# StyleMaster: Towards Flexible Stylized Image Generation with Diffusion
Models

Chengming Xu1 Kai Hu2 Donghao Luo1 Jiangning Zhang1 Wei Li3
Yanhao Ge3 Chengjie Wang1
1Youtu Lab, Tencent 2Carnegie Mellon University 3VIVO

###### Abstract

Stylized Text-to-Image Generation (STIG) aims to generate images based on text
prompts and style reference images. We in this paper propose a novel framework
dubbed StyleMaster for this task by leveraging pretrained Stable Diffusion
(SD), which addresses previous problems such as insufficient style and
inconsistent semantics. The enhancement lies in two novel modules, namely
multi-source style embedder and dynamic attention adapter. In order to provide
SD with better style embeddings, we propose the multi-source style embedder,
which considers both global and local level visual information along with
textual information, which provide both complementary style-related and
semantic-related knowledge. Additionally, aiming for better balance between
the adapter capacity and semantic control, the proposed dynamic attention
adapter is applied to the diffusion UNet in which adaptation weights are
dynamically calculated based on the style embeddings. Two objective functions
are introduced to optimize the model together with denoising loss, which can
further enhance semantic and style consistency. Extensive experiments
demonstrate the superiority of StyleMaster over existing methods, rendering
images with variable target styles while successfully maintaining the semantic
information from the text prompts.

##  1 Introduction

Among all image generation problems, Stylized Image Generation (SIG), in which
images with target styles are required to be generated, possesses significant
scholarly merit. SIG methods can be smoothly applied to extensive regions such
as artistic creation and movie editing, well facilitating real-life cases.
Recently, with the surge in diffusion-based generative models [9, 17], the
research focus has been transferred from traditional style transfer to the
Stylized Text-to-Image Generation (STIG). In this task, one or several style
reference images are given as condition, and various images are generated
based on the style condition along with additional information such as text
prompts. This task, while being more challenging due to mixed input condition,
is more flexible and applicable, thus being of higher practical value.

Figure 1: Left: With our proposed StyleMaster , the diffusion-based stylized
image generation process can avoid issues such as inconsistent semantics and
insufficient style, leading to better generation quality. Right: Our proposed
method can generated consistent and delicate stylized images with different
styles. For simplicity we only present the generated stylized images here. For
the style reference images please refer to the supplementary material.

As a typical STIG method, StyleAdapter [21] is proposed based upon pretrained
Stable Diffusion (SD). The reference images are processed with a style
embedding module and then injected into the diffusion UNet through extra
cross-attention modules, which guide the model to denoise the features
according to style information, resulting in stylized images. While such a
framework shows promising results for handling specific styles and text
prompts, we find that StyleAdapter still suffers from problems such as (1)
insufficient style, in which the generated images do not fully share the same
style with reference images, for complex styles, and (2) inconsistent
semantics, in which semantic objects are leaked from reference images, thus
affecting the semantic consistency between the generated images and the text
prompts. These problems can be attributed to the design of StyleAdapter.
Specifically, StyleAdapter adopts CLIP [16] as source of style information.
Such patch-level representation focuses more on local patterns, while ignoring
global patterns such as which are also important style descriptors. On the
other hand, the adaptation modules in StyleAdapter are applied to all
attention layers. In this way, each aggregation with text prompts would be
biased by the semantic conuterpart contained in the style embedding, thus
resulting in inconsistent semantic.

To this end, we in this paper propose a novel framework for STIG called
StyleMaster with advanced extraction and injection of style information. Our
method inherits the basic pipeline of StyleAdapter, while adopting a specific
design for style embedding extraction and injection respectively. For style
embedding extraction, we propose a multi-source structure to compensate for
the patch-level CLIP information. Specifically, we not only use the commonly-
used CLIP-based patch embedding to represent image information, but also adopt
global-level VGG descriptor, which is engaged in the attention process via
adaptive scaling and shifting. Apart from that, we make use of the semantic
knowledge of reference images contained in their image captions. This highly
abstract semantic information is eliminated from style embedding through the
proposed negative embedding branch. By actively merging these different latent
spaces, the processed style embedding can thus contain more comprehensive
knowledge about the target styles, while avoiding memorizing the semantic
content. For style embedding injection, since fully adapting UNet features
based on style embeddings can distort the semantic information, a simple
solution would be shrinking adapter attachment only to part of UNet, e.g. its
upsampling layers. However, direct shrinkage can hurt the capacity of
adapters, which in turn worsens the insufficient style problem. To solve this
and enhance the module capacity for thorough stylization, we propose the
dynamic attention adapter, in which dynamic weights are generated from the
style embeddings and then used to adapt both the self-attention and cross-
attention layers in the diffusion UNet.

Moreover, we further enhance the model with objectives other than the
commonly-used noise prediction loss for diffusion models. We introduce a gram
consistency loss, in which we augment the reference images with two sets of
transformations, resulting in one set with original style and another set with
a disrupted style. Then gram matrices of estimated denoised result and these
transformed reference images are calculated as their style-aware statistics.
By calculating a triplet loss among them, the model is encourage to generate
images with more robust and consistent styles when processing different
reference images. Additionally, a semantic disentangle loss is utilized to
alleviate the inconsistent semantics problem by contrasting the style
embeddings against reference text embeddings, while keeping them similar to
reference image embeddings.

To show the effectiveness of our proposed method, we conduct extensive
experiments among various styles containing both one-shot and mult-shot
settings. We show that our method can significantly outperform baseline
methods including StyleAdapter, generating correct styles and avoiding
inconsistent semantics. In summary, the contributions of this work are as
follows:

  * •

We propose a novel method for reference-based stylized image generation named
StyleMaster, boosting Stylized Text-to-Image Generation (STIG) with a multi-
source style embedder and a dynamic attention adapter, greatly solving
problems including insufficient style and inconsistent semantics.

  * •

Two objective functions are introduced to enhance our model, including a gram
stylization loss and a semantic disentangle loss, which encourage the model to
decouple the style related information from reference images with other style
unrelated information.

  * •

Extensive experiment results show that our method consistently work well in
both text-to-image and image-to-image tasks with different reference images,
showing the practical value of such a method.

##  2 Related work

Text-to-image diffusion models. Diffusion models have been proven to be a
powerful family of generative models. DDPM [9] originated to propose the
framework by modeling the mapping between Gaussian distribution and image
distribution with the forward diffusion and inverse denoising process. Based
on that Latent Diffusion Model (LDM) [17] largely improved the practical usage
by leveraging diffusion model to latent space instead of pixel space, which
leads to commonly-known text-to-image diffusion models such as Stable
Diffusion (SD), Midjourney and DALLE-3 [1]. Other works focus on improve the
diffusion model structure. For example, DiT [14], MDT [6] and PIXART-$\alpha$
[3] utilize the transformer instead of UNet structure, which can be better
scaled to larger model size. Blattmann et. al. [2] and Zhang et. al. [25]
leverage ideas of Retrieval Augmented Generation (RAG) to generate images
based on other retrived images which provide extra knowledge. Yang et. al.
[22] propose to leverage the LLMs for planning the text-to-image problems. Our
work is built on pretrained SD models, in which the attention mechanism merges
conditional information from text prompts to images. Different from the
previous works, we focus on designing extra attention adaptation so that the
knowledge contained in the style reference images can be smoothly embedded
into the denoising process, leading to stylized images.

Stylized image generation. Among all conditional image generation tasks,
stylized image generation has long been a highlighted one. Most previous works
focus on style transfer, i.e.transfer the style of a content image given
another style refernce image. For example, Gatys et. al. [7] solved this
problem by optimizing the style-related statistics. Li et. al. [12] equipped
this method with explanability. MicroAST [20] proposed to speed up such
framework by abandoning the complex visual encoder and utilizing a dual-
modulation strategy. Yang et. al. [23] leverages diffusion models, in which
the style-aware guidance is used to generate the wanted style. InST [26]
realized style transfer by inverting the content image to noise and then re-
generate it with the condition control of style images. Apart from these style
transfer methods, StyleAdapter [21] proposes a new framework which can
generate images directly from style reference images and text prompts without
content images. Our work mainly follows StyleAdapter to present a generalized
stylization method. Different from StyleAdapter, we analyze the role of style
reference images and text prompts in the generation process. Based on that, we
propose a novel module to extract more representative style embeddings, which
are then injected into noise space with our proposed dynamic adapter.

##  3 Preliminary: Stable Diffusion

Diffusion models are probabilistic models that learn a data distribution
$p_{\theta}\left(\mathbf{x}_{0}\right)$ by progressively denoising a standard
Gaussian distribution, which can be computed by iteratively adding Gaussian
noise to the clean data $\mathbf{x}_{0}$. Stable Diffusion (SD) extends such a
model to text-to-image based on text prompt $p$. With pre-trained VQ-VAE [19]
containing encoder $\mathcal{E}$ and decoder $\mathcal{D}$, SD allows the
model to focus more on the semantic information of data and improves
efficiency. A diffusion UNet is used to predict the noise, in which attention
mechanism is adopted. Specifically, for the $l$-th layer, self-attention is
first used to interact among spatial features: $z^{l}=Attention(W_{Q}^{l}\cdot
z^{l},W_{K}^{l}\cdot z^{l},W_{V}^{l}\cdot z^{l})$, where $Attention$ denotes
the attention operator, $z^{l}$ denotes latent embeddings of the $l$-th layer,
$W_{Q},W_{K},W_{V}$ denotes the projection layers of self-attention. After
that the cross-attention is utilized to merge condition information such as
text prompt: $\hat{z}^{l}=Attention(\hat{W}_{Q_{t}}^{l}\cdot
z^{l},\hat{W}_{K_{t}}^{l}\cdot z_{text},\hat{W}_{V_{t}}^{l}\cdot z_{text})$,
where $z_{text}$ denotes text prompt embedding,
$\hat{W}_{Q},\hat{W}_{K},\hat{W}_{V}$ denotes the projection layers of cross-
attention. The training objective of SD is as follows:

$\mathcal{L}_{noise}=\mathbb{E}_{\mathcal{E}(x),\epsilon\sim\mathcal{N}(0,1),t}\left[\left\|\epsilon-\epsilon_{\theta}\left(z^{t},t\right)\right\|_{2}^{2}\right],$
(1)

where $t$ is uniformly sampled from $\\{0,...,T\\}$, $z^{t}$ denotes noisy
latent at $t$-th timestep.

##  4 Methodology

Figure 2: Overview of our proposed StyleMaster. Concretely, the multi-source
style embedder (Sec. 4.1) aggregates style patterns contained in each
reference image simultaneously from global and local levels. Style embeddings
are then engaged in the denoising procedure through the proposed dynamic
attention adaptation (Sec. 4.2), which guides both the attentions in diffusion
UNet to properly merge style and semantic information from different sources.
During training augmented input images are used as style reference images,
through which objectives as described in Sec. 4.3 are used to optimize the
model. During inference the style reference images are achieves with manual
assignment instead of using augmentation of a specific image.

We focus on reference-based Stylized Text-to-Image Generation (STIG) in this
paper. Formally, a reference style image set
$\mathcal{I}_{s}=\\{\mathbf{I}_{style}^{i}\\}_{i=1}^{N_{s}}$, where $N_{s}$
denotes the number of reference images, together with text prompt $p$ are
given as condition information. $N_{s}$ can be variable among different trials
to describe different style concepts. The model is required to generate image
$\mathbf{I}$ that shares the same style pattern with $\mathcal{I}_{s}$ and
same semantic meaning with text prompt $p$. To solve this task we present a
novel framework named StyleMaster based on SD, as shown in Fig. 2, which will
be introduced in this section.

Figure 3: Left: multi-source style embedder. Right: dynamic attention adapter.

###  4.1 Multi-source style embedder

Given the successful application of text-to-image based on SD, an ideal
reference-based stylization should necessarily depend on a representative
style embeddings. StyleAdapter [21] adopts CLIP visual encoder [16] to extract
the patch-level features from style reference images, which are then processed
with cross-attentions with learnable style tokens. However, StyleAdapter still
suffers from problems like insufficient style and inconsistent semantics,
i.e.the semantic knowledge in reference images are generated during the
denoising procedure. The former problem can be attributed to that CLIP-based
patch embeddings concentrate more on local patterns. In this way, the style
embeddings fail to hold the global style patterns, leading to insufficient
style. Meanwhile, since CLIP image embeddings are aligned with their captions,
they inevitably contain information about contents in the images. Engaging
such information in style embeddings can lead to inconsistent semantics.

To solve these problems, we propose to leverage knowledge from multiple
sources to complement the style embedding. Specifically, given a style image
set $\mathcal{I}_{s}$, we extract three different kinds of features with
pretrained models: (1) local style-aware features: Following StyleAdapter, we
utilize CLIP to encode each style image $\mathbf{I}_{style}^{i}$ into latent
patch tokens $z_{CLIP}^{i}$, which are then concatenated together along the
token dimension, resulted as $z_{CLIP}$. (2) global style-aware features:
Inspired by previous style transfer methods [7], we adopt the gram matrix to
represent global-level style information. Specifically,
$\mathbf{I}_{style}^{i}$ is processed with VGG-19 [18] to get the relu3_1
feature $z_{vgg}^{i}$, using which the gram matrix can be calculated as
$z_{gram}^{i}={z_{vgg}^{i}}^{T}\cdot z_{vgg}^{i}$. Then gram matrices for
different style images are averaged to get the final representation
$z_{gram}$. (3) semantic-aware features: We adopt CLIP text encoder to extract
the text embedding $z_{cap}^{i}$ of captions of $\mathbf{I}_{style}^{i}$,
which is then concatenated into $z_{cap}$. During training, the captions are
provided in the training set, while during inference, we adopt BLIP [11] to
first annotate the style images.

Generally, each token in $z_{CLIP}$ contains local-level style features, while
$z_{gram}$ describes more abstract style information. On the other hand,
$z_{cap}$ contains the semantic information of $\mathcal{I}_{s}$, which should
be eliminated in the final style embedding to avoid inconsistent semantics. To
properly make use of these embeddings, we propose a novel dual-branch
structure as shown in Fig. 3(a). Concretely, several learnable style tokens
$z_{s}$ are first attached to $z_{CLIP}$, with their replication $z^{N}_{s}$
denoted as negative semantic tokens attached to $z_{cap}$:

$\hat{z}_{CLIP}=z_{CLIP}\|_{t}(z_{s}+\delta_{t}),\quad\hat{z}_{caption}=z_{cap}\|_{t}z^{N}_{s}$
(2)

where $\cdot\|_{t}\cdot$ denotes concatenation along the token dimension,
$\delta_{t}$ is the same time embedding of denoising timestep as used in
diffusion UNet. $\hat{z}_{caption}$ is then individually processed with
several transformer layers to aggregate the information between style tokens
and text embedding. For $\hat{z}_{CLIP}$, we adopt a modified version of
transformer layer. Specifically, $z_{gram}$ is first projected to scaling and
shift coefficients. These coefficients are applied to the self-attention
procedure in the same way as adaLN [15]. Before processing the attention
result with FFN, the style token part in $\hat{z}_{caption}$ is subtracted
from the counterpart in $\hat{z}_{CLIP}$.

Such design enjoys three major merits. First, the module is timestep-aware.
Since the style tokens are merged with time embedding, different denoising
steps can thus model different information. Second, the global information in
$z_{gram}$ can guide the model to better concentrate on style-related
knowledge, thus avoiding those irrelevant but repetitive local patterns shared
among style images. Third, the negative semantic tokens generally contain more
abstract content information rather than style. Consequently, subtracting it
from $\hat{z}_{CLIP}$ can help alleviate the problem of inconsistent
semantics. While some captions may describe the style of images, the model can
learn to maintain these knowledge during training thanks to the training
strategy described in Sec. 4.3. By using the proposed module, we can learn
more representative and generalizable style embeddings which can better
facilitate the stylization process described as follow.

###  4.2 Dynamic attention adaptation

The extracted style embedding $z_{s}$ from $\mathcal{I}_{s}$ as described
above can then be used to adapt the pretrained SD to guide the denoising
process based on style information. Thanks to the design of self-attention and
cross-attention mechanism in diffusion UNet, such adaptation can be simply
instantiated as an extra cross-attention module that is parallel to the
original prompt-based cross-attention, as adopted in StyleAdapter. However, we
empirically find that such a method is suboptimal in a large amount of cases,
leading to severe semantic inconsistency. A straightforward solution is cut
down the number of extra attention modules so that only upsample layers in
diffusion UNet are adapted. In this way, the text prompt can dominate the
cross-attention in half of the UNet, consequently resulting in better
semantics in the generated images. However, this can decrease the capacity of
adapters, leading to less preferable stylization. To this end, we propose to
adopt a dynamic adaptation strategy which is applied to both self-attention
and cross-attention (Fig. 3(b)).

Dynamic self-attention adapter. As discussed in previous works [8], the
projected value tensor $W_{V}^{l}\cdot z^{l}$ in the self-attentions
contributes to the texture of generated images. Therefore we introduce a
dynamic self-attention adapter module based on adaIN. Formally, for the $l$-th
self-attention layer, we project $z_{s}$ with a linear layer and adjust
$W_{V}^{l}\cdot z^{l}$ according to statistics of $z_{s}$:

$\hat{\mathbf{V}}^{l}=\mu(f_{proj-SA}^{l}(z_{s}))+\frac{\sigma(f_{proj-
SA}^{l}(z_{s}))}{\sigma(W_{V}^{l}\cdot z^{l})}*(W_{V}^{l}\cdot
z^{l}-\mu(W_{V}^{l}\cdot z^{l}))$ (3)

where $f_{proj-SA}^{l}$ denotes dynamic projection layer, $\mu,\sigma$ denote
mean and standard deviation. By rescaling $\mathbf{V}^{l}$, the information
contained in $z_{s}$ can be directly embedded into the image feature without
destroying the structure and semantic meaning of the generated image.

Dynamic cross-attention adapter. To adapt the cross-attention layers, we
follow the idea of StyleAdapter to adopt the dual-path cross-attention
mechanism. Basically, for the $l$-th cross-attention layer, besides the
original cross-attention performed between text embedding $z_{text}$ and image
embedding $z^{l}$ as in Sec. 3, an extra style-aware cross-attention is added
as $\tilde{z}^{l}=Attention(\hat{W}_{Q_{t}}^{l}\cdot
z^{l},\hat{W}_{K_{s}}^{l}\cdot z_{s},\hat{W}_{V_{s}}^{l}\cdot z_{s})$ with
additional learnable parameters $\hat{W}_{K_{s}}^{l},\hat{W}_{V_{s}}^{l}$.
Then $\hat{z}^{l}+\lambda\tilde{z}^{l}$ is fed into the following feed-forward
networks, where $\lambda$ is learnable coefficient.

To enhance the capacity, we further propose a dynamic cross-attention adapter.
Specifically, we first project the statistics of $z_{s}$ to a layer specific
latent space:

$z_{s}^{l}=f_{proj-CA}^{l}(\mu(z_{s})\|_{c}\sigma(z_{s}))$ (4)

where $f_{proj-CA}$ denotes a linear projection layer, $\cdot\|_{c}\cdot$
denotes concatenation along the channel dimension. Then two weight generators
instantiated as linear layers are applied to $z_{s}^{l}$, resulted in two
dynamic weights $W_{K_{d}}^{l},W_{V_{d}}^{l}$ with dimension $d*d^{l}$. These
two features are reshaped into linear layer weights and used to transform
$z_{s}$. After that the style-aware cross-attention is modified as

$\tilde{z}^{l}=Attention(\hat{W}_{Q_{t}}^{l}\cdot
z^{l},(\hat{W}_{K_{s}}^{l}+W_{K_{d}}^{l})\cdot
z_{s},(\hat{W}_{V_{s}}^{l}+W_{V_{d}})\cdot z_{s})$ (5)

In this way, the key and value projections are partially dependent on $z_{s}$,
resulting in a more complex transformation of $z_{s}$ and leading to better
capacity. To make this module parameter-efficient, we adopt a grouping
strategy, i.e.channels of $z_{s}$ in each group share the same dynamic weight
to produce $W_{K_{d}}^{l}$ and $W_{V_{d}}^{l}$, hence lighter weight
generators can be used to generate dynamic weights.

###  4.3 Training objectives

To train the model such that it can generate images that are conforms to both
the style information from style reference images and semantic information
from text prompts, we introduce a mixed training objectives including three
terms as follows.

$\mathcal{L}=\mathcal{L}_{noise}+\mathcal{L}_{disen}+\mathcal{L}_{style}$ (6)

where $\mathcal{L}_{noise}$ is the noise prediction loss as in Eq. 1.
$\mathcal{L}_{disen}$ denotes a semantic disentangle loss applied to style
embedding $z_{s}$. To ensure that the proposed style embedding model can get
rid of the semantic information contained in $\mathbf{I}_{style}$ when
producing $z_{s}$, $\mathcal{L}_{disen}$ is designed by enlarging the
similarity between $z_{s}$ and $z_{CLIP}$ while decreasing the similarity
between $z_{s}$ and text embedding $z_{text}$. As discussed in Sec. 4.1, the
text embedding represents more abstract semantic information than the image
embedding. Therefore this loss term can help the model avoid the possibility
of semantic leakage, thus leading to better style embedding. On the other
hand, to enhance the style consistency, we propose to regulate the gram matrix
of $\hat{x}^{0}$, which is the noisy estimation from $z^{t}$ and can be
calculated as

$\hat{z}^{0}=\frac{z^{t}-\sqrt{1-\bar{\alpha}^{t}}\epsilon^{t}}{\sqrt{\bar{\alpha}^{t}}},\quad\hat{x}^{0}=\mathcal{D}(\hat{z}^{0})$
(7)

Specifically, we apply several rigid transformations such as random rotation
and cropping to $\mathcal{I}_{S}$ to get a new image $\mathbf{I}_{pos}$ that
has the same style as $\hat{x}_{0}$. Then elastic transformation and color
jitter are also applied to $\mathcal{I}_{S}$. The resulted $\mathbf{I}_{neg}$,
while sharing similar semantic object to $\mathbf{I}_{inp}$, barely inherits
the style from it. Then the objective is a triplet loss which can be written
as

$\displaystyle\delta_{p}$
$\displaystyle=\sum\left|\mathcal{G}(\phi_{vgg}(\hat{x}_{0}))-\mathcal{G}(\phi_{vgg}(\mathbf{I}_{pos}))\right|$
(8) $\displaystyle\delta_{n}$
$\displaystyle=\sum\left|\mathcal{G}(\phi_{vgg}(\hat{x}_{0}))-\mathcal{G}(\phi_{vgg}(\mathbf{I}_{neg}))\right|$
(9) $\displaystyle\mathcal{L}_{style}$
$\displaystyle=max\\{\delta_{p}-\delta_{n}+0.1,0\\}$ (10)

where $\mathcal{G}$ denotes the gram matrix of features. By optimizing this
loss term, the model is encouraged to learn more detailed style information,
thus leading to better results.

##  5 Experiments

###  5.1 Implementation d etail

Dataset. We follow StyleAdapter to adopt LAION-Aesthetic 6.5+ as the training
set, which contains about 600k images. For each input image during training,
we use its augmented variants as the style reference images. For evaluation we
adopt 50 prompts used in StyleAdapter, and select 20 styles covering color,
texture and global layout. More details are presented in the supplementary
material.

Experiment setting. Our experiments cover both one-shot and multi-shot
settings. To make the evaluation more challenging, the shot number, i.e.number
of reference images, varies from 2 to 5 among different styles in the multi-
shot setting. We use all 20 styles for multi-shot experiments and 10 of them
for one-shot experiments.

Training details. We adopt AdamW as optimizer with 1e-5 learning rate. Our
model is trained for 200,000 iterations on 8 V100s with 8 batch size on each
gpu.

Competitor. We include extensive methods as our competitor. For 1-shot
experiment, MicroAST [20], StyleAlign [8], StyTR2 [4], and StyleAdapter [21]
are adopted. For multi-shot experiment, InST [26], LoRA [10], Textual
Inversion (TI) [5] and StyleAdapter are adopted.

###  5.2 Quantitative results

Table 1: Quantitative results for multi-shot setting. For all metrics, the
larger score denotes the better model.

1-shot Methods | Text Sim $\uparrow$ | StyleSim $\uparrow$
---|---|---
MicroAST [20] | 0.299 | 0.529
StyleAlign [8] | 0.276 | 0.645
StyTR2 [4] | 0.298 | 0.541
StyleAdapter [21] | 0.282 | 0.668
Ours | 0.299 | 0.708

Table 2: Quantitative results for multi-shot setting. For all metrics, the
larger score denotes the better model.

multi-shot Methods | Text Sim $\uparrow$ | StyleSim $\uparrow$
---|---|---
InST [26] | 0.196 | 0.692
LoRA [10] | 0.237 | 0.665
TI [5] | 0.268 | 0.678
StyleAdapter [21] | 0.286 | 0.682
Ours | 0.291 | 0.719

Objective quantitative results. For quantitative evaluation we adopt CLIP to
calculate the style similarity between generated images and target style
reference images, and the semantic similarity between generated images and
target text prompts. The results are presented in Tab. 2 and Tab. 2 for 1-shot
and multi-shot respectively. Specifically, MicroAST and StyTR2 share similar
text similarity to ours one-shot experiments. This is because we use
pretrained SD to generate base images for them, thus the basic semantic
meaning is contained in the image. However the gap in terms of style
similarity is much more marginal. Our method outperforms the best baseline
StyleAdapter by 0.04. The results for multi-shot setting is consistent with
one-shot, in which our method is significantly better then the competitors,
thus showing the superiority of the proposed method.

Table 3: User Study for multi-shot setting. For all metrics, the larger score denotes the better model. 1-shot Methods | Text Sim | StyleSim
---|---|---
MicroAST | 3.18 | 2.84
StyleAlign | 2.97 | 2.83
StyTR2 | 3.85 | 3.25
StyleAdapter | 3.16 | 3.30
Ours | 3.20 | 3.80

Table 4: User Study for multi-shot setting. For all metrics, the larger score denotes the better model. multi-shot Methods | Text Sim | StyleSim
---|---|---
InST | 2.11 | 2.68
LoRA | 2.84 | 3.34
TI | 2.95 | 3.03
StyleAdapter | 3.12 | 3.78
Ours | 3.14 | 3.86

Subjective quantitative results. Apart from the quantitative results in the
main paper, we randomly sample 5 images for each method and each style and
conduct a user study. During the study, the participant are required to score
1-5 for each image by considering both semantic fidelity and style
consistency. Then all scores are averaged. The results are shown in Tab. 4 and
Tab. 4, which are generally consistent with the objective results except that
the text similarity of StyTR2 is extremely high with human scores. We think
this may be attributed to the fact that our method still lacks behind the SD
model used to generated base images for StyTR2 in terms of semantic fidelity.
On the other hand, while StyTR2 receives good semantic fidelity, its style
consistency is poor. Our method, on the contrary, can find a good balance for
the trade off between semantic and style.

###  5.3 Qualitative results

Figure 4: One-shot qualitative comparison. Styles from top to bottom: Cezanne,
flat cartoon, expressionism, ink. Prompts from left to right: A bird in a
word; A boy wearing glasses, he is reading a thick book; A cherry blossom. For
detailed reference images please refer to the appendix. Figure 5: Multi-shot
qualitative comparison. Styles from top to bottom: pencil, watercolor, Monet,
impasto. Prompts from left to right: A robot; A modern house with a pool; A
lake with calm water and reflections. For detailed reference images please
refer to the appendix.

We present several uncurated results for both settings in Fig. 4 and Fig. 5.
For one-shot experiment, since we use pretrained SD to first generate the
content images, the style transfer based methods such as StyTR2 and MicroAST
can generally enjoy reasonable semantic consistency. However, such methods can
only inherit the basic color information from reference images rather than the
detailed style information such as shape, texture and layout, thus making them
less preferable. For example, they fail to present the cubism and the curves
in the second and third row. This is because these methods rely on simple
representation to transfer the style-related knowledge from reference images,
which leads to the problem of under-stylization. StyleAlign, on the other
hand, can stylize the images better. For example the images generated by
StyleAlign in the fourth row share similar style patterns with the reference
image. However, we find that StyleAlign suffers from inconsistent semantics
problem, which leads to inconsistent semantic meaning with the text prompt.
Moreover, images generated by StyleAlign seems messy and disordered, which may
be attributed to the uncontrolled shared self-attention between content and
style images. The performance of StyleAdapter is much better than other
competitors, while it is hard for this method to understand complex style
patterns, leading to undesirable results when it comes to ink painting (fourth
row). Compared with these methods, our method can learn appropriate style
information from reference images, e.g. the scattered color patches in the
first and fourth row, and simultaneously keep the images faithful to the
prompts, thus making the best of both worlds.

The multi-shot setting which is more challenging shows similar results. InST
can hardly replicate the style. LoRA and TI suffer from limited style
information. StyleAdapter, while utilizing a specifically-designed pipeline,
shows a tendency to confuse the given styles with the photographic prior
knowledge from pretrained SD. Such phenomenon is most obvious in the first row
of Fig. 5, where pencil drawings are provided as reference images, but
StyleAdapter generates grayscale photos. Our method, thanks to the proposed
multi-source style embedder which can extract more detailed style information
and the dynamic attention adaptation, can generally generate different kinds
of styles with high image quality and semantic fidelity.

Figure 6: Images generated by different model variants.

###  5.4 Ablation study

To further verify the efficacy of our contributions, we conduct several
ablation studies on multi-shot setting. More quantitative and qualitative
ablation studies are provided in the appendix.

Design of style embedding module. We consider two variants together with the
full model for the style embedding module: not using the VGG gram matrix to
regulate the attention layers (w/o VGGToken), and not engaging the reference
image captions in the extraction (w/o NegEmb). The results are shown in the
left three columns of Fig. 6. The style of images generated by model without
VGGToken is generally less mimic. Meanwhile, the model without NegEmb not only
has worse style (first and second row) but also suffers from wrong semantic
meaning (third row).

Design of attention adapter. We illustrate the role of different parts in the
proposed dynamic attention adapter in the middle three columns of Fig. 6. When
the model adopts attention adapter for all UNet attention layers instead of
only the upsample ones, the images generally have problem of wrong semantic
meaning. For example, the clock is missing in the first row, and the cloth
color is wrong in the third row. Also, it is obvious that when only using the
same cross-attention adapter as in StyleAdapter, the generated images show
inconsistent and undesirable styles, which can be attributed to the limited
capacity to such strategy.

Effectiveness of different objective functions. In the right three columns of
Fig. 6 we inspect the efficacy of two objectives introduced in Sec. 4.3. The
results directly support our claim that the gram consistency loss can enhance
the style in generated images and the semantic disentangle loss can make the
model tell apart semantic and style information from reference images, thus
better handling contents in the text prompts.

##  6 Conclusion

We try to solve Stylized Text-to-Image Generation in this paper. A novel model
is proposed to solve the problems of insufficient style and inconsistent
semantics which are suffered by previous methods such as StyleAdapter. The
improvement mainly comes from the multi-source style embedder, in which
multiple sources and used to achieve comprehensive style embeddings and
eliminate the semantic information from style reference images, and the
dynamic attention adapter, in which style embeddings dynamically interact with
attention layers in diffusion UNet. Extensive experiments have been conducted
to show the efficacy of our proposed method as a powerful stylization method,
which can be widely applied to real-life scenarios.

## References

  * [1] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al.  Improving image generation with better captions.  Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023.
  * [2] Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer.  Retrieval-augmented diffusion models.  Advances in Neural Information Processing Systems, 35:15309–15324, 2022.
  * [3] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al.  Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.  arXiv preprint arXiv:2310.00426, 2023.
  * [4] Yingying Deng, Fan Tang, Weiming Dong, Chongyang Ma, Xingjia Pan, Lei Wang, and Changsheng Xu.  Stytr2: Image style transfer with transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11326–11336, 2022.
  * [5] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [6] Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan.  Masked diffusion transformer is a strong image synthesizer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23164–23173, 2023.
  * [7] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016.
  * [8] Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or.  Style aligned image generation via shared attention.  arXiv preprint arXiv:2312.02133, 2023.
  * [9] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [10] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [11] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International conference on machine learning, pages 12888–12900. PMLR, 2022.
  * [12] Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou.  Demystifying neural style transfer.  arXiv preprint arXiv:1701.01036, 2017.
  * [13] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023.
  * [14] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.
  * [15] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville.  Film: Visual reasoning with a general conditioning layer.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * [16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [17] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [18] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [19] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  Advances in neural information processing systems, 30, 2017.
  * [20] Zhizhong Wang, Lei Zhao, Zhiwen Zuo, Ailin Li, Haibo Chen, Wei Xing, and Dongming Lu.  Microast: Towards super-fast ultra-resolution arbitrary style transfer.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2742–2750, 2023.
  * [21] Zhouxia Wang, Xintao Wang, Liangbin Xie, Zhongang Qi, Ying Shan, Wenping Wang, and Ping Luo.  Styleadapter: A unified stylized image generation model without test-time fine-tuning.
  * [22] Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui.  Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms.  arXiv preprint arXiv:2401.11708, 2024.
  * [23] Serin Yang, Hyunmin Hwang, and Jong Chul Ye.  Zero-shot contrastive loss for text-guided diffusion image style transfer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22873–22882, 2023.
  * [24] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023.
  * [25] Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu.  Remodiffuse: Retrieval-augmented motion diffusion model.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 364–373, 2023.
  * [26] Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng Xu.  Inversion-based style transfer with diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10146–10156, 2023.

##  Appendix A Experiment details

#### Competitors.

We follow the original training protocol to train StyleAdapter, and utilize
500 and 1000 iterations for LoRA and TI respectively to achieve suitable
performance and avoid overfitting. To adapt the style transfer methods to our
setting, we first utilize pretrained SD v1.5 to generate an image according to
the text prompt, then transfer its style using the corresponding methods. For
StyleAlign, we first adopt DDIM-inversion [13] to invert reference images back
to noise, and then attach it to other images to be generated.

#### Style reference images.

To make sure our experiments is extensive enough to show the generalization
ability of our method, we manually design the style reference image set, which
are shown in Fig. 7 and Fig. 8, with the search results provided by Google
with key words sets as the notations as depicted in the captions. Our style
reference images cover different style concepts such as artistic contents,
shapes, colors and textures, which can better support our conclusion that the
proposed AnyArt is capable for various cases.

#### Styles used in figures in main paper.

In order to make some results (Fig. 1 and Fig. 6) in main paper simple and
easier to understand, we omit the style reference images and summarize the
used styles here:

  * •

Fig. 1 from first row to fifth row: pixar, ink, wooden, cubism, Van Goah.

  * •

Fig. 6 first row from left to right: Degas, Cezanne, Cubism.

  * •

Fig. 6 second row from left to right: ink, flat cartoon, pencil.

  * •

Fig. 6 third row from left to right: flat cartoon, impasto, watercolor.

Figure 7: Style reference images used in this paper. Each column denotes a
style, in which the images in the first row are used in one-shot experiments,
and all images are used in multi-shot experiments. Notations for each column
from left to right: Cezanne, crayon, cubism, Degas, expressionism, flat
cartoon, ukiyoe, Van Goah, impasto, ink. Figure 8: Style reference images used
in this paper. Each column denotes a style, in which the images in the first
row are used in one-shot experiments, and all images are used in multi-shot
experiments. Notations for each column from left to right: japan anime, Monet,
pencil, pixar, psychedelic, america anime, watercolor, surreal, wooden, surf.

#### Prompts used in figures in main paper.

All qualitative results in the qualitative ablation study of main paper use
the same prompts as in the quantitative evaluation, concretely as follows,

  * •

Fig. 6 first row from left to right: A girl wearing a red dress, she is
dancing; A little boy with glasses and a watch; A smiling little girl.

  * •

Fig. 6 second row from left to right: A puppy sitting on a sofa; An curly-
haired boy; A boy wearing glasses, he is reading a thick book.

  * •

Fig. 6 third row: A man wearing a black leather jacket and a red tie.

##  Appendix B More discussion

#### Limitations.

We would like to highlight two limitations of our method. First, the multi-
source style embedding extraction model relies on a patch-level transformer.
In this way, it is less efficient for this model to process too many style
reference images. While such a scenario is to some extent unrealistic, since
it is generally sufficient to represent a specific style with less than 10
images, solving this problem can be related to improving vision transformer
structures, which can be taken as future works. Second, the proposed method is
only available for style conditions in the form of images. Other forms such as
texts, videos and 3D data are not considered in this work and can be solved in
the future.

#### Broader impacts.

Our work will not lead to significant negative social impacts. Problems such
as privacy invasion and misinformation can be also attributed to normal image
generative models. Solving such problems would be a large future research
topic.

#### Versatility of our method.

To show that our method is generalizable enough, we further apply our method
to pretrained SDXL, which is an advanced version of SD. The results are shown
in Fig. 9 and Fig. 10. We can find that basically the proposed method can
introduce correct style to pretrained SDXL. Since there is significant gap
between the prior knowledge learned by SDXL and SD1.5, the generated images
also show different patterns. The results show that SDXL can better handle
styles regarding lines and colors, while SD1.5 can provide better global-level
styles. Moreover, SDXL can make better balance between style and general image
aesthetic. The human faces generated by SDXL are more proper, thanks to its
larger capacity.

Figure 9: Quantitative results of our proposed method with SDXL in one-shot
setting. Styles used in each row from up to bottom: Cezanne, flat cartoon,
cubism, pencil. Figure 10: Quantitative results of our proposed method with
SDXL in multi-shot setting. Styles used in each row from up to bottom: america
anime, Cezanne, Expressionism, ukiyoe, pencil, surreal, psychedelic.

#### Effectiveness of frequency domain decomposition.

In the multi-source style embedding extraction module, we utilize the discrete
wavelet transform to first decompose the patch level features of style
reference images into low-frequency and high-frequency features. To see how
this process can help our module, we visualize the attention weights for two
sets of reference images among different denoising time steps in Fig. 11.
Three main phenomenon can be concluded: (1) Style embeddings concentrate more
on low frequency features, which is reasonable since low-frequency features
contain information such as color. (2) The patterns of feature usage are
consistent among two prompts for each style, while being different for
different styles. (3) For high-frequency features, different timesteps
generally focus on different information. These results can sufficiently
support our design of decomposing the image features regarding frequency.

Figure 11: Attention weight visualization between style embedding and
frequency domain features of style reference images. Each column represents
the same text prompt and each row represents the same style, which are listed
in the figure. Zoom in for more details.

#### Quantitative results for ablation study.

We present the quantitative results for all variants used in Tab. 5, which are
generally consistent with the qualitative results in the main paper.
Interestingly, we find that when not using dynamic adaptor, the model has a
very different tendency compared with the full model, with much better text
prompt similarity but much worse style similarity. This is reasonable since
the proposed dynamic adaptor greatly strengthens the impact of style embedding
during both self-attention and cross-attention. In this way, the cross-
attention with text prompts is weakened is disguise. In general, adopting
dynamic adaptor can make a good balance between text prompt fidelity and style
fidelity.

Table 5: Quantitative ablation study for multi-shot setting. For all metrics, the larger score denotes the better model. multi-shot Methods | Text Sim | StyleSim
---|---|---
w/o VGGToken | 0.288 | 0.698
w/o NegEmb | 0.281 | 0.696
w/o OnlyUP | 0.286 | 0.702
w/o dynamic adaptor | 0.299 | 0.643
w/o style loss | 0.290 | 0.705
w/o disen loss | 0.286 | 0.694
Ours | 0.291 | 0.719

#### Image-to-image.

Apart from the basic text-to-image task, we also extend our method to image-
to-image task, in which we follow the commonly used pipeline to first and then
adopt the ControlNet-Canny [24] together with our proposed method to generate
a new image with target style. The results are shown in Fig. 12. One can find
that when given different reference images such as Japan anime, American
anime, Von Goah’s painting and pixar anime, the human face in the base image
can flexibly change according to the style, which illustrates the
effectiveness of our method.

Figure 12: Image to image results generated by our model.

##  Appendix C More quantitative results

Figure 13: More quantitative results of our proposed method in one-shot
setting. Styles used in each two rows from up to bottom: ink, impasto, Monet,
Van Goah. Figure 14: More quantitative results of our proposed method in
multi-shot setting. Styles used in each three rows from up to bottom: Cezanne,
flat cartoon, america cartoon. Figure 15: More quantitative results of our
proposed method in multi-shot setting. Styles used in each three rows from up
to bottom: surreal, psychedelic, surf.

We provided more quantitative results in both one-shot and multi-shot settings
in Fig. 13, Fig. 14 and Fig. 15. The images generated with each group of
reference images share consistent style, while correctly showing the target
objects.