Demonstration of stellar intensity interferometry with the four VERITAS
telescopes

A. U. Abeysekara1, W. Benbow2, A. Brill3, J. H. Buckley4, J. L. Christiansen5,
A. J. Chromey6, M. K. Daniel2, J. Davis1, A. Falcone7, Q. Feng3, J. P.
Finley8, L. Fortson9, A. Furniss10, A. Gent11, C. Giuri12, O. Gueta12, D.
Hanna13, T. Hassan12, O. Hervet14, J. Holder15, G. Hughes2, T. B. Humensky3,
P. Kaaret16, M. Kertzman17, D. Kieda1,∗, F. Krennrich6, S. Kumar13, T.
LeBohec1, T. T.Y. Lin13, M. Lundy13, G. Maier12, N. Matthews1,∗, P.
Moriarty18, R. Mukherjee19 M. Nievas-Rosillo12, S. O’Brien13, R. A. Ong20, A.
N. Otte11, K. Pfrang12, M. Pohl21,12, R. R. Prado12, E. Pueschel12, J.
Quinn22, K. Ragan13, P. T. Reynolds23, D. Ribeiro3, G. T. Richards15, E.
Roache2, J. L. Ryan20, M. Santander24, G. H. Sembroski8, S. P. Wakely25, A.
Weinstein6, P. Wilcox9, D. A. Williams14, T. J Williamson15
††∗ email<EMAIL_ADDRESS><EMAIL_ADDRESS>
1Department of Physics and Astronomy, University of Utah, Salt Lake City, UT
84112, USA

2Center for Astrophysics $|$ Harvard & Smithsonian, Cambridge, MA 02138, USA

3Physics Department, Columbia University, New York, NY 10027, USA

4Department of Physics, Washington University, St. Louis, MO 63130, USA

5Physics Department, California Polytechnic State University, San Luis Obispo,
CA 94307, USA

6Department of Physics and Astronomy, Iowa State University, Ames, IA 50011,
USA

7Department of Astronomy and Astrophysics, 525 Davey Lab, Pennsylvania State
University, University Park, PA 16802, USA

8Department of Physics and Astronomy, Purdue University, West Lafayette, IN
47907, USA

9School of Physics and Astronomy, University of Minnesota, Minneapolis, MN
55455, USA

10Department of Physics, California State University - East Bay, Hayward, CA
94542, USA

11School of Physics and Center for Relativistic Astrophysics, Georgia
Institute of Technology, 837 State Street NW, Atlanta, GA 30332-0430

12DESY, Platanenallee 6, 15738 Zeuthen, Germany

13Physics Department, McGill University, Montreal, QC H3A 2T8, Canada

14Santa Cruz Institute for Particle Physics and Department of Physics,
University of California, Santa Cruz, CA 95064, USA

15Department of Physics and Astronomy and the Bartol Research Institute,
University of Delaware, Newark, DE 19716, USA

16Department of Physics and Astronomy, University of Iowa, Van Allen Hall,
Iowa City, IA 52242, USA

17Department of Physics and Astronomy, DePauw University, Greencastle, IN
46135-0037, USA

18School of Physics, National University of Ireland Galway, University Road,
Galway, Ireland

19Department of Physics and Astronomy, Barnard College, Columbia University,
NY 10027, USA

20Department of Physics and Astronomy, University of California, Los Angeles,
CA 90095, USA

21Institute of Physics and Astronomy, University of Potsdam, 14476 Potsdam-
Golm, Germany

22School of Physics, University College Dublin, Belfield, Dublin 4, Ireland

23Department of Physical Sciences, Cork Institute of Technology, Bishopstown,
Cork, Ireland

24Department of Physics and Astronomy, University of Alabama, Tuscaloosa, AL
35487, USA

25Enrico Fermi Institute, University of Chicago, Chicago, IL 60637, USA

High angular resolution observations at optical wavelengths provide valuable
insights in stellar astrophysics [1, 2], directly measuring fundamental
stellar parameters [3, 4], and probing stellar atmospheres, circumstellar
disks [5], elongation of rapidly rotating stars [6], and pulsations of Cepheid
variable stars [7]. The angular size of most stars are of order one milli-
arcsecond or less, and to spatially resolve stellar disks and features at this
scale requires an optical interferometer using an array of telescopes with
baselines on the order of hundreds of meters. We report on the successful
implementation of a stellar intensity interferometry system developed for the
four VERITAS imaging atmospheric-Cherenkov telescopes. The system was used to
measure the angular diameter of the two sub-mas stars $\beta$ Canis Majoris
and $\epsilon$ Orionis with a precision better than 5%. The system utilizes an
off-line approach where starlight intensity fluctuations recorded at each
telescope are correlated post-observation. The technique can be readily scaled
onto tens to hundreds of telescopes, providing a capability that has proven
technically challenging to current generation optical amplitude interferometry
observatories. This work demonstrates the feasibility of performing
astrophysical measurements with imaging atmospheric-Cherenkov telescope arrays
as intensity interferometers and the promise for integrating an intensity
interferometry system within future observatories such as the Cherenkov
Telescope Array.

High-angular resolution optical astronomy is often performed through optical
amplitude interferometry (OAI), which measures the source spatial coherence
function by observing the fringe visibility of interference patterns produced
when light collected at separated telescopes is superimposed [8]. One other
approach is stellar intensity interferometry (SII) that instead uses
correlations of star-light intensity fluctuations recorded with independent
detectors on separated telescopes to measure the spatial coherence [9]. SII
was developed in the late 1950s [10] and resulted in the Narrabri Stellar
Intensity Interferometer (NSII) that was used for observations from 1963 to
1974, providing the first definitive catalog of 32 stellar angular diameters
[11]. Observations with the NSII were limited to only very bright stars with
visual magnitudes less than 2.5. The sensitivity of an SII telescope is
linearly proportional to the telescope area and detector efficiency, and
inversely proportional to the square root of the time-resolution as shown in
the Methods section (see Equation 6). The capabilities of the NSII were
restricted by these factors and concurrent technical advancements in OAI
provided substantial gains in the achievable limiting magnitude using much
smaller telescopes, forestalling further developments in astronomical SII
efforts.

SII has re-emerged as a viable technique for high-angular resolution astronomy
primarily due to the potential of outfitting current and future large diameter
($>10\,$m) telescope arrays with SII capabilities [12, 13]. A suitable SII
instrument requires recording the star-light intensity with nanosecond level
time-resolution and then correlating the intensities between telescope pairs.
The optical path length only needs control with a precision no better than a
few centimeters, determined by the light travel distance over a duration equal
to the detector time resolution. The insensitivity to optical imperfections
allows for SII instrumentation to be installed onto imaging atmospheric-
Cherenkov telescope (IACT) arrays constructed for gamma-ray astronomy. With
mirror diameters typically exceeding 10 m, IACTs are among the largest
astronomical light collectors. IACTs employ fast (f $\sim$ 1.0) optics with
segmented mirrors to reduce cost, and so the large light collection area is
achieved at the expense of imaging resolution, but they remain capable for
performing rapid optical photometry [14] and SII measurements of stars that
are several magnitudes fainter than those accessible to the NSII [15]. There
are several practical arguments for a modern SII observatory that uniquely
complement the current capabilities of OAI. The tolerance to path length
fluctuations allows for observations at all optical wavelengths, even in the U
and B photometric bands that are generally inaccessible to OAI observatories
on account of the required mechanical precision at a fraction of a wavelength.
Furthermore, the baselines between telescopes can be made arbitrarily large in
order to probe unprecedented angular scales as small as tens of
microarcseconds with kilometer length baselines. A significant advantage of
SII is that the technique can be scaled up to an arbitrary number of
telescopes since only digital electronic connections are required [16],
enabling an optical intensity interferometry observatory that operates in a
comparable way to radio interferometry. These realizations have led to several
recent experimental efforts towards a modern intensity interferometer [17, 18,
19, 20], successful SII observations of coherent intensity fluctuations using
two telescopes [21, 22], and distance calibration to the luminous blue
variable P Cyg [23]. We improve upon these observations by directly measuring
angular stellar diameters through fits to the squared visibility-baseline
dependence with an SII system extended to four telescopes to provide six
simultaneous baselines.

Figure 1: The VERITAS array. The left image shows a photograph of VERITAS
located at the Fred Lawrence Whipple Observatory located in Amado, Az. The
array consists of four 12 m diameter telescopes, T1 (front-center), T2
(leftmost), T3 (rightmost), T4 (back-center). The right plot shows a top-down
view of the array with each of the radial telescope separations.

We report here on observations of the two bright ($m_{v}<2.1$), hot (T
$>22,000$ K) B stars $\epsilon$ Ori (Alnilam) and $\beta$ CMa (Mirzam) that
were conducted using an SII system (see Methods) installed onto the Very
Energetic Radiation Imaging Telescope Array System (VERITAS) IACTs shown in
Figure 1 [25]. A total of 4.25 and 5.5 hours of data were taken of $\epsilon$
Ori and $\beta$ CMa, respectively, spanning the nights of December 12-14,
2019, local time. The interferometric (u,v)-plane coverage [24] for both
sources are shown in extended data Figure 3. The observations took place
within a few days from the full moon when VERITAS does not operate as a gamma-
ray instrument as the night sky background light overwhelms the faint
Cherenkov signal. A custom camera is mounted near the focal plane of the
VERITAS telescopes directly in front of the gamma-ray camera to enable SII
observations. The starlight is passed through an interferometric filter with
an effective center wavelength of $\lambda=416\,$nm and bandpass of
$\Delta\lambda=13\,$nm that were chosen to match the peak reflectivity and
quantum efficiency of the mirrors and detector. The filtered starlight is then
detected by a photomultiplier tube. The resulting signal is continuously
digitized and streamed to disk at a rate of 250 MS/s at each telescope.
Correlations between the intensities recorded at each telescope are performed
off-line using a field-programmable gate array. We then analyze the correlated
data by correcting for instrumental and geometrical time delays, background
light effects, and applying noise cuts, to obtain the squared first-order
coherence function $|g^{(1)}(\tau,\textbf{r})|^{2}$ that is proportional to
the squared visibility measured in OAI. It is a function of the path delay
corrected time lag $\tau$ and projected baseline r that is defined as the
separation between the telescopes as viewed from the star. As the star tracks
through the sky, a given telescope pair will span a range of projected
baselines. The amplitude of $|g^{(1)}(\textbf{r})|^{2}$ at $\tau=0$ measures
the spatial coherence that is dependent on the source angular brightness
distribution and projected baseline, reaching a minimum at the baseline $r\sim
1.22\lambda/\theta$ for an observation wavelength $\lambda$ and stellar
angular diameter $\theta$. The angular diameter can then be determined by
fitting the $|g^{(1)}(\textbf{r})|^{2}$ measurements to an appropriate
visibility model for a given source angular brightness distribution (see the
Analysis section for more details). Stars that have a larger angular diameter
will show a more rapid decline in $|g^{(1)}(\textbf{r})|^{2}$ in comparison to
stars with smaller angular diameters.

Figure 2: Temporal and spatial coherence measurements. The top two panels show
the averaged $|g^{(1)}(\tau)|^{2}$ correlation measurements over the full live
time between different pairs of telescopes for $\beta\,$CMa (left) and
$\epsilon$ Ori (right). Each respective telescope pair measurement is color-
coded matching the combinations shown in Figure 1. The uncertainties are given
by the standard deviation of the mean normalized correlation and are estimated
independently for each time-lag. The correlations are ordered by increasing
average baseline from the top, corresponding to decreasing spatial coherence.
The dashed lines show Gaussian fits to the data. The amplitude and
corresponding fit uncertainty of the $|g^{(1)}(\tau)|^{2}$ peak over shorter
time intervals are obtained as a function of the baseline and shown by the
individual points in the bottom panels. The uncertainty is determined by the
square root of the covariance matrix resulting from the fit. These
measurements are fit to a function that approximates the star as a uniform
disk (see Eq. 3) and includes free parameters for the overall normalization
and angular diameter. The shaded area shows the 68 % confidence intervals
determined through the uncertainty in the fit parameters.

Figure 2 presents the temporal (top) and spatial (bottom) correlation
measurements from the observations of $\beta\,$CMa (left) and $\epsilon$ Ori
(right). The top two panels show integrated $|g^{(1)}(\tau)|^{2}$ correlations
for various pairs of the telescopes as a function of $\tau$. The significant
peak at $\tau=0$ is the signal associated with the spatial coherence of the
star. The width of the $|g^{(1)}(\tau)|^{2}$ peak is determined by the
correlation between the combined telescope optics and detector time response
at each telescope. The dashed lines show Gaussian fits to the data. The
Gaussian fit width, representative of the temporal resolution time of the
system, is fixed to 4 ns, as explained in Methods, and the amplitude is left
as a free parameter. The lower panels of Figure 2 show values of
$|g_{\ast}^{(1)}(\textbf{r})|^{2}$ representing the correlation resulting only
from star-light. These values are obtained by scaling the
$|g^{(1)}(\tau,\textbf{r})|^{2}$ amplitude fit by a factor that accounts for
the effect of night sky background light and detector dark current. Each set
of points are shown as a function of the projected baseline and correspond to
an average of 17 minutes of data live-time per measurement. Using uniform-disk
and limb-darkened models for the star angular brightness distribution, we
perform a fit to the data with the stellar diameter and the zero-baseline
correlation as free parameters. The shaded region shows the 68 % confidence
regions obtained from the uniform-disk fit. The value of the zero-baseline
correlation should be on the order of the ratio of the electronic to the
optical bandwidth, each set by the instrumental system. In practice, the
measured correlation can be affected by other factors including the telescope
mirror extent, and spectral absorption or emission lines. We find that these
additional effects are negligible in comparison to our measurement
uncertainty, and thus the zero-baseline value should be the same for both
stars. The zero-baseline normalization fit values are measured to be
$N_{0}=(1.23\pm 0.05)\times 10^{-6}$ for $\beta$ CMa, and $N_{0}=(1.26\pm
0.06)\times 10^{-6}$ for $\epsilon$ Ori, consistent within fitted errors thus
demonstrating the reliability of the system to systematic effects in
observations of different stars.

For $\beta$ CMa, we find a uniform disk diameter of $\theta_{UD}=0.523\pm
0.017\,$mas in agreement with the original NSII measurements of
$\theta_{UD}=0.50\pm 0.03$ mas [11]. In the case of $\epsilon$ Ori, we obtain
$\theta_{UD}=0.631\pm 0.017$ mas, also in agreement with the NSII measurements
($\theta_{UD}=0.67\pm 0.04$ mas). Using a limb-darkened model, given by
Equation 4, we find limb-darkened diameters of $\theta_{LD}=0.542\pm 0.018$
for $\beta$ CMa and $\theta_{LD}=0.660\pm 0.018$ for $\epsilon$ Ori. We note
that the VERITAS stellar intensity interferometer (VSII) gave more precise
angular diameter measurements in comparison to the NSII, with less than a
tenth of the observation time. The improvement is largely due to the greater
light collection areas of the VERITAS mirrors, as well as the increased
instantaneous baseline coverage offered by using multiple telescopes. To our
knowledge, the angular diameters of these stars have not been measured since
the time of the NSII. The current work thus provides confirmation of the NSII
measurements of the stellar angular diameters to better than 5% uncertainty, a
required capability for many science topics described later. Future
observations can further reduce the uncertainty, as the root mean square
fluctuations in the correlation should reduce inversely with the square root
of the observation time. Here, we measure the uncertainty of the
$|g^{(1)}(\tau)|^{2}$ fluctuations at a level of $2.0\times 10^{-8}$ for
several telescope pairs using the entire data set for $\epsilon$ Ori. These
values correspond to an uncertainty in the squared visibility of 1.6 % for the
measured zero-baseline correlation of $N_{0}=1.26\times 10^{-6}$.
Extrapolating these results to fainter stars demonstrates the current
capability to measure squared visibilities with a precision of 4.0 % for stars
of magnitude 2.5 and 10 % for stars with magnitude 3.5 for equivalent
observation time. Direct comparison of these results with the NSII [11] shows
an improvement in the sensitivity by a factor of 6. Future improvements in the
duty cycle and collimation of light through the narrowband filter offer
expected gains in the sensitivity by a factor of 2 to 3, thus offering the
potential for optical intensity interferometry measurements for stars with
$m_{B}\sim 5$.

These two stellar angular diameter measurements with VSII demonstrate the
feasibility of performing optical interferometry with IACT arrays. The
approach developed here using high speed streaming with off-line correlations
demonstrates the technical capability to perform optical intensity
interferometry with tens to hundreds of telescopes, a capability that remains
challenging for modern OAI observatories. We also show that SII observations
increase the scientific output and scope of IACT observatories during bright
moonlight conditions when gamma-ray observations are otherwise limited. The
future Cherenkov Telescope Array (CTA) will employ up to 99 telescopes with up
to kilometer baselines in the Southern hemisphere, and 19 telescopes in the
Northern hemisphere with up to several hundred-meter baselines [26], allowing
for unprecedented angular resolution capabilities approaching the tens of
micro-arcsecond scale. The large number of telescopes would provide hundreds
and possibly thousands of simultaneous baselines. Capability studies of SII on
a CTA-like observatory demonstrate the ability to observe stellar targets
brighter than a limiting visual magnitude of $m_{V}<$ 6 or 7 [27]. Science
opportunities with a future CTA-SII observatory include surveying the angular
diameter of stars larger than $\sim$ 0.05 mas at short visible wavelengths,
measuring the orbital and stellar parameters of interacting binaries [28],
characterizing the effects of gravity darkening and rotational deformation of
rapidly rotating stars [29], and imaging of dark or hot star spots [27]. The
VSII observations presented here demonstrate the core requirements for such an
observatory, thus providing a technological pathway in addition to performing
SII observations with unprecedented sensitivity. Our results demonstrate the
capability to attain squared visibilities at the level of a few percent and
therefore can directly complement ongoing science campaigns pursued by current
OAI observatories [30].

## Correspondence

Correspondence and requests for materials should be addressed to N. Matthews
<EMAIL_ADDRESS>and D. Kieda (dave.kieda@physics.utah.edu).

## Acknowledgements

This research is supported by grants from the U.S. Department of Energy Office
of Science, the U.S. National Science Foundation and the Smithsonian
Institution, by NSERC in Canada, and by the Helmholtz Association in Germany.
We acknowledge the excellent work of the technical support staff at the Fred
Lawrence Whipple Observatory and at the collaborating institutions for their
maintenance of VERITAS and assistance with integrating the SII system. The
authors gratefully acknowledge support from NSF Grants No. AST 1806262, PHY
0960242, and the University of Utah for the fabrication and commissioning of
the VERITAS-SII instrumentation.

This work is dedicated in memory of Paul Nuñez.

## Author Contributions

All authors contributed to the operation of the VERITAS telescopes and the
internal review of the manuscript. N.M. and D.K. developed the SII hardware
used in these observations with assistance from M.D. and G.H. in the
integration with VERITAS; N.M. and D.K. wrote the data acquisition software,
N.M. developed the FPGA correlator software; N.M. proposed the observations;
N.M. and T.H. reduced and analyzed the data; N.M., D.K., T. LeBohec, T.Lin.,
D.R., Q.F., J.D., R.Y.S., U.A., A.F. took the observations; N.M. wrote the
paper with guidance from T. LeBohec., M.D., D.K., T.H., M.P., and J.R.; T.
LeBohec and D.K. provided key insights throughout the entire development of
this work.

## Competing Interests

The authors claim no competing financial interests

## References

  * [1] Monnier, J. Optical interferometry in astronomy. _Rep. on Progress in Phys._. 66, 789-857 (2003)
  * [2] Ridgway, S. _et al._ Revitalizing the Optical/Infrared Interferometry Community in the U.S. _Bull. of the American Astron. Soc._ , 51, 157 (2019)
  * [3] Boyajian, T. S. et. al. Stellar Diameters and Temperatures. III. Main-sequence A, F, G, and K Stars: Additional High-precision Measurements and Empirical Relations. _Astrophys. J._ , 771, 40 (2013)
  * [4] Casagrande, L. et. al. Towards stellar effective temperatures and diameters at 1 per cent accuracy for future surveys. _Mon. Not. R. Astron. Soc._ , 439, 2060-2073 (2014)
  * [5] Kraus, S. et al. Gas Distribution, Kinematics, and Excitation Structure in the Disks around the Classical Be Stars $\beta$ Canis Minoris and $\zeta$ Tauri. _Astrophys. J._ , 744, 19 (2012)
  * [6] van Belle, G. T. Interferometric observations of rapidly rotating stars. _The Astron. and Astrophys. Rev._ , 20, 51 (2012)
  * [7] Kervella, P. et al. Observational calibration of the projection factor of Cepheids: III. The long-period Galactic Cepheid RS Puppis. _Astron. Astrophys._ , 600, A127 (2017)
  * [8] Labeyrie, A., Lipson, S. G., and Nisenson, P. _An Introduction to Optical Stellar Interferometry_. (Cambridge Univ. Press, 2006)
  * [9] Hanbury Brown, R., and Twiss, R. Q. Interferometry of the intensity fluctuations in light. Part I. Basic theory: the correlation between photons in coherent beams of radiation. _Proc. R. Soc. Series A_ 242, 300-324 (1957)
  * [10] Hanbury Brown, R. _The Intensity Interferometer. its applications to astronomy_ (Taylor & Francis, 1974)
  * [11] Hanbury Brown, R., Davis, J. and Allen, L. R. The Angular Diameters of 32 Stars. _Mon. Not. R. Astron. Soc._ , 167, 121 (1974)
  * [12] LeBohec, S., and Holder J. Optical Intensity Interferometry with Atmospheric Cerenkov Telescope Arrays. _Astrophys. J._ 649, 399 (2006)
  * [13] Pilyavsky, G., et al. Single-Photon Intensity Interferometry (SPIIFy): utilizing available telescopes. _Mon. Not. R. Astron. Soc._ 467, 3048-3055 (2017)
  * [14] Lacki, B. C. Cherenkov telescopes as optical telescopes for bright sources: today’s specialized 30-m telescopes? _Mon. Not. R. Astron. Soc._ 416, 3075-3082 (2011)
  * [15] Rou, J., Nuñez, P. D., Kieda, D., and LeBohec, S. Monte Carlo simulation of stellar intensity interferometry. _Mon. Not. R. Astron. Soc._ 430, 3187-3195 (2013)
  * [16] Dravins, D., Lagadec, T., and Nuñez P. Optical aperture synthesis with electronically connected telescopes. _Nat. Comm._ 6, 6852 (2015)
  * [17] Weiss, S. A., Rupert, J. D., and Horch, E. P. Stellar photon correlation detection with the Southern Connecticut stellar interferometer in _Optical and Infrared Interferometry and Imaging VI_ , 107010X (Proc. SPIE 10701, 2018)
  * [18] Zampieri, L. et al. Intensity interferometry with Aqueye+ and Iqueye in Asiago in _Optical and Infrared Interferometry and Imaging V_ , 99070N (Proc. SPIE 9907, 2016)
  * [19] Tan, P. K., Chan, A. H., and Kurtsiefer, C. Optical intensity interferometry through atmospheric turbulence. _Mon. Not. R. Astron. Soc._ , 457, 4291-4295 (2016)
  * [20] Zmija A. et al. LED as laboratory test source for astronomical intensity interferometry. _Opt. Express_ 28, 5248-5256 (2020)
  * [21] Guerin, W. et al. Spatial intensity interferometry on three bright stars _Mon. Not. R. Astron. Soc._ 480, 245-250 (2018)
  * [22] Acciari V. A. et. al. Optical intensity interferometry observations using the MAGIC Imaging Atmospheric Cherenkov Telescopes. _Mon. Not. R. Astron. Soc._ 491, 1540-1547 (2019)
  * [23] Rivet, J. P. et. al. Intensity interferometry of P Cygni in the H$\alpha$ emission line: towards distance calibration of LBV supergiant stars. _Mon. Not. R. Astron. Soc._ 494, 218-227 (2020)
  * [24] Ségransan, D. Observability and UV coverage. _New Astron. Rev._ , 51, 597 - 603 (2007)
  * [25] Kieda D. et al. Augmentation of VERITAS Telescopes for Stellar Intensity Interferometry in _36th International Cosmic Ray Conference_ , 358 (PoS, 2019)
  * [26] Acharyya, A. et. al. Monte Carlo studies for the optimisation of the Cherenkov Telescope Array layout. _Astropart. Phys._ , 111, 35-53 (2019)
  * [27] Nuñez, P. D., Holmes, R., Kieda, D., Rou, J., and LeBohec, S. Imaging submilliarcsecond stellar features with intensity interferometry using air Cherenkov telescope arrays. _Mon. Not. R. Astron. Soc._ , 424, 1006-1011 (2012)
  * [28] Dravins, D., LeBohec S., Jensen, H., and Nuñez, P. D. Optical intensity interferometry with the Cherenkov Telescope Array. _Astropart. Phys._ , 43, 331-347 (2013)
  * [29] Nuñez, P. D. and Domiciano de Souza, A. Capabilities of future intensity interferometers for observing fast-rotating stars: imaging with two- and three-telescope correlations. _Mon. Not. R. Astron. Soc._ , 453, 1999-2005 (2015)
  * [30] Stee et. al. Science cases for a visible interferometer. Preprint at https://arxiv.org/abs/1703.02395 (2017)

## Methods

### Instrumentation

The VERITAS observatory at the Fred Lawrence Whipple Observatory, located in
Amado, AZ, consists of an array of four 12 meter diameter telescopes of the
Davies-Cotton design. The primary mirrors of each telescope consist of 345
hexagonal mirror facets arranged in f/1 optics. SII hardware is mounted onto
an aluminum plate that is installed in front of the VERITAS gamma-ray camera.
Initial tests of the hardware were performed in the laboratory [31] and with
on-sky tests at the StarBase-Utah observatory [32]. The equipment was then
scaled for use on the VERITAS telescopes with successful tests of correlated
intensity fluctuations with two telescopes [33]. On the plate, a 45∘ mirror
redirects the light from the primary onto an optical diaphragm with a diameter
corresponding to approximately 0.1∘ on the sky. The light then passes through
an interferometric filter (SEMROCK 420/5) with a vendor-specified center
wavelength of 420 nm and bandpass of 5 nm. Since the light is not collimated
onto the filter the effective bandpass is broadened due to the large angle of
incident light of up to 26.6$\,{}^{\circ}$. Calculations established the
resulting bandpass to be centered on a wavelength of 416 nm with an effective
width of 13 nm. This effect reduces the overall spectral density throughput,
and correspondingly, the signal-to-noise by approximately a factor of 2. The
light is then detected by a Hamamatsu R10650 photomultiplier tube (PMT) with a
quantum efficiency of $\sim\,$30 % at the observing wavelength [34]. The gain
of the PMT is controlled via battery-powered high voltage. The exact high
voltage delivered to the PMT is set by the duty cycle of a pulse-width
modulator that is connected to the high voltage supply via an optical fiber
[35]. The output current of the PMT is fed into a low-noise trans-impedance
FEMTO HCA-200M-20K-C 200 MHz pre-amplifier. The voltage output of the pre-
amplifier drives a long triaxial cable that is connected to the data
acquisition system. A National Instruments (NI) NI-5761 DC-coupled analog-to-
digital (ADC) converter continuously digitizes the amplified PMT signal at 250
MS/s with 14-bit resolution per sample over a peak to peak voltage range of
1.23 V. Digitized values are passed to a NI PXIe-7965R module which hosts a
Virtex 5 SX95T field-programmable gate array (FPGA). The FPGA is programmed to
downcast the sampled value to an 8-bit integer, and push the data through a
first-in-first-out buffer and then streamed to a 12TB NI-8265 RAID disk array
at a data rate of 250 MB/s per telescope. The ADC sample clock for each
telescope is referenced and phase-locked to a common external 10 MHz clock. To
generate the reference clock, a centrally located White Rabbit (WR) Switch is
used to distribute a 10MHz clock through optical fiber connections to a WR-Len
module located near each of the data acquisition systems. The WR-Len module
takes in the optical fiber signal, and generates the electronic 10 MHz signal
to which the ADC sample clock is referenced.

After observations are completed, the correlation between the stored data of
each telescope is processed off-line by a FPGA-based correlator. The FPGA is
programmed to retrieve the mean intensities of each time stream $\langle
I\rangle=\frac{1}{N}\sum_{i}^{N}{I(t_{i})}$, for the sample $i$, and the cross
correlation between each pair of data streams, c(k) = $\langle
I_{1}(t_{i})I_{2}(t_{i+k})\rangle=\frac{1}{N}\sum_{i}^{N}I_{1}(t_{i})I_{2}(t_{i+k})$,
where $k$ is a digital time-lag inserted between the two channels. The
correlation is calculated over 64 time-lag channels, corresponding to time-
lags of -128 to +124 ns in steps of 4 ns.

### Analysis

From the outputs of the correlator, the second-order coherence function can be
calculated by normalizing the cross correlation by the product of the mean
intensities

$g^{(2)}(\tau,\mathbf{r})=\frac{\langle
I_{1}(t)I_{2}(\mathbf{r},t+\tau)\rangle}{\langle I_{1}(t)\rangle\langle
I_{2}(t)\rangle}$ (1)

where $\mathbf{r}=\mathbf{r_{1}}-\mathbf{r_{2}}$ is the projected baseline
between the two telescopes, $\tau$ is the relative time lag, and the brackets
indicate a statistical average in time assuming a stationary light source. For
chaotic thermal light, such as that from a star, the second-order coherence
function can be written [36] in terms of the first-order coherence
$g^{(1)}(\tau,\textbf{r})$,

$g^{(2)}(\tau,\mathbf{r})=1+|g^{(1)}(\tau,\mathbf{r})|^{2}.$ (2)

The van Cittert-Zernike theorem states that at $\tau=0$, the measured first-
order coherence is equivalent to the Fourier transform of the source angular
intensity distribution in the sky [37]. For randomly polarized light, the
squared modulus of the first-order coherence is reduced by a factor of 1/2. In
the typical case where the detector resolution time $T$ is much longer than
the light coherence time $\tau_{c}\sim 1/\Delta\nu$, where $\Delta\nu$ is the
optical bandwidth, it is reduced by another factor of $\sim\tau_{c}/T$. We
then write the expected zero-baseline coherence as
$N_{0}=\epsilon\tau_{c}/2T$, where $\epsilon$ is a correction factor that
accounts for the shape of the detected light spectral density that sets the
optical bandwidth and corresponding coherence time. The correction factor also
includes potential losses in the coherence signal that, as detailed by
Hanbury-Brown and Twiss [10], are attributed to non-ideal properties of the
filters, detectors and electronic readout system that may influence the
effective electronic or optical bandwidth. Under a uniform disk approximation
for a star of angular diameter $\theta_{UD}$, the squared first-order
coherence can be written as

$|g^{(1)}_{\ast}(\tau=0,r)|^{2}=N_{0}\left|2\frac{J_{1}(x_{UD})}{x_{UD}}\right|^{2}$
(3)

where $x_{UD}=\pi\theta_{UD}r/\lambda$, $|g_{\ast}^{(1)}|^{2}$ is the
coherence due to starlight alone, i.e. $g_{\ast}^{(2)}=\langle
I_{1_{\ast}}I_{2_{\ast}}\rangle/(\langle I_{1_{\ast}}\rangle\langle
I_{2_{\ast}}\rangle)$ where $I_{1_{\ast}}$ and $I_{2_{\ast}}$ are the
starlight intensities at detectors 1 and 2, $J_{1}$ is the Bessel function of
the first kind, and $\lambda$ is the mean observational wavelength. A linear
limb-darkened model [38] was also fit to the data, where the expected squared
first-order coherence is given by

$|g^{(1)}_{\ast}(\tau=0,\textbf{r})|^{2}=N_{0}\left(\frac{1-u_{\lambda}}{2}+\frac{u_{\lambda}}{3}\right)^{-2}\left((1-u_{\lambda})\frac{J_{1}(x_{LD})}{x_{LD}}+u_{\lambda}\sqrt{\pi/2}\frac{J_{3/2}(x_{LD})}{x_{LD}^{3/2}}\right)^{2}$
(4)

where $u_{\lambda}$ is the wavelength-dependent linear limb-darkening
coefficient, and $x_{LD}=\pi\theta_{LD}r/\lambda$, where $\theta_{LD}$ is the
limb-darkened angular diameter.

The measured spatial coherence signal can be affected by detector dark current
and background light. The total signal intensity can be written as a sum of
the starlight and background light sources $I(t)=I_{\ast}+I_{bkg}$ and using
equation 1 it is straightforward to derive the relationship

$|g_{\ast}^{(1)}|^{2}=|g^{(1)}|^{2}\times(1+\beta_{1})(1+\beta_{2})$ (5)

under the assumption that the fluctuations in the starlight-background and
background-background terms are uncorrelated (i.e. $\langle\Delta
I_{1_{\ast}}\Delta I_{2_{bkg}}\rangle=\langle\Delta I_{1_{bkg}}\Delta
I_{2_{\ast}}\rangle=\langle\Delta I_{1_{bkg}}\Delta I_{2_{bkg}}\rangle=0$,
where $\Delta I$ is the AC component of the given source intensity such that
$I(t)=\langle I\rangle+\Delta I(t)$. The terms $\beta_{1}$ and $\beta_{2}$ are
the ratios of the background to starlight intensity $I_{bkg}/I_{\ast}$ for a
given telescope.

Hanbury Brown and Twiss [9] showed that the expected signal to noise ratio of
the correlation due to the spatial coherence of the source, under a first-
order approximation, is given as

$S/N=A\,\alpha\,n\,|g^{(1)}(r)|^{2}\,\sqrt{\frac{\Delta fT_{0}}{2}}$ (6)

where $A$ is the telescope mirror area, $\alpha$ is the quantum efficiency of
the detectors, $n$ is the spectral flux density of the source given in units
of ph s-1 m-2 Hz-1, $\Delta f$ is the electronic bandwidth set by the time-
response of the combined optical and electronic system, and $T_{0}$ is the
observation duration.

For a given run, typically 20 to 30 minutes in total duration, the value of
$\left|g^{(1)}(\tau)\right|^{2}$ is calculated over accumulation frames of 1
second. Any $\left|g^{(1)}(\tau)\right|^{2}$ frames contaminated by correlated
high-frequency radio pickup are removed. The Fourier transform of each frame
is calculated to identify corrupted frames. If the power at the specific noise
frequency is greater than the average power over all other frequencies by a
predetermined threshold cut, the frame is removed. For a typical run, 30 - 40
% of the data is removed as a result of the electronic noise. Additionally,
any frames are removed where the mean PMT current falls below 5 $\mu A$, or
approximately one-half of the total current due to star light. These current
losses are attributed to the attenuation of star light by clouds, or also non-
perfect tracking that results in the collected star-light not centered onto
the detectors. Each frame is shifted in time by the average number of samples
to zero optical path delay to correct for the geometrical optical path-delay.
After path-delay corrections and noise/current cuts are applied, all
$|g^{(1)}(\tau)|^{2}$ frames are averaged together through a weighted mean to
produce a final $|g^{(1)}(\tau)|^{2}$ frame for that run. The weights for each
frame are given by $1/\sigma^{2}_{|g^{(1)}|^{2}}$ where
$\sigma_{|g^{(1)}|^{2}}=\sigma_{\langle I_{1}I_{2}\rangle}/\left(\langle
I_{1}\rangle\langle I_{2}\rangle\right)$, where $\sigma_{\langle
I_{1}I_{2}\rangle}$ is the standard deviation of the correlation over all
time-lag bins for a given frame. Provided sufficient throughput and spatial
coherence, the corrected and averaged results reveal an excess peak at zero
time lag due to the spatial coherence of the source. In our case where the
detector resolution time is much greater than the coherence time of the light,
the shape of the peak is determined by the cross-correlation of the telescope
and detector time response with an amplitude proportional to the amount of
spatial coherence. The Davies-Cotton mirror design of the VERITAS telescopes
introduces an approximate 4 ns spread in the arrival time of photons that
otherwise would arrive synchronously [39]. A PMT detects the photons, with a
characteristic single photo-electron pulse width of 3.7 ns and additional
timing jitter on the order of 1 ns. Simulations of these effects find that the
$\left|g^{(1)}(\tau)\right|^{2}$ peak can be modeled as a Gaussian following
the form

$f(\tau)=Ae^{-\frac{1}{2}(\tau-\tau_{0})^{2}/\sigma_{\tau}^{2}}+C$ (7)

with $A$, $\tau_{0}$, and $C$ left as fit parameters. $A$ is the amplitude of
the coherence peak and directly measures $|g^{(1)}(\textbf{r})|^{2}$,
$\tau_{0}$ is a parameter that accounts for a variable start time between the
separated data acquisition systems, and is constrained within $|\tau|<10$ ns
of zero-lag, and C is another correction parameter that subtracts off any
drift in the mean level of the correlator. The value of the fit width is fixed
to a value of $\sigma_{\tau}=4.0\,$ns determined empirically from the data and
consistent with expectations from simulations. The value of the width was
obtained by allowing it to be a free parameter in the fit of the integrated
$\left|g^{(1)}(\tau)\right|^{2}$ correlation measurements for runs showing a
peak amplitude with a p value less than 3 $\times 10^{-7}$. The value of the
fit width was stable within fit uncertainties for several runs demonstrating
that no significant fluctuations in the relative timing of the acquisition
systems are present.

The value of the fit amplitude and corresponding uncertainty are found for all
runs to measure $\left|g^{(1)}(\tau=0)\right|^{2}$ as a function of the
projected baseline. The projected baseline is calculated from the telescope
positions and source sky position at the time of observation. The value of the
fit amplitude is then multiplied by the scaling factor described in Equation 5
to compensate for night sky background and detector dark current. The mean
level of the background is found by pointing the telescope off source and
recording the average intensity. These off runs are taken before and after
each run on source. The background light corrected values of
$|g^{(1)}_{\ast}|^{2}$ are then fit to a function with the form of Equation 3
in order to obtain the uniform-disk normalization and stellar diameter. Values
of the linear limb-darkening coefficient $u_{\lambda}$ were estimated by
interpolating existing tables that calculate expected values of $u_{\lambda}$
from atmospheric parameters [40]. Using the atmospheric values listed in Table
1, for observations in the B band, a microturbulent velocity of 2 km/s, and
solar metallicity we find values of $u_{\lambda}$ of 0.38 and 0.46 for $\beta$
CMa and $\epsilon$ Ori, respectively.

Source | Spectral Type | $B$ |  $T_{eff}$ (K) |  $log(g)$ (dex) |  $v\sin(i)$ (km/s)
---|---|---|---|---|---
$\beta$ CMa | BII/III | 1.73 | 24000 $\pm$ 500a | 3.43 $\pm$ 0.10a | 20 $\pm$ 7a
$\epsilon$ Ori | B0Ia | 1.51 | 27000 $\pm$ 1000b | 2.90 $\pm$ 0.02b | 65 $\pm$ 15c
Table 1: Atmospheric parameters used to estimate values of the linear limb-
darkening coefficient. The spectral type and B band magnitudes were obtained
from SIMBAD. a Taken from reference [41], b Taken from reference [42], c Taken
from reference [43].

### Extended Data

Figure 3: Coverage of the sources in the (u,v)-plane. Each of the colored
points represent different runs for a given telescope pair.

### Data Availability

The data that support the plots within this paper and other findings of this
study are available from the corresponding author upon reasonable request
(N.M). Due to the extremely large datasets obtained during the observations,
in excess of 40 TB, only the post-correlation data can reasonably be made
available.

### Code Availability

The software used to analyze the data in this study is available upon request
to the corresponding author (N.M.).

## References

  * [31] Matthews, N., Kieda, D., and LeBohec, S. Development of a digital astronomical intensity interferometer: laboratory results with thermal light. _J. of Mod. Opt._ , 65, 1336-1344, (2018)
  * [32] Matthews, N., Clarke, O., Snow, S., LeBohec S., and Kieda D. Implementation of an intensity interferometry system on the StarBase observatory in _Optical and Infrared Interferometry and Imaging VI_ , 107010W (Proc. SPIE 10701, 2018)
  * [33] Matthews, N. et. al. Astrophysical measurements with the VERITAS Stellar Intensity Interferometer in _36th International Cosmic Ray Conference_ , 36 (PoS 2019)
  * [34] Otte, A. N. et. al. The Upgrade of VERITAS with High Efficiency Photomultipliers in _32nd International Cosmic Ray Conference_ (2011)
  * [35] Cardon, R., Matthews, N., Abeysekara, A. U., and Kieda, D. A Fiber Optic Based High Voltage System for Stellar Intensity Interferometry Observations in _36th International Cosmic Ray Conference_ , 358 (PoS 2019)
  * [36] Foellmi, C. Intensity interferometry and the second-order correlation function $g^{(2)}$ in astrophysics. _Astron. Astrophys._ , 507, 1719-1727 (2009)
  * [37] Born, M, and Wolf, E. _Principles of Optics_. Ch. 10 (Pergamon 1980)
  * [38] Hanbury Brown, R., Davis, J., Lake, R. J. W., and Thompson, R. J. The Effects of Limb Darkening on Measurements of Angular Size with an Intensity Interferometer. _Mon. Not. R. Astron. Soc._ , 167, 474-484 (1974)
  * [39] Holder, J. et. al. The first VERITAS telescope. _Astropart. Phys._ , 25, 391 - 401 (2006)
  * [40] Claret, A., and Bloemen, S. Gravity and limb-darkening coefficients for the Kepler, CoRoT, Spitzer, uvby, UBVRIJHK, and Sloan photometric systems. _Astron. Astrophys._ , 529, A75 (2011)
  * [41] Levenhagen, R. S. and Leister, N. V. Spectroscopic analysis of southern B and Be stars. _Mon. Not. R. Astron. Soc._ , 371, 252-262 (2006)
  * [42] Crowther, P. A., Lennon, D. J., and Walborn, N. R. Physical parameters and wind properties of galactic early B supergiants. _Astron. Astrophys._ , 446, 279-293 (2006)
  * [43] Abt H. A., Levato, H., and Grosso, M. Rotational Velocities of B Stars. _Astrophys. J._ , 573, 359-365 (2002)


(cvpr) Package cvpr Warning: Package ‘hyperref’ is not loaded, but highly
recommended for camera-ready version

# ArtAdapter: Text-to-Image Style Transfer using
Multi-Level Style Encoder and Explicit Adaptation

Dar-Yen Chen Hamish Tennent Ching-Wen Hsu
Cardinal Blue
{daryen.chen, hamish.tennent<EMAIL_ADDRESS>
###### Abstract

This work introduces ArtAdapter, a transformative text-to-image (T2I) style
transfer framework that transcends traditional limitations of color,
brushstrokes, and object shape, capturing high-level style elements such as
composition and distinctive artistic expression. The integration of a multi-
level style encoder with our proposed explicit adaptation mechanism enables
ArtAdapte to achieve unprecedented fidelity in style transfer, ensuring close
alignment with textual descriptions. Additionally, the incorporation of an
Auxiliary Content Adapter (ACA) effectively separates content from style,
alleviating the borrowing of content from style references. Moreover, our
novel fast finetuning approach could further enhance zero-shot style
representation while mitigating the risk of overfitting. Comprehensive
evaluations confirm that ArtAdapter surpasses current state-of-the-art
methods.

Figure 1: Our framework is capable of capturing faithful style representation,
from low-level delicate texture to high-level minimalism composition, in
either single or multiple style references, closely adhering to the textual
prompts.

##  1 Introduction

Bridging the realms of artificial intelligence and artistic creativity, text-
to-image (T2I) style transfer [24, 13, 25] stands out as a captivating domain
that masterfully transforms textual descriptions into visually rich and
stylistic representations. The core challenge lies in not just generating
text-resonant images but infusing them with artistic depth and nuance,
spanning from delicate brushstrokes to bold compositional elements—thereby
capturing the essence of artistic vision. Conventional arbitrary style
transfer (AST) methods [53, 9, 21, 7, 4, 8, 28, 15] typically struggle beyond
low-level features such as medium and colors, failing to grasp the more
sophisticated realms of artistic expression. Diffusion approaches [25, 54],
including Textual Inversion [14], DreamBooth [39], and Low-Rank Adaptation
(LoRA) [2, 20], have shown potential in style representation. Yet, these
methods are hindered by laborious finetuning processes and a tendency towards
overfitting, leading to outputs overly influenced by the style references’
content at the expense of the textual context. Especially when dealing with
the unique nature of artworks, these methods face difficulties in style
transfer from a single reference.

Addressing these challenges, we present ArtAdapter, an innovative T2I style
transfer framework utilizing style embeddings derived from a multi-level style
encoder. These style embeddings, representing various layers of artistic
attributes, then intricately interplay with textual context within the text
encoder. The style information is refined further through the novel Explicit
Adaptation mechanism, situated within the cross-attention layers of the
diffusion model. Here, the Explicit Adaptation focuses explicitly on the style
pathway, while leaving the text pathway frozen. This ensures that the artistic
elements, from foundational textures to the distinctive expression of the
artworks, are faithfully and precisely represented in the generated images
without compromising the textual generalizability. Moreover, our approach
incorporates the Auxiliary Content Adapter (ACA) during the training phase.
The ACA plays a vital role by offering weak content guidance, aiding in the
separation of content structure from style references. This ensures that the
final images are not overwhelmed by the content of the style references,
maintaining a clear representation of style elements and the narrative intent
of the text prompts. Furthermore, we introduce a fast finetuning method, which
further refines the model’s ability to capture nuanced style details. This
finetuning is effective for both single- and multi-image style references,
where it achieves detailed style representation with minimal steps, greatly
reducing the time and computational resources typically required. This
innovation addresses the challenge of overfitting and speeds up the adaptation
process. A notable feature of ArtAdapter is its adeptness at style mixing.
Leveraging the multi-level style encoder, ArtAdapter can blend styles from
various references, extracting and combining distinct stylistic elements at
different levels. This enables the production of images that blend a diverse
array of artistic influences, thereby offering remarkable flexibility and
creativity to style transfer.

The principal contributions of our work are summarized as follows: (i) A
groundbreaking T2I style transfer model, ArtAdapter, that harnesses a multi-
level style encoder and the Explicit Adaptation mechanism to capture diverse
levels of style representation, and ensures a subtle balance between style and
textual semantics; (ii) The Auxiliary Content Adapter (ACA) separates content
structure from style references, addressing the issue of content dominance
from style references. (iii) A fast finetuning approach that enables rapid and
effective style adaptation; (iv) ArtAdapter allows for style mixing across
different hierarchical levels, enriching the creative potential of T2I style
transfer.

##  2 Related Work

###  2.1 Text-to-Image Synthesis

Text-to-image (T2I) synthesis has evolved remarkably, with early methods
primarily leveraging Generative Adversarial Networks (GANs) [16]. A paradigm
shift was marked by the introduction of discrete Variational Autoencoders
(VAEs) [36, 12] and autoregressive transformers [46], leading to a more stable
training process and high-quality image generation as demonstrated by Esser
_et al_. [12] and the DALL-E [34]. Subsequent advancements with diffusion
models [19] further refined the generation with a gradual denoising process,
offering enhanced control and fidelity in outputs, as showcased by DALL-E2
[35], Imagen [41], and Stable Diffusion [37]. Recent developments [50, 55, 51]
have extended these models’ capabilities in various context, with innovations
like ControlNet [52] and T2I adapters [31] for spatial semantics, enriching
the conditioning options well beyond textual inputs.

###  2.2 T2I Personalization

In T2I personalization, the central goal is to tailor pretrained models to a
specific concept using a collection of reference images. Innovations like
Textual Inversion [14] and DreamBooth [39] pioneered this direction, learning
text embeddings and optimizing diffusion backbone respectively. The quest for
efficiency has led to the development of Low-Rank Adaptation (LoRA) [20, 2],
which reduces parameters by decomposing residual weights into two low-rank
estimated matrices. SVDiff [17] and Lightweight DreamBooth [40] further refine
this process using Singular Value Decomposition (SVD) and orthogonal
incomplete basis within LoRA weight-space respectively. Perfusion [45]
incorporates Rank-One Model Editing (ROME) [30], exemplifies targeted model
edits aligned with conceptual directions. Meanwhile, HyperDreamBooth [40]
enables rapid adaptation to new concepts through hypernetwork-initialized
rank-1 residuals. Beyond finetuning, various approaches [43, 49] deploy task-
specific components for zero-shot adaptation to novel concepts, including
Taming Encoder [23] and IP-Adapter [51] for content semantics. Interestingly,
T2I style transfer aligns closely with style personalization, treating
artistic style as a unique conceptual entity. Our work contributes to this
evolving landscape by developing a novel zero-shot T2I style transfer
framework whose performance can be further improved through fast finetuning.

###  2.3 Style Transfer

Traditionally, Style Transfer [4, 8, 28] has depended on extracting content
from target images and matching style via second-order statistics [21, 15].
However, the biases [7, 9] inherent in statistics cripple the representations.
IEST [7] and CAST [53], address this issue by introducing contrastive
learning. The advent of vision transformers [11] has facilitated the capture
of long-range features, as seen in StyleFormer _et al_. [48] and StyTr2 [9].
ArtFusion [6] presents an integration of diffusion models [19] into style
transfer, bypassing reliance on statistical matching. Following the booming of
T2I models, InST [54] exemplifies the growing interest in T2I style transfer,
with semantics provided by textual prompt, learning artistic features and
guiding the generating with the CLIP image encoder [33]. Nonetheless, T2I
style transfer confronts challenges such as content from style references
overshadowing the textual context [47]. Addressing this, our framework
introduces an auxiliary component to foster content-style disentanglement,
mitigating the influence of content semantics from style references.

##  3 Approach

Preliminary on T2I Diffusion Models. A Text-to-Image (T2I) diffusion model
comprises two foundation components: the diffusion backbone
$\epsilon_{\theta}$ and the text encoder $c_{\theta}$. The diffusion backbone
meticulously manages a process of adding and removing noise. The inference
process begins with a distribution of random noise, which is gradually
denoised step by step according to the textual prompt to form a coherent
image. The textual prompt, denoted as $y$, first undergoes tokenization and
index-based lookup, linking it to text embeddings $F_{txt}$, a sequence of
vectors. These embeddings are further refined by the text encoder, which
contextualizes the information, producing enriched text embeddings $E_{txt}$
that encapsulate the textual description’s meaning, intent, and subtleties.
Typically, T2I diffusion models employ cross-attention layers to access and
utilize the semantic information contained in $E_{txt}$. The T2I diffusion
models’ objective is defined by the equation [10]:

$\mathcal{L}=\mathbb{E}_{x,y,\epsilon,t}\Bigr{[}||\epsilon_{\theta}(x_{t},t,c_{\theta}(F_{txt}))-\epsilon||^{2}_{2}\Bigr{]},$
(1)

where $x$ is the image, and $\epsilon$ represents the noise component in the
corrupted $x_{t}$ at timestep $t$. This equation aims to estimate the noise
involved in the diffusion process, which is crucial to the gradual denoising
process, aligning samples with the text description.

Overall Framework. In Figure 2, we present the architecture of our ArtAdapter,
utilizing a pretrained diffusion model as its backbone. Our method instructs
the diffusion model in articulating diverse styles through learnable style
embeddings. For encoding the style reference $x_{sty}$ into style embeddings
$F_{sty}$, ArtAdapter employs a multi-level style encoder to process
activations of pretrained VGG network [27]. Subsequently, these style
embeddings are concatenated with text embeddings $F_{txt}$, and processed
through the text encoder to obtain the style encodings $E_{sty}$. To allow the
diffusion backbone to adapt to these style concepts, we introduce the Explicit
Adaptation mechanism within the cross-attention layers. This mechanism
exclusively adapts to style encodings while preserving the integrity of text
encoding. This ensures the robust generalization of the pretrained diffusion
model while enabling precise adaptation to complex style nuances. Furthermore,
our framework incorporates the Auxiliary Content Adapter (ACA) - a key
component during training, that is excluded when inference. ACA provides rough
content structure information to the UNet backbone [38], helping eliminate the
influence of the content semantics in the style reference. Conclusively, our
fast finetuning method is designed to capture more nuanced style
characteristics, applicable to both individual and collective style
references.

Figure 2: Architecture of ArtAdapter. Style embeddings, extracted through a
cascade of a pretrained VGG [27] followed by the multi-level style encoder,
interact with text embeddings in the text encoder. In the cross-attention
layers, the Explicit Adaptation exclusively optimizes the style-related
projection to align outputs with style references. The Auxiliary Content
Adapter provides weak content guidance during training, helping disentangle
the content structure in the style reference. Our approach faithfully captures
the style features without content semantics.

###  3.1 Multi-Level Style Encoder

In style transfer, second-order statistics of VGG [27] activations,
specifically the mean and standard deviation, have been crucial for their
effectiveness in achieving style similarity [22, 15]. These statistics lack
overt content information, enabling more unadulterated style features. A core
of our ArtAdapter lies in the extraction of multi-level style features,
amplifying the expressiveness and interpretability of the style transfer.
Specifically, we select VGG activations - relu3_3, relu4_3, and relu5_3 \- to
represent low-, mid-, and high-level features, respectively. Once we
concatenate these activations’ mean and standard deviation [21, 6], low-,
mid-, and high-leve style embeddings $F_{sty}^{l}$, $F_{sty}^{m}$,
$F_{sty}^{h}$ are extracted using distinct MLPs. These embeddings, akin to
newly learned words for style, interact with the textual context in the text
encoder and present style information to the diffusion backbone. In
particular, our implementation generates 3 embeddings for each level, which
combine to form a style embedding $F_{sty}$ of length 9. These multi-level
style embeddings provide rich style context, enhancing the style fidelity.

###  3.2 Explicit Adaptation

Significant advancements in adapting pretrained diffusion models to new
contexts, notably with efficient methods like LoRA [20], have demonstrated
effective adaptation with fewer parameters. To enhance the integration of
style within the diffusion backbone, we introduce Explicit Adaptation, a novel
adaptation mechanism different from pursuing minimal parameters. Within the
cross-attention layers, the Explicit Adaptation mechanism is meticulously
applied to the key and value projections of the style encodings $E_{sty}$,
while leaving the pathways of the text embeddings $E_{txt}$ frozen. The output
$h$ of K- or V-projection in a cross-attention layer can be represented as:

$h=W\\{E_{sty},E_{txt}\\}+\alpha\Delta W\\{E_{sty},0\\}$ (2)

where $W$ represents the original weights, which are frozen to maintain
existing knowledge, $\alpha$ is a learnable scale, and $\Delta W$ is the
residual weight focusing solely on the style encodings.

The term ”Explicit” signifies a direct and intentional approach, not only in
the operational dynamics of the adaptation but also in the model’s learning
objectives. By this deliberate demarcation, the model’s foundational
competencies in textual interpretation are fortified, ensuring the adaptation
process is sharply concentrated on the incorporation of style nuances hidden
in the style encodings. This innovation enables our ArtAdapter to refine style
representations without compromising the established robust linguistic
knowledge base. The merits of Explicit Adaptation, particularly in its role in
the precise acquisition of fine-grained style features, will be further
dissected and validated in Section 4.4.

###  3.3 Auxiliary Content Adapter

One of the major challenges in T2I style transfer is the unintentional
overpowering of the style reference’s content within the generated images,
eclipsing the textual semantics. Even conditioning on VGG [27] statistics
features without content structure, due to certain highly recurrent patterns
in the dataset, like human faces [6], the model might become prone to
overfitting specific statistical features of content, leading it to transfer
content patterns from the style reference. To address this challenge, we
introduce the Auxiliary Content Adapter (ACA) into our framework, an
implementation of the T2I adapter [31]. It takes the image $x$ as input,
providing the diffusion backbone with essential content cues. This ensures
that the ArtAdapter, including the multi-level style adapter and the Explicit
Adaptation, concentrates solely on capturing pure style features, avoiding any
unwanted diversion towards mimicking content semantics from the style
reference.

To calibrate the influence of ACA, its features are infused within the deepest
input block of the UNet backbone [38], tactically positioned to impact only
the initial denoising stages (20% in this work), where the model delineates
only the rudimentary content outlines. Moreover, to fortify the distinction
between content and style, we subject the ACA’s input to random color
augmentations, such as inversion and color jitter. These augmentations prevent
the ACA from capturing color representations that are rightfully the domain of
style components. This approach confines the ACA’s function to offering only
foundational content structure, stripped of complex details, thus paving the
way for a more nuanced style representation. Crucially, the ACA’s role is
confined to the training process. During inference, the ACA is rendered
obsolete, allowing our ArtAdapter to draw exclusively from the learned style
features, thus ensuring that the generated images present only the desired
style features without content semantics from style references.

Figure 3: Qualitative results. This collection of images exhibits the
ArtAdapter’s capability to present faithful style representation across
diverse artworks without compromising on semantics, showcasing the versatility
and deep understanding of artistic and textual contexts.

###  3.4 Fast Finetuning and Multiple Style References

Building on the Explicit Adaptation mechanism, we introduce a fast finetuning
method to enhance the precision and subtlety of zero-shot style
representation. The crux of this finetuning lies in the following equation:

$h=W\\{E_{sty},E_{txt}\\}+\alpha\Delta W\\{E_{sty},0\\}+\\{\Delta
h_{sty},0\\}$ (3)

$\Delta h_{sty}$ is the residual vector that directly adjusts the style-
related activations, refining the model’s ability to mirror the
characteristics of the style references. During finetuning, all components
besides the vector residuals remain frozen, ensuring an efficient adaptation
process focused on the enhancement of style representation. The principle of
explicit adaptation mechanism ensures that the textual context pathways remain
intact, offering a safeguard against overfitting—a common challenge in single-
image personalization. This finetuning is characterized by its efficiency.
With only minimal extra parameters from the rank-1 residual vector, and a
focus on style operations, the process is expedited to dozens of steps,
typically completed within minutes. Notably, this approach can be easily
extended to multi-reference style transfer. By averaging the style embeddings
from all references, the model can optimize the vector residuals to reflect
the aggregate style.

###  3.5 Style Mixing

Our innovative style mixing approach leverages the multi-level style encoder
to expand the T2I style transfer’s horizons. With a trio of style reference
images, we extract their respective style embeddings at low, mid, and high
levels. These are subsequently constructed into compositional style embeddings
by concatenation. Such a fusion of embeddings enables ArtAdapter to synthesize
a new, mixed style that simultaneously exhibits features from the individual
styles at corresponding hierarchical levels. Furthermore, this method of style
mixing can easily extend to styles that have undergone finetuning, by applying
the residual vectors to the corresponding entries.

##  4 Experiments

Data. For training, we harness the LAION AESTHETICS [42] and WikiArt [32]
datasets. The WikiArt dataset has been captioned using BLIP-2 [26]. For
testing, we adopt a test dataset comprising 35 prompts, including some derived
from Wang _et al_. [47]. We collect 10 styles for both single- and multi-
reference style transfer separately, culminating in a total of 700 image pairs
for the test dataset.

Evaluation Metrics. We employ the CLIP [33] metrics for objective assessment,
including similarity for text and style. The aesthetics score [1] functions as
an indicator, gauging the aesthetic appeal of the generated images.
Additionally, we undertake a user study involving 212 participants utilizing a
7-point scale, with 5 tuples per user, to assess subjective text and style
accuracy, alongside the overall quality.

Implementation Details. This work leverages the SD V1.5 [37] as the backbone.
We employ an AdamW optimizer [29] with a batch size of 16, a learning rate of
1e-4 for the multi-level style encoder and the ACA, and a reduced learning
rate of 1e-7 for Explicit Adaptation residual weights. Specifically, we apply
LoRA [2, 20] to reduce the parameters. During the finetuning, we optimize the
vector residual using a learning rate of 0.02 over 25 iterations, taking
$\sim\\!\\!1$ minute. All experiments are undertaken on a single Nvidia GTX
4090 GPU. For inference, we set the sampling [44] steps to 50, with the
classifier-free guidance scale [18] being fixed at 9. Unless otherwise
specified, all reported results were obtained with finetuning.

Figure 4: Illustration of Style Mixing. The seamless integrations of the two
styles reflect the distinct contributions of style features from different
hierarchical levels to the final images and demonstrate the remarkable
flexibility of ArtAdapter. Figure 5: Qualitative comparison on single style
reference. Our results showcase ArtAdapter’s superior style alignment over
other approaches [53, 9, 54, 5]. Note that the SD [37] column works as content
target images for conventional AST models [53, 9]. Figure 6: Qualitative
comparison on multiple style references. Compared to TI [14] and LoRA [2, 20],
our ArtAdapter provides greater consistency and coherence, while preventing
content borrowed from the style references to the results.

###  4.1 Qualitative Evaluation

The ArtAdapter’s prowess is showcased in Figure 1, where it captures nuanced
styles from various either single or multiple style references, while
faithfully reflecting the textual prompts. This is particularly evident in the
top row of Figure 1, where a clear facial structure is depicted in the style
reference, a challenging scenario often encountered in T2I style transfer.
Remarkably, our model adeptly reproduces the ornate texture and aesthetic of
the style reference without erroneously incorporating the facial structure
into the final results. The bottom row of Figure 1 illustrates ArtAdapter’s
capability to go beyond mere low-level style representation. Here, we observe
not only the replication of black-and-white tones and geometric patterns but,
more crucially, the high-level abstract composition that resonates with the
style reference collection. This demonstrates the multi-level style encoder
and Explicit Adaptation mechanism’s collaborative efficacy in rendering styles
that are both authentic and expressive.

Figure 3 provides further insights into ArtAdapter’s capacity, showcasing
side-by-side comparisons of various styles within the same textual context.
From the precise transfer of textures and tones to the emulation of geometric
shapes and the overall composition, ArtAdapter showcases a deep understanding
of the distinctive artistic concepts inherent in the style references.
ArtAdapter effectively circumvents borrowing content from style references,
creatively reinterpreting artistic elements to yield visually compelling art
that remains faithful to the textual descriptions. This attests to the
efficacy of the explicit mechanisms in our approach, which align the subject
matter with the style, while preventing overfitting that commonly happens in
single-image personalization finetuning.

In summary, our method facilitates T2I style transfer, advancing toward a more
genuine and precise representation of styles. It encapsulates the essence of
the artistic intent, offering a substantial leap.

Type | Approach | Text $\uparrow$ | Style $\uparrow$ | Aesth $\uparrow$
---|---|---|---|---
single ref | CAST | 0.297 | 0.583 | 5.410
StyTr2 | 0.299 | 0.577 | 5.308
InST | 0.235 | 0.660 | 5.255
Ours (zero-shot) | 0.269 | 0.656 | 5.532
Ours | 0.255 | 0.707 | 5.601
multi ref | TI | 0.265 | 0.678 | 5.712
LoRA | 0.254 | 0.678 | 5.701
Ours | 0.258 | 0.680 | 5.610
Table 1: Quantitative Comparison. ArtAdapter demonstrates superior style similarity and aesthetics score, even with zero-shot, while the text similarity is competitive. Model | Text $\uparrow$ | Style $\uparrow$ | Quality $\uparrow$
---|---|---|---
TI | 4.49 | 4.35 | 4.38
LoRA | 3.91 | 4.08 | 4.18
Ours | 4.74 | 4.76 | 4.43
Table 2: Perceptual Comparison. ArtAdapter exhibits preeminence in all
subjective metrics over TI [14] and LoRA [2, 20], highlighting its
effectiveness in T2I multi-reference style transfer based on user preferences.

###  4.2 Style Mixing

Figure 4 demonstrates ArtAdapter’s proficiency in seamlessly mixing two styles
within one image. Achieving this fusion involves applying distinct styles to
different hierarchical levels: one affects low- and mid-level features, while
the other shapes high-level attributes. This mixing reveals the significant
roles of different levels in shaping the final image. The low and mid-levels
primarily dictate tone and texture, as evidenced by the haziness of the ”Mona
Lisa” and Monet’s brushstroke texture. Conversely, high-level features dictate
the overall compositional structure and artistic expression, as exemplified by
the simplistic style in the second row. This multi-level approach underscores
our model’s flexibility and interpretability in style transfer, facilitating a
creative combination of artistic elements.

###  4.3 Comparison with State-of-the-art Methods

Single Style Reference. In Figure 5, our ArtAdapter is compared with
conventional AST models, such as CAST [53] and StyTr2 [9], the diffusion-based
InST [54], and the advanced DALL-E3 [5] in single-reference T2I style
transfer. Utilizing SD [37] as for T2I generation, stylized by AST models, we
have the T2I style transfer results of conventional AST approaches [53, 9].
These models typically struggle to transcend basic color transfer, struggling
with the conveyance of high-level style attributes, particularly abstract
geometric configurations, and holistic style elements. InST [54] tends to
produce results with unnatural style representation and unintended content
borrowings from style references, as evidenced by the robots in the last row.
DALL-E3 [5], leveraging ChatGPT [3] to generate style descriptions, struggles
in the actual reproduction of style features, revealing a bottleneck in prompt
engineering.

Distinctly, our zero-shot ArtAdapter captures the original style’s brushwork,
texture, and overall aesthetic with exceptional fidelity, as evidenced by the
quantitative results in Table 1. This fidelity is further enhanced by the
finetuning, leading to an alignment closely mirroring the original artworks.
Quantitative analyses, as shown in Table 1, corroborate our model’s superior
performance, achieving a style similarity of 0.707 and an aesthetics score of
5.601, which are significantly higher than other methods. This superior
performance is achieved with a minimal trade-off in text fidelity. Without
finetuning, the similarity score and aesthetics score of our zero-shot results
still surpass others [53, 9] except for InST [54], which has a competitive
style similarity but the worst aesthetic score.

Multiple Style Reference. In assessing multi-reference T2I style transfer, our
ArtAdapter is juxtaposed with Textual Inversion (TI) [14] and Low-Rank
Adaptation (LoRA) [20], detailed in Figure 6 and Table 1. Despite the
successes of TI [14] and LoRA [20] in capturing subtle style nuances, they
exhibit overfitting, as evidenced by their dominance of style reference
content, such as the face of Van Gogh, leading to discrepancies with the
textual prompts. Instances such as the top-right corner of the sample manifest
the incoherence in their style transfer, rooted in localized stylization
problems. Moreover, varying degrees of style assimilation within a single
style reference in the bottom row reveals a lack of consistency. These
highlight the instability of TI [14] and LoRA [20] in style transfer. Our
ArtAdapter excels in rendering outputs that accurately embody style features
from the collection and eliminate content borrowing, with consistent
representation in diverse textual contexts. It nicely preserves the inherently
strong generality of SD [37]. Quantitative analysis demonstrates parity in
performance among the methods, highlighting our approach’s efficiency with a
fast finetuning process of $\sim\\!\\!1$ minute.

User Study. To evaluate the efficacy of our multi-reference T2I style transfer
approach, a comprehensive user study, detailed in Table 2, is conducted. The
study unveils a marked discrepancy in text and style precision, with
ArtAdapter achieving 4.74 and 4.76 scores, respectively, thus underscoring its
enhanced adherence to textual prompts and style references, surpassing TI [14]
and LoRA [20] benchmarks. Moreover, the elevated quality score of 4.43
underscores the visual appeal of our generated images. Collectively, these
results indicate that our approach provides an improved T2I style transfer
experience from a user perspective.

Figure 7: Ablation research. These comparisons highlight the importance of
each mechanism in our framework.

###  4.4 Ablation Study

The ablative analysis illustrated in Figure 7 dissects our ArtAdapter,
emphasizing the distinct contributions and impact of each core mechanism on
style transfer performance.

Auxiliary Content Adapter. The first row underscores the crucial role of the
Auxiliary Content Adapter (ACA). In its absence, the model inappropriately
adopts content semantics. The outputs will be dominated by content features in
the style reference—such as the face in ”Mona Lisa”. The ACA’s function to
separate content and style learning is confirmed; it empowers the model to
preserve style features while filtering out content during inference,
guaranteeing that the generated images adhere to the textual prompts.

Explicit Adaptation. In the second and third rows, we scrutinize the
effectiveness of the Explicit Adaptation mechanism. Without this mechanism,
the model shows a marked decrease in its ability to accurately represent style
across different levels. This is evidenced by distorted colors, imprecise
geometric formations, and a lack of depth in capturing the compositional
elements and artistic intent, such as those found in Salvador Dali’s
surrealist works. The explicit adaptation mechanism is thereby shown to be
indispensable for the model’s ability to internalize style contexts,
generating outputs with high style fidelity.

Exclusive Style Encoder. Sole reliance on the multi-level style encoder,
without incorporating adaptation in the backbone, leads to significant
degradation in style representation, especially obvious in the second and
third rows. The style encoder can only capture features such as imprecise
color and rough texture patterns, failing to grasp the more profound stylistic
expressions. This clearly highlights the critical role of the diffusion
backbone in adapting to various style contexts, affirming that the integration
of all components is critical for optimal style transfer performance.

##  5 Conclusion

In this work, our innovative T2I style transfer framework, ArtAdapter, has
effectively demonstrated its ability to synthesize images that faithfully
align with given textual prompts and style references. By employing a multi-
level style encoder for nuanced style capture, the Explicit Adaptation for
effective style integration, and the Auxiliary Content Adapter (ACA) for
content-style separating, our ArtAdapter sets a new standard in achieving both
style and text fidelity. The implementation of our fast finetuning strategy
significantly enhances the efficiency of style alignment, establishing our
model as a benchmark in the realm of T2I style transfer. However, ArtAdapter
does have limitations, particularly in style mixing. We observe that high-
level style embeddings often inadvertently incorporate elements from lower
levels, causing interference in the style mixing process. We aim to refine the
disentanglement of hierarchical style features to improve the authenticity and
precision of style mixing in future works.

## References

  * aes [2022] Clip+mlp aesthetic score predictor.  https://github.com/christophschuhmann/improved-aesthetic-predictor, 2022.
  * db_[2022] Low-rank adaptation for fast text-to-image diffusion fine-tuning.  https://github.com/cloneofsimo/lora, 2022.
  * cha [2023] Chatgpt.  https://openai.com/blog/chatgpt, 2023.
  * An et al. [2021] Jie An, Siyu Huang, Yibing Song, Dejing Dou, Wei Liu, and Jiebo Luo.  Artflow: Unbiased image style transfer via reversible neural flows.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Betker et al. [2023] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh.  Improving image generation with better captions.  2023\.
  * Chen [2023] Dar-Yen Chen.  Artfusion: Controllable arbitrary style transfer using dual conditional latent diffusion models, 2023.
  * Chen et al. [2021a] Haibo Chen, Lei Zhao, Zhizhong Wang, Zhang Hui Ming, Zhiwen Zuo, Ailin Li, Wei Xing, and Dongming Lu.  Artistic style transfer with internal-external learning and contrastive learning.  In _Advances in Neural Information Processing Systems_ , 2021a.
  * Chen et al. [2021b] Haibo Chen, lei zhao, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Ailin Li, Wei Xing, and Dongming Lu.  Artistic style transfer with internal-external learning and contrastive learning.  In _Advances in Neural Information Processing Systems_ , 2021b.
  * Deng et al. [2022] Yingying Deng, Fan Tang, Weiming Dong, Chongyang Ma, Xingjia Pan, Lei Wang, and Changsheng Xu.  Stytr2: Image style transfer with transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Quinn Nichol.  Diffusion models beat GANs on image synthesis.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  2021\.
  * Esser et al. [2021] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021.
  * Fu et al. [2022] Tsu-Jui Fu, Xin Eric Wang, and William Yang Wang.  Language-Driven Artistic Style Transfer.  In _European Conference on Computer Vision (ECCV)_ , 2022.
  * Gal et al. [2023] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Ghiasi et al. [2017] Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, and Jonathon Shlens.  Exploring the structure of a real-time, arbitrary neural artistic stylization network.  In _BMVC_ , 2017.
  * Goodfellow et al. [2014] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  In _Proc. NeurIPS_ , 2014.
  * Han et al. [2023] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang.  Svdiff: Compact parameter space for diffusion fine-tuning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2023.
  * Ho and Salimans [2021] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  In _NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications_ , 2021.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  2020\.
  * Hu et al. [2022] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2022.
  * Huang and Belongie [2017a] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In _ICCV_ , 2017a.
  * Huang and Belongie [2017b] Xun Huang and Serge J. Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017b.
  * Jia et al. [2023] Xuhui Jia, Yang Zhao, Kelvin C. K. Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su.  Taming encoder for zero fine-tuning image customization with text-to-image diffusion models, 2023.
  * Kwon and Ye [2022] Gihyun Kwon and Jong Chul Ye.  Clipstyler: Image style transfer with a single text condition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Kwon and Ye [2023] Gihyun Kwon and Jong Chul Ye.  Diffusion-based image translation using disentangled style and content representation.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023.
  * Liu and Deng [2015] Shuying Liu and Weihong Deng.  Very deep convolutional neural network based image classification using small training sample size.  In _2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR)_ , 2015.
  * Liu et al. [2021] Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Meiling Wang, Xin Li, Zhengxing Sun, Qian Li, and Errui Ding.  Adaattn: Revisit attention mechanism in arbitrary neural style transfer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 6649–6658, 2021.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled Weight Decay Regularization.  In _International Conference on Learning Representations_ , 2019.
  * Meng et al. [2022] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov.  Locating and editing factual associations in GPT.  2022\.
  * Mou et al. [2023] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  _arXiv preprint arXiv:2302.08453_ , 2023.
  * Phillips and Mackintosh [2011] Fred Phillips and Brandy Mackintosh.  Wiki art gallery, inc.: A case for critical thinking.  _Issues in Accounting Education_ , 2011.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning Transferable Visual Models From Natural Language Supervision.  In _ICML_ , 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _Proceedings of the 38th International Conference on Machine Learning_ , 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Razavi et al. [2019] Ali Razavi, Aaron van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  In _MICCAI_ , 2015.
  * Ruiz et al. [2022] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  DreamBooth: Fine Tuning Text-to-image Diffusion Models for Subject-Driven Generation.  In _CVPR_ , 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman.  Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models, 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  2022\.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5b: An open large-scale dataset for training next generation image-text models.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.
  * Shi et al. [2023] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung.  InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning, 2023.
  * Song et al. [2021] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _International Conference on Learning Representations_ , 2021.
  * Tewel et al. [2023] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon.  Key-locked rank one editing for text-to-image personalization.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , 2017.
  * Wang et al. [2023] Zhouxia Wang, Xintao Wang, Liangbin Xie, Zhongang Qi, Ying Shan, Wenping Wang, and Ping Luo.  Styleadapter: A single-pass lora-free model for stylized image generation, 2023.
  * Wu et al. [2021] Xiaolei Wu, Zhihao Hu, Lu Sheng, and Dong Xu.  Styleformer: Real-time arbitrary style transfer via parametric style composition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Xiao et al. [2023] Guangxuan Xiao, Tianwei Yin, William T. Freeman, Frédo Durand, and Song Han.  Fastcomposer: Tuning-free multi-subject image generation with localized attention.  _arXiv_ , 2023.
  * Xu et al. [2023] Xingqian Xu, Jiayi Guo, Zhangyang Wang, Gao Huang, Irfan Essa, and Humphrey Shi.  Prompt-free diffusion: Taking” text” out of text-to-image diffusion models.  _arXiv preprint arXiv:2305.16223_ , 2023.
  * Ye et al. [2023] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang.  Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.  _arXiv preprint arxiv:2308.06721_ , 2023.
  * Zhang et al. [2023a] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2023a.
  * Zhang et al. [2022] Yuxin Zhang, Fan Tang, Weiming Dong, Haibin Huang, Chongyang Ma, Tong-Yee Lee, and Changsheng Xu.  Domain enhanced arbitrary image style transfer via contrastive learning.  In _ACM SIGGRAPH_ , 2022.
  * Zhang et al. [2023b] Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng Xu.  Inversion-based style transfer with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023b.
  * Zhao et al. [2023] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong.  Uni-controlnet: All-in-one control to text-to-image diffusion models.  2023\.

Supplementary Material

##  A Test Dataset

We showcase the style references and prompts utilized in our quantitative
evaluation and user study in Figures 8 and 9, and Table 3. Renowned artworks
have been sourced from https://www.wikiart.org/, while others are synthesized
images collected from https://civitai.com/.

Figure 8: Style reference used in the test dataset for single-reference T2I
style transfer. Figure 9: Collections of styles used in the test dataset for
multi-reference T2I style transfer.

“A robot” | “A curly-haired girl wearing a red dress, she is dancing”
---|---
“A little boy playing football” | “A monkey playing with a banana”
“A dog in the desert” | “A pine tree with a snowman hugging it”
“A bird in a wood” | “A stone with a face carved on it, standing on a pedestal in a museum”
“A kitten sleeping on a pillow” | “A modern house with a pool”
“A daisy with a ladybug on it” | “A house covered with ice and snow”
“Cherry blossom in full bloom” | “People in gondolas on the river”
“solar system and galaxy” | “A woman sitting on grass near a tree”
“A river with rapids and rocks” | “A windmill on a hillside”
“Polar bears on the ice” | “A woman sitting on grass near a tree”
“A snowy mountain peak” | “A castle with steps leading up to it”
“A motorcycle” | “A couple sitting on a swing seeing aurora”
“Sheep in autumn field” | “A woman wearing a baseball cap”
“Volcano eruption” | “Eiffel Tower in heavy lightning storm”
“A white rose” | “A house made of cardboard boxes”
“A stone with a hole in it” | “A bronze sculpture of an old man”
“A portrait of a rugby star” | “A colosseum near a waterfall”
“A girl playing electric guitar” |

Table 3: List of prompts used in the test dataset. Figure 10: Ablative results
of the ACA. Figure 11: Adapting $\alpha$. ArtAdapter enables users to scaling
the $\alpha$, effectively balance textual and style fidelity. Figure 12:
Results with additional structural controls.

##  B Details on User Study

We randomly selected 40 generated images, encompassing the entire spectrum of
multi-reference style collections and prompts. Results from LoRA [20], TI
[14], and our ArtAdapter were presented in a randomized order for users to
compare and rate on a seven-point scale. Responses were solicited from 212
individuals, comprising both GenAI industry professionals and general users.
Each participant evaluated five image tuples randomly drawn from our
selection, yielding an aggregate of 1060 assessments.

##  C Extended Ablation Study

###  C.1 Auxiliary Content Adapter

Figure 10 demonstrates how the Auxiliary Content Adapter (ACA) captures rough
content structures, influencing the final results. When employing ACA during
the denoising process, the content structures from style references will be
presented. Consequently, by omitting ACA during inference, we can effectively
remove these content structures while retaining the style representation.

###  C.2 Adaptive $\alpha$

Figure 11 illustrates the impact of the adaptive $\alpha$, as defined in
Equation 2, during sampling. By adjusting the scaling factor of $\alpha$,
users can tailor the balance between the style elements from style references
and the textual semantics. A higher $\alpha$ scaling emphasizes style traits
in the results, albeit at the expense of textual semantics alignment, while a
lower scaling does the opposite. This flexibility empowers users to generate
results that align with their own preferences.

##  D More Qualitative Results

In Figure 12, we demonstrate how effectively ArtAdapter integrates with the
existing T2I-Adapter [31], showcasing its adaptability in exerting additional
structural controls. Further evaluations of ArtAdapter across a broader range
of styles are presented in Figures 13, 14, and 15. These results underscore
ArtAdapter’s impressive performance across diverse styles.

Figure 13: More results on single-reference T2I style transfer.

Figure 14: More results on single-reference T2I style transfer.

Figure 15: More results on single-reference T2I style transfer.

Figure 16: More results on multi-reference T2I style transfer.