# T-RAX: Transversely Resonant Axion eXperiment

Chang Lee,11footnotetext: Corresponding author. Olaf Reimann

###### Abstract

We propose to use an elongated rectangular waveguide near its cutoff frequency
for axionic dark matter searches. The detector’s large surface area allows for
significant signal power, while its narrow transverse dimension and tapered-
waveguide coupling suppress parasitic modes. The proposed system can fit
inside a solenoid magnet and is sensitive to the QCD-axion in the axion mass
$40-400\,\mu$eV. We describe the theoretical principles of the new design,
present simulation results, and discuss the implementation.

##  1 Introduction

While numerous astronomical observations strongly support the existence of
dark matter [1], its identity remains one of the biggest mysteries of modern
physics. Axions were initially motivated to solve the Strong CP problem [2, 3,
4, 5], but also turned out to be excellent candidates for dark matter [6, 7,
8]. Axions in the mass range 40–400$\,\mu$eV are particularly favored in the
scenarios where the Peccei-Quinn symmetry breaks after cosmic inflation [9,
10, 11, 12, 13, 14].

Unfortunately, most existing axion dark matter searches are not optimized in
this mass range. They utilize axion-photon coupling resonantly enhanced inside
a cavity [15, 16, 17, 18]. While the strategy has been extraordinarily
successful in the mass below $\mathcal{O}(10)~{}\mu$eV, it becomes rapidly
inefficient as the mass increases. The cavity’s volume shrinks inversely
proportional to the axion mass. The shrinking volume reduces the signal power
and increases the search time.

Dielectric haloscopes are promising alternatives. They detect axion-induced
traveling waves from dielectric interfaces [19, 20]. The traveling waves
resonate only in the propagating direction and not in the transverse
dimensions. Their signal power can increase proportionally to the area of the
dielectrics, independent of the axion mass. Wave detection using a horn
antenna or tapered waveguide can couple to the exact transverse mode of
interest, not the unwanted higher-order modes. Unfortunately, the larger
dielectric also requires a larger magnet. In addition, the traveling wave
detection system also requires a long optical path perpendicular to the
magnetic field. As a result, the dielectric haloscopes need a large custom
dipole magnet.

Another recent proposal suggests using a _thin-shell cavity_ : two parallel
conducting plates separated by one half wavelength [21, 22]. The plates
support only the fundamental transverse modes. The cavity’s volume can
increase independent of the frequency in two other dimensions. The design
varies the separation width to scan a wide axion mass range. The detector
coherently sums the signals from multiple ports to filter out the unwanted
high-order modes. The system can also fit in a simple solenoid magnet. We
point out two challenges of the design. First, the tuning mechanism allows
multiple degrees of freedom and is mechanically complicated. Slight
misalignment or tilt of the mechanism splits or shifts the peaks. Synchronized
operation of multiple such cavities will be even more challenging. Second, the
plates still host multiple vertical higher-order modes, TE/TM $1ml$ and $0ml$,
and identifying them with dipole antennas becomes rapidly challenging. These
unwanted modes overlap with the desired modes and limit vertical scale-up. In
addition, coherent signal summing is not proven in a resonant setup.

In this paper, we propose a novel axion detector concept that takes advantage
of both the _thin-shell cavity_ and the dielectric haloscope. Our proposed
Transversely Resonant Axion eXperiment (T-RAX) utilizes an elongated
rectangular waveguide as a large-volume resonator with few higher-order modes.
Our approach adds dielectric slabs to induce longitudinal resonance and to
tune the system. To collect the axion-induced signal, T-RAX uses a tapered
waveguide, which is relatively simple and easy to implement. T-RAX can
potentially sense some QCD-axion models in the centimeter-wave range, but it
is also relatively small and can fit inside a solenoid magnet. In the next
section, we discuss the principle of T-RAX. In Sec. 3, we present the result
from finite element method simulations. In Sec. 4, we discuss how to reduce
the detector profile to fit it inside a solenoid magnet. In Sec. 5, we discuss
the projected sensitivity of T-RAX. Finally, we summarize and conclude in Sec.
6.

##  2 Theory

In this section, we develop a simple model of T-RAX, a dielectric haloscope
operating near a waveguide’s cutoff frequency. First, we show how the axion
induces a strong transverse resonance between two parallel plates. Then we
discuss how this resonating standing wave generates a traveling wave from a
shorted end. Finally, we will see how an additional longitudinal reflection
can further boost the traveling wave power.

###  2.1 Transverse resonance of parallel plates

Fig. 1 displays two infinite conducting plates at $x=0$ and $a$ in free space
with a wave impedance $Z_{0}$. A strong external DC magnetic field $B_{e}$
runs parallel to the plates ($y$-axis). The axion field $a$ couples with
$B_{e}$ and induces an electric field in the vacuum parallel to the plates.
The electric field oscillates with frequency $\nu_{a}=\frac{m_{a}}{h}$, where
$m_{a}$ is the axion mass and $h$ is the Planck constant. The electric field
has an amplitude

$E_{0}=\frac{\alpha}{2\pi\varepsilon_{0}}C_{a\gamma}B_{e}\theta\simeq
1.3\times 10^{-12}\text{\,[V/m]}\frac{B_{e}}{10\text{\,T}}|C_{a\gamma}|.$
(2.1)

Here, $\alpha$ is the fine structure constant, $\varepsilon_{0}$ is the vacuum
electric permittivity, $C_{a\gamma}$ is the model-dependent dimensionless
axion-photon coupling, and $\theta$ is the axion field. We also assume that
the axion is responsible for all dark matter density.

$E_{0}$ excites two transversely travelling waves moving forward and backward,
along the $x$-axis. They have complex amplitudes $F$ and $B$ at $x=0$ and a
transverse wave number $k_{x}$. We calculate $F$ and $B$ by applying the
following boundary conditions at the plates:

$\begin{split}F+B+E_{0}&=0\text{ at }x=0\text{, and}\\\
Fe^{-ik_{x}a}+Be^{ik_{x}a}+E_{0}&=0\text{ at }x=a.\end{split}$ (2.2)

The equations above yields

$\begin{split}F&=-\frac{1}{1+e^{-ik_{x}a}}E_{0}\text{ and}\\\
B&=-\frac{e^{-ik_{x}a}}{1+e^{-ik_{x}a}}E_{0}.\end{split}$ (2.3)

FB$x=a$$x=0$$B_{e}$$x$$z$ Figure 1:  Diagram for the transverse resonance.

The sum of $F$ and $B$ forms a standing wave. The electric field $E^{tr}_{y}$
at the center of the waveguide ($x=\frac{a}{2}$) is

$\begin{split}E^{tr}_{y}&=Fe^{-ik_{x}a/2}+Be^{ik_{x}a/2}\\\
&=-\frac{E_{0}}{1+e^{-ik_{x}a}}(e^{-ik_{x}a/2}+e^{-ik_{x}a/2})\\\
&=-\frac{2E_{0}}{e^{ik_{x}a/2}+e^{-ik_{x}a/2}}\\\
&=-\frac{E_{0}}{\cos{\frac{k_{x}a}{2}}}.\end{split}$ (2.4)

$|E^{tr}_{y}|$ diverges to infinity as $k_{x}a\to\pi$222$E_{0}$ also excites
$k_{x}a=3\pi,5\pi,...$ if the spacing between the plates is wider, _i.e._
$E_{0}$ excites TE30 and TE50 modes. Unfortunately, extracting power from
these modes is practically very challenging., at the TE10 mode cutoff
frequency. The divergence is expected because the parallel plates form a loss-
less cavity with an infinite quality factor. Unlike a cavity, however,
$-\frac{1}{\cos{\frac{k_{x}a}{2}}}\gg 1$ even when $k_{x}a$ is slightly larger
than $\pi$, or slightly above the cutoff frequency, and
$|E^{tr}_{y}|\gg|E_{0}|$. Such a _side-band resonance_ is a unique
characteristic of the axion-induced emission. $|E^{tr}_{y}|$ and the
detectable signal power increases as $k_{x}a\to\pi$. Unfortunately, operation
near the cutoff frequency also requires more precise machining and increases
conduction loss. The optimal spacing between the plates should consider these
factors.

###  2.2 Short-circuited waveguide

$E^{tr}_{y}$ induces a traveling wave from a conducting surface. We consider
the geometry in Fig. 2, yet without the reflection $\Gamma$. A metallic mirror
shorts the parallel plates at the left end ($z=0$). The parallel plates form a
transmission line with a TE10-mode loss-less propagation constant $\beta$. The
right and left traveling waves have complex amplitude $E_{R}$ and $E_{L}$ at
the center of the mirror ($x=\frac{a}{2}$, $z=0$). The boundary condition here
is333We ignore $E_{0}$ on the mirror because we are interested in the case
where $|E^{tr}_{y}|\gg|E_{0}$—. Similarly, we ignore the axion-induced
traveling wave at the interface to the load

$E_{R}+E_{L}+E^{tr}_{y}=0.$ (2.5)

Because $E_{L}=0$, $E_{R}=-E^{tr}_{y}=\frac{E_{0}}{\cos{\frac{k_{x}a}{2}}}$.
The transverse resonance formed inside the parallel plates can source an
axion-induced traveling wave stronger than $E_{0}$.

$E_{R}$$E_{L}$$E_{T}$$Z_{L}$$\Gamma$$\beta$$z=0$$z=d$

mirror

$B_{e}$$Z$$x$$z$$x=a$ Figure 2:  A simple T-RAX.

###  2.3 1D transmission line dielectric haloscope

We introduce a load with an impedance $Z_{L}$ at $z=d$, where $d$ is larger
than the free-space wavelength, and the transverse resonance is hardly
disturbed at the load. Fig. 2 shows the geometry. The reflection at the load
is

$\Gamma=\frac{Z_{L}-Z}{Z_{L}+Z},$ (2.6)

where $Z=\frac{kZ_{0}}{\beta}$ is the TE10 mode wave impedance inside the
transmission line. Eq. 2.5 is still valid, but $E_{L}$ is no longer zero.

The new boundary condition at $z=d$ is

$E_{L}e^{i\beta d}=\Gamma E_{R}e^{-i\beta d}+\Gamma E^{tr}_{y}.$ (2.7)

Solving the boundary conditions yields

$E_{R}=-\frac{1+\Gamma e^{-i\beta d}}{1+\Gamma e^{-2i\beta d}}E^{tr}_{y}.$
(2.8)

$E_{R}$ diverges to infinity when $\Gamma e^{-2i\beta d}\to-1$, _i.e._ $E_{R}$
is maximum where $2\beta d+\angle\Gamma=\pi$. For example, if $\Gamma$ is a
real minus number ($\angle\Gamma=\pi$), the system resonates at $\beta d=\pi$.
We use this relation for tuning later. The resonance is similar to a TE101
mode, but $k_{x}$ is slightly larger than the cutoff wavenumber. A larger
$|\Gamma|$ will yield a stronger longitudinal resonance and increase $E_{R}$.
The operation near the cutoff frequency increases $|\Gamma|$: an empty
waveguide’s TE mode wave impedance diverges near the cutoff frequency, while
that of a material-loaded waveguide is still close to its free space value.

A fraction of $E_{R}$ passes the reflective boundary and delivers a measurable
power to the load. The transmitted wave is

$E_{T}=(1+\Gamma)E_{R}e^{-i\beta d}=\frac{2Z_{L}}{Z_{L}+Z}E_{R}e^{-i\beta d}.$
(2.9)

The time-averaged power on the load is

$\begin{split}P_{\text{sig}}&=\frac{1}{2}\text{Re}\int_{A}E_{T}\times\frac{E_{T}^{*}}{Z_{L}}\\\
&=\frac{2R_{L}}{|Z_{L}+Z|^{2}}\int_{A}|E_{R}|^{2},\end{split}$ (2.10)

where $R_{L}$ is the resistive component of $Z_{L}$: $R_{L}=\text{Re}(Z_{L})$.
$\int_{A}$ is the integration of the TE10 mode over the $xy$ cross-section of
the waveguide. For a waveguide of width $a$ and height $b$, the integration
yields $\frac{ab}{2}$. $\int_{A}$ and $P_{\text{sig}}$ scale inversely
proportional to frequency $\nu$ ($\propto\nu^{-1}$) because the width of the
waveguide has to shrink so that we operate near the cutoff frequency. The same
is true for the example in Sec. 2.2. For comparison, pure dielectric
haloscopes use no transverse resonance, and its signal power is independent of
the frequency ($\propto\nu^{0}$).

We compare $P_{\text{sig}}$ to the axion-induced traveling wave power of a
simple infinite mirror of the same cross-sectional area $ab$:
$P_{0}=\frac{E_{0}^{2}}{2Z_{0}}ab$. We call this ratio a power boost.

$\frac{P_{\text{sig}}}{P_{0}}=\frac{2Z_{0}R_{L}}{|Z_{L}+Z|^{2}}\frac{|E_{R}|^{2}}{{E_{0}}^{2}}.$
(2.11)

Fig. 3 shows the calculated power boost from an example dielectric haloscope
that consists of an 8-mm wide rectangular waveguide and a 1-mm thick sapphire
slab. The 47.96-mm spacing between the mirror and the dielectric maximizes the
power boost at $19\,$GHz. The peak power boost here is 175,000.

Figure 3:  Calculated power boost from a simple T-RAX described in Sec. 2.3.
The system consists of a short-circuited rectangular waveguide and a 1-mm
thick sapphire slab. Table 1 lists the exact dimensions.

So far, we have modeled a simple dielectric haloscope. While the model
presented in this section is insightful and accurate for simple systems, it is
not applicable for more complicated and powerful systems. They have internal
reflections that modify wave propagation. Impedance mismatch boundaries, the
sources of additional reflections, also source additional axion-induced
traveling waves. The simple analytical model does not account for these
contributions. In the next section, we study a more complex geometry using
numerical simulations.

##  3 Simulation

In this section, we explain the detail of the T-RAX simulation. We describe
the simulation geometry and the underlying reasoning behind it. We show
reflectivity and axion signal power from simulation and discuss how to tune
the system to scan a wider mass range.

###  3.1 Geometry

Fig. 4 shows the simulation geometry. A long, thin rectangular waveguide forms
the detector body. Its two vertical walls are at a distance slightly larger
than the free-space half-wavelength, $k_{x}a>\pi$, of our 19 GHz design
frequency. A conductive mirror shorts the waveguide on the left side. Two 1-mm
thick equally-spaced dielectric slabs divide the waveguide volume along the
$z$ axis. Dielectrics can longitudinally reflect the traveling waves and
further enhance the signal power. We choose C-cut sapphires for their high
relative permittivity, low loss, mechanical strength, and availability. Our
simulations show that anisotropy of the sapphire’s permittivity hardly affects
the result. A hexahedrally-tapered waveguide connects the detector section to
a standard waveguide (WR42 in our case).

standardtaperzyx$B_{e}$waveguiderectangularwaveguide Figure 4:  Schematic
T-RAX geometry simulated in this paper, including the dielectrics (brown
color). This image is not to scale. Table 1 lists the exact simulation
dimensions.

###  3.2 Dielectric positioning

The $z$ position of the dielectrics determines the resonance frequency. The
system resonates when the reflectivity is real, i.e., when the phase
difference $\phi$ between the input and the reflected wave is zero, or
$\cos{\phi}=1$. In general, the signal power is maximal when a T-RAX forms a
parallel resonator444The simple model in Sec. 2 does not consider the
dielectric thickness.. For a fixed dielectric permittivity and thickness, we
can find the resonance using the dispersion calculation in Appendix B. The two
dielectric-separated sections form two coupled resonators that have two
resonance modes: anti-symmetric and symmetric. Only the latter couples
effectively with the axion-induced electric field. Fig. 5 shows a clear
correlation between dispersion and reflectivity for the two-dielectric system.

Figure 5:  Dispersion and reflection magnitude of the two dielectric
structure. The $y-$axis on the left shows the $\cos{\phi}$, where $\phi$ is
the phase difference $\phi$ between the input and the reflected waves off two
dielectric T-RAX structure. The resonances, or the reflectivity dips, occur at
$\cos{\phi}=1$. The horizontal pink band represents the passband. Check
Appendix B for the calculation details.

The dispersion also guides how to calculate the spacing to boost the desired
frequency. Given the waveguide geometry, dielectric thickness, and
dielectric’s relative permittivity, one can find spacings that satisfy the
standing wave condition, _i.e._ $\cos{\phi}=1$. For example, 49.108 mm vacuum
spacings between the dielectrics can enhance the signal power at 19 GHz. We
can further optimize the system, for example, by having different gap
distances, using microwave filter theory as in the case of Ref. [23]. However,
for practical purposes, we vary the spacing for tuning rather than
optimization.

###  3.3 Taper design

The taper transforms the TE10 mode of the resonator to that of the standard
waveguide. We want to minimize reflection and mode conversion in the taper. In
our example, the height (the E-plane or the $y$ axis) decreases from 100 to
4.318 mm of WR42. The width (the H-plane or the $x$ axis) increases from 8 to
10.668 mm of WR42. The taper barely converts the TE10 mode signal to TE/TM
$0m$ high order modes because the overlap among them is small. However, a
realistic taper might generate unwanted higher-order modes, and they need to
be carefully monitored in future R&D.

Reflections and mode conversions at impedance mismatches distort the booster
behavior and should be avoided. The H-plane taper changes the TE10-mode wave
impedance. In our example, the resonator’s wave impedance is four times higher
than inside the standard WR42 waveguide. The E-plane taper also adds a
reactance whose magnitude depends on the propagation constant, taper angle,
and waveguide height [24]. Not surprisingly, a longer taper with a lower
launch angle generally reflects less. We empirically find that the taper
height twice its base length insignificantly distorts the power boost peaks.
We plan to optimize the taper using nonlinear profiles [25] or a dielectric
lens in the future.

design frequency | 19 GHz
---|---
TE10 mode cutoff frequency | 18.737 GHz
dielectric | sapphire
relative permittivity | 9.4
thickness | 1 mm
loss $\tan{\delta}$ | $10^{-5}$
|  $6\times 10^{7}$ S/m
waveguide wall conductivity |  $2\times 10^{8}$ S/m
|  $1\times 10^{9}$ S/m
geometry |
rectangular waveguide width ($a$, $x$ axis) | 8 mm
rectangular waveguide height ($b$, $y$ axis) | 100 mm
distance between dielectrics | 49.108 mm
taper length | 300 mm
WR42 waveguide width | 10.668 mm
WR42 waveguide height | 4.318 mm
Table 1:  Key quantities of simulations

###  3.4 Simulation techniques

We simulate the reflectivity and axion-induced power of the proposed geometry
using COMSOL Multiphysics® and its RF module. The 3D simulation geometry is
cut symmetrically along the $x$ and $y$ axes to save simulation resources. The
reference plane for the reflectivity and the signal power is the WR42
waveguide’s TE10-mode port. The simulations include resistive loss of copper
and dielectric loss inside the sapphire. The latter is subdominant because the
field is weak around the dielectrics. To simulate the system’s response under
axion-induced electric field, we distribute the axion-induced current
$J_{a}=\frac{\alpha}{2\pi}C_{a\gamma}B_{e}\dot{\theta}$ vertically, parallel
to the $y$-axis. The current decays slowly along the $z$ axis to suppress the
$\nabla\theta$ contribution. We integrate the time-averaged Poynting vector
normal to the waveguide port. The power boost is the ratio of this integral to
the axion-induced power $P_{0}$ of the waveguide’s cross-sectional area.

###  3.5 Simulation results

Fig. 6 shows the simulation results. The reflectivity (dashed lines) and the
power boost (solid lines) peak at 18.97, 18.98, and 19 GHz. Fig. 7 shows the
vertical field map at the peaks: these frequencies indeed correspond to the
mode-conversion inside the taper, asymmetric, and symmetric resonances of the
detector. The axion-induced power is maximum for the latter case when both
resonators oscillate in phase.

Figure 6:  Reflectivity magnitude (dashed) and axion-induced power (solid)
from T-RAX simulation. Different colors represent the varying surfaces,
simulated with conductivities of $1\times 10^{9}$, $2\times 10^{8}$, and
$6\times 10^{7}$ [S/m] for high-temperature superconductor (HTS), 4K copper,
and room temperature copper.

Figure 7:  Vertical electric field inside the T-RAX at taper-conversion (top,
18.97 GHz), asymmetric-resonance (middle, 18.98 GHz), and symmetric resonance
(bottom, 19.00 GHz) modes. The blue and red color indicate opposite
directions.

The peak signal power largely depends on the resistive loss of copper.
Temperature, surface roughness, and machining precision all affect the
reflectivity and the power boost. We simulate three different surface
conductivities to represent the different surface losses: $6\times 10^{7}$
[S/m] for room temperature, $2\times 10^{8}$ [S/m] for 4 K, and $1\times
10^{9}$ [S/m] for a high-temperature superconductor (HTS) coated system.
Higher conductivity slightly shifts the peak and makes it narrower. The peak
power boost for $2\times 10^{8}$ [S/m] is about 86,000 at 19 GHz. The boost
peak is about 1 MHz wide.

The peak boost power drops at a higher frequency as the sheet resistance
$R_{s}$ increases. We simulate the T-RAX system in a wider frequency range by
scaling the default geometry. The peak power boost decreases by about
$\nu^{-0.64}$, close to the expectation of $\nu^{-\frac{2}{3}}$ from the
anomalous skin effect.

###  3.6 Tuning

As already discussed in Sec. 3.2, the resonant frequency critically depends on
the dielectric position. Moving the dielectric along the waveguide length ($z$
axis) is the most practical way to tune T-RAX because we can not easily change
the dielectric’s permittivity and thickness. The theoretical low-frequency
limit is the cutoff frequency. However, the dielectric spacing (roughly a half
guided-wavelength) rapidly diverges to infinity near the cutoff frequency, and
such a system would not fit inside a real magnet555Our simulation shows that
the elongated waveguide section can also be warped to reduce the overall
profile and fit inside a small solenoid magnet. Such geometry, unfortunately,
is even harder to realize, and we do not propose it as the default design.. In
other words, the available magnet space limits the low-frequency bound.

If we reduce the dielectric spacing, the resonant frequency will increase, and
the transverse resonance and the $P_{\text{sig}}$ will gradually decrease. The
decreasing $P_{\text{sig}}$ sets the high-frequency limit. A simple model
relates the peak boost frequency and the tuning range. From Eq. 2.4,

$\frac{{E^{tr}_{y}}^{2}}{E_{0}^{2}}=\frac{1}{\cos^{2}{\frac{k_{x}a}{2}}}=\frac{2}{1+\cos{k_{x}a}}\simeq\frac{4}{(k_{x}a-\pi)^{2}}.$
(3.1)

The last approximation is valid near the cutoff; $k_{x}a\simeq\pi$. The peak
power boost occurs at $k_{p}$ due to the geometry and the conductor loss. We
want to find $k_{d}$ slightly higher than $k_{p}$, where the signal power or
${E^{tr}_{y}}^{2}$ drops by a fraction $p$:

$p\frac{4}{(k_{p}a-\pi)^{2}}=\frac{4}{(k_{d}a-\pi)^{2}},$ (3.2)

or

$k_{d}a=\sqrt{\frac{1}{p}}(k_{p}a-\pi)+\pi.$ (3.3)

The tuning range $\Delta k$ is the difference between $k_{d}$ and $k_{p}$:

$\Delta k=k_{p}-k_{d}=(\sqrt{\frac{1}{p}}-1)\frac{k_{p}a-\pi}{a}.$ (3.4)

Notice that $\Delta k$ decreases as we operate closer to the cutoff frequency.
Operation near the cutoff frequency significantly increases $P_{\text{sig}}$
at the price of reduced bandwidth.

Numerical simulation with loss provides a more realistic tuning scenario. Fig.
8 shows how the boost frequency shifts if we move the dielectrics. The peak
power boost is about 90,000, and it drops to half around 19.4 GHz. The lower
limit is 18.86 GHz, with 70-mm spacing between the dielectrics. Again, this
limit will decrease with larger spacing, but the larger detector hardly fits
inside a real magnet. We claim a practical tuning range of 500 MHz, and the
relative tuning range $\frac{\delta\nu}{\nu}\approx$ 2.5 %. The dielectric has
to be placed with a $50\,\mu$m precision. Such control is readily feasible
with existing cryogenic stepper motors.

Figure 8:  T-RAX can be tuned over 300 MHz by moving the dielectrics. The
spacing between the dielectrics changes from 28 to 70 mm in 2 mm steps. The
cutoff frequency is 18.737 GHz.

Dielectric displacement is easy to implement and control. We can move the
dielectric via metallic rods. The rods can enter the system i) from the top,
as shown in Fig. 9, through slits on the waveguide’s H-planes, or ii) from the
back through slits on the taper. The first option is easier to implement and
can be used to move the mirror-side dielectric. TE11 mode leaks
insignificantly through the slits. The second option is harder to implement,
but the field in the taper is far weaker and poses less trouble. Dielectric
displacement requires only two actuators on the top and bottom of the
dielectric and does not deform or change the main metallic waveguide
structure.

E-planeH-planetuning rod Figure 9:  Cross-sectional view of T-RAX with its
dielectric. The tuning rod enters the system from the tuning slit on the
H-plane, moves the dielectric (brown), and changes the resonant frequency. The
gaps between the dielectric and the E-plane walls minimally disturb the signal
power.

To scan a broader range, we need to vary the waveguide width. Ref [21] already
suggested using _tuning slits_ along with quarter-wave deep corrugations to
tune the _thin shell cavity_. T-RAX can use the same slits on its waveguide
and taper. As an alternative to the corrugations, flexible conducting
materials such as bellows for foil filling the slit can also be used to
contain the energy loss via surface current. The irregular surface of the foil
does not significantly modify the electromagnetic behavior because TE10 mode
resonance relies mainly on the E-plane walls and less on the H-plane walls or
the mirror at the end. Surprisingly, we don’t need to change the dielectrics
according to the varying width. Our simulations show that gaps between the
dielectric and the E-plane walls do not significantly degrade the power boost,
as long as the dielectric is covering $\sim 10$% of the center of the
waveguide. We observe only a slight shift of the peak power boost frequency
due to the gap. By increasing the 8 mm waveguide width by 1 mm, we can easily
achieve a tuning range $>10$ %. Only the dielectric thickness limits the
tuning range. The waveguide width has to be controlled with a precision of
$\sim\lambda/2Q$, or $1\,\mu$m-level at 20 GHz. Such control is challenging
yet achieved in space telescopes [26].

##  4 Solenoid magnet application

A smaller axion dark matter detector offers several advantages. A solenoid
magnet is usually stronger than its dipole counterparts and is more readily
available from commercial suppliers. Even among the solenoid magnets, smaller
bore ones tend to have a stronger field [27]. Unfortunately, T-RAX in Fig. 4
is still relatively long ($\sim 400\,$mm) and can fit only in large solenoids.

We propose to _fold_ the taper to fit a fixed-width T-RAX inside a small
conventional solenoid magnet, as shown in the left picture of Fig. 10. The
right picture of Fig. 10 shows how a T-RAX array can simultaneously operate
inside a magnet to combine the signal power or scan different mass ranges. The
folding has to reflect or convert the modes minimally. The larger guided
wavelength near the cutoff frequency is particularly problematic [28]. An
E-plane taper can expand to a slightly larger width, e.g., 8.5 mm over the 30
mm length. A $180^{\circ}$ bend with a 10-mm bend radius follows the E-plane
taper. These components reflect -16.5 and -25 dB666Although the $180^{\circ}$
bend appears to be very reflective, many commercial waveguide H-bends
guarantee a return loss above 30 dB with an even shorter bend radius [29].,
which are subdominant to the reflection at the input of a typical amplifier.
The overall profile is about 140 mm, which fits inside a proposed 30-T 160-mm-
bore solenoid magnet [30, 27]. The taper after the bend tolerates more
deformation and can be made of a flexible waveguide for easier packaging. The
taper inside the strong magnetic field can source axion-induced emission and
distort the signal. Fortunately, the transverse resonance inside the E-plane
taper is small, and the signal power is minimally affected.

Figure 10:  A T-RAX with a folded taper (left). An array of T-RAX with folded
tapers installed inside a solenoid magnet (right). The overall profile is 200
mm.

##  5 Projected sensitivity

At the 19 GHz design frequency, T-RAX’s sensitivity to the axion-photon
coupling $C_{a\gamma}$ is

$\begin{split}C_{a\gamma}=15.1~{}&\Big{(}\frac{300\text{\,MeV}/\text{cm}^{3}}{\rho_{a}}\Big{)}^{\frac{1}{2}}\Big{(}\frac{80,000}{\beta^{2}}\Big{)}^{\frac{1}{2}}\Big{(}\frac{8\,\text{mm}\times
100\,\text{mm}}{A}\Big{)}^{\frac{1}{2}}\Big{(}\frac{10\text{\,T}}{B_{e}}\Big{)}\\\
&\times\Big{(}\frac{T_{\text{sys}}}{0.9\,\text{K}}\Big{)}^{\frac{1}{2}}\Big{(}\frac{SNR}{5}\Big{)}^{\frac{1}{2}}\Big{(}\frac{0.85}{\eta}\Big{)}^{\frac{1}{2}}\Big{(}\frac{\Delta\nu_{a}}{19~{}\text{kHz}}\Big{)}^{\frac{1}{4}}\Big{(}\frac{1~{}\text{day}}{\tau}\Big{)}^{\frac{1}{4}}.\end{split}$
(5.1)

Here, $\rho_{a}$ is the local dark matter density, $A$ is the waveguide’s
cross-sectional area, $T_{sys}$ is the system noise temperature, $SNR$ is the
signal to noise ratio, $\eta$ is the efficiency, $\Delta\nu_{a}$ is the axion
line width, and $\tau$ is the integration time.

Fig. 11 shows the possible search range with T-RAX. We scale Eq. 5.1 in the
10–100 GHz range. We assume that a heterodyne receiver at the single quantum
limit dominates $T_{\text{sys}}$. The sensitivity decreases with higher
frequency because the detector width ($\propto\nu^{-1}$) and the power boost
($\propto\nu^{-\frac{2}{3}}$) decrease while $T_{\text{sys}}$ ($\propto\nu$)
increases at high mass. The default geometry can reach $|C_{a\gamma}|=6.5$.
The sensitivity can improve with a taller detector and either the application
of a superconductor [31, 32] or the proposed strong magnet. The lower line
indicates the sensitivity of a 500 mm-tall system inside a 30-T magnetic
field. It has a minimum $|C_{a\gamma}|=1.4$.

The projected scan rate is about 1 MHz per day, counting only the net
measurement time. We need an array of T-RAX to cover the wide mass range. To
cover the whole 10–100 GHz range, we need about a hundred fixed-width T-RAX
with a 2.5 % relative bandwidth. The net measurement time is five hundred days
without considering integration, cool down, or warm-up times. To speed up the
scan, we can operate an array of T-RAX simultaneously in a magnet, especially
at the high frequency where the detector size is smaller. For comparison, 17
variable-width T-RAX (15 % relative bandwidth) can scan the same range, but
their details depend more heavily on the available technology and
implementation.

Finally, the application of advanced detection methods is promising for T-RAX
[33, 34, 35, 36]. T-RAX’s taper can focus large-signal power into a single
port. Thermal noise is negligible in this frequency range at millikelvin
temperature and only the detector’s intrinsic noise matters. As a result, a
large signal-to-noise ratio is achievable. The advanced detection methods will
significantly improve the sensitivity and the scanning speed. The details of
the application are currently under study.

Figure 11:  Projected sensitivity of T-RAX for axion-photon coupling. The
upper section corresponds to the simulated geometry with the peak power boost
around 80,000. The lower shade corresponds to the case with 500 mm height and
a 30-T magnetic field. The figure also shows limits from CAST [37], RBF-UF
[38, 39], HAYSTACK [40, 41], RADES [42], and ORGAN [43]. Image created using
AxionLimits [44].

##  6 Summary

We introduce a novel high-mass axion detector design. The detector consists of
an elongated rectangular waveguide operating near its cutoff frequency.
Dielectric layers inside the waveguide further enhance the axion-induced
electric field via longitudinal resonances. A tapered waveguide collects a
traveling wave signal that passes the dielectric layers. The structure
instruments a large area with few parasitic high-order modes. The detector has
the potential to scan the QCD-axion in a wide mass range from $40-400\,\mu$eV.
The system is small enough to fit inside a solenoid and is relatively simple
and easy to build with existing technology.

## Acknowledgments

The authors thank X. Li, A. E. Ivanov, and S. Heyminck for the helpful
discussion. The authors also thank D. Strom and A. Caldwell for revising
English.

##  Appendix A Loss calculation

This section calculates the loss inside a simple, single-dielectric T-RAX in
Sec. 2.3. The standing wave inside T-RAX is similar to a rectangular TE101 or
circular TM010 mode, except that $k_{x}$ is slightly larger than the cutoff
wavenumber. We borrow the textbook description [45] and list the relevant
parameters here. We point out that critical coupling may not necessarily
optimize the signal-to-noise ratio of T-RAX or a dielectric haloscope due to
their complex internal structure. The work to optimize the signal-to-noise
ratio is ongoing.

First, we calculate the conductor loss $P_{c}$. We assume that the field
distribution is approximately TE101, where resistive energy dissipation on the
waveguide walls dominates the loss.

$P_{c}=\frac{R_{s}E_{R}^{2}\lambda^{2}}{2\eta^{2}}\Big{(}\frac{l^{2}ab}{d^{2}}+\frac{bd}{a^{2}}+\frac{l^{2}a}{2d}+\frac{d}{2a}\Big{)}$
(A.1)

In our geometry, $a\ll d\ll b$, and the dominant loss occurs on the E-plane
walls. This simplifies the expression.

$P_{c}\approx\frac{R_{s}E_{R}^{2}\lambda^{2}}{2\eta^{2}}\frac{bd}{a^{2}}$
(A.2)

Near the TE10 cutoff, $\frac{\lambda}{a}\to\frac{1}{2}$. $\eta$ is the
intrinsic impedance of the filling material. For air, $\eta=Z_{0}\approx
377\,\Omega$. $R_{s}$ is the frequency-dependent sheet resistance of the
waveguide wall. For copper at room temperature, $R_{s}=0.035\,\Omega$ at 19
GHz.

$P_{c}\approx\frac{R_{s}}{8Z_{0}^{2}}E_{R}^{2}bd$ (A.3)

Using $P_{c}$, we calculate the unloaded quality factor $Q_{0}$. The unloaded
quality factor of a TE10l mode inside a rectangular cavity is

$\begin{split}Q_{0}&=\frac{(kad)^{3}b\eta}{2\pi^{2}R_{s}}\frac{1}{(2l^{2}a^{3}b+2bd^{3}+l^{2}a^{3}d+ad^{3})}\\\
&\approx\frac{(kad)^{3}b\eta}{2\pi^{2}R_{s}}\frac{1}{2bd^{3}}=\frac{(ka)^{3}\eta}{4\pi^{2}R_{s}}.\end{split}$
(A.4)

Near the TE10 mode cutoff, $k_{x}a\to\pi$, and

$\lim_{k_{x}a\to\pi}Q_{0}=\frac{\pi^{3}Z_{0}}{4\pi^{2}R_{s}}=\frac{\pi}{4}\frac{Z_{0}}{R_{s}}\approx
8,000.$ (A.5)

Cryogenic temperature and superconductor can significantly reduce $R_{s}$ and
increases $Q_{0}$. Increasing frequency $\nu$ decreases $Q_{0}$ as
$\nu^{-2/3}$ due to anomalous skin effect.

##  Appendix B Dielectric position calculation

The dielectric haloscope’s dispersion determines the resonance condition.
Unfortunately, wave propagation calculation through a dispersive system
quickly becomes complicated.

We use matrix formalism that relates the electric field and its derivative
along the propagation direction [46]. We prefer this formalism because all
elements are real numbers in lossless systems, and its serial concatenation is
a simple matrix multiplication of its constituent matrices. Fig. 12 shows a
wave traveling through a transmission line section with a propagation constant
$\beta$ and a physical thickness $d$. We use an electrical length
$\delta=\beta d$ for simplicity. A matrix $L_{n}$ relates a traveling wave’s
electric field $E$ and its spatial derivative $D=\frac{dE}{dz}$ at left
(subscript $n$) and right (subscript $n+1$) the transmission line section:

$\begin{pmatrix}E_{n+1}\\\ D_{n+1}\end{pmatrix}=L_{n}\begin{pmatrix}E_{n}\\\
D_{n}\end{pmatrix},$ (B.1)

where

$L_{n}=\begin{pmatrix}\cos{\delta}&\frac{\sin{\delta}}{\beta}\\\
-\beta\sin{\delta}&\cos{\delta}\end{pmatrix}.\\\ $ (B.2)

$\delta=\beta d$$\begin{pmatrix}E_{n}\\\
D_{n}\end{pmatrix}$$\begin{pmatrix}E_{n+1}\\\ D_{n+1}\end{pmatrix}$$\beta$
Figure 12:  Wave propagating through a transmission line

We express one-dielectric T-RAX as

$\begin{split}\begin{pmatrix}E_{o}\\\
D_{o}\end{pmatrix}&=L_{d}L_{v}\begin{pmatrix}0\\\ 1\end{pmatrix}\\\
&=\begin{pmatrix}\cos{\delta_{d}}&\frac{\sin{\delta_{d}}}{\beta_{d}}\\\
-\beta_{d}\sin{\delta_{d}}&\cos{\delta_{d}}\end{pmatrix}\begin{pmatrix}\cos{\delta_{v}}&\frac{\sin{\delta_{v}}}{\beta_{v}}\\\
-\beta_{v}\sin{\delta_{v}}&\cos{\delta_{v}}\end{pmatrix}\begin{pmatrix}0\\\
1\end{pmatrix}\\\
&=\begin{pmatrix}\frac{\cos{\delta_{v}}\sin{\delta_{d}}}{\beta_{d}}+\frac{\cos{\delta_{d}}\sin{\delta_{v}}}{\beta_{v}}\\\
\cos{\delta_{d}}\cos{\delta_{v}}-\frac{\beta_{d}}{\beta_{v}}\sin{\delta_{v}}\sin{\delta_{d}}\end{pmatrix}.\\\
\end{split}$ (B.3)

Subscripts $v$ and $d$ indicate vacuum and dielectric-filled waveguide
sections. The last vector, $\begin{pmatrix}0\\\ 1\end{pmatrix}$, is the
boundary condition at the metallic mirror, _i.e._ zero electric field and a
unit electric field gradient. $\begin{pmatrix}E_{o}\\\ D_{o}\end{pmatrix}$ is
the traveling wave exiting the system at the last dielectric surface.

A similar equation below describes wave propagation inside a two-dielectric
T-RAX. For simplicity, we use an identical electrical length $\delta_{v}$
among the mirror, the first dielectric, and the second dielectric.

$\begin{pmatrix}E_{o}\\\
D_{o}\end{pmatrix}=L_{d}L_{v}L_{d}L_{v}\begin{pmatrix}0\\\ 1\end{pmatrix}$
(B.4)

Resonances occur at $D_{o}=0$, _i.e._ when the one-way accrued phase is
$\pm\frac{\pi}{2}$. The total phase accrued during a round trip and a
reflection on the mirror is $2\pi$. Given fixed $\delta_{d}$, $\beta_{d}$, and
$\beta_{v}$, we want to find $\delta_{v}$ near $\pi$ that makes $D_{o}=0$.
Below is an analytic solution for the single-dielectric case:

$\delta_{v}=\tan^{-1}{\Big{(}\frac{\beta_{v}}{\beta_{d}\tan{\delta_{d}}}\Big{)}}.$
(B.5)

For the two-dielectric case, we can find the solution numerically. A second-
order Taylor expansion of $D_{o}$ with $\delta_{v}$ around $\pi$ is a good
approximation.

$\begin{split}D_{o}=&\cos{2\delta_{d}}-\frac{(3\beta_{d}^{2}+\beta_{v}^{2})\cos{\delta_{d}}\sin{\delta_{d}}}{\beta_{v}\beta_{d}}(\delta_{v}-\pi)\\\
&+\Big{(}-\cos^{2}{\delta_{d}}-\cos{2\delta_{d}}+(\frac{\beta_{d}}{\beta_{v}}\sin{\delta_{d}})^{2}\Big{)}(\delta_{v}-\pi)^{2}+\mathcal{O}(\delta_{v}-\pi)^{3}+...\end{split}$
(B.6)

Two $\delta_{v}$ satisfy $D_{0}=0$. The larger one, the symmetric resonance
case, maximizes the power boost.

## References

  * [1] Planck Collaboration, Aghanim, N., Akrami, Y., Ashdown, M., Aumont, J., Baccigalupi, C. et al., _Planck 2018 results - vi. cosmological parameters_ , _A &A_ 641 (2020) A6.
  * [2] R.D. Peccei and H.R. Quinn, _Constraints imposed by $\mathrm{CP}$ conservation in the presence of pseudoparticles_, _Phys. Rev. D_ 16 (1977) 1791.
  * [3] R.D. Peccei and H.R. Quinn, _$\mathrm{CP}$ conservation in the presence of pseudoparticles_, _Phys. Rev. Lett._ 38 (1977) 1440.
  * [4] S. Weinberg, _A new light boson?_ , _Phys. Rev. Lett._ 40 (1978) 223.
  * [5] F. Wilczek, _Problem of strong $p$ and $t$ invariance in the presence of instantons_, _Phys. Rev. Lett._ 40 (1978) 279.
  * [6] J. Preskill, M.B. Wise and F. Wilczek, _Cosmology of the invisible axion_ , _Physics Letters B_ 120 (1983) 127.
  * [7] L. Abbott and P. Sikivie, _A cosmological bound on the invisible axion_ , _Physics Letters B_ 120 (1983) 133.
  * [8] M. Dine and W. Fischler, _The not-so-harmless axion_ , _Physics Letters B_ 120 (1983) 137.
  * [9] S. Borsanyi, Z. Fodor, J. Guenther, K.H. Kampert, S.D. Katz, T. Kawanai et al., _Calculation of the axion mass based on high-temperature lattice quantum chromodynamics_ , _Nature_ 539 (2016) 69.
  * [10] M. Dine, P. Draper, L. Stephenson-Haskins and D. Xu, _Axions, Instantons, and the Lattice_ , _Phys. Rev. D_ 96 (2017) 095001 [1705.00676].
  * [11] V.B.. Klaer and G.D. Moore, _The dark-matter axion mass_ , _JCAP_ 11 (2017) 049 [1708.07521].
  * [12] M. Buschmann, J.W. Foster and B.R. Safdi, _Early-Universe Simulations of the Cosmological Axion_ , _Phys. Rev. Lett._ 124 (2020) 161103 [1906.00967].
  * [13] M. Buschmann, J.W. Foster, A. Hook, A. Peterson, D.E. Willcox, W. Zhang et al., _Dark matter from axion strings with adaptive mesh refinement_ , _Nature Communications_ 13 (2022) 1049.
  * [14] G. Ballesteros, J. Redondo, A. Ringwald and C. Tamarit, _Unifying inflation with the axion, dark matter, baryogenesis and the seesaw mechanism_ , _Phys. Rev. Lett._ 118 (2017) 071802 [1608.05414].
  * [15] P. Sikivie, _Detection rates for “invisible”-axion searches_ , _Phys. Rev. D_ 32 (1985) 2988.
  * [16] P. Sikivie, _Erratum: Detection rates for "invisible"-axion searches_ , _Phys. Rev. D_ 36 (1987) 974.
  * [17] ADMX Collaboration collaboration, _Search for invisible axion dark matter with the axion dark matter experiment_ , _Phys. Rev. Lett._ 120 (2018) 151301.
  * [18] B.M. Brubaker et al., _First results from a microwave cavity axion search at $24\text{ }\text{ }\mu\mathrm{eV}$_, _Phys. Rev. Lett._ 118 (2017) 061302.
  * [19] MADMAX Working Group collaboration, _Dielectric haloscopes: A new way to detect axion dark matter_ , _Phys. Rev. Lett._ 118 (2017) 091801.
  * [20] R. Cervantes, G. Carosi, C. Hanretty, S. Kimes, B.H. LaRoque, G. Leum et al., _Admx-orpheus first search for 70 $\mu$ev dark photon dark matter: Detailed design, operations, and analysis_, 2022.  10.48550/ARXIV.2204.09475.
  * [21] C.-L. Kuo, _Large-volume centimeter-wave cavities for axion searches_ , _Journal of Cosmology and Astroparticle Physics_ 2020 (2020) 010.
  * [22] C.-L. Kuo, _Symmetrically tuned large-volume conic shell-cavities for axion searches_ , _Journal of Cosmology and Astroparticle Physics_ 2021 (2021) 018.
  * [23] A.Á. Melcón, S.A. Cuendis, C. Cogollos, A. Díaz-Morcillo, B. Döbrich, J.D. Gallego et al., _Axion searches with microwave filters: the RADES project_ , _Journal of Cosmology and Astroparticle Physics_ 2018 (2018) 040.
  * [24] L. Lewin, _The e-plane taper junction in rectangular waveguide_ , _IEEE Transactions on Microwave Theory and Techniques_ 27 (1979) 560.
  * [25] J.L. Doane, _Parabolic tapers for overmoded waveguides_ , _International Journal of Infrared and Millimeter Waves_ 5 (1984) 737.
  * [26] R.M. Warden, _Cryogenic nano-actuator for jwst_ , in _Proc. of the 38th Aerospace Mechanisms Symposium_ , 2006.
  * [27] J. Jaeckel, G. Rybka and L. Winslow, _Axion dark matter_ , 2022.  10.48550/ARXIV.2203.14923.
  * [28] N. Marcuvitz, _Waveguide Handbook_ , Institution of Electrical Engineers, GBR (1986).
  * [29] “Waveguide bends & twists.” http://mdllab.com/wp-content/themes/divi-child/pdf/Waveguide-Bends-Twists/waveguidebends_twists.pdf.
  * [30] M.D. Bird, _Ultra-high field solenoids and axion detection_ , in _Microwave Cavities and Detectors for Axion Research_ , G. Carosi and G. Rybka, eds., (Cham), pp. 9–16, Springer International Publishing, 2020.
  * [31] D. Ahn, O. Kwon, W. Chung, W. Jang, D. Lee, J. Lee et al., _Superconducting cavity in a high magnetic field_ , 2020.
  * [32] J. Golm, S. Arguedas Cuendis, S. Calatroni, B. Dobrich, C. Cogollos, J.D. Gallego Puyol et al., _Thin film (high temperature) superconducting radiofrequency cavities for the search of axion dark matter_ , _IEEE Transactions on Applied Superconductivity_ (2022) 1.
  * [33] K.M. Backes, D.A. Palken, S.A. Kenany, B.M. Brubaker, S.B. Cahn, A. Droster et al., _A quantum enhanced search for dark matter axions_ , _Nature_ 590 (2021) 238.
  * [34] L.S. Kuzmin, A.S. Sobolev, C. Gatti, D. Di Gioacchino, N. Crescini, A. Gordeeva et al., _Single photon counter based on a josephson junction at 14 ghz for searching galactic axions_ , _IEEE Transactions on Applied Superconductivity_ 28 (2018) 1.
  * [35] A. Rettaroli, D. Alesini, D. Babusci, C. Barone, B. Buonomo, M.M. Beretta et al., _Josephson junctions as single microwave photon counters: Simulation and characterization_ , _Instruments_ 5 (2021) .
  * [36] A.V. Dixit, S. Chakram, K. He, A. Agrawal, R.K. Naik, D.I. Schuster et al., _Searching for dark matter with a superconducting qubit_ , _Phys. Rev. Lett._ 126 (2021) 141302.
  * [37] CAST collaboration, _New CAST Limit on the Axion-Photon Interaction_ , _Nature Phys._ 13 (2017) 584 [1705.02290].
  * [38] S. DePanfilis, A.C. Melissinos, B.E. Moskowitz, J.T. Rogers, Y.K. Semertzidis, W.U. Wuensch et al., _Limits on the abundance and coupling of cosmic axions at 4.5 $<{m}_{a}<$5.0 $\mu$ev_, _Phys. Rev. Lett._ 59 (1987) 839.
  * [39] C. Hagmann, P. Sikivie, N.S. Sullivan and D.B. Tanner, _Results from a search for cosmic axions_ , _Phys. Rev. D_ 42 (1990) 1297.
  * [40] HAYSTAC collaboration, _Results from phase 1 of the HAYSTAC microwave cavity axion experiment_ , _Phys. Rev. D_ 97 (2018) 092001 [1803.03690].
  * [41] HAYSTAC collaboration, _A quantum-enhanced search for dark matter axions_ , _Nature_ 590 (2021) 238 [2008.01853].
  * [42] CAST collaboration, _First results of the CAST-RADES haloscope search for axions at 34.67 $\mu$eV_, 2104.13798.
  * [43] B.T. McAllister, G. Flower, E.N. Ivanov, M. Goryachev, J. Bourhill and M.E. Tobar, _The organ experiment: An axion haloscope above 15 ghz_ , _Physics of the Dark Universe_ 18 (2017) 67.
  * [44] C. O’Hare, _cajohare/axionlimits: Axionlimits_ , July, 2020.  10.5281/zenodo.3932430.
  * [45] D.M. Pozar, _Microwave engineering; 3rd ed._ , Wiley, Hoboken, NJ (2005).
  * [46] J. Lekner, _Matrix and numerical methods_ , in _Theory of Reflection of Electromagnetic and Particle Waves_ , pp. 224–228, Springer (1987), https://link.springer.com/book/10.1007/978-3-319-23627-8.


#  DemoCaricature: Democratising Caricature Generation with a Rough Sketch

Dar-Yen Chen Subhadeep Koley Aneeshan Sain Pinaki Nath Chowdhury
Tao Xiang Ayan Kumar Bhunia Yi-Zhe Song
SketchX, CVSSP, University of Surrey, United Kingdom.
{s.koley, a.sain, p.chowdhury, t.xiang, a.bhunia<EMAIL_ADDRESS>
https://democaricature.github.io

###### Abstract

In this paper, we democratise caricature generation, empowering individuals to
effortlessly craft personalised caricatures with just a photo and a conceptual
sketch. Our objective is to strike a delicate balance between abstraction and
identity, while preserving the creativity and subjectivity inherent in a
sketch. To achieve this, we present Explicit Rank-1 Model Editing alongside
single-image personalisation, selectively applying nuanced edits to cross-
attention layers for a seamless merge of identity and style. Additionally, we
propose Random Mask Reconstruction to enhance robustness, directing the model
to focus on distinctive identity and style features. Crucially, our aim is not
to replace artists but to eliminate accessibility barriers, allowing
enthusiasts to engage in the artistry.

##  1 Introduction

Ever wondered when you would finally decide to get that personalised
caricature created, perhaps during a holiday? Look no further, this paper is
for you – we strive to democratise caricature [3, 23, 24] generation for
everyone! With a portrait of yourself and a conceptual sketch of how you
envision your caricature, we will automatically generate a high-fidelity
caricature that unmistakably captures your essence [24]. Our aim however is
not to replace artists; after all, the realm of art may be one that AI will
never entirely conquer – so when you find yourself in Paris next, do get your
caricature expertly crafted by a skilled artist!

We commence our study by asking the fundamental question that arises when
scrutinising a caricature – Is this me? (or Obama or Mr. Bean for that matter
in LABEL:fig:teaser?) Indeed, the core challenge in caricature generation is
navigating the delicate balance of infusing abstraction [17] into the process
to achieve that distinctive caricature appearance, while still preserving the
essential identity cues that unmistakably represent the intended person [7].
Over and above all, how do we seamlessly inject your individuality [5] and
creativity [2] into the art generation process, ensuring the resulting
caricature is genuinely your own, rather than one dictated solely by AI?

Our solution lies in your sketch! A single rough sketch is all it takes to
encapsulate your vision for your caricature, as illustrated in
LABEL:fig:teaser. The scientific challenge is clear: regardless of your
artistic skill or the lack of it, how can we design a system to adeptly
generate a plausible caricature while still preserving your identity [24]? And
one more thing, if there is a specific caricature style you prefer, we would
like to accommodate that preference as well.

We most certainly are not pioneers in caricature generation [3, 24, 23]; our
motivation primarily draws from prior art in this field. However, our set of
challenges notably surpasses the technical capabilities of previous systems
[24, 3], particularly those primarily deformation-based [50, 13], which tend
to prioritise style creation over identity. Crucially, these prior systems
often fall short in including “you” in the solution. This deficiency results
in generated caricatures lacking expressiveness and missing interesting
features like local abstraction [17], hairstyle variation [59], and view
changes – all of which can be easily injected into our system with just your
single sketch!

Our approach to modelling the delicate balance between identity and style
relies on the interaction between a novel single-image Text-to-Image (T2I)
personalisation module and a sketch-specific T2I-Adapter [34]. The former
ensures identity, while the latter allows for sketch-controlled caricature
generation. This, of course, is not trivial. Latest single-image T2I
personalisation approaches [10, 43, 54] often grapple with overfitting during
single-image fine-tuning, resulting in a highly specialised yet inflexible
model that lacks generalisation beyond training data. This makes them
especially challenging for them to adapt to the highly exaggerated and
subjective human sketches, which are often Out-of-Distribution (OOD). This
challenge is further exacerbated, as we face the task of merging concepts of
identity and style. If done blindly, this would lead to a blending of
features, resulting in caricatures that lack distinction or skew towards one
aspect at expense of the other.

We thus propose Explicit Rank-1 Model Editing for single-sketch
personalisation, enabling effective learning and the fusion of identity and
style. By incorporating an explicit mechanism, it independently manipulates
the explicit editing of identity and style in the cross-attention layers [34],
with minimal extra parameters while maintaining the integrity of textual
contexts. This provides a more subjective and fine-grained control over
desired concepts, mitigating the overfitting typically encountered.
Furthermore, we introduce Random Mask Reconstruction to enhance the robustness
of distorted shapes. It achieves this by masking random patches of the input
image, compelling the model to focus on crucial identity and style features
over local variations. This capability importantly allows the model to better
handle exaggerated caricature sketches while emphasising the essential learned
features.

Our contributions are: (i) we democratise caricature generation, enabling
individuals to easily create personalised caricatures, from a photo and a
conceptual sketch. (ii) we address the delicate balance between abstraction
and identity via Explicit Rank-1 Editing, offering nuanced control by
selectively applying rank-1 edits to cross-attention layers. (iii) we enhance
system robustness with Random Mask Reconstruction, enabling effective handling
of distorted shapes while emphasising essential identity and style.

##  2 Related Work

Deep Caricature Synthesis:  Caricature synthesis aims to exaggerate or distort
specific facial features for a stylised yet recognisable portrayal of a
subject [11, 13, 1]. Such methods typically involve a deformation stage,
followed by image-to-image translation. Introduced as a GAN [12]-based
framework [3] involving facial landmarks to guide deformations, it was
enhanced by automating control point prediction for warping and embedding a
discriminator, acting as an identity classifier to help in its preservation
[50]. A few subsequent works include diversifying caricature generation to
multiple facial exaggeration types [13], leveraging SENet [22] and spatial
transformer modules to produce high-fidelity warps based on dense warping
field [11], and leveraging StyleGAN [25] with GAN inversion [37, 53] to
propose shape exaggeration blocks for additional control [24]. Towards spatial
manipulation within caricature synthesis, while Semantics CariGAN [6]
leveraged semantic shape transformations for caricature-control from warped
semantic maps, a segmentation-guided dual-domain synthesis framework [1]
combined few-shot GAN [40] with RepurposingGAN [55]. Addressing the
limitations of the deformation-based pipeline, we strive to enhance creative
freedom in caricature synthesis via freehand sketches.

Denoising Diffusion Probabilistic Models (DDPM): Recently, DDPMs [19] have
emerged as the de facto choice for generative modelling, thanks to their high-
fidelity image synthesis potential. Earlier works [19, 18, 35, 41] have
significantly improved text-to-image (T2I) models, such as Imagen [46],
DALL-E2 [39], and Stable Diffusion (SD) [41] – further enhanced by training on
diverse image-caption pair datasets [47, 48]. Harnessing the prior knowledge
of pretrained T2I models, research progressed to guide generation under
additional conditions [64, 62, 61]. For instance, ControlNet [63] and
T2I-Adapter [34] introduced content semantics adapters for targeted tasks such
as pose, depth map, and sketch-conditional synthesis [58], which enhances the
flexibility of the generation process.

T2I Personalisation: With a limited set of reference images, T2I
personalisation aims to adapt pretrained T2I models [46, 41] to specific
concepts, while retaining its generalisability. Among the proposed strategies
[57, 51, 9], while Textual Inversion [10] optimises text embeddings to capture
new concepts, DreamBooth [43] personalises the output by fine-tuning the whole
Stable Diffusion [41] and Imagen [46] models. Research on Parameter-Efficient
Fine-Tuning (PEFT) methods [15], such as LoRA [21, 45] and SVDiff [14],
focuses on reducing the computational burden during model training.
Additionally, CustomDiffusion [30] fine-tunes only cross-attention layers,
while Perfusion [54] introduces Rank-1 Model Editing (ROME) [33] to optimise
the Value-pathway in the cross-attention mechanism. InstantBooth [49] enables
personalised inference with single images. FastComposer [60] uses a novel
image encoder for concept embeddings, while HyperDreamBooth [44] achieves
efficient fine-tuning with a hypernetwork. However, resource-intensive
training may limit their application [49, 60]. We thus offer a rapid and
universal single-image method, that extends personalisation beyond individual
identity images to include reference style images, facilitating synthesis of
stylised caricatures, while costing minimal iterations and parameter overhead.

##  3 Revisiting Text-to-Image Diffusion Models

Overview: Diffusion models [41, 8, 52], rely on two stochastic processes,
termed as forward and backward diffusion [19]. The forward process involves
iteratively adding Gaussian noise to a clean image
${x}_{0}\in\mathbb{R}^{h\times w\times 3}$ over $t$ time-steps, producing a
noisy image ${x}_{t}\in\mathbb{R}^{h\times w\times 3}$ as:
${x}_{t}=\sqrt{\bar{\alpha}_{t}}~{}{x}_{0}+(\sqrt{1-\bar{\alpha}_{t}})\epsilon$,
where, $\epsilon\sim\mathcal{N}(0,\mathbf{I})$ is the added noise,
$\\{\alpha_{t}\\}_{1}^{T}$ represents a predefined noise schedule [19] with
$\alpha_{t}=\prod_{s=1}^{T}\alpha_{s}$ and time-step $t$ value is sampled from
a Uniform distribution $t\sim U(0,T)$. With sufficiently large $T$, $x_{T}$
approximates isotropic Gaussian noise. The backward process entails training a
modified UNet [42] denoiser $F_{\theta}(\cdot)$. It takes the noisy input
${x}_{t}$ and the corresponding time-step $t$ to estimate the input noise
$\epsilon_{t}\approx F_{\theta}({x}_{t},t)$. Once trained with a standard MSE
loss [19], $F_{\theta}$ can reverse the effect of forward diffusion. During
inference, $F_{\theta}$ is applied iteratively for $T$ time-steps on a
randomly sampled 2D Gaussian noise image ${x}_{T}$ to get a cleaner image
${x}_{t-1}$ at each time-step $t$, thus eventually resulting in one of the
cleanest images ${x}_{0}$ resembling the original target distribution [19]

Text-Conditioned Diffusion Model: Diffusion models can generate images
conditioned on different signals (e.g., class labels [20], textual prompts
[41, 39], etc.). Given a textual prompt $p$, the initial step involves its
conversion to the word-embedding space $\mathcal{W}$ on applying a word-
embedding function $\mathbf{W}$. Subsequently, the transformed prompt is
passed through a CLIP [38] text encoder denoted by $\mathbf{T}(\cdot)$, which
produces the text encoding as
$\mathbf{t_{p}}=\mathbf{T}(\mathbf{W}(p))\in\mathbb{R}^{77\times d}$ in the
text encoding space $\mathcal{T}$. This $\mathbf{t_{p}}$ controls the
diffusion process via cross-attention, thus allowing
$F_{\theta}(x_{t},t,\mathbf{t_{p}})$ to perform $p$ controlled denoising on
$x_{t}$.

Stable Diffusion: Latent Diffusion Models (_i.e_., Stable Diffusion) [41]
perform forward and backward denoising in the latent space for [41]. In its
two-stage approach, Stable Diffusion (SD) [41] first trains a variational
autoencoder (VAE) [28], comprising an encoder $E(\cdot)$ and a decoder
$D(\cdot)$ in sequence. $E(\cdot)$ converts the input image
$x_{0}\in\mathbb{R}^{h\times w\times c}$ to its latent representation
$z_{0}\in\mathbb{R}^{{\frac{h}{8}}\times{\frac{w}{8}}\times d}$ [41]. The
forward process adds Gaussian noise to ${z}_{0}$ over $t$ time-steps,
producing a noisy latent
${z}_{t}=\sqrt{\bar{\alpha}_{t}}~{}{z}_{0}+(\sqrt{1-\bar{\alpha}_{t}})\epsilon$.
Later, a UNet [42] denoiser $\epsilon_{\theta}(\cdot)$ is trained to perform
conditional denoising based on textual prompt $p$ directly in the latent space
[41] with loss objective as:

$\mathcal{L}_{sd}=\mathbb{E}_{z_{t},t,\epsilon,p}(||\epsilon-\epsilon_{\theta}(z_{t},t,\mathbf{t_{p}})||_{2}^{2})\vspace{-0.2cm}$
(1)

SD incorporates the text conditioning using $\mathbf{t_{p}}$ into the
denoising process via cross-attention [41] as:

$\small\left\\{\begin{matrix}Q=W_{Q}z_{t};~{}K=W_{K}\mathbf{t_{p}};~{}V=W_{V}\mathbf{t_{p}}\\\
\mathtt{Attention}(Q,K,V)=\mathtt{SoftMax}(\frac{QK^{T}}{\sqrt{d}})\cdot
V\end{matrix}\right.\vspace{-0.2cm}$ (2)

where $W_{Q}$, $W_{K}$, and $W_{V}$ are the learnable projection matrices.
$W_{K}$ and $W_{V}$ linearly projects the text ecoding
$\mathbf{t_{p}}\in\mathbb{R}^{77\times 768}$ to form the “Key” and “Value”
vectors. Whereas, $W_{Q}$ projects the intermediate noisy latents to form the
“Query” maps [41]. The cross-attention map is produced as
$\mathtt{SoftMax}(\frac{QK^{T}}{\sqrt{d}})\cdot V$. Essentially, the cross-
attention map indicates the correspondence between the textual prompt and the
spatial regions of the image [41].

T2I-Adapter: Moving beyond conventional textual conditioning [41], T2I-Adapter
[34] enables a myriad of different spatial conditioning signals (_e.g_.,
segmentation masks, scribbles, sketches, key pose, depth maps, colour palates,
etc., or their weighted combinations) to guide the T2I image generation
process of SD [41]. In practice, T2I-adapter [34] trains a lightweight network
(comprising one convolutional and four residual blocks) that extracts deep
features from spatial conditioning signals at four different scales. Those
extracted conditioning-features are then added with intermediate features of
SD’s UNet decoder at each scale [34] to influence denoising with the given
condition [34].

##  4 Problem Definition and Challenges

Given a reference portrait photo $\mathcal{I}_{p}$ depicting a specific
identity $p$ and a free-hand abstract sketch $\mathcal{S}$ as the query, we
aim to generate a caricature $\mathcal{C}_{p}^{s}$, which should retain the
identity [7] captured in $\mathcal{I}_{p}$, while reflecting the sketch’s
($\mathcal{S}$) influence on its shape. Notably, $\mathcal{I}_{p}$ may
represent an individual not encountered previously, and the free-hand sketch
$\mathcal{S}$ may depict random or highly unconstrained deformations [3] or
shape exaggerations [50], to be reflected in $\mathcal{C}_{p}^{s}$.

Complexity stems from the delicate balance between preserving identity [7] and
introducing sketch-guided shape deformations [6]. Learning the unique identity
from a single reference image is non-trivial, given the risk of overfitting
[10, 43] on limited data. Adding to it is the complexity of generating an
exaggerated shape in accordance with the sketch. Addressing these challenges
requires a robust model capable of learning and generalising [54] from a
single reference image while optimising the trade-off between identity
preservation and shape exaggeration.

Figure 1: Within cross-attention layers, Explicit ROME (Sec. 5.2) edits the
concept entry with trainable target output $\mathbf{o}^{*}$ that encapsulates
the identity features. We also employ a dynamic masking method (Sec. 5.3),
selectively occluding latent regions during training to enhance model
robustness. Additional regularisation (Sec. 5.4) is applied to word embeddings
and text encoding through superclass. During inference, a frozen T2I-sketch-
adapter [34] provides shape guidance, resulting in an output caricature with
the desired identity and shape. A similar training pipeline is used for the
style image as well. We use Eq. 4 to perform sketch+style guided caricature
generation.

##  5 Sketch for Caricature Generation

Overview: Our approach diverges from the prevalent use of pre-trained
StyleGAN’s [25, 26, 27] latent space in facial image editing [36] tasks.
Instead, we opt for a pre-trained text-to-image stable diffusion (SD) model
[41], known for its generalisation [31] and adaptability [10] across diverse
and wild scenarios. Our problem being inherently multi-modal [52], where
$\mathcal{I}_{p}$ is a real photo, $\mathcal{S}$ is a black-and-white sparse
abstract [17] line drawing, and the output caricature ($\mathcal{C}^{s}_{p}$)
typically extends beyond real photo modality, SD [41] becomes an ideal fit as
it excels in handling such scenarios which are less encountered [52] in real
life.

Our personalised text-to-image (T2I) framework involves fine-tuning the SD
model to capture identity in the reference photo $\mathcal{I}_{p}$ and
generate the same identity in various contexts [10]. Consequently, we leverage
an off-the-shelf T2I-Sketch-Adapter [34] to spatially condition the identity-
adapted SD model. This process effectively integrates shape guidance from
sketch, aligning $\mathcal{C}^{s}_{p}$ with the intended shape.

Our caricature generation pipeline extends further to include style [24]
adaptation, by acquiring low-level style features from a single style-
reference image $\mathcal{I}g$ characterised by a specific style $g$. The
resulting output caricature $\mathcal{C}_{p|g}^{s}$, now concurrently
preserves the identity, style, and shape derived from $\mathcal{I}_{p}$,
$\mathcal{I}_{g}$, and $\mathcal{S}$, respectively.

###  5.1 Baseline off-the-shelf Solutions

Given the recent rise of personalised T2I frameworks [46, 41, 43], one can
naively finetune it using a single reference identity image, and further
generate a sketch-conditioned shape-exaggerated output caricature plugging an
off-the-shelf T2I-Sketch-Adapter [10]. Among such frameworks, Textual
Inversion [10] aims to learn a new pseudo word embedding $\mathbf{v}_{*}$
(representing the concept) in $\mathcal{W}$ space by directly optimising the
LDM loss as in Eq. 1 against reference images. Whereas, Perfusion [54] further
adjusts visual representations through ROME [33], modifying the Value-pathway
activation according to the component of all words that are aligned with the
target concept.

Such a naive solution would however suffer from a few challenges. Firstly,
training from a single reference identity ($\mathcal{I}_{p}$) or style
($\mathcal{I}_{g}$) image easily leads to overfitting [54] in word embeddings,
thereby compromising on generalisability to multiple contexts. Secondly,
integrating homologous concepts like identity and style, encounters a
substantial degree of semantic overlap [54]. This results in detrimental
interference (see Fig. 4), causing the concepts to overshadow each other in
sketch+style guided caricature generation. Lastly, being trained on a single
reference image only, it fails to generalise towards imbibing the exaggerated
[50] shape guidance from diverse sketches [17].

Accordingly, we propose three key solutions: (i) _Explicit_ Rank-1 Model
Editing (Sec. 5.2), that edits only at the concept index. Besides preventing
potential interference, it also refines the optimisation scope, rendering the
adaptation process more effective. Secondly, we implement Random Mask
Reconstruction (Sec. 5.3) to enable training with locally masked images,
directing the model’s focus away from local variations and emphasising on key
features. This enhances the model’s resilience to diverse facial shape
constraints [29] crucial for caricature synthesis. Thirdly, we incorporate
additional regularisation (Sec. 5.4) using superclass on word embeddings and
text encoding, to counter overfitting, which ensures the model’s attention
mechanism remains less burdened by the identity [54], allowing more free-form
shape exaggeration, while preserving identity.

###  5.2 Explicit Rank-1 Model Editing

Rank-1 Model Editing (ROME) [33] in NLP considers transformer [56] feed-
forward layers as memory storage. It utilises learnable outputs to edit this
memory, aligning it with the target concept. ROME differentially modifies only
the knowledge related to the target, preserving rest of the pre-trained
model’s memory completely. In T2I [34], textual context is integrated via
cross-attention layers, using _‘Key’_ and _‘Value’_ pathways akin to feed-
forward layers in transformers [54]. Our contribution, Explicit Rank-1 Model
Editing (Explicit ROME), refines T2I models by applying modifications to the
textual encoding locally, specifically _at the position of the concept index_
while _leaving_ other textual contexts untouched.

Given a reference identity photo $\mathcal{I}_{p}$ and a textual prompt $p$ =
‘a photo of a P∗’, we convert $p$ to a series of word embedding vectors
$p_{w}$ through word-embedding layer $\mathbf{W}$ where the word embedding
corresponding to _concept token_ P∗ is replaced with a learnable pesudo word
embedding vector $\mathbf{v^{*}}\in\mathbb{R}^{768}$ for SD v1.5 [41]. It is
initialised from the word embedding of its corresponding superclass word, like
‘man’ or ‘woman’ based on the gender of the identity photo $\mathcal{I}_{p}$.
Position of the concept token is denoted as $\mathtt{c_{i}}$. Next, we use a
CLIP-text encoder to obtain textual encoding (in $\mathcal{T}$ space) as
$\mathbf{t_{p}}=\mathbf{T}(p_{w})\in\mathbb{R}^{77\times 768}$. This
$\mathbf{t_{p}}$ influences the intermediate feature map of SD-UNet through
‘Key’ and ‘Value’ pathways which we edit via Explicit Rank-1 Model Editing in
the next stage.

Considering $W\in\mathbb{R}^{320\times 768}$ (for SD v1.5 [41]) from Eq. 2 as
the embedding matrix for either _‘Key’_ as $W_{K}$ or _‘Value’_ as $W_{V}$ and
$\mathbf{t_{p}}$ as the textual encoding, the standard output $h=Wt_{p}$ is
edited by _Explicit ROME_ as:

$h[\mathrm{\mathtt{c_{i}}}]\leftarrow
h[\mathrm{\mathtt{c_{i}}}]+s\cdot\Phi(\mathbf{t_{p}}[\mathrm{\mathtt{c_{i}}}],i^{*})\cdot\
\mathbf{o}^{*}\vspace{-1mm}$ (3)

where $\Phi(\cdot,\cdot)$ is the cosine similarity function and
$\mathbf{o}^{*}$ is a learnable vector of size $\mathbb{R}^{320}$ (for SD v1.5
[41]). The target input $i^{*}$ is initialised from CLIP [38] text encoding
$\mathbf{t_{p}}$ at $\mathtt{c_{i}}$ index, and at every step is updated
through the exponential moving average [54] as $i^{*}\leftarrow 0.98\cdot
i^{*}+\mathbf{t_{p}}[\mathrm{\mathtt{c_{i}}}]$. The input $i^{*}$ serves as a
prototype for gauging the alignment of various contexts with the learned
identity. The scale $s$ allows modulating the degree of personalisation during
inference, offering more control over results. The similarity
$\Phi(\mathbf{t_{p}}[\mathrm{\mathtt{c_{i}}}],i^{*})$ represents how closely
the input matches $i^{*}$, after the interaction of learnable pseudo word
embedding $\mathbf{v^{*}}$ and other word embeddings. It can adjust what level
of identity features in the caricature should be embedded depending on various
contexts (freehand sketch in our case). Instead of complex Mahalanobis
distance [54] based formulation we utilise the cosine distance to calculate
the similarity, which is an intuitive and effective choice according to the
text encoder CLIP [38]. In all the cross-attention layers’ _‘Key’_ and
_‘Value’_ pathways, we explicitly apply Explicit ROME as in Eq. 3 only at the
index position of the concept token $\mathtt{c_{i}}$. This aligns visual
features with the target concept, therefore preserving other textual contexts,
and consequently ensuring generalisability without compromise.

Similar to adapting to a reference identity photo $\mathcal{I}_{p}$, one can
adapt it for a specific style-image $\mathcal{I}_{g}$ as well, taking
superclass word for style images as ‘comics’, ‘illustration’ etc. In
particular, Eq. 3 can be extended to combine multiple independently trained
concepts as follows:

$h[\mathrm{\mathtt{c_{i}}}]\leftarrow
h[\mathrm{\mathtt{c_{i}}}]+\sum_{j=1}^{J}s_{j}\cdot\Phi(\mathbf{t_{p}}[\mathrm{\mathtt{c_{i}}}],i^{*}_{j})\cdot\mathbf{o}^{*}_{j}\vspace{-1mm}$
(4)

This equation independently treats each concept at its respective
$j^{\text{th}}$ index, preserving unique elements without unintentional
blending. This ensures easier integration of multiple concepts in caricature
synthesis [54], addressing the challenge of blending homologous identity and
style [24].

To sum up, our method has the following trainable parameters: (i) a single
pseudo word embedding $\mathbf{v^{*}}\in\mathbb{R}^{768}$. (ii) the tuple
$\\{\mathbf{o_{K}^{*}},\mathbf{o_{V}^{*}}\\}^{l}$ at each cross-attention
layer $l$ for _‘Key’_ and _‘Value’_ pathway respectively. Every
$\mathbf{o}^{*}$ has a dimension of 320, thus making our Explicit ROME overall
$30\times$ lesser learnable parameters than Perfusion [54].

###  5.3 Random Mask Reconstruction

One of the major challenges of caricature synthesis is recreation of the
reference-style [24], while maintaining the subject’s unique identity [3, 24,
54, 23]. To ensure the seamless reproduction of style and identity in the
output caricature, we introduce random mask reconstruction (RMR) loss. We
hypothesise that random masking of the reference images would shift the
model’s focus from local spatial regions, enforcing it to understand the
global concepts (i.e., style and identity). Given a random masked image, we
pass it through the encoder $E(\cdot)$, to obtain a masked latent image
$z_{0}^{m}$ which after forward diffusion becomes $z_{t}^{m}$. This upon
passing through UNet-denoiser $\epsilon_{\theta}$ conditioned on
$\mathbf{t_{p}}$, the modified SD objective becomes:

$\mathcal{L}^{\mathrm{mask}}_{\text{sd}}=\mathbb{E}_{\mathbf{z}_{t},t,\mathbf{t_{p}},\epsilon}(||(\epsilon-\epsilon_{\theta}(\mathbf{z}_{t}^{m},t,\mathbf{t_{p}}))\odot
M||^{2}_{2})\vspace{-0.1cm}$ (5)

where $M$ is the equivalent latent space binary mask with size same as
$z_{t}$. It is used to impose $\mathcal{L}^{\mathrm{mask}}_{\text{sd}}$ on the
unmasked areas only. In practice, we obtain $M$ via bilinear downscaling from
a randomly sampled mask [16] in the pixel space.

Figure 2: Qualitative comparison with GAN-based deformation models. These
visual results illustrate our method’s higher fidelity and shape flexibility
in caricature synthesis compared to existing method viz. StyleCariGAN [24],
CariGANs [3], and WarpGAN [50].

###  5.4 Concept Regularisation

Any marked deviation of the concept word embedding $\mathbf{v^{*}}$, risks
dominating of the text encoder and attention mechanism by the concept, thus
losing generalisability to sketch-based deformations. We thus apply $l_{2}$
regularisation [10] on the concept word embedding against its superclass word
embedding $S_{c}^{w}$ in $\mathcal{W}$ space to prevent overfitting of the
text encoder. Furthermore, we impose cosine distance-based regularisation loss
between text encodings $\mathbf{t_{p}}$ (using $\mathbf{v^{*}}$) and
$\mathbf{t_{p}^{s_{c}}}$ (using $S_{c}^{w}$) from CLIP textual encoder
$\mathbf{T}$, at the position of concept token $\mathtt{c_{i}}$. Therefore,
the regularisation losses in $\mathcal{W}$ and $\mathcal{T}$ spaces become:

$\begin{split}\mathcal{L}_{\text{reg}}^{\mathcal{W}}=l_{2}(\mathbf{v^{*}},S_{c}^{w})\;;\;\mathcal{L}_{\text{reg}}^{\mathcal{T}}=1-\Phi(\mathbf{t_{p}}[\mathtt{c_{i}}],\mathbf{t_{p}^{s_{c}}}[\mathtt{c_{i}}])\end{split}\vspace{-0.4cm}$
(6)

Finally, the overall training loss becomes
$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{sd}}^{mask}+\lambda_{1}\mathcal{L}_{\text{reg}}^{\mathcal{W}}+\lambda_{2}\mathcal{L}_{\text{reg}}^{\mathcal{T}}$.
Please see Fig. 1 for a summarised overview of training and inference
pipelines.

Figure 3: Comparison with T2I personalisation approaches. Our framework is
stronger in single-image personalisation caricature synthesis against
Perfusion [54] and TI [10].

##  6 Experiments

Datasets. We use the WebCaricature dataset [23] to source identities and
styles. To validate our approach via a quantitative comparison and a user
study, we curate a test dataset encompassing 20 identities, 4 styles, and 12
distinctive edge maps as shapes. These edge maps are extracted from caricature
images of WebCaricature [23], leading to 960 unique caricature pairs for
evaluation. For a fair assessment of our method, the carefully selected
identities encompass a wide-spectrum of race, gender, and age, thereby
upholding diversity and inclusiveness in our evaluation. Analysing qualitative
results, we incorporate amateur freehand sketches, incorporating real user
interpretation into the assessment.

Implementation Details. Our implementation is based on Stable Diffusion v1.5
[41]. We train using AdamW [32] optimiser, with a batch size of $16$, learning
rates $0.2$ and $0.002$ for target outputs and embeddings respectively. Fine-
tuning consists of $40$ and $100$ steps for identities and styles,
respectively. We conduct all experiments on a single NVIDIA GTX $4090$ GPU,
taking $1$ minute for identity and $2$ minutes for style fine-tuning. For
inference, results are sampled with 50 steps along with a classifier-free
guidance [18] scale of $9$. We use the prompts
$\mathtt{``a~{}caricature~{}of~{}[id*]"}$ and
$\mathtt{``a~{}caricature~{}of~{}[id*]~{}in~{}the~{}style~{}of~{}[style*]"}$
to generate caricatures.

Figure 4: Comparison with T2I personalisation approaches with style reference.
Demonstrates our model’s robustness in generating stylised caricatures with
faithful identity and style, surpassing other methods like Perfusion [54] and
TI [10].

###  6.1 Qualitative Evaluation

LABEL:fig:teaser shows the efficacy of our proposed method in generating
caricatures while faithfully conforming to specifications of identity [23],
style [24], and sketch shape [17]. Given a subject, our method demonstrates
its robust caricature synthesis [24] potential in the above half of
LABEL:fig:teaser. It moves beyond the traditional confines of feature scaling
[50, 3, 24] to a paradigm where features can be adjusted and exaggerated with
ease of using sketch-based guidance. Such flexibility reaches into the domain
of fine-grained facial feature manipulation [4] adjusting shape, features
(mouth, ears, nose), expressions, as well as hairstyles, while also attending
to accessories and novel perspectives. From simple one-stroke outlines to
intricate details, our model demonstrates adaptability to varying sketch
complexities. Remarkably, it achieves this without reliance on identity-
tailored components [44], capturing the subtle essence of human faces from
merely a single reference image with only a few fine-tuning steps. Our
framework addresses the challenge of identity preservation while applying
exaggeration and distortion, exemplifying a robust resistance to overfitting.
When constrained by a sketch, the model seamlessly integrates identity into
the shape, ensuring recognisability without apparent visual artefacts, while
maintaining the prior knowledge of the SD [41] model.

Lower half of LABEL:fig:teaser illustrates our model’s ability to harmonise
two conflicting concepts: identity and style, each derived from separate human
likenesses. The objective is to unify them within a single synthesised
caricature face. Diffusion backbones [41] usually struggle with such duality,
yet our model overcomes this, rendering caricatures with high fidelity to both
identity and style elements.

###  6.2 Comparison with SOTA

We benchmark our caricature synthesis against three state-of-the-art (SOTA)
deformation-based models viz. StyleCariGAN [24], CariGANs [3], and WarpGAN
[50]. These models however do not support caricature synthesis with combined
conditioning on identity [3], style [24], and shape [23] like ours. We extend
our comparison to advanced SD-based [41] personalisation models, like Textual
Inversion (TI) [10] and Perfusion [54] as well.

CariGANs [3] and WarpGAN [50] which rely on landmarks and control point
manipulation, clearly show distortions and artefacts in Fig. 2. By leveraging
deep feature-map modulation from StyleGAN [25], StyleCariGAN [24] delivers
higher-fidelity caricatures, yet it is limited to pre-defined scale-based
exaggeration [24], ignoring shape information. On the other hand TI [10] and
Perfusion [54] fail to preserve identity in caricatures due to overfitting
caused by single-image personalisation (Fig. 3). Furthermore, lacking an
effective interaction-control mechanism, they suffer from (Fig. 4) identity
and style ambiguity, thus deviating from corresponding references. Our
Explicit ROME strategy circumvents these pitfalls, ensuring targeted editing
at corresponding positions without disrupting other text and concept encodings
in the cross-attention mechanism [34], as verified by our superior qualitative
results.

Figure 5: Identity Scale Adaptability. Our method provides a dynamic
adjustment of the identity scale $s$, exemplifying flexibility. Figure 6:
Qualitative results of our ablation study

Now, for quantitative evaluation (Tab. 1) , we use CLIP-Score [38] on ID,
Style, and Shape. It measures ID/style-fidelity as the similarity between
generated caricatures and ID/style images using a pre-trained CLIP [38]
encoder, and shape fidelity as the same between edgemaps of generated
caricatures and conditioning sketches. Notably, at the same level of shape
similarity, our results have the highest identity and style similarity at
$0.671$ and $0.576$, which is $3\times$ and $10\times$ faster than Perfusion
[54] and TI [10] respectively. Notably, this was achieved within three minutes
of fine-tuning for identity and style.

Table 1: Quantitative comparison. Quantitative metrics of various approaches and our framework ablative design, reflecting the precise quantitative edge our model holds over existing methods. Methods |  ID $\uparrow$ |  Style $\uparrow$ |  Shape $\uparrow$
---|---|---|---
TI [10] | 0.634 | 0.553 | 0.633
Perfusion [54] | 0.536 | 0.549 | 0.676
Ours (w/o rand mask) | 0.659 | 0.567 | 0.694
Ours (w/o explicit) | 0.664 | 0.530 | 0.661
Ours (Mahalanobis) | 0.666 | 0.574 | 0.663
Ours-full | 0.671 | 0.576 | 0.654

Human Study. We conduct a thorough human study to judge the efficacy of our
method from end-users’ perspective. Specifically, each of the $15$ users were
shown $20$ tuples, each containing $\\{$ID, input sketch, style image, output
caricature$\\}$ from all competing methods, and asked to rate the caricatures
on a discrete scale of $[1,5]$ (worst to best) based on fidelity to input
sketch-shape, style, and ID – resulting in a total of $300$ responses per
method. The final score for each method is calculated from the mean of all its
responses. Our method with high shape-fidelity and identity-preservation,
garners an impressive overall score of $4.1$ (Tab. 2) surpassing others.
Although, the users preferred TI [10] over Perfusion [54] in terms of
identity-preservation, they both score lower compared to ours.

Figure 7: Our model’s capacity to integrate various modalities. Table 2: Human Study Scores. Methods |  ID $\uparrow$ |  Style $\uparrow$ |  Shape $\uparrow$ |  Overall $\uparrow$
---|---|---|---|---
TI [10] | 3.3 | 2.5 | 2.9 | 2.9
Perfusion [54] | 2.8 | 3.1 | 2.4 | 2.7
Ours | 4.4 | 3.8 | 4.2 | 4.1

###  6.3 Ablation Study

Design Choices. Our ablation experiments are depicted in Fig. 6 and Tab. 1.
(i) To judge the impact of explicit editing we exclude it for an experiment,
to observe that caricatures lose defining visual characteristics, dropping
scores to $0.006$ and $0.046$ in identity and style similarities,
respectively, thus proving its significance. (ii) Removing Random mask
reconstruction (Sec. 5.3) results in $0.659$ ($0.553$) for ID (style),
validating its role in reinforcing robustness against local distortions in
personalisation [54]. (iii) The replacement from the Mahalanobis distance to
applying cosine similarity on the Euclidean distance alleviates the need for
cumbersome pre-cached uncentered covariance estimation [54], leading to a more
streamlined training process. More importantly, it causes an apparent
improvement in visual quality, and a slight increase ($0.05/0.002$ in
ID/style) in the similarities as well, thus replacing cosine similarity with a
more efficient choice.

Modalities. While the fourth column of Fig. 7 validates our model’s precision
in preserving identity, the fifth displays our integration of styles with
shapes. Finally, the sixth column highlights our Sketch+ID+Style result,
achieving high fidelity to input ID [23], sketch [17] and style [24].

Impact of Identity Scale ($s$). Fig. 5 shows the influence of identity scale
$s$ on the generated caricatures. Evidently, a higher $s$ tends to retain a
higher proportion of identity traits in a caricature and vice-versa. Around
the sweet spot below $1.40$, users can freely choose this balance as per their
own subjective tastes to obtain coherent personalised caricatures [54]. In all
our experiments we had set $s$ as $1.2$, empirically.

##  7 Conclusion

In conclusion, our work marks a significant leap in democratising caricature
generation, offering individuals an effortless means to craft personalised
artworks with minimal input – just a photo and a conceptual sketch. By
navigating the delicate balance between abstraction and identity, our proposed
Explicit Rank-1 Model Editing and Random Mask Reconstruction, empower users to
seamlessly merge their unique identity and desired artistic style in the
caricature synthesis process. We emphasise that our intention is not to
replace the irreplaceable touch of artists but to remove accessibility
barriers, allowing enthusiasts to engage in the creative realm of caricature
art. More generally, our contribution underscores the potential for AI to
harmoniously collaborate with human creativity, ensuring that art remains a
captivating and inclusive expression for all.

## References

  * Bazazian et al. [2022] Dena Bazazian, Andrew Calway, and Dima Damen.  Dual-Domain Image Synthesis Using Segmentation-Guided GAN.  In _CVPRW_ , 2022.
  * Bhunia et al. [2022] Ayan Kumar Bhunia, Viswanatha Reddy Gajjala, Subhadeep Koley, Rohit Kundu, Aneeshan Sain, Tao Xiang, and Yi-Zhe Song.  Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches.  In _CVPR_ , 2022.
  * Cao et al. [2018] Kaidi Cao, Jing Liao, and Lu Yuan.  CariGANs: Unpaired photo-to-caricature translation.  _ACM TOG_ , 2018.
  * Chen et al. [2020] Shu-Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu.  DeepFaceDrawing: Deep Generation of Face Images from Sketches.  _ACM TOG_ , 2020.
  * Chowdhury et al. [2023] Pinaki Nath Chowdhury, Ayan Kumar Bhunia, Aneeshan Sain, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song.  What Can Human Sketches Do for Object Detection?  In _CVPR_ , 2023.
  * Chu et al. [2021] Wenqing Chu, Wei-Chih Hung, Yi-Hsuan Tsai, Yu-Ting Chang, Yijun Li, Deng Cai, and Ming-Hsuan Yang.  Learning to Caricature via Semantic Shape Transform.  _IJCV_ , 2021.
  * Deng et al. [2019] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In _CVPR_ , 2019.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Quinn Nichol.  Diffusion Models Beat GANs on Image Synthesis.  In _NeurIPS_ , 2021.
  * Dong et al. [2023] Ziyi Dong, Pengxu Wei, and Liang Lin.  DreamArtist: Towards Controllable One-Shot Text-to-Image Generation via Positive-Negative Prompt-Tuning, 2023.
  * Gal et al. [2023] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or.  An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion.  In _ICLR_ , 2023.
  * Gong et al. [2020] Julia Gong, Yannick Hold-Geoffroy, and Jingwan Lu.  AutoToon: Automatic Geometric Warping for Face Cartoon Generation.  In _WACV_ , 2020.
  * Goodfellow et al. [2014] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  In _NeurIPS_ , 2014.
  * Gu et al. [2021] Zheng Gu, Chuanqi Dong, Jing Huo, Wenbin Li, and Yang Gao.  CariMe: Unpaired Caricature Generation with Multiple Exaggerations.  _IEEE T-MM_ , 2021.
  * Han et al. [2023] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang.  SVDiff: Compact Parameter Space for Diffusion Fine-Tuning.  In _ICCV_ , 2023.
  * He et al. [2023] Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang.  Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning.  In _CVPR_ , 2023.
  * He et al. [2021] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll’ar, and Ross B Girshick.  Masked Autoencoders Are Scalable Vision Learners.  In _CVPR_ , 2021.
  * Hertzmann [2020] Aaron Hertzmann.  Why Do Line Drawings Work? A Realism Hypothesis.  _Perception_ , 2020.
  * Ho and Salimans [2021] Jonathan Ho and Tim Salimans.  Classifier-Free Diffusion Guidance.  In _NeurIPSW_ , 2021.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _NeurIPS_ , 2020.
  * Ho et al. [2022] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded Diffusion Models for High Fidelity Image Generation.  _JMLR_ , 2022.
  * Hu et al. [2022] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-Rank Adaptation of Large Language Models.  In _ICLR_ , 2022.
  * Hu et al. [2018] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-Excitation Networks.  In _CVPR_ , 2018.
  * Huo et al. [2018] Jing Huo, Wenbin Li, Yinghuan Shi, Yang Gao, and Hujun Yin.  WebCaricature: a benchmark for caricature recognition.  In _BMVC_ , 2018.
  * Jang et al. [2021] Wonjong Jang, Gwangjin Ju, Yucheol Jung, Jiaolong Yang, Xin Tong, and Seungyong Lee.  Stylecarigan: caricature generation via stylegan feature map modulation.  In _SIGGRAPH_ , 2021.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _CVPR_ , 2019.
  * Karras et al. [2020] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In _ICCV_ , 2020.
  * Karras et al. [2021] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-Free Generative Adversarial Networks.  In _NeurIPS_ , 2021.
  * Kingma and Welling [2014] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  In _ICLR_ , 2014.
  * Koley et al. [2023] Subhadeep Koley, Ayan Kumar Bhunia, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang, and Yi-Zhe Song.  Picture that sketch: Photorealistic image generation from abstract sketches.  In _CVPR_ , 2023.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _CVPR_ , 2023.
  * Li et al. [2023] Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak.  Your diffusion model is secretly a zero-shot classifier.  In _ICCV_ , 2023.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled Weight Decay Regularization.  In _ICLR_ , 2019.
  * Meng et al. [2022] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov.  Locating and Editing Factual Associations in GPT.  In _NeurIPS_ , 2022.
  * Mou et al. [2023] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models.  _arXiv preprint arXiv:2302.08453_ , 2023.
  * Nichol et al. [2022] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen.  GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.  In _ICML_ , 2022.
  * Patashnik et al. [2021] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In _ICCV_ , 2021.
  * Piao et al. [2021] Jingtan Piao, Keqiang Sun, Quan Wang, Kwan-Yee Lin, and Hongsheng Li.  Inverting Generative Adversarial Renderer for Face Reconstruction.  In _CVPR_ , 2021.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning Transferable Visual Models From Natural Language Supervision.  In _ICML_ , 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Robb et al. [2020] Esther Robb, Wen-Sheng Chu, Abhishek Kumar, and Jia-Bin Huang.  Few-Shot Adaptation of Generative Adversarial Networks.  _arXiv preprint arXiv:2010.11943_ , 2020.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  In _MICCAI_ , 2015.
  * Ruiz et al. [2023a] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  DreamBooth: Fine Tuning Text-to-image Diffusion Models for Subject-Driven Generation.  In _CVPR_ , 2023a.
  * Ruiz et al. [2023b] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman.  HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models, 2023b.
  * Ryu [2022] Simo Ryu.  Low-rank Adaptation for Fast Text-to-Image Diffusion Fine-tuning.  https://github.com/cloneofsimo/lora, 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  In _NeurIPS_ , 2022.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs, 2021.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5B: An open large-scale dataset for training next generation image-text models.  In _NeurIPSW_ , 2022.
  * Shi et al. [2023] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung.  InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning, 2023.
  * Shi et al. [2019] Yichun Shi, Debayan Deb, and Anil K Jain.  Warpgan: Automatic caricature generation.  In _CVPR_ , 2019.
  * Sohn et al. [2023] Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al.  StyleDrop: Text-to-Image Generation in Any Style.  _arXiv preprint arXiv:2306.00983_ , 2023.
  * Tang et al. [2023] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan.  Emergent Correspondence from Image Diffusion.  In _NeurIPS_ , 2023.
  * Tewari et al. [2020] Ayush Tewari, Mohamed Elgharib, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt.  Pie: Portrait image embedding for semantic control.  _ACM TOG_ , 2020.
  * Tewel et al. [2023] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon.  Key-Locked Rank One Editing for Text-to-Image Personalization.  In _SIGGRAPH_ , 2023.
  * Tritrong et al. [2021] Nontawat Tritrong, Pitchaporn Rewatbowornwong, and Supasorn Suwajanakorn.  Repurposing GANs for One-shot Semantic Part Segmentation.  In _CVPR_ , 2021.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is All you Need.  In _Advances in Neural Information Processing Systems_ , 2017.
  * Voynov et al. [2023] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman.  P+: Extended Textual Conditioning in Text-to-Image Generation.  _arXiv_ , 2023.
  * Wang et al. [2022] Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen.  Pretraining is All You Need for Image-to-Image Translation.  _arXiv preprint arXiv:2205.12952_ , 2022.
  * Wei et al. [2022] Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Zhentao Tan, Lu Yuan, Weiming Zhang, and Nenghai Yu.  HairCLIP: Design Your Hair by Text and Reference Image.  In _CVPR_ , 2022.
  * Xiao et al. [2023] Guangxuan Xiao, Tianwei Yin, William T. Freeman, Frédo Durand, and Song Han.  FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention.  _arXiv_ , 2023.
  * Xu et al. [2023] Xingqian Xu, Jiayi Guo, Zhangyang Wang, Gao Huang, Irfan Essa, and Humphrey Shi.  Prompt-Free Diffusion: Taking “Text” out of Text-to-Image Diffusion Models.  _arXiv preprint arXiv:2305.16223_ , 2023.
  * Ye et al. [2023] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang.  IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models.  _arXiv preprint arxiv:2308.06721_ , 2023.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding Conditional Control to Text-to-Image Diffusion Models.  In _ICCV_ , 2023.
  * Zhao et al. [2023] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong.  Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models.  In _NeurIPS_ , 2023.

Supplementary Material for
DemoCaricature: Democratising Caricature Generation with a Rough Sketch

##  A Details on Random Mask Reconstruction

We present more implementation details on the random mask reconstruction (RMR)
as shown in Fig. 8. We create the masked image $x_{0}^{m}$ by randomly
occluding several patches with different ratios to simulate a caricature
having local variation. Besides the random occlusion, we also apply id- and
style-specific masks on $M$ to isolate regions of interest when calculating
loss using Eq. 5. Specifically, for identity finetuning, $M$ contains a binary
mask over the background, making the model capture the distinguishing facial
features exclusively. For style reference, $M$ adopts a small value (0.2 in
this work) on the face area, subtly nudging the model to infuse the stylistic
elements from both the background and the face into caricatures.

##  B More Qualitative Results

In the end, we provide additional qualitative results across various fields.
Fig. 9 showcases more caricatures of celebrities. Beyond celebrities, we
exhibit our method’s capability of learning identity and style from artistic
and synthetic 111We collect the synthetic id image from Ruiz _et al_. [44] and
synthetic style reference from https://civitai.com. portraits in Fig. 10 and
Fig. 11 respectively. The results demonstrate the versatility of our
democratising caricature generation, allowing users to flexibly and
artistically create caricatures with the desired identities and styles.

Figure 8: Random Mask Reconstruction. $x_{0}^{m}$ mimics an image with local
variation, a critical feature of caricature. $M$ makes the objective function
focus on the region of interest and ignore the occluded area. Figure 9:
Results on celebrities. Figure 10: Results on famous artworks. Figure 11:
Results on synthetic human faces.