# Negative Differential Resistance in Spin-Crossover Molecular Devices

Dongzhe Li<EMAIL_ADDRESS>Department of Physics, Technical University of
Denmark, DK-2800 Kongens Lyngby, Denmark  Yongfeng Tong  Université Paris
Cité, CNRS, Laboratoire Matériaux et Phénomènes Quantiques UMR7162, 75013
Paris, France  Kaushik Bairagi  Université Paris Cité, CNRS, Laboratoire
Matériaux et Phénomènes Quantiques UMR7162, 75013 Paris, France  Massine Kelai
Université Paris Cité, CNRS, Laboratoire Matériaux et Phénomènes Quantiques
UMR7162, 75013 Paris, France  Yannick J. Dappe  Université Paris-Saclay, CEA,
CNRS, SPEC, 91191 Gif-sur-Yvette, France  Jérôme Lagoute  Université Paris
Cité, CNRS, Laboratoire Matériaux et Phénomènes Quantiques UMR7162, 75013
Paris, France  Yann Girard  Université Paris Cité, CNRS, Laboratoire Matériaux
et Phénomènes Quantiques UMR7162, 75013 Paris, France  Sylvie Rousset
Université Paris Cité, CNRS, Laboratoire Matériaux et Phénomènes Quantiques
UMR7162, 75013 Paris, France  Vincent Repain  Université Paris Cité, CNRS,
Laboratoire Matériaux et Phénomènes Quantiques UMR7162, 75013 Paris, France
Cyrille Barreteau  Université Paris-Saclay, CEA, CNRS, SPEC, 91191 Gif-sur-
Yvette, France  Mads Brandbyge  Department of Physics, Technical University of
Denmark, DK-2800 Kongens Lyngby, Denmark  Alexander Smogunov  Université
Paris-Saclay, CEA, CNRS, SPEC, 91191 Gif-sur-Yvette, France  Amandine Bellec
<EMAIL_ADDRESS>Université Paris Cité, CNRS, Laboratoire Matériaux
et Phénomènes Quantiques UMR7162, 75013 Paris, France

###### Abstract

We demonstrate, based on low-temperature scanning tunneling microscopy (STM)
and spectroscopy, a pronounced negative differential resistance (NDR) in spin-
crossover (SCO) molecular devices, where a Fe${}^{\text{II}}$ SCO molecule is
deposited on surfaces. The STM measurements reveal that the NDR is robust with
respect to substrate materials, temperature, and the number of SCO layers.
This indicates that the NDR is intrinsically related to the electronic
structure of the SCO molecule. Experimental results are supported by density
functional theory (DFT) with non-equilibrium Green’s functions (NEGF)
calculations and a generic theoretical model. While the DFT+NEGF calculations
reproduce NDR for a special atomically-sharp STM tip, the effect is attributed
to the energy-dependent tip density of states rather than the molecule itself.
We, therefore, propose a Coulomb blockade model involving three molecular
orbitals with very different spatial localization as suggested by the
molecular electronic structure.
Keywords: Negative differential resistance, Coulomb blockade, scanning
tunneling microscopy, spin crossover molecule, density functional theory,
Landauer-Büttiker scattering theory

CEMES, Université de Toulouse, CNRS, 29 rue Jeanne Marvig, F-31055 Toulouse,
France Center for Nanostructured Graphene, Technical University of Denmark,
DK-2800 Kongens Lyngby, Denmark

Spin-crossover (SCO) molecules, a family of molecular bistable systems with
easy spin manipulation via external stimuli, have attracted tremendous
attention due to their potential application in molecule-based spintronics
devices 1, 2, 3, 4, 5. Utilizing single SCO molecules as quantum units opens a
new pathway to reach the physical limits of miniaturization and achieve
energy-efficient control and detection of magnetism 6, 7, 8, 9, 10, 11, 12. In
this context, a fundamental understanding of the current-voltage ($I-V$)
behavior is mandatory for the smart design of future organic devices. In
particular, negative differential resistance (NDR), characterized by a
decreasing current with increasing bias in a certain bias voltage window, is
one of the most attractive quantum tunneling phenomena. Stimulated by the
promising electronic applications that the NDR effect can offer, the
functional $I-V$ behavior entailed in the NDR of single-molecule junctions has
been extensively studied during the last two decades 13, 14, 15, 16, 17, 18,
19, 20, 21, 22, 23, 24, 25, 26, 27, 28.

Several NDR mechanisms have been proposed by quantum transport theory and
model analysis, namely resonances in the projected density of states (PDOS) of
the scanning tunneling microscopy (STM) tip 13, voltage drop along the
molecule 14, electron-phonon interaction 15, 27, Coulomb blocking 16, surface
states 19, and orbital symmetry matching 17. However, most reported NDR
effects are restricted to nonmagnetic molecular systems and have not yet been
demonstrated by experiments in SCO systems. The NDR associated with magnetic
orderings may give rise to all-electrical spin manipulation due to versatile
control of the SCO complex via external stimuli. Moreover, while direct
comparison between theory and experiments has become standard for conductance,
thermopower, and Fano-factor studies in molecular junctions, NDR studies in
corresponding molecular junctions are still missing.

In the present Letter, we report the direct experimental observation of
pronounced NDR in the current-voltage characteristics of SCO molecule,
FeII((3,5-(CH3)2Pz)3BH)2 (named FeMPz in the following, Pz refers to
pyrazolyl), deposited on various surfaces. Interestingly, the pronounced NDR
effect appearing for negative bias is robust with respect to substrate
materials (ranging from metal to semi-metal), temperature, and the number of
SCO layers. These results are analyzed and explained by density functional
theory (DFT) with non-equilibrium Green’s functions (NEGF) calculations and a
Coulomb blockade model. Our calculations conclude on two main points: (i) The
mean-field DFT+NEGF approach fails to reproduce molecular NDR, presumably due
to the lack of strong correlation effects; (ii) The simple model, including
Coulomb interactions on the molecule, reproduces experimental observations,
tracing the origin to a specific strongly localized molecular orbital blocking
electron transport through the other (more delocalized) orbitals.

As reported in the literature for sub-monolayer coverage, the FeMPz molecules
self-assemble on various substrates forming one-monolayer-high islands.
Interestingly, using a bias voltage of 0.3 V to acquire STM images (inset of
Fig. 1a-b), two types of molecules (bright and dark) are identified, which can
be correlated to the spin state of the molecules 29, 30. Indeed, the thermal
transition is partial, and at low temperature both spin states co-exist in the
molecular network 31. To investigate the electronic properties of the FeMPz
molecules, we performed scanning tunneling spectroscopy (STS) measurements.
Fig. 1 presents typical $I-V$ and $dI/dV$ curves recorded over FeMPz molecules
on Au(111), Cu(111) and highly ordered pyrolytic graphite (HOPG). At positive
voltages, on bright molecules, a molecular state can be measured at low bias
voltage (Fig. 1a), thus explaining the contrast observed at 0.3 V. The exact
energy position of the molecular orbital cannot be determined as the molecules
are switching between both spin states for bias voltage above 0.5 V as
evidenced by the sharp transition in the $I-V$ curves 30. No other molecular
state is detected for positive bias. At negative bias, and for all substrates
investigated, the STS curves exhibit a clear NDR whose position can vary
between 0.6 and 1.1 V. The NDR energy position can not directly be related to
the spin state. Indeed, the difference observed in Fig. 1b on Cu(111) could
originate from the tip distance being more important over the bright molecule
compared to the dark one during the spectra acquisition. Moreover, we find
that the NDR signature is robust with temperature. As visible in Fig. S1 in
Supporting Information (SI), at liquid N2 temperature, the NDR is observed
both over the bright and dark molecules when adsorbed on Au(111).

The electronic properties of the FeMPz molecules have also been investigated
on several layers of molecules grown on Au(111). As illustrated by the STM
topographic image of Fig. 2a, at coverages above one monolayer, the molecular
films follow a layer-by-layer growth with nicely defined edges. Fig. 2b and c
present the $I-V$ and $dI/dV$ curves measured on the second and third layer,
respectively. One can clearly see that no state is available at positive bias
voltage up to 2 V for the molecules in the second or third layer. Thus, the
second and third layers cannot be imaged at positive bias voltages. At
negative voltages, the NDR is still there but shifted to lower values as the
molecular island height is increased. For molecules in the first monolayer,
the $I-V$ curves acquired are “noisy" due to the molecular switch under the
STM tip (see Fig. 1a). On the contrary, for the second and third layers, the
$I-V$ curves are smooth, and the position of the NDR is reproducible from one
molecule to the other. The maximum value of the NDR is at an energy of
$-1.26\pm 0.07$ eV for molecules in the second layer, and of $-1.42\pm 0.10$
eV for molecules in the third layer.

We now focus on the different possible mechanisms at the origin of the
observed NDR. The first thought is that it can arise from the molecular
switching between the two spin states under the electric field applied between
the tip and the metallic surface. In this case, the current measured as a
function of the bias voltage switches between two extreme curves 32, 33. In
the case of HOPG substrate, by measuring the $I-V$ curve in back and forth
mode (i.e., the bias voltage is swept from $-2$ V to 2 V and then from 2 V to
$-2$ V), abrupt switching is observed in the return curve (see Fig. S2 in SI).
This brutal change is reproducible over various molecules and most probably
originates from the spin state switching.

Prior to and after the STS measurements on the molecules, the electronic state
of the tip has been checked by acquiring spectra on bare Au(111) or bare
Cu(111) to confirm the presence of the Shockley states (i.e., surface states).
In some cases, after measurements over molecules, one can be picked up by the
tip, giving rise to a specific signal. Fig. S3 in the SI reports two examples
of such situations either on Cu(111) or on Au(111). In both cases, the NDR is
observed but once for negative bias and once for positive bias. In the second
case, one could think that the molecule is more strongly coupled to the tip
than the surface, thus explaining the reversing of the $I-V$ curve 34. These
experimental results tend to prove that the NDR signature observed is related
to the intrinsic property of the FeMPz molecule.

In order to explain such phenomena, we performed ab initio quantum transport
calculations for STM-based FeMPz molecular junctions as shown on the upper
panels of Fig. 3 for two considered geometries – with sharp (on the left) or
flat (on the right) STM tips. We used DFT at the PBE level including a van der
Waals correction 35, as implemented in the Siesta 36, 37 code. Subsequent
electron transport calculations were carried out using TranSiesta 38, 39 and
post-processing codes TBtrans and sisl 40, which employ the NEGF formalism
combined with DFT (see SI for computational details).

We start our discussion with the sharp tip geometry. From the zero-bias
transmission function for the LS state ($S=0$), shown in Fig. 3a (in log
scale), we observe a HOMO dominant transport behavior around the Fermi energy
($E_{F}$) with a large HOMO-LUMO gap of more than 3 eV. Due to the weak
coupling between the HOMO-1 orbital and the Au surface, we observe a narrow
transmission resonance at about 0.5 eV below $E_{F}$. This is also reflected
in the spatial distributions shown on insets. The HOMO-1 is strongly localized
on the Fe atom, originating from the Fe $d_{z^{2}}$ orbital (the $Z$ axis
being directed along the molecule axis). The 2-fold degenerate HOMO, on the
other hand, is more delocalized within the molecule (originating from Fe
$d_{xz,yz}$ orbitals, which couple better to ligands) and therefore hybridizes
much strongly with the surface, which results in a broadened transmission peak
just below $E_{F}$.

Next, we consider the calculated current-voltage characteristics, $I-V$. The
effect of applied bias, $V$, was included by shifting the levels of left (STM
tip) and of right (substrate) electrodes symmetrically by $+V/2$ and $-V/2$,
respectively. The Green’s functions of the scattering region (including the
molecule and some Au atoms on both sides, shown on upper panels of Fig. 3)
were then calculated self-consistently at each $V$. Interestingly, the
simulated $I-V$ curve (Fig. 3b) reproduces the experimental observation
reasonably well with a clear NDR signal observed for negative bias. The
observed decrease in the current with voltage is a consequence of the lower
amplitude of the HOMO-related transmission peak, as shown in Fig. S4 in SI.
This peak, moreover, follows almost rigidly the Fermi level of the substrate
when the bias is applied, reflecting a much stronger coupling of HOMO to the
substrate compared to the tip. To get more insight, we plot in Fig. 3c the
PDOS of the tip apex atom (black) and its $s$-orbital contribution (red). A
strong energy-dependent behavior of the PDOS (mainly from $s$ orbital) near
$E_{F}$ is clearly related to the atomically sharp geometry of the tip. In
particular, a significant drop in the apex atom PDOS is observed right above
the Fermi energy, which corresponds to the small negative bias where NDR
appears. This indicates that NDR may indeed be related to the energy-dependent
DOS of the tip apex atom rather than to the molecule itself.

To confirm this point, we further analyzed the coupling matrices, $\gamma$, of
the HOMO orbital to both the surface and the tip (Fig. S5) which should be
proportional to the corresponding PDOS. We find a dip-like feature in the
$\gamma_{\text{Tip}}$ for negative $V$, correlating well with the drop of the
tip atom PDOS discussed above, while the $\gamma_{\text{Surface}}$ is almost
constant. The resonant transmission from the molecular level is given by
$T=4\gamma_{\text{Tip}}\gamma_{\text{Surface}}/(\gamma_{\text{Tip}}+\gamma_{\text{Surface}})^{2}$.
Therefore, one can conclude that the reduction of the HOMO-related
transmission peak and, as a consequence, the NDR originate from the strong
energy-dependent PDOS of the sharp STM tip. A similar effect was also reported
in Ref. 13.

In order to eliminate the electronic effects of the tip, we have therefore
performed transport calculations replacing the sharp STM tip with a flat
surface. The transmission curve, Fig. 3d, looks very similar to the one for
the sharp tip case, while the $I-V$, Fig. 3e, shows essentially different
behavior. The current gradually increases with respect to the bias voltage; no
NDR effect is observed. The PDOS analysis (Fig. 3f) also reveals rather
constant PDOS around $E_{F}$ in this case.

We note that the NDR effect is robust and well reproducible in our experiment,
regardless of tip and substrate. This questions the tip-related mechanism
presented above, rooted in the energy-dependent PDOS for a specific Au tip,
pointing to a mechanism intrinsic to the molecule itself. Moreover, the mean-
field-based DFT approach may not accurately capture strong correlation effects
on SCO molecules coupled weakly to electrodes. We suggest, therefore, that the
Coulomb blockade-like model introduced previously by Muralidharan and Datta 16
can also apply to the present case. The model is based on two kinds of
molecular orbitals characterized by very different spatial localizations,
exactly like in our case of HOMO (relatively delocalized) and HOMO-1 (strongly
localized). The first one (with a large coupling to electrodes) provides the
main transport channel (by holes) at a negative voltage. While the other one
(weakly coupled) acts to block the transport when started, in its turn, to be
emptied due to the Coulomb repulsion on the molecule strongly unfavoring a
double hole configuration.

The model (Fig. 4a) contains four (many-body) molecular states: the ground
state of $N$ electrons, $\ket{0}$ (HOMO and HOMO-1 are occupied), and 3 states
with $N-1$ electrons, $\ket{1}$, $\ket{2}$, and $\ket{3}$, corresponding to a
hole on HOMO1, HOMO2, and HOMO-1 orbitals, respectively. They have energies
$E_{i}(i=0,..3)$ and are characterized, in the sequential tunneling (weak
coupling) transport regime, by their probabilities, satisfying the
normalization condition, $\sum_{i=0,3}P_{i}=1$. In the out-of-equilibrium
situation (applied voltage) the occupations obey the master equation:

$\begin{dcases}\frac{dP_{0}}{dt}&=-P_{0}\sum_{i=1,3}R_{0i}+\sum_{i=1,3}P_{i}R_{i0}\\\
\frac{dP_{i}}{dt}&=-P_{i}R_{i0}+P_{0}R_{0i},~{}~{}~{}~{}~{}i=1,2,3\end{dcases}$
(1)

where $R_{0i}=\sum_{\alpha=L,R}\Gamma_{i}^{\alpha}[1-f_{\alpha}(E_{0}-E_{i})]$
describes the transition rate between the $N$ electron state $\ket{0}$ and
$N-1$ electron states $\ket{i}$ with $\Gamma^{\alpha}$ being the level
coupling strength to the electrode $\alpha$ and $f_{\alpha}(\epsilon)$ \- the
electrode’s Fermi-Dirac distribution functions allowing the jump of an
electron from the molecule to the electrode empty states (the zero energy is
chosen to be the equilibrium Fermi energy, $E_{F}=0$). Opposite transition
rates are given by
$R_{i0}=\sum_{\alpha=L,R}\Gamma_{i}^{\alpha}f_{\alpha}(E_{0}-E_{i})$ and are
associated with the hopping of an electron from electrodes to the molecule.

In the steady state, the probabilities are constant in time and can be written
explicitly:

$\begin{dcases}P_{0}&=\frac{1}{1+\sum_{i=1,3}R_{0i}/R_{i0}}\\\
P_{i}&=P_{0}\frac{R_{0i}}{R_{i0}},~{}~{}~{}~{}~{}i=1,2,3\end{dcases}$ (2)

The electron current from the left electrode to the molecule through the state
$\ket{i}$ is then provided by:

$I_{i}^{L}=-e^{2}/\hbar(R^{L}_{0i}P_{0}-R^{L}_{i0}P_{i}),$ (3)

while the total current is obtained by summing all the contributions,
$I^{L}=\sum_{i=1,3}I_{i}^{L}$.

We can estimate the necessary parameters, energies $E_{i}$ and coupling
constants $\Gamma_{i}^{\alpha}$, from DFT calculations, relating them to the
energies and couplings of corresponding molecular orbitals, so that
$E_{0}-E_{1,2}=\epsilon_{\rm 1,2},E_{0}-E_{3}=\epsilon_{\rm 3}$ and
$\Gamma_{1,2}^{\alpha}=\gamma_{\rm
1,2}^{\alpha},\Gamma_{3}^{\alpha}=\gamma_{\rm 3}^{\alpha}$. Here, indexes 1,
2, 3 correspond to HOMO1, HOMO2, and HOMO-1, respectively. However, based on
the set of parameters directly extracted from DFT, we could not reproduce the
NDR (see Fig. 4b).

It is well known that the mean-filed DFT cannot accurately describe strong
correlation effects due to the self-interaction error. Moreover, standard GGA
functionals turn out to be not accurate enough to predict total energies of
SCO molecule 41, 42. Therefore, the parameters extracted from DFT are expected
to be unreliable for strongly localized HOMO-1. Indeed, by setting
$\gamma_{3,\text{L}}=0.5\gamma_{3,\text{R}}$, we could reproduce the
experimental NDR feature reasonably well, as shown in Fig. 4c. In particular,
when the negative voltage reaches the HOMO-1 level (at about $-0.6$ V), the
three levels starts to be emptied simultaneously. However, due to the weak
coupling of the HOMO-1 and electrodes, the hole stays on HOMO-1 much longer
compared to HOMO orbitals. On the other side, the double-hole occupation (in
all the HOMOs and HOMO-1) is strongly suppressed due to a substantial charging
energy on the molecule (which is encoded in the normalization condition of
occupations of four molecular states involved in the model).

As a result, the occupancies of HOMO-related states are reduced, leading to a
decrease in the associated currents (black solid and dotted lines). Therefore,
the strongly localized HOMO-1 orbital acts as a "blocking state": its small
contribution to the total current is not enough, and the total current start
to decrease, resulting in NDR. Finally, it should be noted that the DFT
parameters (which did not work straightforwardly to produce NDR) were
extracted from the low spin (LS, $S=0$) state, while the NDR is more often
observed experimentally for the high spin (HS, $S=2$) state. The DFT
calculations for the magnetic HS state turned out to be quite delicate for the
Transiesta code (to find correct level positions for frontier molecular
orbitals), especially when the Coulomb+$U$ correction (which is an unknown
parameter in general) is applied. Therefore, we did not manage to obtain
reliable parameters for this case. On the other hand, the proposed arguments
for NDR should remain valid also for the HS state due to the fact that the
same orbitals are involved in the transport: HOMO is mainly from $d_{xz,yz}$
(delocalized) while the HOMO-1 is from $d_{z^{2}}$ (strongly localized). This
is confirmed by DFT+$U$ calculations for free-standing SCO of the HS (see Fig.
S6 in SI). As a result, we can speculate that the model parameters, if
extracted from the HS case, could fit better with those reproducing the NDR.

To conclude, we report on a pronounced NDR effect from Fe${}^{\text{II}}$ SCO
molecules deposited on surfaces, highly robust to substrate materials (Au, Cu,
or HOPG), temperature, and the number of SCO layers. A generic theoretical
model based on the Coulomb blockade mechanism captures the experimental
findings pretty well, tracing the origin of NDR to strongly localized HOMO-1
and rather delocalized HOMO orbitals. We also discuss that the mean-field-
based DFT+NEGF approach fails to capture the intrinsic NDR effect due to
presumably strong correlation effects on the SCO molecule. We believe that our
findings and proposed physical mechanism bear a rather general character and
can apply to other molecular systems with similar electronic properties.

Acknowledgment This project received funding from the European Union’s Horizon
2020 Research and Innovation Programme under grant agreement no. 766726. The
Center for Nanostructured Graphene (CNG) is sponsored by the Danish National
Research Foundation (DNRF103). Part of the calculations was done using HPC
resources from CALMIP (Grant 2022-[P21008]). The authors would like to
acknowledge Marie-Laure Boillot and Talal Mallah for providing the spin-
crossover molecules and Cyril Chacon for experimental support.

The Supporting Information is available free of charge at
https://pubs.acs.org/. Experimental and computational details, spectroscopy of
FeMPz molecules on Au(111) at 78 K and on HOPG for switching, spectroscopy of
FeMPz molecules adsorbed on the tip, bias-dependent transmission functions,
tip and surface coupling strength projected on HOMO, the DFT+$U$ results for
free standing SCO.

## References

  * Ali et al. 2012 Ali, M. E.; Sanyal, B.; Oppeneer, P. M. Electronic structure, spin-states, and spin-crossover reaction of heme-related Fe-porphyrins: a theoretical perspective. _J. Phys. Chem. B_ 2012, _116_ , 5849–5859
  * Bousseksou et al. 2011 Bousseksou, A.; Molnár, G.; Salmon, L.; Nicolazzi, W. Molecular spin crossover phenomenon: recent achievements and prospects. _Chem. Soc. Rev._ 2011, _40_ , 3313–3335
  * Lefter et al. 2016 Lefter, C.; Davesne, V.; Salmon, L.; Molnár, G.; Demont, P.; Rotaru, A.; Bousseksou, A. Charge Transport and Electrical Properties of Spin Crossover Materials: Towards Nanoelectronic and Spintronic Devices. _Magnetochemistry_ 2016, _2_ , 18
  * Molnár et al. 2018 Molnár, G.; Rat, S.; Salmon, L.; Nicolazzi, W.; Bousseksou, A. Spin Crossover Nanomaterials: From Fundamental Concepts to Devices. _Adv. Mater._ 2018, _30_ , 1703862
  * Zhang et al. 2020 Zhang, Y.; Séguy, I.; Ridier, K.; Shalabaeva, V.; Piedrahita-Bello, M.; Rotaru, A.; Salmon, L.; Molnár, G.; Bousseksou, A. Resistance switching in large-area vertical junctions of the molecular spin crossover complex [Fe(HB(tz)3)2]: ON/OFF ratios and device stability. _J. Phys. Condens. Matter_ 2020, _32_ , 214010
  * Gruber and Berndt 2020 Gruber, M.; Berndt, R. Spin-crossover complexes in direct contact with surfaces. _Magnetochemistry_ 2020, _6_ , 35
  * Montenegro-Pohlhammer et al. 2021 Montenegro-Pohlhammer, N.; Sánchez-de Armas, R.; Calzado, C. J. Deposition of the Spin Crossover Fe${}^{\text{II}}$–Pyrazolylborate Complex on Au (111) Surface at the Molecular Level. _Chem. Eur. J_ 2021, _27_ , 712–723
  * Kipgen et al. 2021 Kipgen, L.; Bernien, M.; Tuczek, F.; Kuch, W. Spin-Crossover Molecules on Surfaces: From Isolated Molecules to Ultrathin Films. _Adv. Mater._ 2021, _33_ , 2008141
  * Konstantinov et al. 2021 Konstantinov, N.; Tauzin, A.; Noumbé, U. N.; Dragoe, D.; Kundys, B.; Majjad, H.; Brosseau, A.; Lenertz, M.; Singh, A.; Berciaud, S., et al. Electrical read-out of light-induced spin transition in thin film spin crossover/graphene heterostructures. _J. Mater. Chem. C_ 2021, _9_ , 2712–2720
  * Torres-Cavanillas et al. 2021 Torres-Cavanillas, R.; Morant-Giner, M.; Escorcia-Ariza, G.; Dugay, J.; Canet-Ferrer, J.; Tatay, S.; Cardona-Serra, S.; Giménez-Marqués, M.; Galbiati, M.; Forment-Aliaga, A., et al. Spin-crossover nanoparticles anchored on MoS2 layers for heterostructures with tunable strain driven by thermal or light-induced spin switching. _Nat. Chem._ 2021, _13_ , 1101–1109
  * Boix-Constant et al. 2022 Boix-Constant, C.; García-López, V.; Navarro-Moratalla, E.; Clemente-León, M.; Zafra, J. L.; Casado, J.; Guinea, F.; Mañas-Valero, S.; Coronado, E. Strain Switching in van der Waals Heterostructures triggered by a Spin-Crossover Metal Organic Framework. _Adv. Mater._ 2022, 2110027
  * 12 Gavara-Edo, M.; Córdoba, R.; Valverde-Muñoz, F. J.; Herrero-Martín, J.; Real, J. A.; Coronado, E. Electrical Sensing of the Thermal and Light-Induced Spin Transition in Robust Contactless Spin-Crossover/Graphene Hybrid Devices. _Adv. Mater._ _n/a_ , 2202551
  * Xue et al. 1999 Xue, Y.; Datta, S.; Hong, S.; Reifenberger, R.; Henderson, J. I.; Kubiak, C. P. Negative differential resistance in the scanning-tunneling spectroscopy of organic molecules. _Phys. Rev. B_ 1999, _59_ , R7852–R7855
  * Mujica and Ratner 2001 Mujica, V.; Ratner, M. A. Current–voltage characteristics of tunneling molecular junctions for off-resonance injection. _Chemical Physics_ 2001, _264_ , 365–370
  * Galperin et al. 2005 Galperin, M.; Ratner, M. A.; Nitzan, A. Hysteresis, Switching, and Negative Differential Resistance in Molecular Junctions. _Nano Lett._ 2005, _5_ , 125–130
  * Muralidharan and Datta 2007 Muralidharan, B.; Datta, S. Generic model for current collapse in spin-blockaded transport. _Phys. Rev. B_ 2007, _76_ , 035432
  * Chen et al. 2007 Chen, L.; Hu, Z.; Zhao, A.; Wang, B.; Luo, Y.; Yang, J.; Hou, J. G. Mechanism for Negative Differential Resistance in Molecular Electronic Devices: Local Orbital Symmetry Matching. _Phys. Rev. Lett._ 2007, _99_ , 146803
  * Kang et al. 2010 Kang, N.; Erbe, A.; Scheer, E. Observation of negative differential resistance in DNA molecular junctions. _App. Phys. Lett._ 2010, _96_ , 023701
  * Heinrich et al. 2011 Heinrich, B. W.; Rastei, M. V.; Choi, D.-J.; Frederiksen, T.; Limot, L. Engineering Negative Differential Conductance with the Cu(111) Surface State. _Phys. Rev. Lett._ 2011, _107_ , 246801
  * Perrin et al. 2014 Perrin, M. L.; Frisenda, R.; Koole, M.; Seldenthuis, J. S.; Gil, J. A. C.; Valkenier, H.; Hummelen, J. C.; Renaud, N.; Grozema, F. C.; Thijssen, J. M., et al. Large negative differential conductance in single-molecule break junctions. _Nat. Nanotechnol._ 2014, _9_ , 830–834
  * Xu and Dubi 2015 Xu, B.; Dubi, Y. Negative differential conductance in molecular junctions: an overview of experiment and theory. _J. Phys. Condens. Matter_ 2015, _27_ , 263202
  * Warner et al. 2015 Warner, B.; El Hallak, F.; Prüser, H.; Sharp, J.; Persson, M.; Fisher, A. J.; Hirjibehedin, C. F. Tunable magnetoresistance in an asymmetrically coupled single-molecule junction. _Nat. Nanotechnol._ 2015, _10_ , 259–263
  * Tawfik et al. 2017 Tawfik, S. A.; Weston, L.; Cui, X.; Ringer, S.; Stampfl, C. Near-perfect spin filtering and negative differential resistance in an Fe (II) S complex. _J. Phys. Chem. Lett._ 2017, _8_ , 2189–2194
  * Kuang et al. 2018 Kuang, G.; Chen, S. Z.; Yan, L.; Chen, K. Q.; Shang, X.; Liu, P. N.; Lin, N. Negative Differential Conductance in Polyporphyrin Oligomers with Nonlinear Backbones. _J. Am. Chem. Soc._ 2018, _140_ , 570–573
  * Fung et al. 2019 Fung, E.-D.; Gelbwaser, D.; Taylor, J.; Low, J.; Xia, J.; Davydenko, I.; Campos, L. M.; Marder, S.; Peskin, U.; Venkataraman, L. Breaking Down Resonance: Nonlinear Transport and the Breakdown of Coherent Tunneling Models in Single Molecule Junctions. _Nano Lett._ 2019, _19_ , 2555–2561
  * Jiang et al. 2019 Jiang, Y.; Wei, Y.; Wang, Y.; Ngeywo, K. T.; Hu, Y.; Wang, S.; Pang, K.; Zhang, G.; Li, W.; Jiang, Y. Perfect Spin Filtering in Homobimetallic Ni Complex with High Tolerance to Structural Changes. _J. Phys. Chem. Lett._ 2019, _10_ , 7842–7849
  * Liu and Segal 2020 Liu, J.; Segal, D. Sharp negative differential resistance from vibrational mode softening in molecular junctions. _Nano Lett._ 2020, _20_ , 6128–6134
  * Zhao et al. 2021 Zhao, M.; Zhao, Y.; Xi, Y.; Xu, H.; Feng, H.; Xu, X.; Hao, W.; Zhou, S.; Zhao, J.; Dou, S. X.; Du, Y. Electric-Field-Driven Negative Differential Conductance in 2D van der Waals Ferromagnet Fe3GeTe2. _Nano Lett._ 2021, _21_ , 9233–9239
  * Bairagi et al. 2016 Bairagi, K. et al. Molecular-scale dynamics of light-induced spin cross-over in a two-dimensional layer. _Nat. Commun._ 2016, _7_ , 12212
  * Tong et al. 2021 Tong, Y.; Kelaï, M.; Bairagi, K.; Repain, V.; Lagoute, J.; Girard, Y.; Rousset, S.; Boillot, M.-L.; Mallah, T.; Enachescu, C., et al. Voltage-Induced Bistability of Single Spin-Crossover Molecules in a Two-Dimensional Monolayer. _J. Phys. Chem. Lett._ 2021, _12_ , 11029–11034
  * Kelai et al. 2021 Kelai, M.; Repain, V.; Tauzin, A.; Li, W.; Girard, Y.; Lagoute, J.; Rousset, S.; Otero, E.; Sainctavit, P.; Arrio, M.-A.; Boillot, M.-L.; Mallah, T.; Enachescu, C.; Bellec, A. Thermal Bistability of an Ultrathin Film of Iron(II) Spin-Crossover Molecules Directly Adsorbed on a Metal Surface. _J. Phys. Chem. Lett._ 2021, _12_ , 6152–6158
  * Miyamachi et al. 2012 Miyamachi, T.; Gruber, M.; Davesne, V.; Bowen, M.; Boukari, S.; Joly, L.; Scheurer, F.; Rogez, G.; Yamada, T. K.; Ohresser, P.; Beaurepaire, E.; Wulfhekel, W. Robust spin crossover and memristance across a single molecule. _Nat. Commun._ 2012, _3_ , 938
  * Gerhard et al. 2017 Gerhard, L.; Edelmann, K.; Homberg, J.; Valášek, M.; Bahoosh, S. G.; Lukas, M.; Pauly, F.; Mayor, M.; Wulfhekel, W. An electrically actuated molecular toggle switch. _Nat. Commun._ 2017, _8_ , 14672
  * Schull et al. 2009 Schull, G.; Frederiksen, T.; Brandbyge, M.; Berndt, R. Passing current through touching molecules. _Phys. Rev. Lett._ 2009, _103_ , 206803
  * Grimme 2006 Grimme, S. Semiempirical GGA-type density functional constructed with a long-range dispersion correction. _J. Comput. Chem._ 2006, _27_ , 1787–1799
  * Soler et al. 2002 Soler, J. M.; Artacho, E.; Gale, J. D.; García, A.; Junquera, J.; Ordejón, P.; Sánchez-Portal, D. The SIESTA method for ab initio order-N materials simulation. _J. Phys. Condens. Matter_ 2002, _14_ , 2745
  * García et al. 2020 García, A.; Papior, N.; Akhtar, A.; Artacho, E.; Blum, V.; Bosoni, E.; Brandimarte, P.; Brandbyge, M.; Cerdá, J. I.; Corsetti, F., et al. Siesta: Recent developments and applications. _J. Chem. Phys._ 2020, _152_ , 204108
  * Brandbyge et al. 2002 Brandbyge, M.; Mozos, J.-L.; Ordejón, P.; Taylor, J.; Stokbro, K. Density-functional method for nonequilibrium electron transport. _Phys. Rev. B_ 2002, _65_ , 165401
  * Papior et al. 2017 Papior, N.; Lorente, N.; Frederiksen, T.; García, A.; Brandbyge, M. Improvements on non-equilibrium and transport Green function techniques: The next-generation TRANSIESTA. _Comput. Phys. Commun._ 2017, _212_ , 8–24
  * Papior 2021 Papior, N. sisl: v0.11.0. 2021; https://doi.org/10.5281/zenodo.597181
  * Cirera et al. 2018 Cirera, J.; Via-Nadal, M.; Ruiz, E. Benchmarking density functional methods for calculation of state energies of first row spin-crossover molecules. _Inorg. Chem._ 2018, _57_ , 14097–14105
  * Cirera and Ruiz 2020 Cirera, J.; Ruiz, E. Assessment of the SCAN Functional for Spin-State Energies in Spin-Crossover Systems. _J. Phys. Chem. A_ 2020, _124_ , 5053–5058

Figure 1:  STS measurements on molecules in monolayer islands grown on a)
Au(111), b) Cu(111), and c) HOPG substrates. For a) and b), the $I-V$ and
$dI/dV$ curves have been acquired on molecules appearing either dark or bright
at 0.3 V. The 8.5${\times}$8.5 nm2 STM topographic images with the positions
at which the spectra have been acquired are shown in the inset. a) 0.3 V, 50
pA, b) 0.3 V, 5 pA and c) $-1.5$ V, 20 pA.  Figure 2:  (a) $150{\times}150$
nm2 topographic STM image of FeMPz island up to 3 ML high absorbed on Au(111)
($-1.5$ V, 20 pA). (b) and (c) $I-V$ and $dI/dV$ curves recorded on the second
(black curves) and the third layer (red curves). The presented curves are
averaged over several molecules and measurements (2${}^{\text{d}}$ layer: 20
curves on 4 molecules, 3${}^{\text{rd}}$ layer: 15 curves on 3 molecules)
Figure 3:  NEGF+DFT results: (a) Zero-bias transmission as a function of
energy for the low spin ($S$ = 0) configuration FeMPz molecular junction with
an atomically sharp STM tip. (b) Current-voltage characteristics and (c) PDOS
of the Au apex atom, where the black line denotes the total PDOS while the red
line means the PDOS of the $s$ orbital. (d-f) The same as (a-c) but with a
flat tip. Figure 4:  Model results: (a) The effective three-level model based
on the Coulomb blockade mechanism. Due to the weak coupling between the HOMO-1
and the electrodes, it plays the role of a "blocking orbital", which leads to
NDR. The voltage is assumed to be applied fully to the left electrode (STM
tip) simulating strong coupling of molecular levels to the surface (right
electrode). (b) Simulated $I-V$ curve using parameters (in eV) extracted from
DFT calculations: $\epsilon_{\text{1}}=-0.048$, $\epsilon_{\text{2}}=-0.051$,
$\epsilon_{\text{3}}=-0.518$, $\gamma_{1,\text{L}}=6.995{\times}10^{-5}$,
$\gamma_{1,\text{R}}=1.879{\times}10^{-3}$,
$\gamma_{2,\text{L}}=7.725{\times}10^{-5}$,
$\gamma_{2,\text{R}}=2.010{\times}10^{-3}$,
$\gamma_{3,\text{L}}=1.190{\times}10^{-5}$,
$\gamma_{3,\text{R}}=1.736{\times}10^{-4}$. Here, no NDR is observed. (c) The
same as (b) but $\gamma_{3,\text{L}}=1.190{\times}10^{-5}$,
$\gamma_{3,\text{R}}=2.380{\times}10^{-5}$. A clear NDR is found for negative
bias.


# A Survey for Graphic Design Intelligence

Danqing Huang1† Jiaqi Guo1† Shizhao Sun1† Hanling Tian2§ Jieru Lin3§ Zheng
Hu4§ Chin-Yew Lin1 Jian-Guang Lou1 Dongmei Zhang1 1Microsoft 2Xi’an Jiaotong
University
3Harbin Institute of Technology 4 Central South University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>†Equal contributions. §This work was done when the authors were interns at
Microsoft Research Asia.

###### Abstract

Graphic design is an effective language for visual communication. Using
complex composition of visual elements (e.g., shape, color, font) guided by
design principles and aesthetics, design helps produce more visually-appealing
content. The creation of a harmonious design requires carefully selecting and
combining different visual elements, which can be challenging and time-
consuming. To expedite the design process, emerging AI techniques have been
proposed to automatize tedious tasks and facilitate human creativity. However,
most current works only focus on specific tasks targeting at different
scenarios without a high-level abstraction. This paper aims to provide a
systematic overview of graphic design intelligence and summarize literature in
the taxonomy of representation, understanding and generation. Specifically we
consider related works for individual visual elements as well as the overall
design composition. Furthermore, we highlight some of the potential directions
for future explorations.

##  1 Introduction

Graphic design conveys messages in a visual way. It occurs everywhere in our
daily life, from striking advertisements, attractive movie posters, to
branding logos and stunning user interfaces. An aesthetically-pleasing design
can draw audiences’ attention and improve their engagement.

Figure 1: An overview of graphic design intelligence. The composition of
visual elements produces a design. AI components including representation,
understanding and generation serve as assistants to enhance graphic design in
both element-level and design-level.

However, creating a successful design can be difficult and time-consuming, due
to the enormous space of design choices. For instance, to create a product
advertising poster, a designer often first explores posters for similar
products to form an initial idea. Then, she needs to select or create visual
elements (e.g., shape, color, font & typography) that best match the product
and the poster style. Finally, all the visual elements are composited
properly, such that the resulting poster is compelling and conforms to the
well-developed design practices (e.g., the golden ratio principle). The above
process usually iterates multiple times to finalize the poster. To ease this
creation process, practitioners and researchers have innovated a lot of
professional authoring tools, such as Photoshop, Illustrator, and Sketch, but
it still requires huge human efforts to create designs.

Recently, with the rapid advancements in artificial intelligence (AI), the
field of Graphic Design Intelligence has been emerging, which applies AI to
innovating intelligent services for graphic design (see Figure 1). Research
work in this field can be mainly categorized into three groups. The first line
of work, Graphic Design Representation, aims to learn low-dimensional and
semantic-rich representations for designs. The representations make it
possible to search for designs with targeted functionalities and styles,
accelerating the ideation process. The second line of work, Graphic Design
Understanding, attempts to perform reverse engineering on certain aspects of
designs, such as identifying visual elements, parsing element relations, and
recognizing visual flows. The design understanding capability empowers a rich
set of applications, e.g., visual element recommendations, design quality
assessments, and focus area prediction. The third line of work, Graphic Design
Generation, takes a step further and aims to automate the complicated design
creation process. Given a specification, it automatically generates visual
elements and composites them to form a compelling design. Nowadays, as the
techniques developed, an increasing number of intelligent services have been
integrated into professional tools to boost productivity, and it also raises
an urgent need to comprehensively organize existing literature for a better
overview of graphic design intelligence.

In this survey, we aim to bridge this gap and review the related literature on
graphic design intelligence according to the taxonomy of representation,
understanding, and generation. We also discuss the open questions in this
field and some promising research directions.

##  2 Overview

In this section, we introduce some graphic design basics and summarize the
major components in design intelligence.

###  2.1 Graphic Design Basics

Graphic design is the process of planning and projecting ideas with the use of
rich visual elements. It helps present the content in a more visually-
appealing way. There are many types of design such as publication (book,
magazine), user interface (website, mobile app) and advertising (poster,
banner). As it covers a wide range of fields, this survey focuses on key and
general AI techniques applied in static graphic design under digital media.

#### Visual Elements

are the building blocks of graphic design. Here we list some most frequently-
used elements:

  * •

Shape. It is recognizable objects defined by boundaries such as with lines,
color, or negative space. Shape types are diverse which can be flat (2D) or
solid (3D), geometric or organic, etc.

  * •

Color. It carries meanings such as theme, mood and visual hierarchy using
attributes such as hue, saturation, value and brightness.

  * •

Font & Typography. Fonts with different leading, tracking and kerning can
represent different meanings. Typography arranges type in order to make the
displayed words legible and appealing.

#### Layout and Composition.

Together considering the individual visual elements, we use principles in
layout and composition (e.g., alignment, contrast, repetition) to arrange
elements and create a harmonious design.

###  2.2 AI in Graphic Design

AI technologies have successfully empowered many applications in graphic
design, facilitating human to speed up the design process including materials
searching, ideation, design creation and refinement, etc. From the view of
machine learning, graphic design intelligence can be summarized into three
core components:

  * •

Representation relates to how to encode a design which serves as a backbone to
surface tasks. It is an important factor for the success of applying machine
learning algorithms. (Section 3)

  * •

Understanding focuses on machine comprehension to analyze the syntax and
semantic of a design. It aims to extract the hidden information (e.g. logical
structure, style) from the unstructured design. (Section 4)

  * •

Generation is the process of producing novel visual response (e.g., layouts,
colors, fonts) based on input constraints or queries, which improves design
creativity efficiency. (Section 5)

As shown in Figure 1, each of the three components enhances graphic design in
both visual-element-level and overall design-level. In the following sections,
we will introduce each of them in more details.

##  3 Representation

Representation learning aims to train machine-learning algorithms for learning
useful representations of data, such that building classifiers and predictors
becomes easier and more efficient Bengio et al. (2013). In this section, we
review the literature on learning element-level representations and design-
level representations.

###  3.1 Element-level Representation

This line of work aims to model the unique characteristics of visual elements
and learn their representations accordingly.

#### Shape.

Shapes in graphic design are typically represented as vector graphics, which
can be rendered at arbitrary resolutions (i.e., scale-invariant). To learn a
shape embedding, Lopes Lopes et al. (2019) proposes the first neural model. It
consists of a variational autoencoder (VAE) Kingma and Welling (2013) that
learns to project a rasterized shape into a latent space and an auto-
regressive decoder that generates a shape by predicting a sequence of discrete
commands (e.g., moveTo). The VAE and decoder are optimized independently.
Albeit effective, this approach requires clean discrete commands as labels,
which are difficult to obtain in practice. To mitigate this limitation, Reddy
Reddy et al. (2021a) presents Im2Vec, a VAE-based approach that learns shape
embeddings with indirect supervision from rasterized shapes. The encoder in
Im2Vec takes a rasterized shape as input and generates a latent code. A set of
decoders then predict Bezier paths based on the code to reconstruct the shape.
The paths can be rasterized and composited differentiablely so that Im2Vec can
be optimized in an end-to-end manner without any path supervision. The learned
shape embeddings enable plausible interpolation between samples to create
novel shapes.

#### Color.

Some research efforts have attempted to learn neural representations of color
for palette recommendation and re-colorization tasks. Kim Kim et al. (2022)
regards each color as a discrete token and a color palette as a sentence. They
adopt the fastText embedding technique Bojanowski et al. (2017), to obtain
color embeddings. Qiu Qiu et al. (2022) takes a step forward and attempts to
learn contextualized color embeddings using Transformer Vaswani et al. (2017)
and BERT-style training loss Devlin et al. (2019). Unlike this line of work,
Deshpande Deshpande et al. (2017) aims to learn a disentangled latent color
representation for whole images or designs using VAE. The color
representations can then be used to colorize grayscale images.

#### Font & Typography.

Early work Wang et al. (2015) on typography representations ratsterizes text
of a specific font into an image and leverages the Convolutional neural
network (CNN) to obtain the font embedding. The CNN is trained with the image
reconstruction loss and the font family classification loss. Later work
Balashova et al. (2018) exploits the structural nature of typography and
attempts to learn stroke-based font representations using approaches similar
to those for vector graphics. Liu Liu et al. (2022) instead proposes to model
a glyph as shape primitives enclosed by quadratic curves. They find this
implicit representation well-suited because glyphs have clear boundaries and
can be easily split into parts. Reddy Reddy et al. (2021b) extends this idea
further and develops the multi-implicit representation for glyphs based on the
observations that a complex font can be encoded as a composition of global
implicit functions. The learned font representations can be used for font
retrieval, style transfer, and novel typography creation.

###  3.2 Design-level Representation

This spectrum of work seeks to learn representations of a design based on how
it is composited with visual elements.

#### Layout.

Layout, the positioning and sizing of all the shapes in a design, plays an
important role in graphic design. Deka Deka et al. (2017) renders a layout as
an image and learns a low-dimensional layout embedding with an autoencoder. To
model the geometric relationships between shapes more effectively, Manandhar
Manandhar et al. (2020) proposes to represent a layout as a graph, in which
vertices indicate shapes and edges denote the geometric relationships. A graph
neural network is adopted to encode the layout graph and optimized via the
layout reconstruction loss and a triplet ranking loss. Xie Xie et al. (2021)
introduces CanvasEmb, which takes a richer set of shape properties into
consideration for layout modeling, including type property, geometric
property, and content-related properties. The Transformer encoder is employed
to encode a layout and is trained with the masked language modeling objective
Devlin et al. (2019). The learned layout representations can facilitate
various layout understanding tasks, e.g., layout retrievals, object role
detection, and relation parsing Feng et al. (2022).

#### Overall Design.

A design is often composed of visual elements with multi-modal content. For
instance, a graphic user interface (GUI) includes textual and visual content.
Hence, it is natural to explicitly model this compositionality with modality-
aware feature extractors. He He et al. (2021) proposes ActionBERT to learn
mobile GUI representations. Given a GUI, ActionBERT first tries to detect all
its visual elements through the view hierarchy or by using OCR and object
detection services. Then, ActionBERT leverages a pre-trained text encoder and
vision encoder to extract features for elements with multi-modal content. The
features are finally fed into a Transformer encoder to build the GUI
representation. Rastogi Rastogi et al. (2021) introduces UIBert, which differs
from ActionBERT in two aspects. First, UIBert leverages a richer set of
features from the GUI view hierarchy. Second, UIBert primarily focuses on
representing a single GUI, while ActionBERT supports multiple GUIs as input,
enabling multi-GUI downstream tasks. Fu Fu et al. (2021) targets the same
venue as UIBert and introduces PW2SS. In addition to the multi-modal content
features, PW2SS incorporates a layout embedding module to extract global
layout information. The learned GUI representation can facilitate a wide range
of downstream tasks, e.g., UI retrieval, application classification, icon
classification, etc.

While explicitly modeling a design based on its compositionality is sensible
and effective, it requires access to its internal structure via view hierarchy
or a set of off-the-shelf services, which are not easily accessible and could
be error-prone. To alleviate this shortcoming, some recent work attempts to
treat a design as an image and implicitly learn its compositionality via a
well-designed pre-training task. Specifically, Lee Lee et al. (2022) presents
Pix2Struct, an image-to-text model for design representation. Given a design,
Pix2Struct rasterizes it as an image and obtains its representation with a
vision encoder. To acquire the compositionality knowledge, they propose a
screenshot parsing objective to pre-train the model, which is to predict the
serialized structure of a given design, including its element type, hierarchy,
and textual content. To facilitate the pre-training, they collect an
80-million web page dataset. Though pre-trained on Web GUI, Pix2Struct can
transfer its knowledge to a wide variety of designs, such as mobile GUI, book
covers, and infographics, and it presents promising downstream performance.

###  3.3 Discussion

From the literature review above, one can observe the connection between
element-level work and design-level work is quite loose. This is unexpected
because a design is intentionally composited by its visual elements. The
representations of visual elements are supposed to benefit design
representations, and vice versa. We believe that enhancing this connection
deserves more careful study. In addition, an unlimited number of graphic
designs are available on the internet, and they are treasured resources for
building general-purpose design representation models. However, these designs
do not have precise metadata, such as view hierarchy, shape position and size,
causing a huge challenge for learning representations. Pix2Struct Lee et al.
(2022) makes a pioneer attempt in this direction by discarding metadata, and
there is still a long way to go.

##  4 Understanding

Understanding aims to perform reverse engineering from an unstructured design
and extract useful information such as style and visual flow. In this section,
we introduce several key aspects of design understanding in both element-level
and design-level. We show an example in Figure 2.

###  4.1 Element-level Understanding

Here we summarize the understanding tasks and methods in the context of
individual visual elements.

Figure 2: Design understanding in element-level and design-level.

#### Shape.

Using arbitrary curves, shapes are attached with meanings: (1) Explicit
objects: shapes can represent real-world objects (e.g., plane, apple). A
series of works have focused on sketched shape recognition, which classify
hand-drawn shapes to object categories. Recent approaches have explored
different backbone architectures such as CNN, RNN and Transformer Jongejan et
al. (2016); Lin et al. (2020)) for sketch sequences encoding and
classification. (2) Implicit metaphors: visual metaphor is a creative way to
deliver concepts by implying through visual symbols and juxtaposition, which
is widely used in advertising. As an example shown in Figure 2, the heart-like
shape infers “loves our earth”. Theoretical investigations with user studies
Indurkhya and Ojha (2017) have been conducted to understand visual
metaphors.Creating and interpreting visual metaphors is challenging, which
requires mappings between visual objects and common knowledge. Recently there
are ideation tools to facilitate visual metaphors creation. DARCI Heath et al.
(2014) learns the relatedness between adjectives (e.g., dark, cold) and low-
level image features with neural networks. MetaMap Kang et al. (2021) explores
relevant metaphor examples with extracted semantic features (concept graph),
color and shape for metaphor recommendations.

#### Color.

Progress has been made in exploring color semantics. Color Image Scale
Kobayashi (1991) is a notable attempt to match 130 basic colors and 1170
three-color combinations to 180 keywords. Based on this, Solli Solli and Lenz
(2010) proposes a semantic image descriptor that can be used as a tool in both
image labeling and retrieval. Havasi Havasi et al. (2010) later generates
single colors per meaning to benefit color selection (e.g. “snow” with color
“white”).

#### Font & typography.

Related studies of font and typography can be mainly summarized into the
following two topics: (1) Recognition aims to recognize font face from image.
Features for classification can either be manually extracted, such as local
typographical features, connected component and global texture analysis Sun
(2006), or font embeddings learned using autoencoder Wang et al. (2015) as
mentioned in Section 3.1. (2) Structuring organizes fonts by measuring their
visual similarity instead of listing them in an alphabetical order.O’Donovan
O’Donovan et al. (2014) estimates font similarity by training a model with
predefined features such as italics and thickness. Zhao Zhao et al. (2018a)
learns a font face embedding space and train a multi-task DNN by predicting
font properties.

###  4.2 Design-level Understanding

Now we focus on the overall analysis of a design.

#### Layout.

Layout analysis aims to parse the logical structure (i.e., object roles and
their relations) of an unstructured design. (1) Object roles: Most of previous
works focus on scanned text documents such as academic articles and forms.
Detailed related works can be referred to Binmakhashen Binmakhashen and
Mahmoud (2019). Compared to text documents with simple rectangle grid layouts,
other types of designs include more complex layout variations with free-form
and overlapped objects, which makes layout analysis more challenging and less
explored. Previous works mainly adapt the image detection/segmentation models
and incorporate some unique characteristics of graphic design into modeling.
For example, as object edges provide skeleton information, E3Nets Wu et al.
(2021) embed the edge features along with the RGB channels to train a pixel
segmentation model. Meanwhile, MagicLayouts Manandhar et al. (2021) integrates
the co-occurrence between UI components as a structural prior into the
existing object detector. Hao Hao et al. (2023) further adopts the advanced
Transformer-based detection architecture. (2) Relations: some works further
consider the spatial and semantic relation parsing in a design. Rule-based
methods Xu and Miller (2016) incorporate the Gestalt laws of proximity to
infer the elements groupings in HCI designs or web pages. As for learning-
based methods, CanvasEmb Xie et al. (2021) first pretrains a large-scale
design representation and then predicts the relation of an object pair with
finetuning. Following the same architecture, Shi Shi et al. (2022) recovers
the hierarchical structure by recursively merge objects with an empirical
threshold.

#### Overall design.

It studies the human perception of the overall design that can be a mixed
result from different visual elements. Here we introduce two popular topics,
visual style and visual flow. (1) Visual style. The overall look and feel of a
design needs to cohere with its content. Style labels describing a design
(e.g., artistic, dynamic, energetic, fresh) are selected from crowd-sourcing
or from existing design books. To recognize the style, most of the previous
works view it as image classification Huang et al. (2021). Instead of binary
classification objective, they learn a ranking function to estimate the score
to a style with hand-crafted features Chaudhuri et al. (2013) or
automatically-learned neural features Zhao et al. (2018b). Furthermore, Zhao
Zhao et al. (2018b) conducts a detailed analysis on how various design factors
affect style across different scales (from pixels, regions to objects). (2)
Visual flow. It studies where the viewers look and how their visual flows move
on a design, which can serve as a clue to guide designers to improve their
works. A large amount of works have been built based on eye tracking data
collection and analysis. They rely on hand-crafted features such as human face
and positional bias, relative size and color contrast between object pairs
Pang et al. (2016). Recently deep learning based models Zheng et al. (2018)
generally encode the design as image using CNN and output the saliency map by
predicting the pixel-level importance.

###  4.3 Discussion

In this section, we try to generalize several key aspects in graphic design
understanding. Previous works have considered many similarly-defined tasks but
targeting for different scenarios, where most of the annotated datasets and
proposed models can be hardly reused or transferred. We argue that the
capability of generalization should be more considered in dataset construction
and model design, and more unified frameworks are encouraged for solving these
separate tasks more efficiently. Furthermore, explainability would be useful
in model output, especially for graphic design with complex compositions
(e.g., model is expected to explain the association between the organic style
in Figure 2 and the use of dominant green color).

##  5 Generation

Generation aims to automate the complicated creation process of graphic
designs. Existing studies investigated how to create an individual design
element, i.e., _element-level generation_ , and how to compose different
design elements to produce a complete design, i.e., _design-level generation_.

###  5.1 Element-level Generation

As creating different elements requires different technologies, we categorize
the studies in this line by the design element.

#### Shape.

The shape is usually represented by Scalable Vector Graphics (SVG), which is a
sequence of higher-level commands paired with numerical arguments. We
categorize existing approaches for shape generation by the supervision signal
they utilize. First, some work directly uses _SVG path as supervision_.
Specifically, Lopes Lopes et al. (2019) trains a VAE on raster images to get
the latent code and an autoregressive SVG decoder to predict SVG paths from
the learned latent code; Carlier Carlier et al. (2020) exploits the
hierarchical nature of SVG in both the encoder and decoder of VAE by
disentangling high-level shapes from the low-level commands. Furthermore,
since it is difficult to collect large-scale and high-quality SVG paths, Reddy
Reddy et al. (2021a) proposes using _raster image as supervision_. It trains
an encoder to map a raster image to a latent code which is then decoded into a
SVG, and then uses a differentiable pipeline to render the generated SVG onto
a raster image which is compared to a raster ground truth for training.
Moreover, as language-image encoder (e.g., CLIP Radford et al. (2021)) and
text-to-image generation models (e.g., DALL-E Ramesh et al. (2022) and Stable
Diffusion Rombach et al. (2022)) achieve great success, recent work considers
leveraging _pretrained model as supervision_. Frans Frans et al. (2021)
optimizes the position and colors of the Bezier curves so that the generated
drawings best match the given textual description, where a pretrained CLIP
model is used as a metric for the similarity between the given description and
the generated drawing. Jain Jain et al. (2022) first rasterizes the SVG given
path parameters and then back-propagates the score distillation loss computed
from Stable Diffusion to update paths.

#### Color.

Mainstream approaches for color generation can be classified into two groups,
i.e., _palette-based methods_ and _object-based methods_. Palette-based
methods aim to generate a harmonic and aesthetic color palette, which is
expressed by a limited number of colors in a fixed form. Jahanian Jahanian et
al. (2013) produces the color palette by studying well-known color theories
(including color harmony and color semantics) and quantifying them in a
mathematical framework. Instead of utilizing color theories, Qiu Qiu et al.
(2022) leverages a large-scale dataset of vector graphic documents to learn to
generate the color palette. It models color palettes as a sequence and
proposes a masked color model based on BERT architecture Devlin et al. (2019)
for color sequence completion. After generating color palettes, palette-based
methods rely on human effort or heuristic rules to match the colors in the
color palette to the objects in the graphic design. Differently, object-based
methods directly generate colors for the objects. Specifically, Yuan Yuan et
al. (2021) frames color recommendation for infographics as a conditional
generation problem, where the conditions include the observed colors of
certain objects and non-color features (e.g., tree structure). It employs
Variational AutoEncoder with Arbitrary Conditioning (VAEAC), where which part
is the conditions is controlled by a binary mask vector.

#### Font & Typography.

To make the overall design aesthetically pleasing, the selection of fonts
takes into account the existing information in a graphic design. Choi Choi et
al. (2018) considers the information of the images. It recommends fonts that
convey similar feelings to the given image (e.g., a warm feeling). To achieve
such a goal, it first utilizes an image impression model to map the image into
a semantic space and then proposes a font impression model to calculate the
distance between the given image and each font in the same semantic space.
Jiang Jiang et al. (2019) considers the information of the texts. It
recommends a font to be used in a part (e.g., body) given a font for another
part of a document (e.g., header). Specifically, the font recommendation is
learned from large-scale human-generated font pairs by dual-space kNN and
asymmetric similarity metric learning.

###  5.2 Design-level Generation

When design elements are composed to generate a complete design, lots of
effort goes into the following tasks: 1) arriving at an appropriate _layout_ ,
i.e., the spatial relationships between objects, and 2) optimizing the
_overall design_ choices, i.e., the comprehensive consideration of all the
elements.

#### Layout.

The layout is represented as positions and sizes of all the objects in a
design. We categorize layout generation tasks into three groups by the
constraints they condition on.

_Unconditional generation_ aims to generate high-quality and diverse layouts
without considering any constraints. Li Li et al. (2019) proposes a novel
generative adversarial network (GAN) where the generator uses self-attention
modules to refine randomly-placed 2D graphic objects and the discriminator
maps the generated layout to a wireframe image. Patil Patil et al. (2020)
incorporates the hierarchy information of document layout into Variational
Autoencoder (VAE). Arroyo Arroyo et al. (2021) proposes using Transformer
Vaswani et al. (2017) as the building blocks of VAE to capture high-level
relationships between objects. Jiang Jiang et al. (2022b) proposes to
decompose the decoding process of VAE into two stages, where the first stage
predicts coarse region information and the second stage generates the detailed
placement for each object.

_Generation conditioned on object wireframe_ pays extensive attention to
modeling diverse constraints about object wireframes. First, some work
considers object relationships as the constraint (e.g., the title is on the
top of the image) Yang et al. (2023); Weng et al. (2023); Guo et al. (2021).
Specifically, Lee Lee et al. (2020) proposes using graph neural networks to
model object relationships, and Kikuchi Kikuchi et al. (2021) formulates the
problem as a constrained optimization in the latent space of the model.
Second, Gupta Gupta et al. (2021) studies completion, which generates the
placement of remaining objects given the placement of an initial set of
objects. It represents a layout as a sequence of graphical primitives and
leverages Transformer decoder to predict the next primitive based on existing
primitives in the layout. Besides, Kong Kong et al. (2022) considers object
types and sizes as the constraint. It predicts the masked attributes by
attending to surrounding attributes during training and refines the layout by
masking out low-confident attributes iteratively during inference. Moreover,
Jiang Jiang et al. (2022a) tackles various constraints by a uniform framework.
It represents constraints as sequences of tokens and leverages an identical
encoder-decoder framework with Transformer to handle different constraints.

_Generation conditioned on object content_ focuses on how the object contents
influence layouts. First, Zheng Zheng et al. (2019) considers the relationship
between the layout and the topic, style, and purpose encoded in the visual
content of the image and the meaning of the text. It encodes multi-modal
contents and the structural/categorical attributes of designs and leverages
GAN to model layout distribution. Besides, other work studies how to place
objects over a natural image, where the objects are better not to overlap
significant areas of the image (e.g., human faces) and the overall appearance
of the generated design is harmonious. Specifically, Zhang Zhang et al. (2020)
generates candidate positions and sizes based on the salience map of the
image, and then develops a scoring network to assess the aesthetic quality of
the candidates; Zhou Zhou et al. (2022) proposes a domain alignment module
with an inpainting network and a salience detection network to restore the
natural image from a design, aiming at solving the data shortage problem of
the training; and Cao Cao et al. (2022) adopts cross attention to fuse the
visual information of the image and designs a geometry alignment module to
align the geometric information of the image and the layout representation.

#### Overall Design.

First, Yang Yang et al. (2016) explores a template-based approach. It proposes
a set of topic-dependent templates incorporating aesthetic design principles
and develops a computational framework to automatically crop the image and
optimize the typography and color under the constraints from the template.
Besides, some work investigates multi-stage generation-based approaches, where
layout, font, and color are handled by different modules. Specifically,
Vaddamanu Vaddamanu et al. (2022) develops a genetic algorithm-based
technology to generate layouts by considering visual salience, alignment and
overlap, and then uses the generated layout to contextually select color and
font; Jin Jin et al. (2022) generates layouts by cascaded auto-encoder and
stylizes color and font by a matching-based method. Moreover, Yamaguchi
Yamaguchi (2021) proposes a single-stage generation-based approach, where the
whole design is generated by a single model. It defines a graphic design by a
multi-modal set of attributes associated with a canvas and a sequence of
visual objects such as shapes, images or texts, and trains VAE to learn the
representation.

###  5.3 Discussion

In general, layout generation receives more attention compared to element-
level generation and overall design generation, which are also critical for
creating a pleasing design and worth of more exploration. Besides, how to
control the generation by user requirements is not well-studied. On the one
hand, some practical and friendly ways to express user requirements, e.g.,
text and sketch, do not receive enough attention. On the other hand, it is
worth of further investigation about how to reduce the violation of user
constraints and improve the generation quality at the same time. Furthermore,
some important features for graphic design, which can make the generated
design more attractive and creative, are not discussed by existing studies.
For example, for font, while font face and size have been studied, other
critical attributes, e.g., line space, text alignment and font weight, are
neglected; for color, gradient color and transparency are ignored. Moreover,
existing studies mainly focus on generating a single design. In practice, a
series of designs, e.g., a set of slides and posters, is also widely used. To
generate a series, the consistency between designs should be taken into
account, which receives little attention currently.

##  6 Future

Despite extensive efforts has been made, there remains many unresolved
research problems. In this section, we summarize several promising directions
to follow up.

###  6.1 Benchmarks and Evaluation

Although there are several benchmark datasets (see Table 1), it is still an
urgent need to build a better dataset due to two reasons. The first one is the
_scale_. The largest dataset about design intelligence contains 330k data,
which is much smaller than the datasets in CV and NLP areas. We believe a
sufficient amount of data will promote the development of the algorithms. The
second one is the _coverage_. Current datasets usually support one or two
tasks on a specific type of graphic design. We think a better dataset that
supports as many as tasks and contains various graphic designs will
significantly benefit the learning and analysis of the algorithms. Besides
datasets, evaluation metrics are also critical. To make the evaluation more
comprehensive, we think proposing new metrics based on typical design
principles is promising. Furthermore, as aesthetics is subjective, large-scale
and formal human evaluation should be paid more attention to.

Dataset | Size | Tasks | Annotation
---|---|---|---
DSSE | 200 | Layout analysis | Semantic role labels
VisMet | 350 | Shape metaphor | Online ads images with metaphor keywords
Image ads | 500 | Layout generation | Image banner ads with layout annotation
AdobeVF | 1k | Font recognition | Text images with font class
CTXFont | 1k | Font recognition | Web designs with font class
Imp-1k | 1k | Visual attention | Pixel-wise importance maps
Magazine | 4k | Layout generation | Images of magazine page
Poster | 5k | Style classification | Online posters with style labels
PAT | 10k | Color understanding | Palette-text pairs
InfoVIF | 13k | Color generation | Infographics obtained from design resources sites
Crello | 23k | Color generation, layout generation | Designs with canvas and elements in vector format
InfoPPT | 23k | Layout analysis | Slides with hierarchy
RICO | 91k | Layout analysis, layout generation | UI designs collected from mobile apps
SVG-Icons8 | 100k | Shape generation | SVG icon shapes
PubLayNet | 330k | Layout analysis, layout generation | Images of papers and annotations for elements
SVG-Fonts | 14M | Shape generation, font recognition | Fonts across 62 characters in SFD format
QuickDraw | 50M | Shape recognition, shape generation | User sketches with stroke action and shape class
Table 1: Benchmark datasets related to graphic design.

###  6.2 Large-scale Self-supervised Learning

Large-scale self-supervised learning has mede incredible progress in natural
language and computer vision. It aims to learn salient representations from a
large amount of unlabeled data, where downstream tasks can leverage the
learned features for better performance. While preliminary attempts Xie et al.
(2021); Rastogi et al. (2021); He et al. (2021) have been made for graphic
design along this direction, there are still many challenges and
opportunities.

#### Information Extraction and Fusion.

A design can be viewed as either a pixel image or structured document with
object metadata (e.g., type, hierarchy). Due to such characteristics, there
are several important questions when performing large-scale self-supervised
learning for graphic design. First, as designs are usually stored as pixel
images on the web, it is not easy to transform it into a structured document.
Recently, reverse engineering for graphic design Shi et al. (2022), which
recognizes primitive objects from design images, has been attracting growing
interest. It will be a promising direction to obtain rich structured design
data. Second, whether all the information is critical and necessary. Most of
the existing approaches only consider a subset of metadata while there is
little discussion about the informativeness of different metadata. Third, how
to effectively fuse information in various modalities. It is worth exploring
the combination of design in both structure and pixel formats.

#### Model and Pretext Tasks.

Existing approaches similarly adopt classical Transformer as the backbone and
use masked language modeling (MLM) as the pretext task. We expect more
discussion on different model architectures and pretext tasks. For example,
CNN-based architecture or vision Transformer could be a good choice. For MLM,
it would be better if there is a thorough discussion about which part of
graphic design should be masked. Besides MLM, the completion mechanism, which
is widely used in natural language Brown et al. (2020), is also worth to be
explored for graphic design.

###  6.3 Design Knowledge Formulation and Encoding

Design knowledge, commonly in the form of design principles, serves as the
foundation in graphic design. It is a good reference and standard for
interpreting the design logic (e.g. where to place the object for visual
attraction, what color to use for highlighting the contrast) and evaluating
the quality. Furthermore, it can be used as an indicative feature in
downstream tasks to represent, analyze, and generate designs. There are
several attempts to explicitly express the principles into machine learning
models Kikuchi et al. (2021); Manandhar et al. (2021); Xie et al. (2021), but
they are limited to some principles that are easy to measure, such as
alignment and overlap. Hence, there is still a large exploration space of
better formulating and encoding more design knowledge.

###  6.4 Human-AI Collaboration

AI has enhanced many applications in graphic design Li et al. (2021);
Weingarten et al. (2020) and raised lots of discussions about its positive and
negative impacts. As graphic design is human-centered, it is important to
emphasize the role of AI as an effective assistant. AI should but not be
limited to follow human-centered design principles, respect human’s cognitive
capacities and be responsible to ethics.In each step of the design process
including material searching, ideation, creation and refinement, how AI can
optimally intervene and collaborate with human requires lots of user studies
and experiments. So far the topic of human-AI collaboration is less explored.
We hope more research and development of human-AI collaboration will be
conducted in the future to create an effective and harmonious design
environment for human with maximizing the value of AI.

##  7 Conclusion

This paper presents the first comprehensive survey in the field of graphic
design intelligence. We summarize relevant AI techniques with respect to
representation, understanding, and generation. In each direction, we consider
both the visual-element-level and overall design-level related works.
Moreover, we shed light on several promising future directions. We hope this
survey can provide a clear overview of graphic design intelligence, and
inspire researchers with new discoveries to advance the development of this
field.

## References

  * Arroyo et al. [2021] Diego Martin Arroyo, Janis Postels, and Federico Tombari.  Variational transformer networks for layout generation.  In CVPR, 2021.
  * Balashova et al. [2018] Elena Balashova, Amit H. Bermano, Vladimir G. Kim, Stephen DiVerdi, Aaron Hertzmann, and Thomas A. Funkhouser.  Learning a stroke‐based representation for fonts.  Computer Graphics Forum, 2018.
  * Bengio et al. [2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: a review and new perspectives.  TPAMI, 2013.
  * Binmakhashen and Mahmoud [2019] Galal M. Binmakhashen and Sabri A. Mahmoud.  Document layout analysis: A comprehensive survey.  ACM Comput. Surv., 2019.
  * Bojanowski et al. [2017] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov.  Enriching word vectors with subword information.  TACL, 2017.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 2020.
  * Cao et al. [2022] Yunning Cao, Ye Ma, Min Zhou, Chuanbin Liu, Hongtao Xie, Tiezheng Ge, and Yuning Jiang.  Geometry aligned variational transformer for image-conditioned layout generation.  In ACM MM, 2022.
  * Carlier et al. [2020] Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte.  Deepsvg: A hierarchical generative network for vector graphics animation.  Advances in Neural Information Processing Systems, 2020.
  * Chaudhuri et al. [2013] Siddhartha Chaudhuri, Evangelos Kalogerakis, Stephen Giguere, and Thomas A. Funkhouser.  Attribit: content creation with semantic attributes.  UIST, 2013.
  * Choi et al. [2018] Saemi Choi, Kiyoharu Aizawa, and Nicu Sebe.  Fontmatcher: font image paring for harmonious digital graphic design.  In IUI, 2018.
  * Deka et al. [2017] Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar.  Rico: A mobile app dataset for building data-driven design applications.  In UIST, 2017.
  * Deshpande et al. [2017] Aditya Deshpande, Jiajun Lu, Mao-Chuang Yeh, Min Jin Chong, and David Forsyth.  Learning diverse image colorization.  In CVPR, 2017.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In NAACL, 2019.
  * Feng et al. [2022] Guosheng Feng, Danqing Huang, Chin-Yew Lin, Damjan Dakic, Milos Milunovic, Tamara Stankovic, and Igor Ilic.  Exploring heterogeneous feature representation for document layout understanding.  In ICTAI, 2022.
  * Frans et al. [2021] Kevin Frans, Lisa B Soros, and Olaf Witkowski.  Clipdraw: Exploring text-to-drawing synthesis through language-image encoders.  arXiv preprint, 2021.
  * Fu et al. [2021] Jingwen Fu, Xiaoyi Zhang, Yuwang Wang, Wenjun Zeng, Sam Yang, and Grayson Hilliard.  Understanding mobile gui: from pixel-words to screen-sentences.  arXiv preprint, 2021.
  * Guo et al. [2021] Mengxi Guo, Danqing Huang, and Xiaodong Xie.  The layout generation algorithm of graphic design based on transformer-cvae.  In CONF-SPML, 2021.
  * Gupta et al. [2021] Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry S Davis, Vijay Mahadevan, and Abhinav Shrivastava.  Layouttransformer: Layout generation and completion with self-attention.  In ICCV, 2021.
  * Hao et al. [2023] Xixuan Hao, Danqing Huang, Jieru Lin, and Chin-Yew Lin.  Relation-enhanced detr for component detection in graphic design reverse engineering.  IJCAI, 2023.
  * Havasi et al. [2010] Catherine Havasi, Robyn Speer, and Justin Holmgren.  Automated color selection using semantic knowledge.  In AAAI, 2010.
  * He et al. [2021] Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, and Jindong Chen.  Actionbert: Leveraging user actions for semantic understanding of user interfaces.  AAAI, 2021.
  * Heath et al. [2014] Derrall Heath, David Norton, and Dan Ventura.  Conveying semantics through visual metaphor.  ACM Trans. Intell. Syst. Technol., 2014.
  * Huang et al. [2021] Danqing Huang, Jinpeng Wang, Guoxin Wang, and Chin-Yew Lin.  Visual style extraction from chart images for chart restyling.  ICPR, 2021.
  * Indurkhya and Ojha [2017] Bipin Indurkhya and Amitash Ojha.  Interpreting visual metaphors: Asymmetry and reversibility.  Poetics Today, 2017.
  * Jahanian et al. [2013] Ali Jahanian, Jerry Liu, Qian Lin, Daniel R Tretter, Eamonn O’Brien-Strain, Seungyon Lee, Nic Lyons, and Jan P Allebach.  Automatic design of colors for magazine covers.  In Imaging and Printing in a Web 2.0 World IV, 2013.
  * Jain et al. [2022] Ajay Jain, Amber Xie, and Pieter Abbeel.  Vectorfusion: Text-to-svg by abstracting pixel-based diffusion models.  arXiv preprint, 2022.
  * Jiang et al. [2019] Shuhui Jiang, Zhaowen Wang, Aaron Hertzmann, Hailin Jin, and Yun Fu.  Visual font pairing.  TMM, 2019.
  * Jiang et al. [2022a] Zhaoyun Jiang, Huayu Deng, Zhongkai Wu, Jiaqi Guo, Shizhao Sun, Vuksan Mijovic, Zijiang Yang, Jian-Guang Lou, and Dongmei Zhang.  Unilayout: Taming unified sequence-to-sequence transformers for graphic layout generation.  arXiv preprint, 2022.
  * Jiang et al. [2022b] Zhaoyun Jiang, Shizhao Sun, Jihua Zhu, Jian-Guang Lou, and Dongmei Zhang.  Coarse-to-fine generative modeling for graphic layouts.  In AAAI, 2022.
  * Jin et al. [2022] Chuhao Jin, Hongteng Xu, Ruihua Song, and Zhiwu Lu.  Text2poster: Laying out stylized texts on retrieved images.  In ICASSP, 2022.
  * Jongejan et al. [2016] J. Jongejan, H. Rowley, T. Kawashima, J. Kim, and N. Fox-Gieg.  The Quick, Draw! - A.I. Experiment. 2016\.
  * Kang et al. [2021] Youwen Kang, Zhida Sun, Sitong Wang, Zeyu Huang, Ziming Wu, and Xiaojuan Ma.  Metamap: Supporting visual metaphor ideation through multi-dimensional example-based exploration.  In CHI, 2021.
  * Kikuchi et al. [2021] Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi.  Constrained graphic layout generation via latent optimization.  In ACM MM, 2021.
  * Kim et al. [2022] Eunseo Kim, Jeongmin Hong, Hyuna Lee, and Minsam Ko.  Colorbo: Envisioned mandala coloringthrough human-ai collaboration.  In IUI, 2022.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint, 2013.
  * Kobayashi [1991] Shigenobu Kobayashi.  Color Image Scale.  1991\.
  * Kong et al. [2022] Xiang Kong, Lu Jiang, Huiwen Chang, Han Zhang, Yuan Hao, Haifeng Gong, and Irfan Essa.  Blt: bidirectional layout transformer for controllable layout generation.  In ECCV, 2022.
  * Lee et al. [2020] Hsin-Ying Lee, Lu Jiang, Irfan Essa, Phuong B Le, Haifeng Gong, Ming-Hsuan Yang, and Weilong Yang.  Neural design network: Graphic layout generation with constraints.  In ECCV, 2020.
  * Lee et al. [2022] Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova.  Pix2struct: Screenshot parsing as pretraining for visual language understanding.  arXiv preprint, 2022.
  * Li et al. [2019] Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu.  Layoutgan: Generating graphic layouts with wireframe discriminators.  In ICLR, 2019.
  * Li et al. [2021] Da-Wei Li, Danqing Huang, Tingting Ma, and Chin-Yew Lin.  Towards topic-aware slide generation for academic papers with unsupervised mutual learning.  In AAAI, 2021.
  * Lin et al. [2020] Hangyu Lin, Yanwei Fu, Xiangyang Xue, and Yu-Gang Jiang.  Sketch-bert: Learning sketch bidirectional encoder representation from transformers by self-supervised learning of sketch gestalt.  In CVPR, 2020.
  * Liu et al. [2022] Ying-Tian Liu, Yuan-Chen Guo, Yi-Xiao Li, Chen Wang, and Song-Hai Zhang.  Learning implicit glyph shape representation.  TVCG, 2022.
  * Lopes et al. [2019] Raphael Gontjio Lopes, David Ha, Douglas Eck, and Jonathon Shlens.  A learned representation for scalable vector graphics.  In ICCV, 2019.
  * Manandhar et al. [2020] Dipu Manandhar, Dan Ruta, and John Collomosse.  Learning structural similarity of user interface layouts using graph networks.  In ECCV, 2020.
  * Manandhar et al. [2021] Dipu Manandhar, Hailin Jin, and John Collomosse.  Magic layouts: Structural prior for component detection in user interface designs.  In CVPR, 2021.
  * O’Donovan et al. [2014] Peter O’Donovan, Jundefinednis Lundefinedbeks, Aseem Agarwala, and Aaron Hertzmann.  Exploratory font selection using crowdsourced attributes.  ACM Trans. Graph., 2014.
  * Pang et al. [2016] Xufang Pang, Ying Cao, Rynson W. H. Lau, and Antoni B. Chan.  Directing user attention via visual flow on web designs.  ACM Trans. Graph., 2016.
  * Patil et al. [2020] Akshay Gadi Patil, Omri Ben-Eliezer, Or Perel, and Hadar Averbuch-Elor.  Read: Recursive autoencoders for document layout generation.  In CVPR Workshops, 2020.
  * Qiu et al. [2022] Qianru Qiu, Xueting Wang, Mayu Otani, and Yuki Iwazaki.  Color recommendation for vector graphic documents based on multi-palette representation.  arXiv preprint, 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint, 2022.
  * Rastogi et al. [2021] Abhinav Kumar Rastogi, Blaise Aguera y Arcas, Chongyang Bai, Jindong (JD) Chen, Srinivas Kumar Sunkara, Xiaoxue Zang, and Ying Xu.  Uibert: Learning generic multimodal representations for ui understanding.  In IJCAI, 2021.
  * Reddy et al. [2021a] Pradyumna Reddy, Michaël Gharbi, Michal Lukac, and Niloy J. Mitra.  Im2vec: Synthesizing vector graphics without vector supervision.  In CVPR, 2021.
  * Reddy et al. [2021b] Pradyumna Reddy, Zhifei Zhang, Zhaowen Wang, Matthew Fisher, Hailin Jin, and Niloy Mitra.  A multi-implicit neural representation for fonts.  In Advances in Neural Information Processing Systems, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * Shi et al. [2022] Danqing Shi, Weiwei Cui, Danqing Huang, Haidong Zhang, and Nan Cao.  Reverse-engineering information presentations: Recovering hierarchical grouping from layouts of visual elements.  arXiv preprint arXiv:2201.05194, 2022.
  * Solli and Lenz [2010] Martin Solli and Reiner Lenz.  Color semantics for image indexing.  In CGIV/MCS, 2010.
  * Sun [2006] Hung-Ming Sun.  Multi-linguistic optical font recognition using stroke templates.  In ICPR, 2006.
  * Vaddamanu et al. [2022] Praneetha Vaddamanu, Vinay Aggarwal, Bhanu Prakash Reddy Guda, Balaji Vasan Srinivasan, and Niyati Chhaya.  Harmonized banner creation from multimodal design assets.  In CHI, 2022.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * Wang et al. [2015] Zhangyang Wang, Jianchao Yang, Hailin Jin, Eli Shechtman, Aseem Agarwala, Jonathan Brandt, and Thomas S. Huang.  Deepfont: Identify your font from an image.  In ACM MM, 2015.
  * Weingarten et al. [2020] Evan Weingarten, Michael Meyer, Amit Shkenazi, and On Amir.  Human experts outperform technology in creative markets.  She Ji The Journal of Design Economics and Innovation, 6:301, 10 2020.
  * Weng et al. [2023] Haohan Weng, Danqing Huang, Tong Zhang, and Chin-Yew Lin.  Learn and sample together: Collaborative generation for graphic design layout.  In IJCAI, 2023.
  * Wu et al. [2021] Xingjiao Wu, Yingbin Zheng, Tianlong Ma, Hao Ye, and Liang He.  Document image layout analysis via explicit edge embedding network.  Information Sciences, 2021.
  * Xie et al. [2021] Yuxi Xie, Danqing Huang, Jinpeng Wang, and Chin-Yew Lin.  Canvasemb: Learning layout representation with large-scale pre-training for graphic design.  In ACM MM, 2021.
  * Xu and Miller [2016] Zhen Xu and James Miller.  Identifying semantic blocks in web pages using gestalt laws of grouping.  World Wide Web, 2016.
  * Yamaguchi [2021] Kota Yamaguchi.  Canvasvae: Learning to generate vector graphic documents.  In ICCV, 2021.
  * Yang et al. [2016] Xuyong Yang, Tao Mei, Ying-Qing Xu, Yong Rui, and Shipeng Li.  Automatic generation of visual-textual presentation layout.  TOMM, 2016.
  * Yang et al. [2023] Huiting Yang, Danqing Huang, Chin-Yew Lin, and Shengfeng He.  Layout generation as intermediate action sequence prediction.  In AAAI, 2023.
  * Yuan et al. [2021] Lin-Ping Yuan, Ziqi Zhou, Jian Zhao, Yiqiu Guo, Fan Du, and Huamin Qu.  Infocolorizer: Interactive recommendation of color palettes for infographics.  TVCG, 2021.
  * Zhang et al. [2020] Peiying Zhang, Chenhui Li, and Changbo Wang.  Smarttext: Learning to generate harmonious textual layout over natural image.  In ICME, 2020.
  * Zhao et al. [2018a] Nanxuan Zhao, Ying Cao, and Rynson W. H. Lau.  Modeling fonts in context: Font prediction on web designs.  Computer Graphics Forum, 2018.
  * Zhao et al. [2018b] Nanxuan Zhao, Ying Cao, and Rynson W. H. Lau.  What characterizes personalities of graphic designs?  ACM Trans. Graph., 2018.
  * Zheng et al. [2018] Quanlong Zheng, Jianbo Jiao, Ying Cao, and Rynson W.H. Lau.  Task-driven webpage saliency.  In ECCV, 2018.
  * Zheng et al. [2019] Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH Lau.  Content-aware generative modeling of graphic design layouts.  TOG, 2019.
  * Zhou et al. [2022] Min Zhou, Chenchen Xu, Ye Ma, Tiezheng Ge, Yuning Jiang, and Weiwei Xu.  Composition-aware graphic layout gan for visual-textual presentation designs.  arXiv preprint, 2022.