# The fundamental $1/f$ noise in monolayer graphene

Kirill A. Kazakov  Department of Theoretical Physics, Physics Faculty,
Moscow State University, $119991$, Moscow, Russian Federation

###### Abstract

The quantum indeterminacy caused by non-commutativity of observables at
different times sets a lower bound on the voltage noise power spectrum in any
conducting material. This bound is calculated explicitly in the case of
monolayer graphene. It is found that account of graphene pseudospin/valley
band structure raises the quantum bound by a factor of $7/2$ compared to the
case of spinless charge carriers with a conical energy-momentum dispersion.
The bound possesses all characteristic properties of $1/f$ noise, and its
dependence on the charge carrier density is congruent to the experimentally
observed.

1/f noise, quantum bound, pseudospin, graphene, Schwinger-Keldysh formalism

###### pacs:

42.50.Lc, 72.70.+m

Among numerous fluctuation phenomena found in conducting media, the $1/f$
(flicker) noise is by far the least understood despite its ubiquity and
practical importance. This notion covers all electrical fluctuations whose
power spectral density $S(f)\sim 1/f^{\gamma}$ at low frequencies $f,$ where
the frequency exponent $\gamma$ is around unity johnson ; buckingham ;
bell1980 ; raychaudhuri . On the one hand, this noise limits performance of
microelectronic devices, but on the other hand, measurements of the $1/f$
spectrum are widely used to identify certain physical processes in various
materials and to assess structural properties of samples. An adequate
characterization of the possible $1/f$-spectra is thus vital for these
applications.

It is generally accepted that $1/f$ noise is due to fluctuations in electrical
conductivity caused by various physical processes such as charge carrier
trapping, temperature fluctuations, generation-recombination of carriers in
semiconductors, impurity migration, etc. An important common feature of the
corresponding power spectra is that they all eventually flatten at
sufficiently low frequencies, so that their $1/f$-components cover only a few
frequency decades. At the same time, $1/f$ noise has been experimentally
detected down to frequencies as low as $f=10^{-6.3}$ Hz, with no sign of a
low-frequency cutoff rollin1953 ; caloyannides . As this is far below of any
characteristic frequency of the aforementioned conventional physical
processes, one is led to the problem of origin of fundamental flicker noise,
which is the component of $1/f$ spectrum presumably extending down to $f=0.$
It was suggested long ago by P. Handel that its origin might be in the well-
known quantum infrared divergence in the coupling of electric charges to
photons handel1 ; handel2 . Irrespective of the severe critics that followed
Handel’s derivation, his theory predicts that the frequency exponent is
strictly less than unity, $1>\gamma=1-O(\alpha),$ where $\alpha\approx 1/137$
is the fine structure constant. Though the deviation from unity is tiny, it
ensures convergence of the total noise power in Handel’s theory handel1 ;
handel7 . However, the numerous flicker noise measurements unequivocally
demonstrate that $(\gamma-1)$ can be positive as well as negative (it is
actually predominantly positive, as e.g., in Ref. caloyannides ).

It is the case $\gamma>1$ that makes the problem of fundamental noise so
acute, because of an apparent conflict between the observed absence of a low-
frequency cutoff and finiteness of the noise variance. In fact, a direct
consequence of the celebrated Wiener-Khinchin relation wiener ; khinchin  is
that this variance is equal to $2\int_{0}^{\infty}{\rm d}fS(f),$ which is
formally infinite because the integral diverges at $f=0$ whenever $S(f)\sim
1/f^{\gamma}$ with $\gamma>1.$ Yet, it was shown recently that account of the
quantum nature of charge carrier interaction with photons does resolve this
paradox, though in a way quite different from Handel’s kazakov1 ; kazakov2 .
First of all, it turns out that quantum theory implies certain limitations on
applicability of the Wiener-Khinchin relation to the voltage measurements, and
second, it sets a lower bound on the spectral power of voltage fluctuations
which possesses all characteristic features of the observed $1/f$ noise. In
Refs. kazakov1 ; kazakov2  this bound was calculated explicitly for materials
with a parabolic energy-momentum dispersion of charge carriers, which allowed
a direct comparison of the result with experiment. Such a comparison with
$1/f$ noise measured in InGaAs quantum wells and high-temperature
superconductors has revealed that the observed noise levels are only a few
times as high as the bound established.

An important property of the quantum bound is its inverse proportionality to
the charge carrier mass. This raises a curious question of what the noise
magnitude is in the case of conical energy-momentum dispersion (massless
charge carriers). The answer is that the zero-mass singularity of the power
spectrum for parabolic dispersion transmutes into a zero-momentum singularity
of its decomposition with respect to the charge carrier momentum kazakov3 . A
measurable consequence of this singularity is a sharp peak in the noise
magnitude at small charge carrier density. This result was applied in Ref.
kazakov3  to monolayer graphene, in which case the peak becomes M-shaped on
account of a continuous transition from the electron to hole conductivity. A
comparison with experimental data in GraFETs has shown that the calculated
power spectrum, considered as a function of the charge-carrier density, is
congruent to the observed, though the excess of the measured noise magnitudes
over the bound is more pronounced than in the case of parabolic dispersion. On
the other hand, this comparison was only qualitative, as accurate as a mere
replacement of the dispersion law permits. To obtain a quantitatively reliable
prediction, and in particular, to decide if graphene really exhibits excessive
noise, calls for a more careful account of the underlying physics. The purpose
of this Letter is to make a first step in this direction and to obtain the
quantum bound taking into account the charge carrier pseudospin.

Before going into details of the calculation, it is useful to summarize the
main points of the new approach. Consider a (semi)conducting sample biased so
that a constant-on-average electric field $\bm{E}$ is established within, and
let the voltage across the sample be measured by means of two probes assumed
for simplicity pointlike, $\bm{x}_{1},\bm{x}_{2}$ denoting their position. A
voltage measured at time $t$ is the sum of a mean value
$U_{0}(\bm{x}_{1},\bm{x}_{2})$ and a fluctuation, or noise, $\Delta
U(t,\bm{x}_{1},\bm{x}_{2}).$ The voltage fluctuation $\Delta
U(t,\bm{x}_{1},\bm{x}_{2})$ is an observable to which there corresponds a
Hermitian (Heisenberg) operator $\widehat{\Delta U}(t)$ (for brevity, the
arguments $\bm{x}_{1},\bm{x}_{2}$ in $\widehat{\Delta U}$ are henceforth
omitted). According to Refs. kazakov1 ; kazakov2 , the quantum bound on the
power spectral density of voltage fluctuations, $S_{F}(f),$ arises from the
quantum indeterminacy caused by non-commutativity of observables at different
times. It is this indeterminacy that obstructs applicability of the Wiener-
Khinchin relation to voltage fluctuations, which in turn is the reason why the
quantum bound needs no low-frequency cutoff. In order to find $S_{F}(f),$ one
first defines a correlation function
$S(t-t^{\prime})=\left\langle\widehat{\Delta U}(t)\widehat{\Delta
U}(t^{\prime})\right\rangle,$ where angular brackets denote the quantum-
mechanical average over a fixed state of the system “conducting sample plus
electromagnetic field.” $S(t-t^{\prime})$ can be written as the sum of two
terms, one of which is symmetric under the interchange $t\leftrightarrow
t^{\prime},$ and the other is antisymmetric. As was demonstrated in Refs.
kazakov1 ; kazakov2 , the latter admits a Fourier decomposition despite the
presence of $1/f$-term in the power spectrum, and defines $S_{F}(f)$ according
to

$\displaystyle
S_{F}(f)=\left|\lim\limits_{t_{m}\to\infty}\left\\{\int_{-t_{m}}^{t_{m}}{\rm
d}\tau S(\tau)\sin(\omega\tau)-\frac{1}{t_{m}}\int_{-t_{m}}^{t_{m}}{\rm
d}\tau|\tau|S(\tau)\sin(\omega\tau)\right\\}\right|,\quad\omega=2\pi f.$ (1)

It is a key fact of the present approach that the limit under the modulus sign
in Eq. (1) exists even when $S(\tau)\sim\tau^{\gamma-1}$ for $\tau\to\infty,$
as is the case when $S(f)\sim 1/f^{\gamma}$ at low frequencies.

A straightforward way to compute the expectation value
$\left\langle\widehat{\Delta U}(t)\widehat{\Delta U}(t^{\prime})\right\rangle$
is to use the Schwinger-Keldysh technique schwinger ; keldysh , according to
which it is written as

$\displaystyle\left\langle\widehat{\Delta U}(t)\widehat{\Delta
U}(t^{\prime})\right\rangle={\rm
tr}\left(\hat{\rho}\,\mathcal{T}_{C}\widehat{\Delta u}^{(2)}(t)\widehat{\Delta
u}^{(1)}(t^{\prime})\exp\left\\{-{\rm i}\int_{C}{\rm
d}t\,\hat{w}(t)\right\\}\right),$ (2)

where $\widehat{\Delta
u}(t)=\hat{a}_{0}(t,\bm{x}_{1})-\hat{a}_{0}(t,\bm{x}_{2})-U_{0},$ with
$\hat{a}_{0}$ the scalar electromagnetic potential, $\hat{\rho}$ is the system
density matrix, $\hat{w}$ is the interaction Hamiltonian, and the lowercase
letters denote operators in the interaction picture; the so-called Schwinger-
Keldysh time contour $C$ runs from $t=-\infty$ to $t=+\infty$, and then back
to $t=-\infty,$ with all time instants on the forward branch [designated with
a superscript $(1)$] treated as being in the past with respect to any time
instant on the backward branch [designated with a superscript $(2)$];
$\mathcal{T}_{C}$ denotes operator ordering along this contour.

Let us now specify physical conditions under which the power spectrum of
voltage fluctuations in graphene will be evaluated. As is well known, the
conical energy-momentum dispersion of charge carriers is found in monolayer
graphene geim2009 . Therefore, the sample is a planar monolayer of carbon
atoms arranged in a hexagonal lattice. The energy-momentum dispersion is
approximately conical near the so-called Dirac points which are two corners of
the graphene Brillouin zone wallace1947 . In the approximation where the
electron hopping between nearest sites of the two triangular sublattices is
only taken into account, the effective free electron Hamiltonian reads
semenoff1984

$\displaystyle\hat{H}_{0}=v_{F}\int{\rm
d}^{2}\bm{x}\left[\hat{\Psi}_{1}^{\dagger}(\bm{x})\bm{\sigma}\cdot(-{\rm
i}\hbar\bm{\nabla})\hat{\Psi}_{1}(\bm{x})+\hat{\Psi}_{2}^{\dagger}(\bm{x})\bm{\sigma}^{*}\cdot(-{\rm
i}\hbar\bm{\nabla})\hat{\Psi}_{2}(\bm{x})\right],$ (3)

where the matrix vectors $\bm{\sigma}=(\sigma_{x},\sigma_{y}),$
$\bm{\sigma}^{*}=(\sigma_{x},-\sigma_{y}),$ $\sigma_{x},\sigma_{y}$ being the
standard Pauli matrices in the sample plane, and $\hat{\Psi}_{1,2}(\bm{x})$
are the (pseudo)spinor electron fields, the subscripts 1,2 referring to the
two Dirac points. Thus, the electrons are massless Dirac-like particles with
the energy-momentum dispersion $\varepsilon(\bm{q})=\pm v_{F}|\bm{q}|$ (Dirac
point being at $\bm{q}=0$), so that the charge carriers include two copies of
electrons ($\varepsilon>0$) and holes ($\varepsilon<0$), all characterized by
the same value of Fermi velocity $v_{F}\approx 10^{8}\,$cm/s. Except in the
number of states counting, the ordinary spin of charge carriers does not play
a role in the present consideration, and so the charge carriers are assumed
unpolarized, with all spin indices accordingly suppressed. At last, also for
simplicity, all finite-temperature effects are neglected. From the
experimental standpoint, dependence of the power spectrum on the charge
carrier density is of major interest. As a means to control this density, it
is best to consider the graphene sample as part of a field-effect transistor –
GraFET.

Interaction with the external fields is introduced, as usual, by invoking the
principle of gauge invariance to replace the ordinary space-time derivatives
by covariant ones in the Schr$\ddot{\rm o}$dinger equation. This yields the
interaction Hamiltonian, in the interaction picture,

$\displaystyle\hat{w}(t)=\int{\rm
d}^{2}\bm{x}\left\\{v_{F}\frac{e}{c}\left[\hat{\psi}_{1}^{\dagger}\bm{\sigma}\cdot\hat{\bm{a}}\hat{\psi}_{1}+\hat{\psi}_{2}^{\dagger}\bm{\sigma}^{*}\cdot\hat{\bm{a}}\hat{\psi}_{2}\right]+e\hat{a}_{0}[\hat{\psi}_{1}^{\dagger}\hat{\psi}_{1}+\hat{\psi}_{2}^{\dagger}\hat{\psi}_{2}]\right\\},$
(4)

where $\hat{\bm{a}}$ is the electromagnetic vector potential. Since the charge
carriers are non-relativistic ($v_{F}\ll c,$ the speed of light in vacuum),
$\hat{\bm{a}}$ in Eq. (4) can be replaced by the classical vector potential of
the electric field, $\bm{E},$ established in the sample,

$\displaystyle{\bm{a}}(t,\bm{x})={\rm i}\bm{E}(\bm{x})\frac{{\rm e}^{{\rm
i\lambda t}}-1}{\lambda}\,,\quad\lambda\to 0.$ (5)

The set of lowest-order nonvanishing Schwinger-Keldysh diagrams contributing
to the $1/f$ asymptotic of the power spectrum are drawn in Fig. 1. Each
diagram involves two vertices of the charge carrier interaction with the
external electric field. As was found in Ref. kazakov3 , similar diagrams
generated by the model Hamiltonian that neglects the electron pseudospin
cancel each other, so that the non-trivial contribution comes only from
diagrams involving vertices quadratic in $\bm{E}.$ In contrast, there are no
such vertices in the interaction Hamiltonian (4), but the sum of diagrams in
Fig. 1 no longer vanishes because of the charge carrier pseudospin fully taken
into account by this Hamiltonian. In any case, the power spectrum in the
lowest, forth order with respect to the electron charge is quadratic with
respect to $\bm{E}.$ On the other hand, a consistent evaluation of the higher
order corrections would require explicit account of the charge carrier
collisions kazakov2 .

Figure 1: Non-vanishing diagrams representing $\left\langle\widehat{\Delta
U}(t)\widehat{\Delta U}(t^{\prime})\right\rangle$ in the Schwinger-Keldysh
formalism. Solid and wavy lines depict propagators of the charge carriers and
electromagnetic field, respectively, dashed line – charge carrier density
matrix, dotted line – external electric field. The arrows on the lines show
the energy-momentum flow.

The sum of diagrams in Fig. 1 reads, in relativistic units $\hbar=c=1$ (it is
doubled to account for the equal contributions of the fields $\psi_{1}$ and
$\psi_{2}$)

$\displaystyle S(\tau)$ $\displaystyle=$ $\displaystyle
v^{2}_{F}(4\pi)^{2}e^{4}\frac{\partial^{2}}{\partial\lambda^{\prime}\partial\lambda^{\prime\prime}}\int\frac{{\rm
d}^{4}k}{(2\pi)^{4}}\frac{{\rm d}^{3}\bm{q}}{(2\pi)^{3}}\frac{{\rm
d}^{3}\bm{q}^{\prime}}{(2\pi)^{3}}\varrho(\bm{q},\bm{q}^{\prime})\left({\rm
e}^{{\rm i}\bm{k}\cdot(\bm{x}_{1}-\bm{x}_{2})}-1\right){\rm e}^{{\rm
i}(\bm{q}-\bm{q}^{\prime})\cdot\bm{x}_{1}}$ (6)
$\displaystyle\times\left\\{{\rm e}^{-{\rm i}k^{0}\tau-{\rm
i}(\lambda^{\prime}+\lambda^{\prime\prime})t}G^{(11)}(q-q^{\prime}+k+k^{\prime}+k^{\prime\prime})G^{(12)}(k)\right.$
$\displaystyle\left.\hskip 17.07182pt\times{\rm
tr}\left[D^{(11)}(q+k+k^{\prime}+k^{\prime\prime})\bm{\sigma}\cdot\bm{E}D^{(11)}(q+k+k^{\prime})\bm{\sigma}\cdot\bm{E}D^{(11)}(q+k)\right.\right.$
$\displaystyle\left.\left.\hskip
17.07182pt+D^{(11)}(q^{\prime}-k)\bm{\sigma}\cdot\bm{E}D^{(11)}(q^{\prime}-k-k^{\prime})\bm{\sigma}\cdot\bm{E}D^{(11)}(q^{\prime}-k-k^{\prime}-k^{\prime\prime})\right]\right.$
$\displaystyle\left.\hskip 17.07182pt+{\rm e}^{{\rm i}k^{0}\tau-{\rm
i}(\lambda^{\prime}+\lambda^{\prime\prime})t^{\prime}}G^{(22)}(q-q^{\prime}+k+k^{\prime}+k^{\prime\prime})G^{(21)}(k)\right.$
$\displaystyle\left.\hskip 17.07182pt\times{\rm
tr}\left[D^{(22)}(q+k+k^{\prime}+k^{\prime\prime})\bm{\sigma}\cdot\bm{E}D^{(22)}(q+k+k^{\prime})\bm{\sigma}\cdot\bm{E}D^{(22)}(q+k)\right.\right.$
$\displaystyle\left.\left.\hskip
17.07182pt+D^{(22)}(q^{\prime}-k)\bm{\sigma}\cdot\bm{E}D^{(22)}(q^{\prime}-k-k^{\prime})\bm{\sigma}\cdot\bm{E}D^{(22)}(q^{\prime}-k-k^{\prime}-k^{\prime\prime})\right]\right\\}\Bigr{|}_{\lambda=\lambda^{\prime}=0}$
$\displaystyle+(\bm{x}_{1}\leftrightarrow\bm{x}_{2}),$

where a 4-vector notation is used,
$k=(k^{0},\bm{k}),q=(\varepsilon(\bm{q}),\bm{q}),$
$k^{\prime}=(\lambda^{\prime},\bm{0}),$
$k^{\prime\prime}=(\lambda^{\prime\prime},\bm{0});$ $G$ and $D$ are the
momentum-space Schwinger-Keldysh propagators of temporal photons and charge
carriers, respectively,

$\displaystyle G^{(11)}(k)$ $\displaystyle=$ $\displaystyle\frac{{\rm
i}}{k^{2}+{\rm i}0}=[G^{(22)}(k)]^{*},\quad
G^{(12)}(k)=2\pi\theta(-k^{0})\delta(k^{2}),\quad
G^{(21)}(k)=2\pi\theta(k^{0})\delta(k^{2}),$ $\displaystyle D^{(11)}(p)$
$\displaystyle=$ $\displaystyle{\rm
i}\left\\{p^{0}-v_{F}\bm{\sigma}\cdot\bm{p}+{\rm i}0\right\\}^{-1}={\rm
i}\frac{p^{0}+v_{F}\bm{\sigma}\cdot\bm{p}}{(p^{0}+{\rm
i}0)^{2}-v^{2}_{F}\bm{p}^{2}}\,,\quad D^{(22)}(q)=-{\rm
i}\frac{p^{0}+v_{F}\bm{\sigma}\cdot\bm{p}}{(p^{0}-{\rm
i}0)^{2}-v^{2}_{F}\bm{p}^{2}}\,,$

and the step function $\theta(k^{0})=0$ for $k^{0}\leqslant 0$,
$\theta(k^{0})=0$ for $k^{0}>0$; the trace is over the charge carrier
pseudospin; since the photon heat bath effects are neglected, $G$’s are purely
vacuum; accordingly, the system density matrix $\hat{\rho}$ is reduced to that
of the charge carriers, $\varrho(\bm{q},\bm{q}^{\prime})$ denoting its
momentum-space representation; the latter is normalized by

$\displaystyle\int\frac{{\rm
d}^{3}\bm{q}}{(2\pi)^{3}}\varrho(\bm{q},\bm{q})=1.$ (7)

At last, “$+(\bm{x}_{1}\leftrightarrow\bm{x}_{2})$” means that the preceding
expression is to be added with $\bm{x}_{1}$ and $\bm{x}_{2}$ interchanged.

To extract the low-frequency asymptotic of the power spectrum, we note that at
frequencies $k^{0}$ typical for the flicker noise studies, the photon
4-momentum $k$ associated with the propagators
$G^{(12)},G^{(21)}\sim\delta(k^{2})$ is negligibly small compared to that of
the charge carriers. With a great accuracy, therefore, denominators of the
charge carrier propagators can be simplified as

$(q^{0}+k^{0})^{2}-v^{2}_{F}(\bm{q}+\bm{k})^{2}\approx 2v_{F}|\bm{q}|k^{0},$

etc. Next, the traced expressions in the numerator are of the form

$\displaystyle{\rm
tr}\left[(s^{0}+v_{F}\bm{\sigma}\cdot\bm{s})\bm{\sigma}\cdot\bm{E}(s^{0}\pm\lambda^{\prime}+v_{F}\bm{\sigma}\cdot\bm{s})\bm{\sigma}\cdot\bm{E}(s^{0}\pm\lambda^{\prime}\pm\lambda^{\prime\prime}+v_{F}\bm{\sigma}\cdot\bm{s})\right]$
$\displaystyle=2\bm{E}^{2}s^{0}(s^{0}\pm\lambda^{\prime})(s^{0}\pm\lambda^{\prime}\pm\lambda^{\prime\prime})-2v^{2}_{F}\bm{E}^{2}\bm{s}^{2}(s^{0}\pm\lambda^{\prime})+4v^{2}_{F}(\bm{s}\cdot\bm{E})^{2}(2s^{0}\pm\lambda^{\prime}\pm\lambda^{\prime\prime}),$

where $s=q+k,q^{\prime}-k.$ Neglecting, as before, $\bm{k}$ in comparison with
$\bm{q},\bm{q}^{\prime}$ and assuming for simplicity that the momentum
distribution of charge carriers is isotropic in the sample plane,
$(\bm{s}\cdot\bm{E})^{2}\approx(\bm{q}\cdot\bm{E})^{2}$ in the last term can
be replaced by $\bm{q}^{2}\bm{E}^{2}/2$ (though the factor ${\rm e}^{{\rm
i}(\bm{q}-\bm{q}^{\prime})\cdot\bm{x}_{1}}$ in the integrand of Eq. (6)
explicitly breaks the isotropy, it is not difficult to see from the subsequent
calculations that this only adds terms which vanish unless the charge carrier
is localized on a voltage probe). Written longhand, expression (6) thus takes
the form

$\displaystyle S(\tau)$ $\displaystyle=$
$\displaystyle\frac{4\pi^{2}e^{4}\bm{E}^{2}}{v_{F}}\frac{\partial^{2}}{\partial\lambda^{\prime}\partial\lambda^{\prime\prime}}\int\frac{{\rm
d}^{4}k}{(2\pi)^{3}}\frac{{\rm d}^{3}\bm{q}}{(2\pi)^{3}}\frac{{\rm
d}^{3}\bm{q}^{\prime}}{(2\pi)^{3}}\varrho(\bm{q},\bm{q}^{\prime})\left({\rm
e}^{{\rm i}\bm{k}\cdot(\bm{x}_{1}-\bm{x}_{2})}-1\right)$ (8)
$\displaystyle\times{\rm e}^{{\rm
i}(\bm{q}-\bm{q}^{\prime})\cdot\bm{x}_{1}}\delta(k^{2})\frac{{\rm e}^{-{\rm
i}k^{0}\tau-{\rm
i}(\lambda^{\prime}+\lambda^{\prime\prime})t}\theta(-k^{0})+{\rm e}^{{\rm
i}k^{0}\tau-{\rm
i}(\lambda^{\prime}+\lambda^{\prime\prime})t^{\prime}}\theta(k^{0})}{(q-q^{\prime})^{2}k^{0}(k^{0}+\lambda^{\prime})(k^{0}+\lambda^{\prime}+\lambda^{\prime\prime})}$
$\displaystyle\times\left\\{\left(v_{F}+\frac{k^{0}+\lambda^{\prime}}{|\bm{q}|}\right)\left[\left(v_{F}+\frac{k^{0}}{|\bm{q}|}\right)\left(v_{F}+\frac{k^{0}+\lambda^{\prime}+\lambda^{\prime\prime}}{|\bm{q}|}\right)+v^{2}_{F}\right]\right.$
$\displaystyle\left.\hskip
14.22636pt-\left(v_{F}-\frac{k^{0}+\lambda^{\prime}}{|\bm{q}^{\prime}|}\right)\left[\left(v_{F}-\frac{k^{0}}{|\bm{q}^{\prime}|}\right)\left(v_{F}-\frac{k^{0}+\lambda^{\prime}+\lambda^{\prime\prime}}{|\bm{q}^{\prime}|}\right)+v^{2}_{F}\right]\right\\}\Biggl{|}_{\lambda=\lambda^{\prime}=0}$
$\displaystyle+(\bm{x}_{1}\leftrightarrow\bm{x}_{2}).$

Since $|\bm{k}|=|k^{0}|$ (because of the factor $\delta(k^{2})$), the
integrand is to be expanded with respect to $\bm{k}$ as well as to $k^{0}$
keeping only the leading term, e.g., $\exp[{\rm
i}\bm{k}\cdot(\bm{x}_{1}-\bm{x}_{2})]-1=-[\bm{k}\cdot(\bm{x}_{1}-\bm{x}_{2})]^{2}/2.$
Taking into account also that the expression in the braces is $O(k^{0})$ at
$\lambda^{\prime}=\lambda^{\prime\prime}=0,$ a simple power counting shows
that the integrand is $O(1/k^{0})$ for $k^{0}\to 0,$ as expected. It is also
readily checked that $\lambda^{\prime}$\- or
$\lambda^{\prime\prime}$-differentiation of the time-dependent exponential
factors gives rise to terms which are non-singular at $k^{0}=0.$ These
contribute only to the symmetric part of $S(\tau)$ and are ill-defined, as
this part lacks temporal Fourier decomposition kazakov2 . On the other hand,
the $k^{0}$-integral in the antisymmetric part is well-defined. A
straightforward calculation yields

$\displaystyle S(\tau)=$
$\displaystyle\frac{14v_{F}e^{4}\bm{E}^{2}(\bm{x}_{1}-\bm{x}_{2})^{2}}{3}\int_{0}^{\infty}{\rm
d}k^{0}\frac{{\rm e}^{{\rm i}k^{0}\tau}}{k^{0}}\int\frac{{\rm
d}^{3}\bm{q}}{(2\pi)^{3}}\frac{{\rm d}^{3}\bm{q}^{\prime}}{(2\pi)^{3}}$ (9)
$\displaystyle\times\varrho(\bm{q},\bm{q}^{\prime})\left(\frac{1}{|\bm{q}|}+\frac{1}{|\bm{q}^{\prime}|}\right)\frac{{\rm
e}^{{\rm i}(\bm{q}-\bm{q}^{\prime})\cdot\bm{x}_{1}}+{\rm e}^{{\rm
i}(\bm{q}-\bm{q}^{\prime})\cdot\bm{x}_{2}}}{(\bm{q}-\bm{q}^{\prime})^{2}}\,.$

Following Refs. kazakov1 ; kazakov2 , this expression can be made more
specific by expressing the charge carrier density matrix via the mixed
position-momentum distribution function $R(\bm{r},\bm{Q}),$

$\displaystyle\varrho\left(\bm{Q}-\frac{\bm{p}}{2},\bm{Q}+\frac{\bm{p}}{2}\right)=\frac{1}{\Omega}\int_{\Omega}{\rm
d}^{3}\bm{r}{\rm e}^{{\rm i}\bm{p}\cdot\bm{r}}R(\bm{r},\bm{Q}),$ (10)

where $\Omega$ is the sample volume. $R(\bm{r},\bm{Q})$ vanishes for $\bm{r}$
outside of the sample, hence, $(\bm{q}^{\prime}-\bm{q})=\bm{p}$ is of the
order of the inverse linear sample size which in practice is much larger than
the lattice constant. Therefore, $|\bm{p}|\ll|\bm{q}|$ for all relevant charge
carrier momenta, and so one can set $(1/|\bm{q}|+1|\bm{q}^{\prime}|)\approx
2/|\bm{Q}|.$ The integral over $\bm{p}$ in Eq. (9) is then just a Fourier
decomposition of the Coulomb potential. Finally, the power spectrum of voltage
fluctuations is found by substituting an odd in $\tau$ part of $S(\tau)$ into
Eq. (1). Replacing $\bm{E}^{2}(\bm{x}_{1}-\bm{x}_{2})^{2}$ by $U^{2}_{0}$ and
restoring ordinary units gives

$\displaystyle S_{F}(f)=$ $\displaystyle\frac{7v_{F}e^{4}U^{2}_{0}}{6\pi\hbar
c^{3}|f|}\frac{1}{\Omega}\int_{\Omega}{\rm
d}^{3}\bm{r}\left(\frac{1}{|\bm{r}-\bm{x}_{1}|}+\frac{1}{|\bm{r}-\bm{x}_{2}|}\right)\int\frac{{\rm
d}^{3}\bm{Q}}{(2\pi)^{3}}\frac{R(\bm{r},\bm{Q})}{|\bm{Q}|}\,.$ (11)

In the practically important case of macroscopically homogeneous sample,
$R(\bm{r},\bm{Q})=R(\bm{Q}),$ this expression simplifies to:

$\displaystyle S_{F}(f)=\frac{\varkappa
U^{2}_{0}}{|f|}\,,\quad\varkappa=\frac{7v_{F}e^{4}g}{2\pi\hbar
c^{3}}\int\frac{{\rm
d}^{2}\bm{Q}}{(2\pi\hbar)^{2}}\frac{R(\bm{Q})}{|\bm{Q}|}\,,$ (12)

where the geometric factor $g$ is defined following Ref. kazakov1 ,

$g=\frac{1}{3\Omega}\int_{\Omega}{\rm
d}^{3}\bm{r}\left(\frac{1}{|\bm{r}-\bm{x}_{1}|}+\frac{1}{|\bm{r}-\bm{x}_{2}|}\right).$

Except for the factor $7/2,$ Eq. (12) coincides with the expression derived in
Ref. kazakov3 . Thus, account of the detailed band structure of graphene
raises the lower bound on the power spectrum of flicker noise by a factor of
$7/2=2\times(7/4),$ where the factor $2$ is merely due to doubling of charge
carrier types, and the rest is specific to the charge carrier pseudospin.

For the purposes of comparison with experiment, the reduced power spectrum,
$\varkappa,$ is to be expressed as a function of the charge carrier density,
$n.$ Its analytical expression is easily derived under the assumption that
thermal effects are negligible. According to Eqs. (7), (10), $R(\bm{Q})$ is
normalized by

$\displaystyle\int\frac{{\rm d}^{2}\bm{Q}}{(2\pi\hbar)^{2}}R(\bm{Q})=1.$ (13)

At zero temperature, one has $R(\bm{Q})=4\theta(q_{F}-|\bm{Q})|)/n,$ where
$q_{F}$ is the Fermi momentum (the factor of four accounts for the electron
spin and the two Dirac points). Substitution of this relation in Eqs. (12),
(13) yields

$\varkappa=\frac{7v_{F}e^{4}gq_{F}}{\pi^{2}\hbar^{3}c^{3}n}\,,\quad
n=\frac{q^{2}_{F}}{\pi\hbar^{2}},$

hence,

$\displaystyle\varkappa=\frac{7v_{F}e^{4}g}{\pi^{3/2}\hbar^{2}c^{3}}n^{-1/2}\,.$
(14)

The unbounded growth of $\varkappa$ at small $n$ is a reflection of the
singularity at $\bm{Q}=0$ of the integrand in Eq. (12). Account of the thermal
effect on the charge carrier distribution removes this singularity, in which
case Eq. (14) plays the role of a large-$n$ asymptotic of this function.
However, the value $n=0$ is unattainable in practice anyway because of the
fluctuations in $n$ caused by disorder and charged impurities in the
substrate. The corresponding characteristic density fluctuation, $\delta n,$
depends essentially on the charged impurity concentration, and ranges normally
$10^{11}\,$cm-2 to $10^{12}\,$cm-2 hwang2007 . To give an idea of how Eq. (14)
applies at various temperatures, it is compared in Fig. 2(a) with the function
$\varkappa(n)$ evaluated taking into account thermal effect on the charge
carrier distribution kazakov3 .

Figure 2: (a) A semi-logarithmic plot of the reduced power spectrum,
$\varkappa,$ versus charge carrier density, $n,$ suggesting a singularity of
$\varkappa$ at $n=0$ (sample size $1\times 1\,$$\mu$m${}^{2},$ $g\approx
1.6\times 10^{4}\,$cm-1). Inset: logarithmic plots of $\varkappa(n)$ at
various temperatures showing that $\varkappa(0)$ is actually finite.
$T=300\,$K (dash), $T=88\,$K (dash-dots), and $T=20\,$K (dots); solid line is
the expression (14). (b) $\varkappa(n)$ measured in Ref. pal2011  in a field-
effect device SLG1 (marks, as read off from figure 1(d) of Ref. pal2011 ), and
calculated according to Eq. (14) (solid line). $n>0$ ($n<0$) is the electron
(hole) doping. Dashed curve is obtained by a vertical shift of the solid curve
to ease comparison with the experimental data (in view of the observed
electron-hole asymmetry, the two wings of the curve were shifted
independently).

The correction factor $7/2$ found above is important for identifying the
origin of flicker noise in graphene, as it shifts the quantum bound closer to
the experimentally observed values. This is illustrated in Fig. 2(b) where the
function $\varkappa(n)$ calculated according to Eq. (14) is compared to the
measured in a sample of size $3.5\times 4.5\,$$\mu$m2 pal2011  (in which case
$g\approx 4\times 10^{3}\,$cm-1). It is seen that the two sets of values are
nearly congruent, though the experimental values are three to five times as
large as the quantum bound. One might think that this discrepancy in the
magnitude rules out the quantum indeterminacy as a dominant noise source.
However, there is actually another important factor affecting the noise
magnitude which was not taken into account in the above calculations. Namely,
there is a natural interplay between the power spectrum magnitude and the
value of the frequency exponent, $\gamma.$ As $\gamma$ deviates from unity,
$\varkappa$ in Eq. (14) becomes dimensional,
$\varkappa\to\varkappa(f_{*})^{\delta},$ where $\delta\equiv\gamma-1$ and
$f_{*}$ is a frequency parameter. Its value is determined by the physical
process responsible for the deviation, and is naturally a large number when
measured in hertz. For instance, in the case of charge carrier–phonon
interaction, $f_{*}$ is the ratio of the sound speed to the lattice constant,
$f_{*}\approx 10^{13}$ kazakov2 . Therefore, already a comparatively small
deviation of $\gamma$ can noticeably change the magnitude of $S(f).$ At the
same time, $\gamma$ is not easy to measure accurately. Thus, e.g., a 0.05
error in $\gamma$ will bring a factor of $10^{13\cdot 0.05}\approx 5$ in
$\varkappa.$

Figure 3: (a) M-shaped $\varkappa(n)$s obtained as described in the text
without smearing (solid line), and smeared using a model weight function
$(1+\exp(n-n_{\rm i})/\delta n)^{-1}$ (dashed line). $n_{\rm i}=4\times
10^{12}\,$cm${}^{-2},$ $\delta n=10^{12}\,$cm${}^{-2}.$ (b) Same in a
suspended graphene flake of Ref. pal2011  at $T=300\,$K (solid line, circles),
$T=250\,$K (dashed line, diamonds), and $T=200\,$K (dotted line, boxes).
Suspension reduces the effect of substrate disorder, allowing thereby
resolution of the M-shape.

As is well-know, the experimentally measured $\varkappa(n)$s are often
M-shaped lin2008 ; pal2009 ; heller2010 ; zhang2011 ; pal2011 ; kaverzin2012 .
From the present theory standpoint, this readily follows from another widely
recognized fact that at sufficiently small charge carrier densities, the
charge transport is mixed because both types of charge carriers are present
simultaneously. Specifically, as the electron density drops below $n=n_{\rm
i}\approx 10^{11}\,$cm${}^{-2}-10^{12}\,$cm${}^{-2},$ graphene becomes
inhomogeneous because of the occurrence of hole “puddles,” which grow and
coalesce as the electron density decreases, until they fill up the whole
sample at $n=-n_{\rm i}$ hwang2007 . The fact that the average hole (electron)
density vanishes not at $n=0,$ but at $n=n_{\rm i}$ ($n=-n_{\rm i}$) means
that the left (right) wing of the curve $\varkappa(n)$ is to be shifted to the
right (to the left) by the amount of $n_{\rm i},$ and the true $\varkappa$ be
obtained by adding the two contributions. Strictly speaking, in view of the
sample inhomogeneity, $n\in(-n_{\rm i},n_{\rm i})$ no longer has the meaning
of charge carrier density (it is to be considered rather as a measure of the
gate voltage), and likewise Eq. (14) is not applicable because
$R(\bm{r},\bm{Q})$ in Eq. (11) does depend on $\bm{r}$ within the sample. But
if the puddles are distributed approximately uniformly throughout, and under
the assumption that the graphene band structure remains unchanged, Eq. (14) is
approximately valid. One then obtains curves with M-shaped peaks as shown in
Fig. 3(a). To account for the density fluctuations, $\varkappa(n)$ is to be
smeared on the scale $\delta n.$ The resulting $\varkappa(n)$s are
superimposed on the experimental data of Ref. pal2011  in Fig. 3(b) where
theoretical curves are obtained using the value $n_{\rm i}=5\times
10^{10}\,$cm-2 as estimated in Ref. pal2011 , and $\delta n$ chosen
$10^{11}\,$cm-2 to $10^{12}\,$cm-2 so as to obtain best fits to the
experimental data (the latter is read off from figure 2(c3) of Ref. pal2011
where the sets of marks for different temperatures were separated by shifting
them vertically; unfortunately, the authors of Ref. pal2011  were unable to
specify the shifts,111A.N. Pal, private communication. which does not allow
accurate comparison of the theoretical and observed noise magnitudes). It is
seen that as in the case of $\Lambda$-shapes in Fig. 2(b), the theoretical
M-shaped curves are congruent with the experimentally observed.

To summarize, account of the detailed band structure of graphene raised the
quantum bound kazakov3  on the $1/f$ noise by a factor of $7/2,$ nearing the
experimentally observed values. This result, together with the demonstrated
congruence of the calculated and measured noise magnitudes versus charge
carrier density, sharpens the question of the flicker noise origin in
graphene. It might be readily resolved if estimates of contributions from
other possible noise sources were available. However, a common view on the
flicker noise as originating from fluctuation of the electric charge
distribution around the graphene film still lacks a consistent theory,
remaining purely phenomenological. For instance, a variant of this approach
suggested in Ref. pal2011 , which considers the charge exchange between
graphene and its environment as well as the charge rearrangement within the
environment, involves four freely adjustable parameters including the noise
magnitude. In these circumstances, the issue can be settled only by
measurements which would allow sufficiently accurate determination of the
frequency exponent to reduce the uncertainty in the noise magnitude, and to
verify its dependence on the sample geometry predicted by Eq. (14).

## References

  * (1) J. B. Johnson, Phys. Rev. 26(1), 71 (1925).
  * (2) M. Buckingham, Noise in Electronic Devices and Systems (Ellis Horwood, Chichester, 1983).
  * (3) D. A. Bell, J. Phys. C: Solid St. Phys. 13, 4425 (1980).
  * (4) A.K. Raychaudhuri, Curr. Opin. Solid State Mater. Sci. 60, 67 (2002).
  * (5) B. V. Rollin and I. M. Templeton, Proc. Phys. Soc. B 66, 259 (1953).
  * (6) M. F. Caloyannides, J. Appl. Phys. 45, 307 (1974).
  * (7) P. H. Handel, Phys. Rev. 22, 745 (1980).
  * (8) P. H. Handel, Phys. Rev. Lett. 34, 1492 (1975).
  * (9) P. H. Handel, “Decoherence and conventional Quantum 1/f noise,” Proc. Int. Conf. on Noise and Fluctuations, June 2013 (Montpellier, France) page 169.
  * (10) N. Wiener, Acta Math. 55, 117 (1930).
  * (11) A. Khinchin, Math. Ann. 109, 604 (1934).
  * (12) K.A. Kazakov, Phys. Lett. A384, 126812 (2020).
  * (13) K.A. Kazakov, “Quantum theory of flicker noise: the $1/f$ law as a lower bound on the voltage power spectrum,” arXiv:2007.14481 (2020).
  * (14) K.A. Kazakov, “A quantum bound on the $1/f$ noise in semiconductors with a conical energy-momentum dispersion,” arXiv:2107.01652 (2021).
  * (15) J. Schwinger, J. Math. Phys. 2, 407 (1961).
  * (16) L. V. Keldysh, Zh. Eksp. Teor. Fiz. 47, 1515 (1964) [Sov. Phys. JETP 20, 1018 (1965)].
  * (17) A. H. Castro Neto, F. Guinea, N. M. R. Peres, K. S. Novoselov, and A. K. Geim, Rev. Mod. Phys. 81, 109 (2009).
  * (18) P. R. Wallace, Phys. Rev. 71, 622 (1947).
  * (19) G. W. Semenoff, Phys. Rev. Lett. 53, 2449 (1984).
  * (20) E. H. Hwang, S. Adam, and S. Das Sarma, Phys. Rev. Lett. 98, 186806 (2007).
  * (21) A. N. Pal, S. Ghatak, V. Kochat, E. Sneha, A. Sampathkumar, S. Raghavan, A. Ghosh, ACS Nano 5, 2075 (2011).
  * (22) Y.-M. Lin and P. Avouris, Nano Lett. 8, 2119 (2008).
  * (23) A. N. Pal and A. Ghosh, Phys. Rev. Lett. 102, 126805 (2009).
  * (24) I. Heller, S. Chatoor, J. M$\ddot{\rm a}$nnik, M. A. G. Zevenbergen, J. B. Oostinga, A. F. Morpurgo, C. Dekker, and S. G. Lemay, Nano Lett. 10, 1563 (2010).
  * (25) Y. Zhang, E. E. Mendez, and X. Du, ACS Nano 5, 8124 (2011).
  * (26) A. A. Kaverzin, A. S. Mayorov, A. Shytov, and D. W. Horsell, Phys. Rev. B85, 075435 (2012).


11institutetext: Kyushu University, Fukuoka, Japan 11email:
<EMAIL_ADDRESS>NTT Communication
Science Laboratories, NTT Corporation, Japan

# Shared Latent Space of Font Shapes and Impressions

Jihun Kang 11 Daichi Haraguchi 11 Akisato Kimura 22 Seiichi Uchida 11
0000-0001-8592-7566

###### Abstract

We have specific impressions from the style of a typeface (font), suggesting
that there are correlations between font shape and its impressions. Based on
this hypothesis, we realize a shared latent space where a font shape image and
its impression words are embedded in a cross-modal manner. This latent space
is useful to understand the style-impression correlation and generate font
images by specifying several impression words. Experimental results with a
large style-impression dataset prove that it is possible to accurately realize
the shared latent space, especially for shape-relevant impression words, and
then use the space to generate font images with various impressions.

###### Keywords:

Font shape Font impression Shared latent space.

##  1 Introduction

It is known that different font shapes give different impressions. A recent
paper [4] provides a large font dataset, where many people attach impression
words to individual fonts. Fig. 1 shows several examples of fonts and their
impressions. In the dataset, $1,824$ impression words are attached to the
$18,815$ fonts; therefore, about $10$ different impression words are attached
to each font on average.

This fact suggests that font is a multi-modal medium. A font is presented as
letter images, i.e., visible shapes; simultaneously, it gives various
impressions. If we can believe that the impression can be represented by a set
of words, like Fig. 1, the multi-modality of a specific font is defined as the
pairwise relationship between “a shape set” and “an impression word set.” For
example, the font 4-square has two modalities, the 52 shape images of ‘A’ to
‘z’ and a set of impression words $\\{$heavy, headline, display, $\ldots,\\}$.

Figure 1: Examples of font images with their impression words.
Figure 2: The goal of our research: constructing a latent space representing
semantics shared with font images and their impressions.

Our research aims to construct a latent space representing semantics shared
with the two modalities, i.e., font images and their impressions, to
understand the relationship between them. This latent space enables cross-
modal translations; we can describe a given font image with appropriate
impression words and search for or even generate font images for a given set
of impression words through the obtained latent space. Fig. 2 illustrates the
idea of the shared latent space. Let $\mathbf{X}_{i}$ denote the $i$-th font
(i.e., a set of images from ‘A’ to ‘Z’ of the $i$-th font) and assume a set of
$J_{i}$ impression words
$\mathbf{W}_{i}=\\{w_{i,1},\ldots,w_{i,j},\ldots,w_{i,J_{i}}\\}$ are attached
to the font $\mathbf{X}_{i}$. In the $d$-dimensional shared latent space, we
expect that $f(\mathbf{X}_{i})\sim g(\mathbf{W}_{i})$ for all $i$, where the
functions $f$ and $g$ give $d$-dimensional representations of $\mathbf{X}_{i}$
and $\mathbf{W}_{i}$, respectively. Therefore, the realization of the latent
space is the task of getting the representation functions $f$ and $g$ that
satisfy this proximity condition.

Figure 3: Overview of the proposed method to generate the latent space shared
by two totally different modalities, that is, font shape images and their
impression words.

This paper proposes a novel machine learning-based analysis framework, as
shown in Fig. 3, for constructing a shared latent space. We rely on a cross-
modal network structure, where two modalities $\mathbf{X}_{i}$ and
$\mathbf{W}_{i}$ are fed to two neural networks that realize $f$ and $g$,
respectively. Those two neural networks are co-trained to build a shared
latent space. In order to guarantee that the representations
$f(\mathbf{X}_{i})$ and $g(\mathbf{W}_{i})$ still keep the original
information carried by $\mathbf{X}_{i}$ and $\mathbf{W}_{i}$, our framework
employs an auto-encoder for each modality.

Figure 4: Word vector by aggregating the word vectors of multiple impression
words by word2vec and DeepSets.

(a)

(b)

Figure 5: Possible applications of the shared latent space. (a) Font image
retrieval. (b) Font image generation.

In the proposed framework, we employ DeepSets [26], which can deal with sets
as the input. The word set $\mathbf{W}_{i}$ will have an arbitrary number
$J_{i}$ of impression words and thus cannot deal with a standard data input
scheme. As shown in Fig. 4, DeepSets first converts $J_{i}$ elements
$w_{i,1},\ldots,w_{i,j},\ldots,w_{i,J_{i}}$ of $\mathbf{W}_{i}$ as $J_{i}$
feature vectors $e(w_{i,1}),\ldots,e(w_{i,j}),\ldots,e(w_{i,J_{i}})$
independently and then average them into a single $d$-dimensional vector
$h(\mathbf{W}_{i})=\sum_{j}e(w_{i,j})/J_{i}$. This simple procedure allows us
to deal with an arbitrary number of the set elements, while avoiding the
effect of the order of elements.

If the shared latent space is realized, there are several possible
applications. Fig. 5 shows two main applications; (a) font image retrieval and
(b) font image generation. In both applications, we can specify an arbitrary
number of impression words. Then, (a) existing font images and (b) new font
images are provided with the specified impressions, respectively.

We have to note that the proposed method based on DeepSets also allows us to
understand the shape-relevance of the individual words. A careful observation
of Fig. 1 implies that there are two types of impression words: shape-relevant
and shape-irrelevant. The former is the impression words that are
understandable from the font shape. For example, the impression words, such as
sans-serif and heavy, are shape-relevant. The latter, shape-irrelevant words,
is not understandable from the font shape. For example, soccer attached to the
font international in Fig. 1 is not understandable without knowing the font is
used for the uniform of a soccer team.

Based on the above discussions, we can summarize the main technical
contributions of this paper as follows:

  * •

The proposed method can construct latent space representing semantics shared
with font image and their impression words. By doing this, we can capture the
relationship between font shapes or styles and impression words through the
latent space.

  * •

The latest space derived by the proposed method is generic. It can be applied
to various downstream tasks, such as font image retrieval and generation with
impression words.

##  2 Related Work

###  2.1 Font style and impression

Fonts affect some impressions to readers. In the fields of psychology and
marketing, the relationship between fonts and the impressions has been
analyzed experimentally for long years [5, 7, 12, 17]. For instance, Doyle et
al. [7] have reported that fonts affected the perception of the brand name.

In recent years, the relationship between font styles and various targets
concerning impressions has been analyzed, and font recommendation systems
corresponding impressions of targets have been proposed. Shinahara et al. [18]
have analyzed the relationship between font styles and genre in book and
advertisement. Choi et al. [6] and Shirani et al. [19] develop font
recommendation systems; they select fonts that match the impressions of images
and texts, respectively. O’Donovan et al. [15] have published a dataset with
200 different fonts, each of which is annotated with the degrees of 37
attributes. Chen et al. [4] published a far larger collection of fonts tagged
with impression words and proposed a font retrieval method using word queries.

Font style classification and identification are other important tasks for
dealing with fonts. Font style classification, such as [27], is a task to
classify a font image to one of the styles, such as roman, slant, bold, serif,
sans-serif, etc. Font identification is a task to estimate the name of the
font family, such as Helvetica, from images. DeepFont has been proposed [25]
for this purpose. Haraguchi et al. [9] tackle the font identification task as
a matching problem.

As an extension of various GAN-based font image generation, such as [10, 13],
generation of font images with specific impressions has been recently examined
by Wang et al. [24]. Their Attribute2Font uses the dataset teodonovan to
generates fonts by specifying the degrees of 37 attributes, such as gentle and
fresh.

Those font recommendation, search, style classification, and generation
methods are developed in a more application-oriented manner and do not focus
on the essential correlation between font styles and the impressions;
therefore, they do not deal with the shared latent space. To the authors’ best
knowledge, this is the first attempt to prove that the letter images
$\mathbf{X}_{i}$ and the impression words $\mathbf{W}_{i}$ of the $i$th font
can be embedded into the same $d$-dimensional vector space to satisfy
$f(\mathbf{X}_{i})\sim g(\mathbf{W}_{i})$ as possible for all $i$.

###  2.2 Latent space embedding

In multimodal modeling of images and texts, many attempts have been made for
shared latent space embedding of the images and texts. Socher et al. [20] have
proposed a model that segments and annotates images by mapping images
associated with the words to a latent semantic space. The same group extended
this idea [21] by incorporating a neural network-based representation learning
scheme of the image modality. In this work, the text modality is encoded by a
hand-crafted feature, and then the image modality is mapped to the fixed text
modality. In the works focusing on neural language caption generation [8, 11,
23], images and texts are not embedded into the same latent space explicitly,
but image features by Convolutional Neural Networks (CNNs) are used as an
input of Recurrent Neural Networks (RNNs) that generate textual information.

In the document analysis research field, Almazán et al. [1] have published a
pioneering work that a word image and its textual information are embedded
into the same space for word spotting and recognition even in a zero-shot
manner. Such an embedding strategy is nowadays extended to deal with a tough
multi-modal task, called Text VQA [2]. Sumi et al. [22] realized a shared
latent space between online and offline handwriting sample pairs and proved
that a stroke order recovery is possible via the shared latent space.

###  2.3 Representation learning for a set

When each training sample comprises a different number of elements without any
specific order, some machine learning architecture that accepts a set as its
input sample is necessary. DeepSets [26] has been proposed a simple but
powerful framework to deal with sets as samples. As shown in Fig. 4, DeepSets
treats the elements in an input set independently during its feature
extraction layer and then combines the feature vectors into a single vector by
a simple operation, such as sum, average, and max. In recent years, Saito et
al. [16] have proposed the architecture to use sets by capturing the
properties from the basis of set matching mathematically and have tried novel
fashion item matching using sets.

In this paper, we treat the impression words $\mathbf{W}_{i}$ attached to the
$i$th font as a set. The number of the attached words is different among
fonts, as shown in Fig. 1. In addition, the words have no specific order. We,
therefore, use DeepSets to treat $\mathbf{W}_{i}$ as a set. Note that the
other modality $\mathbf{X}_{i}$ is also comprised of 26 images (from ‘A’ to
‘Z’) and possible to deal with a set; however, in our setup,
$\|\mathbf{X}_{i}\|$ is always 26 and the order of the 26 images is better to
be fixed in alphabetical order. Therefore, we represent $\mathbf{X}_{i}$ as a
stack of 26 images.

##  3 MyFonts dataset [4]

As the font dataset with impression words, we employ the dataset published by
Chen et al., [4]. This dataset, hereafter called the MyFonts dataset,
comprises 18,815 fonts collected at MyFonts.com. As shown in Fig. 1, each font
is tagged with $0\sim 184$ impression words attached by crowd-sourcing. This
means that the impression words have a large variability according to the
crowd-sourcing workers’ subjective bias. The vocabulary size of the impression
words is 1,824. As noted in Section 1, most impression words are shape-
relevant, but some are independent of shapes, such as soccer. (Recall that the
bottom example of Fig. 1 has this word.)

Since we need to train several networks as a function $g(\cdot)$ with
sufficient samples, we removed non-frequent impression words attached to less
than 100 fonts. Consequently, we used 451 impression words in our experiments.
In addition, we removed the dingbat (pictorial symbols) fonts and the circled
fonts from the MyFonts dataset by manual inspections by three persons. We also
removed fonts without any impression words (after the above non-frequent word
removal). Consequently, we used 9,980, 2,992, and 1,223 fonts for training,
validating, and testing, respectively. Hereafter $\mathbf{\Omega}_{\rm
train}$, $\mathbf{\Omega}_{\rm val}$ and $\mathbf{\Omega}_{\rm test}$ denote
the training, validation, and test font sets, respectively. Note that we used
capital letter images of ‘A’ to ‘Z’ in each font.

##  4 Shared Latent Space of Font Shape and Impression

This section provides the method to train our model of Fig. 3 to realize a
shared latent space. The training is organized in a two-step manner for faster
and more accurate convergence. We first perform two independent training
pipelines as an initialization of the cross-modal encoding scheme. More
specifically, we independently train two different autoencoders for font
shapes (the bottom pipeline of Fig. 3) and impression word (the upper
pipeline). The latent vectors of those autoencoders correspond to
$f(\mathbf{X}_{i})$ and $g(\mathbf{W}_{i})$, respectively. Second, the end-to-
end co-training will be performed to embed those latent vectors into the
shared space to satisfy the condition $f(\mathbf{X}_{i})\sim
g(\mathbf{W}_{i})$.

###  4.1 Font shape encoding by autoencoder and its initialization

An autoencoder is used for generating the latent vector $f(\mathbf{X}_{i})$ of
the image modality, i.e., font shapes. As shown in the bottom part of Fig. 3,
the autoencoder accept $\mathbf{X}_{i}$ as its input and generate
$\bar{\mathbf{X}}_{i}$ via an intermediate compressed representation
$f(\mathbf{X}_{i})$. Both of $\mathbf{X}_{i}$ and $\bar{\mathbf{X}}_{i}$ are
26 images (stacked as 26 channels) and expected to be similar with each other,
i.e., $\mathbf{X}_{i}\sim\bar{\mathbf{X}}_{i}$, in order to guarantee that
$f(\mathbf{X}_{i})$ carries the original shape information of $\mathbf{X}_{i}$
sufficiently. Note that $f(\mathbf{X}_{i})$ is emitted from the autoencoder as
a tensor of $7\times 7\times 128$, whereas it is flattened as a $d=7\times
7\times 128=6,272$-dimensional vector in the shared latent space. In the
following, these two representations are not distinguished unless otherwise
mentioned ($g(\mathbf{W}_{i})$ also).

The encoder ($\mathbf{X}_{i}\mapsto f(\mathbf{X}_{i})$) is based on ResNet18
(pre-trained with ImageNet) and the decoder
($f(\mathbf{X}_{i})\mapsto\bar{\mathbf{X}}_{i}$) is comprised of several
deconvolutional layers. (See Section 5.1 for the detail.) They are trained to
minimize the following construction loss function $L_{\rm shape}$:

$L_{\rm
shape}=\sum_{i=1}^{N}{\left\|\mathbf{X}_{i}-\bar{\mathbf{X}}_{i}\right\|},$
(1)

where $N$ is the number of fonts used for training.

It is possible to initialize the autoencoder for the image modality by using
font images $\mathbf{X}_{i}\in\mathbf{\Omega}_{\rm train}$, independently of
the text modality. Specifically, the encoder and the decoder are trained to
minimize $L_{\rm shape}$ for $\\{\mathbf{X}_{i}\\}$. Note that the autoencoder
of the text modality will not be pre-trained, as described later.

###  4.2 Impression word encoding by Word2vec and DeepSets and their
initialization

Like the image modality, an autoencoder is used for generating the impression
word vectors $g(\mathbf{W}_{i})$. However, the text modality requires extra
modules to accept an arbitrary number of impression words
$\mathbf{W}_{i}=\\{w_{i,1},\ldots,w_{i,j},\ldots,\allowbreak w_{i,J_{i}}\\}$
as its input. Moreover, each impression word $w_{i,j}$ should be converted to
a semantic vector, so that similar impression words give similar effects to
the system. Therefore, before the autoencoder the impression word $w_{i,j}$ is
converted to a semantic vector $l(w_{i,j})$ by Word2vec [14] and then in the
front-end of the autoencoder, all the semantic vectors are aggregated to a
single fixed-dimensional vector $h(\mathbf{W}_{i})$ by DeepSets [26]. We treat
$s(\mathbf{W}_{i})=\sum_{j}l(w_{i,j})/J_{i}$ as the original vector
representing the $J_{i}$ impression words and extract the latent vector
$g(\mathbf{W}_{i})$ from it by an encoder.

Word2vec is a well-known word embedding method111If necessary, it is possible
to use an arbitrary word embedding methods, such as fasttext [3] and other
pre-trained models. that generates a fixed-dimensional vector $l(w_{i,j})$ for
a word $w_{i,j}$. If a pair of words $w_{i,j}$ and $w_{i,j^{\prime}}$ have
similar meanings (like bold and thick), we expect $l(w_{i,j})\sim
l(w_{i,j^{\prime}})$. Although the one-hot representation might be a simpler
choice, we use the word embedding method since we can expect that the
semantically-similar impression words give a similar effect on
$g(\mathbf{W}_{i})$. We use word2vec pretrained by Google News dataset and do
not retrain it by our dataset.

The $J_{i}$ semantic vectors $\\{l(w_{i,j})|j=1,\ldots,J_{i}\\}$ are then
converted into a single vector $h(\mathbf{W}_{i})$ by DeepSets. As shown in
Fig. 4, DeepSets has two functions: a trainable encoding scheme, or
representation learning, and an aggregation scheme. The former is a deep
neural network and gives a new representation $e(w_{i,j})$ for the word2vec
vector $l(w_{i,j})$. The later is the simple averaging process
$h(\mathbf{W}_{i})=\sum_{j}e(w_{i,j})/J_{i}$, whereas DeepSets can employ
another pooling process, such as sum and max.

The encoder ($s(\mathbf{W}_{i})\mapsto g(\mathbf{W}_{i})$) is ResNet18 (with
random initialization) and the decoder
($g(\mathbf{W}_{i})\mapsto\bar{s}(\mathbf{W}_{i})$) is composed of several
fully-connected layers (see Section 5.1 for the detail). Specifically, the
encoder and the decoder are trained by minimizing the following loss function:

$L_{\rm
impression}=\sum_{i=1}^{N}\|s(\mathbf{W}_{i})-\bar{s}(\mathbf{W}_{i})\|.$ (2)

One might think that we can skip the encoder in DeepSets, that is, it might be
possible to use the averaged word2vec vectors $s(\mathbf{W}_{i})$ directly as
$h(\mathbf{W}_{i})$. However, the two-step encoding scheme via DeepSets is
necessary. The first encoder in DeepSets plays the role of suppressing the
effect of shape-irrelevant words by setting their vectors smaller. Then, the
second encoder will convert $h(\mathbf{W}_{i})$, which mainly carries the
information of shape-relevant impression words, into the latent vector
$g(\mathbf{W}_{i})$, which is expected to be similar to the font shape vector
$f(\mathbf{X}_{i})$.

Like the image modality, we can initialize this pipeline (i,e., the
autoencoder and DeepSets) independently by minimizing the loss function
$L_{\rm impression}$. Note that we cannot initialize the pipeline by using
$h(\mathbf{W}_{i})$ instead of $s(\mathbf{W}_{i})$. This is because it results
in the trivial solution that
$h(\mathbf{W}_{i})\equiv\bar{h}(\mathbf{W}_{i})\equiv 0$ by abusing the
encoder inside DeepSets.

###  4.3 Co-training for the shared latent space

After the pre-training of the autoencoder for both modality, all the modules
of both modalities are co-trained to realize the shared latent space. From its
purpose to have $f(\mathbf{X}_{i})\sim g(\mathbf{W}_{i})$, we have the
following loss function:

$L_{\rm share}=\sum_{i=1}^{N}\|f(\mathbf{X}_{i})-g(\mathbf{W}_{i})\|.$ (3)

Consequently, the overall loss function $L$ of the proposed method becomes:

$L=L_{\rm shape}+L_{\rm impression}+L_{\rm share}.$ (4)

In the process of minimizing the loss function $L$, the weights of all
autoencoders and DeepSets are trained simultaneously.

##  5 Experimental Results

###  5.1 Implementation details

For the image modality, the encoder ($\mathbf{X}_{i}\mapsto
f(\mathbf{X}_{i})$) is ResNet18 (pre-trained by ImageNet) that have additional
convolution layer whose kernel size is $1\times 1$ and number of channel is
128. The decoder ($f(\mathbf{X}_{i})\mapsto\bar{\mathbf{X}}_{i}$) is $\rm
D_{(512,1,0)}^{1\times 1}$–$\rm R$–$\rm D_{(256,2,1)}^{4\times 4}$–$\rm
R$–$\rm D_{(128,2,1)}^{4\times 4}$–$\rm R$–$\rm D_{(64,2,1)}^{4\times 4}$–$\rm
R$–$\rm D_{(32,2,1)}^{4\times 4}$–$\rm R$–$\rm D_{(26,2,1)}^{4\times 4}$ where
$\rm D$ and $\rm R$ show transpose convolution layer and Relu function
respectively. The parenthesized description shows (channels, stride, padding)
and the superscript shows the kernel size.

For the text (impression word) modality, the encoder
($h(\mathbf{W}_{i})\mapsto g(\mathbf{W}_{i})$) is $\rm F_{(1024)}$–R–$\rm
F_{(2048)}$–R–$\rm F_{(6272)}$, and the decoder
($g(\mathbf{W}_{i})\mapsto\bar{s}(\mathbf{W}_{i})$) is $\rm F_{(2048)}$–R–$\rm
F_{(1024)}$–$\rm R$–$\rm F_{(300)}$ where $\rm F$ shows fully-connected layer.
Note that the parenthesized description shows hidden units.

The entire network is trained by the training font set $\mathbf{\Omega}_{\rm
train}$ (9,980 fonts) and tested by the test set $\mathbf{\Omega}_{\rm test}$
(1,223 fonts). The hyper-parameters and the training epochs are optimized by
$\mathbf{\Omega}_{\rm val}$ (2,992 fonts).

serif, modern, sans-serif, text, magazine, elegant, book, slab-serif,
humanist, fashion, ultra-bold, editorial, contemporary, signage, bold, clean,
poster, condense

Figure 6: 18 words with P@10 $\geq$ 0.5.

serif

modern

sans-serif

text

magazine

Figure 7: tSNE visualization of the shared latent space for the words with
P@10$\geq 0.5$. Each dot corresponds to a font. A red dot corresponds to the
impression word $w$. A blue dot corresponds to a font $\mathbf{X}$ that the
impression word $w$ is attached as its ground-truth.

fun

retro

Figure 8: tSNE visualization of the shared latent space for the words with
<EMAIL_ADDRESS>
###  5.2 Qualitative evaluation 1: Observing the shared latent space via a
font image retrieval task

As shown in Fig. 5 (a), the proposed shared latent space can be used for font
image retrieval by the query of impression words. In this section, we conduct
the retrieval experiment by using the test dataset $\mathbf{\Omega}_{\rm
test}$, to understand the characteristics of the learned latent space.
Especially, we carefully observe the difference between shape-relevant
impression words and shape-irrelevant impression words through this
experiment.

Fig. 6 shows the impression words with higher P@K values. P@K (abbreviation of
“Precision at $K$”) of a word $w$ is a metric for the information retrieval
task. In our task, it evaluates how many the font shapes with the same word
$w$ in their ground-truth are retrieved among the $K$ nearest font shapes in
the shared latent space. Formally,

$\mathrm{P@K}=\frac{1}{K}\sum_{i\in\mathbf{\Omega}_{\rm
test}}\mathbf{1}(w\in\mathbf{W}_{i})\cap\mathbf{1}\left(\|f(\mathbf{X}_{i})-g(w)\|\leq\mathop{\rm
minK}_{i^{\prime}\in\mathbf{\Omega}_{\rm
test}}\|f(\mathbf{X}_{i^{\prime}})-g(w)\|\right),$ (5)

where $\mathbf{1}(\cdot)$ is the indicator function, $\cap$ is the and-
operation, and ${\rm minK}$ is the operator to give the $K$th minimum value.
Note that P@K is the evaluation for a single word query $w$. By the
flexibility of DeepSets, it is possible to have a latent vector $g(w)$ even
for a single word $w$, although our method is trained with a set of
impressions $\mathbf{W}_{i}$ for each font $i$.

If P@K is 1, all the $K$-neighboring font shapes in the shared latent space
have the impression word $w$ in their ground-truth, and therefore the
corresponding shape and word are embedded successfully while satisfying
$f(\mathbf{X})\sim g(w)$. If P@K is near or equal to zero, it means the
corresponding word and shape are distant in the latent space. This might
happen for the shape-irrelevant impression word.

Fig. 6 shows the lists of 18 words with higher P@10 222Although sans-serif and
sans might be the same, we use the original word tag in the MyFonts dataset..
The list shows the font shape-relevant words, such as serif, sans-serif, bold
and condense, as expected. This proves that our framework can learn the shared
latent space successfully. It is noteworthy that the more subjective
impression elegant has a high P@K value. Although the “elegantness” of a font
may vary among people, this result indicates there are common shape-relevant
characteristics about it.

Figure 9: The transition of P@K values along with $K$ for several words.

Figure 10: Generating images from single impression word.

Figure 11: Generating images from multiple impression words.

Fig. 7 visualizes the distribution of the test fonts in $\mathbf{\Omega}_{\rm
test}$ in the shared latent space by tSNE. We selected several impression
words from Fig. 6 and plotted the tSNE of each word. On each tSNE of the word
$w$, the latent vector $f(\mathbf{X})$ of a font shape $\mathbf{X}$ with the
word $w$ is depicted as a blue dot. The font shapes independent of $w$ are
depicted as gray dots. The red dot on tSNE corresponds to $g(w)$, that is, the
latent vector of the impression word $w$. Since it is a “shared” latent space,
we can plot the latent vectors of font shapes and impression words in the same
space.

Fig. 7 proves that the many appropriate font shapes (the blue dots) are
gathering around $g(w)$ (the red dot). This means that the corresponding
shapes and words are embedded closely to each other in the shared latent
space. At the same time, we can also observe that not all blue dots gather
around $g(w)$. This suggests the diversity of the impression; the same
impression word is attached to fonts with very different shapes. In other
words, even for shape-relevant impression words, the font shape is not very
similar, as shown in the font shape examples of bold.

Fig. 8 shows the tSNE visualization of the the latent vectors $f(\mathbf{X})$
for more shape-irrelevant words, fun and retro. As expected from the above
discussion on the shape-relevant words, these shape-irrelevant words show far
more scattered distributions; there is no condensation around $g(w)$ even
after co-training the shared latent space. In fact, P@10 of the two words is
0.1 (i.e., just one correct $\mathbf{X}$ in the ten nearest neighbors). As
shown in the font examples of fun and retro, the shape-irrelevant words do not
correspond to a fixed font shape style, and therefore they do not show the
condensation. Consequently, it is difficult to satisfy the condition
$f(\mathbf{X})\sim g(w)$ for the shape-irrelevant word $w$.

Fig. 9 shows the transition of P@K values of several words along with $K$. The
words are selected from the 18 words with higher P@10 (i.e., shape-relevant
words) and two extra words, retro and fun. From this plot, it is possible to
understand how the font shapes $f(\mathbf{W})$ for each word spread in the
latent space. Although less-frequent words have a handicap in P@K evaluation,
some words, such as sans-serif and fun, keep the same P@K values between
$K=10$ and $100$ – this suggests they spread over from $g(w)$ with almost the
constant density.

###  5.3 Qualitative evaluation 2: Font image generation task

As shown in Fig. 5(b), we can use the shared latent space for generating font
images with specific impressions. Using $g(\mathbf{W}_{i})$ as the input for
the image decoder, we can get the alphabet images from ‘A’ to ‘Z’ at once.
Theoretically, we can specify any impression words (even unlearned words).
However, as we will see in the following experiment, the quality of the
generated font images depends on the “type” of the impression words: shape-
relevance and shape-irrelevance.

Fig. 11 shows the font generation results when a single impression word is
given. For the shape-relevant words with higher P@10, i.e., serif to condense,
legible font images with the specified impression are generated successfully,
thanks to the property $f(\mathbf{X})\sim g(w)$. However, for shape-irrelevant
words, fun and retro, font images with just a neutral style were generated,
due to their largely variable shapes, as shown in Fig. 8.

Fig. 11 shows the generation results when we specify multiple impression
words. The result images generated by specifying two shape-relevant words
become a mixed style successfully. The last example shows the case of the same
words are specified two times; according to the property of DeepSets, we can
strengthen an impression by this strategy.

###  5.4 Quantitative evaluation of the learned latent space

A quantitative evaluation is conducted by using the embedding results of all
font data in $\mathbf{\Omega}_{\rm test}$. Specifically, we use an image
retrieval scenario: Assume that we will give $\mathbf{W}_{i}$ of the $i$th
test font as a query. Then, all the images $\mathbf{X}\in\mathbf{\Omega}_{\rm
test}$ are then ranked by the distance $\|f(\mathbf{X})-g(\mathbf{W}_{i})\|$.
Finally, the rank $r_{i}$ of the correct image $\mathbf{X}_{i}$ among
$|\mathbf{\Omega}_{\rm test}|$ images is determined. The average retrieval
rank $R_{\rm{t}\to\rm{i}}=\sum{r_{i}}/|\mathbf{\Omega}_{\rm test}|$ is the
evaluation metric. Similarly, we have $R_{\rm{i}\to\rm{t}}$ by exchanging
$\mathbf{W}_{i}$ and $\mathbf{X}\in\mathbf{\Omega}_{\rm test}$ to
$\mathbf{X}_{i}$ and $\mathbf{W}\in\mathbf{\Omega}_{\rm test}$, respectively.

To our best knowledge, this is the first attempt at the cross-modal embedding
of font impression and font shape into the shared latent space; therefore,
there is no appropriate comparative baseline for this study. We, therefore,
consider the following ablation cases to confirm the advantage of the proposed
method.

  * •

Proposed: $f(\mathbf{X}_{i})$ and $g(\mathbf{W}_{i})$ are embedded by the co-
trained encoder (Section 4.3).

  * •

Independent: $f(\mathbf{X}_{i})$ and $g(\mathbf{W}_{i})$ are embedded by the
encoders by the initial training steps (Sections 4.1 and 4.2). No co-training
has been made.

  * •

Image $\mapsto$ Text: After initializing the encoders, the encoder for the
text modality is fixed. The encoder and decoder of the image modality are then
trained so that $f(\mathbf{X}_{i})\sim g(\mathbf{W}_{i})$. This setup is very
similar to a well-known previous research named cross-modal transfer [21].

  * •

Text $\mapsto$ Image: The opposite of the Image $\mapsto$ Text setting. After
initializing the encoder for the image modality, the encoder for the image
modality is fixed. The encoder and decoder of the text modality are then
trained so that $f(\mathbf{X}_{i})\sim g(\mathbf{W}_{i})$.

Table 1 shows the evaluation results. This indicates that the proposed method
greatly outperformed the others. It also indicates that both Image $\mapsto$
Text and Text $\mapsto$ Image settings did not work well, which implies that
joint optimization of cross-modal autoencoders was a key for obtaining
meaningful shared representations.

Table 1: Average retrieval rank. (*: A similar setup to [21].) Method | $R_{\rm{i}\to\rm{t}}$ | $R_{\rm{t}\to\rm{i}}$
---|---|---
Independent | 608.7 | 612.5
Image $\mapsto$ Text(*) | 608.1 | 612.2
Text $\mapsto$ Image(*) | 516.9 | 553.0
Proposed | 172.6 | 356.6

##  6 Conclusion

This paper showed that it is possible to realize a shared latent space where a
font shape image and its multiple impression words are embedded as similar
vectors. Through the shared latent space, we can handle font shapes and their
impressions in a unified manner, which can lead us to generate and retrieve
font images with specific fonts and retrieve font images. Our various
experimental results revealed the usefulness of the latent space. It is also
clarified that the embedding quality depends on the shape-relevance of the
impression words — it is rather reasonable because the latent space represents
font shapes, as well as impression words.

## References

  * [1] Almazán, J., Gordo, A., Fornés, A., Valveny, E.: Word spotting and recognition with embedded attributes. IEEE Trans. Patt. Anal. Mach. Intell. 36(12), 2552–2566 (2014)
  * [2] Biten, A.F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C., Karatzas, D.: Scene text visual question answering. In: ICCV (2019)
  * [3] Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguistics 5, 135–146 (2017)
  * [4] Chen, T., Wang, Z., Xu, N., Jin, H., Luo, J.: Large-scale tag-based font retrieval with generative feature learning. In: ICCV (2019)
  * [5] Childers, T.L., Jass, J.: All dressed up with something to say: Effects of typeface semantic associations on brand perceptions and consumer memory. J. Consumer Psychology 12(2), 93–106 (2002)
  * [6] Choi, S., Aizawa, K., Sebe, N.: Fontmatcher: Font image paring for harmonious digital graphic design. In: IUI (2018)
  * [7] Doyle, J.R., Bottomley, P.A.: Mixed messages in brand names: Separating the impacts of letter shape from sound symbolism. Psychology & Marketing 28(7), 749–762 (2011)
  * [8] Fang, H., Gupta, S., Iandola, F., Srivastava, R.K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J.C., et al.: From captions to visual concepts and back. In: CVPR (2015)
  * [9] Haraguchi, D., Harada, S., Iwana Brian, K., Shinahara, Y., Uchida, S.: Character-independent font identification. In: DAS (2020)
  * [10] Hayashi, H., Abe, K., Uchida, S.: GlyphGAN: Style-consistent font generation based on generative adversarial networks. Knowledge-Based Syst. 186 (2019)
  * [11] Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: CVPR (2015)
  * [12] Lewis, C., Walker, P.: Typographic influences on reading. British J. Psychology 80(2), 241–257 (1989)
  * [13] Lyu, P., Bai, X., Yao, C., Zhu, Z., Huang, T., Liu, W.: Auto-encoder guided gan for chinese calligraphy synthesis. In: ICDAR (2017)
  * [14] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: NIPS (2013)
  * [15] O’Donovan, P., Lībeks, J., Agarwala, A., Hertzmann, A.: Exploratory font selection using crowdsourced attributes. ACM Trans. Graphics 33(4), 92 (2014)
  * [16] Saito, Y., Nakamura, T., Hachiya, H.: Exchangeable deep neural networks for set-to-set matching and learning. In: ECCV (2020)
  * [17] Shaikh, D., Chaparro, B.: Perception of fonts: Perceived personality traits and appropriate uses. In: Digital Fonts and Reading, chap. 13. World Scientific (2016)
  * [18] Shinahara, Y., Karamatsu, T., Harada, D., Yamaguchi, K., Uchida, S.: Serif or sans: Visual font analytics on book covers and online advertisements. In: ICDAR (2019)
  * [19] Shirani, A., Dernoncourt, F., Echevarria, J., Asente, P., Lipka, N., Solorio, T.: Let Me Choose: From verbal context to font selection. In: ACL (2020)
  * [20] Socher, R., Fei-Fei, L.: Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora. In: CVPR (2010)
  * [21] Socher, R., Ganjoo, M., Sridhar, H., Bastani, O., Manning, C.D., Ng, A.Y.: Zero-shot learning through cross-modal transfer. In: ICLR (2013)
  * [22] Sumi, T., Iwana, B.K., Hayashi, H., Uchida, S.: Modality conversion of handwritten patterns by cross variational autoencoders. In: ICDAR (2019)
  * [23] Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: CVPR (2015)
  * [24] Wang, Y., Gao, Y., Lian, Z.: Attribute2Font. ACM Trans. Graphics 39(4) (2020)
  * [25] Wang, Z., Yang, J., Jin, H., Shechtman, E., Agarwala, A., Brandt, J., Huang, T.S.: DeepFont: Identify your font from an image. In: ACM Multimedia (2015)
  * [26] Zaheer, M., Kottur, S., Ravanbhakhsh, S., Póczos, B., Salakhutdinov, R., Smola, A.J.: Deep sets. In: NIPS (2017)
  * [27] Zramdini, A., Ingold, R.: Optical font recognition using typographical features. IEEE Trans. Patt. Anal. Mach. Intell. 20(8), 877–882 (1998)