# Font Completion and Manipulation by
Cycling Between Multi-Modality Representations

Ye Yuan, Wuyang Chen, Zhaowen Wang, Matthew Fisher, Zhifei Zhang, Zhangyang
Wang, and Hailin Jin Ye Yuan is with the Texas A&M University. E-mail:
<EMAIL_ADDRESS>Chen and Zhangyang Wang are with the Department of
Electrical and Computer Engineering, The University of Texas at Austin, TX,
78712. E-mail<EMAIL_ADDRESS>Wang, Matthew Fisher,
Zhifei Zhang, and Hailin Jin are with the Adobe Research, San Jose, CA, 95100.
E-mail<EMAIL_ADDRESS>
###### Abstract

Generating font glyphs of consistent style from one or a few reference glyphs,
i.e., font completion, is an important task in topographical design. As the
problem is more well-defined than general image style transfer tasks, thus it
has received interest from both vision and machine learning communities.
Existing approaches address this problem as a direct image-to-image
translation task. In this work, we innovate to explore the generation of font
glyphs as 2D graphic objects with the graph as an intermediate representation,
so that more intrinsic graphic properties of font styles can be captured.
Specifically, we formulate a cross-modality cycled image-to-image model
structure with a graph constructor between an image encoder and an image
renderer. The novel graph constructor maps a glyph’s latent code to its graph
representation that matches expert knowledge, which is trained to help the
translation task. Our model generates improved results than both image-to-
image baseline and previous state-of-the-art methods for glyph completion.
Furthermore, the graph representation output by our model also provides an
intuitive interface for users to do local editing and manipulation. Our
proposed cross-modality cycled representation learning has the potential to be
applied to other domains with prior knowledge from different data modalities.
Our code is available at https://github.com/VITA-Group/Font_Completion_Graph.

###### Index Terms:

Font Completion, Image Synthesis, Graph

Figure 1: Unlike style transfer that learns a mapping from a single style to
another, font completion learns to transfer arbitrary input style and generate
new contents for all the glyph set. Our proposed image-to-graph-to-image
method (Section IV) learns a graph representation (Section III-B) for font
glyphs, which facilitates font completion, interpolation and manipulation
tasks.

##  I Introduction

The design of different fonts plays an essential role in multimedia, which can
not only convey various moods but also preserve the intellectual property
value of designers. A font is a set of glyphs sharing the same style. However,
to design a font, it is hard to keep the style of all glyphs consistent,
especially for languages of significant morphological variations. Therefore,
it is extremely time and resource-consuming for a multimedia designer to
manually create new fonts for a set of glyphs.

Due to the success of visual style transfer works [1, 2, 3], more recently the
research community shifted attention to address the above problem via the font
completion task. Text glyphs, as a common category of visual signals, have a
clear decoupled definition of style (i.e., font) and content (i.e., glyph).
Font completion targets on transferring unseen styles to all glyphs in a font,
by using only one or a few sample characters as input. This is different from
traditional image style transfer [4] which only creates a one-to-one mapping
between two domains (Figure 1). This task has important applications in
typography and design, since it can significantly help speed up the process of
designing or modifying fonts.

Generating glyphs with the same font style could be seen as an image-to-mage
translation problem. Image-to-image translation [5] is arguably the most
successful framework for image synthesis tasks including photo stylization.
Existing image-to-image translation tasks are commonly formulated as
extracting a style feature from a reference image, and combining the style
feature with the content feature of a target image. Combining the power of
deep convolutional neural network (CNN) and generative adversarial network
(GAN), image-to-image often produces high-quality stylization results.
Unpaired image-to-image [6, 7] further generalizes the framework by relaxing
the requirement on paired supervision. Application to font glyphs [8] shows
promising results in transferring the font style in terms of both shape and
texture. However, there are still limitations in the image-to-image framework
for font applications. It is observed in our experiments that the supervised
image-to-image suffers from poor style/content separation capability. More
importantly, the generated results can only capture the global shape transform
but not the high-resolution local details, which is particularly vital to the
font style. The generated images often have unsmooth or blurry boundaries.

One key insight from this paper is to advocate that font images form a very
restricted and well-defined sub-family of general images. In particular, font
glyphs are vector objects defined by parametric 2D outlines (e.g. TrueType
[9]) or strokes (e.g. Metafont [10]). These native font formats provide more
compact and intrinsic representations from geometric parameter manifolds than
pixel representations in the image space. That could be treated as a strong
domain-specific prior that lays the groundwork for the generation of finer
details specific to font images, yet was largely ignored by prior work. To the
best of our knowledge, the joint modeling of parametric and perceptual data
modalities is still an underexplored area in the research community. In this
paper, we aim to investigate how to utilize such multi-modality representation
to facilitate the font completion task by generating glyphs with both faithful
global structure (glyph information) and fine-grained local details (style
information).

To this end, we propose a graph-based glyph representation as an intermediate
stage to improve conventional image-based glyph style transfer. As shown in
Figure 2, the graph has a hierarchical structure, consisting of primary nodes
segmenting glyph strokes, secondary nodes within each stroke, and edges
connecting the entire glyph contour. The primary nodes control the global
glyph structure and the secondary nodes (dense sample points) describe the
local region details. Each node has both coordinates and tangents information
so that the original curve segments can be fully captured. Our graph can
express any connection between strokes. It is a scalable representation
similar to Scalable Vector Graphics (SVG) format, and also a computational
representation easy to integrate with deep learning models. Furthermore, the
graph representation generated by our model also provides an intuitive
intermediate interface for designers to do local editing and manipulation on
existing font glyphs.

Figure 2: We propose to leverage the graph structure of font glyphs as
intermediate representation to address the font completion problem. After data
preparation from SVG (Section V-A), our framework (Section IV) can cycle
between different glyph modalities and thus enable high-quality font
completion, manipulation and interpolation. The proposed graph representation,
serving as a transition state between other modalities including point set and
rasterized image, can capture both global content and local style features. It
is also resolution-free and allows easy manipulation by designers. The graph
has a hierarchical structure consisting of strokes (colored curves) and point
nodes (dots).

Specifically, we leverage four representations of the font glyphs in our
framework: SVG curves, rasterized images, point set (coordinates and
tangents), and graph (with each point in point set as a node and the
connections along contour as edges). We adopt a cross-modality auto-encoder
framework to learn font style features and transitions between different
representations. For font completion (Figure 3), given a reference source
glyph, we first extract the font style vector with an image encoder. Then a
graph constructor for the target glyph synthesizes the stylized target glyph
in its graph representation, which is further converted to the pixel output by
a graph transformer-based image renderer. On the Google Fonts dataset [11],
our method attains noticeable improvement over baseline and previous method in
both quantitative and qualitative evaluation.

To summarize, our contributions include:

  1. 1.

We design a graph representation for font glyphs to better capture style and
glyph information for font completion problem (Section III-B). The graph
representation is scalable and serves as a pivot to train conversion to other
formats (rasterized images, point set, etc.).

  2. 2.

We design a cross-modality auto-encoder framework towards high-quality font
completion (Section IV).

  3. 3.

We collected and processed a font dataset that includes all four modalities
for the first time (Section V-A).

  4. 4.

With the superior properties of our graph representation, we further enable
diverse font tasks, including font interpolation (V-C3) and local manipulation
(V-C2).

##  II Related Work

###  II-A Font Completion and Stylization

Several methods have been proposed to complete a typeface from a small number
of example glyphs. Initial work here focused on analyzing geometric features
and learning a non-linear mapping to interpolate and extrapolate existing
fonts [12, 13]. Early work using deep networks used a VAE encoder to learn a
font embedding that could be decoded into each glyph [14]. More recent deep
learning methods are mostly based on image-to-image translation adversarial
networks [15]. [16] proposed to transfer Latin font styles between rasterized
images by using a glyph completion GAN followed by a stylization network. This
has been generalized to Chinese typefaces also with a GAN-based architecture
[17]. A shape-matching GAN has been developed to help control the stylization
of the generated glyphs [18]. To support the automatic generation of target
typographic font with a small subset of target glyph images, [19] extracted
the natural strokes from glyph images, and constructs glyphs by deploying the
appropriate strokes onto skeletons.

TCN [8] utilized all the existing fonts to train a model that can extract
content features and style features from a rasterized image, and then replace
the style feature to reconstruct images of new glyphs of the same style.
StarFont [20] focused on Chinese font completion and treats each glyph as a
class and abandons reconstruction loss to reduce the total number of example
glyphs needed.

###  II-B Vector Font Learning

A small number of font learning methods operated in the vector domain, such as
SVG-VAE [21], a generative model that takes a rasterized image of icons as
input, and outputs the corresponding icons in scalable vector graphics (SVGs)
format. A method for finding correspondences between glyphs has been developed
that leverages an unsupervised Bezier representation [22]. We build on these
approaches to demonstrate the effectiveness of a model able to jointly operate
in both the raster and vector domains.

###  II-C Graph Representation Learning

In computer vision and graphics, graph-structured representations have
attained widespread use to efficiently represent compositional scenes in
large-scale images. Scene graphs have been used for image retrieval [23] and
generation [24], where nodes are objects and edges give relationships between
objects. Structured representations of visual scenes can also explicitly
encode certain types of properties, such as attributes, object co-occurrence,
or spatial relationships between pairs of objects [25]. 3D shape segmentation
is addressed in [26] by focusing on learning graph convolutions and using a
graph spectral embedding via eigenvector decomposition. In the context of
spectral clustering, [27] learned an auto-encoder that maps the rows of a
graph Laplacian matrix onto the corresponding spectral embedding. We build
upon these approaches and leverage the graph structures of font glyphs for
both cross-modal style translation and to represent font style and content.

##  III Problem Definition and Motivation

###  III-A Font Completion

Font completion is to transfer the font style of one or several glyphs to the
others in a glyph set (see top of Figure 1). For the ease of discussion,
assume we are given a single input glyph image $\bm{x}_{s,c^{\mathrm{in}}}$ of
a specific font style $s$ and glyph content $c^{\mathrm{in}}$, and the font
completion task requires completing the remaining glyphs in the glyph set
$X_{s}=\\{\bm{x}_{s,1},\bm{x}_{s,2},\cdots,\bm{x}_{s,C}\\}$, where $C$ is the
number of predefined glyphs (e.g., $C{=}26$ for Roman capital letters). Taking
$\bm{x}_{s,c^{\mathrm{in}}}$ as input, our model generates the remaining
glyphs $\\{\hat{\bm{x}}_{s,c}|c\neq c^{\mathrm{in}}\\}$ sharing the same font
style $s$ as $\bm{x}_{s,c^{\mathrm{in}}}$.

The objective of our learning process is to obtain a model $f$ parameterized
by $\theta$ that minimizes the difference between $\hat{\bm{x}}_{s,c}$ and
$\bm{x}_{s,c}$ ($c\in[1,C]\setminus\\{c^{\mathrm{in}}\\}$). The overall
formula of this problem is

$\theta^{*}=\operatorname*{argmin}_{\theta}\sum_{s=1}^{S}\sum_{c^{\mathrm{in}}=1}^{C}\sum_{c\neq
c^{\mathrm{in}}}d(\bm{x}_{s,c},f(\bm{x}_{s,c^{\mathrm{in}}},c;\theta)),$ (1)

where $d$ is the distance measure between two images, $s\in[1,S]$ and
$c\in[1,C]$ are the style and content index of a glyph $\bm{x}_{s,c}$,
respectively, and $S$ is the total number of fonts in training data. We mainly
use PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural SIMilarity) between
predicted and ground truth images as evaluation metrics. With the above
optimal $\theta^{*}$, we can obtain
$\hat{\bm{x}}_{s,c}=f(\bm{x}_{s,c^{\mathrm{in}}},c;\theta^{*})$ as the
prediction of $\bm{x}_{s,c}$.

Generating font glyph images with the same style could also be seen as an
image-to-image translation problem, where the style feature from a reference
image is extracted and combined with the content features of target images.
Existing single image-to-image translation methods [1, 4, 2, 3] only learn a
single modality translation. However, as will be shown in Figure 5, the visual
quality of direct image-to-image translation is low when only a single input
glyph is given. This motivates us to look for another font modality with which
the intrinsic structure of font glyph can be better captured.

###  III-B Why Graph Font Representation is Necessary?

In font completion, the input image $\bm{x}_{s,c^{\mathrm{in}}}$ and target
image $\bm{x}_{s,c}$ share the same font style but different glyph contents.
We can infer that model $f$ must learn meaningful style features of input
glyphs and content features of target glyphs. Different from the image-to-
image translation task, we have multiple modalities to choose from to
represent the target glyph, which would be finally converted to glyph image
$\hat{\bm{x}}_{s,c}$. However, it is not immediately clear which font modality
is the most suitable one for modeling. To facilitate the learning process of
Eq. 1, we therefore make the following key observations for the font
completion task.

Disentanglement and characterization of style and content features. The model
$f$ will need to extract style feature from an input glyph
$\bm{x}_{s,c^{\mathrm{in}}}$, and combine the style feature with the content
features of the target glyph $\bm{x}_{s,c}$. Thus, the font completion task
will benefit from learning decoupled global content and local style features
of glyphs. Another key property of font ignored by most previous work in font
generation is that font glyphs are initially designed as vector objects
defined by 2D curves and shapes, which means they have more intrinsic
representation in geometric parameter space than the perceptual representation
in the image space. By design, the geometric parameter space can better
capture local patterns of glyph strokes, such as serif, terminal, bowl, and
ear, which are the key to distinguish different font styles [28]. Therefore,
we expect a good font representation to disentangle content and style
information, and also faithfully capture local style details.

Scalability and manipulability of learned font representation. The learned
representation will be decoded or rendered into a rasterized font glyph image.
Thus, for practical usage, we also need $f$ to learn a representation of good
scalability, i.e., can be easily rendered to glyph images of various
resolutions. Moreover, since human designers usually create new style designs
by referring to existing ones, it is also beneficial to have good
manipulability for the learned representation, i.e., can be easily modified by
designers with interpretable operations.

Based on the above requirements for font completion, we compare different font
representations available to us as summarized in Table I. Specifically, we
have three existing representations of font glyphs: SVG curves, rasterized
images, and point set sampled along the contour (with coordinates and
tangents). Each of these representation has its pros and cons: SVG vector
fonts are scalable and resolution-free, but are not good candidates for
representation learning as they cannot be uniquely expressed with Bezier
curves; rasterized fonts are more perceptual and good for visual learning, but
are limited by fixed resolution and therefore may not capture enough style
details; sampled point set can be easily manipulated and converted from other
formats, but lacks global structure information for glyph content.

TABLE I: Comparison of different font modalities for the needs of font completion task. Modality |  | Content
---
(global)
| Style
---
(local)
Scalable | Manipulable
SVG curves |  |  | ✓ | ✓
Raster image | ✓ |  |  |
Point set |  | ✓ | ✓ | ✓
Graph (proposed) | ✓ | ✓ | ✓ | ✓

###  III-C Font in Graph: Decoupled Style/Content with Scalability &
Manipulation

Motivated by the limitations of existing representations, we want to design a
new font representation that can capture both local and global glyph
appearance, scale to any resolution, and deform easily with the designer’s
manipulation. To this end, we propose a graph-based font glyph representation
which serves as an intermediate stage to facilitate the font completion task.

In our graph formulation, we have a set of nodes corresponding to a sampled
point set whose attributes capture the local font styles, and the edge
connections between nodes define the global glyph structure. As shown in
Figure 2, the graph has a hierarchical structure, consisting of primary nodes
segmenting glyph strokes, secondary nodes describing local details within each
stroke, and edges connecting the entire glyph contour. Each node has
attributes containing both coordinates and tangents information so that the
original vector curve segments can be fully reconstructed. These attributes
also expose a user-friendly interface for manual editing. The graph can
express any connection between strokes. When the graph is partially connected,
it can also represent glyphs with disjoint strokes, e.g., the letter “i”.

As summarized in the last row of Table I, our graph representation satisfies
all the desired properties for font completion. It can be regarded as an
intermediate state between vectorized SVG curves and a rasterized image.
Moreover, with the recent advance of graph deep learning models such as Graph
Transformer Networks [29] and Graph Convolutional Networks [30], the graph
representation is also computationally tractable to integrate with other deep
learning modules. With more intrinsic font glyph information represented in
our graph, we will show how it can help regularize the conventional image
style transfer problem with domain knowledge from a complementary modality,
and facilitate tasks including font completion, interpolation, and
manipulation (see Figure 1).

##  IV Font Completion: Cycling between Multi-modality Representations

Figure 3: Our image-to-graph-to-image method for font completion. Given a
reference source glyph, we will generate all unseen glyphs of the same font
style. To leverage multi-modal font representations, we design a cross-
modality encoder-decoder to transit between different representations. The
image encoder first extracts the font style vector from the source glyph. Then
a point-set decoder together with a graph constructor synthesizes the stylized
target glyph in its graph representation, which is further converted to pixel
output by a graph transformer-based image renderer.

Framework Design Philosophy Given an input image $\bm{x}_{s,c^{\mathrm{in}}}$
of glyph $c^{\mathrm{in}}$ with style label $s$, the goal of font completion
is to generate glyph images $\bm{x}_{s,c}$ for any other glyph $c$ of the same
font style $s$. To leverage the graph-based intermediate representation
(Section III-B), we decompose the overall framework into two stages: 1) the
input glyph image is transformed to glyph graph representations, and 2) the
graph feature is rendered as completed glyph images with the transferred font
style. In short, the intermediate graph representation connects the input and
the glyphs to be completed, and regulates the transferring process.

As shown in Figure 3, our deep network model $f$ consists of four integral
components: an image encoder $f_{e}$, a point set decoder $f_{d}$, a graph
constructor $\mathcal{G}$, and a neural renderer $\mathcal{R}$, which gives

$\hat{\bm{x}}_{s,c}=f(\bm{x}_{s,c^{\mathrm{in}}},c;\theta)=\mathcal{R}\circ\mathcal{G}\circ
f_{d}\circ f_{e}(\bm{x}_{s,c^{\mathrm{in}}},c;\theta).$ (2)

We first extract font style feature with image encoder $f_{e}$, based on which
a number of sampled points of target glyph are generated with point set
decoder $f_{d}$. The discrete point set is further mapped to graph
representation with graph constructor $\mathcal{G}$, which is finally
converted to a rasterized image with neural renderer $\mathcal{R}$. In this
way, we establish the two-way connection between the image modality and graph
modality of font glyphs. The graph representation serves as an intermediate
representation to efficiently capture the structure of glyph and the style of
font.

Image encoder $f_{e}$. The image encoder $f_{e}$ extracts a latent vector
$\bm{z}_{s}$ from a source glyph image $\bm{x}_{s,c^{\mathrm{in}}}$ that
represents the common style shared all the glyphs of style $s$:

$\bm{z}_{s}=f_{e}(\bm{x}_{s,c^{\mathrm{in}}};\theta_{e}),$ (3)

where $f_{e}$ is designed as a convolutional network with parameter
$\theta_{e}$. The objective of learning this image encoder is to disentangle
font style from glyph content, and faithfully capture local style details. To
make the learned feature distinctive of font style, we apply a linear
classifier $h$ on $\bm{z}_{s}$, which guides feature learning with a font
classification loss:

$\displaystyle\mathcal{L}_{\mathrm{cls}}$
$\displaystyle=-\mathop{\mathbb{E}}_{s,c}\left[\mathrm{log}p(s|\bm{x}_{s,c})\right]$
(4) $\displaystyle=-\mathop{\mathbb{E}}_{s,c}\left[\mathrm{log}h_{s}\circ
f_{e}(\bm{x}_{s,c})\right],$ (5)

where $h_{s}$ is the normalized classification scoring function for style
class $s$.

Point set decoder $f_{d}$. Instead of directly generating image or graph of
target glyph $c$ from the style feature $\bm{z}_{s}$, we first employ a point
set decoder $f_{d}$ to output a fixed number of $m$ sample points on the
contour of target glyph $c$:

$\mathbf{P}_{s,c}=f_{d}(\bm{z}_{s},c;\theta_{d}),$ (6)

where $\mathbf{P}_{s,c}{=}[\bm{p}_{1},\cdots,\bm{p}_{m}]$, and a point feature
$\bm{p}_{i}$ of dimension $p$ contains attribute values such as coordinates
and tangents. $f_{d}$ is designed as a multi-layer 1D convolutional network
with parameter $\theta_{d}$. The generated points are densely distributed over
the entire glyph contour, and therefore can serve as candidate node locations
for graph construction in the next step. Such a design reduces the complexity
of graph constructor which otherwise has to build the entire graph from raw
style feature.

Graph constructor $\mathcal{G}$. The goal of a graph constructor $\mathcal{G}$
is to select graph nodes from the point set $\mathbf{P}_{s,c}$ and predict
their connectivity. Suppose there are $n_{1}$ primary nodes each of which is
associated with $n_{2}$ secondary nodes, we then have $n{=}n_{1}{\times}n_{2}$
actual points to be mapped from the $m$ points in $\mathbf{P}_{s,c}$, assuming
$m{\geq}n$. The mapping is represented by a $m{\times}n$ binary matrix
$\mathbf{M}$. As the $n_{2}$ secondary nodes associated with each primary node
are defined to be sequentially connected, the order of the secondary nodes
within each stroke is implied by the mapping matrix $\mathbf{M}$. The
remaining topology variation from the primary nodes can be modeled by a
$n_{1}{\times}n_{1}$ adjacency matrix $\mathbf{A}$, with $A(i,j){=}1$
indicating a directed edge is presented from primary node $i$ to $j$ (i.e.
their corresponding curve segments are connected on the contour), and
$A(i,j){=}0$ otherwise. Note directed graph is necessary to model the closed
loop of the inner and outer outline of a stroke. When there are two primary
nodes on an outline loop, they are connected with a bi-directional edge
$A(i,j){=}A(j,i){=}1$. When there is only one primary node on an outline loop,
a self-linkage $A(i,i){=}1$ is added.

Now we define a directed graph $\bm{g}_{s,c}$ as the graph representation of
glyph $\bm{x}_{s,c}$, whose structure is predicted with graph constructor
$\mathcal{G}$:

$\bm{g}_{s,c}=(\mathbf{M}_{s,c},\mathbf{A}_{s,c})\approx\mathcal{G}(\mathbf{P}_{s,c};\theta_{g},\phi_{c}),$
(7)

where $\mathbf{M}_{s,c}$ and $\mathbf{A}_{s,c}$ are the mapping and adjacency
matrices for glyph $\bm{x}_{s,c}$. The model $\mathcal{G}$ has a backbone
network with parameter $\theta_{g}$ shared by all the glyphs, and two head
networks for each glyph predicting $\mathbf{M}_{s,c}$ and $\mathbf{A}_{s,c}$
with parameter $\phi_{c}$. The shared backbone network is a single layer
Transformer [31] considering the non-local relationship among points
$\mathbf{P}_{s,c}$. The Transformer maps $\mathbf{P}_{s,c}$ of size
$m{\times}p$ to a feature matrix $\mathbf{Q}_{s,c}$ of size $m{\times}n$,
which is fed into the two head networks. The mapping matrix head consists of
$n$ parallel linear layers followed by softmax operation over $m$-dimensional
outputs. The adjacency matrix head applies average pooling of size $n_{2}$
along the column of $\mathbf{Q}_{s,c}$, followed by a multilayer perceptron
(MLP) and a sigmoid activation. Both of the two predicted matrices
$\hat{\mathbf{M}}_{s,c}$ and $\hat{\mathbf{A}}_{s,c}$ of the inferred graph
$\hat{\bm{g}}_{s,c}$ are probabilistic version of their binary format. To
learn the graph constructor, we use point reconstruction loss
$\mathcal{L}_{\mathrm{rec}}$ for the mapped points and log likelihood loss
$\mathcal{L}_{\mathrm{adj}}$ for all the entries of $\mathbf{A}_{s,c}$:

$\displaystyle\mathcal{L}_{\mathrm{rec}}=$
$\displaystyle\mathop{\mathbb{E}}_{s,c}\left[\left\lVert\hat{\mathbf{M}}_{s,c}^{T}\mathbf{P}_{s,c}-\mathbf{N}_{s,c}\right\rVert_{2}\right]$
(8) $\displaystyle\mathcal{L}_{\mathrm{adj}}=$
$\displaystyle-\Sigma_{ij}\mathop{\mathbb{E}}_{s,c}\left[A_{s,c}(i,j)\mathrm{log}\hat{A}_{s,c}(i,j)\right.$
$\displaystyle\left.+(1{-}A_{s,c}(i,j))\mathrm{log}(1{-}\hat{A}_{s,c}(i,j))\right],$
(9)

where $\mathbf{N}_{s,c}$ denotes the aligned node points on the ground truth
graph $\bm{g}_{s,c}$. We cannot apply loss on $\hat{\mathbf{M}}_{s,c}$ since
it is an operation on the point set $\mathbf{P}_{s,c}$ with no predefined
order.

Neural renderer $\mathcal{R}$. Lastly, the rasterized image of target glyph
can be generated by a neural renderer $\mathcal{R}$ conditioned on the
predicted graph $\hat{\bm{g}}_{s,c}$:

$\hat{\bm{x}}_{s,c}=\mathcal{R}(\hat{\bm{g}}_{s,c};\theta_{r,c}).$ (10)

The neural renderer $\mathcal{R}$ is designed as a multi-graph transformer
network [32] with glyph-specific parameter $\theta_{r,c}$. The adjacency
matrix $\hat{\mathbf{A}}_{s,c}$ defines a global graph of primary nodes and
the sequentially connected secondary nodes implied by $\hat{\mathbf{M}}_{s,c}$
produce a local graph for each glyph stroke. The graph attention enforced on
different scales helps to generate glyph images with a smooth global structure
as well as detailed local patterns. The neural renderer is trained by
minimizing the pixel-wise MSE loss:

$\mathcal{L}_{\mathrm{img}}=\mathop{\mathbb{E}}_{s,c}\left[\left\lVert\hat{\bm{x}}_{s,c}-\bm{x}_{s,c}\right\rVert_{2}\right].$
(11)

Note that it is also possible to use a conventional graphics renderer as a
surrogate for the neural renderer by fitting smooth Bezier curves to vectorize
the outline connecting the nodes in $\hat{\bm{g}}_{s,c}$.

Model training There are multiple components ($f_{e}$, $f_{d}$, $\mathcal{G}$,
$\mathcal{R}$) in the overall font completion model $f$, and some components
even have individual parameter instance for each glyph. Therefore, training
all the model parameters jointly is computationally prohibitive. In practice,
we first pre-train $f_{e}$ with $\mathcal{L}_{\mathrm{cls}}$ loss. Then we
jointly train $f_{e}$, $f_{d}$ and $\mathcal{G}$ with additional losses of
$\mathcal{L}_{\mathrm{rec}}$ and $\mathcal{L}_{\mathrm{adj}}$. Finally,
$\mathcal{R}$ is independently trained with loss $\mathcal{L}_{\mathrm{img}}$
for each glyph $c$. During the training of $\mathcal{G}$, we can load and
optimize a subset of glyph-specific parameters $\phi_{c}$ at each time to make
the memory usage independent of glyph set size. This training strategy works
well in our experiments and is scalable to newly added glyphs.

##  V Experiments

###  V-A Dataset

All our experiments are conducted on the Google Fonts dataset [11], and we
follow similar pre-processing steps as in [21] to filter out bad glyphs,
render images and extract font metadata. Glyphs with irregular designs such as
symbols and ornaments are removed to simplify our graph template design (see
below). We consider all the 26 Roman capital letters in the experiments. In
total, we have 2693/55554 valid fonts/glyphs. We use 95% fonts for
train/validation and 5% fonts for testing.

Graph Template. The image, SVG curves, and sample points of a glyph can be
obtained using standard procedures. To train our model, the ground truth for
the graph modality $\bm{g}_{s,c}$ still needs to be created.

For each glyph, we design an automatic procedure to build templates for all
font styles which specifies the number of primary nodes $n_{1}$, the number of
corresponding secondary nodes $n_{2}$, and the connections among all the
primary nodes $\mathbf{A}_{c}$. Following [22], we decompose each glyph into
three levels of hierarchy: contour, stroke, and node. Each stroke corresponds
to a primary node and the nodes on the stroke are secondary nodes. All the
primary nodes on the same contour loop are sequentially connected in the
counterclockwise direction. The primary nodes of different contours are not
connected. We keep the same total number of nodes as 150 for all the glyphs
and distribute the nodes to strokes according to the number of contours and
the relative curve length from their vector representations.

To obtain the node attribute matrix $\mathbf{N}_{s,c}$, we need to align all
the nodes from the graph template for $c$ to the SVG curves of the glyph, and
then calculate the exact coordinates and tangents. The same node should be
aligned to the corresponding position of all glyphs $c$ with different styles
$s$. To do that, we first use the keypoint detection model of [22] to roughly
segment all the strokes based on the glyph image. The nearest points on SVG
curves to those detected points are used as the locations of primary nodes.
Then all the secondary nodes are found by uniformly sampling SVG curves
between each pair of connected primary nodes. Figure 11 shows all the font
glyph modalities in our dataset.

Figure 4: Different data modalities of font glyphs. For each example from left
to right: image, keypoints, point set, graph template and directed graph
representation.

###  V-B Implementations

Encoder/Decoder Learning. The image encoder $f_{e}$ takes a $128{\times}128$
grayscale image as input and encodes a latent vector
$\bm{z}_{s}{\in}\mathds{R}^{128}$ using five Conv2d-BatchNorm-ReLU layers. A
linear mapping with sigmoid function is used for font classifier $h$ in Eq. 5.
The point set decoder $f_{d}$ uses four glyph-conditional
TranposeConv1D-BatchNorm-ReLU layers with the first layer shared by all
glyphs.

The image-to-image baseline (Table II and Figure 5) has the same encoder
structure as $f_{e}$, and its decoder uses five glyph-conditional
TranposeConv2D-BatchNorm-ReLU layers with the first layer shared by all
glyphs. The last layer is followed by a sigmoid function which generates an
output image with the same size as input.

Graph Constructor. We define graph node attributes $\mathbf{N}_{s,c}$ with
2-dim coordinates and 2-dim tangents of the corresponding points on SVG
curves. The adjacency matrix $\mathbf{A}_{s,c}$ reduces to $\mathbf{A}_{c}$ in
our case as the same graph template is used for any given $c$. In this way,
the loss term $\mathcal{L}_{\mathrm{adj}}$ is not activated. The mapping
$\mathbf{M}_{s,c}$ is still learned to align point set with graph template,
from which the relative position of a node on the stroke/contour can be
determined.

Neural Renderer. The neural renderer takes the node attributes
$\mathbf{N}_{s,c}$ (coordinates and tangents) and graph structure
$\mathbf{A}_{c}$ (adjacency matrix) as input. The coordinates and tangents are
translated to a 256-dim feature through a linear function. As $\mathbf{A}_{c}$
is fixed for any given glyph, we use a dictionary lookup table to implement
the graph attentions over strokes and contours defined by $\mathbf{A}_{c}$,
which results in another 256-dim feature. The node feature and edge feature
are concatenated and fed into a four-layer Conv1D network to produce a 128-dim
embedding, which further passes through five TranposeConv2D layers to
reconstruct the output image.

We train all the models on a single GeForce RTX 2080 Ti GPU for 100 epochs,
using Adam with learning rate $1{\times}10^{-3}$, betas $(0.9,0.999)$ and
batch size $64$.

###  V-C Results

####  V-C1 Font Completion

Quantitative Measure. We first quantitatively validate the contribution from
our proposed graph font representation and $\mathcal{L}_{\mathrm{cls}}$.
Specifically, we calculate $\mathrm{MSE}$, $\mathrm{PSNR}$ and $\mathrm{SSIM}$
against the ground truth glyph images, as listed in Table II. By optimizing
the graph reconstruction loss, our method outperforms image-to-image
translation by 35.3%, 29.4%, 8.8% on $\mathrm{MSE}$, $\mathrm{PSNR}$ and
$\mathrm{SSIM}$, respectively. This validates the necessity of introducing our
intermediate font graph representation. The performance can be further
improved by $\mathcal{L}_{\mathrm{cls}}$, i.e., with the learned font style
feature disentangled from glyph content.

TABLE II: Quantitative evaluation of font completion task for img2img baseline and our graph based method using different training losses. $\mathcal{L}_{\mathrm{img}}$ for img2img is the image reconstruction loss defined similarly as our renderer loss in Eq. 11. $\mathcal{L}_{\mathrm{rec}}$ is our graph reconstruction loss in Eq. 8. The classification loss $\mathcal{L}_{\mathrm{cls}}$ in Eq. 5 makes the learned style feature more distinctive of font style.  Method | Loss | $\mathrm{MSE}\downarrow$ | $\mathrm{PSNR}\uparrow$ | $\mathrm{SSIM}\uparrow$
---|---|---|---|---
img2img | $\mathcal{L}_{\mathrm{img}}$ | 0.17 | 8.14 | 0.57
img2graph2img (ours) | $\mathcal{L}_{\mathrm{rec}}$ | 0.11 | 10.53 | 0.62
$\mathcal{L}_{\mathrm{rec}}$ \+ $\mathcal{L}_{\mathrm{cls}}$ | 0.10 | 10.90 | 0.62

Visual Quality. As demonstrated in Figure 5 (one style for each panel),
comparing with previous TCN [8] method, our method (the third row in each
panel) produces better completed glyphs with both clear appearance and
consistent style with the target glyphs (first row in each panel), even if the
targets are unseen during training. In contrast, TCN suffers from serious
blur, artifacts, broken glyph structure, and fails to maintain style
information from the input glyph.

(a) ABeeZee-Italic

(b) Capriola-Regular

(c) Sansita-Italic

(d) Eczar-Regular

(e) Gudea-Italic

Figure 5: Font completion visualization. Three rows in each style panel:
ground truth glyphs (top), results of TCN [8] (middle), and our image-to-
graph-to-image approach (bottom). Red boxes indicate input glyphs. Our method
completes glyphs with clear appearance and consistent style with the target
glyphs.

####  V-C2 Font Manipulation

The goal of font manipulation is to make small modifications to the font style
of a source glyph, and transfer the changes to other glyphs of this font.
Considering that modifying several points on a graph is much easier than
changing a large number of pixels on an image, our proposed graph
representation and graph constructor can serve as powerful tools for human
designers to easily change typography properties like serif, italic, and
weight.

After modifying an original graph $\bm{g}_{s,c^{in}}$ to a modified one
$\tilde{\bm{g}}_{s,c^{in}}$, we can transfer the changes to another glyph
image $\tilde{\bm{x}}_{s,c}$ by inferring an updated style feature
$\tilde{\bm{z}}_{s}$ with either forward or backward propagation:

$\displaystyle\tilde{\bm{z}}_{s}$
$\displaystyle=f_{e}(\mathcal{R}(\tilde{\bm{g}}_{s,c^{in}};\theta_{r,c^{in}});\theta_{e}),$
(12) $\displaystyle\tilde{\bm{z}}_{s}$
$\displaystyle=\operatorname*{argmin}_{\bm{z}_{s}}\left\lVert\mathbf{M}^{T}_{s,c^{in}}f_{d}(\bm{z}_{s},c^{in};\theta_{d})-\tilde{\mathbf{N}}_{s,c^{in}}\right\rVert_{2}.$
(13)

In the back propagation of Eq. 13, we assume the manipulated graph
$\tilde{\bm{g}}_{s,c}$ keeps the same node mapping relation
$\mathbf{M}_{s,c^{in}}$ and only updates the node attributes
$\tilde{\mathbf{N}}_{s,c^{in}}$. In our experiment, the optimization is solved
by Adam optimizer with learning rate $10^{-4}$ for $10^{4}$ steps.

The updated style feature $\tilde{\bm{z}}_{s}$ will then be used to complete
the glyph set (Eq. 7 and Eq. 10). As illustrated in Figure 6, both forward and
backward methods can transfer the manipulated font style to other glyphs
successfully. This demonstrates the capability of the proposed graph
representation to capture even small local style variations, and also to
effectively transfer the style changes to image modality.

Figure 6: Font manipulation results for editing properties of serif, italic
and weight. Three rows in each style panel: ground truth before manipulation
(top), forward-propagation results (middle), back-propagation results
(bottom). Red boxes indicate input glyphs.

####  V-C3 Font Interpolation

Font interpolation is performed to validate the effectiveness of the learned
latent feature $\bm{z}_{s}$ in controlling glyph styles. Given two input
glyphs $\bm{x}_{{s_{1}},c^{\mathrm{in}}}$ and
$\bm{x}_{{s_{2}},c^{\mathrm{in}}}$ of the same content but different styles,
we first extract their font features $\bm{z}_{s_{1}}$ and $\bm{z}_{s_{2}}$
with $f_{e}$. With linear interpolation between the two features
$\lambda\bm{z}_{s_{1}}+(1{-}\lambda)\bm{z}_{s_{2}}$ for $\lambda\in[0,1]$, we
can generate all the glyph set for each interpolation coefficient $\lambda$ as
shown in Figure 7. The smooth transition from one style to another
demonstrates that our proposed graph representation can effectively capture
local glyph styles, and the linear transition in the latent feature space can
also be transferred to the image space.

Figure 7: Font interpolation results. The input glyphs
$\bm{x}_{{s_{1}},c^{\mathrm{in}}}$ and $\bm{x}_{{s_{2}},c^{\mathrm{in}}}$ are
highlighted by red boxes in the top and bottom rows. The middle rows are
interpolated with $\lambda=0.8,0.6,0.4,0.2$ (from top to bottom). The top and
bottom rows also show font completion results from the input glyphs.

##  VI Conclusion

In this work, motivated by the limitation of image modality for font
generation tasks, we proposed a new graph representation for font glyphs. To
better capture both global content and local details, we proposed a cross-
modality auto-encoder framework to leverage the graphs as an intermediate
representation for conversion between different font modalities. We build the
first graph representation dataset for font glyphs by transforming SVG curves
and point sets to hierarchical graphs. Our method achieves significant
performance improvements on font completion in both visual quality and
quantitative metrics, compared with the previous image-to-image translation
methods. Our graph representation also exhibits high scalability and
convenience for manual manipulation, which was demonstrated in font
manipulation and interpolation tasks. In summary, our graph representation and
cross-modality framework provide the font community with a new learning
strategy as well as a new benchmark.

##  Appendix A Graph Templates of All Glyphs

As listed in Table III, we keep the same total number of nodes for all the
glyphs and distribute the nodes to strokes according to the number of contours
and the relative curve length from their vector representations. We also
includes all designed templates in Fig. 11. The templates are consistent among
different font styles.

TABLE III: Decomposition of contours, strokes and nodes in our designed graph templates for all the Roman capital letters. Glyphs | Contours |  #strokes per contour |  #nodes per stroke |  Total #nodes
---|---|---|---|---
A,P,R |  outer inner |  15 3 |  8 10 | 150
D,O,Q |  outer inner |  15 4 |  6 15 | 150
B |  outer inner upper inner bottom |  15 3 3 |  6 10 10 | 150
others | outer | 15 | 10 | 150

##  Appendix B Dataset Statistics

Figure 8: Google Fonts dataset: the number of fonts per letter, distribution
of font categories, styles, and different stroke widths.

After data preprocess, the dataset has an imbalance distribution as shown in
Fig. 8. For some characters (“x”, “y”, “z”, etc.) there are not many glyphs
due to the failure in keypoint detection during template mapping. Besides,
since glyphs of sans serif, normal, and regular stroke width dominate the
dataset, font completion performance of these styles are slightly better than
the others.

##  Appendix C Additional Results

The image completion and graph completion visualizations in Fig.12 also
indicate that part of the font completion error comes from the graph
renderers, and the final completion performance can be improved by switching
to a better graph renderer. We also add the visualizations of the intermediate
graph representation in font completion task Fig. 12. We also compare the
single glyph font completion and multi-glyph font completion performance. As
we could see from Fig. 13, the single glyph font completion has a comparable
performance to multi-graph completion. However, for some font, i.e. ptmono-
PTM55FT, single glyph could not represent the style, therefore font completion
using different glyph as input results in inconsistent results (Fig. 14).

Figure 9: Different data modalities of font glyphs. For each example from left
to right: image, keypoints, point set, graph template and directed graph
representation. (To be continued.)

Figure 10: Different data modalities of font glyphs. For each example from
left to right: image, keypoints, point set, graph template and directed graph
representation. (To be continued.)

Figure 11: Different data modalities of font glyphs. For each example from
left to right: image, keypoints, point set, graph template and directed graph
representation.

(a) ABeeZee-Italic

(b) Capriola-Regular

(c) Sansita-Italic

(d) Eczar-Regular

(e) Capriola-Regular

(f) Gudea-Italic

Figure 12: Font completion visualization. Three rows in each style panel:
ground truth glyphs (top), results of our image-to-graph-to-image approach
(middle), and intermediate graph visualization of our image-to-graph-to-image
approach (bottom). Red boxes indicate input glyphs. Our method completes
glyphs with clear appearance and consistent style with the target glyphs.

Figure 13: Multi-input inference of our image-to-graph-to-image method, where
we use averaged style feature of all input to complete the remaining glyphs.

Figure 14: Failure case of image-to-graph-to-image (font style: ptmono-
PTM55FT).

## References

  * [1] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2414–2423.
  * [2] Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M.-H. Yang, “Universal style transfer via feature transforms,” in _Advances in neural information processing systems_ , 2017, pp. 386–396.
  * [3] Y. Li, M.-Y. Liu, X. Li, M.-H. Yang, and J. Kautz, “A closed-form solution to photorealistic image stylization,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 453–468.
  * [4] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1501–1510.
  * [5] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-resolution image synthesis and semantic manipulation with conditional gans,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8798–8807.
  * [6] M.-Y. Liu, T. Breuel, and J. Kautz, “Unsupervised image-to-image translation networks,” in _Advances in neural information processing systems_ , 2017, pp. 700–708.
  * [7] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [8] Y. Park, J. Lee, Y. Koh, I. Lee, J. Lee, and J. Kang, “Typeface completion with generative adversarial networks,” _arXiv preprint arXiv:1811.03762_ , 2018.
  * [9] _TrueType overview_ , https://docs.microsoft.com/en-us/typography/truetype/.
  * [10] _METAFONT – A system for specifying fonts_ , https://ctan.org/pkg/metafont?lang=en.
  * [11] Google, “Google fonts files.” _https://github.com/google/fonts_ , 2014.
  * [12] R. Suveeranont and T. Igarashi, “Example-based automatic font generation,” in _International Symposium on Smart Graphics_. Springer, 2010, pp. 127–138.
  * [13] N. D. Campbell and J. Kautz, “Learning a manifold of fonts,” _ACM Transactions on Graphics (TOG)_ , vol. 33, no. 4, pp. 1–11, 2014.
  * [14] P. Upchurch, N. Snavely, and K. Bala, “From A to Z: supervised transfer of style and content using deep neural network generators,” _arXiv preprint arXiv:1603.02003_ , 2016.
  * [15] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134.
  * [16] S. Azadi, M. Fisher, V. G. Kim, Z. Wang, E. Shechtman, and T. Darrell, “Multi-content GAN for few-shot font style transfer,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7564–7573.
  * [17] Y. Jiang, Z. Lian, Y. Tang, and J. Xiao, “DCFont: an end-to-end deep chinese font generation system,” in _SIGGRAPH Asia 2017 Technical Briefs_ , 2017, pp. 1–4.
  * [18] S. Yang, Z. Wang, Z. Wang, N. Xu, J. Liu, and Z. Guo, “Controllable artistic text style transfer via shape-matching GAN,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 4442–4451.
  * [19] T. Miyazaki, T. Tsuchiya, Y. Sugaya, S. Omachi, M. Iwamura, S. Uchida, and K. Kise, “Automatic generation of typographic font from small font subset,” _IEEE Computer Graphics and Applications_ , vol. 40, no. 1, pp. 99–111, 2019\.
  * [20] X. Yan, Y. Wang, R. Yi, Z. Sun, and Y. Liu, “Starfont: Enabling font completion based on few shots examples,” in _Proceedings of the 2019 3rd International Conference on Advances in Artificial Intelligence_ , 2019, pp. 1–8.
  * [21] R. G. Lopes, D. Ha, D. Eck, and J. Shlens, “A learned representation for scalable vector graphics,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019.
  * [22] D. Smirnov, M. Fisher, V. G. Kim, R. Zhang, and J. Solomon, “Deep parametric shape predictions using distance fields,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 561–570.
  * [23] J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei, “Image retrieval using scene graphs,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3668–3678.
  * [24] J. Johnson, A. Gupta, and L. Fei-Fei, “Image generation from scene graphs,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 1219–1228.
  * [25] M. Fisher, M. Savva, and P. Hanrahan, “Characterizing structural relationships in scenes using graph kernels,” in _ACM SIGGRAPH 2011 papers_ , 2011, pp. 1–12.
  * [26] L. Yi, H. Su, X. Guo, and L. J. Guibas, “SyncSpecCNN: Synchronized spectral CNN for 3D shape segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2282–2290.
  * [27] F. Tian, B. Gao, Q. Cui, E. Chen, and T.-Y. Liu, “Learning deep representations for graph clustering.” in _AAAI_ , vol. 14. Citeseer, 2014, pp. 1293–1299.
  * [28] S. Coles, _The Anatomy of Type: A Graphic Guide to 100 Typefaces_. Harper Design, 2012.
  * [29] S. Yun, M. Jeong, R. Kim, J. Kang, and H. J. Kim, “Graph transformer networks,” in _Advances in Neural Information Processing Systems_ , vol. 32. Curran Associates, Inc., 2019, pp. 11 983–11 993. [Online]. Available: https://proceedings.neurips.cc/paper/2019/file/9d63484abb477c97640154d40595a3bb-Paper.pdf
  * [30] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. Van Den Berg, I. Titov, and M. Welling, “Modeling relational data with graph convolutional networks,” in _European Semantic Web Conference_. Springer, 2018, pp. 593–607.
  * [31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , vol. 30. Curran Associates, Inc., 2017, pp. 5998–6008. [Online]. Available: https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * [32] P. Xu, C. K. Joshi, and X. Bresson, “Multi-graph transformer for free-hand sketch recognition,” _CoRR_ , vol. abs/1912.11258, 2019. [Online]. Available: http://arxiv.org/abs/1912.11258

|  Ye Yuan received her Ph.D. degree from the Department of Computer Science
and Engineering, Texas A&M University. Before that, she obtained her Bachelor
degree from University of Science and Technology of China. He has also
interned at Warmart AI Lab, Bytedance, and Adobe Research. Her research
interests are broadly in computer vision and machine learning.
---|---
|  Zhaowen Wang received the B.E. and M.S. degrees from Shanghai Jiao Tong
University, China, in 2006 and 2009 respectively, and the Ph.D. degree in ECE
from UIUC in 2014. He is currently a Senior Research Scientist with the
Creative Intelligence Lab, Adobe Inc. His research focuses on understanding
and enhancing images, videos and graphics via machine learning algorithms,
with a particular interest in sparse coding and deep learning.
---|---
|  Matthew Fisher is a senior research scientist at Adobe Research. He
received his PhD from Stanford University supported by a Hertz Fellowship
where he worked with Pat Hanrahan in the Graphics Lab. He completed his
undergraduate degree at the California Institute of Technology working with
Mathieu Desbrun. During a postdoc at Stanford, he has also worked on the data
analytics team at Khan Academy. His research focuses on combining computer
graphics, vision, and machine learning to make it faster and more fun to
complete creative tasks.
---|---
|  Wuyang Chen is a Ph.D. student in Electrical and Computer Engineering at
University of Texas at Austin. He received his M.S. degree in Computer Science
from Rice University in 2016, and his B.S. degree from University of Science
and Technology of China in 2014. Wuyang’s research focuses on addressing
domain adaptation/generalization, self-supervised learning, and AutoML. He has
published 6 papers in CVPR, ICLR and ICML as the first author, and co-authored
more.
---|---
|  Zhifei Zhang is currently a research engineer in Adobe Research. He
obtained the Ph.D. in 2018 supervised by Prof. Hairong Qi in computer
engineering at the University of Tennessee. He received the B.S. and M.S. in
2010 and 2013, respectively, from the Northeastern University and Zhejiang
University, China. His interest lies in deep learning based image synthesis,
computer vision, text detection and recognition, etc.
---|---
|  Zhangyang Wang is currently an Assistant Professor of Electrical and
Computer Engineering at University of Texas at Austin. He received his Ph.D.
degree in ECE from UIUC in 2016, advised by Professor Thomas S. Huang; and his
B.E. degree in EEIS from USTC in 2012. Prof. Wang is broadly interested in the
fields of machine learning, computer vision, optimization, and their
interdisciplinary applications. His latest interests focus on automated
machine learning (AutoML), learning-based optimization, machine learning
robustness, and efficient deep learning. His research is gratefully supported
by NSF, DARPA, ARL/ARO, as well as a few more industry and university grants.
He has co-authored over 130 papers, 2 books and 1 chapter; and has been
granted 3 patents. He has received many research awards and scholarships,
including most recently an ARO Young Investigator award, an IBM faculty
research award, an Amazon research award (AWS AI), an Adobe Data Science
Research Award, a Young Faculty Fellow of TAMU, and four research competition
prizes from CVPR/ICCV/ECCV.
---|---
|  Hailin Jin is a Senior Principal Scientist at Adobe Research. He received
his M.S. and Ph.D. in EE from WUSTL in 2000 and 2003. Between fall 2003 and
fall 2004, he was a postdoc researcher at the CS Department, UCLA. His current
research interests include deep learning, computer vision, and natural
language processing. His work can be found in many Adobe products, including
Photoshop, After Effects, Premiere Pro, and Photoshop Lightroom.
---|---