# Next-Gen Machine Learning Supported Diagnostic Systems for Spacecraft

Athanasios Vlontzos1111Contact Author
Accepted in the AI for Spacecraft Longevity Workshop at IJCAI 2021  Gabriel
Sutherland2 Siddha Ganju 3&Frank Soboczenski4
1Imperial College London, 2Oregon State University, 3NVIDIA Corp, 4King’s
College London
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Future short or long-term space missions require a new generation of
monitoring and diagnostic systems due to communication impasses as well as
limitations in specialized crew and equipment. Machine learning supported
diagnostic systems present a viable solution for medical and technical
applications. We discuss challenges and applicability of such systems in light
of upcoming missions and outline an example use case for a next-generation
medical diagnostic system for future space operations. Additionally, we
present approach recommendations and constraints for the successful generation
and use of machine learning models aboard a spacecraft.

##  1 Introduction

Space agencies have a renewed drive to take human space exploration beyond Low
Earth Orbit (LEO) and into deep space. NASA’s Artemis program outlines a clear
path to return to the Moon and to go beyond to Mars NASA (2017). Additionally,
recent successes in the commercial space sector by major players such as
SpaceX and Blue Origin make human spaceflight more accessible, affordable and
future long term-missions a reality. Yet future long duration spaceflight
require systems that are independent of LEO operations such as constant
communication, the ability to transfer large amounts of data via multiple
systems in a relatively short time-frame or the ability to request and
exchange crew if needed. On Earth, machine learning (ML) and machine
automation is already driving the next industrial revolution and resulted in
fully autonomous industrial processes in domains such as agriculture as well
as manufacturing Ayaz et al. (2019); Yang et al. (2019). Spaceflight itself,
however, is far behind such advances. Here we discuss challenges ML supported
systems face in the space domain as well as the applicability and advantages
of ML systems on a spacecraft. We highlight aforementioned items via an
example of an autonomous medical system and describe an infrastructure for the
successful development of such systems.

##  2 Challenges for ML aboard Spacecraft

Space is hard and manned space exploration is dangerous and unforgiving.
Moreover, the long term effects of human presence in microgravity are still
not fully known. Current space missions do rarely include a medical officer
among the crew and rely on specialists who are also trained in emergency
medicine. While this is sufficient for minor and trained emergency cases, it
does not allow for more serious and complex medical treatments. Health related
checks and emergencies are handled in a tele-medicine regime where
instructions are communicated to the astronauts via ground to spacecraft
channels. In extreme emergencies the astronauts can always make the journey
back on Earth. As missions veer further away from Earth, returning for medical
treatment and relying on simultaneous communications becomes infeasible as
both distance and communication latency increases exponentially. For a
deployed ML system aboard a spacecraft there are several challenges to
consider: a) limited live testing abilities available, and testing in the form
of payloads on missions are expensive; b) systems that are deployed need to be
at a high technology readiness level (TRL) Lavin et al. (2021); c)
environmental effects may influence deployed systems. For example, how
ionizing radiation can affect deployed space capable hardware Page and
Benedetto (2005), affecting the consistency of sensor behavior such as early
or late sensor fusion; leading to potentially corrupted data to be processed
e) payloads are constrained by weight, so shipping large compute
infrastructures is infeasible; f) fully autonomous applications can be found
mainly on controlled environments that assume almost complete access to
information and environmental parameters; g) and more importantly the lack of
labeled data for each task along with limited interpretability and
explainablity of current ML systems add to the complexity. Contrary to that,
any space-faring vehicle is faced with extreme environmental conditions that
not only are hard to control or predict, but in some cases are challenging to
human scientific comprehension. Hence, ML systems must be as robust as
possible to changes of their operating environments.

##  3 Applicability & Advantages

Incorporating autonomous processes in a spacecraft is a complex task as
technology and the associated needs constantly develop. Some key points,
however, are: (1) Reduce Latency and Earth Reliance - A significant amount of
operations aboard modern day spacecrafts and the ISS, require the constant
communication with mission control on Earth Dempsey (2017). As space
exploration expands further away from LEO and the Moon, the delay in
communication represent an insurmountable obstacle for remote guidance,
control and communications. Characteristically, we note that the round-trip
time for current communications with Mars ranges between 5 and 20 minutes
depending on the state of the two planets orbits NASA (2020). ML, thus, can
resolve the dependency on communications and perform the mission critical
information processing aboard. (2) Adapting Maintenance - Modern spacecrafts
both manned and unmanned, constitute extremely complex systems that even with
the use of automated checks are still prone to faults, especially when faced
with extraordinary circumstances. The crew might not be able to solve the
issues by themselves. To tackle this we believe that ML systems can perform
functions that transcend anomaly detection and fault prediction. An ideal
deployed ML solution needs to account for automated maintenance and resolution
of faults both in hardware and in the software of the spacecraft. (3) Reduce
latency for functions that don’t require manual intervention and can be
conducted at scale. For example, automated checksums for data or models (4)
Recent advances in ML like bit quantization, pruning, and hardware
approximations Wang et al. (2019) enable inference on resource constraint edge
devices which can also be similar to the target hardware in space Hiemstra et
al. (2020).

##  4 Medical Use Case

In the previous sections we have set out some challenges as well as advantages
for the use of ML on spacecrafts. In this section we will be exploring the
medical use case. As mentioned before, medical treatment in space relies on
communication with the earth, a reliance we need to remove as we progress into
deep space missions. A major part of medical treatment is the ability to
inspect bodily functions in a non invasive manner through the use of medical
imaging devices. Magnetic Resonance (MRI), CT-Scans and X-Rays are widely used
on earth but are not ideal for space use, as we explain below. To this effect,
we propose the use of Point of Care Ultrasounds (POCUS) as a lightweight non
ionizing imaging solution.

ML Enabled Imaging: Medical imaging devices are often slow, requiring
significant resources to operate and store and depend on the use of ionizing
radiation (e.g. X-Rays/CT Scans) that has negative effects on patients and
doctors alike FDA (2021). One notable exception to the above constraints is
Point of Care Ultrasounds (POCUS). Mobile device enabled Ultrasound (US)
probes are safe for patients and doctors, require minimal resources to operate
—weighting less than 1kg— and offer real time imaging capabilities
GEHealthcare (2021). However, acquiring medically significant images with a
POCUS probe is non trivial and requires expert operators.

As expert users might not always be available in deep space missions we
envision the use of ML enabled POCUS probes that guide the user to medically
relevant areas.

Approaching the task of navigation one can identify straight away the need to
be able to determine where the intelligent agent is with respect to the
patients body. Basic anatomy knowledge by both patient and other crew members
is assumed for the purposes of this application. Navigating to points of
medical interest, though is non trivial. Following medical practice of first
identifying standard planes and anatomical landmarks; reinforcement learning
(RL) solutions of disembodied agents have been proposed in Alansary et al.
(2019); Vlontzos et al. (2019) to perform the above tasks, providing high
accuracy and low computational constraints. Milletari et al. (2019); Hase et
al. (2020); Li et al. (2021) expand the RL methods including the degrees of
freedom of the US probe into the agents action space and optimize directly on
identifying landmarks while controlling a virtual probe. All the above
methods, find themselves limited to the anatomies that they have been trained
on, with their computational burden rising exponentially when trained to find
standard planes or landmarks of multiple anatomies.

As such we believe that a hierarchical approach to the problem would help keep
complexity and computation low enough for space craft applications while
maintaining high enough performance. Approaches like MAX-Q Dietterich (2000)
have withstood the test of time and have theoretical guarantees on convergence
and the ability of the algorithm that given a set of subtasks it can find the
global optimal policy. In short, MAX-Q constructs a hierarchical action tree
that a higher level agent sets out subtasks for other agents. In this fashion
one can design a fully autonomous agent that collects information in form of
images through POCUS navigation, assesses biomarkers based on measurements
derived from landmarks, and regresses the diagnosis.

Furthermore, Eslami et al. (2018); Dinesh and Grauman (2016); Ramakrishnan and
Grauman (2018) have shown that incorporating modules that aid agents ”imagine”
how scenes would look like from different point of views, increases their
performance in scene understanding. Hence, we are firm believers that
incorporating such modules in approaches like Milletari et al. (2019) would
increase their performance capabilities.

Finally, as medical applications are of critical importance, appropriate
checks and balances should be put in place to avoid any harm to the crew. We
believe that a rule based set of parameters should be developed in
collaboration with physicians that would constitute a fallback system. Having
no dependence on learned or inferred information we are able to guarantee a
basic level of care to the astronauts in case all other systems fail. On top
of the safety-net rule based system, a causal inference infrastructure can be
used to assess the the probabilities of causality (Sufficiency and Necessity)
Buesing et al. (2018); Kusner et al. (2017); Louizos et al. (2017); Vlontzos
et al. (2020); Budd et al. (2021). Causal counterfactual inference enable us
to assess the causal links and potential outcomes of treatments providing,
thus, a more informed decision process.

##  5 Infrastructure considerations for ML systems on Spacecraft

Developing novel ML methods to provide mission critical treatments to
astronauts is a hard task. In the previous section we set out some thoughts on
a potential application for medical imaging aboard a spacecraft. In this
section we will be briefly exploring infrastructure considerations that are
directly related to the operation and development of medical imaging ML
solutions. These considerations are also apparent on earth bound ML
applications but gain increased importance due to the edge cases that they are
called to operate on during spaceflight.

Data Collection Medical data acquisition is faced with a series of challenges.
The privacy of the patients donating their data has to be protected throughout
the data acquisition and model development processes. In order to comply with
all legal obligations, we propose the data to have NIHR NIHR (2021) approved
methods in place for making patient data anonymous. While, storage of data
must be in a HIPAA HIPPA (2021) compliant platform. Federated learning also
present promising methods in regards to secure data processing Konečnỳ et al.
(2016). Going beyond the need of anonymization and maintenance we would like
to draw the readers attention to two constraints of increased importance on
medical ML applications

Domain Shift: Domain shift robustness is an open topic of ML research that
focuses on making existing methods robust to distributional shifts from the
underlying training data domain Long et al. (2016); Zhang et al. (2020). In
medical applications and more prominently in medical imaging applications
domain shifts are easier to manifest and harder to overcome. The first major
factor in this phenomenon comes from the equipment used. Medical equipment,
when installed in hospitals and health centers is configured by the seller to
the exact specifications of the attending physicians, as such, two different
doctors using the same base equipment on the same patient can result in two
quite different images. The standardization of medical equipment
configurations in space missions would aid decrease the equipment induced
domain shifts.

Another domain shift source comes from the training patient characteristics.
Different populations exhibit different medical characteristics. Hereditary
traits as well as phenotype derived attributes place an individual in
different medical risk groups and force pathologies to manifest with different
probabilities. These differences, are often picked up by ML algorithms as
unwanted inductive biases, skewing the learned conclusions. In the context of
space missions, astronauts have diverse backgrounds both genetically and in
terms of phenotype, as such models trained on a general population not
representative of the characteristics of the crew can provide skewed
estimations of vital for diagnosis biomarkers. It is imperative, then, that
the deployed models are calibrated to the genetic background and phenotype of
the astronauts.

Anomalous phenomena: As stated in the above motivation, the effects on human
well-being stemming from prolonged exposure to cosmic radiation and other
space related phenomena are not fully understood. It is unknown, hence how the
human body might react to adverse conditions. ML applications cannot, then, be
expected to cover the full range of scenarios, on the contrary they should be
expected to fail when presented with data that constitute an anomalous effect.
In order to combat potential failure cases of the medical systems on-board a
spacecraft, we suggest accompanying any ML algorithm with an anomaly detection
mechanism (perhaps probabilistic or out-of-distribution mechanics), that flags
non standard bio markers Winkens et al. (2020), and an active learning Settles
(2012) feedback pipeline such that new, medical phenomena are incorporated in
the evolution of the medical algorithms on-board.

Computational Considerations High communication latency creates the demand for
on-board computation. However, cosmic radiation and weight offers strict
constraints on the available on-board compute resources. Cosmic radiation is
able to produce errors in modern scale computing units, while weight
restrictions exist in all parts of a mission, from lift off to grounding. ML
applications should, then, be able to function without heavy computational
needs, a non trivial feat especially on medical imaging algorithms. This,
reinforces our case towards the use of POCUS as they have been shown to be
computationally and physically lightweight. Recently Hiemstra et al. (2020)
has shown that NVIDIA’s Jetson Xavier modules are able to withstand a
significant level of proton based radiation, making them optimal candidates
for on-board inference and fine-tuning infrastructure. NASA has also awarded
an SBIR contract to Numem Gharia (2020) to develop a radiation hardened DNN
co-processor for a wide variety of ML applications, from low power machine
vision to healthcare.

##  6 Conclusion

ML supported medical diagnostic systems on spacecraft are necessary for long-
term space mission to overcome limitations in ground-to-spacecraft
communication and lack of qualified medical crew. In this extended abstract,
we outlined the importance of incorporating ML enabled medical applications on
spacecrafts and considered challenges that need to be overcome in terms of
anomaly detection and domain adaptation. Finally, we discussed ideas on how to
augment existing POCUS algorithms such that they constitute a complete
diagnostic system. We gave the example of a hierarchical RL method that also
includes causal inference checks and balances such that the health and safety
of crew members is guaranteed. We hope that with this paper we highlighted the
final frontier for ML applications: space capable ML medical systems.

##  7 Acknowledgments

We would like to thank NASA, the Canadian Space Agency (CSA), European Space
Agency (ESA), NVIDIA, the SETI Institute and Frontier Development Lab (FDL)
for their continuous support and for making this collaboration possible.

## References

  * Alansary et al. [2019] Amir Alansary, Ozan Oktay, Yuanwei Li, Loic Le Folgoc, Benjamin Hou, Ghislain Vaillant, Konstantinos Kamnitsas, Athanasios Vlontzos, Ben Glocker, Bernhard Kainz, et al.  Evaluating reinforcement learning agents for anatomical landmark detection.  Medical image analysis, 53:156–164, 2019.
  * Ayaz et al. [2019] Muhammad Ayaz, Mohammad Ammad-Uddin, Zubair Sharif, Ali Mansour, and El-Hadi M. Aggoune.  Internet-of-things (iot)-based smart agriculture: Toward making the fields talk.  IEEE Access, 7:129551–129583, 2019.
  * Budd et al. [2021] Samuel Budd, Arno Blaas, Adrienne Hoarfrost, Kia Khezeli, Krittika D’Silva, Frank Soboczenski, Graham Mackintosh, Nicholas Chia, and John Kalantari.  Prototyping crisp: A causal relation and inference search platform applied to colorectal cancer data.  In 2021 IEEE 3rd Global Conference on Life Sciences and Technologies (LifeTech), pages 517–521. IEEE, 2021.
  * Buesing et al. [2018] Lars Buesing, Theophane Weber, Yori Zwols, Sebastien Racaniere, Arthur Guez, Jean-Baptiste Lespiau, and Nicolas Heess.  Woulda, coulda, shoulda: Counterfactually-guided policy search.  arXiv preprint arXiv:1811.06272, 2018.
  * Dempsey [2017] Robert C. Dempsey.  The International Space Station: operating an outpost in the new frontier.  National Aeronautics and Space Administration, Lyndon B. Johnson Space Center, 2017.
  * Dietterich [2000] Thomas G Dietterich.  Hierarchical reinforcement learning with the maxq value function decomposition.  Journal of artificial intelligence research, 13:227–303, 2000.
  * Dinesh and Grauman [2016] Jayaraman Dinesh and Kristen Grauman.  Look-Ahead Before You Leap : End-to-End Active Recognition.  ECCV, 2016.
  * Eslami et al. [2018] S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier, Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis.  Neural scene representation and rendering.  Science, 360(6394):1204–1210, 2018.
  * FDA [2021] FDA.  Medical x-ray imaging.  https://www.fda.gov/radiation-emitting-products/medical-imaging/medical-x-ray-imaging, 2021\.
  * GEHealthcare [2021] GEHealthcare.  Point of care ultrasounds.  https://www.gehealthcare.co.uk/products/ultrasound/point-of-care-ultrasound, 2021\.
  * Gharia [2020] Nilesh Gharia.  Dnn radiation hardened co-processor companion chip to nasa’s upcoming high-performance spaceflight computing processor.  https://www.sbir.gov/node/1882269, 2020.
  * Hase et al. [2020] Hannes Hase, Mohammad Farid Azampour, Maria Tirindelli, Magdalini Paschali, Walter Simson, Emad Fatemizadeh, and Nassir Navab.  Ultrasound-guided robotic navigation with deep reinforcement learning, 2020.
  * Hiemstra et al. [2020] D. M. Hiemstra, C. Jin, Z. Li, R. Chen, S. Shi, and L. Chen.  Single event effect evaluation of the jetson agx xavier module using proton irradiation.  In 2020 IEEE Radiation Effects Data Workshop (in conjunction with 2020 NSREC), pages 1–4, 2020.
  * HIPPA [2021] HIPPA.  Hippa data protection guidelines.  https://www.hhs.gov/hipaa/for-professionals/security/laws-regulations/index.html, 2021\.
  * Konečnỳ et al. [2016] Jakub Konečnỳ, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon.  Federated learning: Strategies for improving communication efficiency.  arXiv preprint arXiv:1610.05492, 2016.
  * Kusner et al. [2017] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva.  Counterfactual fairness.  In Advances in Neural Information Processing Systems, pages 4066–4076, 2017.
  * Lavin et al. [2021] Alexander Lavin, Ciarán M. Gilligan-Lee, Alessya Visnjic, Siddha Ganju, Dava Newman, Sujoy Ganguly, Danny Lange, Atılım Güneş Baydin, Amit Sharma, Adam Gibson, Yarin Gal, Eric P. Xing, Chris Mattmann, and James Parr.  Technology readiness levels for machine learning systems, 2021.
  * Li et al. [2021] Keyu Li, Jian Wang, Yangxin Xu, Hao Qin, Dongsheng Liu, Li Liu, and Max Q. H. Meng.  Autonomous navigation of an ultrasound probe towards standard scan planes with deep reinforcement learning, 2021.
  * Long et al. [2016] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.  Unsupervised domain adaptation with residual transfer networks.  arXiv preprint arXiv:1602.04433, 2016.
  * Louizos et al. [2017] Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling.  Causal effect inference with deep latent-variable models.  In Advances in Neural Information Processing Systems, pages 6446–6456, 2017.
  * Milletari et al. [2019] Fausto Milletari, Vighnesh Birodkar, and Michal Sofka.  Straight to the point: reinforcement learning for user guidance in ultrasound, 2019.
  * NASA [2017] NASA.  Nasa artemis program.  https://www.nasa.gov/artemisprogram, 2017.
  * NASA [2020] NASA.  Nasa mars 2020 mission perserverance rover.  http://shorturl.at/xzI25, 2020.
  * NIHR [2021] NIHR.  Data sharing guidelines.  https://www.nihr.ac.uk/documents/nihr-position-on-the-sharing-of-research-data/12253, 2021\.
  * Page and Benedetto [2005] T.E. Page and J.M. Benedetto.  Extreme latchup susceptibility in modern commercial-off-the-shelf (cots) monolithic 1m and 4m cmos static random-access memory (sram) devices.  In IEEE Radiation Effects Data Workshop, 2005., pages 1–7, 2005\.
  * Ramakrishnan and Grauman [2018] S K Ramakrishnan and K Grauman.  Sidekick Policy Learning for Active Visual Exploration.  In EECV, 2018.
  * Settles [2012] Burr Settles.  Active learning.  Synthesis Lectures on Artificial Intelligence and Machine Learning, 6(1):1–114, 2012.
  * Vlontzos et al. [2019] Athanasios Vlontzos, Amir Alansary, Konstantinos Kamnitsas, Daniel Rueckert, and Bernhard Kainz.  Multiple landmark detection using multi-agent reinforcement learning.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 262–270. Springer, Cham, 2019.
  * Vlontzos et al. [2020] Athanasios Vlontzos, Henrique Bergallo Rocha, Daniel Rueckert, and Bernhard Kainz.  Causal future prediction in a minkowski space-time.  arXiv preprint arXiv:2008.09154, 2020.
  * Wang et al. [2019] Erwei Wang, James J. Davis, Ruizhe Zhao, Ho-Cheung Ng, Xinyu Niu, Wayne Luk, Peter Y. K. Cheung, and George A. Constantinides.  Deep neural network approximation for custom hardware.  ACM Computing Surveys, 52(2):1–39, May 2019.
  * Winkens et al. [2020] Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R. Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, Taylan Cemgil, S. M. Ali Eslami, and Olaf Ronneberger.  Contrastive training for improved out-of-distribution detection, 2020\.
  * Yang et al. [2019] Hui Yang, Soundar Kumara, Satish Bukkapatnam, and Fugee Tsung.  The internet of things for smart manufacturing: A review.  IIE Transactions, pages 1–35, 01 2019.
  * Zhang et al. [2020] Yifan Zhang, Ying Wei, Qingyao Wu, Peilin Zhao, Shuaicheng Niu, Junzhou Huang, and Mingkui Tan.  Collaborative unsupervised domain adaptation for medical image diagnosis.  IEEE Transactions on Image Processing, 29:7834–7844, 2020.


#  Automatic Generation of Typographic Font from a Small Font Subset

Tomo Miyazaki, Tatsunori Tsuchiya, Yoshihiro Sugaya, Shinichiro Omachi,
Masakazu Iwamura, Seiichi Uchida, and Koichi Kise Manuscript created September
25, 2015. This work was partially supported by JST, CREST, and JSPS KAKENHI
Grant Numbers 16H02841 and 16K00259.T. Miyazaki, Y. Sugaya, and S. Omachi are
with Tohoku University (e-mail: tomo@iic.ecei.tohoku.ac.jp).T. Tsuchiya was
with Tohoku University and is now with Canon, Inc.M. Iwamura and K. Kise are
with Osaka Prefecture University, Japan.S. Uchida is with Kyushu University,
Japan.

###### Abstract

This paper addresses the automatic generation of a typographic font from a
subset of characters. Specifically, we use a subset of a typographic font to
extrapolate additional characters. Consequently, we obtain a complete font
containing a number of characters sufficient for daily use. The automated
generation of Japanese fonts is in high demand because a Japanese font
requires over 1,000 characters. Unfortunately, professional typographers
create most fonts, resulting in significant financial and time investments for
font generation. The proposed method can be a great aid for font creation
because designers do not need to create the majority of the characters for a
new font. The proposed method uses strokes from given samples for font
generation. The strokes, from which we construct characters, are extracted by
exploiting a character skeleton dataset. This study makes three main
contributions: a novel method of extracting strokes from characters, which is
applicable to both standard fonts and their variations; a fully automated
approach for constructing characters; and a selection method for sample
characters. We demonstrate our proposed method by generating 2,965 characters
in 47 fonts. Objective and subjective evaluations verify that the generated
characters are similar to handmade characters.

###### Index Terms:

Font generation, Active shape model, Computer aided manufacturing, Kanji
character.

##  I Introduction

Automatic generation of Japanese font is important for lowering the costs
associated with creating kanji111Kanji are Chinese characters adapted for the
Japanese language. characters. All of the characters in a font need to be
designed in a particular style and size, a process generally performed by
humans. Moreover, font creation is professional and time-consuming work.

The cost problem is especially crucial in languages that contain a very large
number of characters, such as Chinese, Korean, and Japanese. Most commercial
fonts contain at least 1,006 kanji characters, which are defined as those in
daily use by the Japanese Industrial Standards
Committee222http://www.jisc.go.jp. Chinese and Korean each use more than 6,000
characters. These numbers are far greater than those of other major languages,
e.g., 26 characters in English, 27 in Spanish, 49 in Hindi, 28 in Arabic, and
33 in Russian. Thus, it takes a few years even for a professional designer to
create a font.

Previous work on character generation is shown in Table I. We categorized
previous studies by their inputs and outputs into three problem settings. The
first problem setting is to generate a font using parameters for handwritten
[1, 2, 3, 4, 5, 6] or typographic fonts [7, 8, 9, 10, 11, 12]. The second
problem setting is to generate a font by blending several complete fonts333A
complete font contains a sufficient number of characters for daily use, e.g.,
26 for English and over a thousand for Japanese. for handwritten [13, 14, 15]
or typographic fonts [16, 17, 18, 19]. The third problem setting is to
generate a font by extrapolating characters given a subset of handwritten [20,
21, 22, 23, 24, 25, 26, 27] or typographic fonts [28]. Each problem setting
has been an important research topic.

TABLE I: Categories of related work

In this study, we address the problem of generating a typographic font from a
given subset of characters. From a subset of a typographic font that includes
only a small number of characters, we generate missing characters to obtain a
complete typographic font. Although the proposed method requires a subset of a
particular typographic font, the burden of creating a subset of a font is much
less than creating an entire font. This benefit is highlighted in languages
containing many characters, such as Chinese and Japanese. The proposed method
uses sample characters to extract strokes and constructs characters by
deploying them. Fig. 1 illustrates an overview of the proposed method, which
begins with stroke extraction from sample characters. Strokes are extracted
using the skeletons of the samples. We take the skeletons of a target
character from the skeleton dataset and transform it into the structure of the
target font. Finally, we select some strokes and deploy them to the target
skeletons.

Figure 1: Overview of the proposed method.

The proposed method accepts samples in an image format. For practicality, the
ideal font generation method must accept characters in an image format and
require only a small subset of the font as input. First, we consider the image
format. For the purpose of character recognition, character generation from
samples in an image format is far more useful than from those in a vector
format. Image formats are more convenient for sample collection. For example,
when users encounter text in an unknown font, it is difficult to find samples
in a vector format, whereas an image is available immediately upon taking a
photo of the text. Likewise, sample collection is time-consuming work and
could be an obstacle for the popularization of a font generation method if too
many samples are needed.

Our automatic generation framework makes three main contributions, each of
which is briefly introduced below and described in detail in subsequent
sections.

The first contribution of this study is a stroke extraction method. We propose
a method that extracts character strokes by applying a novel active contour
model that determines the boundaries of strokes to extract. Some strokes are
damaged because of the complexity of characters; therefore, we incorporate a
restoration process for fixing these damaged strokes.

The second contribution of this study is a method for the automatic
construction of characters in two phases: modifying skeletons in the target
font and deploying strokes to them. The proposed method extracts the
transformation parameters. With these parameters, we can unify the skeletons
in size and geometry. Because our method transforms skeletons from a standard
font (such as Mincho) to fit the target font, it is unnecessary to build
skeleton datasets for a large number of fonts. This approach helps us with
variety and scalability. We can generate as many characters as the skeletons
in the dataset will allow (210,000 characters are available at this time). For
automation, we define an energy function to deploy a stroke so that an
appropriate one can be added to the skeleton.

The third contribution of this study is a sample selection method. We address
which characters are suitable for the proposed method and how we can select
them based on the energy function used in character generation. Finding
samples is combinatorial problem, and it is difficult to obtain a solution
analytically. Therefore, we apply a genetic algorithm to find an approximate
solution.

##  II Related work

In this section we review the relevant work illustrated in Table I.

#### Parameter-based methods

The methods in this category use parameters acquired by analyzing fonts. One
of the most famous methods is Metafont[7], which use the shapes of pen tips
and pen-movement paths as parameters for character generation. [8] was
inspired by Metafont. Further, [9, 10] analyzed characters in the Times font
family to extract parameters. Character properties, such as thickness and
aspect ratio, are changed using these parameters. A method for generating
Japanese characters was addressed by Tanaka et al. [11], who created a
database of skeletons of Japanese characters, including kanji. The characters
were generated by placing thick lines in the skeleton data. A famous
noncommercial font444https://www.tanaka.ecc.u-tokyo.ac.jp/ktanaka/Font/ was
created by applying this method. Kamichi extended the skeleton database in
[12].

There have been attempts to analyzing handwritten characters. Bayoudh et al.
used Freeman chain-code to analyze alphabetical characters [1]. Djioua and
Plamondon used a Sigma lognormal model supported by the kinematic theory [2].
An interactive system was developed so that the user can easily fit a Sigma-
lognormal model to alphabetical characters. Wada et al. extracted the
trajectories of alphabetical characters and replaced them using a genetic
algorithm [3]. Zheng and Doermann adopted a thin plate spline to model an
alphabetical character and generated a new alphabetical character by
calculating the intermediate of the two [4]. Handwriting models for robot arms
were developed in [5, 6].

Parameter-based methods generate clean characters. However, they are only
applicable to particular fonts that are analyzed by humans. For instance, the
method in [11] accepts only two fonts: Mincho and Gothic. This drawback comes
from the method’s requirement for precise analysis. In addition, the methods
cannot be applied to Japanese fonts, since over a thousand of characters need
to be parameterized. Moreover, special equipment is required to accurately
parameterize alphabetical characters, such as an interactive system or device
for acquiring trajectories. Unlike these methods, the method proposed in this
study generates characters without requiring precise analysis. Thus, only a
few character sample images are required. As we discussed in the previous
section, preparing sample images is a simple task.

#### Font blending methods

Methods in this category receive several complete fonts and generate a font by
blending them. Xu et al. generated Chinese calligraphy characters using a
weighted blend of strokes in different styles [13]. They decomposed samples
into radicals and single strokes based on rules defined by expert knowledge.
An improved method [14] considers the spatial relationship of strokes. Choi et
al. generated handwriting Hangul characters using a Bayesian network trained
with a given font[15].

Suveeranont and Igarashi addressed a generation of alphabetical characters for
typographic fonts [16]. They generated characters by blending predefined
characters from miscellaneous complete fonts. This method is based on a vector
format, in contrast with the proposed method, which accepts an image format.
Campbell and Kautz learned a manifold of standard fonts of alphabetical
characters [17]. Locations on the manifold represent a new font. Feng et al.
used a wavelet transform to blend two fonts [18]. Tenenbaum and Freeman used
complete fonts as references to form characters from the generation results
[19].

The methods in this approach require human supervision to generate a desired
font; since they automatically blend fonts, the result is not always the
desired font. In order to avoid this problem, blending weights [13] or an
interactive system [16] are implemented. However, sweeping the weights and
using the system require human supervision.

In contrast, the proposed method generates a typographic font with little
human supervision. Instead of blending fonts, we construct characters with
strokes extracted from sample characters in the desired font. Therefore, the
proposed method can directly provide the user with the desired font.

#### Character extrapolation methods

The aim of methods in this category is to extrapolate characters not included
in a given subset. Attempts to complete this process on handwritten fonts can
be found in [20, 21, 22, 23, 24, 25, 26, 27]. Lin et al. generated characters
with components extracted from given characters [20] using an annotated font
in which the positions and sizes of components were labeled. The extraction of
the components was performed on electronic devices so that characters can be
easily decomposed. Zong and Zhu developed a character generation method using
machine learning [21]. They decomposed the given characters into components by
analyzing the orientation of strokes. Components were assigned to a reference
font with a similarity function trained by a semi-supervised algorithm. Wang
et al. focused on the spatial relationships of the character components for
decomposition and generation [22]. An active contour model was used for
decomposition [23].

Character component decomposition is a crucial technique for the methods in
this category. However, most use naive decomposition that rely on spatial
relationships [23, 22, 25, 26, 24] or special devices [20, 27]. It is
difficult to extract components when they are connected or decorated. To the
best of our knowledge, the method in [28] is the only method that is
applicable to characters in fonts with decorations. Saito et al. applied a
patch transform [29] to samples and generated alphabetical characters in wide
range of fonts [28]. However, the generated results did not meet the criteria
for practical use. In this paper, we propose an adaptive active contour model
for component extraction. With the proposed method, we can obtain natural
character strokes even in decorated characters.

##  III Skeleton dataset

First, we address the character skeleton dataset used in this study. We
created the dataset based on GlyphWiki [30], which contains data from more
than 210,000 of characters555It includes characters created by users that are
not used publicly.. GlyphWiki utilizes a wiki format, allowing anybody to
contribute to and maintain the database. Each stroke of a character is stored
in KAGE format [12] which uses four attributes: control points, line type,
start shape, and end shape. There are six line types, seven start shapes, and
15 end shapes. The number of control points depends on line type and is at
most four. The control points and line types lead to a rough stroke, then the
start and end shapes provide details of a stroke. Fig. 2 (a) illustrates a
character in KAGE format in Mincho.

|
---|---
(a) | (b)
Figure 2:  Visualization of the GlyphWiki data and our dataset. (a) Data from
GlyphWiki drawn in Mincho. The red triangles represent the control points. The
line types of the left and right strokes are “curve” and “straight” in the
middle. All start and end shapes are different. (b) The skeletons in our
dataset. The control points, line type, start shape, and end shape are
attributes.

We created skeletons of the strokes in a character using GlyphWiki data. Fig.
2(b) illustrates the skeletons of a character in our dataset. We draw a line
of a skeleton from the first control point to the last control point. An
additional line of a skeleton is drawn according to start shape and end shape.
For example, the end shape is drawn in the middle stroke in Fig. 2(b). We draw
a line according to the line type and the number of control points. A straight
line is drawn when the line type is straight and two control points are given.
A B-spline curve is drawn when the line type is curved and three control
points are given. A straight line and B-spline curve are drawn when the line
type is vertical and four control points are given. We include the attributes
with each skeleton. We extract points on a skeleton by regular sampling from
the start point to the end point. Therefore, a skeleton is composed of a set
of sampling points. By using sampling points, we define a skeleton $S$ as

$S=\left[\begin{array}[]{ccc}\cdots&x_{i}&\cdots\\\ \cdots&y_{i}&\cdots\\\
\cdots&1&\cdots\end{array}\right],$ (1)

where $x_{i}$ and $y_{i}$ represent the $x$\- and $y$-coordinates of $i$th
sampling point, respectively. We adopted homogeneous coordinates.
$P=\left[x,y,1\right]^{\mathrm{T}}$ denotes a vector that represents a
sampling point. Then, $S=\left[\cdots,P_{i},\cdots\right]$. We denote a
skeleton in our dataset by $S$, a skeleton in the samples by $\hat{S}$, and a
transformed skeleton666We described the details of the transformed skeletion
in Section V-A. by $\tilde{S}$ (See Fig. 1).

##  IV Stroke extraction

When the proposed method receives samples as an input, we extract the strokes
of the samples. Fig. 3 shows examples of extracted strokes.

Figure 3: Stroke extraction. The black pixels represent the extracted strokes.

###  IV-A Adjusting skeletons to samples

The proposed method maximally exploits the skeletons of sample characters,
however, the skeletons of samples are not given. The skeletons can be easily
extracted if strokes are separated; however, strokes often overlap. Quality
skeleton extraction is essential for the proposed method. Therefore, inspired
by accurate segmentation applications, such as Grabcut [31], we adopt user
interaction to adjust skeletons in our dataset to the samples so that accurate
skeletons of the samples can be obtained.

We developed a graphical user interface (GUI) for adjusting skeletons, as
illustrated in Fig. 4. Skeletons are displayed with control points on the
sample. The operator adjusts a skeleton by dragging the control points and the
GUI shows these changes in real time.

|
---|---
(a) | (b)
Figure 4:  Skeleton adjustment GUI. (a) Screenshot of the GUI initiating an
adjustment. Note that the skeleton is not matched to the sample at this point
because the skeletons are in Mincho. (b) Adjustment result.

In order to assist the operator, we implemented three automation techniques:
scale adjustment, rotation adjustment, and cooperative move. Firstly, a scale
of skeletons is adjusted to fit the sample. We calculate a scaling factor from
rectangles around the skeleton and sample. The second technique is rotation
adjustment. We extract points from the skeleton by sampling. Likewise, we
extract points from the medial axis of the sample, which is obtained by a
morphological operation. With these points, we use iterative closest point
matching [32] to calculate the rotation matrix. Thirdly, control points move
cooperatively when the start point is moved by hand, e.g., if the start point
moves up, the other control points also move up. The scale and rotation
adjustments are performed only once, before manual adjustment.

We emphasize that skeleton adjustment is not difficult and can be completed in
minutes. It is unnecessary to adjust all of the skeletons in the dataset.
According to our experimental results, approximately 15 samples are sufficient
for the proposed method. Therefore, skeleton adjustment is significantly less
work than creating thousands of characters by hand.

###  IV-B Skeleton relation assignment

We determine the connectivity of the skeletons in the samples. We define
relations between two skeletons as follows:

  * •

continuous: the start or end of a skeleton is connected to the start or end of
another.

  * •

connecting: the start or end of a skeleton is connected to the body of
another.

  * •

connected: the body of a skeleton is connected to the start or end of another.

  * •

crossing: the skeleton crosses over another.

  * •

isolated: the skeleton is isolated from others.

The body is the part of a skeleton without start or end parts. Examples of
each relation are illustrated in Fig. 5.

Two skeletons are in contact when the distance between them is small.
Therefore, we measure the distance $d$ between $S$ and $S^{\prime}$ as

$d=\left\\{\begin{array}[]{ll}\mathop{\rm
min}\limits_{i,j}\|Q_{i,j}-P_{i}\|_{2}+\|Q_{i,j}-P^{\prime}_{j}\|_{2}&\mbox{if
$Q$ exists},\\\ \infty&\mbox{otherwise}.\end{array}\right.$ (2)

$Q_{i,j}$ is the point at which two lines cross, which are vertical to $S$ and
$S^{\prime}$ and through $P_{i}$ and $P^{\prime}_{j}$, respectively. Fig. 6
illustrates an example of $Q$. Using the $\bar{i}$ and $\bar{j}$ that minimize
$d$, we can determine which part of $S$ is in contact with $S^{\prime}$, e.g.,
the start of $S$ is in contact with the part of $S^{\prime}$ when $\bar{i}$ is
small and $\bar{j}$ is large. We use three functions to indicate contact:
$\mathbb{I}_{s}$ for the start, $\mathbb{I}_{e}$ for the end, and
$\mathbb{I}_{b}$ for the body. For $n$ points of a skeleton, the functions
calculate true or false, as follows. $\mathbb{I}_{s}(\bar{i})=\text{True}$ if
$\frac{\bar{i}}{n}<.05$, otherwise False.
$\mathbb{I}_{e}(\bar{i})=\text{True}$ if $\frac{\bar{i}}{n}>.95$, otherwise
False. $\mathbb{I}_{b}(\bar{i})=\text{True}$ if
$.05\leq\frac{\bar{i}}{n}\leq.95$, otherwise False. The parameters $.05$ and
$.95$ were determined experimentally. We assign a relation $R$ to $S$ against
$S^{\prime}$ as follows:

$R=\left\\{\begin{array}[]{ll}\mbox{continuous}&\mbox{if
}d<2(\tau+\tau^{\prime})\\\
&\quad\wedge\left(\mathbb{I}_{s}(\bar{i})\vee\mathbb{I}_{e}(\bar{i})\right)\\\
&\quad\wedge\left(\mathbb{I}_{s}(\bar{j})\vee\mathbb{I}_{e}(\bar{j})\right),\\\
\mbox{connecting}&\mbox{if }d<2(\tau+\tau^{\prime})\\\
&\quad\wedge\left(\mathbb{I}_{s}(\bar{i})\vee\mathbb{I}_{e}(\bar{i})\right)\wedge\mathbb{I}_{b}(\bar{j}),\\\
\mbox{connected}&\mbox{if }d<2(\tau+\tau^{\prime})\\\
&\quad\wedge\mathbb{I}_{b}(\bar{i})\wedge\left(\mathbb{I}_{s}(\bar{j})\vee\mathbb{I}_{e}(\bar{j})\right),\\\
\mbox{crossing}&\mbox{if }d<2(\tau+\tau^{\prime})\\\
&\quad\wedge\mathbb{I}_{b}(\bar{i})\wedge\mathbb{I}_{b}(\bar{j}),\\\
\mbox{isolated}&\mbox{otherwise},\end{array}\right.$ (3)

where $\tau$ represents a thickness of a stroke in a sample. We can draw a
character image with $\tau$. Fig. 7 illustrates the skeletons and created
character with various values of $\tau$. We empirically determined $\tau$ by
creating character images with various $\tau$ and choosing the $\tau$ that
minimizes the hamming distance between the created and sample character
images.

Figure 5:  Skeleton relations of the red skeletons to the black skeletons: (a) continuous, (b) connecting, (c) connected, (d) crossing, and (e) isolated.  Figure 6:  Measurement of the distance between two skeletons. The sampling points are drawn with black circles.  |
---|---
(a) | (b)
Figure 7: (a) Skeletons and (b) created characters with $\tau=1,10,20,40$.

We reconsider the relations obtained above. Since the relations are based on
the skeletons, it is necessary to verify that they are consistent with the
sample images. The relations may occasionally be different from the samples
because of human error during adjustment and the parameters. The parameter
$2(\tau+\tau^{\prime})$ in Eq. (3) is set to a severe value in order to detect
all stroke contacts without omission. Consequently, an incorrect relation may
be assigned.

The procedures for reconsideration are as follows. We classify all pixels of
the sample to the nearest skeleton within the Euclidean distance. Fig. 8 (a)
illustrates the segmentation results. Subsequently, we choose a connected
component from the sample using these results. Then, we determine the number
of connected components obtained. If the relation is isolated, there must be
two connected components. Likewise, if the relation is connecting, connected,
or crossing, there must be only one connected component. This check
complements Eq. (3).

###  IV-C Adaptive active contour model

We propose an adaptive active contour model (AACM) for character stroke
extraction. The AACM is able to distinguish even strokes that have contact
with others, determining the boundary of each stroke. Inspired by Snake [33],
we seek the boundary by optimizing a spline curve. In addition, we incorporate
constraints and adaptive energy modification into the AACM so that strokes can
be extracted from complex characters.

The AACM is defined as a spline curve that minimizes the energy function as

$E_{\text{\it AACM}}=E_{\text{\it int}}+E_{\text{\it img}},$ (4)

where $E_{\text{\it int}}$ and $E_{\text{\it img}}$ take into account the
smoothness of the curves and the fit to objects, respectively. $E_{\text{\it
int}}$ follows [33]. $E_{\text{\it img}}$ must be determined for each task. If
we set $E_{\text{\it img}}$ to $-|\triangle I|$, where $\triangle I$
represents the gradient image of $I$, $E_{\text{\it img}}$ pulls the AACMs to
the edges. For our purposes, it is unnecessary for a boundary to be very close
to its stroke, and instead, may be rather relaxed. If the boundary is too
close, it may cross the stroke. Consequently, the extracted strokes are
damaged. Therefore, in this paper, we define $E_{\text{\it img}}$ as

$E_{\text{\it img}}=G_{v}\ast I_{\text{\it SMP}}+I_{\text{\it SK}},$ (5)

where $G_{v}$ is a Gaussian kernel with variance $v$, and $I_{\text{\it SMP}}$
and $I_{\text{\it SK}}$ are the grayscale images777Pixel values are 0 if they
are background, and 255 otherwise. of the sample and its skeletons,
respectively. $E_{\text{\it img}}$ has a high energy888The pixel value
represents the energy: 0 is the lowest and 255 is the highest. around the
strokes, thus, the boundary remains separated from them. We use the boundaries
of the segmentation results as the initial AACMs. We adaptively modify
$E_{\text{\it img}}$ and set some points that the AACMs must pass through.

|
---|---
(a) | (b)
Figure 8:  Relation check. (a) Skeleton of segmentation results. (b) Sample of
the segmentation results. Focusing on the two segments indicated by the red
circle, the relation is isolated in (a), but there is only one connected
component in (b).

In the case where the relation of the target stroke is isolated, which is the
simplest case, we directly apply an AACM. Fig. 9 illustrates the results of
the extraction of an isolated stroke.

|
---|---
(a) | (b)
Figure 9:  Extraction of an isolated stroke. (a) Initial AACM (blue) and
minimized AACM (orange) on $E_{\text{\it img}}$. (b) Extracted stroke.

In the case where the relation of the target stroke is connecting or
continuous, we force an AACM to go through the point at which the strokes are
in contact. Then, we decrease $E_{\text{\it img}}$ by $1/\beta_{1}$ within the
range of the length $\beta_{2}\tau$ from the point, except for $I_{\text{\it
SK}}$. This decrease operation encourages the AACM to pass through the stroke
intersection. $\beta_{1}$ and $\beta_{2}$ are constant values that were
determined empirically. With the modified $E_{\text{\it img}}$ and the
constraints, we minimize the AACM and extract the target. Fig. 10 demonstrates
the extraction for this case. The AACM naturally passes through the
intersection due to the decreased energy.

|
---|---
(a) | (b)
Figure 10:  Stroke extraction for connecting or continuous stroke relations.
(a) Initial AACM (blue) and minimized AACM (orange) on $E_{\text{\it img}}$.
(b) Extracted stroke.

Where the relation of the target stroke is crossing or connected, we decrease
$E_{\text{\it img}}$ by $1/\beta_{1}$ within the range of
$\beta_{2}\tau_{\text{\it others}}$ from the other strokes. Then, we minimize
the AACM with the modified $E_{\text{\it img}}$ and extract the target along
the minimized AACM.

###  IV-D Stroke restoration

We reshape extracted strokes, which often have defects at points where contact
occurs; see Fig. 11(a). An intuitive method for restoration is the removal of
contours around defects and the connection with cubic Bézier curves. A cubic
Bézier curve is drawn with four points $(P_{1},P_{2},P_{3},P_{4})$. The curve
starts from $P_{1}$ and ends at $P_{4}$. $P_{2}$ and $P_{3}$ serve as control
points that provide a direction to the curve.

We apply cubic Bézier curves to a stroke using POTRACE [34], generating a
number of curves that represent the short parts of the contour. Then, we
remove the curves within a distance of $\tau$ from the extracted stroke; see
Fig. 11(c). We create a cubic Bézier curve between two curves so that the
contour can be continuous; see Fig. 11(d). We set $P_{1}$ and $P_{4}$ of the
new curve to $P_{4}$ and $P_{1}$, respectively, of the remaining curves. Fig.
12 illustrates the new curve. We calculate $P_{2}$ and $P_{3}$ by

$\displaystyle P_{2}$ $\displaystyle=$ $\displaystyle
P_{1}+\gamma(P_{1}-P^{\prime}_{1}),$ (6) $\displaystyle P_{3}$
$\displaystyle=$ $\displaystyle P_{4}+\gamma(P_{4}-P^{\prime}_{4}),$ (7)

where $\gamma$ is a constant value. $P^{\prime}_{1}$ and $P^{\prime}_{4}$
represent the nearest control points from $P_{1}$ and $P_{4}$ in the remaining
curves, respectively. $P_{1}-P^{\prime}_{1}$ and $P_{4}-P^{\prime}_{4}$
represent local gradients. Hence, $P_{2}$ and $P_{3}$ are the points moved
along the local gradients from $P_{1}$ and $P_{4}$, resulting in a smooth
curve.

|
---|---
(a) | (b)
|
(c) | (d)
Figure 11:  Restoration of an extracted stroke. (a) Damaged stroke. (b)
Restored stroke. (c) Contours removed with cubic Bézier curves; red rectangles
represent $P_{1}$ and $P_{4}$, and pink circles represent $P_{2}$ and $P_{3}$.
Cyan lines represent contours. (d) The created curves, indicated by red lines.
Figure 12:  Points of a new Bézier curve. The red curve represents a new curve
and the green curves represent those remaining.

##  V Stroke deployment

We generate characters by deploying strokes on skeletons. Our character
generation method consists of two phases: skeleton modification and stroke
deployment.

###  V-A Skeleton modification

The results of character generation would be strange—even with perfect
strokes— if the style of the skeleton dataset were different from the target
font. Thus, we modify the skeletons to be similar to the target font. A
feasible method for modification is the use of a transformation matrix. We
estimate the transformation matrix from the skeletons of the samples.
Specifically, we seek two transformation matrices: $T_{\text{\it sz}}$ and
$T_{\text{\it aff}}$. $T_{\text{\it sz}}$ adjusts the size of the skeletons
and centroid translation, and $T_{\text{\it aff}}$ adjusts the affine
transformation of the skeletons. Modification is carried out by applying
$T_{\text{\it sz}}$ and $T_{\text{\it aff}}$ to the dataset.

We estimate $T_{\text{\it sz}}$ from the rectangles of the skeletons. Let $H$
and $W$ denote the height and width of a character in the dataset,
respectively. Likewise, let $\hat{H}$ and $\hat{W}$ be the height and width of
the skeleton of a sample character, respectively. With output image size
$I_{w}$ and $I_{h}$, we calculate $T_{\text{\it sz}}$ by averaging size
transformation over the sample characters $\bm{C}=\\{\cdots,c_{i},\cdots\\}$
as:

$T_{\text{\it
sz}}=\frac{1}{\left|{\bm{C}}\right|}\sum_{c_{i}\in\bm{C}}\left(\begin{array}[]{ccc}\frac{\hat{W}_{c_{i}}}{W_{c_{i}}}&0&I_{w}\frac{\hat{W}_{c_{i}}}{2W_{c_{i}}}\\\
0&\frac{\hat{H}_{c_{i}}}{H_{c_{i}}}&I_{h}\frac{\hat{H_{c_{i}}}}{2H_{c_{i}}}\\\
0&0&1\end{array}\right).$ (8)

We estimate $T_{\text{\it\it aff}}$ using the skeletons of the sample and
dataset. However, it is difficult to estimate $T_{\text{\it aff}}$ directly
from the skeletons because of the complex structure of the characters. If the
characters are complex, they have many skeletons. The affine transformations
of skeletons cancel each other. Eventually, $T_{\text{\it aff}}$ becomes a
trivial matrix, such as an identity matrix. To avoid these problems, we divide
the characters into groups and calculate $T_{\text{\it aff}}$ by averaging the
affine transformation matrices obtained from each group. We divide a character
using the relations of skeletons. Groups consist of skeletons whose relations
are continuous, connecting, connected, and crossing. Fig. 13 illustrates the
grouping results.

|  |  |
---|---|---|---
(a) | (b) | (c) | (d)
Figure 13:  Groups of skeletons: (a) skeleton and (b), (c), and (d) groups.

We use function $f_{T}$ to calculate the affine transformation matrix, which
fits stroke $S$ to stroke $S^{\prime}$. We formulate $f_{T}$ based on the
least-squares method, as Eq. (9), which can be solved analytically as Eq.
(10).

$\displaystyle f_{T}(S,S^{\prime})$ $\displaystyle=$ $\displaystyle\mathop{\rm
arg~{}min}\limits_{T\in\bm{T}}\|S-TS^{\prime}\|_{2},$ (9) $\displaystyle=$
$\displaystyle(S^{\top}S)^{-1}S^{\top}S^{\prime},$ (10)

where $\bm{T}$ represents a set of all possible affine transformation
matrices. We calculate $T_{\text{\it aff}}$ by averaging affine transformation
matrices from $S$ to $\hat{S}$ as

$T_{\text{\it
aff}}=\frac{1}{n_{\text{strokes}}}\sum_{i\in\bm{C}}\sum_{k=1}^{n^{i}_{\text{groups}}}\sum_{j\in
G^{i}_{k}}f_{T}(S^{i}_{j},\hat{S}^{i}_{j}),$ (11)

where $n_{\text{strokes}}$ is the number of total strokes in $\bm{C}$. The
number of groups in character $i$ is $n^{i}_{\text{groups}}$. $G^{i}_{k}$ is a
set of stroke indices in group $k$ of character $i$, such as:
$G^{i}_{k}=\\{j\mid S^{i}_{j}\in\text{group $k$}\\}$.

Finally, we modify the skeletons by applying $T_{\text{\it sz}}$ and
$T_{\text{\it aff}}$. A transformed stroke is obtained using the equation

$\tilde{S}=T_{\text{\it aff}}T_{\text{\it sz}}S.$ (12)

###  V-B Stroke deployment

Here, we describe the framework for selecting a stroke that is the most
suitable for a target skeleton $\tilde{S}$. Stroke deployment has an impact on
the appearance of the generated characters; therefore, the appropriate stroke
must be chosen for each target skeleton.

First, we select $\hat{S}$, which fits to $\tilde{S}$, from a set of the
extracted strokes $\hat{\bm{S}}$. Then, we determine the stroke
$\hat{\mathcal{I}}$ corresponding to the selected $\hat{S}$.

$\displaystyle\hat{\mathcal{I}}$ $\displaystyle=$ $\displaystyle f_{\text{\it
img}}(\hat{S}),$ (13) $\displaystyle\hat{S}$ $\displaystyle=$
$\displaystyle\mathop{\rm
arg~{}min}\limits_{\hat{S}_{i}\in\hat{\bm{S}}}E(\hat{S}_{i},\tilde{S}),$ (14)

where $f_{\text{\it img}}$ is a function that gives a stroke corresponding to
$\hat{S}$ and $E$ is an energy function associated with skeletons $\hat{S}$
and $\tilde{S}$. Once $\hat{\mathcal{I}}$ is determined, we apply
$f_{T}(\hat{S},\tilde{S})$ to $\hat{\mathcal{I}}$ and deploy the transformed
$\hat{\mathcal{I}}$ on $\tilde{S}$. We repeat the selection until strokes for
all target skeletons are determined. Finally, a character is generated by
integrating the selected strokes.

We define $E$ as

$\displaystyle
E(\hat{S},\tilde{S})=E_{s}(\hat{S},\tilde{S})+E_{s}(f_{\text{\it
data}}(\hat{S}),f_{\text{\it data}}(\tilde{S}))+$ $\displaystyle
E_{a}(f_{\text{\it data}}(\hat{S}),f_{\text{\it data}}(\tilde{S})),$ (15)

where $f_{\text{\it data}}(S)$ gives the skeleton corresponding to $S$ in the
skeleton dataset. The energy $E$ utilizes three terms in inspecting $\hat{S}$.
The first term measures the distance between skeletons in a sample character
and a target character. When this term is small, two skeletons are similar.
Therefore, $\hat{\mathcal{I}}$ will naturally fit to the target skeleton. The
second term also measures the distance between two strokes, but only using the
dataset. If the skeletons in the dataset are similar, the stroke is favorable
for the target skeleton. This term improves the accuracy of the energy
function. The third term measures distance using adjacent skeletons, making
the distance more global than when focusing on only two skeletons.

The energy $E_{s}$ measures the distance between two skeletons. We define
$E_{s}(S,S^{\prime})$ as

$\displaystyle E_{s}(S,S^{\prime})$ $\displaystyle=$
$\displaystyle\|S^{\prime}-f_{T}(S,S^{\prime})S\|_{1}$ (16)
$\displaystyle+\|S-f_{T}(S^{\prime},S)S^{\prime}\|_{1}$
$\displaystyle+\sum_{k=1}^{3}\mathbb{I}_{k}(S,S^{\prime}),$

where $\mathbb{I}_{k}$ is $1$ if the three attributes of two skeletons—line
type, start shape, and end shape—are different; otherwise, it is $0$.
$\mathbb{I}_{k}$ inspects the line type, start shape, and end shape of the
stroke when $k=1$, $2$, and $3$, respectively. The first term of $E_{s}$
measures the distance between two skeletons. The second term of $E_{s}$ takes
into account the distortion from the transformation $f_{T}(S,S^{\prime})$; a
less distorted skeleton is favored. The third term of $E_{s}$ incorporates
attribute differences. We define $E_{a}$ as

$E_{a}(S,S^{\prime})=\begin{cases}\|S_{\text{\it st}}-S^{\prime}_{\text{\it
st}}\|_{1}+\|S_{\text{\it ed}}-S^{\prime}_{\text{\it ed}}\|_{1}&\text{if
}\bm{S}_{a}\text{exists}\\\ 50&\text{otherwise},\end{cases}$ (17)

where $\bm{S}_{a}=\\{S_{\text{\it st}},S_{\text{\it ed}},S^{\prime}_{\text{\it
st}},S^{\prime}_{\text{\it ed}}\\}$, and $S_{\text{\it st}}$ represent a
skeleton connected to the start point of $S$. In the case where there are
several skeletons connected to $S$ at the start point, we choose one skeleton
whose center is closest to $S$. Likewise, $S_{\text{\it ed}}$ represents a
skeleton connected to the end point of $S$. If $S_{\text{\it st}}$,
$S_{\text{\it ed}}$, $S^{\prime}_{\text{\it st}}$, or $S^{\prime}_{\text{\it
ed}}$ is unfavorable, $E_{a}$ has a large value.

##  VI Sample selection

The proposed method uses $\bm{C}$, a font subset, to generate a large number
of characters. The generation results are deeply influenced by $\bm{C}$.
Therefore, it is important to analyze which characters are suitable elements
for $\bm{C}$. With an optimal $\bm{C}$, we are able to maximize the capability
of the proposed method. We define a process of seeking the optimal $\bm{C}$ as
sample selection.

In sample selection, we use the skeleton dataset to seek $\bm{C}$. Note that
sample images are not used. Since the skeletons in the dataset are fundamental
data, if the skeletons in $\bm{C}$ in the dataset are suitable, $\bm{C}$ in a
target font can be expected to work well.

In order to ensure sample selection feasibility, we seek $\bm{C}$ in a subset
of the dataset that we define as validation characters. In this study, we
adopt the 1,006 characters of kyoiku-kanji containing the elemental characters
of Japanese. Since the number of characters in the dataset exceeds 210,000, it
is time-consuming to use the entire dataset. As we show in experimental
results in Section VII, the proposed method is able to generate acceptable
results with a selected $\bm{C}$, verifying the effectiveness of our approach.

Sample selection is based on a genetic algorithm. There are a large number of
candidates of optimal $\bm{C}$ even with the subset. Candidates undergo
crossover, evaluation and selection processes and an optimal candidate can be
obtained over many iterations of these processes.

We define function $f_{\text{\it selection}}$ that calculates the energy of a
candidate $\bm{C}_{i}$ using a set of skeletons $\bm{S}_{i}$ of $\bm{C}_{i}$.
Let $\bm{S}_{v}$ represent a set of skeletons of validation characters.

$\displaystyle f_{\text{\it selection}}(\bm{S}_{i})$ $\displaystyle=$
$\displaystyle\alpha f_{e}(\bm{S}_{i})+(1-\alpha)f_{r}(\bm{S}_{i}),$ (18)
$\displaystyle f_{e}(\bm{S}_{i})$ $\displaystyle=$
$\displaystyle\sum_{s_{v}\in\bm{S}_{v}}\left(\min_{s\in\bm{S}_{i}}E_{s}(s,s_{v})+E_{a}(s,s_{v})\right),$
(19) $\displaystyle f_{r}(\bm{S}_{i})$ $\displaystyle=$
$\displaystyle|\bm{S}_{i}|+N_{\text{\it cross}}(\bm{S}_{i})+N_{\text{\it
con}}(\bm{S}_{i}),|N|$ (20)

where $|\bm{S}|$ represents the cardinarity. $N_{\text{\it cross}}$ gives the
number of skeletons whose relations are crossing. $N_{\text{\it con}}$ gives
the number of skeletons whose relations are continuous, connecting, and
connected. $f_{r}$ measures the complexity of $\bm{C}$ and serves as a
regularization term. We control the complexity of the samples with a constant
value $\alpha$.

##  VII Experimental results

We demonstrate character generation with the proposed method using kanji. We
used five fonts as the target fonts: Ibara 43), Gcomic 44), Onryo 45), Tsunoda
46), and Zinpen 47), where the indices represent the numbers in Appendix. Fig.
14 shows the original characters in the five fonts.

Figure 14: The Ibara, Gcomic, Onryo, Tsunoda, and Zinpen fonts (from top to
bottom).

The characters in each font have distinctive strokes; for instance, two
parallel lines are used in Zinpen. It is very difficult for existing methods
to generate characters in these characteristic fonts.

We created sample images by drawing characters in a 360-pt font size using the
Qt library999http://www.qt.io/. The size of the images is 500 by 500 pixels.
We implemented the proposed method with C++ and experiments were carried out
on a Windows machine with a dual-core CPU. Fifteen characters were chosen as
the samples in each font. The coefficients were set as follows:
$\beta_{1}=2.0$ and $\beta_{2}=1.5$. We generated 2,965 characters, which
comprise the set of JIS level-1.

For comparison, we used the existing method [28] based on patch transformation
developed by Saito et al. To the best of our knowledge, [28] is the only
method that is applicable to characters in various fonts. [28] divides samples
into grid squares and generates characters by deploying the squares. The
frameworks of the proposed and [28] are similar. Both exploit samples, extract
components of characters, and generate characters. However, the extracted
components are significantly different. The components in [28] are small
pieces of characters that lack meaning, whereas the components in the proposed
method are complete strokes.

We have employed our method for sample selection, the results of which are
summarized in Table II. We extracted strokes of the validation characters from
the 1,006 characters from kyoiku-kanji that are for elementary school
students, as determined by the Japanese Ministry of Education. The initial
candidates are randomly generated. We fixed the number of elements in a
candidate to 15 characters. In each iteration of the algorithm, 20 candidates
survive as good candidates and 150 new candidates are created. The maximum
number of iterations is set to 1,000. We varied $\alpha$ and carried out
sample selection. The minimum $f_{\text{\it selection}}$ and $f_{r}$ are
listed in Table II. The obtained samples are complex characters at high values
of $\alpha$ and simpler at low $\alpha$ values. We use samples at $\alpha=0.6$
as the sample characters in the following experiments.

TABLE II: Sample selection results. $\alpha$ | $f_{\text{\it selection}}$ | $f_{r}$ | samples
---|---|---|---
0.8 | 0.017 | 0.673 |
0.6 | 0.027 | 0.447 |
0.4 | 0.030 | 0.387 |

###  VII-A Character generation results

We generated characters in the five fonts using the proposed and existing
methods, which are shown in Fig. 15. We generated 2,965 characters in each
font, though only 75 characters are shown due to space limitations. Almost all
characters generated by the proposed method appear clean and have good
readability. Moreover, the font characteristics, such as the slant of the
characters in Ibara, are successfully reconstructed. It is not easy to
distinguish the original characters from those generated by the proposed
method at a glance.  In addition, we describe the complexity of the strokes of
the generated characters: 1 (min), 29 (max), 12.5 (avg), and 4.5 (std).

---
(a)
(b)
(c)
(d)
(e)
Figure 15:  The characters generated in five fonts: (a) Ibara, (b) Gcomic, (c)
Onryo, (d) Tsunoda, and (e) Zinpen. From top to bottom in each subfigure,
results are shown for the original characters, the proposed method, and the
method of Saito et al. [28].

###  VII-B Similarity evaluation of the images

We evaluated the generated characters as images. We compared the generated
characters with the original images with the Chamfer distance[35]. Let
$d_{\text{\it cham}}(A,B)$ be the Chamfer distance from image $A$ to image
$B$. The distance is defined as $d_{\text{\it
cham}}(A,B)=\sum_{p\in\hat{A}}\min_{q\in\hat{B}}|p-q|$, where $\hat{A}$ and
$\hat{B}$ are the sets of edge points of $A$ and $B$, respectively. We
obtained $\hat{A}$ and $\hat{B}$ by applying a Canny edge detector [36]. In
general, $d_{\text{\it cham}}(A,B)$ is not equal to $d_{\text{\it
cham}}(B,A)$; therefore, the symmetric formulation is often used

$d_{\text{\it cham}}^{\prime}(A,B)=d_{\text{\it cham}}(A,B)+d_{\text{\it
cham}}(B,A).$ (21)

We employed $d_{\text{\it cham}}^{\prime}$ in this study.

A subset of the generated characters is used for this evaluation.
Specifically, we used the 1,006 characters mentioned above. We resized the
images of the original and generated characters to 100 by 100 pixels and
calculated the Chamfer distance between the generated characters and the
originals. Table III summarizes the average Chamfer distances.

TABLE III: Average Chamfer distance | Ibara | Gcomic | Onryo | Tsunoda | Zinpen
---|---|---|---|---|---
Saito [28] | 4.7 | 4.2 | 4.6 | 6.8 | 3.8
Proposed | 4.1 | 3.7 | 4.2 | 4.1 | 3.2

The proposed method is closer for all five fonts than the existing method. In
particular, the proposed method is greatly superior to the existing method for
the Tsunoda font. Tsunoda has a distinctive skeleton, as can be seen in the
spaces between strokes. Since the proposed method successfully modified the
skeleton for Tsunoda, the generated characters are close to the originals.
These results numerically demonstrate the effectiveness of the proposed
method.

###  VII-C Evaluation using character recognition

We evaluated the generated characters by using them as training data for a
character recognition system. The test data are character images of kyoiku-
kanji in each font. Therefore, the number of classes is 1,006. A simple
recognition method with Chamfer distance is used in this study. We calculated
the Chamfer distance between the training and test data and classified the
test data as the nearest character. Test data are created as images, including
one JPEG-compressed character. We created test data by drawing characters
using a 60-pt font size on 100 by 100 pixel images; the background is white,
and the foreground is black. Table IV summarizes the results.

TABLE IV: Recognition results (%) | Ibara | Gcomic | Onryo | Tsunoda | Zinpen
---|---|---|---|---|---
Saito [28] | 29.0 | 61.5 | 56.5 | 7.4 | 68.8
Proposed | 48.3 | 77.4 | 67.7 | 28.6 | 77.4

The proposed method achieved higher performance than the existing method for
all fonts.

###  VII-D Subjective evaluation

We present the results of two subjective evaluations based on the similarity
of each font to the original characters and the appearance of the generated
characters. Both subjective evaluations were carried out by 14 participants.

The first subjective evaluation is of the similarity between the generated and
original characters. At the beginning of the first subjective evaluation, we
showed the original characters to the participants. Then, 10 idioms consisting
of four characters were displayed. The idioms were made from original
characters, characters generated by the proposed method, or characters
generated by the existing method. We use the mean opinion score (MOS) over the
results. The participants assigned scores ranging from 1 (bad) to 5
(excellent) to the idioms according to their impression of their similarity.
Table V summarizes the results. The proposed method receives higher MOSs than
the existing method. Moreover, the MOSs of the proposed method are relatively
close to the originals.

TABLE V: Results of a subjective evaluation of font similarity to original characters | Ibara | Gcomic | Onryo | Tsunoda | Zinpen
---|---|---|---|---|---
Originals | 4.5 | 4.8 | 4.5 | 4.9 | 4.9
Saito [28] | 1.1 | 1.3 | 1.5 | 1.1 | 1.3
Proposed | 4.3 | 4.4 | 4.6 | 4.3 | 4.6

The second subjective evaluation is of appearance. We asked the participants
to select characters that may have been generated by computers. The test data
consisted of 150 characters from each font, i.e., 50 characters each from the
original characters, those generated by the existing method, and those
generated by the proposed method. Therefore, the test data consist of a total
of 750 characters. Fig. 16 illustrates the test data shown to the
participants.

---
(a)
(b)
Figure 16:  Examples of test data and answers. (a) Test data shown to the
participants. (b) Answers to the test data. Generated characters are gray.

We counted the number of participants who believed the characters to have been
generated artificially. Then, we calculated the averages and normalized the
numbers. Table VI summarizes the results. A character’s appearance is natural
if the value is low and unnatural if it is high. Natural means that the
characters are likely handmade, and unnatural means artificial. Most of
characters generated by the existing method are classified as generated
characters since the results are in the range of 0.77–0.99. The appearance of
the characters generated by the method in [28] is far from handmade. In
contrast, the results for the proposed method are much lower than those for
[28]. In particular, the characters generated by the proposed method for Ibara
and Onryo are close to the original characters. According to the results, it
is difficult to distinguish the original characters and those generated by the
proposed method, even by a human. Therefore, the proposed method successfully
generated characters with a good appearance.

TABLE VI: Results of the subjective evaluation of appearance | Ibara | Gcomic | Onryo | Tsunoda | Zinpen
---|---|---|---|---|---
Originals | 0.06 | 0.02 | 0.04 | 0.01 | 0.01
Saito [28] | 0.96 | 0.9 | 0.86 | 0.98 | 0.94
Proposed | 0.11 | 0.09 | 0.07 | 0.13 | 0.08

###  VII-E Varied font generation

In order to demonstrate the font generation capability of the proposed method,
we performed a generation experiment with 42 fonts: four standard fonts used
in Windows (Gothic, Meiryo, Mincho, and YuMincho), six calligraphy handwriting
fonts 5) – 10), 17 pen handwriting fonts 11) – 24), and 15 artificial fonts
28) – 42). The fonts are varied, and include Mincho, Gothic, clerical,
antique, personal handwriting, professional and handwriting styles. It is
worth noting that the number of fonts used in most existing methods is small;
in particular, we used a significantly large number of handwritten fonts, 23.
In this experiment, we generated 2,965 characters. We illustrate examples of
the results in Fig. 17. The results are promising. The proposed method
generated clean characters and the style of each font is reproduced.

|
---|---
(a) Proposed | (b) Original
Figure 17:  Generation results of the proposed method and the original
characters in various fonts. Each row shows one font.

##  VIII Conclusion

This paper focused on the problem of generating characters for a typographic
font by exploiting a subset of a font and a skeleton dataset. The proposed
method extracts character strokes and constructs characters by selecting and
deploying the strokes to the skeleton. The proposed method successfully
extracts the natural strokes from sample character images. It is worthwhile to
emphasize the importance of stroke extraction from the small number of
character images. The proposed method only requires a font subset—as small as
15 samples—which is a feasible number of samples to collect. With such a small
subset, the proposed method can generate thousands of characters. Sample
collection is further eased because this method can extract characters from
image formats.

An experimental evaluation was conducted with five characteristic fonts that
are difficult to generate with existing methods. We evaluated the generated
characters by objective and subjective evaluations; all results indicated that
the characters generated by our proposed method have a comparable appearance,
usefulness, and readability to the original characters. Furthermore, we
carried out the experiments with subsets of 42 fonts to demonstrate the
generative capability of the proposed method. In our future work, we will
attempt to automatically adjust skeleton samples and conduct experiments with
a larger number and greater variation of fonts.

[List of fonts]

  1. 1.

MS Gothic

  2. 2.

MS Meiryo

  3. 3.

MS Mincho

  4. 4.

MS YuMincho

  5. 5.

Aoyagi kouzan

  6. 6.

Aoyagi reisho

  7. 7.

Eishi kaisho

  8. 8.

aiharahude

  9. 9.

jetblack

  10. 10.

riitf

  11. 11.

azukifont

  12. 12.

chigfont

  13. 13.

gyate

  14. 14.

hosofuwafont

  15. 15.

KajudenB

  16. 16.

KajudenR

  17. 17.

kiloji

  18. 18.

KTEGAKI

  19. 19.

mitimasu

  20. 20.

seifuu

  21. 21.

setofont

  22. 22.

SNsanafon

  23. 23.

TAKUMISFONT-B

  24. 24.

uzurafont

  25. 25.

apjapanesefonth

  26. 26.

KFhimaji

  27. 27.

ruriiro

  28. 28.

AMEMUCHIGOTHIC

  29. 29.

antique

  30. 30.

chogokubosogothic

  31. 31.

dejima-mincho

  32. 32.

GenEiAntique

  33. 33.

hakidame

  34. 34.

IPAexg

  35. 35.

IPAexm

  36. 36.

jk-go-l

  37. 37.

Kazesawa

  38. 38.

migu-1c

  39. 39.

mofuji

  40. 40.

Nasu

  41. 41.

Smart

  42. 42.

yutapon-coding

  43. 43.

Ibara

  44. 44.

Gcomic

  45. 45.

Onryo

  46. 46.

Tsunoda

  47. 47.

Zinpen

## References

  * [1] S. Bayoudh, H. Mouchère, L. Miclet, and E. Anquetil, “Learning a classifier with very few examples: analogy based and knowledge based generation of new examples for character recognition,” in _Machine Learning: ECML 2007_. Springer, 2007, pp. 527–534.
  * [2] M. Djioua and R. Plamondon, “An interactive system for the automatic generation of huge handwriting databases from a few specimens,” in _Pattern Recognition, 19th International Conference on_. IEEE, 2008, pp. 1–4.
  * [3] Y. Wada, H. Kasuga, and K. Sumita, “An evolutionary approach for the generation of diversiform characters using a handwriting model,” in _Pattern Recognition, 16th International Conference on_ , vol. 3, 2002, pp. 131–134.
  * [4] Y. Zheng and D. Doermann, “Handwriting matching and its application to handwriting synthesis,” in _Document Analysis and Recognition, Eighth International Conference on_. IEEE, 2005, pp. 861–865.
  * [5] B. Bai, K.-W. Wong, and Y. Zhang, “An efficient physically-based model for chinese brush,” in _Frontiers in Algorithmics_. Springer, 2007, pp. 261–270.
  * [6] K. W. Kwok, S. M. Wong, K. W. Lo, and Y. Yam, “Genetic algorithm-based brush stroke generation for replication of chinese calligraphic character,” in _2006 IEEE International Conference on Evolutionary Computation_ , 2006, pp. 1057–1064.
  * [7] D. E. Knuth, _The METAFONTbook_. Addison-Wesley Longman Publishing Co., Inc., 1986.
  * [8] E. J. Jakubiak, R. N. Perry, and S. F. Frisken, “An improved representation for stroke-based fonts,” in _Proc. of ACM SIGGRAPH_ , 2006.
  * [9] C. Hu and R. D. Hersch, “Parameterizable fonts based on shape components,” _Computer Graphics and Applications_ , vol. 21, no. 3, pp. 70–85, 2001.
  * [10] T. Hassan, C. Hu, and R. D. Hersch, “Next generation typeface representations: revisiting parametric fonts,” in _Proceedings of the 10th ACM symposium on Document engineering_. ACM, 2010, pp. 181–184.
  * [11] T. Tanaka, H. Iwasaki, K. Nagahashi, and E. Wada, “Making kanji skeleton fonts through compositing parts,” _Transactions of Information Processing Society of Japan_ , vol. 36, no. 9, pp. 2122–2131, 1995, in Japanese.
  * [12] K. Kamichi, “Kage — an automatic glyph generating engine for large character code set,” in _Proceedings of the glyph and typesetting workshop_ , 2004, pp. 85–92, in Japanese.
  * [13] S. Xu, F. Lau, W. K. Cheung, and Y. Pan, “Automatic generation of artistic chinese calligraphy,” _Intelligent Systems, IEEE_ , vol. 20, no. 3, pp. 32–39, 2005.
  * [14] S. Xu, H. Jiang, T. Jin, F. C. M. Lau, and Y. Pan, “Automatic generation of chinese calligraphic writings with style imitation,” _IEEE intelligent systems_ , vol. 24, no. 2, pp. 44–53, 2009.
  * [15] H. Choi, S.-J. Cho, and J. H. Kim, “Generation of handwritten characters with bayesian network based on-line handwriting recognizers,” in _Document Analysis and Recognition. Seventh International Conference on_ , 2003, pp. 995–999.
  * [16] R. Suveeranont and T. Igarashi, “Example-based automatic font generation,” in _Smart Graphics_. Springer, 2010, pp. 127–138.
  * [17] N. D. F. Campbell and J. Kautz, “Learning a manifold of fonts,” _ACM SIGGRAPH_ , vol. 33, no. 4, 2014.
  * [18] J. chao Feng, C. K. Tse, and Y. Qiu, “Wavelet-transform-based strategy for generating new chinese fonts,” in _Circuits and Systems, Proceedings of the International Symposium on_ , vol. 4, 2003, pp. IV–337–IV–340 vol.4.
  * [19] J. B. Tenenbaum and W. T. Freeman, “Separating style and content with bilinear models,” _Neural Comput._ , vol. 12, no. 6, pp. 1247–1283, 2000.
  * [20] J. W. Lin, C. Y. Hong, R. I. Chang, Y. C. Wang, S. Y. Lin, and J. M. Ho, “Complete font generation of chinese characters in personal handwriting style,” in _IEEE 34th International Performance Computing and Communications Conference_ , 2015, pp. 1–5.
  * [21] A. Zong and Y. Zhu, “Strokebank: Automating personalized chinese handwriting generation,” in _Proceedings of the Twenty-Sixth Annual Conference on Innovative Applications of Artificial Intelligence_ , 2014, pp. 3024–3029.
  * [22] Y. Wang, H. Wang, C. Pan, and L. Fang, “Style preserving chinese character synthesis based on hierarchical representation of character,” in _IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2008, pp. 1097–1100.
  * [23] B. Zhou, W. Wang, and Z. Chen, “Easy generation of personal chinese handwritten fonts,” in _IEEE International Conference on Multimedia and Expo_ , 2011, pp. 1–6.
  * [24] C. Shi, J. Xiao, W. Jia, and C. Xu, “Automatic generation of chinese character based on human vision and prior knowledge of calligraphy,” in _Natural Language Processing and Chinese Computing_. Springer, 2012, pp. 23–33.
  * [25] P. Liu, S. Xu, and S. Lin, “Automatic generation of personalized chinese handwriting characters,” in _Digital Home, Fourth International Conference on_ , 2012, pp. 109–116.
  * [26] X. Qiu, W. Jia, and H. Li, “A font style learning and transferring method based on strokes and structure of chinese characters,” in _Computer Science Service System, International Conference on_ , 2012, pp. 1836–1839.
  * [27] S. Ju and J. Shin, “Cursive style korean handwriting synthesis based on shape analysis with minimized input data,” in _High Performance Computing and Communications 2013 IEEE International Conference on Embedded and Ubiquitous Computing_ , 2013, pp. 2231–2236.
  * [28] H. Saito, Y. Sugaya, S. Omachi, S. Uchida, M. Iwamura, and K. Kise, “Generation of character patterns from sample character images,” in _IEICE Technical Report. PRMU_ , vol. 110, no. 467, 2011, pp. 293–298, in Japanese.
  * [29] T. S. Cho, M. Butman, S. Avidan, and W. T. Freeman, “The patch transform and its applications to image editing,” in _CVPR 2008_. IEEE, 2008, pp. 1–8.
  * [30] “Glyphwiki,” http://glyphwiki.org/, in Japanese.
  * [31] C. Rother, V. Kolmogorov, and A. Blake, “”grabcut”: Interactive foreground extraction using iterated graph cuts,” _ACM Trans. Graph._ , vol. 23, no. 3, pp. 309–314, 2004.
  * [32] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _Conference on Computer Vision and Pattern Recognition_ , 2012.
  * [33] M. Kass, A. Witkin, and D. Terzopoulos, “Snakes: Active contour models,” _Int. journal of computer vision_ , vol. 1, no. 4, pp. 321–331, 1988.
  * [34] P. Selinger, “Potrace: a polygon-based tracing algorithm,” http://potrace.sourceforge.net/potrace.pdf, 2003.
  * [35] G. Borgefors, “Hierarchical chamfer matching: A parametric edge matching algorithm,” _Pattern Analysis and Machine Intelligence, IEEE Transactions on_ , vol. 10, no. 6, pp. 849–865, 1988.
  * [36] J. Canny, “A computational approach to edge detection,” _Pattern Analysis and Machine Intelligence, IEEE Transactions on_ , no. 6, pp. 679–698, 1986.