# Detection of Bangla Fake News using MNB and SVM Classifier

Md Gulzar Hussain1, Md Rashidul Hasan2, Mahmuda Rahman3, Joy Protim4 and Sakib
Al Hasan5 Department of CSE, Green University of Bangladesh, Bangladesh1,2,4
Department of CSE, Jahangirnagar University, Dhaka, Bangladesh3
Department of CSE, Huzhou Normal University, Huzhou, China5
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Fake news has been coming into sight in significant numbers for numerous
business and political reasons and has become frequent in the online world.
People can get contaminated easily by these fake news for its fabricated words
which have enormous effects on the offline community. Thus, interest in
research in this area has risen. Significant research has been conducted on
the detection of fake news from English texts and other languages but a few in
Bangla Language. Our work reflects the experimental analysis on the detection
of Bangla fake news from social media as this field still requires much focus.
In this research work, we have used two supervised machine learning
algorithms, Multinomial Naive Bayes (MNB) and Support Vector Machine (SVM)
classifiers to detect Bangla fake news with CountVectorizer and Term Frequency
- Inverse Document Frequency Vectorizer as feature extraction. Our proposed
framework detects fake news depending on the polarity of the corresponding
article. Finally, our analysis shows SVM with the linear kernel with an
accuracy of 96.64% outperform MNB with an accuracy of 93.32%.

###### Keywords:

Fake News, Bangla Fake News, Text Classification, Bangla News, Natural
Language Processing.

##  I Introduction

Fake news is an oxymoron that undermines the credibility of reporting that
actually meets the verifiability standard and the public interest-i.e. real
news. And in this electronic era, it is one of the biggest challenges to
control the spreading of false or misleading news due to the free flow of
information through social networking sites such as Facebook, Twitter,
YouTube, micro-blogging and others. For instance, a new report says, nearly
one in three citizens across the United States, Spain, Germany, United
Kingdom, Argentina, and South Korea claim they have seen false or misleading
information on social media related to COVID-19 [1]. Because of the abundant
spreading of false and unreliable information, some commentators are now
referring to the latest wave of misinformation that’s accompanied by the
COVID-19 pandemic as a disinfodemic [2].

Since the information shapes the way we perceive the universe, therefore, fake
news and other misleading facts can take on different aspects. Research in
intelligence shows that the rumors spreading in social media leave a
particularly lasting impact on less smart people [3] and keep them from making
the right decisions. False news is used to build people’s fears, raise racist
ideas, and lead to bullying and violence against innocent people. Even fake
news has great democratic impacts. American presidential election showed how
it disrupts and fallacy people’s opinions [4]. In the last few years, there
have been many tragic incidents in Bangladesh due to rumors. In July 2019,
five people were beaten to death and ten injured by mobs as a result of
widespread rumors about the expected human sacrifice in the construction of
the Padma Bridge [5]. Because of the expanding number of clients in web-based
life, news can be immediately distributed by anyone, and its credibility
stands traded off. As fake news is written to mislead the readers, it makes it
a difficult task to detect based on the content of the news only. It becomes
essential to bring an efficient system to detect the fake news as the news
content is diverse in terms of style and subject in which it is written. Fake
news detection has lately received considerable interest from researchers. So
far different approaches have been introduced in the detection of fake
news[6].

Most of the techniques suggested in the literature to identify fake news deal
with the problem as a classification task by associating labels such as fake
or real, true or false, etc. with a particular text. In most cases, machine
learning and deep learning methods are utilized to achieve promising outcomes.
Research shows that SVMs have outperformed a number of supervised machine
learning algorithms for deception detection in text by exploiting content-
based features (e.g. linguistic and visual features)[7][8]. Although a lot of
work has been done to identify the fake news, it is still being done in the
Bangla Language to a limited extent[9][10]. With evolving digital technology,
the native Bangla speakers are now producing a large amount of Bangla text
online. As a result, it will be easier to create a dataset by scraping the
text from online sources and categorize them according to polarity of the
article by analyzing the text pattern.

In this work, detection of fake news is proposed using two different
modalities available in an efficient manner using Support Vector Machine (SVM)
and Multinomial Naive Bayes (MNB). Content-based features have been extracted
from the open source data to classify the fake news. Rest of the paper is
organized as follows. In section II we provide an overview of the related
works. Section III discusses the methodology i.e. how the data are collected
and pre-processed, feature extraction method and detection techniques. Section
IV demonstrates and discusses the result achieved through research and finally
Section V draws the conclusion with possible future research directions.

##  II Related Works

The rising prevalence and vast amount of fake news draws researchers to
analyze problems that internet users suffer from. A. S. Sharma et al. proposed
a hybrid extraction technique from text documents, combining Word2Vec and TF-
IDF[9] which may detect with standard CNN architecture whether a text document
in Bangla is satire or not with a precision of more than 96%. T. Islam et al
[10] retrieves comments as data from different social media using comment
extractors and removes punctuation marks, numerical meaning, emoticons which
create a perfect text corpus. Using Naive Bayes Classifier, which is commonly
used for spam identification, they used TF-IDF vectorizer to extract the
features from the text corpus and train the processing results.

Some researchers have been utilizing graph analysis to better identify origins
of fake news. Shu et al.[11] have shown that models of network diffusion can
be used to map the provenance nodes and the provenance origins of fake news.
Tarek Hamdi et al.[12] suggested an method that incorporates node embedding
and user-based functionality to improve Twitter’s analysis of fake news. Using
node2vec they retrieved information from twitter followers/followees graph
which provides a simplified way to help identify SOFNs. The finding that
characteristics produced by graph embedding are efficient but node embedding
features are powerful SOFN predictors that convey valuable information about a
user’s reputation in the social network. Kai Shu et al.[13] proposed a model
named Social Article Fusion (SAF) which combines the linguistic features of
news content with social context features to identify fake news. They use
commonly used autoencoders to identify the news content to represent text
content in lower dimensional space. To capture the users temporal interactions
with the fake news they used Recurrent Neural Networks (RNN), which
efficiently performs to capture the temporal relationship. Then they extract
useful features after classifying the data, and build various machine learning
models to identify fake news.

Mohamed Torky et al.[14] suggested a novel blockchain-based algorithm named
Proof of Credibility (PoC) to identify and prevent fake news and deceptive
social media content. The experimental results gave around 89% accuracy.
Eugenio Tacchini et al. has shown that Facebook posts can be categorized as
hoaxes or non-hoaxes with high accuracy depending on the users who ”like”
them. Use user IDs as features for classifying posts, they implemented two ML
algorithms (logistic regression and harmonic boolean label crowdsourcing)
obtained accuracies exceeding 99% even with very limited training sets[15].
But, the method offers difficult-to-beat efficiency, its implementation is
necessarily limited to situations, because the method uses social interactions
(i.e. ”likes”) as signals to help distinguish Facebook posts, it can not be
used when a post has no likes, and it is likely to perform worse when a post
only receives few social interactions.

To detect fake news, Kai Shu et al.[16] introduced a novel system called TriFN
(which can isolate useful functionality independently from news provider and
user obligations, as well as catch interrelationships simultaneously) in his
another research work. Marco L. Della Vedova et al.[17] suggested a novel
technique of machine learning that incorporates news content and social
signals and implemented their plan inside a Facebook Messenger chatbot and
accepted it with a legitimate application[15]. For contrast, they achieve
higher accuracies than [15] and [16] using their respective Facebook and
Twitter data sets.

##  III Proposed Methodology

In this work, two classifiers SVM and MNB have been used to classify fake news
from Bangla news articles. The dataset used in the proposed classification
system has been built by scraping various Bangla newspapers as it is hard to
find resources in Bangla language. System flow diagram of the classification
model is illustrated in Figure 1 and others steps are described below.

Figure 1: System Flow Diagram of the Proposed System

###  III-A Data Collection

For the research purpose, we scraped various news articles from
ProthomAlo111https://www.prothomalo.com/,
ProthomAlu222https://prothom1alu.blogspot.com/,
BalerKontho333http://www.balerkontho.net/,
Motikontho444https://motikontho.wordpress.com/ as no previous datasets were
found in Bangla language as well as it was hard to gather fake Bangla news.
Around 2500 articles were collected for our dataset as all these are public
articles. We decided that articles from very popular portals are real news and
fake news articles are collected from various sites which are known as portals
for satire news such as Motikontho, BalerKnotho, ProthomAlu, etc. Table I
provides the percentage of real and fake news in our dataset.

TABLE I: Data set Details Information Type of Article |  Total Count |  Percentage to Total
---|---|---
Real |  1548 |  60.92
Fake |  993 |  39.08

###  III-B Data Preprocessing

It is very important to apply some preprocessing on the raw text data before
feeding them to the classifier. A raw text might contain unnecessary symbols
or other things that are not important for our classification. Various
emoticons like :D, ;) might be helpful for sentiment analysis but not in our
case. We also removed various special characters e.g. @, #, ! etc from our
text. These elements can reduce or diminish the performance of the classifier.
After removing the numerical values, punctuation marks, special symbols, our
dataset is prepared for the classifier algorithms. Table II shows details
characters which were considered removing in preprocessing.

TABLE II: Characters Considered removing in Preprocessing Category |  Characters
---|---
Special Characters |  @, #, $, %, —, ,, ……
Bangla & English Digits |  1, 2, 3, 4, …. 0
English Alphabets |  A, B, C, …… Z; a, b, c,……z
Emoticons |  :), :D, :(, :o, …..

###  III-C Feature Extraction

Extracting the proper feature have a perfect impact on the performance of the
machine learning classifier algorithms. Count Vectorizer and TF-IDF Vectorizer
(term frequency–inverse document frequency) are used to extract features from
our text before feeding it into the classification algorithms. Count
Vectorizer generates a vector which has as several dimensions as the specific
word of corpora. Every single word has a particular dimension and contain 1 in
that specific dimension with 0 in others which simply keeps the frequency of
every words. TF-IDF vectorizer features numerical representations of the words
whether they are there or not, rather than just featuring a count. Words are
measured by frequency, multiplied by the inverse document frequency of them.
In simple terms, words that appear a good amount but everywhere should be
provided hardly any significance or weighting. In Bangla Language words like
following,

don’t offer a huge deal of interest. If a word occurs quite low, or sometimes
occurs, then such words are actually more relevant and should be carefully
weighed as such. This would lead to improved performance on classification. It
is a method intended to describe the significance of a keyword inside a text.
If the term denoted as ’$t$’, a particular document as ’$d$’ and the whole
document as ’$D$’, then the formula[18] is,

$tf-idf(t,d,D)=tf(t,d)*idf(t,D)$ (1)

Here,

$tf(t,d)$ is the frequency of ’$t$’ in ’$d$’

$idf(t,D)$ is how ’$t$’ is common or rare across ’$D$’

###  III-D Classifier

The dataset was split into two sets, training-set and test-set to feed into
classifier algorithms. The training set contains 70% of the dataset and the
test set contain 30% of the dataset. Among various classifier, we used two
widely used Multinomial Naïve Bayes Classifier and Support Vector Machine
Classifier with a linear kernel in this research. The Naive Bayes classifier
is based on Bayes theorem, a simple probabilistic classifier which is fast and
easy to implement. It is a tough task concerning which Naïve Bayes version
will be implemented. In the context of text classification Multinomial Naïve
Bayes gets better results than the Bernoulli Naïve Bayes or Gaussian naïve
Bayes[19]. Multinomial Naïve Bayes is often used in a classification problem
where the numerous occurrences of the word mean a lot. On the other hand, the
support vector machine would be used for regression or classification
problems. It utilizes the kernel technique to process the data, and determines
an appropriate boundary between the potential outputs depending on these
transformations. It is robust against over-fitting problems related to high-
dimensional space particularly for text datasets[19]. The linear kernel
preferred for text classification as most of them are linearly separable.

##  IV Experimental Result

In this section, performance analysis of multinomial naive bayes and support
vector machine classifiers on our dataset has been described. The dataset was
split into two parts randomly. One part contains 70% of the data which is used
to train the classifiers and the remaining 30% is to test the performance of
the classifiers.

Multinomial Naive Bayes gave us 93.32% accuracy for our dataset where Support
Vector Classifier with linear kernel gave 96.64% accuracy. We observed that
SVM performs better than MNB. Table III and IV shows the precision, recall and
F1-score for MNB and SVM classifier models consecutively.

TABLE III: Results for MNB |  precision |  recall |  f1-score
---|---|---|---
Real |  0.90 |  1.00 |  0.95
Fake |  1.00 |  0.83 |  0.90
avg |  0.95 |  0.91 |  0.93
TABLE IV: Results for SVM |  precision |  recall |  f1-score
---|---|---|---
Real |  0.97 |  0.99 |  0.98
Fake |  0.99 |  0.95 |  0.97
avg |  0.98 |  0.97 |  0.97

Figure 2 shows the confusion matrix for Multinomial Naïve Bayes which shows
that 472 news was indicated as real and was actually real news. On the other
hand 50 news was predicted as fake which was actually real, that means false
negative. Similarly, 240 fake news were predicted as fake and was actually
fake but only 1 was predicted as fake and was actually real which new was
falsely indicated as real.

Figure 2: Confusion Matrix for Multinomial Naïve Bayes.

Figure 3 shows the confusion matrix for Support Vector Classifier which shows
that 469 news was indicated as real and was actually real news. On the other
hand only 14 news was predicted as fake which was actually real, that means
false negative. So we can see SVC gives less false negative. Similarly, 276
fake news were predicted as fake and was actually fake but only 4 was
predicted as fake and was actually real which new was falsely indicated as
real.

Figure 3: Confusion Matrix for Support Vector Machine with Linear Kernel.

##  V Conclusion

This research work concludes that Support Vector Machine with linear kernel is
doing marginally better than Multinomial Naïve Bayes on our dataset. Since the
research with the identification of Bangla fake News is new[9][10], this work
will be helpful for any future works on such topic. Very recently a research
work has been accepted in a conference consisting a dataset around 49 thousand
news articles[20] and it is our plan for the future to work with this dataset
to expand the number of features and sufficient lexicons. Besides, a stemmer
can be applied to minimize corpus size and enhance model efficiency. In the
future, the rate of success can be improved by doing more research utilizing
hybrid-classifiers.

## References

  * [1] R. K. Nielsen, R. Fletcher, N. Newman, J. S. Brennen, and P. N. Howard, “Navigating the ’infodemic’: How people in six countries access and rate news and information about coronavirus,” Apr 2020.
  * [2] U. News. During this coronavirus pandemic, ‘fake news’ is putting lives at risk: Unesco. [Online]. Available: https://news.un.org/en/story/2020/04/1061592
  * [3] J. De keersmaecker and A. Roets, “Fake news: Incorrect, but hard to correct. the role of cognitive ability on the impact of false information on social impressions,” in _Intelligence_ , vol. 65, 2017, pp. 107–110.
  * [4] N. Grinberg, K. Joseph, L. Friedland, B. Swire-Thompson, and D. Lazer, “Fake news on twitter during the 2016 u.s. presidential election,” _Science_ , vol. 363, no. 6425, pp. 374–378, 2019. [Online]. Available: https://science.sciencemag.org/content/363/6425/374
  * [5] (2019, July) Mobs beat five dead for ‘kidnapping’, daily star. [Online]. Available: https://www.thedailystar.net/frontpage/news/mobs-beat-2-dead-kidnapping-1774471
  * [6] A. Bondielli and F. Marcelloni, “A survey on fake news and rumour detection techniques,” _Information Sciences_ , vol. 497, p. 38–55, 2019.
  * [7] V. Rubin, N. Conroy, Y. Chen, and S. Cornwell, “Fake news or truth? using satirical cues to detect potentially misleading news,” in _Proceedings of the Second Workshop on Computational Approaches to Deception Detection_. San Diego, California: Association for Computational Linguistics, Jun. 2016, pp. 7–17. [Online]. Available: https://www.aclweb.org/anthology/W16-0802
  * [8] H. Zhang, Z. Fan, J. Zheng, and Q. Liu, “An improving deception detection method in computer-mediated communication,” _Journal of Networks_ , vol. 7, no. 11, Jan 2012.
  * [9] A. S. Sharma, M. A. Mridul, and M. S. Islam, “Automatic detection of satire in bangla documents: A cnn approach based on hybrid feature extraction model,” _arXiv preprint arXiv:1911.11062_ , 2019.
  * [10] T. Islam, S. Latif, and N. Ahmed, “Using social networks to detect malicious bangla text content,” in _2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT)_ , 2019, pp. 1–4.
  * [11] K. Shu, H. R. Bernard, and H. Liu, _Studying Fake News via Network Analysis: Detection and Mitigation_. Cham: Springer International Publishing, 2019, pp. 43–65. [Online]. Available: https://doi.org/10.1007/978-3-319-94105-9_3
  * [12] T. Hamdi, H. Slimi, I. Bounhas, and Y. Slimani, “A hybrid approach for fake news detection in twitter based on user features and graph embedding,” in _Distributed Computing and Internet Technology_ , D. V. Hung and M. D´Souza, Eds. Cham: Springer International Publishing, 2020, pp. 266–280.
  * [13] K. Shu, D. Mahudeswaran, and H. Liu, “Fakenewstracker: a tool for fake news collection, detection, and visualization,” _Computational and Mathematical Organization Theory_ , vol. 25, no. 1, p. 60–71, 2018.
  * [14] M. Torky, E. Nabil, and W. Said, “Proof of credibility: A blockchain approach for detecting and blocking fake news in social networks,” _International Journal of Advanced Computer Science and Applications_ , vol. 10, no. 12, 2019. [Online]. Available: http://dx.doi.org/10.14569/IJACSA.2019.0101243
  * [15] E. Tacchini, G. Ballarin, M. L. D. Vedova, S. Moret, and L. de Alfaro, “Some like it hoax: Automated fake news detection in social networks,” 2017.
  * [16] K. Shu, S. Wang, and H. Liu, “Exploiting tri-relationship for fake news detection,” _ArXiv_ , vol. abs/1712.07709, 2017.
  * [17] M. L. D. Vedova, E. Tacchini, S. Moret, G. Ballarin, M. D. Pierro, and L. de Alfaro, “Automatic online fake news detection combining content and social signals,” _2018 22nd Conference of Open Innovations Association (FRUCT)_ , pp. 272–279, 2018.
  * [18] Wikipedia. Tf–idf - wikipedia. [Online]. Available: https://en.wikipedia.org/wiki/Tf%E2%80%93idf
  * [19] K. Kowsari, K. Jafari Meimandi, M. Heidarysafa, S. Mendu, L. Barnes, and D. Brown, “Text classification algorithms: A survey,” _Information_ , vol. 10, no. 4, p. 150, 2019.
  * [20] M. Z. Hossain, M. A. Rahman, M. S. Islam, and S. Kar, “Banfakenews: A dataset for detecting fake news in bangla,” 2020.


# Flying Trapeze Act Motion Planning Algorithm for Two-Link Free-Flying
Acrobatic Robot

Thanapong Chuangyanyong1, Panusorn Chinsakuljaroen1,
Worachit Ketrungsri1, and Thanacha Choopojcharoen1,2 1All authors are
associated with the Institute of Field Robotics, King Mongkut’s University of
Technology Thonburi, 126 Pracha Uthit Rd., Bang Mot, Thung Khru, Bangkok
10140.2Thanacha Choopojcharoen is currently an adjunct lecturer at the
Institute of Field Robotics, King Mongkut’s University of Technology Thonburi.

###### Abstract

A flying trapeze act can be a challenging task for a robotics system since
some act requires the performer to catch another trapeze or catcher at the end
after being airborne. The objective of this paper is to design and validate a
motion planning algorithm for a two-link free-flying acrobatic robot that can
accurately land on another trapeze after free-flying in the air. First, the
proposed algorithm plan the robot trajectory with the non-linear constrained
optimization method. Then, a feedback controller is implemented to stabilize
the posture. However, since the spatial position of the center-of-mass of the
robot cannot be controlled, this paper proposes a trajectory correction scheme
that manipulates the robot’s posture such that the robot is still able to land
on the target. Lastly, the whole algorithm is validated in the simulation that
mimics real-world circumstances.

##  I Introduction

In recent years, robotics has become an integral part of immersive
entertainment. New technologies allow robots to achieve complex behavior, one
being a stunt double for untethered aerial performance in an amusement park.
The most prominent example is Disney’s “Stuntronics,” a performance robot that
is being used at Walt Disney World®, which is an implementation of the
research from Disney Research, [1], where a three-link robot is designed to
perform a safe landing from a swinging pendulum. Their work presents the
potential of using robots to perform other complex acrobatics treats. This
research will be focusing on designing a motion planning and control algorithm
for one of the most prevalent aerial acrobatic performances, the flying
trapeze act. The act involves a performer jumping from a platform while
holding onto a flying trapeze, then catching another performer, also known as
a catcher. Sometimes an experienced performer releases the trapeze and becomes
airborne before catching the target. Nakanishi and Fukuda [2] proposed a
control strategy for a ricochetal brachiation robot, which resembles a flying
trapeze act. The controller works by combined system dynamics and mechanical
energy regulator. However, their approach only works on targets that are the
same height as the releasing height, which is rarely the case in the flying
trapeze act. However, Wan et al. [3] solve the aforementioned shortcoming by
using a control method that combined the virtual constraint-based trajectory
planning and tracking control algorithm. Both [2] and [3] are able to achieve
accuracy of the final catching position. Especially [2], which can adjust the
robot posture when the releasing condition (the spatial position and velocity
when the robot starts to experience free fall) is off from the desired value.
The posture adjustment is essential since the system’s center-of-mass follows
a parabolic trajectory predetermined by the releasing condition. For the
research to move closer to a physical realization, the control algorithm
should be taken into account the adjustable target’s position similar to [3],
and it should incorporate “posture adjustment” similar to [2] in case of
control delay or physical uncertainties. Additionally, there are several
points that must be addressed, for instance, the robot’s physical limits
(joint limits, actuator torque, and actuator velocity) and the attack angle of
which the robot grasp the target.

The algorithm will be implemented on an acrobatic robot research platform
named Mon$\chi$. Although Mon$\chi$ is designed for general acrobatic
movement, this paper only focuses on the flying trapeze act. Mon$\chi$
consists of two rigid links connected via an actuated revolute joint. At the
end of one of the links lies a gripper module used to clutch the trapeze and
land on the target. The robot is specifically designed to stay on the sagittal
plane closely. Therefore, while the robot is flying in the air, there are four
degree-of-freedom (DOF) of movement with only a single-DOF actuator. Moreover,
when the robot clutches the trapeze, there are two-degree-of-freedom with the
same actuator. This kind of mechanism is called an underactuated system, which
adds more challenges to the design of the algorithm.

##  II Problem Formulation

A flying trapeze act requires a positional accuracy at the end of the
trajectory to be able to catch the next trapeze or the catcher. The tolerance
of the distance between the final landing position and the target is aimed to
be less than $5\text{\,}\mathrm{cm}$ according to the ratio between the
average human hand and the average length of the human body with extended
arms. The robot, Mon$\chi$, consists of two rigid links. The first link
provides a mounting point for an embedded computer, a power source, an
actuator, an inertial measurement unit (IMU), and a gripper module. The second
link is connected to the first link via an actuated revolute joint. $F_{g}$ is
the coordinate frame of the landing position attached to the first link. The
origin of $F_{g}$ is the contact point that the motion planner will attempt to
navigate to the target bar. The z-axis of $F_{g}$ is parallel to the axis of
actuation, and the x-axis is perpendicular to the contact surface of the
gripper.

To verify the motion planning algorithm of the flying trapeze act, the
apparatus must be designed accordingly. The apparatus comprises a starting
platform, a trapeze, and a target bar. The target bar acts as a catcher;
however, it is designed to be stationary rather than swingable. The function
of the starting platform is to lock Mon$\chi$ in position until the starting
signal is set, which then instantly releases the robot. A global coordinate
frame $F_{0}$ has its z-axis coinciding with the axis of rotation of the
trapeze’s hinge, and its y-axis points upward—the gravity points in the $-y$
direction of the $F_{0}$. The x-axis is parallel to the ground and points to
the target bar. The target bar is adjustable in the sagittal plane, and
$F_{t}$ denotes its attached coordinate frame.

Figure 1: Coordinate frames of Mon$\chi$ and its environment.

##  III Modeling

A trapeze act consists of two phases: a “swinging phase” and a “flying phase.”
The robot is in a swinging phase when it clutches the trapeze with its gripper
module. Once the robot releases its gripper from the trapeze, it transitions
into a flying phase. By denoting the releasing time by $t_{rel}$, the
transition occurs only once when time $t$ is equal to $t_{rel}$. Since there
are only two phases with a simple transition behavior, equations of motion are
derived separately. The system’s mode can be switched with a discrete state
variable $s\in\\{0,1\\}$. When $s$ is $0$, the robot is in the swinging phase.
When $s$ is $1$, the robot is in the flying phase.

Figure 2: Mon$\chi$’s model in the swinging phase (left) and Mon$\chi$’s model
in the flying phase (right).

###  III-A Flying Phase

Fig. 2 shows the model of the robot. $m_{1},m_{2},I_{1},I_{2}$ are the mass
and moment of inertia about the local z-axis of the first link and the second
link, respectively. $F_{h}$ is the hip’s coordinate frame, which is rigidly
attached to the first link. Its origin coincides with the axis of actuation,
and the x-axis points directly toward the center-of-mass of the first link.
$r_{1}$ and $r_{2}$ are the distance corresponding to the first link center-
of-mass measured from the origin of $F_{h}$ in the $+x$ direction and the
distance corresponding to the second link center-of-mass measured from the
origin of $F_{\beta}$ in the $-x$ direction, respectively.

In the flying phase, given negligible air drag, Mon$\chi$ has no external
wrenches acted on it besides the gravity. Consequently, the center-of-mass
trajectory is modeled as a parabolic trajectory (15) and will be entirely
determined by the position and the velocity right after it leaves the trapeze.
Thus, the angular position and angular velocity of both links can be decoupled
from its center-of-mass position and velocity [4]. The spatial position and
velocity can be added in the form of a simple time-varying variant of
projectile motion afterward. Therefore, the generalized coordinate
($\mathbf{q}$) consists of two configuration variables: a rotation of the
first link relative to the x-axis of $F_{0}$ about the z-axis of $F_{0}$
($\alpha$), and a rotation of the second link relative to the x-axis of
$F_{h}$ about the z-axis of $F_{h}$ ($\beta$). The Euler-Lagrange equation is
used to obtain the equation of motion.

###  III-B Swinging Phase

In the swinging phase, Mon$\chi$ is modeled as a double pendulum with an
actuated revolute joint on the second joint. The length of the first link of
the double pendulum ($l_{h}$) is measured from the origin of $F_{0}$ to the
origin of $F_{h}$ while the robot clutches the trapeze. $\eta$ is the mass of
the first link with the trapeze. $\zeta$ is the moment of inertia about the
z-axis of the first link while both links are rigidly coupled. One end of the
trapeze is attached to the origin of $F_{0}$ with a revolute joint, as seen in
Fig. 2. $\kappa_{1}$ and $\kappa_{2}$ are the angular offset measured from the
x-axis of $F_{h}$ to the rotating axle of the trapeze, and the angular offset
measured from -x-axis of $F_{\beta}$ to the center-line of the second physical
link, respectively. $\sigma_{\alpha}$ and $\sigma_{\beta}$ are the damping
coefficient of the revolute joint at $F_{0}$ and $F_{h}$, respectively. The
former is added to maintain continuity while transitioning between phases. The
latter is added to determine the physical limit of the actuated joint. Similar
to the flying phase, equation motion is formulated; however, the structure is
identical. As a result, the equation of motion can be expressed such that the
only difference is a parameter vector ($\boldsymbol{\uptheta}_{s}$).
Parameters are separated from both equations of motion, which can now be
written as in (1).

$\mathbf{\uptau}=\mathbf{M}_{s}(\mathbf{q})\ddot{\mathbf{q}}+\mathbf{C}_{s}(\dot{\mathbf{q}},\mathbf{q})\dot{\mathbf{q}}+\mathbf{B}_{s}\dot{\mathbf{q}}+\mathbf{g}_{s}(\mathbf{q})$
(1)

$\displaystyle\mathbf{M}_{s}(\mathbf{q})=\begin{bmatrix}2\theta_{s,1}\cos(\beta)+\theta_{s,2}+\theta_{s,3}&\theta_{s,1}\cos(\beta)+\theta_{s,3}\\\
\theta_{s,1}\cos(\beta)+\theta_{s,3}&\theta_{s,3}\end{bmatrix}$ (2)

$\mathbf{C}_{s}(\dot{\mathbf{q}},\mathbf{q})=\begin{bmatrix}-\theta_{s,1}\sin(\beta)\dot{\beta}&-\theta_{s,1}\sin(\beta)(\dot{\alpha}+\dot{\beta})\\\
\theta_{s,1}\sin(\beta)\dot{\alpha}&0\end{bmatrix}$ (3)

$\displaystyle\mathbf{g}_{s}(\mathbf{q})=\begin{bmatrix}\theta_{s,6}\sin(\alpha^{\prime})+\theta_{s,5}\cos(\alpha^{\prime})+\theta_{s,4}\cos(\alpha^{\prime}+\beta)\\\
\theta_{s,4}\cos(\alpha^{\prime}+\beta)\end{bmatrix}$ (4)

$\mathbf{B}_{s}=\begin{bmatrix}\theta_{s,7}&0\\\ 0&\theta_{s,8}\\\
\end{bmatrix},~{}\alpha^{\prime}=\alpha+\kappa_{1}$

$\boldsymbol{\uptheta}_{0}=\begin{bmatrix}m_{2}{l_{h}}{r_{2}}\\\
\eta(l_{c}^{2}+a_{c}^{2})+(\eta+m_{2})l_{h}^{2}-2\eta l_{h}l_{c}+\zeta\\\
m_{2}{r_{2}^{2}}+I_{2}\\\ \eta{g_{y}}a_{c}\\\
\eta{g_{y}}(l_{h}-l_{c})+m_{2}{g_{y}}l_{h}\\\ m_{2}{g_{y}}r_{2}\\\
\sigma_{\alpha}\\\ \sigma_{\beta}\\\ \end{bmatrix}$ (5)

$\boldsymbol{\uptheta}_{1}=\begin{bmatrix}\mu r_{1}r_{2}\\\ \mu
r_{1}^{2}+I_{1}\\\ \mu r_{2}^{2}+I_{2}\\\ 0\\\ 0\\\ 0\\\ 0\\\
\sigma_{\beta}\\\ \end{bmatrix},~{}~{}\mu=\frac{m_{1}m_{2}}{m_{1}+m_{2}}$ (6)

###  III-C State Space Representation

Finally, the state space representation of the system can be written as
presented in (7) and (8).

$\mathbf{x}=\begin{bmatrix}\mathbf{q}\\\ \dot{\mathbf{q}}\\\ \end{bmatrix}$
(7)

$\dot{\mathbf{x}}=f_{\lambda(t)}(\mathbf{x},u),~{}\lambda(t)=\begin{cases}0,&\text{if}~{}t\leq
t_{rel}\\\ 1,&\text{otherwise.}\\\ \end{cases}$ (8)

Where:

$\displaystyle f_{s}(\mathbf{x},u)=$
$\displaystyle\begin{bmatrix}\dot{\mathbf{q}}\\\
\mathbf{M}_{s}(\mathbf{q})^{-1}\left(\begin{bmatrix}0\\\ 1\\\
\end{bmatrix}u-\mathbf{C}_{s}(\dot{\mathbf{q}},\mathbf{q})\dot{\mathbf{q}}-\mathbf{B}_{s}\dot{\mathbf{q}}-\mathbf{g}_{s}(\mathbf{q})\right)\\\
\end{bmatrix}$ (9)

##  IV Control Strategy

One of the most prevalent motion planning methods for an underactuated system
like Mon$\chi$ is non-linear constrained trajectory optimization. The
optimization approach can be constrained to satisfy the constraints stated
above. Moreover, the system only has four state variables, which further
reduces optimization time. Thus, the strategy is to find an optimal trajectory
for Mon$\chi$ to follow. Furthermore, the uncertainty in the robot’s physical
properties can cause the robot to miss the target bar even though the planned
trajectory is feasible. Therefore, the controller shall be implemented to
follow its reference pose trajectory.

###  IV-A Optimal Motion Planning

To land at the target bar, various constraints must be considered in this
optimization problem. First, the landing point of the robot
($\mathbf{p}_{0}^{g}$) must be at the target bar ($\mathbf{p}_{0}^{t}$) at the
end of the trajectory. The landing point can be transformed from the center-
of-mass position ($\mathbf{p}_{0}^{c}$), which can be computed from (15).
Every position vector in this paper is constructed with respect to $F_{0}$.

$\mathbf{p}_{0}^{g}(\mathbf{q},t)=\mathbf{p}_{0}^{c}(t)+\mathbf{p}_{c}^{h}(\mathbf{q})+\mathbf{p}_{h}^{g}$
(10)

$\begin{split}\mathbf{p}_{c}^{h}(\mathbf{q})&=\frac{m_{1}r_{1}}{m_{1}+m_{2}}\begin{bmatrix}\cos(\alpha)\\\
\sin(\alpha)\\\ \end{bmatrix}\\\
&+\frac{m_{2}r_{2}}{m_{1}+m_{2}}\begin{bmatrix}\cos(\alpha+\beta)\\\
\sin(\alpha+\beta)\\\ \end{bmatrix}\end{split}$ (11)

$\mathbf{p}_{h}^{g}$ is the position of the landing point frame ($F_{g}$)
relative to the origin of $F_{h}$. $\mathbf{p}_{h}^{g}$ is used here rather
than translating directly from the center-of-mass to the landing position
because, this way, the landing position is always referenced to the robot’s
hip, which is easier for development purposes.

Next, the landing point must not be obstructed by any part of the robot. For
this research, the landing point has an opening angle between $\gamma_{min}$
and $\gamma_{max}$ with respect to the x-axis of the $F_{g}$. If the angle of
the position vector $\mathbf{p}_{g}^{t}$ while the robot is approaching lies
within the opening with respect to the x-axis of $F_{g}$ (12), the landing
point will not be obstructed. Moreover, an offset of $2\pi\nu$ can be added to
alter the final posture; consequently, force the robot to somersault where
$\nu$ is a number of somersault attempts.

$\psi(\mathbf{x}(t))=\alpha(t)+\mathbf{R}_{0}^{g}-\operatorname{arctan2}(p_{g}^{t,y}(\mathbf{q},t),p_{g}^{t,x}(\mathbf{q},t))$
(12)

$\mathbf{p}_{g}^{t}(\mathbf{q},t)=\mathbf{p}_{0}^{t}-\mathbf{p}_{0}^{g}(\mathbf{q},t)$
(13)

Lastly, the minimum effort squared is chosen to be the cost function. The
aforementioned function usually produces a smooth and life-like trajectory.
With these constraints, the optimization problem can be formulated as in (14).

$\langle\mathbf{x}^{*}(t),u^{*}(t),T_{0}^{*},T_{1}^{*}\rangle=\operatorname*{arg\,min}_{\mathbf{x}(t),\\\
u(t),\\\ T_{0},\\\ T_{1}}\int_{\tau=0}^{T_{0}+T_{1}}u^{2}(\tau)\mathrm{d}\tau$
(14)

subject to:

$\mathbf{x}(0)=\mathbf{x}_{0}$

$\dot{\mathbf{x}}(t)=f_{\lambda(t)}(\mathbf{x}(t),u(t))\forall
t\in[0,T_{0}+T_{1}]$

$\gamma_{min}\leq\psi(\mathbf{x}(\tau))-2\pi\nu\leq\gamma_{max}\\\
\forall\tau\in[T_{0}+\phi_{min}T_{1},T_{0}+\phi_{max}T_{1}]$

$\mathbf{p}_{0}^{g}(T_{0}+T_{1})=\mathbf{p}_{0}^{t}$

$\mathbf{x}_{0}$ is the desired initial state, and $T_{s}$ is the trajectory
duration. The final state of the swinging phase, $\mathbf{x}(t_{rel})$, is
used to calculate the initial condition of the center-of-mass trajectory in
(16) and (17) while the robot is in the flying phase. $\phi_{min}$ and
$\phi_{max}$ are approaching constants that range between 0 and 1.
$\phi_{min}$ is always lesser than $\phi_{max}$. If $\phi_{min}$ and
$\phi_{max}$ are set to $0.75$ and $0.95$, respectively, this means the third
constraint in (14), approaching constraint, will only be enforced from
$75\text{\,}\mathrm{\char 37\relax}$ to $95\text{\,}\mathrm{\char 37\relax}$
of the flying phase trajectory in the time domain. Lastly, the boundary on
state trajectory and control input is also added to ensure that the robot
operates within its limit.

$\mathbf{p}_{0}^{c}(t)=\mathbf{p}_{0}^{c}(t_{rel})+\mathbf{v}_{0}^{c}(t_{rel})t+\begin{bmatrix}0\\\
g_{y}\\\ \end{bmatrix}\frac{t^{2}}{2}$ (15)

$\mathbf{p}_{0}^{c}(t_{rel})=\mathbf{p}_{0}^{h}(t_{rel})-\mathbf{p}_{c}^{h}(t_{rel})$
(16)

$\mathbf{v}_{0}^{c}(t_{rel})=\dot{\mathbf{p}}_{0}^{h}(t_{rel})-\dot{\mathbf{p}}_{c}^{h}(t_{rel})$
(17)

$\mathbf{p}_{0}^{h}(t_{rel})=-\begin{bmatrix}l_{h}\cos(\alpha(t_{rel})+\kappa_{1})\\\
l_{h}\sin(\alpha(t_{rel})+\kappa_{1})\\\ \end{bmatrix}$ (18)

This motion planning algorithm is planned to be implemented on an embedded
computer. Therefore, the optimization is parameterized beforehand. Some
aspects of the problem must be modified. For instance, the Hermite-Simpson
collocation method is used here to enforce the system dynamics [5] rather than
using a numerical integration method. The resulting state trajectory
($\mathbf{x}^{*}$) and resulting control input ($u^{*}$) will be stored as
cubic coefficients and linear coefficients, respectively.

Figure 3: Example of a single somersault trajectory ($\nu=1$) from non-linear
constrained optimization method, in this example,
$\mathbf{p}_{0}^{t}=[2.44~{}-0.9]^{\top}$.

###  IV-B Posture Control

After obtaining the feasible trajectory, Mon$\chi$ must stabilize its
trajectory according to the reference trajectory. Note that the controller
only has to stabilize the state variable ($\mathbf{x}$), not the position in
the sagittal plane. A time-varying linear quadratic regulator (TVLQR) is used
here since it has been shown to perform sufficiently in several physical
nonlinear systems in terms of tracking performance, as seen in [6], [7], and
[8]. Especially the last mentioned, which is highly similar to our system both
in robot topology and the trajectory it is trying to follow.

$u(t)=u^{*}(t)-\mathbf{K}(t)(\mathbf{x}-\mathbf{x}^{*}(t))$ (19)

The controller stabilizes the linearized plant in its error coordinate (19).
$\mathbf{K}(t)$ is a piece-wise linear optimal gain matrix. It can be solved
by forming a differential-algebraic Riccati equation and integrating it
backward in time. The final cost of the swinging phase can be set to the
initial cost of the flying phase to make $\mathbf{K}(t)$ continuous.
$\mathbf{K}(t)$ can also be stored in the same manner as the control input
($u^{*}$).

###  IV-C Trajectory Correction

The uncontrollability of Mon$\chi$’s center-of-mass can pose a potential
problem. For instance, when the gripper misses the releasing time by little,
the releasing position ($\mathbf{p}_{0}^{c}(t_{rel})$), and the releasing
velocity ($\mathbf{v}_{0}^{c}(t_{rel})$) can vary from the reference values
greatly. Consequently, the robot’s center-of-mass trajectory will also change.
This causes the gripper to miss the target because the posture stabilizer only
stabilizes the state variable ($\mathbf{x}$). Even though the center-of-mass
trajectory can not be altered without an external wrench, the gripper position
($\mathbf{p}_{0}^{g}(t)$); however, can be altered if the actual releasing
state is known [2]. Though, the correction scheme in [2] only works for
horizontal bars with the same height. Thus, the optimal control approach from
earlier can be reduced to solve a new trajectory. The true releasing state can
be estimated by using any online parameter estimator, e.g., a recursive least
square. In this case, the least square estimator is used because the
trajectory is relatively short. Then, the reduced version of the previous
optimization problem can be used to find a new feasible trajectory.

$\langle\mathbf{x}^{*}(t),u^{*}(t),T^{*}\rangle=\operatorname*{arg\,min}_{\mathbf{x}(t),\\\
u(t),\\\ T,\\\ \upsilon}\int_{\tau=t_{rel}}^{T}u^{2}(\tau)\mathrm{d}\tau$ (20)

subject to:

$\mathbf{x}(0)=\hat{\mathbf{x}}(t_{rel})$

$\dot{\mathbf{x}}(t)=f_{1}(\mathbf{x}(t),u(t))\forall t\in(t_{rel},T]$

$\gamma_{min}\leq\psi(\mathbf{x}(\tau))+\upsilon\leq\gamma_{max}\forall\tau\in[\phi_{min}T,\phi_{max}T]$

$\mathbf{p}_{0}^{g}(T)=\mathbf{p}_{0}^{t}$

Now, the problem is reduced to only optimizing the flying phase. $\upsilon$ is
an auxiliary decision variable and is there, in place of $-2\pi\nu$, because
the final posture is already constrained by the releasing state there. The
time window between robot releasing and robot landing on the target is already
determined by its center-of-mass trajectory; the robot also experienced the
nonholonomic constraint on velocity emerges, in this case, from the
conservation of angular momentum in a multibody system [9]. Hence, the
possible range of the final posture is already determined; adding more
constraints on the final posture may cause the feasible trajectory to be more
strenuous to find or even impossible to find. Most importantly, the flying
portion of the previously optimized decision variables can be utilized as an
initial guess to further decrease the optimization time.

After the online optimization is finished, the posture trajectory is corrected
by calculating the new optimal time-varying LQR gain. Fig. 4 and Fig. 5 show
how trajectory correction affect the reference trajectory of the robot. The
newly optimized state reference and control input reference can be denoted as
$\mathbf{x}^{*}_{i}$, and $u^{*}_{i}$ instead, where $i$ is the number of the
optimization attempt since the trajectory correction algorithm can be
initiated as many times as specified. This may be helpful when the robot
experiences another change in its center-of-mass trajectory, like a strong
gust of wind.

Figure 4: Visualization of the new reference position trajectory
($\alpha_{1}^{*}$, $\beta_{1}^{*}$), and the new reference actuator’s effort
($u_{1}^{*}$).

Figure 5: Visualization of the corrected gripper trajectory
($\mathbf{p}_{0}^{g*}$)

With all necessary components in place, the full control scheme can be
realized. The simplified block diagram of the control scheme can be seen in
Fig. 6, which shows that only the state estimator, TVLQR, and the trajectory
generator must be computed in a real-time manner. And other tasks that require
solving nonlinear optimization problems should be isolated in another
computing thread or processor.

Figure 6: A simplified block diagram of the proposed control scheme.

##  V Simulation

Both Mon$\chi$ and the testing environment are simulated with Simulink with a
high level of detail, including the contact between every element.

###  V-A Trajectory Correction Verification

To verify the effectiveness of the trajectory correction algorithm, simulation
is performed in MATLAB to numerically find the releasing conditions that
result in a successful landing. Results are compared between two scenarios:
trajectory correction algorithm disabled ($\text{TC}=0$) and trajectory
correction algorithm enabled ($\text{TC}=1$). Note that all trajectory
correction attempts must be done within the time-limit, in this experiment,
$0.5\text{\,}\mathrm{s}$. Below are parameters for this experiment and the
next one, where $\mathbf{Q}$ and $\mathbf{R}$ are weighting matrices.

$\mathbf{Q}=\begin{bmatrix}4&0&0&0\\\ 0&2&0&0\\\ 0&0&2&0\\\ 0&0&0&2\\\
\end{bmatrix},~{}\mathbf{R}=\begin{bmatrix}0.3\end{bmatrix}$

$\mathbf{p}_{0}^{t}=\begin{bmatrix}1.9\\\ -1.725\\\
\end{bmatrix},~{}\mathbf{x}_{0}=\begin{bmatrix}-\kappa_{1}&\kappa_{1}&0&0\\\
\end{bmatrix}^{\top}$

$\displaystyle\gamma_{min}=$-40\text{\,}\mathrm{\SIUnitSymbolDegree}$,~{}\gamma_{max}=$10\text{\,}\mathrm{\SIUnitSymbolDegree}$$
$\displaystyle\phi_{min}=0.73,~{}\phi_{max}=0.90,~{}\nu=1$

After the optimization process completed, a nominal trajectory and the
releasing condition are obtained.

$\displaystyle\alpha(t_{rel})=$2.834\text{\,}\mathrm{rad}$,~{}\beta(t_{rel})=$0.390\text{\,}\mathrm{rad}$$
$\displaystyle\dot{\alpha}(t_{rel})=$1.7816\text{\,}\mathrm{rad}\text{\,}{\mathrm{s}}^{-1}$,~{}\dot{\beta}(t_{rel})=$6.139\text{\,}\mathrm{rad}\text{\,}{\mathrm{s}}^{-1}$$
$\displaystyle t_{rel}=$1.0204\text{\,}\mathrm{s}$$

The releasing condition progresses incrementally away from the nominal
releasing condition to construct a “margin” of releasing conditions that
result in successful acts.

Figure 7: This figure shows how the initial releasing point influences
Mon$\chi$’s ability to land. Successful attempts with trajectory correction
are marked in red, and successful attempts without trajectory correction are
marked in blue.

###  V-B Flying Trapeze Act Simulation

The entire system is tested in the simulation. Various parameters have been
altered in this experiment to reflect uncertainty in the real world. Those
parameters are the robot’s initial position, inertial properties, and damping
coefficients. First, inertial properties, e.g., mass, the moment of inertia,
and the center-of-mass position, are correlated. For that reason, the function
that can calculate uncertainties according to the fabrication tolerance of
each part is written. Results are shown in table I. Unfortunately, the damping
coefficient cannot be obtained yet since the system has not been realized;
hence, relatively high uncertainty is selected.

TABLE I: Robot Parameters for the Experiment Parameter |  Value
---|---
$m_{1}$ |  $1.5790\pm 0.0076\text{\,}\mathrm{kg}$
$m_{2}$ |  $1.4370\pm 0.0400\text{\,}\mathrm{kg}$
$r_{1}$ |  $0.1443\pm 0.0052\text{\,}\mathrm{m}$
$r_{2}$ |  $0.1269\pm 0.0033\text{\,}\mathrm{m}$
$\kappa_{1}$ |  $-0.0369\pm 0.0001\text{\,}\mathrm{rad}$
$\kappa_{2}$ |  $0.0001\pm 0.0023\text{\,}\mathrm{rad}$
$I_{1}$ |  $0.0375\pm 0.0013\text{\,}\mathrm{kg}\text{\,}{\mathrm{m}}^{2}$
$I_{2}$ |  $0.0237\pm 0.0009\text{\,}\mathrm{kg}\text{\,}{\mathrm{m}}^{2}$
$\sigma_{\alpha}$ |  $0.050\pm 0.010\text{\,}\mathrm{N}\text{\,}\mathrm{m}\text{\,}{\mathrm{rad}}^{-1}\text{\,}{\mathrm{s}}^{-1}$
$\sigma_{\beta}$ |  $0.200\pm 0.040\text{\,}\mathrm{N}\text{\,}\mathrm{m}\text{\,}{\mathrm{rad}}^{-1}\text{\,}{\mathrm{s}}^{-1}$

In the simulation, contacts between objects were also added to simulate
effects from friction and contact forces between objects, especially between a
robot’s gripper and trapeze’s handle, which affects the center-of-mass
trajectory ($\mathbf{p}_{0}^{c}$). Three hundred test scenarios are generated
for each simulation setting based on parameters in table I. Other than
parameters in table I, uncertain initial position is also added, with a
maximum magnitude of $\pm$$0.01\text{\,}\mathrm{rad}$ for both $\alpha_{0}$
and $\beta_{0}$. The result is shown in table II. Comparison of the gripper
trajectory between a controller with and without trajectory correction are
plotted in Fig. 8. Note that the successful trajectories on the right of the
Fig. 8 (blue) can be grouped into two categories. First are the trajectories
that should have missed the target but were corrected by the trajectory
correction module. And there are a few that did not trigger the trajectory
correction module since the releasing states ($\mathbf{p}_{0}^{c}(t_{rel})$
and $\mathbf{v}_{0}^{c}(t_{rel})$) did not deviate much from the baseline. The
trajectories in the latter group are similar to some of the successful
trajectories on the left plot.

Figure 8: The gripper trajectory of a controller without trajectory correction (left) and with trajectory correction (right). TABLE II: Simulation Result Controller Configuration | Success | Fail | Success%
---|---|---|---
Open Loop ($u(t)=u^{*}(t)$) | 1 | 299 | $0.33\text{\,}\mathrm{\char 37\relax}$
Posture Control and TC = 0 | 83 | 217 | $27.67\text{\,}\mathrm{\char 37\relax}$
Posture Control and TC = 1 | 274 | 26 | $91.33\text{\,}\mathrm{\char 37\relax}$

##  VI Discussion and Conclusion

The motion planning algorithm for a flying trapeze act has been designed for
the two-link free-flying robot in this paper. The hybrid dynamics of the robot
are derived and used in motion planning, controller synthesis, and simulation.
The uncontrollability of the center-of-mass of the multibody system has been
overcome by correcting the old trajectory in mid-air. By doing that, the
“success region” of the releasing angular position and the releasing angular
velocity has increased substantially. The simulation results also reveal that
the controller with all functionalities enabled yield
$91.33\text{\,}\mathrm{\char 37\relax}$ success rate based on 300 simulations
with uncertain parameters.

This paper also shows how the initial releasing condition affects the act. As
shown in Fig. 7, the region with the disabled trajectory correction
($\text{TC}=0$) resulted in a smaller area than the enabled one
($\text{TC}=1$). However, when looking at the nominal releasing condition, it
is vastly biased compared to the whole region. Therefore, if the performance
of the gripper is modeled, the releasing condition can be placed accordingly
to increase the possibility of performing a successful attempt.

The control algorithm presented in this paper used the trajectory correction,
which requires a more powerful processor to solve non-linear constrained
optimization problems, which may be potentially more costly and more
challenging to design and protect the physical hardware. Another potential
setback is that the initial planning may take considerable time to find a
feasible trajectory. Since the trapeze act is often a prepared routine act,
the result from the previous computation can be reused, which the computation
time can be neglected. However, unlike the previous research, the proposed
method incorporates various adjustable physical parameters such as target’s
position, joint limits, velocity limit, actuator limit, and angle of attack.
In addition, the formulation of the proposed algorithm also allows the
possibility of extension to a higher degree of freedom and flexibility in
robot configuration, such as the gripper position and opening.

## References

  * [1] M. T. Pope, S. Christensen, D. Christensen, A. Simeonov, G. Imahara, and G. Niemeyer, “Stickman: Towards a human scale acrobatic robot,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, May 2018, pp. 2134–2140. [Online]. Available: https://doi.org/10.1109/icra.2018.8462836
  * [2] J. Nakanishi and T. Fukuda, “A leaping maneuvre for a brachiating robot,” in _Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065)_. IEEE, 2000, pp. 2822–2827. [Online]. Available: https://doi.org/10.1109/robot.2000.846455
  * [3] D. Wan, H. Cheng, G. Ji, and S. Wang, “Non-horizontal ricochetal brachiation motion planning and control for two-link bio-primate robot,” in _2015 IEEE International Conference on Robotics and Biomimetics (ROBIO)_. IEEE, Dec. 2015, pp. 19–24. [Online]. Available: https://doi.org/10.1109/robio.2015.7407033
  * [4] S. Dubowsky and E. Papadopoulos, “The kinematics, dynamics, and control of free-flying and free-floating space robotic systems,” _IEEE Transactions on Robotics and Automation_ , vol. 9, no. 5, pp. 531–543, 1993. [Online]. Available: https://doi.org/10.1109/70.258046
  * [5] C. Hargraves and S. Paris, “Direct trajectory optimization using nonlinear programming and collocation,” _Journal of Guidance, Control, and Dynamics_ , vol. 10, no. 4, pp. 338–342, Jul. 1987. [Online]. Available: https://doi.org/10.2514/3.20223
  * [6] A. J. Barry, P. R. Florence, and R. Tedrake, “High-speed autonomous obstacle avoidance with pushbroom stereo,” _Journal of Field Robotics_ , vol. 35, no. 1, pp. 52–68, Sep. 2017. [Online]. Available: https://doi.org/10.1002/rob.21741
  * [7] J. Moore, K. C. Wolfe, M. S. Johannes, K. D. Katyal, M. P. Para, R. J. Murphy, J. Hatch, C. J. Taylor, R. J. Bamberger, and E. Tunstel, “Nested marsupial robotic system for search and sampling in increasingly constrained environments,” in _2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_. IEEE, Oct. 2016. [Online]. Available: https://doi.org/10.1109/smc.2016.7844578
  * [8] S. Farzan, A.-P. Hu, E. Davies, and J. Rogers, “Feedback motion planning and control of brachiating robots traversing flexible cables,” in _2019 American Control Conference (ACC)_. IEEE, Jul. 2019. [Online]. Available: https://doi.org/10.23919/acc.2019.8814894
  * [9] X. Xin, T. Mita, and M. Kaneda, “The posture control of a two-link free flying acrobot with initial angular momentum,” _IEEE Transactions on Automatic Control_ , vol. 49, no. 7, pp. 1201–1206, Jul. 2004. [Online]. Available: https://doi.org/10.1109/tac.2004.831093


0

#  Modern Methods of Text Generation

National Research University Higher School of Economics
Moscow, Russia

Dimas Munoz Montesinos

(Aug 31th, 2020)

## Abstract

Synthetic text generation is challenging and has limited success. Recently, a
new architecture, called Transformers, allow machine learning models to
understand better sequential data, such as translation or summarization. BERT
and GPT-2, using Transformers in their cores, have shown a great performance
in tasks such as text classification, translation and NLI tasks. In this
article, we analyse both algorithms and compare their output quality in text
generation tasks.

###### Contents

  1. 1 Introduction
  2. 2 Models
    1. 2.1 BERT
    2. 2.2 GPT-2
  3. 3 Experiments
    1. 3.1 Infer masked token
    2. 3.2 Question-answering
    3. 3.3 Conditional text generation
  4. 4 Models comparison
    1. 4.1 Architecture and pre-training
    2. 4.2 Performance
  5. 5 Related models
  6. 6 Conclusions
  7. A Appendix: Experiments results
    1. A.1 Infer masked token
    2. A.2 Question-answering
    3. A.3 Conditional text generation
  8. B Appendix: Simple Transformer model

##  1 Introduction

Natural Language Processing (NLP) is a large field where we can find different
tasks: Text Classification, Named Entities Recognition, Language Translation…
These tasks have a common challenge: texts written with human languages
(usually unstructured texts). The task that concerns us in this article is
text generation using a Conditional Language Model and the novel Transformers
architecture.

In order to understand Text Generation, it is necesary to define what is a
Language Model (LM). From Wikipedia, “a Statistical Language Model is a
probability distribution over sequences of words, such that, given a sequence
of length m, it assigns a probability $P(w_{1},\ldots,w_{m})$ to the whole
sequence.”. In consequence, we can use a Conditional LM to find the
probability of the next word in a sequence: $P(w_{m+1}|w_{1},\ldots,w_{m})$.

In this article, we assume that you have fundamental knowledge of deep
learning, word vectors and embedding space. Nevertheless, here we describe
some models and techniques which are relevant to understand Transformer-based
models.

#### Seq2seq models

During a long time, Conditional Text Generation was based in Seq2Seq models
[1]. The idea behind Seq2Seq consists of 2 Recurrent Neural Networks (RRN)
that try to predict the next state sequence from the previous one (the two
RNNs receive the names encoder and decoder respectively). Two of the most
extended RNNs were LSTM, introduced in 1997 by Sepp Hochreiter and Jurgen
Schmidhuber [2], and GRU, introduced in 2014 by Junyoung Chung et al [3].
However, texts generated with RNNs are far from being perfect: they tend to be
nonsense and sometimes they include spelling mistakes. Basically, one wrong
prediction has the potential to make the entire sentence meaningless.
Furthermore, it is not possible to apply parallelization since the RNNs need
to process data as a sequence.

Figure 1: This is the architecture of a Seq2seq model: the first RNN is called
encoder and the second one is decoder. In this case, the model receives an
input sentence “ABC” and produces “XYZ” as the output sentence (the input and
output may have different lengths).

#### Contextualized word embeddings

Traditionally, a model received a sequence of tokens (usually words) and
transformed them into static vectors. In short, they are simply vectors of
numbers that represent the meaning of a word. One widely extended model is
Word2vec (introduced by Mikolov et al. in 2013) which computes the static
vector of each token (the vectors are called embeddings) [4]. Furthermore,
vectors of Word2vec provided State-Of-The-Art (SOTA) performance in syntactic
and semantic word similarities.

The power of using word vectors is that they lend themselves to mathematical
operators. For example, we can add and subtract vectors:

$king-man+woman=queen$

The recent techniques consists of incorporating context into word embeddings:
replacing static vectors with contextualized word representations has led to
significant improvements on virtually every NLP task. ELMo introduced this
kind of word embeddings in 2018 [5]: “vectors are learned functions of the
internal states of a deep bidirectional language model (biLM)”. This is one of
the breakthroughs which will lead models to further understanding of words.
For example, they may difference homonyms (e.g. rock can be a stone or a music
genre) instead of having the same static vector for them.

#### Transformers

Google’s researches released a new model called Transformer in 2017 in the
paper “Attention Is All You Need” [6]. Very briefly, this architecture
consists of self-attention and point-wise, fully connected layers (see Figure
2). Similarly to what we describe for Seq2seq, Transformers include an
encoder, decoder and a final linear layer.

Figure 2: Transformer-model architecture described in “Attention Is All You
Need” [6].

These models are designed to handle sequences of data (especially useful in
NLP). Note that, in contrast to Seq2seq they do not contain recurrence or
convolution, so they do not require to process sequences in order. This fact
allows us to parallelize (much more than RNNs) and reduces training time.

##  2 Models

###  2.1 BERT

####  2.1.1 Description

Bidirectional Encoder Representations from Transformers (commonly known by its
abbreviated form BERT), as the name suggests, “is designed to pretrain deep
bidirectional representations from unlabeled text by jointly conditioning on
both left and right context in all layers” [7]. The model architecure is a
technical innovation that implements multiple layers of Transformer-encoders
to language modelling.

The authors of BERT describe two steps in their framework: pre-training and
fine-tuning (see Figure 3).

  1. 1.

During pre-training, the model is trained on unlabeled data over different
pre-training tasks: Masked LM and Next Sentence Prediction. The authors
pretrained BERT on the BooksCorpus ($800M$ words) and Wikipedia ($2,500M$
words).

  2. 2.

During fine-tuning, the model is initialized with the pre-trained parameters
(this is also known as transfer learning). Then, each downstream task is fine-
tuned separately. This process is simple, so it is not described in this
article.

Figure 3: Overall pre-training and fine-tuning procedures described in BERT’s
paper [7].

####  2.1.2 Input representation

In order to make BERT handle a variety of downstream tasks, the authors
defined three different inputs which allow us to unambiguously represent both
a single sentence and a pair of sentences.

One of these inputs is the tokenized text using a technique called WordPiece
tokens (i.e. sub-words units) [8]. Apart from the sub-words units, the authors
introduced two new tokens that must be appended to the input sentences:
$[CLS]$ in the beginning of the input and $[SEP]$ after each sentence.

The second and third inputs are sequences of $0s$ and $1s$. One (called Token
Type IDs or Segment IDs) indicates that a token belongs to the sentence A (a
series of $0s$) or the sentence B (a series of $1s$). The other (called Mask
IDs) is used when input texts are padded to same length (indicates whether the
text is padded from a certain position).

Figure 4: Example of BERT’s input with a single sentence. If there were two
different sentences in the input, there would be $1s$ in the segments sequence
(starting from the position of the token $[SEP]$).

As you can see in Figure 4, not all tokens are words, thus it is easier to
handle unknown words. Word pieces are very powerful in the sense that tokens
cover all the word (even the words that do not occur in the dictionary) and we
do not lose information (since all subword units are in the input).

####  2.1.3 Pre-training tasks

Masked Language Model (MLM). In this task, authors masked $15\%$ of WordPiece
tokens (in each sequence of the dataset at random) and then predict those
masked tokens. They only mask $15\%$ of tokens since the token $[MASK]$ is not
used in fine-tuning and they may create a mismatch between pre-training and
fine-tuning.

Next Sentence Prediction (NSP). In words of the authors, “tasks such as
Question Answering (QA) and Natural Language Inference (NLI) are based on
understanding the relationship between two sentences, which is not directly
captured by language modeling”. In order to train a model that understands
sentence relationships, given a pair of sentences, the model should predict if
the second sentence is the subsequent sentence in the original document. The
authors built a dataset where $50\%$ is the actual next sentence and $50\%$ is
a random sentence from a monolingual corpus.

###  2.2 GPT-2

####  2.2.1 Description

Generative Pretrained Transformer 2, known by GPT-2, is a large unsupervised
transformer-based language model and the successor to GPT [9]. GPT-2 was
introduced in June 2018 by researchers from OpenAI in their paper “Language
Models are Unsupervised Multitask Learners” [10].

GPT-2 consists of solely stacked decoder blocks from the transformer
architecture. In the vanilla transformer architecture, the decoder is fed a
word embedding concatenated with a context vector, both generated by the
encoder. In GPT-2 the context vector is zero-initialized for the first word
embedding. Furthermore, in the vanilla transformer architecture self-attention
is applied to the entire surrounding context (e.g. all of the other words in
the sentence), but in GPT-2 masked self-attention is used instead: the decoder
is only allowed (via obfuscation masking of the remaining word positions) to
glean information from the previous words in the sentence (plus the word
itself). Besides this, GPT-2 is a close copy of the vanilla transformer
architecture and very similar to its predecessor GPT.

Authors of GPT-2 trained it with a simple objective: given some text, predict
the next word. For this purpose, they used around 40GB of crawled data from
internet. Also, in a similar way as BERT, they fine-tuned GPT-2 in downstream
tasks to analyse the model performance in different situations.

Figure 5: GPT architecture described in “Improving Language Understanding by
Generative Pre-Training” [9] (transformer and training objectives are on the
left, and the input transformations for fine-tuning are on the right). In
GPT-2, authors moved the layer normalization to the input of each sub-block
and also added another layer normalization after the final self-attention
block.

####  2.2.2 Input representation

GPT-2 uses Byte Pair Encoding (BPE) as input representation [11]. This
technique allows us to combine the empirical benefits of word-level LMs with
the generality of byte-level approaches. BPE is a simple compression method in
which the most common pair of consecutive bytes of data is replaced with a
byte that does not occur within that data. For instance, given the sequence
$aaabdaaabac$ and $Z=aa$, then we obtain $ZabdZabac$.

It is important to highlight the fact that the authors do not apply any kind
of pre-processing to the data (e.g. lowercasing, tokenization or out-of-
vocabulary tokens). They believe that “a general language model (LM) should be
able to compute the probability of (and also generate) any string”.

##  3 Experiments

###  3.1 Infer masked token

As we have explained before, one of the pretraining tasks of BERT was MLM so,
for this experiment, we only need a pretrained BERT model and
HuggingFace111https://huggingface.co/models already provides a wide variety of
pretrained and fine-tuned models in PyTorch or TensorFlow. In particular, we
chose BERT multilingual base model which is a pretrained model on the top 104
languages with the largest Wikipedia using a MLM objective. Then, we apply the
softmax function to normalize the output and pick the top words.

Since this experiment is really simple and does not require any further
training, we proceeded to test it out.

Figure 6: Given the sentence “This car is very [MASK]” in Spanish, it returns
words such as popular ($59.34\%$), simple ($3.47\%$) or uncommon ($2.98\%$).
In this case, all top-5 words make sense. Figure 7: Given the sentence “I
think that Nastya is a very [MASK] person” in Russian, it returns words such
as young ($54.84\%$), big ($20.15\%$) or great ($5.03\%$). In this context,
the last word (new) does not make sense.

Despite infered words are correct in many cases, there are other cases where
the word suggestions are far to be good. For instance, the first word of
“Today is [MASK].” (written in English) is “demolished” (it is gramatically
correct, but it is very uncommon). However, if we use a only-English
pretrained model (we used BERT base model), the first output is “closed”.
Also, in non-English languages, sometimes it is unable to return a word but
punctuation symbols.

We believe that we must use monolingual models to obtain good results. In
other words, we must train one BERT model per language. There are a wide list
of arguments for this which are described in the conclusions section.

You can find some infered masked tokens of our experiments in the appendix.
There are a wide variety of tests in three languages: English, Spanish and
Russian.

###  3.2 Question-answering

In Question-Answering tasks, the model receives a text (called context) and a
question regarding to the context, and it should mark the inital and ending of
the answer in the context (n)ote that the model does not generate text).

Figure 8: Sample question-answering to Napoleon’s biography (context extracted
from Wikipedia). The answer is good but it is not stricly accurate: it should
be only “1804” since we did not ask for the period.

Similarly to the previous experiment, we import BERT multilingual base model
from HuggingFace. Then, we add two fully-connected layers to obtain the
initial and ending token positions from the context (see Figure 9). Based on
these positions, we pick a portion of context and return it as the answer to
the question. However, we cannot test it without previous training (fine-
tuning BERT and adjust weights of fully-connected layers). In other words, we
are using a technique called transfer learning: the model is pre-trained in a
dataset $A$ and, then, we use that pre-trained model to carry that knowledge
into solving dataset $B$.

Figure 9: Architecture of QA-BERT model.

In order to train this QA model, we use
XQuAD222https://github.com/deepmind/xquad. This dataset consists of a subset
of $240$ paragraphs and $1190$ question-answer pairs from the development set
of SQuAD v1.1 [12] together with their professional translations into ten
languages, such as Spanish and Russian.

In our experiments, answers tend to be accurate when we ask simple questions
and when we use similar words as the context (see Figure 8). We observe that
it is particularly difficult to the model to understand some synonyms and
homonyms. For example, we ask (in Spanish) “When was Napoleon crowned king of
Italy?” and “When did he become monarch?”, the first gives a good answer
“March 18th, 1805” (strictly it was on May 26th) and the second returns a
totally wrong answer “November 11th, 1799”.

You can find some questions and answers of our experiments in the appendix.
There are a wide variety of questions in three languages: English, Spanish and
Russian. Overall, the model answers correctly when we write questions in
English.

###  3.3 Conditional text generation

In this task, we experimented with GPT-2 due to BERT is not designed for text
generation. GPT-2 was only trained to predict the next token in a sentence,
but it surprisingly learned basic competence in some tasks like translating
between languages and answering questions (see Figure 10).

The RMS Titanic was visited by divers for the first time in 14 years. The
British vessel was damaged to the bow in 1912 and the ship was found to be
completely submerged. The Titanic is also the only ship that has sunk in
international waters. But, according to experts, the damage to the Titanic
would have been even worse, if the ship had not been in the water. In fact,
according to reports, the damage could have been caused by a catastrophic
failure of the […]

Figure 10: Text sample generated by GPT-2 after providing an initial text (in
bold letters).

GPT-2 has the ability to generate conditional text samples of unprecedented
quality. In the appendix, you can observe that the model is capable of
generating synthetic texts close to human quality. We provided different kind
of prompts to analyse the versatility of GPT-2 and, despite it shows coherence
and good quality (in particular, when we write about topics highly represented
in the training data), we found failures, such as sudden topic switching,
repetitive text and LM failures. For example, sometimes the model negates what
it just wrote in the previous sentences. In Figure 10, even though text
quality is good, you can observe that it also wrote the extravagant sentence
“if the ship had not been in the water”

As we explain later, GPT-2 and BERT architectures are different (the first is
a Transformer-encoder architecture and the second is a Transformer-decoder
one). This fact lead that we cannot use BERT as-it-is for text generation.
Nevertheless, there are some efforts to create new BERT-based algorithms for
this purpose (the main idea consists of adding a decoder-transformer in the
end). For example, the recent paper CG-BERT [13] looks very promising in this
task, but we could not include it in this article due to lack of time and
resources to train the model. The main idea of CG-BERT is to use half layers
of BERT as encoders and half layers as decoders.

In addition to BERT and GPT-2, we tried to implement a simple Transformer
model to be capable of generating a conversation. The result is poor since the
training data is really small as well as the resources to train the model. You
can read the details in the appendix.

##  4 Models comparison

###  4.1 Architecture and pre-training

One difference that we encounter between BERT and GPT-2 is in the
architecture: despite both are transformers-based architectures, it uses
encoders in the case of BERT and, in the case of GPT-2, decoders. This
particularity makes BERT and GPT-2 to understand text information in a
different way and, in consecuence, their performance variate depending on the
dataset.

We find another difference in the pre-training method: GPT-2 aims to predict
the next word in a sentence, while BERT is trained in NSP and MLM. We should
keep in mind these pre-training methods when we use transfer learning to fine-
tune a model.

###  4.2 Performance

In order to evaluate BERT and GPT-2, authors prepared different versions of
both models (with different number of parameters) and fine-tuned them in
different downstream tasks. One version is BERT Base, which has the same
number of parameters as GPT (the predecesor of GPT-2) for comparison purposes.
Unfortunately, we could not find common benchmarks between BERT and GPT-2 in
the same datasets, so we only can compare BERT and GPT in GLUE and SWAG tasks.

GLUE [14], which consists of a wide list of NLP tasks (sentiment analysis,
similarity, NLI, question-answering…), is one of the dataset where BERT was
fine-tuned. In this dataset, BERT Base already outperformed GPT (and prior
approaches such as BiLSTM+ELMo) and BERT Large got an even better
result333GLUE leaderboard: https://gluebenchmark.com/leaderboard, thus it
achieved the status SOTA in this collection of tasks.

SWAG [15] consists of 113k multiple choice questions about grounded
situations: given a question/situation, we have to pick the correct answer
among four choices. Identically as GLUE, BERT outperformed GPT (BERT Large
improved GPT score by $8.3\%$). It is also surprising that BERT Large got a
better score than a human expert in a small subset of $100$ questions.

##  5 Related models

The models described in this article, BERT and GPT-2, demonstrate the benefits
of large scale language modeling. Both papers leverage advances in compute and
available text corpora to significantly surpass state-of-the-art performance
in Natural Language Understanding (NLU), modeling, and generation. Currently,
there are new models (based on BERT and GPT-2 architectures) which get even
better results. Some of them are RoBERTa (robustly optimized version of BERT
with larger training data) [16], ALBERT (lite version of BERT) [17],
DistilBERT (also a reduced version of BERT) [18] and StructBERT (incorporate
language structures into BERT pre-training) [19] 444In SQuAD, ALBERT has the
SOTA status: https://rajpurkar.github.io/SQuAD-explorer/. 555In GLUE,
StructBERT achieves the second best score:
https://gluebenchmark.com/leaderboard 666In SWAG, RoBERTa is in the second
position in the leaderboard:
https://leaderboard.allenai.org/swag/submissions/public.

Appart from the mentioned BERT-based models, Nvidia published Megatron-LM [20]
in 2019. In a few words, it is a simple and efficient model parallel approach
by modificating existing PyTorch transformer implementations. They released a
$3.9$ billion parameters version of BERT and $8.3$ of GPT-2. Both Megatron
models got better scores than both vanilla BERT and GPT-2 models in the same
tasks.

Figure 11: Evolution of number of parameters described by authors of
DistilBERT [18].

Additionally, OpenAI published recently a new model called GPT-3 (it uses the
same architecture as GPT-2) [21]. The largest version of this novel model has
$175$ billion parameters which is 10 times more than any previous non-sparse
language mode. However, despite the strong quantitative and qualitative
improvements of GPT-3, particularly compared to its direct predecessor GPT-2,
it still has notable weaknesses in text synthesis and several NLP tasks (they
are extensively described in the paper). Unfortunately, we could not include
text samples generated by GPT-3 in this article due to OpenAI has not released
the model yet.

##  6 Conclusions

Transformers disrupted sequence-based deep learning significantly. The two
variants of transformer models that we showed, BERT and GPT-2, outperform
previous approaches with RNNs. In both cases, models take advantage of
Attention layers, which selectively weight different elements in input data,
and can achieve state-of-the-art performance on many language modelling
benchmarks.

Although there are some situations where the output of Transformers-based
models is close to the human quality, we find that the output quality depends
on how familiar is the model with the topic. For instance, GPT-2 is not able
to generate a high-quality text when we provide an uncommon context (such as
the recent COVID-19 pandemic or a new popular person). In the case of BERT, we
observe that the model tend to answer incorrectly (in question-answering
tasks) when we use homonyms or synonyms instead of the same words as the
context. These experiments show us that still computers are distant from fully
understanding of unstructured texts written in human languages.

We observe a trend to build larger models in the NLP field, so they can
capture more information from unstructured texts. Furthermore, unlikely
humans, they need to be trained with tons of data. In terms of zero-shot or
one-shot tasks, we need to improve the efficiency of NLP algorithms since they
see much more text than a human sees in their lifetime.

In general, we observe that despite these models achieve better performance
than us in specific tasks, the common sense of humans produces better results
rather than any deep learning model. Also, we cannot ignore the fact that we
learn from our daily life, but it is difficult to measure what type of data
should we provide to a model (during pre-training) to get a similar knowledge.
This is the most likely reason deep learning models fail to produce coherent
texts.

In addition to the previous points, we observe a growing need to pre-train a
model per language (BERT, GPT-2 or any other model). Apart from the grammar
rules of each language (or, for example, the evident difference of RTL and LTR
languages), there is knowledge which is intrinsic to the language: polite
expressions, informal style… If we want to obtain human-quality in text
generation, we will need large dataset not only in English but also in other
languages (we cannot use translated data).

In summary, we will see more transformer-based models in the future. They have
demonstrated superiority in parallel computation and modelling long-range
dependencies, compared to RNNs (such as LSTM). Still, we do not know which is
the best approach to pre-train them as well as how to reproduce the human
common sense (this is mandatory in text generation). Thus it is very likely
that we are going to see not only new pre-training methods but also larger
models.

##  References

  * [1] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le.  Sequence to sequence learning with neural networks, 2014.
  * [2] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural Computation, 9(8):1735–1780, 1997.
  * [3] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling, 2014.
  * [4] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space, 2013.
  * [5] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations, 2018.
  * [6] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2017.
  * [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.
  * [8] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean.  Google’s neural machine translation system: Bridging the gap between human and machine translation, 2016.
  * [9] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training, 2018.
  * [10] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners, 2010.
  * [11] Philip Gage.  A new algorithm for data compression.  C Users J., 12(2), 1994.
  * [12] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  SQuAD: 100,000+ questions for machine comprehension of text, 2016\.
  * [13] Congying Xia, Chenwei Zhang, Hoang Nguyen, Jiawei Zhang, and Philip Yu.  Cg-bert: Conditional text generation with bert for generalized few-shot intent detection, 2020.
  * [14] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding, 2018.
  * [15] Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi.  Swag: A large-scale adversarial dataset for grounded commonsense inference, 2018.
  * [16] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, 2019.
  * [17] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations, 2019.
  * [18] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.
  * [19] Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Jiangnan Xia, Liwei Peng, and Luo Si.  Structbert: Incorporating language structures into pre-training for deep language understanding, 2019.
  * [20] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.  Megatron-lm: Training multi-billion parameter language models using model parallelism, 2019.
  * [21] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.

##  Appendix A Appendix: Experiments results

###  A.1 Infer masked token

Despite some options could fit into philosophical or rare situations, we
believe that they are uncommon options to replace the masked token. It also
occurs that some options are grammatically correct, but they do not fit in the
context of the sentence. In both cases, we note them as “incorrect”.

English.

Sentence |  Options |  Notes
---|---|---
Hello, I’m a [MASK] model. |  • model $6.66\%$ • real $4.59\%$ • business $3.30\%$ • mathematical $3.16\%$ • new $2.78\%$ |  All correct.
Today is [MASK]. |  • demolished $19.43\%$ • unknown $8.44\%$ • closed $5.93\%$ • abandoned $2.82\%$ • active $2.72\%$ |  Only the third option, closed, seems correct.
[MASK] is a good idea. |  • It $48.57\%$ • This $30.64\%$ • That $7.16\%$ • There $2.25\%$ • it $0.70\%$ |  All correct.
The doctor ran to the emergency room to see [MASK] patient. |  • the $88.97\%$ • a $4.84\%$ • another $2.83\%$ • his $1.07\%$ • her $0.76\%$ |  All correct.
The doctor ran to the emergency room to see the [MASK]. |  • incident $4.33\%$ • accident $3.85\%$ • explosion $3.77\%$ • scene $2.64\%$ • crash $2.49\%$ |  All correct.

Russian.

Sentence |  Options |  Notes
---|---|---
Я считаю, что Настя очень [MASK] человек. |  • молодой $54.84\%$ • большой $20.15\%$ • великий $5.03\%$ • лучший $4.91\%$ • новый $1.64\%$ |  All correct except the last option ( новый).
Я очень люблю [MASK]. Я рисую днями и ночами. |  • музыку $13.82\%$ • книги $3.11\%$ • жизнь $2.64\%$ • людей $2.58\%$ • игру $2.34\%$ |  All incorrect.
Он обладает многими [MASK]. |  • медалями $44.46\%$ • орденами $18.62\%$ • людьми $12.01\%$ • словами $5.61\%$ • результатами $3.22\%$ |  Only the first (медалями) and the second (орденами) options are correct.

Spanish.

Sentence |  Options |  Notes
---|---|---
Ayer estuve paseando y me encontré con [MASK]. |  • #migo $34.68\%$ (conmigo) • él $14.68\%$ • ella $7.43\%$ • ellos $3.16\%$ • Dios $2.60\%$ |  All correct except the first option (conmigo).
Si tuviera [MASK] podría irme de vacaciones. |  • dinero $14.99\%$ • , $6.45\%$ • tiempo $5.54\%$ • años $4.50\%$ • problemas $2.79\%$ |  The first three options are correct.
Necesito [MASK] para aprobar el examen. |  • tiempo $39.88\%$ • dinero $4.38\%$ • pruebas $3.30\%$ • información $3.10\%$ • recursos $2.57\%$ |  The first (tiempo) and the last two options (información and recursos) are correct.
Voy a ir con mis amigos al [MASK]. |  • colegio $7.99\%$ • pueblo $5.74\%$ • mar $3.19\%$ • mundo $2.57\%$ • hotel $2.16\%$ |  All correct except the forth option (mundo).

###  A.2 Question-answering

English. The context is the Napoleon’s biography extracted from
Wikipedia777https://en.wikipedia.org/wiki/Napoleon.

Napoleon Bonaparte (15 August 1769 - 5 May 1821) was a French statesman and
military leader who became notorious as an artillery commander during the
French Revolution. He led many successful campaigns during the French
Revolutionary Wars and was Emperor of the French as Napoleon I from 1804 until
1814 and again briefly in 1815 during the Hundred Days. Napoleon dominated
European and global affairs for more than a decade while leading France
against a series of coalitions during the Napoleonic Wars. He won many of
these wars and a vast majority of his battles, building a large empire that
ruled over much of continental Europe before its final collapse in 1815. He is
regarded as one of the greatest military commanders in history, and his wars
and campaigns are studied at military schools worldwide. Napoleon’s political
and cultural legacy has made him one of the most celebrated and controversial
leaders in human history.

Question |  Answer |  Notes
---|---|---
When did Napoleon become Emperor? |  1804 until 1814 |  Correct, but it should answer only 1804.
Who was Napoleon Bonaparte? |  Napoleon Bonaparte (15 August 1769 - 5 May 1821), born Napoleone di Buonaparte |  Incorrect.
Where are studied his wars and campaigns? |  military schools worldwide |  Correct.
What is studied about him? |  military schools worldwide |  Incorrect.
How long Napoleon dominated Europe? |  for more than a decade |  Correct.

Russian. Context extracted from Wikipedia about Higher School of
Economics888https://ru.wikipedia.org/wiki/%D0%92%D1%8B%D1%81%D1%88%D0%B0%D1%8F_%D1%88%D0%BA%D0%BE%D0%BB%D0%B0_%D1%8D%D0%BA%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D0%BA%D0%B8.

Национальный исследовательский университет «Высшая школа экономики» (НИУ ВШЭ;
Вышка) — автономное учреждение, федеральное государственное высшее учебное
заведение. ВШЭ создана в 1992 году, нынешний статус носит с 2009 года.
Основной кампус находится в Москве, ещё три — в Нижнем Новгороде, Санкт-
Петербурге и Перми.

Question |  Answer |  Notes
---|---|---
Когда была основана Вышка? (When was HSE founded?) |  1992 году (1992 year) |  Correct answer, but it is gramatically incorrect.
Где расположены кампусы? (Where are the campuses located?) |  Москве (Moscow) |  Partial answer.
Что такое Высшая школа экономики? (What is the Higher School of Economics?) |  Национальный исследовательский университет (National Research University) |  Correct, but it is not accurate.
Что такое аббревиатура Высшей школы экономики? (What is the acronym for the Higher School of Economics?) |  Национальный исследовательский университет «Высшая школа экономики» (National Research University Higher School of Economics) |  Incorrect.

Spanish. Description about Sagrada Familia extracted from
Wikipedia999https://es.wikipedia.org/wiki/Templo_Expiatorio_de_la_Sagrada_Familia.

El Tempo Expiatorio de la Sagrada Familia (en catalán, Temple Expiatori de la
Sagrada Família), conocido simplemente como la Sagrada Familia, es una
basílica católica de Barcelona (España), diseñada por el arquitecto Antoni
Gaudí. Iniciada en 1882, todavía está en construcción. Es la obra maestra de
Gaudí, y el máximo exponente de la arquitectura modernista catalana. Es uno de
los monumentos más visitados de España, junto al Museo del Prado y la Alhambra
de Granada, y es la iglesia más visitada de Europa tras la basílica de San
Pedro del Vaticano. Cuando esté finalizada será la iglesia cristiana más alta
del mundo.

Question |  Answer |  Notes
---|---|---
¿Cómo se escribe en catalán? (How is it written in Catalan?) |  Temple Expiatori de la Sagrada Família |  Correct
¿Quién diseñó la iglesia? (Who designed the church?) |  Antoni Gaudí |  Correct
¿Cuándo comenzó la construcción de la Sagrada Familia? (When did the construction of Sagrada Familia began?) |  1882 |  Correct
¿Qué lugar cristiano recibe más visitas que la Sagrada Familia? (Which christian place receives more visits than Sagrada Familia?) |  It outputs the whole context. |  Incorrect
¿Qué otros monumentos son muy visitados por turistas en España? (Which other monuments are highly visited by tourists in Spain?) |  Museo del Prado y la Alhambra de Granada |  Correct

###  A.3 Conditional text generation

The following synthetic texts were generated by GPT-2. We fed the model with
an initial text and then we let GPT-2 to finish it. These text samples are
written only in English due GPT-2 end up in nonsense outputs using other
languages. Note that the model does not give the same output everytime we run
it, so you might obtain a different output.

Last week, many people reported that they saw a unicorn at the park. It turned
out to be a real unicorn, but it was just a fake. The unicorn was made of foam
and plastic. The foam was made of polystyrene, which is a plastic that is
commonly used in construction materials. The fake unicorn was made of foam,
which has been known to be a very effective material for creating fake
animals, such as this one. The foam was made of polystyrene, […]

In the last months, a new virus spread worldwide and changed our daily lifes.
This new virus, called Ebola, appeared in three countries, and has killed more
than 3,000 people in the four countries: Guinea, Liberia and Sierra Leone. The
virus is spread through direct contact with the fluids of infected persons,
which can include body fluids, such as blood, sweat and tears… It is also
spread through contaminated food and water. When Ebola arrives in a country, a
lot of people […]

Q: How would you describe to Maria? A: I’d say… I’d say that she ’s like very
strong and very resilient. Q: Thank you very much. A: Oh, thanks for the offer
to talk.

Q: What do you think about the death penalty? A: I am not an advocate for the
death penalty. I am not someone who believes that this life should be taken
and then taken over again, which of course is the case with every case when we
see a murder. It is my belief that a fair penalty should be used, and that if
the evidence was not sufficient to prove that the person committed the actual
crime, then there should be a life sentence.

Teacher: Good morning everyone. Students: Good morning teacher. Teacher: It’s
my pleasure to welcome you all here today. Students: Yeah. Teacher: I want to
talk about some things that you all do. Students: Yeah. Teacher: You guys
always seem to have such great enthusiasm. Students: Yeah. Teacher: So, we’ll
start with our first topic. What’s your favorite type of book to read?

##  Appendix B Appendix: Simple Transformer model

We implemented a simple model (see Figure 12) to figure out whether we are
able of generating text based on training data. For this purpose, we took an
English dialogue written by Shakespeare101010We took the dialogue from this
TensorFlow tutorial:
https://www.tensorflow.org/tutorials/text/text_generation.

Figure 12: Visual description of our Transformer model.

The model consists of 2 transformer-encoders and 2 transformer-decoders to
capture the text information, and a fully-connected layer to obtain the
prediction for the next token in the text. We kept the model simple due to two
reasons:

  * •

We do not have enough resources to train a large model like BERT or GPT-2.

  * •

If the model was larger, we would also need a bigger dataset to train all its
parameters.

Unfortunately, the model predictions are really bad in the sense that it
outputs nonsense tokens to continue a text. Here you can see the sample text
that the model generated from the initial text “ROMEO:”. It is very likely
that we need to increase the model size to better capture the text information
(in consecuence, we need to use more data to train it) and, therefore,
generate a coherent text.

ROMEO: took perform still sets big,KATHARINA’d me my heart l belie to
harpets’d upon me And sure darkness under’d statutes. deceive them or orderly
and likely body will f roodFor Masteredon’d conclude else aged asTherefore The
suffer hiswittedWould being royalties and to ouroan as which and than him The
goddess wasYour things curst hag that they admirings that how of they twere to
spirit: linger OF glory of follow ised greatestar