# Teaching Drones on the Fly: Can Emotional Feedback Serve as Learning Signal
for Training Artificial Agents?

Manuela Pollak, Andrea Salfinger, Karin Anna Hummel

###### Abstract

We investigate whether _naturalistic emotional human feedback_ can be directly
exploited as a _reward signal_ for training artificial agents via _interactive
human-in-the-loop reinforcement learning_. To answer this question, we devise
an experimental setting inspired by animal training, in which human test
subjects interactively teach an emulated drone agent their desired command-
action-mapping by providing emotional feedback on the drone’s action
selections. We present a first empirical proof-of-concept study and analysis
confirming that human facial emotion expression can be directly exploited as
reward signal in such interactive learning settings. Thereby, we contribute
empirical findings towards more naturalistic and intuitive forms of
reinforcement learning especially designed for non-expert users.

##  1 Introduction

We study the research question _can emotional feedback serve as learning
signal for intuitively training artificial agents_ on the use case of human-
drone interaction. We present an experimental setting in which a user
intuitively teaches a drone the meaning of commands such as gestures that can
be detected by vision-based recognition (Minh Dang et al. 2020; Hummel,
Pollak, and Krahofer 2019). Learning is based on emotional feedback, a
_naturalistic mode of interaction_ humans are familiar with from conventional
human-to-human and human-to-animal interactions.

Learning Approach. We establish an analogy to _animal training_ , where
animals are instructed to learn the correct action for a given command. Here,
learning is based on evaluative feedback from the animal’s instructor,
traditionally in form of tangible rewards such as the animal’s preferred food
(Vieira de Castro et al. 2021), but often also in form of the instructor’s
emotional response, especially for social animals such as canines. Over time,
the animal learns to correlate the received reward with the correct action.
Similarly, an artificial agent – here, a drone – should interpret the
emotional feedback of the user to learn.

The ability to process emotional cues can be considered as a key requirement
for any socially intelligent agent. At the same time, the intuitiveness of
emotional feedback allows to include users with little technical background in
the training/rewarding task. Thus, we believe that providing a method to
interpret affective evaluative human feedback represents a cornerstone in
social human to machine interaction.

Figure 1: The _interactive machine learning loop_ of our drone training
experiment: The user issues a gesture command $c$ upon which the drone chooses
an action $a$. The user provides emotional feedback which the drone interprets
to derive a reward. In this contribution, we specifically focus on deriving
rewards from interpreting naturalistic emotional feedback.

Contributions. (i) We present the design and implementation of a real-time
interactive machine learning (IML) system based on reinforcement learning
(RL), where rewarding is provided by human emotional feedback. We envision an
artificial agent that makes use of a camera and facial emotion recognition and
exemplify the agent by a small indoor drone that requires to learn a command-
action mapping. Among the core features of the system, a _rewarding algorithm_
is developed that converges fast and allows to cope with different
expressiveness of individual facial emotions. (ii) To study whether emotional
feedback can provide a useful learning signal for the drone, we conduct an
experimental user study, where we mimic the drone’s actions upon receiving a
command as sketched in Fig. 1. Our first results indicate that the approach is
feasible.

##  2 Background and Related Work

We first motivate why emotions play an important role in human-robot
interaction (HRI), summarize major results on the role of feedback in agent
learning, and give an overview of emotion recognition as feedback signal and
direct, naturalistic form of interaction.

The Role of Emotions in HRI. A large fraction of research studies on the role
of emotions in HRI settings has focused on investigating the human’s
preferences of interaction modalities to better understand the robot’s
behavior, such as the studies described in (Reyes, Meza, and Pineda 2019;
Thunberg and Ziemke 2021). Another field of study examines how typically
humanoid robots’ emulated emotion expressions affect human performance. For
example, studies on robot companions for children investigated the effect of
the robot’s displayed emotional response as a _feedback and motivation signal_
for children’s learning performance (Ahmad et al. 2019).

Yet, also the artificial agents’ ability to correctly interpret and react to
human emotions has been acknowledged as a key requirement for socially
intelligent agents and human-machine interaction in the fields of cognitive
robotics, artificial intelligence (AI), and HRI, such as described in
(Chakraborti et al. 2017; Biundo and Wendemuth 2016; Hörnle et al. 2017). The
ability of a robot to infer the emotional state of a human reliably is a
desired goal. Research has targeted the creation of emotional models and
classification schemes of human emotions, and emotion recognition such as
facial emotion recognition (FER) and analysis of body posture as well as brain
activity as surveyed in (Spezialetti, Placidi, and Rossi 2020), yet only few
works have focused on the feasibility of using emotion as reliable signal for
feedback-based machine learning.

Feedback in Agent Learning. In his characterization of the twelve potential
roles that emotion could play in AI, (Scheutz 2004) pointed out the
possibility of _using emotion for learning_ , for example by using emotional
evaluations as $Q$-values in reinforcement learning (RL), as we will attempt
in this work. RL represents the machine learning paradigm employing learning
from _evaluative feedback_.111 _Evaluative_ feedback indicates how good/bad
the taken decision was. This is opposed to _imperative_ feedback that states
what the _correct_ decision would have been (i.e., the paradigm of
_supervised_ machine learning), or _descriptive_ feedback, which states _how_
the learner should adapt its behavior (Sumers et al. 2021). Classically, RL
agents compute a reward score from their interactions with the environment,
using a pre-defined reward function to provide the feedback signal for the
agent. This way, the desired behavior is reinforced by rewarding. However,
this reward function is specific for the given task, and consequently the
agent’s learning capability often hinges on the designer’s capability of
engineering an appropriate reward function for the problem at hand (Sequeira
2013; Russell 2019). To allow for more generic open-world learning, approaches
such as presented in (Thomaz and Breazeal 2008; Knox and Stone 2009;
MacGlashan et al. 2017) propose learning through _human reinforcement_ , i.e.,
enable a human tutor to assess the agent’s behavior and to provide direct
feedback. In past works, this feedback has to be provided in the form of
predefined, numeric reward scores, which is not the natural interaction style
we are aiming for.

Naturalistic Human Feedback. Human-robot interaction (HRI) can reach a next
level if an artificial agent is capable of interpreting the human’s feedback
in a more naturalistic manner. In terms of the RL paradigm, the agent should
be capable of inferring its (numeric) reward scores from humans’ natural
communication channels. For example, (Sumers et al. 2021) devised a learning
approach using _inverse RL_ to learn humans’ latent reward function from
(unconstrained, thus naturalistic) textual feedback captured during a human-
human collaborative game play setting. Other research work studied the
utilization of emotional evaluations as rewards, as (Scheutz 2004) proposed.
(Broekens 2007) indeed already experimented with incorporating humans’
emotional response as a reward signal in RL. However, in this approach
emotional feedback is used only as an _additional reward_ added on top of a
base reward signal the agent obtained from its virtual environment (thus,
essentially only makes the base reward signal more accurate). This is due to
the chosen RL algorithm based on neural networks, which requires a large
number of training iterations and hence is not amenable to continuous human
feedback.

Our approach is similar in spirit to recent endeavors on learning rewards from
_unconstrained naturalistic human feedback_ such as (Sumers et al. 2021). Yet,
we extend the aforementioned related work by utilizing the human’s emotional
response as the _sole source of feedback_ on the artificial agent’s
performance. To overcome the problem of long training efforts, we propose to
choose an RL approach well suitable for real-time learning, notably multi-
armed bandits that require only a few training iterations until convergence.

##  3 Model and Study Design

We study the feasibility of using emotional feedback to train an artificial
agent along the use case of _human-drone interaction_ , where the drone is the
artificial agent as depicted in Fig. 1. The user issues a search command by a
gesture, which results in the drone navigating to the specific object the user
searches for. The drone will learn the correct command-action mapping by
reinforcement learning based on emotional user feedback. More formally, we can
describe the system by an agent-based model.

Agent-based Model. We model the human-drone interaction as a multi-agent
system consisting of

  * •

a user agent $u\in\mathcal{U}$ that can issue a set of commands $\mathcal{C}$,

  * •

a drone agent $d\in\mathcal{D}$ that can execute a set of actions
$\mathcal{A}$,

where $|\mathcal{C}|=|\mathcal{A}|=k$, and the sets $\mathcal{U}$ and
$\mathcal{D}$ represent the sets of users and drones, respectively. During the
_training phase_ the drone agent $d$ learns which action ${a}\in\mathcal{A}$
the user agent $u$ desires when issuing a command $c\in\mathcal{C}$. The
bijective mapping function $f_{u}(c)=a$ expresses the personalized command-
action mapping for a user $u$.

Drone Learning by Emotional Rewards. Initially, the drone does not know the
command-mapping $f_{u}(c)=a$ for any command $c$ and has to learn it by
reinforcements from the respective user $u$. The drone selects an action
$a\in\mathcal{A}$ randomly (independently and identically distributed) and
awaits feedback in form of emotional feedback from user $u$, which results in
a _reward_ $r$. Eventually, the drone will learn the desired command-action
mapping function $f_{u}(c)=a$ for all commands $c$ after experiencing a
sequence of interactions with user $u$ that can be described as
$\langle(c^{0},a^{0},r^{0}),(c^{1},a^{1},r^{1}),\ldots\rangle$. Yet, the
learning outcome is also effected by learning confounders.

Learning Confounders. Naturalistic user interfaces rely on sensor-based
signals and thus require the implementation of user recognition, command
recognition, and feedback recognition and interpretation (feedback in form of
text, visual, audio, etc.). Any such recognition involving the drone hardware
is prone to classification errors, thus, impacting machine learning
performance. In order to reduce this complexity when studying our research
question, we design our experimental study to eliminate confounders, i.e.,
potential error sources that might interfere with the learning problem, here
user and gesture recognition errors and interaction with a real hardware-
embodied agent. Once we have evidence that emotional feedback is a valid
source for teaching, we will extend our study to the hardware drone.

Experimental Study Design. To study the usefulness of naturalistic emotional
feedback and the effectiveness of the rewarding algorithm without side-effects
caused by confounders related to the hardware, we create a Web-based emulated
drone agent. This way, we abstract user-drone interactions to deterministic,
error-free user login (to emulate user recognition), button-clicks (to emulate
gesture-based commands) and video replay of recorded real drone flights (to
emulate actions performed by the drone). The user trains the drone by
providing emotional feedback to the drone based on its action selection.
Further, the user provides labels for every emotional feedback in terms
“positive“ and “negative“ action assessment. Both the desired command-action
mappings and the feedback labels serve as _ground truth data_.

##  4 RL by Emotional Signals

Learning by emotional signals is based on the following steps: (i) capturing
emotional feedback as an assessment of the drone’s action choice by the
drone’s onboard camera, (ii) recognizing emotions by analyzing the user’s
facial expression on video frames resulting in an emotion distribution, and
(iii) deriving a reward score based on the emotion distribution for drone
actions to finally implement the command-action mapping $f_{u}(c)=a$,
personalized for each user $u$.

Capturing Emotional Feedback. The user will try to convey to the drone
visually how (dis)satisfied s/he was with the drone’s action choice for a
predefined feedback time interval. The drone records a video stream of this
evaluative feedback, i.e., a time series of video frames, intended as raw
input for drone learning. Each detected face is analyzed based on facial
emotion recognition. The number of frames to be considered is configurable
(sampling fewer frames may lead to missed emotions, sampling more frames leads
to higher processing effort).

Facial Emotion Recognition. We estimate the facial emotion expressions in each
frame with the facial emotion classifier provided by the fer package (Arriaga,
Valdenegro-Toro, and Plöger 2017; Arriaga et al. 2020), employing the MTCNN
face detector (Zhang et al. 2016) which has been trained on a large-scale
facial expression dataset (Goodfellow et al. 2013). For each frame, fer
predicts a discrete probability distribution characterized by the probability
vector $\mathcal{P}(E)$, which describes how likely the user’s facial
expression corresponds to each of the seven universal facial emotions
established by (Ekman et al. 1987). The emotions are defined in the set $E$,
which we classify as: positive (‘happy’), neutral (‘neutral’, ‘surprise’), and
negative (‘angry’, ‘disgust’, ‘fear’, ‘sad’) feedback.

Inferring the Agent’s Reward. Essentially, the desired _reward signal_ $r$
corresponds to the user’s _level of satisfaction_ , a latent variable that
cannot be directly accessed but is inferred from the output of facial emotion
recognition.

The RL problem requires a scalar reward value $r$. Thus, we define a mapping
of the emotion probability vector $\mathcal{P}(E)$ to $r$ that conveys the
idea of satisfaction or dissatisfaction of the user. $r$ corresponds to the
_emotional valence_ score, representing the extent to which an emotion is
positive or negative (Citron et al. 2014; Kossaifi et al. 2021). In _affective
computing_ terms, this realizes a mapping from Ekman’s basic discrete model to
the pleasure-displeasure scale of the continuous valence-arousal-dominance
model (Mehrabian and Russell 1974).

In detail, the scalar reward $r$ is calculated as the dot product between the
emotion probability vector $\mathcal{P}(E)$, and its corresponding scaling
factors vector $\mathbf{s}$:

$r=\mathcal{P}(E)\cdot\mathbf{s},$ (1)

where $\mathbf{s}$ is configured as $[$+3 (happy), 0 (neutral), +1 (surprise),
-3 (angry), -2 (disgust), -2 (fear), -3 (sad)$]^{T}$. Thus, _positive_
emotions will increase, _negative_ emotions will decrease the resulting score
$r$. Note that alternative calculations are possible, e.g., just selecting the
most probable emotion, yet this yielded worse results in our first
experiments.

Implementing Agent Learning. We formulate our learning problem as a _multi-
armed bandit (MAB) problem_ (Sutton and Barto 2018): Each command
$c\in\mathcal{C}$ represents a context,222Note that the corresponding context
is directly observable and hence does not have to be inferred, thus our
problem corresponds to a classical MAB setting involving multiple bandits
rather than a contextual/associative bandit. thus instantiates an MAB with $k$
arms. Each arm represents one particular action $a\in\mathcal{A}$, whereby we
will denote these actions by indexing their enumeration in vector
$[a_{1},\ldots,a_{k}]^{T}$. The user’s satisfaction with a chosen action $a$
is expressed by the reward $r$. The drone selects the action that is expected
to maximize its collected reward $\sum_{T}r$ over time $T$, in other words,
the action with the highest expected (reward) value.

More precisely, the MAB estimates the actions’ values as $Q_{t}(a)$ by
consistently updating the mean rewards received for each of its actions up to
its current trial step $t$.

Figure 2: Drone agent’s learning evolving over time based on the user’s
interactive feedback, ideal sample use case (taken from our study with three
possible action choices, user ID 13): thick line indicates the desired action.

Fig. 2 depicts an example from our user study for inspecting the drone’s
learning. It is shown how the drone’s action value estimates $Q_{t}(a)$ evolve
with repeated human reinforcement through emotional rewards $r$. At each trial
step (x-axis), the drone selects the action (bandit arm) with the current
maximum action value $Q_{t}(a)$ (y-axis), or randomly samples from the maximum
ones in the case of ties.333We initially implemented classical
$\epsilon$-greedy action selection: In $1-\epsilon$ % of the trials the action
with the max. $Q(a)$ is chosen (_exploitation_), in $\epsilon$ % a random
action is selected (_exploration_) to refine $Q(a)$ estimates. After initial
tests, we chose to disable _exploration_ and realize deterministic action
selection by setting $\epsilon=0$, since in our setting learning will be
actively “steered” by the user. At trial step $t=0$ for this MAB, the drone
thus selects an action randomly out of three actions. This action is not the
desired one and the negative user feedback leads to a decrease of this
action’s value estimate. At trial step $t=1$, the drone selects the correct
action out of the two actions with maximum value, leading to positive user
feedback. From this step on, the drone has learned the correct command-action
mapping.

As can be seen in this example, the MAB is initialized with action values
$Q_{0}(a)$ of _neutral valence_ (0). We also experimented with an _optimistic
initial values_ approach (Sutton and Barto 2018) where all action values are
initially set to $+5$. This forces the MAB to first try out each action at
least once to collect the user’s real feedback, even if the user’s desired
action is selected early-on, as in Fig. 2. To reduce the necessary number of
interactions, we thus decided to use values of neutral valence.

##  5 Experiments

We conduct an experimental user study in the form of a custom Web
app444https://apps.tk.jku.at/he4dc/ emulating the drone in the interactive
learning loop, see Fig. 1.

Experiment Setup. The study considers three command finger gestures and three
actions (i.e., $k=3$): finding a hammer, a book, or a bottle. The drone is
required to learn each of the command-action mappings in one teaching session.
The study participants first specify their desired command-action mapping
(_ground truth_ mapping), then the following teaching loop is performed: (1)
the user selects a gesture command; (2) the (emulated) drone chooses the
action with the maximum action value (or, if multiple actions with maximum
action value exist, chooses one of those actions randomly) and a video is
played emulating the drone’s flight towards the anticipated object to be
found; (3) the user provides real facial emotional feedback recorded by a Web
cam over a time period of 5 seconds and labels his/her emotional response as
positive or negative (_ground truth feedback_).

With this setting, we abstract from the embodied hardware agent to limit the
learning confounders and focus on the evaluation of the usefulness of the
facial emotional feedback per-se. Yet, we performed initial experiments with
our hardware drone (Parrot Anafi) to make sure that video processing (frame
selection and FER-based emotion classification) and taking a video with the
on-board camera of the drone is of comparable video quality. This is important
as we plan to extend the investigation by a lab-experiment with a real drone,
where we will study the influence of practical factors such as light
conditions and camera angle.

(a) Drone teaching success rates.

(b) Reward scores wrt. ground-truth feedback.

Figure 3: Empirical results on drone teaching success.

Video Frame Rate and Reward Calculation. The frame rate of the video stream is
25 frames per second (FPS), which is down-sampled for emotion classification
by only retaining every $j$-th frame. We set $j=12$, which makes real-time
emotion classification feasible that takes about 32-42 seconds per face. We
then compute the _average emotion probability vector_ $\overline{P(E)}$ by
computing the mean of the derived sequence of $\mathcal{P}(E)$s obtained on
the frames of the down-sampled set. The overall emotional reward $r$ is
calculated by applying Eq. 1 to $\overline{P(E)}$.

Study Participants and Sessions. 16 volunteers are included in the discussion,
all with an academic background, among them 56 % female and 44 % male, who
performed in total 21 drone teaching sessions with a total of 157 feedback
rounds.555We did not include sessions not completed by the respective user in
order not to bias the data. Thus, only sessions are included where each of the
three commands has been selected at least once.

Figure 4: Comparison of the reward score distributions of positively and
negatively intended feedback (ground truth feedback) for each study
participant. The boxes in the boxplot show the median and the 25 % and 75 %
quantiles.

##  6 Analysis and Discussion

We study whether our approach based on emotional feedback overall leads to
successful drone learning and whether the scores derived by our fer-based RL
algorithm correlate with the users’ intention (ground truth feedback). We
dissect this result further by analysing the capabilities of fer per-se.
Finally, we quantify the quality of learning, i.e., assess the magnitude of
how well desired and undesired actions can be distinguished in the drone’s
learned action values.

(1) Did users succeed in teaching the drone their desired command-action
mapping? – The majority of users successfully taught the drone agent based on
a few trials only (see Fig. 3(a)): $57$% of the training sessions succeeded to
correctly teach all three command-action mappings and $19$% succeeded in
correctly teaching two out of three; only less than $10$% of the sessions
yielded in no success at all. These first results indicate the overall
feasibility of our approach using emotional response as a reward signal.

(2) Are the reward scores correlated with the ground truth feedback? – We can
analyze whether the reward scores based on Eq. 1 correlate with the ground
truth feedback labels provided by the users (positive or negative rating of
the drone’s chosen action). Fig. 3(b) shows the reward scores wrt. their
corresponding ground truth label across all trials and participants. The
distributions of positively and negatively intended reward scores indeed
differ, as visually observable, which is confirmed by a two-sample Kolmogorov-
Smirnov test that rejects the null hypothesis $H_{0}$ with $p<0.001$ ($H_{0}$:
The positively and negatively intended reward scores come from the same
distribution). Yet, we also see that the reward scores of a considerable
amount of negatively labeled scores have positive values.

To dissect this phenomenon, we analyze the score distributions of each
individual participant, as shown in Fig. 4. We generally observe a visible
difference between the distributions of the positive and negative scores,
indicating that our generic reward score computation (Eq. 1) produces a suited
learning signal. Yet, we find that both positive and negative score medians
differ widely between the different participants. For example, whereas the
positive and negative feedback distributions of the participant with user ID 1
appear rather weakly separated, users with IDs 2–3 and 13–15 show extremely
well separated positive and negative feedback distributions.

Figure 5: 7-d emotional response vectors projected into 2-d for all user IDs
and colored wrt. ground truth feedback (blue: positive, orange: negative).
Figure 6: Facetted view of Fig. 5, outlining selected participants’ 2-d
projection of the 7-d emotional response vector.

### (3) Are the facial emotion classifications correlated with the ground
truth?

– We now evaluate the initial emotional responses before applying the reward
function. Fig. 5 and Fig. 6 visualize the 2-dimensional projections of the
7-dimensional average emotional probability vectors $\overline{P(E)}$, labeled
according to the ground truth feedback labels (positive/negative). The
projection is achieved with _multi-dimensional scaling_ (MDS) (Kruskal and
Wish 2009), a method that aims to preserve the distances between the data
points in the lower-dimensional projection. It can be observed that positive
and negative emotional responses tend to cluster across different subjects.
However, we also note an overlapping area where a clear separation is not
possible, mainly caused by the individual effects of a specific user (User ID
4).

To provide a quantitative assessment of the general correlation between the
emotion vectors and the ground truth labels, we fit a _logistic regression_
model onto the original 7-dim. average emotion vectors $\overline{P(E)}$ to
predict the ground truth feedback labels (positive/negative), which results in
a prediction error of $20$%, which can be interpreted as a quantification of
the overlap between the positive and negative feedback across different users,
likely corresponding to the interspersed area of positively and negatively
labeled emotional responses in Fig. 5. Fig. 6 dissects this 2-dimensional
projection for four selected individual participants, revealing typically
well-separated “clusters” of positive and negative emotional responses, but
also outlining a user with emotional characteristics that are difficult to
distinguish (User ID 1). These results indicate the need for learning a _user-
specific_ feedback inference model, and thus, confirm the fit of our RL-based
approach to the emotional feedback case. Further, our MAB-approach for
rewarding allows to consider _relative distances_ of each individual user’s
feedback scores, thus mitigating the problem of different individual inter-
cluster distances of emotional vectors.

### (4) How well have users been able to teach the drone?

– The MAB-based approach allows the drone to learn a command-action mapping
even when the difference between the average rewards for desired and non-
desired actions is small. In our last analysis, we thus inspect the drone
agent’s _final learning outcome_ at the last time step $T$ of each teaching
session, by quantifying the difference between the participant’s _desired_ and
_undesired_ actions, with actions referred to by their position index in
vector $[a_{1},\ldots,a_{k}]^{T}$. The final action values $Q_{T}(a)$
represent the drone’s “view”, where the _desired_ action $a_{d}$ should
ideally yield the maximum action value estimate $Q_{T}(a_{d})$. Fig. 7
visualizes the difference
$d(a_{i})=Q_{T}(a_{d})-Q_{T}(a_{i})\quad\forall\,1\leq i\leq k$ across all
final command-action mappings in all sessions.

Figure 7:  Difference of action values to the desired action $a_{d}$,
$d(a_{i})=Q_{T}(a_{d})-Q_{T}(a_{i})$, across all command-action combinations
(x-axis) of all sessions (y-axis, labeled by participant ID – note that one
participant may have conducted multiple sessions): yellow cells correspond to
$d(a_{i})=0$ (as given for $a_{i=d}$); the greener the cell, the larger
$d(a_{i})$. Orange color indicates wrongly learned actions.

It can be observed that some sessions show a strong pattern, i.e., $d(a_{i})$
is large for all undesired actions $a_{i\neq d}$, meaning that teaching was
efficient (which corresponds to well-separated feedback scores as given in
Fig. 4). Conversely, negative values of $d(a_{i})$ indicate that a non-desired
action has received a higher action value than the desired action, indicating
a wrongly learned mapping. Based on this color-coding, we can easily identify
unsuccessful teaching, and participants (e.g., user ID 2 and 15) who have been
most effective in successfully instructing the drone as well as participants
who have given rather weak feedback signals (e.g., user ID 1). This again
reveals considerable individual differences among participants.

##  7 Conclusion

We contributed insights on developing artificial agents that learn based on
human emotional feedback, studied along the use case human-drone interaction.
We conducted a user study with 16 participants. Our initial empirical findings
confirm that humans’ facial emotion expressions indeed can be exploited as
reward signals in interactive human-in-the-loop reinforcement learning. For
the majority of the study participants, teaching the drone worked
successfully, supported in particular by our reward function derived from
emotional feedback. The learning process accounts for individual differences
in expressing emotions, without necessitating any further personalization to
individual users.

This is a first study on the feasibility of emotion-based reinforcement
learning for flying robots. Many practical aspects are subject to future
studies, including the limitations of emotional feedback wrt. complex tasks,
the inaccuracies introduced by noise and low-quality video feed, and that
observing human emotion may be a threat to privacy.

## Acknowledgements

We sincerely wish to thank all our study participants. A. Salfinger was funded
by the Austrian Science Fund (FWF) under grant FWF T961-N31.

## References

  * Ahmad et al. (2019) Ahmad, M. I.; Mubin, O.; Shahid, S.; and Orlando, J. 2019.  Robot’s Adaptive Emotional Feedback Sustains Children’s Social Engagement and Promotes Their Vocabulary Learning: A Long-Term Child-Robot Interaction Study.  _Adaptive Behavior_ , 27(4): 243–266.
  * Arriaga et al. (2020) Arriaga, O.; Valdenegro-Toro, M.; Muthuraja, M.; Devaramani, S.; and Kirchner, F. 2020.  Perception for Autonomous Systems (PAZ).  arXiv:2010.14541.
  * Arriaga, Valdenegro-Toro, and Plöger (2017) Arriaga, O.; Valdenegro-Toro, M.; and Plöger, P. 2017.  Real-time Convolutional Neural Networks for Emotion and Gender Classification.
  * Biundo and Wendemuth (2016) Biundo, S.; and Wendemuth, A. 2016.  Companion-Technology for Cognitive Technical Systems.  _KI - Künstliche Intelligenz_ , 30(1): 71–75.
  * Broekens (2007) Broekens, J. 2007.  Emotion and Reinforcement: Affective Facial Expressions Facilitate Robot Learning.  In Huang, T. S.; Nijholt, A.; Pantic, M.; and Pentland, A., eds., _Artifical Intelligence for Human Computing_ , 113–132. Berlin, Heidelberg: Springer Berlin Heidelberg.  ISBN 978-3-540-72348-6.
  * Chakraborti et al. (2017) Chakraborti, T.; Kambhampati, S.; Scheutz, M.; and Zhang, Y. 2017.  AI Challenges in Human-Robot Cognitive Teaming.  arXiv:1707.04775.
  * Citron et al. (2014) Citron, F. M. M.; Gray, M. A.; Critchley, H. D.; Weekes, B. S.; and Ferstl, E. C. 2014.  Emotional Valence and Arousal Affect Reading in an Interactive Way: Neuroimaging Evidence for an Approach-Withdrawal Framework.  _Neuropsychologia_ , 56: 79–89.
  * Ekman et al. (1987) Ekman, P.; Friesen, W. V.; O’Sullivan, M.; Chan, A.; Diacoyanni-Tarlatzis, I.; Heider, K.; Krause, R.; LeCompte, W. A.; Pitcairn, T.; Ricci-Bitti, P. E.; Scherer, K.; Tomita, M.; and Tzavaras, A. 1987.  Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion.  _Journal of Personality and Social Psychology_ , 53(4): 712–717.
  * Goodfellow et al. (2013) Goodfellow, I. J.; Erhan, D.; Carrier, P. L.; Courville, A.; Mirza, M.; Hamner, B.; Cukierski, W.; Tang, Y.; Thaler, D.; Lee, D.-H.; Zhou, Y.; Ramaiah, C.; Feng, F.; Li, R.; Wang, X.; Athanasakis, D.; Shawe-Taylor, J.; Milakov, M.; Park, J.; Ionescu, R.; Popescu, M.; Grozea, C.; Bergstra, J.; Xie, J.; Romaszko, L.; Xu, B.; Chuang, Z.; and Bengio, Y. 2013.  Challenges in Representation Learning: A Report on Three Machine Learning Contests.  In Lee, M.; Hirose, A.; Hou, Z.-G.; and Kil, R. M., eds., _Neural Information Processing_ , 117–124. Berlin, Heidelberg: Springer Berlin Heidelberg.  ISBN 978-3-642-42051-1.
  * Hörnle et al. (2017) Hörnle, T.; Tornow, M.; Honold, F.; Schwegler, R.; Heinemann, R.; Biundo, S.; and Wendemuth, A. 2017.  Companion-Systems: A Reference Architecture.  In Biundo, S.; and Wendemuth, A., eds., _Companion Technology: A Paradigm Shift in Human-Technology Interaction_ , 449–469. Cham: Springer International Publishing.  ISBN 978-3-319-43665-4.
  * Hummel, Pollak, and Krahofer (2019) Hummel, K. A.; Pollak, M.; and Krahofer, J. 2019.  A Distributed Architecture for Human-Drone Teaming: Timing Challenges and Interaction Opportunities.  _Sensors (Basel, Switzerland)_ , 19(6).
  * Knox and Stone (2009) Knox, W. B.; and Stone, P. 2009.  Interactively Shaping Agents via Human Reinforcement.  In Gil, Y.; and Noy, N., eds., _Proceedings of the fifth international conference on Knowledge capture_ , 9. New York, NY: ACM.  ISBN 9781605586588.
  * Kossaifi et al. (2021) Kossaifi, J.; Walecki, R.; Panagakis, Y.; Shen, J.; Schmitt, M.; Ringeval, F.; Han, J.; Pandit, V.; Toisoul, A.; Schuller, B.; Star, K.; Hajiyev, E.; and Pantic, M. 2021.  SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(3): 1022–1040.
  * Kruskal and Wish (2009) Kruskal, J. B.; and Wish, M. 2009.  _Multidimensional scaling_ , volume 11 of _Sage university papers Quantitative applications in the social sciences_.  Newbury Park, Calif.: Sage Publ, [nachdr.] edition.  ISBN 9780803909403.
  * MacGlashan et al. (2017) MacGlashan, J.; Ho, M. K.; Loftin, R.; Peng, B.; Wang, G.; Roberts, D. L.; Taylor, M. E.; and Littman, M. L. 2017.  Interactive Learning from Policy-Dependent Human Feedback.  In Precup, D.; and Teh, Y. W., eds., _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , 2285–2294. International Convention Centre, Sydney, Australia: PMLR.
  * Mehrabian and Russell (1974) Mehrabian, A.; and Russell, J. A. 1974.  _An approach to environmental psychology_.  Cambridge, Mass.: MIT Press.  ISBN 0262130904.
  * Minh Dang et al. (2020) Minh Dang, L.; Min, K.; Wang, H.; Jalil Piran, M.; Hee Lee, C.; and Moon, H. 2020.  Sensor-Based and Vision-Based Human Activity Recognition: A Comprehensive Survey.  _Pattern Recognition_ , 108: 107561.
  * Reyes, Meza, and Pineda (2019) Reyes, M. E.; Meza, I. V.; and Pineda, L. A. 2019.  Robotics Facial Expression of Anger in Collaborative Human–Robot Interaction.  _International Journal of Advanced Robotic Systems_ , 16(1): 172988141881797\.
  * Russell (2019) Russell, S. J. 2019.  _Human compatible: Artificial intelligence and the problem of control_.  New York NY: Viking.  ISBN 9780525558613.
  * Scheutz (2004) Scheutz, M. 2004.  Useful Roles of Emotions in Artificial Agents: A Case Study from Artificial Life.  In _Proceedings of the 19th National Conference on Artifical Intelligence_ , AAAI’04, 42–47. AAAI Press.  ISBN 0262511835.
  * Sequeira (2013) Sequeira, P. 2013.  _Socio-Emotional Reward Design for Intrinsically Motivated Learning Agents_.  Ph.d. thesis, Universidade de Lisboa, Portugal.
  * Spezialetti, Placidi, and Rossi (2020) Spezialetti, M.; Placidi, G.; and Rossi, S. 2020.  Emotion Recognition for Human-Robot Interaction: Recent Advances and Future Perspectives.  _Frontiers in Robotics and AI_ , 7: 145.
  * Sumers et al. (2021) Sumers, T. R.; Ho, M. K.; Hawkins, R. D.; Narasimhan, K.; and Griffiths, T. L. 2021\.  Learning Rewards From Linguistic Feedback.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(7): 6002–6010.
  * Sutton and Barto (2018) Sutton, R. S.; and Barto, A. G. 2018.  _Reinforcement Learning: An Introduction_.  Adaptive Computation and Machine Learning series. MIT Press.  ISBN 9780262039246.
  * Thomaz and Breazeal (2008) Thomaz, A. L.; and Breazeal, C. 2008.  Teachable Robots: Understanding Human Teaching Behavior to Build More Effective Robot Learners.  _Artificial Intelligence_ , 172(6-7): 716–737.
  * Thunberg and Ziemke (2021) Thunberg, S.; and Ziemke, T. 2021.  User-Centred Design of Humanoid Robots’ Communication.  _Paladyn, Journal of Behavioral Robotics_ , 12(1): 58–73.
  * Vieira de Castro et al. (2021) Vieira de Castro, A. C.; Araújo, Â.; Fonseca, A.; and Olsson, I. A. S. 2021.  Improving Dog Training Methods: Efficacy and Efficiency of Reward and Mixed Training Methods.  _PloS one_ , 16(2): e0247321.
  * Zhang et al. (2016) Zhang, K.; Zhang, Z.; Li, Z.; and Qiao, Y. 2016.  Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks.  _IEEE Signal Processing Letters_ , 23(10): 1499–1503.


# Improving Code Autocompletion with Transfer Learning

Wen Zhou  Facebook Inc.
Menlo Park, U.S.A.
<EMAIL_ADDRESS>Seohyun Kim  Facebook Inc.
Menlo Park, U.S.A.
<EMAIL_ADDRESS>Vijayaraghavan Murali  Facebook Inc.
Menlo Park, U.S.A.
<EMAIL_ADDRESS>Gareth Ari Aye  Facebook Inc.
Menlo Park, U.S.A.
<EMAIL_ADDRESS>
###### Abstract

Software language models have achieved promising results predicting code
completion usages, and several industry studies have described successful IDE
integrations. Recently, accuracy in autocompletion prediction improved
12.8%[1] from training on a real-world dataset collected from programmers’ IDE
activity. But what if limited examples of IDE autocompletion in the target
programming language are available for model training? In this paper, we
investigate the efficacy of pretraining autocompletion models on non-IDE, non-
autocompletion, and different-language example code sequences. We find that
these unsupervised pretrainings improve model accuracy by over 50% on very
small fine-tuning datasets and over 10% on 50k labeled examples. We confirm
the real-world impact of these pretrainings in an online setting through A/B
testing on thousands of IDE autocompletion users, finding that pretraining is
responsible for increases of up to 6.63% autocompletion usage.

###### Index Terms:

Machine learning, neural networks, software language models, naturalness, code
completion, integrated development environments, software tools

##  I Introduction

Autocompletion is the most frequently used IDE feature[2]. Significant
attention has been given to improving suggestion prediction through machine
learning[3, 4, 5, 6] by feeding code to models as a sequence of tokens or even
AST nodes [7]. Figure 1 shows an example of autocomplete powered by deep
learning in an IDE. Several recent studies [1] and [8] have demonstrated the
strengths of real-world and weaknesses of synthetic datasets in training and
evaluating autocompletion models. Concept drift between real-world and
synthetic examples can be ameliorated by only showing models real-world
autocompletion selections. But what if autocompletion examples from IDE usage
in the target language are hard to come by?

Figure 1: Example of autocomplete in an IDE.

Consider the various stages of code that could be used for autocompletion
training data. Figure 2 shows the different code datasets. First, we have the
code in IDE- these are snapshots of code that is currently being written in an
IDE. From here, there are two actions that provide two more datasets. First is
after a developer uploads the code to code review and is accepted. This
committed code in central version control provides a more permanent, mature
snapshot of the code. The other is when a developer accepts autocompletion
suggestions from the IDE. The selection, along with the surrounding context,
is logged. This provides us with a supervised dataset for autocompletion.

Even with a large user population, the tool’s vendor may not be able to log
autocompletion events to build a training dataset. But perhaps the vendor is
able to collect random, non-autocompletion code sequences during code
authoring. These would suffice to train a task-agnostic language model (LM)
for autocompletion prediction, but under the assumption that tokens used in
autocompletion follow the same distribution as tokens in arbitrary code
sequences. One flaw in this assumption is that modern IDE autocompletion tools
restrict the token types which can be suggested, commonly disallowing
punctuation, literals, and variable declarations. Furthermore, several recent
studies[1][8] have shown a myriad of differences between random tokens sampled
from source code and those used in autocompletion. Can knowledge from these
unlabeled code sequences transfer to the autocompletion task?

On the other hand, a huge corpus of open-source code commits written by
millions of developers and spanning all major programming languages is
available on GitHub. There is an intuitive relationship between code commits
and developers’ IDE activity since a commit is the first code artifact that a
developer will produce after code authoring. Can knowledge from these commits
transfer to modeling code authoring behaviors?

Figure 2: Different stages of code that could be used for autocompletion
training data. IDE dataset consists of snapshots of source code files
collected during code authoring. Once a developer submits the code to code
review, it becomes part of Facebook’s version control data. From the IDE, a
developer can also choose an autocompletion suggestion, which would be logged
(along with the surrounding context code).

Another potential cause for insufficient labeled data is a non-uniform
programming language usage distribution. While there are undeniably shared
concepts and constructs across programming languages, there are often major
differences in syntax, programming style, and language constructs.
Additionally, language-specific differences in IDE tools may impact
programmers’ autocompletion behaviors. At Facebook, we have a large developer
activity dataset in the Hack programming language containing millions of real-
world autocompletion examples. But there is also a long tail of less
frequently used languages such as OCaml. These languages lack enough labeled
data for robust model training, but they are still used by a sufficient
developer population to warrant autocompletion support. Can knowledge from the
more popular languages transfer to others where labeled data is insufficient?

In this paper, we explore these questions regarding transfer learning for
autocompletion empirically. Advances in Transformer [9, 10, 11, 12, 13]-based
neural models have popularized transfer learning in the deep learning
community. Often transfer learning consists of “pre-training” such a model on
large corpora of unlabeled data in an unsupervised manner and then “fine-
tuning” on a (possibly smaller) corpus of labeled data. The latter fine-tuning
corpus is typically drawn from the same distribution as the test dataset. In
our study, the evaluation task is to predict IDE autocompletion selections
made by programmers in a given language, and our fine-tuning datasets consist
of real-world autocompletion events collected from IDE usage at Facebook. We
explore the effect of transfer learning by pretraining models on non-IDE, non-
autocompletion, and different programming language code sequences.
Specifically, we answer the following research questions:

RQ1: How do autocompletion models benefit from combining unsupervised
pretraining with task-specific fine-tuning? How does their performance improve
across offline and online evaluation?

Our experimental results show that pretraining on code sequences collected
during code authoring and fine-tuning on tokens selected through
autocompletion produces models which outperform training on only one of these
two datasets. We show further that such a model drives greater online tool
usage.

RQ2: What is the impact of pretraining on a large source code dataset obtained
from outside of code authoring? Can these pretrained software language models
be fine-tuned on IDE autocompletion to achieve better accuracy with fewer
real-world examples?

We show that starting from a model pretrained on files appearing in version
control commits decreases the number of real-world autocompletion examples
required to achieve high accuracy. Additionally, our experiments suggest that
there is diminishing marginal benefit to pretraining as the number of real-
world examples grows.

RQ3: Consider the case where a large training corpus is available in one
language but not another. Can pretraining a multilingual model on the language
with more training data benefit the language with less data?

To answer this question, we pretrain a multilingual model on examples from one
language before fine-tuning on examples from another. Our results show that
fewer target-language examples are required to achieve high accuracy after
pretraining on different-language examples. We again observe diminishing
marginal benefit to pretraining on different-language examples as the number
of available target-language examples grows.

Contributions

  1. 1.

We pretrain two transformer software language models GPT-2[9] and BART[11] on
source code files obtained from version control commits and show how their
performance on autocompletion prediction improves through fine-tuning on real-
world IDE code sequences by 2.18%.

  2. 2.

The GPT-2 model is trained on two real-world datasets: code sequences logged
during IDE authoring and autocompletion selections. A third variant is
pretrained on the former and fine-tuned on the latter corpus to demonstrate
how the combination of pretraining and task-specific fine-tuning lead to a
superior model, outperforming the base model by 3.29%.

  3. 3.

We show that pretraining on a different programming langauge boosts accuracy
by 13.1% when comparing a model pretrained on Hack examples and fine-tuned on
10k Python examples versus only training on Python examples.

  4. 4.

We prove that improvements across these three transfer learning
dimensions—task, domain, and language —translate into increased autocompletion
tool usage by 3.86%, 6.63%, 4.64%, respectively, comparing these models
through online A/B tests.

Outline

The rest of this paper is organized to first introduce our experimental setup
in Section II. In this section we describe the corpora, language models, and
evaluation methodologies employed in our study. Section III reports
experimental results, supporting the answers to our research questions.
Section IV, Section V, and Section VI discuss threats to validity, related
work, and future work, respectively. Finally, Section VII concludes with a
summation and key insights for autocomplete tool designers.

##  II Experimental Setup

###  II-A Datasets

This study’s datasets (summarized in Table I) are sourced from real-world
developer activity at Facebook. We focus on two languages, Hack and Python,
with significant developer populations at Facebook. Going forward, we will
refer to each dataset by _[language][dataset]_ (e.g. HackCommit). Figure 2
shows a diagram of the datasets as part of the various stages of the code
cycle.

  1. 1.

_Autocompletion_ : Autocompletion events logged whenever a programmer accepts
a completion suggestion. The accepted suggestion, other unused suggestions
(from static analysis), and the surrounding program context are logged.

  2. 2.

_IDE_ : Snapshots of source code files collected during code authoring. In
addition to capturing the file contents, we collect the cursor position so
that specific code fragments undergoing modification can be identified. This
dataset shares the code authoring domain with _Autocompletion_ , but while
_Autocompletion_ is task-specific, _IDE_ is importantly task-agnostic.

  3. 3.

_Commit_ : The set of files created or modified in a version control commit
and uploaded to Facebook’s code review service. This version control dataset
is included in our investigation to explore transfer learning across domains
in light of the concept drift reported in [1] and [8]. In a typical
programming workflow, version control commits represent the software artifacts
with the closest relationship to IDE code authoring.

  4. 4.

_All_ : A union of _Autocompletion_ and _IDE_ constructed to explore the
effect of mixing labeled data into pretraining.

We train a variety of monolingual models on examples from only one of Hack and
Python as well as several multilingual models on a union of examples from both
languages. When training on a union of Hack and Python examples, we construct
the model vocabulary _V_ = $V_{hack}\cup V_{python}$. Code sequences fed to
our multilingual model are modified by prepending a control code[14] to
indicate the source language.

TABLE I: Various datasets used in this paper. | Python | Hack
---|---|---
| # tokens | # samples | # tokens | # samples
_Autocompletion_ | 543,401,684 | 3,201,299 | 642,886,605 | 3,792,485
_IDE_ | 540,200,385 | 3,201,299 | 639,094,120 | 3,792,485
_Commit_ | - | - | 629,192,335 | 3,452,434

###  II-B Tokenization

One difficulty in modeling source code as compared to natural language is that
code introduces new vocabulary at a far higher rate [15]. Recognizing and
predicting rare and novel tokens from an open vocabulary poses a challenge
when our models are trained to recognize a fixed set of terms. A strategy
explored in [15] is to model code as a sequence of partial tokens. This scheme
allows an open vocabulary of code tokens to be represented using a fixed-size
vocabulary. Another idea from [7] is copy mechanism where out-of-vocabulary
(OOV) tokens are encoded so that model can recognize matches and predict these
terms by reference. In this study we tokenize code fragments in two different
ways:

#### Byte-pair encoding (BPE)

This scheme used in our PLBART model tokenizes source code tokens as a
sequence of partial tokens. Common character sequences are encoded with a
single subword whereas rare character sequences are encoded with longer
sequences of shorter subwords. BPE has been applied successfully across
natural language [16] and programming language [15] modeling.

#### Bigram encoding + copy mechanism

This scheme used in our GPT-2 model tokenizes snake_case and camelCase
identifier tokens as exactly two subtokens (bigrams). We selected this
encoding strategy for online experiments in autocompletion ranking since it
yields a short, fixed-height partial token tree to search during inference. In
our online deployment, we worked with a 100ms latency budget for model
predictions. Searching a height-2 tree is fast enough to meet low latency user
experience requirements in autocompletion.

Concretely, consider a vocabulary _V_ of partial tokens and an example token
_t_ = “fooBarBazQuux”. First _t_ is broken into a list of partial tokens
[“foo”, “Bar”, “Baz”, “Quux”]. Then we choose a split point that cuts the list
into two equal length sublists and join the elements of each list to form a
bigram ($b_{1},b_{2}$) = (“fooBar”, “BazQuux”). Finally the $i^{th}$ unique,
OOV bigram $b_{i}$ $\notin$ _V_ is replaced with a synthetic token <var-i> as
in [17]. The special case of _t_ $\in$ _V_ receives a length-2 encoding of [t,
</t>] where </t> is a synthetic end-of-token identifier.

###  II-C Models

In this paper, we evaluate the effects of transfer learning using two models,
both incorporating Transformer architectures. Since the main focus of this
paper is to examine the impact of transfer learning, we limit our focus to
these two models and do not compare their performance to other state-of-the-
art models. Our experiments leverage:

#### GPT-2

a decoder transformer model [9], which has achieved state-of-the-art
performance in code prediction [6] due to the transformer’s ability to observe
all sequence elements simultaneously with its self-attention block.

#### PLBART

a bidirectional model that utilizes an encoder to encode the context
surrounding the code completion point, as well as a GPT-2-like decoder for
auto-regressive generation [11]. PLBART demonstrates state-of-the-art
performance across a variety of source code modeling tasks in [13].

###  II-D Training

Each of the software language models were trained in two phases. The first
phase is pretraining, in which models are shown a large number of source code
examples drawn from a dataset with concept drift from the evaluation dataset.
The second phase is fine-tuning where models are shown source code examples
drawn from the same distribution as the evaluation dataset. All models were
trained for up to twenty epochs (with early termination at convergence) using
Nvidia Tesla V100 GPUs. The learning rates for pretraining and fine-tuning
were set to $5^{-4}$ and $5^{-6}$ respectively111We reduced the fine-tuning
learning rate after observing models abandoning solutions discovered during
pretraining and overfitting the smaller fine-tuning datasets..

###  II-E Evaluation

We measure the performance of our study’s models through both offline and
online evaluation.

#### Offline evaluation

For offline evaluation, we use 10% of HackAutocompletion and
PythonAutocompletion as held-out test datasets. The evaluation task is for our
autocompletion models to predict users’ real, historical, IDE autocompletion
selections given the surrounding program context and suggestion list (from
static analysis). HackAutocompletion examples have an average of 99.5
candidate suggestions to choose from and PythonAutocompletion examples have an
average of 26.3. The candidate suggestions list allows us to frame our
evaluation as a ranking problem. For each offline measurement, we report top-1
and top-3 accuracy as well as mean reciprocal rank at _k_ = 3 (MRR@3). MRR is
defined as:

$\text{MRR}=\frac{1}{n}\sum_{i=1}^{n}\frac{1}{rank_{i}}$ (1)

where $n$ is the size of test dataset and $rank_{i}$ is the rank of the
correct token predicted by the model as the $i^{th}$ candidate. In our
evaluation, we only consider the top _k_ = 3 results (otherwise the score will
be zero).

#### Online evaluation

The ultimate goal of our research is to improve the developer IDE experience.
While offline evaluation is faster and lower-cost, it’s imperative to test
improvements with real users. In this study, we ran several live A/B
experiments on thousands of Hack developers at Facebook. In each experiment,
developers are randomly assigned to an experiment or control group, and we
measure daily completions per user (DCPU). Using this metric, A/B test
observations are taken as the number of times a given developer in one of
these two groups uses autocompletion on a given day. We conduct each
experiment until it reaches statistical significance of at least 95%222Each
experiment takes approximately two weeks to reach statistical significance..

##  III Results

TABLE II: Production A/B test results. An observation counts the number of completion suggestions accepted by a given developer on a given day. | Pretraining | Fine-tuning | mean | std dev | # unique developers | improvement | p-value
---|---|---|---|---|---|---|---
1 | HackIde | - | 17.78 | 24.98 | 3912 | - | -
HackIde | HackAutocompletion | 18.25 | 27.37 | 3933 | 0.0386 | 0.0238
2 | - | HackAutocompletion | 18.14 | 26.22 | 3002 | - | -
HackIde | HackAutocompletion | 19.34 | 27.77 | 3022 | 0.0663 | 0.0172
3 | - |  100k (HackAll $\rightarrow$ HackAutocompletion) | 16.19 | 22.45 | 3704f | - | -
HackCommit |  100k (HackAll $\rightarrow$ HackAutocompletion) | 16.78 | 24.88 | 3697 | 0.0362 | 0.0494
Figure 3: Production A/B test results of three experiments. In the
“pretraining” test the experiment group model is pretrained on HackIde and the
control group model is not pretrained. In the “fine-tuning” test the
experiment group is fine-tuned on HackAutocompletion and the control group is
not fine-tuned. In the final “diff pretraining” test the experiment group
model is pretrained on version control code commits and the control group
model is not pretrained. For all tests, the experiment group had greater DCPU
at a statistically significant p-value.

RQ1: How do autocompletion models benefit from combining unsupervised
pretraining with task-specific fine-tuning? How does their performance improve
across offline and online evaluation?

_Offline evaluation._ Autocompletion models fine-tuned on labeled data
(HackAutocompletion) outperform models without task-specific fine-tuning
across offline and online evaluation. In offline evaluation, Table III shows
that fine-tuning led to a top-1 accuracy increase from 39.73% to 41.91%. When
labeled examples were mixed into pretraining (HackAll), top-1 accuracy jumped
from 40.25% to 41.6%. The same trends were observed when training Python
autocompletion models (Table IV).

_Online evaluation._ For online evaluation, we trained an autocompletion model
on HackIde. The experiment variant was then fine-tuned on HackAutocompletion
whereas the control variant did not undergo fine-tuning. Experiment 1 in table
II shows that daily completions per user (DCPU) was 3.86% greater in the
experiment group at _p_ = 0.0238.

We conducted a second A/B experiment comparing the better model from the
previous experiment (pretraining on HackIde and fine-tuning on
HackAutocompletion) to a model trained on HackAutocompletion without
pretraining. Experiment 2 in table II shows an even larger improvement over
the model without pretraining—the experiment group DCPU was 6.63% greater at
_p_ = 0.017.

RQ2: What is the effect of pretraining on a large source code dataset obtained
from outside of code authoring? Can pretrained software language models be
fine-tuned on IDE autocompletion to achieve better accuracy with fewer real-
world examples?

_Offline evaluation._ Given limited real-world examples of IDE autocompletion,
pretraining on version control commits can make a big difference. However, as
the number of autocompletion examples grows, the benefit of pretraining
diminishes. These observations held constant across the GPT-2 and PLBART
models we pretrained on a commits dataset.

Table III shows that HackCommit pretraining with HackAutocompletion fine-
tuning outperforms HackAutocompletion by 3.29% top-1 accuracy (39.61% vs.
36.32%). The top-1 accuracy improvement is even greater for PLBART (Table VI):
6.52% (51.06% vs. 44.54%). However, pretraining on HackCommit yields worse
performance compared to IDE code sequences (HackIde). The GPT-2 variant has
1.99% lower top-1 accuracy (39.61% vs. 41.60%) whereas the PLBART variant is
2.17% weaker (51.06% vs. 53.23%). This result aligned with our expectations as
the HackCommit dataset, being sourced from a different stage of software
development, exhibits greater concept drift from autocompletion than code
sequences randomly sampled during code authoring.

_Online evaluation._ We conducted an A/B experiment to explore the real-world
impact of pretraining on code commits. The GPT-2 model trained on 100k HackAll
examples was compared against a variant pretrained on all of the data from
HackCommit and fine-tuned on 100k HackAll examples. Experiment 3 in table II
shows that the pretrained model drove an improvement of 3.62% DCPU at _p_ =
0.049.

_Experiment - varying number of fine-tuned examples._ We also experimented
with stacking the two pretrainings (first HackCommit and then HackAll) before
fine-tuning on HackAutocompletion and found that these models, whether GPT-2
or PLBART, did not show meaningful improvement over a single HackAll
pretraining. To understand why multiple pretrainings did not result in better
performance, we conducted a final experiment in which the number of real-world
fine-tuning examples was varied from 0% to 100%. The results are shown in
Figure 3.

What we found is that pretraining has a diminishing marginal benefit as the
number of fine-tuning examples grows. Given a small number of real-world
examples, pretraining on HackCommit has a major impact. For example, there is
an improvement of 17.25% (37.48% vs. 20.23%) when we limited the number of
autocompletion examples to 25k! However, at a certain point, given enough
training data points drawn from the same distribution as our evaluation
dataset, pretraining on a dataset with domain concept drift is no longer
helpful.

TABLE III: Offline evaluations of mono-lingual GPT-2 models on Hack Pretraining | Fine-tuning | Epochs | Top1 Acc | Top3 Acc | MRR
---|---|---|---|---|---
- | HackAutocompletion | 6 | 0.3632 | 0.5614 | 0.4508
HackIde | - | 18 | 0.3973 | 0.5816 | 0.4787
HackIde | HackAutocompletion | 18+11 | 0.4191 | 0.5987 | 0.4988
HackAll | - | 12 | 0.4025 | 0.5858 | 0.4835
HackAll | HackAutocompletion | 12+7 | 0.4160 | 0.5970 | 0.4963
HackCommit | - | 20 | 0.3479 | 0.5357 | 0.4306
HackCommit | HackAutocompletion | 20+4 | 0.3961 | 0.5857 | 0.4801
HackCommit | HackAll | 20+20 | 0.4026 | 0.5867 | 0.4841
HackCommit |  HackAll $\rightarrow$ HackAutocompletion | 20+20+20 | 0.4145 | 0.5962 | 0.4953
TABLE IV: Offline evaluations of mono-lingual GPT-2 models on Python Pretraining | Fine-tuning | Epochs | Top1 Acc | Top3 Acc | MRR
---|---|---|---|---|---
- | PythonAutocompletion | 6 | 0.5282 | 0.6951 | 0.6029
PythonIde | - | 15 | 0.5610 | 0.7228 | 0.6331
PythonIde | PythonAutocompletion | 15+13 | 0.5751 | 0.7286 | 0.6439
PythonAll | - | 19 | 0.5605 | 0.7188 | 0.6313
PythonAll | PythonAutocompletion | 19+8 | 0.5723 | 0.7253 | 0.6408

RQ3: Consider the case where a large training corpus is available in one
language but not another. Can pretraining a multilingual model on the language
with more training data benefit the language with less data?

_Offline evaluation._ To answer this question, we pretrain models on HackAll,
fine-tune on varying amounts of PythonAll, and evaluate on held-out
PythonAutocompletion examples. The baselines we use for comparison are models
with the same configuration trained on an equal number of PythonAll examples
without any pretraining. This experiment was designed to show whether models
pretrained on Hack data exhibited superior prediction performance on Python
autocompletion examples, indicating knowledge transfer across programming
languages.

Figure 4 shows that the models pretrained on HackAll had better performance
independent of the number of PythonAll examples used in fine-tuning. The
marginal impact was greatest when we limited models to only 10k (13.1% better
top-1 accuracy, 37.11% vs. 24.01%) and 25k (12.6% better top-1 accuracy,
41.26% vs. 28.66%) PythonAll examples. This shows clear evidence of knowledge
transfer across programming languages in autocompletion. The performance of
the model pretrained on HackAll and fine-tuned with 25k and 50k PythonAll
examples is similar to the performance of training from scratch on 50k and
100k PythonAll examples, respectively. This shows that half as many examples
were required for comparable performance after pretraining on an other-
language dataset.

_Results insight._ This is a meaningful insight for IDE autocompletion
developers. Consider the case of providing predictive autocompletion ranking
for less common programming languages (or ones for which real-world training
examples are scarce). Our results show that pretraining on real-world examples
from other languages makes it possible to achieve high accuracy with a
relatively small fine-tuning dataset.

_Experiment - multilingual model._ In a final multilingual LM experiment we
combined the Hack and Python corpora to investigate whether a larger, diverse
pretraining would improve performance. In addition to incorporating multiple
languages in pretraining, we tested fine-tuning on examples from one language
against fine-tuning on examples from multiple. Fine-tuning on a union of Hack
and Python examples MultiAutocompletion led to the best-performing
multilingual model across Hack and Python evaluation. Table V shows
improvements of 0.5% and 1.34% above fine-tuning on HackAutocompletion and
PythonAutocompletion respectively. However, none of the multilingual models
showed significant improvement over the best monolingual models. The best
multilingual model had 0.53% better top-1 accuracy than the best Python model
but showed 0.02% worse top-1 accuracy than the best Hack model. Taken in
combination with our other results, we hypothesize that combining programming
language examples across languages has a diminishing marginal benefit as the
number of examples available in each language grows.

TABLE V: Offline evaluations of multi-lingual GPT-2 models on Hack and Python Base Training | Fine Tuning | Epochs | Hack | Python
---|---|---|---|---
Top1 Acc | Top3 Acc | MRR | Top1 Acc | Top3 Acc | MRR
- | MultiAutocompletion | 7 | 0.3690 | 0.5677 | 0.4569 | 0.5416 | 0.7037 | 0.6142
MultiIde | - | 20 | 0.3981 | 0.5825 | 0.4796 | 0.5614 | 0.7260 | 0.6348
MultiIde | HackAutocompletion | 20+14 | 0.4178 | 0.5973 | 0.4975 | 0.4951 | 0.6829 | 0.5780
MultiIde | PythonAutocompletion | 20+20 | 0.3616 | 0.5541 | 0.4465 | 0.5746 | 0.7274 | 0.6429
MultiIde | MultiAutocompletion | 20+12 | 0.4187 | 0.5987 | 0.4986 | 0.5759 | 0.7278 | 0.6439
MultiAll | - | 14 | 0.4046 | 0.5882 | 0.4859 | 0.5688 | 0.7302 | 0.6408
MultiAll | HackAutocompletion | 14+10 | 0.4178 | 0.5977 | 0.4977 | 0.5321 | 0.7042 | 0.6086
MultiAll | PythonAutocompletion | 14+12 | 0.3703 | 0.5584 | 0.4532 | 0.5791 | 0.7297 | 0.6466
MultiAll | MultiAutocompletion | 14+9 | 0.4190 | 0.5988 | 0.4988 | 0.5804 | 0.7308 | 0.6478
TABLE VI: Offline evaluations of GPT-2 and PLBART in each of 7 configs on Hack | Config | GPT-2 | PLBART
---|---|---|---
Top1 Acc | Top3 Acc | MRR | Top1 Acc | Top3 Acc | MRR
1 | HackCommit | 0.3479 | 0.5357 | 0.4306 | 0.428 | 0.6608 | 0.5312
2 | HackAll | 0.4025 | 0.5858 | 0.4835 | 0.4491 | 0.6863 | 0.5545
3 | HackAutocompletion | 0.3632 | 0.5614 | 0.4508 | 0.4454 | 0.6806 | 0.5498
4 |  HackCommit $\rightarrow$ HackAll | 0.4026 | 0.5867 | 0.4841 | 0.4471 | 0.682 | 0.5514
5 |  HackCommit $\rightarrow$ HackAutocompletion | 0.3961 | 0.5857 | 0.4801 | 0.5106 | 0.7315 | 0.6096
6 |  HackAll $\rightarrow$ HackAutocompletion | 0.4160 | 0.5970 | 0.4963 | 0.5323 | 0.7497 | 0.6296
7 | HackCommit$\rightarrow$HackAll$\rightarrow$HackAutocompletion | 0.4145 | 0.5962 | 0.4953 | 0.5323 | 0.7490 | 0.6293
Figure 4: Offline top-1 accuracy evaluation starting at HackCommit checkpoint
vs random initialization as size of (HackAll $\rightarrow$ HackAutocompletion)
fine-tuning samples increases Figure 5: Offline top-1 accuracy evaluation
starting at HackAll checkpoint vs random initialization as size of PythonAll
fine-tuning samples increases

##  IV Related Work

IDE Autocompletion. Many studies at the intersection of software development
tools and machine learning have investigated next code token prediction and
its application to autocompletion. Earlier attempts at modeling software
languages and completion ranking were based on n-gram language models [4, 18,
19] or probabilistic models using the information from ASTs [20]. With the
advancement of deep learning models, RNNs (and their variants) [7, 15] have
shown promising improvements. More recently, Transformers have achieved state-
of-the-art performance for software language modeling [6, 21]. Galois (Radford
et al., 2019) and TabNine are two additional code completion tools that employ
the GPT-2 Transformer for next token prediction. Furthermore, some studies
have focused on the industry use case of autocompletion for IDE users within a
company. While [5] showed that a pointer mixture network performs well on an
open-source GitHub corpus as well as Google’s internal Dart language corpus,
[8] warns that accuracy achieved on synthetic benchmarks may not translate to
real-world completion performance. Aye et al. [1] demonstrated how training on
real-world developer activity can combat this challenge.

Transfer Learning for Code. Transfer learning has revolutionized natural
language processing (NLP) since the seminal papers on models such as GPT-2
[9], BERT [10] and RoBERTa [22]. These works proposed the idea of pretraining
Transformer models on large, diverse text corpora in an unsupervised manner
and fine-tuning them for specific downstream tasks. Since then several works
have been proposed [12, 23, 24, 25, 26, 27, 28, 13] to apply the idea of
transfer learning in the domain of code.

CodeBERT [12] was one of the first models pretrained on pairs of code and
natural language sequences in order learn a bimodal representation of both
entities. It was applied on natural language code search and code
documentation generation to demonstrate the benefit of transfer learning
compared to training directly on the downstream tasks. A follow-up study [23]
showed that CodeBERT can also transfer knowledge to the problem of automated
program repair.

More recently, PLBART [13] showed that the BART architecture [11] is better-
suited for source code generation tasks compared to BERT-based models. In
comparison to these works, the goal of our paper is not to produce a state-of-
the-art model for any specific task. Rather it is to show how transfer
learning across various dimensions can benefit IDE autocompletion prediction
regardless of the model. In our experiments we used two state-of-the-art
models GPT-2 and PLBART for this purpose, and showed that both models can
benefit from transfer learning.

In a closely related work, Mastropaolo et al. [26] study empirically how the
T5 model [29] behaves when pretrained and fine-tuned on four code-related
tasks. They also make an observation that an unsupervised pretraining phase
helps the model achieve better performance on their set of tasks, such as code
summarization and mutation. In our paper, in addition to observing this effect
in the downstream task of IDE autocompletion prediction, we also explored
transfer learning in other dimensions, such as across programming languages,
and validated our results through online A/B tests.

Outside of code authoring, transfer learning has been enabled by Transformer-
based models in other software artifacts. Lin et al. [24] apply pretrained
BERT models for learning relationships between issues and commits in a
software repository. Sharma et al. [25] detect code smells in programming
languages where sufficient training data is not available by transferring
knowledge from other data-rich languages. Pei et al. [27] propose a tool
called TREX that is better at detecting binary-level function similarity than
state-of-the-art tools owing to transfer learning from binary-level execution
traces.

##  V Threats to Validity

#### Intersection of vocabulary between languages

In our investigation of multilingual software language modeling, one method we
used to reduce the out-of-vocabulary (OOV) problem was to tokenize code
sequences using a bigram encoding. Although this reduced the OOV rate for
individual languages, there was only a small overlap between the different
languages’ vocabularies. Applying BPE or another encoding scheme may have
resulted in more tokens receiving the same encoding across languages which
could increase the efficacy of transfer learning across programming languages.

#### Testing on more languages

We examined the effects of transfer learning on two languages: Hack and
Python. While we saw that transfer learning (over various dimensions) improves
performance, we did not evaluate on other languages. Hack and Python are both
dynamic, object-oriented programming languages. It’s conceivable that
knowledge transfer to a static language like C++ or a different paradigm
language like OCaml could be less effective.

#### Facebook vs open-source

When exploring pretraining on code commits from version control, we leveraged
Facebook’s internal version control repository. It’s possible that some of the
transfer learning effects we observed when fine-tuning on IDE autocompletion
would be lessened if we had instead pretrained on GitHub code commits. In
addition to the greater source code diversity in the GitHub corpus, there is
undoubtedly concept drift between code in open-source and at Facebook. There
may even be different version control usage patterns that would affect
pretraining on code commits.

##  VI Future Work

#### Stronger ablation study on PLBART

For both languages, PLBART outperformed the GPT-2 model by more than 10%.
However, it is difficult to make an apples-to-apples comparison. A
bidirectional model, PLBART leverages context after the predicted token while
GPT-2 only uses the before-cursor context. PLBART also uses BPE instead of a
bigram + copy mechanism encoding. In the future we wish to do a more thorough
ablation study to determine the biggest contributing factors to PLBART’s
performance.

#### Transfer learning across multiple languages

In this paper, we focused on transfer learning between two languages: Hack and
Python. However, there are many other languages that are used in software
engineering. Our experiments showed that pretraining on one language transfers
knowledge to the other. Does the impact of transfer learning grow or diminish
as we add more languages to pretraining? Could pretraining on multiple
languages decrease even further the number of fine-tuning examples needed in
the target language?

#### Transfer learning across source code tasks

The evaluation task for this paper was IDE autocompletion prediction. Could we
leverage our pretrained models for transfer learning to other source code
tasks such as as bug fixing or similar code detection? Furthermore, could the
pretrained model for another task be used effectively as the base model for
autocompletion as well?

##  VII Conclusion

In this paper, we explored ways in which transfer learning can improve
autocompletion. Previous work showed that a training corpus consisting of
real-world examples collected from programmers’ IDE usage leads to the highest
accuracy autocompletion models. But for some tool developers, there may be a
limited number of real-world autocompletion examples in the desired
programming language. This study showed how the power of transfer learning
enables pretraining on non-IDE, non-autoompletion, and different-language
example code sequences before fine-tuning on the autocompletion prediction
task. Our results show that we can reach comparable accuracy while drastically
reduce the number of fine-tuning examples by starting from a pretrained model.
These findings in offline evaluation were confirmed in online A/B experiments
conducted on thousands of software developers at Facebook.

## References

  * [1] G. A. Aye, S. Kim, and H. Li, “Learning autocompletion from real-world datasets,” 2020.
  * [2] G. C. Murphy, M. Kersten, and L. Findlater, “How are java software developers using the eclipse ide?” _IEEE Softw._ , vol. 23, no. 4, pp. 76–83, Jul. 2006\. [Online]. Available: http://dx.doi.org/10.1109/MS.2006.105
  * [3] M. Bruch, M. Monperrus, and M. Mezini, “Learning from examples to improve code completion systems,” in _Proceedings of the the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering_ , ser. ESEC/FSE ’09. New York, NY, USA: ACM, 2009, pp. 213–222. [Online]. Available: http://doi.acm.org/10.1145/1595696.1595728
  * [4] A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu, “On the naturalness of software,” in _Proceedings of the 34th International Conference on Software Engineering_ , ser. ICSE ’12. Piscataway, NJ, USA: IEEE Press, 2012, pp. 837–847. [Online]. Available: http://dl.acm.org/citation.cfm?id=2337223.2337322
  * [5] G. A. Aye and G. E. Kaiser, “Sequence model design for code completion in the modern ide,” 2020.
  * [6] S. Kim, J. Zhao, Y. Tian, and S. Chandra, “Code prediction by feeding trees to transformers,” 2020.
  * [7] J. Li, Y. Wang, M. R. Lyu, and I. King, “Code completion with neural attention and pointer networks,” _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence_ , Jul 2018. [Online]. Available: http://dx.doi.org/10.24963/ijcai.2018/578
  * [8] V. J. Hellendoorn, S. Proksch, H. C. Gall, and A. Bacchelli, “When code completion fails: A case study on real-world completions,” ser. ICSE ’19. IEEE Press, 2019, p. 960–970. [Online]. Available: https://doi.org/10.1109/ICSE.2019.00101
  * [9] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019.
  * [10] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [11] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: Association for Computational Linguistics, Jul. 2020, pp. 7871–7880. [Online]. Available: https://www.aclweb.org/anthology/2020.acl-main.703
  * [12] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “Codebert: A pre-trained model for programming and natural languages,” 2020.
  * [13] W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre-training for program understanding and generation,” in _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics_ , 2021.
  * [14] N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher, “Ctrl: A conditional transformer language model for controllable generation,” 2019.
  * [15] R. Karampatsis and C. Sutton, “Maybe deep neural networks are the best choice for modeling source code,” _CoRR_ , vol. abs/1903.05734, 2019. [Online]. Available: http://arxiv.org/abs/1903.05734
  * [16] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” _CoRR_ , vol. abs/1508.07909, 2015. [Online]. Available: http://arxiv.org/abs/1508.07909
  * [17] N. Chirkova and S. Troshin, “A simple approach for handling out-of-vocabulary identifiers in deep learning for source code,” _CoRR_ , vol. abs/2010.12663, 2020. [Online]. Available: https://arxiv.org/abs/2010.12663
  * [18] T. T. Nguyen, A. T. Nguyen, H. A. Nguyen, and T. N. Nguyen, “A statistical semantic language model for source code,” in _Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering_ , ser. ESEC/FSE 2013\. New York, NY, USA: Association for Computing Machinery, 2013, p. 532–542. [Online]. Available: https://doi.org/10.1145/2491411.2491458
  * [19] V. J. Hellendoorn and P. Devanbu, “Are deep neural networks the best choice for modeling source code?” in _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_ , ser. ESEC/FSE 2017. New York, NY, USA: Association for Computing Machinery, 2017, p. 763–773. [Online]. Available: https://doi.org/10.1145/3106237.3106290
  * [20] P. Bielik, V. Raychev, and M. Vechev, “Phog: Probabilistic model for code,” in _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ser. ICML’16. JMLR.org, 2016, p. 2933–2942.
  * [21] M. Brockschmidt, M. Allamanis, A. L. Gaunt, and O. Polozov, “Generative code modeling with graphs,” _arXiv preprint arXiv:1805.08490_ , 2018.
  * [22] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 2019.
  * [23] E. Mashhadi and H. Hemmati, “Applying codebert for automated program repair of java simple bugs,” 2021.
  * [24] J. Lin, Y. Liu, Q. Zeng, M. Jiang, and J. Cleland-Huang, “Traceability transformed: Generating more accurate links with pre-trained bert models,” 2021\.
  * [25] T. Sharma, V. Efstathiou, P. Louridas, and D. Spinellis, “Code smell detection by deep direct-learning and transfer-learning,” _Journal of Systems and Software_ , vol. 176, p. 110936, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0164121221000339
  * [26] A. Mastropaolo, S. Scalabrino, N. Cooper, D. N. Palacio, D. Poshyvanyk, R. Oliveto, and G. Bavota, “Studying the usage of text-to-text transfer transformer to support code-related tasks,” 2021.
  * [27] K. Pei, Z. Xuan, J. Yang, S. Jana, and B. Ray, “Trex: Learning execution semantics from micro-traces for binary similarity,” 2021.
  * [28] A. Elnaggar, W. Ding, L. Jones, T. Gibbs, T. Feher, C. Angerer, S. Severini, F. Matthes, and B. Rost, “Codetrans: Towards cracking the language of silicone’s code through self-supervised deep learning and high performance computing,” 2021.
  * [29] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _Journal of Machine Learning Research_ , vol. 21, no. 140, pp. 1–67, 2020. [Online]. Available: http://jmlr.org/papers/v21/20-074.html