# Haptic human-human interaction does not improve individual visuomotor
adaptation

Niek Beckers  Department of Biomechanical Engineering, University of Twente,
Enschede, The Netherlands  Cognitive Robotics, Delft University of Technology,
Delft, The Netherlands  Correspondence should be addressed to N.B.
<EMAIL_ADDRESS> Edwin van Asseldonk  Department of Biomechanical
Engineering, University of Twente, Enschede, The Netherlands  Herman van der
Kooij  Department of Biomechanical Engineering, University of Twente,
Enschede, The Netherlands  Department of Biomechanical Engineering, Delft
University of Technology, Delft, The Netherlands

###### Abstract

Haptic interaction between two humans, for example, a physiotherapist
assisting a patient regaining the ability to grasp a cup, likely facilitates
motor skill acquisition. Haptic human-human interaction has been shown to
enhance individual performance improvement in a tracking task with a
visuomotor rotation perturbation. These results are remarkable given that
haptically assisting or guiding an individual rarely benefits their individual
improvement when the assistance is removed. We, therefore, replicated a study
that reported that haptic interaction between humans was beneficial for
individual improvement for tracking a target in a visuomotor rotation
perturbation. In addition, we tested the effect of more interaction time and a
stronger haptic coupling between the partners on individual improvement in the
same task. We found no benefits of haptic interaction on individual
improvement compared to individuals who practised the task alone, independent
of interaction time or interaction strength.

## Introduction

While working to improve their motor abilities, such as walking or grasping a
cup, patients often rely on physical assistance of their physiotherapist.
Joint action, like physical assistance, is a crucial way through which we
learn new skills or transfer knowledge to others [1, 2]. Here, we study the
effect of haptic interaction, in which two partners exchange forces while
performing a joint task, on motor skill acquisition.

To date, the number of studies on the effect of haptic interaction between
humans on individual motor skill acquisition is limited and their results are
dissimilar. Ganesh et al.[3] were, to our knowledge, one of the first to
report that haptic interaction between two partners resulted in better
_individual_ motor performance after haptic interaction compared to
participants who never interacted. They intermittently coupled two partners
with a compliant spring generated by a dual-robot interface while they tracked
the same continuously moving target in a challenging visuomotor rotation
perturbation. A visuomotor rotation perturbation is a motor adaptation
paradigm in which the visual feedback of the arm movement is rotated with
respect to the actual arm movement. Motor performance is initially decreased
when the visuomotor rotation is introduced, but people consistently improve
performance with practice in a visuomotor rotation by compensating for the
visuomotor rotation, a process referred to as motor adaptation [4]. Ganesh et
al.[3] analysed to what extent intermittent haptic interaction influenced each
participant’s individual tracking performance in trials in which the
participants performed the task alone. The individual performance was compared
to a group who never interacted. They showed that intermittent haptic
interaction improved the participant’s _individual_ motor improvement
significantly _more_ and, although not explicitly mentioned by the authors,
initially _faster_ compared to someone who practised the task alone. These
results are encouraging and have potential high impact for designing robot-
assisted motor skill acquisition algorithms, for example for physical
rehabilitation applications.

However, more recent studies report no or context- or task-specific benefits
of haptic interaction on individual motor skill acquisition. Takagi et al. [5,
6] found no benefit on improvement rates in tracking tasks without visual
perturbation, though their tracking task might have been not challenging
enough to elicit significant differences in individual motor improvement. In
an earlier study, we found that haptic interaction did not improve individual
motor adaptation to a velocity-dependent force field while tracking a
continuously moving target [7]. Van der Wel et al.[8] reported that haptically
interacting partners learned a novel coordination task (balancing a stick)
just as quickly as individuals performing the task alone. Using a tracking
task and the same visuomotor perturbation as Ganesh et al.[3], Kager et al.[9]
found no significant effect of haptic interaction on final individual motor
performance. However, their study has a few important differences to Ganesh et
al.[3]: their tracking task was less challenging, they used a different haptic
interaction paradigm and they only tested a limited number of participants.
Lastly, practising a challenging reaching task with nonlinear dynamics while
being haptically connected to an expert appeared to be less beneficial for
subsequent individual motor performance than being connected to a partner with
similar initial skill level [10]. Unfortunately, the final individual skill
levels of their interaction participants were not compared to a solo group
that practised the task without interaction for the same amount of time. Based
on these results – although obtained using different motor tasks – we question
_whether_ haptic human-human interaction indeed benefits an individual’s
visuomotor adaptation as found by Ganesh et al.[3].

It is also unknown _why_ haptic interaction would improve individual motor
adaptation. Adapting to a visuomotor rotation is predominantly driven by
errors between planned and actual movements [4]. The larger the experienced
error, the more participants compensate for those errors in subsequent
movements [11]. Reducing a participant’s movement errors while they learn a
new motor skill using robot-generated haptic assistance rarely transferred
into improved subsequent individual performance compared to participants who
received no haptic assistance [12, 13, 14, 15, 16]. Similarly, several studies
on haptic human-human interaction consistently showed that tracking errors are
significantly smaller _during_ haptic interaction compared to performing the
tracking task alone, particularly if you interact with a partner who is better
than you [3, 5, 6, 10, 9, 7]. Combining these observations, we would expect
little to no benefit of haptic human-human interaction on individual
visuomotor adaptation.

Still, haptic interaction could be a valuable means through which partners
transfer skill or learn from each other. Visually observing another person
learn a motor skill promotes the observer’s own learning [1, 17]. Similarly,
haptic interaction could enable an individual to imitate or observe the
actions of their partner through haptics, facilitating their own motor skill
acquisition. Also, haptically interacting pairs can communicate intentions or
coordinate specialised roles through the interaction force[18, 19, 20, 21,
22]; partners could adopt teaching roles, coaching each other on how to
account for the visuomotor rotation, potentially speeding up their visuomotor
adaptation. However, coordinating roles or communicating intentions would
likely take additional time on top of the motor adaptation task. A joint
action study showed that groups can learn to coordinate actions, but the group
coordination process only occurred gradually, which could hinder task
performance compared to individuals [23]. Still, although the aforementioned
mechanisms could facilitate motor adaptation, we believe it is unlikely that
participants who were naïve to the haptic interaction – like in Ganesh et
al.[3] – were able to use any of these mechanisms to improve their short-term
motor adaptation, especially in the early stages of the adaptation process.

Our goal is to investigate whether haptic interaction with a partner who is
practising the same tracking task indeed results in more and faster individual
performance improvement in a visuomotor rotation perturbation. In addition to
replicating the experiment by Ganesh et al.[3], we also investigate whether
the amount of interaction time (i.e. more trials in which the partners
interact) and the strength of the haptic coupling affect individual
improvement. The participants in Ganesh et al.[3] interacted intermittently in
half of the trials. If haptic interaction indeed benefits individual motor
improvement, more interaction time could, on the one hand, increase these
benefits. On the other hand, as haptic interaction reduces tracking error – a
key training signal for visuomotor adaptation – practising the task always
connected to a partner could yield no individual improvement benefits. To
further test the hypothesised effect of the reduced experienced tracking error
during interaction on visuomotor adaptation, we also tested a group who
interacted through a stronger haptic coupling. A stronger coupling has been
show to reduce tracking errors significantly more than a weaker coupling, in
particular for the inferior-performing partner [6], which could affect
individual motor improvement[13]. Lastly, as the partner’s skill level can
significantly affect a participant’s _individual_ performance [10], we also
analysed the effect of the partner’s relative initial skill level on the
participant’s improvement and improvement rates.

Contrary to Ganesh et al. [3], we found no benefit of haptic human-human
interaction on individual visuomotor adaptation in a tracking task compared to
individuals who practised the task alone. Haptic interaction did not result in
more individual improvement or faster improvement. Increasing the amount of
interaction time or interaction strength also did not improve or impede
individual improvement.

## Methods

Eighty healthy participants were recruited: 40 women and 40 men; age $22.0\pm
2.1\text{\,}\mathrm{yr}$; all except three participants were right-handed
according to the Edinburgh handedness inventory [24]. The participants were
equally distributed over the four experiment groups that we will describe
later on. The participants had no prior experience with studies involving
haptic human-human interaction or visuomotor rotation adaptation paradigms. An
assessment of the study by the Medical Ethical Review Board of the University
of Twente (METC Twente) showed that the study posed minimal risk to the
participants and therefore under Dutch law did not need full ethical review.
All participants provided written informed consent. All participants received
a compensation for their participation, independent of their performance or
whether they completed the study. The experiment lasted approximately two
hours.

### Dual-robot interface

The experiment was performed using a dual-robot interface (see Fig. 1a).
Participants held a handle at the endpoint of their own robot interface with
their preferred hand. Each robot interface allowed hand movements in a planar
circular workspace with a diameter of $20\text{\,}\mathrm{cm}$. Each
participant had their own display that showed the workspace, the target and
their own cursor that they could control by moving the robot interface’s
handle (see Fig. 1b). Cursor movement was scaled to match the real-world
movement of the handle. The coordinate frame of each robot interface was
centred at the centre of the corresponding display. A panel obstructed the
participant’s view on their arm, hand and robot interface. A curtain separated
the partners to prevent social interaction.

Figure 1: Dual-robot interface, display, and visuomotor rotation paradigm. (a)
Each participant held the handle of one of the two identical robot interfaces.
Visual feedback was presented on a display mounted in front of each
participant. The robot interfaces could generate a compliant connection
between the partners to enable haptic interaction. (b) Participants received
visual feedback of their own cursor and a common target; they did not see
their partner’s cursor. (c) Visuomotor rotation: the visual feedback of the
cursor was rotated clockwise with $80\text{\,}\mathrm{deg}$ with respect to
the actual hand movement.

### Tracking task, visuomotor rotation paradigm, and haptic interaction
paradigm

The tracking task and haptic interaction paradigm were similar to the ones
used by Ganesh et al.[3]. All participants tracked a continuously moving
target with their own cursor as accurately as possible in trials with a
duration of $23\text{\,}\mathrm{s}$ each. The target trajectory (in
$\mathrm{mm}$) was defined as a sum-of-sines (see Supplementary Methods for
more details on the target signal design):

$\displaystyle x(t)=28.7\sin$
$\displaystyle\left(0.94t-7.77\right)+27.1\sin\left(1.26t-8.53\right)+23.5\sin\left(1.89t-4.36\right)+18.0\sin\left(2.83t-3.79\right)\text{,}$
(1) $\displaystyle y(t)=27.1\sin$
$\displaystyle\left(1.26t-0.71\right)+25.3\sin\left(1.57t-3.45\right)+21.6\sin\left(2.20t+3.92\right)+16.4\sin\left(3.14t+4.93\right)\textrm{.}$

The tracking signal required hand movements over the robot interface’s full
circular workspace, an average velocity of
$7.9\text{\,}\mathrm{cm}\text{\,}{\mathrm{s}}^{-1}$ and a maximum velocity of
$13.9\text{\,}\mathrm{cm}\text{\,}{\mathrm{s}}^{-1}$. We generated the
tracking signal using different time offsets $t_{0}$ for each trial, which was
randomly chosen from a uniform distribution
$\left(t\in\left[t_{0},t_{0}+20\right]$\mathrm{s}$\textrm{,
}t_{0}\sim\mathcal{U}(0,20)$\mathrm{s}$\right)$ for each trial. As a result,
the target started at different locations of the target trajectory in each
trial to help keep the tracking task challenging and slightly different per
trial.

The participants performed 84 trials divided over four blocks with
$15\text{\,}\mathrm{s}$ of rest between trials and five minutes of rest
between blocks. The first block served as a baseline block in which the
participants tracked the moving target without the visuomotor rotation; hence
the visual movements of the cursor matched the actual hand movements.

We then introduced the same $80\text{\,}\mathrm{deg}$ visuomotor rotation as
used by Ganesh et al.[3] in blocks 2, 3, and 4 to study the effect of haptic
interaction on individual tracking improvement. The cursor movement on the
display was rotated clockwise with $80\text{\,}\mathrm{deg}$ with respect to
the participant’s actual hand movement (see Fig. 1c), which initially results
in a mismatch between the expected cursor movement based on the hand movement
and actual cursor movement. A visuomotor rotation initially degrades tracking
performance, which then improves with practice, typically within a few hours,
as the participants adapt to the different mapping between actual hand
movements and the visual feedback of their hand movements [4].

The haptic interaction paradigm is the same as used by Ganesh et al.[3]. Two
types of trials were used in the experiment: _single_ trials (S), in which the
participants performed the tracking task alone, and _connected_ trials (C). In
the connected trials the partners’ hands were coupled through an compliant
connection (see the detail in Fig. 1a) with a force

$\mathbf{F}_{s}=k_{s}\left(\mathbf{p}_{p}-\mathbf{p}_{o}\right)+b_{s}\left(\dot{\mathbf{p}}_{p}-\dot{\mathbf{p}}_{o}\right)$
(2)

generated by each robot interface. A participant would experience an
interaction force $\mathbf{F}_{s}$ when he/she was at position
$\mathbf{p}_{o}$ and their partner was at position $\mathbf{p}_{p}$. The
coordinate frames of the robot interfaces coincided, so that if both partners
moved along the same trajectory, they would experience no interaction force,
while if a partner moved away from the other partner, they both experienced a
force pulling them toward each other. The stiffness $k_{s}$ was set to
$120\text{\,}\mathrm{N}\text{\,}{\mathrm{m}}^{-1}$ (same as used by Ganesh et
al.[3]) or $250\text{\,}\mathrm{N}\text{\,}{\mathrm{m}}^{-1}$, depending on
the experiment group and the damping was set to
$b_{s}=$7\text{\,}\mathrm{N}\text{\,}{\mathrm{m}}^{-1}$$. The compliant
connection stiffnesses were chosen such that the task required active
tracking: participants could not completely relax and let the interaction
force pull their hand passively along.

We instructed all participants to track the target as accurately as possible
using continuous and smooth movements; their goal was to minimise the tracking
error as much as they could. We also explained the concept of a visuomotor
rotation and made clear that their goal was to accurately track the target
despite the rotation and not to estimate the magnitude of the rotation. We
informed participants that the interaction forces they would sometimes
experience “involved external forces that would sometimes help the task and
sometimes disturb it”. We did not provide explicit information about the
haptic connection. Participants were not allowed to verbally communicate
during the experiment.

### Experiment groups

The eighty participants were equally divided over four groups: (1) a _solo_
group, (2) an ‘intermittent interaction’ group (denoted by _int. int._), (3) a
‘stiff interaction’ group (_stiff int._), and (4) a ‘continuous interaction’
group (_cont. int._). All participants in each group performed the experiment
in gender- and age-matched pairs. The groups performed the same tracking task
with the same visuomotor rotation for the same amount of trials. We only
changed how often participants in a group interacted (i.e. the amount of
connected trials versus the amount of single trials) or the strength of the
coupling between the partners in the connected trials.

The _solo_ participants performed the experiment in pairs, but they were never
connected. Solo participants thus performed performed the tracking task always
alone; they only performed single trials (S). The solo participants served as
a control group.

The _intermittent interaction_ group intermittently interacted through the
connection with a stiffness of
$k_{s}=$120\text{\,}\mathrm{N}\text{\,}{\mathrm{m}}^{-1}$$. Each block
consisted of sequences of alternative single (S) and connected (C) trials,
resulting in the same trial sequence for each of the four blocks:
{SCSCSCSCSCSCSCSCSCSCS} per block. The first single trial of block 2 (i.e. the
first single trial with the visuomotor rotation) was used to assess the
individual initial tracking error in the visuomotor rotation, denoted by
$E_{s,0}$. The intermittent interaction group is similar to the interaction
group of Ganesh et al.[3] and was used for comparison to their results. We
used the same connection stiffness
($k_{s}=$120\text{\,}\mathrm{N}\text{\,}{\mathrm{m}}^{-1}$$) and damping
($b_{s}=$7\text{\,}\mathrm{N}\text{\,}\mathrm{s}\text{\,}{\mathrm{m}}^{-1}$$).
Our trial sequence differs from Ganesh et al.[3], who used a semi-random
sequence of single and connected trials. We chose to alternate the single and
connected trials to have a consistent spacing of single trials (for comparison
with the solo and other experiment groups) throughout the experiment to
capture each participants motor improvement over time.

We increased the connection stiffness $k_{s}$ to
$250\text{\,}\mathrm{N}\text{\,}{\mathrm{m}}^{-1}$ for the _stiff interaction_
group to investigate the effect of higher interaction strength on individual
motor improvement. Pilot tests showed that the higher connection stiffness
resulted in stronger interaction forces compared to the intermittent
interaction group, yet still allowed for independent movement. The stiff
interaction group used the same alternating sequence of single and connected
trials per block as the intermittent interaction group. Similarly, the first
single trial in the visuomotor rotation blocks was used as the individual
initial tracking error $E_{s,0}$.

The _continuous interaction_ group had different trial sequences per block
compared to the int. int. and stiff int. groups. Block 1 consisted of single
trials only. The partners were always connected in the visuomotor rotation
blocks (blocks 2, 3 and the first half of block 4) to investigate the effect
of more interaction time on individual motor improvement in the visuomotor
rotation. Specifically, we used the the following trial sequence for this
group: [{$21\times$S},{$21\times$C},{$21\times$C},{CCCCCCCCC SSSSSSSSSSSS}].
The last 12 trials in block 4 were single trials to measure the final
individual tracking error of each participants in the visuomotor rotation.
Because the first trial in the visuomotor rotation was a connected trial, we
could not measure the participants’ _individual_ initial tracking error
($E_{s,0}$) on initial exposure to the visuomotor rotation as we did for the
intermittent interaction and stiff interaction groups. Therefore, for the
continuous interaction group only, we introduced the visuomotor rotation in
one single trial in block 1 (trial 13) to measure the individual initial
tracking error ($E_{s,0}$). We chose trial 13 in block 1 based on pilot tests.
We found that the single tracking errors stabilised quickly and remained
relatively constant in the trials before trial 13 (see Fig. 2). Eight single
trials with no visuomotor rotation after trial 13 (trials 14–21) were
sufficient to wash out after effects of the visuomotor rotation in trial 13
before block 2 started. The continuous interaction group were coupled with a
coupling stiffness of
$k_{s}=$120\text{\,}\mathrm{N}\text{\,}{\mathrm{m}}^{-1}$$, same as the int.
int. group.

### Analysis

Motor performance is analysed as the tracking error $E$, which is calculated
as the root mean square of the distance between the target and their cursor of
the last $20\text{\,}\mathrm{s}$ of each $23\text{\,}\mathrm{s}$ trial. We
primarily focused our analysis on the single trials in the visuomotor rotation
blocks, unless explicitly stated otherwise. The tracking errors in the single
and connected trials are denoted by $E_{s}$ and $E_{c}$, respectively. The
tracking error in the first single trial in the visuomotor rotation blocks is
referred to as the initial tracking error $E_{s,0}$.

We studied each participant’s individual adaptation to the visuomotor rotation
by analysing their absolute tracking error improvement and improvement rate in
the single trials. Each participant’s single tracking error improvement
$I_{s}$ was calculated as the difference between the initial tracking error
$E_{s,0}$ and the final single tracking error $E_{s,f}$:
$I_{s}=\left(E_{s,0}-E_{s,f}\right)$. We defined each participant’s final
tracking performance $E_{s,f}$ as their mean tracking performance of the last
five single trials in block 4. Differences in individual improvement across
groups were tested using a linear mixed-effect model with improvement as
dependent variable, interaction group $G$ as fixed effect and pair as random
variable using maximum likelihood estimation.

Short-term visuomotor adaptation has been shown to consist of slow- and fast-
adaptation processes [25]. We fitted a function with two exponents – one with
a fast improvement rate $\lambda_{f}$ and another with a slow improvement rate
$\lambda_{s}$, $\lambda_{s}<\lambda_{f}$ – to the single trials of the
visuomotor rotation blocks of each participant:

$E_{s}=a+b_{s}e^{-\lambda_{s}(t-1)}+b_{f}e^{-\lambda_{f}(t-1)}\textrm{,}\\\ $
(3)

where $t$ is the trial number and $a$, $b_{s}$ and $b_{f}$ are constants. The
effect of interaction group on slow and fast improvement rates was tested by
fitting a linear mixed-effects model with the log-transform of either the slow
or fast improvement rate as dependent variable, interaction group $G$ as fixed
effect and participant pair as random factor using maximum likelihood
estimation. Improvement rates were log-transformed to yield improved residual
distributions, which we assessed through visual inspection of the histograms
and QQ-plots.

Haptic interaction with an initially more skilled partner could facilitate a
participant’s individual improvement and improvement rates[26]. We analysed
whether the difference in initial skill level between the partners impacted
their individual improvement ($I_{s}$) and individual improvement rates
($\lambda_{s}$, $\lambda_{f}$). The initial tracking error $E_{s,0}$ was used
as a representation of each participant’s single initial skill level. The
partner’s relative initial tracking error was calculated for each participant
per pair as $\Delta E^{p}_{s,0}=\left(E_{s,0}-E^{p}_{s,0}\right)/E_{s,0}$,
where $E_{s,0}$ is the participant’s initial tracking error and $E^{p}_{s,0}$
is their partner’s initial tracking error. A positive $\Delta E^{p}_{s,0}$
means that the partner’s initial tracking error was lower than the
participant’s own initial tracking error (i.e. the partner’s initial tracking
performance was better). A negative $\Delta E^{p}_{s,0}$ indicates that the
partner’s initial tracking error was worse than the participant’s initial
tracking error (i.e. the participant was initially better). To assess whether
individual improvement changed significantly with interaction group $G$ and
$\Delta E^{p}_{s,0}$, we fitted a linear mixed-effects model with individual
improvement $I_{s}$ as dependent variable, $\Delta E^{p}_{s,0}$ as co-variate,
interaction group $G$ as fixed effect and pair as random variable. The
improvement rates are also fitted with the same linear mixed-effects model
with either $\log(\lambda_{s})$ or $\log(\lambda_{f})$ as dependent variable.
All linear mixed-effects models were fitted using maximum likelihood.

Lastly, previous work showed that a participant’s tracking error in connected
trials $(E_{c})$ depended on their partner’s relative tracking error during
the subsequent single trials[3, 7, 5]. To corroborate these results and check
whether our interaction paradigm yields similar results, we calculated each
participant’s instantaneous improvement in tracking error _during_ haptic
interaction as $\Delta E_{c}=\left(E_{s}-E_{c}\right)/E_{s}$, where $E_{c}$ is
the tracking error in the connected trial and $E_{s}$ is the tracking error in
the immediately following single trial. The connected trial tracking error
$E_{c}$ could be different for the partners in a pair because of the compliant
connection. A participant’s instantaneous improvement in a connected trial
$\Delta E_{c}$ is compared to their partner’s relative tracking error in the
single trial that immediately followed the connected trial: $\Delta
E^{p}_{s}=\left(E_{s}-E^{p}_{s}\right)/E_{s}$, where $E^{p}_{s}$ is their
partner’s single tracking error and $E_{s}$ is the participant’s tracking
error in the same single trial. A positive $\Delta E^{p}_{s}$ means that the
partner’s performance was better and a negative $\Delta E^{p}_{s}$ indicates
that the partner’s single tracking performance was worse than the
participant’s own single trial performance. $\Delta E_{c}$ and $\Delta E_{s}$
are only calculated for the int. int. and stiff int. group because the
calculation requires pairs of connected and single trials (i.e., CS). Note
that $\Delta E^{p}_{s}$, which we calculate for each pair of CS trials in the
visuomotor rotation blocks, is different from $\Delta E^{p}_{s,0}$, which we
only calculate using the first single trial in the visuomotor rotation blocks.

To test the effect of relative partner performance and group on tracking error
improvement in a connected trial, we fitted a linear mixed-effect model using
maximum likelihood with $\Delta E_{c}$ as dependent variable, interaction
group $G$ (with int. int. and stiff int. group) and $\Delta E^{p}_{s}$ and the
quadratic term $(\Delta E^{p}_{s})^{2}$ as predictors and pair $i$ as random
variable to the data (following similar studies[3, 5]):

$\Delta E_{c}=\beta_{0}+\beta_{1}\,\Delta E^{p}_{s}+\beta_{2}\,(\Delta
E^{p}_{s})^{2}+\beta_{3}\,G+\beta_{4}\,\big{(}\Delta E^{p}_{s}\times
G\big{)}+\beta_{5}\,\big{(}(\Delta E^{p}_{s})^{2}\times
G\big{)}+\epsilon_{i}\textrm{,}$ (4)

where $\beta_{0,\ldots,5}$ are the model coefficients and $\epsilon_{i}$ the
unexplained variation of improvement for each pair $i$. We included the square
of $\Delta E^{p}_{s}$ to include the slope increase with relative partner
performance observed in the data.

We found that participant 11 showed relatively high and variable tracking
errors in the last 12 single trials after the connected trials compared to the
other participants in the cont. int. group (see Supplementary Fig. S1d).
However, the participant’s tracking performance during the connected trials
was similar to the other participants in the cont. int. group. This may
indicate that this participant relied too much on the haptic interaction for
tracking in the visuomotor rotation or that he/she was less motivated to
perform the task alone. As this is the only participant in which we observed
this consistently different tracking behaviour, we decided to not include
participant 11 of the cont. int. group in the analysis.

## Results

We investigated whether haptic interaction between two partners facilitated
their individual motor performance improvement of tracking a continuously
moving target – analysed as the tracking error – while perturbed by a
visuomotor rotation of $80\text{\,}\mathrm{deg}$. Participants first tracked
the target in a baseline block without a visuomotor rotation. We then visually
rotated the on-screen cursor movement clockwise with $80\text{\,}\mathrm{deg}$
with respect to the participant’s actual hand movement in the following three
blocks. Participants were naïve to the visuomotor rotation, which initially
resulted in high tracking errors (i.e., low tracking performance) that reduced
(i.e., improved tracking performance) with practice of tracking the target in
the visuomotor rotation. The partners either performed the tracking task alone
in single trails or haptically coupled through a dual-robot interface in
connected trials. We tested four groups to study whether haptic interaction
improved individual tracking performance in the visuomotor rotation: (1) a
baseline group who performed the tracking task alone (solo group), (2) a group
that intermittently interacted (alternating single and connected trials; int.
int. group) through a compliant connection
($k_{s}=$120\text{\,}\mathrm{N}\text{\,}{\mathrm{m}}^{-1}$$), (3) a group that
also intermittently interacted through with a stiffer coupling
($k_{s}=$250\text{\,}\mathrm{N}\text{\,}{\mathrm{m}}^{-1}$$) than the int.
int. group to study the effect of interaction strength on individual motor
improvement (stiff int. group), and (4) a group that continuously interacted
through a compliant connection
($k_{s}=$120\text{\,}\mathrm{N}\text{\,}{\mathrm{m}}^{-1}$$) in all visuomotor
rotation trials to study the effect of more interaction time on individual
motor improvement (cont. int. group). Our analysis focused on the
participants’ individual tracking performance improvement in the single trials
in the visuomotor rotation blocks, unless explicitly stated otherwise.

Figure 2: Tracking error in the single trials ($E_{s}$) in the baseline and
visuomotor rotation blocks (group mean$\pm$s.e.m.). The initial tracking
errors in the first single trial in the visuomotor rotation ($E_{s,0}$) for
all groups are explicitly labelled; note that the cont. int. group performed
one single trial with visuomotor rotation in the baseline block. The vertical
lines delineate the blocks.

### Haptic human-human interaction does not yield more motor improvement or
faster improvement rates

All groups show clear and similar single trial tracking error ($E_{s}$)
improvements in the visuomotor rotation blocks (Fig. 2). A visual inspection
of the data shows only small differences in tracking errors between groups.
Before analysing the participant’s individual improvement in the visuomotor
rotation, we first checked whether the introduction of the visuomotor rotation
initially increased single tracking errors similarly for all groups. We
calculated the initial increase in tracking error using the last single trial
in block 1 (without visuomotor rotation) and first single trial in block 2
(with visuomotor rotation) for the solo, int. int. and stiff int. groups. For
the cont. int. group we calculated the increase in tracking error between
trial 12 (single trial without visuomotor rotation) and trial 13 (single trial
with visuomotor rotation) in block 1. Overall, we found no significant
differences in the initial tracking error increase due to the visuomotor
rotation between groups ($\chi^{2}(3)=2.90$, $p=0.407$).

To analyse the tracking error improvement curves in the visuomotor rotation
blocks, we calculated each participant’s _individual_ single tracking error
improvement $I_{s}$ and each participant’s slow and fast improvement rates
based on their single trials ($\lambda_{s}$ and $\lambda_{f}$, respectively).
We found no significant differences in individual tracking error improvement
between groups (Fig. 3a, effect of group G on improvement: $\chi^{2}(3)=3.22$,
$p=0.360$). These results indicate that haptic interaction does not result in
more individual performance improvement compared to practising the task solo.

In addition, haptic interaction did not result in significantly different slow
and fast improvement rates for the solo, intermittent interaction and stiff
interaction groups, see Fig. 3b (effect of group $G$ on slow improvement rate:
$\chi^{2}(2)=1.09$, $p=0.582$ and fast improvement rate: $\chi^{2}(2)=2.30$,
$p=0.316$). Because the participants in the continuous interaction group did
not perform any single trials in the early stages while adapting to the
visuomotor rotation, we could not extract their individual motor improvement
rates.

Figure 3: Tracking error improvement and improvement rates in the visuomotor
rotation blocks. (a) Single tracking error improvement for all groups. (b)
Individual slow $\left(\lambda_{s}\right)$ and fast $\left(\lambda_{f}\right)$
improvement rates for the solo, intermittent interaction, and stiff
interaction groups. While we used the log-transform of learning rates, we
plotted the non-transformed data here. The markers show the participants’
individual data.

### The effect of the partner’s relative initial skill level on individual
improvement and improvement rates

Although we found no significant effect of haptic interaction on individual
improvement on a group level, some participants could have benefited from the
haptic interaction depending on the difference in initial skill level between
the partners in a pair (i.e., the difference between $E_{s,0}$ of both
partners) [27, 28, 10]. For example, haptic interaction with an initially
highly skilled partner could result in a higher individual improvement $I_{s}$
compared to interaction with a similarly skilled partner or compared to a
participant who practised the task without interaction with a better partner.

To test whether haptic interaction benefited individual improvement on top of
the potential effect of the initial skill difference between partners, we need
to compare the interaction groups to participants who never interacted. The
analysis focuses on the partner’s relative initial error in the first single
trial $(\Delta E^{p}_{s,0})$ before any connected trials occurred in the
visuomotor rotation blocks, and compares $\Delta E^{p}_{s,0}$ with the overall
individual improvement ($I_{s}$, $\lambda_{s}$, and $\lambda_{f}$). We can,
therefore, include the solo group as a control group that never had connected
trials by calculating $\Delta E^{p}_{s,0}$ for the solo pairs and comparing it
to each solo participant’s individual improvement and improvement rates. Note
that although the solo participants performed the experiment in pairs, they
are not considered to be partners, as they were never connected and were thus
not able to influence each other (in principle we could even randomly select
pairs of solo participants for the analysis). For consistency with the
interaction groups, we still refer to the solo participants in a pair as
partners. If haptic interaction has an additional effect on individual
improvement on top of the partner’s relative initial skill level, we would
expect a difference between the solo group and the interaction groups,
depending on the skill difference. Fig. 4 shows each participant’s individual
improvement $I_{s}$ and improvement rates ($\lambda_{s}$ and $\lambda_{f}$)
versus their partner’s relative initial error $\Delta E^{p}_{s,0}$.

A participant’s individual improvement $I_{s}$ depended significantly on their
partner’s relative initial error $\Delta E^{p}_{s,0}$ ($\chi^{2}(1)=121.07$,
$p<10^{-6}$, see Fig. 4a). This indicates that the more skilled their partner
initially was compared to their own initial skill level, the higher the
participant’s own individual improvement $I_{s}$. We found no significant
differences between groups $G$ ($\chi^{2}(3)=3.27$, $p=0.352$). The
interaction effect ($\Delta E^{p}_{s,0}\times G$) was also not significant
($\chi^{2}(3)=6.69$, $p=0.083$). Hence, individual improvement depends
significantly on the partner’s relative initial performance, and this effect
is similar for all groups including the solo group.

Figure 4: The partner’s relative initial tracking error $\Delta
E^{p}_{s,0}=\left(E_{s,0}-E^{p}_{s,0}\right)/E_{s,0}$ versus (a) individual
improvement $I_{s}$ and (b) individual improvement rates ($\lambda_{s}$ and
$\lambda_{f}$). We refer to the two solo participants in each pair as partners
for consistency with the pairs in the interaction groups, although they were
never connected.

The improvement rates did not depend on the partner’s relative initial error
($\Delta E^{p}_{s,0}$) or interaction group $G$ (Fig. 4b). The slow
improvement rate ($\lambda_{s}$) neither depended on the partner’s relative
initial performance ($\chi^{2}(1)=0.33$, $p=0.564$) nor on the interaction
group ($\chi^{2}(2)=1.12$, $p=0.570$). The interaction effect $(\Delta
E^{p}_{s,0}\times G)$ was also not significant ($\chi^{2}(2)=5.08$,
$p=0.078$). Similarly, we found no effect of partner’s relative initial
performance on the fast improvement rate ($\lambda_{f}$; $\chi^{2}(1)=0.22$,
$p=0.639$) and the fast improvement rate did not depend on interaction group
($\chi^{2}(2)=2.57$, $p=0.277$). The interaction effect $(\Delta
E^{p}_{s,0}\times G)$ was also not significant ($\chi^{2}(2)=1.07$, $p=0.586$)
for the fast improvement rate.

### Tracking performance improves during haptic interaction depending on the
partner’s relative single tracking performance

A common finding in similar haptic human-human interaction studies is that
haptic interaction improves individual tracking error _during_ interaction
($\Delta E_{c}$) depending on the partner’s relative tracking error in the
subsequent single trial ($\Delta E^{p}_{s}$) [3, 18, 7, 6]. Although our focus
in this paper is on analysing single trial improvement, we also analysed
$\Delta E_{c}$ versus $\Delta E^{p}_{s}$ for the int. int. and stiff int.
groups to check whether our data corroborates previous studies. Note that this
section is different from the analysis in the previous section, in which we
analysed _individual_ improvement $I_{s}$ based on _single trials_ only for
all groups, whereas here, we analyse the immediate effect of haptic
interaction during the _connected trials_ for the int. int. and stiff int.
groups only. Another difference is that here, $\Delta E^{p}_{s}$ is calculated
for each single trial after a connected trial in the visuomotor rotation
blocks, which is different from the partner’s relative _initial_ error $\Delta
E^{p}_{s,0}$ used in the previous section, which is only calculated using the
first single trial in the visuomotor rotation blocks.

The partner’s relative error and interaction group (int. int. or stiff int.
group) had a significant effect on the tracking error improvement during
interaction $\Delta E_{c}$ (see Fig. 5). Specifically, the quadratic term
$(\Delta E^{p}_{s})^{2}$ (see Eq. 5) had a significant effect on connected
tracking improvement ($\chi^{2}(1)=5.82$, $p=0.016$), indicating that the
connected improvement increased with a greater slope rise with a progressively
better partner. The interaction effect $\big{(}(\Delta E^{p}_{s})^{2}\times
G\big{)}$ was also significant ($\chi^{2}(1)=3.98$, $p=0.046$), indicating
that $(\Delta E^{p}_{s})^{2}$ had a significantly larger effect on $\Delta
E_{c}$ for the stiff int. group compared to the int. int. group. In other
words, interacting through a stiffer coupling resulted in progressively more
connected improvement compared to a weaker coupling, in particular for
participants whose partners were better at the task (participants with $\Delta
E^{p}_{s}>$0\text{\,}\mathrm{\char 37\relax}$$).

Our results also showed that interaction can hinder tracking performance
compared to single performance (i.e. $\Delta E_{c}<$0\text{\,}\mathrm{\char
37\relax}$$). When interacting with a partner whose relative single error was
approximately $\Delta E^{p}_{s}<$-30\text{\,}\mathrm{\char 37\relax}$$ for the
int. int. group and $\Delta E^{p}_{s}<$-39\text{\,}\mathrm{\char 37\relax}$$
for the stiff int. group, the fitted model predicted that interaction would
hinder performance. Interestingly, even interaction with a partner who was
slightly worse at the task (e.g. $$-30\text{\,}\mathrm{\char
37\relax}$\leq\Delta E^{p}_{s}\leq$0\text{\,}\mathrm{\char 37\relax}$$ for the
int. int. group and $$-39\text{\,}\mathrm{\char 37\relax}$\leq\Delta
E^{p}_{s}\leq$0\text{\,}\mathrm{\char 37\relax}$$ for stiff int.) still
resulted in an improvement during interaction ($\Delta
E_{c}\geq$0\text{\,}\mathrm{\char 37\relax}$$).

Figure 5: Tracking error improvement during interaction in a connected trial
($\Delta E_{c}=\left(E_{s}-E_{c}\right)/E_{s}$) versus relative partner
performance in the subsequent single trial ($\Delta
E^{p}_{s}=\left(E_{s}-E^{p}_{s}\right)/E_{s}$) for the int. int. and stiff
int. groups.

## Discussion

We investigated whether haptic interaction between two humans performing the
same target tracking task in a visuomotor rotation perturbation enhanced their
individual motor adaptation in terms of motor improvement and improvement rate
compared to someone who practised the task alone. This work was motivated by
the results of Ganesh et al.[3], who found a significant benefit of haptic
interaction on individual motor improvement in the same motor task. Here, we
repeated their study using the same interaction paradigm and motor adaptation
task. In addition to repeating Ganesh et al.[3], in which we compared a group
in which partners intermittently interacted with a solo group, we also added a
group who spent more time interacting and a group who intermittently
interacted through a stronger haptic connection. In contrast to Ganesh et
al.[3], we found no effect of intermittent interaction on individual
improvement or improvement rate compared to the solo group. More interaction
time (e.g. more connected trials) to allow the participants to benefit more
from the interaction – if those benefits would have been present in our study
– did not improve individual improvement either. Interaction through a
stronger connection also resulted in similar individual improvement. Although
we found an effect of the partner’s initial performance level on individual
improvement, there was no difference between the interaction and solo groups.
Improvement rates did not depend on the partner’s relative skill level.

Our results are in line with observations in motor adaptation literature and
robot-assisted motor skill acquisition studies. A consistent finding in this
field is that movement errors drive motor adaptation; reducing movement
errors, for example through haptic assistance, does generally not facilitate
individual motor improvement [26, 4, 11]. In accordance with other haptic
human-human interaction studies [3, 5, 6, 10, 9, 7], we found that tracking
errors generally were smaller _during_ haptic interaction ($\Delta E_{c}>0$)
depending on the partner’s relative single trial error ($\Delta E^{p}_{s}$),
specifically for participants whose partners had better single tracking
performance. Haptic interaction could be seen as a compliant guidance that
allows each participant to independently perform the task, but still benefit
from the error-correcting guidance of their partner. This gives the
interacting partners an incorrect good impression of their tracking
performance, but reduces tracking error as driving training signal for their
own motor adaptation [26]. This observation is supported by several studies
that showed that robot-generated haptic guidance temporarily improved motor
performance while the participant received the guidance, but did not improve
subsequent individual motor skill acquisition [12, 13, 14, 15, 16].

Although the group-based analysis found no benefit of haptic interaction on
individual improvement, we also investigated whether some participants could
have benefited from the interaction depending on their partner’s relative
initial skill level. Avila-Mireles et al.[10] found that haptic interaction
between two partners who were naïve to the task resulted in more individual
motor skill acquisition benefits compared to interaction with an expert.
Interaction with an expert only helped if the novice had prior solo experience
in the task. Kager et al.[9] found no effect of partner skill level on
individual tracking skill improvement in a tracking task similar to ours. We
did not control the initial skill difference between the partners, resulting
in a range of relative skill differences. We found that haptic interaction
with an initially more skilled partner resulted in higher individual
improvement. However, this effect could also be because participants with a
high initial single tracking error – who are more likely to interact with a
superior-performing partner – had more room to improve. This might be related
to the task and further research is warranted. We used the skill difference
and not each participant’s own absolute initial tracking error, as the skill
difference of the partner could have impacted to what extent haptic
interaction helped or even hindered individual improvement. Lastly, and more
importantly, we found no additional benefit of haptic interaction for
individual improvement on top of the partner’s initial skill level compared to
solo participants (the effect of group $G$ and interaction effect $(\Delta
E^{p}_{s,0}\times G)$ on $I_{s}$ were not significant). Hence, haptic
interaction with a more-skilled or less-skilled partner does not benefit
individual improvement compared to practising the task alone, supporting our
group-based conclusion.

Although we used the same interaction paradigm and motor task as Ganesh et
al.[3], a number of differences in the experiment design and analysis could
explain the different results. First, our solo and intermittent interaction
groups used double the number of participants compared to the solo and
interaction groups of Ganesh et al.[3] (twenty participants or ten pairs per
group in our study, compared to ten participants or five pairs). In addition,
we tested another forty participants – albeit with slight variations of the
interaction paradigm – without any effect on individual improvement.

Second, we calculated each participant’s own single improvement $I_{s}$ on
their own initial tracking error $E_{s,0}$ and own final tracking error
$E_{s,f}$, whereas Ganesh et al.[3] based improvement on the group-mean of the
initial tracking error (see Fig 1c in their paper). As our data showed a
reasonable spread in initial tracking error between participants indicating
differences in initial skill levels (see Supplementary Fig. S1), we believe
that calculating individual improvement based on each individual’s initial and
final tracking performance is a more precise representation for each
individual’s improvement. In fact, if we would have calculated and analysed
each participant’s improvement based on their group-average initial tracking
error, we would find that the solo group improved _more_ than the intermittent
and stiff interaction groups and similar to the continuous interaction group.
Hence, our conclusion would have been that intermittent haptic interaction
would even _impede_ motor improvement compared to practising the task solo
(see Supplementary Fig. S2).

Third, we used an alternating sequence of single and connected trials for the
intermittent interaction group, whereas Ganesh et al.[3] used a semi-
randomised sequence. However, we believe that our trial sequence had little
effect on the participant’s individual tracking improvement, based on a direct
comparison of the solo and continuous interaction groups. These two groups had
two opposite extremes of the possible trial sequences; either all single or
all connected during the initial adaptation phases (blocks 2 and 3). The solo
and continuous interaction groups showed no significant difference in
improvement compared to the other groups, indicating that the trial sequence
or amount of interaction time in our task did not seem to significantly affect
individual improvement.

Lastly, our robot interface’s workspace (diameter of $20\text{\,}\mathrm{cm}$)
was smaller than that of Ganesh et al.[3] (diameter of
$30\text{\,}\mathrm{cm}$) due to hardware limitations. As a result, we had to
design a different target trajectory that required a smaller range of motion.
This resulted in smaller improvement in tracking errors in the single and
connected trials compared to Ganesh et al.[3]. Still, our data showed
prominent yet similar individual improvement curves for all groups (see Fig.
2) and we found significant improvement of tracking error during interaction
in the connected trials similar to Ganesh et al.[3] and other studies using
the same interaction paradigm [5, 7, 6] (Fig. 5).

Although the focus of this paper is on the effect of haptic interaction on
individual motor improvement, we also analysed the tracking error improvement
during interaction in the connected trials compared to the subsequent single
trials (denoted by $\Delta E_{c}$) to check whether the haptic interaction
paradigm yielded similar result as previous haptic human-human interaction
studies work (specifically [6, 7, 3]). The extent to which haptic interaction
helps ($\Delta E_{c}>$0\text{\,}\mathrm{\char 37\relax}$$) or hinders ($\Delta
E_{c}<$0\text{\,}\mathrm{\char 37\relax}$$) tracking performance depends on
the partner’s relative single performance $E^{p}_{s}$. Interaction with a
better partner improved tracking performance. Interestingly, even when a
partner’s relative single error was slightly worse (e.g.,
$$-30\text{\,}\mathrm{\char 37\relax}$<\Delta
E^{p}_{s}<$0\text{\,}\mathrm{\char 37\relax}$$), haptic interaction still
improved performance during interaction. Interaction hindered tracking
performance for a partner relative error of $\Delta
E^{p}_{s}<$-30\text{\,}\mathrm{\char 37\relax}$$. These observations
corroborate previous studies [6, 7, 3]. Lastly, our observation that a stiffer
interaction yield progressively more tracking error benefit is also observed
in previous work [6].

It is likely that the benefits of haptic human-human interaction on motor
skill acquisition are task- and instruction-dependent. For example, Avila-
Mireles et al.[10] found that a uni-manual skill learned during haptic
interaction with a human partner could benefit the participant’s skill level
in the same task that the participant would now perform bi-manually. We only
investigated short-term motor improvement over a $2\text{\,}\mathrm{h}$
session; the effects of haptic human-human interaction on long-term skill
acquisition and retention remain unknown. Lastly, we neither made the
participants explicitly aware of the connection, nor assigned roles, such as
teacher-student. Assigning roles or making partners aware of the connection
could influence their interaction strategies and motor improvement of the
partners[29].

In conclusion, our study found no benefit of haptic interaction between
partners on short-term individual motor improvement in a tracking task with a
large visuomotor rotation. We could not corroborate the findings of Ganesh et
al.[3] on individual motor improvement. Recalling the example of a
physiotherapist assisting a patient to regain motor abilities, we are aware of
the limited generalisability of our in-the-lab-results to realistic motor
tasks. Haptic human-human interaction likely plays an important role in motor
skill acquisition in our lives. Even though we studied three aspects of haptic
human-human interaction in our specific tracking task – interaction time,
interaction strength and partner skill level – further investigation into the
mechanisms behind the possible benefits of haptic human-human interaction is
warranted. Furthermore, as we minimised any social interaction between partner
except through the interaction force, we need to take important aspects of
joint action such as (conscious) motor coordination and role distribution into
account in future studies.

## Data availability

Data and analysis scripts that support the findings of this study are
available through the following link:
https://doi.org/10.4121/uuid:a4f1add7-892d-456e-8b8a-75a708d01f15.

## Acknowledgements

We thank Jeanine van Bruggen, Frits Wiersma and Ingrid van den Heuvel for
their assistance with the experiments. We also thank Jared Atkinson and Victor
Sluiter for their insightful comments on the manuscript. This research is
supported by the Netherlands Organisation for Scientific Research (NWO),
project no. 13524.

## Author contributions statement

N.B. conceived the experiments, conducted the experiments and wrote the
manuscript. All authors analysed and discussed the results and reviewed the
manuscript.

## Additional information

Competing interests The authors declare no competing interests.

## References

  * [1] Sebanz, N., Bekkering, H. & Knoblich, G.  Joint action: Bodies and minds moving together.  _Trends in Cognitive Sciences_ 10, 70–76 (2006).
  * [2] Tomasello, M., Carpenter, M., Call, J., Behne, T. & Moll, H.  Understanding and sharing intentions: The origins of cultural cognition.  _Behavioral and Brain Sciences_ 28 (2005).
  * [3] Ganesh, G., Takagi, A., Osu, R., Yoshioka, T. & Kawato, M.  Two is better than one: Physical interactions improve motor performance in humans.  _Scientific Reports_ 4 (2014).
  * [4] Shadmehr, R., Smith, M. A. & Krakauer, J. W.  Error correction, sensory prediction, and adaptation in motor control.  _Annual Review of Neuroscience_ 33, 89–108 (2010).
  * [5] Takagi, A., Ganesh, G., Yoshioka, T., Kawato, M. & Burdet, E.  Physically interacting individuals estimate the partner’s goal to enhance their movements.  _Nature Human Behaviour_ 1, 0054 (2017).
  * [6] Takagi, A., Usai, F., Ganesh, G., Sanguineti, V. & Burdet, E.  Haptic communication between humans is tuned by the hard or soft mechanics of interaction.  _PLoS Computational Biology_ 14, e1005971–17 (2018).
  * [7] Beckers, N., Keemink, A. Q. L., van Asseldonk, E. & van der Kooij, H.  Haptic human-human interaction through a compliant connection does not improve motor learning in a force field.  In _11th Int. Conf. of EuroHaptics 2018_ , 333–344 (2018).
  * [8] van der Wel, R. P. R. D., Knoblich, G. & Sebanz, N.  Let the force be with us: Dyads exploit haptic coupling for coordination.  _Journal of Experimental Psychology: Human Perception and Performance_ 37, 1420–1431 (2011).
  * [9] Kager, S. _et al._ The effect of skill level matching in dyadic interaction on learning of a tracing task.  In _16th IEEE Int. Conf. on Rehabilitation Robotics (ICORR)_ , 824–829 (2019).
  * [10] Avila Mireles, E. J., Zenzeri, J., Squeri, V., Morasso, P. & De Santis, D.  Skill learning and skill transfer mediated by cooperative haptic interaction.  _IEEE Trans. on Neural Systems and Rehabilitation Engineering_ (2017).
  * [11] Wei, K. & Körding, K. P.  Relevance of error: what drives motor adaptation?  _Journal of Neurophysiology_ 101, 655–664 (2009).
  * [12] Winstein, C. J., Pohl, P. S. & Lewthwaite, R.  Effects of physical guidance and knowledge of results on motor learning: Support for the guidance hypothesis.  _Research Quarterly for Exercise and Sport_ 65, 316–323 (1994).
  * [13] van Asseldonk, E. H. F., Wessels, M., Stienen, A. H. A., van der Helm, F. C. T. & van der Kooij, H.  Influence of haptic guidance in learning a novel visuomotor task.  _Journal of Physiology - Paris_ 103, 276–285 (2009).
  * [14] O’Malley, M. K., Gupta, A., Gen, M. & Li, Y.  Shared control in haptic systems for performance enhancement and training.  _Journal of Dynamic Systems, Measurement, and Control_ 128, 75–11 (2006).
  * [15] Marchal-Crespo, L., Lopez-Oloriz, J., Jaeger, L. & Riener, R.  Optimizing learning of a locomotor task: Amplifying errors as needed.  In _36th Int. Conf. of the IEEE Engineering in Medicine and Biology Society_ , 5304–5307 (2014).
  * [16] Heuer, H. & Lüttgen, J.  Robot assistance of motor learning: A neuro-cognitive perspective.  _Neuroscience & Biobehavioral Reviews_ 56, 222–240 (2015).
  * [17] Mattar, A. G. & Gribble, P. L.  Motor learning by observing.  _Neuron_ 46, 153–160 (2005).
  * [18] Reed, K. B. & Peshkin, M. A.  Physical collaboration of human-human and human-robot teams.  _IEEE Trans. on Haptics_ 1, 108–120 (2008).
  * [19] Groten, R., Feth, D., Klatzky, R. L. & Peer, A.  The role of haptic feedback for the integration of intentions in shared task execution.  _IEEE Trans. on Haptics_ 6, 94–105 (2013).
  * [20] Sawers, A. _et al._ Small forces that differ with prior motor experience can communicate movement goals during human-human physical interaction.  _Journal of NeuroEngineering and Rehabilitation_ 14, 8 (2017).
  * [21] Mojtahedi, K., Whitsell, B., Artemiadis, P. & Santello, M.  Communication and inference of intended movement direction during human-human physical interaction.  _Frontiers in Neurorobotics_ 11, 127–23 (2017).
  * [22] Kucukyilmaz, A., Sezgin, T. M. & Basdogan, C.  Intention recognition for dynamic role exchange in haptic collaboration.  _IEEE Trans. on Haptics_ 6, 58–68 (2013).
  * [23] Knoblich, G. & Jordan, J. S.  Action coordination in groups and individuals: Learning anticipatory control.  _Journal of Experimental Psychology: Learning, Memory, and Cognition_ 29, 1006–1016 (2003).
  * [24] Oldfield, R. C.  The assessment and analysis of handedness: the Edinburgh inventory.  _Neuropsychologia_ 9, 97–113 (1971).
  * [25] Smith, M. A., Ghazizadeh, A. & Shadmehr, R.  Interacting adaptive processes with different timescales underlie short-term motor learning.  _PLOS Biology_ 4, 1035–1043 (2006).
  * [26] Marchal-Crespo, L., Bannwart, M., Riener, R. & Vallery, H.  The effect of haptic guidance on learning a hybrid rhythmic-discrete motor task.  _IEEE Trans. on Haptics_ 1–16 (2013).
  * [27] Guadagnoli, M. A. & Lee, T. D.  Challenge point: a framework for conceptualizing the effects of various practice conditions in motor learning.  _Journal of Motor Behavior_ 36, 212–224 (2004).
  * [28] Marchal-Crespo, L. _et al._ The role of skill level and motor task characteristics on the effectiveness of robotic training: first results.  In _IEEE Int. Conf. on Rehabilitation Robotics (ICORR)_ , 151–156 (2015).
  * [29] Granados, C. & Wulf, G.  Enhancing motor learning through dyad practice: Contributions of observation and dialogue.  _Research Quarterly for Exercise and Sport_ 78, 197–203 (2007).


# GRCNN: Graph Recognition Convolutional Neural Network for Synthesizing
Programs from Flow Charts

Lin Cheng  Zijiang Yang L. Cheng and Z. Yang are with the Department of
Computer Science, Western Michigan University, Michigan, USA (emails:
<EMAIL_ADDRESS>and zijiang.yang@wmich.edu).

###### Abstract

Program synthesis is the task to automatically generate programs based on user
specification. In this paper, we present a framework that synthesizes programs
from flow charts that serve as accurate and intuitive specifications. In order
doing so, we propose a deep neural network called GRCNN that recognizes graph
structure from its image. GRCNN is trained end-to-end, which can predict edge
and node information of the flow chart simultaneously. Experiments show that
the accuracy rate to synthesize a program is 66.4%, and the accuracy rates to
recognize edge and nodes are 94.1% and 67.9%, respectively. On average, it
takes about 60 milliseconds to synthesize a program.

program synthesis, flow chart, specification, graph recognition, CNN.

##  1 Introduction

Program synthesis enables people to program computers without training in
coding. It has been used used in many domains such as data wrangling, graphs,
and code repair [1]. A good example is FlashFill [2], which allows spreadsheet
users to provide a few examples and generates a program that conforms to the
examples.

To synthesize a program, specification must be provided. Specification in
formal language can accurately represent the user intent and are used in
deductive program synthesis [3]. However, very few have the knowledge of
formal language, so it cannot benefit most end users. Under-specification is
used in programming by example [2, 4], and programming by demonstration [5].
Under-specification does not require language knowledge and is accessible by
most end users. However, because there may be more than one program that
satisfies the specification, how to choose the correct program that captures
the user intent is still an open problem.

In this paper, we propose a new technique called GRCNN (Graph Recognition
Convolutional Neural Network) that takes as input flow charts as accurate
specification, and uses deep convolutional neural network (CNN) to analyze a
given image, and compiles the obtained information into program code. GRCNN is
an end-to-end network that shares the computation of a rich convolutional
feature vector and predicts edge and node information simultaneously. A flow
chart is a diagram that represents the workflow of a program. It is used
widely in textbooks to teach coding and illustrate programs. Moreover, flow
charts are intuitive, which allows users to focus on programming logic instead
of language details. Thus they are also frequently used in program designing
stage. Figure 1 shows a flow chart representing the workflow of a function
that computes the absolute value of an input. Because flow charts can
accurately describe programs, synthesizing program by flow chart may precisely
capture users’ intent.

Due to recent progress in deep learning, CNN is powerful enough to detect and
recognize complicated information from image. Therefore, we can obtain a graph
data structure from a flow chart with the help of deep CNN. Because neural
networks are differentiable functions, our method does not suffer from the
combinatorial explosion problem that plagues traditional program synthesis
methods.

Figure 1 givens an overview of our approach. First, an image of flow chart is
resized to a fixed size and fed to GRCNN. Then, GRCNN generates the graph
information including an adjacent matrix for the edges and a list of strings
for the nodes. Finally, we compile the graph information to source code.

0 1 2 00 0 0 10 0 0 10 0 0 0“x >0” “a = x” “a = -x” “return a” if (x >0) a = x
elsea = -x return a  Figure 1: Overview of GRCNN. The input is the flow chart
of abs function. The middle is the adjacent matrix and text of each node
generated by GRCNN. The output is the synthesized source code.

Our evaluation showed that it is feasible to share the convolutional vector
between edge and node networks. Experiments on our synthetic test dataset show
that the accuracy rate to synthesize a program is 66.4%, and the accuracy
rates to predict edges and nodes are 94.1% and 67.9%, respectively.
Experiments on another dataset, which is manually converted from a textbook,
show that that the accuracy rates to synthesize a program is 63.6%, and the
accuracy rates to predict edge and nodes are 72.7% and 81.8%, respectively.
The average time to synthesize a program is about 60 milliseconds.

In summary, the main contributions include:

  * •

We propose a deep neural network that parses graph edge and node information
from flow chart.

  * •

We propose to use flow chart as an accurate and intuitive specification for
program synthesis.

  * •

We have implemented a prototype and conducted empirical study.

conv

conv

conv

+$\cdots$

conv

conv

+Backbone Net

conv

fc

fc

${0}$${1}$${2}$${0}$${0}$${0}$${0}$${1}$${0}$${0}$${0}$${1}$${0}$${0}$${0}$${0}$Edge
Net

conv

conv

conv

$k$ scores$k$ boxesDetection Net

conv

conv

conv

+$\cdots$

conv

conv

+Recognition Net“x>=0” Figure 2:  Architecture of GRCNN

##  2 Related Work

Traditional methodology of program synthesis is to construct a program space
and design search algorithms to find a solution that satisfies the
specification. The program space usually grows exponentially with the size of
the target program. Different methods are proposed to speedup the search. For
example, Flashfill, [2], synthesizes string-transforming programs given input-
output examples. It uses dynamic programming to speedup the search. Morpheus,
[6], enumerates nested queries and prunes by grouping programs with same
input-output pairs. Sketch, [7], and Sqlsol, [8], encode the synthesis problem
into logic constraints and delegate the searching algorithm to modern SMT
solvers. This approach can boost the performance because modern SMT solvers
are implemented for efficiency. Though different methods are proposed to
speedup the synthesis process, the underlying complexity is unchanged and
scalability is still an issue when it comes to large programs. Another issue
is how to capture users’ intent. Typically, synthesis algorithms terminate
when the first solution is found or the best-ranking solution is found. They
may ask a user to provide more examples. However, there is no guarantee that
the solution precisely captures the users’ intent.

Recently researchers propose to use machine learning to speed up the search
for synthesized program. Morpheus [4] uses statistics to rank $R$ program
sketches, and uses the rank to guide the search. DeepCoder [9] augments beam
search with deep learning recommendation, and the speedup is significant.
However, the underlying complexity is still unchanged.

Researchers proposed natural language based program synthesis techniques.
SQLizer, [10], synthesizes SQL queries from natural language. Locascio et al,
[11], synthesizes regular expressions from natural language. GRCNN differs
from these approaches in that our input is different. Flow charts can
accurately specify the users’ intent, while natural language is ambiguous.

Faster RCNN, [12], and LPRNet, [13], are closely related to our work. Two
subnets of GRCNN are built following their ideas. Faster RCNN is a deep neural
network for object detection. It slides a small window on a convolutional
feature and generates box proposals relative to anchors at each position. The
box proposal is used to crop the image for a classifier to detect the class of
the object in it. Because of the shape and positioning of nodes in a flow
chart are different from those in general object-detection jobs, GRCNN chooses
different anchors and methods to select the proposals. LPRNet is a license
plate recognizing deep convolutional neural network. It reads an image of a
license plate and generates a sequence that preserves the spatial order of the
characters. A feature of LPRNet is that it has only one deep convolutional
network, while other work has both a CNN for feature extraction and a
recurrent neuralnetwork for prediction. This feature helps to limit the number
of subnets in GRCNN, since it already has four subnets. Instead of taking a
license plate as input, GRCNN takes as input a crop of image whose boundary is
predicted by another network.

##  3 Network

In this section, we describe the network and its subnet. Then, we describe the
loss function and training details.

The input of GRCNN is an image of a flow chart. A flow chart is a graph
diagram that represents the work flow of program [14]. Standard flow charts
has several shapes for nodes. In this work, we consider all shapes of nodes as
rectangles. Because we use heuristic algorithms to generate ground truth data
for bounding boxes, designing and implementing algorithms for every shape
requires considerable engineering effort . However, given ground truth data
for bounding boxes, our algorithm can be trained the same way to handle other
shapes.

The input flow chart is resized to a fixed size (400 pixels for height and 200
pixels for width) before being fed to the neural network. If both height and
width is less then the fixed size, we pad the image by zeroes. Otherwise, we
interpolate the image to the fixed size. The output of GRCNN is a graph
representation of the flow chart, including an adjacent matrix for the edge
representation and a list of text for content in nodes. In addition to the
original text, we insert an _id_ to the text to match the text with the _id_
-th row of adjacent matrix. In this project, we use an alphabet of 50
characters including English characters in lower case, digit characters,
arithmetic operators and other symbols. The source code that our algorithm
produces supports sequential statements and control structure including IF-
ELSE, WHILE loop, DO-WHILE loop.

Figure 2 is the overview of GRCNN. GRCNN has four parts: backbone network,
edge detection network, node detection network, and node recognition network.
The backbone network takes as input the raw image of the flow chart and
outputs a feature vector. The feature vector is fed to both the edge detection
to produce the adjacent matrix of the flow chart, and the node detection
network to generate the bounding boxes of each node, which is used to crop the
node from the original image. Then, the crop of each node is fed to the node
recognition network to generate the text in the node.

###  3.1 Backbone Network

The backbone network is a deep CNN that takes as input an image of size
$C_{in}\times H\times W$. The output is a rich feature vector, which is later
used as input to the edge network and the node detection network. The backbone
network is a sequence of four basic blocks. Table 1 shows the architecture of
a basic block, which takes as input a feature vector of $C_{in}$ channels and
outputs a feature vector of $C_{out}$ channels and the same height and width.
Each convolutional layer in the basic block is followed by a batch
normalization layer and a ReLU activation layer. Each basic block is followed
by a max-pooling layer, except the third one. During training, a dropout layer
(p=0.1) is added after each pooling layer.

Optionally, the backbone network can be made a residual learning network
(ResNet), [15], by modifying the basic block as follows. We perform a down-
sampling on the input vector with $1\times 1$ convolutional layer to $C_{out}$
channels, and add the result to the original output as new output.

Table 1: Basic Block Architecture. Layer Type | Parameters
---|---
Input | $C_{in}\times H\times W$
Conv2D | $C_{out}/4,3\times 3$
Conv2D | $C_{out}/4,3\times 3$
Conv2D | $C_{out}/4,3\times 3$
Conv2D | $C_{out},3\times 3$
Table 2: Backbone Network Architecture Layer Type | Parameters
---|---
Input | C $\times$ H $\times$ W
Basic Block | 15, 3 $\times$ 3
Max Pool | 2 $\times$ 2
Basic Block | 50, 3 $\times$ 3
Max Pool | 2 $\times$ 2
Basic Block | 200, 3 $\times$ 3
Basic Block | 400, 3 $\times$ 3
Max Pool | 2 $\times$ 2

###  3.2 Edge Network

The edge network takes as input the feature vector generated by the backbone
network and outputs an adjacent matrix, which is the edge representation of
the flow chart. The element of the adjacent matrix at row $i$ and column $j$
has three possible values: 0, 1, 2, which encode no edge, normal edge or YES
branch of a decision node, NO branch of a decision node from node $i$ to node
$j$, respectively. Because the number of nodes in the flow chart may vary, we
pad the adjacent matrix to a fixed size $PAD$ by zeroes. We set $PAD=6$ in
this work. We encode the three values in the adjacent matrix with one-hot-
vectors of length 3. Therefore, the edge network outputs a vector of
$PAD\times PAD\times 3$ scores.

Table 3 shows the architecture of the edge network. It first performs a
convolutional layer activated by a ReLU function. Then, a max pooling layer is
performed followed by two linear layers actived by Tanh function. We observed
the training converges significantly faster when using Tanh activation
function in the linear layers than using other activation functions.

The loss function to train the edge network is the multi-class multi-
classification hinge loss . Equation 1 is the formula of the loss function,
where x is the input vector and y is the target class indices.

$l_{e}(x,y)=\sum_{i,j}\frac{max(0,1-(x[y[j]]-x[i]))}{x.size(0)}$ (1)

Table 3: Edge Network Architecture Layer Type | Parameters
---|---
Conv2D | $400,3\times 3$
ReLu |
Max Pool | $3\times 3$
Linear | $400*16*7\times 400$
Tanh |
Linear | $400\times 6*6*3$
Tanh |

###  3.3 Node Detection Network

The node detection network takes as input the convolutional feature vector
from the backbone network and outputs a set of rectangular node proposals,
each with an objectness score.

The network slides a small network over the input convolutional layer. The
output of the window is fed into two sibling layers: a box-regression layer
and a box-classification layer. Because the small networks work in the sliding
window fashion, they are naturally implemented with convolutional networks.

At each sliding window, we predict one region proposal which encodes the four
coordinates of a box and one score which estimates the probability of whether
the proposal is a node or not. Each region proposal is parameterized relative
to a reference box, called anchor. We parameterize the coordinates of the
bounding boxes following [16]:

$t_{x}=(x-x_{a})/w_{a},t_{y}=(y-y_{a})h_{a},$
$t_{w}=log(w/w_{a}),t_{h}=log(h/h_{a})$
$t_{x}^{*}=(x^{*}-x_{a})/w_{a},t_{y}^{*}=(y^{*}-y_{a})/h_{a},$
$t_{w}^{*}=log(w^{*}/w_{a}),t_{h}^{*}=log(h^{*}/h_{a}),$

where $x$, $y$, $w$, and $h$ is the box’s center point and width and height.
Variables $x$, $x_{a}$ and $x^{*}$ are for the predicted box, anchor box, and
ground-truth box respectively (likewise for $y$, $w$, $h$). This
parameterization converts large integers of bounding box coordinates to
variables close to interval $[-1,1]$, and therefore improves the numerical
performance of the algorithm.

To train the node detection network, we assign a binary class label of being a
node or not to each anchor. We assign a positive number to two kinds of
anchors: (1) The anchor with the highest Intersection-over-Union (IoU) with a
ground-truth box, or (2) an anchor that has an IoU higher than 0.9 with any
ground-truth box. We assign a negative number to an anchor if its IoU is less
than 0.3 for all ground-truth boxes. An anchor that is neither positive nor
negative does not contribute to the train objective. Because the number of
positive labels and negative anchors may be different and therefore the train
may be biased toward one direction, we sample from the more to ensure the same
size of positive and negative labels during training.

We apply binary cross entropy loss to the objectness, and smooth-l1 loss to
the region proposal. The final loss is the sum of the objectness loss and
region proposal loss over all anchors.

For prediction, we choose top 50 anchors with the highest scores, and group
them by the condition that anchors with IoU greater than 0.2 are in the same
group. Then, we choose from each group the highest score as the final
prediction.

Nevertheless, our network differs from the faster RCNN in two ways. (1)
Because the nodes in flow charts have similar size and shape, we use one
anchor, instead many anchors with different size and ratio, to save computing
power. (2) Because the nodes in flow charts do not overlap, we consider
proposals in the same group if one has IoU over a threshold with any other one
in the group. Meanwhile, fast RCNN considers proposals in the same group if
one has IoU over a threshold with the one of highest score.

The architecture of the node detection network is as follows. The intermediate
layer is a Conv2D layer with 3 X 3 kernel and 400 output channels. The
classifier is a Conv2D layer with 3 X 3 kernel and 1 output channel. The
regressor is a Conv2D layer with 3 X 3 kernel and 4 output channels. Each
Conv2D layer is activated by ReLU function.

###  3.4 Node Recognition Network

The node recognition network takes as input the crop of each node and outputs
a vector of size $LENVOC\times 1\times W$, where $LENVOC$ is the vocabulary
size.

Table 4 is the architecture of node recognition network, where the base block
shares the same architecture as the basic block in the backbone network, Table
1,

The output of the node recognition network can be interpreted as probability
distribution over the vocabulary at each position along the width direction.
CTC loss is a function for training sequences problems such as handwriting
recognition or speech recognition. CTC loss does not attempt to learn the
character boundaries, and can be applied if the input is a sequence with some
order. By adoption CTC loss, we do not need to use another recurrent neural
network to predict the text. Instead, we directly predict from the output of
the node recognition network.

When prediction, we use greedy search to decode text from the output vector.

Optionally, a spatial transformer network (STN), [17], can be inserted before
the node recognition network to further adjust the the boundary of the crop.
STN is the network that can be inserted into existing convolutional
architectures, giving neural networks the ability to actively spatially
transform feature maps.

Table 4: Node Recognition Network Architecture Layer Type | Parameters
---|---
Input | C $\times$ H $\times$ W
Basic Block | 64, 3 $\times$ 3
Max Pool | 3 $\times$ 3
Basic Block | 128, 3 $\times$ 3
Max Pool | 3 $\times$ 3
Basic Block | 256, 3 $\times$ 3
Max Pool | 3 $\times$ 3
Basic Block | LENVOC, 3 $\times$ 3

###  3.5 Train and Implementation

The whole network is trained end-to-end. The lost function is the sum of loss
of all sub-networks:

$loss=loss_{edge}+loss_{ndc}+loss_{ndr}+\sum_{node}{loss_{nr}},$ (2)

where $loss_{edge}$, $loss_{ndc}$, $loss_{ndr}$ and $loss_{nr}$ is the loss of
edge network, classifier and regressor of node detection network and node
recognition network, respectively.

We use SGD method to train the network. Learning rate is 0.02, and halved when
the error plateaued. The total epoch is 200.

##  4 Experiment

We designed experiments to answer the following research questions: (1) what
is the accuracy of GRCNN ? (2) what is the inference performance of GRCNN? (3)
Is GRCNN able to synthesize real-world program?

The experiments are conducted on a desktop with Geforce 1070 GPU, Intel i7
CPU, and 16GB memory. GRCNN is implemented with PyTorch.

We tested the following networks.

  * •

GRCNN: which is our basic network

  * •

Separated GRCNN: not sharing the backbone network. See Section 4.4.

  * •

GRCNN with ResNet: add optional residual learning. See Section 3.1.

  * •

GRCNN with STN: add option STN network. See Section 3.4.

  * •

GRCNN with ResNet and STN.

###  4.1 Dataset Generation

We created a synthetic dataset to train and test the GRCNN, since there are no
existing datasets we can use.

A data sample includes a flow chart image in PNG format and a text file
containing the ground truth which is used to train the network. The ground
truth includes the adjacent matrix which represents the edge information, and
the bounding box and text content for each node.

The dataset contains flow charts with 3 to 6 nodes, and 0 to 2 decisions. The
text in each node contains 3 to 9 random characters from an alphabet of 50
characters, including the lower case English characters, digit characters,
arithmetic operators, parenthesis et al. Note that the alphabet can be chosen
freely without changing the essential network architecture. The train and test
dataset contains 9960 and 2490 data samples, respectively.

We draw the flow chart using Graphviz, a popular graph-drawing tool. We set
the maximal width to be 200 pixels and maximal height to be 400 pixels.

When drawing the flow chart, we randomize to cover wide range of data samples.
The width of lines, including nodes boundaries and edges, is randomly chosen
between 1 to 5 pixels. The font size of characters is randomly chosen between
20 to 30. The font color of characters is random RGB color. We draw nodes in
rectangles and design a heuristical algorithm to effectively compute high
accuracy coordinates of bounding boxes.

###  4.2 Accuracy

Table 5: Accuracy in Percentage of GRCNN. The columns are Edge, Sequence, Nodes and Graph accuracy, respectively. | Edge | Squence | Nodes | Graph
---|---|---|---|---
GRCNN | 94.1 | 90.6 | 67.9 | 66.4
+STN | 90.0 | 93.8 | 71.3 | 64.4
+ResNet | 91.9 | 91.3 | 64.8 | 60.3
+STN+ResNet | 90.9 | 93.2 | 70.0 | 63.2
Separated | 93.7 | 91.1 | 68.2 | 65.8
Table 6: Time Cost in Milliseconds of GRCNN . The columns are the time cost of the backbone, edge, node detection, node recognition network and GRCNN, respectively. | BB | Edge | ND | NR | GRCNN
---|---|---|---|---|---
GRCNN | 3.5 | 0.4 | 42.4 | 14.2 | 60.5
+STN | 3.5 | 0.4 | 43.3 | 16.6 | 63.8
+ResNet | 3.5 | 0.4 | 44.8 | 14.0 | 79.1
+STN+ResNet | 3.6 | 0.5 | 47.0 | 16.7 | 84.3
Separated | 3.5 | 0.4 | 42.8 | 15.1 | 61.8
|

    [[0 1 2 0 0 0]
     [0 0 0 1 0 0]
     [0 0 0 1 0 0]
     [0 0 0 0 1 2]
     [0 0 0 0 0 1]
     [0 0 0 0 0 0]]

    if x<y
        a=y
    else
        a=x
    if a<z
        a=z
    return a

|  |

    [[0 1 0 0 0 0]
     [0 0 1 0 0 0]
     [0 0 0 1 0 0]
     [0 0 0 0 1 0]
     [0 0 2 0 0 1]
     [0 0 0 0 0 0]]

    f=1
    i=n
    do
        f=f*i
        i--
    while i>0
    return f

---|---|---|---
Figure 3: Samples of GRCNN prediction. The predicted bounding boxes are drawn
in red lines. The predicted text is drawn above the bounding box in blue
characters. On the right is the predicted adjacent matrix and synthesized
source code.

We measure the Edge Accuracy, Sequence Accuracy, Nodes accuracy and Graph
Accuracy of GRCNN prediction. Edge accuracy is the percentage of images whose
edge is correctly predicted. We say the edge is correctly predicted if the
predicted adjacent matrix is exactly the same as the ground truth. Sequence
accuracy is the percentage of nodes whose text content is correctly predicted
among all nodes in all images. This item shows how well the node recognition
network works for individual nodes. Nodes accuracy is the percentage of images
whose nodes are all correctly predicted. This item shows how well the tool
predicts the nodes as a whole. Graph accuracy is the percentage of images
whose edge and nodes are all correctly predicted.

The accuracy of GRCNN and its subnets are described in the first row of Table
5. The Edge, Sequence, Nodes and Graph accuracy is 94.1%, 90.6%, 67.9% and
66.4%, respectively. Note that the Graph accuracy is the result of joint
probability of all edges and all nodes, so it is lower than Edge and Sequence
accuracy.

We conducted ablation study to identify how well the optional enhancements
are. In Table 5, row 2, 3 and 4 shows the accuracy of GRCNN with different
enhancements. We find that the edge accuracy and graph accuracy decreased with
those enhancements. Our explanation is that the extra trainable weights in the
enhancements caused the network biased toward the nodes detection and
recognition networks, but caused the decrease in edge accuracy, and therefore
caused decrease in the graph accuracy. We also find all STN enhancements
improved the accuracy of node detection and recognition networks.

###  4.3 Performance

Table 6 shows the performance of GRCNN and its subnets. The overall time cost
of GRCNN is about 60 milliseconds, and the performance of other networks are
close to the performance of GRCNN. Among the subnets, the time cost of node
detection network is the major part, 70.1%, of the overall cost. The rest is
the node recognition network (23.5%), backbone network (5.8%), and edge
network (1%).

###  4.4 End-to-End vs Separated Network

Because edges and nodes information do not depend on each other, it is natural
to consider to use two separate networks to predict edges and nodes. We
designed experiments to see how well both ideas work.

We made two clones of GRCNN and modified as follows. For one clone, we disable
the node detection network and node recognition network. For the other clone,
we disable the edge detection network. We trained the two networks separately
with the same dataset and hyper parameters.

The row _Separated_ in Table 5 and 6 shows the accuracy and performance of the
separated networks. The performance are close to GRCNN, therefore it is
feasible to share the computation of the backbone network. The sharing saves
the 3.5 milliseconds, which is 5.8% of the time cost.

###  4.5 Real-world Program Synthesis

Table 7: Result of real-world program synthesis. The columns are program name, Graph accuracy, Edge accuracy, Nodes accuracy, number of nodes, correctly predicted nodes.  program | Graph | Edge | Nodes | #Nodes | Nodes
---|---|---|---|---|---
abs | 1 | 1 | 1 | 4 | 4
swap | 1 | 1 | 1 | 3 | 3
max | 0 | 1 | 0 | 4 | 3
sum | 0 | 0 | 1 | 6 | 6
max3 | 1 | 1 | 1 | 6 | 6
log | 1 | 1 | 1 | 4 | 4
radius | 1 | 1 | 1 | 3 | 3
poly | 1 | 1 | 1 | 6 | 6
factorial | 1 | 1 | 1 | 6 | 6
quadrant | 0 | 0 | 1 | 5 | 5
cntpos | 0 | 0 | 0 | 6 | 3
sum | 7 | 8 | 9 | 53 | 49
percentage | 63.6 | 72.7 | 81.8 | - | 92.5

In order to see how well GRCNN synthesizes real-world programs, we created a
test dataset of 11 programs, which are selected from the problems in a
programming textbook [18].

Table 7 is the testing result. It shows that 70% of programs are correctly
predicted, 70% edges are correctly predicted, 93.6% individual nodes are
correctly predicted.

In Figure 3, we draw two samples to visually demonstrate the input and output
of GRCNN. The first one is the function that finds the max of three numbers.
The second one is the factorial function.

##  5 Conclusion

We presented GRCNN, a deep convolutional neural network that parses graph data
structure from a flow chart, and we automatically generates source code that
matches the flow chart. GRCNN predicts the edge information and nodes
information simultaneously. Experiments show that we can share the computation
of the feature vector. GRCNN achieves 66.4%, 94.1%, 90.6% for graph, edge and
node accuracy, respectively, on our test dataset, and close accuracy on a
real-world dataset.

## References

  * [1] S. Gulwani, “Dimensions in program synthesis,” in _Proceedings of the 12th international ACM SIGPLAN symposium on Principles and practice of declarative programming_. ACM, 2010, pp. 13–24.
  * [2] ——, “Automating string processing in spreadsheets using input-output examples,” _ACM Sigplan Notices_ , vol. 46, no. 1, pp. 317–330, 2011.
  * [3] Z. Manna and R. Waldinger, “A deductive approach to program synthesis,” _ACM Transactions on Programming Languages and Systems (TOPLAS)_ , vol. 2, no. 1, pp. 90–121, 1980.
  * [4] Y. Feng, R. Martins, J. Van Geffen, I. Dillig, and S. Chaudhuri, “Component-based synthesis of table consolidation and transformation tasks from examples,” in _ACM SIGPLAN Notices_ , vol. 52, no. 6. ACM, 2017, pp. 422–436.
  * [5] T. A. Lau and D. S. Weld, “Programming by demonstration: An inductive learning formulation,” in _International Conference on Intelligent User Interfaces: Proceedings of the 4 th international conference on Intelligent user interfaces_ , vol. 5, no. 08. Citeseer, 1998, pp. 145–152.
  * [6] C. Wang, A. Cheung, and R. Bodik, “Synthesizing highly expressive sql queries from input-output examples,” in _ACM SIGPLAN Notices_ , vol. 52, no. 6. ACM, 2017, pp. 452–466.
  * [7] A. Solar-Lezama and R. Bodik, _Program synthesis by sketching_. Citeseer, 2008.
  * [8] L. Cheng, “Sqlsol: An accurate sql query synthesizer,” in _International Conference on Formal Engineering Methods_. Springer, 2019, pp. 104–120.
  * [9] M. Balog, A. L. Gaunt, M. Brockschmidt, S. Nowozin, and D. Tarlow, “Deepcoder: Learning to write programs,” _arXiv preprint arXiv:1611.01989_ , 2016.
  * [10] N. Yaghmazadeh, Y. Wang, I. Dillig, and T. Dillig, “Sqlizer: query synthesis from natural language,” _Proceedings of the ACM on Programming Languages_ , vol. 1, no. OOPSLA, p. 63, 2017.
  * [11] N. Locascio, K. Narasimhan, E. DeLeon, N. Kushman, and R. Barzilay, “Neural generation of regular expressions from natural language with minimal domain knowledge,” _arXiv preprint arXiv:1608.03000_ , 2016.
  * [12] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in _Advances in neural information processing systems_ , 2015, pp. 91–99.
  * [13] S. Zherzdev and A. Gruzdev, “Lprnet: License plate recognition via deep neural networks,” _arXiv preprint arXiv:1806.10447_ , 2018.
  * [14] G. B. Shelly and M. E. Vermaat, _Discovering Computers, Complete: Your Interactive Guide to the Digital World_. Cengage Learning, 2011.
  * [15] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [16] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 580–587.
  * [17] M. Jaderberg, K. Simonyan, A. Zisserman _et al._ , “Spatial transformer networks,” in _Advances in neural information processing systems_ , 2015, pp. 2017–2025.
  * [18] D. M. Etter, _Introduction to C_. Prentice Hall, 1998.