kxl014 N N

# Optimal transport and colossal ionic mechano–conductance in graphene crown
ethers

Subin Sahu  1,2 Justin Elenewski  1,2 Christoph Rohmann  1,2 Michael Zwolak1∗
1Biophysics Group, Microsystems and Nanotechnology Division, Physical
Measurement Laboratory, National Institute of Standards and Technology,
Gaithersburg, MD 20899,2Maryland Nanocenter, University of Maryland, College
Park, MD 20742

###### Abstract

Biological ion channels balance electrostatic and dehydration effects to yield
large ion selectivities alongside high transport rates. These macromolecular
systems are often interrogated through point mutations of their pore domain,
limiting the scope of mechanistic studies. In contrast, we demonstrate that
graphene crown ether pores afford a simple platform to directly investigate
optimal ion transport conditions, i.e., maximum current densities and
selectivity. Crown ethers are known for selective ion adsorption. When
embedded in graphene, however, transport rates lie below the drift-diffusion
limit. We show that small pore strains – 1 % – give rise to a colossal – 100 %
– change in conductance. This process is electromechanically tunable, with
optimal transport in a primarily diffusive regime, tending toward barrierless
transport, as opposed to a knock–on mechanism. Measurements of mechanical
current modulation will yield direct information on the local electrostatic
conditions of the pore. These observations suggest a novel setup for
nanofluidic devices while giving insight into the physical foundation of
evolutionarily–optimized ion transport in biological pores.

###### doi:

<EMAIL_ADDRESS>
††volume: 00††issue: 0 Figure 1: Potential transport mechanisms in graphene
crown ether pores. a. Using partial charges $q_{\mathrm{O}}=-0.24\,e$,
consistent with the electrostatic potential from DFT, the ion transport
mechanism is drift–diffusion. In this case, a K+ (purple sphere) finds an
empty pore and translocates through it; the pore then remains empty again for
several nanoseconds (see the SM). b. At larger partial charge
($q_{\mathrm{O}}=-0.54\,e$ or $q_{\mathrm{O}}=-1.0\,e$), the vicinity of the
charge separation at the pore rim results in an energetic well deep enough to
trap a K+ (light purple). For a current to be present, an incoming K+ (dark
purple) knocks out and replaces the trapped K+. This yields a two-step knock-
on mechanism reminiscent of some biological potassium ion channels. However,
for $q_{\mathrm{O}}=-0.54\,e$, the mechanism shifts towards a majority
drift–diffusion process at moderate (1 %) strain due to a shallowing of the
free energy well. Oxygen, carbon, and hydrogen atoms are small red, grey, and
white spheres, respectively.

## Introduction

The search for universal transport mechanisms is an underlying theme in ion
channel research (1). Nonetheless, even foundational questions – such as the
primary mechanism for selectivity of K+ over Na+ in the potassium ion channel
family – remain contentious: Is the smaller dehydration energy of the K+ ion
responsible, or it is the “snuggle–fit” of K+ in the selectivity filter (2, 3,
4)? Solving these puzzles requires a detailed means to control and
characterize ion transport. Biological ion channels are notoriously difficult
to manipulate, requiring mutagenesis and near–native conditions for
patch–clamp studies, thus limiting experimental characterization and
tunability. These large macromolecules are also theoretically complex,
requiring hefty resources to conduct statistically meaningful simulations. In
contrast, biomimetic synthetic pores – or, as addressed here, pores that
retain core aspects of some biological channels – may provide a simple
platform to explore ion transport mechanisms under a broad range of conditions
(5).

In this work, we demonstrate that graphene crown ether pores (6) have
competing electrostatic and dehydration contributions to transport,
reminiscent of biological ion channel mechanism and selectivity. The discovery
of crown ethers a half–century ago triggered a cascade of research that
evolved into the fields of host–guest and supramolecular chemistry (7). The
isolation of graphene (8) has likewise inspired diverse fields of study,
including those that leverage atomically thin pores for biosensing and
nanoscale separation (9). Graphene crown ether pores lie at the crossroads,
affording a means to address transport and selectivity in a synthetic analog
of biological ion channels. Using a graphene–embedded 18-crown-6 ether pore as
a model system, we examine the mechanism for ion translocation under variable
strain and applied voltage, see Fig. 1. Our results indicate that a few
percent change in pore size can induce a several fold increase in ionic
current, underscoring the sub–nanometer fine tuning between electrostatics and
dehydration seen in biological contexts. We similarly find that small changes
in pore radius and charge can optimize transport, and even alter the
underlying mechanism of ion translocation. These results collectively suggest
that graphene crown ether pores are an effective mechanistic probe for
understanding transport in both biological and artificial ion channels alike.

Figure 2: Colossal ionic mechano–conductance. a. Schematic of graphene on (or
embedded within) a polymeric matrix support (such as $\sim 50$ nm thick epoxy
resin) with a window where the crown ether pore is located. The membrane can
be strained by stretching or bending (e.g., via a piezoelectric actuator
offset from the window). Alternative experimental setups are possible, such as
metallic regions that bind the graphene and are used to apply strain. b.
Relative change in current versus strain in the crown ether pore (for
$q_{\mathrm{O}}=-0.24\,e$ and $-0.54\,e$) at different voltages. At lower
voltages, the current increases substantially with strain, as shown by the
fitted dashed lines. c. Current without strain ($I_{0}$) for different values
of $q_{\mathrm{O}}$ and $V$. Going from $q_{\mathrm{O}}=-0.24\,e$ to
$q_{\mathrm{O}}=-0.54\,e$, the current increases as highly charged oxygen
atoms compensate the loss of waters from the hydration layers. However, going
from $q_{\mathrm{O}}=-0.54\,e$ to $q_{\mathrm{O}}=-1.0\,e$, the current
decreases because a further increase in the pore charge makes it harder for an
ion to escape the potential well in the pore. The typical drift-diffusion
limit – set by access resistance to an uncharged pore with approximate radius
0.1 nm – at 0.25 V is about 0.35 nA. Only the intermediate charge case – the
one most analogous to biological systems – approaches this limit: It is about
a factor of two lower at 2 % strain. Further strain allows it to reach this
limit. The smaller charge case can approach within a factor of 6. The range of
accessible currents is due to the extensive mechanistic leeway permitted by
the intermediate charge. The error bars are the standard error from five
parallel runs.

## Results

Partial charge distribution: Unlike an isolated crown ether, an 18-crown-6
pore in graphene is believed to assume a planar conformation (6). However,
while critical to molecular dynamics (MD) simulations, the charge distribution
and response of this pore is unknown. Prior simulations on this and related
systems assume oxygen partial charges that span between
$q_{\mathrm{O}}=-0.21\,e$ and $q_{\mathrm{O}}=-0.74\,e$ (10, 11, 12, 13, 14,
15), over which transport properties will vary dramatically. To provide a more
comprehensive approach, we reexamine this assignment using density functional
theory (DFT), see the Methods and supplementary materials (SM) for details.
Determinations from electrostatic potential fitting without an ion present –
consistent with additive force fields and the electrostatic environment
predicted by DFT – yield a uniform value of $q_{\mathrm{O}}=-0.24\,e$,
decreasing to $-0.23\,e$ when a K+ is present. Chemically distinct, but oxygen
containing, fragments in OPLS and CHARMM force fields give substantially
higher partial charges, $\geq-0.4\,e$, of which we take
$q_{\mathrm{O}}=-0.54\,e$ as a representative example. An upper bound of
$q_{\mathrm{O}}=-1.0\,e$ is given by Bader analysis, which tends toward larger
values of partial charge and represents the maximum expected from any
electronic structure calculation. Physically reasonable values of
$q_{\mathrm{O}}$ should span the low end of the range $-0.2\,e$ to $-0.6\,e$,
as evidenced by electrostatic fits of $q_{\mathrm{O}}=-0.34\,e$ for the
18-crown-6 alkyl ether and $q_{\mathrm{O}}=-0.54\,e$ for a highly–dipolar
glycine dipeptide backbone carbonyl. The graphene crown ether should be lower
than both. We perform MD simulations using three distinct assignments
$q_{\mathrm{O}}=(-0.24,-0.54,-1.0)\,e$ to capture the full range of potential
behavior, showing both universal features of transport and aspects particular
to different assignments. We test the effect of strain on ion transport using
different applied biases and deformations, well within the elastic limit of
graphene (16, 17).

Ion transport mechanism: Since this pore is tiny and negatively charged, only
K+ contributes to the current in a KCl solution (set at a concentration of 1
mol/L in our case). The mechanism of transport depends on the ion’s binding
strength in the pore, largely regulated by $q_{\mathrm{O}}$ (as an indicator
of the oxygen–carbon charge separation) and atomic structure. The mechanism is
normal drift–diffusion for $q_{\mathrm{O}}=-0.24\,e$, shifting to a knock–on
type for $-0.54\,e$ and $-1.0\,e$ (see the illustration in Fig. 1). For drift-
diffusion, the residence time (on the order of 0.1 ns at 0.25 V) is much
shorter than the delay to the next ion crossing event (on the order of 1 ns at
0.25 V). Conversely, for the two-step knock-on mechanism seen at higher
charge, the residence time (on the order of 1 ns) is much larger than the
delay time (less than 10 ps for most events; see the SM). However, even for
$q_{\mathrm{O}}=-0.54\,e$, drift–diffusion starts to dominate at high strain
and/or voltage, both of which weaken ion binding to the pore.

Ionic mechano-conductance: Figure 2a depicts a potential experimental setup,
in which a graphene sheet is selectively strained by deforming an underlying
substrate. Our simulations mimic these conditions by increasing the
cross–sectional area of the membrane to induce a strain. The relative change
in current as a function of strain, for different values of $q_{\mathrm{O}}$
and applied voltages, is shown in Fig. 2b. At low voltages (0.25 V and 0.5 V),
the ionic current increases rapidly with strain, orders of magnitude larger
than expected from the increase in pore area alone. Instead, this colossal
increase is indicative of a change in the energetics and dynamics of ion
transport. In the large voltage regime (1 V), the current remains constant
with increasing strain when $q_{\mathrm{O}}=-0.54\,e$. This is not altogether
surprising, as a large voltage drop across the membrane suppresses the
influence of the free energy profile on translocating ions. The dependence of
the current on $q_{\mathrm{O}}$ and $V$ is in Fig. 2c. In this case, the
current increases with $q_{\mathrm{O}}$ between $q_{\mathrm{O}}=-0.24\,e$ to
$q_{\mathrm{O}}=-0.54\,e$, while giving a turnover to decreasing current at
higher charge.

Figure 3:  Free energy landscape. Free energy profile, $\Delta
F_{\mathrm{K}}$, for a K+ translocating along the $z$-axis of an 18-crown-6
graphene pore at different strains. The charge on the oxygen atom,
$q_{\mathrm{O}}$, is presented at the top of each plot. The peaks and valleys
in $\Delta F_{\mathrm{K}}(z)$ are due to the balance between dehydration
energy penalties and electrostatic interactions with the charged pore atoms.
For $q_{\mathrm{O}}=-0.54$, there is an additional contribution from the K+
occupying the deep potential well, giving a barrier at $z\approx 0.2$ nm when
the incoming ion attempts to go into the already occupied pore (this is
confirmed by a free-energy profile with only 1 K+ and 1 Cl- present; see the
SM). The illustrations show the position of the K+ in the peaks and valleys.
In general, the free energy profile flattens with increasing – but still small
– strain, tending toward barrierless transport and making it easier for the
ion to translocate. The error bands are the standard error from five parallel
runs.

The energy landscape: Our results indicate that the energetics of pore
permeation lie in a regime where small electromechanical variations modify
current magnitudes and give rise to qualitative changes in behavior. The free
energy of a potassium ion in the vicinity of the pore, $\Delta
F_{\mathrm{K}}$, is approximately the sum of a dehydration energy barrier
$\Delta E_{\mathrm{deh}}$ and three electrostatic terms ($E_{\mathrm{KO}}$,
$E_{\mathrm{KC}}$, and $E_{\mathrm{KK}}$) that capture the interaction of K+
with nearby charged atoms:

$\Delta F_{\mathrm{K}}=\Delta
E_{\mathrm{deh}}+E_{\mathrm{KO}}+E_{\mathrm{KC}}+E_{\mathrm{KK}},$ (1)

where $E_{\mathrm{deh}}=\eta\sum_{i}f_{i}E_{i}$ is the barrier due to
dehydration, $f_{i}$ ($E_{i}$) is the fractional dehydration (energy) of the
$i^{\mathrm{th}}$ hydration layer, and the factor $\eta\approx 1/2$ accounts
for the nonlinear effects (18). Each electrostatic component is
$E_{\mu\nu}=q_{\mu}n_{\nu}q_{\nu}/4\pi\epsilon_{0}\epsilon(z_{\mu\nu})r_{\mu\nu}$,
where $\epsilon(z_{\mu\nu})$ is a position–dependent relative permittivity
(19), $q_{\mu(\nu)}$ is the charge of ion species $\mu(\nu)$, $n_{\nu}$ is the
number of proximate species $\nu$ (6 for oxygen, 12 for carbon, and 1 or 0 for
a nearby K+), $z_{\mu\nu}$
($r_{\mu\nu}=\sqrt{z_{\mu\nu}^{2}+\rho_{\mu\nu}^{2}}$) is the axial (total)
distance between species $\mu$ and $\nu$, and $\rho_{\mu\nu}$ is the
respective radial distance. $E_{\mathrm{KK}}$ is the energetic coupling of an
incoming K+ to a trapped K+. The van der Waals interactions, the coupling to
other solvated ions, and the entropy change should have only marginal
contributions when $\Delta F_{\mathrm{K}}$ varies with strain (e.g., the
entropy change into the pore is important, but this varies little with
moderate strain). In order to quantify the variable electrostatic environment
reflected in $\epsilon(z)$, as well as nonlinear effects in $\Delta
E_{\mathrm{deh}}$ (stronger polarization of bound water molecules as an ion
becomes dehydrated (18)), we employ all–atom MD simulations to find the free
energy profile of K+ around the pore (Fig. 3 and the Methods). Equation 1
explains the key features in the free energy profiles for the range of
$q_{\mathrm{O}}$ under consideration. Together with the MD data, this
expression dissects the contributions to the free energy. We will use this
equation below to characterize the response of ion transport to strain.

Figure 3 shows that when $q_{\mathrm{O}}=-0.24\,e$, a free energy barrier
appears outside the pore due to the progressive dehydration of K+. The barrier
decreases with increasing proximity to the pore opening as the flanking oxygen
atoms begin to compensate for water molecules displaced from the ion’s first
hydration layer. This results in a potential well on both sides of the
membrane. Further dehydration occurs as the ion enters the pore, giving rise
to an overall barrier at $z=0$ since the electrostatic attraction between the
pore oxygen atoms and the K+ ($E_{\mathrm{KO}}$) is insufficient to compensate
for the dehydration energy penalty. This central barrier – and only shallow
adjacent wells – ensure that an approaching K+ will almost always find an
empty pore, and hence the $E_{\mathrm{KK}}$ term will not contribute in this
case. Equation 1 confirms this assignment of the peaks and wells.

This situation is inverted for $q_{\mathrm{O}}=-0.54\,e$, see Fig. 3, where a
deep potential well forms at the center of the pore due to strong
electrostatic attraction with the K+. The central pore acts as a binding site
in this case, ensuring that it is almost always occupied. When K+ approaches
the pore, the presence of an interstitial K+ repels it, yielding a potential
barrier on either side of the pore. A free energy calculation with single K+
and single Cl- confirms this (see the SM). We can estimate $\epsilon_{r}(z)$
using the free energy profile from Fig. 3 and the fractional dehydration from
MD. For the physical range of pore charge, $\epsilon_{r}(z)$ is around 4 to 6
within 0.2 nm of the pore, which is consistent with experimental values for
the dielectric constant within 1 nm of an interface (20). A similar value for
dielectric constant is given in another recent computational study (19).

Barrierless transport: While the transport mechanism is different for
unstrained pores, the colossal mechano–conductance change is present at low
voltage regardless of the partial charge. In both partial charge scenarios,
the free energy profile becomes smoother with increasing strain, tending
toward “barrierless transport” (21, 19): Pulling charged oxygens away from the
pore center reduces their coupling to translocating ions. The colossal change
in conductance indicates that there are optimal structural positions for the
oxygens: Picometer changes in their position do not change the dehydration
contribution to the central barrier (see the SM) but it vastly changes the
transport rate, maintaining exclusion of other ions but enhancing transport
rates by several fold. Biological ion channels can further make use of the
partial charge, engineering not only structural characteristics, but also the
electronic environment. As we discussed earlier, the larger partial charges we
consider – in line with biological channels (22) – show a shift from knock–on
to drift–diffusion mechanisms as transport starts to become barrierless. In
this manner, the electromechanical environment may exert a large effect on the
current.

Mechanical susceptibility: To understand the mechanics underlying this
process, we define the susceptibility, $\chi$, of the free energy to a small
change in pore size,

$\chi=\frac{d\Delta F}{da}=\sum_{\langle\mu\nu\rangle}\frac{\partial\Delta
F}{\partial\rho_{\mu\nu}}\frac{d\rho_{\mu\nu}}{da}\approx
q_{\mathrm{K}}E_{\rho},$ (2)

where $a$ is the effective pore radius, $\langle\mu\nu\rangle$ indicates the
pairs KO and KC (we ignore the role of KK interactions, although this is
important for knock–on aspects of transport), and $d\rho_{\mu\nu}/da\approx
1$. The latter approximation is reasonable since, as the pore size increases,
the oxygen and carbons move outward by the same amount. For simplicity, we
took this for an ion at the origin ($z=0,\rho=0$) where the susceptibility is
just the local radial electric field, $E_{\rho}$, multiplied by the charge of
the translocating ion. Due to the lack of electrostatic screening in the pore
(i.e., $\epsilon(z=0)$ is small) and the proximity of the oxygen atoms, this
field is very large, resulting in a large $\chi$: Equation (1) gives
$\chi\approx 100\,k_{B}T$/nm and $\chi\approx 140\,k_{B}T$/nm for the
$q_{\mathrm{O}}=-0.24\,e$ and $q_{\mathrm{O}}=-0.54\,e$, respectively (using
$\epsilon=3.9$ and $6.4$, which we extract via Eq. (1) and the all-atom MD
data). These two susceptibilities are closer than their difference in charge
would indicate due to the smaller permittivity for $q_{\mathrm{O}}=-0.24\,e$
(the charge $q_{\mathrm{O}}=-0.54\,e$ more strongly attracts counter charges,
specifically K+ and hydrogens on water molecules, that help screen the
interaction.)

The large susceptibility means that even a one picometer change in the pore
size gives a 0.1 $k_{B}T$ to 0.14 $k_{B}T$ change in the free energy. A 1 %
strain gives about 7 pm change in radius. Using this in Eq. (1), free energy
changes by about 0.7 $k_{B}T$ for $q_{\mathrm{O}}=-0.24\,e$ and about 1.0
$k_{B}T$ for $q_{\mathrm{O}}=-0.54\,e$, in line with the all-atom results in
Fig. 3. We note that there are clearly other effects occurring: As we will
discuss below, the $q_{\mathrm{O}}=-0.24\,e$ case is already in a limit where
dehydration and electrostatic effects are comparable, and thus the change in
the free energy profile versus $z$ reflects both (the barrier reduction at
$z\approx 2$ nm is due to a changing dehydration contribution, whereas in
between those two outer barriers, electrostatic effects dominate and $\Delta
F$ increases overall). For $q_{\mathrm{O}}=-0.54\,e$, the potential well
shallows, causing a transition from a knock–on to a diffusive mechanism and
removing the ion–ion repulsion that causes the external barrier. Nevertheless,
in both cases, the electrostatic contribution to the susceptibility captures
the increased free energy for an ion in the pore. While a general
susceptibility will need to include the totality of effects, electrostatics
and dehydration will typically dominate and are key to understanding
amplification.

The free energy profile and its variation (according to $\chi$) determines the
current. In particular, the largest free energy hurdle, $\Delta F^{\star}$, in
the profiles of Fig. 3 are the main factors influencing the conductance. For
$q_{\mathrm{O}}=-0.24\,e$, this will be the jump from about $z=$ 0.1 nm to 0.2
nm and, for $q_{\mathrm{O}}=-0.54\,e$, this will be the jump out of the well
($z=0$) to the maxima (at about $z=0.2$ nm to $z=0.4$ nm depending on strain).
The current will take the form (see Methods)

$I\approx Ae^{-\Delta F^{\star}/k_{B}T},$ (3)

where $A$ is a constant that only weakly depends on pore parameters (such as
size), but we will take it as fixed. The total amplification of the current
then becomes

$\frac{\Delta I}{I}\approx e^{(-\Delta F^{\star}(s)+\Delta
F^{\star}(s=0))/k_{B}T}-1\approx e^{\alpha\,a\,s\,\chi/k_{B}T}-1,$ (4)

where $-\Delta F^{\star}(s)+\Delta F^{\star}(s=0)\approx d\Delta
F/da\cdot\delta a\approx\alpha\,a\,\,s\,\chi$. The parameter $\alpha=1/a\cdot
da/ds$ quantifies the relative response of the pore radius to the application
of a strain. Both MD and DFT predict that $\alpha\approx 2$, meaning that if
the graphene has 1 % strain, the atoms at the pore rim move by about 2 %. This
is due to a distortion of the rim structure since the pore is an effective
impurity and can relieve strain by relaxing angular coordinates. The
straightforward relation in Eq. 4 indicates that the gain in current is the
exponential of the work performed by the applied strain on a K+ within the
pore. For the values in our case, Eq. (4) gives an amplification of 100 % to
200 % for a 1 % strain, in line with the MD results in Fig. 2b.

Figure 4: Optimum ion transport and selectivity. Ionic current versus strain
in a pore with $q_{\mathrm{O}}=-0.24\,e$ and an applied bias of 0.25 V. For
small strain (blue line), the current increases rapidly, commensurate with a
decrease in the outer barriers with increasing strain. This gives an overall
flattening of the free-energy profile. Transport (and selectivity, see below)
is optimal near 3 % strain (green line) when the outer barrier is sufficiently
diminished but the central barrier is not too high. Further increases in
strain continue to increase the central barrier (dehydration remains unchanged
in this strain regime), decreasing the current (red line). Schematics of the
free-energy profiles are above each regime. For every value of strain and
voltage, the selectivity of K+ over Cl- is perfect for all practical purposes,
as the negative partial charge of the oxygens on the pore rim create a strong
barrier for Cl- transport. Separate simulations (each of duration 1 $\mu$s
with an equal mixture of NaCl and KCl at 1 mol/L Cl-) reveal that Na+ does not
cross the pore (except for 0 % strain where a single Na+ crossing event occurs
during the 1 $\mu$s time frame). Due to the near complete exclusion of Na+ in
the time frame of the simulation, we can determine a high-confidence lower
bound on the selectivity, which goes from 15 at 0 % strain to 52 at 3 %
strain, see the text for details. In other words, the Na+ and Cl- exclusion is
maintained while K+ current changes with strain, and the lower bound on
selectivity exactly traces the K+ current in the plot.

Optimum transport: In all parameter regimes, the amplified current is due to
an overall flattening of the free energy profile. Ion transport thus tends
toward barrierless transport for small values of strain and displays a
colossal increase in conductance along the way. However, for
$q_{\mathrm{O}}=-0.24\,e$, the dominant contribution to $\Delta F^{\star}$
shifts from a pathway connecting a minimum (at $z=\pm\,0.1$ nm) and the
external solution to a pathway that hops over the central barrier (the well
minima already lie at about $\Delta F=0$). After this point, $\Delta
F^{\star}$ will continually increase due to decreasing compensation by the
oxygen partial charge. There will be a corresponding decrease in current, see
Fig. 4, thus indicating an optimum near this value of strain – a regime that
may be experimentally accessible.

In separate calculations with an equal mixture of NaCl and KCl (with Cl- at
one molar), we find that only a single Na+ traversed the pore (compare to 47
K+) during 1 $\mu$s of simulation for the unstrained pore at a 0.25 V bias. At
higher strains, only K+ translocate through the pore within the 1 $\mu$s
simulation time. The lack of Na+ transport is due to the larger hydration
energy of Na+ compared to K+, giving it a higher penalty to go through the
pore. The presence of an overall barrier should result in Na+ crossings
following Poisson statistics. Due to rarity of these crossings, we consider
two values for the Na+ current, 0.5 pA and 0.16 pA . The former would give a 5
% probability to detect no Na+ crossings during 1 $\mu$s and thus is a high
confidence upper bound to the Na+ current. The latter gives equal
probabilities for 0 and 1 Na+ crossing and thus represents an estimate of the
Na+ current assuming we are near the threshold to start seeing Na+ crossings,
for which the single crossing at 0 % strain suggests is the case. Using the
upper bound on the Na+ current, the corresponding high-confidence lower bound
on the selectivity ranges from about 15 at 0 % strain to 52 at 3 % strain
(note that the K+ currents in the equal mixtures are 7.5 pA at 0 % strain and
26.2 pA at 3 % strain, which are approximately half the value in the 1 mol/L
KCl case shown in Fig. 4 due to the mixture having half the K+ concentration).
The estimate of selectivity, though, ranges from 47 at 0 % strain to 164 at 3
% strain. Very long simulations (much greater than 10 $\mu$s) will be
necessary to tightly bound the selectivity, but it will be large nonetheless.

For $q_{\mathrm{O}}=-0.54\,e$, the free energy at $z=0$ will not turn from a
well to barrier until about $s=6$ % (i.e., about $\delta a=6\,k_{B}T/\chi$),
with the optimum appearing at potentially larger strains and outside of
measurable regimes. Up to this point, the current will increase with strain as
the free energy profile becomes truly barrierless. In either case, the
dehydration contribution itself (at $z=0$) will not substantially change until
the pore becomes sizable, as indicated by the continuing decrease of the
current in Fig. 4. The selectivity filters of biological channels have partial
charges near $q_{\mathrm{O}}=-0.54\,e$ by virtue of the dipolar peptide
backbone (4, 23, 24, 22), lying within this more strongly coupled regime and
affording them extensive mechanistic leeway. These systems generically exploit
the spatial distribution of charges or functional groups via the protein
structure to achieve optimality. Strain in the graphene crown ether pore
modulates a single collective variable, acting as a model for this
optimization process.

## Discussion

The investigation of 18-crown-6 and other crown ether pores, as well as pores
generally, with respect to strain (the experimental extraction of $\chi$ and
determination of the local interaction environment), voltage, and ion
concentration will shed further light into the behavior of selectivity and
optimal transport. These characteristics underlie the mechanism of
physiological ion channels and are of substantial interest in biology and
medicine (1, 25). This is paralleled by broad applicability to technologies
for the sensing and sequencing of biomolecules (26, 27, 28); molecular sieving
and desalination (29, 30); as well as batteries, fuel cells, and other energy
harvesting devices (31, 32). In all of these applications, the atomic
thickness and physical strength of graphene provide a unique advantage (28,
27) facilitating the detailed examination of molecular processes such ion
dehydration (18, 33). Mechanical modulation of ion transport will thus lend
itself to filtration and other technologies.

Our results indicate the ideal pore characteristics for optimal transport:
Minute changes in pore size or charge give rise to a dramatic change in ionic
conductance as transport becomes barrierless, but further change will decrease
the conductance. These systems also comprise a simple and transparent platform
to understand the evolution of biological ion channels. Working by analogy, it
is possible to identify structural features and charge distributions that
maximize selectivity while maintaining high permeation rates. These
considerations, in turn, constrain which conformational transitions may be
leveraged to regulate transport (34). Layered crown ether graphene pore
heterostructures – mimicking aspects of the selectivity filter in the
potassium channel from Streptomyces lividans (KcsA) – are likewise promising
for workable models of biomolecular channel dynamics. These systems should be
quite sensitive to ionic conditions, potentially gating or doping graphene to
access broad regimes of behavior. The susceptibility to transverse strain is
also generally interesting for artificial porous membranes as it gives
otherwise inaccessible information regarding the local electrostatic and
hydration conditions. This platform will thus open new and general
opportunities for the study, quantification, and tuning of ion transport at
the nanoscale.

## Methods

We determine point charges using both localized–basis and extended plane–wave
DFT schemes. The localized basis calculations (B3LYP/6-31G*) employ a
truncated model of the graphene pore alongside the Gaussian09 code (35),
affording point charges through CHELPG electrostatic potential fitting (36).
In contrast, for plane–wave DFT, we employ VASP(37) and a dispersion–corrected
scheme (PBE-D2/PAW) for a 1.71 nm $\times$ 1.73 nm graphene supercell
containing an 18-crown-6 pore, with periodic images separated by a 2.0 nm
vacuum layer ($5\times 5\times 1$ Monkhorst–Pack mesh, 500 eV cutoff) (38). In
this case, point charges were determined using Bader analysis (39). A
complementary set of plane wave calculations employs a similar pore within a
2.96 nm $\times$ 2.99 nm cell, described using CP2K (40) and a mixed Gaussian
plane–wave basis scheme (PBE-D3/GPW/DZVP/GTH pseudopotentials, 5442 eV
real–space grid cutoff) with $\Gamma$-point sampling and RESP charge fitting
(41). Point charges are derived from optimized geometries, defined where SCF
energy convergence is below $1.0\times 10^{-4}$ eV and forces below 0.1 eV
nm-1. CHELPG fitting yields $q_{\mathrm{O}}=-0.24\,e$ without a K+ present in
the pore and $-0.23\,e$ when a K+ is present. The complementary plane-wave
calculations confirm these assignments. The Bader analysis yields
$q_{\mathrm{O}}=-1\,e$, an upper bound for local partial charges due to its
strong bias toward chemically intuitive, but not electrostatically
representative, distributions. Both MD (below) and DFT give an $\alpha$ of
about 2 (DFT yields a slightly higher value), indicating that the pore rim has
an enhanced response to the strain. Reference fragments of
$q_{\mathrm{O}}=-0.54\,e$ include calculated CHELPG carbonyl charges for the
Ac-Gly-Gly-CH3 peptide (B3LYP/6-31G*), as well as standard OPLS
($q_{\mathrm{O}}=-0.50\,e$) and CHARMM ($q_{\mathrm{O}}=-0.51\,e$) backbone
carbonyl parameters. Graphene should have oxygen partial charges lower than
any of these fragments’ values – as well as the isolated crown ether’s oxygen
partial charges – due to the delocalized nature of the electrons in the carbon
sheet (see the SM for details and extended methods).

We simulate ion transport through the pore using all–atom molecular dynamics
via the NAMD2 code (42) with an integration time step of 1 fs. We take a
simulation cell aspect ratio of 1.2 – the golden aspect ratio – where the
simulation captures both access and pore resistances (43, 44). While access
resistance should be less than a 10 % correction for some parameter regimes,
it must be taken into account when understanding how the drift–diffusion limit
is approached. Its exclusion by choosing inappropriate (either too small or a
poor aspect ratio) simulation cells hinders the comparison with the
drift–diffusion limit (also see the SM).

We employ the adaptive biasing force (ABF) method (45) to calculate the
equilibrium free energy $\Delta F$ profile for a K+ going through the pore. We
compute $\Delta F$ versus $z$ in a cylindrical region of radius 0.1 nm and
length 3 nm centered around the pore. As a K+ approaches the pore, it
experiences either a free energy barrier or a potential well. Assuming a small
applied voltage $V$, the current over a barrier $\Delta F>0$ is (46)

$\displaystyle I_{\mathrm{K}}$
$\displaystyle=2\,q_{\mathrm{K}}\,k_{\text{in}}\,e^{-\Delta
F/k_{B}T}\sinh(q_{\mathrm{K}}V/2k_{B}T),$ (5)

and the current across a potential well $\Delta F<0$ is

$\displaystyle I_{\mathrm{K}}$
$\displaystyle=\frac{2\,q_{\mathrm{K}}\,k_{\text{in}}\,k_{\text{out}}\,c\,e^{-\Delta
F/k_{B}T}\sinh(q_{\mathrm{K}}V/2k_{B}T)}{k_{\text{in}}\,c+k_{\text{out}}\,e^{-\Delta
F/k_{B}T}\,\cosh(q_{\mathrm{K}}V/2k_{B}T)},$

where $k_{\text{in}}$ and $k_{\text{out}}$ are the rate constant for incoming
and outgoing ions and $c$ is the ion concentration in bulk solvent. Both of
these expressions give Eq. 3 for large $\Delta F$. (See the SM for a
additional discussion and results regarding the influence of the free energy
barrier, including a multiple barrier rate model).

## Acknowledgments

S. Sahu, J. Elenewski, and C. Rohmann acknowledge support under the
Cooperative Research Agreement between the University of Maryland and the
National Institute of Standards and Technology Center for Nanoscale Science
and Technology, Award 70NANB14H209, through the University of Maryland.

Author contributions: S.S. analyzed the ion transport behavior and performed
MD simulations. J.E. and C.R. performed DFT calculations. S.S., J.E., and M.Z.
formulated and derived mathematical expressions. All authors developed the
ideas and wrote the manuscript.

Competing interests: The authors declare that they have no competing
interests.

Data and materials availability: All data necessary to evaluate the
conclusions in the paper are present in the paper and/or the Supplementary
Material. Additional data related to this paper may be requested from the
authors.

## References

  * Hille (2001) Hille, B., 2001.  Ion channels of excitable membranes, volume 507.  Sinauer Sunderland, MA.
  * Nimigean and Allen (2011) Nimigean, C. M., and T. W. Allen, 2011.  Origins of ion selectivity in potassium channels from the perspective of channel block.  _J. Gen. Physiol._ 137:405–413.
  * Noskov et al. (2004) Noskov, S. Y., S. Berneche, and B. Roux, 2004.  Control of ion selectivity in potassium channels by electrostatic and dynamic properties of carbonyl ligands.  _Nature_ 431:830.
  * Doyle et al. (1998) Doyle, D. A., J. a. M. Cabral, R. A. Pfuetzner, A. Kuo, J. M. Gulbis, S. L. Cohen, B. T. Chait, and R. MacKinnon, 1998.  The Structure of the Potassium Channel: Molecular Basis of K+ Conduction and Selectivity.  _Science_ 280:69–77.
  * Kowalczyk et al. (2011) Kowalczyk, S. W., T. R. Blosser, and C. Dekker, 2011.  Biomimetic nanopores: learning from and about nature.  _Trends Biotechnol._ 29:607–614.
  * Guo et al. (2014) Guo, J., J. Lee, C. I. Contescu, N. C. Gallego, S. T. Pantelides, S. J. Pennycook, B. A. Moyer, and M. F. Chisholm, 2014.  Crown ethers in graphene.  _Nat. Commun._ 5:5389.
  * Pedersen (1967) Pedersen, C. J., 1967.  Cyclic polyethers and their complexes with metal salts.  _J. Am. Chem. Soc._ 89:7017–7036.
  * Novoselov et al. (2004) Novoselov, K. S., A. K. Geim, S. V. Morozov, D. Jiang, Y. Zhang, S. V. Dubonos, I. V. Grigorieva, and A. A. Firsov, 2004.  Electric field effect in atomically thin carbon films.  _Science_ 306:666–669.
  * Geim and Novoselov (2010) Geim, A. K., and K. S. Novoselov, 2010.  The rise of graphene.  _In_ Nanoscience and Technology: A Collection of Reviews from Nature Journals, World Scientific, 11–19.
  * Heath and Kuroda (2018) Heath, J. J., and M. A. Kuroda, 2018.  First principles studies of the interactions between alkali metal elements and oxygen-passivated nanopores in graphene.  _Phys. Chem. Chem. Phys._ .
  * Grootenhuis and Kollman (1989) Grootenhuis, P. D., and P. A. Kollman, 1989.  Molecular mechanics and dynamics studies of crown ether-cation interactions: free energy calculations on the cation selectivity of dibenzo-18-crown-6 and dibenzo-30-crown-10.  _J. Am. Chem. Soc._ 111:2152–2158.
  * Smolyanitsky et al. (2018) Smolyanitsky, A., E. Paulechka, and K. Kroenlein, 2018.  Aqueous ion trapping and transport in graphene-embedded 18-crown-6 ether pores.  _ACS Nano_ 12:6677–6684.
  * Fang et al. (2019) Fang, A., K. Kroenlein, D. Riccardi, and A. Smolyanitsky, 2019.  Highly mechanosensitive ion channels from graphene-embedded crown ethers.  _Nat. Mater._ 18:76.
  * Wipff et al. (1982) Wipff, G., P. Weiner, and P. Kollman, 1982.  A molecular-mechanics study of 18-crown-6 and its alkali complexes: an analysis of structural flexibility, ligand specificity, and the macrocyclic effect.  _J. Am. Chem. Soc._ 104:3249–3258.
  * Glendening et al. (1994) Glendening, E. D., D. Feller, and M. A. Thompson, 1994.  An ab initio investigation of the structure and alkali metal cation selectivity of 18-crown-6.  _J. Am. Chem. Soc._ 116:10657–10669.
  * Pereira et al. (2009) Pereira, V. M., A. C. Neto, and N. Peres, 2009.  Tight-binding approach to uniaxial strain in graphene.  _Phys. Rev. B_ 80:045401.
  * Lee et al. (2008) Lee, C., X. Wei, J. W. Kysar, and J. Hone, 2008.  Measurement of the elastic properties and intrinsic strength of monolayer graphene.  _Science_ 321:385–388.
  * Sahu et al. (2017) Sahu, S., M. Di Ventra, and M. Zwolak, 2017.  Dehydration as a Universal Mechanism for Ion Selectivity in Graphene and Other Atomically Thin Pores.  _Nano Lett._ 17:4719–4724.
  * Gibby et al. (2018) Gibby, W. A. T., M. Barabash, C. Guardiani, D. Luchinsky, and P. V. E. McClintock, 2018.  The Role of Noise in Determining Selective Ionic Conduction Through Nano-Pores.  _In_ 2018 IEEE 13th Nanotechnology Materials and Devices Conference (NMDC). IEEE, 1–6.
  * Fumagalli et al. (2018) Fumagalli, L., A. Esfandiar, R. Fabregas, S. Hu, P. Ares, A. Janardanan, Q. Yang, B. Radha, T. Taniguchi, K. Watanabe, et al., 2018.  Anomalously low dielectric constant of confined water.  _Science_ 360:1339–1342.
  * Fedorenko et al. (2018) Fedorenko, O., I. K. Kaufman, W. A. T. Gibby, D. Luchinsky, S. K. Roberts, and P. V. E. McClintock, 2018.  Quantized dehydration and the determinants of selectivity in the NaChBac bacterial sodium channel.  _arXiv:1803.07063_ .
  * Bucher et al. (2009) Bucher, D., L. Guidoni, P. Maurer, and U. Rothlisberger, 2009.  Developing Improved Charge Sets for the Modeling of the KcsA K+ Channel using QM/MM Electrostatic Potentials.  _J. Chem. Theory Comput._ 5:2173–2179.
  * Roux (2005) Roux, B., 2005.  Ion Conduction and Selectivity in K+ Channels.  _Annu. Rev. Biophys. Biomol. Struct_ 34:153.
  * Kuang et al. (2015) Kuang, Q., P. Purhonen, and H. Hebert, 2015.  Structure of potassium channels.  _Cell. Mol. Life Sci._ 72:3677.
  * Kandel et al. (2000) Kandel, E. R., J. H. Schwartz, T. M. Jessell, S. A. Siegelbaum, A. J. Hudspeth, et al., 2000.  Principles of Neural Science, volume 4.  McGraw-hill New York.
  * Deamer et al. (2016) Deamer, D., M. Akeson, and D. Branton, 2016.  Three decades of nanopore sequencing.  _Nat. Biotech._ 34:518.
  * Sahu and Zwolak (2019) Sahu, S., and M. Zwolak, 2019.  Colloquium: Ionic phenomena in nanoscale pores through 2D materials.  _Reviews of Modern Physics_ 91:021004.
  * Prozorovska and Kidambi (2018) Prozorovska, L., and P. R. Kidambi, 2018.  State-of-the-Art and Future Prospects for Atomically Thin Membranes from 2D Materials.  _Adv. Mater._ 30:1801179.
  * Joshi et al. (2014) Joshi, R., P. Carbone, F. Wang, V. Kravets, Y. Su, I. Grigorieva, H. Wu, A. Geim, and R. Nair, 2014.  Precise and ultrafast molecular sieving through graphene oxide membranes.  _Science_ 343:752–754.
  * Park et al. (2017) Park, H. B., J. Kamcev, L. M. Robeson, M. Elimelech, and B. D. Freeman, 2017.  Maximizing the right stuff: The trade-off between membrane permeability and selectivity.  _Science_ 356:1137.
  * Wang et al. (2011) Wang, H., Y. Yang, Y. Liang, J. T. Robinson, Y. Li, A. Jackson, Y. Cui, and H. Dai, 2011.  Graphene-wrapped sulfur particles as a rechargeable lithium–sulfur battery cathode material with high capacity and cycling stability.  _Nano Lett._ 11:2644–2647.
  * Siria et al. (2013) Siria, A., P. Poncharal, A.-L. Biance, R. Fulcrand, X. Blase, S. T. Purcell, and L. Bocquet, 2013.  Giant osmotic energy conversion measured in a single transmembrane boron nitride nanotube.  _Nature_ 494:455.
  * Sahu and Zwolak (2017) Sahu, S., and M. Zwolak, 2017.  Ionic selectivity and filtration from fragmented dehydration in multilayer graphene nanopores.  _Nanoscale_ 9:11424–11428.
  * Baker et al. (2007) Baker, K. A., C. Tzitzilonis, W. Kwiatkowski, S. Choe, and R. Riek, 2007.  Conformational dynamics of the KcsA potassium channel governs gating properties.  _Nat. Struct. Mol. Biol._ 14:1089.
  * (35) Frisch, M. J., G. W. Trucks, H. B. Schlegel, G. E. Scuseria, M. A. Robb, J. R. Cheeseman, G. Scalmani, V. Barone, B. Mennucci, G. A. Petersson, H. Nakatsuji, M. Caricato, X. Li, H. P. Hratchian, A. F. Izmaylov, J. Bloino, G. Zheng, J. L. Sonnenberg, M. Hada, M. Ehara, K. Toyota, R. Fukuda, J. Hasegawa, M. Ishida, T. Nakajima, Y. Honda, O. Kitao, H. Nakai, T. Vreven, J. A. Montgomery, Jr., J. E. Peralta, F. Ogliaro, M. Bearpark, J. J. Heyd, E. Brothers, K. N. Kudin, V. N. Staroverov, R. Kobayashi, J. Normand, K. Raghavachari, A. Rendell, J. C. Burant, S. S. Iyengar, J. Tomasi, M. Cossi, N. Rega, J. M. Millam, M. Klene, J. E. Knox, J. B. Cross, V. Bakken, C. Adamo, J. Jaramillo, R. Gomperts, R. E. Stratmann, O. Yazyev, A. J. Austin, R. Cammi, C. Pomelli, J. W. Ochterski, R. L. Martin, K. Morokuma, V. G. Zakrzewski, G. A. Voth, P. Salvador, J. J. Dannenberg, S. Dapprich, A. D. Daniels, O. Farkas, J. B. Foresman, J. V. Ortiz, J. Cioslowski, and D. J. Fox.  Gaussian 09.  Gaussian Inc. Wallingford CT 2009.
  * Breneman and Wiberg (1990) Breneman, C. M., and K. B. Wiberg, 1990.  Determining atom-centered monopoles from molecular electrostatic potentials. The need for high sampling density in formamide conformational analysis.  _J. Comput. Chem._ 11:361.
  * Kresse and Furthmüller (1996) Kresse, G., and J. Furthmüller, 1996.  Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set.  _Phys. Rev. B_ 54:11169–11186.
  * Monkhorst and Pack (1976) Monkhorst, H. J., and J. D. Pack, 1976.  Special points for Brillouin-zone integrations.  _Phys. Rev. B_ 13:5188–5192.
  * Tang et al. (2009) Tang, W., E. Sanville, and G. Henkelman, 2009.  A grid-based Bader analysis algorithm without lattice bias.  _J. Phys. Condens. Matter_ 21:084204.
  * Hutter et al. (2014) Hutter, J., M. Iannuzzi, F. Schiffmann, and J. VandeVondele, 2014.  _Wiley Inderdiscip. Rev: Comput. Mol. Sci._ 4:15.
  * Bayly et al. (1993) Bayly, C., P. Cieplak, W. D. Cornell, and P. A. Kollman., 1993.  A Well–Behaved Electrostatic Potential Based Method Using Charge Restraints For Determining Atom–Centered Charges: The RESP Model.  _J. Phys. Chem._ 97:10269.
  * Phillips et al. (2005) Phillips, J. C., R. Braun, W. Wang, J. Gumbart, E. Tajkhorshid, E. Villa, C. Chipot, R. D. Skeel, L. Kale, and K. Schulten, 2005.  Scalable molecular dynamics with NAMD.  _J. Comput. Chem._ 26:1781–1802.
  * Sahu and Zwolak (2018a) Sahu, S., and M. Zwolak, 2018.  Maxwell-Hall access resistance in graphene nanopores.  _Phys. Chem. Chem. Phys._ 20:4646 – 4651.
  * Sahu and Zwolak (2018b) Sahu, S., and M. Zwolak, 2018.  Golden aspect ratio for ion transport simulation in nanopores.  _Phys. Rev. E_ 98:012404.
  * Hénin and Chipot (2004) Hénin, J., and C. Chipot, 2004.  Overcoming free energy barriers using unconstrained molecular dynamics simulations.  _J. Chem. Phys._ 121:2904–2914.
  * Nelson (2002) Nelson, P. H., 2002.  A permeation theory for single-file ion channels: Corresponding occupancy states produce Michaelis–Menten behavior.  _J. Chem. Phys._ 117:11396–11403.


11institutetext: Computer Science Department
Maharaja Agrasen Institute of Technology
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>nimishamittal1999<EMAIL_ADDRESS>

# Domain Controlled Title Generation with Human Evaluation

Abdul Waheed 0000-0002-7229-3207 Muskan Goyal Corresponding Author
0000-0003-3643-7311
Nimisha Mittal ** 0000-0002-7178-2017 Deepak Gupta 0000-0002-3019-7161 ****

###### Abstract

We study automatic title generation and present a method for generating
domain-controlled titles for scientific articles. A good title allows you to
get the attention that your research deserves. A title can be interpreted as a
high-compression description of a document containing information on the
implemented process. For domain-controlled titles, we used the pre-trained
text-to-text transformer model and the additional token technique. Title
tokens are sampled from a local distribution (which is a subset of global
vocabulary) of the domain-specific vocabulary and not global vocabulary,
thereby generating a catchy title and closely linking it to its corresponding
abstract. Generated titles looked realistic, convincing, and very close to the
ground truth. We have performed automated evaluation using ROUGE metric and
human evaluation using five parameters to make a comparison between human and
machine-generated titles. The titles produced were considered acceptable with
higher metric ratings in contrast to the original titles. Thus we concluded
that our research proposes a promising method for domain-controlled title
generation.

###### Keywords:

Automatic title generation Natural language generation Transformer
Summarization technique T5 model Domain-control Additional token technique
Human Evaluation.

##  1 Introduction

The analysis of literature is a crucial practice for researchers to determine
relevant publications for the research topic. The title of the research papers
becomes really significant due to the availability of scientific papers in
abundance. Researchers can automatically determine the importance of a paper
by its title rather than reading the whole document [1, 2, 3]. The accuracy of
the title influences the number of potential readers and therefore the number
of citations [1, 4]. That’s why it’s important for the researchers to produce
a good title, however, people spend very little time on it [3]. This leads to
non-informative headings that do not take the entire content of the scientific
paper into account.

To establish a title for a scientific paper, one has to understand the
objective and characteristics of the paper, so that the meaning of the paper
can be distilled into the title with only a few words. A suitable title for a
particular paper must convey the core meaning of the paper briefly. The
automatic title generation task has traditionally been closely linked to the
conventional summarization technique because it can be considered as
scientific paper compression to represent its content [5, 6, 7]. For a given
paper, a short and concise description, conveying the complete details of the
text in just a few words, must be made. The task cannot, therefore, be
regarded as simple.

It is crucial to identify the type of information/domain of each text unit to
generate a suitable title. The performance of automatic title generation can
become unsatisfactory due to the differences between the domains of scientific
papers. Such differences include diverse vocabulary, different forms of
grammar, and various ways of expressing identical concepts. Therefore, it is
important to use domain-specific vocabulary to generate the title for
scientific papers. Although automated title creation is not capable of
replacing an author’s skills in creating a title, it is helpful to propose a
title.

In this research, we propose to utilize the text-to-text transformer model (T5
model) for domain-controlled title generation. The text-to-text transformer
model converts all the NLP tasks to a single text-text format where text is
used as input and output. This formatting allows the T5 model to perform a
number of tasks into its framework, including the summarization task. For
domain control, we used an additional token method which is discussed in
Section 3.3. Since an abstract of a scientific paper describes the author’s
work and presents all key arguments and relevant findings [8], we used it to
generate a suitable domain-controlled title for papers. Several researchers
have worked on automatic title generation. We examine some of these works
below.

###  1.1 Related Work

Liqun and Wang proposed a method called DTATG in [9] to produce titles. DTATG
was an unsupervised method that produced syntactically correct titles very
easily. This method used sentence segmentation to draw a limited number of
central sentences that express the core meanings of the text. DTATG created a
dependency tree for each of these sentences and extracted several branches for
trimming purposes with a dependency tree compression model. The authors also
designed a title test to figure out if the trimmed sentence can be used as a
title. The title with the top-ranked score was selected as the final title.
DTATG created titles similar to human-generated titles. However, DTATG was
constrained to the use of central phrases.

In [10], Putra et al. proposed a method for the creation and classification of
rhetorical corpus structures. The experiment was directly integrated into the
role of automatic title generation for scientific papers. Each sentence was
categorized into one of the three groups: OWN_MTHD (method), AIM (purpose),
and NR (not relevant). Firstly the abstract of the scientific papers was
annotated with the corresponding category. Then features like rhetorical
patterns and formulaic lexicon statistics were decided and analyzed for
classification purposes. Finally, several supervised learning algorithms were
utilized to build the classification model. The models were tested using
10-fold cross-validation and resulted in a weighted average F-measure between
0.70-0.79.

Liu, Wei, et al. [11] proposed a sentence2vec-enhanced Quality-Diversity
Automated Summarization (QDAS) model and attempted to implement transfer
learning for the Wikipedia title Generation task. Summaries from paragraphs
were derived by extractive summarization method and sequence labeling data was
provided to the model for title generation. The system involved only general
preprocessing, such as sentence splitting and word segmentation, and could be
implemented in a multilingual environment. The authors fine-tuned the BERT
based CRF model for Wikipedia article title generation.

Chen et al. [12] presented a method for domain adaptation using artificial
titles for title generation. The authors discussed the strategies for
modifying the encoder-decoder model for text generation from a domain marked
source to an unlabeled domain target. Sequential training was utilized in
order to capture the unlabeled target domain’s grammatical form. For the title
generation task, an encoder-decoder RNN model with domain control was used.
The encoder collected data from the source and the decoder learned to
construct summary captions. The source data and unlabeled target domain
information were encoded as their definition representations by a
bidirectional LSTM, and the domain classifier attempted to learn to
distinguish between the representations of two domains.

In [13], Gehrmann, Sebastian et al. proposed a method to generate titles for
short sections of long documents. The authors aimed at designing techniques in
a low-resource system for section title generation. They first picked the most
influential sentence and then performed a removal-based compression on an
extractive pipeline. The Semi-Markov Conditional Random Field compression
method was utilized. The method depended upon unsupervised textual
representations such as ELMo or BERT to eliminate the need for the design of
the complex encoder-decoder.

Most of the above works are centered on how relevant terms can be extracted in
the article for the title output without recognizing the domain of the text.
The domain offers a category of material that reflects the communication
purpose conveyed by a paper to the reader. It is alleged that the sentence
domain in the document could boost the effectiveness of automatic title
generation by including a certain sort of information in the document title.
In this research, we consider domains in the form of information types
communicated by abstracts of scientific papers. Also, we used a transformer-
based architecture model (T5 model) instead of using sequence-to-sequence
models (RNN, LSTM) to perform automatic domain-controlled title generation.
RNN/CNN handle sequences word-by-word sequentially which is an obstacle to
parallelize. Transformers remove recurrence and convolution entirely and
substitute them with self-attention to assess the dependencies between inputs
and outputs. Transformer achieve parallelization by replacing recurrence with
attention and encoding the symbol position in sequence. This in turn leads to
significantly shorter training time and better performance.

###  1.2 Contributions

While considering the above works, we have proposed a domain-controlled
automatic title generation method. The key contributions to this paper are
mentioned as follows:

  1. 1.

Propose a method for automatic domain-controlled title generation using
transformers.

  2. 2.

Provide meta-information about the domain using the additional token
technique.

  3. 3.

Compare the titles generated by the model with human-generated titles by
performing the human evaluation.

The remaining exhibition is the following. The background is presented in
Section 2. The proposed model is discussed in Section 3. Section 4 deals with
the simulation model, followed by Section 5 with results and discussion.
Section 6 addresses the conclusions and possible future directions.

##  2 Background

Natural Language Processing (NLP) is a facet of Artificial Intelligence and is
responsible for computers and machines to understand, interpret, and further
incorporate human languages. NLP is based on different disciplines, like
computer science and machine-translated linguistics. It further attempts to
narrow the communication gap between computers and humans.

###  2.1 Attention

Attention [14] proposed in machine translation alleviates the bottleneck of
seq2seq learning. It allows us to look at hidden states of the source sequence
and they are parsed as an additional input in the form of weighted average to
the decoder. It is not restricted to input sequence and the concept of self-
attention can be incorporated to analyze surrounding words to obtain
contextually sensitive word representations. Transformer architecture [15]
consists of various self-attention layers.

###  2.2 Pretrained language models

The pre-trained word embeddings initialize only the first layer of the model
and the embeddings are generally context-agnostic. Pretraining always learns
the important parameters of deep neural networks and are further fine-tuned
for various tasks. They were proposed in 2015 [16] but their benefits across
various tasks were recently observed. Pretraining involves improved model
initialization that boosts generalization performance and convergence speed
for the tasks. Being language models, they can solely rely on unlabeled
corpora, hence they are beneficial when labeled data is scarce.

###  2.3 Transformer architecture

The Transformer [15] is a model architecture that solves the problem of
parallelization by integrating self-attention techniques with convolutional
neural networks. Before the Transformer was introduced, Convolutional Neural
Networks (CNN) and Recurrent Neural Networks (RNN) were used for sequence
transduction. Similar to RNNs and CNNs, there is an encoder and decoder in the
transformer where each block includes a multi-head attention block, skip
connection, and normalization layer followed by a feed-forward block. With the
help of multi-head attention and positional embedding, the transformer network
has the capability of processing the data parallely. Often, transformer-based
models contain stacked encoders and decoders.

The self-attention mechanism in this model is responsible for correlating each
input token regardless of their position. Initially, all the words are
represented using high dimensional vectors or embeddings. This process is
performed in the bottom-most encoder that is subsequently passed to all the
succeeding encoders which perform self-attention to generate new
representation considering contextual information. The decoder having similar
architecture operates likewise but generates one word at a time by attending
the encoded contextualized representation and previously generated tokens.

##  3 Proposed Method

This section presents the methodology and architecture used in the research. A
detailed overview of the model and the process used to make the overall
concept comprehensible has been given. The present study discusses title
generation as a summarization process. Our suggested architecture comprises
three modules integrating a primary pipeline for summarization: pre-
processing, training of the T5 model, and generation of the title (summary).
The flowchart of the proposed method is shown in Fig. 1 below. Each module is
defined in detail in the following sections.

Figure 1: Flowchart of title generation using T5. Preprocessing involves
mainly tokenization which is based on subword tokenization.

###  3.1 Pre-processing: Subword Tokenization

The tokenization of subwords is a strategy that embraces uncertainty in data.
Subword tokenization breaks the text into subwords. For instance, terms like
smartest are segmented as smart-est, lower as low-er, etc. Transformer based
models depend on subword tokenization algorithms for vocabulary generation.
This method divides OOV(out of vocabulary) terms into subwords and depicts the
term in these subwords. The length of input and output phrases is less than
the tokenization of the character after subword tokenization. This method uses
the most frequently occurring character or sequence iteratively.

###  3.2 T5 Model

T5: Text-to-text-Transfer-Transfer model was proposed by Colin Raffel et al.
[17] to reframe all NLP-based activities to a single text-to-text format where
text strings are both input and output. This formatting enables the T5 model
to cast various tasks into this framework including machine translation,
classification task, regression task, and summarization task. This framework
uses the same model, hyperparameters, and loss function on the tasks
mentioned.

The model architecture is similar to standard vanilla encoder-decoder
transformers with similar computational cost and considering half parameters.
The input sequence is initially mapped to the embedding sequence and
transferred to the encoder afterwards. The encoder comprises a stack of
blocks, and each block consists of two subcomponents: a self-attention layer
and a feed-forward network. Normalization of the layer is introduced to each
subcomponent’s input. Normalization only includes rescaling of the activations
and no additive bias is introduced. This is followed by a residual skip
connection that adds the subcomponent’s input to its output. This model
considers a different position embedding scheme and these are passed as
parameters across all layers of the model.

The structure of the decoder is similar to the encoder except it incorporates
the standard attention process after the self-attention layer encoder’s
output. The self-attention in the decoder is a type of auto-regressive or
causal self-attention that only enables the algorithm to attend to previous
outputs. The final decoder block output is fed into a dense layer with a
softmax output, the weights of which are shared with the input embedding
matrix. Hence, the T5 model is similar to the original Transformer with an
exception of the removal of Layer Norm bias and using different position
embedding methods.

###  3.3 Additional token technique for domain control

Domain control or often called adaptation is an important research area in the
processing of natural languages. The arXiv dataset has a category/domain label
corresponding to each paper. By controlling the domain, title tokens will be
sampled from the local distribution of the domain-specific vocabulary (which
will be a subset of global vocabulary) rather than global vocabulary. The
produced title will therefore be attractive and realistic, closely associated
with its corresponding abstract.

Now, it would be cumbersome and time-consuming to train one separate model for
each type of domain. The solution to this problem is to add a reference to the
desired domain to the input text. The reference acts as a control tag specific
to each type of domain. This helps the model to learn the relation between the
control tags and the following text and only a preferred control tag must be
defined by the user to specify the type of text that is to be generated. This
control tag is the meta-information about the domain that we provide to the
model. The tags are supplied using the additional token method technique.

The additional token method [18] requires the addition of an artificial token
at the end of each abstract which enables the model to pay attention to the
domain of each title and abstract. The model reads the abstract with the
associated domain tag @DOMAIN-NAME. Domain tags are already defined in the
ArXiv metadata file, corresponding to each paper. Although simple, this
approach has proved to be successful [19, 20]. Note that the T5 model does not
include any information about the domain. It includes domain information
through this additional token approach. Now, we need to maximize the
probability of title using abstract and domain.

Domain Controlled,

$loss=-log\sum_{t=1}^{|y|}P(y_{t}^{*}|y_{<t}^{*},x,d)$ (1)

Without domain control,

$loss=-log\sum_{t=1}^{|y|}P(y_{t}^{*}|y_{<t}^{*},x)$ (2)

where,

$y_{t}^{*}=y^{<1>},y^{<2>},...,y^{<n>}$ denotes tokens in generated title

$x=x^{<1>},x^{<2>},...,x^{<m>}$ denotes tokens of the abstract

$d$ denotes domains (examples astro-ph: Astrophysics, cs.CC: Computational
Complexity, cs.CL: Computation and Language)

###  3.4 Training Procedure

Aforementioned, a T5 model takes input and produces output in text format. A
unique prefix tag is used to define and train the model on different tasks.
The format of input text passed to the model consists of three parameters,
prefix, input text, and target text. This makes training of the model simple
as the model only requires a change in the prefix tag to run a specific task.
The format of input can be stated as, ${<}$prefix${>}$:${<}$input_text${>}$.

The first step is to feed the text input into a layer of word embedding that
generates a vector representation of each word. Further, positional encoding
is used to inject positional data into the input embeddings. The Encoder layer
present transforms input sequences to abstract continuous representation. The
decoder is responsible for producing text sequences that are capped off with a
linear and a softmax layer. Moreover, the decoder consists of a start token, a
list of previous outputs that play the role of inputs along with encoder
outputs containing attention information. It stops decoding when a token is
produced as an output.
In our implementation, we used Beam Search with beam size 4 as the decoding
method because it eliminates the likelihood to miss hidden word sequences with
high probability. For instance, consider the abstract and title pair
illustrated below. This abstract is further passed to the T5 model after
appending the appropriate domain tag as represented to predict domain
controlled title of the scientific paper. The training loss graph of the model
for domain-control and without-domain control is shown in Fig. 2.

Abstract:
This paper proves that labelled flows are expressive enough to contain all
process algebras which are a standard model for concurrency. More precisely,
we construct the space of execution paths and of higher-dimensional homotopies
between them for every process name of every process algebra with any
synchronization algebra using a notion of labelled flow. This interpretation
of process algebra satisfies the paradigm of higher dimensional automata
(HDA): one non-degenerate full $n$-dimensional cube (no more no less) in the
underlying space of the time flow corresponding to the concurrent execution of
$n$ actions. This result will enable us in future papers to develop a
homotopical approach to process algebras. Indeed, several homological
constructions related to the causal structure of time flow are possible only
in the framework of flows.
Actual Title:
Towards a homotopy theory of process algebra

The T5 model reads the Abstract appended with the appropriate domain tag
(@DOMAIN-NAME) of the scientific paper.

Input(Abstract):
This paper proves that labelled flows are expressive enough to contain all
process algebras which are a standard model for concurrency. More precisely,
we construct the space of execution paths and of higher-dimensional homotopies
between them for every process name of every process algebra with any
synchronization algebra using a notion of labelled flow. This interpretation
of process algebra satisfies the paradigm of higher dimensional automata
(HDA): one non-degenerate full $n$-dimensional cube (no more no less) in the
underlying space of the time flow corresponding to the concurrent execution of
$n$ actions. This result will enable us in future papers to develop a
homotopical approach to process algebras. Indeed, several homological
constructions related to the causal structure of time flow are possible only
in the framework of flows. @domain: math.AT math.CT
Output(Predicted Title):
Labelled flows and homotopical approach to synchronization algebras

Figure 2: Training Loss

##  4 Simulation Model

###  4.1  Simulation Setup

The entire model is trained and tested on an Nvidia Tesla P100 GPU with 16GB
GPU VRAM using python language. The overall parameters of the model and
dataset are defined below.

###  4.2 Parameters

The T5 model used various parameters in the training and testing process. By
adjusting these parameters, the accuracy of the model can be determined. The
maximum sequence length of the input (abstract) is set to 512 and the maximum
sequence length of output (title) is set at 20. For both training and testing,
a batch size of 8 is used . The model is trained with a learning rate of 1e-5
for a total of 5 epochs. An evaluation occurred once for every 1000 training
steps.

###  4.3 Dataset

The model is trained on the popular ArXiv collection [21]. ArXiv has served
the general and research community with free access to scholarly publications
from the major fields of physics to several computer sciences disciplines and
all that goes between them, including electrical engineering, computational
biology, mathematics, and statistics. This large corpus of data provides
substantial, but often daunting, depth. This dataset contains the original
data from ArXiv. Since the complete dataset is very huge (with 1.7 million
articles), it provides a JSON-format metadata file. For each paper, this file
contains an entry with key features such as names of articles, authors,
domains/categories, abstracts, complete text PDFs, etc. For this research, we
used 1,10,000 samples from the dataset that is further divided into 90% for
training and 10% for the purpose of validation.

##  5 Results and Discussion

###  5.1 Automated Evaluation

Automatic evaluation is performed for evaluating systems when the output is
the text that is generally referred to as a sequence to sequence or string
transduction problem. For evaluating the results, generated text from the
model is compared with target (source) or reference text. There are various
evaluation metrics, but this paper focuses on the ROUGE metric.

Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [22] is an intrinsic
metric that is recall focused. This metric measures the number of n-grams or
words present in the target text summaries appeared in the generated text by
machines. There are three prominent types of Rouge scores including Rouge-N,
Rouge-L, and Rogue-S.

The N-gram overlap between the machine summary and the target summary is
represented in Rouge-N. Rouge-N contains Rouge-1 and Rouge-2, which in
generated and goal text correspond to the contrast of unigrams and bigrams,
respectively. The longest matching sequence of terms using LCS is measured by
Rouge-L. Rouge-S focuses on skip-bigrams based co-occurrence statistics. This
paper uses only Rouge-1 and Rouge-L metrics for comparison. The scores of the
ROUGE metric can be seen in Table 1.

Table 1: Results of the automatic evaluation. ROUGE score with abstract + domain is significantly better than without a domain.  Model |  Data |  ROUGE-1 |  ROUGE-L
---|---|---|---
T5 |  Abstract |  28.5 |  16.6
|  Abstract+Domain |  31.5 |  21.6

We trained the T5 model for title generation in two different ways. In the
first one, we generated the title directly from abstract text where we
observed that the tokens in the generated title were not specific to the
domain of the paper. Hence, to address this issue we controlled the domain
using additional token technique and with this approach we noticed significant
improvement. It can be inferred from Table 1 that ROUGE-1 and ROUGE-L scores
obtained for domain controlled abstracts (Rouge-1: 31.5 and Rouge-L: 21.6) are
considerably higher to the domain-less abstracts.

###  5.2 Human Evaluation

The interest in analyzing NLG (Natural Language Generation) texts in recent
years has grown by contrasting them to a corpus of human texts [23]. As with
other NLP fields, the benefits of automated corpus-based assessment are that
it is theoretically both cheaper and faster than human-based assessment and is
also repeatable. Corpus tests have been first used in the NLG, where the texts
scanned from a corpus have fed the output of the parser into the NLG system
and have then been compared to the original text of the corpus. In the NLG
group, such corpus-based evaluations were often criticized. Reasons for
criticism include the fact that the regeneration of a parsed text is not a
true NLG task; that texts can vary considerably from corpus text but are still
successful in achieving the communication purpose of the method and corpus
texts often are not of good quality enough to shape a practical evaluation.

Human Evaluation is another approach to evaluate NLG (Natural Language
Generation) based systems. This involves performing quality surveys of the
generated output using human annotators. In this approach, generated results
are presented to the people who assess the quality of the text on different
criteria. Using intrinsic and extrinsic approaches [23], human evaluation of
natural language generation systems can be performed. Intrinsic methods seek
to test the performance properties of the system by asking participants about
the fluency of the output of the system in a questionnaire. Extrinsic methods
aim to measure the effect of the system by evaluating the degree to which the
system accomplishes the overarching task for which it was created. Extrinsic
testing is the most time-consuming and cost-intensive of all possible tests;
hence it is very rare.

We conducted an intrinsic human evaluation with 40 participants. The academic
background and field of study of each participant were also recorded. We
prepared a form that paired 2 abstracts with the original and predicted title.
The nature of the title (original or predicted) was not known to any of the
participants. The participants judged and rated each title from 1(very bad) to
7 (very good). The judgments were based on various factors like coherence,
relevance, fluency, semantic adequacy, and overall quality of the title. The
aggregated mean score was first calculated for each factor, and then for each
of the original and predicted titles. The values are shown in Table 2.

In the process of human evaluation, each factor has its own significance.
Coherence determines whether or not the title is semantically meaningful, and
relevance indicates how applicable the title is to the abstract. Two titles
may equally effectively express the underlying intention while varying in
fluency. Therefore, the titles were judged for their readability with fluency
(able to understand the meaning of the title) and semantic adequacy (the title
is a sufficient representative of abstract) as its factors. It is often
difficult for human annotators to differentiate between the different quality
measures. This is why the overall quality of a title is evaluated directly.

The most suitable number of answers can rely on the task itself, but for most
tasks, 7-point ratings are the best. Several studies suggest that 7-point
ratings optimise reliability and discriminatory power [24, 25, 26]. One
7-point likert question has 7 points for discrimination but 4 7-point likert
questions have 4*7=28 discrimination points. Therefore, one 7-point likert
rating was used for each of the 4 titles in human evaluation form.

Table 2: Human evaluation scores. Compared with human-scaled, T5 with domain generated titles are scored better than human-generated titles by human evaluators on almost all parameters. Model |  Coherence |  Relevance |  Fluency |  Semantic Adequacy |  Overall Quality |  Mean Score
---|---|---|---|---|---|---
Human |  4.2 |  4.29 |  4.11 |  4.05 |  4.16 |  4.16
T5 + Domain |  4.53 |  4.63 |  4.58 |  4.43 |  4.6 |  4.55
Scaled |  1.08 |  1.08 |  1.11 |  1.09 |  1.11 |  1.09

It is clearly evident from Table 2 that the predicted title has higher values
of evaluation metrics (coherence, relevance, fluency, semantic adequacy, and
overall quality) in comparison to the original title. Overall, on normalizing
the difference between the values of human-generated and machine-generated
titles, the scaled value is in the range from 1.08 to 1.11. Moreover, the mean
score of the title predicted using the proposed model is greater than human-
generated titles. This distinctly depicts that the title produced using the T5
model and additional token is more appropriate and useful for the authors of
scientific papers.

##  6 Conclusion and Future work

In this research, we proposed the use of the T5 model for automatic domain-
controlled title generation for scientific papers. We have used the ArXiv
dataset for our research and analysis. Scholars, professional authors,
students, and teachers may use this method. Sparseness is the biggest struggle
of the title generation. For a document with several optional words, it is
important to create a brief and succinct title. An abstract of a scientific
paper contains the goal and method of the research, along with all the key
findings. Therefore, we utilized a scientific paper’s abstract to produce the
title for the paper.

We also identify the communication purpose of the authors bypassing the domain
of the paper as the meta-information. The quality of the automatic title
generated improves because the title tokens are sampled from the local
vocabulary of the domain, rather than the global vocabulary. The titles
generated from the model looked satisfactory and realistic.

To analyze the results we used automated and human evaluation. For automated
evaluation, we calculated ROUGE scores to prove that our approach leads to
high performance in the generation of automatic domain-controlled titles. In
the human-subject analysis, we used 40 participants with diverse reading
capabilities and academic backgrounds. We noticed that our generated titles
have higher values of evaluation metrics (coherence, relevance, fluency,
semantic adequacy, and overall quality) in comparison to the original title.

Although the abstract is a brief representation of the paper, writers may have
written the abstract hastily. Therefore, it is suggested to use the complete
paper text along with more refined weighing, title generation, and selection
methods, in order to better capture the prominent tokens of a document in a
particular domain. Another task that can be addressed in the future is to
create new techniques that deliver titles which are more close to human
generated titles.

## References

  * [1] Putra, J.W., & Khodra, M.L., “Automatic Title Generation in Scientific Articles for Authorship Assistance: A Summarization Approach”. Journal of ICT Research and Applications, 11, 253-267, 2017.
  * [2] Hamid R.Jamali, M., & Nikzad, M., “Article title type and its relation with the number of downloads and citations”. Scientometrics, 88, 653-661, 2011.
  * [3] Xu, Han, E. Martin, and Ashesh Mahidadia. “Extractive Summarisation Based on Keyword Profile and Language Model.” In HLT-NAACL (2015).
  * [4] Paiva, C. et al. “Articles with short titles describing the results are cited more often.” Clinics 67 (2012): 509 - 513.
  * [5] Jin, Rong and A. Hauptmann. ”Headline Generation using a Training Corpus”. Carnegie Mellon University (Journal contribution) https://doi.org/10.1184/R1/6606059.v1
  * [6] Witbrock, M. and V. Mittal. “Ultra-summarization (poster abstract): a statistical approach to generating highly condensed non-extractive summaries.” SIGIR ’99 (1999).
  * [7] Kupiec, J. et al. “A trainable document summarizer.” SIGIR ’95 (1995).
  * [8] Putra, J.W.G. & Fujita, K., “Scientific Paper Title Validity Checker Utilizing Vector Space Model and Topics Model”, In Proceedings of Konferensi Nasional Informatika (KNIF), pp. 69-74, 2015.
  * [9] Shao, Liqun, and J. Wang. “DTATG: An Automatic Title Generator based on Dependency Trees.” KDIR (2016).
  * [10] Putra, Jan Wira Gotama, and M. L. Khodra. “Rhetorical Sentence Classification for Automatic Title Generation in Scientific Article.” TELKOMNIKA Telecommunication Computing Electronics and Control 15 (2017): 656-664.
  * [11] Liu, Wei et al. “Multi-lingual Wikipedia Summarization and Title Generation On Low Resource Corpus.” Proceedings of the Multiling 2019 Workshop, co-located with the RANLP 2019 conference, pages 17–25.
  * [12] Chen, Francine R., and Yan-Ying Chen. “Adversarial Domain Adaptation Using Artificial Titles for Abstractive Title Generation.”, In ACL (2019).
  * [13] Gehrmann, Sebastian et al. “Improving Human Text Comprehension through Semi-Markov CRF-based Neural Section Title Generation.” ArXiv abs/1904.07142 (2019): n. pag.
  * [14] Bahdanau D., Cho K., & Bengio Y. “Neural Machine Translation by Jointly Learning to Align and Translate.” ICLR (2015).
  * [15] Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A. N., Polosukhin I. “Attention Is All You Need.”, In Advances in Neural Information Processing Systems (2017).
  * [16] Dai A. M., & Le Q. V. “Semi-supervised Sequence Learning.”, Advances in Neural Information Processing Systems NIPS ’15 (2015).
  * [17] Raffel, Colin et al. “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.” ArXiv abs/1910.10683 (2019): n. pag.
  * [18] Kobus, C. et al. “Domain Control for Neural Machine Translation.” RANLP (2017).
  * [19] Sennrich, Rico et al. “Controlling Politeness in Neural Machine Translation via Side Constraints.” HLT-NAACL (2016).
  * [20] Johnson, M. et al. “Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation.” Transactions of the Association for Computational Linguistics 5 (2017): 339-351.
  * [21] Clement, Colin B. et al. “On the Use of ArXiv as a Dataset.” ArXiv abs/1905.00075 (2019): n. pag.
  * [22] Lin, Chin-Yew. “ROUGE: A Package for Automatic Evaluation of summaries.”, Proceedings of the ACL Workshop: Text Summarization Branches Out (2004).
  * [23] Lee, C. et al. “Best practices for the human evaluation of automatically generated text.” INLG (2019).
  * [24] Miller, G. A.. “The magical number seven plus or minus two: some limits on our capacity for processing information.” Psychological review 63 2 (1956): 81-97 .
  * [25] Green, P. and V. Rao. “Rating Scales and Information Recovery—How Many Scales and Response Categories to Use?” Journal of Marketing 34 (1970): 33 - 39.
  * [26] Lissitz, R. W. and S. Green. “Effect of the number of scale points on reliability: A Monte Carlo approach.” Journal of Applied Psychology 60 (1975): 10-13.