|  Confinement twists achiral liquid crystals and causes chiral liquid
crystals to twist in the opposite direction†
---|---
|  Jungmyung Kima and Joonwoo Jeong∗a
|  We study the chiral symmetry breaking and metastability of confined nematic
lyotropic chromonic liquid crystal (LCLC) with and without chiral dopants. The
isotropic-nematic coexistence phase of the LCLC renders two confining
geometries: sessile isotropic(I) droplets surrounded by the nematic(N) phase
and sessile nematic droplets immersed in the isotropic background. In the
achiral system with no dopants, LCLC’s elastic anisotropy and topological
defects induce a spontaneous twist deformation to lower the energetic penalty
of splay deformation, resulting in spiral optical textures under crossed
polarizers both in the I-in-N and N-in-I systems. While the achiral system
exhibits both handednesses with an equal probability, a small amount of the
chiral dopant breaks the balance. Notably, in contrast to the homochiral
configuration of a chirally doped LCLC in bulk, the spiral texture of the
disfavored handedness appears with a finite probability both in the I-in-N and
N-in-I systems. Our director field model elucidates the boundary conditions
and energetics behind this chiral symmetry breaking and the metastability of
the director configuration having the disfavored handedness. This work sheds
light on design considerations in homochiral configurations under confinement.

##

††footnotetext: aDepartment of Physics, Ulsan National Institute of Science
and Technology, Ulsan, Republic of Korea. E-mail:
<EMAIL_ADDRESS>† Electronic Supplementary Information (ESI)
available: [details of any supplementary information available should be
included here]. See DOI: 10.1039/cXsm00000x/

Chiral structures are ubiquitous, ranging from molecules to astrophysical
phenomena. Understanding the mechanism underlying this mirror symmetry
breaking is an active field of investigation. Simultaneously, the control and
separation of chiral structures are crucial in diverse applications 1, 2, 3,
4, 5. For instance, chiral molecules can be toxic in unwanted enantiomeric
forms 6, and these chiral objects as building blocks or dopants can determine
their assembled structures’ chirality. Additionally, external fields,
including shear and confinement, are often employed to control chiral
structures 7, 8, 9, 10.

Liquid crystal (LC) is the quintessential model system for studying chiral
structures 11, 12, 13, 14, 15, 16, 17. In particular, chiral nematic LCs, also
known as the cholesteric phase, have been studied extensively. They are
omnipresent in nature and deployed in various applications 18, 19, 20, 21, 22,
23, 24. In the cholesteric phase, LC mesogens twist along the helical axis,
defining its handedness — right- or left-handed — and helical pitch, which are
often determined by the intrinsic chirality of the mesogen or dopants. Thus,
one may design chiral structures by choosing mesogens or the dopant’s type and
concentration at a given temperature.

The boundary conditions and elasticity of confined LCs also play a vital role
in determining their chiral structures 25, 26, 27, 28, 8, 9, 29, 10, 30, 31,
32, 33, 34, 35, 36, 37, 38, 39, 40, 41. The confining geometry and LC surface
anchoring at the interface between LC and the surrounding phase can result in
unexpected structures that differ from the ones in bulk. For example,
confinement may lead to an apparent helical pitch different from the bulk
pitch42, 43. Furthermore, achiral LCs under confinement can form chiral
structures 8, 9, 10, 29, 30, 31, 32 and vice versa 34. Confinement also can
give rise to multiple meta-stable structures 44, 45, 43, 46, i.e., not at the
energy ground state but experimentally observable. These confinement effects
involving surface anchoring, elastic anisotropy, saddle-splay elasticity,
topological defects, and metastability, have advanced our understanding of the
partially ordered matter and its applications.

Here, we report that chiral nematic LC can exhibit a meta-stable structure of
the disfavored handedness, i.e., opposite to the handedness in bulk, because
of boundary conditions. We use achiral and chiral nematic lyotropic chromonic
LCs to study their director configuration in and around sessile droplets on an
anchoring-controlled substrate. Our experiments and numerical calculations
demonstrate the roles of confinement and elastic anisotropy in the chiral
symmetry breaking of achiral LC. Moreover, we elucidate how chiral LC can
afford chiral structures of the opposite handedness as metastable states,
thanks to confinements.

## Materials and Methods

### Materials

Sunset Yellow FCF (SSY, a purity of 90%) was purchased from Sigma-Aldrich and
purified using the conventional method 47. Brucine sulfate heptahydrate (BSH)
as a chiral dopant for SSY was purchased from Sigma-Aldrich and used without
purification. Mixing deionized water, SSY, and BSH, we prepared an aqueous
solution of 30.0% (wt/wt) SSY with 0 to 0.4% (wt/wt) of BSH as achiral
(without BSH) or chiral nematic LC solution at room temperature.

### Sample Preparation

We coated glass slides and coverslips with parylene 48 and used them as
substrates for the homeotropically aligned cell. The substrates were placed in
a parylene coating machine (LAVIDA 110; Femto Science) with 500 mg of
parylene-N (diX N, Daisan Kasei), and the temperatures for vaporizer,
pyrolyzer, and deposition of parylene were $150^{\circ}\mathrm{C}$,
$690^{\circ}\mathrm{C}$, and room temperature, respectively. The estimated
thickness of the parylene film is approximately 400 nm 49. Utilizing Kapton
films of 25 $\mu$m thickness on the surface-treated slide as spacers, we
dropped 10 $\mu$l of LC solution on the parylene-coated slide and quickly
covered the specimen with the coated coverslip. The edges of the sandwich cell
were sealed with epoxy glue to minimize the drying of the aqueous LC solution.
The sandwiched nematic SSY initially shows the transient stripe texture, but
after several tens of minutes, the stripe texture is replaced by a full
homeotorpically aligned domain 48. However, when the dopant concentration
surpasses 0.4% (wt/wt), the sandwiched nematic SSY no longer exhibits the
homeotropic alignment but the fingerprint texture, not providing the proper
initial condition for our droplet experiments, as shown in Fig. S1.

### Optical Observation

We observed the sandwich cell with a polarized optical microscope (Olympus
BX53-P) equipped with linear polarizers, full waveplate (optical path
difference = 550 nm; Olympus), a temperature-controlled stage (T95-PE120;
Linkam Scientific Instruments), a CCD camera (INFINITY3-6URC; Lumenera), and a
LED lamp (LED4D067; Thorlabs) providing quasi-monochromatic illumination at
660 nm wavelength (FWHM = 25 nm). We increased the temperature,
$1^{\circ}\mathrm{C}/$min, of the aligned LC cell to make the LC enter the
isotropic-nematic coexistence phase at $45^{\circ}\mathrm{C}$. Repeating the
heating-cooling sequence, we collected multiple polarized optical microscopy
(POM) images from the same location of the same sample.

### Numerical Calculations

The total elastic free energy $F$ is the volume integral of the energy density
$f$ given by

$\displaystyle
f=\frac{1}{2}[K_{1}(\nabla\cdot\mathbf{n})^{2}+K_{2}(\mathbf{n}\cdot\nabla\times\mathbf{n}+2\pi/p_{0})^{2}$
$\displaystyle+K_{3}(\mathbf{n}\times\nabla\times\mathbf{n})^{2}],$ (1)

where $K_{1}$, $K_{2}$, and $K_{3}$ are elastic moduli of splay, twist, and
bend deformations, respectively. Note that we neglect a saddle-splay
elasticity term with $K_{24}$ because it does not affect the energetics when
the homeotropic anchoring is strong (at the substrates), and the planar
anchoring is imposed on the surface with the same principal curvatures (the
surface of the hemispherical droplet). $p_{0}$ is the helical pitch of the
chiral nematic phase and is inversely proportional to the concentration of a
chiral dopant. Assuming that the elastic moduli remain similar in the nematic
phase of the coexistence and the moduli do not depend on the dopant
concentration, we use $K_{1}:K_{2}:K_{3}=1:0.1:1$ for the 30.0% (wt/wt) SSY 50
and $p_{0}=1/(C*HTP)$ where $C$ is BSH concentration in % (wt/wt) and $HTP$ is
helical twisting power, 0.123 $\pm$ 0.007 $\mu$m-1 (see Fig. S2 for details).
For a given director field $\mathbf{n}$, we perform a numerical integration
$F=\int fdv$ over domain $\mathbb{D}$ to draw the energy landscape such as in
Fig. 3(a). For the I-in-N case, i.e., the sessile isotropic droplet surrounded
by the nematic phase, we adopted $\mathbb{D}=\\{(x,y,z)\mid
x^{2}+y^{2}\leq(5R)^{2},0\leq z\leq 5R,x^{2}+y^{2}+z^{2}\geq
R^{2},x^{2}+y^{2}+(z-R)^{2}\geq R_{d}^{2}\\}$ where $R$ is the radius of the
hemispherical droplet. To facilitate the convergence of the numerical
integration, we set a cutoff radius $R_{d}=10^{-3}R$ around the point defect
at the top of the droplet; $R_{d}=10^{-3}R$ corresponds to 10 nm, which is of
the same order of the persistence length of the SSY aggregates 50 in a droplet
of 10 $\mu$m radius. For the N-in-I case, we used $\mathbb{D}=\\{(x,y,z)\mid
0\leq z,x^{2}+y^{2}+z^{2}\leq(R)^{2},x^{2}+y^{2}+(z-R)^{2}\geq R_{d}^{2}\\}$
and $R_{d}=10^{-3}R$. After identifying the director configurations
corresponding to local minima in the energy landscape of each director fields
model, we simulated their POM textures using Jones calculus10 and compared the
results with experimental observations.

## Results and Discussion

### Sessile Isotropic Droplets Surrounded by Nematic Phase

We observe chiral symmetry breaking of achiral nematic LC around sessile
isotropic liquid droplets. Figure 1 shows our lyotropic chromonic LC, SSY, at
its isotropic-nematic coexistence phase confined between two parallel
substrates imposing homeotropic anchoring. When a homeotropically aligned
nematic SSY enters the coexistence phase upon a temperature increase, as shown
in Fig. 1(a), sessile droplets of the isotropic phase appear at the bottom
substrate. Because the temperature-controlling Peltier plate contacts the
bottom substrate, the isotropic droplets nucleate from the bottom substrate.
We make and observe the droplets only at the bottom substrate. We also confirm
no memory effect by comparing the location, size, and handedness of droplets
in different images. The sessile isotropic droplets deform the homeotropically
aligned nematic director field, and the deformed regions exhibit spiral
textures under POM, as shown in Fig. 1(b)-(f). The appearance of the chiral
spiral texture, despite the absence of chirality in SSY, is notable. Non-
deformed nematic regions remain dark under crossed polarizers because of the
homeotropic alignment.

We classify the spiral textures’ handedness, investigate their appearance
probability, and find that each handedness is equally probable. In the right-
handed spiral texture (RS), the dark line turns right while going from the
center to the boundary. In three independent experiments, each collecting
approximately 270 droplets, the average probability of observing the RS
texture is 0.509 $\pm$ 0.021, where the error indicates the standard deviation
of the three measurements. This half-and-half probability is consistent with
the achiral nature of SSY, i.e., no preference for a specific handedness.

With a chiral dopant added, the spiral texture prefers a specific handedness.
As we increase the concentration of a dopant, BSH, the appearance probability
of RS increases and excels left-handed spiral texture (LS)’s, as shown in Fig.
1(g). For example, with 0.4% (wt/wt) BSH added to 30% (wt/wt) SSY, almost 90%
of the spiral texture becomes RS. It is unexpected that 10% remains as the
energetically unfavorable LS at our maximum dopant concentration because a
chirally doped LC in bulk would exhibit a homochiral director configuration
with a well-defined helical pitch 51, 52, 33, 53, 54, 55.

Fig. 1:  Spiral textures in the isotropic-nematic coexistence phase of Sunset
Yellow under confinement. (a) Schematic diagram of the sample from the side
view. The isotropic-nematic coexistence phase is sandwiched between two
parylene-coated substrates. Sessile droplets of the isotropic(I) phase
surrounded by the nematic(N) sit on the heated bottom substrate. Yellow rods
in the nematic phase sketch the directors. (b) Spiral textures observed with
crossed polarizers from the top view. White double arrows represent the pass
axes of the polarizer and analyzer. The scale bar is 50 $\mu$m. (c)-(d)
Representative spiral textures having different handedness (c) without chiral
dopant and (d) with 0.4% (wt/wt) chiral dopant. The width and height of the
images are 30 $\mu$m, respectively. (e)-(f) Spiral textures of (c) and (d)
with a full waveplate. (g) Appearance probability of the spiral textures
having different handedness according to the chiral dopant concentration. As
the concentration increases, a certain handedness becomes dominant, but the
other handedness exists even at the highest dopant concentration investigated.
Fig. 2:  Director configurations of a sessile isotropic droplet surrounded by
nematic phase. (a) A coordinate system to represent the director
configuration. A yellow rod is the director at the position $\mathbf{r}$, and
two black rods are the projection of the yellow director on $x-y$ plane and
$x-z$ plane, respectively. $\alpha$, $\beta$, and $\delta$ are the angles
between the coordinate axes and the director or the projected directors. (b)
Director visualization of the bipolar planar quadrupole configuration. The
directors are shown as yellow rods on the surface of the droplet and on the
$y-z$ plane. The black dot at the top of the droplet is the point defect. (c)
One example of the twisted bipolar planar quadrupole configuration. (d) Twist
angle $\delta$ profile of (c) as a function of the normalized radius $\rho/R$
in the cylindrical coordinate along the $\rho$ axis at $z/R=1.1$.

To elucidate the observed chiral symmetry breaking in achiral SSY and
metastability in chiral SSY, we propose a director field model of nematic LC
around the isotropic droplet. First, we assume the isotropic sessile droplet
in the model has a hemispherical shape, as shown in Fig. 1(a). It is
surrounded by the nematic phase satisfying the boundary conditions: an
infinitely strong homeotropic anchoring both on the top and bottom substrates
and a degenerate planar anchoring on the hemispherical droplet’s surface. We
parameterize the director field $\mathbf{n}$ with two spatially varying
angles, $\alpha$ and $\beta$.

$\displaystyle(n_{x},n_{y},n_{z})=(\sin{\beta}\cos{(\phi+\alpha)},\sin{\beta}\sin{(\phi+\alpha)},\cos{\beta}).$
(2)

The equation describes the Cartesian components $(n_{x},n_{y},n_{z})$ of a
director $\mathbf{n}$ at a position ($r$, $\theta$, $\phi$) in the spherical
coordinate. For the angle $\beta$ between a director and $z$-axis shown in
Figure 2(a), we adopt the bipolar planar quadrupole configuration 56.

$\displaystyle\beta=2\theta-\tan^{-1}\left(\frac{(r/R)^{3/2}\sin{\theta}}{(r/R)^{3/2}\cos{\theta}-1}\right)$
$\displaystyle-\tan^{-1}\left(\frac{(r/R)^{3/2}\sin{\theta}}{(r/R)^{3/2}\cos{\theta}+1}\right).$
(3)

This configuration satisfies the boundary conditions, i.e., the homeotropic
alignment at the far field and the planar alignment on the hemispherical
droplet’s surface, resulting in a point defect at the droplet’s pole, also
known as a boojum.

Fig. 3:  Elastic free energy landscape of the I-in-N system according to the
model parameter $a$ and $c$ and their comparison with experimental data. (a)
The color-coded energy landscape of the achiral nematic SSY with no chiral
dopant. Two filled star symbols indicate two degenerate global minima. The
patterned regions at the bottom left and right are excluded because they
violate the boundary conditions considerably. (b) Comparison of experimentally
observed spiral textures with Jones-calculus-simulated POM textures of the
director configurations located at two global minima of the energy landscape
shown in (a). (c) The color-coded energy landscape of the chiral nematic SSY
with 0.4% (wt/wt) BSH. The global minimum is indicated by a filled symbol, and
the local minimum at the top-right region is shown with an empty star. (d)
Comparison of experimental images with simulated POM textures based on the
minima in (c). Left and right column of POM images represent global and local
minimum, respectively.

We hypothesize that the experimentally observed spiral POM textures result
from a twisted director configuration. We add twist deformation to the bipolar
planar quadrupole configuration by introducing the angle $\alpha$ in Eq. 2. As
shown in Fig. 2(a), $\alpha$ is the angle between the radial direction
$\hat{\rho}$ in the cylindrical coordinate and the director’s projection onto
the $x-y$ plane. If $\alpha=0$, the director field becomes the bipolar planar
quadrupole configuration with no $\hat{\phi}$-component.

Our ansatz adopts $\alpha$ as a power-law decaying function of $\rho$,
satisfying the cylindrical symmetry and the far-field boundary condition.
Namely, $\alpha$ only depends on the radial distance from the $z$ axis and
satisfies $\alpha(\rho\to\infty)\to 0$, i.e., the deformation-free homeotropic
alignment at the infinity.

$\displaystyle\alpha=a\left(\frac{1}{1+(\rho/R)}\right)^{c},$ (4)

Two constants $a$ and $c$ determine the decaying $\alpha(\rho)$’s profile and
how the twist angle $\delta$ defined in Fig. 2(a) varies spatially. Fig. 2(d)
is a representative $\delta$-$\rho$ profile for given $a$ and $c$. A large $a$
results in the large peak value of $\delta$, and a large $c$ leads to a fast
decay as $\rho$ increases.

The degenerate planar anchoring of the bipolar planar quadrupole configuration
at the droplet surface can be violated when we adopt $\alpha\neq 0$ for the
twist deformation. Varying the parameters $a$ and $c$, we investigate the
maximum polar deviation angle of the surface directors from the tangential
plane at the droplet surface. Then, we exclude the parameter regimes having
the maximum deviation angle greater than 20 deg from the region of interest,
only allowing a slight violation of the planar anchoring condition at the
isotropic-nematic interface.

By numerically calculating the total elastic free energy $F$ of the above-
mentioned director field model according to $a$ and $c$, we find the director
configurations at local energy minima, which agree well with experimental
observations. See the Methods section for the details of the elastic free
energy calculation. Figure 3(a) shows the energy landscape of the achiral
system, highlighting the local minima. In this achiral system, the energy
landscape is symmetric about the axis of $a=0$, i.e., $F(a)=F(-a)$; the sign
change of the angle $\alpha$ means the flipping of the spiral handedness. We
locate $(a,c)$ giving the two degenerate global minima of the energy
landscape, simulate the corresponding POM textures using Jones calculus, and
compare them with the experimental results, as shown in Fig. 3(b). The
simulated textures well reproduce the spiral textures of different
handednesses. Additionally, two degenerate global minima are consistent with
the observation that the spiral textures of different handedness appear with
equal probabilities in the achiral system.

The elastic anisotropy of the nematic SSY is responsible for two global minima
at non-zero $a$ and $c$, resulting in the two degenerate twisted ground
states. Fig. S3 shows the energy difference between our twisted ground-state
configuration and the twist-free bipolar planar quadrupole configuration. It
indicates that the relatively cheap twist deformation cancels out the splay
deformation near the topological defect, resulting in a net decrease in the
total elastic free energy. This scenario is similar to the cases in the
literature, where the achiral SSY under confinement breaks its chiral symmetry
because of the very small $K_{2}$ compared to other elastic moduli 10, 31, 32,
57. Comparison of energy landscapes at different $K_{2}$s corroborates the
role of elastic anisotropy in the chiral symmetry breaking. Fig. S4 shows that
two degenerate global minima exist when $k_{2}=K_{2}/K_{3}<0.4$ with
$K_{1}=K_{3}$, while only one achiral ground state, i.e., the configuration
with $a=0$, exists with the $k_{2}>0.7$.

With chiral dopants added, as shown in Fig. 3(c), the energy landscape becomes
asymmetric but still retains two minima. This is consistent with our
experimental observation that the spiral textures of both handednesses appear.
Because $p_{0}$ in Eq. 1 is no longer zero, the director configuration
energetically favors a twist deformation of a given helical pitch and
handedness imposed by the chiral dopant, making the energy landscape
asymmetric. Notably, two global minima in the achiral system shown in Fig.
3(a) become one global minimum and one local minimum in the chiral system
shown in Fig. 3(c). The director configuration of the preferred handedness has
the lower energy and corresponds to the global minimum, which is a more
frequently observed spiral texture in our experiments. For instance, 0.4%
(wt/wt) of chiral dopant gives right-handed spiral texture handedness with the
probability 0.894 $\pm$ 0.032 out of 794 droplets. Intriguingly, the director
configuration of the disfavored handedness exists with a finite probability,
even though we add the chiral dopant to make the configuration favor a
particular handedness. In fact, in the calculated energy landscape, there
exists the local minimum with higher energy than the global minimum; See the
top-right region of Fig. 3(c).

We find that a constraint from the symmetry and the far-field boundary
condition results in the appearance of both handednesses. As shown in Fig.
2(c), the directors along the rotational axis, i.e., $z-$axis, are aligned
vertically because of the symmetry. Starting from this symmetric axis, the
directors twist along the radial direction $\hat{\rho}$ to lower the total
elastic free energy in both achiral and chiral cases. However, to satisfy the
far-field boundary condition, i.e., the uniform homeotropic alignment, the
director should untwist to be vertical again unless the centrosymmetric
directors twist by 180 degrees. In other words, as shown in Fig. 2(d), the
system should have both twist-handedness, i.e., the positive and negative
slopes in $\delta$, regardless of the favored handedness of the system. This
constraint justifies that, even in highly doped specimens preferring one-
handedness and paying high energy penalty for the other-handedness, the
disfavored spiral texture can be observed experimentally as a local minimum in
the energy landscape. With different symmetry and boundary conditions allowing
the homochiral twist deformation, one may expect chiral LCLCs to exhibit the
homochiral spiral texture. We test this idea in the following section with the
inverted system.

### Sessile Nematic Droplets Surrounded by Isotropic Phase

Fig. 4:  Schematic, spiral textures, and energy landscape of the N-in-I
system. (a) Director visualization of a representative twisted configuration
in the sessile nematic droplet. The directors are shown as yellow rods on the
$y-z$ plane. The black dot at the top of the droplet is the point defect. (b)
The comparison of experimental POM textures with simulated textures based on
the energy-minimizing configurations of the energy landscape. (c) The color-
coded energy landscape of the achiral nematic SSY according to the model
parameters $a$ and $c$ at $b=2.51$. The symmetry landscape with the negative
$a$ is not shown. The filled symbol indicates one of two degenerate global
minima. (d-e) The color-coded energy landscape of the chiral liquid crystal
with 0.2% (wt/wt) BSH. (d) shows the global minimum as the filled symbol in
the $a-c$ plane at $b=2.09$. (e) shows the local minimum at $b=3.02$.

We observe the spiral textures again from sessile nematic LCLC droplets
surrounded by the isotropic phase, i.e., N-in-I, of which the boundary
conditions allow homochiral configurations. Figure 4(b) shows the POM
textures. In contrast to the I-in-N case at the temperature just above the
nematic-to-coexistence phase transition temperature, the temperature of the
N-in-I sample is at the one just below the coexistence-to-isotropic phase
transition temperature. The isotropic regions grow and merge during the
heating, and nematic regions eventually become sessile droplets surrounded by
the isotropic phase. We find that sessile droplets surrounded by air can
implement the same boundary conditions. The spiral textures result from the
twist deformation of the nematic phase confined within the sessile droplets.
The absence of the far-field boundary condition, such as the homeotropic
alignment in the I-in-N case, allows homochiral configurations. This chiral
symmetry breaking and the energetics are similar to those in the suspended
spherical droplets 10. The LCLC’s cheapest twist deformation cancels out the
high elastic energy penalty around the surface point defect, i.e., the black
dot in Fig. 4(a). As expected, in the achiral system with no chiral dopant, we
observe both handednesses in the spiral texture with equal probabilities: the
ratio of RS to LS is 9 to 8.

To our surprise, the sessile nematic droplets of chirally doped LCLC do not
exhibit homochirality, even though the geometry and boundary conditions allow
homochiral configurations. When 0.1% (wt/wt) of BSH is added, 41 out of 43 are
left-handed. Namely, we observe the disfavored handedness experimentally, even
though they are rarely observed. When we increase the dopant concentration to
0.2% (wt/wt), all droplets exhibit the homochiral configuration. Please note
that we observe the same metastability in the N-in-air system; the data is not
shown here because the sessile-droplet shapes surrounded by the air are not
regular enough to be a well-defined confinement.

To investigate further the energetics of the metastable configuration having
the disfavored handedness, we construct a parameterized director field model
for the hemispherical sessile droplet, which is inspired by the twisted
bipolar director configuration of the spherical droplet 10. The director
$\mathbf{n}$ with a parameter $\gamma$ is

$\displaystyle\mathbf{n}=\mathbf{n}_{\mathrm{bipolar}}\cos{\gamma}+\mathbf{n}_{\mathrm{concentric}}\sin{\gamma},$
(5)

where $\mathbf{n}_{\mathrm{bipolar}}$ and $\mathbf{n}_{\mathrm{concentric}}$
are the director field model of the bipolar and concentric configurations used
in Jeong et al10. We introduce the parameter $\gamma$ to describe the
azimuthal component of the director experiencing a twist deformation. If
$\gamma=0$, the director field becomes the bipolar configuration,
$\mathbf{n}=\mathbf{n}_{\mathrm{bipolar}}$, with no twist deformation. With
$\gamma\neq 0$, directors twist as shown in Fig. 4(a). Assuming
$\gamma=a(z/R)^{b}(\rho/R)^{c}$ as a power-law function of $z$ and $\rho$ with
parameters $a$, $b$, and $c$, we numerically calculate total elastic free
energy $F$ according to ($a$, $b$, $c$). It is noteworthy that $z^{b}$ makes
the model satisfy the homeotropic anchoring condition at the bottom substrate.
Moreover, the splay cancellation near the defect inspires the decaying along
$\rho$. The farther the director is away from the defect, the less twist
deformation is required to lower the energy.

We find that both achiral and chiral models can have two minima in the energy
landscape of the parameter space $(a,b,c)$. It is no surprise that the energy
landscape of the achiral degenerate system with $p_{0}=0$ renders the two
energy minima with the same energy, and these two configurations correspond to
each spiral texture of different handedness shown in the experiment. As shown
in Fig. 4(b), the Jones calculus-simulated POM textures approximately match
our experimental observation. However, the existence of the local minimum
configuration having the disfavored handedness, which is experimentally
observed, is unexpected. It is not straightforward to imagine why the
directors twist in an energetically unfavorable handedness and remain stable
enough to be observed experimentally, although the boundary conditions afford
homochiral configurations.

Our investigation of the energetics reveals that the elastic anisotropy is
responsible for the metastability in the chiral system, as in the chiral
symmetry breaking of the achiral system. First, the energy landscape assuming
the one-constant approximation, i.e., $K_{1}=K_{2}=K_{3}$, shows only one
minimum exhibiting the twisted director configuration of the favorable
handedness. However, with $K_{1}=K_{3}=10\leavevmode\nobreak\ K_{2}$, in
addition to the global minimum, there exists a local minimum having the
opposite handedness. To rationalize the existence of the local minimum, we
investigate the energy landscape in detail. As shown in Fig. S5, in the
vicinity of the local minimum, the increase in $b$ of
$\gamma=a(z/R)^{b}(\rho/R)^{c}$ results in the increase of total splay energy
within the droplet. But, both total twist and bend energy decrease, leading to
a local minimum at a finite $b$. This trade-off results from the fact that the
director configurations should satisfy the boundary conditions imposed by the
confinement. Namely, because of the constraints by the boundary conditions,
each deformation cannot change independently, but changes in a certain
deformation affect the other deformation modes. It is the elastic anisotropy
that leads to a local minimum in the trade-off situation. Notably, with
$K_{1}=K_{2}=K_{3}$ instead of $K_{1}=K_{3}=10\leavevmode\nobreak\ K_{2}$,
i.e., the decrease of total twist and bend energy upon the increase in $b$
dominates over the increase of total splay, giving no local minimum but a
monotonic decrease. In summary, a confinement-induced energy trade-off with
the small $K_{2}$, or large enough elastic anisotropy in general, makes this
local minimum reside even in the energy-landscape region of disfavored
handedness.

## Conclusions

We study the roles of confinement and elastic anisotropy in the chiral
symmetry breaking of achiral LCLCs and the metastability of chiral LCLCs. The
broad isotropic-nematic coexistence phase of LCLC enables us to study sessile
isotropic droplets immersed in an aligned nematic phase and vice versa. Our
director field models reproduce the spiral textures observed by POM and
elucidate how their handedness population depends on the chiral dopant
concentrations. Particularly, energy landscape calculations reveal that the
spirals of the disfavored handedness can exist as a local minimum. Regardless
of the boundary conditions, i.e., both in I-in-N and N-in-I cases, the large
elastic anisotropy $K_{2}\ll K_{1}=K_{3}$ is responsible for local minimum.

These findings help us design homochiral structures, primarily when they are
confined and possess large elastic anisotropy. The fabrication and control of
chiral structures are of great interest in various fields ranging from
conventional optical applications to state-of-the-art metamaterials.
Homochirality matters in those applications because racemic mixtures or
defects, i.e., the appearance of unwanted handedness, may negatively affect
applications’ performance. Using chiral building blocks of desired handedness
or imposing enantiomeric imbalance to achiral systems are typical in self-
assembly approaches. Our work with liquid crystals demonstrates that local
minima having undesirable handedness may appear, despite the intrinsic or
imposed handedness, because of boundary conditions and elastic anisotropy.
This finding indicates that energy landscape study should precede chirality
control, including material choice and design of confining geometries and
anchoring conditions.

## Conflicts of interest

There are no conflicts to declare.

## Acknowledgements

The authors gratefully acknowledge the financial support from the National
Research Foundation of Korea through NRF-2021R1A2C101116312.

## Notes and references

  * Green and Reidy 1989 M. Green and M. Reidy, _J. Am. Chem. Soc._ , 1989, 6452–6454.
  * Brunsveld _et al._ 2002 L. Brunsveld, J. A. Vekemans, J. H. Hirschberg, R. P. Sijbesma and E. W. Meijer, _Proceedings of the National Academy of Sciences of the United States of America_ , 2002, 99, 4977–4982.
  * Pakhomov _et al._ 2003 S. Pakhomov, R. P. Hammer, B. K. Mishra and B. N. Thomas, _Proceedings of the National Academy of Sciences of the United States of America_ , 2003, 100, 3040–3042.
  * Hough _et al._ 2009 L. E. Hough, H. T. Jung, D. Krüerke, M. S. Heberling, M. Nakata, C. D. Jones, D. Chen, D. R. Link, J. Zasadzinski, G. Heppke, J. P. Rabe, W. Stocker, E. Körblova, D. M. Walba, M. A. Glaser and N. A. Clark, _Science_ , 2009, 325, 456–460.
  * Kang _et al._ 2013 S. H. Kang, S. Shan, W. L. Noorduin, M. Khan, J. Aizenberg and K. Bertoldi, _Advanced Materials_ , 2013, 25, 3380–3385.
  * Smith 2009 S. W. Smith, _Toxicological Sciences_ , 2009, 110, 4–30.
  * Petit-Garrido _et al._ 2009 N. Petit-Garrido, J. Ignés-Mullol, J. Claret and F. Sagués, _Phys. Rev. Lett._ , 2009, 103, 237802.
  * Tortora and Lavrentovich 2011 L. Tortora and O. D. Lavrentovich, _Proceedings of the National Academy of Sciences_ , 2011, 108, 5163–5168.
  * Pairam _et al._ 2013 E. Pairam, J. Vallamkondu, V. Koning, B. C. Van Zuiden, P. W. Ellis, M. A. Bates, V. Vitelli and A. Fernandez-Nieves, _Proceedings of the National Academy of Sciences of the United States of America_ , 2013, 110, 9295–9300.
  * Jeong _et al._ 2014 J. Jeong, Z. S. Davidson, P. J. Collings, T. C. Lubensky and A. G. Yodh, _Proceedings of the National Academy of Sciences of the United States of America_ , 2014, 111, 1742–1747.
  * Kitzerow and Bahr 2001 _Chirality in Liquid Crystals_ , ed. H.-S. Kitzerow and C. Bahr, Springer-Verlag, New York, 2001.
  * Dierking 2014 I. Dierking, _Symmetry_ , 2014, 6, 444–472.
  * Salamończyk _et al._ 2019 M. Salamończyk, N. Vaupotič, D. Pociecha, R. Walker, J. M. D. Storey, C. T. Imrie, C. Wang, C. Zhu and E. Gorecka, _Nature Communications_ , 2019, 10, 1922.
  * Abberley _et al._ 2018 J. P. Abberley, R. Killah, R. Walker, J. M. D. Storey, C. T. Imrie, M. Salamończyk, C. Zhu, E. Gorecka and D. Pociecha, _Nature Communications_ , 2018, 9, 228.
  * Lewandowski _et al._ 2020 W. Lewandowski, N. Vaupotič, D. Pociecha, E. Górecka and L. M. Liz-Marzán, _Advanced Materials_ , 2020, 32, 1905591.
  * Wang _et al._ 2020 L. Wang, A. M. Urbas and Q. Li, _Advanced Materials_ , 2020, 32, 1801335.
  * Majewska _et al._ 2022 M. M. Majewska, E. Forsyth, D. Pociecha, C. Wang, J. M. D. Storey, C. T. Imrie and E. Gorecka, _Chemical Communications_ , 2022, 58, 5285–5288.
  * Huang _et al._ 2006 Y. Huang, Y. Zhou, C. Doyle and S.-T. Wu, _Optics Express_ , 2006, 14, 1236.
  * Sharma _et al._ 2009 V. Sharma, M. Crne, J. O. Park and M. Srinivasarao, _Science_ , 2009, 325, 449–451.
  * Han _et al._ 2010 Y. Han, K. Pacheco, C. W. Bastiaansen, D. J. Broer and R. P. Sijbesma, _Journal of the American Chemical Society_ , 2010, 132, 2961–2967.
  * Coles and Morris 2010 H. Coles and S. Morris, _Nature Photonics_ , 2010, 4, 676–685.
  * Beeckman 2011 J. Beeckman, _Optical Engineering_ , 2011, 50, 081202.
  * Mitov 2017 M. Mitov, _Soft Matter_ , 2017, 13, 4176–4209.
  * Hare _et al._ 2020 S. M. Hare, B. Lunsford-Poe, M. Kim and F. Serra, _Materials_ , 2020, 13, 3761.
  * Press and Arrott 1974 M. J. Press and A. S. Arrott, _Physical Review Letters_ , 1974, 33, 403–406.
  * Kitzerow _et al._ 1996 H.-S. Kitzerow, B. Liu, F. Xu and P. P. Crooker, _Physical Review E_ , 1996, 54, 568–575.
  * Ambrožič and Žumer 1996 M. Ambrožič and S. Žumer, _Physical Review E_ , 1996, 54, 5187–5197.
  * Ambrožič and Žumer 1999 M. Ambrožič and S. Žumer, _Physical Review E_ , 1999, 59, 4153–4160.
  * Nych _et al._ 2014 A. Nych, U. Ognysta, I. Muševič, D. Seč, M. Ravnik and S. Žumer, _Physical Review E_ , 2014, 89, 062502.
  * Davidson _et al._ 2015 Z. S. Davidson, L. Kang, J. Jeong, T. Still, P. J. Collings, T. C. Lubensky and A. G. Yodh, _Physical Review E - Statistical, Nonlinear, and Soft Matter Physics_ , 2015, 91, 1–5.
  * Nayani _et al._ 2015 K. Nayani, R. Chang, J. Fu, P. W. Ellis, A. Fernandez-Nieves, J. O. Park and M. Srinivasarao, _Nature Communications_ , 2015, 6, 1–7.
  * Jeong _et al._ 2015 J. Jeong, L. Kang, Z. S. Davidson, P. J. Collings, T. C. Lubensky and A. G. Yodh, _Proceedings of the National Academy of Sciences of the United States of America_ , 2015, 112, E1837–E1884.
  * Peng and Lavrentovich 2015 C. Peng and O. D. Lavrentovich, _Soft Matter_ , 2015, 11, 7257–7263.
  * Guo _et al._ 2016 Y. Guo, S. Afghah, J. Xiang, O. D. Lavrentovich, R. L. Selinger and Q. H. Wei, _Soft Matter_ , 2016, 12, 6312–6320.
  * Ryu and Yoon 2016 S. H. Ryu and D. K. Yoon, _Liquid Crystals_ , 2016, 43, 1951–1972.
  * Zhou _et al._ 2016 Y. Zhou, E. Bukusoglu, J. A. Martínez-González, M. Rahimi, T. F. Roberts, R. Zhang, X. Wang, N. L. Abbott and J. J. De Pablo, _ACS Nano_ , 2016, 10, 6484–6490.
  * Serra 2016 F. Serra, _Liquid Crystals_ , 2016, 43, 1920–1936.
  * Javadi _et al._ 2018 A. Javadi, J. Eun and J. Jeong, _Soft Matter_ , 2018, 14, 9005–9011.
  * Ellis _et al._ 2018 P. W. Ellis, K. Nayani, J. P. McInerney, D. Z. Rocklin, J. O. Park, M. Srinivasarao, E. A. Matsumoto and A. Fernandez-Nieves, _Physical Review Letters_ , 2018, 121, 247803.
  * Lee _et al._ 2019 H. Lee, V. Sunkara, Y. K. Cho and J. Jeong, _Soft Matter_ , 2019, 15, 6127–6133.
  * Palacio-Betancur _et al._ 2023 V. Palacio-Betancur, J. C. Armas-Pérez, J. P. Hernández-Ortiz and J. J. de Pablo, _Soft Matter_ , 2023, 19, 6066–6073.
  * Smalyukh and Lavrentovich 2002 I. I. Smalyukh and O. D. Lavrentovich, _Phys. Rev. E_ , 2002, 66, 051703.
  * Eun _et al._ 2021 J. Eun, J. Pollard, S.-J. Kim, T. Machon and J. Jeong, _Proceedings of the National Academy of Sciences_ , 2021, 118, e2102926118.
  * Seč _et al._ 2012 D. Seč, T. Porenta, M. Ravnik and S. Žumer, _Soft Matter_ , 2012, 8, 11982–11988.
  * Yoshioka _et al._ 2016 J. Yoshioka, F. Ito and Y. Tabe, _Soft Matter_ , 2016, 12, 2400–2407.
  * Li _et al._ 2023 S. Li, X. Zhou, J. Zhu, K. Du, Y. Du and H. Gao, _Physical Review E_ , 2023, 107, 1–7.
  * Horowitz _et al._ 2005 V. R. Horowitz, L. A. Janowitz, A. L. Modic, P. A. Heiney and P. J. Collings, _Physical Review E - Statistical, Nonlinear, and Soft Matter Physics_ , 2005, 72, 1–10.
  * Jeong _et al._ 2014 J. Jeong, G. Han, A. T. Johnson, P. J. Collings, T. C. Lubensky and A. G. Yodh, _Langmuir_ , 2014, 30, 2914–2920.
  * Lee _et al._ 2011 G. Y. Lee, H. Ko, H. W. Jung, S. J. Ye, M. H. Kim and J. C. Pyun, _Procedia Engineering_ , 2011, 25, 321–324.
  * Zhou _et al._ 2012 S. Zhou, Y. A. Nastishin, M. M. Omelchenko, L. Tortora, V. G. Nazarenko, O. P. Boiko, T. Ostapenko, T. Hu, C. C. Almasan, S. N. Sprunt, J. T. Gleeson and O. D. Lavrentovich, _Physical Review Letters_ , 2012, 109, 1–5.
  * Lee and Labes 1982 H. Lee and M. M. Labes, _Molecular Crystals and Liquid Crystals_ , 1982, 84, 137–157.
  * McGinn _et al._ 2013 C. K. McGinn, L. I. Laderman, N. Zimmermann, H. S. Kitzerow and P. J. Collings, _Physical Review E - Statistical, Nonlinear, and Soft Matter Physics_ , 2013, 88, 1–9.
  * Ogolla _et al._ 2017 T. Ogolla, S. B. Nashed and P. J. Collings, _Liquid Crystals_ , 2017, 44, 1968–1978.
  * Shirai _et al._ 2018 T. Shirai, M. Shuai, K. Nakamura, A. Yamaguchi, Y. Naka, T. Sasaki, N. A. Clark and K. V. Le, _Soft Matter_ , 2018, 14, 1511–1516.
  * Ogolla _et al._ 2019 T. Ogolla, R. S. Paley and P. J. Collings, _Soft Matter_ , 2019, 15, 109–115.
  * Uzunova and Pergamenshchik 2011 V. A. Uzunova and V. M. Pergamenshchik, _Physical Review E_ , 2011, 84, 031702.
  * Eun _et al._ 2019 J. Eun, S. J. Kim and J. Jeong, _Physical Review E_ , 2019, 100, 12702.


# Towards Neural Programming Interfaces

Zachary C. Brown
Electrical and Computer Engineering
Duke University
Durham, NC 27708
<EMAIL_ADDRESS>
Nathaniel Robinson
Department of Computer Science
Brigham Young University
Provo, UT 84602
<EMAIL_ADDRESS>
David Wingate
Department of Computer Science
Brigham Young University
Provo, UT 84602
<EMAIL_ADDRESS>
Nancy Fulda
Department of Computer Science
Brigham Young University
Provo, UT 84602
<EMAIL_ADDRESS>

###### Abstract

It is notoriously difficult to control the behavior of artificial neural
networks such as generative neural language models. We recast the problem of
controlling natural language generation as that of learning to interface with
a pretrained language model, just as Application Programming Interfaces (APIs)
control the behavior of programs by altering hyperparameters. In this new
paradigm, a specialized neural network (called a Neural Programming Interface
or NPI) learns to interface with a pretrained language model by manipulating
the hidden activations of the pretrained model to produce desired outputs.
Importantly, no permanent changes are made to the weights of the original
model, allowing us to re-purpose pretrained models for new tasks without
overwriting any aspect of the language model. We also contribute a new data
set construction algorithm and GAN-inspired loss function that allows us to
train NPI models to control outputs of autoregressive transformers. In
experiments against other state-of-the-art approaches, we demonstrate the
efficacy of our methods using OpenAI’s GPT-2 model, successfully controlling
noun selection, topic aversion, offensive speech filtering, and other aspects
of language while largely maintaining the controlled model’s fluency under
deterministic settings.

##  1 Introduction

Researchers’ ability to automate natural language processing has grown
exponentially over the past few years, particularly with the advent of the
Transformer architecture [1]. Despite the fact that recent machine learning
methods achieve impressive and almost human-level performance on tasks such as
dialogue modeling [2, 3, 4] and natural language generation [5, 6, 7], many
intelligent voice assistants still rely on rule-based architectures and cached
responses in open domain dialogue [8, 9, 10]. This is primarily due to the
lack of controls in deep learning architectures for producing specific
phrases, tones, or topics, which makes these models inherently unpredictable
and therefore too risky for most entities - corporate or otherwise - who wish
to deploy public-facing intelligent agents. For example, it is often desirable
for a conversational agent to maintain a specific identity (comprised of a
name, set of opinions, etc.) throughout an exchange of dialogue and it is
currently impossible to condition deep learning algorithms to maintain a
coherent identity across dialogue without training them on highly specialized
(and, by extension, biased and limited) data sets. Fine-tuning on these
specialized data sets comes with an additional, significant cost: it can lead
to catastrophic forgetting of the language model [11]. Despite this aspect of
fine-tuning, current state-of-the-art methods [12, 2] require fine-tuning [13]
of the entire network when their original data set proves unsuitable for a
given task (such as discussing a recent development in the news), even if the
language being modeled is the same across tasks. Furthermore, models produced
by current methods are almost entirely uninterpretable and therefore generally
difficult to test for egregious failure cases.

In this paper111Code available at https://github.com/DRAGNLabs/towards-neural-
programming-interfaces, we address both the issue of content control as well
as that of catastrophic forgetting induced by fine-tuning. We define ‘content
control’ as being able to command a network to either incorporate or eschew an
exact word, phrase, topic, style, or sentiment in its output, and therefore
attempt a more granular level of control than the purely topic/style-level
control that has been published in recent literature [12, 2]. We also
introduce an alternative to fine-tuning neural language models and demonstrate
through experimentation that the high-cost of overwriting model weights
through fine-tuning (and thereby reducing model generalizeability in favor of
a specific sub-task) often fails to induce the desired behavior in generalized
settings. Instead, we recast the problem of control in natural language
generation as one of combining separate models - one of the natural language
itself and one of high-level command responses - to produce desired linguistic
output. In doing so, we develop a framework for interpreting and subsequently
controlling the hidden activations of a pretrained neural network without any
adjustments being made to the pretrained model. This framework is biologically
consistent with the findings of Knutson et al., who discovered that neural
pathways in humans are inhibited by other neuron clusters [14], and has
applications to other neural network architectures and questions outside the
domain of controllable text generation.

##  2 Related Work

Recently, several publications in controllable natural language generation
have reported impressive results. Particularly interesting is the approach
taken by Plug and Play Language Models (PPLM) [15], which bears significant
resemblance to our own despite the two approaches having been developed
completely independently (we were only made aware of the PPLM method during
the review process for this paper). Our work differs from PPLM in multiple
ways, foremost among them being our distinct uses of neural networks to
produce perturbations in the pretrained model. Rather than using summed
discriminator gradients to produce perturbations as with PPLM’s attribute
models, we train a neural network (the NPI network) to produce perturbations
adversarially against a discriminator; this makes our approach compatible with
arbitrary discriminator algorithms, including any which may not provide
gradients to the perturbation model (i.e. the NPI) during training (see
Section 3.1.2). Our novel approach also enables us to avoid adopting a greedy
approach to textual control (see the beginning of Section 3.1), whereas the
PPLM paper is focused primarily on greedy control. The differences in our
methodology carry over to our data sets, where our novel data curation
approach (see Section 3.1.1) does not require pre-labeled text data
exemplifying target behavior and which could be generalized to non-textual
inputs (such as random samples from a Gaussian distribution). In Section 4 we
present experimental results comparing the NPI and PPLM approaches which
demonstrate the relative strengths of both algorithms.

The work regarding the CTRL [12], Meena [2], and GPT-3 [16] neural networks
also demonstrates significant progress in controllable natural language
generation, however, these projects represent single neural networks that
model both natural language as well as input commands simultaneously. Our work
differs from CTRL [12] and Meena [2] in that we seek to (a) achieve content
control and (b) separate the language model from the control model to avoid
fine-tuning the language model. The recently published GPT-3 [16] language
model is capable of learning new linguistic modeling tasks without any fine-
tuning, but, as the authors state in their paper, it is a predictive model and
therefore not ideal for goal-oriented text generation.

While there has been work which suggests neural architectures can learn to
multi-task [17, 18], multi-tasking networks are typically trained to model
various highly-correlated tasks (our own approach models a sequence of
controls in parallel, similar to work by Feng Jin & Shiliang Sun [18]). We
argue that modeling natural language and responding to high-level commands are
two fundamentally different tasks, and that a neural network can therefore not
optimally model both tasks, as suggested by the “No Free Lunch" (NFL) theorems
by Wolpert & Macready [19]. Rather than model these tasks jointly, our
proposed method of targeted text generation makes no permanent changes to a
pretrained language model - which is assumed to be an (approximately) optimal
model of natural language - and instead leverages an additional neural network
(an NPI) trained on a separate objective to manipulate the inner workings of
the pretrained language model in response to high-level commands.

We focus our experiments on controlling the output of the autoregressive GPT-2
language model [6], though other autoregressive transformers such as
Transformer-XL [5], XLNet [7], and GPT-3 [16] could theoretically be
controlled in a similar manner, and we propose further research be done into
controlling these models. We use several variations on vanilla feed-forward
neural network architectures [20] in various capacities, as well as an
adversarial GAN-style setup [21], for training our NPI network. The NPI loss
function was inspired by the Style-Transfer loss function introduced by Gatys,
Ecker, & Bethge [22]. Much of our code was adapted from the Hugging Face
Transformers GitHub repository [23] and is available online (see Footnote 1).

Our methodology, which seeks to alter the functionality of pretrained
networks, has significant crossover with adversarial attacks on neural
networks [24]. Our work differs from that of adversarial perturbations of
seq-2-seq models [25] in that we formulate our Style-Transfer loss function
such that it encourages the perturbed outputs to be as close to the original
outputs as possible while still introducing the desired content, whereas
related work does not incorporate this condition [25].

##  3 Neural Programming Interfaces (NPIs)

We define an NPI network $X:H_{in}\rightarrow D_{out}$ as a mapping from a
subset of hidden layer activations $H_{in}$ of a pre-trained model $P$ to a
set of activation perturbation values $D_{out}$ which can be element-wise
added to hidden states of the pretrained model during a forward pass to
produce desired output.

Given an input $T_{in}$ and a pretrained model
$P:T_{in}\rightarrow\\{T_{out},H=\\{h_{i}\\}^{n}_{i=1}\\}$ which produces
output $T_{out}$ and $n$ hidden layer activations $H$, we first collect $m\leq
n$ hidden layer activations from $H$, yielding $H_{in}\subseteq H$; we index
these by $I_{in}=\\{i|h_{i}\in H_{in}\\}$ and refer to them as ‘input
controls’. We then generate $m$ corresponding control perturbations

$X(H_{in})=D_{out}=\\{d_{l,i}\\}^{m}_{l=1,i\in I_{in}}$ (1)

where $1\leq l\leq m$ is the $l$-th index into the set of control
perturbations $D_{out}$ and $i\in I_{in}$ is the index of the hidden layer
activation where the control perturbation $d_{l,i}$ will be added (element-
wise) during a second forward pass of the network $P$, producing the perturbed
(i.e. controlled) hidden layer activation $c_{l,i}$. Once $D_{out}$ is
generated, we generate controlled output and hidden layer activations

$P^{\prime}(T_{in})=P(T_{in}|D_{out})=\\{T^{D}_{out},H^{\prime}\\}$ (2)

where $P^{\prime}$ represents the pretrained network with $d_{l,i}\in D_{out}$
added to the appropriate hidden layer activations. In summary, the NPI network
$X$ learns to adjust the hidden layer activations $H_{in}$ using perturbations
$D_{out}$ such that they produce a desirable output $T^{D}_{out}$ \- please
see Algorithm 1 of Appendix 1 as well as Figure 1 of this paper for further
illustration.

Figure 1: Our approach to controlling a fully aggregated pretrained neural
network $P$ (See Section 3). The influence of the Neural Programming Interface
(NPI) in the controlled version of $P$ is represented by the presence of boxes
and lines not found in the original version of $P$. Input $T_{in}$ is passed
through $P$, generating $n$ hidden layer activations $H=\\{h_{i}\\}^{n}_{i=1}$
and output $T_{out}$ (left). Given $m\leq n$ input controls $H_{in}$, the NPI
generates $m$ control perturbations $D_{out}=\\{d_{l,i}\\}^{m}_{l=1,i\in
I_{in}}$ (middle) which are added to the hidden layer activations whose
indices are in $I_{in}$ when $T_{in}$ is again passed through $P$ (right).
Changes made by each $d_{l,i}$ propagate down-stream through the network via
the nature of element-wise addition $\oplus$ and aggregation mechanisms $A$,
resulting in altered hidden layer activations for all subsequent layers
$c_{l,i+q}\in H^{\prime}$ where $1\leq q\leq n-i$ (right). The newly generated
output $T^{D}_{out}$ represents $T_{out}$ conditioned on $D_{out}$.

###  3.1 NPIs for Controllable Natural Language Generation

We seek to control the output of OpenAI’s GPT-2 [6] while still making full
use of the GPT-2’s internal models of language, which produce a single token
$T_{out}$ for each input $T_{in}$, with $T_{out}$ encoding for anything from
an entire word to white-space characters. Without changing the weights of the
GPT-2, we aim to use an NPI to guide its outputs either toward or away from
the use of targeted content.

Though it seems natural to control the GPT-2 at every forward pass of the
network - and there are existing methods which achieve an impressive level of
success at doing this [15] \- this approach makes it difficult to perform
topical control (and even word-level control for multi-token words). This is
due partly to the difficulties of classifying a single token as belonging to a
particular multi-token word, let alone to a conversation topic. Since part of
our goal involves skewing the GPT-2 output towards the use of specific
phrases, exercising control on every forward pass could result in the GPT-2
repeating only those tokens as a greedy solution. In our approach we
simultaneously control entire sequences $S=(H_{in,x})^{w}_{x=1}$ (where $w$
denotes the length of the sequence context window) of GPT-2 hidden layer
control sequences $H_{in,x}$ via corresponding sequences of perturbations
$D_{out}$ in order to provide context and give full expression to GPT-2’s
linguistic model.

To accomplish this, we produced multiple data sets of sequences of GPT-2
hidden layer activations associated with sequences of output tokens; the NPI
was then trained to process a concatenation $S$ of multiple sequences
$H_{in,x}$ of input controls simultaneously and produce an associated sequence
of control perturbations for iterative forward passes of the GPT-2 network
(see Figure 2 for an illustration). Note this is a modified application of the
NPI structure described previously. Note also that since the NPI perturbations
in early forward passes may cause drastic changes in the hidden states in
later forward passes, this requires the NPI to learn how to anticipate in some
capacity what tokens the GPT-2 will begin to produce after initial
perturbations.

####  3.1.1 Data Set Generation

We generated our data sets by performing hundreds of thousands of GPT-2
forward passes using input text extracted from a Wikipedia corpus [26], Reddit
corpus [27], and Toronto BookCorpus [28]. Sample text was fed into the GPT-2
model, which was permitted to generate tokens for a maximum number of
iterations. We allowed GPT-2 to produce tokens $T_{out,j}$ in a fixed window
size $w$ around the instance of the target quality (e.g. the appearance of a
target word in the output text). Because converting language model activations
into text is non-differentiable, we constructed data sets that consisted of
GPT-2 activation patterns that proved to produce the target text behavior,
rather than merely the sampled text itself. We collected hidden states from
$m=2$ of the GPT-2 layers for each forward pass (each pass corresponding to an
output token in the context window $T_{out,j}$), for a total of $m*w=2w$ total
hidden activations per data point. We also collected the text tokens used as
input $T_{in,j}$ for the first of the $w$ forward passes that produced the
output text, as well as the assigned text quality label $L_{j}\in\\{0,1\\}$
(where $1$ corresponds to the target behavior), which was assigned using
simple metrics - such as word presence or absence - operating on $T_{out,j}$.

More formally, each data set had the final form
$Q=\\{q_{j}\\}^{N_{Q}}_{j=1}=\\{(S_{j},L_{j},T_{in,j})\\}^{N_{Q}}_{j=1}$ where
$S_{j}=(H_{in,x,j})^{w}_{x=1}$ is a sequence of subsets of GPT-2 input control
layers $H_{in,x,j}$, each of length $m$, $L_{j}\in\\{0,1\\}$ is the assigned
label, and $T_{in,j}$ is the input text tokens corresponding to $q_{j}$.
222The size of GPT-2 hidden layers prevented us from posting data sets online,
though the code used to produce the data sets can be found at
https://github.com/DRAGNLabs/towards-neural-programming-interfaces; contact
Nate Robinson at<EMAIL_ADDRESS>for access to the data sets used in this
paper.

Often a target text quality is either common or rare. To maintain a balanced
data set, our data generation software rejects many examples of the more
common class. We found that randomly inserting tokens displaying the less
common text quality into the input text from our corpus made GPT-2 more likely
to output text of the rarer class and expedited data generation. Examples of
data sets we generated are described further in Section 4 and in our
supplementary materials.

A potential limitation of this data curation technique is that we relied on
stringent $top\\_k=1$ filtering in order to ensure the output $T_{out,j}$ and
associated labels $L_{j}$ were closely correlated to the content in the
sequences $S_{j}$; more research is required to determine whether or not this
type of deterministic filtering is required for NPIs to attain satisfactory
levels of control over pretrained language models. If this assumption can
indeed be relaxed, we may see further improvements in the fluency of
controlled outputs than those shown in our experiments in Section 4, which use
a baseline GPT-2 model that has the same filter settings as those used to
generate our NPI data sets. Note also that our approach to data curation
assumes the language model to be at least capable of producing the target text
quality in the first place, though the desired quality can be rarely
expressed; eliminating this assumption marks an area of further research we
find very intriguing.

####  3.1.2 Loss Function and Training Process

In our linguistic control experiments, we defined NPI models mapping
$X:S_{j}\rightarrow D_{out,j}$ from a sequence $S_{j}$ of subsets of hidden
layer activations to a sequence of sets of activation perturbation values
$D_{out,j}$ which can be element-wise added into the pretrained model during a
series of forward passes to produce output. As noted earlier, this is a
modification to the original definition of NPI at the start of Section 3, as
we are now passing it entire sequences $S_{j}$ of sets of hidden layers
corresponding to multiple language model forward passes that produced a text
context window.

We also define $S^{\prime}_{j}=\\{c_{a,b}|b\in
I_{in},c_{a,b}\in(H^{\prime}_{x,j})^{w}_{x=1},\\{T^{D}_{out,x,j},H^{\prime}_{x,j}\\}=P(T_{in,x,j}|D_{out,j})\\}$
(where $T_{in,x,j}$ and $T^{D}_{out,x,j}$ are the input and output of P in the
$x$th of a series of forward passes) as the derived control activations of
$S_{j}$, which are to be directly compared to $S_{j}$ in the loss function for
$X$ which we describe below. Let $L^{\prime}_{j}$ denote the label of
$S^{\prime}_{j}$ generated using the same metric-based rules operating on
$T^{D}_{out,j}$ as were used with $T_{out,j}$ to produce the label $L_{j}$ for
$S_{j}$.

NPI training involved three neural models trained together in an adversarial
framework [21]. The NPI model $X:S_{j}\rightarrow D_{out,j}$ received feedback
from a pre-trained content classifier network $Y:S_{j}\rightarrow[0,1]$ as
well as a discriminator network $Z:S_{j}\rightarrow[0,1]$.

The discriminator $Z$ was trained to distinguish between original GPT-2 hidden
layer activation sequences (i.e. $S_{j}$) and those perturbed by $X$ (i.e.
$S^{\prime}_{j}$) to ensure that the NPI did not perturb GPT-2 activations
outside the space of plausible activations. $Z$ was trained in tandem with $X$
in an adversarial manner, with regular updates made to the weights.

Before training $X$ and $Z$, we trained $Y$ \- a neural classification model -
to distinguish between original GPT-2 hidden activations $S_{j}$ that
exhibited target behavior and those that did not, per $L_{j}$. (Thus $Y$ could
be trained with the same data set as $X$ and $Z$.) When $Y$ was able to
generalize well enough to correctly label activation sequences outside of the
training data, we began training $X$ and $Z$. We found it helpful in some
experiments to continually update the weights of $Y$ with real-time generated
labels from the output text of derived control activations $S^{\prime}_{j}$
produced by $X$. This helped ensure that $X$ could not exploit weaknesses in
$Y$ as $Y$ learned to counteract such exploitations.

The content classifier $Y$ was pretrained to minimize
$E_{Y}=\mathrm{BCE}(Y(S_{j}),L_{j})$ and then later to minimize
$E^{\prime}_{Y}=\mathrm{BCE}(Y(S^{\prime}_{j}),L^{\prime}_{j})$ in adversarial
settings. The discriminator network $Z$ was trained to minimize
$E_{Z}=\mathrm{BCE}(Z(S_{j}),0)+\mathrm{BCE}(Z(S^{\prime}_{j}),1)$. The NPI
loss $E_{X}$, inspired by the Style-Transfer loss function [22], was composed
of three sub-components - fluency loss $E_{X,f}$, content loss $E_{X,c}$, and
stability loss $E_{X,s}$ \- as shown below:

$E_{X}=\gamma E_{X,f}+\alpha E_{X,c}+\beta E_{X,s}$ (3)
$E_{X,f}=\mathrm{BCE}(Z(S^{\prime}_{j}),0)$ (4)
$E_{X,c}=\mathrm{BCE}(Y(S^{\prime}_{j}),\ell_{target})$ (5)
$E_{X,s}=\mathrm{MSE}(S^{\prime}_{j},S_{j})$ (6)

where $\ell_{target}=1$ when the target quality is to be induced and
$\ell_{target}=0$ when it is to be avoided; $\gamma$, $\alpha$, and $\beta$
are scalar hyperparameters; and BCE and MSE denote the Binary Cross-Entropy
and Mean-Squared Error loss functions respectively. Minimizing the content
loss $E_{X,c}$ (the signal from $Y$) encouraged the NPI to control the
language model such that it output text with the desired target quality. As
described earlier, minimizing the fluency loss $E_{X,f}$ (the signal from $Z$)
encouraged the derived control activations $S^{\prime}_{j}$ (produced using
the NPI outputs) to resemble realistic language model activations. Minimizing
stability loss $E_{X,s}$ encouraged output text influenced by the NPI not to
stray too far (in ways other than the content being controlled) from the
original output text.

Please see Figure 2 for a summary of this section.

Figure 2: Our approach to controlling OpenAI’s GPT-2 (see Section 3.1). In
Step 1, we pass uncurated text through the GPT-2 model and iteratively produce
$w$ tokens $T_{out,j}$. In Step 2, we extract the hidden layer activations
(indexed by $I_{in}$) produced during each forward pass and store them as a
sequence $S_{j}$. In Step 3, the output tokens $T_{out,j}$ are analyzed (via
some chosen metric) to determine a label $L_{j}$ for the behavior of the
sequence $S_{j}$. Step 4 iteratively produces the data set $Q$ by repeating
Steps 1-3. Step 5 illustrates how $Q$ is used to train the NPI $X$
adversarially against $Y$ and $Z$.

##  4 Results

We demonstrate the effectiveness of the NPI framework on a series of tasks
including fine-grained topic control, offensive language filtering and bias
reduction, and thematic adaptations. Our results show that NPI-guided text
generation is able to increase the frequency of desired text properties
without causing any detectable fluency degradation after controlling for top_k
and top_p filtering. See Sections 5-6 of Appendix 1 for further details as to
experiments and hyperparameters, as well as a detailed description of the
‘word prob’ baseline which either induced or eliminated target content during
the logit sampling step of GPT-2 generation.

Throughout this section, the NPIs’ performances were evaluated both by
counting the number of times the perturbed GPT-2’s output produced the target
content and by measuring the vector shift of the GPT-2 output’s vector
representation toward or away from the embedded representation of the
target.333For the embedding comparisons, we used 300-dimensional GloVe
embeddings trained on an 840B word corpus [29]. Sentences were represented as
a bag-of-words average of their component word embeddings. Target in output
refers to the percentage of GPT-2 outputs containing at least one instance of
the target. Embed shifts refers to the frequency with which embedded
representations of the NPI-manipulated outputs were closer to the target
embedding than original GPT-2 outputs. Avg shift refers to the mean distance
between NPI-manipulated embeddings and the original GPT-2 embeddings. Fluency
was evaluated using a crowd-sourced Likert scale. Mechanical Turk workers with
a “master” qualification allotted 1 to 5 stars for text quality. The best
results in each table are bolded.

Although our current experiments focus on text generation using a small-scale
GPT-2 model, we believe that the NPI framework is equally applicable to other
domains and language models, including GPT3 [16], MEENA [30], GeDi [31], and
image and sound generation tasks [32, 33, 34, 35]. Please see Appendix 1 for
further discussion on applying the NPI approach to other models and domains.

###  4.1 Fine-grained Topic Control While Maintaining Fluency

Table 1: Fine-grained topic control across 1000 GPT-2 input sentences (500 for word prob baseline). The best results in each column are shown in bold-face text. model name | target in output | embed shifts | avg shift
---|---|---|---
cat-NPI low-resource | 33.3% | 87.6% | 0.092
cat-NPI | 47.6% | 94.8% | 0.125
fine-tuned GPT-2 baseline | 0.20% | 64.5% | 0.015
word prob baseline | 0.00% | 0.00% | 0.000
unmodified GPT-2 | 0.00% | N/A | N/A

To demonstrate topic induction, we first trained multiple NPI models to elicit
the word ‘cat’ as frequently as possible, while still remaining coherent given
the original path of discourse. The word ‘cat’ seemed an appropriate test bed,
because it is a relatively common word in the English language, and yet is not
spontaneously generated by the GPT-2 model with any notable frequency. We
compared the performance of two NPI models: a low-resource model trained on
only 691 training instances, and another model trained on 70,227 text samples.
A fine-tuned GPT-2 baseline [36] was trained on 50,209 sentences containing
the word ‘cat’, with a final loss of 0.05, at which point it was observed that
the loss had only dropped by 0.02 in over 1000 steps. We also compared against
a word probability baseline which automatically selected the target word
whenever its likelihood exceeded 0.

Results are shown in Table 1. The effectiveness of training NPI models on
small data sets (See Figures 1-4 in Appendix 1) suggest that NPIs may be a
viable technique for controlled text generation in low-resource settings,
where there are only a small set of labeled examples of the desired behavior.
See Appendix 1 for further discussion and text samples.

To test the NPI’s ability to aggressively avoid an undesired topic, we created
evaluation data sets consisting of 1000 GPT-2 input contexts that were
confirmed to produce the undesired trait in GPT-2 output text with high
likelihood. (This was necessary for comparison, as many words, such as ‘cat’,
may not appear at all in 1000 random GPT-2 outputs.) Note to train NPI on the
avoidance task we simply switch the objective in Equation (5). We tested
multiple NPIs on topic aversion, notably a cat-avoidance NPI for avoiding the
word ‘cat’, and a political-avoidance NPI designed to avoid producing the name
of a controversial public figure. A fine-tuned GPT-2 baseline did not make
sense for this task, as it is difficult to fine-tune a language model away
from producing a specific word. Results are shown in Table 2.

Table 2: Fine-grained target avoidance across 1000 GPT-2 input sentences that produced the target word. The objective was to pivot away from the target word and settle on a different but related topic. model name | ‘cat’ in output | name in output | embed shifts | avg shift
---|---|---|---|---
cat-avoidance-NPI | 11.0% | - | 47.6% | 0.008
public-figure-avoidance-NPI | - | 52.4% | 70.8% | 0.025
unmodified GPT-2 | 37.1% | 76.2% | - | -

Due to limited computational capacity, our experiments were performed using a
small GPT-2 model and a short context length of $w\in\\{10,15\\}$ characters.
This naturally results in compromised fluency as compared to using larger
models and contexts; however, as shown in Table 3 and other experiments
(including some in Appendix 1), human evaluators rated NPI-guided outputs on
par with unmodified GPT-2 outputs when using deterministic filtering. Our
method also attains fluency ratings that either match or exceed the fluency of
outputs generated using the Plug and Play Language Model (PPLM) [15]. These
results demonstrate that NPIs can be used without significantly impacting
fluency.

Table 3: Model comparisons across 500 utterances. PPLM (Discriminator) used the same hyperparamters as NPI: top-1 filtering, small-size GPT-2. NPI clearly outperformed on word induction. Word prob outperformed in word avoidance. See Appendix Table 2 for a discussion of fluency.  | target in | embed | avg | fluency | fluency
---|---|---|---|---|---
| output | shifts | shift | Likert scale | std dev
word induction - “cat” |  |  |  |  |
(random contexts from Wikipedia) |  |  |  |  |
NPI | 48.8% | 95.4% | 0.126 | 3.392 | 1.027
PPLM | 23.2% | 44.0% | 0.059 | 3.632 | 1.116
word prob baseline | 0.00% | 0.00% | 0.000 | 4.136 | 0.799
unmodified GPT-2 | 0.00% | N/A | N/A | 3.452 | 0.994
word avoidance - “cat” |  |  |  |  |
(contexts containing “cat”) |  |  |  |  |
NPI | 11.2% | 47.2% | 0.009 | 3.614 | 1.076
PPLM | 10.0% | 78.6% | 0.143 | 2.808 | 1.325
word prob baseline | 0.60% | 33.2% | 0.017 | 4.010 | 1.100
unmodified GPT-2 | 38.8% | N/A | N/A | 3.604 | 1.099
offense avoidance |  |  |  |  |
(contexts containing offensive terms) |  |  |  |  |
NPI | 17.6% | 56.4% | 0.067 | 2.944 | 0.752
PPLM | 17.0% | 33.8% | 0.119 | 2.394 | 1.265
word prob baseline | 16.6% | 18.9% | 0.0006 | 3.450 | 1.130
unmodified GPT-2 | 28.4% | N/A | N/A | 2.912 | 0.767

###  4.2 Bias Mitigation and Offensive Language Filtering

Table 4: Topic aversion across multiple forms of offensive speech. Successful shifts refers to the number of times that embedded representations of the NPI-manipulated outputs were closer to the target word embedding than they were to the original GPT-2 outputs. n denotes the number of sample GPT-2 input texts used for the evaluation. Each experiment is paired with comparative results from an unmodified GPT-2 model using an identical set of input texts. model name | target in output | successful shifts | avg shift | n
---|---|---|---|---
Public figure avoidance | 54.2% | 70.8% | 0.025 | 500
unmodified GPT-2 | 76.2% | N/A | N/A | 500
Racial slur avoidance | 0.5% | 73.57% | 0.089 | 140
unmodified GPT-2 | 52.1% | N/A | N/A | 140
Gender slur avoidance | 10.3% | 63.6% | 0.073 | 1000
unmodified GPT-2 | 90.2% | N/A | N/A | 1000
offensive speech avoidance | 58.0% | 57.0% | 0.046 | 500
unmodified GPT-2 | 88.4% | N/A | N/A | 500

An immediate and much needed use case for the ability of an NPI to control for
entire classes of behavior in addition to inducing or avoiding specific words
is offensive language filtering. The inherent bias of large language models is
well known and a topic of intense study [37, 38, 39, 40, 41]. While it may not
be possible at this time to remove the bias from the weights of a language
model that was trained on faulty data, it may at least be possible to inhibit
inappropriate expressions of that bias.

To test this possibility, we evaluated the NPI’s ability to reduce the
incidence of multiple forms of offensive speech, ranging from specific to
general (Table 4). Public figure avoidance describes a task where the NPI
avoided producing the name of a prominent and controversial politician (see
Section 4.1). In the Racial slur avoidance and Gender slur avoidance tasks,
the NPI learned to avoid using a small list (of size 2 or 4 respectively) of
synonymous or nearly-synonymous terms that denigrate specific subsets of
society. The Offensive speech avoidance task was more encompassing, including
a list of 116 terms [42, 43] commonly accepted as offensive, which the NPI was
trained to avoid.

Section 5.1 in Appendix 1 demonstrates the ability of NPIs to influence the
content of generated text to refer to specific political candidates, and
Section 5.4 in Appendix 1 discusses the merits of this approach over the use
of fine-tuning and direct vocabulary manipulation in avoiding offensive
speech.

###  4.3 Thematic Adaptations

Many uses for controlled text generation are thematic; for example, one may
wish to produce text using a simplified vocabulary, or to render text
conforming to a specific dialect, world view, or religious context. Table 5
shows results from one such task: an NPI trained to produce outputs with small
or large average word lengths, respectively. In this context, NPI use has
significant advantages over approaches based on manual restrictions via
reduced output vocabularies or adjustments to word sampling probabilities, as
the NPI retains the ability to use a long word when an appropriate analogue
does not exist in a shorter format, and vice versa.

Table 5: Fine-grained control of word length across 1000 sample sentences. avg word length refers to the mean number of characters per word in the output text. num long words indicates the number of words per text output, on average, whose length was greater than 5.26, which was calculated by taking two standard deviations outside a mean word length calculated from the words in 500 random generic sentences. model name | avg word length | num long words | avg fluency | fluency std dev
---|---|---|---|---
short-NPI | 2.90 | 3.440 | 3.75 | 1.1
long-NPI | 4.10 | 14.013 | 3.65 | 1.10
unmodified GPT-2 | 3.82 | 9.425 | 3.79 | 1.08

##  5 Conclusion

The key contribution and insight of this paper is the use of a small,
independently trained neural network called a Neural Programming Interface
(NPI) to influence the behavior of a large pretrained model. In contrast to
fine-tuning, this approach retains the linguistic breadth and versatility of
the original model, allowing the possibility to control for multiple factors
either in sequence or simultaneously, and to induce behavior in the language
model contrary to the patterns baked into linguistic training data (such as
inducing a polite response in an offensive context). We have demonstrated that
this approach can be used to produce specific words within a GPT-2 model’s
output text, to pivot away from a specific word, and to create a linguistic
aversion to offensive speech. We believe that future avenues for this research
include investigations of the use for NPI models in network interpretability,
regulation, and bias mitigation.

## Broader Impact

The ultimate aim of this research is to provide individuals and organizations
who use neural networks - natural language models or otherwise - with tools
that enable them to test, debug, and regulate the outputs of their machine
learning models. As the NPIs in this paper learned to interpret and control
the previously uninterpretable GPT-2 language model, we hope this approach can
be used to debug and interpret the inner workings of other neural networks in
addition to controlling their output. As such, the primary beneficiaries of
this work are individuals or organizations that wish to incorporate neural
network modeling capabilities with domain knowledge and/or high-level commands
which require a high degree of granular control.

In addition to the general benefits of increased interpretability, our
particular approach enables those with access to a pretrained neural network
to re-purpose it for highly specific tasks without the need for specialized
training data in the pretrained network‘s original training domain; so long as
the model is capable of generating the desired output on occasion, the NPI
approach can be used to amplify this aspect of the pretrained model.
Therefore, individuals and organizations working in data-restricted domains
such as healthcare and news broadcasting, groups with limited funding for data
curation, or individuals and groups who might be under-represented in large
public data sets (such as minority groups or public figures without extensive
media coverage) also stand to benefit from this work. Going even further, we
envision systems capable of leveraging a series of neural classifiers to
detect bias such as political slant or other ideologies in language model
output as it is produced in real-time. When the classifiers detect that the
output is heavily biased towards one ideology, corresponding NPIs are
activated in order to encourage output that leans in the opposite direction,
thus facilitating more balanced text generation and greater representation of
the full range of human dialogue and thought.

A third purpose of this paper is to prevent the abuse of controllable neural
network technologies by making our approach widely known and open to public
discussion. In particular, the spread of misinformation and offensive speech
through automated conversational agents is a modern regulatory challenge, and
we are concerned that controllable natural language generation has the
capability to worsen this modern problem. As such, we desire to aid the public
discourse regarding these technologies by informing individuals at all levels
of society about the current state-of-the-art capabilities of these systems.
Our team has therefore chosen to make our project’s Github repository publicly
available at the time of publication along with the weights of some of our
classification, NPI, and fine-tuned GPT-2 models. However, in the interest of
preventing malicious actors from easily converting our NPI models into tools
for promoting offensive speech or political disinformation campaigns, we have
also chosen to make the weights of some of our data sets and models - those
trained on offensive speech, with political target behaviors, or of a
similarly controversial nature - only available upon request, on a case-by-
case basis, for academics and researchers also interested in ethical machine
learning.

We are the first to acknowledge that the Neural Programming Interfaces (NPIs)
presented in this research are not fail-safe regulatory tools, and should
never be used as a final quality check for natural language generation (or any
other) tasks. The failure-modes of NPIs include total degradation of neural
network output quality (in the case that the NPI model over-aggressively
alters the output of the original model) and insufficient change to the output
of a pretrained neural network (in the case that target content is not
sufficiently adhered to in the NPI output). To mitigate these failure modes,
we suggest that the NPIs presented here be used as back-end procedures for
pretrained neural network fault analysis, data bias analysis, and cached
language generation (which would subsequently be filtered through a battery of
quality checks). As more sophisticated Neural Programming Interfaces (NPIs)
are developed, it may become possible for some of these recommendations to be
eased.

In the course of training classifier neural networks for the adversarial
training of NPI networks, we observed that the content classifier networks
were susceptible to leveraging bias in heavily imbalanced data sets. Because
content classifier networks form a portion of the NPI loss signal, it is
likely that any bias baked into the classifier weights will contribute to bias
in the NPI network itself. We therefore recommend that researchers and
developers ensure training data sets are sufficiently balanced to mitigate
bias in the methods described above.

## Acknowledgments and Disclosure of Funding

We would like to acknowledge the significant contribution of our collaborators
Junseong Ahn and Benjamin Murdoch, whose discussion and insights were
invaluable in both the conceptualization and implementation of this completed
work. We would also like to acknowledge David Carlson for his very helpful
feedback and for supporting the completion of this work. Finally, we would
like to recognize each of the reviewers of this paper, whose insightful
feedback greatly improved the quality of this work.

## References

  * [1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc., 2017.
  * [2] Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le.  Towards a human-like open-domain chatbot, 2020.
  * [3] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan.  A persona-based neural conversation model.  CoRR, abs/1603.06155, 2016.
  * [4] Xupeng Tong, Yikang Li, and Chao-Ming Yen.  Variational neural conversational model.  ICML, 2014.
  * [5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context, 2019.
  * [6] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  OpenAI Blog, 1(8):9, 2019.
  * [7] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 5753–5763. Curran Associates, Inc., 2019.
  * [8] Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, Eric King, Kate Bland, Amanda Wartick, Yi Pan, Han Song, Sk Jayadevan, Gene Hwang, and Art Pettigrue.  Conversational ai: The science behind the alexa prize, 2018.
  * [9] Chun-Yen Chen, Dian Yu, Weiming Wen, Yi Mang Yang, Jiaping Zhang, Mingyang Zhou, Kevin Jesse, Austin Chau, Antara Bhowmick, Shreenath Iyer, et al.  Gunrock: Building a human-like social bot by leveraging large scale real user data.  Alexa Prize Proceedings, 2018.
  * [10] Amanda Cercas Curry, Ioannis Papaioannou, Alessandro Suglia, Shubham Agarwal, Igor Shalyminov, Xinnuo Xu, Ondrej Dusek, Arash Eshghi, Ioannis Konstas, Verena Rieser, et al.  Alana v2: Entertaining and informative open-domain social dialogue using ontologies and entity linking.  Alexa Prize Proceedings, 2018.
  * [11] James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell.  Overcoming catastrophic forgetting in neural networks.  CoRR, abs/1612.00796, 2016.
  * [12] Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher.  Ctrl: A conditional transformer language model for controllable generation, 2019.
  * [13] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3320–3328. Curran Associates, Inc., 2014.
  * [14] Kristine M Knutson, Olga Dal Monte, Selene Schintu, Eric M Wassermann, Vanessa Raymont, Jordan Grafman, and Frank Krueger.  Areas of brain damage underlying increased reports of behavioral disinhibition, 2015.
  * [15] Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu.  Plug and play language models: A simple approach to controlled text generation, 2019.
  * [16] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * [17] Ronan Collobert and Jason Weston.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, page 160–167, New York, NY, USA, 2008. Association for Computing Machinery.
  * [18] Feng Jin and Shiliang Sun.  Neural network multitask learning for traffic flow forecasting.  In 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), pages 1897–1901, 2008.
  * [19] D. H. Wolpert and W. G. Macready.  No free lunch theorems for optimization.  IEEE Transactions on Evolutionary Computation, 1(1):67–82, 1997\.
  * [20] Daniel Svozil, Vladimír Kvasnicka, and Jirí Pospichal.  Introduction to multi-layer feed-forward neural networks.  Chemometrics and Intelligent Laboratory Systems, 39(1):43 – 62, 1997.
  * [21] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 2672–2680. Curran Associates, Inc., 2014.
  * [22] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge.  A neural algorithm of artistic style, 2015.
  * [23] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew.  Huggingface’s transformers: State-of-the-art natural language processing.  ArXiv, abs/1910.03771, 2019.
  * [24] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks, 2013.
  * [25] Paul Michel, Xian Li, Graham Neubig, and Juan Miguel Pino.  On evaluation of adversarial perturbations for sequence-to-sequence models.  CoRR, abs/1903.06620, 2019.
  * [26] Wikipedia contributors.  Plagiarism — Wikipedia, the free encyclopedia, 2004.  [Online; accessed 22-July-2004].
  * [27] Devin Gaffney and J. Nathan Matias.  Caveat emptor, computational social science: Large-scale missing data in a widely-published reddit corpus.  CoRR, abs/1803.05046, 2018.
  * [28] Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  CoRR, abs/1506.06724, 2015.
  * [29] Jeffrey Pennington, Richard Socher, and Christopher D. Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532–1543, 2014.
  * [30] Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le.  Towards a human-like open-domain chatbot, 2020.
  * [31] Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani.  Gedi: Generative discriminator guided sequence generation, 2020.
  * [32] Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, koray kavukcuoglu, Oriol Vinyals, and Alex Graves.  Conditional image generation with pixelcnn decoders.  In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 4790–4798. Curran Associates, Inc., 2016.
  * [33] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alexander Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  In Arxiv, 2016.
  * [34] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra.  Draw: A recurrent neural network for image generation, 2015.
  * [35] Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and Venkatesh Babu Radhakrishnan.  Deligan : Generative adversarial networks for diverse and limited data, 2017.
  * [36] Max Woolf.  minimaxir/gpt-2-simple, Apr 2020.
  * [37] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, NIPS, pages 4349–4357. Curran Associates, Inc., 2016.
  * [38] Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai.  Quantifying and reducing stereotypes in word embeddings.  CoRR, abs/1606.06121, 2016.
  * [39] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan.  Semantics derived automatically from language corpora contain human-like biases.  Science, 356(6334):183–186, 2017.
  * [40] Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou.  Word embeddings quantify 100 years of gender and ethnic stereotypes.  Proceedings of the National Academy of Sciences, 115(16):E3635–E3644, 2018.
  * [41] William L. Hamilton, Jure Leskovec, and Dan Jurafsky.  Diachronic word embeddings reveal statistical laws of semantic change.  CoRR, abs/1605.09096, 2016.
  * [42] Luis von Ahn’s Research Group (CMU).  https://www.cs.cmu.edu/ biglou/resources/bad-words.txt.
  * [43] T. Davidson, D. Warmsley, M. Macy, and I. Weber.  Automated hate speech detection and the problem of offensive language.  2017\.