# Photospheric Velocities Measured at Mt. Wilson Show Zonal and Sectoral Flows
Compose the Torsional Oscillations

Roger K. Ulrich  Tham Tran  John Boyden  Dept. of Phys. and Astro., University
of Calif. at Los Angeles 475 Portola Plaza, Los Angeles, CA 90095-1547

###### keywords:

Helioseismology, Observations; Velocity Fields, Interior; Velocity Fields,
Photosphere; Supergranulation

##  1 Introduction

This paper reports on progress in the analysis of the archival data base from
the observations made at the 150-foot Tower Telescope at the Mt. Wilson
Observatory (MWO). A new method of determining large scale flow velocities and
their variations is presented here along with results showing the close
association between surface velocities deviations going E/W and N/S. These new
results will help in the study of the nature of the solar dynamo which is
currently under intense study using complementary helioseismic and feature
tracking methods. Those methods also yield E/W and N/S transverse velocities
that apply to a range of solar interior depths. Small time dependent
variations derived from these velocities are of particular interest.

The totality of the collected magnetogram/Dopplergram observations is a
resource that allows the study of temporal variation in the solar rotation
rate (Howard (1976); Howard et al. (1983a); Gilman and Howard (1984a); Howard
(1984); Gilman and Howard (1984b)). An important finding of these studies was
the identification of an E/W moving pattern of velocity called the Torsional
Oscillations by Howard and Labonte (1980); LaBonte and Howard (1982b, a) and
Howard (1983). Proposals by Schüssler (1981); Yoshimura (1981) suggested the
Torsional Oscillations are produced by Lorenz forces from the solar cycle
magnetic fields. The work by Howard (1983), however, pointed out that the
Torsional Oscillations occur at times and places where there are no strong
magnetic fields.

The Torsional Oscillations have been confirmed through Helioseismology
observations (Kosovichev and Schou (1997); Schou (1999); Howe et al. (2000);
Getling et al. (2021)); (see the review provided by Howe (2009)). Global
dynamo modeling by Guerrero et al. (2016) finds that magnetic tension at the
tachocline lower boundary of the convection zone can generate a dynamo wave
that propagates to the solar surface. Subsequently, Kosovichev and Pipin
(2019) found support for this idea from a Principal Component Analysis
(Jolliffe (2013)) of the zonal acceleration derived from global
helioseismology.

The analysis of the present paper raises additional questions - a) “What is
the direction of the flow?” and b) “What is the reference function that we
should subtract?” The time scale for changes in the flow is an important part
of characterizing the motion. Since we do not know the cause for the dynamics
we prefer to use the terms “zonal flow” for the E/W motions and “sectoral
flow” for the N/S motions. For the Torsional Oscillations a differential
rotation function is subtracted. For the sectoral flow there is a process
called meridional circulation which is a consequence of stellar rotation [see
for example Sect. 42 of Kippenhahn and Weigert (1990)] which generates a
steady flow with upwelling at the equator, surface flow from equator to pole,
downdraft at the pole and a return flow through the solar envelope, not
necessarily at the convective/radiative boundary. Since the total solar
angular momentum is not variable, this flow should also not be variable.
Assuming the flow variations we find are related to the solar cycle, we need
to subtract a similar background circulation function which we will refer to
as the meridional circulation function. We find both the differential rotation
function and the meridional circulation function by averaging over the full
time span of the record we are using. Data outside our time frame could alter
these functions which would alter the variations we determine. We will show in
this paper that the zonal and sectoral pair of flows have variations which are
both part of the Torsional Oscillations.

The full nature of the Torsional Oscillations has not been evident because
only the east-west component of the velocity vector was part of the variation.
Addition of north-south velocity component as part of the process may help
identify the underlying cause. Although surface meridional circulation due to
rotation generally originates in the equatorial zone and moves toward the
poles, the work by Ulrich (2010), especially figure 6, showed that there are
periods when the flow is toward the equator. That work also showed that there
are strong year-to-year changes.

The study by Komm et al. (2018) found some evidence of equator-ward motion of
the sectoral flow. More extensive studies by Komm et al. (2018, 2020); Getling
et al. (2021) have found variations in the sectoral flow in bands that migrate
toward the solar equator during the solar cycle. They have correlated the
regions on the sun with magnetic activity with the strength of these sectoral
flows. There is a general confirmation of the model by Spruit (2003) wherein
excess emission in magnetic regions causes cooling and sinking of the matter
in magnetized area. Studies by Komm et al. (2020); Getling et al. (2021) have
found evidence in support of this model but there remains the difficulty that
the motions are present before there are strong magnetic fields. Our new
method of analysis shows that the zonal and sectoral components are closely
related throughout the solar cycle.

##  2 Early Record – Why Start in 1983

The digital data record of the 150-foot tower telescope at the Mt. Wilson
Observatory begins in 1967 and our project aims to utilize this full record.
Solar differential rotation rates have typically been represented by a three
term series in $\sin(\phi)$ where $\phi$ is the latitude and the lead term is
$A$. Our equation (1) below is typical. However, the calibration of the $A$
coefficient was flawed in the years prior to 1983. The calibration of the
early observations came from a displacement of the entrance slit a known
distance so that a known signal was introduced. This approach which was
introduced by Babcock, H. W. (1953) as part of the magnetograph design
involved the controlled displacement of the spectral line from its centered
position and the measurement of the resulting artificial signal. Because of
variations in the size of the displacement, the calculated calibration
coefficients were not stable so that the determined $A$ rotation rates and the
calibration coefficients are inversely related. While a correction to the $A$
values can be derived by assuming the calibration coefficient is fixed, the
earlier data will remain less reliable and we have concentrated on the data
obtained after 1983.

##  3 Initial Reductions of Surface Velocities

###  3.1 Background

The instrument and measuring system setup and calibration for the Mt. Wilson
Magnetograph is well described by Howard et al. (1983b) and details of the
setup and operation of the system can be found in that reference. We review
some of these details because they influence results we describe here.

The data from the 150-foot tower project applies to the solar surface yielding
magnetic field $B$, Doppler velocity $V$ and line intensity $I$ as functions
of heliographic latitude $\phi$, central meridian angle $\delta L$ which is
the westward angle from the central meridian to the pixel and time $t$. An
important property of this dataset from the 150-foot tower is its early start
date. A complication dealt with in the reduction of this data comes from the
fact that observed data is not on a regular grid of spatial position and the
temporal spacing is irregular both within individual observations and between
successive observations. Consequently interpolations are required to produce
tables of the observables in regularly spaced grids.

The polarization analysis uses a rapidly oscillating K*DP crystal combined
with a Glan-Thompson prism to alternate the state of circular polarization
allowed through the spectrograph. The spectral sampling is done by rectangular
shaped fiber-optic light pipes whose long dimension entrance windows are
perpendicular to the dispersion direction. Paired samples of the spectrum from
the red, $R$, and blue, $B$, wings of the sampled line are sent to two pulse-
counting photomultiplier tubes yielding count values stored in four registers:
$R^{+},R^{-},B^{+},B^{-}$ where the superscript indicates the polarization
state. At regular intervals the sum values are recorded on disk and the
registers are cleared. Each data record includes the four sample values, the
time along with the location of the servoed stage carrying the sampling
rectangles and the position of the servos controlling the position of the
solar image. These packets of data are recorded sequentially on a magnetic
disk. At the end of the observation these records get transferred to magnetic
tape. These files have a name structure where the letter ’M’ is followed by
YYYYMMDD_HHHH_MM.01 where YYYY is the year, MM is the month, DD is the day and
HHHH_MM is the time. We refer to these as the 01 files.

While the 01 files include all the necessary information, they are in a
difficult-to-use format - each data element has a different time, the spacing
between the scan lines is not constant and the elements are not lined up
vertically. We address these problem in steps: 1) Each file is converted to an
internal format we call an MVI file which has three images which are shifted
by differential rotation to be at the same time and to compensate for the
guiding system which is imperfect. These shifts produce a grid which is
rectangular. These images show the magnetic, velocity and intensity variables,
2) For each day the pixels of the MVI images are shifted at constant latitude
using the local rotation rate to a fixed time and are averaged together, 3) A
set of fits files is created for each average and a text file called VMREC is
written. The VMREC file is derived by appling geometric corrections to the
background limb shift function [see section (4)], along with first guess
differential rotation and meridional circulation laws to yield position
dependent line-of-sight velocity deviation and magnetic field variables as
functions of $x,y$ indicies $i$, $j$. Steps to derive an update to the flow
functions from the VMREC file are described in the following sections.

###  3.2 Differential Rotation and Sectoral Flow

A primary output of the observing program at the 150-foot tower telescope has
been the study of the time dependence of the solar surface velocities. The
system provides Doppler shifts of the spectral line which are converted to
rotation rate and N/S flow velocity using the spherical trigonometry
algorithms from Howard and Harvey (1970). In addition we modify the sign for
the N/S velocity by reversing the sign in the southern hemisphere so a
positive value represents flow toward the poles. The largest feature of these
velocities is the differential rotation which has a large East/West component.
Flow variability with small amplitude cannot be detected superposed on the
differential rotation unless the main pattern is subtracted. This has been
done using a polynomial with the coefficients $(A,B,C)$ in the representation

$\Omega(\phi,t)=A(t)+B(t)\sin^{2}(\phi)+C(t)\sin^{4}(\phi)\;\mu\,{\rm rad/s}$
(1)

where $\Omega$ is the rotation rate and $\phi$ is the heliographic latitude.
For many years this polynomial was fitted to each observation. This
representation is imprecise with there being no expectation that the flow
should have this structure. To address this concern and still allow us to
detect small and variable structures we have run the 01 files through a new
initial reduction with the modification that the $B,C$ pair of coefficients is
held fixed. This produces a new archive of MVI files and other products from
which we can retrieve the observed flows but which has a static, first guess
differential rotation offset curve subtracted. The curve we have used is:

$\Delta\Omega_{0}(\phi)=-0.409999*[\sin^{2}(\phi)+1.0216295*\sin^{4}(\phi)]\;\mu\,{\rm
rad/s}\ .$ (2)

Note that the subtracted quantity is negative so the rotation rate is
increased to become more nearly constant with latitude. The flow remaining
after equation (2) has been applied can be described by either a rotation
velocity deviation $\delta V_{\rm rot}$ or as an angular rotation rate
deviation $\delta\omega$. The conversion from angular rotation rate deviation
to rotation velocity deviation is:

$\delta V_{\rm Rot}(\phi,t)=\delta\omega(\phi,t)*R_{\odot}\cos(\phi)\ .$ (3)

We use equation (3) to convert rotation rates or rotation rate deviations to
rotation velocities or rotation velocity deviations and vice versa as needed.

We observe the line-of-sight velocity $V_{\rm los}$ at each point. Using the
geometry as described by Howard and Harvey (1970) we convert this velocity to
a rotation rate $\Omega(\phi,t)$ and a rotation rate deviation:

$\delta\omega_{0}(\phi,t)=\Omega(\phi,t)-\Delta\Omega_{0}(\phi)\ .$ (4)

The values of $\delta\omega_{0}(\phi,t)$ along with magnetic field strength
and line intensity can be read into standard fits files which we pick to be
daily averages with dimension $256\times 256$ in $\delta L$ and $\phi$.

The differential rotation rate offset from Equation (2) was adopted in 1987
and those coefficients do not accurately represent the rotation rate over the
full time series. Consequently, we work with a reference rotation rate
equation (5) and we get rotation rate deviations relative to the actual
30-year average equation (6)

$\delta\omega_{\rm ref}(\phi)=<\Omega(\phi,t)>_{\rm
average}-\Delta\Omega_{0}(\phi)$ (5)
$\delta\omega(\phi,t)=\delta\omega_{0}(\phi,t)-\delta\omega_{\rm ref}(\phi)\
.$ (6)

In order to recover the rotation rates themselves we have to use the sum

$\Omega(\phi,t)=\delta\omega(\phi,t)+\delta\omega_{\rm
ref}(\phi)+\Delta\Omega_{0}(\phi)\ .$ (7)

The original program reduction yielded the first guess angular rotation rate
as in equation (1) whereas our new reduction yields local line-of-sight
velocity deviations relative to the projection of the 30-year average rotation
rate. The new rotation velocity deviation is then

$\delta V_{\rm rot}={d\delta V_{\rm los}\over d\sin(\delta L)}\ .$ (8)

This equation provides a value of $\delta V_{\rm rot}$ for each pixel of the
image.

The first guess background meridional flow law adopted at the same time as the
differential rotation and is:

$V_{\rm mer}(\phi,t)=[\cos({\delta
L})\sin(\phi)\cos(B_{0})-\cos(\phi)\sin(B_{0})]\\{G\sin(\phi)+H\sin(2\phi)\\}$
(9)

where $G=-28.692\,{\rm m\,s}^{-1}$ and $H=35.240\,{\rm m\,s}^{-1}$. The
sectoral velocity deviation we seek, $\delta V_{\rm sec}$, is relative to this
function corrected the same way as $\delta V_{\rm rot}$. This function was
subtracted during the initial reduction and has to be restored now as part of
the determination of the 30-year average.

##  4 New Rotation Rate Algorithm

The method we used previously starts with the initial reduction with the
algorithm described by Howard et al. (1983b) which uses pixels from most of
the solar disk including those near the limb. The intensity gradient
approaching the limb causes scattered light to have a significant effect on
the determined rotation law. The amount of scattered light in each solar image
is determined by recording intensity values for pixels near but outside the
limb. The average from such pixels is used to quantify the scattered light
which after correlation with the measured rotation rate allows the derivation
of a correction to the rotation rate. For observations with high scattered
light, the correction can be large. Our new method avoids the need for this
correction.

The scattered light correction is important near the limb due to the steep
gradient in intensity as the limb is approached. The position of each pixel is
defined by $\phi$, and $\delta L$. The new method avoids the steep gradient
zone by requiring $|\sin(\delta L)|<0.85$ which leaves out the near-limb
pixels. The heliographic geometry and resolution of the observed Doppler
velocities into rotation and meridional circulation was described by Howard
and Harvey (1970) and Thompson (2006).

Our task of determining the zonal and sectoral flow rates is complicated by
another effect known as the limb shift. Due to correlations between vertical
velocity and spectral line intensity due to temperature changes, photospheric
spectral lines are shifted as a function of the angle the emergent ray makes
relative to the local vertical. Both the flow fields and the limb shift
contribute to the Doppler shift velocity, $V_{\rm los}$, of each pixel and we
wish to isolate only the material flow. We find the limb shift by using an
equatorial band 4∘ wide out of the velocity array before the subtraction of
the differential rotation. These entries are minimally influenced by the
sectoral flow and symmetrically influenced by the zonal flow so we can use
this band to derive the limb shift function. The relationship between the limb
shift and the zonal and sectoral material flows was discussed in detail by
Ulrich et al. (1988).

As described in Section(3) the scanning system at the 150-foot tower at Mt.
Wilson produces a data file VMREC which gives position indices $i$ and $j$
along with daily averages of the line-of-sight component of the rotation
velocity deviations, and line-of-sight magnetic field strength in $256\times
256$ arrays. Our working quantities are the line-of-sight component of the
rotation velocity deviations $\delta V_{\rm los}$ and the magnetic fields. We
calculate these for the latitude bands 4∘ wide defined below using our new
method.

Figure 1.: This figure shows a sample daily average magnetogram from Oct. 28,
2003. The average is formed by spatial interpolating to a rectangular of
dimension $256\times 256$ in $\delta L$ and $\phi$ and temporal shifting of
each pixel using differential rotation to where the point would be at 20.0 UT.
The pixel positions marked out with an $X$ are those which were included in
the algorithm for determining the rotation rate. These pixels are in latitude
band number 16 which covers the range $20^{\circ}S<\phi<16^{\circ}S$. In
addition accepted pixels have $|\sin(\delta L)|<0.85$ and have an absolute
magnetic field strength $|B|$ less than 30G. Notice that along the lines where
the magnetic field changes sign there are a number of included pixels because
the field direction is nearly horizontal. Although this figure shows the
magnetogram, the velocities for the shown pixels are extracted at the
corresponding points on the Dopplergram.

Figure 2.: This is the sample latitude band shown in Figure(1). Each selected
pixel is shown with an $X$ symbol at its line-of-sight velocity $V_{\rm los}$
and sine central meridian angle: $\sin(\delta L)$. The red line is the
quadratic fit to those pixels and the fitting coefficients $\alpha$ and
$\beta$ are shown by the blue arrows. The value of $\beta$ is indicated by the
very small but non-zero slope of the arrow. The rms variation of both $\alpha$
and $\beta$ is about 15 m/s overall and the value of $\beta$ is unusually
small in this case. This figure is for a single day of observation while the
quantities used for measurement of the surface flows are averages over six
Carrington rotations.

The new method begins by using the latitude $\phi$ to define bands which are
each $4^{\circ}$ wide:

$j_{\rm band}={\rm Round}((\phi+82)/4)$

and collects pixels within each band. We refer to $j_{\rm band}$ as the band
number. The selected pixels also have $|\sin(\delta L)|<0.85$. The Doppler
shift velocities for each daily observation and each latitude band is then fit
by a quadratic function of $\sin(\delta L)$ yielding three coefficients
$(\alpha,\beta,\gamma)$:

$V_{los}(\delta L)=\alpha+\beta\sin(\delta L)+\gamma\sin^{2}(\delta L)\ .$
(10)

The coefficients are: $\alpha$ is the daily offset velocity due to the
sectoral flow, $\beta$ gives the daily rotation velocity deviation and
$\gamma$ allows the flexibility to match the other two coefficients. The
rotation rate deviation, $\delta V_{\rm rot}(\phi,t)$, is then found from the
average of the $\beta$ values over 6 Carrington rotations.

Each daily result is influenced by supergranulation noise. The only way to
reduce this noise is through averaging. For that reason we combine the daily
results for $\delta V_{\rm rot}$ and $\delta V_{\rm sec}$ in 6 or 12
Carrington rotation averages. When we use 6 CR averages, we apply an even-odd
filter, (0.25,0.50,0.25), to the time series to remove the effect of polar
tilt from the final series.

As an example we show in figure (2) the $\sin(\delta L)$ dependent points from
Oct. 28, 2003 with each point value and the fit to equation (10). This is the
same day and latitude band shown in Figure (1). This particular day
corresponds to Carrington Rotation 2009.1936 and year 2003.8222. The fit
coefficients for latitude band 16 for this day are
$\alpha=17.431,\beta=2.874{\rm\ and\ }\gamma=-71.142$ all with units of m/s
since the $\sin(\delta L)$ factor is dimensionless.

##  5 Data Reduction Steps

The initial step carries out the latitude band fit described in the preceding
section for the 7939 days of observation in the record. Each day produces a
line of output with 42 entries of $\alpha$ and $\beta$. These are combined
into a Microsoft Excel file which is output into two text files, one for the
Northern hemisphere and one for the Southern hemisphere. Entries in these
files are Null if fewer than 8 points are in the latitude band or if the slope
is greater than 100 m/s.

Table 1.: This table gives the angular rotation rate from Equation (7) as a
function of latitude band averaged over the 30 Years of this data set. The
error of the mean, $\delta\Omega$ EoM, is also given based on the variance of
the data values and the number of points included.
---

The time series for each latitude band is cleaned independently using a
procedure based on the variance. The variance for each group of 120 points is
calculated and a quadratic function of time is fitted to them. The point with
the largest absolute deviation from the fit is found and removed and the
variance is recalculated. If this variance has decreased by more than 8%, the
point is replaced by the fitted value and next largest deviation is found and
removed. The test is repeated until the variance changes by less than 8%. The
last deleted point is then restored and the analysis moves to the next group
of 120 points. For each latitude band’s time sequence we have about 50 points
out of the 7939 that are removed this way. The points removed, however, would
have a strongly adverse effect on the derived velocities if left in. The
highest latitude sequences have fewer observations due to the requirement that
the velocities stay below 100 m/s or due to the failure to have more than 8
observations at each accepted latitude and time. The same cleaning process is
applied to the sectoral flow velocities.

Figure 3.: This figure shows the 30-year average of the sectoral velocity as a
function of latitude. This average is the combination of the offset during the
initial reduction and the average of the individual offsets determined with
this new method. See the discussion of the sign convention in the text of
section(5).

Additionally the rotation rate as a function of time and latitude is
calculated by dividing the rotation velocities by $R_{\odot}\cos(\phi)$. Next
we compute the 30-year averages of the rotation velocities, the rotation rates
and the sectoral flow velocities. The 30-year averages of the rotation rates
are given in Table 1. The North/South difference in this table is likely of
solar origin. The sectoral velocities have been treated similarly with the
exception that we have reversed the sign in the southern hemisphere. For
coordinates on a sphere positive velocity values flow from south toward north.
For the sectoral velocities we desire the positive velocities to be flowing
away from the equator and toward the poles. For the southern hemisphere this
requires a sign reversal. In addition the initial reductions subtracted the
final term with $G$ and $H$ in Equation (10) of Ulrich et al. (1988) as
indicated above in Equation (9). To get the average sectoral flow velocity we
have to combine this term and the average offset of the arrays in the archive.
This sum including the sign reversal is shown in Figure (3).

##  6 Rotational and Meridional Flow Maps

Figure 4.: The velocity deviations shown in these two figures have been
derived according to the algorithms described in the previous sections.
Temporal filtering with a (0.25,0.5,0.25) even/odd filter was applied to both
maps prior to plotting. The two components - East West in the zonal plot and
North South in the sectoral plot together constitute the process known as the
Torsional Oscillations. The dashed white line traces the latitude of the
maximum speedup at each time on the zonal map. These line locations are
transfered to the same latitudes in the sectoral plot map. For latitudes near
10∘, the time of maximum poleward flow speed in this second map is a year or
so after the position of this line, but for most of the plot the times
coincide.

The last step in the reduction is the summing of the zonal and sectoral flow
velocity deviations for each latitude band into temporal bins of 6 Carrington
rotations in duration. For each bin of 6 Carrington rotations we obtain 42
values of zonal flow velocity deviations and 42 values of sectoral flow
deviations. Because the time interval for 6 Carrington rotation is not an even
fraction of the year, the x-axis points drift relative to the year.

The flow deviation grids can be plotted as flow maps. These maps are shown in
Figure (4). The similarity between the patterns of speed-up and slow-down for
these two flows is remarkable. The white lines were drawn by eye on the zonal
flow map to correspond roughly with the location of the zonal flow maxima. The
lines were then transfered to the sectoral flow map to allow comparison of the
spatial-temporal patterns. The combined flow patterns are what constitute the
surface pattern of the Torsional Oscillations. The patterns have greater
regularity than has been seen before and sectoral flow extends to higher
latitude than shown by Getling et al. (2021).

##  7 Time Slice Plots - Solar Cycle Dependence

The 2D color plots give an overall view of the behavior of the zonal and
sectoral flows. However the forms of the functions are not readily apparent.
There are several shifts in $\delta V_{\rm sectoral}$ occurring at different
latitudes and showing too many curves on a single plot leads to a spider-web
result where the consistent features cannot be seen. We have previously noted
[Ulrich et al. (2022)] that $\delta V_{\rm sectoral}$ shows upwelling at
sunspot minimum. Based on this idea we have segregated the time slice plots
based on the strength and sign of the high latitude sectoral velocity. The
phase of the solar cycle is well followed with this discriminant.

Figure 5.: Time slice plots of $\delta V_{\rm sectoral}$ versus Latitude for
12 Carrington Rotation averages. The curves are color coded and the times of
the averages are show on the left with the characters of the times having the
same color as the lines. The times are the start and finish for each temporal
average. The choice of panel for showing each plot was based on both the high
latitude flow value and the active latitude flow value. The spacing started
with two lines per panel but then some lines needed to be moved forward or
backward to maintain the most nearly similar shapes.

Figure (5) shows the time slice plots segregated according to the time within
the solar cycle which picks the selection very well according to the high
latitude values of $V_{\rm sectoral}$. Panel A is at the times of minimum with
the strongest polar outflow. The active region bands near
$15^{\circ}-20^{\circ}$ N/S have equatorial direction flows as well. Panels B
and C are transitioning toward solar maximum with the high latitudes going
toward poleward flow and the active region bands also developing stronger
poleward flow. Panel D is after solar maximum but still has the active region
bands flowing poleward while the high latitudes are still strongly poleward in
their flow. Panels E and F are transitioning to solar minimum with the high
latitude flow becoming equatorward in panel F. In both these panels the active
latitude bands are equatorward in their flow. It is worth noting that the time
between solar maximum and solar minimum was longer for cycle 23 than for cycle
22 yet the dynamics of the sectoral flow continued to follow the pattern as
minimum was approached in cycle 23.

##  8 Conclusions

This study has yielded sectoral flow velocities at latitudes which are much
closer to the poles compared to what has been available before. The pattern
seen in Figure (3) is consistent with expectations for the meridional flow
between latitudes of 15∘ and 60∘. Nearer the equator the determination here is
uncertain due to the possible vertical flow which is being interpreted by our
formulations as the projection of a flow along a level surface. The strong
negative values in Figure (3) are no doubt the result of a subsidence at the
equator. Near the poles the return flow from the time variable circulation
currents dominates the observed sectoral flow and includes periods when the
polar upwelling generates strong equatorward flows. It can be hoped that the
new observational results will help the development of a more complete theory
of the solar circulation.

The Torsional Oscillations are well studied and it is well known that the
migration of the pattern from high latitudes to the equator requires most of
two solar cycles. This study has shown that the sectoral flow deviations and
the zonal flow deviations are well synchronized during these cycles. The
Torsional Oscillations evidently involve motions in the N/S direction as well
as the E/W direction. The high latitude polar outflow at sunspot minimum found
by Ulrich et al. (2022) extends to a polar inflow at sunspot maximum. The
pattern of high latitude flow deviations is dependent on the phase of the
sunspot cycle and can provide a new way of predicting the time remaining
before the next sunspot minimum.

#### Acknowledgements

The Mt. Wilson Observatory project was started by the Carnegie Institution of
Washington and financed by that institution until 1984 when management and
funding were transferred to UCLA. Before the transfer funding was provided by
the US Navy through its Office of Naval Research as well as NASA and NSF.
After the transfer NASA and NSF provided most of the support along with
continuing support from ONR. Current support comes from NSF through grant
2000994 while earlier support came from the NSF through grant AGS-0958779 and
NASA through grants NNX09AB12G and HMI subcontract 16165880. Over the years
additional funding has come from these two agencies as well as the ONR and
NOAA.. Ulrich and Boyden are supported by retirement funds while Tran is
supported by the NSF grant. The observatory is managed by the Mt. Wilson
Institute.

## References

  * Babcock, H. W. (1953) Babcock, H. W.: 1953, ‘The Solar Magnetograph.’.  ApJ 118, 387.
  * Getling et al. (2021) Getling, A. V., A. G. Kosovichev, and J. Zhao: 2021, ‘Evolution of Subsurface Zonal and Meridional Flows in Solar Cycle 24 from Helioseismological Data’.  ApJ 908(2), L50.
  * Gilman and Howard (1984a) Gilman, P. A. and R. Howard: 1984a, ‘On the correlation of longitudinal and latitudinal motions of sunspots’.  Sol. Phys. 93, 171–175.
  * Gilman and Howard (1984b) Gilman, P. A. and R. Howard: 1984b, ‘Variations in solar rotation with the sunspot cycle’.  ApJ 283, 385–391.
  * Guerrero et al. (2016) Guerrero, G., P. K. Smolarkiewicz, E. M. de Gouveia Dal Pino, A. G. Kosovichev, and N. N. Mansour: 2016, ‘Understanding Solar Torsional Oscillations from Global Dynamo Models’.  ApJ 828(1), L3.
  * Howard (1976) Howard, R.: 1976, ‘A possible variation of the solar rotation with the activity cycle.’.  ApJ 210, L159–L161.
  * Howard (1983) Howard, R.: 1983, ‘Torsional Oscillations of the Sun’.  Sol. Phys. 82(1-2), 437–437.
  * Howard (1984) Howard, R.: 1984, ‘Solar Rotation’.  ARA&A 22, 131–155.
  * Howard et al. (1983a) Howard, R., J. M. Adkins, J. E. Boyden, T. A. Cragg, T. S. Gregory, B. J. Labonte, S. P. Padilla, and L. Webster: 1983a, ‘Solar Rotation Results at Mount-Wilson - Part Four - Results’.  Sol. Phys. 83(2), 321–338.
  * Howard et al. (1983b) Howard, R., J. E. Boyden, D. H. Bruning, M. K. Clark, H. W. Crist, and B. J. Labonte: 1983b, ‘The Mount Wilson Magnetograph (Report from a Solar Institute)’.  Sol. Phys. 87, 195–+.
  * Howard and Harvey (1970) Howard, R. and J. Harvey: 1970, ‘Spectroscopic Determinations of Solar Rotation’.  Sol. Phys. 12, 23–51.
  * Howard and Labonte (1980) Howard, R. and B. J. Labonte: 1980, ‘The sun is observed to be a torsional oscillator with a period of 11 years’.  ApJ 239, L33–L36.
  * Howe (2009) Howe, R.: 2009, ‘Solar Interior Rotation and its Variation’.  Living Reviews in Solar Physics 6(1), 1.
  * Howe et al. (2000) Howe, R., J. Christensen-Dalsgaard, F. Hill, R. W. Komm, R. M. Larsen, J. Schou, M. J. Thompson, and J. Toomre: 2000, ‘Deeply Penetrating Banded Zonal Flows in the Solar Convection Zone’.  ApJ 533, L163–L166.
  * Jolliffe (2013) Jolliffe, I.: 2013, Principal Component Analysis, Springer Series in Statistics.  New York, NY: Springer.
  * Kippenhahn and Weigert (1990) Kippenhahn, R. and A. Weigert: 1990, Stellar Structure and Evolution.  Berlin, Germany: Springer-Verlag.
  * Komm et al. (2018) Komm, R., R. Howe, and F. Hill: 2018, ‘Subsurface Zonal and Meridional Flow During Cycles 23 and 24’.  Sol. Phys. 293(10), 145.
  * Komm et al. (2020) Komm, R., R. Howe, and F. Hill: 2020, ‘Solar-Cycle Variation of the Subsurface Flows of Active- and Quiet-Region Subsets’.  Sol. Phys. 295(3), 47.
  * Kosovichev and Pipin (2019) Kosovichev, A. G. and V. V. Pipin: 2019, ‘Dynamo Wave Patterns inside of the Sun Revealed by Torsional Oscillations’.  ApJ 871(2), L20.
  * Kosovichev and Schou (1997) Kosovichev, A. G. and J. Schou: 1997, ‘Detection of Zonal Shear Flows beneath the Sun’s Surface from f-Mode Frequency Splitting’.  ApJ 482(2), L207–L210.
  * LaBonte and Howard (1982a) LaBonte, B. J. and R. Howard: 1982a, ‘Are the high-latitude torsional oscillations of the sun real?’.  Sol. Phys. 80, 373–378.
  * LaBonte and Howard (1982b) LaBonte, B. J. and R. Howard: 1982b, ‘Torsional waves on the sun and the activity cycle’.  Sol. Phys. 75, 161–178.
  * Schou (1999) Schou, J.: 1999, ‘Migration of Zonal Flows Detected Using Michelson Doppler Imager F-Mode Frequency Splittings’.  ApJ 523(2), L181–L184.
  * Schüssler (1981) Schüssler, M.: 1981, ‘The solar torsional oscillation and dynamo models of the solar cycle’.  A&A 94, L17+.
  * Spruit (2003) Spruit, H. C.: 2003, ‘Origin of the torsional oscillation pattern of solar rotation’.  Sol. Phys. 213, 1–21.
  * Thompson (2006) Thompson, W. T.: 2006, ‘Coordinate systems for solar image data’.  A&A 449, 791–803.
  * Ulrich (2010) Ulrich, R. K.: 2010, ‘Solar Meridional Circulation from Doppler Shifts of the Fe I Line at 5250 Å as Measured by the 150-foot Solar Tower Telescope at the Mt. Wilson Observatory’.  ApJ 725(1), 658–669.
  * Ulrich et al. (1988) Ulrich, R. K., J. E. Boyden, L. Webster, S. P. Padilla, and H. B. Snodgrass: 1988, ‘Solar rotation measurements at Mount Wilson. V - Reanalysis of 21 years of data’.  Sol. Phys. 117, 291–328.
  * Ulrich et al. (2022) Ulrich, R. K., T. Tran, and J. E. Boyden: 2022, ‘Polar Upwelling at Three Sunspot Minima’.  Res. Notes AAS 6(9), 181.
  * Yoshimura (1981) Yoshimura, H.: 1981, ‘Solar cycle Lorentz force waves and the torsional oscillations of the sun’.  ApJ 247, 1102–1112.


# Select and Attend: Towards Controllable Content Selection in Text Generation

Xiaoyu Shen1,2, Jun Suzuki3,4, Kentaro Inui3,4, Hui Su5
Dietrich Klakow1 and Satoshi Sekine4
1Spoken Language Systems (LSV), Saarland University, Germany
2Max Planck Institute for Informatics, Saarland Informatics Campus, Germany
3Tohoku University 4RIKEN AIP, Japan
5Pattern Recognition Center, Wechat AI, Tencent Inc, China  Work mostly done
while at RIKEN AIP. Correspondence to<EMAIL_ADDRESS>
###### Abstract

Many text generation tasks naturally contain two steps: content selection and
surface realization. Current neural encoder-decoder models conflate both steps
into a black-box architecture. As a result, the content to be described in the
text cannot be explicitly controlled. This paper tackles this problem by
decoupling content selection from the decoder. The decoupled content selection
is human interpretable, whose value can be manually manipulated to control the
content of generated text. The model can be trained end-to-end without human
annotations by maximizing a lower bound of the marginal likelihood. We further
propose an effective way to trade-off between performance and controllability
with a single adjustable hyperparameter. In both data-to-text and headline
generation tasks, our model achieves promising results, paving the way for
controllable content selection in text generation.111The source code is
available on https://github.com/chin-gyou/controllable-selection

##  1 Introduction

Source Sentence: The sri lankan government on Wednesday announced the closure
of government schools with immediate effect as a military campaign against
tamil separatists escalated in the north of the country.
---
Selected :  sri lankan, closure, schools
Text:  sri lanka closes schools .
Selected :  sri lankan, Wednesday, closure, schools
Text:  sri lanka closes schools on Wednesday.
Selected :  sri lankan, closure, schools, military campaign
Text:  sri lanka shuts down schools amid war fears.
Selected :  sri lankan, announced, closure, schools
Text:  sri lanka declares closure of  schools.
Table 1: Headline generation examples from our model. We can generate text
describing various contents by sampling different content selections. The
selected source word and its corresponding realizations in the text are
highlighted with the same color.

Many text generation tasks, e.g., data-to-text, summarization and image
captioning, can be naturally divided into two steps: content selection and
surface realization. The generations are supposed to have two levels of
diversity: (1) content-level diversity reflecting multiple possibilities of
content selection (what to say) and (2) surface-level diversity reflecting the
linguistic variations of verbalizing the selected contents (how to say) Reiter
and Dale (2000); Nema et al. (2017). Recent neural network models handle these
tasks with the encoder-decoder (Enc-Dec) framework Sutskever et al. (2014);
Bahdanau et al. (2015), which simultaneously performs selecting and
verbalizing in a black-box way. Therefore, both levels of diversity are
entangled within the generation. This entanglement, however, sacrifices the
controllability and interpretability, making it diffifcult to specify the
content to be conveyed in the generated text Qin et al. (2018); Wiseman et al.
(2018).

With this in mind, this paper proposes decoupling content selection from the
Enc-Dec framework to allow finer-grained control over the generation. Table 1
shows an example. We can easily modify the content selection to generate text
with various focuses, or sample multiple paraphrases by fixing the content
selection.

Though there has been much work dealing with content selection for the Enc-
Dec, none of them is able to address the above concerns properly. Current
methods can be categorized into the following three classes and have different
limits:

  1. 1.

Bottom-up: Train a separate content selector to constrain the attention to
source tokens Gehrmann et al. (2018), but the separate training of
selector/generator might lead to discrepancy when integrating them together.

  2. 2.

Soft-select: Learn a soft mask to filter useless information Mei et al.
(2016); Zhou et al. (2017). However, the mask is _deterministic_ without any
probabilistic variations, making it hard to model the content-level diversity.

  3. 3.

Reinforce-select: Train the selector with reinforcement learning Chen and
Bansal (2018), which has high training variance and low diversity on content
selection.

In this paper, we treat the content selection as latent variables and train
with amortized variational inference Kingma and Welling (2014); Mnih and
Gregor (2014). This provides a lower training variance than Reinforce-select.
The selector and generator are co-trained within the same objective, the
generations are thus more faithful to the selected contents than Bottom-up
methods. Our model is task-agnostic, end-to-end trainable and can be
seamlessly inserted into any encoder-decoder architecture. On both the data-
to-text and headline generation task, we show our model outperforms others
regarding content-level diversity and controllability while maintaining
comparable performance. The performance/controllability trade-off can be
effectively adjusted by adjusting a single hyperparameter in the training
stage, which constrains an upper bound of the conditional mutual information
(CMI) between the selector and generated text Alemi et al. (2018); Zhao et al.
(2018). A higher CMI leads to stronger controllability with a bit more risk of
text disfluency.

In summary, our contributions are (1) systematically studying the problem of
controllable content selection for Enc-Dec text generation, (2) proposing a
task-agnostic training framework achieving promising results and (3)
introducing an effective way to achieve the trade-off between performance and
controllability.

##  2 Background and Notation

Let $X,Y$ denote a source-target pair. $X$ is a sequence of
$x_{1},x_{2},\ldots,x_{n}$ and can be either some structured data or
unstructured text/image depending on the task. $Y$ corresponds to
$y_{1},y_{2},\ldots,y_{m}$ which is a text description of $X$. The goal of
text generation is to learn a distribution $p(Y|X)$ to automatically generate
proper text.

The Enc-Dec architecture handles this task with an encode-attend-decode
process Bahdanau et al. (2015); Xu et al. (2015). The encoder first encodes
each $x_{i}$ into a vector $h_{i}$. At each time step, the decoder pays
attentions to some source embeddings and outputs the probability of the next
token by $p(y_{t}|y_{1\mathrel{\mathop{\mathchar 58\relax}}t-1},C_{t})$.
$C_{t}$ is a weighted average of source embeddings:

$\begin{split}C_{t}&=\sum_{i}\alpha_{t,i}h_{i}\\\
\alpha_{t,i}&=\frac{e^{f(h_{i},d_{t})}}{\sum_{j}e^{f(h_{j},d_{t})}}\end{split}$
(1)

$d_{t}$ is the hidden state of the decoder at time step $t$. $f$ is a score
function to compute the similarity between $h_{i}$ and $d_{t}$ Luong et al.
(2015).

##  3 Content Selection

Our goal is to decouple the content selection from the decoder by introducing
an extra content selector. We hope the content-level diversity can be fully
captured by the content selector for a more interpretable and controllable
generation process. Following Gehrmann et al. (2018); Yu et al. (2018), we
define content selection as a sequence labeling task. Let
$\beta_{1},\beta_{2},\ldots,\beta_{n}$ denote a sequence of binary selection
masks. $\beta_{i}=1$ if $h_{i}$ is selected and 0 otherwise. $\beta_{i}$ is
assumed to be independent from each other and is sampled from a bernoulli
distribution $\mathbf{B}(\gamma_{i})$222Devlin et al. (2019) have shown that
excellent performance can be obtained by assuming such conditionally
independence given a sufficiently expressive representation of $x$, though
modelling a richer inter-label dependency is for sure beneficial Lei et al.
(2016); Nallapati et al. (2017).. $\gamma_{i}$ is the bernoulli parameter,
which we estimate using a two-layer feedforward network on top of the source
encoder. Text are generated by first sampling $\beta$ from
$\mathbf{B}(\gamma)$ to decide which content to cover, then decode with the
conditional distribution $p_{\theta}(Y|X,\beta)$. The text is expected to
faithfully convey all selected contents and drop unselected ones. Fig. 1
depicts this generation process. Note that the selection is based on the
token-level _context-aware_ embeddings $h$ and will maintain information from
the surrounding contexts. It encourages the decoder to stay faithful to the
original information instead of simply fabricating random sentences by
connecting the selected tokens.

Figure 1: Model will select contents based on $\mathbf{B}(\gamma)$, then
decode with $p_{\theta}(Y|X,\beta)$. Source-text pairs are available for
training, but the ground-truth content selection for each pair is unknown.

For each source-target pair, the ground-truth selection mask is unknown, so
training is challenging. In the following session, we discuss several training
possibilities and introduce the proposed model in detail.

###  3.1 Bottom-up

The most intuitive way is training the content selector to target some
heuristically extracted contents. For example, we can train the selector to
select overlapped words between the source and target Gehrmann et al. (2018),
sentences with higher tf-idf scores Li et al. (2018) or identified image
objects that appear in the caption Wang et al. (2017). A standard encoder-
decoder model is independently trained. In the testing stage, the prediction
of the content selector is used to hard-mask the attention vector to guide the
text generation in a bottom-up way. Though easy to train, Bottom-up generation
has the following two problems: (1) The heuristically extracted contents might
be coarse and cannot reflect the variety of human languages and (2) The
selector and decoder are independently trained towards different objectives
thus might not adapt to each other well.

$\bm{\beta}$ as Latent Variable: Another way is to treat $\beta$ as a latent
variable and co-train selector and generator by maximizing the marginal data
likelihood. By doing so, the selector has the potential to automatically
explore optimal selecting strategies best fit for the corresponding generator
component.

With this in mind. We design $p_{\theta}(Y|X,\beta)$ by changing the original
decoder in the following way: (1) We initialize hidden states of the decoder
from a mean pooling over selected contents to inform the decoder which
contents to cover and (2) Unselected contents will be prohibited from being
attended to:

$\displaystyle\begin{split}&d_{0}=\text{MLP}\left(\frac{1}{n}\left(\sum_{i}^{n}\beta_{i}h_{i}\right)\right)\\\
&\alpha_{t,i}=\frac{e^{f\left(h_{i},d_{t}\right)}\beta_{i}}{\sum_{j}e^{f\left(h_{j},d_{t}\right)}\beta_{j}}\end{split}$
(2)

$d_{0}$ is the initial decoder hidden state and MLP denotes multi-layer-
perceptron.

Since computing the exact marginal likelihood
$\log\mathbb{E}_{\beta\sim\mathbf{B}(\gamma)}p_{\theta}(Y|X,\beta)$ requires
enumerating over all possible combinations of $\beta$ (complexity
$\mathbf{O}(2^{n})$), we need some way to efficiently estimate the likelihood.

###  3.2 Soft-Select

Soft-select falls back on a _deterministic_ network to output the likelihood
function’s first-order Taylor series approximation expanded at
$\mathbb{E}_{\beta\sim\mathbf{B}(\gamma)}\beta$:

$\begin{split}&\log\mathbb{E}_{\beta\sim\mathbf{B}(\gamma)}p_{\theta}(Y|X,\beta)\\\
\approx&\log[p_{\theta}(Y|X,\gamma)+\mathbb{E}_{\beta\sim\mathbf{B}(\gamma)}(\beta-\gamma)p^{\prime}_{\theta}(Y|X,\gamma)]\\\
=&\log p_{\theta}(Y|X,\gamma)\end{split}$

By moving the expectation into the decoding function, we can deterministically
compute the likelihood by setting $\beta_{i}=\gamma_{i}$, reducing complexity
to $\mathbf{O}(1)$. Each attention weight will first be “soft-masked” by
$\gamma$ before being passed to the decoder. soft-select is fully
differentiable and can be easily trained by gradient descent. However, this
soft-approximation is normally inaccurate, especially when
$\mathbf{B}(\gamma)$ has a high entropy, which is common in one-to-many text
generation tasks. The gap between
$\log\mathbb{E}_{\beta\sim\mathbf{B}(\gamma)}p_{\theta}(Y|X,\beta)$ and $\log
p_{\theta}(Y|X,\mathbb{E}_{\beta\sim\mathbf{B}(\gamma)})$ will be large Ma et
al. (2017); Deng et al. (2018). In practice, this would lead to unrealistic
generations when sampling $\beta$ from the deterministically trained
distribution.

###  3.3 Reinforce-Select

Reinforce-select (RS) Ling and Rush (2017); Chen and Bansal (2018) utilizes
reinforcement learning to approximate the marginal likelihood. Specifically,
it is trained to maximize a lower bound of the likelihood by applying the
Jensen inequalily:

$\log\mathbb{E}_{\beta\sim\mathbf{B}(\gamma)}p_{\theta}(Y|X,\beta)\geq\mathbb{E}_{\beta\sim\mathbf{B}(\gamma)}\log
p_{\theta}(Y|X,\beta)$

The gradient to $\gamma$ is approximated with Monte-Carlo sampling by applying
the REINFORCE algorithm Williams (1992); Glynn (1990). To speed up
convergence, we pre-train the selector by some distant supervision, which is a
common practice in reinforcement learning. REINFORCE is unbiased but has a
high variance. Many research have proposed sophisticated techniques for
variance reduction Mnih and Gregor (2014); Tucker et al. (2017); Grathwohl et
al. (2018). In text generation, the high-variance problem is aggravated
because there exists multiple valid selections. Accurately estimating the
likelihood becomes difficult. Another issue is its tendency to avoid
stochasticity Raiko et al. (2015), which we will show in Sec 5.2 that it
results in low content-level diversity.

###  3.4 Variational Reinforce-Select

We propose Variational Reinforce-Select (VRS) which applies variational
inference Kingma and Welling (2014) for variance reduction. Instead of
directly integrating over $\mathbf{B}(\gamma)$, it imposes a proposal
distribution $q_{\phi}$ for importance sampling. The marginal likelihood is
lower bounded by:

$\begin{split}&\log\mathbb{E}_{\beta\sim\mathbf{B}(\gamma)}p_{\theta}(Y|X,\beta)\\\
&=\log\mathbb{E}_{\beta\sim
q_{\phi}}\frac{p_{\theta}(Y,\beta|X)}{q_{\phi}(\beta)}\\\
&\geq\mathbb{E}_{\beta\sim
q_{\phi}}\log\frac{p_{\theta}(Y,\beta|X)}{q_{\phi}(\beta)}\\\
&=\mathbb{E}_{\beta\sim q_{\phi}}\log
p_{\theta}(Y|X,\beta)-KL(q_{\phi}||\mathbf{B}(\gamma))\end{split}$ (3)

By choosing a proper $q_{\phi}$, the bound will be improved and the variance
can be largely reduced compared with REINFORCE. If $q_{\phi}$ equals the
posterior distribution $p_{\theta}(\beta|X,Y)$, the bound is tight and the
variance would be zero Mnih and Rezende (2016). We define
$q_{\phi}(\beta|X,Y)$ as a mean-field distribution parameterized by a set of
global parameters $\phi$ to approach the true posterior distribution. $\phi$,
$\theta$ and $\gamma$ are simultaneously trained by minimizing the last tine
of Eq. 3. $q_{\phi}(\beta|X,Y)$ also allows us to further perform posterior
inference: Given an arbitrary text $Y$ for a source $X$, we can infer which
source contents are included in $Y$ (An example is given in Appendix C).

In Eq.3, the KL divergence term can be computed analytically. As for the
independence assumption, it can be summed over each individual $\beta_{i}$.
The likelihood term is differentiable to $\theta$ but not to $\phi$, we
estimate the gradient to $\phi$ in Eq 3 by applying the REINFORCE estimator:

$\begin{gathered}\nabla_{\phi}\mathbb{E}_{\beta\sim q_{\phi}}\log
p_{\theta}(Y|X,\beta)=\\\ \mathbb{E}_{\beta\sim q_{\phi}}\nabla_{\phi}\log
q_{\phi}(\beta|X,Y)(\log p_{\theta}(Y|X,\beta)-B)\end{gathered}$

$B$ is the control variate Williams (1992). The optimal $B$ would be Weaver
and Tao (2001):

$B^{\ast}=\mathbb{E}_{\beta\sim q_{\phi}}\log p_{\theta}(Y|X,\beta)$

which we set as a soft-select approximation:

$B=\log p_{\theta}(Y|X,\mathbb{E}_{\beta\sim q_{\phi}}\beta)$

We estimate Eq. 3.4 with a single sample from $q_{\phi}$ for efficiency.
Though multiple-sample could potentially further tighten the bound and reduce
the variance Burda et al. (2016); Lawson et al. (2018); Tucker et al. (2019),
it brings significant computational overhead, especially in text generation
tasks where the whole sentence needs to be decoded.

###  3.5 Degree of Controllability

In practice, when treating content selection as latent variables, the model
tends to end up with a trivial solution of always selecting all source tokens
Shen et al. (2018a); Ke et al. (2018). This behavior is understandable since
Eq. 2 strictly masks unselected tokens. Wrongly unselecting one token will
largely deteriorate the likelihood. Under the maximum likelihood (MLE)
objective, this high risk pushes the selector to take a conservative strategy
of always keeping all tokens, then the whole model degenerates to the standard
Enc-Dec and the selection mask loses effects on the generation. Usually people
apply a penalty term to the selecting ratio when optimizing the likelihood:

$\mathcal{L}+\lambda\mathinner{\\!\left\lvert(\bar{\gamma}-\alpha)\right\rvert}$
(4)

$\mathcal{L}$ is the MLE loss function, $\bar{\gamma}$ is the mean of $\gamma$
and $\alpha$ is the target selecting ratio. This forces the selector to select
the most important $\alpha$ tokens for each source input instead of keeping
all of them.

In our VRS model, we can easily adjust the degree of controllability by
limiting an upper bound of the conditional mutual information (CMI)
$I(\beta,Y|X)$ Zhao et al. (2018). Specifically, we can change our objective
into:

$\begin{split}\max_{\phi,\theta,\gamma}\mathbb{E}_{\beta\sim q_{\phi}}\log
p_{\theta}(Y|X,\beta)\\\ -\lambda\mathinner{\\!\left\lvert
KL(q_{\phi}||\mathbf{B}(\gamma))-\epsilon)\right\rvert}\end{split}$ (5)

$\lambda$ is a fixed lagrangian multiplier. Eq. 5 can be proved equal to
maximum likelihood with the constraint $I(\beta,Y|X)=\epsilon$ given proper
$\lambda$ Alemi et al. (2018). A higher $\epsilon$ indicates $\beta$ has more
influences to $Y$ (higher controllability) while always safely selecting all
tokens will lead $I(\beta,Y|X)=0$.333We also tried adding a coverage
constraint to ensure the decoder covers all the selected tokens Wen et al.
(2015); Wang et al. (2019), but we find it brings no tangible help since a
higher CMI can already discourage including redundant tokens into the
selection. It is preferred over Eq. 4 because (a) CMI directly considers the
dependency between the selection and _multiple_ -possible text while limiting
the ratio aims at finding the _single_ most salient parts for each source. (b)
Unlike CMI, limiting the ratio is coarse. It considers only the total selected
size and ignores its internal distribution.

Algorithm 1 Variational Reinforce-Select (VRS)

Parameters: $\theta,\phi,\gamma$

$pretrain\leftarrow$ TRUE

repeat

Sample X,Y from the corpus;

Encode X into $(h_{1},h_{2},\ldots,h_{n})$;

if $pretrain$ then

Update $\phi$ with distant supervision;Update $\theta,\gamma$ by
$\nabla_{\theta,\gamma}$Eq. 3;

else

Update $\theta,\gamma,\phi$ by $\nabla_{\theta,\gamma,\phi}$Eq. 5;

end if

$pretrain\leftarrow$ FALSE if Eq. 3 degrades

until convergence and $pretrain$ is False

In practice, we can set $\epsilon$ to adjust the degree of controllability we
want. Later we will show it leads to a trade-off with performance. The final
algorithm is detailed in Algorithm 1. To keep fairness, we trian RS and VRS
with the same control variate and pre-training strategy.444The only extra
parameter of VRS is $\phi$ which is a simple MLP structure. The actual
training complexity is similar to RS because they both use the REINFORCE
algorithm for gradient estimation.

##  4 Related Work

Most content selection models train the selector with heuristic rules Hsu et
al. (2018); Li et al. (2018); Yu et al. (2018); Gehrmann et al. (2018); Yao et
al. (2019); Moryossef et al. (2019), which fail to fully capture the relation
between selection and generation. Mei et al. (2016); Zhou et al. (2017); Lin
et al. (2018); Li et al. (2018) “soft-select” word or sentence embeddings
based on a gating function. The output score from the gate is a deterministic
vector without any probabilistic variations, so controlling the selection to
generate diverse text is impossible. Very few works explicitly define a
bernoulli distribution for the selector, then train with the REINFORCE
algorithm Ling and Rush (2017); Chen and Bansal (2018), but the selection
targets at a high recall regardless of the low precision, so the
controllability over generated text is weak. Fan et al. (2018) control the
generation by manually concatenating entity embeddings, while our model is
much more flexible by explicitly defining the selection probability over all
source tokens.

Our work is closely related with learning discrete representations with
variational inference Wen et al. (2017); van den Oord et al. (2017); Kaiser et
al. (2018); Lawson et al. (2018), where we treat content selection as the
latent representation. Limiting the KL-term is a common technique to deal with
the “posterior collapse” problem Kingma et al. (2016); Yang et al. (2017);
Shen et al. (2018b). We adopt a similar approach and use it to further control
the selecting strategy.

##  5 Experiments

For the experiments, we focus on comparing (1) Bottom-up generation (Bo.Up.),
(2) soft-select (SS), (3) Reinforce-select (RS) and (4) Variational-Reinforce-
select (VRS) regarding their performance on content selection. SS and RS are
trained with the selecting ratio constraint in Eq. 4. For the SS model, we
further add a regularization term to encourage the maximum value of $\gamma$
to be close to 1 as in Mei et al. (2016). We first briefly introduce the tasks
and important setup, then present the evaluation results.

###  5.1 Tasks and Setup

We test content-selection models on the headline and data-to-text generation
task. Both tasks share the same framework with the only difference of source-
side encoders.

Headline Generation: We use English Gigaword preprocessed by Rush et al.
(2015), which pairs first sentences of news articles with their headlines. We
keep most settings same as in Zhou et al. (2017), but use a vocabulary built
by byte-pair-encoding Sennrich et al. (2016). We find it speeds up training
with superior performance.

Data-to-Text Generation: We use the Wikibio dataset Lebret et al. (2016). The
source is a Wikipedia infobox and the target is a one-sentence biography
description. Most settings are the same as in Liu et al. (2018), but we use a
bi-LSTM encoder for better performance.

Heuristically extracted content: This is used to train the selector for bottom
up models and pre-train the RS and VRS model. For wikibio, we simply extract
overlapped words between the source and target. In Gigaword, as the headline
is more abstractive, we select the closest source word for each target word in
the embedding space. Stop words and punctuations are prohibited from being
selected.

Gigaword | Oracle upper bound of | % unique | % unique | % Effect of | Entropy of
---|---|---|---|---|---
ROUGE-1 | ROUGE-2 | ROUGE-L | Generation | Mask | Selector | Selector
Bo.Up. | 42.61 | 22.32 | 38.37 | 84.28 | 95.87 | 87.91 | 0.360
SS | 33.15 | 14.63 | 30.68 | 82.06 | 96.23 | 85.27 | 0.392
RS | 36.62 | 18.34 | 34.60 | 3.01 | 6.23 | 48.31 | 0.018
VRS | 54.73 | 33.28 | 51.62 | 89.23 | 92.51 | 96.45 | 0.288
Wikibio | ROUGE-4 | BLEU-4 | NIST | Generation | Mask | Selector | Selector
Bo.Up. | 47.28 | 49.95 | 11.06 | 31.57 | 77.42 | 40.78 | 0.177
SS | 41.73 | 43.94 | 9.82 | 63.09 | 89.42 | 70.55 | 0.355
RS | 44.07 | 46.89 | 10.31 | 4.55 | 43.83 | 10.38 | 0.105
VRS | 52.41 | 55.03 | 11.89 | 57.62 | 77.83 | 74.03 | 0.181
Table 2: Diversity of content selection. The % effect of selector is defined
as the ratio of unique generation and mask, which reflects the rate that
changing the selector will lead to corresponding changes of the generated
text.

Choice of $\alpha/\epsilon$: As seen in Sec 3.5, we need to set the
hyperparameter $\alpha$ for RS/SS and $\epsilon$ for VRS. $\alpha$ corresponds
to the selecting ratio. We set them as $\alpha=0.35$ for Wikibio and $0.25$
for Gigaword. The value is decided by running a human evaluation to get the
empirical estimation. To keep comparison fairness, we tune $\epsilon$ to make
VRS select similar amount of tokens with RS. The values we get are
$\epsilon=0.15n$ for Wikibio and $\epsilon=0.25n$ for Gigaword. $n$ is the
number of source tokens. 555$\epsilon$ corresponds to the KL divergence of the
selection mask, which scales linearly with the number of source tokens, so we
set it proportinally w.r.t. $n$.

###  5.2 Results and Analysis

Ideally we would expect the learned content selector to (1) have reasonable
diversity so that text with various contents can be easily sampled, (2)
properly control the contents described in the generated text and (3) not hurt
performance. The following section will evaluate these three points in order.

Diversity: We first look into the diversity of content selection learned by
different models. For each test data, 50 selection masks are randomly sampled
from the model’s learned distribution. Greedy decoding is run to generate the
text for each mask. We measure the entropy of the selector, proportion of
unique selection masks and generated text in the 50 samples. We further define
the “effect” of the selector as the ratio of sampled unique text and mask.
This indicates how often changing the selection mask will also lead to a
change in the generated text. The results are averaged over all test data.
Following Rush et al. (2015) and Lebret et al. (2016), we measure the quality
of generated text with ROUGE-1, 2, L F-score for Gigaword and ROUGE-4, BLEU-4,
NIST for Wikibio. As there is only one reference text for each source, we
report an oracle upper bound of these scores by assuming an “oracle” that can
choose the best text among all the candidates Mao et al. (2015); Wang et al.
(2017). Namely, out of each 50 sampled text, we pick the one with the maximum
metric score. The final metric score is evaluated on these “oracle” picked
samples. The intuition is that if the content selector is properly trained, at
least one out of the 50 samples should describe similar contents with the
reference text, the metric score between it and the reference text should be
high. Table 2 lists the results. We can have the following observations:

  * •

RS model completely fails to capture the content-level diversity. Its selector
is largely deterministic, with a lowest entropy value among all models. In
contrast, the selector from SS, VRS and Bo.Up. have reasonable diversity, with
over $90\%$ and $75\%$ unique selection masks for Gigaword and Wikibio
respectively.

  * •

The selector from VRS has the strongest effect to the generator, especially on
the Gigaword data where modifying the content selection changes the
corresponding text in more than 95% of the cases. RS has the lowest effect
value, which indicates that even with the selecting ratio constraint, its
generator still ignores the selection mask to a large extent.

  * •

The oracle metric score of VRS is much higher than the other two. This is
beneficial when people want to apply the model to generate a few candidate
text then hand-pick the suitable one. VRS has more potential than the other
three to contain the expected text. SS performs worst. The gap between the
soft approximation and the real distribution, as mentioned before, indeed
results in a large drop of performance.

In short, compared with others, the content selector of VRS is _(1) diverse,
(2) has stronger effect on the text generation and (3) with a larger potential
of producing an expected text_.

Controllability: We have shown the content selector of VRS is diverse and has
strong effect on the text generation. This section aims at examining whether
such effect is desirable, i.e., whether the selector is able to properly
control the contents described in the text. We measure it based on the self-
bleu metric and a human evaluation.

The self-bleu metric measures the controllability by evaluating the “intra-
selection” similarity of generated text. Intuitively, by fixing the selection
mask, multiple text sampled from the decoder are expected to describe the same
contents and thereby should be highly similar to each other. The decoder
should only model surface-level diversity without further modifying the
selected contents. With this in mind, for each test data, we randomly sample a
selection mask from the selector’s distribution, then fix the mask and run the
decoder to sample 10 different text. The self-BLEU-1 score Zhu et al. (2018)
on the sampled text is reported, which is the average BLEU score between each
text pair. A higher self-BLEU score indicates the sampled text are more
similar with each other. The results are shown in Table 3. We can see
generations from VRS have a clearly higher intra-selection similarity. SS
performs even worse than RS, despite having a high effect score in Table 2.
The selector from SS affects the generation in an undesirable way, which also
explain why SS has a lowest oracle metric score though with a high score on
content diversity and effect.

Method | Bo.Up. | SS | RS | VRS
---|---|---|---|---
Gigaword | 46.58 | 37.20 | 48.13 | 61.14
Wikibio | 38.30 | 13.92 | 25.99 | 43.81
Table 3: Self-Bleu score by fixing selection mask. Higher means better controllability of content selection Method | Fluency | intra-consistency | inter-diversity
---|---|---|---
Reference | 0.96 | - | -
Enc-Dec | 0.83 | - | -
Bo.Up. | 0.46 | 0.48 | 0.61
SS | 0.27 | 0.41 | 0.54
RS | 0.78 | 0.39 | 0.47
VRS | 0.74 | 0.72 | 0.87
Table 4: Human evaluation on fluency, intra-consistency and inter-diversity of
content selection on DUC 2004.

We further run a human evaluation to measure the text-content consistency
among different models. 100 source text are randomly sampled from the human-
written DUC 2004 data for task 1&2 Over et al. (2007). Bo.Up, SS, RS and VRS
are applied to generate the target text by first sampling a selection mask,
then run beam search decoding with beam size 10. We are interested in seeing
(1) if multiple generations from the same selection mask are paraphrases to
each other (intra-consistent) and (2) if generations from different selection
masks do differ in the content they described (inter-diverse). The results in
Table 4 show that VRS significantly outperforms the other two in both intra-
consistency and inter-diversity. RS has the lowest score on both because the
selector has very weak effects on the generation as measured in the last
section. Bo.Up and SS lay between them. Overall VRS is able to _maintain the
highest content-text consistency_ among them.

Method | R-1 | R-2 | R-L | %Word
---|---|---|---|---
Zhou et al. (2017) | 36.15 | 17.54 | 33.63 | 100
Enc-Dec | 35.92 | 17.43 | 33.42 | 100
SS | 20.35 | 4.78 | 16.53 | 24.82
Bo.Up | 28.17 | 10.32 | 26.68 | 24.54
RS | 35.45 | 16.38 | 32.71 | 25.12
VRS($\epsilon=0$)-pri | 36.42 | 17.81 | 33.86 | 78.63
VRS($\epsilon=0.25$)-pri | 34.26 | 15.11 | 31.69 | 24.36
VRS($\epsilon=0$)-post | 37.14 | 18.03 | 34.26 | 78.66
VRS($\epsilon=0.25$)-post | 56.72 | 33.24 | 51.88 | 24.53
Table 5: Gigaword best-select results. Larger $\epsilon$ leads to more controllable selector with a bit degrade of performance. (-post) means selecting from the posterior $q_{\phi}(\beta|X,Y)$, (-pri) is from the prior $\mathbf{B}(\gamma_{i})$. Method | R-4 | B-4 | NIST | %Word
---|---|---|---|---
Liu et al. (2018) | 41.65 | 44.71 |  | 100
Enc-Dec | 42.07 | 44.80 | 9.82 | 100
SS | 5.10 | 5.73 | 0.24 | 35.12
Bo.Up | 8.07 | 9.52 | 0.42 | 38.79
RS | 42.64 | 45.08 | 10.01 | 34.53
VRS($\epsilon=0$)-pri | 43.01 | 46.01 | 10.24 | 84.56
VRS($\epsilon=0.15$)-pri | 42.13 | 44.51 | 9.84 | 34.04
VRS($\epsilon=0$)-post | 43.84 | 46.60 | 10.27 | 85.34
VRS($\epsilon=0.15$)-post | 49.68 | 52.26 | 11.48 | 34.57
Table 6: Wikibio best-select results.

Performance $\&$ Trade-off: To see if the selector affects performance, we
also ask human annotators to judge the text fluency. The fluency score is
computed as the average number of text being judged as fluent. We include
generations from the standard Enc-Dec model. Table 4 shows the best fluency is
achieved for Enc-Dec. Imposing a content selector always affects the fluency a
bit. The main reason is that when the controllability is strong, the change of
selection will directly affect the text realization so that a tiny error of
content selection might lead to unrealistic text. If the selector is not
perfectly trained, the fluency will inevitably be influenced. When the
controllability is weaker, like in RS, the fluency is more stable because it
will not be affected much by the selection mask. For SS and Bo.Up, the drop of
fluency is significant because of the gap of soft approximation and the
independent training procedure. In general, VRS does properly decouple content
selection from the enc-dec architecture, with only tiny degrade on the
fluency.

Table 5/6 further measure the metric scores on Gigaword/Wikibio by decoding
text from the best selection mask based on the selector’s distribution (set
$\beta_{i}=1$ if $\mathbf{B}(\gamma_{i})>0.5$ and $0$ otherwise). We include
results from VRS model with $\epsilon=0$, which puts no constraint on the
mutual information. We further report the score by generating the best
selection mask from the learned posterior distribution $q_{\phi}(\beta|X,Y)$
for VRS model. Two current SOTA results from Zhou et al. (2017) and Liu et al.
(2018) and the proportion of selected source words for each model are also
included. We have the following observations:

  * •

As the value of $\epsilon$ decreases, the performance of VRS improves, but the
selector loses more controllability because the model tends to over-select
contents (over $75\%$ source words selected). The text-content consistency
will become low.

  * •

Increasing $\epsilon$ sacrifices a bit performance, but still comparable with
SOTA. Especially on Wikibio where the performance drop is minor. The reason
should be that Wikibio is relatively easier to predict the selection but
Gigaword has more uncertainty.

  * •

Increasing $\epsilon$ improves the accuracy of the posterior selection. This
would be useful when we want to perform posterior inference for some source-
target pair.

  * •

Setting $\epsilon=0$ can actually outperform SOTA seq2seq which keeps all
tokens, suggesting it is still beneficial to use the VRS model even if we do
not care about the controllability.

Figure 2: Negative log likelihood (NLL), selection entropy and self-BLEU as
$\epsilon$ changes. NLL and self-bleu on Wikibio are added by 1 for better
visualization. Lower NLL suggests higher performance. Higher entropy/self-BLEU
means higher diversity/controllability.

Figure 2 visualizes how changing the value of $\epsilon$ affects the negative
log likelihood (NLL), entropy of the selector and self-bleu score, which
roughly correlates with performance, diversity and controllability. NLL is
evaluated based on the lower bound in Eq 3 Sohn et al. (2015). We can see as
$\epsilon$ increases, the performance decreases gradually but the content
selection gains more diversity and controllability. In practice we can tune
the $\epsilon$ value to achieve a trade-off.

Source: indian prime minister p.v. narasimha rao ’s promise of more autonomy
for troubled kashmir and his plea for early state elections has sparked a
violent reaction from provincial moslem and opposition parties .
Samples from SS:
t1: indian indian calls for end to violence in kashmir .
t2: indian pm calls for end to violence in afghanistan .
t3: indian pm calls for boycott of pakistan ’s ruling party .
Samples from Bo.Up:
t1: india promises more autonomous more autonomy .
t2: indian pm promises autonomy for kashmir autonomy .
t3: indian pm ’s promise sparks violent reaction .
Samples from RS:
t1: indian pm ’s kashmir promises sparks violent reaction.
t2: indian pm ’s promise sparks violent reaction .
t3: kashmir parties blast pm ’s promise .
Samples from VRS:
t1: indian pm ’s promise on kashmir sparks uproar .
t2: indian pm ’s promise on kashmir sparks protests .
t3: indian pm ’s promise for kashmir sparks controversy .
Source: factory orders for manufactured goods rose #.# percent in september ,
the commerce department said here thursday .
Samples from SS:
t1: u.s. consumer confidence down in january in january.
t2: u.s. wholesale prices up #.# percent in october .
t3: u.s. jobless rate rises to #.# percent in march .
Samples from Bo.Up.:
t1: september u.s. factory orders up #.# percent .
t2: september u.s. factory orders increase .
t3: factory orders up in september .
Samples from RS:
t1: u.s. factory orders up #.# percent in september .
t2: factory orders for manufactured goods rise .
t3: factory orders up in september from the year .
Samples from VRS:
t1: september factory orders up #.# percent .
t2: september factory orders rise #.# percent .
t3: september factory orders increase #.# pct .

Figure 3: Text generation examples from Gigaword. Highlighted words are
selected. t1-3 are sampled from the decoder based on the selected content.
Generations from VRS are more faithful to selected contents.

Generation Example: Figure 3 shows some examples from Gigaword. As can be
seen, decodings from the VRS model are largely consistent with each other, in
most cases only replacing one or two words with corresponding synonyms.
Samples are able to faithfully convey all selected contents. In contrast,
generations from SS. Bo.Up. and RS are unpreditable, differing in both
selected contents and also the way of saying. SS and Bo.Up also suffer more
from the text disfluency. The generations from them are largely uncertain.

##  6 Conclusion

In this paper, we tackle the unaddressed problem of controllable content
selection in text generation. We propose a general framework based on
variational inference that can be potentiall applied to arbitrary tasks. On
both the headline generation and data-to-text tasks, our model outperforms
state-of-the-art models regarding the diversity and controllability of content
selection. We further introduce an effective way to achieve a
performance/controllability trade-off, which can be easily tuned to meet
specific requirement.

## Acknowledgments

We thank anonymous reviewers for valuable comments, thank Aditya Mogadala,
Shun Kiyono, Thomas Mclachlan and other members of the LIAT team at RIKEN AIP
for useful discussions. Xiaoyu Shen is supported by IMPRS-CS fellowship. The
work of J. Suzuki was partly supported by JSPS KAKENHI Grant Number JP19104418
and AIRPF Grant Number 30AI036-8. This work is also partially funded by DFG
collaborative research center SFB 1102.

## References

  * Alemi et al. (2018) Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A. Saurous, and Kevin Murphy. 2018.  Fixing a broken ELBO.  In _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 159–168, StockholmsmÃ€ssan, Stockholm Sweden. PMLR.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _International Conference on Learning Representations_.
  * Burda et al. (2016) Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. 2016.  Importance weighted autoencoders.  In _International Conference on Learning Representations_.
  * Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018.  Fast abstractive summarization with reinforce-selected sentence rewriting.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 675–686. Association for Computational Linguistics.
  * Cornia et al. (2018) Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2018.  Show, control and tell: A framework for generating controllable and grounded captions.  _arXiv preprint arXiv:1811.10652_.
  * Deng et al. (2018) Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander M Rush. 2018.  Latent alignment and variational attention.  _Nips_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _NAACL_.
  * Fan et al. (2018) Angela Fan, David Grangier, and Michael Auli. 2018.  Controllable abstractive summarization.  In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_ , pages 45–54. Association for Computational Linguistics.
  * Gehrmann et al. (2018) Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018.  Bottom-up abstractive summarization.  _EMNLP_.
  * Glynn (1990) Peter W. Glynn. 1990.  Likelihood ratio gradient estimation for stochastic systems.  _Commun. ACM_ , 33(10):75–84.
  * Grathwohl et al. (2018) Will Grathwohl, Dami Choi, Yuhuai Wu, Geoff Roeder, and David Duvenaud. 2018.  Backpropagation through the void: Optimizing control variates for black-box gradient estimation.  In _International Conference on Learning Representations_.
  * Haury et al. (2011) Anne-Claire Haury, Pierre Gestraud, and Jean-Philippe Vert. 2011.  The influence of feature selection methods on accuracy, stability and interpretability of molecular signatures.  _PloS one_ , 6(12):e28210.
  * Hsu et al. (2018) Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018\.  A unified model for extractive and abstractive summarization using inconsistency loss.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 132–141. Association for Computational Linguistics.
  * Jackson (1998) Peter Jackson. 1998.  _Introduction to expert systems_.  Addison-Wesley Longman Publishing Co., Inc.
  * Kaiser et al. (2018) Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. 2018.  Fast decoding in sequence models using discrete latent variables.  In _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 2390–2399, StockholmsmÃ€ssan, Stockholm Sweden. PMLR.
  * Ke et al. (2018) Nan Rosemary Ke, Konrad Żołna, Alessandro Sordoni, Zhouhan Lin, Adam Trischler, Yoshua Bengio, Joelle Pineau, Laurent Charlin, and Christopher Pal. 2018.  Focused hierarchical RNNs for conditional sequence processing.  In _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 2554–2563, StockholmsmÃ€ssan, Stockholm Sweden. PMLR.
  * Kingma et al. (2016) Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016.  Improved variational inference with inverse autoregressive flow.  In _Advances in Neural Information Processing Systems_ , pages 4743–4751.
  * Kingma and Welling (2014) Diederik P Kingma and Max Welling. 2014.  Auto-encoding variational bayes.  In _International Conference on Learning Representations_.
  * Lawson et al. (2018) Dieterich Lawson, Chung-Cheng Chiu, George Tucker, Colin Raffel, Kevin Swersky, and Navdeep Jaitly. 2018.  Learning hard alignments with variational inference.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5799–5803. IEEE.
  * Lebret et al. (2016) Rémi Lebret, David Grangier, and Michael Auli. 2016.  Neural text generation from structured data with application to the biography domain.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1203–1213. Association for Computational Linguistics.
  * Lei et al. (2016) Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016.  Rationalizing neural predictions.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 107–117.
  * Li et al. (2018) Wei Li, Xinyan Xiao, Yajuan Lyu, and Yuanzhuo Wang. 2018.  Improving neural abstractive document summarization with explicit information selection modeling.  _EMNLP_.
  * Lin et al. (2018) Junyang Lin, Xu SUN, Shuming Ma, and Qi Su. 2018.  Global encoding for abstractive summarization.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 163–169. Association for Computational Linguistics.
  * Ling and Rush (2017) Jeffrey Ling and Alexander Rush. 2017.  Coarse-to-fine attention models for document summarization.  In _Proceedings of the Workshop on New Frontiers in Summarization_ , pages 33–42.
  * Liu et al. (2018) Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. 2018.  Building end-to-end dialogue systems using generative hierarchical neural network models.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_ , pages 4881–4888. AAAI Press.
  * Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015.  Effective approaches to attention-based neural machine translation.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1412–1421. Association for Computational Linguistics.
  * Ma et al. (2017) Xuezhe Ma, Yingkai Gao, Zhiting Hu, Yaoliang Yu, Yuntian Deng, and Eduard Hovy. 2017\.  Dropout with expectation-linear regularization.  In _International Conference on Learning Representations_.
  * Mao et al. (2015) Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. 2015.  Deep captioning with multimodal recurrent neural networks (m-rnn).  _ICLR_.
  * Mei et al. (2016) Hongyuan Mei, TTI UChicago, Mohit Bansal, and Matthew R Walter. 2016.  What to talk about and how? selective generation using lstms with coarse-to-fine alignment.  In _Proceedings of NAACL-HLT_ , pages 720–730.
  * Mnih and Gregor (2014) Andriy Mnih and Karol Gregor. 2014.  Neural variational inference and learning in belief networks.  In _Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32_ , ICML’14, pages II–1791–II–1799. JMLR.org.
  * Mnih and Rezende (2016) Andriy Mnih and Danilo J. Rezende. 2016.  Variational inference for monte carlo objectives.  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ICML’16, pages 2188–2196. JMLR.org.
  * Moryossef et al. (2019) Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019.  Step-by-step: Separating planning from realization in neural data-to-text generation.  _NAACL_.
  * Nallapati et al. (2017) Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017.  Summarunner: A recurrent neural network based sequence model for extractive summarization of documents.  In _AAAI_.
  * Nema et al. (2017) Preksha Nema, Mitesh M Khapra, Anirban Laha, and Balaraman Ravindran. 2017.  Diversity driven attention model for query-based abstractive summarization.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 1063–1072.
  * van den Oord et al. (2017) Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. 2017.  Neural discrete representation learning.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 6306–6315. Curran Associates, Inc.
  * Over et al. (2007) Paul Over, Hoa Dang, and Donna Harman. 2007.  Duc in context.  _Information Processing & Management_, 43(6):1506–1520.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Prabhumoye et al. (2019) Shrimai Prabhumoye, Chris Quirk, and Michel Galley. 2019.  Towards content transfer through grounded text generation.  _NAACL_.
  * Qin et al. (2018) Guanghui Qin, Jin-Ge Yao, Xuening Wang, Jinpeng Wang, and Chin-Yew Lin. 2018.  Learning latent semantic annotations for grounding natural language to structured data.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3761–3771.
  * Raiko et al. (2015) Tapani Raiko, Mathias Berglund, Guillaume Alain, and Laurent Dinh. 2015.  Techniques for learning binary stochastic feedforward neural networks.  _ICLR_.
  * Reiter and Dale (2000) Ehud Reiter and Robert Dale. 2000.  _Building natural language generation systems_.  Cambridge university press.
  * Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 379–389. Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 1715–1725.
  * Shen et al. (2018a) Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Sen Wang, and Chengqi Zhang. 2018a.  Reinforced self-attention network: a hybrid of hard and soft attention for sequence modeling.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_ , pages 4345–4352. International Joint Conferences on Artificial Intelligence Organization.
  * Shen et al. (2018b) Xiaoyu Shen, Hui Su, Shuzi Niu, and Vera Demberg. 2018b.  Improving variational encoder-decoders in dialogue generation.  In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Sohn et al. (2015) Kihyuk Sohn, Honglak Lee, and Xinchen Yan. 2015.  Learning structured output representation using deep conditional generative models.  In _Advances in neural information processing systems_ , pages 3483–3491.
  * Stent et al. (2004) Amanda Stent, Rashmi Prasad, and Marilyn Walker. 2004.  Trainable sentence planning for complex information presentation in spoken dialog systems.  In _Proceedings of the 42nd annual meeting on association for computational linguistics_ , page 79. Association for Computational Linguistics.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 27_ , pages 3104–3112. Curran Associates, Inc.
  * Tucker et al. (2019) George Tucker, Dieterich Lawson, Shixiang Gu, and Chris J Maddison. 2019.  Doubly reparameterized gradient estimators for monte carlo objectives.  In _International Conference on Learning Representations_.
  * Tucker et al. (2017) George Tucker, Andriy Mnih, Chris J Maddison, John Lawson, and Jascha Sohl-Dickstein. 2017.  Rebar: Low-variance, unbiased gradient estimates for discrete latent variable models.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 2627–2636. Curran Associates, Inc.
  * Wang et al. (2017) Liwei Wang, Alexander Schwing, and Svetlana Lazebnik. 2017.  Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space.  In _Advances in Neural Information Processing Systems_ , pages 5756–5766.
  * Wang et al. (2019) Wentao Wang, Zhiting Hu, Zichao Yang, Haoran Shi, Frank Xu, and Eric Xing. 2019\.  Toward unsupervised text content manipulation.  _arXiv preprint arXiv:1901.09501_.
  * Weaver and Tao (2001) Lex Weaver and Nigel Tao. 2001.  The optimal reward baseline for gradient-based reinforcement learning.  In _Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence_ , UAI’01, pages 538–545, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
  * Wen et al. (2015) Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015.  Semantically conditioned lstm-based natural language generation for spoken dialogue systems.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1711–1721.
  * Wen et al. (2017) Tsung-Hsien Wen, Yishu Miao, Phil Blunsom, and Steve Young. 2017.  Latent intention dialogue models.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 3732–3741, International Convention Centre, Sydney, Australia. PMLR.
  * Williams (1992) Ronald J. Williams. 1992.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine Learning_ , 8(3):229–256.
  * Wiseman et al. (2018) Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2018.  Learning neural templates for text generation.  _EMNLP_.
  * Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015.  Show, attend and tell: Neural image caption generation with visual attention.  In _International conference on machine learning_ , pages 2048–2057.
  * Yang et al. (2017) Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017\.  Improved variational autoencoders for text modeling using dilated convolutions.  _ICML_.
  * Yao et al. (2019) Lili Yao, Nanyun Peng, Weischedel Ralph, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019.  Plan-and-write: Towards better automatic storytelling.  _AAAI_.
  * Yu et al. (2018) Naitong Yu, Jie Zhang, Minlie Huang, and Xiaoyan Zhu. 2018.  An operation network for abstractive sentence compression.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1065–1076. Association for Computational Linguistics.
  * Zhao et al. (2018) Shengjia Zhao, Jiaming Song, and Stefano Ermon. 2018.  The information autoencoding family: A lagrangian perspective on latent variable generative models.  _UAI_.
  * Zhou et al. (2017) Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2017.  Selective encoding for abstractive sentence summarization.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1095–1104. Association for Computational Linguistics.
  * Zhu et al. (2018) Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018.  Texygen: a benchmarking platform for text generation models.  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, pages 1097–1100. ACM.

##  Appendix A Performance/Controllability trade-off

The trade-off between performance and interpretability has been a long-
standing problem in feature selection Jackson (1998); Haury et al. (2011). The
trade-off exists because it is usually very difficult to accurately find the
exact features needed to make the prediction. Safely keeping more features
will almost always lead to better performance. Some models do succeed in
achieving superior performance by selecting only a subset of the input.
However, they mostly still target at the recall of the selection Hsu et al.
(2018); Chen and Bansal (2018); Shen et al. (2018a), i.e., to select all
possible content that might help predict the target. The final selected
contents reduce some most useful information from the source, but they still
contain many redundant contents (same like our VRS-($\epsilon=0$) as in Table
6 and 5). This makes them unsuitable for controllable content selection. In
text generation, a recent work from Moryossef et al. (2019) shows they could
control the contents by integrating a symbolic selector into the neural
network. However, their selector is tailored by some rules only for the RDF
triples. Moreover, even based on their fine-tuned selector, the fluency they
observe is still slightly worse than a standard seq2seq.

We assume the content selector is the major bottle if we want a model that can
achieve controllability without sacrificing the performance. We can clearly
observe in Table 6 that the performance drop in Wikibio is marginal compared
with Gigaword. The reason should be that the selection on Wikibio is much
easier than Gigaword. The biography of a person almost always follow some
simple patterns, like name, birthday and profession, but for news headlines,
it can contain information with various focuses. In our two tasks, due to the
independence assumption we made on $\beta_{i}$ and the model capacity limit,
the content selector cannot fully fit the true selecting distribution, so the
trade-off is necessary. Improving the selector with SOTA sequence labelling
models like Bert Devlin et al. (2019) would be worth trying.

There are also other ways to improve. For example, we could learn a ranker to
help us choose the best contents Stent et al. (2004). Or we could manually
define some matching rules to help rank the selection Cornia et al. (2018). In
Table 2, we show the VRS model achieves very high metric scores based on an
oracle ranker, so learning a ranker should be able to improve the performance
straightforwardly.

Source: The sri lankan government on Wednesday announced the closure of
government schools with immediate effect as a military campain against tamil
separatists escalated in the north of the country.
Reference: sri lanka closes schools as war escalates .
b1: sri lanka shuts schools as war escalates .
b2: sri lanka closes schools as violence escalates .
b3: sri lanka shuts schools as fighting escalates .
b4: sri lanka closes schools as offensive expands .
b5: sri lanka closes schools as war continues .

Figure 4: Posterior inference example. Highlighted words are selected contents
according to the posterior distribution $q_{\phi}(\beta|X,Y)$. b1-b5 are
decoded by fixing the selected contents.

##  Appendix B Example from Wikibio

Figure 5: Example of content selection in Wikibio. Summary 4 in unnatural
because we force to remove the birthday content which the selector assigns
very high probability to select.

To see how we can manually control the content selection, Figure 5 shows an
example from Wikibio, the model is mostly able to form a proper sentence
covering all selected information. If the selector assigns very high
probability to select some content and we force to remove it, the resulting
text could be unnatual (as in summary 4 in Figure 5 because the model has seen
very few text without containing the birthday information in the training
corpus). However, thanks to the diversity of the content selector as shown in
the previous section, it is able to handle most combinatorial patterns of
content selection.

##  Appendix C Posterior inference

Figure 4 further provides an example of how we can perform posterior inference
given a provided text. Our model is able to infer which source contents are
covered in the given summary. With the inferred selection, we can sample
multiple paraphrases describing the same contents. As seen in Table 6 and 5,
the metric scores are remarkably high when decoding from the posterior
inferred selections (last three rows), suggesting the posterior distribution
is well trained. The posterior inference part could be beneficial for other
tasks like content transfer among text Wang et al. (2019); Prabhumoye et al.
(2019). The described source contents can be first predicted with the
posterior inference, then transferred to a new text.

|  Gigaword |  Wikibio
---|---|---
Vocabulary |  Built with byte-pair segmentation with size 30k |  Built by keeping the most frequent 20k tokens
Word Embedding | Size 300. Initialized with Glove Pennington et al. (2014). OOVs are randomly initialized from a normal distribution
Inputs |  Sequence of source word embeddings |  Sequence of concatenation of source table field, value, position and reverse position embeddings Lebret et al. (2016)
Source Encoder |  Single-layer Bi-LSTM with hidden size 512 |  Single-layer Bi-LSTM with hidden size 500
Target Decoder |  Single-layer LSTM with hidden size 512 |  Single-layer LSTM with hidden size 500
Drop out rate | 0.3 for both encoder and decoder
Decoding Method |  Beam search with beam size 5 |  Greedy decoding. UNK words are replaced with the most attended source token as in Liu et al. (2018)
Mini-batch size |  256 |  128
Optimizer | Adam, $\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=10^{-8},\text{weight decay}=1.2\times 10^{-6}$, gradient clipping in [-5,5]
Initial Learning Rate | 0.0005
Prior Selector |  $\mathbf{B}(\gamma_{i})=\sigma(\textmd{MLP}(h_{i}))$. MLP is a multi-layer perceptron. $h_{i}$ comes from the encoder hidden state
Posterior Selector |  $q_{\phi}(\beta_{i}|X,Y)=\sigma(\textmd{MLP}([h_{i}\circ e(y)]))$. $\circ$ means concatenation
Table 7: Detailed settings of our experiment. $e(y)$ in the posterior selector
is an encoded representation of the ground-truth text. We use a bi-LSTM
encoder. The last hidden state is treated as the representation for the text.