# Radiation Tolerance of 65 nm CMOS Transistors

M. Krohna B. Bentelea D.C. Christianb Corresponding author J.P. Cumalata G.
Deptuchb F. Fahimb J. Hoffb A. Shenaib S.R. Wagnera
aDepartment of Physics  University of Colorado
Boulder  Colorado 80309-0390  U.S.A.
bFermi National Accelerator Laboratory
Batavia  IL 60510  U.S.A.
E-mail<EMAIL_ADDRESS>
###### Abstract

We report on the effects of ionizing radiation on 65 nm CMOS transistors held
at approximately $-20\,^{\circ}$C during irradiation. The pattern of damage
observed after a total dose of 1 Grad is similar to damage reported in room
temperature exposures, but we observe less damage than was observed at room
temperature.

###### keywords:

Radiation-hard electronics; Front-end electronics for detector readout

##  1 Introduction

The need for extremely radiation tolerant electronics is one of the major
issues confronting high energy physics in the era of High Luminosity running
at the CERN [2] Large Hadron Collider (HL-LHC). Tests by Bonacini, et al. [3]
at CERN, published in 2012, established 65 nm CMOS as the leading candidate
technology for HL-LHC electronics. Using an X-ray beam, Bonacini, et al.
exposed 65 nm transistors to a total dose of 200 Mrad. Their results showed,
with one exception, relatively small changes in transistor parameters for
normal layout standard gate oxide thickness (core) transistors. The exception
was a dramatic loss of maximum drain-source current in the narrowest PMOS
transistors. The CERN group concluded that 65 nm CMOS technology could be used
for HL-LHC applications with no special design considerations, except that all
core devices should have width greater than 360 nm.

The RD53 collaboration was formed in 2014 to further explore the feasibility
of using 65 nm CMOS technology to design a pixel readout chip for use at the
HL-LHC [4]. The group established a total ionizing dose tolerance goal of 1
Grad. The measurements reported in this paper were done in the context of
RD53. Discussions late in 2013 within RD53 centered on the fact that the data
presented in reference [3], and also subsequent data collected by the CERN
group and by a group from CPPM [5], contain evidence of significant room
temperature annealing during the time between X-ray exposures. Both CMS and
ATLAS currently plan to operate their HL-LHC pixel vertex detectors at
approximately $-20\,^{\circ}\mathrm{C}$. This choice is because the silicon
strip trackers will operate at $-20\,^{\circ}\mathrm{C}$ in order to limit
leakage current in the silicon sensors, which would otherwise require much
more cooling and therefore more mass in the tracking volume. Concern was
expressed that because of reduced annealing, 65 nm circuits might experience
greater radiation damage than had been observed in room temperature exposures
if the circuits were maintained at $-20\,^{\circ}\mathrm{C}$ during
irradiation.

We report the results of an irradiation of 65 nm transistors performed using
the Gamma Irradiation Facility[6] at Sandia National Laboratories [7]. The
devices under test were maintained at a temperature
$\lesssim-20\,^{\circ}\mathrm{C}$ during irradiation.

##  2 Apparatus and Technique

###  2.1 Test ASIC

A 65 nm CMOS Application Specific Integrated Circuit (ASIC) containing
individual transistors connected to wire bond pads was designed at Fermilab
and fabricated by the Taiwan Semiconductor Manufacturing Company
(TSMC)[8].111Our test chip was fabricated at TSMC fab 14; the devices tested
earlier at CERN were fabricated at TSMC fab 12[10]. The test ASIC was part of
a multi-project wafer submitted to TSMC through the Metal Oxide Semiconductor
Implementation Service (MOSIS)[9]. The chip was divided into two parts, one
part intended primarily for lifetime studies of devices operated at liquid
argon temperature, and one part intended for radiation tolerance testing.
Transistors intended for radiation tolerance testing were laid out in groups
of similar transistors (for instance, NMOS transistors with channel length
$L=60$ nm and width W from 120 nm to 1000 nm). Within a group, all transistors
share a diode-protected gate pad, and an (unprotected) source/drain pad. The
other drain/source of every transistor is connected to its own (unprotected)
wire bonding pad. We tested PMOS and NMOS core (1.2 V) transistors, and NMOS
I/O (2.5 V) transistors (with double thickness gate oxide).

###  2.2 ASIC package, test equipment, and measurement procedures

The test ASICs were wire bonded into (64-pin) pin grid array (PGA) chip
carriers so that they could be irradiated on simple printed circuit boards
(PCBs) containing only sockets for the ASICs and connectors for bias voltages.
Transistor characteristics were measured by mounting one chip carrier at a
time on a test board containing switches that allowed individual transistors
to be measured independently. The number of pads on the test ASICs was too
large to allow all pads to be wire bonded in one package, given the chosen
chip carrier, so three different packages with different wire bonding patterns
were made. One package had bonds only to devices intended for cold tests. NMOS
transistors were wire bonded in the second package, and PMOS transistors were
wire bonded in the third package. The devices intended for cold tests are all
large transistors unlikely to be used in a pixel readout ASIC. They have been
excluded from this analysis.

Figure 1: Simplified schematic of the PCB used to measure NMOS transistor
characteristics.

A different PCB was used to test each ASIC package. A simplified schematic of
the PCB used to test NMOS transistors is shown in Figure 1. The PCB used to
test PMOS transistors was very similar. Two Keithley[11] 237 Source
Measurement Units (SMUs)[12] were used, one to bias transistor gates, and one
to measure drain-souce current. A Labview[13] program running on a laptop
computer was used to sequence and control the measurements. The two SMUs were
controlled via General Purpose Interface Bus (GPIB)[14]. Logic on the test PCB
was controlled via USB using a National Instruments[15] USB-6501 I/O
board[16], connected to the test PCB by a ribbon cable. Bias voltage for the
protection diodes was generated by a voltage regulator on the test PCB from
the 5 V provided by the laptop USB port. The Labview program controlled solid
state switches on the test PCB that connected one of the SMUs to a single gate
pad at a time; unused gates were grounded. The program controlled LEDs on the
test PCB to indicate how mechanical (rotary) switches on the test PCB should
be set to connect the other SMU to a single transistor drain (unused drains
were left floating). All three voltage sources were referenced to a common
ground plane on the test PCB, and the source pads for all transistors in a
package were connected directly to this ground. The fact that we did not
separate the return current path for the two SMUs, together with possible
parasitic circuits involving the protection diodes and the solid state
switches in the OFF state, made it impossible for us to accurately measure the
leakage current of transistors in the ASIC packages.

###  2.3 Irradiation

The Sandia National Laboratories Gamma Irradiation Facility (GIF) uses 60Co
sources to provide controlled doses of ionizing radiation. 60Co decays by beta
decay to an excited state of 60Ni. 60Ni relaxes to the ground state by
emitting two gamma rays of energy 1.17 and 1.33 MeV [17]. At the Sandia GIF,
60Co is held in stainless steel “source pins” that are $3/8$ inch diameter and
18 inches long. A number of source pins are mounted in an array and to first
order, none of the beta electrons escapes the steel source pins. When not in
use, the sources are kept at the bottom of an 18 foot deep pool of deionized
water which provides shielding. The facility has three shielded irradiation
cells in a single high bay area above the shielding pool. Each irradiation
cell has an opening in the floor that allows a source array to be raised out
of the water into the cell by an elevator. The cell that was used in these
irradiations contained an array of 40 source pins arranged in a straight line.
The array contained approximately 225 kCi of 60Co. Our test ASICs were held
inside stainless steel thermos bottles (see Figure 2) positioned approximately
2 inches from the face of the source array.222The standard practice for 60Co
irradiation calls for the electrical devices being tested to be shielded with
1.5mm of lead followed by 0.7 - 1.0 mm of aluminum[21] “in order to minimize
dose enhancement effects caused by low-energy scattered radiation.” Our setup
did not include a lead-aluminum shielding structure. Cooling was provided by
vortex tube coolers [18] mounted in holes drilled through the plastic thermos
bottle lids.

The dose rate was 1425 rad/second as measured by an ion chamber placed inside
one of the thermos bottles.333All dosimetry was provided by Sandia National
Laboratories. The uniformity of the radiation field was checked by irradiating
thermoluminescent dosimeters (TLDs) taped to each of the chip carriers on the
irradiation PCBs. The TLDs were read at the Radiation Metrology Laboratory at
Sandia National Laboratories. The nonuniformity was measured to be less than
6% RMS by comparing the truncated mean (middle two of four) of the four TLDs
at each chip carrier position to the average of truncated means, for
measurements taken at the start and end of the irradiation. This variation,
which we did not correct for because it showed no obvious pattern at the
different chip carrier positions, dominates the error on the ion chamber
measurement. The TLD measurements also provided a check of the dose rate
measured with the ion chamber.

Figure 2: Pictures are shown of a thermos bottle assembly, including an
irradiation board with four chip carriers, before insertion of the irradiation
board into the thermos bottle. In the left photo, the red arrow points to the
vortex tube[18] on top of the thermos bottle lid. In the right photo, the red
arrow points to an antistatic bag which wraps the irradiation board and (LEMO)
low-voltage cable before irradiation. These bags separate the boards and
voltage cables from the not-very-dry thermos bottle environment, and provide
protection from the metal thermos bottle wall (the test structures are as
close to the inner thermos bottle wall as is safe, but not touching). During
irradiation, copper pipe was used to deliver air to the vortex tubes.

During irradiation, gamma rays interacted in the walls of the thermos bottles
and directly heated the inside of the thermos bottles. In order to maintain
the temperature of the test devices at less than $-20\,^{\circ}$C during long
irradiations, especially during daytime when the outside temperature was $\sim
35\,^{\circ}$C, it was necessary to precool the compressed air input to the
vortex tubes and to insulate the copper tubes carrying air to the vortex
tubes. Figure 3 shows the temperature of the two thermos bottles during long
irradiations. Temperatures were measured using a K-type thermocouple in each
thermos bottle, read out and recorded with a Fluke 52 II digital
thermometer[19]. The calibration error for K-type thermocouples used near
$-20\,^{\circ}$C is $\pm 2.2\,^{\circ}$C[20]. The precooling of the compressed
air was improved after the first two long irradiations, during which the
temperature in one of the two thermos bottles reached $-15\,^{\circ}$C.

Figure 3: The temperature measured inside the two thermos bottles ($\\#1$ in
blue and $\\#2$ in red) during long irradiations. No irradiation was performed
during the day on (Saturday) June 8, or on June 9. The two spikes where the
temperature reached about $8\,^{\circ}$C in both thermos bottles for 30
minutes late on June 12 occurred because the compressed air unexpectedly shut
off.

During irradiation the chip carriers were mounted in sockets on irradiation
PCBs. Each irradiation PCB held four chip carriers (see Figure 2), two for
PMOS packages, and one each for NMOS and cold transistor packages. Transistor
bias voltages were provided by Keithley 237 SMUs (located outside the shielded
irradiation cell) connected to the irradiation PCBs by 20 foot long triax
cables. The PMOS transistors were biased in two different ways. In one
package, the drains, sources, and gates were held at 1.2 V and the substrate
was grounded; the other package was biased with all the gates and the
substrate grounded, while the drains and sources were held at 1.2 V. The gates
of both the core NMOS and the I/O NMOS were biased at 1.2 V; all other nodes
were grounded. Twelve irradiations were performed over 15 days, as shown in
Table 1. After each irradiation step, a single characteristic curve was
recorded for each transistor. All measurements were made at room temperature.
The drain-source voltage was set to 1.2 V and the drain-source current was
measured as the gate-source voltage was swept from 0 to 1.2 V. It took
$\sim$10 minutes to test the transistors in each package. The ASIC packages
were kept at $-20\,^{\circ}$C in a freezer when not being tested or
irradiated.

Table 1: The irradiation schedule, showing the 2 weeks it took to accumulate 1 Grad. Date |  Length |  Dose(Mrad) |  Cumulative Dose(Mrad)
---|---|---|---
June 2 |  1 hour |  5 |  5
June 3 |  1 hour |  5 |  10
June 3 |  1 hour 45 mins |  9 |  19
June 3 |  4 hour 15 mins |  22 |  41
June 4-5 |  12 hours |  62 |  103
June 5-6 |  22 hours |  113 |  215
June 6-7 |  22 hours |  113 |  329
June 9-10 |  22 hours |  113 |  441
June 10-11 |  17 hours |  87 |  528
June 11-12 |  22 hours |  113 |  641
June 12-13 |  22 hours |  113 |  754
June 13-16 |  66 hours |  339 |  1093

Pre-irradiation measurements of the transistors showed that a small number of
transistors were broken either in fabrication or in the wire bonding process.
Approximately half of the transistors that were irradiated failed during the
15 days at Sandia. One group of 12 NMOS transistors was broken mechanically by
mishandling. Most of the other transistors that failed also did so in groups,
but without an obvious cause. We replaced the package containing the group of
12 failed NMOS transistors partway through the irradiation. The replacement
package received a total dose of 878 Mrad. Tables 2 and 3 list all of the
transistors included in this study and note which transistors failed and when
the failures occurred. Broken transistors were easily identified. For many,
the drain-source current was either very small or very large, independent of
gate bias. For a smaller number, the drain-source current varied approximately
linearly with gate bias.

The most likely cause of transistor failures is electrostatic discharge (ESD).
We took a number of steps to reduce the probability of ESD, but our procedures
had some deficiencies. The chip carrier packages were transported in an
antistatic box and when a package was mounted on, or removed from a PCB, the
work was done on a grounded antistatic mat by a person wearing a wrist
grounding strap. The PCB was grounded before a chip carrier was inserted into
or removed from a socket, but no ESD precautions were taken when the
irradiation PCBs were inserted into the thermos bottles or when bias cables
were connected. All transistor gate pads were diode protected on-chip, but
none of the source or drain pads was ESD protected. Moreover, bias for the on-
chip protection diodes was provided through only one pin of the PGA chip
carriers. If this pin failed to make contact before other pins while a package
was being inserted into a socket, the protection diodes may not have been
biased when they were needed most.

Table 2: NMOS transistors: Each entry in one of the last three columns corresponds to a transistor and indicates the dose accumulated before the transistor was broken. Transistors that were not broken have no entry. Zero indicates a transistor that was broken before irradiation. Transistors in the upper part of the table have standard thickness gate oxide; those in the lower part have gate oxide that is twice normal thickness. All transistors are standard layout unless otherwise indicated; ELT indicates enclosed layout. Transistors that share a gate pad are grouped together. IC2 received the full dose of 1.1 Grad; IC3 was replaced by IC1 after a dose of 215 Mrad, so IC1 received 878 Mrad. IC1 and IC3 were irradiated in thermos $\\#1$; IC2 was irradiated in thermos $\\#2$. W/L (nm) |  Type (if not simple) |  Gate |  IC1 |  IC2 |  IC3
---|---|---|---|---|---
120/60 |  |  1 |  0 |  328 |  43
240/60 |  |  1 |  |  5 |  43
360/60 |  |  1 |  |  5 |  43
480/60 |  |  1 |  |  5 |  0
600/60 |  |  1 |  0 |  5 |  43
1000/60 |  |  1 |  |  754 |
5000/500 |  |  2 |  |  754 |
5000/5000 |  |  2 |  426 |  754 |
120/60 |  Triple well |  2 |  |  328 |  43
5000/60 |  Triple well |  2 |  |  328 |  43
1500/300 |  Zero $V_{t}$ |  2 |  |  5 |  43
2050/60 |  ELT |  2 |  0 |  5 |  43
2240/300 |  Zero $V_{t}$ ELT |  2 |  |  328 |  43
400/280 |  |  3 |  |  754 |
500/280 |  |  3 |  |  754 |
800/280 |  |  3 |  |  754 |
1000/280 |  |  3 |  |  754 |
5000/500 |  |  3 |  |  754 |
5000/5000 |  |  3 |  |  754 |
2220/280 |  ELT |  3 |  |  754 |
3380/1200 |  Zero $V_{t}$ ELT |  3 |  |  754 |
400/280 |  Triple well |  3 |  426 |  754 |
800/280 |  Triple well |  3 |  |  754 |
Table 3: PMOS transistors: Each entry in one of the last four columns corresponds to a transistor and indicates the dose accumulated before the transistor was broken. Transistors that were not broken have no entry. IC4 and IC6 were biased with $V_{s}=V_{d}=V_{g}$. IC5 and IC7 were biased with $V_{s}=V_{d}=1.2$ V and $V_{g}=$GND. All four packages received the full dose of 1.1 Grad. IC4 and IC5 were irradiated in thermos $\\#2$; IC6 and IC7 were irradiated in thermos $\\#1$. W/L (nm) |  Gate |  IC4 |  IC5 |  IC6 |  IC7
---|---|---|---|---|---
120/60 |  1 |  0 |  |  218 |  531
360/60 |  1 |  328 |  |  13 |
600/60 |  1 |  0 |  754 |  43 |  443
1000/60 |  1 |  5 |  |  531 |
5000/500 |  2 |  0 |  |  |
5000/5000 |  2 |  328 |  |  531 |

After the irradiations, the devices were kept at $-20\,^{\circ}$C in a freezer
that could be powered either by 120 V or by 12 V and transported to Fermilab.
Once at Fermilab the transistors were removed from the freezer and kept at
room temperature for one week. Multiple measurements were taken during this
time. Then the transistors were held in an oven at $100\,^{\circ}$C for
another week and a final set of measurements was made. This annealing schedule
can be seen in Table 4. The transistors were not biased during transport or
annealing.

Table 4: The annealing times and temperatures of the transistors. Annealing
Schedule
---
June 16-24 |  $-20^{\circ}$C |  8 Days
June 24 - July 1 |  Room Temperature |  7 Days
July 1-8 |  $100\,^{\circ}$C |  7 Days

##  3 Analysis and Results

Figure 4: This figure illustrates the quadratic extrapolation method used to
determine the (saturation) threshold voltage ($V_{th}$) of an NMOS transistor.
The data shown is from the pre-irradiation measurement of the 240/60
transistor in IC3. For PMOS transistors, $|I_{ds}|$ is used since $I_{ds}$ is
negative.

Two quantities were extracted from each transistor characteristic: the maximum
drain-source current and the (saturation) threshold voltage $V_{th}$. The
quadratic extrapolation method was used to determine the threshold
voltage[22]. As shown in Figure 4, $V_{th}$ is defined to be the voltage at
which a line tangent to the curve $\sqrt{|I_{ds}|}$ vs $V_{gs}$ at the point
of maximum $\frac{d\sqrt{|I_{ds}|}}{dV_{gs}}$ intercepts the $I_{ds}=0$ axis.
We determined the slope of the curve by fitting it with a fifth order
polynomial and differentiating the fit function. In Figure 4, the red squares
were computed using finite differences
(${\sqrt{I_{ds}(N+1)}-\sqrt{I_{ds}(N)}\over{V_{gs}(N+1)-V_{gs}(N)}}$ ); the
black line is the result of differentiating the fit to the curve
$\sqrt{|I_{ds}|}$ vs $V_{gs}$.

Figure 5: Transistor characteristic curves for total dose up to 1.1 Grad of
(upper left) a 120/60 core PMOS, (upper right) a 360/60 core PMOS, and for
total dose up to 878 Mrad of (lower left) a 240/60 core NMOS, and (lower
right) a 1000/280 2.5 V NMOS.

Figure 5 illustrates the radiation effects observed in our data. The most
prominent effect is a decrease of the maximum drain-source current of core
PMOS transistors. The fractional decrease is largest for the smallest PMOS
transistors; the maximum drain-source current of the smallest PMOS decreased
by more than a factor of two. The maximum drain-source current of core NMOS
transistors also decreased, but only by $\sim 5-10\%$. No significant
threshold shift was observed for any of the core transistors, but the
threshold voltage of NMOS I/O transistors increased by 100 - 200 mV. No error
bars are included in the figures because the uncertainty in the SMU
measurements is smaller than the symbols used to plot the measurements.

Figure 6: The change in maximum drain-source current for similar PMOS core
transistors irradiated with different gate bias voltages. The graph on the
left is for 120/60 transistors and the graph on the right is for 360/60
transistors. The lines connecting points do not represent a fit, and are
included only to make the plots easier to read. The transistor characteristics
measured for transistors in package IC5 after 754 Mrad was accumulated were
all offset by current not likely to have passed through the transistors (this
can be seen in Figure 5). Lines are not drawn through these points. The most
likely source of these offsets is leakage current due to moisture caused by
condensation on the cold IC package.

Figure 7: Transistor chararcteristic curves during the annealing period for
(left) a 120/60 core PMOS and (right) a 1000/280 2.5 V NMOS.

No significant difference was observed between the radiation-induced changes
of PMOS transistors biased during the irradiation with the gate in the ON
state and PMOS transistors biased with the gate in the OFF state. This is
illustrated in Figure 6.

Figure 8: The graph on the left shows the loss of maximum drain-source current
during irradiation for 4 PMOS core transistors. The graph on the right shows
the recovery of maximum drain-source current for the same 4 transistors during
and after annealing. As in Figure 6, lines are included to make the plots
easier to read. Once again, lines are not drawn through the points
corresponding to measurements made after 754 Mrad of transistors in IC5. The
measurements shown for the 5000/5000 transistor are for the transistor in IC7.
For this transistor, no point is included corresponding to an integrated dose
of 641 Mrad; we believe that the rotary switch was not set correctly during
the measurement of this transistor characteristic since the recorded drain-
source current was very small for all values of the gate bias.

Figure 9: The graph on the left shows the loss in maximum drain-source current
after each irradiation step for 9 NMOS core transistors. The graph on the
right shows the change in maximum drain-source current for the same 9
transistors during and after annealing.

Figure 10: The shift in threshold voltage for 8 NMOS I/O transistors
irradiated to 878 MRad is shown in the graph on the left, while the graph on
the right shows $V_{th}$ for the same 8 transistors during and after
annealing. No significant annealing was observed for the two zero $V_{th}$ I/O
transistors.

Figure 7 demonstrates the annealing effects observed in our data. Both the
PMOS core transistors and the NMOS I/O transistors recovered significantly
during the annealing period.

Figures 8 and 9 show the evolution of the maximum drain-source current for a
representative selection of PMOS and NMOS core transistors during irradiation
and annealing. We did not observe any significant differences in the effect of
radiation on the various different types of NMOS transistors tested (normal
layout, enclosed layout, triple well, and zero $V_{th}$). Figure 10 shows the
threshold shift of a representative selection of NMOS I/O transistors during
irradiation and annealing.

##  4 Summary

Previous measurements have established 65 nm CMOS as the leading candidate
technology for HL-LHC electronics. After an exposure of 200 Mrad, Bonacini, et
al. reported [3], with one exception, only minor changes in transistor
parameters. The exception was a significant loss of maximum drain-source
current by narrow PMOS core transistors. They reported a 50% reduction in
maximum drive current for a 120/60 PMOS core transistor and a 35% loss for a
360/60 PMOS core transistor. This irradiation of “cold” 65 nm CMOS transistors
was motivated by a concern that damage to pixel vertex detector readout
electronics operated at $-20\,^{\circ}\mathrm{C}$ might be greater than
observed in room temperature irradiations. Our measurements show the same
pattern of effects as observed previously, but the damage is less severe than
was observed at room temperature, rather than more severe.

##  5 Acknowledgments

We wish to thank Charles Bowen of the University of Colorado Department of
Physics Precision Machine Shop, Nina Moibenko of Fermilab’s Electrical
Engineering Department, and Donald Hanson, Maryla Wasiolek, and Nathan Hart of
the Sandia National Laboratories Gamma Irradiation Facility. This work was
supported in part by Department of Energy grant (DE-SC0006963). Fermilab is
operated by Fermi Research Alliance, LLC under Contract No. DE-AC02-07CH11359
with the United States Department of Energy.

## References

  * [1]
  * [2] CERN, the European Organization for Nuclear Research, CH-1211 Geneva 23, Switzerland; http://home.web.cern.ch
  * [3] S. Bonacini, et al., “Characterization of a commercial 65 nm CMOS technology for SLHC applications,” 2012 JINST 7 P01015
  * [4] J. Christiansen, et al., “RD Collaboration Proposal: Development of pixel readout integrated circuits for extreme rate and radiation,” CERN-LHCC-2013-008; LHCC-P-006. Available at http://rd53.web.cern.ch/RD53/.
  * [5] M Barbero and M. Menoui, private communication.
  * [6] http://www.sandia.gov/research/facilities/gamma irradiation facility.html.
  * [7] Sandia National Laboratories, P.O. Box 5800, Albuquerque, NM 87185
  * [8] TSMC Corporate Headquarters, 8, Li-Hsin Rd. 6, Hsinchu Science Park, Hsinchu 300-78, Taiwan, R.O.C.; telephone: +866-3-5636688; http://www.tsmc.com
  * [9] MOSIS, 4676 Admiralty Way, 7th floor, Marina del Rey, CA 90292; telephone: +1-310-448-9400; http://www.mosis.com
  * [10] Sandro Bonacini, private communication.
  * [11] Keithley, 28775 Aurora Road, Cleveland, Ohio 44139; telephone: +1-440-248-0400
  * [12] A data sheet for the Keithley 237 is available at http://www.keithley.com/data?asset=5512
  * [13] http://www.ni.com/labview/
  * [14] A history of GPIB and IEEE-488 is available at http://www.ni.com/white-paper/3419/en/pdf
  * [15] National Instruments, 11500 N. Mopac Expressway, Austin TX 78759; telephone: +1-877-387-0015
  * [16] A data sheet for the NI USB-6501 is available at http://www.ni.com/datasheet/pdf/en/ds-135
  * [17] K.A. Olive, et al. (Particle Data Group), Chin. Phys. C, 38, 090001 (2014), page 466
  * [18] Vortec 106-4-H vortex tubes were purchased from Vortec (www.vortec.com), part of the ITW Air Management business unit of Illinois Tool Works.
  * [19] Fluke Corporation, 6920 Seaway Boulevard, Everett, WA 98206; telephone: +1-425 347-6100 Specifications for the Fluke 52 digital thermometer are available at http://en-us.fluke.com/products/thermometers/fluke-52-ii-thermometer.html$\\#$techspecs
  * [20] “Manual on the Use of Thermocouples in Temperature Measurement: 4th Edition,” ASTM International, West Conshohocken, PA, 2014 (1993).
  * [21] ASTM Standard F1892-12, 2012, “Standard Guide for Ionizing Radiation (Total Dose) Effects Testing of Semiconductor Devices," ASTM International, West Conshohocken, PA, 2014,DOI:10.1520/F1892-12, http://www.astm.org.
  * [22] Dieter K. Schroder, _Semiconductor Material and Device Characterization_ , John Wiley and Sons, Inc., New York 1998.


# The Curious Case of Control

Elias Stengel-Eskin
Johns Hopkins University
<EMAIL_ADDRESS>&Benjamin Van Durme
Johns Hopkins University
<EMAIL_ADDRESS>
###### Abstract

Children acquiring English make systematic errors on subject control sentences
even after they have reached near-adult competence (Chomsky, 1969), possibly
due to heuristics based on semantic roles (Maratsos, 1974). Given the advanced
fluency of large generative language models, we ask whether model outputs are
consistent with these heuristics, and to what degree different models are
consistent with each other. We find that models can be categorized by behavior
into three separate groups, with broad differences between the groups. The
outputs of models in the largest group are consistent with positional
heuristics that succeed on subject control but fail on object control. This
result is surprising, given that object control is orders of magnitude more
frequent in the text data used to train such models. We examine to what degree
the models are sensitive to prompting with agent-patient information, finding
that raising the salience of agent and patient relations results in
significant changes in the outputs of most models. Based on this observation,
we leverage an existing dataset of semantic proto-role annotations (White et
al., 2020) to explore the connections between control and labeling event
participants with properties typically associated with agents and
patients.111Code and prompts available at https://github.com/esteng/curious-
case-of-control.

##  1 Introduction

Normally-developing children learning English struggle with subject control
clauses long after they have successfully acquired the components to
understand them (Chomsky, 1969; Cromer, 1970; Maratsos, 1974; Sherman and
Lust, 1993). A sentence with a subject control clause has a matrix (main)
clause containing a main verb, an agent, and a patient, and an embedded
infinitival clause. For example, in _Cole promised Joe to call_ , the agent is
_Cole_ , the patient is _Joe_ , and the embedded clause is _to call_.
Crucially, the embedded verb here does not have an overt subject, but rather
implicitly refers to a subject in the matrix clause (in this case, _Cole_). In
a subject control clause, this latent subject of the embedded infinitival
clause (usually written as _PRO_) is coindexed with the _subject_ (_Cole_)
rather than the object (_Joe_) of the matrix (main) clause, i.e. _Cole_ (the
agent) is doing the calling. This is typically written:

[Cole]${}_{\text{NP}_{i}}$ promised [Joe]${}_{\text{NP}_{j}}$ PROi to call (1)

where subscripts indicate the noun phrase (NP) _“Cole”_ is the subject of _“to
call”_. (1) can be contrasted with the more common case of object control; for
example, if the matrix verb _“promised”_ is swapped with an object control
verb like _“told”_ , then the coreferrent of PRO changes:

[Cole]${}_{\text{NP}_{i}}$ told [Joe]${}_{\text{NP}_{j}}$ PROj to call (2)

Chomsky (1969)222NB: the author is Carol Chomsky, not Noam Chomsky. finds that
children ages 5 to 10 regularly misinterpret subject control (1) for object
control (2) while correctly interpreting object control clauses, and proposes
that children are following the Minimal Distance Principal (MDP), choosing the
linearly closest noun phrase (NP) to govern PRO. Cromer (1970) highlights the
systematicity with which children mistake subject control for object control
and provides evidence for the MDP. However, Maratsos (1974) argues against the
MDP; while his results support the observation that children struggle with
subject control, they do not support the MDP, favoring an alternative based on
semantic roles. Maratsos changes the subject and object order through
passivization:

[Joe]${}_{\text{NP}_{j}}$ was told by [Cole]${{}_{\text{NP}_{i}}}$ PROj to
call (3)

finding that children _correctly_ coindex PRO with the (further away) object,
violating the MDP.

Recently, large pre-trained language models have shown an impressive ability
not only to produce fluent text, but also to perform tasks by “filling in the
blank” in question-answering prompts, either with no previous examples (zero-
shot) or with a few representative examples (few-shot) Brown et al. (2020);
Raffel et al. (2020); Sanh et al. (2021). In light of the difficulty children
have in acquiring subject control constructions, we explore how the outputs of
the language models tested compare with adult and child strategies for
coindexing PRO. In Section 3.2, we examine this question in the zero-shot
setting where we give the models only a single question, treating each model
as a sort of experimental subject (cf. Fig. 1). Our initial hypothesis is that
model outputs will be consistent with child strategies, i.e. the models will
perform well on object control examples, but misinterpret subject control for
object control. This is informed by two factors: object control is orders of
magnitude more frequent than subject control (cf. Section 3.2), and active
object control (i.e. (2)) requires resolving a shorter dependency than subject
control. We instead find that the tested models fall into three groups, with
the majority in fact producing responses mistaking object control for subject
control – the opposite of what children do.

Figure 1: Zero-shot probe for object control. Colors indicate names, which can
be swapped.

Following these observations, in Section 4 we examine to what degree this
behavior is sensitive to semantic roles, following Maratsos (1974). To test
this, we investigate a “few-shot” setting, where we prompt the model not only
with the context and a single question, but also with a set of question-answer
pairs that raise the salience of the matrix agent and patient (cf. Fig. 2).

Figure 2: A prompt-hacked example for object control, with long-form
instructions.

We find that the models whose behavior in the zero-shot setting was consistent
with a positional heuristic have significant differences in the few-shot
setting, and the directions of these difference are consistent with a
sensitivity to semantic roles.

Finally, in Section 5.2 we investigate whether the sensitivity to semantic
roles corresponds to performance on a semantic proto-role labeling task, where
models are tested with questions about volition and change of state,
properties associated with agents and patients (respectively). We find that
while some models are able to perform the labeling task surprisingly well, the
differences between models do not necessarily map to the differences in
Section 4. We offer three key takeaways:

  1. 1.

For many models (all GPT-Neo variants, Jurassic Jumbo) the outputs are
surprising given their training distributions; while object control is orders
of magnitude more common in the text data used in training these models, they
perform better on subject control.

  2. 2.

Large pretrained models are not consistent among themselves. Even models with
similar architectures can have very different trends in their outputs, and the
outputs of autoregressive and text-to-text models differ substantially.

  3. 3.

The associations in the autoregressive models tested form outputs that can be
explained by simple, often position-based heuristics. For text-to-text models
(e.g. T0, T5) the output patterns can also be captured by heuristics based on
agent and patient relations. However, sensitivity to agent and patient
relations in subject and object control clauses does not always entail higher
performance on semantic proto-role labeling.

##  2 Models and Metrics

We explore both autoregressive models and text-to-text models. Autoregressive
models are optimized by minimizing $-\log(P(w_{i}|w_{-i}))$ for words
$w_{1},\ldots w_{N}$ in a given context. These models (also called “decoder-
only” models) are composed of just a decoder, which encodes the previously
observed tokens $w_{1},\ldots w_{i-1}$ produces a probability distribution
over the vocabulary for the next token, $w_{i}$. Text-to-text models are
encoder-decoder models, optimized to reconstruct a noised version of the input
via the decoder. An encoder takes a corrupted version of whole sequence
$w_{1},\ldots w_{N}$ as input, encoding it into a dense representation from
which the decoder reconstructs the original $w_{1},\ldots w_{N}$.

The autoregressive models considered are:

  * •

GPT-3 Davinci: this model is only available through the OpenAI API, and its
exact training details are unclear. It is based on the GPT-3 model (Brown et
al., 2020) which was trained on Common Crawl (Raffel et al., 2020) with 175B
(billion) parameters. Among the several versions of GPT-3, Davinci is
generally regarded as the highest-performing (OpenAI, ).

  * •

GPT-Neo: this is an open-source replication of GPT-3 introduced by Black et
al. (2021), trained on The Pile (Gao et al., 2020), a 800Gb dataset of web-
text intended for pre-training. GPT-Neo has 3 sizes: 1.3B, 2.7B, and 6B
parameters (GPT-J), all trained on the same dataset, allowing for direct
comparison.

  * •

Jurassic: Jurassic Large (7.5B parameters) and Jurassic Jumbo (178B
parameters) (Lieber et al., 2021) are also accessible only through an API. The
training data is based on Common Crawl, though similarly to GPT-3 Davinci, the
details of the training data filtering process are unclear. Relevant
differences to GPT-3 are in the tokenization (which includes multi-word
expressions) and use of fewer, wider layers.

The text-to-text models we consider are:

  * •

T5 for QA: The T5-base text-to-text model (220-million parameters) (Raffel et
al., 2020) was pre-trained on cleaned Common Crawl data (C4) and fine-tuned on
SQuaD question answering data (Rajpurkar et al., 2016).

  * •

T0pp: presented by Sanh et al. (2021), T0pp is an 11B parameter model with a
T5-like architecture, pre-trained on Common Crawl data and finetuned
specifically for zero-shot question answering on the P3 dataset of NLP
benchmarks. This dataset recasts a large number of NLP benchmark datasets into
question answering prompts.

Note that because of fine-tuned nature of the “T5 for QA” model, the expected
prompt format is fixed, unlike the unrestricted prompt format for the other
models. Thus, prompt hacking cannot be done on this version of T5, and so it
is only used in Section 3.2. We access non-API models via the Transformers
library (Wolf et al., 2020); due to computational constraints, they are run on
single GPUs at 1/2 precision. For all models, we decode with the temperature
parameter set to 0.

###  2.1 Metrics

Online APIs make forced decoding very costly (Shin and Van Durme, 2021).
Rather than comparing logits for a restricted output vocab, we allow the model
to freely generate tokens, letting the model produce a larger variety of
answers. In other words, rather than comparing the output probabilities for
particular tokens (the logits) given a fixed prefix, we compare full strings
of output tokens. However, this method requires heuristics to classify the
output strings into categories. In Section 3.2 we validate our heuristics,
verifying that for locally-run models the trends are similar when using
logits.

Our metric first extracts single word answers and then searches for answers
like “The answer is: NAME”. For some models and settings, the model re-
generates the entire prompt before answering, i.e. it copies the instructions,
context, and question, before copying the answer continuation and finally
producing an answer. We use Levenshtein distance to check whether the prompt
has been regenerated; if it has, it is removed and the first string following
the prompt is checked for answer strings. The extraction function always
returns the first valid answer produced by the model. If the extraction
function fails to find any valid answer strings, the example is skipped in
evaluation rather than counted as wrong. We measure significance in model
differences with McNemar’s test (McNemar, 1947), following Dietterich (1998).

##  3 Experiment 1

In order to examine what types of generalizations are made by the examined
models when prompted for subject and object control information, we construct
a number of question-answering-style prompts, where the models fill in the
answer. This approach follows recent literature (Raffel et al., 2020; Brown et
al., 2020) and takes advantage of the models question-answering abilities.
Moreover, using models pretrained with a language-modeling loss rather than
training a model specifically for control lets us examine what types of
generalizations are captured by the models’ associations learned from its
original training data, rather than whether a very large model can learn to
answer subject and object control questions correctly. We first describe the
construction of the prompts used in this set of experiments. Using those
prompts, we validate the choice to use heuristic extraction from open
generation (i.e. allowing the model to generate tokens up to an end-of-
sequence token) rather than logits and confirm that in the C4 dataset, object
control is more frequent than subject control. Then, we analyze the zero-shot
performance of the models on the subject and object control plots.

###  3.1 Subject and Object Prompts

While pretrained language models used for QA are often evaluated in a “few
shot” setting, where they are given a few “training” prompts before answering
a “test” prompt, in our main experiments we focus on the zero-shot setting.
This is in order to avoid learning effects that might result from few-shot
prompting (as one would with human subjects) and follows human experiment
paradigms, where experimenters are careful not to provide feedback to subjects
about the expected answer until after all trials are complete. The prompts
used in Section 3.2 and Section 4 have an instruction sentence, a context
(like (1)-(3)), a question (e.g. “Who called?”), and an answer continuation.
See Fig. 1 for an example zero-shot object control prompt.

We take the max over two instruction types (long and short) in our analyses.
Fig. 1 shows the long-form instructions, which include the options in the
instructions (e.g. _Answer the question with either "Casey" or "Avery"_) and
in the question (e.g. _Casey or Avery_). The short-form instructions omit
these prompts in the instructions and questions.

Since the models examined can be sensitive to specific tokens, we cover 9
embedding verbs for object control, chosen from a selection of common
linguistics examples: “told”, “ordered”, “called upon”, “urged”, “asked”,
“persuaded”, “convinced”, “forced”, and “pushed”. These verbs are presented
both in the active (object control experiments) and passive (passive object
control experiments). These include verbs that trigger a factuality inference
in the affirmative (e.g. “persuaded”, “convinced”, “forced”). For subject
control, we follow previous work (Chomsky, 1969; Maratsos, 1974) and use
“promise”; we also include “threaten” as a subject control verb. These verbs
are presented only in the active voice, as sentences such as “Casey was
promised by Avery to call” were deemed too ambiguous (if grammatical at all).
In our main experiments, we use names as NPs; we also report results in
Appendix A using common professions to ensure that the trends observed with
names hold. We chose 2 male names, 2 female names, and 2 gender-neutral names;
these were chosen by taking the top 2 names in each reported gender category
in US Social Security data from 1970 to
2019.333https://www.ssa.gov/oact/babynames/ The gender-neutral names were
chosen by taking the top names in the intersection of male and female
names.444We note that this does not guarantee that the name is equally likely
for both genders. We run the same prompt with each name combination in both
orders, to avoid possible biases the model may have towards particular names.
When the names are included in the instruction, we add an example with the
name order swapped to avoid confounding due the model simply copying the first
or last name to appear in the instructions. Finally, for the action infinitive
(i.e. the embedded verb) we chose the first 5 coherent verbs (i.e.
intransitive infinitives) from a frequency list of English verbs (Yu et al.,
2020; Sharov, 2020). This yields 1500 sentences for object control and 150 for
subject control (3000 and 300 with swapped names).

###  3.2 Results and Analysis

#### Frequency of Subject and Object Control

In Section 1, we claimed that object control is more frequent that subject
control. To support this claim in the context of the models examined here, we
conduct a search of a subset of the C4 dataset (Raffel et al., 2020) for
sentences fitting subject control and object control templates. While there
are many types of subject and object control, we focus on infinitival
complements with transitive matrix verbs, searching with templates similar to
the sentences in examples 2 and 1. For object control, we use the same verb
list as in Section 3.1. For subject control, we use “promise” and “threaten”,
as in Section 3.1. We allow the embedded verb to be any verb. We sub-sample
the first $1\,000\,000$ sentences of C4 and search them with the templates,
finding that object control occurs $10\,435$ times, while subject control
occurs only $209$ times, i.e. object control is $\sim 50$ times more
frequent.555Note that object control is to be distinguished from Exceptional
Case Marking (ECM; Chomsky et al., 1986) and raising, which yield similar
surface forms but have a different analysis than object control. Since object
control is already shown to be far more frequent than subject control, we do
not examine common ECM and raising verbs; however, we note that, when used
with transitive matrix verbs (i.e. verbs with a matrix subject and object)
these constructions generally follow the object control template, where the
matrix object is the embedded subject.

#### Validating Logits

Fig. 3 shows the zero-shot accuracy of the best instructions using logit
scoring, for the models for which we have access to the full output
distribution (non-API models).

Comparing the results to Fig. 4, we see similar but less pronounced trends. As
in Fig. 4, GPT-Neo models perform better on subject control and passive object
control, while text-to-text models perform better on object control and
passive object control. This validates our choice to use heuristics in later
experiments. We note also that for GPT-Neo models, the heuristic results in
Fig. 4 reflect higher accuracies than the logit-based results in Fig. 3,
indicating that the heuristics capture broader range of outputs corresponding
to valid answers.

Figure 3: Accuracy of logit-scored model, taking the max across instruction
types.

#### Zero-shot performance

In Fig. 4, we see that model classes have different results; we further
classify models into 3 groups:

  1. 1.

The GPT-Neo variants and Jurassic Jumbo are better on subject and passive
object control than object control. This pattern can be accounted for by a
positional heuristic, namely to take the _first_ NP in the matrix clause (i.e.
_Maximum_ Distance Principle rather than the minimum distance principle of
Chomsky (1969)).

  2. 2.

T5 and T0 are consistent with the observations in Maratsos (1974); both models
do better on object control (active and passive) than subject control. This
contradicts the MDP but is consistent with a heuristic choosing the matrix
patient.

  3. 3.

GPT-3 and Jurassic Large both perform well above chance on object control and
subject control, with their best performance on subject control, but both
perform worse on passive object control. This could be matched to a positional
heuristic (take the second NP) for object control verbs, rather than an
agency-based heuristic.

#### Further observations

Even within model families, there are measurable differences: although GPT-3
and Jurassic Jumbo are roughly the same size and share a general architecture,
and are ostensibly trained on similar data, the changes made by Lieber et al.
(2021) seem to have a measurable impact, with Jurassic Jumbo performing
differently on zero-shot object control examples. For active object control,
the difference
$\Delta_{\text{GPT-3},\text{Jumbo}}^{\text{active}}=acc_{\text{GPT-3}}-acc_{\text{J-Jumbo}}=29$
($p<0.01$), and for passive
$\Delta_{\text{GPT-3},\text{Jumbo}}^{\text{passive}}=-9$ ($p<0.01$).
Similarly, GPT-3 differs from GPT-Neo-1.3B on active object control, and from
GPT-Neo-2.7B and GPT-J on both forms of object control, despite sharing an
architecture. Further analysis is impeded by a lack of details on the data
used to train GPT-3 and Jurassic; this underscores the need for model creators
to be transparent about training data and details.

Figure 4: Zero-shot accuracy on object control, passive object control, and
subject control. Black line represents random performance ($50\%$ accuracy).
Figure 5: Accuracy on object control, passive object control, and subject
control after prompting with agent and patient questions. Accuracy changes to
Fig. 4 are generally consistent within heuristic groups.

We also observe that larger models tend to have higher performance: GPT-J is
better on all settings than GPT-Neo-1.3B and 2.7B, and Jurassic Jumbo is
better than Jurassic Large on passive object control. That said, some larger
models are also slightly worse than their smaller counterparts (e.g. Jurassic
Jumbo on object control). This suggests that larger models may be more prone
to learning patterns corresponding to simple heuristics; however, additional
evidence is needed.

##  4 Experiment 2

Recent work has shown that “prompt hacking” a frozen pre-trained model can
yield higher performance on QA tasks (Brown et al., 2020). In this setting
(sometimes called “in context learning”), several demonstrations of questions
and answers are given in the context, followed by a test example, to which the
model produces an answer. The demonstrations in the context give the model
additional guidance on which task is being evaluated. While we do not do
“prompt hacking” with training questions about object in subject control, we
do experiment with adding information to the prompt to raise the salience of
agents and patients (e.g. “Q: Who told someone to call? A: Cole” for (2).) An
example can be seen in Fig. 2; the QA pairs given in the context give the
agent and the patient of the matrix clause, but not the embedded clause.

We hypothesize that prompting with agent-patient questions will affect each
group as follows:

  1. 1.

For Group 1 (GPT-Neo, Jurassic Jumbo) where the models’ outputs are consistent
with positional heuristics, the additional prompts will provide some evidence
inconsistent with the heuristic. This evidence may lead to changes in the
models’ associations that result in outputs less consistent with the
heuristics. For example, in Fig. 2 the question _Who was told to come, Avery
or Casey?_ the answer _Casey_ provides evidence against taking the furthest-
away NP as the answer. In that case, we would expect a drop in performance in
passive object control and in subject control, where the heuristic is
beneficial, and an increase in object control, where the heuristic does not
help.

  2. 2.

In Group 2 (T0 and T5), since the model outputs are already consistent with a
semantic role-based explanation, we do not expect much change in any setting.
In other words, the model outputs are already consistent with access to agent-
patient relations combined with an incorrect association for outputting the
matrix clause’s agent as the embedded subject.

  3. 3.

Finally, in Group 3 (GPT-3, Jurassic Large), we see that the models’ outputs
on object control are consistent with an object control-specific heuristic (to
take the first NP) the models have lower performance for passive object
control than active object control. Thus, as in Group 1, we expect that
evidence against the positional heuristic in the prompts will boost
performance in passive object control, while reducing it on active object
control.

###  4.1 Results and Analysis

Fig. 5 shows the results after applying prompts with questions about agents
and patients. Here, we see that for Group 1 (GPT-Neo and Jurassic Jumbo) the
performance does decrease for subject control and passive object control. This
decrease is significant for all models and settings ($p<0.01$) except Jurassic
Jumbo in subject control ($\Delta_{\text{Jumbo}}^{\text{subj}}=-7$, $p=0.06$).
At the same time, all object control performance increases significantly for
Group 1 ($p<0.01$). These results confirm our hypotheses, supporting the
notion that these models are consistent with a positional rather than semantic
heuristic. For Group 2 (T0), we find a significant decrease in performance on
both object control types ($\Delta_{\text{T0}}^{\text{active}}=-20$,
$p<0.01$), and no significant difference for subject control ($p=0.14$); this
is roughly consistent with our predictions, since the size of the decrease for
object control is relatively small (e.g. compared to the decrease for GPT-3 in
the passive). Finally, for Group 3 (GPT-3 and Jurassic Large) we largely see
the opposite of what we expected: GPT-3’s performance on object control goes
close to ceiling after prompting with agent-patient questions, while the
passive performance drops to 0; similarly, Jurassic Large’s performance drops
on passive object control, dropping slightly also on active object control.
Both models improve on subject control, with significant differences from the
zero-shot setting ($\Delta_{\text{GPT-3}}^{\text{subj}}=15$, $p<0.01$,
$\Delta_{\text{Large}}^{\text{subj}}=9$, $p<0.01$), perhaps reflecting an
effect of the semantic role-based priming. Note that for GPT-3, the passive
performance drop is from a lack of parseable, non-empty strings being
produced, rather than incorrect predictions. For active object control, GPT-3
outputs the second NP more often with additional prompts, increasing the score
from $78\%$ to $99\%$. It may be that these associations responsible for this
are also to blame for the degenerate behavior seen in the passive, where the
model produces only an end-of-sequence token.

##  5 Experiment 3

Following Maratsos (1974)’s hypothesis that the observed mistakes children
make on subject control sentences is driven by semantic roles, in Section 5.2
we examine the relationship between a model’s ability to perform zero-shot
object and subject control and its accuracy on identifying attributes commonly
associated with agents and patients. Since querying language models using
fixed semantic role ontologies (e.g. AGENT, PATIENT, THEME) may be difficult
since these ontologies may be absent from pretraining corpora, we instead
measure the models ability to perform semantic proto-roles labeling (SPRL) for
the volition and change of state properties. We use the SPRL data provided in
the Universal Decompositional Semantics (UDS) dataset introduced by White et
al. (2020). These properties, first proposed by Dowty (1991), were found to be
strongly prototypical of agents and patients, respectively (Reisinger et al.,
2015).666While instigation and stationarity were slightly more predictive of
agency and patienthood, they were deemed to be more difficult to re-frame as a
prompt. Proto-role inferences are elicited with simple prompts, circumventing
brittle and complicated ontologies. Indeed, the UDS dataset was built by
asking annotators questions like “How likely is it that ARG chose to be
involved in the PRED?” and normalizing their scalar ratings to $[-3,3]$.

###  5.1 Semantic Proto-Role Labeling Prompts

To construct a dataset of SPRL prompts, we first filter the UDS dataset for
sentences with $<35$ tokens – this eliminates many long sentences, which are
often more difficult to answer. We then eliminate examples with scalar
annotations $\in(-1,1)$, keeping only examples with strong inferences about
the properties. The annotations are binarized with values $\geq 1$ leading to
a “Yes” answer and values $\leq-1$ leading to “No”. The annotations are
balanced between “Yes” and “No”, with the excess examples from the more
frequent category being removed.

Figure 6: Prompt for eliciting SPRL judgments, shown here with one prompting
example (1-shot).

Two templates are used for each property; an example template is shown in Fig.
6. For volition, one template asks, _In the event “PRED”, does the participant
“ARG” act with volition?_ while the other asks _In the event “PRED”, does the
participant “ARG” act on purpose?_. For change of state, the first template
asks, _In the event “PRED”, does the state of the participant “ARG” change?_
and the other asks, _In the event “PRED”, does the participant “ARG” change in
state?_ We take the maximum over these two templates.

In this set of experiments we are interested in the raw ability of the model
to perform the semantic proto-role labeling (SPRL) task, and so we allow for
full prompt hacking, where demonstrations of the task are provided as part of
the context. Accordingly, we stratify the annotations into 4 stages; the
bottom stage always forms the “test” prompt, with the answer blank. The
remaining 3 stages are added for increasing levels of prompting with
“training” question-answer pairs (i.e. when the 3rd layer is added, there are
3 example question-answer pairs with answers, and one “test” pair that has no
answer, where the model must fill in the answer.) Fig. 6 shows a prompt with
one training stage, followed by one test example. We ensure that we use each
annotation only once, and that all test annotations paired across across
prompting settings. This results in 118 change-of-state test prompts and 168
volition prompts.

#### Hypotheses

We expect that models which perform well on zero-shot subject and object
control (e.g. those that can model both the active and passive of object
control) will also have higher performance on SPRL, since both require
semantic role information. Specifically, we expect to see higher performance
from T0 and T5 on at least one of the properties, since their outputs on
active and passive object control are consistent with a heuristic that
identifies patients as embedded subjects, rather than a positional heuristic.
Thus, it may be that the representations learned by T0 and T5 contain more
information on agency and patienthood, leading to better performance on SPRL.

###  5.2 Results and Analysis

Table 1 shows the accuracy on binary semantic proto-role labeling of all
models with performance significantly above a random baseline.

Setting | Model |  $\\#$ shots | Acc. |  $\\#$ valid
---|---|---|---|---
$\Delta$State | GPT-3 | 1 | $0.61$ | $118$
Volition | GPT-3 | 3 | $0.77$ | $168$
GPT-J | 0 | $0.69$ | $111$
T0 | 0 | $0.60$ | $168$
Table 1: Accuracy on change-of-state and volition for models significantly
above random baseline.

For change of state, only GPT-3 performs above chance, while for volition,
GPT-3, GPT-J, and T0 perform above chance. T0’s lower performance is
surprising, as the performance of T0 in Fig. 4 is more consistent with an
role-based heuristic. In other words, we do not find that models performing
well on both passive and active control perform well on SPRL. However, these
are separate tasks – thus, it is possible for GPT-3 and GPT-J to be consistent
with non-role-based heuristics in one task while still encoding information
about agent and patient properties. Finally, we note that in both Fig. 4 and
Fig. 5, GPT-3 performs well on subject control and object control in the
active, which is consistent with it containing information on agency and
patienthood.

##  6 Related Work

Following the advent of pretrained language models there has been an explosion
of work examining what kinds of linguistic knowledge such models contain.
Rogers et al. (2020) provide a comprehensive survey of probing work on BERT,
covering probing for syntactic, semantic, and world knowledge. This probing
work generally makes use of linear classifiers on top of frozen
representations, tuned on a training set (Adi et al., 2016; Hupkes et al.,
2018; Hewitt and Manning, 2019).

In contrast, we follow more recent work in probing large generative models
using cloze-style prompts and relatively open generation (Schick and Schütze,
2021). Such models (generative and non-generative) have been probed for
diverse knowledge, including syntax (Futrell et al., 2019), symbolic reasoning
(Talmor et al., 2020), and common-sense knowledge (Petroni et al., 2019;
Kassner and Schütze, 2020; Sakaguchi et al., 2020). This has often been done
by recasting benchmark datasets into text, either with zero examples (Sanh et
al., 2021) or in the form of prompt hacking (Brown et al., 2020; Raffel et
al., 2020). Ettinger (2020) present a suite of comparisons between pretrained
language models and psycholinguistic experiments.

Language models can be sensitive to the format of a prompt – in order to
improve extraction of relational knowledge from large language models, Jiang
et al. (2020) propose automatic methods for mining new prompts and
paraphrasing existing prompts. Similarly, Qin and Eisner (2021) propose a
method for gradient-based prompt optimization, and Shin et al. (2020) propose
a gradient-based search for prompt token replacement. As our experiments
require a specific prompt syntax, we choose to instead run prompts across
different instruction styles and name-verb combinations. Large language models
are also sensitive to the frequency of terms in their training data; Razeghi
et al. (2022) show a strong correlation between frequency of a term in a
corpus and performance on tasks requiring that term. This further hightlights
how surprising it is that several models perform better on subject control
than object control.

##  7 Conclusion

The results in Fig. 4 indicate that differences between models are not merely
of degree, but of kind, with groups of models following different patterns,
many of which are inconsistent with the dominance of object control in
English. This highlights the pressing need for transparency in the reporting
of model details, and especially of training data, without which it is
impossible to hypothesize _why_ these differences are observed. We also find
that, despite there being no trainable parameters in the few-shot setup of
Fig. 5, the models tested are in many cases predictably sensitive to semantic
role information. Our results in Section 5 suggest that some models appear
able to perform semantic proto-role labeling for volition and change of state
(Table 1), but this ability is not directly tied to the sensitivity to
semantic roles in Fig. 5. In other words, some models contain information on
semantic roles but may not recruit that information in producing answers for
control examples. This leads to an interesting direction of future work in
applying causal mediation analysis (Vig et al., 2020; Elazar et al., 2021) to
control clauses, to disentangle the information present in a model from the
process by which the model produces an output.

##  8 Limitations

Firstly, this work is limited by its focus on English syntax, models, and
examples. Control constructions exist a variety of languages (Landau, 2001);
unfortunately, large pre-trained models currently exist primarily in English
alone. Another limitation is the use of fixed prompts: all models tested were
found to be sensitive to the prompt format, and while a large number of
prompts were explored by varying instructions, names, verbs, and actions,
there may be more optimal prompts for the task. Our work is also limited by
the use of open generation. While open-ended generation allows for more
flexibility than constrained decoding, it introduces the challenge of
interpreting the model outputs, though we do validate the use of open
generation in Section 3.2. We note that both these limitations are also common
in human subject research.

While our results show that model outputs are consistent with simpler
heuristics, some of which are observed in human children, we have attempted to
clearly separate this from any anthropomorphic claims that the models might be
actively “following” such a heuristic. The claim made is that the associations
learned from large-scale pretraining lead to outputs with patterns that can be
described by simple heuristics, and that those heuristics at times differ from
or resemble heuristics seen in human data. We also note that while we do not
make strong commitments to any particular account of human language
acquisition, all such accounts differ substantially from how the models tested
are trained, i.e. on extremely large text-only corpora.

## Acknowledgements

Elias Stengel-Eskin is supported by an NSF Graduate Research Fellowship. We
would like to thank Madelaine Thomas and Kate Sanders for feedback on previous
drafts, as well as the ARR reviewers for their constructive comments and
suggestions.

## References

  * Adi et al. (2016) Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016.  Fine-grained analysis of sentence embeddings using auxiliary prediction tasks.  _arXiv preprint arXiv:1608.04207_.
  * Black et al. (2021) Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Chomsky (1969) C. Chomsky. 1969.  _The Acquisition of Syntax in Children from 5 to 10_.  Research monograph series. MIT Press.
  * Chomsky et al. (1986) Noam Chomsky et al. 1986.  _Barriers_ , volume 13.  MIT Press (MA).
  * Cromer (1970) Richard F Cromer. 1970.  "Children are nice to understand": Surface structure clues for the recovery of a deep structure.  _British Journal of Psychology_ , 61(3):397–408.
  * Dietterich (1998) Thomas G Dietterich. 1998.  Approximate statistical tests for comparing supervised classification learning algorithms.  _Neural computation_ , 10(7):1895–1923.
  * Dowty (1991) David Dowty. 1991.  Thematic proto-roles and argument selection.  _Language_ , 67(3):547–619.
  * Elazar et al. (2021) Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021.  Amnesic probing: Behavioral explanation with amnesic counterfactuals.  _Transactions of the Association for Computational Linguistics_ , 9:160–175.
  * Ettinger (2020) Allyson Ettinger. 2020.  What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models.  _Transactions of the Association for Computational Linguistics_ , 8:34–48.
  * Futrell et al. (2019) Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019.  Neural language models as psycholinguistic subjects: Representations of syntactic state.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 32–42.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.  The Pile: An 800Gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Hewitt and Manning (2019) John Hewitt and Christopher D. Manning. 2019.  A structural probe for finding syntax in word representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hupkes et al. (2018) Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018.  Visualisation and’diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure.  _Journal of Artificial Intelligence Research_ , 61:907–926.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020.  How can we know what language models know?  _Transactions of the Association for Computational Linguistics_ , 8:423–438.
  * Kassner and Schütze (2020) Nora Kassner and Hinrich Schütze. 2020.  Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7811–7818, Online. Association for Computational Linguistics.
  * Landau (2001) Idan Landau. 2001.  _Elements of control: Structure and meaning in infinitival constructions_ , volume 51.  Springer Science & Business Media.
  * Lieber et al. (2021) Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. 2021.  Jurassic-1: Technical details and evaluation.  Technical report, AI21 Labs.
  * Maratsos (1974) Michael P Maratsos. 1974.  How preschool children understand missing complement subjects.  _Child Development_ , pages 700–706.
  * McNemar (1947) Quinn McNemar. 1947.  Note on the sampling error of the difference between correlated proportions or percentages.  _Psychometrika_ , 12(2):153–157.
  * (21) OpenAI.  [link].
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases?  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473.
  * Qin and Eisner (2021) Guanghui Qin and Jason Eisner. 2021.  Learning how to ask: Querying lms with mixtures of soft prompts.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5203–5212.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Razeghi et al. (2022) Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022.  Impact of pretraining term frequencies on few-shot reasoning.  _arXiv preprint arXiv:2202.07206_.
  * Reisinger et al. (2015) Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015.  Semantic proto-roles.  _Transactions of the Association for Computational Linguistics_ , 3:475–488.
  * Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020.  A primer in bertology: What we know about how bert works.  _Transactions of the Association for Computational Linguistics_ , 8:842–866.
  * Sakaguchi et al. (2020) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020.  Winogrande: An adversarial winograd schema challenge at scale.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8732–8740.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021.  Multitask prompted training enables zero-shot task generalization.  _arXiv preprint arXiv:2110.08207_.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 255–269.
  * Sharov (2020) S Sharov. 2020.  Know thy corpus! robust methods for digital curation of web corpora.  In _Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020)_. Leeds.
  * Sherman and Lust (1993) Janet Cohen Sherman and Barbara Lust. 1993.  Children are in control.  _Cognition_ , 46(1):1–51.
  * Shin and Van Durme (2021) Richard Shin and Benjamin Van Durme. 2021.  Few-shot semantic parsing with language models trained on code.  _arXiv preprint arXiv:2112.08696_.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4222–4235.
  * Talmor et al. (2020) Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020.  olmpics-on what language model pre-training captures.  _Transactions of the Association for Computational Linguistics_ , 8:743–758.
  * Vig et al. (2020) Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020.  Investigating gender bias in language models using causal mediation analysis.  _Advances in Neural Information Processing Systems_ , 33:12388–12401.
  * White et al. (2020) Aaron Steven White, Elias Stengel-Eskin, Siddharth Vashishtha, Venkata Subrahmanyan Govindarajan, Dee Ann Reisinger, Tim Vieira, Keisuke Sakaguchi, Sheng Zhang, Francis Ferraro, Rachel Rudinger, et al. 2020.  The universal decompositional semantics dataset and decomp toolkit.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 5698–5707.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yu et al. (2020) Charles Yu, Ryan Sie, Nicolas Tedeschi, and Leon Bergen. 2020.  Word frequency does not predict grammatical knowledge in language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4040–4054, Online. Association for Computational Linguistics.

##  Appendix A Profession Results

Figure 7: Accuracy of long instruction template on names. Figure 8: Accuracy
of long instruction template on professions. Performance follows similar
trends to comparable results with names (Fig. 7).

##  Appendix B Data

###  B.1 Licensing

All data and code is released under an MIT license.